well we had our first network partition on ec2 yesterday evening. we think it was amazon doing maintenance because it happened in both our production and development clusters.
it turns out things went pretty well considering we've done nothing yet to address network partitioning. two islands formed, but never tried to join back up (it turns out we only try to connect nodes when schemafinder starts). after a while each island's schemafinder decided that the other island was dead and removed it from the schema. so we ended up with two disjoint mnesia instances. they each had complete data so they were able to serve. of course, they were also creating data and diverging over time. however right now we're not doing anything that important so it didn't really matter. we just picked one island to survive and restarted the other ones after nuking their mnesia directories.
however the experience has convinced us that we need to prioritize up our network partition recovery strategy. the good news is that we have an out-of-band source of truth about what the set of nodes should be (via ec2-describe-instances), so it should be possible for us to detect a network partition has occurred. the other good news is that we care more about high availability than transactional integrity so we'll be willing to set master nodes on tables based upon simple heuristics.
Subscribe to:
Post Comments (Atom)
2 comments:
Hi, just wanted to learn a bit more about the nature of the partitioning you saw (if you recall) - was it the case that both islands were reachable from outside EC2, but they couldn't reach each other by their internal IPs? Your description makes it sound like this was the case.
Yes we could reach both islands from outside EC2.
We also created a small OTP application which just logged net_adm;ping/1 times to all the other nodes, and we noticed that between redundancy zones the numbers were more volatile and generally larger.
This led us to put all the mnesia nodes in the same redundancy zone, which made things better.
Post a Comment