Our buddy Cooter is going to be giving a talk at OSCON entitled
“Beautiful Concurrency with Erlang"
Congrats cooter.
Monday, April 7, 2008
Wednesday, March 26, 2008
Nodefinder improvements
Matthew Dempsky from Mochi mailed me some suggestions for nodefinder. Now the multicast_ttl is an application parameter so multicast routing can be used; separate send and receive sockets are employed to ensure the right source address on BSD; and the discovery packet is signed using the cookie as a secret which prevents (hopeless but annoyingly logged) connect attempts to nodes with different cookies and also prevents a list_to_atom flood from packets that happen to match.
Thanks Matt!
Thanks Matt!
Friday, March 21, 2008
network partition ... oops
well we had our first network partition on ec2 yesterday evening. we think it was amazon doing maintenance because it happened in both our production and development clusters.
it turns out things went pretty well considering we've done nothing yet to address network partitioning. two islands formed, but never tried to join back up (it turns out we only try to connect nodes when schemafinder starts). after a while each island's schemafinder decided that the other island was dead and removed it from the schema. so we ended up with two disjoint mnesia instances. they each had complete data so they were able to serve. of course, they were also creating data and diverging over time. however right now we're not doing anything that important so it didn't really matter. we just picked one island to survive and restarted the other ones after nuking their mnesia directories.
however the experience has convinced us that we need to prioritize up our network partition recovery strategy. the good news is that we have an out-of-band source of truth about what the set of nodes should be (via ec2-describe-instances), so it should be possible for us to detect a network partition has occurred. the other good news is that we care more about high availability than transactional integrity so we'll be willing to set master nodes on tables based upon simple heuristics.
it turns out things went pretty well considering we've done nothing yet to address network partitioning. two islands formed, but never tried to join back up (it turns out we only try to connect nodes when schemafinder starts). after a while each island's schemafinder decided that the other island was dead and removed it from the schema. so we ended up with two disjoint mnesia instances. they each had complete data so they were able to serve. of course, they were also creating data and diverging over time. however right now we're not doing anything that important so it didn't really matter. we just picked one island to survive and restarted the other ones after nuking their mnesia directories.
however the experience has convinced us that we need to prioritize up our network partition recovery strategy. the good news is that we have an out-of-band source of truth about what the set of nodes should be (via ec2-describe-instances), so it should be possible for us to detect a network partition has occurred. the other good news is that we care more about high availability than transactional integrity so we'll be willing to set master nodes on tables based upon simple heuristics.
Subscribe to:
Posts (Atom)