Nodefinder improvements

Matthew Dempsky from Mochi mailed me some suggestions for nodefinder. Now the multicast_ttl is an application parameter so multicast routing can be used; separate send and receive sockets are employed to ensure the right source address on BSD; and the discovery packet is signed using the cookie as a secret which prevents (hopeless but annoyingly logged) connect attempts to nodes with different cookies and also prevents a list_to_atom flood from packets that happen to match.

Thanks Matt!

network partition ... oops

well we had our first network partition on ec2 yesterday evening. we think it was amazon doing maintenance because it happened in both our production and development clusters.

it turns out things went pretty well considering we've done nothing yet to address network partitioning. two islands formed, but never tried to join back up (it turns out we only try to connect nodes when schemafinder starts). after a while each island's schemafinder decided that the other island was dead and removed it from the schema. so we ended up with two disjoint mnesia instances. they each had complete data so they were able to serve. of course, they were also creating data and diverging over time. however right now we're not doing anything that important so it didn't really matter. we just picked one island to survive and restarted the other ones after nuking their mnesia directories.

however the experience has convinced us that we need to prioritize up our network partition recovery strategy. the good news is that we have an out-of-band source of truth about what the set of nodes should be (via ec2-describe-instances), so it should be possible for us to detect a network partition has occurred. the other good news is that we care more about high availability than transactional integrity so we'll be willing to set master nodes on tables based upon simple heuristics.

walkenfs improvements

I've made some progress towards my goal of making walkenfs usable by those who know nothing about Erlang or Mnesia.

• schemafinder has been open sourced, which provides Mnesia auto-configure.
• mount and umount work (although the mount line is very baroque), so no need to start an Erlang emulator explicitly.
  
• the distribution includes an example init script very close to the one we actually use.
  

Of course walkenfs could still have a showstopping bug in it. In our recent use it was mostly fuse mount options that gave us trouble, and also the shared writeable mmap problem surfaced because rrdtool uses that by default. fortunately with rrdtool, it was a simple configure option to disable mmap. We did have a data corruption scare in development but that turned out to be due to the default redundancy setting being 1 copy (changed that to 3, now that fragmentron can initialize under infeasible conditions). Otherwise, so far so good, knock on wood.