well we had our first network partition on ec2 yesterday evening. we think it was amazon doing maintenance because it happened in both our production and development clusters.
it turns out things went pretty well considering we've done nothing yet to address network partitioning. two islands formed, but never tried to join back up (it turns out we only try to connect nodes when schemafinder starts). after a while each island's schemafinder decided that the other island was dead and removed it from the schema. so we ended up with two disjoint mnesia instances. they each had complete data so they were able to serve. of course, they were also creating data and diverging over time. however right now we're not doing anything that important so it didn't really matter. we just picked one island to survive and restarted the other ones after nuking their mnesia directories.
however the experience has convinced us that we need to prioritize up our network partition recovery strategy. the good news is that we have an out-of-band source of truth about what the set of nodes should be (via ec2-describe-instances), so it should be possible for us to detect a network partition has occurred. the other good news is that we care more about high availability than transactional integrity so we'll be willing to set master nodes on tables based upon simple heuristics.