The Need for Language Specific Packaging

We use apt to manage all our software at our startup, and we like being able to manage components from a heterogeneous set of languages in a uniform way. In other words, some of our software is in Erlang, some is not, and the apt (plus erlrc) lets us treat them similarly. One might guess that I would be against an Erlang-specific packaging solution such as erlware.

Not so! We need erlware and I love it.

At Yahoo we had a proprietary package format that nobody outside Yahoo used. However we could algorithmically convert any CPAN package to our internal package format, so in effect, we had access to all of CPAN. Analogously, companies might use rpm or deb or something else entirely; but by releasing all Erlang software in a standard form, it should be possible to transduce. Right now, when we get stuff from the universe, we end up repackaging them for debian and placing them into our private package archive. Everybody does things slightly differently so it's difficult to generalize this process.

Furthermore, there are exciting benefits to having a single place to look for open-source Erlang software which enforces project quality standards and provides a cross-platform way to start playing around with Erlang.

erlrc erlang factory talk

I'll be giving a talk about erlrc at the Erlang Factory. erlrc is our scheme for integrating the Erlang VM with the OS package manager (we use deb, but it should be possible to integrate with rpm and others as well).

The google code project contains the powerpoint slides as well as other useful documentation.

In the talk I indicate that erlrc is componentized for embedding in other build systems and launch processes, and I also mention that we built our own automake-based build system (which is used for all the languages we develop in, not just Erlang). That system is called framewerk and is also available from google code.

I also claim that once your entire software state is represented in OS packages, simple and effective strategies for deployment and provisioning emerge. Two simple tools that we've developed along these lines are ec2-do and apt-snapshot.

ec2-do is an escript that provides rolling window execution of an arbitrary command across a set of ec2 machines (typically, an ec2 group). When we want to launch we do something like this to sync all the machines with the package archive,

ec2-do -g production -m 1000 -b -d apt-get update

and then something like this to install the package

ec2-do -g production -m 3 -b -d apt-get install <newpackage>

ec2-do is available from the erlrc downloads page.

apt-snapshot is even simpler than ec2-do (just a shell script), but very useful for provisioning when using deb packages. It can associate the current package state of a server with a symbolic tag which can later be restored (possibly from another host). To create a tag we do something like

apt-snapshot create <tag>

and to restore it later we do

apt-snapshot restore <host>:<tag>

We use this in three distinct ways. First, we have a cron job creating regular snapshots on every machine with timestamp tag names in case of snafu (to allow us to rollback). Second, when we launch something particularly risky we create a symbolic tag manually for rollback. Third, when we start up new machines, we get them in the correct state quickly by tagging one of the currently running machines and then telling the new machines to restore that tag from that host (our EC2 images are sensitive to the extra arguments passed to ec2-run-instance, but you can also just use ec2-do for this purpose, since restoring a tag is idempotent).

apt-snapshot is available from the erlrc downloads page.

scaling mnesia with local_content

so we've been spending the last two weeks trying to scale our mnesia application for a whole bunch of new traffic that's about to show up. previously we had to run alot of ec2 instances since we were using ets (RAM-based) storage; once we solved that we started to wonder how many machines we could turn off.

initial indications were disappointing in terms of capacity. none of cpu, network, memory, or disk i/o seemed particularly taxed. for instance, even though raw sequential performance on an ec2 instance can hit 100mb/s, we were unable to hit above 1mb/s of disk utilization. one clue: we did get about 6mb/s of disk utilization when we started a node from scratch. in that case, 100 parallel mnesia_loader processes grab table copies from remote nodes. thus even under an unfavorable access pattern (writing 100 different tables at once), the machines were capable of more.

one problem we suspected was the registered process mnesia_tm, since all table updates go through this process. the mnesia docs do say that it is "primarily intended to be a memory-resident database". so one thought was that mnesia_tm was hanging out waiting for disk i/o to finish and this was introducing latency and lowering throughput; with ets tables, updates are cpu bound so this design would not be so problematic. (we already have tcerl using the async_thread_pool, but that just means the emulator can do something else, not the mnesia_tm process in particular). thus, we added an option to tcerl to not wait for the return value of the linked-in driver before returning from an operation (and therefore not to check for errors). that didn't have much impact on i/o utilization.

we'd long ago purged transactions from the serving path, but we use sync_dirty alot. we thought maybe mnesia_tm was hanging out waiting for acknowledgements from remote mnesia_tm processes and this was introducing latency and lowering throughput. so we tried async_dirty. well that helped, except that under load the message queue length for the mnesia_tm process began to grow and eventually we would have run out of memory.

then we discovered local_content, which causes a table to have the same definition everywhere, but different content. as a side effect, replication is short-circuited. so with very minor code changes we tested this out and saw a significant performance improvement. of course, we couldn't do this to every table we have; only for data for which we were ok with losing if the node went down. however it's neat because now there are several types of data that can be managed within mnesia, in order of expense:

1. transactional data. distributed transactions are extremely expensive, but sometimes necessary.
2. highly available data. when dirty operations are ok, but multiple copies of the data have to be kept, because the data should persist across node failures.
   
3. useful data. dirty operations are ok, and it's ok to lose some of the data if a node fails.

the erlang efficiency guide says "For non-persistent database storage, prefer Ets tables over Mnesia local_content tables.", i.e., bypass mnesia for fastest results. so we might do that, but right now it's convenient to have these tables acting like all our other tables.

interestingly, i/o utilization didn't go up that much even though overall capacity improved alot. we're writing about 1.5 mb/s now to local disks. instead we appear cpu bound now; we don't know why yet.