Automatic .appup file generation

My talk at the Erlang Factory covered several topics, but mostly people were only interested in the last 5 minutes, where I talk about how we automatically generate appup files when we are installing our software. (Makes you wonder how to do a minimum viable talk).

I created a direct (escript) interface to the automatic appup generation that doesn't require you use erlrc to manage your nodes (although, you do need erlrc installed so the escript can access the magic functions). The idea is you invoke this at the right time during whatever installation procedure you are using, and redirect the output to create the appup file (e.g., either when building a new release from an old release, or "just in time" when you are installing a new application). When things go right, it looks something like this:

% ./erlrc-makeappup erlrc 0.1.2 0.2.0 /usr/lib/erlang/lib/erlrc-0.1.2 /Users/pmineiro/tmp/usr/lib/erlang/lib/erlrc-0.2.0
{"0.2.0",
[{"0.1.2",
[{load_module,erlrc},
{load_module,erlrc_boot},
{load_module,erlrc_lib},
{load_module,erlrcdynamic},
{update,erlrcsup,supervisor},
{load_module,erlrcsupnotify},
{load_module,release_handler_1}]}],
[{"0.1.2",
[{load_module,release_handler_1},
{load_module,erlrcsupnotify},
{update,erlrcsup,supervisor},
{load_module,erlrcdynamic},
{load_module,erlrc_lib},
{load_module,erlrc_boot},
{load_module,erlrc}]}]}

That's the appup file that erlrc generates for itself when it is being upgraded from version 0.1.2 to 0.2.0. The (surprisingly simple) appup generation algorithm is outlined in the documentation.

This is available starting in version 0.2.0 of erlrc which is available on google code.

The Need for Language Specific Packaging

We use apt to manage all our software at our startup, and we like being able to manage components from a heterogeneous set of languages in a uniform way. In other words, some of our software is in Erlang, some is not, and the apt (plus erlrc) lets us treat them similarly. One might guess that I would be against an Erlang-specific packaging solution such as erlware.

Not so! We need erlware and I love it.

At Yahoo we had a proprietary package format that nobody outside Yahoo used. However we could algorithmically convert any CPAN package to our internal package format, so in effect, we had access to all of CPAN. Analogously, companies might use rpm or deb or something else entirely; but by releasing all Erlang software in a standard form, it should be possible to transduce. Right now, when we get stuff from the universe, we end up repackaging them for debian and placing them into our private package archive. Everybody does things slightly differently so it's difficult to generalize this process.

Furthermore, there are exciting benefits to having a single place to look for open-source Erlang software which enforces project quality standards and provides a cross-platform way to start playing around with Erlang.

erlrc erlang factory talk

I'll be giving a talk about erlrc at the Erlang Factory. erlrc is our scheme for integrating the Erlang VM with the OS package manager (we use deb, but it should be possible to integrate with rpm and others as well).

The google code project contains the powerpoint slides as well as other useful documentation.

In the talk I indicate that erlrc is componentized for embedding in other build systems and launch processes, and I also mention that we built our own automake-based build system (which is used for all the languages we develop in, not just Erlang). That system is called framewerk and is also available from google code.

I also claim that once your entire software state is represented in OS packages, simple and effective strategies for deployment and provisioning emerge. Two simple tools that we've developed along these lines are ec2-do and apt-snapshot.

ec2-do is an escript that provides rolling window execution of an arbitrary command across a set of ec2 machines (typically, an ec2 group). When we want to launch we do something like this to sync all the machines with the package archive,

ec2-do -g production -m 1000 -b -d apt-get update

and then something like this to install the package

ec2-do -g production -m 3 -b -d apt-get install <newpackage>

ec2-do is available from the erlrc downloads page.

apt-snapshot is even simpler than ec2-do (just a shell script), but very useful for provisioning when using deb packages. It can associate the current package state of a server with a symbolic tag which can later be restored (possibly from another host). To create a tag we do something like

apt-snapshot create <tag>

and to restore it later we do

apt-snapshot restore <host>:<tag>

We use this in three distinct ways. First, we have a cron job creating regular snapshots on every machine with timestamp tag names in case of snafu (to allow us to rollback). Second, when we launch something particularly risky we create a symbolic tag manually for rollback. Third, when we start up new machines, we get them in the correct state quickly by tagging one of the currently running machines and then telling the new machines to restore that tag from that host (our EC2 images are sensitive to the extra arguments passed to ec2-run-instance, but you can also just use ec2-do for this purpose, since restoring a tag is idempotent).

apt-snapshot is available from the erlrc downloads page.