Automatic .app file generation

At our startup, we have our own build system (framewerk) and our deployment framework (erlrc) which play well together. As I learned at the Bay Area Erlang Factory, mostly people have their own processes already, so what they want to extract some of the useful functionality and incorporate it into their way of doing things. This led to exposing automatic .appup file generation from erlrc in a reusable fashion; that technique requires .app files to be correct which is why we automatically generate them in framewerk. To compliment, I've isolated our automatic .app file generation and released it in a standalone form.

The escript is called fwte-makeappfile, and it basically takes a set of Erlang source code which comprise an application and does three things for you: 1) attempts to automatically discover registered processes, 2) attempts to automatically discover all the module names, and 3) attempts to automatically discover the start module for the application (if present). You can override these choices if you don't like them. Here's an example of how it works:

% ./fwte-makeappfile --application nitrogen --description 'Nitrogen Web Framework' --version '0.2009.05.11' ~/src/nitrogen-git/src/**/*.erl
{application,nitrogen,
[{description,"Nitrogen Web Framework"},
{vsn,"0.2009.05.11"},
{modules,[action_add_class,action_alert,action_animate,
action_appear,action_buttonize,action_comet_start,
action_confirm,action_disable_selection,action_effect,
action_event,action_fade,action_hide,
action_jquery_effect,action_remove_class,
action_script,action_show,action_toggle,
action_validate,action_validation_error,element_bind,
element_body,element_br,element_button,
element_checkbox,element_datepicker_textbox,
element_draggable,element_dropdown,element_droppable,
element_file,element_flash,element_google_chart,
element_gravatar,element_h1,element_h2,element_h3,
element_h4,element_hidden,element_hr,element_image,
element_inplace_textbox,element_label,
element_lightbox,element_link,element_list,
element_listitem,element_literal,element_p,
element_panel,element_password,element_placeholder,
element_radio,element_radiogroup,
element_rounded_panel,element_singlerow,
element_sortblock,element_sortitem,element_span,
element_spinner,element_table,element_tablecell,
element_tableheader,element_tablerow,element_template,
element_textarea,element_textbox,element_upload,
element_value,element_windex,element_wizard,mirror,
nitrogen,nitrogen_file,nitrogen_inets_app,
nitrogen_mochiweb_app,nitrogen_project,
nitrogen_yaws_app,sync,validator_confirm_password,
validator_custom,validator_is_email,
validator_is_integer,validator_is_required,
validator_js_custom,validator_max_length,
validator_min_length,web_x,wf,wf_bind,wf_cache,
wf_cache_server,wf_comet,wf_continuation,wf_convert,
wf_counter,wf_email,wf_handle,wf_handle_firstrequest,
wf_handle_postback,wf_handle_postback_multipart,
wf_http_basic_auth,wf_inets,wf_init,wf_mochiweb,
wf_multipart,wf_path,wf_platform,wf_platform_inets,
wf_platform_mochiweb,wf_platform_yaws,wf_query,
wf_redirect,wf_render,wf_script,wf_session,
wf_session_server,wf_session_sup,wf_state,wf_tags,
wf_utils,wf_validation,wf_yaws]},
{registered,[wf_session_server,wf_session_sup]},
{applications,[kernel,stdlib]},
{env,[]}]}
.

We use this in our build system, where the .app file is always automatically generated, but the developer can set overrides if the autodetection is f-ing up. I recommend this as a general strategy: you need to be able to manually specify for edge cases, but you don't want to count on your developers maintaining the routine cases correctly.

Erlang R12B-5 for Fink (Mac OS/X)

It looks like Fink is a bit behind the Erlang releases. Here's an erlang-otp.info file and an erlang-otp.patch file. You can put these in /sw/fink/dists/unstable/main/finkinfo/languages and then do a fink install erlang-otp . I'll ping the maintainer to get these added to Fink.

When you fink install these right now, you'll get prompted because the tarballs are not on the fink mirrors, so you'll have to select "download from original source URL".

Update: if you have pcre installed it can confuse the build. You can use erlang-otp.info.pcre instead. Rename this to erlang-otp.info in the /sw/fink/dists/unstable/main/finkinfo/languages directory.

Automatic .appup file generation

My talk at the Erlang Factory covered several topics, but mostly people were only interested in the last 5 minutes, where I talk about how we automatically generate appup files when we are installing our software. (Makes you wonder how to do a minimum viable talk).

I created a direct (escript) interface to the automatic appup generation that doesn't require you use erlrc to manage your nodes (although, you do need erlrc installed so the escript can access the magic functions). The idea is you invoke this at the right time during whatever installation procedure you are using, and redirect the output to create the appup file (e.g., either when building a new release from an old release, or "just in time" when you are installing a new application). When things go right, it looks something like this:

% ./erlrc-makeappup erlrc 0.1.2 0.2.0 /usr/lib/erlang/lib/erlrc-0.1.2 /Users/pmineiro/tmp/usr/lib/erlang/lib/erlrc-0.2.0
{"0.2.0",
[{"0.1.2",
[{load_module,erlrc},
{load_module,erlrc_boot},
{load_module,erlrc_lib},
{load_module,erlrcdynamic},
{update,erlrcsup,supervisor},
{load_module,erlrcsupnotify},
{load_module,release_handler_1}]}],
[{"0.1.2",
[{load_module,release_handler_1},
{load_module,erlrcsupnotify},
{update,erlrcsup,supervisor},
{load_module,erlrcdynamic},
{load_module,erlrc_lib},
{load_module,erlrc_boot},
{load_module,erlrc}]}]}

That's the appup file that erlrc generates for itself when it is being upgraded from version 0.1.2 to 0.2.0. The (surprisingly simple) appup generation algorithm is outlined in the documentation.

This is available starting in version 0.2.0 of erlrc which is available on google code.

The Need for Language Specific Packaging

We use apt to manage all our software at our startup, and we like being able to manage components from a heterogeneous set of languages in a uniform way. In other words, some of our software is in Erlang, some is not, and the apt (plus erlrc) lets us treat them similarly. One might guess that I would be against an Erlang-specific packaging solution such as erlware.

Not so! We need erlware and I love it.

At Yahoo we had a proprietary package format that nobody outside Yahoo used. However we could algorithmically convert any CPAN package to our internal package format, so in effect, we had access to all of CPAN. Analogously, companies might use rpm or deb or something else entirely; but by releasing all Erlang software in a standard form, it should be possible to transduce. Right now, when we get stuff from the universe, we end up repackaging them for debian and placing them into our private package archive. Everybody does things slightly differently so it's difficult to generalize this process.

Furthermore, there are exciting benefits to having a single place to look for open-source Erlang software which enforces project quality standards and provides a cross-platform way to start playing around with Erlang.