Dukes of Erl

Minor Erlang Interface Tricks

2010-01-14T12:02:00.000-08:00

erlrc requires that you drop a file whose name is your node name and whose contents is the node cookie into a particular directory so that the packaging system can find running Erlang VMs and ask them to do hot-code upgrades. It's not that hard, but I figured I would put some shell scripts into the erlrc google code project demonstrating how to do this and other minor tricks that make it a little nicer to talk to an Erlang VM from the (non-Erlang) command line.
All of these scripts are driven by a POSIX shell syntax configuration file describing an Erlang VM, which can be pretty short. Here's one I'm using right now for my personal stuff:

% cat /usr/share/myerlnode/myerlnode.rc
node_name_file='/etc/erlrc.d/nodes/erlang'

run_extra_args='+A 64 -noshell -noinput -s crypto -s mnesia -eval "case erlrc_boot:boot () of ok -> ok; _ -> init:stop () end" >erlang.out 2>erlang.err &'

node_name_file (the only required config setting) defines what the node name will be (via the basename). This config file drives four scripts:

erlstart-run-erlang: starts an Erlang VM. doesn't do that much, really: creates the node file for erlrc, and sets the heart command in case you want to use heart.
erlstart-remsh: starts a remote shell on an Erlang VM.
erlstart-eval: takes the argument, evals it on an Erlang VM, and prints the result to standard out. very useful shell script glue for maintenance scripts.
erlstart-etop: runs etop on an Erlang VM.

If you put your config file in the default location (/etc/erlstart.rc) or export an environment variable indicating the location (ERLSTART_CONFIG_FILE) then things are pretty zero-configuration. Doing this kind of thing at the (non-Erlang) shell gets very addictive:

% erlstart-eval 'application:which_applications()'
[{anwhereos,"Rest API for time series persistence and retrieval.","0.1.0"},
{drurlyjsclientsrv,"Serve the drurly jsclient from the drurly server.",
                "0.0.1"},
{drurly,"Social sharing server.","2.2.0"},
{mcedemo,"TinyMCE + Nitrogen demo.","1.3.0"},
{inets,"INETS  CXC 138 49","5.0.12"},
{mochiweb,"MochiWeb is an Erlang library for building lightweight HTTP servers.",
       "0.2009.05.26"},
{nitrogen,"Nitrogen web framework for Erlang.","0.2009.05.12.3"},
{nitromce,"A Nitrogen element which corresponds to a TinyMCE editor instance.",
       "4.0.1"},
{sgte,"String template language for Erlang.","0.7.1"},
{signzor,"Erlang library to generate signed printable encodings.","0.0.1"},
{tcerl,"Erlang driver for tokyocabinet.","1.3.1h"},
{webmachine,"An Erlang REST framework.","0.2009.09.24b"},
{erlrc,"Extensible application management.","0.2.3"},
{mnesia,"Mnesia storage API extensions.","4.4.7.6.1"},
{crypto,"CRYPTO version 1","1.5.3"},
{sasl,"SASL  CXC 138 11","2.1.5.4"},
{stdlib,"ERTS  CXC 138 10","1.15.5"},
{kernel,"ERTS  CXC 138 10","2.12.5"}]

erlstart-run-erlang is fairly low level, so for an init script I would do something like the following: first, a little stub installed into /etc/rc.d

#! /bin/sh

# chkconfig: 2345 20 80
# description: Control my personal Erlang node.

exec myerlnodectl "$@"

and then the actual guts installed as myerlnodectl

#! /bin/sh

eval_with_main_node () \
{
  erl -name myerlnodetmp$$ \
      -hidden \
      -setcookie "$cookie" \
      -noshell -noinput \
      -eval "MainNode = list_to_atom (\"$1\"), $2" \
      -s erlang halt
}

get_hostname () \
{
  erl -name myerlnodetmp$$ -setcookie $$ -noshell -noinput -eval '
    [ Host ] = tl (string:tokens (atom_to_list (node ()), "@")),
    io:format ("~s~n", [ Host ])
  ' -s init stop
}

id=`basename "$0"`

if test -d /root
then
  HOME=${HOME-/root}
else if test -d /var/root
  then
    HOME=${HOME-/var/root}
  fi
fi
export HOME

ERLSTART_CONFIG_FILE=${ERLSTART_CONFIG_FILE-/usr/share/myerlnode/myerlnode.rc}
export ERLSTART_CONFIG_FILE

. "$ERLSTART_CONFIG_FILE"

cookie=${cookie-turg}
user=${user-erlang}
hostname=${hostname-`get_hostname`}
node_name=`basename "$node_name_file"`
full_name="$node_name@$hostname"
shutdown_file=${shutdown_file-/var/run/myerlnode.shutting_down}

ERL_CRASH_DUMP=${ERL_CRASH_DUMP-/dev/null}
export ERL_CRASH_DUMP

case ${1-"status"} in
start)
  test "`id -u`" -eq 0 || exec sudo $0 "$@"

  printf "Starting Erlang... "

  if test -f "$node_name_file" &&                                     \
     test true = "`erlstart-eval 'true' 2>/dev/null`" 2>/dev/null
    then
      echo "already started."
      exit 0
    fi

  pid=`eval_with_main_node "$full_name" '
         io:format ("~p", [
           case rpc:call (MainNode, os, getpid, []) of
             { badrpc, _ } -> undefined;
             Pid -> list_to_integer (Pid)
           end ])'`

  test "$pid" -gt 0 2>/dev/null && {
    test -f "$shutdown_file" && {
      oldpid=`cat "$shutdown_file"`
      test "$pid" -eq "$oldpid" && {
        echo "shutdown in progress (pid = '$pid')." 1>&2
        exit 1
      }
    }
  }

  rm -f "$shutdown_file"

  # check for -s shell support
  su -l -s /bin/sh $user -c true >/dev/null 2>/dev/null

  if test $? = 0
    then
      dashs="-s /bin/sh"
    else
      dashs=""
    fi

  ${niceness+ nice -n $niceness}                                      \
  su -l $dashs "$user" -c                                             \
    "env cookie=\"$cookie\" erlstart-run-erlang \"$ERLSTART_CONFIG_FILE\""

  eval_with_main_node "$full_name"                                    \
      "Wait = fun (_, 0) ->
                    failed;
                  (Cont, Max) ->
                    case net_adm:ping (MainNode) of
                      pong ->
                        ok;
                      pang ->
                        timer:sleep (100),
                        Cont (Cont, Max - 1)
                    end
              end,
       DontTellMe = 100,
       ok = Wait (Wait, DontTellMe)" || {
    echo "" 1>&2
    echo "$id: could not connect to node after 10 seconds" 1>&2
    exit 1
  }

  eval_with_main_node "$full_name"                                    \
      "Wait = fun (_, 0) ->
                    failed;
                  (Cont, Max) ->
                    case rpc:call (MainNode, init, get_status, []) of
                      { started, _ } ->
                        ok;
                      { starting, _ } ->
                        timer:sleep (100),
                        Cont (Cont, Max - 1);
                      { Status, _ } ->
                        { failed, Status }
                    end
              end,
       DontTellMe = 100,
       ok = Wait (Wait, DontTellMe)" || {
    echo "" 1>&2
    echo "$id: node did not boot after 10 seconds" 1>&2
    exit 1
  }

echo "done."
;;

stop)
  test "`id -u`" -eq 0 || exec sudo $0 "$@"

  printf "Stopping Erlang... "

  if test ! -f "$node_name_file" ||                                   \
     test true != "`erlstart-eval 'true' 2>/dev/null`" 2>/dev/null
    then
      echo "not running."
      exit 0
    fi

  pid=`erlstart-eval 'os:getpid ()' 2>/dev/null`

  test "$pid" -gt 0 2>/dev/null && {
    printf '%s' $pid > "$shutdown_file"
    chown $user:$user "$shutdown_file"
  }

  erlstart-eval 'init:stop ()' >/dev/null 2>/dev/null

  eval_with_main_node "$full_name"                                    \
      "Wait = fun (_, 0) ->
                    failed;
                  (Cont, Max) ->
                    case net_adm:ping (MainNode) of
                      pong ->
                        timer:sleep (100),
                        Cont (Cont, Max - 1);
                      pang ->
                        ok
                    end
              end,
       DontTellMe = 100,
       ok = Wait (Wait, DontTellMe)" || {
    echo "" 1>&2
    echo "$id: node still responsive after 100 seconds" 1>&2
    exit 1
  }

  rm -f "$node_name_file"

  echo "done."
  ;;

status)
  if test -f "$node_name_file" &&                                     \
     test true = "`erlstart-eval 'true' 2>/dev/null`" 2>/dev/null
    then
      echo "Erlang is running"
    else
      echo "Erlang is not running"
    fi

  ;;

*)
  echo "$id: unknown command $1" 1>&2
  exit 1
esac

exit 0

Hopefully you found that inspirational.

erlrc and rpm

2009-12-23T10:57:00.000-08:00

Two of the Dukes now work at OpenX, which is starting to dip its toe into the Erlang waters. They use CentOS so they agreed to fund porting framewerk and erlrc to rpm; previously I'd only used them with deb.

A bit of background: erlrc is a set of Erlang modules and shell scripts that are designed to be integrated into packaging hooks so that installation, upgrade, or removal of a package causes corresponding hot-code activity to happen inside registered Erlang VMs on the box. Since erlrc was designed to easily integrate with multiple package managers, getting it to work with rpm was mostly about me understanding rpm's package hooks model. The result is an experience like this,


% sudo yum -q -y install lyet
erlrc-start: Starting 'lyet': (erlang) started
% sudo yum -q -y remove lyet
erlrc-stop: Stopping 'lyet': (erlang) unloaded

i.e., installing an rpm causes a running Erlang VM on the box to hot-code load the new modules and start the associated application, and removing the rpm causes the associated application to be stopped and the corresponding modules to be hot-code unloaded.

If you use fw-template-erlang than the appropriate packaging hooks are added for you automatically, both for deb and now rpm. However even manual creation of rpm spec files is pretty easy:

erlrc-stop is called in %preun if the installation count indicates removal
erlrc-upgrade is called in %preun if the installation count indicates upgrade
erlrc-start is called in %posttrans

Also, the erlrc shell scripts want to know the previously installed version, so I call rpm -q in a %pretrans hook and save the result. Longer term, erlrc should probably ask the Erlang VM it is talking to what version is running to eliminate the need for this argument (I was a bit surprised that rpm doesn't provide this argument to the package hook like debian does; it seems very useful for creating version-specific upgrade fixes).

Linear Programming with Erlang

2009-10-06T16:03:00.001-07:00

So you have to solve a linear program, so naturally the first language you think of is Erlang. Actually, it's not a natural first choice for most people, but if you are solving a linear program as part of an automatic control strategy for an internet facing application, the choice is better motivated.

Since I faced this situation recently I wrote a binding for GLPK for Erlang. Writing port drivers is a drag so I actually wrote a program to generate the C and Erlang for me. Perhaps with the new FFI these sort of games will not be necessary, but I was happy with the approach because I anticipate experimenting with several linear programming packages, which should be much easier to accommodate.

Available at Google code.

Removing warnings from ct_expand

2009-09-10T10:52:00.000-07:00

I use ct_expand alot (can you tell?). The released version outputs warnings about unused variables, which are harmless, except ... that it conditioned me to ignore unused variable warnings from the compiler. Then the other day I had a real bug which the compiler was warning me about (via an unused variable) that took me quite a bit of time to find. Lesson learned: make sure that a correct compile is as quiet as possible, so problems stand out.

Ulf gave me some pointers on how to prevent ct_expand from emitting these warnings. Basically, ensure that the temporary variables created by ct_expand are prefixed with an underbar. Here's a patch:


--- src/ct_expand.erl   5 Jun 2009 21:07:32 -0000       1.1
+++ src/ct_expand.erl   10 Sep 2009 04:12:07 -0000
@@ -139,6 +139,9 @@
                  {Fname, Arity} = erl_syntax_lib:analyze_function(Form),
                  VarNames = erl_syntax_lib:new_variable_names(
                               Arity,
+                               fun (N) ->
+                                 list_to_atom ("_V" ++ integer_to_list (N))
+                               end,
                               erl_syntax_lib:variables(Form)),
                  {Form, [{function, Fname},
                          {arity, Arity},

Application Variables during Upgrade

2009-09-08T11:33:00.000-07:00

A quick note related to my previous post: it turns out the application controller gives "old values" for application variables during the code upgrade process. Therefore the call to


element (2, application:get_key (drurly, modules))

returns the old list of modules, i.e., is missing new modules. ct_expand to the rescue! I get the list of modules at compile time from the application specification via


  Modules = 
    ct_expand:term (
      begin
        { ok, [ { application, drurly, PropList } ] } = 
          file:consult ("drurly.app"),
        case proplists:get_value (modules, PropList) of
          Y when is_list (Y) -> Y
        end
      end
    ),

For this to work, you must ensure the application specification is generated prior to compiling the module with this function in it.

Dynamically Loading Webmachine Resources

2009-08-26T21:19:00.001-07:00

I've been using webmachine lately, which is fabulous for REST server development. It has a modular concept of resources and the design could be summarized as a "RESTlet container". Because Erlang has hot code loading I was interested in doing this dynamically; in other words, I want to write a module which provides a webmachine resource, hot code load it into the system, and have webmachine start dispatching requests to it.

The good news is, you can change the dispatch list at any time, via

application:set_env (webmachine, dispatch_list, NewDispatchList).

This does not upset webmachine and it will see the change immediately. So for a single application leveraging webmachine, this seemed as simple as having a code_change handler execute something like

application:set_env
  (webmachine,
   dispatch_list,
   lists:foldl (fun (Mod, Acc) ->
                  case catch Mod:dispatch_rules () of
                    { 'EXIT', _ } -> Acc;
                    X -> X ++ Acc 
                  end
                end,
                [],
                element (2, application:get_key (drurly, modules)))).

(drurly is the name of the application in this case). Now my modules export a function like


dispatch_rules () ->
  [ { [ "clip" ], ?MODULE, [] },
    { [ "clip", id ], ?MODULE, [] }
  ].

if they are webmachine resources.

The problem is that order is important: the webmachine_dispatch_list is consulted in order, and the first match is executed. To solve this I created a sort function which orders the dispatch rules by "specificity"

path_spec_priority ('*') -> 3;
path_spec_priority (X) when is_atom (X) -> 2;
path_spec_priority (X) when is_list (X) -> 1.

dispatch_specificity ({ PathSpecA, _, _ },
                      { PathSpecB, _, _ }) ->
  case erlang:length (PathSpecA) - erlang:length (PathSpecB) of
    X when X > 0 ->
      true;
    X when X < 0 ->
      false;
    _ ->
      PrioPathSpecA = [ path_spec_priority (X) || X <- PathSpecA ],
      PrioPathSpecB = [ path_spec_priority (X) || X <- PathSpecB ],

      case PrioPathSpecA =< PrioPathSpecB of
        false ->
          false;
        true ->
          FullPathSpecA = [ { path_spec_priority (X), X } || X <- PathSpecA ],
          FullPathSpecB = [ { path_spec_priority (X), X } || X <- PathSpecB ],

          FullPathSpecA =< FullPathSpecB
      end
  end.

Basically:

Longer dispatch paths come first.
If two dispatch paths have equal length, the more specific one comes first, where specificity is defined by examining the elements left-to-right, with
- string literals are most specific
- atoms except '*' are the next most specific
- '*' is the least specific
If two dispatch paths have equal length and specificity, sort by Erlang term order (effectively, break ties arbitrarily)

The code change handler now becomes:

application:set_env
  (webmachine,
   dispatch_list,
   lists:sort
     (fun dispatch_specificity/2,
      lists:foldl (fun (Mod, Acc) ->
                     case catch Mod:dispatch_rules () of
                       { 'EXIT', _ } -> Acc;
                       X -> X ++ Acc 
                     end
                   end,
                   [],
                   element (2, application:get_key (drurly, modules))))).

This has proven sufficient for a single application running webmachine.

For multiple applications that want to run under the same webmachine, I suspect the right way to go is to have a gen_server which

contains webmachine dispatch configuration "fragments" per key, where the key is intended to be the application name;
accepts commands to replace or delete a particular fragment by key;
for any replace or delete, rebuilds the complete dispatch list by concatenating all fragments together and sorting by specificity, and then updates the webmachine application env var.

For multiple applications that want to run under different webmachines, well unfortunately webmachine uses some global application settings under the fixed atom webmachine and thus currently, like the highlander, there can be only one. (In fact, you can only listen on one ip/port combination with webmachine right now. I might have to patch it to accept multiple ip/port combinations to listen to, since a standard trick of mine is to have nginx handle both regular and ssl connections and connect to a different back-end port to indicate whether or not the connection is secure.)

Metaprogramming with ct_expand

2009-08-22T13:21:00.000-07:00

Yesterday I posted about putting arbitrary Erlang terms into HTTP cookies. I suggested that constructing the base64 codec at compile time was an excellent application for ct_expand, but I didn't provide any details. Therefore, here is a follow-up.

ct_expand provides a simple interface to Erlang metaprogramming: evaluating an arbitrary Erlang term at compile time and substituting the results into the source code during compilation. This is especially useful for initializing a data structure at compile time, and in particular, it can be used to construct the forward and inverse maps for the base64 codec. We will specify our codec by providing a list of 64 unique characters, and ct_expand will do the rest. The following code is functionally identical to the termcookie module presented previously.


-module (termcookie2).
-compile ({ parse_transform, ct_expand }).
-export ([ decode/2,
          encode/2 ]).

-define (CODEC, "abcdefghijklmnopqrstuvwxyz"
               "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
               "0123456789.,").

%
% Public
%

decode (Encoded, Secret) when is_binary (Encoded) ->
 <<Signature:28/binary, Payload/binary>> = Encoded,
 Signature = to_base64 (crypto:sha ([ Payload, Secret ])),
 erlang:binary_to_term (from_base64 (Payload)).

encode (Term, Secret) ->
 Payload =
   to_base64
     (erlang:term_to_binary (Term,
                             [ compressed,
                               { minor_version, 1 } ])),
 Signature = to_base64 (crypto:sha ([ Payload, Secret ])),
 <<Signature/binary, Payload/binary>>.

%
% Private
%

to_base64 (Bin) when (8 * byte_size (Bin)) rem 6 =:= 0 ->
 to_base64_padded (Bin);
to_base64 (Bin) when (8 * byte_size (Bin)) rem 6 =:= 2 ->
 to_base64_padded (<<Bin/binary, 0:16>>);
to_base64 (Bin) when (8 * byte_size (Bin)) rem 6 =:= 4 ->
 to_base64_padded (<<Bin/binary, 0:8>>).

to_base64_padded (Bin) ->
 << <<(element (N + 1,
                ct_expand:term (
                  begin
                    64 = length (?CODEC),
                    64 = length (lists:usort (?CODEC)),
                    list_to_tuple (?CODEC)
                  end
                )
               )
      ):8>>
    || <<N:6>> <= Bin >>.

from_base64 (Bin) ->
 << <<(element
         (N + 1,
          ct_expand:term
            (element
              (2,
               lists:foldl
                 (fun (X, { K, T }) ->
                    { K + 1, setelement (X + 1, T, K) }
                  end,
                  { 0, erlang:make_tuple (256, -1) },
                  ?CODEC)
              )
            )
         )
      ):6>>
     || <<N:8>> <= Bin >>.

Some notes:

Lines 6-8 contain the specification of the codec.
Lines 43-44 are compile-time assertions on the codec specification, namely that it consists of 64 unique characters. If you modify the specification to violate these assertions, the module will not compile, although the resulting error message will be nearly unintelligible (try it!).
Line 45 constructs the forward mapping from the specification; the result is a (constant) tuple which is consulted at run time.
Lines 56-65 construct the inverse mapping from the specification; the result is a (constant) tuple which is consulted at run time.

Note you cannot use any functions from the current module inside the ct_expand:term/1 argument, because the current module has not been compiled yet! If you need to do something really complicated and don't like an unwieldy inline expression you can place helper code in a separate module which is compiled first.

The resulting software is easier to maintain than the original version, because if the codec needs to be changed, only the specification is modified. For instance we can replace lines 5-8 with


-define (CHARS, "abcdefghijklmnopqrstuvwxyz"
               "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
               "0123456789.,").
-define (CODEC,
        ct_expand:term (
          fun () ->
            random:seed (1, 2, 3),
            [ X ||
              { _, X } <-
                lists:sort (
                  [ { random:uniform (), Y }
                    || Y <- ?CHARS
                  ]
                )
            ]
          end ()
        )).

which results in a proper codec utilizing a permutation of the original codec. Note the call to random:seed/3 is happening at compile time, and is setting the random seed of the compilation process. Therefore this is a stable codec definition. (Unfortunately, it doesn't really increase the opacity of the encoding scheme; the bits in an encoded Erlang term are highly degenerate so any interested party would be able to deduce the permutation given enough cookies).

Erlang Terms in Cookies

2009-08-21T16:33:00.000-07:00

Erlang is an increasingly popular choice for web development. Such projects tend to heavily leverage HTTP cookies, and because Erlang defines an external format for any Erlang term, it turns out to be very easy to store an arbitrary term with a cookie (within cookie size limitations), which can be a useful trick.

The technique outlined here generates a signed but not encrypted cookie. That means it's fairly simple for anyone possessing one of your cookies to determine the contents, but difficult for them to forge a novel cookie. The latter is typically important for the application, but has additional importance here because we will be calling erlang:binary_to_term/1 on the cookie value and passing arbitrary data to erlang:binary_to_term/1 is a bad idea (for example, this could cause an extremely large memory allocation).

Here's the code:


-module (termcookie).
-export ([ decode/2,
      encode/2 ]).

%
% Public
%

decode (Encoded, Secret) when is_binary (Encoded) ->
<<Signature:28/binary, Payload/binary>> = Encoded,
Signature = to_base64 (crypto:sha ([ Payload, Secret ])),
erlang:binary_to_term (from_base64 (Payload)).

encode (Term, Secret) ->
Payload =
to_base64
 (erlang:term_to_binary (Term,
                         [ compressed,
                           { minor_version, 1 } ])),
Signature = to_base64 (crypto:sha ([ Payload, Secret ])),
<<Signature/binary, Payload/binary>>.

%
% Private
%

to_base64 (Bin) when (8 * byte_size (Bin)) rem 6 =:= 0 ->
to_base64_padded (Bin);
to_base64 (Bin) when (8 * byte_size (Bin)) rem 6 =:= 2 ->
to_base64_padded (<<Bin/binary, 0:16>>);
to_base64 (Bin) when (8 * byte_size (Bin)) rem 6 =:= 4 ->
to_base64_padded (<<Bin/binary, 0:8>>).

to_base64_padded (Bin) ->
<< <<(to_base64_char (N)):8>> || <<N:6>> <= Bin >>.

to_base64_char (N) when N >= 0, N =< 25 -> $a + N;
to_base64_char (N) when N >= 26, N =< 51 -> $A + (N - 26);
to_base64_char (N) when N >= 52, N =< 61 -> $0 + (N - 52);
to_base64_char (62) -> $.;
to_base64_char (63) -> $,.

from_base64 (Bin) ->
<< <<(from_base64_char (N)):6>> || <<N:8>> <= Bin >>.

from_base64_char (N) when N >= $a, N =< $z -> N - $a;
from_base64_char (N) when N >= $A, N =< $Z -> 26 + (N - $A);
from_base64_char (N) when N >= $0, N =< $9 -> 52 + (N - $0);
from_base64_char ($.) -> 62;
from_base64_char ($,) -> 63.

We are taking advantage of the fact that erlang:binary_to_term/1 will ignore extra bytes at the end, which allows us to mindlessly pad for base 64 encoding.

If you really like to squeeze the last few drops of efficiency out of code, you can change those to_base64_char/1 and from_base64_char/1 functions into tuple lookups. If you are extra cool you can use Ulf Wiger's ct_expand parse transform to construct the tuples at compile time from a specified character list.

This code will throw an exception if anything is amiss with the input, including a signature fail.


% erl
Erlang (BEAM) emulator version 5.6.5 [source] [async-threads:0] [kernel-poll:false]

Eshell V5.6.5  (abort with ^G)
1> crypto:start ().
ok
2> termcookie:encode ({ "omg", erlang, rulz }, "wazzup").
<<"nQCwmuMgeK3bTPzBqKDSmSylIciaG2GdAWadB21NzaagzxjSyw5NzaaeCNvSEGaa">>
3> termcookie:decode (termcookie:encode ({ "omg", erlang, rulz }, "wazzup"), "wazzup").
{"omg",erlang,rulz}
4> termcookie:decode (termcookie:encode ({ "omg", erlang, rulz }, "huh"), "wazzup").
** exception error: no match of right hand side value <<"nQCwmuMgeK3bTPzBqKDSmSylIcia">>
 in function  termcookie:decode/2

Implementing a DSL in Erlang

2009-08-08T23:00:00.001-07:00

If you are willing to abide by Erlang syntax, then you can leverage erl_parse, erl_scan, and erl_eval to quickly whip up a domain specific language (DSL). You can manipulate the semantics via a combination of transformations on the parse tree (with the help of erl_syntax) and interception of function calls (which comes with erl_eval).

For instance, suppose we want a domain specific language which is just like Erlang, except that it has destructive assignment. This can be done in three steps: 1) parse the input using the Erlang parser, 2) transform the parse tree so that match expressions are rewritten as a special local function call, and 3) eval the result, intercept the special local function call and implement the destructive assignment by recursive eval.

Here's the code:


-module (dsl).
-export ([ compile/1,
           run/2 ]).

%-========================================================-
%-                   Public                               -
%-========================================================-

compile (String) ->
  { ok, Scanned, _ } = 
    erl_scan:string (maybe_append_dot (String)),
  { ok, Parsed } = erl_parse:parse_exprs (Scanned),
  { dsl, transform (Parsed) }.

run ({ dsl, Parsed }, Bindings) ->
  eval (Parsed, Bindings).

%-========================================================-
%-               eval callbacks                           -
%-========================================================-

local ('_assign', [ Pattern, Expr ], Bindings) ->
  { value, Rhs, NewBindings } = 
    eval ([ Expr ], Bindings),
  MatchExpr = 
    erl_syntax:match_expr (Pattern, 
                           erl_syntax:abstract (Rhs)),
  { value, _Lhs, MatchBindings } = 
    eval ([ erl_syntax:revert (MatchExpr) ],
            erl_eval:new_bindings ()),
  { value, 
    Rhs, 
    destructive_merge (NewBindings, MatchBindings) }.

%-========================================================-
%-                  Private                               -
%-========================================================-

destructive_merge (Bindings, MatchBindings) ->
  lists:foldl
    (fun ({ Key, Val }, Acc) ->
       erl_eval:add_binding 
        (Key, 
         Val, 
         erl_eval:del_binding (Key, Acc))
     end,
     Bindings,
     erl_eval:bindings (MatchBindings)).

eval (Parsed, Bindings) ->
  erl_eval:exprs (Parsed, 
                  Bindings, 
                  { eval, fun local/3 }).

maybe_append_dot (String) ->
  case lists:last (string:strip (String, 
                                 right)) of
    $. -> String;
    _ -> String ++ "."
  end.

postorder (F, Tree) ->
  F (case erl_syntax:subtrees (Tree) of
       [] -> 
         Tree;
       List -> 
         erl_syntax:update_tree 
           (Tree,
            [[ postorder (F, Subtree)
             || Subtree <- Group ]
            || Group <- List ])
     end).

transform (Parsed) ->
  F = fun (Node) ->
        case erl_syntax:type (Node) of
          % match_expr: rewrite as (destructive) assignment
          match_expr ->
            erl_syntax:application 
              (none,
               erl_syntax:atom ("_assign"),
               [ erl_syntax:match_expr_pattern (Node),
                 erl_syntax:match_expr_body (Node) ]);
          _ ->
            Node
        end
      end,

  [ erl_syntax:revert (postorder (F, X)) 
    || X <- Parsed ].

dsl:compile/1 scans and parses the string in the standard way, but transforms the resulting parse tree rewriting match expressions as calls to a fictitious local function _assign/2. This local function call is intercepted by specifying a local function handler to erl_eval:exprs/3. At that point (lines 22-33), the sequence is: 1) evaluate the right hand side using the current bindings to reduce the right hand side to a constant, 2) evaluate a match of the left hand side to the constant expression given an empty set of bindings, and 3) merge the new bindings with the existing bindings, overwriting as necessary. By proceeding in this fashion, we can still error out if we have a structural mismatch, or if an unbound variable is used on a right hand side.


% erl
Erlang (BEAM) emulator version 5.6.5 [source] [async-threads:0] [kernel-poll:false]

Eshell V5.6.5  (abort with ^G)
1> c (dsl).
{ok,dsl}
2> dsl:run (dsl:compile ("A = 7, A = A + 3"), erl_eval:new_bindings ()).
{value,10,[{'A',10}]}
3> dsl:run (dsl:compile ("B = 7, A = A + 3"), erl_eval:new_bindings ()).
** exception error: variable 'A' is unbound
4> dsl:run (dsl:compile ("A = 7, { _, A } = A + 3"), erl_eval:new_bindings ()).
** exception error: no match of right hand side value 10

Ets Ordered Set Select Efficiency

2009-07-26T11:54:00.000-07:00

Ets (and tcerl) ordered_sets have the useful property that range queries on them can be resolved more efficiently than full-table scan. For instance, given the operation


ets:select (Tab, [ { { { foo, 1, '_' }, bar }, [], [ '$_' ] } ])

only the range of keys corresponding to 3-element tuples whose first two elements are 'foo' and 1 will be examined, and this is potentially far less than the entire set of keys.

Recently I've been working on a project that requires a range of partially bound keys. This would correspond to an operation such as


ets:select (Tab, [ { { { foo, '$1', '_' }, bar }, [ { '>=', '$1', 1 }, { '=<', '$1', 5 } ], [ '$_' ] } ])

and so I made the modifications to tcerl to be able to detect match conditions of this form and constrain the query range. I started to wonder if I was reinventing the wheel and whether ets had sophisticated match specification analysis under the hood somewhere. I asked the mailing list but answers were inconclusive so I decided to do some measurements. Using the following module:


-module (etsselect).
-compile (export_all).

populate_ets (Tab, N) ->
 ets:delete_all_objects (Tab),
 lists:foreach (fun (M) ->
                  ets:insert (Tab, { { foo, M div 10, M }, bar })
                end,
                lists:seq (1, N)).

repeat_factor (Size) when Size < 1000 -> 100;
repeat_factor (Size) when Size < 100000 -> 30;
repeat_factor (_) -> 10.

select (MatchSpec, Tab, Repeat) when Repeat > 0 ->
 ets:select (Tab, MatchSpec),
 select (MatchSpec, Tab, Repeat - 1);
select (_, _, _) ->
 ok.

select_time (MatchSpec) ->
 Tab = ets:new (?MODULE, [ ordered_set ]),
 try
   [ { N,
       (fun () ->
          populate_ets (Tab, N),
          { Time, _ } = timer:tc (?MODULE, select, [ MatchSpec, Tab, Repeat ]),
          Time / Repeat
        end) ()
     }
    || X <- lists:seq (1, 9),
       N <- [ trunc (math:pow (5, X)) ],
       Repeat <- [ repeat_factor (N) ]
   ]
 after   
   ets:delete (Tab)      
 end.

I did some timing tests with different match specifications and table sizes. The match specifications were:


[{ '_', [], [ '$_' ] }] % unconstrained
[{{{ foo, 1, '_' }, bar }, [], [ '$_' ] }] % prefix bound
[{{{ foo, '$1', '_' }, bar }, 
   [{ '>=', '$1', 1 }, { '=<', '$1', 1 }],
   [ '$_' ]
 }] % prefix range bound

The results (using R12B5) indicate that the prefix range bound condition is not being optimized by ets.

The tcerl timings indicate the prefix range bound condition is being optimized (they also indicate much worse constant factors then ets).

Note I haven't released these changes yet, but I will soon as 1.3.1h. Versions of tcerl prior to 1.3.1h will do a full table scan in the prefix range bound case.

osmos

2009-07-01T13:59:00.000-07:00

I just released the first version of osmos, a pure Erlang library that provides on-disk ordered set tables which allow thousands of updates per second with ACID properties.

It achieves that update rate using a rather different structure from a traditional B-tree based table (like the ones used by most RDBMSs or provided by DBM libraries like BDB or tokyocabinet): an incremental merge sort with user-defined merge semantics.

Motivation

Ordinarily, the rate of updates to an on-disk table is limited by the need to seek around and update an index in place. With a typical seek time on the order of 10 ms, this makes it challenging to scale past about 100 updates/s. Most strategies for going beyond that involve some kind of partitioning, either over multiple disks, multiple machines, or both.

However, a few key observations^[1] point to a different strategy:

The reason for updating an index on every write is the expectation that reads are much more frequent than writes, so that read efficiency is the dominating factor. But if writes are far more frequent than reads, you can use some kind of lazy updating to delay the work until absolutely necessary, and combine the work from multiple updates.
An extreme example of a write-dominated, lazily updated database is a full-text inverted index for a search engine. To build one, you might typically read in billions of term-document pairs, sort them by term using some form of external merge sort, and then create a highly optimized index before ever handling a single query.
A merge sort can operate continuously, by injecting new records in sorted batches, and then merging the batches as necessary to maintain a set of sorted files with exponentially increasing sizes. And crucially, this kind of incremental merge sort process can allow relatively efficient searches of the data while it's operating, by binary-searching each sorted file. (An example of this is the incremental indexing provided by Lucene.)

This gives you an ordered set table with a slight increase in the cost of a search (with N records, maybe an extra factor of log N). But the cost of a write is tiny, and mostly delayed: about log N future comparisons during merging, and log N future disk writes, but since all the disk writes are sequential, they will be buffered, and writing to the table requires no explicit seeking at all.^[2]

User-defined merging

Things get even more interesting when you let the user control how records are merged. In the osmos model, there is at most one record for any given key; if two records with the same key are encountered during the merge sort, the user's merge function is called to merge the two records into a single record.

The merge function can be any function


  Merge(Key, EarlierValue, LaterValue)  -> MergedValue

that is associative, i.e.,


  Merge(K, Merge(K, V1, V2), V3) =:=
  Merge(K, V1, Merge(K, V2, V3))

for any key K and any consecutive sequence of values V1, V2, V3.

This allows a wide variety of semantics for writing to the table. For example:

If the merge function always returns the later value, then a write replaces any previous value, like an ordinary key-value store.
If the values are integers, and the merge function returns the sum of the two values, then writing to the table acts like transactionally incrementing a counter.

Similarly, you could use any associative function of two numbers; you could apply such a function to each element of a vector of numbers; or you could apply a different function to each element. For example, to keep a minimum, maximum, and average over some set of keys, you could use something like:


  merge(_K, N1, N2)
      when is_number(N1), is_number(N2) ->
    {min(N1, N2), max(N1, N2), N1 + N2, 2};
  merge(_K, N1, {Min2, Max2, Sum2, Count2})
      when is_number(N1) ->
    {min(N1, Min2), max(N1, Max2),
     N1 + Sum2, 1 + Count2};
  merge(_K, {Min1, Max1, Sum1, Count1}, N2)
      when is_number(N2) ->
    {min(Min1, N2), max(Max1, N2),
     Sum1 + N2, Count1 + 1};
  merge(_K, {Min1, Max1, Sum1, Count1},
            {Min2, Max2, Sum2, Count2}) ->
    {min(Min1, Min2), max(Max1, Max2),
     Sum1 + Sum2, Count1 + Count2}.

This lets you write single numbers as values, but read back either {Min, Max, Sum, Count} tuples (if more than one number has been written for a key) or single numbers (if that was the only value written). To do this with an ordinary key-value table and multiple writers would require expensive transactions, but with osmos, operations like this are no more expensive than replacement, but still ACID.

As you can see, keeping statistics for reporting (when the reports are queried infrequently relative to data collection) is one of the killer applications for a merge sort table.

Among the possibilities for even wackier merge operations are:

Always return the earlier value. (What was the first value that occurred for this key?)
Take the union of two sets. (What are all the values that have occurred for this key?)
Take the intersection of two sets. (What values have always occurred for this key?)
Multiply two NxN matrices, e.g., to keep track of a series of rotations applied to a vector in R^N.

Compose a series of arbitrary operations applied to a space of replaceable objects, e.g.:


  merge(_K, _, V) when ?is_value(V) ->
    V;
  merge(_K, V, Op) when ?is_value(V),
                        ?is_operation(Op) ->
    do_operation(Op, V);
  merge(_K, V, Ops) when ?is_value(V),
                         is_list(Ops) ->
    lists:foldl (fun (Op, V) ->
                   do_operation(Op, V)
                 end,
                 V,
                 Ops);
  merge(_K, Op1, Op2) when ?is_operation(Op1),
                           ?is_operation(Op2) ->
    [Op1, Op2];
  merge(_K, Op, Ops) when ?is_operation(Op),
                          is_list(Ops) ->
    [Op | Ops];
  merge(_K, Ops, Op) when is_list(Ops),
                          ?is_operation(Op) ->
    Ops ++ [Op];
  merge(_K, Ops1, Ops2) when is_list(Ops1),
                             is_list(Ops2) ->
    Ops1 ++ Ops2.

The values could be employee records, and the operations could be things like, “change street address to X,” “increase salary by Y%.” Using this pattern, you can get extremely cheap transactional safety for any single-key operation, as long as your merge function implements it.

API

The basic API is quite simple:


  {ok, Table} = osmos:open(Table, [{directory, D}, {format, F}])

to open a table named Table with the format F in the directory D;


  ok = osmos:write(Table, Key, Value)

to write a record to the table;


  case osmos:read(Table, Key) of
    {ok, Value} -> ...;
    not_found   -> ...
  end

to read the record for a key; and


  ok = osmos:close(Table)

to close the table.

You can also iterate over a range of keys in chunks using osmos:select_range/5 and osmos:select_continue/3. The results from a select provide a consistent snapshot of the table, meaning that the results always reflect the contents of the table at the time of the original call to select_range. (In other words, any writes that happen between subsequent calls to select_continue won't affect the results.)

A table format is a record with the following fields:

block_size::integer(): block size of the table in bytes, controlling the size of disk reads (which are always whole blocks), and the fanout of the on-disk search trees.
key_format::#osmos_format{}: on-disk format for keys. (A pair of functions to convert some set of terms to binaries and back.)
key_less::(KeyA, KeyB) -> bool(): comparison function defining the order of keys in the table. Takes two native-format keys, and returns true if the first argument is less than the second argument.
value_format::#osmos_format{}: on-disk format for values.
merge::(Key, EarlierValue, LaterValue) -> MergedValue: the merge function described above.
short_circuit::(Key, Value) -> bool(): function which allows searches of the table to be terminated early (short-circuited) if it can be determined from a record that any earlier records with the same key are irrelevant.
delete::(Key, Value) -> bool(): function controlling when records are deleted from the table.

There are several pre-canned formats available from the function osmos_table_format:new/3, or you can build your own #osmos_table_format{} record as needed.

Performance

The file tests/osmos_benchmark_1.erl in the source distribution contains a benchmark that uses variable-length binaries as keys (some more frequent than others, with an average length of 15 bytes), and nonnegative integers as values, encoded in 64 bits, where merging takes the sum of the two values. One process writes random keys and values as fast as it can, while another process reads random keys with a 10 ms sleep between reads.

I ran the benchmark for 15 minutes on a 2.2 GHz dual-core MacBook, and got the following numbers:

5028735 total records written, for an average of 5587 writes/second (including the time to compute random keys, etc.)
an average of 109.8 microseconds per write call, which would mean a theoretical maximum write rate of 9111 writes/second (for this table format, machine, etc.)
the median time per write call was 30 microseconds, and the 90th percentile was 54 microseconds, indicating that the vast majority of writes are extremely quick
an average of 1900 microseconds (1.9 ms) per read call
the median time per read call was 979 microseconds, and the 90th percentile was 2567 microseconds

I reran the benchmark for 2 hours on the same machine, and got the following numbers:

17247676 total records written, for an average of 2396 writes/second
an average of 329.7 microseconds per write call, for a theoretical maximum write rate of 3033 writes/second
the median time per write call was 39 microseconds, and the 90th percentile was 88 microseconds
an average of 13076 microseconds (13 ms) per read call
the median time per read call was 6081 microseconds, and the 90th percentile was 34094 microseconds

The table had about 400 MB of data on the disk (in 7 files) at the end of the 2-hour run. This shows that read performance does start to suffer a bit as the amount of data on the disk grows, but writes remain very fast. (In fact, if there were no reads competing with writes, I wouldn't expect the write times to degrade even that much, since all that's happening synchronously during a write call is a buffered write to a journal file, and an insert into an in-memory tree.)

[1] I have to thank my good friend Dave Benson for introducing me to these ideas, and the generalized merge sort table. His excellent library GSK provides a very similar table API (GskTable) for people who want to write single-threaded servers in C. (I ripped off several of his design ideas, including prefix compression and the variable-length integer encoding.)
[2] Of course there may be implicit seeking due to the need to access multiple files at the same time. But for sequential access, the OS and disk hardware can mitigate this to a large degree as long as the number of files isn't too large.

Keeping it simple with flatula

2009-06-17T15:53:00.000-07:00

Paul has blogged about overcoming mnesia performance issues in the past, but I don't think we've talked much about the ultimate strategy -- keeping data out of mnesia altogether.

When we first started serving ads, we stored information about every single ad impression in a huge mnesia database, for retrieval on click, and for building behavioral profiles. Almost needless to say, this didn't scale very far. We spent many a day last summer delving into mnesia internals, fixing corrupted table fragments after node crashes, bemoaning how long it took new nodes to join the schema under heavy load, and so on.

One of the simplest and most effective changes that got us out of this mess was not to store any per-impression data in mnesia at all -- instead, we started logging the data to flat files on disk, and storing a small pointer to the data in a cookie so we could read it back the next time we saw the user. Hardly a revolutionary solution . . . it's well-known that disk seeking is the enemy of performance. The hardest part was coming to realizations like, "Hmm, I guess we don't really care if a node goes down and we lose part of that data!"

We've open-sourced one of the main components that enabled this strategy: flatula, an Erlang application that manages write-once "tables" that are really just collections of flat files. It looks a bit like dets, except that it doesn't support deletions, updates, or iteration, and you can't make up the keys. But when you don't need those things, it's hard to imagine a more efficient way to store data.

If you'd like to learn more, there's a brief tutorial on the Google Code site.

Let parse transform

2009-06-10T15:31:00.001-07:00

So the problem of intermediate variable naming came up again on erlang questions.

Subject:Versioned variable names
From: Attila Rajmund Nohl
Date: Tue, 9 Jun 2009 17:12:34 +0200

Hello!

I think there wasn't any grumbling this month about the
immutable local variables in Erlang, so here's real world
code I've found just today:


% Take away underscore and replace with hyphen
MO1 = re:replace(MO, "_", "-", [{return, list}, global]),
MO2 = toupper(MO1),
% Replace zeros
MO3 = re:replace(MO2,
                 "RX0",
                 "RXO",
                 [{return, list}, global]),
% Insert hyphen if missing
MO5 = case re:run(MO3, "-", [{capture, none}]) of
  nomatch ->
    insert_hyphen(MO3);
  match ->
MO3
end,

...

Mikael Pettersson pointed out that this really has less to do with immutable local variables and more to do with the lack of a let expression. That was insightful, and since a let expression can be considered syntactic sugar for a lambda expression, I realized that a parse transform could provide let like functionality. Let is a reserved keyword in Erlang so I used lyet instead.

Essentially the parse transform rewrites

lyet:lyet (A = B, C)

(fun (A) -> C end) (B)

so the above code could be rewritten as


Result = lyet:lyet (
  % Take away underscore and replace with hyphen
  MO = re:replace(MO, "_", "-", [{return, list}, global]),
  MO = toupper(MO),
  % Replace zeros
  MO = re:replace(MO,
                  "RX0",
                  "RXO",
                  [{return, list}, global]),
  % Insert hyphen if missing
  case re:run(MO, "-", [{capture, none}]) of
    nomatch ->
      insert_hyphen(MO);
    match ->
      MO
  end),

You must provide at least one argument to lyet:lyet. All but the last argument to lyet:lyet must be an assignment, and the last argument has to be a single expression (but you can use begin and end for a block of expressions inside the lyet). As you can see above, you can reuse a variable name across the assignment arguments to lyet:lyet. You can even use lyet:lyet on the right hand side of the assignments, or as part of the expression argument. Some examples of usage are present in the unit test.

Update: per Ulf's suggestion, the parse transform also recognizes the local call let_ in addition to the remote call lyet:lyet. It definitely looks nicer with let_.

The software is available on Google code.

Automatic .app file generation

2009-05-19T15:35:00.000-07:00

At our startup, we have our own build system (framewerk) and our deployment framework (erlrc) which play well together. As I learned at the Bay Area Erlang Factory, mostly people have their own processes already, so what they want to extract some of the useful functionality and incorporate it into their way of doing things. This led to exposing automatic .appup file generation from erlrc in a reusable fashion; that technique requires .app files to be correct which is why we automatically generate them in framewerk. To compliment, I've isolated our automatic .app file generation and released it in a standalone form.

The escript is called fwte-makeappfile, and it basically takes a set of Erlang source code which comprise an application and does three things for you: 1) attempts to automatically discover registered processes, 2) attempts to automatically discover all the module names, and 3) attempts to automatically discover the start module for the application (if present). You can override these choices if you don't like them. Here's an example of how it works:

% ./fwte-makeappfile --application nitrogen --description 'Nitrogen Web Framework' --version '0.2009.05.11' ~/src/nitrogen-git/src/**/*.erl
{application,nitrogen,
[{description,"Nitrogen Web Framework"},
{vsn,"0.2009.05.11"},
{modules,[action_add_class,action_alert,action_animate,
action_appear,action_buttonize,action_comet_start,
action_confirm,action_disable_selection,action_effect,
action_event,action_fade,action_hide,
action_jquery_effect,action_remove_class,
action_script,action_show,action_toggle,
action_validate,action_validation_error,element_bind,
element_body,element_br,element_button,
element_checkbox,element_datepicker_textbox,
element_draggable,element_dropdown,element_droppable,
element_file,element_flash,element_google_chart,
element_gravatar,element_h1,element_h2,element_h3,
element_h4,element_hidden,element_hr,element_image,
element_inplace_textbox,element_label,
element_lightbox,element_link,element_list,
element_listitem,element_literal,element_p,
element_panel,element_password,element_placeholder,
element_radio,element_radiogroup,
element_rounded_panel,element_singlerow,
element_sortblock,element_sortitem,element_span,
element_spinner,element_table,element_tablecell,
element_tableheader,element_tablerow,element_template,
element_textarea,element_textbox,element_upload,
element_value,element_windex,element_wizard,mirror,
nitrogen,nitrogen_file,nitrogen_inets_app,
nitrogen_mochiweb_app,nitrogen_project,
nitrogen_yaws_app,sync,validator_confirm_password,
validator_custom,validator_is_email,
validator_is_integer,validator_is_required,
validator_js_custom,validator_max_length,
validator_min_length,web_x,wf,wf_bind,wf_cache,
wf_cache_server,wf_comet,wf_continuation,wf_convert,
wf_counter,wf_email,wf_handle,wf_handle_firstrequest,
wf_handle_postback,wf_handle_postback_multipart,
wf_http_basic_auth,wf_inets,wf_init,wf_mochiweb,
wf_multipart,wf_path,wf_platform,wf_platform_inets,
wf_platform_mochiweb,wf_platform_yaws,wf_query,
wf_redirect,wf_render,wf_script,wf_session,
wf_session_server,wf_session_sup,wf_state,wf_tags,
wf_utils,wf_validation,wf_yaws]},
{registered,[wf_session_server,wf_session_sup]},
{applications,[kernel,stdlib]},
{env,[]}]}
.

We use this in our build system, where the .app file is always automatically generated, but the developer can set overrides if the autodetection is f-ing up. I recommend this as a general strategy: you need to be able to manually specify for edge cases, but you don't want to count on your developers maintaining the routine cases correctly.

Erlang R12B-5 for Fink (Mac OS/X)

2009-05-12T22:56:00.001-07:00

It looks like Fink is a bit behind the Erlang releases. Here's an erlang-otp.info file and an erlang-otp.patch file. You can put these in /sw/fink/dists/unstable/main/finkinfo/languages and then do a fink install erlang-otp . I'll ping the maintainer to get these added to Fink.

When you fink install these right now, you'll get prompted because the tarballs are not on the fink mirrors, so you'll have to select "download from original source URL".

Update: if you have pcre installed it can confuse the build. You can use erlang-otp.info.pcre instead. Rename this to erlang-otp.info in the /sw/fink/dists/unstable/main/finkinfo/languages directory.

Automatic .appup file generation

2009-05-09T22:18:00.000-07:00

My talk at the Erlang Factory covered several topics, but mostly people were only interested in the last 5 minutes, where I talk about how we automatically generate appup files when we are installing our software. (Makes you wonder how to do a minimum viable talk).

I created a direct (escript) interface to the automatic appup generation that doesn't require you use erlrc to manage your nodes (although, you do need erlrc installed so the escript can access the magic functions). The idea is you invoke this at the right time during whatever installation procedure you are using, and redirect the output to create the appup file (e.g., either when building a new release from an old release, or "just in time" when you are installing a new application). When things go right, it looks something like this:

% ./erlrc-makeappup erlrc 0.1.2 0.2.0 /usr/lib/erlang/lib/erlrc-0.1.2 /Users/pmineiro/tmp/usr/lib/erlang/lib/erlrc-0.2.0
{"0.2.0",
[{"0.1.2",
[{load_module,erlrc},
{load_module,erlrc_boot},
{load_module,erlrc_lib},
{load_module,erlrcdynamic},
{update,erlrcsup,supervisor},
{load_module,erlrcsupnotify},
{load_module,release_handler_1}]}],
[{"0.1.2",
[{load_module,release_handler_1},
{load_module,erlrcsupnotify},
{update,erlrcsup,supervisor},
{load_module,erlrcdynamic},
{load_module,erlrc_lib},
{load_module,erlrc_boot},
{load_module,erlrc}]}]}

That's the appup file that erlrc generates for itself when it is being upgraded from version 0.1.2 to 0.2.0. The (surprisingly simple) appup generation algorithm is outlined in the documentation.

This is available starting in version 0.2.0 of erlrc which is available on google code.

The Need for Language Specific Packaging

2009-05-01T10:30:00.000-07:00

We use apt to manage all our software at our startup, and we like being able to manage components from a heterogeneous set of languages in a uniform way. In other words, some of our software is in Erlang, some is not, and the apt (plus erlrc) lets us treat them similarly. One might guess that I would be against an Erlang-specific packaging solution such as erlware.

Not so! We need erlware and I love it.

At Yahoo we had a proprietary package format that nobody outside Yahoo used. However we could algorithmically convert any CPAN package to our internal package format, so in effect, we had access to all of CPAN. Analogously, companies might use rpm or deb or something else entirely; but by releasing all Erlang software in a standard form, it should be possible to transduce. Right now, when we get stuff from the universe, we end up repackaging them for debian and placing them into our private package archive. Everybody does things slightly differently so it's difficult to generalize this process.

Furthermore, there are exciting benefits to having a single place to look for open-source Erlang software which enforces project quality standards and provides a cross-platform way to start playing around with Erlang.

erlrc erlang factory talk

2009-04-29T10:08:00.000-07:00

I'll be giving a talk about erlrc at the Erlang Factory. erlrc is our scheme for integrating the Erlang VM with the OS package manager (we use deb, but it should be possible to integrate with rpm and others as well).

The google code project contains the powerpoint slides as well as other useful documentation.

In the talk I indicate that erlrc is componentized for embedding in other build systems and launch processes, and I also mention that we built our own automake-based build system (which is used for all the languages we develop in, not just Erlang). That system is called framewerk and is also available from google code.

I also claim that once your entire software state is represented in OS packages, simple and effective strategies for deployment and provisioning emerge. Two simple tools that we've developed along these lines are ec2-do and apt-snapshot.

ec2-do is an escript that provides rolling window execution of an arbitrary command across a set of ec2 machines (typically, an ec2 group). When we want to launch we do something like this to sync all the machines with the package archive,

ec2-do -g production -m 1000 -b -d apt-get update

and then something like this to install the package

ec2-do -g production -m 3 -b -d apt-get install <newpackage>

ec2-do is available from the erlrc downloads page .

apt-snapshot is even simpler than ec2-do (just a shell script), but very useful for provisioning when using deb packages. It can associate the current package state of a server with a symbolic tag which can later be restored (possibly from another host). To create a tag we do something like

apt-snapshot create <tag>

and to restore it later we do

apt-snapshot restore <host>:<tag>

We use this in three distinct ways. First, we have a cron job creating regular snapshots on every machine with timestamp tag names in case of snafu (to allow us to rollback). Second, when we launch something particularly risky we create a symbolic tag manually for rollback. Third, when we start up new machines, we get them in the correct state quickly by tagging one of the currently running machines and then telling the new machines to restore that tag from that host (our EC2 images are sensitive to the extra arguments passed to ec2-run-instance, but you can also just use ec2-do for this purpose, since restoring a tag is idempotent).

apt-snapshot is available from the erlrc downloads page .

scaling mnesia with local_content

2008-08-06T18:19:00.001-07:00

so we've been spending the last two weeks trying to scale our mnesia application for a whole bunch of new traffic that's about to show up. previously we had to run alot of ec2 instances since we were using ets (RAM-based) storage; once we solved that we started to wonder how many machines we could turn off.

initial indications were disappointing in terms of capacity. none of cpu, network, memory, or disk i/o seemed particularly taxed. for instance, even though raw sequential performance on an ec2 instance can hit 100mb/s, we were unable to hit above 1mb/s of disk utilization. one clue: we did get about 6mb/s of disk utilization when we started a node from scratch. in that case, 100 parallel mnesia_loader processes grab table copies from remote nodes. thus even under an unfavorable access pattern (writing 100 different tables at once), the machines were capable of more.

one problem we suspected was the registered process mnesia_tm, since all table updates go through this process. the mnesia docs do say that it is "primarily intended to be a memory-resident database". so one thought was that mnesia_tm was hanging out waiting for disk i/o to finish and this was introducing latency and lowering throughput; with ets tables, updates are cpu bound so this design would not be so problematic. (we already have tcerl using the async_thread_pool, but that just means the emulator can do something else, not the mnesia_tm process in particular). thus, we added an option to tcerl to not wait for the return value of the linked-in driver before returning from an operation (and therefore not to check for errors). that didn't have much impact on i/o utilization.

we'd long ago purged transactions from the serving path, but we use sync_dirty alot. we thought maybe mnesia_tm was hanging out waiting for acknowledgements from remote mnesia_tm processes and this was introducing latency and lowering throughput. so we tried async_dirty. well that helped, except that under load the message queue length for the mnesia_tm process began to grow and eventually we would have run out of memory.

then we discovered local_content, which causes a table to have the same definition everywhere, but different content. as a side effect, replication is short-circuited. so with very minor code changes we tested this out and saw a significant performance improvement. of course, we couldn't do this to every table we have; only for data for which we were ok with losing if the node went down. however it's neat because now there are several types of data that can be managed within mnesia, in order of expense:

transactional data. distributed transactions are extremely expensive, but sometimes necessary.
highly available data. when dirty operations are ok, but multiple copies of the data have to be kept, because the data should persist across node failures.
useful data. dirty operations are ok, and it's ok to lose some of the data if a node fails.

the erlang efficiency guide says "For non-persistent database storage, prefer Ets tables over Mnesia local_content tables.", i.e., bypass mnesia for fastest results. so we might do that, but right now it's convenient to have these tables acting like all our other tables.

interestingly, i/o utilization didn't go up that much even though overall capacity improved alot. we're writing about 1.5 mb/s now to local disks. instead we appear cpu bound now; we don't know why yet.

Tokyocabinet and Mnesia

2008-06-23T21:09:00.000-07:00

As Daisy has already indicated, it is now possible to plug arbitrary storage strategies into Mnesia. For those who are familiar with mnesia_access, this is different; mnesia_access only covers reads and writes, not schema manipulations, and has other deficiencies that it render it useless for adding a new storage type in practice (what mnesia_access is great for is changing the semantics of mnesia operations, e.g., mnesia_frag). This project lets you make tables that are essentially indistinguishable from the built-in mnesia table types (ram_copies, disc_copies, disc_only_copies).

Anyway our goal was to get a good on-disk ordered_set table type, since we've found ordered_sets very useful for the kinds of problems we're solving but we're tired of being limited by memory. After looking around for a while Tokyocabinet emerged as our favorite for underlying implementation. We considered BDB and libmysql, but Tokyocabinet seemed simple and faster, and had a more accommodating license. So we ported Tokyocabinet to Erlang and then used the above storage API to connect to Mnesia.

As a side benefit Tokyocabinet might also be preferred to dets even for set-type applications because of the lack of file size limit and high performance. Tokyocabinet actually has a set-type storage strategy that we'd like to define an Erlang Term Store for, but as of this post, the set-type store doesn't support cursor positioning based upon a key, which makes the implementation of next tedious (although not impossible). So I'm waiting on the author to add that call; if that happens, we could have a nicer on-disk set-type table as well.

Everything is available on google code: mnesiaex (storage API) and tcerl (erlang port of Tokyocabinet).

Also Daisy (aka Joel Reymont) is really nice to work with.

generating app files

2008-06-15T11:32:00.000-07:00

We automatically generate our .app files around here from analysis of the source code. However this trick is buried inside fw-template-erlang which most (all?) people outside our group do not use. I thought I'd highlight how it works.

A typical OTP application specification file looks like:


{ application, walkenfs,
[
  { description, "Distributed filesystem." },
  { vsn, "0.1.7" },
  { modules, [ walkenfssup, walkenfsfragmentsrv, walkenfs, walkenfssrv ] },
  { registered, [ walkenfsfragmentsrv ] },
  { applications, [ kernel, stdlib   ] },
  { mod, { walkenfs, [] } },
  { env, [ { linked_in, false }, { mount_opts, "" }, { mount_point, "/walken" }, { prefix, walken }, { read_data_context, async_dirty }, { write_data_context, sync_dirty }, { read_meta_context, transaction }, { write_meta_context, sync_transaction }, { attr_timeout_ms, 0 }, { entry_timeout_ms, 0 }, { reuse_node, true }, { start_timeout, infinity }, { stop_timeout, 1800000 }, { init_fragments, 7 }, { init_copies, 3 }, { copy_type, n_disc_only_copies }, { buggy_ets, auto_detect }, { port_exit_stop, true } ] }

]
}.

There are alot of bits here that can filled in automatically. It helps to think in terms of the template:


{ application, @FW_PACKAGE_NAME@,
[
  { description, "@FW_PACKAGE_SHORT_DESCRIPTION@" },
  { vsn, "@FW_PACKAGE_VERSION@" },
  { modules, [ @FW_ERL_APP_MODULES@ ] },
  { registered, [ @FW_ERL_APP_REGISTERED@ ] },
  { applications, [ kernel, stdlib @FW_ERL_PREREQ_APPLICATIONS@ ] },
  @FW_ERL_APP_MOD_LINE@
  { env, @FW_ERL_APP_ENVIRONMENT@ }
  @FW_ERL_APP_EXTRA@
]
}.

This is an "automake" style template where @VARIABLE@ indicates something that will be substituted.

Now we can say where each of these templates values comes from when using fw-template-erlang:

FW_PACKAGE_NAME: specified by the developer. if you're using automake you've had to specify this already via AC_INIT, so it's good to reuse that rather than require duplicate specification.
FW_PACKAGE_SHORT_DESCRIPTION: specified by the developer.
FW_PACKAGE_VERSION: specified by the developer; like the package name, if you're using automake this information is already contained in the arguments to AC_INIT so reuse that.
FW_ERL_APP_MODULES: generated by analysis of the source code directory.
FW_ERL_APP_REGISTERED: generated by analysis of the source code directory.
FW_ERL_PREREQ_APPLICATIONS: specified by the developer.
FW_ERL_MOD_LINE: partially generated by analysis of the source code directory, partially specified by the developer (see below).
FW_ERL_APP_ENVIRONMENT: specified by the developer.
FW_ERL_APP_EXTRA: specified by the developer. this is for unusual extra directives like included applications.

So the interesting bits come down to FW_ERL_APP_MODULES, FW_ERL_APP_REGISTERED, and FW_ERL_MOD_LINE being generated automatically via inspection of the source code.

First, a note on intervention. Anytime you are attempting to automate a task that was previous done by humans, it's helps to allow a human to override any of the automatic settings that you generate by default. That way, you can be correct only 95% of the time and still be a huge timesaver. In practice we've found that although it easy to come up with code examples that flummox the automatic strategies, such code is never actually written by people in the normal course of their work. In fact, we've found that application files we get from the universe are often missing registered process names that our automatic strategy finds. So empirically we're running at 100% correct so far, but fw-template-erlang still contains the capability to override any automatically generated value.

Now the actual trick to computing all the stuff is that Erlang contains an Erlang compiler as a library. Here's an example of leveraging this to generate all the module names from a set of files; any file that contains a

-fwskip()

directive will be ignored.


-module (find_modules).
-export ([ main/1 ]).

is_skipped ([]) -> false;
is_skipped ([ { attribute, _, fwskip, _ } | _ ]) -> true;
is_skipped ([ _ | Rest ]) -> is_skipped (Rest).

print_module (Dir, F) ->
case compile:file (F, [ binary, 'E', { outdir, Dir } ]) of
{ ok, Mod, { Mod, _, Forms } } ->
case is_skipped (Forms) of
  true ->
    ok;
  false ->
    port_command (get (three), io_lib:format ("~p~n", [ Mod ]))
end;
_ ->
ok
end.

main ([ Dir | Rest ]) ->
ok = file:make_dir (Dir),

try
  Three = open_port ({ fd, 0, 3 }, [ out ]),
% ugh ... don't want to have to change all these function signatures,
% so i'm gonna be dirty
  put (three, Three),
  lists:foreach (fun (F) -> print_module (Dir, F) end, Rest)
after
  { ok, FileNames } = file:list_dir (Dir),
  lists:foreach (fun (F) -> file:delete (Dir ++ "/" ++ F) end, FileNames),
  file:del_dir (Dir)
end;
main ([]) ->
Port = open_port ({ fd, 0, 2 }, [ out ]),
port_command (Port, "usage: find-modules.esc tmpdir filename [filename ...]\n"),
halt (1).

The output is on file descriptor 3 in order to isolate the desired output from various messages being output by the Erlang code. This was originally an escript but to be compatible with older versions of Erlang we changed it to run as follows:


#! /bin/sh

# NB: this script is called find-modules.sh

# unfortunately, escript is a recent innovation ...

ERL_CRASH_DUMP=/dev/null
export ERL_CRASH_DUMP

erl -pa "${FW_ROOT}/share/fw/template/erlang/" -pa "${FW_ROOT}/share/fw.local/template/erlang" -pa "${top_srcdir}/fw/template/erlang/" -pa "${top_srcdir}/fw.local/template/erlang" -noshell -noinput -eval '
find_modules:main (init:get_plain_arguments ()),
halt (0).
' -extra "$@" 3>&1 >/dev/null

In practice this gets executed like


find src -name '*.erl' -print | xargs find-modules.sh "$tmpdir" |
perl -ne 'chomp;
     next if $seen{$_}++;
     print ", " if $f;
     print $_;
     $f = 1;'

Ok, that was the easy one! In fact, the problem of finding all the module names seems so easy that one could be tempted to solve it with grep (hmmm: what about preprocesser directives ?). However the point is that nothing parses Erlang as well as Erlang, so why reinvent the wheel.

Computing the mod line in the application specification is just slightly harder. Basically we scan source code for a module which has a

-behaviour(application)

attribute and assume that is the start module. The Erlang compiler allows behaviour to be (mis)spelt American-style so we allow that as well.


-module (find_start_module).
-export ([ main/1 ]).

is_application ([]) -> false;
is_application ([ { attribute, _, behaviour, [ application ] } | _ ]) -> true;
is_application ([ { attribute, _, behavior, [ application ] } | _ ]) -> true;
is_application ([ _ | Rest ]) -> is_application (Rest).

is_skipped ([]) -> false;
is_skipped ([ { attribute, _, fwskip, _ } | _ ]) -> true;
is_skipped ([ _ | Rest ]) -> is_skipped (Rest).

find_start_module (_, []) ->
ok;
find_start_module (Dir, [ F | Rest ]) ->
case compile:file (F, [ binary, 'E', { outdir, Dir } ]) of
{ ok, Mod, { Mod, _, Forms } } ->
  case is_application (Forms) and not is_skipped (Forms) of
    true ->
      port_command (get (three), io_lib:format ("~p~n", [ Mod ]));
    false ->
      find_start_module (Dir, Rest)
  end;
_ ->
  find_start_module (Dir, Rest)
end.

main ([ Dir | Rest ]) ->
ok = file:make_dir (Dir),

try
Three = open_port ({ fd, 0, 3 }, [ out ]),
% ugh ... don't want to have to change all these function signatures,
% so i'm gonna be dirty
put (three, Three),
find_start_module (Dir, Rest)
after
{ ok, FileNames } = file:list_dir (Dir),
lists:foreach (fun (F) -> file:delete (Dir ++ "/" ++ F) end, FileNames),
file:del_dir (Dir)
end;
main ([]) ->
Port = open_port ({ fd, 0, 2 }, [ out ]),
port_command (Port, "usage: find-modules.esc tmpdir filename [filename ...]\n"),
halt (1).

If there are no applications in the set of files being analyzed, we output nothing for the mod line in the application file, otherwise we output

{ mod, { @FW_ERL_APP_START_MODULE@, @FW_ERL_APP_START_ARGS@ } },

where FW_ERL_APP_START_MODULE is the output of the above command and FW_ERL_APP_START_ARGS is indicated by the programmer (we use [] by default for FW_ERL_APP_START_ARGS and in practice never use the field). Obviously, if multiple modules implementing the application behaviour are found then this will not work. Our style has been to put each application we make in a separate directory to facilitate automatic analysis.

Anyway now we can tackle the more challenging problem of finding registered processes. There are many library calls that end up registering a process name; the ones we recognize are:

supervisor child specs that contain calls of the form

{ _, start, [ { local, xxx }, ... ] } -> xxx being registered
{ _, start_link, [ { local, xxx }, ... ] } -> xxx being registered

function calls of the form

Module:start ({ local, xxx }, ...) -> xxx being registered
Module:start_link ({ local, xxx }, ...) -> xxx being registered

calls to erlang:register (xxx, ...) -> xxx being registered

There's alot of code here but the general idea is the same: walk the forms generated by compile and pattern match on one of the above cases, outputting discovered registered process names to file descriptor 3.


-module (find_registered).
-export ([ main/1 ]).

%% looks like fun_clauses are the same as clauses (?)
print_registered_fun_clauses (Clauses) ->
print_registered_clauses (Clauses).

%% looks like icr_clauses are the same as clauses (?)
print_registered_icr_clauses (Clauses) ->
print_registered_clauses (Clauses).

print_registered_inits (Inits) ->
lists:foreach (fun ({ record_field, _, _, E }) ->
               print_registered_expr (E);
                 (_) ->
               ok
             end,
             Inits).

print_registered_upds (Upds) ->
lists:foreach (fun ({ record_field, _, _, E }) ->
               print_registered_expr (E);
                 (_) ->
               ok
             end,
             Upds).

% hmmm ... pretty sure patterns are not supposed to have side-effects ...
print_registered_pattern (_) -> ok.
print_registered_pattern_group (_) -> ok.

print_registered_quals (Qs) ->
lists:foreach (fun ({ generate, _, P, E }) ->
               print_registered_pattern (P),
               print_registered_expr (E);
                 ({ b_generate, _, P, E }) ->
               print_registered_pattern (P),
               print_registered_expr (E);
                 (E) ->
               print_registered_expr (E)
             end,
             Qs).

print_registered_expr ({ cons, _, H, T }) ->
print_registered_expr (H),
print_registered_expr (T);
print_registered_expr ({ lc, _, E, Qs }) ->
print_registered_expr (E),
print_registered_quals (Qs);
print_registered_expr ({ bc, _, E, Qs }) ->
print_registered_expr (E),
print_registered_quals (Qs);

%% Ok, here's some meat:
%% the "supervisor child spec" rule
%% { _, start, [ { local, xxx }, ... ] } -> xxx being registered
%% { _, start_link, [ { local, xxx }, ... ] } -> xxx being registered
print_registered_expr ({ tuple, _,
                     Exprs=[ _,
                       { atom, _, Func },
                       { cons,
                         _,
                         { tuple, _, [ { atom, _, local },
                                       { atom, _, Name } ] },
                         _ } ] }) when (Func =:= start) or
                                       (Func =:= start_link) ->
port_command (get (three), io_lib:format ("~p~n", [ Name ])),
print_registered_exprs (Exprs);

print_registered_expr ({ tuple, _, Exprs }) ->
print_registered_exprs (Exprs);
print_registered_expr ({ record_index, _, _, E }) ->
print_registered_expr (E);
print_registered_expr ({ record, _, _, Inits }) ->
print_registered_inits (Inits);
print_registered_expr ({ record_field, _, E0, _, E1 }) ->
print_registered_expr (E0),
print_registered_expr (E1);
print_registered_expr ({ record, _, E, _, Upds }) ->
print_registered_expr (E),
print_registered_upds (Upds);
print_registered_expr ({ record_field, _, E0, E1 }) ->
print_registered_expr (E0),
print_registered_expr (E1);
print_registered_expr ({ block, _, Exprs }) ->
print_registered_exprs (Exprs);
print_registered_expr ({ 'if', _, IcrClauses }) ->
print_registered_icr_clauses (IcrClauses);
print_registered_expr ({ 'case', _, E, IcrClauses }) ->
print_registered_expr (E),
print_registered_icr_clauses (IcrClauses);
print_registered_expr ({ 'receive', _, IcrClauses }) ->
print_registered_icr_clauses (IcrClauses);
print_registered_expr ({ 'receive', _, IcrClauses, E, Exprs }) ->
print_registered_icr_clauses (IcrClauses),
print_registered_expr (E),
print_registered_exprs (Exprs);
print_registered_expr ({ 'try', _, Exprs0, IcrClauses0, IcrClauses1, Exprs1 }) ->
print_registered_exprs (Exprs0),
print_registered_icr_clauses (IcrClauses0),
print_registered_icr_clauses (IcrClauses1),
print_registered_exprs (Exprs1);
print_registered_expr ({ 'fun', _, Body }) ->
case Body of
  { clauses, Cs } ->
    print_registered_fun_clauses (Cs);
  _ ->
    ok
end;

%% Ok, here's some meat:
%% Module:start ({ local, xxx }, ...) -> xxx being registered
%% Module:start_link ({ local, xxx }, ...) -> xxx being registered

print_registered_expr ({ call,
                     _,
                     E={ remote, _, _, { atom, _, Func } },
                     Exprs=[ { tuple, _, [ { atom, _, local },
                                           { atom, _, Name } ] } | _ ] })
                        when (Func =:= start) or
                             (Func =:= start_link) ->
port_command (get (three), io_lib:format ("~p~n", [ Name ])),
print_registered_expr (E),
print_registered_exprs (Exprs);

%% Ok, here's some meat:
%% erlang:register (xxx, ...) -> xxx being registered

print_registered_expr ({ call,
                     _,
                     { remote,
                       _,
                       { atom, _, erlang },
                       { atom, _, register } },
                     Exprs=[ { atom, _, Name } | _ ] }) ->
port_command (get (three), io_lib:format ("~p~n", [ Name ])),
print_registered_exprs (Exprs);

print_registered_expr ({ call, _, E, Exprs }) ->
print_registered_expr (E),
print_registered_exprs (Exprs);
print_registered_expr ({ 'catch', _, E }) ->
print_registered_expr (E);
print_registered_expr ({ 'query', _, E }) ->
print_registered_expr (E);
print_registered_expr ({ match, _, P, E }) ->
print_registered_pattern (P),
print_registered_expr (E);
print_registered_expr ({ bin, _, PatternGrp }) ->
print_registered_pattern_group (PatternGrp);
print_registered_expr ({ op, _, _, E }) ->
print_registered_expr (E);
print_registered_expr ({ op, _, _, E0, E1 }) ->
print_registered_expr (E0),
print_registered_expr (E1);
print_registered_expr ({ remote, _, E0, E1 }) ->
print_registered_expr (E0),
print_registered_expr (E1);
print_registered_expr (_) ->
ok.

print_registered_exprs (Exprs) ->
lists:foreach (fun (E) -> print_registered_expr (E) end, Exprs).

print_registered_clauses (Clauses) ->
lists:foreach (fun ({ clause, _, _, _, Exprs }) ->
                 print_registered_exprs (Exprs);
                   (_) ->
                 ok
               end,
               Clauses).

print_registered_forms (Forms) ->
lists:foreach (fun ({ function, _, _, _, Clauses }) ->
                 print_registered_clauses (Clauses);
                   (_) ->
                 ok
               end,
               Forms).

is_skipped ([]) -> false;
is_skipped ([ { attribute, _, fwskip, _ } | _ ]) -> true;
is_skipped ([ _ | Rest ]) -> is_skipped (Rest).

print_registered (Dir, F) ->
case compile:file (F, [ binary, 'E', { outdir, Dir } ]) of
{ ok, _, { _, _, Forms } } ->
  case is_skipped (Forms) of
    true ->
      ok;
    false ->
      print_registered_forms (Forms)
  end;
_ ->
  ok
end.

main ([ Dir | Rest ]) ->
ok = file:make_dir (Dir),

try
Three = open_port ({ fd, 0, 3 }, [ out ]),
% ugh ... don't want to have to change all these function signatures,
% so i'm gonna be dirty
put (three, Three),
lists:foreach (fun (F) -> print_registered (Dir, F) end, Rest)
after
{ ok, FileNames } = file:list_dir (Dir),
lists:foreach (fun (F) -> file:delete (Dir ++ "/" ++ F) end, FileNames),
file:del_dir (Dir)
end;
main ([]) ->
Port = open_port ({ fd, 0, 2 }, [ out ]),
port_command (Port, "usage: find-modules.esc tmpdir filename [filename ...]\n"),
halt (1).

It's definitely possible to fool the above code. For instance:


Hidden = therealname,
erlang:register (Hidden, SomePid).

However in practice developers tend to use either explicitly name their registered process or at worst obscure them with a macro (which, by going through the Erlang compiler, we get expanded for free), so this automatic strategy has been working very well.

You can download fw-template-erlang from google code and get the complete source code from above plus some intuition about how it is called. One thing that's been on our TODO list is to combine all these analysis into one pass on the source code; currently we pass through the source code three times, which is clearly suboptimal, although the process is speedy enough that it is only noticeable when analyzing larger projects (like, when we converted yaws and mnesia to fw-template-erlang for our internal use).

Erlang is so clear.

2008-06-13T10:11:00.000-07:00

Ah the clearness of erlang.

Someone on the yaws list gets errors and asks:

Can any one please explain me what this error is about and how to
overcome this one?

and gets the reply

Well - it's fairly clear. It says:

{undef,[{compile,file,
["/home/phanikar/.yaws/yaws/default/m1.erl",
[binary,return_errors, ...

That means that when .yaws page has been transformed into an
.erl page, the yaws server cannot compile the generated .erl file
You need to have the erlang compiler in the load path

I'm going to assume the author was being sarcastic about that being clear, I mean I get that it couldn't compile something.. but.. no hint of what reasons it couldn't compile.

maybe to quote the awesome tude of of the dynamic window manager guys:

This keeps its userbase small and elitist. No novices asking stupid questions.

Well despite some of these problems erlang is super awesome. It just needs a little more new blood to inject new attitudes of usability [ or some other language will evbentually just steal it's good ideas, which would be a shame.. we have them now. ] Also new attitudes around easier sharing of libraries. [ the erlware guys get that. ].

I'm pretty proud to see Uncle Jesse and the gang here trying to contribute improvements all the time. Better mnesia on disk tables in the works, yaws and erlang patches, erlrc, automatic mnesia fragmentation mangment and loads more. I mean we're building a big and real distributed mnesia backed system so we're learning a lot, and we're certainly not the only ones contributing lately.

So, as a novice who asks stupid questions, I'm telling the rest of you out there not to give up if you run into a few weird error messages or attitudes from language scholars who have been so entrenched they don't see what seems weird at first. Despite those few small speed bumps, there is some really powerful awesome stuff going on here. So, don't let em be elitist barge in there and get some erlang for yourself.

P.S. Pattern matching rules!!!

I'm in ur Erlangz, upgrading ur applicationz

2008-06-11T11:24:00.000-07:00

Hot code loading is one of the sexiest features of Erlang; after all, who wants downtime? If you're running mnesia, it's also very convenient, because as we've learned mnesia does not like to be shut down (more precisely, it makes no attempt to incrementally replicate changes across brief outages, instead doing full table (re)copying; something probably nobody cares about because they never turn it off).

Erlang's hot code loading support consists of low level language and runtime features, and a high level strategy embodied by release handler. The latter is a very Java-esque strategy of bundling up the complete new state of the system and upgrading everything. Without being normative let's just say that we enjoy using our OS package manager to manage the software on our boxes, and so we were interested in integrating Erlang hot code loading with the OS package manager.

The result is erlrc, which when operating looks something like this:

pmineiro@ub32srvvmw-199% sudo apt-get install egerl
Reading package lists... Done
Building dependency tree    
Reading state information... Done
The following packages will be upgraded:
egerl
1 upgraded, 0 newly installed, 0 to remove and 34 not upgraded.
Need to get 0B/113kB of archives.
After unpacking 0B of additional disk space will be used.
WARNING: The following packages cannot be authenticated!  egerl
Install these packages without verification [y/N]? y
(Reading database ... 48221 files and directories currently installed.)
Preparing to replace egerl 4.0.1 (using .../archives/egerl_4.1.0_all.deb) ...
Unpacking replacement egerl ...
erlrc-upgrade: Upgrading 'egerl': (cb8eec1a1b85ec017517d3e51c5aee7b) upgraded
Setting up egerl (4.1.0) ...
erlrc-start: Starting 'egerl': (cb8eec1a1b85ec017517d3e51c5aee7b) already_running

Basically, I issued an apt command, the result was to hot upgrade the corresponding Erlang application on the running Erlang node on the machine.

In addition to integration with the OS package manager, erlrc also features an extensible boot procedure (basically, using a boot directory instead of a boot file to determine what to start, which makes packaging easier); and in addition erlrc will automatically generate appup files which correspond to most of the common cases outlined in the appup cookbook. Of course, Erlang packages made with fw-template-erlang are automatically instrumented for erlrc (but the instrumentation is inert if erlrc is not installed).

If you're interested, there's more extensive documentation on google code.

Cooter to talk “Beautiful Concurrency with Erlang" at OSCON

2008-04-07T09:07:00.000-07:00

Our buddy Cooter is going to be giving a talk at OSCON entitled

“Beautiful Concurrency with Erlang"

Congrats cooter.

Nodefinder improvements

2008-03-26T21:53:00.000-07:00

Matthew Dempsky from Mochi mailed me some suggestions for nodefinder. Now the multicast_ttl is an application parameter so multicast routing can be used; separate send and receive sockets are employed to ensure the right source address on BSD; and the discovery packet is signed using the cookie as a secret which prevents (hopeless but annoyingly logged) connect attempts to nodes with different cookies and also prevents a list_to_atom flood from packets that happen to match.

Thanks Matt!

network partition ... oops

2008-03-21T21:24:00.000-07:00

well we had our first network partition on ec2 yesterday evening. we think it was amazon doing maintenance because it happened in both our production and development clusters.

it turns out things went pretty well considering we've done nothing yet to address network partitioning. two islands formed, but never tried to join back up (it turns out we only try to connect nodes when schemafinder starts). after a while each island's schemafinder decided that the other island was dead and removed it from the schema. so we ended up with two disjoint mnesia instances. they each had complete data so they were able to serve. of course, they were also creating data and diverging over time. however right now we're not doing anything that important so it didn't really matter. we just picked one island to survive and restarted the other ones after nuking their mnesia directories.

however the experience has convinced us that we need to prioritize up our network partition recovery strategy. the good news is that we have an out-of-band source of truth about what the set of nodes should be (via ec2-describe-instances), so it should be possible for us to detect a network partition has occurred. the other good news is that we care more about high availability than transactional integrity so we'll be willing to set master nodes on tables based upon simple heuristics.

walkenfs improvements

2008-03-15T15:08:00.000-07:00

I've made some progress towards my goal of making walkenfs usable by those who know nothing about Erlang or Mnesia.

schemafinder has been open sourced, which provides Mnesia auto-configure.
mount and umount work (although the mount line is very baroque), so no need to start an Erlang emulator explicitly.
the distribution includes an example init script very close to the one we actually use.

Of course walkenfs could still have a showstopping bug in it. In our recent use it was mostly fuse mount options that gave us trouble, and also the shared writeable mmap problem surfaced because rrdtool uses that by default. fortunately with rrdtool, it was a simple configure option to disable mmap. We did have a data corruption scare in development but that turned out to be due to the default redundancy setting being 1 copy (changed that to 3, now that fragmentron can initialize under infeasible conditions). Otherwise, so far so good, knock on wood.

mnesia and ec2

2008-03-12T21:39:00.000-07:00

Mnesia is Erlang's built-in multi-master distributed database. It's one of the reasons why we chose Erlang for our startup. And while it is mostly a great fit for EC2, it needs some tweaks to work. In particular we wanted the following to be automatic:

Discover nodes entering or leaving the system
Have them join the Mnesia database
Have them take responsibility for a portion of the data

We've already talked about and published solutions to the first problem (nodefinder) and third problem (fragmentron). Now I just published a solution to the second problem on google code (schemafinder). It took a while to get this part out partially because we kept finding small problems with it, but mostly because we're very busy.

It's actually surprisingly complicated, given that at the root the way to add a node to a running database is to call mnesia:change_config/2 as mnesia:change_config (extra_db_nodes, NodeList). However while adding nodes is easy removing nodes is harder. Detecting nodes to remove is not that hard; we go with the (EC2-centric) strategy of periodically updating a record in a table to "stay alive" (and also a way to mark clean shutdown); nodes that fail to check in eventually are considered dead. To handle multiple node failure we patch mnesia to provide mnesia_schema:del_table_copies/2, the multiple table analog to mnesia:del_table_copy/2. To guard against rejoining the global schema after having been removed but without having removed one's mnesia database directory (this is a no-no), we check with the other database nodes to see if they think we've been removed.

Even with all this complication, we have yet to address the problem of handling network partitioning automatically. We'll see if we have time to address that before it bites us in the butt.

Now available on google code.

controlling mnesia db size

2008-03-05T21:13:00.000-08:00

genexpire something quick that we rolled together to satisfy our paranoia about out-of-control databases (ram based dbs are especially worrisome). Basically, it periodically runs and enforces a maximum size for each database fragment (which depends upon how many fragments are on the node; in practice this means you get the space of your most crowded node in a distributed config. With fragmentron things are balanced to within one fragment so with enough fragments this is ok.)

Now available on google code.

Procfs

2008-03-02T22:53:00.001-08:00

Our next sprint's theme is "monitoring", so this weekend I thought I'd slap together a procfs style filesystem with fuserl. I've got the contents of erlang:processes (), erlang:process_info (), erlang:ports (), erlang:port_info (), and erlang:system_info () now exposed in a filesystem format. This is less useful than I envisioned on Friday, but it was pedagogical.

Now available on google code.

Erlang and Automake

2008-02-24T11:06:00.000-08:00

There were some inquiries on the erlang-questions mailing list recently about using Gnu Automake with Erlang. That reminded me to release a build system we've been using for development.

My fascination with build systems predates my obsession with Erlang so it is more general than just Erlang. It is organized around project templates, and there is Erlang project template that has been getting pretty well developed. Every time we figure out a new best practice around Erlang development we incorporate it into the template so that all of our projects benefit. So for instance now we have cover integration, eunit integration, and edoc integration.

It is now available on google code.

New book from Erlang Training

2008-02-22T10:14:00.000-08:00

So, the guys Erlang Consulting and Training are making a new book.

and on the erlang-questions mailing list:

On Feb 21, 2008, at 1:50 PM, Francesco Cesarini wrote:
Our main goal is to cover Erlang in depth. Make sure that newbies struggling with recursion can easily understand it or developers being exposed to pattern matching for the first time use it optimally. The contents of the book are pretty much outlined by the introductory Erlang courses Jan-Henry and I have been giving in the last decade, and based on the knowledge and experience gained when teaching. We know [where] students struggle...

On Feb 21, 2008, at 4:29 PM, some guy wrote:
It's just that, no matter how good it is, with Joe's book already out there - it's not going to be a must read for me.

On Feb 21, 2008, at 3:17 PM, someone else wrote:
Another book on just the Erlang programming language would not find a place on my bookshelf.

These responses kind of made me upset.

I applaud Francesco, and Jan-Henry in getting another book out there. Obviously they have learned from their history areas where new comers to erlang have troubles, and they're making a book to plug that hole. They are of course asking you to look back on your experience coming into erlang, think about what concepts you had troubles with and let them know, this would both help confirm the topics for their book, and possibly give them new topics that are perhaps different than they have been exposed to because they come from people
who don't buy erlang training courses.

Just because you don't need the book now doesn't mean you dont' have something to add.

And sure, we all have frustrations about the OTP stuff, it seems like it might be very very good, and then again it seems pretty opaque. I encourage anyone reading this who knows more OTP to publish more stuff about it.

[ I know I'm calling myself RoscoeOTPColtrain, but right now I'm more roscoe than OTP, but perhaps I'll get there. ]

As a scripter in the internet era I've always looked for high level tools that help me build scalable reliable systems as quickly and as error free as possible.

As a manager I've grown to look for tools that help reduce the errors caused by teams of varying skill level trying to make deadlines that have little basis in reality.

I believe erlang is a good fit. As mentioned on erlang questions list It's missing libraries, [ we all should be addressing this. ] it's missing the ability to share code easily amongst the community. [ erlware guys are trying to address this. ] It's missing a lot of depth of documentation especially with respects to OTP.

"see the OTP design principles", is it's own running gag around my office. "Hey, where is the coffee maker?!?" "See the OTP design principles."

Erlang provides some great tools to help regular code monkeys do great things, with less errors. It comes at a cost of learning some interesting programing paradigms which of course seem trivial to elitist programing apes.

So, being a procedural and sometimes OO script monkey for most of my life. [ the joy of programing being denied to me cause I always end up managing. :) ] I found once I bridged the concepts of erlang I totally love it. I feel that there are plenty of others out there who are like me that could also feel the same way.

and last, quoting yarivs "damn, I love pattern matching"

Automatic Node Discovery

2008-02-13T20:34:00.003-08:00

One of the first problems we ran into on EC2 was having Erlang nodes find each other. Erlang ships with a node discovery strategy based upon a static hosts file. On EC2, the idea is to start and stop nodes frequently; and EC2 will typically hand out different hostnames each time (I guess that repeats are possible but unlikely). So a static hosts file won't work.

Fortunately Erlang is ready for the set of nodes to change constantly, it is just not spelled out how to find them. I've noticed this about Erlang: often an excellent design is coupled with a simplistic stub implementation. It's the "Tommy" school of software: even the most casual observer is supposed to be able to figure out the next step.

Initially we thought we'd use multicast UDP for discovery, which was extremely easy to do but then we discovered that EC2 does not support multicast. So then I overthought the problem and came up with a discovery protocol based upon using S3 as a blackboard. That worked but was overkill, since just parsing the output of ec2-describe-instances contains all the information needed, plus it allows EC2 security groups to define the Erlang node groups. This is convenient because the EC2 security group(s) can be determined dynamically from the instance metadata.

As an aside, the parsing of the output of ec2-describe-instances is done with a list comprehension and built-in pattern matching only (parse_host_list/2 and parse_host_list/4).

All three strategies are now available on google code. If you're on EC2, ec2nodefinder is the recommendation. Note, you can still in addition use a static hosts file (or any other node discovery strategy), e.g., if you have some non-EC2 hosts you'd like to connect as well.

howdy

2008-02-12T21:25:00.000-08:00

we're the dukes. basically 3 guys at a brand new startup (more about that later). we want to have a go with Erlang and EC2 as the two seem to go together pretty nicely. anyway we thought it'd be fun to blog about things as we figure them out, mostly about cloud computing with Erlang, but occasionally maybe about some dramatic startup stuff like not being able to make payroll.

walkenfs on google code

2008-02-12T20:57:00.001-08:00

We Dukes are into cloud computing. We especially like our machines to be operationally symmetric so we don't have to think that hard. Erlang makes that easy but unfortunately sometimes we have to deal with software not written in Erlang.

In particular Fess started to put together a monitoring system lately using rrdtool which is great but wants to operate on files. We'd done all this work to make mnesia play nicely on EC2 (more on that later) and suddenly we seemed back at square one. We considered storing data in mnesia and rendering files at the last minute for rrdtool; but then we noticed fuse. Sweet! Now we can store data in mnesia and make it look like a filesystem at the same time.

First we needed Erlang bindings for fuse, so I wrote those. That turned out to involve writing C code, a moderately frustrating experience that served as a nice reminder why we don't call ourselves the "Dukes of C". Once the nasty bits were subdued, writing the actual Erlang code corresponding to a filesystem via mnesia was comparatively quick and painless. I'm still trying to figure out the best way to support posix locking (right now, I'm looking for a satisfying way to have the locks release when an entire node dies unexpectedly), but everything else seems to be working.

It still needs alot of polish. As a filesystem one would expect a mkfs tool and the ability to put entries in /etc/fstab that work. However it's usable as part of an Erlang system right now.