Tokyocabinet and Mnesia

As Daisy has already indicated, it is now possible to plug arbitrary storage strategies into Mnesia. For those who are familiar with mnesia_access, this is different; mnesia_access only covers reads and writes, not schema manipulations, and has other deficiencies that it render it useless for adding a new storage type in practice (what mnesia_access is great for is changing the semantics of mnesia operations, e.g., mnesia_frag). This project lets you make tables that are essentially indistinguishable from the built-in mnesia table types (ram_copies, disc_copies, disc_only_copies).

Anyway our goal was to get a good on-disk ordered_set table type, since we've found ordered_sets very useful for the kinds of problems we're solving but we're tired of being limited by memory. After looking around for a while Tokyocabinet emerged as our favorite for underlying implementation. We considered BDB and libmysql, but Tokyocabinet seemed simple and faster, and had a more accommodating license. So we ported Tokyocabinet to Erlang and then used the above storage API to connect to Mnesia.

As a side benefit Tokyocabinet might also be preferred to dets even for set-type applications because of the lack of file size limit and high performance. Tokyocabinet actually has a set-type storage strategy that we'd like to define an Erlang Term Store for, but as of this post, the set-type store doesn't support cursor positioning based upon a key, which makes the implementation of next tedious (although not impossible). So I'm waiting on the author to add that call; if that happens, we could have a nicer on-disk set-type table as well.

Everything is available on google code: mnesiaex (storage API) and tcerl (erlang port of Tokyocabinet).

Also Daisy (aka Joel Reymont) is really nice to work with.

generating app files

We automatically generate our .app files around here from analysis of the source code. However this trick is buried inside fw-template-erlang which most (all?) people outside our group do not use. I thought I'd highlight how it works.

A typical OTP application specification file looks like:

{ application, walkenfs,
[
  { description, "Distributed filesystem." },
  { vsn, "0.1.7" },
  { modules, [ walkenfssup, walkenfsfragmentsrv, walkenfs, walkenfssrv ] },
  { registered, [ walkenfsfragmentsrv ] },
  { applications, [ kernel, stdlib   ] },
  { mod, { walkenfs, [] } },
  { env, [ { linked_in, false }, { mount_opts, "" }, { mount_point, "/walken" }, { prefix, walken }, { read_data_context, async_dirty }, { write_data_context, sync_dirty }, { read_meta_context, transaction }, { write_meta_context, sync_transaction }, { attr_timeout_ms, 0 }, { entry_timeout_ms, 0 }, { reuse_node, true }, { start_timeout, infinity }, { stop_timeout, 1800000 }, { init_fragments, 7 }, { init_copies, 3 }, { copy_type, n_disc_only_copies }, { buggy_ets, auto_detect }, { port_exit_stop, true } ] }

]
}.

There are alot of bits here that can filled in automatically. It helps to think in terms of the template:

{ application, @FW_PACKAGE_NAME@,
[
  { description, "@FW_PACKAGE_SHORT_DESCRIPTION@" },
  { vsn, "@FW_PACKAGE_VERSION@" },
  { modules, [ @FW_ERL_APP_MODULES@ ] },
  { registered, [ @FW_ERL_APP_REGISTERED@ ] },
  { applications, [ kernel, stdlib @FW_ERL_PREREQ_APPLICATIONS@ ] },
  @FW_ERL_APP_MOD_LINE@
  { env, @FW_ERL_APP_ENVIRONMENT@ }
  @FW_ERL_APP_EXTRA@
]
}.

This is an "automake" style template where @VARIABLE@ indicates something that will be substituted.

Now we can say where each of these templates values comes from when using fw-template-erlang:

1. FW_PACKAGE_NAME: specified by the developer. if you're using automake you've had to specify this already via AC_INIT, so it's good to reuse that rather than require duplicate specification.
   
2. FW_PACKAGE_SHORT_DESCRIPTION: specified by the developer.
3. FW_PACKAGE_VERSION: specified by the developer; like the package name, if you're using automake this information is already contained in the arguments to AC_INIT so reuse that.
4. FW_ERL_APP_MODULES: generated by analysis of the source code directory.
5. FW_ERL_APP_REGISTERED: generated by analysis of the source code directory.
   
6. FW_ERL_PREREQ_APPLICATIONS: specified by the developer.
7. FW_ERL_MOD_LINE: partially generated by analysis of the source code directory, partially specified by the developer (see below).
8. FW_ERL_APP_ENVIRONMENT: specified by the developer.
9. FW_ERL_APP_EXTRA: specified by the developer. this is for unusual extra directives like included applications.

So the interesting bits come down to FW_ERL_APP_MODULES, FW_ERL_APP_REGISTERED, and FW_ERL_MOD_LINE being generated automatically via inspection of the source code.

First, a note on intervention. Anytime you are attempting to automate a task that was previous done by humans, it's helps to allow a human to override any of the automatic settings that you generate by default. That way, you can be correct only 95% of the time and still be a huge timesaver. In practice we've found that although it easy to come up with code examples that flummox the automatic strategies, such code is never actually written by people in the normal course of their work. In fact, we've found that application files we get from the universe are often missing registered process names that our automatic strategy finds. So empirically we're running at 100% correct so far, but fw-template-erlang still contains the capability to override any automatically generated value.

Now the actual trick to computing all the stuff is that Erlang contains an Erlang compiler as a library. Here's an example of leveraging this to generate all the module names from a set of files; any file that contains a

-fwskip()

directive will be ignored.

-module (find_modules).
-export ([ main/1 ]).

is_skipped ([]) -> false;
is_skipped ([ { attribute, _, fwskip, _ } | _ ]) -> true;
is_skipped ([ _ | Rest ]) -> is_skipped (Rest).

print_module (Dir, F) ->
case compile:file (F, [ binary, 'E', { outdir, Dir } ]) of
{ ok, Mod, { Mod, _, Forms } } ->
case is_skipped (Forms) of
  true ->
    ok;
  false ->
    port_command (get (three), io_lib:format ("~p~n", [ Mod ]))
end;
_ ->
ok
end.

main ([ Dir | Rest ]) ->
ok = file:make_dir (Dir),

try
  Three = open_port ({ fd, 0, 3 }, [ out ]),
% ugh ... don't want to have to change all these function signatures,
% so i'm gonna be dirty
  put (three, Three),
  lists:foreach (fun (F) -> print_module (Dir, F) end, Rest)
after
  { ok, FileNames } = file:list_dir (Dir),
  lists:foreach (fun (F) -> file:delete (Dir ++ "/" ++ F) end, FileNames),
  file:del_dir (Dir)
end;
main ([]) ->
Port = open_port ({ fd, 0, 2 }, [ out ]),
port_command (Port, "usage: find-modules.esc tmpdir filename [filename ...]\n"),
halt (1).

The output is on file descriptor 3 in order to isolate the desired output from various messages being output by the Erlang code. This was originally an escript but to be compatible with older versions of Erlang we changed it to run as follows:

#! /bin/sh

# NB: this script is called find-modules.sh

# unfortunately, escript is a recent innovation ...

ERL_CRASH_DUMP=/dev/null
export ERL_CRASH_DUMP

erl -pa "${FW_ROOT}/share/fw/template/erlang/" -pa "${FW_ROOT}/share/fw.local/template/erlang" -pa "${top_srcdir}/fw/template/erlang/" -pa "${top_srcdir}/fw.local/template/erlang" -noshell -noinput -eval '
find_modules:main (init:get_plain_arguments ()),
halt (0).
' -extra "$@" 3>&1 >/dev/null

In practice this gets executed like

find src -name '*.erl' -print | xargs find-modules.sh "$tmpdir" |
perl -ne 'chomp;
     next if $seen{$_}++;
     print ", " if $f;
     print $_;
     $f = 1;'

Ok, that was the easy one! In fact, the problem of finding all the module names seems so easy that one could be tempted to solve it with grep (hmmm: what about preprocesser directives ?). However the point is that nothing parses Erlang as well as Erlang, so why reinvent the wheel.

Computing the mod line in the application specification is just slightly harder. Basically we scan source code for a module which has a

-behaviour(application)

attribute and assume that is the start module. The Erlang compiler allows behaviour to be (mis)spelt American-style so we allow that as well.

-module (find_start_module).
-export ([ main/1 ]).

is_application ([]) -> false;
is_application ([ { attribute, _, behaviour, [ application ] } | _ ]) -> true;
is_application ([ { attribute, _, behavior, [ application ] } | _ ]) -> true;
is_application ([ _ | Rest ]) -> is_application (Rest).

is_skipped ([]) -> false;
is_skipped ([ { attribute, _, fwskip, _ } | _ ]) -> true;
is_skipped ([ _ | Rest ]) -> is_skipped (Rest).

find_start_module (_, []) ->
ok;
find_start_module (Dir, [ F | Rest ]) ->
case compile:file (F, [ binary, 'E', { outdir, Dir } ]) of
{ ok, Mod, { Mod, _, Forms } } ->
  case is_application (Forms) and not is_skipped (Forms) of
    true ->
      port_command (get (three), io_lib:format ("~p~n", [ Mod ]));
    false ->
      find_start_module (Dir, Rest)
  end;
_ ->
  find_start_module (Dir, Rest)
end.

main ([ Dir | Rest ]) ->
ok = file:make_dir (Dir),

try
Three = open_port ({ fd, 0, 3 }, [ out ]),
% ugh ... don't want to have to change all these function signatures,
% so i'm gonna be dirty
put (three, Three),
find_start_module (Dir, Rest)
after
{ ok, FileNames } = file:list_dir (Dir),
lists:foreach (fun (F) -> file:delete (Dir ++ "/" ++ F) end, FileNames),
file:del_dir (Dir)
end;
main ([]) ->
Port = open_port ({ fd, 0, 2 }, [ out ]),
port_command (Port, "usage: find-modules.esc tmpdir filename [filename ...]\n"),
halt (1).

If there are no applications in the set of files being analyzed, we output nothing for the mod line in the application file, otherwise we output

{ mod, { @FW_ERL_APP_START_MODULE@, @FW_ERL_APP_START_ARGS@ } },

where FW_ERL_APP_START_MODULE is the output of the above command and FW_ERL_APP_START_ARGS is indicated by the programmer (we use [] by default for FW_ERL_APP_START_ARGS and in practice never use the field). Obviously, if multiple modules implementing the application behaviour are found then this will not work. Our style has been to put each application we make in a separate directory to facilitate automatic analysis.

Anyway now we can tackle the more challenging problem of finding registered processes. There are many library calls that end up registering a process name; the ones we recognize are:

• supervisor child specs that contain calls of the form
• { _, start, [ { local, xxx }, ... ] } -> xxx being registered
• { _, start_link, [ { local, xxx }, ... ] } -> xxx being registered
• function calls of the form
• Module:start ({ local, xxx }, ...) -> xxx being registered
• Module:start_link ({ local, xxx }, ...) -> xxx being registered
• calls to erlang:register (xxx, ...) -> xxx being registered

There's alot of code here but the general idea is the same: walk the forms generated by compile and pattern match on one of the above cases, outputting discovered registered process names to file descriptor 3.

-module (find_registered).
-export ([ main/1 ]).

%% looks like fun_clauses are the same as clauses (?)
print_registered_fun_clauses (Clauses) ->
print_registered_clauses (Clauses).

%% looks like icr_clauses are the same as clauses (?)
print_registered_icr_clauses (Clauses) ->
print_registered_clauses (Clauses).

print_registered_inits (Inits) ->
lists:foreach (fun ({ record_field, _, _, E }) ->
               print_registered_expr (E);
                 (_) ->
               ok
             end,
             Inits).

print_registered_upds (Upds) ->
lists:foreach (fun ({ record_field, _, _, E }) ->
               print_registered_expr (E);
                 (_) ->
               ok
             end,
             Upds).

% hmmm ... pretty sure patterns are not supposed to have side-effects ...
print_registered_pattern (_) -> ok.
print_registered_pattern_group (_) -> ok.

print_registered_quals (Qs) ->
lists:foreach (fun ({ generate, _, P, E }) ->
               print_registered_pattern (P),
               print_registered_expr (E);
                 ({ b_generate, _, P, E }) ->
               print_registered_pattern (P),
               print_registered_expr (E);
                 (E) ->
               print_registered_expr (E)
             end,
             Qs).

print_registered_expr ({ cons, _, H, T }) ->
print_registered_expr (H),
print_registered_expr (T);
print_registered_expr ({ lc, _, E, Qs }) ->
print_registered_expr (E),
print_registered_quals (Qs);
print_registered_expr ({ bc, _, E, Qs }) ->
print_registered_expr (E),
print_registered_quals (Qs);

%% Ok, here's some meat:
%% the "supervisor child spec" rule
%% { _, start, [ { local, xxx }, ... ] } -> xxx being registered
%% { _, start_link, [ { local, xxx }, ... ] } -> xxx being registered
print_registered_expr ({ tuple, _,
                     Exprs=[ _,
                       { atom, _, Func },
                       { cons,
                         _,
                         { tuple, _, [ { atom, _, local },
                                       { atom, _, Name } ] },
                         _ } ] }) when (Func =:= start) or
                                       (Func =:= start_link) ->
port_command (get (three), io_lib:format ("~p~n", [ Name ])),
print_registered_exprs (Exprs);

print_registered_expr ({ tuple, _, Exprs }) ->
print_registered_exprs (Exprs);
print_registered_expr ({ record_index, _, _, E }) ->
print_registered_expr (E);
print_registered_expr ({ record, _, _, Inits }) ->
print_registered_inits (Inits);
print_registered_expr ({ record_field, _, E0, _, E1 }) ->
print_registered_expr (E0),
print_registered_expr (E1);
print_registered_expr ({ record, _, E, _, Upds }) ->
print_registered_expr (E),
print_registered_upds (Upds);
print_registered_expr ({ record_field, _, E0, E1 }) ->
print_registered_expr (E0),
print_registered_expr (E1);
print_registered_expr ({ block, _, Exprs }) ->
print_registered_exprs (Exprs);
print_registered_expr ({ 'if', _, IcrClauses }) ->
print_registered_icr_clauses (IcrClauses);
print_registered_expr ({ 'case', _, E, IcrClauses }) ->
print_registered_expr (E),
print_registered_icr_clauses (IcrClauses);
print_registered_expr ({ 'receive', _, IcrClauses }) ->
print_registered_icr_clauses (IcrClauses);
print_registered_expr ({ 'receive', _, IcrClauses, E, Exprs }) ->
print_registered_icr_clauses (IcrClauses),
print_registered_expr (E),
print_registered_exprs (Exprs);
print_registered_expr ({ 'try', _, Exprs0, IcrClauses0, IcrClauses1, Exprs1 }) ->
print_registered_exprs (Exprs0),
print_registered_icr_clauses (IcrClauses0),
print_registered_icr_clauses (IcrClauses1),
print_registered_exprs (Exprs1);
print_registered_expr ({ 'fun', _, Body }) ->
case Body of
  { clauses, Cs } ->
    print_registered_fun_clauses (Cs);
  _ ->
    ok
end;

%% Ok, here's some meat:
%% Module:start ({ local, xxx }, ...) -> xxx being registered
%% Module:start_link ({ local, xxx }, ...) -> xxx being registered

print_registered_expr ({ call,
                     _,
                     E={ remote, _, _, { atom, _, Func } },
                     Exprs=[ { tuple, _, [ { atom, _, local },
                                           { atom, _, Name } ] } | _ ] })
                        when (Func =:= start) or
                             (Func =:= start_link) ->
port_command (get (three), io_lib:format ("~p~n", [ Name ])),
print_registered_expr (E),
print_registered_exprs (Exprs);

%% Ok, here's some meat:
%% erlang:register (xxx, ...) -> xxx being registered

print_registered_expr ({ call,
                     _,
                     { remote,
                       _,
                       { atom, _, erlang },
                       { atom, _, register } },
                     Exprs=[ { atom, _, Name } | _ ] }) ->
port_command (get (three), io_lib:format ("~p~n", [ Name ])),
print_registered_exprs (Exprs);

print_registered_expr ({ call, _, E, Exprs }) ->
print_registered_expr (E),
print_registered_exprs (Exprs);
print_registered_expr ({ 'catch', _, E }) ->
print_registered_expr (E);
print_registered_expr ({ 'query', _, E }) ->
print_registered_expr (E);
print_registered_expr ({ match, _, P, E }) ->
print_registered_pattern (P),
print_registered_expr (E);
print_registered_expr ({ bin, _, PatternGrp }) ->
print_registered_pattern_group (PatternGrp);
print_registered_expr ({ op, _, _, E }) ->
print_registered_expr (E);
print_registered_expr ({ op, _, _, E0, E1 }) ->
print_registered_expr (E0),
print_registered_expr (E1);
print_registered_expr ({ remote, _, E0, E1 }) ->
print_registered_expr (E0),
print_registered_expr (E1);
print_registered_expr (_) ->
ok.

print_registered_exprs (Exprs) ->
lists:foreach (fun (E) -> print_registered_expr (E) end, Exprs).

print_registered_clauses (Clauses) ->
lists:foreach (fun ({ clause, _, _, _, Exprs }) ->
                 print_registered_exprs (Exprs);
                   (_) ->
                 ok
               end,
               Clauses).

print_registered_forms (Forms) ->
lists:foreach (fun ({ function, _, _, _, Clauses }) ->
                 print_registered_clauses (Clauses);
                   (_) ->
                 ok
               end,
               Forms).

is_skipped ([]) -> false;
is_skipped ([ { attribute, _, fwskip, _ } | _ ]) -> true;
is_skipped ([ _ | Rest ]) -> is_skipped (Rest).

print_registered (Dir, F) ->
case compile:file (F, [ binary, 'E', { outdir, Dir } ]) of
{ ok, _, { _, _, Forms } } ->
  case is_skipped (Forms) of
    true ->
      ok;
    false ->
      print_registered_forms (Forms)
  end;
_ ->
  ok
end.

main ([ Dir | Rest ]) ->
ok = file:make_dir (Dir),

try
Three = open_port ({ fd, 0, 3 }, [ out ]),
% ugh ... don't want to have to change all these function signatures,
% so i'm gonna be dirty
put (three, Three),
lists:foreach (fun (F) -> print_registered (Dir, F) end, Rest)
after
{ ok, FileNames } = file:list_dir (Dir),
lists:foreach (fun (F) -> file:delete (Dir ++ "/" ++ F) end, FileNames),
file:del_dir (Dir)
end;
main ([]) ->
Port = open_port ({ fd, 0, 2 }, [ out ]),
port_command (Port, "usage: find-modules.esc tmpdir filename [filename ...]\n"),
halt (1).

It's definitely possible to fool the above code. For instance:

Hidden = therealname,
erlang:register (Hidden, SomePid).

However in practice developers tend to use either explicitly name their registered process or at worst obscure them with a macro (which, by going through the Erlang compiler, we get expanded for free), so this automatic strategy has been working very well.

You can download fw-template-erlang from google code and get the complete source code from above plus some intuition about how it is called. One thing that's been on our TODO list is to combine all these analysis into one pass on the source code; currently we pass through the source code three times, which is clearly suboptimal, although the process is speedy enough that it is only noticeable when analyzing larger projects (like, when we converted yaws and mnesia to fw-template-erlang for our internal use).

Erlang is so clear.

Ah the clearness of erlang.

Someone on the yaws list gets errors and asks:

Can any one please explain me what this error is about and how to
overcome this one?

and gets the reply

Well - it's fairly clear. It says:

{undef,[{compile,file,
["/home/phanikar/.yaws/yaws/default/m1.erl",
[binary,return_errors, ...

That means that when .yaws page has been transformed into an
.erl page, the yaws server cannot compile the generated .erl file
You need to have the erlang compiler in the load path

I'm going to assume the author was being sarcastic about that being clear, I mean I get that it couldn't compile something.. but.. no hint of what reasons it couldn't compile.

maybe to quote the awesome tude of of the dynamic window manager guys:

This keeps its userbase small and elitist. No novices asking stupid questions.

Well despite some of these problems erlang is super awesome. It just needs a little more new blood to inject new attitudes of usability [ or some other language will evbentually just steal it's good ideas, which would be a shame.. we have them now. ] Also new attitudes around easier sharing of libraries. [ the erlware guys get that. ].

I'm pretty proud to see Uncle Jesse and the gang here trying to contribute improvements all the time. Better mnesia on disk tables in the works, yaws and erlang patches, erlrc, automatic mnesia fragmentation mangment and loads more. I mean we're building a big and real distributed mnesia backed system so we're learning a lot, and we're certainly not the only ones contributing lately.

So, as a novice who asks stupid questions, I'm telling the rest of you out there not to give up if you run into a few weird error messages or attitudes from language scholars who have been so entrenched they don't see what seems weird at first. Despite those few small speed bumps, there is some really powerful awesome stuff going on here. So, don't let em be elitist barge in there and get some erlang for yourself.

P.S. Pattern matching rules!!!

I'm in ur Erlangz, upgrading ur applicationz

Hot code loading is one of the sexiest features of Erlang; after all, who wants downtime? If you're running mnesia, it's also very convenient, because as we've learned mnesia does not like to be shut down (more precisely, it makes no attempt to incrementally replicate changes across brief outages, instead doing full table (re)copying; something probably nobody cares about because they never turn it off).

Erlang's hot code loading support consists of low level language and runtime features, and a high level strategy embodied by release handler. The latter is a very Java-esque strategy of bundling up the complete new state of the system and upgrading everything. Without being normative let's just say that we enjoy using our OS package manager to manage the software on our boxes, and so we were interested in integrating Erlang hot code loading with the OS package manager.

The result is erlrc, which when operating looks something like this:

pmineiro@ub32srvvmw-199% sudo apt-get install egerl
Reading package lists... Done
Building dependency tree    
Reading state information... Done
The following packages will be upgraded:
egerl
1 upgraded, 0 newly installed, 0 to remove and 34 not upgraded.
Need to get 0B/113kB of archives.
After unpacking 0B of additional disk space will be used.
WARNING: The following packages cannot be authenticated!  egerl
Install these packages without verification [y/N]? y
(Reading database ... 48221 files and directories currently installed.)
Preparing to replace egerl 4.0.1 (using .../archives/egerl_4.1.0_all.deb) ...
Unpacking replacement egerl ...
erlrc-upgrade: Upgrading 'egerl': (cb8eec1a1b85ec017517d3e51c5aee7b) upgraded
Setting up egerl (4.1.0) ...
erlrc-start: Starting 'egerl': (cb8eec1a1b85ec017517d3e51c5aee7b) already_running

Basically, I issued an apt command, the result was to hot upgrade the corresponding Erlang application on the running Erlang node on the machine.

In addition to integration with the OS package manager, erlrc also features an extensible boot procedure (basically, using a boot directory instead of a boot file to determine what to start, which makes packaging easier); and in addition erlrc will automatically generate appup files which correspond to most of the common cases outlined in the appup cookbook. Of course, Erlang packages made with fw-template-erlang are automatically instrumented for erlrc (but the instrumentation is inert if erlrc is not installed).

If you're interested, there's more extensive documentation on google code.