Wednesday, August 26, 2009

Dynamically Loading Webmachine Resources

I've been using webmachine lately, which is fabulous for REST server development. It has a modular concept of resources and the design could be summarized as a "RESTlet container". Because Erlang has hot code loading I was interested in doing this dynamically; in other words, I want to write a module which provides a webmachine resource, hot code load it into the system, and have webmachine start dispatching requests to it.

The good news is, you can change the dispatch list at any time, via
application:set_env (webmachine, dispatch_list, NewDispatchList).
This does not upset webmachine and it will see the change immediately. So for a single application leveraging webmachine, this seemed as simple as having a code_change handler execute something like
application:set_env
(webmachine,
dispatch_list,
lists:foldl (fun (Mod, Acc) ->
case catch Mod:dispatch_rules () of
{ 'EXIT', _ } -> Acc;
X -> X ++ Acc
end
end,
[],
element (2, application:get_key (drurly, modules)))).
(drurly is the name of the application in this case). Now my modules export a function like

dispatch_rules () ->
[ { [ "clip" ], ?MODULE, [] },
{ [ "clip", id ], ?MODULE, [] }
].
if they are webmachine resources.

The problem is that order is important: the webmachine_dispatch_list is consulted in order, and the first match is executed. To solve this I created a sort function which orders the dispatch rules by "specificity"
path_spec_priority ('*') -> 3;
path_spec_priority (X) when is_atom (X) -> 2;
path_spec_priority (X) when is_list (X) -> 1.

dispatch_specificity ({ PathSpecA, _, _ },
{ PathSpecB, _, _ }) ->
case erlang:length (PathSpecA) - erlang:length (PathSpecB) of
X when X > 0 ->
true;
X when X < 0 ->
false;
_ ->
PrioPathSpecA = [ path_spec_priority (X) || X <- PathSpecA ],
PrioPathSpecB = [ path_spec_priority (X) || X <- PathSpecB ],

case PrioPathSpecA =< PrioPathSpecB of
false ->
false;
true ->
FullPathSpecA = [ { path_spec_priority (X), X } || X <- PathSpecA ],
FullPathSpecB = [ { path_spec_priority (X), X } || X <- PathSpecB ],

FullPathSpecA =< FullPathSpecB
end
end.
Basically:
  • Longer dispatch paths come first.
  • If two dispatch paths have equal length, the more specific one comes first, where specificity is defined by examining the elements left-to-right, with
    • string literals are most specific
    • atoms except '*' are the next most specific
    • '*' is the least specific
  • If two dispatch paths have equal length and specificity, sort by Erlang term order (effectively, break ties arbitrarily)
The code change handler now becomes:
application:set_env
(webmachine,
dispatch_list,
lists:sort
(fun dispatch_specificity/2,
lists:foldl (fun (Mod, Acc) ->
case catch Mod:dispatch_rules () of
{ 'EXIT', _ } -> Acc;
X -> X ++ Acc
end
end,
[],
element (2, application:get_key (drurly, modules))))).
This has proven sufficient for a single application running webmachine.

For multiple applications that want to run under the same webmachine, I suspect the right way to go is to have a gen_server which
  • contains webmachine dispatch configuration "fragments" per key, where the key is intended to be the application name;
  • accepts commands to replace or delete a particular fragment by key;
  • for any replace or delete, rebuilds the complete dispatch list by concatenating all fragments together and sorting by specificity, and then updates the webmachine application env var.
For multiple applications that want to run under different webmachines, well unfortunately webmachine uses some global application settings under the fixed atom webmachine and thus currently, like the highlander, there can be only one. (In fact, you can only listen on one ip/port combination with webmachine right now. I might have to patch it to accept multiple ip/port combinations to listen to, since a standard trick of mine is to have nginx handle both regular and ssl connections and connect to a different back-end port to indicate whether or not the connection is secure.)

3 comments:

justin said...

I do think order based dispatch is just wrong, and specificity is a much better idea in general. It would be nice to push this back to webmachine. You could provide a sort function perhaps.

Paul Mineiro said...

I think order-based is the safest thing for a general library to implement, because it allows the user to implement any concept of priority on top of it.

However I agree that the concept of specificity defined in the post will probably capture most user's needs, so getting it back into webmachine as a sort function makes sense. I'll ping them.

Paul Mineiro said...

The application controller returns old data during a code change handler, solution is here: http://dukesoferl.blogspot.com/2009/09/application-variables-during-upgrade.html