Tokyocabinet and Mnesia

As Daisy has already indicated, it is now possible to plug arbitrary storage strategies into Mnesia. For those who are familiar with mnesia_access, this is different; mnesia_access only covers reads and writes, not schema manipulations, and has other deficiencies that it render it useless for adding a new storage type in practice (what mnesia_access is great for is changing the semantics of mnesia operations, e.g., mnesia_frag). This project lets you make tables that are essentially indistinguishable from the built-in mnesia table types (ram_copies, disc_copies, disc_only_copies).

Anyway our goal was to get a good on-disk ordered_set table type, since we've found ordered_sets very useful for the kinds of problems we're solving but we're tired of being limited by memory. After looking around for a while Tokyocabinet emerged as our favorite for underlying implementation. We considered BDB and libmysql, but Tokyocabinet seemed simple and faster, and had a more accommodating license. So we ported Tokyocabinet to Erlang and then used the above storage API to connect to Mnesia.

As a side benefit Tokyocabinet might also be preferred to dets even for set-type applications because of the lack of file size limit and high performance. Tokyocabinet actually has a set-type storage strategy that we'd like to define an Erlang Term Store for, but as of this post, the set-type store doesn't support cursor positioning based upon a key, which makes the implementation of next tedious (although not impossible). So I'm waiting on the author to add that call; if that happens, we could have a nicer on-disk set-type table as well.

Everything is available on google code: mnesiaex (storage API) and tcerl (erlang port of Tokyocabinet).

Also Daisy (aka Joel Reymont) is really nice to work with.