Monday, June 23, 2008

Tokyocabinet and Mnesia

As Daisy has already indicated, it is now possible to plug arbitrary storage strategies into Mnesia. For those who are familiar with mnesia_access, this is different; mnesia_access only covers reads and writes, not schema manipulations, and has other deficiencies that it render it useless for adding a new storage type in practice (what mnesia_access is great for is changing the semantics of mnesia operations, e.g., mnesia_frag). This project lets you make tables that are essentially indistinguishable from the built-in mnesia table types (ram_copies, disc_copies, disc_only_copies).

Anyway our goal was to get a good on-disk ordered_set table type, since we've found ordered_sets very useful for the kinds of problems we're solving but we're tired of being limited by memory. After looking around for a while Tokyocabinet emerged as our favorite for underlying implementation. We considered BDB and libmysql, but Tokyocabinet seemed simple and faster, and had a more accommodating license. So we ported Tokyocabinet to Erlang and then used the above storage API to connect to Mnesia.

As a side benefit Tokyocabinet might also be preferred to dets even for set-type applications because of the lack of file size limit and high performance. Tokyocabinet actually has a set-type storage strategy that we'd like to define an Erlang Term Store for, but as of this post, the set-type store doesn't support cursor positioning based upon a key, which makes the implementation of next tedious (although not impossible). So I'm waiting on the author to add that call; if that happens, we could have a nicer on-disk set-type table as well.

Everything is available on google code: mnesiaex (storage API) and tcerl (erlang port of Tokyocabinet).

Also Daisy (aka Joel Reymont) is really nice to work with.


just_ziz said...

Such a nice work!

Are you going to include tokyocabinet transaction support to tcerl, or use Erlang code to provide isolation levels?

besquared said...

I have been pretty enamored with tokyo cabinet (since it was QDBM) and now I'm getting into erlang due to messaging systems like ejabberd.. I can't believe I ran into this post.. it's like.. a 1 in a million find for me..

I had a few questions about erlang and its distributed database frontend Mnesia

1) What kind of queries/operations does an ordered_set table type allow?
2) Can mnesia support large databases? Say, several terabytes in size across a cluster of dozens of machines
3) Can it support a large number of queries? Say millions a month?

Krzysztof (Christopher) Kliƛ said...

A respectable piece of work - thank you for making it open source!