Tuesday, February 12, 2008

walkenfs on google code

We Dukes are into cloud computing. We especially like our machines to be operationally symmetric so we don't have to think that hard. Erlang makes that easy but unfortunately sometimes we have to deal with software not written in Erlang.

In particular Fess started to put together a monitoring system lately using rrdtool which is great but wants to operate on files. We'd done all this work to make mnesia play nicely on EC2 (more on that later) and suddenly we seemed back at square one. We considered storing data in mnesia and rendering files at the last minute for rrdtool; but then we noticed fuse. Sweet! Now we can store data in mnesia and make it look like a filesystem at the same time.

First we needed Erlang bindings for fuse, so I wrote those. That turned out to involve writing C code, a moderately frustrating experience that served as a nice reminder why we don't call ourselves the "Dukes of C". Once the nasty bits were subdued, writing the actual Erlang code corresponding to a filesystem via mnesia was comparatively quick and painless. I'm still trying to figure out the best way to support posix locking (right now, I'm looking for a satisfying way to have the locks release when an entire node dies unexpectedly), but everything else seems to be working.

It still needs alot of polish. As a filesystem one would expect a mkfs tool and the ability to put entries in /etc/fstab that work. However it's usable as part of an Erlang system right now.

1 comment:

Matthew said...

There seems to be considerable chatter about capacity limits of mnesia. One person I speak with says 4GB, another 7GB.

What do you think about walkenfs storage capacity?