[flud-devel] flud goes zfec

[Alen Peacock]

I've long maintained that LDPC/LDGM erasure codes were a good fit for
a system such as flud (http://flud.org), and that their better
encode/decode performance was a good tradeoff for the slightly higher
average inefficiency ratio when decoding.

I now wish to recant all that.

It is true that LDPC/LDGM can produce an /average/ inefficiency ratio
that is very acceptable, even when using only a small number of
blocks.  My mistake was in assuming that the variance in this average
could be controlled through careful choice of parameters.  Based on
some preliminary experiments, that seemed to be true, but I've
recently done some better testing, inspired by anomolies that I was
seeing for decode inefficiency in flud, and realized that there wasn't
a configuration that could provide an acceptable variance, at least
not when using a small number of blocks (see thread at
http://flud.org/pipermail/flud-devel_flud.org/2007-September/000048.html).

flud is now using the zfec library generously provided by zooko
(http://allmydata.org/source/zfec/), which is also used in tahoe.  So
far, it is working fabulously.  It was fairly easy to replace flud's
ldpc lib with zfec, though the semantics (especially for decode) are
quite different.  flud performs as before, except that the blocks it
produces will naturally be incompatible, and that instead of needing
(k*inefficiency_ratio)+uncontrollable_variance blocks to decode, it
now only needs exactly k (yay!).

The next release of flud was going to consist mainly of a bunch of
ldpc-related encoder fixes.  Now, thanks to zfec, those issues are
already resolved and we can get some other [more interesting] stuff in
before the next release.

Alen