[flud-devel] DHT justifications (was: DHT Performance? Design?)
zooko at zooko.com
Tue Nov 6 20:34:03 PST 2007
> flud recently switched
> to zfec for erasure coding (created by zooko for Tahoe). Its
> performance is quite satisfactory compared to rizzo, but of course has
> to pay the python wrapper penalty. You can find benchmarks in the
> expected places
For the record, I started zfec by copying Rizzo's code. I tried to
simplify and optimize his code, but I never did solid benchmarks of
the result, or if I did I don't remember and I no longer have my notes.
Also, you might be interested in this ticket:
http://allmydata.org/trac/tahoe/ticket/167 -- try out Jerasure
Professor James Plank has published a new library that has
implementations of various erasure coding algorithms, Jerasure:
It would be fun to benchmark these against the current zfec
and if some algorithm in there is faster, switch the internals of
zfec to use it instead.
I benchmarked an earlier library of Professor Plank's (with the help
of someone named Alen, as it happens) for Mnet back in the day, and
it was slower than Rizzo for the parameters that I was interested in
at the time. Maybe the new one is faster. Also the parameters that
I am interested in has changed...
By the way, I've enjoyed following this discussion. Tahoe currently
doesn't have a DHT implementation at all. There is a DHT design note
-- http://allmydata.org/trac/tahoe/browser/docs/denver.txt -- but we
currently aren't attempting to scale up the number of active nodes.
We're currently focussed on the "friendnet" and "proprietary grid"
use cases -- http://allmydata.org/trac/tahoe/wiki/UseCases .
More information about the flud-devel