Cloud Durability?

Cloud computing might not be able to deliver on its high-availability promises? Who would’ve thunk it?

The future is not to rely on a single entity to provide hosting, scaling, and reliability, but rather to rely on a multitude of distinct entities. That way, if one or two go down — or even if many go down — service continues uninterrupted, as if by magic.

Posted in decentralization, resilience | 3 Comments »

Dozenal FTW!

I’m happy to present the world’s first online dozenal conversion calculator.[1]

For a number of years, I’ve entertained myself in conversations with friends and family by taking the preposterous position that we’d have been better off, maths-wise, if we’d all been born with 12 fingers instead of 10.

Since that clearly didn’t happen, I’ve maintained hope by vowing that if I ever get my hands on a time machine, one of my first acts-of-historical-correction would be to fly back to early caveman days, find the first people who started counting with their hands, and teach them to eschew fingers in favor of finger segments (of which there are conveniently 12), using the thumb of the same hand to touch the current count on the appropriate finger segment.[2]

With this method and using the second hand as a 12′s placeholder for counts greater than 12, you can actually achieve the incredible sum of 144 (which is “100″ in dozenal) using nothing more than two hands!

But, alas, I have no time machine. So I will instead have to settle for the world’s FIRST online dozenal/decimal calculator.

For me, the most alluring argument in favor of the dozenal system revolves around factorization. A base-10 measuring system can only be factored into integer halves and fifths. A base-12 number system, on the other hand, is easily divisible into halves, thirds, fourths, and sixths. And when it comes to measurement, there really aren’t any more important divisors than thirds and fourths. What did you do to the Rule of Thirds, base-10? Seriously, metric-system, show me where to make the mark for 3.3333333… on the wall.

In fact, when one considers number-of-factors as the basic measure of a base-N system, it quickly becomes apparent that base-12 gets you the most bang-for-the-buck. You have to jump all the way up to base-60 to get your next most important factorizations, but base-60 is a bit unwieldy.[3] Base-2, Base-3, Base-8, Base-10, Base-15, Base-16, Base-24, Base-30, etc — all have their unique features and settings where they are the right choice, but none are as naturally ready for human consumption as Base-12.

Still not convinced that Base-12 is Teh Rawk? Imagine replacing timekeeping with a Base-10 system. I know, I know, there are those who have tried it (French Revolutionaries, crazy internet dudes), but really, let’s think about this for just a minute. With a base-10 timekeeping system (for example, 100-minute hours), you can’t say anything equivalent to “see you at 20 after 5″ — at least not without sounding like a moron: “see you at 5:33-and-a-third?” You’d be left with units of half-base10-hours and fifth-base10-hours. Trust me, that’d suck.

If you still aren’t sure that I’m right (even though I am), take a minute or two to watch a bit of Schoolhouse Rock for some real edumacation.

See also the Dozenal Society of America.

(Note: this post has little, if anything, to do with flŭd decentralized backup – except perhaps that both rely on the art of non-base10 mathematics).

[1] At least, it is the first online dozenal-decimal calculator that I am aware of, and, apparently, the first that google is aware of too. Searches for “dozenal|duodecimal|base12|base-12 [online] calculator” don’t yield anything meaningful for me, although the Dozenal Society of Great Britain does have a downloadable calculator for MSWindows available. Also, this online calculator is really more of a converter than anything. There are generic radix integer converters out there that can convert between many different bases, such as this or this, but none can even handle the simplest non-integer values. If you are aware of another online dozenal/decimal conversion calculator, please let me know so that I can retract my outrageous claims and give due credit. UPDATE 2009-03-14: Ray Greaves points to his very fine base-2, 3, 8, 10, and 16 converter in the comments below, which had many other bases (including 12) added sometime in 2008 (not sure if it beats the Jan 12 2008 date of this posting, but either way, check it out, it’s a very nice piece of work).

[2] I haven’t worked out the details of how I’d actually go about reasoning with cavemen. If anyone knows of studies on the comparative efficacy of different teaching techniques on primitive man (perhaps performed in secret government cloning labs), forward plzkthx.

[3] Just ask the Babylonians. Actually, base-60 remains very useful today in some settings, such as timekeeping. And its large cousin, base-360, is a very close friend of navigators and geometers.

Posted in humor, mathematics | 11 Comments »

flŭd backup 0.2.0 released

Just a quick note in case you missed it: 0.2.0 was released last night.

Posted in flud releases | 1 Comment »

Decentralized Resilience

I’ve written at some length about the importance of resilience in flŭd’s design, and how complete decentralization is a key component of that resiliency. One of flŭd’s hallmark goals is to back up data in such a way that it would be virtually impossible to lose it — even if a very powerful adversary (including an oppressive government regime or an extensive natural disaster) disables large portions of the flŭd network.

Most software is not designed to have this type of resiliency, and the excuse is simple: most software does not face adversarial forces. Or, at least, its designers think that it won’t (an assumption that many times leads to disaster).

There is one class of software, however, which meets adversity as part of its raison d’être: malware. Now, of course, flŭ­d’s purposes are at the polar opposite of software such as the storm botnet, but I can’t help admire, at least from the standpoint of technology, some of the self-preservation techniques employed by malware such as Nugache in avoiding eradication. It seems that many anti-malware researchers share my reluctant admiration. From SearchSecurity.com:

Dittrich, one of the top botnet researchers in the world, has been tracking botnets for close to a decade and has seen it all. But this new piece of malware, which came to be known as Nugache, was a game-changer. With no [centralized command-and-control] server to target, bots capable of sending encrypted packets and the possibility of any peer on the network suddenly becoming the de facto leader of the botnet, Nugache, Dittrich knew, would be virtually impossible to stop.

Posted in decentralization, resilience | Comments Off

flŭd backup 0.1.1 released

flŭd version 0.1.1 is now available.

This is mainly a bugfix release focusing on the graphical frontend:

  • fludrules.init, used as a template to select/exclude a default set of files in $HOME on first run of fludclient, was missing from distributed packages.
  • changes in wx (from version 2.6 to 2.8) caused the rendering of the DirCtrlCheckbox custom widget to be too tall and too thin, making fludclient difficult to use on systems with 2.8 due to strange default window sizes

In addition, 0.1.1 is now available via YUM for Fedora 8 (in addition to Fedora 6, and via APT for Ubuntu 7.10).

Detailed release notes are available

Posted in flud releases | Comments Off

flŭd backup 0.1.0 released

flŭd backup 0.1.0 is now available (via tarball, YUM/RPMs, apt/debs, or SVN snapshot)

Improvements include: trust system / peer affinity added, replaced LDPC decoder with RS (zfec), better packaging (YUM/apt), and various bugfixes. For details see the release notes.

Posted in flud releases | Comments Off

Eternal Storm

If you haven’t heard of the Storm Botnet yet, chances are you will soon. With between one and 50 million nodes infected with this trojan, it forms the world’s most powerful supercomputer. Criminal elements are believed to control Storm, but that’s not why it is interesting.

The Storm Botnet is fascinating because of its resilience. As Bruce Schneier points out, antivirus companies haven’t figured out a way to put a dent in its propagation or effectiveness, even though they have known about it for almost a year, and even though it has already been used to send millions (perhaps billions) of spam emails and carry out several high-profile DDoS attacks. Getting rid of such a beast would clearly be very lucrative work for the antivirus industry, yet no one has devised a successful solution.

What makes this thing so indestructible? Schneier explains a key component:

Rather than having all hosts communicate to a central server or set of servers, Storm uses a peer-to-peer network for C2. This makes the Storm botnet much harder to disable. The most common way to disable a botnet is to shut down the centralized control point. Storm doesn’t have a centralized control point, and thus can’t be shut down that way.

This technique has other advantages, too. Companies that monitor net activity can detect traffic anomalies with a centralized C2 point, but distributed C2 doesn’t show up as a spike. Communications are much harder to detect.

One standard method of tracking root C2 servers is to put an infected host through a memory debugger and figure out where its orders are coming from. This won’t work with Storm: An infected host may only know about a small fraction of infected hosts — 25-30 at a time — and those hosts are an unknown number of hops away from the primary C2 servers.

And even if a C2 node is taken down, the system doesn’t suffer. Like a hydra with many heads, Storm’s C2 structure is distributed.

In combination with some of its other novel features, Storm’s decentralization gives it a level of immunity to shutdown previously unseen anywhere.

It’s a shame that a malicious tool like Storm is providing such a stunning demonstration of the advantages of decentralization in the wild. But these techniques are not limited to malicious software. Indeed, major components of the Internet itself use the same techniques. flŭd backup is 100% decentralized for the same reasons: the data that you backup with flŭ­d should be as indestructible as possible. Decentralization is the most effective path to that goal, and Storm has provided us with ample evidence that such a scheme can be extremely effective.

Posted in decentralization, resilience | Comments Off

flŭd backup 0.0.2 released

flŭd backup 0.0.2 is now available (via tarball, RPMs, debs, or SVN snapshots)

A slew of improvements and features have been added. For details see the release notes.

Screenshots are also available.

Posted in flud releases | Comments Off

P2P’s Skype-induced Blackeye, or Why Diversity is Good

On August 16th, Skype disappeared for two days and the company blamed the outage on Skype’s p2p network, the nodes of which received updates worldwide at the same time, restarted, and overwhelmed the system.

Rather than blaming the decentralized nature of their p2p network, Skype should be pointing the finger of blame at the true cause of this extended outage: Skype’s own centralized control and distribution of their software, and Skype’s closed, proprietary protocol, which ensures that no clients other than the official Skype client software exist. The Skype ecosystem is homogeneous, and as such faces problems similar to those found during the famous potato famine in Ireland, or the current problems facing genetically homogeneous banana crops: a population that lacks diversity lacks resilience to disease and adversity. When all the individuals in a population are clones or close relatives of each other, every single member of the population is susceptible to a single disease, and it can spread like wildfire. In Skype’s case, the massive simultaneous restart of nodes apparently sparked a constant flame of log-in requests, exposing another weakness in Skype’s architecture: since Skype must centrally manage accounts for business reasons, all account management, including log-in authentication, must be centrally performed. Authentication servers in Skype are distributed, but not decentralized. Centralized components do not scale well, especially during heavy spikes in usage patterns.

Contrast this with the gnutella or bittorrent networks, in which no single entity controls all the clients. There are perhaps dozens of different active clients for each network, and if there is a fatal bug in one version of one client, it only affects the portion of the population running that version of that client. Diversity is good for the ecosystem, and always will be.

I’ve claimed before that decentralization can eradicate service outages. But simply decentralizing some portion of a system clearly does not make it immune to serious outage problems. Decentralization without open protocols (and usually by corollary, open software) is not decentralization at all. Decentralization is not made of magic pixie dust, but it will overcome this type of catastrophe if applied correctly.

This is why the flŭ­d protocol is open, and why the software is open source; we hope that others will eventually write clients not only for other languages and platforms, but clients that implement different strategies for maximizing their efficiency and trading relationships.

Hamstringing a p2p application with centralized SPOFs (single points of failure), and then blaming decentralization as the cause of problems that really stem from centralized bottlenecks is, at the very least, disingenuous. But I understand Skype’s predicament — it’s much harder to say, “our system failed because our business model fundamentally weakened our technology” than it is to say “we had a bug in our p2p application.”

Posted in decentralization, resilience | 2 Comments »

I haz a flud

I cannot resist posting this lolcat reference that has, at once, everything and nothing to do with flÅ­d backup. From gothamist.

i haz a flud

Posted in Uncategorized | Comments Off