Eric Raymond on “Escalating complexity and the collapse of elite authority”

“Open source” software guru Eric S. Raymond blogs on  how the ever-increasing complexity of the world is leading to catastrophic inability of elite authority to cope:

Our “educated classes” cannot bring themselves to come to grips with the fact that fundamentalist Islam has proclaimed war on us. They have run the economy onto recessionary rocks with overly-clever financial speculation and ham-handed political interventions, and run up a government deficit of a magnitude that has never historically resulted in consequences less disastrous than hyperinflation. And I’m not taking conventional political sides when I say these things; Republicans have been scarcely less guilty than Democrats.

In the first month of a new decade, unemployment among young Americans has cracked 52% and we’re being officially urged to believe that an Islamic suicide bomber trained by Al-Qaeda in Yemen was an “isolated extremist”.

One shakes one’s head in disbelief. Is there anything our “educated classes” can’t fuck up, any reality they won’t deny? [Will Collier] fails, however to ask the next question: why did they fail?

The obvious and most tempting hypothesis for a libertarian student of history like myself is that the Gramscian damage caught up with them. And I think there’s something to that argument, especially when the President of the U.S. more beloved of those “educated classes” than any other in my lifetime routinely behaves exactly as though he’d been successfully conditioned to believe the hoariest old anti-American tropes in the Soviet propaganda arsenal. And is praised for this by his adoring fans!

I think there’s much more to it than that, though. When I look at the pattern of failures, I am reminded of something I learned from software engineering: planning fails when the complexity of the problem exceeds the capacity of the planners to reason about it. And the complexity of real-world planning problems almost never rises linearly; it tends to go up at least quadratically in the number of independent variables or problem elements.

I think the complexifying financial and political environment of the last few decades has simply outstripped the capacity of our “educated classes”, our cognitive elite, to cope with it. The “wizards” in our financial system couldn’t reason effectively about derivatives risk and oversimplified their way into meltdown; regulators failed to foresee the consequences of requiring a quota of mortgage loans to insolvent minority customers; and politico-military strategists weaned on the relative simplicity of confronting nation-state adversaries thrashed pitifully when required to game against fuzzy coalitions of state and non-state actors.

[…]

In retrospect, I think race- and class-blind meritocracy bought us about 60 years (1945-2008) of tolerably good management by Western elites. The meritocracy developed as an adaptation to the escalating complexity of 20th-century life, but there was bound to be a point at which that adaptation would run out of steam. And I think we’ve reached it. The “educated classes” are adrift, lurching from blunder to blunder in a world that has out-complexified their ability to impose a unifying narrative on it, or even a small collection of rival but commensurable narratives. They’re in the exact position of old Soviet central planners, systemically locked into grinding out products nobody wants to buy.

My readers might well ask, at this point, “Great. Does this mean we’re screwed?” If a meritocracy drawn from the brightest, best-educated people in history can’t cope with our future, what do we do next?

The answer is, I think implied by three words: Adapt, decentralize, and harden. Levels of environmental complexity that defeat planning are readily handled by complex adaptive systems. A CAS doesn’t try to plan against the future; instead, the agents in it try lots of adaptive strategies and the successful ones propagate. This is true whether the CAS we’re speaking of is a human immune system, a free market, or an ecology.

Since we can no longer count on being able to plan, we must adapt. When planning doesn’t work, centralization of authority is at best useless and usually harmful. And we must harden: that is, we need to build robustness and the capacity to self-heal and self-defend at every level of the system. […]

The people who will resist the end of the engineered society, the managed economy and the paternalist state are, of course, the “educated classes” themselves. Having become accustomed to functioning as the aristocracy of our time, they will believe in anything except their own obsolescence as rulers. It remains to be seen how much longer events will permit their delusions to continue.

Indeed.

UPDATE: some nuggets from the comments can be seen by hitting “Read more”. And yes, “sophomore” (literally “wise fool”) is not a bad description of the self-appointed “anointed/enlightened”…

Scott Says:
January 5th, 2010 at 9:40 pm
ESR, you should be promoted as a national treasure. Really.

I have two points. The first is that I don’t buy the inherent implication of the term “educated classes”. That is to say I completely agree that the “educated classes” think/believe they are “educated” and the intellectual superiors. I also agree that the “unwashed masses” likely think/believe that also. But in my completely biased, limited, and arrogant experience, the members of the “educated classes” I have ever met in the real and came to know through the Information Age are morons. Smart maybe, but idiots. Is there a proper word for that yet? We need one.

Further, I believe they have committed possibly what “they”, themselves, would likely offer as the ultimate crime of their predecessor -ocracy: they have become their own “good ole boy network”. On both sides of the aisle, and in about the only reflection of bipartisanship, they work tirelessly to block the entry of America’s actual, legitimate intellects into their power structure. And this happens at all levels of governance in this country.
[…]

William H. Stoddard Says:
January 5th, 2010 at 10:57 pm
Scott: You ask if there is a name for people who are smart, but idiots. J. R. R. Tolkien offers us one: “wise fool” (a literal translation of the Greek sophomoros from which we get “sophomore,” a student who has learned just enough to be overconfident). And he offers us a classic portrait of the type in Saruman the White, to whom Gandalf applies that epithet.

Buck O’Fama Says:
January 5th, 2010 at 9:47 pm
I don’t think meritocracy is the problem, for meritocracy implies there is a reason for someone to be in the position he is in. The problem is today we have pseudo-meritocracy, which means that many are in high positions not because they were good at something but have credentials and degrees from universities which are regarded as a substitute for being good at something. But many of these people are hyper-educated nitwits, good at reading books and parroting back what they were fed but no good at thinking. Obama is the poster boy for the problem. His only response to problems is the poop his professors told him, but when that doesn’t work he has no Plan B nor any idea of how to come up with one. You can see this attitude in so many other areas: the nitwits who packaged low-quality mortgages together and came up with a triple AAA rated bond…. How the hell does that work? It’s because they all blindly relied on a stupid statistical formula which had some invalid assumptions in it. Anyone who bothered to THINK would’ve seen it was nonsense. But fewer and fewer people these days seem to know how to do that.

Peter A. Taylor Says:
January 5th, 2010 at 10:32 pm
bruce wrote:

> Obama et al are playing a moral dominance game based on: ‘I’m morally better than you because of my voluntary poverty of mere simple-minded patriotism’.

I’ve been thinking along the same lines. For your reading pleasure, I give you “The Market for Sanctimony (or Why We Need Yet Another Space Alien Cult).”

esr Says:
January 5th, 2010 at 11:20 pm>http://home.earthlink.net/~peter.a.taylor/midrash.htm

Wow. Lots of good insight and cut-through-the-bullshit there, and I’m not saying that because you quote me a lot either. You’ve just joined the very select group of people who can clearly do the kind of reasoning I do about human behavior as well or better than I do; the only two others I can think of offhand are David Friedman and Eliezer Yudkowsky. I have only minor disagreements, mostly about the usefulness of Christianity.

I can answer one of your questions. When you ask of a hypothetical black man “What institution could you walk into without having your color tallied up as a credit to the institution?” he could honestly answer “the open-source movement”. We truly don’t give a f*ck, and I once personally cooperated with a black Linux hacker to squash somebody who wanted to import racial politics and racial sanctimony into the movement. (The somebody was and still is a friend of mine, so it was kind of uncomfortable to squash her. But the black man and I both understood it to be necessary.)