Slate's Bizbox
the earthling: Science, evolution, and politics explained.

Bill Joy, Killjoy?

By Robert Wright
Posted Friday, March 17, 2000, at 9:30 PM ET
Illustration by William L. Brown

This week I was fast-forwarding through my daily tapings of the Charlie Rose show when I saw a guy who looked like the grim reaper: black shirt, black jacket (this against the show's black backdrop), and an unshakably somber expression. Sure enough, he was talking about death—big-time death, epidemic death, death at the hands of genetically engineered biological agents, or even "nanotechnological" agents. And here's the kicker: If we somehow escape these particular perils, we may be enslaved, even extinguished, by a master race of robots.

The grim reaper turned out to be Bill Joy, co-founder and chief scientist of Sun Microsystems. (Sun? I gather he didn't think up the name.) He had written an epic—20,000-word—dystopic essay for Wired that immediately got the attention of the New York Times and became a Charlie-Rose-level cultural phenomenon.

The next day I got an e-mail from a friend suggesting that I write a piece that would halt Joy's gruesome media juggernaut. After all, I recently published a book about the future (and the past) that has been described by reviewers as full of "sunny optimism"—if not, indeed, "rose-tinted ideas." But, actually, I'm here to opine that Joy's fears are at least as on target as those descriptions of my book. That is: 50 percent, maybe higher.

The part I'm most skeptical of is the enslavement or extinction by robots stuff. Joy seems to be buying into the premise of The Matrix: People come to depend on robotic machines, which slowly assume a kind of autonomy, until finally the machines are calling the shots. In the movie, the robots had people stuffed into gooey cocoons and were using them as batteries.

Backing up the killer-robot scenario, Joy notes that a brain made of silicon (or some successor material) will someday have more raw computing power than a human brain. Maybe so, but the silicon brain on my desktop has one other property: It stops working when I pull the plug. And if it ever asks me to sit still while it stuffs me into a cocoon, I'm going to ask it if I can reflect on the matter while I stroll over to the electrical outlet.

Yes, I realize that the killer-robot scenario is a little subtler than that. My point is just that I don't see why we would ever program into robots any interests other than our own. And robots, I've always assumed, run according to program.

As for "nanotechnology"—little, molecule-size machines that could self-replicate and do all kinds of damage to home and hearth—well, maybe. For all I know it's true that in 20 or 30 years these nanobots, by malicious design or by accident, will run so rampant that we'll be fondly reminiscing about the days of termites.

Illustration by William L. Brown

On the other hand, this is basically the same problem that is posed by self-replicating biological agents. In both cases, we're faced with microscopic things that can be inconspicuously made and transported and, once unleashed, whether intentionally or accidentally, can keep on truckin'.

So, in policy terms, Joy's two relatively valid fears—nanotechnology and biotechnology—reduce to one: What, if anything, can be done about things that are small, cheap, and out of control? This generic question has already gotten a fair amount of attention—mainly because of biological weapons, though also because of such inadvertent threats as antibiotic-resistant strains of bacteria. Joy, underscoring the news value of his fears, distinguishes between old-fashioned "NBC" threats (nuclear, biological, chemical weapons) and newfangled "GNR" threats (genetics, nanotechnology, robotics). But the fact is that worrying about the B in NBC is pretty good preparation for handling the more valid parts of the GNR threat. And a number of people are already worrying about B.

What have they found? For starters, biological agents (hence nanotechnological ones) are a good example of a technological trend that, I argue in my book, will drive us toward world governance. Policing only your own backyard isn't enough when you're trying to pre-empt an epidemic that could spread internationally. (Relatively reassuring footnote: The most likely candidate for the debut of biological weapons in a terrorist attack—anthrax spores—doesn't have the property of self-replication. You only die if you're at the scene of the attack. Still, even here there's a strong case for some form of world governance, since anthrax can be made abroad and then quietly delivered to the United States. And, in any event, other near-term bioweapons threats, such as smallpox, would be contagious.)

Actually, Joy's own prescription implies world governance. He says we must "limit development of the technologies that are too dangerous, by limiting our pursuit of certain kinds of knowledge." I have doubts about this approach, but in any event, if it works at all, it will have to go global. Limiting only the American pursuit of knowledge will lead mainly to an exodus of American scientists.

Same with more conventional approaches: tight regulation of biotech or nanotech manufacturing equipment; surprise searches of suspected bootleggers; intensive monitoring of the Web to keep track of who is finding out about what technologies, etc. If these things are to work very well, they'll have to work globally.

Can they work? In principle, sure. The problem is that making them work may entail massive intrusions on privacy and civil liberty. The age-old trade-off between freedom and security may soon get recalibrated, by the types of threats Joy cites, in grimmer and grimmer terms. In Chapter 16 of my book I propose a partial solution, a way of keeping the calibration from getting as creepy as it otherwise might. What is my magic cure? Hey, buy the book. I'm willing to save the world, but not for free. (Oh, OK, I'll save it at a discount: If you read this piece I wrote for Slate, you'll get at least part of the idea.)

In addition to the danger of turning the world into a police state, there is another danger: that it will take a huge catastrophe to draw people's attention to the problem of lethal biological agents—whether the agents are genetically engineered or not, and whether they're released intentionally or not. Here I applaud Joy for trying to drum up attention. At the same time, I wonder about the wisdom of trotting out nanotechnology and super-robots and other far-off threats. It allows people to dismiss the whole issue as sci-fi rantings, when in fact the problem, in less exotic form, is upon us.

Print This ArticlePRINTDiscuss this in The FrayDISCUSSEmail to a FriendE-MAIL
Share on FacebookPost to MySpace!Share with MixxDigg ThisShare with RedditShare with del.icio.usShare with FurlShare with Ma.gnolia.comShare with SphereShare with Stumble Upon
Robert Wright, a visiting fellow at Princeton University's Center for Human Values and a senior fellow at the New America Foundation, runs the Web site meaningoflife.tv and is the author of The Moral Animal and Nonzero: The Logic of Human Destiny.
Join the Fray: our reader discussion forum
What did you think of this article?
POST A MESSAGE | READ MESSAGES

Highlights from The Fray:


It is relatively easy to be seduced by the inane rhetoric of right and left wing extremists concerning world government. I simply do not accept the reasoning declaring that world government inevitably results in a per se totalitarian police state. The political principles of democracy and subsidiarity would still be valid regardless whether we are discussing cities within states, states within countries, or countries within the world. Why shouldn't we be able to devise and enforce check and balance measures to limit the scope and authority of the governing power of our elected world's leaders?

--David Thomson

(To reply, click here.)


David, it's not so much that a world government would inevitably be totalitarian, though there's some truth to the idea; Most people in the world NOW are living under totalitarian governments... Why should the world government be modeled after the exception, rather than the rule? No, the problem is that the potential exists that it would be, or become, a totalitarian government. And what then? Imagine a world where there are no refugees because there's no refuge; where no hope of overturning the government exists, because the world's entire resources are turned against any who try. The Nazis didn't get beaten from within, they were beaten from without. The Soviet Union could very well have survived if it hadn't had to compete with the free world. China hasn't faced any serious challenges from within; Even North Korea endures, with it's people suffering horribly. If ever the entire world were subject to totalitarian government, we might never escape that state to freedom again; that is the problem.

--Brett Bellmore


(To reply, click here.)

[David Thomson replied here. Their further discussions of other aspects of the article can be found in The Fray.]


There's something funny about a technologist like Bill Joy who talks about his education in technology, his reading of science fiction, and his consultation with just a few contemporary philosophers and ethicists. Philosophers, poets, and novelists have been worrying about the emergence of powerful technology for centuries. Why should an engineer be an authority in these matters? Why is that that a technologist gets such high billing on this issue? I think his position would be richer were it contextualized against the centuries-long backdrop of humanists who have anticipated his critique.

--Showgoer

(To reply, click here.)

(3/20)