Human Nature: Science, Technology, and Life.

Freeway Autopilots

Maybe one lesson of Northwest Flight 188 is that driving should be more like flying. There are tricky parts that require close attention and adjustment—takeoff and landing for planes, local navigation for cars—and then there are parts that can be automated: cruising altitude for planes, freeway driving for cars. A freeway autopilot would have to keep you in your lane until your exit. It wouldn't zip around from lane to lane like a jerk. But I'm guessing most of us could live with that.

More here.

Bombs, Innovation, and Afghanistan

In today's New York Times, James Dao reports that Afghan IEDs

are becoming more common and more sophisticated with each week, American military officers say. This year, bomb attacks on coalition troops in Afghanistan have spiked to an all-time high, with 465 in May alone, more than double the number in the same month two years before. At least 46 American troops have been killed by I.E.D.'s this year, putting 2009 on track to set a record in the eight-year war. ... At the current rate, I.E.D. attacks on Afghan forces could reach 6,000 this year, up from 81 in 2003, an American military official said.

At least three of the lessons we drew from Iraq seem to apply in Afghanistan. First, IEDs enable insurgents to strike with a level of precision that would be impossible from a distance. Second, IEDs can be assembled from inexpensive, readily available components, such as fertilizer, artillery shells, and cell phones. Third, instead of risking human lives, you can hunt or disable IEDs with dogs or robots. The bomber isn't risking his life. Why risk yours?

Afghan insurgents are exploiting the same cheap technology that worked in Iraq. According to Dao, "The bombs are often made with fertilizer and diesel fuel, but some use mortar shells or old mines that litter the countryside. Some bombs are set off when vehicles pass over pressure plates. Others require remote control, like a cellphone. Still others detonate with a button or a wire touched to a battery." Likewise, we're using familiar detection methods: dogs, robots, and drones.

So what has changed? One difference is a lower level of technology in Afghanistan. Sometimes this works to the insurgents' advantage:

With few paved roads, Afghanistan is even more fertile territory for I.E.D.'s. than Iraq, where hard pavement often forced insurgents to leave bombs in the open. Not so in Afghanistan, where it is relatively easy to bury a device in a dirt road and cover the tracks.

But it can also make them vulnerable. U.S. officials tell Dao that IED deployment networks connect top layers of "financiers, logistical experts, bomb designers and trainers" with lower layers of "bomb planters, often villagers or nomadic herdsmen paid $10 or less to dig holes and serve as spotters." The weak link is the top layer. In Afghanistan, there may be fewer people with the expertise to run such networks than there were in Iraq.

To get the key players, you have to operate like a crime scene investigator. Dao reports:

Like a police forensic unit and a bomb squad rolled into one, Lieutenant Brown's 25-member team not only disarms I.E.D.'s but also scours sites—more than 50 this year—for telltale signatures of a bomb. Soil samples, electrical parts, fingerprints and photographs are sent for analysis, and detailed reports are compiled in a central database.

This is one of the main questions being tested in Afghanistan: Can forensic investigation and a pooled database unravel IED networks? Can high-tech police work catch the experts and organizers instead of settling for the suckers who plant the bombs? IEDs, like drones, are an evolving story of measures and countermeasures, technologies of destruction and technologies of detection. We don't know how the story will turn out. But we know which weapon will prevail. It won't be a device. It will be a process, a talent, and an attitude: innovation.

Married to the Machines

Shankar Vedantam is a people person. I don't mean that in the ordinary sense, as in, "Do I look like a @#$% people person?" I mean that, in addition to being delightful company, he writes about people. He's interested in how we think.

So when he writes about machines, as he did last week, something funny is going on. His topic was the recent Metro train crash in Washington, D.C., which killed nine people. How did it happen?

One theory is the automation paradox:

The more reliable the system, the more likely it is that humans in charge will "switch off" and lose their concentration, and the greater the likelihood that a confluence of unexpected factors that stymie the algorithm will produce catastrophe. ... After the previous fatal accident on Metro, in which a train overshot the Shady Grove station on an icy night, the National Transportation Safety Board found that the driver of the train had reported overshooting problems at earlier stops but was told not to interfere with the automated controls.

In that case, automation researcher Greg Jamieson points out, "For a year before the accident, the transit authority had put in position a directive that you were not to drive the train in manual." Vedantam concludes: "No matter how clever the designers of automated systems might be, they simply cannot account for every possible scenario, which is why it is so dangerous to eliminate human ‘interference.' "

This is the problem we discussed after the February plane crash that killed 50 people near Buffalo, N.Y. Initial evidence indicated that the pilot misunderstood what the autopilot was doing, and, by overriding the machine, caused the crash. Further evidence presented at a May hearing confirms that

the plane, which was collecting ice on its windshield and wings, was slowing to an unsafe speed. But when a warning system began vibrating the control column to get their attention, the captain pulled the nose up when he should have pushed it down. ... [C]onfronting the vibrating column, called a stick shaker, was probably something new and startling. The airline that was contracted with Continental Airlines to make the one-hour commuter flight, Colgan Air, said on Wednesday that it had given the crew simulated training in the activation of the stick shaker, but not in the next step, activation of the stick pusher, which takes control and pushes the nose of the plane down. In this instance, the stick pusher kicked in shortly after the captain pulled instead of pushed. "I don't see any evidence that he ever understood the situation he was in," said Dr. Dismukes ...

Shortly after the Buffalo crash, I outlined three possible responses to such disasters. One was take the controls away from the machines, on the grounds that difficult conditions require human attention and judgment. The opposite approach was to take the controls away from the humans, on the grounds that pilots can't be trusted to override the machine's superior judgment. A third, hybrid solution was to teach the humans how to read and collaborate with the machine's intentions.

The third approach seems to be the one most clearly supported by the evidence in the Buffalo crash: Flight crews must be trained to interpret and interact with their autopilots. Vedantam makes a similar point about automated systems in general: "Several studies have found that regular training exercises that require operators to turn off their automated systems and run everything manually are useful in retaining skills and alertness." We have to know when to second-guess our machines and how to operate without their help. Sometimes, they'll err fatally unless we intervene. But our intervention can itself be fatal. The key is to understand when to step in and when to butt out. That's the role of human intelligence in a machine-controlled world.

It's fun to go to summer sci-fi movies and wonder whether humans or machines would prevail in a mortal showdown. But in the real world, machines aren't our enemies. They're our collaborators. If those 50 people in Buffalo died in a fight between a human and a machine, it wasn't a fight chosen by either side. It was a misunderstanding. And since we're the ones who made the machines, it's our job to teach one another how to work with them, around them, and without them.

God Is My Autopilot

That's one possibility emerging from for last week's passenger jet crash in Buffalo, N.Y. So far, the investigation has found no mechanical or autopilot failure. To the contrary, a reconstruction based on flight-recorder data indicates that the crew may have caused the crash by mismanaging or overriding the autopilot. Here are the theories in play as of this morning.

Theory 1: Take the controls away from the machine. The argument here is that difficult conditions require human attention and judgment. "The pilot did not disengage the autopilot after encountering what was noted to be ‘significant ice'—disregarding recommendations from the [National Transportation Safety Board] and his own airline," the Associated Press reports. Accordingly, the NTSB will study "whether the recommendation should be a requirement."

Theory 2: Teach the humans how to read and collaborate with the machine's intentions. This is the scenario we looked at yesterday: that the plane's crew misunderstood what the autopilot was doing and why, and that this misunderstanding caused the crash. Today's Wall Street Journal adds more evidence to this scenario:

The safety board, among other issues, is looking into why [the airline's] training programs apparently stop short of allowing pilots in simulators to feel the stick-pusher activate. ... The device is intended to automatically prevent the plane from going into a stall by pointing the nose down to regain speed. Safety experts worry that unless pilots understand and feel what happens when the stick-pusher goes into action in a simulator, they may not react properly when it activates during an in-flight emergency. In 2004, a [same-company] commuter jet went out of control at a high altitude, both its engines shut down, and it ultimately crashed, killing both pilots. Among other mistakes, the pilots fought the stick-pusher. ... [T]he board specifically called for a blue-ribbon panel of experts to examine ways to make pilots more familiar and comfortable with the operation of stick-pusher systems.

Theory 3: Take the controls away from the humans. This is a more radical version of Theory 2. On this view, human pilots shouldn't be trusted to override the machine's superior judgment. To prevent such intervention, we can engineer flight controls to let a computer override the pilot. Today's New York Times explains how:

The crash last week could renew a debate about how much authority the crew should have over an airplane. In fly-by-wire aircraft—in which the crew's control is through a computer rather than a direct mechanical link to flight control surfaces—one manufacturer, Airbus, interposes computer logic between the human and the machine. If the pilot's manipulation of the controls is too severe, the computer will specify a more moderate path.

Yesterday, going on Theory 2, I implied that the remedy for what happened in Buffalo might be to upgrade autopilots so that humans, at the speed of conversation, could query the machine's reasons for acting. But maybe that's the long-term remedy. Maybe the short-term remedy is to give the machine the final say.

The Fatal Silence of Autopilots

Horrifying update posted this afternoon by the New York Times, following up last night's report by the Wall Street Journal:

A re-creation of the last moments of the plane that crashed in Buffalo Thursday night, based on data from the "black boxes," shows that the crew may have overreacted to an automatic system that was trying to protect the aircraft from flying too slowly and crashing from an aerodynamic stall. ... investigators have developed a theory that after the automatic system pointed down the nose of the plane to generate speed, the crew may have overreacted by yanking back on the yoke and pointing the nose too high ...

The plane, Continental Connection Flight 3407, from Newark to Buffalo, was flying on auto-pilot. ... [W]hen the "stick-pusher," which takes control of the plane and points the nose down, activated and the autopilot kicked off, the crew tried to increase power. Apparently there was not enough altitude or time to recover control, however. The stick-pusher may have activated at a speed higher than normal because it added a margin of safety to account for icing conditions, investigators say.

In other words, the human crew misunderstood what the autopilot was doing and why, and this misunderstanding caused the crash. If the autopilot had been a human being, a five-second conversation could have resolved the misunderstanding. But autopilots can't talk. They're designed to perform calculations and adjust the plane without humans having to understand their reasons.

If this theory of the crash turns out to be true, it'll go down in history as a tragic flaw in early generations of artificial intelligence, at the cost of 50 lives.

Armed Robotry

I've been meaning to get back to this Cornelia Dean piece from last week's NYT Science Times. It's about one of my favorite topics: military robots. Except it confounds some of my assumptions, which makes it all the more worth thinking about.

First off: The "killing machines" I keep writing about are just drones. They're fully controlled (except for malfunctions and weather) by human pilots. Dean is talking about something way more unnerving: machines that make their own killing decisions. I had assumed that for safety reasons, this kind of technology was still confined to the computer equivalent of drawing boards. Wrong. Army software contractor Ronald Arkin tells Dean that armed mechanical border guards are already on the job in Israel and South Korea. Here in the United States, the Army is paying Arkin and others to explore, among other things, how to design such robots to "operate within the bounds imposed by the warfighter." In other words, before we give them guns, we'd better figure out how to keep them from screwing up royally or turning on us.

What's really interesting about Arkin is that he directly contradicts my paranoid prejudice. It's not the armed robots I should worry about. It's the armed humans. Dean summarizes his argument:

In a report to the Army last year, Dr. Arkin described some of the potential benefits of autonomous fighting robots. For one thing, they can be designed without an instinct for self-preservation and, as a result, no tendency to lash out in fear. They can be built without anger or recklessness, Dr. Arkin wrote, and they can be made invulnerable to what he called "the psychological problem of ‘scenario fulfillment,' " which causes people to absorb new information more easily if it agrees with their pre-existing ideas.

His report drew on a 2006 survey by the surgeon general of the Army, which found that fewer than half of soldiers and marines serving in Iraq said that noncombatants should be treated with dignity and respect, and 17 percent said all civilians should be treated as insurgents. More than one-third said torture was acceptable under some conditions, and fewer than half said they would report a colleague for unethical battlefield behavior. Troops who were stressed, angry, anxious or mourning lost colleagues or who had handled dead bodies were more likely to say they had mistreated civilian noncombatants, the survey said.

That makes sense: In war, emotion is more hindrance than help. Same goes for my previous speculation that pilots will become more brutal as they're insulated from physical risk. Arkin's data suggest that in fact, exposure to physical risk makes troops more aggressive, not less. Again, the theory makes sense: You shoot first and ask questions later when failure to shoot jeopardizes your safety. Take the ego out of it—make you a robot instead of a person—and the self-protective instinct to shoot first disappears.

That leaves the problem of ethics. Hormones, mirror neurons, socialization, and love, among other things, make most people reluctant to kill one another. Robots lack these inputs. Will they be ruthless? Arkin's answer, as related by Dean, is that "because rules like the Geneva Conventions are based on humane principles, building them into the machine's mental architecture endows it with a kind of empathy."

Well, I wouldn't go that far. It's not empathy, exactly. But maybe empathy isn't so hot as a guide to behavior in combat. Maybe one lesson of the Army's Iraq survey is that empathy too easily morphs into tribalism. Maybe mechanical soldiers programmed with ethical rules, like the machines of I, Robot, are more likely to behave decently.

But then comes the hitch: What happens when the grainy realities of war defy the simplicity of the robot's program? What happens when the hard part isn't restraining yourself from firing on civilians, but distinguishing them from enemy forces in the first place? That's where Arkin's dream bogs down. He admits it would be hard for robots to recognize physical changes that entail moral changes, such as an enemy fighter with a wound or a white flag. And that's basic stuff compared to the multiplying subtleties of modern counterinsurgency. It's not as though al-Qaida hands out uniforms. Is the guy with the backpack a student or a terrorist? Is the woman across the street chubby or wearing a belt full of explosives?

Here's my preliminary take on Arkin's idea: He's right that we can and should substitute robots for humans in some lethal jobs. Where the categories are clear and cold reason is crucial, let the robots do the guarding and killing. But don't give the early generations of robots any jobs that require nuanced judgments about who's a bad guy and who isn't. And be prepared for the bad guys to learn the loopholes in the robots' algorithms. If the robots respect white flags, the terrorists will use white flags. If the robots presume women are civilians, the terrorists will use women. That's what terrorists do: They study our habits and exploit them. It's a human skill. And it will take humans, not robots, to defeat them.

I, LawnBott

If I had a nickel for every time I've read the word "robot" in a headline about new technology, I'd ... well, given the current price of metals, I'd melt down all those nickels, sell the ingredients, and become a very rich man. Journalists and PR people use the word "robot" to mean anything from HAL to a remote-controlled toy car. Actually, robots come in various degrees. The revolution we're seeing in mechanization isn't so much in the proliferation of robots as in their increasing autonomy.

Case in point: Two stories from this morning's news batch.

First we have an AP story about a "Bum Bot" designed to disperse vagrants from an iffy neighborhood in Atlanta. It belongs to Rufus Terrill, a local bar owner and ex-Marine. The story says Terrill used to patrol the area on foot, but "guns were stuck in his face several times. His wife suggested he patrol a safer way - using a robot." In the AP photo, the robot looks like a small tank, about half as tall as Terrill. It weighs 300 pounds and has a camera and water cannon. (Terrill says he's never used the cannon.) The robot's exterior has been "nicked by rocks, bricks and other objects people Terrill was rousting have thrown at it."

The point of the robot, it seems, is to take the physical risks formerly taken by its human owner. Any guns that might previously have been stuck in his face now have to be pointed at his tank instead, which doesn't have quite the same effect. There's no report of the tank having been shot, but, as the story says, it has taken its share of rocks and bricks. That's fine. It's part of the plan. Sticks and stones may break my drones, but they can't hurt me.

In this way, the Bum Bot is a lot like the thousands of drones currently deployed by the U.S. military. The enemy can't kill American soldiers who patrol war zones from a safe distance via remotely-operated unmanned vehicles.

The tricky thing about drones, as I've noted before, is that they can desensitize you to the battlefield. I mean literally desensitize you: Your physical senses have no direct contact with what you're looking or shooting at. Can the same thing happen to civilians who use private security drones at home? Apparently so. "It's just like a video game," says Terrill, describing how he operates the Bum Bot. The Atlanta police warn that he might be prosecuted if he uses the water cannon. But there's no such constraint on the use of drones in Afghanistan, Pakistan, or Iraq.

The chief constraint on the Bum Bot isn't legal or moral. It's technological. The Bum Bot isn't really a robot. It's controlled entirely by the handheld remote, and it has no voice other than Terrill's, which he projects through an integrated walkie-talkie. Without constant human direction, the machine does nothing.

If you want to get closer to the cutting edge of robotics, so to speak, you're better off looking at a technology that's already well-commercialized: robotic lawn mowers. Today's New York Times salutes a new product, the Kyodo LawnBott LB3500, which mows your lawn by itself. Here's the manufacturer's description:

It operates automatically, and autonomously by means of its intelligent computer and a perimeter cable. It can move freely within an enclosed area, detecting the faint signal transmitted by the perimeter cable located on the ground, defining the areas to be mowed; it can also work without a perimeter cable as working area is enclosed by a fence or small border at least 4 inches tall. ... [I]t leaves its docking station and starts mowing your yard in a random direction. It will mow in a straight line until it bumps into an obstacle, such as a tree or flower pot, or until it runs over its perimeter cable, then it stops, backs up, turns and takes off again.

Well, at least it needs a human to recharge its batteries, right?

Wrong. The company explains:

When the batteries start running low, or at the end of its cutting cycle, the mower will search out the perimeter cable and follow it back to its docking station to recharge. After charging, it heads back out on its own! ... With the new LawnBott, you have One Less Thing to Worry About.

Well, yes. But you also have one more thing to worry about: Your lawnmower running amok while you're at the office. No human hassle means no human control.

Kyodo says the LB3500 comes with enhanced safety features: "a higher sensitivity, free-floating, 360° bumper shell, blade stop proximity sensor, and an on-board alarm system should an unauthorized user pick up the Lawnbott." Still, we're talking about a slicing machine that runs around by itself and can't even be stopped by power depletion. LawnBotts.com points out that "robotic lawnmowers are many times safer than its manual counterparts just because you eliminate the human needing to be around it while it's operating." This is the same sense in which military drones are safer than manned vehicles and weapons: They protect their owners. But if you're not the owner, look out.

In the AP story about Terrill's bar, some of the locals complain that the Bum Bot is "intimidating." They have no idea what's coming.