Human Nature: Science, Technology, and Life.

Freeway Autopilots

Maybe one lesson of Northwest Flight 188 is that driving should be more like flying. There are tricky parts that require close attention and adjustment—takeoff and landing for planes, local navigation for cars—and then there are parts that can be automated: cruising altitude for planes, freeway driving for cars. A freeway autopilot would have to keep you in your lane until your exit. It wouldn't zip around from lane to lane like a jerk. But I'm guessing most of us could live with that.

More here.

Married to the Machines

Shankar Vedantam is a people person. I don't mean that in the ordinary sense, as in, "Do I look like a @#$% people person?" I mean that, in addition to being delightful company, he writes about people. He's interested in how we think.

So when he writes about machines, as he did last week, something funny is going on. His topic was the recent Metro train crash in Washington, D.C., which killed nine people. How did it happen?

One theory is the automation paradox:

The more reliable the system, the more likely it is that humans in charge will "switch off" and lose their concentration, and the greater the likelihood that a confluence of unexpected factors that stymie the algorithm will produce catastrophe. ... After the previous fatal accident on Metro, in which a train overshot the Shady Grove station on an icy night, the National Transportation Safety Board found that the driver of the train had reported overshooting problems at earlier stops but was told not to interfere with the automated controls.

In that case, automation researcher Greg Jamieson points out, "For a year before the accident, the transit authority had put in position a directive that you were not to drive the train in manual." Vedantam concludes: "No matter how clever the designers of automated systems might be, they simply cannot account for every possible scenario, which is why it is so dangerous to eliminate human ‘interference.' "

This is the problem we discussed after the February plane crash that killed 50 people near Buffalo, N.Y. Initial evidence indicated that the pilot misunderstood what the autopilot was doing, and, by overriding the machine, caused the crash. Further evidence presented at a May hearing confirms that

the plane, which was collecting ice on its windshield and wings, was slowing to an unsafe speed. But when a warning system began vibrating the control column to get their attention, the captain pulled the nose up when he should have pushed it down. ... [C]onfronting the vibrating column, called a stick shaker, was probably something new and startling. The airline that was contracted with Continental Airlines to make the one-hour commuter flight, Colgan Air, said on Wednesday that it had given the crew simulated training in the activation of the stick shaker, but not in the next step, activation of the stick pusher, which takes control and pushes the nose of the plane down. In this instance, the stick pusher kicked in shortly after the captain pulled instead of pushed. "I don't see any evidence that he ever understood the situation he was in," said Dr. Dismukes ...

Shortly after the Buffalo crash, I outlined three possible responses to such disasters. One was take the controls away from the machines, on the grounds that difficult conditions require human attention and judgment. The opposite approach was to take the controls away from the humans, on the grounds that pilots can't be trusted to override the machine's superior judgment. A third, hybrid solution was to teach the humans how to read and collaborate with the machine's intentions.

The third approach seems to be the one most clearly supported by the evidence in the Buffalo crash: Flight crews must be trained to interpret and interact with their autopilots. Vedantam makes a similar point about automated systems in general: "Several studies have found that regular training exercises that require operators to turn off their automated systems and run everything manually are useful in retaining skills and alertness." We have to know when to second-guess our machines and how to operate without their help. Sometimes, they'll err fatally unless we intervene. But our intervention can itself be fatal. The key is to understand when to step in and when to butt out. That's the role of human intelligence in a machine-controlled world.

It's fun to go to summer sci-fi movies and wonder whether humans or machines would prevail in a mortal showdown. But in the real world, machines aren't our enemies. They're our collaborators. If those 50 people in Buffalo died in a fight between a human and a machine, it wasn't a fight chosen by either side. It was a misunderstanding. And since we're the ones who made the machines, it's our job to teach one another how to work with them, around them, and without them.

God Is My Autopilot

That's one possibility emerging from for last week's passenger jet crash in Buffalo, N.Y. So far, the investigation has found no mechanical or autopilot failure. To the contrary, a reconstruction based on flight-recorder data indicates that the crew may have caused the crash by mismanaging or overriding the autopilot. Here are the theories in play as of this morning.

Theory 1: Take the controls away from the machine. The argument here is that difficult conditions require human attention and judgment. "The pilot did not disengage the autopilot after encountering what was noted to be ‘significant ice'—disregarding recommendations from the [National Transportation Safety Board] and his own airline," the Associated Press reports. Accordingly, the NTSB will study "whether the recommendation should be a requirement."

Theory 2: Teach the humans how to read and collaborate with the machine's intentions. This is the scenario we looked at yesterday: that the plane's crew misunderstood what the autopilot was doing and why, and that this misunderstanding caused the crash. Today's Wall Street Journal adds more evidence to this scenario:

The safety board, among other issues, is looking into why [the airline's] training programs apparently stop short of allowing pilots in simulators to feel the stick-pusher activate. ... The device is intended to automatically prevent the plane from going into a stall by pointing the nose down to regain speed. Safety experts worry that unless pilots understand and feel what happens when the stick-pusher goes into action in a simulator, they may not react properly when it activates during an in-flight emergency. In 2004, a [same-company] commuter jet went out of control at a high altitude, both its engines shut down, and it ultimately crashed, killing both pilots. Among other mistakes, the pilots fought the stick-pusher. ... [T]he board specifically called for a blue-ribbon panel of experts to examine ways to make pilots more familiar and comfortable with the operation of stick-pusher systems.

Theory 3: Take the controls away from the humans. This is a more radical version of Theory 2. On this view, human pilots shouldn't be trusted to override the machine's superior judgment. To prevent such intervention, we can engineer flight controls to let a computer override the pilot. Today's New York Times explains how:

The crash last week could renew a debate about how much authority the crew should have over an airplane. In fly-by-wire aircraft—in which the crew's control is through a computer rather than a direct mechanical link to flight control surfaces—one manufacturer, Airbus, interposes computer logic between the human and the machine. If the pilot's manipulation of the controls is too severe, the computer will specify a more moderate path.

Yesterday, going on Theory 2, I implied that the remedy for what happened in Buffalo might be to upgrade autopilots so that humans, at the speed of conversation, could query the machine's reasons for acting. But maybe that's the long-term remedy. Maybe the short-term remedy is to give the machine the final say.

The Fatal Silence of Autopilots

Horrifying update posted this afternoon by the New York Times, following up last night's report by the Wall Street Journal:

A re-creation of the last moments of the plane that crashed in Buffalo Thursday night, based on data from the "black boxes," shows that the crew may have overreacted to an automatic system that was trying to protect the aircraft from flying too slowly and crashing from an aerodynamic stall. ... investigators have developed a theory that after the automatic system pointed down the nose of the plane to generate speed, the crew may have overreacted by yanking back on the yoke and pointing the nose too high ...

The plane, Continental Connection Flight 3407, from Newark to Buffalo, was flying on auto-pilot. ... [W]hen the "stick-pusher," which takes control of the plane and points the nose down, activated and the autopilot kicked off, the crew tried to increase power. Apparently there was not enough altitude or time to recover control, however. The stick-pusher may have activated at a speed higher than normal because it added a margin of safety to account for icing conditions, investigators say.

In other words, the human crew misunderstood what the autopilot was doing and why, and this misunderstanding caused the crash. If the autopilot had been a human being, a five-second conversation could have resolved the misunderstanding. But autopilots can't talk. They're designed to perform calculations and adjust the plane without humans having to understand their reasons.

If this theory of the crash turns out to be true, it'll go down in history as a tragic flaw in early generations of artificial intelligence, at the cost of 50 lives.

Night of the Living Dad

For ages, we've been telling children that ghosts aren't real. But the Department of Defense has just put out a request for proposals to create what are, in effect, virtual ghosts. Another truism of parenting is about to become untrue.

More here.

Armed Robotry

I've been meaning to get back to this Cornelia Dean piece from last week's NYT Science Times. It's about one of my favorite topics: military robots. Except it confounds some of my assumptions, which makes it all the more worth thinking about.

First off: The "killing machines" I keep writing about are just drones. They're fully controlled (except for malfunctions and weather) by human pilots. Dean is talking about something way more unnerving: machines that make their own killing decisions. I had assumed that for safety reasons, this kind of technology was still confined to the computer equivalent of drawing boards. Wrong. Army software contractor Ronald Arkin tells Dean that armed mechanical border guards are already on the job in Israel and South Korea. Here in the United States, the Army is paying Arkin and others to explore, among other things, how to design such robots to "operate within the bounds imposed by the warfighter." In other words, before we give them guns, we'd better figure out how to keep them from screwing up royally or turning on us.

What's really interesting about Arkin is that he directly contradicts my paranoid prejudice. It's not the armed robots I should worry about. It's the armed humans. Dean summarizes his argument:

In a report to the Army last year, Dr. Arkin described some of the potential benefits of autonomous fighting robots. For one thing, they can be designed without an instinct for self-preservation and, as a result, no tendency to lash out in fear. They can be built without anger or recklessness, Dr. Arkin wrote, and they can be made invulnerable to what he called "the psychological problem of ‘scenario fulfillment,' " which causes people to absorb new information more easily if it agrees with their pre-existing ideas.

His report drew on a 2006 survey by the surgeon general of the Army, which found that fewer than half of soldiers and marines serving in Iraq said that noncombatants should be treated with dignity and respect, and 17 percent said all civilians should be treated as insurgents. More than one-third said torture was acceptable under some conditions, and fewer than half said they would report a colleague for unethical battlefield behavior. Troops who were stressed, angry, anxious or mourning lost colleagues or who had handled dead bodies were more likely to say they had mistreated civilian noncombatants, the survey said.

That makes sense: In war, emotion is more hindrance than help. Same goes for my previous speculation that pilots will become more brutal as they're insulated from physical risk. Arkin's data suggest that in fact, exposure to physical risk makes troops more aggressive, not less. Again, the theory makes sense: You shoot first and ask questions later when failure to shoot jeopardizes your safety. Take the ego out of it—make you a robot instead of a person—and the self-protective instinct to shoot first disappears.

That leaves the problem of ethics. Hormones, mirror neurons, socialization, and love, among other things, make most people reluctant to kill one another. Robots lack these inputs. Will they be ruthless? Arkin's answer, as related by Dean, is that "because rules like the Geneva Conventions are based on humane principles, building them into the machine's mental architecture endows it with a kind of empathy."

Well, I wouldn't go that far. It's not empathy, exactly. But maybe empathy isn't so hot as a guide to behavior in combat. Maybe one lesson of the Army's Iraq survey is that empathy too easily morphs into tribalism. Maybe mechanical soldiers programmed with ethical rules, like the machines of I, Robot, are more likely to behave decently.

But then comes the hitch: What happens when the grainy realities of war defy the simplicity of the robot's program? What happens when the hard part isn't restraining yourself from firing on civilians, but distinguishing them from enemy forces in the first place? That's where Arkin's dream bogs down. He admits it would be hard for robots to recognize physical changes that entail moral changes, such as an enemy fighter with a wound or a white flag. And that's basic stuff compared to the multiplying subtleties of modern counterinsurgency. It's not as though al-Qaida hands out uniforms. Is the guy with the backpack a student or a terrorist? Is the woman across the street chubby or wearing a belt full of explosives?

Here's my preliminary take on Arkin's idea: He's right that we can and should substitute robots for humans in some lethal jobs. Where the categories are clear and cold reason is crucial, let the robots do the guarding and killing. But don't give the early generations of robots any jobs that require nuanced judgments about who's a bad guy and who isn't. And be prepared for the bad guys to learn the loopholes in the robots' algorithms. If the robots respect white flags, the terrorists will use white flags. If the robots presume women are civilians, the terrorists will use women. That's what terrorists do: They study our habits and exploit them. It's a human skill. And it will take humans, not robots, to defeat them.