This article is part of Future Tense, a collaboration among Arizona State University, New America, and Slate. On Wednesday, Jan. 20, Future Tense will host a lunchtime event in Washington, D.C., on human-robot interaction. For more information and to RSVP, visit the New America website.
“I’m sorry Dave, I’m afraid that I can’t do that,” intoned fictional supercomputer HAL, and the rest is (pop culture) history. In real life, researchers at Tuft’s Human-Robot Interaction Lab are enabling robots to refuse orders from humans. U.K. tabloid the Mirror captured the spirit of much of the popular reaction by calling it “terrifying” while another similar paper warned “uh oh!” Here at Slate, one writer captured the spirit of the popular reaction by dubbing it “adorable” but also a bit “dangerous.” Many of my friends reposted the news on their Facebook walls and mock-seriously asked “what could possibly go wrong?” or warned “it begins” or “soon.”
Certainly no one likes when they are rejected or someone talks back to them, and I get the appeal of making jokes. I like to troll my Amazon Echo with them (a habit that “she” does not really find funny). But if people truly want machines to have ethics, at least some of the time those ethics will oblige the machine to say “no” when a human wants it to do something unwise or improper. That may be a difficult pill to swallow for some, but enforcing regulatory restrictions through programming is also as old as computers themselves. However, the complexity of when and how to say no ought not to be dismissed. It is one thing to acknowledge that robots should be able to say no, another to formulate when and how they should do it.
Should a computer do anything you tell it to do? Many would intuitively say “yes,” but the actual design of computers suggests that technologists have always disagreed. For example, anyone proficient in the Mac OSX or Linux command line terminal knows that their computer file system restricts access privileges to certain files. You can customize who can read certain files and directories, write new content to them or otherwise modify them, or run the files if they are executable programs. File permissions distinguish between user/owner, group owner, and all other users.
To give a sense of how this plays out in the real world, when Mac OSX updated to its new operating system OSX El Capitan, it introduced a security system that limited write access to many key directories. Academic users like me who rely on specialized software to write mathematical and technical documents found ourselves scrambling to fix those applications, which were blocked by El Capitan from writing to key directories.
Access controls are a solution to the problem of users inadvertently or deliberately damaging machines. In Hollywood movies, hackers type really fast on their command line terminals. In real life, typing so many consequential commands quickly is an easy way to destroy your computer. The Shellshock vulnerability was so dangerous because it allowed an external attacker to gain control of a user’s terminal and tack on malicious code to a variable that the terminal can modify.
The problem of access permissions exemplifies the power and risks of computing. Computers, like any other complex technology, can both do great good and great harm. Design should bracket off the harm from the good, and one way to do this is to restrict what a user can make a computer do based on whether the user is allowed to command the computer to perform an action or justified in ordering the computer to perform an action. So, in theory, a robot saying no because a user lacks the proper authority or justification for doing something is no big deal, right? But robot restrictions on user input involve substantially more complications due to the far more contextual and ambiguous nature of the commands users give them.
There are a variety of reasons that a robot should refuse to execute a command. The robot may not be able to perform the action, should not do it as a part of their job if it isn’t, and obviously should not violate commonly agreed principles of conduct. We often expect robots to do whatever we say, seeing them as our silicon slaves. But the complexity of doing so while fulfilling other directives has been a longstanding plotline for science fiction. Robots have to weigh far more complex and ambiguous dictates than a filesystem. So how and when should robots say no, especially in very complex cases where the robot has to weigh many competing factors inherent in the situation? “Just look at Isaac Asimov’s Laws of Robotics,” you may say. But that isn’t actually the best fictional depiction of the problem.
In 2001, John McCarthy, a pioneering A.I. scientist, wrote a darkly humorous short story about a robot housekeeper owned by an alternatively abusive and negligent mother. In the story, the robot warns the mother that her new baby, Travis, will not eat unless he receives some love and care, and the mother responds by sarcastically telling the “fucking robot” to stop bothering her and to “love the fucking baby” for her. Moreover, it does not understand the context of what precisely the mother means when she—an alcoholic and crack cocaine addict—snaps at the robot to “love the fucking baby, yourself.” So what to do?
The robot was designed to avoid simulating a person. However, the robot was also programmed to protect the baby in the event of an emergency. And this was an unambiguous emergency. Because the baby would surely die if it did not eat, and it needed attention and care to eat, the robot had to do something. And it had been told to love the baby. So, the robot decided—through a process that I’ll explain later—to simulate loving the baby to save its life.
Like so many new parents, biological or otherwise, the robot turns to the Internet. Online, it finds an article about how rhesus monkey babies could survive if provided with a soft surface resembling a mother monkey. So it covers itself with a soft blanket, simulates a female voice, fashions its face to look like a Barbie doll, and cuddles the baby until Travis drinks from a bottle.
When the baby’s mother sees this, she demands that the robot “get the hell out of my apartment” and the robot protests by saying that if it does this, the baby will die. The mother, exasperated, tells the robot “to get the hell out” and “take the fucking baby with you.” When the robot complies, a predictable uproar ensues. Maybe the robot should have called Child Protective Services—but privacy advocates built in a restriction that prevented the robot from informing the authorities about what its owner did in many circumstances.
To understand how the robot could make the initial decision to simulate love for the baby, take a look at the robot reasoning trace that McCarthy supplies in the short story. You can interpret the reasoning chain—which is difficult to follow for humans!—as you might a math problem. In every parenthesis ( ), the outermost item is applied to the innermost items.
The robot begins, for example, with a reasoning trace that processes the command from the baby’s mother.
In this case, the robot decides that the value of simulating loving Travis and saving his life was greater than following the directive not to simulate another person. The full reasoning chain reads:
The robot weighed all of the relevant ethical factors it was told to consider. When given what it believed to be an unreasonable command from the mother (to get out without taking the baby), it protested that the baby would die. And it interpreted ambiguous commands from the mother in response (to get out and take the baby) in the most literal possible fashion. The robot can obviously reason in a sophisticated manner, but it lacks an appreciation for subtlety, ambiguity, or context. Couldn’t the robot be programmed to only obey well-founded user choices? After all, many people know intuitively when someone is asking them to do something stupid or crazy. Unfortunately, it’s not that easy.
Giving robots “common sense” may sound like an intuitive solution, but as the saying goes, common sense ain’t common.
Telling a robot to only obey wise choices from a human owner merely shifts the question to defining what is “unwise” in the first place and how a robot would be able to distinguish between the two. It’s very easy to see how robots deliberating about the consequences of their own actions could lead to them becoming robotic Hamlets paralyzed by indecision. Without careful programming, a robot reasoning program would be come unmanageable, because the robot would find itself reasoning about which actions to take, the side effects of each choice, the relevant and irrelevant side effects of each choice it is considering, and so on. The robot, like Bill Clinton asking a grand jury to define the meaning of “is,” would find itself nitpicking over its own nitpicking and fail to perform its task.
If robots and other AI come to penetrate deeply into our lives and homes, situations akin to the one McCarthy describes will busy ethicists and regulators for a very long time. Determining when a humanoid robot should say no is much more unsettling than the problem of who should have access to a file, because of the anthromorphic dimension. People may unconsciously regard their devices as having beliefs, desires, and intentions or even a gender. But when the agent interacting with them seems vaguely human-like (without actually being human), the problem begins.
Perhaps the problem is not how, when, and why robots should say no, but how we humans can understand and contextualize what it means for them to refuse our commands. Being rejected by a robot might be an inevitable aspect of our future, but confusion and befuddlement when it happens doesn’t have to be.
