Rise of the Machines: A Computer That Learns Games by Watching Us Play

Future Tense
The Citizen's Guide to the Future
July 12 2012 1:11 PM

Rise of the Machines: A Computer That Learns Games by Watching Us Play

New software teaches a computer grid-based games by showing it video footage of game play.

Photo by HENNING KAISER/AFP/Getty Images

We’ve been able to “play the computer” in games for years. But what if, instead of telling them what to do and how to play, we let computers learn from watching us? And what if they used that information to beat us consistently?

It’s not time to invoke Skynet just yet, but researcher Lukasz Kaiser at Paris Diderot University has flipped the logic of computer game play with the help of some basic software, video recognition, and an algorithm.


Kaiser’s software feeds video footage of winning, losing, and illegal moves from five grid-based games to a computer. The computer then analyzes the minimal sets of footage and intuits the exact rules for Breakthrough, Gomoku, Connect Four, tic-tac-toe, and a game we've never heard of called Pawn Whopping (if you have, tell us in the comments). Kaiser’s relatively simple laptop can analyze the footage in less than a minute, separating material like human hands from the important stuff, and learn the exact rules of the game from scratch, with information rather than direction. Once the computer learns rules for a game like Connect Four, it’s almost impossible to beat. Remember how we humans lose every time at rock paper scissors?

“There is strong theoretical evidence that it will generalize to the other problems,” Kaiser writes of the method he developed. “Some of those problems might require hierarchical, structured learning or a form of probabilistic formulas, and in the future we intend to consider such extensions. But already the presented technique significantly improves the state of the art in learning from visual input.”

Translation: It may be a little while before computers are learning more complex games like chess just from watching us play, but we’re moving into an age where, with limited visual input, a machine can understand the rules. Hopefully that ability can prime machines to help us with all sorts of problems—not put us in checkmate.

Future Tense is a partnership of SlateNew America, and Arizona State University.

Ben Johnson is the producer of Marketplace Tech from American Public Media.



The Ebola Story

How our minds build narratives out of disaster.

The Budget Disaster That Completely Sabotaged the WHO’s Response to Ebola

PowerPoint Is the Worst, and Now It’s the Latest Way to Hack Into Your Computer

The Shooting Tragedies That Forged Canada’s Gun Politics

A Highly Unscientific Ranking of Crazy-Old German Beers


Welcome to 13th Grade!

Some high schools are offering a fifth year. That’s a great idea.


The Actual World

“Mount Thoreau” and the naming of things in the wilderness.

Want Kids to Delay Sex? Let Planned Parenthood Teach Them Sex Ed.

Would You Trust Walmart to Provide Your Health Care? (You Should.)

  News & Politics
Oct. 22 2014 9:42 PM Landslide Landrieu Can the Louisiana Democrat use the powers of incumbency to save herself one more time?
Continuously Operating
Oct. 22 2014 2:38 PM Crack Open an Old One A highly unscientific evaluation of Germany’s oldest breweries.
Dear Prudence
Oct. 23 2014 6:00 AM Monster Kids from poorer neighborhoods keep coming to trick-or-treat in mine. Do I have to give them candy?
  Double X
The XX Factor
Oct. 22 2014 4:27 PM Three Ways Your Text Messages Change After You Get Married
  Slate Plus
Tv Club
Oct. 22 2014 5:27 PM The Slate Walking Dead Podcast A spoiler-filled discussion of Episodes 1 and 2.
Oct. 22 2014 11:54 PM The Actual World “Mount Thoreau” and the naming of things in the wilderness.
Future Tense
Oct. 22 2014 5:33 PM One More Reason Not to Use PowerPoint: It’s The Gateway for a Serious Windows Vulnerability
  Health & Science
Bad Astronomy
Oct. 23 2014 7:30 AM Our Solar System and Galaxy … Seen by an Astronaut
Sports Nut
Oct. 20 2014 5:09 PM Keepaway, on Three. Ready—Break! On his record-breaking touchdown pass, Peyton Manning couldn’t even leave the celebration to chance.