Rise of the Machines: A Computer That Learns Games by Watching Us Play

Future Tense
The Citizen's Guide to the Future
July 12 2012 1:11 PM

Rise of the Machines: A Computer That Learns Games by Watching Us Play

New software teaches a computer grid-based games by showing it video footage of game play.

Photo by HENNING KAISER/AFP/Getty Images

We’ve been able to “play the computer” in games for years. But what if, instead of telling them what to do and how to play, we let computers learn from watching us? And what if they used that information to beat us consistently?

It’s not time to invoke Skynet just yet, but researcher Lukasz Kaiser at Paris Diderot University has flipped the logic of computer game play with the help of some basic software, video recognition, and an algorithm.


Kaiser’s software feeds video footage of winning, losing, and illegal moves from five grid-based games to a computer. The computer then analyzes the minimal sets of footage and intuits the exact rules for Breakthrough, Gomoku, Connect Four, tic-tac-toe, and a game we've never heard of called Pawn Whopping (if you have, tell us in the comments). Kaiser’s relatively simple laptop can analyze the footage in less than a minute, separating material like human hands from the important stuff, and learn the exact rules of the game from scratch, with information rather than direction. Once the computer learns rules for a game like Connect Four, it’s almost impossible to beat. Remember how we humans lose every time at rock paper scissors?

“There is strong theoretical evidence that it will generalize to the other problems,” Kaiser writes of the method he developed. “Some of those problems might require hierarchical, structured learning or a form of probabilistic formulas, and in the future we intend to consider such extensions. But already the presented technique significantly improves the state of the art in learning from visual input.”

Translation: It may be a little while before computers are learning more complex games like chess just from watching us play, but we’re moving into an age where, with limited visual input, a machine can understand the rules. Hopefully that ability can prime machines to help us with all sorts of problems—not put us in checkmate.

Future Tense is a partnership of SlateNew America, and Arizona State University.

Ben Johnson is the producer of Marketplace Tech from American Public Media.



Forget Oculus Rift

This $25 cardboard box turns your phone into an incredibly fun virtual reality experience.

The Congressional Republican Digging Through Scientists’ Grant Proposals

Renée Zellweger’s New Face Is Too Real

Sleater-Kinney Was Once America’s Best Rock Band

Can it be again?

Whole Foods Is Desperate for Customers to Feel Warm and Fuzzy Again

The XX Factor

I’m 25. I Have $250.03.

My doctors want me to freeze my eggs.

The XX Factor
Oct. 20 2014 6:17 PM I’m 25. I Have $250.03. My doctors want me to freeze my eggs.

Smash and Grab

Will competitive Senate contests in Kansas and South Dakota lead to more late-breaking races in future elections?

George Tiller’s Murderer Threatens Another Abortion Provider, Claims Free Speech

These Companies in Japan Are More Than 1,000 Years Old

  News & Politics
The World
Oct. 21 2014 3:13 PM Why Countries Make Human Rights Pledges They Have No Intention of Honoring
Oct. 21 2014 4:33 PM Walmart Is Killing the Rest of Corporate America in Solar Power Adoption
The Vault
Oct. 21 2014 2:23 PM A Data-Packed Map of American Immigration in 1903
  Double X
The XX Factor
Oct. 21 2014 3:03 PM Renée Zellweger’s New Face Is Too Real
  Slate Plus
Behind the Scenes
Oct. 21 2014 1:02 PM Where Are Slate Plus Members From? This Weird Cartogram Explains. A weird-looking cartogram of Slate Plus memberships by state.
Brow Beat
Oct. 21 2014 1:47 PM The Best Way to Fry an Egg
Future Tense
Oct. 21 2014 4:14 PM Planet Money Uncovers One Surprising Reason the Internet Is Sexist
  Health & Science
Climate Desk
Oct. 21 2014 11:53 AM Taking Research for Granted Texas Republican Lamar Smith continues his crusade against independence in science.
Sports Nut
Oct. 20 2014 5:09 PM Keepaway, on Three. Ready—Break! On his record-breaking touchdown pass, Peyton Manning couldn’t even leave the celebration to chance.