Your Video-Game Console Knows a Lot More About You Than You Thought

The art of play.
March 7 2012 6:40 AM

Your Kinect Is Watching You

The amazing, disturbing things your gaming console can learn about you.

Kinect for Xbox 360.
A general view at Kinect for Xbox 360 Launch Party held at a private residence on October 23, 2010 in Beverly Hills, California.

Jason Merritt/Getty Images for Xbox.

There is a wave of concern—completely justified, to my mind—over the privacy implications of our increasing reliance on Facebook and Google. What most people don’t realize, however, is that these issues are dwarfed by the potential for privacy invasion that’s presented by a seemingly innocuous platform: video games.

Consider the Kinect, the Microsoft console that sold 8 million units in its first 60 days of release. This inexpensive, book-sized panel has the ability to create a realistic, virtual likeness of the player. In doing so, it creates a delightful interface to play games—instead of hitting a button to kick a ball, you kick your foot and the digital character on screen mimics your movements. How does the Kinect produce this dazzling immersive experience? By capturing every move you make.

People choose to post personal information on Facebook and Google. Game platforms like the Kinect, by contrast, continuously observe your nonverbal behavior. Movements and gestures may seem harmless to share with others, but decades of psychological research demonstrate that the way you move is more revealing than what you say.

Advertisement

More than 40 years ago, UCLA professor Albert Mehrabian demonstrated that nonverbal behavior constitutes a majority of face-to-face communication. More importantly, nonverbal behavior is automatic. Though we can all watch what we say, very few of us can consistently regulate our subtle movements and gestures.

Though it’s designed for gaming, the Kinect can be modified to track other behaviors as well. As such, scientists throughout the world, including my team at Stanford’s Virtual Human Interaction Lab, are starting to study what the Kinect and other gaming systems reveal about you. I’ve been studying digital footprints—the behavioral residues left behind in video games and other virtual worlds—for almost 10 years. During that time I have collaborated with Fortune 500 executives, military officials, and educators. Our ability to automatically classify someone’s nonverbal cues—the essence of their identity, psychological state, and behavior—could prove extremely beneficial. It’s also not to be taken lightly.

In my lab, a team working with Konica Minolta has developed a system to automatically detect learning. In one study we used the Kinect to gather nonverbal data during one-on-one, student-teacher interactions. Later, we used that data to predict the students’ test scores. The early results are encouraging, though preliminary, with around 10 movements (those relating to the shoulder and elbow, for instance) being the most predictive. What makes this type of experiment so powerful is the “bottom up” nature of the research. Instead of looking for specific known gestures, like nodding or pointing, we can mathematically uncover subtle movement patterns, many of which would not be noticed by the human eye. Just imagine if teachers, based on a small sample of their students’ nonverbal behavior, could instantly detect which students needed extra attention or specialized assignments.

Scientists at the University of Southern California have already used devices much simpler than the Kinect to diagnose classroom behavior. By looking at nothing more than the direction a child’s head was pointed—a snap for a system like the Kinect—Skip Rizzo and his colleagues were able to detect hyperactivity disorders like ADHD. A system examining nonverbal behavior in real time could use this data to automatically diagnose kids in school.

Other scientists have developed applications for detecting behavior in the home. Jaeyong Sung and his colleagues at Cornell University are looking at whether a Kinect could assist people by recognizing their activities—automatically detecting, for example, if someone is brushing her teeth, cooking, or opening a pill container. The models were highly accurate (84 percent) at categorizing behavior for people who the system had seen previously. But even more impressive was how it categorized new people. If someone new visits your home and walks past the video-game console, it still recognized that person’s behavior 64 percent of the time.

TODAY IN SLATE

Politics

Blacks Don’t Have a Corporal Punishment Problem

Americans do. But when blacks exhibit the same behaviors as others, it becomes part of a greater black pathology. 

I Bought the Huge iPhone. I’m Already Thinking of Returning It.

Scotland Is Just the Beginning. Expect More Political Earthquakes in Europe.

Lifetime Didn’t Think the Steubenville Rape Case Was Dramatic Enough

So they added a little self-immolation.

Two Damn Good, Very Different Movies About Soldiers Returning From War

Medical Examiner

The Most Terrifying Thing About Ebola 

The disease threatens humanity by preying on humanity.

Students Aren’t Going to College Football Games as Much Anymore, and Schools Are Getting Worried

The Good Wife Is Cynical, Thrilling, and Grown-Up. It’s Also TV’s Best Drama.

  News & Politics
Weigel
Sept. 20 2014 11:13 AM -30-
  Business
Business Insider
Sept. 20 2014 6:30 AM The Man Making Bill Gates Richer
  Life
Quora
Sept. 20 2014 7:27 AM How Do Plants Grow Aboard the International Space Station?
  Double X
The XX Factor
Sept. 19 2014 4:58 PM Steubenville Gets the Lifetime Treatment (And a Cheerleader Erupts Into Flames)
  Slate Plus
Slate Picks
Sept. 19 2014 12:00 PM What Happened at Slate This Week? The Slatest editor tells us to read well-informed skepticism, media criticism, and more.
  Arts
Brow Beat
Sept. 20 2014 1:52 PM Julian Casablancas’ New Album Sounds Like the Furthest Thing From the Strokes
  Technology
Future Tense
Sept. 19 2014 6:31 PM The One Big Problem With the Enormous New iPhone
  Health & Science
Bad Astronomy
Sept. 20 2014 7:00 AM The Shaggy Sun
  Sports
Sports Nut
Sept. 18 2014 11:42 AM Grandmaster Clash One of the most amazing feats in chess history just happened, and no one noticed.