Why Students Prefer to Learn From a Machine

What's to come?
Aug. 13 2014 12:07 PM

Why Students Prefer to Learn From a Machine

Robo-readers aren’t as good as human readers—they’re better.

Children on tablets.
Bonus: Machines assign less homework than teachers do, too.

Photo courtesy of Shutterstock.

In April 2012, Mark D. Shermis, then the dean of the College of Education at the University of Akron, made a striking claim: “Automated essay scoring systems” were capable of evaluating student writing just as well as human readers. Shermis’ research, presented at a meeting of the National Council on Measurement in Education, created a sensation in the world of education—among those who see such “robo-graders” as the future of assessment, and those who believe robo-graders are worse than useless.

The most outspoken member of the second camp is undoubtedly Les Perelman, a former director of writing and a current research affiliate at the Massachusetts Institute of Technology. “Robo-graders do not score by understanding meaning but almost solely by use of gross measures, especially length and the presence of pretentious language,” Perelman charged in an op-ed published in the Boston Globe earlier this year. Test-takers who game the programs’ algorithms by filling pages with lots of text and using big words, Perelman contended, can inflate their scores without actually producing good writing.

Perelman makes a strong case against using robo-graders for assigning grades and test scores. But there’s another use for robo-graders—a role for them to play in which, evidence suggests, they may not only be as good as humans, but better. In this role, the computer functions not as a grader but as a proofreader and basic writing tutor, providing feedback on drafts, which students then use to revise their papers before handing them in to a human.

Advertisement

Instructors at the New Jersey Institute of Technology have been using a program called E-Rater in this fashion since 2009, and they’ve observed a striking change in student behavior as a result. Andrew Klobucar, associate professor of humanities at NJIT, notes that students almost universally resist going back over material they’ve written. But, Klobucar told Inside Higher Ed reporter Scott Jaschik, his students are willing to revise their essays, even multiple times, when their work is being reviewed by a computer and not by a human teacher. They end up writing nearly three times as many words in the course of revising as students who are not offered the services of E-Rater, and the quality of their writing improves as a result. Crucially, says Klobucar, students who feel that handing in successive drafts to an instructor wielding a red pen is “corrective, even punitive” do not seem to feel rebuked by similar feedback from a computer.

A close look at one of the growing number of independent studies of automated writing feedback provides some clues as to what might be going on among NJIT students. Khaled El Ebyary of Alexandria University in Egypt and Scott Windeatt of Newcastle University in Britain published the study in the International Journal of English Studies; it looks at the effects of a robo-reader program called Criterion on the writing of education students learning to teach English as a foreign language. The students in the study received Criterion’s feedback on two drafts of essays submitted on each of four topics.

The computer program appeared to transform the students’ approach to the process of receiving and acting on feedback, El Ebyary and Windeatt report. Comments and criticism from a human instructor actually had a negative effect on students’ attitudes about revision and on their willingness to write, the researchers note. By contrast, interactions with the computer produced overwhelmingly positive feelings, as well as an actual change in behavior—from “virtually never” revising, to revising and resubmitting at a rate of 100 percent. As a result of engaging in this process, the students’ writing improved; they repeated words less often, used shorter, simpler sentences, and corrected their grammar and spelling. These changes weren’t simply mechanical. Follow-up interviews with the study’s participants suggested that the computer feedback actually stimulated reflectiveness in the students—which, notably, feedback from instructors had not done.

Why would this be? First, the feedback from a computer program like Criterion is immediate and highly individualized—something not usually possible in big classes like those at Alexandria University, the site of the study by El Ebyary and Windeatt. Second, the researchers observed that for many students in the study, the process of improving their writing appeared to take on a gamelike quality, boosting their motivation to get better. Third, and most interesting, the students’ reactions to feedback seemed to be influenced by the impersonal, automated nature of the software.

This may seem paradoxical. When critics like Les Perelman of MIT claim that robo-graders can’t be as good as human graders, it’s because robo-graders lack human insight, human nuance, human judgment. But it’s the very non-humanness of a computer that may encourage students to experiment, to explore, to share a messy rough draft without self-consciousness or embarrassment. In return, they get feedback that is individualized, but not personal—not “punitive,” to use the term employed by Andrew Klobucar of NJIT.

Evidence of this peculiar advantage of technology can be found in a field outside education. Public health professionals have long known that people will more readily disclose sensitive information to a computer than to a person. When typing their answers on a keyboard, rather than looking a questioner in the eye, respondents reveal more about their health problems, acknowledge that they’re suffering from more symptoms (especially psychiatric symptoms), admit more HIV risk behaviors and confess more drug use. (Fun fact: Women confess a greater number of sexual partners when asked by a computer, while men drop the macho act and admit fewer.)

The “disinhibition effect” produced by technology, writes Adam Joinson, a professor at the University of the West of England, emerges whenever an individual has reason to feel anxiety or self-consciousness, or worries about being evaluated. And anxiety, self-consciousness, and worries about evaluation are just the emotions that, sad but true, many people feel around learning. Research has repeatedly shown that many students experience these uncomfortable emotions in relation to writing, as well as to math, science and foreign languages.

Precisely because learning can be so emotionally fraught, a nonjudgmental computer may motivate students to try, to fail, and to improve more than almost any human. Just don’t let the robo-reader give out grades.

This story was produced by the Hechinger Report, a nonprofit, nonpartisan education-news outlet based at Teachers College, Columbia University. Future Tense is a partnership of Slate, New America, and Arizona State University.

Annie Murphy Paul is a fellow at the New America Foundation and the author of the forthcoming book Brilliant: The Science of How We Get Smarter.

TODAY IN SLATE

Politics

Blacks Don’t Have a Corporal Punishment Problem

Americans do. But when blacks exhibit the same behaviors as others, it becomes part of a greater black pathology. 

I Bought the Huge iPhone. I’m Already Thinking of Returning It.

Scotland Is Just the Beginning. Expect More Political Earthquakes in Europe.

Lifetime Didn’t Think the Steubenville Rape Case Was Dramatic Enough

So they added a little self-immolation.

Two Damn Good, Very Different Movies About Soldiers Returning From War

Medical Examiner

The Most Terrifying Thing About Ebola 

The disease threatens humanity by preying on humanity.

Students Aren’t Going to College Football Games as Much Anymore, and Schools Are Getting Worried

The Good Wife Is Cynical, Thrilling, and Grown-Up. It’s Also TV’s Best Drama.

  News & Politics
Weigel
Sept. 19 2014 9:15 PM Chris Christie, Better Than Ever
  Business
Moneybox
Sept. 19 2014 6:35 PM Pabst Blue Ribbon is Being Sold to the Russians, Was So Over Anyway
  Life
Inside Higher Ed
Sept. 19 2014 1:34 PM Empty Seats, Fewer Donors? College football isn’t attracting the audience it used to.
  Double X
The XX Factor
Sept. 19 2014 4:58 PM Steubenville Gets the Lifetime Treatment (And a Cheerleader Erupts Into Flames)
  Slate Plus
Slate Picks
Sept. 19 2014 12:00 PM What Happened at Slate This Week? The Slatest editor tells us to read well-informed skepticism, media criticism, and more.
  Arts
Brow Beat
Sept. 19 2014 4:48 PM You Should Be Listening to Sbtrkt
  Technology
Future Tense
Sept. 19 2014 6:31 PM The One Big Problem With the Enormous New iPhone
  Health & Science
Medical Examiner
Sept. 19 2014 5:09 PM Did America Get Fat by Drinking Diet Soda?   A high-profile study points the finger at artificial sweeteners.
  Sports
Sports Nut
Sept. 18 2014 11:42 AM Grandmaster Clash One of the most amazing feats in chess history just happened, and no one noticed.