Inside Higher Ed

Does Duolingo Pass the Test?

Carnegie Mellon partners with the language-learning app to make it good enough to prep students for the TOEFL.

Photo illustration by Slate. Photo by Chloe Fan via Wikimedia Commons.

This story originally appeared in Inside Higher Ed.

Carnegie Mellon University is partnering with Duolingo, one of its spinoffs, to see if a 20-minute, $20 test is sufficient to prove international students’ English proficiency. Duolingo, meanwhile, hopes its test can upend the market.

Duolingo, a crowdsourced Web translation project created by researchers in Carnegie Mellon’s computer science department, spun off from the university in November 2011 to become a venture capital-backed startup. The company now offers language learning Web and smartphone apps.

Last week, the company released Duolingo Test Center, a stand-alone app intended for language learners interested in more than deepening their vocabularies. Duolingo is breaking into what the company’s head of marketing, Gina Gotthilf, described as the “archaic” industry of language proficiency tests, starting with English.

To study in the U.S., most international students have to produce evidence that they can speak and write in English, usually by taking a standardized test. The IELTS and TOEFL, two such tests, have long dominated the landscape of English proficiency tests, but they can cost hundreds of dollars, take hours to complete, and require students to travel to approved test centers on specific dates.

Duolingo, in comparison, promises a 20-minute, $20 test that can be completed at home on a computer or smartphone—and produce equally representative results of a student or job-seeker’s English proficiency. 

“The test-taking industry—especially for language certification—is pretty much a monopoly,” Gotthilf, who also heads the company’s international development efforts, said in an interview. “It just seems really silly that people can’t get jobs or a chance to study somewhere because they can’t take these tests.”

In the absence of proctors, Duolingo requires test-takers to snap a picture of a valid photo ID, then one of themselves. Using the built-in camera on the computer or smartphone, the app then matches photos to test-takers and monitors them for suspicious behavior such as excessive background noise or glances at something outside the camera’s field of vision. The recording is later verified by a third-party proctoring service.

The test consists of four types of questions. Test-takers have to pick out English words from a lineup that in one example included “boax,” “champoo” and “dac”; listen to a sentence and transcribe it; record themselves reading; and complete fill-in-the-blank questions. The test adapts itself to the test-taker’s level of proficiency, becoming easier or more difficult in response to correct or incorrect answers.

Some of the test’s content comes from the public domain—a passage from Moby Dick appeared in one question—but the test also pulls in data from Duolingo’s learning app, which processes about one test each second, said Burr Settles, a lead scientist and software engineer with the company.

“What the machine is trying to learn here is how well you know English, and it explores the space of sophistication in the language and quickly tries to zero in on where you are,” Settles said. Twenty minutes is enough for the adaptive algorithms to build a profile of the test-taker, he said, adding that it also pushes the limit for how long someone is willing to stare at his or her smartphone.

Google is supporting the launch, which may explain why the app is only available on the Web or on Android. An iOS version—as well as French and Spanish tests—is in the works.

Back to Carnegie Mellon

Building a new language proficiency test only gets Duolingo halfway there. Unless colleges, universities, and employers in English-speaking countries accept the test, it will remain a novelty.

Duolingo hopes Carnegie Mellon’s clout can spark widespread adoption. During the upcoming admission cycle, the university will encourage its international applicants to take Duolingo’s test in addition to other standardized tests. By 2016, Duolingo’s test may qualify as an admission requirement on its own.

Duolingo already has some preliminary data to suggest the test holds up to scrutiny. A study released this May by University of Pittsburgh assistant professor Feifei Ye—and sponsored by Duolingo—found a statistically significant connection between strong Duolingo and TOEFL test scores.

Carnegie Mellon enrolls more than 12,000 undergraduate and graduate students. Last fall, about one-third of them, or 4,121, came from countries other than the United States, according to a university report.

While institutions generally require international students to achieve a certain IELTS or TOEFL score, not all of those who do can use English at a university level.

Educational Testing Service, the nonprofit that offers the TOEFL, was this summer rocked by scandal after the BBC uncovered widespread fraud at test centers. The British government responded by removing ETS from the list of approved test providers and stripping 60 institutions of their ability to host international students.

Thomas A. Ewing, a spokesman for ETS, declined to comment on Duolingo’s plans, instead stressing the company’s dedication to security. The TOEFL, he pointed out, is accepted by more than 9,000 institutions and agencies worldwide, and the test itself uses security measures such as biometric voice recognition. “Providing standardized testing on a global basis requires such efforts especially when universities, businesses and governments are depending upon those scores as part of making important decisions,” Ewing said in an email. “Without such safeguards and standards for quality, global acceptance is difficult, if not impossible.”

Duolingo sees real vulnerability in the market leader, and they’re not the only ones. The investigation “definitely added” to the company’s decision to capture a slice of the standardized test market, Gotthilf said. Earlier this summer, Pearson also introduced its Global Scale of English, calling it the first “globally recognized standard in English.”

Universities often use their own safeguards to ensure international students don’t misrepresent their language skills—particularly in graduate admissions. John Lehoczky, Carnegie Mellon’s interim executive vice president, sits on the steering committee for the university’s master’s degree program in computational finance. The department videoconferences with its applicants, which he said can serve to validate test scores. “I would say we certainly find some students whose TOEFL scores are surprising,” Lehoczky said. “One wonders how they might have done so well on the exam—but I’m certainly not willing to connect that to fraud.”

Still, the app is not yet ready to replace the other tests, Lehoczky said. The TOEFL generates its final score by adding up how students performed in the listening, reading, speaking, and writing sections, and prospective students need to do well in all four. Duolingo also quizzes test takers in those areas but scores them on a 1-to-10 scale. “At this point it’s not a fit vehicle for undergraduate admissions at Carnegie Mellon,” Lehoczky said, adding that he hoped Duolingo will update the app to include a score breakdown.

The university’s partnership with its spinoff runs deeper than just the language proficiency test. The university is in the early stages of examining whether it can use the data gathered by Duolingo’s language learning tool in the Simon Initiative, Carnegie Mellon’s open-access database on student learning.

Carnegie Mellon would have to resolve “significant privacy issues” before the data could be put to use, Lehoczky said, “but this is part of a broader vision that the university is trying to promote.”