I have seen the future, and that’s official. The Good Judgment Project, a 5-year-old government-funded exercise in world affairs forecasting, has found that volunteers like me, working in independent groups, can be strangely good at gazing into liquid crystal screens and divining the future. Let me explain.
Who doesn’t want to know more about what’s coming? I’ve thought about how to get better at seeing the future for a while now, so I was intrigued a couple of years ago when I came across an online tournament for testing your ability to make accurate forecasts. I signed up for the Good Judgment Project in 2013 and took part for two seasons.
The U.S. intelligence community, which consists of 16 agencies and has a budget of $50 billion per year (at least for the budget that is made public), has been forced into some soul-searching in recent years. After the first Gulf War in 1991, it found that Saddam Hussein was much further along in nuclear weapons development than it had realized. Then Sept. 11 happened and, despite some pretty clear warnings about possible airplane attacks, the intelligence community (also known as the IC) failed to identify specific plotters or stop them. The community embarrassed itself again in the run-up to the second Gulf War in 2003 with too-confident pronouncements about Iraqi weapons of mass destruction, which were—of course—shown to be completely wrong.
After that last debacle became apparent in the post-invasion years, a branch of the IC, the Intelligence Advanced Research Projects Activity, agreed to fund a number of private researchers to run tournaments that enlisted volunteers to do their best to answer fairly tricky questions about future world events.*
For example: Will North Korea test another nuclear device before the end of 2014? Or: Will there be a violent confrontation between China and one or more of its neighbors by March 2014 in the South China Sea? All of the questions required research, discussion, and probabilistic answers.
One of the teams of private researchers created a very successful formula for finding good forecasters. This was the Good Judgment Project, a collaboration between researchers at UC–Berkeley and the University of Pennsylvania. After the researchers found that some volunteers were particularly good at forecasting, they decided to put the most successful 2 percent of volunteers on teams with one another in the following year (if they agreed to take part again). Lo and behold, the teams of “superforecasters” were even better than when they worked alone, apparently feeding on one another’s shared commitment to the competition.
I certainly didn’t expect to become a superforecaster when I signed up for the GJP in 2013. After a slow start, I became fairly serious about my involvement as the year progressed. Each tournament year is about nine months long. I was placed on a team of a dozen random strangers and we were all encouraged to introduce ourselves on the team forum—the whole tournament process is conducted online. (The tournament organizers did provide a couple of chances for a real-life meet and greet, but I didn’t attend any of these events.) For each day of the tournament, I spent half an hour to an hour reading questions about world affairs, doing research, discussing with my teammates, and making my forecasts. Good, clean nerdy fun.
It got exciting for me when I found out a couple of months in that our team was in second place out of 25 teams. It’s always fun to be good at something, and my team’s progress kept my interest strong until the end. During my first season, we were given an individual score and a team score—running tallies of how accurate our forecasts were.
For some questions there was a lot of online discussion and frequent changes in our forecasts. Constantly checking back in and adjusting as new information came to light were key to keeping scores good. Collaboration kept things fun, but, I found out later, collaborating too much could promote “groupthink” and negatively affect scores.
Anyway, right at the end of the first year my team fell from second place to sixth place and my personal score dropped precipitously as some key long-term questions were resolved and I did abysmally.
It was a pleasant surprise for me, then, to learn after my first season that I was invited to participate in the next year as a “superforecaster,” meaning that I was in the top 2 percent of all forecasters in my year. (I continue to believe that this designation may have been a mistake, but they haven’t revoked it yet.)
When the Cal and UPenn researchers tallied up all the superforecaster scores, they found that teams of superforecasters were quite good at predicting the future. In fact, when tallied up, they found that teams of superforecasters were about 30 percent better at seeing the future than the intelligence community experts doing their best on exactly the same questions. (This information was classified but leaked.)
Yes, a bunch of ragtag non-expert volunteers beat the best that the $50-billion-a-year intelligence community could offer. The volunteers were unpaid. We each received a $250 gift certificate from Amazon for our nine months of volunteering, but I didn’t even know about that little perk when I signed up as a volunteer.
So what happened? Why were some forecasters pretty dang good at forecasting? Philip Tetlock, at UPenn, and one of the primary research leads for GJP, has co-authored a new book all about this: Superforecasting: The Art and Science of Prediction. The book is a fairly light read about the tournament and how regular people—yes, you—can get better at seeing the future.
Tetlock says superforecasters share some qualities, like being actively open-minded, seeking new data, updating one’s opinion when new data requires it, and seeking out reliable expert advice to inform one’s own opinions. Above all, however, he suggests that we can get better at forecasting by tracking our successes and failures, and adjusting accordingly.
Tetlock made a name for himself in the ’90s with his research on expert political judgment. A common summary of his work went something like this: “Tetlock found that most experts were no better than chimps at making accurate forecasts about world events.” Tetlock didn’t actually say this verbatim, but he avers in his new book that it’s actually close enough to what he did find that he doesn’t mind it too much.
He also found that, in general, the more popular and voluble the expert the less accurate he or she is. In his previous book, Expert Political Judgment, he describes these findings and sketches two basic psychological types to make sense of what makes for good political judgment and forecasting: the hedgehog and the fox.
Hedgehogs have one big idea through which they view the world, and they are reticent to change their worldview with new facts. In other words, hedgehogs are pretty dogmatic. And they’re usually terrible forecasters.
Foxes, on the other hand, adhere to no large philosophy; instead, they survey the world of ideas and facts and constantly update their worldview based on new information. They’re not very flashy or generally very good speakers. But they can be very good forecasters.
The most profound finding in Tetlock’s work, however, is the discovery that teams of forecasters do better than individuals, and better than prediction markets, and better than professionals in the intelligence community. So teams of superforecasters are the closest thing we’ve found to a modern crystal ball.
The implications of this finding are potentially huge as we think through the possible applications of this fairly easy-to-assemble small crowd for specific purposes. Why can’t we call up an Internet “smart mob” on demand, as sci-fi author David Brin, has speculated about? Why can’t superforecaster teams help inform key national security decisions, as Tetlock suggests? Why can’t we call up a team to help make major career or love decisions?
Well, we can’t yet, but there’s no reason to think that these possibilities won’t become real before too long. We’re already doing it in many ways in consulting crowds of experts for tricky scientific or engineering problems, and recent breakthroughs in molecular biology were made using crowds.
Before too long you too could have your own crystal ball.
This article is part of Future Tense, a collaboration among Arizona State University, New America, and Slate. Future Tense explores the ways emerging technologies affect society, policy, and culture. To read more, visit the Future Tense blog and the Future Tense home page. You can also follow us on Twitter.
*Correction, Nov. 19, 2015: This article originally misidentified the Intelligence Advanced Research Projects Activity as the Intelligence Advanced Research Projects Agency. (Return.)
