Where do forecasts come from, and which ones are the most accurate?

Frustrated Slate sters wanted to know: Where to go for weather on the Web? Web forecasts seem "wildly divergent," huffed a New York-based editor, making it "tough to pack for trips." A Seattle scribe claimed MSNBC's predictions were more "upbeat" than Yahoo!'s. (Try this forecast, Yahoo!, High 205, Low 13.5.) Bitterness mounted as we sought to define "Partly Sunny." An investigation was in order.

I chose two cities—Boston (where I live) and Seattle (Slate's home base)—and set out to test a few online forecasters: Weather.com, AccuWeather.com, MSNBC, CNN, Yahoo!, the National Weather Service, and, as a local entry, Boston's WCVB-TV home page. I logged on each night to check next-day predictions for highs, lows, and conditions, then compared these guesses to the actual weather.

Where Do Forecasts Come From?

After just two days, I reached my first conclusion: AccuWeather, MSNBC, and CNN were posting identical forecasts. Close reading revealed this fine print on the MSNBC page: "MSNBC Weather is provided by AccuWeather." Aha! And CNN AccuWeather Senior Vice President Michael Steinberg explains that his massive meteorological hive, located in Pennsylvania, indeed provides forecasts for CNN.com, hundreds of other sites, and about 750 newspapers (including the Washington Post). Sometimes AccuWeather gets credited on the sites, other times it doesn't. Steinberg was quick to point out that while AccuWeather posts its forecasts on other sites, Weather.com (from the Weather Channel—AccuWeather's arch-nemesis) doesn't license out its forecasts—you must visit its site. This explains, says Steinberg, why Weather.com is No. 1 in weather site traffic, while Accu is only No. 2.

Steinberg said any forecast discrepancies between AccuWeather.com and the sites it services arise only from the fact that AccuWeather updates more frequently. But a deeper issue had been raised: Where do forecasts come from? Do they all spring forth from a tiny weather cabal? Sort of.

According to both Steinberg and Jody Fennell, a VP at Weather.com, basically all raw weather data comes from the federal National Weather Service. A few local TV stations own their own radars (Doppler 5000!), but to put together a radar mosaic, or certainly for satellite data, everyone counts on the NWS. It's how you interpret the raw data (from radars, satellites, weather station observers, and so on) that determines your forecast. In fact, most NWS data is available to the public for free. (With a bit of research, a green screen, and a mustache, you could be your own weatherman.) But the major forecasting groups use their own proprietary software, with specially chosen algorithms, to form predictions. That's the system for Weather.com, AccuWeather, your local station, and pretty much everybody else: NWS data is fed through various computer models. Fennell told me Weather.com's local temperature forecasts go straight from the software to the site, with no human intervention at all. The NWS makes forecasts from its data using software, too. Yahoo! says its forecasts come from an international company called Weathernews Inc., which gets its data from, yes, the NWS.

Who's Got the Best Forecasts?

OK, the same data gets fed through different software models. But whose forecasts are best? According to AccuWeather's Steinberg, there is no industry watchdog. Occasionally, an analysis will show up in the press or in scientific literature, but most outlets review their own forecasts internally. For more than 13 years, AccuWeather has compared its forecasts for the D.C. area to the forecasts from the NWS and claims it's beaten the NWS in 156 out of 158 months. What does that mean? AccuWeather looked at the average temperature for each day, then determined whose forecast was further off and by how many degrees.

I tried a similar test, but used high temperatures instead of average. Over eight nights, I looked at forecasts for the next day's high temp in Boston and Seattle, then compared these forecasts to the actual temperatures recorded by the government.

First I measured absolute values: the number of degrees by which each guess was wrong. If Yahoo! guessed 2 degrees too high on one day, and 4 degrees too low on the next, the absolute value of degrees-off was 6. I added this up for all eight days, then divided to find a per-day average, heretofore known as the "Total Degrees off Average."