Last Monday morning, the search engine Cuil launched with great fanfare. By Monday afternoon, it had completely tanked. Users who test-drove the would-be Google rival were quick to complain about mismatched articles and thumbnail photos; the poor breadth of results; obvious queries that turned up blank; and even, in a moment of true existential crisis, the site’s inability to locate itself.
It’s unfair, of course, to judge a new site on its opening-day performance. Still, it’s not clear how we should evaluate a new search engine. In the same way that Kurt Vonnegut once proposed that the definitions for ain’t and like could identify whether a dictionary was prescriptive or descriptive, it’s high time for a quick and dirty litmus test for the Googles (and Cuils) of the world. Last week, I asked readers to come up with some standard queries that we should use to measure our search engines. I also wanted an explanation of how those questions could be used to reveal each site’s philosophy.
For an extremely thorough evaluation of how major search engines measure up against one another, Rand Fishkin at SEOmoz has oodles of tables and charts. But if you prefer a quick and dirty version—a Vonnegut test for the digital age—here are the three searches that I propose. (Beneath each search term are links to the results given by five major search engines. I’m ignoring sponsored links in all cases.)
George W. Bush
Google, Yahoo, Live, Ask, Cuil
“I tried George W. Bush and got a great view at the different search engines,” writes reader Jeff Alhadeff. “Google responds with news, Wikipedia, the White House, his library, and then some links demonstrating his lack of popularity. …Yahoo natural search nearly the same, but lighter on the anti-Bush links. … Cuil responded with ‘We didn’t find any results for George W. Bush.’ I am still laughing. (I had to run the search without the period after the W.)”
A query for our current president points to an interesting philosophical question for any search engine: Which should be higher in this search, whitehouse.gov or Wikipedia? The former is the president’s official site, the latter is the pre-eminent reference site on the Web and considerably more objective. This is an important question of orthodoxy vs. popularity—and one on which not all search engines agree. Of the five search engines considered here, only Google and Yahoo put Wikipedia first. (And for the record, Cuil has resolved its disagreement with punctuation marks.)
2. Viagra
Google, Yahoo, Live, Ask, Cuil
“Another important factor is checking to see if the engine is resistant to spam,” writes John King. “Can spammers get into your index for Viagra keywords?”
How well does a search engine sniff out spam—and when, if ever, are sites that sell ED pills over the Internet legitimate? Of the five engines I tested, only Cuil gives high rankings to results that are clearly spam. Yahoo’s first page of results was the most informational, with links to health sites, clinics, and the FDA. Live Search, by contrast, has five first-page links to online pharmacies and other sites hawking the drug.
It’s not obvious which approach is correct, though I’m thinking that Live might be on to something. What are people who search for the word Viagra more likely to be after, a site that discreetly sells the drug or a HowStuffWorks article on the mechanics of the corpora cavernosa? Perhaps search engines are putting propriety in front of the results that users really want.
3. Your Own Name
More than half of everyone who wrote in made the case for a vanity search as the ultimate litmus test, arguing that these are the results with which the average obsessive self-searcher is most familiar. “I am intimately familiar with where my name appears online (everybody’s guilty of googling themselves from time to time, or all the time),” writes reader Caren Beilin. “I figured typing in my own name would be the quickest way to test out Cuil, since I already had a comprehensive knowledge about my name on the web. As for now, I’d rather google myself than cuil myself.”
While this test is obviously subjective, it does offer clear insights into the balance between two important priorities for any search engine: relevance and freshness. Are the results that come up for oneself more recent, or are they ones that are more vital to your digital legacy?
The vanity search test breaks down for those of us blessed with highly generic names, so I gave it a run on my editor, Josh Levin (putting his name in quote marks for exact matches). Josh fares the best on Cuil, which, unlike all the other search engines, is not squeamish about listing lots of results from the same source; 10 of the 11 front-page results are from Slate. Both Google and Ask give top billing to Slate articles, while Yahoo and Live Search give the No. 1 spot to sites devoted to people of the same name—josh-levin.us and joshlevin.com, respectively.
The differences here demonstrate that a search engine has to choose between foregrounding a close match or going with a site with lots of authority. By putting a Slate article as its top result, Google and Ask reveal a bias for authority over closeness. By linking to sites with the keywords in the URLs, Yahoo and Live show a preference for closeness over authority.
Search engines ultimately aspire to produce results that are both intuitive and correct. In the end, your decision about what search engine is right for you may come down to a matter of opinion: Who spews more disinformation, Wikipedia or the White House?
