Will Robots Steal Your Job?
We wrote a computer program that replicates Jason Kottke. How does Robottke compare to the Web's best link blogger?
I asked my colleague Chris Wilson to build a robotic Jason Kottke because the human version looked like he could use a break. Kottke launched his eponymous site in 1998, back when blogging was just getting started. The Web has changed enormously in that time, but Kottke.org remains as vital as ever. Part of why I like the site is that Kottke and I share many interests: science, technology, Michael Lewis, typography, The New Yorker, David Foster Wallace, The Wire, and North Korea. But lots of people blog about those subjects. Kottke's genius is his ability to find links that elude the rest of us. Every day, I come across hundreds of URLs on Twitter, Facebook, email, and dozens of blogs. But of the five or six links Kottke posts on a typical day, two or three are brand new to me, and almost every one points me toward an awe-inspiring corner of the Web.
There used to be many Kottkes. During the heyday of blogging—between the late 1990s and about 2006—Kottke-like "link blogs" were a primary format. Sites like John Barger's Robot Wisdom served a useful purpose for readers—there was new stuff online everyday; you went to a link blog to find the best of it—while allowing authors to demonstrate their curatorial mastery over the Web.
There are still several popular link blogs today; Boing Boing, Daring Fireball, and The Drudge Report are canonical examples. But the format is on the decline. Barger, for instance, no longer blogs—instead, he tweets and shares links using his RSS reader. Sites like Reddit, Digg, and Hacker News also allow readers to submit links and rank them by votes. Other services scan blogs, Twitter, and Facebook in search of the most popular stuff (see Gabe Rivera's Techmeme, Memeorandum, and Mediagazer). Several times a day, I get an email from a wonderful service called Summify that sends me the most-shared links among the people I follow on Twitter. Kottke himself now spends less time blogging than building a link-collecting service called Stellar, which lets you follow the things people liked on Twitter, Flickr, YouTube, and Vimeo.
It's obvious why these new systems have superseded Kottke-style blogs. The handmade link blog depends on one person's—or a handful of people's—ability to sort a never-ending stream of new content. "Some days, you just don't want to do it," Kottke says. "You look at so much stuff everyday and it all becomes kind of the same—all equally interesting or uninteresting. It's hard to maintain that sense of discovery, that little hit that you get when you find something that you haven't seen before. I've posted 15,000, maybe 20,000 links since I started. I've been whittling down the discovery space of things that are going to be new and interesting."
In computer science parlance, Kottke doesn't scale. That's a shame. While services that collect popular stuff online are useful, they lack any editorial sensibility. The links on Techmeme and Summify represent a horde's view of the Web. The material on Kottke represents one guy's indispensible take. The Web ought to have both kinds of aggregators, but I'd love to see more people starting link blogs that offer a clear editorial vision. But how do you get more of something so hard to do?
Enter Robottke. Over the last few weeks, Chris Wilson has been building a machine that aims to automatically generate links you might find on Kottke.org. Robottke isn't meant to replace flesh-and-blood Kottke; we just want to come up with a list of items that Jason Kottke might link to each day.
You can check out Robbotke here. How does it work? We began by crawling all the sources that Jason Kottke is likely to look at every day—we look at all the sites he links to, and all the stuff that people he follows on Twitter are sharing. The hard part is choosing the best, most Kottke-like links from Robottke's collection. It's helpful that the human Kottke meticulously tags all of his posts with keywords. When Robottke finds a link, it searches for topics that it knows Kottke likes—the more it finds, the higher the article ranks.
The results can be charming: As I write this, Robottke's top link is to a video that explores the scientific mystery of how a bike can stay balanced when it rolls along without a rider. Robottke can also miss the mark wildly: Other links in today's edition of Robottke include a 2010 Washington Post op-ed about the cost of the Iraq war (too old, too political for Kottke), an interview with Michael Moore (also too political), and a link to the traffic stats page of the Cynical-C Blog (oops—that tricked our algorithm).
My verdict: Robottke is far from perfect, but it's not bad, either. With some more effort and artificial-intelligence expertise, I believe Robottke stands a good chance of succeeding.
When I asked Jason Kottke what he thought, he was diplomatic. "Um," he began, and then paused for a few seconds. "I think some of them are good and some are way off. The biggest thing is that it's not necessarily about the links. People tell me that what they like about my site is what I say about the links—it's how I present the links, how I sell them in a certain sense. And this doesn't do a very good job of that at all."
Farhad Manjoo is Slate's technology columnist and the author of True Enough: Learning To Live in a Post-Fact Society. You can email him at firstname.lastname@example.org and follow him on Twitter.
Photograph of Robbie Allen courtesy Robbie Allen.