If you’ve spent any of the past week binge-watching the second season of Netflix’s House of Cards, you’ll know that there’s a lot of talk about the “Deep Web.” (If not, be warned: Spoilers ahead.) When our stalwart and unkempt reporter Lucas Goodwin (Sebastian Arcelus) wants to dig into the dark deeds of nefarious Vice President Frank Underwood (Kevin Spacey), a techie friend helps Goodwin get onto the Deep Web and make contact with the hacking underworld, where he connects with hacker Gavin Orsay—except that Orsay has already been turned by the feds and is being used to entrap Goodwin. You can’t trust anyone on this Deep Web. Goodwin’s techie friend brags that the Deep Web is 96 percent of the Internet, with us plebes only seeing a mere 4 percent.
You might be thinking, “Whoa, the Deep Web is 96 percent of all Internet content? I must be missing out!” But I wish this number would go away. That figure refers to an entirely different definition of the Deep Web, one created back in 2001 that simply referred to anything that couldn’t be reached by crawling links. By that I mean dynamically generated Internet content without stable URLs or that required set cookies in order to view—anything that you couldn’t reliably get to just by clicking a permanent link. Online library catalogs, for one example, subscription sites like JSTOR, or sites that produce content via typed search queries, or this Hangman game. Search engines have gotten better at crawling this content, though much of the work is an exercise in avoiding crawling too much of it. That dumb Hangman game can produce more unique URLs than the entirety of Slate’s website.
Aside from gated subscription sites, however, many producers of this so-called Deep Web content want their information to be publicly available via Google. Google offers ways for websites to indicate where such content is and how best to crawl it. Some sites are indifferent to whether Google indexes them, but Google will try to crawl them anyway.
Sometimes the definition of Deep Web has been extended to include non-publicly accessible sites, such as organizational intranets—think the Department of Transportation or Dunder Mifflin. And although there’s probably some juicy stuff hidden in those mountains of publicly inaccessible Web pages, it’s not exactly the mountain of crime that House of Cards makes it out to be (unless you’re looking at New Jersey’s intranet). Nor is it anything like the virtual reality counterculture described as the Deep Web (and “DeepArcher”) in Thomas Pynchon’s recent Bleeding Edge (but then, accuracy was certainly not Pynchon’s intent).
In the House of Cards context, the Deep Web is a Wild West of illegality and shady activity, sometimes called “Darknet.” This is a distinct subset of the Deep Web, which only includes sites that are publicly accessible (not on a firewalled intranet, that is) but lack DNS addresses (like slate.com) or a known IP address (Slate’s is 18.104.22.168, last I checked). Those DNS and IP addresses form the backbone of the public Internet routing infrastructure. This Darknet is stuff that is not available through that infrastructure without going through customized and anonymized Darknet routing.
This Darknet is what you can reach via decentralized, anonymized nodes via a number of networks including Tor (short for The Onion Router—it has nothing to do with torrents) and I2P (Invisible Internet Project). Client tools (that also go by the same names) connect you to these networks while obscuring the source of your traffic. They encrypt and route traffic through random nodes in the decentralized network in order to make it difficult to identify where any piece of traffic is actually coming from. You are “off the grid,” so to speak.
What all this means is that people using Darknet are people who don’t want to be identified or caught using Darknet. The most notorious site of the entire Darknet is/was the black market the Silk Road, which was shut down by the feds last year when the alleged founder and operator (who went by the nom de Web Dread Pirate Roberts) was charged with drug trafficking.