ExtremeTech explains: All about the dark web, and how to use it

Security surveillance unlock privacy

If you’ve paid any attention to online marketplaces for illegal goods like the now-defunct Silk Road or the FBI’s investigations into criminal in cyberspace, chances are you’ve heard the term “dark web.” Curious about what it means? You’ve come to the right place.

The dark web is sometimes called onionland because of its content accessible only using services like Tor. The rest of the internet is simply referred to as the clearweb, since it isn’t generally encrypted.

How does the dark web work?

The dark web works just about the same as the regular internet: it uses the same TCP/IP framework to transmit HTTP and FTP traffic within and between networks, over the same phone, cable or FiOS lines that carry regular internet traffic. Content on the dark web consists of HTML webpages and their assets, just like it does on the rest of the web. In fact, under the hood, the dark web is the same as the regular web, with two important exceptions that also distinguish the dark web from the deep web.

First: the dark web isn’t indexed by search engines. Second, content on the dark web can’t be accessed with regular web browsing software alone; additional software is required to make the networks talk to one another.

This is because content on the dark web is hosted on overlay networks, which are physically connected to the internet but aren’t accessible to web crawlers. That relative inaccessibility is because the dark web uses a complete, but fundamentally different, network addressing system than the web addresses most of us know and use. Browsers like Chrome and Firefox are programmed to access website files using the DNS index, which turns a file’s unique address on its unique server into a string of text that you can type into your address bar. Sites indexed by the DNS registry are accessible via top-level domains like .com and .org, among others. After ICANN opened up the suffixing system to other strings of text, we started to see web addresses that look like home.cern and bit.ly — but you can still type those into your address bar and get to a website, because they’re in the official DNS registry. Dark websites don’t participate in the DNS system, and web crawlers don’t have the software to get onto the dark web, so the dark web and the clearweb don’t really cross-pollinate.

Content obscured in this way can still be accessed, but you need the right software. It’s a bit like a Wi-Fi network that doesn’t broadcast its SSID: you can only get access if you already know exactly how to find it. Some content accessible only through Tor is hosted at a .onion pseudo-top-level domain, which means that in the right software, you might type in foobar.onion and get to the Foobar dark website.

Such software, including the Tor browser bundle, is capable of bridging the differences in network behavior between the dark web and the clearweb. But that only works when you’re using a compatible browser and have the right encryption. Tor, Freenet and I2P are the most commonly cited examples of software capable of accessing the dark web. Typing a .onion address into your Chrome address bar won’t get you anywhere. Furthermore, many if not most .onion sites are generated sixteen-character “non-mnemonic” alphanumeric strings, rather than being composed of words like most clearweb URLs.

There also exists a difference in the path web traffic takes on the clearnet versus the dark web. Tor is valuable because it sends your own web traffic through multiple different network nodes, masking its origin and destination. There’s significant overlap between VPNs and the dark web; both services use encryption and multiple network nodes to anonymize traffic. But VPNs deal with clearweb sites that participate in the DNS system, while dark web browsers deal with domains not recognized by ICANN.

What is the dark web used for?

The structure of the dark web makes it anonymizing, which means that first and foremost, it’s used for anonymous communication and web browsing. This accounts for the vast majority of network traffic through Tor. Why seek out anonymity? To read and write about things that might get you in trouble, like political dissent or whistleblowing. The same technology that enables Tor is capable of tunneling out from behind the Great Firewall of China, and the US government contributes to the development of such software.

Anonymity also brings out those who wish to do illegal things. A 2014 study found that of the different kinds of sites on the dark net, there are more markets devoted to drugs and guns than any other kind of dark site, including forums, bitcoin laundering, hacking, fraud, whistleblowing and even regular old porn.

To paraphrase Jim Jeffries, if you want to murder someone, you can’t just walk up to Pier 31 and shout “GUNS, WHO WANTS TO SELL ME SOME GUNS!?” But with a website like an evil eBay that lists weapons and other contraband for sale, all of a sudden you don’t have to know someone with “black market connections.” You just have to be able to install some software.

Tor hidden services are the other thing the dark web does, and they’re what gives the dark web its shady reputation. Hidden services refers to dark sites where both the host and the visitor are anonymous to one another. That technology enables dark web sites that host illegal content to persist. Hidden services account for only 1.5% of the Tor network volume. But the overwhelming majority of resources requested over Tor hidden services — fully 80% of that traffic — were requests from child abuse sites. Outgoing traffic from the dark web flowed mainly between botnets and their hidden control servers. More detail on Tor’s traffic patterns and how much of its total bandwidth is used for illegal activities is available in a blog post by the Tor project.

The dark web is notoriously dodgy territory for both buyers and sellers. Law enforcement has been chipping away at the nominal anonymity afforded by software like Tor, and anything of interest on the dark web is as likely to be a scam as it is to be a honeypot. Between social engineering and software vulnerabilities, it is a realm best accessed while wielding some trustworthy anti-malware.

For a long time, the Silk Road was the biggest game in darknet commerce. It allowed users to sell a great many illegal things, and inspired a number of similarly designed copycat markets. Transactions there were conducted in bitcoins and other virtual currency, and then goods were shipped through the mail. But a high-profile bust and ensuing court case put several Silk Road admins in jail. The media spotlight has impinged on the Silk Road’s relative obscurity, reducing its value as a black marketplace.

From Tumblr.

While Uncle Sam contributes to the development of Tor and similar anonymity resources, the government is also known to take more of a proprietary approach, considering even the dark web to be within American jurisdiction when site hosting is in question. The FBI paid Carnegie Mellon to crack Tor in pursuit of a criminal case. They even waded into the muck and ran a huge sting operation on Playpen, a darknet child porn site — by taking over control of the site and running it for weeks as a poisoned well to catch its users.

lIKE ()orShare