Google only indexes a tiny fraction of the internet. By some estimates, the web contains 500 times more content than what Google returns in search results. The links that Google and other search engines return when you type in a query is known as the “surface web,” while all the other, non-searchable content is referred to as the “deep web” or “invisible web”.
Most of that information is hidden simply because the vast majority of users won’t find it relevant. Much of it is tucked away in databases that Google is either not interested in or barred from crawling. A lot of it is old and outdated. The contents of iPhone apps, the files in your Dropbox account, academic journals, court records, and private social media profiles are all examples of data that aren’t necessarily indexed by Google but still exist on the internet.
Much of this article revolves around the use of anonymity networks like Tor, which are used to access the dark net. Internet providers can detect when Tor is being used because Tor node IPs are public. If you want to use Tor privately, you can use either a VPN or Tor Bridges (Tor nodes that are not publicly indexed). US Tor users in particular may want to use a VPN, which will be faster and more reliable.
Recent changes in US legislation mean internet providers are free to sell and share data on their customers, including their browsing habits. When using a VPN, your ISP will not be able to see that you are connected to a Tor entry node, only an encrypted tunnel to a VPN server.
The deep web is often confused with the dark net, also called dark web, black web, and black net. Put simply, the deep web is all of the information stored online that isn’t indexed by search engines. You don’t need any special tools or a dark net browser to access the deep web; you just need to know where to look. Specialized search engines, directories, and wikis can help users locate the data they’re looking for.
Many of the best general deep web search engines have shut down or been acquired, like Alltheweb and CompletePlanet. Still, a few are hanging around to get you started:
- DeeperWeb – Deep web search engine that leverages Google Search
- The WWW Virtual Library – The original index of the web, but more of a directory than a search engine.
- Surfwax – Indexes RSS feeds. Not certain this is still working…
- IceRocket – Searches the blogosphere and Twitter
These are all okay, but specialized search engines tend to be better than general ones for finding info on the deep web. If you’re looking for a court case, for example, use your state or country’s public records search. If you need academic journals, check out our article on using deep web search engines for academic and scholarly research. The more specific you can be, the better, or else you’ll just end up with the same search results that you would find on Google. If you need a specific file type, like an Excel file or a PDF, learn how to specify searches for that type of file (e.g. type “filetype:PDF” in your DeeperWeb query).
The dark net is a small part of the deep web that is kept hidden on purpose. Websites and data on the dark web do typically require a special tool to access. The type of site most commonly associated with the dark web are marketplaces where illicit goods such as narcotics, firearms, and stolen credit card numbers are bought and sold. The darkest corners are used to hire hitmen, engage in human trafficking, and exchange child pornography. More than that, though, the dark web contains content and data that can be accessed with anonymity. It could be a blog, forum, chat room, or private gaming server.
The beauty of the dark net is anonymity. No one knows who anyone else is in the real world, so long as they take the necessary precautions. Users are free from the prying eyes of governments and corporations.
The dark web and Tor are often used by journalists and whistleblowers to exchange sensitive information, including Edward Snowden himself. The Ashley Madison data dump, for instance, was posted to a site only accessible to Tor users.