Pierluigi Paganini wrote a great blog post today. The Deep Web (or Invisible Web) is all the information on the World Wide Web not reported by normal search engines. It's HUGE. According to several researchers, the major search engines index only a very small portion of the overall web content, about a quarter of one percent. Call it censorship if you want but the vast majority of the web is unknown to most web users. The Deep Web is a network of interconnected systems, they are not indexed, and are 500 times larger than the current web. Whoa Nellie, 500 times? Yup.
How is that possible? Web search engines use so called "crawlers" to index the pages they can see, but there is a LOT that they cannot see, like web pages that change all the time (dynamic content), pages that are not linked in any way, whole websites that are password-protected, and more.
You can see it as a parallel web with a lot more information that is valuable for private companies, governments, and of course also cybercrime. For instance, the recently stolen Target credit card information is for sale on invitation-only websites in the Deep Web. If you want to learn more about this, check out his blog post that will take you for a first trip into the abyss.
