The Invisible Web: Searching Deep and Wide

There are 2 kinds of websites... Common and the Invisible.

The "invisible web", or deep web, is a part of the World Wide Web that is not so easily searched. It is not listed with any search engines, but you can still get to them, if you know how and where to look. Alot of the invisible web is in databases from Universities , or you might take a look at Google Scholar. If you google your topic, then add the word 'database", you can usually get something that's on the invisible web. Most the web is found this way. This is usually a more focused web search.

Did you know there is more of the web that is invisible, than there is through common search engines ? The web that we commonly get to by search engines , called 'surface web', is only about 167 terabytes. The Library of Congress itself is only 11 terabytes big. The invisible web is about 91, 000 terabytes big. What is crawled by common search engines like Google, Yahoo, are just a smigen of what's on the web. Google alone crawls about 8 billion pages... "Big Planet", another search engine for the deep web estimates the deep web is about 500 times bigger than what the regular search comes up with.

In order to get to the invisible web, you need to do a 'federated search' of the databases. InfoMine is one such search engine for the deep web. Web Harvesting is done by humans who run into interesting websites....Stumble Upon is an example of this. The University of Michigan has a large database that you can search through, as does the University of Virginia. The invisible web stores a wealth of information the common search engines can't see in it's crawling of websites.

Sources to get to the Invisible Web

Science.gov: anything in the scientific field.

Complete Planet: this has over 70,000 searchable databases and specialty search engines.

ResourceShelf: This website has all kinds of info on the invisible web. The search engine 'Direct Search' is part of this site.

PubMed: This can get you millions and millions of medical info on anything from common to rare conditions and diseases

FindArticles: This will get you to leading acidemic, industry and general interest articles with over 10 million articles at your fingertips.

Educators Referance Desk: Large database on lesson plans, full text expert digest reports, and more.

Super Searchers Web Page: This offers a growing collection of links to university databases, and recommended by the world's leading web researchers.

WebData: Browse through the list of searchable databases.

There are hundreds more 'invisible web' search engines out there, such as : CloserLooksearch and NorthernLight. Google and see what you can find!