There are hundreds of search engines and search directories that contain Invisible Web content. What is more, there are even ones that claim to be an exhaustive collection of such documents. Three of the most popular search engines for the Invisible Web are Direct Search, The Invisible Web, and Complete Planet. They index mainly, but not only, Invisible Web content.
If you are searching for content that is limited to a particular topic only (e.g. programming), a great time-saver are topical search engines, because they return results related to the selected area. There are search engines for almost every topic you can think of – from gardening to nuclear weapons. A nice list of topical search engines can be found.
Meta search engines return combined results, harvested by a syndicate of search engines, so if Google does not have a particular page in its index but this page is indexed by Yahoo or another search engine, you will get it retrieved in the results. Metasearch engines were popular especially before the advent of Google but now they are regaining popularity. Examples of popular metasearch engines are Copernic, Beaucoup.
Some of the search engines offer a directory service as well and there you can browse by topic. Besides the directory services of search engines, there are specialized search directories. Search directories are an important place to submit your site to.
Once upon a time the distinction between a search directory and a search engine was clear but today, when search directories provide search tools and search engines offer lists of topical links, there is no sharp boundary between the two services. Basically, the difference is that search engines crawl the Web to find pages, while people submit their pages to search directories. Search directories are collections of links that are organized hierarchically by topic – for instance the top-level topics are entertainment, business, education, technology, etc. These topics are further divided into subtopics, which in turn have subtopics of their own, etc. In the above example, subtopics of technology, for example could be computers, biotechnology, personal tech, etc. One of the advantages of search directories is that their content is reviewed by humans and irrelevant pages are excluded from the listings.
Some of the most popular search directories are the Open Directory Project DMOZ, The Invisible Web Directory, Librarians’ Index to the Internet, About, Infomine, Yahoo!, the Directory Service, Google etc.
Databases, Specialized Portals and Reference Sources
Invisible Web Databases, specialized portals and reference sources can be an extremely valuable resource especially for very specific stuff. While search engines and search directories list only links to pages (and it happens that these links are broken), specialized databases and portals generally contain the pages themselves, so it is less likely to encounter a broken link or a missing document. There are specialized databases and portals for many topics, and if you write “medical database”, for example, Google will display a long list of medical databases only. Now you can go to the URL of the database and search from there. Another example of a specialized database is find articles, which contains over 5 million articles most of which are not indexed by the major search engines. A valuable source for virtual reference is The Internet Public Library. Of course, there are many other resources that can be quoted but I leave the fun of discovering them to you.