Skip to Main Content

Libraries & Cultural Resources

Research Guides

Guide to Research and Writing for the Academic Study of Religion

Invisible Web

Now that you know how to find information on the Web using search engines and directories, the bad news is that much of the Web is what is called invisible. Search engines, in particular, are limited from finding a large portion of the Web, a portion often figured to be twice as large as the visible Web. While directories do not have the same limitations with the invisible Web, since they are created by humans, they are still primarily accessing the visible Web.

 

Why are Parts of the Web Invisible?

There are many reasons that Web pages can be invisible to search engines:

 

UNKNOWN PAGE

If a Web page is not linked to in already indexed pages then a search engine’s spiders cannot find it.

PRIVATE WEB

Some sites are blocked intentionally in order to remain confidential or to ensure that users pay to access them. All password protected sites are invisible to search engines. Unless you have access rights (such as being a University of Calgary student for University of Calgary Library resources on the Internet) you will not be able to use private Web sites.

UNREADABLE FORMAT

Some information is invisible because it is in a file format that particular search engines cannot read, although most search engines now can find pages in pdf, word, excel and powerpint . 

DYNAMIC PAGES

When accessing a database through the Web (such as the Library catalogue) it is often easier for the site to create pages as they are needed, based on each individual search, rather than store all the possible responses. These pages that are created on demand are called dynamic pages and are not stored anywhere and so cannot be searched by search engines. 

EXCLUDED PAGES

Some Web sites are enormous and many search engines will index only the first or first few pages of each site. This means that pages that are buried in a site are missed. There are some search engines that index much further, doing what is called a deep crawl, though some sites remain too large to be indexed comprehensively. The number of pages that are catalogued per site varies from engine to engine which means the ‘depth’ of your search will vary with each search engine.

PAGES DELIBERATELY EXCLUDED BY OWNERS

A web page creator can add "meta tags", unseen to the user,  that will prevent search engine crawlers from indexing their page.‘

There is a wealth of information in the invisible Web, primarily hidden in databases and file types that search engines exclude. Though a search engine can find a database for you, it often cannot search that database, so all the information that it contains remains invisible unless you search it directly.

To make full use of the Web, being able to access the invisible Web is essential!

 

 

Finding Information on the Invisible Web

When searching for information on the invisible Web you are primarily searching for useful database.. Some directories exist whose main purpose is directing users to invisible Web resources. They can be a good place to start your invisible Web search.  Remember also to use the Databases that the University of Calgary Libraries have licenced or  purchased for your use:

  • OAIster - union catalogue of academic digital resources, links to images of actual works.  OAIster is now part of Worldcat and automatically searched when you conduct a Worldcat search.

 

Note, you can also use subject-specific directories to locate databases and other invisible Web resources.

Libraries & Cultural Resources