FACTOID # 7: The top five best educated states are all in the Northeast.
 Home   Encyclopedia   Statistics   States A-Z   Flags   Maps   FAQ   About 


FACTS & STATISTICS    Advanced view

Search encyclopedia, statistics and forums:



(* = Graphable)



Encyclopedia > Web spider

See WebCrawler for the specific search engine of that name.

A web crawler (also known as web spider) is a program which browses the World Wide Web in a methodical, automated manner. A web crawler is one type of bot. Web crawlers not only keep a copy of all the visited pages for later processing - for example by a search engine but also index these pages to make the search narrower.

In general, the web crawler starts with a list of URLs to visit. As it visits these URLs, it identifies all the hyperlinks in the page and adds them to the list of URLs to visit. The process is either ended manually, or after a certain number of links have been followed.

Web crawlers typically take great care to spread their visits to a particular site over a period of time, because they access many more pages than the normal (human) user and therefore can make the site appear slow to the other users if they access the same site repeatedly.

For similar reasons, web crawlers are supposed to obey the robots.txt protocol, with which web site owners can indicate which pages should not be spidered.

The procedure of following links and not submitting queries to databases causes much content to be ignored: the deep web.

See also: Google, PageRank, Data mining

External links

  • InternetAdSales.com: Robots, Spiders, Crawlers and HTTP_User_Agents (http://www.internetadsales.com/modules/wfsection/index.php?category=23) - Comprehensive listing of common web crawlers
  • Google Dance Tool (http://www.google-dance-tool.com/) - A tool to help webmasters determine when Google's webcrawler is crawling the web

  Results from FactBites:
Funnel-web spiders (0 words)
Funnel-web spiders are found in eastern Australia, including Tasmania, in coastal and highland forest regions - as far west as the Gulf Ranges area of South Australia.
The Sydney Funnel-web Spider (Atrax robustus) occurs in New South Wales, from Newcastle to Nowra and west to Lithgow.
Female funnel-web spiders spend most of their life in their burrows, but do occasionally hunt on the surface at night.
Build a Web spider on Linux (3332 words)
Web spiders are software agents that traverse the Internet gathering, filtering, and potentially aggregating information for a user.
Web spiders and scrapers are simply another form of software robot or agent (as coined by Alan Kay in the early 1980s).
Web spiders and scrapers are useful applications, and, therefore, you can find a variety of different types in use for both good and evil.
  More results at FactBites »



Share your thoughts, questions and commentary here
Your name
Your comments

Want to know more?
Search encyclopedia, statistics and forums:


Press Releases |  Feeds | Contact
The Wikipedia article included on this page is licensed under the GFDL.
Images may be subject to relevant owners' copyright.
All other elements are (c) copyright NationMaster.com 2003-5. All Rights Reserved.
Usage implies agreement with terms, 1022, m