Crawler (aka Robot or Spider)

March 24, 2006

Crawler (aka Robot or Spider) A script running for a search engine that travels the web, through hyperlinks, storing URLs and indexing content of pages it encounters. These results are then ranked and appear in Search engine result pages (SERPs)