Crawler (aka Robot or Spider)
March 24, 2006
Crawler (aka Robot or Spider) A script running for a search engine that travels the web, through hyperlinks, storing URLs and indexing content of pages it encounters. These results are then ranked and appear in Search engine result pages (SERPs)
