Also called a web crawler, spider, it’s a software that has been specifically designed to automatically browse the internet. Google uses a number of crawlers to constantly map all the sites on the internet and use it in its search engine.
previous post