A text file that’s included on a site; it directs search engines to include/exclude certain pages from their indexes. These can be submitted manually to various search engines to ensure that only the latest version is followed no matter what the “crawl cycle” is.
previous post