Automated website crawlers are powerful tools to help crawl and index content on the web. As a webmaster, you may wish to guide them towards your useful content and away from irrelevant content. The methods described in these documents are the de-facto web-wide standards to control crawling and indexing of web-based content. They consist of the robots.txt file to control crawling, as well as the r