robots.txt - WikipediaExample of a simple robots.txt file, indicating that a user-agent called "Mallorybot" is not allowed to crawl any of the website's pages, and that other user-agents cannot crawl more than one page every 20 seconds, and are not allowed to crawl the "secret" folder. robots.txt is the filename used for implementing the Robots Exclusion Protocol, a standard used by websites to indicate to visiting web
