The Web Robots PagesA Standard for Robot Exclusion Table of contents: Status of this document Introduction Method Format Examples Example Code Author's Address Status of this document This document represents a consensus on 30 June 1994 on the robots mailing list (robots-request@nexor.co.uk), between the majority of robot authors and other people with an interest in robots. It has also been open for discussion on the
