Robots Exclusion ProtocolNetwork Working Group M. Koster Internet-Draft Stalworthy Computing, Ltd. Intended status: Draft Standard G. Illyes Expires: January 2, 2020 H. Zeller L. Harvey Google July 01, 2019 Robots Exclusion Protocol draft-rep-wg-topic-00 Abstract This document standardizes and extends the "Robots Exclusion Protocol" <http://www.robotstxt.org/> method originally defined by Martijn Koster in 1996 for servic
