GitHub - google/robotstxt: The repository contains Google's robots.txt parser and matcher as a C++ library (compliant to C++11).

The Robots Exclusion Protocol (REP) is a standard that enables website owners to control which URLs may be accessed by automated clients (i.e. crawlers) through a simple text file with a specific syntax. It's one of the basic building blocks of the internet as we know it and what allows search engines to operate. Because the REP was only a de-facto standard for the past 25 years, different impleme

[netcraft3]

Googleのrobots.txtパーサーがオープンソースで公開されてる。

[braitom]

Googleのrobots.txtパーサーがオープンソース化された。C++製。