検索エンジンのロボットによるクロールをブロックするrobots.txtについてGoogleのJohn Mueller氏がTwitterとGoogle BuzzでTIPSを公開しています。 Robots-tip: crawlers cache your robots.txt; update it at least a day before adding content that is disallowed. Q&A in Buzz. If you add a new subdirectory on your site and simultaneously add a disallow for it in your robots.txt file, it’s possible that crawlers will not see the disallow before they crawl t
![robots.txtの更新は1日1回 + noindex / nofollow / noarchive / nosnippet / noydir / noodp タグのおさらい](https://cdn-ak-scissors.b.st-hatena.com/image/square/f00f4804fe59acd36d8e9d02e50d10db7dc9a119/height=288;version=1;width=512/https%3A%2F%2Fwww.suzukikenichi.com%2Fblog%2Fwp-content%2Fuploads%2Fsplash_1200_675.png)