サクサク読めて、アプリ限定の機能も多数!
トップへ戻る
買ってよかったもの
www.robotstxt.org
About /robots.txt In a nutshell Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. It works likes this: a robot wants to vists a Web site URL, say http://www.example.com/welcome.html. Before it does so, it firsts checks for http://www.example.com/robots.txt, and finds: User-agent: * Disallow: / The "User-agent
Robots Database The Robots Database lists robot software implementations and operators. Robots listed here have been submitted by their owners, or by web site owners who have been visited by the robots. A listing here does not mean that a robot is endorsed in any way. For a list of User-Agents (including bots) in the wild, see www.botsvsbrowsers.com This robots database is currently undergoing re-
A Standard for Robot Exclusion Table of contents: Status of this document Introduction Method Format Examples Example Code Author's Address Status of this document This document represents a consensus on 30 June 1994 on the robots mailing list (robots-request@nexor.co.uk), between the majority of robot authors and other people with an interest in robots. It has also been open for discussion on the
Web Robots (also known as Web Wanderers, Crawlers, or Spiders), are programs that traverse the Web automatically. Search engines such as Google use them to index the web content, spammers use them to scan for email addresses, and they have many other uses. On this site you can learn more about web robots. About /robots.txt explains what /robots.txt is, and how to use it. The FAQ answers many frequ
Frequently Asked Questions This is a list with frequently asked questions about web robots. Select the question to go to the answer page, or select on the eye icon after the question to show the answer in this page. About WWW robots What is a WWW robot? What is an agent? What is a search engine? What kinds of robots are there? So what are Robots, Spiders, Web Crawlers, Worms, Ants? Aren't robots b
このページを最初にブックマークしてみませんか?
『The Web Robots Pages』の新着エントリーを見る
j次のブックマーク
k前のブックマーク
lあとで読む
eコメント一覧を開く
oページを開く