Msnbot, discobot, and Slurp are all called out specifically, so those user-agents will only pay attention to the directives in their sections of the robots.txt file. If the file contains a rule that applies to more than one user-agent, a crawler will only pay attention to (and follow the directives in) the most specific group of instructions. In a robots.txt file with multiple user-agent directives, each disallow or allow rule only applies to the useragent(s) specified in that particular line break-separated set. Within a robots.txt file, each set of user-agent directives appear as a discrete set, separated by a line break: Together, these two lines are considered a complete robots.txt file - though one robots file can contain multiple lines of user agents and directives (i.e., disallows, allows, crawl-delays, etc.). These crawl instructions are specified by “disallowing” or “allowing” the behavior of certain (or all) user agents. In practice, robots.txt files indicate whether certain user agents (web-crawling software) can or cannot crawl parts of a website. The REP also includes directives like meta robots, as well as page-, subdirectory-, or site-wide instructions for how search engines should treat links (such as “follow” or “nofollow”). The robots.txt file is part of the the robots exclusion protocol (REP), a group of web standards that regulate how robots crawl the web, access and index content, and serve that content up to users. Robots.txt is a text file webmasters create to instruct web robots (typically search engine robots) how to crawl pages on their website.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |