How the rules are applied
- The crawler uses the group whose User-agent matches its name (Googlebot, Bingbot…); if none matches, the * group. Several groups naming the same crawler are merged.
- Within the group, the longest matching Allow or Disallow path wins. On a tie, Allow wins.
- * matches any characters and $ anchors the end of the URL. Paths are case-sensitive.
- An empty Disallow allows everything. A robots.txt answering 404 means no restrictions; a 5xx makes Google pause crawling.
What else is reported
Sitemap lines, Crawl-delay (ignored by Google, read by Bing and Yandex), rules written before any User-agent, unknown directives, and a Noindex line, which Google stopped supporting in 2019. This is RankCortex's reading of RFC 9309; each crawler implements the standard its own way.
Frequently asked questions
Does Disallow keep a page out of Google?
No. It stops crawling, not indexing. A blocked URL can still be indexed from links, without its content. To keep a page out of search results, allow crawling and use a noindex meta tag.
Why test Googlebot and not just *?
A group written for Googlebot replaces the * group for Google entirely, which surprises many site owners. Test each crawler you care about.