How to use it
- Start from "allow everything" and only block what crawlers really shouldn't fetch: admin areas, internal search results, cart and checkout pages.
- The file has one group for all crawlers (*) and, if you tick them, one group blocking the AI crawlers you chose. A group for a named crawler replaces the * group for that crawler, so rules meant for everyone must be repeated there if you edit the file by hand.
- List your sitemaps with full URLs.
- Upload the file as /robots.txt at the root of each host (https://example.com/robots.txt), then check it with the robots.txt tester.
AI crawlers
You can add rules for crawlers that collect content for AI models (GPTBot, CCBot, Google-Extended, ClaudeBot and others). Blocking them is a policy choice; it doesn't affect Google Search rankings.
Frequently asked questions
Will robots.txt hide private pages?
No. It's public and only asks polite crawlers not to fetch URLs. Protect private content with a login.
Should I block CSS and JavaScript?
No. Search engines need them to render pages; blocking them can hurt how pages are understood.