Robots.txt Generator
Build a robots.txt file and choose which AI crawlers may use your site, one by one.
User-agent: * Disallow: /admin/ Disallow: /cart/ Sitemap: https://example.com/sitemap.xml
Training vs search crawlers
- Training crawlers gather pages to train future models. Blocking them is a request not to use your content that way.
- Search and user crawlers fetch pages to answer a question now, often with a link back. Blocking them can remove you from those answers.
- Crawler names change over time; check each company's documentation for the latest list.
Frequently asked questions
What does robots.txt do?
It tells well-behaved crawlers which parts of your site they may visit. It sits at yoursite.com/robots.txt. It is a request, not a lock: bad bots can ignore it, so never rely on it to hide private pages.
Should I block AI crawlers?
It depends on what you want. Blocking training crawlers (like GPTBot, ClaudeBot and Google-Extended) asks them not to use your pages for training. Blocking search and user crawlers (like OAI-SearchBot or PerplexityBot) can stop your site appearing or being cited in those AI answers.
Will blocking Google-Extended hurt my Google rankings?
No. Google says Google-Extended only controls use for Gemini training and grounding; Google Search uses Googlebot, which this setting does not touch.
Where do I put the file?
Upload it to the root of your website so it loads at https://yoursite.com/robots.txt.
Related free tools
llms.txt generator · UTM link builder · Character counter · Favicon generator