AI robots.txt builder
Pick which AI crawlers can read your site. Search and retrieval bots help you get cited. Training bots are a separate choice. Copy the rules when you are done.
robots.txt is a request, not a lock. Well-behaved crawlers follow it, but it does not stop a bot that ignores it, and some providers treat user-requested visits differently. To truly restrict access, use authentication or firewall rules.
Add these lines to the robots.txt at the root of your domain. A crawler follows the most specific group that names it, so a named bot here will ignore your User-agent: * rules. Put any site-wide paths you want it to respect in the private paths box.
How to use this
- Start with Recommended. It allows the bots that fetch pages for AI answers and search, and blocks one crawler with a weaker track record.
- Decide on training bots. Allowing them may help models learn about your brand over time. Blocking them keeps your content out of future training sets that honor the rule.
- Add any private paths, such as checkout or account pages, that allowed bots should skip.
- Copy the output and add it to /robots.txt on your live domain. Keep your existing rules for other crawlers.
What to do next
- Open yourdomain.com/robots.txt in a browser to confirm the new rules are live.
- Check your server or CDN logs over the next few weeks to see which AI bots visit and which pages they request.
- Make sure a firewall or bot-protection setting is not blocking the bots you just allowed. That is a common cause of “we allowed it but nothing changed”.
- Run the AI crawlability checker on your key pages.
Common questions
Will blocking GPTBot remove me from ChatGPT answers?
Not by itself. OpenAI uses separate agents for training (GPTBot), search indexing (OAI-SearchBot) and user-requested visits (ChatGPT-User). Blocking GPTBot is about training. To appear in ChatGPT search results, OAI-SearchBot needs access.
Does Google-Extended affect AI Overviews?
Google says Google-Extended does not affect inclusion or ranking in Google Search. AI Overviews are part of Search and use content crawled by Googlebot. Google-Extended controls use of your content for Gemini models.
Do all AI bots respect robots.txt?
Major providers say their crawlers follow it, but there is no enforcement, and some bots have been reported to ignore it. User-requested fetches can also be handled under different rules. For real access control, use authentication or bot rules at your CDN or firewall.
More free tools
Track this every day.
Seeqly asks your buyer questions on every AI assistant daily, with citations and share of voice. ChatGPT, Google AI Overviews, Gemini, Claude, Perplexity and Grok. Try Basic free for 7 days.