What is robots.txt?
The file governing which bots may crawl what — now the control panel deciding your participation in AI training and AI answers.
Definition
robots.txt is the plain-text file at your domain root telling crawlers which paths they may fetch. Once a routine SEO artifact, it became strategic in the AI era: it's where you allow or block GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, and Google-Extended individually — separate decisions about training inclusion and answer-time retrieval.
Why it matters
Misconfigured robots.txt is the most common self-inflicted AI visibility wound. CDN 'block AI bots' toggles and 2023-era blanket rules silently exclude brands from ChatGPT and Perplexity answers to this day. The 2026 mainstream position for brands seeking visibility: allow retrieval/search bots (OAI-SearchBot, PerplexityBot, ChatGPT-User), decide policy-by-policy on training bots (GPTBot, Google-Extended, ClaudeBot), and re-audit quarterly because new bots keep appearing.
Frequently asked
Does robots.txt legally stop scraping?
No — it's a convention, not an enforcement mechanism. Major AI companies publicly honor it; bad actors ignore it. For enforcement, use bot management at the CDN/WAF layer.
What robots.txt setup maximizes AI visibility?
Explicitly allow the search/retrieval bots, keep sitemaps declared, and make sure no wildcard rule accidentally covers AI user-agents. Then verify with server logs that the bots actually crawl.
Related terms
- Retrieval-Augmented Generation (RAG)The architecture where an LLM retrieves relevant documents first, then generates its answer from them — the engine of citable AI search.
- Rich ResultsEnhanced search listings powered by structured data — stars, FAQs, products — and a proving ground for machine-readable content.
- AI CitationsThe linked sources an AI assistant credits when generating an answer. Being cited is the AI-era equivalent of ranking #1.
- AI CrawlerBots operated by AI companies — GPTBot, ClaudeBot, PerplexityBot, Google-Extended — that fetch web content for model training or live answer retrieval.