Definition
robots.txt is a plain text file at the root of a website, such as example.com/robots.txt, that gives crawlers instructions using the Robots Exclusion Protocol. Rules are grouped by user-agent and use Allow and Disallow lines to specify paths. Reputable crawlers follow these rules, but the file cannot force compliance.
For example, Bizbloom might disallow GPTBot, allow OAI-SearchBot and Googlebot, and block all crawlers from its /admin path. A single broad rule like “User-agent: * Disallow: /” left over from a staging site would block every compliant crawler from the whole site.
Why it matters
robots.txt is where most decisions about AI crawler access are made, and small errors can have large effects. A wildcard rule meant for one bot can block the search crawlers that help your pages appear in AI answers. It controls crawling, not access, and is not a security tool: it does not remove pages that are already indexed or stop bots that ignore it. Review it whenever your crawler policy changes.
What to do
- Open your robots.txt and list every rule that affects AI crawlers by name or wildcard.
- Check that the search crawlers you want to allow are not blocked by broader rules.
- Test changes before deploying, and recheck the file after site migrations or platform updates.
Common questions
Does robots.txt stop AI companies from using my content?
It asks compliant crawlers not to fetch specified paths, and major AI companies say their documented crawlers respect it. It cannot stop crawlers that ignore it, and it does not affect content already collected. It also does not control copies of your content published on other sites.
Can robots.txt remove a page from Google?
No. robots.txt prevents crawling, not indexing. A blocked page can still appear in results if other sites link to it, usually without a description. To remove a page from the index, allow crawling and use a noindex directive, or use Search Console's removal tool for urgent cases.