Definition
An AI crawler is a bot operated by an AI company that requests pages from websites. Each crawler identifies itself with a user-agent name and usually has a stated purpose: collecting training data, building a search index, or fetching a page because a user asked the assistant to read it.
OpenAI, for example, documents GPTBot for training and OAI-SearchBot for search, and other companies publish their own crawler names. If Bizbloom blocks every bot in robots.txt to stop training use, it may also stop the search crawlers that help its pages appear in AI answers.
Why it matters
AI crawlers are the route by which your pages reach many AI systems. If the crawler that powers an assistant's search feature cannot fetch your pages, those pages are less likely to be retrieved or cited when buyers ask questions. Because each crawler has a different job, a single blanket rule can have effects you did not intend. Reviewing crawlers by name lets you make separate decisions about training, search and on-demand fetching.
What to do
- List the AI crawlers each vendor documents and note the stated purpose of each.
- Check robots.txt, CDN and firewall rules for each crawler by name, not just wildcard rules.
- Review server logs to confirm which AI crawlers actually request your pages.
Common questions
Does blocking one AI crawler block all AI assistants?
No. Each crawler is controlled separately by its user-agent name. Blocking a training crawler does not necessarily block a search crawler from the same company, and it has no effect on crawlers from other companies. Review each one by name and decide based on its documented purpose.
Where can I find the list of AI crawlers to allow or block?
Most AI companies publish their crawler names, purposes and, in some cases, IP ranges in their developer or help documentation. These lists change, so check the vendors' own pages periodically rather than relying on a static list, and compare them with the bots that appear in your server logs.