Definition
A user-agent is a text string included in the headers of every HTTP request that identifies the software making the request. Browsers send user-agents describing the browser and operating system, and crawlers send their own names, such as Googlebot, GPTBot, OAI-SearchBot or PerplexityBot. robots.txt rules are written against these names.
For example, a line in Bizbloom's server log might show a request with a user-agent containing “GPTBot” and a link to OpenAI's documentation. That suggests a visit from OpenAI's training crawler, but anyone can send that string, so the request should be verified.
Why it matters
User-agents are how you tell AI crawlers apart in logs and how you address them in robots.txt. Getting the names right matters: a typo or outdated name means a rule does nothing. Because anyone can fake a user-agent, scrapers sometimes pose as well-known bots. Verifying important crawlers against the operator's published IP ranges, where available, keeps your analysis and firewall rules accurate.
What to do
- Copy exact crawler names from each vendor's documentation into robots.txt rules.
- Group server log requests by user-agent to see which bots visit and how often.
- Verify requests claiming to be important bots against published IP ranges or reverse DNS.
Common questions
How do I verify that a request is really from GPTBot or Googlebot?
Check the request's IP address against the IP ranges the operator publishes. Google also supports verification through a reverse DNS lookup. OpenAI publishes IP ranges for its crawlers. If the user-agent claims to be a known bot but the IP does not match, treat the request as unverified.
Where can I see user-agents for my website?
Server access logs record the user-agent for every request, as do most CDN logs. Log analysis tools can group requests by user-agent so you can see which crawlers visit. Tag-based analytics tools usually do not show crawler visits, since most bots do not run their scripts.