What is User-Agent?
The identity string a bot or browser sends with each request — how you distinguish GPTBot from Googlebot from a fake in your logs.
Definition
A user-agent is the identity string sent with every HTTP request. Bots declare themselves through it — GPTBot, ClaudeBot, PerplexityBot, Googlebot — which is what makes per-bot robots.txt rules and log analysis possible.
Why it matters
User-agents are the observability layer of AI visibility. Your server logs, filtered by AI user-agents, answer questions no analytics tool can: which assistants crawl you, which pages they fetch, how often, and whether your robots.txt intentions match reality. A retrieval bot repeatedly fetching a product page is evidence that page is sourcing live answers. Since scrapers fake user-agents, verify serious decisions against published IP ranges.
Frequently asked
Which AI user-agents should I look for in logs?
At minimum: GPTBot, OAI-SearchBot, ChatGPT-User (OpenAI); ClaudeBot, Claude-User (Anthropic); PerplexityBot, Perplexity-User; Google-Extended and Google-CloudVertexBot; Bytespider; Amazonbot; Meta's crawlers. The roster grows quarterly.
Can I serve different content by user-agent?
Serving bots different content than humans is cloaking — penalized in search and trust-destroying with AI providers. Serve everyone the same rendered truth; vary only what robots rules legitimately control.
Related terms
- Unlinked MentionA brand named without a hyperlink. Nearly worthless in old SEO accounting — now a first-class visibility asset AI systems learn from.
- AI CitationsThe linked sources an AI assistant credits when generating an answer. Being cited is the AI-era equivalent of ranking #1.
- AI CrawlerBots operated by AI companies — GPTBot, ClaudeBot, PerplexityBot, Google-Extended — that fetch web content for model training or live answer retrieval.
- AI OverviewsGoogle's AI-generated summaries shown above traditional results, now appearing on roughly half of searches and dramatically reducing clicks to websites.