What is X-Robots-Tag?
An HTTP header carrying robots directives — the way to noindex PDFs, images, and any non-HTML resource.
Definition
The X-Robots-Tag is an HTTP response header carrying the same directives as the meta robots tag — noindex, nofollow, nosnippet, max-snippet — but applicable to any file type, since it lives in the header rather than the HTML. It's how you control indexing of PDFs, images, feeds, and API responses.
Why it matters
Non-HTML assets are a blind spot in AI-era content governance: whitepapers, internal PDFs, and export files get crawled, indexed, and retrieved into AI answers like any page — sometimes surfacing outdated pricing or draft material. X-Robots-Tag is the control for that surface. Inversely, check that valuable PDFs you want cited aren't accidentally carrying a noindex header from a CDN or plugin default.
Frequently asked
When should I use X-Robots-Tag instead of meta robots?
For anything that isn't an HTML page — PDFs, images, video files — or when you need directives applied at scale by server/CDN configuration rather than per-template.
Do AI crawlers respect X-Robots-Tag?
Bots that honor robots conventions read it; retrieval pipelines built on search indexes inherit its effect, since noindexed resources don't enter the index they query.
Related terms
- AI CitationsThe linked sources an AI assistant credits when generating an answer. Being cited is the AI-era equivalent of ranking #1.
- AI CrawlerBots operated by AI companies — GPTBot, ClaudeBot, PerplexityBot, Google-Extended — that fetch web content for model training or live answer retrieval.
- AI OverviewsGoogle's AI-generated summaries shown above traditional results, now appearing on roughly half of searches and dramatically reducing clicks to websites.
- AI Search VisibilityHow often and how prominently a brand appears in answers generated by AI assistants like ChatGPT, Gemini, Perplexity, and Google AI Overviews.