Definition
The X-Robots-Tag is an HTTP header a server sends with a response to give search engines indexing instructions. It supports the same directives as the robots meta tag, such as noindex, nofollow and nosnippet. Because it is sent in the header, it works for files that are not HTML pages, like PDFs and images.
For example, Bizbloom might host old pricing sheets as PDFs that it no longer wants in search results. It cannot add a meta tag to a PDF, so it configures the server to send X-Robots-Tag: noindex with those files instead.
Why it matters
The X-Robots-Tag is the practical way to apply indexing rules to PDFs, images and other files, which can otherwise stay in search indexes and be retrieved long after they are outdated. Because it is set at the server level, it can also be applied accidentally across a whole directory or site. Checking response headers for key pages helps catch rules that quietly remove content from search.
What to do
- Check response headers of key pages and files for any X-Robots-Tag directives.
- Use X-Robots-Tag noindex for outdated PDFs or files you do not want in search.
- Make sure affected files are not blocked in robots.txt, so crawlers can see the header.
Common questions
When should I use X-Robots-Tag instead of a meta robots tag?
Use X-Robots-Tag for non-HTML files, such as PDFs, images and videos, where you cannot add a meta tag. It is also useful for applying rules across many URLs through server configuration. For standard HTML pages, either method works, but avoid conflicting instructions between them.
How do I check if a page has an X-Robots-Tag?
Inspect the HTTP response headers using your browser's developer tools, a command-line request or an SEO crawler. Google Search Console's URL Inspection tool also reports whether indexing is blocked. Look for any header named X-Robots-Tag and note the directives it contains.