What is Noindex?
A directive telling search engines not to store a page in their index — which also removes it from the indexes AI retrieval depends on.
Definition
Noindex is a directive (meta robots tag or X-Robots-Tag header) instructing search engines to exclude a page from their index. The page can still be crawled and linked; it just won't appear in results.
In the AI era, the blast radius is bigger than intended: retrieval-based assistants query search indexes, so a noindexed page is also invisible to most AI answer pipelines.
Why it matters
Accidental noindex is a classic silent killer — a staging flag shipped to production, a plugin default, a template inheritance. Classic SEO audits catch it eventually via ranking drops; in AI search there's no ranking report to alarm you, just absence from answers. Audit noindex deliberately: it should mark genuinely private or thin pages, never anything you'd want an assistant to cite.
Frequently asked
Does noindex block AI training crawlers?
Not by itself — noindex governs indexing, not crawling. To keep content out of AI training, use robots.txt rules for specific bots (GPTBot, Google-Extended). To keep it out of AI answers, noindex mostly works because retrieval leans on search indexes.
What's the difference between noindex and robots.txt disallow?
Disallow blocks crawling but the URL can still be indexed from links; noindex allows crawling but forbids indexing. For true removal, allow the crawl and serve noindex.
Related terms
- Natural Language Processing (NLP)The field of computing that parses and generates human language — the machinery deciding what your content 'means' to search and AI systems.
- AI CitationsThe linked sources an AI assistant credits when generating an answer. Being cited is the AI-era equivalent of ranking #1.
- AI CrawlerBots operated by AI companies — GPTBot, ClaudeBot, PerplexityBot, Google-Extended — that fetch web content for model training or live answer retrieval.
- AI OverviewsGoogle's AI-generated summaries shown above traditional results, now appearing on roughly half of searches and dramatically reducing clicks to websites.