Definition
Noindex is an instruction, placed in a robots meta tag or an X-Robots-Tag HTTP header, that tells search engines not to index a page. The page can still be crawled and visited, but compliant search engines will drop it from their results. The crawler must fetch the page to see the directive.
For example, Bizbloom might add noindex to thin tag pages and internal search results. A common mistake is also blocking those URLs in robots.txt, which prevents crawlers from fetching the pages, so they never see the noindex and the URLs can remain indexed.
Why it matters
Noindex controls which pages are available to search indexes, and some AI assistants retrieve through those indexes. An accidental noindex on a key page, often left over from a staging site or template change, can remove it from search and reduce its chances of appearing in AI answers. Used deliberately, noindex keeps low-value pages from competing with the pages you want systems to find.
What to do
- Crawl your site and list every page carrying a noindex directive.
- Confirm no important page carries noindex from templates, plugins or staging settings.
- Make sure noindexed pages are not also blocked in robots.txt.
Common questions
What is the difference between noindex and robots.txt disallow?
Disallow in robots.txt asks crawlers not to fetch a URL. Noindex asks search engines not to include a page in their index. A disallowed page can still be indexed if other sites link to it, and a crawler cannot see a noindex on a page it is not allowed to fetch.
Does noindex block AI crawlers?
Noindex is aimed at search indexes, and AI crawlers may treat it differently. It is not designed to stop crawling or training use. To control AI crawler access, use robots.txt rules for each crawler by name, and check each vendor's documentation for the directives it respects.