The llms.txt Reality Check: 10% of Sites Have One. 97% Are Never Fetched.

We built a free llms.txt generator, and we're about to tell you the file it produces is probably not being read by anyone. That's not a contradiction — it's what taking data seriously looks like in a field full of cargo cults. Here's what the measurements actually say about llms.txt, why we still think you should spend the two minutes, and where those hours of "AI optimization" effort should really go.

1. The adoption-consumption gap, quantified

Two datasets tell the whole story. On the supply side, llms.txt adoption climbed roughly 8.8× to reach about one in ten websites — 8.7% of the top 1,000 (Rankability, June 2026), 10.13% across SE Ranking's 300,000-domain crawl, remarkably consistent across site sizes. Thousands of teams did the work.

On the demand side, Ahrefs pointed log analysis at the question every guide skipped: does anything fetch these files? Their answer, from May 2026 data: 97% of llms.txt files received zero requests. Of the requests that did occur, GPTBot accounted for 4.51%, ClaudeBot 0.80%, DeepseekBot 0.02%. The AI search crawlers that matter — GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, Google-Extended — overwhelmingly skip the file and crawl your HTML directly.

And the platform silence is itself data: no major LLM provider has publicly committed to llms.txt as a signal. Google's John Mueller compared it to the meta keywords tag. When the companies that would have to read the file decline to say they read it, believe them.

2. Why the gap exists (it's not a conspiracy)

The proposal (Jeremy Howard, 2024) is elegant: a curated markdown map at your domain root so context-limited models skip the crawling mess. But it solves a problem the big providers solved differently. OpenAI, Google, and Anthropic already operate web-scale crawlers and indexes; their retrieval pipelines rank passages from full HTML, and they've spent years hardening ranking systems against self-declared metadata — which is exactly what llms.txt is. A file where every site describes its own importance is the meta keywords tag with better formatting: useful in a cooperative world, unrankable in an adversarial one.

The place llms.txt does work confirms the pattern: coding agents. When Cursor or Claude Code points at your docs site, it's not ranking the open web — it's navigating one trusted site with a small context window. A curated map is exactly what it needs, and those tools fetch /llms.txt and /llms-full.txt today. Cooperative context, real consumer.

3. The honest case for doing it anyway

So why did we ship a generator? Because the expected-value math still lands slightly positive — if you hold it honestly:

• Cost is ~zero. Two minutes with our generator, no risk, no maintenance beyond occasional updates.
• Developer-tool audiences read it today. If you publish documentation and your buyers use coding agents, this is a real (small) channel now.
• It's an option on adoption. If any provider flips it on, publishers with clean files are first in line. Cheap insurance is fine as long as you know it's insurance.
• The exercise has side effects. Writing one page that says what your site is and which pages matter is clarifying — most teams discover they can't do it, which is its own finding.

What we won't do is what a chunk of our industry is doing: selling llms.txt as an AI-visibility silver bullet. If someone's pitch leads with it, ask them for fetch logs. That request settles a lot of pitches.

4. Where the real leverage is: the checklist bots actually hit

Server logs show what AI crawlers really consume, and it's boring: your robots.txt, your raw HTML, your load times. That's the surface to optimize, in this order:

1. Unblock the retrieval bots. Audit robots.txt and CDN bot settings for OAI-SearchBot, PerplexityBot, and ChatGPT-User. Blanket "block AI" rules from 2023–24 are still silently deleting brands from AI answers — the most common, most fixable failure we see.
2. Serve content in raw HTML. Most AI crawlers don't execute JavaScript. If your pages are client-rendered shells, you're invisible regardless of every other choice. This is the JavaScript rendering problem, and SSR/prerendering is the fix.
3. Structure for extraction. One clear H1, real H2 sections, answer-first paragraphs, lists and tables for anything comparative — the shapes RAG pipelines lift whole.
4. Keep index hygiene. No stray noindex, clean canonicals, fast responses. Retrieval sits on search indexes; index problems are visibility problems.
5. Add entity schema. Organization + sameAs, Article with authors — hygiene that helps machines resolve who you are (with honestly mixed evidence on direct citation lift, per the schema data).

Every item on that list is checked automatically by our free AI crawlability checker — paste your page source, get a 0–100 score across 14 checks. Unlike llms.txt, this is the path AI bots verifiably walk every day.

5. How to test any GEO tactic before believing it

llms.txt is a case study in a broader discipline this young field badly needs. Before adopting any "AI visibility" tactic, run it through three questions:

Is there consumption evidence? Not adoption evidence — consumption. Ten percent of the web adopting llms.txt proved nothing; one log study measuring fetches settled the question. Server logs, provider documentation, and controlled tests outrank every listicle.

Does the mechanism survive adversarial thinking? If a tactic only works when every site is honest, providers won't trust it. Corroboration-based signals (reviews, third-party mentions, cited data) survive; self-declaration signals mostly don't.

Can you measure the outcome? The only endpoint that matters is whether assistants mention and cite you more for the prompts your buyers ask. That's measurable — sample the prompts on a schedule, track mention and citation rates, and let your own data promote or kill each tactic. That measurement loop is precisely what Seeqly exists to run — and it's how we caught ourselves before overselling the very file this article is about.

Sources

• PPC Land — llms.txt adoption rises 8.8× but 97% of files get zero AI requests (Ahrefs data)
• Rankability — llms.txt adoption: 8.7% of the top 1,000 (June 2026)
• SE Ranking — llms.txt: why brands rely on it and why it doesn't work (300k-domain study)
• aeo.press — The state of llms.txt in 2026
• Limy — llms.txt in 2026: the full guide (agent/IDE usage)