Does llms.txt Help AI Visibility? How to Evaluate It
What llms.txt is, how it differs from robots.txt, and how to test whether it helps your readers or tools before investing in it.
What llms.txt is for
The llms.txt project describes a convention for presenting information to language-model tools. Read the current proposal before implementing it; conventions and consuming tools can change.
A documentation team might use a curated file to direct a tool toward a getting-started guide, an API reference, and version-specific limitations. The value comes from the selection and accuracy of those links, not from the filename alone.
How it differs from robots.txt
Robots.txt communicates crawl rules to cooperating crawlers. It is not a password or an access-control mechanism. An llms.txt file is a content aid, not a replacement for those rules.
Google's robots.txt documentation explains the distinction between controlling crawling and removing a page from search. Sensitive material needs authentication. Listing it in either file does not make it private.
| Question | Relevant check |
|---|---|
| Can a crawler request the page? | Robots rules, server and CDN behavior |
| Is the page eligible for indexing? | Status, noindex directives and canonical |
| Is the content readable? | Returned HTML, headings and links |
| Does a specific tool use llms.txt? | Tool documentation and observed requests |
Build a small version with a clear purpose
Start with your most useful public pages. Give each link a description that helps someone choose the right resource. Avoid exporting every URL on the site without prioritization; that recreates the navigation problem the file is meant to solve.
Keep links current, avoid authentication-only destinations, and decide who owns the file when documentation changes. A small accurate index is easier to maintain than an automatically generated list full of obsolete pages.
Test usage in server logs
If you have access to request logs, filter for the file's exact path and a defined date range. Record response status and user agent. Distinguish your own tests, uptime monitors, and ordinary browsers from requests you can reasonably attribute to a tool.
A user-agent string can be imitated. Verify a crawler's identity using its provider's documented method where available. Even a verified request establishes retrieval only; it does not show whether the content influenced an answer.
Test usefulness with a repeatable task
For a tool you control, give it the same documentation question with and without the curated index. Record whether it finds the correct version, cites the right page, and completes the task. Keep the model and other settings constant where possible.
This is a local evaluation of that tool and task. It is not evidence of a ranking change across public AI search services. If you cannot identify a likely consumer, treat the file as a small maintenance experiment rather than a major growth project.
Fix these things before expanding the file
Repair broken public links, inaccessible documentation, missing page titles, and contradictory product information. Make sure your primary content is available in readable HTML and reachable through ordinary navigation.
Google states that "special AI text files are not needed for its AI search features." That is a reason to keep the fundamentals ahead of speculative work.
You can use the Seeqly llms.txt generator to draft a file. Review its output as you would any navigation document, then track whether your intended consumers use it.