Crawlers & access

What is llms.txt?

A proposed Markdown file at your domain root that points AI tools to your most useful pages.

Definition

llms.txt is a proposed convention for a plain Markdown file placed at a site's root, such as example.com/llms.txt. It gives AI tools a short summary of the site and a curated list of links to important pages, sometimes with brief descriptions. It is a community proposal, not an adopted web standard.

For example, Bizbloom could publish an llms.txt file listing its product overview, pricing, documentation and API reference. There is no confirmed adoption of the file by major AI assistants, so it may help some developer tools but should not be expected to change answers in ChatGPT or Gemini.

Why it matters

llms.txt is often discussed as a way to improve AI visibility, so it helps to be precise about what it does. Support varies, it is not a ranking signal, and it does not replace robots.txt, indexing rules or crawlable pages. It is low effort to create and may be useful for tools that read it. Treat it as an optional extra, not a fix for pages assistants cannot find.

What to do

  • Fix crawl access and content gaps before spending time on llms.txt.
  • If you create one, list your most useful pages with short, accurate descriptions.
  • Check server logs to see whether any crawlers actually request the file.

Common questions

Does llms.txt improve AI search visibility?

There is no confirmed evidence that major AI assistants use llms.txt when choosing sources or writing answers. Some developer tools and smaller systems may read it. It does not hurt to publish one, but it should not be relied on to change how assistants describe or cite your brand.

Is llms.txt the same as robots.txt?

No. robots.txt tells crawlers which paths they may request and is widely respected by well-behaved bots. llms.txt is a proposed file that lists recommended pages for AI tools. It does not grant or block access, and it does not replace any robots.txt or noindex rules.

See this on your own brand.

Track the questions your buyers ask across ChatGPT, Google AI Overviews, Gemini, Claude, Perplexity and Grok. Start with Basic free for 7 days.