llms.txt

llms.txt is a proposed plain-text file, written in Markdown and placed at a site’s root, that gives language models a curated index of a site’s most useful pages. Jeremy Howard of Answer.AI proposed it on September 3, 2024; it is a community proposal, not an official web standard.

llms.txt in Practice

The evidence on whether crawlers read it is thin. One publisher that had run an llms.txt since May 2026 examined four days of IP-verified server logs: AI crawlers requested its robots.txt 120 times and its XML sitemaps 43 times, and llms.txt zero times ([Prosopo](https://prosopo.io/blog/llms-txt/)). Google’s Gary Illyes said in July 2025 that Google doesn’t support llms.txt and isn’t planning to ([Art Digital Media](https://artdigitalmedia.co.uk/blog/llms-txt-explained)), and John Mueller likened it to the old keywords meta tag — a site owner’s own claim about what the site is about ([Codersera](https://codersera.com/blog/llms-txt-complete-guide-2026/)). The one bright spot: documentation-reading coding agents, with Claude Code fetching it more than any AI retrieval bot in one dataset ([Marqeable](https://www.marqeable.com/blog/does-llms-txt-work/)). Adoption estimates vary widely; one survey of 300,000 domains found about 10% publishing one ([Link Building HQ](https://www.linkbuildinghq.com/blog/should-websites-implement-llms-txt-in-2026/)). A SaaS company with extensive API docs, like Ledgerly, is the case where it’s most plausible to help, since coding agents read docs.


How to Check Whether llms.txt Gets Read

Check your server logs for requests to /llms.txt from verified crawler IP ranges — the approach the Prosopo study used. A file with no requests is a clear signal that, for now, it’s not changing anything about how those crawlers read your site.


Why llms.txt Matters

It costs minutes and carries no known penalty, so adding one is harmless — but it’s low priority next to robots.txt access, clean HTML, and structured data, which crawlers demonstrably do use. Treat it as optional groundwork for agents, not an AI-visibility lever.


Terms Related to llms.txt

robots.txt for AI Crawlers ·
Context Window ·
Chunking (Content)
LLM Visibility


FAQ

Should I add an llms.txt file?
If it takes minutes, there’s little downside, but the best available evidence shows major AI crawlers largely ignore it for search and citation. Fix robots.txt access first.