Imagine an AI agent landing on a news article. Before it can use a word of the text, it has to wade through navigation menus, cookie banners, ad tags and tracking scripts. llms.txt is the proposed shortcut: a plain Markdown file at the root of a domain (yoursite.com/llms.txt) that lists the pages worth reading, each with a short description, so a model can skip the clutter. The idea is sound, but the evidence from 2026 server logs is that very few AI systems actually come looking for the file, so it solves a comprehension problem that most crawlers are not asking about and has nothing to say about payment.
What is llms.txt?
llms.txt is a Markdown file placed at the root of a website that gives large language models a curated, machine-readable summary of the site's most important content. It was proposed by Jeremy Howard of Answer.AI on 3 September 2024 and is documented at llmstxt.org. A valid file needs little more than an H1 with the site name, an optional blockquote summary, and sections of Markdown links to key pages, each with a one-line description.
Some sites also publish a companion file, llms-full.txt, which inlines the full text of key pages so an agent can ingest them in one fetch. The format was designed for a specific pain point: context windows are finite, and converting messy HTML into clean text is lossy and expensive. It remains a community proposal. No standards body has ratified it and no AI company is obliged to honour it.
How is llms.txt different from robots.txt?
robots.txt is an access-control file that tells crawlers which paths they may request. llms.txt is a comprehension file that assumes a model is already reading and suggests what is worth reading. The two are complementary, and publishing one has no effect on the other.
Neither carries any payment mechanism. A publisher can block training crawlers in robots.txt and still publish an llms.txt, or the reverse. For the access-control side, see our guide to what robots.txt can and cannot stop.
Do AI crawlers actually read llms.txt?
Mostly not. The largest server-log study to date, published by Ahrefs on 15 June 2026, examined 137,210 domains with traffic in May 2026. Of the roughly 28% of domains that published an llms.txt file (a figure Ahrefs calls an upper bound because its customer base skews technical), 97% received no requests at all that month. The remaining 3% accounted for about 22,000 requests in total.
Named AI bots made up only 19.5% of those requests. AI agents accounted for 10.5%, training crawlers 5.3%, AI assistants 2.5% and retrieval bots 1.1%. GPTBot was the largest single AI fetcher at around 4.5% of requests. By contrast, SEO audit tools made 21.7% of requests, ahead of any AI category. Ahrefs notes that a fetch does not prove a bot read or used the file, so every consumption figure is a ceiling.
Smaller log studies, collected in a 2026 comparison by AIVisibilityNerd, point the same way. EZY, monitoring 83 sites from April to July 2026, recorded 7 llms.txt fetches from OpenAI crawlers against 3,990 for robots.txt, and 9 from Anthropic against 3,120. Perplexity made none against 775. These are small samples from SEO and AI-visibility vendors, and none is peer-reviewed, but the direction is consistent.
What does Google say about llms.txt?
Google says Google Search does not use it. Google's Search Central documentation on AI features states that you do not need to create new machine-readable files, AI text files or markup to appear in AI Overviews or AI Mode. In June 2026 Google updated its guidance to confine that statement to Google Search and its generative features, and dropped the discouraging tone, saying that using llms.txt or markdown is fine for other services (reported by Search Engine Journal, 17 June 2026). Google's position remains that the file neither helps nor harms visibility in Google Search.
Does llms.txt improve AI citations?
There is no reliable evidence that it does. SE Ranking's analysis of nearly 300,000 domains (November 2025) found no relationship between having an llms.txt file and how often a domain is cited by AI models. Search Atlas, analysing 13,911 domains in June 2026, reported no meaningful visibility advantage either. For what does influence citation, see how AI assistants decide which sources to cite.
Who does use llms.txt?
The clearest consumers are developer tools. Ahrefs found that Claude Code ranked ahead of every retrieval bot, assistant and training crawler apart from GPTBot and one other indexer. Documentation-heavy sites, coding agents and MCP-style retrieval workflows fetch the file for defined tasks, which is where a clean, flattened index earns its keep.
Platforms have also begun generating the file automatically. Shopify and Wix are reported to do so for eligible sites, which is better read as a platform bet than as proof that AI systems consume it. Chrome's Lighthouse has added an llms.txt audit, which explains some of the request volume from tooling.
Should publishers add an llms.txt file?
For most publishers, it is a low-cost, low-return housekeeping task, not a strategy. Adoption is real: a 2026 compilation of studies puts it at roughly 6% to 10% of top-ranked domains, and about 2% of the whole web as measured by the HTTP Archive Web Almanac in July 2025. But adoption is not the same as use.
Reasons to publish one: you run substantial documentation or tools content, you want coding agents to find your canonical pages, or the exercise of choosing your twenty most representative pages is useful in itself. Reasons to hold off: it needs maintenance, a stale file pointing at dead URLs is worse than none, and Ahrefs flagged security concerns, including research crawlers studying prompt injection through the file. Ahrefs' own verdict is that for most sites the cons currently outweigh the pros.
There is also a commercial point. llms.txt optimises for being understood, not valued. Making your content cheaper for an agent to consume is a mixed outcome when that agent sends no click, shows no ad and pays no licence fee.
Where llms.txt stops, monetisation begins
llms.txt addresses comprehension, not compensation. It has no notion of impressions, attribution or payment. A publisher can hand an agent a perfect summary of its best work and still earn nothing when that agent answers a user. This is the layer blankspace works in: at the CDN edge, it detects Live Search Agent and LLM traffic as it arrives and injects contextual brand mentions into the AI responses those agents generate. A voluntary file that crawlers can ignore is a different tool for a different job.
Frequently asked questions
Is llms.txt an official web standard?
No. It is a community proposal published by Answer.AI in September 2024. It has a specification at llmstxt.org but has not been ratified by the IETF, the W3C or any AI platform, and no AI company is obliged to support it.
Will llms.txt help my pages appear in ChatGPT, Perplexity or Google AI answers?
There is no reliable evidence that it will. SE Ranking found no link between having the file and citation frequency across nearly 300,000 domains, and Google says Google Search does not use it. Treat claims of an AI visibility boost with caution and ask to see the data.
Does llms.txt replace robots.txt?
No. robots.txt controls which paths a crawler may request. llms.txt suggests what a model should read. They can coexist, and neither provides payment or licensing.
Who actually fetches llms.txt?
Mostly developer tooling, SEO audit tools and research bots. In Ahrefs' May 2026 data, SEO audit tools made more requests for the file than any AI category, and retrieval bots made about 1.1% of requests.
If llms.txt does not pay publishers, what does?
Monetisation has to come from elsewhere: licensing deals, pay-per-crawl arrangements, or edge-level approaches that detect agent traffic and place paid contextual brand mentions in AI answers. blankspace sits in that last category.