Picture a Disallow rule for a well-known AI bot sitting in your robots.txt, and then picture that bot's requests arriving in your logs anyway. That is not a hypothetical. It is the pattern measured across publisher sites in 2026, and it follows from how the file was designed. robots.txt is read by the crawler, interpreted by the crawler and obeyed at the crawler's discretion. The standard that formalised it says as much: RFC 9309, published by the IETF in September 2022, states that its rules "are not a form of access authorization" and that the protocol is "not a substitute for valid content security measures". A polite bot stays out and an impolite one does not, so the file records your intent without enforcing it.
What is robots.txt, and what was it built to do?
robots.txt is a plain text file at the root of a domain that tells automated clients which paths they should not fetch. It was designed in the 1990s for cooperative search crawlers, which had every reason to comply because being welcome on the web was the point of their business. It became an IETF Proposed Standard, RFC 9309, in 2022, but the standardisation described existing behaviour and added no enforcement. Nothing in the protocol lets a server verify who is asking or stop a client that ignores the answer.
Do AI crawlers actually obey robots.txt?
Some do and some do not, and the gap is measurable. TollBit, which sits between AI bots and publisher sites, reported in February 2026 that 30 per cent of AI scrapes in the fourth quarter of 2025 ignored explicit robots.txt permissions, as covered by Media Copilot. Its Q1 and Q2 2026 report, covered by Search Engine Journal in August 2026, looked specifically at sites that had written disallow rules for named AI fetchers. On European sites, it found that around 15 per cent of identified AI page-fetching agents reached URLs that robots.txt told them not to, with ChatGPT-User, Bytespider and Youbot each reported at roughly half. Claude-User and Perplexity-User were reported far lower in Europe, at single figures to low teens, and the regional pattern differs in North America.
Two cautions apply. These figures describe TollBit's own network of publisher sites, not the whole web, and different write-ups of the same report round the ChatGPT-User figure slightly differently (about half, or 54 per cent), so treat the direction of travel as the finding rather than the decimal. And TollBit sells access control to publishers, so it has an interest in the problem being visible. Even so, the pattern is consistent with what the operators themselves say.
Why can a bot ignore robots.txt and still be within its own rules?
Because some AI operators do not treat user-triggered fetches as crawling. OpenAI documents ChatGPT-User as an agent that fetches pages when a person asks ChatGPT about them, and its documentation states that because those requests are initiated by a user, robots.txt rules may not apply. OpenAI runs separate agents for other jobs: GPTBot for training and OAI-SearchBot for search inclusion, each with its own controls. Anthropic, by contrast, documents compliance with robots.txt across its bots.
This matters because the fastest-growing category of AI traffic is exactly this kind: an agent acting for a person who is waiting for an answer. A Disallow line aimed at that agent may be read by the operator as advice about crawling, not a rule about a user's request. Publishers who assume one file governs all AI access are therefore working from a wrong model, and the answer depends on which operator and which bot.
What about bots that hide their identity?
robots.txt works by matching a user-agent name, so it only constrains bots that announce who they are. In August 2025, Cloudflare published tests in which it created new domains with robots.txt files disallowing all automated access, then asked Perplexity about them. It reported that Perplexity's declared crawler was blocked, but that undeclared traffic using a generic Chrome browser string still fetched the content. Cloudflare estimated that stealth traffic at 3 to 6 million requests a day alongside 20 to 25 million daily requests from the declared Perplexity-User agent, and de-listed Perplexity as a verified bot. Perplexity has disputed Cloudflare's characterisation. Whatever the merits of the dispute, the mechanism it exposed is structural: a rule keyed to a name cannot stop a client that does not use its name.
What can robots.txt still do?
Quite a lot, within limits. Most major AI companies publish their crawler names, and the biggest declared crawlers generally respect the file. It is the cheapest way to state your position, and it is where separate rules for training and search can live. Blocking rates show publishers still use it: one 2026 tracker of the top 1,000 sites, aggregated from several sources by Presenc AI, put GPTBot blocking at about a quarter of sites and OAI-SearchBot at around 5 per cent, which suggests most blockers are separating training from search. That secondary aggregation should be treated as indicative, not definitive.
Two further developments are trying to make the declaration layer richer. Cloudflare's Content Signals Policy lets a site state preferences for search, AI input and AI training inside robots.txt. At the IETF, the AI Preferences working group has produced a vocabulary draft, at version 08 as of 14 September 2026, that defines categories such as train-ai, ai-use and search. It is still a working draft with no consensus statement and no RFC number. Both improve how clearly you can say what you want. Neither makes a bot listen.
What actually stops an AI crawler?
Enforcement needs to happen where requests can be verified, which means the server or the CDN edge. Cloudflare's own change is the clearest signal: from 15 September 2026, new domains onboarding to Cloudflare block Training and Agent crawlers by default on pages that display ads, while allowing Search, and existing free customers who have not changed their settings are covered too. That is a network-level block, so it does not depend on the crawler's cooperation.
Verification is the other half. Web Bot Auth, an IETF-track proposal, lets a bot sign its requests cryptographically so a site can confirm who sent them instead of trusting a user-agent header. IP-range checks against operators' published lists do a similar job today. Once identity is verified at the edge, a publisher can choose between three outcomes: block the request, charge for it, or allow it and monetise the retrieval. blankspace operates at this layer, classifying Live Search Agent traffic at the CDN edge, because a policy that can be enforced is the precondition for any of those outcomes being commercial rather than aspirational.
How should publishers set this up?
Treat robots.txt as the declaration layer and keep it accurate. Name the crawlers you care about, separate training from search and retrieval if that reflects your wishes, and check what the ChatGPT-User line actually does given OpenAI's stated position. Then audit what your CDN is already doing on your behalf, since defaults are changing. Finally, decide the enforcement policy at the edge for the traffic that matters, using verified identity rather than declared names. The file says what you would like, and the edge decides what happens.
Frequently asked questions
Does robots.txt legally block AI companies from scraping?
Not by itself. RFC 9309 describes robots.txt as a set of rules that are not a form of access authorization, and it is a technical convention rather than a contract. Its legal weight varies by jurisdiction and is being tested in ongoing disputes, so publishers should treat it as evidence of intent and not as protection.
Does ChatGPT-User follow robots.txt?
OpenAI documents ChatGPT-User as a user-triggered agent and states that robots.txt rules may not apply to it because a person initiates the request. TollBit's 2026 data found it reaching disallowed URLs on roughly half of the European sites that named it. OAI-SearchBot and GPTBot are separate agents with their own controls.
Will blocking AI crawlers in robots.txt remove me from AI answers?
It depends on which crawler you block. Blocking a training crawler such as GPTBot does not necessarily affect search inclusion, which OpenAI handles through OAI-SearchBot. Blocking search or retrieval agents can reduce your visibility in AI answers, so decide the two separately.
Can a bot lie about its user agent?
Yes. A user-agent string is whatever the client says it is, so a bot can present itself as an ordinary browser. Cloudflare reported exactly this in its August 2025 Perplexity investigation, which Perplexity disputed. Cryptographic request signing and IP verification at the edge exist to close that gap.
Should I still keep a robots.txt file for AI bots?
Yes. The major operators publish crawler names and most of the large declared crawlers respect the file, so it remains the cheapest way to state your policy and to separate training from search. Just do not rely on it as your only control.