A robots.txt file has always answered one question: may this bot fetch this path? The Content Signals Policy adds a second: once the bot has the page, what may it do with it? A publisher can mark three uses - search indexing, feeding a live AI answer, and training a model - as yes or no in a single line. Cloudflare published the policy on 24 September 2025 under a CC0 licence and added it to the managed robots.txt files of its customers, so many sites carry content signals without anyone having chosen them. Since then the picture has become clearer and less flattering: Google's John Mueller said in July 2026 that the directive has no effect on any crawler he knows of, Cloudflare has moved its own enforcement towards bot categories, and the IETF is working on a rival standard.
How does the Content Signals Policy work?
The policy is a set of machine-readable directives inside robots.txt that describe how content may be used after it has been fetched. Ordinary robots.txt governs access to a path. Content signals govern purpose. The standard is published at contentsignals.org and is free for any site, CDN or vendor to implement, which is why it is described as a standard and not a Cloudflare product.
A publisher writes one comma-delimited line, for example "Content-Signal: search=yes, ai-train=no". Each signal takes a value of yes or no, and leaving a signal out means no preference has been stated, not a yes or a no. Cloudflare is explicit that the signals are preferences and not enforceable rules.
What are the three content signals?
There are three. The search signal covers building a traditional search index that links back to the source. The ai-input signal covers feeding content into an AI system to produce a real-time answer, the retrieval step behind AI Overviews and chatbot responses. The ai-train signal covers training or fine-tuning a model.
Separating ai-input from ai-train is the useful idea. A publisher can accept being quoted in a live answer, where a citation and a click are at least possible, while refusing to be absorbed into a model's weights.
What did Cloudflare set as the default?
For sites using Cloudflare's managed robots.txt, the default is search=yes and ai-train=no, with ai-input left unset. Cloudflare said at launch that it did not want to guess customers' preferences on live AI answers. That omission matters, because ai-input is the signal that decides whether your page can be read to answer a question, and it is the one most publishers have never consciously set.
Cloudflare also said at launch that Google had been told in advance but had not committed to honouring the signals. That has not changed in any way a publisher can rely on.
What has changed since launch?
Three developments matter more than the original announcement.
The first is Google's response. On 6 July 2026, Google's John Mueller wrote in a Reddit thread that, as far as he knows, the Content-Signal directive has no effect on any crawler or LLM, and that it adds bloat and maintenance to a robots.txt file. Search Engine Roundtable reported the comment the same day. It is one Google employee's view in a forum thread and not a Google policy statement, but it is the clearest public signal yet that the largest search operator is not treating the directive as binding.
The second is Cloudflare's own shift. On 1 July 2026 it launched new AI traffic options for all customers, including the free tier, built around three bot categories: Search, Agent and Training. It also began testing a use field that extends content signals with three values: immediate, reference (the default) and full, ranging from interacting with content without storing it to summarising and reproducing it. Cloudflare describes these as stated preferences, not blocks. What it does enforce is bot status: bots that abuse the signals can lose Verified status, and Verified no longer means allowed by default.
The third is the standards track. The IETF's AI preferences work has produced a draft, draft-ietf-aipref-attach-05, dated 19 August 2026 and authored by Gary Illyes of Google and Martin Thomson of Mozilla. It defines a Content-Usage field that can be sent as an HTTP header or written as a robots.txt rule, and it relies on a separate vocabulary draft. It is still an Internet-Draft, not an RFC, so it is work in progress. Content Signals and Content-Usage address the same problem with different syntax, and publishers should expect to carry both until one settles.
Does the Content Signals Policy have legal weight?
Cloudflare's managed file includes a notice that any restriction expressed through content signals is an express reservation of rights under Article 4 of the EU's 2019 Copyright Directive (Directive 2019/790). The aim is to turn a polite request into a documented declaration, so that a crawler training on content marked ai-train=no cannot claim it had no notice.
This is untested. Courts and regulators may decide that a robots.txt line is not an effective machine-readable reservation, or that it binds no one. Combined with site terms of service it can strengthen a later claim, but it is a notice mechanism and not a guarantee. Publishers with a real dispute should take legal advice and not rely on the file.
Do AI companies actually obey robots.txt signals?
Compliance is uneven, and the fetchers that matter most for live answers are the least predictable. TollBit's State of the Bots report for the first half of 2026, as reported by Search Engine Journal on 14 August 2026, found that roughly 15% of identified AI page-fetching agents reached URLs that European sites had disallowed in robots.txt. These are user-triggered fetchers that load a page when a person asks a question, and a few named agents, including ChatGPT-User, Bytespider and Youbot, accounted for most of the bypasses.
Two caveats apply. TollBit counts any request to a disallowed URL as a bypass, and OpenAI's own documentation says robots.txt may not apply to user-initiated requests, so some of that behaviour reflects stated policy and not concealment. The figure comes through a secondary report and not the full study. If classic disallow rules are unevenly obeyed by live fetchers, a finer-grained preference line is unlikely to be obeyed more reliably.
Should publishers use it?
For most publishers, yes, with modest expectations. Adding signals costs nothing, records a position in a form that reputable operators can read, and helps build a documented reservation of rights. Three practical points follow.
Decide ai-input deliberately. Setting it to no may reduce your presence in AI answers, and setting it to yes permits the read without any payment, so neither is a neutral default.
Check what you already publish. If you use Cloudflare's managed robots.txt, you may be asserting preferences you never chose.
Do not rely on the file for protection. Mueller's comment and the fetcher data both point the same way: a signal is a request, and a bot that ignores robots.txt or presents as a browser is unaffected by it.
Where enforcement and monetisation actually happen
A content signal is a request made in a file. What happens to an AI request is decided at the network edge, where it arrives. There, a crawler can be verified against its operator's published IP ranges and signatures rather than trusted on its user-agent string, and a publisher can apply a real decision: block it, charge for access, or allow the read and monetise it. For the September 2026 changes to Cloudflare's own defaults, see our guide to Cloudflare's AI crawler defaults.
blankspace works at this layer. It detects and verifies Live Search Agent traffic at the CDN edge and places contextual brand mentions in the AI response, so an allowed read can earn revenue and not only consume bandwidth. Content signals and edge enforcement are complementary: the signal states the rule, and the edge is where it is applied and where permitted traffic can be paid for.
Frequently asked questions
Is the Content Signals Policy the same as robots.txt?
No. It is an extension that lives inside robots.txt. Classic robots.txt says whether a crawler may access a path. Content signals add a statement about how the content may be used once accessed, for search, for AI answers or for AI training, using a Content-Signal line with yes or no values.
Does Google honour Cloudflare's content signals?
Google has not committed to doing so. Cloudflare said at launch that Google had advance notice but had not said whether it would respect the signals, and on 6 July 2026 John Mueller said he knew of no crawler or LLM that acts on the directive. Treat it as unsupported by Google unless it announces otherwise.
What is the difference between ai-input and ai-train?
The ai-input signal covers using content to generate a real-time AI answer, the live retrieval behind AI Overviews and chatbot responses. The ai-train signal covers using content to train or fine-tune a model. They are separate so a publisher can allow quoting in live answers while refusing use as training data, or the reverse.
How is the Content Signals Policy different from the IETF Content-Usage proposal?
Both let a publisher state AI usage preferences. Content Signals is a Cloudflare-led standard already deployed in many managed robots.txt files. Content-Usage is an IETF Internet-Draft, still work in progress, that can be sent as an HTTP header or a robots.txt rule and is co-authored by engineers from Google and Mozilla. Neither is enforcement.
Do content signals help publishers make money from AI traffic?
No. Content signals govern permission and cannot charge for the access they grant. Earning revenue from AI reads requires acting on the request at the edge, through access charges or by placing contextual advertising inside the AI response, neither of which a robots.txt directive can do.