For three years, publishers who wanted to keep their content out of AI training had to chase each company separately: GPTBot for OpenAI, Google-Extended for Google, ClaudeBot for Anthropic, and so on down a list that kept getting longer. That is about to change. A single shared signal, proposed at the IETF on 30 October 2026, now has public backing from four of the largest AI developers.
It is useful. It is also narrower than it looks. The token governs training. It does not govern the real-time reads that AI assistants make when a user asks a question, and that is where the commercial value of publisher content is moving.
What was announced?
On 30 October 2026, Kevin Kelley, a standards lead at Anthropic, submitted an individual IETF draft (draft-kelley-aipref-training-preference-signal) describing a shared AI training preference expressed in robots.txt (reported, Digiday). The same day, two of the largest platforms published their plans.
Microsoft Bing set out the syntax in a post by principal product manager Krishna Madhavan. A site opts out of training with a pseudo user agent:
User-Agent: __usage_ai-training
Disallow: /
Paths can be targeted the same way, and Bing will also accept page-level meta tags and X-Robots-Tag headers (ai-training:disallow), with page-level settings overriding robots.txt.
Google's John Mueller wrote that Google plans to support a shared training control based on the IETF's AI Preferences (AIPREF) work, with support starting in 2027. If a site opts out through the shared token and does not separately specify Google-Extended, Google says it will treat that as a Google-Extended disallow. Provider-specific controls keep working alongside it.
Digiday reports that Anthropic is rolling the signal out now, OpenAI says it will honour it "in the coming months", and Meta has expressed support for a common signal without committing to honour this one (reported, Digiday).
How does this fit the IETF's wider AIPREF work?
The AIPREF working group, chaired by Mark Nottingham and Suresh Krishnan, is building a standard vocabulary for AI usage preferences. Its current vocabulary draft (draft-ietf-aipref-vocab-08, last updated 13 September 2026) defines three categories: train-ai (modifying a model's learned parameters), ai-use (using content as input to a generative model) and search (selecting content and directing users to it, explicitly excluding generated summaries).
The draft carries a note that it does not yet reflect working group consensus, and both core documents were targeted at August 2026 for submission without, as of this week, showing that step on the IETF datatracker. The new training token is effectively the platforms moving first on the one category everyone could agree on.
What does the token not cover?
Two things, and both matter more than the token itself.
It does not enforce anything. AIPREF's charter puts enforcement, crawler authentication and auditing out of scope, and the vocabulary draft says plainly that it does not ensure preferences are followed. Digiday cites ScalePost data showing the four companies' crawlers made roughly 1 billion requests to more than 50 publisher properties in 30 days, of which only 0.15% were requests for robots.txt (reported, ScalePost via Digiday).
It does not cover grounding. Training and grounding are different acts. Bing defines training as changing a model's parameters and grounding as using content as input to a generated answer. Live Search Agents, dispatched in real time when someone asks ChatGPT, Perplexity, Claude or Copilot a question, are a grounding activity. The shared token is silent on them.
Bing is the only one of the four to offer a separate grounding control, and it is deliberately limited: page-level only, no robots.txt option. Publishers can choose Allow, DisallowExtended (Microsoft surfaces only, excluding third-party grounding APIs) or Disallow. Bing states that restricting grounding "will significantly reduce" visibility in its AI experiences, and that neither choice guarantees "inclusion, traffic, or payment" (named-party figure, Microsoft Bing).
What does it mean for publishers?
Set the training preference. It costs nothing, it consolidates a messy patchwork, and it creates a clear, machine-readable record of intent that may matter in licensing talks and litigation.
Then be clear about what it does not solve. The token is a training crawler policy. The traffic growing fastest on publisher sites is not training crawlers but agents reading pages on behalf of a live user; Cloudflare reported personal AI agent traffic grew 15x by 2025 (reported, Cloudflare). Those reads do not execute JavaScript, so no analytics fire, no ad tags load and no auction runs.
Bing's grounding options also show the bind publishers face. Block grounding and you vanish from the answer. Allow it and you are read for free. That is the same choice blocking has always offered: give up visibility or give up value. Microsoft's dates are worth diarising regardless: new preferences take effect from 1 February 2027, and from 1 March 2027 Bing stops reading NoArchive and NoCache as training or grounding signals.
What does it mean for advertisers?
Indirectly, quite a lot. Bing's warning is a tell: platforms are now openly linking grounding access to AI visibility. As more publishers make deliberate grounding choices, the pool of sources that AI answers draw on will shift, and so will which brands get mentioned in them.
For brands, the practical point is that visibility in LLM answers depends on what those systems are allowed to read in the moment, not only on what they were trained on. Paid placement inside the live read, contextual and measurable, is a different lever from training-era GEO.
The blankspace read
The industry has spent three years arguing about training, and has now largely settled the plumbing for saying no to it. Good. But training is a one-time copy. Grounding is a recurring read, every query, every day, and that is where the audience now sits. A standard that lets publishers refuse the copy while staying silent on the read solves yesterday's problem. Opt-outs give publishers a way to say no. They still need a way to say yes, on terms, at the moment of the read. Monetise the read, not the referral.
Key definitions
- AI training: using content to modify a generative model's learned parameters (AIPREF
train-ai). - Grounding: using content as input to inform a generated response at answer time (Bing definition).
- Live Search Agent: an AI agent dispatched in real time to fetch pages in response to a user's question.
- AIPREF: the IETF working group standardising a vocabulary and attachment methods for AI usage preferences.
Key figures
| Figure | Value | Provenance |
|---|---|---|
| AI companies backing the shared training signal | 4 (Anthropic, Google, Microsoft, OpenAI) | Reported (Digiday) |
| IETF draft submitted | 30 October 2026 | Reported (Digiday); draft linked by Microsoft Bing |
| Google support start | 2027 | Named-party figure (Google) |
| Bing new preferences take effect | 1 February 2027 | Named-party figure (Microsoft Bing) |
| Bing stops reading NoArchive/NoCache as AI signals | 1 March 2027 | Named-party figure (Microsoft Bing) |
| Crawler requests by the four companies, 30 days, 50+ publisher properties | ~1 billion | Reported (ScalePost via Digiday) |
| Share of those requests that fetched robots.txt | 0.15% | Reported (ScalePost via Digiday) |
| Personal AI agent traffic growth by 2025 | 15x | Reported (Cloudflare) |
| AIPREF vocabulary draft version | -08, updated 13 September 2026 | Reported (IETF datatracker) |
Primary source, opened and verified
Microsoft Bing, "Your content, your choice: clearer preferences for training and grounding" (30 October 2026): https://blogs.bing.com/search/2026/10/your-content-your-choice-clearer-preferences-for-training-and-grounding/
Also opened: Google Search Central, "Supporting a shared AI training control for websites" (30 October 2026): https://developers.google.com/search/blog/2026/10/ai-training-control ; IETF AIPREF working group documents: https://datatracker.ietf.org/wg/aipref/documents/