Picture a request arriving at your server labelled Google-Agent. Under the old model you check the IP range, run a reverse DNS lookup, and hope the answer is conclusive. Under Web Bot Auth, the request carries a digital signature made with a private key that only the operator holds, and your server (or your CDN) checks it against a public key the operator publishes at a known address. If the signature verifies, the identity is proven. If it does not, the label is just text. That shift, from a claimed identity to a provable one, is the whole point of the protocol, and it is the first access-control mechanism in the AI crawler toolkit that does not rely on the bot's good manners.
How does Web Bot Auth work?
Web Bot Auth applies RFC 9421 (HTTP Message Signatures, an IETF Proposed Standard from February 2024) to automated traffic. The bot operator generates a key pair and publishes the public half as a JSON Web Key Set at a /.well-known/http-message-signatures-directory URL on a domain it controls. Every outbound request then carries three headers: Signature-Agent, which points to that key directory; Signature-Input, which describes what was signed; and Signature, which holds the signature itself. The receiving server fetches the directory, caches it according to its Cache-Control header, and verifies the signature.
Because the check is cryptographic, it survives the three failure modes that undermine older methods. A user-agent string can be typed by anyone. Reverse DNS suits crawlers with stable infrastructure but is awkward for agents running on shared cloud ranges. IP allowlists drift as cloud egress ranges change and are often shared with VPNs and proxies.
Is Web Bot Auth an official standard yet?
Not yet. This is the point most vendor explainers blur, and it matters for anyone writing it into a procurement document or a bot policy.
The IETF chartered a Web Bot Auth working group on 23 October 2025, according to a status review by Nerd Level Tech dated 12 August 2026. That review reported no adopted working-group documents at that point, a milestone for two standards-track specifications that had passed on 30 April 2026, and a further milestone for a Best Current Practice document due 31 August 2026. It also recorded a direction poll at IETF 126 in Vienna on 22 July 2026 that drew 22 yes, 6 no and 5 no opinion.
Since then the core document has moved. On 1 September 2026 the individual submission was re-indexed as draft-ietf-webbotauth-httpsig-protocol-00, titled "HTTP Message Signatures for automated traffic", with revision 01 following on 29 October 2026. The IETF Datatracker lists Thibault Meunier of Cloudflare and Sandor Major of Google as authors. At the time of writing the document is a working-group draft with a blank "Intended RFC status" field, and it expires on 2 May 2027 if not revised. In plain terms: the protocol is now on the IETF's track, but it is not an RFC, and details can still change. Google itself describes its own implementation as experimental for exactly that reason.
Which bots sign their requests?
Google's Google-Agent signs some requests, but only some. Google's documentation (last updated 13 October 2026) says only a subset of Google-Agent requests are signed, that they authenticate as https://agent.bot.goog, and that the public keys sit at agent.bot.goog/.well-known/http-message-signatures-directory. Google states that not every Google user agent uses the protocol and advises site owners to keep verifying by IP address, reverse DNS and user-agent string alongside it. Search Engine Journal reported the experiment on 5 May 2026, quoting the same caveat that a missing signature does not mean a bot is rogue.
OpenAI's help documentation says ChatGPT agent signs every outbound HTTP request following RFC 9421, with a Signature-Agent value of https://chatgpt.com. Cloudflare's signed agents programme, announced on 28 August 2025, listed ChatGPT agent alongside Goose from Block, Browserbase and Anchor Browser, and added Cloudflare's own Browser Rendering.
The pattern to take from this is that signing is strongest among agents acting for a user, and weakest among the traditional crawlers that still generate most of the volume. No source reviewed for this article shows GPTBot, ClaudeBot or the main search crawlers signing, so treat any claim that "AI crawlers are now cryptographically verified" as overstated.
Who verifies signatures on the publisher side?
Verification is already built into several platforms. Cloudflare validates signatures at the edge as part of its bot classification. Nerd Level Tech reports that Akamai, HUMAN and Vercel also verify signed requests, with Vercel starting in August 2025, and that AWS WAF added support on 21 November 2025, applying to CloudFront distributions only. Where a platform requires it, a publisher may need to add an allow rule for verified traffic. Publishers on other infrastructure can verify at the origin using open-source libraries such as those in Cloudflare's web-bot-auth research repository.
The practical operating rule is to run signature verification in parallel with your existing methods. Cache the key directory per its Cache-Control header, remove keys that have been rotated out, and fall back to IP and reverse DNS when a signature is absent or a directory is unreachable.
What does Web Bot Auth not do?
It is easy to over-read a verified signature, so the limits are worth stating plainly.
First, it proves the operator, not the person. The working-group charter puts authenticating the end user out of scope, so a valid signature from an agent tells you which company's infrastructure made the request and nothing about who asked it to.
Second, it does not catch bots that do not sign. Distinguishing non-participating bots is also out of scope, so stealth crawlers that mimic ordinary browsers remain a separate problem for conventional bot detection and, in some jurisdictions, for legislation.
Third, it is identity, not permission. A verified bot is still subject to whatever your robots.txt, Content Signals declarations and paywall say, and a signature does not oblige anyone to pay.
Fourth, implementations are not fully aligned. Nerd Level Tech notes that vendor documentation disagrees on the format of the Signature-Agent header, with Cloudflare treating one form as a failure that Google and OpenAI use differently, and that Cloudflare does no replay-database checking and recommends short expiry windows. Test against the specific bots you care about rather than assuming one validator handles all of them.
Where does it fit in a publisher's access-control stack?
It helps to separate three layers. The declaration layer, made up of robots.txt, Content Signals and Really Simple Licensing, states what you want. The identity layer, where Web Bot Auth sits, establishes who is asking. The enforcement and commercial layer, which includes pay-per-crawl and response-level monetisation, decides what happens to the request. Each upper layer is only as trustworthy as the one beneath it: a price for access means little if a crawler can lie about who it is.
For the part of AI traffic where a live search agent fetches a page server-side and composes an answer without the reader visiting, verified identity is the first step towards knowing who is reading, and therefore who could be charged or matched with a sponsor. blankspace works at the CDN edge to recognise that traffic and attach revenue to it, and stronger identity signals make that recognition more reliable. The signature itself does not create revenue; it makes the counterparty known.
Frequently asked questions
Is Web Bot Auth a replacement for robots.txt?
No. Robots.txt declares which URLs a crawler may fetch, while Web Bot Auth proves who is making the request. They answer different questions and apply together. A signed crawler that ignores a disallow rule is still ignoring it.
Is Googlebot cryptographically verified?
Not on the evidence available. Google's documentation says only a subset of Google-Agent requests are signed, and recommends keeping IP address, reverse DNS and user-agent verification alongside Web Bot Auth. Treat a missing signature as inconclusive rather than as proof of a rogue bot.
Can Web Bot Auth be used to charge crawlers?
It is not a payment mechanism. It makes the identity behind a pay-per-crawl or licensing arrangement credible, but charging happens through other layers such as HTTP 402 responses and the platforms built on them.
What goes wrong when a bot rotates its keys?
Verification fails if your system caches the key directory on a fixed schedule instead of honouring Cache-Control. Refresh on expiry, drop keys that have been removed, and fall back to the legacy verification path if the directory cannot be fetched, rather than blocking outright.
Should I call Web Bot Auth a standard in a policy document?
Describe it as an IETF working-group draft that is widely deployed. The core document reached working-group status on 1 September 2026 and was at revision 01 on 29 October 2026, but it is not an RFC, so wording that implies a finished standard would be inaccurate.