When an AI crawler requests a page a publisher has put a price on, Cloudflare stops the request at its network edge and answers with an HTTP 402 Payment Required response and a header stating the price. The crawler can either walk away or retry the request with a header agreeing to pay, at which point it receives the content. Cloudflare records the billing event, acts as merchant of record, aggregates the charges, bills the AI company and pays the publisher. That is the whole mechanism: a paywall for machines, applied by the network rather than the website, so a publisher never has to build a payments stack or negotiate crawler by crawler.
Pay per crawl launched in private beta on 1 July 2025 and sits inside Cloudflare's wider set of AI-traffic controls, which already reach a large share of the web. On 1 July 2026, Cloudflare's second Content Independence Day, the company extended the model in two directions that matter to publishers: a shift from charging per crawl towards charging per use, and a new set of defaults, arriving 15 September 2026, that block certain AI crawlers on ad-supported pages unless the site owner chooses otherwise.
How does Cloudflare pay per crawl work?
Pay per crawl gives a domain owner three choices for any given crawler. Allow grants free access. Charge requires payment at a flat, domain-wide, per-request price the publisher sets. Block refuses the crawler outright. When a crawler hits a charged URL without payment intent, Cloudflare returns the 402 response and a crawler-price header. To buy the page, the crawler repeats the request with a header that agrees the price, or sends a maximum-price header up front so it only pays where the cost sits under its ceiling.
Because Cloudflare is the merchant of record, the publisher does not invoice anyone. Cloudflare authenticates the crawler, logs each paid request that returns a successful response, tallies the events across the period, charges the AI company and distributes the earnings. The appeal for smaller publishers is that this removes both the technical build and the commercial negotiation, which is why the licensing conversation had until now been open only to the largest media groups with legal teams.
What is pay per use, and how is it different?
Pay per crawl charges for the fetch. Pay per use, the evolution Cloudflare set out on 1 July 2026, charges when the content actually creates value inside an AI product, not merely when it is retrieved. The distinction is the point: a crawler might read a hundred pages and use one, and a per-crawl fee captures the read while a per-use fee captures the moment the content appears in an answer.
Cloudflare is starting pay per use with two launch partners, Ceramic.ai and You.com. When a publisher opts in, it is paid when its content surfaces in Ceramic's AI search results, or when You.com accesses a piece of its premium content. Alongside this, Cloudflare is testing a new content-use signal in robots.txt that extends its Content Signals policy with a fourth field describing what a bot may keep and reshare: use=immediate (interact but store nothing), use=reference (index, excerpt and link back, the new default), and use=full (summarise and reproduce). Bots that reproduce content in full can no longer hold Verified status.
What changes on 15 September 2026?
From 15 September 2026, Cloudflare is setting new defaults built around a three-way taxonomy of AI behaviour rather than a single "AI bot" category. It splits automated AI traffic into Search (crawling to index a site so it can be found later), Agent (a bot acting in real time on a person's behalf, such as a chat fetch or a browser-use agent) and Training (crawling to train or fine-tune a model). On pages that display ads, Training and Agent crawlers will be blocked by default, while Search stays allowed, on the logic that an ad is a signal the publisher intended a human to land and see it.
The change applies to new domains onboarding to Cloudflare, new sites added by existing customers, and all existing free customers. Multi-purpose crawlers that combine search with training, such as Googlebot, Applebot and BingBot, are judged by all of their behaviours and blocked under the most restrictive rule that applies, so a site that blocks Training will also block those crawlers. Any publisher that wants different behaviour can opt out in its security settings at any point before 15 September and keep its current configuration.
Does pay per crawl actually pay publishers?
The infrastructure is now operating at real scale. Cloudflare says its AI crawl control returns more than one billion 402 responses a day across its customer base, and that content owners signed over 50 major licensing agreements in the past year. Early participants named around the launch include Condé Nast, TIME, the Associated Press, BuzzFeed, Reddit, Pinterest, Stack Overflow, The Atlantic and Fortune. Cloudflare has also added an Attribution Business Insights dashboard so publishers can see crawler appetite and value, and a Monetization Gateway that settles charges in stablecoins over the open x402 protocol.
The honest caveat is that a charge only lands when the AI company agrees to pay. A crawler is free to refuse the 402 and leave, and the model depends on operators identifying themselves honestly and holding Verified status. It works best where an AI company wants the content enough to pay for it and is willing to be named. For the long tail of live reads, where an agent fetches a page to answer a single question and no payment relationship exists, blocking is the only lever pay per crawl offers, and blocking an agent on an ad-supported page can remove the publisher from the very answer that now replaces the click.
What pay per crawl does not solve for publishers
Cloudflare's move validates a thesis the industry is converging on: compensation has to attach to the use of content, not just its retrieval, which is exactly what pay per use reaches for. The gap it leaves is the answer surface itself. Permission and charging at the crawl decide whether a bot may take the content; they do not monetise the AI response that a reader actually sees, and for most publishers most of the time there is no paying, Verified crawler at the door to charge.
This is the layer blankspace works in. Rather than pricing or blocking the fetch, blankspace detects live search agent traffic at the CDN edge and injects contextual brand facts into the AI response, so the read is monetised at the point of the answer regardless of whether the crawler has signed a payment agreement. The two approaches are complementary: a publisher can license and charge the crawlers willing to pay through Cloudflare, and still monetise the far larger volume of live answer traffic that no per-crawl fee reaches. The strategic error would be to treat an ad-page block as the end of the story, because the traffic that skips the block is where the answer, and the value, now lives.
Frequently asked questions
Is Cloudflare pay per crawl free for publishers to use?
Pay per crawl is offered through Cloudflare and does not require the publisher to build any payments infrastructure, because Cloudflare acts as the merchant of record, charges the crawler and distributes the earnings. The publisher sets a flat per-request price and Cloudflare handles authentication, billing and settlement. It launched in private beta in July 2025, so availability and terms depend on a site's Cloudflare plan and access to the programme.
How is pay per crawl different from Cloudflare's Content Signals policy?
Content Signals is a preference layer written into robots.txt that tells crawlers how a publisher wants content used, using signals such as search, ai-input and ai-train, and now a content-use field. It expresses a wish and is not self-enforcing. Pay per crawl is the enforcement and payment layer: it actually returns an HTTP 402, withholds the content until payment is agreed, and moves money. One declares intent, the other collects.
Will the 15 September 2026 change block Google from my site?
Only if you choose to block Training. From that date, on ad-supported pages, Cloudflare blocks Training and Agent crawlers by default for new and free customers, and judges multi-purpose crawlers such as Googlebot by all their behaviours under the most restrictive rule. Search behaviour stays allowed by default, and any publisher can opt out in security settings before the date to keep classic search crawling untouched.
Does pay per crawl stop AI answers from using my content?
Not on its own. Pay per crawl governs whether a crawler may fetch a page and whether it pays to do so, but a large share of AI reads come from live agents answering a single question, and where a publisher blocks rather than charges those agents it can drop out of the answer entirely. Charging captures value only from crawlers willing to pay and be verified, so it does not, by itself, monetise every AI response that draws on a site.
What is the difference between pay per crawl and pay per use?
Pay per crawl charges a fee each time a crawler fetches a priced page. Pay per use charges when the content actually creates value in an AI product, for example when it appears in an AI search result or is drawn on to answer a query. Cloudflare began rolling pay per use out on 1 July 2026 with launch partners Ceramic.ai and You.com, positioning it as a fairer match between payment and the point at which content is genuinely used.