← Back to blogTollBit vs Cloudflare pay per crawl vs ad injection: a publisher's guide to monetising AI bots

TollBit vs Cloudflare pay per crawl vs ad injection: a publisher's guide to monetising AI bots

TollBit and Cloudflare charge AI bots for access. Ad injection earns from the read itself. In 2026 the access side stopped being a two-vendor question and became infrastructure, with Cloudflare, AWS and Akamai all billing crawlers natively and Cloudflare moving from per-crawl to per-use pricing. Here is what each model actually captures, and where all of them leak.


Pick any of these models and you are answering the same question: at what moment does the publisher get paid. TollBit bills at the request, when a crawler asks for the page. Cloudflare has spent 2026 moving that billing moment forward, from the fetch to the point where the content is used inside an AI answer. Content-layer ad injection, the model blankspace uses, never bills the bot at all and instead runs an auction on the read, earning from a paid brand fact placed in the text the agent parses. The three are not really competitors so much as three different meters attached to three different points in the same pipe, which is why publishers who understand their traffic mix usually end up running more than one.

What changed in 2026: access charging became infrastructure

Four figures: 52% of crawler requests were for AI training as of June 2026, up from 22% in spring 2025; around 30% of Q4 2025 AI bot scrapes bypassed robots.txt; roughly 40 third-party scrapers were named reselling publisher content; and blankspace reports an average 5 CPM on Live Search retrievals in its own accounts.

The original version of this comparison treated access charging as something two specialist vendors did. That is no longer the shape of the market.

On 1 July 2026, a year after launching Pay Per Crawl, Cloudflare declared the per-fetch charge insufficient and set out Pay Per Use, which compensates publishers when their content actually creates value inside an AI product rather than when a bot collects it. It launched with two demand partners, Ceramic.ai and You.com: an opted-in publisher is paid when its content appears in Ceramic's AI search results, or when You.com accesses a piece of its premium content.

On 17 June 2026, AWS made the same capability native to its own edge. Any publisher behind CloudFront and the AWS Web Application Firewall can now charge AI agents per request, with the crawler receiving an HTTP 402 Payment Required response carrying the price and settling in USDC through the x402 protocol. It is generally available at no cost beyond standard WAF pricing. Cloudflare's equivalent Monetization Gateway remains waitlist-only.

Underneath both sits an open standard. x402 revives the 402 Payment Required status code that has sat reserved and unused since HTTP/1.1 in 1997. Coinbase contributed the protocol to the Linux Foundation in April 2026, and the x402 Foundation now counts AWS, Cloudflare, Anthropic, Circle and more than twenty other members. Akamai, meanwhile, has built on its bot-management heritage to apply policy per bot, per path and per price, with monetisation delivered through partnerships with TollBit and with Skyfire, whose Know Your Agent and KYAPay layer verifies bot identity so content is only served to authorised, paying agents.

The practical consequence for publishers is that the access decision has moved from "which vendor do I sign with" to "what does my CDN already do by default, and have I configured it".

TollBit

TollBit operates a paywall and marketplace for AI bots. A crawler requesting a publisher page is intercepted and met with a fee the publisher sets, priced by AI company, content category and usage type, with a summarisation licence distinguished from a full-display licence. Access is logged and AI companies are invoiced. TollBit states that publishers keep their full set rate and that it charges AI customers a transaction fee on top rather than taking a revenue share. It runs across more than 3,000 sites and is distributed through Arc XP, the Washington Post's publishing platform.

Its second function has become as valuable as its first. TollBit's State of the Bots reporting is now one of the few public measurement series in the category, and TIME has used TollBit traffic data as leverage in licensing negotiations with OpenAI and Perplexity. Knowing precisely who is taking what, and how often, is what makes any of these conversations possible.

Strengths: granular per-use pricing, control by company and content type, credible measurement, and a clean transaction for AI companies that want to be compliant. Limitation: it depends on crawlers being willing to identify themselves and pay. An agent that fetches anonymously to answer one user prompt and declines the charge is not monetised, and the model remains access control rather than advertising.

Cloudflare: from pay per crawl to pay per use

Cloudflare became the first major infrastructure provider to block AI crawlers by default in 2025 and introduced pay per crawl as a third option between allow and block. Publishers set a price, AI companies decide whether to pay, a billing event is recorded when a crawler makes an authenticated request with payment intent and gets a successful response, and Cloudflare aggregates and pays out. Because it sits in front of a very large share of the web, its defaults move the market on their own.

Two changes matter this year. The pricing unit is moving from the crawl to the use, as described above. And from 15 September 2026, Cloudflare's defaults for new and free sites will block mixed-use crawlers, those blending search indexing with AI training and agent retrieval, from pages that carry advertising, while continuing to permit genuine search indexing. That forces AI companies to declare crawler purpose properly, which is the precondition for pricing purpose separately. It is also a deadline: publishers on Cloudflare should check now what their configuration will do to their ad-supported pages in September.

Strengths: default protection, network-level scale, no integration beyond being a customer, and the most credible route to per-use pricing at scale. Limitations: it still monetises crawlers that authenticate and accept a price, the intermediary takes a cut that independent analysts have estimated at around 30%, and Pay Per Use currently depends on a small set of participating AI companies rather than the whole demand side.

Content-layer ad injection

Ad injection earns from the content rather than the access. Live Search Agents parse text and ignore JavaScript, so the monetisation has to happen in the text the agent reads, server-side, before the agent processes the page. This is the layer blankspace operates in. When a Live Search Agent retrieves an eligible page, blankspace runs a sub-50ms auction at the CDN edge and injects a relevant, accurate brand fact into the content as editorial context. Human readers see nothing different. Advertisers target by IAB content category, market and the specific prompt clusters users ask, and revenue is attributed the same day. In live blankspace accounts the average CPM has benchmarked at around $15, which is blankspace's own reported figure from its own accounts rather than an industry benchmark.

Strengths: it monetises the anonymous Live Search retrievals the access models miss, it requires no agreement from the AI company, and it pays per high-intent retrieval rather than per generic access. It is also, in principle, the same billing moment Cloudflare is now moving towards: value from the use of the content, not the act of fetching it. Limitation: it monetises retrieval and reading rather than bulk training ingestion, so it complements rather than replaces a licensing or access strategy aimed at training crawlers.

The limitation all three share

Flow showing two paths to the same page: a compliant crawler identifies itself and pays through an access meter such as TollBit, Cloudflare or AWS, while a third-party scraper bypasses robots.txt entirely and reaches the publisher as an unidentified request that pays nothing.

No model on this page monetises a bot that refuses to play. TollBit's "Pipes are Leaky" report found that around 30% of total AI bot scrapes in Q4 2025 bypassed explicit robots.txt permissions, named roughly 40 third-party scrapers reselling publisher content, and found in tests across 30 high-authority sites that some scrapers retrieved paywalled articles in full. Many of those tools advertise evasion capabilities and ignore robots.txt by default.

That matters for how you read every claim in this comparison. Access models price the compliant crawler. Content-layer advertising monetises the read whether or not the reader announced itself, which covers more of the leakage, but neither approach makes an unauthorised bulk scrape disappear. Enforcement, legal action and identity standards such as Web Bot Auth and Skyfire's Know Your Agent are doing the work that pricing alone cannot.

How the three compare

Comparison of access charging against content-layer ad injection: access charging bills the fetch or the use, needs the bot to identify itself and works best on identified training crawlers, while ad injection bills the read, needs no agreement from the AI company and works best on anonymous Live Search Agents.

Access models charge AI companies to reach your content and work best on identified training crawlers, which now account for the majority of the problem: 52% of crawler requests were for AI training as of June 2026, up from 22% in spring 2025. Ad injection earns from contextual brand mentions inside the content and works best on anonymous Live Search Agents answering live user questions. TollBit and Cloudflare put a price on the door, and Cloudflare is now also trying to put a meter inside the room. blankspace only ever earned from what happens inside the room.

A publisher can sensibly block low-value scrapers, charge identified training crawlers through an access marketplace or their CDN, and monetise Live Search retrievals through content-layer advertising, all at once, because each addresses a different category of bot and a different billing moment.

How to choose

Look at your AI traffic by type before you look at vendors, using server-side measurement, because client-side analytics cannot see agents that never run JavaScript.

If training crawlers from a few named companies dominate, an access marketplace or your CDN's native billing captures more, and the configuration work is small. If Live Search Agents are your largest and fastest-growing category, which is increasingly the case, content-layer advertising is the only model that monetises them. If you are on Cloudflare or AWS, part of this decision has already been made for you by a default, so the immediate task is to check what that default is. And whichever you choose, assume a meaningful share of scraping will route around it, and price accordingly rather than counting on full capture.

Frequently asked questions

Is TollBit or Cloudflare pay per crawl better?

Neither is universally better, and Cloudflare's offer has changed. Cloudflare gives you default protection and network scale with no separate integration if you are already a customer, and is moving from per-crawl to per-use pricing. TollBit gives you more granular pricing by company and usage type, plus measurement data that has proved useful in licensing negotiations. Both charge for access and both take or add a fee.

What is the difference between pay per crawl and pay per use?

Pay per crawl bills the fetch: a bot requests a page and pays a set price for access. Pay per use bills the outcome: the publisher is paid when the content actually appears in or grounds an AI product, or when an agent buys premium information to complete a task. Cloudflare set out this shift on 1 July 2026, launching with Ceramic.ai and You.com as its first demand partners.

Can I use ad injection and access charging together?

Yes, and most publishers should. They monetise different traffic and different moments. Access charging bills identified training crawlers for entry, while ad injection earns from anonymous Live Search retrievals that would otherwise pass through free. Running both captures more of your total AI traffic than either alone.

Do access models monetise ChatGPT and Perplexity live searches?

Only partially. When a Live Search Agent fetches a page anonymously to answer a user prompt and does not accept an access charge, the access model earns nothing from that visit. Content-layer advertising is designed to monetise exactly those retrievals, and Cloudflare's move to per-use pricing is an attempt to reach the same value from the access side.

How much do the intermediaries take?

TollBit states it does not take a revenue share and instead adds a transaction fee for AI customers. Independent analysts have estimated Cloudflare's pay-per-crawl cut at around 30%, and some licensing marketplaces at around 15%. AWS charges nothing beyond standard WAF pricing for its x402 integration. blankspace operates a revenue share, with publishers receiving 70%.