Before the number comes the unit. A publisher setting a price for AI access is choosing between three different things to sell: the fetch, which is the request itself; the use, which is the content shaping an answer; and the appearance, which is a citation underneath one. They command different amounts and, more importantly, they differ in whether you can check that you were paid correctly. Only once that choice is made does the rate become a tractable question, answered by what your infrastructure permits, what the buyer on the other end is authorised to spend, and which tier of your library the requested page sits in. Most publishers get stuck because they try to pick the number first.
Why there is no rate card to copy
Every mechanism on the market asks the publisher to name a price and none of them publishes what other publishers named.
Cloudflare's pay per crawl remains in closed beta. Its documentation, last updated on 28 July 2026, tells site owners to set a default per crawl price charged on each successful content retrieval, with a stated minimum of one tenth of a US cent, and adds only that publishers should "consider your content value and expected crawler volume". There is no suggested range. Cloudflare acts as merchant of record and pays out monthly, subject to settlement periods and minimum thresholds.
TollBit works the same way from the other end. Publishers set their own rates per thousand pages accessed against two standard licences, Summarization and Full Display, keep all of the rate they set, and TollBit adds a transaction fee paid by the AI buyer rather than taking a revenue share. Rules can then vary that rate by bot, page, keyword, time window and directory.
The closest thing to a benchmark anyone has published is an outcome, not a rate. Toshit Panigrahi, TollBit's chief executive and co-founder, told Digiday in April 2026 that nearly 20 per cent of the network's roughly 7,000 publisher sites had earned money from its bot paywall, in amounts ranging from hundreds of dollars to tens of thousands a month. That is a spread of results across thousands of different prices, content libraries and crawl volumes. It tells you the range of the prize. It does not tell you what to type into the box.
The three units you might be selling, and which one you can audit
A fetch is a request that returns content. You can see it in your own logs, you can count it, and you can match a payment to it. Cloudflare's pay per crawl, TollBit's bot paywall and open payment rails such as x402 all price the fetch.
A use is the content actually informing an answer. Cloudflare has said it is moving its default emphasis from pay per crawl towards pay per use, with pilots involving Ceramic.ai, which pays publishers when their content appears in its AI search results, and You.com, which lets an agent pay on demand for a specific piece of premium content. Google is running its own version, the AI contribution pilot, which pays when a page is judged to have contributed significantly to a response in AI Overviews, AI Mode or the Gemini app. Being linked after the answer has been written does not qualify.
An appearance is a citation. Nobody currently pays for one.
A use is the better proxy for value. It is also the one you cannot verify. Once a page has left your server you have no independent way of observing whether it shaped an answer, so a pay-per-use arrangement means accepting the platform's arithmetic. Publishers inside Google's pilot get a monthly earnings figure in Search Console with no breakdown of how it was calculated. One executive with knowledge of the programme described it to Digiday as "quite black box", and a source close to the situation said early returns had been "peanuts" relative to advertising revenue. David Buttle, founder of the publisher AI coalition Spur, read the pilot as a hedge rather than a market, though he allowed that it at least establishes the precedent that usage events convey value.
The practical implication is uncomfortable but simple. The unit that pays better is the one you cannot check. The unit you can check is the one you should price first, because it is the only one where your number and your invoice are the same number.
What the floor and the ceiling look like in September 2026
The floor is set by infrastructure. One tenth of a US cent is Cloudflare's stated minimum. Anything at that level is symbolic rather than commercial for most sites.
The ceiling is set by the buyer, and it is lower than most publishers assume, because the buyer is increasingly an agent with a spending limit rather than a procurement team. Suganthan Mohanadasan, co-founder of Snippet Digital, published a working x402 demonstration on 15 September 2026 in which his site charged one US cent per page and his own buying agent ran with a cap of five US cents per call and a daily allowance of twenty-five cents. When he set the per-call cap below the page price, the agent refused the purchase outright and exited rather than negotiating. That refusal is the ceiling made visible.
The payment plumbing around those agents is arriving quickly. The x402 Foundation became operational under the Linux Foundation on 14 July 2026 with 40 members including Visa, Mastercard, Google, AWS, Stripe and Cloudflare. Amazon made Bedrock AgentCore payments generally available on 18 August 2026. Cloudflare announced its Monetization Gateway on 1 July 2026 and Wallets on 4 August 2026, both still gated behind waitlists, with reserved wallet handles unable to hold funds yet.
Two cautions belong next to that. First, volume on these rails is not evidence of demand: Decrypt reported on 13 September 2026 that TRM Labs had examined 198.9 million x402 settlements and found most of the payment volume was not coming from AI agents, with scheduled scripts, load tests and self-dealing producing similar records. Second, and more bluntly, GPTBot, ClaudeBot, PerplexityBot and Googlebot do not currently turn up at your origin with a wallet. Requiring payment from them today does not produce revenue. It produces a block.
So the honest working band for a fetch is tenths of a cent to a few cents. Above a few cents you are not pricing access, you are declining it with extra steps.
Why a single flat price is the worst option available
This is the one part of the question with real evidence behind it.
Richard Archer, Soheil Ghili and Nima Haghpanah published "Pay-Per-Crawl Pricing for AI: The LM-Tree Agent" on arXiv on 1 April 2026. They tested pricing strategies against real content from HardwareLuxx, a German technology publisher, using 8,939 articles and 80,451 buyer queries with willingness to pay calibrated from actual AI crawler traffic. Their adaptive pricing agent delivered a 65 per cent revenue gain over a single static price, a 47 per cent gain over two-category pricing, and a 40 per cent gain over the publisher's own eight-segment editorial taxonomy.
The last of those three numbers is the interesting one. The publisher's existing categories, built for human readers and ad sales, were themselves leaving 40 per cent on the table. The model found distinctions the taxonomy did not carry: articles about flagship GPUs commanded higher prices even though "flagship" was not a category the publisher had ever defined.
Two caveats are worth stating plainly. This is a simulation with willingness to pay calibrated from crawler behaviour, not audited transacted revenue. And it tests one technology publisher's library, which may be unusually easy to tier.
Even discounted heavily, the direction holds: uniform pricing is the weakest strategy, and your own content taxonomy is probably not the right basis for a rate card. Both platforms already support the alternative. Cloudflare allows dynamic pricing through a crawler-price response header or a Worker that sets prices by request properties. TollBit allows pricing rules by page, directory, keyword and bot.
A method for setting your first number
Work out what a page earns from a human. Take your session revenue per thousand and divide by a thousand. That figure is a reference point, not a price. A crawl is not a pageview: it may feed one answer or a hundred, and it delivers no ad impression and no reader.
Work out your crawl-to-human ratio. TollBit's State of the Bots reporting put AI bot activity at one bot visit for every 31 human visits in the fourth quarter of 2025, up from one in 50 in the second quarter. Arc XP, The Washington Post's publishing platform arm, reported a 300 per cent year-on-year jump in AI-driven bot traffic across its CDN and found media and publishing sites seven times more likely than the average site to see it. Your own ratio decides whether a tenth of a cent is trivial or material.
Discount for redundancy. Cloudflare's data indicates more than half of AI crawler traffic re-fetches pages that have not changed. A flat per-fetch price charges for that repetition, which is good for you until the buyer notices and starts caching, at which point your volume falls. Pricing the first fetch of a changed page differently from the fiftieth fetch of a static one is a more durable position than charging a high flat rate.
Tier the library before you price it. Keep explainers, evergreen guides and anything whose job is discovery freely accessible, because citation still has value and a paywalled explainer simply disappears from answers. Price original datasets, proprietary research, archives and tools, which are the things an agent needs specifically in order to finish a task and cannot substitute from elsewhere.
Set it, log every 402, and move it. Nothing about the first number is binding, and neither platform requires renegotiation to change it. The information you want is the shape of the demand curve, which you only get by moving the price and watching what stops buying.
What a price per crawl cannot do for you
A rate is a term of access. It is not demand, and publishers should be clear-eyed about which of the two they are short of.
For publishers with leverage, pricing is a negotiating instrument and it appears to work. People Inc's chief executive told investors that blocking crawlers through Cloudflare had helped bring AI companies to the negotiating table, as Press Gazette reported on 4 September 2026. For everyone else, the per-crawl price is an experiment run against buyers who mostly cannot pay it yet, and the realistic near-term outcome is a smaller AI footprint rather than a new revenue line.
That gap is why the fetch is not the only unit worth pricing. The moment an AI agent retrieves a publisher page to build an answer is also an ad opportunity, and an impression served into that retrieval at the CDN edge is priced in the currency publishers already understand, settles on the publisher's own logs, and does not depend on the buyer holding a wallet. That is the model blankspace operates. It answers a different question from the one this article poses, and it is not a substitute for a licensing position: a publisher with the leverage to negotiate a deal should still negotiate one.
Frequently asked questions
What is the minimum price a publisher can charge an AI crawler?
On Cloudflare's pay per crawl, the documented minimum is one tenth of a US cent per successful retrieval, charged when the crawler receives an HTTP 200 response. TollBit sets rates per thousand pages accessed rather than per request and does not publish a floor. Open payment rails such as x402 have no floor beyond what the settlement network makes economic.
Is there a going rate for a crawl in 2026?
No. No exchange, clearing price or published average exists for per-crawl access, and every mechanism on the market asks the publisher to name a price rather than quoting one. The only public figures are outcome ranges, such as TollBit's report that nearly 20 per cent of its roughly 7,000 publisher sites earn between hundreds and tens of thousands of dollars a month. Anyone quoting a market rate is quoting their own ask.
Should a publisher charge the same price for every page?
The evidence says no. Yale researchers testing pricing strategies against 8,939 articles from a German technology publisher found an adaptive per-article approach returned 65 per cent more revenue than a single static price and 40 per cent more than the publisher's own editorial categories. Both Cloudflare and TollBit support rules that vary price by page, directory or crawler, so flat pricing is a choice rather than a constraint.
Will charging for crawls stop a publisher being cited in AI answers?
For most content, yes, and that is the real cost. The major crawlers do not currently arrive with payment credentials, so a price on a page functions as a block, and blocked pages are not retrieved, summarised or cited. The usual resolution is to tier: leave discovery content open and price only the material an agent genuinely cannot substitute.
Is pay per crawl better than pay per use?
They trade off differently. Pay per use tracks value more closely, because content informing an answer is worth more than content merely downloaded, which is why Cloudflare is piloting it with Ceramic.ai and You.com and why Google's AI contribution pilot is structured that way. Pay per crawl tracks something you can verify yourself. Publishers inside Google's pilot receive a monthly figure with no calculation attached, which is the structural weakness of every usage-based scheme: you are auditing nothing.