Blog

United Airlines has rewritten its Contract of Carriage to ban AI agents from booking, changing or servicing its tickets, even when a human approves every step. Agents can still search and show United fares. The line it draws between the read and the transaction is the one publishers and brands now have to price.

New SparkToro and Similarweb data puts visits to the web's top million sites 12.2% below their 2022 peak, even though the online population grew by about a quarter. People visit less per head; the reading has not stopped, it has moved somewhere visit counters cannot see.

OpenAI is testing a "Claim your business in ChatGPT" hub that would show verified businesses how often they appear in ChatGPT answers and how many clicks follow. It would be the first first-party measure of brand visibility inside an LLM, and it leaves out the prompts, the competitors and the publishers whose content built the answer.

Ziff Davis says organic Google referrals fell about 37% year on year in Q3 2026, while Google's own Search Console counted roughly 2.7 billion AI impressions for its content, nearly ten times its search clicks. The read is growing. The referral that used to pay for it is not.

Not yet. Meta paused its new commercial terms on 5 November 2026 after publisher, political, court and regulator pressure, and its existing terms stay in force. The scraping clause may return in some form, so publishers should keep the contract risk on the register while robots.txt and edge controls do the day-to-day work.

Anthropic, Google, Microsoft and OpenAI are lining up behind a single robots.txt signal that lets publishers opt out of AI training across providers. It is a real step forward for training crawlers, and it says nothing about the live reads that happen every time an AI answers a question.

USA TODAY Co.'s third quarter put two lines side by side: AI partnerships and licensing up 24.8%, search-fed indirect advertising down 32%. The money from machines is real, but it arrives as deals, not as a price on every read.

Meta AI is the assistant built into WhatsApp, Instagram, Facebook and Messenger, and since April 2026 it has run on Meta's own Muse Spark model rather than Llama. For publishers it is above all an access-control question: Meta documents several separate crawlers, one of which may bypass robots.txt, and it has published no referral data to show that any traffic comes back.

A new analysis shows that Google's step-by-step instructions for verifying Googlebot accept any rented Google Cloud server. The same research finds that most traffic claiming to be certain AI agents is fake, which means publishers cannot price, block or monetise AI reads until they can prove who is reading.

Microsoft Copilot answers questions using pages from the Bing index, and it cites publishers far more often than it sends them visitors. Bing now reports citations in Webmaster Tools but not clicks, so publishers have to measure the gap themselves.

The New York Times and three other US dailies have sued Brave, alleging it crawls their journalism, bypasses paywalls and sells the content to AI companies through a real-time search API. The case shows that the machine read already has a market price; the publisher is just not the one collecting it.

Siri AI is Apple's Gemini-backed assistant, in beta since 14 September 2026, and it answers web questions from Apple's own crawl of publisher sites. Sites are included by default unless they opt out, and Apple publishes no citation or traffic reporting, so publishers can control the use of their content but cannot yet measure or monetise it.

GEO ROI cannot be read from one dashboard, because most AI influence never produces a tracked click. A credible number comes from combining four layers - crawl and citation visibility, branded search lift, AI-attributed traffic and self-reported or modelled revenue - and being honest about which parts are estimated.

AI deep research agents - from ChatGPT, Gemini, Claude and Perplexity - read dozens of web pages per question and hand the user a finished, cited report. Publishers supply the raw material, but the user rarely visits, no ad loads, and no payment follows.

Web Bot Auth lets a publisher check a bot's cryptographic signature instead of trusting its user-agent string. It is live at Cloudflare, Akamai, HUMAN, Vercel and AWS WAF, signed by OpenAI's ChatGPT agent and partly by Google, but the IETF specification is still an unfinished working-group draft.

Cloudflare's Content Signals Policy lets publishers declare in robots.txt whether content may be used for search, live AI answers or AI training. It is a cheap statement of intent, but Google has said it has no effect on any crawler it knows of, and it earns nothing from the AI traffic it permits.

AI browsers put an agent between the reader and your page. It reads, summarises and acts on the user's behalf while presenting itself as ordinary Chrome, so it slips past robots.txt, user-agent rules and many paywalls. The category is already consolidating - OpenAI shut Atlas in August 2026 - but the behaviour is moving into mainstream browsers.

Google AI Mode is a Gemini-powered chat inside Search that answers a query in one conversation instead of returning a page of links. Controlled experiments now show it cuts clicks to publishers sharply, and on 28 October Google confirmed it is taking ads into the answer itself. The content still feeds the reply; the visit, the impression and the affiliate link increasingly do not happen.

RSL is an open standard that lets publishers state the price and terms for AI use of their content in a machine-readable file. About 1,500 organisations back it and Cloudflare and Akamai support it, but no AI company has been named as a licensee. Until a buyer signs, it is an asking price - useful, but not yet revenue.

llms.txt is a Markdown file that gives AI models a curated map of your site. Adoption has grown fast, but Ahrefs found that 97% of the files it examined received no requests in a month, and Google says its search systems ignore it. Publishers can add one cheaply as a comprehension aid, but it will not lift AI citations or earn a penny.

Yes, and the category has moved from pilot to line item. Future reports more than 30 GEO clients, Ziff Davis and Time sell it, and the Daily Mail is running 12 to 15 tests ahead of a product launch. What nobody can sell is a guaranteed placement in an AI answer, so it prices as premium branded content and consultancy, not as a performance buy.

The UK's Automated Online Software (Access and Transparency) Bill and the US Stealth Bot Prohibition Act both try to make scrapers say who they are. That helps publishers deal with the bots that already identify themselves, but a law cannot verify anyone - and the fakes are the real problem.

Google says it will place more links higher in AI Overviews and AI Mode, but it has published no click data to show that this sends readers to publishers. Treat the changes as a visibility upgrade that is still unproven as a traffic recovery.

Not yet, and not cleanly. Publishers have survived early motions and one large authors' settlement has been paid out, but no court has yet ruled on the core news-publisher claim, and the first AI Overviews antitrust suits have just been thrown out.

Yes, but less than the advice online suggests. Large studies show AI assistants cite newer pages than Google ranks, yet the typical cited page is still years old, and freshness only counts when the content has genuinely changed.

Your server logs record every AI crawler and agent visit that your analytics tool never sees. A short filter on user agent strings, checked against each vendor's published IP ranges, shows who is reading your site, why, and how often.

DeeperDive is Taboola's AI answer engine that sits on a publisher's own site and earns revenue from ads placed inside its answers. It monetises humans who are already on the page, not the AI crawlers reading the site from outside, which is why the two models complement rather than compete.

Common Crawl is a free public archive of the web that feeds many AI training datasets, and blocking its CCBot crawler stops future collection but does not remove what is already stored. Most large news publishers already block it. The decision is a trade-off between control and open-web presence, not a technical default.

Registration walls trade volume for known audiences, and as search referrals shrink more publishers are willing to make that trade. The evidence says registration lifts conversion and programmatic yield, but only when the wall is soft, the value exchange is clear and the lost traffic was low-value anyway.

Sometimes, and the evidence is thinner than the headlines suggest. Independent tests have caught assistants citing a Yahoo or MSN copy of a story instead of the newsroom that reported it, and Google and Bing give partners opposite advice on how to stop it. The strongest fix sits in the syndication contract, not in the markup.

No - Google classes Google-Agent as a user-triggered fetcher, and its documentation says fetchers of that kind generally ignore robots.txt. Publishers who want to restrict it need identity checks and access controls at the server or CDN edge, not a Disallow line.

Bots now outnumber people online, AI harvesters have grown from 8 to 13 per cent of web sessions in a year, and chatbots still send publishers less than 1 per cent of their referrals. These are the AI bot traffic statistics that matter in 2026, each tied to a named source.

Agentic commerce is AI agents researching, comparing and buying on a shopper's behalf. The first big attempt at in-chat checkout stalled in 2026, but the discovery stage, where agents read publisher content, is growing fast and still goes unpaid.

AI share of voice is your brand's mentions as a proportion of all brand mentions across a fixed set of category prompts, measured separately for each assistant. It only means something if you sample enough, repeat the runs and hold the method constant. Here is the formula, the IAB's August 2026 standard, and what to do with the number.

CDN-layer monetisation earns revenue from a request at the network edge, before any page code runs. It matters because the AI agents now reading publisher sites never execute the JavaScript that browser-side monetisation depends on. In 2026 it has split into two distinct models: charging for access, and earning from the content once access is granted.

AI bots do not appear in Google Analytics because they never run the JavaScript that records a visit - and the bots that do get recorded are quietly deleted by a filter you cannot switch off. The read still happened, and you still earned nothing from it. Here is why the blind spot exists and how to see your AI traffic.

Generative engine optimisation (GEO) is the practice of making your content easy for AI assistants to find, trust and quote. It competes for citations inside answers, not rankings on a results page, and Google now says that for its own AI features it is still SEO.

No. robots.txt is a request, not a barrier, and TollBit's 2026 data shows a meaningful share of AI fetches reaching pages the file told them to avoid. Some bots are not even bound by it by their own operators' account, so enforcement has to happen at the network edge.

Yes, and in volume. A new study of more than 106,000 citations found that 25.4 per cent led to pages disclosing a commercial interest, that 86 per cent of best-product answers cited at least one, and that ten review publishers supplied roughly a quarter of them. The layer AI assistants trust for buying advice is an affiliate layer, and the assistants are draining the clicks that pay for it.

On 2 September 2026 a federal judge spared Google's ad exchange from a break-up and ordered six years of behavioural fixes instead: forced interoperability with Prebid and rival ad servers, real-time bid-data sharing, and a court-appointed monitor. Publishers polled by Digiday scored the package five out of ten - real accountability, but the earliest meaningful change is not expected before late 2027.

AI bots requesting your pages now split into four groups, not three: training crawlers, retrieval crawlers, live agents, and a growing set that will not identify itself at all. This updated reference names the current user agents, from Google-CloudVertexBot and Meta-ExternalFetcher to Bytespider and Grok, and sets out which ones actually honour the rules you set for them.

The IAB Tech Lab Agent Registry checks that an AI agent's operator is a real, privacy-compliant company with a working endpoint - it does not check that the agent is entitled to sell any particular publisher's inventory. Six months after its March 2026 launch it remains a small, mostly self-reported directory, and the one feature that would answer a publisher's actual question is still listed as a future roadmap item rather than something the registry does today.

AI visibility tools now split into two camps: those that sample what ChatGPT, Gemini and Perplexity say about you, and those that log when AI agents fetch your pages. Profound, Semrush, Ahrefs, Otterly, Peec and Microsoft Clarity each sit somewhere on that line. None of them agree on the numbers, and none of them turns a read into revenue.

AdCP is an open standard that lets AI agents discover, plan, buy and report on advertising in one shared language. Version 3.0, published in April 2026, stretched it from media buying into brand identity, governance and ads inside AI assistants. It is now the most complete agentic ad protocol available - and still the subject of an open dispute with IAB Tech Lab.

No. CoMP is IAB Tech Lab's standard for how AI systems and content owners describe content and its terms to one another, and the specification states plainly that licensing terms, token issuance, counting and payment are all out of scope. It is the vocabulary for a transaction, not the transaction itself. A publisher still has to block first and negotiate separately.

Nobody, formally. The three specifications that police this on the open web all anchor authorisation to a domain, and a generated answer does not have one. The rival proposal now splitting the industry, adagents.json, does not close that gap either, which leaves publishers with a commercial question they cannot yet answer in a machine-readable file.

Yes. No regulator has written a rule specific to AI assistants, and none needs to: the FTC, the CAP Code and the Digital Services Act are all media-neutral and already bite. The unsolved problem is not whether a label is required but what a label looks like when the paid element is a sentence inside generated text rather than a unit beside it.

Four competing standards now exist for letting software pay for a resource without a human at the checkout: Google's AP2, OpenAI and Stripe's ACP, Stripe and Tempo's MPP, and x402. Only one of them settles in public, and blockchain analysts at TRM Labs found that agents move roughly 5,000 to 11,000 US dollars a month across it. The plumbing is real. The money is not there yet.

No market rate for a crawl exists in September 2026, and anyone quoting one is quoting their own ask. What does exist is a floor of one tenth of a US cent on Cloudflare, buying agents with per-call budgets measured in single cents, and peer-reviewed evidence that a single flat price is the worst of the available strategies. Price the unit you can audit, tier it by content, and treat the first number as an experiment.

Microsoft publishes no Copilot-specific metrics and Google will not separate AI Mode delivery in reporting, yet both serve existing campaigns into their assistants automatically. ChatGPT is the exception, bought on purpose through OpenAI, Criteo, StackAdapt or, since 10 September 2026, Amazon DSP. Perplexity, Meta AI and Claude sell nothing.

The bandwidth bill is almost always trivial. A mid-sized publisher serving ten million crawler requests a month is looking at roughly fifty dollars of egress, and on a flat-rate CDN it is zero. The cost that actually shows up is bot management, and at Politico it now takes a quarter of hosting spend.

Google launched Search profiles in the US on 4 June 2026 and cut the eligibility threshold to 35,000 social followers on 16 August. Claiming one takes an afternoon and rewrites your Knowledge Panel, which is the entity record machines read when they decide who you are. The traffic it promises runs through Discover, so treat it as identity work rather than an audience plan.

Mostly, no. Assistants reproduce a publisher's distinctive reporting far more often than they credit it, and when they do name a source they frequently name the wrong one. A McGill audit published in March 2026 found Gemini covered the substance of a news article in 81% of responses while naming the outlet in 6%. Treat attribution as a variable you measure, not a benefit you receive.

Preferred Sources lets a reader pin your publication so your content surfaces more often, and carries a preferred badge, in Top Stories, AI Overviews and AI Mode. Since 20 August 2026 you can embed a two-line button that captures the selection without sending the reader to a Google settings page. Ship it, because it costs an afternoon. Do not forecast against it, because Google publishes no data that would let you.

Selling access to your content is lawful. Selling it to one buyer and locking every rival out is where the legal risk starts, and a filing made in Manhattan on 28 August 2026 is the first serious test of that line. Publishers negotiating AI deals should be reading the exclusivity clause more carefully than the price.

On 1 September 2026 the US Department of Justice told a New York court that training large language models on copyrighted text is fair use, and that the market dilution theory publishers rely on is deeply flawed. If Judge Stein agrees, the legal leverage behind every publisher licensing negotiation weakens at once. What survives is operational, not legal.

From 15 September 2026, Cloudflare blocks Training and Agent crawlers by default on pages that display ads, and judges multi-purpose crawlers by the strictest rule that applies to them. Any site blocking Training will also block Googlebot, Applebot and Bingbot. Two tools shipped in late August change how publishers should prepare, and Cloudflare's own executives describe the point of all this as negotiating leverage rather than revenue.

Google finished rolling out its AI opt-out control to every website in the world on 31 August 2026. It removes you from AI Overviews, AI Mode and AI summaries in Discover without touching your blue-link ranking. Almost nobody will use it, because Google reports AI impressions and withholds AI clicks, so no publisher can price what they would be giving up.

Publisher ad supply fell by up to 40% year on year in the second quarter of 2026, because AI answers and social feeds have removed the pageviews that generate ad calls. Prices have risen to partly compensate, with UK eCPMs up around 30%, but scarcity pricing recovers only part of the loss and nobody can say where the ceiling sits. The publishers who come through this are rebuilding around fewer, better-evidenced impressions rather than trying to buy the volume back.

There is no agreed way to count an ad that an AI agent reads on a user's behalf, and no agreed way to credit it when that user later buys. The IAB standardised the organic half of the problem in August 2026 and has an attribution framework due in November. Until that lands, the useful question for a buyer is not what a dashboard reports but what a vendor is willing to disclose.

The Prebid Sales Agent is free, open-source software that turns your inventory into an endpoint AI buyer agents can discover, negotiate with and buy from without a human in the loop. Prebid released it in January 2026 and it runs on the Ad Context Protocol. Almost no publisher can yet describe what happens when a buyer agent actually calls one, which is the real reason adoption is slow.

On one platform, yes, and by a wide margin: publishers with an OpenAI licence collected 48 per cent more ChatGPT citations per page than unlicensed publishers in June 2026. The premium disappears on Google and Perplexity, the causation is unproven, and the referral traffic that once came with a deal has almost entirely evaporated.

Collective AI licensing lets a collecting society license thousands of publishers' content to AI developers under one agreement, so publishers too small to negotiate directly can still get paid. The UK scheme run by PLS, CLA and ALCS opened for publisher opt-in in March 2026. No AI developer has been publicly named as a licensee yet, and until one is, the scheme is a supply side without a demand side.

Google's News AI programme pays a few hundred publications for rights it is acquiring from everyone else for nothing. The cash is real and budgeted, but the terms reported so far - two years, a non-disclosure agreement, a no-sue clause - quietly convert a regulatory win into a private settlement. Whether to sign turns on what else you have, not on whether the cheque clears.

Amazon Content Partners is the first AI-era programme built for the long tail rather than the top fifty publishers, and it is free to join. It is also not a licensing deal: it pays in affiliate commission, ad-stack access and hosting credits rather than in any share of AI revenue. Whether to join comes down to how much you think the access you are granting is worth.

Perplexity, Microsoft, Amazon, Google, Meta and OpenAI all now pay publishers something, but through four incompatible mechanisms: a share of subscription revenue, a fee per grounded answer, a flat licence, and benefits in kind. Only two of them pay a stated percentage. None of it is yet offsetting the search traffic it displaced.

ChatGPT Ads is no longer a test. It runs a relevance-weighted second-price auction across nine markets, sells on impressions, clicks and actions through a self-serve manager with no minimum spend, and appears on roughly a quarter of commercial prompts. Publishers supply the answers it is sold against and receive none of the money.

Technically, yes. Time is already selling FAQ-formatted ads inside the machine-readable copies of its pages that AI agents read, and charging a premium for them. The constraint is permission, not technology: on 11 August 2026 Perplexity blocked those ads from influencing its index, called the practice deceptive and warned that publishers who run them risk a lower trust score.

AWS WAF AI traffic monetization lets any publisher behind CloudFront put a price on an AI bot's request and collect it at the network edge, in stablecoins, with AWS taking no cut of the revenue. It launched on 15 June 2026 and it turns charging for crawler access from a vendor negotiation into a console setting. The harder question is whether the agents reading your site are built to pay.

There are now three ways into an AI answer: earn the citation, buy a labelled ad slot from the assistant, or place your brand inside the publisher content the assistant retrieves. Most brands are further out than they think. Webflow's audit of 2,000 sites found the median company appeared in 16% of relevant AI answers and was cited with a link only 6% of the time.

Serving markdown to AI bots reliably cuts the tokens a crawler spends on your page by 80 to 90 per cent. What it does not reliably do is get you cited more often, and one analysis of 1.6 million AI citations found no citation benefit at all. Markdown is worth doing as an efficiency and control measure, and only worth doing at all if you have decided what happens to the traffic once it arrives.

A stealth crawler is an automated tool that scrapes a site without declaring what it is or why it is there, typically by presenting as an ordinary browser on a residential IP address. On large news sites, roughly a quarter of all traffic is estimated to be stealth crawling that publishers never register as bot traffic at all. New York and Congress are both moving to outlaw it, and the objections to those bills are more serious than most publishers realise.

Content Telemetry is an open standard from the publisher-led SPUR Coalition that defines five events an AI system should report back every time it uses a publisher's content: retrieved, grounded, cited, displayed and engaged. Published in draft on 12 June 2026 with revised specifications due by 10 August, it is the first serious attempt to standardise what happens to content after the crawl. Its weakness is that the AI companies have to agree to send the data.

Trusted Server is IAB Tech Lab's open-source framework for moving ad execution out of the browser and into publisher-controlled edge infrastructure. The proof of concept has been open for publisher testing since February 2026, and publishers expect live production tests late in 2026 or early in 2027. It is the industry's formal admission that the browser can no longer be trusted to carry the monetisation layer.

Today it does. Integral Ad Science and DoubleVerify both bucket declared AI agents inside general invalid traffic, so those impressions are stripped out and not paid for. That classification has already cost at least one US publisher an entire buyer's budget, and better bot detection will not fix it.

No. Apart from Google's and Apple's crawlers, no major AI crawler executes JavaScript. GPTBot, ClaudeBot, PerplexityBot and the rest read the raw HTML your server returns and nothing else, so any content assembled in the browser is invisible to the assistants answering questions about your market.

AI assistants cite what they can retrieve, trust and extract cleanly, and that is no longer the same thing as what ranks. Ahrefs found only 38% of Google AI Overview citations came from top-10 pages in April 2026, down from 76% nine months earlier, and 86% of the most-cited sources are not shared between ChatGPT, Perplexity and AI Overviews. Citation has become its own discipline, and it has to be measured engine by engine.

Blocking is no longer a free option. Academic research now puts the cost of a robots.txt block at around 7% of weekly traffic within six weeks, while citation data shows blocking does not reliably keep you out of AI answers. The workable position in 2026 is selective: block the crawlers that take and give nothing, monetise the ones that carry real audience, and treat your CDN default as the decision that actually matters.

AI licensing pays a few dozen brand-name publishers real money and almost everyone else nothing. News Corp clears around $50m a year, Wiley booked $49m in its 2026 financial year, and the long tail is never offered a deal at all. Marketplaces and collectives arrived in 2026 to fix that, but the concentration has not gone away.

TollBit and Cloudflare charge AI bots for access. Ad injection earns from the read itself. In 2026 the access side stopped being a two-vendor question and became infrastructure, with Cloudflare, AWS and Akamai all billing crawlers natively and Cloudflare moving from per-crawl to per-use pricing. Here is what each model actually captures, and where all of them leak.

A nationally representative Ipsos study for the AOP finds an AI answer is only trusted as much as the publisher it cites, from above 90% when the source is trusted down to about 10% when it is not. The value has moved to the read, not the referral.

SEO earns a ranked link a human clicks. GEO earns a mention inside the answer an AI assistant writes, where a click is rare. By 2026 the two reward different pages: the overlap between Google's top links and the sources AI tools cite has fallen from around 70% to under 20%, so most publishers now have to win both.

A Wall Street Journal report says Reddit, USA Today, Reuters, Politico, The Economist and People Inc. are reconsidering how much access to give Google as AI Overviews cut referral traffic. The story underneath the threat is a pricing problem: publishers are read by Google's AI at full scale and paid almost nothing for the read.

No. Blocking the bots that scrape your content to train models - GPTBot, Google-Extended, ClaudeBot - has no effect on your Google search rankings, because none of them is the crawler that builds Google's index. The real risk is subtler: block the wrong bot and you vanish from AI answers instead, and blocking rarely stops your content being cited anyway.

Agentic shopping squeezes publisher affiliate revenue from both ends. When an AI agent researches and buys inside the conversation the publisher's link is never clicked, so no commission is earned; and on the purchases that still route through a link, networks led by Amazon are cutting the rate. The affiliate channel that funded a generation of review and recommendation sites is being quietly disintermediated.

AI content licensing marketplaces are intermediary platforms that let publishers license their work to AI companies without negotiating a separate deal with each one. They open a door for the long tail that bilateral agreements ignore, but they differ sharply on who pays, what they take, and whether they value training, citations or live reads. This is how the main marketplaces work and where they stop short.

Google Offerwall is a free tool in Ad Manager and AdSense that lets a visitor choose how to unlock content - watch a rewarded ad, answer a survey, make a micropayment or sign up - instead of hitting a hard paywall. Google built it because AI search summaries are cutting publisher traffic, and it now optimises timing with machine learning. It only earns from the humans who still arrive, though, so the agent reads that never render a page stay unmonetised.

Agentic advertising is programmatic media trading run by autonomous AI agents that plan, buy, sell and optimise campaigns while humans set only the goals and guardrails. For publishers it shifts the contest from filling ad slots to being discoverable and negotiable by buyer agents through a seller agent that represents your inventory in machine-to-machine deals. Expose structured, priced, machine-readable inventory now, or risk being invisible to the buyers doing the deciding.

GA4 added a native AI Assistant channel in May 2026, but it misses Perplexity and cannot see the 60 to 70 per cent of AI clicks that arrive with no referrer. To measure AI traffic properly you need a custom channel group, a maintained regex, and an honest account of what analytics still cannot show you.

Most publishers still earn nothing from AI bot traffic, even though AI crawlers now generate the majority of web requests and the visitors they send back convert far better than search. The value sits in the intent behind the read, not the request count. Here is what sets it and how to price your own inventory.

Google's AI Overviews now answer most informational queries on the results page, and the referrals that used to follow have fallen by roughly a third for many publishers. This is a structural loss, not a ranking dip, and the response is a three-part shift: defend high-intent visits, get cited by the AI, and earn from the agents now reading your pages directly.

Answer engine optimisation (AEO) is the practice of structuring content so answer engines - Google AI Overviews, ChatGPT, Perplexity, voice assistants - use it as the direct answer to a question. It overlaps heavily with GEO, and the useful distinction is that AEO is about winning the answer slot while GEO is about shaping the citations inside a generated response.

Applebot-Extended is the robots.txt control Apple gives publishers to keep their content out of Apple Intelligence and Siri training. It never crawls anything itself; it only governs how pages Applebot has already fetched can be used. Blocking it stops AI training use while leaving your site fully searchable in Spotlight, Siri and Safari.

TollBit is a two-sided marketplace that puts a paywall in front of AI crawlers, letting publishers charge bots per request while AI companies buy compliant access. It sits at the request layer, sets a price on the crawl, and settles the money. It does not monetise the answer the bot then generates - which is where the value increasingly lands.

Cloudflare pay per crawl is a marketplace that lets a publisher charge AI crawlers a set price for each request, enforced at the network edge with the HTTP 402 Payment Required response. On its second Content Independence Day, Cloudflare extended the idea towards pay per use and set a 15 September 2026 default that blocks training and agent crawlers on ad-supported pages.

Google-Extended is a robots.txt control token, not a crawler. It lets publishers stop their content training Google's Gemini and Vertex AI models with no Google Search penalty. What it cannot do is remove you from AI Overviews or AI Mode, because those are served from the search index by the ordinary Googlebot.

ClaudeBot is Anthropic's web crawler, the bot that collects public web content to help train the Claude models. Publishers can allow or block it independently in robots.txt, but the sharper question is economic: ClaudeBot currently reads more pages and returns fewer visitors than any other major AI crawler.

Query fan-out is the technique behind Google's AI Mode and other AI search engines: a single question is silently broken into dozens - or, in deep research modes, hundreds - of background searches, each pulling from different sources. For publishers it multiplies how often your content is read while collapsing the clicks you receive towards zero.

ChatGPT Search is the live web retrieval mode inside ChatGPT that answers a question with a synthesised response and a short list of cited links, instead of returning a page of results. For publishers it is the single largest source of AI referral traffic and the AI surface most likely to read their content, yet it returns a referral for only a fraction of what it crawls, and OpenAI has confirmed it will not share its new advertising revenue with publishers.

The crawl-to-refer ratio counts how many pages an AI platform crawls for every visitor it sends back. Cloudflare data puts the imbalance in the tens of thousands to one for the heaviest operators, against single digits for traditional search. It is the cleanest single number proving that AI reads publisher content far more than it returns traffic.

NLWeb is an open Microsoft project that turns any website into a conversational AI app and a Model Context Protocol server, so people and agents can query your content in plain language. It runs on data publishers already publish, Schema.org and RSS, and lets sites join the agentic web on their own terms. What it does not settle is how a publisher gets paid when the agent takes the answer and the reader never lands.

Schema markup helps AI systems read and understand your content, but the strongest 2026 evidence shows it does not directly cause AI citations. Authority, content quality and clean retrievable text are the real drivers, and thin or generic schema can even reduce visibility. Treat structured data as comprehension hygiene, not a citation lever.

Grok is xAI's AI assistant, built natively into X and available at grok.com, and by early 2026 it had become the third most-used chatbot in the United States. For publishers it is one of the most extractive AI surfaces in the market: it reads the open web and live X posts to answer questions, returns almost no referral traffic, and runs a retrieval crawler that routinely ignores robots.txt by presenting itself as an ordinary human visitor.

Whether AI systems can access your subscriber content depends almost entirely on how your paywall is built. Publishers using client-side JavaScript overlay paywalls are delivering the full article text to any requesting client before the subscription prompt fires - because the HTML leaves the server before JavaScript runs. Hard, server-side authentication is the only architecture that reliably stops both traditional AI crawlers and the newer generation of AI browsers.

Advertising inside LLM responses now takes three distinct forms: platform-native ads sold by the AI company, product-feed placements served inside shopping queries, and content-layer ads placed in the publisher content an assistant reads. Here is how each works, who gets paid, and what the 2026 landscape actually looks like.

A zero-click search is one that ends without the user clicking through to any website, because the answer is delivered on the results page itself. In early 2026 around two thirds of Google searches end this way, and the rate climbs past 80% when an AI Overview appears. The visit that used to fund the page is increasingly never made.

Information Agents are one layer of a complete agentic commerce stack Google announced at I/O 2026. Discovery, answer, basket, payment, execution - all Google owned, all terminating inside Google's environment. Here's where the publisher position sits, and why the contractual layer is the next twelve months of monetisation.

The Google-Agent user agent string can be spoofed. The IP range, reverse DNS, and Web Bot Auth signatures can't. Three verification methods, in increasing order of reliability, and why the cryptographic layer changes the publisher position more than the agent product launch itself.

Publisher training bot policies cover the first AI dependency. They don't cover the second. Here's the four week plan we're recommending to our publisher partners - and why the Q4 commercial conversations are being framed this summer.

At I/O 2026, Google announced Information Agents - persistent AI processes that scan the live web on a user's behalf. It's Live Search Agent traffic, productised, shipping this summer.

At I/O 2026, Google announced Information Agents - persistent AI processes that scan the live web on a user's behalf. It's Live Search Agent traffic, productised, shipping this summer.

Automated traffic is now more than half of the web, AI crawlers make tens of billions of requests a day, and publishers lose an estimated $2 billion a year to AI bots. Here are the key AI bot traffic statistics for 2026, with sources.

Advertising in AI answers can be brand safe, but the risks differ from display and depend on the model you use. Here is what changes in a generated answer, and which approach gives advertisers and publishers the most control.

The AI bots visiting your site fall into three groups: training crawlers, retrieval crawlers, and live agents. Here is a 2026 reference to the names you will see, from GPTBot to ChatGPT-User, and what to do about each.

A Live Search Agent is a bot an AI assistant sends to read a specific page in real time to answer a user's question. It is different from a training crawler, it never runs JavaScript, and it is invisible to standard analytics. Here is why it matters to publishers.

AI Overviews answer queries in place, and the referral traffic they remove is not coming back in its old form. Here is what the data shows, why the loss is structural, and the three-part response that protects revenue and opens a new line from AI agent traffic.

TollBit and Cloudflare pay per crawl charge AI bots for access. Ad injection earns from the read itself. Here is how the three models differ, which traffic each one captures, and why most publishers will end up using more than one.

AI licensing deals pay a small number of very large publishers well, and almost everyone else little or nothing. Here is why the headline agreements stay out of reach for most publishers, and what the uncompensated majority can do instead.

There are three ways to monetise AI bot traffic: license your content, charge crawlers for access, or inject paid contextual brand mentions into the pages AI agents read. Licensing suits a handful of large publishers. Here is how each model works, and which one captures the Live Search Agent traffic the others miss.

We launched a free domain audit this week. Before you request one, here's exactly what the report shows: what's real data, what's modelled, and why we're being transparent about the difference.

Google quietly added a new bot that bypasses robots.txt by design. And 23 major publishers are blocking the Wayback Machine, while their own journalists depend on it. Two developments every publisher commercial team needs to understand.

Your domain audit includes a brand share of voice table. Here's what the multiplier actually means, how to interpret high and low scores, and how to use the data in your next advertiser conversation.

Publishers are making editorial and commercial decisions on incomplete data. The fastest-growing segment of their audience doesn't appear in Google Analytics.

$15.02 average CPM from AI agent monetisation in live publisher accounts. Publishers with that same traffic on their sites are currently earning $0 from it.

ChatGPT, Claude, and Perplexity are now the first stop for millions of purchasing decisions. Here's what that means for publishers and brands.

AI retrieval events aren't bot visits. They're audience data: high-intent signals connected to real purchase decisions. The publishers who start reading them will understand their readership better than anyone.

A practical guide for content publishers ready to turn AI bot traffic into a new revenue stream - from integration to optimisation.

To understand the cost, you need to understand what publishers have spent two decades building on top of HTML.

Early data from Blankspace campaigns suggests AI-cited brand mentions drive purchase intent at rates significantly above traditional digital channels.