Common Crawl works one step upstream of the AI labs. A nonprofit fetches billions of pages a month into a public archive, and labs then filter that archive into training sets such as C4, FineWeb and Dolma. CCBot is the crawler that does the fetching. It reads robots.txt, so a two-line rule opts a site out of future crawls, but copies already archived stay where they are. That gap between future access and past collection is the centre of the dispute between publishers and Common Crawl in 2026.
What is Common Crawl?
Common Crawl is a nonprofit that publishes an open repository of web crawl data, free for anyone to download. It has archived billions of pages every month since 2008, according to Press Gazette, and Search Engine Journal reported that its July 2026 crawl held 2.14 billion pages. The data is used by researchers, search startups and AI developers, which is what makes it different from a crawler operated by a single AI company.
It is also funded differently from a commercial crawler. Press Gazette reports that it is primarily funded by the Elbaz Family Foundation Trust, and The Atlantic reported (as relayed by PPC Land) that OpenAI and Anthropic each donated 250,000 dollars in 2023.
How does Common Crawl end up inside AI models?
Labs rarely train on the raw archive. They filter it into derived datasets, and Search Engine Journal names C4, RefinedWeb, RedPajama, Dolma and FineWeb among them. Two data points show how central the archive has been:
- OpenAI's GPT-3 paper listed filtered Common Crawl as about 60 per cent of that model's training mix, a figure the New York Times cited in its 2023 lawsuit against OpenAI.
- A Mozilla audit of language models released between 2019 and 2023 found that 64 per cent had trained on Common Crawl data in some form, as reported by Search Engine Journal.
Two cautions apply. Labs stopped disclosing their training mixes around 2023, so the current share is unknown. And being in the archive does not mean a page is in a model, because every training set filters the crawl differently.
What is CCBot and how does it behave?
CCBot is Common Crawl's crawler. According to Common Crawl's own FAQ, it checks robots.txt before fetching any page, honours the Crawl-delay directive, backs off when a server returns 429 or 5xx responses, and does not execute JavaScript or use cookies. Its requests can be verified through reverse DNS on crawl.commoncrawl.org, with current IP ranges published as JSON.
Because CCBot does not run JavaScript, it sees the HTML the server sends first. Any content a site delivers in the initial HTML response, including article text sitting behind a client-side overlay, is visible to it. That is the technical reason the paywall question below keeps recurring.
Does blocking CCBot remove content already in the archive?
No. A User-agent: CCBot group with Disallow: / in robots.txt stops future collection, and CCBot rechecks the file periodically. It does not delete earlier captures. Search Engine Journal's coverage of the publisher dispute states the point plainly: blocking stops future collection but does not affect content already in the archive.
Removal is a separate process, and it is the contested one. Common Crawl's executive director Rich Skrenta has said the archive format is immutable, so affected URLs are removed or filtered from later crawls and made inaccessible through its public tools and indices rather than edited out of existing files. Common Crawl's FAQ points publishers to its opt-out registry, which lists large news groups including the BBC, The Guardian, the Financial Times, the Washington Post, News Corp, DMG Media, Advance Publications, the Associated Press, Le Monde, Reuters and Hearst, plus a News/Media Alliance entry covering more than 900 news sites, per Press Gazette.
Why are publishers in dispute with Common Crawl?
Two events frame it. In November 2025, The Atlantic's Alex Reisner reported that Common Crawl's archives contained full text from paywalled outlets and that removal requests had not been fully honoured. As relayed by PPC Land, the New York Times sent a removal notice in July 2023, its court filing referred to at least 16 million unique records, and Reisner still found many of its articles. The same report said the Danish Rights Alliance was told in December 2024 that about half of its requested content had been removed. Common Crawl denied lying to publishers and denied that CCBot scrapes behind paywalls. The Atlantic's original article could not be accessed for this piece, so those details are second-hand.
In June 2026, Digital Content Next, a US trade body for digital publishers, sent a cease-and-desist letter dated 3 June to the Common Crawl Foundation. It demands that Common Crawl stop collecting copyrighted, paywalled or subscriber-only content from DCN members and remove what it already holds. DCN argues that copyright law is not an opt-out regime. Its lawyers are also examining whether Common Crawl's statements to publishers about removal were inaccurate or misleading. Skrenta declined to comment on the letter specifically. No court ruling or settlement had been reported at the time of writing.
How many publishers block CCBot?
Blocking is already mainstream among news publishers. Search Engine Journal, citing BuzzStream, reported that 75 per cent of top US and UK publishers block AI training crawlers. Citing HTTP Archive figures from Chris Green, it also reported that about 492,000 sites name CCBot in robots.txt and roughly 95 per cent of those mentions are blocks. Cloudflare has blocked AI crawlers by default on new domains since 1 July 2025, which pushes the default for many newer sites in the same direction.
Prevalence is not the same as effectiveness. Robots.txt is a voluntary standard, and a block only works if the crawler honours it. CCBot publishes its behaviour and states that it does, but a rule written today only shows up in next month's crawl.
Should your site block CCBot?
There is no evidence that quantifies what blocking or allowing CCBot does to a brand's visibility in AI answers, so this is a judgement about control, not a proven ranking lever. The considerations run in two directions.
When does blocking make sense?
Blocking fits publishers whose commercial model depends on exclusivity or on licensing revenue: subscription businesses, outlets negotiating AI licensing deals, and any group already litigating or preparing to. A block is also a low-cost signal of intent for future disputes, because it records that the publisher did not consent to collection from a given date. Publishers with a paywall that relies on client-side scripts should check what the first HTML response actually contains before assuming it is protected.
When does allowing or waiting make sense?
Allowing CCBot fits publishers whose goal is broad presence in what models know about their brand, topic or product, and who monetise through reach rather than access. The honest trade-off is that open-web presence in training data may help a model recognise an entity, but nobody has published reliable evidence of the size of that effect, and archived copies cannot be recalled later. A publisher that is undecided should treat the choice as one-way: allowing is reversible going forward, but collected data is not.
How do you block CCBot and check it worked?
Add this to robots.txt:
User-agent: CCBot
Disallow: /
Then check three things. Fetch the file as CCBot would see it, confirm that your CDN or bot-management layer is not serving a different robots.txt or challenge page to that user agent, and review your server logs for CCBot requests after the change, verifying any hits by reverse DNS rather than by user-agent string alone. Spoofed user agents are common, so the string on its own proves nothing.
A CDN-edge layer, the same place blankspace operates, can apply different rules to different crawlers without touching the origin. That makes it practical to block CCBot while treating retrieval agents and licensed partners differently.
Where does Common Crawl fit alongside retrieval and licensing?
Common Crawl is a training-side source. It does not decide whether a live assistant fetches your page when a user asks a question, and blocking CCBot does not change how Googlebot or retrieval agents such as ChatGPT-User behave. For the distinction, see how the major AI crawlers are classified in the AI crawler and bot user agent list, and why robots.txt alone has limits in the page on whether robots.txt can stop AI crawlers. Training access, retrieval access and paid licensing are three separate levers, and a sensible policy sets each one deliberately.
Frequently asked questions
Is Common Crawl the same as Google's crawler?
No. Common Crawl is an independent nonprofit whose archive anyone can download, while Googlebot crawls for Google Search. Blocking CCBot has no effect on Googlebot, and Google's AI controls are handled through separate tokens.
Will blocking CCBot remove my articles from AI models?
No. It stops future collection by CCBot only. Pages already archived stay in earlier crawls, and models already trained are unaffected. Removal from the archive depends on Common Crawl's own opt-out and filtering process.
Does Common Crawl scrape paywalled content?
The claims conflict. The Atlantic reported that full text from paywalled outlets appeared in the archive, while Common Crawl's executive director has denied that CCBot scrapes behind paywalls. CCBot does not execute JavaScript, so whether a paywall is visible to it depends on what the server sends in the first HTML response.
Should small publishers bother blocking CCBot?
It is a two-line change with no cost to search traffic, so many do. The reason to hold off is a deliberate strategy of maximum open-web presence. Either way, make the decision on purpose and record the date.
Is there a legal route to get content removed from Common Crawl?
Publishers can submit to Common Crawl's opt-out registry and request removal. DCN has gone further with a cease-and-desist letter arguing that copyright law is not an opt-out system. No ruling had been reported at the time of writing, so the legal position remains untested.