A contract clause can do what a crawler cannot: turn access into a permission. That was the worry when Meta proposed, in October 2026, to add language to its commercial terms letting it send bots to websites "associated with" a business account. Meta has since removed the word "train", moved the effective date, and then, on 5 November, paused the whole package with no new date. The clause is not in force today. It is also not withdrawn, and Meta has not said whether it will come back, which is why the wording and the legal arguments around it are still worth understanding.
Are Meta's new commercial terms in force?
No. On 5 November 2026 Meta removed the preview of the proposed terms and posted a Business Help Center update saying the changes "are now paused and will not take effect on November 20, 2026", according to Nieman Lab. Four documents were paused: the Meta Commercial Terms, Product Catalog Terms, Self-Serve Ad Terms and Ad Creative Generative AI Terms. Existing terms remain in place.
A Meta spokesperson told Nieman Lab the company is "pausing the changes as we continue to engage with stakeholders and take our time to get this right." PPC Land notes that Meta has not named the stakeholders, set a deadline, said which provisions it will rewrite, or linked the pause to any court order or regulator. A pause is not a repeal.
What did the proposed clause say?
The original text, reported by Nieman Lab on 15 October, authorised Meta to use automated software such as scrapers, bots and crawlers to access domains, pages and URLs "associated with your products or services", and said Meta could use that content to "provide, develop, improve, research, train" its products. Business accounts could not opt out before the effective date, which was then 30 October.
On 16 October Meta published a revision. According to Nieman Lab, it removed "train" and the phrase "Meta may use any content from such domains", narrowed the purposes to "provide, develop, improve, research, and maintain advertising and business products", and added security alongside integrity in the closing purpose. The scope, sites "associated with" a business account, stayed the same. The revised text did not mention robots.txt.
Why did experts say the revision did not settle it?
Three legal specialists quoted by Nieman Lab pointed at the same gap. James Grimmelmann, professor of digital and information law at Cornell Law School, said the revision did not address his concerns and that the scope language could still let Meta scrape entire sites. Matt Topic, a media and IP attorney at Loevy & Loevy, said "develop" could still cover training. Benjamin Sobel, a law professor at the University of Wisconsin, said robots.txt expresses a site owner's preference but does not by itself create an enforceable right to stop scraping.
The point is structural. If a publisher agrees in a contract that Meta may crawl, a robots.txt disallow can be argued to be a preference rather than a bar. That is an argument, not a ruling.
What does Meta say the terms were for?
Meta's position, set out by communications chief Andy Stone and a company spokesperson, is that the crawling language applied "only to our ads and business products", that it does not change how Meta uses web crawlers, and that it gives Meta no new rights that supersede robots.txt preferences. Meta also said it was folding its Ad Creative Generative AI Terms into the general commercial terms for clarity. The European Publishers Council replied that verbal assurances were not enough because the revised wording gave no binding guarantee, and Jason Kint of Digital Content Next called Meta's correction "a farce", as reported by The Next Web.
Who pushed back, and what has happened since?
The sequence matters because it shows what moved Meta. The dates below come from Nieman Lab and PPC Land.
On 19 October the European Publishers Council wrote to EU competition chief Teresa Ribera and tech chief Henna Virkkunen asking them to suspend the changes, and UK publishers raised the issue with the Competition and Markets Authority. On 23 October Meta moved the Commercial Terms and Self-Serve Ad Terms effective date to 20 November. In late October US senators Adam Schiff and Marsha Blackburn wrote to Mark Zuckerberg asking him to withdraw the terms. On 28 October Stone said Meta would complete a full consultation with "industry groups and leaders" before implementing, even if that ran past 20 November.
Courts and regulators then acted. PPC Land reports that a court in Oslo temporarily blocked the terms on 28 October and a court in Sao Paulo suspended the crawling clause on 3 November. The CMA opened an investigation on 30 October into whether Meta is abusing a dominant position by imposing exploitative terms on publishers and advertisers. Neither the court orders nor the CMA investigation has been reported as lifted. The News/Media Alliance's Danielle Coffey called the proposal "outrageous and probably illegal".
Which Meta crawlers does robots.txt control today?
Robots.txt remains the technical control Meta documents. Meta's developer documentation lists several crawlers, including Meta-ExternalAgent, Meta-WebIndexer, Meta-ExternalFetcher, Meta-ExternalAds and facebookexternalhit, each with a different job. The page on what Meta AI means for publishers maps these to Meta's products, and the AI crawler user agent list is the reference for current tokens.
Two cautions apply. A robots.txt rule is a request that compliant crawlers choose to honour, which is why robots.txt alone cannot stop AI crawlers. And Meta's documentation says some user-initiated fetchers may not follow robots.txt, so a disallow does not cover every Meta request.
What should publishers do while the terms are paused?
The risk has moved from imminent to latent, so the response is monitoring and preparation, not panic. Take legal advice before acting, because this article is general information and not legal advice.
First, keep the inventory. List every Meta business account, ad account and Business Suite login across your organisation, including those held by individual sections or commercial teams. If the clause returns, it attaches to business accounts.
Second, watch for the next notice. Meta says it will share "a further update when we have more to share". Record the date and wording of any new terms as shown to your own account, because wording can differ by edition and change quickly, as it did between 15 and 16 October.
Third, check your contracts. If you license content to AI companies, review whether exclusivity or usage restrictions could conflict with granting any crawling permission through account terms. The guide to selling exclusive AI access sets out where conflicts tend to arise.
Fourth, join the collective effort if it suits you. The pushback that produced the pause was coordinated through trade bodies and regulators, and they are the channel through which any consultation will run.
Can publishers control Meta's access at the edge?
A contract question and a technical question are separate. Even if terms granted Meta a licence, a publisher still decides what its servers return. Bot management at the CDN layer can identify requests from Meta's declared user agents, check them against published IP ranges, and then allow, throttle, block or serve a licensed or paid response. That does not decide whether Meta could claim a contractual right, but it makes access visible and leaves a record of what was requested and when.
Visibility matters because the original text allowed Meta to "create test credentials" to reach sites. Meta declined to tell Nieman Lab whether those would be used to get past paywalls or registration walls. Detecting undeclared or credentialled access is the hard problem described in the pages on stealth crawlers and AI crawlers and paywalled content. It is also where blankspace works, at the CDN edge, though the contractual question is for lawyers rather than infrastructure.
How does this fit the wider fight over AI access?
Meta's proposal tried to obtain crawl access through account terms, not through a licence negotiated with the publisher. The overview of AI content licensing explains why only a minority of publishers receive deals, and the question of whether to block or monetise AI crawlers is the wider frame. The backlash suggests that bundling a crawl permission into tools publishers depend on is hard to defend, which is a useful precedent for the next platform that tries it.
Frequently asked questions
Are Meta's new commercial terms in effect?
No. Meta paused the proposed changes on 5 November 2026, and its Help Center update says they will not take effect on 20 November. Existing terms remain in place, and Meta has set no new date.
Did Meta remove the AI training wording?
Yes, from the October proposal. Meta removed "train" on 16 October and limited the purposes to advertising and business products. Experts told Nieman Lab that "develop" and the broad scope could still cover training, and the whole package has since been paused.
Does robots.txt still stop Meta's crawlers?
Meta says it respects robots.txt preferences and that the terms did not override them. Robots.txt remains a voluntary signal, and Meta's documentation says some user-initiated fetchers may not follow it. It should be paired with server-side controls.
Could the clause come back?
It could. Meta has said only that it will keep consulting stakeholders and share more later, and has not said which provisions it will rewrite. Court orders in Norway and Brazil and a UK CMA investigation are reported as continuing.
What should publishers do now?
Keep an inventory of Meta business accounts, monitor for any re-issued terms and record the wording shown to your account, review AI licensing contracts for conflicts, and keep bot controls current at the server or CDN. Take legal advice before any decision to accept or leave.