Apple runs two crawlers that share one name between them. Applebot fetches and renders your pages for Spotlight, Siri and Safari search. Applebot-Extended never requests a single URL. It is a signal rather than a spider - a secondary user agent whose only job is to decide whether the content Applebot already holds may be used to train Apple's foundation models. A publisher who adds a Disallow rule for Applebot-Extended in robots.txt opts out of that training pipeline while remaining fully indexed and discoverable across Apple's search surfaces.
That split matters more in mid-2026 than it did when Apple introduced Applebot-Extended in June 2024. On 8 June 2026, the same day Apple confirmed at WWDC26 that Siri now runs partly on Google Gemini models, Apple rewrote its "About Applebot" documentation to state plainly that crawled data may train the foundation models behind Apple Intelligence, Siri, Services and Developer Tools - and may also supply real-time context for AI-generated answers. Applebot-Extended is the lever publishers pull to stay out of the first of those uses.
What is Applebot-Extended?
Applebot-Extended is a control token, not a crawler. Apple's own documentation is explicit that it "does not crawl webpages" and is "used solely to determine how data already crawled by Applebot can be used". The data collection is done by the primary Applebot user agent. Applebot-Extended sits on top of that, letting a publisher say: you may index my pages, but you may not use them to train your AI models.
Introduced in June 2024, ahead of Apple Intelligence shipping as a product, it is Apple's answer to the same problem OpenAI, Google and Anthropic each addressed with their own opt-out tokens: separating the crawl a search engine needs from the far broader permission an AI trainer wants. The design assumption is that most publishers want to keep the search traffic and control the training use independently, so Apple made them two distinct signals rather than one.
How is Applebot-Extended different from Applebot?
Applebot is the crawler. It fetches, renders and indexes pages, processes JavaScript, CSS and XHR requests in a browser-like environment, and powers results in Spotlight, Siri and Safari. If you block Applebot in robots.txt, you disappear from those surfaces.
Applebot-Extended controls use, not access. Blocking it changes nothing about whether Apple can crawl or index you; it changes only whether your already-crawled content can feed Apple's foundation model training. This is why the two must be reasoned about separately. Disallowing Applebot removes you from Apple search. Disallowing Applebot-Extended keeps you in Apple search but out of Apple's training data. Publishers routinely want the second without the first.
One quiet but important detail from the updated documentation: if a site's robots.txt does not mention Applebot but does mention Googlebot, Applebot follows the Googlebot rules. That fallback does not extend to Applebot-Extended, which is only ever honoured when it is named explicitly. A publisher who has opted out of Google-Extended has not, by default, opted out of Apple's training use. The Apple token has to be written in by hand.
What changed in Apple's June 2026 documentation?
The 8 June 2026 revision formalised what had been implicit since Apple Intelligence launched. Three additions stand out for publishers.
First, training use is now stated outright. Crawled data "may also be used to help train Apple foundation models powering generative AI features", where earlier versions spoke only of search indexing. Allowing Applebot now means contributing to a training pipeline unless you opt out.
Second, Apple documented a real-time retrieval use that is separate from training. Applebot data "may be used to provide additional context and up-to-date content when AI models are used to generate output" - the example given is answering broad knowledge questions in Siri and Search, with output that "may include links to sources and websites used to help generate the answer". That is retrieval-augmented generation, and it is governed by a different control: the nosnippet tag, not Applebot-Extended.
Third, Apple added handling for paywalled content and non-HTML resources, plus a clarification that Applebot does not follow the crawl-delay directive and instead throttles itself based on server response. The net effect is that Apple now exposes a control set broadly comparable to Google's and Bing's, split across training, retrieval and access.
How do you block Applebot-Extended?
Blocking training use takes one rule in your robots.txt file, using Apple's documented syntax:
User-agent: Applebot-Extended
Disallow: /
Narrow the path if you only want to protect part of the site. This directive stops the training use and nothing else; per Apple, pages that disallow Applebot-Extended "can still be included in search results" through Spotlight, Siri and Safari.
The training opt-out does not cover the real-time retrieval use. To keep a page out of the context Apple draws on for AI-generated answers, apply the nosnippet meta tag in the HTML head, or the Applebot-specific variant:
<meta name="applebot" content="nosnippet">
For PDFs, images and other non-HTML files where a meta tag has nowhere to live, Apple now honours the X-Robots-Tag HTTP response header, scoped to its crawler:
X-Robots-Tag: applebot: nosnippet
Paywalled publishers have a fourth lever. Marking a page with the schema.org isAccessibleForFree property set to false keeps it eligible for search results while telling Apple not to use that content as context for AI output. All of these controls act at the page level; Apple does not support section-level exclusion via the hasPart property, so it is the whole page or nothing.
Does blocking Applebot-Extended remove you from Siri and Spotlight?
No, and this is the single most misunderstood point. Apple's documentation states that even if you disallow Applebot-Extended and tag content with nosnippet, "your content will remain discoverable through Spotlight, Siri, and Safari". The three controls - robots.txt for crawl access, Applebot-Extended for training use, and nosnippet for retrieval context - are independent levers that do not override one another. Only disallowing the primary Applebot user agent removes a site from Apple's index entirely. Publishers who want to protect their content from AI use without losing Apple search traffic can do exactly that, and most who act choose precisely this combination.
How many publishers block Applebot-Extended?
Adoption is meaningful but far from universal. A June 2026 analysis of 107 prominent sites with a parseable robots.txt found 31 blocking Applebot-Extended, a 29% rate and the fifth highest among the nine major AI crawlers measured. Blocking clustered among large news and media organisations - the same cohort, including The New York Times, The Guardian, CNN, Reuters and Condé Nast titles, that moved early against OpenAI's GPTBot after its August 2023 launch.
The strategic caveat, well documented by 2026, is that blocking a training crawler is not the same as controlling how your content surfaces. A BuzzStream study of roughly four million AI citations, published in April 2026, found that blocking AI crawlers rarely stops AI systems from citing publisher content, because models can draw on material indexed before the rule was added or reach it through search results rather than a direct crawl. An opt-out governs the training input. It does not govern the answer.
Applebot-Extended vs Google-Extended: how do they compare?
The two are close structural parallels and worth reading together. Both are training-only control tokens that a search company bolted on top of its main crawler so publishers could refuse AI training without refusing search. Google split Googlebot from Google-Extended; Apple split Applebot from Applebot-Extended. Neither token crawls; each only governs use of content the primary crawler has already fetched.
The differences are in scope and mechanics. Google-Extended governs training and grounding for Gemini and Vertex AI. Apple layers three signals rather than one - Applebot-Extended for training, nosnippet for real-time retrieval context, and isAccessibleForFree for paywalled material - giving finer control across distinct AI uses. Apple also applies the Googlebot-fallback rule to its primary crawler but not to Applebot-Extended, so opting out of Google's training does not carry over to Apple's. Publishers auditing their AI posture need to name each token explicitly; there is no universal "block AI training" switch.
Where this leaves publishers, and where blankspace fits
Applebot-Extended is a clean, well-designed control, and publishers should use it deliberately. But it is worth being precise about what it can and cannot do. It is a declared preference, honoured on trust, and it governs one use: training. The newer retrieval use - where Apple surfaces your content inside a Siri or Search answer, sometimes with a link, often without a click back - is governed by a separate honour-system tag, and the industry evidence is that opting out of the crawl does not reliably stop the citation.
That gap is the structural problem across every AI surface, not just Apple's. The read still happens, the answer still surfaces, and the publisher whose content fuelled it receives neither the visit nor, in most cases, any compensation. blankspace works on the other side of that gap: at the CDN edge, where the live-answer read can be detected as it happens and turned into a monetisable, attributable event rather than an unpaid training or retrieval input. Robots.txt and nosnippet decide what you permit. Edge enforcement decides what you can actually see and charge for. The two are complementary, and publishers taking Apple's opt-outs seriously are exactly the ones who benefit from closing the enforcement gap underneath them.
Frequently asked questions
Does Applebot-Extended crawl my website?
No. Applebot-Extended never requests a page. Apple's documentation states it "does not crawl webpages" and exists solely to determine how content already collected by the primary Applebot user agent may be used - specifically, whether it can train Apple's foundation models. The crawling is done by Applebot; Applebot-Extended is a use-permission signal layered on top.
Will blocking Applebot-Extended hurt my Apple search visibility?
No. Disallowing Applebot-Extended stops your content being used to train Apple Intelligence, but Apple confirms affected pages "can still be included in search results" via Spotlight, Siri and Safari. Search indexing and AI training are governed by separate robots.txt signals, so you can opt out of one without losing the other. Only blocking the primary Applebot user agent removes you from Apple search.
What is the difference between Applebot-Extended and the nosnippet tag?
They control two different Apple uses. Applebot-Extended governs whether your content trains Apple's foundation models. The nosnippet meta tag governs whether your content is used as real-time context when Apple generates an AI answer in Siri or Search. Apple's June 2026 documentation treats these as distinct pipelines, so a publisher who wants to opt out of both needs to set both signals.
Does opting out of Google-Extended also opt me out of Apple's AI training?
No. The tokens are separate and must each be named explicitly. Apple's crawler falls back to Googlebot rules only for the primary Applebot user agent, and that fallback does not apply to Applebot-Extended. If your robots.txt blocks Google-Extended but never mentions Applebot-Extended, Apple can still use your content for training.
If I block Applebot-Extended, can Siri still cite my content in answers?
Potentially, yes. Blocking Applebot-Extended addresses training use, not real-time retrieval, and 2026 research on roughly four million AI citations found that blocking training crawlers rarely stops AI systems from citing content already indexed or reachable through search. To limit retrieval use specifically you need the nosnippet tag, and even declared opt-outs are honoured on trust rather than technically enforced at the point the answer is generated.