Opt-Out Tokens: Decline AI Training, Keep Search

Opt-out tokens decline AI training per operator without leaving search: Google-Extended, Applebot-Extended and noai opt out of training, not search.

What are opt-out tokens, and which operators offer them?

Opt-out tokens are robots.txt product tokens that decline AI training per operator while its search crawler keeps indexing and citing you. Two exist as of 6 July 2026 — Google-Extended and Applebot-Extended — among the 41 crawlers and policy tokens in this site's AI Crawler Registry (AGENTS WELCOME, 2026). Google-Extended withholds crawled content from training future Gemini models and from grounding them at prompt time; it "doesn't have a separate HTTP request user agent string" (Google, 2026). Applebot-Extended "does not crawl webpages" and opts content out of training the foundation models behind Apple Intelligence (Apple, 2026). A Disallow for either blocks no traffic — it governs what the operator may do with content already fetched.

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

Does a training opt-out remove a site from search?

No — a training opt-out is not a search block. Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal" (Google, 2026); pages disallowing Applebot-Extended "can still be included in search results" (Apple, 2026); OpenAI documents GPTBot (training) and OAI-SearchBot (search) as independent settings (OpenAI, 2026). Blocking the search crawler instead — Googlebot, Applebot or OAI-SearchBot — removes you from that engine's results and its AI answers: sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers" (OpenAI, 2026).

What do the noai and noimageai directives do?

The noai and noimageai directives signal at page level that text and images should not be used for AI; DeviantArt introduced them on 11 November 2022 as a robots meta tag and an X-Robots-Tag header (DeviantArt, 2022). They are adoption-dependent: honoring them is each operator's choice, nothing enforces them at the edge, and none of the 41 registry entries documents a noai opt-out (AGENTS WELCOME, 2026) — a statement of intent, not a control.

<meta name="robots" content="noai, noimageai">

How do you opt out a single AI crawler in robots.txt?

Name the crawler's product token in its own group and disallow it — User-agent: GPTBot plus Disallow: / for OpenAI training, User-agent: ClaudeBot for Anthropic; 35 of the 41 registry entries document a robots.txt Disallow as their opt-out mechanism (AGENTS WELCOME, 2026). RFC 9309 sets two rules: a crawler obeys the group matching its product token and falls back to * only "if no matching group exists", and the rules "are not a form of access authorization" (IETF, 2022). A Disallow placed only under * never reaches a crawler with its own group, and compliance stays the crawler's choice — seven registry entries, Bytespider and Perplexity-User among them, are documented as not honoring robots.txt. For enforcement, pay-per-crawl answers at the edge.

Opt-out tokens — frequently asked questions

Does Google-Extended affect Google Search?

No. Google states that Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal"; disallowing it withholds content from Gemini training and grounding only. It is a training opt-out, not a search block.

Does Applebot-Extended block Applebot?

No. Applebot-Extended does not crawl webpages; it is a token that opts content out of training Apple's foundation models, and pages that disallow it can still be included in search results. Blocking Applebot itself removes you from Spotlight, Siri and Safari search.

Can I keep ChatGPT search but refuse OpenAI training?

Yes. OpenAI documents GPTBot (training) and OAI-SearchBot (search) as independent robots.txt settings: allow OAI-SearchBot to stay in ChatGPT search answers and disallow GPTBot to opt out of training.

Can I opt out of Anthropic training but keep Claude search?

Yes. Anthropic documents ClaudeBot (training), Claude-SearchBot (search) and Claude-User (fetches on a user's behalf) as separate user agents that honor robots.txt: disallow ClaudeBot in its own group and leave the other two allowed.

Why does a Disallow under User-agent: * not reach GPTBot?

Because RFC 9309 makes a crawler obey the group that matches its product token and fall back to the wildcard group (User-agent: *) only if no matching group exists. If GPTBot has its own group, rules under User-agent: * are invisible to it — put the Disallow in the GPTBot group.

Related: Google-Extended and other directives defined · robots.txt-AI blocks AI crawlers per user-agent · each crawler's opt-out mechanism and block-vs-allow rationale · weigh the trade-off in should you block AI? · back to AI access economics.

← Access Economics · .md