Insights ·
What is __usage_ai-training? The shared robots.txt AI training opt-out
The short version
__usage_ai-training is a shared robots.txt policy token that lets a site state, in one group, whether its content may be used to train AI models. Google says it intends to support it in early 2027, Microsoft Bing will honor it from February 1, 2027, and OpenAI has said it will honor the shared signal too. It states a preference; it does not block crawling, and Google says the matching Google-Extended control does not affect Search inclusion or ranking. The practical step is to decide your training preference on its own, then write it down explicitly, because a User-Agent: * group does not count.
What is __usage_ai-training?
On October 30, 2026, Kevin Kelley of Anthropic published an IETF Internet-Draft, “Machine-Readable Training Preference Signal”. It adds no new robots.txt grammar. It defines a policy token, __usage_ai-training, that goes in an ordinary User-Agent line. A Disallow rule under that token states a preference against AI training for the matching paths, and an Allow rule states a preference in favor. Path matching follows RFC 9309, so the longest matching rule wins.
Three details are easy to miss. The token is not a crawler name and not a crawling directive: crawlers still follow their own groups. A User-Agent: * group must not be read as matching it, so a blanket rule says nothing about training. And if no group names the token, no preference is expressed. The draft is an individual submission with no formal IETF standing yet.
SEOAST Impact Rating: A2, plan for it. The platforms confirmed it, but it only states a preference, it does not change crawling or Search, and support starts in 2027. The Impact Rating is SEOAST’s own editorial rating, not a Google severity level.
Who has said they will honor the AI training token, and when?
Google updated its common crawlers page on October 30 to say it intends to support the shared machine-readable training preference signal in early 2027. John Mueller added that if a site opts out with the token and does not set Google-Extended, Google will treat it as a Google-Extended disallow, and that existing company-specific controls will keep working alongside the shared signal.
Microsoft Bing published the same day. It will honor AI training preferences in robots.txt through the same token from February 1, 2027, and it adds page-level controls: an ai-training:disallow value in a Bingbot meta tag or X-Robots-Tag header, plus ai-grounding:disallow and ai-grounding:disallowextended for grounding. From March 1, 2027, Bing stops reading NoArchive and NoCache as training or grounding preferences.
OpenAI has said it will honor the shared signal and calls the effort voluntary, as reported by Search Engine Roundtable and Thurrott. The draft itself came from Anthropic. Not everyone agrees on the details: Nate Hake, who takes part in the working group, says significant disputes remain.
Does __usage_ai-training affect Google Search?
Not according to Google. Its documentation says Google-Extended does not impact a site’s inclusion in Google Search and is not used as a ranking signal, and an opt-out through the shared token is treated as a Google-Extended disallow. Search Console’s separate control for generative AI features in Search is unchanged; that is a different decision with different costs.
Bing is the one to watch. Its new grounding preference is separate from training, and Microsoft says blocking grounding may reduce visibility in its AI experiences, while traditional Bing rankings stay the same.
How do I add the AI training preference to robots.txt?
Add a group that names the token and lists the paths you mean. The draft’s own example is User-Agent: __usage_ai-training, then Disallow: /, then Allow: /press-kit/. That states a preference against training for the whole site except a press kit. To state a preference in favor, use Allow: / instead.
Keep the groups you already have for named crawlers. The draft says more specific expressions take precedence over general ones, and Google says its company-specific controls continue to work. Nothing is enforced until 2027, but writing the group now makes your intent explicit before those dates.
What to check on your site this week
These checks come from the draft and from the Google and Bing announcements.
- Decide your AI training preference separately from your search and answer-engine preferences.
- List the named training crawlers you already block or allow, and make the new group say the same thing.
- Do not rely on a User-Agent: * group to express a training preference; it does not count.
- If you use NoArchive or NoCache to keep content out of Bing’s AI features, plan the move before March 1, 2027.
- Re-test robots.txt after any edit, so a typo in one group does not change what answer-engine crawlers can read.
What SEOAST checks, and what it does not
SEOAST audits one URL from its server HTML. Its robots.txt check tests the audited path for Googlebot. Its AI crawler access check is scored: it tests whether robots.txt lets OAI-SearchBot, ChatGPT-User, Claude-SearchBot and PerplexityBot read the site root and the audited path. Training crawlers such as GPTBot, ClaudeBot and Google-Extended are reported but never move the score, which matches the split these providers are now making between training and answers.
The shared training token is not part of that check today. SEOAST does not query ChatGPT, Perplexity, Gemini or Google AI Overviews, does not track citations and does not read your analytics.
Questions
What is __usage_ai-training?
A robots.txt policy token from an October 30, 2026 IETF draft. A group that names it states your preference for or against AI training, path by path.
Does __usage_ai-training block AI crawlers?
No. The draft says it is not a crawling directive. To stop a crawler fetching pages, use that crawler’s own User-Agent group.
Does opting out of AI training affect Google Search?
Google says Google-Extended does not affect Search inclusion or ranking, and it treats an opt-out through the shared token as a Google-Extended disallow.
When will Google and Bing honor the AI training token?
Google says early 2027. Microsoft Bing says from February 1, 2027.
Sources
- Machine-Readable Training Preference Signal (draft-kelley-aipref-training-preference-signal-00) — IETF Datatracker
- Google’s common crawlers: Google-Extended — Google for Developers
- Crawling documentation changelog — Google for Developers
- Supporting a shared AI training control for websites — Google Search Central Blog
- Your content, your choice: clearer preferences for training and grounding — Microsoft Bing Blogs
- Google Also Adopts New AI Training Control — Search Engine Roundtable
- Anthropic, Google, Microsoft, and OpenAI Agree to Let Sites Opt Out of AI Training — Thurrott