Insights ·
Cloudflare’s “Disallow AI Training” and Google’s AI opt-out: what changed for GEO in September 2026
The short version
Two controls that used to be tangled are now separate. Cloudflare’s Disallow AI Training setting (September 15) tells Google and Apple not to train on your content while their search crawlers keep indexing you. Google’s Search Console control (worldwide since August 31) decides whether you appear in AI Overviews and AI Mode at all. Opting out of training does not remove you from AI answers, and opting out of AI answers does not change your regular rankings. Check what your robots.txt says now, because many Cloudflare sites were migrated to the new setting automatically.
What Cloudflare changed on September 15
Cloudflare added a Disallow AI Training option to its crawler controls. Instead of blocking a crawler outright, it writes a no-training preference into robots.txt using the tokens Google and Apple publish for that purpose: Google-Extended and Applebot-Extended. Googlebot, Applebot and Bingbot keep crawling for search.
The distinction matters because Google and Apple each use one crawler for search and training. Blocking Googlebot to stop training also takes a site out of Google Search. The -Extended tokens are how those companies let a site say yes to one and no to the other.
- Sites that had chosen Block, or Block on pages with ads, under the older training settings were moved to Disallow AI Training automatically. New ad-monetized domains get it by default.
- Training crawlers that run separately from search, from Amazon, Anthropic, Meta and OpenAI, are still blocked outright.
- Bing has no robots.txt no-training token yet. Microsoft’s support is targeted for early 2027.
Training, AI search and Google’s AI features are three separate decisions
This is where most robots.txt files go wrong. Each decision has its own control, and none of them implies the others.
- Model training. Controlled by training tokens: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot and others. Blocking these does not remove you from search or from AI search answers.
- AI search retrieval. Controlled by the crawlers answer engines fetch with: OAI-SearchBot for ChatGPT search, PerplexityBot, Claude-SearchBot. Block these and those engines cannot cite the page.
- Google’s AI Overviews and AI Mode. These draw on Google’s search index, so Google-Extended does not control them. The Search Console generative AI control does.
What Search Console now shows, and what it leaves out
Since August 31 every Search Console property can open generative AI performance reports covering AI Overviews, AI Mode and AI Overviews in Discover. They report impressions and pages, broken down by country, device and date. They do not include clicks.
The same rollout added a control that lets a site stop appearing in, and grounding, those AI features. Google says the setting is not used as a ranking signal outside the AI features, but a site that opts out gives up the impressions and traffic those features send.
What to check this week
Five checks, in order. None takes more than a few minutes.
- Open yoursite.com/robots.txt. If you use Cloudflare, confirm whether Google-Extended and Applebot-Extended rules now appear there, and that they reflect a decision you actually made.
- Make sure no rule blocks the AI search crawlers you want to be cited by: OAI-SearchBot, PerplexityBot and Claude-SearchBot. A copied wildcard block is the usual culprit.
- Open the generative AI report in Search Console and note this month’s AI Overviews and AI Mode impressions for your key pages. That is your baseline.
- Decide deliberately on the Search Console AI control. Leaving it on is the default for a reason; turning it off is a business decision, not an SEO fix.
- Audit the pages you most want cited. Crawl access is only the first gate; content that exists before JavaScript runs, clean structure and direct answers come next.
Where SEOAST fits
SEOAST reads your robots.txt on every audit and reports which AI crawlers it blocks, including Google-Extended, Applebot-Extended, GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot and PerplexityBot. It reports them as facts rather than faults, because blocking training is a legitimate choice. It then checks the rest of what a page needs to be quoted.
SEOAST does not query ChatGPT, Perplexity, Gemini or Google AI Overviews, and it does not read your Search Console data. Use the Search Console report for what Google shows; use SEOAST for whether each page is ready to be shown.
Questions
Does blocking Google-Extended remove my site from AI Overviews?
No. AI Overviews and AI Mode draw on Google’s search index. Google-Extended covers use of your content for Gemini models. The control for AI Overviews and AI Mode is in Search Console.
Will Cloudflare’s Disallow AI Training setting hurt my Google rankings?
It is designed not to. It leaves Googlebot, Applebot and Bingbot free to crawl for search and only adds no-training preferences for Google-Extended and Applebot-Extended.
Can I see clicks from AI Overviews in Search Console?
Not in the generative AI reports. They show impressions and pages by country, device and date, with no click data.
Does Bing honor a no-training rule yet?
Not yet. Microsoft’s support for a robots.txt no-training preference is targeted for early 2027.
Sources
- Cloudflare Lets Sites Disallow AI Training Without Blocking Googlebot — Search Engine Journal
- Google Search Console AI performance reports and Search generative AI control rolling out globally — Search Engine Land
- Introducing Search Generative AI performance reports in Search Console — Google Search Central Blog