Insights ·
Cloudflare will grade bots on obeying robots.txt. Is your file saying what you mean?
The short version
On October 16, 2026, Cloudflare said it is continuously checking whether Verified bots honor robots.txt and HTTP 429 responses, and will give each bot an explainable trust band that customers can later use to decide how its traffic is handled. A day earlier, Cloudflare and OpenAI reported that a pilot using Cloudflare signals helped OpenAI’s search crawler find hundreds of millions of new pages while skipping missing, redirecting and duplicate URLs. Both make the same point: your robots.txt and your status codes are now read as instructions that bots are measured against, so they should say exactly what you mean.
What Cloudflare announced
In a post on October 16, 2026, Cloudflare described a system that continuously checks whether bots on its Verified bots list respect the preferences sites publish, mainly robots.txt and HTTP 429 (Too Many Requests) responses. Verified bots are bots that identify themselves accurately and securely, sign their requests with Web Bot Auth, and do only what they say they will do.
For robots.txt, the system follows RFC 9309 and looks for repeated patterns of ignoring a site’s rules rather than single violations. For 429s, it watches whether a bot slows down or pauses, or keeps the same request rate. The result is an explainable trust band that shows how consistently a bot respects preferences and which behavior informed the assessment. Ignoring 429s lowers a bot’s band; backing off raises it. Cloudflare says the band is not a permanent judgment.
Cloudflare is running the system internally for now while it builds a history of observations. It plans to let all customers choose the trust threshold that decides how Verified bot traffic is handled, and to publish scores on Cloudflare Radar. It gave no dates for either step.
SEOAST Impact Rating: A2, plan for it. The change comes from Cloudflare itself and affects how bots are judged against your rules, but nothing changes for your site today and there is no penalty for publishers. The Impact Rating is SEOAST’s own editorial rating, not a Cloudflare, Google or OpenAI severity level.
What the Cloudflare and OpenAI pilot found
On October 15, Cloudflare and OpenAI published results from a research pilot that started in July 2026. Cloudflare passed signals from its network to OpenAI’s search crawler to help it find useful pages; OpenAI evaluated and ranked those pages independently. With a participating cohort of about 1% of Cloudflare’s network, the pilot helped OpenAI discover and index hundreds of millions of new pages.
Much of the gain came from filtering out waste. The post names bots repeatedly requesting files that do not exist, sites that redirect almost any address to the homepage, and different addresses that serve the same content. Change signals tell the crawler when a page is worth another visit, and a single harmful page, such as a fake login form, can be kept out without penalizing the whole domain.
Two limits are stated plainly. A signal about a page does not grant permission to access it, so your robots.txt still governs. And the research uses metadata for search and discovery, not page content for model training. The next phase will test faster change signals and measure what happens after content is discovered. No general availability was announced.
Why this puts your robots.txt and status codes under a brighter light
Until now, robots.txt was a request that each bot chose to honor or not, and nobody published a record of which did. Cloudflare is turning compliance into a measured, explainable signal. That only works in your favor if your file says what you actually want. A blanket rule written years ago, a group for one crawler token that should cover another, or a file that returns an error page instead of rules can all be read by a well-behaved bot as permission or refusal you never intended.
The pilot makes the same point about URLs. The patterns it filtered out, missing pages that redirect to the homepage and duplicate addresses for one page, are ordinary technical SEO problems. A crawler with limited time per site now has an outside signal telling it to skip them.
What to check this week
None of this needs a new tool. It needs a careful read of files you already have.
- Open your robots.txt and confirm it returns a 200 with plain-text rules, not an HTML error page or a redirect to the homepage.
- Read each user-agent group as a bot would. AI companies publish separate tokens for different jobs, so decide per token, and check that no catch-all rule blocks a crawler you meant to allow.
- When you need a bot to slow down, answer with HTTP 429 rather than a 403 or an empty 200. A 429 is the signal Cloudflare says it is watching.
- Make missing pages return a real 404 or 410, not a redirect to the homepage.
- Consolidate duplicate addresses for the same page with redirects or a self-referencing canonical.
- If you use Cloudflare, review the bot and AI crawler controls you already have; Cloudflare points to Bot Management and AI Crawl Control as the place to manage this today.
What SEOAST checks, and what it does not
SEOAST does not monitor bot behavior, read your server logs or see your Cloudflare settings. SEOAST does not query ChatGPT, Perplexity, Gemini or Google AI Overviews, does not track citations and has no AI visibility score, so it cannot tell you whether any crawler obeyed your rules.
SEOAST reads your robots.txt when it audits a URL: it tests whether Googlebot may fetch the audited path and reports which AI crawlers, including GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot, are disallowed at the site root. From the server HTML of that one URL it also checks the status code and redirect hops, whether robots meta and X-Robots-Tag allow indexing, and whether the page declares a canonical. Those are the instructions bots will be measured against.
Questions
What did Cloudflare announce on October 16, 2026?
A system that continuously checks whether Verified bots honor robots.txt and HTTP 429 responses and gives each bot an explainable trust band. It is internal for now; Cloudflare plans to let customers set a trust threshold and to publish scores on Radar, with no dates given.
Does the Cloudflare and OpenAI pilot override my robots.txt?
No. The companies state that a signal about a page does not grant permission to access it, and that the research uses metadata for discovery, not page content for model training.
Do I need to change anything today?
Nothing is required. It is a good week to confirm your robots.txt says what you mean, that missing pages return 404 or 410, and that you use 429 when you need a bot to slow down.
Can SEOAST tell me whether a bot ignored my robots.txt?
No. SEOAST audits one URL from its server HTML and reads your robots.txt. It does not see bot traffic, server logs or Cloudflare data.