AI Crawler Access and robots.txt

What robots.txt does

A robots.txt file at your site root tells crawlers which pages they are allowed to request. It is read before any page is fetched, so a disallow rule keeps a bot out entirely. This is different from a noindex meta tag, which lets the bot read the page but keeps it out of search results.

Each bot has its own token

Every crawler announces itself with a user agent string. You can allow or block each one separately. Common examples:

• Googlebot — Google Search
• OAI-SearchBot — ChatGPT search (separate from GPTBot, which governs training)
• PerplexityBot — Perplexity's index
• Claude-SearchBot — Claude web search
• Bingbot — Bing and Microsoft Copilot
• Google-Extended — Gemini model training (not search)

Why sites accidentally block AI crawlers

Many robots.txt files were written years ago with a "disallow all unknown bots" rule. When AI crawlers arrived, those rules blocked them without anyone noticing. A common pattern is allowing Googlebot and Bingbot while blocking everything else.

SEOAST reads your robots.txt and reports which AI crawlers it blocks. That report is factual, not a verdict: blocking them is a legitimate choice, and the audit does not score you down for it.

Training versus search

Several AI companies publish separate tokens for search and for training. For example, OpenAI uses OAI-SearchBot for ChatGPT search results and GPTBot for model training. Blocking one does not block the other. Google-Extended controls Gemini training but does not remove pages from Google Search or AI Overviews.

What SEOAST checks

The robots_txt and ai_crawler_access checks fetch your robots.txt and report which rules apply to the URL you audited. If a disallow blocks the page for Googlebot, the audit fails that check. If it blocks an AI crawler, that is reported separately.