robots.txt for AI Crawlers

robots.txt for AI crawlers is the practice of using the robots.txt file at a domain’s root to allow or block individual AI crawlers by user-agent token — typically separating training crawlers from the search and retrieval crawlers that determine whether a site can be cited.

robots.txt for AI Crawlers in Practice

The standard setup treats training and search as separate decisions: disallow the training crawlers (GPTBot, ClaudeBot, Google-Extended) while allowing the search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) so the site stays eligible for AI citations ([DataImpulse](https://dataimpulse.com/blog/robots-txt-ai-crawlers/)). Three pitfalls recur. A robots.txt that predates AI crawlers can block them through a catch-all disallow rule ([Vercel](https://vercel.com/kb/guide/agentic-commerce-readiness)). User-initiated fetchers such as ChatGPT-User and Perplexity-User may not follow robots.txt at all. And a CDN can override the file: one open-source project’s July 2026 audit found its site serving Cloudflare-managed content that disallowed ClaudeBot, GPTBot, Google-Extended, and others, regardless of the project’s own intent ([GitHub issue](https://github.com/MarkusNeusinger/anyplot/issues/9633)).


How to Audit Your robots.txt for AI Crawlers

This is an audit rather than a metric. Read the live file at yourdomain.com/robots.txt, then compare it against server logs to confirm blocked bots stop arriving and allowed ones appear. Verify identities against each operator’s published IP ranges, since user agents can be spoofed. Free auditing tools check roughly 20 or more AI user-agent tokens against a site’s file.


Why robots.txt for AI Crawlers Matters

robots.txt is the one control that all the major compliant AI crawlers document and honor, which makes it the real decision point between training exposure and search visibility. Errors are silent: a misconfigured file doesn’t break anything visibly — you just stop appearing in AI answers.


Terms Related to robots.txt for AI Crawlers

GPTBot ·
OAI-SearchBot ·
Google-Extended
llms.txt


FAQ

Does blocking AI crawlers hurt my Google rankings?
Blocking tokens like GPTBot or Google-Extended doesn’t affect Google Search; Google states Google-Extended is not a ranking signal. Blocking Googlebot, however, would remove you from Google’s index.