ClaudeBot

ClaudeBot is Anthropic’s web crawler for collecting public web content used in model development. Anthropic runs two other bots alongside it: Claude-User, which fetches pages when a user asks Claude about a link, and Claude-SearchBot, which indexes content to improve search result quality.

ClaudeBot in Practice

In July 2024, ClaudeBot visited the repair site iFixit about a million times in 24 hours; the crawling stopped after the site added ClaudeBot to its robots.txt ([The Register](https://www.theregister.com/2024/07/30/taming_ai_content_crawlers/)). That one incident shows both sides of the picture: crawl volume can be heavy, and the bot honors robots.txt. Anthropic’s documentation states that its bots respect robots.txt directives, supports the non-standard `Crawl-delay` extension for throttling (for example, `Crawl-delay: 1`), and cautions that blocking its IP addresses may not work as a lasting opt-out because it stops the bots from reading robots.txt at all ([Anthropic](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)). The older tokens Claude-Web and anthropic-ai are deprecated, and ClaudeBot honors any rules previously set for them.


How ClaudeBot Activity Is Tracked

Filter server logs for the ClaudeBot, Claude-User, and Claude-SearchBot user agents separately, since each signals a different purpose. A spike from ClaudeBot suggests training-oriented crawling; Claude-User requests mean a person asked Claude about a specific page.


Why ClaudeBot Matters

Anthropic notes that disabling Claude-SearchBot or Claude-User may reduce a site’s visibility in Claude’s search and user-directed answers — so a blanket “block Anthropic” rule trades training exposure for retrievability. Decide per bot, not per company.


Terms Related to ClaudeBot

GPTBot ·
PerplexityBot ·
robots.txt for AI Crawlers
LLM Visibility


FAQ

Should I block ClaudeBot?
That depends on whether you object to training use. Blocking ClaudeBot addresses training collection; Claude-SearchBot and Claude-User are separate controls that affect search visibility.