ClaudeBot
Collects public web content for training future Claude models. Anthropic consolidated the older anthropic-ai and Claude-Web tokens into this single ClaudeBot identity in July 2024.
Specs
| Vendor | Anthropic |
| Category | MEMORY |
| robots.txt token | ClaudeBot |
| Renders JavaScript | HTTP only |
| Honors robots.txt | yes |
| Reverse-DNS pattern | *.anthropic.com |
| IP-range source | https://claude.com/crawling/bots.json |
User-Agent string
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)Considerations
- Pure HTTP fetcher — no JS execution.
- The most-singled-out AI bot in our May 2026 audit: six sites block only ClaudeBot while serving every other AI bot normally. Four of the six are eBay properties.
- Older robots.txt files that block anthropic-ai or Claude-Web should be updated — those tokens are no longer in use.
robots.txt recipe
User-agent: ClaudeBot
Disallow: /Sources: Anthropic · Crawler FAQ · Anthropic IP ranges