ClaudeBot
| Vendor | Anthropic |
| Type | AI training crawler |
| robots.txt token | ClaudeBot |
| JavaScript rendering | No — HTTP-only fetcher |
| Honors robots.txt | Yes |
| Vendor docs | support.anthropic.com/en/articles/8896518 |
User-Agent strings
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)Older / deprecated tokens (no longer in production):
anthropic-ai/0.x(deprecated July 2024)Claude-Web(deprecated July 2024)
Purpose
Collects public web content for training future Claude models. Distinct from Claude-User (live retrieval) and Claude-SearchBot (search index). Anthropic publicly confirmed in mid-2024 that ClaudeBot consolidates the previous anthropic-ai and Claude-Web bots into one identity.
Network identity
- Reverse-DNS hostname pattern:
*.anthropic.com - IP ranges: documented in Anthropic's support article; subject to
change. Verify by rDNS.
In our audit
The most-singled-out AI bot in our 100-site sample:
- Blocked at 62 sites as part of the AI-bot cluster.
- **Blocked at 6 additional sites alone** while the other AI bots
were allowed: ebay.com, ebay.de, ebay.co.uk, kleinanzeigen.de (the four eBay properties), nike.com, canadiantire.ca.
The "ClaudeBot only" pattern at the eBay properties strongly suggests an explicit organizational policy rather than a one-click WAF preset. A WAF preset would block training bots from multiple vendors as a group; the ClaudeBot-only block is hand-rolled.
Our v3 rendered audit further qualified this finding: when a real Chromium arrives at any of the five sites with any bot UA, the page fails to render. The "ClaudeBot only" block in v2 was the visible tip of a broader anti-JS-bot stance at those sites.
How to allow / block
To opt out of Claude training:
User-agent: ClaudeBot
Disallow: /To opt out of training but allow live retrieval (recommended for ecommerce):
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-User
Allow: /Quirks
- The consolidation of
anthropic-ai,Claude-Web→ClaudeBotin
July 2024 means older robots.txt files blocking those names should be updated.
- Pure HTTP fetcher — no JS execution.
- Anthropic publishes a three-bot framework
(ClaudeBot / Claude-User / Claude-SearchBot) with distinct purposes. Many sites block ClaudeBot as if all three are one bot.