Claude-SearchBot
| Vendor | Anthropic |
| Type | Search-index crawler |
| robots.txt token | Claude-SearchBot |
| JavaScript rendering | No — HTTP-only fetcher |
| Honors robots.txt | Yes |
| Vendor docs | support.anthropic.com |
User-Agent strings
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; +https://anthropic.com/claude-searchbot)Purpose
Builds the Claude search index — the corpus Claude consults when a user asks a question with the web-search tool enabled. Roughly the Anthropic equivalent of OpenAI's OAI-SearchBot.
The three-bot framework (ClaudeBot · Claude-User · Claude-SearchBot) gives Anthropic the most granular crawler taxonomy of any AI vendor in 2026. Each token has a distinct purpose:
ClaudeBot→ training (HTTP only)Claude-SearchBot→ search index (HTTP only)Claude-User→ live retrieval (JS-capable)
How to allow / block
To appear in Claude's search-tool results:
User-agent: Claude-SearchBot
Allow: /To exclude:
User-agent: Claude-SearchBot
Disallow: /Quirks
- Newer than ClaudeBot; some older robots.txt files only block the
consolidated training token and leave Claude-SearchBot unconfigured (which is fine — it allows indexing for Claude's web-search results).
- HTTP-only. The same SSR caveat applies as for GPTBot — pages that
rely on JS to materialize content are invisible to it.