SEARCH · Anthropic

Claude-SearchBot

Claude's web search index

Builds the Claude search index — the corpus Claude consults when a user asks a question with the web-search tool enabled. The Anthropic equivalent of OAI-SearchBot.

Specs

VendorAnthropic
CategorySEARCH
robots.txt tokenClaude-SearchBot
Renders JavaScriptHTTP only
Honors robots.txtyes
Reverse-DNS pattern*.anthropic.com
IP-range sourcehttps://claude.com/crawling/bots.json

User-Agent string

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; +https://anthropic.com/claude-searchbot)

Considerations

  • Newer than ClaudeBot. Some older robots.txt files only block the training token and leave Claude-SearchBot unconfigured (which means it can index, often the desired outcome for visibility).
  • HTTP-only — same SSR caveat as for GPTBot and OAI-SearchBot.

robots.txt recipe

User-agent: Claude-SearchBot
Allow: /
Related bots: ClaudeBot · Claude-User · Googlebot

Sources: Anthropic · Crawler FAQ

← Back to directory