Bot reference catalog

One-page summaries of 30 common search, AI/LLM, and social-preview crawlers.

← bot index · audit index

Claude-User

Vendor
Anthropic
Type
AI live-fetch
JavaScript rendering
yes
Honors robots.txt
yes
VendorAnthropic
TypeLive retrieval / on-demand fetcher
robots.txt tokenClaude-User
JavaScript renderingYes (headless browser via BrowserBase-style infrastructure)
Honors robots.txtYes
Vendor docsdocs.anthropic.com/en/docs/agents-and-tools/web-search

User-Agent strings

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +https://anthropic.com/claude-user)

Purpose

Live retrieval bot — fetches URLs in real time when a Claude user asks a question that requires current web information, or when Claude's agent tools (browser, search) follow a link. Each request corresponds to an active user query and a user-visible response.

Distinct from ClaudeBot (training) and Claude-SearchBot (index). This three-bot framework is Anthropic's clearest signal that site owners can opt out of training while still participating in Claude's real-time product features.

Network identity

reverse-DNS.

In our audit

Claude-User was not part of our v2 HTTP matrix (we tested ClaudeBot instead, the training-side bot). It was tested in v3 with a real Chromium browser at 15 sites.

At walmart.com and samsung.com, Claude-User received the same full rendered HTML as real Chrome. At eBay properties, nike.com, and canadiantire.ca it failed to render — those sites' anti-bot stance extends to JS-capable bots regardless of UA. At amazon.co.uk the trusted-crawler allowlist excluded it; it received a 200-byte stub while Googlebot received the full pre-rendered page.

How to allow / block

To allow Claude live retrieval (recommended for ecommerce):

User-agent: Claude-User
Allow: /

To block:

User-agent: Claude-User
Disallow: /

Quirks

out of training but allows Claude-User still participates in Claude's live answers.

(agent tool calls, "browse a page"). May not render JS for every fetch — Anthropic has stated they keep this efficient.

Sources