Brave Search Crawler
Brave Search is one of the few search engines with its own independent index (not licensed from Google or Bing). The crawler builds that index.
Specs
| Vendor | Brave |
| Category | SEARCH |
| robots.txt token | BraveBot |
| Renders JavaScript | RENDERS JS |
| Honors robots.txt | yes |
User-Agent string
(Brave's crawler does not advertise a distinguishing User-Agent.)Considerations
- Brave's policy is to send a generic browser UA to avoid being discriminated against by sites that allow only Googlebot. The robots.txt token BraveBot is the control surface site owners get.
- Brave has stated that if a URL is not crawlable by Googlebot (per robots.txt), Brave's crawler will not crawl it either — effectively a 'follow Googlebot's policy' fallback.
- Because there is no distinguishing HTTP UA, sites cannot detect Brave's crawler in their access logs without knowing its IP ranges.
robots.txt recipe
User-agent: BraveBot
Disallow: /Sources: Brave Search · Crawler FAQ