Meta-ExternalAgent
Meta's primary AI-training crawler. Feeds the Llama family of foundation models and Meta AI features across Facebook, Instagram, WhatsApp, and Threads.
Specs
| Vendor | Meta |
| Category | MEMORY |
| robots.txt token | Meta-ExternalAgent |
| Renders JavaScript | HTTP only |
| Honors robots.txt | yes |
User-Agent string
meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)Considerations
- Aggressive crawling pattern with high request volumes versus traditional search crawlers.
- HTTP-only — no JS, no CSS processing.
- Distinct from facebookexternalhit (link-preview bot) — blocking the training crawler does not break your FB / IG link previews.
robots.txt recipe
User-agent: Meta-ExternalAgent
Disallow: /Sources: Meta · Crawlers documentation