Meta-ExternalAgent
| Vendor | Meta (Facebook, Instagram, WhatsApp, Threads) |
| Type | AI training crawler |
| robots.txt token | Meta-ExternalAgent |
| JavaScript rendering | No |
| Honors robots.txt | Yes (since 2024) |
| Vendor docs | developers.facebook.com/docs/sharing/webmasters/crawler |
User-Agent string
meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)Purpose
Meta's primary crawler for AI training data. Feeds the Llama family of models and Meta AI features across Facebook, Instagram, WhatsApp, and Threads.
Introduced in 2024 as part of Meta's AI training crawler taxonomy. Distinct from FacebookExternalHit (link-preview bot, predates LLMs) and Meta-ExternalFetcher (live-retrieval).
Quirks
- Aggressive crawling pattern with high request volumes vs traditional
search crawlers.
- HTTP-only — does not execute JavaScript or process CSS.
- Sites that block AI training but want to keep their FB/IG link
previews need to allow FacebookExternalHit while blocking Meta-ExternalAgent.
How to allow / block
To opt out of Llama training:
User-agent: Meta-ExternalAgent
Disallow: /To opt out of training but keep FB/IG link previews:
User-agent: Meta-ExternalAgent
Disallow: /
User-agent: FacebookExternalHit
Allow: /