Meta-ExternalFetcher
| Vendor | Meta |
| Type | Live retrieval / on-demand fetcher |
| robots.txt token | Meta-ExternalFetcher |
| JavaScript rendering | Yes (sometimes) |
| Honors robots.txt | Yes — user-prompted, not a bulk crawler |
| Vendor docs | developers.facebook.com/docs/sharing/webmasters/crawler |
User-Agent string
meta-externalfetcher/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)Purpose
Companion to Meta-ExternalAgent. Fetches a URL right now because a Meta AI user just asked a question requiring web grounding. Each request maps to a user action — not a bulk crawl. The token exists to let publishers express different policies for "use my content to train models" vs "fetch my content live for a user asking right now".
Quirks
- User-prompted only — does not crawl on its own schedule.
- May render JS depending on the flow.
- Will sometimes appear with the original user's UA prepended; Meta's
docs explain the replacement behavior.
How to allow / block
To allow live retrieval (recommended for ecommerce):
User-agent: Meta-ExternalFetcher
Allow: /