facebookexternalhit
Fetches a URL whenever someone shares it on Facebook, Instagram, Messenger, or WhatsApp to extract og:title, og:image, og:description and render the preview card. Predates the LLM era — around since ~2010.
Specs
| Vendor | Meta |
| Category | |
| robots.txt token | facebookexternalhit |
| Renders JavaScript | HTTP only |
| Honors robots.txt | partial |
User-Agent string
facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)Considerations
- Reads only the HTML head and a small body window — not designed to crawl deep.
- Per Meta's webmaster docs, the crawler may bypass robots.txt when performing security or integrity checks (malware / malicious-content detection). Normal share-driven fetches honor robots.txt.
- SPA sites that don't SSR og:* meta tags ship broken FB / IG previews.
robots.txt recipe
User-agent: facebookexternalhit
Allow: /Sources: Meta · Sharing webmasters