Long-form pieces on bot-readability, the strange afterlife of dynamic rendering, the May 2026 cross-bot audit, and the day-to-day of running a prerender layer in production.
The plain-language version of WebMCP: what it is, the HTML attributes and JavaScript it's built from, why the page has to render first, and how PrerenderProxy brings tools to a CMS that can't add them itself.
DEEP DIVEA citation-first teardown of WebMCP in mid-2026: the imperative and declarative APIs, the Draft-CG-Report reality, WebKit's oppose, the one real attack paper, an FAQ, and the honest good and bad.
ANALYSISGoogle's documented Googlebot crawl limit now reads 2MB, down from 15MB. The fetch-vs-index nuance most write-ups get wrong, who is actually at risk, and why byte order rewards lean, content-first HTML.
PRACTICEInject schema, fix canonicals, convert soft 404s, shape responses per-bot at the prerender layer when the origin CMS won't. Cloaking risk addressed directly with production-grade safety rails.
OBSERVABILITYThe structured-log schema PrerenderProxy emits per request, the Vector → ES → Combot.ai pipeline, five anomaly alerts with thresholds. GDPR PII handling and VRL syntax notes baked into the recipe.
ENGINEERINGWorking code for nginx, Cloudflare, Fastly, Vercel, AWS, Apache + OpenResty Lua. The three-step protocol, the published vendor IP-range JSON sources, the operational tips most rDNS posts skip.
POSITIONThe decision framework that says "block training, allow retrieval" was written for publishers. For brands, the math inverts. The parametric-recall mechanism, the 2026 numbers, the strongest counter-argument addressed directly.
DEEP DIVEEight years from Google I/O recommendation to formal deprecation to AI-era revival. The compute-economics re-reading of 2024, the strongest counter-argument, and a 2027 forecast.
RESEARCH62 sites block AI crawlers wholesale. 6 single-target ClaudeBot. 5 still dynamic-render. The full pattern map from our May 2026 audit.
DECISIONThe Cloudflare "Block AI Bots" preset blocks both training and live-retrieval bots. Here's how to write a more honest robots.txt — and what it costs.
ENGINEERINGFastly VCL + Puppeteer + a custom warm-cache crawler. Why we picked each piece, what we'd do differently, and the operating cost at our scale.