Long-form pieces on bot-readability, the strange afterlife of dynamic rendering, the May 2026 cross-bot audit, and the day-to-day of running a prerender layer in production.
Inject schema, fix canonicals, convert soft 404s, shape responses per-bot at the prerender layer when the origin CMS won't. Cloaking risk addressed directly with production-grade safety rails.
OBSERVABILITYThe structured-log schema PrerenderProxy emits per request, the Vector → ES → Combot.ai pipeline, five anomaly alerts with thresholds. GDPR PII handling and VRL syntax notes baked into the recipe.
ENGINEERINGWorking code for nginx, Cloudflare, Fastly, Vercel, AWS, Apache + OpenResty Lua. The three-step protocol, the published vendor IP-range JSON sources, the operational tips most rDNS posts skip.
POSITIONThe decision framework that says "block training, allow retrieval" was written for publishers. For brands, the math inverts. The parametric-recall mechanism, the 2026 numbers, the strongest counter-argument addressed directly.
DEEP DIVEEight years from Google I/O recommendation to formal deprecation to AI-era revival. The compute-economics re-reading of 2024, the strongest counter-argument, and a 2027 forecast.
RESEARCH62 sites block AI crawlers wholesale. 6 single-target ClaudeBot. 5 still dynamic-render. The full pattern map from our May 2026 audit.
DECISIONThe Cloudflare "Block AI Bots" preset blocks both training and live-retrieval bots. Here's how to write a more honest robots.txt — and what it costs.
ENGINEERINGFastly VCL + Puppeteer + a custom warm-cache crawler. Why we picked each piece, what we'd do differently, and the operating cost at our scale.