Most AI assistants don't run JavaScript, and search engines render it unevenly — so they reach a half-built shell of your pages, or skip them. PrerenderProxy gives each AI assistant and search engine your real, fully-rendered page, tuned to what it can use: clean HTML, the right structured data, shaped per bot.
Open. Works with the site you already have.
<html>
<head><title></title></head>
<body>
<div id="root"></div>
<script src="/app.js"></script>
</body>
</html>
<title>Atlas Pack 30L — €149</title> <meta name="description" content="…"> <script type="application/ld+json"> { "@type":"Product","offers":{"price":"149"} } </script> <h1>Atlas Pack 30L</h1> <p>Waterproof, …</p>
62 of the top 100 e-commerce sites block at least one major AI · only 14 are fully readable · audited May 2026
Raw data, screenshots, methodology, per-site cards → audit/2026-05-ecommerce-100
PrerenderProxy runs at the edge of your existing CDN. It detects every legitimate search and AI crawler — Google, Bing, GPTBot, ChatGPT-User, ClaudeBot, Claude-User, Perplexity, Applebot — and serves them a Puppeteer-rendered HTML snapshot of the same page your customers see. Your SPA stays exactly as it is. Your bot story converges to one HTML response.
Reverse-DNS + IP-range validation against every vendor's published bot list. UA-only matching is a footgun — we don't ship it.
The rendered HTML is exactly what your hydrated SPA would show — no special pleading, no drift. Compatible with Google's 2024 guidance and AI crawlers' 2026 reality.
Snapshots warm proactively from your sitemap and live in your existing Fastly or Cloudflare cache. Crawlers get edge-cache-speed responses on hit; origin load drops; no SSR migration required.
Block training bots, allow live-retrieval bots, allow search-index bots — independently, per UA, by reverse-DNS. The right policy for AI shopping visibility in 2026.
Every render emits a structured event. Hit rate, error rate, render time, and per-bot success — straight into Elasticsearch / Grafana / whatever you already run.
All config is plain JSON + VCL + Markdown. No bespoke DSL, no opaque admin UI. Designed for iterative AI-led development: any change is one PR, one diff, one human review.
The agentic web needs pages that expose tools, not just content. When your CMS can't add tool attributes or let you register JavaScript, PrerenderProxy can inject WebMCP tool declarations at the render layer — the same no-rewrite pattern, aimed at agents that act. Read the explainer →
A request hits your existing CDN. Crawlers are routed to a Puppeteer service that runs your real JavaScript; everyone else gets your SPA shell as normal. Both paths render the same page — one in the user's browser, one at the edge.
VCL classifies the request: real user → SPA path; verified bot → prerender path. Verification is rDNS + IP-range, never UA-only.
Cache hit → edge-cache response, no compute. Cache miss → Puppeteer renders the page using your real SPA + JS bundle, then caches the HTML for next time.
The bot gets the same DOM your customers would see after hydration — title, meta, JSON-LD, Product / Offer schema, full body — in one fetch.
A nightly crawler keeps the snapshot fresh against your sitemap. Every render emits a metric line. Drift between bot HTML and user HTML is alarmed, not tolerated.
Google deprecated dynamic rendering in 2024 on the assumption that everyone would migrate to SSR. Most didn't — and AI crawlers showed up that don't execute JavaScript at all. The technique came back under new names. We just write down honestly what the trade-offs are.
Serving fully-formed HTML helps far beyond AI assistants. The same edge-prerendered page an AI crawler can finally read also reaches search engines that render JavaScript less reliably than Google — and even helps Googlebot, which renders JavaScript well but defers it to a separate, budget-limited pass, index your content faster and cheaper.
renders JS — but static still wins on speed & budget
In February 2026 Google's documented crawl limit dropped from 15 MB to 2 MB per file — Googlebot now indexes only the first 2 MB of a page. Lean prerendered HTML stays well inside that budget; a bloated client-rendered bundle can push real content past the cutoff. And Googlebot does render JavaScript, so "Google can't read your app" is false — the value of static HTML is that your content lands in the first crawl wave, sparing both crawl budget and the separate render budget Google spends on JavaScript. Google's own docs put it plainly: "server-side or pre-rendering is still a great idea… not all bots can run JavaScript."
render JS less reliably — so static = they see you
Outside Google, JavaScript rendering is partial and inconsistent. Microsoft has long advised serving Bing prerendered HTML rather than relying on its JS rendering; DuckDuckGo leans on Bing's index, so the same limits apply; Yandex renders selectively and says to keep critical content in the initial HTML; Brave runs its own index with rendering depth it doesn't publicly confirm. The safe read: don't bet visibility on client-side rendering.
mostly don't render JS — empty without prerendering
The major AI crawlers — GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot and others — mostly don't execute JavaScript. The most-cited study (Vercel × Merj, across 500M+ GPTBot fetches) found no evidence of JS execution: they read your JavaScript as text but never run it. A client-rendered page looks blank to them, even when Googlebot renders it fine. (Google's Gemini, which rides Googlebot, and Applebot, on WebKit, are the exceptions.)
AI knowledge reaches models through three modes — training, search, and fetch. Fully-formed HTML helps in every one. See the three knowledge modes — training, search, fetch.
Crawlers that build training corpora mostly don't run JavaScript, so static HTML is what gets captured into a model's memory. Content missed at training time can't be added to that model later.
One fix, two channels: AI answer engines reading live results and classic search engines indexing your page both get your full content instead of an empty shell.
When an AI assistant fetches your URL mid-answer, it pulls the page without rendering — so a JavaScript-only page returns an empty shell. Prerendered HTML means it reads your real page: headings, links, prices, availability.
The plain-language version: what WebMCP is, the HTML attributes and JavaScript it's made of, why the page has to render first, and how a rigid CMS still gets tools — at the edge.
DEEP DIVEA citation-first teardown: what registerTool really does, where the spec is a TODO, the single-engine reality, the one real attack paper, an FAQ, and the honest good and bad.
PRACTICEWhen the CMS can't ship modern SEO + AI metadata, the prerender layer can. Per-bot response shaping with the cloaking-risk safety rails worked out.
OBSERVABILITYStructured-log schema, Vector → ES → Combot pipeline, five anomaly alerts with thresholds, GDPR PII handling.
ENGINEERINGnginx, Cloudflare, Fastly, Vercel, AWS, Apache + OpenResty Lua. The three-step protocol, the vendor IP-range JSON sources, the operational tips most posts skip.
POSITIONThe block-training framework was written for publishers. For brands, the math inverts. The parametric-recall mechanism, the 2026 numbers, the strongest counter-argument addressed.
DEEP DIVEGoogle deprecated it. The AI crawlers brought it back. A history of a technique that refused to die — with the 2027 forecast.
RESEARCHThe single biggest finding from our May 2026 audit: most large e-commerce sites have voluntarily made themselves invisible to AI shopping.
DECISIONTraining vs live retrieval vs search index. They're not the same thing. How to write a robots.txt that blocks the first and welcomes the second.
BEHIND THE SCENESWhy we chose Puppeteer over Playwright, what Fastly VCL lets us do that Cloudflare doesn't, and the operating cost of a self-hosted prerender layer.
We send our full 100-site audit dataset (every site × every bot, raw HTML, screenshots) and a one-page benchmark of your own site against the cohort. No pricing pages yet — we're picking the first ten partners by hand.