DEEP DIVE

The strange afterlife of dynamic rendering, 2018–2026

2026-05-19 · 12 min read · Joni Kautto

This is a history of a web technique that was endorsed by Google in 2018, became standard practice through 2023, was formally deprecated in 2024, and quietly came back in 2026 under different names — driven by a wave of crawlers Google's deprecation never anticipated. It's also a history of how the same idea keeps surfacing as engineering teams hit the same wall every few years: the bot side of the web and the user side of the web have drifted apart, and we have to do something about it.

The strongest counter-argument is engaged with directly in §5. I think the side I argue is the right one for sites that cannot migrate to native SSR within their planning horizon, which is most of them.

2018 — the technique gets a name

Dynamic rendering had been practiced for years before Google formally named it. The Prerender.io service had been around since 2013, Rendertron emerged from Google in 2017, and ad-hoc snapshot caches lived in every large e-commerce stack. What Google I/O 2018 did was bless the pattern as an official SEO strategy. The motivation was straightforward: Googlebot's renderer at the time used a deferred two-stage pipeline. HTML was crawled immediately. JavaScript was rendered later — sometimes much later — and large JS-heavy sites were therefore indexed incompletely and slowly. The fix was to give crawlers a pre-cooked HTML response and let Google's WRS off the hook.

The implementation pattern that became canonical:

  1. Detect a crawler by User-Agent.
  2. Route its request through a headless-Chrome service.
  3. Cache the rendered HTML.
  4. Serve the cached HTML to crawlers; serve the live SPA to users.

By 2020 most large e-commerce stacks had a version of this — branded as Prerender.io subscriptions, internal "snapshot service" projects, Cloudflare Always-Online, or Fastly snapshot caches. The pattern was load-bearing for a generation of single-page apps that would otherwise have ranked terribly.

2024 — Google says stop

Two things changed by the early 2020s. Googlebot's renderer became near-real-time — the deferred queue that made dynamic rendering necessary in 2018 had largely disappeared. And the drift problem started showing up in production: a small but visible class of sites were getting indexed with content that no longer existed in the live app, because the snapshot pipeline was running a slightly old build. Google's spam team formalized the position: serve one HTML to everyone, do it server-side, do not maintain a separate snapshot.

The 2024 deprecation was specific. It did not say "dynamic rendering doesn't work". It said "the conditions that made dynamic rendering necessary in 2018 are gone, and the drift risk is now larger than the indexing benefit." Most large sites took the advice and started planning Next.js / Nuxt / Angular Universal migrations. Some of those migrations completed. Many didn't.

2024–2025 — a new class of crawler shows up

While the SEO industry was still digesting the deprecation, a different conversation was happening: AI labs started launching production crawlers. GPTBot in mid-2023. ClaudeBot in 2024. PerplexityBot around the same time. By late 2024 every major AI vendor had at least one production crawler, often three — one for training, one for search-index, one for live retrieval.

One specific technical fact reshaped the entire problem: the major AI crawlers generally do not execute JavaScript. Vercel's December 2024 analysis ("The rise of the AI crawler") measured roughly 569 million GPTBot and 370 million ClaudeBot requests across their CDN in a single month — and the training crawlers in that traffic did not render JS. GPTBot downloaded JavaScript files about 11.5 % of the time but never executed them. Live-retrieval bots (ChatGPT-User, Claude-User, Perplexity-User, Meta-ExternalFetcher) occasionally invoke a headless browser when a user-driven task requires it, but treat rendering as optional rather than the default path. The implication for CSR-heavy sites is the same either way: a page that depends on JS to materialize its content is invisible to the bulk of AI-crawler traffic.

This broke the implicit assumption behind Google's 2024 deprecation. The deprecation said "you can stop pre-rendering because Googlebot can read your JS now." That premise applied to Googlebot specifically. It did not apply to GPTBot, ClaudeBot, PerplexityBot, or the third-party crawlers feeding Common Crawl. A CSR-only SPA that Googlebot could render perfectly in 2025 was invisible to the AI crawler ecosystem.

2026 — the revival, under different names

The pattern reasserted itself. Sites that had migrated to SSR (the right move per Google) didn't notice anything change; their bot story was already correct. Sites that hadn't — and that's most of the web — discovered their AI-search visibility quietly collapsing. The fix re-emerged, but the language shifted:

  • "Dynamic rendering" became "edge rendering" or "AI-readability layer".
  • Prerender.io kept its name and added explicit AI-bot configs.
  • Cloudflare and Vercel shipped one-click integrations.
  • Self-hosted stacks (Puppeteer + a cache, sometimes with a queue) re-appeared at scale.

Our May 2026 audit of the world's top 100 e-commerce sites found five doing classical dynamic rendering at meaningful ratios — amazon.co.uk serves Googlebot a fully pre-rendered 28 KB page while real Chrome gets a 200-byte stub, a 30× content multiplier. Coupang, shopping.yahoo.co.jp, uniqlo, canadiantire and a long tail of smaller sites do the same thing at smaller ratios. The technique Google deprecated two years ago is alive at retail sites visited by hundreds of millions of people every month.

5 · The strongest counter-argument — and why I still land where I land

The honest case against the position above has three parts plus a forward-looking forecast. Each is worth engaging with on its own terms before reaching for the defense.

Agentic fragmentation makes rDNS allowlists fragile. The world is no longer "Googlebot + Bingbot from known IP ranges". Headless agents triggered by ChatGPT, Claude, Perplexity, and a long tail of bespoke retrieval tools fetch from residential proxies, VPNs, and serverless pools where reverse-DNS gives nothing useful. The implication some draw is "therefore your allowlist will miss them and they will get the empty SPA shell, undermining the visibility you were trying to protect." The implication is correct about the failure mode and wrong about the remedy. The remedy is the inverse default: unknown UAs receive the SSR-rendered HTML, not the SPA shell. The bot path becomes the default; the SPA path is the exception, served only to clients that pass active fingerprint and behavior checks as human. The result is that even an unrecognised AI agent from a residential proxy still receives the indexed version of your content. The drift CI/CD cost the critique raises is correct, expensive, and the operational reality we live with — it is also engineerable with the SHA-comparison alarms documented elsewhere in this series.

The compute-economics re-reading of 2024. The standard story about Google's 2024 deprecation is "WRS got good enough that the workaround is unnecessary." That reading is incomplete. The cost side mattered at least as much: when Google asked the ecosystem to migrate to SSR, part of what was being said was "stop offloading the hydration of your client-side bundle onto our compute budget." The 2018 blessing of dynamic rendering had inadvertently subsidised the SPA-bloat era — sites shipped progressively heavier JavaScript on the implicit premise that Google would do the rendering work. By 2024 the cost of rendering hundreds of billions of pages at Google-scale was no longer absorbed quietly; the deprecation was a transfer of that compute burden back to the publisher, dressed in the language of architectural correctness. This re-reading explains why the official guidance leans so hard on "server-rendered HTML is best for everyone" rather than just "WRS can handle JS now." Both sentences are in the deprecation post; they target different audiences.

The drift trap is real. Maintaining two paths to a single URL — one for users, one for bots — does require synchronised CI/CD testing and per-route content-equivalence checks. When the bot HTML and the user-hydrated DOM drift apart, you get the failure mode that ends in Google catching the discrepancy and treating the site as cloaking. The defense is operational, not architectural: the bot version is produced from the same component tree the user hydrates (not a hand-built variant), SHA-equivalence checks alarm on drift, and the rendering path is the SSR pipeline applied to a Puppeteer client rather than a separate snapshot generator. Run this discipline and the trap doesn't close. Skip it and you become amazon.co.uk at 30× content ratio — a finding from our own audit that should function as a cautionary tale, not a template.

A 2027 forecast worth tracking. By next year, edge platforms (Vercel, Cloudflare, Fastly, Shopify) may productise an "Agent-Native Syndication" pattern: when a verified AI bot UA arrives at a product URL, the edge routes the request away from the presentation layer and serves a structured JSON or Markdown representation of the core entities (Product, Price, Inventory, Reviews) — likely at a standard path such as /.well-known/agent-data. If that pattern arrives at the protocol layer, the relative weight of HTML-based dynamic rendering shifts, because the formal path becomes additive to it. Two reasons I think the shift lands later than 2027. First, the bots that most need such a syndication path today — GPTBot, ClaudeBot, OAI-SearchBot, Claude-SearchBot — are already content with plain HTML; the protocol needs a defector at vendor scale (most likely Anthropic, whose web-search tool already prefers structured documents), and defectors of that size move on years not months. Second, the long tail of agents the fragmentation critique correctly worries about will not honor an .well-known/agent-data contract — they will keep scraping HTML because their input is whatever the page renders. So even with formal Agent-Native Syndication in place, the HTML path remains the floor of compatibility. PrerenderProxy is that floor.

The practical answer for 2026 doesn't change: render the HTML, serve it to everyone, alarm on drift, allowlist with rDNS where you can, and serve SSR to unknown UAs by default. The afterlife continues.

6 · Where we land

Universal SSR is the right destination for any site that can migrate to it within the planning horizon. Google's 2024 guidance was correct then and is correct now. Next.js, Nuxt, SvelteKit, Angular Universal, Remix — if you can pick one of these and ship the migration in two quarters, do it.

For sites that cannot migrate that quickly, dynamic rendering — done with reverse-DNS verified bot detection, with the bot HTML produced from the exact same component tree the user hydrates, with drift alarmed in observability — remains a defensible bridge. It is what Google's guidelines explicitly allow when same-content is honored. It is what amazon.co.uk does today at 30× content ratio. It is what we operate.

The strange afterlife probably ends along the path described in §5's 2027 forecast: edge platforms make it a one-click feature, the pattern absorbs into the default deployment pipeline, and "dynamic rendering" goes from being a named SEO strategy to an unremarkable detail of how the edge serves bots. The technique survives. The label fades. We are roughly two years from the label fading.

Until then, eight years and counting.

Companion reading: The May 2026 audit findings · Should you block AI bots? · Building PrerenderProxy