Top-30 Cross-Bot HTML Parity Audit · May 2026

1 · What this audit measures

For each of the Top-30 most-visited websites worldwide (Similarweb, April 2026), we send the same plain GET / request from ten different User-Agent headers and compare the HTML each one receives back. Two humans (Chrome desktop & mobile), two traditional search bots (Googlebot, Bingbot), and six AI/LLM crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Applebot).

The Chrome desktop response is the baseline. For every other UA we record: HTTP status, byte size, the <title>, <h1>, JSON-LD blocks, visible-text length, and a SHA-256 of the normalized visible text. "100% same" means the visible-text hash is byte-identical to Chrome; lower ratios indicate either an empty SPA shell, a login wall, or a divergent variant.

A 403 / 429 / Cloudflare challenge is recorded as a finding, not retried via a residential proxy — because the question is precisely "what does this UA see today?". All requests are HTTP-level only (no JS execution), which mirrors what GPTBot / ClaudeBot / PerplexityBot actually do today (Vercel, 2025: none of the major AI crawlers execute JavaScript).

2 · Why dynamic rendering is back on the table in 2026

Google formally deprecated dynamic rendering (serving a pre-rendered HTML to bots and a JS app to humans) in 2024 and continues to recommend SSR / SSG / hydration instead. But the 2025–2026 reality has shifted the debate:

• AI crawlers don't render JavaScript. A Vercel analysis of 500M+ bot fetches showed GPTBot, ClaudeBot, and PerplexityBot all skip JS execution — they read whatever HTML the server first returns, then move on.
• CSR-only SPAs are invisible to LLMs. A site that ships an empty <div id="root"> shell and hydrates client-side will show ~0 chars of visible text to those crawlers, even if Googlebot can still render it.
• So the "two-HTML" pattern returns under a new name. Edge pre-rendering (Cloudflare/Fastly), Next.js SSR, Nuxt SSR, Angular Universal, and dedicated services like PrerenderProxy all produce the same outcome: crawlers see fully-formed HTML. Whether you call it dynamic rendering, prerendering, or SSR-for-bots, the architectural goal is identical — give every crawler the static, indexable version of the page.
• Cloaking only triggers if the bot-version differs from the user-version in content, not in form. Search engines explicitly carve out an exception for pre-rendering as long as the same content is shown.

The audit below is therefore a sanity check on the world's biggest sites: do they actually achieve content parity across human + search + AI user-agents? Or are some bots already getting a thinner, blocked, or divergent view of the web?

3 · Headline numbers

4 · Key findings

4.1 · Hard AI-bot blocks (the gatekeeping sites)

Seven sites return HTTP 4xx to one or more declared AI crawlers from this server's IP while serving Chrome normally. This is the closest thing to a "no robots" policy actually enforced at the edge — and it splits cleanly by vendor:

• #11 wikipedia.org (claudebot, applebot)
• #13 yahoo.com (claudebot, chatgpt_user, oai_searchbot, perplexitybot, gptbot)
• #14 yandex.ru (claudebot, perplexitybot)
• #17 linkedin.com (gptbot)
• #21 cloud.microsoft (applebot)
• #24 live.com (applebot)
• #25 ebay.com (claudebot, applebot, chatgpt_user, oai_searchbot, perplexitybot, gptbot)

Note yahoo.com blocks every one of the six AI bots (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot) while still giving Chrome a full SSR page. linkedin.com blocks only GPTBot (their well-publicized stance against AI training data scraping). ebay.com blocks every declared AI bot UA and the search bots — a near-total "humans-and-direct-traffic only" posture from the front page.

4.2 · Sites blocking declared Googlebot / Bingbot from a random IP

Five sites refuse the canonical Googlebot or Bingbot User-Agent when it comes from an IP outside the official bot ranges. The lesson: never trust the User-Agent on its own — these sites verify reverse-DNS or maintain an IP allowlist on top of UA matching.

• #7 x.com (bingbot)
• #9 tiktok.com (googlebot_smartphone, bingbot)
• #11 wikipedia.org (googlebot_smartphone, bingbot)
• #21 cloud.microsoft (googlebot_smartphone, bingbot)
• #25 ebay.com (googlebot_smartphone, bingbot)

Wikipedia is the most public-spirited example: its anti-abuse stack rejects "Googlebot" claims from non-Google IPs with the same 403 you'd give a curl script — a healthy default that more sites should copy.

4.3 · Pure SSR — the same bytes for everyone

16 of the 30 sites are fully UA-neutral: no blocks, no challenges, every user-agent gets a response. Crucially, 14 of them serve byte-identical visible text (SHA-256 matches) to at least one AI crawler and Chrome — proof that a single SSR pipeline is feeding both browsers and bots:

• #4 instagram.com (chatgpt_user, oai_searchbot, perplexitybot, gptbot)
• #5 chatgpt.com (claudebot, applebot, chatgpt_user, oai_searchbot, perplexitybot, gptbot)
• #6 reddit.com (applebot, oai_searchbot, perplexitybot)
• #9 tiktok.com (claudebot, applebot, chatgpt_user, oai_searchbot, perplexitybot, gptbot)
• #11 wikipedia.org (chatgpt_user, oai_searchbot, perplexitybot, gptbot)
• #12 yahoo.co.jp (claudebot, chatgpt_user, oai_searchbot, perplexitybot, gptbot)
• #14 yandex.ru (chatgpt_user, oai_searchbot, gptbot)
• #15 gemini.google.com (claudebot, applebot, chatgpt_user, oai_searchbot, perplexitybot, gptbot)
• #16 amazon.com (claudebot, applebot, chatgpt_user, oai_searchbot, perplexitybot, gptbot)
• #18 baidu.com (applebot)
• #20 naver.com (claudebot, applebot, chatgpt_user, oai_searchbot, perplexitybot, gptbot)
• #22 netflix.com (claudebot, applebot, oai_searchbot, perplexitybot, gptbot)
• #24 live.com (claudebot, chatgpt_user, oai_searchbot, perplexitybot, gptbot)
• #26 bilibili.com (applebot)
• #27 temu.com (claudebot, applebot, chatgpt_user, oai_searchbot, perplexitybot, gptbot)
• #28 twitch.tv (claudebot, applebot, chatgpt_user, oai_searchbot, perplexitybot, gptbot)
• #29 dzen.ru (claudebot, chatgpt_user, oai_searchbot, perplexitybot, gptbot)
• #30 microsoft.com (claudebot, applebot, chatgpt_user, oai_searchbot, perplexitybot, gptbot)

This is the modern post-deprecation pattern Google asked for in 2024: one HTML response, served to everyone, no UA forking. Amazon, Naver, Netflix, Microsoft, Temu, Twitch, and Gemini all run this way.

4.4 · "Dynamic rendering by another name" — bots get MORE than humans

A handful of sites still return dramatically more content to declared bots than to a real Chrome browser. The most extreme is baidu.com: Chrome gets a 357-character shell and a script that hydrates the search UI; Googlebot, Bingbot, GPTBot, ClaudeBot and PerplexityBot all get a ~250 KB fully-rendered HTML — the classic pre-render-for-bots pattern that Google formally deprecated but the rest of the world quietly keeps using.

• #13 yahoo.com Chrome=2521 chars → applebot=436%, googlebot_smartphone=442%, bingbot=441%
• #18 baidu.com Chrome=357 chars → claudebot=69488%, chatgpt_user=69488%, oai_searchbot=69488%, perplexitybot=69488%, gptbot=69488%, bingbot=69488%
• #20 naver.com Chrome=126 chars → googlebot_smartphone=815%
• #23 pinterest.com Chrome=19 chars → claudebot=394%, applebot=394%, googlebot_smartphone=394%, perplexitybot=394%, bingbot=394%

Whether this counts as "cloaking" depends on whether the content is the same. In Baidu's case the same query box and same brand are present in both responses, just rendered server-side for crawlers — which is exactly the "edge pre-rendering" pattern that PrerenderProxy implements for client sites in Finland.

4.5 · IP-level blockades (everything blocked, not bot-specific)

Four sites returned a hard 4xx to every single UA we tried — including a real Chrome. From this server's IP (a Hetzner datacenter address) they treat all traffic as untrusted regardless of User-Agent:

• #5 chatgpt.com
• #6 reddit.com
• #19 bet.br
• #25 ebay.com

These are not informative findings about UA-specific behavior — they're informative about the broader trend of "block all datacenter IPs at the edge". From a residential IP all four would likely respond normally; they're included so the data set is honest about which results are constrained by our origin.

5 · Site × UA heatmap

Green = byte-identical to Chrome. Yellow = 50–90% text parity. Blue = 10–50% (likely a thinner variant or partial render). Red text = HTTP block (403/429/etc.). Click a domain to jump to the per-site breakdown.

6 · Per-site detail

7 · How to read the SHA column

Each cell shows the first 10 characters of the SHA-256 of the normalized visible text (HTML stripped, whitespace collapsed). If two UAs share a hash, they got byte-identical content. If they differ, even by a single user-specific token, the hashes will diverge — useful for spotting personalization, A/B-test buckets, or country-specific bodies served to one UA family but not another.

8 · About PrerenderProxy

PrerenderProxy is the open custom-built pre-rendering layer behind a number of Finnish high-traffic sites (Elisa, Yritystele, …). It sits at the Fastly edge, detects crawler user-agents, and serves Puppeteer-rendered HTML snapshots to bots while regular users get the live SPA. The conceptual model is identical to what the Top 30 above are doing internally with Next.js / Nuxt / Angular Universal SSR — the difference is that PrerenderProxy retrofits the same property onto sites that can't easily migrate to a framework-native SSR path. See / for the dashboard (basic-auth required).