Audit documents

Companion writing to the May 2026 cross-bot audit — technical findings, blog post, case study, dynamic-rendering summary.

← docs index · audit index

What eBay, Nike, and Amazon UK actually show the AI bots you'd swear they'd be courting

May 2026 · PrerenderProxy

Open a fresh ChatGPT, ask it "where can I buy a 65-inch OLED TV under $1500 in the US", and watch closely which retailers show up in the answer. Best Buy? Costco? Home Depot? Macy's? Lowe's? Don't be surprised if none of them do. Open a Perplexity tab and try Sephora, Ulta, IKEA, John Lewis. Same outcome.

This isn't an AI quality problem. It's a configuration choice on the retailer side that 62 of the world's 100 largest e-commerce sites made in roughly the same way over the past 18 months — and the consequences for product visibility in AI-driven shopping are now showing up at scale.

We spent May running three increasingly careful audits of the front pages of the top 30 most-visited sites (general) and the top 100 e-commerce sites in the world. Each site was probed with up to ten different User-Agent headers — real Chrome, mobile Chrome, Googlebot, Bingbot, GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Applebot — plus, for fifteen high-signal cases, a real Chromium browser rendering the page with each bot UA and screenshots saved.

The full data and reports are public:

Here are the four findings that surprised us most.

1 · 62 of 100 e-commerce sites block AI bots the same way

The dominant pattern across our 100-site sample is not the long tail of "this site does its own thing." It's a remarkably uniform cluster: 62 sites block exactly the same four user-agents in exactly the same way — GPTBot, ChatGPT-User, ClaudeBot, and PerplexityBot all get a ~150 KB HTTP 403 with a Cloudflare interstitial.

That homogeneity is the story. Sixty-two independent retailers did not arrive at the same policy via parallel evolution. They turned on a common preset — Cloudflare's "Block AI Bots" managed ruleset, AWS WAF's equivalent, or one of the bot-management vendors' presets — and let the defaults make the decision.

The cost of that default is precise:

user just asked ChatGPT a question that needs a fresh web result. If you block it, your products do not appear in that answer.

All three are live retrieval bots, completely distinct from the training bots that share the cluster (GPTBot, ClaudeBot). The training bots take your content into a model that may never directly recommend your products. The live retrieval bots are the AI-shopping equivalent of a customer walking into your store.

Sixty-two retailers blocked both kinds together because the WAF preset treats them as one category. They are not.

2 · Four eBay properties block ClaudeBot specifically (and only ClaudeBot)

Six of our 100 sites block exactly one AI bot. That one bot is ClaudeBot in every case. Four of the six sites are eBay-owned:

Plus nike.com and canadiantire.ca, two independent decisions.

This is not a WAF preset — no preset blocks ClaudeBot alone. It's a named, hand-rolled rule. Anthropic specifically is unwelcome at eBay properties; OpenAI and Perplexity are tolerated. The reasons aren't in the public record. The shape of the rule, however, is very public: a single User-Agent token, denied at the edge, applied consistently across the org's properties.

This finding made it into our v2 report as "the ClaudeBot vendetta." We expected to leave it there. Then we ran v3.

3 · v3 changed the story — eBay actually blocks every JS-capable bot

v2 was an HTTP audit. We sent one request per UA, captured the body, moved on. When we did the same probe with a real Chromium browser in v3 — real JavaScript engine, real network idle wait, real screenshots — the "ClaudeBot only" story collapsed.

At ebay.com, ebay.de, kleinanzeigen.de, nike.com, and canadiantire.ca, zero of six bot UAs successfully rendered content. Every JS-capable client identifying as Googlebot, Bingbot, Applebot, ChatGPT-User, Claude-User, or Perplexity-User got either a 4xx, a timeout, or a tiny challenge body. The 200 OK that v2 saw was deceptive: the HTTP layer returned a polite 200 with a near-empty body or a challenge page that v2's "is it 4xx?" detector didn't flag.

The corrected interpretation: eBay's ClaudeBot block is a public-facing expression of a much wider anti-AI-bot posture. When a real browser arrives with any bot UA, the deeper layers of eBay's stack reject it. ClaudeBot stands out only because Anthropic's UA was the one that didn't get the polite 200 — they got the explicit 4xx. The others got a slower no.

This is why v2 → v3 mattered. HTTP-only audits flatten a multi-layered defense into one column.

4 · Dynamic rendering — the technique Google deprecated in 2024 — is alive and well in 2026

The biggest single shock from v2: amazon.co.uk serves Googlebot a fully pre-rendered ~900 KB HTML page while serving real Chrome a ~30 KB shell. A 30× content multiplier in favor of crawlers. Shopping.yahoo.co.jp does a 7–8× multiplier. Coupang does 5×. Uniqlo does 3×. Canadian Tire does 6× for AI bots specifically.

This is the dynamic rendering pattern Google formally deprecated in 2024 — "serve one HTML to everyone." It did not go away. It just stopped being mentioned in the official guidance because Google would prefer you migrate to SSR, and the deprecation removed the official endorsement of the workaround.

What v3 added: Amazon UK's discrimination isn't humans-vs-bots. It's trusted bots vs AI bots. Googlebot, Bingbot, and Applebot all get the full pre-rendered version (~28 KB rendered text in our headless test). ChatGPT-User, Claude-User, and Perplexity-User get a 200–400 byte stub. Amazon UK has an internal allowlist of which crawlers deserve the indexed version. That is a defensible policy choice — you can argue Google indexes your products for searchers, so they get the SEO version — but it bakes in a multi-year head-start for the older crawlers over the new AI ones.

Why this matters — and what to do about it

The combination is bleak. Most large retailers run the AI-blocking WAF preset and still depend on Google indexing the well-formed HTML. The retailers that do not block AI bots — walmart.com, target.com, samsung.com, apple.com, ulta.com, shein.com, alibaba.com, newegg.com, otto.de, decathlon.com, nordstrom.com — are the ones whose products will keep appearing in ChatGPT, Claude, and Perplexity shopping answers through 2026 and 2027. They are also, not coincidentally, the ones whose homepages serve byte-identical HTML to every UA: a single SSR pipeline, no special pleading for any crawler.

The actionable list for the other 62:

1. Read the rule you turned on. Cloudflare's "Block AI Bots" bundles training and live-retrieval bots together. Untangle them. Block GPTBot and ClaudeBot if you want to opt out of training. Allow ChatGPT-User, Claude-User, and Perplexity-User — those are the bots that bring you customers. 2. Verify by reverse-DNS, not UA. UA spoofing is trivial. Every legitimate AI vendor publishes the hostnames its bots resolve from. 3. Stop dynamic rendering. If your bot version is 30× the size of your user version, you are doing something Google said to stop doing two years ago. Move to SSR; the cleanest implementations on our top-100 list are walmart, samsung, and ikea-the-corporate-site (a different story).

For everyone running a site that's not yet on this scale: the most durable choice in 2026 is the same as the 2024 advice. Serve one HTML response to everyone. Make it the SSR-rendered one. If your stack can't do that natively, PrerenderProxy and similar edge prerendering tools exist to retrofit the property until the migration is done.

Full methodology, raw data, and per-site cards: links above. Comments and corrections welcome — we will re-run this audit in November.