A/B testing on ecommerce storefronts without hurting Core Web Vitals

Your team spends a quarter getting product detail pages green on Core Web Vitals. Growth then installs a client-side experimentation tool, and Largest Contentful Paint (LCP) on the exact page under test slides back past 2.5 seconds. When conversions dip, nobody can say whether the variant caused it or the tool did. Assigning the variant before the response is built removes that ambiguity.

Here's what the browser-side approach costs you, and how to run the same test with Routing Middleware and Vercel Flags instead.

Copy link to headingWhy client-side A/B testing on an ecommerce storefront hurts Core Web Vitals

A browser-side tool offers two ways to apply a variant, each affecting a different performance metric.

The first is the anti-flicker snippet, which runs in four steps:

1. Define a CSS class that sets opacity: 0 on the page.
2. Add that class to document.documentElement from a synchronous script, before anything renders.
3. Arm a failsafe timer, commonly set to four seconds.
4. Remove the class once the tool finishes bucketing the visitor, or once the timer fires.

Between the second step and the fourth, the browser paints nothing, and LCP absorbs the whole wait. The 2.5-second LCP threshold leaves no room for a bucketing round trip on a page that was already close to it.

Without the snippet, post-paint changes can increase Cumulative Layout Shift (CLS). The variant lands as a DOM swap after first paint, elements move, and CLS climbs past the 0.1 Good threshold.

A clean CLS score on a snippet-protected page deserves suspicion, because CLS counts only movement the browser can observe. Shifts behind an opacity: 0 mask go unrecorded while the shopper still waits through them. Helly Hansen saw CLS above 2.8 before migrating to Vercel, and virtually eliminated it afterward. Speedway Motors moved a banner promotion A/B test into Global Config and cut CLS by 50%.

Copy link to headingWhat you need to run an ecommerce A/B test at the edge

You’ll need four components:

• Next.js 16 or later: Version 16 renames middleware.ts to proxy.ts and runs it on the Node.js runtime only. If you need the Edge runtime, stay on the deprecated middleware.ts convention.
• Routing Middleware: It runs before the cache, so the rewrite picks the variant before anything is rendered or stored. It's priced on the fluid compute model.
• The Flags SDK: Install the flags package and the @flags-sdk/vercel adapter, and set a FLAGS_SECRET environment variable to sign precompute codes and authenticate the Flags Explorer.
• Analytics packages: @vercel/analytics records conversion events, and @vercel/speed-insights reports field Core Web Vitals.

To migrate an existing project, run npx @next/codemod@latest middleware-to-proxy ., which renames the file, the named export, and the config flags that carried the old name. The Shirt Shop template wires the SDK, the toolbar, and analytics together on a product page, so clone it rather than starting from an empty file.

Copy link to headingHow to A/B test a storefront with Routing Middleware and Vercel Flags

Each experiment touches three files: a flag definition, a rewrite, and a page component that branches.

Copy link to heading1. Declare the flag with its options

Give every experiment a flag() declaration with a key, an options array, and an adapter:

import { flag } from 'flags/next';
import { vercelAdapter } from '@flags-sdk/vercel';

export const pdpLayout = flag<'control' | 'treatment'>({
  key: 'pdp-layout',
  options: [
    { label: 'Control', value: 'control' },
    { label: 'Treatment', value: 'treatment' },
  ],
  adapter: vercelAdapter(),
});

export const productFlags = [pdpLayout] as const;

The options array lets precompute encode values compactly into the URL, and it's what generatePermutations reads when it builds static pages. Export the experiment flags as one group and keep demo-only flags out of it, since every extra option multiplies pages nobody will look at.

Copy link to heading2. Rewrite the request in proxy.ts

Call precompute on the flag group, then rewrite to a path that carries the returned code:

import { type NextRequest, NextResponse } from 'next/server';
import { precompute } from 'flags/next';
import { productFlags } from './flags';

export const config = { matcher: ['/product/:path*'] };

export async function proxy(request: NextRequest) {
  const code = await precompute(productFlags);

  const nextUrl = new URL(
    `/${code}${request.nextUrl.pathname}${request.nextUrl.search}`,
    request.url,
  );

  return NextResponse.rewrite(nextUrl, { request });
}

Scope the matcher to the routes under experiment, or the rewrite runs on every request, static assets included. Wrap your identify function in dedupe from flags/next so the stable-ID cookie is read once per request no matter how many flags ask for it.

Copy link to heading3. Render the variant in a Server Component

Pass the code and the flag group to the flag function inside the rewritten route, then branch on the result:

import { productFlags, pdpLayout } from '../../../../../flags';

export default async function Page({
  params,
}: {
  params: Promise<{ code: string; slug: string }>;
}) {
  const { code } = await params;
  const layout = await pdpLayout(code, productFlags);

  return layout === 'treatment' ? <NewPdp /> : <CurrentPdp />;
}

Because the rewrite is internal, the shopper never sees the code segment, and no client-side flag fetch runs after paint. Vercel Flags hashes the entity value with xxHash32, so a shopper keeps the same variant across page loads and sessions without any extra storage. That hash is seeded per flag, which is why a banner assignment tells you nothing about the same shopper's button color.

Copy link to headingKeep your CDN cache intact across ecommerce A/B tests

Vercel's content delivery network (CDN) builds its cache key from the request method, the request URL, the host domain, the deployment URL, and the scheme. Cookies aren't in it, which is the detail most teams miss.

A response carrying set-cookie isn't stored at all, whatever its Cache-Control header says. Neither is a response whose Vary header names Cookie, since a near-unique value per visitor would give almost every request its own cache entry. Bucket by cookie without precompute, and every experiment request comes from compute.

Precompute puts the discriminator in the path instead. Each combination of flag values encodes to one signed code, so /abc123/product/blue-hoodie is a different cache entry from /def456/product/blue-hoodie. The page stays static, the CDN serves it from the nearest region, and the rewrite is the only request-time work. Splitting the cache that way multiplies your pages, so size the groups deliberately:

• Pre-build the high-traffic routes: Call generatePermutations inside generateStaticParams so every combination exists before the first shopper arrives.
• Let lower-traffic routes fill in on demand: Leave dynamicParams at its default of true and Incremental Static Regeneration (ISR) builds combinations on the first request.
• Split site-wide flags from page-level flags: Two groups keep a header experiment from multiplying against a product detail page experiment.

Three flags with two, two, and three options already give twelve variants of every matched page. ISR also needs the encoded URL to stay under 1,024 characters, which is the real ceiling on how many flags one group can hold. Locale behaves the same way. Put it in the path, and it joins the cache key at no cost; put it in a cookie, and the page leaves the cache.

Copy link to headingWhat to A/B test on an ecommerce storefront, and when to stop

Aim at the friction your storefront data already shows.

Checkout is a useful starting point. Seventeen percent of shoppers have abandoned an order because checkout was too complicated, and the average 2024 checkout flow runs 5.1 steps and 11.3 form fields against a workable optimum of around eight. Unexpected extra costs drive 40% of abandonment, and slow delivery accounts for another 20%. You can communicate both costs and delivery estimates on the cart page before checkout. Product detail page layout and promotional banner placement move earlier still, before the add-to-cart click decides whether a visitor enters the funnel at all.

Sample size scales with the inverse square of the effect you want to detect, so halving that effect quadruples the visitors you need. At a 5% baseline conversion rate and a 10% relative lift target, each variant needs roughly 31,000 visitors for a two-sided test at 95% confidence and 80% power.

For a fixed-horizon test, set the sample size in advance and wait until it is reached before deciding on a winner. Repeated significance checks on a fixed-horizon test inflate the false-positive rate well above the 5% you planned for, since every look is another chance to cross the line by accident. To watch a test continuously and stop it early, use a sequential method such as the mixture sequential probability ratio test, which yields always-valid p-values.

Copy link to headingHow to measure an ecommerce A/B test without losing Core Web Vitals coverage

Speed Insights has no variant dimension, but precompute gives every flag combination its own path, so the path breakdown does the same job. Check your tier first. The free tier reports only the Real Experience Score and LCP, CLS, and viewing individual Core Web Vitals requires Speed Insights Plus, which costs $10 per project per month on Pro and is included on Enterprise.

For the conversion side, emit your flag values to the DOM so Web Analytics annotates page views and events with them automatically, then fire your conversion event:

import { track } from '@vercel/analytics';

track('add-to-cart');

With flags annotated, Web Analytics compares add-to-cart and checkout rates by variant, while the path breakdown shows how each variant impacts performance.

Copy link to headingHow to troubleshoot ecommerce A/B tests that give back the performance win

Three mistakes undo the gain, and all three are avoidable before the first test ships.

The first is a decide function that awaits a distant database, which collapses the latency argument for edge assignment. Keep flag values and segment data in Global Config, where the vast majority of reads complete within 15 ms at the 99th percentile (P99) and often in less than 1 ms, or in the stable-ID cookie itself.

The second is mixing strategies. Running some experiment routes through precompute and others through Vary: Cookie produces cache behavior nobody can reason about, because half the storefront is static and half is quietly served from compute. Settle on path encoding for every experiment route before the first test goes live.

The third is dead flags. Cleanup never happens on its own, so write the removal pull request when you create the flag and put a sprint deadline on it. Vercel documents the full teardown. Strip the conditional from your components, deploy to preview to confirm the winning path still works, then archive the flag with one command:

vercel flags archive pdp-layout --yes

Whether archiving clears the cached permutations generated for that flag from the ISR cache isn't documented, so revalidate the affected routes afterward rather than assuming stale variant paths have gone.

Copy link to headingNext steps

Once proxy.ts and the flag group exists, a new experiment costs one flag() declaration and one branch in a page component. That is how Helly Hansen went from one A/B test a month to five or six per sprint. Deploy the Shirt Shop template to get the wiring in place on a working product page, or start a new Vercel project if you're adding edge assignment to a storefront you already run.

Copy link to headingRead more

• A/B Testing on Vercel
• Running an A/B test with the Vercel CLI
• Routing Middleware
• Precompute with the Flags SDK
• Improve Cumulative Layout Shift on Vercel
• Purging Vercel CDN cache