How to run A/B tests with Next.js and Vercel

4 min read

Sep 9, 2022

Running A/B tests is hard.

We all know how important it is for our business–it helps us understand how users are interacting with our products in the real world.

However, a lot of the A/B testing solutions are done on the client side, which introduces layout shift as variants are dynamically injected after the initial page load. This negatively impacts your websites performance and creates a subpar user experience.

To get the best of both worlds, we built Edge Middleware: code that runs before serving requests from the edge cache. This enables developers to perform rewrites at the edge to show different variants of the same page to different users.

Today, we'll take a look at a real-world example of how we used Edge Middleware to A/B test our new Templates page.

Redesigning the Template Marketplace

Earlier this year, we kickstarted the redesign of our Templates page.

Our old Templates page had a limited scope that focused mainly on the different frameworks that were supported on Vercel:

Old version: limited selection of templates that focused mainly on framework starters.

For the new version, we wanted to build a marketplace to showcase all the different types of applications that can be built on Vercel – blogs, e-commerce storefronts, etc.

We also improved search to allow for partial matching of search terms and added category filters to make discovering new templates much easier:

New version: wider selection of templates with fuzzy search & filters.

Since this was a major change of one of our most popular pages, we decided to break down the launch into multiple release phases:

Early Access: Only 20% of visitors to /templates will see the new variant.
Public Beta: An equal 50:50 split between the new and old variant.
General Availability: Public launch–everyone gets the new variant.

We used Vercel's Edge Middleware to stagger our launch into 3 phases.

We leveraged Edge Middleware to perform the A/B test at the edge, which allowed us to gather insightful user feedback without sacrificing performance.

Catch-all Routes & On-demand Revalidation

Since we were using Next.js, we decided to create the new templates marketplace using the following optional catch-all route: /templates/[[...slug]].tsx.

This allowed us to programmatically generate new static pages when new templates are added using On-Demand Incremental Static Regeneration (ISR) and navigate between them using shallow-routing.

Here are some examples:

/templates/next.js/nextjs-boilerplate: Next.js boilerplate template
/templates/svelte/sveltekit-boilerplate: SvelteKit boilerplate template

We also generated dynamic landing pages for each filter option to capture long-tail SEO traffic:

/templates/next.js: Next.js templates
/templates/blog: Blog templates
/templates/tailwind: Templates that use Tailwind CSS

We then moved the old templates marketplace to the /templates-old.tsx route.

Experimenting at the Edge

To perform the A/B test, we created a middleware.ts file with the following code:

middleware.ts

import { type NextRequest, NextResponse } from 'next/server';

// make sure the middleware only runs when
// the requested url starts with `/templates`
export const config = {
  matcher: ['/templates(.*)'],
};

const THRESHOLD = 0.2; // initial threshold for the new variant (20%)
const COOKIE_NAME = 'tm_var'; // name of the cookie to store the variant

export function middleware(req: NextRequest) {
  // get the variant from the cookie
  // if not found, randomly set a variant based on threshold
  const variant =
    req.cookies.get(COOKIE_NAME) || (Math.random() < THRESHOLD ? 'new' : 'old');

  const url = req.nextUrl.clone();

  // if it's the old variant, rewrite to the old templates marketplace
  if (variant === 'old') {
    url.pathname = '/templates-old';
  }

  const res = NextResponse.rewrite(url);

  // set the variant in the cookie if not already set
  if (!req.cookies.get(COOKIE_NAME)) {
    res.cookies.set(COOKIE_NAME, variant);
  }
  return res;
}

Middleware code to perform the A/B test at the edge.

Here's a step by step of what the middleware does:

Tags each new visitor to the /templates page with either the old or new variant based on the current threshold (0.2 for the first stage)
If the user is part of the old variant, performs a rewrite to the /templates-old.tsx route
Saves the user's variant in the tm_var cookie to ensure the user gets served the same version in subsequent visits

Avoiding layout shift with Edge Middleware

As we developed the new templates page, we ran into an interesting problem–data fetching had to be done on the client side for three reasons:

We were using query parameters to preserve the filter state
The pages were generated statically
You can't read query parameters inside getStaticProps

To avoid layout shift, we decided to use skeleton loaders to buffer the loading state before showing the templates that match the configured filters.

For instance, if you go to vercel.com/templates?framework=svelte, you first receive a set of skeleton loaders while the data is fetched on the client side, and then the list of Svelte templates:

However, for the dynamic landing pages for each filter option (e.g. vercel.com/templates/svelte), we wanted to avoid skeleton loaders since we already know the set of templates that match the given filter at build time.

To get the best of both worlds, we used Edge Middleware. Since Edge Middleware is executed before a request is processed on a site, we are able to detect if there are any query parameters present right when the user requests the page.

Then, if there are query parameters present, we perform a rewrite to /templates/skeleton, which is a special route that shows a skeleton loader.

Here's the middleware code from earlier, with the addition of this logic:

middleware.ts

import { type NextRequest, NextResponse } from 'next/server';

// make sure the middleware only runs when
// the requested url starts with `/templates`
export const config = {
  matcher: ['/templates(.*)'],
};

const THRESHOLD = 0.2; // initial threshold for the new variant (20%)
const COOKIE_NAME = 'tm_var'; // name of the cookie to store the variant

export function middleware(req: NextRequest) {
  // get the variant from the cookie
  // if not found, randomly set a variant based on threshold
  const variant =
    req.cookies.get(COOKIE_NAME) || (Math.random() < THRESHOLD ? 'new' : 'old');

  const url = req.nextUrl.clone();

  // if it's the old variant, rewrite to the old templates marketplace
  if (variant === 'old') {
    url.pathname = '/templates-old';
  } else {
    // for the new variant, we need to perform a rewrite to /template/skeleton
    // if there are query paramters in the request URL
    const hasQueryParams = req.nextUrl.search.length > 0;

    if (hasQueryParams) {
      url.pathname = `/templates/skeleton`;
    }
  }

  const res = NextResponse.rewrite(url);

  // set the variant in the cookie if not already set
  if (!req.cookies.get(COOKIE_NAME)) {
    res.cookies.set(COOKIE_NAME, variant);
  }
  return res;
}

Middleware code to perform an edge rewrite to avoid layout shift.

Then, on the client side, we detect when the data is finished fetching with SWR and render the final list of cards:

templates/[[...slug]].tsx

import useSWR from "swr"
import { useRouter } from "next/router"
import { Card, PlaceholderCard, CardProps } from '@/components/cards'

export default function Templates({ skeleton } : { skeleton: boolean }) {
  const router = useRouter()
  const { data } = useSWR<CardProps[]>("/api/templates")
  
  // if skeleton is true and the router is still loading, show a skeleton
  if (!data || (skeleton && !router.isReady)) {
    return (
      <div>
      	{[...Array(6).keys()].map((i: number) => (
        	<PlaceholderCard key={i}/>
        ))}
      </div>
		)
  }
  
  return (
    <div>
    	{data.map((d: CardProps, i) => (
       <Card key={i} data={d} />
       ))}
    </div>
  )
}

export function getStaticProps(context) {
  const skeleton = context.params?.slug?.includes('skeleton')
  return {
    props: {
      skeleton,
    },
  }
}

Simplified version of the code that shows how we use SWR and middleware to conditionally render loader skeletons on the client side if there are query parameters present in the request URL.

DX Platform

Managed Infrastructure

Open Source

Use Cases

Users

Tools

Company

How to run A/B tests with Next.js and Vercel

Redesigning the Template Marketplace

Catch-all Routes & On-demand Revalidation

Experimenting at the Edge

Avoiding layout shift with Edge Middleware

Collecting data & iterating

Tracking conversion rates with Heap

Tracking search & filter usage Stats with Algolia

Edge Middleware: Run A/B tests without the tradeoffs