How to run A/B tests with Next.js and Vercel

Running A/B tests is hard.

We all know how important it is for our business–it helps us understand how users are interacting with our products in the real world.

However, a lot of the A/B testing solutions are done on the client side, which introduces layout shift as variants are dynamically injected after the initial page load. This negatively impacts your websites performance and creates a subpar user experience.

To get the best of both worlds, we built Edge Middleware: code that runs before serving requests from the edge cache. This enables developers to perform rewrites at the edge to show different variants of the same page to different users.

Today, we'll take a look at a real-world example of how we used Edge Middleware to A/B test our new Templates page.

Link to headingRedesigning the Template Marketplace

Earlier this year, we kickstarted the redesign of our Templates page.

Our old Templates page had a limited scope that focused mainly on the different frameworks that were supported on Vercel:

Old version: limited selection of templates that focused mainly on framework starters.

For the new version, we wanted to build a marketplace to showcase all the different types of applications that can be built on Vercel – blogs, e-commerce storefronts, etc.

We also improved search to allow for partial matching of search terms and added category filters to make discovering new templates much easier:

New version: wider selection of templates with fuzzy search & filters.

Since this was a major change of one of our most popular pages, we decided to break down the launch into multiple release phases:

Early Access: Only 20% of visitors to /templates will see the new variant.
Public Beta: An equal 50:50 split between the new and old variant.
General Availability: Public launch–everyone gets the new variant.

We used Vercel's Edge Middleware to stagger our launch into 3 phases.

We leveraged Edge Middleware to perform the A/B test at the edge, which allowed us to gather insightful user feedback without sacrificing performance.

Link to headingCatch-all Routes & On-demand Revalidation

Since we were using Next.js, we decided to create the new templates marketplace using the following optional catch-all route: /templates/[[...slug]].tsx.

This allowed us to programmatically generate new static pages when new templates are added using On-Demand Incremental Static Regeneration (ISR) and navigate between them using shallow-routing.

Here are some examples:

/templates/next.js/nextjs-boilerplate: Next.js boilerplate template
/templates/svelte/sveltekit-boilerplate: SvelteKit boilerplate template

We also generated dynamic landing pages for each filter option to capture long-tail SEO traffic:

/templates/next.js: Next.js templates
/templates/blog: Blog templates
/templates/tailwind: Templates that use Tailwind CSS

We then moved the old templates marketplace to the /templates-old.tsx route.

Link to headingExperimenting at the Edge

To perform the A/B test, we created a middleware.ts file with the following code:

middleware.ts

import { type NextRequest, NextResponse } from 'next/server';

// make sure the middleware only runs when
// the requested url starts with `/templates`
export const config = {
  matcher: ['/templates(.*)'],
};

const THRESHOLD = 0.2; // initial threshold for the new variant (20%)
const COOKIE_NAME = 'tm_var'; // name of the cookie to store the variant

export function middleware(req: NextRequest) {
  // get the variant from the cookie
  // if not found, randomly set a variant based on threshold
  const variant =
    req.cookies.get(COOKIE_NAME) || (Math.random() < THRESHOLD ? 'new' : 'old');

  const url = req.nextUrl.clone();

  // if it's the old variant, rewrite to the old templates marketplace
  if (variant === 'old') {
    url.pathname = '/templates-old';
  }

  const res = NextResponse.rewrite(url);

  // set the variant in the cookie if not already set
  if (!req.cookies.get(COOKIE_NAME)) {
    res.cookies.set(COOKIE_NAME, variant);
  }
  return res;
}

Middleware code to perform the A/B test at the edge.

Here's a step by step of what the middleware does:

Tags each new visitor to the /templates page with either the old or new variant based on the current threshold (0.2 for the first stage)
If the user is part of the old variant, performs a rewrite to the /templates-old.tsx route
Saves the user's variant in the tm_var cookie to ensure the user gets served the same version in subsequent visits

Link to headingAvoiding layout shift with Edge Middleware

As we developed the new templates page, we ran into an interesting problem–data fetching had to be done on the client side for three reasons:

We were using query parameters to preserve the filter state
The pages were generated statically
You can't read query parameters inside getStaticProps

To avoid layout shift, we decided to use skeleton loaders to buffer the loading state before showing the templates that match the configured filters.

For instance, if you go to vercel.com/templates?framework=svelte, you first receive a set of skeleton loaders while the data is fetched on the client side, and then the list of Svelte templates:

However, for the dynamic landing pages for each filter option (e.g. vercel.com/templates/svelte), we wanted to avoid skeleton loaders since we already know the set of templates that match the given filter at build time.

To get the best of both worlds, we used Edge Middleware. Since Edge Middleware is executed before a request is processed on a site, we are able to detect if there are any query parameters present right when the user requests the page.

Then, if there are query parameters present, we perform a rewrite to /templates/skeleton, which is a special route that shows a skeleton loader.

Here's the middleware code from earlier, with the addition of this logic:

middleware.ts

import { type NextRequest, NextResponse } from 'next/server';

// make sure the middleware only runs when
// the requested url starts with `/templates`
export const config = {
  matcher: ['/templates(.*)'],
};

const THRESHOLD = 0.2; // initial threshold for the new variant (20%)
const COOKIE_NAME = 'tm_var'; // name of the cookie to store the variant

export function middleware(req: NextRequest) {
  // get the variant from the cookie
  // if not found, randomly set a variant based on threshold
  const variant =
    req.cookies.get(COOKIE_NAME) || (Math.random() < THRESHOLD ? 'new' : 'old');

  const url = req.nextUrl.clone();

  // if it's the old variant, rewrite to the old templates marketplace
  if (variant === 'old') {
    url.pathname = '/templates-old';
  } else {
    // for the new variant, we need to perform a rewrite to /template/skeleton
    // if there are query paramters in the request URL
    const hasQueryParams = req.nextUrl.search.length > 0;

    if (hasQueryParams) {
      url.pathname = `/templates/skeleton`;
    }
  }

  const res = NextResponse.rewrite(url);

  // set the variant in the cookie if not already set
  if (!req.cookies.get(COOKIE_NAME)) {
    res.cookies.set(COOKIE_NAME, variant);
  }
  return res;
}

Middleware code to perform an edge rewrite to avoid layout shift.

Then, on the client side, we detect when the data is finished fetching with SWR and render the final list of cards:

templates/[[...slug]].tsx

import useSWR from "swr"
import { useRouter } from "next/router"
import { Card, PlaceholderCard, CardProps } from '@/components/cards'

export default function Templates({ skeleton } : { skeleton: boolean }) {
  const router = useRouter()
  const { data } = useSWR<CardProps[]>("/api/templates")
  
  // if skeleton is true and the router is still loading, show a skeleton
  if (!data || (skeleton && !router.isReady)) {
    return (
      <div>
      	{[...Array(6).keys()].map((i: number) => (
        	<PlaceholderCard key={i}/>
        ))}
      </div>
		)
  }
  
  return (
    <div>
    	{data.map((d: CardProps, i) => (
       <Card key={i} data={d} />
       ))}
    </div>
  )
}

export function getStaticProps(context) {
  const skeleton = context.params?.slug?.includes('skeleton')
  return {
    props: {
      skeleton,
    },
  }
}

Simplified version of the code that shows how we use SWR and middleware to conditionally render loader skeletons on the client side if there are query parameters present in the request URL.

Link to headingCollecting data & iterating

Throughout our Early Access and Public Beta phases, we collected a series of data points that helped us understand how our new Templates Marketplace was performing in the real world.

Link to headingTracking conversion rates with Heap

We used Heap to track our conversion rates: how likely is a user to successfully deploy a template after seeing it on the new vs. old Templates pages.

Based on the data from Heap, we were able to make some tweaks to our implementation to increase conversion rates by 16% with the new variant.

Link to headingTracking search & filter usage Stats with Algolia

Algolia's Search Without Results feature was also instrumental for us to understand which templates users were interested in. Based on the data, we were getting >30 searches a week for WordPress templates that yielded no results.

This prompted us to add a few WordPress templates before releasing the Templates Marketplace to General Availability.

Link to headingEdge Middleware: Run A/B tests without the tradeoffs

Edge Middleware allowed us to streamline the release of our new Templates Marketplace by putting the new version behind a feature flag until it's fully tested and optimized.

Check out our Templates Marketplace today to jumpstart your app development process with our pre-built solutions.

Agent Stack

Core Platform

Tools

Learn

Build

Explore