Production architecture for a RAG chatbot on Vercel

Behind the chat window, a chatbot built with retrieval-augmented generation (RAG) is three backend jobs. A streaming API holds a response open for the length of every answer, retrieval and reranking run inside that same request, and a vector database has to answer the retrieval query fast enough that nobody notices it happened. All three, plus the UI, can run as one deployment on Vercel.

Two mechanisms make that work. Vercel Functions on Fluid compute hold the stream and bill on Active CPU, which pauses whenever the code awaits tokens, so a thirty-second answer costs the compute of the work in it, not thirty seconds of wall clock. And the vector database provisions alongside the app, in the same region as the functions, so the retrieval query is one intra-region round trip.

Building an AI chat app with RAG and source citations on Vercel walks through the build itself, with the full code.

Copy link to headingContents

• What the four layers are, and what runs each one
• How long-lived streams work on Fluid compute
• How citations stream before the first token
• How one gateway carries generation and reranking
• How region pinning answers retrieval latency
• When containers are still the right call
• How to migrate from a split deployment
• What the reference architecture includes

Copy link to headingThe four layers of a RAG chatbot on Vercel

Each layer has a job, a requirement that job imposes, and a home on the platform.

Three layers deploy as one application (Next.js in these examples).

The request path stays short, with the client reaching the function through the nearest point of the global network and the function reaching the vector store inside one cloud region.

Copy link to headingLong-lived streams on Fluid compute

A single chat request holds an open Server-Sent Events connection for the duration of retrieval, reranking, model streaming, and citation emission, which can run to tens of seconds. On per-invocation serverless, that is one full instance per user for the length of the response, mostly parked on await.

Fluid compute changes the economics through two separate mechanisms. Active CPU billing means the CPU meter runs while code executes and pauses whenever the function awaits I/O, whether a database read, an upstream API, or a token stream. In-instance concurrency means one function instance serves multiple concurrent streams, which spreads the instance's provisioned-memory cost across every open chat.

maxDuration defaults to 300 seconds on Fluid compute, 800 seconds is generally available on Pro and Enterprise, 1800 seconds is in beta, and work with no ceiling belongs in Vercel Workflows. A stream that reaches the ceiling ends mid-answer, so pair a long ceiling with an upstream abort for stalled providers. See configuring duration.

Vercel Functions support Node.js streaming responses and the Web ReadableStream API, which is what the AI SDK writes into. There is no separate streaming runtime to provision.

Copy link to headingCitations stream before the first token

The client uses useChat from @ai-sdk/react and renders citations from source-url parts on each message. On the server, retrieval and reranking come first:

import { rerank } from 'ai';
import { gateway } from '@ai-sdk/gateway';
import { vectorStore } from '@/lib/vector';

export async function retrieveTop(query: string) {
  // Scope the query to the requesting user's document permissions
  const candidates = await vectorStore.query(query);

  const { ranking } = await rerank({
    model: gateway.rerankingModel('cohere/rerank-v4-fast'),
    query,
    documents: candidates.map((c) => c.text),
    topN: 5,
  });

  return ranking.map((r) => candidates[r.originalIndex]);
}

Query the store, rerank the candidates, return the top five.

The route writes the retrieved sources into the stream as parts, then merges the model's token stream into the same response. Shapes follow the AI SDK streaming data reference.

import {
  convertToModelMessages,
  createUIMessageStream,
  createUIMessageStreamResponse,
  streamText,
  toUIMessageStream,
} from 'ai';
import { gateway } from '@ai-sdk/gateway';
import { retrieveTop } from '@/lib/retrieval';
import { buildSystemPrompt } from '@/lib/prompt';
import type { UIMessage } from 'ai';

export async function POST(req: Request) {
  const { messages }: { messages: UIMessage[] } = await req.json();
  const query = messages
    .at(-1)!
    .parts.filter((part) => part.type === 'text')
    .map((part) => part.text)
    .join(' ');

  const top = await retrieveTop(query);

  const stream = createUIMessageStream({
    execute: ({ writer }) => {
      for (const source of top) {
        writer.write({
          type: 'source',
          value: {
            type: 'source',
            sourceType: 'url',
            id: source.id,
            url: source.url,
            title: source.title,
          },
        });
      }

      const result = streamText({
        model: gateway('openai/gpt-5-mini'),
        system: buildSystemPrompt(top),
        messages: convertToModelMessages(messages),
      });

      writer.merge(toUIMessageStream({ stream: result.stream }));
    },
  });

  return createUIMessageStreamResponse({ stream });
}

Source parts go out before generation begins, and the token stream merges into the same response.

The user sees where the answer will come from while the model is still writing it, and because each source is attached to the message that cites it, every turn's citations resolve against that turn's retrieval.

Copy link to headingOne gateway for generation and reranking

AI Gateway runs on the same platform as the functions that call it, so generation and reranking arrive through one endpoint, one authentication surface, and one spend and observability view, with automatic retries and fallbacks across providers. Swapping the generation model is a one-line edit that never touches retrieval code.

Rerank models from multiple providers are in the catalog, and swapping one is the same one-line edit. See the reranking reference for the call shapes.

Copy link to headingPin the vector store to the function region

Retrieval latency is the strongest objection to this design, and it goes away when the vector store runs in the same region as the functions that query it. Neon and Upstash Vector let you choose the provider region when you create the database, so pick the region your functions run in and the vector query is one intra-region round trip. Time it from a deployed function before committing. Intra-region reads give the same region-level locality a container next to the database gets.

Locality settles the latency question, and Active CPU billing changes what the waiting costs. A 20-second streamed answer bills for the fraction of a second of CPU it used, plus provisioned memory shared across every concurrent stream, where an always-on server bills for all 20 seconds of every stream it holds. For bursty, idle-heavy chat traffic, that difference dominates the bill.

For document metadata, run Postgres in the same region. With Neon, pgvector and relational tables share one database, so retrieval and metadata joins are a single round trip. Upstash Vector provisions natively as well, and external stores such as Pinecone connect with an API key, in whichever of the provider's regions you select at creation.

Copy link to headingWhen containers are still the right call

Use containers on Azure Container Apps, AWS ECS or Fargate, or Google Cloud Run when:

• Guaranteed warm caches: Fluid compute instances persist and reuse in-process caches across invocations, but instance lifetime is not guaranteed. If cache-miss cost dominates request latency and the cache must be warmed at boot and held for hours, a container you control wins.
• Self-hosted GPU models: a custom embedding model on GPUs you host is container territory.
• VPC-resident compute: for the outbound direction, reaching services inside a private VPC from Vercel Functions, Secure Compute provides a dedicated private network with VPC peering on Enterprise. When policy requires the compute itself to reside inside your VPC, containers are the answer.
• Platform gravity: a mature container platform that already owns identity, networking, and deploys, which the chatbot must inherit. That is a requirement like any other, and fighting it costs more than the consolidation saves.

If none of those conditions apply, a separate container tier adds an operational surface without a latency gain, and without a capability this workload uses.

Copy link to headingMigrating from a split deployment

A split deployment puts the chat UI on one platform and the retrieval API in containers next to the vector database, which means a second deployment surface with its own TLS, CORS, and firewall configuration, a second identity plane, and a second observability stack. Consolidating removes those, and the move can happen incrementally:

1. Move the API routes first: the chat route and retrieval logic in this guide are plain AI SDK application code that fits the existing Vercel project serving the UI. If the current API is Python or another stack, this step is a port rather than a copy, and the container API stays up while both run.
2. Keep the vector store where it is, then re-region it: the function can query the existing store cross-region during the transition, at a latency cost you can measure. Moving the store last means either a provider-side region move or a re-index, and a re-index means re-embedding the corpus, which costs embedding calls and a cutover window.
3. Roll back at the route level: while both APIs exist, the rollback is pointing the UI's transport back at the container endpoint, a one-line change.

The chat and retrieval code stays plain AI SDK code afterward, and runs anywhere Node.js does. The Vercel-specific pieces are Fluid compute billing and the ingestion pipeline, the parts a split deployment assembles from a worker service and a queue.

Copy link to headingReference architecture

The full stack for a customer-support chatbot with citations. The UI, the chat API, and the retrieval logic deploy, authenticate, and scale as one application. Next.js is the worked example, and any framework Vercel supports fits the same shape.

• Frontend: Next.js App Router, AI SDK useChat, rendering source-url parts as citation links.
• Chat API: a route handler on Vercel Functions with Fluid compute, maxDuration sized to the longest response, streaming UI message parts.
• Retrieval: vector query against a store in the same region as the function, scoped to the requesting user's permissions so no one receives a citation to a document they cannot open.
• Reranking: rerank() through AI Gateway with a reranking model from the catalog.
• Generation: streamText through AI Gateway, model selected per environment or per user tier.
• Ingestion: chunking, embedding, and indexing run on Vercel Workflows as durable steps. The build guide walks through the pipeline.
• Source metadata: Postgres in the same region for document URLs, titles, permissions, and audit records.
• Observability: Vercel Observability for function traces, AI Gateway for prompt and completion logs. Support conversations contain customer data, so set retention and redaction policy for those logs before launch.
• Identity: your auth provider for end users, and OIDC federation in place of long-lived cloud keys for backend calls that need cloud IAM.

Copy link to headingFAQ

Does a chat stream cost compute while it waits on the model? Active CPU pauses during I/O waits, including token streams, so no CPU bills while a response waits. Provisioned memory bills for the instance, which concurrent streams share.

What happens when a response hits maxDuration? The stream ends at the ceiling (300 seconds by default on Fluid compute, 800 generally available on Pro and Enterprise, 1800 in beta). Size the ceiling to your longest answer and abort stalled upstream streams early.

Which vector databases support region pinning? Native Marketplace integrations (Neon, Upstash Vector) let you choose the provider region at creation. External stores like Pinecone connect by API key in whichever of their regions you select. Match the function region either way.

When do I still need a container platform? GPU-hosted models, compute that must reside inside your VPC, guaranteed-lifetime warm caches, or an existing container platform the chatbot must inherit. For reaching into a VPC from Vercel, see Secure Compute.

Copy link to headingNext steps

• Building an AI chat app with RAG and source citations on Vercel: the build walkthrough for this architecture, with the full code
• Fluid compute: the concurrency and Active CPU billing model this architecture leans on
• Streaming functions: response streaming behavior and limits
• AI Gateway reranking: the rerank API, supported models, and REST endpoints
• AI SDK chatbot docs: useChat, transports, and message parts
• The Agent Stack: the assembled pieces behind this stack, with guides for agent-shaped workloads
• What is retrieval-augmented generation (RAG): the concepts underneath the architecture