Building an AI chat app with RAG and source citations on Vercel

An AI chat app that can answer from your own documents, stream its responses, and cite sources the reader can click to check can be built and deployed in a single Vercel project.

Each of those capabilities has a mechanism behind it:

• Streaming: Vercel Functions on Fluid compute hold a response open for the length of an answer, and Active CPU billing pauses whenever the code awaits I/O, whether a database read or the token stream, so waiting bills no CPU.
• Answering from your documents: retrieval-augmented generation (RAG) runs inside the chat request, a vector lookup over a database provisioned alongside the app, plus a reranking pass through AI Gateway, the same endpoint that serves the model.
• Citations the reader can check: the model cites [id] tags that resolve against the exact sources retrieval returned, and those sources stream to the client with the message that cites them.
• Staying current: the chunking, embedding, and indexing behind the corpus runs on Vercel Workflows as durable steps that retry and survive redeploys, never inside a request.

The examples use Next.js, but nothing here requires it. The AI SDK has bindings for React, Svelte, and Vue, and the rest of the stack works from any framework Vercel supports.

Production architecture for a RAG chatbot on Vercel covers the decision side of the same stack, including when a container platform is still the right call.

Copy link to headingOverview

In this guide, you'll learn how to:

• Map the four layers of a production RAG chat app to the Vercel platform primitives
• Set up a Next.js chat route with AI SDK streaming and clickable source citations
• Retrieve context from a vector database and rerank results through the AI Gateway
• Run document chunking, embedding, and indexing with Vercel Workflows
• Compare the Vercel architecture with a container-based build
• Apply production best practices for timeouts, billing, security, retrieval quality, and ingestion

Copy link to headingArchitecture at a glance

A typical production RAG chat app has four layers, and each maps to a platform primitive.

• UI and chat transport: a Next.js App Router app with the AI SDK useChat hook. Streaming happens over the standard HTTP response, no WebSocket infrastructure required.
• Model access and retrieval quality: AI Gateway, one endpoint across OpenAI, Anthropic, Google, and open-weight models, which also exposes reranking as a modality, a request type alongside chat and embeddings.
• Vector storage and metadata: a vector database connected to the project. Neon with pgvector and Upstash Vector provision natively through the Vercel Marketplace and inject their connection credentials as environment variables. External stores like Pinecone connect with an API key.
• Ingestion and background work: Vercel Workflows for chunk, embed, and index pipelines, built on Vercel Queues underneath.

Copy link to headingPrerequisites

• A Vercel account and a project in the framework of your choice. The examples use Next.js on the App Router.
• The packages the guide uses: npm install ai @ai-sdk/react @ai-sdk/gateway workflow zod.
• A vector database: Neon or Upstash Vector provisioned through the Vercel Marketplace, or an external store reachable with an API key. Create a lib/vector.ts module that wraps your store's query, upsert, and delete-by-prefix calls, since every snippet below imports it.
• Comfort reading TypeScript.

Working with a coding agent? Paste this prompt to have it prepare the project:

Copy link to headingStream grounded answers from the chat route

The chat route does three things in order. It retrieves context for the user's question, tells the model to answer only from that context and cite it, and streams the result. The stream construction follows the AI SDK streaming data reference.

The citation contract lives in the system prompt, so it comes first:

export function buildSystemPrompt(chunks: { id: string; text: string }[]) {
  return [
    'You are a support assistant. Answer using only the sources provided.',
    'Cite sources by their [id] tag inline after each claim.',
    'If the sources do not contain the answer, say so.',
    '',
    'Sources:',
    ...chunks.map((chunk) => `[${chunk.id}] ${chunk.text}`),
  ].join('\n');
}

The system prompt: answer only from the sources, cite by [id].

The [id] tags the prompt demands are the same ids the route streams as source parts, which is what makes rendering a lookup later. The route itself retrieves, writes the source parts, and merges the model's stream into the same response:

import {
  convertToModelMessages,
  createUIMessageStream,
  createUIMessageStreamResponse,
  streamText,
  toUIMessageStream,
} from 'ai';
import { gateway } from '@ai-sdk/gateway';
import { buildSystemPrompt } from '@/lib/rag/prompt';
import { retrieveContext } from '@/lib/rag/retrieve';
import type { UIMessage } from 'ai';

export async function POST(req: Request) {
  const { messages }: { messages: UIMessage[] } = await req.json();
  const query = messages
    .at(-1)!
    .parts.filter((part) => part.type === 'text')
    .map((part) => part.text)
    .join(' ');

  const { chunks, sources } = await retrieveContext(query);

  const stream = createUIMessageStream({
    execute: ({ writer }) => {
      for (const source of sources) {
        writer.write({
          type: 'source',
          value: {
            type: 'source',
            sourceType: 'url',
            id: source.id,
            url: source.url,
            title: source.title,
          },
        });
      }

      const result = streamText({
        model: gateway('openai/gpt-5-mini'),
        system: buildSystemPrompt(chunks),
        messages: convertToModelMessages(messages),
      });

      writer.merge(toUIMessageStream({ stream: result.stream }));
    },
  });

  return createUIMessageStreamResponse({ stream });
}

The chat route: retrieve, write source parts, stream the model into the same response.

• Timeout ceilings: the default maxDuration on Fluid compute is already 300 seconds, 800 seconds is generally available on Pro and Enterprise, 1800 seconds is in beta, and work with no ceiling belongs in Vercel Workflows. Pair long ceilings with an upstream abort so a stalled provider stream does not hold the request open. See configuring duration.
• Billing: Active CPU is the default billing model for I/O-bound functions. CPU bills while code runs and pauses while it awaits a database, an upstream API, or a token stream. Provisioned memory bills for the instance's lifetime, and concurrent chats share one instance.
• Security: this endpoint is public, and every request triggers embedding, reranking, and generation. Protect it before launch with your auth layer, rate limiting, or Vercel Firewall rules.

Routing the model call through AI Gateway adds automatic provider retries and fallbacks, spend monitoring, and a one-line model change that never touches retrieval code. You can swap the model string for any model in the catalog.

Copy link to headingRetrieve and rerank the context

Vector search returns chunks by embedding similarity, which is a fast approximation of relevance. A reranker scores each query-document pair directly, so it catches matches that similarity misses. The pattern is to over-fetch and prune. Pull 20 candidates from the vector index, let the reranker keep the best 5, and send only those to the model. The call shapes follow the AI Gateway reranking reference.

import { embed, rerank } from 'ai';
import { gateway } from '@ai-sdk/gateway';
import { vectorIndex } from '@/lib/vector';

export async function retrieveContext(query: string) {
  const { embedding } = await embed({
    model: gateway.textEmbeddingModel('openai/text-embedding-3-small'),
    value: query,
  });

  const candidates = await vectorIndex.query({
    vector: embedding,
    topK: 20,
    includeMetadata: true,
  });

  const { ranking } = await rerank({
    model: gateway.rerankingModel('cohere/rerank-v4-fast'),
    query,
    documents: candidates.map((c) => c.metadata.text),
    topN: 5,
  });

  const top = ranking.map((r) => candidates[r.originalIndex]);

  return {
    chunks: top.map((c) => ({ id: c.id, text: c.metadata.text })),
    sources: top.map((c) => ({ id: c.id, title: c.metadata.title, url: c.metadata.url })),
  };
}

Embed the query, over-fetch candidates, rerank to the top five.

• Latency: the rerank call is a serial hop between retrieval and generation, so it adds to time to first token on every request. It helps most on large or noisy corpora and on top of cheap embedding models. Skip it, or rerank only long candidate lists, when your corpus is small and curated.
• Multi-turn queries: the raw last message is not always the query. In a longer conversation, "what about the enterprise plan?" retrieves nothing useful on its own, so production apps often condense the conversation into a standalone query with a small model call before embedding. Start without it, add it when multi-turn quality drops.

Copy link to headingResolve citations against retrieved sources

The route streamed the sources as parts, and the system prompt made the model cite by [id]. On the client, the [id] tags in the text resolve against the source-url parts on the same message, so rendering is a lookup, and a citation can only point at a source retrieval returned. Hook shapes follow the AI SDK chatbot reference.

The lookup itself is a small helper. An [id] with a matching source becomes a link, and an [id] the model invents matches nothing and stays plain text, visibly broken instead of clickable:

export function renderWithCitations(
  text: string,
  sources: Map<string, { url: string; title?: string }>,
) {
  return text.split(/(\[[^\]]+\])/g).map((segment, i) => {
    const match = segment.match(/^\[([^\]]+)\]$/);
    const source = match ? sources.get(match[1]) : undefined;
    if (!source) return <span key={i}>{segment}</span>;
    return (
      <a key={i} href={source.url} title={source.title} target="_blank" rel="noreferrer">
        {segment}
      </a>
    );
  });
}

An [id] tag becomes a link only when a matching source exists.

The page builds each message's source map from that message's own parts, then renders its text through the helper:

'use client';
import { useChat } from '@ai-sdk/react';
import { DefaultChatTransport } from 'ai';
import { useState } from 'react';
import { renderWithCitations } from './citations';

export default function ChatPage() {
  const { messages, sendMessage, status } = useChat({
    transport: new DefaultChatTransport({ api: '/api/chat' }),
  });
  const [input, setInput] = useState('');

  return (
    <div>
      {messages.map((message) => {
        const sources = new Map(
          message.parts
            .filter((part) => part.type === 'source-url')
            .map((part) => [part.id, { url: part.url, title: part.title }]),
        );

        return (
          <div key={message.id}>
            <strong>{message.role === 'user' ? 'You' : 'Assistant'}</strong>{' '}
            {message.parts.map((part, i) =>
              part.type === 'text' ? (
                <span key={i}>{renderWithCitations(part.text, sources)}</span>
              ) : null,
            )}
          </div>
        );
      })}
      <form
        onSubmit={(event) => {
          event.preventDefault();
          if (!input.trim()) return;
          sendMessage({ text: input });
          setInput('');
        }}
      >
        <input
          value={input}
          onChange={(event) => setInput(event.target.value)}
          disabled={status !== 'ready'}
        />
      </form>
    </div>
  );
}

Each message resolves its own [id] tags against its own source parts.

Each message carries its own sources, so turn five's citations can never resolve against turn one's documents.

Copy link to headingRun ingestion on Vercel Workflows

Chunking a large PDF, embedding hundreds of chunks, and writing them to the index is work that outlives any request, so it runs as a workflow instead. Functions marked 'use workflow' orchestrate. Each helper marked 'use step' becomes a durable, retried unit that survives redeploys, and by default, a failed step retries up to three times without redoing the steps before it.

import { fetchDocument, chunkDocument } from '@/lib/rag/documents';
import { embedChunks, indexChunks } from './steps';

export async function ingestDocument(documentId: string) {
  'use workflow';

  const doc = await fetchDocument(documentId);
  const chunks = await chunkDocument(doc, { size: 800, overlap: 100 });
  const embedded = await embedChunks(chunks);
  await indexChunks(documentId, doc, embedded);
}

The workflow reads as the pipeline it is: fetch, chunk, embed, index.

The steps carry the work. Each one is a durable, retriable unit:

import { embedMany } from 'ai';
import { gateway } from '@ai-sdk/gateway';
import { vectorIndex } from '@/lib/vector';

export async function embedChunks(chunks: { index: number; text: string }[]) {
  'use step';
  const { embeddings } = await embedMany({
    model: gateway.textEmbeddingModel('openai/text-embedding-3-small'),
    values: chunks.map((chunk) => chunk.text),
  });
  return chunks.map((chunk, i) => ({ ...chunk, embedding: embeddings[i] }));
}

export async function indexChunks(
  documentId: string,
  doc: { title: string; url: string },
  embedded: { index: number; text: string; embedding: number[] }[],
) {
  'use step';
  await vectorIndex.deleteByPrefix(`${documentId}:`);
  await vectorIndex.upsert(
    embedded.map((chunk) => ({
      id: `${documentId}:${chunk.index}`,
      vector: chunk.embedding,
      metadata: { text: chunk.text, title: doc.title, url: doc.url },
    })),
  );
}

Embed in one batch, then delete and re-upsert the document's chunks.

Start a run from the upload endpoint or a cron. Directive syntax follows the Workflow SDK docs.

import { start } from 'workflow/api';
import { ingestDocument } from '@/app/workflows/ingest';

export async function POST(req: Request) {
  const { documentId } = await req.json();
  await start(ingestDocument, [documentId]);
  return Response.json({ status: 'queued' });
}

The upload handler returns immediately and the workflow does the heavy work. fetchDocument and chunkDocument carry the step directive in their own module.

• Batched embeddings: embedMany embeds all chunks in one step, so a provider rate limit retries one batch call instead of replaying a per-chunk loop from the start. Shard into multiple steps for very large documents so retry granularity matches failure granularity.
• Deterministic IDs, delete before upsert: IDs like documentId:index make retried writes idempotent, which durable delivery requires, and deleting the document's chunks before upserting keeps a re-ingested, shorter document from stranding orphans that retrieval would keep returning.
• A thin upload handler: parsing PDFs in the upload request blocks the user and hits request limits. The handler's only job is to enqueue.

Copy link to headingHow the stack compares to a container build

The same four needs exist on any platform. The difference is how much arrives assembled.

The container column is a real architecture that works. Every row of it is also a component to build, secure, and pay for before the first chat ships.

A container that streams answers is allocated and billed for the life of the instance, whether tokens are flowing or not, and platforms that bill per request still allocate CPU for the full request duration. On Fluid compute, Active CPU bills while code executes and pauses while the function awaits the token stream, and one instance serves many concurrent chats. A chat request spends most of its wall-clock time awaiting tokens, which is exactly the case this billing model favors.

Portability cuts both ways. The chat route, retrieval, and UI in this guide are plain AI SDK and Next.js code that run anywhere Node.js runs. The Workflows pipeline is Vercel-specific, and its container equivalent is the worker service, queue, and retry logic in the table. Vercel is not a replacement for every containerized workload. What it removes for this one is the worker service, the queue backing store, and the streaming plumbing.

Copy link to headingBest practices

• Keep the retrieval prompt strict: cite by [id], refuse to answer outside the sources. Reranking only helps if the model actually uses the top chunks.
• Route resilience through AI Gateway: provider fallbacks and retries keep a provider outage from taking the chat down, and spend monitoring catches runaway usage.
• Chunk with overlap: 600 to 1000 tokens with 10 to 15 percent overlap is a solid starting point for documentation and support content. Tune against your corpus.
• Match ingest and query embedding models: a mismatch degrades retrieval silently. Change both together, and re-embed when you change either.
• Rebuild, don't accumulate: re-ingest changed documents with delete-before-upsert so stale chunks leave the index when the source shrinks.

Copy link to headingNext steps

• AI SDK docs: the useChat hook, UI message streams, and source parts used throughout this guide
• AI Gateway reranking: the rerank() call, supported models, and the Cohere-compatible REST endpoints
• Cost-aware model routing with AI Gateway: picking and switching models without overpaying for easy requests
• Vercel Workflows: durable steps and observability for the ingestion pipeline
• Understanding vector databases for AI apps: how to choose and operate the store this stack retrieves from
• Production architecture for a RAG chatbot on Vercel: the decision-level view of this stack, including when containers still win
• The Agent Stack: the assembled pieces this guide draws on, with guides for agent-shaped workloads
• What is retrieval-augmented generation (RAG): the concepts behind the pipeline you just built