--- title: Building an AI chat app with RAG and source citations on Vercel description: A production stack for AI chat with retrieval, reranking, source citations, and background ingestion on Vercel using Next.js and the AI SDK. url: /kb/guide/building-ai-chat-app-with-rag-and-citations-on-vercel canonical_url: "https://vercel.com/kb/guide/building-ai-chat-app-with-rag-and-citations-on-vercel" last_updated: 2026-07-29 authors: Vercel related: - /docs/functions - /docs/fluid-compute - /docs/ai-gateway - /docs/workflows - /docs/frameworks - /kb/guide/rag-chatbot-production-architecture-on-vercel - /docs/ai-gateway/modalities/reranking - /docs/queues - /docs/functions/configuring-functions/duration - /docs/vercel-firewall - /docs/deployments/preview-deployments - /kb/guide/cost-aware-model-routing-with-ai-gateway - /kb/guide/understanding-vector-databases-for-ai-apps - /blog/agent-stack - /kb/agent-stack - /kb/guide/what-is-rag install_vercel_plugin: npx plugins add vercel/vercel-plugin --- An AI chat app that can answer from your own documents, stream its responses, and cite sources the reader can click to check can be built and deployed in a single Vercel project. Each of those capabilities has a mechanism behind it: - **Streaming:** [Vercel Functions](https://vercel.com/docs/functions) on [Fluid compute](https://vercel.com/docs/fluid-compute) hold a response open for the length of an answer, and [Active CPU](https://vercel.com/docs/fluid-compute) billing pauses whenever the code awaits I/O, whether a database read or the token stream, so waiting bills no CPU. - **Answering from your documents:** retrieval-augmented generation (RAG) runs inside the chat request, a vector lookup over a database provisioned alongside the app plus a reranking pass through [AI Gateway](https://vercel.com/docs/ai-gateway), the same endpoint that serves the model. - **Citations the reader can check:** the model cites `[id]` tags that resolve against the exact sources retrieval returned, and those sources stream to the client with the message that cites them. - **Staying current:** the chunking, embedding, and indexing behind the corpus runs on [Vercel Workflows](https://vercel.com/docs/workflows) as durable steps that retry and survive redeploys, never inside a request. The examples use [Next.js](https://nextjs.org/), but nothing here requires it. The [AI SDK](https://ai-sdk.dev/) has bindings for React, Svelte, and Vue, and the rest of the stack works from [any framework Vercel supports](https://vercel.com/docs/frameworks). [Production architecture for a RAG chatbot on Vercel](https://vercel.com/kb/guide/rag-chatbot-production-architecture-on-vercel) covers the decision side of the same stack, including when a container platform is still the right call. ## Contents - [What the four layers are, and what runs each one](#architecture-at-a-glance) - [What to set up before you start](#prerequisites) - [How to stream grounded answers from the chat route](#stream-grounded-answers-from-the-chat-route) - [How to retrieve and rerank the context](#retrieve-and-rerank-the-context) - [How citations resolve against retrieved sources](#resolve-citations-against-retrieved-sources) - [How to run ingestion on Vercel Workflows](#run-ingestion-on-vercel-workflows) - [How the stack compares to a container build](#how-the-stack-compares-to-a-container-build) - [Best practices](#best-practices) ## Architecture at a glance A typical production RAG chat app has four layers, and each maps to a platform primitive. - **UI and chat transport:** a Next.js App Router app with the AI SDK `useChat` hook. Streaming happens over the standard HTTP response, no websocket infrastructure required. - **Model access and retrieval quality:** AI Gateway, one endpoint across OpenAI, Anthropic, Google, and open-weight models, which also exposes [reranking](https://vercel.com/docs/ai-gateway/modalities/reranking) as a modality, a request type alongside chat and embeddings. - **Vector storage and metadata:** a vector database connected to the project. [Neon](https://vercel.com/marketplace/neon) with pgvector and [Upstash](https://vercel.com/marketplace/upstash) Vector provision natively through the [Vercel Marketplace](https://vercel.com/marketplace) and inject their connection credentials as environment variables. External stores like Pinecone connect with an API key. - **Ingestion and background work:** Vercel Workflows for chunk, embed, and index pipelines, built on [Vercel Queues](https://vercel.com/docs/queues) underneath. ## Prerequisites - A Vercel account and a project in the framework of your choice. The examples use Next.js on the App Router. - The packages the guide uses: `npm install ai @ai-sdk/react @ai-sdk/gateway workflow zod`. - A vector database: Neon or Upstash Vector provisioned through the [Vercel Marketplace](https://vercel.com/marketplace), or an external store reachable with an API key. Create a `lib/vector.ts` module that wraps your store's query, upsert, and delete-by-prefix calls, since every snippet below imports it. - Comfort reading TypeScript. Working with a coding agent? Paste this prompt to have it prepare the project: ### Agent prompt ```txt Set up a Next.js App Router project for the guide at vercel.com/kb/guide/building-ai-chat-app-with-rag-and-citations-on-vercel. Install ai, @ai-sdk/react, @ai-sdk/gateway, workflow, and zod. Add a lib/vector.ts wrapper for my vector database with query, upsert, and delete-by-prefix functions, and confirm AI Gateway access works with a one-line generateText call. ``` ## Stream grounded answers from the chat route The chat route does three things in order. It retrieves context for the user's question, tells the model to answer only from that context and cite it, and streams the result. The stream construction follows the [AI SDK streaming data reference](https://ai-sdk.dev/docs/ai-sdk-ui/streaming-data). The citation contract lives in the system prompt, so it comes first: ``export function buildSystemPrompt(chunks: { id: string; text: string }[]) { return [ 'You are a support assistant. Answer using only the sources provided.', 'Cite sources by their [id] tag inline after each claim.', 'If the sources do not contain the answer, say so.', '', 'Sources:', ...chunks.map((chunk) => `[${chunk.id}] ${chunk.text}`), ].join('\n'); }`` _The system prompt: answer only from the sources, cite by_ `_[id]_`_._ The `[id]` tags the prompt demands are the same ids the route streams as source parts, which is what makes rendering a lookup later. The route itself retrieves, writes the source parts, and merges the model's stream into the same response: `import { convertToModelMessages, createUIMessageStream, createUIMessageStreamResponse, streamText, toUIMessageStream, } from 'ai'; import { gateway } from '@ai-sdk/gateway'; import { buildSystemPrompt } from '@/lib/rag/prompt'; import { retrieveContext } from '@/lib/rag/retrieve'; import type { UIMessage } from 'ai'; export async function POST(req: Request) { const { messages }: { messages: UIMessage[] } = await req.json(); const query = messages .at(-1)! .parts.filter((part) => part.type === 'text') .map((part) => part.text) .join(' '); const { chunks, sources } = await retrieveContext(query); const stream = createUIMessageStream({ execute: ({ writer }) => { for (const source of sources) { writer.write({ type: 'source', value: { type: 'source', sourceType: 'url', id: source.id, url: source.url, title: source.title, }, }); } const result = streamText({ model: gateway('openai/gpt-5-mini'), system: buildSystemPrompt(chunks), messages: convertToModelMessages(messages), }); writer.merge(toUIMessageStream({ stream: result.stream })); }, }); return createUIMessageStreamResponse({ stream }); }` _The chat route: retrieve, write source parts, stream the model into the same response._ - **Timeout ceilings:** the default `maxDuration` on Fluid compute is already 300 seconds, 800 seconds is generally available on Pro and Enterprise, 1800 seconds is in beta, and work with no ceiling belongs in [Vercel Workflows](https://vercel.com/docs/workflows). Pair long ceilings with an upstream abort so a stalled provider stream does not hold the request open. See [configuring duration](https://vercel.com/docs/functions/configuring-functions/duration). - **Billing:** [Active CPU](https://vercel.com/docs/fluid-compute) is the default billing model for I/O-bound functions. CPU bills while code runs and pauses while it awaits a database, an upstream API, or a token stream. Provisioned memory bills for the instance's lifetime, and concurrent chats share one instance. - **Security:** this endpoint is public, and every request triggers embedding, reranking, and generation. Protect it before launch with your auth layer, rate limiting, or [Vercel Firewall](https://vercel.com/docs/vercel-firewall) rules. Routing the model call through AI Gateway adds automatic provider retries and fallbacks, spend monitoring, and a one-line model change that never touches retrieval code. You can swap the model string for any model in the [catalog](https://vercel.com/ai-gateway/models). ## Retrieve and rerank the context Vector search returns chunks by embedding similarity, which is a fast approximation of relevance. A reranker scores each query-document pair directly, so it catches matches that similarity misses. The pattern is to over-fetch and prune. Pull 20 candidates from the vector index, let the reranker keep the best 5, and send only those to the model. The call shapes follow the [AI Gateway reranking reference](https://vercel.com/docs/ai-gateway/modalities/reranking). `import { embed, rerank } from 'ai'; import { gateway } from '@ai-sdk/gateway'; import { vectorIndex } from '@/lib/vector'; export async function retrieveContext(query: string) { const { embedding } = await embed({ model: gateway.textEmbeddingModel('openai/text-embedding-3-small'), value: query, }); const candidates = await vectorIndex.query({ vector: embedding, topK: 20, includeMetadata: true, }); const { ranking } = await rerank({ model: gateway.rerankingModel('cohere/rerank-v4-fast'), query, documents: candidates.map((c) => c.metadata.text), topN: 5, }); const top = ranking.map((r) => candidates[r.originalIndex]); return { chunks: top.map((c) => ({ id: c.id, text: c.metadata.text })), sources: top.map((c) => ({ id: c.id, title: c.metadata.title, url: c.metadata.url })), }; }` _Embed the query, over-fetch candidates, rerank to the top five._ - **Latency:** the rerank call is a serial hop between retrieval and generation, so it adds to time to first token on every request. It helps most on large or noisy corpora and on top of cheap embedding models. Skip it, or rerank only long candidate lists, when your corpus is small and curated. - **Multi-turn queries:** the raw last message is not always the query. In a longer conversation, "what about the enterprise plan?" retrieves nothing useful on its own, so production apps often condense the conversation into a standalone query with a small model call before embedding. Start without it, add it when multi-turn quality drops. ## Resolve citations against retrieved sources The route streamed the sources as parts, and the system prompt made the model cite by `[id]`. On the client, the `[id]` tags in the text resolve against the `source-url` parts on the same message, so rendering is a lookup, and a citation can only point at a source retrieval returned. Hook shapes follow the [AI SDK chatbot reference](https://ai-sdk.dev/docs/ai-sdk-ui/chatbot). The lookup itself is a small helper. An `[id]` with a matching source becomes a link, and an `[id]` the model invents matches nothing and stays plain text, visibly broken instead of clickable: `export function renderWithCitations( text: string, sources: Map, ) { return text.split(/(\[[^\]]+\])/g).map((segment, i) => { const match = segment.match(/^\[([^\]]+)\]$/); const source = match ? sources.get(match[1]) : undefined; if (!source) return {segment}; return ( {segment} ); }); }` _An_ `_[id]_` _tag becomes a link only when a matching source exists._ The page builds each message's source map from that message's own parts, then renders its text through the helper: `'use client'; import { useChat } from '@ai-sdk/react'; import { DefaultChatTransport } from 'ai'; import { useState } from 'react'; import { renderWithCitations } from './citations'; export default function ChatPage() { const { messages, sendMessage, status } = useChat({ transport: new DefaultChatTransport({ api: '/api/chat' }), }); const [input, setInput] = useState(''); return (
{messages.map((message) => { const sources = new Map( message.parts .filter((part) => part.type === 'source-url') .map((part) => [part.id, { url: part.url, title: part.title }]), ); return (
{message.role === 'user' ? 'You' : 'Assistant'}{' '} {message.parts.map((part, i) => part.type === 'text' ? ( {renderWithCitations(part.text, sources)} ) : null, )}
); })}
{ event.preventDefault(); if (!input.trim()) return; sendMessage({ text: input }); setInput(''); }} > setInput(event.target.value)} disabled={status !== 'ready'} />
); }` _Each message resolves its own_ `_[id]_` _tags against its own source parts._ Each message carries its own sources, so turn five's citations can never resolve against turn one's documents. ## Run ingestion on Vercel Workflows Chunking a large PDF, embedding hundreds of chunks, and writing them to the index is work that outlives any request, so it runs as a workflow instead. Functions marked `'use workflow'` orchestrate. Each helper marked `'use step'` becomes a durable, retried unit that survives redeploys, and by default a failed step [retries up to three times](https://workflow-sdk.dev/docs/foundations/errors-and-retries) without redoing the steps before it. `import { fetchDocument, chunkDocument } from '@/lib/rag/documents'; import { embedChunks, indexChunks } from './steps'; export async function ingestDocument(documentId: string) { 'use workflow'; const doc = await fetchDocument(documentId); const chunks = await chunkDocument(doc, { size: 800, overlap: 100 }); const embedded = await embedChunks(chunks); await indexChunks(documentId, doc, embedded); }` _The workflow reads as the pipeline it is: fetch, chunk, embed, index._ The steps carry the work. Each one is a durable, retriable unit: ``import { embedMany } from 'ai'; import { gateway } from '@ai-sdk/gateway'; import { vectorIndex } from '@/lib/vector'; export async function embedChunks(chunks: { index: number; text: string }[]) { 'use step'; const { embeddings } = await embedMany({ model: gateway.textEmbeddingModel('openai/text-embedding-3-small'), values: chunks.map((chunk) => chunk.text), }); return chunks.map((chunk, i) => ({ ...chunk, embedding: embeddings[i] })); } export async function indexChunks( documentId: string, doc: { title: string; url: string }, embedded: { index: number; text: string; embedding: number[] }[], ) { 'use step'; await vectorIndex.deleteByPrefix(`${documentId}:`); await vectorIndex.upsert( embedded.map((chunk) => ({ id: `${documentId}:${chunk.index}`, vector: chunk.embedding, metadata: { text: chunk.text, title: doc.title, url: doc.url }, })), ); }`` _Embed in one batch, then delete and re-upsert the document's chunks._ Start a run from the upload endpoint or a cron. Directive syntax follows the [Workflow SDK docs](https://workflow-sdk.dev/docs). `import { start } from 'workflow/api'; import { ingestDocument } from '@/app/workflows/ingest'; export async function POST(req: Request) { const { documentId } = await req.json(); await start(ingestDocument, [documentId]); return Response.json({ status: 'queued' }); }` _The upload handler returns immediately and the workflow does the heavy work._ `_fetchDocument_` _and_ `_chunkDocument_` _carry the step directive in their own module._ - **Batched embeddings:** `embedMany` embeds all chunks in one step, so a provider rate limit retries one batch call instead of replaying a per-chunk loop from the start. Shard into multiple steps for very large documents so retry granularity matches failure granularity. - **Deterministic IDs, delete before upsert:** IDs like `documentId:index` make retried writes idempotent, which durable delivery requires, and deleting the document's chunks before upserting keeps a re-ingested, shorter document from stranding orphans that retrieval would keep returning. - **A thin upload handler:** parsing PDFs in the upload request blocks the user and hits request limits. The handler's only job is to enqueue. ## How the stack compares to a container build The same four needs exist on any platform. The difference is how much arrives assembled. | **What the workload needs** | **Assembled on a container platform** | **In one Vercel project** | | --------------------------- | ---------------------------------------------------------- | -------------------------------------------------------------------------------------------------- | | Streaming chat requests | Container service, load balancer, autoscaling policy | [Vercel Functions](https://vercel.com/docs/functions) on Fluid compute, billed on Active CPU | | Background ingestion | Worker service, a queue (SQS or Redis-backed), retry logic | [Vercel Workflows](https://vercel.com/docs/workflows) with durable, retried steps | | Model access and reranking | Per-provider SDKs or a self-hosted proxy | [AI Gateway](https://vercel.com/docs/ai-gateway), one endpoint with fallbacks and spend monitoring | | Deploys and previews | CI pipelines and environments you maintain | Git push, [preview deployments](https://vercel.com/docs/deployments/preview-deployments) | _The container column is a real architecture that works. Every row of it is also a component to build, secure, and pay for before the first chat ships._ A container that streams answers is allocated, and billed, for the life of the instance, whether tokens are flowing or not, and platforms that bill per request still allocate CPU for the full request duration. On Fluid compute, Active CPU bills while code executes and pauses while the function awaits the token stream, and one instance serves many concurrent chats. A chat request spends most of its wall-clock time awaiting tokens, which is exactly the case this billing model favors. Portability cuts both ways. The chat route, retrieval, and UI in this guide are plain AI SDK and Next.js code that run anywhere Node.js runs. The Workflows pipeline is Vercel-specific, and its container equivalent is the worker service, queue, and retry logic in the table. Vercel is not a replacement for every containerized workload. What it removes for this one is the worker service, the queue backing store, and the streaming plumbing. ## Best practices - **Keep the retrieval prompt strict:** cite by `[id]`, refuse to answer outside the sources. Reranking only helps if the model actually uses the top chunks. - **Route resilience through AI Gateway:** provider fallbacks and retries keep a provider outage from taking the chat down, and spend monitoring catches runaway usage. - **Chunk with overlap:** 600 to 1000 tokens with 10 to 15 percent overlap is a solid starting point for documentation and support content. Tune against your corpus. - **Match ingest and query embedding models:** a mismatch degrades retrieval silently. Change both together, and re-embed when you change either. - **Rebuild, don't accumulate:** re-ingest changed documents with delete-before-upsert so stale chunks leave the index when the source shrinks. ## FAQ **Do long streaming answers hit function timeouts?** The Fluid compute default is 300 seconds of `maxDuration`, 800 seconds is generally available on Pro and Enterprise, and 1800 seconds is in beta. Work with no ceiling belongs in [Vercel Workflows](https://vercel.com/docs/workflows). **Does a chat request cost compute while it waits on the model?** [Active CPU](https://vercel.com/docs/fluid-compute) pauses during I/O waits, including token streams, so no CPU bills while a response waits. Provisioned memory bills for the instance, which concurrent chats share. **Which vector database should I use?** Neon (pgvector) and Upstash Vector provision natively through the [Vercel Marketplace](https://vercel.com/marketplace). External stores like Pinecone connect with an API key. Pick one in the same region as your functions. **Can ingestion really run without a container or worker service?** Yes. [Vercel Workflows](https://vercel.com/docs/workflows) runs multi-step pipelines as durable, retried steps with no ceiling on run duration, built on [Vercel Queues](https://vercel.com/docs/queues). ## Next steps - [AI SDK docs](https://ai-sdk.dev/docs): the `useChat` hook, UI message streams, and source parts used throughout this guide - [AI Gateway reranking](https://vercel.com/docs/ai-gateway/modalities/reranking): the `rerank()` call, supported models, and the Cohere-compatible REST endpoints - [Cost-aware model routing with AI Gateway](https://vercel.com/kb/guide/cost-aware-model-routing-with-ai-gateway): picking and switching models without overpaying for easy requests - [Vercel Workflows](https://vercel.com/docs/workflows): durable steps, pricing, and observability for the ingestion pipeline - [Understanding vector databases for AI apps](https://vercel.com/kb/guide/understanding-vector-databases-for-ai-apps): how to choose and operate the store this stack retrieves from - [Production architecture for a RAG chatbot on Vercel](https://vercel.com/kb/guide/rag-chatbot-production-architecture-on-vercel): the decision-level view of this stack, including when containers still win - [The Agent Stack](https://vercel.com/blog/agent-stack): the assembled pieces this guide draws on, with [guides](https://vercel.com/kb/agent-stack) for agent-shaped workloads - [What is retrieval-augmented generation (RAG)](https://vercel.com/kb/guide/what-is-rag): the concepts behind the pipeline you just built