--- title: Production architecture for a RAG chatbot on Vercel description: Architect a production RAG chatbot on Vercel Functions with Fluid compute, AI Gateway, and a region-pinned vector store. url: /kb/guide/rag-chatbot-production-architecture-on-vercel canonical_url: "https://vercel.com/kb/guide/rag-chatbot-production-architecture-on-vercel" last_updated: 2026-07-29 authors: Vercel related: - /docs/functions - /docs/fluid-compute - /kb/guide/building-ai-chat-app-with-rag-and-citations-on-vercel - /docs/ai-gateway/modalities/reranking - /docs/workflows - /docs/functions/configuring-functions/duration - /docs/networking/secure-compute - /docs/frameworks - /docs/observability - /docs/oidc - /docs/functions/streaming-functions - /blog/agent-stack - /kb/agent-stack - /kb/guide/what-is-rag install_vercel_plugin: npx plugins add vercel/vercel-plugin --- Behind the chat window, a chatbot built with retrieval-augmented generation (RAG) is three backend jobs. A streaming API holds a response open for the length of every answer, retrieval and reranking run inside that same request, and a vector database has to answer the retrieval query fast enough that nobody notices it happened. All three, plus the UI, can run as one deployment on Vercel. Two mechanisms make that work. [Vercel Functions](https://vercel.com/docs/functions) on [Fluid compute](https://vercel.com/docs/fluid-compute) hold the stream and bill on [Active CPU](https://vercel.com/docs/fluid-compute), which pauses whenever the code awaits tokens, so a thirty-second answer costs the compute of the work in it, not thirty seconds of wall clock. And the vector database provisions alongside the app, in the same region as the functions, so the retrieval query is one intra-region round trip. [Building an AI chat app with RAG and source citations on Vercel](https://vercel.com/kb/guide/building-ai-chat-app-with-rag-and-citations-on-vercel) walks through the build itself, with the full code. ## Contents - [What the four layers are, and what runs each one](#the-four-layers-of-a-rag-chatbot-on-vercel) - [How long-lived streams work on Fluid compute](#long-lived-streams-on-fluid-compute) - [How citations stream before the first token](#citations-stream-before-the-first-token) - [How one gateway carries generation and reranking](#one-gateway-for-generation-and-reranking) - [How region pinning answers retrieval latency](#pin-the-vector-store-to-the-function-region) - [When containers are still the right call](#when-containers-are-still-the-right-call) - [How to migrate from a split deployment](#migrating-from-a-split-deployment) - [What the reference architecture includes](#reference-architecture) ## The four layers of a RAG chatbot on Vercel Each layer has a job, a requirement that job imposes, and a home on the platform. | **Layer** | **What it needs** | **On Vercel** | | ---------------------------- | ------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Chat UI | Streaming state, message rendering, citations as they arrive | The [AI SDK](https://ai-sdk.dev/) `useChat` hook, rendering `source-url` parts | | Streaming chat API | A response held open for the length of each answer, without paying for the wait | A Vercel Function on Fluid compute, streaming over Server-Sent Events | | Retrieval and reranking | A vector query plus a reranking pass inside the request path | The same function, with [reranking through AI Gateway](https://vercel.com/docs/ai-gateway/modalities/reranking) | | Vector storage and documents | Low-latency reads from the API tier | A store provisioned in the function's region, natively through the [Vercel Marketplace](https://vercel.com/marketplace) (Neon, Upstash Vector) or connected externally | _Three layers deploy as one application (Next.js in these examples)._ The request path stays short, with the client reaching the function through the nearest point of the global network and the function reaching the vector store inside one cloud region. ## Long-lived streams on Fluid compute A single chat request holds an open Server-Sent Events connection for the duration of retrieval, reranking, model streaming, and citation emission, which can run to tens of seconds. On per-invocation serverless, that is one full instance per user for the length of the response, mostly parked on `await`. Fluid compute changes the economics through two separate mechanisms. [Active CPU](https://vercel.com/docs/fluid-compute) billing means the CPU meter runs while code executes and pauses whenever the function awaits I/O, whether a database read, an upstream API, or a token stream. In-instance concurrency means one function instance serves multiple concurrent streams, which spreads the instance's provisioned-memory cost across every open chat. `maxDuration` defaults to 300 seconds on Fluid compute, 800 seconds is generally available on Pro and Enterprise, 1800 seconds is in beta, and work with no ceiling belongs in [Vercel Workflows](https://vercel.com/docs/workflows). A stream that reaches the ceiling ends mid-answer, so pair a long ceiling with an upstream abort for stalled providers. See [configuring duration](https://vercel.com/docs/functions/configuring-functions/duration). Vercel Functions support Node.js streaming responses and the Web `ReadableStream` API, which is what the AI SDK writes into. There is no separate streaming runtime to provision. ## Citations stream before the first token The client uses `useChat` from `@ai-sdk/react` and renders citations from `source-url` parts on each message. On the server, retrieval and reranking come first: `import { rerank } from 'ai'; import { gateway } from '@ai-sdk/gateway'; import { vectorStore } from '@/lib/vector'; export async function retrieveTop(query: string) { // Scope the query to the requesting user's document permissions const candidates = await vectorStore.query(query); const { ranking } = await rerank({ model: gateway.rerankingModel('cohere/rerank-v4-fast'), query, documents: candidates.map((c) => c.text), topN: 5, }); return ranking.map((r) => candidates[r.originalIndex]); }` _Query the store, rerank the candidates, return the top five._ The route writes the retrieved sources into the stream as parts, then merges the model's token stream into the same response. Shapes follow the [AI SDK streaming data reference](https://ai-sdk.dev/docs/ai-sdk-ui/streaming-data). `import { convertToModelMessages, createUIMessageStream, createUIMessageStreamResponse, streamText, toUIMessageStream, } from 'ai'; import { gateway } from '@ai-sdk/gateway'; import { retrieveTop } from '@/lib/retrieval'; import { buildSystemPrompt } from '@/lib/prompt'; import type { UIMessage } from 'ai'; export async function POST(req: Request) { const { messages }: { messages: UIMessage[] } = await req.json(); const query = messages .at(-1)! .parts.filter((part) => part.type === 'text') .map((part) => part.text) .join(' '); const top = await retrieveTop(query); const stream = createUIMessageStream({ execute: ({ writer }) => { for (const source of top) { writer.write({ type: 'source', value: { type: 'source', sourceType: 'url', id: source.id, url: source.url, title: source.title, }, }); } const result = streamText({ model: gateway('openai/gpt-5-mini'), system: buildSystemPrompt(top), messages: convertToModelMessages(messages), }); writer.merge(toUIMessageStream({ stream: result.stream })); }, }); return createUIMessageStreamResponse({ stream }); }` _Source parts go out before generation begins, and the token stream merges into the same response._ The user sees where the answer will come from while the model is still writing it, and because each source is attached to the message that cites it, every turn's citations resolve against that turn's retrieval. ## One gateway for generation and reranking AI Gateway runs on the same platform as the functions that call it, so generation and reranking arrive through one endpoint, one authentication surface, and one spend and observability view, with automatic retries and fallbacks across providers. Swapping the generation model is a one-line edit that never touches retrieval code. Rerank models from multiple providers are in the [catalog](https://vercel.com/ai-gateway/models?capabilities=reranking), and swapping one is the same one-line edit. See the [reranking reference](https://vercel.com/docs/ai-gateway/modalities/reranking) for the call shapes. ## Pin the vector store to the function region Retrieval latency is the strongest objection to this design, and it goes away when the vector store runs in the same region as the functions that query it. Neon and Upstash Vector let you choose the provider region when you create the database, so pick the region your functions run in and the vector query is one intra-region round trip. Time it from a deployed function before committing. Intra-region reads give the same region-level locality a container next to the database gets. Locality settles the latency question, and Active CPU billing changes what the waiting costs. A 20-second streamed answer bills for the fraction of a second of CPU it used, plus provisioned memory shared across every concurrent stream, where an always-on server bills for all 20 seconds of every stream it holds. For bursty, idle-heavy chat traffic, that difference dominates the bill. For document metadata, run Postgres in the same region. With [Neon](https://vercel.com/marketplace/neon), pgvector and relational tables share one database, so retrieval and metadata joins are a single round trip. [Upstash](https://vercel.com/marketplace/upstash) Vector provisions natively as well, and external stores such as Pinecone connect with an API key, in whichever of the provider's regions you select at creation. ## When containers are still the right call Use containers on Azure Container Apps, AWS ECS or Fargate, or Google Cloud Run when: - **Guaranteed warm caches:** Fluid compute instances persist and reuse in-process caches across invocations, but instance lifetime is not guaranteed. If cache-miss cost dominates request latency and the cache must be warmed at boot and held for hours, a container you control wins. - **Self-hosted GPU models:** a custom embedding model on GPUs you host is container territory. - **VPC-resident compute:** for the outbound direction, reaching services inside a private VPC from Vercel Functions, [Secure Compute](https://vercel.com/docs/networking/secure-compute) provides a dedicated private network with VPC peering on Enterprise. When policy requires the compute itself to reside inside your VPC, containers are the answer. - **Platform gravity:** a mature container platform that already owns identity, networking, and deploys, which the chatbot must inherit. That is a requirement like any other, and fighting it costs more than the consolidation saves. If none of those conditions apply, a separate container tier adds an operational surface without a latency gain, and without a capability this workload uses. ## Migrating from a split deployment A split deployment puts the chat UI on one platform and the retrieval API in containers next to the vector database, which means a second deployment surface with its own TLS, CORS, and firewall configuration, a second identity plane, and a second observability stack. Consolidating removes those, and the move can happen incrementally: 1. **Move the API routes first:** the chat route and retrieval logic in this guide are plain AI SDK application code that fits the existing Vercel project serving the UI. If the current API is Python or another stack, this step is a port rather than a copy, and the container API stays up while both run. 2. **Keep the vector store where it is, then re-region it:** the function can query the existing store cross-region during the transition, at a latency cost you can measure. Moving the store last means either a provider-side region move or a re-index, and a re-index means re-embedding the corpus, which costs embedding calls and a cutover window. 3. **Roll back at the route level:** while both APIs exist, the rollback is pointing the UI's transport back at the container endpoint, a one-line change. The chat and retrieval code stays plain AI SDK code afterward, and runs anywhere Node.js does. The Vercel-specific pieces are Fluid compute billing and the ingestion pipeline, the parts a split deployment assembles from a worker service and a queue. ## Reference architecture The full stack for a customer-support chatbot with citations. The UI, the chat API, and the retrieval logic deploy, authenticate, and scale as one application. Next.js is the worked example, and any [framework Vercel supports](https://vercel.com/docs/frameworks) fits the same shape. - **Frontend:** Next.js App Router, AI SDK `useChat`, rendering `source-url` parts as citation links. - **Chat API:** a route handler on Vercel Functions with Fluid compute, `maxDuration` sized to the longest response, streaming UI message parts. - **Retrieval:** vector query against a store in the same region as the function, scoped to the requesting user's permissions so no one receives a citation to a document they cannot open. - **Reranking:** `rerank()` through AI Gateway with a reranking model from the catalog. - **Generation:** `streamText` through AI Gateway, model selected per environment or per user tier. - **Ingestion:** chunking, embedding, and indexing run on [Vercel Workflows](https://vercel.com/docs/workflows) as durable steps. The [build guide](https://vercel.com/kb/guide/building-ai-chat-app-with-rag-and-citations-on-vercel) walks through the pipeline. - **Source metadata:** Postgres in the same region for document URLs, titles, permissions, and audit records. - **Observability:** [Vercel Observability](https://vercel.com/docs/observability) for function traces, AI Gateway for prompt and completion logs. Support conversations contain customer data, so set retention and redaction policy for those logs before launch. - **Identity:** your auth provider for end users, and [OIDC federation](https://vercel.com/docs/oidc) in place of long-lived cloud keys for backend calls that need cloud IAM. ## FAQ **Does a chat stream cost compute while it waits on the model?** [Active CPU](https://vercel.com/docs/fluid-compute) pauses during I/O waits, including token streams, so no CPU bills while a response waits. Provisioned memory bills for the instance, which concurrent streams share. **What happens when a response hits** `**maxDuration**`**?** The stream ends at the ceiling (300 seconds by default on Fluid compute, 800 generally available on Pro and Enterprise, 1800 in beta). Size the ceiling to your longest answer and abort stalled upstream streams early. **Which vector databases support region pinning?** Native Marketplace integrations (Neon, Upstash Vector) let you choose the provider region at creation. External stores like Pinecone connect by API key in whichever of their regions you select. Match the function region either way. **When do I still need a container platform?** GPU-hosted models, compute that must reside inside your VPC, guaranteed-lifetime warm caches, or an existing container platform the chatbot must inherit. For reaching into a VPC from Vercel, see [Secure Compute](https://vercel.com/docs/networking/secure-compute). ## Next steps - [Building an AI chat app with RAG and source citations on Vercel](https://vercel.com/kb/guide/building-ai-chat-app-with-rag-and-citations-on-vercel): the build walkthrough for this architecture, with the full code - [Fluid compute](https://vercel.com/docs/fluid-compute): the concurrency and Active CPU billing model this architecture leans on - [Streaming functions](https://vercel.com/docs/functions/streaming-functions): response streaming behavior and limits - [AI Gateway reranking](https://vercel.com/docs/ai-gateway/modalities/reranking): the rerank API, supported models, and REST endpoints - [AI SDK chatbot docs](https://ai-sdk.dev/docs/ai-sdk-ui/chatbot): `useChat`, transports, and message parts - [The Agent Stack](https://vercel.com/blog/agent-stack): the assembled pieces behind this stack, with [guides](https://vercel.com/kb/agent-stack) for agent-shaped workloads - [What is retrieval-augmented generation (RAG)](https://vercel.com/kb/guide/what-is-rag): the concepts underneath the architecture