--- title: How to architect an AI evaluation dashboard on Vercel description: Map eval orchestration, traces, and run storage to AI Gateway, Observability, and Marketplace Postgres, and learn when self-hosted LiteLLM or Langfuse fits better. url: /kb/guide/ai-evaluation-dashboard-architecture-on-vercel canonical_url: "https://vercel.com/kb/guide/ai-evaluation-dashboard-architecture-on-vercel" last_updated: 2026-07-28 authors: Vercel related: - /docs/ai-gateway - /docs/observability - /docs/frameworks/nextjs - /docs/functions - /docs/workflows - /docs/queues - /docs/vercel-blob - /docs/ai-gateway/authentication-and-byok - /docs/ai-gateway/authentication-and-byok/api-keys - /docs/ai-gateway/observability-and-spend/observability - /docs/ai-sdk - /docs/ai-gateway/observability-and-spend/custom-reporting - /docs/observability/observability-plus - /docs/ai-gateway/security-and-compliance/zdr - /docs/fluid-compute - /docs/ai-gateway/models-and-providers/model-fallbacks install_vercel_plugin: npx plugins add vercel/vercel-plugin --- ## Introduction An AI evaluation dashboard has to solve three problems at once. It needs multi-provider model access, request-level visibility into every model call, and persistent storage for eval runs and artifacts. A common self-hosted stack covers all three with LiteLLM as the model proxy, Langfuse for traces, and a database behind them. It also means three services you deploy, upgrade, and monitor yourself. This guide maps each part of that stack to Vercel, and covers when staying self-hosted is the right call. It addresses the infrastructure layer and leaves the grading logic and data model to you. ## Overview In this guide, you'll: - Map the parts of an eval dashboard (frontend, orchestration API, model calls, storage) to Vercel primitives - Compare AI Gateway's coverage against what a LiteLLM proxy provides - Plan around Observability's log retention window - Decide when self-hosted LiteLLM or Langfuse is still the right call - Work through the hosting decision in order, from data sovereignty to batch runtime to eval-native features ## Reference architecture Start by mapping each piece of your current stack to its Vercel counterpart. LiteLLM's proxy role moves to [AI Gateway](https://vercel.com/docs/ai-gateway), Langfuse's request-level tracing maps to [Observability](https://vercel.com/docs/observability), and your eval data goes into a managed Postgres you provision through Vercel. The one piece with no equivalent is LiteLLM's load-aware routing, since AI Gateway's fallback is a static preference order. | Self-hosted component | On Vercel | | ---------------------------------------------- | ----------------------------------------------------- | | LiteLLM proxy (routing, fallback, keys, spend) | AI Gateway | | Langfuse request-level tracing | Observability | | Database for eval runs and scores | Managed Postgres (Neon, Supabase, or another partner) | | Worker for batch evals | Vercel Workflows | | Object storage for artifacts | Vercel Blob | The dashboard splits into four parts: - Frontend dashboard: Charts, run detail views, and an artifact browser need a place to render. On Vercel, this is a [Next.js](https://vercel.com/docs/frameworks/nextjs) app built from standard server components and API routes. - Eval orchestration: Something has to accept a run request, fan out model calls, and write results. Request-shaped endpoints like these run as [Vercel Functions](https://vercel.com/docs/functions). - Model calls: Where you run LiteLLM today for unified access and fallback, AI Gateway takes over with one key, ordered fallback across providers, and `modelAttempts` metadata recording which model served each call. - Run and artifact storage: Eval definitions, run metadata, and per-sample scores need a database that outlives any log retention window, the same role your self-hosted database plays today. Provision it with one command, `vercel install neon`, or pick another [Marketplace](https://vercel.com/marketplace) partner. The resource is created, connected to your project, and its credentials are pulled into your environment. Two patterns sit just outside this core. Batch evals that exceed a single function's execution window run on [Vercel Workflows](https://vercel.com/docs/workflows). You decompose the run into function-sized steps, and Workflows makes the whole run durable. Steps retry independently, and runs pause, resume, and persist across restarts and deployments with no limit on total duration, while each step stays within a function's bounds. If Workflows isn't a fit, you have alternatives. [Vercel Queues](https://vercel.com/docs/queues), the primitive that powers Workflows, gives you direct control over message publishing and consumption. A Marketplace partner like [Inngest](https://vercel.com/marketplace/inngest) or [Trigger.dev](http://Trigger.dev) runs the job as-is, without restructuring. Or run a worker you host. Large artifact blobs (full transcripts, image outputs, embeddings) belong in [Vercel Blob](https://vercel.com/docs/vercel-blob) rather than in a Postgres row. ## What AI Gateway replaces in a LiteLLM setup If you're running LiteLLM today, you're probably using it for unified API access, ordered fallback, key management, spend tracking, request logging, or a mix of these. AI Gateway covers the first four, with one key-management gap noted below. The fifth, request logging, is covered by Observability in the next section. - Unified API: One endpoint and one auth surface for hundreds of models. Inside a Vercel deployment there are no provider keys to store or rotate, because your project authenticates with its [OIDC token](https://vercel.com/docs/ai-gateway/authentication-and-byok). Static API keys work anywhere, including outside Vercel. - Ordered fallback: The `providerOptions.gateway.models` array sets the fallback sequence per request, and the `order` option controls provider preference within each model. The response includes `modelAttempts`, so you can record which model and provider served the call, plus any failed attempts. - Key management: [AI Gateway API keys](https://vercel.com/docs/ai-gateway/authentication-and-byok/api-keys) are created and revoked through the Vercel API, CLI, or dashboard. Through the API, a key can be scoped to a project and given an expiry date, and any key can have a spend budget with a daily, weekly, or monthly refresh, or a fixed lifetime cap. Budgets are soft caps, since the check runs when a request starts. The gap: LiteLLM virtual keys also carry per-key rate limits and model allowlists, and AI Gateway keys have no equivalent for those. - Spend monitoring: Per-project and per-key usage, request counts, time to first token, token counts, and spend live in the [Usage and Requests views](https://vercel.com/docs/ai-gateway/observability-and-spend/observability), scoped to team or project. Fallback configuration is set per request. This [AI SDK](https://vercel.com/docs/ai-sdk) call tries the primary model first and falls back in order: `import { streamText } from 'ai'; const result = streamText({ model: 'anthropic/claude-fable-5', prompt, providerOptions: { gateway: { models: ['anthropic/claude-opus-4.8', 'google/gemini-3.1-pro-preview'], }, }, });` AI Gateway does not store eval scores, link a model response to a graded outcome, retain prompt and response content, or join runs across time. That work belongs in Postgres. ## What Observability gives you, and what to watch for Observability surfaces the request log AI Gateway produces. Each request is recorded with its model, token counts, cost, latency, and time to first token, in a sortable log, grouped by project or API key, that you can export. For production debugging, this covers the request-level metadata visibility you would otherwise get by sending LiteLLM's traces to Langfuse. For eval runs specifically, [Custom Reporting](https://vercel.com/docs/ai-gateway/observability-and-spend/custom-reporting) attaches metadata at request time. Pass tags (up to 10 per request) and a user ID through `providerOptions.gateway`, or the `ai-reporting-tags` and `ai-reporting-user` headers, then query the report API for cost, tokens, and request counts grouped or filtered by tag. Tagging each request with its eval run ID gives you per-run spend with no reporting pipeline to build. Before you build on it, know that Custom Reporting is in beta, tag writes and report queries are billed, the report API requires a Pro or Enterprise plan, and report data covers the whole account during the beta. Three things to know before you build on this log. First, the log holds request metadata, not content. AI Gateway does not retain prompts or outputs after a request completes, so the only copy of a prompt and response pair you control is the one your orchestration code writes to Postgres at request time. Second, the metadata itself expires. Without [Observability Plus](https://vercel.com/docs/observability/observability-plus), request data is kept for 12 hours on Hobby, 1 day on Pro, and 3 days on Enterprise. Observability Plus, available on paid Pro and Enterprise teams, extends retention to 30 days. If your evals reference that metadata beyond the window, copy it into Postgres at write time as well. Third, Observability is built around production request traffic, not eval-specific concepts like graded samples, datasets, or regression diffs. The eval dashboard you build is what makes those concepts first-class. ## When the self-hosted stack still wins There are three reasons to stay self-hosted. - Data sovereignty: On Pro and Enterprise plans, AI Gateway supports [zero data retention routing](https://vercel.com/docs/ai-gateway/security-and-compliance/zdr) through the `zeroDataRetention` option, which restricts requests to providers with ZDR agreements. Certain models are excluded from that restriction, and requests involving them go through with no ZDR guarantee applied rather than erroring, so check the exclusion list before relying on the option. And if contracts require that prompt and response data never traverse a third-party gateway at all, retention guarantees don't address that. A self-hosted LiteLLM deployment on infrastructure you control is the answer. - Compute you control: Duration alone is no longer the dividing line, since Vercel Workflows handles batch runs that outlast a single function. Self-hosting makes sense when the eval work itself needs infrastructure you choose, such as GPUs for local model scoring or data that can't leave your network. You can still front the model calls with AI Gateway from that worker. - Eval-native features: Langfuse ships dataset management, prompt versioning, and scored experiment comparison as first-class features. If those are core to your dashboard and you don't want to build them, keeping Langfuse for tracing and evaluation while routing model calls through AI Gateway, or calling providers directly with your own keys, is reasonable. None of these three reasons forces an all-or-nothing choice. Even if one piece stays on your infrastructure, AI Gateway gives model calls one key and ordered fallback, Observability produces the request log with nothing to deploy, a managed Postgres stores eval data with no database to operate, and Workflows keeps batch runs durable with no worker to run. In the strict sovereignty case, the dashboard itself can still run on Vercel, limited to the metadata and scores that are allowed to leave your network. Transcripts and run content stay in storage you control. ## Making the decision Run the decision in this order: 1. Does any contract or compliance requirement forbid third-party gateways for prompt or response data? If yes, self-host the proxy layer, keep eval orchestration and artifact storage wherever the data is allowed to flow, and put only the dashboard and its permitted metadata on Vercel. 2. Can your batch evals decompose into function-sized Workflow steps? If they can, Workflows runs them natively. If you'd rather not restructure the job, a Marketplace partner like Inngest or [Trigger.dev](http://Trigger.dev) runs it as-is. Self-host the worker only when the eval work must run on infrastructure you control. 3. Are Langfuse's eval-native features (datasets, scored experiments, prompt versions) core to what you're building, or does your dashboard already own that surface? If your dashboard owns it, AI Gateway plus your own Postgres covers the rest. For teams without strict data-residency requirements, the Vercel-hosted dashboard with AI Gateway and a managed Postgres means three fewer services to deploy, upgrade, and monitor (LiteLLM, Langfuse, and the database behind them), while keeping per-key budgets, per-run reporting tags, and fallback control. Vercel Functions run on [Fluid compute](https://vercel.com/docs/fluid-compute) by default for new projects. Multiple invocations share an instance, and CPU is billed only when active, so a dashboard fanning out I/O-bound eval calls pays memory rates but not CPU rates while requests wait on databases, external APIs, or model responses. The same setup extends to multi-tenant work. An agency serving multiple clients can run every client's evals through one gateway, using per-key budgets and reporting tags to attribute cost per client inside its own dashboard. During the Custom Reporting beta, report queries return account-wide data, so per-client views are something your dashboard builds, not something you delegate to the report API. ## Resources and next steps - Start with the [AI Gateway documentation](https://vercel.com/docs/ai-gateway). - Read the [model fallbacks reference](https://vercel.com/docs/ai-gateway/models-and-providers/model-fallbacks) for fallback configuration and the full `modelAttempts` response schema, the shape your orchestration code stores for each sample. - Check [Custom Reporting](https://vercel.com/docs/ai-gateway/observability-and-spend/custom-reporting) for current pricing on tag writes and report queries, which are omitted here because beta pricing can change. - Review the [Observability Plus limitations table](https://vercel.com/docs/observability/observability-plus#limitations). Even with Plus's 30-day retention, runtime log queries are capped at a 14-day selection window. - Learn [Vercel Workflows](https://vercel.com/docs/workflows). Its hooks primitive pauses a run until an external event arrives, which fits human-in-the-loop grading steps. - See [Fluid compute](https://vercel.com/docs/fluid-compute) for per-plan maximum durations, the number that decides whether an eval batch fits in one function or needs Workflows. - Browse the [Vercel Marketplace](https://vercel.com/marketplace) for managed Postgres options, and see [Vercel Blob](https://vercel.com/docs/vercel-blob) for artifact storage.