Vercel AI Gateway and LangSmith both show token, latency, and cost dashboards, which makes them easy to mistake for two takes on the same observability problem. That resemblance is misleading, because AI Gateway is a routing and reliability layer that sits in the request path, while LangSmith is a tracing and evaluation layer that reads what happened after the call.
AI Gateway observability logs tokens, time to first token (TTFT), latency, and spend at the proxy layer. LangSmith observability tracks token usage, latency, errors, cost, and feedback scores from traces.
The dashboards overlap, but the jobs don't. AI Gateway owns routing and reliability, LangSmith owns tracing and evaluation, so some production stacks need one layer, while others need both.
Copy link to headingKey takeaways
Vercel AI Gateway is a routing and reliability layer, and LangSmith is a tracing and evaluation layer, so they solve different problems at different points in the request path.
Both log per-request token and cost data, but AI Gateway captures it with zero instrumentation because it sits in the request path.
In a week-long A/B test, Vercel AI Gateway cut Cline's P99 streaming latency by 10% to 14% and dropped API error rates 43.8% against its previous router.
The two run together through the AI SDK, with LangSmith tracing the SDK call while AI Gateway logs the proxy event.
Most production teams building multi-provider agents eventually need both layers.
Copy link to headingWhat Vercel AI Gateway does
Vercel AI Gateway is a managed Node.js service, deployed across our distributed infrastructure, that intercepts every outbound large language model (LLM) request and handles routing, failover, credentials, and cost tracking through a single endpoint.
It handles four things through a single endpoint:
A unified API across hundreds of models spanning language, embedding, reranking, image, and video
Dynamic routing between providers, with automatic failover when one degrades
Centralized credentials, including bring-your-own-key (BYOK), at zero markup and no platform fee on inference
Per-request logging of tokens, TTFT, latency, and spend, with no instrumentation required
Routing overhead targets sub-20ms, while our content delivery network (CDN) handles trillions of requests and sits at the core of AI Gateway. The service processes billions of tokens per day.
On Vercel deployments, gateway authentication doesn't even need a manual key, since apps authenticate through a Vercel-generated OpenID Connect (OIDC) token, VERCEL_OIDC_TOKEN, linked to the project automatically.
AI Gateway deliberately leaves distributed trace spans, evaluations, prompt versions, and human feedback annotation to other tools, which is where LangSmith comes in.
Copy link to headingWhat LangSmith does
LangSmith is a framework-agnostic observability and evaluation platform that receives structured trace data from AI SDK calls, or any OpenTelemetry (OTel)-compatible instrumentation, and provides distributed tracing, evaluation pipelines, prompt management, and human annotation tooling.
It works with the AI SDK, OpenAI SDK, Anthropic SDK, LlamaIndex, CrewAI, and Pydantic AI through the traceable wrapper or OTel export, and LangChain and LangGraph are optional.
Its observability and evaluation core covers four things:
Full distributed traces with span hierarchy, from the root generation through model calls and tool executions
Offline and online evaluations with human, code, LLM-as-judge, and pairwise evaluators
Prompt Hub for versioned prompt management
Annotation queues for collecting structured human feedback
The platform has grown past that core into agent hosting, no-code agent creation, and ephemeral code execution, but this comparison scopes to the observability and evaluation core, which is where it overlaps with AI Gateway.
Copy link to headingHow AI Gateway and LangSmith compare
The cleanest way to place them is by when they act. AI Gateway intercepts the HTTP request before it reaches a provider, and LangSmith receives structured trace data after the SDK call completes. One is a network proxy, the other an observability backend, so they never compete for the same slot in the request path. The only real overlap is usage logging, since both emit dashboards of token counts, latency, and cost.
Mapping the boundary against concrete engineering responsibilities rather than product categories makes the split obvious:
Only per-request token counts and cost appear in both products. On top of that shared row, LangSmith adds configurable dashboards, P50 and P99 latency percentiles, webhook and PagerDuty alerting, and, since February 2026, cost tracking that extends beyond LLM calls to tool calls and third-party API calls.
Gateway latency is a common objection in architecture reviews, since teams worry a proxy will slow direct provider calls. Production data points the other way. In Cline's week-long A/B test splitting global traffic between its previous router and AI Gateway, P99 streaming latency improved by 10% to 14% across its most-used models and API error rates dropped 43.8%. The network path explains it. For in-cloud workloads, most of the route from application to inference provider stays on our private managed backbone and off the public internet.
A gap on the AI Gateway side calls for a gateway, and a gap on the LangSmith side calls for an observability platform.
Copy link to headingWhen to use AI Gateway, LangSmith, or both
Which layers you need follows from the question you're trying to answer.
Copy link to headingWhen AI Gateway alone covers the need
If you're deploying a multi-provider application where cost governance and failover are the primary concerns, AI Gateway’s built-in per-request dashboard of tokens, TTFT, spend, and model used, plus the Custom Reporting API, often covers monitoring on its own.
Zo Computer was retrying 7.5% of messages and erroring on another 4.6%, roughly one in eight requests needing a retry or failing, before it replaced hand-maintained retry logic and fallback routing with AI Gateway and improved AI reliability 20 times. Its monitoring need at that stage was cost and uptime.
Okara had the same need at higher volume, running four billion tokens a day across a stack that previously required eight separate provider SDKs, so it needed provider consolidation and same-day model access, not trace inspection.
Copy link to headingWhen LangSmith addresses what a gateway cannot
LangSmith earns its place when the question shifts from whether a request succeeded to why an agent made a bad decision, such as which tool call returned an unexpected value or which prompt version regressed quality.
A semantic-context failure, like a tool returning { status: 2, amount: 45000 } without enough meaning for the agent to interpret it, needs trace and application context the proxy event doesn't carry.
A gateway shows you that something failed. LangSmith shows you where in the execution graph it failed, and gives you the eval and annotation tooling to fix it.
Copy link to headingWhen production agents require both
Most production agents need multi-provider failover as a reliability requirement and eval loops with prompt regression testing as a quality requirement, and those needs sit at different points in the stack.
Wrap the AI SDK with LangSmith's wrapAISDK helper so every generateText and streamText call is traced to LangSmith, while the model string routes the same request through AI Gateway, which logs it at the proxy level and emits cost metadata. Both run at once, and the wiring takes a few lines:
import * as ai from 'ai';import { wrapAISDK } from 'langsmith/experimental/vercel';
const { generateText } = wrapAISDK(ai);
const { text } = await generateText({ model: 'anthropic/claude-opus-4.8', // routes through AI Gateway prompt: 'Summarize this document.',});LangSmith reads LANGSMITH_TRACING=true and LANGSMITH_API_KEY from the environment (the older LANGCHAIN_* names still work as aliases). AI Gateway authenticates separately through AI_GATEWAY_API_KEY, or through the OIDC token on Vercel deployments.
Copy link to headingRun routing and evaluation as separate layers on Vercel
Routing and evaluation are separate lifecycle concerns, and they fail differently when they're missing. Skip the gateway and you absorb provider outages as application errors. Skip evaluation and you ship quality regressions you can't see.
We build the routing side of that stack and treat LangSmith as a supported observability integration alongside Langfuse and Braintrust, a downstream consumer of AI SDK telemetry.
On Vercel, the routing and reliability layer comes as managed infrastructure:
Automatic failover: AI Gateway routes across providers and fails over when one degrades, which is how Cline cut API error rates 43.8%.
Zero-markup pricing with BYOK: You pay provider list rates on inference with no platform fee, whether you use Vercel-managed credentials or your own key.
Spend controls: Per-key budgets cap spend, and the Custom Reporting API tracks cost by tag, user ID, or quota entity ID.
Zero-instrumentation logging: Tokens, TTFT, latency, and spend are captured for every request because the gateway sits in the path.
AI SDK telemetry: The same SDK call can stream traces to LangSmith, Langfuse, or Braintrust, so the evaluation layer plugs in without extra wiring.
In practice, get routing and failover stable first, then add evaluation once the request path is reliable. Teams that invert that order end up debugging infrastructure failures through an eval UI built for quality ones, and lose time on both.
Start a new project at vercel.com/new and route your first model call through AI Gateway, then wrap the AI SDK with your observability provider when you're ready to add evaluation. The AI Gateway docs cover the routing layer end to end.
Copy link to headingFAQ about AI Gateway vs. LangSmith
Copy link to headingDoes Vercel AI Gateway replace the need for LangSmith?
No, they run in parallel. AI Gateway provides provider routing, automatic retries and failover, BYOK, and spend monitoring, while LangSmith handles distributed tracing, evaluations, and prompt versioning at the SDK layer. They solve different problems at different points in the request path.
Copy link to headingCan I use LangSmith without Vercel AI Gateway?
Yes. LangSmith is framework-agnostic and integrates with the AI SDK, OpenAI SDK, Anthropic SDK, and any OTel-compatible instrumentation, whether or not requests are proxied through AI Gateway.
Copy link to headingDoes adding AI Gateway increase latency?
In practice no, and in some architectures it reduces latency. The routing overhead target is sub-20ms, and Cline's production A/B test found P99 streaming latency improved by 10% to 14% after switching from its previous router, because traffic routes on our private backbone rather than the public internet.
Copy link to headingHow does pricing differ between the two tools?
AI Gateway charges zero markup on tokens, so you pay provider list rates whether you use Vercel-managed credentials or BYOK. LangSmith's Plus plan is $39 per seat per month with 10,000 base traces included, then usage-based pricing beyond that. The two models add together.
Copy link to headingDo AI Gateway observability features cover the same ground as LangSmith?
Partially. AI Gateway logs every request (tokens, TTFT, latency, spend, model, provider) with no instrumentation and exposes a Custom Reporting API for grouping by tag, user ID, or quota entity ID.
LangSmith adds span-level traces, eval pipelines, prompt versioning, annotation queues, and P50/P99 dashboards with alerting, all of which are out of scope for a routing proxy.