Picture this: A user reports a confidently wrong answer, the on-call engineer opens the dashboards, and every graph is green. The request returned a 200 OK in under two seconds, and traditional application performance monitoring (APM) has no column for "the answer was wrong." That gap is the reason large language model (LLM) apps need their own observability layer.
Seven tools cover different parts of the problem, and each one answers a different production question. This guide maps the seven to the question each is built to answer.
Key takeaways:
Traditional APM misses the failure mode that matters: A 200 OK response can carry a hallucinated answer, and request-level monitoring reads it as success.
Your AI gateway is the first observability layer: It answers cost, latency, and provider-health questions before you need a dedicated eval platform.
The AI SDK exports over OpenTelemetry: Traces flow to supported LLM observability backends, and AI Gateway attaches per-request cost and fallback metadata automatically.
Namespaces differ across tools: The OpenTelemetry GenAI (
gen_ai.) and OpenInference (llm.) conventions use different attribute prefixes, so tool choice affects which backends can parse your spans.The right pick usually matches your existing surface: If you already run Datadog, LangChain, or an open-source mandate, that instrumentation surface tends to decide the tool for you.
Copy link to headingWhat is AI observability?
AI observability is the practice of tracing, measuring, and evaluating LLM applications in production. It tells teams not only whether a request succeeded but whether the output was correct. It extends traditional APM with trace fidelity for non-deterministic outputs, token-level cost attribution, and evaluation signals that score response quality alongside delivery status.
The distinction that matters is the failure mode it is built to catch. AI observability captures the prompt, response, tool calls, and agent steps behind each request. That is what makes the silent failures visible.
Copy link to headingThe three layers of LLM observability
Traditional monitoring answers "is it working?" LLM observability also has to answer "is it correct?", and those are different systems.
There are three layers, and conflating them is where tool evaluations go wrong:
Gateway metrics: Cost, latency, token counts, and provider health, aggregated across every model you route to.
Trace capture: Prompt and response content, plus the tool calls and agent steps around each request.
Eval signals: LLM-as-judge scoring combined with human annotation, for regression detection over time.
Each layer answers a different class of question. If your dominant question is which model is slow and which feature costs the most, start at the gateway. If your question is whether the output is correct, you need evals.
The tipping point is the moment when "it's slow" stops being the problem and "it's wrong" becomes the problem. APM can tell you the request took two seconds. It cannot tell you the response was hallucinated.
The seven tools below sort cleanly once you know which layer your question lives in.
Copy link to headingThe 7 best AI observability tools at a glance
The tools split across those three layers, and none of them cover all three the same way. Some start at the gateway and add tracing. Some start at tracing and add a full eval pipeline. A few correlate LLM traces with the rest of your infrastructure. Before comparing them, it helps to fix the axes that separate a tool you keep from one you outgrow.
We weighted six things that decide fit for a production LLM stack:
Layers covered: Whether the tool handles gateway metrics, trace capture, eval signals, or a combination.
Vercel AI SDK integration: How the tool ingests spans from the AI SDK, and how much code that wiring takes.
Open source and self-hosting: Whether you can run it in your own infrastructure, and under which license.
OpenTelemetry namespace: Whether it emits or expects
gen_ai.,llm., or a proxy-based schema, which governs portability.Latency overhead: What the tool adds to the request path, where the vendor documents it.
Free tier and pricing: Where the free tier ends and what the first paid step costs.
Each tool below is organized by the question it answers best, starting with the layer that catches the first wave of production incidents.
Copy link to heading1. Vercel AI Gateway and AI SDK tracing
Every request routed through AI Gateway gets gateway-layer observability with no instrumentation code. That covers requests by model, time to first token (TTFT), input and output token counts, per-request cost, and metadata describing which provider served the request and any fallback attempts.
Over the BFCM 2025 window, more than 24 million AI requests passed through the gateway. The Usage tab shows aggregate request data. The Requests tab adds per-project and per-API-key summaries with request count, average tokens, P75 duration, P75 TTFT, and cost. Logs is sortable and exportable.
For spend attribution, the Custom Reporting API breaks costs out by tag and user through providerOptions.gateway.tags and providerOptions.gateway.user.
Above the gateway sits AI SDK 7 tracing. Registered once at application startup with registerTelemetry, the SDK emits OpenTelemetry (OTel) spans for model calls, tool executions, agent steps, embeddings, and errors, with onStepStart and onStepFinish callbacks for multi-step agent runs.
Here’s how:
import { registerTelemetry } from 'ai';import { DatadogTelemetry } from '@ai-sdk/datadog';// or Langfuse, Braintrust, Raindrop, Sentry, Laminar, LangSmithregisterTelemetry(DatadogTelemetry());
After that call, every generateText, streamText, generateObject, and streamObject in the app is traced without further configuration. When an anomaly needs a closer look, getGenerationInfo() returns the per-generation details.
AI SDK 7 names Datadog, Langfuse, Braintrust, Raindrop, Sentry, Laminar, and LangSmith as OTel export backends, so the instrumentation you register here feeds any tool below. The reliability gains compound.
Zo Computer ran a 7.5% retry rate before switching to AI Gateway routing, then improved reliability 20x. Co-founder Rob Cheung said, "We didn't even know what we were missing until after we switched." At Gamma, "every model automatically gets production-grade observability from its first request."
One caveat is worth knowing up front. The AI SDK emits proprietary ai. attributes alongside its OTel output, so backends parsing only gen_ai. miss SDK-specific span data.
Pricing: Gateway and AI SDK observability are included on all Vercel plans from the first deployed request. Extended data retention needs Observability Plus.
Best for: Teams on Vercel that want gateway and trace observability without a third-party tool, or that want a single instrumentation foundation feeding any tool below.
Copy link to heading2. Langfuse
Langfuse is OpenTelemetry-native and accepts OTLP over HTTP/JSON and HTTP/protobuf, with gRPC not yet supported.
It defines 10 observation types (event, span, generation, agent, tool, chain, retriever, evaluator, embedding, and guardrail) and layers on LLM-as-judge evaluation, prompt management with version tracking, datasets and experiments, and asynchronous SDKs that add no latency to the request path.
The Vercel AI SDK integration wires LangfuseSpanProcessor from @langfuse/otel into instrumentation.ts alongside @vercel/otel, and the JS/TS SDK v4, generally available (GA) in September 2025, was rebuilt on OTel for exactly this path. Prompt management links to AI SDK traces so you can track performance per prompt version.
You can self-host under the MIT license with Docker Compose or Kubernetes and Terraform, or use Langfuse Cloud in EU, US, Japan, and HIPAA regions. The project has more than 21,000 GitHub stars, stores data in open-source ClickHouse, and was acquired by ClickHouse.
The MIT license is the real differentiator here. For a compliance-sensitive org, data portability and the option to run the whole stack in-house outweigh a slicker managed-only alternative.
Pricing: Hobby is free at 50k units per month with 30-day retention, Core is $29 per month with 90-day retention, and Pro is $199 per month with three-year retention.
Best for: Teams that want full eval depth with self-hosting and data portability.
Copy link to heading3. LangSmith
LangSmith traces LangChain and LangGraph apps with no code changes, using environment variables. Its July 2026 OTLP support accepts OTel traces from other stacks. It adds annotation queues with auto-assignment, production alerting, and prompt management, and LangSmith Engine, announced at Interrupt 2026, clusters failures automatically and proposes fixes.
For the Vercel AI SDK, pass AISDKExporter from langsmith/vercel as the traceExporter in registerOTel, or use LangSmithTelemetry for AI SDK 7 with langsmith>=0.7.2. AI SDK traces now render in the LangSmith Messages view.
The tradeoff is the backend. It is proprietary and closed-source, and self-hosting is Enterprise-only on Kubernetes in AWS, GCP, or Azure, so an open-source mandate rules it out. If your team already lives in LangChain or LangGraph, the zero-code tracing usually justifies that constraint.
Pricing: Developer is free at 5,000 traces per month with 14-day retention, and Plus is $39 per seat per month.
Best for: Teams already on LangChain or LangGraph that want tracing with no code changes and can accept a proprietary backend.
Copy link to heading4. Braintrust
Braintrust wrapAISDK adds automatic logging with cost tracking and evaluation across AI SDK v3 through v6 without code changes. Its distinctive move is closing the loop between production and testing: traces convert into test cases with one click, and Loop generates custom scorers from natural language.
Real-time drill-down surfaces the slowest calls and the highest token consumption, with error patterns, model-level breakdowns, and cost attribution down to individual users and features.
On Vercel, you can skip the SDK path and add an OTel Trace Drain in Team Settings through the Vercel Marketplace integration, then create instrumentation.ts and call registerOtel. Braintrust ships a native OTel exporter with automatic LLM span conversion and OpenLLMetry support.
The design leans toward one workflow in particular. If your primary need is trace-first debugging, a trace-native tool fits better, but if you want production traffic feeding an offline eval dataset, this is the shortest path from trace to test case.
Pricing: Starter is free at 1 GB of processed data and 10k eval scores per month with 14-day retention, and Pro is $249 per month.
Best for: Teams whose primary workflow is feeding production traces into offline eval datasets.
Copy link to heading5. Arize Phoenix
Phoenix is an Elastic License v2 (ELv2) open-source, local-first debugging environment: OTel-based tracing, LLM-as-judge evaluation, a prompt playground with model comparison and replay of traced calls, versioned datasets, and PXI, an AI agent for debugging traces and iterating on prompts.
Auto-instrumentors for the Vercel AI SDK are available, and the managed counterpart, Arize AX, adds Agent Graph visualization and production monitoring.
The schema is the thing to account for before you commit. Phoenix uses OpenInference with the llm. attribute prefix (llm.input_messages, llm.token_count.total), while OpenTelemetry GenAI uses the gen_ai. prefix.
Arize labels OTel GenAI as "not on a stable release; subject to change at any time" and positions OpenInference as "Stable." The practical consequence is that Phoenix traces are not natively interchangeable with backends that ingest pure OTel GenAI spans, such as Datadog's OTLP ingest.
A later migration to a gen_ai.-native backend can mean attribute mapping. Account for that gap up front, and Phoenix gives you the most detailed local trace view in this list.
Pricing: AX Free is 25k spans per month with 15-day retention, and AX Pro is $50 per month.
Best for: Teams that want a local, open-source debugging environment with detailed trace visualization before adding a managed backend.
Copy link to heading6. Helicone
Helicone takes the proxy approach: change the base URL on your LLM provider call, and you get observability, cost and latency tracking, agent tracing, evaluations, LLM routing with automatic fallbacks, and automatic response caching. A Vercel AI SDK provider is available.
The cost of that setup is spelled out, which is rare in this category. The distributed proxy adds an average of 50 to 80 ms per call, the only documented latency figure in this list. That is the price of a zero-instrumentation swap, and for a prototyping team it is usually the right trade, though a latency-sensitive production path may not absorb it.
The main platform is Apache-2.0 open source and self-hostable (Y Combinator W23), with the AI gateway component open-sourced separately in Rust, and SOC 2, HIPAA, and GDPR compliance land on the Team and Enterprise plans.
Pricing: Hobby is free at 10k requests per month, Pro is $79 per month, and Team is $799 per month.
Best for: Teams prototyping fast that want observability from a single base-URL swap and can absorb the latency overhead.
Copy link to heading7. Datadog LLM Observability
Datadog traces agent workflows end-to-end, calculates cost per request automatically, runs LLM-as-judge evaluations, detects hallucinations and prompt injection, flags personally identifiable information (PII), and correlates LLM traces with APM traces. Hence, a slow retrieval step traces back to the database query behind it. Cluster visualization flags drift.
Its OTLP ingest supports OTel GenAI semantic conventions v1.37+, gated behind OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental. Vercel Log Drains and Trace Drains feed Datadog directly.
Datadog announced a dedicated page per Vercel project in its Serverless view, with Logs and Traces tabs plus real user monitoring (RUM). The pull here is correlation. If your slowest LLM calls are usually downstream of a retrieval or database step, keeping LLM traces in the same backend as your infrastructure traces removes the cross-system stitching that stretches out incidents.
Pricing: Langfuse lists a free tier at 40k LLM spans per month with 15-day retention.
Best for: Organizations already paying for Datadog APM whose primary debugging need is correlating LLM traces with infrastructure metrics.
Copy link to headingHow to choose the right AI observability tool
Match the tool to the question you are trying to answer. The dominant failure mode in your app routes the decision:
Cost and provider reliability dominate: Start with Vercel AI Gateway, since it is already built in.
You need trace content and want open source: Choose Langfuse, or Arize Phoenix for local debugging.
Your team runs LangChain or LangGraph: Choose LangSmith for zero-code tracing.
You want traces feeding an eval dataset workflow: Choose Braintrust.
You want zero-code instrumentation and can absorb 50 to 80 ms: Choose Helicone.
You are on Datadog APM and need LLM-plus-infrastructure correlation: Choose Datadog LLM Observability.
Schema portability varies even when every tool here claims OpenTelemetry support. Phoenix uses OpenInference with the llm. prefix, while Datadog, Langfuse, and Braintrust are listed as Vercel AI SDK observability integrations alongside optional OpenTelemetry support on the latest GenAI conventions.
Check schema compatibility before you commit. That is the constraint most likely to surface during a later migration.
Copy link to headingStart at the gateway, then add the layer your failures demand
Gateway metrics are available to every team on Vercel from the first deployed request, and they answer the questions behind early incidents: cost spikes, latency regressions, and provider outages. Adding a purpose-built eval tool is the right next step once "is it slow?" stops being the dominant failure mode and "is it correct?" takes over.
The per-request metadata AI Gateway attaches is already structured for export to any tool in this comparison. Here is what that gives an engineering team without a custom instrumentation layer:
AI Gateway observability: Requests by model, TTFT, per-request cost, and provider plus fallback metadata, captured with zero instrumentation code.
AI SDK telemetry:
registerTelemetry, registered once at startup, traces every model call, tool execution, and agent step over OpenTelemetry.Custom Reporting API: Spend attribution by tag and user through
providerOptions.gateway.tagsandproviderOptions.gateway.user.OpenTelemetry export:
@vercel/oteland Vercel Drains route AI SDK spans to Datadog, Langfuse, Braintrust, and the rest of the backends above.Provider routing with fallbacks: Automatic failover with per-request metadata already shaped for downstream analysis.
Copy link to headingFrequently asked questions about AI observability tools
Copy link to headingWhat makes LLM observability different from traditional APM?
Traditional APM monitors deterministic systems where the same input produces the same output and a 200 OK means success. LLM apps break those assumptions, since the same prompt can return different outputs and a clean response can contain a hallucinated answer. LLM observability adds trace fidelity for non-deterministic outputs and evaluation signals that score whether the output was right.
Copy link to headingDoes the Vercel AI SDK work with all these tools?
Yes. The AI SDK registers a telemetry integration once at startup with registerTelemetry and exports over OpenTelemetry. The wiring varies by tool: LangfuseSpanProcessor for Langfuse, AISDKExporter for LangSmith, wrapAISDK for Braintrust, and OTel Trace Drains for Datadog and Braintrust.
Copy link to headingDo I need a dedicated AI observability tool from day one?
No. Teams can start with AI Gateway, which captures requests by model, TTFT, token counts, and per-request cost out of the box. That covers the first wave of production questions around cost, latency, and provider reliability. Add a purpose-built eval tool when output quality becomes the primary concern.
Copy link to headingAre OpenTelemetry GenAI spans portable across these tools?
Portability depends on the schema. Arize Phoenix uses OpenInference with the llm. prefix, while OpenTelemetry GenAI uses the gen_ai. prefix, so migrating Phoenix traces to a gen_ai.-native backend like Datadog can require attribute mapping. Verify schema compatibility before you commit.