Picture this: A user reports a confidently wrong answer, the on-call engineer opens the dashboards, and every graph is green. The request returned a 200 OK in under two seconds, and traditional application performance monitoring (APM) has no column for "the answer was wrong." That gap is the reason large language model (LLM) apps need their own observability layer.

Seven tools cover different parts of the problem, and each one answers a different production question. This guide maps the seven to the question each is built to answer.

**Key takeaways:**

- **Traditional APM misses the failure mode that matters:** A 200 OK response can carry a hallucinated answer, and request-level monitoring reads it as success.

- **Your AI gateway is the first observability layer:** It answers cost, latency, and provider-health questions before you need a dedicated eval platform.

- **The AI SDK exports over OpenTelemetry:** Traces flow to supported LLM observability backends, and [AI Gateway](https://vercel.com/docs/ai-gateway) attaches per-request cost and fallback metadata automatically.

- **Namespaces differ across tools:** The OpenTelemetry GenAI (`gen_ai.`) and OpenInference (`llm.`) conventions use different attribute prefixes, so tool choice affects which backends can parse your spans.

- **The right pick usually matches your existing surface:** If you already run Datadog, LangChain, or an open-source mandate, that instrumentation surface tends to decide the tool for you.

## [Copy link to heading](#what-is-ai-observability)**What is AI observability?**

AI observability is the practice of tracing, measuring, and evaluating LLM applications in production. It tells teams not only whether a request succeeded but whether the output was correct. It extends traditional APM with trace fidelity for non-deterministic outputs, token-level cost attribution, and evaluation signals that score response quality alongside delivery status.

The distinction that matters is the failure mode it is built to catch. AI observability captures the prompt, response, tool calls, and agent steps behind each request. That is what makes the silent failures visible.

### [Copy link to heading](#the-three-layers-of-llm-observability)**The three layers of LLM observability**

Traditional monitoring answers "is it working?" [LLM observability](https://vercel.com/docs/ai-gateway/observability-and-spend/observability) also has to answer "is it correct?", and those are different systems.

There are three layers, and conflating them is where tool evaluations go wrong:

- **Gateway metrics:** Cost, latency, token counts, and provider health, aggregated across every model you route to.

- **Trace capture:** Prompt and response content, plus the tool calls and agent steps around each request.

- **Eval signals:** LLM-as-judge scoring combined with human annotation, for regression detection over time.

Each layer answers a different class of question. If your dominant question is which model is slow and which feature costs the most, start at the gateway. If your question is whether the output is correct, you need evals.

The tipping point is the moment when "it's slow" stops being the problem and "it's wrong" becomes the problem. APM can tell you the request took two seconds. It cannot tell you the response was hallucinated.

The seven tools below sort cleanly once you know which layer your question lives in.

## [Copy link to heading](#the-7-best-ai-observability-tools-at-a-glance)**The 7 best AI observability tools at a glance**

The tools split across those three layers, and none of them cover all three the same way. Some start at the gateway and add tracing. Some start at tracing and add a full eval pipeline. A few correlate LLM traces with the rest of your infrastructure. Before comparing them, it helps to fix the axes that separate a tool you keep from one you outgrow.

We weighted six things that decide fit for a production LLM stack:

- **Layers covered:** Whether the tool handles gateway metrics, trace capture, eval signals, or a combination.

- **Vercel AI SDK integration:** How the tool ingests spans from the [AI SDK](https://vercel.com/docs/ai-sdk), and how much code that wiring takes.

- **Open source and self-hosting:** Whether you can run it in your own infrastructure, and under which license.

- **OpenTelemetry namespace:** Whether it emits or expects `gen_ai.`, `llm.`, or a proxy-based schema, which governs portability.

- **Latency overhead:** What the tool adds to the request path, where the vendor documents it.

- **Free tier and pricing:** Where the free tier ends and what the first paid step costs.

The table below lines up all seven on those axes. Read it for the namespace column and the layers column first, since those two decide the most about whether a tool fits your stack.

| Tool | Layers covered | Vercel AI SDK integration | Open source / self-host | OTel namespace | Latency overhead | Free tier |
| --- | --- | --- | --- | --- | --- | --- |
| Vercel AI Gateway \\+ AI SDK | Gateway \\+ trace | Built in (`registerTelemetry`) | No (platform feature) | `gen_ai.` \\+ `ai.` | Not documented | Included on all plans |
| Langfuse | Trace \\+ eval | `LangfuseSpanProcessor` | MIT, self-host | `gen_ai.` (aims for compliance) | None (async SDKs) | 50k units/mo |
| LangSmith | Trace \\+ eval | `wrapAISDK` | No; Enterprise-only self-host | OTLP ingest (July 2026\\) | Not documented | 5,000 traces/mo |
| Braintrust | Trace \\+ eval | `BraintrustExporter` / `BraintrustSpanProcessor`, or OTel Trace Drain | No (SaaS) | `gen_ai.` | Not documented | 1 GB \\+ 10k scores/mo |
| Arize Phoenix | Trace \\+ eval | Auto-instrumentor | ELv2, local-first | OpenInference (`llm.`) | Not documented | AX Free: 25k spans/mo |
| Helicone | Gateway \\+ trace | Base URL swap | Apache-2.0, self-host | Proxy-based | 50 to 80 ms | 10k requests/mo |
| Datadog LLM Observability | Trace \\+ eval \\+ APM | Vercel Drains | No | `gen_ai.` v1.37+ | Not documented | 40k LLM spans/mo |

Each tool below is organized by the question it answers best, starting with the layer that catches the first wave of production incidents.

## [Copy link to heading](#1.-vercel-ai-gateway-and-ai-sdk-tracing)**1\. Vercel AI Gateway and AI SDK tracing**

Every request routed through AI Gateway gets gateway-layer observability with no instrumentation code. That covers requests by model, time to first token (TTFT), input and output token counts, per-request cost, and metadata describing which provider served the request and any fallback attempts.

Over the BFCM 2025 window, [more than 24 million AI requests](/blog/bfcm-2025) passed through the gateway. The Usage tab shows aggregate request data. The Requests tab adds per-project and per-API-key summaries with request count, average tokens, P75 duration, P75 TTFT, and cost. Logs is sortable and exportable.

For spend attribution, the [Custom Reporting API](/blog/unified-reporting-for-your-ai-spend) breaks costs out by tag and user through `providerOptions.gateway.tags` and `providerOptions.gateway.user`.

Above the gateway sits [AI SDK 7](/blog/ai-sdk-7) tracing. Registered once at application startup with `registerTelemetry`, the SDK emits OpenTelemetry (OTel) spans for model calls, tool executions, agent steps, embeddings, and errors, with `onStepStart` and `onStepFinish` callbacks for multi-step agent runs.

Here’s how:

```
import { registerTelemetry } from 'ai';
import { DatadogTelemetry } from '@ai-sdk/datadog';
// or Langfuse, Braintrust, Raindrop, Sentry, Laminar, LangSmith

registerTelemetry(DatadogTelemetry());
```

After that call, every `generateText`, `streamText`, `generateObject`, and `streamObject` in the app is traced without further configuration. When an anomaly needs a closer look, `getGenerationInfo()` returns the per-generation details.

AI SDK 7 names Datadog, Langfuse, Braintrust, Raindrop, Sentry, Laminar, and LangSmith as OTel export backends, so the instrumentation you register here feeds any tool below. The reliability gains compound.

Zo Computer ran a 7.5% retry rate before switching to AI Gateway routing, then [improved reliability 20x](/blog/how-zo-computer-improved-ai-reliability-20x-on-vercel). Co-founder Rob Cheung said, "We didn't even know what we were missing until after we switched." At [Gamma](/blog/gamma-builds-design-first-agents-with-vercel), "every model automatically gets production-grade observability from its first request."

One caveat is worth knowing up front. The AI SDK emits proprietary `ai.` attributes alongside its OTel output, so [backends parsing only](https://community.vercel.com/t/opentelemetry-gen-ai-semantic-conventions-support/29859) `gen_ai.` miss SDK-specific span data.

**Pricing:** Gateway and AI SDK observability are included on all Vercel plans from the first deployed request. Extended data retention needs Observability Plus.

**Best for:** Teams on Vercel that want gateway and trace observability without a third-party tool, or that want a single instrumentation foundation feeding any tool below.

## [Copy link to heading](#2.-langfuse)2\. Langfuse

Langfuse is OpenTelemetry-native and accepts OTLP over HTTP/JSON and HTTP/protobuf, with gRPC not yet supported.

It defines [10 observation types](https://langfuse.com/docs/observability/overview) (event, span, generation, agent, tool, chain, retriever, evaluator, embedding, and guardrail) and layers on LLM-as-judge evaluation, prompt management with version tracking, datasets and experiments, and asynchronous SDKs that add no latency to the request path.

The [Vercel AI SDK integration](https://langfuse.com/integrations/frameworks/vercel-ai-sdk) wires `LangfuseSpanProcessor` from `@langfuse/otel` into `instrumentation.ts` alongside `@vercel/otel`, and the JS/TS SDK v4, generally available (GA) in September 2025, was rebuilt on OTel for exactly this path. [Prompt management](https://langfuse.com/changelog/2024-11-17-vercel-ai-sdk-prompt-mgmt) links to AI SDK traces so you can track performance per prompt version.

You can [self-host](https://langfuse.com/self-hosting) under the MIT license with Docker Compose or [Kubernetes](https://vercel.com/docs/integrations/external-platforms/kubernetes) and Terraform, or use Langfuse Cloud in EU, US, Japan, and HIPAA regions. The project has more than 21,000 GitHub stars, stores data in open-source ClickHouse, and was acquired by ClickHouse.

The MIT license is the real differentiator here. For a compliance-sensitive org, data portability and the option to run the whole stack in-house outweigh a slicker managed-only alternative.

**Pricing:** Hobby is free at [50k units per month](https://langfuse.com/pricing) with 30-day retention, Core is $29 per month with 90-day retention, and Pro is $199 per month with three-year retention.

**Best for:** Teams that want full eval depth with self-hosting and data portability.

## [Copy link to heading](#3.-langsmith)3\. LangSmith

[LangSmith](https://smith.langchain.com/) traces LangChain and LangGraph apps with no code changes, using [environment variables](https://vercel.com/docs/cli/env). Its July 2026 OTLP support accepts OTel traces from other stacks. It adds annotation queues with auto-assignment, production alerting, and prompt management, and [LangSmith Engine](https://docs.langchain.com/langsmith/engine), announced at Interrupt 2026, clusters failures automatically and proposes fixes.

For the [Vercel AI SDK](https://ai-sdk.dev/providers/observability/langsmith), import `wrapAISDK` from `langsmith/experimental/vercel` and wrap the `ai` module to trace every call with no further code changes, using `langsmith>=0.3.63`. Traces land in the LangSmith dashboard without adopting any LangChain framework.

The tradeoff is the backend. It is proprietary and closed-source, and self-hosting is Enterprise-only on Kubernetes in AWS, GCP, or Azure, so an open-source mandate rules it out. If your team already lives in LangChain or LangGraph, the zero-code tracing usually justifies that constraint.

**Pricing:** Developer is free at 5,000 traces per month with 14-day retention, and Plus is $39 per seat per month ([pricing](https://www.langchain.com/pricing-langsmith)).

**Best for:** Teams already on LangChain or LangGraph that want tracing with no code changes and can accept a proprietary backend.

## [Copy link to heading](#4.-braintrust)4\. Braintrust

[Braintrust](https://www.braintrust.dev/) adds automatic logging with cost tracking and evaluation for AI SDK 7, wired through `registerTelemetry` with its `BraintrustExporter` (Next.js) or `BraintrustSpanProcessor` (Node.js) and no application code changes. Its distinctive move is closing the loop between production and testing: traces convert into test cases with one click, and Loop generates custom scorers from natural language.

Real-time drill-down surfaces the slowest calls and the highest token consumption, with error patterns, model-level breakdowns, and cost attribution down to individual users and features.

On Vercel, you can skip the SDK path and add an OTel Trace Drain in Team Settings through the [Vercel Marketplace integration](https://www.braintrust.dev/blog/vercel-marketplace), then create `instrumentation.ts` and call `registerOtel`. Braintrust ships a [native OTel exporter](https://www.braintrust.dev/articles/best-llm-evaluation-tools-integrations-2025) with automatic LLM span conversion and OpenLLMetry support.

The design leans toward one workflow in particular. If your primary need is trace-first debugging, a trace-native tool fits better, but if you want production traffic feeding an offline eval dataset, this is the shortest path from trace to test case.

**Pricing:** Starter is free at 1 GB of processed data and 10k eval scores per month with 14-day retention, and Pro is $249 per month ([pricing](https://www.braintrust.dev/pricing)).

**Best for:** Teams whose primary workflow is feeding production traces into offline eval datasets.

## [Copy link to heading](#5.-arize-phoenix)5\. Arize Phoenix

[Phoenix](https://github.com/arize-ai/phoenix) is an Elastic License v2 (ELv2) open-source, local-first debugging environment: OTel-based tracing, LLM-as-judge evaluation, a prompt playground with model comparison and replay of traced calls, versioned datasets, and PXI, an AI agent for debugging traces and iterating on prompts.

Auto-instrumentors for the [Vercel AI SDK](https://arize.com/docs/ax/concepts/otel-openinference/overview) are available, and the managed counterpart, Arize AX, adds Agent Graph visualization and production monitoring.

The schema is the thing to account for before you commit. Phoenix uses OpenInference with the `llm.` attribute prefix (`llm.input_messages`, `llm.token_`[`count.total`](http://count.total)), while OpenTelemetry GenAI uses the `gen_ai.` prefix.

Arize [labels OTel GenAI](https://arize.com/docs/phoenix/tracing/concepts-tracing/otel-openinference/semantic-conventions) as "not on a stable release; subject to change at any time" and positions OpenInference as "Stable." The practical consequence is that Phoenix traces are not natively interchangeable with backends that ingest pure OTel GenAI spans, such as Datadog's OTLP ingest.

A later migration to a `gen_ai.`\\-native backend can mean attribute mapping. Account for that gap up front, and Phoenix gives you the most detailed local trace view in this list.

**Pricing:** AX Free is [25k spans per month](https://arize.com/pricing/) with 15-day retention, and AX Pro is $50 per month.

**Best for:** Teams that want a local, open-source debugging environment with detailed trace visualization before adding a managed backend.

## [Copy link to heading](#6.-helicone)6\. Helicone

[Helicone](https://github.com/helicone/helicone) takes the proxy approach: change the base URL on your LLM provider call, and you get observability, cost and latency tracking, agent tracing, evaluations, LLM routing with automatic fallbacks, and automatic response caching. A Vercel AI SDK provider is available.

The cost of that setup is spelled out, which is rare in this category. The distributed proxy adds an [average of 50 to 80 ms per call](https://www.helicone.ai/blog/the-complete-guide-to-LLM-observability-platforms), the only documented latency figure in this list. That is the price of a zero-instrumentation swap, and for a prototyping team it is usually the right trade, though a latency-sensitive production path may not absorb it.

The main platform is Apache-2.0 open source and self-hostable (Y Combinator W23), with the AI gateway component open-sourced separately in Rust, and SOC 2, HIPAA, and GDPR compliance land on the Team and Enterprise plans.

**Pricing:** Hobby is free at [10k requests per month](https://www.helicone.ai/pricing), Pro is $79 per month, and Team is $799 per month.

**Best for:** Teams prototyping fast that want observability from a single base-URL swap and can absorb the latency overhead.

## [Copy link to heading](#7.-datadog-llm-observability)7\. Datadog LLM Observability

[Datadog](https://www.datadoghq.com/) traces agent workflows end-to-end, calculates cost per request automatically, runs LLM-as-judge evaluations, detects hallucinations and prompt injection, flags personally identifiable information (PII), and correlates LLM traces with APM traces. Hence, a slow retrieval step traces back to the database query behind it. Cluster visualization flags drift.

Its OTLP ingest supports [OTel GenAI semantic conventions v1.37+](https://www.datadoghq.com/blog/llm-otel-semantic-convention), gated behind `OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental`. Vercel Log Drains and Trace Drains feed Datadog directly.

At [DASH 2026](https://www.datadoghq.com/blog/dash-2026-new-feature-roundup-observability), Datadog announced a dedicated page per Vercel project in its Serverless view, with Logs and Traces tabs plus real user monitoring (RUM). The pull here is correlation. If your slowest LLM calls are usually downstream of a retrieval or database step, keeping LLM traces in the same backend as your infrastructure traces removes the cross-system stitching that stretches out incidents.

**Pricing:** Datadog's [free tier](https://www.datadoghq.com/pricing/) covers 40k LLM spans per month with 15-day retention.

**Best for:** Organizations already paying for Datadog APM whose primary debugging need is correlating LLM traces with infrastructure metrics.

## [Copy link to heading](#how-to-choose-the-right-ai-observability-tool)How to choose the right AI observability tool

Match the tool to the question you are trying to answer. The dominant failure mode in your app routes the decision:

- **Cost and provider reliability dominate:** Start with Vercel AI Gateway, since it is already built in.

- **You need trace content and want open source:** Choose Langfuse, or Arize Phoenix for local debugging.

- **Your team runs LangChain or LangGraph:** Choose LangSmith for zero-code tracing.

- **You want traces feeding an eval dataset workflow:** Choose Braintrust.

- **You want zero-code instrumentation and can absorb 50 to 80 ms:** Choose Helicone.

- **You are on Datadog APM and need LLM-plus-infrastructure correlation:** Choose Datadog LLM Observability.

Schema portability varies even when every tool here claims OpenTelemetry support. Phoenix uses Arize's OpenInference package with the `llm.` prefix, while Datadog, Langfuse, and Braintrust are listed as Vercel AI SDK observability integrations alongside optional OpenTelemetry support on the latest GenAI conventions.

Check schema compatibility before you commit. That is the constraint most likely to surface during a later migration.

## [Copy link to heading](#start-at-the-gateway,-then-add-the-layer-your-failures-demand)Start at the gateway, then add the layer your failures demand

Gateway metrics are available to every team on Vercel from the first deployed request, and they answer the questions behind early incidents: cost spikes, latency regressions, and provider outages. Adding a purpose-built eval tool is the right next step once "is it slow?" stops being the dominant failure mode and "is it correct?" takes over.

The AI Gateway [processes billions of tokens per day](/blog/how-ai-gateway-runs-on-fluid-compute), and the per-request metadata it attaches is already structured for export to any tool in this comparison. Here is what that gives an engineering team without a custom instrumentation layer:

- **AI Gateway observability:** Requests by model, TTFT, per-request cost, and provider plus fallback metadata, captured with zero instrumentation code.

- **AI SDK telemetry:** `registerTelemetry`, registered once at startup, traces every model call, tool execution, and agent step over OpenTelemetry.

- **Custom Reporting API:** Spend attribution by tag and user through `providerOptions.gateway.tags` and `providerOptions.gateway.user`.

- **OpenTelemetry export:** `@vercel/otel` and Vercel Drains route AI SDK spans to Datadog, Langfuse, Braintrust, and the rest of the backends above.

- **Provider routing with fallbacks:** Automatic failover with per-request metadata already shaped for downstream analysis.

Start a new project at [vercel.com/new](https://vercel.com/new), or browse [vercel.com/templates](https://vercel.com/templates) for AI apps wired up with gateway observability from the first request.

## [Copy link to heading](#frequently-asked-questions-about-ai-observability-tools)Frequently asked questions about AI observability tools

### [Copy link to heading](#what-makes-llm-observability-different-from-traditional-apm)What makes LLM observability different from traditional APM?

Traditional APM monitors deterministic systems where the same input produces the same output and a 200 OK means success. LLM apps break those assumptions, since the same prompt can return different outputs and a clean response can contain a hallucinated answer. LLM observability adds trace fidelity for non-deterministic outputs and evaluation signals that score whether the output was right.

### [Copy link to heading](#does-the-vercel-ai-sdk-work-with-all-these-tools)Does the Vercel AI SDK work with all these tools?

Yes. The AI SDK registers a telemetry integration once at startup with `registerTelemetry` and exports over OpenTelemetry. The wiring varies by tool: `LangfuseSpanProcessor` for Langfuse, `wrapAISDK` for LangSmith, a `BraintrustExporter` or `BraintrustSpanProcessor` for Braintrust, and OTel Trace Drains for Datadog.

### [Copy link to heading](#do-i-need-a-dedicated-ai-observability-tool-from-day-one)Do I need a dedicated AI observability tool from day one?

No. Teams can start with AI Gateway, which captures requests by model, TTFT, token counts, and per-request cost out of the box. That covers the first wave of production questions around cost, latency, and provider reliability. Add a purpose-built eval tool when output quality becomes the primary concern.

### [Copy link to heading](#are-opentelemetry-genai-spans-portable-across-these-tools)Are OpenTelemetry GenAI spans portable across these tools?

Portability depends on the schema. Arize Phoenix uses OpenInference with the `llm.` prefix, while OpenTelemetry GenAI uses the `gen_ai.` prefix, so migrating Phoenix traces to a `gen_ai.`\\-native backend like Datadog can require attribute mapping. Verify schema compatibility before you commit.