Early production AI agent deployments often revealed a gap in observability: HTTP 200 dashboards looked clean even as users received incorrect answers.
Traditional observability tells you when systems are broken. Agent observability tells you when they’re succeeding for the wrong reasons.
In this guide, we explore agent observability in depth, walk through what to instrument, where silent failures hide, and how you build the layer that catches them before they reach your users.
Key takeaways:
Traditional services fail loudly with exceptions and 5xx codes, while agents fail quietly by returning HTTP 200 responses that can contain hallucinated content.
Individual requests mislead on their own because agent failures emerge from the sequence of turns, so the session is the real unit of observability.
Capturing prompts and completions makes debugging trivial but expands trace payloads, so teams capture full content in debug and sampled content in production.
A production agent observability stack comprises six layers: tracing, event capture, metrics, evaluation, guardrails, and session correlation, each addressing a distinct question.
The platform beneath a team determines how much of the observability stack it must build itself, since primitives like AI SDK telemetry and OpenTelemetry exports eliminate that work.
Copy link to headingWhat is agent observability?
Agent observability is the practice of tracing and measuring autonomous AI agents in production. It extends traditional application performance monitoring with session-scoped tracing, token cost attribution, evaluation integration, and debugging workflows built for non-deterministic, multi-step agent workloads.
The signals it captures aren’t just whether a request succeeded but also whether the agent took the right path to the answer, called the right tools, and produced a response that’s grounded rather than confidently wrong.
Copy link to headingWhat are the differences between traditional observability and agent observability?
The biggest operational difference is what counts as a failure mode. Traditional services fail loudly. They throw exceptions, return 5xx codes and time out at the load balancer. Agents fail quietly. They return HTTP 200 with hallucinated content, call the wrong tool but get a plausible-looking response, or partially succeed in a way that appears to be complete success.
The signals diverge across five dimensions that teams have to instrument for differently:
Teams often underweight the trace structure dimension. A traditional call chain is something a human can read line by line. An agent’s execution graph with branching and loops doesn’t fit on a whiteboard.
This means the trace UI you pick determines whether you can debug at all once production traffic gets messy.
Copy link to headingWhy agent observability matters for production teams
Agent observability earns its keep across four operational dimensions that traditional monitoring doesn’t try to cover:
Semantic failure detection: Standard error-rate dashboards don’t tell you whether the answer was grounded or whether the tool call was appropriate. Agent observability captures the reasoning path, allowing teams to ask, “Did this succeed correctly?” rather than “Did this succeed?”
Non-deterministic debugging: The same prompt produces different outputs across runs. Teams without the reasoning chain captured at execution time end up trying to reproduce failures from prompts that no longer fail the same way. Capturing the run is the only path to debugging it.
Cost visibility per session, not per request: Retry loops, context window growth, and tool re-invocations push token usage in ways that look fine at the call level and disastrous at the session level. Without session-scoped cost attribution, teams discover budget overruns from invoices, not dashboards.
Compliance and audit trails for regulated workloads: Durable records around prompts, tool calls, and outputs are required for internal review and external audit in regulated industries. The audit trail isn’t an add-on; it’s the runtime requirement that determines whether the agent can ship at all.
What ties these together is session-scoped tracing. Individual requests in an agent system are misleading on their own. The conversation is the unit that matters because failures emerge from the sequence of turns, not from any single one.
Teams that get this right structure their observability around sessions from day one, and the work compounds. Teams that retrofit it later spend months stitching span IDs and conversation IDs together while trying to answer questions the schema wasn’t designed to answer.
The instrumentation that supports those four operational benefits comes from a stack of telemetry primitives, each answering a different question about what the agent did.
Copy link to heading6 core components of an AI agent observability stack
A production agent observability stack combines six layers:
Tracing
Event capture
Metrics
Evaluation
Guardrails
Session correlation
Together, they give teams a clear picture of how an agent behaves over time.
Let’s see how.
Copy link to heading1. Distributed tracing with typed spans
Distributed tracing for AI agents extends the standard span model with operation types that map to agent execution. Model calls, tool invocations, workflows, and retrieval steps each get their own span type.
The OpenTelemetry GenAI Semantic Conventions define attributes for requests, token usage, and responses, ensuring traces remain consistent across whatever telemetry backend a team uses.
The first priority is per-call span context. Without it, you can see a trace fail, but you can’t tell which model call, which tool, or which retrieval step was responsible.
The honest tradeoff is content capture. Including prompts, completions, and tool arguments in the trace makes debugging trivial and the trace payload expensive at high request volumes. The pattern that works is selective capture.
Full content in debug environments, sampled or redacted content in production traces. Pay for the depth where the depth pays off, not on every request.
Copy link to heading2. Structured event capture
Structured event capture records what happened at each step of execution. Duration, token usage, tool calls, intermediate outputs, and the final response.
The OTel GenAI Events specification defines events as point-in-time occurrences within a span, which is the right model for capturing tool-call success and failure, step latency, and partial-success states without forcing them into the span hierarchy.
The failure pattern teams hit is treating events as logs. They aren’t. Logs are unstructured and grep-able. Events are typed records you can aggregate, filter, and join against traces.
Step lifecycle callbacks in Vercel’s AI SDK make this clean for agent workflows because they emit per-step data alongside trace context, so the event and the span share the same identifier. That correlation is what lets a team move from “something went wrong at step three” to “here’s the prompt, tool call, and intermediate output for step three” in one query.
Copy link to heading3. Metrics and alerting
Metrics turn individual agent runs into operational trends. The OTel GenAI metrics specification covers token usage, request latency, and error rates, with attributes that fit agent workloads more naturally than request-only monitoring does.
Teams should monitor these four metrics together:
Error rate
P95 latency
Token cost per session
Tool-call success rate
These metrics are most useful as a set because any one of them can move for benign reasons. Latency climbs when the model is busy; token cost climbs when context grows; error rate moves with provider outages.
Together, they tell you whether something structural is changing or whether one provider is just having a bad day. Where this breaks down is in alerting on agent quality. Quality isn’t a metric you can threshold cleanly, which is why evaluation belongs in a separate layer that runs alongside the metrics, not on top of them.
Copy link to heading4. Evaluation and quality scoring
Evaluation gives teams a repeatable way to score agent behavior. The signal applies to a single span, a full trace, or an entire multi-turn session, depending on what the agent is meant to do. Scoring at the wrong level produces misleading results.
The teams that do this well combine automated scoring with sampled human review and feed the results back into prompts, guardrails, and regression tests.
Deterministic checks cover structured criteria:
Did the JSON validate?
Did the tool call match the expected schema?
Did the response include the required fields?
Open-ended quality scoring covers the harder cases. Was the answer grounded, was the reasoning sound, did the agent stop when it should have stopped? Both signals sit next to traces and metrics in the production feedback loop, not as a separate offline activity that nobody acts on.
Copy link to heading5. Guardrail monitoring
Guardrail monitoring tracks the checks that run alongside agent execution. Policy violations, unsafe tool attempts, confidence thresholds, and prompt injection signals all belong here.
What separates guardrails from evaluation is timing:
Evaluation runs later as part of the ongoing quality review
Guardrail monitoring belongs on the active response path, blocking or modifying responses before they reach the user.
A reliable pattern is recording guardrail decisions in the same trace as the output they influenced. Without that correlation, you can’t tell whether a guardrail caught a real problem or fired on a benign request.
Over time, the guardrail telemetry becomes the most useful signal for tuning the guardrails themselves. Teams that skip the correlation end up with guardrails that either over-block legitimate requests or under-block the cases they were meant to catch. And they can’t tell which without rebuilding the relationship between guardrail events and outputs from scratch.
Copy link to heading6. Session and multi-agent correlation
Session correlation links related traces across a conversation or a multi-agent workflow. Agent systems rarely operate in isolation, especially when one model hands off work to another tool or agent before the session completes. OpenTelemetry context propagation via trace and span identifiers enables session-scoped views.
This matters most in multi-agent debugging and drift detection. When an orchestrator agent calls a research agent, which calls a retrieval tool, which calls another model, you have four traces that mean nothing in isolation.
Connected by shared identifiers, they become one execution graph you can read. The same view also catches semantic drift between turns. Early responses asserting X and later responses contradicting X only become visible when you can read the full session in a single connected view.
Teams that treat the session as the primary unit of observability from the start avoid the rewrite that hits teams who started with request-level observability and later tried to retrofit conversation context.
The components above describe what to instrument. In the next section, we examine the operational practices that determine whether the instrumentation performs its job once production traffic hits it.
Copy link to heading5 best practices to implement AI agent observability for engineering teams
The instrumentation only matters if the operating practices around it support the signals. The five practices below are what every engineering team running agents should adopt before scale exposes the gaps.
Copy link to headingInstrument typed spans for every agent step
A common pitfall is teams instrumenting the model call and treating tool calls, retrieval steps, and intermediate decisions as black boxes. When that pattern hits production, every failure produces the same useless trace. “Something went wrong in the agent loop.”
Per-step spans give you an execution map. When an agent runs branches across model calls, tools, and retrieval, typed spans make it obvious which step produced what. The OpenTelemetry GenAI conventions provide the schema. Where content capture belongs is a per-environment decision, not a global one. Sampled production traffic, full debug traces.
Copy link to headingPropagate trace context across every service boundary
Multi-service agent systems are hardest to debug when the trace fragments are at service boundaries. Teams hit this when an agent’s tool call hits an internal microservice that doesn’t propagate context. The resulting trace shows the request entering a black hole and returning a few seconds later.
Propagating trace context through every downstream service keeps the execution graph connected. That continuity matters more as agent systems grow. If you do this from day one, you can follow one connected trace through the components that shaped the final output.
The teams that don’t do this end up correlating logs by timestamp during incidents, which is the slowest debugging mode there is.
Copy link to headingSet token cost budgets at the session level
Token usage is invisible at the call level until the invoice arrives. A single model invocation may look efficient, whereas the broader workflow loops between two tools, grows context across turns, or repeatedly invokes the same retrieval without making progress.
The first time a team gets surprised by a token bill, it’s almost always a session-level pattern that looked fine at every individual call.
With session-level accounting, you expose those patterns earlier. Combined with span-level token attribution, this provides teams with a clear view of where costs accumulate within a multi-step run. Set the per-session budget and alert before it blows through, not after.
Copy link to headingTreat the session as the unit of observability, not the request
A request can look healthy on its own while the larger conversation drifts. Teams instrument at the request level because that’s how HTTP services work, and then discover the agent’s behavior only makes sense across turns. By the time they realize, they’ve shipped a metric layer that can’t answer the questions that actually matter.
Session-scoped observability keeps the full interaction connected. Task completion, multi-turn quality, and behavioral traceability: these are session-level signals. If your team needs to reconstruct how an agent arrived at an answer for auditing or debugging, the session is the unit that ties all spans together.
Copy link to headingRun continuous evaluation as part of production pipelines
Evaluation as a one-time offline activity becomes useless the moment your prompts, models, or data sources change. A common mistake is treating evaluation as a pre-launch gate, then skipping it after launch. Models drift. Prompts get updated. Tool integrations change. The eval that passed last month doesn’t tell you anything about this week.
Production evaluation works best as an ongoing signal. Automated scoring on sampled production traffic, human review on the rest, results fed back into prompts and guardrails. Over time, that turns observability from a debugging tool into part of the quality loop.
Copy link to headingHow Vercel powers AI agent observability for engineering teams at scale
Vercel supports agent observability through AI SDK telemetry primitives and OpenTelemetry-based export into whatever observability backend an engineering team already runs.
The combination captures per-call details, aggregate behavior, and multi-step workflow context while letting traces flow into the tooling teams already trust.
Copy link to headingBuilt-in telemetry on every AI SDK call
If you adopt the AI SDK without telemetry, you end up flying blind on every model call. Manual instrumentation per call site is the alternative, and the friction it adds means most teams skip it on the calls that matter most. A common mistake is leaving instrumentation off the exact call teams later need to inspect.
The AI SDK includes telemetry support via the telemetry option for generateText, streamText, and related calls. That gives teams a direct path to attaching tracing metadata to model calls without writing the instrumentation layer themselves:
const result = await generateText({ model: 'openai/gpt-5.6-sol', prompt: 'Summarize the latest order status.', telemetry: { isEnabled: true, functionId: 'order-summarizer', recordInputs: true, recordOutputs: true, }, runtimeContext: { userId: 'u-123', sessionId: 's-456' },});The recordInputs and recordOutputs flags also let teams decide whether to capture prompt and response content on a per-call basis. For sensitive workloads, leaving them off preserves structural telemetry without storing the payloads.
Copy link to headingZero-instrumentation cost and latency dashboards via AI Gateway
Multi-provider teams hit the same problem. OpenAI in one app, Anthropic in another, an open-source model somewhere else. Three providers mean three billing portals, three latency reports, three sets of error logs.
Aggregating that view is real work, and most teams don’t do it until they’ve already lost track of what they’re paying for and where the latency is coming from.
AI Gateway provides observability for LLM API calls routed through it. Latency, token cost, and error rates are consolidated in a single dashboard across providers, which is the operational view teams otherwise build for themselves.
For follow-up analysis, the getGenerationInfo() API returns generation details by ID, so teams can connect aggregate views to per-generation investigations when an anomaly occurs.
Copy link to headingMulti-step agent visibility with step lifecycle callbacks
Multi-step runs fail in ways that look identical at the surface. The agent returned something, but which step produced the wrong tool call? Which retrieval came back empty? Which model call exhausted the context window? Without per-step telemetry, every multi-step failure becomes a guessing game, and the guessing gets harder as the workflow grows.
The AI SDK exposes onStepStart, onStepEnd, onEnd, and the steps property on the result, so teams can observe intermediate work as a run progresses:
import { generateText, isStepCount } from 'ai';
const result = await generateText({ model: openai('gpt-5.6-sol'), tools: { /* ... */ }, stopWhen: isStepCount(5), onStepStart({ stepNumber, messages, steps }) { // ... }, onStepEnd({ stepNumber, toolCalls, usage, performance }) { // ... }, onEnd({ usage, steps }) { // usage is aggregated across all steps }, telemetry: { isEnabled: true },});Those callbacks collect workflow-level usage and step-level context in the same execution path. When combined with telemetry metadata such as functionId, they make multi-step debugging actionable rather than speculative.
Copy link to headingOpenTelemetry export to any observability backend
Teams with an existing observability stack hit a real problem when adopting AI agents. Bolting on a separate vendor for agent telemetry means correlating across two systems during incidents. This is exactly what observability is supposed to prevent.
The right answer isn’t a new tool; it’s getting agent traces into the backend the team already runs and trusts.
The @vercel/otel package initializes OpenTelemetry instrumentation for apps running on Vercel Functions. For AI SDK calls, @ai-sdk/otel registers the AI SDK’s OpenTelemetry integration so model calls, agent steps, and tool executions can emit GenAI spans.
import { registerOTel } from '@vercel/otel';import { registerTelemetry } from 'ai';import { OpenTelemetry } from '@ai-sdk/otel';
export function register() { registerOTel({ serviceName: 'my-agent-app', });
registerTelemetry(new OpenTelemetry());}Vercel Drains can then forward trace data in OpenTelemetry format to supported native integrations or a custom endpoint. That keeps AI SDK agent telemetry in the same observability workflow as application and infrastructure traces, provided the team has configured the appropriate trace drain or integration.
Copy link to headingAI-powered anomaly investigation with Vercel Agent
Telemetry helps you see anomalies. Connecting the anomaly to a root cause across spans, events, metrics, and logs during an incident is the slow part. Teams either build the investigation runbooks themselves or they accept that the first hour of every incident is stitching signals together by hand.
With Vercel Agent, you extend observability with AI-assisted investigations. For agent workloads, that means moving from anomaly detection to a readable summary of likely causes, without manually stitching every signal together. The investigation layer builds on the telemetry already in place and provides a starting point for the actual fix, rather than a dashboard the team has to interpret on its own.
Copy link to headingShip reliable AI agents with end-to-end observability on Vercel
Traditional observability tells you when systems are broken. Agent observability tells you when they’re succeeding for the wrong reasons. The teams that ship reliable agents are deciding what counts as success per session, then building the telemetry stack that catches the failures their definition of success would miss.
The platform beneath them determines how much of that stack they must build themselves.
Here’s how Vercel collapses the agent observability stack onto a small set of primitives engineering teams can adopt without writing their own instrumentation layer:
AI SDK with built-in telemetry:
telemetryon everygenerateTextandstreamTextcall gives teams span context, metadata, and selective content capture without a custom instrumentation layer.AI Gateway with cross-provider observability: Unified latency, token cost, and error reporting across every LLM provider routed through the gateway, with
getGenerationInfo()for per-generation drill-down when anomalies surface.Step lifecycle callbacks:
onStepEndandonStepStartexpose intermediate steps in multi-step runs, so teams can correlate workflow-level usage with the exact step where behavior changed.@vercel/oteland Vercel Drains: OpenTelemetry export wires agent traces into the observability backend the team already runs, so agent telemetry sits alongside application and infrastructure traces.Vercel Agent investigations: AI-assisted incident investigation moves teams from anomaly detection to a readable causal summary, without manually stitching together every signal during the first hour of an incident.
Start a new project to put these patterns into practice, or browse Vercel templates for AI agent examples already wired up for production observability.
Copy link to headingFrequently asked questions about agent observability
Copy link to headingHow does agent observability differ from traditional APM for AI workloads?
Traditional APM focuses on latency, error rates, and throughput. Agent observability adds token attribution, tool tracing, reasoning-step visibility, and quality evaluation, so teams can review how a response was produced and whether it succeeded for the right reasons, not whether it returned an HTTP 200.
Copy link to headingWhat is the minimum instrumentation needed for a production AI agent?
Start with telemetry on each generateText or streamText call, plus onStepEnd for step-level visibility. That setup gives teams span context, metadata, and a clear view of what happened at each step before expanding into session correlation and continuous evaluation.
Copy link to headingHow do teams handle PII in agent traces without breaking compliance?
The AI SDK’s recordInputs: false and recordOutputs: false flags let teams preserve structural telemetry without storing prompt or response content for a given call. Content capture stays on only where debugging value outweighs the data-handling constraints, not by default.
Copy link to headingCan existing observability backends receive agent traces from Vercel?
Yes. Vercel supports OpenTelemetry-based export through @vercel/otel and Vercel Drains. Teams already using external observability tooling can keep agent traces within the same monitoring workflow as the rest of the application, rather than fragmenting visibility across systems.