---
title: Trace and debug eve agent runs
description: Inspect eve traces in Vercel Agent Runs, control captured content, export OpenTelemetry spans, and test fixes with regression evals.
url: "https://vercel.com/kb/guide/trace-and-debug-eve-agent-runs"
published: 2026-11-02
last_updated: 2026-11-02
authors: Ben Sabic
install_vercel_plugin: npx plugins add vercel/vercel-plugin
---

eve's OpenTelemetry spans record model calls and tool execution. Use them to investigate wrong answers, slow steps, or unexpected token usage.

On Vercel, Agent Runs receives traces without a custom instrumentation file once sampling is configured. Capture policies determine which spans and content are available. You can also export traces to Datadog, Langfuse, or Braintrust.

## What an eve run records

An eve session can contain several turns. Each root agent turn starts a trace with an `agent.step` span for each loop step and the model and tool calls nested beneath it:

```text
invoke_agent <agent>
  └── agent.step
        ├── chat <model>
        └── execute_tool <tool>
```

Skill loads appear as `execute_tool eve:load-skill` spans. If eve receives the caller's trace context, a local subagent's first turn joins that trace beneath the dispatch span. Without that context, it starts a separate trace.

Remote subagents start separate traces and link their first turn back to the caller. Later turns in child sessions start new traces.

Use these eve trace schema v4 attributes to connect related spans:

| Attribute                | What it identifies                                     |
| ------------------------ | ------------------------------------------------------ |
| `gen_ai.conversation.id` | The root session and its child sessions, across traces |
| `agent.run.id`           | One agent session, not the full conversation           |
| `gen_ai.tool.call.id`    | One tool call, matched with its approval spans         |
| `error.type`             | The class of a tool failure                            |
| `gen_ai.usage.*`         | Token counts on model spans                            |
| `agent.usage.*`          | Token counts on step and tool spans                    |

Model spans carry input, output, and cached token counts, and step spans add cost when Vercel AI Gateway served the call. Your trace content policy decides whether a span also holds prompts, responses, tool arguments, and tool results.

You can inspect the same spans locally before you deploy. `eve dev` writes traces to `.eve/traces/v1`, `eve traces list` lists them, and `eve traces --verbose` expands each span with all of its attributes:

```bash
eve traces list
eve traces --verbose
```

## Find and inspect runs in Agent Runs

Agent Runs is eve's built-in OpenTelemetry destination on Vercel. Preview and production deployments export to it by default without an instrumentation file.

During beta, Agent Runs is available on all plans with 30-day retention. Usage follows [Always-on Tracing pricing](https://vercel.com/docs/tracing/always-on-tracing#usage-and-pricing).

### Turn on trace sampling

When `eve deploy` creates a new Vercel project, it configures 100% trace sampling for all environments. Existing projects collect no traces until they have a sampling rule, so open your project's [**Settings → Tracing**](https://vercel.com/d?to=%2F%5Bteam%5D%2F%5Bproject%5D%2Fsettings%2Ftracing&title=Go+to+Tracing+settings), then:

1. Select **Add Sampling Rule**.
   
2. Choose **All Environments**, **Production**, or **Preview**.
   
3. Set **Rate** to `100%` to sample every matching request.
   
4. Leave **Request Path Prefix** blank to sample every path, then select **Save**.
   

Vercel applies the first matching rule, which you can update in the dashboard or with [`vercel traces config`](https://vercel.com/docs/cli/traces#manage-trace-sampling-rules). Set its rate to 100%.

If you restrict sampling by path, cover `/eve/v1` (or a named prefix such as `/eve/support/v1`), `/.well-known/workflow`, and any custom channel routes. Agents using dynamic schedules also need `/.well-known/eve/v1/schedules/consumer`.

Missing routes can leave gaps. Sampling does not restore spans or content excluded by eve's policies.

### Inspect a run in the dashboard

Open your project and go to [**Observability → Agent Runs**](https://vercel.com/d?to=%2F%5Bteam%5D%2F%5Bproject%5D%2Fobservability%2Fagent-runs&title=Agent+Runs). The overview charts runs by trigger (such as Slack or HTTP) and token usage split into input, output, and cached tokens. Below the charts, a table lists each run with its triggering message, tokens, turn count, and duration.

Select a run to see its model, trigger, deployment, and per-turn details:

- Timing for every step, including skill loads and individual tool calls
  
- The turn's input and output
  
- The model's reasoning
  
- Each tool call with its arguments and results
  
- Input, cached, and output token counts
  

Prompts, responses, reasoning, and tool payloads appear only when recorded. Models may not expose reasoning, and capture policies can exclude content.

### Inspect a run from the terminal

Use [`vercel agent-runs`](https://vercel.com/docs/cli/agent-runs) in a terminal or coding agent's shell. Replace the team and project names below, then choose a run ID from `list`:

```bash
vercel agent-runs projects --scope my-team
vercel agent-runs list --scope my-team --project my-app
vercel agent-runs inspect run_1234567890 \
  --scope my-team --project my-app
vercel agent-runs trace run_1234567890 \
  --scope my-team --project my-app \
  --json --max-field-length 0
```

Commands query production by default. Add `--environment preview` for preview runs and use the same environment when inspecting the selected run. `inspect` returns metadata, lifecycle events, usage, and subagents.

`trace` returns the recorded conversation. Combine `--json` with `--max-field-length 0` to preserve full field values; Markdown output still shortens long content without `--json`.

For additional pages, follow the continuation commands printed by the CLI. These retain the relevant run and filter context. Untruncated fields do not combine pages or restore content that was never recorded.

If your editor uses [Vercel MCP](https://vercel.com/docs/agent-resources/vercel-mcp), it can inspect runs with `list_agent_run_projects`, `list_agent_runs`, `get_agent_run`, and `get_agent_run_trace`.

Trace results can be paginated or unavailable. Follow the [response and pagination instructions](https://vercel.com/docs/agent-resources/vercel-mcp/tools/agent-runs/get_agent_run_trace#response-and-pagination) before treating an empty page as evidence that a run had no activity.

## Control what traces capture

eve controls trace content at two levels. The process-wide `tracePolicy` in `otel()` sets the maximum content any OpenTelemetry destination can receive. Each destination's `exportPolicy` can only narrow that content.

### Know the default

Channels set a session's audience to `public`, `private`, or `unknown` when creating it. Audience determines content capture, not who can access traces. The default policy retains metadata and captures content as follows:

| Environment           | Audience               | Content recorded   |
| --------------------- | ---------------------- | ------------------ |
| Development           | Any                    | Inputs and outputs |
| Preview or production | `public`               | Inputs and outputs |
| Preview or production | `private` or `unknown` | Metadata only      |

Inputs cover model prompts, tool arguments, and channel input. Outputs cover model responses, reasoning, tool results, and error details.

Metadata includes span names, timing, model and tool names, and reported token usage. These fields remain available when content capture is disabled.

### Set the process-wide policy

Declare `otel()` once, in `agent/instrumentation/otel.ts`. This policy keeps the default rules for public and private sessions and drops traces for `unknown`\-audience sessions outside development:

```typescript
import { otel } from "eve/instrumentation/otel";

export default otel({
  resource: {
    "deployment.environment": process.env.VERCEL_ENV ?? "development",
  },
  tracePolicy: ({ audience, environment }) => {
    if (audience === "unknown" && environment !== "development") {
      return { emit: false };
    }
    const recordContent =
      audience === "public" || environment === "development";
    return {
      emit: true,
      recordInputs: recordContent,
      recordOutputs: recordContent,
    };
  },
});
```

Keep the policy deterministic because eve can evaluate it more than once while preparing a session. Its resolved decision sets the session's default Workflow run-data retention when the session starts.

If the policy drops a trace or excludes inputs or outputs, eve asks the Workflow backend to delete stored run payloads and streams when the run ends. Completed transcripts then become unreadable on backends that support deletion.

Set `experimental.workflow.retention: "default"` in `defineAgent()` to retain run data for the backend's default period independently of tracing. Destination `exportPolicy` filters do not change this retention decision.

Deleting run data leaves local traces and exported telemetry intact.

### Narrow what Agent Runs receives

Default-export `agentRuns()` to set the policy for the Agent Runs destination.

This policy keeps the spans it receives, removes inputs and outputs from non-public sessions, and drops the custom `customer.id` attribute:

```typescript
import { agentRuns } from "eve/instrumentation/otel";

export default agentRuns({
  exportPolicy: {
    span: ({ audience }) =>
      audience === "public"
        ? { emit: true }
        : { redact: true, inputs: true, outputs: true },
    attribute: ({ key }) =>
      key === "customer.id" ? { emit: false } : { emit: true },
  },
});
```

The `span` callback keeps, drops, or redacts each span. Return `{ emit: true }` to keep it or `{ emit: false }` to drop it. Use `{ redact: true }` with `inputs`, `outputs`, or both to remove content. If the callback throws, Agent Runs drops the span.

The `attribute` callback can keep, remove, or replace a value. Return `{ emit: true }`, `{ emit: false }`, or `{ replace: true, value }` for each action. Use it to filter custom attributes outside eve's built-in redaction.

Input redaction removes eve's prompt, instruction, document, and tool-argument attributes. Output redaction removes response, reasoning, tool-result, and exception details, plus status messages. The span's status code is preserved.

Redaction in `agentRuns()` affects only Agent Runs. To turn Agent Runs off, default-export `disableInstrumentation()` from `eve/instrumentation` in the same file.

Local traces from `eve dev` include content by default and obey the process-wide policy. Set `EVE_TRACES_CONTENT=off` in `.env.local` to redact local inputs and outputs. This setting affects only local traces.

**Warning:** Captured model and tool content is visible to project members with trace access. If your agent handles personal, sensitive, or regulated data, review what it records and whether your privacy disclosures need to cover it.

## Export traces to your observability tool

To send traces somewhere besides Agent Runs, add one file per destination under `agent/instrumentation/` and default-export `otelIntegration()` from each.

eve already emits OpenTelemetry spans, so you don't register a tracer provider yourself. Every `otelIntegration()` destination shares the process-wide `tracePolicy`, and Agent Runs keeps receiving spans alongside it.

### Export to Datadog

This example sends eve traces to Datadog APM over OTLP/HTTP.

#### 1\. Install the exporter

Install `@vercel/otel`, which provides the OTLP exporter:

#### 2\. Add your Datadog credentials

The exporter below builds Datadog's Vercel OTLP endpoint from `DD_SITE`. Set this variable to your Datadog site, such as `datadoghq.com` or `datadoghq.eu`.

Add the site and API key as production environment variables:

```bash
vercel env add DD_SITE production
vercel env add DD_API_KEY production
```

#### 3\. Declare the destination

Add the exporter in this instrumentation file:

```typescript
import { OTLPHttpProtoTraceExporter } from "@vercel/otel";
import { otelIntegration } from "eve/instrumentation/otel";

const site = process.env.DD_SITE ?? "datadoghq.com";

export default otelIntegration({
  exportPolicy: {
    span: () => ({ redact: true, inputs: true, outputs: true }),
  },
  traceExporter: new OTLPHttpProtoTraceExporter({
    url: `https://vercel.integrations.otlp.${site}/v1/traces`,
    headers: { "dd-api-key": process.env.DD_API_KEY! },
  }),
});
```

The `exportPolicy` keeps span names, timing, and reported token usage in Datadog, but removes prompts, responses, and tool payloads. Delete it if your `tracePolicy` already limits content to what Datadog may store.

eve wraps `traceExporter` in its own batching processor, so you don't add one.

#### 4\. Deploy and send a message

Deploy the agent to production:

```bash
eve deploy
```

Send your agent a message, then open Datadog APM and find the service named after your agent. eve sets `service.name` to the agent's name unless you set `OTEL_SERVICE_NAME`. Root agent turns appear as separate traces, with the same `agent.step`, `chat`, and `execute_tool` spans you see in Agent Runs.

Datadog's managed-platform endpoint doesn't attach host metadata, so features such as the Infrastructure Host List stay empty for these traces. If you already run an OpenTelemetry Collector with the Datadog exporter, point `url` at your Collector's OTLP/HTTP traces endpoint.

### Export to Langfuse

Langfuse accepts OTLP over HTTP/protobuf at `/api/public/otel/v1/traces`. Authenticate with Basic auth built from your project's public and secret keys. The `x-langfuse-ingestion-version: 4` header selects its v4 ingestion model.

Install `@vercel/otel` if needed, then set `LANGFUSE_PUBLIC_KEY` and `LANGFUSE_SECRET_KEY` in your deployment environment. Add the exporter below:

```typescript
import { OTLPHttpProtoTraceExporter } from "@vercel/otel";
import { otelIntegration } from "eve/instrumentation/otel";

const auth = Buffer.from(
  `${process.env.LANGFUSE_PUBLIC_KEY}:${process.env.LANGFUSE_SECRET_KEY}`,
).toString("base64");

export default otelIntegration({
  traceExporter: new OTLPHttpProtoTraceExporter({
    url: "https://cloud.langfuse.com/api/public/otel/v1/traces",
    headers: {
      Authorization: `Basic ${auth}`,
      "x-langfuse-ingestion-version": "4",
    },
  }),
});
```

The example URL is Langfuse's EU region. Use `https://us.cloud.langfuse.com` for the US region, or your own host for a self-hosted instance.

### Export to Braintrust

Braintrust accepts OTLP at `https://api.braintrust.dev/otel/v1/traces`, or `https://api-eu.braintrust.dev/otel/v1/traces` on the EU data plane.

Install `@vercel/otel` if needed and add `BRAINTRUST_API_KEY` to the deployment environment. Requests use that key as a Bearer token.

Set `x-bt-parent` to select the destination project. In the file below, replace `my-eve-agent` with your Braintrust project name:

```typescript
import { OTLPHttpProtoTraceExporter } from "@vercel/otel";
import { otelIntegration } from "eve/instrumentation/otel";

export default otelIntegration({
  traceExporter: new OTLPHttpProtoTraceExporter({
    url: "https://api.braintrust.dev/otel/v1/traces",
    headers: {
      Authorization: `Bearer ${process.env.BRAINTRUST_API_KEY}`,
      "x-bt-parent": "project_name:my-eve-agent",
    },
  }),
});
```

This OTLP integration sends only content allowed by `tracePolicy`.

The registry command `eve add instrumentation/braintrust` installs a different provider: `braintrustEveInstrumentation()` from `braintrust`. It uses lifecycle events and does not apply `tracePolicy` or `exportPolicy`.

In `braintrust@3.42.0`, that provider records inputs and outputs regardless of audience and listens for event names used before eve 0.84. Check its compatibility with your eve version before installing it.

## Investigate a wrong answer and add a regression eval

Trace the answer back through the model and tool calls to find where it diverged from the available evidence. Suppose an order-support agent tells a customer "Order 4417 shipped yesterday" while the order is still being packed.

### 1\. Find the run

Filter the Agent Runs table by time and trigger, or list runs from the terminal and pick the run ID:

```bash
vercel agent-runs list --scope my-team --project order-agent
```

Under the default policy, deployed traces for private or unknown-audience sessions contain metadata only. You can still see the order of tool calls and which calls failed.

To inspect prompts and results, reproduce the conversation with `eve dev` using test data. Local traces include content by default unless your policy excludes it. Select the saved trace with `eve traces list`, then inspect its ID:

```bash
eve traces show your_trace_or_conversation_id
```

### 2\. Walk the steps in order

Open the run and read each turn's `agent.step` spans from top to bottom. For each step, compare what the model received with what it decided:

- In the `chat` span, compare the prompt with the model's response. Check whether it calls a tool or answers directly.
  
- Read each `execute_tool` span's arguments and result. Use its `error.type` attribute to investigate failures.
  
- Check slow skill loads and tool calls for timeouts using the step timings.
  

In this example, step 1's `chat` span answered directly with no `execute_tool` span beneath it. The model stated an order status without calling `get_order_status`.

If the tool had failed, inspect the call and the answer that followed. Fixing the tool alone does not correct an invented order status.

### 3\. Fix the cause

Match the fix to the step that failed:

| First bad step                     | How to fix it                                                                                                             |
| ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------- |
| Model skipped a tool it needed     | Tighten the tool's description, or add an instruction such as "Always call `get_order_status` before describing an order" |
| Model passed wrong tool arguments  | Narrow the tool's input schema, or show the expected format in the description                                            |
| Tool returned an error or bad data | Fix the tool code or the upstream service, then check `error.type` for other failures                                     |
| Skill load or tool call timed out  | Raise the timeout, or split the work into smaller tool calls                                                              |

### 4\. Test the fix with an eval

Create a regression eval that checks the lookup and the expected status. Use a test agent with fictional order `4417` fixed at `status: "packing"` so changes to real order data cannot change the result.

Each app needs one eval config in its `evals/` directory:

```typescript
import { defineEvalConfig } from "eve/evals";

export default defineEvalConfig({});
```

The path under `evals/` gives this eval its ID: `orders/status-uses-tool`.

The assertion checks the tool result's `status` field. Adapt it to your output schema:

```typescript
import { defineEval } from "eve/evals";
import { includes } from "eve/evals/expect";

export default defineEval({
  description: "Look up order 4417 and report its packing status.",
  tags: ["regression"],
  async test(t) {
    const turn = await t.send("Where is order 4417?");
    t.succeeded();
    t.calledTool("get_order_status", {
      input: { orderId: "4417" },
      output: { status: "packing" },
      count: 1,
    });
    t.check(turn.reply, includes("4417"));
    t.check(turn.reply, includes(/packing|being packed/i));
  },
});
```

Run the eval locally with the test agent's model credentials and fixture data. `eve eval` starts its own agent server and makes real model requests.

For preview runs, [configure target authentication](https://eve.dev/docs/evals/targets#authentication) with a service credential and use the same test data. Protected deployments may need an automation bypass secret.

Replace the URL below with your preview deployment:

```bash
eve eval orders/status-uses-tool
eve eval orders --url https://order-agent-git-fix-status-my-team.vercel.app
```

The command exits with `0` when every gate passes and `1` when an eval fails. Configuration errors return `2`. Add `--strict` to also fail when a soft assertion scores below its threshold.

These checks catch a missing lookup, an unexpected fixture result, or a reply that omits the packing status. The wording check is limited; use additional cases or a [gated judge](https://eve.dev/docs/evals/assertions) to assess whether the full answer agrees with the tool result.

Eval reporters export results separately from traces and do not use `tracePolicy`. Use fictional test data or a destination approved to store the eval content.

Add a reporter to send results to the same service as your traces:

```typescript
import { defineEvalConfig } from "eve/evals";
import { braintrustReporter } from "eve/evals/reporters";

export default defineEvalConfig({
  reporters: [braintrustReporter({ projectName: "order-agent" })],
});
```

`braintrustReporter()` requires the `braintrust` package and `BRAINTRUST_API_KEY`. It exports the eval description, answer, errors, and assertion details. Use `--skip-report` to run evals without uploading results.

`datadogReporter({ projectName })` sends results to a Datadog LLM Observability Experiment. Install `dd-trace` and set `DD_API_KEY`, `DD_APP_KEY`, and `DD_SITE`.

Datadog reporter defaults include prompts, outputs, expected answers, and errors. Configure its [recording options](https://eve.dev/docs/evals/reporters#datadog) for the content your destination may store.

## Upgrade notes for eve 1.0

This guide is written for eve 1.0, which is currently a release candidate (`1.0.0-rc.1`). If you're upgrading a project from eve 0.83 or earlier, these related APIs were renamed. The [Upgrade to 1.0](https://eve.dev/docs/guides/upgrade) guide has the full list.

| Area                   | eve 0.83 and earlier                                           | eve 0.84 and later, including 1.0                                      |
| ---------------------- | -------------------------------------------------------------- | ---------------------------------------------------------------------- |
| Eval helpers           | `noFailedActions`, `calledSubagent`, `loadedSkill`, `parked()` | `noFailedCalls`, `calledAgent`, `calledSkill`, `awaitingInput()`       |
| Instrumentation events | `tool.call.*`, `turn.completed`, `model.call.completed`        | `call.requested`, `call.settled`, `turn.settled`, `model.call.settled` |
| CLI display flags      | `--tools`, `--reasoning`                                       | `--show-tools`, `--show-reasoning`                                     |

Code that matches events by name, such as a stream reader or dashboard query, can compile cleanly and still never fire after the rename. Search for the old names in your instrumentation, scripts, and dashboards when you upgrade.

## Next steps

- Connect eve to other services with the [Agent Stack guides](https://vercel.com/kb/agent-stack).
  
- Review [Agent Runs sampling and access](https://vercel.com/docs/agent-runs) for production tracing.
  
- Configure [OpenTelemetry](https://eve.dev/docs/observability/otel) resources, samplers, and Node.js instrumentation.
  
- Write judge-graded and multi-turn checks with [eve evals](https://eve.dev/docs/evals).
  
- Query runs from your terminal with [`vercel agent-runs`](https://vercel.com/docs/cli/agent-runs).

## Related Documentation

- [eve](https://eve.dev/docs)
- [Observability](/docs/observability)

## More eve guides

- [Build a software factory with eve](/kb/guide/eve-software-factory): Foreman, a software factory built on eve. It turns GitHub and Linear work items into reviewed draft pull requests through four agent stations, triaging issues and fixing its own red CI along the way.
- [How to add tools to your eve agent](/kb/guide/how-to-add-eve-tools): Add tools to an eve agent by creating a TypeScript file under agent/tools/ with defineTool, and gate sensitive ones on human approval with approval.
- [Running background workers on Vercel](/kb/guide/background-workers-on-vercel): Run background jobs on Vercel with waitUntil, Cron Jobs, Queues, Workflows, Daemons, or Sandbox. Compare the options and pick the right one for your workload.