Build an incident response sre agent with eve

Investigate production issues without leaving Slack. Mention @sre in an alerts channel, or let it respond to alerts automatically. It gathers relevant context from Datadog, Vercel, GitHub, and Slack, tests two to four likely root causes, and reports its conclusions in the thread. The response covers impact, timing, affected systems, and next steps, with each finding linked to its source.

The agent is built on eve, a filesystem-first framework for durable backend agents from Vercel. Slack, GitHub, Datadog, and Vercel authenticate through Vercel Connect, with read-only tokens on GitHub and Datadog, and Vercel Blob and AI Gateway authenticate with your project's OpenID Connect (OIDC) token. This guide covers deploying the template, connecting your tools through Vercel Connect, and extending the agent with team runbooks and webhook triggers.

Deploy the template now, or read on for a deeper look at how investigations work.

Copy link to headingQuick start with an AI coding agent

If you're working with an AI coding agent like Claude Code or Cursor, you can use this prompt to have it help you build the agent:

Copy link to headingVercel Plugin

The Vercel Plugin turns your AI coding agent (e.g., OpenAI Codex, Claude Code, or Cursor) into a Vercel expert. It adds skills, slash commands, and current knowledge of the tools this template uses, including Vercel Connect, Vercel Blob, and AI Gateway. The plugin is optional; it isn't required to use eve or to follow this guide.

npx plugins add vercel/vercel-plugin

Copy link to headingHow investigations work

The agent follows one investigation loop from an incoming signal to an evidence-backed Slack response. Here’s how it works:

1. Start a session. A Slack mention, direct message, watched channel message, or webhook starts an eve session.
2. Load the procedure. The agent loads deep-investigation for an alert or incident, or a focused skill for another task.
3. Gather context. It reads the supplied alert data and retrieves only the Vercel, Slack, Datadog, and GitHub context needed to explain the symptom.
4. Test hypotheses. A deep investigation generates two to four plausible root-cause hypotheses and tests each hypothesis against available evidence.
5. Record findings. The agent records each novel, decision-relevant finding with a source URL.
6. Report to Slack. The final response summarizes impact, timing, root-cause evidence or uncertainty, blast radius, and next actions.

The agent treats alert context as a starting point rather than a conclusion. It bounds Datadog queries with an explicit time window. When relevant, it explores Vercel Observability deployments, build logs, runtime errors, runtime logs, Web Analytics, and eve Agent Runs. It reads GitHub when a code or delivery change matters and reads Slack when the request points to context outside the active thread.

Evidence belongs to one turn. Recording the same finding and URL twice returns the existing evidence identifier instead of adding a duplicate. A later message starts with an empty evidence set, which prevents findings from one investigation from appearing in another.

Copy link to headingSetup and deployment

Copy link to headingWhat you need before deploying

You need the following accounts and permissions:

• A Vercel account
• A Slack workspace where you can install an app
• A GitHub account or organization with repositories the agent can inspect
• A Datadog organization and a personal access token or service access token

For local development, you also need Node.js 24 or newer, pnpm, and the Vercel CLI.

Copy link to headingDeploy to Vercel

The private Blob store supports channel watches and custom skills. Mentioned Slack investigations and webhook investigations can run without those optional features.

Copy link to headingConnecting Slack, GitHub, Vercel, and Datadog

Vercel Connect provides a guided setup for enabling Slack, GitHub, Vercel, and Datadog. Use read-only permissions for the tokens used for GitHub and Datadog.

For Datadog, select the “Access token” setup using a read-only service access token or personal access token. If your Datadog organization does not use the default US site, set the DD_SITE environment variable on the Vercel project. The default is datadoghq.com.

After deployment, invite the Slack app to your alerts channel and tell it to get started:

@sre watch this channel for incoming alerts and investigate them

After you approve the request, the bot evaluates each eligible top-level message in the channel. When a message warrants investigation, the bot starts a thread, tests likely causes, and posts its conclusions with supporting evidence.

Copy link to headingConnecting additional tools

Slack, GitHub, Vercel, and Datadog provide enough context for many investigations. You can configure additional observability and incident-management tools through Vercel Connect and eve.

To add an MCP server for an on-call provider like incident.io or PagerDuty, pass the provider's MCP server URL to vercel connect create, complete the prompted authentication flow, and attach the returned connector UID with vercel connect attach.

Then add a connection file under agent/connections/ that uses defineMcpClientConnection and authenticates with connect() from @vercel/connect/eve. Automated webhook and channel-watch sessions need a connection method that supports application credentials; per-user OAuth connections are available only in sessions started by a Slack user.

See eve’s integrations for a full list of natively supported connections.

Copy link to headingRun the agent locally

First, clone the repository created under your GitHub account and install its dependencies:

git clone <your_repository_url_here>
cd sre
pnpm install

Confirm that the Vercel CLI is authenticated, then link the repository and pull the project environment variables:

vercel whoami
vercel link
vercel env pull

vercel env pull writes a short-lived OpenID Connect (OIDC) token to .env.local. Re-run the command when the token expires.

Start the eve terminal user interface (TUI):

pnpm dev

Run /model once if the TUI asks you to link a model provider. Slack events continue to reach the deployed project through Vercel Connect, so test Slack mentions against a preview or production deployment.

Before deploying changes, run the validation and test suites:

pnpm validate
pnpm test

pnpm validate runs the Ultracite lint and format check, TypeScript type checking, and eve info. Run pnpm fix to apply supported fixes.

Deploy changes with eve:

pnpm exec eve deploy

eve deploy wraps the production deployment. Do not replace it with a raw vercel deploy --prod command.

Copy link to headingHow sre works

Copy link to headingHypothesis-based investigations

Mention @sre in a channel that contains the app, or send it a direct message. The current message and visible conversation become the initial investigation context. After you invoke the bot, it:

1. Hydrates context: usually, the context in the initial message is not enough. The bot will use its connected tools to load any context from links or references in the initial message to determine what information might be useful before starting an investigation.
2. Forms hypotheses: after enough context has been gathered to understand the problem, the agent forms hypotheses for the alert/incident being investigated.
3. Investigates hypotheses and records evidence: the agent will investigate each hypothesis independently and record specific evidence with links to support its conclusions.
4. Draws conclusions: after investigating all hypotheses and reviewing the evidence, the agent will draw an overall conclusion regarding the alert/incident at hand and post the conclusion in Slack.

Copy link to headingWatching channels

In addition to mentions and direct messages, sre supports listening to specific Slack channels and automatically investigating messages posted to those channels, such as alerts. Ask the bot to watch a channel to get started:

@sre watch this channel for incoming messages that need investigation.

The bot asks for approval before storing the watch in a private Vercel Blob. Each eligible top-level message starts a session in its own thread. The agent investigates the message when it warrants investigation or briefly explains why it does not.

Watched messages can include app-authored bot_message posts, such as alerts delivered by another Slack app. The agent ignores thread replies, its own messages, empty messages, and unsupported message subtypes. The bot must belong to the watched channel.

Copy link to headingCreate runbooks and handoff summaries

sre can create and execute runbooks for performing specific checks for alerts or incidents. If you want to guide the agent on how to respond to specific types of alerts or incidents, ask it to write a runbook:

@sre let's create a runbook. When you are investigating "Elevated 5xx error rate on <service>" alerts, make sure to do the following:
...

Additionally, when connected to a third-party on-call provider, such as PagerDuty or incident.io, you can use sre’s built-in skills to prepare on-call handoff documents at the end of a shift, or ask the agent to suggest follow-ups after an incident is resolved.

Copy link to headingCode walkthrough

Copy link to headingChannels and session authentication

agent/channels/slack.ts serves Slack events at /eve/v1/slack. Mentions and direct messages use Slack user authentication. An authorless app message in a watched channel receives a service principal based on its workspace, channel, and timestamp.

agent/channels/webhook.ts exposes POST /v1/investigate. The route verifies WEBHOOK_SECRET, validates the request body, and uses waitUntil to start the Slack investigation after returning 202:

args.waitUntil(
  startInvestigation(parsed.data, args.to, slack).catch((error) => {
    console.error("[sre/investigation] webhook session start failed", {
      error,
    });
  })
);

return new Response(null, { status: 202 });

The webhook session creates a service principal with a random identifier, posts an initial investigation card to slackChannel, and sends the payload to the agent as seed context.

Copy link to headingInvestigation instructions and skills

agent/instructions/instructions.md requires source-backed conclusions, absolute timestamps, bounded Datadog queries, and focused tool use.

The built-in skills cover:

Copy link to headingCustom skills and runbooks

In addition to static skills, the agent can create custom skills directly in Slack. Global skills act as runbooks and can apply to any session of the agent. A personal skill applies to one Slack user and overrides a global skill with the same name.

Copy link to headingRead-only access by default

The Datadog connection authenticates through Vercel Connect as the application. Its allowlist limits the model to 19 operations across metrics, logs, traces, monitors, dashboards, notebooks, hosts, services, Real User Monitoring (RUM), and Continuous Integration (CI) pipeline data.

export default defineMcpClientConnection({
  url: `https://${DATADOG_MCP_HOST}/v1/mcp?toolsets=core,software-delivery`,
  auth: connect({
    connector: DATADOG_CONNECTOR,
    principalType: "app",
    displayName: "Datadog",
  }),
  tools: {
    allow: [...DATADOG_MCP_TOOLS],
  },
});

The GitHub extension uses the read-only repo-explorer preset and excludes gist operations. It can inspect commits, pull requests, files, blame, and GitHub Actions without changing a repository.

Copy link to headingEvidence tools

evidence_record stores one normalized finding, source type, source label, and HTTP URL. It assigns a stable identifier such as evidence-1 and deduplicates identical finding and URL pairs.

const existing = current.items.find(
  (candidate) =>
    candidate.sourceUrl === normalized.sourceUrl &&
    candidate.finding === normalized.finding
);

if (existing) {
  return { item: existing, recorded: false, state: current };
}

The agent can remove evidence that becomes erroneous, superseded, redundant, or irrelevant. It must not remove a finding because it invalidates a hypothesis.

The Slack completion handler retrieves the turn's evidence and appends it to the answer as a Block Kit footer:

const evidence = getInvestigationEvidence(data.turnId);

if (evidence.length > 0) {
  await ctx.thread.post(
    buildInvestigationResultMessage(data.message, evidence)
  );
}

Copy link to headingCustomize the agent

eve discovers capabilities from the agent/ filesystem. Add or remove a supported file to change the capabilities available to the model.

Copy link to headingAdvanced integrations

Copy link to headingTriggering investigations with a webhook

You can also start investigations with webhooks. This is useful when integrating with third-party systems, like PagerDuty and incident.io.

Webhooks are verified using a static webhook secret for security. Generate a secret with at least 32 bytes of entropy:

node -e "console.log(require('crypto').randomBytes(32).toString('hex'))"

Add the generated value to the production environment at the CLI prompt, then redeploy:

vercel env add WEBHOOK_SECRET production
pnpm exec eve deploy

Do not place the secret in source code, logs, request payloads, or chat. Send it in either the x-sre-webhook-secret header or an Authorization: Bearer header. The direct header takes priority when both are present.

Export the same secret in the caller's environment, then send the request:

curl --request POST "https://<your_deployment_hostname_here>/v1/investigate" \
  --header "content-type: application/json" \
  --header "authorization: Bearer $WEBHOOK_SECRET" \
  --data '{
    "title": "Checkout error rate increased",
    "description": "HTTP 500 responses increased in production.",
    "slackChannel": "C0123456789",
    "metadata": {
      "service": "checkout-api",
      "environment": "production",
      "monitorId": "123456",
      "incidentId": "123456"
    }
  }'

The endpoint accepts the following fields:

A 202 Accepted response confirms authentication, body validation, and background scheduling. Slack session creation can still fail afterward. The endpoint returns 400 for invalid JSON or an invalid body, and 401 for a missing or incorrect secret.

If Deployment Protection applies to the endpoint, configure Protection Bypass for Automation. Send its secret in the x-vercel-protection-bypass header in addition to WEBHOOK_SECRET.

Copy link to headingAutomatically investigate incidents created in incident.io

Create an incident.io workflow that invites sre to a new incident channel and calls the investigation webhook. This starts the investigation in the channel where responders are already working.

Complete the setup in the section above on configuring webhooks before configuring incident.io. You need the deployed investigation URL and the value of WEBHOOK_SECRET.

Copy link to headingStore the webhook secrets

1. Open Settings > Secrets in incident.io.
2. Create a secret containing the same WEBHOOK_SECRET value used by the Vercel project.
3. (Optional) Create another secret containing the Protection Bypass for Automation value if your project uses Deployment Protection.

Store these values as incident.io secrets instead of placing them directly in the workflow definition for security.

Copy link to headingCreate the workflow

1. Open Workflows and create a workflow that runs when an incident is created or changed.
2. Add conditions that require the incident status to be Active or Triage.
3. Add a condition that requires the incident to have been reported within the last minute.
4. Add an Invite a Slack bot user step and select sre.
5. Add a Send a webhook step after the invitation step.

The status and time conditions reduce the chance that a later incident update starts another investigation. Adjust them if your incident lifecycle uses different statuses, but keep a condition that distinguishes a new incident from a later update.

Copy link to headingConfigure the webhook step

Set the webhook method to POST and use the following endpoint:

https://<your_deployment_hostname_here>/v1/investigate

Add these headers with values from the incident.io secrets:

Use incident.io's Insert variable control to map incident data into the request body. Select the incident's Slack channel ID for slackChannel, not its channel name. Keep the bot invitation before the webhook step so sre belongs to the channel when the investigation starts.

{
  "title": "incident_name_here",
  "description": "incident_summary_here",
  "slackChannel": "incident_slack_channel_id_here",
  "metadata": {
    "incidentId": "incident_id_here",
    "incidentUrl": "incident_url_here",
    "reportedAt": "reported_at_here",
    "severity": "incident_severity_here"
  }
}

Replace each placeholder through Insert variable. Keep only metadata that helps the agent identify the affected system, time window, and source incident.

Copy link to headingTest the workflow

1. Ensure that the workflow you created has been saved and published.
2. Create an incident.io test incident.
3. Confirm that incident.io invites sre to the incident channel.
4. Confirm that the webhook step receives 202 Accepted.
5. Confirm that the incident channel receives an investigation card and thread.

A 202 Accepted response confirms that Vercel authenticated and scheduled the request. If no thread appears, confirm that the workflow supplied a Slack channel ID and inspect the Vercel runtime logs for [sre/investigation] webhook session start failed.

Copy link to headingTroubleshooting

Copy link to headingSlack mentions do not get a response

Symptom: You mention @sre, but the app does not reply.

Cause: The app is not in the channel, the deployment failed, or the Slack trigger does not point to /eve/v1/slack.

Fix: Invite the app to the channel, confirm that the deployment succeeded, and verify the connector trigger path.

Copy link to headingA Datadog investigation cannot read data

Symptom: The agent cannot find monitors, metrics, logs, or traces that exist in Datadog.

Cause: The connector UID, access-token permissions, or DD_SITE value does not match the Datadog organization.

Fix: Confirm DATADOG_CONNECTOR, verify that the token can read the required data, and set DD_SITE for non-default sites.

Copy link to headingGitHub results are missing

Symptom: The agent cannot find a repository, pull request, commit, or workflow run.

Cause: The GitHub connector installation does not include that repository.

Fix: Review the repositories available to the connector installation, then retry the investigation.

Copy link to headingA webhook returns 400 or 401

Symptom: POST /v1/investigate returns 400 Bad Request or 401 Unauthorized.

Cause: A 400 response means the JSON or request schema is invalid. A 401 response means the secret is missing or does not match WEBHOOK_SECRET.

Fix: Send valid JSON with title and slackChannel, then confirm that the caller and production environment use the same secret.

Copy link to headingA webhook returns 202, but no thread appears

Symptom: The caller receives 202 Accepted, but Slack does not show an investigation.

Cause: The response confirms scheduling, not successful Slack delivery. The channel ID may be invalid, the bot may not belong to the channel, or session creation may have failed.

Fix: Confirm slackChannel, invite the app to the channel, and inspect runtime logs for [sre/investigation] webhook session start failed or [sre/slack] failed to post investigation answer.

Copy link to headingA watched channel does not start investigations

Symptom: A new top-level message does not start a thread.

Cause: The channel is not watched, the bot is not a member, the message is a reply, or Blob membership lookup failed.

Fix: Mention @sre and ask it to watch the channel again, approve the action, and confirm that the private Blob store is connected.

Copy link to headingAn investigation has no evidence links

Symptom: The final response does not contain an evidence footer.

Cause: The agent did not record a decision-relevant finding with an HTTP source URL during that turn.

Fix: Provide a Datadog monitor, dashboard, trace, log, Slack, or GitHub link that grounds the investigation, then ask the agent to investigate again.

Copy link to headingA custom skill does not load

Symptom: A saved runbook is not available on the next matching request.

Cause: Blob storage is unavailable, the skill description does not match the request, or the skill was created during the current turn and is not loadable until the next turn.

Fix: Confirm that the private Blob store is connected, check for [sre/custom-skills] custom skills unavailable in runtime logs, and start a new request that matches the skill description.

Copy link to headingRelated resources

• Explore the eve SRE template source to review the agent’s code.
• Read the eve documentation to learn how eve agents work.
• Learn how to use Vercel Connect and the Vercel Connect CLI to configure the agent’s connections.
• Review Vercel Blob documentation to understand how channel watches and custom skills are stored.
• See the Vercel CLI documentation for project and deployment commands.
• Explore the GitHub tools eve extension for its read-only repository tools.
• Review the Datadog MCP server for supported observability tools.
• Follow the incident.io webhook guide to trigger incident investigations.