--- title: Build an incident response sre agent with eve description: sre is an incident response agent for Slack. It investigates production issues using a hypothesis-driven approach and returns verifiable evidence from connected sources. url: /kb/guide/eve-incident-sre-agent canonical_url: "https://vercel.com/kb/guide/eve-incident-sre-agent" published: 2026-08-25 last_updated: 2026-08-26 authors: Zach Cowan related: - /docs/connect - /docs/vercel-blob - /docs/agent-resources/vercel-plugin - /docs/observability - /docs/cli - /docs/deployment-protection/methods-to-bypass-deployment-protection/protection-bypass-automation - /docs/cli/connect install_vercel_plugin: npx plugins add vercel/vercel-plugin --- Investigate production issues without leaving Slack. Mention `@sre` in an alerts channel, or let it respond to alerts automatically. It gathers relevant context from Datadog, Vercel, GitHub, and Slack, tests two to four likely root causes, and reports its conclusions in the thread. The response covers impact, timing, affected systems, and next steps, with each finding linked to its source. The agent is built on [eve](https://eve.dev/), a filesystem-first framework for durable backend agents from Vercel. Slack, GitHub, Datadog, and Vercel authenticate through [Vercel Connect](https://vercel.com/docs/connect), with read-only tokens on GitHub and Datadog, and [Vercel Blob](https://vercel.com/docs/vercel-blob) and [AI Gateway](https://vercel.com/ai-gateway) authenticate with your project's OpenID Connect (OIDC) token. This guide covers deploying the template, connecting your tools through Vercel Connect, and extending the agent with team runbooks and webhook triggers. Deploy the template now, or read on for a deeper look at how investigations work. ## Quick start with an AI coding agent If you're working with an AI coding agent like Claude Code or Cursor, you can use this prompt to have it help you build the agent: ### Agent prompt ```txt I want to deploy the eve sre template as an incident investigation agent. Clone https://github.com/vercel-labs/eve-sre-agent-template, read docs/setup-for-agents.md, and follow its instructions. Configure read-only GitHub and Datadog access, deploy with eve, and stop after verifying that a Slack mention receives a response. ``` ### Vercel Plugin The [Vercel Plugin](https://vercel.com/docs/agent-resources/vercel-plugin) turns your AI coding agent (e.g., OpenAI Codex, Claude Code, or Cursor) into a Vercel expert. It adds skills, slash commands, and current knowledge of the tools this template uses, including Vercel Connect, Vercel Blob, and AI Gateway. The plugin is optional; it isn't required to use eve or to follow this guide. ```bash npx plugins add vercel/vercel-plugin ``` ## How investigations work The agent follows one investigation loop from an incoming signal to an evidence-backed Slack response. Here’s how it works: 1. Start a session. A Slack mention, direct message, watched channel message, or webhook starts an eve session. 2. Load the procedure. The agent loads `deep-investigation` for an alert or incident, or a focused skill for another task. 3. Gather context. It reads the supplied alert data and retrieves only the Vercel, Slack, Datadog, and GitHub context needed to explain the symptom. 4. Test hypotheses. A deep investigation generates two to four plausible root-cause hypotheses and tests each hypothesis against available evidence. 5. Record findings. The agent records each novel, decision-relevant finding with a source URL. 6. Report to Slack. The final response summarizes impact, timing, root-cause evidence or uncertainty, blast radius, and next actions. The agent treats alert context as a starting point rather than a conclusion. It bounds Datadog queries with an explicit time window. When relevant, it explores [Vercel Observability](https://vercel.com/docs/observability) deployments, build logs, runtime errors, runtime logs, Web Analytics, and eve Agent Runs. It reads GitHub when a code or delivery change matters and reads Slack when the request points to context outside the active thread. Evidence belongs to one turn. Recording the same finding and URL twice returns the existing evidence identifier instead of adding a duplicate. A later message starts with an empty evidence set, which prevents findings from one investigation from appearing in another. ## Setup and deployment ### What you need before deploying You need the following accounts and permissions: - A [Vercel account](https://vercel.com/signup) - A Slack workspace where you can install an app - A GitHub account or organization with repositories the agent can inspect - A Datadog organization and a personal access token or service access token For local development, you also need Node.js 24 or newer, pnpm, and the [Vercel CLI](https://vercel.com/docs/cli). ### Deploy to Vercel | Provisioned | Sets | | ------------------------------------------------- | ---------------------- | | Slack connector with trigger path `/eve/v1/slack` | `SLACK_CONNECTOR` | | GitHub connector | `GITHUB_CONNECTOR` | | Datadog connector | `DATADOG_CONNECTOR` | | Vercel connector | `SRE_VERCEL_CONNECTOR` | | Private Vercel Blob store | Blob credentials | The private Blob store supports channel watches and custom skills. Mentioned Slack investigations and webhook investigations can run without those optional features. ### Connecting Slack, GitHub, Vercel, and Datadog Vercel Connect provides a guided setup for enabling Slack, GitHub, Vercel, and Datadog. Use read-only permissions for the tokens used for GitHub and Datadog. For Datadog, select the “Access token” setup using a read-only [service access token](https://docs.datadoghq.com/account_management/service-access-tokens/) or [personal access token](https://docs.datadoghq.com/account_management/personal-access-tokens/). If your Datadog organization does not use the default US site, set the `DD_SITE` environment variable on the Vercel project. The default is `datadoghq.com`. After deployment, invite the Slack app to your alerts channel and tell it to get started: ```plaintext @sre watch this channel for incoming alerts and investigate them ``` After you approve the request, the bot evaluates each eligible top-level message in the channel. When a message warrants investigation, the bot starts a thread, tests likely causes, and posts its conclusions with supporting evidence. ### Connecting additional tools Slack, GitHub, Vercel, and Datadog provide enough context for many investigations. You can configure additional observability and incident-management tools through [Vercel Connect](https://vercel.com/docs/connect) and [eve](https://eve.dev/). To add an MCP server for an on-call provider like incident.io or PagerDuty, pass the provider's MCP server URL to `vercel connect create`, complete the prompted authentication flow, and attach the returned connector UID with `vercel connect attach`. Then add a connection file under `agent/connections/` that uses `defineMcpClientConnection` and authenticates with `connect()` from `@vercel/connect/eve`. Automated webhook and channel-watch sessions need a connection method that supports application credentials; per-user OAuth connections are available only in sessions started by a Slack user. See eve’s [integrations](https://eve.dev/integrations?filter=connection) for a full list of natively supported connections. ### Run the agent locally First, clone the repository created under your GitHub account and install its dependencies: ```bash git clone cd sre pnpm install ``` Confirm that the Vercel CLI is authenticated, then link the repository and pull the project environment variables: ```bash vercel whoami vercel link vercel env pull ``` `vercel env pull` writes a short-lived OpenID Connect (OIDC) token to `.env.local`. Re-run the command when the token expires. Start the eve terminal user interface (TUI): ```bash pnpm dev ``` Run `/model` once if the TUI asks you to link a model provider. Slack events continue to reach the deployed project through Vercel Connect, so test Slack mentions against a preview or production deployment. Before deploying changes, run the validation and test suites: ```bash pnpm validate pnpm test ``` `pnpm validate` runs the Ultracite lint and format check, TypeScript type checking, and `eve info`. Run `pnpm fix` to apply supported fixes. Deploy changes with eve: ```bash pnpm exec eve deploy ``` `eve deploy` wraps the production deployment. Do not replace it with a raw `vercel deploy --prod` command. ## How sre works ### Hypothesis-based investigations Mention `@sre` in a channel that contains the app, or send it a direct message. The current message and visible conversation become the initial investigation context. After you invoke the bot, it: 1. **Hydrates context:** usually, the context in the initial message is not enough. The bot will use its connected tools to load any context from links or references in the initial message to determine what information might be useful before starting an investigation. 2. **Forms hypotheses**: after enough context has been gathered to understand the problem, the agent forms hypotheses for the alert/incident being investigated. 3. **Investigates hypotheses and records evidence**: the agent will investigate each hypothesis independently and record specific evidence with links to support its conclusions. 4. **Draws conclusions**: after investigating all hypotheses and reviewing the evidence, the agent will draw an overall conclusion regarding the alert/incident at hand and post the conclusion in Slack. ### Watching channels In addition to mentions and direct messages, sre supports listening to specific Slack channels and automatically investigating messages posted to those channels, such as alerts. Ask the bot to watch a channel to get started: ```plaintext @sre watch this channel for incoming messages that need investigation. ``` The bot asks for approval before storing the watch in a private Vercel Blob. Each eligible top-level message starts a session in its own thread. The agent investigates the message when it warrants investigation or briefly explains why it does not. Watched messages can include app-authored `bot_message` posts, such as alerts delivered by another Slack app. The agent ignores thread replies, its own messages, empty messages, and unsupported message subtypes. The bot must belong to the watched channel. ### Create runbooks and handoff summaries sre can create and execute runbooks for performing specific checks for alerts or incidents. If you want to guide the agent on how to respond to specific types of alerts or incidents, ask it to write a runbook: ```plaintext @sre let's create a runbook. When you are investigating "Elevated 5xx error rate on " alerts, make sure to do the following: ... ``` Additionally, when connected to a third-party on-call provider, such as [PagerDuty](https://www.pagerduty.com/) or [incident.io](https://incident.io/), you can use sre’s built-in skills to prepare on-call handoff documents at the end of a shift, or ask the agent to suggest follow-ups after an incident is resolved. ## Code walkthrough ### Channels and session authentication `agent/channels/slack.ts` serves Slack events at `/eve/v1/slack`. Mentions and direct messages use Slack user authentication. An authorless app message in a watched channel receives a service principal based on its workspace, channel, and timestamp. `agent/channels/webhook.ts` exposes `POST /v1/investigate`. The route verifies `WEBHOOK_SECRET`, validates the request body, and uses `waitUntil` to start the Slack investigation after returning `202`: ```typescript args.waitUntil( startInvestigation(parsed.data, args.to, slack).catch((error) => { console.error("[sre/investigation] webhook session start failed", { error, }); }) ); return new Response(null, { status: 202 }); ``` The webhook session creates a service principal with a random identifier, posts an initial investigation card to `slackChannel`, and sends the payload to the agent as seed context. ### Investigation instructions and skills `agent/instructions/instructions.md` requires source-backed conclusions, absolute timestamps, bounded Datadog queries, and focused tool use. The built-in skills cover: | Skill | Purpose | | ----------------------- | --------------------------------------------------------------------- | | `deep-investigation` | Generate and test hypotheses for an alert or incident | | `hypothesis-validation` | Validate one supplied hypothesis | | `suggest-follow-ups` | Recommend ranked follow-up work from established incident context | | `on-call-handoff` | Summarize incidents, changes, and open actions for the next responder | ### Custom skills and runbooks In addition to static skills, the agent can create custom skills directly in Slack. Global skills act as runbooks and can apply to any session of the agent. A personal skill applies to one Slack user and overrides a global skill with the same name. ### Read-only access by default The Datadog connection authenticates through Vercel Connect as the application. Its allowlist limits the model to 19 operations across metrics, logs, traces, monitors, dashboards, notebooks, hosts, services, Real User Monitoring (RUM), and Continuous Integration (CI) pipeline data. ```typescript export default defineMcpClientConnection({ url: `https://${DATADOG_MCP_HOST}/v1/mcp?toolsets=core,software-delivery`, auth: connect({ connector: DATADOG_CONNECTOR, principalType: "app", displayName: "Datadog", }), tools: { allow: [...DATADOG_MCP_TOOLS], }, }); ``` The GitHub extension uses the read-only `repo-explorer` preset and excludes gist operations. It can inspect commits, pull requests, files, blame, and GitHub Actions without changing a repository. ### Evidence tools `evidence_record` stores one normalized finding, source type, source label, and HTTP URL. It assigns a stable identifier such as `evidence-1` and deduplicates identical finding and URL pairs. ```typescript const existing = current.items.find( (candidate) => candidate.sourceUrl === normalized.sourceUrl && candidate.finding === normalized.finding ); if (existing) { return { item: existing, recorded: false, state: current }; } ``` The agent can remove evidence that becomes erroneous, superseded, redundant, or irrelevant. It must not remove a finding because it invalidates a hypothesis. The Slack completion handler retrieves the turn's evidence and appends it to the answer as a Block Kit footer: ```typescript const evidence = getInvestigationEvidence(data.turnId); if (evidence.length > 0) { await ctx.thread.post( buildInvestigationResultMessage(data.message, evidence) ); } ``` ## Customize the agent eve discovers capabilities from the `agent/` filesystem. Add or remove a supported file to change the capabilities available to the model. | Goal | Edit | | -------------------------------------------------------- | ----------------------------------------------- | | Change investigation and reporting behavior | `agent/instructions/instructions.md` | | Change an investigation or handoff procedure | `agent/skills` | | Add a team runbook at runtime | Ask the agent to create a runbook | | Add additional observability tools or incident platforms | Add a new connection under `agent/connections/` | | Change allowlisted Datadog tools | `agent/connections/datadog.ts` | | Change allowlisted GitHub tools | `agent/extensions/github.ts` | | Change the model or reasoning effort | `agent/agent.ts` | ## Advanced integrations ### Triggering investigations with a webhook You can also start investigations with webhooks. This is useful when integrating with third-party systems, like [PagerDuty](https://www.pagerduty.com/) and [incident.io](https://incident.io/). Webhooks are verified using a static webhook secret for security. Generate a secret with at least 32 bytes of entropy: ```bash node -e "console.log(require('crypto').randomBytes(32).toString('hex'))" ``` Add the generated value to the production environment at the CLI prompt, then redeploy: ```bash vercel env add WEBHOOK_SECRET production pnpm exec eve deploy ``` Do not place the secret in source code, logs, request payloads, or chat. Send it in either the `x-sre-webhook-secret` header or an `Authorization: Bearer` header. The direct header takes priority when both are present. Export the same secret in the caller's environment, then send the request: ```bash curl --request POST "https:///v1/investigate" \ --header "content-type: application/json" \ --header "authorization: Bearer $WEBHOOK_SECRET" \ --data '{ "title": "Checkout error rate increased", "description": "HTTP 500 responses increased in production.", "slackChannel": "C0123456789", "metadata": { "service": "checkout-api", "environment": "production", "monitorId": "123456", "incidentId": "123456" } }' ``` The endpoint accepts the following fields: | Field | Required | Purpose | | ------------ | -------- | ----------------------------------------------------------------------------------------------- | | title | Yes | Investigation label shown in Slack | | slackChannel | Yes | Target Slack channel ID | | description | No | Summary of the issue to be investigated | | metadata | No | Additional metadata from external systems, like alerts, incident information, or other metadata | A `202 Accepted` response confirms authentication, body validation, and background scheduling. Slack session creation can still fail afterward. The endpoint returns `400` for invalid JSON or an invalid body, and `401` for a missing or incorrect secret. If Deployment Protection applies to the endpoint, configure [Protection Bypass for Automation](https://vercel.com/docs/deployment-protection/methods-to-bypass-deployment-protection/protection-bypass-automation). Send its secret in the `x-vercel-protection-bypass` header in addition to `WEBHOOK_SECRET`. ### Automatically investigate incidents created in incident.io Create an incident.io workflow that invites `sre` to a new incident channel and calls the investigation webhook. This starts the investigation in the channel where responders are already working. Complete the setup in the section above on configuring webhooks before configuring incident.io. You need the deployed investigation URL and the value of `WEBHOOK_SECRET`. #### Store the webhook secrets 1. Open [**Settings > Secrets**](https://app.incident.io/settings/secrets) in incident.io. 2. Create a secret containing the same `WEBHOOK_SECRET` value used by the Vercel project. 3. (Optional) Create another secret containing the Protection Bypass for Automation value if your project uses Deployment Protection. Store these values as incident.io secrets instead of placing them directly in the workflow definition for security. #### Create the workflow 1. Open [**Workflows**](https://app.incident.io/settings/workflows/definitions) and create a workflow that runs when an incident is created or changed. 2. Add conditions that require the incident status to be **Active** or **Triage**. 3. Add a condition that requires the incident to have been reported within the last minute. 4. Add an **Invite a Slack bot user** step and select `sre`. 5. Add a **Send a webhook** step after the invitation step. The status and time conditions reduce the chance that a later incident update starts another investigation. Adjust them if your incident lifecycle uses different statuses, but keep a condition that distinguishes a new incident from a later update. #### Configure the webhook step Set the webhook method to `POST` and use the following endpoint: ```plaintext https:///v1/investigate ``` Add these headers with values from the incident.io secrets: | Header | Required | Value | | ---------------------------- | ------------------------------------- | -------------------------------------- | | `Content-Type` | Yes | `application/json` | | `x-sre-webhook-secret` | Yes | The secret containing `WEBHOOK_SECRET` | | `x-vercel-protection-bypass` | When Deployment Protection is enabled | The automation bypass secret | Use incident.io's **Insert variable** control to map incident data into the request body. Select the incident's Slack channel ID for `slackChannel`, not its channel name. Keep the bot invitation before the webhook step so `sre` belongs to the channel when the investigation starts. ```json { "title": "incident_name_here", "description": "incident_summary_here", "slackChannel": "incident_slack_channel_id_here", "metadata": { "incidentId": "incident_id_here", "incidentUrl": "incident_url_here", "reportedAt": "reported_at_here", "severity": "incident_severity_here" } } ``` Replace each placeholder through **Insert variable**. Keep only metadata that helps the agent identify the affected system, time window, and source incident. #### Test the workflow 1. Ensure that the workflow you created has been saved and published. 2. Create an incident.io test incident. 3. Confirm that incident.io invites `sre` to the incident channel. 4. Confirm that the webhook step receives `202 Accepted`. 5. Confirm that the incident channel receives an investigation card and thread. A `202 Accepted` response confirms that Vercel authenticated and scheduled the request. If no thread appears, confirm that the workflow supplied a Slack channel ID and inspect the Vercel runtime logs for `[sre/investigation] webhook session start failed`. ## Troubleshooting ### Slack mentions do not get a response **Symptom:** You mention `@sre`, but the app does not reply. **Cause:** The app is not in the channel, the deployment failed, or the Slack trigger does not point to `/eve/v1/slack`. **Fix:** Invite the app to the channel, confirm that the deployment succeeded, and verify the connector trigger path. ### A Datadog investigation cannot read data **Symptom:** The agent cannot find monitors, metrics, logs, or traces that exist in Datadog. **Cause:** The connector UID, access-token permissions, or `DD_SITE` value does not match the Datadog organization. **Fix:** Confirm `DATADOG_CONNECTOR`, verify that the token can read the required data, and set `DD_SITE` for non-default sites. ### GitHub results are missing **Symptom:** The agent cannot find a repository, pull request, commit, or workflow run. **Cause:** The GitHub connector installation does not include that repository. **Fix:** Review the repositories available to the connector installation, then retry the investigation. ### A webhook returns `400` or `401` **Symptom:** `POST /v1/investigate` returns `400 Bad Request` or `401 Unauthorized`. **Cause:** A `400` response means the JSON or request schema is invalid. A `401` response means the secret is missing or does not match `WEBHOOK_SECRET`. **Fix:** Send valid JSON with `title` and `slackChannel`, then confirm that the caller and production environment use the same secret. ### A webhook returns `202`, but no thread appears **Symptom:** The caller receives `202 Accepted`, but Slack does not show an investigation. **Cause:** The response confirms scheduling, not successful Slack delivery. The channel ID may be invalid, the bot may not belong to the channel, or session creation may have failed. **Fix:** Confirm `slackChannel`, invite the app to the channel, and inspect runtime logs for `[sre/investigation] webhook session start failed` or `[sre/slack] failed to post investigation answer`. ### A watched channel does not start investigations **Symptom:** A new top-level message does not start a thread. **Cause:** The channel is not watched, the bot is not a member, the message is a reply, or Blob membership lookup failed. **Fix:** Mention `@sre` and ask it to watch the channel again, approve the action, and confirm that the private Blob store is connected. ### An investigation has no evidence links **Symptom:** The final response does not contain an evidence footer. **Cause:** The agent did not record a decision-relevant finding with an HTTP source URL during that turn. **Fix:** Provide a Datadog monitor, dashboard, trace, log, Slack, or GitHub link that grounds the investigation, then ask the agent to investigate again. ### A custom skill does not load **Symptom:** A saved runbook is not available on the next matching request. **Cause:** Blob storage is unavailable, the skill description does not match the request, or the skill was created during the current turn and is not loadable until the next turn. **Fix:** Confirm that the private Blob store is connected, check for `[sre/custom-skills] custom skills unavailable` in runtime logs, and start a new request that matches the skill description. ## Related resources - Explore the [eve SRE template source](https://github.com/vercel-labs/eve-sre-agent-template) to review the agent’s code. - Read the [eve documentation](https://eve.dev/docs/introduction) to learn how eve agents work. - Learn how to use [Vercel Connect](https://vercel.com/docs/connect) and the [Vercel Connect CLI](https://vercel.com/docs/cli/connect) to configure the agent’s connections. - Review [Vercel Blob documentation](https://vercel.com/docs/vercel-blob) to understand how channel watches and custom skills are stored. - See the [Vercel CLI documentation](https://vercel.com/docs/cli) for project and deployment commands. - Explore the [GitHub tools eve extension](https://github-tools.com/frameworks/eve-extension) for its read-only repository tools. - Review the [Datadog MCP server](https://docs.datadoghq.com/bits_ai/mcp_server/) for supported observability tools. - Follow the [incident.io webhook guide](https://docs.incident.io/workflows/webhooks) to trigger incident investigations.