--- title: "eve Incident Response Agent" description: "sre investigates production issues using a hypothesis-driven approach and outputs verifiable evidence from connected sources." url: "https://vercel.com/templates/eve/eve-incident-response-agent" links: repository: "https://github.com/vercel-labs/eve-sre-agent-template" demo: "https://vercel.com/kb/guide/eve-incident-sre-agent" deploy: "https://vercel.com/new/clone?demo-title=eve+Incident+Response+Agent&demo-description=sre+investigates+production+issues+using+a+hypothesis-driven+approach+and+outputs+verifiable+evidence+from+connected+sources.&demo-url=https%3A%2F%2Fvercel.com%2Fkb%2Fguide%2Feve-incident-sre-agent&project-name=sre&repository-name=sre&repository-url=https%3A%2F%2Fgithub.com%2Fvercel-labs%2Feve-sre-agent-template%2Ftree%2Fmain&from=templates&products=%5B%7B%22type%22%3A%22blob%22%2C%22access%22%3A%22private%22%7D%5D&connect=%5B%7B%22type%22%3A%22slack%22%2C%22env%22%3A%22SLACK_CONNECTOR%22%2C%22triggers%22%3Atrue%2C%22triggerPath%22%3A%22%2Feve%2Fv1%2Fslack%22%7D%2C%7B%22type%22%3A%22github%22%2C%22env%22%3A%22GITHUB_CONNECTOR%22%7D%2C%7B%22type%22%3A%22datadog%22%2C%22env%22%3A%22DATADOG_CONNECTOR%22%7D%2C%7B%22type%22%3A%22vercel%22%2C%22env%22%3A%22SRE_VERCEL_CONNECTOR%22%7D%5D&demo-image=%2F%2Fimages.ctfassets.net%2Fe5382hct74si%2F2palFAjg8CYiPZQAHmfto8%2F6aba3a8137b5673c08620f9311526459%2Fimage.png" --- # eve Incident Response Agent sre investigates production issues using a hypothesis-driven approach and outputs verifiable evidence from connected sources. sre is an [eve](https://eve.dev) incident response agent for Slack. It brings together the observability data you need to debug an alert or incident across Datadog, GitHub, Vercel, and other tools. Investigations are read-only by default. Mention `@sre`, watch a channel for alerts, or invoke it from an external system with a webhook. The agent checks hypotheses against live signals and records each finding with a source link. Replies start with the answer, then the supporting evidence. ## How it works An investigation starts from one of three places. - A Slack mention or direct message. - A new top-level message in a watched Slack channel. - A `POST /v1/investigate` webhook call. The agent records novel, decision-relevant findings with source links. Each investigation turn has its own evidence set. The agent records each finding-and-URL pair once, then adds recorded evidence to the Slack result. Automated runs use webhook metadata and service-authenticated tools. They do not use per-user OAuth. Channel watch and custom skills are optional. Both store state in Vercel Blob. The Blob store is required only for these two features. ## Set up the project Click Deploy to clone the repository, create a Vercel project, and provision the connectors and storage. | Provisioned | Sets | | ------------------------------------------------- | ----------------------------------- | | Slack connector with trigger path `/eve/v1/slack` | `SLACK_CONNECTOR` | | GitHub connector, read-only install | `GITHUB_CONNECTOR` | | Datadog connector | `DATADOG_CONNECTOR` | | Vercel MCP connector | `SRE_VERCEL_CONNECTOR` | | Private Vercel Blob store | Vercel Blob environment variables | For a complete CLI setup with your agent, use [`docs/setup-for-agents.md`](https://github.com/vercel-labs/eve-sre-agent-template/tree/main/docs/setup-for-agents.md). After deployment: 1. Invite the Slack app to a channel. 2. Mention `@sre` and confirm that the app replies. 3. To enable automated runs via a webhook, follow [`docs/automate-investigations.md`](https://github.com/vercel-labs/eve-sre-agent-template/tree/main/docs/automate-investigations.md). ## Local development ```bash pnpm install vercel link vercel env pull pnpm dev ``` ## Environment configuration | Variable | Required | Default | What it does | | ------------------- | ----------------- | --------------- | ------------------------------------------------------------------------------------------------------- | | `SLACK_CONNECTOR` | No | `slack/sre` | Connector UID for the Slack channel and bot Web API. | | `GITHUB_CONNECTOR` | No | `github/sre` | Connector UID for the GitHub tools extension. | | `DATADOG_CONNECTOR` | No | `datadog/sre` | Connector UID for the Datadog MCP tools. | | `SRE_VERCEL_CONNECTOR` | No | `vercel/sre` | Connector UID for the Vercel MCP tools. | | `WEBHOOK_SECRET` | No | none | Shared secret for `POST /v1/investigate`. Use a long random value from Node crypto (`randomBytes(32)`). | | `DD_SITE` | No | `datadoghq.com` | Datadog site for MCP. Examples are `datadoghq.eu` and `us5.datadoghq.com`. | Copy [`.env.example`](https://github.com/vercel-labs/eve-sre-agent-template/tree/main/.env.example) to `.env` to set local environment variables. ## Automation endpoint `POST /v1/investigate` accepts a title, a Slack channel, an optional description, and optional metadata. It returns `202` and starts a Slack investigation. The endpoint uses `WEBHOOK_SECRET`. Send the secret with `x-sre-webhook-secret` or `Authorization: Bearer`. ## Customize the agent - Edit `agent/instructions/instructions.md` to change general behavior. - Edit the built-in skills in `agent/skills/` to change investigation and handoff procedures. - Add tools or connections for other operational systems using Vercel Connect. - Integrate with `agent/channels/webhook.ts` to invoke the agent from any external system (see [`docs/automate-investigations.md`](https://github.com/vercel-labs/eve-sre-agent-template/tree/main/docs/automate-investigations.md)). ### Create runbooks in Slack Ask `@sre` to create a runbook for a recurring alert or incident. Global runbooks apply to every session. Personal runbooks apply only to the requesting Slack user and override global runbooks with the same name. The agent loads a saved runbook on a later matching request, not during the request that creates it. Private Vercel Blob storage is required to save and load runbooks. ## Verify changes ```bash pnpm validate pnpm test ``` ## Troubleshooting - If Slack mentions do not arrive, confirm that the trigger path is `/eve/v1/slack`. - If a webhook returns `401`, confirm that the caller and the deployment use the same `WEBHOOK_SECRET`. - If an investigation does not start, confirm that the bot is a member of `slackChannel`. - If custom skills fail to save, confirm that the private Blob store is connected. - If Vercel tools request authorization, complete the Vercel Connect sign-in for the Slack user. Automated runs do not have a user identity, so they cannot use this connection. ## Learn more - [eve documentation](https://eve.dev/docs/introduction) - [Automate investigations with generic webhooks](https://github.com/vercel-labs/eve-sre-agent-template/tree/main/docs/automate-investigations.md) - [Vercel Connect](https://vercel.com/docs/connect) - [Vercel Blob](https://vercel.com/docs/vercel-blob) ## Explore more templates - [eve software factory](https://vercel.com/templates/eve/eve-software-factory) - [eve marketing team](https://vercel.com/templates/eve/eve-marketing-team) - [eve personal agent](https://vercel.com/templates/nuxt/eve-personal-agent) - [All eve templates](https://vercel.com/templates/eve) ## Related Templates ### [eve Marketing Team](https://vercel.com/templates/eve/eve-marketing-team) Marketing team in Slack: one lead, five specialists. Blog drafts in Notion, posts in Typefully, campaigns in Resend, on approval. ### [eve Sanity Copilot](https://vercel.com/templates/eve/eve-sanity-copilot) Give your team a Sanity copilot in Slack. It runs GROQ, edits schemas, manages releases, and asks before it touches production. ### [eve Software Factory](https://vercel.com/templates/eve/eve-software-factory) Software factory built on eve: AI agents work each stage of the development loop, and people make the judgment calls. --- ## Additional documentation The [Vercel Knowledge Base](https://vercel.com/kb) contains guides and answers to common questions about Vercel, including deployment, framework configuration, domains, caching, and troubleshooting.