A payment flow or an agent loop usually ships as a multi-step process within a single function invocation. It holds together until the 300-second default runs out, and then a timeout or a crash erases the in-memory state and forces a restart from step one.

A bigger timeout only moves the deadline. Workflow orchestration removes the failure mode instead, by keeping execution state in a durable record rather than in process memory.

This guide covers what an orchestrator coordinates, how durable execution works underneath it, and where the step boundary belongs in your own code.

Key takeaways:

- Workflow orchestration is the centralized coordination of a multi-step process, where one orchestrator sequences the calls, tracks every state transition, retries failures, and keeps the full execution history.

- Durable execution is what a job queue lacks, because it checkpoints completed steps so a retry never re-runs them.

- Serverless functions lose all in-memory state on timeout or crash, so multi-step work needs checkpoints rather than longer timeouts.

- The step boundary is the design decision that matters most, since anything with a side effect has to sit inside a step for the event log to record and replay it.

- Assuming durable execution requires a dedicated orchestration cluster is the expensive default rather than a technical requirement.

## [Copy link to heading](#what-is-workflow-orchestration)What is workflow orchestration?

Workflow orchestration is the centralized coordination of a multi-step process across distributed services. One orchestrator holds both the flow definition and the execution state, sequences the calls, retries failed steps, and keeps a record of what happened and why. The workers it calls stay stateless and independent.

The word carrying the weight in that definition is state. Durable execution means that when a step fails or the host crashes, the system resumes from the last recorded checkpoint instead of starting over. Any process spanning more than one network call needs that memory, because without it, you can't tell a partial failure from no progress at all.

The clearest way to see what orchestration buys is to compare it against the other three patterns teams reach for when work has to happen outside a request.

### [Copy link to heading](#how-workflow-orchestration-differs-from-queues,-schedulers,-and-choreography)How workflow orchestration differs from queues, schedulers, and choreography

All four patterns coordinate work, and each one fails differently. Where execution state lives at the moment something breaks, decides how much recovery logic you end up writing yourself:

| Pattern | Where state lives | Failure recovery | Flow visibility |
| --- | --- | --- | --- |
| Orchestration | Event log of every step | Resumes from the last checkpoint | Explicit, queryable from one place |
| Job queue | Per message only | Redelivers the message, the application handles the rest | None across steps |
| Scheduler (cron) | Nothing between runs | Handled by the job itself | Trigger times only |
| Choreography | Scattered across services | Handled service by service | Implicit, reconstructed from logs |

A job queue dispatches a message and leaves retry logic, state tracking, and failure coordination to the application, including where a message goes once it stops being retryable, which is what a [dead letter queue](https://vercel.com/i/dead-letter-queue) exists to answer.

At-least-once delivery means every side effect has to be idempotent, and the pressure comes from outside too, since Stripe retries failed [webhook deliveries](https://docs.stripe.com/webhooks) for up to three days in live mode. Cron gives you less again, because it keeps no record of the previous run and leaves recovery after a crash to the job itself. Both work when a task can safely start over, and neither helps when it can't.

Choreography removes the coordinator entirely. Each service emits events and other services react, so the business process exists only as the sum of those reactions, and finding where one order got stuck means correlating logs across every service that touched it.

Orchestration makes the same flow explicit and queryable from one place, which is why production systems often pair the two, orchestrating stateful flows and leaving loose fan-outs to choreography, the same split the [saga pattern](https://vercel.com/i/saga-pattern) draws between its two coordination models.

### [Copy link to heading](#where-data-pipeline-orchestrators-fit)Where data pipeline orchestrators fit

A data pipeline orchestrator schedules batch jobs that move and transform data along a fixed graph, which is a different job from coordinating an application process. [Apache Airflow](https://airflow.apache.org/docs/apache-airflow/stable/index.html) is the standard example, built on static directed acyclic graphs (DAGs), and its own documentation draws the boundary plainly, describing a tool made for finite batch workflows rather than infinitely running event-based ones.

That suits analytics pipelines with a defined start, end, and schedule. Order processing, payment flows, and agent loops are application workflows, and they need branching decisions while the run is in flight, external signals arriving at unpredictable times, and waits of indefinite length that a static graph can't express.

Teams running both are common, and the split usually falls along exactly that line.

### [Copy link to heading](#benefits-of-workflow-orchestration-for-engineering-teams)Benefits of workflow orchestration for engineering teams

Orchestration changes four operational properties that teams otherwise hand-roll one incident at a time:

- Partial failure stops costing a full restart: A run that fails on its fifth step retries only that step, because the first four are recorded as complete and their results are returned from the log. Without checkpointing, every retry re-executes and re-pays for work that already succeeded.

- Waiting becomes ordinary control flow: A run that pauses for a webhook or a human reviewer holds no function open and polls no database on a timer. A one-hour wait and a one-week wait are the same line of code, which changes what durations are worth modeling at all.

- The process becomes queryable from one place: The orchestrator knows which steps have been completed, what each returned, and where the run currently sits. Answering "where did order 4471 stop" becomes a lookup rather than a log correlation exercise.

- Retry and backoff logic leaves application code: Retries, backoff, and step-level replay move into the runtime, so application code holds business logic rather than a hand-written state machine wrapped around it.

None of that earns its keep on the first run. The return arrives at volume, when the failure rate of any single external dependency multiplied by daily run count guarantees that some run is mid-flight when a deploy goes out or a model API returns a 500\\.

A useful threshold is whether a process has two or more of four properties, meaning multiple dependent steps, calls to external services, execution time that can outlast a single invocation, and a requirement that a partial failure not restart everything.

Below that bar, orchestration is overhead. Above it, the alternative is writing an orchestrator yourself out of status tables and retry columns.

## [Copy link to heading](#core-components-of-a-workflow-orchestration-system)Core components of a workflow orchestration system

Underneath the pattern, a durable execution engine is a small number of parts that have to work together. What each part exposes to your code decides how much orchestration plumbing you write by hand. Five components carry that work, and each one answers a different question about how a run survives interruption.

### [Copy link to heading](#the-append-only-event-log)The append-only event log

The event log is the source of truth for the execution state. Before the application observes the result of any step, that step's inputs and outputs are written to the log, along with sleeps, hook registrations, and errors. Every state transition becomes a recorded event.

This ordering is the part that teams get wrong when they build it themselves. Writing the record after the application acts on a result leaves a window where a crash loses the fact that the step ran at all, which is how duplicate charges happen. Recording before observation closes that window, at the cost of a write on the hot path of every step.

### [Copy link to heading](#deterministic-replay)Deterministic replay

When a crash or a redeploy interrupts a run, the system replays the log to rebuild the exact execution state it had. Steps already marked complete are skipped, and their recorded outputs are handed back to your code rather than the underlying service getting called a second time.

Replay only works if workflow-level code is deterministic, which is a real constraint rather than a detail. Anything with a side effect has to live inside a step where the log can record it. That covers API calls and database writes, and also random values and timestamps, which produce different results on every replay and silently fork execution if left at the workflow level.

The same log that makes replay possible doubles as a complete record of what the run did, which is what makes a failed run readable afterward instead of guesswork.

### [Copy link to heading](#step-checkpointing-and-retries)Step checkpointing and retries

The step is the atomic unit of progress. Each one is a stateless function with built-in retries that survive network errors and process crashes, and each completed step is a checkpoint the run can resume from.

Retry behavior needs failure classification to be useful. In the Workflow SDK, steps retry three times by default and `maxRetries` adjusts that per step, while `RetryableError` marks a transient failure and accepts a `retryAfter` delay for [exponential backoff](https://vercel.com/i/exponential-backoff).

`FatalError` marks a permanent failure that skips the retry queue entirely. A bad API key doesn't become valid on the third attempt, and treating it as retryable turns a fast, loud failure into a slow, quiet one.

### [Copy link to heading](#suspension-through-sleep-and-hooks)Suspension through sleep and hooks

Calling `sleep("30s")` inside a durable workflow writes a checkpoint to durable storage and returns. The invocation ends there and the compute is released. Thirty seconds later, any available function instance resumes from that checkpoint, across deploys and host restarts. Sleep durations run from minutes to months.

A hook is a wait that ends on an event rather than a clock. A workflow can pause until an external system calls back through a webhook, until a human reviewer responds, or until a slow third-party API returns. The run stays suspended for however long that takes, so a slow counterparty only extends the pause.

### [Copy link to heading](#deployment-version-pinning)Deployment version pinning

A run stays bound to the deployment version that started it. A deploy on Tuesday leaves a run from last week resuming on last week's code, because deterministic replay needs the same code that wrote the log to read it back.

This gives you a clean upgrade boundary, where in-flight runs finish on the version they started with and new runs pick up the latest deployment. It also imposes a discipline, because a breaking change to a step's inputs or outputs affects only new runs, so schema changes need to stay compatible for as long as the longest-lived run is still open.

## [Copy link to heading](#how-vercel-powers-workflow-orchestration-for-engineering-teams)How Vercel powers workflow orchestration for engineering teams

Most durable execution systems ask you to run a cluster of their servers, backed by a database, with a fleet of workers alongside it. That's a second distributed system to scale and keep healthy before your first workflow does anything.

Durable execution doesn't require any of that. [Framework-defined infrastructure](/blog/framework-defined-infrastructure) takes the other route, where the platform reads your application code and provisions what the code needs, and Workflows applies the same idea to long-running work.

### [Copy link to heading](#orchestration-that-lives-in-application-code,-not-a-separate-cluster)Orchestration that lives in application code, not a separate cluster

Standing up an orchestration cluster means a second codebase, a second deploy target, and a second on-call surface, all before the first workflow does anything useful. For a team of five shipping an agent, that cost lands entirely on the people who were supposed to be building the product.

[Vercel Workflows](https://vercel.com/docs/workflows) makes the orchestrator your own code. Marking an async function with `"use workflow"` makes it durable, and marking a called function with `"use step"` makes it an isolated, retryable unit of progress:

```
import { sleep } from "workflow";

export async function processOrder(orderId: string) {
  "use workflow";
  const order = await chargeCard(orderId);
  await sleep("1h");
  return await sendReceipt(order);
}

async function chargeCard(orderId: string) {
  "use step";
  return await payments.charge(orderId);
}

async function sendReceipt(order: Order) {
  "use step";
  return await email.send(order.email);
}
```

There's no queue wiring, no scheduler, and no separate orchestration definition. The function is the flow.

### [Copy link to heading](#steps-run-on-the-same-function-infrastructure-as-your-requests)Steps run on the same function infrastructure as your requests

Running orchestration on a separate infrastructure means paying for a control plane that sits idle between runs and reasoning about two scaling models during an incident.

On Vercel, each step executes as its own [Vercel Functions](https://vercel.com/docs/functions/limitations) invocation on [Fluid compute](https://vercel.com/docs/fluid-compute), dispatched through [Vercel Queues](https://vercel.com/docs/queues), with managed persistence holding the state and event logs. The same infrastructure that serves your requests runs your steps, which means one deploy target instead of two. Step inputs, outputs, and stream chunks are encrypted before they leave your deployment by default.

### [Copy link to heading](#waits-that-consume-no-compute)Waits that consume no compute

Function duration is the constraint that pushes teams toward orchestration in the first place. Vercel Functions default to 300 seconds across plans and reach 800 seconds on Pro and Enterprise, with an extended maximum of [30 minutes](https://vercel.com/docs/functions/configuring-functions/duration) in beta for supported Node.js and Python runtimes. Long work fits inside that window, but work that has to survive a crash inside that window does not.

Suspension solves the duration problem differently. Fluid compute bills Active CPU from [$0.128 per hour](https://vercel.com/docs/functions/usage-and-pricing) and only while your code is running, so a workflow waiting a full day on a payment webhook executes nothing and accrues no compute cost for that day. Stored run state bills separately as [retained data](https://vercel.com/docs/workflows/pricing), which is the tradeoff worth knowing about before modeling waits in months.

### [Copy link to heading](#durable-agents-that-can-pause-for-a-human-reviewer)Durable agents that can pause for a human reviewer

Agent loops fail mid-iteration when a model API returns a 500, and without checkpointing, the retry replays every prior iteration and re-pays for every token. Approval flows have the same shape, where the agent needs to stop before a write and wait for a person who may not respond today.

AI SDK 7 addresses both through [`WorkflowAgent`](https://ai-sdk.dev/docs/agents/workflow-agent), which runs the agent loop inside a workflow, so each tool call becomes a durable step. Tools marked `needsApproval` suspend the run until a reviewer responds, with no custom state store and no polling loop. Higher-risk flows can opt into hash-based message authentication code (HMAC) signing, which binds the original tool inputs to the approval token so arguments can't be altered between request and resumption. Vercel's knowledge base walks through the pattern end-to-end in a [human-in-the-loop moderation guide](https://vercel.com/kb/guide/building-human-in-the-loop-agents-for-community-moderation-with-durable-workflows).

### [Copy link to heading](#run-history-without-writing-instrumentation)Run history without writing instrumentation

Debugging a multi-step process usually means correlating logs across services by timestamp during an incident, which is the slowest debugging mode available.

Because durability requires recording every step anyway, the run history comes with the runtime. The Vercel dashboard shows traces, logs, and metrics per run, with pause, replay, and time-travel debugging on top of the recorded state. These primitives run at production volume. Since the October 2025 beta and through general availability in April 2026, Workflows has processed over [100 million runs](/blog/a-new-programming-model-for-durable-execution) and 500 million steps across more than 1,500 teams. [Mux](/blog/how-mux-shipped-durable-video-workflows-with-their-mux-ai-sdk) shipped the same directives inside its own SDK, so callers get a durable pipeline from a normal import.

## [Copy link to heading](#five-best-practices-for-implementing-workflow-orchestration)Five best practices for implementing workflow orchestration

The runtime handles retries, checkpoints, and replay on its own. What it can't handle is code that isn't safe to replay, effects that aren't safe to repeat, or work that didn't need a workflow in the first place.

### [Copy link to heading](#draw-the-step-boundary-around-every-side-effect)Draw the step boundary around every side effect

The most common way a durable workflow behaves non-deterministically is a `fetch` call or a database write sitting at the workflow level rather than inside a step. Nothing fails immediately, which is what makes it dangerous. The bug appears on the first replay, when the call runs a second time.

The step boundary is the line between recorded and re-executed, so anything you'd be unhappy to see happen twice belongs inside a step.

### [Copy link to heading](#make-every-step-idempotent-before-you-rely-on-retries)Make every step idempotent before you rely on retries

Automatic retries are only safe if repeating a step is safe. A step that charges a card, sends an email, or POSTs to an external service can be retried after a network error that occurred after the external system already committed the write.

Those effects need an [idempotency key](https://vercel.com/i/what-is-idempotency) the downstream service honors, or a dedupe record written under a unique constraint before the effect runs. Retries make failures survivable, and idempotency is what makes the retries themselves safe.

### [Copy link to heading](#classify-failures-instead-of-retrying-everything-the-same-way)Classify failures instead of retrying everything the same way

Retrying a 401 three times produces three identical failures and delays the alert that would have told someone the credential expired. Retrying a 429 immediately makes the rate limit worse.

Mapping status codes to error types at the step boundary fixes both, with a fatal error for client errors that'll never succeed and a retryable error carrying an explicit delay for rate limits and upstream 5xx responses. The cost is a few lines of classification per external call, which buys back faster failure signals.

### [Copy link to heading](#suspend-on-time-and-events-instead-of-polling)Suspend on time and events instead of polling

The pattern to avoid is a workflow that wakes on a short interval to check whether an external event has arrived. It burns compute on every check, adds latency equal to half the polling interval, and puts steady read load on whatever it polls.

Modeling the wait directly avoids all three, with a sleep for known durations and a hook for unknown ones. The tradeoff is that the external system now has to signal you, so an integration that offers no webhook still needs polling, and that polling should be bounded rather than open-ended.

### [Copy link to heading](#send-single-step-work-to-a-queue-instead-of-a-workflow)Send single-step work to a queue instead of a workflow

Logging, cache invalidation, and a discrete notification are single-step background jobs with no dependencies between them. Wrapping them in a workflow adds event storage and replay overhead for state that has nothing to track.

The split worth holding to is that anything needing to happen off the request path goes to a queue, anything that has to survive a crash midway through a sequence goes to a workflow, and anything that runs on a clock goes to cron.

## [Copy link to heading](#ship-long-running-work-that-survives-a-crash)Ship long-running work that survives a crash

The multi-step process inside a single function invocation works until it doesn't, and the moment it stops working is a timeout or a crash that erases everything the run had accomplished. Longer timeouts postpone that moment without changing it. Checkpoints remove it, and the design decision that follows is where to draw the step boundary rather than which orchestration cluster to operate.

Here is how Vercel collapses durable execution onto primitives an engineering team can adopt without running a second system:

- Vercel Workflows with `"use workflow"` and `"use step"`: Durable execution defined in application code, with automatic retries, deterministic replay from the last completed step, and no orchestration server to provision or operate.

- Sleep and hooks on Fluid compute: Runs suspend for minutes to months without holding compute, and resume on any available instance across deploys, so waiting on a webhook or a reviewer costs no Active CPU.

- Vercel Functions and Vercel Queues underneath: Steps execute as isolated function invocations dispatched through Queues with managed persistence for state and event logs, on the same infrastructure and deploy target as your application.

- `WorkflowAgent` in AI SDK 7: Agent loops run as durable steps with tool-level approval gates that suspend for hours or days, plus optional HMAC signing binding tool inputs to the approval token.

- Built-in run observability: Every step input, output, sleep, and error is recorded without instrumentation, with per-run traces, replay, and time-travel debugging in the dashboard.

[Start a new project](https://vercel.com/new) and get a durable workflow running on your first `git push`, or follow the [Workflows docs](https://vercel.com/docs/workflows) to add one to an application you already run.

## [Copy link to heading](#frequently-asked-questions-about-workflow-orchestration)Frequently asked questions about workflow orchestration

### [Copy link to heading](#what-is-the-difference-between-workflow-orchestration-and-a-message-queue)What is the difference between workflow orchestration and a message queue?

A queue dispatches individual messages with at-least-once delivery and leaves state management to the application. An orchestrator tracks the full execution graph, so it knows which steps completed, what they returned, and where to resume after a failure.

### [Copy link to heading](#does-durable-workflow-execution-require-a-separate-server)Does durable workflow execution require a separate server?

Not on Vercel. The Workflows runtime executes inside the same serverless function infrastructure as your application code, using Vercel Queues for dispatch and managed persistence for state, so there's no orchestration cluster to provision or operate.

### [Copy link to heading](#how-long-can-a-workflow-sleep-or-pause)How long can a workflow sleep or pause?

Runs can sleep or wait on external events for minutes to months, with no ceiling equivalent to a function timeout. Duration itself isn't a billed dimension. A long wait accrues a retained-data cost for the stored run state rather than execution cost.

### [Copy link to heading](#what-happens-to-an-in-flight-workflow-when-new-code-is-deployed)What happens to an in-flight workflow when new code is deployed?

It finishes on the deployment that started it, uninterrupted. The constraint that follows is versioning discipline. A step's inputs and outputs have to stay readable by runs that began before the deploy, so a breaking change belongs in a new workflow function rather than an edit to the existing one.

### [Copy link to heading](#when-should-i-use-a-queue-instead-of-a-workflow)When should I use a queue instead of a workflow?

A queue fits work that is one step and depends on nothing before it, like writing an analytics event or generating a thumbnail after upload. A workflow earns its event-storage overhead when steps depend on each other, when a run has to survive a crash midway, or when a step waits on an external event.