The kitchen is flaky

In every lesson up to now, the kitchen step has done what it said and returned cleanly. That's because we wrote it that way. Reality is less polite.

Real services flake. Networks blip. APIs throw 503s for reasons their own engineers can't explain. The whole reason workflows are interesting is that they handle this without you writing a single line of retry code.

We're going to make acknowledgeKitchen fail 30% of the time on purpose. Then place a few orders. The dashboard will show steps failing and getting retried. The customer will never notice.

Outcome

Make acknowledgeKitchen throw intermittently. Bump its maxRetries to 5. Watch the runtime retry the step automatically and the customer's order complete normally.

Fast Track

1. In workflows/steps/acknowledge-kitchen.ts, add if (Math.random() < 0.3) throw new Error(...) before the status update.
2. Add acknowledgeKitchen.maxRetries = 5 after the function definition.
3. Place several orders. Watch some steps fail and retry in the dashboard while the orders still complete.

Hands-on exercise

1. Add flakiness.

workflows/steps/acknowledge-kitchen.ts:

import { updateStatus } from "@/lib/orders-store";
import type { Order } from "@/lib/pizza";
 
export async function acknowledgeKitchen(order: Order): Promise<void> {
  "use step";
 
  if (Math.random() < 0.3) {
    throw new Error("Kitchen printer jammed. Retrying.");
  }
 
  await new Promise((resolve) => setTimeout(resolve, 200));
 
  updateStatus(order.id, "in-the-kitchen");
}
 
acknowledgeKitchen.maxRetries = 5;

Two changes. The Math.random() check throws an ordinary Error about 30% of the time. The runtime treats any uncaught exception as retryable, so the step gets re-enqueued automatically.

acknowledgeKitchen.maxRetries = 5 bumps the retry count from the default of 3 (4 total attempts) to 5 (6 total attempts). It's a property on the step function. Set it once, after the function is defined.

Try It

Place five or six orders.

In the dashboard, some runs look like the ones you've seen. Others look like this:

processOrder                running
  ├─ sendOrderConfirmation    completed   389ms
  └─ acknowledgeKitchen       running
       attempts:
         1. failed   "Kitchen printer jammed. Retrying."
         2. completed   211ms

The step failed, the runtime waited briefly, the step ran again with the same inputs, the second attempt succeeded. The workflow keeps going. Marge never knew there was a problem.

If you get really unlucky, you might see a step fail three times in a row before succeeding. Or get sketchy enough to fail all six attempts, at which point the workflow fails. With 30% flakiness and 6 attempts, the probability of all attempts failing is 0.3^6 ≈ 0.07%. Rare. But possible.

Now drop maxRetries to 1 just to see it happen:

acknowledgeKitchen.maxRetries = 1;

Place orders. About 30% of them now fail outright on the kitchen step. The workflow shows up red in the dashboard with the error attached. Customer experience: bad. You'd never ship this. But it makes the difference between 1 retry and 5 retries visceral.

Set maxRetries back to 5 before moving on.

Commit

feat(workflow): introduce kitchen flakiness with maxRetries

Done-When

• acknowledgeKitchen throws an Error roughly 30% of the time
• acknowledgeKitchen.maxRetries = 5 is set after the function definition
• You've watched a step fail and retry in the dashboard
• You've toggled maxRetries between 1 and 5 to see the difference

Solution

import { updateStatus } from "@/lib/orders-store";
import type { Order } from "@/lib/pizza";
 
export async function acknowledgeKitchen(order: Order): Promise<void> {
  "use step";
 
  if (Math.random() < 0.3) {
    throw new Error("Kitchen printer jammed. Retrying.");
  }
 
  await new Promise((resolve) => setTimeout(resolve, 200));
 
  updateStatus(order.id, "in-the-kitchen");
}
 
acknowledgeKitchen.maxRetries = 5;

Two lines added, retry behavior unlocked. We didn't write a backoff function. We didn't add a queue. We didn't track attempts in a database. The directive bought us all of it.