Vercel Logo

The lunch order

Marguerite is still on hold. You know why Harborview got blocked: Preorder rush gives every IP 10 preorders per 10 minutes, and her whole office is one IP. She's polite about it, the way people are polite right before they switch trucks. You need the office through, and you need it now. Let's do the fast thing first, because it's the thing everybody does.

Outcome

You can roll back a bad Firewall change from the audit log, and Preorder rush lets a 30-person office through with a cost you chose on purpose.

The fastest fix

The rule is blocking customers, so turn it off:

vercel firewall rules disable "Preorder rush" --yes
vercel firewall publish --confirm-name claw-and-order

With the rule off, there's no window to wait out. Restock and send Harborview:

npm run restock
npm run lunch -- --target "$URL"
Results
  201    30

Thirty rolls for Harborview. Marguerite says thanks and hangs up. Then the preorder bot runs, the way it does every morning at 11:

npm run scalper -- preorder --target "$URL"
Results
  201            120
  409            80

Sold out. You fixed one office by handing the bot every roll that was left. Bot Protection is still on log from 5.1's drill mode, so nothing else stood in the way. On a real morning Bot Protection would be on challenge. You'd still have pulled the one layer built for this endpoint.

Roll it back

You could re-enable the rule. But in a real incident you've usually changed three things in a hurry and remember two of them. The Firewall keeps every published configuration, and you can restore any of them:

  1. Open the project's Firewall page in the dashboard.
  2. From the menu at the top right (currently ⋯), open the audit log (currently View Audit Log).
  3. Use the date and time selectors to find the version from before you disabled Preorder rush. It's the one right before your last publish.
  4. Select Restore, then confirm (currently Restore Configuration).

Check the result from the terminal:

vercel firewall rules inspect "Preorder rush"

Enabled, 10 per 10 minutes, exactly as it was. A restore brings back the whole configuration from that version, so anything you published after it is gone too. That's what you want after a panic, and it's why the next fix goes in carefully.

The real options

Allowlist Harborview's IP. A bypass rule, like TidePay's, would let Harborview skip Preorder rush. It also lets anything else on that network skip it, and it breaks quietly the day their IT department changes providers. On Hobby you're already out of custom rules.

Change the key. Outside Enterprise, rate limits key by IP or JA4 (limits). Keying by JA4 puts every customer on the same browser into one bucket, which is Harborview's problem for the whole city. Counting per phone number is what 3.2 does, on Pro and Enterprise.

Raise the ceiling. Give each IP enough room for a big office. It works on every plan, and it has an obvious cost: the bot gets that much room too.

Before touching the number, check what's behind it. Custom rules run before managed rulesets, so since 4.1 every preorder that clears Preorder rush still meets Bot Protection's browser challenge, and a script fails it.

That changes the rate limit's job. In 3.1 it stood alone between the bot and the rolls. Now it caps what one address can take before Bot Protection steps in, so we can size it for the biggest honest customer.

Raise it

Harborview has 30 people. Give a little room for retries:

vercel firewall rules update "Preorder rush" --rate-limit-requests 40 --yes
vercel firewall diff
vercel firewall publish --confirm-name claw-and-order

diff shows exactly one change, the request count. That's the habit that would have saved you 20 minutes ago.

Hands-on exercise 5.2

Make the rushed fix, undo it, then make the careful one.

Requirements:

  1. Disable Preorder rush and publish. Restock and run the lunch rush, then the preorder bot.
  2. Restore the configuration from before the disable in the dashboard's audit log.
  3. Confirm Preorder rush is enabled at 10 requests per 600 seconds.
  4. Raise it to 40 requests per 600 seconds, review the diff, and publish.
  5. Restock, wait out the window, run the lunch rush on its own, then (after another window) the preorder bot on its own.
  6. Snapshot your rules and commit, with a message that says what the bot gains.

Implementation hints:

  • Both scripts share your IP, so the second one inherits the first one's count. Restocking resets the rolls. The Firewall's count keeps running.
  • vercel firewall traffic list --since "$SINCE" --action log --dimension rule shows what the log-mode Bot Protection would have challenged.
  • In production, Bot Protection is on challenge. It's on log only for these drills, and 5.3 switches it back.

Try It

The lunch rush, on its own, after a restock:

npm run lunch -- --target "$URL"
Harborview Insurance lunch rush → https://claw-and-order.vercel.app  (30 requests)
Only point this at a deployment you own.

  Gus G.         201
  Marguerite H.  201
  Dot I.         201
  ...

Results
  201    30

Thirty rolls for Harborview. Now the cost. After a fresh window, mark the time so the report skips Harborview's requests, then send the preorder bot:

export SINCE=$(date -u +%Y-%m-%dT%H:%M:%SZ)
npm run scalper -- preorder --target "$URL"
vercel firewall traffic list --since "$SINCE" --action log --dimension rule
Results
  201            40
  429            160

Forty rolls, in drill mode, instead of 120. The traffic report shows Bot Protection logged those same requests: on challenge, they'd have been stopped before reaching the route.

Commit

vercel firewall rules list --json > firewall/rules.json
git add firewall/rules.json
git commit -m "chore(firewall): raise preorder ceiling to 40 per IP per 10 min" \
  -m "Fits a 30-person office on one IP. Script bots that clear the ceiling still meet Bot Protection; headless-browser bots get up to 40. Disabling the rule outright let the bot sell us out; restored from the audit log."

The second -m is for the next person who reads git log firewall/rules.json and wonders why the limit is so generous. Probably you.

Done-When

  • You've seen the preorder bot sell out the truck with Preorder rush disabled
  • You've restored the previous configuration from the audit log
  • Preorder rush allows 40 requests per 600 seconds
  • The lunch rush gets 30 201s on its own, and the bot gets about 40
  • firewall/rules.json is committed with a message that names the cost

Troubleshooting

You can't find the right version in the audit log. Each publish is a version, so a busy session leaves several. Match the time to your git log firewall/rules.json commits, or to when you ran rules disable. If you restore the wrong one, open the audit log again and restore the right one.

The lunch rush still gets 429s after the change. You're inside the window from an earlier run, and the IP's count carried over. Wait 10 minutes and run it again. vercel firewall traffic inspect ip <your-ip> --since 15m shows how many requests your IP sent recently.

Bot Protection logged nothing for the bot. Check its action with vercel firewall bot-management. If it's off, set it to log for the drill.

Solution

The rushed fix, the rollback, and the careful fix:

# The rushed fix
vercel firewall rules disable "Preorder rush" --yes
vercel firewall publish --confirm-name claw-and-order
npm run restock
npm run lunch -- --target "$URL"
npm run scalper -- preorder --target "$URL"
 
# Undo: dashboard > Firewall > ⋯ > View Audit Log > Restore
vercel firewall rules inspect "Preorder rush"
 
# The careful fix
vercel firewall rules update "Preorder rush" --rate-limit-requests 40 --yes
vercel firewall diff
vercel firewall publish --confirm-name claw-and-order
 
vercel firewall rules list --json > firewall/rules.json
git add firewall/rules.json
git commit -m "chore(firewall): raise preorder ceiling to 40 per IP per 10 min"

Marguerite's team gets lunch, and the bot gets a ceiling. The next time a rule goes wrong in a hurry, the audit log has the last version that worked.

Was this helpful?

supported.