Vercel Logo

Your first rule

Pinchy's has parked across the street from Claw & Order four days running. They show up ten minutes after you do, and their sign says "Lobster Rolls (Better)." The scraper from 1.3 told them exactly where to go. Let's write a rule that catches it and nothing else, then shut the door.

Outcome

The Pinchy's scraper rule denies PinchyScout with a 403 on / and /location, after log mode showed it matches nothing else.

Log before you block

A custom rule is a condition plus one of the actions from 1.2. Every rule in this course that matches visitors by who they seem to be starts with Log, which records each match and lets the request through. A sloppy condition is much cheaper to discover in a log entry than in an email from a customer.

The rule that's too easy

Look at the scraper's user agent again:

PinchyScout/2.1 (+https://pinchys.example/bot)

It ends in /bot. The tempting rule is "user agent contains bot." Let's write that one first and see what it catches.

In the dashboard, open the project's Firewall page, then open its configuration (currently ⋯ > Configure) and add a new rule (currently Add New... > Rule). Fill it in:

FieldValue
NamePinchy's scraper
IfUser Agent contains bot
ThenLog

Save the rule, then review and publish the change. Rules take effect when you publish, with no redeploy.

Now send the scraper, plus one request pretending to be Google's crawler:

npm run scalper -- scrape --target "$URL" --count 5
curl -s -o /dev/null -w "%{http_code}\n" \
  -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" \
  "$URL/"

Every request still returns 200, because log rules never block. The traffic report shows what the rule would have done:

vercel firewall traffic list --since 1h --action log --dimension user-agent --dimension path

The rule matched PinchyScout, and it matched the Googlebot request too. Search engines say bot in their user agents, and so do plenty of monitoring tools and link previewers. If this rule were set to deny, the truck would vanish from search results by Friday.

Narrow it down

Pinchy's scraper calls itself PinchyScout, and it only wants today's spot, which shows on /location and in the home page's <h1>. Edit the rule so the name and one of those paths both have to match:

FieldValue
NamePinchy's scraper
IfUser Agent contains PinchyScout
ANDRequest Path is any of /, /location
ThenLog

Save and publish, then note the time so the next report only covers the new version:

export SINCE=$(date -u +%Y-%m-%dT%H:%M:%SZ)

Send the same traffic again, then inspect the rule by its id:

npm run scalper -- scrape --target "$URL" --count 5
curl -s -o /dev/null -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" "$URL/"
 
vercel firewall traffic list --since "$SINCE" --dimension rule --json
vercel firewall traffic inspect rule <rule-id> --since "$SINCE" --group-by user-agent

The rule's id is the value of its row in the rules list from traffic list --json. traffic inspect takes the id, not the name.

One user agent: PinchyScout/2.1. Drop --since and Googlebot shows up too. This is expected because the rule kept its id when you edited it, so its history includes the broad version. If anything you care about shows up after the edit, stop here and fix the condition.

Flip it to deny

The log says the rule catches the scraper and nothing else. Open it and change Then from Log to Deny. Save, review, and publish.

On Pro and Enterprise, a timeframe dropdown (currently for) appears next to Deny and starts at one minute. Clear it for now. That timeframe locks out your own IP, and the next step adds it on purpose.

Pro and Enterprise: make it stick

Optional: Pro and Enterprise

Persistent actions are available on Pro and Enterprise. On Hobby, skip to the exercise. Nothing later depends on this step.

A plain deny judges each request on its own, so Pinchy's can still wander the rest of the site. A persistent action blocks the client's whole IP for a time you pick after the first match. Try the shortest one, because the scraper's IP is yours:

vercel firewall rules update "Pinchy's scraper" --action deny --duration 1m --yes
vercel firewall publish --confirm-name claw-and-order
npm run scalper -- scrape --target "$URL" --count 1
curl -s -o /dev/null -w "%{http_code}\n" "$URL/preorder"
vercel firewall persistent-actions list

The scraper got its 403, and so did a plain curl to a page the rule doesn't mention. Your browser is locked out too for the minute, which is expected because it shares the scraper's IP. Against a real scraper on somebody else's server, that's the setting you want. Here, turn it off once the minute is up: in the dashboard, set the timeframe next to Deny (currently labeled for) back to none, then publish.

Hands-on exercise 2.1

Write the rule, watch it over-match, fix it, then make it block.

Requirements:

  1. Create a custom rule named Pinchy's scraper with User Agent contains bot and a Log action. Publish it.
  2. Run the scraper with --count 5 and send one fake Googlebot request to /. Confirm the rule logged both.
  3. Change the conditions to User Agent contains PinchyScout AND Request Path is any of /, /location. Publish and set $SINCE.
  4. Use vercel firewall traffic inspect to confirm the rule only matched PinchyScout/2.1 since the edit.
  5. Change the action to Deny and publish.
  6. Pro and Enterprise, optional: add a 1-minute persistent action, watch it lock out your IP, then turn it off.
  7. Snapshot your rules to a file and commit it.

Implementation hints:

  • Condition and operator names in the dashboard may differ a little from the ones here. Look for the user agent and the request path in the If dropdown.
  • Copy PinchyScout from the scraper's user agent so a typo doesn't sink the rule. Matching ignores case, so capitals don't matter.
  • Traffic can take a minute to show up in the CLI. Run the report again before you change anything.

Try It

Check both pages for both visitors:

UA="PinchyScout/2.1 (+https://pinchys.example/bot)"
 
curl -s -o /dev/null -w "pinchy   /          %{http_code}\n" -A "$UA" "$URL/"
curl -s -o /dev/null -w "pinchy   /location  %{http_code}\n" -A "$UA" "$URL/location"
curl -s -o /dev/null -w "browser  /          %{http_code}\n" "$URL/"
curl -s -o /dev/null -w "browser  /location  %{http_code}\n" "$URL/location"
pinchy   /          403
pinchy   /location  403
browser  /          200
browser  /location  200

Pinchy's is out on both pages. Everyone else gets lunch. The "browser" rows send curl's own user agent, which is close enough for this check. Open /location in your real browser too. The truck is still there.

Commit

Rules live in the Firewall, outside your repo. Save a snapshot so the change has a history:

mkdir -p firewall
vercel firewall rules list --json > firewall/rules.json
git add firewall/rules.json
git commit -m "chore(firewall): deny PinchyScout on / and /location"

From here on, every lesson that changes a rule ends with a fresh snapshot. When a rule misbehaves in Section 5, git log firewall/rules.json will tell you when it changed and why.

Done-When

  • A rule named Pinchy's scraper is published with the Deny action
  • Its conditions are User Agent contains PinchyScout AND Request Path is any of /, /location
  • traffic inspect on the rule shows only PinchyScout/2.1 since your edit
  • PinchyScout gets 403 on / and /location, and a request without that user agent gets 200 on both
  • Pro and Enterprise, optional: you've seen a persistent action block your IP, and the rule is back to a plain deny
  • firewall/rules.json is committed

Troubleshooting

The rule doesn't match, or the scraper still gets 200 after you switched to Deny. Check that you published it. Saving a rule stages the change, and the review step is what makes it live. vercel firewall diff lists anything still waiting.

Every page returns 403 long after the scraper stopped. A persistent action is still on the rule, so each new scraper run blocks your IP again. Check with vercel firewall rules inspect "Pinchy's scraper" and remove the timeframe.

traffic inspect says there's no traffic for the rule. The window is too short or the id is wrong. If you opened a new terminal, $SINCE is empty; use --since 1h. Copy the id from traffic list --dimension rule --json, since traffic inspect doesn't take the rule's name.

Solution

The finished rule:

FieldValue
NamePinchy's scraper
IfUser Agent contains PinchyScout
ANDRequest Path is any of /, /location
ThenDeny

The commands that prove it:

export SINCE=$(date -u +%Y-%m-%dT%H:%M:%SZ)   # right after publishing the narrowed rule
npm run scalper -- scrape --target "$URL" --count 5
vercel firewall traffic list --since "$SINCE" --dimension rule --json
vercel firewall traffic inspect rule <rule-id> --since "$SINCE" --group-by user-agent
 
curl -s -o /dev/null -w "%{http_code}\n" -A "PinchyScout/2.1 (+https://pinchys.example/bot)" "$URL/location"
 
mkdir -p firewall
vercel firewall rules list --json > firewall/rules.json
git add firewall/rules.json
git commit -m "chore(firewall): deny PinchyScout on / and /location"

Pinchy's is locked out by name. That holds right up until Pinchy's changes its name, which is what happens next.

Was this helpful?

supported.