Your first rule
Pinchy's has parked across the street from Claw & Order four days running. They show up ten minutes after you do, and their sign says "Lobster Rolls (Better)." The scraper from 1.3 told them exactly where to go. Let's write a rule that catches it and nothing else, then shut the door.
Outcome
The Pinchy's scraper rule denies PinchyScout with a 403 on / and /location, after log mode showed it matches nothing else.
Log before you block
A custom rule is a condition plus one of the actions from 1.2. Every rule in this course that matches visitors by who they seem to be starts with Log, which records each match and lets the request through. A sloppy condition is much cheaper to discover in a log entry than in an email from a customer.
The rule that's too easy
Look at the scraper's user agent again:
PinchyScout/2.1 (+https://pinchys.example/bot)It ends in /bot. The tempting rule is "user agent contains bot." Let's write that one first and see what it catches.
In the dashboard, open the project's Firewall page, then open its configuration (currently ⋯ > Configure) and add a new rule (currently Add New... > Rule). Fill it in:
| Field | Value |
|---|---|
| Name | Pinchy's scraper |
| If | User Agent contains bot |
| Then | Log |
Save the rule, then review and publish the change. Rules take effect when you publish, with no redeploy.
Now send the scraper, plus one request pretending to be Google's crawler:
npm run scalper -- scrape --target "$URL" --count 5
curl -s -o /dev/null -w "%{http_code}\n" \
-A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" \
"$URL/"Every request still returns 200, because log rules never block. The traffic report shows what the rule would have done:
vercel firewall traffic list --since 1h --action log --dimension user-agent --dimension pathThe rule matched PinchyScout, and it matched the Googlebot request too. Search engines say bot in their user agents, and so do plenty of monitoring tools and link previewers. If this rule were set to deny, the truck would vanish from search results by Friday.
Narrow it down
Pinchy's scraper calls itself PinchyScout, and it only wants today's spot, which shows on /location and in the home page's <h1>. Edit the rule so the name and one of those paths both have to match:
| Field | Value |
|---|---|
| Name | Pinchy's scraper |
| If | User Agent contains PinchyScout |
| AND | Request Path is any of /, /location |
| Then | Log |
Save and publish, then note the time so the next report only covers the new version:
export SINCE=$(date -u +%Y-%m-%dT%H:%M:%SZ)Send the same traffic again, then inspect the rule by its id:
npm run scalper -- scrape --target "$URL" --count 5
curl -s -o /dev/null -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" "$URL/"
vercel firewall traffic list --since "$SINCE" --dimension rule --json
vercel firewall traffic inspect rule <rule-id> --since "$SINCE" --group-by user-agentThe rule's id is the value of its row in the rules list from traffic list --json. traffic inspect takes the id, not the name.
One user agent: PinchyScout/2.1. Drop --since and Googlebot shows up too. This is expected because the rule kept its id when you edited it, so its history includes the broad version. If anything you care about shows up after the edit, stop here and fix the condition.
Flip it to deny
The log says the rule catches the scraper and nothing else. Open it and change Then from Log to Deny. Save, review, and publish.
On Pro and Enterprise, a timeframe dropdown (currently for) appears next to Deny and starts at one minute. Clear it for now. That timeframe locks out your own IP, and the next step adds it on purpose.
Pro and Enterprise: make it stick
A plain deny judges each request on its own, so Pinchy's can still wander the rest of the site. A persistent action blocks the client's whole IP for a time you pick after the first match. Try the shortest one, because the scraper's IP is yours:
vercel firewall rules update "Pinchy's scraper" --action deny --duration 1m --yes
vercel firewall publish --confirm-name claw-and-order
npm run scalper -- scrape --target "$URL" --count 1
curl -s -o /dev/null -w "%{http_code}\n" "$URL/preorder"
vercel firewall persistent-actions listThe scraper got its 403, and so did a plain curl to a page the rule doesn't mention. Your browser is locked out too for the minute, which is expected because it shares the scraper's IP. Against a real scraper on somebody else's server, that's the setting you want. Here, turn it off once the minute is up: in the dashboard, set the timeframe next to Deny (currently labeled for) back to none, then publish.
Hands-on exercise 2.1
Write the rule, watch it over-match, fix it, then make it block.
Requirements:
- Create a custom rule named
Pinchy's scraperwith User Agent containsbotand a Log action. Publish it. - Run the scraper with
--count 5and send one fake Googlebot request to/. Confirm the rule logged both. - Change the conditions to User Agent contains
PinchyScoutAND Request Path is any of/,/location. Publish and set$SINCE. - Use
vercel firewall traffic inspectto confirm the rule only matchedPinchyScout/2.1since the edit. - Change the action to Deny and publish.
- Pro and Enterprise, optional: add a 1-minute persistent action, watch it lock out your IP, then turn it off.
- Snapshot your rules to a file and commit it.
Implementation hints:
- Condition and operator names in the dashboard may differ a little from the ones here. Look for the user agent and the request path in the If dropdown.
- Copy
PinchyScoutfrom the scraper's user agent so a typo doesn't sink the rule. Matching ignores case, so capitals don't matter. - Traffic can take a minute to show up in the CLI. Run the report again before you change anything.
Try It
Check both pages for both visitors:
UA="PinchyScout/2.1 (+https://pinchys.example/bot)"
curl -s -o /dev/null -w "pinchy / %{http_code}\n" -A "$UA" "$URL/"
curl -s -o /dev/null -w "pinchy /location %{http_code}\n" -A "$UA" "$URL/location"
curl -s -o /dev/null -w "browser / %{http_code}\n" "$URL/"
curl -s -o /dev/null -w "browser /location %{http_code}\n" "$URL/location"pinchy / 403
pinchy /location 403
browser / 200
browser /location 200Pinchy's is out on both pages. Everyone else gets lunch. The "browser" rows send curl's own user agent, which is close enough for this check. Open /location in your real browser too. The truck is still there.
Commit
Rules live in the Firewall, outside your repo. Save a snapshot so the change has a history:
mkdir -p firewall
vercel firewall rules list --json > firewall/rules.json
git add firewall/rules.json
git commit -m "chore(firewall): deny PinchyScout on / and /location"From here on, every lesson that changes a rule ends with a fresh snapshot. When a rule misbehaves in Section 5, git log firewall/rules.json will tell you when it changed and why.
Done-When
- A rule named
Pinchy's scraperis published with the Deny action - Its conditions are User Agent contains
PinchyScoutAND Request Path is any of/,/location traffic inspecton the rule shows onlyPinchyScout/2.1since your edit- PinchyScout gets
403on/and/location, and a request without that user agent gets200on both - Pro and Enterprise, optional: you've seen a persistent action block your IP, and the rule is back to a plain deny
firewall/rules.jsonis committed
Troubleshooting
The rule doesn't match, or the scraper still gets 200 after you switched to Deny. Check that you published it. Saving a rule stages the change, and the review step is what makes it live. vercel firewall diff lists anything still waiting.
Every page returns 403 long after the scraper stopped. A persistent action is still on the rule, so each new scraper run blocks your IP again. Check with vercel firewall rules inspect "Pinchy's scraper" and remove the timeframe.
traffic inspect says there's no traffic for the rule. The window is too short or the id is wrong. If you opened a new terminal, $SINCE is empty; use --since 1h. Copy the id from traffic list --dimension rule --json, since traffic inspect doesn't take the rule's name.
Solution
The finished rule:
| Field | Value |
|---|---|
| Name | Pinchy's scraper |
| If | User Agent contains PinchyScout |
| AND | Request Path is any of /, /location |
| Then | Deny |
The commands that prove it:
export SINCE=$(date -u +%Y-%m-%dT%H:%M:%SZ) # right after publishing the narrowed rule
npm run scalper -- scrape --target "$URL" --count 5
vercel firewall traffic list --since "$SINCE" --dimension rule --json
vercel firewall traffic inspect rule <rule-id> --since "$SINCE" --group-by user-agent
curl -s -o /dev/null -w "%{http_code}\n" -A "PinchyScout/2.1 (+https://pinchys.example/bot)" "$URL/location"
mkdir -p firewall
vercel firewall rules list --json > firewall/rules.json
git add firewall/rules.json
git commit -m "chore(firewall): deny PinchyScout on / and /location"Pinchy's is locked out by name. That holds right up until Pinchy's changes its name, which is what happens next.
Was this helpful?