# How to block bots from OpenAI GPTBot

**Author:** Lee Robinson

---

OpenAI has an AI-based crawler (also referred to as “GPTBot”) that fetches and indexes website content, which is then used to power the training data of models like GPT series, o series, and more. According to [our research](https://vercel.com/blog/the-rise-of-the-ai-crawler), OpenAI generates hundreds of millions of monthly requests, making it the most active AI crawler on the web.

## Good bots vs. bad bots

- **Good Bots**: Examples include Googlebot, Applebot, and Bingbot. They’re typically transparent about their intentions, respect robots.txt, and help improve search engine visibility.
  
- **Bad Bots**: These bots may scrape content without permission, inflate server usage, or perform malicious activities. Even certain “legitimate” AI crawlers can become unwanted if they exceed fair-use limits or repeatedly crawl error pages (e.g., excessive 404 requests).
  

We'll focus on the OpenAI GPTBot here.

## What is the Vercel Firewall?

[Vercel Firewall (WAF)](https://vercel.com/docs/concepts/functions/edge-middleware/firewall) is a Web Application Firewall service that lets you:

- **Log or block requests** that match certain criteria (IP address, user agent, request path, geolocation, etc.).
  
- **Challenge suspicious traffic** with an automated check (e.g., requiring the visitor’s browser to pass a JavaScript challenge).
  
- **Rate limit** excessive or malicious requests.
  

All configuration changes are applied globally within ~300ms, and can be rolled back instantly.

## Identifying the Bot

### User Agent

OpenAI identifies itself in the `User-Agent` header. Check your Firewall traffic for entries like `Mozilla/5.0 (compatible; GPTBot/1.0; +https://openai.com/gptbot)`.

If you see repeated requests from that User-Agent, you can be confident it’s OpenAI. Note that AI crawlers occasionally use different sub-variants of user agents over time.

## Taking action with the Firewall

Vercel Firewall allows a few approaches:

### Rate Limit

If you don’t mind occasional indexing by OpenAI models, but want to cap how frequently it crawls your site, you can **rate limit** requests from the “GPTBot” user agent. This helps prevent resource overload while still allowing some AI-based traffic.

### Challenge

If you want to slow down less-dedicated bots, you can **challenge** requests from certain user agents. The challenge forces a short browser-based security check. Most legitimate human traffic (with real browsers) will pass automatically, but a bot that can’t solve or respond to the challenge will be blocked.

### Persistent Blocking (No Charge)

If you want to fully prevent GPTBot from crawling your site—and avoid incurring data transfer or function usage for these requests—you can **persistently block** it. Requests that match your block rule won’t reach your Vercel Functions or static pages, so you won’t be charged.

## Templates

Vercel provides several [Firewall Templates](https://vercel.com/templates/vercel-firewall) you can clone or learn from:

1. [**Rate Limit API Requests**](https://vercel.com/templates/other/rate-limit-api-requests-firewall-rule)
   
2. [**Block OFAC-Sanctioned Countries**](https://vercel.com/templates/other/block-ofac-sanctioned-countries-firewall-rule)
   
3. [**Block WordPress URLs**](https://vercel.com/templates/other/block-wordpress-urls-firewall-rule)
   
4. [**Block Bad Bots**](https://vercel.com/templates/other/block-bad-bots-firewall-rule)
   
5. [**Block AI Bots**](https://vercel.com/templates/other/block-ai-bots-firewall-rule)
   

To block OpenAI GPTBot specifically, you can start with the “Block AI Bots Firewall Rule” template and modify it for the defined user agent.

## Example: Creating a Custom Rule to Block OpenAI GPTBot

1. **Navigate to “Firewall”** in your Vercel project dashboard.
   
2. **Click “Add Rule”** (or “Create Rule” if using templates).
   
3. **Select the “Block AI Bots” template** (or a blank custom rule).
   
4. **Match Condition**: For “User Agent” contains “GPTBot”.
   
5. **Action**: Choose “Deny” (to block) or “Challenge” (to verify) or “Rate Limit” (to limit).
   
6. **Review & Publish** changes.
   

**Note**: Once published, changes take effect globally in ~300ms. You can always roll back if you block or challenge traffic unintentionally.

## Resources

- [**How Google Handles JavaScript Throughout the Indexing Process**](https://vercel.com/blog/how-google-handles-javascript-throughout-the-indexing-process)
  
- [**The Rise of the AI Crawler**](https://vercel.com/blog/the-rise-of-the-ai-crawler)
  

These posts provide background on how advanced bots handle JavaScript, distribution, and crawling inefficiencies (e.g., excessive 404s).

---

[View full KB sitemap](/kb/sitemap.md)