AI crawlers are getting more aggressive. If you've seen your Vercel or Netlify bill spike without a corresponding increase in real users, there's a good chance an AI training crawler is behind it. Our analysis of 20 popular developer platforms found that most aren't blocking the crawlers responsible for these cost spikes.
This guide gives you copy-paste configs for every major platform to block the ones that matter most.
Before you start, find out which AI crawlers are currently allowed to access your site:
Check my site free → Full bot list on GitHub →There are two reasons to block AI crawlers:
That said: robots.txt is fast, free, zero-risk, and effective against the major crawlers you're most likely to encounter. Start there.
Add this to your robots.txt file at the root of your domain (https://yourdomain.com/robots.txt):
# Block all major AI training crawlers
User-agent: GPTBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: anthropic-ai
Disallow: /
User-agent: Claude-Web
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: meta-externalagent
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Amazonbot
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: cohere-ai
Disallow: /
User-agent: DiffBot
Disallow: /
User-agent: FacebookBot
Disallow: /
# Allow Google and Bing (comment out if you want to block them too)
User-agent: Googlebot
Allow: /
User-agent: Bingbot
Allow: /
User-agent: *
Allow: /Where to put it:
robots.txt in your repo root.public/ directory.After deploying, verify at crawl-check that the bots now show as blocked.
Vercel can block crawlers at the edge via response headers. Add this to your vercel.json:
{
"headers": [
{
"source": "/(.*)",
"headers": [
{
"key": "X-Robots-Tag",
"value": "noai, noimageai"
}
]
}
]
}The noai and noimageai directives are the HTTP header equivalent of robots.txt for AI crawlers — they're honored by the major crawlers alongside the file-based approach.
For a harder server-side block using Vercel middleware:
// middleware.ts
import { NextResponse } from 'next/server'
import type { NextRequest } from 'next/server'
const AI_BOTS = [
'gptbot', 'chatgpt-user', 'oai-searchbot', 'ccbot',
'anthropic-ai', 'claude-web', 'claudebot', 'meta-externalagent',
'bytespider', 'perplexitybot', 'amazonbot', 'applebot-extended',
'cohere-ai', 'diffbot', 'facebookbot',
]
export function middleware(request: NextRequest) {
const ua = request.headers.get('user-agent')?.toLowerCase() ?? ''
if (AI_BOTS.some(bot => ua.includes(bot))) {
return new NextResponse('Access denied', { status: 403 })
}
return NextResponse.next()
}
export const config = {
matcher: '/((?!_next/static|_next/image|favicon.ico).*)',
}If your site is behind Cloudflare, you can block AI crawlers at the network level before requests ever reach your origin — completely free with any Cloudflare plan.
Option A: WAF Custom Rule (recommended)
Go to Security → WAF → Custom Rules → Create rule, then configure:
Rule name: Block AI crawlers
Expression:
(http.user_agent contains "GPTBot") or
(http.user_agent contains "ChatGPT-User") or
(http.user_agent contains "anthropic-ai") or
(http.user_agent contains "ClaudeBot") or
(http.user_agent contains "Claude-Web") or
(http.user_agent contains "meta-externalagent") or
(http.user_agent contains "Bytespider") or
(http.user_agent contains "PerplexityBot") or
(http.user_agent contains "Amazonbot") or
(http.user_agent contains "CCBot") or
(http.user_agent contains "cohere-ai") or
(http.user_agent contains "DiffBot")
Action: BlockOption B: Page Rule (older, simpler)
For a quick block on the whole domain without WAF:
Security Level → Under Attack mode temporarily, or use a Transform Rule
to block specific user-agents via the header value.The WAF approach is better — it's per-rule, logged, and doesn't affect real users.
Add to your server block:
server {
# ... your existing config ...
# Block AI training crawlers
if ($http_user_agent ~* "(GPTBot|ChatGPT-User|OAI-SearchBot|CCBot|anthropic-ai|Claude-Web|ClaudeBot|meta-externalagent|Bytespider|PerplexityBot|Amazonbot|Applebot-Extended|cohere-ai|DiffBot|FacebookBot)") {
return 403;
}
}Or using a map block for better performance (evaluate once per request):
http {
map $http_user_agent $is_ai_bot {
default 0;
"~*GPTBot" 1;
"~*ChatGPT-User" 1;
"~*anthropic-ai" 1;
"~*ClaudeBot" 1;
"~*Claude-Web" 1;
"~*meta-externalagent" 1;
"~*Bytespider" 1;
"~*PerplexityBot" 1;
"~*Amazonbot" 1;
"~*CCBot" 1;
"~*cohere-ai" 1;
"~*DiffBot" 1;
}
server {
if ($is_ai_bot) {
return 403;
}
}
}After editing, reload Nginx: sudo nginx -s reload
Add to your .htaccess file:
RewriteEngine On
# Block AI training crawlers
RewriteCond %{HTTP_USER_AGENT} (GPTBot|ChatGPT-User|OAI-SearchBot|CCBot|anthropic-ai|Claude-Web|ClaudeBot|meta-externalagent|Bytespider|PerplexityBot|Amazonbot|Applebot-Extended|cohere-ai|DiffBot|FacebookBot) [NC]
RewriteRule .* - [F,L]If mod_rewrite isn't available, use mod_setenvif instead:
<IfModule mod_setenvif.c>
SetEnvIfNoCase User-Agent "GPTBot|ChatGPT-User|anthropic-ai|ClaudeBot|meta-externalagent|Bytespider|PerplexityBot|Amazonbot|CCBot|cohere-ai|DiffBot" bad_bot=1
<IfModule mod_authz_core.c>
<RequireAll>
Require all granted
Require not env bad_bot
</RequireAll>
</IfModule>
</IfModule>Drop this middleware in before your routes:
// ai-bot-block.js
const AI_BOTS = [
'gptbot', 'chatgpt-user', 'oai-searchbot', 'ccbot',
'anthropic-ai', 'claude-web', 'claudebot', 'meta-externalagent',
'bytespider', 'perplexitybot', 'amazonbot', 'applebot-extended',
'cohere-ai', 'diffbot', 'facebookbot',
]
function blockAiBots(req, res, next) {
const ua = (req.headers['user-agent'] || '').toLowerCase()
if (AI_BOTS.some(bot => ua.includes(bot))) {
return res.status(403).json({ error: 'Access denied' })
}
next()
}
module.exports = blockAiBots// app.js
const express = require('express')
const blockAiBots = require('./ai-bot-block')
const app = express()
app.use(blockAiBots) // Apply before all routes
// ... your routes ...For Fastify, add it as a hook:
const AI_BOTS = [
'gptbot', 'chatgpt-user', 'oai-searchbot', 'ccbot',
'anthropic-ai', 'claude-web', 'claudebot', 'meta-externalagent',
'bytespider', 'perplexitybot', 'amazonbot',
'cohere-ai', 'diffbot',
]
fastify.addHook('onRequest', async (request, reply) => {
const ua = (request.headers['user-agent'] || '').toLowerCase()
if (AI_BOTS.some(bot => ua.includes(bot))) {
return reply.status(403).send({ error: 'Access denied' })
}
})Use the crawl-check tool to confirm the bots now show as blocked on your domain. It checks 12+ major AI crawlers against your live robots.txt in seconds.
New AI crawlers appear every few months. The ai-crawler-block-list repo is updated regularly and includes configs for all platforms in this guide. Watch the repo to get notified of new additions.
robots.txt blocks well-behaved crawlers. For real enforcement — and to see which bots were costing you money before you blocked them — you need server-level logging. If you're on Vercel, CrawlGuard connects to your Vercel log drain and shows you per-bot request counts and estimated cost attribution.
robots.txt stops future crawls, but it doesn't tell you what was hitting you before. CrawlGuard connects to your Vercel log drain and shows per-bot cost attribution — so you can see exactly what you were paying for.
Get early access to CrawlGuard → Check my site now →