← Blog  ·  September 2026

How to Block AI Crawlers in robots.txt (GPTBot, ClaudeBot, Meta-externalagent, and more)

The complete reference. Every major AI crawler, the exact User-Agent string, whether it actually respects the rules, and a copy-paste block-all snippet.

AI training crawlers are now among the most active bots on the web. Unlike search crawlers, they provide no direct benefit to most site owners — no search rankings, no referral traffic. They consume bandwidth, can spike hosting bills, and scrape content for model training without attribution.

This is the complete reference for blocking them via robots.txt. It covers every major AI crawler, the exact User-Agent string you need, compliance track records, and what to do when robots.txt isn't enough.

Copy-paste: block all major AI crawlers

Add this to your robots.txt file at the root of your site. This blocks all major AI training and retrieval crawlers as of 2026:

# Block AI training crawlers
# OpenAI
User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: OAI-SearchBot
Disallow: /

# Anthropic
User-agent: ClaudeBot
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: Claude-Web
Disallow: /

# Meta
User-agent: meta-externalagent
Disallow: /

User-agent: FacebookBot
Disallow: /

# ByteDance / TikTok
User-agent: Bytespider
Disallow: /

# Common Crawl
User-agent: CCBot
Disallow: /

# Perplexity
User-agent: PerplexityBot
Disallow: /

# Google (AI training only — does NOT block Google Search)
User-agent: Google-Extended
Disallow: /

# Apple
User-agent: Applebot-Extended
Disallow: /

# Amazon
User-agent: Amazonbot
Disallow: /

# Cohere
User-agent: cohere-ai
Disallow: /

Want a maintained, community-updated version of this block list? See github.com/lior-seats/ai-crawler-block-list — includes an auto-generated robots.txt snippet that's kept up to date as new crawlers are identified.

Per-bot reference table

Every major AI crawler, with exact User-Agent strings and compliance notes:

Does blocking actually work?

For most crawlers: yes, mostly. The major companies (OpenAI, Anthropic, Google, Apple) have made public commitments to robots.txt compliance and have business incentives to honor them — they don't want to be blocked at the CDN/firewall level by every publisher simultaneously.

The exceptions:

Bytespider (ByteDance)

Multiple independent analyses have found Bytespider crawling pages explicitly disallowed in robots.txt. If you need to reliably block it, you need server-level or CDN-level blocking, not just robots.txt. The User-Agent string also sometimes varies — look for strings containing Bytedance, bytespider, or TikTokBot.

Wildcards and unknown crawlers

Adding User-agent: * with Disallow: / would block everything — including Googlebot, which you almost certainly don't want. The per-crawler approach above is safer. It's also worth checking logs for User-Agents that don't match known bots but exhibit bot-like behavior (high request rates, no referrer, sequential URL patterns).

Proxy-based crawling

Some crawlers (Perplexity was documented doing this in 2024) route requests through residential proxy networks, masking the bot User-Agent entirely. robots.txt cannot defend against this. It requires rate limiting or bot detection at the infrastructure level.

How to verify your rules are working

The most common mistake: updating robots.txt and assuming it worked, without checking.

Three things can silently break your robots.txt update:

• You edited the wrong file, and the deployed version is still the old one
• A CDN or caching layer is serving a cached version of the old robots.txt
• A syntax error in the robots.txt causes parsers to skip the new rules

Use the free AI Crawler Check tool to verify your live robots.txt. It fetches your actual deployed file (not a local copy), parses it, and checks each major AI crawler against your rules. You get a per-bot status in about 5 seconds.

For ongoing verification after changes:

# Fetch your live robots.txt and inspect it
curl -s https://yourdomain.com/robots.txt | grep -A2 "GPTBot\|ClaudeBot\|meta-externalagent"

# Test whether a specific path would be allowed for a given bot
# (using Google's robots.txt tester in Search Console is the most reliable method)

Google Search Console also has a robots.txt tester — go to Settings → robots.txt. It lets you enter any User-Agent and URL to see whether that agent is blocked or allowed by your rules.

When robots.txt isn't enough

robots.txt is a declaration. It works for bots that choose to honor it. For complete protection, add server-level blocking:

Nginx

if ($http_user_agent ~* "(GPTBot|ClaudeBot|anthropic-ai|meta-externalagent|Bytespider|CCBot|PerplexityBot)") {
    return 403;
}

Cloudflare (WAF rule)

In Cloudflare WAF → Custom Rules, create a rule with:

Field: User Agent
Operator: contains
Value: GPTBot
(repeat for each bot, OR conditions)
Action: Block

Vercel (vercel.json headers)

Vercel doesn't support User-Agent blocking natively in vercel.json. Use Vercel's Edge Middleware or the Vercel WAF (Pro/Enterprise) for reliable blocking. An alternative: Next.js middleware can return a 403 based on the incoming User-Agent header.

Apache (.htaccess)

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} GPTBot [OR]
RewriteCond %{HTTP_USER_AGENT} ClaudeBot [OR]
RewriteCond %{HTTP_USER_AGENT} anthropic-ai [OR]
RewriteCond %{HTTP_USER_AGENT} meta-externalagent [OR]
RewriteCond %{HTTP_USER_AGENT} Bytespider
RewriteRule .* - [F,L]

Keep your block list up to date

New AI crawlers appear regularly as new labs launch products. The block list in this post will be updated, but for an always-current version, see the ai-crawler-block-list repository — it tracks new crawlers and generates an up-to-date robots.txt snippet.

For monitoring whether AI crawlers are actually hitting your site — and getting alerts when new ones appear — see how AI crawlers are spiking hosting bills and the CrawlGuard waitlist below.

← All articles  ·  AI Crawler Check tool  ·  Block list repo  ·  CrawlGuard waitlist