← Blog  ·  October 2026

How to Block AI Crawlers With robots.txt (GPTBot, ClaudeBot, and More)

AI companies are crawling your site to train their models. Here's exactly how to stop them.

AI training crawlers operate differently from search engine bots. Googlebot crawls to index your content so users can find it — which benefits you. AI training crawlers crawl to ingest your content into datasets used to train commercial models — with no reciprocal benefit unless you've licensed the content. Several major AI companies now provide dedicated User-agent strings for their crawlers, which makes it straightforward to block them via robots.txt.

This article covers the current list of known AI crawler User-agents, the exact robots.txt syntax to block each, and how to verify your rules are working.

How robots.txt blocking works

The robots.txt file lives at the root of your domain (https://yourdomain.com/robots.txt) and uses a simple directive syntax. Each rule block starts with a User-agent line identifying which crawler the rule applies to, followed by one or more Disallow directives specifying which paths to block.

To block a crawler from your entire site:

User-agent: GPTBot
Disallow: /

To block a crawler from specific paths only:

User-agent: GPTBot
Disallow: /private/
Disallow: /api/

The full AI crawler list (with current User-agents)

Note: Googlebot itself (the standard search index crawler) is separate from Google-Extended. Blocking Google-Extended stops Google from using your content for AI training while preserving normal search indexing.

The complete robots.txt block — copy and paste

Add this to your robots.txt file to block all major AI training crawlers:

# Block AI training crawlers

# OpenAI
User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

# Anthropic
User-agent: ClaudeBot
Disallow: /

User-agent: anthropic-ai
Disallow: /

# Meta
User-agent: meta-externalagent
Disallow: /

User-agent: FacebookBot
Disallow: /

# Google AI (does NOT affect standard Googlebot search indexing)
User-agent: Google-Extended
Disallow: /

# Perplexity
User-agent: PerplexityBot
Disallow: /

# ByteDance
User-agent: Bytespider
Disallow: /

# Common Crawl (open dataset used to train many AI models)
User-agent: CCBot
Disallow: /

# Amazon
User-agent: Amazonbot
Disallow: /

# Apple AI
User-agent: Applebot-Extended
Disallow: /

# Cohere
User-agent: cohere-ai
Disallow: /

Blocking all AI crawlers in one directive

If you want a more compact format and are comfortable blocking any unrecognized bot, you can use a catch-all and then selectively allow known good bots. This is more aggressive but cleaner:

# Block all crawlers by default
User-agent: *
Disallow: /

# Allow standard search engines
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

User-agent: DuckDuckBot
Allow: /

User-agent: Slurp
Allow: /

Blocking specific sections instead of the whole site

If you publish content you want indexed by search engines but have sections you don't want in AI training datasets — documentation, forums, user-generated content, premium content — you can block those paths specifically:

User-agent: GPTBot
Disallow: /docs/
Disallow: /forum/
Disallow: /users/

User-agent: ClaudeBot
Disallow: /docs/
Disallow: /forum/
Disallow: /users/

This is useful for content platforms that want Google to index their content for SEO but don't want that same content used for AI training.

How to deploy your robots.txt update

Static sites (GitHub Pages, Netlify, Vercel)

Add or edit robots.txt in the root of your repository and push. The file will be served at https://yourdomain.com/robots.txt automatically.

# Verify it's live
curl -s https://yourdomain.com/robots.txt | grep -A2 "GPTBot"

Next.js / Node.js

In Next.js 13+, create app/robots.ts or place a static robots.txt in the public/ directory.

// app/robots.ts
import { MetadataRoute } from 'next'

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      { userAgent: 'GPTBot', disallow: '/' },
      { userAgent: 'ClaudeBot', disallow: '/' },
      { userAgent: 'Google-Extended', disallow: '/' },
      // add remaining bots
      { userAgent: ['Googlebot', 'Bingbot'], allow: '/' },
    ],
    sitemap: 'https://yourdomain.com/sitemap.xml',
  }
}

WordPress

Use the Yoast SEO or Rank Math plugin's robots.txt editor (SEO → Tools → File editor), or edit robots.txt directly in your web root if it exists as a static file. Note: WordPress generates a virtual robots.txt if a physical one doesn't exist — check which applies to your installation.

Verifying that your blocks are working

After deploying, confirm two things: the file is accessible and the directives are correct.

# Check file is accessible
curl -s https://yourdomain.com/robots.txt

# Verify GPTBot is blocked
curl -s https://yourdomain.com/robots.txt | grep -A2 "GPTBot"
# Expected output:
# User-agent: GPTBot
# Disallow: /

You can also use our free crawl-check tool to verify which bots are allowed or blocked on your site, and to see the full parsed robots.txt as different crawlers would interpret it.

What about meta tags?

For pages where you want AI-crawler-specific controls without modifying robots.txt, you can use the X-Robots-Tag HTTP header or a <meta name="robots"> tag:

<!-- Prevent AI crawlers from using this page for training -->
<meta name="robots" content="noai, noimageai">

The noai directive is recognized by some AI crawlers to signal that the content should not be used for training, even if the crawler is allowed to access the page for other purposes. Adoption is currently inconsistent across providers — robots.txt remains the most reliable method.

← All articles  ·  Crawl checker  ·  CrawlGuard