There are now more than a dozen active AI training crawlers regularly hitting websites. Some consume significant bandwidth. Most offer no benefit to site owners — no search rankings, no referral traffic, just content scraping for model training. And writing a correct robots.txt that blocks all of them requires knowing the exact User-Agent string for each one.
The free robots.txt generator handles this for you. Pick a preset, review the output, copy it. Done in under a minute.
Three reasons come up repeatedly:
AI crawlers — especially meta-externalagent and Bytespider — have been the dominant driver of unexpected hosting bill spikes for independent developers. They crawl aggressively, often hitting pages at rates that resemble a small DDoS. On a pay-per-GB hosting plan, the cost is real and immediate.
Your content feeds LLM training datasets whether you want it to or not, unless you explicitly opt out. robots.txt is the standard, documented mechanism for doing so. Major AI labs have publicly committed to honoring it — OpenAI, Anthropic, Google, and Apple all have. For well-behaved bots, it works.
Search crawlers index your content and drive traffic back to you. AI crawlers take content and surface it in chat interfaces that remove the user's need to ever visit your site. The value exchange is asymmetric. Many developers have concluded there's simply no reason to opt in.
robots.txt is a declaration of intent, not a technical barrier. The protocol works by convention: bots fetch your /robots.txt file before crawling and respect the rules it contains — if they choose to.
What it reliably does:
What it cannot do:
For the vast majority of AI crawler traffic, robots.txt is the right first step and handles the bots most likely to be causing problems.
The robots.txt generator gives you four presets:
Blocks all 13 known AI training and retrieval bots. Leaves Google Search, Bing, and other search crawlers alone.
Blocks everything except Googlebot, Bingbot, and a small whitelist of search crawlers. Maximum restriction.
Generates a permissive robots.txt. Useful as a starting point if you want to allow everything and add specific exclusions.
Toggle individual bots. Block the training crawlers you care about and allow the ones you want to permit.
Three steps:
robots.txt at the root of your siteFree tool. No account required. Covers 13 AI crawlers across 3 sections. Takes under a minute.
Open robots.txt generator →The generator covers every major AI training and retrieval crawler as of October 2026:
| Bot | Company | User-Agent | Respects robots.txt? |
|---|---|---|---|
| GPTBot | OpenAI | GPTBot |
Yes |
| ChatGPT-User | OpenAI | ChatGPT-User |
Yes |
| OAI-SearchBot | OpenAI | OAI-SearchBot |
Yes |
| ClaudeBot | Anthropic | ClaudeBot |
Yes |
| anthropic-ai | Anthropic | anthropic-ai |
Yes |
| meta-externalagent | Meta | meta-externalagent |
Partial |
| Bytespider | ByteDance | Bytespider |
No |
| CCBot | Common Crawl | CCBot |
Yes |
| PerplexityBot | Perplexity | PerplexityBot |
Partial |
| Google-Extended | Google-Extended |
Yes | |
| Applebot-Extended | Apple | Applebot-Extended |
Yes |
| Amazonbot | Amazon | Amazonbot |
Yes |
| cohere-ai | Cohere | cohere-ai |
Yes |
Google-Extended controls whether Google uses your content for Gemini and AI Overviews. It is a completely separate User-Agent from Googlebot. Adding it to your robots.txt has zero impact on your search rankings.
If you prefer to write or edit your robots.txt by hand, here are the exact directives for the bots most commonly cited in bandwidth spike reports:
User-agent: GPTBot
Disallow: /Block both User-Agents — Anthropic uses both, and the legacy one still shows up in logs:
User-agent: ClaudeBot
Disallow: /
User-agent: anthropic-ai
Disallow: /This one is most frequently associated with hosting bill spikes. Block it explicitly — don't rely on a wildcard User-agent: * rule to catch it:
User-agent: meta-externalagent
Disallow: /For a complete block list with all 13 bots, use the generator — it outputs a clean, correctly-formatted file with one click. Writing it manually at this scale introduces typo risk in User-Agent strings.
User-Agent: GPTbot (lowercase b) will not match GPTBot. robots.txt User-Agent matching is case-insensitive in the spec, but implementation varies across parsers — and more importantly, you won't get any error when a string is wrong; the bot simply isn't blocked. The generator handles the correct casing for all 13 bots.
After deploying your robots.txt, verify the live file is what you intended. Three things silently break robots.txt updates:
/robots.txt, not /static/robots.txt)The free AI Crawler Check tool fetches your live robots.txt, parses it, and tells you which major AI crawlers are currently blocked vs. allowed. It takes about 5 seconds and requires no account.
Enter your domain and the tool checks your live robots.txt against all major AI crawlers instantly. Free, no signup.
Check my site → robots.txt generator →robots.txt blocking tells crawlers to stay away. But it doesn't tell you whether they actually are — or whether new ones have appeared that aren't covered by your current rules.
New AI crawlers emerge as new labs launch products. A bot that didn't exist six months ago might be hitting your site today. Static robots.txt rules don't update themselves.
CrawlGuard monitors your site's crawler activity in real time, alerts you when new bots appear, and tracks whether blocked bots are actually staying away. It's designed for developers who want ongoing visibility, not just a one-time config.
Join the CrawlGuard waitlist →← All articles · robots.txt generator · AI Crawler Check · CrawlGuard waitlist