A developer posted to Hacker News last year that their Vercel bandwidth bill jumped from $40 to $500 in a single month. Their traffic was the same as always. Their users were fine. What changed was that several major AI companies had started crawling the entire web — aggressively, repeatedly, and by default with no bandwidth limits — and their site happened to be in the path.
This is happening more frequently. Here's why, what's different about AI crawlers versus search crawlers, and how to find out which bot is responsible for your bill.
Search crawlers like Googlebot have operated under informal norms for years. They crawl politely: they respect Crawl-delay in robots.txt, they don't hammer a server all at once, and they're indexing for a product (search) that directly benefits site owners. Google has real incentive to stay in publishers' good graces.
AI training crawlers have different incentives. They're building datasets for model training, not search indexes. They need as much text as possible, as quickly as possible, to hit training deadlines. And critically: they don't necessarily benefit site owners the way search indexing does.
The behavioral differences are significant:
The traditional assumption is: add a robots.txt rule and bots stop. For Googlebot, this largely works. For AI crawlers, the situation is more complicated.
First, most robots.txt files don't include AI crawlers at all. The standard template most developers start with has rules for Googlebot, maybe Bingbot, and a wildcard User-agent: * with paths to disallow. It was written before GPTBot existed.
Second, AI companies use multiple User-Agent strings. OpenAI crawls under GPTBot for training and ChatGPT-User for real-time browsing. Blocking one doesn't block the other. Anthropic uses both ClaudeBot and anthropic-ai. Meta uses meta-externalagent and has historically also crawled under FacebookBot.
Third — and this is the one that surprises developers — even if your robots.txt is correct, you may not know whether it's actually working. Bots don't confirm compliance. You write the rules, but unless you're monitoring your server logs and matching User-Agents against your rules, you have no idea if the requests are actually stopping.
| Bot | Company | User-Agent | Typical Aggressiveness |
|---|---|---|---|
| GPTBot | OpenAI | GPTBot |
High — full-web crawl, repeated |
| ClaudeBot | Anthropic | ClaudeBot |
Medium–high |
| anthropic-ai | Anthropic | anthropic-ai |
Medium (legacy crawler) |
| meta-externalagent | Meta | meta-externalagent |
Very high — most cited in bill spikes |
| Bytespider | ByteDance | Bytespider |
High — documented ignoring rules |
| PerplexityBot | Perplexity | PerplexityBot |
Medium |
| Google-Extended | Google-Extended |
Low–medium (respects rules) |
meta-externalagent (Meta's crawler) stands out in the data. It's consistently cited in the most severe hosting bill spikes because it runs at high concurrency and has historically been less restrained than crawlers from companies with a longer search-engine track record. It was also one of the last major AI crawlers to get widely added to block lists — meaning most sites that blocked GPTBot years ago still had open doors to Meta's bot.
The fastest way: check your robots.txt configuration against the actual list of AI crawlers that exist. Use the free AI Crawler Check tool — paste your domain and it will fetch your live robots.txt and tell you which major AI crawlers are currently allowed in.
For bandwidth diagnosis, you need server logs. On platforms that give you log access:
# Find top bandwidth consumers by User-Agent in nginx/Apache logs
cat access.log | grep -oP '"[^"]*"$' | sort | uniq -c | sort -rn | head -20
# More targeted: find all AI crawler requests
grep -E "(GPTBot|ClaudeBot|anthropic-ai|meta-externalagent|Bytespider|PerplexityBot)" access.log | wc -l
# Bandwidth estimate per bot (approximate, bytes-based)
grep "GPTBot" access.log | awk '{sum += $10} END {print sum/1048576 " MB"}'On Vercel: Vercel doesn't expose raw access logs on most plans. You can instrument your Next.js middleware to log User-Agents to an external service, or use Vercel's Firewall (Pro plan) which does show bot traffic breakdowns.
On AWS CloudFront: enable access logging to S3, then query with Athena using the standard CloudFront partition schema. Filter on cs-user-agent.
Adding rules for the major AI crawlers takes five minutes and can meaningfully reduce bot traffic. At minimum, add these to your robots.txt:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: anthropic-ai
Disallow: /
User-agent: meta-externalagent
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Google-Extended
Disallow: /After updating, use the crawl-check tool to verify the rules are actually present and parseable in your live robots.txt. The most common mistake is updating the wrong file, having a caching layer serve the old version, or a subtle robots.txt syntax error that causes parsers to skip the rule.
For a complete reference guide to robots.txt AI crawler blocking — including a full per-bot table, notes on compliance, and every major User-Agent string — see How to Block AI Crawlers in robots.txt.
The free AI Crawler Check tool fetches your live robots.txt and tells you which major AI crawlers currently have access to your site — in about 5 seconds.
Check your robots.txt → Get CrawlGuard monitoring← All articles · AI Crawler Check tool · CrawlGuard waitlist