AI crawler traffic has exploded over the past 18 months. GPTBot, ClaudeBot, meta-externalagent, and Bytespider now routinely account for a meaningful share of server requests on public-facing applications — and unlike Googlebot, many of these crawlers don't respect crawl budget signals and hit dynamic routes that cost real money on serverless platforms.
There's a reasonable debate about whether blocking AI crawlers is always the right move. But we were curious about a more specific question: what do popular developer platforms do on their own sites? Do the companies selling you infrastructure to run your app — and in some cases billing you per-invocation when AI crawlers spike your function calls — block those same crawlers on their own domains?
We fetched robots.txt from 20 popular developer platforms and checked each one against seven major AI crawlers. The findings were surprisingly consistent.
We fetched https://<domain>/robots.txt directly for each platform on September 22, 2026, following redirects. For each platform-and-bot combination we looked for a dedicated User-agent stanza and applied the most-specific-match rule from RFC 9309. A bot is Blocked if it has Disallow: / in its applicable stanza; Partially Blocked if it has specific path restrictions but no Disallow: /; and Allowed if it either has an explicit Allow: /, no dedicated stanza, or the applicable wildcard stanza doesn't block the root.
Legend: ✅ Blocked · ⚠️ Partially Blocked · ❌ Allowed / Not Blocked
| Platform | GPTBot | ClaudeBot | anthropic-ai | meta-externalagent | Bytespider | CCBot | PerplexityBot | Summary |
|---|---|---|---|---|---|---|---|---|
| vercel.com | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | No AI-specific stanzas; sets ai-train=no via Content-Signal only |
| netlify.com | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | Explicitly welcomes AI crawlers; dedicated Allow: / per bot with comments |
| stripe.com | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | No AI-specific stanzas; general * rule restricts only internal paths |
| heroku.com | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | WordPress-generated file; no AI stanzas; ai-train=no via Content-Signal |
| github.com | ⚠️ | ⚠️ | ⚠️ | ⚠️ | ✅ | ⚠️ | ⚠️ | Only platform with meaningful AI restrictions. Bytespider fully blocked. GPTBot/ClaudeBot/anthropic-ai/PerplexityBot limited to marketing pages; repo content disallowed. |
| gitlab.com | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | No AI-specific stanzas; standard repo path restrictions under * |
| cloudflare.com | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | Dedicated Allow: / for GPTBot, Claude-Web, CCBot, PerplexityBot; also links to llms.txt |
| notion.so | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | No AI-specific stanzas; blocks SEO tools (AhrefsBot, SemrushBot) but not AI crawlers |
| linear.app | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | No AI-specific stanzas; only /api/ and /cdn-cgi/ disallowed |
| railway.app | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | No AI stanzas; blocks dashboard/account routes; ai-train=no via Content-Signal |
| render.com | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | Most comprehensive AI listing: 15+ bots explicitly given Allow: / with minor marketing path restrictions |
| supabase.com | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | GPTBot, ClaudeBot, Google-Extended, PerplexityBot explicitly given Allow: / |
| planetscale.com | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | User-agent: * / Allow: / with Content-Signal welcoming AI training |
| neon.tech | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | Allow: / globally; Content-Signal opts in to AI training |
| fly.io | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | Allow: / globally; Content-Signal opts in to AI training and AI input |
| resend.com | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | Allow: / globally; Content-Signal opts in to AI training |
| postmarkapp.com | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | Most detailed AI policy of any platform checked; every major bot gets its own Allow: / stanza with inline content-signal commentary |
| twilio.com | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | No AI-specific stanzas; Content-Signal sets ai-train=no only |
| sendgrid.com | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | Redirects to twilio.com/robots.txt — same policy as above |
| digitalocean.com | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | No AI-specific stanzas; only auth/login paths disallowed under * |
GitHub is the sole exception in the entire dataset. They fully block Bytespider — ByteDance's crawler behind TikTok — with Disallow: /. Every other platform, for every other bot, results in either "allowed" or at most partial path restrictions. Out of 140 combinations (20 platforms × 7 bots), that's a block rate of 0.7%.
Vercel's robots.txt includes Content-Signal: search=yes, ai-input=yes, ai-train=no — a signal saying "don't use our content to train AI models." That's a policy preference, not enforcement. There is no Disallow for GPTBot, ClaudeBot, meta-externalagent, or any other AI crawler. Vercel's serverless platform is commonly cited as a platform where AI crawlers spike your bill through function invocations they trigger — and yet Vercel itself leaves all those crawlers free to do exactly that on its own domain. The ai-train=no signal is respected only by compliant crawlers (most of the big ones claim they honor it), but there is no actual gate.
GitHub created dedicated stanzas for GPTBot, OAI-SearchBot, ClaudeBot, anthropic-ai, and PerplexityBot that allow only their marketing pages (/about, /pricing, /solutions, etc.) while disallowing all repository-level paths — commits, branches, blame, archives, raw files, and more. They went further with Bytespider and issued a complete Disallow: /. This is a notably different approach from every other platform in the list, and it's worth asking what GitHub knows about crawler behavior that others don't.
Netlify, Cloudflare, Render, Supabase, Postmark, Fly.io, Neon, and Resend are not just silent about AI crawlers — they're actively inviting them. Netlify's robots.txt opens with a comment: "Netlify welcomes AI agents and crawlers." Postmark has the most detailed AI robots.txt of any platform checked, with individual stanzas, Content-Signal annotations, and inline documentation explaining the policy for every major bot. These platforms have clearly decided that broad AI discoverability is good for their businesses.
Meta's AI crawler (used by Meta AI and related products) has no restrictions on any of the 20 platforms we checked. Not a single one. It's also frequently cited in discussions about aggressive crawling behavior. If you want to block it on your own site, none of the platforms you're likely using as infrastructure have set an example.
The data above covers platform marketing sites, not the apps running on those platforms. If you're running an app on Vercel, Railway, or Render, your robots.txt is entirely your own — and separate from theirs. The findings here are about developer platform landing pages, not hosted applications.
To audit your own site: fetch your robots.txt directly at https://yourdomain.com/robots.txt and look for specific User-agent stanzas for each bot you care about. If there's no stanza for a bot, it falls under your User-agent: * rule. If there's no User-agent: * either, every bot can crawl everything.
If you want to block a specific crawler, you need a stanza like:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: meta-externalagent
Disallow: /The tricky part is knowing what's actually visiting you and how much of your traffic — and cost — it's responsible for, before you decide whether and what to block.
Run a free robots.txt audit against all major AI crawlers, see exactly which ones have access to which paths on your site, and get a copy-paste block snippet if you want it.
Check your site → Get monitoring alerts← All articles · AI Crawler Check tool · CrawlGuard waitlist