← Blog  ·  October 2026

Vercel Log Drains: How to Track AI Crawler Costs Per Bot

Your Vercel bandwidth bill spiked. Log Drains will tell you exactly which AI crawler did it.

Vercel's bandwidth pricing starts at $0.15–$0.40/GB depending on your plan. When AI crawlers like GPTBot, ClaudeBot, and meta-externalagent hit your site, they generate requests — and those requests consume bandwidth. The problem is that Vercel's dashboard only shows you total bandwidth. It doesn't break it down by requester, bot, or User-Agent.

Vercel Log Drains solve this. They pipe every request — including every bot hit — to an endpoint you control, in real time. This guide explains how they work and how to use them to build a per-bot cost breakdown.

What are Vercel Log Drains?

A Vercel Log Drain is a webhook that Vercel calls with every log event from your deployments. This includes HTTP access logs — one event per request, with the full request metadata: method, URL, status, response size, and critically, the User-Agent header.

You configure a drain by giving Vercel an HTTPS endpoint URL. Vercel sends batched JSON payloads to that URL as your site receives traffic. You own the endpoint and decide what to do with the data.

How to enable a Log Drain

Log Drains are configured per-team in your Vercel dashboard:

1. Go to your Vercel team settings (not project settings — team-level settings)
2. Navigate to Integrations → Log Drains
3. Click Add Log Drain
4. Enter your endpoint URL (must be HTTPS)
5. Select Sources: Access Logs — this gives you HTTP request logs, which contain User-Agent
6. Choose the projects to include, or select all projects
7. Save — drain activates immediately

Vercel will start sending POST requests to your endpoint within seconds of the next deployment receiving traffic.

What the payload looks like

Each POST from Vercel contains a JSON array of log entries. Access log entries look like this:

[
  {
    "id": "01ARZ3NDEKTSV4RRFFQ69G5FAV",
    "message": "GET /api/data 200 1234ms",
    "timestamp": 1728054000000,
    "source": "edge",
    "projectId": "prj_abc123",
    "deploymentId": "dpl_xyz789",
    "host": "yourdomain.com",
    "path": "/api/data",
    "statusCode": 200,
    "duration": 1234,
    "responseBodySize": 8421,
    "userAgent": "Mozilla/5.0 ...",
    "method": "GET",
    "scheme": "https",
    "region": "iad1"
  }
]

The field you care most about for AI crawler attribution is userAgent. It contains the raw User-Agent string of the requester. responseBodySize tells you how many bytes were transferred — this is what drives bandwidth costs.

Identifying AI crawlers from User-Agent strings

Here are the User-Agent patterns for the major AI crawlers:

A simple classification function in Node.js:

const AI_BOTS = [
  'GPTBot',
  'ChatGPT-User',
  'ClaudeBot',
  'anthropic-ai',
  'meta-externalagent',
  'Bytespider',
  'PerplexityBot',
  'CCBot',
  'Diffbot',
  'YouBot',
];

function classifyUserAgent(ua) {
  if (!ua) return null;
  for (const bot of AI_BOTS) {
    if (ua.includes(bot)) return bot;
  }
  return null;
}

Calculating per-bot bandwidth cost

Once you've classified the User-Agent, you can accumulate responseBodySize per bot and convert to cost:

// Vercel Pro bandwidth pricing: ~$0.40/GB
const COST_PER_GB = 0.40;

function bytesToCost(bytes) {
  const gb = bytes / 1_000_000_000;
  return gb * COST_PER_GB;
}

// Example: accumulate over a batch of log entries
function processBatch(entries, botBandwidth) {
  for (const entry of entries) {
    const bot = classifyUserAgent(entry.userAgent);
    if (bot) {
      botBandwidth[bot] = (botBandwidth[bot] || 0) + (entry.responseBodySize || 0);
    }
  }
}

// Output
function report(botBandwidth) {
  for (const [bot, bytes] of Object.entries(botBandwidth)) {
    const cost = bytesToCost(bytes);
    console.log(`${bot}: ${(bytes / 1e6).toFixed(1)} MB — est. $${cost.toFixed(4)}`);
  }
}

Building the drain endpoint

Your drain endpoint needs to be fast — Vercel will retry if it times out. Keep it simple: receive the payload, parse the entries, store the aggregated data, return 200 quickly.

// Express.js endpoint — runs on any Node server
app.post('/api/drain', express.json({ limit: '10mb' }), async (req, res) => {
  // Respond immediately — don't await heavy processing
  res.json({ ok: true });

  const entries = Array.isArray(req.body) ? req.body : [];
  for (const entry of entries) {
    const bot = classifyUserAgent(entry.userAgent);
    if (bot) {
      await db.insert({
        botName: bot,
        url: entry.path,
        sizeBytes: entry.responseBodySize || 0,
        hitAt: new Date(entry.timestamp),
      });
    }
  }
});

A few implementation notes:

• Respond before processing. Vercel considers a drain delivery successful only when you return a 2xx. If you await heavy DB writes before responding, you'll hit Vercel's 10-second timeout on retries. Respond with { ok: true } immediately and process asynchronously.
• Authenticate the drain. Anyone who discovers your endpoint URL can send fake log data. Use a secret token in the URL (/api/drain?token=your-secret) or verify Vercel's HMAC signature if you're on Enterprise.
• Handle batching. Vercel batches multiple log entries per POST. Always treat the body as an array.
• Store aggregates, not raw rows. If you receive 100,000 requests per day, storing one DB row per request is expensive. Consider aggregating by (bot, day) in a summary table, or using a time-series approach.

What to do with the data

Once you have per-bot bandwidth data, you have several options:

1. Block the expensive bots

Add the high-cost bots to your robots.txt. Start with the ones consuming the most bandwidth. A Disallow: / directive for a specific User-Agent costs nothing and can immediately stop the bleeding.

User-agent: GPTBot
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: ClaudeBot
Disallow: /

Note: you need to block each User-Agent string separately. User-agent: * disallows paths for all bots but should only be used if you want to block everything including search engines.

2. Rate-limit at the edge

Vercel's Firewall can rate-limit by User-Agent. This is more nuanced than a blanket robots.txt block — you can allow the bot to crawl, but cap it at a bandwidth-safe rate.

3. Set an alert threshold

If a bot's daily bandwidth cost exceeds your threshold (e.g., $1/day = ~2.5 GB), you want to know. A simple daily scheduled job that queries your drain database and sends you a notification when any single bot exceeds the threshold gives you early warning before it becomes a billing crisis.

Putting it together: a minimal cost monitoring setup

Here's the full architecture for a minimal AI crawler cost monitor using Vercel Log Drains:

1. Drain endpoint — a lightweight HTTP server (Node, Python, Go) that receives Vercel drain payloads and stores per-bot bandwidth aggregates in a database
2. Vercel configuration — Log Drain pointed at your endpoint, Access Logs source selected, all projects included
3. Dashboard query — a simple query that groups bytes by bot name and converts to estimated cost
4. Alert — a daily or hourly check that fires when any bot's accumulated cost exceeds your threshold

The actual volume of data is small. Even at 100,000 requests/day, if you only store AI crawler hits (typically 1–5% of total traffic), you're looking at a few thousand rows per day — easily handled by any database.

The thing Vercel doesn't tell you

Vercel's built-in analytics show you traffic sources like "Direct", "Search", and "Social". They do not surface bot traffic separately, and they certainly don't break it down by crawler company. The only way to get bot-level attribution is through Log Drains or a third-party log analysis tool.

This means most developers on Vercel are flying blind on AI crawler costs. They see the bandwidth total and assume it's user traffic. When the bill spikes, they look at their feature changes, their deployment frequency, their user acquisition. The actual culprit — a newly aggressive AI crawler — is invisible.

Log Drains are the only Vercel-native mechanism to close this gap.