Why Your LLM API Bill Is Unpredictable (And How to Fix It)

You ship a feature that uses GPT-4o. Three weeks later you have a $2,000 bill and no idea which feature caused it. This is a solvable problem — it just requires instrumentation that OpenAI doesn't provide by default.

The visibility gap in LLM billing

OpenAI's billing dashboard gives you total spend per day. Anthropic's is similar. Neither tells you which endpoint, which feature, or which user was responsible for a spike.

Compare this to how infrastructure costs work. AWS Cost Explorer lets you tag resources and see spend by tag — by service, by environment, by team. CloudWatch gives you per-Lambda invocation metrics. You can pinpoint exactly which part of your system is costing what.

LLM APIs have no equivalent. You get a daily total, a running monthly total, and a PDF invoice. That's it.

What happens when you don't monitor

How to add LLM cost monitoring today

The pattern is straightforward: wrap your LLM client calls with a function that records usage and calculates cost before returning the response.

Python (OpenAI)

from openai import OpenAI
import time

client = OpenAI()

# Prices per 1K tokens (update as rates change)
MODEL_PRICES = {
    "gpt-4o":       {"input": 0.0025, "output": 0.010},
    "gpt-4o-mini":  {"input": 0.000150, "output": 0.000600},
}

def tracked_chat(feature: str, model: str, messages: list, **kwargs):
    start = time.time()
    response = client.chat.completions.create(
        model=model, messages=messages, **kwargs
    )
    elapsed = time.time() - start
    usage = response.usage
    prices = MODEL_PRICES.get(model, {"input": 0, "output": 0})
    cost = (usage.prompt_tokens / 1000 * prices["input"] +
            usage.completion_tokens / 1000 * prices["output"])
    
    # Log to your metrics system
    record_cost(feature=feature, model=model, cost_usd=cost,
                prompt_tokens=usage.prompt_tokens,
                completion_tokens=usage.completion_tokens,
                latency_ms=elapsed * 1000)
    return response

TypeScript (Anthropic)

import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic();

const MODEL_PRICES: Record = {
  "claude-3-5-sonnet-20241022": { input: 0.003, output: 0.015 },
  "claude-3-haiku-20240307":    { input: 0.00025, output: 0.00125 },
};

async function trackedMessage(
  feature: string,
  params: Anthropic.MessageCreateParamsNonStreaming
) {
  const response = await client.messages.create(params);
  const prices = MODEL_PRICES[params.model] ?? { input: 0, output: 0 };
  const cost =
    (response.usage.input_tokens / 1000) * prices.input +
    (response.usage.output_tokens / 1000) * prices.output;

  recordCost({ feature, model: params.model, costUsd: cost,
               inputTokens: response.usage.input_tokens,
               outputTokens: response.usage.output_tokens });
  return response;
}

What to record and where to store it

At minimum, capture per call: feature name, model, prompt tokens, completion tokens, cost in USD, timestamp. This gives you the raw material to answer questions like:

• Which feature accounts for what percentage of total spend?
• What is the daily cost trend for a specific feature?
• Which model change saved the most money?
• Which users are disproportionately expensive to serve?

Storage option	Latency overhead	Query capability	Cost
Postgres (existing DB)	~2ms	Full SQL	Free if you have it
ClickHouse (self-hosted)	~1ms	Excellent for aggregations	~$20-40/mo
DataDog custom metrics	~5ms async	Good dashboards	$0.05/metric/month
Simple JSON log file	<1ms	grep + jq	Free

For most teams starting out, Postgres is the right answer. You almost certainly already have it, the schema is simple, and you can get per-feature aggregations with a single GROUP BY query.

Setting up alerts

Once you have per-feature data, alerting is straightforward. The two most useful alert types:

Daily spend threshold

Alert when any single feature exceeds a daily cost threshold. A simple cron job running once per hour that queries SUM(cost_usd) WHERE feature = 'X' AND date = today() and fires a Slack message when it exceeds your limit works well at small scale.

Cost rate spike (z-score)

Alert when a feature's cost in the current hour is more than 2 standard deviations above its 30-day hourly average. This catches runaway batch jobs and unexpected traffic spikes without requiring you to set manual thresholds for every feature.

The ongoing maintenance burden

Building this infrastructure yourself is a few days of work — and then an ongoing maintenance burden as model prices change, new models are released, and your feature set grows. You'll need to:

• Update price constants when OpenAI or Anthropic changes rates
• Add new models to the price table as you adopt them
• Maintain the storage schema as you add new tracking dimensions
• Keep the alert thresholds calibrated as your traffic scales

This is solved infrastructure. The building blocks are simple enough that most teams do build it themselves — but it's the kind of work that lives in a maintenance purgatory once it's deployed.