Keeping LLM costs from quietly wrecking your margins

LLM cost control is the kind of problem that does not exist until suddenly it is the only thing finance wants to talk about. A feature ships, usage grows, and the model bill goes from a rounding error to a real line item that scales with every active user. The trap is that none of this shows up in a load test. It shows up in production, a month later, when the invoice arrives and someone asks why inference costs more than the rest of the infrastructure combined.

Know where the money actually goes

Before optimizing anything, I instrument. You cannot manage a cost you cannot see, and most teams have no idea which features, which users, or which prompts drive their spend. Token counts per request, broken down by feature and by model, are the minimum. Once I had that view on a project the picture was lopsided in the usual way: a small number of paths accounted for most of the cost, often because they stuffed huge contexts into an expensive model for a job a cheaper one could have done.

The single most useful number is cost per successful task, not cost per token. Tokens are an input. What you actually care about is how much it costs to get a user the outcome they wanted. A 'cheaper' model that fails half the time and triggers retries is not cheaper. Measuring at the task level keeps you honest about that.

Right-size the model for the job

The default move people reach for is the biggest, smartest model for everything, because it is the safest choice when you are still figuring the feature out. That makes sense early. It stops making sense once the feature is stable and you understand the actual difficulty of each request. Plenty of work in a typical pipeline is classification, extraction, or simple formatting that a small fast model handles fine at a fraction of the price.

I route by difficulty. Easy, well-defined tasks go to a cheap model. Hard, open-ended reasoning goes to the expensive one. Sometimes I use a cheap model first and only escalate when its confidence is low or a validation step fails. The routing logic costs a little engineering effort up front and then pays for itself continuously, which is exactly the trade you want.

Cut tokens before you cut models

A lot of spend is just waste in the context. Prompts grow over time as people paste in more examples and more instructions, most of which stopped earning their place long ago. Retrieval steps pull in more documents than the model needs. Conversation history gets appended forever instead of summarized. Every one of those is tokens you pay for on every single call.

Some of the cuts that have paid off the most for me:

• Trim system prompts ruthlessly and test that quality holds. Half the instructions are often dead weight.
• Cap and summarize conversation history instead of replaying the whole thing each turn.
• Retrieve fewer, more relevant chunks rather than padding the context to feel safe.
• Cache responses for identical or near-identical requests, which is free money on repetitive workloads.
• Use prompt caching where the provider supports it, so the stable part of your prompt is not re-billed every call.

Put hard limits in place

Cost optimization is ongoing, but cost disasters are sudden, and the two need different defenses. A retry loop gone wrong, an agent stuck calling itself, or a single abusive user can produce a bill that dwarfs your normal monthly spend in a few hours. I treat spend like any other resource that can run away: per-user rate limits, per-request token ceilings, and a global budget alarm that pages someone when the daily rate looks wrong.

These limits feel pessimistic when everything is calm. They are also the difference between a bad day and a genuinely expensive incident. I would rather a runaway process fail loudly against a cap than succeed quietly at burning the quarter's budget.

Revisit it, because the math keeps moving

Model prices and capabilities change fast. A routing decision that was correct six months ago may be wrong now because a cheaper model got good enough, or an expensive one dropped in price. I put a recurring reminder on the calendar to re-benchmark the main paths against current options. It is dull work and it consistently finds savings, because the ground keeps shifting under whatever you decided last time.

None of this is glamorous and none of it requires deep magic. Measure at the task level, match the model to the difficulty, stop paying for tokens you do not use, and cap the failure modes that can run away. Do that and the model bill becomes a predictable cost you manage, instead of a surprise that shows up to wreck your margins when you are least expecting it.