Right-sizing AI costs: stop paying frontier prices for grocery math

The first AI bill is cute. The sixth one, after your automation caught on and volume went up 40x, funds a difficult meeting. This is the predictable second act of every successful AI deployment, and it has a predictable cause: using one big model for everything, because that is how the prototype was built.

The industry consensus formed hard on this through 2025: as one analyst memorably put it, paying flagship rates to classify support tickets is hiring a PhD mathematician to add up your grocery bill. Production teams that implement tiered model routing routinely report inference cost reductions of 40–60% with little or no quality loss — and the price spread between model tiers keeps widening, so the prize keeps growing.

The one metric: cost per successful outcome

Before touching architecture, fix the accounting. Token prices are noise; the signal is cost per successful outcome — per resolved ticket, per qualified lead, per processed invoice. It bundles model price, retries, escalations, and failure rates into one number a CFO can love or veto.

Two systems can have identical token bills where one costs $0.09 per resolved ticket and the other $0.71 — because the cheap-looking one fails more, retries more, and escalates more. Without this metric you will "optimize" your way into worse economics.

The architecture: a triage desk, not a genius

The pattern that wins in production is boring and looks like a hospital triage desk:

Request
│
[Classifier / rules]  ← cheap, fast, sub-cent
│
├── ~70–85%: routine ────► small model (classify, extract, draft)
├── ~10–25%: complex ────► frontier model (reasoning, edge cases)
└── low confidence ──────► human queue
│
[Log cost + outcome per run]

Most real workloads are dominated by routine calls — classification, extraction, formatting, summarizing, first-draft text. Small models (the flagship vendors' own mini/haiku/flash tiers, or self-hosted options at serious volume) handle these at a fraction of flagship price. The frontier model earns its rate only on the genuinely hard slice.

Those bars are illustrative of the pattern reported across 2025–2026 production case studies — your ratios depend on your task mix. The ordering, in our experience, holds.

The four levers, in order of effort

1. Route by task, not by habit. Start with rules, not ML: task type, input length, customer tier. A config file that says "classification → small, contract analysis → frontier" captures most of the win in an afternoon.

route:
  classify_ticket:   small    # high volume, low stakes
  extract_invoice:   small    # structured, verifiable
  draft_proposal:    frontier # customer-facing, judgment
  fallback:          escalate_to_human

2. Cache your prompts. Your system prompt and few-shot examples repeat on every call. Provider prompt caching discounts that repeated prefix heavily — up to 90% on some platforms. If your prompts are long and your volume is real, this is free money.

3. Batch the non-urgent. Overnight enrichment, weekly summaries, backfills — provider batch APIs run them at roughly half price in exchange for latency you were not using anyway.

4. Cap the output. Set explicit output limits and terse formats for machine-consumed responses. Output tokens cost multiples of input tokens, and chatty JSON is pure waste.

When small models lose

Right-sizing is not "always go cheap." Small models genuinely underperform on multi-step reasoning, ambiguous instructions, and long-horizon agent work — and a cheap model that fails 15% more can be more expensive per successful outcome once retries and human cleanup are counted. That is the metric doing its job. Route down aggressively, measure honestly, and promote a task back up the tier ladder the moment its failure math says so.

The quarterly re-check

Model prices and capabilities are moving fast enough that routing tables rot. Once a quarter: re-run your eval set against current model tiers, check cache hit rates, and re-price the top three workflows. It is an hour of work that has paid for itself every single time we have done it — the cheap tier keeps getting better, and yesterday's "needs frontier" task is often today's small-model task.

Cost discipline is not the glamorous part of AI automation. It is the part that decides whether the program survives contact with the CFO. Build the triage desk, watch cost per outcome, and let the models compete for your workload.