Disclosure: I'm a Solutions Architect at DigitalOcean, so when I mention DigitalOcean products it's an informed-insider view, not a neutral one — I flag those moments clearly and keep the analysis vendor-neutral wherever the topic allows. The opinions here are my own.
Why this series exists
A few years ago, the hard part of shipping an LLM feature was getting the model to do the thing at all. In 2026 that's mostly solved — the model works on day one. The hard part is everything after: the bill that arrived 3× higher than you budgeted, the nagging question of whether you picked the right model, the latency your users feel, and the architecture you quietly committed to the first time you pasted in an API key.
I'm a Solutions Architect. I spend my days helping teams run inference in production, and the same five problems come up in almost every conversation. This series is those five — written the way I'd walk through them with you on a call, not the way a pricing page would.
Think of it as a field guide. Each part stands on its own; read the one that's biting you today. Read them in order and they tell a single story: here's what you're feeling → here's why → here's how to think about it → here's how to architect for it.
The five parts
-
Why Your LLM Bill Is 3× What You Expected — The advertised token price is a floor, not a ceiling. Output tokens, hidden reasoning tokens, non-production traffic, verbosity, and long-context surcharges compound into the bill nobody budgeted for. Where the money actually goes, and the four levers that bring it back.
-
Choosing the Right Model for Your Use Case — Model choice is the single biggest lever on both cost and quality. Establish your accuracy floor first, then find the smallest model that clears it. With deep dives into translation, RAG, code, and customer support — and why benchmarks are screening filters, not verdicts.
-
Prompt Caching in Practice — From 7% to 74% — The highest-leverage cost and latency optimization most teams under-use, and the one-line mistake that quietly kills it. Includes a benchmark I ran myself: moving a single field took cache hit rate from 0% to 99% and cut input cost 92%.
-
The Real TCO of a Full-Stack AI Application — Every comparison online compares token prices. None compares the whole stack — compute, vector DB, storage, networking, egress, ops. That's where the real number lives, with a transparent model you can re-run yourself.
-
Multi-Provider Routing Is Not a Problem — It's Your Architecture — Serious teams are multi-provider by design. Why that's the right instinct, how to route by constraint, and why "goodput" — requests that meet the SLO with a correct answer — is the metric that actually matters.
How I wrote it
Three commitments run through all five parts:
- Start from your problem, not a product. Every part opens with something you've actually felt, not a feature.
- Every number is sourced or measured. Where I could, I ran the benchmark myself rather than quote a vendor figure — the caching numbers in Part 3 and the cost model in Part 4 are mine, and the harnesses are open so you can reproduce them.
- Honest about trade-offs. Including where the tools I work on fit, and where they don't. A recommendation you can't trust isn't worth reading.
If you're a CTO staring at a surprising invoice, an AI engineer deciding what to ship, or a platform engineer designing the inference layer — this is written for you. Start anywhere.
Series — Inference in Production
- Series intro — Inference in Production: A Five-Part Field Guide (you are here)
- Part 1 — Why Your LLM Bill Is 3× What You Expected
- Part 2 — Choosing the Right Model for Your Use Case
- Part 3 — Prompt Caching in Practice — From 7% to 74% Hit Rate
- Part 4 — The Real TCO of a Full-Stack AI Application
- Part 5 — Multi-Provider Routing Is Not a Problem — It's Your Architecture
