If your LLM bill surprised you last month, it probably was not the flashy features. It was the quiet stuff you never show the user: bloated system prompts, oversized retrieval chunks, tool outputs pasted back into the model, and history that keeps getting re-injected on every turn.<\/p>\n
On one client system, 72 percent of tokens per request were invisible to the user. They were paying for embeddings to over-retrieve, a fat system message, 10 pages of doc snippets, previous 8 turns, and a verbose JSON tool response the model never needed. Everyone was trying to shorten the final answer. Wrong target.<\/p>\n
What most teams misunderstand<\/p>\n
Think in layers. A single call usually looks like this:<\/p>\n
1) Static system content
\n– Policy text, tone, tooling instructions
\n– Typical waste: 800 to 3k tokens every turn because no one separates static and dynamic parts or caches them<\/p>\n
2) Conversation state
\n– Full transcript vs compact state
\n– Typical waste: linear growth per turn when you reattach every message<\/p>\n
3) Retrieval context
\n– Top K chunks, often from page-level retrieval with large overlap
\n– Typical waste: 3k to 10k tokens because of aggressive K, no re-rank, and pasting full documents<\/p>\n
4) Tool results
\n– JSON blobs, CSV dumps, screenshots turned into captions
\n– Typical waste: 1k to 6k tokens because you returned full rows and all fields<\/p>\n
5) Model output
\n– Usually shorter than everything above, yet gets all the blame<\/p>\n
Trade-offs you should acknowledge<\/p>\n
Failure modes I keep seeing<\/p>\n
1) Put a hard token budget per request
\n– Example: 2.5k input, 400 output. Split input as 600 system, 600 history summary, 1.0k retrieval, 300 tool results
\n– If any layer breaches its budget, trim that layer first instead of truncating blindly at the end<\/p>\n
2) Stop paying for fat system prompts every turn
\n– Split system content into static core and per-request deltas
\n– Use provider prompt caching where available and keep the static segment byte-identical
\n– Remove policies that can live in code. Keep only what the model must read<\/p>\n
3) Fix retrieval before you tune prompts
\n– Use small chunks with low overlap. 600 to 1,200 tokens per chunk is a good starting band
\n– Add a cross-encoder re-ranker to cut K down to 3 to 5 chunks, not 10 to 20
\n– Extractive compression: include only quoted spans + a short header per chunk
\n– Prefer section-level retrieval over page-level. Pages inflate tokens and dilute relevance
\n– If you need summaries, compress with a cheaper model and keep links back to original spans<\/p>\n
4) Stop dumping tool results into the LLM
\n– Ask tools to return only the fields you need
\n– Cap rows. If you need to reason about aggregates, compute them in code
\n– For structured tasks, use function calling to request precise fields, then render user-facing text on the server<\/p>\n
5) Conversation state that does not grow unbounded
\n– Maintain a rolling summary of facts + decisions. Keep references to full messages off-model
\n– Store machine-readable state separately. Only pass diffs or keys, never the entire object each turn
\n– TTL older turns aggressively unless the user explicitly asks to revisit<\/p>\n
6) Output control that the model actually follows
\n– Prefer JSON schema or function calling with enums and short field names over free text
\n– Set stop sequences to cut standard boilerplate. For example, stop at \\n\\n if you only want a single paragraph
\n– Max tokens should be set. Do not let the model free run because you are scared of truncation. If the task needs more, plan multiple short calls<\/p>\n
7) Router patterns that earn their keep
\n– Use a small model for classification and task routing
\n– Only escalate to a large model when uncertainty or task type requires it
\n– Make the router cheap and deterministic. Thresholds should be tuned offline with evals, not in prod by vibes<\/p>\n
8) Make caching real, not theoretical
\n– Keep cacheable segments stable: same tool order, no timestamps, no random IDs
\n– Providers now support partial message caching. Segment prompts so that static chunks are cacheable
\n– Log cache hit rates by path. If you are not seeing 30 to 60 percent hits on static-heavy prompts, you are probably invalidating accidentally<\/p>\n
9) Tokenization discipline
\n– Measure real token counts per layer with the exact tokenizer used by the provider
\n– Strip boilerplate. Collapse whitespace and markdown noise
\n– Replace verbose labels with short keys. Replace repeated legal text with a single reference line if policy allows
\n– Avoid base64 or large inline binary-in-text. Pass references and fetch out of band<\/p>\n
10) Multimodal is a silent tax
\n– Images often explode into long captions. Downsample, crop to regions, or OCR to text if you only need fields
\n– Do not send 10 screenshots when 2 will do. One client cut 70 percent of vision tokens by cropping to the table region they actually needed<\/p>\n
11) Retry strategy with budgets
\n– Cap total tokens per user action across retries
\n– Retry with a smaller context, not the same bloated one
\n– Add loop guards for function calling. Detect repetitive tool requests and stop<\/p>\n
This pattern consistently drops input tokens by 40 to 70 percent without quality loss. In some cases, quality improves because the model is not drowning in irrelevant context.<\/p>\n
A real case from Q4:<\/p>\n
Your numbers will differ by provider, but the pattern holds. The big win is almost always input reduction, not squeezing the final answer.<\/p>\n
If your LLM bill is driven by invisible tokens and the fixes above feel doable but not trivial in your stack, that is normal. I help teams design token-aware architectures, set budgets that hold in production, and bring costs down without gutting quality. If you are running into similar issues, this is exactly the kind of thing I help teams fix when systems start breaking at scale.<\/p>\n","protected":false},"excerpt":{"rendered":"
The real problem If your LLM bill surprised you last month, it probably was not the flashy features. It was the quiet stuff you never show the user: bloated system… <\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[7],"tags":[17,15,14],"class_list":["post-76","post","type-post","status-publish","format-standard","hentry","category-ai-cost-optimization","tag-ai-cost","tag-ai-system-design","tag-llm"],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/angirash.in\/blog\/wp-json\/wp\/v2\/posts\/76","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/angirash.in\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/angirash.in\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/angirash.in\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/angirash.in\/blog\/wp-json\/wp\/v2\/comments?post=76"}],"version-history":[{"count":0,"href":"https:\/\/angirash.in\/blog\/wp-json\/wp\/v2\/posts\/76\/revisions"}],"wp:attachment":[{"href":"https:\/\/angirash.in\/blog\/wp-json\/wp\/v2\/media?parent=76"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/angirash.in\/blog\/wp-json\/wp\/v2\/categories?post=76"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/angirash.in\/blog\/wp-json\/wp\/v2\/tags?post=76"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}