If your latency is spiky, your OpenAI or self-hosted bill is creeping up, and your team keeps telling you they “turned on caching” already, I have a guess: you have a string-equality cache on the prompt and nothing else. That buys you a small win and hides the real opportunities.<\/p>\n
I keep seeing the same pattern. Teams push RAG or agent features, usage grows, then cost and tail latency bite. The fix is rarely a new model. It is proper caching across the LLM stack.<\/p>\n
Why it happens:
\n– Caches are bolted on after launch, not designed as part of the architecture.
\n– Keys are naive. They ignore system prompt versions, tool schemas, corpus versions, locale, or sampling params.
\n– Fear of staleness blocks useful caches. So teams pick the only “safe” cache – exact prompt match – which has low hit rates.<\/p>\n
What most teams misunderstand:
\n– You do not need a single cache. You need layers that match the cost profile: embeddings, retrieval, tools, final generation.
\n– Staleness is not binary. Use soft TTL with background refresh and versioned keys instead of global disables.
\n– Semantic caches can work if you gate them properly. But you must treat them like a product feature, not a toggle.<\/p>\n
Think in layers. Each layer has a different key, TTL, and failure mode.<\/p>\n
Failure mode: if you forget this, a trending query can spawn hundreds of identical LLM calls at once.<\/p>\n
Key must include at least:
\n– tenant_id and user segment (or locale) to avoid cross-tenant leaks
\n– model_id@version and sampling params digest (temperature, top_p, seed)
\n– system_prompt_digest and tool_schema_digest
\n– safety_policy_version
\n– normalized user input<\/p>\n
Normalization matters. Trim whitespace, lowercase where safe, collapse numeric formatting, sort JSON keys, freeze tool schema serialization. I have seen 10 to 20 percent hit-rate gains from just cleaning inputs.<\/p>\n
Failure modes:
\n– Model upgrade without versioned keys returns inconsistent outputs from old cache entries.
\n– Cross-locale reuse leads to wrong tone or units.<\/p>\n
Verifier options:
\n– Fast model check: ask a small model to validate reuse with a yes\/no verdict.
\n– Retrieval overlap check: run retrieval for the new query, compare top-k doc IDs with cached answer sources. Require overlap >= X%.<\/p>\n
TTL: short. Think minutes to a few hours, unless your domain is static.<\/p>\n
Failure modes:
\n– Domain drift and temporal queries. Do not reuse “What is today\u2019s rate” style questions.
\n– Hidden context. If answers depend on user profile, bake a profile_version into the key or skip semantic caching.<\/p>\n
Keys:
\n– tool_name + stable-serialized input + upstream dataset version
\n– TTL: long for slow-changing data, short for live data with its own SLA<\/p>\n
Failure modes:
\n– Assuming idempotence where it does not exist. If the tool queries a live API, treat it as non-deterministic.<\/p>\n
Sub-layers:
\n– Embedding cache: key on content_hash + embed_model_version. TTL effectively infinite until doc changes.
\n– ANN query cache: key on quantized query embedding bucket + corpus_version + retrieval_params. TTL short.
\n– Re-ranker cache: key on candidate_IDs + re_ranker_model@version.<\/p>\n
Corpus versioning is the unlock. Every ingest or update increments a collection_version. Include it in all keys. No need for expensive invalidation jobs.<\/p>\n
Failure modes:
\n– Ignoring re-ranker changes. If you switch cross-encoder versions, stale rankings hurt quality.<\/p>\n
Failure modes:
\n– Over-personalization. Keep stable templated answers and apply light post-processing for name or locale to maintain reuse.<\/p>\n
Failure modes:
\n– KV state blow-up under high concurrency. Enforce eviction policies per session and ceiling memory per GPU.<\/p>\n
Create a library that every service uses. Do not let teams hand-roll keys.<\/p>\n
Key components I include in most systems:
\n– tenant_id
\n– model_id@version
\n– system_prompt_digest
\n– tool_schema_digest
\n– sampling_params_digest
\n– query_normalized or query_embedding_bucket
\n– corpus_or_dataset_version
\n– locale and policy_version
\n– persona_or_profile_version when personalization affects content<\/p>\n
Combined, I regularly see 30 to 60 percent lower token spend and 25 to 50 percent lower p95 latency. The remaining tail usually comes from long generations or cold tool paths.<\/p>\n
Scaling risk if you skip this: at 100 QPS with 1k-token outputs, you will either melt your quota or spend a lot for nothing. Caching is the only way to bend that curve without strangling features.<\/p>\n
If this sounds familiar and you are juggling cost, latency, and correctness, I have designed and fixed these caching layers for teams running real traffic. If you want an external set of hands to make this work without six months of experiments, this is exactly the kind of thing I help teams ship when systems start breaking at scale.<\/p>\n","protected":false},"excerpt":{"rendered":"
The silent reason your LLM bill is 2x higher than it should be If your latency is spiky, your OpenAI or self-hosted bill is creeping up, and your team keeps… <\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[3],"tags":[17,24,15],"class_list":["post-97","post","type-post","status-publish","format-standard","hentry","category-ai-architecture","tag-ai-cost","tag-ai-latency","tag-ai-system-design"],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/angirash.in\/blog\/wp-json\/wp\/v2\/posts\/97","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/angirash.in\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/angirash.in\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/angirash.in\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/angirash.in\/blog\/wp-json\/wp\/v2\/comments?post=97"}],"version-history":[{"count":0,"href":"https:\/\/angirash.in\/blog\/wp-json\/wp\/v2\/posts\/97\/revisions"}],"wp:attachment":[{"href":"https:\/\/angirash.in\/blog\/wp-json\/wp\/v2\/media?parent=97"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/angirash.in\/blog\/wp-json\/wp\/v2\/categories?post=97"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/angirash.in\/blog\/wp-json\/wp\/v2\/tags?post=97"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}