The fastest way to blow your LLM budget is to keep shoving yesterday’s conversation back into the prompt on every turn. I keep seeing teams call that \u201cstateful,\u201d then wonder why latency spikes, costs drift, and answers degrade as context gets noisy. The other extreme is teams forcing \u201cstateless\u201d everywhere and then building a Rube Goldberg device of caches and retries to fake memory, which collapses the first time you have concurrency or a partial outage.<\/p>\n
This post is not theory. It is the pattern that has held up across customer support copilots, analytics agents, and workflow automation in production. The short version: keep compute stateless, keep data stateful, and make the boundary between them explicit and metered.<\/p>\n
Why it happens in real systems:
\n– People conflate product memory with model context. Not the same thing.
\n– LLM latency and cost are visible, storage and state sprawl are invisible until it is too late.
\n– Framework defaults encourage stuffing everything into the prompt because it is the fastest way to get a demo working.<\/p>\n
What most teams misunderstand:
\n– Stateless is a property of workers, not the product. You can be operationally stateless while being product-stateful by reconstructing state on demand.
\n– Memory is not a blob. There are types of state that need different stores, different lifetimes, and different consistency rules.<\/p>\n
Think in four buckets of state:<\/p>\n
1) Request state
\n– Correlation ID, tenant, auth, flags, deterministic routing input. TTL: minutes.
\n– Store: in headers and a lightweight KV like Redis.<\/p>\n
2) Session state
\n– Conversation turns and tool call results for a live session. TTL: hours to days.
\n– Store: Redis or Dynamo with versioned upserts. Summarize aggressively.<\/p>\n
3) Domain state
\n– Facts, profiles, documents, knowledge. TTL: weeks to forever, but versioned.
\n– Store: primary DB plus vector DB, both namespaced and access-controlled.<\/p>\n
4) Model policy state
\n– System prompts, routing rules, tools registry, safety config. TTL: deployed versions.
\n– Store: config service or feature flags with immutable versions.<\/p>\n
Core pattern:
\n– Workers are stateless. They build context on each request using a Context Builder service that pulls from the stores above and enforces token budgets.
\n– All writes go through an Event Log or an append-only audit trail. Readers materialize views into session and domain stores. You can replay or debug.
\n– Use idempotency keys for tool execution. Key = session_id + step_index + tool_name.
\n– Apply optimistic concurrency for session memory. Version field with compare-and-swap so two parallel turns do not clobber each other.<\/p>\n
Trade-offs and failure modes:
\n– Latency tax from state hydration. If your Context Builder hits 4 stores per request, your p95 suffers. Fix with batching and prefetch.
\n– Stale memory. Eventual consistency can surface outdated facts. For critical operations use a read-your-writes path or lock a short-lived session sequencer.
\n– Memory bloat. Unbounded transcripts and embedding every message will quietly 10x your bill. Cap, summarize, dedupe.
\n– Cross-tenant leakage. Misused vector DB filters or shared namespaces can leak content. Namespaces should be enforced at the SDK and schema level.
\n– Retry storms. Without idempotency, retries call tools again, charge external APIs twice, and double-insert state.<\/p>\n
1) Make state explicit in the API
\n– Define input contract: what state is allowed and how much of it. Anything not in the contract does not reach the prompt.
\n– Version prompts and tools. Never hot-edit a system prompt without a new version.<\/p>\n
2) Two-tier memory
\n– Tier A: short-term in Redis, TTL hours, keep last N turns and a rolling 1k token summary.
\n– Tier B: long-term in DB + vector DB. Store structured facts, not raw chat. Summarization jobs convert transcripts into facts with citations.<\/p>\n
3) Context budgeter as a gatekeeper
\n– Hard caps: max tokens per source, max chunks per retrieval, max profile size.
\n– Soft caps: decay older facts unless pinned. If you think you need 20k-token context every turn, you probably need better summarization.<\/p>\n
4) Idempotency and concurrency
\n– Every tool call has an idempotency key. On retry, you look up results and return them.
\n– Session writes use compare-and-swap with a version. If conflict, rehydrate and reapply.<\/p>\n
5) Deterministic retrieval
\n– Retrieval should be stable for the same input. Use query templates and tie-breakers. Randomness in retrieval destroys debuggability.<\/p>\n
6) Observability on state, not just model
\n– Emit state metrics: tokens_from_history, tokens_from_docs, retrieval_hit_rate, state_load_ms, state_write_ms.
\n– Track token tax ratio: tokens_from_state divided by total tokens. Keep it under 30 percent for most apps.<\/p>\n
7) Data safety by default
\n– Per-tenant namespaces everywhere. Encrypt at rest. Do not store raw PII in vector DB; store IDs and fetch the PII from the primary DB on demand.<\/p>\n
8) Output caching with a state fingerprint
\n– Cache key = normalized question + fingerprint(state_version, retrieval_ids, policy_version).
\n– Gains are huge for repeated Q&A and analytics workloads.<\/p>\n
9) Summarize with structure, not prose
\n– Summaries should be key facts plus citations and a recency score. Prose summaries drift and are hard to diff.<\/p>\n
10) Cold starts and tail latency
\n– Warm the Context Builder with connection pools and batched lookups. For chat, prefetch likely next-turn state after sending the previous answer.<\/p>\n
You do not need perfect numbers. You need orders of magnitude.<\/p>\n
Assumptions for illustration only:
\n– Input tokens cost 5 dollars per 1M tokens.
\n– Output tokens cost 15 dollars per 1M tokens.<\/p>\n
Scenario A: push full 12k-token history every turn
\n– 50k sessions per day, 6 turns per session.
\n– Input tokens from history alone: 12k x 6 x 50k = 3.6B tokens.
\n– Cost: 3.6B x 5 \/ 1M = 18k dollars per day. About 540k per month, just for repeated history.<\/p>\n
Scenario B: 1k summary + 2k fresh context per turn
\n– 50k sessions per day, 6 turns per session.
\n– Input tokens from history: 1k x 6 x 50k = 300M.
\n– Cost: 300M x 5 \/ 1M = 1.5k dollars per day. About 45k per month.<\/p>\n
That gap funds your retrieval infra and observability and still leaves money on the table. Also, shorter prompts are faster and usually more accurate because noise is lower.<\/p>\n
Latency impacts:
\n– Hydration using 4 stores at 40 ms each in p50 is not 160 ms in p95. Long tails compound. Batch, parallelize, and prefer one round trip per store.<\/p>\n
Scaling risks:
\n– Token tax grows with engagement. If you do not cap it, growth turns into margin compression. Make the budgeter a hard gate.<\/p>\n
If you are seeing token creep, flaky memory, or tail-latency spikes from state hydration, you are not alone. I help teams redesign the state boundary, put guardrails on context, and get cost and latency under control without gutting product quality. If you want a second set of eyes on your architecture or a quick cost model, reach out.<\/p>\n","protected":false},"excerpt":{"rendered":"
The fastest way to blow your LLM budget The fastest way to blow your LLM budget is to keep shoving yesterday’s conversation back into the prompt on every turn. I… <\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[3],"tags":[17,16,15],"class_list":["post-62","post","type-post","status-publish","format-standard","hentry","category-ai-architecture","tag-ai-cost","tag-ai-scalability","tag-ai-system-design"],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/angirash.in\/blog\/wp-json\/wp\/v2\/posts\/62","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/angirash.in\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/angirash.in\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/angirash.in\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/angirash.in\/blog\/wp-json\/wp\/v2\/comments?post=62"}],"version-history":[{"count":1,"href":"https:\/\/angirash.in\/blog\/wp-json\/wp\/v2\/posts\/62\/revisions"}],"predecessor-version":[{"id":82,"href":"https:\/\/angirash.in\/blog\/wp-json\/wp\/v2\/posts\/62\/revisions\/82"}],"wp:attachment":[{"href":"https:\/\/angirash.in\/blog\/wp-json\/wp\/v2\/media?parent=62"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/angirash.in\/blog\/wp-json\/wp\/v2\/categories?post=62"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/angirash.in\/blog\/wp-json\/wp\/v2\/tags?post=62"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}