The demo worked. The production launch didn’t.

The pattern is predictable. The RAG demo looks great in a room with five people. Then you hit 200 to 800 QPS and everything wobbles. Tail latency explodes, answers drift, the vector database gets “mysteriously” slow, token usage triples, and someone suggests a bigger instance class like that is the system design.

I have been pulled into too many incidents where the root cause was not a single bug. It was a stack of small design choices that only fail when real users show up.

Where RAG breaks and why

• Latency spikes at P95 and P99
  • ANN recall tuned for accuracy in a PoC turns into CPU-bound graph walks at scale. Filters tank performance because the index was not built with filter-aware partitioning.
• Token blowups
  • topK multiplied by aggressive chunk sizes and generous overlap quietly turns into 4x context size. Your LLM bill tracks it.
• Rate limits cascade
  • One provider throttles. Your retries amplify load. A single slow dependency backpressures the whole pipeline.
• “It worked yesterday” retrieval drift
  • Silent embedding version bumps or index merges change scores. No one notices until accuracy slides.
• Caches miss when you need them most
  • Query-variant noise destroys hit rates. Rerank results are not cached at all. Hot paths stay hot.

Why this happens:

• PoC designs are optimized for single-user accuracy, not concurrency. Every step is serial, uncapped, and stateful.
• Teams treat the vector DB like a magic search engine. ANN parameters, filtering strategy, and memory residency matter a lot under load.
• No stage budgets. If retrieval decides to burn 800 ms today, everyone else pays.
• Reranking and tool calls get bolted on later. Fan-out grows by accident.

What most teams misunderstand:

• RAG is a system, not a component. You cannot fix it with “better embeddings” alone.
• Tail latency is a design input, not a metric you look at after the launch.
• The correct topK is a per-query decision with a latency budget, not a constant.

How RAG actually fails under load

Let’s walk the pipeline the way a request experiences it.

1) Query normalization and embedding

• Failure mode: per-request embedding on CPU with no batching. At 300 QPS you saturate cores and add 150 to 400 ms.
• Failure mode: cross-region embedding API. Add 60 to 120 ms of network tax for no reason.
• Failure mode: silent embedding model upgrade. Recall shifts, reranker works harder, answers drift.

Trade-offs:

• Self-host a small, fast embedding model and batch. Or pay API tax and cap QPS with aggressive caching. You cannot have low latency and no control over rate limits without a plan.

2) Vector retrieval

• Failure mode: topK set to 40 “for quality.” Chunks are 600 tokens with 150 overlap. You just added thousands of tokens to context. P95 moves by seconds.
• Failure mode: HNSW ef_search cranked up to hit offline recall. Works at 10 QPS. At 400 QPS, CPU pins and queries start queueing.
• Failure mode: filters on high-cardinality fields without pre-partitioned shards. ANN graph walks most of the index then fails late on the filter.
• Failure mode: memory misses to disk. A single cold shard spikes to 1 to 2 seconds.

Trade-offs:

• You can target high recall or predictable latency. Production systems pick a recall floor with ef_search and K that fit a 150 to 250 ms budget, then use a reranker.

3) Dedup and rerank

• Failure mode: cross-encoder on 100 candidates. Reranker is accurate and slow. Tail latency becomes reranker-bound.
• Failure mode: no early cut. You rerank near-duplicates from aggressive chunking and overlap.

Trade-offs:

• Cap rerank candidates to 20 to 50. Use MMR or simple BM25 pre-cut to remove redundancy. Cache rerank results by query hash and candidate set.

4) Synthesis

• Failure mode: context stuffing. All retrieved chunks go in, even the weak ones. Token count and hallucination both go up.
• Failure mode: provider rate limits at P95. Your retries stack, the queue grows, then your autoscaler reacts too late.

Trade-offs:

• Structured prompts with explicit instructions for citation and refusal. Compression step for long contexts. Model choice by route, not global default.

5) Caching and orchestration

• Failure mode: reply cache only. Retrieval is recomputed every time. Hot questions never get cheaper.
• Failure mode: no admission control. Spikes take the system down instead of letting some requests degrade.

Trade-offs:

• Multi-layer caching with invalidation hooks and TTLs tied to content updates. Circuit breakers and load shedding. Degrade modes that keep the product usable.

Practical fixes that work

These are the design moves that consistently make RAG reliable at scale.

Set stage budgets and enforce them

• Define an end-to-end SLO like P95 2.5 s, and assign budgets:
  • Retrieval 200 ms
  • Rerank 120 ms
  • Synthesis 1.8 s
  • Everything else 380 ms
• Timebox each stage. If retrieval hits 200 ms, return what you have. If rerank hits 120 ms, stop and continue. This alone removes the worst tails.

Move to two-stage retrieval

• Stage 1 recall: cheap lexical or sparse retrieval to 100 to 200 candidates within 80 to 120 ms. BM25 or SPLADE are fine. Keep it filter-aware.
• Stage 2 precision: vector ANN on those candidates with MMR and metadata scoring, K between 8 and 16.
• Result: fewer tokens, higher precision, and predictable latency. In one migration we cut topK from 20 to 8, dropped P95 by 600 ms, and increased judged relevance by 7 points.

Fix chunking and metadata

• Chunk by boundaries users care about. Sections or headings at 300 to 600 tokens. Overlap 10 to 15 percent max.
• Keep doc_id, section_id, and page in metadata. Use these to dedup before rerank.
• Precompute short abstracts per section. Rerank on abstracts, expand to full text only for synthesis. This cuts rerank FLOPs a lot.

Treat embeddings like a versioned dependency

• Pin model versions. Store version in the index. Dual-write when you upgrade and run a shadow index until metrics are stable.
• Batch and quantize. Small 384 to 768 dim models with FP16 or INT8 are fine for most enterprise RAG.
• Keep embedding service in the same region as the vector DB and application.

Tune your vector database for your workload

• Partition by tenant or security domain to keep working sets hot.
• Pre-filter with scalar conditions, then ANN on the reduced set. Or build filter-aware shards.
• Set ef_search dynamically based on in-flight load. Cap it under pressure.
• Measure memory residency. If your hot shards are not fully in RAM, you are choosing inconsistency by design.

Rerank sanely

• Use a small cross-encoder (e.g., MiniLM). Cap candidates to 20 to 50. Cache results keyed by query hash + candidate ids.
• Early dedup by doc_id and section. You rarely need three near-identical chunks from one page.

Control tokens

• Hard budget the context. If the context exceeds N tokens, compress or drop the weakest chunks. Do not bump the model context window as a fix.
• Strip boilerplate. Logging, legalese, and repeated instructions sneak into contexts and cost real money.

Caching that actually helps

• Query normalization cache: lowercase, strip punctuation, collapse whitespace, light synonym map.
• Retrieval cache: key on normalized query + filter set + embedding version. TTL by content freshness. Invalidate on doc updates.
• Rerank cache: key on query + candidate ids. Small and high hit rate on trending queries.
• Answer cache: for truly repeated questions. Store citations so you can audit.

Orchestration and resilience

• Put deadlines on every call. Retries with backoff and jitter. Circuit breakers for providers and the vector DB.
• Admission control with priority queues. Let low-priority traffic degrade instead of taking everyone down.
• Hedged requests for flaky providers if you can afford the duplicate cost at the tail.

Observability you will actually use

• Per-stage latency and error budgets with P50, P95, P99. Emit ef_search, K, candidate counts, and token counts as first-class metrics.
• Distributed traces tagged with query_id, tenant_id, embedding_version, index_alias.
• Quality telemetry: clickthrough on citations, doc coverage, and rerank win rate. These are your recall proxies.
• Content lineage: which index, which doc version, which chunk ids were used for an answer. You will need this in incident reviews.

Load test with the right shape

• Use real query distributions with long-tail and bursty traffic. Flat RPS tests lie.
• Soak for hours. Watch memory growth, index compaction, and eviction behavior.
• Run chaos: kill a vector DB node during a burst. Verify your degradation path works.

Business impact you can measure

• Cost: topK 40 vs 8 on 600-token chunks can turn a 1k-token prompt into 4k to 6k. That is a 3x to 5x jump in LLM cost plus slower answers. Teams feel this in the bill and the NPS.
• Reliability: one provider rate limit without admission control often creates a 5 to 10 percent error spike. With budgets and degradations it becomes a 1 to 2 percent soft quality dip that most users never see.
• Throughput: moving embedding to a batched service in-region routinely cuts 100 to 200 ms and frees 20 to 40 percent CPU for application threads.
• Accuracy: two-stage retrieval with light rerank tends to beat brute-force vector with high K, because you remove redundancy and keep the model focused.

Key takeaways

• Treat RAG like a system. Stage budgets, deadlines, and backpressure are non-negotiable.
• Pick a retrieval strategy that fits a 150 to 250 ms budget. Two-stage beats brute-force vector for most products.
• Control topK and chunking or your token bill will control you.
• Version your embeddings and indices. Dual-write and shadow deploy upgrades.
• Cache the steps that are expensive and stable. Retrieval and rerank caches pay back fast.
• Tune your vector DB with the same rigor you tune a primary datastore. Memory residency and filter strategy decide your P95.
• Build observability around query distribution and document lineage, not just API errors.
• Design degrade paths before you need them. The first spike will come at the worst moment.

If this sounds familiar

If your RAG starts to fray under load, you are not alone. The fixes are mostly architectural and they compound. If you want someone to audit the pipeline, set budgets, and tune the retrieval stack without burning three quarters, this is exactly the kind of work I do when systems start breaking at scale.