The painful pattern

A team ships a slick internal demo. It answers questions, writes code, summarizes PDFs. The room nods. Then you wire it to real data, real users, real SLAs, and it crumbles. Latency spikes, answers drift, permissions leak, costs explode, adoption stalls. I have watched this movie in banks, SaaS, and logistics. The trailer is great. The film is unwatchable.

If this sounds familiar, you are not cursed. You are seeing the gap between a prompt-level prototype and a production system that has to respect data shape, workflows, governance, and unit economics.

Where the gap shows up and why

• RAG assistants that nail a curated question set but fail on everyday queries because retrieval recall is weak and the corpus changes hourly
• “Autonomous agents” that complete happy-path tasks, then spin in tool loops when APIs timeout or return partial data
• Summarization that looks smart on one doc, then hallucinates on template-heavy docs or mixes tenants
• Code assistants that feel magical in staging and then trip on repository permissions, monorepo scale, or latency budgets

Why this happens in real systems:

• Demos hide non-functional requirements. The happy path ignores latency, cost ceilings, tenancy, auditing, or failure handling.
• Retrieval and data plumbing are harder than model calls. Most “LLM quality” complaints are retrieval, chunking, and ranking problems wearing a different jacket.
• Misaligned evaluation. Teams use vibe checks instead of measurable rubrics, so they ship uncertainty.
• No safety rails. There is no permissions model, no degradation strategy, no timeouts, no canaries.
• Unit economics are an afterthought. The demo uses the largest model, no caching, full transcripts, and generous context.

What most teams misunderstand:

• Model choice is rarely the bottleneck. Data quality, retrieval, routing, and product UX do more work than jumping from Model A to Model B.
• “Value” is not accuracy in isolation. Value is, roughly: value per query = success rate × savings or revenue per success − cost per query. Aggregate across volume and you get a P&L, not a vibe.
• Production AI is systems engineering. If you would not ship a backend service without SLOs, canaries, metrics, and runbooks, do not ship an LLM without them.

Technical deep dive: design it like a system, not a prompt

Architecture that actually survives production

Think in layers, each with budgets and fallbacks:

1. Request shaping
   • Intent typing and policy checks up front
   • Budget assignment per intent: max tokens, max latency, max spend, allowed tools
   • Early exits for unsupported or risky requests
2. Grounding and retrieval
   • Hybrid search: lexical + dense. Dense alone is brittle on numeric, legal, or code tokens.
   • Chunking by structure, not arbitrary tokens. Preserve headings, tables, and lists. Use windowed overlap only where structure is weak.
   • Reranking with a small cross encoder. Rerank 50 to 5. Gains are large and predictable.
   • Permissions at index time when possible. Store ACLs with vectors and filter before scoring. Query-time ACL joins are slow and leaky.
   • Freshness paths: streaming updates for hot collections, batch for cold. Track index lag as a first-class metric.
3. Orchestration and tools
   • Router to choose small model vs large model, RAG vs task-specific tool, based on intent and confidence
   • Deterministic tools for deterministic needs: lookups, math, policy checks. Do not let the model “reason” about exchange rates.
   • Timeouts, retries with jitter, and circuit breakers on every tool call. No infinite agent loops.
4. Generation
   • Short, structured prompts. Use schemas and constrained decoding where possible.
   • System prompts versioned and tested. No silent edits in prod.
   • Summaries built from extracted facts, not free-form synthesis, in high-stakes workflows.
5. Evaluation and safety
   • Confidence scoring that is explainable: combine retrieval scores, coverage checks, and self-check prompts with thresholds per intent
   • Refusal and human handoff when confidence is low or policy triggers
   • Red teaming and jailbreak tests in CI, not once at launch
6. Observability and control
   • Metrics: p50/p95 latency per stage, retrieval nDCG@k, tool success rate, cache hit rate, cost per request, refusal rate, escalation rate
   • Tracing across the chain: request ID flowing through router, retriever, reranker, tool calls, LLM
   • Feature flags and canary percentages per intent

Common failure modes I keep seeing

• Retrieval mismatch
  • Embeddings trained on generic corpora fail on code tokens, legal cites, and SKU identifiers
  • Chunking by size instead of structure breaks answers that need table- or section-level coherence
  • Vector store filters that do not match your real ACLs, causing cross-tenant leaks under pressure
• Prompt state and token bloat
  • Unbounded chat history with every turn doubles your bill and latency, then truncates the one message you needed
  • Multi-tool agents carry forward irrelevant context and create compounding hallucinations
• Latency and reliability traps
  • Reranker hosted in a different region adds 300 ms to every call
  • Tool timeouts set longer than your SLO, so user-facing latency is guaranteed to breach
  • Vendor failover not tested. When the primary LLM blips, everything 500s
• Evaluation theater
  • One polished golden set that the team memorizes. No blind sets, no failure clustering
  • Judge LLM scoring without human calibration, so the metric looks stable while user trust drops

Evaluation that predicts production

Build an eval harness before the launch party:

• Goldens and blinds
  • Curate goldens from real logs. Create blind sets monthly from fresh traffic. Freeze them.
• Rubrics per intent
  • Exactness where needed, semantic match where flexible, policy adherence everywhere. Use structured rubrics, not 1 to 5 vibes.
• Multi-judge with calibration
  • LLM judge plus 10 to 20 percent human spot checks. Track agreement. If judge drifts, halt rollouts.
• System-level metrics
  • Retrieval nDCG@5, answer support rate (% answers supported by a cited source), refusal accuracy, escalation precision
• Cost and latency curves
  • Plot quality vs cost across models, prompts, and k values. Force trade-off decisions with data.
• Safety and jailbreak suite
  • Red team prompts in CI. PII leakage, policy violations, prompt injection. Fail the build if it regresses.

Practical fixes that close the gap

Start with a value function, not a model name

Write the unit economics on one line and hold the team to it:

• Target: reduce support handle time by 25 percent at p95 latency under 1.2 seconds, with cost per resolved ticket under 20 cents, refusal rate under 5 percent
• Or: increase self-serve resolution by 15 points with deflection quality above 0.8 rubric score and zero cross-tenant leaks

Design for budgets

• Latency budget by stage: retrieval 150 ms, rerank 120 ms, generation 300 ms, tools 300 ms total. Leave headroom.
• Token budget: fixed system prompt under 400 tokens, dynamic context under 1,200, output under 300. Trim ruthlessly.
• Cost ceiling: set per-intent ceilings and hard stop when exceeded. Log and refuse gracefully.

Retrieval that holds up

• Use hybrid search and a reranker. You can get 10 to 20 points of recall without touching the base LLM.
• Index with structure
  • Segment docs by headings, keep tables as blocks, store section paths as metadata. Apply 20 to 40 token overlaps only when structure is weak.
• Permissions baked in
  • Store ACLs in the index and filter pre-score. If you must do query-time joins, cache group membership aggressively.
• Freshness SLO
  • Track index lag. If lag exceeds threshold, annotate answers with staleness warnings or refuse.

Routing and model mix

• Use a small model for intent, safety, and simple Q&A. Escalate to a larger model only when confidence is low or task is complex.
• Deterministic paths for deterministic needs. Price lookups, policy checks, data retrieval should not go through creative models.
• Cache everything you can
  • Request normalization plus semantic cache for repeated questions
  • Chunk-level cache for reranker features
  • Prompt template cache with versioning

Confidence and refusal that users trust

• Confidence score = g(retrieval scores, coverage of required fields, self-check vote) with per-intent thresholds. No mystery numbers.
• If below threshold, show sources, ask a clarifying question, or escalate. Do not guess.

Rollout like you mean it

• Dark launch with shadow traffic. Compare side by side with your baseline.
• Canary by intent and tenant. Ramp on low-risk intents first.
• Alerts on cost and quality. If cost per successful outcome spikes or answer support rate drops, auto rollback.
• Runbooks
  • LLM vendor outage: failover path tested weekly
  • Vector store lag: switch to lexical only with stricter refusal logic
  • Tool degradation: disable long-running tools and rewrite prompts to direct users to safe paths

When not to use RAG or a chat UI

• If the task is form-like, build a form. Then use the model behind the scenes for extraction or classification.
• If ground truth is a database, call the database. Generate only the explanation layer.
• If policy must be exact, encode the policy and let the model do retrieval of clauses, not the decision.

Business impact: what changes on the P&L

Cost

Per-request cost is predictable if you measure it:

• cost = LLM input tokens × $/token + LLM output tokens × $/token + reranker calls + vector queries + tool calls + egress
• Biggest levers
  • Shorter prompts and tighter context
  • Mixture of models with routing
  • High cache hit rates on common queries
  • Rerank instead of asking the LLM to sort long contexts

I have seen teams cut cost per answer from 60 cents to 12 cents by moving from a single large model to small-model routing, adding reranking, and trimming context by 50 percent with no quality loss.

Performance and adoption

• p95 latency is the trust killer. If users wait 3 seconds unpredictably, they stop using it. Get to a stable sub 1.5 seconds p95 for most internal tools. External user tolerance is lower.
• Adoption tracks refusal and correction rate more than “wow” moments. If your refusal is honest and your corrections are quick, usage grows.

Scaling risks

• Vendor concentration. If 90 percent of queries hit a single frontier model, you have pricing and outage risk. Build a second source now, not during an incident.
• Data governance blast radius. Poor ACL design turns a minor bug into a board issue. Index-time filtering plus tenant-level canaries reduce risk.
• Opex creep. As features accrete, token use drifts up. Budget guards and cost tests in CI prevent slow bleed.

Key takeaways

• Demos optimize for delight. Products must optimize for latency, cost, permissions, and failure handling.
• Retrieval quality beats model upgrades for most enterprise use cases.
• Route to the smallest model that can do the job, and keep deterministic work deterministic.
• Build an eval harness with goldens, blinds, calibrated judges, and safety tests before GA.
• Set explicit budgets for latency, tokens, and spend per intent. Enforce them with gates and alerts.
• Confidence and refusal logic are product features, not afterthoughts.
• Roll out with canaries and have runbooks for the three outages you will see.

If you are stuck in the demo trap

If this resonates and your prototype is not turning into measurable lift, I can help. I work with teams to restructure retrieval, set budgets, design routing, and build the eval and observability you need to ship with confidence. This is exactly the kind of work that gets systems out of slideware and into production without blowing up cost or trust.