The uncomfortable truth about AI incidents

The scariest production incidents I have worked were not caused by a bad deploy. They were caused by a correct system producing the wrong thing. Logs were clean, tests were green, nothing obvious changed. Yet the model started fabricating policy terms, the RAG system stopped finding the right clause, or the agent looped until a timeout.

If you have ever stared at a perfect 200 OK with a useless answer inside, you know the feeling. Debugging AI is not debugging code. It is debugging behavior.

Why this happens and where it shows up

You see this in:

• RAG pipelines that intermittently miss obvious context
• Multi-step agents that work in staging but wedge in production traffic
• Safety filters that silently strip the one paragraph the answer depends on
• Fine-tuned models that drift after a data refresh

Why it happens in real systems:

• Non-determinism. Sampling, retrieval order, timeouts, retries. Same input, different path.
• Hidden state lives in the data. Index contents, chunking, reranker thresholds, cache keys, vendor model updates.
• Long dependency chains. A single response depends on model version, prompt, tools, retrieval, chunker, tokenizer, latency, policy.
• Weak contracts. Natural language in, natural language out. The contracts are soft unless you make them hard.

What most teams misunderstand:

• Prompts are not code until you version them like code.
• Evals are not unit tests unless they pin the entire context and scoring method.
• Setting temperature to zero does not remove non-determinism in retrieval or tool responses.
• Vendor upgrades are breaking changes even when the API version stays the same.

Technical deep dive: where the bodies are buried

Think in terms of an inference causal cone. Every production answer is a function of:

• Input text and metadata
• Pre-processing and normalization
• Retrieval stack. Embedding model, chunker, index, filters, reranker
• Prompt template and system policies
• Model version and decoding params
• Tool calls. API versions, schemas, latency, retries
• Post-processing and validators
• Caching and fallbacks

Common failure modes:

• Retrieval miss. Chunker changed, embeddings re-computed on a new model, reranker threshold too strict, query builder regression.
• Context truncation. Token budget overruns cut the header that defined the schema, not the body. I have seen a payroll assistant fail only on long documents because the metadata header with column units was always trimmed.
• Silent vendor change. Provider swapped a backend model under the same name. Your JSON formatting rate drops 10 percent overnight.
• Tool contract mismatch. LLM generates a field the tool ignores. You get a 200 OK with a defaulted value that propagates wrong results.
• Cache poisoning. Cache key excludes model or prompt version, so you serve stale answers after a change.
• Safety filters with side effects. Redaction removes entity anchors that your follow-up step depends on.
• Concurrency artifacts. Timeouts flip a retry path that skips reranking. Latency variance equals behavior variance.

Trade-offs you actually feel:

• Determinism vs quality. Lower temperature improves repeatability but can degrade reasoning or synthesis.
• Recall vs precision in retrieval. More docs reduce hallucination risk but increase truncation risk and latency.
• Strict schemas vs flexibility. Hard JSON schemas cut incident rate but raise refusal rates unless prompts and few-shot examples are tuned.
• Observability depth vs cost. Full traces with token logs and document snapshots speed MTTR, but storage and PII risk grow.

Practical ways to make debugging tractable

1) Version everything that changes behavior

• Prompts, system policies, few-shots
• Embedding model, chunker, tokenizer settings, reranker model and thresholds
• Query builder logic and filters
• Tool schemas and API versions
• Safety rules

Stamp these into every trace. Include content hashes of retrieved docs and the final context window.

2) Build real inference traces, not just logs

For each request capture:

• Correlation id and user cohort
• Model name, parameters, seed if supported
• Full rendered prompt and token counts, plus truncation flags
• Retrieval query, top-k with ids, scores, and doc version hashes
• Tool calls with input, output, duration, and retry count
• Validators and post-processing steps with pass or fail reasons
• Final output with schema validation result

Store at least 24 to 72 hours at full fidelity, longer for sampled traffic. Redact PII at the edge.

3) Determinism budget

• Temperature 0 or 0.2 for critical paths. Fix top_k and top_p where supported.
• Seed the sampler if the provider supports it and your compliance team signs off.
• For high-risk flows, freeze retrieval via a short-lived cache keyed by user input hash + index version + model version.

4) Replay and bisect like you mean it

• Keep a replay harness that can re-run the full stack with pinned versions and dependency snapshots.
• When an incident occurs, diff traces between a good and bad run. Look for changes in reranker decisions, truncation, or tool schema mismatches.
• Shadow traffic new versions and compare outputs with model-assisted scoring before you flip the switch.

5) Guardrails as contracts, not vibes

• Use JSON schema or function calling with strict validators. Reject and retry on schema violations with a small repair prompt.
• Add budget-aware retries. One retry with a different prompt or lower temperature is often enough. Do not loop blind.
• Circuit breakers for tools and retrieval. If a tool exceeds p95 by X ms, switch to a safe fallback or degrade gracefully.

6) Evals that reflect your product, not a leaderboard

• For RAG, measure: retrieval hit rate at k, context precision, groundedness, and final answer utility. Separate these. They fail for different reasons.
• Build a golden set from your own docs. Pin specific document ids and passages so you can detect retrieval drift.
• Model-graded evals are fine, but calibrate with human audits and track agreement. Regressions that only show up in production usually reveal eval blind spots.

7) Data pipeline hygiene

• Deterministic chunking. Changing chunk rules is a breaking change. Version and migrate.
• Embedding model migrations behind a flag. Dual-write and compare search quality before cutover.
• Indexing jobs emit a manifest with dataset version, embedding model, and date. That goes into every inference trace.

8) Incident response for AI, not just services

• On alert, snapshot a failing request including the full context and retrieved doc hashes before you try a fix.
• Maintain a prompt patch bank and a retrieval aggression toggle. These give you fast mitigations while you find root cause.
• Postmortem template includes eval gaps and new trace fields to add. Every incident should improve your observability.

Business impact you can feel in the P&L

• Cost. Without traces, teams burn engineer days reproducing sporadic failures. With full-fidelity traces, MTTR often drops from days to hours. The storage bill is smaller than one lost weekend of senior time.
• Revenue and risk. A 5 percent drop in retrieval hit rate can kill a support deflection target or produce wrong financial recommendations. Silent vendor model changes are real. If you cannot detect them in minutes, you will detect them in churn.
• Scaling risk. As traffic grows, tail latency widens and triggers more fallbacks. If fallbacks change behavior, you get emergent bugs at volume. Plan for p95 and p99, not averages.

I have watched a team ship a harmless prompt tweak that increased context length by 12 percent. At scale, this pushed more requests over the token budget, which chopped off few-shot examples. Formatting compliance dropped, which caused retries, which spiked latency, which widened truncation. It looked random until we graphed truncation rate next to error rate. Five minutes to fix once visible, two weeks to guess without traces.

Key takeaways

• Treat prompts, retrieval, and tools as versioned, testable components
• Log the rendered prompt, not just the template
• Make truncation and retrieval decisions observable
• Build a replay harness and use it to bisect incidents
• Use strict schemas and budgeted retries to stabilize behavior
• Evals must reflect your exact product and dataset, not generic benchmarks
• Expect vendor changes and detect them fast with canaries and drift monitors

If this is biting you right now

If your team is chasing intermittent AI bugs with no clear repro, you are not alone. This is exactly the kind of work I do for teams when systems start failing in ways normal dashboards do not explain. Happy to look at a trace, or help set up the observability and evals that make these problems boring again.