If your team is stuck swapping models every month and your roadmap keeps slipping, you are likely chasing the wrong thing. The biggest misconception I see from leadership is thinking AI implementation is about picking the right model. In production, the model choice is rarely the bottleneck. The system around the model is.<\/p>\n
I have walked into orgs where a working PoC became a support nightmare: latency spikes, cost blowups, hallucinations in the weirdest edge cases, and no way to prove if a “fix” actually fixed anything. The model was fine. The architecture, guardrails, and feedback loops were not.<\/p>\n
Why this happens:
\n– API-first marketing makes it look easy to bolt an LLM onto anything
\n– PoCs hide the tail: single-user, curated data, no adversarial inputs, no concurrency
\n– Org structure puts AI under “innovation” instead of production engineering<\/p>\n
What most teams misunderstand:
\n– AI does not remove design constraints. It shifts them. You trade perfect determinism for coverage and speed. That requires policy, evaluation, and observability.
\n– Quality is a distribution. If you do not define the operating envelope and acceptable failure modes, you will get unpredictable outcomes at scale.<\/p>\n
Think in components. A production LLM stack that behaves:<\/p>\n
1) Policy and budget layer
\n– Route by task type, risk class, and allowed cost\/latency envelope
\n– Kill switches and circuit breakers tied to SLO breaches<\/p>\n
2) Retrieval and tools
\n– RAG with hybrid search (BM25 + embeddings) and index versioning
\n– Document quality gates, chunking strategy that matches tasks, and source provenance captured at query time
\n– Tools behind a permissioned router. Do not give the model a firehose<\/p>\n
3) Orchestrator
\n– Planner that decides single call vs multi-step tool use
\n– Structured outputs via JSON schema or function calling to avoid brittle parsing<\/p>\n
4) Verification and safety
\n– Lightweight validators: schema checks, rule-based sanity checks, domain constraints
\n– Safety filters and prompt-injection defenses before tool execution
\n– Optional LLM-as-judge for higher-cost QA on critical paths with sampling<\/p>\n
5) Observation and data flywheel
\n– Log prompts, tools, retrieved sources, token counts, latency percentiles, cost per request, and final user outcomes
\n– Golden datasets and scenario suites. Track pass@1, calibration, hallucination tags, and groundedness
\n– Shadow traffic and A\/B gating on real traffic with rollback hooks<\/p>\n
Trade-offs you need to call explicitly:
\n– RAG vs fine-tuning: choose RAG when knowledge changes often or requires provenance. Fine-tune when style or narrow pattern adherence dominates
\n– Long context vs narrow retrieval: long context looks easy and expensive. Better retrieval plus shorter prompts is usually cheaper and more stable
\n– Single-call vs agentic loops: loops amplify error and cost. Use them only where decomposition clearly improves success rate
\n– Vendor model vs open weights: vendor gives velocity and managed reliability. Open weights give control and cost predictability, but require infra maturity<\/p>\n
Common failure modes I keep seeing:
\n– Prompt bloat kills cache hit rates and inflates cost silently
\n– Index drift. You re-embed monthly while the knowledge changes daily
\n– Tail latency from tool use and retries. No timeouts. No fallbacks
\n– Silent regressions when the model or tool versions change and there is no contract test
\n– RAG returns plausible but wrong passages due to embedding mismatch with the task<\/p>\n
1) Define the operating envelope
\n– For each task, set target quality, max latency, and budget per request
\n– Write an Acceptable Failure Charter. What is allowed to be wrong, and what must be escalated<\/p>\n
2) Build a minimal but real evaluation stack in 2 weeks
\n– 200 to 500 labeled scenarios across core tasks and adversarial inputs
\n– Rubrics that measure groundedness, task completion, and safety. Do not chase generic “accuracy”
\n– Pre-merge offline eval. Canary 1% traffic with rollback. No exceptions<\/p>\n
3) Version everything
\n– Prompt, tool schema, index, and model version in logs and telemetry
\n– Ship like code. If you cannot revert a prompt in under 5 minutes, you are not ready<\/p>\n
4) Fix retrieval first
\n– Clean sources. Aggressive dedup. Domain-specific chunking. Hybrid search
\n– Store citations and confidence per segment. Penalize low-quality sources in post-ranking
\n– Re-index cadence based on content change rate, not calendar<\/p>\n
5) Control cost and latency at the edge
\n– Budget per request enforced by policy. If over budget, degrade gracefully
\n– Prompt caching keyed on normalized intent and retrieved doc hashes
\n– Truncation with utility-aware summarization, not pure token cuts
\n– Batch embeddings and tool calls where possible. Timeouts everywhere<\/p>\n
6) Human-in-the-loop where it matters
\n– Add an escalation UI for low-confidence or high-risk actions
\n– Capture corrections and outcomes to feed back into evals and future fine-tunes<\/p>\n
7) Production observability
\n– Dashboards: token cost per route, P95 latency, cache hit rate, tool failure rate, hallucination rate on sampled traffic
\n– Alerts on SLO breach tied to auto-rollback of the last changed artifact<\/p>\n
Small, concrete practices I have used:
\n– We cut cost by 37% just by removing decorative system prompts and enabling response caching on the top 30 intents
\n– Swapping from naive cosine to hybrid retrieval lifted groundedness by 12 points with no extra tokens
\n– A simple schema validator caught 70% of downstream exceptions that used to show as model “flakiness”<\/p>\n
If your team is firefighting cost spikes, silent regressions, or hallucinations that only show up with real users, you do not need another model. You need a production architecture and the guardrails to run it. This is the kind of work I do for teams when PoCs need to become revenue-grade systems. Happy to compare notes and point you at a path that fits your stack.<\/p>\n","protected":false},"excerpt":{"rendered":"
The painful truth: your AI problem is not the model If your team is stuck swapping models every month and your roadmap keeps slipping, you are likely chasing the wrong… <\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[10],"tags":[23,26,15],"class_list":["post-113","post","type-post","status-publish","format-standard","hentry","category-ai-strategy","tag-ai-deployment","tag-ai-eval","tag-ai-system-design"],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/angirash.in\/blog\/wp-json\/wp\/v2\/posts\/113","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/angirash.in\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/angirash.in\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/angirash.in\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/angirash.in\/blog\/wp-json\/wp\/v2\/comments?post=113"}],"version-history":[{"count":0,"href":"https:\/\/angirash.in\/blog\/wp-json\/wp\/v2\/posts\/113\/revisions"}],"wp:attachment":[{"href":"https:\/\/angirash.in\/blog\/wp-json\/wp\/v2\/media?parent=113"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/angirash.in\/blog\/wp-json\/wp\/v2\/categories?post=113"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/angirash.in\/blog\/wp-json\/wp\/v2\/tags?post=113"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}