Your RAG Demo Worked. Your Production Retrieval Is Still Guessing.

Most RAG pilots look fine in a workshop. Someone asks a tidy question, the answer cites a tidy chunk, and the room nods. Then the same pattern hits a real queue: messy tickets, outdated PDFs, conflicting policies, and a model that sounds sure while retrieving the wrong paragraph.

That gap is not a model problem first. It is a retrieval and evaluation problem. You shipped a demo of search-plus-generation. You did not ship a measured answer path.

This piece is about treating RAG like a product with a quality bar: what to measure before you scale, how to build a golden set that reflects real work, and when to stop adding vector databases and start fixing the corpus.

Why the demo lied

Demos select for questions the corpus can answer. Production selects for the questions people actually ask under pressure.

• The pilot uses a clean knowledge base. Operations uses SharePoint folders named "final_v7_USE_THIS".
• The pilot asks one question. Production asks follow-ups that depend on prior context the retriever never saw.
• The pilot scores "sounds good". Production needs "correct enough to act", with a citation a human can check in ten seconds.

If you only evaluate fluency, every model looks competent. If you evaluate grounded correctness against a fixed set of hard cases, most early RAG stacks look unfinished. That is useful information. Treat it as a gate, not an embarrassment.

What "good" means for enterprise RAG

Pick an outcome the business already owns. Claims triage. Policy answers for contact centre. Engineering runbooks. Then define good in operational language:

1. Grounded. The answer is supported by retrieved evidence, not model memory.
2. Complete enough. Missing a critical exception is a fail, even if the tone is polished.
3. Actionable. The user can take the next step without another tool hop, or the system clearly says it cannot help.
4. Auditable. You can show which chunks were used, when, and under which version of the corpus.

Accuracy without citations is a chat toy. Citations without accuracy are theatre. You need both.

Build a golden set before you buy another platform

A golden set is a small, painful collection of real questions with expected answers or expected evidence. Fifty to two hundred cases is enough to start if they are nasty on purpose.

Source them from:

• Tickets that already burned time
• Policy questions that create rework when answered wrong
• Cases where two documents disagree and a human knows which one wins
• "I don't know" cases, so the system learns to abstain

Have the domain owner label them, not only the AI team. Record the preferred source document and the failure mode you care about (wrong doc, stale doc, partial answer, hallucinated number).

Re-run the golden set on every change to chunking, embeddings, filters, prompts, or corpus refresh. If you cannot re-run it in CI or a scheduled job, you do not have a platform. You have a hobby.

Measure retrieval separately from generation

Teams often tune the prompt while the retriever is the bottleneck. Split the scorecard.

Retrieval. For each golden question, did the top-k chunks include the evidence a human would need? Track hit rate at k, and whether the winning doc ranked first or fifth. If retrieval misses, generation cannot save you honestly.

Generation. Given the right chunks (or a controlled mix), does the answer stay faithful, refuse when incomplete, and keep the house style? This is where prompt and model choice matter.

If retrieval hit rate is weak, stop rewriting system prompts. Fix chunk boundaries, metadata filters, access control, and document hygiene first.

Corpus quality beats clever chunking

Fancy splitting will not rescue a folder of contradictory PDFs. Before you scale RAG:

• Name an owner for each source collection
• Mark authoritative versions and retire the rest
• Strip boilerplate that pollutes similarity search (nav bars, repeated disclaimers)
• Attach metadata the retriever can filter: product, region, effective date, audience

A smaller, curated corpus with dates and owners will beat a lake of everything indexed overnight. This is the same discipline as the lean architecture argument in When Your AI Architecture Becomes a Liability: shared infrastructure only earns its keep when a real use case needs it.

A ninety-day path that survives contact with operations

Days 1 to 14. Pick one workflow. Freeze scope. Build the first golden set with the business owner. Baseline the current manual answer quality and time.

Days 15 to 45. Stand up a thin RAG path: one corpus, one identity model, logging of query plus retrieved chunk ids, human review on a sample. No multi-index routing yet.

Days 46 to 90. Raise the bar until golden-set gates pass for your release criteria. Only then expand corpus coverage or add a second use case that shares the same retrieval pain.

By day 90 you should be able to answer: what is our hit rate on hard cases, what fails when we refresh documents, and who owns a wrong answer in production.

Release discipline for RAG

Treat prompt, chunker, embedding model, and corpus snapshot as deployable artifacts:

• Version them
• Run the golden set before promote
• Keep a rollback to the last known good pair of prompt plus index
• Sample live traffic weekly for new failure modes and fold them into the set

Shadow AI tools will keep answering from memory. Your enterprise path has to win by being checkable, not by being louder. That same "approved lane beats the ban" idea shows up when you bring unmanaged GenAI under control; RAG is where you make the approved lane trustworthy.

Frequently asked questions

How big should the first golden set be? Start with 50 hard, labelled cases from real work. Grow as you find new failure modes. A thousand easy questions will not protect you.

Do we need a dedicated evaluation platform? Not on day one. Spreadsheets plus a scripted runner are fine until two teams share the same harness. Buy tooling when the manual run becomes the bottleneck.

When is RAG the wrong pattern? When the answer lives in a system of record API, not in documents. Prefer structured lookups for balances, entitlements, and live status. Use RAG for policy, procedure, and knowledge that actually sits in prose.

What if the model refuses too often? That can be a feature. Tune abstention with the business owner. A clean "I cannot find a supporting source" beats a fluent wrong answer every time.