RAG that reaches production, not just the demo

Our AI team has shipped retrieval systems for document processing, business intelligence and content automation, and the pattern repeats: the demo takes a week, the production system takes months, and the gap is never the model. The model is the most replaceable component in the stack.

The demo lies politely

A demo answers questions you already know it can answer, over documents you just indexed, for a user who forgives. Production gets the other questions. Before any fine-tuning conversation, we build an evaluation set from real queries — including the ambiguous and the unanswerable — and measure retrieval on its own: if the right passage isn't in the context window, the generation step is decorating a wrong answer.

• Chunking follows document structure, not token counts — a table row split in half retrieves as noise.
• Metadata filters do more work than embeddings in most business corpora: date ranges, document types, access rights.
• Hybrid retrieval (keyword + vector) beats either alone on names, codes and part numbers — the queries businesses actually run.
• Ingestion is a pipeline with idempotent re-runs and failure queues, because source documents change and re-index days are real.

Answers carry receipts

Every production answer cites the passages it drew from, and 'I don't have that' is a first-class response — cheaper than a confident invention reaching a customer. Grounding failures get logged and reviewed the way crashes are, because for an AI product, they are crashes.

Successful AI products are built through reliable engineering, not experimentation alone.

None of this is glamorous. All of it is why the system still answers correctly in month six, after the corpus has doubled and the model has been swapped twice. Retrieval is a data engineering problem wearing an AI costume — treat it that way and it ships.