Why production RAG is a distributed systems problem

The demo-to-production gap

A RAG prototype that works against 50 PDFs in a notebook will fail in ways that have nothing to do with the language model. At enterprise scale — hundreds of thousands of documents, dozens of formats, continuous updates — the failure modes are all infrastructure: stale indexes, inconsistent embeddings after model upgrades, chunking strategies that fragment critical context across boundaries, and retrieval latency that makes the product unusable.

The model is the easy part. The system around it is where production RAG lives or dies.

Ingestion is the first bottleneck

A production ingestion pipeline needs to handle format heterogeneity (PDF, DOCX, HTML, scanned images, email threads), extract structure (headings, tables, metadata), and produce chunks that preserve semantic coherence. This isn’t a batch job you run once. Documents arrive continuously, get updated, get deleted. The pipeline needs to be idempotent, version-aware, and resumable after failure.

We settled on a queue-based architecture with content-addressed deduplication. Each document gets a stable hash; re-ingestion only triggers reprocessing when content actually changes. This sounds obvious, but without it, index refreshes at scale become a source of phantom duplicates and stale retrieval results that silently degrade answer quality.

Index consistency under continuous updates

When you shard a vector index across multiple partitions for throughput, you introduce a consistency window. A document can be partially indexed — some chunks visible, others not — during a refresh cycle. If a user queries during that window, they get incomplete context, which the model then confidently synthesizes into a wrong answer.

We handle this with atomic swap semantics at the partition level: new chunks are written to a shadow segment, and the segment becomes queryable only when the full document is indexed. The tradeoff is slightly higher write latency, but retrieval never returns partial document state.

Retrieval latency is a product problem

Users expect sub-second responses. But a hybrid retrieval path — dense vector search, sparse BM25, reranking, permission filtering — can easily take 800ms+ if not carefully pipelined. We run dense and sparse retrieval in parallel, merge candidate sets before reranking, and cache hot query embeddings. Permission filtering happens post-retrieval but pre-context-assembly, which avoids the cost of embedding permission metadata into the vector space itself.

The alternative — encoding permissions into embeddings — creates an index explosion when permission structures change. It’s a common early design choice that becomes expensive to undo.

Evaluation isn’t optional infrastructure

Without retrieval evaluation, you’re flying blind. We maintain per-domain evaluation sets: known queries with expected source passages. Every deployment runs a retrieval quality check before going live — measuring recall@k, precision, and mean reciprocal rank against the evaluation set. If quality regresses, the deployment blocks.

This catches chunking regressions, embedding model drift, and index corruption before users see degraded answers. It’s the same principle as integration tests in traditional software, but applied to the retrieval layer.

The mental model shift

Teams building production RAG need to think of the retrieval layer as infrastructure — not a feature of the model. It needs monitoring, alerting, capacity planning, and graceful degradation. The model layer consumes whatever context the retrieval layer assembles. If retrieval is broken, no amount of prompt engineering will save the output.