Why RAG Systems Fail in Production — and How to Diagnose Them

A production RAG system is a pipeline of discrete stages, and each stage can fail independently. Ingestion converts source documents into chunks, embeds them and writes them to a vector store. Retrieval takes a query, embeds it and fetches the top-k nearest chunks. Reranking reorders those chunks by relevance. Context assembly selects and formats the chunks into the prompt. Generation passes the assembled prompt to the LLM and produces the answer. Evaluation scores the answer against a quality rubric. A failure at any stage propagates downstream: bad chunks produce bad embeddings, which produce bad retrieval, which produces a bad prompt, which produces a bad answer.

The mistake teams make is treating 'the answer is wrong' as a single failure. It is not. A wrong answer can originate at ingestion (the chunk that contained the answer was split in half and lost the critical sentence), at retrieval (the right chunk existed but was not in the top-k), at reranking (the right chunk was retrieved but demoted below the prompt limit), at context assembly (the right chunk was in the prompt but the model attended to the wrong part), or at generation (the model had the right context but hallucinated anyway). Each cause has a different fix, and the fixes are not interchangeable. Fixing the chunking strategy does nothing if the real problem is the reranker.

The diagnostic workflow starts by naming the layers and instrumenting each one. Every query should produce a trace: the query text, the retrieved chunks with their scores and ranks, the reranked order, the assembled prompt, the model response and the evaluation score. Without per-layer traces, diagnosis is guesswork — the team changes chunking, then retrieval, then the prompt, hoping something works. With traces, the team can see exactly which layer produced the failure and fix that layer. The LangChain community's trace-to-fix practice is that a trace is the first thing to collect when a RAG system fails.

The single most important diagnostic question for a RAG system is: did the retrieval return the right documents, and did the generation use them correctly? These are two separate failures with two separate fixes. If the retrieval returned the right documents but the answer is wrong, the problem is in generation — the prompt, the model, or the context formatting. If the retrieval did not return the right documents, the problem is upstream of generation, and no amount of prompt engineering will fix it. Conflating the two wastes time: the team tunes the prompt when the real problem is the chunking strategy.

The separation is made by inspecting the retrieved chunks for a failed query. If the correct answer is present in one of the retrieved chunks, retrieval succeeded and the failure is in generation. If the correct answer is not in any retrieved chunk, retrieval failed — and the next question is whether the correct chunk exists in the index at all. If it exists but was not retrieved, the embedding or search is the problem. If it does not exist, the ingestion or chunking is the problem. This three-way split — retrieval failed vs generation failed vs the chunk does not exist — narrows the fix from 'something in the pipeline' to a specific layer and a specific action.

The comparison table below maps the two failure classes to their signals, causes and fixes. The key operational discipline is to always check retrieval first: pull the retrieved chunks for the failing query, read them, and judge whether the evidence is there. Only when the evidence is present but the answer is wrong should the team move to generation diagnosis. This order prevents the common anti-pattern of endlessly tuning prompts for a retrieval problem that no prompt can solve.

Ingestion and chunking failures are the most insidious because they are invisible at query time. The system returns an answer, the answer is wrong, and the team spends hours tuning retrieval and prompts — but the real problem is that the chunk containing the answer was never created, or was created in a way that made it unsearchable. A fixed-size chunker that splits a paragraph mid-sentence can separate a condition ('only for customers in India') from the claim it modifies ('free shipping is available'), producing a chunk that says 'free shipping is available' without the qualifying condition. The retrieval returns this chunk, the model reads it, and the answer is technically derived from the context but semantically wrong.

Chunking strategy failures appear as queries where the correct document exists in the source corpus but the correct information is not in any single chunk. A document that answers 'what is the refund policy for electronics?' may have the refund policy in paragraph 3 and the electronics category in paragraph 7 — a chunker that splits by paragraph puts each in a separate chunk, and neither chunk alone answers the query. The fix is not better retrieval; it is chunking that preserves semantic completeness — larger chunks, overlap between chunks, or structure-aware chunking that keeps related sections together. The diagnostic test is to search the source document manually and confirm whether the answer exists in a single retrievable unit.

Ingestion staleness is a separate failure: the index was built from a version of the documents that is now outdated. A policy document was updated on Monday, the RAG index was last refreshed on the previous Friday, and the system returns the old policy. The retrieval 'succeeds' — it returns the chunk with the old policy — but the answer is wrong because the source has changed. The signal is a mismatch between the source document's last-modified date and the index's build date. The fix is scheduled or event-driven re-indexing, and the diagnostic check is to compare the retrieved chunk against the current source document for the failing query.

Retrieval diagnosis starts with a simple question: for the failing query, what did the vector search return? Pull the top-k chunks, read them, and judge whether the correct evidence is among them. If it is not, the retrieval failed. The next step is to search the index directly — not through the embedding, but by keyword or metadata — to determine whether the correct chunk exists in the index at all. If it exists but the embedding search did not surface it, the embedding model or the search parameters are the problem. Common causes include an embedding model that does not handle the query's vocabulary, a top-k that is too small, or a similarity metric that does not match the embedding space.

Reranking failures are subtler. The retrieval returns the correct chunk, but the reranker demotes it below the prompt's context limit. The top-k from the vector search includes the correct chunk at position 3, the reranker reorders it to position 12, and the context assembly only includes the top 5 chunks — so the correct evidence is cut. The signal is a discrepancy between the retrieval rank and the reranked rank for the correct chunk. The fix is to inspect the reranker's inputs and outputs: is the reranker scoring relevance correctly, is the context limit set too low, or is the reranker model itself the problem? Reranking diagnosis requires logging both the pre-rerank and post-rerank order for every query.

Hybrid search failures — combining vector and keyword search — appear when one search leg dominates the other. A query that uses a specific product code ('SKU-12345') may retrieve poorly with vector search (the embedding does not capture the exact code) but perfectly with keyword search. If the hybrid search weights vector search too heavily, the keyword match is buried. The diagnostic is to run each search leg independently and compare: if the correct chunk appears in keyword results but not in the combined results, the hybrid fusion is the problem. The fix is to tune the fusion weights or use a rank fusion method that does not penalise single-leg matches.

Sometimes the RAG system fails not because of a pipeline bug but because the source corpus does not contain the answer. The user asks a question that the indexed documents do not cover, and the system either hallucinates (generates a plausible but ungrounded answer) or refuses (says it cannot answer). Both are correct behaviours in different contexts: a refusal is honest and safe; a hallucination is a failure. The diagnostic question is whether the failure is a retrieval gap (the answer exists in the corpus but was not retrieved) or a knowledge gap (the answer is not in the corpus at all). The test is to search the source corpus manually — if the answer is not there, no retrieval fix will help.

Knowledge gaps require a different response than retrieval failures. The fix is to add the missing information to the corpus — a new document, a FAQ entry, or a knowledge base article — not to tune the pipeline. The system should also be designed to recognise knowledge gaps and refuse gracefully rather than hallucinate. A well-grounded RAG system includes a 'no relevant context' detection: if the top retrieved chunks have low similarity scores, the system responds with 'I do not have enough information to answer this' rather than generating from the model's parametric memory. This is a generation-layer design decision, not a retrieval fix.

The operational discipline is to track knowledge gaps as a category separate from bugs. A knowledge gap is a content problem — the corpus is incomplete — and the fix is owned by the content team, not the engineering team. A retrieval failure is an engineering problem — the pipeline is not finding what exists. Conflating the two sends engineers to tune retrieval for a problem that only the content team can fix. The trace should distinguish: if the source corpus does not contain the answer, log it as a knowledge gap with the query text, so the content team can prioritise what to add.

Context utilization measures whether the model used the evidence that was provided. A RAG system can retrieve the correct chunks, assemble them into the prompt, and still produce an answer that ignores the context — the model falls back to its parametric memory and generates a plausible but ungrounded response. The signal is a faithfulness score that compares the answer against the retrieved context: if the answer contains claims not supported by the context, faithfulness is low. Faithfulness is the core quality metric for RAG generation — it distinguishes 'the model answered from the context' from 'the model answered from memory and the context was decorative'.

Faithfulness evaluation requires a judgement method: human evaluation (a person reads the answer and the context and marks unsupported claims), LLM-as-judge (a second LLM evaluates whether each claim in the answer is supported by the context), or a Natural Language Inference model (a trained entailment classifier). Human evaluation is the gold standard but does not scale; LLM-as-judge is the practical choice for periodic evaluation, with the caveat that the judge model can share biases with the generation model. The evaluation rubric must define what counts as 'supported' — a direct quote, a paraphrase, a logical inference — and what counts as a violation — a claim with no basis in the context, a claim that contradicts the context, or a claim that adds facts the context does not contain.

Context ordering is a related factor that affects utilization. Research on LLM context handling shows that models can exhibit 'lost in the middle' effects — information in the middle of a long context is less likely to be used than information at the beginning or end. The diagnostic is to test the same query with different context orderings: if the answer changes when the correct chunk moves from the middle to the beginning, the model has a position bias. The fix is to place the most relevant chunks at the beginning or end of the context, not in the middle — a context assembly decision, not a retrieval or generation decision.

A trace is the raw material for diagnosis; a corrective action is the output. The gap between the two is the analysis step: reading the trace, identifying the failing layer, forming a hypothesis about the cause and designing a fix. The trace-to-fix workflow is a structured process that prevents the common anti-pattern of changing things at random and hoping the answer improves. Each step produces evidence — the trace, the hypothesis, the fix, the verification — so the team accumulates a record of what failed and what fixed it, which becomes the input to regression tests.

The workflow starts with collecting the trace for the failing query: the query text, the retrieved chunks with scores and ranks, the reranked order, the assembled prompt and the model response. The next step is to read the retrieved chunks and judge whether the correct evidence is present — this is the retrieval-vs-generation split. If the evidence is missing, the team searches the index to determine whether the chunk exists; if it does not, the team checks the source document to determine whether the information exists at all. Each branch leads to a specific layer and a specific fix, not a generic 'improve the RAG system'.

The fix is verified by re-running the failing query and checking the trace. If the fix worked, the trace now shows the correct evidence retrieved and the answer faithful to the context. The final step — the one that prevents the same failure from returning — is to write a regression test: an assertion that the correct chunk appears in the top-k for this query, or that the answer is faithful for this query. The regression test is added to the evaluation suite and run on every future change. Without this step, the same failure returns the next time someone changes the chunking strategy or the prompt template.

A regression test for a RAG system is an assertion about a specific query: the correct chunk must appear in the top-k, or the answer must be faithful to the retrieved context, or the answer must match the expected output within a quality threshold. Each failure that the team diagnoses and fixes should produce a regression test, so the fix is protected against future changes. Without regression tests, a chunking change that fixes query A silently breaks query B, and the team does not discover it until a user reports it — the same failure mode, repeated.

The regression test suite is the RAG system's safety net. It is run on every change to the pipeline — a new embedding model, a new chunking strategy, a new prompt template, a new reranker. The suite should cover the known failure cases (the queries that have failed before) plus a sample of normal queries (to detect regressions in queries that have not failed). A suite that only covers known failures will not detect new regressions; a suite that only covers normal queries will not protect the known fixes. The balance is a judgement: the suite should be large enough to catch regressions but small enough to run on every change without slowing the team down.

The test types map to the failure layers. Retrieval regression tests assert that the correct chunk appears in the top-k for a query — they catch embedding changes, top-k changes and reranker changes. Generation regression tests assert that the answer is faithful to the retrieved context — they catch prompt changes, model changes and context ordering changes. End-to-end regression tests assert that the final answer matches the expected output — they catch any change that affects the user-facing answer. Each type protects a different layer, and all three are needed. The OpenTelemetry instrumentation that collects traces in production can also feed the regression suite: production queries that fail become regression tests for the next release.