Where RAG Fails: Understanding the Limitations of Retrieval-Augmented Generation
RAG helps LLMs answer questions using your own data, but it doesn't guarantee correct answers. Here's how the pipeline works and where it commonly breaks down.
Ask a large language model something specific to your business, and it will usually do one of two things: admit it doesn't know, or answer anyway with something that sounds right but isn't. Neither is useful. Models are trained on a fixed snapshot of data up to some cutoff date, so they have no visibility into your contracts, your internal docs, or anything published after training ended.
Retrieval-Augmented Generation, or RAG, is the standard fix for this. Instead of relying only on what the model memorized during training, RAG looks up relevant information from an external source and hands it to the model before it answers. That's the whole idea: give the model something to read, rather than asking it to answer from memory alone.
It's a useful technique, and it's also frequently described as if it solves the problem completely. It doesn't. RAG reduces certain kinds of errors and introduces its own. This post covers what RAG is, how a basic pipeline works, and where it tends to fail in practice.
What RAG Is
RAG combines two things: a retrieval system that finds relevant text, and a language model that generates an answer using that text as context. Instead of trusting the model's internal memory, the system fetches supporting material first and passes it along with the question.
The model isn't getting smarter here. It's getting better inputs. That distinction matters, because it explains both why RAG works and why it has limits: the output is only as good as what gets retrieved.
Why RAG Was Introduced
Most businesses don't need a model that knows about the world in general. They need one that knows their world: legal documents, contracts, invoices, product manuals, support history, maybe hours of recorded calls or video. That data doesn't live inside a model's weights, and retraining a model every time a document changes isn't practical.
The solution the industry settled on is indexing: convert documents into a searchable format ahead of time, so the right piece can be pulled at query time instead of feeding the model everything at once. This searchable format is typically a vector database. Getting there requires embeddings, numerical representations of text that place similar meanings close together, even when the wording is different, which is what enables semantic search instead of plain keyword matching.
How a Basic RAG Pipeline Works
A basic RAG pipeline follows a fairly fixed sequence of steps:
Ingest content: collect the raw source material: PDFs, PPTs, audio, video, websites.
Convert to text: audio and video need transcription first, usually producing VTT or SRT files.
Chunking: split large documents into smaller pieces that an embedding model can process meaningfully.
Generate embeddings: run each chunk through an embedding model to capture its semantic meaning.
Store in a vector database: options include Qdrant, Pinecone, Weaviate, Milvus, pgvector, and Chroma DB.
Similarity search: when a user asks a question, it's embedded too, and the database returns the chunks closest to it in meaning.
Pass to the LLM: the retrieved chunks go into the prompt alongside the user's question.
Generate a response: the model answers using the retrieved context.
Eight steps, and each one can affect the quality of the final answer. That's the part worth paying attention to, since most explanations of RAG stop at step 8 and treat the pipeline as a solved problem.
Where RAG Works Well
RAG is genuinely strong in a specific kind of scenario: when the answer already exists somewhere in your source material, and the task is really about finding and presenting it.
Internal documentation and support: employees querying HR policies, product manuals, or support knowledge bases.
Searchable archives: finding a specific clause across hundreds of contracts.
Content summarization: pulling relevant context from one or many documents quickly.
In these cases, RAG is usually reliable, because the job is closer to search than reasoning.
Why RAG Sometimes Gives Incorrect Answers
The problem is that a RAG pipeline has several independent stages, and a failure at any one of them degrades the final answer, often without any visible sign that something went wrong. The response still reads fluently. It just isn't necessarily correct.
Poor Retrieval and Missing Context
The model can only answer well if retrieval found the right material to begin with. A vague query, or a weak semantic match, can pull back chunks that are only loosely related to the question. If the necessary context never makes it into the prompt, the model has no way to give a fully correct answer, no matter how capable it is.
Poor Chunking and Its Impact on Responses
How a document gets split into chunks has a direct effect on what the model sees.
Chunks that are too large carry excess irrelevant text, which dilutes the signal.
Chunks that are too small can lose meaning entirely. A sentence like "He declined the offer" is not useful on its own if the previous sentence, which explains who "he" is, ends up in a different chunk.
There isn't a fixed chunk size that works everywhere. It depends on document structure, and getting this right typically takes iteration rather than a one-time setting.
Context Window Limitations
Every LLM has a maximum number of tokens it can process at once, its context window. You can't hand the model an entire large document and expect it to fit. If a similarity search retrieves more text than the context window allows, some of it gets dropped, and there's no guarantee the dropped portion was the least important part. A legal question answered from half a contract is still an answer, just not necessarily a complete one.
Hallucinations Even With RAG
A common assumption is that RAG eliminates hallucinations. It doesn't; it reduces them. The model can still generate incorrect details if the retrieved context is incomplete, noisy, or contradictory. Because generation is based on learned probability patterns rather than direct lookup, the model can blend retrieved text with patterns from training and produce something that sounds plausible but isn't accurate. Retrieval improves grounding. It doesn't guarantee correctness.
Keeping Knowledge Bases Up to Date
A RAG system is only as current as its index. If a company updates a policy but the vector database still holds the old document, the system will answer confidently using outdated information, and there's no built-in mechanism that flags this as wrong. Keeping a knowledge base current requires ongoing re-indexing as source documents change, not a one-time ingestion step.
When RAG Is Not the Right Solution
RAG is not a universal fix. Tasks that require exact reasoning, such as calculations, or tasks that require live data, such as current stock prices, aren't well suited to retrieval-based approaches. Some problems are better handled by a database query, an API call, or a direct calculation than by searching documents for an answer. If a human would reach for a calculator or a live data source to answer a question, that's usually a sign the AI system needs a tool call, not a RAG pipeline.
Summary
RAG addresses a real limitation of LLMs: their inability to access information outside their training data. It's effective when the answer exists somewhere in indexed material and the task is close to search-and-summarize. But it depends on multiple components working correctly together: retrieval quality, chunking strategy, context window limits, and an up-to-date knowledge base. A weakness in any one of these can produce a confident but incorrect answer. Understanding where RAG is likely to fail is what separates a demo that works on clean examples from a system that holds up in production.
