What is RAG? Retrieval-Augmented Generation Explained

How does RAG differ from a generic AI assistant?

A generic AI assistant generates answers from its training data, which may or may not include your internal documents, policies, or industry-specific knowledge. RAG, by contrast, pulls answers from a predefined knowledge base you control. It does not invent responses; it retrieves and synthesises them from the content you provide. This makes it faster and more reliable for domain-specific queries, but it is only as good as the documents you feed it.

For example, a legal Q&A system using RAG will answer questions about your firm’s contracts by referencing the actual clauses in your repository. A generic chatbot, however, might hallucinate an answer based on general legal knowledge, which could be inaccurate or irrelevant to your jurisdiction. The trade-off is clear: RAG excels in precision for known, documented information, but it cannot answer questions outside the scope of your knowledge base.

What goes wrong in practice is that teams assume RAG will cover every possible query. If your knowledge base is incomplete, outdated, or poorly structured, the system will either return irrelevant answers or admit it cannot find a match. This is not a flaw in RAG—it is a limitation of the data you provide. The mechanism is sound, but the output depends entirely on the quality of your input.

Why does chunking make or break RAG accuracy?

Chunking is the process of splitting your documents into smaller, manageable pieces before indexing. Each chunk is then retrieved independently when a query is made. The size of these chunks directly impacts accuracy: smaller chunks improve precision by allowing the system to pinpoint exact information, but they risk losing the broader context needed to understand a query. Larger chunks preserve context but may include irrelevant details, reducing retrieval accuracy.

Consider a 500-word paragraph about a company’s HR policy. If you split it into 50-word chunks, the system might retrieve a single sentence about holiday entitlement but miss the surrounding context about eligibility criteria. If you use 200-word chunks, the system may retrieve the entire section, including irrelevant details about sick leave, which could dilute the relevance of the answer. The trade-off is between granularity and coherence.

What goes wrong is that teams often default to a one-size-fits-all chunking strategy. Poor chunking leads to irrelevant or incomplete answers because the system either cannot find the right context or retrieves too much noise. The solution is to experiment with chunk sizes and overlaps, but this requires time and testing to get right. Without it, your RAG system will underperform, regardless of the quality of your documents.

What happens when retrieval ignores your permissions?

Permission-aware retrieval is the mechanism that filters content based on user access rights before generating an answer. Without it, RAG will retrieve and surface any document in the knowledge base, regardless of who is asking the question. This is a critical oversight for organisations handling sensitive data, as it can lead to accidental exposure of confidential information.

For example, a finance team’s RAG system might be designed to answer queries about budget reports. If the system ignores permissions, it could retrieve and display HR documents that are restricted to the HR team. The trade-off here is that adding permission-aware retrieval increases the complexity of the system, requiring additional infrastructure to map users to their access rights and filter content accordingly.

What goes wrong is that teams often prioritise speed and simplicity over security. Unrestricted retrieval can expose sensitive or confidential data, leading to compliance breaches or reputational damage. In regulated industries like finance or healthcare, this is not just a technical issue—it is a legal and ethical one. The mechanism for permission-aware retrieval exists, but it must be implemented from the start, not bolted on later.

Why do citations matter in RAG—and how are they generated?

Citations in RAG link the generated answer to the exact source chunks used to produce it. This mechanism improves trust by allowing users to verify the information and trace it back to its origin. Without citations, RAG answers are essentially black boxes: you cannot audit where the information came from or whether it is accurate. The trade-off is that generating citations requires additional processing to map answers to their sources, which can slow down the system.

For example, a RAG system for medical guidelines might answer a question about dosage instructions by citing specific paragraphs from a clinical manual. This not only builds trust but also allows healthcare professionals to cross-reference the information with the original source. The mechanism for citation generation involves tracking which chunks were retrieved and used in the final answer, then embedding those references into the response.

What goes wrong is that teams often treat citations as an afterthought. Missing or incorrect citations reduce transparency and auditability, making it difficult to trust the system’s outputs. In high-stakes environments like healthcare or legal, this can have serious consequences. The solution is to design citation generation into the system from the beginning, ensuring that every answer is traceable to its source.

Can RAG handle real-time or frequently updated data?

RAG can handle real-time or frequently updated data, but only if the knowledge base is re-indexed or updated to reflect changes. The mechanism for this involves periodically refreshing the index to include new or revised documents. The trade-off is that real-time updates improve accuracy but increase latency and cost, as the system must constantly process and re-index new information.

For example, a news aggregation RAG system might need to update its knowledge base hourly to include the latest articles. In contrast, a static document repository, such as a company’s internal policies, might only need quarterly updates. The frequency of updates depends on the volatility of your data and the tolerance for stale information.

What goes wrong is that teams often underestimate the operational overhead of keeping a knowledge base current. Stale data leads to outdated or incorrect answers, which can be just as problematic as missing data. The solution is to establish a clear update schedule and automate the re-indexing process where possible. Without this, your RAG system will quickly become a liability rather than an asset.

What are the hidden costs of implementing RAG?

The hidden costs of RAG go beyond the initial development price. Document preprocessing, indexing, retrieval infrastructure, and ongoing maintenance all add up. For instance, cleaning and structuring your documents to ensure they are RAG-ready can take weeks or even months, depending on the volume and complexity of your data. The trade-off is that cloud-based RAG reduces upfront costs but increases long-term operational expenses, as you pay for storage, compute, and data transfer.

A concrete example: a team might budget for the development of a RAG system but fail to account for the time required to clean, deduplicate, and format their documents. This preprocessing step is often the most time-consuming part of the project, as it involves manual review and standardisation. Without it, the RAG system will struggle to retrieve accurate or relevant answers.

What goes wrong is that poorly planned RAG implementations become expensive and slow. Teams may cut corners on preprocessing or indexing, leading to a system that is either inaccurate or too sluggish to be useful. The mechanism for avoiding this is to treat RAG as a data project first and a software project second. The quality of your data will determine the success of your system, not the sophistication of the technology.

When should you avoid RAG entirely?

RAG is not a one-size-fits-all solution. It is not suitable for tasks requiring creativity, open-ended reasoning, or real-time decision-making. The mechanism of retrieving and synthesising information from a predefined knowledge base works well for factual, document-based queries, but it struggles with tasks that require original thought or adaptability. The trade-off is that generic AI may perform better for tasks outside RAG’s strengths, such as brainstorming, storytelling, or complex problem-solving.

For example, using RAG for creative writing would be a poor fit, as it can only regurgitate existing content rather than generate new ideas. In contrast, using RAG for internal knowledge retrieval, such as answering questions about company policies or procedures, is a natural use case. The mechanism is designed for precision, not innovation.

What goes wrong is that teams force RAG into the wrong use case, leading to poor results and wasted resources. If your goal is to generate novel content or make real-time decisions based on dynamic data, RAG is not the right tool. The solution is to evaluate your use case carefully and choose the technology that aligns with your objectives. Sometimes, the best answer is to avoid RAG entirely.