Everyone has tried asking a general AI model about their own business and watched it confidently invent an answer. Retrieval-augmented generation, or RAG, is the technique that fixes this: instead of relying on what the model memorised during training, it looks up the relevant passages from your documents first and answers from those, with citations.
RAG is the foundation of nearly every useful internal AI assistant and customer-facing knowledge bot. Here is how it works and when it is the right tool.
How RAG works
- Ingest. Your documents (policies, manuals, past tickets, product data, contracts) are split into chunks and converted into embeddings, numerical representations of meaning, stored in a vector database.
- Retrieve. When someone asks a question, it is converted the same way and the most relevant chunks are found.
- Generate. The question and the retrieved chunks are given to the language model, which answers using only that material and cites where it came from.
The result is an assistant that knows your business, stays current when documents change, and can show its sources.
Where RAG is useful
- Internal help desks: HR, IT and operations questions answered from policies and runbooks.
- Customer support: accurate answers from product documentation and past resolutions, with a human handoff for anything uncertain.
- Sales enablement: instant answers on pricing, specifications and case studies from approved material.
- Compliance and legal: finding the relevant clause across hundreds of contracts, with the source shown.
Why it beats fine-tuning for most businesses
Fine-tuning changes the model's behaviour; it does not reliably teach it facts, and it must be redone when information changes. RAG keeps knowledge in documents you already maintain, updates instantly, and can enforce access permissions so people only get answers from material they are allowed to see. For factual business knowledge, RAG is almost always the right choice.
What makes a RAG system good or bad
The model is rarely the problem. Quality depends on how documents are chunked, how retrieval is tuned, whether metadata and permissions are respected, and how the system behaves when it does not find a good answer. A good system says 'I could not find this' rather than guessing. Evaluation with a set of real questions and known answers is essential before launch and after every change.
Building it
A focused RAG assistant on a defined document set takes 4–8 weeks: ingestion pipeline, retrieval tuning, a chat interface or integration into Slack or your help desk, permissions, logging and an evaluation set. It frequently sits inside a wider AI agent that can also take actions. The AI agent explainer covers that combination.
Getting your documents ready
The quality of a RAG system is capped by the quality of its sources. Before building, audit the document set: remove superseded versions, mark the authoritative copy of each policy, add dates and owners, and break very long documents into logically titled sections. Contradictions between documents produce contradictory answers, so resolve them. This cleanup is often the most valuable outcome of the project regardless of the AI, because it fixes a knowledge problem the business already had.
Permissions, privacy and audit
- Carry document-level permissions through to retrieval so people only receive answers from material they can already read.
- Keep an audit log of questions, retrieved sources and answers for review and compliance.
- Decide whether query data may leave your environment and choose the model deployment accordingly.
- Set a retention policy for logs that matches your data protection obligations in the UK, EU or US.
- Give users a way to flag a wrong answer so the evaluation set grows from real use.
These are standard practices in any well-built system and the questions any serious vendor should be able to answer on day one.
RAG versus a simple search box
Keyword search finds documents; RAG answers questions. The difference matters when the answer is spread across several documents, when staff do not know the right terminology to search for, or when the question needs synthesis ('what is our policy on X for customers in Y?'). The best systems combine both: hybrid retrieval that uses keywords and meaning together, with the model composing an answer and citing the passages so the user can read the originals.
Frequently asked questions
Does RAG send my documents to OpenAI or Anthropic?
The retrieved passages are sent to the model provider at query time under their API terms, which for business plans do not train on your data. Fully private deployments with self-hosted models are possible for sensitive data.
How accurate is it?
On well-structured document sets with good retrieval, very high, and crucially the citations let users verify. Accuracy drops with messy, contradictory or outdated documents, which is a content problem worth fixing anyway.
Can it handle PDFs, spreadsheets and scanned documents?
Yes, with appropriate extraction and OCR in the ingestion pipeline. Structured data is often better served by a database query tool alongside RAG.
Founder & CEO of Devontix Solutions. Absaar leads strategy, automation planning and product thinking across every build, from the first discovery call to the running system. More about Absaar