Large language models are good at language but know nothing about your business. Ask a general chatbot about your refund policy and it will either refuse or, worse, invent something plausible. Retrieval-augmented generation (RAG) fixes this by giving the model the right information at the moment it answers.
How it works in three steps
- Index: your documents — policies, product sheets, SOPs — are split into passages and converted into embeddings, numerical representations of meaning.
- Retrieve: when someone asks a question, the system finds the passages most similar in meaning to the question.
- Generate: the model writes an answer using only those passages, and cites which ones it used.
The citation step matters more than it sounds. It lets the person asking check the source, and it gives you a way to measure whether the system is answering from the right material.
Signs you need it
- The same questions are answered repeatedly by people who have better things to do.
- Answers exist in documents, but finding them takes too long.
- New staff take months to become self-sufficient because knowledge is scattered.
Where it goes wrong
Most RAG failures are retrieval failures, not model failures: outdated documents, passages split in the wrong places, or search that misses the relevant section. That's why we build a test set of real questions before launch and measure both retrieval and answer quality. If the right passage isn't retrieved, no model will produce the right answer.
A good first project is narrow: one team, one document set, a clear definition of a correct answer. Prove it there, then expand.