Library guide · Retrieval-augmented generation
Give your AI a library card.
RAG lets an AI look things up in your own documents before it answers, so replies are accurate, current and backed by sources. Here's how it works, where it pays off and how to start.
Your knowledge, ready to be looked up. Hover a card.
Your AI is taking a closed-book exam
A large language model only knows what it read during training. Ask it about your refund rules, this month's prices or a client contract and it has nothing to look at, so it guesses. RAG turns that into an open-book exam.
Closed book AI on its own
- Knowledge frozen at its training date
- Has never seen your contracts, prices or policies
- Fills gaps with confident guesses
- Can't show where an answer came from
Closed bookOpen book
RETURNS POLICY · v3 · p.4
§2.1 Large items
Sofas, beds and wardrobes may be returned within 60 days of delivery. A collection fee applies.
up to its training date
Your company
Open book AI + RAG
- Reads your latest documents at question time
- Answers with your real prices and rules
- Says "I don't know" when sources are silent
- Cites the exact document it used
So, what is RAG?
RAG /ræɡ/ noun, abbr. Retrieval-Augmented Generation
- A way of building AI where the system first retrieves relevant passages from your own sources, then adds them to the question, and only then generates an answer based on that evidence.
- Business sense. An AI assistant that answers from your company's knowledge instead of from memory, and can show you where each answer came from.
Usage: "We put RAG on our support docs, and the bot finally quotes the right warranty terms."
Origin: Introduced in a 2020 research paper by Patrick Lewis and colleagues at Facebook AI Research (now Meta AI), University College London and New York University.
See also: embeddings · vector database · grounding · fine-tuning
Search your documents for the few passages that match the question.
Paste those passages into the prompt, with rules like "answer only from these".
The language model writes the answer in plain words and cites its sources.
A customer types a question into your website chat, help desk or internal tool.
An embedding model turns the question into a long list of numbers that captures its meaning, not just its words.
[ 0.12, −0.48, 0.91, 0.07, −0.33, 0.64, … 1,536 numbers ]A vector database compares those numbers with every chunk of your documents and finds the closest matches. Good systems also run a keyword search and merge the two.
The top few passages, usually three to five, are pulled out. Low-scoring matches are thrown away.
The passages are pasted into the prompt together with clear rules. This is the "augmented" part.
RULES: Answer only from SOURCES. Cite them. If unsure, say so. SOURCES: [1] Returns §2.1 … [2] Fees table … QUESTION: Can I return a sofa after 40 days?
The language model writes a clear answer from the sources and shows which ones it used, so anyone can check it.
How AI finds the right page: meaning as a map
An embedding model places every chunk of your documents on a giant "map of meaning". Chunks about similar things end up close together, even when they use different words. A question is placed on the same map, and the nearest chunks are retrieved.
Try a question
Retrieved chunks
- Pick a question to see what the system pulls out.
Simplified 2D illustration. Real embeddings have hundreds or thousands of dimensions, and the scores here are examples.
Before search: cut the book into index cards
A RAG system doesn't search whole documents. It cuts them into small pieces called chunks and searches those. How you cut decides what the AI can find. Try it.
returns-policy-v3.pdf · page 4
Hover a chunk to inspect it.
Split on headings and paragraphs first, then by size. One idea per chunk beats a fixed character count.
Store each chunk with its title, date, owner and who may see it. That powers citations and permissions.
When a document is updated, re-index it automatically so the AI never quotes last year's version.
Same question. Two very different answers.
Here's a support bot for a fictional furniture store, Bramble Home. Their policy allows 60-day returns on large items. Watch what happens with RAG off, then on.
Where RAG pays off in a business
If a team keeps answering questions from documents, RAG can help. Eight places it earns its keep. Hover or tap a card to flip it.
Ask: "Does my plan include on-site repair?"
Reads: help centre, policies, solved tickets
Win: faster first replies and fewer escalations, with sources agents can check
Ask: "What's our parental leave policy?"
Reads: HR handbook, intranet, policy PDFs
Win: HR stops answering the same twenty questions every week
Ask: "Which case study fits a mid-size logistics client?"
Reads: case studies, price sheets, product specs
Win: reps answer RFPs and objections in minutes, not days
Ask: "Which vendor contracts auto-renew in Q1?"
Reads: signed contracts, clause library
Win: clause search in seconds with page references; a lawyer still signs off
Ask: "What's the expense limit for client dinners?"
Reads: finance policies, audit rules, regulator circulars
Win: consistent answers with citations auditors can follow
Ask: "How do I set up VPN on my new laptop?"
Reads: runbooks, wiki pages, ticket history
Win: routine tickets solved by self-service
Ask: "Will this cover fit a 3-seater sofa?"
Reads: product catalogue, size charts, FAQs
Win: fewer pre-sale chats and fewer wrong-size returns
Ask: "What does error E42 mean on the X200 pump?"
Reads: technical manuals, service bulletins
Win: technicians fix more faults on the first visit
RAG by the numbers
RAG has become the default way companies connect AI to their own data. It also isn't magic: how well you build it decides how often it's right.
DATE DUE · 2024
of enterprise AI deployments used RAG in 2024, up from 31% a year earlier.
Menlo Ventures, State of GenAI in the Enterprise 2024
DATE DUE · 2024
of production models were fine-tuned. Most teams ground models with retrieval instead.
Menlo Ventures, State of GenAI in the Enterprise 2024
DATE DUE · 2030
projected RAG market by 2030, up from $1.94B in 2025, a 38.4% yearly growth rate.
MarketsandMarkets, RAG Market report
DATE DUE · 2025
hallucination rate in legal research: GPT-4 on its own vs the best RAG-based tool tested.
Stanford RegLab & HAI, J. Empirical Legal Studies 2025
DATE DUE · 2025
of answers from the weakest RAG legal tool still contained a hallucination. Retrieval quality matters.
Stanford RegLab & HAI, J. Empirical Legal Studies 2025
DATE DUE · 2020
the year RAG was introduced, in a paper by Lewis et al. from Facebook AI Research, UCL and NYU.
Lewis et al., NeurIPS 2020
RAG, fine-tuning or a long prompt?
There are three common ways to make an AI "know" your business. They solve different problems. Pull out a drawer.
Answering from documents and data that change: policies, products, contracts, tickets.
Searches your sources at question time and gives the model only the relevant passages.
It is only as good as your search quality and the documents you feed it.
Teaching a consistent style, format or narrow task, like writing in your brand voice or classifying tickets.
Trains the model further on hundreds or thousands of your examples.
Knowledge goes stale until you retrain, it cannot cite sources, and each retrain costs time and money.
One-off questions about a handful of documents, or quick prototypes.
Pastes whole documents into the prompt every time someone asks.
Slow and expensive at scale, accuracy can drop in very long prompts, and there is no per-user access control.
Rule of thumb: start with RAG for knowledge. Add fine-tuning later only if you need a very specific style or task. They work well together.
What's inside a RAG system
Seven layers, from your raw files at the bottom to a safe, checked answer at the top. You don't need all the fancy parts on day one, but you do need every layer.
- Guardrails & evaluation
Citations, "say I don't know" rules, PII filters, test question sets and live monitoring.
- Language model
Writes the answer from the retrieved passages. GPT, Claude, Gemini, Llama or Mistral.
- Retriever & reranker
Hybrid search (keywords + meaning), reranking, and filters so people only see what they're allowed to.
- Vector database
Stores the embeddings for fast similarity search. pgvector, Pinecone, Weaviate, Qdrant or Elasticsearch.
- Embedding model
Turns every chunk and every question into numbers that capture meaning.
- Ingestion & chunking
Connectors, OCR for scans, cleaning, chunking and metadata. Usually the biggest job.
- Data sources
Google Drive, SharePoint, Notion, your CRM, help desk, PDFs and databases.
Errata: six mistakes that make RAG fail
Most disappointing RAG projects don't fail because of the AI model. They fail on data, search and testing. Here's what to fix.
Dump every file in, exactly as it is.
Correction: Clean first. Remove duplicates and old versions, and choose one source of truth per topic.
Use one chunk size for every document.
Correction: Chunk by structure: headings for policies, rows for tables, question-and-answer pairs for FAQs.
Rely on vector search alone.
Correction: Use hybrid search (meaning + keywords) and a reranker, so product codes and names are found too.
Let everyone retrieve everything.
Correction: Copy file permissions into the index and filter results per user, so salaries stay with HR.
Launch without a test set.
Correction: Collect 50–200 real questions with known answers and score every change against them.
Let it answer even when the sources are silent.
Correction: Require citations, and tell the model to say "I don't know" when nothing relevant is found.
Does your business need RAG? Check in 30 seconds.
-
Does your team answer the same questions from documents every week?
-
Does your knowledge live in PDFs, wikis, drives, tickets or a CRM?
-
Do answers need to be accurate and traceable to a source?
-
Does the information change often (prices, policies, products)?
-
Is there someone who owns and updates that content?
LIBRARY CARD · RAG FIT
0/5 yes
Answer the questions to see your result.
Let's give your AI a library card.
Tell us where your knowledge lives. We'll map the quickest RAG pilot, the data you'll need and what it would take to launch. Free 30-minute call.
Questions people ask about RAG
RAG (retrieval-augmented generation) is a way to make an AI answer from your own information. Before it replies, the system searches your documents for the relevant passages and hands them to the AI, which writes the answer from that evidence and can show its sources.
A general chatbot answers from what it learned in training, which stops at a cut-off date and doesn't include your private data. A RAG system looks up your current documents every time someone asks, so it can quote your real policies, prices and contracts and point to where each answer came from.
They solve different problems. RAG is best for knowledge that changes and needs sources; fine-tuning is best for teaching a model a style, format or narrow task. Most businesses start with RAG and add fine-tuning only if needed. In Menlo Ventures' 2024 survey, 51% of enterprise AI deployments used RAG, while only 9% of production models were fine-tuned.
No. It reduces them a lot but doesn't remove them. A 2025 Stanford study of legal research tools found that RAG-based products hallucinated 17–33% of the time, compared with 43% for GPT-4 on its own. Clean data, hybrid search, required citations, "I don't know" rules and regular testing close much of the remaining gap.
Almost any text your business has: PDFs, Word files, wikis, Google Drive or SharePoint, help-desk tickets, emails, product catalogues, CRM notes and database records. Scanned documents need OCR first, and tables work best when they are indexed row by row.
It can be, if it's designed for it. Your documents stay in your own storage and vector database, and you can choose models and hosting that don't train on your data, including private cloud or on-premise setups. The index should also copy your existing permissions, so each person only retrieves what they're allowed to see.
It depends on how many sources you connect, how clean the data is and your security needs. A focused pilot on one knowledge base is a small project; company-wide systems with many integrations, permissions and evaluation cost more. Running costs are mainly hosting, the vector database and model usage per question. A short assessment gives you a firm quote.
A pilot on one well-organised knowledge base typically takes 4–8 weeks, including data preparation, testing and a supervised launch. Larger rollouts take longer, mostly because of integrations, permissions and content clean-up rather than the AI itself.
A vector database stores embeddings (the number versions of your text) and finds the closest matches very quickly. For smaller collections, a vector extension in a database you already run, such as PostgreSQL with pgvector, is often enough. Dedicated options like Pinecone, Weaviate or Qdrant help at larger scale.
Pick one use case with lots of repeated questions, gather and clean the documents behind it, and write down 50–100 real questions with correct answers. Build a small pilot, test it against those questions, launch to a small group with sources visible, then expand. Aigentora can run this with you, starting with a free assessment.




