Why LLMs need your data
LLMs are brilliant generalists that know nothing about your organisation. Retrieval-augmented generation fixes that by looking things up first.
Learning objectives
- Explain the knowledge limits of LLMs
- Compare prompt-stuffing, RAG and fine-tuning
- Describe the indexing and query halves of a RAG pipeline
- Identify the common failure points of RAG
All examples are Python and run in a free notebook such as Google Colab. Where an example calls an LLM, it uses a small call_llm() helper (Module 5) that you can point at Claude, GPT, Gemini or a local model. You'll need your own API key for generation; everything else runs free.
1.1 What an LLM doesn't know
A large language model is trained on a huge snapshot of public text. Out of the box it knows nothing about:
- Your private information — policies, contracts, product specs, tickets, wikis.
- Anything after its training cut-off — new prices, regulations, decisions.
- Exact details it saw rarely — and it may confidently invent them (hallucinate).
Ask a general assistant "What's our refund window for damaged items?" and the best it can do is guess what companies usually do.
1.2 Three ways to give a model your knowledge
| Approach | How it works | Good for | Limits |
|---|---|---|---|
| Put it in the prompt | Paste the relevant document into the context | One-off questions; small document sets | Doesn't scale to thousands of documents; cost grows with every token |
| Retrieval-augmented generation (RAG) | Search your documents for the most relevant passages and put only those into the prompt | Large, changing knowledge bases; answers with citations | Only as good as the retrieval step |
| Fine-tuning | Further train the model on your examples | Style, format, specialised behaviour | Poor at adding facts that change; expensive to update (see the Fine-tuning course) |
For "answer questions from our documents", RAG is usually the right starting point: it's cheaper, easier to update (just change the documents) and lets you show sources.
1.3 The RAG pipeline
Every RAG system has two halves:
- Indexing (done ahead of time): load documents → split them into chunks → turn each chunk into an embedding → store chunks and embeddings in an index.
- Query time (on every question): embed the question → retrieve the most similar chunks → put them into a prompt with the question → the LLM writes an answer grounded in those chunks, ideally with citations.
1.4 Where RAG goes wrong
When a RAG system gives a bad answer, the cause is usually one of these — and each module of this course tackles one:
| Failure | Symptom | Module |
|---|---|---|
| Poor embeddings or similarity search | Irrelevant passages come back | 2 |
| Bad chunking | The answer is split across chunks, or chunks lack context | 3 |
| Weak retrieval | Exact terms (product codes, names) are missed | 4 |
| Ungrounded generation | The model ignores the passages or adds facts | 5 |
| No evaluation | You can't tell if a change helped | 6 |
For each, choose prompt-stuffing, RAG or fine-tuning: (a) an HR assistant answering from 400 policy documents updated monthly; (b) summarising one 20-page contract; (c) making every output follow your house style for product descriptions.
Show answers
(a) RAG — large, changing knowledge base. (b) Put it in the prompt — one document fits easily. (c) Fine-tuning (or a strong prompt with examples) — it's about style, not facts.
Key takeaways
- LLMs don't know your private, recent or rare information.
- RAG retrieves relevant passages and puts only those into the prompt.
- RAG suits large, changing knowledge; fine-tuning suits style and behaviour.
- Indexing happens ahead of time; retrieval and generation happen per question.
- Most RAG failures come from embeddings, chunking, retrieval, grounding or missing evaluation.