Retrieval that works

Getting the right passage into the prompt is most of the battle. This module covers the techniques that make retrieval reliable.

4.1 Lexical vs semantic search

4.2 Hybrid search

Hybrid search runs both and merges the results. A simple, robust way to merge ranked lists is reciprocal rank fusion (RRF): each document scores 1/(k + rank) in each list, and scores are added up.

Python · reciprocal rank fusiondef rrf(rankings, k=60):
    """rankings: list of ranked lists of doc ids, e.g. [bm25_ids, vector_ids]"""
    scores = {}
    for ranked in rankings:
        for rank, doc_id in enumerate(ranked, start=1):
            scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)

print(rrf([["d3", "d1", "d7"], ["d1", "d9", "d3"]]))   # d1 and d3 rise to the top

4.3 Reranking

Retrieval is a funnel. First, cheap search pulls back many candidates (say, the top 50–150). Then a reranker — a model that reads the query and each candidate together — scores them more accurately, and you keep the best handful (say, 5–20) for the prompt. Rerankers are slower per document, which is why they're applied only to the shortlist.

4.4 Contextual retrieval

A chunk like "Revenue grew 3% over the previous quarter" is ambiguous on its own: which company? which quarter? In 2024 Anthropic described contextual retrieval: before indexing, an LLM writes a short sentence situating each chunk within its document, and that context is prepended to the chunk for both embedding and BM25. In Anthropic's experiments, measured on the top 20 retrieved chunks:

Results will vary with your data — but the lesson generalises: give chunks context, combine search methods, and rerank.

4.5 Improving the query

• Query rewriting: turn a chatty follow-up ("what about for furniture?") into a standalone query using the conversation history.
• Multi-query: generate several phrasings and merge results.
• Decomposition: split a complex question into sub-questions and retrieve for each. Agentic retrieval systems — where an LLM plans and runs several searches — take this further.

4.6 Filters and permissions

Apply metadata filters before or during retrieval: date ranges, document types, regions — and, crucially, access control. A RAG system must never retrieve a document the user isn't allowed to see; the LLM cannot be trusted to keep secrets it has been shown. (See OWASP LLM08 in the AI Security course.)

Key takeaways

• BM25 catches exact terms; vectors catch meaning — use both.
• Reciprocal rank fusion is a simple, robust way to merge rankings.
• Retrieve broadly, then rerank a shortlist.
• Adding context to chunks reduced retrieval failures substantially in Anthropic's tests.
• Rewrite conversational queries; filter by metadata and permissions.