Information Retrieval Fundamentals: The Theory Every RAG System Is Built On

Information Retrieval Fundamentals

Developer Diaries: Building with Retrieval-Augmented Generation

we already established why RAG exists: to give language models access to knowledge beyond what’s frozen in their weights. But we glossed over a detail that turns out to be the entire engineering core of any RAG system — the “R.”

Retrieval isn’t a new invention that arrived alongside LLMs. It’s a field with over sixty years of research behind it, called Information Retrieval (IR). Every modern RAG pipeline — no matter how modern its embedding model or how large its LLM — is standing on IR foundations that predate deep learning entirely. This chapter builds those foundations properly, so that later chapters on vector databases, chunking, and rerankers make sense as refinements of established ideas, not magic.

Short History of Information Retrieval

IR as a discipline predates the internet by decades:

1950s–60s — early work on automatic indexing and the vector space model, driven by the need to search scientific and legal document collections.
1970s — Gerard Salton’s SMART system formalized much of classical IR theory, including term weighting and the vector space model still referenced today.
1990s — the arrival of the web, and with it TREC (Text REtrieval Conference), which gave the field standardized benchmarks and drove rapid progress in ranking algorithms like BM25.
2000s — commercial web search matures; PageRank and link-based signals join text-based relevance scoring.
2010s — neural approaches to IR emerge: word embeddings, then transformer-based dense retrieval, begin to challenge purely lexical methods.
2020s — IR and generative AI converge. Dense retrieval, vector databases, and hybrid search become the backbone of RAG systems.

The point worth internalizing: RAG did not invent search — it repurposed it. The retrievers in a modern RAG stack are direct descendants of ideas from the 1970s, now paired with neural embeddings and LLMs.

Documents, Queries, and Collections

Three terms form the vocabulary of every IR system, and they map directly onto RAG:

Document — a discrete unit of retrievable content. In classical IR this might be a web page or news article; in RAG it’s typically a chunk of a source file (we’ll cover chunking strategy in a later chapter).
Query — the user’s information need, expressed as text. In RAG, this is usually the user’s question, sometimes reformulated before retrieval.
Collection (or Corpus) — the full set of documents available to search. In RAG, this is your knowledge base — the documents you’ve ingested and indexed.

The retrieval task, formally, is: given a query and a collection, return the subset of documents most likely to satisfy the information need behind the query. Everything else in this chapter is about how to define “most likely” and how to measure whether you got it right.

What Makes a Result “Relevant”?

Relevance sounds intuitive but is surprisingly hard to pin down precisely, and getting it wrong is the single biggest silent failure mode in RAG systems.

A document is relevant to a query if it contains information that helps satisfy the user’s underlying intent — not merely if it shares words with the query. This distinction matters enormously:

A document can share zero words with a query and still be highly relevant (a passage about “automobiles” is relevant to a query about “cars”).
A document can share many words with a query and be completely irrelevant (a document that merely mentions the query terms in passing, without addressing the actual question).

Classical IR treated relevance largely as a topical match. Modern systems increasingly treat it as intent match — which is precisely the gap that semantic search and, later, LLM-based reranking were built to close.

Measuring Retrieval Quality: Precision, Recall, and F1

You can’t improve what you can’t measure. These three metrics are the backbone of IR evaluation, and they matter just as much when you’re debugging why your RAG pipeline is retrieving the wrong chunks.

Precision answers: of the documents I retrieved, how many were actually relevant?

\[\text{Precision} = \frac{\text{Relevant documents retrieved}}{\text{Total documents retrieved}}\]

Low precision means your retriever is returning a lot of noise alongside the useful results — the LLM now has to sift signal from clutter, which increases the risk of it latching onto irrelevant context.

Recall answers: of all the relevant documents that exist in the collection, how many did I actually retrieve?

\[\text{Recall} = \frac{\text{Relevant documents retrieved}}{\text{Total relevant documents in collection}}\]

Low recall means your retriever is missing relevant information entirely — and no amount of clever prompting can help the LLM answer using a document it never saw.

F1 Score combines both into a single number, useful when you need one metric to optimize against:

\[F1 = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}\]

There’s an inherent tension here: retrieving more documents tends to improve recall (you’re less likely to miss something relevant) but hurts precision (you’re pulling in more noise). Tuning this trade-off — often via the number of chunks (top-k) you retrieve — is one of the first real design decisions in any RAG pipeline.

Ranking: From Matching to Ordering

Precision and recall tell you whether you retrieved the right documents, but they say nothing about order. In practice, order matters enormously — both because users (and LLMs) pay more attention to what appears first, and because most systems only pass a limited number of top results forward.

Ranking is the process of scoring documents by estimated relevance and ordering them accordingly. Early IR systems ranked using term-frequency-based scores (which we’ll build up to below); modern systems often rank using a combination of lexical scores, semantic similarity, and dedicated reranking models. In a RAG context, ranking quality directly determines what context the LLM sees — and in what order, which can itself bias generation.

The Search Pipeline

It helps to see the full pipeline before diving into individual components:

Raw Documents
     │
     ▼
 Preprocessing (tokenization, normalization)
     │
     ▼
   Indexing  ──────────────▶  Index Structure
                                     │
Query ──▶ Query Processing ─────────▶│
                                     ▼
                              Matching & Scoring
                                     │
                                     ▼
                                  Ranking
                                     │
                                     ▼
                              Top-K Results

Every retriever you’ll ever build — lexical, semantic, or hybrid — is an instance of this same pipeline. What changes across approaches is how documents are represented in the index and how matching/scoring is computed.

Indexing: Making Search Fast

Searching a collection by scanning every document at query time doesn’t scale. Indexing solves this by pre-processing the collection into a structure optimized for fast lookup.

The classic structure is the inverted index — instead of mapping documents to the words they contain, it maps each word to the list of documents containing it (along with position and frequency information). This flips the search problem: rather than scanning every document to check for a query term, you look the term up directly and get its document list instantly.

In modern dense/semantic retrieval, the analogous structure is a vector index (e.g., using approximate nearest neighbor algorithms like HNSW), which we’ll cover in depth in a future chapter on vector databases. Same underlying goal — precompute structure so query time is fast — different representation.

Boolean Retrieval

The earliest practical retrieval model, and still conceptually useful today. Boolean retrieval treats queries as logical expressions over terms — AND, OR, NOT — and returns documents that satisfy the expression exactly.

For example, a query like machine AND learning NOT deep would return only documents containing both “machine” and “learning,” while excluding any that also mention “deep.”

Boolean retrieval’s strength is precision and predictability — you get exactly what you asked for, logically. Its weakness is that it has no concept of degree of relevance: a document either matches or it doesn’t, and there’s no ranking among matches. This limitation directly motivated the development of term-weighting schemes.

Term Frequency (TF)

The first step beyond Boolean matching is asking: how much does a document talk about a given term? A document mentioning “retrieval” ten times is probably more focused on that concept than one mentioning it once.

\[\text{TF}(t, d) = \frac{\text{Number of times term } t \text{ appears in document } d}{\text{Total number of terms in } d}\]

TF alone has an obvious flaw: extremely common words (“the,” “is,” “of”) will have high frequency in nearly every document, without carrying any discriminating information about relevance. That gap is exactly what IDF was designed to fix.

Inverse Document Frequency (IDF)

IDF measures how informative a term is across the whole collection — rare terms carry more signal than common ones.

\[\text{IDF}(t) = \log \left( \frac{\text{Total number of documents}}{\text{Number of documents containing term } t} \right)\]

A term that appears in nearly every document (like “the”) gets an IDF close to zero — it contributes almost nothing to distinguishing relevant documents from irrelevant ones. A term that appears in only a handful of documents (like “transformer” in a general corpus) gets a high IDF — its presence is a strong relevance signal.

TF-IDF

Multiplying the two gives you a term weight that rewards terms that are frequent within a document but rare across the collection — exactly the terms most useful for distinguishing relevant documents from irrelevant ones.

\[\text{TF-IDF}(t, d) = \text{TF}(t, d) \times \text{IDF}(t)\]

A document’s overall relevance score for a query is typically computed by summing the TF-IDF weights of the query terms that appear in it. TF-IDF was, for decades, the default scoring function in search engines and remains a strong, cheap baseline today — and a useful mental model even after you move to more sophisticated methods.

BM25

BM25 (“Best Matching 25”) is a refinement of TF-IDF that emerged from the TREC era and remains, remarkably, the default lexical scoring function in most modern search infrastructure — including the “sparse” side of many production RAG hybrid retrievers.

BM25 improves on raw TF-IDF in two key ways:

Term frequency saturation — it recognizes that the tenth occurrence of a word shouldn’t count as much as the second. Its scoring function grows with term frequency but flattens out, rather than increasing linearly forever.
Document length normalization — longer documents naturally contain more term occurrences by chance. BM25 explicitly corrects for this, so a term appearing 5 times in a 50-word document counts differently than 5 times in a 5,000-word one.

You don’t need to memorize BM25’s full formula to use it well, but it’s worth knowing the intuition: it’s TF-IDF, made more robust to document length and diminishing returns on repeated terms. When you see “BM25” mentioned alongside vector search in a RAG stack, this is the lexical workhorse doing that half of the job.

Lexical Search vs Semantic Search

Everything covered so far — Boolean retrieval, TF-IDF, BM25 — falls under lexical search: matching based on exact or near-exact term overlap between query and document. It’s fast, interpretable, and excellent at exact-match cases (product codes, names, specific jargon) — but it fundamentally cannot bridge a vocabulary gap. A query for “car” won’t lexically match a document that only says “automobile,” even though they mean the same thing.

Semantic search addresses this by representing text as dense vectors (embeddings) that capture meaning rather than exact wording, and retrieving documents whose vectors are closest to the query’s vector in that meaning-space. This is what allows a RAG retriever to find conceptually relevant content even when the surface wording is completely different from the query — the capability we gestured at back in Chapter 1’s comparison of RAG to traditional search.

The trade-off: semantic search can occasionally retrieve content that’s topically similar but not actually what the user needs, since it’s optimizing for meaning-proximity rather than exact term match — and it loses precision on things like exact codes, IDs, or rare proper nouns that lexical search handles perfectly.

Hybrid Search

Given that lexical and semantic search fail in different, largely non-overlapping ways, the natural engineering answer is to combine them — this is hybrid search, and it’s become close to a default choice in production RAG systems.

A typical hybrid setup runs both a lexical retriever (e.g., BM25) and a semantic retriever (dense vector similarity) over the same query, then merges the two ranked lists — often using a fusion method like Reciprocal Rank Fusion (RRF), which combines rankings without needing the raw scores to be on comparable scales.

The practical payoff: hybrid search gets you the exact-match reliability of BM25 (great for names, codes, specific terminology) and the conceptual reach of semantic search (great for paraphrased or loosely worded questions) — covering each other’s blind spots.

Closing Thoughts

Every retriever you’ll build in this series — no matter how modern the embedding model behind it — is doing one of the things covered in this chapter: matching, weighting, ranking, or some hybrid combination of them. Understanding these fundamentals means that when your RAG system retrieves the wrong chunk, you’ll know whether the fix is a scoring problem, a coverage problem, or a fundamentally different retrieval strategy — rather than guessing.