Embeddings Explained: How Machines Turn Meaning Into Numbers

Embeddings

Developer Diaries: Building with Retrieval-Augmented Generation

In previous post, we drew a line between lexical search (matching words) and semantic search (matching meaning), and promised that meaning-based matching deserved its own deep dive.

Embeddings are the single idea that makes semantic search, and by extension most of modern RAG, possible. Once you understand what an embedding actually is — not just “a vector,” but what that vector represents and why distances between vectors mean something — a huge amount of what feels like magic in RAG systems becomes mechanical and predictable.

What Are Embeddings?

An embedding is a numerical representation of a piece of text (a word, sentence, or document) as a list of numbers — a vector — typically with anywhere from a few hundred to a few thousand dimensions.

The important part isn’t the numbers themselves; it’s what they’re constructed to preserve: pieces of text with similar meaning get vectors that are close together, and pieces of text with different meaning get vectors that are far apart.

"a happy dog"      ──▶  [0.21, -0.08, 0.44, ...]
"a joyful puppy"   ──▶  [0.19, -0.11, 0.41, ...]   ← close to the above
"quarterly taxes"  ──▶  [-0.55, 0.62, -0.02, ...]  ← far from both

An embedding model is trained specifically to produce this property. Feed it text, and it outputs a vector — that’s the entire interface. Everything downstream (search, clustering, recommendation) is just geometry performed on these vectors.

Embedding Space and Semantic Meaning

All the vectors an embedding model produces live in the same embedding space — a high-dimensional coordinate system where every possible piece of text has a location. Think of it as a map, except instead of two dimensions (latitude, longitude) it might have 768 or 1536.

What makes this map useful is that distance in this space corresponds to semantic similarity. Synonyms cluster together. Related concepts sit near each other even without shared vocabulary — “physician” and “doctor” end up close, and so do “physician” and “hospital,” even though none of those words overlap textually. Unrelated concepts end up far apart.

This is precisely the property lexical search (Chapter 2) lacks. TF-IDF and BM25 can only ever measure word overlap; embedding space measures something closer to conceptual proximity. That’s the whole reason semantic search can answer “how do I fix a flat tire” using a document titled “roadside puncture repair guide” — zero shared words, but nearly identical location in embedding space.

It’s worth being precise about what “meaning” means here, though: an embedding model doesn’t understand text the way a human does. It’s learned statistical regularities — which words and phrases tend to appear in similar contexts — and encoded those regularities as geometry. That’s usually a very good proxy for meaning, but it’s a learned approximation, not true comprehension, and it can be fooled by unusual phrasing or domain-specific jargon the model never saw enough of during training.

Dense vs Sparse Embeddings

Embeddings come in two structurally different flavors, and it’s easy to conflate them if you’ve only worked with one.

Sparse embeddings are long vectors — often as long as the entire vocabulary, sometimes tens of thousands of dimensions — where almost every value is zero. Each dimension typically corresponds to a specific word or token, and a non-zero value means that word is present (and how strongly, e.g. via TF-IDF or BM25 weighting). Classical IR scoring, from Chapter 2, effectively operates over sparse vector representations, even though it’s rarely described that way. Sparse embeddings are interpretable — you can look at which dimension fired and know exactly which word caused it — but they inherit the same vocabulary-gap problem as lexical search.

Dense embeddings are short vectors — typically 256 to 4096 dimensions — where nearly every value is non-zero, and no single dimension corresponds to a specific word. The meaning is distributed across the whole vector; no individual number is individually interpretable. This is what modern neural embedding models (which we’ll cover by name in Chapter 5) produce, and it’s what enables the synonym- and paraphrase-bridging behavior described above.

	Sparse	Dense
Typical size	Tens of thousands of dims	256–4096 dims
Non-zero values	Very few	Nearly all
Interpretability	High (maps to specific words)	Low (distributed meaning)
Handles synonyms/paraphrase	Poorly	Well
Exact term/code matching	Excellent	Weaker

This is the same trade-off from Chapter 2’s lexical-vs-semantic discussion, just restated at the representation level — which is why hybrid search (combining both) remains a common production pattern rather than a stopgap.

Contextual Embeddings

Early embedding approaches assigned one fixed vector per word, regardless of context — the word “bank” got the same vector whether it appeared in “river bank” or “savings bank.” This is an obvious problem: the same word can mean entirely different things depending on surrounding context.

Contextual embeddings, introduced by transformer-based models, solve this by generating a different vector for a word depending on the sentence it appears in. “Bank” in “I sat by the river bank” and “bank” in “I deposited money at the bank” now get distinct embeddings, positioned near “riverside” and “financial institution” respectively, because the surrounding words shape the representation.

This shift — from static, one-vector-per-word representations to context-dependent ones — is arguably the single biggest leap in embedding quality over the last decade, and it’s the foundation every modern embedding model (which we’ll name explicitly in Chapter 5) is built on.

Sentence, Passage, and Document Embeddings

So far we’ve talked about embeddings somewhat abstractly, but in a RAG system you rarely embed single words — you embed whole units of retrievable text. The terminology shifts slightly depending on the size of that unit, and the distinction matters for how well retrieval performs.

Sentence embeddings represent a single sentence as one vector, capturing its overall meaning rather than word-by-word detail. Useful for fine-grained retrieval — matching a specific claim or question to a specific sentence.

Passage embeddings represent a chunk of text — a paragraph or a handful of sentences — as one vector. This is the unit most RAG systems actually retrieve against, since it balances enough context to be meaningful against small enough size to stay focused (we’ll cover chunking strategy, and why chunk size matters enormously, in a later chapter).

Document embeddings represent an entire document as a single vector. This is useful for coarse-grained tasks like document clustering or routing a query to the right document before a finer passage-level search — but a single vector for an entire long document tends to blur together many different sub-topics, which is usually too lossy for precise retrieval on its own.

The general pattern in production RAG: embed at the passage level for retrieval precision, and reserve document-level embeddings for coarser filtering or routing steps.

Query Embeddings

The user’s question also has to become a vector before it can be compared against anything — this is the query embedding, and it’s usually produced by the same embedding model used for the documents, so that both live in the same space and distances between them are meaningful.

One subtlety worth knowing: some embedding models are trained asymmetrically — they use a different internal representation for “this is a query” versus “this is a passage to be searched,” even though both come out as vectors of the same size. This is because a question and its answer often don’t look linguistically similar even when they’re a great semantic match (“What causes rust?” versus “Iron oxidizes in the presence of oxygen and moisture”) — models trained with this query/passage distinction in mind tend to retrieve noticeably better than ones that treat both as generic, interchangeable text.

Embedding Dimensions

The dimensionality of an embedding is simply how many numbers make up its vector — 384, 768, 1024, and 1536 are all common sizes depending on the model.

More dimensions generally mean more capacity to represent nuanced distinctions in meaning — but this comes at a real cost: higher storage requirements, slower similarity computations at scale, and, past a certain point, diminishing returns on retrieval quality (a phenomenon related to the “curse of dimensionality,” where distances between points become less discriminative as dimensions increase without bound).

In practice, dimensionality is a fixed property of whichever embedding model you choose — it’s not something you tune independently — so this becomes a factor in model selection (covered in depth in Chapter 5) rather than something to optimize on its own. Techniques like dimensionality reduction or model-native “Matryoshka” embeddings (which support truncating a vector to a smaller size with graceful quality loss) are increasingly common ways to trade off storage and speed against retrieval quality without switching models entirely.

Similarity Search

Once documents and queries are both vectors in the same space, retrieval becomes a geometry problem: find the document vectors closest to the query vector. Three distance/similarity measures dominate in practice.

Cosine similarity measures the angle between two vectors, ignoring their magnitude — it asks “do these two vectors point in the same direction?” rather than “how long are they?”

\[\text{cosine similarity}(A, B) = \frac{A \cdot B}{\|A\| \, \|B\|}\]

Dot product is the raw, unnormalized version of the same idea — it factors in vector magnitude as well as direction, which some embedding models are specifically trained to exploit (a longer vector can represent a more “confident” or emphatic match).

\[A \cdot B = \sum_{i=1}^{n} A_i B_i\]

Euclidean distance measures straight-line distance between two points in the vector space, the same way you’d measure distance on a 2D map, just generalized to many dimensions.

\[d(A, B) = \sqrt{\sum_{i=1}^{n} (A_i - B_i)^2}\]

Cosine similarity is the most common default for text embeddings, because it’s insensitive to vector length — which matters since embedding magnitude often reflects something incidental (like text length) rather than meaning. Which measure to use is typically dictated by the specific embedding model’s training setup — using the wrong one can silently degrade retrieval quality even though nothing throws an error.

Embedding Drift

A subtlety that catches a lot of production RAG systems off guard: embeddings are not a stable, universal coordinate system. Two different embedding models — or even two versions of the same model — will place the same piece of text in different locations in their respective spaces. There’s no shared “true north.”

This has a very concrete practical consequence, sometimes called embedding drift: if you re-embed your document collection with an upgraded model version but forget to re-embed something (or mix vectors from two model versions in the same index), similarity comparisons become meaningless — you’d be measuring distance between vectors that were never placed in a shared space to begin with. It’s a silent failure mode: nothing crashes, retrieval just quietly gets worse, because the geometry underlying “closeness” no longer means what it’s supposed to.

The practical rule: whenever you change embedding models, re-embed your entire collection — queries and documents both — rather than assuming compatibility.

Evaluating Embedding Quality

Given that embedding quality directly determines retrieval quality (garbage geometry means garbage nearest-neighbor results), it’s worth knowing how the field actually measures whether an embedding model is any good, rather than trusting benchmarks blindly.

MTEB (Massive Text Embedding Benchmark) — the de facto standard leaderboard, evaluating embedding models across dozens of tasks (retrieval, classification, clustering, semantic similarity) and multiple languages. When you see an embedding model advertised with a benchmark score, it’s very often an MTEB result.
Retrieval-specific metrics — recall@k (of the top-k retrieved results, how many are actually relevant — an extension of the recall concept from Chapter 2) and nDCG (Normalized Discounted Cumulative Gain, which rewards relevant results appearing higher in the ranking, not just present somewhere in the list).
Task-specific evaluation — general benchmarks like MTEB are a useful signal, but they don’t guarantee a model performs well on your domain-specific text (legal documents, medical records, internal codebases). Building a small evaluation set from your own data — even 50–100 representative query/relevant-document pairs — is the most reliable way to know whether an embedding model actually works for your use case before committing to it in production.

The theme worth carrying forward: embedding quality isn’t something you eyeball from a demo working well on a few example queries — it’s something you measure, the same disciplined way Chapter 2 taught us to measure retrieval quality generally.

Closing Thoughts

Embeddings are the translation layer between human language and the geometry that makes semantic search possible. Every concept in this chapter — dense versus sparse representations, contextual embeddings, similarity measures, drift — shows up directly in the design decisions you’ll make when building a RAG pipeline: which model to embed with, what to embed at what granularity, and how to know if it’s actually working.