Free lesson · AI Engineering

What Are Embeddings?

Easy11 min readaillmrag

What Are Embeddings?

The Feature That Made AlgoInsight's Search Actually Work

Early in AlgoInsight's development, we built a search bar for the platform. A student could type "how do I handle race conditions in Go" and the system would find relevant lessons. The first version was keyword search - standard Elasticsearch, matching the words in the query against the words in the lesson content.

It was terrible. Not because Elasticsearch is bad - it is excellent at what it does. It was terrible because the students' queries and the lessons' text used different words for the same concepts. A student would search "how to make API calls faster" and get zero results, even though there was an entire module on latency optimisation. The lesson never used the phrase "make API calls faster." It used "latency reduction," "response time optimisation," and "throughput improvement." Keyword search cannot bridge that gap because it operates on tokens, not on meaning.

The fix was embeddings. We replaced keyword matching with semantic search - converting both the query and the lesson content into numerical representations that capture meaning, not surface text, and then finding the content whose meaning is closest to the query's meaning. "Make API calls faster" and "latency reduction" land in the same region of the embedding space, even though they share zero words.

That upgrade took our search relevance from 61% to 89% - measured as "did the top result answer the student's question?" - and it is the technology that will underpin everything you build in Modules 6 through 8 of this course. Retrieval-augmented generation (RAG), semantic search, recommendation systems, clustering, and classification all sit on top of embeddings. This topic gives you the conceptual foundation; the rest of the module makes it operational.

From Text to Geometry

An embedding is a list of numbers - a vector - that represents a piece of text in a high-dimensional space. The key property: texts with similar meanings get vectors that are close together in this space, and texts with different meanings get vectors that are far apart.

That sentence is the entire concept. Everything else in this topic is unpacking what "close" means, how the vectors are computed, and why this property is useful for engineering.

Let me make it concrete. Imagine you embed three sentences:

"How do I reverse a linked list?" → [0.12, -0.34, 0.87, ... 1,536 numbers]
"Reversing a singly linked list in Java" → [0.11, -0.33, 0.86, ... 1,536 numbers]
"Best restaurants in downtown Portland" → [-0.71, 0.45, -0.22, ... 1,536 numbers]

The first two vectors are nearly identical - their numbers differ by tiny amounts - because the sentences mean nearly the same thing. The third vector is completely different because the sentence is about an unrelated topic. If you could visualise 1,536-dimensional space (you cannot, but the math works the same way), the first two points would be neighbours and the third would be on the other side of the space.

This is not a metaphor. This is the literal mathematical property that embedding models are trained to produce. The model learns, from billions of text pairs, which texts are semantically related and which are not, and it adjusts its internal parameters so that related texts produce nearby vectors. The training objective is geometric: push similar things together, push dissimilar things apart.

Important: Embeddings are not generated by the same models you have been using. GPT-4, Claude, and the other chat models generate text. Embedding models generate vectors. They are different models with different architectures, different APIs, and different pricing. When you call OpenAI's text-embedding-3-small, you are calling a model that is purpose-built for producing embeddings - not a chat model repurposed for the task.

Distance Metrics: Measuring "Close"

Once you have vectors, you need a way to measure how close two vectors are. Three metrics dominate, and understanding when to use each matters for both quality and performance.

Cosine Similarity

The most common metric for text embeddings. Cosine similarity measures the angle between two vectors, ignoring their length. Two vectors pointing in the same direction have cosine similarity 1.0 (identical meaning). Two vectors pointing in perpendicular directions have cosine similarity 0.0 (unrelated). Two vectors pointing in opposite directions have cosine similarity -1.0 (in practice, text embeddings rarely produce negative similarity).

import numpy as np

def cosine_similarity(a: np.ndarray, b: np.ndarray) -> float:
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

# Similar texts: high similarity
sim_related = cosine_similarity(embed("reverse a linked list"), 
                                 embed("reversing a singly linked list"))
# → ~0.92

# Unrelated texts: low similarity  
sim_unrelated = cosine_similarity(embed("reverse a linked list"),
                                    embed("best restaurants in Portland"))
# → ~0.08

Why cosine? Because it is length-invariant - a long document and a short query can be meaningfully compared, since the metric cares about direction (meaning), not magnitude (how many words). This property makes it the default choice for search and retrieval, where queries are short and documents are long.

Dot Product

Cosine similarity without the normalisation step - dot(a, b) instead of dot(a, b) / (|a| * |b|). Faster to compute (no norms), and equivalent to cosine similarity when vectors are already normalised (which many embedding models guarantee). If your embedding model outputs unit-length vectors, dot product and cosine similarity give the same ranking, and dot product is strictly cheaper.

Euclidean Distance

The straight-line distance between two points in the vector space. Lower distance = more similar (opposite convention from cosine, where higher = more similar). Less common for text embeddings because it is sensitive to vector magnitude, but standard in some domains (image embeddings, scientific data).

Metric	When to use	Convention
Cosine similarity	Default for text embeddings, especially search	Higher = more similar
Dot product	When embeddings are pre-normalised (faster)	Higher = more similar
Euclidean distance	Non-text domains, or when magnitude matters	Lower = more similar

Tip: In practice, you will use cosine similarity for almost everything in this course. The vector databases in topic 5.3 default to it, the retrieval pipelines in Module 6 assume it, and the embedding models most commonly used produce vectors optimised for cosine comparison. Learn dot product and Euclidean for completeness; reach for cosine by default.

What Embeddings Capture (and What They Do Not)

The power of embeddings comes from what they encode. Understanding the boundaries prevents a class of bugs where engineers expect embeddings to do work they cannot.

What They Capture Well

Semantic similarity. "Happy" and "joyful" land near each other. "Machine learning" and "ML" land near each other. This is the core use case and the strongest property.

Topical relatedness. A paragraph about PostgreSQL indexing and a paragraph about database query optimisation land in the same region, even if they share few words. The embedding model learned that these topics co-occur and relate.

Analogical structure. The classic example: embed("king") - embed("man") + embed("woman") ≈ embed("queen"). The embedding space encodes relationships between concepts as geometric directions, not just individual concept positions.

What They Capture Poorly

Negation. "The server is running" and "The server is not running" produce vectors that are very close - because the sentences share most of their words and topic. The embedding does not reliably encode the critical semantic inversion. This is a known limitation and it matters for retrieval: a search for "server not running" will retrieve documents about servers running, because the embeddings are similar.

Temporal and logical ordering. "A happened before B" and "B happened before A" produce similar embeddings. The sequential structure is lost in the embedding.

Precise quantitative information. "The latency is 50ms" and "the latency is 5000ms" embed similarly - both are about latency. The specific numbers are not reliably encoded in the vector geometry.

Very long texts. Embedding models have context limits (typically 512 to 8,192 tokens depending on the model). A 10,000-word document must be chunked before embedding, and the chunking strategy - which you will build in topic 5.4 - determines how well the embeddings represent the full document.

graph TD
    A[Input text] --> B[Embedding Model<br/>purpose-built for<br/>vector generation]
    B --> C[Dense vector<br/>e.g. 1536 dimensions]
    C --> D{Comparison<br/>via distance metric}
    D --> E[Cosine similarity<br/>angle between vectors]
    D --> F[Dot product<br/>faster when normalised]
    D --> G[Euclidean distance<br/>straight-line distance]
    
    C --> H[What is captured]
    H --> H1[Semantic similarity]
    H --> H2[Topical relatedness]
    H --> H3[Analogical structure]
    
    C --> I[What is NOT captured]
    I --> I1[Negation]
    I --> I2[Temporal ordering]
    I --> I3[Precise quantities]

This diagram shows the embedding pipeline. Input text passes through an embedding model purpose-built for vector generation, producing a dense vector of, for example, 1536 dimensions. The vector can be compared using cosine similarity, dot product, or Euclidean distance. The diagram also shows what embeddings capture well - semantic similarity, topical relatedness, and analogical structure - and what they do not capture well - negation, temporal ordering, and precise quantities.

Visualising the Embedding Space

You cannot see 1,536 dimensions, but you can project them down to 2 or 3 dimensions using dimensionality reduction techniques (t-SNE or UMAP are the standard tools). The projection loses information - the 2D plot cannot represent all the relationships the full-dimensional space encodes - but it reveals structure that builds intuition.

When you project a set of embeddings from different topics, you see clusters: all the database-related texts cluster together, all the networking texts cluster together, all the career-advice texts cluster together. Within each cluster, more specific sub-clusters emerge: inside the database cluster, PostgreSQL texts separate from MongoDB texts. The embedding space has structure - not random scatter, but organised geometry that reflects semantic relationships.

This visualisation is not just pedagogical - it is a debugging tool. When your semantic search returns irrelevant results, embed the query and the returned documents, project them, and look. If the query lands in the wrong cluster, your query is ambiguous. If the query and a wrong result are neighbours, your embedding model does not distinguish those topics well enough. If the clusters themselves are tangled, your embedding model may be the wrong choice for your domain. Module 13's evaluation discipline applies here too: embeddings are a component with measurable quality, and that quality varies by model, by domain, and by the text you are embedding.

Embedding Dimensions: What 1,536 Numbers Actually Encode

A question engineers frequently ask when they first encounter embeddings: why 1,536 dimensions? Why not 100, or 10,000?

The short answer is that dimensionality is a capacity trade-off. More dimensions give the embedding more "room" to encode distinctions between texts. With 10 dimensions, you can distinguish broad categories (technology vs cooking vs travel) but not fine-grained differences (PostgreSQL vs MySQL). With 1,536 dimensions, the space has enough structure to separate not just topics but subtopics, styles, and nuances - "PostgreSQL query optimisation" lands in a detectably different place from "PostgreSQL backup configuration."

But more dimensions are not free. Each additional dimension adds 4 bytes of storage per vector (at float32 precision). At 1,536 dimensions, each vector is ~6 KB. At 3,072 dimensions, it is ~12 KB. Across a million vectors, that is 6 GB versus 12 GB - the difference between fitting in one server's memory and requiring distributed storage. Distance computation also scales linearly with dimensions: more dimensions means slower similarity calculations, though modern hardware (SIMD instructions, GPU-accelerated search) mitigates this significantly.

The current generation of embedding models offers dimensions ranging from 256 (Nomic Embed at its smallest) to 3,072 (OpenAI text-embedding-3-large). The sweet spot for most production applications is 768–1,536 dimensions: enough capacity to capture the distinctions that matter for retrieval, without the storage and compute overhead of the highest-dimensional models. Topic 5.2 covers the selection framework in detail; here, the point is that dimensions are the capacity of the vector to represent meaning, and like all capacity decisions, oversizing is wasteful and undersizing loses information.

One last nuance worth filing: you cannot interpret individual dimensions. Dimension 742 does not mean "Python-ness" or "urgency." The meaning is distributed across all dimensions collectively - it is the direction the vector points in 1,536-dimensional space that carries meaning, not any individual coordinate. This is different from traditional feature engineering, where each feature has a named interpretation, and it is why embeddings require geometric operations (cosine similarity) rather than element-wise inspection.

The Role of Embeddings in the AI Engineering Stack

Embeddings are the bridge between text (which humans use) and geometry (which computers search efficiently). From this module forward, they appear everywhere:

Semantic search (this module). Convert queries and documents to vectors, find the closest documents. The foundation of the search bar that fixed AlgoInsight.

Retrieval-Augmented Generation (Module 6). The "retrieval" in RAG is embedding-based search. Your RAG pipeline embeds the user's question, retrieves the most relevant document chunks by vector similarity, and feeds them into the LLM's context. Every RAG system you build in this course sits on the embeddings infrastructure from this module.

Classification and clustering. Embed a set of documents, cluster them by similarity, label the clusters. A lightweight alternative to fine-tuning a classifier - and Module 13's evaluation can measure whether it is good enough for your use case.

Anomaly and drift detection. If new content's embeddings land far from your existing clusters, it is either a new topic or an error. Production monitoring systems (Module 15) can use embedding distance as a signal.

graph LR
    E[Embeddings] --> S[Semantic Search<br/>Module 5]
    E --> R[RAG Retrieval<br/>Module 6]
    E --> C[Classification<br/>& Clustering]
    E --> A[Anomaly Detection<br/>Module 15]
    E --> AG[Agent Memory<br/>Module 10]
    
    S --> R
    R --> AG

This diagram shows how embeddings connect to other modules in the course. Embeddings feed into semantic search in Module 5, RAG retrieval in Module 6, classification and clustering, anomaly detection in Module 15, and agent memory in Module 10. Semantic search feeds into RAG retrieval, which feeds into agent memory, showing the progressive dependency chain.

Interview: Embeddings come up in AI Engineering interviews in two forms. The conceptual form: "Explain what an embedding is and why it matters for RAG." The answer is the core property - text to vector, similar meanings to nearby vectors, enabling semantic search over documents the LLM has never seen. The applied form: "Your semantic search returns irrelevant results for negated queries like 'not Python.' Diagnose." The answer reaches for the negation limitation - embeddings encode topic (Python) but not negation (not), so "not Python" retrieves Python content. The fix is either hybrid search (topic 5.5) that combines keyword and semantic matching, or query reformulation (topic 4.4) that rephrases the negation as a positive query for the intended topic.

Exercise: See the Geometry

Exercise: Build intuition for embeddings with hands-on exploration:

Install the OpenAI SDK (or use any embedding provider). Embed 10 sentences: 3 about data structures, 3 about cooking, 3 about travel, and 1 deliberately ambiguous ("Java" - the language or the island?). Compute pairwise cosine similarity. Confirm that within-topic pairs are high (~0.8+) and across-topic pairs are low (~0.2–0.4).

Test the negation limitation: embed "the server is running" and "the server is not running." Compute similarity. If it is above 0.85, you have confirmed the limitation with your own data.

Test the analogy property: embed "king", "queen", "man", "woman." Compute embed("king") - embed("man") + embed("woman") and find which of your embeddings is closest to the result.

Challenge: embed your Module 1 system prompts. Cluster them by cosine similarity. Do they cluster by task type (extraction vs classification vs conversation)? If yes, the embeddings capture your task taxonomy. If not, what does that tell you about how the embedding model sees your prompts?

Challenge 2: compute the cost of embedding 100,000 documents at 500 tokens each using OpenAI's text-embedding-3-small pricing. Compare to the monthly cost of the LLM calls that will use those embeddings in a RAG pipeline. Which component is cheaper by how much?

What You Should Take From This Lesson

An embedding is a vector - a list of numbers - that places text in a geometric space where semantic similarity maps to spatial proximity. Cosine similarity is the default metric for measuring that proximity. Embeddings capture semantic meaning, topical relatedness, and analogical structure well; they capture negation, temporal ordering, and precise quantities poorly. They are the bridge between human language and computational search, and they underpin semantic search, RAG retrieval, classification, and monitoring across the rest of this course. Understanding their strengths tells you what to build on them; understanding their limitations tells you where to add complementary techniques.

Next: the embedding model matters. Different models produce different quality vectors at different costs and dimensions, and the choice affects every downstream component. Topic 5.2 maps the landscape and gives you the selection framework.

Back to AI Engineering