Embeddings from Scratch

§ 01 · HookThe Magic of Semantic Similarity

Run this code:

from sentence_transformers import SentenceTransformer
from sklearn.metrics.pairwise import cosine_similarity

model = SentenceTransformer("all-MiniLM-L6-v2")

sentences = [
    "The dog chased the cat",
    "A puppy ran after a kitten",
    "The stock market crashed today",
]

emb = model.encode(sentences)
sim = cosine_similarity(emb)

print(f"Sentences 1 & 2: {sim[0][1]:.3f}")   # 0.547
print(f"Sentences 1 & 3: {sim[0][2]:.3f}")   # 0.048

Sentences 1 and 2 score 0.547. Sentences 1 and 3 score 0.048, more than ten times lower and close to zero. Here's the part that should make you stop: sentences 1 and 2 share no words at all. "Dog" and "puppy" are different words. "Chased" and "ran after" are different phrases. "Cat" and "kitten" are different tokens. Yet the model clearly places them together, and places the stock market somewhere else entirely.

The mechanism behind this is embeddings: a way of turning meaning into geometry. It's not a niche academic idea. Embeddings sit underneath semantic search engines, every RAG system we'll build in Phase 5, and every vector database in Phase 3. They're also how the Transformer from Article #02 represents every token internally.

§ 02 · The Core IdeaMeaning as Geometry

An embedding is a dense vector (Article #02, §03: a list of decimal numbers where nearly every position carries information) that represents the meaning of a piece of text. Typical sizes run from a few hundred to a few thousand numbers. Every word, sentence, or document can be mapped to a point in this high-dimensional space.

Nobody can picture 384 dimensions, and that's fine; the researchers who build these models can't either (notice that §07 flattens everything to 2D just to put it on a screen). What matters is that distance and direction work the same way in 384 dimensions as on a sheet of paper. One helpful way to think about it: each dimension is roughly one more question the model has learned to answer about the text. Is it formal or casual? Technical or everyday? About a place or a person? Positive or negative? There are hundreds of these "questions" layered together, each adding one coordinate, and in reality no single dimension lines up neatly with one human idea the way "formality" does in this simplified picture.

Similar meanings land at nearby points. Distance in this space is semantic distance.

Here's what that looks like in real word vectors (GloVe, 100 dimensions):

• "king" and "queen" score 0.75. "king" and "carrot" score 0.09.
• "Paris" relates to "France" the way "Berlin" relates to "Germany": the country-to-capital relationship is a consistent direction in space.
• The most famous consequence: king − man + woman ≈ queen. You can do arithmetic on meaning.

import gensim.downloader as api
kv = api.load("glove-wiki-gigaword-100")   # ~130 MB, downloaded once

print(kv.most_similar(positive=["king", "woman"], negative=["man"], topn=1))
# [('queen', 0.7698...)]

print(kv.most_similar(positive=["paris", "germany"], negative=["france"], topn=1))
# [('berlin', 0.8846...)]

print(kv.most_similar(positive=["delhi", "japan"], negative=["india"], topn=1))
# [('tokyo', 0.8396...)]

This isn't hand-waving. Subtract one vector from another, add a third, and the nearest word in the whole vocabulary really is "queen", or "berlin", or "tokyo". Direction in space encodes relationships.

Why engineers care

Embeddings turn a fuzzy problem, "find things that mean the same as this query", into a precise geometric one: find the nearest points in a vector space. No keyword matching, no SQL LIKE, no hand-written synonym lists. Just geometry.

That's exactly what a vector database stores (Phase 3) and what RAG uses to fetch relevant context (Phase 5). Remember the open-book exam from Article #01, §15? Embeddings are how the system finds the right pages of the book.

§ 03 · The FoundationWord2Vec: Where Modern Embeddings Began

In 2013, researchers at Google published Word2Vec, the method that showed word embeddings could be learned at scale. You don't need to implement it, but its core idea answers a question intuition alone can't: why do embeddings work at all?

The distributional hypothesis

Word2Vec rests on an old idea from linguistics, summed up by J. R. Firth in 1957: "You shall know a word by the company it keeps." Words that appear in similar contexts tend to have similar meanings.

Take two sentences: "The cat sat on the mat." and "The dog sat on the rug." "cat" and "dog" appear in almost identical surroundings: both come after "the" and before "sat". After millions of examples like this, the method concludes that these words play the same role, so their vectors should be close. Nobody labelled either one as an "animal". The signal came entirely from context. It's the same trick as Article #01's self-labelling training data: the text supplies its own lessons.

How it learns: skip-gram training

Word2Vec's skip-gram version trains a shallow neural network (a single hidden layer, the opposite of the many-layered networks in Article #01, §05) on a simple task: given a centre word, predict the words around it. The interesting part is what's left over afterwards. The network's weight matrix has one row per word, and after training, those rows are the embeddings. The prediction task is just a way to make useful rows appear.

The limitation that leads to the next step

Word2Vec gives each word one fixed vector. "bank" gets a single embedding forever, whether it appears in "river bank" or "savings bank". This is the polysemy problem (polysemy means one word having several meanings): one vector tries to average all of them. It's the same "bank of the river" puzzle from Article #01, §10, and it's exactly what the Transformer's attention solves.

§ 04 · Scaling UpFrom Word to Sentence Embeddings

Word2Vec gives you one vector per word. But RAG pipelines, semantic search, and document retrieval need one vector per sentence, paragraph, or document.

The naive approach: average the word vectors

Take the vector for every word in a sentence and average them. It works for simple cases, but it has a fatal flaw: word order disappears. "Dog bites man" and "man bites dog" contain the same three words, so they produce the same average. Article #02 showed the same order problem for Transformers, and solved it with position information.

The Transformer solution: contextual embeddings

A Transformer encoder (Article #02, §06) reads the whole sentence at once, with attention. So the same word gets a different vector depending on its neighbours: "bank" next to "river" comes out differently from "bank" next to "account". To squeeze the sentence into one vector, you mean-pool: average all the token vectors number by number, so dozens of token vectors collapse into one sentence vector that blends a bit of everything. Because each token vector already reflects its context, the average keeps far more meaning than Word2Vec averaging did.

Sentence-BERT (2019)

BERT on its own wasn't very good at sentence-level similarity. In 2019, Nils Reimers and Iryna Gurevych fine-tuned it specifically for that job using a siamese network: two identical copies of the same network, sharing the same weights, each reading one sentence of a pair. Think of identical twins who each read a different book and then compare notes on how similar the books felt. Trained on many sentence pairs, the result was a 768-number vector per sentence that captured meaning well across topics. all-MiniLM-L6-v2, the model we use in this article, comes from the same family: a smaller, faster model trained with the same sentence-transformers approach.

§ 05 · Hands-on CodeGenerating Embeddings with sentence-transformers

Install the library and open a Jupyter notebook. We'll build semantic search from the ground up, with no database and no index, just vectors and cosine similarity.

$ pip install sentence-transformers scikit-learn

Encoding, and what comes out

from sentence_transformers import SentenceTransformer
import numpy as np

# Small, fast, and good. Downloads about 90 MB the first time, then it's cached.
model = SentenceTransformer("all-MiniLM-L6-v2")

sentences = [
    "Machine learning is a subset of artificial intelligence",
    "Deep learning uses neural networks with many layers",
    "I love eating pizza on Friday nights",
    "The best pasta recipe uses fresh tomatoes",
]

embeddings = model.encode(sentences)
print(embeddings.shape)
# (4, 384)   4 sentences, 384 numbers each

print(embeddings[0][:6].round(3))
# [-0.046 -0.004  0.07   0.036  0.049 -0.03 ]

print(np.linalg.norm(embeddings[0]).round(3))
# 1.0   this model already scales every vector to length 1

Measuring similarity: cosine similarity

Cosine similarity measures the angle between two vectors and ignores how long they are; only direction matters. It's built on the dot product you met in Article #02, §04. Two vectors pointing the same way score 1.0. Two at a right angle score 0. Two pointing in opposite directions score −1.0.

Picture two people in a room, each pointing at something. Cosine similarity only asks "are they pointing the same way?" It doesn't care if one has a longer arm. That's why it's the standard way to compare embeddings: two sentences can produce vectors of different lengths for incidental reasons, but if they mean the same thing, they point the same way.

from sklearn.metrics.pairwise import cosine_similarity

sim = cosine_similarity(embeddings)

for i, s1 in enumerate(sentences):
    for j, s2 in enumerate(sentences):
        if i < j:
            print(f"{sim[i][j]:.3f} | {s1[:38]:<38} ↔ {s2[:38]}")

# 0.415 | Machine learning is a subset of artifi ↔ Deep learning uses neural networks wit
# 0.016 | Machine learning is a subset of artifi ↔ I love eating pizza on Friday nights
# 0.033 | Machine learning is a subset of artifi ↔ The best pasta recipe uses fresh tomat
# 0.021 | Deep learning uses neural networks wit ↔ I love eating pizza on Friday nights
# 0.121 | Deep learning uses neural networks wit ↔ The best pasta recipe uses fresh tomat
# 0.273 | I love eating pizza on Friday nights   ↔ The best pasta recipe uses fresh tomat

The two AI sentences score 0.415, the two food sentences 0.273, and every tech-versus-food pair lands between 0.016 and 0.121. The within-topic pairs clearly win, even though none of those scores is anywhere near 1.0. That's normal: with this model, closely related sentences often score between 0.3 and 0.7, and unrelated ones near zero. Compare scores with each other, not against a fixed idea of what "similar" should look like.

Semantic search from scratch

This is where it all comes together: a working semantic search engine in a dozen lines. The same pattern is the retrieval half of every RAG system.

query = "how do neural networks learn?"
query_emb = model.encode([query])

documents = [
    "Backpropagation is the algorithm that trains neural networks",
    "Paris is the capital of France",
    "Gradient descent minimises the loss function during training",
    "The Eiffel Tower was built in 1889",
    "Weights in a neural network are updated using gradients",
    "Sourdough bread requires a live starter culture",
]

doc_embs = model.encode(documents)
scores = cosine_similarity(query_emb, doc_embs)[0]

for score, doc in sorted(zip(scores, documents), reverse=True):
    print(f"{score:.3f}  {doc}")

# 0.672  Backpropagation is the algorithm that trains neural networks
# 0.463  Weights in a neural network are updated using gradients
# 0.377  Gradient descent minimises the loss function during training
# 0.088  Paris is the capital of France
# 0.032  Sourdough bread requires a live starter culture
# 0.006  The Eiffel Tower was built in 1889

The top three results are all about how networks train, which is exactly right. And look at the gap: the third-best relevant document (0.377) scores more than four times higher than the best irrelevant one (0.088). The word "learn" doesn't appear in any of them. The ranking comes from meaning, through geometry.

Batch encoding for large datasets

large_corpus = [f"Document number {i}" for i in range(10_000)]

embeddings = model.encode(
    large_corpus,
    batch_size=64,              # encode 64 texts at a time
    show_progress_bar=True,
    normalize_embeddings=True,  # length-1 vectors: cosine similarity = plain dot product
)
print(embeddings.shape)
# (10000, 384)

normalize_embeddings=True rescales every vector to length 1 before you store it. Once every vector has length 1, the cosine formula (which normally divides by both lengths) reduces to a plain dot product, the cheapest comparison there is. This particular model already does that for you, as experiment 1 showed, but many models don't, so it's a good habit. It's the same optimisation the vector databases in Phase 3 rely on.

§ 06 · Production OptionGenerating Embeddings with the OpenAI API

When you move from prototype to production, or when retrieval quality really matters, you may switch to a hosted embedding API. OpenAI's is a common choice, and the interface is simple.

$ pip install openai

from openai import OpenAI

client = OpenAI()  # reads OPENAI_API_KEY from your environment

response = client.embeddings.create(
    model="text-embedding-3-small",
    input=[
        "What is retrieval augmented generation?",
        "How does RAG work in LangChain?",
    ],
)

vectors = [item.embedding for item in response.data]
print(len(vectors[0]))                # 1536
print(response.usage.total_tokens)      # 15   (billed by the token, as in Article #03)

§ 07 · Making It VisibleVisualising Embedding Space

384-dimensional vectors are invisible. PCA (Principal Component Analysis) finds the few directions along which the data varies most and projects everything onto them, so 384 dimensions can be flattened to the 2 you can plot. Some structure is lost, but a surprising amount survives.

$ pip install matplotlib scikit-learn

import matplotlib.pyplot as plt
from sklearn.decomposition import PCA
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")

sentences = [
    "Machine learning models learn from data",          # Tech
    "Neural networks are inspired by the human brain",  # Tech
    "Python is the most popular language for AI",       # Tech
    "Pizza is a beloved Italian dish",                  # Food
    "Pasta with tomato sauce is a classic meal",        # Food
    "Risotto requires constant stirring",               # Food
    "Football is the most popular sport globally",      # Sport
    "A cricket match can last five days",               # Sport
    "Swimming is an excellent full-body workout",       # Sport
]
colors = ["#0E6E62"] * 3 + ["#8A5E0F"] * 3 + ["#B23A26"] * 3   # tech, food, sport

embeddings = model.encode(sentences)
reduced = PCA(n_components=2).fit_transform(embeddings)

plt.figure(figsize=(9, 6))
for i, (x, y) in enumerate(reduced):
    plt.scatter(x, y, color=colors[i], s=130, edgecolors="white", linewidths=1.5)
    plt.annotate(sentences[i][:28], (x, y), fontsize=8, ha="center",
                 xytext=(0, 7), textcoords="offset points")
plt.axis("off")
plt.tight_layout()
plt.savefig("embedding_clusters.png", dpi=150, bbox_inches="tight")
plt.show()

This is what §02 promised: three clean clusters, without any topic labels. For bigger or messier datasets, try t-SNE (t-distributed Stochastic Neighbor Embedding). It's slower and its axes mean nothing on their own, but it's often better at pulling separate groups visibly apart. One catch: its perplexity setting must be smaller than the number of points, so for our nine sentences it has to be tiny.

from sklearn.manifold import TSNE

# perplexity must be less than the number of points. Use ~3 for 9 sentences;
# values between 5 and 50 are typical once you have hundreds of points.
reduced = TSNE(n_components=2, perplexity=3, random_state=42).fit_transform(embeddings)

§ 08 · Engineering DecisionsChoosing the Right Embedding Model

Your choice of embedding model directly affects retrieval quality, speed, cost, and storage. Here's what to weigh:

• Dimensions. More numbers can capture more nuance but cost more to store and search: 384, 768, 1,024, 1,536, 3,072.
• Maximum input length. all-MiniLM-L6-v2 reads at most 256 tokens; all-mpnet-base-v2 reads 384. Anything longer is silently cut off by default, which is one reason RAG systems split documents into chunks (Phase 5).
• Languages. English-only models produce poor vectors for Hindi, Tamil, or Chinese. Article #03 showed how much older tokenisers struggle with Indian scripts, and embedding models inherit their tokeniser's strengths and weaknesses. Use a multilingual model for non-English text, and test it on your own users' languages.
• Task type. Retrieval compares a short query with a long document (asymmetric). Similarity compares two texts of a similar kind (symmetric). A model trained for one can do noticeably worse at the other, because the shapes it learned to compare don't match what you give it. The open-source BGE family (from the Beijing Academy of Artificial Intelligence) is strong at retrieval; classic SBERT models are strong at similarity.
• Benchmarks. The MTEB leaderboard (huggingface.co/spaces/mteb/leaderboard) ranks embedding models across many tasks and languages. The original MTEB benchmark covered 56 datasets, and it has grown a lot since. Check it before picking a production model, then test the top few on your data, because leaderboard rank doesn't always carry over.

§ 09 · SummaryWhat You Now Understand