Introduction

Keyword search is cheap, dumb, fast.

Asking LLMs about every row is smart, slow, expensive.

What’s a Middle Path, padawan? Vector Search! It’s not a panacea and has its issues. Here’s the recipe:

1. Turn every row (note, document, chunk of text) into a list of numbers, a vector, using an embedding model which gives these vectors a place in some $n$-dimensional space.
   Texts with similar ‘meanings’ get similar numbers.
2. Turn your question into a vector using the same model.
3. Find the $k$ stored vectors ‘closest’ or most similar to the question’s vector. Return those rows.

Note that you create this vector index of your rows of data first. That’s the upfront computational cost you’ll pay. No free lunches here (well maybe with Keyword Search but it’s dumb.)

Turning Text into Vectors

Yeah sure it’s taking “Mary had a little lamb” turning it into some array of numbers. But how do you accomplish this?

The Setup

Let’s take three sentences:

• A: “Mary had a little lamb”
• B: “Mary had a small goat”
• C: “Mary had a little carburetor”

Any human can tell you A and B mean nearly the same thing and C is nonsense. But C shares more words with A than B does. We need a method that puts A closer to B than to C.

Baby Steps: One Word, One Dimension

Let’s say your vocabulary (the list of words you’re willing to deal with) has $|V| = 30{,}000$ words. Give each word its own axis and you end up with a 30,000-dimensional space. The word “lamb” becomes a vector that’s all zeros except for a single $1$ in the “lamb” slot:

$$\mathbf{e}_{\text{lamb}} = [0, 0, \dots, 0, \underbrace{1}_{\text{lamb}}, 0, \dots, 0] \in \mathbb{R}^{30{,}000}$$

This is called a one-hot vector ($\because$ only one slot is “hot”). A sentence is just the sum of its words’ one-hot vectors (a “bag of words”). Each of our sentences has five words, so each sentence vector has length $5$.

Now you count all the shared words between a pair of sentence vectors get a baby measure of similarity.

$$\begin{aligned} \text{Similarity}(\mathbf{A}, \mathbf{B}) & = \frac{|\{\text{mary}, \text{had}, \text{a}\}|}{5} & = \frac{3}{5} = 0.6 \\ \text{Similarity}(\mathbf{A}, \mathbf{C}) & = \frac{|\{\text{mary}, \text{had}, \text{a}, \text{little}\}|}{5} & = \frac{4}{5} = 0.8 \end{aligned}$$

Well that’s shitty. The carburetor won! What happened? Every pair of different words is at a right angle (orthogonal) in our embedding space¹, so

• “lamb” is exactly as similar to “goat” as it is to “carburetor”
  (Zero, both times.)
• “little” has nothing to do with “small”

So in our baby scheme,

$$\text{Similarity}(\mathbf{e}_{\text{lamb}}, \mathbf{e}_{\text{goat}}) = 0 = \text{Similarity}(\mathbf{e}_{\text{lamb}}, \mathbf{e}_{\text{carburetor}})$$

This is the problem with Keyword Search expressed all fancy vector-like! Also, you create a lot of sparse vectors: lots of dimensions/slots but all except one are zero. Let’s do a little better.

A Toy Space

What if each slot/axis/dimension meant something?

Let’s say we hand-pick four dimensions based on the classes of things: $[\text{animal}, \text{size}, \text{farm}, \text{machine}]$. And every word in our text gets a score $(0,1) \in \mathbb{R}$

“Mary had a” is in all three sentences, so ignore it. Lets say that we type that “lamb” is very much an animal (0.9), kind of small (0.6), very farm-y (0.8), and definitely not a machine (0.0). We do this for all other words.

$$\begin{aligned} \text{little} & = [0.0, 0.9, 0.0, 0.0] \\ \text{small} & = [0.0, 0.8, 0.0, 0.1] \\ \text{lamb} & = [0.9, 0.6, 0.8, 0.0] \\ \text{goat} & = [0.9, 0.1, 0.8, 0.0] \\ \text{carburetor} & = [0.0, 0.1, 0.0, 0.9] \end{aligned}$$

Now words have neighbors:

$$\begin{aligned} \text{Similarity}(\text{little}, \text{small}) &\approx 0.992 \quad \text{(nearly identical)} \\ \text{Similarity}(\text{lamb}, \text{goat}) &\approx 0.929 \quad \text{(very similar)} \\ \text{Similarity}(\text{lamb}, \text{carburetor}) &\approx 0.049 \quad \text{(basically unrelated lol)} \end{aligned}$$

Where do those numbers come from?

I said $\text{Similarity}()$ above but that’s the Cosine Similarity, a commonly used measure. There are others.

$$\cos(\mathbf{a}, \mathbf{b}) = \frac{\mathbf{a} \cdot \mathbf{b}}{\lVert \mathbf{a} \rVert \, \lVert \mathbf{b} \rVert}$$

Step 1: Dot product. Multiply matching slots, then add:

$$\begin{aligned} \text{lamb} \cdot \text{goat} & = (0.9)(0.9) + (0.6)(0.1) + (0.8)(0.8) + (0.0)(0.0) \\ & = 0.81 + 0.06 + 0.64 + 0 \\ & = 1.51 \end{aligned}$$

Step 2: Length of each vector. Square each slot, add, take the square root (this is the Euclidean Vector Norm $\lVert\mathbf{v}\rVert_2$; there are others.)

$$\begin{aligned} \lVert \text{lamb} \rVert & = \sqrt{0.9^2 + 0.6^2 + 0.8^2 + 0.0^2} = \sqrt{0.81 + 0.36 + 0.64} = \sqrt{1.81} \approx 1.345 \\ \lVert \text{goat} \rVert & = \sqrt{0.9^2 + 0.1^2 + 0.8^2 + 0.0^2} = \sqrt{0.81 + 0.01 + 0.64} = \sqrt{1.46} \approx 1.208 \end{aligned}$$

Step 3: Divide.

$$\cos(\text{lamb}, \text{goat}) = \frac{1.51}{1.345 \times 1.208} = \frac{1.51}{1.626} \approx 0.929$$

Most of that $1.51$ comes from the “animal” ($0.81$) and “farm” ($0.64$) slots, where both words score high. The “small” slot barely helps ($0.06$) because goats aren’t especially small. Do the same with “carburetor” and the dot product is just $(0.6)(0.1) = 0.06$, since carburetors score zero on animal and farm. That’s why it lands at $0.049$!

Now make a sentence vector by averaging its word vectors (ignore the common “Mary has a”)

$$\begin{aligned} \mathbf{A} & = \tfrac{1}{2}(\text{little} + \text{lamb}) & = [0.45, 0.75, 0.40, 0.00] \\ \mathbf{B} & = \tfrac{1}{2}(\text{small} + \text{goat}) & = [0.45, 0.45, 0.40, 0.05] \\ \mathbf{C} & = \tfrac{1}{2}(\text{little} + \text{carburetor}) & = [0.00, 0.50, 0.00, 0.45] \end{aligned}$$

Why average?

It’s the simplest way to squash a sentence of any length into one vector.

• Fixed size. A 3-word sentence and a 300-word note both need to become one vector of the same dimension, or you can’t compare them.
• Same space. The average is still a point in $[\text{animal}, \text{size}, \text{farm}, \text{machine}]$, so you can compare a sentence to a word or another sentence.
• Middle of the meanings. The average sits at the centroid of its words. “Little lamb” lands between “small-ish” and “animal/farm-y”, which is roughly what it means!

Why not just add them up? For Cosine Similarity it makes no difference since Cosine ignores length:

$$\cos(\mathbf{a} + \mathbf{b}, \mathbf{c}) = \cos\left(\tfrac{1}{2}(\mathbf{a} + \mathbf{b}), \mathbf{c}\right)$$

It matters for the Dot Product and Euclidean Distance though: summing makes longer sentences “bigger” and they’d look closer to everything. Averaging removes that.

What you lose: word order. “Dog bites man” and “man bites dog” get the exact same vector.

Real models still average (mean pooling, see “Under the hood” below), but they average the transformer’s context-aware vectors. Each token has already soaked up its neighbors, so order and context are baked in before the averaging happens. Some models skip averaging altogether and use a special token’s vector (like [CLS]) as the summary of the whole sentence.

And the ranking flips!

$$\begin{aligned} \text{Similarity}(\mathbf{A}, \mathbf{B}) &\approx 0.966 \\ \text{Similarity}(\mathbf{A}, \mathbf{C}) &\approx 0.580 \end{aligned}$$

B has fewer words in common with A but wins, because “small” sits next to “little” and “goat” sits next to “lamb”. C still gets some credit: it really does share “little”. Similarity is only as good as what the dimensions capture.

We now have dense vectors. Fixed, smaller number of dimensions, and almost every entry is non-zero. Woo!

But we did the hard work of saying “Lamb is 0.9 similar to goat and 0.0 similar to carburetor”. We can’t do this for all the words.

How can we be lazy and use computer?

The Real Space: Learned Embeddings

An embedding model is a neural network trained so that texts with similar meaning get nearby vectors. It decides on its own what the dimensions are. It is unsupervised.

Here are some popular and small models.

Model	Dimensions	What it was trained on
OpenAI text-embedding-3-small	1,536	not disclosed
MedCPT	768	PubMed search logs
all-MiniLM-L6-v2	384	1 billion general sentence pairs2

Look at MedCPT: you can train specialized embedding models too! So you don’t have to use the general purpose OpenAI text model, for example, when you are dealing with scientific contexts or clinical notes.

So with that OpenAI model, “Allergic to PCN” might become in the models embedding space of 1,536 dimensions.

$$[0.021, -0.113, 0.087, \dots] \in \mathbb{R}^{1,536}$$

Important Things

• You can’t just pick a model and use it blindly! What they are trained on is super-important!
• No single number in a vector means anything!
  Only the distances between vectors mean something.

The cool thing is that the network learns synonyms for free. How? This is via the Distributional Hypothesis: “You shall know a word by the company it keeps.”

During training,

• “Lamb” and “goat” both show up next to “farm”, “graze”, “milk”, and “kid”, so the model nudges their vectors toward each other.
• “Little” and “small” can be swapped in almost any sentence, so same deal.

And so on. Repeat this a few billion times and similar things end up close together. See the last section on this page for how this works (i.e. how you end up with that embedding vector above.)

Under the hood

A model like this:

1. Chops text into tokens.
   Tokens are words or pieces of words. “Mary had a little lamb” becomes [mary, had, a, little, lamb]. The cool thing is that a rare word like “carburetor” might become [car, ##bur, ##etor], so the model never hits an “unknown word”. There’s a lot more action going on here but that’s the gist.
2. Looks up a starting vector for each token in a big learned table.
3. Runs them through a transformer, which adjusts each token’s vector based on the words around it. That’s how “bank” in “river bank” and “bank account” end up different.
4. Averages the token vectors into one vector for the whole sentence (called mean pooling).
5. Scales it to length 1 (called normalizing). You do this because you get similarity for the price of a simple dot product.

Measuring Similarity

Once everything is a vector, “similar” means “close”. There are three common ways to measure closeness. See this page for Cosine Similarity, Euclidean Distance, and Dot Products.

Length One Vectors

The Trick

You use length-one vectors so you get Cosine Similarity for cheap with simple Dot Products (higher is better):

$$\lVert \mathbf{a} \rVert = \lVert \mathbf{b} \rVert = 1 \implies \mathbf{a} \cdot \mathbf{b} = \cos\theta \qquad \text{(Higher is more similar)}$$

You can also get Euclidean Distance like so (lower is better):

$$\lVert \mathbf{a} - \mathbf{b} \rVert = \sqrt{2 - 2\cos\theta} = \sqrt{2 - 2\cdot\mathbf{a}\cdot\mathbf{b}} \qquad \text{(Lower is more similar)}$$

Let’s say length-one query $\mathbf{q} = (0.8, 0.6)$. Let’s say we have three vectors stored in our database, all with length 1:

	Vector	Cosine	Euclidean	Inner/Dot Product
A	(1, 0)	0.80	0.63	0.80
B	(0.6, 0.8)	0.96	0.28	0.96
C	(0, 1)	0.60	0.89	0.60

Let’s look at how $\mathbf{B}$ does agaisnt the query.

• Cosine for B: $\frac{0.8 \times 0.6 + 0.6 \times 0.8}{1 \times 1} = 0.96$ which is also the Inner/Dot product.
• Euclidean distance for B: $\sqrt{(0.8 - 0.6)^2 + (0.6 - 0.8)^2} = \sqrt{0.08} \approx 0.28$.

All three agree that B is closest. That’s not a coincidence! When every vector has length 1, all three measures give the same ranking.

This is the Math part. Let’s look at a real-enough example of using these to Deliver Shareholder Value™.

---

Footnotes

1. Easier to imagine this in three dimensions. Go ahead and close your eyes and see it. ↩

2. TODO: WTF does “sentence pairs” mean? ↩