Vector Search Implementations & Considerations

Imagine a table of clinical notes that looks like this.

Note	PID	Date	Text	Correct Answer
N1	P001	2026-03-15	Pt started on lisinopril 10 mg daily for HTN; A1c 8.2% on 3/15, up from 7.1%.	no
N3	P002	2026-02-10	Hives after amoxicillin in 2019. Penicillin allergy documented.	yes
N4	P003	2026-04-01	No known drug allergies. Denies penicillin allergy.	no (denied)
N5	P004	2026-04-20	Mother has a penicillin allergy; patient tolerates amoxicillin.	no (it’s the mother)
N6	P002	2026-05-02	Rule out penicillin allergy; skin testing scheduled.	unknown
N8	P003	2026-06-11	Annual physical. Feels well. Labs pending.	no
N9	P006	2026-06-20	Allergic to PCN (rash as a child).	yes

Our query is “Which notes say the patient is allergic to penicillin?”. Let’s say we know that the correct answers are N3 and N9.

Recap of why LIKE searches would suck

SELECT note_id FROM notes WHERE text ILIKE '%penicillin%';

This returns N3, N4, N5, and N6. That’s one right answer and three wrong ones, and it misses N9 completely. Two separate failures:

• The word shows up without the fact. N4 says “denies penicillin allergy”. N5 is about the patient’s mother. N6 is a maybe. All contain the word “penicillin”.
• The fact shows up without the word. N9 says “PCN”, a common abbreviation for penicillin. LIKE has no idea those are the same thing.

This is why we like Vector Search. It has its own drawbacks tho.

Embedding Models

With Vector Seach you do the hard work upfront. What is this work? In the example above, you turn the notes into embeddings based on some model — you can certainly build one but you download one and use it. Here’s some Python from our “Mary had a little lamb” example earlier.

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")
A, B, C = model.encode(
    [
        "Mary had a little lamb",
        "Mary had a small goat",
        "Mary had a little carburetor",
    ],
    normalize_embeddings=True,
)

A.shape  # (384,) <--- embedding model's dimensions
A @ B    # 0.80700004
A @ C    # 0.6693007
B @ C    # 0.5810183

In the case of the notes table, “PCN” and “penicillin” both show up near “allergy”, “rash”, and “amoxicillin”. Do this for every row in your data, store the vectors, and you’ve got the “vector” half of a Vector Database. Cool beans!

The Query

The query again is: “penicillin allergy”. We calculate these with the same 384-dimensional embedding model from our toy example.

note	text	cosine similarity	distance
N3	penicillin allergy documented	0.995	0.10
N9	allergic to PCN	0.990	0.14
N8	annual physical	0.616	0.88

N9 is now a near-perfect match even though it shares no words with the query!

This is Clustering

There is no semantic consideration here. Vector Search is a clustering technique. The above looks very similar to kNN! See the section below for why Vector Search doesn’t ‘understand’ semantics.

What Embeddings Miss

Embeddings can miss logical modifiers that flip a sentence’s meaning. In our baby example earlier, if you searched “Mary did not have a big lamb”, you’d likely get back “Mary had a little lamb”! In the case of notes:

Modifier	Example	What actually matters
Negation	”Denies penicillin allergy”	it’s a no
Who it’s about	”Mother has a penicillin allergy”	it’s not the patient
Certainty	”Rule out penicillin allergy”	nobody knows yet
Time	”Had a penicillin allergy as a child, since resolved”	not anymore
Numbers	”A1c 8.2%” vs. “A1c 7.1%“	very different clinically

To illustrate, here’s Cosine Similarity to “penicillin allergy”:

note	text	similarity	allergic?
N3	Hives after amoxicillin in 2019. Penicillin allergy documented.	0.992	yes
N9	Allergic to PCN (rash as a child).	0.990	yes
N4	No known drug allergies. Denies penicillin allergy.	0.938	no: denied
N5	Mother has a penicillin allergy; patient tolerates amoxicillin.	0.907	no: the mother

N4 and N5 score really high even though they’re wrong. They’re about penicillin allergies, and that’s all the vector captures.

On a negation benchmark (NevIR), most retrieval models did no better than chance!

What can we do? Use vectors to retrieve a short list of candidates, and then let an LLM read those few and decide. This is called Retrieval Augmented Generation (RAG). More later.

---

Speeding Things Up

Exact Search can be very slow. The obvious way to find the nearest vectors is to compute the distance from the query to every single row, then sort. This is called exact, brute-force, or flat search. Say you have 1,000,000 chunks of text, each a 384-number vector. One distance takes 384 multiplications (and additions). So one query is

$$1{,}000{,}000 \times 384 \approx 384 \text{ million multiply-adds}$$

May be fine for one query but not for 1,000 users searching at once, or for a billion rows. And it grows in a straight line: double the data, double the time. It’s like finding the person in a stadium who looks most like you by walking up to every single seat!

What can we do?

Approximate Nearest Neighbors (ANN)

ANN search makes a deal: give up the guarantee of finding the exact best answer, in exchange for being way faster. You build an index (a helper data structure) ahead of time, and at query time you only look at a small fraction of the rows.

How do you know if the approximation is any good? One metric is Recall@k which is the share of the true top $k$ that the index actually returned.

How do you know the true top $k$?

Run exact brute-force search offline on a sample of queries (a few hundred to a few thousand). It’s slow, but you pay that cost only once at evaluation time, never when serving. Public benchmarks (SIFT1M, GloVe, ann-benchmarks) ship with this ground truth precomputed.

So for example,

Exact search says the true top 10 are rows #1 through #10.
The index returns 10 rows: 9 of those, plus 1 that’s actually #14.
Recall@10 = $\frac{9}{10} = 0.90$.

As a rule of thumb, for most applications, 0.95+ recall at 100x the speed is a great tradeoff. Again, it depends.

Here are a few families of ANN indexes.

Family	TLDR	Analogy
Clustering (e.g. IVF)	group vectors into buckets; only search the buckets nearest the query	only check the library aisles that match your topic
Hashing (e.g. LSH)	a hash function that sends nearby vectors to the same bin	sorting mail by zip code
Compression (e.g. PQ)	store a shortened version of each vector so comparisons are cheaper	comparing thumbnails instead of full photos
Graphs (e.g. HNSW)	link each vector to its neighbors; walk the links toward the query	asking friends-of-friends for directions

Let’s look at the last one. It’s all graphical and nice.

ANN-HNSW: Hierarchical Navigable Small World

Lots of words! Let’s break it down.

“Small World” — This is the Six Degrees of Separation idea. You’ve heard that any two people on Earth are connected by about six handshakes. That’s a small world network: even though it’s huge, you can get from anywhere to anywhere in a few hops, because some people have friends who are far away.

“Navigable” — You build a graph where every vector is a node and and is linked to a small handful of nearest neighbors. You then use Greedy Search:

1. Start at some node.
2. Look at all its neighbors. Compute each one’s distance to the query.
3. If one of them is closer to the query than where you are, hop to it.
4. Repeat until none of your neighbors is closer. You’re done: that’s your answer.

So say you’re lost in a city and want to get to the stadium. At every intersection, take whichever street gets you closest to it. You never look at a map of the whole city, just the streets right in front of you.

Problem: if you start far away and every link is short, it takes forever. Like crossing the country using only local streets.

This is where you add Hierarchies as ‘highways’! You build several layers of graphs all stacked upon each other.

• Layer 2 (top) contains just a few nodes, with very long links. Interstate highways.
• Layer 1 contains only some of the nodes, with longer links. Main roads.
• Layer 0 (bottom) contains every node, linked to its close neighbors. Local streets.

To search, start at the top and drive on the highway until you’re close, then exit onto main roads, then local streets.

How does a node decide how high it goes? Randomly, like flipping a coin when it’s added. Every node is on layer 0. Roughly half (or fewer) also go on layer 1. Half of those also go on layer 2. And so on. So each layer up has way fewer nodes. (If you know what a skip list is, it’s the same trick.)

Example

To keep it simple, pretend each “vector” is just one number, and distance is just the difference between two numbers. Here are 13 stored points:

P1	P2	P3	P4	P5	P6	P7	P8	P9	P10	P11	P12	P13
0.05	0.15	0.22	0.31	0.40	0.48	0.55	0.61	0.67	0.74	0.82	0.90	0.97

And here’s the HNSW index built over them:

Query: $q = 0.70$. Which point is nearest? We always start at the top layer’s entry point, P2.

Layer	Standing at	Neighbors checked (distance to 0.70)	Move?
2	P2 (0.55 away)	P11: 0.12	yes → P11
2	P11 (0.12)	P2: 0.55	no. Drop down a layer
1	P11 (0.12)	P8: 0.09	yes → P8
1	P8 (0.09)	P5: 0.30, P11: 0.12	no. Drop down a layer
0	P8 (0.09)	P7: 0.15, P9: 0.03	yes → P9
0	P9 (0.03)	P8: 0.09, P10: 0.04	no. Done: P9

Answer: P9 (0.67), which is in fact the true nearest point. We computed distances to 7 of the 13 points.

Now saving 6 doesn’t sound like much. But because each layer has far fewer nodes than the one below, the number of hops grows roughly like $\log(N)$, not $N$. For a million points, that’s the difference between a million comparisons and a few thousand.

Why It’s “Approximate”

In our 1-D toy example, greedy walking always works. In 384 dimensions, it can get stuck: you reach a node where none of its neighbors are closer to the query, but a closer node exists somewhere else in the graph that you can’t see from where you are.

One fix is to not be so greedy on the bottom layer. Instead of only tracking the single best node, keep a list of the best candidates found so far and keep exploring their neighbors too. The size of that list is called ef_search. It also lets you return $k$ results instead of just 1!

Example: from a database search with 50,000 random vectors and $k=10$, you can get:

ef_search	10	32	256
Recall@10	0.76	0.96	1.00

You see the speed-vs-accuracy tradeoff here. What is that table saying?

With k = 10 and ef_search = 32, HNSW finds, on average, 96% of the true 10 nearest neighbors.

This may be a reasonable value of $k$; going to 256 might be slow/expensive. Note that ef_search is not the only “knob” you tweak!

Knob	What it controls	Turn it up and…
M	how many links each node gets	better recall, more memory, slower to build
ef_construction	how hard to look for good neighbors while building	better graph, slower to build
ef_search	how many candidates to keep while searching	better recall, slower queries

Once again: the tradeoff is computational cost versus recall.

Pros and Cons

Good	Bad
very fast queries (roughly $\log N$)	uses a lot of memory: stores every vector plus all the links
high recall with the right settings	approximate: can miss true neighbors
you can add new vectors without rebuilding	deletes are awkward
works with any distance function	the index can run before your WHERE filters (next section)

Pre and Post Filtering Gotcha

Say you only want notes from April onward.

SELECT note_id, array_cosine_similarity(embedding, $q) AS sim
FROM notes
WHERE note_date >= DATE '2026-04-01'
ORDER BY sim DESC
LIMIT 3;

• Without an index, you’ve got exact/flat search. The backend may filter to April+ notes and will then rank them. You get N9, N6, N4. Three rows, as asked.
• With an HNSW index, the index runs first. It returns the 3 nearest notes overall, N3, N9, N6. Then the WHERE throws out N3 because it’s from February. You get N9, N6. Only two rows, even though you asked for three and a valid third one (N4) exists!

This is called post-filtering: search first, filter after. Pre-filtering is filter first, then search.

What can you do?

• Over-fetch: ask the index for more than $k$ (say, 30), filter, then keep the top 3.
• Filter first, search exactly: if the filter leaves only a few thousand rows, brute-force is plenty fast.
• Partition: e.g. keep one index per patient, so “this patient’s notes” is never a filter at all.

Some actual queries in the next section.