Retrieval-Augmented Generation (RAG)

Use vector search to find the relevant passages, paste them into an LLM’s prompt, and tell it to answer only from them.

Question: “Does P002 have a penicillin allergy?”

1. Retrieve. Turn the question into a vector. Find the nearest 2 chunks: N3 and N6. (N8, “Annual physical”, is far away.)

2. Augment. Build a prompt:

   Answer only from the notes below. Cite note IDs.
   If they do not say, say so.
   
   [N3, 2026-02-10] Hives after amoxicillin in 2019. Penicillin allergy documented.
   [N6, 2026-05-02] Rule out penicillin allergy; skin testing scheduled.
   
   Question: Does P002 have a penicillin allergy?

3. Generate. The LLM answers: “A penicillin allergy is documented after hives with amoxicillin in 2019 [N3]. Skin testing is scheduled to rule it out [N6].”

The LLM handles the logic that vectors can’t (negation, uncertainty), and it (hopefully) answers from your data instead of whatever it memorized during training.

Garbage In, Garbage Out

RAG is only as good as its retrieval. If the right note isn’t in the top $k$, the LLM never sees it.

Design Considerations

Chunking

A long note crammed into a single vector gets “averaged out”. Take this discharge summary:

Admitted with chest pain. Troponin negative. History: HTN, type 2 diabetes. Allergies: penicillin (anaphylaxis). Discharge medications: metformin 500 mg twice daily…

As one vector, the allergy line is one topic among five. It barely moves the vector, so a search for “penicillin allergy” might not find it.

So split documents into chunks and embed each one. But be careful:

• No overlap: ”…Allergies: penicillin” lands in chunk 2 and “(anaphylaxis)…” lands in chunk 3. The drug and the reaction got separated!
• With overlap: each chunk repeats the end of the previous one. Chunk 3 now starts with “Allergies: penicillin (anaphylaxis)…”. The full fact appears whole in at least one chunk.

Choice	Common practice
Size	a few hundred tokens. Smaller is more precise, larger keeps more context
Overlap	10–20% of the chunk, so a fact cut at a boundary appears whole once
Boundaries	split at sentences, paragraphs, or section headings. Never mid-word
Context	add the note type, date, and section heading to the front of each chunk

Evaluating Retrieval

Build a labeled set (The Truth): questions paired with the notes that should come back.

Question	Relevant notes
Is the patient allergic to penicillin?	N3, N9
What was the most recent HbA1c?	N1
Was a drug allergy denied or ruled out?	N4, N6

Then eval on two metrics:

1. Precision@k: of the top $k$ results, what fraction are relevant?
   ”How much junk did I get?”
2. Recall@k: of all the relevant notes, what fraction made it into the top $k$?
   ”How much did I miss?”

Question 1 (“Is the patient allergic to penicillin?”) with $k = 3$:

Method	Top 3	Relevant in top 3	Precision@3	Recall@3
Keyword (BM25)	N4, N3, N5	N3	1/3 = 0.33	1/2 = 0.50
Vector	N3, N9, N6	N3, N9	2/3 = 0.67	2/2 = 1.00

Average these over all your questions. Build 20–50 real questions before you choose a model, chunk size, or $k$. Don’t pick by vibes lol.

Note: this “recall@k” (did we find the relevant notes?) is different from the ANN “recall@k” earlier (did the index find the same results as exact search?). Same formula, different “ground truth”.

Practical Matters

The data flows so far is

$$\text{Notes} \rightarrow \text{Note Chunks} \rightarrow \text{Vectors} \rightarrow \text{Indices}$$

Where the Indices can be HNSW for vectors or BM25 for keywords. Every arrow is something that can go stale.

• Vectors leak text.
  An attack recovered 92% of 32-token inputs exactly from their embeddings alone, and 89% of full names from clinical notes. Embeddings are not anonymized. Protect them like the notes they came from!
• Deletion
  Deleting a note must also delete its chunks and vectors.
• Edits
  An edited note needs new vectors. The old vector still matches the old text.
• Indexes
  DuckDB’s full-text index does not update when the table changes. You must rebuild it after loading new notes.

Improving Recall

• Query expansion (adding synonyms) improves Recall for keyword search.
• Adding more connected nodes in HNSW.

Remember there are two “recalls”. Fix the right one!

The index missed things (ANN recall: did HNSW find what exact search would have?)

• Turn the HNSW knobs up. Raise M (more links per node), ef_construction (a better-built graph) and ef_search (more candidates kept while searching). In class, ef_search 10 → 32 → 256 took recall from 0.76 → 0.96 → 1.00. You pay in memory and speed.
• Just do exact search. For a few thousand chunks (or one patient’s notes) brute force is fast enough and recall is 1.00 by definition.
• Over-fetch when filtering. The index runs before the WHERE, so ask for $5k$ or $10k$ results, filter, then keep the top $k$. Or filter first and do exact search on what’s left.

The right note was never “close” (relevance recall: did we find the notes that answer the question?)

• Bigger $k$, then rerank. Retrieve 50–100 cheaply, then let a slower, smarter model (a cross-encoder or an LLM) read each (query, chunk) pair and keep the best 5. Bigger $k$ buys recall, reranking buys back the precision.
• Hybrid search. Run BM25 and vector search and merge the rankings with RRF. Embeddings blur exact tokens like drug names, dosages, MRNs and ICD codes. BM25 nails them. Each catches what the other misses.
• Rewrite the query. Have an LLM expand abbreviations (“PCN” → “penicillin”), fix typos, or split a compound question into parts.
• Multi-query. Have an LLM write 3–5 paraphrases of the question, search with each, and take the union.
• HyDE (Hypothetical Document Embeddings). Have an LLM write a fake note that would answer the question (“Patient reports hives after amoxicillin…”) and embed that. A note-shaped query lands closer to real notes than a question does. It doesn’t matter if the fake note gets facts wrong: it’s only used to search.
• Pick (or train) a better model. Use a domain model like MedCPT for clinical text, or fine-tune on your own (question, relevant note) pairs from the labeled set.
• Use the model’s prefixes. Some models (e.g. E5) expect query: on questions and passage: on documents. Skip them and recall quietly drops.
• Chunk better. Smaller chunks, overlap, and a context header on each chunk (see Chunking). A fact buried in a giant chunk gets averaged away.
• Embed more than one view of each chunk. Also embed an LLM-written summary, or questions the chunk answers. Any match points back to the original chunk.

warning

More recall usually means more junk (lower precision). Measure both on your labeled set after each change!