Vector Operations

Keep it Real

This is for all Real vectors. Complex vectors have their own… shall we say complexities 😎.

Vector Norm

This is a function that takes a vector and maps it to $[0, \infty)$ so you get an idea of the ‘size’ or ‘length’ of the vector.

$$\|a\|_p = \left( \sum_{i=1}^{n} |a_i|^p \right)^{1/p}$$

That’s the p-norm, the general formula. Using that, and for $p \in \{1,2,\infty\}$, you get these norms:

Norm	Formula
Manhattan	$\lVert a \rVert_1 = \lvert a_1 \rvert + \lvert a_2 \rvert + \cdots + \lvert a_n \rvert$
Euclidean	$\lVert a \rVert_2 = \sqrt{a_1^2 + a_2^2 + \cdots + a_n^2}$
Infinity AKA The Fuck It Norm	$\lVert a \rVert_\infty = \max_i \lvert a_i \rvert$

TODO: When does one use each?

Vector Distance

The Euclidean Distance is just the straight line distance between the arrow tips of the vectors.

$$\lVert \mathbf{a} - \mathbf{b} \rVert = \sqrt{\sum_{i=1}^{n} (a_i - b_i)^2}$$

Like Cosine Similarity below, it is sometimes used to measure the similarity between two vectors. Lower means more similar.

---

Inner Product

An Inner Product is pretty general. For vectors $\mathbf{a}$ and $\mathbf{b}$ and some weight $W$,

$$\langle \mathbf{a}, \mathbf{b} \rangle_W = \sum_k w_k a_k b_k \quad \forall\space w_k > 0$$

It has some properties.

1. Symmetric: $\langle \mathbf{a}, \mathbf{b} \rangle = \langle \mathbf{b}, \mathbf{a} \rangle$
2. Linear:
   $\langle \mathbf{a} + \mathbf{b}, \mathbf{d} \rangle = \langle \mathbf{a}, \mathbf{d} \rangle + \langle \mathbf{b}, \mathbf{d} \rangle$ and
   $\langle c\mathbf{a} + \mathbf{b}, \mathbf{d} \rangle = c\langle \mathbf{a}, \mathbf{d} \rangle + \langle \mathbf{b}, \mathbf{d} \rangle$
3. Positive: $\langle \mathbf{a}, \mathbf{a} \rangle > 0$ unless $\mathbf{a} = \mathbf{0}$

Dot Product

This is a specific kind of Inner Product where the weight is one. That’s really it.

$$\langle \mathbf{a}, \mathbf{b} \rangle = \sum_k a_k b_k$$

So if $\mathbf{a}=(1,2,3)$ and $\mathbf{b}=(4,5,6)$,

$$\begin{align*} \langle \mathbf{a}, \mathbf{b} \rangle & =1(4)+2(5)+3(6) \\ & =4+10+18 \\ & =\boxed{32} \end{align*}$$

Easy-peasy. But it says something about geometry which is cool. Keep reading.

Cosine Similarity

This is another way of using the Dot Product. Let’s use Euclidean norms here. You can express the Dot Product as:

$${\langle\mathbf{a},\mathbf{b}\rangle} = {\|\mathbf{a}\|\|\mathbf{b}\|\cos(\theta)}$$

So basically,

$$\underbrace{\langle\mathbf{a},\mathbf{b}\rangle}_{\text{Dot Product}} = \underbrace{\sum_{i=1}^{n}a_i b_i}_{\text{Component Calculation}} = \underbrace{\|\mathbf{a}\|\|\mathbf{b}\|\cos(\theta)}_{\text{Geometric Calculation}}$$

Let’s use the Geometric Calculation and express $\cos(\theta)$ as

$$\cos\theta = \frac{\mathbf{a} \cdot \mathbf{b}}{\lVert \mathbf{a} \rVert \, \lVert \mathbf{b} \rVert}$$

This tells us the angle between $\mathbf{a}$ and $\mathbf{b}$. Dividing by the lengths throws away magnitude and keeps only direction. The result is always in $[-1, 1]$: $1$ means same direction, $0$ means orthogonal, $-1$ means opposite.

From baby trig that $\cos(90^\circ) = 0$, which you can take to mean that they’re not similar at all. They’re orthogonal¹. This is why cosine is the go-to similarity measure in all manner of Machine Learning things like LLM embeddings.

If it’s $1$, they’re lying on top of each other and pointing in the exact same direction. If it’s $-1$ they’re pointing in the opposite direction. Easy-peasy.

---

Other Stuff

Dot and Inner Products

Every Dot product is an Inner Product but the reverse is not necessarily true. Duh.

In ML and vector databases, “inner product” almost always just means the dot product. E.g. DuckDB’s array_inner_product is just the dot product.

Angles and Lengths

Let’s use $\mathbf{a} = (3, 4)$ and $\mathbf{b} = (4, 3)$. Both have length $5$ (Euclidean norm: square root of sum of squares.)

$$\mathbf{a} \cdot \mathbf{b} = 3 \cdot 4 + 4 \cdot 3 = 24 \qquad \cos\theta = \frac{24}{5 \times 5} = 0.96$$

Now make $\mathbf{b}$ ten times longer: $\mathbf{b}' = (40, 30)$, length $50$. Same direction.

$$\mathbf{a} \cdot \mathbf{b}' = 3 \cdot 40 + 4 \cdot 30 = 240 \qquad \cos\theta = \frac{240}{5 \times 50} = 0.96$$

Aha! The Dot Procut grew 10x but the cosine didn’t change at all. So it depends on what you’re trying to do.

Unit/Length-One Vectors

If both vectors have length 1 (unit vectors), the denominator is $1 \times 1$, so

$$\lVert \mathbf{a} \rVert = \lVert \mathbf{b} \rVert = 1 \implies \mathbf{a} \cdot \mathbf{b} = \cos\theta$$

This is why embedding models usually normalize their output to length 1: you get cosine similarity at the cheap price of a plain dot product. Neato.

Here’s another cool advantage. Euclidean Distance is related to Cosine Similarity in length-one vectors like so:

$$\lVert \mathbf{a} - \mathbf{b} \rVert = \sqrt{2 - 2\cos\theta} = \sqrt{2 - 2\cdot\mathbf{a}\cdot\mathbf{b}}$$

Cross Products

These work in 3D for the most part and will give you a new vector that is orthogonal/perpendicular to the plane of the two input vectors. I.e., unlike a Dot Product which gives you a scalar (fancy for “number”), the Cross Product yields a new vector.

I’ve never used them for anything. Read this for more.

Eigenvectors and Eigenvalues

These just vectors (of course) but relate to matrics.

TODO: Finish this.

Footnotes

1. This word being a (highly eyeroll-inducing) TechBro favorite… ↩