Vector Spaces

A Vector Space is a bit abstract. It is a set $V$ of all objects of interest that can be added together or scaled ‘safely’ — this is the Linearity in Linear Algebra. For something to be called a Vector Space, it must satisfy some axioms.

Linearity Axioms

If $u,v,w$ are objects embedded in this space,

Addition Axioms

1. $u + v = v + u$
   Order doesn’t matter (Associativity)
2. $(u + v) + w = u + (v + w)$
   Grouping doesn’t matter (Commutativity)
3. There is some ‘zero vector’ such that $u + 0 = u$
4. For any $u$ there is a $-u$ such that $u + (-u) = 0$
   Vectors can be “cancelled out”

Multiplication Axioms

If $a,b$ are scalars (numbers) then,

1. $a(bv) = (ab)v$
   Scaling behaves as you’d expect
2. $1v = v$
   Scaling by 1 leaves the object alone

Distribution Axioms

1. $a(u + v) = au + av$
2. $(a+b)u = au + bu$

Closure Axioms

1. If $u,v,w \in V$ then $(u + v + w) \in V$
2. If $a \in F$ and $u \in V$ then $av \in V$ where $F$ is the field the vector space is ‘over’ (usually $\mathbb{R}$).

Why do these Rules Matter?

Because you have an abstraction that can study structure. That’s what Linear Algebra is really about (according to me) and why Linear Algebra is so powerful across fields like Quantum Mechanics, ML, etc. Linearity is a very nice property to work with if you can finagle it.

What’s a Vector then?

“A Vector Space is a collection of all vectors” seems circular (because it is.) A vector is just an object that obeys the rules of the Vector Space we defined above. It’s an object that “lives” in its home of Linearity which strictly enforces the (primarily adding and scaling) rules above.

• Polynomials form a “Polynomial Vector Space” because the addition and scaling rules work.
• Functions can be vectors in vector spaces too, for the same reason (e.g. all degree-2 Polynomials, all continuous functions)
• Matrices too, for the same reason.

Again, why does this matter? Because a vector is not just an arrow or a list of numbers. Those are the coordinates of the vector. $\left[\begin{smallmatrix}3 \\ -17 \end{smallmatrix}\right]$ is just the representation of a vector. Change the coordinate system and this representation changes. Which brings us to…

The Coordinate System

Imagine these in 2D as ‘tiling’ a vector space. Imagine making an even grid with those long pieces from Erector Sets. You can shear/smush them only at certain angles. Now imagine that the long pieces can only stretch or shrink lengthwise. That’s kinda what these are. Linearity yo!

These describe the location of the vectors in the space. Thing is: you can have several coordinate systems (basis vectors). Think of the Mercator and other projections of our beautiful little planet. Same space, different ‘views’.

We do pick $\left[\begin{smallmatrix}1 \\ 0 \end{smallmatrix}\right]$ and $\left[\begin{smallmatrix}0 \\ 1 \end{smallmatrix}\right]$ in a 2D space for the nice and simple Cartesian coordinates from school. But these are not the only basis vectors.

Finding the Right Coordinate System

Picking the ‘right’ basis vectors/coordinate system is super important for Math and Computation (and meaning)!

Think of what PCA does. It picks the basis vectors. Each principal component is composed of $(w_1x_1 + w_2x_2 + ...)$. Now each Principal Component uses new basis vectors $u_n$ which are composed of $(w_1, w_2, ..., w_n)$. This is the vector space redone based on variances of $x_n$!

Another example. Fourier Transforms create a new coordinate system that simplifies a signal. Think of an audio waveform as a collection of signals over time $t$ like $t_1 = [0.2,0.7,-10.5]$, $t_2 = [0.4,0.6,-0.5]$ and so on. Signals form a vector space since you can add and scale them. Your simple basis vectors could be $[1,0,0], [0,1,0], [0,0,1]$ — capture how much of each portion of the signal exists. But the Fourier Transform uses $sin(\omega t)$ and $cos(\omega t)$ as the coordinate axes!

This ‘redone’ system changes the question. Instead of asking “how much of each signal exists” you are asking “how much of each frequency does the signal contain?” Instead of arbitrary coordinates, we use coordinates aligned with oscillations.

Orthogonality and Normality

You can pick your own basis vectors but they must (a) span the entire Vector Space and (b) be independent. That’s all. So $\left[\begin{smallmatrix}1 \\ 0 \end{smallmatrix}\right]$ and $\left[\begin{smallmatrix}1 \\ 1 \end{smallmatrix}\right]$ are fine. But they are not orthogonal to each other.

You get a nice and clean and simple geometry when you pick orthogonal vectors like $\left[\begin{smallmatrix}1 \\ 0 \end{smallmatrix}\right]$ and $\left[\begin{smallmatrix}0 \\ 1 \end{smallmatrix}\right]$. Those are perpendicular to each other (the dot product is zero) and you are dealing with unit lengths (their distance is 1).

Orthonomality means both these things. It simplfies things because the coordinates do not become entangled with each other when you’re measuring and operating on things in the Vector Space.

• Matrix Inverses become Transposes
• Lengths simpify: For a vector $v$, $||v||^2 = c_1^2 + c_2^2 + ...$
• Projections simplify:

Linear Operators & Transformations

These are ‘instructions’ on what to do with the vectors in a vector space.

People will talk about the two interchangeably but there’s a subtlely. Every Linear Operator is a Linear Transformation but the reverse is not true. All operators and transformations are functions.

With operators, you can always compose transformations. Consider an operator $T: V \rightarrow V$. Now you can do $T^2$, $T^3$, $e^T$, and polynomials $p(T)$. If $p(x) = 2x^2 + 3x + 2$ then you have $p(T) = 2T^2 + 3T + 2I$ (don’t forget the Identity Matrix!)

You cannot always compose transformations. Consider some arbitrary transformation $T: \mathbb{R^2} \rightarrow \mathbb{R^3}$. TODO…

What’s a Matrix then?

With all this in mind, what’s a Matrix?

It encodes how transformations specified by the Linear Operator act on the defined basis space.

• It’s not just a “table of numbers”!
• It’s not a simple “collection of vectors”!
• It is not the transformation itself.

It describes where to ‘send’ the Basis Vectors! Each column is the coordinate representation of the operator acting on one basis vector. The matrix is the collection of all transformed basis vectors. Different basis vectors? Different matrices.