Product of vectors that produce scalars.
Dot product and inner product are products between vectors that give us additional structures on vector spaces.
So far our discussions on subspaces and basis focused only on vector addition and scalar multiplication. But there is one more operation...
Dot product!
The dot product \[ \begin{bmatrix} x_1 \\ \vdots \\ x_n \end{bmatrix} \cdot \begin{bmatrix} y_1 \\ \vdots \\ y_n \end{bmatrix} = x_1 y_1 + \cdots x_n y_n \] produces a scalar (number).
It gives us one more thing we can do with vectors. In mathematics, we call this an additional "structure".
We can verify that the dot product operation is "linear" in each factor.
I.e., for two vectors $\mathbf{u},\mathbf{v} \in \mathbb{R}^n$ and a real number $r$, \[ \begin{aligned} (r \mathbf{u}) \cdot \mathbf{v} &= r (\mathbf{u} \cdot \mathbf{v}), &&\text{and} & \mathbf{u} \cdot (r\mathbf{v}) &= r (\mathbf{u} \cdot \mathbf{v}), \end{aligned} \]
Similarly, \[ \mathbf{u} \cdot (\mathbf{v}_1 + \mathbf{v}_2) = (\mathbf{u} \cdot \mathbf{v}_1) + (\mathbf{u} \cdot \mathbf{v}_2) \]
This linearity property is important, as it tells us that the dot product operation interacts "nicely" with linear combinations: The dot product of a vector with a linear combination is exactly the weighted sum of dot products.
It is easy to see that for any $\mathbf{v} \in \mathbb{R}^n$, \begin{align*} \mathbf{v} \cdot \mathbf{0} &= 0 &&\text{and} & \mathbf{0} \cdot \mathbf{v} &= 0 \end{align*}
However, these are not the only cases where the dot product of two vectors can be zero.
Indeed, the dot product of two nonzero vectors can still be zero. E.g., \[ \begin{bmatrix} 2 \\ 0 \end{bmatrix} \cdot \begin{bmatrix} 0 \\ 3 \end{bmatrix} = 2 \cdot 0 + 0 \cdot 3 = 0. \] It is easy to come up with other examples, and they show us that the dot product behave very differently from product of real numbers.
In general, from the fact that $\mathbf{u} \cdot \mathbf{v} = 0$ we cannot conclude that $\mathbf{u} = \mathbf{0}$ or $\mathbf{v} = \mathbf{0}$.
Recall that vectors in $\mathbb{R}^2$ or $\mathbb{R}^3$ represents geometric vectors, i.e., directed line segments.
A "directed" line segment is simply a line segment with an assigned direction. The two end points are called its initial point and terminal point respectively, and we usually draw an arrowhead from the initial point to the terminal point to indicate the direction.
For two nonzero vectors $\mathbf{u}$ and $\mathbf{v}$, \[ \mathbf{u} \cdot \mathbf{v} = \| \mathbf{u} \| \, \| \mathbf{v} \| \, \cos(\theta) \] where $\theta$ is the angle between the two vector.
In particular, for two unit vectors $\mathbf{u}, \mathbf{v} \in \mathbb{R}^n$ (i.e., $\| \mathbf{u} \| = \| \mathbf{v} \| = 1$), the equation above can be simplified into \[ \mathbf{u} \cdot \mathbf{v} = \cos(\theta), \] which is between $-1$ and $1$.
Here, the word "perpendicular" still mean the two directed line segments form a 90 degree angle when $n = 2$. The same idea still work for higher dimension as two vectors are always contained in a plane.
In this sense, the dot product between two unit vectors measure the extend to which the two directions agree with each other.
We can also say $\mathbf{u}$ is orthogonal to $\mathbf{v}$.
This is simply a generalization of the idea of two line segments forming a 90 degree angle into the higher dimension. Note, however, that the definition also include the cases where one or both are zero vectors.
The dot product is one example of the more general "inner product".
Here, we focus on real-valued inner products. It is possible to have inner product that have complex values, in which case a different set of properties are specified.
It is easy to see the dot product satisfies all these conditions. Therefore the dot product is a special kind of inner product. However there are other possible inner products.
We can verified that \[ \left\langle \begin{bmatrix} u_1 \\ u_2 \end{bmatrix} \,,\, \begin{bmatrix} v_1 \\ v_2 \end{bmatrix} \right\rangle = 4 u_1 v_1 + 3 u_1 v_2 + 3 u_2 v_1 + 4 u_2 v_2 \] satisfies the properties listed above.
However, the function \[ g \left( \begin{bmatrix} u_1 \\ u_2 \end{bmatrix} \,,\, \begin{bmatrix} v_1 \\ v_2 \end{bmatrix} \right) = 1 u_1 v_1 + 2 u_1 v_2 + 3 u_2 v_1 + 4 u_2 v_2 \] will not define an inner product. Can you see why?
For what choices of $a,b,c,d$, will the formula \[ \left\langle \begin{bmatrix} u_1 \\ u_2 \end{bmatrix} \,,\, \begin{bmatrix} v_1 \\ v_2 \end{bmatrix} \right\rangle = a u_1 v_1 + b u_1 v_2 + c u_2 v_1 + d u_2 v_2 \] define an inner product?
Note inner product space is the pairing of a vector space (subspace) and a choice of inner product. It does not make much sense to ask if a vector space is an inner product space.
All the examples we have seen so far are, implicitly, inner product spaces, e.g., $\mathbb{R}^n$ together with the dot product. Similarly, each subspace of $\mathbb{R}^n$ also inherits this inner product structure.
Once an inner product is chosen, we can redefine the notion of orthogonal vectors in terms of the inner product.
In general, we say $\mathbf{u},\mathbf{v}$ are orthogonal if \[ \langle \mathbf{u} , \mathbf{v} \rangle = 0. \]
The most important concept related to orthogonality, in this course, is "orthogonal basis".
In a sense, an orthogonal basis is a more natural choice of basis for a subspace. After all, in the plane, we prefer choice of $x$-axis and $y$-axis forming a 90 degree angle.