Don't forget about "dot" products

So far our discussions on subspaces and basis focused only on vector addition and scalar multiplication. But there is one more operation...

Dot product!

The dot product $\begin{bmatrix} x_1 \\ \vdots \\ x_n \end{bmatrix} \cdot \begin{bmatrix} y_1 \\ \vdots \\ y_n \end{bmatrix} = x_1 y_1 + \cdots x_n y_n$ produces a scalar (number).

It gives us one more thing we can do with vectors. In mathematics, we call this an additional "structure".

Review: properties of dot products

We can verify that the dot product operation is "linear" in each factor.

I.e., for two vectors $\mathbf{u},\mathbf{v} \in \mathbb{R}^n$ and a real number $r$ , $\begin{aligned} (r \mathbf{u}) \cdot \mathbf{v} &= r (\mathbf{u} \cdot \mathbf{v}), &&\text{and} & \mathbf{u} \cdot (r\mathbf{v}) &= r (\mathbf{u} \cdot \mathbf{v}), \end{aligned}$

Similarly, $\mathbf{u} \cdot (\mathbf{v}_1 + \mathbf{v}_2) = (\mathbf{u} \cdot \mathbf{v}_1) + (\mathbf{u} \cdot \mathbf{v}_2)$

This linearity property is important, as it tells us that the dot product operation interacts "nicely" with linear combinations: The dot product of a vector with a linear combination is exactly the weighted sum of dot products.

About zero dot product

It is easy to see that for any $\mathbf{v} \in \mathbb{R}^n$ , $\begin{align*} \mathbf{v} \cdot \mathbf{0} &= 0 &&\text{and} & \mathbf{0} \cdot \mathbf{v} &= 0 \end{align*}$

However, these are not the only cases where the dot product of two vectors can be zero.

Indeed, the dot product of two nonzero vectors can still be zero. E.g., $\begin{bmatrix} 2 \\ 0 \end{bmatrix} \cdot \begin{bmatrix} 0 \\ 3 \end{bmatrix} = 2 \cdot 0 + 0 \cdot 3 = 0.$ It is easy to come up with other examples, and they show us that the dot product behave very differently from product of real numbers.

In general, from the fact that $\mathbf{u} \cdot \mathbf{v} = 0$ we cannot conclude that $\mathbf{u} = \mathbf{0}$ or $\mathbf{v} = \mathbf{0}$ .

Geometric interpretation

Recall that vectors in $\mathbb{R}^2$ or $\mathbb{R}^3$ represents geometric vectors, i.e., directed line segments.

A "directed" line segment is simply a line segment with an assigned direction. The two end points are called its initial point and terminal point respectively, and we usually draw an arrowhead from the initial point to the terminal point to indicate the direction.

For two nonzero vectors $\mathbf{u}$ and $\mathbf{v}$ , $\mathbf{u} \cdot \mathbf{v} = \| \mathbf{u} \| \, \| \mathbf{v} \| \, \cos(\theta)$ where $\theta$ is the angle between the two vector.

In particular, for two unit vectors $\mathbf{u}, \mathbf{v} \in \mathbb{R}^n$ (i.e., $\| \mathbf{u} \| = \| \mathbf{v} \| = 1$ ), the equation above can be simplified into $\mathbf{u} \cdot \mathbf{v} = \cos(\theta),$ which is between $-1$ and $1$ .

$\mathbf{u} \cdot \mathbf{v} = 1$ when the two are equal.
$\mathbf{u} \cdot \mathbf{v} = -1$ when the two have opposite directions
$\mathbf{u} \cdot \mathbf{v} = 0$ when they are "perpendicular" to one another.

Here, the word "perpendicular" still mean the two directed line segments form a 90 degree angle when $n = 2$ . The same idea still work for higher dimension as two vectors are always contained in a plane.

In this sense, the dot product between two unit vectors measure the extend to which the two directions agree with each other.

Orthogonal vectors

Two vectors

$\mathbf{u}$ and

$\mathbf{v}$ in

$\mathbb{R}^n$ are said to be orthogonal to each other, if

$\mathbf{u} \cdot \mathbf{v} = 0.$

We can also say $\mathbf{u}$ is orthogonal to $\mathbf{v}$ .

This is simply a generalization of the idea of two line segments forming a 90 degree angle into the higher dimension. Note, however, that the definition also include the cases where one or both are zero vectors.

General inner product

The dot product is one example of the more general "inner product".

An inner product defined on

$\mathbb{R}^n$ is a function

$\langle \cdot,\cdot \rangle$ that takes in two vectors and produce a real number such that

$\langle \mathbf{u}, \mathbf{v} \rangle = \langle \mathbf{v}, \mathbf{u} \rangle$ for any $\mathbf{u},\mathbf{v} \in \mathbb{R}^n$ . (Symmetric)
$\langle r_1 \mathbf{u}_1 + r_2 \mathbf{u}_2, \mathbf{v} \rangle = r_1 \langle \mathbf{u}_1, \mathbf{v} \rangle + r_2 \langle \mathbf{u}_2, \mathbf{v} \rangle$ for any $\mathbf{u}_1,\mathbf{u}_2,\mathbf{v} \in \mathbb{R}^n$ and $r_1,r_2 \in \mathbb{R}$ . (Linear)
$\langle \mathbf{v}, \mathbf{v} \rangle > 0$ for any $\mathbf{v} \ne \mathbf{0}$ . (Positive-definite)

Here, we focus on real-valued inner products. It is possible to have inner product that have complex values, in which case a different set of properties are specified.

It is easy to see the dot product satisfies all these conditions. Therefore the dot product is a special kind of inner product. However there are other possible inner products.

Exercises

We can verified that $\left\langle \begin{bmatrix} u_1 \\ u_2 \end{bmatrix} \,,\, \begin{bmatrix} v_1 \\ v_2 \end{bmatrix} \right\rangle = 4 u_1 v_1 + 3 u_1 v_2 + 3 u_2 v_1 + 4 u_2 v_2$ satisfies the properties listed above.

However, the function $g \left( \begin{bmatrix} u_1 \\ u_2 \end{bmatrix} \,,\, \begin{bmatrix} v_1 \\ v_2 \end{bmatrix} \right) = 1 u_1 v_1 + 2 u_1 v_2 + 3 u_2 v_1 + 4 u_2 v_2$ will not define an inner product. Can you see why?

Advanced exercise

For what choices of $a,b,c,d$ , will the formula $\left\langle \begin{bmatrix} u_1 \\ u_2 \end{bmatrix} \,,\, \begin{bmatrix} v_1 \\ v_2 \end{bmatrix} \right\rangle = a u_1 v_1 + b u_1 v_2 + c u_2 v_1 + d u_2 v_2$ define an inner product?

Inner product space

Suppose

$V$ is a vector space (e.g. a subspace of

$\mathbb{R}^n$ ) and

$\langle \cdot, \cdot \rangle$ is an inner product defined on

$V$ . Then the pair

$(V,\langle \cdot, \cdot \rangle)$ is called an inner product space.

Note inner product space is the pairing of a vector space (subspace) and a choice of inner product. It does not make much sense to ask if a vector space is an inner product space.

All the examples we have seen so far are, implicitly, inner product spaces, e.g., $\mathbb{R}^n$ together with the dot product. Similarly, each subspace of $\mathbb{R}^n$ also inherits this inner product structure.

General orthogonality from inner products

Once an inner product is chosen, we can redefine the notion of orthogonal vectors in terms of the inner product.

In general, we say $\mathbf{u},\mathbf{v}$ are orthogonal if $\langle \mathbf{u} , \mathbf{v} \rangle = 0.$

Orthogonal basis

The most important concept related to orthogonality, in this course, is "orthogonal basis".

A orthogonal basis for an inner product space is a basis in which vectors are mutually orthogonal.

In a sense, an orthogonal basis is a more natural choice of basis for a subspace. After all, in the plane, we prefer choice of $x$ -axis and $y$ -axis forming a 90 degree angle.

Dot product and inner product