\documentclass{article}
\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{enumerate}

\usepackage{hyperref}

\usepackage{ifthen}
\usepackage{ulem} % for strikethrough \sout{...} command
\usepackage{color}

\usepackage{wasysym}

\setlength{\unitlength}{0.1in}

\setlength{\oddsidemargin}{-0.25in}
\setlength{\evensidemargin}{-0.25in}
\setlength{\topmargin}{-0.45in}
\setlength{\headheight}{0.0in}
\setlength{\headsep}{0.0in}
\setlength{\topskip}{0.0in}
\setlength{\textheight}{9.6in}
\setlength{\textwidth}{7in}

\newcommand{\bdot}{\, {\scriptstyle{\bullet}} \,}
\newcommand{\B}[1]{{\bf {#1}}}
\newcommand{\la}{\langle}
\newcommand{\ra}{\rangle}

\newcommand{\myspace}{\parbox[c][0.25in]{0.25in}{\hfill}}
\newcommand{\varspace}[2]{\mbox{\parbox[c][#1]{#2}{\hfill}}}
\newcommand{\force}{{\noindent $\mbox{\;}$}}

\newcommand{\diagquotient}[2]{\displaystyle{{#1 \atop \;}\hspace*{-0.1in} \mbox{\put(0,0){\line(2,1){2}}} \hspace*{0.1in} {\; \atop #2}}}


\newcommand{\F}{\mathbb{F}}
\newcommand{\E}{\mathbb{E}}
\newcommand{\K}{\mathbb{K}}
\newcommand{\Z}{\mathbb{Z}}
\newcommand{\R}{\mathbb{R}}
\newcommand{\Q}{\mathbb{Q}}
\newcommand{\C}{\mathbb{C}}

\newcommand{\LEMMA}{\vspace{0.05in} \noindent \fbox{\large\bf Lemma:}  }
\newcommand{\COR}{\vspace{0.05in} \noindent \fbox{\large\bf Corollary:}  }
\newcommand{\PROP}{\vspace{0.05in} \noindent \fbox{\large\bf Proposition:}  }
\newcommand{\DEF}{\vspace{0.05in} \noindent \fbox{\large\bf Definition:}  }
\newcommand{\THM}{\vspace{0.05in} \noindent \fbox{\large\bf Theorem:}  }
\newcommand{\EX}{\vspace{0.05in} \noindent \fbox{\large\bf Example:}  }
\newcommand{\bproof}{\vspace{0.025in} \noindent {\bf Proof:} }
\newcommand{\eproof}{$\blacksquare$ \vspace{0.05in}}

\newcommand{\vsp}{\vspace{0.05in}}

\begin{document}

%\pagestyle{empty}

\noindent
{\bf Math 5230}
\hfill {\Large \bf Structures Related to Inner Product Spaces} \hfill 
{\bf Linear Algebra}

\vspace{0.15in}

There are various generalizations of the notation of an inner product space. We will stick to real and complex inner product spaces. These are vector spaces equipped with an {\it inner product}. This gives us a way to compute things like lengths and angles. Once we have an inner product in hand, we can define related structures like a norm and metric. 

But first we should recall some facts about the field of complex numbers: $\mathbb{C} = \{ a+bi \;|\; a,b \in \mathbb{R} \}$ ($i$ is the imaginary root, so $i^2=-1$). Given $z = a+bi \in \mathbb{C}$, we call $\mathfrak{Re}(z)=a$ the real part of $z$ and $bi$ the imaginary part. Let $z=a+bi$ and $w=c+di$. We have $z+w=(a+c)+(b+d)i$ and $zw=(ac-bd)+(ad+bc)i$.
The complex numbers also are equipped with the operation of {\bf conjugation} defined by $\bar{z} = \overline{a+bi} = a-bi$ (we flip the sign between the real and imaginary parts). Conjugation is an involution on $\mathbb{C}$. In other words, $\overline{z+w}=\bar{z}+\bar{w}$, $\overline{zw} = \bar{z}\,\bar{w}$, and $\bar{\bar{z}}=z$. Notice also that $z=\bar{z}$ if and only if $z=\mathfrak{Re}(z) \in \R$ (i.e. $z$ is self conjugate only when it is real). Notice $z\bar{z} = (a+bi)(a-bi)=a^2+b^2 \in \mathbb{R}_{\geq 0}$. The modulus of
a complex number is defined as $|z| = \sqrt{z\bar{z}} = \sqrt{a^2+b^2}$. If we think of $z=a+bi$ as a point $(a,b)$ in $\mathbb{R}^2$, then $|z|$ is the ``length'' of $z$ (i.e.,
 the distance from $z$ to the origin). 

\vspace{0.05in}

Let us fix a particular a subfield of the complex numbers, say $\F$. Consequently $\F$ will contain (at least) all of the rational numbers, $\Q$, and be closed under addition, subtraction, multiplication, and division (not by zero). Usually, we have in mind $\F=\R$ or $\F=\C$. Also, let $V$ be a vector space over $\F$. 

\vspace{0.05in}

\DEF Let $V$ be equipped with a mapping $\langle \,\cdot\, , \,\cdot\, \rangle : V \times V \to \mathbb{F}$ (called an {\it inner product}) such that\dots
\begin{enumerate}[(a)]
\item {}[Sesquilinear] {\small (one and a half times linear)} $\langle \,\cdot\, , \,\cdot\, \rangle$ is linear in the first slot and conjugate linear in the second slot.\footnote{Some people follow the opposite convention and have their inner product conjugate linear in the first slot and linear in the second slot. This is common in matrix theory and especially in Physics.} 

%\vspace{-0.175in}

\force \hfill $\langle \B u + \B v, \B w \rangle = \langle \B u, \B w \rangle + \langle \B v, \B w \rangle$, \quad 
$\langle \B u, \B v + \B w \rangle = \langle \B u, \B v \rangle + \langle \B u, \B w \rangle$, \quad
$\langle c\B v, \B w \rangle = c\langle \B v, \B w \rangle$, \quad
 and \quad $\langle \B v, c\B w \rangle = \bar{c} \langle \B v, \B w \rangle$.
 \hfill\force

When $\mathbb{F}$ is a subfield of the real numbers (such as $\mathbb{R}$ itself), conjugation does not do anything. In this case, the inner product is linear in both slots (i.e. {\it bilinear}).

\item {}[Conjugate Symmetric] \quad $\langle \B v, \B w \rangle = \overline{\langle \B w, \B v \rangle}$.

Again, if $\mathbb{F}$ is a real field, conjugation does nothing. In such a case, we have plain old {\it symmetry}.

\item {}[Positive Definite] Notice by conjugate symmetry, we have $\langle \B v, \B v \rangle = \overline{\langle \B v, \B v \rangle}$ so $\langle \B v, \B v \rangle \in \mathbb{R}$. We require

\force \hfill $\langle \B v, \B v \rangle \geq 0$ \quad and \quad $\langle \B v, \B v \rangle=0$ if and only if $\B v = \B 0$. \hfill \force
\end{enumerate}

\vspace{-0.05in}

\noindent
where $\B u, \B v, \B w \in V$ and $c \in \F$.
If so, $V$ is an {\bf inner product space} equipped with inner product  $\langle \,\cdot\, , \,\cdot\, \rangle$. In particular, when $\F=\C$ we say $V$ is a complex inner product space and when $\F=\R$ we say $V$ is a real inner product space.

\vsp

\DEF Let $V$ be equipped with a mapping $\| \,\cdot\, \|:V \to \mathbb{R}$ (called a {\it norm}) such that\dots
\begin{enumerate}[(a)]
\item {}[Positive Definite] \quad $\| \B v \| \geq 0$ \quad and \quad $\| \B v \| = 0$ if and only if $\B v = \B 0$
\item {}[Respects Scaling] \quad $\| c\,\B v \| = |c|\,\| \B v \|$
\item {}[Triangle Inequality] \quad $\| \B v + \B w \| \leq \| \B v \| + \| \B w \|$
\end{enumerate}

\vspace{-0.05in}

\noindent where $\B v \in V$ and $c \in \F$. If so, $V$ is called a {\bf normed space}.

\vsp

\THM Every inner product space $V$ has the structure of a normed space. Specifically, let $V$ be an inner product space. Then $\| \B v \| = \sqrt{\langle \B v, \B v \rangle}$ for all $\B v \in V$ is a norm on $V$. 

\vspace{0.05in}

\noindent
{\it Note:} Given a normed space, its norm does not necessarily come from an inner product. 

\vsp

The first part of the definition of a norm follows from the inner product's positive definiteness. The second part is an easy consequence of the scalar multiple part of sesquilinearity. The triangle inequality quickly follows from the following (rather important) result:

\vsp

\THM (Cauchy-Schwarz Inequality) Let $V$ be an inner product space. For all $\B v, \B w \in V$, $|\langle \B v, \B w\rangle | \leq \| \B v \| \, \| \B w \|$.

\vspace{0.05in}

\bproof There are many proofs of this famous inequality. Here we present a quirky one. First, note if an inner product $\langle \B v,\B w\rangle=z$ is not a real number, we can multiply it by $\zeta = \bar{z}/|z|$ so that $\langle \zeta\B v,\B w\rangle = \zeta \langle \B v,\B w\rangle = \bar{z}/|z| \cdot z = |z|^2/|z|=|z| \in \R$. Moreover, $|\zeta| = |\bar{z}|/|z|=1$ so that $|\langle \zeta \B v, \B w\rangle | = |\zeta| \cdot |\langle \B v, \B w\rangle |= |\langle \B v,\B w\rangle|$ and $\|\zeta \B v\| = |\zeta|\cdot\|\B v\| = \|\B v\|$. Thus the inequality holds for $\B v$ and $\B w$ if and only if it holds for $\zeta\B v$ and $\B w$. So without loss of generality we may assume $\langle \B v,\B w\rangle$ is a real number.

Let $x$ be any {\it real} number. Notice $0 \leq \| x\B v+\B w\|^2 = \langle x\B v+\B w,x\B v+\B w \rangle = x^2 \langle \B v,\B v\rangle + x\langle \B v,\B w\rangle +x\langle \B w,\B v\rangle + \langle \B w,\B w\rangle$ where we used sequilinearity to pull apart the inner product. Keep in mind that $x$ is real, so $\bar{x}=x$ (i.e., conjugation does nothing). Also, we have assumed $\langle \B v,\B w\rangle$ is real, so $\langle \B w,\B v\rangle = \overline{\langle \B v,\B w\rangle} = \langle \B v,\B w\rangle$. Thus our expression simplifies to $\|\B v\|^2 x^2+2\langle \B v,\B w\rangle x+\|\B w\|^2$.

If we let $A=\|\B v\|^2$, $B=2\langle \B v,\B w\rangle$, and $C=\|\B w\|^2$. Then $0 \leq \|x\B v+\B w\|^2=Ax^2+Bx+C = y$ is a quadratic in $x$ with real coefficients $A,B,C$.
Since $y \geq 0$, our quadratic cannot have distinct real roots. This means that the discriminant for the quadratic $B^2-4AC = (2\langle \B v,\B w\rangle)^2-4\|\B v\|^2\|\B w\|^2 \leq 0$ (recall that $B^2-4AC$ is the part of the quadratic formula inside the square root). Thus $(\langle \B v,\B w\rangle)^2 \leq (\|\B v\| \cdot \|\B w\|)^2$. Square rooting, we have $|\langle \B v,\B w\rangle |\leq \|\B v\| \, \|\B w\|$. \eproof

\vsp

Now the norm's triangle inequality easily follows. Note: For real numbers $a,b$, we always have $a \leq |a| = \sqrt{a^2} \leq \sqrt{a^2+b^2} = |a+bi|$ (i.e., for a complex number $z$, $\mathfrak{Re}(z) \leq |z|$). Therefore,

\vspace{-0.1in}

$$\| \B v +\B w\|^2 =   \langle \B v+\B w,\B v+\B w \rangle = \langle \B v,\B v\rangle + \langle \B v,\B w\rangle + \langle \B w,\B v\rangle + \langle \B w,\B w\rangle = \|\B v\|^2 + \langle \B v,\B w\rangle + \overline{\langle \B v,\B w\rangle} + \|\B w\|^2$$ 
$$= \|\B v\|^2 + 2\,\mathfrak{Re}(\langle \B v,\B w\rangle) + \|\B w\|^2 \leq \|\B v\|^2 + 2|\langle \B v,\B w\rangle | + \|\B w\|^2$$
$$ \leq \|\B v\|^2 + 2\|\B v\| \|\B w\| + \|\B w\|^2 = (\|\B v\|+\|\B w\|)^2$$
where our last inequality was the Cauchy-Schwarz inequality. Square-rooting yields $\|\B v+\B w\| \leq \|\B v\|+\|\B w\|$.

\vsp

An important consequence of the Cauchy-Schwarz inequality is that for nonzero $\B v, \B w \in V$ in a real inner product space, we can define the notion of the {\it angle} between $\B v$ and $\B w$. In particular, notice Cauchy-Schwarz says $\dfrac{|\langle \B v, \B w \rangle |}{\| \B v \|\,\| \B w \|} \leq 1$. This means $-1 \leq \dfrac{\langle \B v, \B w \rangle}{\| \B v \|\,\| \B w \|} \leq 1$ so that $\theta =\mathrm{arccos}\left( \dfrac{\langle \B v, \B w \rangle }{\| \B v \|\,\| \B w \|} \right)$ makes sense. We note that $0 \leq \theta \leq \pi$ where $\theta=0$ (respectively $\pi$) if $\B v$ and $\B w$ point in the same (respectively opposite) direction(s). We also have the familiar identity: $\langle \B v, \B w \rangle = \| \B v \| \, \|\B w\| \cos(\theta)$ (even when $\B v$ or $\B w$ is $\B 0$ so that $\theta$ is ill defined).

\vsp

\DEF Let $X$ be a set equipped with a mapping $d:X \times X \to \mathbb{R}$ (called a {\it metric}) such that\dots
\begin{enumerate}[(a)]
\item {}[Positivity] \quad $d(x,y) \geq 0$ \quad and \quad $d(x,y)=0$ if and only if $x=y$
\item {}[Symmetry] \quad $d(x,y)=d(y,x)$
\item {}[Triangle Inequality] \quad $d(x,z) \leq d(x,y)+d(y,z)$ 
\end{enumerate}

\vspace{-0.05in}

\noindent
where $x,y,z \in X$. If so, $X$ is called a {\bf metric space}.

\vsp

\THM Every normed space $V$ has the structure of a metric space. Specifically, let $V$ be a normed space. Then $d(\B v,\B w) = \|\B v - \B w\|$ for all $\B v, \B w \in V$ is a metric on $X$.

\vsp

\noindent
{\it Note:} However, given a metric space, its metric does not necessarily come from a norm. Also, if you study manifold theory, be aware that manifold theorists and differential geometers often use the term {\it metric} to mean something quite different. We are using {\it metric} in the topological sense.

\vsp

Once again, the first part of the definition ultimately flows from positive definiteness: $d(\B v,\B w)=\|\B v-\B w\|=0$ if and only if $\B v-\B w=\B 0$ (i.e., $\B v=\B w$). Symmetry is also easy: $d(\B v,\B w)=\|\B v-\B w\| = \|-(\B w-\B v)\| = |-1| \cdot \|\B w-\B v\| = d(\B w,\B v)$. Finally, the metric version of the triangle inequality follows from the norm version: 

%\vspace{-0.05in}

\force \hfill
$d(\B u,\B w) = \|\B u-\B w\| = \|\B u-\B v+\B v-\B w\| \leq \|\B u-\B v\|+\|\B v-\B w\| = d(\B u,\B v)+d(\B v,\B w)$. 
\hfill \force

\vsp

In a linear algebra course, we are not typically concerned with issues of limits and continuity (i.e., analysis issues). However, it is worth (briefly) mentioning that a metric allows one to define a topological structure on our space $X$. In particular, $O \subseteq X$ is {\it open} if given any point $x_0 \in O$ we have that points arbitrarily close to $x_0$ also belong to $O$. More concretely, $O$ is open if for every $x_0 \in O$, there exists some $\epsilon >0$ such that whenever $d(x,x_0)<\epsilon$, we have $x \in O$. The collection of open subsets of $X$ form a {\it topology} which allows one to explore notions such as continuity, limits, connectedness, and more. We will not pursue this further here.

\vsp

\noindent
{\bf Summing up:} Every inner product space gives us a compatible normed space structure. Every normed space gives us a compatible metric space structure. Every metric space gives us a compatible topological space structure. None of these implications hold in reverse (in general). Inner product spaces have many layers of rich structure. In such a space, linear algebra, geometry, analysis, and topology all have something to tell us.\footnote{Functional analysis begins with a study of these structures (inner product, normed, and metric spaces). Such a space in which all Cauchy sequences converge is called complete. A complete normed space is called a Banach space. A complete inner product space is called a Hilbert space. These kinds of spaces play a big role in quantum mechanics and elsewhere.}

\newpage

\EX For $\B x = (x_1,\dots,x_n), \B y = (y_1,\dots,y_n) \in \C^n$ the mapping $\langle \B x, \B y \rangle = \B x \bar{\B y}^t =  \sum\limits_{i=1}^n x_i \bar{y_i}$ is an inner product on $\mathbb{C}^n$. This is called the {\bf dot product} or the standard inner product. If we consider $\R^n$ instead of $\C^n$, we have $\B x \bdot \B y = \sum\limits_{i=1}^n x_iy_i$ for $\B x = (x_1,\dots,x_n), \B y = (y_1,\dots,y_n) \in \R^n$. In this case, the norm is $\|\B x\| = \sqrt{x_1^2+\cdots+x_n^2\strut}$ and the corresponding metric (i.e. distance function) is $d(\B x, \B y) = \sqrt{(x_1-y_1)^2+\cdots+(x_n-y_n)^2\strut}$. These are the standard Euclidean norm and metric.

\vsp

\vsp

\EX (Related to the previous example.) Let $A = (a_{ij}), B=(b_{ij}) \in \mathbb{C}^{m \times n}$. We let $B^* = \bar{B}^t = \overline{B^t}$ denote the conjugate transpose of $B$. Define $\langle A,B \rangle = \mathrm{tr}(AB^*) = \sum\limits_{i=1}^m\sum\limits_{j=1}^n a_{ij}\overline{b_{ij}}$. This is an inner product on $\C^{m \times n}$ called the Frobenius inner product. If we pull apart a matrix row by row (or column by column), we can think of $\C^{m \times n}$ as $\C^{mn}$. With such an identification, the Frobenius inner product on $\C^{m \times n}$ is really nothing more than the dot product on $\mathbb{C}^{mn}$. In the case of real matrices, $\mathbb{R}^{m \times n}$, this is just $\langle A,B \rangle = \mathrm{tr}(AB^T)$. In particular, for $A \in \mathbb{R}^{m \times n}$, $\|A\| = \sqrt{\sum\limits_{i=1}^m \sum\limits_{j=1}^n (a_{ij})^2}$.

\vsp

\noindent
We also have more interesting examples built from integration. 

\vsp

\EX For fixed real numbers $a<b$, let $C[a,b] = \{ f:[a,b] \to \mathbb{R} \;|\; f \text{ is continuous\,}\}$ (i.e., continuous real valued functions defined on the closed interval $[a,b]$). This is a real vector space. Next, fix a positive continuous {\it weight} function $W(x) \in C[a,b]$ (so $W(x)> 0$ for $a \leq x \leq b$). For any $f(x),g(x) \in C[a,b]$, we define the inner product: 

\vspace{-0.25in}

$$\displaystyle \langle f(x),g(x) \rangle = \int_a^b f(x)g(x)W(x)\,dx.$$ 

\noindent
While bilinearity and symmetry are easily established, showing positive definiteness requires careful thought and referencing continuity. Note that for any $f(x) \in C[a,b]$, $f(x)^2 \geq 0$ and so $f(x)^2W(x) \geq 0$ thus $\displaystyle \langle f(x),f(x) \rangle = \int_a^b f(x)^2W(x)\,dx \geq 0$. Next, if $f(x_0) \not=0$ for some $x_0 \in [a,b]$, then $f(x_0)^2 >0$.
Thus $f(x_0)^2W(x_0)>0$. Analysis tells us that a continuous function (such as $f(x)^2W(x)$) that is positive at a point must also be positive on some open interval about that point. Eventually this allows us to conclude $\displaystyle \langle f(x),f(x) \rangle =  \int_a^b f(x)^2W(x)\,dx >0$. Thus $\langle f(x),f(x)\rangle =0$ only if $f(x)=0$. For the space of polynomials\footnote{Note that for $f(x) \in C[a,b]$, if there are real numbers $a_n,\dots,a_0 \in \mathbb{R}$ such that $f(x)=a_nx^n+\cdots+a_0$, we say $f(x)$ is a polynomial function. Since real polynomials are determined by their values on any closed interval, it turns out that $\mathbb{R}[x]$ is isomorphic with the subspace of these polynomial functions on $[a,b]$. We often abuse such an isomorphism and use {\it polynomials} and {\it polynomial functions} interchangably. If we were considering polynomial functions on a finite field, this would no longer work. Over finite fields, there are infinitely many polynomials but only finitely many polynomial functions.}, we can loosen our restriction on our weight function. In that context, we only need $W(x)$ to be a non-zero, non-negative continuous function.

\vsp

\vsp

The above example is a first step into the world of {\it orthogonal polynomials}. For example, choosing $[a,b]=[-1,1]$ and $W(x)=1$ so that $\displaystyle \langle f(x),g(x) \rangle = \int_{-1}^1 f(x)g(x)\,dx$ and then running the Gram-Schmidt process on $1$, $x$, $x^2$, $\dots$, one gets: $1$, $x$, $x^2-1/3$, $x^3-3/5x$, $\dots$. These polynomials (up to a certain normalization) are Legendre polynomials which show up in numerous mathematical and scientific applications.

\vsp

\vsp

\noindent {\bf\Large Coordinates:}

\vsp

Let $V$ be a (finite dimensional) inner product space with basis $\beta = \{\B v_1,\dots, \B v_n \}$. Given $\B v, \B w \in V$, we have $\B v = \sum\limits_{i=1}^n  b_i\B v_i$ and $\B w = \sum\limits_{i=1}^n c_i\B v_i$ for some scalars $b_i,c_i \in \mathbb{F}$ (i.e., $[\B v]_\beta = [b_1\;b_2\;\cdots\;b_n]^t$ and $[\B w]_\beta = [c_1\;c_2\;\cdots\;c_n]^t$). We have:

\vspace{-0.25in}

$$\langle \B v, \B w \rangle = \left\langle \sum\limits_{i=1}^n b_i\B v_i, \sum\limits_{j=1}^n c_j\B v_j \right\rangle = \sum\limits_{i=1}^n \sum\limits_{j=1}^n b_i\overline{c_j} \langle \B v_i, \B v_j \rangle .$$

\noindent If we define a square matrix $A = (a_{ij})$ by $a_{ij} = \langle \B v_i, \B v_j \rangle$, then $\langle \B v, \B w \rangle = \sum\limits_{i=1}^n\sum\limits_{j=1}^n b_i a_{ij} \overline{c_j} =  ([\B v]_\beta)^T A \, \overline{[\B w]_\beta}$. Here $A$ is called the matrix of the inner product (relative to the basis $\beta$). 

\vsp

The matrix for the dot product (relative to the standard basis) is just the identity matrix. Also, not all matrices arise as matrices of inner products. For example, because of conjugate symmetry we must have $A=A^*$ (i.e., $A$ is {\it Hermitian}). Over the reals, this amounts to requiring $A=A^T$ (i.e., $A$ is {\it symmetric}). In addition to the symmetry requirement, the matrix must also satisfy: $\B x^* A\, {\B x} \geq 0$ for all $\B x \in \mathbb{F}^{n\times 1}$ and $\B x^* A\, {\B x} = 0$ only if $\B x=\B 0$ (i.e., $A$ is {\it positive definite}).

\vsp

\DEF Let $V$ be an inner product space. We say $\B v, \B w \in V$ are {\bf orthogonal} (or {\bf perpendicular}) if $\langle \B v, \B w \rangle=0$. If $S \subseteq V$ and for all $\B v, \B w \in S$ and $\B v \not= \B w$ we have $\langle \B v, \B w \rangle=0$, then we say $S$ is an {\bf orthogonal} set. If $\B u \in V$ and $\|\B u \|=1$, we say $\B u$ is a {\bf unit vector}. Notice any non-zero vector $\B v \in V$ can be {\bf normalized} (i.e. turned into a unit vector) by rescaling: $\B u = \B v / \|\B v\|$. If $S$ is an orthogonal set of unit vectors, $S$ is called an {\bf orthonormal} set. 

\vsp

The zero vector is weird. Since $\langle \B 0, \B v \rangle=0$ for any $\B v \in V$, the zero vector is orthogonal to every vector -- including itself! In fact, by positive definiteness, the zero vector is the {\it only} vector that is orthogonal to itself.

\vsp

\LEMMA Let $S$ be an orthogonal subset of an inner product space $V$. Then $S$ is linearly independent if and only if $\B 0 \not\in S$. In particular, orthonormal sets are always linearly independent.

\bproof
If $S$ is linearly independent, $\B 0 \not\in S$ since any set containing the zero vector is linearly dependent. Now suppose $\B 0 \not \in S$ and consider $\B v_1,\dots \B v_n \in S$ (distinct vectors) and $c_1,\dots,c_n \in \F$ such that $c_1\B v_1+\cdots+c_n\B v_n=\B 0$. Since $S$ is orthgonal, we have $\langle \B v_i,\B v_j \rangle=0$ for $i\not=j$. Thus
$0 = \langle \B 0,\B v_j \rangle = \langle c_1\B v_1+\cdots+c_n\B v_n, \B v_j \rangle = c_1\langle \B v_1,\B v_j \rangle+\cdots+c_n\langle \B v_n,\B v_j \rangle = 0+\cdots+0+c_j\langle \B v_j,\B v_j \rangle+0+\cdots+0$. But $\B 0 \not\in S$ means $\B v_j\not=\B 0$ so by positivity $\langle \B v_j,\B v_j \rangle \not=0$. Therefore, $c_j=0$ (for any $j=1,\dots,n$). Consequently $S$ is linearly independent.

Finally, if $S$ is orthonormal, all its elements are unit vectors (i.e., length 1). This means $\B 0 \not\in S$ (and orthonormal implies orthogonal), so $S$ is linearly independent.
\eproof

The above proof indicates that coordinates should be particularly nice when working with orthogonal bases. Let $\beta = \{\B v_1,\dots, \B v_n\}$ be an orthogonal basis for a finite dimensional inner product space $V$. Let $\B v=c_1\B v_1+\cdots+c_n\B v_n$ for $c_1,\dots,c_n \in \F$ (i.e., the coordinates of $\B v$ are $[c_1\;\dots\;c_n]^T$). Then, just like the calculation in our proof above, $\langle \B v, \B v_i \rangle = c_1\langle \B v_1,\B v_i\rangle +\cdots+c_n\langle \B v_n,\B v_i\rangle = 0+\cdots+0+c_i\langle \B v_i,\B v_i\rangle+0+\cdots+0$. This means $c_i = \frac{\langle \B v,\B v_i\rangle}{\langle \B v_i,\B v_i\rangle}$ so that $\B v = \sum\limits_{i=1}^n \frac{\langle \B v,\B v_i\rangle}{\langle \B v_i,\B v_i\rangle} \B v_i$ (i.e., $\B v$'s $i^\text{th}$ $\beta$-coordinate is $\frac{\langle \B v,\B v_i\rangle}{\langle \B v_i,\B v_i\rangle}$). 


When $\beta$ is orthonormal (i.e., $\langle \B v_i,\B v_j\rangle = \delta_{ij}$ which is $1$ if $i=j$ and $0$ otherwise), we simply have $\B v = \sum\limits_{i=1}^n \langle \B v,\B v_i\rangle \B v_i$ (i.e., $\B v$'s $i^\text{th}$ $\beta$-coordinate is $\langle \B v,\B v_i\rangle$ -- that is -- just the inner product with the appropriate basis element). For example, the (standard) coordinates of $\B v = [1\;2\;3] \in \mathbb{R}^{1\times 3}$ are $\B v \bdot \B i =1$, $\B v\bdot \B j=2$, and $\B v\bdot \B k=3$. Also, if $\beta$ is orthonormal, notice that the matrix of the inner product relative to $\beta$ is $A=[a_{ij}]$ where $a_{ij} = \langle \B v_i,\B v_j \rangle = \delta_{ij}$. In other words, the matrix of an inner product relative to an orthonormal basis is just the identity matrix.

Lengths also work nicely with orthogonal sets. Suppose $S=\{\B w_1,\dots,\B w_n \}$ is orthogonal. Then $$\|\B w_1+\cdots+\B w_n \|^2 = \left\langle \sum\limits_{i=1}^n \B w_i, \sum\limits_{j=1}^n \B w_j \right\rangle = \sum\limits_{i=1}^n \sum\limits_{j=1}^n \langle \B v_i, \B v_j \rangle = \sum\limits_{i=1}^n \langle \B w_i,\B w_i \rangle = \|\B w_1\|^2+\cdots+\|\B w_n\|^2$$ where one of the sums drops out because $\langle \B w_i,\B w_j \rangle=0$ if $i \not= j$. This then implies that if $\beta = \{\B v_1,\dots,\B v_n \}$ is an orthonormal basis, then $\| \B v\|^2 = \sum_{i=1}^n \|\langle \B v, \B v_i \rangle \B v_i\|^2 = \sum\limits_{i=1}^n |\langle \B v, \B v_i \rangle|^2$ (i.e. the length of $\B v$ is the sum of the squares of the lengths of its coordinates.

\vsp

\vsp

\noindent {\bf\Large Gram-Schmidt Orthgonalization:} 

\vsp

Seeing that orthogonal (or even better orthonormal) bases are so nice. One might ask if these actually exist? Perhaps surprisingly, for finite dimensional inner product spaces, they always do. This is a consequence of the Gram-Schmidt orthgonalization procedure. First, we need to be reminded about orthogonal projections.

\DEF Let $V$ be an inner product space and $\B v, \B w \in V$ with $\B v \not=\B 0$. Let $\mathrm{proj}_{\B v}(\B w) = \dfrac{\langle \B w, \B v \rangle}{\|\B v\|^2}\B v$. This is the {\bf orthogonal projection} of $\B w$ onto $\B v$.

\vsp

If we let $\B p = \mathrm{proj}_{\B v}(\B w)$, then $\langle \B w - \B p, \B v \rangle = \langle \B w, \B v \rangle - \langle \B p, \B v \rangle = \langle \B w, \B v \rangle -  \frac{\langle \B w, \B v \rangle}{\|\B v\|^2} \langle \B v, \B v \rangle =  \langle \B w, \B v \rangle- \langle \B w, \B v \rangle=0$. In other words, we have $\B w = (\B w-\B p)+\B p$ where $\B w-\B p$ is orthogonal to $\B v$ and $\B p$ is parallel to (i.e., a scalar multiple of) $\B v$. Thus projections give us a way to pull apart vectors into parallel and perpendicular parts.

\vsp

Gram-Schmidt is amounts to successively removing parallel parts of vectors leaving only the orthogonal bits. Let $\alpha = \{ \B w_1,\dots, \B w_n \}$ be a linearly independent set. Recursively define a set $\beta =\{ \B v_1,\dots, \B v_n \}$ as follows:

\vspace{-0.1in}

$$\B v_1 = \B w_1 \qquad \qquad \B v_{j+1} = \B w_{j+1} - \mathrm{proj}_{\B v_1}(\B w_{j+1}) - \cdots - - \mathrm{proj}_{\B v_n}(\B w_{j+1})$$

\vspace{-0.1in}

This means each new $\B v_i$ is $\B w_i$ with its orthogonal projections onto previous stuff removed. Notice $\mathrm{span}\{\B w_1,\dots, \B w_j \}$ $= \mathrm{span}\{\B v_1,\dots,\B v_j \}$ for each $j=1,\dots,n$. We can see this since each new vector $\B v_{j+1}$ is equal to $\B w_{j+1}$ modulo the span of previous vectors. Thus swapping out $\B w_{j+1}$ with $\B v_{j+1}$ does not effect the span. Also, we have:

$\langle \B v_{j+1},\B v_k\rangle 
= \langle \B w_{j+1},\B v_k \rangle - \sum_{\ell=1}^j  \langle \mathrm{proj}_{\B v_\ell}(\B w_{j+1}), \B v_k \rangle
= \langle \B w_{j+1},\B v_k \rangle - \sum_{\ell=1}^j   \dfrac{\langle \B w_{j+1}, \B v_\ell \rangle}{\|\B v_\ell \|^2} \langle \B v_\ell, \B v_k \rangle$\\ 
$\phantom{\langle \B v_{j+1},\B v_k\rangle} = \langle \B w_{j+1},\B v_k \rangle - \dfrac{\langle \B w_{j+1}, \B v_k \rangle}{\|\B v_k \|^2} \langle \B v_k, \B v_k \rangle
= \langle \B w_{j+1},\B v_k \rangle - \langle \B w_{j+1}, \B v_k \rangle =0$ 

\noindent
where the sum collapses since orthogonality of previous vectors implies $\langle \B v_\ell, \B v_k \rangle=0$ for $\ell \not=k$. Thus each new $\B v_{j+1}$ is orthogonal to the previous $\B v_k$'s. 

\THM (Gram-Schmidt) Let $\alpha = \{\B w_1,\dots,\B w_n\}$ be a basis for a finite dimensional inner product space $V$. Define $\B v_1=\B w_1$ and then $\B v_{j+1} = \B w_{j+1}-\sum_{\ell=1}^j \mathrm{proj}_{\B v_\ell}(\B w_{j+1})$. Then for each $1 \leq j \leq n$, $\mathrm{span}\{\B w_1,\dots,\B w_j\}=\mathrm{span}\{\B v_1,\dots,\B v_j\}$. Also, $\beta = \{\B v_1,\dots,\B v_n\}$ is an orthogonal basis for $V$. Moreover, normalizing the vectors in $\beta$ yields an orthonormal basis for $V$. In particular, every finite dimensional inner product space has an orthonormal basis.

\vsp

As an addendum to this process, we note that if we apply the Gram-Schmidt process to a linearly {\it dependent} set, it still works -- after a minor modification. Suppose $\B w_{j+1}$ depends linearly on previous vectors. Then when we attempt to compute $\B v_{j+1}$, its formula will yield $\B 0$. Ultimately this means we can still apply the process to a linearly dependent set. Modification for linearly dependent sets: If we get $\B v_j=\B 0$, then we should toss out this vector (and $\B w_j$) and continue as usual. Thus we detect and then ignore the dependent stuff. We also note that some people, calculate each new vector and immediately normalize (to be a unit vector). If one does this, $\|\B v_\ell\|^2=1$ in the above calculations and our output is already an orthonormal basis.

\vsp

%It is worth mentioning that if you start with merely a spanning set, Gram-Schmidt will still produce an orthogonal basis -- if we toss out the zero vector anytime it appears (if a vector is linearly dependent on previous vectors, once we subtract out projections, nothing will be left).

In the end there is not a lot of variety in the world of finite dimensional inner product spaces. First, we define the notion of isometry (what it means to be essentially the ``same'' inner product space).

\DEF A mapping $T:V \to W$ is {\bf morphism} of inner product spaces if $T$ is linear and it preserves inner products. Specifically, for all $\B v_1,\B v_2 \in V$, we have $\langle T(\B v_1), T(\B v_2) \rangle = \langle \B v_1,\B v_2\rangle$ (the first inner product is the inner product in $W$ and the second is the inner product in $V$). When $T$ is an isomorphism (i.e., an invertible morphism), we say $T$ is an {\bf isometry}.

\vsp

The Gram-Schmidt process guarantees that every finite dimensional inner product space $V$ has an orthonormal basis $\beta$. The coordinate map $[\cdot]_\beta:V \to \mathbb{F}^{n\times 1}$ is easily seen to be an isometry when $\mathbb{F}^{n \times 1}$ is given the standard inner product (i.e., the dot product). Thus {\it every $n$-dimensional real inner product space is essentially just $\mathbb{R}^n$} (with its standard dot product) in disguise. Likewise, every $n$-dimensional complex inner product space is essentially just $\mathbb{C}^n$ (with its standard dot product). Just as with finite dimensional vector spaces, when dealing with finite dimensional inner product spaces, every computation and question can be realized in terms of column vectors (with the usual addition, scalar multiplication, and dot product).

\vsp

 %Define a linear map $\varphi$ that sends $\B v_i$ to $\B e_i$ for $i=1,\dots,n$. This map is an isomorphism (since we send a basis to a basis) and $\varphi(\langle \B v, \B w \rangle) = \varphi(\B v) \bdot \varphi(\B w)$ (it preserves inner products). Thus $V \cong \mathbb{F}^n$ (isomorphic not only as vector spaces but also as inner product spaces). So in a sense, the only inner product one can have on a finite dimensional inner product space is the dot product (up to isomorphism).

\EX Let $S = \{\B w_1, \B w_2, \B w_3 \} = \{ (4,-1,1), (8,-1,3), (5,-2,-4) \} \subseteq \R^3$.

\begin{itemize}
\item $\B v_1 = \B w_1 = (4,-1,1)$
\item $\B v_2 = \B w_2 - \dfrac{\B w_2 \bdot \B v_1}{\|\B v_1 \|^2}\B v_1 = (8,-1,3)-\dfrac{36}{18}(4,-1,1) = (0,1,1)$
\item $\B v_3 = \B w_3 - \dfrac{\B w_3 \bdot \B v_1}{\|\B v_1\|^2}\B v_1 - \dfrac{\B w_3 \bdot \B v_2}{\|\B v_2\|^2}\B v_2 = 
           (5,-2,-4)-\dfrac{18}{18}(4,-1,1)-\dfrac{-6}{2}(0,1,1) = (1,2,-2)$
\end{itemize}

We have $\mathrm{span}(S) = \mathrm{span}\{\B v_1,\B v_2,\B v_3\}$ but our new spanning set is an orthogonal set (without zero vector). This implies that both $S$ and our new set are linearly independent sets. Since $\dim(\R^3)=3$, these are bases for $\R^3$. Specifically, $\{(4,-1,1),(0,1,1),(1,2,-2)\}$ is an orthogonal basis for $\R^3$. 
We can normalize our set and have that $\left\{ \frac{1}{3\sqrt{2}}(4,-1,1), \frac{1}{\sqrt{2}}(0,1,1), \frac{1}{3}(1,2,-2)\right\}$ is an orthonormal basis for $\R^3$.

\vsp

\EX Consider $S = \{\B w_1, \B w_2, \B w_3, \B w_4 \} = \{ (1,0,0,i), (1+i,1,1,1+i),(3+i,1,1,1+3i),(0,1+2i,0,1) \} \subseteq \C^4$. 

\begin{itemize}
\item $\B v_1 = \B w_1 = (1,0,0,i)$
\item $\B v_2 = \B w_2 - \dfrac{\B w_2 \bdot \B v_1}{\|\B v_1 \|^2}\B v_1 = (1+i,1,1,1+i) - \dfrac{2}{2}(1,0,0,i) = (i,1,1,1)$
\item Notice that $\B w_3 - \dfrac{\B w_3 \bdot \B v_1}{\|\B v_1\|^2}\B v_1 - \dfrac{\B w_3 \bdot \B v_2}{\|\B v_2\|^2}\B v_2 = (3+i,1,1,1+3i) - \dfrac{6}{2}(1,0,0,i) - \dfrac{4}{4}(i,1,1,1) = (0,0,0,0)$. 

This means that $\B w_3 \in \mathrm{span}\{\B v_1,\B v_2 \} = \mathrm{span}\{\B w_1,\B w_2 \}$. So we can toss out $\B w_3$ (and $\B 0$). 
\item $\B v_3 = \B w_4 - \dfrac{\B w_4 \bdot \B v_1}{\|\B v_1\|^2}\B v_1 - \dfrac{\B w_4 \bdot \B v_2}{\|\B v_2\|^2}\B v_2 = (0,1+2i,0,1) - \dfrac{-i}{2}(1,0,0,i)-\dfrac{2+2i}{4}(i,1,1,1) = \dfrac{1}{2}(1,1+3i,-1-i,-i)$ 
\end{itemize}

\vspace{0.1in}

\noindent
We have $\mathrm{span}(S) = \mathrm{span}\left\{ (1,0,0,i), (i,1,1,1), \frac{1}{2}(1,1+3i,-1-i,-i) \right\}$ where our new set is an orthogonal basis for $\mathrm{span}(S)$. 
If we normalize our new vectors, we have $\alpha = \left\{ \frac{1}{\sqrt{2}}(1,0,0,i), \frac{1}{2}(i,1,1,1), \frac{1}{\sqrt{14}}(1,1+3i,-1-i,-i) \right\}$ which is an orthonormal basis for $\mathrm{span}(S)$.

\vsp

\EX  Consider the inner product $\displaystyle \langle f,g \rangle = \int_{-1}^1 f(x)g(x)\,dx$ defined on $\mathbb{R}[x]$. If we run the Gram-Schmidt algorithm (unnormalized) on $\{1,x,x^2,\dots \}$ and then rescale them so that $\|P_n(x)\|^2=\dfrac{2}{2n+1}$, we will get a collection of polynomials known as {\it Legendre polynomials}. Let us denote theses by $P_0(x),P_1(x),$ etc. We calculate the first few Legendre polynomials:

\begin{itemize}
\item $f_0(x) = 1$. We need $2(?)^2=(?)^2\int_{-1}^1 1^2\,dx = (?)^2\|f_0(x)\|^2=\dfrac{2}{2(0)+1}$, so $?=1$ and thus $P_0(x)=f_0(x)=1$.
\item $f_1(x) = x - \dfrac{\langle x,1 \rangle}{\|1\|^2}1 = x - \dfrac{\int_{-1}^1 x \cdot 1 \,dx}{\int_{-1}^1 1^2 \,dx}\,1 = x - \dfrac{0}{2}\,1 = x$. Using the Legendre normalization, we need\\ \force\hfill $\dfrac{2}{3}(?)^2 = \|(?)^2f_1(x)\|^2 = \dfrac{2}{2(1)+1}$ so $?=1$ and thus $P_1(x)=f_1(x)=x$.
\item $f_2(x) = x^2 - \dfrac{\langle x^2,1 \rangle}{\|1\|^2}1 - \dfrac{\langle x^2,x \rangle}{\|x\|^2}x = x^2 - \dfrac{\int_{-1}^1 x^2 \cdot 1 \,dx}{\int_{-1}^1 1^2 \,dx}\,1 - \dfrac{\int_{-1}^1 x^2 \cdot x \,dx}{\int_{-1}^1 x^2 \,dx}\,x = x^2 -\dfrac{2/3}{2}\,1 - \dfrac{0}{2/3}x = x^2-\dfrac{1}{3}$.\\ Again, we need $\dfrac{8}{45}(?)^2 = (?)^2\int_{-1}^1 (x^2-1/3)^2\,dx=\|?\,f_2(x)\|^2 = \dfrac{2}{2(2)+1}=\dfrac{2}{5}$ so $(?)^2=9/4$ and thus $?=3/2$. Therefore, $P_2(x) = (3/2)f_2(x) = \dfrac{3}{2}x^2-\dfrac{1}{2}$.
\end{itemize}

\vsp

\noindent {\bf\Large Connection with Dual Spaces:} 

\vsp

Notice that since the inner product is linear in its first slot, for any $\B w \in W$ we have $\langle \cdot\,,\B w \rangle :W \to \mathbb{F}$ is a linear map. In other words, the mapping $f$ defined by $f(\B v) = \langle \B v,\B w\rangle$ is a dual vector (i.e., $\langle \cdot\,,\B w\rangle \in W^*$). In fact, when $\beta = \{\B w_1,\dots,\B w_m\}$ is an orthonormal basis, $\beta^* = \{ \langle \cdot\,,\B w_1 \rangle, \dots, \langle \cdot\,,\B w_m \rangle \}$ is its dual basis. Why? Denoting $\B w_j^* = \langle \cdot\,,\B w_j \rangle$, we have $\B w_j^*(\B w_i) = \langle \B w_i,\B w_j\rangle = \delta_{ij}$. 

\vsp

Suppose $T:V \to W$ is a linear transformation of finite dimensional vector spaces where $\alpha = \{\B v_1,\dots,\B v_n\}$ is a basis for $V$ and $\beta=\{ \B w_1,\dots,\B w_m\}$ is an orthogonal basis for $W$. Then the $(i,j)$-entry of the coordinate matrix $[T]_\alpha^\beta$ is $\B w_i^*(T(\B v_j)) = \langle T(\B v_j), \B w_i \rangle$.

\vsp

Working over a real field, we can make this cleaner. If $\mathbb{F}$ is a subfield of the real numbers, the inner product is bilinear and symmetric. Thus the $(i,j)$-entry of the coordinate matrix $[T]_\alpha^\beta$ (still assuming $\beta$ is orthonormal) could be written as $\langle \B w_i,T(\B v_j) \rangle$. Also, in this case, the association of $\B w \in W$ with $\langle \B w,\,\cdot\rangle = \langle \cdot\,,\B w\rangle \in W^*$ is actually an isomorphism. For any finite dimensional vector space, $W$, we have $W \cong W^*$. But this isomorphism is not ``natural'' in a some technical sense I won't get into (the isomorphism is basis dependent -- we usually show $W \cong W^*$ by choosing a basis for $W$ and relating it to a dual basis for $W^*$). It turns out that picking an isomorphism between $W$ and $W^*$ is almost the same thing as choosing an inner product on $W$. For a finite dimensional real inner product space, its inner product can allow us (with some care) to treat vectors in our space and its dual as interchangeable.


\end{document}
