Positive (Semi)Definite Matrices

PreviousGeometrical Representation NextMatrix Interpretation

Last updated 11 months ago

Positive (Semi)Definite Matrices

Symmetric Matrix

When the matrix entries are the same across the diagonal.

A = A^T

Positive Definite or Positive Semi-definite

It's defined for symmetric matrix $M$ .

An $n\times n$ symmetric real matrix $M$ is said to be positive-definite if $x^TMx > 0$ for all non-zero $\mathbf {x}$ in $\mathbb {R} ^{n}$ . Formally

Positive Semi-Definite

An $n\times n$ symmetric real matrix $M$ is said to be positive semi-definite if $x^TMX \geq 0$ for all non-zero $\mathbf {x}$ in $\mathbb {R} ^{n}$ .

What does positive definite mean? It's similar to saying for example in one dimensional that $kx^2$ should be positive $\forall x$ which means $k>0$ . Similarly, for higher dimensions, to get the higher dimensional parabola always positive we need a positive definite matrix.

PSD from the lens of dot-product

Let's see

We can consider $x^TAx$ as dot product $(x^T)Ax$ . Now PSD means $(x^T)(Ax) \geq 0$ which means $x^T$ and $Ax$ form an acute angle between them i.e point in the same directoin. Hence, it means that $A$ is a transformation that keeps $Ax$ in the same direction as $x$ .

Properties of PD or PSD matrices

All the eigenvalues of a positive definite matrix are positive. A matrix is positive definite if it’s symmetric and all its eigenvalues are positive.

As established all the transformed vectors Ax are in same direction as x for positive dot product. Let's consider some eigen vector $v_i$ with it's eigen value $\lambda_i$ . Now we know that $Av_i = \lambda_iv_i$ , so for $(v_i^T)(Av_i) \geq 0 \implies v_i^T\lambda_uv_i \geq 0 \implies \lambda_i ||v_i|| \geq 0 \implies \lambda_i \geq 0$

\text{det}(A) = \prod_{n=1}^N \lambda_n

So if all the $\lambda_n$ are positive, determinant will also be positive.

Decomposition

If $A$ is Positive semi-definite if and only if it can be decomposed as follows:

A = B^TB

Also

A = LL^T

Where $L$ is a lower triangular matrix. And this is called Cholesky decomposition.

PSD Matrices defining Ellipsoid

Let's say we have $Q(x) = x^TAx$ , be a positive definite form. Then the set

\{x \in \mathbf R^n: Q(x)=c\}

where $c$ is some constant. This set of points forms an ellipsoid.

Resrouces

Why does PSD matrices have to be symmetric?

Full Rank Matrix

A matrix is said to have full rank if its rank equals the largest possible for a matrix of the same dimensions, which is the lesser of the number of rows and columns.

Resources

Talks about multiple properties of PSD and also gives insight about Hessians of functions as they are PSD