Positive (Semi)Definite Matrices

Symmetric Matrix

When the matrix entries are the same across the diagonal.

A=ATA = A^T

Positive Definite or Positive Semi-definite

It's defined for symmetric matrix MM.

An n×nn\times n symmetric real matrix MM is said to be positive-definite if xTMx>0x^TMx > 0for all non-zero x\mathbf {x} in Rn\mathbb {R} ^{n}. Formally

Positive Semi-Definite

An n×nn\times n symmetric real matrix MM is said to be positive semi-definite if xTMX0x^TMX \geq 0 for all non-zero x\mathbf {x} in Rn\mathbb {R} ^{n}.

What does positive definite mean? It's similar to saying for example in one dimensional that kx2kx^2should be positive x\forall xwhich means k>0k>0. Similarly, for higher dimensions, to get the higher dimensional parabola always positive we need a positive definite matrix.

PSD from the lens of dot-product

Let's see

We can consider xTAxx^TAx as dot product (xT)Ax(x^T)Ax. Now PSD means (xT)(Ax)0(x^T)(Ax) \geq 0 which means xTx^T and AxAx form an acute angle between them i.e point in the same directoin. Hence, it means that AA is a transformation that keeps AxAx in the same direction as xx.

Properties of PD or PSD matrices

All the eigenvalues of a positive definite matrix are positive. A matrix is positive definite if it’s symmetric and all its eigenvalues are positive.

As established all the transformed vectors Ax are in same direction as x for positive dot product. Let's consider some eigen vector viv_iwith it's eigen value λi\lambda_i. Now we know that Avi=λiviAv_i = \lambda_iv_i , so for (viT)(Avi)0    viTλuvi0    λivi0    λi0(v_i^T)(Av_i) \geq 0 \implies v_i^T\lambda_uv_i \geq 0 \implies \lambda_i ||v_i|| \geq 0 \implies \lambda_i \geq 0

The determinant of a positive definite matrix is always positive, so a positive definite matrix is always nonsingular.

det(A)=n=1Nλn\text{det}(A) = \prod_{n=1}^N \lambda_n

So if all the λn\lambda_nare positive, determinant will also be positive.

Decomposition

If AA is Positive semi-definite if and only if it can be decomposed as follows:

A=BTBA = B^TB

Also

A=LLTA = LL^T

Where LL is a lower triangular matrix. And this is called Cholesky decomposition.

PSD Matrices defining Ellipsoid

Let's say we have Q(x)=xTAxQ(x) = x^TAx, be a positive definite form. Then the set

{xRn:Q(x)=c}\{x \in \mathbf R^n: Q(x)=c\}

where cc is some constant. This set of points forms an ellipsoid.

Resrouces

Very comprehensive study of PSD Matrices. Relating it to dot product, quadratic programming, and visualization.

Why does PSD matrices have to be symmetric?

Full Rank Matrix

In linear algebra, the rank of a matrix A is the dimension of the vector space generated (or spanned) by its columns.This corresponds to the maximal number of linearly independent columns of A. This, in turn, is identical to the dimension of the vector space spanned by its rows. Rank is thus a measure of the "nondegenerateness" of the system of linear equations and linear transformation encoded by A.

A matrix is said to have full rank if its rank equals the largest possible for a matrix of the same dimensions, which is the lesser of the number of rows and columns.

Resources

  • Talks about multiple properties of PSD and also gives insight about Hessians of functions as they are PSD

Last updated