> For the complete documentation index, see [llms.txt](https://theshank.gitbook.io/ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://theshank.gitbook.io/ai/linear-algebra/positive-semi-definite-matrices.md). # Positive (Semi)Definite Matrices ## Symmetric Matrix When the matrix entries are the same across the diagonal. ![](/files/z4k2RMQhzZ2cT05Ub1Oj) $$ A = A^T $$ ## Positive Definite or Positive Semi-definite It's defined for symmetric matrix $$M$$. An $$n\times n$$ symmetric real matrix $$M$$ is said to be positive-definite if $$x^TMx > 0$$for all non-zero $$\mathbf {x}$$in $$\mathbb {R} ^{n}$$. Formally **Positive Semi-Definite** An $$n\times n$$ symmetric real matrix $$M$$ is said to be positive semi-definite if $$x^TMX \geq 0$$ for all non-zero $$\mathbf {x}$$in $$\mathbb {R} ^{n}$$. **What does positive definite mean?** It's similar to saying for example in one dimensional that $$kx^2$$should be positive $$\forall x$$which means $$k>0$$. Similarly, for higher dimensions, to get the higher dimensional parabola always positive we need a positive definite matrix. ### PSD from the lens of dot-product Let's see We can consider $$x^TAx$$ as dot product $$(x^T)Ax$$. Now PSD means $$(x^T)(Ax) \geq 0$$ which means $$x^T$$ and $$Ax$$ form an acute angle between them i.e point in the same directoin. Hence, it means that $$A$$ is a transformation that keeps $$Ax$$ in the same direction as $$x$$. ### Properties of PD or PSD matrices **All the eigenvalues of a positive definite matrix are positive. A matrix is positive definite if it’s symmetric and all its eigenvalues are positive.** As established all the transformed vectors Ax are in same direction as x for positive dot product. Let's consider some eigen vector $$v\_i$$with it's eigen value $$\lambda\_i$$. Now we know that $$Av\_i = \lambda\_iv\_i$$ , so for $$(v\_i^T)(Av\_i) \geq 0 \implies v\_i^T\lambda\_uv\_i \geq 0 \implies \lambda\_i ||v\_i|| \geq 0 \implies \lambda\_i \geq 0$$ **The determinant of a positive definite matrix is always positive, so a positive definite matrix is always** [**nonsingular**](https://mathworld.wolfram.com/NonsingularMatrix.html)**.** $$ \text{det}(A) = \prod\_{n=1}^N \lambda\_n $$ So if all the $$\lambda\_n$$are positive, determinant will also be positive. **Decomposition** If $$A$$ is **Positive semi-definite if and only if it can be decomposed as follows:** $$ A = B^TB $$ Also $$ A = LL^T $$ Where $$L$$ is a lower triangular matrix. And this is called Cholesky decomposition. ## PSD Matrices defining Ellipsoid Let's say we have $$Q(x) = x^TAx$$, be a positive definite form. Then the set $$ {x \in \mathbf R^n: Q(x)=c} $$ where $$c$$ is some constant. This set of points forms an ellipsoid. ### Resrouces {% embed url="" %} Very comprehensive study of PSD Matrices. Relating it to dot product, quadratic programming, and visualization. {% endembed %} {% embed url="" %} {% embed url="" %} #### Why does PSD matrices have to be symmetric? {% embed url="" %} ### Full Rank Matrix In [linear algebra](https://en.wikipedia.org/wiki/Linear_algebra), the **rank** of a [matrix](https://en.wikipedia.org/wiki/Matrix_\(mathematics\)) A is the [dimension](https://en.wikipedia.org/wiki/Dimension_\(vector_space\)) of the [vector space](https://en.wikipedia.org/wiki/Vector_space) generated (or [spanned](https://en.wikipedia.org/wiki/Linear_span)) by its columns.This corresponds to the maximal number of [linearly independent](https://en.wikipedia.org/wiki/Linearly_independent) columns of A. This, in turn, is identical to the dimension of the vector space spanned by its rows. Rank is thus a measure of the "[nondegenerateness](https://en.wikipedia.org/wiki/Degenerate_form)" of the [system of linear equations](https://en.wikipedia.org/wiki/System_of_linear_equations) and [linear transformation](https://en.wikipedia.org/wiki/Linear_transformation) encoded by A. A matrix is said to have **full rank** if its rank equals the largest possible for a matrix of the same dimensions, which is the lesser of the number of rows and columns. ## Resources * Talks about multiple properties of PSD and also gives insight about Hessians of functions as they are PSD {% embed url="" %}