Norms

Norms are mathematical concepts to measure the length of vector.

Norm is any function that satisfy following properties:

  • f(x)=0f(x)=0for all x=0x=0

  • f(x+y)>f(x)+f(y)f(x+y)>f(x)+f(y), triangle inequality

  • f(αx)=αf(x)f(\alpha x)=|\alpha|f(x)for all scalar α\alpha

Example of Norms

L^p Norm

LPL^Pnorm is given as

xp=(ixip)1p||x||_p = (\sum_i|x_i|^p)^{\frac{1}{p}}

So, L2L^2is euclidean norm. Many times, square of L2L^2norm is used. Many times denoted simply by x||x||. For vector xx

x=xTx||x|| = x^Tx

Many times L1L^1used when difference between zero and non-zero elements are more important.

Max Norm

It is denoted by LL^\infty. It is simply the absolute value of element with largest magnitude.

x=maxixi||x||_\infty = \max_i |x_i|

Frobenius Norm

This is norm used for measure size of a matrix.

AF=i,jAi,j2=Tr(AAT)||A||_F = \sqrt{\sum_{i,j}A_{i,j}^2} = \sqrt{\text{Tr}(AA^T)}

Properties:

  • LpL_p norms are not differentiable at origin, because of using x||x|| in the norm, which is not differentiable at origin.

  • For 0p10 \leq p \leq 1, LpL_p isn't convex as it doesn't follow the triangle inequality. Using LpL_p induces sparsity in the features.

Last updated