> For the complete documentation index, see [llms.txt](https://theshank.gitbook.io/ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://theshank.gitbook.io/ai/linear-algebra/matrix-calculus.md). # Matrix Calculus Gradients, Jacobians, etc in Matrix algebra ## Gradient $$f: \mathcal{R}^d \rightarrow \mathcal{R}$$ where $$f$$ is some function. $$ \hat y = f(\mathbf w) $$ Here $$\mathbf w \in \mathcal{R}^d$$, now for some reason we have want the derivative of $$f$$ wrt to each elements of $$\mathbf w$$, that will be called gradient. Represented as follows: $$ \nabla\_\mathbf w f(\mathbf w) = \[\frac{\partial f}{\partial w\_1}, \dots, \frac{\partial f}{\partial w\_d}]^T $$ The $$\text{grad}$$ $$\nabla\_\mathbf w f(\mathbf w)$$is supposed to be also a horizontal vector of same dimension as $$\mathbf w$$. * **Note that the** $$f(\mathbf w)$$ is scalar values function but $$\nabla\_\mathbf w f(\mathbf w)$$ is actually a vector values function. * The gradients are perpendicular to the contour lines of the curve $$f$$. * The gradient of $$f$$points in the direction of the steepest ascent. Why? Think using directional derivatives. {% hint style="info" %} Something I get confused it gradient should be a row or column vector? **Look here** {% endhint %} ### Gradient in Matrix, Vector forms Let's say that $$f( \mathbf w) = \mathbf w\cdot \mathbf x = \mathbf w^T\mathbf x = \mathbf x^T \mathbf w$$ which is simply a linear function. Here x is some input vector of dimension same as w. Then we have $$ \nabla\_{\mathbf w} f(\mathbf w) = \nabla\_\mathbf w (\mathbf w^T \mathbf x) = \mathbf x $$ ## Jacobian Now let $$f: \mathcal{R}^n \rightarrow \mathcal{R}^m$$. In this case we would have a jacobian $$J$$ $$ J = \begin{bmatrix} \frac{\partial \mathbf f}{\partial x\_1} & \dots & \frac{\partial \mathbf f}{\partial x\_n}\\ \end{bmatrix} \\ \= \begin{bmatrix} \nabla^T f\_1\\ \dots\\ \nabla^T f\_m\\ \end{bmatrix} \\ \= \begin{bmatrix} \frac{\partial f\_1}{\partial x\_1} & \dots & \frac{\partial f\_1}{\partial x\_n}\\ \vdots & \dots & \vdots\\ \frac{\partial f\_m}{\partial x\_1} & \dots & \frac{\partial f\_m}{\partial x\_n}\\ \end{bmatrix} $$

Dimension of reulting Jacobian shape based on function and input shape

{% hint style="info" %} ~~Note: Jacobian and Gradient are transpose of each other. So, if you try to calculate derivative of single value function with respect to a vector using Jacobian, it would be a row vector. So to get a gradient you have transpose it.~~ -- Not sure about this anymore {% endhint %} ## Chain Rule ### Vector Let $$\mathbf{f, g}$$ be two vector valued function and $$x$$ be a scalar. Then $$ \nabla\_x \mathbf f = \frac{\partial \mathbf {f(g(}x))}{\partial x} = \frac{\partial \mathbf f}{\partial \mathbf g}\frac{\partial \mathbf g}{\partial x} $$ Now if there are multiple paramteres i.e. it's a vector $$\mathbf x$$. Then it's $$ \nabla\_{\mathbf x} \mathbf f = \frac{\partial \mathbf {f(g(x))}}{\partial \mathbf x} = \frac{\partial \mathbf f}{\partial \mathbf g}\frac{\partial \mathbf g}{\partial \mathbf x} $$

Chain Rule for different dimension of variables.

## Resources {% embed url="" %} {% embed url="" %} {% embed url="" %} {% embed url="" %}