Gradient
f : R d → R f: \mathcal{R}^d \rightarrow \mathcal{R} f : R d → R f f f
y ^ = f ( w ) \hat y = f(\mathbf w) y ^ ​ = f ( w ) Here w ∈ R d \mathbf w \in \mathcal{R}^d w ∈ R d f f f w \mathbf w w
∇ w f ( w ) = [ ∂ f ∂ w 1 , … , ∂ f ∂ w d ] T \nabla_\mathbf w f(\mathbf w) = [\frac{\partial f}{\partial w_1}, \dots, \frac{\partial f}{\partial w_d}]^T ∇ w ​ f ( w ) = [ ∂ w 1 ​ ∂ f ​ , … , ∂ w d ​ ∂ f ​ ] T The grad \text{grad} grad ∇ w f ( w ) \nabla_\mathbf w f(\mathbf w) ∇ w ​ f ( w ) w \mathbf w w
Note that the f ( w ) f(\mathbf w) f ( w ) ∇ w f ( w ) \nabla_\mathbf w f(\mathbf w) ∇ w ​ f ( w )
The gradients are perpendicular to the contour lines of the curve f f f
The gradient of f f f
Let's say that f ( w ) = w â‹… x = w T x = x T w f( \mathbf w) = \mathbf w\cdot \mathbf x = \mathbf w^T\mathbf x = \mathbf x^T \mathbf w f ( w ) = w â‹… x = w T x = x T w
∇ w f ( w ) = ∇ w ( w T x ) = x \nabla_{\mathbf w} f(\mathbf w) = \nabla_\mathbf w (\mathbf w^T \mathbf x) = \mathbf x ∇ w ​ f ( w ) = ∇ w ​ ( w T x ) = x Jacobian
Now let f : R n → R m f: \mathcal{R}^n \rightarrow \mathcal{R}^m f : R n → R m J J J
J = [ ∂ f ∂ x 1 … ∂ f ∂ x n ] = [ ∇ T f 1 … ∇ T f m ] = [ ∂ f 1 ∂ x 1 … ∂ f 1 ∂ x n ⋮ … ⋮ ∂ f m ∂ x 1 … ∂ f m ∂ x n ] J = \begin{bmatrix}
\frac{\partial \mathbf f}{\partial x_1} & \dots & \frac{\partial \mathbf f}{\partial x_n}\\
\end{bmatrix} \\
= \begin{bmatrix}
\nabla^T f_1\\
\dots\\
\nabla^T f_m\\
\end{bmatrix} \\
= \begin{bmatrix}
\frac{\partial f_1}{\partial x_1} & \dots & \frac{\partial f_1}{\partial x_n}\\
\vdots & \dots & \vdots\\
\frac{\partial f_m}{\partial x_1} & \dots & \frac{\partial f_m}{\partial x_n}\\
\end{bmatrix} J = [ ∂ x 1 ​ ∂ f ​ ​ … ​ ∂ x n ​ ∂ f ​ ​ ] = ​ ∇ T f 1 ​ … ∇ T f m ​ ​ ​ = ​ ∂ x 1 ​ ∂ f 1 ​ ​ ⋮ ∂ x 1 ​ ∂ f m ​ ​ ​ … … … ​ ∂ x n ​ ∂ f 1 ​ ​ ⋮ ∂ x n ​ ∂ f m ​ ​ ​ ​
Dimension of reulting Jacobian shape based on function and input shape Chain Rule
Vector
Let f , g \mathbf{f, g} f , g x x x
∇ x f = ∂ f ( g ( x ) ) ∂ x = ∂ f ∂ g ∂ g ∂ x \nabla_x \mathbf f = \frac{\partial \mathbf {f(g(}x))}{\partial x} = \frac{\partial \mathbf f}{\partial \mathbf g}\frac{\partial \mathbf g}{\partial x} ∇ x ​ f = ∂ x ∂ f ( g ( x )) ​ = ∂ g ∂ f ​ ∂ x ∂ g ​ Now if there are multiple paramteres i.e. it's a vector x \mathbf x x
∇ x f = ∂ f ( g ( x ) ) ∂ x = ∂ f ∂ g ∂ g ∂ x \nabla_{\mathbf x} \mathbf f = \frac{\partial \mathbf {f(g(x))}}{\partial \mathbf x} = \frac{\partial \mathbf f}{\partial \mathbf g}\frac{\partial \mathbf g}{\partial \mathbf x} ∇ x ​ f = ∂ x ∂ f ( g ( x )) ​ = ∂ g ∂ f ​ ∂ x ∂ g ​
Full Jacobian Calculation
Chain Rule for different dimension of variables. Resources
