Automatic Differentiation

Frameworks such as tensorflow or pytorch use auto-differentiation to find the gradient for back propagation of gradients.

Two types of auto-differentiation:

  • Forward Mode

  • Reverse Mode

Concise Comparison of Forward vs Backward Model

vJp and backward pass

Hessians in AD

Intuitive way of learning why the hessian vector product is all that we need?

Last updated