Automatic Differentiation

Frameworks such as tensorflow or pytorch use auto-differentiation to find the gradient for back propagation of gradients.

Two types of auto-differentiation:

  • Forward Mode

  • Reverse Mode

Tutorial on Automatic Differentiation
LogoModule 3: The Reverse Mode of Automatic Differentiation — Auto-eD documentation

Concise Comparison of Forward vs Backward Model

LogoForward Automatic Differentiation

vJp and backward pass

Hessians in AD

LogoHow to compute Hessian-vector products? | ICLR Blogposts 2024
PreviousActivation FunctionsNextSoftmax Classifier and Cross Entropy

Last updated