Automatic Differentiation

Frameworks such as tensorflow or pytorch use auto-differentiation to find the gradient for back propagation of gradients.

Two types of auto-differentiation:

• Forward Mode

• Reverse Mode

Tutorial on Automatic Differentiationtheoryandpractice.org

Module 3: The Reverse Mode of Automatic Differentiation — Auto-eD documentationauto-ed.readthedocs.io

Concise Comparison of Forward vs Backward Model

Forward Automatic Differentiationliqimai.github.io

vJp and backward pass

Hessians in AD

How to compute Hessian-vector products? | ICLR Blogposts 2024iclr-blogposts.github.io

Intuitive way of learning why the hessian vector product is all that we need?

ChatGPT - JAX Autodiff ExplanationChatGPT