Automatic Differentiation

Frameworks such as tensorflow or pytorch use auto-differentiation to find the gradient for back propagation of gradients.

Two types of auto-differentiation:

• Forward Mode

• Reverse Mode

Tutorial on Automatic Differentiation

Module 3: The Reverse Mode of Automatic Differentiation — Auto-eD documentation

Concise Comparison of Forward vs Backward Model

Forward Automatic Differentiation

vJp and backward pass

Hessians in AD

How to compute Hessian-vector products? | ICLR Blogposts 2024