Implicit Function Theorem for optimization

Intuition of Implicit differentiation

Implicit differentiation is mainly used for when we have implicit functions.

Think about a function $g(x,y)=c$, now this function implicitly defines a relation between the variables x and y. $x^2+y^2 = 1$ is an example of such a function, where x and y are points on circle of radius 1.

Now, the point of implict differentiation is that in a very you can define a explicit function in a very local region of the graph such that: $y=h(x)$. but this hold for only very near points of $(x,y)$. And how you find this $y=h(x)$ is given by implicit differentiation the contraint

So you differentiaite on both side and you get

$$dg = \frac{\partial g}{\partial x} dx + \frac{\partial g}{\partial y}dy = 0 \\ \quad \\ \frac{dy}{dx} = - \frac{\frac{\partial g}{\partial x}}{\frac{\partial g}{\partial y}}$$

Now this $\frac{dy}{dx}$ defines a locally applicable $y=h(x)$, as you can just integrate the differentiable equation to find the h(x).

This is called Implicit Function Theorem.

Optimization with implicit differentiationMathematics Stack Exchange

Implicit Gradient Differentiation — TorchOpt 0.7.4.dev20+g960fb0a documentationtorchopt.readthedocs.io

Using HvP and CG, to calculate gradients involving implicit gradients

ChatGPTChatGPT

Resources

The Implicit Differentiation Formulas - Mathonlinemathonline.wikidot.com