> For the complete documentation index, see [llms.txt](https://theshank.gitbook.io/ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://theshank.gitbook.io/ai/bayesian-deep-learning.md).

# Bayesian Deep Learning

Using Bayesian inference in deep learning

### Bayesian Neural Network

The idea behind Bayesian Neural Network is to model the network's wights $$W$$as a distribution $$p(W|D)$$conditioned on the training data $$D$$, instead of a deterministic point estimates. \
By placing a prior over the weights e.g. $$W \sim N(0,I)$$, the network training can be interpreted as determining a posterior over the weights given the training data: $$p(W|T)$$. \
However, evaluating this posterior is not tractable without approximation techniques. 

### Bayesian Inference

![](https://1877261540-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LFDuA0A2VRqmT31Blrq%2F-LlT2FQ6R6RKA5rKZA0O%2F-LlTSp1tJ7eXCJGn1myc%2Fimage.png?alt=media\&token=18362b6f-52a3-4afd-a047-20ed146b4db7)

### Priors in weights in NNs

We can apply this process to neural networks and come up with the probability distribution over the network weights, $$w$$ , given the training data, $$p ( w|D )$$ .

![](https://1877261540-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LFDuA0A2VRqmT31Blrq%2F-LlTZjvGCp3NkTKArVHb%2F-LlT_fy7o7EO0hNyizH2%2Fimage.png?alt=media\&token=736e4a01-a2a6-4058-b10c-60f4afa1a627)

![](https://1877261540-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LFDuA0A2VRqmT31Blrq%2F-LlTZjvGCp3NkTKArVHb%2F-LlTa6nKz6EV02GxfRiJ%2Fimage.png?alt=media\&token=0946b997-b648-43ce-ae5c-2bb934260eb2)

![](https://1877261540-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LFDuA0A2VRqmT31Blrq%2F-LlTZjvGCp3NkTKArVHb%2F-LlTaCD3XXiqUEujebiL%2Fimage.png?alt=media\&token=32cff9ea-9b5f-424b-a7c9-6bbaa64c6c04)

![](https://1877261540-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LFDuA0A2VRqmT31Blrq%2F-LlTZjvGCp3NkTKArVHb%2F-LlTaLZoOQ-O11yzCH4H%2Fimage.png?alt=media\&token=6ff04d98-dd5e-4d67-8aeb-e5e2de625152)

![](https://1877261540-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LFDuA0A2VRqmT31Blrq%2F-LlTZjvGCp3NkTKArVHb%2F-LlTaRYjclGbHU4tApKS%2Fimage.png?alt=media\&token=72fd391e-a3ae-410f-86a8-e9618b25c7ac)

{% file src="/files/-LlTZWdQKWq6T8XMqpX9" %}
Bayesian Learning of weights in NN
{% endfile %}

#### Take away

When you add the **"Regularization or Weight Decay",** it simply assumes that my weights follows zero centered gaussian distribution. Hence, it is a way to incorporate a prior.  

### Resources

{% embed url="<https://slideslive.com/38923183/deep-learning-with-bayesian-principles>" %}

Neurips 2019 talk - **See at 19:00**, interesting

* <https://www.cs.cmu.edu/afs/cs/academic/class/15782-f06/slides/bayesian.pdf>
* <https://stats.stackexchange.com/a/335422>