---
description: Using Bayesian inference in deep learning
---

# Bayesian Deep Learning

### Bayesian Neural Network

The idea behind Bayesian Neural Network is to model the network's wights $$W$$as a distribution $$p(W|D)$$conditioned on the training data $$D$$, instead of a deterministic point estimates. \
By placing a prior over the weights e.g. $$W \sim N(0,I)$$, the network training can be interpreted as determining a posterior over the weights given the training data: $$p(W|T)$$. \
However, evaluating this posterior is not tractable without approximation techniques.&#x20;

### Bayesian Inference

![](https://1877261540-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LFDuA0A2VRqmT31Blrq%2F-LlT2FQ6R6RKA5rKZA0O%2F-LlTSp1tJ7eXCJGn1myc%2Fimage.png?alt=media\&token=18362b6f-52a3-4afd-a047-20ed146b4db7)

### Priors in weights in NNs

We can apply this process to neural networks and come up with the probability distribution over the network weights, $$w$$ , given the training data, $$p ( w|D )$$ .

![](https://1877261540-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LFDuA0A2VRqmT31Blrq%2F-LlTZjvGCp3NkTKArVHb%2F-LlT_fy7o7EO0hNyizH2%2Fimage.png?alt=media\&token=736e4a01-a2a6-4058-b10c-60f4afa1a627)

![](https://1877261540-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LFDuA0A2VRqmT31Blrq%2F-LlTZjvGCp3NkTKArVHb%2F-LlTa6nKz6EV02GxfRiJ%2Fimage.png?alt=media\&token=0946b997-b648-43ce-ae5c-2bb934260eb2)

![](https://1877261540-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LFDuA0A2VRqmT31Blrq%2F-LlTZjvGCp3NkTKArVHb%2F-LlTaCD3XXiqUEujebiL%2Fimage.png?alt=media\&token=32cff9ea-9b5f-424b-a7c9-6bbaa64c6c04)

![](https://1877261540-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LFDuA0A2VRqmT31Blrq%2F-LlTZjvGCp3NkTKArVHb%2F-LlTaLZoOQ-O11yzCH4H%2Fimage.png?alt=media\&token=6ff04d98-dd5e-4d67-8aeb-e5e2de625152)

![](https://1877261540-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LFDuA0A2VRqmT31Blrq%2F-LlTZjvGCp3NkTKArVHb%2F-LlTaRYjclGbHU4tApKS%2Fimage.png?alt=media\&token=72fd391e-a3ae-410f-86a8-e9618b25c7ac)

{% file src="https://1877261540-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LFDuA0A2VRqmT31Blrq%2F-LlT2FQ6R6RKA5rKZA0O%2F-LlTZWdQKWq6T8XMqpX9%2Fbayesiandeeplearning.pdf?alt=media&token=e354e20f-20f6-482d-9422-512cd330e262" %}
Bayesian Learning of weights in NN
{% endfile %}

#### Take away

When you add the **"Regularization or Weight Decay",** it simply assumes that my weights follows zero centered gaussian distribution. Hence, it is a way to incorporate a prior. &#x20;

### Resources

{% embed url="https://slideslive.com/38923183/deep-learning-with-bayesian-principles" %}

Neurips 2019 talk - **See at 19:00**, interesting

* [https://www.cs.cmu.edu/afs/cs/academic/class/15782-f06/slides/bayesian.pdf](https://www.cs.cmu.edu/afs/cs/academic/class/15782-f06/slides/bayesian.pdf)
* [https://stats.stackexchange.com/a/335422](https://stats.stackexchange.com/a/335422)
