Bayesian Deep Learning

Using Bayesian inference in deep learning

Bayesian Neural Network

The idea behind Bayesian Neural Network is to model the network's wights $W$as a distribution $p(W|D)$conditioned on the training data $D$, instead of a deterministic point estimates. By placing a prior over the weights e.g. $W \sim N(0,I)$, the network training can be interpreted as determining a posterior over the weights given the training data: $p(W|T)$. However, evaluating this posterior is not tractable without approximation techniques.

Bayesian Inference

Priors in weights in NNs

We can apply this process to neural networks and come up with the probability distribution over the network weights, $w$ , given the training data, $p ( w|D )$ .

259KB

bayesiandeeplearning.pdf

pdf

Bayesian Learning of weights in NN

Take away

When you add the "Regularization or Weight Decay", it simply assumes that my weights follows zero centered gaussian distribution. Hence, it is a way to incorporate a prior.

Resources

Neurips 2019 talk - See at 19:00, interesting

• https://www.cs.cmu.edu/afs/cs/academic/class/15782-f06/slides/bayesian.pdf
• https://stats.stackexchange.com/a/335422