> For the complete documentation index, see [llms.txt](https://theshank.gitbook.io/ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://theshank.gitbook.io/ai/bayesian-deep-learning/probabilistic-view.md).

# Probabilistic View

A neural network can be viewed as probabilistic model $$p(y|x,w)$$ . For classification, $$y$$ is a set of classes and $$p(y|x,w)$$ is a categorical distribution. For regression, $$y$$ is a continuous variable and $$p(y|x,w)$$ is a Gaussian distribution.

Given a training dataset $$D={x^i,y^i}$$ we can construct the likelihood function $$p(D|w)=∏\_ip(y^i, x^i |w)$$ which is a function of parameters $$w$$ . Maximizing the likelihood function gives the maximimum likelihood estimate (MLE) of $$w$$. The usual optimization objective during training is the negative log likelihood. For a categorical distribution this is the *cross entropy* error function, for a Gaussian distribution this is proportional to the *sum of squares* error function. MLE can lead to severe overfitting though.

Multiplying the likelihood with a prior distribution $$p(w)$$ is, by Bayes theorem, proportional to the posterior distribution $$p(w|D)∝p(D|w)p(w)$$ . Maximizing $$p(D|w)p(w)$$ gives the maximum a posteriori (MAP) estimate of $$. Computing the MAP estimate has a regularizing effect and can prevent overfitting. The optimization objectives here are the same as for **MLE plus a regularization term coming from the log prior**.

Both MLE and MAP give point estimates of parameters. If we instead had a full posterior distribution over parameters we could make predictions that take weight uncertainty into account. This is covered by the posterior predictive distribution $$p(y|x,D)=∫p(y|x,w)p(w|D)dw$$ in which the parameters have been marginalized out. This is equivalent to averaging predictions from an ensemble of neural networks weighted by the posterior probabilities of their parameters **w**.