We all know that there are two types of statistics: frequentist and Bayesian. Frequentist approaches treat parameters to be estimated as fixed quantities and the data we observe to come from a data generating process based on these quantities. The Bayesian approach assumes that we don’t want to just estimate the parameters as fixed points but rather believes that these parameters are random values and one needs to estimate a distribution.
As the name indicates, the Bayesian approach relies on Bayes Rule and tries to estimate the following:
p(θ|y) = p(y|θ)p(θ)/p(y)
Since the denominator does not depend on θ, one often calculates the posterior distribution as:
p(θ|y) ∝ L(θ;y)π(θ)/p(y)
and if we take the log-scale we get:
ln{p(θ|y)} = ;(θ;y)+ln{π(θ)} – c
where l() is the log likelihood, π(θ is the prior, and c is a constant.
The canonical Bayesian example is that of a coin flip. Our prior distribution for a coin flip is typically distributed from the Beta(α,β) distribution. The actual coin number of flips observed is modelled based on the Binomial(n,θ) where n is the number of coin flips and the parameter of interest θ is the true probability of getting a heads. This formulation is often used since the posterior can be calculated with a closed-form solution as Beta(y+α,n-y+β), where y is the number of heads that appear in the data.
If we used informative priors to set the Beta(α,β) distribution so it would be centered around .5 (i.e. α=30 and β=30), then the posterior would still indicate that the true heads probability distribution is close to 0.5; if we use an uninformed prior (i.e. α=1 and β=1) then the likelihood function and the posterior distribution are equivalent. In other words, data rules when priors are uninformative.
For more information, see the following Stata Bayesian analysis documentation as well as the following two videos. The videos are nice as they not only tell you the logic behind Bayesian analysis but also how to run programs in Stata to answer these questions with real data using commands such as the bayesmh approach.