- Posterior $\propto$ prior $\times$ likelihood; recognize the family.
- Beta for Bernoulli counts, Gamma for Poisson counts: the observed counts add to the prior's parameters.
- Point estimates follow from the loss: mean, median, mode.
- The predictive distribution averages the model over the posterior.
the mean of a Gaussian (Sec. 3.4)
judged on that one problem
conjugate priors, exponential family (3.5, 3.6)
Travel times $y_1^n$, independent given $\theta$:
\[\clk{p(y_i\mid\theta)=\frac{1}{\sqrt{2\pi\sigma^2}}\,e^{-(y_i-\theta)^2/(2\sigma^2)}}.\]With no information about $\theta$, take $\cpr{p(\theta)\propto1}$ on $\bbR$.
Since $\int_{-\infty}^{\infty}1\,d\theta=\infty$, this is not a distribution: an improper prior. It is acceptable when the posterior it produces is proper. An improper prior need not be uniform.
Predict: what shape does the posterior have, and where is its centre?
$n=4$, $\bar y=12$, $\sigma^2=4$
Four days of travel times with $\bar y=12$ and $\sigma^2=4$:
\[\cpo{\Theta\mid y_1^4\sim\cN(12,\,1)}.\]Chapter 2: $\cml{\bar Y\sim\cN(\theta^*,\sigma^2/n)}$, the estimator over repeated samples of $n$ days.
Today: $\cpo{\Theta\mid y_1^n\sim\cN(\bar y,\sigma^2/n)}$, the parameter given this one sample.
By iterated expectations, with $Y_{n+1}\independent Y_1^n\mid\Theta$,
\[\E[Y_{n+1}\mid y_1^n]=\E\big[\E[Y_{n+1}\mid\Theta]\,\big|\,y_1^n\big]=\E[\Theta\mid y_1^n]=\bar y.\]The mean of tomorrow's travel time is $\E[Y_{n+1}\mid y_1^n]=\bar y$.
Predict: is tomorrow's travel time more or less spread out than $\Theta$ is?
Condition on $\Theta$ first, then average over the posterior:
\[\begin{aligned} \cpd{\var(Y_{n+1}\mid y_1^n)}&=\E\big[(Y_{n+1}-\bar y)^2\,\big|\,y_1^n\big]\\ &=\E\Big[\E\big[(Y_{n+1}-\bar y)^2\,\big|\,\Theta,y_1^n\big]\,\Big|\,y_1^n\Big]\\ &=\E\big[\sigma^2+(\Theta-\bar y)^2\,\big|\,y_1^n\big]\\ &=\cml{\sigma^2}+\cpo{\var(\Theta\mid y_1^n)}=\cml{\sigma^2}+\cpo{\sigma^2/n}. \end{aligned}\]The third line uses $\E[(Y_{n+1}-\bar y)^2\mid\Theta=\theta]=\sigma^2+(\theta-\bar y)^2$, the variance plus the squared offset of the mean. The predictive distribution is Gaussian, so its mean and variance determine it.
Prior $\Theta\sim\cN(\theta_0,\tau_0^2)$, with mean $\theta_0$ and variance $\tau_0^2$:
\[\begin{aligned} \cpr{\text{Prior:}}\quad &\cpr{p(\theta)\propto\exp\left(-\frac{(\theta-\theta_0)^2}{2\tau_0^2}\right)},\\[2pt] \clk{\text{Likelihood:}}\quad &\clk{p(y_1^n\mid\theta)\propto\exp\left(-\frac{\sum_i(y_i-\theta)^2}{2\sigma^2}\right)}. \end{aligned}\]Predict: what family is the posterior in? Is it narrower or wider than the prior?
prior $\cN(10,1)$; $n=4$, $\bar y=12$, $\sigma^2=4$
Prior $\Theta\sim\cN(\theta_0,\tau_0^2)$ and travel times $Y_i\mid\Theta=\theta\sim\cN(\theta,\sigma^2)$:
\[\begin{aligned} \cpr{\text{Prior:}}\quad &\cpr{p(\theta)\propto\exp\left(-\frac{(\theta-\theta_0)^2}{2\tau_0^2}\right)},\\[4pt] \clk{\text{Likelihood:}}\quad &\clk{p(y_1^n\mid\theta)\propto\exp\left(-\frac{\sum_i(y_i-\theta)^2}{2\sigma^2}\right)},\\[4pt] \cpo{\text{Posterior:}}\quad &\cpo{p(\theta\mid y_1^n)}\propto\clk{\exp\left(-\frac{\sum_i(y_i-\theta)^2}{2\sigma^2}\right)}\times\cpr{\exp\left(-\frac{(\theta-\theta_0)^2}{2\tau_0^2}\right)}. \end{aligned}\]The exponent is a quadratic in $\theta$.
Claim. If $p(x)\propto e^{-f(x)}$ with $f(x)=ax^2-bx+c$ and $a>0$, then $X\sim\cN\!\left(\dfrac{b}{2a},\dfrac{1}{2a}\right)$.
Proof.
\[ax^2-bx+c=\frac{\left(x-\frac{b}{2a}\right)^2}{2\cdot\frac{1}{2a}}+C,\qquad C\text{ free of }x.\]Applied to the posterior. Expanding the two squares in $\theta$,
\[a=\clk{\frac{n}{2\sigma^2}}+\cpr{\frac{1}{2\tau_0^2}},\qquad b=\clk{\frac{n\bar y}{\sigma^2}}+\cpr{\frac{\theta_0}{\tau_0^2}},\] \[\boxed{\cpo{\Theta\mid y_1^n}\sim\cN\!\left(\frac{\clk{\frac{n\bar y}{\sigma^2}}+\cpr{\frac{\theta_0}{\tau_0^2}}}{\clk{\frac{n}{\sigma^2}}+\cpr{\frac{1}{\tau_0^2}}},\;\frac{1}{\clk{\frac{n}{\sigma^2}}+\cpr{\frac{1}{\tau_0^2}}}\right)}\]Write $\Theta\mid y_1^n\sim\cN(\mu_1,\tau_1^2)$. With precision the reciprocal of variance,
\[\cpo{\frac{1}{\tau_1^2}}=\cpr{\frac{1}{\tau_0^2}}+\clk{\frac{n}{\sigma^2}},\qquad\cpo{\mu_1}=\frac{\cpr{\frac{1}{\tau_0^2}\,\theta_0}+\clk{\frac{n}{\sigma^2}\,\bar y}}{\cpr{\frac{1}{\tau_0^2}}+\clk{\frac{n}{\sigma^2}}}.\]The mean is a weighted average of the prior mean and $\bar y$, with weights the prior precision $\cpr{1/\tau_0^2}$ and the data precision $\clk{n/\sigma^2}$: the same shape as the Beta mean, whose weights were $\alpha+\beta$ and $n$.
Predict: travel times with $\sigma^2=4$, prior $\theta_0=10$, $\tau_0^2=1$, and $n=4$ days with $\bar y=12$. Where does $\mu_1$ land?
Data precision $4/4=1$, prior precision $1$, posterior precision $2$: $\Theta\mid y_1^4\sim\cN(11,\,1/2)$.
Same prior $\Theta\sim\cN(10,1)$ and $\sigma^2=4$, same $\bar y=12$, but now $n=16$ days.
Predict: sketch the new posterior next to the four-day one before computing it.
- (a) Find the posterior mean and variance.
- (b) For which $n$ do the prior and the data get equal weight?
- (c) What happens to $\mu_1$ and $\tau_1^2$ as $n\to\infty$?
- (a) Find the posterior mean and variance.
Prior precision $\cpr{1}$, data precision $\clk{4}$: $\cpo{\Theta\mid y_1^{16}\sim\cN(11.6,\,0.2)}$.
Four times the data: the data's weight rises from $1/2$ to $4/5$, and the variance falls from $1/2$ to $1/5$.
- (b) For which $n$ do the prior and the data get equal weight?
- (c) What happens to $\mu_1$ and $\tau_1^2$ as $n\to\infty$?
The prior's precision stays fixed while the data's grows with $n$, so the prior's share of the weight goes to zero.
Take the prior $\Theta\sim\cN(0,\tau_0^2)$. The posterior mean (also its mode and median) and the MLE:
\[\cpo{\hat\theta_B}=\bar y\left(\frac{\tau_0^2}{\tau_0^2+\sigma^2/n}\right),\qquad\cml{\hat\theta_{\mle}}=\bar y.\]$\cpo{\hat\theta_B}$ is $\bar y$ shrunk toward the prior mean $0$ by a factor in $(0,1)$: with $\tau_0^2=1$ and $\sigma^2=4$, $1/2$ at $n=4$ and $4/5$ at $n=16$.
With the prior $\Theta\sim\cN(0,\tau_0^2)$,
\[\cpo{\hat\theta_B}=\bar y\left(\frac{\tau_0^2}{\tau_0^2+\sigma^2/n}\right),\qquad\cml{\hat\theta_{\mle}}=\bar y.\]To compare them, judge both in the frequentist framework: a true value $\theta^*$, and each estimator a function of the random data $Y_1^n$.
Predict: which of the two is unbiased? Which has the smaller variance?
The MLE is unbiased with $\mse(\cml{\hat\theta_{\mle}})=\var(\bar Y)=\sigma^2/n$, and by the Cramér–Rao bound no unbiased estimator does better.
\[\begin{aligned} \text{Bias:}\quad &\bias(\cpo{\hat\theta_B})=\E[\cpo{\hat\theta_B}]-\theta^*=-\theta^*\left(\frac{\sigma^2/n}{\tau_0^2+\sigma^2/n}\right),\\[4pt] \text{Variance:}\quad &\var(\cpo{\hat\theta_B})=\frac{\sigma^2}{n}\left(\frac{\tau_0^2}{\tau_0^2+\sigma^2/n}\right)^2,\\[4pt] \text{MSE:}\quad &\mse(\cpo{\hat\theta_B})=\bias(\cpo{\hat\theta_B})^2+\var(\cpo{\hat\theta_B}). \end{aligned}\]As $\tau_0^2$ grows, the bias term falls and the variance term rises.
Take $\sigma^2/n=1$. Predict: as $\tau_0^2$ runs from $0$ (the prior alone) to $\infty$ (the MLE), how do $\bias^2$ and $\var$ move? Is there a $\tau_0^2$ whose MSE falls below $\sigma^2/n=1$?
below the MLE for every $\tau_0^2>0$;
minimum $0.5$ at $\tau_0^2=1$.
above the MLE for $\tau_0^2<1.5$, below it beyond;
minimum $0.8$ at $\tau_0^2=4$.
For $\sigma^2/n=1$, both minima sit at $\tau_0^2=(\theta^*)^2$. In general, $\tau_0^2=(\theta^*)^2$ gives
\[\begin{aligned} \mse(\cpo{\hat\theta_B})&=\frac{(\theta^*)^2\,\sigma^2/n}{(\theta^*)^2+\sigma^2/n}\\ &<\frac{\sigma^2}{n}=\mse(\cml{\hat\theta_{\mle}}). \end{aligned}\]| Frequentist | Bayesian | |
|---|---|---|
| Parameter | fixed $\theta^*$ | random $\Theta$ |
| What is random | the data, over samples | $\Theta$, given the data |
| Output | a point $\hat\theta(\bx)$ | a posterior $p(\theta\mid\bx)$ |
| Prior | none | required |
| Accuracy | bias, variance, MSE; Cramér–Rao floor | posterior variance |
True or false? Decide with a neighbour before the answers appear.
(a) $\hat\theta_B$ is biased, so it is worse than $\hat\theta_{\mle}$.
(b) Letting $\tau_0\to\infty$ turns $\hat\theta_B$ into $\hat\theta_{\mle}$.
(c) The posterior variance $1/(n/\sigma^2+1/\tau_0^2)$ is smaller than $\sigma^2/n$.
(d) For $\theta^*=0$ the Bayesian estimator is unbiased.
(e) A frequentist can say “the probability that $\theta^*$ lies in $[10.5,11.5]$ is $0.9$.”
| Statement | Answer | Why |
|---|---|---|
| (a) biased, so worse | False | MSE decides; an unbiased estimator can be inferior |
| (b) $\tau_0\to\infty$ gives the MLE | True | the shrinkage factor goes to $1$ |
| (c) posterior variance below $\sigma^2/n$ | True | precisions add; the extra certainty comes from the prior |
| (d) unbiased when $\theta^*=0$ | True | the bias is proportional to $\theta^*$ |
| (e) “$\pr(\theta^*\in[10.5,11.5])=0.9$” | False | $\theta^*$ is fixed; the statement is about the procedure |
Given a likelihood, a conjugate prior is a family of distributions such that the posterior is in the same family.
| Likelihood | Data summary | Prior | Posterior |
|---|---|---|---|
| $\ber(\theta)$ | $y=\sum_iy_i$ | $\bet(\alpha,\beta)$ | $\bet(\alpha+y,\beta+n-y)$ |
| $\poi(\lambda)$ | $S=\sum_iy_i$ | $\gam(\alpha,\beta)$ | $\gam(\alpha+S,\beta+n)$ |
| $\expo(\theta)$ | $y=\sum_iy_i$ | $\gam(\alpha,\beta)$ | $\gam(\alpha+n,\beta+y)$ |
| $\cN(\theta,\sigma^2)$, $\sigma^2$ known | $\bar y$ | $\cN(\theta_0,\tau_0^2)$ | $\cN(\mu_1,\tau_1^2)$ |
Conjugate for one observation implies conjugate for $n$ iid observations: update $n$ times, one observation at a time.
Waiting times between incidents, $Y_i\mid\theta\sim\expo(\theta)$, rate $\theta$ per day; $y=\sum_{i=1}^ny_i$ is the total wait.
The incident rate $\theta$ has prior $\cpr{\gam(2,2)}$, with mean $1$ per day. Then $\clk{n=5}$ waits are observed, totalling $\clk{y=10}$ days:
\[\begin{aligned} \cpr{\text{Prior:}}\quad &\cpr{p(\theta)\propto\theta^{2-1}e^{-2\theta}},\\ \clk{\text{Likelihood:}}\quad &\clk{p(y_1^5\mid\theta)=\theta^{5}e^{-10\theta}},\\ \cpo{\text{Posterior:}}\quad &\cpo{p(\theta\mid y_1^5)\propto\theta^{\cpr{2}+\clk{5}\cpr{-1}}e^{-(\cpr{2}+\clk{10})\theta}}. \end{aligned}\] \[\boxed{\cpo{\Theta\mid y_1^5\sim\gam(7,12)}}\]The posterior mean $7/12=0.58$ per day lies between the prior mean $1$ and the MLE $n/y=0.5$.
A likelihood $p(y\mid\theta)$ is in the exponential family if
\[p(y\mid\theta)=\exp\left(\clk{a(y)}^Tb(\theta)+\clk{f(y)}+g(\theta)\right),\]where $\clk{a},b,y,\theta$ may be vectors and $\clk{f},g$ are scalar; $b(\theta)$ is the natural parameter. Gaussian, Beta, Gamma and Binomial are members.
Your turn: write the Bernoulli pmf $\theta^y(1-\theta)^{1-y}$ in this form. What are $\clk{a}$, $b$, $\clk{f}$, $g$?
Write the Bernoulli pmf $\theta^y(1-\theta)^{1-y}$ as
\[p(y\mid\theta)=\exp\left(\clk{a(y)}^Tb(\theta)+\clk{f(y)}+g(\theta)\right).\]What are $\clk{a}$, $b$, $\clk{f}$, $g$?
so $\clk{a(y)=y}$, $b(\theta)=\ln\dfrac{\theta}{1-\theta}$ (the log odds), $\clk{f(y)=0}$, $g(\theta)=\ln(1-\theta)$.
For $n$ iid observations,
\[\begin{aligned} p(y_1^n\mid\theta)&\propto\exp\left(\sum_{i=1}^n\clk{a(y_i)}^Tb(\theta)+\clk{n}\,g(\theta)\right)\\ &=\exp\left(\clk{t(y_1^n)}^Tb(\theta)+\clk{n}\,g(\theta)\right), \end{aligned}\]with $\clk{t(y_1^n)=\sum_{i=1}^na(y_i)}$, the sufficient statistic. The MLE and the posterior depend on the data only through $\clk{t}$, whose size does not grow with $n$.
The sufficient statistic of $n$ iid observations is $\clk{t(y_1^n)=\sum_{i=1}^na(y_i)}$.
Match the prior to the shape of the likelihood, with two parameters $\cpr{\nu}$ and $\cpr{m}$ in the places of $\clk{t(y_1^n)}$ and $\clk{n}$:
\[\begin{aligned} \clk{\text{Likelihood:}}\quad &\clk{p(y_1^n\mid\theta)}\propto\exp\left(\clk{t(y_1^n)}^Tb(\theta)+\clk{n}\,g(\theta)\right),\\ \cpr{\text{Prior:}}\quad &\cpr{p(\theta)}\propto\exp\left(\cpr{\nu}^Tb(\theta)+\cpr{m}\,g(\theta)\right),\\ \cpo{\text{Posterior:}}\quad &\cpo{p(\theta\mid y_1^n)}\propto\exp\left((\cpr{\nu}+\clk{t(y_1^n)})^Tb(\theta)+(\cpr{m}+\clk{n})\,g(\theta)\right). \end{aligned}\]Bernoulli: $b(\theta)=\ln\dfrac{\theta}{1-\theta}$, $g(\theta)=\ln(1-\theta)$, and $y=\sum_iy_i$.
\[\begin{aligned} \cpr{\text{Prior:}}\quad &\cpr{p(\theta)}\propto\exp\left(\cpr{\nu}\,b(\theta)+\cpr{m}\,g(\theta)\right)=\theta^{\cpr{\nu}}(1-\theta)^{\cpr{m}-\cpr{\nu}}\;\;\text{is }\cpr{\bet(\cpr{\nu}+1,\cpr{m}-\cpr{\nu}+1)},\\[6pt] \cpo{\text{Posterior:}}\quad &\cpo{\Theta\mid y_1^n}\sim\cpo{\bet(\cpr{\nu}+\clk{y}+1,\;\cpr{m}+\clk{n}-\cpr{\nu}-\clk{y}+1)}. \end{aligned}\]The exponents of last time's $\bet(\alpha,\beta)$ are $\cpr{\nu}=\alpha-1$ and $\cpr{m}-\cpr{\nu}=\beta-1$: the prior looks like $\cpr{m}$ earlier observations with statistic $\cpr{\nu}$.
Next: random vectors and the multivariate Gaussian, Chapter 4.