3 Bayesian Parameter Estimation

3.1 From Prior to Posterior

In the Bayesian philosophy, unknown parameters are viewed as random. So, our knowledge about the parameter can be encoded as a distribution. The distribution representing our belief before observing the data is called the prior distribution. After we observe the data, our belief changes, resulting in the posterior distribution.

Specifically, the components of a Bayesian estimation problem are:

• Data \({{\boldsymbol{x}}}\): The data is a realization of a random variable \({{\boldsymbol{X}}}\). The distribution of \({{\boldsymbol{X}}}\) depends on a parameter \(\Theta\).

• Parameter \(\Theta\): The parameter of the distribution of \({{\boldsymbol{X}}}\), which is unknown, and hence a random variable in the Bayesian framework.

• Joint and marginal distributions \(p\): A joint distribution \(p_{{{\boldsymbol{X}}},\Theta}\) and its marginals \(p_{{{\boldsymbol{X}}}}\) and \(p_\Theta\).

The steps of Bayesian estimation of a parameter \(\theta\) are:

1. Identifying the prior distribution, \(p_\Theta(\theta)\). This is called the prior because it encodes our beliefs about \(\Theta\) before seeing any data.

2. Collecting data \({{\boldsymbol{x}}}\) and forming the likelihood: \(p_{{{\boldsymbol{X}}}|\Theta}({{\boldsymbol{x}}}|\theta)\)

3. Finding the posterior distribution \(p_{\Theta|{{\boldsymbol{X}}}}(\theta|{{\boldsymbol{x}}})\) as \[\begin{equation} \begin{split} p_{\Theta|{{\boldsymbol{X}}}}(\theta|{{\boldsymbol{x}}}) & = \frac{p_\Theta(\theta)p_{{{\boldsymbol{X}}}|\Theta}({{\boldsymbol{x}}}|\theta)}{p_{{\boldsymbol{X}}}({{\boldsymbol{x}}})}, \end{split} \end{equation}\] The distribution \(p_{\Theta|{{\boldsymbol{X}}}}\) is called the posterior distribution since it encodes our knowledge about the parameter after observing the data. Usually, since the distribution is clear from the argument, we drop the subscripts of \(p\), writing the above equation as \[\begin{equation} \begin{split} p(\theta|{{\boldsymbol{x}}}) & = \frac{p(\theta)p({{\boldsymbol{x}}}|\theta)}{p({{\boldsymbol{x}}})}, \end{split} \end{equation}\]

Normalizing distributions. Finding the posterior distribution requires computing the integral \(p({{\boldsymbol{x}}}) = \int_{\theta} p(\theta)p({{\boldsymbol{x}}}|\theta)d\theta\). Since we have to compute an integral anyway, we might as well drop all multiplicative terms that are constant in \(\theta\) and then normalize the final distribution. In particular, \(p({{\boldsymbol{x}}})\) is one such term. So we often first find a function proportional to \(p(\theta|{{\boldsymbol{x}}})\) as \[\begin{equation} p(\theta|{{\boldsymbol{x}}})\propto p(\theta)p({{\boldsymbol{x}}}|\theta), \end{equation}\] where we can also drop constant terms in \(\theta\) from \(p(\theta)\) and \(p({{\boldsymbol{x}}}|\theta)\). We can then normalize the result by integration. This is often difficult to do. Sometimes, given this function, we can identify the distribution. More generally, we can use computational methods, such as Markov Chain Monte Carlo, or approximation methods, such as variational inference, as we will see later. Finally, in certain cases, we can find what we need without any integration. For example, if our goal is to find the value of \(\theta\) maximizing \(p(\theta|{{\boldsymbol{x}}})\).

Note that since we are interested in \(\Theta\), we can drop multiplicative terms that are constant with respect to \(\theta\), such as \(\binom{n}{s}\), in the above example.

Now that we have the posterior distribution, we can answer questions about the parameter, for example, What is the probability that \(0.4 < \Theta < 0.6\)? \[\begin{equation} \begin{split} \int_{0.4}^{0.6}p(\theta|{{\boldsymbol{x}}})d\theta \end{split} \end{equation}\]

3.2 Bayesian Point Estimates

Having the complete distribution for \(p_{\Theta|{{\boldsymbol{X}}}}(\theta|{{\boldsymbol{x}}})\) is useful since it provides the probability for different values for \(\theta\). But sometimes we want to estimate \(\Theta\) with a single value \(\hat\theta=\hat\theta({{\boldsymbol{x}}})\) as a function of the data, similar to maximum likelihood. It is very important to note that in the Bayesian framework, the data is given and the estimate is known (and not random). The best choice for \(\hat\theta\) then depends on how we characterize the estimation error:

In the table, \(I(condition)\) is 1 if the condition is satisfied and is 0 otherwise.

We prove the first case in the table. Let \(\bar\theta =\mathop{\mathrm{{\mathbb{E}}}}[\Theta|{{\boldsymbol{x}}}]\). We have \[\begin{align} \mathop{\mathrm{{\mathbb{E}}}}[(\hat\theta - \Theta)^2 | {{\boldsymbol{x}}}] &= \mathop{\mathrm{{\mathbb{E}}}}[((\hat\theta - \bar\theta) - (\Theta-\bar\theta ))^2 | {{\boldsymbol{x}}}] \\ &= \mathop{\mathrm{{\mathbb{E}}}}[(\hat\theta - \bar\theta)^2 - 2(\hat\theta - \bar\theta)(\Theta-\bar\theta) + (\Theta-\bar\theta)^2 | {{\boldsymbol{x}}}] \\ &= (\hat\theta - \bar\theta)^2 - 2(\hat\theta - \bar\theta)\mathop{\mathrm{{\mathbb{E}}}}[(\Theta - \bar\theta)|{{\boldsymbol{x}}}] + \mathop{\mathrm{{\mathbb{E}}}}[(\Theta-\bar\theta)^2 | {{\boldsymbol{x}}}] \\ &= (\hat\theta - \bar\theta)^2 + \mathop{\mathrm{{\mathbb{E}}}}[(\Theta-\bar\theta)^2 | {{\boldsymbol{x}}}] \\ &= (\hat\theta - \bar\theta)^2+ \mathop{\mathrm{Var}}(\Theta | {{\boldsymbol{x}}}) \\ &\ge \mathop{\mathrm{Var}}(\Theta|{{\boldsymbol{x}}}), \end{align}\] and the lower bound on the error is achieved when \(\hat\theta=\bar\theta\).

3.3 Posterior Predictive Distribution

Given \(n\) iid samples, \(y_1^n=(y_1,\dotsc,y_n)\), we are often interested in the distribution of the next (unobserved) value, \(p_{Y_{n+1}|Y_1^n}(y_{n+1}|y_1^n)\). This distribution is referred to as predictive posterior. We have \[\begin{align} p(y_{n+1}|y_1^n) &= \int p(y_{n+1},\theta|y_1^n)d\theta\\ &= \int p(\theta|y_1^n) p(y_{n+1}|\theta,y_1^n) d\theta\\ &= \int p(\theta|y_1^n) p(y_{n+1}|\theta) d\theta, \end{align}\] where we have used the fact that \(Y_{n+1}{\mathrel{\perp\mkern-10mu\perp}}Y_1^n|\Theta\). We have thus written the predictive posterior in terms of two known distributions.

We may also ask about the expected value of \(Y_{n+1}\) given \(y_1^n\), i.e., \(\mathop{\mathrm{{\mathbb{E}}}}[Y_{n+1}|y_1^n]\). We can find this by first finding \(p(y_{n+1}|y_1^n)\) explicitly. But it is often easier to use the law of iterated expectations, since \(y_1^n\) influences \(Y_{n+1}\) through \(\Theta\). Recall that \[\begin{equation} \mathop{\mathrm{{\mathbb{E}}}}[\mathop{\mathrm{{\mathbb{E}}}}[Y|X]]=\mathop{\mathrm{{\mathbb{E}}}}[Y],\qquad \mathop{\mathrm{{\mathbb{E}}}}[\mathop{\mathrm{{\mathbb{E}}}}[Y|X,Z=z]|Z=z]=\mathop{\mathrm{{\mathbb{E}}}}[Y|Z=z]. \end{equation}\] Hence, \[\begin{equation} \mathop{\mathrm{{\mathbb{E}}}}[Y_{n+1}|y_1^n] = \mathop{\mathrm{{\mathbb{E}}}}[\mathop{\mathrm{{\mathbb{E}}}}[Y_{n+1}|\Theta,y_1^n]|y_1^n] =\mathop{\mathrm{{\mathbb{E}}}}[\mathop{\mathrm{{\mathbb{E}}}}[Y_{n+1}|\Theta]|y_1^n], \end{equation}\tag{3.29}\] where the last step follows from the fact that \(Y_{n+1}{\mathrel{\perp\mkern-10mu\perp}}Y_1^n|\Theta\), implying that \(\mathop{\mathrm{{\mathbb{E}}}}[Y_{n+1}|\Theta,y_1^n]=\mathop{\mathrm{{\mathbb{E}}}}[Y_{n+1}|\Theta]\).

3.4 Gaussian Prior and Likelihood

Suppose that we want to estimate the mean of a Gaussian distribution with known variance, \[\begin{equation} p(y_i|\theta)=\frac{1}{\sqrt{2\pi\sigma^2}}e^{-\frac{(y_i-\theta)^2}{2\sigma^2}} \end{equation}\tag{3.30}\] given iid data \(\{y_1,\dotsc,y_n\}\).

Improper priors. Assuming that we have no information about this mean, it makes sense to choose the prior \[\begin{equation} p(\theta)\propto 1. \end{equation}\] But since the integral \(\int_{-\infty}^\infty 1 d\theta=\infty\), this does not lead to a valid distribution. Nevertheless, such a choice is acceptable, if the posterior is a valid distribution. Such priors are called improper priors. An improper prior does not necessarily have to be uniform.

For the posterior predictive variance, we have \[\begin{equation} \mathop{\mathrm{Var}}(Y_{n+1}|y_1^n) = \sigma^2+\sigma^2/n. \end{equation}\]

It can be shown that \(Y_{n+1}|y_1^n\) has a Gaussian distribution, and as we know its conditional mean and variance, we have \[\begin{equation} Y_{n+1}|y_1^n \sim {\mathcal{N}}(\bar y,\sigma^2+\sigma^2/n) \end{equation}\] From the variance, we can see that there are two sources of uncertainty. One is the inherent randomness in \(Y\), quantified by \(\sigma^2\) and the other is the result of the uncertainty of our estimate of the mean, quantified by \(\sigma^2/n\).

We now consider the same problem with a proper Gaussian prior. Note that below as \(\tau_0\to\infty\), the proper prior below tends to the improper prior \(p(\theta)\propto 1.\)

3.5 Conjugate Priors

Given a likelihood function, a conjugate prior is a family of distributions such that the posterior is in the same family. The posterior is proportional to prior times likelihood, \[\begin{equation} p(\theta|y_1^n)\propto p(\theta)\,p(y_1^n|\theta), \end{equation}\] so when this product, as a function of \(\theta\), has the form of a member of the family, the posterior is that member. The family then also gives the normalizing constant, and the integral \(\int p(\theta)p(y_1^n|\theta)\,d\theta\) never has to be computed. Conjugate priors also often let us read the parameters of an informative prior as pseudo-observations added to a flat, possibly improper, prior. We have already met three conjugate pairs: the Beta prior for Bernoulli trials (Example 3.3), the Gamma prior for Poisson counts (Example 3.6), and the Gaussian prior for a Gaussian mean with known variance (3.52).

In each pair below, the posterior’s parameters combine the prior’s parameters with a summary of the data.

• Bernoulli/Beta: \(\theta\) is the probability of success, and \(y_i=1\) marks a success and \(y_i=0\) a failure, so \(y=\sum_{i=1}^n y_i\) is the number of successes and \(n-y\) the number of failures. \[\begin{align} p(y_i|\theta)&=\theta^{y_i}(1-\theta)^{1-y_i}&\mathop{\mathrm{\mathrm{Ber}}}(\theta)\\ p(y_1^n|\theta)&=\theta^y(1-\theta)^{n-y}&&\\ p(\theta)&\propto \theta^{\alpha-1}(1-\theta)^{\beta-1}&\mathop{\mathrm{\mathrm{Beta}}}(\alpha,\beta) \\ p(\theta|y_1^n)&\propto \theta^{y+\alpha-1}(1-\theta)^{n-y+\beta-1}&\mathop{\mathrm{\mathrm{Beta}}}(y+\alpha,n-y+\beta) \end{align}\] A \(\mathop{\mathrm{\mathrm{Beta}}}(\alpha,\beta)\) prior is therefore equivalent to starting from the uniform prior \(\mathop{\mathrm{\mathrm{Beta}}}(1,1)\) and having observed \(\alpha-1\) successes and \(\beta-1\) failures.

• Poisson/Gamma: \(y_i\) is the number of incidents at a fire station on day \(i\) and \(\theta\) the average number of incidents per day, so \(y=\sum_{i=1}^n y_i\) is the total number of incidents in \(n\) days. \[\begin{align} p(y_i|\theta)&=\frac{\theta^{y_i}e^{-\theta}}{y_i!}&\mathop{\mathrm{\mathrm{Poi}}}(\theta)\\ p(y_1^n|\theta)&\propto\theta^y \exp(-n\theta)&\\ p(\theta)&=\frac{\beta^\alpha}{\Gamma(\alpha)} \theta^{\alpha-1}\exp(-\beta\theta)&\mathop{\mathrm{\mathrm{Gamma}}}(\alpha,\beta)\\ p(\theta|y_1^n)&\propto \theta^{y+\alpha-1}\exp(-(n+\beta)\theta)&\mathop{\mathrm{\mathrm{Gamma}}}(y+\alpha,n+\beta) \end{align}\] A \(\mathop{\mathrm{\mathrm{Gamma}}}(\alpha,\beta)\) prior is therefore equivalent to starting from the flat improper prior \(\mathop{\mathrm{\mathrm{Gamma}}}(1,0)\) and having observed \(\alpha-1\) incidents in \(\beta\) days. In the Gamma density \(\mathop{\mathrm{\mathrm{Gamma}}}(\alpha,\beta)\), \(\alpha\) is the shape and \(\beta\) the rate, and the mean is \(\alpha/\beta\).

• Exponential/Gamma: \(y_i\) is the time, in days, from the \((i-1)\)th to the \(i\)th incident at the same fire station and \(\theta\) the average number of incidents per day, so \(y=\sum_{i=1}^n y_i\) is the time taken by \(n\) incidents. \[\begin{align} p(y_i|\theta)&=\theta \exp(-\theta y_i)&\mathop{\mathrm{\mathrm{Exp}}}(\theta)=\mathop{\mathrm{\mathrm{Gamma}}}(1,\theta)\\ p(y_1^n|\theta)&=\theta^n \exp(-\theta y)&\\ p(\theta)&=\frac{\beta^\alpha}{\Gamma(\alpha)} \theta^{\alpha-1}\exp(-\beta\theta)&\mathop{\mathrm{\mathrm{Gamma}}}(\alpha,\beta)\\ p(\theta|y_1^n)&\propto \theta^{n+\alpha-1}\exp(-(y+\beta)\theta)&\mathop{\mathrm{\mathrm{Gamma}}}(n+\alpha,y+\beta) \end{align}\] A \(\mathop{\mathrm{\mathrm{Gamma}}}(\alpha,\beta)\) prior is therefore equivalent to starting from the flat improper prior \(\mathop{\mathrm{\mathrm{Gamma}}}(1,0)\) and having observed \(\alpha-1\) incidents in \(\beta\) days.

• Gaussian/Gaussian: \(y_i\) is the \(i\)th of \(n\) measurements of an unknown quantity \(\theta\), each with noise of known variance \(\sigma^2\), and \(\bar y=\frac1n\sum_{i=1}^n y_i\) is their average. \[\begin{align} p(y_i|\theta)&\propto \exp\left(-\frac{(y_i-\theta)^2}{2\sigma^2}\right)&{\mathcal{N}}(\theta,\sigma^2)\\ p(y_1^n|\theta)&\propto \exp\left(-\frac{\sum_{i=1}^n(y_i-\theta)^2}{2\sigma^2}\right)&\\ p(\theta)&\propto \exp\left(-\frac{(\theta-\theta_0)^2}{2\tau_0^2}\right)&{\mathcal{N}}(\theta_0,\tau_0^2)\\ p(\theta|y_1^n)&\propto \exp\left(-\frac{(\theta-\theta_1)^2}{2\tau_1^2}\right)&{\mathcal{N}}(\theta_1,\tau_1^2), \end{align}\] where \[\begin{align} \theta_1 &=\frac{\frac{1}{\tau_0^2}\theta_0+\frac{1}{\sigma^2/n}\bar y}{\frac{1}{\tau_0^2}+\frac{1}{\sigma^2/n}},\\ \frac{1}{\tau_1^2} &= \frac{1}{\tau_0^2}+\frac{1}{\sigma^2/n}. \end{align}\] The precisions add, and the posterior mean is the precision-weighted average of the prior mean and \(\bar y\). A \({\mathcal{N}}(\theta_0,\tau_0^2)\) prior is therefore equivalent to starting from the flat improper prior \(p(\theta)\propto 1\) and having observed \(\sigma^2/\tau_0^2\) measurements with average \(\theta_0\).

If a prior is conjugate for the likelihood of a single observation, it is also conjugate for the likelihood of many iid observations. Updating with \(n\) iid observations is equivalent to updating \(n\) times with one observation at a time: the posterior after \(y_1\) is in the family, so it serves as the prior for \(y_2\), and so on.

3.6 The Exponential Family (EF)

For a random variable \({{\boldsymbol{Y}}}\) with parameter \({\boldsymbol{\Theta}}\), \(p({{\boldsymbol{y}}}|{\boldsymbol{\theta}})\) is said to be from the exponential family if it has the form \[\begin{equation} p({{\boldsymbol{y}}}|{\boldsymbol{\theta}}) = \exp\left(\langle{\textcolor{forestgreen}{{{\boldsymbol{a}}}({{\boldsymbol{y}}})}},{{\boldsymbol{b}}}({\boldsymbol{\theta}})\rangle+f({{\boldsymbol{y}}})+g({\boldsymbol{\theta}})\right), \end{equation}\tag{3.84}\] where \({{\boldsymbol{a}}},{{\boldsymbol{b}}},{{\boldsymbol{y}}},{\boldsymbol{\theta}}\) can be vectors (in bold) and \(f,g\) are scalar functions. The function \({{\boldsymbol{b}}}({\boldsymbol{\theta}})\) is the natural parameter. The data and the parameter meet only through the inner product \(\langle{\textcolor{forestgreen}{{{\boldsymbol{a}}}({{\boldsymbol{y}}})}},{{\boldsymbol{b}}}({\boldsymbol{\theta}})\rangle\): \(f\) depends on \({{\boldsymbol{y}}}\) alone and \(g\) on \({\boldsymbol{\theta}}\) alone. Below, quantities computed from the data are in green, the prior’s parameters in blue, and the posterior in red.

The exponential family includes many common distributions, among them the Bernoulli, binomial (with a known number of trials), Poisson, exponential, Gaussian, Beta, and Gamma. For likelihoods in this family, the data can be summarized in a statistic of fixed size, and the conjugate prior can be written down directly.

Sufficient statistics. Suppose that we have \(n\) iid observations. The likelihood is a product, and a product of exponentials is the exponential of the sum: \[\begin{align} p({{\boldsymbol{y}}}_1^n|{\boldsymbol{\theta}}) &= \prod_{i=1}^n p({{\boldsymbol{y}}}_i|{\boldsymbol{\theta}})=\exp\left(\Big\langle{\textcolor{forestgreen}{\sum_{i=1}^n {{\boldsymbol{a}}}({{\boldsymbol{y}}}_i)}},{{\boldsymbol{b}}}({\boldsymbol{\theta}})\Big\rangle+\sum_{i=1}^n f({{\boldsymbol{y}}}_i)+{\textcolor{forestgreen}{n}}g({\boldsymbol{\theta}})\right)\\ &\propto \exp\left(\langle{\textcolor{forestgreen}{{{\boldsymbol{t}}}({{\boldsymbol{y}}}_1^n)}},{{\boldsymbol{b}}}({\boldsymbol{\theta}})\rangle+{\textcolor{forestgreen}{n}}g({\boldsymbol{\theta}})\right), \end{align}\] where \({\textcolor{forestgreen}{{{\boldsymbol{t}}}({{\boldsymbol{y}}}_1^n)}}={\textcolor{forestgreen}{\sum_{i=1}^n {{\boldsymbol{a}}}({{\boldsymbol{y}}}_i)}}\) is the sufficient statistic. The factor \(\exp\left(\sum_i f({{\boldsymbol{y}}}_i)\right)\) does not involve \({\boldsymbol{\theta}}\), so it is absorbed into the proportionality constant. The maximum-likelihood estimate maximizes the likelihood over \({\boldsymbol{\theta}}\), and the posterior multiplies the likelihood by the prior and normalizes over \({\boldsymbol{\theta}}\). Both use the likelihood only as a function of \({\boldsymbol{\theta}}\), so a factor free of \({\boldsymbol{\theta}}\) changes neither, and both depend on the data only through \({\textcolor{forestgreen}{{{\boldsymbol{t}}}({{\boldsymbol{y}}}_1^n)}}\). Within the model, the rest of the information in \({{\boldsymbol{y}}}_1^n\) is irrelevant for estimating \({\boldsymbol{\theta}}\). The size of \({\textcolor{forestgreen}{{{\boldsymbol{t}}}({{\boldsymbol{y}}}_1^n)}}\) does not grow with \(n\).

For the exponential distribution, \({\textcolor{forestgreen}{a(y)=y}}\) (Example 3.16), so \({\textcolor{forestgreen}{t(y_1^n)=\sum_{i=1}^ny_i}}\), the total waiting time. This is why the posterior in Example 3.15 depends on the data only through \(n\) and \(y\).

Bayesian estimation with conjugate priors. The conjugate prior of an exponential family has the form of the likelihood in \(\theta\), with two parameters \({\textcolor{royalblue}{{\boldsymbol{\nu}}}}\) and \({\textcolor{royalblue}{m}}\) in the places of \({\textcolor{forestgreen}{{{\boldsymbol{t}}}({{\boldsymbol{y}}}_1^n)}}\) and \({\textcolor{forestgreen}{n}}\): \[\begin{align} p({{\boldsymbol{y}}}_i|{\boldsymbol{\theta}})&\propto \exp\left(\langle{\textcolor{forestgreen}{{{\boldsymbol{a}}}({{\boldsymbol{y}}}_i)}},{{\boldsymbol{b}}}({\boldsymbol{\theta}})\rangle+g({\boldsymbol{\theta}})\right)\\ p({{\boldsymbol{y}}}_1^n|{\boldsymbol{\theta}})&\propto \exp\left(\langle{\textcolor{forestgreen}{{{\boldsymbol{t}}}({{\boldsymbol{y}}}_1^n)}},{{\boldsymbol{b}}}({\boldsymbol{\theta}})\rangle+{\textcolor{forestgreen}{n}}g({\boldsymbol{\theta}})\right)&\\ p({\boldsymbol{\theta}})&\propto \exp\left(\langle{\textcolor{royalblue}{{\boldsymbol{\nu}}}},{{\boldsymbol{b}}}({\boldsymbol{\theta}})\rangle+{\textcolor{royalblue}{m}}g({\boldsymbol{\theta}})\right)&{\textcolor{royalblue}{Dist({\boldsymbol{\nu}},m)}}\\ p({\boldsymbol{\theta}}|{{\boldsymbol{y}}}_1^n)& \propto \exp\left(\langle{\textcolor{royalblue}{{\boldsymbol{\nu}}}}+{\textcolor{forestgreen}{{{\boldsymbol{t}}}({{\boldsymbol{y}}}_1^n)}},{{\boldsymbol{b}}}({\boldsymbol{\theta}})\rangle+({\textcolor{royalblue}{m}}+{\textcolor{forestgreen}{n}})g({\boldsymbol{\theta}})\right)&{\textcolor{firebrick}{Dist({\boldsymbol{\nu}}+{{\boldsymbol{t}}}({{\boldsymbol{y}}}_1^n),m+n)}}, \end{align}\] where \(Dist\) refers to a specific type of distribution, determined by \({{\boldsymbol{b}}}\) and \(g\). Multiplying two exponentials of this form adds their coefficients of \({{\boldsymbol{b}}}({\boldsymbol{\theta}})\) and of \(g({\boldsymbol{\theta}})\), so the posterior has the same form as the prior, and the prior is conjugate. The posterior adds the sufficient statistic \({\textcolor{forestgreen}{{{\boldsymbol{t}}}({{\boldsymbol{y}}}_1^n)}}\) to \({\textcolor{royalblue}{{\boldsymbol{\nu}}}}\) and the sample size \({\textcolor{forestgreen}{n}}\) to \({\textcolor{royalblue}{m}}\).

Pseudo-observations. The parameters in conjugate priors can be interpreted as representing pseudo-observations by comparing the forms of \(p({{\boldsymbol{y}}}_1^n|{\boldsymbol{\theta}})\) and \(p({\boldsymbol{\theta}})\). In particular, \({\textcolor{royalblue}{{\boldsymbol{\nu}}}}\) plays the same role as \({\textcolor{forestgreen}{{{\boldsymbol{t}}}({{\boldsymbol{y}}}_1^n)}}\) and \({\textcolor{royalblue}{m}}\) the same role as \({\textcolor{forestgreen}{n}}\): the prior acts like \({\textcolor{royalblue}{m}}\) earlier observations whose sufficient statistic is \({\textcolor{royalblue}{{\boldsymbol{\nu}}}}\).