TIME: 2.5 min. ASK: What value would you report at $x_\star$? How did you use all the points? PAUSE: Take two or three answers. SAY: We are already choosing a function, even if we have not named it. TRANSITION: Let us name the objects in that choice.
Regression learns a function
\[\cD=\{(\bx_i,y_i)\}_{i=1}^{n},\qquad \bx_i\in\bbR^d,\quad y_i\in\bbR.\]
Choose $f_{\btheta}:\bbR^d\to\bbR$ so that
\[\widehat y_\star=f_{\btheta}(\bx_\star)\]
predicts a new response.
features $\bx$
model $f_{\btheta}$
prediction $\widehat y$
Learning means choosing parameters $\btheta$ from data.
TIME: 1.5 min. ASK: Which objects are observed, and which one must be learned? REVEAL: Data, prediction rule, then parameters. TRANSITION: We need a score for deciding which parameters are good.
Learning as empirical risk minimization
data $(\bx_i,y_i)$
predictions $f_{\btheta}(\bx_i)$
loss $\ell(y_i,f_{\btheta}(\bx_i))$
risk
Prediction error on future data:
\[R(\btheta)=\E\!\left[\ell\!\left(Y,f_{\btheta}(X)\right)\right].\]
We would minimize $R(\btheta)$, but the true data distribution is unknown.
Training data give a computable proxy:
\[\widehat R(\btheta)=\frac1n\sum_{i=1}^n\ell\!\left(y_i,f_{\btheta}(\bx_i)\right)\]
TIME: 2.5 min. ASK: What prediction error would we minimize if we knew the population? PAUSE: Ask why $R(\btheta)$ is unavailable. REVEAL: Population risk, the unknown distribution, then empirical risk as a computable proxy. SAY: Training loss stands in for prediction error on the underlying distribution. TRANSITION: ERM still allows far too many functions.
Many functions can fit the same data
Which function should we trust at a new $x$?
hypothesis class $\cH$ $+$ loss
TIME: 1.5 min. ASK: Which curve is best? What information is missing? REVEAL: Candidate functions one at a time. SAY: Data alone do not answer this. We choose a hypothesis class $\cH$ and a loss. This restriction is an inductive bias. TRANSITION: Our first restriction is linearity in the parameters.
The intercept is just the weight on an always-on feature.
\[\bx=(1,x_1,\ldots,x_d)^\top.\]
“Linear” describes the parameters, not necessarily the raw input.
TIME: 1.5 min. ASK: Where did the intercept go? REVEAL: The augmented feature $x_0=1$. SAY: This notation scales directly to many features. TRANSITION: Check what linear in parameters really permits.
Poll: linear in the parameters?
Model
Linear in $\btheta$?
A
$\theta_0+\theta_1x$
Yes
B
$\theta_0+\theta_1x+\theta_2x^2$
Yes
C
$\theta_0+\theta_1\sin x$
Yes
D
$\theta_0+\exp(\theta_1x)$
No
Vote before the answer appears.
Test: can we write $f_{\btheta}(x)=\btheta^\top\boldsymbol\phi(x)$ with fixed features $\boldsymbol\phi(x)$?
A curved prediction can still be linear in $\btheta$.
TIME: 3 min. ASK: Vote A through D. Which models are linear regression? PAUSE: Ask one student to justify B and another to justify D. REVEAL: A, B, C yes. D no. TRANSITION: With the model class fixed, we still need to choose one line.
Three lines enter. Which one wins?
A, B, or C?
Measure every vertical miss, then combine them.TIME: 1.5 min. ASK: Which line looks best? What numerical criterion could settle the vote? PAUSE: Let students propose distance summaries. REVEAL: The residual-based ruler. TRANSITION: First define one miss precisely.
Residuals are the vertical misses
What should count as an error?
Each vertical segment is a residual.
$e_i>0$: point above line $e_i<0$: point below line
But positive and negative misses can cancel.
TIME: 1.5 min. ASK: Why not simply add the residuals? PAUSE: Wait for “cancellation.” REVEAL: Residual segments, then sign convention. TRANSITION: Squaring prevents cancellation and defines our objective.
large residuals receive disproportionately more weight
Multiplying by $1/n$ would not change the minimizer.
TIME: 2 min. ASK: What properties does squaring buy us immediately? REVEAL: Objective, three consequences, then the quadratic growth. SAY: Squaring prevents cancellation, but it also gives large misses extra influence. TRANSITION: Now try to beat the optimum by eye.
Demo 1: can you beat least squares?
interceptslope
What would you change?
Better. Can we do more?
One last adjustment?
Least-squares optimum
TIME: 4 min. ASK: On each state, should slope rise or fall? Should the intercept rise or fall? PAUSE: Take a show of hands before advancing. REVEAL: Poor line, two improvements, then the optimum and SSE sequence. SAY: Visual tuning got close, but optimization certifies the winner. TRANSITION: Matrix notation lets us derive that certificate.
outlier heightTIME: 3 min. ASK: Predict how the fitted line will move as one point rises. PAUSE: Before each state, ask how the squared contribution changes. REVEAL: Residuals 2, 5, 10 and losses 4, 25, 100. SAY: Squaring gives smooth optimization, but it amplifies extreme residuals. TRANSITION: Stack all n equations into one matrix equation.
TIME: 2.5 min. ASK: What must the dimensions of $X\btheta$ be? REVEAL: $X$ maps parameter space into label data space, then show the residual and norm form. SAY: $X\btheta$ lies in $\bbR^n$, and soon we will restrict it further to $\operatorname{col}(X)$. TRANSITION: Use a tiny system to make every object concrete.
TIME: 4 min. ASK: Compute all three quantities with a neighbor. PAUSE: Give two minutes. Ask for the prediction vector first. REVEAL: Full calculation. SAY: Keep this $X$ and $y$. We will return to them twice. TRANSITION: Before optimizing, ask what predictions $X\btheta$ can ever produce.
As $\btheta$ varies, $X\btheta$ ranges over
\[\operatorname{col}(X)=\operatorname{span}\{\boldsymbol x^{(1)},\boldsymbol x^{(2)}\}\subseteq\bbR^3.\]
Usually, $\by$ is not in that subspace.
TIME: 2.5 min. ASK: Why can $X\btheta$ not reach every point in $\bbR^3$ here? REVEAL: Linear combination, column space, then $y$ outside it. SAY: Regression is now a nearest-point problem. TRANSITION: The nearest point has a familiar geometric name.
Least squares is an orthogonal projection
Which reachable prediction is closest to $\by$?
The least-squares prediction is the projection of $\by$.
The leftover residual is perpendicular to the whole subspace.
TIME: 2.5 min. ASK: On the first state, where would you place the closest point? REVEAL: Projection, then residual and right angle. SAY: The geometry gives the optimality condition without calculus. TRANSITION: Translate “perpendicular to the column space” into algebra.
What must be orthogonal to what?
At the optimum, the residual $\boldsymbol e$ is perpendicular to
the response $\by$
the parameter vector $\widehat{\btheta}$
every column of $X$
every row of $X$
C. Every column of $X$
\[(\boldsymbol x^{(j)})^\top\boldsymbol e=0\quad\text{for every }j\]
\[\boxed{X^\top\boldsymbol e=\boldsymbol 0}\]
TIME: 2 min. ASK: Vote A through D. PAUSE: Ask a student to connect the answer to $\operatorname{col}(X)$. REVEAL: C and the compact matrix equation. TRANSITION: Substitute the residual definition and solve.
Normal means perpendicular: these equations encode the projection condition.
TIME: 3 min. ASK: Predict the next algebraic line before each reveal. REVEAL: Substitute, expand, rearrange. SAY: These are not a separate trick. They are the geometry written in symbols. TRANSITION: Under a rank condition, the equations have one closed-form solution.
$X$ must have full column rank, so $X^\top X$ is invertible.The unique parameter vector whose prediction $X\widehat{\btheta}$ is closest to $\by$.
In numerical software, solve the system or use a factorization. Do not explicitly form an inverse.
TIME: 1.5 min. ASK: What assumption did we use when multiplying by $(X^\top X)^{-1}$? SAY: Full column rank gives a unique parameter vector. Stable software uses QR or related solvers. TRANSITION: Return to our three-point example and close the loop.
Your turn: solve least squares
Work with a partner for about 5 minutes
Given
\[X=\begin{bmatrix}0&1\\2&0\\1&1\end{bmatrix}\hspace{2mm}\by=\begin{bmatrix}-1\\[2pt]1\\[2pt]0\end{bmatrix}\]
VERIFY $X^\top\boldsymbol e=?$What does your result mean geometrically?
1–3. Build the normal-equation pieces
\[X^\top X=\begin{bmatrix}5&1\\1&2\end{bmatrix},\qquad(X^\top X)^{-1}=\frac{1}{9}\begin{bmatrix}2&-1\\-1&5\end{bmatrix}\]
\[X^\top\by=\begin{bmatrix}2\\[2pt]-1\end{bmatrix}\]
4. Solve for the parameters
\[\widehat{\btheta}_{\mathrm{LS}}=(X^\top X)^{-1}X^\top\by=\begin{bmatrix}5/9\\[2pt]-7/9\end{bmatrix}\]
5–6. Predict, then form the residual
\[\begin{aligned}\widehat{\by}=X\widehat{\btheta}_{\mathrm{LS}}&=\begin{bmatrix}-7/9\\[2pt]10/9\\[2pt]-2/9\end{bmatrix},\\[3mm]\boldsymbol e=\by-\widehat{\by}&=\begin{bmatrix}-2/9\\[2pt]-1/9\\[2pt]2/9\end{bmatrix}.\end{aligned}\]
\[X^\top\boldsymbol e=\begin{bmatrix}0&2&1\\1&0&1\end{bmatrix}\begin{bmatrix}-2/9\\[2pt]-1/9\\[2pt]2/9\end{bmatrix}=\begin{bmatrix}0\\[2pt]0\end{bmatrix}.\]
Residual $\perp\operatorname{col}(X)$ The numerical solution confirms the projection geometry.
TIME: 5 min. FIRST REVEAL: Say: “Now you are going to solve the whole least-squares problem yourselves. Work with the person next to you. Start with $X^\top X$, its inverse, and $X^\top\by$. Then find $\widehat{\btheta}$, the predictions, the residual, and finally $X^\top\boldsymbol e$. Use exact fractions. You have about five minutes.” PAUSE: Give students the full working time. After about four minutes, say: “If you already have $\widehat{\btheta}$, keep going. Make sure you get all the way to $X^\top\boldsymbol e$.” REVEAL 2: Ask for $X^\top X$, its inverse, and $X^\top\by$. Reveal each ingredient, then say: “Check these against your work before we continue.” REVEAL 3: Ask: “Using those pieces, what did you get for $\widehat{\btheta}$?” Take an answer, then reveal. REVEAL 4: Ask for $\widehat{\by}$, then for $\boldsymbol e=\by-\widehat{\by}$. Reveal after responses. REVEAL 5: Ask: “What happens when we compute $X^\top\boldsymbol e$? Why did we expect zero?” WAIT: Connect to the projection geometry. SAY: “$X^\top\boldsymbol e=0$ means the residual is orthogonal to every column of $X$, hence perpendicular to $\operatorname{col}(X)$.” TRANSITION: The closed-form solution gives exactly the projection predicted by the geometry.
\[\nabla^2 J(\btheta)=2X^\top X\succeq 0.\]
The objective is convex.
orthogonal residual $\Updownarrow$ normal equations $\Updownarrow$ zero gradient
TIME: 2.5 min. ASK: What condition characterizes the minimum of this convex quadratic? REVEAL: Gradient, zero-gradient equation, then the positive-semidefinite Hessian. SAY: The Hessian confirms convexity. Geometry and calculus identify the same global optimum. TRANSITION: A formula is not always the best computational route.
TIME: 1.5 min. ASK: Which route would you prefer when $X$ is too large for a direct factorization? SAY: Both optimize the same least-squares objective. The notes' update absorbs the factor of $2$ into the learning rate $\rho$. TRANSITION: Step size controls whether the iterative path crawls, converges, or explodes.
Demo 2: one objective, three learning rates
Predict each path.
$J(\btheta)=(\theta_0-1)^2+4(\theta_1-2)^2$
Stable, but slow.
Fast convergence.
Overshoots and diverges.
$\eta$TIME: 4 min. ASK: Before each reveal, predict the trajectory for the learning rate. PAUSE: Let students trace a path in the air from the common start. REVEAL: Small, useful, then too-large learning rate. SAY: The objective is convex, yet an unstable step size can still prevent convergence. TRANSITION: We know how to minimize squared error. Why squared error in the first place?
How do we compute $\widehat{\btheta}$?
$\checkmark$ Closed form or gradient descent
What can we say statistically about $\widehat{\btheta}$?
TIME: 0.5 min. ASK: We know how to compute the estimate. What questions remain about its behavior across possible datasets? REVEAL: The statistical question. SAY: This connects directly to our earlier lecture on bias, variance, and estimator quality. TRANSITION: Begin with assumptions on only the first two moments of the error.
\[\E[\bY\mid X]=X\btheta^*,\qquad\cov(\bY\mid X)=\sigma^2 I_n.\]
The errors have mean zero, common variance, and zero covariance across observations.
No Gaussian assumption yet
We have specified only the mean and covariance of the error.
As in the notes, treat $X$ as fixed and assume full column rank.
TIME: 2 min. ASK: Does a mean and covariance determine a full distribution? PAUSE: Wait for “no.” REVEAL: Moment implications, then the no-Gaussian warning. SAY: Uncorrelated errors with common variance are enough for the next results. TRANSITION: Apply these assumptions to the least-squares estimator.
TIME: 2.5 min. ASK: Which frequentist property are we checking when we compute $\E[\widehat{\btheta}]$? REVEAL: Move expectation through the linear estimator, substitute the model mean, then conclude. SAY: Across repeated datasets generated at the same fixed $X$, the estimator centers on $\btheta^*$. TRANSITION: Unbiasedness gives the center. Now compute the spread.
TIME: 2.5 min. ASK: In the covariance rule $\cov(A\bY)=A\cov(\bY)A^T$, what is $A$ here? REVEAL: Covariance transformation, substitute $\sigma^2I_n$, simplify, then connect variance to MSE. SAY: This reuses the frequentist bias-variance identity from the estimator-properties lecture. TRANSITION: Unbiasedness and covariance lead to an optimality theorem.
Gauss–Markov: how good is OLS?
\[\E[\boldsymbol\varepsilon]=\boldsymbol 0,\qquad\cov(\boldsymbol\varepsilon)=\sigma^2I_n,\]
OLS is the best linear unbiased estimator of $\btheta^*$.
Linear means linear in the observed response:
\[\widetilde{\btheta}=A\bY.\]
B
Best
L
Linear
U
Unbiased
E
Estimator
For any vector $\bu$, $\bu^\top\widehat{\btheta}$ has the smallest variance among linear unbiased estimators of $\bu^\top\btheta^*$.
Gaussianity is not required for Gauss–Markov
TIME: 2.5 min. ASK: In BLUE, what does “linear” refer to? REVEAL: The response-linear form, the acronym, then the scalar-combination consequence. SAY: Gauss–Markov compares OLS with all linear unbiased competitors. It uses only the first two error moments. TRANSITION: Now strengthen the model from moment assumptions to a full distribution.
The Gaussian model adds a full distribution
General model
\[\begin{aligned}\E[\boldsymbol\varepsilon]&=\boldsymbol0,\\\cov(\boldsymbol\varepsilon)&=\sigma^2I_n.\end{aligned}\]
Moments only.
TIME: 3 min. ASK: At a fixed $x$, what varies from one possible observation to another? REVEAL: Start with moment assumptions, then add Gaussianity and the full distributions. SAY: The Gaussian model strengthens the general model. It does not replace the mean and covariance assumptions. TRANSITION: State that progression explicitly before forming the likelihood.
From moments to a Gaussian distribution
General model: mean and covariance assumptions
\[\E[\boldsymbol\varepsilon]=\boldsymbol0,\qquad\cov(\boldsymbol\varepsilon)=\sigma^2I_n.\]
$+$ assume the full error distribution
Gaussian model: full distributional assumption
\[\boldsymbol\varepsilon\sim\cN(\boldsymbol0,\sigma^2I_n)\quad\Longrightarrow\quad\bY\sim\cN(X\btheta^*,\sigma^2I_n).\]
The stronger assumption enables likelihoods and full sampling distributions.
TIME: 1.5 min. ASK: Which line adds information beyond the first two moments? REVEAL: Add the Gaussian distribution, then the statistical consequences. SAY: Everything proved under the general model remains true. Gaussianity gives additional results. TRANSITION: Begin those results with the likelihood.
From one observation to the likelihood
\[p(y_i\mid\bx_i,\btheta,\sigma^2)=\frac{1}{\sqrt{2\pi\sigma^2}}\exp\!\left[-\frac{(y_i-\btheta^\top\bx_i)^2}{2\sigma^2}\right].\]
\[p(\by\mid X,\btheta,\sigma^2)=\prod_{i=1}^{n}p(y_i\mid\bx_i,\btheta,\sigma^2).\]
TIME: 2.5 min. ASK: What operation combines independent densities? PAUSE: Wait for “multiply.” REVEAL: Product likelihood and diagram. SAY: Independence is what lets us multiply the observation densities. The observed data determine how plausible each parameter value is. TRANSITION: Logs turn the product into the sum we already know.
Maximum likelihood reveals least squares
Under independent, constant-variance Gaussian noise: $\boldsymbol\varepsilon\sim\cN(\mathbf0,\sigma^2I_n)$.
The constants and the positive factor $1/(2\sigma^2)$ do not change the optimizer.
TIME: 3 min. ASK: Stop before the reveal. For fixed variance, what quantity must shrink when log-likelihood grows? PAUSE: Let students identify the squared-residual sum. REVEAL: The equivalent argmin. TRANSITION: We can now state the central equivalence.
Maximum likelihood estimates both parameters
Mean parameters
\[\widehat{\btheta}_{\mathrm{ML}}=(X^\top X)^{-1}X^\top\by=\widehat{\btheta}_{\mathrm{LS}}.\]
under independent Gaussian noise with common variance.
TIME: 2 min. ASK: Which unknown quantity remains after maximizing over $\btheta$? REVEAL: The OLS estimate, the variance MLE, then the equivalence statement. SAY: The variance MLE divides the residual sum of squares by $n$. This differs from the later unbiased variance estimate that uses degrees of freedom. TRANSITION: The squared residuals also explain a practical vulnerability.
What else does Gaussianity buy us?
\[\E[\widehat{\btheta}]=\btheta^*,\qquad\cov(\widehat{\btheta})=\sigma^2(X^\top X)^{-1}.\]
Because $\widehat{\btheta}$ is a linear transformation of Gaussian $\bY$,
\[\boxed{\widehat{\btheta}\sim\cN\!\left(\btheta^*,\sigma^2(X^\top X)^{-1}\right)}.\]
General model gives unbiasedness, covariance, and Gauss–Markov.
Gaussian model additionally gives sampling distribution, likelihood, and stronger efficiency results.
TIME: 2 min. ASK: Why can we now identify the full distribution of $\widehat{\btheta}$? REVEAL: The Gaussian sampling distribution, then the general-versus-Gaussian comparison. SAY: In the frequentist statement, $\btheta^*$ is fixed and $\widehat{\btheta}$ varies across datasets. TRANSITION: Use the likelihood curvature to obtain a stronger optimality bound.
Gauss–Markov best among linear unbiased estimators
Gaussian CRLB attains the lower bound among unbiased estimators
TIME: 2.5 min. ASK: Compare the covariance from the general model with the inverse Fisher information. REVEAL: The bound, Fisher information calculation, equality, then the comparison. SAY: This is the calculation represented by Example 5.5. The Gaussian assumption strengthens the comparison class beyond linear estimators. TRANSITION: So far $\btheta$ has remained fixed but unknown. Bayesian inference changes what carries uncertainty.
From a point estimate to a posterior
So far, $\btheta$ has been unknown but fixed.
What if we represent uncertainty about $\btheta$ probabilistically?
\[\by\mid\btheta,\sigma^2\sim\cN(X\btheta,\sigma^2I_n).\]
\[p(\btheta,\sigma^2)\propto\frac1{\sigma^2}.\]
Equivalently,
\[p(\btheta)\propto1,\quad p(\sigma^2)\propto\frac1{\sigma^2},\]
with $\btheta$ and $\sigma^2$ independent.
Bayesian inference seeks
\[p(\btheta,\sigma^2\mid\by),\]
not only a point estimate.
TIME: 2 min. ASK: In the frequentist analysis, which object was random and which parameter was fixed? REVEAL: The Bayesian question, likelihood and prior, then posterior factorization. SAY: The prior is the exact non-informative choice used in the notes. TRANSITION: Condition on $\sigma^2$ and complete the square in $\btheta$.
Posterior for $\btheta\mid\by,\sigma^2$
On this slide, $\sigma^2$ is treated as fixed and known.
Same likelihood as before, now used in Bayes’ rule.
\[\widehat{\btheta}=(X^\top X)^{-1}X^\top\by.\]
\[\begin{aligned}(\by-X\btheta)^\top(\by-X\btheta)&=\underbrace{(\btheta-\widehat{\btheta})^\top X^\top X(\btheta-\widehat{\btheta})}_{\textcolor{#087F80}{\text{depends on }\btheta}}\\[-1mm]&\quad+\underbrace{\textcolor{#8c8c8c}{(\by-X\widehat{\btheta})^\top(\by-X\widehat{\btheta})}}_{\textcolor{#737373}{\text{constant with respect to }\btheta}}.\end{aligned}\]
With a flat prior, the posterior is centered at the least-squares estimate.
TIME: 5 min total for the five reveals. REVEAL 1: Bayes' rule and the flat prior. SAY: Let us slow this derivation down. We want the posterior of $\btheta$, conditional on the data and $\sigma^2$. Bayes' rule says the posterior is proportional to the likelihood times the prior. The prior on $\btheta$ is flat, so it does not favor one value over another. The posterior shape therefore comes entirely from the likelihood. REVEAL 2: The Gaussian likelihood. SAY: This is the same Gaussian likelihood we used for maximum likelihood. Before, we maximized it. Now we use it to construct a distribution over $\btheta$. REVEAL 3: The completing-the-square identity. ASK: Which term still depends on $\btheta$? SAY: This is the vector version of rewriting a quadratic as a squared distance plus a constant. The first term depends on $\btheta$ and is centered at $\widehat{\btheta}$. The second term contains no $\btheta$, so it only changes the normalizing constant. REVEAL 4: Compare the kernel with a multivariate Gaussian. SAY: The quadratic has the Gaussian form. Its center is $\widehat{\btheta}$, and its precision is $(1/\sigma^2)X^\top X$. Inverting the precision gives covariance $\sigma^2(X^\top X)^{-1}$. REVEAL 5: The final posterior. SAY: Conditioned on $\sigma^2$, the posterior is Gaussian. Its center is the least-squares estimate, and its covariance quantifies uncertainty around that estimate. TRANSITION: This has almost the same form as the sampling distribution derived earlier, but its interpretation differs.
Observed $\by$ is fixed. $\btheta$ is uncertain and described by the posterior.
Question Given the observed data, which $\btheta$ values are plausible?
The covariance has the same algebraic form. The random object and the conditioning are different.
TIME: 2 min. ASK: In each statement, what is fixed and what is random? SAY: The frequentist statement describes repeated samples of the estimator when the parameter is fixed. The Bayesian statement conditions on the observed data and describes uncertainty about the parameter. The covariance matrices match under the flat prior, but the probability statements answer different questions. TRANSITION: The joint posterior also represents uncertainty in the noise variance.
$m$ is the number of model parameters, equivalently the number of columns of $X$.The posterior represents uncertainty about both $\btheta$ and $\sigma^2$.
TIME: 1.5 min. ASK: Why does the scale use $n-m$ rather than $n$? SAY: Estimating $m$ mean parameters consumes $m$ degrees of freedom. We will use this distribution without deriving its normalization. TRANSITION: Posterior sampling turns these two uncertainty statements into predictions.
Instead of plugging in one pair $(\widehat{\btheta},\widehat\sigma^2)$, posterior prediction propagates uncertainty in both parameters into $y_{\mathrm{new}}$.
Frequentist: $\btheta$ is fixed; $\widehat{\btheta}$ varies across datasets. Bayesian: after observing $\by$, $p(\btheta\mid\by)$ represents uncertainty about $\btheta$.
TIME: 2 min. ASK: At which steps do we propagate parameter uncertainty rather than only observation noise? REVEAL: Sample $\sigma^2$, then $\btheta$, then a new response; finish with the practical comparison. SAY: Repeating the three steps produces samples from the posterior predictive distribution. TRANSITION: Assemble the complete story of Sections 5.1 through 5.3.
One estimator: optimization, geometry, computation
Optimization
\[\min_{\btheta}\|\by-X\btheta\|_2^2\]
Minimize squared training error.
Geometry
Project $\by$ onto $\operatorname{col}(X)$.
Computation
\[\widehat{\btheta}=(X^\top X)^{-1}X^\top\by\]
or gradient descent with
\[\rho X^\top(\by-X\btheta^{(t)}).\]
$\displaystyle X^\top(\by-X\widehat{\btheta})=\boldsymbol0$ links all three views.
TIME: 2.5 min. ASK: Explain the same estimator in one sentence from each viewpoint. PAUSE: Call on three students, one per column. SAY: The normal equations are simultaneously an optimization condition, an orthogonality statement, and the target of the computation. TRANSITION: The statistical views also describe the same model, with progressively stronger assumptions.
Different questions about the same model: $\bY=X\btheta+\boldsymbol\varepsilon$.
TIME: 2 min. ASK: Which conclusions survive without Gaussianity, and which require it? REVEAL: Read left to right from weaker assumptions to posterior uncertainty. SAY: These are complementary analyses of one linear-regression model, not disconnected techniques. TRANSITION: Stop at Section 5.3 and preview where Lecture II begins.
Next: what happens when fitting the data too well becomes the problem?
Linear Regression II
§ 5.4 Regularization
§ 5.5 Bias, variance, overfitting, and model selection
§ 5.6 Stochastic gradient descent
Post-class reading: §5.2 Standardization and Example 5.3.
TIME: 0.5 min. ASK: Leave the opening question for Lecture II unanswered. SAY: Assign Standardization and Example 5.3 as post-class reading. Lecture II begins at Section 5.4. TRANSITION: End.
Reserve: what if the normal equations are singular?
\[X=\begin{bmatrix}1&2\\2&4\\3&6\end{bmatrix}\]
Can we recover a unique $\btheta$ from the predictions $X\btheta$?
Inspect the columns before calculating.
\[\boldsymbol x^{(2)}=2\boldsymbol x^{(1)}\]
The columns are linearly dependent, so $X$ does not have full column rank.
\[\det(X^\top X)=0.\]
TIME: Reserve. ASK: Diagnose the issue without multiplying matrices. REVEAL: Column dependence and singularity. TRANSITION: Show two parameter vectors with identical predictions.
The prediction is identifiable. The parameter vector is not.
In fact, every $\btheta$ satisfying $\theta_1+2\theta_2=2$ gives the same prediction.
TIME: Reserve. ASK: Verify both matrix products. Can you describe all equivalent parameter vectors? SAY: Redundant features create a flat direction in parameter space. TRANSITION: Connect redundancy to nonunique least-squares solutions.
Reserve: the rank-deficiency chain
redundant columns in $X$
$X^\top X$ is singular
least-squares parameters may be nonunique
predictions can still be unique
The inverse formula requires full column rank. Other solution concepts belong to later sections.
TIME: Reserve. ASK: Which link in the chain breaks the closed-form inverse? SAY: Stop at the rank diagnosis. Remedies belong to later sections.