Linear Regression II

Regularization, overfitting, bias, variance, and noise

Sarvin Motamen

University of Virginia

October 8, 2026

Opening, 30 seconds. This follows least squares, gradient descent, Gaussian regression, and Bayesian regression from Lecture I. Today we ask which fitted predictor generalizes and how to compute it efficiently. Source: course notes, Sections 5.4–5.6.

Which model would you trust?

Question: What should our goal be?

We have observed 12 noisy outcomes. A new outcome has yet to arrive.

Question: Which would you trust on new data?

A smooth quadratic fit

A degree-11 interpolating fit

Zero training error can coexist with poor prediction. The dashed curve is the true mean relationship.

Training observations Held-out observations

2–3 minutes. Reveal 1: invite goals, especially accurate future prediction. Reveal 2: ask students to vote and identify the regions where they distrust the flexible curve. Reveal 3: compare held-out points, not only training fit. All observations are synthetic: mean .45+.65x-1.25x squared, Gaussian noise SD .27. The fits use the same 12 observations.

Fitting the data is not the whole goal

\[\min_{\btheta}\;\|\by-X\btheta\|_2^2.\] This measures residuals at the observed inputs.

Question: Can we discourage unnecessarily complicated solutions?

\[\text{data fit}+\text{complexity penalty}\]

The model should explain patterns that persist in new data.

2 minutes. Connect the nearly zero residuals to the least-squares objective. A polynomial is still linear in its coefficients: the design matrix contains powers of x. We retain least squares as a data-fit term and change how we choose among plausible fits. Next: specify a coefficient penalty.

Regularization

\[\cL(\btheta)=\underbrace{\cte{\|\by-X\btheta\|_2^2}}_{\text{fit to data}}+\underbrace{\cpr{\lambda\|\btheta\|_2^2}}_{\text{complexity penalty}}.\]

Fit the observed responses well.

Prefer coefficients with smaller squared length.

\[\lambda\geq0\quad\text{sets the strength of the penalty.}\]

At $\lambda=0$, recover ordinary least squares.

2 minutes. Ask how a larger coefficient vector affects each term. The squared coefficient length is a model preference, not a direct count of wiggles. A fixed feature representation connects coefficients to fitted curves. The displayed objective penalizes every coefficient; implementations often leave the intercept unpenalized.

What does $\lambda$ do?

Question: What do you expect as $\lambda$ increases?

Interpolates the training points.

$\|\widehat{\btheta}\|_2\approx765$.

Accepts some residual error.

$\|\widehat{\btheta}\|_2\approx0.667$.

Strong shrinkage toward zero.

$\|\widehat{\btheta}\|_2\approx0.195$.

Larger $\lambda$ gives a stronger preference for smaller coefficients.

Same degree-11 polynomial basis, data, and axes in every state.

$\lambda$ (log scale) $\lambda=$ $\|\widehat{\btheta}\|_2\approx$ 3 minutes. Ask before revealing any curve. Discuss the disappearance of noise-following oscillations, then the eventual flattening. The generator fits actual ridge models, not schematic curves. The basis is (1,x,...,x to power 11), so the large unregularized norm reflects cancellation among large coefficients. Lambda values are meaningful for this fixed basis and summed-error normalization only.

Two ways to say the same thing

\[\min_{\btheta}\;\|\by-X\btheta\|_2^2+\lambda\|\btheta\|_2^2.\] Pay a price for coefficient length. \[\min_{\btheta}\;\|\by-X\btheta\|_2^2\quad\text{subject to }\|\btheta\|_2^2\leq t.\] Limit the allowed coefficient length.

For each $\lambda$ there is a $t$ giving the same solution, and vice versa: $t=\|\widehat{\btheta}(\lambda)\|_2^2$.

$\lambda$ (log scale)$\lambda=$

$t$$t=$

2 minutes. Explain that for a penalized optimum, choose t as its squared norm. Conversely an active constrained solution has a Lagrange multiplier. The correspondence concerns matching optima, not lambda=t or a universal reciprocal relation. In inactive or degenerate cases uniqueness of the correspondence needs care. Next: solve the penalized problem.

Ridge regression: solution

\[\cL(\btheta)=\|\by-X\btheta\|_2^2+\lambda\|\btheta\|_2^2.\]

\[\nabla\cL(\btheta)=-2X^T(\by-X\btheta)+2\lambda\btheta.\]

\[\nabla\cL(\widehat{\btheta})=0\quad\Longleftrightarrow\quad(X^TX+\cpr{\lambda I})\widehat{\btheta}=X^T\by.\]

\[\boxed{\widehat{\btheta}_{\rm ridge}=(X^TX+\cpr{\lambda I})^{-1}X^T\by}.\]

$\displaystyle\widehat{\btheta}_{\rm LS}=(X^TX)^{-1}X^T\by.$Only the added $+\lambda I$ changes the system.

With $X^TX=I$: $\widehat{\btheta}_{\rm LS}=X^T\by$ and $\widehat{\btheta}_{\rm ridge}=\frac{1}{1+\lambda}X^T\by$.

3 minutes. Have students supply the penalty gradient. Reveal one algebraic step at a time. Multiplication by X transpose collects residual contributions across observations. Unlike OLS, the ridge residual need not be orthogonal to every design column: X transpose residual=lambda theta. OLS inverse requires full column rank; the next slide removes that restriction for positive lambda.

A useful side effect

Question: What if $X^TX$ is singular?

Example: $X=\begin{bmatrix}1&2\\2&4\\3&6\end{bmatrix}$, $\by=(2,4,6)$. The second column is twice the first, so $X^TX$ is singular.

Least squares has a whole line of minimizers: every $\btheta$ with $\theta_1+2\theta_2=2$ fits exactly.

Adding $\lambda\|\btheta\|_2^2$ turns the valley into a bowl with one minimum, since $X^TX+\lambda I$ is invertible for every $\lambda>0$.

Ridge both constrains coefficients and stabilizes the linear system.

$\lambda$ (log scale)$\lambda=$ 2 minutes. Recall duplicate columns from Lecture I. Adding lambda I shifts eigenvalues without changing eigenvectors. Zero eigenvalues become positive, so the solution is unique. This assumes every coefficient receives the positive penalty shown in the formula. Avoid a formal proof; ask which bar was problematic before regularization.

Ridge through a Bayesian lens

\[\by\mid\btheta,\sigma^2\sim\cN(X\btheta,\sigma^2I).\]

\[\btheta\mid\sigma^2\sim\cN\!\left(\bzero,\frac{\sigma^2}{\lambda}I\right).\]

Smaller coefficient vectors have higher prior density.

\[p(\btheta\mid\by,\sigma^2)\propto\exp\!\left[-\frac{\|\by-X\btheta\|_2^2+\lambda\btheta^T\btheta}{2\sigma^2}\right].\]

\[\text{MAP estimate}=\text{posterior mode}=\widehat{\btheta}_{\rm ridge}.\]

Regularization expresses prior information about plausible coefficients.

3 minutes. Hold sigma squared fixed, as the notes do. Reveal likelihood, prior, posterior kernel, then mode. Contours show equal prior density; the center is zero. Multiplication of likelihood and prior adds quadratic terms. The posterior is Gaussian with covariance sigma squared times the inverse of X transpose X+lambda I, so its mode and mean agree. Source: Section 5.4.

Do we always want “small”?

Question: What if we want unimportant features to disappear entirely?

\[\btheta=(2.8,\ \cgy{0.05},\ \cgy{-0.03},\ 1.9,\ \cgy{0.01}).\]

\[\btheta=(2.8,\ \cbr{0},\ \cbr{0},\ 1.9,\ \cbr{0}).\]

Shrink coefficients and allow exact zeros: a sparse, more interpretable predictor. 2 minutes. The coefficient vector is schematic, not the result of a fitted optimization problem. Ask what a zero coefficient means for prediction. The visual change highlights selection of features; important coefficients may also shrink. Next compare the actual penalties that create different behavior.

Lasso

\[\min_{\btheta}\;\|\by-X\btheta\|_2^2+\lambda\|\btheta\|_2^2.\] Shrink coefficients. \[\min_{\btheta}\;\|\by-X\btheta\|_2^2+\lambda\|\btheta\|_1.\] \[\|\btheta\|_1=\sum_{i=1}^m|\theta_i|.\]

Shrink coefficients and create exact zeros.

\[\min_{\btheta}\;\|\by-X\btheta\|_2^2\quad\text{subject to}\quad\|\btheta\|_1\leq t.\]

As for ridge, the penalty strength corresponds to a constraint size.

Lasso has no general matrix-inverse solution like ridge. Efficient numerical methods exist.

2 minutes. Have students compare squaring with taking absolute values near zero. Explicitly discuss both constrained and penalized forms. Lasso does not guarantee zeros for every coordinate or dataset; it encourages them. For orthonormal designs there is soft thresholding, but the general solution is numerical. Next use two-dimensional geometry.

Why does lasso create zeros?

Question: Where can the smallest loss contour touch the constraint?

A smooth circle usually touches away from an axis.

The diamond has corners on the axes.

A corner solution has a coefficient exactly zero.

3 minutes. The loss is proportional to (theta1-2.6) squared+1.6(theta2-.55) squared. The circle and diamond both have radius 1.4 on their coordinate axes. Draw the first touching contour conceptually: a corner can receive a range of supporting slopes. Exact zeros are common, not inevitable. The rounded constraint can also meet an axis in special cases.

Bayesian view: ridge vs lasso

\[p(\btheta\mid\sigma^2)\propto e^{-\lambda\|\btheta\|_2^2/(2\sigma^2)}.\]

Circular equal-density contours.

MAP = ridge; mean and mode coincide.

\[p(\btheta\mid\sigma^2)\propto e^{-\lambda\|\btheta\|_1/(2\sigma^2)}.\]

Diamond equal-density contours.

MAP = lasso; the mean need not be sparse.

2 minutes. Use the same Gaussian likelihood in both columns. The difference is the prior. For lasso the posterior is not generally Gaussian and its mode is not generally its mean. A continuous prior can yield a zero MAP coefficient without positive prior point mass at zero. Source: Section 5.4 Bayesian interpretations.

A whole family of regularizers

\[R_q(\btheta)=\|\btheta\|_q^q=\sum_{i=1}^m|\theta_i|^q,\qquad p(\btheta\mid\sigma^2)\propto e^{-\lambda R_q(\btheta)/(2\sigma^2)}.\]

$q=2$ ridge

$q=1$ lasso

$q=\tfrac12$ stronger axis preference

Density falls away from zero. Small $q$ gives relatively longer tails along axes.

For $0<q<1$, the penalty is nonconvex and $\|\cdot\|_q$ is not a norm.

2–3 minutes. Redraw of the notes' Lq figure in the reference deck's contour style. All shapes are level sets of sum absolute coefficient to power q. Reveal the identities before discussing sparse axes. Smaller q can emphasize sparsity but makes optimization nonconvex. Next return to the original question: why can changing the training objective improve squared test error?

But why should adding bias ever help?

\[\E[(Y-f(\bx))^2\mid\bX=\bx].\]

Question: Why did we deliberately change the squared-error objective?

Discuss what changes when we evaluate a new response rather than fit an observed one.

Least squares Ridge dashed: true mean

“Adding bias” means making the estimator statistically biased—not adding a bias term to the loss.

2 minutes. Invite hypotheses without giving the decomposition yet. A training fit and a future response depend on different randomness. The ridge curve is biased under the original Gaussian linear model, but the relevant goal is prediction risk. The next slides separate noise in a future response from instability of the fitted model.

Prediction has a ceiling

Question: Even if we knew the true mean exactly, could we predict $Y$ perfectly?

\[\ybar(x)=\E[Y\mid X=x].\] Each blue outcome is one realization around that mean.

The remaining variation is intrinsic / irreducible error.

2 minutes. The sideways Gaussian is centered at the mean on the regression relationship; it is not centered at an observed point. Vertical position is response, horizontal width is density, not another x value. The Gaussian is an illustration, while the next decomposition needs only finite second moments. Irreducibility is relative to the current inputs X; additional informative inputs could change it.

Where does prediction error come from?

At a fixed input $\bx$, first hold the predictor $f$ fixed.

\[\E[(Y-f(\bx))^2\mid\bX=\bx]=\underbrace{\cpd{\var(Y\mid\bX=\bx)}}_{\text{intrinsic noise}}+\underbrace{\cro{(\ybar(\bx)-f(\bx))^2}}_{\text{error in our predictor}}.\]

Question: Which part can choosing $f$ change?

\[f^*(\bx)=\ybar(\bx)=\E[Y\mid\bX=\bx].\] The second term becomes zero. Intrinsic noise remains.

More training data do not remove $\var(Y\mid\bX=\bx)$ for these inputs.

2–3 minutes. Use E[(Z-c) squared]=Var(Z)+(EZ-c) squared for the conditional distribution. If useful, add and subtract the conditional mean verbally: the cross term is zero. The notes distinguish this test error of a given fitted function from expected test error of a learning procedure, which also averages over training datasets. Transition to that extra randomness.

Our fitted model is also random

Same hypothesis class: quadratics.

Same loss: least squares.

New independent noise in each dataset.

Question: Why are the fits different if our algorithm did not change?

The training dataset $\cD$ is random, so the learned function $f_{\cD}$ is random.

Dashed: the same true mean. Colors: independent datasets and their fits.

3 minutes. Each dataset has 16 fixed inputs and fresh independent Gaussian noise SD .30. Reveal the second and third independently fitted curves, keeping truth and axes unchanged. In the notes, M consists of the fixed hypothesis class and empirical loss. Empirical risk minimization picks $f_{\cD}$ from that class; changing $\cD$ changes the chosen predictor.

Bias vs variance: see it first

Question: Classify each learning procedure from its repeated fits.

High bias
low variance

Low bias
high variance

Low bias
low variance

Dashed: truth. Thin: individual fits. Thick: the average fit across datasets.

Bias compares the average fit with truth. Variance compares individual fits with their average. 3 minutes, including a one-minute pair discussion before the answer reveal. These are schematic repeated-fit ensembles designed to isolate the two ideas. Ask students to identify the average curve before using the labels. High variance concerns changes across datasets, not noise around a single fitted curve. Next give exact mathematical definitions at a fixed input.

Bias–variance decomposition

For a fixed input $\bx$, average over independent training datasets and a new response.

\[\underbrace{\E_{\cD,Y}[(Y-f_{\cD}(\bx))^2\mid\bX=\bx]}_{\text{expected test error}}=\cpd{\underbrace{\var(Y\mid\bX=\bx)}_{\text{irreducible error}}}+\cro{\underbrace{(\E_{\cD}f_{\cD}(\bx)-\ybar(\bx))^2}_{\text{bias}^{2}}}+\cam{\underbrace{\var_{\cD}(f_{\cD}(\bx))}_{\text{variance}}}.\]

Variation around the true mean.

Average prediction misses truth.

Predictions vary across datasets.

Neither unbiasedness nor low variance alone guarantees the smallest test error.

3 minutes. Read the expectation subscripts carefully. The future outcome is independent of the training set conditional on this test input. Both the model family and fitting rule are fixed. The identity does not require a Gaussian model. There is a swapped-term sentence in the notes after the intermediate variance identity; the displayed definitions here follow the correct identity. Do not derive on the slide.

Watch bias and variance happen

Question: What changes when we change only model flexibility?

Fits cluster together but miss the curvature of the truth. The model can represent the truth, and the fitted curves remain relatively stable. Extra freedom follows sample-specific fluctuations. Fits vary more across datasets.

Eight independently sampled datasets, each with $n=16$ and the same noise level.

polynomial degree 3 minutes. These are actual least-squares fits on the same eight datasets used in the previous repeated-sampling demonstration. Ask for a prediction before each reveal. One realization looking close to truth is weaker evidence than stability across repetitions. The true mean is quadratic in all states, and all coordinate axes stay fixed.

What happens as flexibility increases?

More flexibility: bias generally falls.

More flexibility: variance generally rises.

More training data generally reduce variance.

Same quadratic procedure, now $n=150$.

Bias$^2$ Variance; repeated-data estimate

2–3 minutes. The curves estimate integrated squared bias and variance over a fixed x grid from 1200 independent noise samples, with the same 16 training inputs. More flexibility removes the missed quadratic curvature, after which bias is already near zero. The larger-data panel uses eight independent n=150 samples. More data can also reduce finite-sample bias, depending on the procedure. Expected test error falls when bias or variance falls while intrinsic noise stays fixed. These are tendencies, not universal monotonic laws. Training error with more data can rise as memorization becomes harder.

Underfit, good fit, overfit

Question: What failure, if any, do you see?

Insufficient flexibility misses the pattern: underfit / high bias. Captures the underlying form without interpolating each fluctuation. Training fit is exact, but noise sensitivity is high: overfit / high variance.

Illustrative quadratic truth with noisy observations.

Truth: $Y=ax+W$, $W\sim\cN(0,\sigma^2)$; train at $x_1=1$, $x_2=2$; test at $x_3=3$.

\[\begin{aligned}f_0(x)&=0,\\[1mm]f_1(x)&=\frac{y_1+2y_2}{5}\,x,\\[1mm]f_2(x)&=\frac{4y_1-y_2}{2}\,x-\frac{2y_1-y_2}{2}\,x^2.\end{aligned}\]

Question: Which model has zero training error? Does it predict best at $x=3$?

The flexible quadratic can fit both noisy observations exactly even when the true mean is linear.

3 minutes. Invite labels before discussing each revealed fit; the title identifies the sequence but not which visual is which. The first six states use the requested quadratic illustration, with a question state and a label state for each model. The final state faithfully presents the notes' actual Example 5.6, which has linear truth, two training points, and extrapolation to x=3. Assume sigma small relative to the signal when judging the zero predictor. The following slide compares exact risks.

Training error tells only half the story

Question: If we only inspect training error, what model would we choose?

\[\frac1n\sum_{i=1}^n(y_i-f_{\cD}(\bx_i))^2.\]

Training error rewards fitting the same observations that chose the parameters.

New responses were not used to choose them.

Expected training error Expected test error

Example 5.6: exact expectations at $x_3=3$

	Train	Noise	Bias$^2$	Variance	Test total
$f_0=0$	$\frac52a^2+\sigma^2$	$\sigma^2$	$9a^2$	$0$	$9a^2+\sigma^2$
$f_1$: linear	$\sigma^2/2$	$\sigma^2$	$0$	$\frac95\sigma^2$	$\frac{14}5\sigma^2$
$f_2$: quadratic	$0$	$\sigma^2$	$0$	$18\sigma^2$	$19\sigma^2$

Both fits are unbiased, but the quadratic prediction variance at $x=3$ is ten times larger.

\[\text{expected training error}=\frac1n\sum_{i=1}^n\E_{\cD}[(Y_i-f_{\cD}(\bx_i))^2].\]

Lower training error does not establish lower expected test error.

3 minutes. Reveal test error after a vote. First two states are a 1200-repetition simulation averaged over test inputs in [-1,1], with intrinsic noise .09. Final state uses the notes' exact example at x=3, so these are different evaluations. The flexible predictor has zero training error but expected test loss 19 sigma squared versus 2.8 sigma squared for the correctly specified linear model. The zero predictor can have larger bias when the signal dominates noise.

Regularization is a bias–variance knob

Question: When is accepting bias worthwhile?

Shrinkage reduces variation across datasets but moves the average prediction toward zero.

Test error improves when the variance reduction exceeds the increase in squared bias.

Variance Bias$^2$ Total test error

Expected test error $=$ bias$^2$ $+$ variance $+$ irreducible error.

For $\by=X\btheta+\boldsymbol\varepsilon$, with $\E\boldsymbol\varepsilon=0$, \[\E_{\cD}[\widehat y_{\rm new}]=\bx_{\rm new}^T(X^TX+\lambda I)^{-1}X^TX\btheta.\]

If $X^TX=I$, then

\[\E[\widehat y_{\rm new}]=\frac{\bx_{\rm new}^T\btheta}{1+\lambda},\qquad\bias=-\frac{\lambda}{1+\lambda}\bx_{\rm new}^T\btheta.\]

\[\var(\widehat y_{\rm new})=\frac{\sigma^2\|\bx_{\rm new}\|_2^2}{(1+\lambda)^2}.\]

The mean shrinks toward zero for either sign of the true response. Variance also falls.

\[\cL_b=\|\by-X\btheta\|_2^2+b\theta_2^2.\] \[\begin{aligned}\widehat\theta_1&=\frac{b(y_1+2y_2)+8y_1-2y_2}{5b+4},\\\widehat\theta_2&=\frac{-4y_1+2y_2}{5b+4}.\end{aligned}\]

For $h=1+5b/4$,

\[\E[\text{train error}]=\frac{\sigma^2}{2}(1-h^{-1})^2,\] \[\text{bias}^2=0,\quad\var(f_{\cD}(3))=\frac{9\sigma^2}{5}(1+9h^{-2}),\] \[\E[\text{test error}]=\frac{\sigma^2}{5}(14+81h^{-2}).\]

Example 5.6: penalize only curvature.

As $b\to\infty$, the unbiased linear fit remains: no large-$b$ underfit for this selective penalty.

3–4 minutes. First plot uses an exact orthonormal-design example: true prediction 1, unregularized prediction variance .6, noise .2. Risk=.2+(lambda/(1+lambda)) squared+.6/(1+lambda) squared; optimum lambda=.6. State 4 reproduces Example 5.7 without the notes' sign-specific less-than statement. State 5 includes the regularized row of Example 5.6. To agree with the printed coefficient/error formulas, the displayed objective is SSE+b theta2 squared: the notes' leading half on SSE otherwise requires replacing b by 2b. The displayed training error remains half-SSE as in the table.

Poor prediction does not always mean overfitting

Question: Would more data fix this?

Question: Would more regularization fix this?

The chosen family cannot express the periodic relationship over this range.

Changing the model family can matter more than tuning its complexity.

Dashed: sinusoidal mean. Red: a poorly matched linear family.

2 minutes. The notes warn that a fixed polynomial family is a poor global match to y=a sin x+noise over a wide range. A polynomial can approximate a sine on a bounded interval with sufficient degree; do not claim otherwise. The linear red curve illustrates family mismatch. More data may estimate the best member more accurately without removing approximation bias; regularization alone cannot add periodic structure.

How do we choose the model in practice?

Nature’s true model.

The true expected test error of each candidate.

Question: So how do we choose?

Candidates might differ in degree, $\lambda$, or the model family.

2 minutes. All candidates in the opening visual were compared with truth only because this was a simulation. In real data the conditional mean and true risk are unknown. Ask students for a way to mimic future prediction before revealing the three roles. These are disjoint datasets in the same target prediction setting; random splits assume suitable iid observations.

Train, validate, test

Repeated test-based choices adapt to that test set. Its error then loses its role as an honest evaluation on unseen data. 3 minutes. Ask where theta belongs and where lambda belongs. For example fit three ridge candidates on training, choose the lowest validation MSE, then evaluate the selected model once on test. Preprocessing must be fitted using training data; apply those learned transforms to validation/test. If refitting after selection, fix the procedure before final test evaluation. The test set cannot be used for another round of model selection. Source: Section 5.5 data splits.

One more problem: computation

\[\nabla\widehat\cL(\btheta)=-\frac2n\sum_{i=1}^n(y_i-\bx_i^T\btheta)\bx_i.\] Every update reads all $n$ observations.

Question: What if $n$ is enormous?

Could we update using one randomly sampled observation, or a small batch, at a time?

2 minutes. Use the normalized empirical squared error so the full gradient is the average per-example gradient. Full gradient work is roughly O(nm) per step for dense data; one sampled example costs O(m). Sampling a batch offers an intermediate cost and noise level. Before writing the update, follow the notes' root-finding argument.

Before SGD: finding a root

Question: Which way should we move to reach the zero?

\[\theta^{(t+1)}=\theta^{(t)}-a_t f(\theta^{(t)}).\] $a_t>0$ controls the step size.

If $f(\theta)>0$, move left.

If $f(\theta)<0$, move right.

Repeating the correction can approach $\theta^*$.

Bounded increasing illustration: $f(\theta)=\tanh(\theta-1)$.

2–3 minutes. Notes assume a bounded function, a unique root, locally increasing behavior, and an appropriately close initialization. These are part of a convergence setup, not a guarantee for an arbitrary function and start. Ask for the sign-based correction before the reveal. The illustrated steps use .5/t; later steps get smaller. The root remains at theta=1 on all related slides.

How large should the steps be?

$a_t=1/t$: enough cumulative movement.

$a_t=1/t^2$: can shrink too quickly and stop short.

Question: What should shrinking steps preserve?

Shown with common scale $1/2$; vertical offsets separate the tracks.

\[\underbrace{\sum_{t=1}^{\infty}a_t=\infty}_{\text{enough total movement}},\qquad\underbrace{\sum_{t=1}^{\infty}a_t^2<\infty}_{\text{accumulated noise is controlled}}.\] 2–3 minutes. Displayed paths use a common constant .5 multiplying 1/t and 1/t squared; multiplying by a positive constant does not alter either sum condition. After 40 steps theta is about 1.32 for harmonic steps and 2.07 for squared-harmonic steps, from start 2.8. The harmonic path keeps making progress beyond the finite picture. Square summability controls cumulative noise effects under the other stochastic-approximation assumptions; the noise observations themselves do not disappear.

What if the direction is noisy?

\[\E[F(\theta)\mid\theta]=f(\theta).\] Individual observations can have the wrong sign.

Question: Can repeated noisy corrections still find the zero?

\[\theta^{(t+1)}=\theta^{(t)}-a_tF(\theta^{(t)}).\]

Diminishing steps can average out noise and approach the root.

Red: noisy function samples. Orange: iterates in order (vertical offset is time).

2–3 minutes. The simulation adds independent bounded uniform noise in [-1.2,1.2] to tanh(theta-1) and uses .5/t steps. Ask students to identify a point with the wrong sign. The descending display offset orders the iterates; it is not a function value. Unbiased directions and the two step-size sums are not sufficient without regularity and stability assumptions. Next replace this noisy direction with a one-example regression gradient.

This is stochastic gradient descent

\[\cL(\btheta)=\E[(y-\bx^T\btheta)^2],\qquad\nabla\cL(\btheta)=\E[-2(y-\bx^T\btheta)\bx].\]

\[f(\btheta)=\E[-2(y-\bx^T\btheta)\bx],\] \[F(\btheta)=-2(y-\bx^T\btheta)\bx,\] \[f(\btheta)=\E[F(\btheta)].\] One random observation provides a noisy gradient.

Realization of $F$ from sampled observation $i$: \[F_i(\btheta)=-2(y_i-\bx_i^T\btheta)\bx_i.\] \[\btheta^{(t+1)}=\btheta^{(t)}-\eta_tF_i(\btheta^{(t)})\] \[=\btheta^{(t)}+2\eta_t(y_i-\bx_i^T\btheta^{(t)})\bx_i.\]

Set $a_t=2\eta_t$ to obtain the notes’ update:

\[\boxed{\btheta^{(t+1)}=\btheta^{(t)}+a_t\underbrace{\bigl(y_i-\bx_i^T\btheta^{(t)}\bigr)}_{\text{error}}\underbrace{\bx_i}_{\text{feature direction}}}.\]

Prediction: $\bx_i^T\btheta^{(t)}$. Update direction: (error)$\,\bx_i$.

3 minutes. In the population objective, lowercase x,y denote a random observation. Interchanging gradient and expectation assumes suitable integrability. For finite data, uniform sampling makes the one-example gradient unbiased for the averaged empirical gradient, conditional on the current iterate. The raw squared loss yields a factor 2; the notes absorb it into the learning rate. A mini-batch averages sampled per-example gradients.

Gradient descent vs SGD

All data per update.

Exact empirical gradient.

Expensive update.

One point / mini-batch.

Noisy gradient.

Cheap update.

Full gradient Stochastic gradient

Noise buys computational efficiency.

2 minutes. Both paths share a start and one quadratic loss surface. Paths come from actual deterministic and stochastic iterations for a quadratic population objective with Hessian diag(1,3), minimum (.3,-.2), and randomly perturbed per-sample centers. This is a schematic loss example, not the earlier polynomial ridge data. Full-gradient steps use .22; SGD steps decrease. The finite noisy path illustrates motion, not a finite-time convergence guarantee.

The whole story

2 minutes. Ask students to narrate one link in the chain. Least squares addresses fitting, regularization controls a model preference, bias-variance describes repeated-sampling risk, validation chooses complexity without knowing truth, and SGD addresses computational cost. None eliminates conditional response variation given the current inputs. Exit question: If training MSE falls after increasing degree, what additional evidence is needed before trusting the new model?