Part 3: How can we make accurate predictions based on data, and how accurate can they be?
Probabilistic Machine Learning
University of Virginia Farzad Farnoud
October 15, 2026
Today we're going to study classification. In contrast to regression, which predicts a numerical value, classification predicts a label for an observation from a discrete set of possible labels. We'll build models that assign probabilities to those labels and fit them using labelled examples.
Goal: learn to predict the label of a new input $\bx$.
Rain tomorrow
$\bx$: humidity, pressure $y\in\{0,1\}$: dry or rainy
The next word
$\bx$: features of the text so far $y$: next token, one of $K$
Classification starts with an input, a vector of features, and a label from a finite set of classes. We observe training examples whose labels are known. The goal is to use those examples to learn a rule that predicts the label of a new input.For rain tomorrow, the features can be today's humidity and pressure, and the label is zero for a dry day or one for a rainy day. For the next word, the input represents the text so far and the label is one token from a vocabulary of K tokens. The same setup covers both examples.
A probabilistic approach
A purely predictive classifier outputs only a label $\htmlClass{hl-yhat}{\hat y(\bx)}$. A probabilistic classifier estimates $\htmlClass{hl-pyx}{p(y\mid\bx)}$ for each label, then predicts the most probable one.
Probabilities say how sure the prediction is.
Smooth dependence on the parameters lets gradient descent fit the model.
Sampling generates data, such as the text an LLM writes.
Suppose we know $p(y\mid\bx)$. Predict the most probable class:
A purely predictive classifier gives us just a label for each input. A probabilistic classifier gives the probability of each possible label instead. For rain, we need the probability of rain; for the next token, one probability for each token in the vocabulary. We can still predict the most probable label.These probabilities say how sure the prediction is, so a decision can weigh them against its costs. They also give us a way to learn and a way to generate. Smooth dependence on the parameters lets gradient descent fit the model, while the number of mistakes changes in jumps. Sampling from the probabilities generates data: a language model writes text by sampling each next token.Suppose we already know the probability of each class for a given input. (We will learn how to estimate these probabilities later.) The curve on the right shows the probability of rain from afternoon humidity. We predict rain where that probability exceeds one half and no rain otherwise (here this happens at humidity equal to 70%). In general, we pick the class with the highest probability. This rule minimizes the chance of misclassification.
The mind map of the deck. Entering pulls back from the opening slides to the whole map.This lecture has three parts. We start with generative models, which describe inputs and labels together by modeling their joint distribution. Then we turn to discriminative models, which model the label given the input directly. We fit one by maximum likelihood, with gradient descent. Finally, we move beyond two classes to many, including the next-token predictions made by language models.The map zooms into the generative-classifiers branch.Let's start with generative models.
Parametrize $p(y\mid\bx)$ without modeling $p(\bx\mid y)$.
Example: logistic regression.
There are two ways to arrive at the probability of a label given the input. The generative way models how common each class is and how the inputs of each class are spread, and then turns these around with Bayes' rule. Because it describes the inputs themselves, such a model can also generate new data. The discriminative way skips the distribution of the inputs and models the probability of the label directly, as one curve.
Which label is more likely at 70% humidity?
Example (illustrative): $y=1$ for rain tomorrow; $x$ is the afternoon humidity in percent.
Priors: $\pi=p(y=1)=\frac13$ and $p(y=0)=\frac23$.
Joint densities: $\htmlClass{hl-joint}{w_j(x)=p(y=j)\,p(x\mid y=j)=p(x,y=j)}$. By Bayes' rule $\htmlClass{hl-bayes}{p(y=j\mid x)=w_j(x)/p(x)}$, so the class with the larger $w_j(x)$ is the more likely one.
Since $\htmlClass{hl-cmp}{w_0(70)>w_1(70)}$, predict no rain.
We now follow the generative route on one input, the afternoon humidity, to predict rain tomorrow. Its first ingredient is the prior, how common each label is before the humidity is observed. Here dry days are twice as common as rainy ones, as the bars on the right show.The second ingredient is the distribution of the humidity within each class. On both rainy and dry days the humidity is Gaussian with the same spread, but rainy days are centred at a higher humidity. These class-conditional densities, in the middle panel, say which humidities are typical of each label.Multiplying each class-conditional density by its prior gives the joint density of the humidity and that label, in the bottom panel. The dry curve now peaks twice as high as the rainy one, because dry days are twice as common. Comparing the two joint densities at a humidity is enough to decide which label is more likely there. By Bayes' rule, each posterior is its joint density divided by the density of the humidity, and that divisor is the same for both labels.Now evaluate both joint densities at seventy percent humidity, the dashed line in the bottom panel. The dry one is the larger, so the prediction is no rain. The next slide turns the joint densities into the probability of rain at every humidity, the posterior curve.
Normalize the joint densities to get the posterior
For any humidity $x$, the joint densities of the previous slide add up to $\htmlClass{hl-px}{p(x)=w_1(x)+w_0(x)}$, so
with the sigmoid $\htmlClass{hl-sig}{\sigma(u)=\frac{1}{1+e^{-u}}}$.
As $x$ varies, $\sigma\big(\ln(w_1(x)/w_0(x))\big)$ traces the curve in step 4.
For the densities of the previous slide, $\pi=\frac13$ and the Gaussian constants are equal, so
\[\begin{aligned}\ln\frac{w_1(x)}{w_0(x)}&=-\ln2+\ln p(x\mid y=1)-\ln p(x\mid y=0)\\&=-\ln2-\tfrac1{200}(x-80)^2+\tfrac1{200}(x-60)^2\\&=\frac{x-70}{5}-\ln2.\end{aligned}\]
Comparing the joint densities decided between the labels at one humidity. Normalizing them gives the probability of each label at every humidity. The two joint densities add up to the density of the humidity, so dividing the joint density of rain by their sum gives the probability of rain.This probability has a standard form. Divide the top and the bottom by the joint density of rain. What remains depends on the humidity only through the log of the ratio of the two joint densities, and the expression is the sigmoid function of that log ratio. The same form returns for LDA and becomes the model itself in logistic regression.As the humidity varies, the sigmoid of the log ratio traces the posterior curve in step four. It rises with the humidity and crosses one half where the two joint densities cross in step three. For the two Gaussians of this example, the log ratio is a linear function of the humidity, as the Derivation drawer shows.
Gaussian class-conditionals in $d$ dimensions
With $\bx\in\bbR^d$, each class has a Gaussian density $\bX\mid Y=j\sim\cN(\bmu_j,\Sigma_j)$ and prior probability $p(y=j)=\pi_j$.
Illustration: $x_1$ is humidity and $x_2$ is pressure, both rescaled. Rain has higher humidity and lower pressure. Compare two covariance choices:
Previously, we assumed the input was one-dimensional. Here we extend it to a multidimensional input, for example humidity and pressure. In this illustration both features are rescaled. Rainy days have higher humidity and lower pressure. Both classes have positively correlated features, shown by the tilted ellipses. On the left they share the same covariance, including the off-diagonal terms. On the right, the rainy class spreads farther in humidity and less in pressure. Shared versus different covariances will decide whether the classification boundary is linear or curved.
LDA: Gaussian classes with a shared covariance
Suppose $\Sigma_j=\Sigma$ for every class. By Bayes' rule, with the joint density $w_j(\bx)=\pi_j\,p(\bx\mid y=j)$ of slide 7,
For the Gaussian, $\ln p(\bx\mid y=j)=-\tfrac12(\bx-\bmu_j)^T\Sigma^{-1}(\bx-\bmu_j)$ plus a normalizing term that depends only on $\Sigma$. Dropping that term and $-\ln p(\bx)$, which are the same for every class ($\doteq$),
The quadratic term $\bx^T\Sigma^{-1}\bx$ is the same for every class and cancels when log-posteriors are compared. What remains is linear in $\bx$.
Linear discriminant analysis, LDA, assumes every class has the same covariance matrix. Taking logs in Bayes' rule, the log posterior of a class is the log of its joint density minus the log density of the input. The Gaussian's normalizing term and the log density of the input are the same for every class, and dropping them leaves the log prior minus half the squared Mahalanobis distance from the input to that class's mean. Sigma inverse weights that distance according to the shared covariance.Expand that squared Mahalanobis distance. Each log posterior then has a term quadratic in the input, a term linear in the input, and a constant term. Because the covariance is shared, the quadratic term is identical for every class.When the log posteriors of two classes are compared, that shared quadratic term cancels. The remaining difference is linear in the input. The next slide turns this difference into the classification rule and its straight boundary.
LDA: the decision boundary
For two classes, subtracting the log-posteriors of slide 10 gives
Figure: $\Sigma_{11}=\Sigma_{22}=1$ and $\Sigma_{12}=\Sigma_{21}=0.4$.
Compare $p(y=1\mid\bx)$ and $p(y=0\mid\bx)$:
\[\hat y(\bx)=\htmlClass{hl-rule}{\begin{cases}1,&\bbeta^T\bx+\cgamma>0,\\0,&\bbeta^T\bx+\cgamma<0.\end{cases}}\]
Classes tie on $\htmlClass{hl-hyp}{\bbeta^T\bx+\cgamma=0}$, a hyperplane (a line in the plane).For two classes, the difference of the two log posteriors is linear in the input, with coefficients beta and an offset gamma built from the means, the shared covariance and the class probabilities. On the right, both classes have the same tilted spread because their shared covariance has nonzero off-diagonal entries.The boundary between the classes is where the two log posteriors are equal, so where the linear function is zero: a line in this plane. With equal priors it passes through the midpoint between the two means. Its normal is sigma inverse times their difference, so with correlated features the line need not be perpendicular to the segment joining the means.In any number of dimensions, the boundary is the set where one linear function of the input is zero, a hyperplane: a line for two features, a plane for three.
LDA's posterior is logistic
The log-posteriors differ by $\htmlClass{hl-lin}{\bbeta^T\bx+\cgamma}$, so $p(y=0\mid\bx)/p(y=1\mid\bx)=e^{-(\bbeta^T\bx+\cgamma)}$, and the two posteriors add up to 1:
Each probability level is a parallel hyperplane $\bbeta^T\bx+\cgamma=\text{constant}$; the figure marks levels 0.25, 0.5 and 0.75.
Direction: $\bbeta$ is normal to the boundary and points toward the fastest increase in $p(y=1\mid\bx)$.
Position: for fixed $\bbeta$, $\htmlClass{hl-gam}{\cgamma}$ shifts the boundary. Its signed distance from the origin along $\bbeta$ is $\htmlClass{hl-dist}{-\cgamma/\|\bbeta\|}$.
LDA gives more than a boundary: it gives the probability of each class. The difference of the two log posteriors is the linear function of the previous slide, and the two posteriors add up to one. Together these make the posterior the sigmoid of that linear function. On the right, the shading shows the probability of class one, and the dashed lines, its level lines, are the inputs where it equals one quarter and three quarters; they are parallel to the boundary.The vector beta points perpendicular to each probability level line, toward increasing probability of class one. Its direction determines which way the probability rises, and its length controls how quickly it changes across the boundary.For a fixed beta, gamma shifts every level line together. The boundary is where beta dot x plus gamma equals zero; its signed distance from the origin along beta is minus gamma divided by the length of beta.
Explore the LDA posterior
Explore and explain.
1. Vary $\piOne$: how do the joint densities and boundary change?
2. Drag $\muZero$ and $\muOne$: what changes?
3. At $\piOne=0.5$, then $0.6$, vary $s$ and $\rho$ in the shared covariance $\Sigma=s^2\begin{pmatrix}1&\rho\\\rho&1\end{pmatrix}$. How do the boundary and posterior change?
$p(y=0\mid\bx)\propto\piZero p(\bx\mid y=0)$$p(y=1\mid\bx)\propto\piOne p(\bx\mid y=1)$Joint density along the line from $\muZero$ to $\muOne$
Perform these experiments with the figure, and explain what you observe. First, keep the class means, shared spread, and correlation fixed, and vary the prior probability of class one. Watch the decision boundary and the class probabilities as the prior changes.Next, keep the prior and shared covariance fixed. Drag the brown and teal markers for the two means. How do the position and direction of the boundary change? Try moving a mean in more than one direction.Finally, vary the shared standard deviation and the correlation. The correlation tilts both class ellipses and can rotate the boundary, while the joint density curves below change with them. Compare how the boundary moves with a prior of one half and then point six.
QDA, one covariance per class
With one $\htmlClass{hl-sigj}{\Sigma_j}$ per class the quadratic term stays, and the boundary $\ln p(y=1\mid\bx)=\ln p(y=0\mid\bx)$ is of the form
a curve in the plane: quadratic discriminant analysis (QDA).
Alternatively, add the quadratic features $\bphi(\bx)=(x_1,\ldots,x_d,\ \htmlClass{hl-prod}{x_ix_j\text{ for }1\le i\le j\le d})$. A linear rule in $\htmlClass{hl-phi}{\bphi(\bx)}$ has the same form as LDA's rule and gives a quadratic boundary in $\bx$, without fitting a QDA density model.
For general Gaussian covariances, Bayes' rule gives $\ln p(y=j\mid\bx)\doteq\ln\pi_j-\tfrac12\ln|\Sigma_j|-\tfrac12(\bx-\bmu_j)^T\Sigma_j^{-1}(\bx-\bmu_j)$.
Here $|\Sigma_1|=|\Sigma_0|=0.84$ and $\piOne=\piZero=\frac12$, so the log-determinant and prior terms cancel:
\[\begin{aligned}A&=-\tfrac12\big(\Sigma_1^{-1}-\Sigma_0^{-1}\big),\\ \bbeta&=\Sigma_1^{-1}\muOne-\Sigma_0^{-1}\muZero,\\ \cgamma&=-\tfrac12\muOne^T\Sigma_1^{-1}\muOne+\tfrac12\muZero^T\Sigma_0^{-1}\muZero.\end{aligned}\]
Quadratic discriminant analysis, QDA, lets each class keep its own covariance matrix. Then the quadratic terms of the two log posteriors differ, and they no longer cancel. The boundary, where the two log posteriors are equal, is the set where a quadratic function of the input is zero: a curve. Both class ellipses are tilted by their nonzero covariance; class one is also stretched along humidity and squeezed along pressure, and the boundary bends around it.There is another way to get a quadratic boundary. Add the squares and the pairwise products of the input coordinates as features, then fit a linear rule in that expanded feature vector. The rule is linear in the new features but quadratic in the original input.
Maximum likelihood for LDA and naive Bayes
Let $n_j$ be the number of training examples in class $j$. Maximum likelihood gives the class fraction and within-class averages:
Use every entry of $\hat\Sigma$: $d(d+1)/2$ covariance parameters.
Naive Bayes with shared variances
Keep only the diagonal of $\hat\Sigma$. Equivalently, $\hat D_{kk}=\frac1n\sum_i(x_{ik}-\hat\mu_{y_i,k})^2$: the pooled within-class sample variance of feature $k$ (MLE; $d$ parameters).
So far, we have assumed the distribution is given. In reality, we must learn it from data, for which we will use MLE. We fit LDA from labelled examples. The maximum-likelihood class probability is the fraction of training examples in that class, and each class mean is the average of its examples. To estimate one shared covariance, center each observation at its own class mean, form its outer product, and average those products over all n observations. Naive Bayes with shared variances keeps only the diagonal of this estimate, one pooled variance per feature.
Full LDA vs naive Bayes
Both Gaussian classes share a covariance. Naive Bayes ignores correlations between features.
Full LDA: fit the pooled covariance $\hat\Sigma$.
Naive Bayes: use $\hat D$ with $\hat D_{kk}=(\hat\Sigma)_{kk}$ and zero off-diagonal entries, so
Both boundaries are lines. LDA fits $d(d+1)/2$ covariance entries; this shared-variance naive Bayes model fits $d$.
Both fits use the same class means and the same training points.
Full LDA keeps the correlations between input features in the shared covariance matrix. This Gaussian naive Bayes model keeps the same class means and the same individual feature variances, but sets the off-diagonal covariances to zero. That makes the features independent within each class. Both models have straight decision boundaries, yet the fitted lines differ on the same training points. Naive Bayes estimates fewer covariance parameters, while full LDA can use the correlations when the data support them.
Your turn: fewer rainy days
The densities of slide 7, with $\pi=\frac14$.
(a) What is $p(y=1\mid x)$ at 70% humidity, where the two class-conditional densities cross?
(b) Which way does the boundary move, and where is it?
(c) What is $p(y=1\mid x)$ at 80%?
Solve the following problem. The solution is on the next slide.
Your turn: fewer rainy days (a) to (c)
Same class densities as slide 7, with rain prior $\pi=\frac14$.
(a) Rain probability at 70%(b) Decision boundary(c) Rain probability at 80%
Solution
(a) The class-conditional densities are equal at $x=70$, so the posterior equals the prior: $\frac14$.
(b) $\ln\big(w_1(x)/w_0(x)\big)=0.2x-14-\ln 3$ is zero at $x=70+5\ln 3\approx75.49$: the boundary moves toward the rainy mean, to about 75.5%.
Generative models done: the map pulls back and enters discriminative models.That completes the generative classifiers. With Gaussian classes that share a covariance, the posterior is the sigmoid of a linear function of the input.The second part takes that form, makes it the model, and fits it to data.
Exponential-family class-conditionals
With Gaussian class-conditionals that share a covariance, the posterior is the sigmoid of a linear function of $\bx$: \[\htmlClass{hl-gs}{p(y=1\mid\bx)=\sigma(\bbeta^T\bx+\cgamma)}\]
Is this form limited to Gaussian class-conditionals?
No! It holds for the exponential family with the same base term $f(\bx)$:
If the posterior form is the same, why not model that posterior directly?
The first part followed the generative route. With Gaussian class-conditionals that share a covariance, the posterior probability of class one is the sigmoid of a linear function of the input, with coefficients beta and offset gamma. On the right is the one-input example of slide eight: the two joint densities underneath, faint and scaled, and the posterior that Bayes' rule makes of them.Is this form limited to Gaussian class-conditionals?No. It holds for any exponential-family class-conditional whose statistic is the input itself, as long as every class has the same base term f of x. Each class has its own natural parameter, eta j, and its own normalizing term, g j.By Bayes' rule, the log ratio of the posteriors is the log ratio of the class probabilities plus the log ratio of the class-conditionals. The base term cancels, and what remains is linear in the input.So the posterior is again a sigmoid of a linear function. Beta is the difference of the natural parameters, and gamma is the log ratio of the class probabilities plus the difference of the g terms.The posterior has the same form for all of these class-conditionals. So should we model that posterior directly, without the class densities? Think about it before going on.
What each route estimates
Both routes end in $p(y=1\mid\bx)=\sigma(\bbeta^T\bx+\cgamma)$, with $\bx\in\bbR^d$.
Generative (LDA)
Estimate $\piOne$, $\muZero$, $\muOne$, $\Sigma$, then compute $\bbeta$ and $\cgamma$: $\htmlClass{hl-nlda}{1+2d+d(d+1)/2}$ numbers.
Discriminative (logistic regression)
Estimate $\bbeta$ and $\cgamma$ directly: $\htmlClass{hl-nlr}{d+1}$ numbers.
$d$
LDA
logistic regression
2
8
3
10
76
11
100
5,251
101
The posterior, and so the decision, depends on the parameters only through $\bbeta$ and $\cgamma$: logistic regression estimates these $d+1$ numbers directly.
The two routes estimate different things. The generative way, LDA, estimates the class probability, both means and the shared covariance, and then computes beta and gamma from them. The discriminative way, logistic regression, skips the class densities and estimates beta and gamma directly.The two differ in size. The covariance has a number of free entries that grows with the square of the dimension, while logistic regression needs one coefficient per input dimension plus the offset, so the gap widens quickly as the dimension grows. The posterior, and so the decision, uses the parameters only through beta and gamma.
The sigmoid function
\[\sigma(u)=\frac{1}{1+e^{-u}}\]
Its values lie in $\htmlClass{hl-range}{(0,1)}$: $\htmlClass{hl-pos}{e^{-u}>0}$, so the denominator exceeds 1.
The sigmoid increases in $u$: its derivative is $\sigma'(u)=\dfrac{e^{-u}}{(1+e^{-u})^2}>0$.
The sigmoid appeared on both routes, so it is worth knowing well. It takes any real number, which we will call a score, and returns a number that grows with the score. On the right is its graph.The exponential is always positive, so the denominator is always greater than one: every value of the sigmoid lies strictly between zero and one, never reaching either.A score of zero lands exactly in the middle, because the exponential of zero is one.The curve is symmetric about that middle point: flipping the sign of a score turns its value into one minus that value, so a score and its negative give probabilities that add up to one. The two marked points show this for one pair of scores; the Derivation drawer has the algebra.Far from zero in either direction, the curve flattens, close to zero on the left and close to one on the right. So the sigmoid turns any real score into a probability: large negative scores mean near certainty of class zero, and large positive ones near certainty of class one.
With one input $x$, $p_{\btheta}(y=1\mid x)=\frac12$ at the midpoint $\htmlClass{hl-midx}{x=-\cgamma/\cbeta}$.
Example: $\cbeta=1$ and $\cgamma=-1$, so the midpoint is $x=1$.
Question: Double both $\cbeta$ and $\cgamma$: what changes?
The midpoint stays at 1, and the curve is steeper.
With $\htmlClass{hl-g1}{\cgamma=1}$ the midpoint moves to $-1$, with the same steepness.
$\cbeta$$\cgamma$Logistic regression takes the form that LDA arrived at and makes it the model: the probability of class one is the sigmoid of a linear function of the input, with the coefficients and the offset as its parameters. With a single input, the curve crosses one half where the linear part is zero, the midpoint. The curve on the right has beta equal to one and gamma equal to minus one, so its midpoint is at one.Suppose both beta and gamma are doubled. Before going on, predict what happens to the midpoint and to the shape of the curve.Doubling both beta and gamma leaves the midpoint where it was. The linear part now grows twice as fast with x, so the curve rises from near zero to near one over half the distance: it is steeper. The dashed curve is the original.Changing gamma alone moves the curve sideways. With beta back at one and gamma raised from minus one to one, the midpoint moves left and the shape is unchanged; the dashed curve is still the original. So beta alone sets the steepness, and the midpoint, which depends on both, sets where the curve rises.
Odds and log-odds
For a probability $p$, the odds are $\htmlClass{hl-odds}{\dfrac{p}{1-p}}$ and the log-odds are $\htmlClass{hl-lo}{\ln\dfrac{p}{1-p}}$.
probability $p$
0.1
0.25
0.5
0.75
0.9
odds
$1/9$
$1/3$
$1$
$3$
$9$
log-odds
$-2\ln 3$
$-\ln 3$
$0$
$\ln 3$
$2\ln 3$
The log-odds undo the sigmoid: \[\htmlClass{hl-hs}{p=\sigma(u) = \dfrac{1}{1+e^{-u}}}\iff \htmlClass{hl-inv}{\ln \frac{p}{1-p}=u}\]
One more unit of $x_j$, with the other features fixed, adds $\htmlClass{hl-add}{\cbeta_j}$ to the log-odds and multiplies the odds by $\htmlClass{hl-exp}{e^{\cbeta_j}}$.
The inverse of the sigmoid is the log-odds function. The odds of an event is its probability divided by the probability that it does not happen, and the log-odds is the logarithm of the odds. On the right is the log-odds as a function of the probability: negative below one half, zero at one half, and positive above it, without bound toward either end.The table pairs a few probabilities with their odds and log-odds. Probabilities symmetric about one half have reciprocal odds and log-odds of opposite sign, and each step along the row multiplies the odds by the same factor.If a probability is the sigmoid of a score, its log-odds are that score, and the other way round. So the two undo each other: the sigmoid turns a score into a probability, and the log-odds turn the probability back into the score.Applied to the logistic model, the log-odds give back the linear function of the input: logistic regression is the model in which the log-odds are linear in the input. So each coefficient has a direct reading. One more unit of a feature, with the other features fixed, adds its coefficient to the log-odds, which multiplies the odds by the exponential of the coefficient.
Decision boundary and level sets
$p_{\btheta}(y=1\mid\bx)\ge\frac12$ exactly when $\htmlClass{hl-ge}{\bbeta^T\bx+\cgamma\ge0}$: the boundary is the hyperplane $\htmlClass{hl-hyp}{\htmlClass{dk1}{\bbeta^T\bx+\cgamma=0}}$, and $\bbeta$ is perpendicular to it.
Each level set $p_{\btheta}(y=1\mid\bx)=\sigma(u)$ is the parallel hyperplane $\htmlClass{hl-lev}{\bbeta^T\bx+\cgamma=u}$, at signed distance $\htmlClass{hl-dist}{u/\|\bbeta\|}$ from the boundary.
Question: Double both $\bbeta$ and $\cgamma$. What happens to the boundary and the level sets?
Scaling $\bbeta$ and $\cgamma$ by $c$ keeps the boundary and divides each level set's distance from it by $c$.
Illustrative samples, labels drawn from the model.
$c$ (log scale)$c=$
For two points $\bx,\bx'$ on the boundary, $\bbeta^T\bx+\cgamma=\bbeta^T\bx'+\cgamma=0$, so $\bbeta^T(\bx-\bx')=0$: $\bbeta$ is perpendicular to every direction within the boundary.
Write a point as $\bx=\bx_0+d\,\bbeta/\|\bbeta\|$, with $\bx_0$ on the boundary and $d$ its signed distance from it. Then
so the level set $\bbeta^T\bx+\cgamma=u$ lies at $d=u/\|\bbeta\|$.
Replacing $\bbeta,\cgamma$ by $c\bbeta,c\cgamma$ with $c>0$ leaves the boundary $\bbeta^T\bx+\cgamma=0$ unchanged and multiplies $\|\bbeta\|$ by $c$, so each distance $u/\|\bbeta\|$ is divided by $c$.
The solid line marks where the model assigns probability one half to class one. This is the decision boundary: on the side beta points toward, that probability is above one half and we predict class one; on the other side, we predict class zero. In any dimension, the boundary is a hyperplane; here, it is a line.The dashed lines marked 0.25 and 0.75 each join points with the same probability of class one. They are level sets, each a hyperplane parallel to the decision boundary. For a fixed probability sigma of u, the score beta transpose x plus gamma equals u. Its level set is at signed distance u divided by the length of beta from the boundary.Suppose we double both beta and gamma. What happens to the solid decision boundary and the dashed probability level sets?The boundary stays put: doubling both beta and gamma leaves the score zero at exactly the same points. For any fixed probability, the level set's distance from the boundary is inversely proportional to the length of beta. Doubling beta therefore halves that distance. The 0.25 and 0.75 lines move toward the boundary, so probability rises from 0.25 to 0.75 over half as much distance. Try the slider.
Maximum likelihood for logistic regression
Maximum likelihood chooses $\cbeta$ and $\cgamma$ to make the observed labels as probable as possible. Equivalently, minimize the negative log-likelihood:
Let's try it by hand first: Use the sliders to make the negative log-likelihood as small as you can: that $\btheta=(\cbeta,\cgamma)$ is the ML estimate.
At the current $\cbeta$ and $\cgamma$, the negative log-likelihood is .
$\cbeta$$\cgamma$each day's −log-likelihood (smaller is better)To fit beta and gamma by maximum likelihood, choose values that make the observed labels as probable as possible. For our running example, given the humidities, the model treats the thirty-two rain labels as independent, so their probabilities multiply. Minimizing the negative log-likelihood is equivalent; taking the negative log turns that product into the sum on the left. A rainy day contributes the log of one over its predicted probability of rain; a dry day contributes the same expression using its predicted probability of no rain. The curve on the right shows the model's predicted probability of rain at each humidity for the current beta and gamma. Both start at zero, so the curve is flat at one half. The current negative log-likelihood appears above the plot.Use the sliders to make the negative log-likelihood above the plot as small as you can. The beta and gamma that achieve this are the maximum-likelihood estimate. You can see each day's contribution by checking the box under the plot: a longer bar means the model assigned less probability to that day's observed label. The bars add up to the total. Try to minimize it before going on.The revealed curve is the maximum-likelihood fit. At each recorded humidity, the fraction of rainy days gives an empirical rain probability. The sigmoid follows these fractions as closely as it can under the total negative log-likelihood. It does not pass through every fraction, as the next slide makes clear.
KL divergence and cross-entropy
At humidity $x$, $r_x$ rainy days among $n_x$ days give observed probabilities $\hat p(y=1\mid x)=r_x/n_x$ and $\hat p(y=0\mid x)=1-r_x/n_x$. The model predicts $p_{\btheta}(y=1\mid x)=\sigma(\cbeta x+\cgamma)$.
Minimize the KL divergence from observed to model distributions, weighted by the number of days $n_x$:
Observed entropy is fixed by the data, so minimizing KL divergence also minimizes cross-entropy: the negative log-likelihood from slide 26.
Another way to view this fit is to compare the observed and predicted rain probabilities at each recorded humidity. The empirical rain probability is the fraction of rainy days at that humidity. At fifty percent humidity, none of the ten days was rainy, so the observed fraction is zero. The teal points show all five fractions; the purple sigmoid shows the model's prediction at the fitted beta and gamma.Each rain probability defines a Bernoulli distribution over rain and no rain. To compare the observed distribution with the model's distribution, we can minimize their KL divergence, weighting each humidity by its number of days.Cross-entropy equals the entropy of the observed distribution plus its KL divergence from the model. The observed entropy is fixed by the data. So minimizing the weighted KL divergences is equivalent to minimizing the cross-entropy, or negative log-likelihood, from slide twenty-six. The fitted sigmoid does not pass through every observed point, but it gives the smallest weighted KL divergence among the sigmoids this model can produce.
Interpreting the loss function
For each day $i$, use the probability the model assigned to the label that actually occurred, $p_{\btheta}(y_i\mid\bx_i)$. The total loss is
On the plot, $q$ is that assigned probability for one day. A high $q$ gives a small loss; a low $q$ gives a large loss. Maximum likelihood minimizes the total.
At 90% humidity, the fitted model assigns 0.859 to rain and 0.141 to no rain. A rainy day costs 0.152 nats; a dry day costs 1.961.Yet another way to view loss is to take the probability the model assigned to the label that actually occurred for each day, then take the log of its reciprocal. Add these costs to get the loss. Let's assume the observed class is class 1 and q is the probability that the model assignes to class 1. A larger q means the classifier assigned a large probability to the correct class, leading to a smaller cost. On the other hand, as q approaches zero, the model is assigning more probability to class 1, and is confident but wrong, leading to a cost that rises sharply.At ninety percent humidity, the fitted model assigns probability about 0.859 to rain and 0.141 to no rain. If the day was rainy, its cost is about 0.152 nats, which is small. If it was dry, the classifier strongly disagrees with the observation, contributing a cost of about 1.961 nats. The unlikely observed label contributes much more.
Gradient of the negative log-likelihood
Call the negative log-likelihood the loss $\htmlClass{hl-nllx}{\cL(\btheta)=-\ln p_{\btheta}(y_1,\dots,y_n\mid\bx_1,\dots,\bx_n)}$. With $\htmlClass{hl-xt}{\tbx=(\bx,1)}$ and $\htmlClass{hl-th}{\btheta=(\bbeta,\cgamma)}$, so that $p_{\btheta}(y=1\mid\bx)=\sigma(\btheta^T\tbx)$, its gradient is
The equations $\htmlClass{hl-eq}{\sum_i\big(y_i-\htmlClass{hl-sg}{\sigma(\btheta^T\tbx_i)}\big)\tbx_i=\bzero}$ are nonlinear in $\btheta$, so $\hat\btheta$ is computed iteratively, by gradient descent: $\htmlClass{hl-gd}{\btheta^{(t+1)}=\btheta^{(t)}-\rho\,\nabla\cL(\btheta^{(t)})}$.
The sigmoid's derivative is $\sigma'(u)=\dfrac{e^{-u}}{(1+e^{-u})^2}=\sigma(u)\big(1-\sigma(u)\big)$.
Write $p=p_{\btheta}(y=1\mid\bx)=\sigma(\btheta^T\tbx)$. By the chain rule, $\nabla_{\btheta}p=p(1-p)\,\tbx$, so
The error $y-p$: the label minus the predicted probability.
The input $\tbx$, with a 1 in the place of the offset $\cgamma$.
Summing over the days gives $\nabla\cL(\btheta)$.
Call the negative log-likelihood L. To minimize it, we need its gradient with respect to beta and gamma. Append a one to each input and collect beta and gamma in theta; then the score is theta transpose times the extended input. The gradient is a sum of each extended input multiplied by the prediction error, label minus predicted probability, with a minus sign.Compare least-squares with logistic regression. Both gradients have the form error times input. Least squares uses the observed value minus a linear prediction; logistic regression uses the label minus the predicted probability. An example predicted accurately has error near zero, so it makes only a small contribution.Setting this gradient to zero does not give linear equations or a closed-form estimate, because the sigmoid depends on theta. We therefore use gradient descent: each step subtracts the gradient. Equivalently, each day's contribution adds its extended input times its error. For a rainy day assigned probability near zero, the error is near one. Its contribution pushes beta in the direction of that day's input and increases gamma, raising the score and probability of rain there. For a dry day assigned probability near one, the error is near minus one, so its contribution goes the other way.
Gradient descent on the rain data
From $\btheta^{(0)}=\bzero$, take gradient-descent steps with sizes $1/32000$ for $\cbeta$ and $1/320$ for $\cgamma$: one step size, $1/320$, with humidity measured in tens of percent.
Question: Where will the curve settle?
$t$ (log scale)$t=$
It approaches the ML estimate of slide 26, $\htmlClass{hl-rm}{(\cbeta,\cgamma)\approx(0.115,-8.511)}$.
Start with beta and gamma at zero. The curve on the left is flat at one half. Each gradient-descent step updates both parameters; beta's numerical step size is one hundredth of gamma's because humidity is measured in percent. The right panel plots average loss against the number of steps on a logarithmic scale. Where do you expect the curve and loss to settle?After one hundred steps, the probability curve tilts upward, but its slope is still shallow. The loss has fallen only a little from its initial value.After ten times as many steps, the curve steepens and the loss falls further. The pale curves show earlier fits.After another tenfold increase, the curve approaches the maximum-likelihood fit from slide twenty-six. Its midpoint sits near seventy-four percent humidity, and the loss approaches the dashed minimum. The next slide explains why the path takes thousands of steps.
Centering the input
Same 32 days and step sizes as slide 30, with humidity $x$ in percent and $u=x-70$: within 0.001 of the minimum loss per day after 4,483 steps on $x$, after 123 steps on $u$.
On $x$, the fitted midpoint $-\cgamma/\cbeta$ is near 74, so a change in $\cbeta$ requires $\cgamma$ to follow. On $u$, the midpoint is near 4, and the two parameters are less coupled.
The left panel uses humidity in percent. Its loss contours form a narrow tilted valley: changing beta moves the curve's midpoint, so gamma must change with it, and gradient descent advances slowly. The right panel uses humidity centered at seventy percent. Now the midpoint is closer to zero in the new coordinate, so beta and gamma interfere less and the contours are less stretched. With the same step sizes, the centered run gets within 0.001 of the minimum loss per day in one hundred twenty-three steps, versus four thousand four hundred eighty-three on the raw input.
Stochastic gradient descent
Instead of summing the errors from all 32 days, select one day and update the curve from that day's error. The ring marks the selected day; the dashed curve is before its update, and the solid curve is after.
Selected SGD step$t=$
Using centered humidity $u=x-70$, each step adds $(y_i-p_{\btheta}(y=1\mid u_i))u_i/500$ to $\cbeta$ and $(y_i-p_{\btheta}(y=1\mid u_i))/5$ to $\cgamma$.
Stochastic gradient descent updates the curve from one training day at a time. Use the slider to follow the selected day and the change it causes. The right panel shows the loss averaged over all thirty-two days. What will happen when the first selected day is dry at fifty percent humidity?The selected day was dry at fifty percent humidity. The model predicted one half before the update, so its error is negative. The solid curve moves down at that humidity. The dashed curve shows where it was before this one-day update.Now the selected day is rainy at ninety percent humidity. Its positive error raises the curve there. Both beta and gamma change, so the curve also changes at other humidities.This time a dry day at ninety percent humidity pulls the curve down there. Notice that the loss over all days rises after this step: one day can move the fit away from the full-data minimum.Continue through the first thirty-two selected days. The curve generally moves toward the maximum-likelihood fit, but each one-day update can raise or lower the total loss. The second pass uses the same days in a new order.
Logistic regression and LDA compared
Both give $\htmlClass{hl-post}{p(y=1\mid\bx)=\sigma(\bbeta^T\bx+\cgamma)}$.
LDA
Estimates $\piOne$, $\muZero$, $\muOne$, $\Sigma$: 5,251 numbers at $d=100$. When the classes are Gaussian with a shared covariance, it uses that knowledge and estimates the boundary more efficiently.
Logistic regression
Estimates $\bbeta$ and $\cgamma$: 101 numbers at $d=100$. It does not model $p(\bx\mid y)$, so its estimate does not depend on the Gaussian assumption holding.
Efron compares the two: “The efficiency of logistic regression compared to normal discriminant analysis,” JASA, 1975.
Both models give the probability of class one as a sigmoid of a linear score. At one hundred features, LDA estimates five thousand two hundred fifty-one parameters, including two means and a shared covariance. When its Gaussian assumptions hold, it can estimate the decision boundary more efficiently. Logistic regression estimates only one hundred one parameters, beta and gamma, and does not depend on the class inputs being Gaussian.Part 2 done: the map pulls back; the click zooms into part 3.That finishes the two-class part. Logistic regression models the probability of class one directly; maximum likelihood fits beta and gamma by minimizing cross-entropy with gradient descent.Next we give each possible class its own score. Softmax turns those scores into probabilities, taking us from rain or no rain to predicting the next word of a text.
Softmax regression
$K$ classes, $y\in\{1,\dots,K\}$. Each class $k$ has a score $\htmlClass{hl-z}{z_k=\bbeta_k^T\bx+\cgamma_k}$, called its logit, and
So $\ln p(y=k\mid\bx)=z_k-\ln\sum_je^{z_j}$: a logit is a log-probability up to a constant shared by all classes, and $z_k-z_j=\ln\frac{p(y=k\mid\bx)}{p(y=j\mid\bx)}$.
Adding the same constant to every logit leaves $p$ unchanged.
With $K=2$: $p(y=2\mid\bx)=\frac{e^{z_2}}{e^{z_1}+e^{z_2}}=\htmlClass{hl-sd}{\sigma(z_2-z_1)},$
logistic regression with $\htmlClass{hl-bd}{\bbeta=\bbeta_2-\bbeta_1}$ and $\cgamma=\cgamma_2-\cgamma_1$.
$z_1$$z_2$$z_3$For K classes, softmax regression gives each class a linear score, called a logit. In the three-class plot, the bars on the left are the logits; the bars on the right are the resulting probabilities. Softmax exponentiates each logit and divides by their sum, so the probabilities add to one. A single logit is a log probability up to a constant shared by all classes; the difference between two logits is their log-odds. Use the sliders to see how changing a logit shifts the probabilities.Now add the same constant to all three logits. Each exponential gains the same factor, which cancels in softmax. The logit bars move together, but the probabilities remain unchanged.With two classes, divide the softmax numerator and denominator by the exponential of the first logit. The probability of class two becomes the sigmoid of the logit difference. That difference is linear in the input, with beta and gamma equal to the differences between the two classes' coefficients. So two-class softmax is logistic regression.
Softmax regression in two dimensions
Three classes, $\bx\in\bbR^2$: $\bbeta_1=(1,0)$, $\bbeta_2=(0,1)$, $\bbeta_3=\bzero$ and every $\cgamma_k=0$, so the logits are $\htmlClass{hl-lg}{(x_1,x_2,0)}$. Each class is the most probable where its logit is the largest.
The regions are bounded by the lines where two logits tie.
At $\bx=(\ln 3,\ln 2)$ the logits are $(\ln 3,\ln 2,0)$, as on slide 35: probabilities $(1/2,1/3,1/6)$.
Classes $k$ and $j$ tie where $\htmlClass{hl-tie}{(\bbeta_k-\bbeta_j)^T\bx+\cgamma_k-\cgamma_j=0}$, a hyperplane, as for two classes.The logits are x one, x two, and zero. Each shaded region shows where its class has the largest logit. Class three wins where both coordinates are negative.The diagonal marks a tie between classes one and two. The negative coordinate axes mark ties with class three. Only the segments separating winning regions are decision boundaries.At the marked input, exponentiating the logits gives three, two, and one. Softmax assigns probabilities one half, one third, and one sixth, so class one wins.In general, two linear logits tie on a hyperplane. The decision boundary is the part where those classes also beat the others.
With $K=2$, the gradient for class 2 is slide 29's gradient, with $\htmlClass{hl-i2}{I(y_i=2)}$ as the label and $\htmlClass{hl-p2}{p_{\btheta}(2\mid\bx_i)}$ as $p_{\btheta}(y=1\mid\bx_i)$.
The log-probability of class $y$ is $\ln p_{\btheta}(y\mid\bx)=z_y-\ln\sum_je^{z_j}$.
The logit $z_k=\bbeta_k^T\bx+\cgamma_k=(\bbeta_k,\cgamma_k)^T\tbx$ has gradient $\tbx$ with respect to $(\bbeta_k,\cgamma_k)$, and no other logit depends on them.
By the chain rule each example contributes $-\big(I(y_i=k)-p_{\btheta}(k\mid\bx_i)\big)\tbx_i$ to $\nabla_{(\bbeta_k,\cgamma_k)}\cL$.
For K classes, maximum likelihood still minimizes the negative log probability of each example's observed class. The loss is the sum shown here.The two cards show the same pattern: the gradient is minus the sum of error times extended input. For class k, the error is one minus the predicted probability of k when the label is k, and zero minus that probability otherwise. Each class's beta and gamma therefore receive a contribution from every example.When K is two, the probability of class two is the sigmoid of the difference between its logit and class one's. The class-two error and gradient are exactly those of binary logistic regression.
Next-token prediction as classification
A language model predicts the next token $w_t$ from the context $w_{1:t-1}$: a classification with $\htmlClass{hl-kv}{K=V}$ classes, one per token of the vocabulary $\cV$.
As we will see later, a neural network turns the context into a feature vector $\bphi(w_{1:t-1})\in\bbR^m$, and the last step is
with one row of $\mW$ and one entry of $\bb$ per token.
With $V=10{,}000$ and $m=1{,}000$ features, $\mW$ and $\bb$ hold $\htmlClass{hl-count}{10{,}000\times1{,}001=10{,}010{,}000}$ parameters.
Next-token prediction is a classification problem: the possible labels are vocabulary tokens, and the input is the text so far. The text so far is “Tomorrow it will.” A neural network turns this context into a feature vector; we'll see the network's structure later. Its last layer assigns one linear logit to each token; softmax turns the logits into predicted probabilities for the next token.The last layer needs one row of weights and one offset for every vocabulary token. With ten thousand tokens and one thousand features, that is ten million ten thousand parameters in the last layer alone.
The last layer of a language model
With $\bphi$ fixed, fitting $\mW$ and $\bb$ by maximum likelihood is softmax regression with inputs $\bphi$: slide 37's loss and gradient with $\htmlClass{hl-phit}{\tbphi=(\bphi,1)}$ in place of $\tbx$,
with $\mW_k$ the row of token $k$ and $\tbphi_t=\tbphi(w_{1:t-1})$.
Its last layer is softmax regression on features the network learns.Hold the feature vectors fixed. Then fitting the last layer's weights and offsets is softmax regression. The input in the gradient is the feature vector with a one appended for the offset, and the error for token k is its observed indicator minus its predicted probability. We sum those contributions over token positions. In a neural language model, gradient descent learns the features and the last layer together.The diagram highlights the last layer. It maps learned features to one linear logit per token and applies softmax. The same cross-entropy loss trains both that layer and the network producing the features.
Your turn: softmax by hand
“Tomorrow it will ___”: the next word is one of the tokens rain, snow, clear, with logits $\bz=(\ln 2,0,0)$.
(a) The three probabilities.
(b) The loss if the next word is “snow”, in nats and in bits.
(c) The probabilities after adding 3 to every logit.
Suppose we want to predict the next token when the text so far is “Tomorrow it will.” We will focus on turning logits into probabilities. The possible next tokens are rain, snow, and clear, with logits log two, zero, and zero. First find their probabilities. Then find the loss if the observed token is snow, in nats and bits. Finally, add three to every logit and predict what happens to the probabilities. Pause here before the solution.
Your turn: softmax by hand
Logits $\bz=(\ln 2,0,0)$ for rain, snow, clear. (a) The probabilities. (b) The loss for “snow”, in nats and bits. (c) The probabilities after adding 3 to every logit.
Solution
(a) $e^{\bz}=(2,1,1)$, sum 4: probabilities $(0.5,0.25,0.25)$.
(c) $e^{\bz+3}=e^3(2,1,1)$: the same $(0.5,0.25,0.25)$.
Let's work through the three parts.Exponentiating the logits gives two, one, one. They sum to four, so the probabilities are one half for rain and one quarter each for snow and clear.If the observed token is snow, the model gave it probability one quarter. Its loss is log four: about 1.386 nats, or two bits.Adding three to every logit multiplies all three exponentials by the same factor, which cancels in softmax. The probabilities stay one half, one quarter, one quarter.
Perplexity
A language model's training loss is the cross-entropy averaged over the positions of a text (slide 27).
Over a text of $T$ tokens, the average loss, measured in bits per token, is
Perplexity $\htmlClass{hl-ppl}{2^{\bar\ell}}$ is the effective number of equally likely choices with that loss; it need not equal the vocabulary size.
Observing snow costs 2 bits, so its one-token perplexity is 4. The four bars are a hypothetical uniform comparison; the model on the left still has three tokens. A uniform $\htmlClass{hl-v}{10{,}000}$-token model has perplexity 10,000.
At each position in a text, the language model gives a probability to the next token that actually occurred. Its loss is the average log of one over those probabilities, the same cross-entropy used for the rain labels. The bars on the left show the exercise model's probabilities for rain, snow, and clear after “Tomorrow it will.”With base-two logs, average loss is measured in bits per token, and perplexity is two raised to that average. For the observed word snow, the model assigns probability one quarter. Its loss is two bits, and its one-token perplexity is four.The right panel is a hypothetical uniform comparison: four equally likely tokens each have probability one quarter, giving the same two-bit loss. The actual model still has only three tokens. Perplexity is the exponentiated loss, not the number of possible tokens. Over a whole text, we exponentiate average loss; for a uniform ten-thousand-token model, this gives ten thousand.
Temperature
\[p_T(k)=\htmlClass{hl-ptk}{\frac{e^{z_k/T}}{\sum_je^{z_j/T}}},\qquad T>0,\ \text{applied when generating.}\]
$\htmlClass{hl-t1}{T=1}$ is the model. Question: Lower $T$: more spread or less?
$T$
rain
snow
clear
1
0.5
0.25
0.25
$T$
rain
snow
clear
1
0.5
0.25
0.25
1/2
0.667
0.167
0.167
$T$
rain
snow
clear
1
0.5
0.25
0.25
1/2
0.667
0.167
0.167
2
0.414
0.293
0.293
$T<1$ sharpens it, and as $T\to0$ all mass goes to the largest logit.
$T>1$ flattens it, and as $T\to\infty$ it becomes uniform. The order of the tokens never changes.
$T$ (log scale)$T=$ During generation, the model samples the next token from its softmax probabilities. Temperature changes these probabilities by dividing every logit by T before softmax. At T equal to one, we get the original probabilities shown in the table. Predict what happens if we lower T: do the probabilities spread out or concentrate on the leading token?At temperature one half, dividing the logits by one half doubles their differences. Rain, the leading token, rises from 0.5 to about 0.667; snow and clear fall to about 0.167 each. As temperature approaches zero, softmax concentrates on the largest logit.At temperature two, the logit differences shrink. Rain falls to about 0.414, while snow and clear each rise to about 0.293. As temperature grows, the distribution approaches uniform. Since dividing by a positive T preserves the order of logits, temperature changes the probabilities but not their ranking.
Summary
Generative classifiers model $p(\bx\mid y)$ and $p(y)$; LDA and naive Bayes fit class means and different forms of shared covariance.
Exponential-family class-conditionals can yield linear log-odds; logistic regression models that posterior directly.
Conditional maximum likelihood gives the cross-entropy loss. Gradient descent fits it, faster on a centered input.
Softmax regression handles $K$ classes, and a language model's last layer is softmax regression with $K=V$.
Further reading: the notes' Sec. 6.2.4, 6.2.6 and 6.4 (slide 45).
Next: Language models I, starting from Sec. 1.3: the chain rule, n-gram models, smoothing, and perplexity.
Generative classifiers model inputs within each class and use Bayes' rule. Gaussian classes with a shared covariance lead to linear log-odds, which logistic regression models directly. Maximum likelihood minimizes cross-entropy; its gradient has the form error times input, and centering can make gradient descent much faster. For several classes, softmax turns one logit per class into probabilities. A neural language model uses that same softmax regression as its last layer. The next lecture builds language models from the chain rule, n-grams, smoothing, and perplexity.
Further reading
Sec. 6.2.4, more on maximum likelihood for LDA and naive Bayes.
Sec. 6.2.6, Class-conditionals from the exponential family.
Sec. 6.4, Risk minimization and loss functions: the 0-1 loss, the logistic loss as a function of the signed distance, the hinge loss and the SVM.
Optional: Sec. 6.2.5, Generative model for discrete features.
These notes extend four parts of the lecture. Section 6.2.4 works through maximum-likelihood fitting for LDA and naive Bayes. Section 6.2.6 shows when exponential-family class-conditionals produce linear log-odds. Section 6.4 compares zero-one, logistic, and hinge losses and develops support vector machines. Section 6.2.5, on discrete features, is optional.
Discriminative models done: the map pulls back and enters optimization and losses.That completes part two. Starting from the generative classifiers, we arrived at a direct model of the label given the input: the sigmoid of a linear function, whose log-odds are linear in the input. Its decision boundary is a hyperplane, and the length of beta sets how sharply the probability changes across it. We fit it by maximum likelihood, with gradient descent.Part three looks more closely at the loss behind that fit, the cross-entropy, compares it with other losses for a linear classifier, and adds a penalty that keeps the fit finite on separable data.
Separable data
Four days, with humidity centered at 70: $u=x-70=-20,-10,10,20$ and rain $y=0,0,1,1$, with log-odds $\cbeta u+\cgamma$. Then $\htmlClass{hl-neg}{\partial\cL/\partial\cbeta<0}$ at every $\btheta$, so no $\btheta$ has zero gradient.
Data are separable when some $\btheta$ has $\bbeta^T\bx_i+\cgamma>0$ exactly at the class-1 points. They are separable at $\htmlClass{hl-u0}{u=0}$.
Along $\htmlClass{hl-dir}{\btheta=(\cbeta,0)}$:
$\cbeta$
0
0.1
0.2
0.3
0.5
$\cL$
2.773
0.880
0.290
0.102
0.0135
Every term of $\cL$ is positive at every $\btheta$, and $\cL(\cbeta,0)\to0$ as $\cbeta\to\infty$, so no $\btheta$ minimizes $\cL$: the ML estimate does not exist.
With $p_i=p_{\btheta}(y=1\mid u_i)$, the gradient of slide 27 gives $\partial\cL/\partial\cbeta=-\sum_i(y_i-p_i)u_i$.
Every term $(y_i-p_i)u_i$ is positive at every $\btheta$: the days with $u_i<0$ have $y_i=0<p_i$, and the days with $u_i>0$ have $y_i=1>p_i$. So $\partial\cL/\partial\cbeta<0$ everywhere.
Take four days, with the humidity centered at seventy percent: two dry days below it and two rainy days above it. On these days the derivative in beta is negative at every theta, as the Derivation drawer shows, so the gradient is never zero and no theta can be a minimum. Behind this is a special property of the four days: a threshold at the middle humidity puts every dry day on one side and every rainy day on the other. Data like this are called separable.Look at the loss along the direction where gamma is zero. On the right is the loss as a function of beta: it keeps falling as beta grows.It approaches zero but never reaches it, because every day's term stays positive at any finite theta. So no theta minimizes the loss, and the maximum-likelihood estimate does not exist: there is nothing for gradient descent to converge to. It keeps increasing beta, and the next slide follows where it goes.
Gradient descent on separable data
The four days of slide 33, from $\btheta=\bzero$ with step size $\rho=1/300$.
steps $t$
1
$\cbeta$
0.1
steps $t$
1
10
$\cbeta$
0.1
0.235
steps $t$
1
10
100
$\cbeta$
0.1
0.235
0.429
steps $t$
1
10
100
1,000
10,000
$\cbeta$
0.1
0.235
0.429
0.652
0.881
$\cbeta$ grows by about 0.2 for every tenfold increase in the number of steps.
As $\cbeta$ grows, $p_{\btheta}(y=1\mid u)$ tends to 0 and 1 at the four days: a certain prediction from four observations.The previous slide showed that no theta minimizes the loss on the four separable days. Gradient descent can still be run from zero. After the first step, the curve is a gentle sigmoid centred on the boundary between the dry and the rainy days.With more steps, beta keeps growing, though each step adds less than the one before. The curve steepens around the boundary, and the loss keeps falling. The lighter curves on the right are the earlier iterates.Ten times as many steps steepen the curve further: the dry days' probability of rain falls toward zero, the rainy days' rises toward one, and the loss falls with them.The growth slows down but never ends: beta gains roughly the same amount for every tenfold increase in the number of steps, and the curve approaches a step at the boundary.In the limit the model gives the dry days probability zero of rain and the rainy days probability one. That is certainty, claimed from a handful of days, which is not a reasonable conclusion from so little data.
Convexity of the cross-entropy loss
\[\nabla^2\cL(\btheta)=\sum_i\htmlClass{hl-var}{p_i(1-p_i)}\,\htmlClass{hl-outer}{\tbx_i\tbx_i^T},\qquad p_i=p_{\btheta}(y=1\mid\bx_i),\]
\[\bv^T\nabla^2\cL(\btheta)\,\bv=\htmlClass{hl-qf}{\sum_ip_i(1-p_i)\,(\bv^T\tbx_i)^2\ge0}.\]
So $\cL$ is convex: any $\btheta$ with $\nabla\cL=\bzero$ is a global minimum, and gradient descent with a small enough step approaches the minimum value.
On separable data the loss is still convex but has no minimizer (slide 33).
From slide 27, $\nabla\cL(\btheta)=-\sum_i(y_i-p_i)\,\tbx_i$, and $\nabla_{\btheta}p_i=p_i(1-p_i)\,\tbx_i$ (the sigmoid's derivative and the chain rule). The labels $y_i$ do not depend on $\btheta$, so
For any $\bv$, $\bv^T\tbx_i\tbx_i^T\bv=(\bv^T\tbx_i)^2$, and $p_i(1-p_i)>0$.
Gradient descent found the minimum on the rain data because the cross-entropy loss is convex. Differentiating the gradient once more gives the Hessian, a sum of outer products of the inputs, each weighted by the variance of that example's Bernoulli label under the model, which is positive. Any quadratic form in this matrix is a sum of nonnegative terms, so the Hessian is positive semidefinite, and that makes the loss convex. For a convex loss, a point where the gradient vanishes is a global minimum, and gradient descent with a small enough step gets as close to the minimum value as we like. On separable data convexity still holds, but there is no minimum to reach: the loss of the four separable days on the right, along the direction where gamma is zero, keeps decreasing toward zero.
Loss functions for a linear classifier
Conditional maximum likelihood gave cross-entropy. We can use other losses for the same linear score.
Labels $\htmlClass{hl-pm}{y\in\{-1,1\}}$ (the convention of Sec. 6.4) and the prediction $\htmlClass{hl-sign}{\hat y(\bx)=\operatorname{sign}(\bbeta^T\bx+\cgamma)}$.
Each loss is a function of $\htmlClass{hl-m}{y(\bbeta^T\bx+\cgamma)}=\htmlClass{hl-yd}{y\,d\,\|\bbeta\|}$, with $d$ the signed distance of slide 25. It is positive exactly when $\bx$ is classified correctly.
Learning minimizes the average loss over the data,
So far, logistic regression was fitted by maximum likelihood. The notes' section on loss functions takes a second view: choose a loss that says how bad each prediction is, and minimize its average. For this view the two labels are coded as plus one and minus one, so the brown points of the example on the right are now labelled minus one, and the classifier predicts the sign of the linear part. Every loss here depends on a point only through its label times the linear part. That product is the signed distance from the boundary, scaled by the length of beta and multiplied by the label, so it is positive exactly when the point is on its correct side.Learning then means finding the beta and gamma with the smallest average loss over the data. The next slides compare three losses, each a function of that one product.
It counts mistakes; how far a point lies from the boundary does not matter.
Minimizing it is NP-hard in general (Sec. 6.4.1), so differentiable losses are used instead.
The most natural loss counts mistakes: one for a point on the wrong side of the boundary, and zero for a point on its correct side. On the right it is the step, drawn against the label times the linear part. A point just across the boundary costs as much as a point far across it.Minimizing the number of mistakes over all linear classifiers is computationally hard in general; the notes cite the result that it is NP-hard. Its slope is also zero everywhere except at the boundary, so gradient descent has nothing to follow. In practice, differentiable losses, for which efficient algorithms exist, are minimized instead.
It is positive for every point, and smaller the farther a correctly classified point lies from the boundary.
The cross-entropy loss of slide thirty-one is one of these losses. Recode the label zero as minus one. A day with rain costs the log of one over the sigmoid, a dry day costs the log of one over one minus the sigmoid, and both become the same expression in the label times the linear part. On the right, the logistic loss joins the zero-one step; the marked points are the values in the table.Unlike the zero-one loss, it never reaches zero: even a correctly classified point pays a little, and the farther it lies on its correct side, the less it pays. A point on the wrong side pays more the farther away it is. The loss is smooth, so gradient descent can minimize it, as it did on the rain data.
It is 0 once $\htmlClass{hl-ge1}{y(\bbeta^T\bx+\cgamma)\ge1}$: on the correct side, at least $\htmlClass{hl-margin}{1/\|\bbeta\|}$ from the boundary, the margin.
On separable data with small $\clambda$, it picks the separating boundary with the largest margin (Sec. 6.4.3).
The hinge loss charges points on the wrong side, and points on the correct side that lie close to the boundary, and nothing beyond that. Since the product is the distance times the length of beta, the hinge loss is zero for every point at least one over the length of beta from the boundary on its correct side; that distance is called the margin. On the right, the hinge is the straight line that bends to zero.Adding a penalty on the squared length of beta gives the support vector machine. A short beta means a wide margin. On separable data, any boundary that keeps every point outside the margin has zero hinge loss, and among those a small penalty picks the one whose margin is widest. The notes draw this maximum-margin boundary, and mention the kernels that make the method nonlinear.
L2-regularized logistic regression
\[\cL_{\clambda}(\btheta)=\cL(\btheta)+\htmlClass{hl-pen}{\cpr{\clambda\|\bbeta\|^2}},\qquad\clambda>0,\ \htmlClass{hl-gnp}{\cgamma\text{ not penalized}}\]
The logistic loss of slide 38 with the SVM's penalty. With both classes in the data, $\cL_{\clambda}$ has a minimizer, separable data or not.
Four days ($\htmlClass{hl-g0}{\hat\cgamma_{\clambda}=0}$ by symmetry):
$\clambda$
10
25
50
100
$\hat\cbeta_{\clambda}$
0.1847
0.1341
0.1007
0.0715
At $\clambda=50$: $p_{\btheta}(y=1\mid u)=0.732$ at $u=10$ and $0.882$ at $u=20$.
$\clambda/100$ (log scale)$\clambda=$ The same penalty fixes logistic regression on separable data. Add a penalty on the squared length of beta to the cross-entropy loss, as in ridge regression, and leave the offset unpenalized. When both classes appear in the data, this loss grows without bound in every direction of theta: either beta grows and the penalty with it, or gamma grows and one class's probabilities go to zero. So it has a minimizer for every positive lambda, separable data or not, unlike the plain loss on separable data shown earlier. On the right is the regularized fit for the four separable days at the displayed lambda.As lambda grows, the fitted beta shrinks toward zero. The offset stays at zero because the four days are symmetric about the boundary.At a moderate lambda the fit is gentle: the rainy day nearest the boundary gets a probability of rain well below one, and the farther one gets more. The slider under the plot varies lambda.
Bayesian view: L2 regularization as a Gaussian prior
Prior $\htmlClass{hl-prior}{\bbeta\sim\cN(0,\sigma_0^2I)}$, flat in $\cgamma$. Then
so the MAP estimate minimizes $\cL_{\clambda}$ (slide 40) with $\htmlClass{hl-lam}{\clambda=1/(2\sigma_0^2)}$; $\clambda=50$ is $\sigma_0^2=0.01$.
Ridge (Sec. 5.4)
Prior $\cN\big(0,(\sigma^2/\clambda)I\big)$, $\sigma^2$ the noise variance MAP $\htmlClass{hl-ridge}{(X^TX+\clambda I)^{-1}X^T\by}$
Logistic
Prior $\htmlClass{hl-pv}{\cN\big(0,\tfrac{1}{2\clambda}I\big)}$ MAP by gradient descent on $\cL+\clambda\|\bbeta\|^2$
The likelihood increases forever in $\cbeta$; the prior's tail brings the posterior back down, so the posterior has a mode.
The four days, $\cgamma=0$, $\clambda=50$.
Regularization also has a Bayesian reading, as ridge regression did in chapter five. Put a Gaussian prior with mean zero on beta and a flat prior on gamma. The negative logarithm of the posterior is then the cross-entropy loss plus the prior's quadratic penalty, so the most probable theta, the MAP estimate, minimizes the regularized loss, with lambda set by the prior variance. On the right, for the four days, are the prior and the likelihood as functions of beta, with gamma at zero: the prior is a density in beta, and the likelihood is the probability of the four labels. The density curves are divided by ten to share a readable vertical scale with the likelihood.The correspondence is the same as for ridge regression: a smaller prior variance means a larger lambda and a stronger pull toward zero. The difference is computational. Ridge has a closed-form MAP estimate, while logistic regression needs gradient descent, as the unregularized fit did.The picture shows why the regularized estimate exists. The likelihood keeps increasing in beta and never peaks, which is why maximum likelihood fails. Multiplying by the prior, whose tail falls off quickly, brings the product back down, so the posterior, a density in beta like the prior, has a mode, at the value of beta that the regularized fit found.
