Probabilistic Machine Learning

Course notes

Links to the complete notes and individual chapters are listed below and are updated as needed, with the chapter notes being possibly more up-to-date. To ensure you always access the latest versions, please view them via the links below rather than downloading local copies.

Complete notes: Estimation and Probabilistic Learning PDF

Estimation

Regression, classification, and missing data

Graphical models

Computational methods

Generative AI

Why a probabilistic view?

Information and uncertainty, which underlie both statistical estimation and machine learning, can be represented via probability in a robust and versatile way. Unknown quantities can be cast as random variables and their relationships to each other and to known information as joint distributions. This provides a unifying framework for setting up estimation and machine learning problems, where we can state our assumptions clearly, design methods, and evaluate performance.

What topics will we study?

We will start with estimation, which can be defined as the problem of learning about the world from data (e.g., finding the chance of getting a disease given one’s genetic make-up) or drawing conclusions about relationships (e.g., what are the best predictors of academic success?). We will then learn about machine learning problems such as regression and classification, where the goal is to predict an unknown quantity, e.g., the price of a house, based on some relevant information. We will also learn how to deal with situations when part of the data is missing. Finally, we will discuss computational methods, which help tackle difficult problems via approximation.

Pre-requisites

• Fluency in basic probability (e.g., APMA 3100) is needed for the course. You should be comfortable with 70-80% of Chapter 0. In the second week of class, you will be asked to pass a test of basic probability with at least 70%. You will have the chance to retake the test if you score below 70%.

  • Example Probability Review Test

  You can also refer to these pages I developed for a different course to review probability, although these don’t cover everything we’ll need:

  • Basics
  • Random variables, distributions, and independence
  • Expectation and variance
  • Conditional probability
  • And here are some other resources:
    • An online textbook for probability theory: https://www.probabilitycourse.com
    • Random Processes for Engineers by Bruce Hajek
• Familiarity with linear algebra.
• The programming exercises are based on Python.

Textbooks and other resources

The main resources are lectures and the pdf notes posted on this page. But you may find the following useful:

1. Probabilistic Machine Learning: An introduction by Kevin P. Murphy, 2022.
2. Deep Learning by Ian Goodfellow et al, 2015.
3. Information Theory, Inference, and Learning Algorithms by David MacKay, 2003.
4. Pattern Recognition and Machine Learning by Christopher M. Bishop, 2006.
5. Probabilistic Graphical Models: Principles and Techniques by Daphne Koller and Nir Friedman, 2009.

Previous offerings

• Fall 2025