ECE 6501/4502, CS 6501: Probabilistic Machine Learning (Fall 2025)

Welcome! In this course, we’ll study estimation and machine learning from a probabilistic point of view.

Why a probabilistic view?

Information and uncertainty, which underlie both statistical estimation and machine learning, can be represented via probability in a robust and versatile way. Unknown quantities can be cast as random variables and their relationships to each other and to known information as joint distributions. This provides a unifying framework for setting up estimation and machine learning problems, where we can state our assumptions clearly, design methods, and evaluate performance.

What topics will we study?

We will start with estimation, which can be defined as the problem of learning about the world from data (e.g., finding the chance of getting a disease given one’s genetic make-up) or drawing conclusions about relationships (e.g., what are the best predictors of academic success?). We will then learn about machine learning problems such as regression and classification, where the goal is to predict an unknown quantity, e.g., the price of a house, based on some relevant information. We will also learn how to deal with situations when part of the data is missing. Finally, we will discuss computational methods, which help tackle difficult problems via approximation.

Course objectives:

1. Use joint distributions and graphical models to describe relationships between known and unknown quantities
2. Describe, identify, and apply frequentist and Bayesian estimation methods
3. Construct and apply learning models
4. Apply computational methods such as expectation-maximization and Monte Carlo sampling
5. Perform approximate inference using variational methods
6. Quantify fundamental limits on estimation and learning given available data

Pre-requisites:

• Fluency in basic probability (e.g., APMA 3100) is needed for the course. You should be comfortable with 70-80% of Chapter 0. In the second week of class, you will be asked to pass a test of basic probability with at least 70%. You will have the chance to retake the test if you score below 70%.

  • Example Probability Review Test

  You can also refer to these pages I developed for a different course to review probability, although these don’t cover everything we’ll need:

  • Basics
  • Random variables, distributions, and independence
  • Expectation and variance
  • Conditional probability
  • And here are some other resources:
    • An online textbook for probability theory: https://www.probabilitycourse.com
    • Random Processes for Engineers by Bruce Hajek
• Familiarity with linear algebra.
• The programming exercises are based on Python.

What will help you excel:

The most important factor is remaining engaged in the class. In particular, office hours are often underutilized. Ask questions in class when they arise; not doing so can prevent you from following the lecture and understanding the subsequent material as well. Reach out to the instructors and the TA when you need help.

Note to Undergraduate students: You do not need instructor permission to enroll in this course. But fluency in probability is an important prereq and if your foundation in probability is not strong, you will not be able to fully benefit from the course.

Activities and Grading Scheme:

Tentative Grading Scheme:

HW/Labs = 50%; Quizzes/In-class activities = 20%; Exam = 20%; Project = 10%

Comments:

• The lowest grade from the HW/Labs category and the lowest grade from the Quizzes/ICAs category will be dropped.
• Late homework submissions will incur a penalty of 10% per day, for up to two days.
• Collaboration on homework is allowed, but you must list your collaborators and write up your solutions independently.
• The exam will be take-home, given toward the end of the semester. Collaboration or discussion of problems are not permitted.
• The goal of the group project is to enhance your understanding of the material covered in the class and cultivate your own perspectives towards data, estimation, and machine learning. The project can be a focused study of specific topic; or an application of the methods we study in the course in an interesting way. More information about the project is available in a separate document.
• Bonus points may be awarded for consistent class participation, exceptional work, and other contributions to the class.

Course Notes:

Links to the complete notes and individual chapters are listed below and are updated as needed, with the chapter notes being possibly more up-to-date. To ensure you always access the latest versions, please view them via the links below rather than downloading local copies.

Estimation and Probabilistic Learning

0. Review of Probability
1. Probability, Inference, and Learning
2. Frequentist Parameter Estimation
3. Bayesian Parameter Estimation
4. Multivariate random variables
5. Linear Regression
6. Linear Classification
7. Expectation-Maximization
8. Basics of Graphical Models
9. Independence in Graphical Models
10. Parameter Estimation in Graphical Models
11. Inference in Graphical Models
12. Inference in Hidden Markov Models
13. Factor Graphs and Sum/Max-product Algorithms
14. Markov Chains
15. Sampling Methods
16. Variational Inference
17. Appendix

Textbooks and other resources:

The main resources are lectures and the pdf notes posted on this page. But you may find the following useful:

1. Probabilistic Machine Learning: An introduction by Kevin P. Murphy, 2022.
2. Deep Learning by Ian Goodfellow et al, 2015.
3. Information Theory, Inference, and Learning Algorithms by David MacKay, 2003.
4. Pattern Recognition and Machine Learning by Christopher M. Bishop, 2006.
5. Probabilistic Graphical Models: Principles and Techniques by Daphne Koller and Nir Friedman, 2009.

---