Machine Learning, 10-701 and 15-781, 2005

Carlos Guestrin and Tom Mitchell
School of Computer Science, Carnegie Mellon University

Spring 2005 (Starting Monday Jan. 10th)

Class lectures: Mondays & Wednesdays from 1:30-2:50 in Wean Hall 7500

Review sessions: Thursdays 5:00 in Doherty Hall 1212



It is hard to imagine anything more fascinating than automated systems that improve their own performance. The study of learning from data is commercially and scientifically important. This course is designed to give a graduate-level student a thorough grounding in the methodologies, technologies, mathematics and algorithms currently needed by people who do research in learning and data mining or who may need to apply learning or data mining techniques to a target problem. The topics of the course draw from classical statistics, from machine learning, from data mining, from Bayesian statistics and from statistical algorithmics.

Students entering the class should have a pre-existing working knowledge of probability, statistics and algorithms, though the class has been designed to allow students with a strong numerate background to catch up and fully participate.


Page links


Instructors


Teaching Assistants

In order to facilitate the interaction between students and TAs in this large class, we have split the class into four groups, and assigned each group to one of the four TAs. This TA is your "first point of contact" for course-related issues. Please send any questions, concerns, etc. to your point of contact:
Last names starting with A-D: contact Daniel (neill@cs)
Last names starting with E-Le: contact Kaustav (kaustav@cs)
Last names starting with Li-P: contact Derek (dhoiem@cs)
Last names starting with Q-Z: contact Zhenzhen (woomy@cs)


Class Assistants

Textbooks


Course Website (this page)


Grading


Collaboration policy

Homeworks will be done individually: each student must hand in their own answers. It is acceptable, however, for students to collaborate in figuring out answers and helping each other solve the problems. We will be assuming that, as participants in a graduate course, you will be taking the responsibility to make sure you personally understand the solution to any work arising from such collaboration. You also must indicate on each homework with whom you collaborated. The final project may be completed by small teams.


Late homework policy


Homework regrades policy

If you feel that we have made an error in grading your homework, please turn in your homework with a written explanation to Sharon, and we will consider your request. Please note that regrading of a homework may cause your grade to go up or down.


Homework assignments

Final project

For project milestone, roughly half of the project work should be completed. A short, graded write-up will be required, and we will provide feedback.

Lecture schedule (and online slides when available)


Module

Material covered

Online material and links

Dates and Instructor
Module 1: Basics
(2 Lectures)
  • What is learning?
    • Version spaces
    • Sample complexity
    • Training set/test set split
  • Point estimation
    • Loss functions
    • MLE
    • Bayesian
    • MAP
    • Bias-Variance trade off

Jan 10:

Introduction

Function Approximation and Version Spaces

Jan 12:

Point Estimation& Linear Regression

Link to the regression applet: http://www.mste.uiuc.edu/users/exner/java.f/leastsquares/

 Jan 10: Tom Mitchell
Jan 12: Carlos Guestrin
Module 2: Linear models
(2 Lectures)
  • Naive Bayes
  • Logistic regression
  • Discriminative v. Generative models
 Jan 19:
Naive-Bayes Classifiers

Jan 24:
Logistic Regression

Draft Chapter on Naive-Bayes and Logistic Regression
Jan 17: MLK Day, no classes
Jan 19: Tom Mitchell
Jan 24: Tom Mitchell
 Module 3: Non-linear models
Model selection
(5 Lectures)
  • Neural Nets
  • Overfitting
  • Instance-based learning
    • K-nearest neighbors
    • Kernels
  • Decision trees
  • Boosting
  • MDL
Jan 26:
Logistic Regression and Bias/Variance

Jan 31:
Neural Networks

Feb 2:
Neural Networks/Cross Validation

Feb 9:
Decision Trees/MDL/Boosting
Boosting Paper
Adaboost Applet

Feb 14:
Instance-Based Learning
KNN Applet

Jan 26: Tom Mitchell
Jan 31: Tom Mitchell
Feb 2: Carlos Guestrin
Feb 7: Mid-mini break - no classes
Feb 9: Tom  Mitchell
Feb 14: Carlos Guestrin 
Module 4: Margin-based approaches
(2 Lectures)
  • SVMs
  • Kernel trick
Feb 16:
SVM slides
SVM Applets
Hearst 1998: High Level Presentation
Burges 1998: Detailed Tutorial
Burges (cleaner ps version)

Feb 21:
Kernels for SVM slides
SVM Applets


Feb 16: Carlos Guestrin
Feb 21: Carlos Guestrin
Module 5: Learning theory
(3 Lectures)
  • Sample complexity
  • PAC learning
  • Error bounds
  • VC-dimension
  • Margin-based bounds
  • Large-deviation bounds
    • Hoeffding's inequality, Chernoff bound
  • Mistake bounds
  • No Free Lunch theorem
 Feb 23:
PAC learning

Feb 28:
PAC/VC Slides
Learning with Kernels
Example of More Complex Bounds (Zhang ML 2002)

Mar 2:
Mistake bounds
Mid-Review Comments
Feb 23: Tom Mitchell
Feb 28: Carlos Guestrin
Mar 2: Tom and Carlos

Spring break

March 7-11

Mid-term Exam

All material thus far

March 14
Module 6: Structured models
(4 Lectures)
  • HMMs
    • Forwards-Backwards
    • Viterbi
    • Supervised learning
  • Graphical Models
    • Representation
    • Inference
    • Learning
    • BIC

Mar 16: Mar 21:

Mar 23:

 

Mar 28:

 

 Mar 16: Carlos Guestrin
Mar 21: Carlos Guestrin
Mar 23: Carlos Guestrin
Mar 28: Carlos Guestrin
Module 7: Unsupervised
and  semi-supervised learning
(4 Lectures)
  • Expectation Maximization (EM)
    • for training Bayes nets
    • for training HMMs
    • mixture of Gaussians
  • K-means
  • Combining labeled and unlabeled data
    • EM
    • reweighting labeled data
    • Co-training
    • unlabeled data and model selection
  • Dimensionality reduction
    • PCA, SVD

Mar 30:

 Apr 4:

 Apr 6:
 Apr 11:
 Mar 30: Tom Mitchell
Apr 4: Tom Mitchell
Apr 6: Tom Mitchell
Apr 11: Tom Mitchell
Module 8: Invited lectures in advanced topics
(5 Lectures)
  • Combining graphical models and margin-based approaches
  • Markov decision processes
  • Reinforcement learning
  • Tackling very large datasets
  • Active learning
  • Overview of follow-up classes
Apr 13: Apr 18:
Apr 20:
 Apr 25:
Apr 27:
 Apr 13: Carlos Guestrin
Apr 18: Ron Parr
Apr 20: Ron Parr
Apr 25: Daniel Neill
Apr 27: Tom Mitchell

Final Exam

All material thus far

May 9, 1:00-4:00pm

Review session schedule


Date
Time
Place
Instructor
Topic
Tue Jan. 11
5:00pm ~ 6:30pm
NSH 3305
Daniel
Review of Basic Probability Concepts
Thurs Jan. 20
5:00pm ~ 6:30pm
Doherty 1212
Zhenzhen
Thurs Jan. 27
5:00pm ~ 6:30pm
Doherty 1212
Kaustav
Naive Bayes and Logistic Regression
Thurs Feb. 3
5:00pm ~ 6:30pm
Doherty 1212
Derek
Thurs Feb. 10
5:00pm ~ 6:30pm
Doherty 1212
Daniel
Decision Trees / Boosting
Thurs Feb. 17
5:00pm ~ 6:30pm
Doherty 1212
Derek
HW2 / KNN / Exam Review (Mid 2003)
Thurs Feb. 24
5:00pm ~ 6:30pm
Doherty 1212
Kaustav
SVM
Thurs Mar. 3
5:00pm ~ 6:30pm
Doherty 1212
Zhenzhen
PAC Learning / Midterm Review
Thurs Mar. 10
5:00pm ~ 6:30pm
Doherty 1212
Spring Break
Spring Break
Thurs Mar. 17
5:00pm ~ 6:30pm
Doherty 1212
Zhenzhen
Thurs Mar. 24
5:00pm ~ 6:30pm
Doherty 1212
TBD
TBD
Thurs Mar. 31
5:00pm ~ 6:30pm
Doherty 1212
TBD
TBD
Thurs Apr. 7
5:00pm ~ 6:30pm
Doherty 1212
TBD
TBD
Thurs Apr. 14
5:00pm ~ 6:30pm
Doherty 1212
TBD
TBD
Thurs Apr. 21
5:00pm ~ 6:30pm
Doherty 1212
Zhenzhen
Thurs Apr. 28
5:00pm ~ 6:30pm
Doherty 1212
TBD
TBD


Exam Schedule


Additional Resources

Here are some example questions here for studying for the midterm/final. Note that these are exams from earlier years, and contain some topics that will not appear in this year's final. And some topics will appear this year that do not appear in the following examples.


Note to people outside CMU

Feel free to use the slides and materials available online here. Please email the instructors with any corrections or improvements. Additional slides and software are available at the Machine Learning textbook homepage and at Andrew Moore's tutorials page.