Machine Learning Fall 2007

Mon., Sep. 10:

• Lecture: What's ML, Point estimation [Slides] [Annotated]
• Mathematica Demonstration The Mathematica demonstrations require the newest version of Mathematica (Version 6) which can be obtained from MyAndrew.
• Additional Reference: Andrew Moore's basic probability tutorial
• Readings: Bishop 2.1, Appendix B

Wed., Sep. 12:

• Lecture: Gaussians, Linear Regression, Bias-Variance Tradeoff, Overfitting, What's ML revisited. [Slides] [Annotated]
• Readings: Bishop 1.1 to 1.4, Bishop 3.1, 3.1.1, 3.1.4, 3.1.5, 3.2, 3.3, 3.3.1, 3.3.2
• Completely Optional: Joey's quickly written notes on the matrix MLE for regression. [PDF] [Mathematica6 Notebook] If there are any typos or mistakes please let me know .

Mon., Sep 17:

• Lecture (Eric Xing): Naive Bayes, Gaussian Naive Bayes [Slides] [Annotated]
• Readings: Bishop 1.3, 1.5, 3.2, Mitchell's Chapter on Naive Bayes and Logistic Regression (Sections 1 and 2)

Wed., Sep 19:

• Lecture: Overfitting, What's learning revisited, Generative v. Discriminative, Logistic Regression [Slides] [Annotated]
• Required Reading: Mitchell's Chapter on Naive Bayes and Logistic Regression (All sections)
• Optional Reading: Ng and Jordan's NIPS 2001 paper on Discriminative versus Generative Learning [pdf] [ps]

Mon., Sep 24:

• Lecture: Logistic Regression [Slides] [Annotated]
• Readings: Bishop - 4.0, 4.2, 4.3, 4.4, 4.5

Wed., Sep. 26:

• Lecture: Decision Trees [Slides] [Annotated]
• Readings: (Bishop - 1.6) Information Theory
• (Bishop - 14.4) Tree-based Models
• Recommended Reading: Quantities of Information Wikipedia entry
• Recommended Reading: Nils Nilsson's Chapter (All Sections): Decision Trees
• Optional Review of Boolean Logic/DNF: Nils Nilsson's Chapter Boolean Functions (first 4 pages)

Mon., Oct. 1:

• Lecture: Boosting [Slides] [Annotated]
• Readings: (Bishop 14.3) Boosting
• Schapire Boosting Tutorial
• Optional Reading: Multi-class AdaBoost paper, by Zhu, Rosset, Zou, and Hastie.
• Additional resource: Schapire Boosting Tutorial Video.

Wed., Oct. 3:

• Homework 1 is due at the beginning of lecture.

• Lecture: Cross Validation, Simple Model Selection, Regularization, MDL, Neural Nets [Slides] [Annotated]
• Readings: (Bishop 1.3) Model Selection / Cross Validation
• (Bishop 3.1.4) Regularized least squares
• (Bishop 5.1) Feed-forward Network Functions
• Optional Reading: Ron Kohavi's paper, A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection.
• Additional Resource: Minimum Description Length website

Mon., Oct. 8:

• Lecture: Neural Nets [Slides]
• Readings: (Bishop 5.1) Feed-forward Network Functions
• (Bishop 5.2) Network Training
• (Bishop 5.3) Error Backpropagation

Wed., Oct. 10:

• Lecture: Neural Nets (cont), Instance-based Learning [Slides] [Annotated]
• Readings: (Bishop 2.5) Nonparametric Methods

Mon., Oct. 15:

• Lecture: SVMs [Slides] [Annotated]
• Readings: (Bishop 6.1,6.2) Kernels
• (Bishop 7.1) Maximum Margin Classifiers
• Hearst 1998: High Level Presentation
• Burges 1998: Detailed Tutorial
• (Optional) Platt 1998: Training SVMs with Sequential Minimal Optimization
• Additional Resource: Smola video tutorial on SVM (see Part 3)
• Additional Resource: Scholkopf video tutorial on kernels
• Additional Resource: http://www.svms.org

Wed., Oct. 17:

• Lecture: SVMs - The Kernel Trick [Slides] [Annotated]
• Additional Resource: http://www.kernel-machines.org

Mon., Oct. 22:

• Lecture: Learning Theory [Slides] [Annotated]
• Readings:Goldman's COLT survey, sections 1-3.1
• Avrim Blum's course handout on tail inequalities
• (Optional) John Langford's tutorial on generalization bounds
• (Optional) Littlestone's original (excellent) paper on the Mistake Bound model: Learning Quickly When Irrelevant Attributes Abound: A New Linear-Threshold Algorithm
• Additional Resource: Langford video tutorial on generalization bounds
• Additional Resource: John Shawe-Taylor video tutorial on statistical learning theory
• Additional Resource: http://www.learningtheory.org

Wed., Oct. 24:

• Lecture: Learning Theory, Midterm review [Slides] [Annotated]

Mon., Oct. 29:

• Lecture: Bayes nets - Representation [Slides] [Annotated]
• Readings: (Bishop 8.1,8.2) Bayesian Networks

Wed., Oct. 31:

• Lecture: Bayes nets - Representation (cont.), Inference [Slides] [Annotated]
• Readings: (Bishop 8.1,8.2) Bayesian Networks

Mon., Nov. 5:

• Lecture: BNs inference, HMMs [Slides] [Annotated]
• Readings: (Bishop 8.4.1,8.4.2) - Inference in Chain/Tree Structures
• Rabiner's Detailed HMMs Tutorial

Wed., Nov. 7:

• Lecture: HMMs, Bayes Nets - Structure Learning [Slides] [Annotated]
• Readings: Additional Reading: Heckerman BN Learning Tutorial
• Additional Reading: Tree-Augmented Naive Bayes paper

Mon., Nov. 12:

• Lecture: BNs Structure learning, Clustering - K-means [Slides] [Annotated]
• Readings: (Bishop 9.1, 9.2) - K-means, Mixtures of Gaussian

Wed., Nov. 14:

• Guest Lecture: Online Learning (Avrim Blum) [Slides]

Mon., Nov. 19:

• Lecture: EM [Slides] [Annotated]
• Readings: (Bishop 9.3, 9.4) - EM
• Neal and Hinton EM paper
• Ghahramani, "An introduction to HMMs and Bayesian Networks"

Mon., Nov. 26:

• Lecture: EM (cont.) and Principal Component Analysis (PCA) [Slides] [Annotated]
• Readings: Shlens' PCA tutorial
• Optional reading: Wall et al. 2003 - PCA for gene expression data

Wed., Nov. 28:

• Lecture: Markov Decision Processes (MDPs) [Slides] [Annotated]
• Readings: Kaelbling et al. Reinforcement Learning tutorial