10701/15781 Machine Learning

Syllabus and (tentative) Course Schedule

Date	Lecture	Topics	Readings and useful links	Handouts
Module 1
Intro to Functional Approximation
Mon 9/12	Lecture 1: Overview. Slides, (Annotated Slides)	Overview of Machine Learning What is Machine Learning? Elements of machine learning Decision tree learning	Mitchell: Chap 1,3 Decision Tree Learning [Applet] Leo Breiman (2001). Random Forests. Machine Learning Volume 45, Number 1.
Wed 9/14	Lecture 2: Nonparametric methods. Slides, (Annotated Slides)	Non parametric learning methods Univariate prediction without using a model: good or bad? instance based "learning": the Bayes optimal classifier. spectrum graph methods	Keinosuke Fukunaga (1990), Hypothesis Testing. In Introduction to Statistical Pattern Recognition (pp. 51-58). Academic Press Prof., Inc., San Diego, CA, USA. Keinosuke Fukunaga (1990), Nonparametric Density Estimation. In Introduction to Statistical Pattern Recognition (pp. 254-276). Academic Press Prof., Inc., San Diego, CA, USA. Keinosuke Fukunaga (1990), Nonparametric classification and error estimation. In Introduction to Statistical Pattern Recognition (pp. 300-309). Academic Press Prof., Inc., San Diego, CA, USA. Ting Liu, Charles Rosenberg, Henry A. Rowley (2007), Clustering Billions of Images with Large Scale Nearest Neighbor Search, wacv, pp.28, Eighth IEEE Workshop on Applications of Computer Vision (WACV'07) Herwig Lejsek, Bjorn Por Jonsson, and Laurent Amsaleg (2011). NV-Tree: nearest neighbors at the billion scale. In Proceedings of the 1st ACM International Conference on Multimedia Retrieval (ICMR '11). ACM, New York, NY, USA, , Article 54 , 8 pages. Andrew Y. Ng, Michael I. Jordan, Yair Weiss. On Spectral Clustering: Analysis and an algorithm. In Proceedings of NIPS'2001. pp.849-856 Jianbo Shi and Jitendra Malik (2000). Normalized Cuts and Image Segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 22, 8 (August 2000), 888-905.
Preliminaries: Learning linear separation functions
Mon 9/19	Lecture 3: Generative versus discriminative classifers. Slides , (Annotated Slides)	Naive Bayes vs Logistic Regression Design of an experiment	Andrew Ng and Michael Jordan (2002). On discriminative vs. generative classifiers: A comparison of Logistic Regression and Naive Bayes. Neural Information Processing Systems 2002.	HW 1 out
Wed 9/21	Lecture 4: Linear regression and sparsity. Slides (Annotated Slides)	Linear regression Introduction to sparsity and structure.	Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. J. Royal. Statist. Soc B., Vol. 58, No. 1, pages 267-288. Tibshirani, R., Saunders, M., Rosset, S., Zhu, J. and Knight, K. (2005), Sparsity and smoothness via the fused lasso. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 67: 91-108. Yuan, M. and Lin, Y. Model selection and estimation in regression with grouped variables. J. Royal. Statist. Soc B., Vol. 68, No. 1, pages 49-67.
Into the nonlinear world and theoretical foundations of supervised learning
Mon 9/26	Lecture 5: Neural Networks Slides (Annotated Slides)	Neural networks and deep learning	Hinton, G. E., Osindero, S. and Teh, Y. (2006). A fast learning algorithm for deep belief nets. Neural Computation, 18, pp 1527-1554. Mohamed, A., Dahl, G. and Hinton, G. E. (2011). Acoustic Modeling using Deep Belief Networks. IEEE Trans. on Audio, Speech, and Language Processing. Susskind,J., Memisevic, R., Hinton, G. and Pollefeys, M. (2011). Modeling the joint density of two images under a variety of transformations.. IEEE Conference on Computer Vision and Pattern Recognition.
Wed 9/28	Lecture 6: Computational Learning Theory Slides (Annotated Slides)	Computational and Learning theory PAC learning VC dimensions Structural risk minimization	Vapnik, V. (1999) An Overview of Statistical Learning Theory, IEEE transactions on Neural Networks, Vol. 10, pp. 988-99.
Mon 10/3	Lecture 7: Overfitting and model selection Slides (Annotated Slides)	Overfitting and model selection	Burges, Christopher(1998). A Tutorial on Support Vector Machines for Pattern Recognition. Data Min. Knowl. Discov. 2(2): 121-167 [Section 1 and 2] E.P. Xing, M.I. Jordan and R.M. Karp (2001), Feature selection for high-dimensional genomic microarray data, Proceedings of the Eighteenth International Conference on Machine Learning. Andrew Y Ng. On Feature Selection: Learning with Exponentially many Irrelevant Features as Training Examples. In Proceedings of the Fifteenth International Conference on Machine Learning, 1998.	HW 1 due, HW 2 and data out
Unsupervised learning: Clustering
Wed 10/5	Lecture 8: Clustering Slides (Annotated Slides)	Clustering K-means Distance metrics	Arthur, D. and Vassilvitskii, S. (2007). k-means++: The advantages of careful seeding. 2007 ACM-SIAM symposium on Discrete algorithms (SODA). pp 1027-1035. E.P. Xing, A.Y. Ng, M.I. Jordan and S. Russell, Distance Metric Learning, with application to Clustering with side-information, Advances in Neural Information Processing Systems 16 (NIPS2002), (eds. Becker et al.) MIT Press, 521-528, 2002.
Mon 10/10	Lecture 9: Expectation Maximization Slides (Annotated Slides)	Probabilistic models for clustering: Expectation-maximization	Bilmes, Jeff. (1998). A gentle tutorial of the EM algorithm and its application to parameter estimation for Gaussian mixture and hidden Markov models. TR-97-021, Computer Science Division, U.C. Berkeley.
Wed 10/12	Lecture 10: Infinite Mixture Models Slides (Annotated Slides)	Infinite Clusters Dirichlet processes Introduction to sampling methods	Escobar, M.D. and West, M. Bayesian density estimation and inference using mixtures. Journal of the american statistical association 1995, pp 577-588. MacEachern, S.N. and Muller, P. Estimating mixture of Dirichlet process models. Journal of Computational and Graphical Statistics 1998, pp 223-238.
Structured Inference: Graphical Models
Mon 10/17	Lecture 11: Hidden Markov Models Slides (Annotated Slides)	Sequential Labeling: Hidden Markov Model	L. R. Rabiner. A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition. Proceedings of the IEEE, Vol. 77, No. 2 (06 February 1989), pp. 257-286.	HW 2 due, HW 3 out, Project proposal due
Wed 10/19	Lecture 12: Conditional Random Fields Slides (Annotated Slides)	Conditional Random Field: a discriminative HMM	John Lafferty, Andrew McCallum, Fernado Pereira (2001). Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data. 2001 International Conference on Machine Learning (ICML).
Mon 10/24	Lecture 13: Bayesian Networks Slides (Annotated Slides)	Bayesian Networks A Pedigree of people Exact inference algorithm	Bishop: Chap 8
Wed 10/26	Midterm Exam	open book, open notes, no computers	Will cover lectures upto 10/19
Mon 10/31	Lecture 14: Inference and Learning for Bayesian Networks Slides (Annotated Slides)	Inference and Learning for Bayesian Networks		HW 3 due
Wed 11/2	Lecture 15: Undirected Graphical Models and Approximate Inference Slides (Annotated Slides)	Undirected Graphical Models and Approximate Inference	Xing, E.P. and Jordan, M.I. and Russell, S. (2003). A generalized mean field algorithm for variational inference in exponential families. UAI 2003.	HW 4 out
Alternative strategies of learning
Mon 11/7	Lecture 16: PCA versus Topic models Slides (No annotated slides)	Subspace learning: nonprobabilistic vs probabilistic approaches Principal component analysis Topic models	David M. Blei, Andrew Y. Ng, Michael I. Jordan (2003). Latent Dirichlet Allocation. Journal of Machine Learning Research, Vol. 3 , pp. 993-1022.
Wed 11/9	Lecture 17: Support Vector Machines Slides (Annotated Slides)	Support Vector Machines	Burges, Christopher(1998). A Tutorial on Support Vector Machines for Pattern Recognition. Data Min. Knowl. Discov. 2(2): 121-167
Applications
Mon 11/14	Lecture 18: Structured Sparsity in Genetics (Lecturer: Seyoung Kim) Slides	Structured sparsity in genetics		Project mid-term report due
Wed 11/16	Lecture 19: Generative Latent Variable Models of Text (Lecturer: Jacob Eisenstein) Slides	Social media modeling and analysis via latent space models		HW 4 due, HW 5 out
Advanced Topics
Mon 11/21	Lecture 20: Advanced topics in maximum-margin learning Slides (Annotated Slides)	The kernel trick Maximum-entropy discrimination Maximum-margin markov networks
Wed 11/23	No class for Thanksgiving Break
Mon 11/28	Lecture 21: Max-margin learning of graphical models Slides (Annotated Slides)	Maximum-entropy discrimination Maximum-margin markov networks
Wed 11/30	Lecture 22: Ensemble Methods: Boosting from weak learners Slides (Annotated Slides)	Boosting: ensemble of weak learners
Mon 12/5	Lecture 23: Reinforcement Learning Slides	Reinforcement Learning Review What is machine learning? Elements of machine learning		HW 5 due
Wed 12/7	No class!	No class!
Thu 12/8	Poster session	NSH atrium 2:30-6:30pm		Project final report due
Tue 12/13	Final Exam	Doherty Hall 2210, 1:00-4:00pm	One A4 sheet of paper allowed, Closed book, CLosed notes.