10701/15781 Machine Learning

Syllabus and (tentative) Course Schedule

Date	Lecture	Topics	Readings and useful links	Anouncements
Mon 9/10	Lecture 1: Overview, Decision Trees Logistics, Introduction, Decision Tree Lecturer: Eric Xing / Aarti Singh	Overview of Machine Learning What is Machine Learning? Elements of machine learning Decision tree learning	Mitchell: Chap 1,3 Decision Tree Learning [Applet] Leo Breiman (2001). Random Forests. Machine Learning Volume 45, Number 1.
Wed 9/12	Lecture 2: MLE & MAP. Slides Lecturer: Aarti Singh		Mitchell: Naive Bayes, Logistic Regression	HW 1 out
Mon 9/17	Lecture 3: Bayes Estimator, Naive Bayes Slides Lecturer: Aarti Singh		Mitchell: Naive Bayes, Logistic Regression Bishop: Chapter 1.2.4
Wed 9/19	Lecture 4: Logistic Regression, Generative versus Discriminative model Slides Lecturer: Eric Xing		Bishop Ch 3,4 Andrew Ng and Michael Jordan (2002). On discriminative vs. generative classifiers: A comparison of Logistic Regression and Naive Bayes. Neural Information Processing Systems 2002.
Mon 9/24	Lecture 5: Linear regression. Slides Lecturer: Eric Xing		Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. J. Royal. Statist. Soc B., Vol. 58, No. 1, pages 267-288. Tibshirani, R., Saunders, M., Rosset, S., Zhu, J. and Knight, K. (2005), Sparsity and smoothness via the fused lasso. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 67: 91-108. Yuan, M. and Lin, Y. Model selection and estimation in regression with grouped variables. J. Royal. Statist. Soc B., Vol. 68, No. 1, pages 49-67.
Wed 9/26	Lecture 6: Instatnce Based Learning Slides Lecturer: Eric Xing		Bishop: Chap 2.5 Fukunaga (Intro to Statistical PR) Hypothesis Test Non-parametric density estimation Non-paramteric Classification Tutorial on an instance of instance based learning: Locally weighted regression- Andrew Moore Herwig Lejsek, Bjorn Por Jonsson, and Laurent Amsaleg (2011). NV-Tree: nearest neighbors at the billion scale. In Proceedings of the 1st ACM International Conference on Multimedia Retrieval (ICMR '11). ACM, New York, NY, USA, , Article 54 , 8 pages.
Mon 10/1	Lecture 7: Bias, Variance, Model Selection. Slides Lecturer: Aarti Singh		Bishop: Chap 1, 2 Mitchell: Chap 5, 6 Matlab demo code for understanding overfitting Model comparison and Occam's Razor,Chapter 28 from David Mackay's book Model selection and Minimum Description Length principle,Mark Hansen and Bin Yu, J. Amer. Statist. Assoc. vol.96,746-774, 2001
Wed 10/3	Lecture 8: SVM Slides Lecturer: Aarti Singh		SVM [Applet] Bishop: Chap 6, 7 Burgess tutorial The Sequential Minimal Optimization page, by Platt Pegasos: Primal Estimated Sub-Gradient Solver for SVM Fast and efficient online SVM
Mon 10/8	Lecture 9: SVM Slides Lecturer: Aarti Singh		Maximum-entropy Discrimination, Jaakolla et al. Max-margin Markov network, Taskar et al. Laplace Max-margin Markov networks, Zhu et al.
Wed 10/10	No class			HW 2 out
Mon 10/15	Lecture 10: Clustering and EM Slides Lecturer: Eric Xing		Bishop Ch: 9 Arthur, D. and Vassilvitskii, S. (2007). k-means++: The advantages of careful seeding. 2007 ACM-SIAM symposium on Discrete algorithms (SODA). pp 1027-1035. E.P. Xing, A.Y. Ng, M.I. Jordan and S. Russell, Distance Metric Learning, with application to Clustering with side-information, Advances in Neural Information Processing Systems 16 (NIPS2002), (eds. Becker et al.) MIT Press, 521-528, 2002.
Wed 10/17	Midterm Exam
Mon 10/22	Lecture 11: EM: mixture model and HMM Slides Lecturer: Eric Xing		Bilmes, Jeff. (1998). A gentle tutorial of the EM algorithm and its application to parameter estimation for Gaussian mixture and hidden Markov models. TR-97-021, Computer Science Division, U.C. Berkeley.
Wed 10/24	Lecture 12: HMM Slides Lecturer: Eric Xing		Rabiner, Lawrence R. (1989). A Tutorial on Hidden Markov Model and selected Applications in Speech Recognition Proceedings of the IEEE Volume 77, Issue 2, pp 257-286
Mon 10/29	Lecture 13: Graphical Model Slides Lecturer: Eric Xing		Bishop: Chap 8 Kevin Murphy's tutorial BayesNet Toolbox in Matlab by Kevin Murphy	HW2 Due
Wed 10/31	Lecture 14: Inference and Learning in GM Slides Lecturer: Eric Xing			HW 3 out
Mon 11/5	Lecture 15: Advanced Learning Theory I Slides Lecturer: Aarti Singh		Mitchell: Ch 7	Midway Report Due
Wed 11/7	Lecture 16: Advanced Learning Theory II Slides Lecturer: Aarti Singh		Mitchell: Ch 7 VC dimension tutorial by Andrew Moore Structural Risk Minimization
Mon 11/12	Lecture 17: Boosting Slides Lecturer: Aarti Singh		Bishop: Sec 14.3 Boosting homepage Schapire: Boosting Tutorial, Video Adaboost Applet
Wed 11/14	Lecture 18: Active learning Slides Guest Lecturer: Burr Settles		A free PDF of Burr's book on active learning (through the CMU network), the DUALIST demo, and other useful links at: active-learning.net	HW 4 out
Mon 11/19	Lecture 19: Latent Space Analysis SVD and Topic Models Slides Lecturer: Eric Xing		Eric Xing: Topic Models, Latent Space Models, Sparse Coding, and All That: A systematic understanding of probabilistic semantic extraction in large corpus, ACL 2012 Tutorial David Blei, et.al.: Latent Dirichlet Allocation, Journal of Machine Learning Research 3 (2003) 993-1022 Bishop Chap 12.1 pp 561~569: Principal Component Analysis Hyvarinen, A.: Independent Component Analysis: Algorithms and Application Neural Networks 2000
Mon 11/26	Lecture 20: Reinforcement Learning Slides Lecturer: Eric Xing		Sutton, Richard and Barto, Andrew: Reinforcement Learning: an introduction, MIT Press, Cambridge, MA, 1998
Wed 11/28	Lecture 21: Spectral Clustering Slides Lecturer: Aarti Singh		Spectral Clustering Tutorial by Ulrike von Luxburg	HW4 due
Mon 12/17, 5:30pm to 8:30pm	Final Exam (GHC 4401)