10708 Probabilistic Graphical Models

Lecture Schedule

Lectures are held on Mondays and Wednesdays from 12:00-1:20 pm in GHC 4307.

Date	Lecture	Scribes	Readings	Anouncements
Monday, Jan 11	Lecture 1: Introduction to GM - Slides	Yuxing Zhang, Tianshu Ren Notes	Required (no reading summary): Jordan Graphical Models E. Airoldi Getting Started in Probabilistic Graphical Models	Scribe Template
Module 1: Representation
Wednesday, Jan 13	Lecture 2: Directed GMs: Bayesian Networks - Slides	Lidan Mu, Lanxiao Xu Notes	Required (please bring your reading summary): Jordan Textbook, Ch. 2 (Section 2.1) Optional: Koller and Friedman Textbook, Ch. 3
Monday, Jan 18	No Lecture due to MLK day.
Wednesday, Jan 20	Lecture 3: Representation of Undirected GM - Slides	Longqi Cai, Man-Chia Chang Notes	Required (please bring your reading summary): Jordan Textbook, Ch. 2 (Section 2.2 - end) A. Fischer and C. Igel, An Introducton to Restricted Boltzmann Machines Optional: B. A. Cipra, An Introduction to the Ising Model Koller and Friedman Textbook, Ch. 4 S. Gould et al., Decomposing a Scene into Geometric and Semantically Consistent Regions C. K. Chow and C. N. Liu, Approximating Discrete Probability Distributions with Dependency Trees
Module 2: Classical Methods of Inference & Learning
Monday, Jan 25	Lecture 4: Parameter Estimation in Fully Observed BNs - Slides	Natalie Klein, Purvasha Chakravarti, Dipan Pal Notes	Required (please bring your reading summary): Koller and Friedman Textbook, Ch. 17 (Section 17.2 - 17.4) Jordan Textbook, Ch. 8 Optional: T. Park and G. Casella, The Bayesian Lasso R. Tibshirani, Regression Shrinkage and Selection via the Lasso Parameter Priors for Directed Acyclic Graphical Models and the Characterization of Several Probability Distributions A Characterization of the Dirichlet Distribution Through Global and Local Parameter Independence
Wednesday, Jan 27	Lecture 5: Learning fully observed directed GM - Slides, Whiteboard	Yuan Li, Yichong Xu, Silun Wang Notes	Required (please bring your reading summary): Jordan Textbook, Ch. 9 (Section 9.1 - 9.2) Optional: Structural learning using neighborhood selection:High-dimensional graphs and variable selection with the Lasso Chow-Liu tree seminal paper:Approximating discrete probability distributions with dependence trees	Homework 1 is out (Jan 30). Due on Feb 15 at 12 noon.
Monday, Feb 1	Lecture 6: Learning fully observed undirected GM - Slides, Whiteboard	Akash Bharadwaj, Sumeet Kumar, Devendra Chaplot Notes	Required: Jordan Textbook: Ch. 9 (Section 9.3 - 9.5), Ch. 20 F. Sha and F. Pereira, Shallow Parsing with Conditional Random Fields Optional: Graphical lasso J. Lafferty et al., Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data Koller and Friedman Textbook, Ch. 20 (Section 20.1 - 20.3)
Wednesday, Feb 3	Lecture 7: Exact Inference - Slides	Keith Maki, Anbang Hu, Jining Qin Notes	Required: Jordan Textbook: Ch. 3, Ch. 4 Optional: Shun-Zheng Yu, Hidden semi-Markov models Koller and Friedman Textbook, Ch. 9, Ch. 10
Monday, Feb 8	Lecture 8: Learning Partially observed models - Slides	Cuong Nguyen, Anirudh Vemula, Ankit Laddha Notes	Required (please bring your reading summary): Jordan Textbook, Ch. 11 Optional: A. Dempster et al., Maximum Likelihood from Incomplete Data via the EM Algorithm Some interesting aspects of EM Koller and Friedman Textbook, Ch. 19 (Section 19.1, 19.2.2, 19.2.3)
Module 3: Popular Graphical Models in Action
Wednesday, Feb 10	Lecture 9: Discrete sequential models + CRFs - Slides, Whiteboard	Pankesh Bamotra, Xuanchong Li Notes	Required (please bring your reading summary): A tutorial on HMMs CRF Tutorial by Hanna Wallach Optional: S. Kumar and M. Hebert. Discriminative Random Fields. IJCV, 2006 Structured Prediction and Learning in Computer Vision. CVPR 2011 Tutorial.
Monday, Feb 15	Lecture 10: Gaussian graphical models and Ising models: modeling networks - Slides	Xiongtao Ruan, Kirthevasan Kandasamy Notes	Required (please bring your reading summary): J. Friedman et al., Sparse Inverse Covariance Estimation with the Graphical Lasso M. Kolar et al., Estimating Time-Varying Networks Optional: A. Dempster, Covariance Selection N. Meinshausen and P. Buhlmann, High-Dimensional Graphs and Variable Selection with the Lasso M. Kolar et al., Graph Estimation From Multi-attribute Data A. Parikh et al., TREEGL: reverse engineering tree-evolving gene networks underlying developing biological lineages	Homework 1 due at 12 noon
Wednesday, Feb 17	Lecture 11: Factor Analysis and State Space Models - Slides	Yu Zhang, Syed Zahir Bokhari, Rahul Nallamothu Notes	Required (please bring your reading summary): Jordan Textbook, Ch. 14 Jordan Textbook, Ch. 15 Optional: D. Blei et al., Dynamic Topic Models A. Ng, Factor Analysis Lecture Notes G. Welch and G. Bishop, An Introduction to the Kalman Filter Z. Ghahramani et al., Variational Learning for Switching State-Space Models R.P. Roesser, A discrete state-space model for linear image processing	Project proposal due at 12 noon
Module 4: Approximate Inference
Monday, Feb 22	Lecture 12: Variational Inference: Loopy Belief Propagation - Slides	Jing Chen, Yulan Huang, Yu Fang Chang Notes	Required (please bring your reading summary): Yedidia et al., Generalized Belief Propagation Wainwright and Jordan, Graphical Models, Exponential Families, and Variational Inference, Sec. 4.1 Optional: Murphy et al., Loopy Belief Propagation for Approximate Inference: An Empirical Study	Homework 2 is out. Due on Mar 16 at noon.
4-5:30pm Friday, Feb 26, Porter Hall 125C	Lecture 13: Mean Field Approximation & Topic Models - Slides	Shichao Yang, Haoqi Fan, Mengtian Li Notes	Required (please bring your reading summary): Xing et al., A Generalized Mean Field Algorithm for Variational Inference in Exponential Families Wainwright and Jordan, Graphical Models, Exponential Families, and Variational Inference, Ch. 5 (Section 5.1 - 5.3)
Monday, Feb 29	Lecture 14: Theory of VariationalInference: Inner and Outer Approximation - Slides	Chieh Lo, Wei-Chiu Ma, Qi Guo Notes	Required (please bring your reading summary): M. Wainwright and M. Jordan, Variational Inference in Graphical Models: The View from the Marginal Polytope Optional: M. Wainwright and M. Jordan, Graphical Models, Exponential Families, and Variational Inference, Sec. 3 and Sec. 4
Wednesday, Mar 2	Lecture 15: Approximate Inference: Monte Carlo methods - Slides	Binxuan Huang, Yotam Hechtlinger, Fuchen Liu Notes	Required: Jordan Textbook, Ch. 21 Optional: Mackay Textbook, Ch. 29 (Section 29.1 - 29.3) D. Mackay, Introduction to Monte Carlo Methods
Monday, Mar 7	No Lecture due to CMU spring break.
Wednesday, Mar 9	No Lecture due to CMU spring break.
Monday, Mar 14	Lecture 16: MCMC - Slides, Whiteboard	Yining Wang, Renato Negrinho Notes	Required: Mackay Textbook, Ch. 29 (Section 29.4 - 29.10) Optional: Iain Murray's PhD thesis, pp. 20-55 Mackay. Introduction to Monte Carlo Methods C. Bishop, Pattern Recognition and Machine Learning, Ch. 11 R. Neal, Slice Sampling, Annals of Statistics, 2003 C. Geyer, Practical Markov chain Monte Carlo, Statistical Science 7(4): 473-492. 1992.
Wednesday, Mar 16	Lecture 17: Case study with approximate inference - Slides	Yanyu Liang, Chun-Liang Li, Mengxin Li Notes	Required (please bring your reading summary): D. Blei et al., Latent Dirichlet Allocation T. Griffiths and M. Steyvers, Finding Scientific Topics Optional: A. Ahmed and E. P. Xing, On Tight Approximate Inference of Logistic-Normal Admixture Model.	Homework 2 due at 12 noon
Module 5: Nonparametric Bayesian Models
Monday, Mar 21	Lecture 18: Dirichlet Process and Dirichlet Process Mixtures - Slides	Chiqun Zhang, Hsu-Chieh Hu Notes	Required: Y. W. Teh, Dirichlet Process Optional: Y. W. Teh, Dirichlet Processes: Tutorial and Practical Course, MLSS 2007.
Wednesday, Mar 23	Lecture 19: Indian Buffet Process - Slides, Whiteboard	Kai-Wen Liang, Han Lu Notes	Required: Thomas L. Griffiths and Zoubin Ghahramani, The Indian Buffet Process: An Introduction and Review, JMLR 2011.	Midway report due at 12 noon
Monday, Mar 28	Lecture 20: Gaussian Processes - Slides	Sai Ganesh Notes	Required: C. Rasmussen and C. Williams, GPML, Preface + Ch. 2.2-2.4 C. Rasmussen and Z. Ghahramani, Occam's razor, NIPS 2001. Optional: Wilson PhD Thesis, 2014, Ch. 1, 2. C. Rasmussen and C. Williams, GPML, Ch. 3, 5 D. MacKay, Introduction to Gaussian Processes, 1998.
Module 6: Spectral Graphical Models
Wednesday, Mar 30	Lecture 21: Spectral Learning for Graphical Models - Slides	Maruan Al-Shedivat, Wei-Cheng Chang, Frederick Liu Notes	Required: D. Hsu, S. Kakade, and T. Zhang A Spectral Algorithm for Learning Hidden Markov Models Optional: L. Song et al., Nonparametric Latent Tree Graphical Models: Inference, Estimation, and Structure Learning A. Parikh, L. Song, and E. Xing, A Spectral Algorithm for Latent Tree Graphical Models L. Song et al., Hilbert Space Embeddings of Hidden Markov Models L. Song et al., Kernel Embeddings of Latent Tree Graphical Models	Homework 3 is out (Mar 29). Due on Apr 13 at 12 noon.
Monday, Apr 4	Lecture 22: Introduction to Hilbert Space Embeddings and Kernel GM - Slides	Kevin Lin Notes	Required: L. Song. Learning via Hilbert space embedding of distributions, Sec. 2.1, 2.2, 3.1, 3.2 Optional: L. Song, J. Huang, A. Smola, and K. Fukumizu. Hilbert Space Embedding of Conditional Distributions with Applications to Dynamical Systems
Module 7: Optimization view of Graphical Models
Wednesday, Apr 6	Lecture 23: Graph-induced structured input/output models - Slides	Raied Aljadaany, Shi Zong, Chenchen Zhu Notes	Required: X. Chen et al. Smoothing Proximal Gradient Method for General Structured Sparse Regression S. Kim, E. P. Xing. Statistical Estimation of Correlated Genome Associations to a Quantitative Trait Network Optional: S. Kim, E. P. Xing. Tree-Guided Group Lasso for Multi-Task Regression with Structured Sparsity
Monday, Apr 11	Lecture 24: Max-margin learning of GMs - Slides	Po-Wei Wang, Eric Wong, Achal Dave Notes	Required: B. Taskar, C. Guestrin, and D. Koller, Max-Margin Markov Networks J. Zhu and E. Xing, Maximum Entropy Discrimination Markov Networks
Wednesday, Apr 13	Lecture 25: Regularized Bayesian learning of GMs - Slides	Tzu-Ming Kuo Notes	Required: J. Zhu, N. Chen, and E. P. Xing, Bayesian Inference with Posterior Regularization and Applications to Infinite Latent SVMs, JMLR 2014. Optional: K. Ganchev, J. Graça, J. Gillenwater, and B. Taskar, Posterior Regularization for Structured Latent Variable Models, JMLR 2010.	Homework 3 due at 12 noon; Homework 4 is to be released.
Module 8: Deep Learning
Monday, Apr 18	Lecture 26: Deep neural networks and GMs - Slides	Hayden Luse Notes	Required: L. Deng, A Tutorial Survey of Architectures, Algorithms, and Applications for Deep Learning, Transactions on Signal and Information Processing 2013. Optional: Y. Bengio, Learning Deep Architectures for AI, Foundations and Trends in Machine Learning, 2009. G. Hinton, A Practical Guide to Training Restricted Boltzmann Machines, Tech Report, 2010. M. A. Ranzatto, Y. L. Boureau and Y. LeCun, Sparse Feature Learning for Deep Belief Networks, NIPS, 2007.
Wednesday, Apr 20	Lecture 27: Hybrid Graphical Models and Neural Networks - Slides	Jakob Bauer, Rohan Varma, Otilia Stretcu Notes	Required: Ossama Abdel-Hamid et.al., Deep Segmental Neural Networks for Speech Recognition Optional: Waleed Ammar et.al., Conditional Random Field Autoencoders for Unsupervised Structured Prediction, NIPS 2014.
Module 9: Scalable Approaches for Graphical Models
Monday, Apr 25	Lecture 28: Distributed Algorothms for ML - Slides	Joe Runde, Michael Muehl Notes	Required: H. Wang et al., Large Scale Distributed Sparse Precision Estimation, NIPS 2013. Optional: T. Cai et al., A constrained l_1 minimization approach to sparse precision matrix estimation, JASA 2011.
Wednesday, Apr 27	Lecture 29: Distributed Systems for ML - Slides	Petar Stojanov, Christoph Dann Notes	Required: Yuan, Jinhui, et al. "Lightlda: Big topic models on modest computer clusters." Proceedings of the 24th International Conference on World Wide Web. International World Wide Web Conferences Steering Committee, 2015. Optional: Wei, Jinliang, et al. "Managed communication and consistency for fast data-parallel iterative analytics." Proceedings of the Sixth ACM Symposium on Cloud Computing. ACM, 2015. Ho, Qirong, et al. "More effective distributed ml via a stale synchronous parallel parameter server." Advances in neural information processing systems. 2013.	Homework 4 due at 12 noon.