10708 Probabilistic Graphical Models

Lecture Schedule

Lectures are held on Mondays and Wednesdays from 12:00-1:20 pm in GHC 4307.

Date	Lecture	Scribes	Readings	Anouncements
Wednesday, Jan 18	Lecture 1 (Eric) - Slides - Annotated - Video Introduction to GM		Required (no reading summary): Jordan Graphical Models E. Airoldi Getting Started in Probabilistic Graphical Models	Scribe Template
Module 1: Representation
Monday, Jan 23	Lecture 2 (Eric) - Slides Directed GMs: Bayesian Networks	Notes	Required: Jordan Textbook, Ch. 2 (Section 2.1) Optional: Koller and Friedman Textbook, Ch. 3
Wednesday, Jan 25	Lecture 3 (Eric) - Slides - Annotated - Video Representation of Undirected GM	Notes	Required: Jordan Textbook, Ch. 2 (Section 2.2 - end) A. Fischer and C. Igel, An Introducton to Restricted Boltzmann Machines Optional: B. A. Cipra, An Introduction to the Ising Model Koller and Friedman Textbook, Ch. 4 S. Gould et al., Decomposing a Scene into Geometric and Semantically Consistent Regions C. K. Chow and C. N. Liu, Approximating Discrete Probability Distributions with Dependency Trees	HW1 is out Friday, Jan 27
Module 2: Classical Methods of Inference & Learning
Monday, Jan 30	Lecture 4 (Eric) - Slides - Annotated - Video Exact inference Elimination and message passing The sum product algorithm	Notes	Required: Jordan Textbook, Ch. 3-4 Optional: Shun-Zheng Yu, Hidden semi-Markov models Koller and Friedman Textbook, Ch. 9, Ch. 10 Tom Minka, Divergence measures and message passing
Wednesday, Feb 1	Lecture 5 (Eric) - Slides - Annotated - Video Generalized Linear Models Maximum Likelihood Estimation Sufficient Statistics	Notes	Required: Jordan Textbook, Ch. 8 Optional: Structural learning using neighborhood selection:High-dimensional graphs and variable selection with the Lasso GLIM for longitudinal data: Liang & Zeger, Longitudinal data analysis using generalized linear models
Monday, Feb 6	Lecture 6 (Eric) - Slides - Annotated - Video Learning fully observed BN	Notes	Required: Jordan Textbook: Ch. 9 (Section 9.1 - 9.2) Koller and Friedman Textbook, Ch. 17 (Sections 17.1 - 17.4) Optional: Parameter Priors for Directed Acyclic Graphical Models and the Characterization of Several Probability Distributions A Characterization of the Dirichlet Distribution Through Global and Local Parameter Independence
Wednesday, Feb 8	Lecture 7 (Eric) - Slides - Video Learning fully observed undirected GM	Notes	Required: Jordan Textbook: Ch. 9 (Section 9.3 - 9.5), Ch. 20 Jerome Friedman, Trevor Hastie and Robert Tibshirani. Sparse inverse covariance estimation with the graphical lasso. Optional: J. Lafferty et al., Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data	HW1 is due Friday, Feb 10
Monday, Feb 13	Lecture 8 (Manuela) - Slides EM and partially observed GM	Notes	Required: Jordan Textbook, Ch. 11 Optional: A. Dempster et al., Maximum Likelihood from Incomplete Data via the EM Algorithm Some interesting aspects of EM Koller and Friedman Textbook, Ch. 19 (Section 19.1, 19.2.2, 19.2.3)
Module 3: Popular Graphical Models in Action
Wednesday, Feb 15	Lecture 9 (Manuela) - Slides Discrete sequential models: HMM vs. CRF	Notes	Required: A tutorial on HMMs CRF Tutorial by Hanna Wallach Optional: S. Kumar and M. Hebert. Discriminative Random Fields. IJCV, 2006 Structured Prediction and Learning in Computer Vision. CVPR 2011 Tutorial. Continuous-time HMMs.
Monday, Feb 20	Lecture 10 (Bryon) - Slides Gaussian graphical models Ising models Modeling networks	Notes	Required: N. Meinshausen and P. Buhlmann, High-Dimensional Graphs and Variable Selection with the Lasso M. Kolar et al., Estimating Time-Varying Networks Optional: A. Dempster, Covariance Selection M. Kolar et al., Graph Estimation From Multi-attribute Data A. Parikh et al., TREEGL: reverse engineering tree-evolving gene networks underlying developing biological lineages	Proposal is due Monday, Feb 20
Wednesday, Feb 22	Lecture 11 (Bryon) - Slides Factor analysis (FA) State space models (SSM)	Notes	Required: Jordan Textbook, Ch. 14, Ch. 15 Optional: A. Ng, Factor Analysis Lecture Notes D. Blei et al., Dynamic Topic Models G. Welch and G. Bishop, An Introduction to the Kalman Filter Z. Ghahramani et al., Variational Learning for Switching State-Space Models Structured inference networks for SSMs: arXiv:1609.09869	HW2 is out Friday, Feb 24
Module 4: Approximate Inference
Monday, Feb 27	Lecture 12 (Eric) - Slides Variational Inference: Loopy Belief Propagation Ising models	Notes	Required: Yedidia et al., Generalized Belief Propagation Wainwright and Jordan, Graphical Models, Exponential Families, and Variational Inference, Sec. 4.1 Optional: Murphy et al., Loopy Belief Propagation for Approximate Inference: An Empirical Study
Wednesday, Mar 1	Lecture 14 (Eric) - Slides - Video Theory of Variational Inference: Inner and Outer Approximation	Notes	Required: M. Wainwright and M. Jordan, Variational Inference in Graphical Models: The View from the Marginal Polytope Optional: M. Wainwright and M. Jordan, Graphical Models, Exponential Families, and Variational Inference, Sec. 3 and Sec. 4
Monday, Mar 6	Lecture 13 (Willie) - Slides - Video Variational Inference: Mean Field Approximation Topic Models	Notes	Required: Xing et al., A Generalized Mean Field Algorithm for Variational Inference in Exponential Families Wainwright and Jordan, Graphical Models, Exponential Families, and Variational Inference, Ch. 5 (Section 5.1 - 5.3) Optional: Variational inference tutorial (NIPS 2016)
Wednesday, Mar 8	Lecture 15 (Eric) - Slides - Video Approximate Inference: Monte Carlo methods	Notes	Required: Jordan Textbook, Ch. 21 Optional: David MacKay's Textbook, Ch. 29 (Section 29.1 - 29.3) D. MacKay, Introduction to Monte Carlo Methods	HW2 is due Friday, Mar 10
Monday, Mar 13	No Lecture due to CMU spring break.
Wednesday, Mar 15	No Lecture due to CMU spring break.
Monday, Mar 20	Lecture 16 (Eric) - Slides Markov Chain Monte Carlo	Notes	Required: David MacKay's Textbook, Ch. 29 (Section 29.4 - 29.10) Optional: Iain Murray's PhD thesis, pp. 20-55 Mackay. Introduction to Monte Carlo Methods C. Bishop, Pattern Recognition and Machine Learning, Ch. 11 R. Neal, Slice Sampling, Annals of Statistics, 2003 C. Geyer, Practical Markov chain Monte Carlo, Statistical Science 7(4): 473-492. 1992.
Wednesday, Mar 22	Lecture 17 (Avinava) - Slides Optimization and Monte Carlo methods	Notes	Required: R. Neal. MCMC using Hamiltonian dynamics Optional: S. Patterson, Y. W. Teh. Stochastic Gradient Riemannian Langevin Dynamics on the Probability Simplex.
Module 5: Deep Learning
Monday, Mar 27	Lecture 18 (Maruan) - Slides An overview of DL building blocks Similarities and differences between NNs & GMs Ways to combine GMs and NNs	Notes	Required: Deep learning book, Ch. 6.2-5, 20.3-4 Optional: Deep Boltzmann machines: Salakhutdinov and Hinton, 2009 Deep exponential families: Ranganath et al., 2015 Structured prediction energy networks: Belanger and McCallum, 2016 Structured inference networks for nonlinear SSMs: Krishnan et al., 2016 On the Origin of Deep Learning: Wang and Raj, 2017	Midway report is due Monday, Mar 27
Wednesday, Mar 29	Lecture 19 (Zhiting) - Slides Inference & learning in DL: VAEs and GANs Ways to incorporate domain knowledge into deep neural networks Deep models for NLP applications	Notes	Required: Deep learning book, Ch. 20.9-10 Harnessing deep neural nets with logic rules: Hu et al., 2016. Optional: Variational Autoencoders: Kingma and Welling, 2014. Sanjeev Arora's blog post on GANs. Generative Adversarial Nets: Goodfellow et al., 2014. Wasserstein GAN: arXiv:1701.07875.	HW3 is out Friday, Mar 31
Monday, Apr 3	Lecture 20 (Xiaodan) - Slides Convolutional and recurrent neural networks Attention and memory mechanisms Applications in computer vision	Notes	Required: Deep learning book, Ch. 9.1-4, 10.1-2 Recurrent models of visual attention: Mnih et al., 2014. Optional: Reinforcement learning tutorial by David Silver Fully convolutional networks: Long, Shelhamer, Darrel, 2015. Long-term recurrent convolutional networks: Donahue et al., 2015. Tree-structured LSTM: Tai, Socher, Manning, 2015 Deep variation-structured RL: Liang, Lee, Xing, 2017.
Module 6: Scalable Approaches
Wednesday, Apr 5	Lecture 21 (David) - Slides Distributed Algorithms for ML		Required: LightLDA: Big Topic Models on Modest Computer Clusters: Yuan et al., 2015. Parallel Coordinate Descent for L1-Regularized Loss Minimization: Bradley et al., 2011. Optional: HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent: Niu et al., 2011. On Model Parallelization and Scheduling Strategies for Distributed Machine Learning: Lee et al., 2014.
Monday, Apr 10	Lecture 22 (Qirong) - Slides Distributed Systems for ML	Notes	Required: More effective distributed ML via a stale synchronous parallel parameter server: Ho et al., 2013. Optional: Managed Communication and Consistency for Fast Data-Parallel Iterative Analytics: Wei et al., 2015. STRADS: A Distributed Framework for Scheduled Model Parallel Machine Learning: Kim et al., 2016. Lighter-Communication Distributed Machine Learning via Sufficient Factor Broadcasting: Xie et al., 2016.
Module 7: Advanced Graphical Models
Wednesday, Apr 12	Lecture 23 (Avinava) - Slides Nonparametric Bayesian Graphical Models Dirichlet and Hierarchical Dirichlet Processes Indian Buffet Process	Notes	Required: Y. W. Teh, Dirichlet Process Thomas L. Griffiths and Zoubin Ghahramani, The Indian Buffet Process: An Introduction and Review, JMLR 2011. Optional: Video lecture by Y.W. Teh: Dirichlet Processes: Tutorial and Practical Course, MLSS 2007.	HW3 is due & HW4 is out Friday, Apr 14
Monday, Apr 17	Lecture 24 (Zhiting) - Slides Nonparametric Bayesian Graphical Models Gaussian process (Deep) kernel learning	Notes	Required: C. Rasmussen and C. Williams, GPML, Preface + Ch. 2.2-2.4 C. Rasmussen and Z. Ghahramani, Occam's razor, NIPS 2001. Optional: Wilson, Hu, Salakhutdinov, Xing: Deep kernel learning. Al-Shedivat et al.: Learning Scalable Deep Kernels with Recurrent Structure. Wilson, Hu, Salakhutdinov, Xing: Stochastic Variational Deep Kernel Learning.
Wednesday, Apr 19	Lecture 25 (Eric) - Slides Max-margin Graphical Models		Required: B. Taskar, C. Guestrin, and D. Koller, Max-Margin Markov Networks J. Zhu and E. Xing, Maximum Entropy Discrimination Markov Networks Optional:	HW4 is due Friday, Apr 21
Monday, Apr 24	Lecture 26 (Pengtao) - Slides Regularized Bayesian Graphical Models		Required: J. Zhu, N. Chen, and E. P. Xing, Bayesian Inference with Posterior Regularization and Applications to Infinite Latent SVMs, JMLR 2014. K. Ganchev, J. Graça, J. Gillenwater, and B. Taskar, Posterior Regularization for Structured Latent Variable Models, JMLR 2010.
Wednesday, Apr 26	Lecture 27 (Eric) - Slides Spectral & Kernel Graphical Models		Required: D. Hsu, S. Kakade, and T. Zhang A Spectral Algorithm for Learning Hidden Markov Models L. Song. Learning via Hilbert space embedding of distributions, Sec. 2.1, 2.2, 3.1, 3.2 Optional: Song, Liu, Parikh, Xing: Nonparametric latent tree graphical models. L. Song, J. Huang, A. Smola, and K. Fukumizu: Hilbert Space Embedding of Conditional Distributions with Applications to Dynamical Systems L. Song et al., Nonparametric Latent Tree Graphical Models: Inference, Estimation, and Structure Learning. A. Parikh, L. Song, and E. Xing: A Spectral Algorithm for Latent Tree Graphical Models. L. Song et al.: Hilbert Space Embeddings of Hidden Markov Models. L. Song et al.: Kernel Embeddings of Latent Tree Graphical Models.

Monday, May 1	Class Presentation			Final report is due Monday, May 8