Module	Lectures, readings, online materials	Homeworks, Exams
		Mon, Jan 15: *No Class, MLK Day*
Introduction: Hierarchical Bayes	Lecture 1: 1/17/07 Basics of hierarchical Bayesian models Reading: Christian Robert, The Bayesian Choice, 2nd Edition, Chapter 10, Springer, 2001 (hardcopy). Basics of hierarchical Bayesian models. William DuMouchel, Hierarchical Bayes Linear Models for Meta-Analysis. Technical Report Number 27, National Institute of Statistical Sciences, 1994. [pdf] Andrew Gelman, ``Multilevel (hierarchical) modeling: what it can and can't do.'' Technometrics, to appear, 2005 [pdf]	Wed, Jan 17: Scribe Notes 1 (Kyung-Ah) Scribe Notes 2 (Kyung-Ah)
	Lecture 2: 1/22/07 Bayesian Admixtures Latent Dirichlet Allocation Genetic Admixture The mixed membership model Reading: IR: Blei, D., Ng, A., and Jordan, M. Latent Dirichlet Allocation. Journal of Machine Learning Research. 3:993-1022. Genetics: Daniel Falush, Matthew Stephens, and Jonathan K. Pritchard (2003), Inference of Population Structure Using Multilocus Genotype Data: Linked Loci and Correlated Allele Frequencies. Networks: E. Airoldi, D. Blei, E.P. Xing and S. Fienberg, A Latent Mixed Membership Model for Relational Data. Workshop on Link Discovery: Issues, Approaches and Applications (LinkKDD-2005). Elena Erosheva, Steve Fienberg, and John Lafferty, Mixed membership models of scientific publications, Proceedings of the National Academy of Sciences, Vol. 101, Suppl. 1, April 6, 2004.	Mon, Jan 22: Scribe Notes 2 (Wenjie)
Advanced Topics in Learning I: Model Selection	Lecture 3: 1/24/07 Variable selection in hierarchical Bayesian models Reading: Ed George and Robert McCulloch (1993). "Variable selection by Gibbs sampling." JASA, 88, 881-889. [pdf] Xinlei Wang and Edward I. George (2004). A Hierarchical Bayes Approach to Variable Selection for Generalized Linear Models. Technical report SMU-TR-321, Department of Statistics, Southern Methodist University. [pdf] J.E. Griffin and P.J. Brown (2005). Alternative prior distributions for variable selection with very many more variables than observations. Technical report, Dept. of Statistics, University of Warwick. [pdf]	Wed, Jan 24: Scribe Notes 3 (Fan)
	Lecture 4: 1/29/07 Information theory and model selection I: Information criteria Introduction to information theory some basic definitions coding theory Reading: T. Cover and J. Thomas, Information Theory. Chaps 2, 3, 5.	Mon, Jan 29: Scribe Notes 4 (Fan)
	Lecture 5: 1/31/07 Information theory and model selection II: Minimum description length Reading: Mark Hansen and Bin Yu (2001). Model selection and Minimum Description Length principle. J. Amer. Statist. Assoc. vol. 96, 746-774.	Wed, Jan 31: Scribe Notes 5 (?)
	Lecture 6: 2/5/07 Bayesian Model Selection BIC Bayes factor Reading: Kass, R.E. and Raftery, A.E. (1995). Bayes factors. Journal of the American Statistical Association, 90, 773-795.	Mon, Feb 5: Scribe Notes 6 (Wilson)
Advanced Topics in Learning II: Conditional Graphical Models and Margin-based Learning	Lecture 7: 2/7/07 Conditional Random Fields Basic formulation Likelihood-based training Reading: John Lafferty, Andrew McCallum, Fernando Pereira, Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data, Proc. 18th International Conf. on Machine Learning. Fei Sha and Fernando Pereira (2003).   Shallow Parsing with Conditional Random Fields.   In Proceedings of Human Language Technology-NAACL 2003.	Wed, Feb 7:   Scribe Notes 7 (Ramesh)
	Lecture 8: 2/12/06 Optimal margin principle and Lagarangian Duality Support vector learning Duality and convex optimization Reading: Burgess tutorial	Mon, Feb 12: Scribe Notes 8 (Guang)
	Lecture 9: 2/14/07 Maximum margin Markov Networks structured output Reading: Taskar, B., Guestrin, C., & Koller, D. (2004). Max-margin Markov networks. Advances in Neural Information Processing Systems. Ioannis Tsochantaridis , Thorsten Joachims , Thomas Hofmann , Yasemin Altun, Large Margin Methods for Structured and Interdependent Output Variables, The Journal of Machine Learning Research, 6, p.1453-1484.	Wed, Feb 14: Scribe Notes 9 (?)
	Lecture 10: 2/19/07 Maximum margin learning of generative models Gaussian mixture model HMM Reading: Fei Sha and Lawrence K. Saul (2006).   Large margin Gaussian mixture modeling for phonetic classification and recognition.  Proc. of ICASSP 2006, pages 265-268. Fei Sha and Lawrence K. Saul (2007).   Large margin hidden Markov models for automatic speech recognition. To appear in Advances in Neural Information Processing Systems 19, 2007.	Mon, Feb 19: Scribe Notes 10 (Jason)
	Lecture 11: 2/21/07 New perspective of generative and discriminative models example 1 example 2 Reading: Tom Minka, Principled Hybrids of Generative and Discriminative Models, 2006.	Wed, Feb 21:
Sampling methods for inference and learning	Lecture 12: 2/26/07 Monte Carlo EM Markov Chain Monte Carlo: Metropolis Hasting and Gibbs Sampling The data-augmentation algorithm Reading: Andrieu, C., de Freitas, N., Doucet, A., and Jordan, M. (2003). An introduction to MCMC for machine learning. Machine Learning, 50:5-43.	Mon, Feb 26: Project proposals (1 page) due Scribe Notes 12-13 (Wenjie)
	Lecture 13: 2/28/07 Sampling high-dimensional models Collapsed Gibbs sampling Case study of the Gibbs Motif sampler Reading: J. S. Liu. The collapsed Gibbs sampler with applications to a gene regulation problem. Journal of the American Statistical Association, 89, 958-966, 1994. J. S. Liu, A. F. Neuwald, and C. E. Lawrence. Bayesian models for multiple local sequence alignment and Gibbs sampling strategies. Journal of the American Statistical Association, 90, 1156-1170, 1995.	Wed, Feb 28:
	Lecture 14: 3/7/07 Sampling and model selection Approximation of marginal likelihood Reverse-jump MCMC Reading: Richardson, S. and Green, P. J. (1997). On Bayesian analysis of mixtures with unknown number of components (with discussion). Journal of the Royal Statistical Society, Series B, 59, 731--92. Green, P. (1995). Reversible jump Markov chain Monte Carlo computation and Bayesian model determination. Biometrica 82, 711--732.	Wed, Mar 7:
	Lecture 15: 3/?/07 (note covered) Sampling random fields Learning Markov random fields Approximating the partition function Contrastive divergence More advanced sampling methods Reading: Iain Murray and Zoubin Ghahramani, Bayesian Learning in Undirected Graphical Models, 2005.	?, Mar ?:
	No class	Spring Break
Bayesian nonparametric models	Lecture 15: 3/19/07   The stick-breaking process (warming up) T. S. Ferguson. A Bayesian analysis of some nonparametric problems, Annals of Statistics, 1, 209-230, 1973.	Mon, Mar 19:
	Lecture 17: 3/21/07 Dirichlet process The infinite limit of mixture model The Polya urn process The stick-breaking process Reading: Required: Hemant Ishwaran and Lancelot F. James. Gibbs sampling methods for stick-breaking priors, Journal of the American Statistical Association, March 2001, Vol. 96, No. 453, Supplemental: D. Blackwell and J. MacQueen. Ferguson distributions via Polya urn schemes, Annals of Statistics, 1, 353-355, 1973. C. Antoniak. Mixtures of Dirichlet processes with applications to Bayesian nonparametric problems, Annals of Statistics, 2, 1152-1174, 1974. J. Sethuraman. A constructive definition of Dirichlet priors, Statistica Sinica, 4, 639-650, 1994.	Wed, Mar 21:
	Lecture 18: 3/26/07 Inference on Dirichlet process Sampling algorithms Truncation and variational approximation Reading: M. Escobar and M. West. Bayesian density estimation and inference using mixtures, Journal of the American Statistical Association, 90, 577-588, 1995. R. Neal. Markov chain sampling methods for Dirichlet processes mixture models, Technical Report 9815, Department of Statistics, University of Toronto, 1998. D. M. Blei and M. I. Jordan. Variational methods for the Dirichlet process,  Proceedings of the 21st International Conference on Machine Learning (ICML), 2004. [Long version].	Mon, Mar 26:
	Lecture 19: 3/28/07 Learning DP Empirical Bayesian learning of DP Reading: J. D. McAuliffe, D. M. Blei and M. I. Jordan. Nonparametric empirical Bayes for the Dirichlet process mixture model, Statistics and Computing, 16, 5-14, 2006.	Wed, Mar 28:
	Lecture 20: 4/2/07 Hierarchical DP and Hidden Markov DP Theory application in IR and Genetics Reading: Y. W. Teh, M. I. Jordan, M. J. Beal and D. M. Blei. Hierarchical Dirichlet processes, Journal of the American Statistical Association, 101, 1566-1581, 2006. E. P. Xing, K.-A. Song, M. I. Jordan, and Y. W. Teh. Bayesian multi-population haplotype inference via a hierarchical Dirichlet process mixture, Proceedings of the 23rd International Conference on Machine Learning (ICML), 2006. D. M. Blei, T. Griffiths, M. I. Jordan, and J. Tenenbaum. Hierarchical topic models and the nested Chinese restaurant process, In S. Thrun, L. Saul, and B. Schoelkopf (Eds.), Advances in Neural Information Processing Systems (NIPS) 16, 2004.	Mon, Apr 2:
	Lecture 21: 4/4/07 Hierarchical DP and hidden Markov DP Theory application in IR and Genetics Reading: E. P. Xing and K. Sohn, A New Nonparametric Bayesian Model for Genetic Inference in Open Ancestral Space, CMU-MLD Technical Report 06-111.	Wed, Apr 4:
	Lecture 22: 4/9/07 Gaussian processes I Bayesian logistic regression Gaussian processes Reading: C. Williams.  Prediction with Gaussian processes: from linear regression to linear prediction and beyond, in ``Learning and Inference in Graphical Models,'' MIT Press, 1999.	Mon, Apr 9:
Advanced topics in Variational Inference	Lecture 23: 4/11/07   Propagation algorithms I Review of BP and GBP Tree-reweighted BP and bounds of the likelihood Reading:	Wed, Apr 11:
	Lecture 24: 4/16/07 Propagation algorithms II Assumed density filtering Expectation propagation Reading:	Mon, Apr 16:
	No class	Apr 18-23
	Lecture 25: 4/25/07  Mean field approximation I Naive mean on Ising model The Latend Dirichlet Allocation model of the Variational Bayesian learning algorithm           Reading:	Wed, Apr 25:
	Lecture 26: 4/30/07 Mean field approximation II The generalized mean field theory Variational message passing Reading: E.P. Xing, M.I. Jordan, and S. Russell, A generalized mean field algorithm for variational inference in exponential families, Uncertainty in Artificial Intelligence (UAI2003). Winn, J. and Bishop, C. (2005). Variational message passing. Journal of Machine Learning Research. 6:661-694	Mon, Apr 30:
	Lecture 27: 5/2/07 Theory of variation inference I: from an optimization-theoretic perspective Duality and Marginal Polytope VI via Log-determinant optimization Reading: Wainwright, M. and Jordan, M. "Variational inference in graphical models: The view from the marginal polytope" (2003). Wainwright, M. and Jordan, M. Graphical models, exponential families, and variational inference. (2003)	Wed, May 2:
	Lecture 28: 5/7/07   Theory of variation inference II: from an information-geometry perspective Alpha-Divergence Devergence-minimization: a unifying view of VI Reading: Tom Minka, Divergence Measures and Message Passing, 2006	Mon, May 7:
	Final Exam	Wed, May 9: Presentation