Information Processing and Learning

10-704, Spring 2015

Akshay Krishnamurthy, Aarti Singh

Teaching Assistant: Kirthevasan Kandasamy
Class Assistant: Sandra Winkler

Lecture:

Tuesday and Thursday, 1:30 - 2:50 pm, Wean 4623

Recitation:

Monday, 6-7 pm, Doherty 2105

Office hrs:

Aarti: Wednesday 12-1pm GHC 8207
Akshay: Friday 3-4pm GHC 7507
Kirthevasan: Monday 4-5pm GHC8015 or 6-7pm DH2105

Course Description:

What's the connection between how many bits we can send over a channel and how accurately we can classify documents or fit a curve to data? Is there any link between decision trees, prefix codes and wavelet transforms? What about max-entropy and maximum likelihood, or universal coding and online learning?

This inter-disciplinary course will explore these and other questions that link the fields of information theory, signal processing, and machine learning, all of which aim to understand the information contained in data. The goal is to highlight the common concepts and establish concrete links between these fields that enable efficient information processing and learning.

We will do a short but introductory review of basic information theory, including entropy and fundamental limits of data compression, data processing and Fano's inequalities, channel capacity, and rate-distortion theory. Then we will dig into the connections to learning including: estimation of information theoretic quantities (such as entropy, mutual information, and divergence) and their applications in learning, information theoretic lower bounds for machine learning problems, duality of max entropy and maximum likelihood, connections between clustering and rate-distortion theory, universal coding and online learning, active learning and feedback channels, and more.

We expect that this course will cater to both students that have taken a basic information theory course and those that have not.

Emails:

The class mailing list is 10704-announce@cs. If you wish to email only the instructors, the email is 10704-instructors@cs
We have added all those who registered for the course to the mailing list. If you are NOT receiving these announcements, you can subscribe via the 10704-announce list page

Prerequisites:

Fundamentals of Probability, Statistics, Linear Algebra and Real analysis

Textbooks:

Elements of Information Theory, T.M. Cover and J.A. Thomas.
Information Theory, Inference and Learning Algorithms, David J.C. MacKay.
Modern Coding Theory, A. Montanari and R. Urbanke

Grading:

Homeworks (40%)
Project (25%)
Online Q&A (10%)
Two Short Quiz (20%)
Scribing (5%)

Homeworks:

All homeworks, quizzes and solutions are posted here.

Course Project:

Information about the course project is available here.

Schedule of Lectures:

Lecture schedule, scribed notes and HWs are available here.

Tentative Schedule:

Week 1:	Basics of information theory
Week 2:	Some applications in machine learning
Week 3:	Estimation of information theoretic quantities
Week 4:	Maximum entropy distributions and exponential families, I-geometry
Week 5:	Source coding and compression
Week 6:	Model selection and connections to source coding
Week 7:	Universal source coding and online learning
Week 8:	Sufficient statistics, Information Bottleneck principle
Week 9:	Channel coding and Redundancy-Capacity Theorem
Week 10:	Fano's Inequality and minimax theory
Week 11:	Minimax lower bounds for parametric and nonparametric problems
Week 12:	Strong data processing inequalities and minimax lower bounds
Week 13:	Classification and hypothesis testing
Week 14:	Graphical models, LDPC codes, and active learning