Instructor: Avrim Blum (Wean 4107, avrim@cs.cmu.edu, x8-6452).

Office Hours: Stop by or make an appointment.

Credits: 12 Units

Course Description: This course will focus on theoretical aspects of machine learning. We will examine questions such as: What kinds of guarantees can one prove about learning algorithms? What are good algorithms for achieving certain types of goals? Can we devise models that are both amenable to mathematical analysis and make sense empirically? What can we say about the inherent ease or difficulty of learning problems? Addressing these questions will require pulling in notions and ideas from statistics, complexity theory, cryptography, game theory, and empirical machine learning research.

Prerequisites: Either 15-681/781 Machine Learning, or 15-750 Algorithms, or a strong Theory/Algorithms background.

Evaluation and Responsibilities: Grading will be based on 5 or 6 homework assignments, a take-home final, class participation, and a class presentation or project.  Because this course has no TA, students from time to time will also be asked to help with the grading of assignments.

General structure of the course: We will use roughly 2/3 of the lectures to cover "core" topics in this area, and then will diverge in the remaining 1/3 based on student interest. Here is a rough outline of the "core" portion (some bullets correspond to multiple lectures):

• Basic models: consistency, PAC, and Mistake-bound
• simple algorithms and hardness results
• the Weighted-Majority Algorithm
• the Winnow Algorithm and the Infinite-Attribute model
• Occam's razor: when can we be confident about our predictions?
• VC-dimension, uniform convergence
• Margins and support-vector machines
• Weak-learning vs. Strong-learning
• Classification noise and the Statistical-Query model
• Fourier analysis
• Algorithmic hardness results, relations to cryptography
• Active learning / membership queries

Text: Kearns and Vazirani, "An introduction to computational learning theory" plus papers and notes for topics not in the book. (Roughly half of the topics are in the book)