15-849 | Schedule

Date	Plan	Readings	Presenters
01/17 Mon	Week 1: Martin Luther King Day; No Class [ slides ]
01/19 Wed	Week 1: Course Introduction [ slides ]	(Required) How to Read a Paper
01/24 Mon	Week 2: Introduction to Deep Neural Networks [ slides ]
01/26 Wed	Week 2: Introduction to Deep Learning Systems [ slides ]
01/31 Mon	Week 3: Systems for Deep Learning [ slides ]	(Required) TensorFlow: A System for Large-Scale Machine Learning (Required) PyTorch: An Imperative Style, High-Performance Deep Learning Library (Optional) JANUS: Fast and Flexible Deep Learning via Symbolic Graph Execution of Imperative Programs	Zhihao Jia, Junli Cao
02/02 Wed	Week 3: Systems for Classical ML [ slides ]	(Required) Scaling Distributed Machine Learning with the Parameter Server (Required) XGBoost: A Scalable Tree Boosting System (Optional) A Tensor Compiler for Unified Machine Learning Prediction Serving	Giulio Zhou, Zhihao Zhang
02/07 Mon	Week 4: ML Compilation [ slides ]	(Required) TVM: An Automated End-to-End Optimizing Compiler for Deep Learning (Required) Halide: Decoupling Algorithms from Schedules for Easy Optimization of Image Processing Pipelines (Optional) MLIR: A Compiler Infrastructure for the End of Moore's Law	Ashwin Venkatram, Bohan Hou
02/09 Wed	Week 4: ML Compilation [ slides ]	(Required) Learning to Optimize Tensor Programs (Required) Ansor: Generating High-Performance Tensor Programs for Deep Learning (Optional) Tensor Comprehensions: Framework-Agnostic High-Performance Machine Learning Abstractions	Elias Jaeaesaari, Yijia Zhang
02/14 Mon	Week 5: Graph-Level Optimizations [ slides ]	(Required) TASO: Optimizing Deep Learning Computation with Automatic Generation of Graph Substitutions (Required) PET: Optimizing Tensor Programs with Partially Equivalent Transformations and Automated Corrections (Optional) Equality Saturation for Tensor Graph Superoptimization	Zhihao Jia
02/16 Wed	Week 5: RL for Device Placement and Graph Optimizations [ slides ]	(Required) Device Placement Optimization with Reinforcement Learning (Required) Transferable Graph Optimizers for ML Compilers (Optional) A Hierarchical Model for Device Placement	Zhihao Jia, Yuanxin Wang
02/21 Mon	Week 6: Data and Model Parallelism for Distributed Training [ slides ]	(Required) Beyond Data and Model Parallelism for Deep Neural Networks (Required) GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding (Optional) GSPMD: General and Scalable Parallelization for ML Computation Graphs (Optional) Ray: A Distributed Framework for Emerging AI Applications	Bowen Chen, Jiajun Wan
02/23 Wed	Week 6: Pipeline Parallelism for Distributed Training [ slides ]	(Required) PipeDream: Generalized Pipeline Parallelism for DNN Training (Required) GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism (Optional) Memory-Efficient Pipeline-Parallel DNN Training	Weizheng Xu, Yuxuan Zheng
02/28 Mon	Week 7: Memory-Efficient Training: Tensor Rematerialization [ slides ]	(Required) Training Deep Nets with Sublinear Memory Cost (Required) Checkmate: Breaking the Memory Wall with Optimal Tensor Rematerialization (Optional) Dynamic Tensor Rematerialization	Erhu He, Senyu Tong
03/02 Wed	Week 7: Memory-Efficient Training: Zero Redundancy [ slides ]	(Required) ZeRO: Memory Optimizations Toward Training Trillion Parameter Models (Required) ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning (Optional) ZeRO-Offload: Democratizing Billion-Scale Model Training	Siyu Chen, Arav Agarwal
03/07 Mon	Week 8: Spring Break; No Class [ slides ]
03/09 Wed	Week 8: Spring Break; No Class [ slides ]
03/14 Mon	Week 9: Transformers and Language Models [ slides ]	(Required) Transformer: Attention is All you Need (Required) BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (Optional) Language Models are Unsupervised Multitask Learners	Tony Xi, Vincent Xu
03/16 Wed	Week 9: Guest Lecture: Deepak Narayanan (Microsoft Research) [ slides ]	(Required) Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM	Deepak Narayanan
03/21 Mon	Week 10: Graph Neural Networks [ slides ]	(Required) Semi-Supervised Classification with Graph Convolutional Networks (Required) Inductive Representation Learning on Large Graphs (Optional) Relational inductive biases, deep learning, and graph networks	Hui Chen, Paola A Buitrago
03/23 Wed	Week 10: Guest Lecture: Autotuning Production ML Compilers [ slides ]	(Required) A Flexible Approach to Autotuning Multi-pass Machine Learning Compilers (Optional) A Learned Performance Model for Tensor Processing Units	Phitchaya Mangpo Phothilimthana (Google Brain)
03/28 Mon	Week 11: Guest Lecture: Practical Reinforcement Learning on the Internet [ slides ]	(Required) Learning in situ: a randomized experiment in video streaming	Francis Yan (Microsoft Research)
03/30 Wed	Week 11: Model Compression [ slides ]	(Required) Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding (Required) The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks (Optional) Comparing Rewinding and Fine-tuning in Neural Network Pruning	Rulin Shao, Muyang Li
04/04 Monday	Week 12: ML Hardware [ slides ]	(Required) Fast Algorithms for Convolutional Neural Networks (Required) In-Datacenter Performance Analysis of a Tensor Processing Unit (Optional) Design Automation for Efficient Deep Learning Computing	Balamurugan Marimuthu
04/06 Wed	Week 12: Guest Lecture: Disruptive Research on Distributed ML Systems [ slides ]	(Required) sensAI: ConvNets Decomposition via Class Parallelism for Fast Inference on Live Data (Optional) Blink: Fast and Generic Collectives for Distributed ML	Guanhua Wang (UC Berkeley)
04/11 Mon	Week 13: ML for systems [ slides ]	(Required) The Case for Learned Index Structures (Required) Ithemal: Accurate, Portable and Fast Basic Block Throughput Estimation using Deep Neural Networks (Optional) Chip Placement with Deep Reinforcement Learning	Prasoon Patidar, Sudershan Boovaraghavan
04/13 Wed	Week 13: Federated learning [ slides ]	(Required) Communication-Efficient Learning of Deep Networks from Decentralized Data (Required) Towards Federated Learning at Scale: System Design (Optional) Federated Multi-Task Learning	Charlie Hou
04/18 Mon	Week 14: Guest Lecture: The library to accelerate training and inference of DNN at scale [ slides ]		Minjia Zhang (Microsoft)
04/20 Wed	Week 14: Final Presentation [ slides ]
04/25 Mon 04/27 Wed	Week 15: Final Presentations

15-849: Machine Learning Systems

Schedule