CMU Artificial Intelligence Seminar Series sponsored by Fortive
Tuesday, Apr 30, 2021
Time: 12:00 - 01:00 PM ET
Noah Smith -- Language Models: Challenges and Progress
Relevant Paper(s):
- GENIE: A Leaderboard for Human-in-the-Loop Evaluation of Text Generation
- RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models
- Grounded Compositional Outputs for Adaptive Language Modeling
- Random Feature Attention
Abstract: Probabilistic language models are once again foundational to many advances in natural language processing research, bringing the exciting opportunity to harness raw text to build language technologies. With the emergence of deep architectures and protocols for finetuning a pretrained language model, many NLP solutions are being cast as simple variations on language modeling. This talk is about challenges in language model-based NLP and some of our work toward solutions. First, we'll consider evaluation of generated language. I'll present some alarming findings about humans and models and make some recommendations. Second, I'll turn to an ubiquitous design limitation in language modeling -- the vocabulary -- and present a linguistically principled, sample-efficient solution that enables modifying the vocabulary during finetuning and/or deployment. Finally, I'll delve into today's most popular language modeling architecture, the transformer, and show how its attention layers' quadratic runtime can be made linear without affecting accuracy. Taken together, we hope these advances will broaden the population of people who can effectively use and contribute back to NLP.
Bio: Noah Smith is a Professor in the Paul G. Allen School of Computer Science & Engineering at the University of Washington, as well as a Senior Research Manager at the Allen Institute for Artificial Intelligence. Previously, he was an Associate Professor of Language Technologies and Machine Learning in the School of Computer Science at Carnegie Mellon University. He received his Ph.D. in Computer Science from Johns Hopkins University in 2006 and his B.S. in Computer Science and B.A. in Linguistics from the University of Maryland in 2001. His research interests include statistical natural language processing, machine learning, and applications of natural language processing, especially to the social sciences. His book, Linguistic Structure Prediction, covers many of these topics. He has served on the editorial boards of the journals Computational Linguistics (2009-2011), Journal of Artificial Intelligence Research (2011-present), and Transactions of the Association for Computational Linguistics (2012-present), as the secretary-treasurer of SIGDAT (2012-2015 and 2018-present), and as program co-chair of ACL 2016. Alumni of his research group, Noah's ARK, are international leaders in NLP in academia and industry; in 2017 UW's Sounding Board team won the inaugural Amazon Alexa Prize. He was named an ACL Fellow in 2020, "for significant contributions to linguistic structure prediction, computational social sciences, and improving NLP research methodology." Smith's work has been recognized with a UW Innovation award (2016-2018), a Finmeccanica career development chair at CMU (2011-2014), an NSF CAREER award (2011-2016), a Hertz Foundation graduate fellowship (2001-2006), numerous best paper nominations and awards, and coverage by NPR, BBC, CBC, New York Times, Washington Post, and Time.