CMU Artificial Intelligence Seminar Series ^{sponsored by}

Tuesday, April 5, 2022

Time: 12:00 - 01:00 PM ET

Uri Alon -- Neuro-Symbolic Language Modeling with Automaton-augmented Retrieval

Relevant Paper(s):

• Neuro-Symbolic Language Modeling with Automaton-augmented Retrieval

Abstract: Retrieval-based language models (R-LM) model the probability of natural language text by combining a standard language model (LM) with examples retrieved from an external datastore at test time. While effective, a major bottleneck of using these models in practice is the computationally costly datastore search, which can be performed as frequently as every time step. In this talk, I will present RetoMaton - retrieval automaton - which approximates the datastore search, based on (1) clustering of entries into "states", and (2) state transitions from previous entries. This effectively results in a weighted finite automaton built on top of the datastore, instead of representing the datastore as a flat list. The creation of the automaton is unsupervised, and a RetoMaton can be constructed from any text collection: either the original training corpus or from another domain. Traversing this automaton at inference time, in parallel to the LM inference, reduces its perplexity, or alternatively saves up to 83% of the nearest neighbor searches over kNN-LM (Khandelwal et al., 2020), without hurting perplexity.