[Paper Reading]A Bit of Progress in Language Modeling

11731, Machine Translation, Paper reading
  • Smoothing
    • Simple interpolation
      $$ \scriptsize P_{interpolate}(w\;\vert\;w_{i-2}w_{i-1})\;=\;\lambda P_{trigram}(w\;\vert\;w_{i-2}w_{i-1})\;+\;(1-\lambda)(\mu P_{bigram}(w\;\vert\;w_{i-1})\;+\;(1-\mu)P_{unigram}(w)) $$
    • Katz smoothing
      • Based on Good-Turing formula, if count > 0
        $$ n_r\;is\;the\;frequency\;of\;frequency $$
      • back off, otherwise
    • Backoff Kneser-Ney smoothing
    • Interpolated Kneser-Ney smoothing
Share