Language Model File Formats

BINDUMP File Format

Whenever a Nist file with more than 1000 Bigrams is read, a filename.BINDUMP is created with a binary, machine independent, fast-load version of this language model. The BINDUMP contains all the information in the original language model, but can be up to 10 times smaller and load 20 times faster. Upon loading a Nist file, the system first checks for a BINDUMP file and reads it rather than the extended version. This is important to know as if you edit a nist file you will only see the changes if you delete the BINDUMP file before you load the lm the next time.

Both, BINDUMP and normal files can be read from files compressed with gnuzip. The ending .gz is found automatically, don't add it to the filename.

Nist File Format

This is the standart nist format for language models. Also read the section on the BINDUMP format, especially when editing Nist files. Everything up to the line marked with \data\ is a comment. Next come the number of unigrams, bigrams and trigrams. When you build your own language models, always make sure these numbers are exact or JANUS will stop with an error message.

<s> is the marker for the beginning of an utterance.
</s> is the marker for the end of an utterance.
The value in the first column ist the log monogram, bigram and trigram probability. The value in the last column is the backoff value, that is used in case the corresponding bigram (in case of the monogram section) or trigram (in case of the bigram section) is unseen.

A file always has at least one unigram and one bigram. Trigrams can be missing. If there are no trigrams, there should not be backoff-values for bigrams either.


logP(w1,w2,w3) 
     = logP(w1,w2,w3)               if trigram w1,w2,w3 in list
     = backoff(w1,w2) + logP(w2,w3) if bigram w2,w3 in list
     = backoff(w2) + logP(w3)       else

Example:

This is a silly little nist file. Put in here when and where
the file was built, and if possible on which database using
which program.

\data\
ngram 1=4
ngram 2=4
ngram 3=3

\1-grams:
-99.0 <s>   0.0
-00.0 </s> -99.0
-3.5  ja   -1.5
-3.5  nein -1.5

\2-grams:
-00.0 <s>  </s> -0.5
-99.0 </s> <s>  -0.6
-4.2  <s>  ja   -0.5
-6.7  ja  </s>  -0.5

\3-grams:
-3.2 ja ja </s>
-2.3  <s> ja ja
-3.3  nein ja ja

\end\