Global Voices Malagasy-English Parallel Corpus

Download - release 12.06

Usage

• Simple utilities to convert to text are available on Github:
  git clone https://github.com/vchahun/teny.git
pip install -r teny/requirements.txt # install lxml
• To extract text from the XML format, use the totext.py tool:
  bzcat train.xml.bz2 | python teny/totext.py mlg > train.mlg.txt # extract the Malagasy text
bzcat train.xml.bz2 | python teny/totext.py eng > train.eng.txt # extract the English text

• To split the full corpus into train/development/evaluation, use split.py:
  bzcat corpus.xml.bz2 | python teny/split.py 01 03 > test.xml # select days 1-3
bzcat corpus.xml.bz2 | python teny/split.py 04 04 > dev.xml # select day 4
bzcat corpus.xml.bz2 | python teny/split.py 05 31 > train.xml # select days 5-31
• To obtain MT versions of the development/evaluation corpora, use sample.py:
  bzcat test.xml.bz2 | python teny/sample.py 11 > mt-test.xml # select ~1/11 document
bzcat dev.xml.bz2 | python teny/sample.py 2 > mt-dev.xml # select ~1/2 document

Details

More data will be released periodically (~100 articles are published every month on Global Voices)

• Approximate size of the complete corpus: 3,000 documents / 100,000 sentences / 2M English words.
• Days 1-3/4/5-31 of every month were selected for evaluation/development/training.
• The smaller versions for MT development/evaluation are sampled by choosing documents with identifier ≣ 0 (mod n) where n is selected to reduce the size of the sample to ~1000 sentences.

Sentence aligner used: Gargantua (Improved unsupervised sentence alignment for symmetrical and asymmetrical parallel corpora, F. Braune & A. Fraser, COLING 2010)

License

The original content was published under a Creative Commons Attribution-Only license.

Acknowledgments

This work was supported by the Army Research Office (grant number W911NF-10-1-0533).