Kurdish Multilingual Parallel Corpus

Aligned corpora are essential resources with various applications in natural language processing, particularly in Machine Translation. Such a resource is composed of pairs of sentences or chunks of sentences in two languages or dialects which are translations and thus, useful to train a model for automatic translation. One can imagine how tedious and expensive the task of manual translation can be!

Our approach to automatically retrieve identical news articles

Kurdish, as a less-resourced language, lacks large parallel corpora which are required for further progress in creating statistical and neural machine translation systems. To tackle this problem in a viable way, we create a large parallel text for Kurmanji and Sorani dialects of Kurdish and extend it to include Kurdish-English translations using multilingual news websites. Although there are many multilingual and multi-dialectal news websites for Kurdish, none of them provide their content in an inter-operable way. In other words, an identical news article is translated differently and oftentimes, the translations are not even linked together.

Parallel corpus

Our parallel corpus contains three manually-aligned corpus in Sorani-Kurmanji, Sorani-English and Kurmanji-English in various formats, namely Translation Memory eXchange file format (.tmx), parallel annotated text useful for ParaConc and raw parallel texts (.txt). In the latter, each line corresponds to the same line in the other aligned file. This corpus contains 12,327 translation pairs in the two major dialects of Kurdish, Sorani and Kurmanji. We also provide 1,797 and 650 translation pairs in English-Kurmanji and English-Sorani.

It should be noted that machine translation methods, particularly those based on neural networks, have voracious appetite for data. As a consequence, these datasets may be more suitable for evaluation purposes, fine-tuning or even for trainings for low-resource setups.

The following shows a sample of the corpus:

Nota bene

• If you are interested in machine translation in general, and in Kurdish language processing, in particular, check out this paper entitled Towards Machine Translation for the Kurdish Language.
• Shortly after this project in August 2020, the Microsoft Translation Service added Sorani and Kurmanji to its service. See https://www.bing.com/translator.
• Our approach in this project can be applied to any other language/dialect, or even further enrich the resource for Kurdish. If you are interested in the task, please check out the annotation guidelines at here.
• For the alignment task, check out InterText. Check out this short tutorial video.

Cite this corpus

Download this parallel corpus at https://github.com/KurdishBLARK/InterdialectCorpus. If you use this resource, please cite the following paper:

@misc{ahmadi2020leveraging,
      title={Leveraging Multilingual News Websites for Building a Kurdish Parallel Corpus}, 
      author={Sina Ahmadi and Hossein Hassani and Daban Q. Jaff},
      year={2020},
      eprint={2010.01554},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}