This distribution contains a corpus of Arabic-English parallel sentences, which
were extracted automatically from two monolingual corpora: Arabic Gigaword Second
Edition (LDC2006T02) and English Gigaword Second Edition (LDC2005T12). The data
was extracted from news articles published by Xinhua News Agency and Agence
France Presse and was obtained using the automatic parallel sentence identification
method described in the following publication: Dragos Stefan Munteanu, Daniel
Marcu, 2005. Machine
Translation Performance by Exploiting Non-parallel Corpora, Computational Linguistics,
The corpus contains 1,124,609 sentence pairs; the word count on the English side is approximately 31M words. The sentences in the parallel corpus preserve the form and encoding of the texts in the original Gigaword corpora.
For each sentence pair in the corpus the authors provide the names of the documents
from which the two sentences were extracted, as well as a confidence score (between
0.5 and 1.0), which is indicative of their degree of parallelism. The parallel
sentence identification approach is designed to judge sentence pairs in isolation
from their contexts, and can therefore find parallel sentences within document
pairs which are not parallel. The fact that two documents share several parallel
sentences does not necessarily mean the documents are parallel.
In order to make this resource useful for research in Machine Translation (MT),
the authors made efforts to detect potential overlaps between this data and the standard
test and development data sets used by the MT community. The NIST 2002-2005
MT evaluation data sets contain several articles from Xinhua News Agency and
Agence France Presse. Sentence pairs in this distribution that have a 7-gram
overlap with a sentence pair in a NIST MT evaluation set or sentence pairs coming
from documents whose names are similar to those in the NIST MT sets are marked
with a negative confidence score.
For an example of the data in this publication, please examine this image of text data.
Portions © 1994-2004 Agence France Presse, © 1995-2004 Xinhua News
Agency, © 2005, 2006, 2007 Trustees of the University of Pennsylvania