Multiple-Translation Arabic (MTA) Part 1 was produced by Linguistic Data Consortium (LDC) catalog number LDC2003T18 and ISBN 1-58563-276-7.
To support the development of automatic means for evaluating translation
quality, the LDC was sponsored to solicit ten sets of human translations for a
single set of Arabic source materials. The LDC was also asked to produce
translations from various commercial-off-the-shelf-systems (COTS, including
commercial Machine Translation (MT) systems as well as MT systems available on
There are a total of two sets of COTS outputs and one output set from a TIDES 2002 MT Evaluation participant,
which is representative for the state-of-the-art research systems.
To see if automatic evaluation systems such as BLEU track human assessment,
the LDC has also performed human assessment on the two COTS outputs and the TIDES research
system. The corpus includes the assessment results for one of the two COTS systems, the assessment results for the
TIDES research system, and the specifications used for conducting the assessments.
Source Data Selection
Two sources of journalistic Arabic text were selected to provide the Arabic material:
- Xinhua News Service: 66 news stories (files: artb_500 - artb_565)
- AFP News Service: 75 news stories (files: artb_S01 - artb_S06, artb_001 - artb_069)
(total: 141 stories)
There are 141 source files, and 1,792 translation files (12 of the 13 systems produced translations for all 141 source files,
while one system produced translations for only 100 of the 141 Arabic stories).
The Xinhua data was drawn from the Xinhua News Agency's Arabic newswire
feed in October 2001. The AFP Data was drawn from the LDC's Arabic Newswire Part 1).
The story selection from the two newswire collections was controlled by story
length: all selected stories contain between 700 and 1,500 Arabic characters.
The overall count of Arabic words (excluding markup) is shown in the following table by source:
For the Arabic data, there are approximately 23K-words, while for the English translations, there are 366K-words in total and 163K unique words.
Source Data Preparation for Human Translation
The original source files used CP-1256 encoding for the Arabic characters, and
SGML tags for marking sentence and paragraph boundaries and other information
about each story. The source files were later converted to UTF8 encoding. To make things easier for the translators,
nearly all sgml tags were removed or replaced by "plain text" markers.
Human Translation Procedure and Quality Assessment
Each initially selected translation team received the translation guidelines
and a sample pair of source and translation (excluded from the final release)
for review. After the team said that they understood the task requirements and would be willing to participate
in the project, 75 AFP news stories were sent to them as a first installment
In accordance with the guidelines, each translation team was asked to return
the first six AFP stories for quality checking.
This was to ensure that the translation team had indeed understood and
was following the guidelines and the translation quality was acceptable. The
LDC sent the translations back to the translation team for any deviations from
the guidelines or quality issues detected.
Subsequent translation submissions were continuously monitored for conformance
and quality. Once the full set of translations was complete, a final pass of reformatting and validation was
carried out, to assure alignability of segments, and to convert the translated
texts into SGML format.
Each translation team was also asked to fill out and return a questionnaire to
describe their procedures and professional background.
Machine Translation Procedure
Complete sets of automatic MT translations were also produced by submitting
the 141 stories to each of the two publicly-available MT systems.
Starting from the original SGML text format, special alterations were made to
the files on an as-needed basis, so that they would be accepted and handled correctly by the various systems. Also, the systems differed in
terms of the input and retrieval methods required to submit the source data
for translation and to save the translated text in alignable form.
Human Assessment Procedure
The goal of this effort is to evaluate the quality of TIDES
research, human translation teams and commercial
off-the shelf (COTS) systems. Translations are evaluated on the basis of
adequacy and fluency. Adequacy
refers to the degree to which the translation communicates information present
in the original source language text. Fluency refers to the degree to which
the translation is well-formed
according to the grammar of the target language.
There are no updates available at this time.
Portions © 2001-2002 Xinhua News Agency, © 1998-2000 Agence France Press, © 2003 Trustees of the University of Pennsylvania