NIST 2005 Open Machine Translation (OpenMT) Evaluation, Linguistic
Data Consortium (LDC) catalog number LDC2010T14 and isbn
1-58563-556-1, is a package containing source data, reference
translations, and scoring software used in the NIST 2005 OpenMT
evaluation. It is designed to help evaluate the effectiveness of
machine translation systems. The package was compiled and scoring
software was developed by researchers at NIST, making use of
newswire source data and reference translations collected and
developed by LDC.
The objective of the NIST OpenMT evaluation series is to support
research in, and help advance the state of the art of, machine
translation (MT) technologies -- technologies that translate text
between human languages. Input may include all forms of text. The
goal is for the output to be an adequate and fluent translation of
The MT evaluation series started in 2001 as part of the DARPA TIDES
(Translingual Information Detection, Extraction) program. Beginning
with the 2006 evaluation, the evaluations have been driven and
coordinated by NIST as NIST OpenMT. These evaluations provide an
important contribution to the direction of research efforts and the
calibration of technical capabilities in MT. The OpenMT evaluations
are intended to be of interest to all researchers working on the
general problem of automatic translation between human languages. To
this end, they are designed to be simple, to focus on core
technology issues, and to be fully supported. The 2005 task was to
evaluate translation from Chinese to English and from Arabic to
Additional information about these evaluations may be found at
the NIST Open
Machine Translation (OpenMT) Evaluation web site.
This evaluation kit includes a single Perl script (mteval-v11b.pl)
that may be used to produce a translation quality score for one (or
more) MT systems. The script works by comparing the system output
translation with a set of (expert) reference translations of the same
source text. Comparison is based on finding sequences of words in the
reference translations that match word sequences in the system output
translation. More information on the evaluation algorithm may be
obtained from the paper detailing the
a Method for Automatic Evaluation of Machine Translation (Papineni et
The included scoring script was released with the original
evaluation, intended for use with SGML-formatted data files, and is
provided to ensure compatibility of user scoring results with results
from the original evaluation. An updated scoring software package
(mteval-v13a-20091001.tar.gz), with XML support, additional options
and bug fixes, documentation, and example translations, may be
the NIST Multimodal
Information Group Tools website.
This corpus consists of 100 Arabic newswire documents, 100 Chinese
newswire documents, and a corresponding set of four separate human
expert reference translations. Source text for both languages was
collected from Agence France-Presse and Xinhua News Agency in December
2004 and January 2005.
The reference translations included in this corpus have not
previously been publicly available. Arabic source text from December
2004 has been available in LDC's Arabic Gigaword releases beginning
Edition (LDC2006T02), and from January 2005 beginning with
Edition (LDC2007T40). Chinese source text from Xinhua December
2004 has been available in LDC's Chinese Gigaword releases beginning
Edition (LDC2005T14), and from Xinhua January 2005 and AFP
For each language, the test set consists of two files: a source and
a reference file. Each reference file contains four independent
translations of the data set. The evaluation year, source language,
test set (which, by default, is "evalset"), version of the
data, and source vs. reference file (with the latter being indicated
by "-ref") are reflected in the file name.
DARPA TIDES MT and NIST OpenMT evaluations used SGML-formatted test
data until 2008 and XML-formatted test data thereafter. The files in
this package are provided in both formats.
Sample text file
containing excerpts from different xml files included in this corpus,
including reference translations and source text for a single newswire
document. The file is encoded in UTF-8.
No updates are available at this time.
Portions © 2004-2005 Agence France Presse, ©
2004-2005 Xinhua News Agency, © 2005, 2006, 2007, 2010
Trustees of the University of Pennsylvania.