This file contains the documentation for GALE Phase 1 Arabic
Blog Parallel Text, Linguistic Data Consortium (LDC)
catalog number LDC2008T02, ISBN 1-58563-462-X.
Blogs are posts to informal web-based journals of varying topical content.
GALE Phase 1 Arabic Blog Parallel Text was prepared by the LDC and consists
of 102K words (222 files) of Arabic blog text and its English translation from
thirty-three sources. This release was used as training data in Phase 1 of the
DARPA-funded GALE program.
The task of preparing this corpus involved four stages of work: data scouting,
data harvesting, formatting, and data selection.
Data scouting involved manually searching the web for suitable blog text. Data
scouts were assigned particular topics and genres along with a production target
in order to focus their web search. Formal annotation guidelines and a customized
annotation toolkit helped data scouts to manage the search process and to track
Data scouts logged their decisions about potential text of interest (sites,
threads and posts) to a database. A nightly process queried the annotation database
and harvested all designated URLs. Whenever possible, the entire site was downloaded,
not just the individual thread or post located by the data scout.
Once the text was downloaded, its format was standardized (by running various
scripts) so that the data could be more easily integrated into downstream annotation
processes. Original-format versions of each document were also preserved. Typically
a new script was required for each new domain name that was identified. After
scripts were run, an optional manual process corrected any remaining formatting
The selected documents were then reviewed for content suitability using a
semi-automatic process. A statistical approach was used to rank a documents
relevance to a set of already-selected documents labeled as good. An annotator
then reviewed the list of relevance-ranked documents and selected those which
were suitable for a particular annotation task or for annotation in general.
Those newly-judged documents in turn provided additional input for the generation
of new ranked lists.
Manual sentence units/segments (SU) annotation was also
performed on a subset of files following LDCs
Quick Rich Transcription specification.
Three types of end of sentence SU are
- statement SU
- question SU
- incomplete SU
After files were selected, they were reformatted into a human-readable translation
format, and the files were then assigned to professional translators for careful
translation. Translators followed LDCs GALE Translation guidelines, which describe
the makeup of the translation team, the source, data format, the translation
data format, best practices for translating certain linguistic features (such
as names and speech disfluencies), and quality control procedures applied to
Translators were instructed to return a 50-sentence sample as soon as it was
completed. The sample was reviewed by LDCs bilingual language specialists.
Subsequent deliveries were subject to quality controls as described in the translation
guidelines. Low quality translations were returned to the translators for revision.
All final data are in Tab Delimited Format (TDF). TDF is
compatible with other transcription formats, such as the Transcriber
format and AG format, and it is easy to process.
Each line of a TDF file corresponds to a speech segment and
contains 13 tab delimited fields:
1 file unicode
2 channel int
3 start float
4 end float
5 speaker unicode
6 speakerType unicode
7 speakerDialect unicode
8 transcript unicode
9 section int
10 turn int
11 segment int
12 sectionType unicode
13 suType unicode
A source TDF file and its translation are the same except that
transcript in the source TDF is replaced by its English translation.
All data are encoded in UTF8.
This work was supported in part by the Defense Advanced Research Projects Agency,
GALE Program Grant No. HR0011-06-1-0003. The content of this publication does
not necessarily reflect the position or the policy of the Government, and no
official endorsement should be inferred.
For an example of the data in this corpus, please examine these screen captures(jpg) of the text:
Portions © 2005-2007, 2008 Trustees of the University of Pennsylvania