July 2010 Newsletter

New Corpora

LDC Standard Arabic Morphological Analyzer (SAMA) Version 3.1

NIST 2004 Open Machine Translation (OpenMT) Evaluation

Announcements

2010 Publication Pipeline Update
Membership Year (MY) 2010 has included a strong selection of publications including updates to the Arabic and Chinese treebanks, Spanish telephone speech and transcript data from the Fisher collection, and Chinese word n-grams collected from the web. Please consult our corpus catalog for a full list of publications distributed by LDC. As we are now in the second half of this membership year, we would like to provide information on what publications you can expect for the remainder of MY2010. Our pipeline includes the following:

Arabic Treebank Part 1 Version 4.1 ~ a revision of Arabic Treebank: Part 1 v 3.0 (POS with full vocalization + syntactic analysis) (LDC2005T02) (ATB1), according to the new Arabic Treebank (ATB) annotation guidelines. The Arabic Treebank project consists of two distinct phases: (a) Part-of-Speech (POS) tagging which divides the text into lexical tokens, and gives relevant information about each token such as lexical category, inflectional features, and a gloss, and (b) Arabic Treebanking which characterizes the constituent structures of word sequences, provides categories for each non-terminal node, and identifies null elements, co-reference, traces, etc. on-terminal node. Arabic Treebank Part 1 Version 4.1 represents the manual revision of the syntactic tree annotation in ATB1, the automatic revision and updating of certain part-of-speech tags, and the manual revision of certain targeted POS tags (function words, in particular). The source data consists of 734 newswire stories from Agence France Presse.

Microsoft Research India POS-Tagged Bengali - to support the task of Part-of-Speech Tagging (POS) and other forms of data-driven linguistic research on Indian languages in general, Microsoft Research India has developed POS labeled data for Hindi, Bengali, and Sanskrit as a part of the Indian Language – Part-of-Speech Tagset (IL-POST) project. The corpora are based on the IL-POST framework. IL-POST is a POS-tagset framework which has been designed to cover the morph-syntactic details of Indian languages. It supports a three-level hierarchy of Categories, Types and Attributes. The Bengali corpus consists of two different levels of information for each lexical token: (a) lexical category and types, and (b) set morphological attributes and their associated values in the context. The data consists of 7168 manually annotated sentences (102933 words) targeted to cover written modern standard Bengali from various sources, including blogs, Multikulti, and Wikipedia. .

TRECVID 2006 Keyframes and Transcripts ~ TREC Video Retrieval Evaluation (TRECVID) is sponsored by NIST to promote progress in content-based retrieval from digital video via open, metrics-based evaluation. The keyframes in this release were extracted for use in the NIST TRECVID 2006 Evaluation.  The source data includes approximately 158.6 hours of English, Arabic and Chinese language video data collected by LDC from NBC, CNN, MSN, New Tang Dynasty TV, Phoenix TV, Lebanese Broadcasting Corp.,  and China Central TV.  The keyframes were selected by going to the middle frame of the shot boundary, then parsing left and right of that frame to locate the nearest I-Frame. This then became the keyframe and was extracted. Keyframes have been provided at both the subshot (NRKF) and master shot (RKF) levels.

Uda Walawe Asian Elephant Vocalizations ~ partially-annotated corpus of Asian Elephant communication/vocalization. The data set contains vocalizations primarily by adult female and juvenile Asian elephants. This corpus is intended to enable researchers in acoustic communication of elephants and other species to compare acoustic features and repertoire diversity to this population. Of particular interest is whether there may be regional dialects that differ among Asian elephant populations in the wild and in captivity. A second interest is in whether structural commonalities exist between this and other species that shed light on underlying social and ecological factors shaping communication systems.

2010 Subscription Members are automatically sent all MY2010 data as it is released.  2010 Standard Members are entitled to request 16 corpora for free from MY2010.   Non-members may license most data for research use.

Updated LDC Data Sheets and Papers Pages
LDC is pleased to announce that both our LDC Data Sheets and LDC Papers pages recently have been updated.  On our Data Sheets page, you'll find our growing collection of LDC Data Sheets, each of which highlights a key aspect of the Consortium’s research and development tasks.  Recent additions include a data sheet covering Arabic and English treebanking at LDC and one that provides an overview of LDC's role in sponsored projects.   Our updated papers page contains several papers from LREC2010:  Seventh International Conference on Language Resources and Evaluation, as well as other conferences and journals, dating from 1998 forward.  Most papers are available for download in pdf format; presentations slides and posters are available for several papers as well.

On our Papers page, you can read about LDC's efforts to apply treebank annotation to Arabic broadcast news (Maamouri et al).  Broadcast news (BN) transcript data posed new challenges; for instance, the transcript data included metadata which conveys information in addition to the text of what is being said.   Some forms of metadata were ignored, such as indications of coughs or laughter, while others, such as speech effects including discourse markers and word fragments, were annotated.  Annotators also had to handle indistinct audio signal wherein speech could be heard, but not fully understood, so the words could only be inferred from context rather than from the audio signal.  In these cases, the annotation must convey information not contained in the audio signal that accounts for the annotation in that region. The improved Arabic Treebank (ATB) pipeline and revised annotation guidelines proved robust enough to carry out this task with few changes. This paper discusses where some adaptation was necessary and describes the overall pipeline as used in the production of BN ATB data.

Additionally, you can learn about LDC's role in resource creation for the Knowledge Base Population (KBP) Track of the Text Analysis Conference (TAC) organized by NIST (Simpson et al).  The KBP track of TAC is a hybrid descendant of the TREC Question Answering track and the Automated Content Extraction (ACE) evaluation program and is designed to support development of systems that are capable of automatically populating a knowledge base with information about entities mined from unstructured text. An important component of the KBP evaluation is the Entity Linking task, where systems must accurately associate text mentions of unknown Person (PER), Organization (ORG), and Geopolitical (GPE) names to entries in a knowledge base. This paper describes the 2009 resource creation efforts, with particular focus on the selection and development of named entity mentions for the Entity Linking task evaluation.