September 2012 Newsletter
Monday, September 17, 2012
New Corpora
GALE Chinese-English Word Alignment and Tagging Training Part 1 -- Newswire and Web
Announcements
The Future of Language Resources: LDC 20th Anniversary Workshop Summary
Thanks to the members, friends and staff who made our 20th Anniversary Workshop (September 6-7) a fruitful and fun experience. The speakers -- from academia, industry and government – engaged participants and provoked discussion with their talks about the ways in which language resources contribute to research in language-related fields and other disciplines and with their insights into the future. The result was much food for thought as we enter our third decade.
Visit the workshop page for the proceedings and to learn more about the event.
English Treebanking at LDC
As part of our 20th anniversary celebration, the coming newsletters will include features that provide an overview of the broad range of LDC’s activities. This month, we'll examine English treebanking efforts at LDC. The English treebanking team is lead by Ann Bies, Senior Research Coordinator. The association of treebanks with LDC began with the publication of the original Penn English Treebank (Treebank-2) in 1995. Since that time the need for new varieties of English treebank data has continued to grow, and LDC has expanded its expertise to address new research challenges. This includes the development of treebanked data for additional domains including conversational speech and web text as well as the creation of parallel treebank data.
Speech data presents unique challenges not inherent in edited text such as speech disfluency and hesitations. Penn Treebank contains conversational speech data from the Switchboardtelephone collection which has been tagged, dysfluency-annotated, and parsed. LDC’s more recent publication, English CTS Treebank with Structural Metadata, builds on that annotation and includes new data. The development of that corpus was motivated by the need to have both structural metadata and syntactic structure annotated in order to support work on speech parsing and structural event detection. The annotation involved a two-pass approach to annotating metadata, speech effects and syntactic structure in transcribed conversational speech: separately annotating for structural metadata, or structural events, and for syntactic structure. The two annotations were then combined into a single aligned representation.
Also recently, LDC has undertaken complex syntactic annotation of data collected over the web. Since most parsers are trained using newswire, they achieve better accuracy on similar heavily edited texts. LDC, through a gift from Google Inc., developed English Web Treebank to improve parsing, translation and information extraction on unedited domains, such as blogs, newsgroups, and consumer reviews. LDC’s annotation guidelines were adapted to handle unique features of web text such as inconsistent punctuation and capitalization as well as the increased use of slang, technical jargon and ungrammatical sentences.
LDC and its research partners are also involved in the creation of parallel treebanks used for word alignment tasks. Parallel treebanks are annotated morphological and syntactic structures that are aligned at sentence as well as sub-sentence levels. These resources are used for improving machine translation quality. To create such treebanks, English files (translated from the source Arabic or Chinese) are first automatically part-of-speech tagged and parsed and then hand-corrected at each stage. The quality control process consists of a series of specific searches for over 100 types of potential inconsistency and parser or annotation error. Parallel treebank data in the LDC catalog includes the English Translation Treebank: An Nahar Newswire whose files are parallel with those in Arabic Treebank: Part 3 v 3.2
English treebanking at LDC is ongoing; new titles are in progress and will be added to our catalog.