Obtaining DataUsing DataProviding DataCreating Data
About LDCMembersCatalogProjectsPapersLDC OnlineSearchContact UsUPennHome

LDC Catalog | By Type and Source | By Year | Top Ten | Projects | Catalog Search


English Gigaword Fifth Edition

Item Name: English Gigaword Fifth Edition
Authors: Robert Parker, David Graff, Junbo Kong, Ke Chen and Kazuaki Maeda
LDC Catalog No.: LDC2011T07
ISBN: 1-58563-581-2
Release Date: Jun 17, 2011
Data Type: text
Data Source(s): newswire
Project(s): GALE
Application(s): information retrieval, language modeling
Language(s): English
Language ID(s): eng
Distribution: 3 DVD
Member fee: $0 for 2011 members
Non-member Fee: US $6000.00
Reduced-License Fee: US $3000.00
Extra-Copy Fee: US $600.00
Non-member License: yes
Online documentation: yes
Licensing Instructions: Subscription Members, Standard Members, Non-Members
Citation: Robert Parker, et al.
2011
English Gigaword Fifth Edition
Linguistic Data Consortium, Philadelphia

Introduction

English Gigaword Fifth Edition, Linguistic Data Consortium (LDC) catalog number LDC2011T07 and isbn 1-58563-515-4, is a comprehensive archive of newswire text data that has been acquired over several years by the LDC at the University of Pennsylvania. The fifth edition includes all of the contents in English Gigaword Fourth Edition (LDC2009T13) plus new data covering the 24-month period of January 2009 through December 2010.

The seven distinct international sources of English newswire included in this edition are the following:

  • Agence France-Presse, English Service (afp_eng)
  • Associated Press Worldstream, English Service (apw_eng)
  • Central News Agency of Taiwan, English Service (cna_eng)
  • Los Angeles Times/Washington Post Newswire Service (ltw_eng)
  • Washington Post/Bloomberg Newswire Service (wpb_eng)
  • New York Times Newswire Service (nyt_eng)
  • Xinhua News Agency, English Service (xin_eng)

The seven letter codes in the parentheses above include the three-character source name abbreviations and the three-character langauge code (eng) separated by an underscore (_) character. The three-letter language code conforms to LDCs internal convention based on the ISO 639-3 standard.

Data

The following table sets forth the overall totals for each source. Note that Total-MB refers to the quantity of date when unzipped (approximately 26 gigabytes), Gzip-MB refers to compressed file sizes as stored on the DVD-ROMs and K-wrds refers to the number of whitespace-separated tokens (of all types) after all SGML tags are eliminated:

Source #Files Gzip-MB Totl-MB K-wrds #DOCs
afp_eng 146 1732 4937 738322 2479624
apw_eng 193 2700 7889 1186955 3107777
cna_eng 144 86 261 38491 145317
ltw_eng 127 651 1694 268088 411032
nyt_eng 197 3280 8938 1422670 1962178
wpb_eng 12 42 111 17462 26143
xin_eng 191 834 2518 360714 1744025
TOTAL 1010 9325 26348 4032686 9876086

Sponsorship

This work was supported in part by the Defense Advanced Research Projects Agency, GALE Program Grant No. HR0011-06-1-0003. The content of this publication does not necessarily reflect the position or policy of the Government, and no official endorsement should be inferred.

Samples

For an example of the data in this corpus, please review this sample file.

Content Copyright

Portions © 1994-2010 Agence France Presse, © 1994-2010 The Associated Press, © 1997-2010 Central News Agency (Taiwan), © 1994-1998, 2003-2009 Los Angeles Times-Washington Post News Service, Inc., © 1994-2010 New York Times, © 2010 The Washington Post News Service with Bloomberg News, © 1995-2010 Xinhua News Agency, © 2003, 2005, 2007, 2009, 2011 Trustees of the University of Pennsylvania

About LDC | Members | Catalog | Projects | Papers | LDC Online | Search / Help | Contact Us | UPenn | Home | Obtaining Data | Creating Data | Using Data | Providing Data

Contact: ldc@ldc.upenn.edu

(c) 1992-2010 Linguistic Data Consortium, University of Pennsylvania. All Rights Reserved.