Chinese Web 5-gram Version 1, Linguistic Data Consortium (LDC) catalog number
LDC2010T06 and isbn 1-58563-539-1, was created by researchers at Google Inc. It consists of
Chinese word n-grams and their observed frequency counts generated from over
800 million tokens of text. The length of the n-grams ranges from unigrams (single
words) to 5-grams. This data should be useful for statistical language modeling
(e.g., segmentation, machine translation) as well as for other uses.
Included with this publication is a simple segmenter written in Perl using the
same algorithm used to generate the data.
N-gram counts were generated from approximately 883 billion word tokens of
text from publicly accessible web pages. This data set contains only n-grams
that appeared at least 40 times in the processed sentences. Less frequent n-grams
were discarded. While the aim was to identify and collect only Chinese language
pages, some text from other languages is incidentally included in the final
Data collection took place in March 2008; no text that was created on or after
April 1, 2008 was used to develop this corpus.
The input character encoding of documents was automatically detected, and all
text was converted to UTF-8. The data was tokenized by an automatic tool, and
all continuous Chinese character sequences were processed by the segmenter.
The following types of tokens are considered valid:
- A Chinese word containing only Chinese characters.
- Numbers, e.g., 198, 2,200, 2.3, etc.
- Single Latin tokens, such as Google, &ab, etc.
Extent of Data
- File sizes: approx. 30 GB compressed (gzip'ed) text files
- Number of tokens: 882,996,532,572
- Number of sentences: 102,048,435,515
- Number of unigrams: 1,616,150
- Number of bigrams: 281,107,315
- Number of trigrams: 1,024,642,142
- Number of fourgrams: 1,348,990,533
- Number of fivegrams: 1,256,043,325
Sample screen shot
Portions © 2008 Google Inc., © 2010 Trustees of the University of Pennsylvania