Indian Language Part-of-Speech Tagset: Bengali, Linguistic Data
Consortium (LDC) catalog number LDC2010T16 and isbn 1-58563-561-8, is a
corpus developed by Microsoft Research (MSR) India to support the task
of Part-of-Speech Tagging (POS) and other data-driven linguistic
research on Indian Languages in general. It is created as a part of the
Language Part-of-Speech Tagset (IL-POST) project, a collaborative
effort among linguists and computer scientists from MSR India, AU-KBC
(Anna Universtiy, Chennai), Delhi University, IIT Bombay, Jawaharlal
Nehru University (Delhi) and Tamil University (Tamilnadu).
The goal of the IL-POST project is to provide a common tagset framework
for Indian Languages that offers flexibility, cross-linguistic
compatibility and resuability across those languages. It supports a
three-level hierarchy of Categories, Types and Attributes. The corpus
mainly consists therefore of two different levels of information for
each lexical token: (a) lexical Category and Types, and (b) set
morphological attributes and their associated values in the context.
Bengali (also referred to as Bangla) is a member of the
Eastern Indo-Aryan language group. It
is native to the region of Bengal which consists of Bangladesh, the
Indian state of West Bengal, and parts of the Indian states of Tripura
and Assam. It is spoken by more than 210 million people as a first or a
second language with around 100 million speakers in Bangladesh, about
85 million speakers in India, and others in immigrant communities in
the United Kingdom, USA and the Middle East.
This corpus contains 7168
sentences (102933 words) of manually annotated text from modern
standard Bengali sources including blogs, Wikipedia,
a portion of the EMILLE/CIIL
The annotated data is
structured into two folders, Bangla1 (3684 sentences, 51091 words) and
Bangla2 (3484 sentences, 51842 words), which represent the two stages
in which the data was annotated. All annotated data is provided in both
xml and text files. Each data file contains between 3,000-5,000 words.
The XML file contains metadata about the material, such as language,
encoding and data size.
The Annotation Guidelines for Bangla included in this release contain a detailed description
of the annotation methodology.
Additional information, updates, bug fixes may be available in the
LDC catalog entry for this corpus at
Please examine this XML excerpt for an example of the data in this corpus.
Portions © 2010 Microsoft Research Labs India, Pvt. Ltd., © 2010
Trustees of the University of Pennsylvania