Indian Language Part-of-Speech Tagset: Hindi, Linguistic Data
Consortium (LDC) catalog number LDC2010T24 and isbn 1-58563-571-5, is
a corpus developed by Microsoft Research (MSR) India to support
the task of Part-of-Speech Tagging (POS) and other data-driven
linguistic research on Indian Languages in general. It is created as
a part of the
Language Part-of-Speech Tagset (IL-POST) project, a collaborative
effort among linguists and computer scientists from MSR India, AU-KBC
(Anna University, Chennai), Delhi University, IIT Bombay, Jawaharlal
Nehru University (Delhi) and Tamil University (Tamilnadu).
The goal of the IL-POST project is to provide a common tagset framework for
Indian Languages that offers flexibility, cross-linguistic compatibility and
reusability across those languages. It supports a three-level hierarchy of Categories,
Types and Attributes. The corpus mainly consists therefore of two different
levels of information for each lexical token: (a) lexical Category and Types,
and (b) set morphological attributes and their associated values in the context.
Hindi is the official language of India and a member of the Indo-Aryan language
group. It is spoken mainly in the northern states of Rajasthan, Delhi, Haryana,
Uttarakhand, Uttar Pradesh, Madhya Pradesh, Chhattisgarh, Himachal Pradesh,
Jharkhand and Bihar as well as in much of central India and in communities in
Africa, Australia, New Zealand, the Middle East, Europe and North America. Hindi
is the first or second language of more than 500 million people.
This corpus contains 4859 sentences (98,450
words) of manually annotated Hindi text randomly collected from the Microsoft
Hindi Research Corpus, sourced from the publisher WebDunia.
All annotated data is provided in both xml
and text files. The xml files are contained in the "XML_files" folder
and the text files in the "text_files" folder. Each data file contains
between 900-5,000 words. The XML file contains metadata about the material, such
as language, encoding and data size.
The Annotation Guidelines for Hindi, included in this release, contain a detailed
description of the annotation methodology. The Annotation Tool Guideline 1.0,
also included in this publication, describes the annotation interface developed
for the IL-POST framework; the tool is not included in this corpus.
Additional information, updates, bug fixes may be available in the
LDC catalog entry for this corpus at
Portions © 2010 Microsoft Research Labs India, Pvt. Ltd., ©
2010 Trustees of the University of Pennsylvania