Introduction

This file contains documentation for 2005 NIST Language Recognition Evaluation, Linguistic Data Consortium (LDC) catalog number LDC2008S05 and isbn 1-58563-477-8.

The goal of the NIST (National Institute of Standards and Technology) Language Recognition Evaluation (LRE) is to establish the baseline of current performance capability for language recognition of conversational telephone speech and to lay the groundwork for further research efforts in the field. NIST conducted two previous evaluations in 1996 and 2003. For the 2005 LRE, the emphasis was on research directed toward a general base of technology to be ported to various language recognition tasks with minimum effort and the development of the ability to make more difficult discriminations between similar languages and dialects of the same language. That focus augmented the traditional evaluation goals, those being:

1. to drive the technology forward
2. to measure the state-of-the-art
3. to find the most promising algorithmic approaches

The task evaluated was the detection of a given target language or dialect. From a test segment of speech and a target language or dialect, the system to be evaluated determined whether the speech was from the target language or dialect. The evaluation consisted of speech from the following languages and dialects:

• English (American)
• English (Indian)
• Hindi
• Japanese
• Korean
• Mandarin (Mainland)
• Mandarin (Taiwan)
• Spanish (Mexican)
• Tamil

The 2005 NIST Language Recognition Evaluation Plan, which includes a description of the evaluation tasks, is included with this release. Further information regarding this evaluation is also available at the NIST Language Recognition Evaluation website.

Data

Each speech file is one side of a "4-wire" telephone conversation represented as 8-bit 8 kHz mulaw data. There are 11,106 speech files in sphere (.sph) format for a total of 73.2 hours of speech. The speech data was compiled from LDC's CALLFRIEND corpora and from data collected by Oregon Health and Science University, Beaverton, Oregon.

Each test segment was prepared using an automatic speech activity detection algorithm to identify areas and durations of speech. The test segments were stored in SPHERE file format, one segment per file. Unlike previous evaluations, areas of silence were not removed from the segments. Segments were chosen to contain a specified approximate duration of actual speech. Auxiliary information was included in the SPHERE headers to document the source file, start time, and duration of all excerpts that were used to construct the segment.

The test segments contain three nominal durations of speech: 3 seconds, 10 seconds, and 30 seconds. Actual speech durations vary, but were constrained to be within the ranges of 2-4 seconds, 7-13 seconds, and 25-35 seconds, respectively. Note that this refers to duration of actual speech contained in segments as determined by the speech activity detection algorithm; signal durations in general are longer due to areas of silence in the segments. Shorter speech duration test segments are subsets of longer speech duration test segments; i.e., each 10-second test segment is a subset of a corresponding 30-second test segment, and each 3-second test segment is a subset of a corresponding 10-second segment. Performance was evaluated separately for test segments of each duration.

NIST recommends using data from the 1996 and 2003 evaluations as development data. This data may be found in 2003 NIST Language Recognition Evaluation, LDC2006S31. Because the 1996 and 2003 evaluations did not cover Indian-accented English, this release includes a development data set of Indian-accented English.

Samples

• 3 second
• 10 second
• 30 second

Content Copyright

Portions © 2005 Oregon Health and Science University, © 1996, 2008 Trustees of the University of Pennsylvania