BOLT Information Retreival
The BOLT Information Retrieval (BOLT IR) task required computer systems to process a set of natural language queries written in English (“What do people think about social media?”, “What caused the Euro crisis?”), locate and extract short answers from a large collection of multilingual discussion forum threads, and translate foreign language answers into English.
To support the BOLT IR evaluation, LDC developed these natural language queries and generated a set of common sense rules for assessing responses to each query. LDC then manually reviewed system responses to each query assessing the relevance and utility of the information returned.
These manual relevance judgments were then used to create quantitative and qualitative feedback for system developers to further improve their system’s retrieval technology in subsequent IR evaluations.
Annotation Guidelines
Redundancy Assessment Guidelines