TDT3 Annotation Guide

Annotation Instructions

Your task is to review news stories from a variety of sources, and to label (annotate) each one as relevant or not relevant to a set of pre-defined topics. There are a total of 11 news sources for TDT3 annotation:

	English	Mandarin
Newswire	NYT APW	Zaobao Xinhua
Radio	PRI The World VOA English	VOA Mandarin
Television	CNN Headline News ABC World News Tonight MSNBC News With Brian Williams NBC Nightly News	---

The timespan of the stories is October 1 - December 31, 1998.

In total there will be 60 topics relating to news events, listed in 3 topic lists of 20 topics each. These topics have been selected by the LDC, according to the selection criteria outlined here.

Most stories will match none of the topic definitions; some will match one; a very few will match more than one. Your task is to apply all relevant topics to each story. You will identify the story as NO (this is achieved by simply hitting the submit button in the interface), YES, BRIEF, or REJECT. Use the YES label for stories that discuss the topic in a substantial way. Use the BRIEF label for stories that make only a passing reference to the topic. To choose betweeen YES and BRIEF, use the following criteria:

YES for a topic if at least 10% of the story is devoted to that topic.
BRIEF for a topic if less than 10% of the story is devoted to that topic. There is no such thing as too brief a reference. Any mention is worth noting with BRIEF.

Please REJECT a story if:

it contains more than one story (for newswire files)
it is missing the first part of the story (for newswire files)
it is corrupted (for newswires: lots of gibberish; in a foreign language)
it has font/formatting problems (headline appears in text font or story appears in headline font)
it is obviously NOT NEWS (a commercial; an advertisement; an obvious preview)

Use Comments generously when rejecting files. Also use Comments to specify an "oddity" about an article. For instance, if the segmentation seems odd, but could be correct, label it or not as appropriate and comment on it.

The listing for each topic includes helpful, but incomplete, information about the derivative event. Dates or date ranges are included where they are relevant. A link to each topic's seed story is also available. Use this story as a reference only, an example of the language or vocabulary used in discussing this topic. You may take advantage of the headline information associated with each article, but cannot rely on that to be complete. Topics may be mentioned in the middle of long articles.

There is more specific information related to topic definition below. If you have questions about how to label a specific story, refer to Stephanie Strassel, Kara Rennert, and Nii Martey.

Topics

"Topic" is defined in a special way specifically for TDT research. For the purposes of this project, topics refer to specific events or activities, such as the crash of a China Airlines airplane in Taipei, Taiwan on February 16, 1998, and encompass all facts, events and activities that are directly related to them. Here is the definition of topic and a few other essential terms, as used in TDT research:

TOPIC- A topic is an event or activity, along with all directly related events and activities. A set of 60 topics will be defined for the TDT3 corpus.
EVENT- An event is something that happens at some specific time and place, and the unavoidable consequences. Specific elections, accidents, crimes and natural disasters are examples of events.
ACTIVITY- An activity is a connected set of actions that have a common focus or purpose. Specific campaigns, investigations, and disaster relief efforts are examples of activities.
STORY- A story is a newswire article or a segment of a news broadcast with a coherent news focus. They must contain at least two independent, declarative clauses.

Rules of Interpretation

Topics generally fall into a few general categories. As an aid to topic labeling consistency and to make the process efficient and accurate, please use the following guidelines in expanding from an event (the list provided in your labeling interface) to a topic.
1. Elections:
Examples - New people in office, new public officials, change in governments or parliaments (in other countries), voter scandals.
The event might be the confirmation of a new person into office, the activity around voting in a particular place and time, the opposing parties' or peoples' campaigns, or the election results. The topic would be the entire process, nominations, campaigns, elections, voting, ceremonies of inauguration.

2. Scandals/Hearings:
Examples - Monica Lewinsky, Kenneth Starr's investigations.
The event could be the investigation, independent counsels assigned to a new case, the discovery of a potential scandal, the subpoena of political figures. The topic would include all pieces of the scandal or the hearing including the allegations or the crime, the hearings, the negotiations with lawyers, the trial (if there is one), and even media coverage.

3. Legal /Criminal Cases:
Examples - crimes, arrests, cases.
The event might be the crime, the arrest, the sentencing, the arraignment, the search for a suspect. The topic is the whole package; crime, investigation, searches, victims, witnesses, trial, counsel, sentencing, punishment and other similarly related things.

4. Natural Disasters:
Examples - tornado, snow and ice storms, floods, droughts, mud-slide, volcanic eruptions.
The event would include causal activity (El Nino, in many cases this year) and direct consequences. The topic would also include; the declaration of a Federal Disaster Area, victims and losses, rebuilding, any predictions that were made, evacuation and relief efforts.

5. Accidents:
Examples - plane- car- train crash, bridge collapse, accidental shootings, boats sinking.
The event would be causal activities and unavoidable consequences like death tolls, injuries, loss of property. The topic includes mourners pursuit of legal action, investigations, issues with responsible parties (like drug and alcohol tests for drivers etc.)

6. Ongoing violence or war:
Examples - terrorism in Algeria, crisis in Iraq, the Israeli/Palestinian conflict.
In these cases the event might be a single act of violence, a series of attacks based on a single issue or a retaliatory act. The topic would expand to include all violence related to the same people, place, issue and time frame. These are the hardest to define, since war is often so complex and multi-layered. Consequences or causes often include (and would therefore be topic relevant) preparations for fighting, technology, weapons, negotiations, casualties, politics, underlying issues.

7. Science and Discovery News:
Examples - John Glenn being sent back into space, archaeological discoveries.
The event is the discovery or the decision or the breakthrough The topic, then, would include the technology developed to make this event happen, the researchers/scientists involved in the process, the impact on every day life, all history and research that was involved in the discovery.

8. Finances:
Examples - Asian economy, major corporate mergers.
The topic here could include information about job losses, impacts on businesses in other countries, IMF involvement and sometimes bail out, NYSE reactions (heavy trading BECAUSE Tokyo closed incredibly low). Again, anything that can be defined as a CAUSE of the event or a direct consequence of the event are topic-relevant.

9. New Laws :
Examples - Proposed Amendments, new legislation passed.
While the event may be the vote to pass a proposed amendment, or the proposal for new legislation, the topic includes the proposal, the lobbying or campaigning, the votes (either public voting or House or Senate voting etc.), consequences of the new legislation like protesting or court cases testing it's constitutionality.

10. Sports News :
Examples - Olympics, Super Bowl, Figure Skating Championships, Tournaments.
The event is probably a particular competition or game, and the topic includes the training for the game or competition, announcements of (medal) winners or losers, injuries during the game or competition, stories about athletes or teams involved and their preparations and stories about victory celebrations.

11. MISC. News :
Examples - Dr. Spock's Death, Madeleine Albright's trip to Canada, David Satcher's confirmation.
These events are not easily categorized but might trigger many stories about the event. In these cases, keep in mind that we are defining topic as the seminal event and all directly related events and activities. (include here causes and consequences) If the event is the death of someone, the causes (illness) and the consequences (memorial services) will all be on topic. A diplomatic trip topic would include plans made for the trip, results of the trip (a GREAT relationship with Canada??) would be on topic.

Administrative Tracking

The annotation interface will automatically assign the next available file to you after you have chosen the appropriate language and topic list. You must record the file you're annotating on your timesheet. We need to keep careful records, both programmatic and on paper. While you are working within a file, you will be able to back up and make changes to your labels. Once you log out, the "rejects" you have identified will be labeled as such forever in the interface, so please be especially careful when labeling these.

Last modified: Mon Jul 26 15:17:48 1999