These pages document various aspects of corpus creation at the Linguistic Data Consortium. Readers should not view this as an attempt to prescribe methodology for all corpus creation. It is, instead, a presentation of current practice at LDC based on our experiences.

Note that this page is still under construction. We have provided an outline of the sections we plan to add and will fill it in as we are able. We hope these pages -- even the outline -- will encourage feedback from communities of corpus users.

1. Background (Dave Graff, Shudong Huang, ed.)
   1. Audio, video data formats (Dave Graff)
      1. sampling format (sample size, rate, channels)
      2. fidelity
      3. existing formats (.sph, .au, .wav, .riff, .aiff, raw, .ra(m)), needs, platforms
      4. tools
      5. compression
   2. Text Data Formats (see also annotation format specifications and DTDs)
      1. intro (Steven Bird)
      2. human-readable vs. machine-readable text (Dave Graff)
      3. markup languages (SGML, HTML, XML, specialized) (Steven Bird)
      4. tokenization (Dave Graff)
      5. character encoding (Kevin Walker)
         1. character set vs. font
         2. national standards vs. Unicode vs. proprietary/non-standard
         3. tools
      6. compression (Dave Graff)
      7. tools (Dave Graff)
2. Data Collection
   1. Text (each section includes tools)
      1. WWW (Kevin Walker)
      2. News Feeds (Kevin Walker)
      3. scan (Suzanne D., Christopher Cieri)
      4. keyboard (Christopher Cieri)
      5. closed-captioning (Dave Graff)
      6. QC (Jon Wright)
   2. Audio (each section includes tools)
      1. Wideband (Masato Kobayashi)
      2. Telephone(David Miller)
      3. Broadcast News (Dave Graff)
      4. QC (Dave Graff)
3. Creating Annotated Data Resources
   1. corpus specification and selection
   2. collection
   3. transcription
   4. translation
   5. annotation
   6. quality control
   7. format specifications
   8. DTDs
   9. permissions
4. Building Paradigmatic Data: Lexicons
   1. corpus specification/structure of entries (Masato Kobayashi, Shudong Huang)
   2. QC (Masato Kobayashi, Shudong Huang)
   3. tools (Christopher Cieri, Dave Graff, Steven Bird)
5. Management of corpus building effort (Stephanie Strassel)
   1. Annotation Training
   2. Consistency
   3. Multi-annotator projects
      1. skill set for annotation team
      2. managing team
      3. general QC
6. Data Dissemination (Masato Kobayashi, Christopher Cieri, ed.)
   1. Documentation for corpus
   2. IPR/informed consent (Andy Cole, Shannon Sears)
   3. Directory/file structure, data schema (Jon Wright, Masato Kobayashi)
      1. file/directory naming limitations/best practices
      2. cross platform issues