A Linguistic Archive on the Web

jacobson@idf.ext.jussieu.fr
boydm@vjf.cnrs.fr
 

Abstract

The fundamental technical options adopted in designing the archive are the use of standard XML and XML tools, Unicode, and standard browsers. All archived text materials are coded in XML, conformant to a DTD. Views on the documents when they are browsed or queried are defined by XSLT stylesheets, which select and format material from the text database and return it to clients running standard web browsers.

All software developed for the archive is open-source. We present three aspects of the archive here: (1) the document data structure, defined by a DTD, (2) the use of XSLT as a query language, and (3) the client-server architecture, including the software which permits the client browser to access the sound resource.

1. Introduction

The archived materials -- text and sound -- are accessed using standard web browsers. Browsing gives access to a choice of views on the data (transcription with or without translations or interlinear glosses, translations in either French or English), either sequentially or in response to other queries (all utterances containing a selected morpheme, etc.). In all cases, simultaneous access is provided to the recorded sound.

The LACITO Archive at present comprises a hundred or so texts in a score of languages, mainly languages of New Caledonia and of Nepal, and is accessible on the LACITO intranet. A dozen texts in five languages are available for public browsing on the archive website (http://lacito.archive.vjf.cnrs.fr). These texts and and the archive software can be downloaded for local use.

In the present paper we present the methods used for structuring and querying the archive. The documents are marked-up in XML (eXtensible Markup Language). We begin with the DTD (Document Type Definition), which completely defines their structure. We then present the XSLT (eXtensible Stylesheet Language Transformations) stylesheets which define the user interface, select and organize the data to be furnished in response to queries, and define its format. Finally we present the architecture of the system which makes the documents accessible to web clients using a standard browser.

2. Document structure: the DTD

The levels of segmentation recognized by the DTD are the <TEXT>, the utterance <S>, the word <W> and the morpheme <M>. The number and kind of levels of segmentation, and the appropriateness of assigning a particular type of annotation to a particular level, may depend on the purpose and the presuppositions of the linguist as well as on the nature of the language and the genre of the document; we would not necessarily expect to impose a standard here. Our current annotation is designed to facilitate the study of the documents as connected text, and as a resource for lexicography. The same sound resources and perhaps basic transcriptions could be exploited for phonetic research, but this would require a different annotation.

2.1 Metadata

2.2 Time-alignment

SoundIndex2, a tool which associates a text editor with a sound editor, has been developed in Tcl/Tk to facilitate the process of time-alignment on a variety of platforms. It takes as input any well-formed XML document regardless of DTD and associates time-anchoring elements user-specified text elements (e.g. <S>, <W>, <FOO>). Document import and export is handled by an XML parser to guarantee well-formedness. (Sample screen.)

2.3 Transcription

Many of our early documents were simply computerized versions of earlier publications in linguists' traditional 3-line interlinear format. In these "legacy" documents, only W-level transcription is provided, with some separable morphemes implicitly marked up by hyphens. This is adequate for straightforward browsing, but for lexical searches it is clearly preferable to have "go" explicitly marked up as an <M> element than to have to extract it from a <W> element like "going" or even "go-ing".

In our Nepal texts, loanwords from the national language are marked up as <FOREIGN> elements. A similar mechanism could be used to indicate personal or place-names, etc. This markup has the advantage of being usable when the marked-up item does not constitute a whole text segment -- for example when the loan is only part of a <W> element. In an M-level transcription, such items could be marked up using the type attribute of the <M> element.

2.4 Translations

In general, lower-level glosses (also marked up as <TRANSL> elements) are provided (if at all) at only one of the W- or M- levels. In our texts, W-level glosses are often in relatively plain language (<W><FORM>hiptu</FORM><TRANSL>he.hit.him</TRANSL></W>) whereas M-level glosses tend to be more technical:
<M><FORM type="pastem">hipt</FORM><TRANSL>hit</TRANSL></M>
<M><FORM type="vsuffix">u</FORM><TRANSL type="meta">3obj</TRANSL></M>
Translations have the attribute lang for the target language, and may have the further attribute type="meta" to indicate metalanguage glosses, traditionally formatted in small caps.

2.5 Other issues

        <S id="s1" who="A"><AUDIO start="0.00" end="2.00"/>
                <FORM>I haven't finished.</FORM></S>
        <S id="s2" who="B"><AUDIO start="1.00" end="3.00"/>
                <FORM>I've already started.</FORM></S>

3. Using XSLT as a query language

The LACITO DTD defines a range of text annotations including what is sometimes called "interlinear text" -- which we regard as simply one possible view on multilingual glossed text. This and other views on XML documents structured according to the LACITO DTD are defined in XSLT stylesheets, which are available on our website. Here we will present XSLT stylesheets which define more general queries, extracting data and processing it in a flexible way.

An XML document is a text database with a tree structure. Operations like selecting, sorting and counting data -- defined for relational databases in languages like SQL -- may be defined for an XML document in XSLT. An XSLT stylesheet defines a transformation of an XML tree into a result tree, which may also contain formatting information such as (in the present case) HTML markup.

3.1 Data selection

1. Example 1: To list all the occurring morphemes in a document, we define an <xsl:template> element which searches the XML tree starting from the root until it reaches the <M> nodes. This is expressed by the XPath expression .//W/M/FORM , supplied as the value of the select attribute of the <xsl:template> element. The <xsl:copy-of> element then copies each selected node to the result tree.

3.2 Sorting

2. Example 2: Once the required data is selected, further processing can be envisaged. To sort the data, an <xsl:sort> element can be added to the stylesheet, producing a new, sorted result tree. XSLT provides the usual sorting options for numerical data and for character data in a variety of languages.

3. Example 3: To sort in the order defined in a parameter file, we have defined the Java function mysort.mysortvalue(String s, String level). To call this function via the <xsl:sort> element, we need only replace the value . of the select attribute by the value number(java:mysort.mysortvalue(string(.), string('1'))). The <xsl:sort> element can be repeated for as many keys as are required, producing a new result tree. (The sort in the example classes aspirated initials after unaspirated ones and ignores vowel prosodies except to distinguish homographs.)

4. Example 4: To suppress duplicates, we have defined the Java function mydistinct.distinct(NodeIterator ni). All of the values of .//W/M/FORM are placed in the variable forms. In the element <xsl:for-each>, we replace the value .//W/M/FORM with the value java:mydistinct.distinct($forms), producing a new result tree.

3.3 Formatting HTML output

5. Example 5: To prepare our wordlist for presentation to the client web browser, we add HTML tags (<html>, <head>, <body>, <ul>, <li>, etc.) to our output. The new result tree can be interpreted by a web browser.

3.4 Browsing and further querying

6. Example 6: We would like the user to be able to follow up by selecting an item from the list of distinct morphemes and passing it as a parameter to a stylesheet which defines a further processing operation, in this case a morpheme search. To this end, we place each list item in an anchor. When an item is selected by the user, its value is passed to the find.xsl stylesheet, and the XSLT processor is called to execute the search. The anchored list items have the following form: <a href="/servlets/xslservlet?XML=data.xml&XSL=find.xsl&WHAT=xyz">xyz</a>.

4. Architecture for web dissemination

• unlimited distribution without a physical support.
• rendering including Unicode support.
• multimedia.
• interactive user interface.

4.1 Server-side

• The archived XML documents are stored on the server.
• The sound resources are stored on the server, in any one of the usual formats (AIFF, WAV, AVI, etc.). We currently use a compressed format, MPEG2 Layer 3 (*.MP3), to minimize transfer time.
• XSLT stylesheets (which are also XML documents) are stored on the server.
• The only specific processing program installed on the server is a Java servlet which calls the Apache project's Xalan XSLT processor. The XSLT processor applies the XSLT stylesheets to the XML documents, outputting HTML which can be interpreted directly by a web browser.

4.2 Client-side

• The client-side browser interprets and displays the HTML produced by the servlet.
• A Java applet declared in the HTML handles the sound resource, downloading it from the server and executing it in the browser's Java Virtual Machine environment.
• A JavaScript script, also declared in the HTML, handles communication between the applet and the browser. Progress reports from the applet as it executes the sound ("beginning of segment", "end of segment", "end of play") are received by the script and trigger display events (highlighting of appropriate text segments).
• When the user selects anchored text elements in the display, the browser activates links in the anchor which transmit requests for further treatments to the servlet or to the applet.

Conclusion

All archive documents are structured in XML. We have presented the LACITO DTD, which covers a range of text annotations including time-alignment, transcriptions and translations at different levels of segmentation (granularities). This DTD has a few idiosyncrasies related to particular linguists' analyses of particular language data, but its general structure is straightforward. Adherence to the principle of logically structured text makes it relatively easy to transform the markup if desired. Our only authoring tool, SoundIndex, is DTD-independent.

After experimenting with CGI scripts, non-standard XML query languages, etc., we are currently handling all data processing using an XSLT processor and stylesheets. Here again, standardization has led to the availability of excellent XSLT processors which can be extended to handle additional functions as needed. In the present paper, we have concentrated on stylesheets which define data extraction and ordering operations in XSLT -- that is, on the use of XSLT as a query language. This is the area in which we have made the most progress recently. We have presented for the workshop repository an example of a morpheme index, progressively incorporating minor extensions to the XSLT standard. In the future, we expect to pursue these techniques in linking text documents to lexicons. Two lexicons of Nepal languages have recently been converted to XML in prototype form.

Finally, in the area of web-dissemination we have adopted the web browser as our unique access tool and added the functions required for exploiting sound resources, an area not yet covered by web standards. We have presented the software elements which constitute the web architecture of the LACITO Archive above.

Reference:
Jacobson, Michel, Boyd Michailovsky, John B. Lowe. 2001. Linguistic documents synchronizing sound and text. Speech annotation and corpus tools. Special issue: Speech Communication. [Preprint: http://lacito.archivage.vjf.cnrs.fr/article.pdf]