1
Which or what?
A study of interrogative determiners in present-day English
Bas Aarts, Evelien Keizer, Mariangela Spinillo and Sean Wallis
Survey of English Usage
University College London
The determiner slot in English interrogative Noun Phrases allows for three possible occupants, namely
which, what and whose, and their variants ending in -ever: whichever, whatever and whosever. In this
paper we will be looking only at the first two of these, which and what, in structures like the following:
Which films do you like? / What films do you like? These expressions are close in meaning, so the
question arises what factors influence the choice of one over the other determiner. Our aim in this paper
is twofold: first we will demonstrate how interrogative NPs such as those exemplified above can be
retrieved from the British component of the International Corpus of English (ICE-GB), using Fuzzy
Tree Fragments (FTFs). The second aim of the paper is to show how the search results can be used to
investigate what determines the choice of determiner in NPs like those shown above.

2
The TALANA annotated corpus for French: some experimental results
Abeillé Anne, Clément Lionel, Kinyon Alexandra, Toussenel François
TALANA-UFRL. Université Paris 7
{abeille, clement, kinyon, ftoussen}@linguist.jussieu.fr
This paper presents the first linguistic results exploiting the new annotated corpus for French developed
at Talana-Paris 7 (Abeille & al 00). The corpus comprises 1 million words fully annotated and
disambiguated for parts of speech, inflectional morphology, compounds and lemmas, and partially
annotated with syntactic constituents. It is representative of contemporary normalized written French,
and covers a variety of authors and subjects (economy, literature, politics, etc.), with extracts from
newspapers ranging from 1989 to 93.
After explaining how this corpus was built, we present some linguistic results obtained when searching
the corpus for lexical or syntactic frequencies, for lexical or syntactic preferences, and explain why we
think some of these results are relevant both for theoretical linguistics and psycholinguistics.

3
On the search for metadiscourse units
Annelie Ädel
annelie.adel@eng.gu.se
English Department
Göteborg University, Sweden
1. Introduction
To what extent do writers anchor their discourse in the current discourse situation and make the
presence of the writer and/or reader overt? What types of situations are at hand when writers refer to
their text as text, to themselves or to their readers? To what extent are texts monologic or dialogic?
These types of questions have recently attracted a lot of attention within research on metadiscourse.
Metadiscourse refers to discourse about on-going discourse and is interesting to study from the
perspective of how the writer's or reader's presence in the text are made explicit. It has been studied by
various branches of linguistics, for example text linguistics (Mauranen 1993, Markkanen et al 1993) –
also in a historical perspective (Taavitsainen 2000), pragmatics (Hyland 1998) and genre studies
(Bäcklund 1998, Bondi 1999) and is rapidly becoming a dynamic field of research.
One particularly interesting perspective is the study of cultural differences in the use of
metadiscourse. Several researchers have shown that metadiscourse typically differs across cultures (e g
Markkanen et al 1993, Mauranen 1993, Vassileva 1998, Bäcklund 1998). Cultural differences in
writing have been studied within the field of contrastive rhetoric, on the basis of the primary hypothesis
that there are culture-specific patterns of writing, and that these cause interference in L2 writing (see
Connor 1996:90). In this paper, I will report ongoing research into the use of metadiscourse in written
argumentative texts by native and non-native speakers of English. The study is corpus-based and
comparative, contrasting the use of metadiscourse by Swedish advanced learners’ writing in English to
the writing of native speakers of British and American English.1 All writers are university students. The
argumentative essays are full-length, and are available within the framework of the International
Corpus of Learner English (Granger 1993). In case there may be cultural differences in the use of
metadiscourse, the British English and American English parts of the control corpus are kept separate.
One of the aims of my thesis is to investigate metadiscoursal patterns with explicit
reference to the writer or reader. Some examples of metadiscourse specify discourse acts that the writer
intends to perform are, for instance:
(1a) to introduce a topic or state an aim: In this essay I will discuss some of the
problem...; We must now consider the pros and cons of Britain joining "The Single
Market".
(1b) to sum up a discussion: To make a short summary of what I have been trying to
say in this essay…; I have presented some of the most important benefits of drug
legalization…
(1c) to close the topic: …one may conclude that the American people does not approve
of political leaders with low moral standards; The conclusion one might draw is rather
depressing…
I will first discuss the definition of metadiscourse and how I delimit the area. This is not unproblematic
and metadiscourse has been defined differently by different researchers (see e g Markkanen et al 1993
and Mauranen 1993). The focus of the paper is on a subcategory within the broader field of
metadiscourse which will be referred to as ‘metatext'. Metatext is text about the current text, and
includes references to the evolving text itself rather than its subject matter. Since I am concerned with
personalised examples here, this means linguistic elements through which the writer comments on his
or her own discourse actions (see examples (1a-c) above). The method used in this work will only
capture personalised types of metadiscourse, where the writer and/or reader have been explicitly
mentioned in the texts, including expressions such as as I showed above and I will give an example, but
not as shown above, to exemplify or this essay will show, etc. First person pronouns I and we, which are
1 The native speaker corpus, called LOCNESS also has one British English part consisting of A-level essays,
which has been excluded from the investigation. The Swedish subcorpus will be referred to as SWICLE in the
following. Examples will be marked either (Swicle), (BrE) or (AmE).
4
potential explicit references to the writer and/or the reader, have been retrieved from the corpus
material and will be considered in the present paper.
Further questions that will be posed within the span of the paper are: Who do the
pronouns I and we include? How frequent are metatext units across corpora? What do they look like?
and How are the metatext units distributed in the texts? The comparative perspective involves
investigating the differences and similarities in the use of metadiscourse between learners and native
speakers. It should be stressed that this is work in progress, so no definite analyses will be given.
2. The definition and delimitation of metatext
All uses of I or we are not metatextual. I also occurs when the writer appears in the text to express his
or her feelings, or to talk about personal experiences. I call this category involvement, and do not
include it in the concept of metadiscourse. The way the term is used here is somewhat different from
Chafe's (1982) definition. Chafe refers to involvement with the audience as typical for a speaker, and
‘detachment’ from the audience as typical for a writer, and the concept of involvement includes a range
of linguistic features that are more prevalent in speaking2. Involvement in the sense it is used in this
paper adds a narrative quality, or the writer as a person in the discourse-external world adds some
personal experience to the discourse, or expresses his or her personal feelings or attitudes towards
phenomena in the ‘real world'. Involvement including we stresses shared personal experiences in
discourse-external situations. One would not expect a great deal of involvement in argumentative
essays, but this type is extremely frequent in the learner material.
I make a distinction between two subcategories within metadiscourse, which are
metatext and commentary. Commentary has to do with writer-reader relations. It refers to features used
to address readers directly, and draw them into an implicit dialogue (Vande Kopple 1988). I plan to
carry out a future study, looking at the following commentary features: second person pronoun you
(your), questions, exclamations, and imperative forms, and possibly also lexical items such as dear
reader (vocatives). Speaker attitude is not dealt with here, although it may have to do with the writerreader
relationship and especially with persuasion. In integrative approaches to metadiscourse (e g
Markkanen et al 1993), however, the writer's attitude to what is said is included.
In order to distinguish metatext and commentary from other uses of the pronouns, I
also make a distinction between (a) whether the action takes place within the world of discourse, or (b)
within object language, dealing with the ‘real world'. This can be compared to the distinction between
metalanguage and object language in linguistics, an idea which originates with Roman Jakobson (see
for instance 1980:81-92). Jakobson described metalanguage as a language in which we speak about the
verbal code itself. It was contrasted to object language, in which we deal with objects or items that are
external to language itself. Similarly, in a text perspective, we can speak of discourse-internal versus
discourse-external phenomena. Thus, one basic question I examine when analysing the data is: Is the
ongoing discourse in focus or other, ‘worldly', activities that are external to the text?
The notion of current text (cf Mauranen 1993) is also important to metadiscourse,
meaning that we are interested in how texts refer to themselves, and not to other texts. Texts about
other texts (alluding to or commenting on other texts) are described by the concept of intertextuality,
which is not our concern here. This is also the reason why quote markers, or introductions to reported
speech, which are often included in models of metadiscourse, will be disregarded in the present study.
Reporting the speech of others or quoting from other sources than the current text are intertextual rather
than metatextual activities.
Criteria or parameters for metadiscourse are generally not specified by researchers; it is
usually simply referred to as ‘text about text'. However, I have extracted features, which work well for
the analysis of the type of personalised, explicit types of metadiscourse present in my collection of
corpus data. When looking at an occurrence of I, I decide whether it refers to the writer of the current
text, or to an experiencing subject outside the realm of discourse. When the examples contain we, the
question to ask is whether the pronoun includes [+writer persona] and [+reader persona], and not some
other group that does not directly participate in the ongoing discourse. These parameters are
summarised in figure 1 below. To a great extent, the parameters concern rhetorical roles taken on by
the writer and offered to the reader.
2 These features are for example first and second person reference; the presence of the speaker's mental processes;
monitoring by the speaker of the communication channel; emphatic particles that express enthusiastic involvement
in what is being said, like just and really; vagueness and hedges; and direct quotes (see Chafe 1982).
5
Fig. 1. Parameters for metatext.
Metatext
[+current discourse]
[+world of discourse]
[+writer persona] ( I, we)
[+reader persona] ( we)
In order to analyse an element as metatext, we need to ask whether it refers to the current discourse,
whether it deals with the world of discourse, and, more specifically, concerning first person singular,
whether I refers to the writer presenting him- or herself as a writer. Concerning first person plural we, it
should include the writer persona and/or the reader persona, and no other persons that do not belong in
the world of discourse.
3. Explicit references to the writer and reader: What do the pronouns refer to?
The first and second pronouns are most typically used to refer to specific individuals identified in the
situation of communication (Biber et al 1999:328). They may point to ‘the one who is speaking’ and
‘the one(s) who is/are being addressed'” (Wales 1996:51). These pronouns help us to capture the
current writer and the current reader of the current text. In my search for metadiscourse, I look for what
Quirk et al (1985:347) call ‘specific reference', i e when the first and second person pronouns are used
to refer to the writer(s) and the reader(s), those directly involved in the discourse situation.
The first person singular pronoun I is “unambiguous in referring to the speaker/writer”
(Biber et al 1999:329), in contrast to we, which often involves a “fluidity and ambivalence of meaning”
(Wales 1996:58). In my data, first person singular I points to the writer of the current text in the
majority of cases, except when it occurs in quoted material or reported speech, or for example when the
pronoun it has been misprinted and lost its last letter.
In order to classify an example as metadiscourse, not only has the I to point to the
writer of the current text, but the action that the I performs has to be discourse-internal. The verb
discuss, for example, involves a speech act which is quite frequently performed in argumentative
essays. In the first example, however, the action does not take place within the realm of the current
discourse:
(2) One does not need to read the papers to notice how the antagonism towards
immigrants has increased. Lately I have discussed the increasing hostility towards
immigrants with my friends, relatives and fellow workers. Almost everyone I spoke to
wants tougher immigration rules. (Swicle)
The discussion in example (2) is held between the writer as a person in ‘real life’ and text-external
individuals and it counts as involvement. Another example including the verbs discuss and analyse,
which does concern the writer and the reader is the following, classified as metatext:
(3) [] and secondly that the executive had been strengthened vis-à-vis the parliament. I
will briefly discuss the Prime Ministers role and then elaborate on the Presidents
functions. Then I will analyse each presidency showing how the presidents role has
evolved. (BrE)
Examples (10)-(16) in section 5 give further examples of metatext having I as subject.
3.1 We and metatext
Let us consider some examples of the more complex first person pronoun we. Most occurrences in my
material are not metadiscoursal, but relate to discourse-external phenomena. There is a fairly small
number of examples that refers to the writer (but not as a writer) and other persons (who are not the
readers of the current text), as in the following example:
(4) [], but 2 a.m. struck and he bad [sic] to go. We talked some more in the lobby but
we had to keep our voices down, out of respect. I wanted him to spend the night
because [] (AmE)
Examples of this kind have been left out. What I am primarily interested in is the type that has been
called ‘inclusive authorial we’ (Quirk et al 1985:350). A small part of the overall material is of this
kind (see section 4), for example:
6
(5) students ask themselves the question; is it worth the cost and the great effort it takes
to study at the university? Therefore we are now going to look at some advantages of
higher education, both in a short-term perspective and also over a longer period
(Swicle)
The effect of the fact that both the writer and the reader are addressed in this example is that cooperation
is emphasised. The writer is showing his or her helpfulness and will to guide the reader in the
discourse, thereby bonding with the reader, as it were.
One subgroup of the so-called ‘exclusive we', according to Wales (1996) is the
collective we, indicating several writers. This type does not occur in the selected material, since there
are only single writers of the essays. A second subgroup is ‘editorial we', which is restricted to very
few occurrences in the present material, for example “In the course of this essay, we shall attempt to
analyse whether this is a belief founded in reality and, if it is, why it should cause such fear.” (BrE).
Here, the single author uses the plural form for his own discourse actions. Quirk et al (1985:350)
explain the motivation for using this type as a “desire to avoid I, which may be felt to be somewhat
egotistical”.
Another special usage of we is found in several of the literary essays of the British
English material, where we is equivalent to ‘we the audience', as in the following extract:
(6) Through Caligula's own dialogue, and through the opinions of Cherea, we are
shown that there is a logic to Caligula's approach. Caligula insists on following this
logic through to the completion (BrE)
This reference is made explicit in another instance, through apposition: “We the audience can identify
with her view” (BrE). These types have been omitted. Since the reader is excluded, it could be argued
that these are not cases of metatext. Before leaving this category, I will bring up one more example,
which is intertextual (throughout the play) rather than metatextual, discussing how readers of another
text might interpret that other text:
(7) In the play 'Caligula' Camus is dealing with the themes of death and the absurd, and
throughout the play we can see different characters reacting in different ways. (BrE)
The criterion that metatext instances should refer to the current text is not fulfilled here, but the writer
is referring to another text, which belongs to a different genre even. Also, unless the writer assumes
that the reader of his essay has read (or seen) the play, the we does not include the reader of the current
text. Several examples of ‘we the audience’ type can be found in the British English material (as many
as 75 out of a total of 262, which is nearly 30 per cent), but none at all in the other corpora. In this case,
the use of first person plural we is probably different in the British English material due to the presence
of a large number of literary argumentative essays, illustrating how different topics may influence
linguistic features.
A number of examples are ambiguous, for example: “[] However this system is once
again proved wrong as Cacambo, Candide's manservant remains faithful disproving Martin's theory
that Cacambo would run off with the money collected from Elderado. We can therefore assume that
Voltaire is attacking Optimism in its context of a system, just as he criticises other systems in the book,
such as the Church system, the military system and the caste system.” (BrE). Here, does we refer to the
readers of Candide or to the writer and reader of the current text? Probably the latter, since it is possible
to insert [from what I have said so far] in connection to the instance.
As in the we the audience example, it is fairly often the case in the material that the
pronoun is followed by a specification, which makes it easier to identify what it is intended to refer to.
The ‘generic we', which is by far the most frequent type in the SWICLE and American English
material, is sometimes modified by prepositional phrases and apposition: in Sweden/in the
US/all/people/women. In some cases, a subsection of the whole is referred to by the addition of asphrases,
e g as individuals/as borrowers. This type has also been called ‘rhetorical we’ (used in a
collective sense: e g nation, party, see Quirk et al (ibid.)), and is particularly frequent in the SWICLE
essays, presupposing solidarity with the reader.
In example (8) below, the persons the writer is talking about are ‘people in general’
(also specified by most of us). This includes the writer him- or herself, but not in the role of a writer but
rather as an experiencing person in the world. Moreover, other people than the reader are included.
7
(8) Most of us are rather selfish and if we are frank we have to confess that we can not
compare ourselves with Mother Theresa when unselfishness is considered. (Swicle)
It can be argued that this example is not metadiscoursal. It is still fairly persuasive, and it seems as if
the writer very much would like to include the reader in the group of selfish mortals, but the most of
still leaves some freedom to the reader to decide for him- or herself whether the description fits or not.
This type of ‘freedom’ given to the reader, however, is rare. As Clark and Ivanic
(1997:165) have pointed out, “[i]n building the dialogue with readers, writers in all genres often take
for granted that readers are going to share their beliefs and values [], for example, [] by using the
pronoun ‘we'. In this way they position their readers as consenting, part of an ‘usness’ that is hard to
resist []”. This is particularly true in the case of rhetorical we. The ‘usness’ is particularly hard to resist
in a phrase such as as we all know, which includes everyone, also the reader. These examples have
been classified as commentary.
Interestingly, one example in the material seems to contain some awareness of the fact
that we can be quite powerful in its inclusiveness:
(9) So, in the immediate future eastern Europe is an enormous market for consumers.
At the same time "we" receive alarm reports from all over the world we have millions
of people longing for things we have taken for granted. (Swicle)
The pronoun is put within quotation marks, which seems to have the effect of hedging the statement
made by the writer. Thus, the reader is also made aware that the all-inclusiveness of we can be
questioned.
4. Frequencies of pronouns and metatext units
There are huge differences in the overall frequencies between the three groups, in particular concerning
I. What is most striking is that the Swedish learner material has overwhelmingly higher frequencies of
first person pronouns.
Table 1. The overall frequency of I across corpora
Corpus Approx. number of words
in corpus
Raw frequency Frequency per 10,000
words
SWICLE 204,630 1,851 90
LOCNESS BrE 95,508 83 9
LOCNESS AmE 149,767 649 43
Table 2. The overall frequency of we across corpora
Corpus Approx. number of words
in corpus
Raw frequency Frequency per 10,000
words
SWICLE 204,630 1,893 93
LOCNESS BrE 95,508 262 27
LOCNESS AmE 149,767 426 28
We is the most preferred first person pronoun, except in the American English corpus, which uses the
more individualistic I to a greater extent.3 The preference in the British English data for we over I (27
versus 9) is not in accordance with the results of a study by Vassileva (1998:167) on academic writing,
who found that “the ‘I’ perspective clearly dominates in English” in comparison to the ‘we’
perspective. The main part of her English corpus consists of research articles written by speakers of the
British English variety. The reason why academic writing and the British English argumentative essays
differ remains to be solved.
The overuse of I by Swedish learners in contrast to native speakers has been pointed
out by Altenberg (1997) and by Ringbom (1996, as reported in Altenberg 1997:127). Previous research
on learner writing has also found that learner writers within the ICLE project generally are much more
overtly present within the discourse than native speaker writers (Altenberg 1997 and Petch-Tyson
1998), suggesting that this may be a general learner strategy.
3 However, first person plural forms are most frequent in the American English data in the accusative and
possessive forms.
8
The table below reveals the great difference in the number of metatext units per
100,000 words including I across corpora, particularly in the SWICLE in relation to the native speaker
corpora.
Table 3. The frequency of I involved in metatext units compared across corpora
Corpus Approx. number of words
in corpus
Raw frequency Frequency per
100,000 words
SWICLE 204,630 275 134
LOCNESS BrE 95,508 20 21
LOCNESS AmE 149,767 55 37
These results raise several questions, for instance whether the proportions are similar across corpora
also for impersonal types of metadiscourse, or if the great preponderance in the learner material is due
to the general tendency among Swedish learners (as evidenced in this corpus) to use writer (and reader)
visibility in their writing. In other words, do the data reflect the general picture regarding proportions in
the use of metadiscourse, or are they dependent on the overuse of I in the SWICLE, noted above?
Table 4. The frequency of we involved in metatext units compared across corpora
Corpus Approx. number of words
in corpus
Raw frequency Frequency per 100,000
words
SWICLE 204,630 57 28
LOCNESS BrE 95,508 26 27
LOCNESS AmE 149,767 25 17
In the British English material, there are more metatext units involving we than I, whereas the Swedish
learner essays (in particular) and the American English material have the greatest figures in connection
to I.
The Longman grammar (Biber et al 1999:334) comments on the ‘unexpectedly high
frequency’ of we/us in their categories of news and academic prose as being “connected with the
multiple uses of this pronoun in written prose: to make generalisations, and to refer to the author and
reader”. When comparing the overall figures for we and the frequencies of we involved in metatext
elements, we can see that the latter use is far less frequent in the present argumentative essays.
There is a great difference in the learner material, where the metatext units involving
we only amount to about a fifth of the units including I (57 versus 274). The overall occurrences of the
pronouns I and we in the SWICLE, however, are roughly the same, even with a slight predominance of
we. One conclusion to draw from this, is that the Swedish learners, in their use of personalised
metatext, very much prefer the pronoun I to we. The native speakers, on the other hand, use the
pronouns more or less to the same extent. The British English material has slightly more occurrences of
we (26 instead of 20 in relation to I), whereas the American English material about twice as many
metatext units including I (55, versus 25 for we occurrences).
If we take normalised figures into account, it is still clear that the Swedish learners
overuse personalised metatext, particularly with first person singular I as subject, provided that we
regard the native speaker data as the norm.
The results also show that there is a difference between the American and British
English components of the LOCNESS corpus. The data suggest that writers in the American English
variety use more metadiscourse (at least metatext) in their argumentative writing compared to writers
who use the British English variety. It may be the case that writing conventions concerning
metadiscourse differ in the two cultures, in which case Swedish learners would be more inclined
towards (or influenced by?) the American English style, and a lot less so towards the British English.
5. What do the metatext units look like?
In this section, I will briefly bring up one characteristic of the learner essays that seems to me to be
important. In many respects, the learner essays are more tentative, hedged, and polite in their use of
metatext compared to the native speaker examples.4 For example, try and would like to are very
4 This may also partly depend on cultural differences in writing. Vassileva (1998:168), studying English, German,
French, Russian and Bulgarian academic writing, notes that “[i]n German, the author's intentions are, more often
than not, mitigated (downtoned, hedged) by means of modal verbs, predominantly ich möchte ‘I would like’ []”.
Writing in Swedish may also follow this ‘mitigating’ convention, which may be transferred by the learners to their
9
common in the learners’ metatext units. Try is the most frequent verb after say, but it does not occur at
all in the native speaker material. Examples including I are the following:
(10) In this essay I will try to give some examples of what imagination and dreaming
have come to today with some references to the past, and I will also try to answer the
question as to wether there is a place for imagination and dreaming in our modern
world dominated by technology and science, or not. (Swicle)
(11) Should we accept all foreign cultures or discriminate some of them? In what way
would our defence of other forms of Swedish culture and our attitude to foreigners and
foreign things? I will try to answer these questions, as well as I can, but first I will try
to describe the situation of the Swedish language today. (Swicle)
(12) To make a short summary of what I have been trying to say in this essay,
technology will never make imagination and dreams unnecessary for two main
reasons. (Swicle)
The first two examples announce what is going to follow in the discourse5: in (10) the writer will
perform the discourse act of giving examples and answer an important topical question, and in (11)
answer topical questions and describe a situation to the reader. These examples are placed at the
beginnings of the essays they occur in.
Instead of giving anaphoric reference, example (12) points backwards in the discourse6
and the writer announces that a summary of the main points that have been made in the previous
discourse is to come. The what I have been trying to say gives a modest and even insecure impression,
as if the writer doubts his or her ability to get his or her ideas across in writing.
Also note the occurrence of the phrase in this essay in both (10) and (12), which
reflects the fact that essays in the learner material generally have a high degree of reflexivity, or
awareness of the discourse as discourse. This is expressed particularly often in the first paragraphs, as
in: “This paper is supposed to deal with immigrants coming to Sweden. To be more précis I am going
to write about refugees, the biggest group of people who come here today.” (Swicle). This helps
explain why the verb choose is fairly frequent in the metatext units in the learner material; the writers
explicitly comment on the choice of topic for the essay.
The polite formula I would like to also has high frequencies in the learner texts.
Example (13) below introduces the topic and tells the reader what the writer is going to focus on in the
essay. Note the extreme situation awareness in the adverbs in my essay today.7
(13) One of Anita Brookner's books is called 'A friend from England'. I read it last
autumn, so I might have forgotten some names. If so, I've got to re-name them. In my
essay today I would like to concentrate on the relationship between Heather and
Rachel. I will also discuss in what way this was an important personal relationship.
(Swicle)
The insecure and hedged quality of many learner texts is also noticeable here, in that the writer
explicitly comments on the fact that he or she may have forgotten some names in the novel he or she is
about to discuss. However, the solution to this problem, to make up new names, is also brought up.
Another example involving I would like to is the following:
(14) To become a dog owner is one of the greatest gifts in life. I will support this
statement by discussing three positive effects. I would like to call the first one
writing in English. In this contrastive study on authorial presence by means of first person pronouns, Vassileva
found different cultural norms.
5 This type has been called Announcement (Crismore et al 1993), preview (Crismore and Farnsworth 1990) and
Advance Labelling (Tadros 1993).
6 Types in which the writer tells the reader what he or she has already done in the discourse have been called
Reminders (Crismore et al 1993), reviews (Crismore and Farnsworth 1990) and Recapitulation (Tadros 1993).
7 Petch-Tyson (1998:112) examines the subcategory ‘reference to situation of writing/reading’ within the broader
concept of writer/reader visibility in texts, including here, now and this essay. In a comparison of samples from the
Dutch, Finnish, French and Swedish learner subcorpora within the ICLE project, both here and this essay were
used about twice as often by the Swedish learners.
10
friendship. It is said that dogs are our best friends and I think that there lies a great deal
of truth behind this statement. (Swicle)
In this example, the writer labels the first of three effects which are important supporting arguments for
the writer's thesis that a dog is one of the greatest gifts in life. Naturally, the writer could have skipped
the metatext element marked by italics above, and been content by just having ‘The first one is
friendship'. The writer could also have chosen to express more certainty by using modal will instead of
would like to.
(15) is another example of I would like to, announcing to the reader that some
examples supporting the writer's argument are to come:
(15) In this essay, which is of course wholly unscholarly, I would like to point out
some examples, mostly by female writers where woman's place in society - or rather
woman's possibility of controlling her won social position - is illustrated by the actions
of the "archetypal English literary heroine". (Swicle)
The writer explicitly makes a point of the notion that his or her essay only provides the layperson's
view on the topic (in this essay, which is of course wholly unscholarly), thus telling the reader
something about how to interpret the text. The writer is careful not to present him- or herself as an
expert in the field. This could be interpreted as if the learner writer does not feel confident in the role of
an argumentative essay writer or with the possible objectivity and learnedness that traditionally
accompany this task. A similar example is the following:
(16) Of course, you may wonder what the true reason for heavy taxation on these
pleasures is. Is it really a concern about the average Swede's health and well being or is
it, as many suspect, a concern about the government finances that is the reason? I have
not the competence required to answer this question, but if you ask me if I think that
taxes and restrictions are the right methods to keep people off the bottle and away from
the cigarette, the answer is no. (Swicle)
A question is posed, but the writer chooses not to answer it on the grounds that he or she is not an
authority on the area. In the following coordinated clause, however, the question is rephrased so that
the writer feels that he or she rightly can answer it. Note the interactive expression if you ask me, which
would also be classified as metatext.
6. How are the metatext units distributed in the texts?
The program WordSmith Tools (http://www.liv.ac.uk/~ms2928/index.htm) has been used to study the
distribution of metatext elements. It gives the percentage of how far into the text file the search word
occurs. The spread of the search word is examined in each text, starting at 1 and ending at 100 per cent.
All percentages have been compiled and divided into tenths (1-10, 11-20, 21-30, etc). Measuring
textual distribution in terms of percentages and not paragraphs can be criticised in several ways, but
looking at the figures and tables in terms of percentages still gives a good picture of the spread of the
metatext units through entire collections of texts.8
In the Swedish learner corpus (see diagram 6a below), there is a clear peak in
metatextual units including first person I in the first parts of the essays. As many as 34 per cent of the
overall number of metatext occur from 1 to 20 per cent into the essays. A certain rise is also noticeable
at the very end, in the last percentage section.
There are few occurrences of metatext having I as subject in the British English
material, and the great majority of units occurs in the very first section (see diagram 6c below).
Although the data are scarce, the overall picture is similar to the distribution in the Swedish learner
essays. In the American English material (diagram 6b), on the other hand, the highest peak is found in
the last section, 91-100 per cent into the texts, but also in the first and third sections.
As expected, the results show that the beginnings and the ends of the essays are
important sections for metadiscourse. Further analysis will reveal whether there are patterns in the use
of individual verbs, for example whether some verbs have a preference for occurring at a certain point
in argumentative essays. Thus, questions to be answered are: Which of the metatextual verbs occur
8 However, it may be the case, for instance, that paragraph length differs across corpora, which could influence the
figures.
11
most frequently at the beginning or at the end of texts? Why is that? What do the forms look like? What
functions do they have in the texts?
Diagrams 6 a-c and 7 a-c. The distribution of metatext units involving I and we in the three corpora.
Diagram 6a. The distribution of
occurrences of I involving
metatext in SWICLE
Diagram 6b. The distribution of
occurrences of I involving
metatext in US LOCNESS
Diagram 6c. The distribution of
occurrences of I involving
metatext in UK LOCNESS
Diagram 7a. The distribution of
occurrences of we involving
metatext in SWICLE
Diagram 7b. The distribution of
occurrences of we involving
metatext in US LOCNESS
Diagram 7c. The distribution of
occurrences of we involving
metatext in UK LOCNESS
If we compare the patterns of distribution of metatext involving I and we in the individual corpora, the
neatest match is in the British English data, in which the highest numbers are in the very first section (8
occurrences in both cases, see diagrams 6c and 7c above).
The figures in the American English data are only partly similar (diagrams 6b and 7b).
There is a peak at the end for both pronouns, but it starts already at 81-90% for I, and the numbers are
twice as high. The increase of metatext including I at the beginning do not correspond to a rise in the
results including we. The top score for metatext involving we, instead, occurs at 41-50 per cent.
The distribution of metatext with I and we as subjects is basically reversed in the
SWICLE material (diagrams 6a and 7a). In the case of I, there are two evident tops at the very
beginning and at the very end of the essays. In the data involving we, on the other hand, the lowest
numbers occur in those places, and the rise takes place at 21-30 and 31-40 per cent.
7. Conclusion
Some evidence that the three groups represented differ both quantitatively and qualitatively in their use
of metatext as well as in their overall use of first person pronouns has been presented in this paper. Any
definite answer to the reason why the differences between the corpora exist – in particular with regard
to the learner essays versus the native speaker texts – will have to be left until the material has been
analysed in detail.
However, one partial and tentative answer may be that the differences are related to
different cultural conventions for writing. First of all, there is a possibility that the conventions for
using metatext in argumentative writing are not the same in Swedish and English. In addition, there
seem to be differences with regard to writer visibility and use of metatext among the two varieties of
English in the native speaker corpus. For example, there is a clear dissimilarity in the use of I in the
American and British English parts (43 versus 9 occurrences per 10,000 words, see table 1 above). This
suggests that there may be cultural differences involved, and the division of the LOCNESS corpus into
two parts made for the present investigation is endorsed.
To some extent, the presence of the discourse participants in texts and the use of
pronouns are also a question of politics (see Wales 1996:84). The critical discourse perspective (see e g
12
Fairclough 1995) deals very much with ideology in texts, which may be done for example by asking
questions such as: Is the agent behind the text visible? Is the position taken by the writer explicitly
stated as belonging to the writer or is it rendered as a general truth or common knowledge? In this field,
linguistic forms and structures are studied in terms of power and ideology, such as the agentless passive
construction which, in contrast to personalised expressions, leaves causality and agency unclear –
possibly with the aim of conscious hedging or even deception.
Acknowledgement
I would like to thank professor Karin Aijmer for reading and commenting on my paper.
References
Altenberg B 1997 Exploring the Swedish component of the International Corpus of Learner English. In
Lewandowska-Tomaszcyk B, Melia P J (eds) PALC'97: Practical applications in language corpora.
Lódz, Lódz University Press, pp 119-132.
Bäcklund I 1998 Metatext in professional writing: a contrastive study of English, German and Swedish.
Texts in European Writing Communities 3. Uppsala, Uppsala Universitet.
Biber D, Johansson S, Leech G, Conrad S, Finegan E 1999 The Longman grammar of spoken and
written English. London, Longman.
Bondi M 1999 English across genres: language variation in the discourse of economics. Modena,
Edizioni Il Fiorino.
Chafe W L 1982 Integration and involvement in speaking, writing, and oral literature. In Tannen D (ed)
Spoken and written language: exploring orality and literacy. Norwood, Ablex, pp 35-53.
Clark R, Ivanic R 1997 The politics of writing. London and New York, Routledge.
Connor U 1996 Contrastive rhetoric: cross-cultural aspects of second-language writing. Cambridge,
Cambridge University Press.
Crismore A, Farnsworth R 1990 Metadiscourse in popular and professional science discourse. In Nash
W (ed) The writing scholar: studies in academic discourse. Written Communication Annual: An
international survey of research and theory: Volume 3. Newbury Park/London/New Delhi, Sage
Publications, pp 118-136.
Crismore A, Markkanen R, Steffensen M S 1993 Metadiscourse in persuasive writing: a study of texts
written by American and Finnish university students. Written Communication 10: 39-71.
Fairclough N 1995 Critical discourse analysis: the critical study of language. London and New York,
Longman.
Granger S 1993 The International Corpus of Learner English. In Aarts J, de Haan P, Oostdijk N (eds)
English language corpora: design, analysis and exploitation. Amsterdam, Rodopi, pp 51-71.
Hyland K 1998 Persuasion and context: the pragmatics of academic discourse. Journal of Pragmatics
30: 437-455.
Jakobson R 1980 The framework of language. Michigan, Michigan Studies in the Humanities.
Markkanen R, Steffensen M , Crismore A 1993 Quantitative Contrastive study of metadiscourse:
problems in design and analysis of data. Papers and Studies in Contrastive Linguistics 28: 137-152.
Mauranen A 1993 Cultural differences in academic rhetoric: a textlinguistic study. Frankfurt am Main,
Peter Lang.
Petch-Tyson S 1998 Writer/reader visibility in EFL written discourse. In Granger S (ed) Learner
English on computer. London and New York, Longman, pp 107-118.
Quirk R, Greenbaum S, Leech G, Svartvik J 1985 A comprehensive grammar of the English language.
London, Longman.
Taavitsainen I 2000 Metadiscursive practices and the evolution of early English Medical writing 1375-
1550. In Kirk J M (ed) Corpora galore: analyses and techniques in describing English. Papers from the
nineteenth international conference on English language research on computerised corpora (ICAME
1998). Amsterdam and Atlanta, Rodopi, pp 191-207.
Tadros A 1993 The pragmatics of text averral and attribution in academic texts. In Hoey M (ed) Data,
description, discourse: papers on the English language in honour of John McH Sinclair. London,
HarperCollins, pp 98-114.
Vande Kopple W J 1988 Metadiscourse and the recall of modality markers. Visible Language XXII:
233-27.
Vassileva I 1998 Who am I/who are we in academic writing?: A contrastive analysis of authorial
presence in English, German, French, Russian and Bulgarian. International Journal of Applied
Linguistics, Vol 8, No 2: 163-190.
Wales K 1996 Personal pronouns in present-day English. Cambridge, Cambridge University Press.

13
Discourse particles in contrast
Karin Aijmer
English Department
Göteborg University
There are several new avenues to the study of discourse particles (well, now, actually, in fact, etc ) such
as the study of discourse particles across different languages.
Cross-linguistic comparison of discourse particles represents what Carlson calls ‘an extreme case of
contrastive analysis’ providing ‘potential evidence for an underlying set of shared (perhaps universal)
functional distinctions (Carlson 1984: 68). The external contrastive analysis implies that
correspondences are established between elements in different languages.
Investigating translation equivalents of discourse particles can be regarded as a typical corpus problem
since it involves looking for all the possible equivalents in order to establish rules for how they are
used. Translation data show how transfer of the semantic content results in a rich variety of lexical
items. The results may be analysed in terms of formal, semantic, stylistic and pragmatic equivalence.
For this study I have used the English-Swedish Parallel Corpus, a 2 million word corpus containing
translations from English into Swedish and Swedish into English to initiate fine-grained cross-linguistic
comparisons between languages and to test hypotheses (Aijmer et al 1996). For example, on the basis
of theories of grammaticalisation it is possible to make cross-linguistic predictions that can be tested on
the basis of the corpus.
References
Aijmer K, Altenberg B, Johansson M 1996 Text-based contrastive studies in English. Presentation of a
project. In Aijmer K, Altenberg B, Johansson M (eds). Languages in contrast. Papers from a
symposium on text-based cross-linguistic studies. Lund: Lund University Press. 73-85.
Carlson L 1984. ‘Well’ in dialogue games: A discourse analysis of the interjection ‘well’ in idealized
conversation. Amsterdam and Philadelphia: John Benjamins


14
John is a man of (good) vision: enrichment with evaluative meanings*
Takanobu Akiyama
University of Lancaster
1. Introduction

In human communication, the linguistic form of an utterance often provides the hearer with only very
skeletal information, and thus the hearer requires pragmatic inferential processing for interpretation of
assumptions that are communicated by the utterance. It is reasonable to state that this pragmatic
inferential processing is crucially contingent on contexts. In general, however, the term context tends to
be restricted either to the discourse or co-text or to the physical situation. Relevance theory (Sperber and
Wilson 19952), in order to avoid this ad hoc explanation of the context, defines it as a subset of the
hearer's beliefs and assumptions about the world which interacts with newly received information.
This paper, taking these into account, attempts to provide an account of pragmatic enrichment with
evaluative adjectival meanings which occurs in interpretation of man of + N construction (e.g. John is a
man of (good) vision). Pragmatic enrichment (or simply ‘enrichment') is a pragmatic inferential process
of fleshing out the logical form of an utterance in order to recover the explicature of the utterance.
Interpretation of phrases such as a man of sense, taste, vision, property, etc. consistently requires
enrichment processing with evaluative adjectival meanings, but there has been no research into the
mechanism under which this sort of enrichment processing occurs. This problem will be discussed in the
framework of ‘relevance theory’ on the lines proposed by Sperber and Wilson (19952); this theory
appears to be well equipped to make headway in the murky area involving various pragmatic phenomena.
Yet, our examination is also carried out on the basis of data retrieved from the British National Corpus
(BNC), because an introspection-based analysis alone does not afford an opportunity to construct a truly
convincing argument.
The discussion below is organised as follows. Section 2 is devoted to a preliminary discussion of
certain indispensable pragmatic concepts relating to arguments in this paper and to the establishment of a
consistent perspective for analysing the target construction. Here we will attempt to define the term
‘context', because one may presume that enrichment processing with evaluative meanings in the target
construction is independent of the context, if the term ‘context’ is merely confined to external (i.e.
linguistic and physical) contexts. Section 3 presents a survey of the target construction on the basis of
data retrieved from the British National Corpus (BNC). We will classify phrases that give rise to
evaluative enrichment into two types. Section 4 discusses how enrichment processing in the man of + N
construction is motivated by relevance theory and establishes a hypothesis which is concerned with what
the hearer's evaluative viewpoint is based on. Section 5 concludes this paper with a brief summary.
2. Preliminary discussion

2.1. Explicature and implicature: two levels of communicated assumptions
Before embarking on an analysis of enrichment with evaluative adjectival meanings, it is necessary to
make explicit certain basic concepts relating to pragmatic processing and to establish, as far as possible,
a consistent perspective for analysing it.
Sperber and Wilson (19952) and followers of their theory (e.g. Blakemore (1987, 1992, 1995)) sort
hearers’ assumptions regarding what is communicated by utterance into two levels, viz. explicature and
implicature. Explicature is the first level of assumption in utterance interpretation, at which the hearer
fleshes out the under-determined form produced by the speaker to a fully elaborated, and accordingly
explicit propositional form. At this level of assumption, the hearer's inference is based on the original
utterance form, and is usually recovered by three types of processing, viz. disambiguation, reference
assignment, and enrichment. Implicature, on the other hand, is the second level of assumption
communicated by the utterance, where the hearer's inference is not based upon the original utterance
form but upon the explicature, and thus the implicature can be quite removed from the original utterance
form. A distinction between explicature and implicature can be hence drawn in terms of whether the
assumption is contingent on the original utterance form or not. To enhance our understanding of the
distinction between these two communicated assumptions, we will examine a breakfast conversation,
during a scene in which Philip (husband), Jane (wife), and David (their son) are eating slices of toast.
(1) Philip: David, another piece of this.

15
Jane: I think he's had enough. (BNC: KCH 430)
In the conversation above, both explicature and implicature are requisite for the interpretation of each
utterance. Let us suppose that Jane's interpretation of Phillip's first utterance is: ‘David, you can have
another piece of toast.’ This assumption can be called explicature, because it is recovered in terms of the
enrichment of ‘you can have’ and the reference assignment of the pronoun ‘this’ with ‘toast', motivated
by the under-determinacy of the original utterance. Moreover, Philip's interpretation of Jane's utterance
could be that ‘I think David has had enough toast already’ (viz. explicature), and then ‘If he eats any more
he'll be sick. So don't give him the toast’ (viz. implicature). In Jane's utterance, the foregoing three types
of processing to recover the explicature can be found: the pronouns ‘I’ and ‘he’ refer to the speaker (i.e.
Jane) and David, respectively (viz. reference assignment), the verb ‘have had’ is to be construed as ‘have
consumed', not ‘have possessed’ (viz. disambiguation), and some word meanings (i.e. toast and already)
are added (viz. enrichment).
At issue here is what the recovery of these communicated assumptions is based on. Let us confine
ourselves to discussing explicature for simplicity of exposition. We skate over implicature here.
2.2 Perspectives on ‘context': external contexts
As we have seen in the conversation (1) above, the original utterance form often provides only a very
skeletal clue as to the explicature to be recovered. It will thus be reasonable to state that the process of
fleshing out the original utterance form into an explicature crucially hinges on contexts. In the
conversation (1), for instance, we will have to make much more effort in recovering these explicatures if
we do not know that this conversation occurs during a breakfast scene. Further, Philip's first utterance,
i.e. ‘David, another piece of this', could not be interpreted properly, unless we knew from the context that
the participants of the conversation were eating toast. The explication of explicature therefore stands in
need of a consistent perspective on the term ‘context'.
‘Context’ has been a very widely used term in linguistics and, as a consequence, any account of its
meaning will be required to specify exactly how it is being used. Yet, many of the previous analyses of
the context, while aiming to reify this abstract term, have mainly directed attention to external contexts
and several types of them have been put forward, e.g. linguistic context (co-text), physical context
(setting), etc. The linguistic context (co-text) will be the most easily postulated as a definition of context.
It refers to the sounds, words, phrases, and so on, which come immediately before and after a particular
phrase or piece of text and help to explain its meaning. The major role of the linguistic context for
interpretation of utterances will be concerned with cohesion (c.f. Verschueren (1999: 104)). In other
words, hearers attempt to solve reference assignment, disambiguation, etc. by referring back to earlier
discourse or projecting toward a future linguistic context in order to give coherence to utterance
interpretation. In the conversation (1), for instance, interpretation of Jane's utterance ‘I think he's had
enough’ will require as a linguistic context Philip's previous utterance ‘David, another piece of this', to
resolve a reference assignment of ‘he’ as ‘David'. The physical context (setting) can be defined as ‘the
spatio-temporal location of the utterance, i.e. as the particular time (moment) and particular place at
which speaker gives utterance and the particular time and place at which the hearer hears or reads the
utterance’ (Allan (1986: 36)). This type of context is, thus, in particular concerned with the reference
assignment of deictic expressions of place (e.g. ‘here', ‘there') and of time (e.g. ‘yesterday', ‘last week').
Suppose that you are told by someone ‘Sit here; but don't sit here, please'. You will be thrown into
confusion by the indeterminacy of ‘here’ in this utterance unless the speaker designates two locations (e.g.
two chairs) alternatively.
It is indubitable that utterance interpretation is very highly determined by contextual factors, but it will
be inadequate to direct attention to only the linguistic and the physical contexts, say the external contexts.
In the following utterances (2) and (3), the explicatures recovered by means of enrichment processing do
not seem to be dependent on those external contexts.
(2) a. Whatever else the mysterious Arianna might be, she was clearly a woman of taste.1
(BNC: JY7 4224)
b. = …, she was clearly a woman of good/*bad taste.
(3) a. Wordsworth was a man of feeling, for an Englishman, but well compared to Oor Rabbie quite
frankly he was nothing to write home about. (BNC: B38 112)
b. = Wordsworth was a man of strong/*weak feeling, …


Hereafter, italics in corpus sentences quoted from the BNC are inserted to highlight relevant parts of the
example.


16
Application of those external (i.e. the linguistic and the physical) contexts to the explication of the
enrichment in (2b) and (3b) will be both pointless and stultifying. In the next section, in order to avoid
this impasse, we will turn attention to a ‘psychological construct’ as per Sperber and Wilson (19952).
2.3. A psychological construct as a context
While the emphasis in the previous analyses of context has been placed mainly on the external contexts
in which an utterance is made, Sperber and Wilson (19952:15ff) lay stress on the (internal) psychological
construct, viz. the subset of the hearer's current knowledge, beliefs, assumptions, hypothesis, and
cultural and social conventions about the world. Of interest here is that Sperber and Wilson do not
categorise contexts into several types but unify them into a single one that is supposed to exist in our
mind. In their view, understanding an utterance is a matter of integrating the proposition it expresses with
a context of existing beliefs and assumptions.
The set of premises used in interpreting an utterance […] constitutes what is generally known as the
context. A context is a psychological construct, a subset of the hearer's assumptions about the
world. (Sperber and Wilson 19952: 15)2
This will come close to saying that the ‘psychological construct’ here subsumes a notion ‘schema’ in
cognitive psychology (cf. Blakemore 1992: 17: Sperber and Wilson 19952:138). The schema is ‘a
structured cluster of concepts; usually it involves generic knowledge and may be used to represent events,
sequences of events, percepts, situations, relations, and even objects’ (Eysenck and Keane 20004: 252).
In what follows I will claim that the psychological construct of the schema goes hand in hand with the
enrichment processing of the construction ‘man of + noun’ (see (2-3) above).
3. Enrichment in ‘Man of + N (noun)’
3.1. Classification of the target construction
First of all let us start our analyses of enrichment of the target construction by classifying this
construction according to the meaning of nouns following the man of. Our examination in the following
discussion is based on data from the BNC, because an introspection-based analysis alone does not afford
an opportunity to construct a truly convincing argument. The BNC contains 2060 examples of a
collocation man of in 970 different texts, and 260 types and 651 tokens extracted from these 2060
examples are ‘man of + noun (N)'. These 260 types of ‘man of + N’ fall roughly into the following three
categories: (I) the N characterises the man's activity or occupation (e.g. a man of letters); (II) the N
denotes a group, nationality etc (e.g. a man of French); (III) the N denotes the man's essential feature or
his trait, which is based on his physical/psychological/social properties. (e.g. a man of sense; a man of
principles)3
Category (I) includes as its member a man of action, commerce, letters, business, law, peace, etc. The
nouns in this category characterise the man's activity, which leads to his occupation. For instance, a man
of action denotes a man whose whole life is a soldier, sportsman, political leader, etc.; a man of peace
denotes a man whose life is to bring about peace; a man of letters denotes a man who writes works of
literature or writes about literature. Category (II) has as members a man of Europe, India, Borneo, China
etc. In what follows, these two categories will be excluded from our consideration.
Category (III), which is the object of our investigation, refers to a man's physical/psychological/social
characteristic or some essential feature of his character. Some nouns appearing in this category bring
about enrichment with evaluative meanings for interpretation of the whole phrase (e.g. a man of (good)
sense). In my investigation there are 41 phrases belonging to this category; the examples and their
frequencies are described in Table 1. Here we should notice that not all the phrases in the category (III)
undergo enrichment with evaluative meanings for the interpretation.


Blakemore (1992: 87), following Sperber and Wilson, defines the term context as ‘the beliefs and
assumptions the hearer constructs for the interpretation of an utterance either on the basis of her
perceptual abilities or on the basis of the assumptions she has stored in memory or on the basis of her
interpretation of previous utterances'.


There are a small number of phrases in which N refers to a man's name (e.g. a man of John). Yet, for
this very small number of members, I do not set up another category.


17
Table 1 Phrases and Frequencies of Category (III)
phrase freq. phrase freq.
man of honour 19 man of experience 2
man of integrity 10 man of principles 2
man of principle 9 man of quality 2
man of power 8 man of ability 1
man of reason 8 man of ambition 1
man of influence 7 man of anger 1
man of property 7 man of caution 1
man of vision 5 man of contradictions 1
man of courage 4 man of excitement 1
man of culture 4 man of family 1
man of genius 4 man of fidelity 1
man of knowledge 4 man of goodwill 1
man of sense 4 man of humour 1
man of taste 4 man of ideas 1
man of character 3 man of individuality 1
man of dignity 3 man of passions 1
man of feeling 3 man of spirit 1
man of learning 3 man of self-discipline 1
man of passion 3 man of strength 1
man of standing 3 man of surprise 1
man of energy 2
3.2. Analysis of ‘man of + N’
3.2.1. Phrases of truisms
Phrases which obviously yield enrichment with evaluative meanings are listed up in Table 2. These
phrases are retrieved from Table 1. Most of the phrases refer to a man's physical or psychological
properties and others make reference to a man's social properties. Phrases concerning a man's physical
or psychological properties are a man of power, vision, knowledge, sense, taste, character, feeling,
energy, experience, principles, ability, ideas, spirit, and strength. Phrases in relation to a man's social
properties are a man of property, culture, standing, quality, and family.
Table 2 Phrases which Obviously Yield Enrichment with Evaluative Meanings for their Interpretation
phrase freq. phrase freq.
man of power 8 man of energy 2
man of property 7 man of experience 2
man of vision 5 man of principles 2
man of culture 4 man of quality 2
man of knowledge 4 man of ability 1
man of sense 4 man of family 1
man of taste 4 man of ideas 1
man of character 3 man of spirit 1
man of feeling 3 man of strength 1
man of standing 3
Phrases listed in Table 2 obviously give rise to evaluative enrichment for their interpretation. All Ns
following man of in these phrases, which refer to human physical, psychological or social properties are
considered as being the endowment of all humans ubiquitously. Another common denominator among
these Ns is that in their lexical level they do not denote a specific (e.g. high or low; good or bad) degree.
Power, for example, basically denotes a particular ability of the body or mind but does not specify its
degree in its lexical level. Yet power denotes a high degree sense in the man of + N construction with

18
enrichment processing.
(4) a. Pope Gregory the Great had spoken of taming the wild unicorn, symbol of the man of power.
(BNC: HPT 616)
b. = …, symbol of the man of great/*low power.
Property, culture, standing, quality, and family are usually assumed to be possessed by humans in their
social life. For instance, a man of property, in its linguistically decoded logical form without enrichment
processing, denotes a man who owns things. This logical form itself does not carry any new information.
A close look at the data in table 2 reveals that Ns following the collocation man of commonly denote
properties of human beings. Moreover I reiterate that these properties are assumed as ubiquitous among
humans. It is of significance therefore that the phrases in Table 2 can result in a ‘truism', if we do not
interpret these phrases with enrichment processing. A ‘truism’ is a statement that is obviously true, in
particular one that does not say anything important.
On the other hand, sentences such as He is a man of anger, passion, excitement, goodwill, and surprise,
all of which also make reference to a man's psychological property, are not likely to give rise to
enrichment with evaluative meanings. This is because these Ns following man of clearly contain a
gradable adjectival feature inherent in their basic senses. Anger, for example, denotes a ‘strong’ feeling
of resentment, and thus denotes a strong feeling of wanting to harm or criticise someone because they
have done something unfair. Besides, man of honour, integrity, courage, genius, dignity, caution,
contradictions, fidelity, and self-discipline do not need enrichment processing with evaluative meanings
for their interpretation. This will be because properties depicted by these Ns following man of are
considered as distinctive or unique rather than ubiquitous among humans.
(5) a. "I am a man of honour as well as a royal emissary." (BNC: H9C 1638)
b. It is "mental bombast" as opposed to verbal bombast, and it is a "fault of which none but a man of
genius is capable". (BNC: CDL 1530)
3.2.2. Phrases of Vagueness
There are some more phrases that require enrichment processing with evaluative meanings for their
interpretation in the man of + N construction: these are listed in Table 3. It will be normal and expected in
our social life for humans to have properties described by Ns following man of in this table. However, we
cannot say that every human being is assumed to possess these properties equally. Therefore
linguistically decoded logical forms of these phrases do not obviously give rise to truism but rather
vagueness of their meanings, if these phrases are interpreted without enrichment. Further it is not
obvious but likely that enrichment processing with evaluative meaning for the phrase interpretation in
Table 3 occurs. It follows from this that the possibility of occurrence of enrichment processing in the
man of + N construction is a matter of degree.
Table 3. Phrases which are Likely to Yield Enrichment with Evaluative Meanings for their Interpretation
phrase freq.
man of reason 8
man of influence 7
man of learning 3
man of humour 1
man of individuality 1
It will be admissible to state that the scales of properties described by Ns in these phrases are vague, and
this vagueness provides a motivation for enrichment processing.
(6) a. He was a man of influence in the literary world. (BNC: AC3 1486)
b. = He was a man of great/*small influence in the literary world.
(7) a. I'm a man of individuality where garden embellishment is concerned. (BNC: ACX 184)
b. = I'm a man of strong/*weak individuality…
In (6a) the meaning of the logical form of a man of influence is vague. In other words, it is not obvious
how much influence the man in question has in the literary world, if enrichment processing is not carried
out. In (7a) a man of individuality in its logical form also denotes a vague sense, because the meaning of
the lexical item individuality, i.e. the quality that makes someone different from all other people, does not

19
carry a specific sense in a degree.
I claim that a common motivation for enrichment shared by truism-types of phrases (e.g. a man of taste)
and vagueness-types of phrases (e.g. a man of influence) is a lack of relevance, a technical notion about
communication put forward by Sperber and Wilson (19952), which will give a unitary account of
mechanism of enrichment processing in man of + N construction. In the next section we will discuss the
relationship between this notion and enrichment processing in the construction under discussion.
4. Relevance Theory Approach to Enrichment Processing in Man of + N Construction

As it has been established that the N which appears in phrases in Table 2 commonly has essential
properties (e.g. a man of taste), it will be reasonable to state that these features of Ns provide a clue to the
exploration of the process of enrichment in ‘man of + N’ construction. When the noun denotes essential
properties of human beings and is neutral as to quality/value in its basic sense, the construction ‘man of +
N’ will become a kind of truism. So information in the truism will not cause the hearer to add any new
information, or strengthen his/her existing assumption, or change her/his mind. In this section, we will
discuss mechanism of enrichment processing of the target construction in the framework of ‘relevance
theory’ on the lines proposed by Sperber and Wilson (19952).
4.1. Communicative Principle of Relevance
According to the relevance theory, ‘every aspect of communication and cognition is governed by the
search for relevance’ (Wilson and Sperber 1998: 9). This theory assumes that human communication and
cognition are relevance oriented; we pay attention to information that seems relevant to us. The technical
notion ‘relevance’ is measured in terms of the relationship between ‘contextual effects’ and ‘processing
effort'. Contextual effects are a kind of added set of conclusions that newly received information gives
rise to, by interacting with a context, in order for the information to be relevant to the hearer. There are
three ways in which newly received information may interact with the context to give rise to a contextual
effect: (i) it may combine inferentially with contextual assumptions to yield a contextual implication; (ii)
it may strengthen an existing assumption; (iii) it may contradict and lead to the elimination of an existing
assumption (see Blakemore (1992: 30), (1995: 445); Sperber and Wilson (19952: 108ff); Wilson and
Sperber (1998: 8)). Other things being equal, the more contextual effects there are, the greater the
relevance of particular information. However, these contextual effects, say context-dependent benefits in
communication, do not occur without any cost but require ‘processing effort', and therefore ‘An
assumption is relevant to an individual to the extent that the effort required to achieve these positive
cognitive effects is small’ (Sperber and Wilson 19952: 266). In other words, contextual effects need to be
economically achieved, and thus the harder a hearer has to try to interpret the information, the less
relevant it is. To summarise, a highly relevant utterance has large contextual effects for small processing
effort, and an utterance of small relevance has a processing effort which exceeds its potential contextual
effect.
Taking these notions into account, Sperber and Wilson (19952) enunciate the following fundamental
principle about human communication:
(8) (Communicative) Principle of Relevance: every ostensive stimulus communicates a presumption of
its own optimal relevance. (Sperber and Wilson 19952: 158)
This will come close to saying that every ostensive (i.e. deliberate and overt) communication involves a
presumption that utterance will have adequate contextual effects for the minimum necessary processing.
Let us turn back to the discussion of our original problem. We have seen that the ‘man of + N’
construction, several instances of which give rise to enrichment with evaluative meanings for their
interpretation (e.g. a man of taste), would be understood as a truism, if the phrase were interpreted
without the enrichment processing. (That is, every man can be assumed to have some kind of taste.) In
this case, the processing effort for the truism-interpretation will be minimal because this interpretation is
merely dependent on the hearer's lexical knowledge of the lexical items (i.e. man, of, and taste), but at the
same time this truism-interpretation also yields a minimal contextual effect. There will be no relevance
in this interpretation. Yet, in order to follow the communicative principle of relevance, the hearer tries to
find enough contextual effects and adds some meaning (e.g. good in a man of good taste). In this case,
the processing effort is likely to be small, but the contextual effect will be adequately large to avoid a
truism and to characterise the man in some positive, informative way. The contextual effect in ‘a man of
good taste’ is applied to case (i), viz. ‘it (i.e. newly received information) may combine inferentially with

20
contextual assumptions to yield new conclusions'.
Moreover, analogous comments hold in the case of a man of influence', which is a member of the
category depicted in Table 3 above. The interpretation of this phrase, if it does not involve enrichment
processing, will yield vagueness, or indeterminacy in the meaning of this phrase (some degree of
influence, whether great or small). Indeterminacy of meaning motivates enrichment processing, and thus
a man of influence will be enriched to a man of considerable/great influence. Since the linguistically
decoded logical form a man of influence brings about vagueness rather than a truism, it yields a small
contextual effect with minimal processing effort. However, the hearer tries to make explicit the meaning
of this phrase in order to solve the vagueness of the logical form, and thus he/she enriches the phrase
meaning. Let us here recapitulate the foregoing discussion of the two scales (i.e. contextual effect and
processing effort) in relation to enrichment processing of the target construction in the form of an outline
summary in Table 4.
Table 4. Two Scales Relating to Enrichment Processing in ‘a man of taste/influence’
Logical Form and Explicature Contextual Effect Processing Effort
John is a man of taste minimal minimal
(as a logical form [i.e. truism])
John is a man of great taste (adequately) large small
(as an explicature)
John is a man of influence small minimal
(as a logical form [i.e. vagueness])
John is a man of great influence (adequately) large small
(as an explicature)
What is common to both these phrases (i.e. a man of taste and a man of influence) for their interpretation
is that enrichments are motivated by the lack of relevance of the linguistically decoded logical forms of
their original utterances. Comparison of the contextual effects of the logical forms between a man of
taste and a man of influence suggests that the lack of relevance is a matter of degree.
4.2. Advantage-disadvantage hypothesis
So far we have discussed how enrichment is processed in the ‘man of + N’ construction, but our
discussion has not accounted for the fact that meanings added in explicature of this target construction
tend to have positive evaluative meanings (e.g. good in a man of good taste). In order to shed light on
this problem, I put forward the following hypothesis: an ‘advantage-disadvantage’ hypothesis.
(9) Advantage-disadvantage: The hearer's evaluative viewpoint is based on the
‘advantage-disadvantage’ that he/she conceives of in his/her knowledge or assumptions about the
world (e.g. schemata).
‘Advantage-disadvantage’ refers to a human's psychological response to situations in which things
happen. Further, I claim that a hearer's judgement of advantage/disadvantage is made in terms of a
human's schemata (or encyclopaedic knowledge) in his/her beliefs and assumptions about the world. We
can apply the advantage-disadvantage hypothesis to the elucidation of enrichment processing in the ‘man
of + N’ construction (e.g. a man of (good) taste). Possession of properties such as taste, sense, influence,
learning will be advantageous for a human's life and thus it will be evaluated as positive. It is for this
reason that phrases such as a man of taste or a man of influence usually receive positive evaluative
meanings for their explicatures.
In order to make the preceding hypothesis defensible, we must examine more examples.
(10) a. This milk has a smell.
b. = This milk has a *good/bad smell.
The state in which milk has a smell is supposed to be abnormal and the milk in this abnormal state will be
unhealthy (i.e. disadvantage). This inferential process on the basis of the advantage-disadvantage
hypothesis gives rise to the additional sense of bad, i.e. a negative evaluative meaning, to the explicature
presented in (10b). On this basis, the foregoing hypothesis seems to be on the right track.
5. Conclusion
This paper has attempted to elucidate the enrichment processing with evaluative meanings in the man of
+ N construction. Our corpus investigation of the target construction reveals that phrases which yield

21
enrichment with evaluative meanings can split into two types, viz. the truism type (e.g. a man of taste)
and the vagueness type (e.g. a man of influence). The common motivation for enrichment between these
two types is a lack of relevance. That is, the linguistically decoded logical forms of these types of phrases
cannot bring about contextual effects, which should be large in context in order for the newly impinging
information (i.e. utterance) to be relevant. In order to avoid the lack of relevance, or in other words, to
follow the communicative principle of relevance, the hearer enriches the logical form to give rise to an
appropriate contextual effect. Moreover, we found that enrichment of the man of + N construction tended
to require positive evaluative meanings (e.g. a man of good taste). Consequently I put forward the
advantage-disadvantage hypothesis for the explication of the hearer's evaluative viewpoint. One may
claim that this hypothesis comes into conflict with the principle of relevance methodologically, because
the principle of relevance is supposed to be a unitary principle which governs all human communication.
However, the hypothesis I have provided is not a rule but a description of a tendency regarding the
hearer's evaluative viewpoints. While the hearer's evaluative viewpoint tends to be based on normal
states in his/her schemata, the choice of the normal state which is utilised for interpretation of an
utterance is contingent on the hearer's contextual assumption at a given moment, in order to follow the
principle of relevance.
The advantage-disadvantage hypothesis is likely to be applied to the explication of evaluative
enrichment in other constructions or even in constructions in other languages. With regard to the
possibility of generalisation of this hypothesis, however, we have to wait for further research.
*This is a revised version of Akiyama (2000: Unpublished paper).
References
Akiyama, T. 2000 Pragmatic enrichment in the man of + N construction. Unpublished paper. University
of Lancaster.
Allan, Keith (1986) Linguistic Meaning, vol.1. London: Routledge & Kegan Paul.
Blakemore, D. 1987 Semantic constraints on relevance. Oxford, Blackwell.
Blakemore, D. 1992 Understanding utterances: an introduction to pragmatics. Oxford, Blackwell.
Blakemore, D. 1995 Relevance Theory. In Verschueren, J., J. Östman, J. Blommaert (eds.) Handbook of
pragmatics, manual. Amsterdam: John Benjamins. p.p. 443-52.
Eysenck, M. W., M. T. Keane 20004 Cognitive psychology: a student handbook. Hillsdale, N. Jersey:
Lawrence Erlbaum.
Sperber, D., D. Wilson 19952 Relevance: communication and cognition, Oxford: Blackwell.
Verschueren, Jef (1999) Understanding Pragmatics. London: Arnold
Wilson, D., D. Sperber 1998 Pragmatics and Time. In Carston, R., S. Uchida (eds.) Relevance theory:
applications and implications, Amsterdam: John Benjamins Publishing Company. 1-22

22
Word order variations and spoken man-machine dialogue in French : a corpus
analysis on the ATIS domain
Jean-Yves Antoine, Jérôme Goulian
VALORIA, University of South Britanny
IUP Vannes, rue Yves Mainguy, F-56000 Vannes, France
Phone : +33 2 97 68 32 10 — Email : {Jean-Yves.Antoine,Jerome.Goulian}@univ-ubs.fr
WWW : http://www.univ-ubs.fr/valoria/antoine
1. Introduction
During the last decade, spoken man-machine dialogue has known significant improvements that
should lead shortly to the development of real use systems. In spite of these indisputable advances,
numerous limitations restrict still the expansion of common use spoken dialogue systems. In
particular, present researches in spoken man-machine communication lack seriously genericity. Most
of spoken dialogue systems are indeed concerned by a unique application domain : transport
information (ATIS domain). This task is very restricted, what allows the achievement of ad hoc
processing methods that ignore most of the structure of the sentence — see for instance (Minker,
1999) for an overview concerning speech understanding. Although these approaches show a
significant robustness on spontaneous speech, their portability to other application domains remains
an open issue (Hirschman, 1998) : one should reasonably assume that less restricted tasks require a
more detailed linguistic analysis.
As a result, future advances of man-machine communication depend on the improvement of current
spoken language models. In our opinion, corpus linguistics should be of great help for the
development of such improved models :
- the analysis of large task-specific corpora should provide a precise characterisation of the
linguistic phenomena that occur in the concerned application domain. This characterisation is very
useful for the achievement of a system prototype1, but should be helpful for evaluation purposes
too (Antoine et al, 1999).
- the comparison of different corpora should assess usefully the linguistic variabilities ¾ and their
causes : task influence, familiarisation with the task, kind of user,... ¾ that should occur in spoken
man-machine dialogue. It should therefore provide answers to the important problem of genericity.
In this paper, we present a corpus analysis which concerns specifically word order variations in
spoken French. This analysis has been carried out on a corpus of spoken man-man dialogue — the Air
France corpus (Morel et al., 1989) — that corresponds to the ATIS domain. At first, this paper
presents briefly the problem of word order freedom and its implications for natural language
processing. We then detail the main results of this corpus analysis from a strictly linguistic point of
view. In particular, we assess the influence of the familiarisation with the task by means of a
comparison of two subparts of the corpus (see section 3). We finally discuss the consequences of these
linguistic observations on the achievement of spoken dialogue systems.
2. Word order and natural language processing
Word order is an important question for human language technologies. For instance, the problem of
word order freedom originated the development of dependency grammars (Tesnière 1990, Mel'cuk
1988) in response to some weaknesses of standard phrase structure grammars2.
1 Several works have shown for instance that the errors of probabilistic language models may be the result of
systematic failures on a restricted number of linguistic phenomena.
2 This controversy is stille open : see for instance (Rambow and Joshi, 1994) or (Pollard and Sag, 1994).
23
Likewise, stochastic language models (N-grams) depend to a large extent on word order. As a result,
any increase of word order variations should increase harmfully the perplexity of the language model.
Generally speaking, two kinds of word order freedom should be distinguished (Holan et al., 2000) :
- weak word order freedom ¾ called freedom of constituent order within a continuous head domain
by Holan ¾ where a constituent is free to move in several places but remains always continuous.
The corresponding utterance respects therefore the constraint of projectivity. For instance :
(1) on the morning Paul used to go shopping.
- global word order freedom which corresponds on the opposite to a relaxation of continuity. In
such cases, some extracted elements are allowed to appear out of the constituent they are supposed
to belong to : the corresponding utterance is therefore non projective. Consider for instance the
following wh-extraction (Hudson, 2000) :
(2) who do you think that Mary claims that Sarah likes
Global word order freedom concerns above all free word order languages (Russian, Finnish, Czech,...)
whereas rigid languages (English for instance) are more concerned by weak variability (Holan et al.,
2000). Written French should be considered as a rigid word order language (Covington, 1990).
Spoken French is however hardly identifiable to written French (Blanche-Benveniste et al., 1990).
There is therefore no linguistic evidence that spontaneous spoken French presents a word order
variability that is only restricted to weak variation.
Besides the important question of the processing of non projective structures (global word order
freedom), weak variability constitutes a not inconsiderable problem for spoken language technologies.
Since speech recognition provides usually several hypothetical utterances (N-best sequences), any
increase of ambiguity / perplexity due to weak variability should affect noticeably the robustness of the
system. The variability of spontaneous spoken French is therefore an important problem from a
computational point of view. The corpus analysis detailed in this paper aims precisely at answering
this question on a specific task domain.
3. Air France corpus
This analysis has been carried out on a speech corpus which was transcribed from the recording of
real dialogues between a hostess of an air transport information service (ATIS domain) and several
customers (Air France corpus). This corpus represents 103 dialogues that correspond to 5149 speech
turns and 49703 words. It has been divided into two corpora which correspond respectively to
individual customers and travel agents (figure 1) in order to assess the influence of the familiarisation
with the task.
Table 1 — Description of the Air France corpus.
Corpus number of
dialogues
number of
speech turns
number of
words
familiarisation
with the task
individual customers 68 3676 n.c. low
travel agents 35 1473 n.c. high
Total 103 5149 49703 —
This corpus does not correspond to a man-machine interaction, but on the contrary to a dialogue
between two humans. As a matter of fact, our purpose is to characterise the real usages that should be
modelled by spoken dialogue systems.
4. Corpus analysis
We have made an exhaustive inventory of all the extractions that occur in the Air France corpus.
Every observed phenomenon has been characterised according to several features (Gadet, 1992).
1) direction of the extraction ¾ anteposition (movement of an element to the left of the utterance) or
postposition (movement to the right),
24
2) kind of construction ¾ we have distinguished the following kind of extracted constructions :
- simple inversion (extraction without any specific linguistic mark) :
sur Héraklion on n'a qu'un seul tarif special
(AF.II.17.O14
)
[ for Heraklion we have only one unique special fare ]
- dislocations (extraction marked by a clitic)
le visa on l'a eu au consulat (AF.I.48.C6)
[ the visa we have obtained it at the consulate ]
- presentative structures (among which cleaved sentences)
j'ai quelqu'un qu'est allé prendre des billets charters pour moi (AF.I.43.C9)
[I have (there is) someone that went and took charter tickets for me]
c'est une personne de nationalité tunisienne qui a eu ce billet (AF.I.4.O7)
[This is a Tunisian that had this ticket ]
3) syntactic function of the extracted element ¾ subject, argument, adjunct or finally sentence
complement ¾ also called associés (associated elements) in (Blanche-Benveniste, 1997).
4) projectivity ¾ continuous or discontinuous extraction.
These linguistic features have been characterised by their frequencies of occurrence in the Air France
corpus. Since the notion of sentence is not relevant in spoken French (Blanche-Benveniste et al.,
1990), speech turns has been used as unit of segmentation for the computation of these probabilities.
5. Quantitative importance of word order variations in spontaneous spoken French
At first, we present some conclusions that can be drawn from the observation of the whole corpus. The
influence of the familiarisation with the task will be discussed in the following section, which
concerns the comparison between the two sub-corpora (individual customers and travel agents).
First of all, spontaneous spoken French seems to be ¾ given the considered task ¾ noticeably more
flexible than written French. Table 2 shows3 indeed that a not inconsiderable part (13.6%) of the
speech turns presents at least one word order variation.
Table 2 — Frequency of word order variations in the Air France corpus (mean number of speech
turns presenting at least one extraction).
Corpus mean
frequency
standard
variation
minimal
frequency on a
dialogue
maximal frequency
on a dialogue
individual customers 14.9% 6.9% 0.0 % 29.8 %
travel agents 10.1% 8.2% 0.0 % 30.8 %
Total 13.6 % 7.5 % 0.0 % 30.8 %
Besides, the statistical distribution of these frequencies presents a high standard deviation. The use of
word order variations is therefore very variable from a dialogue to an other. It is quite difficult to
explain this variability by means of a unique cause. For instance, dialogic context (negociation,
reformulating,...) is undoubtedly a noticeable source of variability, but one might reasonably assume
that idiosyncratic factors can intervene too. Anyway, it appears that word order variations are rather
common in spoken French. This is why they can not be ignored in the prospect of a robust spoken
3 This table and all the following ones present global results computed on the whole corpus as well as particular
results observed on the sub-corpora that concern respectively “individual customers” and “travel agents”. The
comparison of these last two corpora will be discussed in section 8.
25
man-machine communication. Fortunately, a detailed analysis of the observed extractions shows that
the latter respect to a certain extent some rigid word order constraints.
6. Constrained extractions : projectivity and SVO canonical order
As shown by table 3, most of word order variations correspond unsurprisingly (Gadet, 1992) to
antepositions (82.5 % of the observed variations). This difference between antepositions and
postpositions is statistically significant (c2 test4 of an identical distribution : CHIAF = 0.997).
Table 3 — distribution of the extractions according to their direction (mean percentage of
antepositions and postpositions).
Corpus anteposition postposition standard
deviation
individual customers 82.9% 17.1% 18.4%
travel agents 81.2% 18.8% 24.1%
Total 82.5 % 17.5 % 20.4 %
This predominance of the antepositions can be related to the distribution of the extracted elements
according to their syntactic function (table 4). Word order variations concern above all subjects (30.7
% of the observed variations), sentence complements (30.0 %), adjuncts (27.4%), whereas
subcategorized arguments represent only 12.0 % of the observed variations. This lesser occurrence of
argument extractions is statistically significant (Student mean test5 of identical distribution of subject
and arguments : Tsub/arg = 3.652 ; T(0.01) = 2.600). On the contrary, there is no significant
difference between the three other functions ( Student mean test : Tsub/adj = 0.911 ; Tsub/scpl =
1.059 ; T(0.1) = 1.652).
Table 4 — distribution of the word order variations according to the syntactic function of the
extracted element.
Corpus subjects arguments adjuncts sentence
complements
individual Mean 29.6% 12.6% 28.8% 29.0%
customers (St. Dev.) (26.0%) (15,2 %6). (24.4%) (22.4%)
travel agents Mean 34.6% 9.4% 22.5% 33.5%
(St. Dev.) (35.6%) (16,1 %) (30.2%) (28.3%)
Total Mean 30.7 % 12.0 % 27.4 % 30.0 %
(St. Dev). (29.6 %) (15.5 %) (26.5 %) (24.5 %)
This distribution seems to be coherent from a linguistic point of view. Generally speaking, written
French follows a canonical SVO (subject-verb-object) order. Since adjuncts or sentence complements
are not concerned by this ordering constraint, they are relatively free to move inside the sentence.
Likewise, subject extractions follow in most cases a SVO order since they correspond very frequently
to an anteposition (table 5). Subject extraction is consequently rather free. On the opposite, the
position of arguments is rigidly fixed by the SVO canonical order. Argument extractions are thus
unsurprisingly less frequent in our corpus.
Table 5 — distribution of the subject extractions according to their direction (mean percentage of
antepositions and postpositions).
4 see (Dudewicz & Mishra, 1988)
5 see (Dudewicz & Mishra, 1988).
6 This value of the standard deviation, which is greater than the corresponding mean value, shows simply that
these distributions do not follows a Gauss distribution.
26
Corpus subject anteposition subject postposition standard
deviation
Individual customers 82.9 % 17.1 % n.c.
Travel agents 77.3 % 22.7 % n.c.
Total 80.6 % 19.4 % 20.4 %
All things considered, most of the observed extractions preserve the canonical SVO order (table 6a).
Thus, in spite of a frequent use of extractions, spoken French infringe hardly some fundamental
ordering constraints.
The inventory of non projective structures supports clearly this observation. Discontinuous extractions
are indeed very rare in the Air France corpus (table 6b) : non projective structures, which are very
embarrassing for most of parsers or language models, do not concern therefore spoken French.
Table 6 — relative importance of the extractions that follow a canonical SVO order (6a : left) and
relative importance of projective extractions (6b : right)
Corpus % of extractions
with SVO order
% of speech turns
with SVO order
% of continuous
extractions
% of continuous
speech turns
customers 90.4 % 98.6 % 97.5 % 99.5 %
travel agents 90.2 % 99.0 % 98.4 % 99.8 %
Total 90.3 % 98.7 % 97.7 % 99.6 %
In conclusion, spoken man-machine dialogue in the ATIS domain seems to be noticeably concerned
by a weak word order variability that preserves nevertheless a SVO canonical order, whereas global
word order freedom is not really observed.
7. Functions and extracted structures
Extracted structures follow some regularities that should be usefully considered for computational
purposes. Table 7 presents for instance the distribution of word order variations according to the
extracted construction used. This distribution presents a rather high dispersion. It is however possible
to distinguish simple inversion as the most frequent extracted construction (60.6 % of the extractions).
This predominance is statistically significant (Student mean test of identical distributions : T = 4.473
; T(0.01) = 2.600). On the opposite, the difference between dislocations (24.9 %) and presentative
structures (among which cleaved sentences ; 13.2 %) is not statistically significant (Student mean test
of identical distributions : T = 1.366 ; Tinv(1.366) = 0.174).
Table 7 — distribution of the word order variations according to their structure.
Corpus simple
inversions
dislocations presentatives
(cleaved sentences)
other
constructions
individual Mean 60.8 % 24.0 % 13.8% 1.4 %
customers (St. Dev.) (27.0 %) (17.3 %) (22.7%) (8.8 %)
travel agents Mean 60.2 % 28.3 % 10.5% 1.0 %
(St. Dev.) (36.0 %) (36.1%) (20.6%) (8.1 %)
Total Mean 60.6 % 24.9 % 13.2 % 1.3 %
(St. Dev). (30.2 %) (25.5 %) (22.1 %) (8.6 %)
A detailed analysis of these distributions according to the syntactic function of the extracted element
provides further conclusions on these structural regularities. Thus, most of subjects and to a lesser
extent most of arguments extractions are linguistically marked (dislocations and cleaved structures :
table 8).
Table 8 — distribution of subjects and arguments extractions according to their structure.
27
Corpus subject extractions argument extractions
inversion dislocation + presentative inversion dislocation + presentative
customers 4.2 % 95.8 % 30.5 % 69.5 %
travel agents 6.1 % 93.9 % 44.4 % 55.6 %
Total 4.6 % 95.4 % 32.7 % 67.3 %
This predominance of marked extractions for the argument function is statistically significant (c2 test
on a not significant predominance : CHIAF = 0.945). On the opposite, adjuncts and sentence
complements extractions corresponds almost always to a simple inversion (table 9).
Table 9 — distribution of adjuncts and arguments extractions according to their structure.
Corpus adjunct extractions sentence complement extractions
inversion dislocation +
presentative
inversion dislocation +
presentative
customers 97.1 % 2.9 % 100.0 % 0.0 %
travel agents 95.3 % 4.7 % 100.0 % 0.0 %
Total 96.8 % 3.2 % 100.0 % 0.0 %
Once again, these observations are coherent from a linguistic point of view. On the one hand (subject
or argument extraction), cleaved structures or clitics in dislocations compensate partially for an
eventual change of the canonical SVO order. On the other hand, adjuncts or sentence complements
extraction does not require such marked constructions, since their position is relatively free.
8. Influence of the familiarisation with the task
In the previous sections, we have only considered global observations on the whole Air France corpus.
Now, any significant difference between the “customers” and the “travel agents” corpora may show an
interesting influence of the familiarisation with the task on word order variations.
The distinction between these two corpora seems a priori relevant. Dialogues are indeed more direct
with travel agents, whereas negotiations and reformulatings are noticeably more frequent with
individual customers. This observation should be put together with the fact that dialogues with travel
agents are shorter than with the other ones. A Wilcoxon-Mann-Withney test7 shows that this
difference is statistically significant (Z = 3.819 ; Z(0.01) = 2.576).
Table 10 — dialogue length according to kind of user (mean number of speech turns per dialogue).
Corpus mean dialogue length Standard deviation
individual customers 54.1 33.5
travel agents 42.1 26.4
The familiarisation with the task has therefore a noticeable influence on the dialogue structure. Does
this influence concern extractions too ? An exhaustive comparison of the results detailed on tables 2 to
9 suggests that word order variations are independent of this familiarisation. Student mean tests (table
11) show indeed that there is no statistically significant influence on the different features that should
characterise word order variations. This observation is obviously interesting for genericity purposes.
Table 11 — Statistical tests (Student mean test) of significance of a feature difference between the
“individual customers” and “travel agents” corpora.
Feature T T(0.1) Tinv(T)
frequency of occurrence 0.628 0.532
7 see (Dudewicz & Mishra, 1988)
28
direction 0.284 0.777
kind of construction inversion 0.212 0.833
dislocation 0.943 0.348
presentative 0.715 1.660 0.476
syntactic function subject 0.503 0.616
argument 0.220 0.827
adjunct 0.213 0.832
sentence complement 0.154 0.878
projectivity 0.380 0.705
9. Conclusion : extractions and NLP for man-machine communication
Since they ignore usually most of the syntactic structure of the sentence, current spoken dialogue
systems have not been concerned so far by the problem of word order freedom. They circumvent
indeed this problem thanks to ad hoc approaches that take advantage of the very restricted nature of
the considered task. This would not be the case with richer applications or finer dialogue models. As a
result, the question of word order freedom will arise soon because of the increasing need8 of a more
detailed language modeling. Now, this corpus analysis provides several lessons on word order freedom
that are interesting from a computational point of view.
First of all, the question of discontinuity (global word order freedom), which is very embarrassing for
natural language processing, does not concern fortunately spoken French in the ATIS domain. Since
discontinuous extractions appear to be very rare, the parsing of these non projective structures does
not constitute a relevant problematic for future researches on spoken dialogue systems.
On the opposite, the processing of weak word order freedom should meet an increasing importance as
more complex applications will be considered by spoken man-machine communication. The frequent
occurrence of extracted constructions in the Air France corpus shows that this question should not be
disposed of anymore. Our inventory of several structural regularities (canonical SVO order, specific
use of each extracted construction) suggests fortunately that the robust processing of extractions is not
an impossible task.
Finally, this corpus analysis does not revealed any influence of the familiarisation with the task on
word order variations. This is an interesting result that guarantees to some (restricted) extent the
genericity of spoken dialogue systems. This study did not investigate however the independence of
word order variations from the application domain. In order to answer this important question, we are
currently analysing two additional corpus whose application domain is tourism information. Since the
corresponding tasks are clearly less restricted than in the ATIS domain, we hope to obtain interesting
conclusions on genericity. Preliminary results suggest that word order variations are rather
independent from the application domain, but also that other factors (degree of interactivity for
instance) should affect noticeably the frequency of occurrence of spoken extractions (Antoine, 2001).
One aim of this paper was to show the benefit that spoken man-machine communication should obtain
from a rigorous analysis of representative corpora. Besides the question of word order variations, we
hope that this paper came up ¾ at least partially ¾ to this expectation.
Acknowledgements
The authors are thankful to Agnès Hamon and Valérie Monbet (SABRES Laboratory of Applied
Statistics, Vannes, France). They were a great assistance for the achievement of the statistical tests of
significance presented in this paper.
8 For instance, see (Chelba and Jelinek, 2000) for an illustration of the need of strutured language models in
speech recognition
29
References
Antoine J-Y Goulian J 2001 Linguistique de corpus et ingénierie des langues appliquée à la CHM
orale : étude des phénomènes d'extraction en français parlé sur deux corpus de dialogue oral finalisé,
TAL, Hermès Paris France (submitted).
Antoine J-Y Siroux J Caelen J Villaneau J Goulian J Ahafhaf M 2000 Obtaining predictive results
with an objective evaluation of spoken dialogue systems : experiments with the DCR assessment
paradigm, In proceedings of the 2nd conference on Language Ressources and Evalaution,
LREC'2000, Athens, Greece.
Blanche-Benveniste C Bilger M Rouget C and van den Eynde K 1990 Le français parlé : études
grammaticales, CNRS Editions Paris France.
Blanche-Benveniste C 1997 Approches de la langue parlée en français, Orphys Paris France.
Chelba C Jelinek F 2000 Structured language modeling, Computer Speech and Language 14(4) pp.
283-332, Academic Press London UK.
Covington M 1990 A dependency parser for variable-order languages, research Report AI-1990-01,
University of Georgia, USA.
Dudewicz E J Mishra S N 1988 Modern mathematical statistics, Wiley series in probability and
mathematical statistics, John Wiley & Sons New-York, USA.
Gadet F 1992 Le français populaire, PUF Paris France.
Hirschman L 1998 Language understanding evaluations : lessons learned from MUC and ATIS. In
proceedings of the 1st conference on Language Ressources and Evalaution, LREC'98, Granada
Spain, pp 117-122.
Holan T, Kubon, Oliva K, Plátek M 2000 On complexity of word order, TAL 41(1) pp. 273-300,
Hermès Paris.
Hudson R 2000 Discontinuity, TAL 41(1) pp. 15-56, Hermès Paris.
Mel'cuk 1988 Dependency syntax : theory and practice, State University of New York Press, Albany,
USA.
Minker W, Waibel A, Mariani J 1999 Stochastically based semantic analysis, Kluwer ac. Amsterdam
the Netherlands.
Morel M-A 1989 Analyse linguistique de corpus, Publications de la Sorbonne Nouvelle Paris France.
Pollard C, Sag I 1994 Head-driven Phrase Structure Grammar, University of Chicago Press, Chicago,
USA.
Rambow O, Joshi A 1994 A formal look at dependency grammars and phrase-structure grammars
with special considerations of word-order phenomena, In Wanner L. (ed.) Current issues in Meaning-
Text Theory, Pinter London UK.
Tesnière L 1959 Elements de syntaxe structurale, Klincksiek Paris France.

30
Sociopragmatic annotation:
New directions and possibilities in Historical Corpus Linguistics
Dawn Archer and Jonathan Culpeper
Department of Linguistics and Modern English Language, Lancaster University
{d.archer, j.culpeper}@lancaster.ac.uk
The bias of computer searches towards form (e.g. a letter or string of letters) can be a major difficulty
for linguistic analyses of texts. In particular, how form relates to context in interactive texts tends to be
overlooked. Corpus linguists have been seeking to improve our understanding of the relationships
between forms (e.g. collocations, lexical networks, keywords); in other words, they focus on text and
co-text. However, little work has considered context in terms of situational, sociological and cultural
phenomena. Consequently, our aim in this paper is to demonstrate how a sophisticated annotation
scheme can help bridge the gap between text and contexts.
Our work is located in the area of historical corpus linguistics, an area where there are particular
difficulties relating to the retrieval of contextual factors (e.g. the possibility of asking speakers is not
available!) and where little work has been done on annotation of any kind. Specifically, we will
introduce an annotated sub-section of the Corpus of English Dialogues (CED), constructed by Jonathan
Culpeper and Merja Kytö (Uppsala University). The Sociopragmatic Corpus, as it is called, covers a
120 year time span (1640-1760), and consists of more than 240,000 words of “authentic” and
constructed speech drawn from two text-types: trial proceedings and drama. We will then describe how
the annotation system that we have developed:
(i) Accommodates the investigation of language set in various context(s) (for example,
speaker/hearer relationships, social roles, and sociological characteristics such as gender), and
(ii) Treats contexts as dynamic (cf. other annotation systems, such as the spoken sub-section of
the BNC, which concentrate upon the relatively static characteristics of speakers).
More generally, this paper shows how computational linguistics can be used in pragmatics research.

31
A corpus for interstellar communication
Eric Atwell eric@comp.leeds.ac.uk http://www.comp.leeds.ac.uk/eric/
John Elliott jre@comp.leeds.ac.uk http://www.comp.leeds.ac.uk/jre/
Centre for Computer Analysis of Language And Speech,
School of Computing,
University of Leeds,
Leeds, Yorkshire, LS2 9JT England.
1. Introduction: SETI, the Search for Extra-Terrestrial Intelligence
Many researchers in Astronomy and Astronautics believe the Search for Extra-Terrestrial Intelligence
is a serious academic enterprise, worthy of scholarly research and publication (e.g. Burke-Ward 2000,
Couper and Henbest 1998, Day 1998, McDonough 1987, Sivier 2000, Norris 1999), and large-scale
research sponsorship attracted by the SETI Institute in California. Most of this research community is
focussed on techniques for detection of possible incoming signals from extra-terrestrial intelligent
sources (e.g. Turnbull et al 1999), and algorithms for analysis of these signals to identify intelligent
language-like characteristics (e.g. Elliott and Atwell 1999, 2000).
However, recently debate has turned to the nature of our response, should a signal arrive and be
detected. For example, the 50th International Astronautical Congress devoted a full afternoon session
to the question of whether and how we should respond to an initial message identified to be of extraterrestrial
origin. Interestingly, we (the authors of this paper) were the only corpus linguists present at
this session: the Congress seemed to assume that the design of potentially the most significant
communicative act in history should be decided by astrophysicists. We believe that others should be
aware of and contribute to what is effectively a corpus design project; and that the Corpus Linguistics
research community has a particularly significant contribution to make.
2. Past ideas on how to signal our existence to extra-terrestrials
Speculations about how to signal our existence to extraterrestrials began at least a century ago. Early
ideas focussed on pictorial messages, transmitted visually by drawing over very large expanses of the
Earth's surface. “For example, the Pythagorean theorem could be illustrated visually during the
daytime by clearing vast expanses of forest in Siberia to show the areas surrounding a right-angled
triangle. Or during the night, canals dug into the Sahara desert in the shape of a circle could be filled
with kerosene; when lit, the flames would provide a pictorial signal of our existence.” (Vakoch 1998a).
More recently, the Pioneer and Voyager spacecraft, sent to explore planets in our solar system but then
left to drift out into interstellar space, carried messages to any extraterrestrials who might intercept
them in their travels beyond the solar system. On the Pioneer plaque, an outline of the Pioneer
spacecraft is seen behind figures of two humans. At the bottom of the plaque, the same spacecraft is
shown in a smaller scale as it passes through the solar system on its journey from Earth. A diagram of
fifteen converging lines shows the Earth's location in time and space in relation to prominent pulsars.
(Sagan et al 1972, Vakoch 1998a). The Voyager spacecraft each bear similar diagrams, and in addition
a record (with player and encoded instructions on how to play) illustrating basics of human knowledge
of mathematics and physics, and a wide variety of pictures of our world. (Sagan 1978, Vakoch 1998a).
There have also been attempts to deliberately transmit messages from the Earth's surface. Most
notably, in 1974 astronomers at the Arecibo radio-telescope in Puerto Rico sent a signal of 1,679 radiowave
pulses to M13, a star-cluster 25,000 light-years away. 1679 is the product of two prime numbers,
23 and 73; arranging the pulses into a rectangle of 23 columns by 73 rows creates a pictogram showing
a radio-dish, a human, and some basic scientific information. (Couper and Henbest 1998, Vakoch
1998a).
32
3. Current SETI ideas on message construction
The Arecibo experiment was a deliberate attempt at message transmission. Humanity has been
transmitting radio signals on a much larger scale for decades, since radio transmissions intended for
terrestrial reception are also beamed into outer space; thus an extra-terrestrial first encounter with
human culture may well be through accidental reception of television and radio broadcasts, as foreseen
in the novel and subsequent film Contact (Sagan 1988). Reception of such “unintended” messages
may prompt Extra-Terrestrials to initiate first contact; but many in the SETI research community (e.g.
Vakoch 1999) feel it is important to plan a more deliberately designed, well-thought-out response
message.
(Vakoch 1998b) argues for “... the need for more intensive investigations of the linguistic aspects of
SETI before a message is received”. (Vakoch 1998c, p705) also identifies several benefits of beginning
work on construction of a reply message immediately, even before an incoming extraterrestrial
message has been received and recognised:
“(1) concretely understanding the challenge of creating an adequate reply; (2) helping decode messages from
extraterrestrials; (3) creating interstellar compositions as a new form of art; (4) having a reply ready in case we
receive a message; (5) providing a sense of concrete accomplishment; (6) preparing for an active search strategy; and
(7) gaining public support for SETI.”
In 1974 a signal of 1,679 bits was considered potentially significant and challenging to technology of
the time, e.g. it took three minutes to transmit; a quarter of a century later, we are used to processing
messages of megabytes, gigabytes, or bigger in terrestrial communication networks such as the
Internet. It is clear that we could look beyond a single pictogram or collection of diagrams, to design a
much larger Corpus of data to represent humanity. (Vakoch 1998c) advocates that the message
constructed to transmit to extraterrestrials should include a broad, representative collection of
perspectives rather than a single viewpoint or genre; this should strike a chord with Corpus Linguists
for whom a central principle is that a corpus must be “balanced” to be representative.
The consensus at the 50th International Astronautical Congress seemed to be to transmit an
encyclopaedia summarising human knowledge, such as the Encyclopaedia Britannica, to give ET
communicators an overview and “training set” key to analysis of subsequent messages. Furthermore,
this should be sent in several versions in parallel: the text; page-images, to include illustrations left out
of the text-file; and perhaps some sort of abstract linguistic representation of the text, using a functional
or logic language (Ollongren 1999, Freudenthal 1960).
4. Enriching the message corpus with multi-level linguistic annotations
The idea of “enriching” the message corpus with annotations at several levels should also strike a chord
with Corpus Linguists. Natural language exhibits highly complex multi-layered sequencing, structural
and functional patterns, as difficult to model as sequences and structures found in more traditional
physical and biological sciences. Corpus Linguists have long known this, on the basis of evidence such
as the following:
·  Language datastreams exhibit structural patterns at several interdependent linguistics levels,
including: phonetic and graphemic transcription, prosodic markup, part-of-speech
wordclasses, collocations, phraseological and collegational patterns, semantic word-sense
classification, syntax or grammatical phrase structure, functional dependency structure,
semantic predicate structure, pragmatic references, discourse or dialogue structure,
communication act or speech act patterns.
·  Even within one such linguistic level, structural analysis is complex, with further
interdependent sublevels. For example, the European Expert Advisory Group on Language
Engineering Standards (EAGLES) report on parsing annotations (Leech et al 1996) recognises
at least 7 separate yet interdependent sublayers of grammatical analysis which a full parser
should aim to recognise; yet none of the state-of-the-art parsers evaluated in (Atwell 1996,
Atwell et al 2000a) were capable of providing all 7 layers of analysis in their output. Different
parsers analysed different subsets of these sublayers of grammatical information, making
cross-parser comparisons and performance evaluations difficult if not meaningless.
33
·  Furthermore, linguistic analysis at one level may depend on or require other levels of
linguistic information; for example, (Demetriou and Atwell 2001) demonstrated that lexicalsemantic
word-tagging subsumes or combines several knowledge sources including thesaurus
class, semantic field, collocation preferences, and dictionary definition.
·  Some corpora have been annotated with several layers or levels of linguistic knowledge in
parallel; for example, the SEC corpus (Taylor and Knowles 1988) has speech recordings,
transcriptions, prosody markup, PoS-tags, parse-trees; the ISLE corpus (Menzel et al 2000,
Herron et al 1999, Atwell et al 2000b) has language-learner speech recordings, transcriptions,
corrections, prosody, expert evaluations. Other annotations can be added automatically by
software, e.g. semantic tags (Demetriou and Atwell 2001), ENGCG Constraint Grammar
dependency structures (Karlsson et al 1995, Voutilainen et al 1996).
5. Natural language learning
In the 1980s, most NLP researchers used their ‘expert intuitions’ to guide development of large-scale
grammars; a language model was essentially an `expert system’ encoding the knowledge of a human
linguistics expert. This kind of knowledge model was harder to `scale up’ to cover more and
more language data, and it relied on existing expert knowledge. More recently, this has given way to
the use of corpora or large text samples, some of which are annotated or `tagged’ with expert analyses.
Tagged and parsed corpora can be used by linguists as a testbed to guide their development of
grammars (see, for example Souter and Atwell 1994); and they can be used to train Natural Language
Learning or data-mining models of complex sequence data. Several initiatives are under way to collect
language datasets for language modelling research, for example, ICAME, the International
Computer Archive of Modern and medieval English (based in Bergen); ELRA, the European
Language Resources Association (based in Paris); LDC, the Linguistic Data Consortium (based at the
University of Pennsylvania).
A growing number of NLP researchers are looking into ways to utilise these new training-set
resources: the Association for Computational Linguistics has established a Special Interest Group
in Natural Language Learning (machine-learning of language sequence-patterns from corpus data)
which holds annual conferences, e.g. CoNLL'2000. Given appropriate annotated Corpus data, many
NLP problems can be generalised to “mappings” between linguistic levels of analysis, for example:
·  Word-class identification (mapping words into syntactic/semantic sets or classes), e.g. (Atwell
and Drakos 1987, Hughes 1993, Finch 1993, Hughes and Atwell 1994, Teahan 1998)
·  Part-of-Speech wordtagging (mapping word-sequences onto wordclass-tag sequences), e.g.
(Leech et al 1983, Atwell 1983, Eeg-Olofsson 1991, Brill 1993, Atwell et al 1984, 2000a);
·  Sentence-structure analysis or parsing (mapping word- and/or word-class sequences onto
parses), e.g. (Sampson et al 1989, Atwell 1987, 1988, 1993, Black et al 1993, Bod 1993,
Briscoe 1994, Jelinek et al 1992, Joshi and Srinivas 1994, Magerman 1994, O'Donoghue
1993, Schabes, Roth and Osborne 1993, Sekine and Grishman 1995)
·  Semantic analysis or word-sense tagging (mapping word-sequences onto semantic tags or
meaning-analyses), e.g. (Demetriou 1993, Demetriou and Atwell 1994, 2001, Bod et al 1996,
Kuhn and de Mori 1994, Weischedel et al 1993, Wilson and Rayson 1993, Wilson and Leech
1993, Jost and Atwell 1993)
·  Machine Translation (mapping a source-language word sequence onto a target-language wordsequence),
e.g. (Brown et al 1990, Berger et al 1994, Gale and Church 1993)
·  Speech-to-text recognition (mapping a speech signal onto a phonetic and graphemic
transcription word-sequence), e.g. (Demetriou and Atwell 1994, Giachin 1995, Jelinek 1991,
Kneser and Ney 1995, Yamron 1994, Young and Bloothooft 1997).
34
Researchers have tried casting these NLP mapping subtasks in terms of Natural Language Learning
models, such as Hidden Markov Models (HMMs), Stochastic Context Free Grammar (SCFG) parsers,
Data-Oriented Parsing (DOP) models. The complex patterns found in language data call for
sophisticated stochastic modelling. For example, Hidden Markov Models have become widely used in
Language Engineering applications because they are well-understood and computationally tractable
(e.g. Young and Bloothooft 1997, Manning and Schutze 1999, Jurafsky and Martin 2000, Huang 1990,
MacDonald 1997, Elliott et al 1995, Woodward 1997). Although (Chomsky 1957) famously
demonstrated that a finite-state model is a theoretically inadequate approximation for certain aspects of
language modelling, Language Engineers have come to realise that HMMs can be adapted to work
most of the time, and that the theoretically problematic cases alluded to by Chomsky are infrequent
enough in “real” applications to be ignored in practice. Language Engineering researchers have been
searching for higher-level models which effectively extend Hidden Markov Models in limited ways
without extending the computational cost prohibitively, for example higher-order Markov models,
limited stochastic context-free grammars, hybrid statistical/knowledge-based models. Linguists have
found ‘Universal’ features which appear to be common to and characteristic of all human languages,
(e.g. Zipf 1935, 1949); but few of these have been stated in terms of or related to stochastic models.
We know how to extract low-level linguistic patterns from raw text using unsupervised learning
algorithms (e.g. Atwell and Drakos 1987, Hughes 1993, Finch 1993, Hughes and Atwell 1994, Elliott
and Atwell 1999, 2000, Elliott et al 2000a,b, 2001, Manning and Schutze 1999, Jurafsky and Martin
2000); a “Rosetta Stone” key to English, annotated with rich linguistic analyses, should help ET
communicators map between symbols and meanings using supervised as well as unsupervised learning
algorithms.
6. A corpus linguistics SETI advisory panel
Astronomers have not sought to consult Corpus Linguists on the design of this Corpus for Interstellar
Communication; but we can and should make an informed contribution. The parallel corpus and multiannotated
corpus are not new concepts to Corpus Linguistics. We have a range of standards and tools
for design and annotation of representative corpus resources. Furthermore, we know which analysis
schemes are more amenable to supervised learning algorithms; for example, the BNC tagging scheme
and the ICE-GB parsing scheme have been demonstrated to be machine-learnable in a tagger and
parser respectively.
An Advisory Panel of corpus linguists could design and implement an extended Multi-annotated
Interstellar Corpus of English. The following are ideas for the Advisory Panel to consider:
·  augment the Encyclopaedia Britannica with a collection of samples representing the diversity
of language in real use. Candidates include the LOB and/or BNC corpus;
·  as an additional “key”, transmit a dictionary aimed at language learners which has also been a
rich source for NLP learning (e.g. Demetriou and Atwell 2001); a good candidate would be
LDOCE, the Longman Dictionary of Contemporary English, which uses the Longman
Defining Vocabulary;
·  supply our ET communicators with several levels of linguistic annotation, to give them a
richer training set for their natural language learning attempts. We suggest that initial (i) raw
text and (ii) page-images should be augmented with some or all of (iii) XML markup, (iv)
PoS-tagging, (v) phrase structure parses, (vi) dependency structure analyses, (vii) coreference
markup, (viii) dialogue act markup, (ix) semantic analyses.
·  Add translations of the English text into other human languages; although the International
Astronautical Congress seemed to assume Humanity should be represented by English,
multilingual annotations may actually be useful in natural language learning algorithms.
This calls for a large-scale corpus annotation project, which may not seem immediately justifiable to
computational linguistics research funding bodies such as the UK Engineering and Physical Sciences
Research Council (EPSRC). However, the International Astronautical Congress also discussed plans to
35
proactively make interstellar contact using existing astronautical technology, by firing a satellite-based
laser cannon at a range of nearby (in astronomical terms) potentially suitable targets. If this succeeds
and we receive a message back, the need for our Interstellar Corpus Advisory Panel becomes more
urgent.
Of course, this Interstellar Corpus Advisory Panel should be chaired by an acknowledged expert in
English grammar and semantics (eg Quirk et al 1972, 1985, Wilson and Leech 1993, Leech 1969,
1971, 1974, 1983, 1994), English language learning (e.g. Leech 1986, 1994, Quirk et al 1972, 1985 ),
and corpus design, implementation, annotation, standardisation, and analysis (e.g. Leech et al 1983,
1996, Atwell et al 1984, Garside et al 1987, Black et al 1993, Leech 1991, 1992, 1993a,b): Professor
Geoffrey Leech.
7. References
Atwell E 1983 Constituent likelihood grammar. ICAME Journal 7: 34-65.
Atwell E 1987 A parsing expert system which learns from corpus analysis. In Meijs W (ed) Corpus
Linguistics and Beyond: Proceedings of the ICAME 7th International Conference on English Language
Research on Computerised Corpora, Amsterdam, Rodopi, pp227-235.
Atwell E 1988 Transforming a parsed corpus into a corpus parser. In Kyto M, Ihalainen O, Risanen M
(eds) Corpus Linguistics, Hard and Soft: Proceedings of the ICAME 8th International Conference on
English Language Research on Computerised Corpora. Amsterdam, Rodopi, pp61-70.
Atwell E 1993 Corpus-based statistical modelling of English grammar. In Souter C, Atwell E (eds),
Corpus-based computational linguistics: proceedings of the 12th conference of the International
Computer Archive of Modern English. Amsterdam, Rodopi, pp195-214.
Atwell E 1993 Linguistic constraints for large-vocabulary speech recognition. In Atwell E (ed),
Knowledge at Work in Universities: Proceedings of the second annual conference of the Higher
Education unding Council's Knowledge Based Systems Initiative. Leeds University Press, pp26-32.
Atwell E 1996 Machine learning from corpus resources for speech and handwriting recognition. In
Thomas J, Short M (eds), Using corpora for language research: studies in the honour of Geoffrey
Leech, Harlow, Longman, pp151-166.
Atwell E 1996 Comparative evaluation of grammatical annotation models. In Sutcliffe R, Koch H,
McElligott A (eds), Industrial Parsing of Software Manuals. Amsterdam, Rodopi.
Atwell E, Leech G, Garside R 1984 Analysis of the LOB corpus: progress and prospects. In Aarts J,
Meijs W (eds) Corpus Linguistics: Proceedings of the ICAME 4th International Conference on the Use
of Computer Corpora in English Language Research, Amsterdam, Rodopi, pp40-52.
Atwell E, Demetriou G, Hughes J, Schiffrin A, Souter C, Wilcock S 2000 A comparative evaluation of
modern English corpus grammatical annotation schemes. ICAME Journal 24: 7-23.
Atwell E, Howarth P, Souter C, Baldo P, Bisiani R, Bonaventura P, Herron D, Menzel W, Morton R,
Wick H 2000 User-guided system development in interactive spoken language education. Natural
Language Engineering, 6(3): 188-202.
Atwell E, Drakos N 1987 Pattern recognition applied to the acquisition of a grammatical classification
system from unrestricted English text. In Maegaard B (ed), Proceedings of the Third Conference of
European Chapter of the Association for Computational Linguistics. New Jersey, Association for
Computational Linguistics, pp56-63.
Atwell E, Hughes J, Souter C 1995 Automatic extraction of tagset mappings from parallel-annotated
corpora. In Tzoukermann E, Armstrong S (eds) From text to tags - issues in multilingual language
analysis: Proceedings of EACL-SIGDAT. Dublin, Association for Computational Linguistics, pp10-17.
36
Berger A, Brown P, Cocke J, Pietra S, Pietra V, Gillett J, Lafferty J, Mercer R, Printz H, Ures L 1994
The candide system for machine translation. In Proceedings of the ARPA workshop on Human
Language Technology. San Mateo, Morgan Kaufmann, pp152-157.
Black E, Garside R, Leech G (eds) 1993 Statistically-driven computer grammars of English: the
IBM/Lancaster approach. Amsterdam, Rodopi.
Bod R 1993 Using an annotated corpus as a stochastic grammar. In Proceedings of the 6th EACL.
Utrecht, Association for Computational Linguistics.
Bod R, Bonnema R, Scha R 1996 A data-oriented approach to semantic interpretation. In Proceedings
of the ECAI'96 workshop on corpus-oriented semantic analysis. Budapest, ECAI.
Brill E 1993 A corpus-based approach to language learning. PhD thesis, University of Pennsylvania.
Briscoe E 1994 Prospects for practical parsing of unrestricted text: robust statistical parsing techniques.
In Oostdijk N, de Haan P (eds) Corpus-based research into language. Amsterdam, Rodopi.
Brown P, Cocke J, Pietra S, Pietra V, Jelinek F, Lafferty J, Mercer R, Roossin P 1990 A statistical
approach to machine translation. Computational Linguistics 16(2): 79-85.
Burke-Ward R 2000 Possible existence of extra-terrestrial technology in the solar system. . Journal of
the British Interplanetary Society 53(1&2): 1-12.
Chomsky N 1957 Syntactic structures. The Hague, Mouton.
Couper H, Henbest N 1998 Is anybody out there? London, Dorling Kindersley.
Day P (ed) 1998 The search for extraterrestrial life. Oxford, Oxford University Press and the Royal
Institution.
Demetriou G 1993 Lexical disambiguation using CHIP. In Proceedings of the 6th EACL. Utrecht,
Association for Computational Linguistics, pp431-436.
Demetriou G, Atwell E 1994 Machine-learnable, non-compositional semantics for domain independent
speech or text recognition. In Proceedings of the 2nd Hellenic-European Conference on Mathematics
and Informatics (HERMIS), Athens.
Eeg-Olofsson M 1991 Word-class tagging: some computational tools. PhD thesis, University of Lund.
Elliott R, Lakhdar A, Aggoun J, Moore R 1995 Hidden Markov models: estimation and control.
London, Springer-Verlag.
Elliott J, Atwell E 1999 Language in signals: the detection of generic species-independent intelligent
language features in symbolic and oral communications. In Proceedings of the 50th International
Astronautical Congress. Amsterdam, paper IAA-99-IAA.9.1.08.
Elliott J, Atwell E 2000. Is there anybody out there?: The detection of intelligent and generic languagelike
features. Journal of the British Interplanetary Society 53(1&2): 13-22.
Elliott J, Atwell E, Whyte B 2000 Language identification in unknown signals. In Proceedings of
COLING'2000 18th International Conference on Computational Linguistics, Saarbrucken, Association
for Computational Linguistics (ACL) and San Francisco, Morgan Kaufmann Publishers, pp1021-1026.
Elliott J, Atwell E, Whyte B 2000 Increasing our ignorance of language: identifying language structure
in an unknown signal. In Daelemans W (ed) Proceedings of CoNLL-2000: International Conference on
Computational Natural Language Learning. Lisbon, Association for Computational Linguistics.
37
Elliott J, Atwell E, Whyte B 2001 A toolkit for visualisation of combinational constraint phenomena in
linguistically interpreted corpora. In Proceedings of CLUK'4: Computational Linguistics in the United
Kingdom. Sheffield.
Finch S 1993 Finding structure in language. PhD thesis, Edinburgh University.
Freudenthal H 1960 LINCOS, Design of a language for cosmic intercourse. North Holland.
Gale W, Church K, 1993 A program for aligning sentences in bilingual corpora. Computational
Linguistics 19(1): 75-102.
Garside R, Leech G, Sampson G (eds) 1987 The computational analysis of English: a corpus-based
approach. London, Longman.
Giachin E 1995 Phrase bigrams for continuous speech recognition. In Proceedings of ICASSP'95.
Detroit.
Herron D, Menzel W, Atwell E, Bisiani R, Daneluzzi F, Morton R, Schmidt J 1999 Automatic
localization and diagnosis of pronunciation errors for second language learners of English. In
Proceedings of EUROSPEECH'99: 6th European Conference on Speech Communication and
Technology. Budapest.
Huang X 1990 Hidden Markov models for speech recognition. Edinburgh, Edinburgh University Press.
Hughes J 1993 Automatically acquiring a classification of words PhD thesis, University of Leeds.
Hughes J, Atwell E 1994 The automated evaluation of inferred word classifications. In Cohn A (ed)
Proceedings of ECAI'94: 11th European Conference on Artificial Intelligence. Chichester, John Wiley,
pp535-539.
Jelinek F 1991 Self-organised language modeling for speech recognition. In Waibel A, Lee K (eds),
Readings in speech recognition. San Mateo, Morgan Kaufmann, pp450-506.
Jelinek F, Lafferty J, Mercer R 1992 Basic methods of probabilistic context-free grammars. In Laface
P, de Mori R (eds) Speech recognition and understanding. Berlin, Springer-Verlag, pp347-360.
Joshi A, Srinivas B 1994 Disambiguation of Super Parts of Speech (or Supertags): almost parsing. In
Proceedings of COLING'94. Kyoto.
Jost U, Atwell E 1993 Deriving a probabilistic grammar of semantic markers from unrestricted English
text. In Lucas S (ed) Grammatical Inference: theory, applications, and alternatives, IEE Colloquium
Proceedings 1993/092. London, Institute of Electrical Engineers, pp91-97.
Jurafsky D, Martin J 2000 Speech and language processing. Prentice-Hall
Karlsson F, Voutilainen A, Heikkila J, Anttila A (eds) 1995 Constraint grammar. Berlin, Mouton de
Gruyter.
Kneser R, Ney H 1995 Improved backing-off for n-gram language modelling. In Proceedings of IEEE
ICASP'95. Detroit, pp49-52.
Kuhn R, de Mori R 1994 Recent results in automatic learning rules for semantic interpretation. In
Proceedings of the International Conference on Spoken Language Processing. Yokohama, pp75-78.
Leech G 1969 Towards a semantic description of English. London, Longman.
Leech G 1971 Meaning and the English verb. London, Longman.
Leech G 1974 Semantics. Harmondsworth, Penguin.
38
Leech G 1983 Principles of pragmatics. London, Longman.
Leech G 1986 Automatic grammatical analysis and its educational applications. In Leech G, Candlin C
(eds) Computers in English language teaching and research: selected papers from the British Council
Symposium. London, Longman, pp204-214.
Leech G 1991 The state of the art in corpus linguistics. In Aijmer K, Altenberg B (eds) English corpus
linguistics: essays in honour of Jan Svartvik. London, Longman, pp8-29.
Leech G 1992 100 million word of English: the British National Corpus (BNC). Language Research
28(1): 1-13
Leech G 1993 100 million words of English. English Today 33: 9-15
Leech G 1993 Corpus annotation schemes. Literary and Linguistic Computing. 8(4): 275-281.
Leech G 1994 Students’ grammar – teachers’ grammar – learners’ grammar. In Bygate M, Tonkyn A,
Williams E (eds) Grammar and the language teacher. Prentice Hall International.
Leech G, Garside R, Atwell E 1983 The automatic grammatical tagging of the LOB corpus. ICAME
Journal 7: 13-33.
Leech G, Barnett R, Kahrel P 1996 EAGLES Final Report and guidelines for the syntactic annotation
of corpora, EAGLES Report EAG-TCWG-SASG/1.5. http://www.ilc.pi.cnr.it/EAGLES96/home.html
MacDonald I 1997 Hidden Markov and other models for discrete-valued time series. London,
Chapman and Hall.
Magerman D 1994 Natural language parsing as statistical pattern recognition. PhD thesis, Stanford
University.
Manning C, Schutze H 1999 Foundations of statistical natural language processing. Cambridge, MIT
Press.
McDonough T 1987 The search for extra-terrestrial Intelligence. John Wiley and Sons.
Menzel W, Atwell E, Bonaventura P, Herron D, Howarth P, Morton R, Souter C 2000 The ISLE
corpus of non-native spoken English. in Gavrilidou M, Carayannis G, Markantionatou S, Piperidis S,
Stainhaouer G (eds) Proceedings of LREC2000: Second International Conference on Language
Resources and Evaluation. Athens, European Language Resources Association (ELRA), vol.2 pp.957-
964.
Norris R 1999 How old is ET? In Proceedings of 50th International Astronautical Congress.
Amsterdam, paper IAA-99-IAA.9.1.04.
O'Donoghue T 1993 Reversing the process of generation in systemic grammar. PhD thesis, University
of Leeds.
Ollongren A 1999 Large-size message construction for ETI. In Proceedings of the 50th International
Astronautical Congress. Amsterdam, paper IAA-99-IAA.9.1.06.
Quirk R, Greenbaum S, Leech G, Svartvik J 1972 A grammar of contemporary English. London,
Longman.
Quirk R, Greenbaum S, Leech G, Svartvik J 1985 A comprehensive grammar of the English language.
London, Longman.
Sagan C (ed) 1978 Murmers of earth: the voyager interstellar record. New York, Random House.
Sagan C 1988 Contact. London, Legend.
39
Sampson G, Haigh R, Atwell E 1989 Natural language analysis by stochastic optimisation. Journal of
Experimental and Theoretical Artificial Intelligence 1: 271-287.
Schabes Y, Roth M, Osborne R 1993 Parsing of the Wall Street Journal with the inside-outside
algorithm. In In Proceedings of the 6th EACL. Utrecht, Association for Computational Linguistics,
pp341-46.
Sekine S, Grishman R 1995 A corpus-based probabilistic grammar with only two non-terminals. In
Proceedings of IWPT International Workshop on Parsing Technologies. Prague University.
Sivier D 2000 SETI and the historian: methodological problems in an interdisciplinary approach. .
Journal of the British Interplanetary Society 53(1&2): 23-26.
Souter C, Churcher G, Hayes J, Hughes J, Johnson S 1994 Natural language identification using
corpus-based models. HERMES Journal of Linguistics. 13: 183-204.
Souter C, Atwell E 1994 Using parsed corpora: a review of current practice. In Oostdijk N, de Haan P
(eds) Corpus-based research into language. Amsterdam, Rodopi, pp143-158.
Taylor L, Knowles G 1988 Manual of information to accompany the SEC corpus: The machine
readable corpus of spoken English. University of Lancaster: Unit for Computer Research on the
English Language. Available from http://kht.hit.uib.no/icame/manuals/sec/INDEX.HTM
Teahan B 1998. Modelling English text. PhD Thesis, University of Waikato, New Zealand.
Voutilainen A, Jarvinen T 1996 Using the English constraint grammar parser to analyse a software
manual corpus. In Sutcliffe R, Richard, Koch H, McElligott A (eds.) Industrial parsing of software
manuals. Amsterdam, Rodopi, pp57-88.
Turnbull M, Smith L, Tarter J 1999 Project Phoenix: Starlist2000. In Proceedings of the 50th
International Astronautical Congress. Amsterdam, paper IAA-99-IAA.9.1.02.
Vakoch D 1998a Pictorial messages to extraterrestrials, SETIQuest 4(1): 8-10 (Part 1), 4(2): 15-17
(Part 2).
Vakoch D 1998b Constructing messages to extraterrestrials: an exosemiotic perspective. Acta
Astronautica 42(10-12): 697-704.
Vakoch D 1998c The dialogic model: representing human diversity in messages to extraterrestrials.
Acta Astronautica 42(10-12): 705-710.
Vakoch D 1999 Communicating scientifically formulated spiritual principles in interstellar messages.
In Proceedings of the 50th International Astronautical Congress. Amsterdam, paper IAA-99-
IAA.9.1.10.
Weischedel R, Meteer M, Schwarz R, Ramshaw L, Palmucci J 1993 Coping with ambiguity and
unknown words through probabilistic models. Computational Linguistics 19(2): 359-382.
Wilson A, Rayson P 1993 The automatic content analysis of spoken discourse. In Souter C, Atwell E
(eds), Corpus based computational linguistics. Amsterdam, Rodopi, pp215-226.
Wilson A, Leech G 1993 Automatic content analysis and the stylistic analysis of prose literature. Revue
Informatique et Statistique dans les Sciences Humaines 29: 219-234.
Young S, Bloothooft G (eds) 1997 Corpus-based methods in language and speech processing.
Dordrecht/Boston, Kluwer Academic Publishers.
Zipf G 1935 The psycho-biology of language. Boston, Houghton-Mifflin
Zipf G 1949 Human behaviour and the principle of least effort. New York, Addison Wesley.

40
From EAGLES to CT tagging: a case for re-usability of resources
Manuel Barbera
SSLMIT Trieste
b.manuel@inrete.it
1. Abstract
Re-usability has been recently identified as one of the main requisites a corpus annotation project
must accomplish (cf. Garside, Leech and McEnery 1997: 5; Leech - Wilson 1999, etc.). This subject,
developed mainly to favour morphosyntactic tagging of corpora, naturally holds more general
implications for corpus linguistics as a whole, and the need for resources to «be reusable,
interchangeable, shareable» (Monachini and Calzolari 1999: 149) is now strongly agreed upon even at
an institutional level. It is not by chance that international initiatives in this sense have multiplied in the
last few years (cf. Monachini and Calzolari 1999: 149-150). In fact, beside the obvious economic and
practical reason, there is also a more theoretic attitude toward a more “ecological” and “democratic”
conception of linguistic computer sciences, where resources can be shared by fields of diverse nature.
Corpus Taurinense (CT) annotation in connection with EAGLES standard guidelines is, I believe,
an excellent example of this approach from many points of view. Not only, in fact, was the CT-tagset
conceived according to standards that would allow the CT to be (re)used for extremely different
purposes (cf. § 4), but its very conception is an example of how experiences previously accumulated in
rather diverse fields may be re-used (cf. § 3). The CT, in fact, is a POS tagged corpus of old Italian
formulated for prevalently linguistic-philological aims. EAGLES standards, instead, have been
introduced for eminently practical (commercial, economic, etc.) purposes. The fact that technologies
designed for society can become valuable to humanistic research, in a sort of cycle and recycle of
intercommunication between the two sectors, is a rather new, fortunate situation.
The present paper will present a short documentation of an example of this lucky match.
2. The Corpus Taurinense (CT).
Before fully entering into the subject, it is worthwhile dedicating a few introductory notes to the
Corpus Taurinense; while it is certainly not necessary in this paper to spend time on the EAGLES
guidelines, so familiar to all.
The Corpus Taurinense1, as I have already mentioned, is a tagged corpus of old Italian texts (more
specifically, old Florentine dated between 1251 and 1300) of 258,310 tokens (for 19,235 forms) which
has been developed by Carla Marello and me2. The choice of texts was not our responsibility, however,
since the CT is truly the annotated reincarnation, improved in the tokenization, of the Padua Corpus,
which is a subset of the collection of texts of the TLIO (Tesoro della lingua italiana delle origini) under
costruction at the OVI (Opera del vocabolario Italiano)3. This collection was made available by Pietro
Beltrami and chosen by Lorenzo Renzi and Giampaolo Salvi (cf. Renzi 1998: 29) as the base for the
compilation of “ItalAnt – Grammatica dell'italiano antico”, a syntax of Old Italian which is considered
an ideal prosecution of the “Grande grammatica italiana di consultazione” (Renzi - Salvi 1988, 1991,
1995).
The linguistic annotation which we have implemented is a morphosyntactic tagging, additionally
enriched by lemmatic annotation. At present, we are working on the disambiguation of the
transcategorizations and on the treatment of multiword entries (Barbera and Marello 2000). In the
future we hope to be able to add at least a third level of annotations of textual nature.
The CT is available for UNIX/LINUX on the Corpus Work Bench (CWB) system , built by IMS
Stuttgart (cf. Christ and Schulze 1996). A demo of an Internet query interface is already online and
1 Its name, analogously to the Padua Corpus, which will be discussed shortly, is taken from the place where the co-financed
group is located, namely Torino (Turin, Italy, in Latin Augusta Taurinorum). We could not have simply called it “Corpus di
Torino” because, aside from our love for Latin, there already exists a corpus of texts in English put together by students from the
University of Turin which is internationally known in studies of applied linguistics as, in fact, the “Turin Corpus”.
2 In the field of research co-financed by the CNR “Per una grammatica testuale dell'italiano antico”, directed by Bice Morara
Garavelli, and coordinated with “Ricerche linguistiche sull'italiano antico”, directed by Lorenzo Renzi.
3 Cf. OVI homepage at the URLs http://www.ovisun199.csovi.fi.cnr.it/italnet/OVI or
http://www.lib.uchicago.edu/efts/ARTFL/projects/OVI.
41
under testing at Stuttgart4. The CT, thanks to the versatility of the CQP (Corpus Query Processor) of
the IMS Corpus Work Bench, allows for the simultaneous display and query of both linguistic (lemma,
POS, morphosyntax) and philological annotations (e.g. corrections, text structure, etc.)
3. From EAGLES to the CT
In the first place, EAGLES guidelines have provided us in the planning phase of the works with
many useful insights which have helped in building an efficient tagset. Monachini and Calzolari (1996)
collects a lot of previous experience in this sector, which has been even more useful since corpus
annotation is a branch of computational linguistics that, until now, has rarely worked on “old” corpora,
so that we had few specific previous experiences5 to base our work on.
In particular, in order to create a tagset suitable for old Italian it was necessary to keep the
peculiarities of the language and of the documentation in mind as they related to the computational
automatisms and the needs of linguistic analyses. Thus, various operations have become necessary.
Often they were complex and would have had to be completely “invented” had the EAGLES guidelines
not offered us appropriate solutions. This is not the place to discuss the details of these matters. I
would, however, like to offer at least a few practical examples of this, restricting the discussion to one
specific fragment of the tagset, the “pronominal” area6.
A more general problem is the division in the tagset into Hierarchy Defining Features (HDF) and
Morphosyntactic Features (MSF). Since only HDF are constructed in a typed hierarchy in this kind of
architecture, while MSF are freely applied to typed tags, the general criteria hold that each class of
alternative features which is POS-specific be arranged in a hierarchy and that each class of alternative
features which is applied to various POS be classified as MSF. A simple comparison of EAGLES
tagsets for German (ELM-DE) and for modern Italian (ELM-IT), however, reveals how these criteria
can be freely modified and shaped in every language according to their requirements for lemmatization
and the problems of disambiguation. The ELM-DE scheme, in fact, may seem complex, yet it was the
best choice in this respect, as demonstrated by the way the IMS Stuttgart has used it.
ELM-DE
PRON personal refl poss demo idf rel interrog
sg;pl sg;pl sg;pl inflect non-inflect sg;pl sg;pl
1;2;3 1;2;3 sg;pl
du mich seines dieser mancher man die wen
+MSF +gend
+case
+case +gend
+case
+gend
+case
+gend
+case
– +gend
+case
+gend
+case
DET poss demo idf rel interrog
sg;pl sg;pl inflect non-inflect inflect non-inflect
sg;pl sg;pl
seine dieser manche manch dessen welchen wessen
+MSF +gend
+case
+gend
+case
+gend
+case
– – +gend
+case
–
Table 1.
For Italian, however, as demonstrated by the ELM-IT solutions, a simpler division was preferred.
4 Cf. the “CT-WWW-Demos des Corpus Query Processor CQP” homepage at the URL http://www.ims.unistuttgart.
de/projekte/CQPDemos/italant/. Access is reserved, but it can be freely granted by asking me (b.manuel@inrete.it) or
Carla Marello (marello@cisi.unito.it).
5 Moreover the Penn-Helsinki parsed corpus of Middle English (http://www.ling.upenn.edu/mideng/) and the Tycho Brahe
parsed corpus of Historical Portuguese (http://www.ime.usp.br/~tycho/corpus/index.html) which are perhaps the most relevant
experiences in this sector, are both treebanks, and present, therefore, problems which are often different from ours. We do,
however, know of some experiments on morphological tagging of Old Italian texts at the CiBIt (Centro interuniversitario
Biblioteca Italiana Telematica) in Pisa: http://cibit.humnet.unipi.it/index_ra.htm.
6 This area of the tagset was dealt with specifically in Parallela IX Congress (Barbera 2000); for a general standard description
of the CT-Tagset cf. Barbera 2000/2001.
42
ELM-IT
PRON pers poss dem indf int rel excl
strg weak
nom obl obl
io mi me mio quello alcuni che? che che!
+MSF +pers
+gend
+numb
+pers
+gend
+numb
+pers
+gend
+numb
+pers
+gend
+numb
+gend
+numb
+gend
+numb
+gend
+numb
+gend
+numb
+gend
+numb
DET poss dem indf int rel excl
1,2,3
mio quello alcuni che? che che!
+MSF +gend
+numb
+gend
+numb
+gend
+numb
+gend
+numb
+gend
+numb
+gend
+numb
Table 2
Having seen the EAGLES proposals, we began to consider the idea that the scheme of the tagset, in
its typed and non-typed components, could be correlated to the scheme of the lemmary, so the system
could be optimized, while considering the specific difficulties Old Italian presented. These are not only
due to the variations introduced by diverse philological practices used in the editions initially used, but
also to the fluid and not yet prescribed nature of the original texts. The result is a staggering number of
graphic and linguistic variations of all forms and the creation of multiple problems in identifying
tokens, especially in relation to the presence of pronominal clitics (and less frequently adverbial
clitics), particularly abundant in this state of language.
The aim was to render the annotation as distinct and suitable to old Italian as possible, while using
the least number of tags necessary7:
CT-Tagset
PD pers poss dem indf int rel excl
strg weak strg weak strg weak
nom obl obl
io mi me mio ÷ma quello ne alcuni che? che che!
+MSF +pers
+gend
+numb
+pers
+gend
+numb
+pers
+gend
+numb
+pers
+gend
+numb
+pers
+gend
+numb
+pers
+gend
+numb
– +pers
+gend
+pers
+gend
+pers
+gend
+pers
+gend
Table 3
In doing so, as can be seen from the results, we have also managed to remain closer than the ELMIT
to the native “naive”8 grammatical tradition. The modern perspective of re-usability of computer
data, in fact, has underlined more than once that «it is a good idea for adnotation schemes to be based
as far as possible on consensual or theory-neutral analyses of the data» (Leech 1997: 7).
From this point of view the principal novelty of the CT-Tagset, compared to ELM-IT, has been to
re-organize the pronominal area (Pronouns and Determiners) in one single POS, which we have called
“PD”, prevalently morphologically based (leaving the syntactic level to a later and different phase), and
with an internal organization studied expressly from the point of view of the tagset's structure and its
suitability to Old Italian.
It was a pleasant surprise for us, as well as a confirmation that we had worked in the right direction,
to find that Geoffrey Leech and Andrew Wilson, in the most recent Guidelines, published after the
formulation of ELM-IT, had reached conclusions similar to ours:
7 In fact, the computational advantages offered by a limited tagset are well-known. For example, if the tagset contains no more
than 70 hierarchical tags, the annotated corpus will be most effective as training corpus for a stochastic annotator (cf. Heid 1998).
8 Cf. also the linguistical notion of “concetto ingenuo” worked out by Graffi 1991.
43
The parts-of-speech Pronoun, Determiner and Article heavily overlap in their formal and functional
characteristics and different analyses for different languages entail separating them out in different ways. For
the present purpose, we have proposed placing Pronouns and Determiners in one ‘super-category', recognizing
that for some descriptions it may be thought best to treat them as totally different part-of-speech. There is also
an argument for subsuming Articles under Determiners. The present guidelines do not prevent such a
realignment of categories, but do propose that articles (assuming they exist in a language) should always be
recognized as a separate class, whether or not included within determiners. (Leech - Wilson 1999: 63-64).
They then concluded that “the requirement is that the descriptive scheme adopted should be
automatically mappable into the present one” (ibidem): and our scheme certainly and easily is.
4. From CT towards the future
This brings us back to the second point of view of re-usability which we bore in mind as we
planned the CT. From this other perspective, as information retrieval is concerned, since the CT tagset
is fully EAGLES-conformant, information provided by the CT is fully comparable with the main
modern language tagged corpora, without even exiting from the same working system environment.
That the interlinguistic comparison is so greatly facilitated is obvious. But the typological
perspective is certainly not the only one to benefit from this approach.
For example, if historical linguists and historians of the Italian language have annotated corpora of
Old and Modern Italian, which can be easily compared, they could verify the empirical basis of their
theories more easily, or even make new discoveries. A small example of what new observations are
made possible by this corpus is found in the study of multiword entries that I am pursuing. I have
discovered that collocations of the structure “dalla parte mia/tua/…” with variable ending are
apparently unknown to thirteenth-century Florentine: in our corpus only the type “dalla mia/tua/…
parte” with variable internal element.
Last but not least, corpora annotated in the same way for diverse chronological phases of the Italian
language, could be used for lexical acquisition by historical dictionaries. An elementary example could
be the evidence of the change in the subcategorization frame of many verbs from the thirteenth century
to the modern language. This can be only roughly studied when simple concordance programmes are
used. Moreover, since the CT, through the mediation of the Padua Corpus, already constitutes in itself a
reuse of the lexicographic resources from the OVI, its ability to return this favour by making a new
source of information retrieval available, is another demonstration of that non vicious circle of
resources from which our small consideration on re-usability began.
5. Conclusions
At the beginning of this contribution we underlined how technologies designed for society could
become valuable for humanistic research. One of the great merits of computational studies, in fact, has
been just that: to have built a bridge between these two worlds, which previously had relatively little
contact. The example of the CT, in this way, is, I hope, even more noteworthy in that it involves
historical linguistics and philology, subjects that until now have benefited from this profitable
circulation of resources less then other doctrines, such as logic, applied linguistics and lexicography.
6. References
Barbera M 2000 Pronomi e determinanti nell'annotazione dell'italiano antico. La POS “PD” del
Corpus Taurinense. Paper presented at Neuntes österreisch-italienisches Linguistentreffen / Nono
incontro italo-austriaco dei linguisti PARALLELA IX. “Text - Variation - Informatik / Testo -
variazione - informatica”, Salzburg, 1.-4. November 2000.
Barbera M 2000/2001 Italiano antico e linguistica dei corpora: un Tagset per ItalAnt, in VI Convegno
Internazionale SILFI “Tradizione & Innovazione”: La linguistica e filologia italiana alle soglie di
un nuovo millennio. Gerhard-Mercator-Universität Duisburg 28.06.-02.07.2000. Atti.
(Forthcoming).
Barbera M, Marello C 1999/2001 L'annotazione morfosintattica del Padua Corpus: strategie adottate e
problemi di acquisizione. Paper presented at Italiano antico e corpora elettronici. Padova, 19-20
febbraio 1999. Incontro seminariale. Forthcoming in Revue romane 36(1).
Barbera M, Marello C 2000 (Forthcoming in Revue française de linguistique appliquée.) Entrées de
multimots et étiquetage de parties du discours dans le Corpus Taurinense. Paper presented at AFLA
2000, Paris, 6-8 juillet 2000.
Christ O, Schulze BM 1996 CWB. Corpus Work Bench, Ein flexibles und modulares Anfragesystem
für Textcorpora. In Feldweg H, Hinrichs E (eds) Lexikon und Text. Tübingen, Niemeyer.
Garside R, Leech G, McEnery A (eds) 1997 Corpus annotation. Linguistic information from computer
44
text corpora, London - New York, Longman.
Graffi G 1991 Concetti ‘ingenui’ e concetti ‘teorici’ in sintassi. Lingua e stile 26: 347-363.
Heid U. 1998 Annotazione morfosintattica di corpora ed estrazione di informazioni linguistiche. Paper
presented at Annotazione morfosintattica di corpora e costruzione di banche di dati linguistici.
Torino, 26-XI-1998.
Leech G, Wilson A 1999 Standards for tagsets. In van Halteren 1999, pp. 55-80.
Monachini M, Calzolari N 1996 Synopsis and comparison of morphosyntactic phenomena encoded in
lexicons and corpora. A common proposal and application to European languages. Pisa, EAGLES
Document EAG-CLWG-MORPHSYN/R, May 1996.
Monachini M 1996 ELM-IT: EAGLES Specifications for Italian morphosyntax. Lexicon specifications
and classification guidelines. Pisa, EAGLES Document EAG-CLWG-ELM-IT/F, May.
Renzi L 1998 Perché una grammatica dell'italiano antico: una presentazione. In Renzi L (ed), ITALANT:
per una Grammatica dell'Italiano Antico. Padova, Centro Stampa di Palazzo Maldura, pp 21-32.
Renzi L, Salvi G (eds) 1988, 1991, 1995 Grande grammatica italiana di consultazione I.-III. Bologna,
Il Mulino.
Teufel S 1996 ELM-EN. EAGLES Specifications for English morphosyntax. Draft version. Stuttgart,
EAGLES Document, July.
Teufel S, Stöckert Ch 1996 ELM-DE. EAGLES Specification for German morphosyntax. Lexicon
specification and classification guidelines. Stuttgart, EAGLES Document EAG-CLWG-ELMDE/
F, März.
van Halteren H (ed) 1999 Syntactic wordclass tagging. Dordrecht - Boston - London, Kluver Academic
Publishers, Text, speech and language technology 9.

Optimisation of corpus-derived probabilistic grammars
Anja Belz
CCSRC, SRI International
23 Millers Yard, Mill Lane
Cambridge CB2 1RQ, UK
1 Overview
This paper examines the usefulness of corpus-derived probabilistic grammars as a basis for the automatic
construction of grammars optimised for a given parsing task. Initially, a probabilistic context-free
grammar (PCFG) is derived by a straightforward derivation technique fromthe Wall Street Journal (WSJ)
Corpus, and a baseline is established by testing the resulting grammar on four different parsing tasks. In
the first optimisation step, different kinds of local structural context (LSC) are incorporated into the basic
PCFG. Improved parsing results demonstrate the usefulness of the added structural context information.
In the second optimisation step, LSC-PCFGs are optimised in terms of grammar size and performance for
a given parsing task. Tests showthat significant improvements can be achieved by the method proposed.
The structure of this paper is as follows. Section 2 discusses the practical and theoretical questions
and issues addressed by the research presented in this paper, and cites existing research and results in the
same and related areas. Section3 describes howLSC-grammars are derived fromcorpora, defines the four
parsing tasks on which grammars are tested, describes data and evaluation methods used, and presents
a baseline technique and baseline results. Section 4 discusses and describes different types of LSC and
demonstrates their effect on rule probabilities. Methods for deriving four different LSC-grammars from
the corpus are described, and results for the four parsing tasks are presented. It is shown that all four
types of LSC investigated improve results, but that some lead to overspecialisation of grammars. Section 5
shows that LSC-grammars can be optimised for grammar size by a generalisation technique that at the
same time seeks to optimise parsing performance for a given parsing task. An automatic search method
is described that carries out a search for optimal generalisations of the given grammar in the space of
partitions of nonterminal sets. First results are presented for the automatic search method that showthat
it can be used to reduce grammar size and improve parsing performance.
Parent node information is shown to be a particularly useful type of LSC, and the results for the
complete parsing task achieved with the corresponding grammar are better than any previously published
results for comparable unlexicalised grammars. Preliminary tests for LSC grammar optimisation show
that it can drastically reduce grammar size and significantly improve parsing performance. In one set
of experiments, a partition was found that increased the labelled F-Score for the complete parsing task
from72.31 to 74.61, while decreasing grammar size from21,995 rules and 1,104 nonterminals to 11,254
rules and 224 nonterminals. Results for grammar optimisation by automatic search of the partition space
showthat improvements in grammar size and parsing performance can be achieved in this way, but do not
come close to the big improvements achieved in preliminary tests. It is concluded that more sophisticated
search techniques are required to achieve this.
2 Background and related research
The research reported in this paper covers a range of issues: (i) corpus-derived grammars; (ii) the usefulness
of structural context information in making parsing decisions; (iii) automatic construction methods
for specialised grammars that take corpus-derived grammars as a starting point; (iv) the (in)adequacy
of PCFGs as a grammar formalism; and (v) the question of whether parsing strategies that do without
lexical information can come closer to the performance of lexicalised systems. Each of these issues will
be discussed in more detail over the following sections.
Corpus-derived grammars. Over the last five years, a range of research projects—e.g. Charniak
(1996), Cardie & Pierce (1998), Johnson (1998, 2000), Krotov et al. (2000)—have looked at probabilistic
grammars that have been directly derived frombracketted corpora (or treebanks, hence the term
46
“treebank grammar” coined by Charniak, 1996). The basic idea in grammar derivation fromcorpora is
simple. For each distinct bracketting found in a corpus, a grammar rule is added to the grammar and the
rule's probabilityis derived in some way (often by maximum-likelihoodestimation with some smoothing
method) fromthe frequency of occurrence of the bracketting in the corpus. For instance, the bracketting
(NP (DT the) (NN tree)) would yield the production rule NP ! DT NN.
However, because the number of rules in grammars derived in this entirely straightforwardmanner is
infeasibly large at least in the case of the WSJ Corpus, and because their parsing performance moreover
tends to be poor, some techniques are usually applied to reduce grammar size and to improve performance.
All approaches edit the corpus in some way, e.g. eliminating single child rules, empty category
rules, functional tags, co-indexation tags, and punctuation marks. Different compaction methods (such
as eliminatingrules with frequency less than some n) have been investigated that reduce the size of grammars
without too much loss of performance (in particular by Charniak and Krotov et al.). To improve
parsing performance, e.g. Charniak relabels auxiliary verbs with a separate POS-tag and incorporates a
“right-branching correction” into the parser to make it prefer right-branching structures.
As a result of such techniques, the final grammars for which performance results are reported tend to
have little in common with the rule set underlying the corpus fromwhich they were derived.
Several other grammar building and training methods are similar to treebank grammar construction:
Bod&Scha's DOP1 method which extracts tree fragments rather than rules fromcorpora, MBL2 methods
(Daelemans et al.) for building parsing systems fromcorpora, and—more generally—any method that
estimates the likelihood of brackettings (or of brackettings converted into taggings) froma corpus, since
such methods directly utilise both the brackettings and their frequencies as found in the corpus.
The existing results for corpus-derived grammars that do not undergo significant further development
demonstrate their limitations: they cannot compete with state-of-the-art parsing results (see Section 2). It
will be argued in this paper that grammars directly extracted fromcorpora do, however, provide a useful
starting point for further automatic grammar construction methods.
Context-free grammars that incorporate structural context. It is frequently observed (e.g. Manning
& Sch¨utze (1999, p. 416ff)) that PCFGs are inadequate as a grammar formalism because of the
very strong independence assumptions inherent in them, reflecting on the one hand a complete lack of
lexicalisation, and on the other a lack of structure dependence.
It is true that in conventional PCFGs the probability of, say, a given NP bracketting is independent of
the identity of the head noun as well as its structural context (e.g. whether the NP is in subject or object
position). However, this independence is not due to the formal characteristics of PCFGs, but rather to the
way they tend to be used. If the set of nonterminals of a PCFG does not distinguish between, say, NPs in
subject position and NPs in object position, then the probabilities of any rules containing the nonterminal
NP are necessarily independent of the subject/object distinction.
However, it is straightforward to introduce such a dependence into a PCFG by splitting the category
NP into two categories NP-SBJ and NP-OBJ. Similarly, categories (nonterminals) can be divided on the
basis of lexemes, lexical categories or semantic classes.
PCFGs may not be able to accommodate lexical and structural informationin themost elegant fashion,
but the point here is not about representational elegance and efficiency. Rather, the fact that PCFGs encode
languages that make up the formal class of context-free languages is entirely separate fromtheir ability
to reflect the dependence of rule probabilities on lexical and structural context.
Examining different kinds of structural context within the PCFG framework (as done in this paper)
has two advantages: firstly, there are polynomial-time algorithms for finding most likely parses, and
secondly, there is a simple measure of the complexity added to a grammar by the introduction of a
piece of structural information such as the subject/object distinction, namely the resulting increase in the
number of rules in the grammar.
Automatic grammar construction. It is sometimes observed that deriving probabilistic grammars
fromcorpora in the way described above is not an automatic grammar learning method because all that
is done is to extract the PCFG that underlies the corpus and is encoded in its sentences, brackettings and
occurrence frequencies. As was pointed out above, creating a grammar in this way is simply one of many
ways to utilise the brackettings and frequencies of corpora, a feature shared with many computational
learning approaches to automatic grammar construction. However, as previously mentioned, the limitations
of grammars directly extracted fromcorpora indicate that using themas starting points for further
grammar development is the more useful approach.
1Data-Oriented Parsing.
2Memory-Based Learning.
47
Grammar/Parser Grammar Size performance (WSJ unseen)
LR UR LP UP CB
Fully lexicalised:
Collins (2000) – 90.1 – 90.4 – 0.73
Charniak (2000) – 90.1 – 90.1 – 0.74
Collins (1999) – 88.5 – 88.7 – 0.92
Collins (1997) – 88.1 – 88.6 – 0.91
Charniak (1997) – 87.5 – 87.4 – 1.00
Magerman (1995) SPATTER – 84.6 – 84.9 – 1.26
Nonlexicalised:
Charniak (1996) 10,605 – 78.8 – 80.4 –
without frequency 1 rules 3,943 – 78.2 – 80.7 –
Krotov et al. (2000) 15,420 74.1 77.1 77.4 80.6 2.13
without frequency 1 rules 6,514 74.4 77.5 76.9 80.2 2.18
WSJ 15–18 treebank PCFG 6,135 69.1 – 71.4 – 2.67
Table 1: Performance of comparable lexicalised and nonlexicalised grammars on full parsing.
Reference Method LP LR F-Score
Lexicalised:
Tjong KimSang et al. (2000) Systemcombination 94.2 93.6 93.9
Mu˜noz et al. (1999) SNoW 92.4 93.1 92.8
XTAG Research Group (1998) XTAG + Supertagging 91.8 93.0 92.4
Ramshaw &Marcus (1995) Transformation Based Learning 91.8 92.3 92.0
Veenstra (1998) MBL 89.0 94.3 91.6
Nonlexicalised:
Argamon et al. (1999) MBL 91.6 91.6 91.6
Cardie&Pierce (1998) Error-Driven Grammar Pruning 90.7 91.1 90.9
WSJ 15–18 treebank PCFG 89.2 87.6 88.4
Table 2: Performance of comparable lexicalised and nonlexicalised grammars on NP-chunking.
Creating a starting point for grammar learning in this way is particularly useful because context-free
grammars cannot be learnt fromscratch fromdata. At the very least, an upper bound must be placed on
the number of nonterminals allowed. Even when that is done, there is no likelihood that the grammars
resulting from an otherwise unsupervised method will look anything like a linguistic grammar whose
parses can provide a basis for semantic analysis3.
Parsing with(out) lexical information. Corpus-derived grammars tend to be nonlexicalised PCFGs,
hence the existing research cited above can be seen as investigations into the results that can be achieved
in parsing without taking into account lexical information.
In syntactic parsing tasks, nonlexicalised methods are generally outperformed by lexicalised approaches.
In the case of complete (non-shallow) parsing, nonlexicalised methods are outperformed by
large margins. Table 1 shows an overview of state-of-the-art nonlexicalised and lexicalised results for
statistical parsing systems (U/LR= Un-/Labelled Recall, U/LP = Un-/Labelled Precision, see Section 3).
For comparison, the last rowof the table shows this paper's baseline result for the complete parsing task
(see Section 3.4).
In NP-chunking, a shallow syntactic parsing task that has become a popular research topic over the
last decade (for details see Section 3.2 below), nonlexicalised systems also tend to lag behind lexicalised
ones, although by much smaller margins. Table 2 shows a range of results for the baseNP chunking task
and data set given by Ramshaw & Marcus (1995). Again, the corresponding baseline result from this
paper is included in the last row. It is clear from this overview that the difference between lexicalised
and unlexicalised systems is far smaller for this parsing task than for complete parsing.
There are several reasons for investigating how well parsers can do without lexicalisation. Apart
3Any linguistic CFG can be converted into a normal form that encodes the same set of sentences, but whose derivations and
substructures are not semantically meaningful.
48
fromthe theoretical interest, optimisinggrammars before adding lexicalisationmay improve their overall
performance, as lexicalised systems often performworse than comparable nonlexicalised systems when
the lexical component is taken out. E.g. Collins (1996) includes results for the system with lexical
information removed, which reduces LR from85.0 to 76.1 and LP from85.1 to 76.6 in one test – worse
than the comparable results reported below in Section 4.3 (78.78 and 77.16). Furthermore, the results
shown in Tables 1, 2 and 4 indicate that shallowparsing tasks require lexical information to a far lesser
extent than nonshallowones, so that the added expense of lexicalisation might be avoidable in the case
of such tasks.
3 Grammars, parsing tasks, data and evaluation
3.1 Grammars fromcorpora
The basic procedure used for deriving PCFGs fromWSJ Sections 15–18 can be summarised as follows4:
1. In the first step, the corpus is iteratively edited by deleting (i) brackets and labels that correspond
to empty category expansions; (ii) brackets and labels containing a single constitutent that is not
labelled with a POS-tag; (iii) cross-indexation tags; (iv) brackets that become empty through a
deletion; and (v) functional tags.
2. In the second step, each remaining bracketting in the corpus is converted into a production rule.
The rules are divided into nonlexical ones (those that form the grammar), and lexical ones (those
that formthe lexicon).
3. In the final step, a complete PCFG is created. The set of lexical rules is converted into a lexicon
with POS-tag frequency information. The set of nonterminals is collected from the set of rules.
Each set is sorted, the number of times each item occurs is determined, and duplicates are removed.
Probabilities P for rules N !  are calculated fromthe rule frequencies C by Maximum
Likelihood Estimation (MLE): PMLE(N ! ) = C(N!)
Pi
C(N!i)
3.2 Four parsing tasks
Results are given in the following and subsequent sections for four different parsing tasks:
1. Full parsing: The task is to assign a complete parse to the input sentence. Afull parse is considered
100%correct if it is identical to the corresponding parse given in the WSJ Corpus.
2. Noun phrase identification: The task is to identify in the input sentence all noun phrases, nested
and otherwise, that are given in the corresponding WSJ Corpus parse.
3. Complete text chunking: This task was first defined in Tjong KimSang & Buchholz (2000), and
involves dividing a sentence into flat chunks of 11 different types. The target parses are derived
fromWSJ parses by a deterministic conversion procedure.
4. Base noun phrase identification: First defined by Abney (1991), this task involves the recognition
of non-recursive noun phrase chunks (so-called baseNPs). Target parses are derived from WSJ
parses by a simple conversion procedure.
3.3 Data and evaluation
Sections 15–18 of theWall Street Journal (WSJ) corpus were used for grammar derivation, and Section 01
from the same corpus was used for testing parsing performance. Parsing performance was tested with
the commonly used evalb program by Sekine & Collins5. The program evaluates parses in terms of
the standard PARSEVAL evaluation metrics Precision, Recall and crossing brackets. For a parse P and a
corresponding target parse T, Precision measures the percentage of brackets in P that match the target
brackettings in T. Recall is the percentage of brackettings from T that are in P. Crossing brackets gives
the average number of constituents in one parse tree that cross over constituent boundaries in the other
tree. (See e.g. Manning&Sch¨utze (1999), p. 432–434.)
For Precision and Recall there are unlabelled and labelled variants. In the latter, both the pair of
brackets and the constituent label on the bracket pair have to be correct for the bracketting to be correct,
4Throughout this paper, WSJ refers to the PENN II Treebank version.
5Available fromhttp://cs.nyu.edu/cs/projects/proteus/evalb/.
49
whereas in the unlabelled variant only the brackets have to be correct. In this paper, unless otherwise
stated, Precision and Recall always mean Labelled Precision and Recall, in particular, all new results
presented are the labelled variants. Precision and Recall are commonly combined into a single measure,
called F-Score, given by (2 +1 )Precision Recall= 2 (Precision +Recall). In this paper,  = 1
throughout.
All grammars tested are nonlexicalised, therefore input sentences are sequences of POS-tags not
words. In the tests, sentences of a length above 40 words (consistently close to 7.5%of all sentences in
a corpus section) were left out. All grammars are formally probabilistic context-free grammars (PCFGS).
The parsing package LoPar (Schmid (2000)) was used to obtain Viterbi parses for data sets and grammars.
If LoPar failed to find a complete parse for a sentence, a simple grammar extension method was
used to obtain partial parses instead.
3.4 Baseline
A baseline grammar “BARE” was extracted from WSJ Sections 15–18 by the method described in Section
3.1, applied to the four parsing tasks defined in Section 3.2, and tested and evaluated as desribed in
the preceding section. This yielded the following set of results which forms the baseline for the purpose
of this paper. (Results include 9 partial parses.)
Full parsing NP identification BaseNP chunking Complete text chunking
LR LP F LR LP F LR LP F LR LP F
69.08 71.43 70.24 74.97 81.62 78.15 87.6 89.21 88.4 89.63 88.99 89.31
4 Introducing structural context into PCFGs
4.1 Different types of structural context
In this section, the effects of introducing three different types of structural context (SC) into PCFG BARE
are examined: (i) the grammatical function of phrases, (ii) their depth in the parse tree, and (iii) the category
of the parent phrase. All three types of structural context are local to the immediate neighbourhood
of the phrase node for which they provide the expansion probabilityconditions. Other local SC types that
could be considered include position among the children of the parent node, and identity of immediate
sibling nodes. Useful nonlocal SC types might be the identity of more distant ancestors than the parent
node and of more distant sibling nodes.
Grammatical function. As mentioned above, the WSJ corpus subdivides standard phrase categories
such as NP by attaching functional tags to themthat reflect the grammatical function of the category, e.g.
NP-SBJ and NP-OBJ. However, the corpus is not consistently annotated in this fashion (the same type of
phrase may have zero, one or more functional tags). Parsing results for grammar FTAGS might be better
if the grammar is derived froma more consistently annotated corpus.
The rule that expands a noun phrase to a personal pronoun is a strong example of the extent to which
grammatical function can affect expansion probabilities. In the WSJ, 13.7% of all NPs expand to PRP
as subject, compared to only 2.1% as object. Of all object NPs, 13.4% expand to PRP as first object,
compared to 0.9%as second object (source: Manning&Schuetze, 1999. p. 420).
Depth of embedding. The depth of embedding of a phrase is determined as follows. The outermost
bracketting (corresponding to the top of the parse tree) is at depth 1, its immediate constituents are at
depth 2, and so on. In the parsed sentence (S (NP (DT the) (NN cat)) (VP (VBD sat) (PP
(IN on) (NP (DT the) (NN mat))))), S is at depth 1, the first occurrences of NP and VP are at
depth 2, the first occurrences of DT and NN as well as VBD and PP at depth 3, IN and the second NP at
depth 4, and the second occurrences of DT and NN are at depth 5.
It is not obvious that the depth of embedding of a phrase captures linguistically meaningful parts of
its local structural context. However, different phrases of the same category do occur at certain depths
with higher frequency than at others. This is most intuitively clear in the case of NPs, where subject NPs
occur at depth 2, whereas object NPs occur at lower depths.
More surprisingly, VPs too have preferences for occurring at certain levels. Table 3 (previouslyshown
in Belz (2000, p. 49)) provides clear evidence of this. The first column shows the six most frequent WSJ
VP expansion rules, the second column shows their canonical probabilities (calculated over all WSJ VP
rules). The remaining columns show how these probabilities change if they are made conditional on
depths of embedding 2–7. For each depth, the highest rule probability is highlighted in boldface font,
50
Depth of Embedding
2 3 4 5 6 7
p(VP!TOVP) 0.089 0.004 0.067 0.136 0.127 0.135 0.130
p(VP!MD VP) 0.056 0.075 0.043 0.055 0.062 0.050 0.047
p(VP!VB NP) 0.054 0.001 0.036 0.052 0.073 0.088 0.096
p(VP!VBNPP) 0.039 0.004 0.049 0.047 0.042 0.044 0.055
p(VP!VBZ VP) 0.038 0.069 0.034 0.037 0.025 0.023 0.021
p(VP!VBDS) 0.026 0.090 0.016 0.005 0.005 0.004 0.003
Table 3: Rule probabilities at different depths of embedding for 6 common VP rules.
and the second highest in italics. At depth 2, for instance, the most likely rule is the one with the fourth
highest canonical probability, and at depth 5, the second most likely rule is the one with the third highest
canonical probability. In fact, there is only one depth (4) at which rule probablities appear in their
canonical order, which shows howstrongly even VP rules are affected by depth of embedding.
Parent node. The parent node of a phrase is the category of the phrase that immediately contains
it. In (S (NP (DT the) (NN cat)) (VP (VBD sat) (PP (IN on) (NP (DT the) (NN
mat))))) S is the parent of NP and VP, VP is the parent of PP, which is the parent of NP. Thus, distinguishing
between NP-S (an NP with S as its parent) and NP-PP captures part of the subject/object
distinction.
The advantage of using parent node information was previously noted by Johnson6 (1998).
4.2 Four LSC-Grammars
Grammars incorporating local structural context—or LSC grammars—were extracted fromthe corpus
by the same procedure as described in Section 3.1 above, except that during Step 2, each bracket label
that is not a POS tag was annotated with a tag representing the required type of LSC.
Four different grammars were derived in this way, PCFGs FTAGS, DOE, PN and DOEPN. All four
grammars incorporate the functional tags present in the WSJ Corpus. Additionally, for grammar DOE,
each nonterminal was annotated with a tag representing the depth of embedding at which it was found,
for grammar PN, nonterminals were annotated with tags encoding their parent node, and for grammar
DOEPN, nonterminals were given both depth and parent node tags. The resulting grammars are signifi-
cantly larger than the baseline grammar BARE. Grammar sizes and numbers of nonterminals (excluding
POS tags) are as follows:
Grammar Type BARE FTAGS DOE PN DOEPN
Size (n rules) 6,135 10,118 21,995 16,480 33,101
Nonterminals 26 147 1,104 970 4,015
4.3 Performance on parsing tasks
In calculating labelled bracketting Recall and Precision for the LSC-grammar results, all labels starting
with the same category prefix, e.g. NP, are considered equivalent (standard in evalb). The idea is that
the additional information encoded in the LSC-tags attached to category labels helps select the correct
parse, not that it should be retained in the annotation for further analysis. Table 4 shows parsing results
for the unseen data in WSJ Section 01 (the results for baseline grammar BARE are also included for
comparison). Best F-Scores are highlighted in boldface font, and second-best F-Scores in italics.
The best results in Table 4 are better than those reported by Charniak (1996) and Krotov et al. (2000),
even though the previous results were obtained after using ca. 10/11 of the WSJ corpus as a training set
(compared to 3/25 used here):
UF LF
Krotov et al. (2000) 79.12 76.09
Charniak (1996) 79.59 –
PN-Grammar 80.51 77.96
6Johnson calls it grandparent node, but means the same thing.
51
Grammar Type BARE FTAGS DOE PN DOEPN
Partial parses 9 9 25 20 62
Full parsing:
LR 69.08 71.41 72.72 78.78 74.33
LP 71.43 73.06 71.9 77.16 70.61
F-Score 70.24 72.23 72.31 77.96 72.42
Crossing brackets 2.76 2.51 2.53 1.91 2.59
%0 CBs 32.34 35.43 35.75 44.40 37.0
NP identification:
LR 74.97 77.22 78.2 83.86 81.02
LP 81.62 81.02 77.56 81.22 74.30
F-Score 78.15 79.07 77.88 82.52 77.51
BaseNP chunking:
LR 87.6 87.35 87.02 90.27 87.05
LP 89.21 88.68 87.03 89.52 84.11
F-Score 88.4 88.01 87.02 89.89 85.55
Complete text chunking:
LR 89.63 89.49 89.17 90.84 89.24
LP 88.99 88.64 87.28 89.46 85.85
F-Score 89.31 89.06 88.21 90.14 87.51
Table 4: Parsing results for the four LSC-grammars and WSJ Section 01.
Incorporating different types of LSC affects results for the four parsing tasks in different ways. It is
clear fromthe results in Table 4 that some kinds of contextual information are useful for some tasks but
not for others. For example, adding parent phrase information improved results (from grammar BARE
to grammar PN) by almost 8 points (F-Score 70.24 to 77.96) for the complete parsing task, by about 4.5
points (F-Score 78.15 to 82.52) for NP identification, by 1.5 points (F-Score 88.4 to 89.89) for baseNP
chunking, and by just under one point (F-Score 89.31 to 90.14) for complete text chunking.
It is likely that adding depth of embedding information indiscriminately (as in grammars DOE and
DOEPN) results in overspecialisation. Looking at results for seen data (part of the training corpus) con-
firms this. Table 5 shows results for the baseline grammar and the four LSC grammars on WSJ Section 15,
i.e. one of the sections used during grammar derivation. On seen data, grammar DOEPN performs best
on all parsing tasks. Tables 4 and 5 together imply that adding depth of embedding information for all
depths to all rules simply overfits the training data and results in undergeneralisation.
Similarly, it is likely that not all the information added in the four LSC grammars is useful for all
parsing tasks. Distinguishing 27 depths of embedding is probably too much for all parsing tasks, e.g.
distinguishingdepths above 20 is generally unlikelyto be useful, as the occurrence of rules at such depths
is rare. Techniques for eliminating the information that makes no useful contribution for a given parsing
task are discussed in the following section.
5 Automatic optimisation of LSC-Grammars
5.1 Initial assumptions
If it is true that some of the LSC information added to the grammars tested so far makes little or no contribution
to a grammar's performance on a given parsing task, then it should be possible to reduce grammar
size without loss of parsing performance by selectively taking out some of the added information. At
the same time, if it is true that some of the LSC-grammars are overspecialised (overfit the data), then it
should be possible to improve the grammar's performance by selectively generalising them.
As pointed out above in Section 4.3, it is clear fromthe LSC results that adding different kinds of LSC
information to a grammar has different effects on the results for different parsing tasks. It should therefore
be possible to optimise a grammar for a given parsing task by selectively taking out the information
that is not useful for the given task. The idea behind the experiments reported in the following section
was to see to what extent the LSC grammars can be optimised in terms of size and parsing performance
by grammar partitioning for each of the parsing tasks.
52
Grammar Type BARE FTAGS DOE PN DOEPN
Partial parses 0 0 0 0 0
Full Parsing:
LR 71.48 75.15 82.81 84.64 90.39
LP 75.03 78.64 84.86 85.94 91.43
F-Score 73.21 76.86 83.82 85.29 90.91
Crossing brackets 2.57 2.15 1.37 1.31 0.75
%0 CBs 34.48 41.85 56.31 57.33 73.46
NP identification:
LR 76.54 79.26 84.51 87.46 91.17
LP 84.89 85.61 88.79 88.75 92.61
F-Score 80.5 82.31 86.6 88.1 91.88
BaseNP chunking:
LR 90.21 90.28 92.68 94.40 95.99
LP 92.59 92.70 94.54 95.66 97.19
F-Score 91.38 91.47 93.60 95.03 96.59
Complete text chunking:
LR 91.68 91.67 93.59 94.25 96.45
LP 92.46 92.56 94.19 95.02 96.84
F-Score 92.07 92.11 93.89 94.63 96.64
Table 5: Parsing results for the LSC-grammars and WSJ Section 15 (seen data).
5.2 Preliminary definitions
The addition of structural context as described in previous sections can be viewed in terms of split
operations on nonterminals, e.g. in the FTAGS grammar, the nonterminal NP is split into NP-SUBJ and
NP-OBJ (among others). This results in grammar specialisation, i.e. the new grammar parses a subset
of the set of sentences parsed by the original one. The reverse, replacing NP-SUBJ and NP-OBJ with a
single nonterminal NP, can be seen as a merge operation, and results in grammar generalisation, i.e. the
new grammar parses a superset of the sentences parsed by the original one.
An arbitrary number of such merge operations can be represented by a partition on the set of nonterminals
of a grammar. A partition is defined as follows.
Definition1 Partition
A partition of a nonempty set A is a subset  of 2A such that ; is not an element of  and
each element of A is in one and only one set in .
PCFGs can be defined as follows.
Definition2 Probabilistic Context-Free Grammar (PCFG)
APCFGs is a 4-tuple (W;N ;NS;R), whereW is a set of terminal symbols fw1; : : :wug, N
is a set of nonterminal symbols fn1; : : : nvg, NS  N is a set of start symbols fns
1; : : : ns
vg,
und R is a set of rules with associated probabilities f(r1; p(r1)); : : : (r1; p(rx))g. Each rule
r is of the form n ! , where  is a sequence of terminals and nonterminals. For each
nonterminal n, the values of all p(n ! i) sumto one.
Given a PCFG G = (W;N ;NS;R) and a partition N = fN1; : : :Nvg of the set of nonterminals
N, the partitioned PCFG G0 = (W;N0;NS0
;R0) is derived by the following procedure:
1. Assign a newnonterminal name to each of the non-singleton elements of N.
2. For each rule ri in R, and for each nonterminal nj in ri, if nj is in a non-singletonelement of N,
replace it with the corresponding new nonterminal.
3. Find all rules in R of which there are multiple occurrences as a result of the substitutions in Step 2,
sumtheir frequencies and recalculate the rule probabilities.
53
If start symbols are permitted to be merged with non-start symbols, then there are two ways of
determining the probability of a rule expanding the nonterminal resulting fromsuch a merge: either its
frequency is the sum of the frequencies of all nonterminals in the merge set, or it is the sum of just the
frequencies of the start symbols in themerge set. The latter optionwas chosen in the tests reported below.
5.3 “Proof of concept”
The discussion and results in this section provide preliminary confirmation of the prediction made in
Section 4.3 that for the different LSC grammars there exist (non-trivial) partitions that outperform the
original base grammar. More formally, the “proof of concept” provided below shows the following for
most of the grammar/task combinations:
Given a base grammar G = (W;N ;NS;R) and a parsing task P, a partition of the set of
nonterminals N can be found such that the derived grammar G0 = (W;N0;NS0
;R0)
1. is smaller than G (i.e. jR0j < jRj), and
2. performs better than G on P.
Some of the five LSC-PCFGs can be derived by partition fromone of the others. For example, BARE
can be derived from all others, FTAGS can be derived from DOE, PN and DOEPN, and DOE and PN
can both be derived from DOEPN. This means that for some of the grammars, the results given in
Section 4.3 in themselves show that there exists at least one (non-trivial) partition that is smaller than
and outperforms the original grammar. E.g. for the baseNP chunking task, the partition that derives PN
fromDOEPN achieves nearly a 3 point improvement (F-Score 87.63 to 90.23), while reducing grammar
size from33,101 rules to 16,480, and the number of non-terminals from4,015 to 970.
In the remainder of this section it is shown that there are other partitions of the DOE grammar that
improves its performance and reduces its size.
Grammar type Depth bands Grammar size Nonterminals
DOE 1, 2, . . . 27 21,995 1,104
1, 2, 3, rest 12,933 312
1, 2, rest 11,254 224
1, rest 10,165 170
FTAGS – 10,118 147
BARE – 6,135 26
Table 6: Sizes and depth bands of DOE grammar and 5 of its partitions.
Fromthe parsing results for the DOE grammar it appears that indiscriminately adding depth of embedding
information does not help improve parsing performance for shallow parsing tasks on unseen
data: while there is a significant improvement for the complete parsing task (F-Score 70.24 to 72.31),
the F-Scores for the other three parsing tasks are worse. That there is any improvement shows that some
useful information is added. It is likely that distinguishing all depths simply leads to overspecialisation
of the grammar, resulting in a large increase in parse failures on the one hand, and the selection of
bad, previously unlikely, parses on the other. If this is so then partitioning the DOE grammar in a way
equivalent to distinguishing broader depth bands rather than each individual depth will improve results.
To test this hypothesis, three different partitions of the DOE grammar were created. The partitions
(too large to be shown in their entirety) correspond to distinguishing between the different depths shown
in the second column of Table 6, e.g. in the case of the fourth row, all nonterminals NT-n with a depth tag
n greater than 1 are merged into a single nonterminal NT-rest. The last two columns show the number
of rules and nonterminals in each grammar. The last two rows show the corresponding numbers for the
BARE and FTAGS grammars (DOE-type grammars all incorporate functional tags).
The partitionedDOEgrammars all improve results (compared to grammars BARE, FTAGS, and DOE)
for the full parsing task, with the DOE-f1, 2, restg grammar performing the best. For the NP identifi-
cation task, grammar DOE achieved a worse F-Score than grammar BARE, yet all the partitioned DOE
grammars achieve a better F-Score than grammar BARE, with the DOE-f1, 2, restg grammar again performing
the best. On the baseNP chunking task and the complete text chunking task, grammar BARE
performs the best, but all the derived DOE grammars outperformthe nonpartitioned DOE grammar. On
54
Grammar Type BARE FTAGS DOE DOE-f1,rg DOE-f1,2,rg DOE-f1, 2, 3, rg
Full Parsing:
LR 69.08 71.41 72.72 73.35 74.13 74.14
LP 71.43 73.06 71.9 74.48 75.1 74.73
F-Score 70.24 72.23 72.31 73.91 74.61 74.43
Crossing brackets 2.76 2.51 2.53 2.32 2.19 2.21
%0 CBs 32.34 35.43 35.75 38.83 39.1 38.56
NP identification:
LR 74.97 77.22 78.2 77.39 77.95 78.29
LP 81.62 81.02 77.56 81.20 81.31 80.85
F-Score 78.15 79.07 77.88 79.25 79.59 79.55
BaseNP chunking:
LR 87.6 87.35 87.02 87.82 87.73 87.74
LP 89.21 88.68 87.03 88.93 88.59 88.11
F-Score 88.4 88.01 87.02 88.37 88.16 87.93
Complete text chunking:
LR 89.63 89.49 89.17 89.58 89.71 89.70
LP 88.99 88.64 87.28 88.54 88.52 88.29
F-Score 89.31 89.06 88.21 89.06 89.11 88.99
Table 7: Parsing results of DOE grammar and 5 of its partitions.
{ {0}, {1}, {2} }
{ {0,2}, {2} } { {1,2}, {0} } { {0,1}, {2} }
{ {0,1,2} } { {0,1,2} } { {0,1,2} }
Figure 1: Partition tree for a set with three elements.
the baseNP chunking task, the BARE grammar's F-Score is closely matched by the DOE-f1, restg grammar.
These results show that partitions can be found that not only drastically reduce grammar size but
also significantly improve parsing performance on a given parsing task.
5.4 Search for optimal partition of LSC-Grammars
Given. A PCFG G = (W;N ;NS;R), a data set D, and a set of target parses DT for D.
Search space. The search space is defined as the partition tree for the set of nonterminals N in the
given grammar G. Each node in the tree is one of the partitions of N, such that each node's partition has
fewer elements than all of its ancestors, and the partition at each node can be derived fromits parent by
merging two elements of the parent's partition.
The single node at the top of the tree is the trivial partition corresponding to N itself. Each node is
the parent of 1
2(n2 
level reduces the number of states by one. The complete partition tree for a set with three elements looks
as shown in Figure 1.
Search method. The partition tree is searched top-down by a variant of beam search. A list of the
n current best candidate partitions is maintained (initialised to the trivial partition). For each of the n
current best partitions a subset of size b of its children in the partition tree is generated and evaluated
(b thus defines the width of the beam). From the set of current best partitions and the newly generated
candidate partitions, the n best elements are selected and formthe new current best set. This process is
iterated until either no new partitions can be generated that are better than their parents, or the lowest
level of the partition tree is reached.
In the current version of the evaluation function, only the F-Score achieved by candidate solutions
on the test data is taken into account. Search stops if in any iteration (depth of the partition tree) no
55
solution is found that outperforms the current best solutions. That is, size is not explicitly evaluated at
all. Candidate solutions are evaluated on a subset of the test data, because evaluating each candidate
solution on all 1,993 sentences of WSJ Section 01 makes the cost of the search procedure prohibitive.
There are three variable parameters in the partition tree search procedure: (i) the number n partitions
(nodes in the tree) that are further explored, (ii) the size x of the subset of the test data that candidate
solutions are evaluated on, and (ii) the width b of the beam.
5.5 Results for LSC-Grammar optimisation by search of partition tree
Table 8 shows some results for automatic optimisation experiments carried out for grammar PN and the
baseNP chunking and complete text chunking tasks. The first three columns showthe variable parameters
b (beamwidth), n (size of list of best solutionsmaintained), and x (size of data subset used in evaluation).
The fourth column shows the number of runs results are averaged over, and the fifth and sixth columns
show the number of iterations and evaluations carried out before search stopped. Column 7 gives the
average number of nonterminals the best solution grammars had, and column 8 their average evaluation
score. The last two columns show the overall change in F-Score (calculated on all of WSJ Section 01)
and grammar size for the given grammar and parsing task.
Var. Parameters Runs Iter. Eval. Nonterms F-Score (sub) F-Score +/- Size +/-
b n x
Grammar: PN; Grammar Size: 16,480/970
Task: BaseNP chunking; F-Score: 89.89
100 2 50 4 4 45 968.25 95.93 +0.032 (89.92) -0.25
100 10 50 4 6.75 341.5 967.25 97.25 +0.048 (89.94) -2
500 1 50 4 5.25 499 967.5 97.49 +0.06 (89.95) -2.25
Grammar: PN; Grammar Size: 16,480/970
Task: Complete Text Chunking; F-Score: 90.14
1,000 1 10 4 5 523.75 967 100.00 +0.06 (90.2) -0.75
Table 8: Results for automatic optimisation tests.
Current results showinsensitivityto the precise values of parameters b and n. What appears to matter
is just the total number of evaluations, results being better the more candidate solutions are evaluated.
Results indicate a greater sensitivity to the value of x: a data subset size of 10 is clearly too small, as
search quickly finds solutions with an F-Score of 100 and then stops (last rowof Table 8).
Overall, results are not nearly as good as might have been expected after the preliminary tests described
above. Only small numbers of nonterminals were merged, and small improvements achieved,
before search stopped. However, the fact that every single run achieved an F-Score improvement and
almsot all runs resulted in a decrease in grammar size even for small numbers of merged nonterminals
indicates that the basic approach is right, but that some way has to be found of overcoming the local optima
on which search in the reported experiments stopped, by widening the width of the beam, changing
the evaluation function, or by using a more sophisticated search method.
6 Conclusions and further research
The first part of this paper looked at the effect of adding three different kinds of local structural context
—grammatical function, parent node and depth of embedding—to a basic PCFG derived fromthe Wall
Street Journal Corpus. Grammars were tested on four different parsing task differing in complexity and
shallowness. Results showed that all three types of context improve performance on the complete parsing
task, but that only parent node information improves performance on all parsing tasks. The PCFG with
parent node information was particularly successful and achieved better results on the complete parsing
task than the best previously published results for nonlexicalised grammars and WSJ corpus data.
In the second part of the paper, a newmethod for optimising PCFGs was introduced that has the effect
of overcoming overspecialisation by generalising grammars. It was shown that partitions can be found
that drastically reduce grammar size and significantly improve parsing performance. First results were
reported for applying an automatic search method to a PCFG that incorporates parent node information,
56
and the tasks of baseNP chunking and complete text chunking. Results are promising, but indicate that in
order to achieve radical improvements in parsing performance and grammar size, a different evaluation
function and/or more sophisticated search methods may be required.
7 Acknowledgements
The research reported in this paper was carried out as part of the Learning Computational Grammars
Project funded under the European Union's TMR programme (Grant No. ERBFMRXCT980237).
References
Krotov A, Hepple M, Gaizauskas R, Wilks Y. 2000. Evaluating two methods for treebank grammar
compaction. Natural Language Engineering, 5(4):377–394.
Belz A. 2000. Computational Learning of Finite State Models for Natural Language Processing. Ph.D.
thesis, COGS, University of Sussex.
Cardie C, Pierce D. 1998. Error-driven pruning of treebank grammars for base noun phrase identification.
In Proceedings of COLING-ACL ’98, pp 218–224.
Manning C, Sch¨utze H. 1999. Foundations of Statistical Natural Language Processing. MIT Press.
Tjong Kim Sang E, Buchholz S. 2000. Introduction to the CoNLL-2000 shared task: Chunking. In
Proceedings of CoNLL-2000 and LLL-2000, pp 127–132.
Charniak E. 1996. Tree-bank grammars. Technical Report CS-96-02, Department of Computer Science,
Brown University.
Charniak E. 1997. Statistical parsing with a context-free grammar and word statistics. In Proceedings
of NCAI-1997, pp 598–603.
Charniak E. 2000. Amaximum-entropy-inspired parser. In Proceedings of NAACL-2000, pp 132–139.
Schmid H. 2000. LoPar: Design and implementation. Bericht des Sonderforschungsbereiches
“Sprachtheoretische Grundlagen f¨ur die Computerlinguistik” 149, Institute for Computational Linguistics,
University of Stuttgart.
Ramshaw L, Mitchell M. 1995. Text chunking using transformation-based learning. In Proceedings of
the Third ACL Workshop on Very Large Corpora, pp 82–94. Association for Computational Linguistics.
Collins M. 1997. Three generative, lexicalised models for statistical parsing. In Proceedings of ACL
and EACL ’97, pp 16–23.
Johnson M. 1998. The effect of alternative tree representations on tree bank grammars. In Proceedings
of the Joint Conference on New methods in Language Processing and Computational Natural Language
Learning (NeMLaP3/CoNLL'98), pp 39–48.
CollinsM. 1999. Head-driven statisticalmodels for natural languageparsing. Ph.D. thesis, Department
of Computer and Information Science, University of Pennsylvania.
CollinsM. 2000. Discriminative reranking for natural language parsing. In Proceedings of ICML 2000.
Abney S. 1991. Parsing by chunks. In Berwick R, Abney S, Tenny C (eds), Principle-Based Parsing,
pp 257–278. Kluwer.
57

58
“But this formula doesn't mean anything!”
Ylva Berglund, Uppsala University
Oliver Mason, University of Birmingham
When working with quantitative measures of even simple statistics, one is sometimes confronted by
colleagues who fail to see the significance of such studies, claiming there was no meaning in a
statistical formula. Even when formulae are used to measure some textual parameter, the results may be
difficult to interpret. What does it actually mean if a text has an average word length of 4.67 and an
TTR of 0.34?
In this paper we start from earlier research into measuring the performance of non-native speakers (or
rather writers). We have previously shown (Proc of TaLC 2000, to appear) that automatic stylistic
assessment can be used to distinguish between different kinds of texts, both those of different genres
and, with even better results, native versus non-native authorship. Taking into account a variety of
‘surface’ parameters which were measured for essays written by Swedish learners of English at
University level, we will now look at a possible mapping between a number of numerical values and
the quality of the essay, which cannot be directly quantified itself. Using PCA and cluster analysis we
try to demonstrate what gives an English text the ‘Englishness', and how non-native speakers develop
their language ability in this ‘textual parameter space'. We also hope to show that even if a formula as
such does not mean anything by itself, quantitative measures are a valuable means that can be used to
enhance our understanding of intuitive judgements about texts.

59
Comparing cohesive devices: a corpus-based analysis of conjunctions in
written and spoken learner discourse
Roumiana Blagoeva
Department of English and American Studies
Sofia University St. Kliment Ohridski
1.The aims of this paper are: to describe the process of developing two learner corpora of English
at Sofia University; to mention some problems associated with the collection of the data as well as
some of the applications of corpora of this type, and to present the findings from the research carried
out so far.
2. The International Corpus of Learner English (ICLE) project, launched at the University of
Louvain in Belgium to compile an international corpus of written learner language was joined by the
Bulgarian team in 1996. To conform to the requirement for comparability all the teams participating
in the project are collecting the same type of data, the only difference between the sub-corpora being
the learners’ mother tongue. At present the Bulgarian sub-corpus contains about 112 000 words,
which is 55% of the total amount of data required, and work on it is still going on.
The data consist largely of argumentative essays written by Sofia University students at the beginning
of their second academic year, so they can be described as adult advanced learners of English. Each of
the 112 learners contributed about 1000 words on one or two of the essay topics suggested by the
ICLE members.
In 1995, a complementary project was conceived in Louvain to compile a corpus of spoken learner
language. The Louvain International Database of Spoken English Interlanguage (LINDSEI) project is
the first of its kind and was soon joined by several other countries, including Bulgaria. In order to
ensure comparability of the data each sub-corpus was to contain transcripts of 50 fifteen-minute
interviews with non-native university students of English. The Bulgarian sub-corpus has already been
compiled, and the transcription and keying-in of the data has been fully completed. The amount of
words collected approaches 110 000 including the speech of the interviewers, who are native speakers
of English.
3. A prime concern of this corpus creation activity is to collect data from a homogeneous
population. Among the variables that need to be controlled are learning environment, age, mother
tongue, stage of learning and nature of the task. The relevant biographical information about each
contributing learner, such as years of English at school, stay in an English speaking country,
knowledge of other foreign languages, use of reference tools, is encoded in a learner profile
questionnaire which learners fill in(Granger, 1994: 26).
The group of learners who contributed to the Bulgarian sub-corpora is homogeneous in terms of all
the variables listed above. The subjects are all second-year students of English and American Studies
at Sofia University; aged 20 to 22; the reference tools used when writing the essays are monolingual
dictionaries; their nationality, mother tongue and language spoken at home is Bulgarian with only 2
exceptions where one of the parents speaks Turkish at home; the medium of instruction at school and
at university has been English and Bulgarian; and very few students (only 6) have spent from four
months up to two years in an English speaking country.
Another essential consideration in developing the corpora is the representativity of the data to be
collected. In compiling the two Bulgarian sub-corpora there was no selection of contributors on the
basis of academic record or any personal preferences on the part of teachers and students. The essay
writing as well as the interviews, their transcription and conversion into electronic form were
incorporated in the students’ written and oral assignments. In this way all Bulgarian students of
English and American Studies at Sofia University could participate in the two projects and they did
their best to fulfil the tasks. There are inevitable differences in the previous learning experience of the
subjects, but I consider them an advantage rather than a hindrance because this fact practically
excludes the influence of one and only one teaching strategy on the learners, thus, rendering the data
representative of a much wider population than the one sampled. It should also be noted that since the
ratio between male and female students at the Department is 1:10, the number of female participants
60
is much higher (90%) than the number of male ones. If the two sexes were to be represented equally
the collection of data would have extended over a much longer period of time.
4. A learner corpus is very different from a native corpus because of the nature of the material
collected. A native corpus contains data from a natural language and can be used on its own for the
investigation of characteristic features of this language. A learner corpus presents evidence of an
interlanguage; and an interlanguage, regardless of its stages of development, can only be a simulation
of the natural language that is the target aimed at in the process of FLT. Therefore, any learner corpus
would be of little value on its own, but it could be a useful tool for investigating a particular
interlanguage when compared to a relevant native corpus. The choice of this native-speaker corpus is
dependent on the aims of FLT. In preceding decades comparisons were largely carried out between
learner language and the norm of the target language described in grammar books and dictionaries.
Isolated examples of TL and IL were analysed with the purpose of finding erroneous structures and
elements in the IL. So analyses tended to overlook the fact that learner language is not characterised
only by misuse, but also by over/underuse of words, syntactic structures and discourse features. This is
especially true for the highest levels of FLA, where errors are rare but we still feel that learners have
not achieved near-native competence and learner production still differs from what native speakers
would produce in similar situations. If the final goal of FLT/FLA is to achieve an ability to use the TL
the way it is used by native speakers for the fulfillment of certain real-life tasks then a study of
interlanguage will always need a corpus of authentic samples of the foreign language to compare with
learner production.
5. For this reason two computerized native-speaker corpora of nearly the same amount of data
were also developed at Sofia University. The written native language corpus is a collection of
newspaper articles and essays on various topics; the spoken native language corpus contains
transcripts of interviews, dialogues, announcements, extracts from radio programmes etc. All of the
texts in these two corpora are used as teaching materials in- or outside class, and in test papers. The
choice of text types and sources is justified by the fact that the most immediate contact Bulgarian
students have with the English language is established through teaching and the media rather than
personal contact with native speakers. Moreover, the students are trained to be specialists in English
language and culture and on graduating should be able to communicate in English in a variety of
situations as well-educated people. Therefore, a comparison of their production with the types of
native corpora mentioned is relevant for the present study.
6. Having such electronic collections of natural textual data enables us to carry out research on a
large scale and to explore characteristic features of interlanguages across a variety of backgrounds in a
quick and reliable way. Instead of analysing isolated, invented sentences we now have the opportunity
to explore language in use with its purposes and functions in mind.
“… for us the actual text, not the invented sentence, must be the essential linguistic unit … In the coming
millennium, this prospect can now finally be documented and clarified by working with very large corpora of
authentic texts, whereby we can hope to uncover some of the vital and delicate missing links between ‘language’
and ‘text'.” (Beaugrande, 2000)
As noted by Halliday (Halliday, 1976:2), “a text does not consist of sentences; it is realized by, or
encoded in sentences”. Hence, only the study of longer stretches of discourse can give an insight into
the resources that English has for creating texture.
7. Some of the first searches applied to the Bulgarian sub-corpora are connected to the use of
conjunctive elements by the learners. Conjunctions seem to be the most explicit way of establishing
relations in a text since they indicate how what is to follow is systematically connected to what has
gone before in the text (Halliday, 1976: 227). They are a means that text producers use “to exert
control over how relations are recovered and set up by the receivers” (Beaugrande, 1983: 74).
Conjunctive elements with their intrinsic function to signal relations in a text are rather different from
the rest of the lexicon in a language: they are relatively independent of context in the sense that we
can expect them to be present in a text no matter what the particular topic of this text is. The study of
their use by foreign learners of English can provide the first step to the understanding of the ideas that
learners have about text structure and to the investigation of their ability to construct a text.
8. The present analysis examines fifty words and phrases that, according to Halliday's (Halliday:
242-243) classification of conjunctive relations express additive, adversative, causal and temporal
relations, in a text (and, being one of the most frequently used additive conjunctions, is the subject of
61
a separate study and is excluded from this analysis). Using WordSmith Tools (Scott, 1997),
concordances were produced for the fifty conjunctions in each of the four corpora. In Corpus 3,
containing the learner spoken data, the interviewers’ words were manually excluded and only the
interviewees’ speech was taken into account. This slightly reduced the size of this corpus so the
frequency of occurrence of each item was calculated as a percentage of the total number of tokens in
the corpora. The results of this first search are summarised in Table 1.
Percentage of occurrences in each corpus
Number of
conjunctions
studied
Corpus 1
Learner written
100 000 words
Corpus 2
Native written
100 000 words
Corpus 3
Learner spoken
70 000 words
Corpus 4
Native spoken
100 000 words
50
3,627
1,894
4,328
3,364
Table 1. Frequency of occurrence of 50 cohesive devices in four corpora
8.1. There are two striking facts that these figures reveal: first, the greater use of connectors by
both native speakers and learners in speaking than in writing; and, second, a clear overuse of
conjunctions by learners in written and spoken production respectively in comparison with native
speakers.
8.1.1. A number of scholars studying spoken language (e.g. Labov, 1972; Chafe, 1979; Cicourel,
1981; Goffman, 1981; Biber, 1995) have reached the conclusion that conjuncts are more formal and
therefore more typical of written rather than spoken language because the speaker is usually less
explicit than the writer. It is true that the speaker can resort to a number of means of expression such
as gestures, posture, tone, etc., but it is equally true that in most situations the speaker is under the
pressure of time-limitations as he observes his interlocutor and has to respond quickly to his reactions,
sometimes modifying what he is saying and making it clearer and more concise. The writer, on the
contrary, can construct a text at leisure, can spend more time on the choice of particular words and
syntactic structures, or can go over the whole text and edit it in many different ways. This allows the
writer to avoid certain unnecessary repetitions (including the repetitions of formal connectors) and to
find other means of structuring his text in a logical way.
Whatever the reason, I am fully aware of the fact that working with corpora of 100 000 words each
cannot give a detailed picture of the existing state of written and spoken language. What this study
presents is based only on the observations of the raw data from the discourse stretches in the particular
corpora under investigation. I believe that the extension of the data and further analyses and
collaboration with the other teams may explain this phenomenon.
8.1.2. In trying to provide reasons for each deviant use of the target language by learners we most
often turn to the factors influencing FLA and the psycholinguistic processes central to second
language learning as determined by Selinker (1972: 37).
The overuse of conjunctions by learners of English could be due to some teaching/learning strategies.
In most classrooms, and more specifically in Bulgaria, speaking and writing are distinguished as
different skills and are trained separately and very often independently. Teaching materials used in
English classes and in the special writing courses place too great an emphasis on text structure and
the importance of formal connectors in general while at the same time leave very little space for other
ways of achieving coherence. This may lead learners to overgeneralization of some TL rules and may
bring them to the conclusion that a well-written and logically structured text can be produced only if
such language items are abundantly present in it. Such an idea results in a tendency on the part of
learners to “paste” connectors mechanically to the texts they produce in an attempt to give them some
“shape” (Blagoeva, 2000: in press).
By contrast, the development of speaking skills usually goes along the lines of free discussions on
suggested topics. Little instruction is given on how to speak coherently and to participate effectively in
the act of communication. Greater attention is paid to grammatical accuracy and pronunciation as
well as to knowledge connected with the topics discussed. It should be noted, however, that the
contributors to the two learner corpora are highly influenced by long and constant exposure to written
forms of language. Since our Department offers simultaneous courses in Linguistics and English and
62
American Literature the students’ work is mostly directed towards reading and writing tasks. This
probably accounts for the overuse of conjunctive elements and the more formal register they use in
their speech.
9. The overall overuse of conjunctions by the learners could be misleading if concrete examples
were not examined in greater detail. Table 2 presents the occurrences of the top 15 connectors in
descending order of frequency.
Percentage of occurrences in each corpus
Top 15 Conjunctions Corpus 1
Learner written
100 000 words
Corpus 2
Native written
100 000 words
Corpus 3
Learner spoken
70 000words
Corpus 4
Native spoken
100 000 words
But 0,632 0,426 0,988 0,632
Because 0,203 0,062 0,534 0,238
So 0,132 0,033 0,432 0,429
However 0,108 0,032 0 0,012
Then 0,107 0,090 0,234 0,299
For instance \ For example 0,098 0,017 0,062 0,058
Though \ Although 0,091 0,077 0,058 0,072
Of course 0,066 0,020 0,074 0,107
Therefore 0,056 0 0 0,020
Thus 0,050 0,002 0,004 0
Rather 0,040 0,016 0,020 0,025
Actually 0,035 0,013 0,235 0,135
In fact 0,033 0,014 0,122 0,059
Yet 0,022 0,015 0,003 0,010
I mean 0,006 0 0,245 0,144
Table 2. Frequency of occurrence of the top 15 connectors
9.1. Even a cursory glance at the first two columns shows that the greater number of instances of
conjunctions in the written learner production are almost evenly distributed among all the 15 most
frequently used connectors. In a recent paper Blagoeva (2000: in press) also reports that the results
obtained from the Bulgarian written learner data confirm the findings made by Granger (1994: 27-28)
about the overuse of the same connectors in the writing of French learners. (For further details see
Blagoeva, 2000: in press)
9.2 The data extracted from the spoken corpus, however, demonstrates distribution of connectors
in the speech of the learners different from that of the native speakers. Two of the connectors, so and
then, have nearly the same ratio in Corpus 3 and Corpus 4 probably because they are also perceived by
the learners as pause fillers. But, because, in fact, I mean, actually seem to be favourites of Bulgarian
speakers of English and it is quite understandable if NL interference as a factor influencing learners’
choices of words is considered. The English in fact can introduce “a contradiction or an opinion
which is different from something that has just been said” (COBUILD,1994) and is therefore
classified as adversative, contrastive conjunction (Halliday, 1976: 242-243 ). The Bulgarian
translation equivalent of in fact, YV &#227;WQRVW, means only in reality and is used to introduce some
clarification or to add details to a previously mentioned statement (Bulgarian Language Dictionary,
1994). In this way it functions rather as an additive, and since this is the more frequent relation in a
text it is quite natural to find more instances of this connector in the Bulgarian learner speech.
The other conjunctions that show greater frequency in the Bulgarian-English interlanguage have
functions similar to those of their translation equivalents in the NL of the learners. But in English and
no in Bulgarian, for example, have the same contrastive adversative function; because and zaštoto in
English and Bulgarian respectively “introduce the reason for a statement or the answer to a ‘why’
question” and are both causal conjunctions. There seems to be no place for confusion here and it is
true that no instances of misuse were encountered. Still NL interference could work not only because
of formal differences between languages but also through the transfer of different speaking or writing
habits in the mother tongue due to some cultural differences. One hypothesis at this point is that
Bulgarians tend to be more explicit when stating reasons or objections. Yet such a conclusion could be
63
confirmed only after comparisons with relevant Bulgarian native corpora, which are still being
compiled at Sofia University.
At the same time several connectors on the list point to a tendency for underuse in learner speech.
Obviously, some language items are used at the expense of others whenever students do not feel
confident enough in their knowledge of the foreign language.
10. One could argue that the higher or lower frequency of formal connectors in learner language,
as long as they are used correctly, may not lead to serious communication breakdowns. Still, it is my
view that it could interfere with a receiver's comprehension of a text and could contribute to the
artificiality of learner English. At an advanced stage of FLA students should be made aware that they
tend to stick to some language structures and should be encouraged to turn to other means of
achieving cohesion.
Another salient point is connected with the differences between spoken and written language and the
choice of teaching materials for the development of speaking and writing skills. By revealing
characteristic features of learner language, corpus analysis studies offer ways of diagnosing the true
learners’ needs for the different purposes of communication. The results of such studies can turn the
attention of teachers to the fact that speaking is not equivalent to mere talking but is a special skill
that can be trained in a systematic way. Naturally, further corpus-based investigation of other
discourse features is likely to be the way forward to developing learner resources that will bring
interlanguages closer to the kind of language used by native speakers of English.
References:
Andreichin L, et al 1994 % OJDUVNL7 ONRYHQ5H&#254;QLN>%XOJDULDQ/DQJXDJH'LFWLRQDU\@Sofia: Nauka
i izkustvo,.
Beaugrande R 2000 Text linguistics at the millennium: Corpus data and missing links.Text, 20, 2000.
Beaugrande R, Dressler W 1983 Introduction to text linguistics. London and New York: Longman.
Biber D 1995 Variarion across speech and writing. Cambridge; Cambridge University Press.
Blagoeva R 2000 Comparing cohesive devices: conjunctions and other cohesive relations and their
place in the Bulgarian-English interlanguage. Paper presented at Third international conference for
research in European studies, Veliko Turnovo, Bulgaria.
Chafe W L 1979 The flow of thought and the flow of language in (ed) T. Givon.
Cicourel A 1981 Language and the structure of belief in medical communication in (eds) B. Sigurd
and J. Startvik, Proceedings of AILA 81 Studia Linguistica 5: 71-85.
Goffman E 1981 Forms of talk. Oxford: Basil Blackwell.
Granger S 1994 The learner corpus: a revolution in applied linguistics. English Today 39: 25-29.
Halliday M A K, Hasan R 1976 Cohesion in English. London and New York, Longman.
Labov W 1972 Sociolinguistic Patterns Philadelphia: University of Pensilvania Press.
Scott, M. 1997. Wordsmith Tools. Version 2.Oxford: Oxford University.
Selinker L 1972. Interlanguage. International Review of Applied Linguistics 10: 209-31.
Sinclair J, ed. in chief. 1994. Collins Cobuild English Language Dictionary. London: HarperCollins
Publishers.

64
Frame Semantics as a framework for describing polysemy and syntactic
structures of English and German motion verbs in contrastive computational
lexicography
Hans C. Boas1
ICSI/UC Berkeley
This paper addresses the question of how to account for verbal polysemy from a contrastive point of
view. By examining the syntactic and semantic distribution of arguments of a selected number of
English and German motion verbs, I intend to demonstrate the usefulness of Fillmore's (1982) Frame
Semantics for describing verbal argument realization patterns across languages. In this connection it
will be shown that frame-semantic descriptions offer a unified way of relating the full range of lexical
units2 that instantiate the same semantic concept. In addition to these theoretical considerations,
practical applications of the frame-semantic approach to lexical organization are discussed.
1. Polysemy of English and German motion verbs
The data in (1) – (2) illustrate a small range of senses associated with the motion verbs run and walk
expressed in terms of distinct syntactic argument realization patterns. In (1a), run is used in a Selfmotion
sense to describe a situation in which a Self-mover (Julie) arrives at a Goal (the store) as the
result of her moving under her own power.3 In (1b), run is used in a Caused-motion sense to describe a
situation in which an Agent (Julie) causes a Theme (Pat) to end up in a location, in this case a Goal (off
the street).
(1) a. Julie ran to the store.
b. Julie ran Pat off the street.
(2) a. Rod walked to the door.
b. Rod walked Melissa to the door.
The semantics of walk in (2a) is similar to the semantics of run in (1a) in that it describes the motion
of a Self-mover (Rod) towards a Goal (the door). Following the terminology developed by Johnson et
al. (2001), I classify the usages of run and walk in (1a) and (2a) as Self-motion. Walk differs from run
in two respects. First, the manner of motion expressed by walk is of a slower nature than that expressed
by run.4 Second, the semantics of walk in (2b) differs from the semantics of run in (1b) in terms of
contact between the two event participants and their relation to each other. That is, whereas run in (1b)
incorporates a notion of force, (2b) does not. In contrast to the Caused-motion semantics attributed to
run in (1b), the Cotheme semantics associated with the use of walk in (2b) implies that the two event
participants (i.e., frame elements), Rod (the Self-mover) and Melissa (the Cotheme), started walking
together from an unmentioned common Source along an unmentioned Path to their final destination,
the Goal (to the door).
In German, the basic types of situations described by run and walk in (1a) and (2a) are typically
expressed by rennen and gehen, respectively. (3a) illustrates that the basic Self-motion sense of rennen
is the translation equivalent of the basic Self-motion sense of run in (1a). Note, however, that although
the basic Self-motion sense of run shows considerable semantic and syntactic overlap with the basic
Self-motion senses of rennen, there is no such overlap between run in (1b) and rennen in (3b).
(3) a. Tina rannte zum Geschäft.
Tina ran to the store
‘Tina ran to the store.’
1 The research reported here has been made possible by a postdoctoral fellowship from the “Deutscher Akademischer
Austauschdienst” (German Academic Exchange Service) under the “Gemeinsames Hochschulprogramm III von Bund und
Ländern” Program for conducting research with the members of the FrameNet project (http://www.icsi.berkeley.edu/~framenet)
(NSF Grant No. IRI-9618838, P.I. Charles Fillmore) at the International Computer Science Institute in Berkeley, California.
2 A lexical unit is a word in one of its senses.
3 For the sake of clarity, names of frame elements (i.e., semantic roles) and semantic frames are capitalized.
4 The difference in speed between run and walk is classified by Levin (1993: 265) as a difference in manner of motion. This leads
her to classify run and walk as “manner of motion verbs.“
65
b. *Tina rannte Enno von der Straße ab.
Tina ran Enno from the street off
c. Tina drängte Enno (beim Rennen) von der Straße ab.
Tina pushed Enno (while running) from the street off
‘Tina pushed Enno off the street (while running).’
A comparison between the Caused-motion sense associated with run in (1b) and the sentence in (3b)
shows that German rennen is not conventionally associated with a Caused-motion sense. The
phenomenon exemplified by the distribution of run and rennen in (1b) and (3b) is a case of what has
been called “divergence” in recent work on machine translation (cf., e.g., Dorr (1990) and Heid
(1994)). Divergences are cases in which different languages use different means to convey a given
meaning. In the case of German rennen, this means that the translation equivalent of the Causedmotion
sense associated with run in (1b) is expressed by a different type of verb in German, in this case
abdrängen in (3c).
A careful comparison of the Caused-motion meaning of run in (1b) with the meaning conveyed by
abdrängen in isolation in (3c) shows that the semantics of the two verbs do not exhibit an exact
overlap. That is, abdrängen without further specifications does not encode the manner in which the
Theme (i.e., Enno in (3c)) has been caused to move to its end location. Information about the manner in
which the caused-motion activity took place must be supplied by a separate phrase beim Rennen.
The comparison between the sentences in (1) and (3) shows that English and German verbs may differ
with respect to how the semantics of Caused-motion is lexicalized. Whereas English may supply its
motion verbs with a specific syntactic frame to express Caused-motion semantics, German does not
allow for such an option with rennen. The language provides a different type of verb to express
Caused-motion and leaves open the option of specifying the particular manner in which it happened.
Building on Talmy's (1985) terminology from work on motion expressions, I refer to the type of
realization of Caused-motion semantics exemplified by run in (1b) as construction-framed semantics.
That is, the abstract semantic concept of Caused-motion in (1b) is lexicalized in terms of a construction
specific syntactic frame occurring with the same verb as the basic sense, i.e., run. German abdrängen
in (3c) is an example of what I refer to as verb-framed semantics. In this case, the caused-motion
semantics is not lexicalized in terms of a specific syntactic frame occurring with the same lexical unit
expressing the basic sense. Instead, it is a lexicalized concept inherent to the semantics of a different
lexical unit, in this case abdrängen.5
Turning to the German translation equivalents of the two senses of walk in (2) above note that the use
of gehen in (4a) exhibits the same basic Self-motion sense as walk in (2a). A comparison between (2a)
and (4a) illustrates that in contrast to walk which is associated with a construction-framed Cotheme
semantics, gehen does not exhibit this pattern. Instead, German forces the use of a different verb,
namely begleiten, in (4c), to express the Cotheme semantics exhibited by walk in (2b).
(4) a. Bernd ging zur Tür.
Bernd walked to the door
‘Bernd walked to the door.’
b. *Bernd ging Anna zur Tür.
Bernd walked Anna to the door
c. Bernd begleitete Anna zur Tür.
Bernd accompanied Anna to the door
‘Bernd accompanied Anna to the door.’
The difference in lexicalization patterns of Cotheme semantics in (2) and (4) shows parallel properties
to the differences in lexicalization patterns of Caused-motion semantics observed above in (1) and (3).
That is, whereas the Cotheme semantics is lexicalized in terms of a construction-framed semantics with
walk, German chooses to lexicalize the translation equivalent in terms of a verb-framed semantics by
employing the verb begleiten (cf. (4c)).
This section has shown that English and German motion verbs may differ with respect to how abstract
semantic concepts are lexicalized. It has been shown that English and German motion verbs display
different types of polysemy networks, i.e., they are not all associated with an equal amount of different
semantic concepts. The following section addresses the issue of describing the similarities and
differences exhibited by run, walk, rennen, and gehen with a set of devices that allows for crosslinguistic
abstractions across different lexicalization patterns.
5 To be more precise, the Caused-motion semantics is already lexicalized in the German verb drängen. In this case, the separable
prefix ab serves as a preverb responsible for specifying the path and goal of the caused-motion semantics.
66
2. The role of Frame Semantics in contrastive lexicography
Most traditional approaches to lexicographic descriptions regard the notion of headword as central to
the organization of dictionaries and list the different senses associated with a headword in a lexical
entry. For each of the individual senses associated with a headword, traditional dictionaries list
information about its meaning, its usage, register, etc. (cf. Atkins 1995: 26). Whereas this approach to
documenting polysemy of lexical units typically centers around a dictionary example for each sense of
a word in order to exemplify the context in which it is used, here I explore an alternative type of lexical
organization for bilingual polysemy structures of the type illustrated in (1)-(4) above. Adopting ideas
from previous work on lexical organization (e.g., Fillmore & Atkins (1992), Heid (1994), Atkins
(1995), and Fontenelle (2000)), I propose that the different senses of English and German motion verbs
are related to each other in terms of frame semantic descriptions.
2.1 Frame Semantics
Charles Fillmore's (1982) Frame Semantics centers around the idea that in order to understand the
meanings of words in a language one must first have knowledge of the semantic frames, or conceptual
structures, that underlie their usage. Frames serve as a type of cognitive structuring device that provide
the background knowledge and motivation for the existence of words in a language as well as how they
are used in discourse.6 Fillmore's most often cited example of a frame is the commercial transaction
frame which involves a scenario with different frame elements such as Buyer, Seller, Goods, and
Money which participate in a commercial transaction. In this frame,
“one person acquires control or possession of something from a second person, by agreement, as a result of
surrendering to that person a sum of money. The needed background requires an understanding of property ownership,
a money economy, implicit contract, and a great deal more.” (Fillmore & Atkins 1992: 78)
Lexical units belonging to this frame are verbs such as buy, sell, spend, or charge, nouns such as
price, goods, or money, and adjectives such as cheap and expensive. While all of these lexical units
belong to the same semantic frame (the commercial transaction frame), a specific choice of a lexical
unit reveals a particular perspective from which the commercial transaction frame is viewed. Consider
the following examples.
(5) a. Miriam bought a book (from Collin) (for $20).
b. Collin sold a book (to Miriam) (for $20).
Both sentences in (5) describe the same commercial transaction event but from different perspectives.
Whereas (5a) views the transaction from the viewpoint of the buyer, (5b) views the transaction from
the perspective of the seller. The main point is that the two verbs buy and sell both refer to the same
underlying frame and evoke the same type of underlying knowledge about commercial transaction
events. Note also, that the syntactic realization of the individual frame elements differs depending on
the type of verb employed to describe the commercial transaction event. Whereas buy requires
syntactic realization of the frame elements Buyer and Goods, the Seller and the Price do not have to be
realized syntactically as is indicated by parentheses. Sell requires the Seller and the Goods to be
realized at the syntactic level, but leaves the realization of the Buyer and the Price an option.
A complete frame semantic description of a lexical unit belonging to the commercial transaction
frame includes not only information about the types of frame elements that make up the underlying
frame, but also information about how these frame elements are realized at the syntactic level. For
example, the lexical entry for the verb buy includes information about the fact that the frame element
Buyer has to be realized as a NP in subject position, whereas the frame element Goods has to be
realized as a NP in object position. The entry also records the fact that the frame elements Seller and
Price may optionally occur in postverbal position.
Capturing semantic and syntactic information about lexical units in terms of frame semantic
descriptions facilitates the creation of inventories of lexical units according to the types of frames to
which they belong. This type of lexical organization differs from that of traditional dictionaries in that
in a frame semantics dictionary the “concept of ‘headword’ becomes obsolete, for the whole frame is
the definiendum,” as Atkins (1995: 27) points out. Note also that the frame-semantic approach to
dictionary organization has practical advantages for the everyday dictionary user. By switching the
6 For a detailed survey of the main concepts underlying Frame Semantics, see Petruck (1996).
67
definitional burden from the level of the individual sense listed under the category of a headword to a
frame-semantic level, it becomes easier to understand the entire structured background of knowledge
that underlies the usage of a word. Since the description of a semantic frame also includes a list of the
words that evoke the frame, the dictionary user has access to the interrelationships that hold between
the class of words belonging to a common semantic frame. This means that understanding the meaning
of a single word based on a frame semantic description facilitates a more direct understanding of all of
the words belonging to the same frame.7
The next section illustrates the advantages of a frame-semantic approach to lexical organization for
describing the different polysemy structures of English and German motion verbs discussed above.
2.2 Describing contrastive polysemy structures
We are now in a position to contrast systematically the polysemy structures of English and German
motion verbs on the basis of frame semantic principles. In our frame semantic treatment of the data in
(1)-(4), there is one important feature that sets our approach apart from traditional approaches to
bilingual lexicography. That is, our lexico-semantic descriptions do not refer to the notion of headword
as the structuring element of our dictionary. This means that instead of referring to a specific sense of a
headword only, the description of a lexical unit will be stated in terms of a frame as structuring device.
By turning to this alternative level of lexical organization, it becomes possible to make higher order
generalizations about meanings of words across different languages.
In order to tease apart the relationships that hold between the individual senses of the verbs surveyed
in section 1, consider first the “basic” senses of English run and walk. As pointed out above, the usages
in (1a) and (2a) are instances of Self-motion in which “a living being (the Self-mover) moves under its
own power in a directed fashion.” (Johnson et al. (2001: 148)) Using the terminology of Johnson et al.
(2001), we identify in (6) the frame elements belonging to the Self-motion frame as follows. The frame
element Self-mover is a living being which moves under its own power (i.e., Julie in (6a) and Rod in
(6b)). The frame element Goal gives information about where the Self-mover ends up as a result of its
motion (i.e., to the store in (6a) and to the door in (6b)).8
(6) a. Julie ran to the store.
b. Rod walked to the door.
Capturing the information about the distribution of frame elements of the self-motion frame as they
are realized by run and walk in (6) results in partial sets of simplified frame-semantic descriptions as in
(7).9
(7) Partial frame-semantic descriptions of Self-motion senses of run and walk
a. runSelf-Motion [ Self-mover Goal ]
NP PP
b. walkSelf-Motion [ Self-mover Goal ]
NP PP
The simplified partial frame-semantic descriptions in (7) identify each verb as belonging to the Selfmotion
frame and give information about the syntactic realization of the two frame elements Selfmover
and Goal. Whereas Self-motion is realized as an NP, the Goal is realized as a PP with both
verbs. Next, consider the corresponding German verbs rennen and gehen in (3a) and (4a), repeated in
7 By including sets of example sentences exemplifying the use of a word in context, the dictionary user also has access to
information about the entire range of syntactic realization patterns of frame elements. In the Berkeley FrameNet database (for
detailed descriptions, see Lowe et al. (1997), Baker et al. (1998), Fillmore & Atkins (1998), and Johnson et al. (2001)), each
lexical entry includes corpus examples from the British National Corpus that have been annotated with semantic tags
representing frame elements (see Gahl (1998)).
8 Besides Self-mover and Goal, this frame also includes the frame elements Source, Path, Manner, Distance, and Area. For more
details, see Johnson et al. (2001: 148 – 150).
9 Note that in a full-fledged frame-semantic description of FrameNet, lexical entries include the entire range of corpus-attested
syntactic patterns exhibited by a lexical unit, including information about how frame elements are realized by each syntactic
pattern. They also include semantically annotated corpus sentences serving as examples (see Lowe et al. (1997), Baker et al.
(1998), and [http://www.icsi.berkeley.edu/~framenet] for details).
68
(8), and their corresponding simplified partial frame-semantic descriptions in (9) identifying them as
belonging to the Self-motion frame.
(8) a. Tina rannte zum Geschäft.
b. Bernd ging zur Tür.
(9) Partial frame-semantic descriptions of Self-motion senses of rennen and gehen
a. rennenSelf-Motion [ Self-mover Goal ]
NP PP
b. gehenSelf-Motion [ Self-mover Goal ]
NP PP
A comparison of the partial frame-semantic descriptions in (7) and (9) shows that all four verbs evoke
the Self-motion frame and exhibit the same type of syntactic realization of the frame elements Selfmover
and Goal. Taking our observations to a higher level of abstraction, we arrive at a generalization
about how the frame elements belonging to the Self-motion frame are realized by the four verbs in the
two languages as in (10).
(10) The Self-motion frame as a common structuring device for English and German
Self-mover
Goal
Self-Motion
runSelf-Motion [Self-mover Goal] rennenSelf-Motion [Self-mover Goal]
NP PP NP PP
The top of the diagram in (10) contains a fragment of the Self-motion frame and illustrates how the
individual English and German verbs instantiate the respective frame elements of that frame. The
arrows connecting the verb's individual frame semantic descriptions with the frame elements of the
underlying Self-motion frame illustrate the mapping between the frame elements of the Self-motion
frame and their syntactic realizations in the two languages. A comparison of the mapping properties
between the frame elements of the Self-motion frames and run and gehen in (10) shows that the two
verbs have identical mapping properties, i.e., they map the Self-mover as a preverbal NP and the Goal
as a postverbal PP. Similar observations can be made for the mapping of frame elements between walk
and gehen discussed in (7b) and (9b) above.
So far, it has been demonstrated how the frame elements of the Self-motion frame are realized in
similar ways by run, walk, gehen, and rennen. The next section turns to a discussion of cases in which
verbs from different languages exhibit different types of mapping of frame elements due to a difference
in lexicalization patterns of semantic frames.
In section 1 it was shown that whereas run is associated both with a Self-motion and a Caused-motion
frame, German rennen does not display a Caused-motion use parallel to that of run. Instead, German
offers a verb-framed lexicalization of Caused-motion (i.e., abdrängen) to describe those types of
situations that are expressed by the construction-framed Caused-motion sense of run. Adopting
Johnson et al.'s (2001: 132) terminology to describe the Caused-motion frame, we can say that “an
Agent causes a Theme to undergo directed motion” which may be “described with respect to a Source,
Path and/or Goal.” The frame elements relevant for describing the Caused-motion senses of run and
abdrängen include Agent, Theme, and Goal.10 The following diagram illustrates how these three frame
elements are realized by the Caused-motion sense of run and the Caused-motion sense of abdrängen.
10 Other frame elements included in the Caused-motion frame are Source, Path, Distance, and Area (cf. Johnson et al. (2001: 131-
133).
69
(11) The Caused-motion frame as a common structuring device for English and German
Agent
Theme
Goal
Caused-motion
runCaused-motion[Agent Theme Goal ] abdrängen Caused-motion [Agent Theme Goal ]
NP NP PP NP NP PP
(12) a. Julie ran Pat off the street.
b. Tina drängte Enno von der Straße ab.
Tina pushed Enno from the street off
‘Tina ran Enno off the street.’
Note that the frame-semantic descriptions of the Caused-motion senses of run and abdrängen in (11)
express similar types of scenarios as exemplified in (12). When comparing the similarities and
differences between the two diagrams in (10) and (11), we see that the notion of semantic frame offers
a convenient way of comparing and contrasting distributions of semantic concepts among different
lexical units.11 In particular, our frame-semantic descriptions give information about the use of run to
express both Self-motion as well as Caused-motion, and the use of gehen only to express Self-motion.12
The advantage of organizing a bilingual dictionary along frame-semantic concepts should be clear by
now. For example, users of bilingual dictionaries requiring information about how to express a specific
semantic concept in a different language, are offered multiple ways of gaining access to the
information.
The first possibility includes looking up a specific word in the target language in order to see whether
it may be used in the same pattern as the word in the source language. In the case of Caused-motion,
the dictionary user might expect that because run and rennen are associated with similar Self-motion
senses, the two verbs share a similar usage pattern when it comes to expressing Caused-motion. By
looking up the caused-motion sense of run, the dictionary user would then get to a description of the
Caused-motion frame and subsequently discover that there is no Caused-motion equivalent for rennen
but that he must use abdrängen. In this case, a frame-semantic description underlying a diagram such
as (11) allows the dictionary user to understand more easily the meaning of abdrängen, because it is
described in terms of the same type of underlying structuring devices, namely the Caused-motion frame
and its frame elements. Furthermore, by providing example sentences such as (12) which include
information about how the given sense of a word is used in context, the dictionary user gains full
access to usage examples from both languages.
The second way of accessing the desired information needed to express a given situation has to do
with cases in which a dictionary user is not completely sure about what types of words to use in either
language. Here, the frame semantic dictionary serves as a combination of dictionary and thesaurus. The
user is able to consult the frame-dictionary and look up lists of descriptions of semantic frames
including the types of words belonging to the frame. Based on the definition of frames and in
combination with examples illustrating the use of the individual words belonging to the frame, the
dictionary user can pick the word which best describes the situations in mind.
Take, for example, our comparison of walk and gehen in (2b) and (4b) above. We have seen that
whereas walk is associated with a construction-framed Cotheme semantics, gehen is not. To be more
precise, whereas walk is associated with a sense describing the motion of two distinct objects (Selfmover
and Cotheme) moving towards a goal, gehen does not allow for such a construction-framed
11 Note that abdrängen by itself does not characterize in detail the manner in which the Caused-motion is carried out. That is, it
only lexicalizes the force dynamics of Caused-motion (pushing). In contrast, run X off does not only lexicalize the abstract force
dynamic semantics of Caused-motion in terms of a syntactic frame, but it also gives information about the manner in which the
Caused-motion activity is carried out.
12 The proposals put forward in this paper are in contrast to many generative accounts of verbal polysemy such as Jackendoff
(1972), Pustejovsky (1995), and Rappaport Hovav & Levin (1998). These accounts typically suggest that verb meanings should
be split up into different verb class clusters and subsequently reduced to include only very minimal lexical semantic information.
In order to arrive at multiple verb meanings, these accounts propose to employ different sets of generative mechanisms to ensure
the generation of different verb senses and their related argument realization patterns from underspecified minimal lexical
entries. With respect to the range of application of generative accounts of verb meaning, Weigand (1998: viii) points out that “the
generative approach (...) reaches its limits in so far as the rule-governed mode-oriented approach, in principle, cannot tackle all
the varieties and idiosyncrasies of language use and therefore remains restricted to a subset of examples.”
70
association with the semantics of the Cotheme frame. Instead, it calls for a different verb, begleiten, to
express the same type of Cotheme semantics. This is illustrated by the following diagram.
(13) The Motion-Cotheme frame as a common structuring device for English and German
accompanyCotheme [S.-mover Cotheme Goal ] begleitenCotheme [S.-mover Cotheme Goal ]
NP NP PP NP NP PP
Self-mover
Cotheme
Goal
Cotheme
walkCotheme [Self-mover Cotheme Goal ]
NP NP PP
(14) a. Rod walked Melissa to the door.
b. Rod accompanied Melissa to the door.
c. Bernd begleitete Anna zur Tür.
‘Bernd accompanied Anna to the door.’
At its center, diagram (13) contains a partial set of frame elements from the Motion-Cotheme frame.13
The arrows linking the frame-descriptions of the individual verbs to the Motion-Cotheme frame
indicate that each of the three verbs are associated with the semantics of the Motion-Cotheme frame, as
well as how the frame elements are realized by each verb respectively.
Coming back to the problems that a dictionary user is faced with, we might consider a German
speaker who wishes to describe a Motion-Cotheme scenario. By consulting the frame-index, the
speaker looks up the Motion-cotheme frame description and sees two possibilities for expressing such a
scenario in English. This is the point where example sentences exemplifying the usage of the respective
words in the frame become crucial. For example, when choosing between the construction-framed
lexicalization pattern of the Motion-Cotheme frame with walk or the verb-framed lexicalization pattern
of the Motion-Cotheme frame with accompany, the dictionary user may want to emphasize the fact that
the Motion-Cotheme scenario was carried out by means of walking. In this case, he chooses the
construction-framed lexicalization of the Motion-Cotheme semantics with walk (cf. (14a)). In contrast,
if the speaker chooses to remain silent about the manner of motion, he chooses the verb-framed
lexicalization of the Motion-Cotheme semantics with accompany (cf. (14b)) which exhibits the same
type of Motion-Cotheme lexicalization pattern as begleiten (i.e., verb-framed) (cf. (14c)).14
The third possibility of gaining access to information about how a specific concept is expressed in a
language is by making reference to individual frame elements. For example, when the dictionary user is
interested in expressing information about a person's motion, he may look up the definition for the
frame element Self-mover. Based on this definition, the dictionary user has automatic access to all
semantic frames that include this frame element in their frame description. For the example frames
discussed in this paper, this means that referring to Self-mover offers direct access to the lexical items
belonging to the Self-motion (e.g., run, walk, rennen, and gehen), Caused-motion (e.g., run, walk, and
abdrängen), and Cotheme-motion (e.g., walk, accompany, and begleiten) frames, among many others.
Offering multiple ways of accessing information about the distribution of lexical items in a bilingual
dictionary demonstrates the usefulness of a frame-semantic approach for lexical organization. In
contrast to traditional dictionaries employing the notions of headword and isolated examples to guide
the dictionary user in finding the proper translation equivalent for a specific sense of a word, bilingual
13 The frame elements of this frame include Self-mover, Cotheme, Source, Path, Goal, Manner, Distance, and Area (cf. Johnson
et al. (2001: 133–135).
14 Note that although all three verbs walk, accompany, and begleiten are all associated with Motion-Cotheme semantics, they
describe the semantics of the Motion-Cotheme from different angles. That is, accompany and begleiten do not explicitly mention
the manner of motion, whereas the Cotheme use of walk makes explicit reference to the manner of motion. According to Snell-
Hornby‘s (1983: 33-35) classification of verb descriptivity, verbs like accompany and begleiten exhibit a variable degree of
descriptivity, whereas the Cotheme sense of walk exhibits a low degree of descriptivity. In this connection, see also Leisi‘s
(1975:77) discussion of “expressive verbs.”
71
dictionaries employing the notion of semantic frames facilitate the finding and understanding of lexical
items because the common device for understanding meaning is the whole semantic frame.
Using semantic frames as structuring devices also facilitates the learning of polysemous structures
since the types of polysemy exhibited by a word in the source language might not be mirrored by a
similar polysemy network of the corresponding word in the target language.15 By constructing bilingual
dictionaries along frame semantic guidelines and supplementing them with a large number of corpus
examples exemplifying the various usages of a lexical item in context, it may become possible to
circumvent a major problem for users of bilingual dictionaries pointed out by Snell-Hornby (1983:
215): “It is perhaps the stumbling-block of the conventional bilingual dictionary that it operates with
words in isolation, yet functions according to the principle of working equivalence, whereby a context
would be required.”
3. Practical applications for bilingual computational lexicography and educational tools
Employing frame-semantic principles for the construction of bilingual dictionaries requires effective
means of representing vast amounts of lexical information. Rather than being confined to traditional
representation tools such as printed materials, a dictionary design employing an electronic database
architecture will facilitate the representation and looking up of lexical units, their frame semantic
descriptions, and semantic relationships among frames (e.g., inheritance and blending (cf. Fillmore &
Atkins (1998)). A basis for building such a bilingual electronic database incorporating the main ideas
proposed in section 2 is the mono-lingual Berkeley FrameNet database for English.16 Without going
into the details of its full architecture (see Lowe et al. (1997) and Baker et al. (1998) for details), I will
briefly outline how some of its features may be incorporated into a computer-based bilingual framesemantic
database.17
At the center of such a database is the description of a frame, its frame elements, and the lexical units
belonging to that frame. For each lexical unit of a language, a lexical entry contains a “traditional”
definition combined with a frame semantic description with an exhaustive list of the semantic and
syntactic combinatorial properties. In addition, the lexical entry includes annotated corpus examples
exemplifying the syntactic valence patterns in which the frame elements occur. As mentioned above,
the database user has the possibility to access information about translation equivalents of a lexical
item in different ways. The first option consists of an alphabetic list of abstract frames that contains a
description of each frame as well as its frame elements and the lexical units for both languages that
participate in this frame. By clicking on a frame name, the user is informed of the frame description
and may then proceed to gather information about individual English and German lexical items that are
members of this frame.18
The second option of accessing information starts from an alphabetical list containing all lexical units
(different lists for English and German). By clicking on a lexical unit, the user will see its lexical entry
and may proceed from there either directly to the corresponding lexical unit(s) of the other language or
to the full description of the semantic frame. At the level of semantic frames, the user is informed of all
of the lexical units from both languages that participate in the frame and may choose the corresponding
lexical item of the other language from there.
Under the third option, as outlined in section 2.2, the dictionary user may access information about
lexical items and their translation equivalents by choosing a frame element from an alphabetical list of
frame elements in order to gain access to an overview of all of the frames in which the frame element
occurs. From this list, individual frame descriptions may be accessed for an overview of lexical items
that instantiate this frame element.
The various ways of accessing frame-semantic descriptions of lexical items and their translation
equivalents presented in this section constitute only a small set of options for representing lexical
information in a bilingual frame-semantic database. The important point is that in a database that is as
15 See, for example, our discussion of run and rennen in section 1. There we have seen that run is associated with both a Selfmotion
sense as well as a Caused-motion sense. In contrast, rennen is only associated with a Self-motion sense.
16 See http://www.icsi.berkeley.edu/~framenet
17 For similar proposals, see Fontenelle (2000) on bilingual lexical data bases combining frame semantics and Meaning-Text
Theory (Mel'cuk et al. (1984)). Heid (1994) presents an outline of the DELIS project which is aimed at developing lexicographic
tools for corpus-based lexicography. Within this project, Frame Semantics is employed to develop semantic descriptions of
lexicon fragments for English, French, Italian, Danish, and Dutch lexical items. The proposals put forward in this paper share
many theoretical and practical considerations underlying the architecture of DELIS.
18 An example of this is the simplified representation in (13) to which the dictionary user would get after clicking on “Motion-
Cotheme.” In addition to information regarding which lexical units belong to this frame, the database user may also have access
to full-fledged frame-semantic descriptions of the individual lexical items by clicking on them.
72
flexible as the type outlined above, the possibilities for accessing different types of relevant
information is manifold. Moreover, dealing with polysemous structures across languages is facilitated
because semantic frames offer a convenient way of structuring polysemy in terms of a unified
descriptive vocabulary.
Finally, consider the advantages a frame-semantic bilingual database can offer to the field of second
language acquisition, especially with respect to educational tools needed for foreign language
instruction. Traditional learning tools such as printed textbooks are limited in the size and scope of
exercises they offer to students. Incorporating a bilingual frame-semantic database into electronic
learning tools for foreign language pedagogy would not only offer students access to more efficient
ways of learning vocabulary by being able to relate to a common structuring device, i.e., semantic
frames. It would also give foreign language teachers the opportunity to design individual exercises for
students incorporating different types of pedagogically relevant information from the database that are
needed for specific learning tasks not covered by the standard learning software employed in the
classroom. Lastly, with semantically annotated example sentences from corpora, students would be
offered the opportunity of learning the vocabulary of a foreign language in context. Such an
opportunity would allow students to increase their understanding of the usage patterns of the respective
lexical items as opposed to simply learning vocabulary by memorizing lists consisting of only “words”
and their translation equivalents.19
4. Summary
This paper has outlined the theoretical and practical advantages of adopting Fillmore's Frame
Semantics for the description of different types of polysemy networks in English and German.20 By
examining the syntactic and semantic distribution of arguments of a selected number of English and
German motion verbs, this paper has shown how semantic frames can be employed as unified
structuring devices for bilingual dictionaries and databases. By describing lexical units with respect to
their underlying semantic frames, there is no need for the traditional notions of ‘headword’ or ‘basic
sense’ as organizational devices for structuring the lexicon. Frame Semantics can thus be regarded as a
true semantic metalanguage for both linguistic theory and applied lexicography, because it refers to
scenarios typically shared by speakers of all languages.
5. References
Atkins B 1995 The role of the example in a frame semantics dictionary. In Shibatani, M, Thompson, S
(eds), Essays in Semantics and Pragmatics in Honor of Charles J. Fillmore. Amsterdam/Philadelphia,
Benjamins, pp 25-42.
Baker C, Fillmore C, Lowe J 1998 The Berkeley FrameNet Project. In Proceedings of ACL/COLING
1998.
Dorr B 1990 Solving Thematic Divergences in Machine Translation. In Proceedings of the 28th Annual
Conference of the Association of Computational Linguistics.
Fillmore C 1982 Frame Semantics. In Linguistic Society of Korea (ed.), Linguistics in the Morning
Calm. Seoul, Hanshin, pp 111-138.
Fillmore C, Atkins B 1992 Toward a frame-based lexicon: The semantics of RISK and its neighbors. In
Lehrer A, Kittay E (eds), Frame, fields, and contrasts: New essays in semantic and lexical
organization. Hillsdale, Erlbaum, pp 75-102.
19 Since lexical entries contain semantically annotated corpus examples showing how a lexical item is used in context, a framesemantic
database would also greatly increase what Neubert (2000) calls the “five parameters of translational competence.”
These include “(1) language competence, (2) textual competence, (3) subject competence, (4) cultural competence, and, last but
not least, (5) transfer competence.” (Neubert (2000: 6))
20 The full range of polysemy networks of English and German motion verbs is, of course, much larger. Creating inventories of
the frame semantic descriptions of these verbs and comparing their different sense distributions is a task that is far beyond the
limits of this paper. For an example of a detailed study discussing the polysemy networks of two related verbs, see Fillmore &
Atkins (2000) for a comparison of English crawl and French ramper.
73
Fillmore C, Atkins B 1998 FrameNet and Lexicographic Relevance. In Proceedings of the First
International Conference on Language Resources and Evaluation, Granada.
Fillmore C, Atkins B 2000 Describing Polysemy: The Case of ‘Crawl.’ In Ravin Y, Laecock C (eds),
Polysemy. Oxford, Oxford University Press, pp 91-110.
Fontenelle T 2000 A Bilingual Lexical Database for Frame Semantics. International Journal of
Lexicography: 232-248.
Gahl S 1998 Automatic extraction of subcategorization frames for corpus-based dictionary-making. In
Fontenelle T, Hiligsman P, Michiels A, Moulin A, Theissen S. (eds), Euralex ’98 Proceedings. 8th
International Congress of the European Association for Lexicography. Li&#232;ge, Université de Li&#232;ge, pp
445-52.
Heid U 1994 Contrastive Classes – Relating Monolingual Dictionaries to Build an MT Dictionary. In
Kiefer F, Kiss G, Pajzs J (eds), Papers in Computational Lexicography – COMPLEX 1994. Budapest,
Research Institute for Linguistics/Hungarian Academy of Sciences, pp 115-126.
Jackendoff R 1972 Semantic Interpretation in Generative Grammar. Cambridge, Mass., MIT Press.
Johnson C, Fillmore C, Wood E, Ruppenhofer J, Urban M, Petruck M, Baker C 2001. The FrameNet
Project: Tools for Lexicon Building. Manuscript. Berkeley, CA, International Computer Science
Institute.
Leisi E 1975 Der Wortinhalt. Seine Struktur im Deutschen und Englischen. Heidelberg, Carl Winter.
Levin B 1993 English Verb Classes and Alternations. Chicago, Chicago University Press.
Lowe J, Baker C, Fillmore C 1997 A frame-semantic approach to semantic annotation. In Tagging Text
with Lexical Semantics: Why, What, and How? Proceedings of the Workshop. Special Interest Group
on the Lexicon, Association for Computational Linguistics, pp 8-24.
Mel'cuk I et al. 1984 Dictionnaire Explicatif et Combinatoire du Français Contemporain. Montréal,
Presses de l'Université Montréal.
Neubert A 2000 Competence in Language and Translation. In Schäfner C, Adab B (eds), Developing
Translation Competence. Amsterdam/Philadelphia, Benjamins, pp 3-18.
Petruck M 1996 Frame Semantics. In Östman J-O, Blommaert J, Bulcaen C (eds), Handbook of
Pragmatics. Amsterdam/Philadelphia, Benjamins, pp 1-13.
Pustejovsky J 1995 The Generative Lexicon. Cambridge, Mass., MIT Press.
Rappaport Hovav M, Levin B 1998 Building Verb Meaning. In Butt M, Geuder W (eds), The
Projection of Arguments. Stanford, CSLI Publications, pp 92-124.
Snell-Hornby M 1983 Verb descriptivity in English and German: A contrastive study in semantic
fields. Heidelberg, Carl Winter.
Talmy L 1985 Lexicalization Patterns. In Shopen T (ed.), Language typology and syntactic description.
Cambridge, Cambridge University Press, pp 57-147.
Weigand E 1998 Foreword. In Weigand E (ed.), Contrastive Lexical Semantics.
Amsterdam/Philadelphia, Benjamins, pp vii-ix.

74
Nouns and their prepositional phrase complements in English
Rhonwen Bowen
Department of English
Göteborg University
1. Introduction
This paper concerns one aspect of work in progress on a PhD thesis on the complementation of nouns
in English (See Bowen (forthcoming)). The purpose of the thesis is to give a description of nouns and
their complements which will prove of interest to linguists but will also be of interest, from a
pedagogical point of view, to learners of English. Although there are extensive surveys of the noun
phrase and its integral parts in grammars1 and research alike, the complementation of nouns is often
only discussed in connection with verb and adjective complementation2. Recent traditional grammars,
Greenbaum (1996) for example, discuss noun complementation more fully. Biber et. al. (1999:604-
656) discuss the different structural types of post-nominal patterns including: that-complement
clauses, exemplified here in (1); to-infinitival complement clauses shown in example (2); whcomplement
clauses shown in (3) and of + -ing constructions shown in example (4). Further, in Biber
et. al. (1999:634), prepositional phrases are included in postmodification and dealt with separately. In
the present study, however, the term complement is used to denote the postnominal elements which are
bound elements selected by a head noun, including both clausal complements as in examples (1) - (3)
and prepositional phrase complements (PPC) as in examples (4) - (7):
1) the fact that he will win
2) his need to win,
3) the decision whether it is worthwhile
4) the thought of losing the match
5) the involvement of the teacher
6) her reliance on drugs
7) their belief in the dollar
The focus of this paper is on the prepositional phrase complements illustrated in examples (5)
– (7). Moving away from prototypical complements like the examples above, this paper explores other
aspects of complementation. One consideration of interest is the type of noun which permits a PPC. In
other words, do only nominalisations allow PPCs or are other nouns found with complements and to
what extent do nominalisations "inherit" their complementation patterns? Another question addressed
in this paper is the type of preposition found heading a PPC. Also of interest are the number of
complements which are permissible with one noun and the question of the order of complements when
more than one is present. Finally, instances of postponement of the complements will be discussed.
These issues are addressed in this paper.
2. Method
In order to analyse and study the patterns of nouns used in English today, a corpus of authentic
sentences was collected. As a point of departure, all the nouns requiring all types of post-nominal
elements, i.e. both phrasal and clausal elements were extracted from two learners' dictionaries of
English. This extraction included 1904 nouns. As a qualitative rather than quantitative study of nouns
is part of the aim, a 25% sub-set of these nouns was systematically selected to form the basis of this
study. Sentences including examples of the subsequent 476 nouns have been extracted from the
CobuildDirect and BNC corpora. The frequency of the nouns in the corpora varied considerably which
entailed that for some of the nouns, a complete search was possible, e.g. seizure with 166 instances in
the CobuildDirect corpus, whereas for extremely frequent nouns such as place with 30,337 instances, a
percentage of the examples was randomly extracted.
1 See Quirk et. al. (1985:1238ff)
2 See Herbst (1988:266) for a discussion of previous research.
75
An analysis of the sentences was conducted with the help of a number of syntactic and
semantic criteria3 in order to establish complementhood. The distinction between complements and
adjuncts is far from clear cut and should be seen in terms of a cline. PP-adjuncts which are not bound
to the head noun are usually markers of description such as the wines of France or a book of poems or
PPs denoting place and time such as a book on the floor or the car in the garage. At the other end of
the cline, PPCs are typically bound to their heads as in a ban on smoking, an introduction to linguistics
or the request for money. Between these prototypical cases, the cline includes partitive complements
such as a chunk of cheese, and intermediate complements such as the cradle of free enterprise. For 65
of the nouns, the prepositional phrases functioned as clear cut adjuncts as in a blaze of colour or the
shank of the screwdriver and as such were excluded from the survey. The analysis showed that 411
nouns were found with PP-complements.
3. Which nouns take complements?
In order to determine if patterns of complement-taking nouns (CTNs) exist, the nouns were grouped
into three word formation categories. The first group comprises derived nouns which contain
distinguishable overt suffixes as in decision and ability and which have been labelled "derived". The
second group comprises nouns which have the same form as other word classes, i.e. they have no overt
suffixation as in hope, right and answer. These are referred to as either verb or adjective "linked"
nouns. The last group includes nouns which are non-derived, i.e. they have no evident connection to
other words classes, such as idea and fact. The reader must keep in mind that a division of this kind is,
however, problematic and that the focus of this work is on complementation and not morphology as
such. Therefore, there are some cases where the categorisation is difficult, e.g. proclivity with an overt
suffix but with no corresponding verb or adjective. The word formation division of the 411 corpus
nouns which take complements is illustrated in table 1:
Table 1 Morphological distribution of corpus nouns
Derived Nouns Linked Nouns Non-Derived
Nouns
196 118 97
Additionally, both the derived and linked categories of nouns have been further sub-divided into nouns
with deverbal as in reliance on, argument against, de-adjectival such as allergy to, evidence on or
denominal derivation such as relationship between. The derived and linked nouns together represent
77% of all the CTNs. The category of non-derived CTNs are also found with all types of PPCs such as
moratorium on, antidote to, clue to, spokesperson for.
In general, it can be said that the derived and linked nouns ”inherit” the same prepositions as
the verbs and adjectives from which they are related, e.g. adherence to, belief in and collision with. For
some nouns, such as ability which predominantly takes to-infinitival clause complements (cf. adjective
able to-v), a variety of PPCs have been found, e.g. the jumping ability of your animal; his ability at
mathematics; his ability for schoolwork and the children's ability on language skill. In a few cases
there are differences in complementation patterns such as the adjectives weak at/in/from/with and fond
of compared to weakness for and fondness for. The alternation between prepositions for one and the
same noun may also entail a change in meaning such as adoration of and adoration for.
4. Which prepositions head PPCs?
The following table illustrates a break-down of the nouns which take complements headed by different
prepositions. The nouns are divided into their morphological categories and the number of nouns is
supplied for each preposition. The right-hand column supplies the total number of nouns for each
individual preposition. It must be kept in mind that one and the same head noun can take more than
one type of PPC as in, the accusation of/by/against NP, or a search of/for/by NP.
3 Following, amongst others, Aarts (1997), Allerton (1982), Chomsky (1970), Collins (1991),
Haegeman (1994), Jackendoff (1977) and Radford (1988).
76
Table 2 Distribution of nouns and prepositions
Preposition Derived
Nouns
Linked
Nouns
Non-derived
Nouns
Total
of 186 121 96 403
for 36 27 21 84
to 35 15 11 61
by 39 12 1 52
on 25 14 4 43
from 21 18 1 40
against 17 16 3 36
between 18 11 7 36
in 18 10 6 34
with 15 7 4 26
about 13 6 1 20
over 4 13 2 19
towards 10 5 1 16
at 6 4 1 11
behind 1 - - 1
into 1 - - 1
It is noteworthy that of the 411 CTNs in the corpus, 403 or 98% take of-PPCs. The prevalence of of-
PPCs can be explained by the versatility of this preposition, i.e. that it is found across the full spectrum
of of-phrases. Its use is frequent in subject- and/or object-related complements but also in partitive
complements and intermediate complements. For the remaining prepositions, their use is not so much
a question of a cline but one which can involve subcategorisation as in solution to and dependence on
or where the meaning of the preposition is decisive a book on linguistics or a misunderstanding about
the money. The by-PPCs differ in that they only represent subject-related complements as in the ruling
by the court.
5. Number and preferred order of complements
For the vast majority of CTNs, only one complement is typically present. The PPC of a head noun may
represent subject- or object-related complements as in the love of God or be subcategorised by the noun
as in her belief in her daughter but also combinations of two or three complements do occur:
8) However, his accusation of corruption against the Prime Minister and her Cabinet members
results in the Government. . . (CobuildDirect times/10).
9) Thus the elevation of potters from the humble status of craftsmen to bona fide artists was
given its greatest boost in. . . (CobuildDirect ukmags/03).
In example (8) the head noun accusation is followed by two complements, an of-PPC and an against-
PPC, whereas in example (9) the head noun elevation has no fewer than three complements represented
by an of-PPC followed by a from-PPC and finally a to-PPC. In the literature, for example, Huddleston
(1984:261) states that "NPs with more than one complement are generally rather infrequent". The
corpus investigation shows that of the 411 CTNs in this study, 76 head nouns were found with double
complements. This figure may give the impression that double complements are fairly frequent, but
the 250 sentences found with this construction represent a very small section of the corpus material.
The number of complements found with a head noun is restricted partly by the type of head noun but
also by the length of the PPCs.
Having established that some nouns take more than one complement, it is of interest to
determine whether there is a preferred order between complements. In cases where two complements
are present, as in, the exclusion of members by the committee the object-related complement of
members precedes the subject-related complement, by the committee. This order is found in the
majority of double-complement sentences, however, examples in the corpus show that the order is by
no means fixed:
77
10) . . . the seizure by Moscow of the provinces of Bukavina and Bessarabia. (CobuildDirect
bbc/06).
11) The seizure of two vessels by armed Chinese security forces who. . . (CobuildDirect
oznews/01).
12) The Contact Group believes recognition of Bosnia by Mr Milosevic would . . . (CobuildDirect
oznews/01)
13) This success was due to a persistent advocacy and a recognition by others of his sincere
concern. (CobuildDirect ukbooks/08)
Other nouns found with alternating order of complements include: acquisition, assault, ban and
boycott, to name a few. All the nouns found with alternating order of complements were either derived
or linked head nouns; no non-derived nouns were found with more than complement.
From the corpus investigation, it appears that there are a number of influencing factors which
determine the order of complements. To begin with, it is difficult to postulate anything concrete about
text-type when the sentences are taken out of context, but it is obvious that the ordering of elements to
give focus or end-weight is a deciding factor particularly in journalese, for example. Also the length of
the PPCs appears to influence the order so that shorter PPCs tend to precede longer ones as can be seen
in examples (10) – (13).
6. Postponement of complements
The usual position of prepositional complements in English is adjacent to the head noun. Cases where
the complement has been moved rightwards to the end of the sentence do occur in the corpus, although
they are rather rare. Radford (1988:448) illustrates this phenomenon thus:
14) [A review -] has just appeared of my latest book.
Using Radford's terminology the dash in example (14) represents a "gap" which the extraposed PP has
left behind4. Examples from the corpus include:
15) A ban also has been introduced recently on the purchase of shares by senior officials although
residents say that many city leaders succeeded in getting rich before it took effect.
(CobuildDirect bbc/06).
16) The search is still going on for the two gunmen who carried out the shooting. (CobuildDirect
ukspok/04)
In example (15) the on-PPC has been moved rightwards from the head noun ban, and in example (16)
the for-PPC has been similarly moved away from search. In the case of example (16) the
postponement may have been used to avoid the imbalance of a heavy subject NP followed by a
relatively light predicate. In the case of example (15) this is not the case, however.
This construction is rare in the corpus with only 66 examples found. Not all the examples are
as clear as examples (15) and (16); a certain amount of ambiguity exists:
17) The ban was imposed on Florey, of Bracknell, Berks, at a Jockey club disciplinary hearing.
(CobuildDirect today/11)
18) And each time you spend on your card, a small donation is made to a worthwhile cause.
(CobuildDirect sunnow/17)
From the examples in the corpus, it can be concluded that the typical syntactic function of the NP in
these constructions is that of subject and that the majority of NPs were found with indefinite articles as
determiners.
7. Summary
4 Scholars differ in their approach to this phenomenon, see Culicover & Rochemont for a ”basegenerated”
model.
78
The above sub-sections show the complexity of noun phrases and their complements and give some of
the tentative results of this investigation. In summary, it can be said that approximately 80% of the
CTNs in this material were derived or linked to verbs and adjectives. This entails that a considerable
number of CTNs do take complements in spite of their lack of connection to other word classes. The
majority of head nouns ”inherit” the same complementation patterns as the verbs and adjectives to
which they are related. From the 411 nouns which have been found to take PP complements, 98%
were found with of-PPCs. Head Nouns are typically found with one complement, however, when both
subject- and object-related complements are present, factors such as end-weight and focus influence the
order. Few examples of PPC-postponement were found in the material but as the most common
function of the NP is that of subject, the use of postponement avoids instances of imbalance in a
sentence where a heavy NP subject is followed by a relatively light predicate.
References
Advanced Learners’ Dictionary of Current English. 1995 & 2000. Oxford, Oxford University Press.
Aarts B, 1997 English Syntax and Argumentation. London, Macmillan Press Ltd.
Allerton D.J, 1982 Valency and the English Verb. London, Academic Press.
Biber D, Johansson S, Leech G, Conrad S, Finegan E 1999 Longman Grammar of Spoken and Written
English. London, Longman.
Bowen R, (forthcoming) Noun Complementation in Present-day English.
Burnard L, 1995 The British National Corpus.
Chomsky N, 1970 Remarks on Nominalization in Readings. English Transformational Grammar. In
Jacobs R, and Rosenbaum P (eds), pp 184-221.
Collins Cobuild English Language Dictionary. 1987 & 1995 London, Harper Collins Publishers Ltd.
Collins P.C, 1991 Cleft and Pseudo-cleft Constructions in English. London, Routledge.
Culicover P.W, Rochemont M.S, 1990 Extraposition and the complement principle. Linguistic Inquiry,
21:23-47.
Greenbaum S, 1996 The Oxford English Grammar. Oxford, Oxford University Press.
Haegeman L, 1991 & 1994 Introduction to Government & Binding Theory. Oxford, Blackwell.
Herbst T, 1987 A Valency model for nouns in English. Journal of Linguistics (24): 265-301.
Huddleston R, 1984 Introduction to the Grammar of English. Cambridge, Cambridge University Press.
Jackendoff R, 1977 X-bar Syntax: A Study of Phrase Structure. Cambridge, Massachusetts, MIT.
Longman Dictionary of Contemporary English. 1987 & 1995 London, Longman.
de Mönnink I, 2000 On the Move The mobility of constituents in the English noun phrase: a multi
method approach. Amsterdam & Atlanta, Rodopi.
Radford A, 1988 Transformation Grammar A First Course. Cambridge, Cambridge University Press.
Sinclair J, 1987 The CobuildDirect Corpus.

79
From dictionary to corpus to self-organizing dictionary: learning valency
associations in the face of variation and change
Ted Briscoe
Computer Laboratory
University of Cambridge
ejb@cl.cam.ac.uk
http://www.cl.cam.ac.uk/users/ejb
1. Introduction
I use the term valency in an extended sense as a relatively theory-neutral term to refer to lexical
information concerning a predicate's realization as a single or multiword expression (such as a phrasal
verb), the number and type of arguments that a particular predicate requires, and the mapping from these
syntactic arguments to a semantic representation of predicate-argument structure which also encodes the
semantic selectional preferences on these arguments. Thus, I use the term valency (frame) to subsume
(syntactic) subcategorization and realization, argument structure, selectional preferences on arguments, and
linking and/or mapping rules which relate the syntactic and semantic levels of representation.
For example, the verb, believe, in its primary sense and one (rare) realization takes a NP subject, NP object
and infinitival VP complement where the subject is interpreted as the `believer', and therefore will typically
denote the kind of entity capable of belief, the object NP is interpreted as the subject of the infinitival VP,
and the proposition denoted by their composition is interpreted as `the belief', as in Most voters believe the
election to have been seriously mishandled. Most grammatical frameworks treat valency almost entirely as
a lexical property of predicates, although the inventory of valency frames, which varies between both major
syntactic categories and between languages, can be described somewhat independently of individual words.
Some theories, such as Construction Grammar (e.g. Goldberg, 1995) argue that the valency frame itself, or
`construction' contributes aspects of the overall meaning; for example, the dative frame is said to denote a
`change of possession': hence, The barman slid Fred the beer but not ?The barman slid the end of the table
the beer. Such examples raise complications for a theory of the association between valency frames and
predicates because it appears that the meaning of slide is coerced to entailing `change of possession’ by
virtue of insertion in the dative frame. Therefore a predicate's participation in alternant valency frames
can result in predictable modification of meaning.
Abstracting over specific lexically-governed particles and prepositions and specific predicate selectional
preferences, but including some `derived' / `alternant' semi-productive, and therefore only semipredictable,
bounded dependency constructions, such as particle or dative movement, there are at least 163
valency frames associated with verbal predicates in (current) English (Briscoe, 2000). In this paper, I will
review the work that my colleagues and I have done to learn (semi-)automatically this very large number
of associations between individual verbal predicates and valency frames.
Access to a comprehensive and accurate valency lexicon is critical for the development of robust and
accurate parsing technology capable of recovering predicate-argument relations (and thus logical forms)
from free text or transcribed speech. Without this information it is possible to `chunk’ input into phrases
but not to distinguish arguments from adjuncts or resolve most phrasal attachment ambiguities.
Furthermore, for statistical parsers it is not enough to know the associations of predicates to valency
frames, it is also critical to know the relative frequency of such associations given a specific predicate.
Such information is a core component of that required to `lexicalize’ a probabilistic parser, and it is now
well–established that lexicalization is essential for accurate disambiguation (e.g. Collins, 1997, Carroll et
al, 1998). While state-of-the-art wide-coverage grammars of English, capable of recovering predicateargument
structure and expressed as a unification-based phrase structure grammar, have on the order of
1000 rules, it is clear that the number of associations between valency frames and predicates needed in a
lexicon for such a grammar will be much higher.
80
2. The dictionary-based approach
In the mid-eighties automated corpus analysis was not good enough to derive useful descriptions of the
grammatical contexts in which predicates occurred, so we resorted to enhancing the valency information
painstakingly gathered by lexicographers and available to us in the machine-readable versions of advanced
learners’ dictionaries, such as the Longman Dictionary of Contemporary English (LDOCE) (Boguraev and
Briscoe, 1987; Boguraev et al, 1987). The resulting lexicon and subsequent similar efforts (e.g. Comlex
syntax distributed by the LDC, Grishman et al, 1994) have high precision (approx. 95%) but disappointing
recall (appox. 76% for ANLT, 84% for Comlex) which means that despite the large amount of
lexicographical and linguistic resources deployed 24-16% of the required associations between predicates
and valency frames were omitted for an open-class vocabulary of about 35,000 words (Briscoe and
Carroll, 1997). However, given that there are more than 60,000 associations between words and valency
frames (ignoring word sense differences) in the ANLT dictionary, perhaps the result is not so surprising.
Many of the omitted associations are quite unremarkable; for example combining the associations from
ANLT and Comlex would still leave the sentence types with seem in 1) (based on cited examples)
unanalyzed:
1a) It seemed to Kim insane that Sandy should divorce.
1b) That Sandy should divorce seemed insane to Kim.
1c) It seemed as though Sandy would divorce.
1d) Kim seemed to me (to be) quite clever / a prodigy.
1e) (For Kim) to leave seemed to be silly.
1f) The issue now seems resolved.
In addition, neither of these projects yielded (public-domain) lexicons which associated predicates (i.e.
word senses as opposed to word forms) with valency frames, nor did they record the relative frequency of a
frame given a specific word, nor did they fully specify the mapping to argument structure or the selectional
preferences on the arguments.
Though some of these limitations might be overcome by further work, inherently, the dictionary-based
approach is untenable, because such associations vary, both absolutely and also in relative strength,
depending on subject domain and genre (Roland and Jurafsky, 1998), and they change over time at a rate
consonant with lexical sense change rather than grammatical change. For example, swing has a core sense
of spatial motion with optional volitional / causative components, explaining its occurrence with locative
and/or source/goal locational arguments: Gay swung her into the saddle (after the LOB corpus). However,
in the financial domain, the extended sense of (share) price movements on a one-dimensional range is more
common: UAL's shares swung violently between an all time low of $6 and $36 in yesterday's trading (after
the WSJ). Sense specialization, such as that of swinger to mean sexually promiscuous, leads to new
valency associations: Kim swung with much of NY's literary society in the course of a long weekend (after
the Guardian). Therefore, at least the relative frequency of a valency frame varies depending on the relative
frequency of the sense of a word, and in many cases valency frames are different under sense extensions.
For example, with swing the selectional preferences and prepositional items are different for the financial
usages, while with the example of slide above, or the following attested example (due to Annie Zaenen)
She smiled herself an upgrade, the entire valency frame is only available under the extended sense.
3. The corpus-based approach
By the mid-nineties it was possible to reliably extract accurate phrasal analyses as well as part-of-speech
(PoS) tags from corpora, and thus to identify the realization of predicate forms in phrasal contexts (e.g.
Abney, 1996). That is, it became possible to `chunk’ PoS-tagged input into verb groups, bare
(unpostmodified) noun phrases, prepositional phrases, and so forth. Around this time, we developed a
system to acquire the associations between predicate forms and valency frames based on on our own
intermediate parsing technology (Briscoe and Carroll, 1997).
81
The system consists of :
·  A tagger, a first-order HMM PoS and punctuation tag disambiguator, is used to assign and rank
tags for each word and punctuation token in sequences of sentences (Elworthy, 1994).
·  A lemmatizer is used to replace word-tag pairs with lemma-tag pairs, where a lemma is the
morphological base or dictionary headword form appropriate for the word, given the PoS
assignment made by the tagger. We use an enhanced version of the GATE project stemmer
(Cunningham et al, 1995).
·  A probabilistic parser, trained on a treebank, returns ranked analyses (Briscoe and Carroll, 1993)
using a grammar written in a feature-based unification grammar formalism which assigns
‘intermediate’ phrase structure analyses to the PoS tag ‘lattices’ returned by the tagger (Briscoe and
Carroll,1995; Carroll and Briscoe, 1996).
·  A pattern extractor which extracts local syntactic frames, including the syntactic categories and
head lemmas of constituents, from sentence subanalyses which begin/end at the boundaries of
(specified) predicates.
·  A pattern classifier which assigns patterns to valency frames or rejects patterns as unclassifiable
on the basis of the feature values of syntactic categories and the head lemmas in each pattern.
·  A lexical filter which evaluates sets of frames gathered for a (single) predicate, constructing
putative lexical entries and filtering the latter on the basis of their reliability and likelihood.
For example, the sentence in 2a) is PoS-tagged as in 2b) and the parser returns the highest ranked analysis,
shown as a labelled bracketing in 2c). Assuming that we are acquiring an entry for attribute, the extractor
yields the pattern in 2d), which is classifiable as a known NP-NP-PSing valency frame with additional
lexical information such as the preposition and heads of the NP arguments and of the NP and VP
arguments of the PP, further parameterizing the frame hypothesized. In mapping from the analysis in 2b)
to a valency frame, the classifier is able to repackage information from the fairly simple X-bar PSG used by
the intermediate parser. In effect, the parser factors information between VSUBCAT and PSUBCAT
features and the classifier looks at these, and sometimes lexical information, to deduce, in this case, that
the appropriate frame is one in which the prepositional complement is a non-finite progressive clause.
2a) He attributed his failure, he said, to no-one buying his book.
2b) he_PPHS1 attribute_VVD his_APP$ failure_NN1 ,_, he_PPHS1 say_VVD ,_, to_II no-one_PN
buy_VVG his_APP$ book_NN2
2c) (Tp
(V2 (N2 he_PPHS1)
(V1 (V0 attribute_VVD) )
(N2 (DT his_APP$)
(N1
(N0 (N0 failure_NN1)
(Ta (Pu ,_,)
(V2 (N2 he_PPHS1)
(V1 (V0 say_VVD))) (Pu ,_,)))))
(P2
(P1 (P0 to_II)
(N2 no-one_PN)
82
(V1 (V0 buy_VVG)
(N2 (DT his_APP$) (N1 (N0 book_NN2)))))))))
2d) ((((he:1 PPHS1)) (VSUBCAT NP_PP) ((attribute:6 VVD) ((failure:8 NN1))
((PSUBCAT SING) ((to:9 II)) ((no-one:1 0 PN)) ((buy:11 VVG)))))
We call the level of analysis exemplified in 2c) `intermediate’ because the parser finds singly rooted trees
rather than simply chunking the input. However, many attachment decisions are `canonical'; for example,
the attachment of the comma-delimited text adjunct (Ta) he said as a nominal postmodifier is incorrect but
convenient for this application because it hides the text adjunct from the pattern extractor which computes
the local syntactic context of attribute. The parser returns the highest ranked analysis using a purely
structural probabilistic model for ranking alternative analyses. This makes training the parser on realistic
amounts of data and using it in a domain-independent fashion feasible. However, it also means that patterns
extracted from the highest ranked analysis are noisy because quite often the parser has no mechanism for
choosing the correct analysis. For example, the correct analysis for 3a) is shown in 3c) and the correct
analysis for 3b) in 3d).
3a) He looked up the word.
3b) He looked up the hill.
3c) (Tp (V2 (N2 he_ PPHS1) (V1 (V0 (V0 look_VVD) (P0 up_RP)) (N2 (DT the_AT) (N1 (N0
word_NN1)))))
3d) (Tp (V2 (N2 he_ PPHS1) (V1 (V0 look_VVD) (P2 (P1 (P0 up_RP) (N2 (DT the_AT) (N1 (N0
hill_NN1)))))))
However, the parser cannot reliably select between the derivations in 3c) and 3d) because it does not have
access to any lexical information such as the likelihood of look up being a phrasal verb or the differing
selectional restrictions on the NP as either PP or verbal argument.
The classifier rejects some noisy patterns which do not conform to the known valency frames for English.
However, many classifiable patterns are still incorrect for the reasons noted above, so the filter accrues
evidence for associations between the specified predicate form and specific valency frames. These are then
filtered using a statistical confidence test which utilizes the overall error probability that a particular frame
will be hypothesized, and the amount of evidence for an association of that frame with the predicate form in
question. Specifically, we think of occurrences of predicate forms with putative frames as a sequence of
independent (Bernoulli) trials and use the error probability, Pe
i, that a predicate form will be associated with
an incorrect valency frame, i, to formulate the null hypothesis. The probability of an event with probability
p happening exactly m times out of n such trials is given by the binomial distribution:
P(m,n,p) = n! ¸ m!(n - m)! Pm (1 - p)n -m
The probability of an event happening m or more times is:
P(m +,n,p) = Sk=m n P(k,n,p)
So P(m +,n Pe
i,) is the probability that m or more occurrences of valency frame i will be associated with a
predicate form occurring n times. The threshold for rejecting this null hypothesis was set to 0.05 yielding a
95% or better confidence that a high enough proportion of frames has been seen, given the underlying
error probability, to accept the hypothesis that the predicate form really is associated with the frame.
The error probability for a given frame i was estimated by:
Pe
i = ( 1 - (|pred in frame i| ¸ |preds|)) (|frames for i| ¸ |frames|)
where the counts for frames were obtained by running the parser and extractor software on the entire
Susanne corpus (Sampson, 1995) and the estimates of the numbers of predicates associated with frame i is
obtained by counting the number of predicates in the ANLT lexicon paired with that frame. Suppose that
83
the parser and extractor predict that verb tokens are associated with frame i &#190; of the time, and that only &#188;
of the verb types in ANLT are associated with i, then the error probability for associating verbs with i will
be slightly over half. If the frame is only hypothesized &#188; of the time but linked to verb types &#190; of the time,
the error probability will be one eighth. Note, however, that the estimate of the true incidence of the
association, being dictionary-based, doesn't take account of the relative frequency of tokens of the verb
types.
The performance of the system can be evaluated by recording true positives (TP) and true negatives (TN),
that is, cases where the filter correctly accepts or rejects an association between a predicate form and a
frame, and false positives (FP) and false negatives (FN), where the filter incorrectly accepts or rejects such
an association. The incidence of the four outcomes can be calculated either by comparing the lexical
entries produced by the system to a set of accurate lexical entries, or to a manually compiled set of entries
based on the same data that the system used to acquire the entries. The advantage of the former approach is
that it is quicker – we can create accurate entries by intelligently merging the information in the ANLT and
Comlex lexicons. However, the disadvantages are that we have no way of knowing whether the data the
system used actually exemplified all the frames in the resulting lexical entries and, since these lack
information about the relative frequency of frames for specific predicates, no way of knowing whether the
system has acquired accurate frequencies. Therefore, we supplement this method with manual analysis of
the data given to the system. This allows us to measure type recall and precision against the gold standard
entries and the corpus analysis and token recall and ranking accuracy against the corpus analysis, as defined
below:
Type recall: TP ÷ (TP + FN types in dict/corpus)
Type precision: TP ÷ (TP + FP types in dict/corpus)
Token recall: TP ÷ (TP + FN tokens in corpus)
Ranking accuracy: % pairs of TPs whose ranking by rel. Freq. is same as in corpus
For 14 pseudo-randomly chosen verbs and training data of between 200 and 1000 exemplars per verb, the
system achieved the results shown in the table below:
Dictionary (14 vbs) Corpus (7 vbs)
Type precision 65.7% 76.6%
Type recall 35.5% 43.4%
Token recall 80.9%
Ranking Accuracy 81.4%
These results are considerably worse than the (type) precision and recall results for ANLT and Comlex,
reported above, but nevertheless were promising. Analysis of the system behaviour showed that for
associations seen less than 10 times, the binomial hypothesis test performed no better than chance.
Therefore, much of our subsequent work has focused on how to improve the filtering and entry
construction component of the system.
4. Better filtering
Briscoe, Carroll and Korhonen (1997) demonstrated that iteratively optimizing the error probabilities used
in the above experiment on the basis of the correct entries resulted in a system yielding nearly 9%
improvement in type precision, over 20% improvement in type recall, and a 10% improvement in ranking
accuracy on a further 20 test verbs. This result shows that the dictionary-based estimation of error
probabilities is far from optimal. However, the other critical problem with the binomial hypothesis test --
that it deals poorly with low frequency events -- remains. Korhonen, Gorrell and McCarthy (2000)
compared results using the binomial test, the log likelihood ratio test and an empirically determined
thresholding scheme. A log likelihood test has been proposed by Dunning (1993) as appropriate for nonnormally
distributed data with many rare events. However, on this task the log likelihood test performed
significantly worse than the binomial, resulting in more FNs for high frequency frames, more FPs for
medium frequency frames and no improvement in performance on low frequency frames. The thresholding
84
scheme simply involved converting the hypothesized counts for each frame for a given predicate form into
a conditional probability for each frame given the predicate, P(framei | predicatej ) using the maximum
likelihood estimate (MLE, i.e. the ratio of the count for predicatej + framei over the count for predicatej )
and then rejecting any association with a probability lower than an empirically determined optimum on
non-test data. This resulted in a 24% improvement in precision and a 2% improvement in recall. However,
thresholding still results in many FNs for low frequency associations.
The reason why hypothesis testing does not work well for this task is that not only is the underlying
distribution Zipfian, but also there is very little correlation between the unconditional distribution on
valency frames independent of specific predicates and the conditional distribution of frames given a
specific predicate. For example, believe occurs mostly with a sentential complement, but the sentential
complement frame, in general, is rare. Therefore, any test that involves reference to the unconditional
distribution, filtering hypotheses about the conditional distributions, will perform badly. Unfortunately, the
same observation also undermines any attempt to use the unconditional distribution as the prior in a
Bayesian scheme or to smooth the conditional distributions to compensate for the known poor performance
of MLE on rare / unseen events. Korhonen (2000), compares the performance of the MLE and
thresholding scheme smoothed with the conditional distributions for a predicate related to those whose
associations the system is attempting to learn.
One simple approach to smoothing one distribution with another to infer the probabilities of unseen events
is linear interpolation (e.g. Manning and Schutze, 1999:218f). The conditional probability P(framei |
predicatej ) is smoothed using the conditional probability of P(framei | pre dicatek ) where pre dicatek
stands in some specified relationship to pre dicatej according to the formula below:
P(framei | predicatej   1 (P(framei | predicatej  2 (P(framei | predicatek))
ZKHUHWKH i denote weights which sum to 1, and can be optimized with held out data so that most of the
smoothed probability for P(framei | predicatej ) is determined by its MLE estimate. Korhonen (2000)
demonstrates that the distribution of frames given predicates is better correlated for hypernyms and
synonyms of target predicates (derived from WordNet) than for the unconditional distribution. She, then,
compares results for acquiring frames for 60 test verbs from 10 semantically-defined classes (based on
Levin's (1993) classification), using as a baseline MLE thresholding with no smoothing and comparing this
to linear interpolation of the MLE estimates with the unconditional distiribution for all verb frames and also
with the merged conditional distributions for 3 other verbs from the same class. Her results are shown in
the table below:
Baseline Unrelated smoothing Sem.-related Smoothing
Type Precision 78.5% 71.4% 87.8%
Type Recall 63.3% 64.1% 68.7%
Ranking Accuracy 79.2% 67.6% 84.4%
The measures shown are based on comparison with the corpus data used by the system. The baseline
performance is better than smoothing with the poorly correlated unconditional distribution. However,
smoothing against the conditional distributions of semantically related predicates results in a significant
improvement in performance. All but 3 of the 151 low frequency FNs rejected by MLE and thresholding
exceed the threshold after semantically-related smoothing, suggesting that this approach provides an
effective way of dealing with low frequency associations.
There is, of course, a cost to the smoothing approach, as it is necessary to obtain the smoothed distributions
for all the different classes of predicates exemplified in the English lexicon. However, based on analysis
and extension of Levin's (1993) classification, it seems likely that the total number of classes needed for
good performance on this task is unlikely to exceed 50, so it should be possible to seed such a system by
obtaining about 200 conditional distributions semi-automatically for smoothing purposes. The underlying
reason why this approach is effective is that similar predicates have similar, though by no means identical,
`paradigms’ of associations to valency frames, because they tend to undergo the same types of frame
alternation processes as other predicates in the same class.
85
This more semantically-driven approach to learning valency associations, seems to be getting us closer to
the tested performance of existing manually-derived valency dictionaries such as ANLT or Comlex, with
the added advantage that we recover the relative frequencies of these associations. However, it also raises
the issue of polysemy and the what precisely the associations are between. I argued initially that
associations hold between predicate senses and specific frames. However, our corpus-based work so far
has resulted in associations between target predicate forms and frames. In order to classify predicate forms
into semantic classes, Korhonen used the predominant sense of the form as defined by WordNet . However,
it is clear that the results obtained with the technique would be improved if it were possible to classify
occurrences of predicate forms into appropriate senses. Similarly, the corpus-based work so far has resulted
in the recovery of lists of lemmas which occur as the heads of arguments in frames with given predicates,
but the induction of selectional preferences from these lists requires their sense disambiguation too.
5. Predicate and argument sense disambiguation
The performance of word sense disambiguation systems (e.g. Wilks and Stevenson, 1998) has improved to
the point where it is viable to integrate such technology into our system. McCarthy (1997) and McCarthy
and Carroll (2000) develop a system which utilizes the WordNet semantic hierarchy on nouns and the lists
of head lemmas in argument slots in valency frames returned by the pattern extractor to infer a probability
distribution on semantic classes occurring in a given argument position in a given frame for specific
predicates. This probability distribution characterizes the selectional preference(s) of the predicate on that
argument.
The approach utilizes the minimum description length principle (MDL e.g. Li and Vitanyi, 1998) and aims
to find the smallest model over semantic classes which describes the data (list of head lemmas) most
succinctly. The head lemmas are assigned to WordNet semantic classes and counts of classes exemplified
for a particular frame argument are maintained and propagated through the hierarchy. If a lemma is
ambiguous between classes then the counts are evenly distributed between these classes. For example, for
the direct object argument in the transitive frame for eat, lemmas such as food and chicken might be seen.
In WordNet food is classified as a substance and is itself a semantic class food, while chicken is variously
classified as food, bird and person with only the food class itself classified as a substance. Therefore, the
counts would be divided and propagated so that food and substance each obtained 1 and 1/3 counts, while
bird and person obtained 1/3 each. These counts are turned into MLE estimates, but the low probability
classes are often filtered out using MDL since they add to model complexity without compressing the
description of the lemmas that occur.
The resulting selectional preference distributions have been used for WSD on nouns occurring in frame
slots with competitive results of around 70% precision (Kilgariff and Rosenzweig, 2000), and also to
identify whether a specific predicate participates in a frame alternation. For example, many verbs with a
causative component to their meaning participate in the so-called causative-inchoative alternation,
exemplified in 4).
4a) Kim broke the window.
4b) The window broke.
4c) Kim galloped the horse.
4d) The horse galloped.
Identifying the precise class semantically is difficult if not impossible (e.g. Levin and Rappaport, 1995),
probably because the alternation is semi-productive and partly driven by item familiarity (e.g. Goldberg,
1995). An alternative is to look for near identical selectional preference distributions on argument slots
between valency frames putatively related by such alternations. In this case, we would expect the direct
object slots in 4a and c) to share similar distributions to the subject slots in 4b) and 4d), respectively. This
approach to such alternations has several potential advantages from the perspective of learning valency
frames for predicates, such as more economic representation of valency lexicons, statistical estimation of
the semi-productivity of alternation rules and thus using them as a further aid in the induction of low
frequency frame associations (e.g. Briscoe and Copestake, 1999), and perhaps most importantly inferring
86
when the basic sense of a predicate will be systematically and predictably modified by association with a
frame, as with the example of slide and dative movement discussed in the introduction. McCarthy (2000)
reports results of an experiment with two alternation rules (causative-inchoative and conative) which
yielded a 75% accuracy rate at classifying predicates as participants in these alternations. Most errors were
FPs and it may be once again that polysemy of the predicate forms accounts for much of this noise. Most
recently McCarthy in unpublished work has begun to disambiguate verb forms into WordNet defined
senses using the distribution of nouns in argument slots which the verb form has been associated with. If
this work is successful, it should be possible to associate associations with predicate senses directly, and
thus to reduce some of the noise inherent in the work that relies on sorting predicates into semantic classes
before identifying the frames they are associated with, the selection preferences on these frames, and so
forth.
6. Related work
There seems to have been very little work done on corpus-based acquisition of valency frame associations
prior to the nineties. Seminal work was done by Brent (1991) establishing the importance of precise cues
and introducing the binomial hypothesis test. Manning (1993) extended this work by performing finite-state
text chunking before extracting patterns for frames. Ushioda et al (1993) were the first to attempt to extract
the relative frequency of different frames. These systems recognized a maximum of 23 frames and had
performance rates of around 80% token recall, in line with the original system we developed which
recognized 161 frames. More recent papers are Gahl (1998), Carroll and Rooth (1998), Lapata (1999),
Stevenson and Merlo (1999) and Zeman and Sarkar (2000). None of these utilizes a set of frames as large
as ours or reports results suggesting more accurate performance. Carroll and Rooth dispense with
hypothesis testing and use estimation maximization without thresholding, prefiguring the results of
Korhonen, Gorrell and McCarthy (2000) to some extent. Zeman and Sarkar also compared the log
likelihood ratio to the binomial hypothesis test and preferred the latter, but start from preparsed data. Lapata
and Stevenson and Merlo focus on learning which verbs participate in a small number of alternations.
7. The future: self-organizing dictionaries
The recent experiments we have undertaken are based on applying the pattern extractor to 20M words of
the BNC (Leech, 1992). There is no reason, in principle, for us not to parse more data, however, the overall
performance of the system would be improved if parse selection accuracy were better. Carroll, Minnen
and Briscoe (1998) demonstrate that using the valency frame associations acquired by our system to rerank
the derivations returned by the parser improves its accuracy by about 10%. Incrementally integrating this
and other lexical information into the probabilistic ranking of derivations is a current avenue of research.
Similarly, there are other incremental improvements that can and will be made to the different subsystems
being developed for identifying predicate senses, selectional preferences and alternation behaviour.
Nevertheless, no matter how much data is analysed however accurately, this data will still be inadequate
from a statistical perspective for the acquisition of an accurate and comprehensive valency lexicon. In the
limit, the arguments that I made against the dictionary-based approach also apply to the corpus-based
approach when it is applied to a finite corpus. Zipf (1949) demonstrated that several distributions derived
from natural language approximate to power laws in that the probability mass is distributed non-linearly
between types with a few of the most frequent types taking the bulk of the probability mass and a very long
tail of rare types. Both the unconditional distribution of valency frames and the conditional distributions of
frames given specific predicates are approximately Zipfian (e.g. Korhonen, Gorrell and McCarthy, 1998).
Two conclusions that can be drawn from this are: 1) that, because the power law is scaling invariant, any
finite sample will not be representative in the statistical sense, and 2) that power law distributions are very
often a clue that we are not sampling from a stationary source but rather from a dynamical system (e.g.
Casti, 1994). Baayen (1991) develops a dynamical model of word generation which predicts a Zipfian
distributed vocabulary. Briscoe (2001a) develops a more general model of language as a dynamical system,
the aggregate output of a changing population of partly heterogeneous generative grammars. From this
perspective it is not surprising that classical statistical models of learning, which rely on representative
87
samples from stationary sources, do not perform optimally. A better model of a valency lexicon, given
these observations, is of an adaptive self-organizing knowledge base which continually monitors data to
update associations and the strengths of associations between predicates and valency frames. The
techniques outlined in this paper will need to be improved before we will be able to meaningfully develop
such a model. However, such models can be developed as an extension of the statistical techniques used
here by augmenting these techniques with the ability to incrementally update and renormalize the
conditional distributions acquired from data (see Briscoe 2001b) for an implementation of Bayesian
incremental updating of parameter values.
References
Abney, S. 1996. Part-of-speech tagging and partial parsing. In Church, K., Young, S. and Bloothoft, G.
Corpus-based Methods in Speech and Language, Dordrecht, Kluwer.
Baayen, H. 1991. A stochastic process for word frequency distributions. Proc of 29th Assoc. For Comp.
Ling. 271-278.
Boguraev, B.K. and Briscoe, E.J. 1987. Large lexicons for natural language processing: utilising the
grammar coding system of the Longman Dictionary of Contemporary English, Computational
Linguistics 13.4, 219-240.
Boguraev, B.K., Briscoe, E.J., Carroll, J., Carter, D. and Grover, C. 1987. The derivation of a
grammatically-indexed lexicon from the Longman Dictionary of Contemporary English, Proc. of 25th
Assoc. for Comp. Ling., 193-200, Morgan Kaufmann, Palo Alto, CA.
Brent, M. 1991. Automatic acquisition of subcategorization frames from untagged text. Proc. of 29th
Assoc. For Comp.Ling., 209-214, Morgan Kaufmann, Palo Alto, CA.
Briscoe, E.J. 2000. Dictionary and System Subcategorisation Code Mappings, Ms. Computer Laboratory.
Briscoe, E.J. 2001a. Evolutionary perspectives on diachronic syntax, Diachronic Syntax: Models and
Mechanisms. (eds) Pintzuk, S., Tsoulas, G. and Warner, A.}Oxford University Press, Oxford.
Briscoe, E.J. 2001b. Grammatical Acquisition and Linguistic Selection. In Linguistic evolution through
language acquisition: formal and computational models. (ed.) Briscoe, E.J. Cambridge University Press,
Cambridge.
Briscoe, E.J. and Carroll, J.A. 1993. Generalized probabilistic LR parsing of natural language
(corpora) with unification-based grammars. Computational Linguistics, 19.1, 25-60.
Briscoe, E.J. and Carroll, J.A. 1995. Developing and Evaluating a Probabilistic LR Parser of Part-of-
Speech and Punctuation Labels, 4th Int. Workshop on Parsing Technologies (IWPT95) Morgan Kaufmann,
Palo Alto, CA.
Briscoe, E.J. and Carroll, J.A. 1997. Automatic extraction of subcategorisation from corpora. Proc. of 5th
Assoc. For Comp. Ling. Conf. on Applied Nat. Lg. Proc., Morgan-Kaufmann. Palo Alto, CA.
Briscoe, E.J. and Copestake, A.A. 1999. Lexical Rules in Constraint-based Grammars, Computational
Linguistics, 25.4, 487-526.
Carroll, J.A. and Briscoe, E.J. 1996. Apportioning development effort in a probabilistic LR parsing system
through evaluation. 2nd Conference on Empirical Methods in Natural Language Processing, 92-100,
Morgan-Kaufmann. Palo Alto, CA.
88
Carroll, J.A. and McCarthy, D. 2000. Word sense disambiguation using automatically acquired verbal
preferences. In Computers and the Humanities. Senseval Special Issue, 34, 1-2.
Carroll, J.A., Minnen, G., and Briscoe, E.J. 1998. Can subcategorisation probabilities help a statistical
parser? 6th Workshop on Very Large Corpora, 35-41, Morgan-Kaufmann. Palo Alto, CA.
Carroll, G. And Rooth, M. 1998. Valence induction with a head-lexicalized PCFG. 3rd Conference on
Empirical Methods in Natural Language Processing, Granada, Spain.
Casti, J.L. 1994. Complexification. Harper Collins, New York.
Collins, M. 1997. Three generative lexicalised models for statistical parsing, Proc of 35th Assoc. for
Comp. Ling. 16-23, Morgan-Kaufmann. Palo Alto, CA.
Cunningham, H., Gaizauskas, R. \& Wilks, Y. 1995. A general architecture for text engineering (GATE) -
a new approach to language R&D. Research memo CS-95-21, Department of Computer Science,
University of
Sheffield, UK.
Elworthy, D. 1994. Does Baum-Welch re-estimation help taggers? Proc. of 4th Conf. Applied Nat. Lang.
Processing, Morgan-Kaufmann. Palo Alto, CA.
Gahl, S. 1998. Automatic extraction of subcategorization frames from a part-of-speech tagged corpus.
Proc. of Assoc. For Comp. Ling. Morgan-Kaufmann. Palo Alto, CA.
Goldberg, A. 1995. A construction grammar approach to argument structure, Chicago UP.
Grishman, R., Macleod, C. and Meyers, A. 1994. Comlex syntax: building a computational lexicon. Int.
Conf. on Computational Linguistics, 268-272, Kyoto, Japan
Kilgariff, A. and Rosenzweig, J. 2000. English Senseval: report and results. Ms. ITRI, Brighton, UK
(www.itri.bton.ac.uk)
Korhonen, A. 2000. Using semantically motivated estimates to help subcategorization acquisition. Proc. of
Jnt. Conf. on Empirical Methods in NLP and Very large Corpora, Morgan-Kaufmann, Palo Alto, CA.
Korhonen, A., Gorrell, G. and McCarthy, D. 2000. Statistical Filtering and Subcategorization Frame
Acquisition. Proc. of Jnt. Conf. on Empirical Methods in NLP and Very large Corpora, Morgan-
Kaufmann, Palo Alto, CA.
Lapata, M. 1999. Acquiring lexical generalizations from corpora: a case study for diathesis alternations.
Proc. of 37th Assoc. For Comp. Ling. 397-404, Morgan-Kaufmann, Palo Alto, CA.
Leech, G. 1992. 100 million words of English: the British National Corpus. Language Research 28.1, 1-13.
Levin, B. 1993. English verb classes and alternations. Chicago Univ. Press, Chicago.
Levin, B and Rappaport, H. 1995. Unaccusativity, MIT Press, Cambridge, MA.
Li, M. and Vitanyi, P.1998. An Introduction to Kolmogoroff Complexity
and Its Applications, Springer-Verlag, Heidelberg.
Manning, C. 1993. Automatic acquisition of a large subcategorisation dictionary from corpora. Proc. of
31st Assoc. for Comp. Ling. 235-242, Morgan-Kaufmann, Palo Alto, CA.
89
Manning, C. and Schutze, H. 1999. Foundations of Statistical Natural Language Processing. MIT Press,
Cambridge MA.
McCarthy, D. and Korhonen, A. 1998. Detecting verbal participation in diathesis alternations. Proc. of 36th
Assoc. For Comp. Ling., Morgan-Kaufmann. , Palo Alto, CA.
McCarthy, D. 2000. Using semantic preferences to identify verbal participation in role switching
alternations. Proc. of 1st Nth. Am. Assoc. For Comp. Ling. Morgan-Kaufmann. , Palo Alto, CA.
Roland, D. and Jurafsky, D. 1998. How Verb Subcategorization Frequencies are Affected by Corpus
Choice. Proc. of 36th Assoc. for Comp. Ling., Morgan-Kaufmann, Palo Alto, CA.
Sampson. G. 1995. English for the Computer. Oxford University Press, Oxford.
Stevenson, S. and Merlo, P. 1999. Automatic verb classification using distributions of grammatical
features. Proc. of 9th Conf. Of Eur. Assoc. For Comp. Ling. 45-52., Morgan-Kaufmann, Palo Alto, CA.
Ushioda, A., Evans, D., Gibson, T. and Waibel, A. 1993. The automatic acquisition of frequencies of verb
subcategorization frames from tagged corpora. SIGLEX ACL Workshop on the Acquisition of Lexical
Knowledge from Text, Columbus, Ohio.
Wilks, Y. and Stevenson, M. 1998. Word sense disambiguation using optimised combinations of
knowledge sources. Proc. of 36th Assoc. for Comp. Ling., Morgan-Kaufmann, Palo Alto, CA.
Zeman, D. and Sarkar, A. 2000. Automatic extraction of subcategorization frames for Czech. Proc. of Int.
Conf. On Comp. Ling., 691-697, Saarbrucken, Germany.
Zipf, G. 1949. Human Behavior and the Principle of Least Effort. Addison-Wesley, Cambridge, MA.

90
Semi-automatic tagging of intonation in French spoken corpora
Estelle Campione & Jean Véronis
Equipe DELIC
Université de Provence
29, Avenue Robert Schuman, 13100 Aix-en-Provence (France)
Estelle.Campione@up.univ-aix.fr, Jean.Veronis@up.univ-mrs.fr
Abstract
The transcription of spoken corpora using the punctuation of the written language is far
from satisfactory. On the other hand, the manual transcription of prosody is an extremely
time-consuming activity, which requires highly specialised experts, and is prone to errors
and subjectivity. Automating the prosodic transcription of corpora can be interesting both
in terms of effort reduction, and in terms of objectivity of the markup. Full automation is
not achievable in the current state of the technology, but we present in this paper a
technique that automates critical steps in the process, which results in a substantial
annotation time reduction, and improves the objectivity and coherence of the annotation.
In addition, the necessary human phases do not require a highly specific training in
phonetics, and can be achieved by syntax students and corpus workers. The technique is
applied to French, but most of the modules are language-independent, and have been
tested on other languages.
Keywords: spoken corpora, annotation, prosody, intonation, French
1. Introduction
The transcription of spoken corpora is a difficult issue. It has been noted many times that transcriptions
using written language punctuation are unsatisfactory and misleading (Blanche-Benveniste & Jeanjean,
1987; Leech, 1997), since the set of written punctuations is far from parallel to that of prosodic
phenomena in speech. Leech (1997:90) calls the transcription of spoken language using ordinary
orthography (and written punctuation) “a pseudo-procedure the only excuse for which is that it would
be prohibitively expensive to attempt anything else”.
Because of this inadequacy, some teams like ours have developed transcription conventions that do not
make use of any of the written punctuations. For reasons of feasibility, these conventions usually mark
only a very limited subset of prosody phenomena. In our case, for example, a minimalist stance has
been taken and only pauses are marked (Blanche-Benveniste & Jeanjean, 1987; Blanche-Benveniste,
1990). However, this type of transcription is not entirely satisfactory either, because ambiguities appear
in the resulting “text”. In French, for example, most discourse markers can belong to another category
and fulfil another function. Very often meaning and context are not enough to find the correct
interpretation, which requires prosodic clues. In the example below, quoi can be a discourse marker
(more or less similar to you see or I mean), but also a pronoun (what). The unpunctuated fragment, je
ne sais pas quoi can therefore be interpreted either as I don't know what or as I don't know, you see:
écrire un un petit euh je sais pas quoi un petit recueil qui qui explique comment les étapes qu'il faut suivre
Another common ambiguity consists in “floating” segments, usually complements, that can be attached
to what comes either before or after in the utterance (see Bilger et al., 1997):
elle arrive moi je m'en vais &#224; une demi-heure pr&#232;s on travaille pas ensemble
In a noticeable proportion of cases, the interpretation spontaneously adopted by corpus users is the
wrong one. Sometimes they do not even notice the double reading.
91
The only satisfactory solution would be to faithfully transcribe the prosody of utterances1, but this was
attempted only for a handful of corpora (e.g. the London-Lund Corpus or the Lancaster/IBM Spoken
English Corpus). Prosodically transcribed corpora are totally lacking for most languages (such as
French). The reasons for this shortage lies in the difficulty of prosodic transcription stressed by many
authors. It is highly time-consuming and requires from the annotators a type of phonetic-oriented
competence that is not common among syntax scholars. In addition, the very subjective nature of
prosodic labelling reduces the trustworthiness of the results or requires careful control by counterexperts
thus increasing the cost yet more.
Automating the prosodic transcription of corpora can be interesting both in terms of effort reduction,
and in terms of objectivity of the markup. Full automation is not achievable in the current state of the
technology, but we present in this paper a technique that automates critical steps in the process, which
results in a substantial annotation time reduction, and improves the objectivity and coherence of the
annotation. In addition, the necessary manual steps do not require a highly specific training in
phonetics, and can be achieved by syntax students and corpus workers. The technique is applied to
French, but most of the modules are language-independent, and have been tested on other languages.
2. Overview
Many schools have developed prosodic theories and annotation schemes and there is no consensus on a
transcription system. It has even been said that each new monograph introduces a different coding
system (Hirst, 1979; Mertens, 1990:159). ToBI (Silverman et al., 1992) has gained a wide popularity
for American English, but it is not easy to adapt to other languages, even to other varieties of English
(Nolan & Grabe, 1997; Leech, 1997). ToBI labelling also relies on linguistic judgements made by
experts and is consequently difficult to carry out automatically, although research in that direction is
underway (Wightman & Ostendorf 1992; Ostendorf & Ross, 1997).
More importantly, ToBI-like systems are too detailed for many linguistic purposes, especially for
syntactic studies. Leech (1997:89) makes a distinction between spoken language corpora and speech
corpora, the former consisting of usually large, naturally occurring samples of continuous language or
discourse, the latter referring to “laboratory speech”, usually words or sentences out of context. While a
“narrow” transcription can be useful and even necessary for speech corpora, and phonetic studies, a
high density of fine-grained prosodic symbols (typically one or more per word) is likely to be difficult
to read and to blur the important facts in spoken corpora. As far as intonation is concerned, for
example, most of the smaller melodic movements are the result of local phonotactic rules and
contraints (such as the number of syllables in a word or the position of the lexical stress), and do not
reflect communicative choices from the speaker (for French, see for example Hirst & Di Cristo, 1984;
Di Cristo, 1999a and b). The following utterance, for example, can be either conclusive:
(la maison du voisin)L
[the neighbor's house]
or continuative:
(la maison du voisin)H
In the first case, the speaker uses a final falling tone (L) to show that he/she is temporarily finished, and
that this is a place for the interlocutor to take over. In the second case, the speaker marks his/her
intention to continue by means of a final rising tone (H), which implicitly invites the interlocutor not to
interrupt.
In both cases, the inner movements of the sequence are controlled by the syntactic and lexical
organisation of the utterance, and the speaker has no real choice concerning them (if no word is
accented):
(la maison) H L(du voisin) H
If the utterance becomes longer, phonotactic rules impose a breakdown into smaller, less prominent
prosodic groups:
1 Of course we do no claim that it would solve all ambiguities in spoken corpora (no more than punctuation solves all
ambiguities in the written language).
92
(la maison) H L(du fils) H L(du voisin) H
[the neighbor's son's house]
It seems to us that, apart from being the only feasible approach, a broad prosodic transcription, marking
only the major prosodic events is sufficient and more readable for most uses of spoken corpora.
We will be as theory-neutral as possible, and simply consider that utterances are composed of
consecutive prosodic segments, delimited by pauses and large pitch movements. These segments can
be prosodically autonomous or depend on their neighbours. In the example above, we made the implicit
assumption that the segment la maison du voisin was isolated from the rest of the discourse, for
example by a long pause (marked --) :
(la maison du voisin)H --
However, the same sequence of words with the same rising intonation could be prosodically coupled
with the next segment, as in the following dislocation :
(la maison du voisin)H (elle a br&#251;lé) L --
[the neighbour's house, it burned down]
Prosodic segments must therefore be grouped together into prosodic units. Prosodic units have an
internal prosodic cohesion, and are independent from each other in the discourse flow.
Our goal is therefore to detect the prosodic segments, to tag them and group them into prosodic units.
Five steps are involved in the process:
(1) pauses are automatically detected in the speech signal;
(2) the fundamental frequency or F0 curve is stylised in order to eliminate its smaller, irrelevant
details;
(3) the stylised curve is reduced to a sequence of discrete symbols encoding the pitch movements;
(4) the recording is orthographically transcribed and synchronised to pauses and major pitch
movements;
(5) the sequence of melodic movements is filtered and translated to a final prosodic coding.
The orthographic transcription is entirely manual. The other steps are automatic, but require some hand
correction, as summarised in Figure 1.
Pause detection (automatic + manual correction)
Stylisation of F0 curve (automatic + manual correction)
Discretisation of pitch movements (automatic)
Orthographic transcription (manual)
Prosodic coding (automatic)
Figure 1. Overview of transcription and annotation process
3. Pause detection
The first step of the annotation process consists in an automatic detection of (silent) pauses. As simple
as it may seem, this task is far from straightforward to perform by automatic means. Long pauses can
be interrupted by ambient noise (frequent outside laboratory conditions). On the other hand, very short
93
pauses are extremely difficult to distinguish from plosives. For example, some very brief breathing
pauses can be as short as 60 ms, which is shorter than many occurrences of plosives.
We use a pause detector based on F0 detection, which behaves reasonably well in terms of robustness
to ambient noise. Using a threshold of 350 ms, very few false detections occur. However, some very
short pauses are not detected, and must be added by hand. The task is not particularly difficult, with the
help of a graphic editor that enables the visualisation of the signal and playback segment by segment.
Typically the manual correction takes ca. one hour for a 15 minutes recording, and does not require a
highly specialised phonetic expertise. At the end of this phase, silent pauses are categorised into three
groups:
(1) very short pauses (< 350 ms, coded ^)
(2) short pauses (&#179; 350 ms and < 1.5 s, coded -)
(3) long pauses (&#179; 1.5 s, coded --).
Even if no subsequent prosodic treatment is planned, this technique leads to a much greater reliability
in pause transcription than direct transcription from a tape recorder, and is therefore advisable in any
spoken corpus work. Pause transcription is a very difficult exercise when done entirely manually, and
we have noticed that most linguists, even highly competent ones, tend to miss many pauses, especially
when they are coupled with other phenomena (such as hesitation or syllable lengthening). For example,
in the fragment that we will use as a running example in the next sections:
on fait pas que le pressing on fait aussi la blanchisserie plus la blanchisserie d'ailleurs - les draps les
nappes la restauration
the pause (marked with a dash) was missed by the (skilled) linguists who transcribed and verified the
corpus.
The part in bold is three-way ambiguous. The form d'ailleurs is either a locative adverb (from
elsewhere) or a discourse marker (similar to actually) which can be attached to what comes before or
after. The three interpretations are therefore:
1. We don't do only dry cleaning, we also do the laundry plus laundry from other places: sheets,
tablecloths, catering...
2. We don't do only dry cleaning, we also do the laundry. More the laundry actually: sheets, tablecloths,
catering...
3. We don't do only dry cleaning, we also do the laundry. More the laundry. Actually sheets, tablecloths,
catering...
The pause is an important clue for the disambiguation, but is not sufficient in itself. The intonation
contour must be taken into account.
4. Stylisation of the F0 curve
F0 curves can be seen as the combination of a macroprosodic component governed by syntactic and
pragmatic rules, which reflects intonational intention of the speaker, and a microprosodic component
which is entirely dependent on the effects of the particular phonemes in the utterance (lowering of F0
for voiced obstruents, etc.). Stylization consists in extracting the macroprosodic component from the
F0, while the microprosodic component is factored out. Various stylization methods have been
proposed since the sixties (Cohen & t'Hart, 1965; t'Hart, Collier, & Cohen, 1990; D'Alessandro &
Mertens, 1995; Fujisaki & Hirose, 1982; Taylor, 1994; etc.), and rely on more or less complex models.
The method used in this work (MOMEL, standing for MOdélisation de MELodie) was proposed by
Hirst & Espesser (1993) (see also Hirst, Di Cristo & Espesser, 2000). It has some appealing features
compared to other methods:
(1) it is language-independent;
(2) it does not require any pre-segmentation of the signal (e.g. in syllables);
(3) it does not require any training on the data;
(4) it performs automatically with a very good success rate;
(5) the stylised curve is perceptually undistinguishable from the original.
94
The technique consists in reducing the intonation contour to a series of target points, which represent
the relevant pitch movements (Figure 2). Once interpolated by a quadratic spline curve (unvoiced
segments are interpolated so that the resulting curve presents no discontinuities), the series of target
points produces an F0 contour perceptually undistinguishable from the original, apart from a few
detection errors that must be corrected by hand. A quantitative assessment showed that the algorithm
produces about 5% of errors. A large part of these errors (approximately 3%) were moreover
systematically of two or three different types, in particular missing targets in transitions from voiced to
voiceless segments of speech, which suggests that an improved algorithm could probably eliminate the
majority of them (Campione, forthcoming).
Again, the correction is easy to perform and does not require a specialised training. The signal can be
played segment by segment, and the original can be compared to the version re-synthesised using the
stylised curve. If the two differ perceptually, the target points can be moved via a graphic interface
until the re-synthesis is judged similar to the original. The correction phase takes around one hour for a
15 minute recording, as for the pause detection. For the moment, the two are done separately due to the
use of different tools, but we plan to integrate them in the future, thus reducing the total correction time
substantially.
les draps les nappes la restaur ation on fait pas que le
pressing
on fait aussi la
blanc hisserie
plus la
blanchisserie
d'ail
leurs
-
Figure 2. Stylisation and discretisation of the F0 curve
5. Discretisation of pitch movements
The next step consists in converting the target points into a sequence of discrete symbols encoding the
pitch movements. The requirements we set for this operation are as follows:
(1) the set of symbols should be as small as possible;
(2) it should be possible to generate from the sequence of symbols an F0 curve perceptually
undistinguishable from the original;
(3) it should be language independent.
After experimentation with several coding systems, including INTSINT (Hirst, 1991; Hirst & Di
Cristo, 1998), we have developed a mathematical model that enables a reduction of the initial curve to
an alphabet of 7 symbols without substantial loss of information. The model is based on the
observation that the distribution of target points is approximately normal (Campione & Véronis, 1998).
Details of the model are outside the scope of this paper, but the reader can find a description in Véronis
& Campione (1998).
The alphabet of symbols is as follows:
L+ large falling movement;
L medium falling movement;
L- small falling movement;
S very small or null movement;
H- small rising movement;
H medium rising movement;
H+ large rising movement.
95
These symbols have no phonological value, and consist only in an extremely compact representation of
the F0 curve. We showed, by an evaluation on a large multilingual database (4 hours 20 minutes of
speech, 50 speakers, 5 languages), that the encoding enables regeneration of ca. 99% of points at less
than 2 semi-tones than the original (Véronis & Campione, 1998). The F0 curves re-generated from the
encoding using the mathematical model are therefore virtually undistinguishable perceptively from the
original.
The model has interesting properties. In particular, movements of the same amplitude in (semi-tones)
do not necessarily have the same coding, depending on the place at which they occur in the speaker's
range, thus reflecting the fact that pitch variation towards the extremes requires more articulatory effort
than pitch variation in the speaker's medium area. As a consequence, the model also predicts the
downdrift effect which is actually observed in speech (Figure 3) without requiring a specific downdrift
parameter.
L
H
L L
H
H
Figure 3. Downdrift effect
6. Orthographic transcription
The speech signal, already segmented at pauses (see section 3), is further segmented at large pitch
movements (coded H+ or L+). In the example above, four new breakpoints are inserted (three H+ and
one L+)2, thus delimiting six segments:
_______ (H+) _______ (H+) _______ (H+) _______ (L+) _______ (pause) _______ (pause)
The corpus is then orthographically transcribed, using a graphic interface that enables playback
segment by segment. During this phase, two additional prosodic phenomena are manually encoded, but
only when they occur at the end of a segment:
(1) accents (coded *)
(2) final syllable lengthening (coded :).
Another important information results from the transcription itself, and consists in filled pauses
(hesitations) which are transcribed as special lexical items (euh).
These cues are necessary for the correct interpretation of pitch movements in the last step (see section
7). In the example above, the segment by segment transcription yields:
on fait pas que* (H+)
le pressing (H+)
on fait aussi la blanchisserie (H+)
plus la blanchisserie (L+)
d'ailleurs (pause)
les draps les nappes la restauration (pause)
Orthographic transcription using this strategy is less time-consuming than the usual technique with a
tape recorder, and more reliable. The pre-segmentation in small units, that can be replayed at will,
facilitates the transcriber's task, and helps avoiding errors (missing hesitations, repeats, etc.).
2 This is not the norm. For the sake of brevity, the example was chosen because it contained several interesting phenomena in
a short time span.
96
Transcribing and correcting a 15 minutes recording takes about two hours, as opposed to three hours
with no assistance.
7. Prosodic coding
We distinguish two types of prosodic events, depending on their scope. A first type of prosodic event is
of a punctual nature: it consists of a change of pitch or a plateau at the end of the segment, regardless of
the sequence of pitch movements in that segment. In the final segment of our example:
les draps les nappes la restauration
the rising movement on the last syllable of restauration (H-) is enough to indicate continuation (in the
context of the following pause). The sequence of preceding movements in the segment consists of a
rising-falling alternation, governed by syntactic and lexical constraints.
Other prosodic events bear on an entire segment. In this case, the normal rising-falling alternation is
replaced by a continuous rising, falling or flat sequence on the entire segment. In the example above,
the sequence
plus la blanchisserie
is composed only of falling movements (L- L L+). It has a communicative role since it indicates, in
this particular case, that the speaker has an afterthought and corrects what she has just said. It is
confirmed by the next segment
d'ailleurs
whose flat intonation (coded S) is typical of a discourse marker that “tags on” the preceding segment.
In French, the combination of a downstepped intonation followed by a plateau is a common strategy for
self-correction or retrospective message modification. The available prosodic information (including
presence of accent) therefore disambiguates among the three interpretations listed in Section 3, in
favour of the following:
We don't do only dry cleaning, we also do the laundry. More the laundry actually: sheets, tablecloths,
catering...
The first step of the algorithm marks the prosodic events at the end of each segment using three codes:
 rising (H-, H, H+)
 falling (L-, L, L+)
 flat (S)
Example :
les draps les nappes la restauration
Braces indicate that a prosodic event bears on an entire segment (for segments of more than two
syllables):
{plus la blanchisserie}
The output of this first step on our example yields:
on fait pas que* le pressing on fait aussi la blanchisserie {plus la blanchisserie} d'ailleurs 
les draps les nappes la restauration 
The second step of the algorithm consists in grouping together the prosodic segments into prosodic
units, or, to put it another way, to detect the prosodic boundaries between units. This step requires that
all the available clues are taken into account:
(1) pitch movements;
(2) silent pauses;
(3) accents;
(4) final syllable lengthening;
97
(5) filled pauses (hesitations).
The interaction of these clues is complex. For example, in French, a falling movement is perceived as a
conclusive boundary when it is followed by a short or long pause, but not if it is preceded by a syllable
lengthening, or a filled pause. On the other hand, rising movements followed by a short or long pause
mark continuative boundaries, even if they are preceded by a syllable lengthening, or a filled pause.
In order to maximise readability, prosodic units are separated by paragraph marks. Each prosodic unit
is preceded by its start time in seconds. In addition, redundancies are removed. Since accents always
appear with a rising intonation, the arrow following the accent mark (*) is removed. In the same way,
flat or falling movements before an internal pause are not marked (unless they bear on an entire
segment), because they can be interpreted only as hesitation. An example of output on a larger sample
is given Figure 4.
/
  YRLOj
/
  EHQMHWUDYDLOOHGDQVXQSUHVVLQJ 
  
  RQ IDLW SDV TXH OH SUHVVLQJ  RQ IDLW DXVVL OD EODQFKLVVHULH  ^SOXV OD EODQFKLVVHULH` 
GDLOOHXUVOHVGUDSVOHVQDSSHVODUHVWDXUDWLRQ 
  
  RQ IDLWEHDXFRXSGH FRORQLHVEHDXFRXSGH  GH FKRVHVFRPPHoD RQWUDYDLOOHSRXUOD
SROLFHSRXUODJHQGDUPHULHHXKRQWUDYDLOOHSRXUEHDXFRXSGHPRQGH 
  
  RQDEHDXFRXSGHPDUFKpV GRQFFHVWSDVpYLGHQW 
  
  ^SDUFH TXLO \ D GHV MRXUV R LO \ D`   SDV GH ERXORW  LO \ D GHV MRXUV R LO \ D GX
ERXORW 
  
  FRPPHSDUWRXW 
  
  GRQFRQHVWGHX[ 
  
  PRLHWPDFROOqJXH+D\DW 
  
  RQVHQWHQGELHQ RQDXQHERQQHDPELDQFHGDQVOHQWUHSULVHGRQFMHSHQVHTXHFHVW
TXDQGPrPHDVVH]DVVH]ELHQ 
  
  ^TXDQGLO\DXQHERQQHHQWHQWH` SDUFHTXHOHERXORWIDXWIDXWUHFRQQDvWUHRQQ\YDSDV
SDUSODLVLU 
  
  RQ\YDSDUREOLJDWLRQHXKGRQFHXKPRLMHWRXFKHjDX[GHX[ jODEODQFKLVVHULH
HWDXSUHVVLQJ 
  
  SDUFHTXHPDFROOqJXHQDSDVOD ODTXDOLILFDWLRQDXQLYHDXGXSUHVVLQJGRQFFHVWSRXU
oDTXHOOH\WRXFKHSDVSRXUOHPRPHQW 
  
Figure 4. Example of prosodic transcription
8. Conclusion and future work
The strategy and algorithms outlined in this paper enable semi-automatic transcription of prosodic
information in spoken corpora. The transcription aimed at is a “broad” prosodic transcription, in which
only the major prosodic events are annotated. We claim that a broad transcription is more suitable for
corpus-based syntactic and pragmatic studies, since most of the smaller melodic movements are the
result of local phonotactic rules and lexical contraints. A “narrow” annotation (in addition to being
impossible to carry out on a large scale) would be difficult to read and unnecessary for many purposes.
At the moment, several separate tools are used in the various phases of the transcription. An obvious
direction for further development would consist in integrating these tools into a single “prosodic
98
annotation workstation”. This would reduce the transcription time substantially, since the two phases of
manual correction (pause detection and F0 stylisation) could be merged, and accomplished during the
orthographic transcription itself. We estimate that, using an integrated environment, the transcription
and correction time of a 15 minute recording could be reduced to about three hours. This figure is
similar to the time currently required for the orthographic transcription and correction alone using a
simple tape recorder. Therefore, it seems possible to add very useful prosodic information to spoken
corpora at little or no extra cost. In addition, our strategy provides a segment by segment alignment of
the transcription with the audio signal, which can be useful for many purposes (e.g. listening to the
fragment corresponding a concordance line).
Other directions of research are concerned with the fine-tuning of the various tools. For example, the
discretisation of the pitch movements uses two parameters, the mean frequency and variance of target
points for a speaker. Currently, these values are computed for the entire recording, but one of the
features of spontaneous speech is the presence of switches in speaking style, with bursts of greater (or
smaller) variation in F0. A relatively simple pre-processing could enable us to segment the recording
into sections of coherent F0 mean and variance. Other tools could also be included in the processing
chain. For example, we have started experimenting the use of a filled pause detector, which could assist
in the transcription of this important parameter. Although the detector used was developed for Japanese
(Goto, Itou & Hayamizu, 1999), and would require tuning and adaptation for French, preliminary
results are encouraging.
Acknowledgements
In this study we made use of the Transcriber software developed by Claude Barras (DGA) and the Signaix speech tools
developed by Robert Espesser (CNRS). We are particularly indebted towards Robert Espesser for his help and assistance
throughout this project. We thank Masataka Goto for making his filled pause detector available to us, although extensive
experimentation has not been possible yet. We are also grateful to Claire Blanche-Benveniste, Albert Di Cristo, José Deulofeu,
Daniel Hirst and Frédéric Sabio for their advice and comments. Remaining errors are of course ours.
References
Bilger M, Blasco M, Cappeau P, Pallaud B, Sabio F, Savelli M-J 1997 Transcription de l'oral et
interprétation ; illustration de quelques difficultés. Recherches sur le français parlé, 14:57-86.
Blanche-Benveniste C (ed) 1990 Le français parlé : études grammaticales. Paris, CNRS éditions.
Blanche-Benveniste C, Jeanjean C 1987 Le français parlé : transcription et édition. Paris, Didier
Erudition.
Campione E, Véronis J 2001 Une évaluation de l'algorithme de stylisation mélodique MOMEL.
Travaux Interdisciplinaires du Laboratoire Parole et Langage d'Aix-en-Provence, 19 [in press].
Campione E, Véronis J 1998 A statistical study of pitch target points in five languages. In Proceedings
of the 5th International Conference on Spoken Language Processing (ICSLP'98), Sidney, pp 1391-
1394.
Cohen A, t'Hart J, 1965 Perceptual Analysis of Intonation Pattern. In Proceedings of 5&#232;me Congr&#232;s
International d'Acoustique, Li&#232;ge, 1-4.
D'Alessandro C, Mertens P 1995 Automatic Pitch Contour Stylisation Using a Model of Tonal
Perception. Computer, Speech and Language, 9:257-288.
Di Cristo A 1999a Le cadre accentuel du français : essai de modélisation : premi&#232;re partie. Langues,
2(3):184-205.
Di Cristo A 1999b Le cadre accentuel du français : essai de modélisation : seconde partie. Langues,
2(4): 258-269.
Fujisaki H, Hirose K 1982 Modelling the dynamic characteristics of voice fundamental frequency with
application to analysis and synthesis of intonation. In Proceedings of 13th International Congress of
Linguists, Tokyo, pp 57-70.
Goto M, Itou K, Hayamizu S 1999 A Real-time Filled Pause Detection System for Spontaneous Speech
Recognition. In Proceedings of the 6th European Conference on Speech Communication and
Technology (Eurospeech ’99), Budapest, pp.227-230.
Hirst, DJ 1979 The transcription of English intonation. Studia Phonetica, 17:29-39.
99
Hirst, DJ 1991. Intonation models. Towards a third generation. In Proceedings of ICPhS, I, pp 305-310.
Hirst DJ, Di Cristo A 1984 French Intonation : a parametric approach. Die Neueren Sprachen,
83(5):554-569.
Hirst DJ, Di Cristo A 1998 A survey of intonation systems. In Hirst DJ, Di Cristo, A (eds). Intonation
Systems: A Survey of Twenty Languages. Cambridge, Cambridge University Press, pp 1-44.
Hirst DJ, Di Cristo A Espesser R 2000 Levels of representation and levels of analysis for the
description of intonation systems. In Horne M (ed), Prosody: Theory and Experiment. Dordrecht,
Kluwer Academic Publishers.
Hirst DJ, Espesser R 1993 Automatic Modelling of Fundamental Frequency Using a Quadratic Spline
Function. Travaux de l'Institut de Phonétique d'Aix-en-Provence, 15:75-85.
Leech G, McEnery A, Wynne M 1997 Further Levels of Annotation. In Garside R, Leech G, McEnery
A (eds), Corpus Annotation : Linguistic Information from Computer Text Corpora, London, Longman,
pp 85-101.
Mertens P 1990 Intonation. In Blanche-Benveniste C (ed), Le français parlé: études grammaticales.
Paris, CNRS Edition, pp 159-176.
Nolan F, Grabe E 1997. Can ’ToBI’ transcribe intonational variations in British English? In
Proceedings of ESCA Workshop Intonation: Theory, Models and Applications, Athens, pp 259-262.
Ostendorf M, Ross K 1997 A multi-level model for recognition of intonation labels. In Sagisaka,
Campbell and Higuchi (eds), Computing Prosody. Springer, Berlin, pp 291-308.
Silverman K, Beckman M, Pitrelli J, Ostendorf M, Wightman C, Price P, Pierrehumbert J, Hirschberg J
1992 ToBI: a standard for labelling English prosody. In Proceedings of ICSLP'92, Banff, pp 867-870.
Svartvik J (ed) The London Corpus of Spoken English: Description and Research. Lund, Lund
University Press.
t'Hart J, Collier R, Cohen A 1990 A Perceptual Study of Intonation, an experimental-phonetic
approach to speech melody, Cambridge, Cambridge University Press.
Taylor LJ, Knowles G 1988 Manual of information to accompany the Spoken English Corpus.
Technical Report, UCREL, University of Lancaster.
Taylor P 1994 The Rise/Fall/Connection Model of Intonation. Speech Communication, 15(1,2):169-
186.
Véronis J, Campione E 1998 Towards a reversible symbolic coding of intonation. In Proceedings of the
5th International Conference on Spoken Language Processing (ICSLP'98), Sidney, pp 2899-2902.
Wightman CW, Ostendorf M 1992 Automatic Recognition of Intonational Features. In Proceedings of
the International Conference on Acoustics, Speech, and Signal Processing, San Francisco, pp 221-224.

100
An attempt to improve current collocation analysis
Pascual Cantos-Gómez
Universidad de Murcia (Spain)
Abstract
Our goal is to review the main stream of research on collocation analysis and try to overcome
some of its intrinsic problems, such as the optimal span and the reason for undesired collocates
(statistically significant collocates, though lexically and semantically not related with the node word).
The idea is not just to calculate significant collocates of a chosen node word or to elucidate which is the
best statistical procedure to achieve this goal. However, we shall focus on the way words socialise with
other words, forming complex network-like structures or units: lexical constellations; something that
cannot be explained using current collocation analyses.
1. Introduction
Conventional or prepatterned expressions are often loosely referred to as collocations. However,
since the term collocation is also used in a stricter sense to denote a special kind of lexical relationship,
it is convenient to clarify these concepts.
In its broadest sense, collocation is more or less equivalent to: recurrent word combination. In the
Firthian tradition, however, it is generally used in a stricter sense: a collocation consists of two or more
words which have a strong tendency to be used together. According to Firth (1968: 181), “collocations
of a given word are statements of the habitual or customary places of that word.” That they are
habitually co-occurring lexical items or mutually selective lexical items (Cruse 1986: 40). For example,
in English you say rancid butter, not sour butter, or burning ambition instead of firing ambition.
Both interpretations imply a syntagmatic relationship between linguistic items, but whereas the
broader sense focuses on word sequences in texts, the stricter one goes beyond this notion of textual cooccurrence
and emphasises the relationship between lexical items in language (Greenbaum 1974: 80).
It follows that the former, but not necessarily the latter, includes idioms, compounds and complex
words, and that the latter extends also to discontinuous items.
There are also intermediate uses of the term, where collocations can be thought of as lying on a
continuum. At one end of the continuum lie the idioms like as soft as butter. At the other end of the
continuum lie free combinations. These are word combinations that can be formed by simply applying
grammatical rules. For example, the table would be the result of the grammatical rule that states that
the definite article precedes a countable noun.
The Firthian notion of collocation is thus a more extensive lexical concept than recurrent word
combination. Consequently, methodologically it is viewed as essentially a probabilistic phenomenon
consisting of identifying statistically significant collocations and excluding fortuitous combinations.
This means that collocation analysis can be dealt with quantitatively and to some extent also
automatically.
Sinclair, within the Firthian tradition, defines collocation as “the occurrence of two or more words
within a short space of each other in a text” (1991: 170). In collocation analysis, interest normally
centres on the extent to which the actual pattern of these occurrences differs from the pattern that
would have been expected, assuming a random distribution of forms. Any significant difference can be
taken as, at least, preliminary evidence that the presence of one word in the text affects the occurrence
of the other in some way.
In what follows, we shall briefly discuss: (1) the ways in which actual and expected patterns of cooccurrence
can be computed and compared, and (2) the measures that can be applied to the results to
assess their significance and explore their implications.
2. Extracting Collocations
A significant collocation can be defined in statistical terms as the probability of one lexical item
co-occurring with another word or phrase within a specified linear distance or span being greater than
might be expected from pure chance.
It is not our goal here to discuss and compare the various measures, though some points are
important to note. Probably, the most commonly used statistical measures to determine collocate
significance are z-score (Berry-Rogghe 1973), t-score and mutual information (Church and Hanks
1990). All three significance measures can be used to highlight words that appear to be most strongly
101
collocated with the node word. Strong collocates are somehow equally highlighted by all three
measures; however, z-score and the mutual information measure artificially inflate the significance of
low-frequency co-occurring words because of the nature of their formulae.
There are important differences between the information provided by the three measures: more,
perhaps, between t-score and the other two than between z-score and mutual information themselves. It
is difficult, if not impossible, to select one measure that provides the best possible assessment of
collocates, although there has been ample discussion of their relative merits (see, for example, Church
et al. 1991; Clear 1993; or Stubbs 1995). It is probably better to use as much information as possible in
exploring collocation and to take advantage of the different perspectives provided by the use of more
than one measure.
Geffroy et al. (1973) produced a formula for the strength of collocation (called C) which took into
account both the frequency of co-occurrence and the proximity of the collocates to each other. They
employed cut-off points for minimum values of C and f (frequency) such as 5 and 20, respectively.
Similarly, Smadja et al. (Smadja and McKeown 1990, Smadja 1991) take into account word
distance as well as word strength (spread) for a measure of word association (height) within a limited
span: 5 words on either side (see Martin et al. 1983). By means of XTRACT, Smadja proposes an
approach for automatically acquiring co-occurrence knowledge from statistical analysis of large
corpora. Smadja assumes that two words are associated if they appear in the same sentence and are
separated by less than five words. For each pair of collocating words, a vector of 10 values is created.
In a flexible collocation, the words may be inflected, the word order may vary and the words can be
separated by any number of intervening words (i.e. to take steps, took immediate steps, steps were
taken). If the word order and inflections are fixed and the words are in sequence, the collocation is said
to be rigid (e.g. International Human Rights).
Kita et al. (1994) also propose a mathematical approach to automatically compiling collocations:
the cost criteria. However, their approach differs slightly from the others mentioned so far in that they
consider collocations to be cohesive word clusters, including idioms, frozen expressions and compound
words. This approach focuses just on linearly fixed multiple word units, being unable to determine
discontinuous co-occurrences and/or word associations. Kita et al.'s measure relies on the length of the
collocate or word sequence and on its frequency. This approach is much more in the line of research
that understands collocation within the broader definition (see above) and consequently fails to account
for word associations, lexical relations or discontinuous collocations.
There are also other approaches such as Lafon's use of combinatorics to determine collocational
significance (Lafon 1984), Daille's approach to monolingual terminology extraction (Daille 1995) or
Dunning's log-likelihood measure also known as g-square or g-score (Dunning 1993), among others.
3. Collocations Revisited
In what follows, we shall re-examine some basic notions regarding collocations with reference to
real data. This, we are confident, will give us a more comprehensive view of the complex
interrelationships (semantic, lexical, syntactic, etc.) between co-occurring linguistic items.
Three starting assumptions were made: (1) we understand collocates in the stricter sense, as
already discussed: that is, words that have a strong tendency to co-occur around a node word in a
concordance; (2) in order to avoid being biased in favour of any data, we concentrated on the
occurrences of a lemma within a single semantic domain: the Spanish noun mano (extracted from the
CUMBRE Corpus -16.5 million tokens-, by SGEL); and (3) no optimal span was taken for granted.
This explains why we took the full sentence as the concordance (though important collocates can be
missed due to anaphoric reference) .
To make sure that we were actually focusing on the same semantic contexts, we first extracted all
instances (full sentences) containing mano, both in singular and plural form and next, classified all
occurrences semantically by means of the various definitions for mano.
This preliminary distributional analysis allowed us to isolate the various meanings and to
concentrate on individual meanings or identical semantic contexts. In the analysis that follows, we shall
concentrate on one of the meanings of mano (“layer of paint, varnish or other substance put on a
surface at one time”).
As already noted, no preliminary assumptions were made regarding relevant or optimal window
size or span. We simply started by taking the whole sentence in the belief that full sentences are more
likely to contain complete ideas, meanings, etc. Fixing a span beforehand might often distort the unity
of meaning or complete meaning in a sentence. We tend to think that words outside the range of a
sentence are not likely to affect the meaning unity of other sentences, nor are they strongly associated
to node words within other sentences.
102
The distribution of significant collocates (z-scores) results in the following graph (Figure 1),
where the x-axis displays the position relative to the node word and the y-axis the number of
statistically significant collocates.
In order to get a more accurate and neater visual representation of the distribution of the significant
collocates than the zigzag curves above, we normalised the data of the y-axis (number of significant
collocates), by dividing the number of significant collocates found at each position by its positioning
with respect to the node word (Figure 2). The y-axis represents the relative number of significant
collocates. Figure 2 visually represent the extent of influence of the node on its immediate
environment: the lexical gravity. Note that our notion of gravity differs radically from the one proposed
by Sinclair et al. (1998; see also Mason 1997): (1) we do not calculate the extent of influence of the
node on its immediate verbal environment, but take the node itself; (2) Mason's one is a more heavily
calculated statistic one and (3) instead of displaying gravity as crater shapes, we do it in peak shapes.
The graph stresses the attraction of MANO(layer of paint, etc.), which is greatest in the immediate
environment and wears off with distance, which is indeed an already established fact.
If we observe and compare the actual gravity with its trendline, we realize that there is a kind of
correlation between the relative number of significant collocates and the node distance, particularly
close around the node word. Intuitively, we can with some confidence state that the most relevant
collocational context for MANO(layer of paint, etc) is between -8 and +6. It is also interesting that data
around the node tends to have a more normal distribution than overall. However, evidence shows that
fixing an optimal span could be misleading as the appearance of significant collocates is likely to
exceed these pre-established limits.
Fig.1. D istribution o f significant collocates vs trendline (z-score: mano)
-1
0
1
2
3
4
5
6
-1 40 -120 -100 -80 -6 0 -40 -20 0 20 40
Pos itioning
Number of significant collocates
Fig. 2. Gravity vs trendline ( mano)
-0 ,5
0
0 ,5
1
1 ,5
2
2 ,5
3
-140 -120 -100 -80 -60 -40 -20 0 20 40
Pos itioning
Relative number of sgnificant collocates
103
Against the background of this data, it becomes clear that the distribution of significant collocates
is not necessarily normal. This has important consequences as it speaks against generic assumptions on
optimal spans (see i.e. Jones and Sinclair 1973; Martin et al. 1983; Smadja 1989; or Berber 1997,
among others). Phillips is right to claim that “it is probably sensible to err in favour of overinclusiveness”
(Phillips 1989: 23).
This led us to state two hypotheses on collocates: H1: each word has a unique idiosyncratic
linguistic behaviour (different attraction strength or power), at least lexically speaking; and H2: not all
significant collocates of a node word are actually attracted by it, only some. Others are likely to be
attracted by other significant collocates within the same context (sentence, phrase, concordance line,
etc.).
Regarding H1, it is clear that our empirical data is not sufficient to accept or reject this hypothesis.
However, other related research such as Mason's (1997) gravity study and Sinclair et al. (1998) clearly
support it. They talk about the importance of establishing a reasonable span for the collocational
analysis of a word. They go even further by saying that there is a need to have some means of
justifying the span to be used for each word. The key is to discover the extent of node effect on its
environment (whether the optimum span varies according to the node's grammatical class, semantic
range, between word-forms of a lemma, richness of lexis and so on, remains to be seen). These
findings clearly indicate a change compared with those of Martin et al. (1983) and Jones and Sinclair
(1973), who fixed the span of influence of the node on its environment, irrespective of the grammatical
class of the node, its semantic range, etc, and concluded that the optimal span is -5 +5 and -4 +4,
respectively. Summing up, this means that each word has a distinctive range of influence on other
words and we, necessarily, need to determine the gravity of each word in order to arrive at or calculate
its significant span.
H2 states that not all significant collocates are actually attracted by the same word or node; only
some are. Others are likely to be attracted by other significant collocates. This hypothesis does not
focus on the span; it is even span-independent. What matters here is not just the span or extent of the
influence of the node on its immediate environment but the fact that collocates (statistically significant
ones) within the same environment are not necessarily attracted by the same node. This introduces a
striking difference with regard to H1. By H2, we understand that collocates within the same
environment build ordered and structured frames: not flat lexical frames (traditional view of collocates,
see H1), but complex interrelated hierarchies similar to constellations consisting of a nucleus (e.g. the
Sun) which attracts various other stars with each star attracting various other moons. Collocates form
lexical conceptual multi-dimensional frames: lexical constellations. Constellations themselves can be
substructures (subsets) of others (e.g. the Solar System as part of the Milky Way) or superstructures
(supersets) subsuming other structures (e.g. the Solar System containing Jupiter and its Moons).
Both hypotheses seem plausible, particularly the first one, if we consider the main stream and
recent advances made in corpus linguistic research on collocational analysis (see e.g. Daille 1995;
Mason 1997; Smadja 1992; Samdja et al. 1996; Sinclair et al. 1998). However, H1 somehow fails to
comfort to one major psychological feature of linguistic data: efficiency of storage of lexical items
(Worff 1991). This speaks against the economising role or purpose of language production (Coulmas
1981, Cowie 1981, Nattinger 1980, 1988, etc.). There are features that indicate that the effect or
influence of a word on others varies significantly among words (see Berry-Rogghe 1974). However,
the range of variability cannot be unlimited or very large. Consequently, H1 is, at least to its full extent,
not fully plausible psychologically, as humans have a limited memory and storage capacity.
H2 seems in this respect more realistic, both psychologically and linguistically. Evidence shows
that words exert some influence on others, forming units (semantic units, tone units, syntactic
structures, such as phrases, etc.). Furthermore, these units are to some extent cognitively, spatially and
temporarily limited, though not necessarily universally fixed or equally limited. Several principles
speak in favour of this: minimal attachment, right association and lexical preferences (Allen 1995: 160-
2). This means that the attraction domain of the node needs to be not just limited but also constrained,
though not necessarily fixed or universally predetermined: the extent of influence is likely to vary
depending on grammatical category, semantic range, etc. This assumption significantly reduces the
range of variability of the various possible attraction domains (optimal spans) of node words. In
addition, this also fits in with the unification-based assumption of the constellation principle: ordered
hierarchical elements forming autonomous structures, sub-structures or super-structures.
This explains the appearance of significant collocates for MANO(layer of paint, etc.) at positions
-99, -89, -38, -28, -19 and +20, among others. If they are not attracted by the node word, then which
items exert their influence on them and attract them? We might explain these facts as follows: the high
z-scores categorise them as significant collocates within the sentence contexts of MANO(layer of paint,
etc.); however, they are likely to be attracted by other words, not necessarily by mano. That is, these
104
collocates are part of another unit, not the immediate unit or vicinity of the node word.
From the data, it becomes apparent that, for instance, profesora (-118) -a significant collocate
within the concordance sentence of MANO(layer of paint, etc.)- might not be semantically attracted by
mano, but by other lexical items within the same context, e.g. by Universidad (position -115). And,
Autónoma (-114) is probably attracted by Universidad (position -115), too. The reason for these
significant collocates, within the context sentences of MANO(layer of paint, etc.), is not related to the
influence the noun mano exert on them, but is due to the fact that they are attracted by other items.
Indeed, they seem to be part of a different unit (i.e. part of a different phrase). This means that the
direct influence of words on others is not just limited but also somehow structured.
To illustrate this, let us take one concordance sentence for MANO(layer of paint, etc.): “Si algún
tono no tiene la intensidad deseada, aplique otra {MANO} de pintura” (“If the shade hasn't got the
desired intensity, apply another layer of paint”). This mano sentence contains just six statistically
significant collocates: algún, tono, intensidad, deseada, aplique and pintura, distributed from -9 to +2
(a positively skewed influence distribution). This gives the following collocational distribution:
Positioning
-10 -9 -8 -7 -6 -5
Freq. 1 Si algún tono no tiene la
Positioning
-4 -3 -2 -1 +1 +2
Freq. 1 intensidad deseada aplique otra de pintura
A simple flat collocational analysis would just reveal that there are six collocates likely to be
statistically significant for mano. Furthermore, if we had fixed the optimum span to -5 +5 or –4 +4, we
would have missed at least: algún and tono.
To know more about the possible lexical hierarchy, we started by calculating the expected cooccurrence
probability of all possible significant collocate tuples (combinations) within the same
sentence, that is, the probability that event x occurs, given that the event y has already occurred. For
example, taking the occurrence data for tono and intensidad we get:
Tono Intensidad
Intensidad 11 Tono 11
Without intensidad 1037 Without tono 640
1048
651
P(intensidad*tono) = 11 / 1048 = 0.0105
P(tono*intensidad) = 11 / 651 = 0.0169
This shows that tono is lexically less tied to intensidad and more likely to appear without it,
whereas intensidad is much more dependent on tono and less likely to occur without it. This
information reveals that tono is lexically less tied to intensidad and more likely to appear without it,
whereas intensidad is more dependent on tono. The probability figures indicate that the lexical
attraction is mutual and bi-directional, but not equidistant: tono exerts a greater influence and attraction
on intensidad than vice versa. Sinclair already realised this fact (1991: 115-116), coining the terms
upward and downward collocation. These two types of collocation are explained using the term node
(for the word that is being studied) and the term collocate (for any word that occurs in the specified
environment of a node). Coming back to our previous example, if we take tono as the node and
intensidad as the collocate, what we have is downward collocation: collocation of a node (tono) with a
less frequent word (intensidad). On the contrary, if intensidad is the node and tono the collocate then
we have upward collocation. The systematic difference between upward and downward collocation is
that the former is the weaker pattern in statistical terms, and the words tend to be elements of
grammatical frames, or subordinates. Downward collocation by contrast gives us a semantic analysis of
words (Sinclair 1991: 116).
Once the attraction direction procedure was established (upward or downward collocation), we
determined all possible tuples of mano by means of the given statistically significant collocates
(namely, tono, intensidad, deseada, aplique and pintura). We discarded algún as it is not a content
word or open-class lexical item. Potentially, the possible tuples or combinations within the same
context sentence range for mano from single-word ones up to six-word clusters (the total number of
105
statistically significant collocates plus the node word mano), that is, the sum of all combinations of r
objects (number of co-occurring items) from n (set of statistically significant collocates). Thus, the
combination of r objects from n is calculated by means of: C(n, r) = n! / (n – r)!r!
This totals an overall of 63 possible order-independent combinations for mano (6 + 15 + 20 + 15 +
6 + 1). However, the singletons (single-word combinations) are practically irrelevant as they just
indicate their own probability of occurrence in the corpus, irrespective of co-occurrence with other
items.
According to the data and probability calculations for tono and intensidad above, we believe that
the attraction direction is somehow inherently determined by the observed frequency (consider also
Sinclair's notion of downward and upward collocation (Sinclair 1991: 115-116)). This means that,
between two items, the one that occurs more often in a language model (i.e. corpus) is likely to exert a
greater influence on the less occurring one(s). This explains why we do not care about the actual order
of the combinations as this is to some extent predetermined by the observed frequencies of their
constituents.
In order to concentrate on the really significant combinations, we took a number of preliminary
decisions. First, all singletons were discarded as they do not give any information as to which other
items they combine with. Next, we discarded all combinations with an occurrence frequency = 1. The
reason is that all significant collocates occur, at least once, that is, whenever they co-occur in the
concordance sentence itself. Consequently, combinations with a frequency lower than 2 are irrelevant,
as they do not contribute anything to their association power with other items. So, among all potential
combinations, we discarded the hapax tuples (combinations with an occurrence frequency = 1) and the
singletons (single words). This resulted in the following significant combinations for mano:
TOKEN(S) FREQ(corpus) PROB(corpus)
mano,pintura 17 6.19820E-14
tono,intensidad 11 4.01060E-14
pintura,tono 8 2.91680E-14
mano,tono 7 2.55220E-14
mano,intensidad 6 2.18760E-14
intensidad,deseada 3 1.09380E-14
mano,aplique 2 7.29200E-15
tono,deseada 2 7.29200E-15
tono,intensidad,deseada 2 4.40306E-22
We find that the most likely tuple for mano is mano-pintura. This means that, in fact, mano is
responsible for its co-occurrence with pintura, but not necessarily the other way round, as mano is
more likely to be found than pintura. This also applies to the tuple tono-intensidad, producing these
two ordered pairs:
mano (pintura
)
tono (intensidad
)
This means that the collocate intensidad, though significant within the context of MANO(layer of
paint, etc.), is not directly attracted by mano but by another significant collocate (tono) within the same
context. The third most likely combination is pintura-tono. Here we have now the missing link between
mano-pintura and tono-intensidad. The two previous combinations can be merged and reduced into a
single structure, giving:
mano (pintura
(tono
(intensidad
)
)
)
The next tuple mano-tono is already entailed in the structure above, and indicates that mano does
indeed attract tono. However, evidence shows that it is actually pintura that exerts a greater influence
on tono, or, in other words, mano attracts tono by means of pintura. If it were not for pintura, it is
unlikely that tono would be a collocate of mano or, at least, directly attracted by it. A similar case is
that of mano-intensidad. The next combination, intensidad-deseada, enlarges the structure into:
106
mano (pintura
(tono
(intensidad
(deseada
)
)
)
)
Mano-aplique brings along a new directly attracted collocate of mano:
mano (pintura
(tono
(intensidad
(deseada
)
)
)
)
(aplique
)
and finally, tono-deseada and tono-intensidad-deseada which are already entailed in the structure
above.
We conclude that, in this example sentence, within the context of MANO(layer of paint, etc.),
mano is the dominant collocate or node as it directly attracts pintura and aplique and indirectly attracts
most of collocates (tono, intensidad and aplique), except for deseada. The resulting collocational
hierarchy is:
MANO (pintura
(tono
(intensidad
(deseada
)
)
)
)
(aplique
)
Or in a more visual mode, very similar to a 3D star constellation:
4. Conclusions
Constellations represent lexical relations among collocates in a structured hierarchical way versus
the standard simple representation of most collocational analyses. It is precisely the plain vision of
standard collocational work that misses two main features of collocational behaviour: (1) their non-
107
linear dependency: not all collocates are necessarily attracted by the same node word and, furthermore;
(2) those that co-occur within a determined unit (phrase, sentence, etc.) form a kind of complex lexical
hierarchy.
It is also interesting that by means of constellational analysis we might also get rid of the optimum
span dilemma, as this analysis always takes the sentence unit. What matters is not the span, but the
lexical hierarchy collocates form within a linguistic unit (sentence, phrase, concordance line, etc.). The
lexical hierarchies are neither predetermined nor assumed (as happens in most semantic networks or
word sense hierarchies); they result from the co-occurrence probabilities and the attraction calculations,
which might lead to divergences depending on the corpus or language samples analysed.
The potential implementation of this constellation analysis could, of course, go much further by
means of incorporating lemmatisation and part-of-speech tagging. This would prevent duplicate lexical
relations (i.e. homonymy, polysemy) and part-of-speech ambiguity, and it would refine frequencies,
producing more accurate constellational models.
Notice that we have not tackled here the problem of which statistical collocation model or method
to use, as this is not a basic issue for constellational analysis. Any statistical model would, in principle,
do; this is up to the researcher's preferences. Of course, much research is still needed to totally
configure this new trend in collocational analysis and to sort out hierarchical discrepancies/similarities
(e.g. very close statistical significances).
We are confident that this new view within collocational analysis can be a positive contribution.
The approach presented is transparent in nature and empirically testable, producing a good description
of the storehouse of vocabulary items. That is, how words are connected through their associations and
how lexical knowledge is schematic and associative. This could lead to improved results in
computational lexicography (development of collocational dictionaries, automatic construction and
typification of proto-senses, etc.), language pedagogy (the teaching/learning of collocational
knowledge for L2 learners) and Machine Translation (particularly to EBMT), to mention some
important applications.
Similarly, there are also various research interests within the lexical constellation framework. This
includes the capturing and/or mapping of syntactic structures (i.e. exploring whether lexical
constellations capture the representation of syntax) by means of various linguistic models
(Compositional Grammars, Construction Grammars, etc.) or specific parsing strategies (e.g. Data
Oriented Parsing).
References
Allen J 1995 Natural Language Understanding. Redwood: The Benjamin/Cummings Publishing
Company.
Berber Sardinha AP 1997 Lexical Co-occurrence: A Preliminary Investigation into Business English
Phraseology. Letras & Letras, 13/1: 15-23.
Berry-Rogghe GLM 1973 The Computation of Collocations and their Relevance in Lexical Studies. In
AJ Aitken, R Bailey, N Hamilton-Smith (eds) The Computer and Literary Studies. Edinburgh:
Edinburgh University Press.
Berry-Rogghe GLM 1974 Automatic Identification of Phrasal Verbs. In J.L. Mitchell (ed) Computers
in the Humanities. Edinburgh: Edinburgh University Press.
Church KW, Hanks P 1990 Word Association Norms, Mutual Information and Lexicography.
Computational Linguistics, 16/1: 22-9.
Church KW, Gale W, Hanks P, Hindle D 1991 Using Statistics in Lexical Analysis. In U. Zernik (ed)
Lexical Acquisition: Exploiting On-line Resources to Build a Lexicon. Hillsdale, NJ: Lawrence
Erlbaum Associates, 115-164.
Clear J 1993 From Firth Principles: Computational Tools for the Study of Collocation. In M. Baker et
al. (eds) Text and Technology. Amsterdam: Benjamins, 271-292.
Coulmas F 1981 Introduction: Conversational Routine. In F Coulmas (ed) Conversational Routine:
Explorations in Standardized Communication Situations and Pre-patterned Speech. The Hague:
Mouton, 1-17.
Cowie AP 1981 The Treatment of Collocations and Idioms in Learners's Dictionaries”. Applied
Linguistics, 2: 223-235.
Daille B 1995 Combined Approach for Terminology Extraction: Lexical Statistics and Linguistic
Filtering. UCREL Technical Papers. Volume 5. Lancaster: University of Lancaster Press.
Dunning T 1993 Accurate Methods for the Statistics of Surprise and Coincidence. Computational
Linguistics, 19/1: 61-74.
Firth J 1968 A Synopsis of Linguistic Theory 1930-1955. In FR Palmer (ed) Selected Papers of J.R.
108
Firth 1952-59. Bloomington: Indiana University Press, 1-32.
Geffroy A, Lafon P, Seidel G, Tournier M 1973 Lexicometric Analysis of Co-occurrences. In AJ
Aitkien, R Bailey and N Hamilton-Smith (eds) The Computer and Literary Studies. Edinburgh:
Edinburgh University Press.
Greenbaum S 1974 Some Verb-Intensifier Collocations in American and British English. American
Speech, 49: 79-89.
Jones S, Sinclair J 1973 English Lexical Collocations. A Study in Computational Linguistics. Cahiers
de Lexicologie, 23/2: 15-61.
Kita K, Kato Y, Omoto T, Yano Y 1994 Automatically Extracting Collocations from Corpora for
Language Learning. In A Wilson, T McEnery (eds) UCREL Technical Papers. Volume 4.
Corpora in Language Education and Research. A Selection of Papers from TALC94. University
of Lancaster. 53-64.
Lafon P 1984 Dépouillements et statistiques en lexicométrie. Geneva: Slatkine-Champion.
Martin W, Al B, van Sterkenburg P 1983 On the Processing of a Text Corpus: From Textual Data to
Lexicographical Information. In R Hartmann (ed) Lexicography: Principles and Practice.
London: Academic Press.
Mason O 1997 The Weight of Words: An Investigation of Lexical Gravity. In Proceedings of
PALC'97, 361-375.
Nattinger J 1980 A Lexical Phrase Grammar for ESL. TESOL Quarterly, 14: 337-344.
Nattinger J 1988 Some Current Trends in Vocabulary Teaching. In C. Carter and M. McCarthy (eds)
Vocabulary and Language Teaching. London: Longman, 62-82.
Phillips M 1989 Lexical Structure of Text. Birmingham: ELR/University of Birmingham.
Sinclair J 1991 Corpus, Concordance, Collocation. Oxford: Oxford University Press.
Sinclair J, Mason O, Ball J, Barnbrook G 1998 Language Independent Statistical Software for Corpus
Exploration. Computers and the Humanities, 31: 229-255.
Smadja FA 1989 Lexical Co-occurrence: The Missing Link. Literary and Linguistic Computing, 4/3:
163-168.
Smadja FA 1992 XTRACT: An Overview. Computers and the Humanities, 26/5-6: 399-414.
Smadja FA 1991 Macro-coding the Lexicon with Co-occurrence Knowledge. In U Zernik (ed) Lexical
Acquisition: Exploiting On-line Resources to Build a Lexicon. Hillsdale, NJ: Lawrence Erlbaum
Associates, 165-189.
Smadja FA, McKeown KR 1990 Automatically Extracting and Representating Collocations for
Language Generation. Proceedings of the 28th Annual Meeting of the Association for
Computational Linguistics, 252-259.
Smadja FA, McKeown KR, Hatzivassiloglou V 1996 Translating Collocations for Bilingual Lexicons:
A Statistical Approach. Computational Linguistics, 22/1: 1-38.
Stubbs M 1995 Collocations and Semantic Profiles: On the Cause of the Trouble with Quantitative
Methods. Functions of Language, 2/1: 1-33.
Wolff JG 1991 Towards a Theory of Cognition and Computing. Chichester: Ellis Horwood.

119
The Design of Czech Lexical Database
)UDQWL&#227;HNýHUPiN
Institute of Czech National Corpus, Philosophical Faculty, Prague, Czech Republic
Frantisek.Cermak@ff.cuni.cz
Jana Klímová
Institute of Czech Language, Czech Academy of Science, Prague, Czech Republic
Jana.Klimova@ff.cuni.cz
Karel Pala
Dept. of Information Technologies,
Faculty of Informatics, Masaryk University Brno, Czech Republic
pala@fi.muni.cz
9ODGLPtU3HWNHYL&#254;
Institute of Theoretical and Computational Linguistics
Philosophical Faculty, Charles University, Prague, Czech Republic
9ODGLPtU3HWNHYL&#254;#IIFXQLF]
1. Introduction
The aim of the paper is to present a conception of Czech Lexical Database (CLD) that should later
become a basis for the new representative Czech dictionary. The main purpose of this enterprise is to
build a representative Czech Lexical Database that would serve as a source of lexical information and
also as a partial knowledge representation in various NLP applications (Ingria, Boguraev, Pustejovsky,
1992, p.341-365)1. The basic units in CLD can be either single lemmata like G PKRXVH or standard
collocations as e.g. vysoká škola (university). The assumed size of the designed CLD is approximately
50 000 entries and we set as our primary task to concentrate on Czech verbs as much as possible, i.e.
the number of the described verbs should be about 20 000 (40 % of CLD size, when the estimated
number verbs in Czech is about 40 000 items). This is based on the fact that the verbs represent the
main relational element in natural languages around which the other elements, mostly nouns, are
centered.
2. The basic structure of CLD
It can be described by a DTD (most probably designed in XML) that will consist of the following
parts (fields, see e.g. Faber, Usón 1999, p.20)2:
a1) <phonological (phonetic) information> about the sound structure of the expressions
constituting a given entry. This in fact means that we will be trying to develop the (parallel) speech
database for Czech that would form a data collection for building the algorithms able to process speech
signals, as e.g. speech recognition, synthesis and encoding, as well as recognition and verification of
the speaker. The speech database data will be appropriately included in the lexical database. Some
interesting tasks have to be solved in this respect: particularly, additional word forms will have to be
generated by the speech synthesis module since it would be virtually impossible to account for all
forms of all words in the lexical database: there are approximately 5–5.5 million word forms in Czech).
a2) <morphological information> about the structure of the entry (for Czech) – it will yield the
information about POS and all the respective grammatical categories associated with that POS plus the
information about the basic segmentation. For nouns this can be captured by <a code of the respective
inflectional paradigm> since we assume that a morphological analyzer/generator AJKA will be
integrated into CLD (6HGOi&#254;HN )3 that would offer the morphological information on demand
1 Ingria R, Boguraev B, Pustejovsky J 1992 Dictionary/Lexicon. in Encyclopedia of Artifical
Intelligence (ed. by Shapiro S. C.), New York, John Wiley, pp.341-365.
2 Faber P, Usón R, M 1999 Constructing a Lexicon of English Verbs. Berlin – New York, de Gruyter.
3 6HGOi&#254;HN50RUIRORJLFN&#234;DQDO\]iWRUSUR&#254;H&#227;WLQX0RUSKRORJLFDODQDO\VHUIRU&]HFK'LSORPD
Thesis, Faculty of Informatics, Brno.
120
(dynamically). For verbs this typically includes 8 categories (attributes): <negation>, <person>,
<number>, <tense>, <mode>, <voice>, <aspect> and <gender>. Their values would be accessed
dynamically through the <inflectional paradigm code of the verb>. To get this information the
morphological analyzer/generator will be used in a similar way as for nouns. Also word formation
information will be included here in a subfield and it should show the relevant formally justified links
between the respective entries including their semantic consequences, cross POS relations like the
direction of derivation (as in práce &#197; pracovat (work &#197; to work)). This poses a task how to formulate
the word derivation rules as formally as possible (see below Klímová, Pala, 2000, p.987-991),
a3) <senses1...n> where for each sense the following should be given:
a3.2) <semantic features> that can be associated with an entry – possibly based on EuroWordNet
Top Ontology and hypero/hyponymy hierarchies (trees) or their parts (subtrees or clusters) (Vossen,
1999)4. It is to be examined how large parts of the trees or subtrees can be employed – we estimate that
the plausible number of the nodes used here may be about 5,
a3.3 <descriptions using genus proximum (hypero/hyponymic relationships)> and distinguishers
(differentia specifica), will be given typically for the entries containing nouns. In fact the genus
proximum definitions can be viewed as subsets of the hypero/hyponymy trees where only two nodes
are considered. The distinguishers represent a sort of problem: in our view they quite successfully resist
to the attempts to formalize them. This can be demonstrated by the fact that the particular dictionaries
differ most in the way in which they treat the distinguishers – there is no general agreement as to what
distinguishers should or should not be selected and included in the particular entries.
a3.4 <semantic classes> – for verbs the genus proximum definitions may not work as reliably as
for nouns or they may be used reliably only for a small number of them, therefore we suggest here to
indicate a semantic class a verb belongs to. In this respect we are preparing a semantic classification of
Czech verbs similar to Levin‘s (Levin, 1995)5 though in Czech this task seems to appear more
complicated because of the category of aspect (thanks to this Czech verbs regularly occur in pairs). On
the other hand, it is also obvious that the semantic classes of verbs are closely related to the valency
frames (verb frames) and we set as our task to reflect these links in the database as well.
a3.1 <synset> that can be found for a given lexical unit (entry, lemma), possibly in WordNet
fashion. The reason for having synsets follows from the fact that the relation of synonymy (and
antonymy) can serve as one of few relatively reliable ways of the characterization of meaning
a4) <syntactic information> about the combinatorial properties of the entry and the expressions
that are related to it. It is obvious that typically the syntactic properties of the given item are strongly
related to the particular sense of the entry and they distinguish it from the other senses. The information
given in this field will be captured through <valency frames> for all the POS where it makes sense, i.e.
for verbs, nouns, adjectives, numerals and also some adverbs. It is evident that in this respect we have
to distinguish syntactic (or superficial) valency frames that in Czech will include the combinatorial
information about the morphological cases (there are seven of them in Czech) and semantic (or deep)
valency frames containing the necessary information about the semantic cases (roles) that are expressed
by the morphological cases. The notation linking syntactic and semantic valencies is indicated in the
examples below, however we take it as preliminary since the final inventory of deep cases for Czech
has not been established yet (see e.g. Fillmore and Atkins, 1998, p. 417-423)6 .
That is not all, in our view it is also very useful to include the particular lexical information in the
valency frames. Typically, it is not enough to know just the respective values of morphological
(superficial) cases but their lexical "cast" as well, see e.g. the vital difference between two accusatives
occurring in GU&#229;HWYUXFHNQLKXKROGDERRNLQWKHKDQG and GU&#229;HWWYDr (to mantain, keep the shape). It
can be objected that the semantic valencies should capture these differences in the senses but for
practical NLP applications it certainly appears to be practical to have this kind of information in CLD
in an explicit form.
a5) <local contexts>, i.e. contexts typical of the given entry, e.g. hezká dívka (pretty girl), or
šikovný chlapec (smart boy) etc., they will be obtained from the corpus.
a6) <examples or typical uses>, e.g. GU&#229;HWNQLKXY UXFHWRKROGDERRNLQWKHKDQGRWR&#254;LWKODYXWR
turn the head), we should get them from the corpus texts,
a7) <collocations> with appropriate subcategorizations, i.e. an attempt has to be made to find a
4 Vossen P et al. 1999 Final Report on EuroWordNet-2, 2D041. CD ROM, v.1, Amsterdam, University
of Amsterdam.
5 Levin B 1995 English Verb Classes and Alternations. Chicago, The University of Chicago Press.
6 Fillmore Ch, Atkins B 1998 FrameNet and Lexicographic Relevance, in Proceedings of the First
National Conference on Language Resources and Evaluation. (eds. Rubio A, Gallardo N, Castro R,
Tejada A) , vol. 1, Paris, ELRA, pp. 417-423.
121
semantic classification of the collocations. We will not go into details here but it can be shown that it
would be very useful to have e.g. the verbal collocations classified in accordance with the semantic
classes of verbs mentioned above. Similar technique can be applied to the noun collocations as well but
we are aware of the fact that this task will require a lot of corpus data and their laborious analysis.
a8) <pragmatic information> – more structured information about the register and stylistic properties of
the item including the regional information and other data, however, we would like to cover only the
basic types of this information,
a9) <origin> – i.e. short etymological information related to the item,
a10) <logtype> – this field will include the information about the logical type of the item based on
the Transparent Intensional Logic (TIL) (Materna 20007, Pala 2000, p.109-114)8. In TIL the types are
built on the ramified theory of types which, we hope, may lead to formally more consistent semantic
representations of NL expressions. This together with hierarchic hypero/hyponymic structures will
enable us to use CLD also as a part of the knowledge representation systems. We would like to try to
establish the relations between EuroWordNet Top Ontology and Type Ontology as defined within TIL.
This should yield more precise and less arbitrary semantic classifications for the semantic hierarchies,
semantic relations and semantic features as well though on the other hand we are aware that this
enterprise may also lead to some problems, e.g. it may be feasible only for some entries or parts of
speech (verbs, nouns, adjectives, adverbs).
a12) <encyclopedic information> – can be included in CLD where it would be useful or even
necessary for possible NLP applications, this may hold e.g. for the entries that are related to the
information technologies. The question is whether we should look for a kind of knowledge
representation language that would enable to represent the encyclopedic information or to approach the
problem pragmatically and to follow the present encyclopedic resources in their current form. In the
examples below we give only some arbitrary explanations but in the beginning it appears to be more
reasonable to follow the latter path.
3. Resources for CLD
Thanks to favourable situation with regard to Czech National Corpus (CNK, at Charles
University in Prague, Faculty of Arts – (abbrev. FF UK) and corpus ESO (at Masaryk University in
Brno, Faculty of Informatics – (abbrev. FI MU) we assume that the building of CLD can be based
mainly on the CNK and ESO data. Other resources would be used as well, particularly the two existing
Czech dictionaries:
1) large Dictionary of Literary Czech (66-ý)9,
2) and smaller Dictionary of Written Czech (66ý)10.
Additional resources will have to be sought as well, particularly other existing dictionaries,
especially the terminological ones. We are also aware of the fact that a well founded reader program
has to be established sooner or later that should be later closely connected with the preparation of the
New Czech Dictionary.
4. Tools
Thanks to the interesting results in the NLP research being performed both at Charles University
(Institute of Czech National Corpus, Institute of Formal and Applied Linguistics, Institute of
Theoretical and Computational Linguistics) in Prague and Masaryk University in Brno (Faculty of
Informatics, NLP Laboratory) we have at our disposal a basic set of tools that can be used for building
CLD.
Particularly, we will take advantage of the morphological analyzer AJKA (mentioned above),
parsers (DIS and GT, äi&#254;NRYi3RSHOtQVN&#234;1HSLOS-+RUiN6PU&#229;S-50)11,
desambiguators (2OLYD3HWNHYL&#254;HWDOS-8)12, corpus manager and graphical interface using
7 Materna P 2000 Type-theoretical analysis as a preparation of analyzing expressions of a natural
language. Prague - Brno, Faculty of Informatics MU, manuscript, pp.110.
8 Pala K 2000 Word Senses and Semantic Representations - Can We Have Both? in Proceedings of
TSD 2000, Berlin, Springer Verlag, pp.109-114.
9 6ORYQtNVSLVRYQpKRMD]\ND&#254;HVNpKR'LFWLRQDUy of Written Czech Language) 1960, Praha, Academia.
10 6ORYQtNVSLVRYQp&#254;H&#227;WLQ\'LFWLRQDU\RI/LWHUDU\&]HFK3UDKD$FDGHPLD
11 äi&#254;NRYi(3RSHOtQVN&#234;/1HSLO05HFRJQLWLRQDQG7DJJLQJRI&RPSRXQG9HUE*URXSVLQ
Czech. in Proceedings of CoNLL-2000 and LLL-2000, Lisbon, ACL New Brunswick, pp.219-
225.+RUiN$6PU&#229;3/DUJH6FDOH3DUVLQJRI&]HFKLQ3URFHHGLQJVRI(IILFLHQF\LQ/DUJH-Scale
Parsing Systems Workshop, COLING'2000, Universitat des Saarlandes, Saarbruecken, pp.43-50.
12 Oliva K,3HWNHYL&#254;9HWDO7KH/LQJXLVWLF%DVLVRID5XOH-Based Tagger of Czech. in
122
client-server architecture (Rychlý, 2000)13, dictionary editor and a browser based on XML format that
can handle any dictionary converted into XML. The modified version of the browser can be also used
for processing any WordNet data in a way that is now possible with Polaris database (Vossen, 1999).
Other tools include various conversion programs, programs for corpus maintenance and corpus
preparation, heuristic programs for obtaining valency frames from corpus texts, Czech morphological
database and programs for automatic word derivation that would capture the word derivation chains
like X&#254;LWWRWHDFK–X&#254;HQtWHDFKLQJ–X&#254;LWHOWHDFKHU–X&#254;LWHONDVKH-teacher) –X&#254;HQ&#234;VFKRODUO\–
X&#254;HQHF VFKRODU – výuka (tuition, lesson) etc. It has to be decided whether these data should be
included in CLD directly or rather whether they would be obtained dynamically from the
morphological module (Klímová, Pala, 2000, p.987-991)14. We touched this question above when
discussing the morphological information about the entries. In the examples given in the next section
we have not tried to show the word formation semantic information because we hope that it can be
obtained dynamically as the output of the word formation engine that is presently being built for Czech
at NLP Lab. at Faculty of Informatics MU.
5. Conclusions
In this short contribution we have presented the underlying assumptions from which building
Czech Lexical Database can start. We are aware of the fact that a number of the discussed points will
have to be elaborated more deeply and systematically to obtain fully applicable results. This explains
the fact that the examples of the entries are in several points rather tentative skeletons than full and
complete entries. However, it is our hope that the described techniques, resources and tools will allow
us to reach our goal.
5.1 An example15
An example of the entry for the Czech verb GU&#229;HWKROGfollows:
HQWU\GU&#229;HW!!
<mf <k5> <eA> <nS> <pI> <t*> <m*> <aI> <paradigm code>>
<sense:1 <def: uchopovat rukou, mít v ruce>>
<val: <ag<k1&#254;ORY Nc1>> <obj<k3Q FRc4> <ins<<k7vc6> <k1rucec6>>>
<synset: <uchopovat> <mít v ruce> >
VIHDW&#254;LQQRVW!!:1-like)
<semclass: <3.1> %(verbs of holding and keeping)
FRQWH[WVGU&#229;HWGYH HGSLVWROLYUXFH!!H[DPSOHVIURPFRUSXV
<collocation: <> <> > %(+ semantic class of a collocation)
<style: written>
<etym: … >
ORJW\SHY]WDKPH]LGY PDLQGLYLGXLUHODWLRQ-in-intension between two individuals>
<encycUXNDMH&#254;iVWOLGVNpKRW ODQHERURERWD!!
VHQVHGHIE&#234;WSHYQRXVRX&#254;iVWtMLQpKRREMHNWX!!
<val <obj<k1c4> <ins<<k7c6> <k1c6>>>
V\QVHWE&#234;WIL[RYiQ!E&#234;WXSHYQ Q!!
<sfeat <...> <mero...> %(WN-like)
<semclass: <3.2> %(verbs of holding )
FRQWH[WVRPtWNDGU&#229;t!!H[DPSOHVIURPFRUSXV
FROORFDWLRQVK HEtNGU&#229;t!!!VHPDQWLFFODVVRIDFROORFDWLRQ
<style: written>
<etym: … >
<logtype: vlastnost individua, property of an individual >>
Proceedings of TSD 2000, Berlin, Springer Verlag, pp.3-8.
13 5\FKO&#234;3.RUSXVRYpPDQD&#229;HU\DMHMLFKHIHNWLYQtLPSOHPHQWDFH&RUSXV0DQDJHUVDQGWKHLU
Effective Implementation). Ph.D. Dissertation, Brno, Faculty of Informatics MU.
14 Klímová J, Pala K 2000 Application of WordNet ILR in Czech Word-formation. In Proceedings of
LREC Conference, Athens, ELRA, pp.987-991.
15 Note that in the examples English is used to describe the fields, however, we doubt whether it makes
sense to translate Czech lexical data within the fields. We decided not give the translations and to offer
the necessary explanations during the poster session when discussing the particular points.
123
HQF\FXSHYQ QtREMHNWXVHSURYiGtOHSHQtP]DWOX&#254;HQtP!!
<sense:3 <def: zachovávat tvar>>
<val <obj<k1klouboukc1>> <forma<k1tvarc4> >>
<synset: <] VWiYDWYSHYQpPWYDUX!QHP QLWIRUPX!!
<sfeat <...> <...> > %(WN-like)
<semclass: <3.3> %(verbs of maintaining and keeping shape)
FRQWH[WVYODV\GU&#229;tID]yQX!!H[DPSOHVIURPFRUSXV
<collocations:SXN\GU&#229;t!!!UR]W tGLWSRGOHW\S 
<style: written>
<etym: … >
<logtype: vlastnost individua, property of an individual >>
<encyc: platí pro objekty jako šaty, vlasy> >
<sense:4 <def: zachovávat polohu>>
<val <ag<k1&#254;ORY Nc1>> <obj<k1W ORc4>> <mod<k6xMURYQ > >>
V\QVHWE&#234;WYHVWHMQpSROR]H!QHP QLWSRORKX!!
<sfeat <...> <...> > %(WN-like)
<semclass: <4.1> %(verbs of keeping a position)
FRQWH[WVGU&#229;HWW ORY]S tPHQ !!H[DPSOHVIURPFRUSXV
FROORFDWLRQVGU&#229;HWKODYXQDGYRGRX!!!VHPDQWLFFODVVRIDFROORFDWLRQ
<style: written>
<etym: … >
<logtype: vlastnost individua, property of an individual >>
HQF\FSODWtSURSRORKXOLGVNpKRW OD!!
VHQVHGHIYODVWQLWS GX!!
<val <ag<k1vlastníkc1>> <obj<k1majetekc4> > >
V\QVHWYODVWQLWS GXVSUDYRYDWPDMHWHN!!
<sfeat <...> <...> > %(WN-like)
<semclass: <2.1> %(verbs of possesion)
FRQWH[WVGU&#229;HW]DKUDGX!!H[DPSOHVIURPFRUSXV
<collRFDWLRQVGU&#229;HWE\W!GU&#229;HWG P!!VHPDQWLFFODVVRIDFROORFDWLRQ
<style: written>
<etym: …. >
<logtype: <vztah mezi individui, relation-in-intension>>
HQF\FYODVWQtNHPMH&#254;ORY NREMHNWHPQHPRYLWRVW!!
<sense:7 <def: >>
<val <ag<k1&#254;ORY Nc1>> <obj<k1]Yt DWDc4> >>
V\QVHWS VWRYDW!FKRYDW!!
<sfeat <...> <...> > %(WN-like)
<semclass: <5.1> %(verbs of growing)
FRQWH[WVEDEL&#254;NDGU&#229;tVOHSLFH!!H[DPSOHVIURPFRUSXV
FROORFDWLRQVGU&#229;HWGRE\WHN!!!VHPDQWLFFODVVRIDFROORFDWLRQ
<style: written>
<etym: … >
<logtype: <vztah mezi individui, relation-in-intension>>
HQF\FREMHNW\MVRX]Yt DWD!!
<sense:8 <def: rezervovat>>
<val <ag<k1&#254;ORY Nc1>> <obj<k1&#254;ORY Nc3>> <loc<k1místoc4>> >
<synset: <rezervovat, obsadit> >
<sfeat <...> <...> > %(WN-like)
<semclass: <6.2> %(verbs of reservation and booking)
FRQWH[WVGU&#229;tQiPPtVWRYSR DGt!!H[DPSOHVIURPFRUSus)
FROORFDWLRQVGU&#229;HWNRPXPtVWR!!!VHPDQWLFFODVVRIDFROORFDWLRQ
<style: written>
<etym: … >
<logtype: <vztah mezi individui, relation-in-intension> >
124
<encyc: platí pro místo v GRSUSURVW HGNXQHERYVH]QDPX!!
<sense:8 <def: oblíbit si koho, preferovat koho>>
<val <ag<k1&#254;ORY Nc1>> <obj <<k7na> <k1&#254;ORY NDF!!_Y FNtenisc4>>>
<synset: < oblíbit si koho, preferovat koho > >
<sfeat <...> <...> > %(WN-like)
<semclass: <7.2.> %(verbs of emotional attitudes, liking)
FRQWH[WVRQQDQLGU&#229;t!!H[DPSOHVIURPFRUSXV
FROORFDWLRQVGU&#229;HWQDVHVWUX!!!VHPDQWLFFODVVRIDFROORFDWLRQ
<style: written>
<etym: … >
<logtype: <vztah mezi individui, relation-in-intension> >
HQF\FSODWtSURPtVWROLGLQHER&#254;ORY NDDQ MDN&#234;REOtEHQ&#234;REMHNWSHQt]H!!
VHQVHGHIGU&#229;HWVN&#234;P!!
<val <ag<k1&#254;ORY Nc1>> <obj<k7sc7> k1&#254;ORY Nc7>> >
V\QVHWGU&#229;HWV N&#234;PGU&#229;HWVSROXE&#234;WV N&#234;PYSDUW !!
<sfeat <...> <...> > %(WN-like)
<semclass: <8.4> %(verbs of social grouping)
FRQWH[WVGU&#229;HWSDUWX!!H[DPSOHVIURPFRUSXV
FROORFDWLRQVGU&#229;HWVNRPXQLVW\!!!VHPDQWLFFODVVRIDFROORFDWLon)
<style: written>
<etym: … >
<logtype: <vztah mezi individui, relation-in-intension> >
<encyc: platí pro místo v GRSUSURVW HGNXQebo v seznamu> >
<sense:9 <def: uchopovat rukou, mít v ruce>>
<val: <ag<k1&#254;ORY Nc1>> <obj<k3Q FRc4> <ins<<k7vc6> <k1rucec6>>>
<synset: <uchopovat> <mít v ruce> >
VIHDW&#254;LQQRVW!!:1-like)
<semclass: <3.1> %(verbs of holding and keeping)
FRQWH[WVGU&#229;HWGYH HGSLVWROLYUXFH!!H[DPSOHVIURPFRUSXV
<collocation: <> <> > %(+ semantic class of a collocation)
<style: written>
<etym: … >
ORJW\SHY]WDKPH]LGY PDLQGLYLGXLUHODWLRQ-in-intension between two individuals>
HQF\FUXNDMH&#254;iVWOLGVNpKRW ODQHERURERWD!!
5. 2 Example of the entry for Czech noun hlava (head):
<entry <hlava>>
<mf <paradigm code: 47a> <<k1> <gF> <nS> <c1>> >
<sense:1 <defgenprox&#254;iVWW OD!!
<difspec: >
<val <<k1hlavac1> <k1&#254;ORY NDc2>> <<k2psíc1> <k1hlavac1>> >
<synset: <kebule> <palice> <šiška>>
VIHDWKRORW OR!PHURW ORR&#254;LQRVWYi H~VWD!!:1-like
<contexts: <lidská hlava> > %(examples from corpus)
<collocations: <c1> <c2> …<cn> > %(+ semantic class of a collocation)
<styl: written>
<etym: ... >
<logtype: property of an individual>
HQF\FSODWtSURKODYX&#254;ORY ND]Yt HWHQHERURERWD!!
<sense:2 <def: rozum, mysl>>
<difspec: >
<val <<k1hlavac1> <k7nac4> <k1fyzikuc4>> <<k2chytrác1> <k1hlavac1>> >
125
V\QVHWP\VO!Y GRPt!!
<sfeat <hyper: abstr> > %WN-like
<contexts: <chytrá hlava> > %(examples from corpus)
<collocation: <c1> <c2> …<cn> > %(+ semantic class of a collocation)
<styl: written>
<etym: … >
<logtype: vlastnost individua, property of an individual>
<encyc: sídlo myšlení, inteligence u lidí> >
<sense:3 <defgenprox: šéf, vedoucí skupiny>>
<difspec: >
<val <<k1hlavac1> <k1podnikuc2> > >
V\QVHWERVV!Qi&#254;HOQtN!!
VIHDWK\SHUKXP!K\SRSRG t]HQ&#234;!!:1-like
<contexts: <hlava mafie> > %(examples from corpus)
<collocations: <hlava rodiny> <c2> …<cn> > %(+ semantic class of a collocation)
<style: <written>
<etym: … >
<logtype: vlastnost individua, property of an individual>
<encyc: v KLHUDUFKLFNpRUJDQL]DFLILUP YOiG !!
VHQVHGHIJHQSUR[S HGQt&#254;iVWS HGP WX!!
<difspec: >
<val <<k1hlavac1> <k1kolac2>> >
<synset: <výstupek> <> >
<sfeat: <hyper: objekt> <hypo: > > %WN-like
<contexts: <hlava šroubu> > %(examples from corpus)
<collocations: <c1> <c2> …<cn> > %(+ semantic class of a collocation)
<style: written>
<etym: … >
<logtype: vlastnost individua, property of an individual >
<encyc: technical component> >
<sense:5 <defgenprox: jednotka textu >>
<difspec: >
<val <<k1hlavac1> <k1zákonac2 >>>
<synset: <oddíl, kapitola v knize> <> >
<sfeat: <holo: kniha> <mero: paragraf> > %WN-like
<contexts: <hlava 22> > %(examples from corpus)
<collocations: <> <> > %(+ semantic class of a collocation)
<style: written>
<etym: … >
<logtype: vlastnost individua, property of an individual >
HQF\FREVDKNQLK\VH&#254;OHQtQDMHGQRWN\– kapitoly, hlavy >

126
Using a corpus of school children's writing to investigate the development of
vocabulary diversity
N. CHIPERE, D. MALVERN, B. RICHARDS and P. DURAN
School of Education, University of Reading, UK
Abstract
The paper shows how a corpus-based approach can be used to investigate the
development of vocabulary diversity during the school years. The theoretical and
pedagogical motivations for the investigation are outlined and advantages of using a
corpus-based approach are discussed. The problem of the text-length effects on typetoken
ratios is presented, followed by a description of a recent mathematical solution
to the problem (Richards and Malvern, 1997). Data for the investigation consisted of
a corpus of 899 narrative essays from school children aged 8 to 15. The essays were
grouped into three three age categories (Key Stages 1, 2 and 3) and within each Key
Stage, essays were further categorised according to a maximum of eight possible
levels of writing ability as defined by the British National Curriculum for English.
An analysis was carried out for each Key Stage to determine the relationship between
level of linguistic ability and vocabulary diversity. The paper presents results from
the analysis and discusses some theoretical and pedagogical implications. Future
applications of the mathematical model to the investigation of diversity in categories
of linguistic form other than vocabulary are discussed.
1 Introduction
Computer-based corpus analysis can reveal meaningful patterns in first language development during
the school years. Results are reported here from an analysis of vocabulary diversity in the writing of
school children aged eight to fifteen. The analysis is part of a three-year funded research project which
seeks to identify quantitative measures of the written language skills of school children. The paper
begins by outlining some key theoretical and pedagogical motivations for studying children's writing.
A discussion of some of the methodological issues involved is then followed by a report of the study
and its results.
2 Theoretical and pedagogical motivations for the study
Recent policy changes in first language pedagogy in England are compelling linguists to reconsider
long-held assumptions about language development. It has been assumed that procedural grammatical
competence is attained without instruction by age five or soon thereafter. This assumption is congruent
with previous educational practice which excluded the teaching of first language grammar. However,
the recent ‘National Literacy Strategy’ now requires the grammar of English to be taught in English
schools. This policy shift, brought about by public concerns with standards of literacy, makes an
implicit but clear theoretical statement about grammatical development: that it is not complete by age
five and that it can be facilitated by formal instruction.
The British government has therefore adopted a view of grammatical development which can loosely
be described as empiricist. The empiricist view is that language is learned with feedback from
experience (e.g. Salzinger, 1975 and Sampson, 1999). However, it is a nativist axiom that competence
in language is attained without assistance very early in life (e.g. Pinker, 1994 and Marcus, 1994) and
that, as far as the purely grammatical component of linguistic competence is concerned, the role of
experience is merely to trigger innate knowledge. While empiricists might therefore regard the attempt
to teach first language grammar as fully justified, nativists might see it as misguided and redundant.
The theoretical and pedagogical importance of settling the question needs no emphasis. However, it is
not easy to tease empiricist and nativist claims apart on the basis of pre-school language acquisition
data. In view of the evidence that grammatical development continues during the school years (e.g.
Karmiloff-Smith 1986 and Perera 1986), it is necessary to extend the investigation to cover this period
as well. It is also necessary to use both experimental and observational methods of data collection. For
instance, one of the authors reports variations in the procedural grammatical competence of 18 year-old
native English speakers and relates these variations statistically to differences in academic ability
(Chipere, in press and Chipere, 1999). The literature reviews in the just cited work indicate that levels
of procedural grammatical competence are also statistically related to levels of formal education. While
127
this evidence supports the empiricist view, the necessarily tight focus of experimental studies restricts
the range of subjects and materials. Corpus-based analyses can complement experimental studies by
providing a wider coverage of subjects and materials.
Corpus-based analyses can also produce data which has pedagogical applications. For instance, corpus
analysis might reveal linguistic features which characterise good versus poor writing among school
children. It is known that skilled writers use more diverse vocabulary than less skilled writers. Corpus
analysis techniques can make it possible to measure vocabulary diversity scores for pieces of children's
writing and these scores could then enable teachers to decide how much time and effort a given pupil
should spend in improving vocabulary knowledge. Vocabulary diversity scores might also be useful for
assessment purposes. There is a growing interest in the possibility of computerised assessment of
writing and vocabulary diversity scores might inform automated assessment of lexical richness.
3 Framework of analysis
Thus there are both theoretical and pedagogical motivations for studying language development during
the school years. Some of the methodological issues surrounding such an investigation now need to be
discussed. The framework of analysis which has been adopted for the current project is that of Biber
(1988). This framework ‘is based on the assumption that strong co-occurrence patterns of linguistic
features mark underlying functional dimensions.’ (Biber, 1988:12). For instance, conversational
interaction represents a functional dimension of language which is characterised by a given pattern of
co-occurring linguistic features. This pattern differs markedly from the pattern which chacterises, for
instance, the delivery of technical information. Biber's analytical procedure therefore involves
calculating the frequencies of a large number of selected linguistic features and then deriving a set of
functional dimensions through factor analysis. Biber proposes that one application of his framework is
in composition research. It might be the case, for instance, that good and poor writing are marked by
different co-occurrence patterns. Grabe and Biber (1987, cited in Biber, 1988) found only small
differences between good and poor essays in their study. However, it is possible that large differences
will be found if a) the sample represents the whole range of writing ability and b) texts are analysed for
the whole range of linguistic features known to mark language development.
These two considerations inform the current project. It is intended to analyse the writing produced by
children who represent a wide range of writing ability. Their writing will be then be analysed in terms
of quantitative measures of language development reported in the literature (e.g. Johnson, 1944; van
der Geest, Gerstel, Appel and Tervoort 1973; Barnes, Gutfreund, Satterly and Wells, 1983; Fletcher
and Peters, 1984; Bennett-Kastor, 1988; Klee, 1992 and Snow, 1996). This literature identifies the ratio
of different types of words to the total number of words in a text, or type-token ratio (TTR) as one of
the most important indicators of language development. TTR is taken as a measure of vocabulary
diversity and it is usually expected that TTRs will be positively correlated with other measures of
language development.
4 The type-token ratio
A serious flaw in the calculation of TTR, however, has been identified by several writers from as early
as Chotlos (1944) to Biber (1988). Richards and Malvern (1997) provide an extensive discussion of the
issues and propose a solution which will be described presently. The following paragraphs summarise
the key points in that discussion.
TTR is calculated by dividing the number of different types (V) with the total number of tokens (N) in
a text. Many researchers have mistakenly assumed that the ratio is constant over a given text. Richards
(1987) shows that the ratio is closely related to text length with the following simple demonstration.
Consider a simple case of a two-word text in which the same word occurs twice. In that case, TTR = 1
type divided by 2 tokens = 0.5. Now consider the case of a three-word text in which the same word
occurs three times. Then, TTR = 1 type divided by 3 tokens = 0.3. In a four-word text with the same
word occurring 4 times, TTR = 1 type divided by 4 tokens = 0.25. Finally, in a five-word text with the
same word occurring five times, TTR = 1 type divided by 5 tokens = 0.2. Thus five texts which have
exactly the same range of vocabulary yield five different values of TTR. Additionally, longer texts will
tend to produce smaller TTRs.
Richards and Malvern show how failure to recognise this flaw has resulted in contradictory research
findings in the child development literature. Examples are cases where the text length effect produces
results which indicate a) no differences in TTRs taken from transcripts of children at different levels of
128
language development; b) lower TTRs for more advanced versus less advanced children and c) a lack
of correlation between TTR and other measures of language development. Several researchers have
long been aware of the problem, however, and have tried to correct it. Solutions have taken the form of
controls on text length or transformations of TTR. As shown below, all these solutions are either
inherently flawed or subject to practical limitations which make them inappropriate for the analysis of
children's writing.
5 Controls on text length
In the child language development literature, Stickler (1987) proposes standardising text length by
using 50 utterances taken from the middle of a transcript. However, this method does not eliminate the
text length effect, since more advanced children produce more words per utterance than less advanced
children. Thus, not only will the measure of lexical diversity be distorted, but there is a possibility that
the TTR values of more advanced children will be smaller than those of less advanced children. A
demonstration of this anomaly is provided in Richards and Malvern (1997: 26).
Another solution is to standardise the number of tokens. While this solution does eliminate text length
effects, there are practical problems. Standardising text length is practicable for a given corpus but it is
difficult to arrive at a standard text length which can be applied to all corpora. Thus the standard text
lengths have varied from 1000 tokens (Wachal and Spreen, 1973 and Hayes & Ahrens, 1988) to 400
tokens (Biber, 1988 and Klee, 1992) to 350 tokens (Hess et al, 1986) to 50 tokens (Stewig, 1994). This
variation is problematic because TTRs which are calculated on the basis of shorter text lengths will be
higher than those calculated from longer text lengths. It is therefore not possible to compare the two
sets of TTRs. The difficulty of arriving at a universal standard cannot be solved simply by consensus
because of wide variations in the lengths of transcripts from different sources. For instance, many
transcripts of child language data are much shorter than those from adult language data. Standards
based on the length of child language transcripts will therefore involve wasting a considerable amount
of adult language data and possibly reducing the reliability of the measure. The crux of the problem is
that TTR continues to fall with increasing text length and measuring TTR at any one point is inherently
unsatisfactory.
The final solution based on standardising text length to be discussed here is the Mean Segmental Type
Token Ratio or MSTTR (Johnson, 1944). This measure involves calculating the mean TTR for
consecutive equal-length segments of text. The advantage of this method over standardising the
number of tokens is that a) the size of the smallest transcript in a corpus can be used as the size of the
segment and b) nearly all the data are used. However, a problem remains in that it is not possible to
compare cross-corpus MSTTRs based on different-sized segments, since MSTTRs based on short
segments will be higher than those based on longer segments. Thus, while MSTTR might appear to be
an elegant solution, it too does not fully overcome the problem of text length.
6 Transformations of TTR
Attempts to transform TTR in various ways also fail to eliminate the effect of text length. Guiraud
(1960) divides the number of types by the square root of the number of tokens to derive root type token
ratio or RTTR. Herdan (1960) divides the logarithm of the number of types by the logarithm of the
number of tokens to obtain the Bilogarithmic TTR. Carroll (1964) divides the number of types by twice
the square root of the number of tokens to derive corrected type token ratio or CTTR. Ultimately,
however, none of these transformations overcome the effect of text length, since “any apparent
reduction of the relationship with sample size is an artefact of the change in scale and will be
accompanied by a reduction in the sensitivity of the measure due to the use of smaller units” (Richards
and Malvern, 1997: 33).
7 Mathematically modelling diversity
Most of the solutions discussed above, with the exception of MSTTR, fail to eliminate text length
effects because they do not utilise the fact that diversity in a text is better represented in terms of a
curve described by values of TTR taken at successive points along the length of the text rather than in
terms of a single value taken at one point. A more successful class of solutions has focussed on
mathematically modelling the way TTR falls with increasing token counts. A detailed discussion of the
development of these models is provided in Richards and Malvern (1997). They present an equation
which describes the family of curves obtained when TTR values are plotted against token counts.
These curves lie between the two extremes of total diversity and zero diversity. In the case of total
diversity, the number of types equals the number of tokens throughout the text and TTR = 1 at each
129
successive point along the abscissa, resulting in a straight line with a zero slope. In the case of zero
diversity, the total number of types is 1 throughout the text and TTR = 1/(N)umber of tokens for
increasing values of N. The result is a curve which falls steeply from an initial value of 1 along the
ordinate and then gradually flattens as it asymptotically approaches the abscissa. The TTR-Token
curves of different texts will therefore lie between the two extremes with increasing lexical diversity
represented by increasingly shallower slopes and decreasing diversity represented by increasingly
steeper slopes. The equation found by Malvern and Richards to describe this family of curves is:
Fig. 1
ú
ú
&#251;
&#249;
&#234;
&#234;
ë
é
- ÷
&#248;
ö ç
&#232;
&#230; 1
D
N
2 + 1
N
D
= TTR
2
1
where TTR = type-token ratio, N = number of tokens and D is a constant with serves as the index of
diversity.
8 Implementation and Validation of D
An algorithm for computing values of D from transcripts is described in McKee, Malvern and Richards
(2000). Points for a TTR-Token curve are obtained by calculating type-token ratios for increasing
values of N from N = 35 to N = 50. Each point is averaged from 100 sub-samples drawn randomly
from the text without replacement. D is then obtained through a curve-fitting procedure. The algorithm
has been implemented in a C program called vocd, also described in McKee et al, which runs on UNIX,
PC and Macintosh platforms as part of the CLAN suite of programs (McWhinney, 2000). D has been
validated in a number of analyses on corpora containing data from first and foreign language learning
and academic writing (Malvern and Richards, 2000). The current project seeks to extend the
application of D to the analysis of first language development during the school years. A description of
a study carried out as part of the project now follows. The study was concerned with discovering
patterns in the development of vocabulary diversity in school children aged between 8 and 15.
9 Background to the study
There are well-established differences in the written language abilities of school-going children. These
differences are found between and within age groups. Between-group differences suggest that language
development continues during the school years while within-group differences suggest individual
differences in language development. It is interesting to find out how such differences might be related
to objective measures of language development. Children's writing in England has been assessed by a
government body called the Qualifications and Curriculum Authority. QCA mark schemes for the
assessment of writing focus on three major aspects of written language: Purpose and Organisation,
which is concerned with discourse level aspects; Style, which is concerned with sentence structure and
vocabulary and Punctuation, which is concerned with the use of punctuation conventions and spelling.
These criteria are applied by markers who give a global score for each of the three aspects. The three
scores are then added up and used to classify each script in terms of eight levels of writing ability. The
assessment is therefore qualitative in nature. However, QCA has recently developed a quantitative
instrument consisting of a set of coding frameworks for writing.
The frameworks are used by trained markers to count the frequencies of selected features in 100-word
samples of scripts. These features include correct and incorrect uses of various punctuation marks,
types of spelling errors, word tokens belonging to various word classes, subordinate and co-ordinate
clauses and so on. The fact that the coding is done manually limits the size of the sample per child and
the variety of features which can be studied. The current project grew out of an attempt by one of the
authors to automate the coding process. The study reported here measured vocabulary diversity as a
first step towards automated analysis.
10 Aims of the study
The primary aim was to analyse the lexical diversity of school children and gauge the extent to which it
is sensitive to age and ability level.
130
11 Materials
899 narrative essays at least 50 words long were analysed. The essays were obtained from various
schools in England. They cover a cross section of three age groups referred to as Key Stages 1, 2 and 3
in the English education system (i.e. age groups 8, 11 and 14 years) and seven (out of a possible eight)
levels of writing ability. All the students were asked to write a narrative essay beginning with the
sentence ‘The gate was always locked, but on that day it was open …'.
12 Data Preparation
10 markers assigned a score to the scripts on the basis of National Curriculum Level descriptors (see
QCA 2000 for instance). The markers were unaware of the age or ability level of the pupils. Scores for
each essay were assigned separately by least two markers and later averaged to obtain the final score.
In a few cases, scores from different markers diverged considerably and the final score was decided
through negotiation. The final score was then used to assign each script to one of eight National
Curriculum Levels of writing ability. A breakdown of the numbers of scripts in each Key Stage and
Ability Level are shown on Table 1 in terms of the percentage of the total number of scripts in each
category. The table needs some explanation. Data for each Key Stage is presented in a column which
is further divided into two columns. The first of these columns, which should be read vertically, shows
the percentage of pupils in that Key Stage who were assigned to the different levels. The second
column, which should be read horizontally, shows the percentage of pupils in each Level who belonged
to different Key Stages.
Table 1 % of scripts in each Key Stage and Level
Key Stage 1 Key Stage 2 Key Stage 3 % of all scripts
% KS1 % Level % KS2 % Level % KS3 % Level
Level 1 11 92 1 8 0 0 4
Level 2 68 76 19 21 2 2 31
Level 3 19 30 30 47 15 23 23
Level 4 2 3 36 57 25 40 22
Level 5 0 0 13 33 27 68 12
Level 6 0 0 1 4 24 96 6
Level 7 0 0 0 0 7 100 2
% of all scripts 32 - 43 - 25 - 100
The graded scripts were converted into machine-readable form by a typist. It was necessary to correct
spelling errors in order to prevent spelling errors from being treated as different types and thereby
inflating the token counts of poor spellers. Spelling errors were corrected using a special utility
program. The procedure for correcting spelling errors was as follows.
Firstly, a list of all the words in the scripts was compiled. Any words found in the list which were not
also found in a dictionary list were considered as potential spelling errors by the program. It was then
up to the human editor to decide if a specific word was indeed a spelling error and if so, what the
correct spelling ought to be. Instances of spelling error in the corpus were then sought, found and
edited through a search and replace dialogue box. This method had the advantage of speed over manual
correction involving reading through the scripts. In addition, the method provided assurance that all the
spelling errors flagged by the program were accounted for. However, there are at least two
disadvantages with the method. Firstly, cases where correctly spelled words were used incorrectly,
such as homophones, were missed. However, the margin of error thus incurred was deemed acceptable.
Secondly, the scripts were altered in such a way that the original spelling could not be recovered.
Refinements will be made to the utility program in future to overcome both problems.
13 Analytical Procedure
Essays were analysed using vocd (Malvern and Richards, 2000) via the CLAN interface. Memory
limitations in the software meant that the scripts could only be analysed in batches of fifty at a time.
After all the scripts had been processed, the output files from vocd were concatenated into one file and
another utility program was used to extract values of D for each essay and produce a spreadsheet of all
the results.
131
14 Results
D values from all 899 scripts were subjected to a 2-way ANOVA with Key Stage and Level as the
independent variables. Main effects were obtained for both Key Stage F(2) = 221.8, p<0.001 and Level
F(2) = 92.965, p<0.001. Mean values of D are plotted on Figure 2 and standard deviations of values D
are shown on Table 2 by Key Stage, Level, by Key Stage collapsed over Level and by Level collapsed
over Key Stage.
Figure 2
There was no significant interaction between Key Stage and Level F(7) = 1.344, p<0.226 and Level
appeared to have a greater effect size than Key Stage, judging from the mean square error generated by
each factor: the mean square error due to Level = 419.866 while that due to Key Stage = 454.406.
Table 2 Standard Deviations by Key Stage and Level
A surprising result was that Key Stage 2 consistently obtained higher mean values of D than Key Stage
3. However, the difference between Key Stage 2 and 3 was not significant. A post-hoc Tukey test
shows that while there are significant differences between Key Stage 1 and Key Stage 2, p<0.001 and
between Key Stage 1 and 3, p<0.001, there are no significant differences between Key Stage 2 and 3,
p>0.5. Additionally, while there are significant differences at the p<0.01 level between each of Levels
1, 2, 3 and all other levels, there were no significant differences between Levels 4, 5, 6 and 7. (NB.
There are only 3 data points contributing to the very high mean of D in Key Stage 2 Level 6 and this
high value should not therefore be treated as a significant trend).
Key Stage 1 Key Stage 2 Key Stage 3 Level stdev
Level 1 12.376 8.006 - 12.351
Level 2 15.762 21.927 6.522 18.764
Level 3 16.426 20.833 15.809 20.448
Level 4 19.344 25.835 16.335 24.012
Level 5 - 23.535 15.693 19.969
Level 6 - 40.931 17.985 21.38
Level 7 - - 14.370 14.37
Key Stage stdev 16.517 25.866 17.753 -
Mean Values of D for each Key Stage
0
20
40
60
80
100
120
140
L1 L2 L3 L4 L5 L6 L7
Level
Mean D
KS1
KS2
KS3
132
15 Discussion
The results indicate that vocabulary diversity is significantly related to age and writing ability. Of the
two factors, writing ability appears to be have a greater effect size. Rather surprisingly, KS2 obtained
higher scores of diversity than KS3. The reasons for this are not clear. One possibility is a dip in
performance which is known to occur at Key Stage 3. This dip has been variously ascribed to the
trauma of moving from primary school to secondary or school or to maturational influences. However,
there might also be a linguistic explanation. It may be that older pupils produce more coherent
discourse which requires a greater level of lexical repetition and therefore results in lower diversity
scores. If that possibility were true, then it might indicate a limit in the sensitivity of vocabulary
diversity as a measure of writing ability. However, these comments are all purely speculative. To
explain the result more fully, it is necessary at least to analyse Key Stage 4 scripts and obtain a wider
view of the trend.
The effects of writing ability and age do not in themselves discriminate between different theories of
language development, since few would deny that vocabulary grows during the school years and that
rates of growth may differ across individuals. However, these effects do raise significant questions for
deeper investigation. It has been proposed that vocabulary growth during the school years can be
accounted for in terms of the development of derivational morphology (Anglin, 1993). By analysing
the co-occurrence of morphologically related forms in a text, a future study might determine whether
increasing knowledge of derivational morphology accounts for the observed increases in vocabulary
diversity. If lexical diversity is found to be an effect of morphological productivity, then the observed
effects of age and writing ability would suggest that morphological knowledge grows gradually during
the school years but at different rates for different individuals. That observation might have both
theoretical and pedagogical implications which would need to be explored.
A second question for further research is also suggested. Inasmuch as considerable grammatical
information is now stated in lexical terms, current linguistic theories posit a close relationship between
the lexicon and sentence level grammar. This relationship is not a purely descriptive artefact. Bates and
Goodman (1997) have shown that there is a close relationship between lexical and grammatical
development in first language acquisition, aphasia and on-line sentence processing. It has also been
shown, by means of both corpus and on-line sentence processing data, that syntactic structures are
often intimately associated with specific lexical properties (eg. MacDonald, 1994 and Trueswell 1996).
The question which arises is whether the differences in lexical diversity reported here might be related
in some way to differences in sentence level grammatical development. A syntactically annotated
corpus would make it possible to address the question. If such a relationship does exist, there would be
theoretical and pedagogical implications to explore.
16 Conclusion
The paper has shown that the study of language development during the school years is a rich area for
corpus-based investigation. It can help to discover objective developmental trends and it can also
yield results of potential pedagogical value. It also offers methodological challenges for the corpusbased
approach. It was shown how an apparently simple measure, vocabulary diversity, requires rather
sophisticated modelling. The use of mathematical modelling to facilitate quantitative analysis of
corpora will no doubt increase as linguistics develops further as a quantitative discipline. The question
arises whether the model described here can be generalised to the study of diversity in forms of
language other than vocabulary. For instance, syntactic annotation would allow the measurement of
syntactic diversity while pragmatic annotation would allow the measurement of the diversity of
pragmatic functions in a text. If these kinds of analysis yield useful results, the analysis of diversity
could well provide a general method of measuring linguistic productivity for various pedagogical,
clinical and other applications.
References
Anglin J 1993 Vocabulary development: a morphological analysis. Monographs for the Society for
Research in Child Development. Serial No. 238, 58(10).
Barnes S, Gutfreund M, Satterly D, Wells CG 1983 Characteristics of adult speech which predict
children's language development. Journal of Child Language 3(10): 65-84.
133
Bates E, Goodman J 1997 On the inseparability of grammar and the lexicon: Evidence from
acquisition, aphasia and real-time processing. In Altmann G (Ed.), Special issue on the lexicon,
Language and Cognitive Processes, 12(5/6): 507-586.
Bennett-Kastor T 1988 Analyzing children's language: methods and theories. Oxford, Blackwell.
Biber D 1988 Variation across speech and writing. Cambridge, CUP.
Carroll J 1964 Language and thought. Englewood Cliffs, Prentice-Hall.
Chipere N (in press) Variations in Native Speaker Competence: Implications for First Language
Teaching. Language Awareness.
Chipere N (1999) Processing Embedding in Humans and Connectionist Models. Unpublished PhD
thesis. Cambridge University.
Chotlos J 1944 Studies in language behavior: IV. A statistical and comparative analysis of individual
written language samples. Psychological Monographs 56, 75-111.
Fletcher P, Peters J 1984 Characterizing language impairment in children: an exploratory study.
Language Testing 1, 33-49.
Guiraud P (1960) Problemes et methodes de la statistique linguistique. Dordrecht, D. Reidel.
Grabe P, Biber D 1987 Freshman student writing and the contrastive rhetoric hypothesis. Paper
presented at SLRF 7, University of Southern Carlifornia. Cited in Biber D 1988 Variation across
speech and writing. Cambridge, CUP.
Hayes D, Ahrens M 1988 Vocabulary simplification for children. Journal of Child Language 15(2):
395-410.
Herdan,G 1960 Type-token mathematics: A textbook of mathematical linguistics. The Hague, Mouton.
Hess C, Sefton K, Landry R 1986 Sample size and type-token ratios for oral language of preschool
children. Journal of Speech and Hearing Research 32: 536-40.
Johnson W 1944 Studies in language behavior: I. A program of research. Psychological Monographs
56: 1-15.
Karmiloff-Smith A (1986) Some fundamental aspects of language develoment after five. In Fletcher P,
Garman M (eds) Language Acquisition. Cambridge, CUP, pp 455-474.
Klee T 1992 Developmental and diagnostic characteristics of quantitative measures of children's
language production. Topics in Language Disorders 12(2): 28-41.
McDonald M (1994) Probabilistic Constraints and Syntactic Ambiguity Resolution. Language and
Cognitive Processes 9: 157-201.
McKee G, Malvern D, Richards B 2000 Measuring vocabulary diversity using dedicated software.
Literary and Linguistic Computing 15(3): 323-338.
MacWhinney B (2000) The CHILDES Project Vol 1: Tools for Analysing Talk – Transcription Format
and Programs. New Jersey, Lawrence Erlbaum.
Malvern D, Richards B (2000) A new method of measuring lexical diversity in texts and conversations.
TEANGA 19: 1-12.
Marcus G (1993) Negative evidence in language acquisition. Cognition 46: 53-85.
Perera K (1986) Language acquisition and writing. In Fletcher P, Garman M (eds) Language
Acquisition. Cambridge, CUP, pp 494-518.
Pinker S 1994 The language instinct: How the mind creates language. New York, Morrow.
QCA (2000) English Tests: Mark Schemes. London, HMSO.
Richards B Malvern D 1997 Quantifying lexical diversity in the study of language development. The
University of Reading, The New Bulmershe Papers.
Richards B (1987) Type/token ratios: what do they really tell us? Journal of Child Language 14: 201-9.
Salzinger K (1975) Are Theories of Competence Necessary? In Aaronson D, Rieber R (eds)
Developmental Psycholinguistics and Communication Disorders. Annals of the New York Academy of
Sciences 263: pp 178-196.
Sampson (1999) Educating Eve. London, Continuum International.
Snow C 1996 Change in child language and child linguists. In Coleman H, Cameron L (eds) Change
and language. Clevedon, BAAL in association with Mulitlingual Matters, pp 75-88.
Stewig J 1994 First graders talk about paintings. Journal of Educational Research 87(5): 309-16.
Stickler K 1987 Guide to analysis of language transcripts. Eau Claire, Thinking Publications.
Trueswell J 1996 The role of lexical frequency in syntactic ambiguity resolution. Journal of Memory
and Language 35: 566-585.
van der Geest T, Gerstel R, Appel R, Tervoort B 1973 The child's communicative competence:
language capacity in three groups of children different social classes. The Hague, Mouton.
Wachal R, Spreen O 1973 Some measures of lexical diversity in aphasic and normal language
performance. Language and Speech (16): 169-81.

134
Approaching irony in corpora
Claudia Claridge (Greifswald)
1. The problem of irony
1.1 In pluribus unum?
What actually is irony? We all usually recognize an ironic utterance when we come across one,
but explaining why we do that and how irony works is not easy. The English lexeme “irony” is
polysemic (cf. OED), covering among others the figure-of-speech sense, irony of situation and Socratic
irony (pretense, dissimulation). Conversational irony is also a very varied phenomenon: its force ranges
from comic irony to more offensive sarcasm (Leech 1983: 143) and it can be realized in many different
surface forms, which means that practically any utterance (depending on the context) can be used for
the purpose of irony. This makes it interesting, but also hard to grasp, in particular for corpus linguists.
It has long been realized that the traditional definition of irony as an utterance “in which the
intended meaning is the opposite of that expressed by the words used” (OED) is inappropriate for
characterizing, let alone explaining, the concept. Let me illustrate this with some examples taken from
Sperber/Wilson (1981: 300), set in the context of two people caught in a downpour, one of them
producing one of the following utterances:
(1) What lovely weather.
(2) It seems to be raining.
(3) I'm glad we didn't bother to bring an umbrella.
(4) Did you remember to water the flowers?
Only (1) and (3) can be seen to express a meaning opposite to the one really intended or to the
actual state of affairs (itself an important distinction). However, (2) and (4) in their literal sense are
highly irrelevant remarks in the given context and thus force an ironic interpretation.1 But their literal
meaning is not thereby cancelled, demonstrating that irony is not a clear example of non-literal
language (such as metaphor). (2) achieves its inappropriateness both by stating the blindingly obvious
and by doing it through understatement. In fact, irony is parasitic on other rhetorical means, also using
hyperbole, rhetorical questions, metaphors, paradoxes, shifts of register/style (e.g. excessive politeness)
etc. for its effect (Barbe 1995, Leech 1983: 82). Thus, these phenomena can serve as irony signals in
the same way as facial expressions, intonation, laughing etc. sometimes do. Yet irony signals
themselves are not constitutive of irony (Lapp 1992: 29): utterances without any such clear signals can
be ironic (cf. 4), while utterances containing them can be absolutely serious. Some (few?) instances of
irony, however, signal themselves rather clearly by virtue of being lexicalised, e.g. a fine friend (called
common irony by Barbe). Moreover, ironic inconsistency can formally manifest itself on several levels,
being inherent in one word only (cf. lovely in (1)), in the whole statement/utterance (cf. (4)), and/or in
the general incompatibility of statement and context (cf. Barbe 1995: 29).
Both the speaker's and the hearer's perspective have to be taken into account in a discussion of
ironic utterances. On the one hand, ironic intention is part of the speaker meaning, but the hearer has to
recognize the irony for it to be successful. It is important to keep these two aspects apart, because of the
fact of unintentional irony (Barbe 1995: 78, Hamamoto 1998: 262), which is inferred by the hearer. It
may be that this inference is connected to the irony-of-situation meaning.
I will not answer here the question posed at the beginning (which would be most daring after such
a brief discussion!), but what seems to apply to all instances of irony is that it points to some
inconsistency or incongruity which the hearer is expected to notice, this inconsistency being located
either on the level of the utterance or on the level of both utterance and situation (Barbe 1995: 16). This
inconsistency can and often does, but need not, consist in an opposition between the literal and some
other meaning. All in all, irony may be seen as a polysemous concept, with a range of more or less
prototypical realizations and various meanings or functions.
1.2 The importance of being ironic
People must be using irony for some purpose that cannot be served equally well by non-ironic
expressions. There must be a clear gain through surplus meaning to outweigh the riskiness of the
strategy (namely that the irony, and thus the real force of the utterance, may not be understood).
1 I will ignore Sperber/Wilson's interpretation of these examples as echoic for the moment.
135
Basically, irony is or transports an attitude, in the majority of cases a negative one. Ironic
utterances are (proto)typically used to convey some criticism on the part of the speaker (Levinson
1983: 161), if in an indirect way. Dews et al. (1995) have experimentally uncovered the following
reasons for selecting irony: (i) to be funny, (ii) to soften the edge of an insult, (iii) to show oneself in
control of one's emotions, and (iv) to avoid damaging one's relationship with the addressee. While (ii)-
(iv) all perform face-saving functions for either the speaker, or the hearer, or both, (i) highlights the
strong connection of irony to humour in general (cf. Attardo 1994: 7 for a placement of irony within
the semantic field of humour). Of course, jocular language can also be seen in face, i.e. politeness,
contexts (cf. Brown/Levinson 1987). However, I think the underlying connection between (i) and the
other functions is rather to be found in the aggressive element inherent in humour and laughter, which
links up with the FTA implicit in (ii)-(iv). Thus, a certain balance between aggression and social
control seems to be the hallmark of irony. Depending on the cultural context, the general balance will
be tilted more towards one or the other of these poles. Sarcasm, which has to be seen as a type of irony,
sacrifices social control for effective aggressive attack – indicating that restricting irony's function to
face-saving might not be appropriate.
In more concrete terms, ironic utterances can take the following forms. A very common form is
saying something positive to express a negative attitude, i.e. to voice some criticism. Opposite meaning
is not the only method for encoding ironic criticism, however (cf. the examples above). On the other
hand, there are also ironic compliments (Dews et al. 1995: 348), in which a negative statement is used
to express a positive value judgement (praise-by-blame). This makes the (supposed) compliment at the
very least ambivalent, as it colours it with some negative touch (e.g. envy); some kind of unexpressed
criticism is hidden within the compliment. Ironic utterances can be directed at (i.e. take as their victim)
the hearer(s), some third (absent) party, the speaker him-/herself or the general situation. The latter case
seems to favour the humorous function of irony. Speaker-directed irony can be seen as a means of
emotional release.
1.3 Theoretical treatments of irony
This is not the place to go into detail about theories of irony, but some points should be
mentioned briefly. Most modern theories have tried to move away (more or less successfully) from
traditional rhetoric's characterization of irony as meaning something different to what is said, in
particular the opposite of what is said (i.e. substitution theories) (cf. Lapp 1992).
Pragmatic treatments promise to be the most successful approaches, as irony is after all a
conversational phenomenon constituting meaning in context and in interaction. Grice (1975: 312)
described irony as flouting the first maxim of Quality, thereby producing the intended meaning by
conversational implicature. Sperber/Wilson (1981), finding fault with Grice's approach, propose the
interpretation of ironic utterances as echoic mentions of previous propositions, adding the speaker's
evaluation of the latter. The sources of the echo are extremely varied and thus vague – actual
utterances, thoughts, opinions, real or imagined sources etc. (Sperber/Wilson 1981: 310) – presenting a
problem for the usefulness of the theory in the analysis of real data (and also of its explanatory power).
The pretense theory of irony (Clark/Gerrig 1984) assumes that the ironic speaker is pretending to be
someone else who might hold the opinion expressed in the ironic utterance, mocking both the opinion
and the person it is attributed to. Both of the latter theories foreground the attitude of and evaluations
made by the speaker (the psychological aspect), which is certainly very important for irony. As hinted
at above, irony has also been seen in the context of politeness in language. Brown/Levinson
(1987:221ff) list irony as one of their off-record strategies, giving it a face-saving function. Leech
(1983) even proposes an Irony Principle (“If you must cause offence, at least do so in a way which
doesn't overtly conflict with the P[oliteness]P[rinciple], but allows the hearer to arrive at the offensive
point of your remark indirectly, by way of implicature”, p. 82), which helps to avoid open conflict and
thus rescues the Politeness Principle on a different level. The last two theories emphasize more the
social functions of irony.
Analysing irony in pragmatic terms of course presupposes the existence of data, which leads us to
the next problem.
2. The problem of data
What strikes one when looking through many publications on irony is, on the one hand, the lack
or rarity of authentic data, and, on the other, the little linguistic context given for the cited examples.
Data found in the literature generally falls into three classes:
136
(i) Invented instances of ironic utterances, based on the researcher's intuition of what are good
examples of irony. This is especially common in approaches in some way connected to the
Gricean paradigm.
(ii) Experimental data: psychological/psycholinguistic results concerning the processing, understanding
and evaluation of irony. Experiments are usually based on invented examples, cf. (i).
(iii) Authentic data sampled by tape recording or, in some cases, taken down as notes by the author.
Each of these has its problems. As we have seen above, irony is a rather multi-facetted
phenomenon and it is not likely that one researcher would come up with a sufficient range of ironic
instances to mirror real-life variety. Individual differences of irony perception, or ironic preferences,
will also play a role here. Thus, approach (i) is rather narrow and may not be adequate for supporting a
comprehensive theory of irony. In so far as (ii) is based on data of the first kind, it shares the problems
mentioned above. However, the resulting experimental data is certainly very important and not
invalidated by its restricted basis. The third approach, in particular if pursued within a conversation
analysis context, seems much better fitted to capture the concept of irony in all its variety. The
problems here are that this method might not produce enough data (in terms of sheer frequency) and
that the resulting data might not have a wide enough range (especially in the sociolinguistic sense). The
latter concern is based on the fact that one will usually ask friends and (good) acquaintances for
permission to make tape recordings (i.e. people of the researcher's own social and educational
background)2 and, potentially more disturbing, one might subconsciously prefer to approach those
people who have a certain reputation for irony (to increase the likelihood of sufficient data).
The second problem is the question of the contextual embedding of the examples cited in the
literature. This is a serious problem, in view of the strong contextual dependence of irony. A short, oneto-
two sentence long description of the larger extralinguistic context is usually given. As to the
linguistic context, it is often only the ironic utterance as such that is provided; where there is more, it is
usually the preceding utterance which triggers the ironic retort. The larger conversational context, in
particular the response to the ironic utterance3, is notably absent. As important functions of irony
include criticism on the one hand and getting the criticism across in a face-saving manner on the other,
the response would seem to be an important part of the data. Furthermore, a larger preceding context
might be interesting with respect to the echoic-mention theory, at least in so far as the echo can relate to
an actual utterance not too far removed in time.
Basically, I think the most promising approach would be a mixture of several approaches,
combining data collection and psycholinguistic methods (experiments, questionnaires).4 A clear
emphasis should be placed on a large amount and a wide range of authentic data, however. While the
researcher's own recordings and field notes are valuable (partly because of better access to contextual
knowledge), it would be a shame to ignore the opportunity offered by corpora to broaden our picture of
irony in use, with regard both to the possible variety of ironic expressions and to the social spread of
users of irony.
3. Irony and corpora
In 1996 McEnery/Wilson (98f) stated that there had been relatively little corpus-based research in
pragmatics and discourse analysis so far, something which seems to me still to hold true today. This is
doubtless due to the fact that corpus-linguistic methods and pragmatic problems do not on the whole
meet easily: it is hard to have the computer automatically search and find a phenomenon that does not
have a (range of) corresponding surface structure(s). Irony is no exception here: as stated above, almost
any utterance can be used ironically and there are no unambiguous irony markers. Nevertheless it is
possible to find irony in corpora and some methods together with their results will be presented below.
As I am interested in irony as a (mostly) conversational phenomenon, I will restrict myself to
spoken corpus data here. I will draw on the spoken part of the BNC, the Wellington Spoken Corpus
(WSC) and the Santa Barbara Corpus of Spoken American English (SBC, part of ICE-USA)5 in order
to cover several varieties of English.
2 The largest-scale study that I am aware of in this respect is Hartung (1998) on irony in conversational German. He used tape
recordings of 14 conversations with a total length of 18.5 hours, which yielded the amazing amount of 302 instances of irony.
The recorded participants were all young adults with a middle-class background and university education (ibid. p. 63-69).
3 Sometimes this is added in the explanation in the surrounding text.
4 It might also be a good idea to feed authentic examples into these experimental methods.
5 A partial pre-publication distributed by the Linguistic Data Consortium. The WSC is published on the ICAME CD.
137
3.1 “Explicit” irony
The expression “explicit irony” is taken from Barbe (1995), who uses it to describe phrases such
as “it is ironic that …” when employed by writers (of letters to the editor, in her data) to comment on
some fact or course of events. She hypothesizes that these uses are based on the writers’ understanding
of implicit irony (ibid. 142) and thus can help find out about people's concept of irony. This might in
fact be so, and looking for expressions including terms like ironic in spoken contexts might also lead
one to situations where irony is actually being employed. The presence of such a metacomment might
even mean that something in the communication is going ‘wrong', which is potentially interesting, as
understanding irony can be a problem. Thus I looked for the following words in the spoken corpora:
irony/ies, ironic, ironical, ironically, sarcasm, sarcastic, sarcastically. While the SBC yielded no instances6,
there were 11 in the WSC and 114 in the BNC, on the whole more from the irony-field than
from the sarcasm-field. Irony, ironical and ironically are most commonly used in the irony-of-situation
meaning similar to Barbe's examples and as in (5):
(5) The problem is, of course, that we <pause> the irony is that we are now in a period where we have a much bigger potential
workforce who we are not employing as we might. (BNC KRE)
This is a sense that seems to be rather salient in an English, or at least British, context. In contrast
to Barbe, however, I doubt whether that particular usage tells us anything much about people's
understanding of verbal irony. Occurrences where people comment on verbal irony are therefore
potentially more enlightening. (6) is about the inappropriateness of irony in some situations or with
some kinds of people, here by a café manager (AU) to her customers.
(6) BY: <[>i don't <.>see</.> well you say you're nice to them i can't</[></{> imagine you being nasty with them
<latch>
AU: well short or abrupt or something or just not NICE not smiling <,,>
BY: <O>exhales</O> yeah <,> i suppose it's times like that you just <.>even</.> you can't even be ironic with them
can you you <.>can't</.> <latch>
AU: yeah <latch>
BY: you can't even <.>say</.> <latch>
AU: oh well there's <{><[>SO much work to do</[>
BY: <[><laughs>god i'm grumpy today you</[></{> can't even say</laughs> <,> <{><[>that's all</[> (WSC
dpc207)
I am not quite sure whether the third statement after the ironic instance (italicised) is intended as
an example of an ironic remark. An instance similar to (6) is presented by a lecturer who had to restrain
her “sarcastic tongue” because her “students cannot handle it because it's a different kind of situation
there are different vulnerabilities going on in that situation so i had to learn to challenge students in
ways that were constructive” (WSC mul030). If she was talking about school students, especially very
young ones, that would not be surprising, but she seems in fact to be talking about university students.
Another instance clearly mentions the criticising function of irony or sarcasm:
(7) DA: it's taken quite a long time to get there but they seem to be heading in the right direction at least i like to think that a
couple of sarcastic references er from various people of this university actually helped them along the right way pointed
out the errors of their ways i <.>the</.> i think they've been getting it from quite a number of different sides actually
(WSC dgz064)
In contrast to ironic(ally), sarcastic(ally) was found in contexts where verbal irony was actually
present or talked about. This might indicate that sarcasm (being more aggressive) is the more salient
type of irony and that as a consequence the word might be extended to cover all forms of irony,
including the less critical ones. The examples could lead one to the latter conclusion, as the comments
referred to often do not sound very aggressive. Sarcastic is used to report other people's use of irony
(8) or to (meta)comment on one's own use of it, as in (9), where it refers to an incongruity between a
statement and the actual state of affairs.
(8) There was as much in the loft as there was on the whole first floor. <pause> My the boss said a few things under his
breath, came down the steps and we went back into the van. Cos we had to repack it you see because it was going to take
a-- it was going to take probably not <pause> it was going to go right to the end with this lot on. So up it went higher.
<pause> Not the six foot he'd thought, seven or eight foot we went up and started to pack away again. A little bit sarcastically
my boss said er to the owners There's nothing else that we haven't seen is there? <laugh>. <pause> Well have
you seen the gar--? Well your wife said there w-- Oh well you'd better come and have a look, you see. (BNC KNC)
6 This is not surprising, as it is very small with only 14 conversations.
138
(9) Ken (…) By the way er we may er be gratified to know that erm <pause> thankfully Labour lost the ninety eighty
seven election but in, in it's manifesto in nineteen eighty seven Labour proposed annual elections in local
government.
PS000 Hm.
Ken don't <pause> recall, well in fact I'm being sarcastic because I know for a, we know that it wasn't in the <voice
quality: laughing> manifesto <end of voice quality> for nineteen ninety two. <pause> What the impact of annual
elections for local government would be on turn out is difficult to say. I suspect disastrous. (BNC F7T)
Cases of (potential) misunderstanding are another environment for sarcastic, as is visible in (10),
where a statement is first taken as ironic by the hearer (cf. unintentional irony), but then the
misunderstanding is cleared up.
(10) Caroline <pause> Don't walk away cos I'm connected to you. Okay we're going to <-|-> <unclear> canteen <-|->.
Lyne <-|-> You know what you have to <-|-> do.
Caroline And <unclear> food. Lucy's looking very nice in a very nice skirt today. <laugh>
Lyne You being sarcastic?
Caroline No I <unclear> she's looking very nice in a nice skirt. No I like it so I said it.
Lyne Oh. (BNC KP3)
Caroline's supposed ironic statement in fact contains possible irony markers that might have led
Lyne to that interpretation, namely the great positive emphasis (twice very nice) and the laugh
following her statement. If Caroline is honest in her denial, this is an example of irony markers being
present in non-ironic statements. Of course, Lyne may also have used mutual knowledge (Caroline's
usual assessment of Lucy, previous talks about Lucy) in reaching her conclusion; however, this is not
accessible via the corpus data. Note that this is an example where missing following context might
enforce an ironic meaning that is not there.
Finally, a rather unfortunate fact has to be mentioned. It is often the case in the BNC (as in (11))
that <unclear>-markup precedes or follows (or both) the mention of irony/sarcasm, so that it is
impossible to put any sensible interpretation on these instances.7 (11) seems to present a harmless,
jocular context between friends, in that case contrasting with the offensive meaning of sarcastic.
(11) Cassie Erm <pause> what is this tape? <pause> Have you got band practice tonight then Dan?
PS6U1 <unclear>
Cassie <laugh> <pause> Don't be sarcastic with me matey.
PS6U1 Give me a kiss.
Cassie Are you alright?
PS6U1 Yeah, think so. <laugh> (BNC KP4)
3.2 Commonly used ironic expressions
Certain words/expressions are more likely to occur in ironic remarks than others. Kreuz/Roberts
(1995: 25), for example, have drawn up a “random irony generator” consisting of hyperbolic
combinations of adverbs such as absolutely, certainly, perfectly, really with “extreme positive
adjectives” like adorable, brilliant, gorgeous, magnificent, the best … in my life etc. If one randomly
asks native speakers for their intuition on which words are often used ironically, they suggest
expressions like great, how funny, big deal, haha, you don't say etc. - again positive terms in the
majority. Seto (1998: 244ff), who assumes that a semantic reversal from an overdose of positive
meaning to the negative opposite is taking place, also lists some linguistic devices that are typical of
irony: single words such as genius, or miracle, modifiers/intensifiers (real, nice, such), superlatives,
exclamations, focus topicalisation and excessive politeness. These examples obviously reflect the most
typical or most easily noticed type of irony, that of blame by praise/saying something positive to
convey criticism. The examples just mentioned partly overlap with Barbe's (1995: 22ff) concept of
“common irony” for phrases which always, even out of context, induce an ironic interpretation, e.g.
real winner, fine friend, likely. In contrast, “nonce irony” (ibid. 18ff) represents original creative
instances of irony which are not in habitual use. The existence of such commonly used ironic
expressions of course makes it possible to search for these in corpora, using the examples found in the
literature, examples provided by asking native speakers, and words/phrase marked as potentially ironic
in dictionaries. An OED search turned up the following as labelled “frequently ironic/sarcastic” or the
like:
(12) ironic: that's a laugh, knight in shining armour, what's the big idea?, Great British people, it's a great life, mein Herr,
Merrie England(er), outpost, prop (up), go/pass etc. to one's reward, royal, sage, sir, sweetness and light, I will thank you
7 This problem also applies to examples of the type discussed in 3.2.
139
to do so-and-so, elegant variation, to have a good etc. war, in his/her etc. wisdom, wise guy, wonders will never cease, yet
(clause-final), favour
sarcastic: erudite, fine gentleman, firm (n.), genteel, martyr, merciful, oh-so, pain, sacred
I decided to do pilot searches using some of the terms mentioned above, concentrating mainly on
the overly positive ones. From the OED list I only checked clause-final yet and oh(-)so, as most of the
others did not seem very relevant for spoken contexts. None of these two has yielded any ironic results
so far, though there are so many instances in the BNC that I would not want to be definite on this point
at the moment. The often quoted fine friend also yielded no instances in any of the corpora; there were
a few possibly ironic examples, such as doing a fine job (BNC KCN), be on (sic) fine form (BNC
KDA), but on the whole the corpus evidence has not pointed to an ironic specialization of fine so far.
Let me now discuss a few of the examples found. (13) is taken from a conversation between
Susan and Anne (colleagues) about the state of the Royal family and the way they are treated by the
press. It is an example of word level irony (great), which is supported by its cotext, the conglomerate
of positive (and somewhat belittling) descriptive terms, the short form of the name and the superfluous
information about Philip's marital status. The irony, which has a benevolent mocking character (and is
after all not directed at anybody present in the situation), lies in the incongruity between the epithet and
the general public evaluation of Prince Philip.
(13) Susan: … but this Royal Family, I E the, the Royal Family with which I grew up and Anne did were really sweet nice
little Windsors who behaved themselves <pause> and that was what was, went into our psychic and there was the odd
crack about Phil the Great who's the Queen's husband, you know and how he perhaps had an eye for the ladies, but there
was never any photographs of him being <unclear> or any evidence that it might have gone further than that (BNC J40)
In contrast, (14) represents a more confrontational form of irony, with 12-year-old Paul criticizing
his parents, Kevin and Ruth, for not being very helpful. Several features contribute to the ironic effect:
individual positive words (fantastic, great, lot), the exclamation, and the focus topicalization (cf. Seto
1998). Paul's mother is not impressed by the criticism and Paul loses the ensuing short argument.
(14) Kevin You might get an extra merit mark <-|-> if you do an extra session. <-|->
Ruth <-|-> I don't really know. <-|->
Paul I don't!
Ruth I don't really know.
Paul Oh fantastic! That's a great lot of <-|-> help you are. <-|->
Ruth <-|-> It'll do <-|-> you good to do G. <pause>
Paul It won't do me good to do pipsqueak questions. <pause>
Ruth If they're pipsqueak why are you asking me?
Paul I'm not <pause dur=23> Two A three. Two A means two times two is four. (BNC KD0)
In (15), with the same participants as (14), a quasi lexicalised ironic expression is used, i.e. an
example of common irony (Barbe 1995). Evaluative big deal is practically not used in a positive sense
any more. Such specialization (and/or overuse) can weaken the ironic force of the utterance (which
might be increased again by repetition, an irony marker, cf. big, big deal in BNC KPV). While the
criticism here is probably not very strong, it leads (after the comment by third party Kevin) to a first,
unfortunately <unclear> response, and then to a justificatory response by the “victim” of irony, Paul.
(15) Ruth Well they'll have to be equal prizes wouldn't they?
Paul I've got it! <pause> Whichever team wins <pause> th-- the cha-- children can give out the Christmas presents.
Ruth What Christmas presents?
Paul Those presents up there.
Ruth Big deal!
Kevin Can't fool you.
Paul <-|-> <unclear> <-|->
Ruth <-|-> Well I've <-|-> never <unclear> <-|-> <unclear> prize <-|->
Paul <-|-> It's only <-|-> <pause> it's only fun, I mean what are you gonna give them? <pause dur=11> The team that
wins <-|-> gets their presents. <-|-> (BNC KD0)
The next two examples have exclamatory structure (what a …, cf. also how …), which is also a
common ironic device. The whole formulation of the utterance in (16) contributes to the ironic effect
(especially brave, let loose) as does the situational inappropriateness: nowadays, it tends to be the press
that is let loose on other people, and the trustees of the Tyneside Cinema Board are probably rather
“harmless” people, more likely to be attacked by the press than the other way round. The trustees
apparently do not fancy the task required of them very much, and Peter (the chief executive) thus feels
it necessary to make conciliatory remarks in response.
140
(16) Peter <pause> Er it will be at ten. Ten till two or something like that. It's also very good if board members can attend
to er <pause> not merely to support the staff er and to celebrate the event but also, if necessary, to talk to the
press or to <pause> er engage er with guests and so on and so forth. Er it would be much appreciated if you were
able to attend. <pause>
Roger Any other other business?
Colin What a brave person you are to let a board of trustees loose on the press.
Peter Well we we we'll have an army of people to stand by you and <-|-> guide you and <-|->
PS000 <-|-> <laugh> <-|->
Peter nudge you should you say anything. (BNC F7A)
In (17), a conversation between two young women, the exclamation form is combined with word
level irony and an imitated accent, which is additionally intended to mock the absent victim of the
irony. AL's irony sounds somewhat bitter, as she has apparently been criticized before for not doing
well certain sales behaviour that the American is boasting of.
(17) AL: no i was talking to this guy um who's from knox
BQ: <[1><O>laughs</O></[1></{1> <[2><laughs>yeah</laughs></[2></{2> mhm
AL: um mark and he's um living in wellington for the summer cos he's being a lawyer
BQ: oh yeah
AL: working as a summer law clerk lucky bastard anyway he um he used to work at tauranga mcdonald's and like he
was a crew trainer and assistant this and worked his way up you know <with pseudo American accent>wow what
a hero</with pseudo American accent> and um he said that he used <.>to</.> cos i was talking about that suggestive
selling you know how they always used to tell me off for not being suggestive <laughs>enough</laughs> and
um he said he used to go to people um when they bought like a small coke he'd say um would you like a twenty
four pack of chicken mcnuggets with that or would you like eight big macs and four cheeseburgers with that
<O>laughs</O> and just take the piss which i think is really good <O>laughs</O> (WSC DPF028)
Searches for particular terms were often not successful with respect to these terms themselves
being used ironically, but turned up other ironic material in their vicinity, thus broadening the scope of
findings. In a House of Commons debate a certain Wilson MP produces the following statement (18),
which makes a very ironic impression. The irony lies on the utterance and the contextual level, with the
speaker alluding to irony of situation, namely the absurd fact that while attempts at reducing
regulations are being pursued, new ones are being passed all the time. Through his choice of words and
register (great enthusiasm, diving, anxious, merrily getting on with it) Wilson is also poking gentle fun
at his colleagues.
(18) Wilson <unclear> here was that er we're debating banks and banking regulations <pause> building soc-- soc-- society
orders, auditors regulations and financial services rules. Madam Deputy Speaker I didn't mention <pause> those
<pause> at all in the speech I gave last week which was so warmly received by the house. <pause> Here are all
my colleagues rushing upstairs with great enthusiasm, diving into the committee room, anxious to get on
to curb the ever growing number of rules and regulations and whilst they're upstairs merrily getting on
with it, here we are downstairs passing more <pause> things which we say we don't want to do.
PS000 Will the honourable member give way?
Wilson Yes. (BNC JSF)
The next example is interesting, as it sounds to me like a good example of the echoic-mention
theory. The bold-faced statement by Ian might be taken to echo what people (parents?, teachers?) said
about himself and Edmund in their youth, and which in the meantime has been proved wrong (cf. the
italicised statements further down). Ian is therefore criticizing (albeit in a good-humoured way) those
people who did not believe in their abilities in the past through the contrast between beliefs and the
actual state of affairs, while also putting himself in a good light. The absence of any linguistic
indicators of irony in this ironic utterance is noticeable, with the accompanying laughter8 the only sign
of a non-serious intention.
(19) Ian <-|-> Yeah. <-|-> Well i-- i-- I always say that er it always comes a surprise that I was good at anything. And I
always think back to the er the time I think Edmund and I were sitting in the back garden here <pause> and
deciding that we'd go off and join the paratroopers
Noel <-|-> <laugh> <-|->
Enid <-|-> Really? <-|->
Ian <voice quality: laughing> because we weren't going to get <end of voice quality> any O levels.
Enid Oh!
Ian And that was <voice quality: laughing> about all we were good for <end of voice quality>.
Enid <laugh> I'm <-|-> sorry. <-|->
Noel <-|-> <unclear> <-|->
Enid How funny. <-|-> But in fact you did get your O levels didn't you? <-|->
8 It is striking how often mark-up for laughing is present around ironic utterances. This might be a further way of looking for
irony.
141
Ian <-|-> So I remember <unclear> well I got a handful <-|->
Enid Yes I <-|-> remember <-|->
Ian <-|-> <unclear> <-|->
Enid Mm.
Noel And Edmund got a BA. (BNC KC0)
My last example is an instance of self-directed irony, i.e. the speaker ‘criticizing’ himself, here in
the presence of others. The speaker at a church funds meeting makes a little grammatical mistake and
proceeds to apologize for this in a very marked fashion, with hyperbole and borrowings from the
religious register. Of course, the remark is rather jocular, but statements of this kind can function as a
sort of preventive self-defence, pre-empting criticism from others. This might also be a case of Dews et
al.'s function of the speaker showing himself to be in control (cf. 1.2 above), not of his emotions but of
the social situation after a faux pas.
(20) F87PS000 (…) Moderator I've asked for this new clause to come in immediately after seven because of what in accepting
seven we've just done. (…) Now let me read the words for the s-- benefit of those who don't have them <pause> add new
section eight and rem-- renumber eight urge his majesty's government to give continuing and careful concern to the many
situations in which lack of financial resources are still causing elderly people grave hardship. Having read that I
immediately repent me in dust and ashes for having committed a dreadful grammatical error. I was so caught up in
my plurals or situations in hardship that I didn't notice that the subject in more senses than one is a singular lack, and the
verb should be is and not are, therefore I must ask the indulgence of the general assembly to change the verb. (BNC F87)
3.3 Unusual collocations
Louw (1993: 163) has argued that violations of semantic prosodies point either to the presence of
irony or to the suspicion that the author might be insincere. One of his examples is “bent on selfimprovement”
(David Lodge), contrasting with the fact that its usual collocates are overwhelmingly
unpleasant or outright negative (e.g. mischief, resigning, destroying) (ibid. 164ff).9 Thus, instances of
unusual collocations might be another way to unearth irony in corpora. However, Louw also points to
problems posed by this approach, namely that semantic prosodies might be difficult to access through
human intuition and, therefore, not that easy to look for and find in corpora (172f). This is a valid point,
as I found out all too readily. There are not that many terms with clear prosodies that immediately
spring to mind. Commit is one such clear example, as it has in one of its uses (transitive and without
following to) a wonderfully unambiguous negative semantic prosody, being followed by such words as
suicide, crime, atrocity, adultery, offence, arson, robbery10 etc. One might then assume that if it
occurred with a positively evaluated object (such as favour), this might in fact produce irony. However,
none of the spoken corpora yielded any unusual combinations for this word. Another search with set in
and set about, for which Sinclair (1991: 74ff) had found rather clear patterns, was also unsuccessful.
I then decided on a slightly more systematic, though still random, approach. I checked through
the entries of the BBI Dictionary beginning with the letters f, g, h, i, j, k, l, m (random choice) to find
promising candidates for a corpus search. Most searches (e.g. honour (v.), hark, flood (v.), frugal,
languish, loyal, interlude, forgive) did not yield any results11, either because the semantic prosodies
were not pronounced/unambiguous enough in the first place or because there were no violations.
However, there were a few successful, or at least interesting, hits such as the following example:
(21) Maggie I'm gonna borrow your husband a minute <-|-> <unclear> sit on the floor. <-|->
Betty <-|-> Alright. Just watch him cos he's <-|-> liable to be a bit amorous. <laugh>
Maggie <-|-> Right Dave? <-|->
PS000 <-|-> <unclear> chuck you out <unclear> <-|->
Betty <-|-> Oh dear dear dear. <-|-> (BNC KBE)
To be liable to is an expression with a tendency to combine with words denoting
unpleasant/disagreeable things or actions and it also has a legalistic touch. Among the nouns
prominently collocating with liable are bye-law, servitude, disqualification, negligence, damages,
indictment, debts, defendant, prosecution, flooding etc. Others collocates include criminally,
disciplinary, legally, penal and so on. A bit amorous is certainly the odd one out in this list, and the
whole utterance above is certainly humorous, perhaps also ironic. Betty makes this statement about her
husband (David) in the presence of her daughter (Sally) and her friends (Rose, Julie, Maggie). If irony
9 This effect seems to be similar to register or stylistic clashes, which can also produce irony. However, I do not think that the
result of a semantic clash need always be ironic or insincere; it might just be a general humorous effect, which I would also see
as a more likely interpretation of the Lodge passage in question.
10 Those were among the first 13 items of a collocates listing (based on mutual information) for commit done on the whole BNC.
11 These searches were only done with the BNC, because it is the only one big enough to yield viable collocation data.
142
is involved here the victim is David, who does not react to it (unless the unattributed, and unfortunately
mostly unclear PS000 utterance is by him). David is described in the BNC text header as a disabled
unemployed man at the age of 55, while Maggie (who wants to ‘borrow’ him) is a 32-year-old shop
assistant. The incongruity between their respective ages might play a role here for an ultimate
interpretation of this exchange.
The next example, which concerns verge (n.), was found incidentally during a search for
marriage. Just like liable, verge has a rather negative semantic prosody, combining with collocates
such as extinction, bankruptcy, starvation, tears, collapse, nervous (breakdown), death, war etc., with
retirement being the only noun among the first 50 collocates that can take on a positive or a negative
evaluation. The phrase on the verge of marriage then at the very least seems somewhat unusual.
(22) Liz Yes. I mean, I know it's hard <pause> it's hard doing <pause> <-|-> erm <-|->
PS000 <-|-> Yes! <-|-> <laugh>
PS000 And she wasn't going to marry, she never really considered marrying Rivers did she? <-|-> It said she was
on the verge of marriage. <-|->
PS000 <-|-> I heard she <unclear>. <-|->
PS000 She was <-|-> almost <-|->
PS000 <-|-> Er <-|->
PS000 <-|-> hypnotized <-|->
Liz <-|-> Yeah. <-|-> (BNC K60)
However, the phrasing seems to produce neither humour nor irony in this instance. Another
problem here is that it is a kind of quote occurring in the context of a literature lecture/discussion (with
the lecturer, Liz, and two other (unknown) participants), so that one would have to take into
consideration how the phrase is used and intended in the original literary work. At any rate, this is not a
truly conversational example for a collocational clash.
The medium might in fact play an important role. Most of Louw's examples are taken from the
written language, and it could be that unusual collocations intended to transport a special meaning and
attitude are more common in carefully drafted written language. My last example is taken from an
academic book publication.
(23) When Tikhon was placed under house arrest in June 1922, one of these movements, the Living Church, was given numerous
concessions by the regime, and at first looked set to take over the role and some of the property of the Orthodox.
Trotsky went so far as to call the agreement “an ecclesiastical NEP”, implying a similar tolerance to that meted out to
“kulaks” or to Nepmen, but this was a superficial and short-sighted judgement redolent with propaganda. The year 1922 in
fact marked the start of the regime's long-term siege of the Church. (BNC A64)
The combination of tolerance and mete out, whose most prominent collocates are punishment,
treatment, justice (the last two especially in their negative instantiations), certainly leads to an ironic
effect, used for the prototypical function of criticism.
4. Evaluation and outlook
The above investigation has shown that it is possible to find instances of irony in corpora, and,
what is more, instances of a very varied kind. The examples presented above do in fact all express an
attitude, in many cases a somewhat negative or the prototypical critical one. They range in force from
being humorous (e.g. (13)) to conveying confrontational criticism (e.g. (14)). The examples exhibit the
diverse targets of ironic utterances: at the speaker him-/herself (20), at the situation as such (18), at
the/a present addressee as the victim (14, 16, 21?), and at an absent victim (13, 17, 19). Formally, the
irony is realized at the word level (13), at utterance level (16) or in a certain incongruity between the
content utterance and the situation (19). Thus, the variety of irony treated in the literature can be
illustrated with authentic corpus material. Moreover, the ironic instances found also show a nice social
spread. They occurred both in informal contexts (family, friends, e.g. (14), (19), (21)) and in more
formal ones (institutional settings, e.g. (18), (20)), with work-related contexts perhaps in the middle
(13, 16). Irony is produced in the examples by people of different ages (from their teens (14) to their
seventies (19)), and from different spheres of life (teachers, students, production workers, shop
assistants, chief executives, housewives, civil servants). While basic sociolinguistic information is
available in some corpora (relatively informative in the BNC, less so in the WSC), much contextual
information that would be useful for the interpretation of irony is missing (e.g. details about personal
relationships, mutual knowledge etc.). The cotext of the examples can be useful in this respect,
however (cf. (19)).
All three approaches pursued here are useful to a certain extent. Searching for explicit irony can
give information about people's understanding of irony and it can even find conversations with
143
(supposedly) ironic remarks. It is a restricted approach, however, as the frequency of such instances is
rather limited. The collocational method is probably the most complicated and also the least promising.
At any rate, it is hard to pursue as long as no more is known about semantic prosodies than is the case
at present. The best way seems to be to look for instances of irony identified by native speakers and
dictionaries, hoping that this will also turn up unexpected examples – because it has to be admitted that
this approach contains the danger of finding only what one was looking for in the first place, i.e. a circular
approach. The method has to be fine-tuned of course, e.g. by using a comprehensive dictionary
database (the OED in its present edition might not have been the best choice here) and employing native
speaker intuitions in a more systematic way by collecting them via a structured questionnaire technique.
The size of the corpus used is also very important. The examples quoted above are almost all
from the BNC, with just a few from the WSC and none at all from the SBC. The WSC proves that
careful searching can make use of a one-million-word corpus for investigating irony, but anything below
that (cf. SBC) is pushing the researcher's luck.
References
British National Corpus (BNC)
Santa Barbara Corpus of Spoken American English Part 1 (SBC)
Wellington Spoken Corpus (WSC)
Attardo S 1994 Linguistic Theories of Humor. Berlin/New York, Mouton de Gruyter.
Barbe K 1995 Irony in context. Amsterdam/Philadelphia, Benjamins.
Benson M, Benson E, Ilson R 1997 The BBI dictionary of English word combinations. Amsterdam/
Philadelphia, Benjamins.
Brown, P, Levinson, S 1987 Politeness. Some universals in language usage. Cambridge, Cambridge
University Press.
Clark H, Gerrig R J 1984 On the pretense theory of irony. In Journal of experimental psychology 113:
121-126.
Dews S, Kaplan J, Winner E 1995 Why not say it directly? The social functions of irony. In Discourse
Processes 19: 347-367.
Grice H P 1991 (1975) Logic and Conversation. In Davis S (ed), Pragmatics: A reader. New
York/Oxford, Oxford University Press, pp. 305-315.
Hamamoto H 1998 Irony from a cognitive perspective. In Carston, R, Uchida, S (eds), Relevance
Theory: Applications and implications. Amsterdam/Philadelphia, Benjamins, pp. 257-270.
Hartung M 1998 Ironie in der Alltagssprache. Opladen/Wiesbaden, Westdeutscher Verlag.
Kreuz, R J, Roberts, R M 1995 Two cues for verbal irony: Hyperbole and the ironic tone of voice. In
Metaphor and Symbolic Activity 10(1): 21-31.
Lapp E 1992 Linguistik der Ironie. Tübingen, Narr.
Leech G 1983 Principles of Pragmatics. London/New York, Longman.
Levinson S 1983 Pragmatics. Cambridge, Cambridge University Press.
Louw B 1993 Irony in the Text or Insincerity in the Writer? The Diagnostic Potential of Semantic
Prosodies. In Baker, M, Francis, G, Tognini-Bonelli, E (eds), Text and technology: In honour of
John Sinclair. Philadelphia/Amsterdam, Benjamins, pp.157-176.
McEnery T, Wilson A 1996 Corpus linguistics. Edinburgh, Edinburg University Press.
Oxford English Dictionary (OED) 2nd edition, 1994 Oxford, Oxford University Press.
Seto K 1998 On non-echoic irony. In Carston, R, Uchida, S (eds), Relevance Theory: Applications and
implications. Amsterdam/Philadelphia, Benjamins, pp. 239-255.
Sinclair J 1991 Corpus, concordance, collocation. Oxford, Oxford University Press.
Sperber D, Wilson D 1981 Irony and the use-mention distinction. In: Cole P (ed), Radical pragmatics.
New York, Academic Press, pp. 295-318.

144
Corpus-based empirical analysis of form, function and frequency of characters
used in Bangla
Niladri Sekhar Dash and Bidyut Baran Chaudhuri
Computer Vision and Pattern Recognition Unit
Indian Statistical Institute
203, Barrackpore Trunk Road
Calcutta - 700 035, India
Email: {niladri/bbc@isical.ac.in}
Abstract
In this paper an attempt is made to understand formal and functional aspects of Bangla characters
used in the written texts compiled in a sample monitor corpus designed systematically from language
data collected from various text documents published within 1980 and 1995. The purpose of this study
is to understand the form and function of the characters, trace their behavioural peculiarities, and if
possible, find out the reasons of such peculiarities. The study focuses on the formation of the
characters, their structural change in case of compound and cluster formation, their contextual use,
statistical analysis of their occurrence, and their position in words. The study also encompasses the
use of different punctuation marks in the texts. Finally, some possible areas of application of such
analysis are identified.
Key words: character, corpus, grapheme, allograph, vowel, consonant, cluster, frequency, statistics,
punctuation.
1. Introduction
The Bangla1 language corpus, used for the following empirical analysis, is represented by a set of
unique orthographic symbols (letters) arranged in a specific pattern with appropriate punctuation
marks for proper reading and comprehension of the language users. Most of the linguistic features of
the language are captured by the character2 set that leads the written text to be potentially
representative of the spoken form of the language. The study of the characters constituting the corpus
is important for accounting their pattern of use in different context of the texts as well for
comprehension of the general characteristics of the language. Thus, multi-layered information of the
characters can be important and necessary contribution to Natural Language Processing (NLP),
Computational Linguistics (CL), Optical Character Recognition (OCR), key-board design, Word
Sense Disambiguation (WSD), Parts-of-speech Tagging, cryptography, language teaching, Machine
Translation (MT), besides other applied and interdisciplinary studies. Moreover, it can provide insight
about how language is used by different users in different domains of knowledge representation.
For functional and structural analysis of characters used in Bangla, we have considered a
written corpus of 4 million words comprising printed texts of nearly 85 disciplines, published within
1980 and 1995 (Dash and Chaudhuri 2000). To make our corpus non-skewed and sufficiently
representative, we have tried to develop it after the standards proposed by Sinclair (1991), Atkins at
al. (1992), Biber (1993) and others. The relative frequency of various published documents in the
corpus is determined following the report of a survey conducted by the Central Institute of Indian
Languages (CIIL), Mysore, as stated below:
1 Among the languages used in the world today, Bangla is fifth (after English, Chinese, Spansh and Hindi) in strngth in ragard to the
number of speakers. It is the national language of Bangladesh and one of the national languages of India, spoken by the people of
West Bengal and Tripura, two states of India. Moreover, a sizable number of people living in Asia, Europe and America also use this
language.
2 The term character is used here in linguistic or paleographic sense to mean different orthographic symbols used for writing of a
natural language. In most cases, it includes letters, punctuation marks and other similar symbols.
145
Texts from genre %-age Texts from genre %-age
Mass media 30 % Literature 15 %
Social science 15 % Natural science 15 %
Commerce 10 % Fine arts 5 %
Translation 5 % Others 5 %
Table 1: Percentage of texts belonging to different genres included in the corpus
Moreover, an electronic Bangla dictionary of around 60,000 words is used for the current purpose.
Thus, the corpus and the dictionary are used as representative linguistic data-base for description as
well as for verification of hypotheses about the use of characters in the language. They provide
information of use, shape, design, size, occurrence and change of the characters as well as insight into
the role of punctuation marks employed in writing.
The paper is organised as follows: section 2 highlights some unique features of Bangla
characters while section 3 gives an account of shape analysis of characters in isolation as well as
within words. It also gives some idea of tier division, compounding and clustering of characters. In
section 4, we provide a few frequency counts of characters in the corpus while in section 5, we try to
provide a brief survey of the use of punctuation marks. The importance of such study is evaluated and
some application areas are identified in section 6.
2. Bangla characters
In printed Bangla texts, five types of character are noted:
(i) vowel grapheme
(iii) consonant grapheme
(ii) vowel allograph3
(iii) consonant graphic variant
(v) graphic compound, and
(vi) consonant cluster.
Scholars like Banerji (1919), Chakravarty (1938), Diringer (1953), Sen (1993) and others
more or less agreed that Bangla characters are developed from the proto-Bangla type. The present
character set is claimed to be evolved from the hand-written letters inscribed in various Old Bangla
documents and manuscripts (Sen 1993: 24). But, through the years, notable modification on the shape
of the characters is introduced at different points of time. By the 15th and 16th century, the characters
appeared to be fully developed. Indeed, during the 17th and 18th century, no change is registered on
their design and structure. In the 19th century, the shape of the characters has been stereotyped by the
introduction of printing press (Diringer 1953: 365). When the character set was mechanically
designed and stratified for printing, the scope of structural modification was drastically reduced.
However, that the first character set designed by Wilkinson in the 18th century, has been modified to
give more recent shape (Bandyopadhay 1981: 47). Vidyasagar made an attempt to rearrange
characters more scientifically and proposed 52 characters (12 vowels and 40 consonants) in place of
earlier 50 characters (16 vowels and 34 consonants) (Bandyopadhaya 1981: 100). As a result, the total
number of character has been reduced and some consonant clusters are simplified in form
(Mukhopadhaya 1981: 102). It is noted that all Bangla vowels, except a, have allographs. Probably,
allographs were meant to speed up handwriting, because basic vowels, compared to their respective
allographs, take more time, space and energy. It is statistically observed that most recurrently used
allograph is most simplified in shape and most suitably positioned in writing system (Dash and
Chaudhuri 2000).
Structurally, Bangla characters are cursive and twisted as Diringer (1953: 363) observes:
3 The term allograph is a cover term that includes all graphic variants found in the script. The variants are generally called vowel
signs. Recently, Babulanam and Beena (1999) have mentioned them as shorthand signs which, however, we ignore for possibility of
confusion with real shorthand signs.
146
"The Bengali was a peculiar cursive script with circular or semi-circular signs, hooks or hollow triangles
attached to the left of the tops of the vertical strokes. The triangle itself is a modification of the top stroke with a
semi-circle below, and this form is connected with the common form of thick top-strokes, rounded off at both
ends".
Ploughing through the corpus, the following features of Bangla character set are noted:
(i) It has 12 vowels, 20 allographs and 39 consonants along with nearly 280 consonant
clusters which are read and written from left to right sequence in word formation.
(ii) In printed texts, the vowel e has two allographs: one is with and the other is without the
’head line’ (a short horizontal line put above most basic characters). The contexts of their
use are also different.
(iii) A consonant or cluster can use only one allograph at a time.
(iv) Unlike vowel, in certain contexts, a consonant may be silent in utterance despite being
physically present in the text.
(v) The shape of an allograph is not grapheme dependent. However, in some cases it is
changed depending on the shape of a consonant. For instance, the shape of allograph of
the vowel u is changed when used with consonant g and sh, and cluster nt etc.
(vi) The linear position of allographs with respect to graphemes is not uniform. They can
occur before or after, above or below the consonants or clusters.
(vii) Consonants also register some graphic variants (called reph and raphalaa, yaphalaa,
vaphalaa etc.) which are generally used in cluster formation.
(viii) A single grapheme most often represents a single phoneme with a slight phonetic
variation (e.g. ii (long) and i (short) denote [i], u (short) and uu (long) denote [u], j and y
denote [Ô], sh (palatal), s. (retroflex) and s (dental) denote [&#242;], n (alveolar) and n (dental)
represent [n] etc.)
(ix) Most of the consonants can join physically to form a cluster. Generally, clusters are
formed by joining two consonants. However, clusters of three or four consonants are also
possible. There are nearly 280 clusters of which those made by two consonants are
around 240. Cluster of three and four consonants are nearly 35 and 5, respectively.
(x) The sentence terminal markers consist of purnacched (full stop) [< Skt. purna "full" +
cched "pause"], interrogation mark and exclamation mark. Most punctuation marks used
in Bangla are borrowed from English punctuation system.
3. Shape analysis of characters
Generally, the vowels and consonants are considered ’basic characters’ because of their independent
existence and their role in governing the use of other characters in the texts. The entity and role of
vowel allographs and consonant graphic variants are measured by their use within words or
morphemes. The formation of graphic compounds is caused by the change of vowel allographs used
with consonants or clusters. The consonant clusters are formed by joining two or more consonant
graphemes the role of which is context-bound (because their role in the script can be properly studied
when put in the context of words).
3.1 Characters in isolation
The shape of the basic characters is a mixture of straight line, circular and semi-circular curves, thick
dot and conic shapes. All these shapes are not of equal length although the height of most of the basic
147
characters are identical. Moreover, the lines and curves are not always used in their full length in
every occasion of character formation. Sometimes, the full length of a line or curve, sometimes the
half of them, or sometimes just a portion of a line or curve is used for designing basic characters.
However, the shape and design of some basic characters (ii, kh, g, gh, ch, j, th, ph, bh, s etc.) are more
complex than that of other characters. The reason of their complexity probably lies in their process of
formation where all the properties (i.e. dots, curves, straight lines and conic sections) are used.
The head line (shirorekhaa) is considered as an important feature of the characters because it
acts as a line of demarcation at the time of tier division (discussed in section 3.3) of the characters. It
is a property by which the basic characters can be grouped into two broad classes:
·  characters with head line (32 in number)
·  characters without head line (14 in number).
Moreover, according to the arrangement of different structures and properties, the basic
characters can be grouped into three major classes:
·  characters formed with linear structures arranged in different angles (15 in number)
·  characters formed by dot and curve shapes (11 in number), and
·  characters having both kind of shapes (26 in number).
The use of vertical line is maximum in the formation of the basic characters. Nearly 33 basic
characters have vertical line in its full horizontal span. It is observed that while some characters (n, b,
r, dh) contain vertical line at their rightmost side, some characters (c, ch, T, Dh, Rh, d) have it on
their left most side. Similarly, while some characters (aa, jh) use vertical line twice where the second
line is placed just parallel to the first one, some other characters (u, uu, ch, D, d, R) use only a halflength
vertical line in their shape design. Moreover, the width of a character is not always
proportionate to its height. While some characters (g, l, sh) are wider than their height some others
(N, n) are less wide than their height. For some characters (k, b, r ) the width and height are nearly
equal
For automatic recognition an analysis of shape similarity of characters is required since some
basic characters (a/aa, u/uu, o/tt, kh/th, k/ph, t/bh, l/n, sh/n, b/r, D/R, T/Dh/Rh, g/p, y/gh/s, ks/hm) are
nearly similar in shape. They are generally considered as confusing characters because one can easily
be confused with the other either by man or in machine recognition problem.
3.2 Characters in string
Characters used within words sometimes differ from their features noted in their isolation, because
context can add some more features. Generally, the following factors control their roles in context:
·  restrictions in positional use,
·  modifications in original form or shape, and
·  limitations in functional role
These factors generally lead vowels to be converted into allographs at word intermediate and
final positions despite their use in original shape at word-initial position. Therefore, the occurrence of
vowels (i, o) in basic shape at word-middle or final position carries separate implication (e.g., a
particle for emphasis). Similarly, some consonants (n, R, Rh, y) cannot occur at word-initial position
because of the restriction in their positional use. Moreover, the variant of the consonant t (called
khandata "half-t") is not entitled to accompany a vowel allograph. Therefore, whenever such situation
arises (particularly at the time of using case markers) it changes into the basic character (t).
The consonant r generally allows its two variants (reph and raphalaa) to occur only in cluster
formation. While reph occurs above a consonant, raphalaa finds place at the bottom of the character.
Moreover, reph is too weak to cause any structural change of the character but raphalaa is strong
enough to modify the basic shape of some consonants.
148
3.3 Tier division of characters
An analysis of tier division of the characters is important for understanding actual behaviour of the
characters within the words. In Bangla words, the characters are arrayed in three tiers: upper, middle,
and lower. While the upper tier contains signature of basic characters and allographs, some consonant
graphic variants (candrabindu, reph etc.), the middle tier contains the bulk of character shape, and the
lower tier contains some allographs (u-kaar, ri-kaar etc.) and some consonant graphic variants
(raphalaa, vaphalaa etc.). The allographs, when used with consonants or clusters, are distributed in
all three tiers (Pal and Chaudhuri 1995). This kind of analysis has helped us in OCR system
development in Bangla.
3.4 Graphic compounds
The graphic compounds, generally formed by joining two or three graphemes with allographs or
graphic variants, can either be a combination of consonant and allograph, or a combination of
consonant, graphic variant and allograph. In this process some change takes place in the original form
of the characters. Compared to basic characters these are complex in form and design. It is noted that:
(a) the allograph of u, when used with consonants, creates three different graphemedependent
compound shapes: (i) with consonant r or a cluster with raphalaa (e.g., dr, gr,
shr, br etc.) the allograph changes its shape and is attached at the right hand side of
consonant or cluster. The notable point is, the change takes place only with consonant r,
either in its original shape or in graphic variants, (ii) with consonant sh, g and cluster nt,
it changes its shape and is attached just at the bottom of consonant or cluster, and (iii)
with the consonant h it entirely merges with the grapheme making the shape of the
grapheme little more twisted.
(b) the allograph of uu also goes through structural change when used with consonant
grapheme r and clusters with raphalaa (e.g. gr, shr). The allograph changes thoroughly
in shape and is attached on the right hand side of the character.
(c) the allograph of ri goes through a directional change (not structural change) when used
with consonant h. Here, the allograph changes its direction from horizontal to vertical
and is attached to the right hand side of grapheme.
3.5 Consonant clusters
The consonants in clusters (formed by joining with other consonants) undergo three types of structural
change:
(i) the shape of the graphemes are entirely modified to generate a new shape (e.g., ks, kt, kr,
ng, nc, tt, tr, hm etc.)
(ii) the shapes of the graphemes are partly modified. We have found that for nearly 45
clusters, the shape of the first grapheme is modified, while for nearly 60 clusters, the
shape of the last grapheme is affected.
(iii) three (nearly 35) or four consonants (nearly 5) can also join to form a cluster, where the
last grapheme (either in full or in part) is attached at the bottom or right hand side of the
cluster.
Recently, some compounds and clusters are simplified in shape for transparency and easy
access in typewriter and computer (Sarkar 1993: 42), which is yet to be accepted widely by printing
organisations.
4. Some quantitative findings
149
The introduction on different sub-disciplines like quantitative linguistics, stylometrics, applied
linguistics, forensic linguistics etc. has raised a demand for different statistical and quantitative
analysis of the occurrence of characters in the texts of a language for making various observation and
hypotheses, developing primers for language learners, designing tools for CL and NLP etc. The use of
statistics on language study is rejuvenated once the computer accessible huge corpus is available to the
investigators. A corpus with a huge collection of empirical data with innumerable variations of use of
different characters can easily be subjected to quantitative analysis which allows us to discover which
characters are more regularly used in the language and which occur rarely. It allows us to get a precise
picture of the frequency and rarity of particular character, and thus helps us to determine their relative
normality or abnormality. This has led Yule (1964: 15) to comment that linguists without adequate
knowledge of statistical information about different properties of language can make mistakes in
handling linguistic data as well as in observations.
Probably, the Bangla language has been first quantitatively studied by Chatterji (1926/1993).
He made a frequency count of lexical items in a Bangla dictionary and in some writings of Old
Bangla. Bhattacharya (1965), on the other hand, has made different statistical analysis of phonemes,
syllables, words and sentences on a collection of prose texts. Similarly, Das et al. (1984) have made
some statistical studies on global character occurrence in Bangla, Assamese and Manipuri on some
selected texts. The efforts of Mallik and Nara (1994, 1996) are mostly centred around the writings of
the poet Tagore. To the best of our knowledge, multi-dimensional quantitative analysis on Bangla
corpus can be credited to Chaudhuri and Dash (1998) and Chaudhuri and Ghosh (1998).
Before starting frequency count, some issues regarding character identification should be
resolved. The decisions thus made are followed throughout the study on corpus at the time of
programming which saved us from problems of wrong observation or deductions. The decisions are as
follows:
·  at character level frequency count, importance is given on each character's position in
the corpus.
·  each grapheme, allograph, graphic variant, consonant cluster and graphic compound is
considered as a single character.
·  the percentage of vowel is obtained by adding the occurrence of their basic as well as
allographs, while that of consonants is counted by adding the occurrence of their basic as
well as their graphic variants (where applicable).
·  for uniformity in processing and identification of the characters, the punctuation marks
are uniformly separated from the characters in the texts.
The following section presents the frequency statistics of different characters used in corpus
along with some discussions on the findings. Among statistical methods, frequency count is the most
straight-forward and rudimentary approach to work with quantitative data. In this process the
characters in corpus are classified according to a particular scheme and an arithmetical count is made
on the number of items within corpus which belong to each classification of the scheme. The
frequency count provides number of occurrences of each type used in corpus. The results may be used
for estimating position of different characters in script, to design primers for language users, as well
as to develop different tools for computer implementation. Moreover, various hypotheses presented by
earlier scholars about the use and occurrence of characters are also verified by the findings. Keeping
these factors in mind, we have counted a few simple statistics of the characters which are given below:
(i) Among total number of characters used in the corpus, the occurrence of aa is maximum
followed by e, r and i. Among the vowels, aa (11.965%) including allographs comes
first, followed by e (9.793%), i (7.745%), u (2.379%) and o (2.027%), while among the
consonants, r (8.633%) is maximally used followed by n (5.033%), k (4.898%), t
(4.312%), b (3.800%), s (2.942%), l (2.866%), m (2.826%), p (2.562%), y (2.143%) and
d (2.127%). The occurrence of r is maximum in corpus probably because of the
150
occurrence of its two variants. Similarly, the percentage of use of t is increased due to the
presence of its variant. Among the first 10 most recurrently used characters in the
corpus, 6 are consonants while the remaining 4 are vowels and all these characters are
easier to articulate than others present in the language. In Hindi corpus also, the vowel
aa occurs maximally in the script (Tripathi 1971: 26).
(ii) The occurrence of vowel (39.63%) and consonant (52.76%) consists nearly 92.39% of
the total characters used in the corpus. Though there are nearly 280 types of cluster in
the script, their use is quite less (07.61%). The percentage of cluster is higher in a
similar context if the text is written in chaste (saadhu) version which is older than the
colloquial (calita) version now in vogue. This supports our argument that Bangla is
gradually simplified in utterance and pronunciation and the clusters are gradually
replaced by single consonants.
(iii) It is found that words starting with the consonant k (9.81%) is highest in the language
followed by p (8.68%), b (8.58%), s (8.24%), e (5.43%), aa (4.85%), n (4.64%), m
(4.63%), t (4.50%), d (4.47%) and h (4.45%). Moreover, words starting with consonants
(81.52%) is more in number than words starting with vowels (18.48%). It provides an
interesting insight into the nature of the language. Out of top ranking 20 characters, the
vowels are 5, while the remaining 15 are consonants which are easy to articulate. It
supports the commonly used statement that Bangla is easier to speak and perhaps
sweeter to listen than many other Indian languages.
(iv) Among allographs, the occurrence of aa (34.20%), similar to Hindi (Khan et al. 1991:
272), is highest in the script followed by e (29.44%), i (18.75%), u (06.64%) and o
(04.50%), consecutively. The counting of a is not possible as it has no allograph, though
our common belief is that the occurrence of a would have been highest in the corpus,
because in most cases a consonant or cluster if not attached with any allograph carries an
inherent [] with it. However, this hypothesis can only be authenticated once a speech
corpus is analysed.
(v) The use of cluster in language is gradually reduced because in corpus the number of
words without cluster is much more (81.57%) than words having one (14.25%), two
(2.72%) or three (1.00%) clusters. Words having more than three clusters are very rare,
mostly tatsama compounds. The statistics hints for almost complete loss of clusters from
the language in future. Among the first 20 most frequently used consonant clusters, pr,
ks, tr, st, sv, ny, sth, gr, by, jn and shy can occur at any position of words, while the
remaining clusters can occur only at word-intermediate and/or final position. Among
these, pr (8.16%) occurs the most because of its maximum occurrence at word-initial
position, and the graphemes comprising the cluster, are quite frequent in the language.
(vi) Among consonant graphic variants raphalaa (37.94%), yaphalaa (26.66%) and reph
(22.70%) are highest in occurrence. They are used in the script for the purpose of cluster
formation. Probably, because of their recurrent use in the language they are made simple
in shape so that they can be used easily in the script. Here we argue with statistical
support that because of their recurrent use in script they are most simple in shape and
designed to make writing easier and faster.
(vii) Following Miller et al. (1958) the statistics of relative frequency of use of punctuation
marks in Bangla corpus is taken. It is noted that the use of comma, like that of English
(Bayraktar et al. 1998), is highest (22.32%) followed by purnacched (17.26%),
semicolon (15.27%), hyphen (8.89%), note of interrogation (7.38%), colon (6.16%) and
note of exclamation (04.36%), respectively.
5. Use of punctuation marks
151
Generally, some syntactic and semantic properties of a sentence are controlled by punctuation marks
as they are used in texts to mark out strings of words into manageable groups. The primary role of
punctuation marks is to show the pause of breath in a sentence besides clarifying meanings or, in
some cases, preventing wrong meaning being deduced form a sentence. Traditionally, two functions of
punctuation marks are considered: (i) grammatical function where they help the construction or
structure of a sentence, and (ii) rhetorical function where they help to deduce the hidden implication
of a statement. However, Crystal (1995) has defined four functions: grammatical, prosodic, rhetoric
and semantic function.
The role of punctuation marks in Bangla is not yet scientifically estimated though there have
been some attempts by Saraswati Press (1956), Roy (1989), Chakravarti (1994), Chatterji (1973),
Bhattacharya (1999) and others. We here try to present an estimate by citing examples from the
corpus.
The punctuation marks most commonly used in Bangla to divide a piece of prose writing are:
purnacched, semicolon, comma, colon, note of interrogation, note of exclamation, apostrophe,
quotation mark, brackets (round, braces and square), dash, ellipsis, hyphen and space, besides arrow
mark, percentage mark, equal mark, therefore mark, underline, implication mark etc. Moreover,
various mathematical and geometric symbols are used in scientific books and articles. Full stop,
therefore, marks the main division into sentences, semicolon joins sentences, and comma which is
most flexible in use, separates smaller elements with the least loss of continuity. Brackets and dashes
also serve as separators - often more strikingly than commas. In the following sections only a brief
outline is given:
·  In Bangla a purnacched is regularly used as a terminal marker at the end of a sentence
which is a statement (not question or exclamation) either in descriptive, declarative,
narrative or imperative sense.
·  The use of comma has a lot of variation in practice. Its primary role is to give detailed
description to the structure of sentences, especially longer ones, and to make their
meaning clear. Too many commas can be distracting; too few can make a piece of
writing difficult to read or, worse, difficult to understand. It is widely used to separate
main clauses of a compound sentence when they are not sufficiently close in meaning or
content to form a continuous unpunctuated sentence, and are not distinct enough to
warrant a semicolon. Moreover, it is used for a short time break during utterance that
indicates a pause between parts of a sentence, or dividing items in a list, string of figures
etc. Besides, it is used in pairs to separate elements in a sentence that are not part of the
main statement, to stop for a short while in various sections within a sentence, to provide
a slight pause between words in a sentence and between dates of days, month and year.
Generally, this technique is not practised in Bangla though our corpus contains some
such instances. It is used in numeral of four or five figures, to separate each group of
figures starting from the right; before the reporting of speech; and sometimes used in
place of parenthesis where it is used just before and after the parts of text.
·  Colon is generally used to introduce a quotation or a list of items; or to separate clauses
when the second clause expands or illustrates the first. It acts to separate main clauses
when there is a step forward from the first to the second, especially from introduction to
main point, from general statement to example, from cause to effect, or from premises to
conclusion; to deliver the terms (name of books, persons, countries or a statement of
people) that have been mentioned in the preceding words; to introduce a list of items; to
indicate time in hours, minutes and seconds in writing; and between numbers in a
statement of proportion or ratio.
·  Semicolon is of intermediate value between a comma and a full stop as it denotes a timebreak
which is longer than comma but shorter than full stop. The main function of
semicolon is to unite sentences which are closely associated or which are complement or
parallel to each other in some way; to join two sentences different in meaning or content;
152
to divide some parts of a sentence meaningfully where each part has commas for its own
use etc. Moreover, it is used as a stronger division in a sentence that already includes
division by means of commas and when the name and designation of some persons are to
be shown in a sentence.
·  The question mark is primarily used to indicate that the sentence is an interrogative one.
Sometimes it is used within brackets in the middle of a sentence to express uncertainty or
doubt about a fact, word or phrase immediately following or preceding it.
·  The exclamation mark is generally used at the end of a sentence to denote a sense of
exclamation. Moreover, it is also used after words within a sentence to express absurdity,
command, warning, contempt, disgust, emotion, pain, encouragement, wish, regret,
wonder, admiration, surprise, grief etc.
·  Apostrophe is used to indicate omission of letters or numbers; to denote loss or
contraction of some letters in words; and to denote loss of digits in years.
·  Generally, two types of quotation mark are used in Bangla to indicate direct speech and
quotation: single and double quotation. While single quotation is used to quote or
mention title of books or other things; to denote special symbols or words used in texts;
to mean that a word or phrase is cited in the sense of pun etc.; the double quotation is
used to denote speech or dialogue in a story or novel; and to quote other's speech
correctly in a piece of writing. It is noted that the closing quotation mark should come
after any punctuation mark which is a part of the quoted matter, but before any mark
which is not.
·  In corpus the use of three types of brackets are noted: parentheses, brace and square
bracket. Among these, use of braces is very rare, mostly for denoting some mathematical
symbols or notations while use of parentheses and square bracket is almost regular.
Parentheses are used to enclose explanation and extra information or comment; to give
reference or citation to any date or event or work or person etc.; and to enclose reference
letter and number. Square brackets are used to enclose extra information which is
attributive to something, someone or some place; and to convey special kinds of
information, especially when parentheses are used for other purposes. For instance, in
standard Bangla dictionaries they are used to give the etymologies at the end of the
entries.
·  The purpose of using dash in writing or printing is to mark a break in words; or to
represent omitted letters or words. It is also used in place of parenthesis. Sometimes a
single dash is used to indicate a pause or hesitation in speech; to introduce an
explanation or expansion of what comes before it; and to indicate omitted words
especially slang or coarse words in reported speech.
·  The use of dot is very rare in Bangla. Generally, it is used to denote abbreviation or
shortening of some portion of words. The use of ellipsis is also very less in Bangla.
Generally, it is used to indicate fumbling of speakers or incompleteness of sentences
which are already started. Moreover, if one does not want to quote the full texts of some
writings one can use this sign in those places of the text where it is left.
·  Asterisk is used as a reference mark in the footnote to explain some ideas or to denote
source of some texts. It is used just immediately before or after word or sentence form
which one can directly refer to the footnote. It is also used to denote some words or
characters silent in texts; to emphasise on or to draw attention of readers to a particular
item in the text. For instance, in a list of books, some can be marked with asterisk to
denote that these books are either very important or rare etc. Sometimes, more than one
asterisk marks are used at a time to indicate a break or lapse of a section of an article or
text.
153
· The alternative sign (generally called as stroke) is used to denote break of lines of a
poetry, to describe two or more related words or items etc. In linguistic analysis it is used
to denote phoneme or phonetic segment or a syllable.
·  The space between words is never considered as punctuation mark but its role in the text
is of equal importance like other punctuation marks. What is a measured pause in speech
is probably a calculated space in writing. It provides gap between two consecutive words
in a writing to identify words in a sentence or texts.
The hyphen is probably the most complex punctuation mark used in Bangla. It's role is not
yet fully defined as there is no regularity in its use in the language. In standard Bangla dictionaries it
is considered as a sign which joins two syllables or words. This definition is partly true as it captures
only a fragment of the multiple roles of hyphen as noted in corpus. Its use in compounds is arbitrary,
especially when elements of compounds are of one syllable. Except for some unavoidable situations, it
is randomly used. In some occasions it is not used though needed, on the contrary, it is used in those
occasions where it is not required. In the corpus more than twenty types of use are noted as given
below. A hyphen is generally used:
·  more often in routine and occasional couplings, especially when reference to the sense of
separate elements is considered important and unavoidable. It is used between
compounds of two nouns (cor-Daakaat "thief and robber"), two adjectives (rogaa-moTaa
"thin and thick"), noun and adjective (man-gaRaa "fabricated"), pronoun and noun (seidin
"that day"), and cardinal adjective and noun (tin-purus "three generation") etc.
·  between two proper names (seli-kiTs "Shelley-Keats"), words of similar meaning
(Taakaa-paysaa "rupees and pennies"), department and post (krisi-mantri "agriculture
minister"), institution and position (skul-maasTaar "school teacher"), place and
occasion, (baarlin-alimpik "Berlin Olympic"), for direction (uttar-pashcim "North-
West") etc.
·  to connect words having syntactic link (kathaay-kathaay-raag-karaa mejaaj "gettingangry-
in-every-word temper"), to link compounds and phrases used attributively
(haajaar-haat-kaali "goddess Kali with a garland made of thousand cut-off hands"), to
denote a sense of continuation (co-o-o-o-r "thief"), to indicate sounds of music or
musical instruments (taa-dhin-dhin-taa), to write some names of non-Indian origin
(maao-se-tung "Mao-Tse-Tung") etc.
·  for all similar words to stop repetition of the second constituent when the second
constituent of compounds of a sentence are common. The second constituent is used only
in the last word in the sentence (raajya juRe shramik-, bekaar-, bidyut-, khaadya-, jalsamasyaa
dekhaa diyeche "The problem of labour, unemployment, electricity, food,
water are noted in the state"). Hyphen here performs the role of concurrence.
·  between prefixes and nouns, (ku-najar "ugly look"), between monosyllabic noun and
suffix (paa-Ti "the leg"), between monosyllabic pronoun and suffix (ka-Taa "how
many"), in compounds where the first part is a single letter word (bhu-prakriti
"geographical nature") etc.
·  between reduplicated words (maajhe-maajhe "sometimes"), onomatopoeic words (jhanjhan
"tinkling"), echo words (bis-Tis "poison etc."), for exclamatory expression (ho-yaaT
"what"), for emphasis (maa-i "mother herself"), for abbreviated words (bi-bi-si "BBC"),
for avoiding awkward collision of homophonous characters (taap-prabar "heat-strong")
etc.
·  when an inflected word form is used as noun and is added with further suffixes for
linguistic analysis (moder-er byabahaar kameche "Use of 'moder' is declined") etc.
154
· between proper name and case ending or suffix which is added with proper name of
person (MilTan-er "of Milton"), institution, (aai.es.aai-te "in ISI"), book (myaakbeth-er
"of Macbeth"), newspaper (sTeTsmyaan-e "In the Statesman"), day (sombaar-e "on
Monday"), year (1947-e "in 1947") etc. However, such use is not a regular feature of the
language. Generally, case endings and suffixes are attached with proper names without
inserting a hyphen in between.
·  to retain the original structure of some words unchanged when they use suffixes (pad-er
"of words", desh-er "of Desh") though without this hyphenation the grammatical value
of the words would not have been changed
·  whenever a case marker is added with words ending with half-t (bhabisyat-e "in future").
·  after Bangla characters when they are subjected to grammatical or linguistic analysis (a-
Taa baanglaar pratham varna "a is the first letter in Bangla").
·  in vowel allographs (u-kaar "u-allograph") for grammatical reasons, and between some
consonants and their place of articulation (dantya-sa "dental-s").
·  with affixes and case markers when used for linguistic analysis. Usually, prefixes are
written with a hyphen immediately after them (pr-, bi-), while suffixes and case markers
are written with a hyphen immediately before them (-der, -ke, -te ). Hyphen works here
as mark of their identity
·  between numbers (2-3), between years (1986-1990), and between numbers and words (6-
phuT "6-feet").
·  when native post-positions or case markers are used with non-nativised foreign words
(mesin-er "of machine"). Such kind of use is very rare in corpus. Generally, nativised
foreign words use native post-positions or case markers without hyphen (skuler "of
school").
·  when some English group verbs (pick up, by pass etc.) are transliterated in Bangla (pikaap,
baai-paash).
·  when a single-letter name is used to hide somebody's identity (ka-baabu "Mr. X").
·  after an abbreviated word with a colon to indicate that the full form of the word is
deliberately omitted (pr:- "question").
·  between words which are not compounds but which by their peculiar combination either
denote a sense of hesitation (dicchi-debo "dilly-dally"), or a sense of request (baabaabaachaa
"appeasing"), or a sense of adverb (saat-taaRaataaRi "in a haste"), or a sense of
pun (jyoti-hin pashchimbanga "West Bengal without light").
As noted above the use of hyphen in Bangla is full of varieties. It serves as a means for
dissolving lexical ambiguity embedded within the surface forms. Moreover, it helps us to find actual
meaning of words or phrases as well as stops us from deducting wrong meaning. The corpus has cited
some interesting instances where deletion or addition of hyphen changes the meaning of words. Table
(2) shows some examples.
Without hyphen Meaning With hyphen Meaning
Asukh illness a-sukh not happy
KaTaa brownish yellow ka-Taa some
PaaTaa plank paa-Taa the leg
Amrita nectar a-mrita not dead
155
Aakaar shape aa-kaar a-allograph
CaaTaa lick caa-Taa the tea
Ekaar alone e-kaar e-allograph
Maar kill maa-r of mother
Kushaasan mat of Kush grass ku-shaasan bad ruling
Table 2: Change of meaning of the words with/without hyphen
Similarly, displacement of hyphen changes the meaning of words. For instance, the word
surat-ranga means "game of coition" while sur-taranga means "waves of music", or akhyaatanaamaa
means "notorious" while a-khyaatanaamaa means "non-famous" etc.
To divide a word with a hyphen at the end of a line is altogether a different matter because it
is not a regular feature of spelling. It is more common in print, where the text has to be accurately
spaced and the margin has to be justified. With little care it can be avoided totally in hand-written,
typed or word-processed text materials. In printing the words need to be divided carefully and
consistently taking account of the appearance and structure of words.
6. Conclusion
Researchers who are studying the evolution of thought process in human societies, believe that
development of language and script may also influence human cognitive powers. Script is a form of
knowledge representation in which the use of grapheme makes demand on humans to code and
decode knowledge, convert auditory sounds into visual symbols, think deductively and order words to
construct sentences. The study of characters used in a language is important and useful for developing
various tools for NLP and CL such as OCR system, cryptography, key-board design for typewriter and
printing, telegraphic codes design, spell-checker design, dictionary preparation, machine translation,
information-theoretic analysis of language, language teaching etc. Even from pure linguistic point of
view such study can help both primary and secondary language users to know how the characters are
designed and used in the language.
Acknowledgement: The Ministry of Information and Technology, Govt. of India is thanked for
providing the corpus for studies presented here.
Bibliography
Atkins S, Clear J, Ostler N 1992 Corpus Design Criteria. Literary and Linguistic Computing 7(1): 1-
16.
Babulanam S M, Beena K F 1999 The User-Oriented Bengali Easy Orthography. Computers and the
Humanities 33: 241-245.
Bandyopadhyay C (ed.) 1981 Dui Shataker Bangla Mudran O Prakashan (The printing and
publishing history of Bangla in last two centuries). Calcutta, Ananda Publishers.
Banerjee R D 1919 The Origin of the Bengali Script. Calcutta, Calcutta University Press.
Bhattacharya N 1965 Some Statistical Studies of the Bangla Language. Unpublished PhD thesis,
Indian Statistical Institute, Calcutta.
Bhattacharya S 1999 Tista Ksanakal: Biramcihna O Anyanya Prasanga (Wait for a While: The
Bangla Punctuations and other Issues). Calcutta, Ananda Publishers.
Bayraktar M, Say B, Akman V 1998 An Analysis of English Punctuation: The Special Case of
Comma. International Journal of Corpus Linguistics 3(1): 33-58.
Biber D 1993 Representativeness in Corpus Design. Literary and Linguistic Computing 8(4): 243-
257.
Chakravarti N (ed.) 1994 Bangla: Ki Likhben, Kena Likhben (Bangla: What to Write and Why to
Write). Calcutta, Ananda Publishers.
Chakravarty S N 1938 Development of the Bengali Alphabet from the Fifth Century AD to the End of
the Muhammadan Rule. Journal of the Royal Asiatic Society of Bengal 4: 351-391.
Chatterji S K 1973 Bangla Bhasatattver Bhumika (An Introduction of the Bangla Linguistics).
Calcutta, Calcutta University Press.
156
Chatterji S K 1926/1993 The Origin and Development of the Bengali Language. Calcutta, Rupa
Publications.
Chaudhuri B B, Dash N S 1998 Bangla Script: A structural Study. Linguistics Today (2)1:1-28.
Chaudhuri B B, Ghosh S 1998 A Statistical Study of Bangla Corpus. in the Proceedings of
International Conference of Computational Linguistics, Speech and Document Processing
(ICCLSDP'98 ): C32-37.
Crystal D 1995 The Cambridge Encyclopaedia of the English Language. Cambridge, Cambridge
University Press.
Das G, Bhattacharya S, Mitra S 1984 Representing Assamese, Bengali and Manipuri Text in Line
Printer and Daisy-Wheel Printer. Journal of the Institution of Electronics and Telecommunication
Engineers 30: 251-256.
Dash N S, Chaudhuri B B 2000 The Process of Designing A Multidisciplinary Monolingual Sample
Corpus. International Journal of Corpus Linguistics. (forthcoming)
Diringer D 1953 The Alphabet: A Key to the History of Mankind. London, Hutchinson's Scientific and
Technical Publications.
Khan I, Gupta S K, Rizvi S H S 1991 Statistics of Printed Hindi Text Graphemes: Preliminary
Results. Journal of the Institute of Electronics and Telecom Engineering 37(3): 268 -275.
Mukhopadhaya B 1981 Bangla Mudraner Car Yug (Four Era of Bangla Printing History) in
Bandyopadhaya C (ed.) DuiSataker Bangla Mudran O Prakashan (The printing and publishing
history of Bangla in last two centuries). Calcutta, Ananda Publishers.
Mallik B P, Nara T (eds) 1994 Gitanjali: Linguistic Statistical Analysis. ILCAA, Tokyo University
Press.
Mallik B P, Nara T (eds) 1996 Sabhyatar Sankat: Linguistic Statistical Analysis. Calcutta: Rabindra
Bharati University.
Miller G A, Newman E B, Friedman E A 1958 Length-Frequency Statistics for Written English.
Information and Control 2 (1): 370-389.
Pal U, Chaudhuri B B 1995 Computer Recognition of Printed Bangla Script. International Journal of
Systems Science 26(3): 2107-2123.
Roy A K 1989 Unis Sataker Bangla Gadya Sahitya: Ingreji Probhab (The Bengali Prose Literature of
the 19th Century: The Impact of English). Calcutta, Jignasa Prakashani.
Saraswati Press 1956 Rules for Compositors and Readers. Calcutta, Saraswati Press.
Sarkar P 1993 Bangla Bhasar Yuktabyanjan (Consonant Clusters in Bangla Language). Bhasa 1(1):
23-45.
Sen S 1993 Bhasar Itivrittva (The History of Language). Calcutta, Ananda Publishers.
Sinclair J 1991 Corpus, Concordance, Collocation. Oxford, Oxford University Press.
Tripathi J N 1971 A statistical Analysis of Devnagari (Hindi) text graphemes. Journal of Institute of
Electronics and Telecom Engineers 17(1): 25-27.
Yule G U 1964 The Statistical Study of Literary Vocabulary. Cambridge, Cambridge University Press.
157

158
Contrasting causal connectives on the Speaker Involvement Scale
Liesbeth Degand
University of Louvain, Louvain-la-Neuve
Germanic studies
Place B. Pascal, 1
B-1348 Louvain-la-Neuve
e-mail: degand@exco.ucl.ac.be
Henk Pander Maat, Utrecht
University of Utrecht
Department of Dutch
Trans 10
NL-3512 JK Utrecht
e-mail: h.pandermaat@let.uu.nl
In our talk we would like to contrast Dutch and French causal connectives on the Speaker
Involvement Scale (Degand & Pander Maat, to appear; Pander Maat & Degand, to appear). This scale
is an alternative proposal to classify the use and meaning of connectives. Going beyond dichotomous
and trichotomous classifications, we have proposed to represent (causal) coherence relations and
connectives in a scalar way This scalar representation reflects the fact that (causal) connectives are not
strictly domain specific, but that they nevertheless impose constraints on the contexts in which they can
occur, with some contexts being more “natural” than others. In addition, a number of causal
connectives seem to take an intermediate position between the traditional categories. According to us,
this situation is an indication for the need of a scalar perspective on the spectrum reaching from nonvolitional
causality in the content domain to epistemic and speech act causality. The scale we have
developed is one of speaker involvement (SI), on which the inherent expressive power of connectives
can be represented. Our hypothesis is that the different causal relations can be ordered along a scale
from minimal to maximal speaker involvement. SI refers to the degree to which the present speaker is
implicitly involved in the construal of the causal relation. More specifically, SI increases with the
degree to which both the causal relation and the related segments vehicle assumptions and actions of
the present speaker. Four characteristics of coherence relations may enhance the prominence of speaker
assumptions in the relation, and thus enhance the level of Speaker Involvement of the relation: The
involvement of a conscious protagonist, a lack of isomorphism between the relation and states of affairs
in the real world, proximity of the relation to the present speaker and the time of speaking, and the
implicit vs. explicit realisation of the protagonist. The different causal relations we distinguish are, in
order of increasing SI: causal non-volitional and volitional content relations; causality-based and noncausality
based epistemic relations, and causal speech-act relations.
Point of departure of our contrastive analysis are connectives which are very close in meaning and
seem to be easily substitutable by one another within one language, like French donc and d&#232;s lors
(‘so/hence'), car and parce que (‘for/because'), or Dutch dus and daarom, or between languages like
the supposed translational equivalents puisque and aangezien (‘since'). In addition, these connectives
also show highly diverging frequencies in a newspaper corpus. The question we would like to tackle in
our presentation is whether these divergences in frequency and subtle meaning differences are reflected
in diverging SI distribution patterns.
This presupposes to uncover the semantic profile of the given connectives as well as their
interaction with the surrounding discourse. In our view, analyses of the SI potential inherent to
connectives cannot do without systematic corpus analyses combining distributional data and semantic
intuitions. In our presentation, we will develop how we proceed to determine the SI of a connective and
how this SI level accounts for the substitution effects created in (1) (when addressed to a traffic agent
who pulls you over because you ignored a ‘one way’ traffic sign), while (2) is perfectly acceptable.
(1a) Ik had haast, #dus /daarom hield ik me niet aan het inrijverbod.
(1b) J'étais pressé, #donc /c'est pourquoi j'ai pris le passage interdit.
(1c) I was in a hurry, ‘so’ / ‘that's why’ I ignored the one way sign.
(2a) Ik had haast, dus /daarom nam ik een taxi.
(2b) J'étais pressé, donc /c'est pourquoi j'ai pris un taxi.
(2c) I was in a hurry, ‘so’ / ‘that's why’ I took a taxi.
References
Degand L, Pander Maat H to appear A contrastive study of Dutch and French causal connectives on the
Speaker Involvement Scale. In Verhagen A, van de Weijer J (eds) Levels in Language and Cognition.
Pander Maat, H. & Degand, L. in press Scaling causal relations and connectives in terms of Speaker
Involvement. Cognitive Linguistics.

159
Corpus-based identification of temporal organisation in discourse
Anne Le Draoulec, Marie-Paule Péry-Woodley
ERSS, Université Toulouse Le Mirail
1. Introduction
We report here on a corpus-based study of a particular form of discourse organisation, inspired by
the notion of temporal discourse frames. In Charolles’ theory of Discourse Framing (Encadrement du
Discours, Charolles 1997), a discourse frame is described as the grouping together of a number of sentences
which are linked by the fact that they must be interpreted with reference to a specific criterion, realised in a
frame-initial introducing expression. For instance, as regards perspective framing, According to X, …
provides an essential element for the interpretation of the proposition which follows, and also potentially of
several subsequent propositions – as frame-introducing expressions are characterised by their ability to
extend their scope beyond the sentence in which they appear (cf. Péry-Woodley 2000).
In the case which concerns us – temporal frames –, the introducing expressions are mostly adverbial
expressions such as today, subordinate clauses such as when he left, and prepositional phrases such as {in |
until | since | about…} 1989. Temporal expressions can occur in various places in the sentence, indeed they
constitute a typical case of these mobile elements whose position (initial, median or final) has interested
many generations of linguists (cf. Bally 1944, Firbas 1972, Givón 1979 inter alia). The best studied
contrast is between initial and final position: a functional difference has been established between
autonomous initial adverbials, which as adjoints have a scene-setting role outside the proposition, and final
adverbials which have no autonomy and express a circumstance only modifying the proposition. The
distinction is also expressed in terms of syntactic integration in the sentence: initial position is equated with
non-integration, final position with integration. Yet position is not the whole story as regards autonomy, or
integration: there is a complex interplay between position and punctuational detachment (a comma is a
visual indication of detachment which argues for non-integration). Combining position and punctuation
yields the following possibilities: initial-detached, initial-non-detached, final-detached, final-non-detached.
However, initial position is recognised as having a major cognitive impact (cf. Lambrecht 1994), and in any
case, for this position, punctuation seems somewhat erratic. On the other hand, there is a real difference
between a detached and a non-detached final adverbial: detachment goes against the integration associated
with final position.
What is the relation between these classical theoretical accounts, and discourse framing? Charolles
(1997, p.15) asserts that frame introducers are sentence adjoints, and suggests that they most often appear
in initial position. As his focus is on the elaboration of the notion of discourse framing, he does not go into
a systematic analysis of the nature of introducers. His intuition about frame introducers seems perfectly
well founded, as there is an obvious link between autonomy (or non-integration) of an expression, and its
ability to open a frame: the characteristic of a frame introducer is indeed to scope over several sentences.
Our objectives in this corpus study are twofold:
(a) Gather data to document the scoping potential of sentence initial temporal expressions, as well as
validate the postulate of a difference in scoping ability linked to integration1. A first contribution of
this study will be to provide a fairly systematic examination, in an extensive corpus, of phenomena
which have until now been observed separately, in theoretical mode mostly and with made up
examples as concerns integration, and on the basis of limited illustrative data for the issue of scope. At
this stage, we choose to concentrate on the most clearly-contrasted cases: initial (detached or non-
1 An earlier corpus-based study (Thompson 1985), though set in a different theoretical framework,
established the same kind of position-linked difference in the case of initial and final purpose clauses.
160
detached) and final non-detached temporal expressions. Within these, we focus on prepositional
phrases (PPs), for methodological reasons which we explain in the next section.
(b) Examine the extension of frames. The left boundary is established by the introducer. But are there
linguistic clues to the final boundary of a temporal frame?
2. Corpus and method
Our initial corpus, the Atlas corpus, is composed of a single text: the book "Atlas de la France
scolaire"2. This choice was motivated by the fact that, as a geographical and historical account of schooling
in France, this book is essentially structured in a spatial and temporal way.
2.1. Preliminaries
Our decision to focus our analysis on PPs reflects in the first instance a methodological choice: the
choice to simplify the parameters to be taken into account for the automatic identification of temporal
expressions. Yet it is not completely arbitrary, as we observed that the temporal expressions in our corpus
were mostly PPs. For instance, a cursory survey came up with an extremely small number of temporal
subordinate clauses. Our method can be said to be semi-automatic: the initial automatic detection based on
lexical markers was followed by a manual selection of the expressions we viewed as relevant. For the
lexical markers, we constituted two lists: one of temporal nouns, based on a study by Berthonneau (1989),
and another containing temporal prepositions. We constructed our search filters3 in such a way that we
could detect:
- PPs composed of any preposition followed by a date (represented by a regular expression) or a
temporal noun. Our list of nouns includes 78 items, among which: année, jour, février, si&#232;cle, moment,
période, début.
- PPs introduced by a temporal preposition, i.e. a preposition which does not require a temporal noun to
form a temporal expression: avant, apr&#232;s, pendant, durant, lors, d&#232;s, depuis, jusque.
We are well aware that these patterns leave out a number of temporal expressions, for instance those
composed with a non specialised preposition and an event noun (&#224; la rentrée des classes, etc.), or those
which are not introduced by a preposition (l'année précédente, deux ans apr&#232;s). Our familiarity with the
corpus allows us to feel confident that our patterns cover the great majority of the temporal PPs that could
be detected. In a study centred on discourse phenomena, we needed extensive context, measured in a
discursively relevant unit: we opted for retaining extracts of three paragraphs, one on either side of the one
containing the PP.
Among the PPs identified, the following were manually eliminated as non-relevant:
- non-temporal PPs: expressions such as en m&#234;me temps in line 13 of example (1), which does not have a
temporal meaning in this context;
- temporal expressions which only bear on one sentence constituent (such as a noun or an adjective), or
which are embedded in subordinate or parenthetical clauses. These expressions, which have a status
below the sentence, seem to be of limited interest for a study of scoping properties.
This sorting task was mostly concerned with eliminating non-relevant items, but occasionally, as in the
analysis of the extended example (1) below, we have also retained expressions which, although not
detected by our automatic procedure, are also relevant for our study. For example, we consider trente ans
plus tard, (line 6), as sharing with the automatically identified surrounding PPs the discourse properties
under study4.
2 Hérin & Rouault 1994. We thank P. Enjalbert et N. Malandain (Greyc, University of Caen) for making
this text (56346 words) available to us.
3 This search was executed with Yakwa, a text analysis tool developed by L. Tanguy (an illustrated
description is available on www.univ-tlse2.fr/erss/membres/tanguy/Yakwa.html). We are very grateful to
L. Tanguy for his readiness to help.
4 Such an adjustment is somewhat arbitrary. Yet, in some cases, it is perhaps less so than allowing technical
considerations to strictly determine the selection of expressions.
161
2.2. Analysis of an example
We begin by presenting an extract from our corpus, which will serve as a basis for explaining our
analytical approach (temporal PPs appear in italics):
(1)
1. Si l'on définit la démocratisation de l'enseignement par l'acc&#232;s d'un nombre croissant de jeunes aux formations et
2. aux diplômes de l'enseignement secondaire et de l'enseignement supérieur, il y a bien eu, en une génération,
3. démocratisation du syst&#232;me éducatif français. En effet, entre la fin des années 1950 et le début des années 1990,
4. le nombre des collégiens a triplé, celui des lycéens quadruplé et celui des étudiants sextuplé. Les taux de
5. scolarisation des adolescents et jeunes adultes ont considérablement augmenté: en 1958-59, &#224; peine 20% des
6. jeunes de dix-huit ans étaient scolarisés; trente ans plus tard, la proportion dépasse 80% (apprentis compris) et
7. dans le m&#234;me temps le nombre des admis au baccalauréat est passé d'&#224; peine 60 000 &#224; plus de 400 000.
8. Toutes les catégories sociales ont bénéficié de l'allongement des scolarités et de l'acc&#232;s de plus en plus ouvert aux
9. formations des cycles secondaires et supérieurs: le nombre des étudiants français d'origine ouvri&#232;re était de
10. 30 000.au début des années 1960, il est de 130 000 au début des années 1990; celui des étudiants provenant
11. des catégories sociales aisées, patrons, professions libérales, cadres supérieurs, est passé dans le m&#234;me temps de
12. 100 000 &#224; 350 000 environ. Mais le rapprochement de ces derniers chiffres, s'il confirme l'ouverture des niveaux
13. supérieurs de formation aux jeunes des milieux économiquement modestes, indique en m&#234;me temps les limites
14. de la démocratisation du syst&#232;me éducatif. Certes, la proportion d'étudiants d'origine ouvri&#232;re (salariés agricoles
15. et personnels de service compris) dans les universités a presque doublé au cours des vingt derni&#232;res années;
16. mais elle n'est encore que de 12%, alors que plus de 40% des jeunes d'âge scolaire sont d'origine ouvri&#232;re. Et
17. pr&#232;s du tiers des étudiants des universités proviennent des milieux aisés, alors que les enfants des familles de
18. cadres supérieurs et des professions libérales ne forment gu&#232;re que 10% de la population scolarisable.
We will concentrate here on the analysis of two sets of tokens which lend themselves particularly
well to exhibiting the functional difference between sentence initial and sentence final temporal expressions
(from now on ITEs and FTEs). The first set consists of the ITEs: entre la fin des années 1950 et le début
des années 1990, en 1958-59, trente ans plus tard, dans le m&#234;me temps. The second one consists of the
FTEs: au début des années 1960, au début des années 19905. The first set examplifies the framing
characteristics described in the introduction. Entre la fin des années 1950 et le début des années 1990 opens
a frame which extends beyond the sentence where it appears: the interpretation criterion thus provided
scopes over a number of sentences which constitute a segment, within which the next three ITEs (en 1958-
59, trente ans plus tard, dans le m&#234;me temps) open sub-frames6. This can be visually represented in the
following manner:
There is no such temporal framing with the expressions in the second set. The organisation is
provided by the anaphoric link between the clauses, and the FTEs are only temporal modifiers within the
clause (in other words, they only provide a specification).
5 dans le m&#234;me temps in the same paragraph is not dealt with in our study as it is in a median position, a
position which will be examined in further work. Though it appears to function here in the same way as an
FTE, we do not want to prejudge the equivalence of these two positions.
6 The interpretation whether an ITE opens a new frame or a sub-frame depends crucially on world
knowledge: it is world knowledge that tells us that the periods 1958-59 and trente ans plus tard are
included in entre la fin des années 1950 et le début des années 1990.
Entre la fin des années 1950 et le début
des années 1990,
en 1958-59,
trente ans plus tard,
dans le m&#234;me temps
162
3. The scope of ITEs
As one of our main objectives is to investigate the scoping properties of temporal expressions, our
first task is to examine the way in which ITEs structure the text. And insofar as they open segments which
can extend over several sentences, the first question is to determine what may signal the final boundary of
such segments. In other words, what indicates to the reader that the interpretation criterion provided by the
ITE no longer applies? This preoccupation leads us to establish a distinction between ITEs opening a
frame, and ITEs opening a sub-frame: the above example (line 3 to 7) shows that only the former can have
an extended scope, whereas the scope of the latter is clearly limited to individual items of the enumeration.
In our corpus, we have identified a number of parameters associated with the end of a frame (shown
in our examples by ¨). Two major types of clues are directly connected to time: the first is occurrence of
another temporal expression referring to a time not included in the period denoted by the frame introducer;
the second is change of tense. Another clue of a more general nature is change of paragraph.
Further work will be needed to determine precisely the interaction between these clues. We can
however already present some observations. It might seem that the presence of an ITE introducing a new
frame should be a sufficient clue to a closing frame boundary. And yet, in our corpus, an overwhelming
number of these cases display a change of tense as well, as in the example below which combines the two
parameters: the opening of a new frame is marked both by another ITE (en 1989) and by a change from a
perfect to a present tense.
(2)
En juin 1992, 747 500 candidats se sont présentés &#224; l'examen, dont 35 000 candidats individuels ; pr&#232;s des trois
quarts ont été reçus ; mais pour les candidats individuels le taux de réussite a été &#224; peine de 50 %. Pour la série
coll&#232;ge (85 % de l'ensemble des candidats), 76 % des candidats des établissements scolaires ont obtenu le brevet,
ceux des coll&#232;ges privés sous contrat réussissant mieux que ceux des coll&#232;ges publics (85 % de taux moyen de
succ&#232;s dans les premiers, 74 % dans les seconds). ¨
En 1989, tant les collégiens du privé que ceux du public ont de meilleurs résultats dans les départements des
académies de l'Ouest o&#249; les él&#232;ves du privé sont nombreux, d'Orléans -Tours, Reims et Grenoble, ainsi que dans les
Midis aquitain et méditerranéen.
We may wonder why we do not find in our corpus any example of a date change without a tense change,
even when this would be possible (as in (2) where the present could easily be replaced by a perfect). A
hypothesis might be that tense should reflect the temporal difference, and indeed this is what happens in
many cases. However, our above example intriguingly contradicts this hypothesis: the change from perfect
to present does not follow a temporal logic, as the second frame is temporally anterior. What is at work
here is definitely a discursive logic: change of tense is a way to make very clear to the reader that she is
moving to a new segment.
Change of tense does not inherently guarantee a frame boundary. But in cases when, in the absence
of a new temporal expression, the tense changes from non-present to present, this can suffice to indicate
such a boundary, as the use of the present is then generally an implicit reference to the time of utterance.
The following example is a case in point, with a transition from perfect to present, made even more explicit
by the temporal adjective actuelle in the NP la tendance actuelle.
(3)
Depuis le début des années 1960, la composition du corps enseignant a été diversifiée : les disciplines, catégories
indiciaires et grades ont été multipliés, avec l'apparition de CAPES et de CAPET artistiques et techniques et la
création de certains corps enseignants (PEGC et plus récemment, PLP1 et PLP2). ¨ M&#234;me si la tendance actuelle
est &#224; la simplification, les grades restent nombreux (pr&#232;s d'une quinzaine) de m&#234;me que les statuts (titulaire, titulaire
académique, titulaire remplaçant, stagiaire, non -titulaire). Le corps professoral demeure hétérog&#232;ne.
Rather than change of tense, the relevant clue is therefore the transition from non-present to present7.
7 This is a somewhat simplified account: with depuis, things are much more complex, as the period referred
to includes the present time. And in French, as opposed to English, this property may lead to a present tense
being used to express a situation which extends from past to present: Depuis 1993, j'habite &#224; Toulouse /
Since 1993, I have been living in Toulouse (English requires a perfect). In example (3), the present tense in
la tendance actuelle est is a “true” present which only refers to the time of utterance, and implies that we
163
Another example is given in (4), where two different past tenses (perfect and imperfect) coexist
within the same frame opened by De la fin du si&#232;cle dernier jusqu' aux années 1950, and contrast with the
present tense which is necessarily outside this frame.
(4)
De la fin du si&#232;cle dernier jusqu' aux années 1950, l'école primaire a été le pilier du syst&#232;me scolaire français. Elle
inculquait les connaissances de base, lire, écrire et compter, qui serviraient toute la vie. Elle avait aussi pour mission
de former les citoyens de la République. Elle délivrait le certificat d'études qui, pour le plus grand nombre, attestait
de la réussite des études et marquait l'entrée dans le monde du travail. ¨
Les sessions du certificat d'études n'ont plus lieu.
The third frame boundary clue mentioned above is change of paragraph, which is often regarded as
linked to topic change (cf. Longacre 1979, Bessonat 1988). In our data, it appears to be indeed a relevant
clue, as in examples (2) and (4). It is however not necessary, as shown in example (3), where a change of
frame occurs without a change of paragraph. Is it sufficient? We have no case when a new paragraph is still
to be interpreted as part of an earlier introduced frame: so a new paragraph seems to be a sufficient clue to
indicate a frame boundary. Is it sufficient by itself? In most cases, the new paragraph is associated to some
of the above clues. But we also find instances where a paragraph change, in the absence of any temporal
clue, implies a frame boundary. In example 5, the paragraph change is accompanied neither by a new
temporal expression, nor by a tense change:
(5)
Malgré cette progression rapide et générale, les écarts restent sensibles entre les académies et les départements. En
1990, la proportion de bacheliers par classe d'âge n'atteint pas 40 % dans les académies de la grande couronne
parisienne. Elle se situe entre 40 et 45 % dans le Nord, en Lorraine et en Alsace, dans les académies de Toulouse,
Limoges, Grenoble, Rennes ainsi qu'en Corse. En Île -de -France les écarts sont écrasants entre Paris (67 % de
bacheliers dans la classe d'âge) et la Seine-Saint-Denis (29 %) Dans l'ensemble, la moitié sud du pays continue &#224;
avoir de plus fortes proportions de bacheliers que la moitié nord, Bretagne et, &#224; présent, Lorraine exceptées. Mais ces
différences s'atténuent. ¨
L'allongement des scolarités, dont l'augmentation continue du nombre des bacheliers donne la mesure, se traduit par
la fréquence sans cesse croissante des scolarités longues, poursuivies jusqu'au terme des études secondaires, sans
préjuger des entrées de plus en plus nombreuses dans les formations supérieures.
At this point in our study, we may feel that we have satisfactory confirmation of the specific
functional role of ITEs. By extending their scope over several sentences, in conditions that we have gone
some way towards exploring, they play a part in discourse organisation and more specifically discourse
segmentation. We have focused on ITEs, as we started from the accepted view that FTEs can only give a
time specification, and not open a frame. In this perspective, it should be out of the question for FTEs to
share the scoping properties of ITEs. And yet, reconsideration of the behaviour of FTEs in the corpus
questions this initial dichotomy. We find examples of FTEs which appear to have extended scope:
(6)
Le nombre d'él&#232;ves par classe stagne dans les coll&#232;ges depuis plusieurs années : la baisse sensible du nombre
d'él&#232;ves s'accompagne en effet de nombreuses suppressions de postes d'enseignants, récupérés pour les lycées.
(7)
En moyenne, 600 &#224; 700 écoles ont été fermées chaque année depuis 1980 : quelques -unes par fusion entre
l'ancienne école de filles et l'ancienne école de garçons ; d'autres dans le cadre d'un regroupement pédagogique
intercommunal ; les plus nombreuses par baisse des effectifs au -dessous du seuil de fermeture d'une école réduite au
fil des années &#224; une seule classe […]
(8)
Plus des trois quarts des 800 000 él&#232;ves qui étaient en 5e en 1991-92 sont passés en 4e. La proportion est en
augmentation continue depuis quatre ou cinq ans. En revanche, les orientations vers les préparations aux CAP de
l'enseignement technique (2 % des él&#232;ves) sont de moins en moins fréquentes - pr&#232;s de 10 % des él&#232;ves sont
maintenant orientés vers les 4e technologiques.
In each of these examples, one might think that the FTE provides an interpretation criterion which applies
to the next sentence(s). Section 4 below proposes a closer look at the data in order to establish whether one
are outside the frame introduced by depuis (which would be confirmed by the translation with a present
tense in English: the present tendency is...).
164
can legitimately consider that FTEs have scoping properties, thus questioning the classical view of the
discourse role linked to sentence position.
4. Do FTEs have scoping properties?
On closer examination, it appears that the sentences which might appear to be within the scope of
an FTE always enter into specific relations with the sentence specified by this FTE. We approach these
discourse relations within the framework of RST (Mann & Thompson 1988), without presenting the model,
but only providing the elements which are necessary for our purposes. In examples (6) and (7), we identify
respectively relations of Non-Volitional Cause and Elaboration: in the sentence(s) after the colon, elements
are given which provide a cause for the situation described in the proposition specified by the FTE, or
develop this proposition. As these relations do not in any way involve temporal considerations, it seems
logical that the temporal specification continues to apply. The same is true of the Contrast relation in (8): a
Contrast relation clearly extracts the elements in opposition, while all others are assumed to remain
constant. Here, the contrast is established between two school orientations, and by default the time period is
assumed to remain the same. We would therefore argue that what is a stake here is not a scoping property
of FTEs, but a default continuation of a temporal specification. Nevertheless, we are aware that this point
remains excessively based on the theoretical postulate that we precisely wanted to validate with our data. It
would undoubtedly be more satisfactory to find empirical evidence to support our argument.
In order to eliminate the possibility of temporal specification by default, we must envisage relations
with temporal implications. It is in such cases that the scoping property may be fully thrown into light.
Sequence is such a relation: by definition, it involves progress in time. If we find a temporal expression
which extends over several sentences in a Sequence relation, we can be sure that this extension corresponds
to a frame scope, and not to a default continuation: it cannot be a default if the relation implies a moving
forward of the temporal reference point. We re-examined our data in this perspective, looking for this kind
of configuration. In accordance with our hypothesis, we found no FTE exhibiting what would constitute
evidence of a scoping property. Unfortunately, we did not find positive evidence either, i.e. ITEs extending
their scope over sentences in a Sequence relation. It became clear to us that this could be due to the
expository nature of our corpus. So with the same method of identification of temporal PPs, we investigated
a corpus made up of film summaries and film reviews8, more likely to contain narrative elements. Indeed,
in this new corpus, the configuration we were looking for was frequent, and, in accordance with our
expectations, only with ITEs. We give below two relevant examples:
(9)
Flash-back : pendant l'été 1945, les bombardiers américains B-29 déversent des tonnes de bombes incendiaires sur la
ville de Kobe. D'innombrables victimes périssent dans un gigantesque incendie. Parmi les rares survivants, dans un
paysage d'Apocalypse, Seita un jeune garçon de quatorze ans prot&#232;ge sa petite s&#339;ur de quatre ans, Setsuko. [...]
(10)
Au début des années 50, le po&#232;te Pablo Neruda, sous le coup d'un mandat d'arr&#234;t au Chili parce que communiste,
arrive en exil en Italie avec sa femme et s'installe dans une petite île du Sud. Mario Ruoppolo, fils de p&#234;cheur en
chômage, est recruté par le chef du bureau de poste local comme facteur auxiliaire dont Neruda sera l'unique client.
Apr&#232;s une prise de contact assez froide, les deux hommes sympathisent […]
In both cases, we have a sequence of events which must be interpreted as taking place in the period referred
to by the ITE.
The existence of such examples, together with the absence of similar patterns with FTEs, at last
provides clear evidence of what we wanted to show: only for ITEs is it appropriate to use the notion of
scope.
Another dimension is opened up by some work in progress seeking to compare the potential of ITEs
for extending their scope and delimiting segments of text, with what happens with certain kinds of titles.
Our Cinema corpus provides us with several examples of interest to this question. They contain temporal
expressions which, although their status is unclear and needs investigating, share some characteristics with
titles (in particular they constitute an autonomous punctuational unit): what is worth noting is that these
temporal expressions behave very much like ITEs in their ability to scope over a – potentially large –
8 The Cinema corpus (81397 words), extracted from two Internet sites, the television channel Canal+ and
the daily newspaper Libération.
165
number of sentences. Thus, Juillet 1914 in (11) works in a very similar way to pendant l'été 1945 in (9) or
Au début des années 50 in (10).
(11)
Juillet 1914. Le château du vieux comte Pascal de Sainteville dans la France profonde entre "campagne bucolique"
et "montagne magique". Pierre Mercadier, riche rentier et professeur dilettante d'histoire, Paulette, sa femme, qui
"n'aime rien ni la musique ni l'amour" et leurs enfants, dont Gabriel, 12 ans, débarquent pour l'été chez leur oncle
Pascal. Ils sont déçus de constater que le comte, ruiné, a d&#251; louer une partie du château aux Pailleron : Ernest,
ouvrier puis contremaître qui a épousé Blanche, la fille du patron, avant de devenir patron lui-m&#234;me, Suzanne, leur
fille et Yvonne, son amie pianiste. Le charme de Blanche séduit Pascal, Pierre et Gabriel, mais Pierre a la préférence.
Gabriel entreprend son initiation aux choses de l'amour avec Suzanne et Yvonne sans craindre de les faire souffrir
alternativement. [...]
5. Conclusion
The corpus analysis has enabled us to throw light on several aspects of temporal discourse framing.
As concerns the distinction between initial and final temporal expressions, the homogeneous (expository)
nature of the Atlas corpus was at once a disadvantage and an advantage. It did not provide data capable of
clearly establishing the specificity of ITEs as regards scoping. But it revealed that all the contexts that could
be interpreted as pointing to a similar scoping behaviour of ITEs and FTEs displayed the same kind of
discourse relation. In turn, this observation led us to formulate the hypothesis that texts with a narrative
component would be more likely to provide suitable evidence. And indeed this was the case. This little
detour urged us to look into the relationship between discourse framing and another form of text
organisation, i.e. discourse relations, as regards temporal structure. Furthermore, it clearly points to the
necessity to take genre into account in any corpus-based study.
As a preliminary analysis, the work presented here simplifies the issue in dealing exclusively with
the two most distinct cases. To be exhaustive, it will need to be completed by a study of median and
detached final temporal expressions. Our intuition is that their scoping behaviour is similar to that of FTEs,
but this needs to be supported by a corpus analysis such at this one.
Our second objective was to investigate the linguistic signalling of frame boundaries. We have
started exploring three clues which seem to be commonly involved in the identification of the end of a
frame. But we are very conscious that this exploration must be pursued much further, if it is to account for
the complexity of the data.
We remained somewhat evasive as to the first clue: occurrence of a temporal expression referring to
a new time. In our examples the frame boundary coincided with the opening of a new frame by a second
ITE. Yet we found examples (which we did not mention for the sake of simplicity) where what appeared as
a new segment only contained a temporal specification: that this should be enough to indicate the end of the
preceding frame shows that the end of a frame does not necessarily signify the opening of a new one. This
has to do with the fact that the theory of discourse framing constitutes a partial model of text organisation:
as not all segments are frames, some other inter-segment organising principles must intervene, such as
those (mentioned above) that call upon discourse relations. Furthermore, we came across some examples
where the discourse relation alone seemed to imply the end of a frame, in particular when a comment is
made on information presented within a frame9.
As regards the second clue (change of tense from non-present to present), the observations we made
are indeed representative of the data in our corpus. However, given that in the French tense system, the
perfect for instance is related to both present and non-present, the interaction between tenses is likely to
lead to a much more complex situation
As we are interested in text segmentation, the third clue (change of paragraph) should be looked at
in relation to other forms of visual segmentation in the light of studies such as those of Virbel (1989, 2001)
and Nunberg (1990).
9 As in the following example, where the comment on what has happened depuis quelques années (“for the
last few years”) must clearly be interpreted outside this same temporal frame:
Depuis quelques années, en prenant pour base de référence l' annonce ministérielle de l'objectif d'amener les trois
quarts des jeunes de chaque classe d'âge aux niveaux de formation correspondant &#224; la classe Terminale de lycée
(1985), les caractéristiques principales du syst&#232;me éducatif français sont en cours de rapide mutation. Quelques
chiffres donnent la mesure de ces changements : […]
166
A final comment: given our objective of finding clues to final boundaries, we acted as though these
boundaries could always be unambiguously established. But we realise that there are also many cases when
the boundary is left unspecified, without it hampering the reading process. There is nothing surprising
about this, as underspecification is a general linguistic principle. However it would be worth quantifying
this tendency with regard to temporal framing, and extending the quantitative study to different types of
framing, such as the perspective framing mentioned in the introduction.
References
Bally C 1944 Linguistique générale et linguistique française, 2&#232;me édition. Berne, Francke.
Berthonneau A-M 1989 Composantes linguistiques de la référence temporelle - Les compléments de temps,
du lexique &#224; l'énoncé. PhD thesis, Université de Paris VII.
Bessonat B 1988 Le découpage en paragraphes et ses fonctions. Pratiques 57: 81-105.
Charolles M 1997 L'encadrement du discours : Univers, champs, domaines et espaces. Nancy, Université
de Nancy2.
Firbas J 1972 On the Interplay of prosodic and Non-prosodic Means of Functional Sentence Perspective. In
Fried U (ed), The Prague School of Linguistics and Language Teaching. London, Oxford University Press.
Givón T 1979 On Understanding Grammar. New-York / San Francisco / London, Academic Press.
Lambrecht K 1994 Information structure and sentence form. Topic, focus and the mental representations of
discourse referents. Cambridge, CUP.
Longacre RE 1979 The paragraph as a grammatical unit. In Givón T (ed) Syntax and semantics vol 12.
Discourse and syntax. New-York / London, Academic Press, pp 115-134.
Luc C, Virbel J to appear 2001 Le mod&#232;le d'architecture textuelle - Fondements et expérimentation.
Verbum 23 (1).
Mann WC, Thompson SA 1988 Rhetorical Structure Theory: toward a functional theory of text
organization. Text 8 (3): 243-281.
Nunberg G 1990 The Linguistics of Punctuation. Menlo Park, CSLI.
Péry-Woodley M-P 2000 Cadrer ou centrer son discours ? Introducteurs de cadres et centrage. Verbum 22
(1): 59-78.
Thompson S A 1985 Grammar and Written Discourse: Initial vs. Final Purpose Clauses in English. Text 5
(1-2): 55-84.
Virbel J 1989 The Contribution of Linguistic Knowledge to the Interpretation of Text Structures. In J.
André J, Quint V, Furuta RK (eds), Structured Documents. Cambridge, CUP.

167
Measuring morphological productivity:
Is automatic preprocessing sufficient?
Stefan Evert & Anke Lüdeling,
Institut für Maschinelle Sprachverarbeitung, University of Stuttgart, Germany.
e-mail: {evert, anke}@ims.uni-stuttgart.de
1. Morphological productivity
In this paper we want to focus on a small facet of morphological productivity: on quantitative measures
and their applicability to “real life” corpus data.1 We will argue that – at least for German – there are at
present no morphological systems available that can automatically preprocess the data to a quality
necessary to apply statistical models for the calculation of productivity rates.2
Before coming to the quantitative aspects we want to clarify the notion morphological productivity.
Morphological productivity has long been a topic in theoretical morphology (see for example Schultink
1961, Aronoff 1976, van Marle 1985, and Plag 1999). It has been defined in many ways. We choose a
definition by Schultink (1961, p. 113) which contains three aspects that are important to us:
We see productivity as a morphological phenomenon as the possibility for language users to coin
unintentionally an in principle unlimited number of new formations, by using the morphological procedure that
lies behind the form-meaning correspondence of some known words.3
The three important aspects are unintentionality, unlimitedness, and regularity. They are all
interdependent. The first aspect – unintentionality – helps us to distinguish between productivity
(which is a linguistic rule-based notion) and creativity (which is a general cognitive ability and cannot
be captured within morphology alone): Words formed by productive processes are often not recognized
or noticed as new words (this is true for speaker and hearer) while words formed by other (creative)
processes are carefully produced and are perceived as new words. The second aspect is unlimitedness –
if productive word formation patterns are in principle unlimited, it is not possible to give a finite list of
words (some implications of this are discussed below). Both unlimitedness and unintentionality require
that the words formed by a given process are morphosyntactically and semantically regular.
Theoretical and descriptive works on word formation mostly focus on what Baayen (1992) calls the
qualitative aspect of productivity: the morphological, phonolocical, syntactic, semantic and other
restrictions of a specific word formation process are studied. The adjective-forming suffix –bar
(roughly comparable to English –able), for example, takes as bases transitive activity verbs (lesbar
“readable” from transitive lesen “to read”, but not *schlafbar from intransitive schlafen “to sleep” or
*wissbar from stative wissen “to know”), the noun-forming circumfix Ge- -e does not take prefix verbs
(Geschimpfe “continued or iterative scolding“ from simplex schimpfen “to scold” but not
*Gebeschimpfe from prefixed beschimpfen “to insult”). The goal is an intensional description of the
possible bases for a given word formation process. Next to qualitative approaches to productivity,
quantitative approaches are suggested (Baayen 1992, Baayen and Lieber 1991, Baayen 2001). These
aim at calculating the probability of finding a new word formed by a given morphological process in a
text once a given amount of text is sampled (see Section 2).
What is the relevance of quantitative approaches to productivity? What can quantitative approaches tell
us that qualitative approaches cannot? At first glance the qualitative view of productivity differs
fundamentally from the quantitative view: A pattern X can be fully productive in the qualitative
analysis – it applies to all possible bases – but unproductive in the quantitative analysis, if there is a
finite number of bases and all the words that pattern X can possibly form from these bases have been
sampled. The probability of encountering new words formed by that word formation process is then 0.
But qualitative and quantitative approaches really complement each other. First of all, quantitative
approaches cannot be used without careful linguistic interpretation, as we will see below. From the
1 We illustrate our points with German data from the StZ corpus, which contains roughly two years of the newspaper Stuttgarter
Zeitung (1991 – 1993, 36 million tokens). We suspect that the problems that arise when corpus data is used for quantitative
approaches to productivity are similar for other languages.
2 We will not deal with the statistical models or the mathematics behind them in this paper. Excellent work on morphological
productivity has been done by Harald Baayen in a series of articles over the last 10 years (starting with Baayen 1992). He gives a
detailed overview over the LNRE models that can be used to calculate productivity in Baayen (2001).
3 Our translation of “Onder produktiviteit als morfologisch fenomeen verstaan we dan de voor taalgebruikers bestaande
mogelijkheid door middel van het morfologisch procédé dat aan de vorm-betekeniscorrespondentie van sommige hun bekende
woorden ten grondslag ligt, onopzettelig een in principe niet telbaar aantal nieuwe formaties te vormen.”
168
opposite perspective, quantitative studies help us to find out more about the nature of word formation
processes: Baayen and Neijt (1997) show, for example, that the words formed by some word formation
patterns really belong to two different distributions: the lexicalized words behave statistically different
from the productive words. This insight is made possible by studying the shape of the distribution and
analysing its anomalies. See also Baayen and Lieber (1991) for an overview of how linguistic
knowledge and quantitative results profit from each other.
In addition to being a valuable facet of the linguistic study of productivity, quantitative approaches also
make a significant contribution to computational morphology. Many applications (for example machine
translation, dialogue systems, text-to-speech systems) have to deal with unseen text. This involves not
only parsing unseen sentences but also analysing new words. Because of productivity it is not possible
to have a finite lexicon containing all the words of a language. Quantitative productivity studies help us
determine which patterns can be listed and for which patterns rules have to be formulated.
In Section 168 we will introduce a few statistical notions necessary for the understanding of
productivity measures and sketch how statistical models dealing with productivity work. These
methods depend on corpus data. But using corpus data is quite problematic, as discussed in Section 3.
The data have to be thoroughly preprocessed before quantitative measures can be applied. We will
investigate how automatic preprocessing compares with manual preprocessing in Section 170.
2. Vocabulary growth curves
A minimal understanding of the statistical ideas is essential as a background for the following sections.
We do not have the space to introduce the statistical assumptions and models in detail and have
therefore tried to define the necessary notions rather intuitively. For a formal introduction to LNRE
models refer to Baayen (2001).
For a quantitative approach to productivity, we need a mathematical definition of the degree of
productivity, based on observable quantities. Following Schultink's definition of morphological
productivity, we define the vocabulary of a morphological process as the number of types (i.e different
lemmata) that the process can potentially generate. A productive pattern is, in theory, characterised by
an infinite vocabulary (cf. the notion of unlimitedness in Schultink's definition), whereas a totally
unproductive pattern is expected to have a finite, and often quite small, vocabulary.
In order to estimate the vocabulary size S of pattern X from a corpus, we look at the subset of the
corpus consisting of all tokens generated by X (e.g. all tokens with the suffix –bar), in the order in
which they appear in the corpus. Let V be the number of different types among the first N tokens in the
subset. Plotting V against N, we obtain vocabulary growth curves as shown in Figure 1. If we had an
infinite amount of data, we would eventually sample the entire vocabulary of X, and V would converge
to S. The left panel of Figure 1 shows the typical shape of the vocabulary growth curve of an
unproductive process. After enough data has been sampled, the curve flattens out and converges to a
constant value, the full vocabulary size S. In the case of an infinite vocabulary, V would continue to
grow indefinitely (see the right panel in Figure 1).
Figure 1: Typical shapes of vocabulary growth curves (idealised).
Baayen (1992) uses the slope P of the vocabulary growth curve after the entire corpus has been
sampled as a measure of productivity (P is referred to as the productivity index of pattern X).
Intuitively speaking, P is the probability that the next token generated by X is a previously unseen type,
169
i.e. a new complex form. Under certain simplifying assumptions we find that P = N
n1 , where n1 is the
number of hapax legomena (words occurring only once, cf. Baayen 2001, Chapter 2).
Unfortunately, P cannot be used to compare the productivity of affixes with substantially differing
sample sizes. If we look at the left panel in Figure 1, we see that for the full sample, P=0 as we would
expect from a completely unproductive process. However, had we looked at the first 15 or 20% of the
curve (e.g. because we are working on a small corpus, or sampling a rare pattern), we would have
observed a much larger value of P. Larger, perhaps, than that of the productive process in the right
panel (for the full sample). Since we do not know where in the curve we are, we cannot simply
compare P values for processes of different sizes.
In order to compare different processes, we therefore need to be able to extrapolate the value of P to
larger sample sizes N, or, equivalently, extrapolate the shape of the vocabulary growth curve. We
cannot rely on standard statistical models for this purpose because the type frequency distributions of
productive patterns are so-called LNRE distributions (for “large number of rare events”), in which lowfrequency
types (including hapax legomena, but also types occurring two, three, etc. times) account for
a major part of the vocabulary. Ordinary statistical techniques are based on assumptions that do not
hold for LNRE distributions (the most prominent one being the law of large numbers). Hence, Baayen
(2001) introduces specialised models, for which parameter estimates are obtained from the counts of
low-frequency types in the corpus (the frequency spectrum). These models can then be used to
extrapolate the vocabulary growth curves to larger values of N. Baayen's comparison of the degrees of
productivity of different patterns is based on the predicted vocabulary sizes.
Any errors in the type counts have a direct influence on the model parameters, and thus on the
predicted vocabulary size. It is therefore essential to correct the input data for the errors described in
Section 3.4
3. Problems with corpus data
Since we want to compute the probability P of a new complex word formed by a given word formation
pattern in a given text type (for example newspaper data or technical text) and because the productivity
of word formation patterns is highly dependent on text type (neoclassical words, for example are much
more likely in scientific texts than in everyday language, see also Baayen 1994) it is necessary to
sample a corpus of that text type. Dictionary data cannot be used for our purpose for two reasons:
(i) Dictionaries often contain obsolete words but typically do not contain regular newly formed words.
(ii) The advanced statistical models introduced in Section 2 are based on vocabulary growth curves that
can only be computed from corpus data.
If we want to apply the measures described in Section 2 to real corpus data we encounter a number of
problems. Lüdeling, Evert, and Heid (2000) describe these problems and show that the data have to be
preprocessed before they can be used. In this section we want to summarise the problematic factors
briefly, using two adjective forming suffixes (–sam and –bar) for illustration. The intuition is that –sam
is unproductive while –bar is productive. If we simply extract all adjectives ending in the letters sam
and bar from the STZ corpus and calculate P from the frequency spectra we get very similar graphs
that suggest that both processes are productive (the solid lines in Figure 2). A closer look at the data
reveals that the data have to be corrected for the following factors:
·  incorrect data
o mistagged items
o misspelt items
o corpus structure: repeated articles and sections influence the frequency distribution
·  linguistic factors: many complex words that look like they are formed by the word formation
process in question are really formed by other processes:
o compounds: these make up the largest portion of the hapax legomena of most word
formation patterns. We have to distinguish between complex words where the
4 Baayen (2001) also describes more sophisticated models (adjusted and mixture distributions) which are estimated from the
(known) shape of a pattern's vocabulary growth curve. Just as it is the case for the simpler models, errors in the type counts have
a significant influence on the shape of the vocabulary growth curve and hence on the predicted vocabulary size.
170
compounding happens before the derivation (with the structure ((stem+stem)+affix))
and cases in which something attaches to an already affixed word
(stem+(stem+affix)). The former cases have to be counted as new types while the
latter cases have to be counted as instances of the affixed word. We will discuss an
example in Section 170.
o other complex bases: likewise we have to distinguish between cases in which the
affix in question attaches before other affixes or after them. For –bar adjectives that
also contain a prefix, for example, we have to decide whether the prefix is attached to
the –bar adjective (like the negation prefix un- which attaches to verzichtbar in
unverzichtbar “indispensable” and thus should not be counted as a new type for the
-bar pattern) or to the verb which is the base for –bar adjectivisation (fahrbar
“ridable” is formed from the simplex verb fahren “to ride” while befahrbar
“passable” is formed from the prefix verb befahren “to pass over, to drive on” – in
this case we have two new types).
o words that “accidentally” end in the same letters: Balsam “balm”, for example, does
not contain the suffix –sam.
o words formed by creative rather than by productive processes: kinobetriebsam is a
creative formation merging betriebsam “busy” and Kinobetrieb “movie theatre
business”.
As we emphasised in Section 2, the correct application of the guidelines is so important because many
of the problematic cases are hapax legomena and thus have a direct influence on the productivity
index P.
In Lüdeling, Evert, and Heid (2000) we manually corrected the data for –sam and –bar according to
these guidelines. The dotted lines in Figure 2 show the vocabulary growth curves for the corrected data.
They now conform to intuition: -sam is clearly unproductive while the curve for -bar still shows a
productive pattern.
Figure 2: Raw and manually corrected vocabulary growth curves for –sam and –bar.
4. Automatic preprocessing vs. manual preprocessing
It is not feasible to correct the input for all word formation processes manually since some word
formation patterns are very large. It would, therefore, be desirable to preprocess the input
automatically. In this section we want to explore how automatic preprocessing compares to manual
correction. We will find that automatic preprocessing – at least with the tools that are available to us –
is not yet a usable alternative to manual preprocessing.
Before we describe the programs we used we want to recapitulate the requirements for morphological
preprocessing: in order to perform like a human corrector, a morphology program would have to be
able to (a) find spelling and tagging mistakes, (b) make a morpheme analysis, and (c) provide the
171
correct hierarchical structure for complex words consisting of more than two morphemes. At least for
German, there are no programs available that meet these requirements for unseen words.5
Because no fully fledged morphology analyser is available, we wanted to find out how much
preprocessing can be done automatically with the available tools. We chose:
·  MORPHY, a freely available German morphological analyser (see Lezius, Rapp, and Wettler
1998). It was developed mainly for the analysis and lemmatisation of inflected forms, as well
as context-sensitive part-of-speech tagging. In our experiment we used the built-in lexicon of
MORPHY together with a simple heuristic compound analysis.
·  DMOR, a two-level morphology system developed at the IMS Stuttgart (cf. Schiller 1996).
Like MORPHY, it is intended mainly for generating and analysing inflected forms, but it
includes compound analysis as well: compounds are analysed as sequences of stems and
linking elements (somewhat restricted by simple heuristic patterns). The DMOR lexicon
contains 65,000 stems (mostly simplexes, but some complex words are listed).
Neither program analyses derivational morphology. The built-in stem lexicon of DMOR is
considerably larger than that of MORPHY.
We manually corrected the output for a number of word formation patterns and compared the
vocabulary growth curves of uncorrected (raw), manually corrected and automatically corrected data.
Figure 4 shows the manually and automatically corrected plots for –bar and –sam. The automatically
corrected curves lie between the uncorrected curve and the manually corrected curve in both cases. For
–bar, the results produced by MORPHY (dashed lines) are hardly different from the uncorrected curve;
DMOR (broken lines) performs somewhat better. For –sam, the automatically corrected curves are
much closer to the manually corrected curve. Qualitatively, the autocorrected curves are similar to the
manual result and hint at an unproductive process (although not as clearly as the manually corrected
data). Why does automatic correction work so much better for -sam than for -bar? This is due to the
fact that almost all of the hapax legomena in the –sam spectrum are compounds involving only very
few head types. The compound analysis components of DMOR and MORPHY seem to be able to deal
with these cases quite well. The hapax legomena in the –bar spectrum are much more diverse: un-
prefixations, many spelling errors and very few compounds – since MORPHY and DMOR do not have
derivation analysis components, they cannot correct these cases.
Figure 4: Vocabulary growth curves for –bar and –sam.
The next process we look at is the diminutive suffix –chen which combines with nouns (and sometimes
adjectives) to form nouns. Here again we have a pattern where the automatically corrected curves are
5 There are a number of morphology programs which analyse complex words
http://services.canoo.com/MorphologyBrowser.html
http://wortschatz.uni-leipzig.de/
http://www.linguistik.uni-erlangen.de/cgi-bin/orlorenz/dmm.cgi
As far as we can see from the available versions, they all work with finite lexicons. There are no downloadable tools or word
lists. None of them provides hierarchical structure.
172
somewhere between the uncorrected curve and the manual curve, with DMOR (with its larger lexicon)
giving considerably better results than MORPHY (Figure 5). Note, however, that the manually
corrected curve has become much shorter, i.e. the number of tokens has been reduced. This is due to
the fact that many words “accidentally” end in –chen (such as Groschen “ten-pfennig piece”, Drachen
“dragon”, or Zeichen “sign”). Because of the resulting smaller sample size the manually corrected
curve cannot be compared directly to the other curves (compare Section 2). However, the shape of the
former suggests that –chen is much less productive than the automatically corrected curves would
predict.
Figure 5: Vocabulary growth curves for –chen.
To summarize the results so far: The improvement achieved by automatic preprocessing differs
considerably for the various processes. It never reaches the quality of manual preprocessing: in the case
of –bar there is little improvement over the raw data, in the case of –sam the automatically corrected
data comes close to the manual curve, and for –chen, automatically corrected curves are approximately
in the middle between the uncorrected curve and the manual curve. For –chen, we also observe the
largest difference between the two morphology systems. Since we cannot predict how good the results
are compared to manually corrected data, automatic preprocessing is no real alternative for manual
correction. However, it would seem that the morphology systems always lead to some improvement
and should therefore always be applied (manual checks may then be done on the automatically cleanedup
data).
However, when we look at another diminutive pattern (suffixation with –lein) and a process that is
semantically similar (compounding with klein “small”), we see a strikingly different pattern (Figure 6).
The vocabulary growth curves of klein compounds show the familiar pattern of a gradual improvement
from raw to manually corrected data. If we look at the plots for –lein, the uncorrected curve and the
manually corrected curve are very similar to those of the klein compounds. However, when we
automatically correct the data with DMOR we obtain a curve that is far below the manual curve, and
hence predicts a considerably lower productivity than we actually find. This is due to the fact that
DMOR contains the noun stem Lein “flax” and therefore analyses many derived words as compounds
with the head Lein.
173
Figure 6: Vocabulary growth curves for klein and –lein.
One could argue that the Lein/-lein homography might be a single example, and can easily be corrected
if we disallow reduction to the simplex Lein. The curves are given in Figure 7 which shows the familiar
pattern where the automatically corrected curves are between the manually corrected curve and the
uncorrected curve.
Figure 7: Vocabulary growth curves for –lein (DMOR curve corrected).
However, Figure 8 shows the same pattern for the noun-forming affix –tum. Again, the curve produced
by MORPHY is above the reference curve, whereas the curve produced by DMOR is below the
reference curve. –tum is an interesting case: DMOR finds roughly the same number of types as the
human annotator, but a considerably larger number of tokens. Although this would explain the
seemingly lower productivity indicated by the DMOR curve, the actual types are different, due to two
factors: as in the case of –chen, there are many words ending with the letter sequence tum which are not
derived by the suffix –tum (examples are Datum “date”, Faktum “fact”, or Quantum “quota”) – DMOR
mistakenly counts these types. The other factor has to do with hierarchical structure: There are many
compounds of the type NOUN+reich+tum which DMOR reduces to Reichtum “wealth”. The correct
structure, however, is ((NOUN+reich)+tum) which means that these cases have to be counted as
different types (an example is Kinderreichtum “having many children” which is derived from
kinderreich “prolific”, and not a compound of Kinder “children” and Reichtum). Here we need careful
linguistic analysis – a simple automatic solution is not possible.
174
Figure 8: Vocabulary growth curves for –tum.
5. Conclusion
Corpus data are necessary for the application of statistical models to the quantitative analysis of
morphological productivity. We summarised Lüdeling, Evert, and Heid (2000), who showed that
corpus data have to be thoroughly preprocessed before they can be used in these models. Since manual
preprocessing is not feasible for large word formation patterns, we wanted to find out whether
automatic preprocessing is a usable alternative. We found that although automatic correction, based on
the currently available morphology systems, yields an improvement over the uncorrected data in many
cases, it is no replacement for manual corrections.
In some cases, automatic preprocessing is possible – if only as a basis for further manual correction.
However, the over-compensation we observed for –lein and –tum shows that fully automatic
preprocessing may produce misleading results. Without a manually corrected reference curve we do not
know where the automatic curves for a given word formation process lie. This means that we cannot
even use automatic preprocessing as a reliable basis for manual correction. Only a morphology system
which, in addition to derivation and compounding components, includes a model of the order in which
processes operate on a simplex form (producing a hierarchical analysis of complex words) can be
expected to lead to sufficiently reliable automatic correction results for quantitative studies of
productivity.6
6. References
Aronoff M 1976 Word Formation in Generative Grammar. Cambridge, MIT Press.
Baayen R H 1992 Quantitative aspects of morphological productivity. Yearbook of Morphology1991:
109 – 150.
Baayen R H 1994 Derivational productivity and text typology. Journal of Quantitative Linguistics 1:
16 – 34.
Baayen R H 2001 Word Frequency Distributions. Dordrecht, Kluwer Academic Publishers.
Baayen R H, Lieber R 1991 Productivity and English derivation: a corpus-based study. Linguistics 29:
801 – 843.
Baayen R H, Neijt A 1997 Productivity in context: a case study of a Dutch suffix. Linguistics 35: 565
– 587.
Lezius W, Rapp R, Wettler M 1998 A freely available morphological analyser, disambiguator, and
context sensitive lemmatizer for German. In Proceedings of the COLING-ACL, pp 743 – 747.
6 Work leading to such a morphology system is currently under way in the DeKo project (Schmid et al 2001).
175
Lüdeling A, Evert S, Heid U 2000 On measuring morphological productivity. In Proceedings of the
KONVENS 2000, Ilmenau, pp. 57 – 61.
Plag I 1999 Morphological Productivity. Structural Constraints in English Derivation. Berlin,
Mouton de Gruyter.
Schiller A 1996 Deutsche Flexions- und Kompositionsmorphologie mit PC-KIMMO. In Hausser R
(ed), Linguistische Verifikation. Dokumentation zur Ersten Morpholympics 1994. Tübingen, Niemeyer
Schmid T, Lüdeling A, Säuberlich B, Heid U, Möbius B 2001 DeKo: Ein System zur Analyse
komplexer Wörter. In: Proceedings der GlDV, Giessen
Schultink H 1961 Produktiviteit als morphologisch fenomeen. Forum der Letteren: 110 – 125.
van Marle J 1985 On the Paradigmatic Dimension of Morphological Productivity. Dordrecht, Foris.

176
Linguistic clues for corpus-based acquisition of lexical dependencies
Cécile Fabre, Didier Bourigault
ERSS - UMR 5610
Université de Toulouse Le Mirail, 5 allées A. Machado, 31058 Cedex, France.
{cfabre,bourigault}@univ-tlse2.fr
1 SYNTEX: a tool for the extraction of lexical dependencies
Disambiguation of prepositional phrase attachment is a crucial issue for all NLP applications
that need textual resources enriched with syntactic knowledge. We are confronted to this problem in the
process of designing SYNTEX, a shallow parser specialised in the extraction of lexical dependencies
(such as adjective/noun, or verb/noun associations) from French technical corpora. These word-to-word
associations will be used as material for the construction of semantic classes on a distributional basis.
In this context, the first step towards the automatic discovery of such dependencies is to determine to
which word a preposition must be attached. For example, in the phrase disséquer le plateau rocheux en
chevron, taken from a corpus in the domain of geomorphology1, the preposition en may potentially be
attached to any of the three words disséquer, plateau, rocheux, as verbs, nouns or adjectives (and also
adverbs) may govern a prepositional phrase.
As lexico-syntactic information is part of what we want to extract from the text, we cannot
rely on prior lexical resources: it is our belief, based on in-depth studies of corpora from technical
domains, that words exhibit idiosyncratic uses from one domain to the other, not only at the semantic
level, but also regarding their syntactic properties. As a consequence, our parser relies as much as
possible on corpus-based information to solve the ambiguities of syntactic attachment. Our second
claim is that such disambiguation process can be performed on the basis of linguistic clues, with only
limited use of statistical measures.
In this paper, we describe a method which relies on the search for linguistic indications to
perform syntactic disambiguation. Our parser is based on two main ideas: first, the detection of
unambiguous contexts is used as a starting point for the processing of ambiguous contexts. Second, the
notion of productivity (more reliable than the simpler notion of frequency) is proposed to assess the
strength of a word/preposition association: we define the productivity of a (word,preposition) pair as
the ability of the word to appear with this preposition in various contexts. These two basic ideas are
refined with the help of further linguistic clues. In particular, we use morphological information and we
take into account semantic similarity between the words to evaluate the likelihood of the association
between a word and a preposition.
We first describe the modules of SYNTEX which perform prepositional phrase attachment. We
present the various information that the analyser exploits to resolve ambiguity, focusing on the notion
of productivity. We then analyse the linguistic relevance of this empirical notion of productivity: is it
feasible to use productivity measure as a means to differentiate between various levels of lexical
relations, and particularly to draw a frontier between arguments and non-arguments? The paper will
present the first results in favour of this hypothesis.
2 Description of the strategy for PPs attachment
Prepositional attachment resolution is usually considered as the first procedure that permits the
delimitation of phrases for automatic parsing (Brent 1993, Basili et al. 1999). This is crucial for NLP
applications such as text retrieval or information extraction (Grefenstette 1994). Our objective is also to
provide data about lexico-syntactic relations between words that will help to construct semantic
resources. Classes of words can be defined on the basis of their distributional properties, following the
propositions of Harris (Harris et al., 1999).
For example, in the geomorphology corpus, we are able to identify a set of four nouns -
alluvions, sable, dépôts, cendres - all sharing at least two lexico-syntactic contexts with each other:
- same V PREP N contexts:
1 All the examples in the paper are extracted from this geomorphology corpus. We are grateful to
Dani&#232;le Candel (INALF) who has made it available.
177
disparaître sous (des alluvions, du sable, des dépôts)
enfouir sous (des alluvions, les dépôts)
creuser dans (les alluvions, le sable, les cendres)
tailler dans (des alluvions, des cendres)
- same N PREP N contexts:
manteau de (alluvions, dépôts)
banc de (alluvions, sable)
These four nouns indeed appear to be semantically close, all denoting some sort of sediment.
The extraction of syntactically and lexically related pairs of words is therefore a means to discover
corpus-specific classes with semi-automatic techniques. Research initiated in French towards this goal
has suffered from the lack of automatic parsers for this language: (Habert et al. 1996), (Assadi and
Bourigault 1995) were both taking as input the results of a NP parser, LEXTER (Bourigault, 1994), and
were not able to exploit data concerning verbs. SYNTEX provides an extension to LEXTER by identifying
all types of lexical dependencies involving verbs, nouns and adjectives.
2.1 An inductive approach
The identification of lexical dependencies in texts may be performed by different techniques.
The first consists in the projection of a-priori lexical resources. There are major well-known problems
with this approach: firstly, such resources, which should not only contain information related to
argument structure but also deal with complementation and modification phenomena, are simply not
available. The process of constructing such lexical bases is long and costly, but most of all,
constructing a-priori resources is an endless task since idiosyncratic uses of words are found in corpora.
Work on technical corpora demonstrates that such texts exhibit a great variety of lexico-syntactic
properties regarding how words associate with each others. Of course, corpora contain also highly
predictible data: for example, we find that in the geomorphology domain, the verb débiter associates
with the preposition en without a determiner, from occurrences such as débiter en blocs, en chicots, en
granules, en lamelles, etc. This construction is described in any French dictionary. But corpus
observations also show many configurations that are totally idiosyncratic. This is especially true
concerning noun modification. To illustrate this point, we can point among numerous examples to the
existence of a lexical pattern of the form N pour det N with the non argument-taking, non relational,
headnoun salle (room), (in: salle pour l'étude des minéraux, salle pour la détermination des argiles). In
this corpus, nouns very often appear with the prepositions en (vallée en canyon, section en cluse) and &#224;
(section &#224; méandre, cirque &#224; source), which are much more frequently used as postverbal prepositions
in other types of texts. Since these properties do not concern argument selection and are very unstable
from one corpus to another, such data cannot be listed to be used on any type of texts. Another
argument against the use of prior resources is the variety of prepositional attachment patterns that exist
for a single lexeme. If subcategorization, complementation and modification phenomena (Grimshaw
1990) are all taken into account, it appears that many verbs and nouns can potentially associate with a
great range of prepositions. This is the case even in a limited domain. For example, the verb accumuler,
besides the transitive construction, gets constructed with six different prepositions, corresponding to
locative and instrumental interpretations (e.g. dans le creux, derri&#232;re l'obstacle, sur le névé). As a
consequence, the listing of all complementation alternatives – if possible at all – will not in itself be
suffichient to reduce ambiguity.
Alternatively, other approaches are proposed to learn this information from corpora in order to
avoid the preconstruction of lexical resources. Our strategy is in line with research adopting a
distributional methods to solve the ambiguities of syntactic analysis. (Brent 1993) was the first to take
into account lexical association measures in texts to identify verbs' arguments. A similar approach has
been adopted by (Hindle and Rooth 1993) or (Manning 1993), who defined methods to discover verbs'
subcategorization frames in texts. More recently, (Federici et al. 1999) combine the tabula rasa
approach and inductive technics for the parsing of Italian texts. Similarly, our parser uses information
extracted from the whole corpus to infer local decisions, exploiting lexical redundancy in technical
corpora to acquire patterns of prepositional attachment. But the novelty of our approach can be
characterized as the conjunction of three options:
- SYNTEX deals not only with subcategorization but also with any type of lexical dependency
involving prepositional attachment,
- it maximizes the use of linguistic clues, limiting the development of statistical methods to
perform disambiguation,
178
- it is mainly based on the productivity measure, a key notion to reach a decision in ambiguous
texts.
2.2 Disambiguation rules
SYNTEX is based on inductive learning of complementation properties. The parser looks for
triplets (governor, preposition, governee) linked by a dependency relation2. This relation may
correspond to subcategorization patterns - as in the triplets (pénétrer, dans, pore) or (accessible, &#224;,
bâteau) - or it may illustrate non argumental associations, such as verb + circumstant – (déplacer, &#224;,
vitesse) -, noun + expansion (côte, &#224;, fjord), etc. Learning is performed in two steps: first, properties of
lexical associations are acquired from unambiguous contexts; second, they are used as indications to
solve ambiguous cases throughout the corpus. This strategy is very close to the approach described in
(Federici et al. 1999) for shallow parsing of italian. More generally, it is not uncommon to see parsers
exploiting unambiguous contexts to limit the complexity of the analysis. The novelty of SYNTEX is in
the criteria used to identify the triplets likely to correspond to genuine dependency relations.
2.2.1 Detection of unambiguous contexts
Learning is first performed by detecting attachment zones in the corpus. Such zones are
delimited to the right by a preposition and to the left by a frontier, which may not, or may rarely be
crossed over by prepositional attachment. Such frontiers are punctuations, verbs, prepositions other
than de and &#224; or typographical items such as parentheses. In the following examples, these zones are
enclosed within brackets and underlined.
i. On tend de plus en plus &#224; ]insérer cette science dans] une géographie physique globale
ii. transformations du relief des versants], conséquences des actions de l' homme sur] le sol
Obsviously, most frontiers are not entirely reliable. The prepositional link can be established
over interpolated verbal phrases, over prepositions, etc. as in the two following examples, where the
preposition and its governor are underlined:
iii. La puissance est utilisée , on s'en souvient , en partie par le transport de la charge
iv. On peut mesurer la vitesse d'infiltration d'une goutte d'eau déposée sur la roche par sa
disparition de la surface
The segmentation module is therefore, for the moment, very rudimentary, but it enables us to
considerably limit the complexity of the attachment procedure. As a first approximation, we have
measured the silence due to "preposition jump" (a preposition finds its governor over a preposition
other than de or &#224;) as around 5% on one of our technical corpora, which is a relatively small
proportion.
Within these zones, all lexical units are viewed as potential governors of the preposition.
Example ii contains three nominal candidates (conséquences, actions, homme), corresponding to three
potential triplets: (conséquence, sur, sol), (action, sur, sol), (homme, sur, sol). One condition must be
met for these triplets to be used in the learning process: they have to be found in unambiguous contexts
- containing no other potential governor for the preposition. For example, the triplet (glisser,sur,bord)
is extracted from the following unambiguous context:
v. des nappes de gravité ]glissent sur] le bord des surélévations
2.2.2 Acquisition of reliable dependency relations
Unambiguous cases are thus the starting point of the parser. The purpose of the acquisition
module is then to use unambiguous contexts as clues for the resolution of ambiguous cases. Yet, we
cannot consider all information found in unambiguous contexts as reliable. Several criteria are taken
into account. Given a triplet (Gvr, Prep, Gvee) found in an ambiguous context, it is considered as a
possibly genuine lexical relation if:
2 The governor is the word governing the prepositional phrase. It belongs to the categories verb,
adjective or noun. The governee is the word governed by the preposition. It may be a noun (as in
insérer dans une géographie) or an infinitive (as in tendance &#224; former). Both appear in lemmatized
form in our results.
179
·  Rule 13: The same triplet has been found in an unambiguous context.
Example: in the ambiguous context indique une lég&#232;re tendance &#224; l'enfoncement, where the
preposition may be governed by the verb indique or the noun tendance, the latter is considered as the
more probable governor, because the triplet (tendance, &#224;, enfoncement) has been found in an
unambiguous context.
·  Rule 2: The pair (Gvr, Prep) is productive.
Productivity of a (Gvr, Prep) pair equals the number of different governees with which the pair
(Gvr, Prep) occurs in the corpus. A word is considered as productive with a given preposition, if it
combines with at least two different governees. For example, given the following unambiguous
contexts found in the corpus, it appears that the verb disséquer is productive with the preposition en,
with a productivity of 5.
Figure 1: productivity of a (governor, preposition) pair
The productivity measure allows us to assess the likelihood of a word being used as a
governor of a preposition on a more reliable basis than the simpler frequency measure. High
productivity indicates that the governor regularly associates with this preposition, in various
occurrences. To illustrate this point, we can oppose three cases:
- unfrequent and unproductive association of a word and a preposition: the string annuler en général is
found only one time in the corpus. In this case, the occurrence corresponds to the association of a verb
and an adverbial phrase (meaning generally), and it does not indicate that the verb is constructed with
the preposition en.
- frequent but unproductive association: the strings an en moyenne, cro&#251;tes en Afrique, allonger dans la
direction are each found three times in the corpus. These findings may correspond to genuine lexical
dependencies but they cannot be used to infer a regular association between the head and the
preposition.
- productive (and necessarily frequent) association: the pair (disséquer, en) seems to indicate a regular
relationship between the verb and the preposition because they appear together in various contexts.
·  Rule 3: A word morphologically linked to Gvr is productive with the preposition.
The exploitation of such morphological links is useful because of the richness of the
morphological system in French. Relations between subcategorization properties of verbs and
argument-taking nominals are not systematic. Yet, they are sufficiently frequent to provide a clue for
disambiguation. We use a lexical resource, Verbaction4, which provides an inventory of process
nominals that are constructed on a verbal base. This information is used to solve ambiguities such as
the following:
vi. la formation &#224; blocs ayant soliflué plus rapidement, ]avec glissement rapide sur] l' ar&#232;ne lente
The only information that the corpus provides on the two potential pairs (glissement, sur) and
(rapide, sur), is that the verb glisser, morphologically related to the noun glissement, has been found as
governor of the preposition sur in eight unambiguous contexts (example v is one of them).
This indication is used to make the hypothesis that glissement may be a governor of the preposition sur.
·  Rule 4: A word semantically linked to Gvee has been found as governee of the pair (Gvr, prep).
A word is considered as semantically related to another, if both have at least two governors in common
in the corpus. They share some distributional properties. They are also said to be semantic neighbours.
3 The numbering of the rules does not indicate priority of application.
4 This morphological resource has been compiled by Nabil Hathout at the National Institute of French
Language (INaLF).
disséqué parfois en récif
disséquer en récif
disséqué en dents de scie
disséquer en terrasses
disséqués en bois de renne
disséquée en chevron
récif
dent
terrasse prod=5
bois
chevron
&#197; disséquer en
180
vii. nous] terminerons par quelques notes sur] la morphologie de la lune et de Mars .
The noun notes has been found as governor in an unambiguous context with a semantic neighbour of
the noun morphologie, namely forme. The proximity of the two nouns morphologie and forme has been
established on account of their sharing two unambiguous contexts:
viii. compréhension de + det + (forme, morphologie)
ix. s'intéresser &#224; + det (forme,morphologie)
The construction of semantic classes, which is the objective of our work, is thus also sketched out
during the syntactic analysis to provide indications for prepositional attachment.
2.2.3 Resolution of ambiguous contexts
Ambiguous cases are solved by using this combination of clues. In the following example,
rules 1 (same triplet) and 2 (productivity) are used to choose between the three potential governors.
x. L'érosion a disséqué le plateau rocheux en chevrons.
The verb is productive (prod=5) and it has been found with the same governee in an
unambiguous context. The noun plateau has been found only one time with the preposition en in an
unambiguous context, with another governee (des plateaux en interfluve); the adjective rocheux is not
found with this preposition.
This resolution module is currently under development. At the moment, precision is 86%,
which is very satisfactory, but the recall measure is only 60%. These results have been obtained by
comparing the results of SYNTEX to prepositional attachment manually performed on several thousands
occurrences in three different corpora. This relatively low recall corresponds to two situations: in the
first case, no potential governor has been found in the attachment zone. Recall must therefore be
increased by improving the segmentation module, through the definition of more flexible frontiers for
the attachment zones. In the second case, no indication could be used to choose between several
potential governors. It is due to lack of corpus evidence, so we must consider developping a default
strategy, or using some amount of prior knowledge when all corpus-specific information has been
exploited.
3 Productivity: a measure that helps to detect different levels of lexical dependency?
A further question is at issue in this experiment: what levels of linguistic information are we
able to point out from the observation of lexical associations in corpora? More precisely, we want to
know if the strength of the association between a governor and the preposition, that we have measured
in terms of productivity, can be used to describe the type of relation - subcategorization or adjunction -
that holds between the two words. (Brent 1993) claims that there is a connection between frequency
of occurrence and type of complementation: two words are more frequently associated if they are
associated by a grammatical relation. According to this view, heuristics based on frequency of
cooccurrence should therefore enable us to make this fundamental distinction between arguments and
adjuncts. On the contrary, (Basili et al. 1999) think that this distinction cannot et should not be made by
automatic means, because adjuncts equally contribute to the verb semantics and are as regularly
associated with the verb as arguments. Observations made on corpora indeed show that the frontier
between the two types of complementation is very difficult to draw, even when we want to manually
determine prepositional attachment. Yet, we wanted to know if it was feasible to go beyond the simple
diagnosis of prepositional attachment, and to rely on the productivity measure to try and detect
different types of prepositional phrases. The last part of this paper is devoted to the presentation of the
first results regarding this issue.
3.1 Variety of lexical dependencies
In the previous sections, we have encountered examples of prepositional attachments that
illustrate the diversity of semantic relations between a governor and a PP. SYNTEX resolves
prepositional attachments corresponding to different types of lexical associations, namely:
- an argument-taking element with its argument
verb: s'enfoncer dans + det + (alluvions,eau, fond, surface)
adj: sujette &#224; + det + (bouleversement, gel,variation, émiettement)
noun: déversement dans + det + (bassin, cuvette)
181
- an argument-taking element with a complement
verb: disparaître dans (le bassin,le gouffre,le lac,le puits)
noun: ruissellement en (nappes, rigoles, films)
- non argument-taking element with a complement
adj: active dans + det + (la baie, la zone)
noun: équilibre entre (forces,puissances)
noun: vallée &#224; (flancs,replats)
All these relations prove to be useful for the construction of semantic classes: words may of
course be grouped because they share arguments, or they may be grouped because they are arguments
of the same words. But non-argumental relations are also useful for the construction of homogenous
sets of words. To illustrate this point, we can point to the fact that nouns that appear in the same list of
complements in the previous examples are closely related, such as the three nouns nappes (sheet),
rigoles (rills), films (films), or the four nouns bassin (basin), gouffre (chasm), lac (lake), puits (well).
As a consequence, it would be very useful to propose some clues in order to differenciate
between these types of dependencies. One track that we are currently following consists in the
measurement of different types of productivity. Our objective is to find criteria to differentiate between
argument relations and other levels of complementation relations. With this in mind, we have tried to
compare two types of productivity: the productivity of governor-preposition pairs, exemplified so far,
and the productivity of preposition-governees pairs. In the latter case, saying that the governee is
productive with a preposition means that it occurs in the scope of various governors. For example, the
pair (par, mer), separated by a determiner, is productive because it occurs under the government of six
different verbs:
battues
coupé
déposé + par la mer prod=6
envahie
occupée
recouvert
Figure 2: productivity of a (preposition,governee) pair
We have compared the lexical information that is acquired from these two different criteria.
Our first conclusions are illustrated by the observation of two prepositions: sur and &#224;, in verbal and
nominal contexts.
3.2 Verbal complementation: V sur det N phrases
We compare two lists: the first one (figure 3) is made up of 15 verbs that are found to be most
productive as governors of the preposition sur (+ determiner) in our corpus. The first line of the table
indicates that the verb reposer has been found before the preposition sur in unambiguous contexts with
23 different right contexts, which are all nouns (reposer sur le banc, reposer sur la couche, etc.).
governor governees prod
reposer banc,couche,crit&#232;re,critérium,dos,fond,galet,horizon,lit,marne,mesure,plancher,précédent,pét
ition de principe,reg,remarque,restitution,roche,sable,socle,substratum,surface,étude
23
renseigner climat,condition,constitution,degré,direction,intensité,morphogen&#232;se,mouvement,nature,pro
venance,rapport,relief,sens,topographie,valeur,évolution
16
situer bord,cr&#234;te,dôme,emplacement,face,front,ligne,passage,plan,trajet,versant,équateur 12
emporter accumulation,altération,amplitude,creusement,exportation,mode,proportion,roche,élément,ér
osion
10
rencontrer croupe,côte,flanc,granit,niveau,paroi,pente,revers,roche 9
trouver bord,emplacement,glacier,mer,partie,pente,plan&#232;te,rivi&#232;re,roche 9
établir année,bloc,couverture,fond,marne,permien,pénéplaine,socle,surface 9
appuyer chronologie,connaissance,couverture,interstratification,pierre,pointement,étude,îlot 9
glisser bord,couche,flanc,fond,neige,pente,plaque,substratum 8
opérer arbre,mod&#232;le, pente, sol, échantillon, élément 6
poser glacier, plan&#232;ze, plaque, pupitre, socle, sol 6
tomber glacier, interfluve, plan&#232;te, région, sol, versant 6
fonder distinction, glacio-eutatisme, granulométrie, inégalité, loi, rep&#232;re 6
insister composition, fixisme, inefficacité, ouvrage, rôle, épigénie 6
localiser
sur +
det
bordure, contact, côte, emplacement, retombée, versant 6
Figure 3: 15 most productive verbal governors with the preposition sur
182
The second list (figure 4) is made up of the 15 nouns that are found most productive
(productivity>=4) as governees of the preposition sur in our corpus after verbs. The first line of the
table indicates that the noun fond (bottom) has been found after the preposition sur with 15 different
right contexts (affleurer sur le fond, ancrer sur le fond, etc.).
governors governee prod
affleurer,ancrer,arr&#234;ter,balayer,concrétionner,frotter,glisser,reposer,rouler,stratifier,transp
orter,traîner,triturer,élever,établir
fond 15
accélérer,arriver,cascader,condenser,descendre,disperser,déboucher,faire,former,glisser,i
nfluer,observer,opérer,rencontrer,trouver
pente 15
affleurer,descendre,exercer,faire,lire,localiser,paralyser,passer,remonter,situer,tomber,tra
vailler,épandre
versant 13
déposer,développer,emporter,faire,fixer,manifester,rencontrer,reposer,trouver roche 9
désintégrer,obtenir,reposer,réfléchir,voir,égaliser,épancher,établir surface 8
agir,basculer,opérer,poser,rouler,séjourner,tomber sol 7
déferler,exister,localiser,rencontrer,retrouver,régulariser côte 6
coller,exercer,plaquer,produire,rencontrer,réfléchir paroi 6
compléter,effectuer,indiquer,pouvoir,repérer,étudier terrain 6
déplacer,manquer,trouver,échelonner,étendre partie 5
effectuer,glisser,situer,trouver bord 4
affleurer,glisser,jouer,rencontrer flanc 4
aligner,épancher,étaler,étendre kilom&#232;tre 4
couler,peser,reposer,étaler lit 4
carboniser,détruire,situer,submerger
sur +
det
passage 4
Figure 4: nouns most productive as governees with the preposition sur
If we compare the two tables, we see that verbs that are most productive with the preposition sur all
instantiate one of these two cases:
- the PP headed by the preposition sur is subcategorized by the verb (reposer, renseigner,
situer, emporter, appuyer, opérer, poser, fonder, insister),
- the PP denotes a localization that is expected given the semantics of the verb, which are all
spatial verbs (rencontrer, trouver, établir, glisser, tomber, localiser).
In the second list, we see that all PPs (sur le fond, sur la pente, sur le versant) tend to behave as
autonomous phrases, conveying spatial information.
3.3 Noun complementation: N &#224; N phrases
The second illustration regards the study of simple nouns, with no morphological link to a
verb, related by the preposition &#224;. We wanted to know whether different types of &#224; N expansions would
emerge by the application of the two productivity measures. The parser extracted 50 nominal governors
and 54 nominal governees with the preposition &#224;. A subset of these results (productivity >= 3) is
presented in figures 5 and 6.
governor prepositional link governees
carte &#224; 1.10 000, 1.200 000,1.80 000
cas &#224; Java, Montserrat, Nantasket
cas &#224; + dét lahar, pays; pays-bas
craie &#224; bélemnite,micraster,silex
cr&#234;te &#224; Porolithon,cheminée,clocheton
côte &#224; falaise,fjord,plage,ria,skjär,structure
kilom&#232;tre &#224; + dét Nord,dizaine,pôle
méthode &#224; + dét potassium,strontium,uranium
roche &#224; diaclase,feldspath,feldspatho&#239;de,grain
roche &#224; + dét extérieur,minéral,soleil
région &#224; cuesta,nappe,permafrost,plateaux,saison,sous-sol
zone &#224; cristal,pergélisol,pluie
Figure 5: (noun, &#224;) productive pairs, with or without a determiner
governors prepositional link governees
ablation; plage; terrasse &#224; + dét aval
degré; maximum; éprouvette &#224; + dét dessous
Ouest; actif; haut; rigole &#224; droite
calcaire; ensemble; granit; granite; gr&#232;s; leucogranit; roche &#224; grain
183
gneiss; granit; micaschiste &#224; mica
altitude; base; forme &#224; peine
dépression; glacis; gr&#232;s; zone &#224; + dét pied
courant; glacis; plan &#224; + dét sens
affaire; ciselure; face &#224; + dét surface
concavité; côte; côté; pente &#224; + dét vent
Figure 6: (&#224;, noun) productive pairs, with or without a determiner
First, a few remarks about these two tables. The results are not as good as those found on verbs,
certainly because this second type of dependency patterns is more unstable, less recurrent than
subcategorization patterns. Both tables contain erroneous triplets, illustrating some defects of the
analysis. Some errors are due to the tagging procedure (actif, haut should have been tagged as
adjectives). Others also come from the non recognition of verbal phrases, such as &#234;tre le cas: the noun
cas cannot be considered as an autonomous governor. Despite these problems, if we observe the
(preposition, governee) pairs, we can see that the two tables exhibit rather different lexical
relationships. Productive governees appear in three structures:
- prepositional locutions (&#224; peine, au sens)
- circumstancial PPs (&#224; l'aval, au dessous, &#224; droite, au pied, &#224; la surface)
- in only 2 out of 10 cases, N &#224; N compounds (&#224; mica, &#224; grain).
As for productive governors, they are heads of N &#224; N compounds in 9 out of 14 cases. In these
cases, the prepositional expansion denotes a qualification (according to the classification made by
(Cadiot 1997)).
These first observations indicate that the productivity of the governor is an indication that we are
dealing with PPs which are cohesive with the head noun or verb. Conversely, a high productivity of
(preposition, governee) pairs is rather an indication for autonomous PPs.
4 Conclusion
This paper reports the results obtained by our parser, SYNTEX, in the task of prepositional
attachment disambiguation. The attachment strategy, which does not limit the focus to
subcategorization patterns but processes any type of prepositional dependency involving verbs, nouns
and adjectives, is based upon a combination of linguistic clues, namely: productivity of a (word,
preposition) pair, evidence about morphologically related words or about words showing similar
distributional behaviours. These first results indicate that the productivity measure, which finds echoes
in other areas of linguistic research (see for example (Bayyen 1996) in morphology), is a very reliable
criterion to assess the likelihood of a prepositional attachment and to extract lexical patterns form texts.
Further work is needed to improve recall, and particularly to find further indications to make a decision
between several potential governors when the system lacks corpus evidence.
In this paper, we have also reported our first observations concerning the use of the
productivity measure in the differenciation of arguments and non-arguments among prepositional
phrases. The opposition between cohesive and autonomous phrases, that emerges from the data we
have presented concerning both verbal and nominal patterns, must be further investigated. But these
first results certainly indicate a contrast between the PP phrases that are detected by these two
productivity measures. Our next objective is to integrate this distinction in the disambiguation strategy.
5 References
Assadi H, Bourigault D 1995 Classification d'adjectifs extraits d'un corpus pour l'aide &#224; la
modélisation des connaissances. In Actes des 3&#232;mes Journées internationales d'analyse des données
textuelles (JADT95), Rome.
Basili R, Pazienza MT, Vindigni M 1999 Adaptive Parsing and Lexical Learning, in Proceedings
of VEXTAL'99, Venise.
Bayyen RH, Renouf A 1996 Chronicling the times: productive lexical innovations in an english
newspaper. Language, 72(1): pp 69-96
Bourigault D 1994 Lexter, un logiciel d'extraction de terminologie. Application &#224; l'acquisition des
connaissances &#224; partir de textes. Unpublished PhD thesis, Ecole des Hautes Etudes en Sciences
Sociales, Paris.
Brent M 1993 From Grammar to Lexicon: Unsupervised Learning of Lexical Syntax.
Computational Linguistics, 19(2): 243-262.
184
Cadiot P 1997 Les prépositions abstraites en français. Paris, Armand Colin/Masson.
Federici S, Montemagni S, Pirrelli V, Calzolari N 1998 Analogy-based extraction of lexical
knowledge from corpora: the SPARKLE experience. In Proceedings of the first international
conference on Linguistic Resources and Evaluation, Grenada, pp 75-82.
Grefenstette G 1994 Exploration in Automatic Thesaurus Discovery. Londres, Kluwer Academic
Publishers.
Grimshaw J 1990 Argument structure. Cambridge, Massachussets, the MIT Press.
Habert B, Naulleau E, Nazarenko A 1996 Symbolic word-clustering for medium-size corpora. In
Proceedings of the 16th International Conference on Computational Linguistics, COLING,
Copenhague, pp 490-495.
Harris Z, Gottfried M, Ryckman T, Mattick Jr P, Daladier A, Harris T, Harris S 1989 The Form of
information in science, analysis of immunology sublanguage. Kluwer Academic Publisher, Dordrecht.
Hindle D, Rooth M 1991 Structural Ambiguity and Lexical Relations. In Proceedings of the 29th
meeting of the association for Computational Linguistics, ACL, Morristown, pp 229-236.
Manning D 1993 Automatic acquisition of a large subcategorization dictionary from corpora. In
Proceedings of the 31th meeting of the association for Computational Linguistics, ACL, Columbus, pp
235-242.

512
CORIS/CODIS: A corpus of written Italian based on a defined and
a dynamic model
R. Rossini Favretti, F. Tamburini and C. De Santis
CILTA - University of Bologna - Italy
{rossini,tamburini,desantis}@cilta.unibo.it
A corpus of written Italian – CORIS – has been under construction at the Centre for Theoretical and
Applied Linguistics of Bologna University (CILTA) since 1998 and will soon be completed and made
available on-line. The project aims at creating a representative and sizeable general reference corpus of
contemporary Italian designed to be easily accessible and user-friendly. CORIS contains 80 million
running words and will be updated every two years by means of a built-in monitor corpus. It consists of
a collection of authentic texts in electronic form chosen by virtue of their representativeness of written
Italian.
It is aimed at a broad spectrum of potential users, from Italian language scholars to Italian and foreign
students engaged in linguistic analysis based on authentic data and, in a wider prospective, all those
interested in intra- and/or interlinguistic analysis. Besides the defined model, a dynamic model
(CODIS) has been designed, which allows the selection of subcorpora pertinent to specific research and
also the size of every single subcorpus, in order to adapt the corpus structure to different comparative
needs. A number of tools have been developed, both for corpus access and for corpus POS tagging and
lemmatisation.

185
Going out in style?
Shall in EU legal English
Richard Foley, Researcher
University of Lapland
Rovaniemi, Finland
1. Introduction
The ambiguity of shall vis-&#224;-vis may and must and its excessive and inconsistent use, or
“traditional promiscuity” (Garner 1998:940), in legal English, have attracted the attention of
draftspersons and reformers of legal language in the United States, Canada, the United Kingdom and
Australia. The evidence brought to bear in arguments for reform in these jurisdictions typically relies
on court decisions in disputes over the meaning of the word and the intuitive perceptions of usage held
by the reformer. The present study describes research which expands the debate on shall to a new
context – the legal language of the European Union – and in applying the systematic usage-based
techniques of corpus linguistics (Biber 1996:172) to the issue introduces a comparatively innovative
methodology as well.
The problem of shall can be attributed in considerable measure to tradition. This is seen as a
reluctance to depart from tried and true formulae, which is described by Mellinkoff (1963:294) as a
fear of change: “Lurking in the dark background is the always present, rarely voiced lawyer's fear of
what will happen if he is not “precise” in the way that the law has always been “precise.” Adherence to
tokens of legalese such as shall not only sustains the myth of precision in legal language but also
perpetuates a style and language that differentiates the genre from that of other professions (Bhatia
1993:101-2) and, by extension, general usage. The tradition of precision instils in the legal profession a
prescriptivist orientation to language, exercised both consciously and subconsciously on new writers,
e.g., draftspersons and practicing lawyers. While the legal profession seeks to contain complexity in
response to and anticipation of litigation arising from ambiguity, the linguist in contrast, is - and must
be - content to describe language practice, however complex, through a principled search for
regularities in a representative corpus.
As a new legal order in which language is uniquely unfettered by tradition and, at the same
time, the object of a singularly well-resourced attempt at language engineering, the EU provides an
interesting crucible for language reform and descriptivist-prescriptivist perspectives. For example, a
semantic analysis of shall and the modals in general is essential for machine translation initiatives
(Svendsen 1991) and an understanding of the use of shall will figure prominently both stylistically and
semantically in implementing the policy of transparency, i.e., making the legal instruments of the EU
more readable for the average citizen. The paper undertakes to inform these concerns by addressing the
following questions:
1) Have the problems of ambiguity identified in the literature for shall in common-law jurisdictions
“come over” from the United Kingdom and continued in EU legal English?
2) Is there evidence of the use of promiscuous shall in EU legislation, e.g., use as a mere stylistic
marker, as has also been suggested in the literature concerning other jurisdictions? In other words, is
one of the ‘senses’ of shall null?
3) What is the frequency of shall in EU legislative language compared to that in other English-speaking
jurisdictions? What is its frequency vis-&#224;-vis general usage? To what extent does shall serve as a sign
making legal language exclusionary of the average citizen?
The paper will proceed in the next section with a discussion of the problem of shall in
English-speaking jurisdictions and a few of the solutions proposed. This serves to contrast the
perspectives of the linguist and the lawyer and provides necessary background for the analyses. The
third section describes the materials and methods used in the study. This is followed by a presentation
of the analyses and a concluding section, which discusses the findings with a view to informing future
research.
186
2. The problem of shall
The lawyer and linguist both strive to understand the meaning of shall, but all that they share
in this pursuit is the token on the page. To the lawyer, shall is a word of authority - a word conferring
rights and obligations and prohibitions - whose function is to impose an obligation (Thornton 1979:86-
7) and to do so unambiguously, in keeping with the Golden Rule of drafting:
”[T]he competent draftsman makes sure that each recurring word or term has been used consistently.
He carefully avoids using the same word or term in more than one sense....In brief, he always expresses
the same idea in the same way and always expresses different ideas differently.” Dickerson The
Fundamentals of Legal Drafting §2.3.1, at 15-16 (2d ed. 1986) cited in Garner (1998:940).
Drafting guidelines such as the above that provide the lawyer with prescribed usage; usage
deviating from this norm -- the source of the problem -- takes the form of entries in dictionaries of legal
English and court cases. For example, the ambiguity of shall is attested in the following entry in
Black's Law Dictionary, which after a paragraph substantiating obligatory shall continues:
But [shall] may be construed as merely permissive or directory (as equivalent to may), to carry out the
legislative intention in cases where no right or benefit to any one depends on its being taken in the
imperative sense. Wisdom v. Board of Sup'rs of Polk County, 236 Iowa 669, 19 N.W. 2d 602, 607, 608
(Black 1990:1375)
More extensive evidence can be had through an investigation of cases involving disputes over the
meaning of shall, a source suggested in the entry above. One such study is that by Kimble (1992), who
after reviewing over one hundred cases involving disputes over the meaning of shall concluded: “In
summary, I'm afraid that shall has lost its modal meaning - for drafters and for courts. Drafters use it
mindlessly. Courts read it any which way” (p.72).
To the linguist, shall is a modal auxiliary, a lexico-semantic category studied in great detail
(e.g., Leech (1971), Coates (1983), Perkins (1983), Palmer (1986)) to describe and account for the
variety of meanings it exhibits. With the establishment of the computer corpus linguistics paradigm
(Leech 1992:107-111), the evidence invoked in such investigations will typically be derived from a
systematic empirical investigation of a corpus or corpora, although researchers may still rely on their
own intuition regarding language usage. The dimensions of the evidence used in the two professions
are depicted below in terms of two continua: anecdotal -systematic and usage – intuition.
Intuition
Usage
Systematic Anecdotal
1
4
2
3
Figure 1 Dimensions of linguistic evidence
The entry in Black's Law Dictionary for shall could be placed in quadrant 1. It cites cases in
which courts have been called on to decide whether shall means obligation or permission - usage –but
such cases are not systematic evidence for the study of shall. Kimble's (1992) study can be placed
somewhat more to the systematic edge of the quadrant, although a collection of cases clearly cannot
represent the typical range of use. Quadrant 2 represents corpus linguistics research. Noteworthy
examples relevant in the present context are the studies of Coates (1983) and Trosborg (1997).
Quadrant 3 describes the approach of many studies of linguistic competence in the 1970s, in particular
the introspective investigations of syntax that figured prominently in the development of
transformational grammar. Quadrant 4 comprises systematic studies using native informants, which
frequently complemented introspective approach.
187
A second crucial difference between the lawyer and linguist is that they have a different
conception of meaning. To be sure, both may seek to distinguish dictionary senses of words, but
linguistic studies of modality (Coates 1983) have yielded acceptable models of meaning in terms of
clines, or gradients of senses running between core and peripheral meanings, which are doubtless at
variance with the categorical approach seen in the drafting guideline.
In the main, solutions to the problem of shall have sought to restrict the word to a single sense,
its original sense of obligation. Garner (1998:940) distinguishes eight senses of shall:
(1a) “the court. . . shall enter an order for the relief prayed for. . . .”
(1b) “Service shall be made on the parties”
(1c) “The debtor shall be brought forthwith before the court that issued the order.”
(1d) “Such time shall not be further extended except for cause shown”
(1e) “Objections to the proposed modification shall be filed and served on the debtor.”
(1f) “The sender shall have fully complied with the requirement to send notice when the sender
obtains electronic confirmation.”
(1g) “The secretary shall be reimbursed for all expenses.”
(1h) “Any person bringing a malpractice claim shall, within 15 days after the date of filing the action,
file a request for mediation.”
Only the first of these is deemed acceptable, the criterion being that the grammatical subject must be
person on whom the obligation has been imposed. Garner objects to shall in (1b) on the grounds that a
duty is being imposed on an abstract thing, e.g., service, and to (1c) because a duty is being imposed on
an unnamed actor; the agent is not specified. The lack of specification made possible by the agentless,
short passive in English is not, however, in any way, a consequence of the author's choice of shall. The
choice of passive voice is a syntactic, not lexico-semantic, consideration. In fact, agents are readily
identifiable: both verbs, ‘serve notice on’ and ‘bring before the court’ imply action on the part of a
bailiff at the request of the court. In each case a duty is being imposed and it is possible to determine
the nature of that duty, the person or persons who are to perform it and the object of that performance.
Shall tells the reader in each case that a duty is involved. The approach confounds syntactic and
semantic criteria and yields three senses where there is only one.
While the foregoing will suffice to illustrate Garner's argumentation, (1d) merits comment as
a case that is undoubtedly of the kind which prompted Bowers (1989:294) to conclude that shall is
generally “used as a kind of totem, to conjure up some flavour of the law.” The function of the verb in
(1d) form is to indicate completed action (establishment of compliance) in future time (when the sender
obtains electronic confirmation), and no interpretation of obligation is possible.
On the evidence of intuitive and empirical linguistic studies, respectively, Bowers (1989:34)
and Trosborg (1997:136) propose that shall be restricted to indicating obligation where a human agent
is specified or easily recoverable from context. This argument rests on the concept of a master speech
act (Kurzon 1986:19), according to which the enactment clause in a legislative instrument establishes a
global illoctionary force of obligation. Used in a non-agentive context in the enactment clause, shall
actually undermines the purpose of the legislation in two respects: first, as the law is always speaking,
if shall is interpreted as a future tense it creates perpetual futurity, meaning the provision will never
come into force; second, “all cases where shall is propositional and non-agentive in fact weaken the
superordinate force of the Act by suggesting that there is yet a further step to be taken before the
enacted clause becomes reality” (Bowers 1989:34).
Language engineering, in particular machine translation, also can be seen to embraces a
similar interest in isolating a single sense for shall and the modals. The following is an analysis of
modality that has been presented as Euroversal by Svendsen (1991:276).
Possible Necessary “others”
Epistemic possibility Necessity presumption
Deontic permission compulsion obligation
Subject-oriented ability Volition resolution
Table 1 Eurotra matrix for English modal meanings
188
The senses distinguished encompass the meanings to be expressed in the official languages of the EU
and presumably one form and one form only should be assigned to each for each language. Shall
certainly would be at home in “deontic obligation.” This would amount to termification of the modals
using Euroversal concepts and an agreed form – a term – in each language.
In the case of the modals, however, describing and prescribing a single sense is perhaps more
daunting a task than in the case of legal terms of art proper. Significantly, the importance of context in
determining meaning has been recognized by lawyers and linguists alike. For example, Asprey
(1992:82) states on the question of replacing shall with must: “the reason why it is difficult to replace
shall with a word that has all these subtle meanings is that shall never did it in the first place. Not on its
own. It did in context.” The problems with specifying the meaning(s) of words of authority have a
linguistic basis in that, as a closed system, their meanings are “reciprocally defining: it is less easy to
state the meaning of any individual item that to define it in relation to the rest of the system” (Quirk et
al 1972:46). Ideally, methods of investigation and evidence should accommodate both conceptions of
meaning.
Another proposed solution to the problem of shall is that it be replaced by must. This would
accomplish little semantically, as the entry in Black's indicates: “but this [mandatory] meaning of the
word is not the only one, and it is often used in a merely directory sense, and consequently is a
synonym for the word “may” Black 1990:1019). The meaning of must will clearly have to be contained
as well before it meets the needs of the legislator. Interestingly for the present study, Asprey (1992:77)
suggests replacing shall with must in the sense of obligation on stylistic grounds, asserting that the use
of shall “puts lawyers out of step with the language of the general community”. This proposal will be
taken up in conjunction with frequency analyses below.
3. Material and methods
The present study relies on four principal sources of data comprising a variety of corpora. The
first is a corpus of EU primary and secondary legislation (EULEG) compiled by the author for research
on modality in EU legal English. Table 2 gives a breakdown of the current composition of the corpus.
Text type No. of texts Word count Avg. word count
per text
Treaty 1 47102 47102
Regulations 4 39863 9966
Directives 4 55360 13840
Decisions 2 16488 8244
Total 11 158813 14437
Table 2 Composition of EULEG
The texts were obtained in electronic form from EUR-LEX the WWW-based database of
European legislation. Although the texts are not official translations, the likelihood that there are
discrepancies in the use of the modal verbs vis-&#224;-vis the authoritative version as published in the
Official Journal was considered negligible. The ready availability of text in electronic form in a
searchable database was seen as outweighing this criterion of authoritativeness.
Legislation in the EU can be divided into primary and secondary. Primary legislation
comprises the Treaties. There are three treaties establishing the European Communities and a number
of conventions and documents by which these have been amended. Complementing the Treaties are the
instruments of secondary legislation, which comprise Regulations, Directives and Decisions.
Regulations are directly applicable in that no national measures are required for a Regulation to
become binding on the citizens of a Member State and that a Member State cannot undertake measures
that would prevent the application of a Regulation. A directive is binding as to the result to be
achieved; the form and methods by which these are implemented may be decided by the authorities in
each Member State. Decisions are generally of an administrative nature and implement other
Community rules. A decision is binding in its entirety on those to whom it is addressed
The text types chosen represent the genre of primary and secondary legislation exhaustively,
whereby they should adequately represent the linguistic distribution of modal verbs. In light of Biber's
189
(1993:252) criteria for representativeness, the figures in the table suggest that a larger number of
samples, perhaps smaller in size, might be equally or more representative than those presently included.
However, at this stage in the research, the author has opted to include the entire text, even in the case of
the Treaty, for a number of reasons. First, stratified sampling without regard to the division of the texts
into recitals, enactment section and annexes would have made it impossible to investigate the
frequency of shall in the enactment section, which is crucial in light of the proposals by Bowers and
Trosborg referred to above. Second, the entire text provides the most representative sample for
intertextual comparison with corpora from other English-language jurisdictions. While the genre of
legislation is in the main similar, no consistent structure could be observed which would have justified
stratified sampling by text section across the jurisdictions.
The texts in the corpus have not been chosen entirely at random, and EULEG can be described
as opportunistic (Leech 1991:10) inasmuch as several of the texts had been gathered for teaching
purposes and terminological research. On the whole, however, the texts do not represent any particular
field of activity or time frame; a study of the frequency of modals along such parameters is beyond the
scope of the present research. The regulations span economics, social security, the directives data
protection and environmental protection and the decisions trade in bananas. Given the nature of the
linguistic feature - modal verbs - it is unlikely that any bias could be introduced by the subject matter.
There is no evidence that one field of activity imposes more obligations and prohibitions or confers
more rights than another.
The second major source of material comprises the translations of the instruments in EULEG
in Swedish, Finnish, French and German. These furnish parallel corpora which are instrumental in
disambiguating the occurrences of shall. For example, the archetypical sense of ‘obligation’ (Crystal
and Davy 1969:206-207) stands out in bold relief where one finds it translated with the Agentgen+ BE +
present passive participle construction in Finnish. French and German, in turn, serve to reveal
occurrences of shall in which futurity but not obligation is intended and in which the use of shall
instead of will in the third person is unmotivated.
The use of the translations is predicated on the assumption that translators deverbalize the
original message in the source text - whatever language this may be in - and expresses it in their native
language. This is depicted as the “language-free semantic representation” in Figure 2 below:
Clause in
source text
1) clause structure,
MOOD and lexical choices
Language-free
semantic
representation
Clause in
target text
6) Speech act
2) Propositional content
3) Thematic structure
4) Register features
5) Illocutionary force
Figure 2 The Process of Translation (adapted from Bell 1991:56)
The thesis here is that, through the agency of the translator, the text of a piece of legislation in each of
the official languages of the EU can be seen as encoding the same meaning. Due to lexical, syntactic
and semantic differences among the languages, however, these features will be expressed in different
ways. A serious qualification on the potential of deverbalization is in order, however, for translators
often rely excessively on the surface structure. Evidence for translationese, represented by the dashed
line between “Clause in source text and “clause in target text” in Figure 2, has been substantiated by
Schmied and Schäffler (1996:48) in translation corpora of English and Norwegian. Trosborg (1997:
159) acknowledges the phenomenon with regard to shall in legal language. Beeth and Fraser (1999:76)
identify institutional pressures for perpetuating inadequate translations:
One of the dangers of many of the translating tools like the Translator's Workbench (TWB) is that they
provide the translator with ready-made segments of text in the target language (lifted from earlier
documents), making it much easier to stay on the surface of a document. And yet in our hearts we know
that what was an adequate translation for the document from which the segment originated is unlikely
to be as adequate for the document we have before us now.
190
A further complication in using parallel corpora is that it is not possible to establish the source
and target text: as all are equally authentic and ‘source’ would imply an untoward political primacy.
Article 53 of the Treaty on European Union reflects this principle: “This Treaty, drawn up in a single
original in the Danish...Spanish languages, the texts in each of these languages being equally
authentic...” The organizational reality mirrors this political reality in that it is nowhere stated for a
particular piece of legislation which language it has been first drafted in and therefore which has served
as the basis for the translations found in EUR-LEX. On balance, while one must be somewhat
circumspect regarding the level of abstraction at which translators operate, the parallel texts represent a
significant resource to tap systematic usage.
The following examples from Regulation 95/48/EC illustrate the use of translations:
(2a) en whereas the name given to the European currency shall be the 'euro`; whereas the euro as the
currency of the participating Member States shall be divided into one hundred sub-units with
the name 'cent`;
(2b) fr que le nom de la monnaie européenne sera «euro»; que l'euro, qui sera la monnaie des États
membres participants, sera divisé en cent subdivisions appelées «cent»;
The corresponding Swedish text uses skall, a cognate of shall, which expresses futurity and obligation,
as well as kommer att, a pure future.
(2c) sv Namnet p&#229; den europeiska valutan skall vara "euro". Som valuta för de deltagande
medlemsstaterna kommer euron att delas upp i ett hundra underenheter med namnet cent.
The English word ‘given’ allows an agent (‘by the participating Member States'), making obligation
and thus shall possible. The French translation using the future tense suggests a strictly temporal use.
Swedish skall indicates both obligation and futurity. In the second instance, the use of the future
kommer att, coupled with the evidence from French, suggests that at least the second shall in (4a) is
superfluous and that is or will would suffice. This is a reasonable interpretation also inasmuch as there
is no recoverable agent; ‘shall be divided’ is the equivalent of ‘equals'.
The following is an example in which the use of the Finnish construction BE + Present passive
participle, which indicates obligation unambiguously, argues in favor of recovering an agent and
therefore accepting the use of shall.
(3a) en The amount shall be credited to the account of the creditor in the denomination of his account,
with any conversion being effected at the conversion rates.
(3b) fi määrä on hyvitettävä velkojan tilille hänen tilinsä valuuttayksikön määräisenä
[the-amount is to-be-credited of-the-creditor to-the-account…]
As the third question in the introduction indicates, the present study seeks to investigate the
distinctive features of the genre of legislative English and examine the characteristics of EULEG in
comparison with comparable legislation from other jurisdictions. On the intuitive level, shall certainly
seems to meet the criteria of frequency, distinctiveness and precision described by Crystal and Davy
(1991:228-9). Following is a description of the several pilot corpora or representative texts that have
been compiled to enable comparisons of EULEG with for the stylistic analysis.
Corpus No. of texts Word count Avg word count per
text
AMLEG 1 5365 5365
CANLEG 2 35213 17607
BRITLEG 1 6592 6592
FINNLEG 6 59316 9886
Tot 9 - -
Table 3 Pilot corpora
191
Clearly, great caution is called for in undertaking an analysis on the basis of a single text.
Shall is a frequent enough linear feature (Biber 1993:252), however, to sustain a tentative assumption
that even a single text may have some value as a sample. At the level of delicacy sought, they do have
something to offer. As the author is interested in translationese, a corpus of translations from Finnish
into English has also been compiled.
The fourth principal source of data comprises the frequency of the occurrence of modals in the
Brown and LOB corpora. The former comprises 1,013737 word of American, the latter 1,013,644
words of British English, and they are used here to represent the use of modals in the standard
language.
4. Analysis
In her study of shall in British statutes, Trosborg (1997) concluded that even though shall has
been defined as a modal verb expressing legal obligation, “ ...65.1% of the observed instances of shall
occurred with non-human subjects which could not be given orders or assigned obligations” (pp. 105-
106). This semantic analysis is essential in overcoming a monolithic concept of or belief in a one-toone
correspondence of form and meaning in legal language. This is reflected in Crystal and Davy's
statement: “Shall is invariably used to express what id to be the obligatory consequence of a legal
decision, and not simply as a marker of future tense, which is its main function in other varieties
(1969:206-7) and can be inferred in studies focusing on general usage, e.g., Coates (1983), which
simply cite the legal or quasi-legal meaning of shall.
An examination of the frequency of shall in the corpus by section of text is depicted in Figure
3 below.
Frequency of deontic modals by section
0
5
10
15
20
25
Recitals Enactment Annex
Frequency (per 1000 words)
shall may must
Figure 3 Frequency of deontic modals by text section
This frequency analysis of shall prompted examination of word frequencies overall in the
corpus and revealed that the relative frequency of shall in EULEG places it squarely among
semantically void function words. The relevant frequencies are summarized in the table below:
Word Frequency (per 1000 words)
Corpus Enactment
The 90 100
Of 55 58
To 29 30
In 26 26
And 22 23
shall 15 20
a 14 15
192
Table 4 Comparison of most frequent tokens in EULEG by section of text
This evidence suggests that the occurrences of shall in the corpus merit closer analysis along the lines
of Trosborg's study. It seems implausible that each occurrence of shall entails the imposing of an
obligation. To investigate this hypothesis empirically, a random sample of 574 occurrences of shall was
drawn from the enactment sections of the legislation in EULEG and analyzed interactively for the
occurrence of a human subject, and where the sentence was in the passive and had a human agent,
whether the agent was expressed in a by clause or had to be recovered from context. The percentages
obtained are presented in Table 5 below:
Sentence type
Active Passive
Subject agentless by + Agent
Human 40% 12% 3%
Non-human 39% 6 % -
Table 5 Use of shall in EULEG with human subject
If one adopts relatively strict criteria for agency, e.g., that the logical and grammatical subject
coincide and that the subject be human, 60% of the occurrences of shall analyzed are unmotivated,
indicating that legislation in the EU is only slightly more fastidious in this regard than that analyzed by
Trosborg. If the passives are included as agents recoverable from context, then some 45% of the
occurrences of shall in this data are unmotivated, an improvement over British statutes but nevertheless
a level of promiscuous use that is certainly in no way reconcilable with the Golden Rule of drafting
cited above.
Perhaps the more significant observation to emerge in the analysis is that the criterion of
human subject, which Bowers and Trosborg and, to an extent, Garner argue for, proves as difficult to
specify as discrete senses of shall. The following are examples from the sample.
(4a) It shall be for the controller to ensure that paragraph 1 is complied with
(4a) might not fit the “human subject” criterion, especially if the text were being analyzed by a
machine, but is clearly equivalent to the “controller shall ensure...”
(4b) below represents the reverse case. This main clause would presumably meet the criterion of human
subject, but the wider context “if it sees fit” renders the issue of obligation irrelevant in giving the
authority discretion.
(4b) If it sees fit, the authority shall seek the views of data subjects or their representative
Here, it is instructive to consult the parallel corpora. The Finnish version uses voi ‘can/may; Swedish
uses skall, and German and French the present tense. The Swedish may be influenced by the English
text. Obligation is not plausible.
(4c) fi Jos viranomainen katsoo aiheelliseksi, se voi hankkia [If the authority sees fit, it can/may obtain the views]
(sv skall de holt ein, fr recueille,)
The four examples that follow are also potentially problematic, and must be accounted for in any
attempt to assign shall to a particular sense. It is unclear whether Trosborg would have included any of
these as human subjects (1997:105-106). The first three were counted in the present analysis; (4g) was
not. Yet the actions involved - publishing, calculating, taking into account and creating a state in which
requirements are provided in written form – all require a human agent.
(4d) The report shall be made public
(4e) The turnover of an undertaking concerned within the meaning of Article 1 shall be calculated by adding together
(4f) The decision to publish shall take due account of the legitimate interest of undertakings
(4g) the requirements shall be in writing or in another equivalent form.
The analysis to follow addresses the third question in the introduction by examining the
frequency of shall and modal in EU legislative language compared to that in other English-speaking
jurisdictions and in general usage. The focal question is the extent to which modality in general and
193
shall in particular as the most frequently occurring modal contribute to making legal language a
distinctive genre, one perhaps exclusionary of the average citizen.
Figure 4 below depicts the frequencies of modal auxiliaries in legal and general usage:
Modals in Legal and General Usage
0
2
4
6
8
10
12
14
would will can could may should must might shall
Frequency (per 1000 words)
Modals
Legal General
.
Figure 4 Frequency of modal auxiliaries in legal and general usage
General usage is represented by frequencies from the LOB and Brown corpora as described in
the previous section. Legal usage is based on 200,000 words comprising EULEG, AMLEG, CANLEG
and BRITLEG. Spearman's rank correlation coefficient for the differences between the nine auxiliaries
was calculated and found to be significant at the 99% level.
The frequency of shall here proves to be distinctive as a stylistic feature in terms of both criteria
presented by Crystal and Davy (1969:21): that is, it occurs most frequently within the variety of
language in question and is less shared by other varieties. Indeed, a nearly clean break can be
established between the general language and legal language. In Coates’ (1983:189-9) analysis, shall in
the sense of obligation is “virtually restricted to formal legal contexts. Its fossilisation is demonstrated
by the fact that there are no examples in the informal spoken language of the Survey and only one in
the more colloquial written language of the Lancaster fiction texts.”
The following analysis compares the individual corpora to one another and each to general
usage.
Modals in Legal Language
0
2
4
6
8
10
12
14
16
18
shall may must
Frequency (per 1000 words)
EULEG AMLEG CANLEG
BRITLEG FINNLEG
Figure 5 Deontic modals in legal corpora
Chi-squared was calculated for the frequencies of modals in each corpus vis-&#224;-vis the Brown
and LOB corpora and found to be significant. Without shall, however, none of the texts or corpora
differ significantly from the general language. The analysis places EU legislation well within the
family of English-speaking jurisdictions. The high frequency of shall in the translation corpus
FINNLEG merits further investigation. A first observation is that the translators have learned their
legal English well.
194
5. Discussion
The foregoing analyses have shed light on the questions presented in the introduction. From the
standpoint of what might be termed legal hygiene, the research to date has revealed no ambiguities
which might compromise a citizen's rights. This finding suggests the need for analysis at a greater level
of delicacy or drawing more extensively on the parallel corpora for interpretations. With regard to
linguistic hygiene, or the promiscuous use of shall, the analysis revealed superfluous use on the order
of fifty percent. More significantly, the analysis revealed difficulties with the proposed solution of
restricting shall to meaning obligation when imposed on a human subject. While there is much work
yet to be done in building EULEG, this finding can be seen as an outcome of corpus linguistics abiding
interest in systematic empirical investigation. The lawyer must accept what the linguist has come to
realize: usage is complex and must be investigated accordingly. Yet, the descriptive analysis raises a
salient prescriptive issue: Might stylistic use of shall, perhaps a habit perpetuating the exclusionary
nature of the genre, introduce ambiguities of obligation into the text for the (monolingual) English
reader where, at least on the evidence of other languages, none was intended? In light of the analyses
presented here, the modals in general and shall in particular pose significant challenges to machine
translation. The need for language engineering is patent, the prospects for success somewhat less
obvious.
Descriptive analysis of the frequency of modals in EULEG and comparable texts or corpora
established that legislative language in the EU has more in common with legal genres in other English
jurisdictions than with the general language. Although hardly surprising as such, legislative language
merits further empirical study given the European Union's express interest in transparency, which is
comparable to the Plain Language Movement in the English-speaking countries. Worth noting in this
regard is that it is the frequency of shall alone, rather than that of may or must, that binds these
populations together statistically.
Obsolescent in the sense of obligation in general language, shall clearly signals to the reader
that he or she is dealing with a distinctive genre. Replacing shall with must, as mentioned earlier,
would be a problematic enterprise semantically. Even if the number of justifiable uses of a modal of
obligation were cut by half, the frequency of familiar must would rise to the point where the word
would differ distinctively from that in the general language and, in this respect, risk becoming a new
shall.
References
Asprey M 1992 Shall Must Go. Scribes Journal of Legal Writing 3(79): 79-83.
Beeth H, Fraser B 1999 The hidden life of translators. Translation and Terminology 2: 76-96
Bell R 1991 Translation and Translating. London,Longman
Bhatia V 1993 Analysing Genre: Language Use in Professional Settings. London,Longman
Biber D 1996 Investigating Language use through corpus-based analyses of association patterns.
International Journal of Corpus Linguistics, 1(2): 171-197.
Biber D 1993 Representativeness in Corpus Design. Literary and Linguistic Computing 8(4): 243-257.
Bowers F 1989 Linguistic Aspects of Legislative Expression. Vancouver, University of British
Columbia Press.
Coates J 1983 The Semantics of the Modal Auxiliaries. London & Canberra,Croom Helm.
Crystal D, Davy D 1969 Investigating English Style. London, Longman.
Crystal D 1991 “Stylistic Profiling” In Aijmer K, Altenberg B (eds) English Corpus Linguistics.
London and New York,Longman, pp. 121-238.
Garner B 1998 A Dictionary of Modern Legal Usage. Oxford,OUP
Kimble J 1992 The Many Misuses of “Shall”. Scribes Journal of Legal Writing 3(61): 65-75
Kurzon D 1986 It is Hereby Performed...Explorations in Legal Speech Acts. Amsterdam,John
Benjamin Publishing.
Leech G 1971 Meaning and the English Verb. London,Longman.
Leech G 1991 The state of the art in corpus linguistics. In Aijmer K, Altenberg B (eds) English Corpus
Linguistics. London and New York,Longman, pp. 8-29.
195
Leech G 1992 Corpora and theories of linguistic performance. In Svartvik J (ed) Directions in Corpus
Linguistics. Berlin, Mouton de Gruyter, pp. 105-122.
Mellinkoff D 1963 The Language of the Law. Boston & Toronto, Little, Brown and Company.
Palmer F 1986 Mood and Modality. Cambridge,CUP.
Perkins, M 1983 Modal Expressions in English. London,Frances Pinter.
Quirk R, Greenbaum S, Leech G, Svartvik J 1972 A Grammar of Contemporary English. London,
Longman.
Schmied J, Schäffler H 1996 Approaching translationese through parallel and translation corpora. In
Percy, C, Meyer C, and Lancashire I (eds) Synchronic corpus liguistics. Amsterdam,Rodopi, pp. 41-55.
Svendsen U 1991 On the translation of modality in an MT-System Part I. Le Langage et l'Homme
25(4): 273-280.
Thornton G 1979 Legislative Drafting. London,Butterworths
Trosborg A 1997 Rhetorical Strategies in Legal Language. Tübingen, Gunter Narr Verlag

196
Translating passives in English and Swedish: a text linguistic perspective
Anna-Lena Fredriksson
anna-lena.fredriksson@eng.gu.se
English department
Göteborg University, Sweden
1 Introduction
The aim of this work-in-progress report is to study whether the textual and grammatical structure of
passive clauses is retained or altered in translations between English and Swedish. The passive is
interesting to study in this perspective since it is a multifunctional category closely linked to textual
structure. It constitutes a useful text-structuring tool in that it makes it possible to ‘thematize a Given
element […] whilst simultaneously focalizing a New actor by shifting it to the end of the clause
[agentful passives]’ (Granger 1983:299).1 Moreover, it allows the verb to occur without an agent
(agentless passives).
The study focuses on translation pairs in which passive clauses in English and Swedish original texts
provide the starting-point. Four alternative situations are possible in the translated texts: a) no change in
syntax – no change in thematic structure (1a-b), b) change in syntax – no change in thematic structure
(2a-b), c) no change in syntax – change in thematic structure (3a-b), and d) change in syntax – change
in thematic structure (4a-b):
(1a) A wall of polythene sheeting was nailed up to screen the area from the rest of the house, and
domestic catering operations were transferred to the barbecue in the courtyard.
(1b) Ett plastsjok spikades upp för att skärma av byggplatsen fr&#229;n resten av huset, och hush&#229;llets
matlagning flyttades ut till grillen p&#229; g&#229;rden.2 (PM1)
(2a) He was instructed to be 'as conspiratorial as possible' in order to minimise the dangers of
police penetration.
(2b) Han fick instruktioner att vara "s&#229; konspiratorisk som möjligt" för att minimera riskerna för
polisinfiltration. (CAOG1)
(Lit: He got instructions to be …)
(3a) Varje &#229;r utdelas cirka 300 Nordstjärneordnar och -medaljer till förtjänta utlänningar.
(Lit: Every year, are-given around 300 orders and medals …)
(3b) Every year, around 300 orders and medals of the Northern Pole Star are given to deserving
foreigners. (GAPG1)
(4a) Till och med ensamhetskänslan försvagades även om han aldrig blev helt fri
fr&#229;n den.
(Lit: Even the-loneliness-feeling was-diminished even though he …)
(4b) Work even diminished his feelings of loneliness, although he could not make them disappear
altogether. (KF1)
Textual structure and thematic patterning in translations is an area which is relatively unexplored, but
interesting for a number of reasons. Ventola (1995:88) remarks that changes in thematic structure in
translations ‘may create somewhat different meanings. The readers are forced to focus on different
1 For text linguistic discussions of the passive, see e.g. Duskov&#224; 1971, Granger 1983, Halliday
1994:126, Péry-Woodley 1991, Sundman 1987: 360f. Reference grammars also bring up textual
functions of the passive, see Quirk et al 1985:1390f and Teleman et al 1999:4:380.
2 All examples are taken from the ESPC (see p. 2). The original text is given first as (a) and the
translation as (b). The source text passive and the corresponding expression in the translation have been
italicised. The text code referring to each corpus text, here PM1, has been given after each translation
(see the list of references). In cases where the Swedish text differs from the English, a word-for-word
gloss is provided for the part of the clause under discussion.
197
things – orientation to ‘the starting-point’ in the forthcoming text is different'. Sometimes it may even
be argued that original and translated texts ‘are not saying the same thing’ (ibid.).
Enkvist (1984:48f) draws attention to the importance of contrastive work combining text linguistics
and grammar. Such work can focus on, for example, ‘the interplay between the syntactic structure and
the information structure of the clause and sentence, that is, the way in which the syntactic structure is
brought into harmony with the desired distribution of old and new information in the sentence and the
text'. Contrastive studies in English and Swedish or Norwegian on thematic structure include for
example Altenberg (1998) and Hasselg&#229;rd (1998).
2 Definition of theme
As a point of departure for the thematic analysis I have chosen Halliday's definition of theme according
to which ‘[t]he Theme is the element which serves as the point of departure of the message; it is that
with which the clause is concerned. The remainder of the message, the part in which the Theme is
developed, is called […] the Rheme’ (Halliday 1994:37). The theme comes in first position in the
clause extending ‘up to (and including) the first element that has a function in transitivity. This element
is called the ‘topical Theme'; so we can say that the Theme of the clause consists of the topical Theme
together with anything else that comes before it’ (Halliday 1994:53).
In declarative clauses, the theme typically conflates with the subject, which is then referred to as an
unmarked theme. Marked themes are realised by adjuncts or complements. Furthermore, the
information structure often coincides with the thematic structure, so that the theme conveys given
information and the rheme new information.
The fact that the topical theme can consist of an adverbial or a prepositional phrase only (forming a
marked theme) complicates the analysis of the thematic structure of passive clauses since such
elements occur quite frequently in thematic position in the material analysed. Therefore, I have chosen
to use a somewhat altered definition and disregard the presence of such adjuncts in order to see the
passive subject as part of the theme. Referring to work by Ravelli and Matthiesen, Altenberg
(1998:116) points out that there have been suggestions within systemic linguistics to allow the theme to
include everything in the clause up to the finite verb. Such a definition might be useful for further
analysis of this type of data.
A further problem and a difference from English is that Swedish is a verb-second language. According
to the V2 constraint, Swedish normally accepts only one clause element before the finite verb in
declarative clauses, which results in inversion of the subject and the verb when the initial element is a
non-subject (cf. Teleman et al:1999:4:4ff. and Altenberg 1998). In such cases the subject becomes part
of the rheme, while it is a thematic element in English. Apart from the V2 constraint, the order of
thematic elements is generally similar in Swedish and English (cf. Teleman et al. 1999:4:25, 380, 390).
Theme-rheme patterns are important in guiding the reader through the text. Daneš has proposed a
model of three main text-structuring patterns (here taken from Fries 1995:7f). The first is called ‘linear
thematic progression'. Here, ‘the content of the Theme of a second sentence (Theme 2) derives from
the content of the previous Rheme (Rheme 1), the content of Theme 3 derives from Rheme 2, etc’
(Fries 1995:7). In the second pattern called ‘Theme iteration', ‘the same Theme enters into relation
with a number of different Rhemes. The result of this type of thematic progression is that the Themes
in the text constitute a chain of (typically) co-referential items which extends through a sequence of
sentences or clauses’ (ibid.). Thirdly, in ‘progression with derived Themes’ different themes are
derived from a hypertheme, a superordinate notion.
3 Material
The data used is derived from the combined parallel/translation corpus English-Swedish Parallel
Corpus (ESPC).3 Six non-fiction texts were analysed: three Swedish original text samples and three
English original text samples with their respective translations. Each text contains 10-15.000 words,
resulting in a total of approximately 75.000 words in source texts and about the same number of words
in translations. Although this is a fairly small sample, a total of 795 instances of passive constructions
were retrieved and analysed.
3 For a description of the corpus, see Altenberg, Aijmer & Svensson 1999.
198
It may be argued that material consisting of text samples instead of complete texts is not ideal for a
textlinguistic study (Johansson 1998:11). However, since the present study does not analyse textual
structure across whole texts, but is restricted to the thematic and information structure in the context of
the sentence containing a passive construction, the size of each text sample does not seem to be of
major importance.
The types of passives analysed are English (agentful and agentless) passives formed with the auxiliary
be and the past participle, and the three types Swedish (agentful and agentless) passives: the s-passive
formed by adding the inflection –s to the active form of the verb, and the periphrastic passive formed
with bli or vara as auxiliaries together with the past participle of the main verb, analogous with the
English be-passive.
4 Translation equivalents
In a previous study (Fredriksson forthcoming) I found that there is a wide range of possible translation
equivalents when an English passive is translated into Swedish, or a Swedish passive into English. The
results of the present study also show a wide variety of translation possibilities.
Table 1. Translation equivalents in Swedish translations (EO 67
4
ST category no %
s-passive 149 49
paraphrase
bli/vara-passive
45
44
15
14
direct active 25 8
active + man 13 4
omission 9 3
adjectival construction 9 3
nominalisation 8 3
other 3 1
Total 305 100
Frequency per 1000 words of 305 EO passives: 8.13
In translations from English into Swedish (Table 1), the great majority of English original passives are
rendered as passives in the Swedish translations; they follow what we may call a ‘passive-to-passive
principle'. If the two passive categories are grouped together, they make up as much as 63%.
Paraphrasing is an alternative rather commonly used. This group contains for example active renderings
that are not the closest possible translation of the passive verbs, but are contextually suitable choices
with similar meanings. Direct active counterparts of passive constructions are used in 8% of all
occurrences, which indicates that this is not a preferred translation strategy. A difference between this
and many of the other translation equivalents is that it entails changes in thematic structure since the
clause elements are re-arranged (agent – subject, etc). Finally, it is interesting to note that the
translation alternative called ‘active + man’ which consists of combinations of the generic man (cf.
French on, German man, English one) and the active form of the verb occurs with only 4% frequency.
This choice is often mentioned as a potential competitor to the agentless passive. Here, however, it is
only one among several active alternatives.
Table 2 shows the frequency of translation equivalents in English translations of Swedish passive
constructions. The passive-to-passive principle is strong in this direction as well. The group ‘omission’
is surprisingly large. It contains two types of omissions; those where the passive construction has been
left out in the translation, and those where the translator has chosen not to leave out the whole clause or
sentence(s).
An interesting point about the frequency of source text passives is that the Swedish passive is more
frequent than the English passive in this material. The English passive is usually regarded as a strong
category used with high frequency, especially in non-fiction genres (see e.g. Baker 1992:102, Biber
1988:250f), whereas the Swedish passive has been considered weaker. A commonly suggested reason
4 English original text is referred to as EO, Swedish original text as SO, English translations as ET, and
Swedish translations as ST.
199
is the use of the generic man combined with an active verb. In Fredriksson (forthcoming) based on
fiction texts passives are more frequent in English texts than in Swedish. The present study shows
reversed results.
Table 2. Translation equivalents in English translations (SO (7
ET category no %
be-passive
omission
paraphrase
non-finite construction
direct active
nominalisation
other
285
70
45
38
27
21
5
58
14
9
8
6
4
1
Total 491 100
Frequency per 1000 words of 491 SO passives: 13.09
5 Thematic and grammatical structure in translations
The default case would be retaining both the grammatical and textual structures of the source text in the
target text. However, the translator can choose to change either the thematic structure or the syntactic
structure, or both.
Table 3. Distribution of grammatical and thematic structure in ST.
Grammatical
structure
Thematic structure Total
+ change - change
+ passive 43 (48%) 149 (69%) 192
- passive 47 (52%) 66 (31%) 113
Total 90 (100%) 215 (100%) 305
Table 3 shows that the default case, i.e. no textual, no syntactic change (showed as -change, +passive in
the table), occurs in a majority of cases (149 instances) in translations from English into Swedish. In
31% of all cases has the translator used a non-passive structure (-passive) in order to retain the textual
structure. The total number of unchanged thematic structures is 215, whereas that of changed structures
is 90. In the column showing changed textual structure (+change), we find that the distribution of
passives and non-passives is nearly equal. We note that in 52% of the occurrences the textual change is
accompanied by a syntactic change as well. The latter would probably produce a translation that differs
from the source text to a considerable extent.
Table 4. Distribution of grammatical and thematic structure in ET.
Grammatical
structure
Thematic structure Total
+ change - change
+ passive 57 (32%) 228 (73%) 285
- passive 120 (68%) 86 (27%) 206
Total 177 (100%) 314 (100%) 491
Table 4 shows the distribution in English translations. Here too, the default case, with unchanged
thematic and syntactic structure is most frequent with 228 instances. The textual structure is preserved
but not the passive in 27% of the cases. The total number of preserved thematic structures is 314 as
compared with 177 instances of changed thematic structure. A difference from the other direction of
translation is that the distribution within the group of changed thematic structure is uneven. The
translators have changed both the thematic and syntactic structure in 68% of the total number of
changed occurrences.
200
6 Preserved thematic structure
The tendencies presented in the previous sections will now be illustrated with a few examples in order
to look into the various translation strategies in more detail. We start by looking at a default case, i.e.
one in which the grammatical as well as the thematic structure is retained in the translation (5a and b).
The English agentful passive is rendered as a Swedish s-passive, the most frequent type of Swedish
passive.
(5a) It was a life with all the cheerful violence of a cartoon film. Boys brawl with each other, get
involved in territorial gang fights, are chased down the street with razor blades by drunken
uncles, whack each other over the head with Christmas trees, have rock fights and snowball
fights.
(5b) Det var en värld som var lika uppsluppet v&#229;ldsam som en tecknad film.
Pojkar br&#229;kar, blir indragna i gängslagsm&#229;l, jagas längs gatan av fulla farbröder med rakblad,
sl&#229;r varandra i huvudet med julgranar, kastar sten och är invecklade i snöbollskrig. (RF1)
The translation neatly follows the textual structure of the original text. The theme boys/pojkar is the
theme of the whole sentence. In the model of thematic progression outlined by Daneš (see section 2),
this pattern is called theme iteration. The theme is constant but the rhemes are different.
The second most frequent group of translation equivalents in translations in both languages is called
‘paraphrase’ and contains a mixture of constructions. A large amount of them are expressions in which
the translator has used an active form of a verb that is not the closest translation of the passive verb, but
still conveys similar meaning:
(6a) Our initial excitement had turned into anti-climax as the plans became more and more dogeared
and, for one reason or another, the kitchen remained untouched. Delays had been caused
by the weather, by the plasterer going skiing, by the chief breaking his arm playing football
on a motor-bike, by the winter torpor of local suppliers.
(6b) Den första ivern hade förvandlats till antiklimax när ritningarna blev allt skrynkligare medan
köket av n&#229;gon anledning förblev orört.
Förseningarna berodde p&#229; vädret, p&#229; att stuckatören var borta och &#229;kte skidor, p&#229; att
murarbasen hade brutit armen medan han spelade fotboll p&#229; motorcykel, p&#229; leverantörernas
vintertröghet. (PM1)
(Lit: The-delays depended on the-weather, …)
A passive translation corresponding to the source text would have been possible – förseningarna hade
orsakats av vädret …- but the chosen alternative is less formal. The thematic structure with theme
iteration is retained in the translation.
(7a) Tre kockar finns redan p&#229; plats. Hovtraktören och Erich Schaumburger, hans köksmästare
även till vardags. Och en extrakock som kan behöva läras upp.
(Lit: … who might need be-taught up.)
(7b) Three cooks are already at work. The Restaurateur and Erich Shaumburger, who is also his
regular Chief Cook. Plus an extra cook, who might need the training. (GAPG1)
In (7b) the translator has used a nominalisation instead of a passive, although a passive could have
worked just as well. It is difficult to explain why this choice has been made, since both texts have the
same thematic structure. Another type of syntactic change is illustrated in (8b) where we find an
adjectival construction instead of a verb:
(8a) Till bondest&#229;ndet uts&#229;g varje härad en riksdagsman. Valbara var bara de självägande bönderna
och kronans &#229;bor. Adelns bönder var uteslutna liksom de stora obesuttna samhällsgrupperna
p&#229; landsbygden.
(Lit: The-nobility's peasants were excluded …)
(8b) For the Estate of the Peasants every rural district known as a ‘härad', or hundred, elected one
member of the Riksdag. Only peasant proprietors and crown tenants were eligible. The
nobility's peasants were ineligible as were the large tenantless social groups of the
countryside. [AA1]
201
The closest English equivalent of the Swedish verb phrase var uteslutna is were excluded/expelled.
However, the author expresses a contrast between the preceding sentence with the adjective valbara
[‘eligible'] and var uteslutna [‘were excluded']. By using an antonym of the preceding adjective -
eligible/ineligible - instead of, for example, exclude or expel the translator makes the contrast even
more marked.
7 Altered thematic structure
We will now look at a few examples in which the translators have chosen to deviate from the thematic
structure of the original text. Thus, the following examples are all non-default cases. Many of the
instances of retained syntactic structure despite thematic re-structuring are due to the fact that Swedish
is a verb-second (V2) language whereas English is not. Temporal and spatial adverbials are highly
frequent in initial position in the Swedish texts which causes frequent alterations, one of which is found
in (9a and b)
(9a) &#197;bo blev den nya maktens säte och handelsplats. Här byggdes ocks&#229; den första domkyrkan för
biskopen i Finland.
(Lit:
(9b) &#197;bo became the seat of the new power and a centre of trade. Here, too, the earliest cathedral
for the Bishop of Finland was erected. (AA1)
Here we find a marked theme, här/here and the remainder of the clause forms the rheme, which is
where we find differences between the original and the translated texts. The subject is rather long and
the differences in the rhematic structure create a different perspective for the reader. In the ET was
erected is focalised in end position and is hence the most rhematic, whereas in the SO den första
domkyrkan för biskopen i Finland/the earliest cathedral for the Bishop in Finland’ is most focalised.
This difference is due to the V2 constraint. Ventola (1995:87) discusses a parallel situation in German
and says that ‘[t]he further the subject is pushed from the first verbal element the less likely it is any
longer encoding ‘Given’ information, but instead its ability as a [sic] encoder for ‘New’ information
increases'.
(10a) The Bolsheviks saw the Civil War from the beginning as part of a great Allied plot. In reality,
the revolt of the Czechoslovak Legion had been prompted not by the Allies but by fears for its
own survival after attempts by Leon Trotsky, now Commissar for War, to disarm it.
(10b) Redan fr&#229;n början betraktade bolsjevikerna inbördeskriget som en del av en stor allierad
komplott. I själva verket var det inte de allierade som hade framkallat den tjeckiska legionens
revolt, utan dess oro för den egna överlevnaden sedan Lev Trotskij, nu folkkommissarie för
krigsärenden, försökt avväpna legionen. (CAOG1)
(Lit: In reality, was it not the allied who had prompted the Czechoslovak Legion's revolt, but
…)
Example (10b) illustrates the use of what I have called a ‘direct active', which is meant to be the closest
active alternative to the passive. The clause elements have been re-arranged in accordance with ‘the
traditional active-passive correspondence’ , i.e. the passive agent corresponds to the active subject, and
the passive subject to the active object. Consequently, the translation has a changed thematic structure;
the theme/subject connects to the rheme of the previous sentence (a great Allied plot). The pattern
called linear thematic progression (see section 2) means that an idea expressed in the rheme of the
preceding clause or sentence is picked up and forms a new theme.
(11a) Gjorde han det skulle han bli presenterad för de rätta personerna. Immanuel, som var omgiven
av ot&#229;liga fordringsägare och hotats med gäldstugan, tyckte sig ha litet att förlora p&#229; att resa.
Han ans&#229;g sig dock inte ha r&#229;d att ta med familjen.
(Lit: Immanuel, who was surrounded by impatient creditors …)
(11b) He urged Immanuel to come to Turku, where he could introduce him to the right people.
Surrounded by impatient creditors and threatened by debtor's prison, Immanuel felt he had
little to lose by going. He told his family they would follow as soon as circumstances allowed.
(KF1)
The original text contains a passive in an embedded relative clause which has been turned into a
fronted non-finite clause in the translation. The Swedish alternative appears to be relatively unmarked,
whereas the same elements in the English text are given greater prominence. In the original, the theme
202
and subject, Immanuel, is the same theme as occurs in the preceding sentence, there is hence theme
iteration, which facilitates the progression of discourse. In contrast, the re-organisation of the translated
text with a postponed subject breaks this progression.
(12a) These were not the kind of gentlemen farmers who spent their winters on the ski slopes or
yachting in the Caribbean. Holidays here were taken at home during August, eating too much,
enjoying siestas and resting up before the long days of the vendange.
(12b) Det här var inga godsägare som tillbragte vintrarna i skidbacken eller p&#229; en lustjakt i Karibiska
havet.
Semester hade man hemma i augusti: d&#229; &#229;t man för mycket, njöt l&#229;nga siestor och vilade upp
sig inför den l&#229;nga skördesäsongen. (PM1)
(Lit: Vacation had one at-home in August: …)
The combination of an active verb-form and the generic pronoun man as subject (12b) is often
described as a useful alternative to agentless passives in Swedish. This translation possibility is
however not among the most frequent in the data. The reason for employing it in this exampel might be
that the Swedish verb ta [‘take'] is unlikely to be used in the passive in collocation with semester
[‘holidays']. The thematic element is the same in both texts and provides a cohesive link with the
preceding sentence according to the principle of linear thematic progression. An idea expressed in the
rheme of the preceding clause or sentence is picked up and forms a new theme (holidays). In
accordance with the Swedish V2 constraint, the subject is placed after the verb. The addition of a
subject in non-initial position and the fronting of the complement in the translation mean that the theme
of this clause is marked whereas the one in the English source text is unmarked.
8 Conclusion
The aim of this paper has been to study how textual and grammatical structure is retained or altered in
translations. It is interesting to see that a range of syntactic alternatives are used in translations in both
directions. Some are less frequent than others, but can still indicate certain tendencies. Some syntactic
changes appear to have been made for keeping the thematic structure intact, while in other cases both
structures have been changed. However, the main tendency is that the default case with no changes is
the most common. This suggests that a variety of translation strategies are in use and further studies
will describe such strategies and in more detail.
Acknowledgements
This work was carried out with funding from the Bank of Sweden Tercentenary Foundation. I wish to
thank Professor Karin Aijmer for her valuable comments on earlier versions of this paper.
References
Primary sources
(a) Swedish corpus texts
&#197;berg A 1985 Sveriges historia i fickformat. Stockholm, Natur & kultur. Translated by Elliot G as A
concise history of Sweden. Stockholm, Natur & kultur, 1991. [AA1]
Arvidsson G, Gullers P 1991 Vad händer p&#229; slottet?. Stockholm, Gullers. Translated by Write Right as
Inside the royal palace. Stockholm, Gullers, 1991. [GAPG1]
Fant K 1991 Alfred Bernhard Nobel. Stockholm, Norstedts. Translated by Ruuth M as Alfred Nobel: a
biography. New York, Arcade Publishing, 1993. [KF1]
(b) English corpus texts
Andrew C, Gordievsky O 1990 KGB. The inside story. London, Hodder & Stoughton. Translated by
Malmsjö K as KGB inifr&#229;n. Stockholm, Albert Bonniers Förlag, 1991. [CAOG1]
Mayle P 1989 A year in Provence. London, Hamish Hamilton. Translated by Wiberg C as Ett &#229;r i
Provence. Malmö, Richter, 1991. [PM1]
Ferguson R 1991 Henry Miller: a life. London, Hutchinson. Translated by Lindgren N as Henry Miller:
ett liv. Stockholm, Wahlström & Widstrand, 1992. [RF1]
Secondary sources
Altenberg B 1998 Connectors and sentence openings in English and Swedish. In Johansson S, Oksefjell
S (eds), Corpora and cross-linguistic research: theory, method, and case studies. Amsterdam/Atlanta,
Rodopi, pp 115-139.
203
Altenberg, B., K. Aijmer & M. Svensson. 1999. The English-Swedish Parallel Corpus: Manual.
Department of English, University of Lund. (Also at http://www.englund.lu.se/research/espc.html.).
Baker M 1992 In other words: a coursebook on translation. London/New York, Routledge.
Biber D 1988 Variation across speech and writing. Cambridge, Cambridge University Press.
Duškov&#224; L 1971 On some functional and stylistic aspects of the passive voice in present-day English.
Philologica Pragensia 14(3): 117-143.
Enkvist N E 1984 Contrastive linguistics and text linguistics. In Fisiak J (ed), Contrastive linguistics:
prospects and problems. Berlin/New York/Amsterdam,
Mouton Publishers, pp 45-67.
Fredriksson A-L forthcoming A contrastive study of English and Swedish passives in a textual
perspective. In Byrman G, Levin M, Lindquist, H (eds), Korpusar i forskning och undervisning.
Fries P H 1995 A personal view of theme. In Ghadessy M (ed), Thematic development in English texts.
London/New York, Pinter, pp 1-19.
Granger S 1983 The be + past participle construction in spoken English with special emphasis on the
passive. Amsterdam, North-Holland.
Halliday M A K 1994 An introduction to functional grammar. Second edition. London, Arnold.
Hasselg&#229;rd H 1998 Thematic structure in translation between English and Norwegian. In Johansson S,
Oksefjell S (eds), Corpora and cross-linguistic research: theory, method, and case studies.
Amsterdam/Atlanta, Rodopi, pp 145-67.
Johansson S 1998 On the role of corpora in cross-linguistic research. In Johansson S, Oksefjell S (eds),
Corpora and cross-linguistic research: theory, method, and case studies. Amsterdam/Atlanta, Rodopi,
pp 3-24.
Péry-Woodley M-P 1991 French and English passives in the construction of text. Journal of French
Language Studies 1(1):55-70.
Quirk R, Greenbaum S, Leech G, Svartvik J 1985 A comprehensive grammar of the English language.
London, Longman.
Sundman M 1987 Subjektval och diates i svenskan. &#197;bo, &#197;bo Academy Press.
Teleman U, Hellberg S, Andersson E 1999 Svenska Akademiens grammatik. 1-4. Stockholm, Svenska
Akademien (Norstedts Ordbok i distr).
Ventola E 1995 Thematic development and translation. In Ghadessy M (ed), Thematic development in
English texts. London/New York, Pinter, pp 85-104.

204
Phraseological approach to automatic terminology extraction from a bilingual
aligned scientific corpus
Frérot Cécile1*, Rigou Géraldine2, Lacombe Annik3
1 University of Paris 7-Denis Diderot, 2 place Jussieu, F-75251 Paris Cedex 05,
France, frerot@cicrp.jussieu.fr
2 INRA-CRJ, Unité Centrale de Documentation, Translation and Terminology Unit,
F-78352 Jouy-en-Josas Cedex, France, rigou@jouy.inra.fr, tel: +33 (0) 1.34.65.24.54,
fax: +33 (0) 1.34.65.22.72
3 INRA-CRJ, Unité Centrale de Documentation, Translation and Terminology Unit,
F-78352 Jouy-en-Josas Cedex, France, alacombe@jouy.inra.fr, tel:
+33 (0) 1.34.65.24.55
* This work was carried out during a training period at the INRA Translation and Terminology Unit as
part of a post-graduate degree (DESS Industrie de la Langue et Traduction Spécialisée), UFR Etudes
Interculturelles de Langues Appliquées
Abstract: The linguistic knowledge represented in specialised dictionaries should not be restricted to a
collection of terms (single and complex terms). It should include phraseological units, i.e. more or less
fixed multiword expressions –often called collocations– which cause serious problems for translators
and technical writers, since translating them into the target language is rendered difficult by the
syntactic and lexical characteristics of each Language for Specific Purposes (LSP). Corpus-based
terminology acquisition tools make it easier to identify and collect these units and to generate
terminology resources that better meet the needs of users.
We present the results of an automatic terminology extraction from a French-English aligned corpus
with the aim of developing a scientific translation and writing tool for French researchers with
emphasis on the phraseological dimension. This experiment was conducted at the Translation and
Terminology Unit of the French National Institute for Agricultural Research (INRA), in collaboration
with two researchers working in automatic terminology extraction and bilingual terminology
alignment. LEXTER extracted term candidates from a French-language scientific corpus and TRINITY
identified the translation candidates in the English corpus made up of the translations of the French
texts. The translator-terminologists exploited the results of the extraction using the hypertext interface
for validation of a database.
After describing the different stages of the experiment, from preparing the corpus to data
processing, we explain how we validated and exploited the results. We focus on collocations and the
problems linked to their identification and terminographic description from a translation perspective
and consider the problems inherent to phraseology in LSP when efforts are made to improve natural
language processing (NLP) tools.
Keywords: terminology extraction, aligned corpus, phraseology, LSP, scientific translation.
205
1 Introduction
Despite considerable terminographic work carried out to collect the vocabulary of specialised
subject fields, deficiencies still exist in emerging and/or evolving subject fields. Available
terminological resources do not fully meet the needs of technical writers and translators. Resources are
often limited to terminological units, whether they are simple or complex terms, and exclude
phraseological units. However, a large number of problems are raised by the linguistic environment of
terms, i.e. the syntagmatic extension of the term to the sentence (Blais, 1993). The ability to deal with
this linguistic environment is absolutely necessary for language specialists eager to respect the type of
language used by the experts. Consequently, the study of lexical units other than nouns (e.g. verbs,
adverbs and adjectives) is of utmost importance. Nevertheless, in spite of a growing interest in
problems linked to identifying and collecting phraseology in LSP (Language for Special Purposes),
there are still few electronic or print resources available for translators.
Corpus-based terminology acquisition tools such as terminology extractors make it easier to
generate terminological resources. These tools help identify simple and complex terminological units
as well as expressions –often known as collocations– specific to a linguistic community and including
other lexical units such as verbs, adjectives and adverbs. Since we placed the emphasis on usage linked
to collocations, we took into account the definition given by Pesant and Thibault (1993) who define
collocations as being different parts of speech which appear together in a text and combine to form
expressions fixed by usage. Two examples of collocations extracted from our corpus are
technologically valuable lactic acid bacteria and radio-tagged river trout.
The present work describes the results of an automatic terminology extraction from a bilingual
aligned scientific corpus. We aimed both at evaluating how well these tools performed and at
analysing how they could be used to develop a scientific writing tool meeting the needs of French
researchers at the National Institute for Agricultural Research (INRA).
2 Terminology extraction
2.1 Context of the experiment
The experiment was conducted by the Translation and Terminology Unit in collaboration with
Didier Bourigault1 who developed LEXTER, a tool for terminology extraction, and David Hull2 who
developed TRINITY, a tool for bilingual word alignment. The experiment aimed at increasing the
terminological database currently used by the translators of the Unit. In the long term, we intend to
develop a computer-based English writing tool for the French researchers of the Institute. Nowadays,
the vast majority of publications are written in English. It is of vital importance to publish one's
results, especially in emerging subject fields where competition is fierce, and the level of English is one
of the criteria for acceptance of papers. However, whereas researchers often have a good command of
their field's terminology, they often encounter great difficulty in expressing their ideas clearly and, as a
result, in writing a syntactically coherent text. At INRA, they have the support of in-house translators
to write their papers. In order to compensate for the lack of terminological data, the translators search
on-line bibliographical databases constituting immense corpora, from which they extract the
terminology and phraseology they cannot find in dictionaries.
The experiment aimed to collect terminology from a French-English translation corpus by
validating the results of an automatic extraction. The analysis of noun phrases (NP) extracted by
LEXTER helped us identify the terms and the extensions of these NPs. We then turned to the French list
of nouns and verbs to collect verb phrases belonging mainly to general language and frequently used by
researchers.
2.2 Preparation of the corpus
This extraction was carried out from an aligned bilingual corpus made up of French to English
translations. The corpus represented a total of 340,000 words and included, in decreasing order:
research papers, press releases and publications aimed at the general public, a software user's guide,
presentation leaflets, a licence contract and summaries of monographs. The corpus comprised texts
from different subject fields of which the most represented were: agricultural sciences, soil sciences,
hydrobiology, environment, biometry and modelling, plant breeding and genetics, plant disease and
1 Didier Bourigault, Equipe de Recherche en Syntaxe et Sémantique, Université Toulouse le Mirail
2 David Hull, Xerox Europe Research Centre, Grenoble
206
weed science. Paragraph marks in both the French and English texts were used to manually align the
corpus. We also removed from the corpus any element likely to generate noise during the extraction
process (bibliographical references, mathematic symbols, figures, etc.).
2.3 Data processing
Xerox tools were used to align the corpus sentence by sentence and tag it.
2.3.1 LEXTER3
LEXTER extracted term candidates (TC) from the tagged French corpus. TCs designate words and
multiwords likely to be terminological units or collocations. The first stage in the process is a
morphological analysis of the texts followed by the identification of noun phrases of maximal length,
using linguistic rules of boundary detection. In the second stage, LEXTER uses a range of parsing rules
to extract from each noun phrase of maximal length a set of sub-phrases which are likely to constitute
terms by virtue of their position in the noun phrase (Bourigault, 1994). After statistical filtering,
LEXTER supplies a list of TCs.
2.3.2 TRINITY4
TRINITY is an alignment system using statistical word alignment techniques to automatically
construct a bilingual word and phrase lexicon from a collection of translated sentences. Statistical
word alignment algorithms use common regular co-occurrence patterns between source and target
language words to establish links between occurrences of these words in individual sentence pairs
(Hull, to be published).
2.3.3 RESULTS: HYPERTEXT INTERFACE FOR VALIDATION
LEXTER automatically generates a list of term candidates corresponding to different grammatical
categories (noun, adverb, adjective, verb) as well as adjective phrases and noun phrases. The LEXTER
hypertext interface for validation gives access to a list of term candidates either in decreasing frequency
or alphabetical order, depending on the lexical units mentioned above (see Appendix A). It is also
possible to view the contexts in which the term candidates appear. Thanks to the output of the TRINITY
system, the term candidates and the translation candidates are presented in their aligned contexts (see
Appendix B). The number of occurrences is available for each item of data. For each noun phrase, the
hypertext interface gives access to its extensions, i.e. the term candidates in which the noun phrase is
the syntactic head or expansion. The interface also makes it possible to use a validation scale which we
determined on the basis of French and English word segmentation as well as terminology and
translation relevance. Lastly, French term candidates and their English equivalents can be entered into
a lexicon if they are considered relevant.
3 Analysis of the results
3.1 Noun phrase analysis
3.1.1 CREATION OF TERMINOLOGICAL ENTRIES
The structure of the lexicon supplied by the hypertext interface made it possible to enter only
French term candidates and their English equivalents together with comments. Since we aimed to
collect additional information useful to both translators and researchers, we created a terminological
entry that would best meet their needs and based its structure on the existing entry of the translators’
database. Therefore, we kept some of the fields (namely: Term, Variant –several, Context, Subject
field, Descriptor, Linguistic note) and added two collocation fields (Noun collocation and Verb
collocation). We removed the Definition field (see Appendix C). The aim was not to formulate
terminological definitions but to collect defining statements in order to understand the terms. However,
it should be noted that identifying defining statements proved quite unproductive. The reason for this
unproductivity lies in the fact that the main type of publications contained in the corpus was the
research paper: research papers relate to the expert-expert communicative setting and concepts are
rarely defined since experts are assumed to have the same or very similar level of expertise.
3 For further details, see BOURIGAULT D 1994
4 For further details, see HULL D (to be published)
207
3.1.2 LEVELS OF ANALYSIS
We selected noun phrases according to three approaches: terminology, translation and scientific
writing. The specific needs of translators and researchers were also considered.
Translators must use the appropriate terminology as well as the linguistic expressions specific to a
subject field which reflect the language as used by researchers. This is rendered difficult by the
diversity of INRA's subject fields. The selection of noun phrases is influenced by the linguistic skills
and scientific expertise of the translator. This usually depends on the translator's experience (namely
beginner, experienced, sub-contractor). Since translators need to understand the concept referred to by
the author, we collected defining statements whenever possible.
As regards researchers, they are more concerned with phraseology than terminology. Two main
reasons explain why they often have great difficulty in clearly expressing their ideas in English: their
command of the target language can be quite limited and, secondly, they tend to apply French syntax,
hence the likelihood of loan translations. Their knowledge of LSP phraseology is often very poor even
when it comes to general language expressions such as to put forward hypotheses, to conduct research,
to take into account, hence the focus on collecting verb phrases belonging to LGP (see 3.2). Generally
speaking, researchers encounter difficulty when combining specialised terms with words from general
language. From a theoretical point of view, this raises the issue of "the separation of terms on the one
hand and the linking elements from LGP on the other [which] may not be maintained uncritically"
(Picht, 1987).
After identifying these needs, we selected noun phrases on the basis of the issues raised by:
- translators: are these noun phrases or collocations specialised enough to be selected? Is this
collocation an original feature linked to the author's style or a rather set expression worth
being collected?
- terminologists: does this noun phrase designate a specific concept?
- researchers: which words are used to describe the steps of an experiment? Which is the
appropriate verb used to describe results, a comparison or cause and effect? Which
expressions would a native English speaker use? Is there a specific expression used to convey
a given idea? What is the English equivalent of that specific French verb?
We analysed each noun phrase as follows: the term candidates and the translation candidates were
analysed, as well as head and expansion productivity (see Appendix D), and the French and English
contexts were read through to collect defining statements and collocations LEXTER would not have
extracted.
3.1.3 IDENTIFICATION AND RECORDING OF COLLOCATIONS
The analysis of collocations has pointed out the lexical and syntactic differences between the two
languages, as well as their unpredictability (Heid and Freibott, 1991) when translating from French into
English. We bore these linguistic facts in mind when selecting collocations, which we define as usual
associations of several words linked by prepositions and referring to one or several notions. We
focused on prepositions since translating the phrases they are part of is a major problem for researchers.
This definition excludes complex terms with patterns such as noun + noun or adjective + noun.
The main difficulty consists in differentiating a term from a collocation when analysing a sequence
of lexical units. The difference involves a change from a conceptual organisation to a linguistic
environment. Consequently, collocations are subjected to greater variation than terms, which is also
related to the author's own style, to the creative power of phraseology (Blampain, 1993). It is difficult
to determine whether experts in a given subject field often use a collocation or whether it is pure
stylistic "extravagance". Frequency was not considered to be a criterion of selection, as most of the
corpus belongs to a very specialised discourse, the terminology and phraseology of which are not
widespread (Béjoint and Thoiron, 1992). Moreover, the corpus is of limited size and is therefore not
representative of scientific discourse.
To record collocations, it is necessary to consider the intellectual processes involved in translating
and writing. Translators and researchers probably try to answer the following question: What words or
verbs usually combine to describe a given process, to show results? (Cohen, 1992) rather than What
words or verbs usually combine? Consequently, concept is crucial and provides some answers as to
whether users should have to access one collocation referring to several notions together or each of
these notions separately.
208
In the following collocation: addition de lipides &#224; la ration de la truie en lactation, the various
notions referred to mean that three different entries need to be created (addition de lipides / ration /
truie en lactation), but the unpredictability of the collocation led us to enter addition de lipides &#224; la
ration de la truie en lactation into the Noun collocation field of the three terminological entries. As it
is more relevant and reliable to search a term rather than a collocation when querying a database, a
collocation will be entered under the keyword (the base of the collocation) as well as the co-occurrent.
Example: Noun collocation: dry matter content under the entries content and dry matter.
To record all the collocations related to a term in a single field would make the content of the entry
difficult for users to read and access to the information would be delayed. Therefore, we decided to
create two collocation fields to improve the way in which noun collocations and verb collocations are
identified.
3.2 Verb analysis
The extraction was performed from the French corpus and concerned simple verbs. TRINITY in this
case did not align the translation candidates in English, which involved us:
- reading the French contexts to analyse the linguistic environment of the verb and identify
complex verb phrases,
- reading the English contexts to identify the equivalents.
3.2.1 CREATION OF VERB ENTRIES
The structure of the verb entry differs from that of the NP terminological entry. We integrated the
following fields: French verb, English equivalent, Linguistic note but removed the Definition, Subject
field and Descriptor fields. In fact, we focused on the information relating to the ways in which these
verbs function and are used in context. We created a dozen fields to record linguistic combinations in
which the verb appears (see Appendix E). These combinations are more or less fixed and constitute
sentence segments that are interesting from a translation point of view.
In order to facilitate access to information, we created an entry for each verb phrase containing a
support verb such as mettre which produces approximately 20 complex verb structures.
3.2.2 SELECTION OF VERBS
From the list of "verb candidates", we selected those frequently used in original research papers due
to their predominance among the documents translated or reviewed by the Translation Unit. An
original research paper displays an IMRaD structure composed of the following sections:
Introduction
Materials and methods (experimental protocol)
Results
(and)
Discussion
We first excluded specialised verbs referring to a specific notion in a subject field for several
reasons: these verbs only create minor difficulties in translation, there is very little morphological or
syntactic variation between both languages and researchers usually know the English equivalents of
specialised verbs, as they do for specialised terms. This hypothesis was confirmed by analysing the list
of specialised French verbs, which was relatively limited, and their English equivalents, as shown by
the following examples: cloner (to clone), coder (to code).
We selected verbs belonging to general language which create difficulties for researchers when
translating them, as well as support verbs such as mettre which produce approximately 20 complex
verb structures (mettre en comparaison, mettre en évidence…). Here are some of the verbs we
analysed: montrer, constituer, réaliser, conduire, mener, estimer, permettre, varier, sembler,
apparaître, entraîner, rechercher.
3.2.3 TERMINOGRAPHIC DESCRIPTION
The verb entry should not be limited to a lexicon, i.e. a verb and its equivalent(s) in the target
language (L'Homme, 1993). Problems in translating complex verbs are related to the syntactic
structures specific to French and English. Users therefore need more information than is usually
209
provided for a simple or complex term. It is necessary to record how the verb functions from a
linguistic point of view and to:
- describe how the verb functions syntactically (What kind of complement can be used
alongside this verb? What preposition is required with this verb? Is it a transitive or
intransitive verb?),
- provide semantic information by proposing a synonym to specify the meaning of the French
verb (the French synonym is usually the most similar morphologically to the English
equivalent),
- give an example of the verb in context, as it occurs in a sentence.
Interestingly, the English equivalents of the complex verb structures having a support verb as their
nucleus, such as mettre, are often simple verbs (mettre en évidence: to demonstrate), and the French
verb has several equivalents in English (entraîner: to cause, to lead to, to result in).
3.2.4 PROSPECTS FOR VERB ANALYSIS
The analysis of NPs had evidenced errors in the extraction performed by LEXTER. For example, the
NP list consists of a large number of sequences for which compte de is a head ([tenir] compte de
l'accumulation préférentielle d'assimilats, [rendre] compte de l'orientation des particules, [prendre] en
compte des contraintes techniques). These sequences, had they been well identified, would have
allowed verb phrases such as rendre compte de, tenir compte de, prendre en compte to be extracted.
Therefore, from the list of nouns extracted by LEXTER, we selected those presumably having high cooccurrence
productivity with verbs.
Examples: hypoth&#232;se: admettre une ~, émettre une ~, faire l'~ que, rejeter une ~, etc.
résultats: commenter des ~, discuter des ~, obtenir des ~, diffuser des ~, etc.
g&#232;nes: identifier des ~, localiser des ~, introgresser des ~, porter des ~, etc.
Of the nouns with high co-occurrence productivity with verbs, it appears that some belong to
general language and are combined with verbs from general language (admettre une hypoth&#232;se), while
some are terms (g&#232;ne) and are combined either with verbs from general language (identifier des g&#232;nes)
or with specialised verbs (introgresser des g&#232;nes). These examples illustrate the fact that general
language intermingles with LSP in scientific documents and raises the issue of the supposed dichotomy
between LGP and LSP, and the widely acknowledged point of view which considers as a boundary the
limit between highly specialised jargon and words from general language (Bourigault and Jacquemin,
2000).
On the basis of the list of nouns extracted by LEXTER, we will further analyse complex verb
structures starting with nouns from general language, since this type of structure raises major problems
for INRA researchers when writing their papers. We will continue with verb structures including
terms.
Recording these different types of verb structures raises the problem of how they can be accessed.
For the time being, the structures we have analysed do not allow us to define the best approach to
recording them: it is still unclear whether access should be given to the whole verb structure or to the
noun and verb separately depending on what information we will provide. According to the
information available in the corpus, we could provide two kinds of entries: a specific term with the
various verbs it can be combined with, or a verb and the kinds of complements with which it can be
used (L'Homme, 1997). This will probably need to be discussed with future users as researchers and
translators may not go about searching databases in the same way.
4 Conclusion and prospects
In the light of the corpus size and heterogeneity, the quality of the extraction, in French as well as in
English, is irrefutable. Terminology extractors are useful to translators considering the very little time
they have to enter terms into a terminological database. Moreover, this kind of extraction allows
experienced translators to record NPs they would not have entered otherwise, judging them as being
too basic. As for verbs, using LEXTER enables verb phrases to be identified rapidly within a context. It
would be interesting to perform a similar automatic extraction with English verbs.
210
We will continue exploiting the results of the extraction and, when the database has a sufficient
number of entries, we will carry out a test in the field with researchers so as to make sure it meets their
needs.
The work carried out on verbs could be improved to better describe verbs in scientific discourse
which, in turn, would make it possible to determine appropriate conceptual classes of co-occurrents and
classify verbs according to their usage. A more detailed analysis of verbs in LSP could contribute to
improving corpus-based terminology acquisition tools and, more generally, to improving NLP.
Once the users’ needs are taken into consideration and the analysis is no longer restricted to noun
terms, other parts of speech (verb, adverb, adjective) emerge: it is then necessary to adopt a
phraseological approach.
5 References
Bejoint H, Thoiron P 1992 Macrostructure et microstructure dans un dictionnaire de collocations en
langue de spécialité. Terminologie et traduction 2/3: 513-522.
Blais E 1993 Le phraséologisme. Une hypoth&#232;se de travail. Terminologies nouvelles 10: 50-56.
Blampain D 1993 Notions et phraséologie. Une nouvelle alliance ? Terminologies nouvelles 10: 43-49.
Bourigault D 1994 Lexter, un logiciel d'extraction de terminologie. Application &#224; l'acquisition des
connaissances &#224; partir de textes. Th&#232;se en informatique linguistique, Ecole des Hautes Etudes en
Sciences Sociales, Paris.
Bourigault D, Jacquemin C 2000 Construction de ressources terminologiques. In J-M. Pierrel,
Ingénierie des langues. Paris, Herm&#232;s Sciences Publications, pp 215-233.
Cohen B 1992 Méthodes de repérage et de classement des cooccurrents lexicaux. Terminologie et
traduction 2/3: 505-511.
Heid U, Freibott G 1991 Collocations dans une base de données terminologique et lexicale. Meta
36(1): 77-91.
Hull D (unpublished) Software tools to support the construction of bilingual terminology lexicons. In
Bourigault D, Jacquemin C, L'Homme M-C (eds). Recent advances in Computational Terminology.
London, John Benjamins.
L'homme M-C 1993 Le verbe en terminologie : du concept au contexte. L'actualité terminologique
26(2) 17-19.
L'homme M-C 1997 Méthode d'acc&#232;s informatisé aux combinaisons lexicales en langue technique.
Meta 42(1): 15-23.
Pesant G, Thibault E 1993 Terminologie et cooccurrence dans la langue du droit. Terminologies
nouvelles 10: 23-35.
Picht H 1987 Terms and their LSP environment – LSP phraseology. Meta 32(2) 149-155.
211
APPENDIX A: Hypertext interface for validation: TCs (NPS) by decreasing frequency
APPENDIX B: TCs and translation candidates in aligned contexts
212
APPENDIX C: Terminological entry
APPENDIX D: Head and expansion productivity
213
APPENDIX E: Verb entry

214
The METER corpus: A corpus for analysing journalistic text reuse
Robert Gaizauskas†, Jonathan Foster‡, Yorick Wilks†, John Arundel‡, Paul Clough†,
Scott Piao†
Departments of Computer Science† and Journalism‡
University of Sheffield, Sheffield, S1 4DP
(contact: R.Gaizauskas@dcs.shef.ac.uk; fax:(0114) 222 1810)
Abstract
As a part of the METER (MEasuring TExt Reuse) project we have built a new type of comparable
corpus consisting of annotated examples of related newspaper texts. Texts in the corpus were manually
collected from two main sources: the British Press Association (PA) and nine British national
newspapers that subscribe to the PA newswire service. In addition to being structured to support
efficient search for related PA and newspaper texts, the corpus is annotated at two levels. First, each of
the newspaper texts is assigned one of three coarse, global classifications indicating its derivation
relation to the PA: wholly derived, partially derived or non-derived. Second, about 400 wholly or
partially derived newspaper articles are annotated down to the lexical level, indicating for each
phrase, or even individual word, whether it appears verbatim, rewritten or as new material. We
envisage that this corpus will be of use for a variety of studies, including detection and measurement of
text reuse, analysis of paraphrase and journalistic styles, and information extraction/retrieval. To
illustrate these potential uses we briefly describe some work we have done with the corpus to develop
algorithms for detecting text reuse.
1. Introduction
The aim of the METER (MEasuring TExt Reuse) project1 is to investigate how text is reused in the
production of newspaper articles from newswire sources and to determine whether algorithms can be
discovered to detect and quantify such reuse automatically. It is to be hoped that results will generalise
beyond the newspaper-newswire scenario and provide broader insights into the nature of text derivation
and paraphrase; but the newspaper-newswire scenario provides an ideal initial case study, and one with
considerable potential practical application – see below.
To assist in this study it was necessary to create a comparable corpus2 consisting of a selection of
newswire texts and newspaper articles reporting the same stories, in some cases derived from the
newswire texts and in some cases not. Because the Press Association, the major British domestic
newswire service, is a collaborator in the METER project and have provided us with unrestricted
access to their newswire service, we have used their archive as the source newswire for our corpus and
texts from a variety of their subscribers in the British press as the candidate derived texts. Having
assembled the corpus and annotated it to assist in our study of text reuse, we believed the corpus would
be of wider interest to the corpus linguistics, natural language processing and language engineering
communities, and hence decided to package and release the corpus on its own. This paper describes the
design, structure and contents of the corpus, and illustrates its potential by briefly describing some
experiments we have carried out using it. The METER Corpus is available free-of-charge for research
purposes.
It should be stressed that the METER corpus is a pioneering corpus for the study of text reuse and that
as such is no doubt flawed and limited in various ways. Resource limitations have meant limiting the
size of the corpus and the amount of interannotator verification carried out on the annotations. Ideas as
to how it should be annotated continued to evolve during the process of annotation, which means that
complete consistency across annotations has probably not been achieved. Our hope is that despite these
1 For further details of the METER project, see: http://www.dcs.shef.ac.uk/nlp/funded/meter.html.
2 Johansson et al. (1996: 3) define comparable corpus as: “corpora consisting of parallel original and translated
texts in the same languages”.
215
limitations the corpus will still prove useful to others, even if only as a starting point for designing a
better resource.
2. Text Reuse in the British Press
The Press Agency (PA) is the national news agency for the UK and Ireland. It provides regional,
national and international news 24 hours a day, 365 days a year to its media customers throughout
Britain. On a daily basis, the PA sources 1,500 news, sport and feature stories and one hundred news
and sport photographs to the newspaper industry. The PA also supplies listings information in, for
example, finance, arts and entertainment and television. In addition, the PA supplies text to specific
news and sport internet websites, as well as providing news copy and information for other leading
commercial and public sector organisations. Those using its services include national and international
newspapers, regional morning and evening papers and terrestrial radio and television broadcasters.
Thousands of weeklies, periodicals and magazines also receive various types of PA output.
Through its services, the PA quite clearly performs a critical function for the British media, whether
they operate in print, audio or electronic forms. Because of its ongoing supply of both domestic and
national news, it is clear that the PA has a critical role in setting the news agenda through its capacity to
distribute salient information rapidly to news organisations. The PA, which has been operating since
the nineteenth century, is in a unique position in British media industry and is widely regarded as a
most credible, authoritative and trustworthy journalistic source for the newspaper and broadcast
enterprises.
Being a primary supplier, the news issued by the PA is widely reused, either directly or indirectly, in
British newspapers. Even if not directly using the PA copy, journalists invariably refer to the agency's
newswire service during report production. This ensures verification of the ’facts’ in a story and also
facilitates effective ’copy tasting’ decisions (copy tasting is the process of assessing which of all the
news available at a given time should be included in the current newspaper edition). Therefore, rich
“real” examples of text reuse can be expected from a collection of PA copy and related newspaper
articles.
The study of text reuse has, aside from its intrinsic academic interest, a number of potential
applications. Like most newswire agencies, the PA does not monitor the uptake or dissemination of
copy they release because tools, technologies, and even the appropriate conceptual framework for
measuring reuse are unavailable. For the PA, potential applications of accurately measuring reuse of
their text include: 1) monitoring of source take-up to identify unused or little used stories; 2)
identifying the most reused stories within the British media; 3) determining customer dependencies on
PA copy, and 4) new methods for charging customers based upon the amount of copy reused. This
could create a fairer and more competitive pricing policy for the PA3.
3. Construction of the METER corpus
The texts of the METER corpus were collected manually from the PA online service and the paper
editions of nine British newspapers - The Sun, Daily Mirror, Daily Star, Daily Mail, Daily Express,
The Times, The Daily Telegraph, The Guardian and The Independent4.
Building a general newspaper corpus was beyond the time and resource limitations of the METER
project, so we have limited the corpus to just two domains: British law court reporting and show
business stories. Court stories were chosen because of the substantial amount of data available in both
newspapers and PA and because of their regular recurrence in British news. Court stories also revolve
around “facts” such as the name of the accused, the charge and the location of a trial or inquest, with
limited scope for journalistic interpretation. This information is found in both newspaper and PA
reports, even when newspapers do not use PA as a source. Courts generally sit from Monday to Friday;
therefore PA copy was collected for these days and newspaper versions of the same story appearing on
the succeeding day were also chosen. Court cases reported by newspapers for which PA did not
3For a more comprehensive summary of the PA and public access to a portion of their newswire, see the PA
website: http://www.ananova.com.
4 The first five of these papers are published in tabloid format and are viewed as the “popular” press; the latter four
are referred to as “broadsheets” and viewed as the “quality” press.
216
produce copy were ignored. Stories were collected for cases that lasted just one day, as well as for
cases that stretched over much longer periods.
The other domain included in the corpus is show business and entertainment news. This was chosen to
contrast with the court domain. Show business press exhibits a more expansive style, with greater
freedom of journalistic expression and interpretation. Show business stories tend to be reported in a
more frivolous, light-hearted manner. Like court reporting, show business news is also a stable and
recurring feature of contemporary British news. The show business news stories, however, form a
secondary collection to that of court stories and less data from this domain is included in the corpus.
Just as practical restrictions on corpus construction limited the scope of news domains included in the
corpus, so too did they limit the temporal extent of the material included. Too narrow a date range
might have lead to a biased sample, but too wide a range would lead to too much material. In the end
we settled on gathering material from a one year period. The text collection spans 24 days for the law
court reporting and 13 days for show business stories from 12 July 1999 to 21 June 2000.
PA stories are classified under a number of discrete, identifiable news categories, such as Courts,
Showbiz (show business), Politics, Education, etc. Under each of these categories, stories are further
classified into sub-categories, such as Courts (Axe), Courts (Strangle) and Courts (Gamekeeper). Each
of the sub-categories refers to an individual story, incident or event. Such a sub-category is called a
catchline. For each catchline, the PA follows its development and keeps releasing updated reports
throughout a day. Each report occupies a single web page – termed a PA page. Therefore, a catchline
contains one or more PA pages.
On each day in the study, all PA catchlines and associated pages relating to courts or show business
were identified through the PA categories, Courts and Showbiz. The PA pages under a selected
catchline were downloaded into separate electronic files. For each selected PA catchline, the final
southern editions of the nine British national newspapers from the next day were then examined. The
newspaper reports about law and court stories were compared against the PA catchlines and all those
that the PA had covered the day before identified. These newspaper articles were then manually
scanned into separate electronic files (and later manually examined and spell-corrected). Original paper
copies were used in constructing the METER Corpus because web-published versions of the stories
were neither reliably available nor, even if available, reliably identical to the published paper copy,
which is still viewed as the definitive form of publication for a newspaper.
Table 1 gives general statistical information about the METER corpus. As discussed, the corpus
consists of two main parts, law and court reports and show business reports. These contain 1,430 texts
(458,992 words) and 287 texts (76,158 words) respectively – here we use the term text to refer either to
a PA page or a newspaper article. In terms of the PA-sourced and newspaper-sourced texts, 773 PA
pages (239,679 words) versus 944 newspaper articles (295,471 words) were included. Of the PAsourced
texts, 611 courts texts and 112 show business texts are associated with 205 Court catchlines
and 60 Showbiz catchlines respectively.
Source Domain Total
Law and Court Show Business
Words Texts Words Texts Words Texts
WD PD ND WD PD ND
PA 206,354 661 (205 catchlines) 33,325 112 (60 catchlines) 239,679 773 (265
catchlines)
Other 1,269 0 3 2 0 0 0 0 1,269 5
Times 34,794 24 41 46 2,966 5 7 2 37,760 125
Star 14,021 15 28 27 7,590 10 19 7 21,611 106
Express 21,956 17 27 18 5,270 1 8 5 27,226 76
Mirror 17,359 22 32 28 4,211 7 11 6 21,570 106
Mail 31,686 21 29 7 6,414 0 7 7 38,100 71
Guardian 38,499 12 46 37 3,805 4 6 3 42,304 108
Telegraph 45,768 30 62 35 2,985 6 7 2 48,753 142
Sun 18,597 18 37 15 6,010 4 24 7 24,607 105
Independent 28,689 7 37 46 3,582 2 7 1 32,271 100
Total 458,992 1,430 76,158 287 535,150 1,717
Table 1: Statistics of the METER corpus
217
The primary aim of constructing the corpus was to provide a resource for studying reuse of text
between the PA and various subscribing newspapers, and not to provide a resource to study how the
same story is handled differentially across the British press. Nevertheless, there are a significant
number of stories in the corpus which are carried across multiple newspapers. Figure 1 illustrates the
distribution of shared catchlines across newspapers. Most catchlines have associated with them only a
single newspaper article; but more than ten in the courts domain are to be found in all nine of the
newspapers represented in the corpus.
Figure 1: Distribution of shared catchlines across newspapers
The total effort expended in creating the corpus was about two person-years. Gathering stories,
converting to electronic form and manual annotation took about one person-year. Subsequent OCR
error checking and correction, organising the electronic texts into a structure, designing the mark-up
scheme and electronic annotation took approximately another person-year.
4. Structure of the METER corpus
An important issue in corpus construction is deciding upon an appropriate structure in which to store
the data, so as to facilitate human and machine access to it. In the current METER corpus, the texts (PA
pages plus newspaper articles) are arranged in a tree structure. Texts are clustered according to their
origin, topic and date of release. This structure provides a unique identifier for each text within the
corpus. Figure 2 illustrates the overall structure of the METER corpus (lower level details are shown
only for the Courts domain – the Showbiz part of the corpus is structured similarly).
0
10
20
30
40
50
60
70
1 2 3 4 5 6 7 8 9 10
Number of newspapers reporting the same catchline
Courts dom ain
Showbiz domain
Lowest
level of
alignment
..
21.06/00
..
Showbiz Courts
Catch
line N
Catch
line 1 ..
annotated
Catch
line N
Catch
line 1 ..
meter
corpus
.. Page 1 Page
N
News
paper
1
News
paper
N
news
papers
.. annotated rawtext
21.06.00 12.07.99 ..
Courts
rawtext
12.07.99
PA
Showbiz
218
Figure 2: Topology of the METER corpus
As shown in Figure 2, the PA and newspaper stories are stored in a six-level tree structure. At the
topmost level, the corpus is divided according to the source of materials, PA texts under one branch
and related newspaper texts under the other. At the second level, both branches split into two according
to the domain of the materials, Courts stories under one branch and Showbiz stories under the other. At
the third level, all branches bifurcate again, into raw and annotated sub-branches. The raw sub-branch
contains exact copies of all texts as downloaded from the PA website or scanned in and OCR errorcorrected
from the newspapers; the annotated sub-branch contains marked-up versions of the texts as
described in more detail in section 5 below.
The fourth level of the tree classifies the texts by the date of release. All texts issued on the same day,
both in the PA and newspaper divisions, are grouped together under the same directory, which takes the
date as its name, in the form day.month.year, e.g. 12.04.99. Note, however, that the newspaper texts in
the METER corpus are released one day later than the corresponding PA texts. However, for the sake
of retaining a parallel data structure between PA and newspapers, the PA dates are used as directories
for both PA and newspaper sections. Thus newspaper texts are actually released a day later than their
directories indicate.
At the fifth level, i.e. within date, in both the PA and newspaper divisions the texts are grouped into
catchlines. However, it should be noted that the texts under the same catchline are related in different
ways within the PA and newspaper halves. In the PA division, all the PA pages reporting a given story
on the same date are stored under the catchline under which the PA released them. In the newspaper
division, all the newspaper versions of a given story are stored beneath the PA catchline for that story,
for example, the Times', Sun's and Telegraph's versions of reports about a killing.
The sixth and final level of the corpus contains the leaf nodes of the tree structure, the actual text files
which make up the corpus content. Each text is given a filename that conveys basic information about
it. The general naming convention for PA texts is: catchline+PA-page-number+{story-type}.txt (“+”
here indicates concatenation, “{}” indicate optionality). catchline and PA page have been previously
explained. story-type indicates whether a PA text is a “non-standard” press release, such as nightlead,
snap, sub, etc. For example, the filename sergeant1snap.txt refers to a text which is a single sentence
giving urgent, breaking news about an incident or event involving a sergeant. Table 2 lists PA text
types included in the METER corpus.
On the other hand, the filenames given to newspaper texts consist of three parts:
catchline+filecode_newspaper-name.txt. The second component filecode is a unique number assigned
to each newspaper article for identification. For example, a newspaper filename
Example filename Code File type description
meter_corpus/PA/courts
/21.06.00/sergeant/ sergeant1.txt
A page of a story released by the PA
(number indicates page).
meter_corpus/PA/courts
/21.06.00/sergeant/ sergeant1lead.txt
lead A page of PA copy that summarises the
major aspects of a story.
meter_corpus/PA/courts
/21.06.00/sergeant/sergeant1nl.txt,
meter_corpus/PA/courts
/22.11.99/inqheart/inqheart1ld.txt
nl or ld As lead - compiled in the afternoon or
evening. Especially useful for the next
day's daily newspapers.
meter_corpus/PA/courts
/21.06.00/sergeant/ sergeant1snap.txt
snap A single sentence giving urgent,
breaking news.
meter_corpus/PA/courts
/21.06.00/sergeant/ sergeant1sub.txt
sub A page of copy that develops previous
material sent by the PA.
meter_corpus/PA/courts
/07.10.99/wife/wifecorr.txt
corr Amendment to earlier copy provided.
meter_corpus/PA/courts
/16.07.99/care/care1nlcorr.txt
nlcorr Nightlead correction.
meter_corpus/PA/showbiz
/14.12.99/mccartney/mccartney1ff.txt
ff Fact file - A series of bullet points to
accompany an existing story.
Table 2: Different PA file types
219
thomas353_telegraph.txt refers to a newspaper article from the Telegraph newspaper under the
catchline of thomas with the file code 353.
The parallel structure of the PA and newspaper divisions in the METER corpus facilitates manual and
automatic search for related PA and newspaper texts. Furthermore, the indicative directory and file
names make it possible to retrieve basic information about text(s) automatically, e.g. data source, text
domain, date of release and topic, etc. This information may be obtained directly from the corpus
structure and text filenames, without the requirement to look inside any of the texts either at text
content or embedded markup. Such information provides one straightforward route to exploit the
corpus; however, all this information, and more, is also available from metadata embedded in the
annotated portion of the corpus.
5. Annotation of the METER corpus
Leech (1997: 4) suggests that “corpora are useful only if we can extract knowledge or information from
them”. While raw texts contain useful data, except basic information such as word frequency and
collocations, it is difficult to extract more complex information automatically from them. Such
information needs to be encoded explicitly in the corpus.
In order to increase the utility of the METER corpus, we have encoded information pertaining to text
reuse in the newspaper section of the corpus. Each of the newspaper articles has been manually
classified into one of three general categories, indicating degree of derivation from the PA. In addition,
approximately 400 newspaper articles have been subject to detailed annotation down to the sentence,
phrase, or even word level. The annotation of the METER corpus is described in the following subsections.
Due to limitations of project resource and time, all annotations were carried out by one person, a
professional journalist. However, we are in the process of getting second judgements from another
expert to verify the choices made by the original journalist in the general classification task, for 5% of
texts in each category. Resources permitting we will validate the more detailed annotations later.
5.1. General classification at the document level
As mentioned earlier, every newspaper text in the METER corpus shares a topic with one of the PA
catchlines. However, the newspaper articles are related to their PA counterparts in a variety of ways.
Some of them use whole pieces of PA text without any change; some of them modify PA texts to fit
their specific requirements; some of them supplement PA materials with other content that is not to be
found in the PA materials; and some do not appear to have consulted the PA at all, even though the PA
provided relevant materials for the story.
In order to capture general information about the reliance of newspapers on the PA, each newspaper
text is classified by an expert journalist into one of the following three categories:
a) Wholly derived (WD) – all content of the target text is derived only from the PA.
b) Partially derived (PD) – some content of the target text is derived from the source text. Other
sources have also been used.
c) Non-derived (ND) – no content of the target text is derived from the source text. Although
verbatim and rewritten text may appear in the target text, the context, overlap of entities or use of
source text is not indicative of reuse.
Note that this classification is based upon judgements concerning the source of the content in the
newspaper article, not simply upon surface criteria, such as presence of a certain number or length of
shared tokens. Since we are interested in studying the mechanisms of text reuse, we cannot begin by
presuming what we hope to discover; i.e. we cannot begin by defining text reuse in terms of surface
linguistic criteria. Otherwise, we can never hope to discover more than our initial definition. Instead we
rely on the judgement of expert journalists who bring years of experience to bear, comprising
specialised linguistic and world knowledge.
In a wholly derived newspaper text, all of the facts in it can be mapped, with varying degrees of
directness, to PA text(s) under the shared catchline. In the most direct cases, the whole or part of a PA
220
text is copied verbatim to form the newspaper text. In the other cases, a PA text is modified in various
ways before being deployed in a newspaper article, including change of word order, substitution of
synonyms, and paraphrase. In such cases, the relation of the newspaper text and its counterpart PA
text(s) is often not so clear, sometimes even difficult for a human to infer.
In a partially derived newspaper text, part of the text can be mapped to the corresponding PA text(s),
but for other parts no related materials can be found in the PA texts. In other words, newspaper articles
in this category contain new facts not found in the PA. This category represents an intermediate degree
of dependency of newspaper texts on the PA. Within this category, the level of dependency varies
considerably, from the majority of text being derived to only one or two sentences being derived from
the PA.
The last category covers those newspaper articles that are written independently from PA. Note that we
are considering newspaper and PA texts covering the same event. This means that the PA provides
coverage of the event, but that it has not been utilised by the newspaper. Instead, the newspaper has
used other journalistic sources. This category represents the null dependency of the newspaper on the
PA. The number of texts in each category can be obtained from Table 1.
This three-way classification is encoded in the METER corpus as described in section 5.3 below. With
such information explicitly available, the METER corpus can be used for training/evaluating
algorithms for detecting text reuse in journalistic domain, studying journalistic styles, etc.
5.2. Detailed classification at the lexical or phrasal level
In addition to the document level classification of the whole of the newspaper portion of the METER
corpus, detailed annotation of about 400 of the wholly or partially derived newspaper articles was also
carried out. In this annotation, individual words, phrases or sentences in the newspaper texts were
tagged with information about their derivation from the PA.
The detailed classification parallels the text-level classification discussed in the previous section. Three
categories were used:
1) Verbatim: text that is reused from PA word-for-word in the same context;
2) Rewrite: text that is reused from PA, but paraphrased to create a different surface appearance. The
context is still the same;
3) New: text not appearing in PA or apparently verbatim or rewritten, but used in a different context.
Of these three categories, the rewrite appears in various forms, such as change of word order or
paraphrase. Such modification of the PA texts occurs due to four main reasons: a) the PA text may be
re-written to comply with the house style of the newspaper; b) it may be re-written to fit the space
available in the newspaper; c) information dispersed throughout the PA texts may be re-arranged into a
single, coherent sequence; d) a PA text may be dramatised by replacing neutral words with more
dramatic words. Accordingly, a PA text can be modified in the following ways:
1) Rearrangement of word/phrase/sentence order or position;
2) Substitution of original terms with synonyms or other context dependent substitutable terms;
3) Deletion of original materials;
4) Insertion of minor new materials (e.g. addition of words like by in passivisation),
In the annotation, newspaper materials falling into any one of the above four categories were tagged as
rewrite. There are some controversial cases in which subjective judgement had to be made. On the
other hand, the tagging of the verbatim and new materials was generally straightforward, although
minor mistakes are inevitable for a manual annotation. All the information about the dependency of
newspaper materials on the PA is encoded in an SGML annotation scheme, as described in the
following sub-section.
5.3. Annotation of the METER corpus
The METER corpus is annotated in SGML (Standard Generalised Markup Language) to comply with
the international mark-up standard (Goldfarb, 1990) and a customised SGML DTD (Document Type
221
Definition) was developed for it. The DTD provides a framework for recording general information
about the texts (such as date, catchline, etc.) as well as for recording information specifically pertaining
to reuse. Figure 3 illustrates the structure of the METER SGML DTD document5.
As shown in Figure 3, the METER mark-up scheme consists of three levels of tags. The topmost is the
header which keeps general information about files, including file name, source newspaper, newspaper
page number on which the report appeared, date of release and catchline. For newspaper articles it also
includes the text level classification of the article as wholly derived, partially derived, or non-derived.
This header mark-up is found in every text in the whole corpus.
The second level consists of two elements: title and body of the document. The title is optional.
The third layer consists of three elements: verbatim, rewrite and new (for their definitions, see section
5.2). They are sub-elements of body, and apply to any individual tokens or token sequences in the
newspaper article, including single words, phrases, sentences and punctuation marks. Generally the
annotation focuses on words rather than punctuation marks. But the latter were included in the
annotation, for they may have significant role in some cases, e.g. in the recognition of quotations.
The detailed annotation was carried out in two stages. First, a journalist analysed and classified the
contents of about 400 newspaper articles on paper. Later, the classification was transcribed into tags in
the electronic version of the corpus using an annotation tool. Figure 4 displays a sample of annotated
newspaper text.
In this example the first line informs an SGML parser of the document type and the location of the
DTD file. The header indicates that the sample is a newspaper report about a court story related to the
PA catchline “banker” on the 4th page of the Telegraph released on 16 July 1999 and is wholly derived
from the PA. The text body is broken into segments each of which is tagged with one of the three
5 In SGML, the DTD document defines the constituent parts of a document in terms of objects, known as elements.
Each element may have parameters called attributes.
Figure 3: Structure of the METER DTD
<METER document> (required)
Header filename: filename of the text (required)
attributes: newspaper: the newspaper name (required)
domain: courts or showbiz (required)
classification: either wholly-derived, partially-derived or non-derived
(optional)
pagenumber: the newspaper page number (optional)
date: the date of publication (required)
<Title> (optional)
No attributes
<Body> (required)
No attributes
<Verbatim> (optional)
Attributes: PA_src: the source PA sentence(s) (optional)
<Rewrite> (optional)
Attributes: PA_src: the source PA sentence(s) (optional)
<New> (optional)
Attributes: PA_src: the source PA sentence(s) (optional)
222
categories: verbatim, rewrite and new. The value of the attribute PA_src, which indicates location of
the PA source sentence(s), is now blank, but may be completed in a subsequent release of the corpus.
Figure 4: A sample annotated newspaper article from the METER Corpus
6. Preliminary experiments with the METER corpus
Currently, the corpus is being used in the METER project for training and evaluating algorithms to
detect text reuse. Our initial investigations have utilised the document-level annotations only and have
addressed the following task: given a PA text and a candidate derived text, determine whether the text
is wholly derived, partially derived or non-derived. We have tried three approaches to this task: the
dotplot, n-gram overlap and text alignment. Details can be found in Clough et al. (2001); here we just
give a brief overview of the work to illustrate how the corpus can be useful.
The dotplot is a tool adapted from the biological domain to visualise the similarities and differences
between input streams of either text in electronic format, or software code (Helfman, 1993). This
technique was tested on METER texts from the three categories: wholly derived, partially derived and
non-derived. It was found that when compared with candidate PA source texts, newspaper texts from
the three categories could generally be identified by distinct dotplot patterns. However, a disadvantage
of the dotplot is that without more complex processing, no quantitative similarity value is produced
above and beyond the visual image. Building the dotplot for large input streams is also computationally
expensive.
An alternative approach is to measure similarity between texts by simply measuring the number of
word n-grams shared between them. From initial experiments, we found that derived texts appeared to
share more n-grams of lengths 3 words and above. This follows the intuition that derived texts share
longer matching strings than non-derived texts. However, we also found some instances of non-derived
Original PA version
BANKER'S BITTERNESS LED TO SYSTEMATIC THEFTS<
By Lyndsay Moss, PA News<
A middle-aged banker who stole more than &#163;270,000 from his bosses because he resented younger staff being promoted
over his head, was jailed for four years today.<
Trusted Derek Boe, 48, used some of the money to splash out on holidays, buy a car and a caravan, and pay for expensive
home improvements.<
Telegraph version:
A BANKER who stole more than &#163;270,000 from his bosses because he resented younger staff being promoted over his head,
was jailed for four years yesterday.
Derek Boe, 48, used some of the money for holidays, to buy a car and a caravan, and to pay for home improvements.
Annotated Telegraph version:
<!DOCTYPE meterdocument SYSTEM "meter_corpus/dtds/meter.dtd">
<meterdocument filename="meter_corpus/newspapers/annotated/courts/16.07.99/banker/banker125_telegraph.sgml",
newspaper="telegraph", domain = "courts", classification="wholly-derived", pagenumber="4", date="16.07.99",
catchline="banker">
<body>
<verbatim PA_src="">A </verbatim>
<verbatim PA_src="">BANKER who stole more than </verbatim>
<rewrite PA_src="">&#163;270,000 </rewrite>
<verbatim PA_src="">from his bosses because he resented younger staff being promoted over his head,
was jailed for four years </verbatim>
<rewrite PA_src="">yesterday. </rewrite>
<verbatim PA_src="">Derek Boe, 48, used some of the money </verbatim>
<rewrite PA_src="">for </rewrite>
<verbatim PA_src="">holidays, </verbatim>
<rewrite PA_src="">to </rewrite>
<verbatim PA_src="">buy a car and a caravan, and </verbatim>
<rewrite PA_src="">to </rewrite>
<verbatim PA_src="">pay for </verbatim>
<verbatim PA_src="">home improvements. </verbatim>
</body>
223
texts which contained shared 10-grams (due to shared directly quoted text). Using the METER corpus
we trained the classifier by selecting optimal threshold values for various parameters. Results in testing
ranged from 50-70% correct in classifying documents as wholly derived, partially derived or nonderived.
The final approach to this classification task that we have explored so far is text alignment (see, e.g.,
Manning and Schütze (1999) for a review of statistical alignment techniques). Given a candidate
derived newspaper text, we first carry out best-match alignment at the sentence level. Then we estimate
the dependency of the whole candidate derived text on the source, using parameters derived from the
METER corpus. Preliminary results show 80-90% correct classification, depending on the setup. The
METER corpus has provided invaluable data for the development and evaluation of this algorithm.
These experiments are but a few examples of potential applications of the METER corpus. When it
becomes widely available to the research community, we envisage that a much wider range of
applications will be found for it.
7. Conclusion
In this paper, we have described a new corpus – the METER corpus – built as part of a project to
investigate text reuse in the world of newspaper journalism. It contains texts from the domains of law
courts and show business reporting. The texts were collected from the Press Association and nine
British national newspapers. The data were manually collected and classified by a professional
journalist. All of the newspaper articles are classified at the document level based on their dependency
on the PA. Some of the newspaper articles are also annotated at the phrasal or even lexical level to
indicate material that is verbatim, rewritten or new.
This is an innovative corpus resource for the communities of corpus linguistics and natural language
processing/engineering. The critical role of the PA in the contemporary British media industry and the
breadth of the newspaper sources from which the data were collected all warrant that the METER
corpus is highly representative of contemporary media in the given domains. The fact that all of the
data in the corpus were manually selected and tuned guarantees high quality and reliability. Until now,
no corpora marked up with information about text reuse have been reported. The METER corpus fills
this gap. We envisage that the corpus will be of use for a range of research, including the study of text
reuse, plagiarism, journalistic style, lexicon building, document clustering and language generation.
Acknowledgements
The authors would like to acknowledge the UK Engineering and Physical Sciences Research Council
for financial support for the METER project (GR/M34041). We would also like to thank the Press
Association for supplying us with access to their newswire archive and for discussions regarding text
reuse in the British press. Finally we would like to thank Andrea Setzer for supplying and helping us to
customise the annotation tool we used to annotate the METER corpus.
References:
Clough P, Gaizauskas R, Piao S, Wilks Y 2001 METER: MEasuring TExt Reuse. Department of
Computer Science Reseach Memorandum CS-01-03, University of Sheffield.
Goldfarb C 1990 The SGML Handbook. Oxford, Oxford University Press.
Helfman J 1993 Dotplot: A Program for Exploring Self-Similarity in Millions of Lines of Text and
Code. Journal of Computational and Graphical Studies 2(2): 153-174.
Johansson Stig, Ebeling Jarle 1996 Exploring the English-Norwegian parallel corpus. In Percy Carol
E., Meyer Charles F., Lancashire Ian (eds), Synchronic corpus linguistics. Amsterdam-Atlanta,
GA, Rodopi B. V., pp 3-15.
Leech Geoffrey 1997 Introducing corpus annotation. In Garside Roger, Leech Geoffrey, McEnery
Anthony (eds), Corpus annotation – linguistic information from computer text corpora. London &
New York, Longman, pp 1-18.
Manning C, Schütze H 1999 Foundations of Statistical Natural Language Processing. Cambridge,
MA, MIT Press.

224
ROSETTA: Rhetorical and semantic environment for text alignment
Hatem Ghorbel, Afzal Ballim, Giovanni Coray
LITH-MEDIA group
Swiss Federal Institute of Technology
IN Ecublens, 1015 Lausanne, Switzerland
Phone:+41-21-693 52 83 Fax:+41-21-693 52 78
{hatem.ghorbel, afzal.ballim, giovanni.coray}@epfl.ch
Abstract
In the framework of machine translation of multilingual parallel texts, the technique
of alignment is based on statistical models and shallow linguistic parsing methods.
When addressing the problem to corpora where different versions are derived or
interpreted from the same source, we need further criteria to consider the forms of
disparity between these versions. In this article we propose a content-driven
approach based on the semantic and pragmatic structure of texts to aid in the process
of alignment and comparison between their versions.
1. Introduction
Alignment is the process of establishing the relationship between the different subparts of two or more
comparable documents. Much of the early work on alignment is still used as the basis for more
advanced systems. These methods are mainly based on statistical models of translated texts, with some
based on word or character frequencies, others on string occurrences. Whereas previous work in
alignment has viewed texts as essentially a flat stream of characters or words, other approaches have
incorporated some structural properties of the documents as further criteria. For example the logical
structure of the documents (e.g., sections, chapters, titles, etc.) is often used. In this paper, we propose
an extension to the existing methods where we consider the semantic and pragmatic structure of texts as
the basic criteria to aid in the process of forming a correspondence between a parallel pair.
We are focusing on parallel corpora where different versions are derived from the same material (intra
or inter lingual documents). The practical goal of alignment is to give experts, students or ordinary
users a tool that facilitates the on-line comparative analysis of ancient texts and to navigate through the
various components of the different versions. Some experiments on ancient manuscripts of medieval
French have shown the limits of statistical alignment due to the considerable variation of these versions,
which exhibit omissions, insertions and substitutions that range from words to sentences and sometimes
to larger spans of texts. Despite these lexical and syntactic variations, the semantic content i.e the
meaning is kept invariable. This presumes that these versions have very close semantic structure. We
argue this hypothesis with the fact that each text holds a semantic content within its lexicon and its
linguistic structure. Generally, this content remains invariable when the text is translated or when a
further version is reproduced. The variation would be at the lexical level or at the level of details added
or omitted. The main ideas and the intentions of the writer are however kept the same. Based on this
hypothesis, if we manage to model the semantic content by means of an appropriate discourse structure,
we can compare the content from the perspective of such a structure.
As a very end goal for alignment, we intend to provide an environment to facilitate the analysis of
ancient texts and to enable data access of contextually relevant materials such as slides of original
books, commentaries from rare books, annotations added by domain experts. This meta-information is
often divorced from the original versions; thus making it easily available to users provides a powerful
educational mechanism for specialized studies.
2. Background
When the idea of fully automated high quality translation was given up by most of researchers, semiautomatic
and assisted translations became the goal of most of the current projects of translation. To
support such approaches many linguistic resources (machine-readable dictionaries, thesauri, tagged
corpora, etc.) and knowledge bases (Wordnet) were built. Parallel corpora were important bases of
225
multilingual resource that complements previous classical tools of translation either for human or for
machine. Basing on translated segments in parallel corpora, some approaches of memory-based or
example-based machine translation were developed (Sato & Nagao 1990, Sumita et al. 1990). Parallel
corpora were used then for other natural language applications, for instance in translation consistency
checking, cross-language information retrieval, document comparison and other multilingual
applications. The problem of alignment and establishing correspondences between segments of
documents became an issue and an end in itself.
3. The alignment problem
3. 1. Alignment of same-text versions
Same-text versions are different presentations of texts derived or based on the same original document.
These versions represent different same-language or cross-language translations preserving semantic
content i.e. meaning. Usually these versions are interpretations of epic stories that tell about gods or
about the adventures of great heroes. They often represent the values and ideals of an entire country or
people. An automatic alignment among this kind of texts establishes a mapping that relates the content
of one text to the content of the other, wherein the subject of the mapping are interpretations of the
same span of the original text. This is an interesting problem that is closely related to the alignment of
different language translations of a common source. Nevertheless, in the latter case, the translation is
usually so regular and homogeneous that a great deal of success is achieved using existing techniques
(see state of the art). Unfortunately, most of this work has focused on cross-language translations and
very few projects have considered alignment of different versions of ancient text. Own et al. (1998)
have focused on the alignment of Iliad, Odyssey, and bible versions in the framework of the HEARER
HOMER project.
Our motivation to perform this sort of alignment is to introduce a new tool that supports
1. comparative navigation within a corpus of parallel versions of ancient texts enabling a comparative
analysis of the translation style, the linguistic and the geo-linguistic features and the dialectal
properties of the language.
2. an elaboration of the comprehension of the documents by grouping parallel segments from various
versions that can be found in another textual or graphical form.
3. random access of information through the different textual streams and retrieval of content and
meta-data attached to the translation content from the different versions, basing on a search of an
original document or an alternate translation.
3. 2. Case study: medieval texts
In this project we are interested in French medieval manuscripts, in particular the manuscripts produced
between the XIIth and the XVth century. These manuscripts are sets of different conversions over the
time of the same original texts. They are written by authors -often unknown- having different cultures
and skills. Each manuscript reflects, thus, its own cultural and geo-linguistic features that depend on a
particular civilization. As a first step, we worked with extracts from the same versions of the
manuscript "Ovide moralisé" (versions of Geneva, Paris, Lyon, and Rouen) as well as versions
translated to Latin and modern French by domain specialists. While manipulating these manuscripts we
have noticed that:
1. the structure is very different from one version to the other. Although there seems to be a certain
isomorphism in the structure within one class of versions (verse or prose), the variation is striking
when comparing two versions from different classes. These irregularities in structure make it quite
difficult even to perform a manual alignment.
2. Sentence and paragraph structure is difficult to detect in many versions due to the loss of
punctuation or uncertainty about the sentence structure.
3. Apart from the variation in the structure between the different classes of documents, the variation
in the linguistic level contains the following attributes:
226
- Morphological variation is quite abundant in all the versions. For instance the word
"plusieurs" can be found as "plusiours". The conjugated verbs particularly cover a great
number of variants of dialectal, orthographic or analogue (to modern French) origins.
- Semantic variations are less abundant but quite important, particularly among versions from
different classes. These variations are mainly the use of synonyms or other group of words
having the same realization (e.g. mal / mauvaise, prouffit / enseignement).
- The pragmatic and discourse structures are very close even if in some versions we find more
elaborated parts than in others, or in some cases more detail is given, particularly when it
comes to a description or an argumentation. Despite this disparity in the expressiveness of the
language, the core of the content is kept invariant among all the converted texts.
After some experiments, we found that the alignment among same class versions can be done using
standard techniques, particularly for texts of the verse class. This is due to the following two reasons:
1. The document structure is very close or in some cases identical: A poem is composed of titles (to
announce a beginning of a division), verses (organized line by line) and annotations in the margins
(often transcribed within the verse structure with a special markup). The graphical objects are
inserted randomly in the document without a prior order. The most significant variation in this
level is the omission or the insertion of a title, or a block of annotation or at most a block of verse
that does not exceed a couple of lines in the worst cases.
2. The linguistic variation is limited to the morphological level and in many cases to the semantic
level where a simple substitution of a word by its synonym has taken place.
As previously noted, in this particular case we are dealing with a classical problem of alignment where
statistical approaches have proved their success. The TALCC aligner (Ballim et al. 1998) has been used
and achieved about 90% success when respecting a same generic model of documents. The TALCC
aligner is based on the Gale and Church algorithm with the addition of the logical structure of
documents as a further criterion of alignment (see state of the art).
3. 3. The alignment problem with verse/prose documents
The first problem that we were faced with when comparing verse and prose versions was the document
structure. The structure of a verse manuscript is composed of titles and verses organized line by line.
Each verse may be a clause or in rare cases a whole sentence. Punctuation does not exist originally in
the material but is introduced by experts. Annotations on the margins are transcribed within the verse
structure with a special markup. Instead, a prose is a block of text organized in a paragraph structure
each one starting with a title that describes that content. Sentences within the paragraphs are usually
separated by original punctuation though in some cases it is difficult to detect (figure 1).
Linguistic variation
Whereas verses are mainly segments of sentences expressed in an artistic style enclosing a form of
rhyme, the content in sentences in a prose is more compact and expressed with a lighter descriptive
style. Obviously, the linguistic construction is different in many ways; the lexicon submits to the usual
morphological variation that can exist between two versions as described before, however the order is
not preserved. Syntactically the whole construction is different: a sentence in the prose version can
correspond to one or more sentences in the verse version. A word itself in one version can be
elaborated to a whole verse or even a block of verses. Usually the prose version is more compact than
the verse one with a ratio of nearly 3/4.
Pragmatic variation
When it comes to the pragmatic structure, the difference is noticed on the level of depth of explanation
and expressiveness. It has been noticed that the verse versions are enriched with more elaboration,
restatements, argumentation, details (temporal, location, manner, circumstances, etc) and descriptions
(comparison, causative events). The comments and personal interpretations are often more abundant in
the verse versions. This explains the difference in size between the two manuscripts. Nevertheless, it
has also been noted that the salient segments are kept invariant and submits only lexical variations or
synonymy substitution.
227
Verses in medieval French Prose in medieval French
Se l'escripture ne me ment
Tout est pour nostre enseignement
Quant qu'il a es livres escript
Soient bien ou mal li escript
Toutes escriptures soient bonnes et mauvaises
sont pour nostre prouffit et doctrine faittes.
Figure 1: Example of alignment of verse and prose versions in medieval French.
4. State of the art
The problem of alignment seems to have first been raised when Brown (1988) and his colleagues tried
to build a probabilistic model for automatic translation. Debili (1992) faced the same problem when he
planned to set up dictionaries of bilingual expression transfers and synonyms. The alignment problem
was then treated as only second or peripherical. Many authors now set the alignment problem in a more
global framework. For instance, Warwick (1989) places the alignment in the context of the
implementation of lexicographic tools for linguists and translators, or, more recently, as an aid to the
evaluation of translation quality.
A good deal of work has already been done on alignment (Brown et al. 1991), (Gale and Church 1991)
and (Simard et al. 1992). Since then several other approaches have been used, both for sentences, word
and character alignment (Papageorgiou et al. 1994; McEnery et al. 1995). All these methods are mainly
based on statistics, some based on word frequencies, others on characters occurrences.
Gale & Church's character-based algorithm propose a method which uses only internal information
and does not consider any hypothesis on the lexical content of the sentences. Authors started from the
observation that the length of sentences in the source text and its translation in the target text are
strongly correlated: short sentences tend to be translated into short sentences and long sentences into
long sentences. Furthermore, it seems that there exists a rather constant ratio between the length of
sentences from a language to another in terms of number of characters. This method has been tested on
a bilingual corpus of 15 economic reports published by the Swiss Banks Union in English, French and
German, for a total of 14'680 words, 725 sentences and 188 paragraphs in English and their
corresponding numbers in the two other languages. This method makes it possible to correctly align the
total amount of sentences, except for 4 % of them. The same number of errors has been found in the
English-French and English-German alignments, showing that the method is relatively language-
independent. The model proposed by Gale & Church has also been tested on a much more important
sampling, of 90 million words, taken from the official Canadian Hansards corpus.
Brown's word-aligner algorithm combines sentences according to the number of words included in
each sentence. This algorithm is described by its author as a development of Gale & Church's
algorithm, which computes the length of sentences from the number of their characters. The data of the
official Hansards (McEnery et al. 1996) corpus of the Canadian parliament official decrees have first
been converted into a unique English corpus and another French corpus. Each of these corpora has
been fragmented into token and these tokens have been combined into groups called sentences.
Moreover, auxiliary data such as the numbering of Parliamentary Sessions, the name of speakers, the
time index and the ordering of questions were used to add comments throughout the text. Each of these
comments can be used as an anchor point in the alignment process. The alignment of anchor points is
made in two passes, first for the main anchor points, then for the secondary ones.
Chen's alignment model (Chen 1996) is built from a sample of data aligned in two languages, English
and French, and tested on samples taken from the Hansards. This model, conceived on an ad hoc basis
in the framework of Bayes' paradigm allows to take into account frequent cases where sentences don't
align in a uniform way, in a 1:1 ratio, but rather in a 2:1 ratio. The search strategy used, which is that of
dynamic programming, allows a linear search in the length of the corpus. This strategy includes a
separated mechanism to process a large number of suppressed sentences in one or the other version of a
bilingual corpus.
Martin Kay's method (Kay, 1993) is based on an EM (Estimation and Maximization) type of algorithm.
Here, the alignment of sentences depends on the alignment of words. That of words depends on the
similarity of their distribution. The method proposed by Kay is relatively more complicated to
implement than the other methods already mentioned.
228
Simard and his colleagues give a simple method, which attempts to align texts on the basis of related
words (cognates). This method seems to give satisfactory results but its drawback is that it can only
operate on rather closely related language pairs. Cognates are words, which are almost identical in both
languages (artificial/artificiel). This thus implies that both languages must be written with the same
alphabet (Simard et al. 1992).
There have been some innovative works that incorporated further criteria in alignment such as the
linguistic knowledge and the structural properties of the documents. The use of linguistic knowledge
covers mainly the process of parsing (Dagan et al. 1996; Matsumoto et al. 1993) and tagging (Van der
Eijk, 1993). Kupiec (1993) proposes an algorithm for finding nominal syntagms matching each other in
a bilingual corpus. In this algorithm, syntagms are thus recognized with the aid of a specific program
and the correspondences between these syntagms are determined with an algorithm based on simple
statistical techniques. The use of external linguistic resources mainly bilingual dictionaries is quite
efficient in identifying lexical anchors (Catizone et al. 1989; Warwick & Russel 1990; Debili &
Sammouda 1992).
Structure-driven methods consider the text as structured flow of information and manipulate this metainformation
about the organization of the text structure to aid in the process of alignment. Ballim and
al. (1998) developed an aligner which takes advantage of the global structure that many documents
have (e.g., sections, chapters, titles, etc.) This structural information is integrated with other similarity
metrics such as: number of characters in PCDATA, cognates, bilingual terms and parts of speech to
decide the correspondence between parallel segments. Tests and evaluations have showed that the
structure-driven alignment is efficient with isomorphic documents having the same generic logical
structure. However it was much more difficult to deal with non-isomorphic documents although
referring to the same generic logical structure. In the same framework of structure-driven alignment
Romary & BonHomme (2000) have used the TEI annotation guidelines to calculate the best alignment
pairs from the multilingual texts at division, paragraph and sentence level.
5. Semantic and pragmatic approach
The success of one approach or another depends strongly on the nature of documents. When aligning
the Hansard corpus for instance, statistical approaches were enough the reach a great performance
(nearly 97% with the Chen algorithm). This is mainly due to the specificity of the translation between
French and English, which is often located in the word and sentence level. When it comes to structured
documents, the logical structure is a further hint to guide the process of alignment. That what has been
shown for instance by Ballim et al. in some experiments with Systematic Corpus of federal Laws from
the Swiss federal Chancellery. However, when we address the problem of alignment with other types
of documents having different properties, we must find further criteria of similarities. Still little work
has focused on the linguistic properties of the documents because they are difficult and complex to
compute and to deal with. Nevertheless in some cases alignment becomes a task that is coupled with a
semantic and pragmatic understanding of the content of the texts. It is the case of the different versions
of ancient texts where even a manual alignment needs experts’ competence.
Thus our semantic and pragmatic approach looks at texts from the perspective of their discourse
structure rather than from the textual flow of data. The discourse structure is the way text is organized
to insure coherence and cohesion of its content. Most of the discourse theories (Hobbs 1985; Grosz &
Sidner, 1986; Mann & Thompson 1988) consider texts as a set of segments related by means of
pragmatic relations. Parallel texts express similar semantic content i.e. share a same meaning, therefore
their discourse and pragmatic structure must be very close. Hence this structure could be considered as
a further criterion to detect similarities where classical approaches failed.
5. 1. Description of the discourse structure
In our hypothesis we have adopted the Rhetorical Structure Theory (RST) (Mann & Thompson 1988)
as a model of discourse organization. RST is a descriptive theory about the organization of natural
texts, characterizing their structure basically in terms of a closed set of relations called rhetorical
relations that may hold between their parts. The term rhetorical is not limited to the relations that have
a rhetorical sense but can be extended to other kinds of relations such as semantic, pragmatic, logical or
even very special domain-dependent relations. Texts are decomposed into non-overlapping units called
discourse segments. Each segment is related to a span of segments by means of a relation and is called
a nucleus or a satellite (there are a few exceptions to this rule: some relations can join two nucleus
229
segments, they are called multinuclear relations). The distinction between nuclei and satellites comes
from the empirical observation that the nucleus expresses what is more essential to the writer's purpose
than the satellite, and that the nucleus of a relation is comprehensible independent of the satellite, but
not vice versa. Text coherence in RST is assumed to arise from a set of constraints applied on the
nucleus, on the satellite and on their combination. For example in the following sentence:
Although we obediently ate everything our mother prepared, my sister and I much preferred to eat our
fruit crisp.
We detect a concession relation, the situation described in the nucleus (second clause of the example) is
in contrast to that presented in the satellite. It is about a violated expectation.
The model of discourse structure we are using obeys the constraints put forth by Mann and Thompson
(1988) and Marcu (1996). It is a binary tree whose terminal nodes represent the elementary units and
non-terminal nodes represent the relations holding between spans of texts (In figure 2 Arrows are
pointing to the nucleus spans).
5. 2. Semantic-pragmatic annotation
Recent developments in computational linguistic have created the means for the automatic derivation
of rhetorical structures of unrestricted texts. Marcu (1996) suggested an algorithm that uses cue phrases
and a simple notion of semantic similarity in order to hypothesize rhetorical relations among the
elementary units. Nevertheless, these algorithms are still domain dependent and the efficiency is their
main drawback.
To structure our texts, we proceeded by a manual annotation of a sample corpus (versions of Geneva
and Paris) to evaluate the complexity of the task. We have fixed a taxonomy of relations where each
class is composed of subclasses of more specific relations. We distinguish three main classes, semantic
relations, inter-personal and textual relations. Semantic or informational relations are mainly relations
used to describe how information is conveyed for instance elaboration, comparison, circumstance,
condition and causative. Inter-personal or planning relations are relations that hold a pragmatic
intention for example interpretation, evidence, explanation and argumentation. Textual relations are
rather relations that have an influence on logical structure of the text for instance list, conclusion,
disjunction, conjunction, summary, joint, topic-drift and sequence. Such classification gives more
freedom to annotators to choose the relations according to their own understanding, and permits to
build a similarity measure in the process of comparison (Ghorbel 2001). The first task of the annotation
is the process of segmentation. Unlike previous work where segmentation is basically situated in the
clause level, we focused on a more global view; the sentence level and in some cases on larger blocks
of texts. This kind of macro segmentation allows us to define the elementary units of the discourse
structure and eventually the units of the alignment process. The larger the segments are, the easier the
computation of correspondence is, but the less precise the alignment is. On the other hand, considering
very short segments we will end up with very large trees and the problem of complexity becomes
important.
The second task of annotation consists of grouping the elementary units together by means of either a
mononuclear or a multinuclear relation. This process will create spans of texts or discourse segments
Figure 2: RST model of discourse structure.
230
related in the form of an ordered binary tree (figure2). Within this tree we can detect certain paths
formed by the nuclear nodes. This path structure will play an important role in the alignment process.
Unlike previous work (Marcu for summarization (1998), for automatic translation (2000), for essay
scoring (2000) and Cristea (1998) for anaphora resolution) where the whole text is represented as a
single tree, and since we are working with long texts, we found it more appropriate to consider the texts
as a forest of trees. Separation between trees is viewed as a topic shift in the texts. Still this concept of
separation between trees is subjective as it depends on the annotator, but it does not have adverse
effects since in the alignment process a tree from the source text can be aligned with more than one tree
from the target text.
6. Multi-criteria alignment
6. 1. semantic-pragmatic structure alignment
We have shown that the semantic and pragmatic structure of the discourse is very important to find
anchor points and hints to align segments of similar texts. We propose in this section some algorithmic
approaches to use the knowledge stored in the suggested model to detect similarities and make
correspondence between segments.
Problem formulation
Consider a document D1 to be aligned with D2. D1 and D2 are respectively represented as forests of n
and m trees as follows:
D1 ={T1
1 , T 1
2 ,…, T 1
n }
D2 ={ 2
1 T , 2
2 T ,…, 2
m T }
Each tree T d
i (the ith tree of the document d) is composed of text spans structured in a binary tree
structure. The terminal units of d
i T are text segments ordered from left to right as they appear as text
sentences in the document. We denote these segments as follows:
d
i T ={ d
i
sr , d
i
s
1 + r , …, d
i
s
1 - s , d
i
ss } where ir and is are respectively the left and the right boundaries of the
tree d
i T .
We call the salient path (SP) in a tree the path followed when we navigate from the root to the terminal
elementary units and choosing the nucleus nodes when coming through a relation node. In a tree there
exists only one SP if and only if all the chosen relations are mononuclear and ( is - ir +1) SPs if all
chosen relations are multinuclear. In the general case the number of SPs, let's denote this number p ,
p then ranges between 1 and ( is - ir +1). Each SP points to a terminal elementary text segment. Let's
denote by d
i T) the set these terminal units in a tree, then we have:
d
i T) ={ d
j s / i j i s r &#163; &#163; } where card( d
i T) )= p .
Pragmatically, the set d
i T) stands for the span of the text in the tree i of the document d, which is
considered as far as it is seen by the annotators when segmenting and attributing the nucleus and
satellite properties to segments, for the main part that helps readers understand the sense and the
content of the tree i. For example in figure 2, ir =2, is =8, d
i T) ={ d s5 }, p =1, the SP is hence the path of
the 5th segment.
We call the k-nearest neighborhood of the SP the k satellite segments having the minimum distance
with SP. We define the distance dm as the metric distance which is defined from d T * d T to the set of
natural numbers where d T = U
n i
d
i T
.. 1 =
. The metric distance between two segments is given as following;
231
dm ( d
i s , d
j s )= |j-i|. We define also ds as the structural distance applied between the nodes of a tree and
which has the value of the length of the minimum path between the nodes. The length of a path in the
tree is given by the number of nodes it contains between its two extremities.
The process of alignment
The semantic and pragmatic annotation as described previously models the text as a forest of trees.
Each tree holds in its structure a set of segments related to each other by means of semantic or
pragmatic relations and ordered by their pragmatic importance in the way they hold and convey
information. This is modeled by the hierarchical structure and the concept of satellite/nucleus. The first
step in the process of alignment is to define a mapping M that establishes the correspondence between
the trees composing the two documents. M can be defined according to three approaches:
Approach 1: alignment based on the SP segments
The window of comparison of the texts is limited to certain specific segments, in this case only those
that form the set d
i T) . M is defined as following: M( d
i T )= ’ d
j T iif S( d
i T) , ’ d
j T) ) h &#163; where S is the
similarity measure and h is an empirical threshold. S is calculated basing on the lexical distance
between the streams of characters and referring to thesaurus database (see next section).
Approach 2: alignment based on the SP and its k-nearest neighbor segments
The window of comparison of the texts is limited to the segments that form the set d
i T) and their knearest
neighbors. The k-nearest neighbors can be obtained either by considering the M( d
i T )= ’ d
j T iif
D( d
i T) , ’ d
j T) ) h &#163; where D is the similarity distance measure and h is an empirical threshold. The knearest
neighbors can be obtained applying the metric distance or the structural distance. In both cases
we search for the k-nearest satellite to the SP. In the case the structural distance, when the found
neighbor is a sub-tree, we investigate its local SP which points to the first neighbor, the next ones will
be recursively determined according the first neighbor.
Approach 3: alignment based on the nature of the relations
Relations are classified according to their semantic and pragmatic similarities. We distinguish three
main classes, semantic relations, inter-personal and textual relations. Semantic or informational
relations are mainly relations used to describe how information is conveyed for instance elaboration,
comparison, circumstance, condition and causative. Inter-personal or planning relations are relations
that hold a pragmatic intention for example interpretation, evidence, explanation and argumentation.
Textual relations are rather relations that have an influence on logical structure of the text for instance
list, conclusion, disjunction, conjunction, summary, joint, topic-drift and sequence. Each subclass can
itself be more detailed. This approach of alignment of trees is based on a statistical comparison of the
frequency and the order of relations.
6. 2. lexical alignment of segments
Despite of the great variation of the conversion from one version to another, there still exists a certain
local similarity between words, particularly proper nouns, nouns, adjectives and verbs. Certain words
might be subjected to tiny morphological variations (change i/y, s/z, etc). For these reasons, we found
the word comparison (Brown et al. 1991) approach more relevant than character comparison. This form
of comparison is used only between certain segments of the text hypothesized by the structural
alignment (according to the previous approaches) to give a further heuristic of similarity.
Each segment is reduced to a word list where noisy data (article, determiners, pronouns etc) is
eliminated. A first approach is to apply a stemming algorithm and then compare the obtained stems.
The main drawback of this method is that inaccuracy in rules can map different forms to same stem. A
second approach consists in estimating the Levenshtein distance between each couple of words taking
into consideration some particular rules when estimating the costs of modification (substitution,
omission, and addition). A vector of words’ distance is hence obtained. The similarity distance measure
is the cost of the minimum path through the previous vector.
232
Besides the lexical similarity, we observed a kind of semantic similarity between the words used in the
different versions of the manuscript, for example mal/mauvais, prouffit/enseignement, etc. A thesaurus
that provides synonymy relation between words is in construction. When comparing two segments, the
synonymy relation is checked between each couple of words and the vector of words’ distance will be
modified.
7. Conclusions and future work
When dealing with comparable documents where manual alignment needs an elaborated and deep
understanding of the content, automatic alignment becomes a difficult task. Human interaction is then
needed to aid in the process structuring and modeling the content. In this framework, we proposed a
content-driven alignment whose heuristic is partially based on a manual human annotation of the
semantic and pragmatic structure of documents. The main drawback of this annotation is the
subjectivity of this task that depends closely on the profile of the annotator. Much of the current work
in devoted to define a simplified golden standard and to automate some sub-tasks in order to facilitate
the annotator's task.
We have also presented in this article some approaches of content-driven alignment based on the
semantic structure of the texts. The alignment process establishes a mapping between spans of texts
represented in a tree structure by comparing the significant (from a semantic view) segments. Therefore
the semantic structure built upon the text limits the window and drives the order of comparison. The
lexical comparison still remains an important heuristic of similarity. Preliminary results seem
encouraging and much of the future work will be focused on the evaluation of each approach of
alignment proposed (structural, lexical and thesaurus-based alignment) and on the integration of these
heuristics. The alignment at the segment level is also one of the interesting points we are investigating,
so that the granularity of the correspondence and the comparison could be refined.
References
Ballim A, Coray G, Linden A, Vanoirbeek C 1998 The use of automatic alignment on structured
multilingual documents. In Proceedings of the Seventh International Conference on Electronic
Publishing, Saint Malo, pp 464-475.
Brown P, Della Pietra S, Della Pietrs V, Mercer R 1988 A statistical approach to language translation.
In Proceedings of the 12th International Conference on Computational Linguistics, Budapest,
Hungary, pp 1-6.
Brown P, Lai J, Mercer R 1991 Aligning sentences in parallel corpora. In Proceedings of the 29th
Annual Meeting of the Association for Computational Linguistics, Berkeley, California, USA, pp 169-
176.
Burstein J, Marcu D 2000 Towards Using Text Summarization for Essay-Based Feedback. Septi&#232;me
Conference Annuelle sur Le Traitement Automatique des Langues Naturelles TALN'2000. Lausanne,
Switzerland, pp 51-59.
Catizone R, Russell G, Warwick S 1989 Deriving translation data from bilingual texts. In Zernik U
(eds), Proceedings of the first Lexical Acquisition Workshop Detroit, Mich, USA.
Chen S 1996 Building Probabilistic Models for Natural Language. PhD thesis Harvard University.
Cranias L, Papageorgio H, PiperidisS 1994. A matching technique in example-based machine
translation. In Proceedings of the Fifteenth International Conference on Computational Linguistics,
Kyoto, Japan, pp100-104.
Cristea D, Ide N, Romary L 1998 Veins theory: A model of global discourse cohesion and coherence.
In Proceedings of the 17th International Conference on Computational Linguistics and the 36th annual
meeting of the Association for Computational Linguistics (COLING, ACL), Montreal, Canada.
Dagan I 1996 Bilingual word alignment and lexicon construction. Tutorial notes, 34th Annual meeting
of the Association for Computational Linguistic, Santa Cruz, California.
Debili F, Sammouda E 1992 Appariement des phrases de textes bilingues. In Proceedings of the 12th
International Conference on Computational Linguistics, Nantes, France, pp 517-538.
Gale W, Church K 1991 A program for aligning sentences in bilingual corpora. In Proceedings of the
29th Annual Meeting of the Association for Computational Linguistics, Berkeley, California, USA, pp
177-184.
233
Ghorbel H (forthcoming) 2001 Guidelines for semantic annotation. Internal report, Swiss Federal
Institute of Technology IN Ecublens, 1015 Lausanne, Switzerland.
Grosz B, Sidner C 1986 Attention, Intentions, and the Structure of Discourse. Computational
Linguistics. 12(3):175-204.
Hobbs J 1985 On the coherence and structure of discourse. Center of the study of language and
information, CSLI-85-37, Leland Stanford Junior University.
Kay M, Roescheisen M 1993 Text-translation alignment. Computational Linguistics. 19(1):121-142.
Kupiec J 1993 An algorithm for finding noun phrase correspondences in bilingual corpora. In
Proceedings of the 31st Annual Meeting of the ACL, Columbus, Ohio, 17-22.
Mann W, Thompson S 1988 Rhetorical Structure Theory: Toward a functional theory of text
organization. Text. 8(3): 243-281.
Marcu D 1997 The Rhetorical Parsing, Summarization, and Generation of Natural language Texts.
PhD Thesis. Department of Computer Science, University of Toronto, Canada.
Marcu D 1998 Improving summarization through rhetorical parsing tuning. In Proceedings of The sixth
workshop on very large corpora, Montreal, Canada. pp 206-215.
Marcu D, Carlson L, Watanabe M 2000 The Automatic Translation of Discourse Structures. The 1st
Annual Meeting of the North American Chapter of the Association for Computational Linguistics
(NAACL'2000), Seattle, Washington, pp 9-17.
Matsumoto Y, Ishimoto H, Utsuro T 1993 Structural matching of parallel texts. In 31st Annual Meeting
of Computational Linguistics, Columbus, Ohio, pp23-30.
McEnery A, Oakes P 1995 Cognate extraction in the Crater project. In Proceedings of the EACLSIGDAT
workshop, Dublin, pp 77-86.
McEneryA, Wilson, A 1996 Corpus Linguistic. Edinburgh, Edinburgh University Press.
Owen C, Makedon F, Steinberg T 1998 Parallel text alignment. Research and Advanced Technology
for Digital Libraries, Special Issue containing invited ECDL'98 papers, Computer Science Lecture
Note Series, Springer Verlag, Christos Nikolaou and Constantine Stephanidis (eds) pp 235-260.
Romary L, BonHomme P 2000 Parallel alignment of structured documents. In Véronis, J. (eds.).
Parallel Text Processing. Kluwer Academic Publishers, Dordrecht.
Sato, S, Nagao M 1990 Towards memory-based translation. In Proceedings of the 12th International
Conference on Computational Linguistic COLING'90, Helsinki, pp 247-252.
Simard M, Foster G, Isabelle P 1992 Using cognates to align sentences in bilingual corpora. In
Proceedings of the Fourth International Conference on Theoretical and Methodological Issues in
Machine Translation, Monreal, Canada, pp 67-81.
Sumita E, Hitoshi I, Hideo K 1990 Translating with examples: a new approach to Machine Translation.
In the Third International Conference on Theoretical and Methodological Issues in Machine
Translation of Natural Language, Austin, Texas, pp. 203-12.
Van der Eijk P 1993 Automating the acquisition of bilingual terminology. In sixth Conference of the
European Chapter of the Association of Computational Linguistics, Utrecht, The Netherlands, pp 113-
119.
Véronis J 2000 Alignement de corpus multilingues. In Pierrel, J-M (eds). Ingénierie des langues.
Editions Herm&#232;s, Paris.
Warwick S, Russel G 1990 Bilingual concordance and bilingual lexicography. In Proceedings of
Eurolax'90, Malaga, Spain.

234
Is that a fact? A corpus study of the syntax and semantics of the fact that
Solveig Granath, Karlstad University
1. Introduction
Even though corpora have been used in language studies for the past few decades, up until now very
few grammars of English have based their descriptions of the language on the wealth of information
that can be gathered from them. There are indications that this is beginning to change. For example,
Biber et al (1999) use a corpus-based approach in their handbook, the Longman Grammar of Spoken
and Written English. It can be expected that this new approach will bring about both a revision and a
refinement of descriptions of structure and patterns of usage in present-day English. The advantages of
using corpora as the basis of grammatical descriptions should be obvious to anyone interested in
grammar: not only is it possible to provide examples of actual use instead of invented sentences, but in
addition, information on frequency of use, style, and the meaning of different features can be included.
The use of the fact that (i.e. the fact followed by an appositional that-clause) is a case in point. Learner
grammars tend to stress that the fact is used to link a preposition to a that-clause, since prepositions
may not govern that-clauses in English (Svartvik & Sager 1996:331, Hasselg&#229;rd et al. 1998:349).
Sometimes it is added that the use of fact presupposes that the that-clause expresses a fact. According
to this description, fact can thus be regarded both as a function word and as a content word. As a
reflection of this, it is interesting to note that the first two entries of fact in Collins Cobuild English
Dictionary (1995) refer to the syntactic use rather than the meaning of the word. Thus the first entry
says that “you use the fact that after some verbs or prepositions … to link the verb or preposition with a
clause”, and in the second entry the reader is informed that “you use the fact that instead of a simple
that-clause either for emphasis or because the clause is the subject of your sentence”. Only in the fifth
entry is a semantic definition of the word given: “when you refer to something as a fact… you mean
that you think it is true or correct”. In Quirk et al (1985:1001-02) the fact is even referred to as a
marginal subordinator, which stresses its syntactic function over and above its semantic meaning.
Whereas grammars and dictionaries thus demonstrate the need for the fact that in certain syntactic
patterns, usage handbooks often denounce the use of the fact as “deadwood” (Perrin 1942: 515, 161):
“The fact that is very often a circumlocution for which that alone would do as well: He was quite
conscious [of the fact] that his visitor had some other reason for coming.” Even quite modern usage
guides will warn users that the fact that “is often inserted quite superfluously into a sentence”, and
exhort the user to avoid the phrase, because it is “ugly and pretentious” (Kahn and Ilson 1985:233). It is
noteworthy that Quirk et al. (1985:659, 1002) also refer to prepositional phrases ending in the fact that
as “stylistically clumsy”.
Both grammars and handbooks thus tend to regard the fact primarily as a function word, which fulfils
little semantic function in the clause. This is not supported by Mair (1988), who draws two major
conclusions from his corpus study of spoken and written English: first, “the function of the fact is
primarily semantic: it indicates that the speaker takes the following clause to refer to a fact,” and
second, “the fact that is not a mere variant of the conjunction that but a genuinely suppletive form
which substitutes for that in contexts where the latter is ruled out” (Mair 1988:70). The present study is
an attempt to shed light on some syntactic and semantic features of the fact that. The paper will begin
with a presentation of the frequency with which the fact that appears in different functions in the
clause. Next, a closer analysis will be made of the use of the phrase in its functions as subject, subject
complement, prepositional complement, and object of a transitive verb. One feature that is rarely noted
is that the phrase in itself is not syntactically frozen, and examples of this will be given in section 7
below. Finally, the question whether fact is semantically meaningful or just a function word empty of
meaning will be discussed (section 8).
The results presented in the quantitative part (section 2) are based on two million-word corpora, FLOB
and Frown, the Freiburg update of LOB and Brown with 1991 and 1992 used as sampling years. A
pilot study of LOB and Brown, which contain texts that antedate FLOB and Frown by 30 years,
showed results that were approximately the same. This makes it reasonable to assume that the figures
are fairly representative, at least of these two varieties of English, but possibly also of English in
general. The three sections that discuss the fact that in subject position, as subject complement, and as
the complement of a preposition are likewise based on FLOB and Frown, whereas the three sections
that deal with the fact that in object position and the syntax and semantics of the phrase have culled
235
examples not only from these two corpora, but also from a number of British and American newspaper
and news broadcast corpora. The majority of examples are from written texts, but in the later sections, a
few examples of spoken English are included.
2. Survey of the functions of the fact that in the clause
The two corpora on which the first part of this study is based, the Frown corpus of American English
and FLOB which is a similar corpus of British texts, are rather small by comparison with some of the
corpora in use today, but since they contain a variety of texts representing many different text
categories, it can be assumed that they reflect general tendencies in the language fairly well. The total
number of the fact that was somewhat higher in the British corpus (121 in FLOB as compared to 87 in
Frown), but when it comes to syntactic function of the fact that, the results are similar (figure 1). Thus,
in close to half of the sentences in both corpora, the fact that is the complement of a preposition,
something that is reflected in grammatical descriptions of this feature, since we saw above that this is
the context in which the fact is most frequently mentioned in grammars today.
The second most common function of the fact that is that of subject; however, almost equally frequent
is the fact as the head of an object clause, something that is all but ignored in grammatical descriptions.
Syntactic accounts generally classify that-clauses as nominal clauses, which means that their function
is similar to NPs (e.g. Quirk et al. 1985:1048). It is therefore curious that a subset of the transitive verbs
that occur with the fact + a that-clause will take a that-clause only as the complement of a head noun
(see below, section 6). The least common grammatical function is that of subject complement. It should
be kept in mind that the figures are based on written corpora, and that they might be somewhat
different in a corpus of spoken English.
3. The fact that as subject
Clauses in subject position are often felt to be awkward in English, since they violate the principle of
end-weight, which requires that “the part of the sentence following the verb should be as long as, and
preferably longer than, the part that precedes the verb” (Quirk et al. 1985:1040). This is normally
resolved by extraposition of the clause, which means that the clause is moved to the end of the sentence
and it takes its place as the subject, so that (1b) is generally preferred to (1a):
(1) a. That the cable networks and the advertisers that found them will continue to encourage the trend is far
from certain.
b. It is far from certain that the cable networks and the advertisers that found them will continue to encourage
the trend.
If the that-clause is an appositional clause, however, extraposition is blocked, since it cannot be used in
subject position when the extraposed constituent is an NP (2):
(2) * It is a real advantage the fact that Harry doesn't have a cart of his own.
There are certain conditions under which a subject the fact that can move to clause-final position. This
happens for instance when the predicate phrase consists of a clause-initial adjective (often in the
comparative) plus the copula, as in (3a). The participle of a verb may also occur in clause-initial
position, in which case auxiliary be follows the rest of the verb phrase (3b):
Figure 1. Survey of the functions of the fact that in
the clause (in percentages)
0
20
40
60
Subject Subj compl Object Prep compl
Frown
FLOB
236
(3) a. More worrying, I fear, is the fact that most people will have terrible trouble just understanding what
Condren is trying to argue. (Frown J58 144)
b. Complicating the issues of social class is the fact that in the United States there is a large overlap between
lower-middle-class, working-class, and lower-class membership (Frown J49 175)
As can be seen from table 1, clause-final subjects are generally longer than clause-initial subjects, but it
is still striking that when the fact that is in its normal subject position, it often “outweighs” the
predicate part of the clause, as in (4), where the subject consists of 24 words and the verb phrase of
only 3 (see also example (16a,b) below).
(4) The fact that she is still somewhat tentative in the role and that her command of English is rather less secure
than her
arabesques, are minor blemishes. (FLOB C11 125)
Table 1. Survey of position and length of subject the fact that in Frown and FLOB
Frown FLOB
Clause-initial subject
No. of sentences 19 21
Length 7 – 34 words 6 - 24 words
Average length 15 words 13 words
Clause-final subject
No. of sentences 11 3
Length 17 – 41 words 14 – 35 words
Average length 31 words 21 words
It is often the case, however, that these clauses have long and information-loaded predicates in addition
to the long subjects, as witness (5), which has a subject part consisting of 34 words, and a verb phrase
containing as many as 37 words.
(5) The fact that this is still going on, in both this and related semidetached systems (the term comes from the
fact
that only one of the stars is in contact with the critical surface), means that accretion flows onto the
companion have played a role in the orbital dynamics and that this has fed back into the stellar evolution
through the alteration of the mass and boundary conditions on the stars. (Frown J07 64)
The explanation for the reversal of weight in sentences such as (4) most likely has to do with
information structure, so that even if the subject in a sentence like (4) contains several items of new
information, they are presented as “facts”, i.e. as something given that cannot be questioned by the
addressee. It is interesting to note that an initial subject that-clause will also be understood factively,
i.e. presupposed to be true, even when it is not preceeded by the fact. This is shown in (6), where only
(6c) presupposes that Smith had in fact arrived (see Kiparsky & Kiparsky 1970:167f, Granath
1997:36f).
(6) a. The UPI reported that Smith had arrived.
b. It was reported by the UPI that Smith had arrived.
c. That Smith had arrived was reported by the UPI.
In conclusion, when the fact that functions as the subject of the clause, the principle of end-weight no
longer applies, so that we may find long and heavy elements with a high information load at the
beginning of a sentence, contrary to the normal place of such elements in English. This is confirmed by
the results quoted in Mair (1988:65), who found that there is a tendency to have the fact that rather than
only that in English subject clauses that have not undergone extraposition. It seems that the fact may
serve as a structural marker of a long constituent, and that it will signal to the listener/reader to expect
the verb phrase to appear late in the clause. As such, it serves a communicative purpose, and omission
of it would enhance the complexity of the sentence and make it more difficult to comprehend, as is
shown in (7a) and (7b).
(7) a. That patients still have to wait 21 weeks to get into hospital for treatment and 12 weeks for day surgery
demonstrates yet again that the health service has not got enough money to treat patients in a reasonable
time.
b. The fact that patients still have to wait 21 weeks to get into hospital for treatment and 12 weeks for day
surgery demonstrates yet again that the health service has not got enough money to treat patients in a
reasonable time." (FLOB A14 192)
237
4. The fact that as subject complement
The least common function of the fact that is that of subject complement. Only one instance was found
in Frown, compared to four in FLOB. Not only is the fact that rather rare in this function, it also seems
that semantically, the fact serves only to reinforce the statement somewhat, and that the message is
communicated almost as forcefully if it is omitted. The reader is welcome to test this on the following
two examples from FLOB (my slashes).
(8) a. “the most obvious encumbrance on this picture is /the fact/ that it is woefully late” (FLOB G49 140)
b. Another component of her success was /the fact/ that she rated well under both the RORC and American
CCA rules of the time. (FLOB E18 23)
Considering the low number of sentences in which the fact that was used in this function, it seems
likely that this construction is less frequent than that-clauses (without an introductory fact) as subject
complement.
5. The fact that as the complement of a preposition
Standard grammatical descriptions of English stress that prepositions cannot govern that-clauses in
English (see e.g. Svartvik & Sager 1996:331, Hasselg&#229;rd et al. 1998:349, Quirk et al. 1985:658f).1 This
means that some other syntactic device is needed to link clauses to prepositional phrases. One such
device is “to use an appositive construction with a ‘general’ noun such as fact” (Quirk et al. 1985:659).
Both in grammars and in English usage guides, this usage is referred to as “clumsy” and “superfluous”,
and the language user is told to avoid it if possible (for examples of how this can be done, see (9) and
(11a) below). The advice from handbooks is usually simply to leave out the sequence preposition +
fact, or in the case of the two prepositions despite and due to, to replace them with a subordinator such
as although or because. However, evidence from the corpora shows that this is rarely an option for the
user: in most cases where this structure is used, no other alternative exists.
In Frown, 40 out of 87 sentences with the fact that were of the structure preposition + the fact. The
number was even higher in FLOB, so that this structure was found in 65 out of 121 sentences. This
means that this is by far the most common use of the fact that. In each corpus, there were only a
handful of cases where the writer would have had the choice of leaving out the preposition + the fact.
This was for instance the case in the four sentences where the preposition was the complement of an
adjective (content with, grateful for, oblivious of/to, responsible for) and it was possible with two of the
four nouns postmodified by preposition + the fact that in Frown and FLOB: recognition of, concern
about. However, only in two cases would it have been possible to leave out preposition + fact as the
complement of a verb, namely comfort oneself with and hint at; in all other cases an ungrammatical
construction would have been the result.2 Some examples of sentences where omission of the prepphrase
would have been possible are given in (9) (my slashes).
(9) a. I am grateful /for the fact/ that we know so little about his life. (FLOB G08 202)
b. In reality, its merger with another organization was recognition /of the fact/ that it carried “high
negatives” in public opinion polls. (Frown D09 202)
c. In the closing pages of the text, his attempted self-renunciation hints /at the fact/ that to abolish the
reflective ego is to cease to be a human subject (FLOB J62 166)
In about one fifth of the sentences, the fact was the complement of a preposition with no governing
adjective, noun or verb. The most common among these were passive sentences, where the fact that
took on the function of agent (16 out of a total of 105 sentences). Mair (1988:64f) interprets this as an
application of the principle of end-weight, i.e. that long constituents are moved to the end of the clause.
Again, it does indeed seem as if the language user sometimes has a choice, so that transforming (10a)
into an active clause (10a') does not really produce a more complex structure. In other cases, the
principle of given and new in the functional sentence perspective makes the passive stylistically the
only viable alternative (cf. 10b and 10b').
1 This rule does admit a number of different exceptions – indeed far more than might be expected – so that among the
prepositions discussed in this section, except, given and than can indeed take that-clause complements. Syntacticians normally
get around this problem by reclassifying these words, so that except that is regarded as a complex subordinator, besides and than
are given a double classification (as a preposition and as a subordinator), and given is referred to as a marginal preposition (e.g.
Quirk et al. 1985:667) precisely because it may indeed precede a that-clause. Further information about that-clauses as
complements of prepositions can be found in Christophersen (1979) and Seppänen & Granath (forthcoming).
2 See Granath 1997 for a detailed account of the omission of prepositions in verb complementation.
238
(10) a. The value of the posters must have been enhanced by the fact that they were created not by an industry
but by an individual. (FLOB E35 213)
a'. The fact that they were created not by an industry but by an individual must have enhanced the value of
the posters.
b. This vulnerability is underlain by the fact that in 1988 66 per cent of goods and services were exported.
(FLOB J42 33)
b'. The fact that in 1988 66 per cent of goods and services were exported underlies this vulnerability.
Other prepositions found with the fact that in the two corpora (in order of frequency) were despite, due
to, given, than, aside from, beside(s), and except for. Of these, besides (but not beside), except and
given (and more marginally than) will allow a following that-clause with no mediating noun (11a).
With the majority of prepositions complementing verbs and phrases of various kinds, this possibility
does not exist (11b,c).
(11) a. In the year 600 the civilizations of Spain and Mexico were roughly comparable except /for the fact/ that
the former had profited from the invention of the wheel. (Frown N13 55)
b. … much of this stemmed from the fact that he rarely took on the more general questions of the
organization (Frown J60 116)
c. I didn't make anything of the fact that we didn't have a particularly good time in bed. (FLOB K21 283)
In summary, out of the 105 sentences where the fact that was the complement of a preposition, there
were only 15 altogether where it would have been possible to leave out preposition + fact. Another
consideration is of course what will happen to the meaning of the sentence if this is done. Removing
the fact that will certainly take away some of the communicative force of the sentence, which can be
seen e.g. in the examples in (9) and (11a).
6. The fact that as direct object
It has already been pointed out above (section 2) that that-clauses are traditionally classified as nominal
clauses and thus put on a par with noun phrases syntactically. We would therefore expect that when the
fact that serves as the object of a transitive verb, a that-clause on its own would do, even if it would be
less emphatic than a construction with the fact. However, when one begins to consider corpus evidence,
it is clear that this is not at all the case. There are several issues that deserve description here. First,
even a very brief search of one of the big corpora will turn up quite a large number of the structure verb
+ the fact that, which means that this is a common structure in English. Furthermore, it is one that is
generally ignored in grammars. Second, whereas there are a number of verbs where the fact can be said
to be optional syntactically, there are others with which a that-clause complement is rare, and yet a
further group where this possibility does not seem to exist at all. It is possible that this area of English
syntax has been neglected precisely because it has been impossible to investigate the patterns in which
these verbs occur without access to large corpora.3
For this part of the paper, several CD-ROMs of British and American newspapers and broadcast news
were used. A pilot study of some of these quickly turned up more than 250 verbs followed by the fact
that. In order to delimit the study somewhat, the results in this section are primarily based on the 120
verbs of this type found in The Guardian/The Observer 1999. Altogether, this corpus comprises
approximately 40 million words. Other corpora were consulted to determine whether the fact could be
considered to be optional or obligatory. It is possible that further corpus searches will revise the results
somewhat, so that some of the verbs in group 3 (verbs that obligatorily require an NP head of the
(appositional) clause) will have to be moved to group 2 (verbs where a that-clause complement is an
alternative to the fact that).
The first group consists of verbs that occasionally take the fact as an object head of a that-clause, but
where the predominant structure is for the clause to follow the verb directly. This applies to roughly
one fourth of the 120 verbs in the survey (table 2). One interesting aspect is that when the Collins
Cobuild Dictionary entries for the 120 verbs were checked, the 28 verbs in this group were all
3 The corpus that Mair (1988) used for his paper comprised about 840,000 words, which was a large corpus at the time, but it is
small by today's standards. It is noteworthy that of the nine verbs for which Mair claims that the plain that-clause is rejected
(give, face, ignore, bring home, disguise, resent, obscure, raise, discuss) (Mair 1988:65), only two have been found not to take a
plain that-clause in the present study, namely obscure and raise. This underscores the value of large corpora, since native speaker
intuition cannot be trusted. Some speakers may very well reject what other speakers say, and it is impossible even for trained
linguists to be able to imagine all the contexts in which a word may be used.
239
categorised as taking a “report-clause”, i.e. a that-clause, whereas the remaining 92 verbs were only
marked as taking NP objects.
Table 2. Transitive verbs for which that-clause complementation is the normal pattern and the fact that less often used
-----------------------
accept consider forget mind recognise stress
acknowledge dispute grasp note regret take (into account)
add emphasise illustrate notice report volunteer
appreciate establish lament proclaim reveal
believe explain mention publicise spot
-----------------------
In the second group of verbs, two types can be distinguished: for a few verbs the fact that occurs with
almost the same frequency as a plain that-clause, but for the majority of verbs, the fact that is in fact
the predominant pattern, with the plain that-clause a rare alternative. The reason why these two types
have been collapsed into one group is that even the relatively large corpora used as the basis of this
survey are too small to establish with any certainty how these verbs should be divided up. Thus reflect
(in the sense ‘mirror') occurs 46 times with the fact that and 6 times with a plain that-clause (12a, b);
face occurs once with a that-clause, 13 times with the fact that, but in addition to that, it occurs with a
number of other NP heads, such as accusations, allegations, claims, complaints, criticism, the
possibility, the realisation (12c, d). Other verbs, such as like, love, and hate were extremely rare with
that-clause complements, and the examples that were found of this structure stem from spoken corpora.
(12) a. The book looks old-fashioned. … Perhaps this reflects that the hey-day of the circus in Britain is long
gone. (The Guardian, 17 April 1999, p. 9)
b. The Bath course records high scores in the value-added measure of academic teaching. This reflects the
fact that a high proportion of students with low entry qualifications are graduating with upper seconds and
first-class honours. (The Guardian, 9 Nov 1999, p. 10)
c. ‘I think it is a damning indictment of our society that my daughter is now having to face that you cannot
trust any word spoken or written by a politician, let alone a secretary of state or prime minister. (The
Guardian, 21 Aug 1999, p. 2)
d. It is scary enough to face the fact that you yourself are faking it. (The Guardian, 30 Jun 1999, p. 8)
Table 3 lists the verbs that were found to take predominantly the fact that rather than the plain thatclause.
This group comprises close to half the 120 verbs, or 51 in all. It needs to be stressed that for
some reason, the possibility of using a that-clause complement is not indicated in Collins Cobuild
English Dictionary, even though this is one of the first dictionaries to be based on a large corpus,
namely the enormous Bank of English.
Table 3. Transitive verbs with which the fact that predominates but which occasionally take a that-clause complement
----------------------
absorb celebrate disguise can't help neglect rumble
advertise change disregard hide omit sell
alter cite dread highlight overlook can't stand
avoid conceal enjoy ignore query take (into account)
belie cover up escape include reflect uncover
bemoan credit expose like reinforce underline
bring home, into criticise face love relish underscore
focus, out (into deplore handle mark resent welcome
the open) discuss hate mask respect
----------------------
The group of verbs that poses the greatest problems to explain in terms of syntactic behaviour is the
third group, presented in table 4. In the material consulted for this study, these verbs were found to
occur only with a that-clause as the apposition of a head noun. The fact was obviously found with all
these verbs, but for some of them, a number of other head nouns were also found, so that dismiss, for
instance, was recorded with well over 20 different NPs, such as speculations, accusations, hints,
rumours etc. This group consists of 41 verbs, that is approximately one third of the verbs studied, but it
should be remembered that further research will probably find that at least some of them will indeed
also take plain that-clause complements.
240
Table 4. Transitive verbs which only take a that-clause complement with a mediating NP such as the fact
----------------------
address cloud dodge invent obscure promote
answer (inanimate concern drop italicise outweigh raise
subject) confront duck keep secret overcome rue
begrudge decry exploit laud overshadow share
blame defend forgive miss pardon splash
camouflage dislike hold sth against negate pinpoint square
cloak dismiss illuminate obfuscate plug withhold
----------------------
It is difficult to account for the reasons why these verbs should behave differently from the verbs in
table 2. One possibility is that semantics has something to do with it. Language often works by
analogy, so that words that have the same meaning tend to adopt similar syntactic patterns. It is
apparent that many of the words are semantically close in meaning, so that we find groups of verbs that
are converse terms, such as conceal/disclose, praise/condemn, welcome/regret, admit/deny, like/dislike,
support/question etc. If meaning could be used to explain syntactic structure, we would expect that
verbs with a similar meaning should behave identically syntactically, but this is not the case. Thus, in
group 2 (verbs that take both the fact that and a plain that-clause) we find conceal, hide, and disguise,
whereas in group 3 (verbs that do not take a that-clause complement) we find camouflage, cloak, cloud
and obscure. One possibility that this suggests is that frequency matters, so that less frequent verbs
more rarely take plain that-clauses. The scope of this paper has not made it possible to look more
carefully into this, but even a quick look at table 4 indicates that at least some of the verbs are not at all
uncommon in English, for instance blame, concern and defend (example 13). The reader is welcome to
try to verify whether it is possible to omit the fact in these three sentences.
(13) a. ‘... She shouldn't blame the fact that she's a woman, or me,’ he gabbles, with more detail than is strictly
necessary. (The Guardian 31 jul 1998, p 2)
b. Mr Clinton's most serious legal problems concerned the fact that he and Ms Lewinsky gave conflicting
stories about how and why she gave back the presents he had given her (The Guardian 21 Aug 1998, p 13)
c. But Bassay asserts that classroom application should not be the overwhelming criterion for research and
defends the fact that academics make so much use of it. (The Guardian, 17 Mar 1998, p 6)
The present paper can only begin to outline the problems encountered in the analysis of these verbs.
The problem appears to call for a syntactic rather than a semantic explanation, since a number of the
verbs in group 3 will take a plain that-clause complement in other languages, such as Swedish. The fact
that the structures are in some sense idiosyncratic in English is another reason why it is important to
have a fuller account of these verbs than has so far been the case: there is at present no place where
learners can find out about the erratic behaviour of these verbs, since not even dictionaries account
fully for their colligations.
What we are left with at present is a system that in Matthews’ terms is not codified but rather an area of
partial codification, for which there are no explicit rules (Matthews 1981:20-21). To a syntactician, this
is of course not the ideal situation, nor is it a happy situation for the language learner. But, as corpus
linguists, working with actual language use and not the ideal speaker-hearer, we will probably have to
accept that there are areas of a language that cannot be wholly explained in terms of one system or
another. Still, the topic of this section, the fact that as the complement of transitive verbs, is an area
where more research is needed both to provide a fuller account of this usage for learners, and in order
to see if in the end perhaps it is possible to find at least some determining factors that can explain what
to us looks like largely idiosyncratic behaviour.
7. Internal syntactic variation in the phrase the fact that
One aspect of language that has become more obvious to linguists after the introduction of corpusbased
research is that speakers only partly have an open choice when it comes to selecting vocabulary,
syntactic forms etc. Much of what we say actually consists of pre-fabricated or semi-preconstructed
chunks of words, which work together in phrases. This is usually referred to as the idiom principle
(Sinclair:1991 110-115). A typical feature of idioms is their frozenness, i.e. lack of syntactic variation
in number, tense, voice etc. It seems that the fact that would serve well as an example of this, since all
mention of the phrase includes the definite article, the subordinator that, and fact in the singular.
However, even though this is indeed the predominant structure of the phrase, some examples will
demonstrate that it is in fact not invariable.
241
Although the definite article is by far the most common determiner, this and that occur occasionally,
both when the that-clause is restrictive, as in (14a and b), and when it is non-restrictive, as in (14c):
(14) a. We haven't tried to hide this fact that, come November, we would have banned them ourselves. (The
Observer, 19 Sept. 1999, p. 1)
b. He kept this up in meetings and letters, despite that fact that we were never together in Paris. (The
Guardian, 21 July 1998, p. 16)
c. Despite this fact, that language is so much more than just a huge store of isolated words, what you
remember of learning your own native language is likely to be limited to memories of just that – the learning
of new words. (Heny 1998:190)
Examples where the indefinite article is used are rare. Instead we find a fact that in the phrase to know
for a fact that, where the that-clause must be interpreted as the object of the verb know. An extraposed
that-clause also often follows a fact in clauses of the type It is a fact that (…). In neither of these cases
is the that-clause appositional. What we find instead is that a fact is often followed by a relative thatclause.
The only example in the present investigation where the that-clause could be taken to be an
appositional clause with the head noun a fact was actually one where fact was post-modified by a
relative clause:
(15) The clash between Schroder and Blair also underscored a fact that yesterday's Florence summit of the centre
left had been intended to disguise: that there remains a huge gulf in thinking between Britain's post-
Thatcherite Labour party and America's post-Reaganite Democrats on the one hand and the socialist and
social democrat parties on the continent on the other. (The Guardian, 23 Nov. 1999, p. 2)
An alternative analysis of the clause structure is to regard the that-clause as the direct object of the verb
underscore, which, as was shown above (table 3), belongs to the group of verbs that may take a thatclause
with no mediating noun. From a stylistic point of view it also seems logical that the determiner
of fact is definite, since in the majority of cases it refers to something presupposed and thus given
information.
As regards the number of fact, the singular by far outnumbers the plural, for the simple reason that in
general, only one “fact” is mentioned. Occasionally, we find the plural facts with appositive
postmodification, something that is also noted by Quirk et al. (1985: 1261). This obviously occurs only
when there is more than one appositive clause. However, with two or more appositive that-clauses, it is
possible to use either the singular fact or the plural facts, as witness the examples in (16), where (a) and
(b) contain that-clauses in apposition to the subject, and the ones in (c) and (d) belong to the object of
the clause:
(16) a. The fact that England did not even practice penalties, and that Hoddle characteristically refused to
accept this as an error, left the coach looking conceited and complacent (The Observer, 4 Oct 1998, p. 5)
b. The facts that the roads carried 81 per cent of all freight last year and that 86 per cent of passenger miles
are taken in cars and vans cut no ice. (The Observer, 21 Nov. 1999, p. 28)
c. He ignores the fact that Geffen reputedly offered her $1million for it, and that her previous album Pretty
On The Inside was a powerful musical landmark two years previously. (The Observer, 5 jul 1998, p. 7)
d. But such notions ignore the facts that Ramsay has already made three short films and that two of them
have won prizes at Cannes. (The Guardian, 14 Aug. 1999, p. 4)
Finally, the language user also has the option of omitting the subordinator that after fact. One could
assume that that would be left out more often in informal than in formal contexts, but corpus data does
not confirm this (cf. (17a and b), both of which contain the phrase lament the fact, where that is omitted
in the longer example from the business section (17b)).
(17) a. He laments the fact that you can't trust politicians these days. (The Guardian, 1 Nov 1999, p. 12)
b. Fellow Tory Eric Forth, a former minister, lamented the fact the Opposition was ‘conniving with his
government’ to pass the legislation (The Guardian, 26 Oct 1999, p. 29)
A cursory glance at the sentences where that is omitted after fact indicates that this happens more
frequently when the embedded clause has a pronoun subject, but it is not restricted to this type of
subject, as is shown in (18). Neither does there seem to be a limitation on that-omission due to the
function of fact in the clause, so that we find it both when the fact is a subject (18a), an object (18b),
and a prepositional complement (18c).
(18) a. The fact the President must step down in 2000 … does not matter. (The Observer, 16 May 1999, p. 22)
b. These examples of Germany's innovative and corporate success cannot, however, disguise the fact the
economy inherited by Gerhard Schroder from Helmut Kohl has serious structural problems (The Guardian,
10 June 1999, p. 19)
242
c. After he was discharged the police asked him ‘to keep quiet about the fact a tank had crushed students …
(The Guardian, 2 June 1999, p. 12)
In conclusion, corpus evidence shows that the fact that is not the invariable phrase that it appears to be
in grammatical descriptions and according to information given in dictionaries.
8. The semantics of fact
The seminal article FACT by Kiparsky & Kiparsky (1971) presented the idea that certain predicates are
factive, meaning that their complements are presupposed to be true. One criterion used for testing
factivity was to negate the matrix verb. The predicate was classified as factive if the presupposition
remained intact under negation unless explicitly contradicted (Kiparsky & Kiparsky 1971:351-52,
Leech 1974:301-317). Thus I'm sorry that he lost his job and I'm not sorry that he lost his job both
presuppose that he lost his job. In a sense then, such predicates can be said to be open to objective
verification, that is, one should be able to show that they are facts. According to Kiparsky & Kiparsky
(1971:145), “[o]nly factive predicates can have as their objects the noun fact with a gerund or thatclause.”
Typical examples of factive predicates are regret, grasp, take into account, ignore, mind, and
deplore. Predicates such as these make up the majority of the verbs in section 6 above, at least in the
two groups that predominantly occur with the fact that. We also saw that that-clauses in subject
position tend to be given a factive reading whether or not the fact is overtly stated. Therefore, one
major conclusion to be drawn from the present investigation is that the fact is indeed used because the
appositional clause is presupposed to be true.
More interesting, perhaps, is whether fact is losing some of its meaning, and whether it is sometimes
used for syntactic reasons alone. Even Kiparsky & Kiparsky (1971:147fn) refer to a colleague who had
informed them “that for him factive and non-factive predicates behave in most respects alike and that
even the word fact in his speech has lost its literal meaning and can head clauses for which no
presupposition of truth is made.” Such usage of fact is what language guide books react against, so that
a sentence such as I certainly do not accept the fact that Sir Patrick was remotely influenced by the
timing of the leadership election (Radio 4) receives the comment that “[i]f someone does not accept a
fact, then in their eyes it is not a fact” (Blamires 1998:117). The author goes on to suggest that
suggestion should replace fact in this case. The way fact is described in grammars, e.g. by being
referred to as a marginal subordinator (Quirk et al. 1985:1001-02) also indicates that the word has lost
much of its original meaning, and that today, it can be used as a function word devoid of meaning.
What indications are there in the corpora that this is true?
Starting from Kiparsky & Kiparsky's list of non-factive verbs, we find that believe is a typical example
of such a predicate. According to their rule, then, it should never occur with the fact as an object.
Nevertheless, several examples of this construction were found (19a,b). Included here is also an
example with buy in the sense ‘believe, accept’ (19c).
(19) a. Her mother, Amanda, 24, said: ‘It's been very traumatic for her and I am just glad she's OK. But I just
can't believe the fact that she was able to get hold of something like this in the first place. (The Guardian,
15 Dec 1999, p. 6)
b. They paid a big price for that, though, and I think they paid it in the sense that Heidstra – he talks about a
Bronco-like car leaving the scene of this crime, and if Christopher Darden and the prosecution can make the
jury question the timing, but believe the fact that there's a car leaving, then his testimony has the effect of
suggesting Simpson at the scene. (CNN Morning News, 26 jul 1995)
c. Quite frankly, it is not a tax break for the rich. … It would be first dollar coverage, it would be a high
deductible, it would be very, very affordable for those people and unfortunately I just cannot buy the fact
that it is a tax break for the rich. (CNN Domestic News, 25 Apr 1996)
If we take a closer look at these examples, it is evident that (19a) expresses a fact that can be verified.
The mother's use of can't believe the fact is just her way of signalling incredulity. The second example,
(19b), might be a little more difficult to explain in terms of factuality. If you are going to make
someone believe something, this usually signifies that what you make them believe is not true.
However, the context here is the courtroom, and the fact referred to is an event that is a fact for the
prosecution, whose task it is to make the jury believe the facts they present.4 Neither of these examples
4 Greetham (1999:3) outlines three “degrees of truth” from the rhetoric of litigation: raw facts, the common ground taken as
evidence by both parties, truefacts, the facts appropriated by only one party, which “had acquired a new level of revealed truth
by being frankly partisan, and thus more true than mere facts”, and factoids, presented by the opposing party but not espoused as
facts by one's own side.
243
can therefore be explained as sentences where the fact functions as a prop word devoid of meaning.
The last example, on the other hand, is a contradiction, because the speaker states emphatically that “it
is not a tax break for the rich”, a “fact” that he immediately says he does not believe in. In this
particular case the fact is probably more of a rhetorical flourish, used to add emphasis to what the
person is saying.
It ought not to be possible to refer to things that are counterfactual (i.e. made up or imagined) as
“facts”, but (20) demonstrates precisely this. In both these cases, fact fulfils a syntactic function, since
neither lay money on nor invent can take that-clause complements.5 Thus it does seem that we have two
cases here of the fact that as a marginal subordinator.
(20) a. I'll lay money on the fact that he's found some other way to work out his aggressions. (Walters, Minette
1999 The Dark Room Pan Books, p. 447)
b. When asked whether he still works behind the bar, Martin says: 'I used to love it, but I don't do it any
more. Tim invented the fact that he worked behind the bar, but he never did. (The Observer, 19 Sep 1999,
p. 4)
Two other verbs often mentioned as typical examples of non-factive verbs are claim and assume. Claim
is a verb that signals that the complement is the speaker's subjective opinion, and assume refers to a
hypothetical condition. Consequently, it is not be possible to verify the truth of the complement of
either of these verbs (21a,b). In (21c), where the predicate is hear, it is obvious that the speaker uses
the fact to refer to something he does not believe in.
(21) a. Bobby Murcer, Mantle Teammate: Well, I didn't know Gehrig and Ruth – I just heard about those guys. I
knew Mickey Mantle, so I can honestly claim the fact that I think Mickey Mantle was the greatest. (CNN
World News, 15 Aug 1995)
b. Eli Noam: It's true for the present, but if you look at the market share of broadcasters generally, and even
assuming the fact that they will be able with new technology to squeeze more broadcast channels into their
broadcast signals, even assuming that, they still are – will be big players, but among many, many other
competitors. (NPR Morning: Business, 9 Aug 1995)
c. At Marty's Center Tap, UAW members are not optimistic about reaching a contract agreement and Mark
Samp says it will take a tremendous effort to overcome the effects of a strike, which has been terrible for
everyone.
Mark Samp: I get tired of hearing the fact that people believe that, well, you know, time heals all wounds.
(NPR Morning Edition: Domestic News, 9 Jan 1996)
It is significant that all these examples are from spoken language, and the speakers have thus not been
able to edit their utterances, which means that they could all be explained as performance errors.
However, if we assume that the speakers used the fact for some purpose, then it is noteworthy that in
none of the sentences in (21), the fact is needed for syntactic reasons: assume, claim and hear all
normally take that-clause complements. So why is the fact used here at all? I would like to suggest that
in the context, it has a certain pragmatic force, so that in the examples in both (20) and (21) the fact is
used to strengthen the force of what is said. The fact that examples such as these are found in corpora
also supports the suggestion made by Kryk (1981) that the logical analysis in terms of truth conditions
of these utterances should be replaced by a pragmatic approach, and the factive/non-factive dichotomy
by a scale of factive and “not-so-factive” predicates. Whether the fact is necessary to avoid an
ungrammatical construction, as in (20), or just added as some kind of rhetorical flourish, as in (21),
speakers make use of it in order to get a message across, a message that would certainly be less forceful
without the fact.
9. Conclusion
The present paper has attempted to demonstrate that the use of the fact that in present-day English
cannot be explained by referring to its syntactic function alone, but semantic as well as pragmatic
meaning needs to be taken into account in a description of its function in utterances. This serves as an
example to demonstrate in what way a corpus-based approach to syntax can be used to enhance
existing accounts. Traditional syntax can be amended, so that a narrow focus on structure can be
replaced by a multifunctional approach. Instead of classifying words as function words or content
words, a common procedure in introductory texts in semantics, we will have to acknowledge that very
few words act as function words only. The result is a grammar where there is a synthesis of syntactic,
semantic and pragmatic rules.
244
References
Biber D, Johansson, S, Leech, G, Conrad, S, Finegan, E 1999 Longman Grammar of Spoken and
Written English. Harlow, Pearson Education.
Blamires H 1998 The Cassell Guide to Common Errors in English. London, BCA.
Christophersen P 1979 Prepositions Before Noun Clauses in Present-Day English. In Chesnutt, M,
Faerch C, Thrane T, Caie G D, Essays Presented to Knud Schibsbye. Copenhagen: Akademisk Forlag,
229-234.
Granath S, 1997 Verb Complementation in English: Omission of Prepositions before that-clauses and
to-infinitives. Göteborg, Acta T. Universitatis Gothoburgensis.
Greetham D 1999 Facts, Truefacts, Factoids; or, Why Are They Still Saying Those Nasty Things about
Epistemology? Yearbook of English Studies 29:1-23.
Hasselg&#229;rd H, Johansson S, Lysv&#229;g P 1998 English Grammar: Theory and Use. Oslo,
Universitetsforlaget.
Heny F 1998 The Structure of Sentences. In Clark V P, Eschholz P A & Rosa A F (eds) Language:
Readings in Language and Culture. Boston and New York, Bedford/St Martin's, 189-224.
Kahn J E, Ilson R (eds) 1985 The Right Word at the Right Time: A Guide to the English Languge and
How to Use It. London, Reader's Digest Association.
Kiparsky P, Kiparsky C 1971 Fact. In Bierwisch, M & Heidolph K E (eds), Progress in Linguistics: A
Collection of Papers. The Hague, Mouton, 143-173.
Kryk B 1982 The Relation between Predicates and Their Sentential Complements, A Pragmatic
approach to English and Polish Studia Anblica Posnaniensia: An International Review of English
Studies, 14:103-120.
Leech G 1974 Semantics. Penguin Books.
Mair C 1988 In Defense of the fact that: A Corpus-Based Study of Current British Usage. Journal of
English Linguistics 21:1, 59-71.
Matthews P H 1981 Syntax. Cambridge, Cambridge University Press.
Perrin P G 1942 Writer's Guide and Index to English. Chicago, Scott, Foresman and Company.
Seppänen A, Granath S (forthcoming) That-Clauses and the Complementation of Prepositions.
Sinclair J 1991 Corpus, Concordance, Collocation. Oxford, Oxford University Press.
Quirk R, Greenbaum S, Leech G, Svartvik J 1985 A Comprehensive Grammar of the English
Language. London and New York, Longman.

245
Building a text corpus for representing the variety of medical language
Benoît Haberta, Natalia Grabarb, Pierre Jacquemartb, Pierre Zweigenbaumb
a LIMSI-CNRS & Université Paris 10
b DIAM - Service d'Informatique Médicale/DSI, Assistance Publique – Hôpitaux de Paris
& Département de Biomathématiques, Université Paris 6
Abstract
The representation of specialized domains in reference corpora does not always cater for the internal
diversity of genres. Similarly, most sublanguage studies have focussed on domain specialization, largely
leaving genre an implicit choice that received less individual attention. Specialized domains, though,
display a large palette of text genres. Medicine is a case in point, and has been the subject of much work in
Natural Language Processing. We therefore endeavored to build a large corpus of medical texts with a
representation of the main genres found in that domain. We propose a framework for designing such a
corpus: an inventory of the main genres of the domain, a set of descriptive dimensions and a standardized
encoding of both meta-information (implementing these dimensions) and content. We present a proof of
concept demonstrator encoding an initial corpus of text samples according to these principles.
Keywords:
Specialized language, genres, medicine, French, natural language processing, TEI, CES, XML, XSL.
1 Introduction
The representation of specialized domains in reference corpora does not always cater for the internal
diversity of genres. This is the case for instance in Brown and LOB, where the domain is the entry key –
the small size of these corpora also accounts for this. Similarly, domain specialization seems to have been
the focal criterion considered in all sublanguage studies performed in the eighties (e.g., (Grishman &
Kittredge, 1986)), whereas genre was largely an implicit choice that received less individual attention.
Specialized domains, though, display a large palette of text genres. This is the case of medicine.
Medical narratives, including discharge summaries and imaging reports, have been the most studied type of
text (Sager et al., 1987; Friedman, 1997; Rassinoux, 1994; Zweigenbaum & Consortium MENELAS, 1994).
Short problem descriptions, such as signs, symptoms or diseases, have been the subject of much attention
too, in relation to standardized vocabularies (Tuttle et al., 1998). Some authors have also examined
abstracts of scientific literature (Grefenstette, 1994). And indeed, web pages are today the most easily
available source of medical documents. These documents vary both in form and in content; it has even been
showed that within a single document, subparts can consistently display very different language styles
(Biber & Finegan, 1994). The natural language processing (NLP) tools that have been tailored for one
document type may therefore be difficult to apply to another genre (Friedman, 1997)1. All these genres are
found in the same overall domain: medicine. A large palette of genres can also be found within each
medical specialty: domain and genre are clearly distinct factors of textual variety.
This diversity has consequences for the design and development, or simply for the use, of natural
language processing tools for medical information processing. Without better informed knowledge about
the differential performance of natural language processing tools on a variety of medical text types, it will
be difficult to control the extension of their application to different medical documents. We propose here to
provide a basis for such informed assessment: the construction of a corpus of medical texts. We address this
task for French language texts, but we believe the same reasoning and methods and part of the results are
applicable to other languages too.
This text corpus must be useful for testing or training NLP tools 2. It must provide a variety of medical
texts: diversity must be obtained in addition to mere volume, since our specific aim is to represent the many
different facets of medical language. We need to characterize this diversity by describing it along
1The precision of French taggers evaluated within the framework of GRACE (Adda et al., 1999), measured in relation to a manually
tagged reference corpus, similarly shows significant variations depending on the part of the corpus under examination (Illouz, 1999).
This corpus containing 100,000 words has been compiled from extracts from Le Monde (2 extracts), and from literary texts: memoirs
(2 extracts), novels (6 extracts), essays (2 extracts). Thus an extract from memoirs results in important variations, positive and
negative, among the taggers.
2Taggers, shallow parsers which are able to build syntactic representations for any kind of text, named entities recognizers, etc.
246
appropriate dimensions: origin, genre, domain, etc. These dimensions have to be documented precisely for
each text. This documentation must be encoded formally, as meta-information included with each
document, so that sub-corpora can be extracted as needed to study relevant families of document types.
Finally, text contents must also be encoded in a uniform way, independently of the many formats
documents were written in originally.
We present here a framework for designing a corpus of medical texts representing genre variety: a set
of genres and descriptive dimensions, inspired in part from previous relevant literature, a standardized
encoding of both meta-information (implementing these dimensions) and content, using the TEI XML
Corpus Encoding Standard (Ide et al., 1996), and an initial set of text samples encoded according to these
principles. This work takes place in the context of a larger corpus collection initiative, project CLEF
(www.biomath.jussieu.fr/CLEF/), whose goal is to build a large, diversified corpus of French texts and to
distribute it widely to researchers.
After a brief review of related work (section 2), we explain in turn each of the main phases of the
design of our corpus: (i) assessing document diversity, choosing dimensions to characterize this diversity,
and implementing them in a standard XML DTD (section 3); (ii) selecting the main classes of documents
we want to represent and documenting them with these dimensions, then populating the corpus with texts
(section 4). We also explain how sub-corpora can be extracted from the corpus (section 4.3).
2 Taking into account corpus state of the art
The evolution of corpus techniques and standards in the past ten years makes it difficult to reuse
existing (medical) corpora.
The development of standards for the encoding of textual documents has been the subject of past
initiatives in many domains (electronic publishing, aeronautics, etc.), using the SGML formalism, and now
its XML subset. The Text Encoding Initiative was a major international effort to design an encoding
standard for scholarly texts in the humanities and social sciences, including linguistics and natural language
processing. It produced document type definitions (DTDs) which have been complemented with a Corpus
Encoding Standard (CES) (Ide et al., 1996). The CES DTD is therefore the natural format for encoding a
corpus that is targeted at NLP tools.
Beyond bibliographic description, descriptive dimensions for characterizing text corpora have been
proposed by Sinclair (Sinclair, 1996) and Biber (Biber, 1994) among others. A related strand of work is that
around the standardization of meta-information for documenting web pages (Dublin Core Metadata
Initiative, 1999); but this covers more limited information than that we shall need. In the medical
informatics domain, the standardization efforts of bodies such as HL7 (Dolin et al., 2000) and CEN (Rossi
Mori & Consorti, 1999) focus on clinical documents for information interchange: both their aim and
coverage are different from ours.
The available medical corpora we are aware of do not match the criteria underlying current standards.
Firstly, medical textbooks and scientific literature have been collected in project LECTICIEL (Lehmann et
al., 1995) for French for Special Purposes learning. A set of software tools were available to study the
various parameters of documents from the corpus (lexical choices, grammatical connectors, text
organization: titles, parts...). Users could add new texts to the database and compare them with the existing
sub-corpora. The encoding standard however is an obsolete one and the resulting corpus far too small by
our current expectations. Secondly, one medical corpus was specifically built for the purpose of linguistic
study: MEDICOR (Vihla, 1998). Although its focus is on published texts (articles and books), with no
clinical documents, it is an example of the kind of direction that we wish to take. Unfortunately the initial
version of the corpus provides limited documentation about the features of each document (intended
audience, genre and writer qualification), which is planned to be extended. Thirdly, even though very large
collections of medical texts indeed exist within hospital information systems – the DIOGENE system being
among the earliest ones (Scherrer et al., 1996) – the issue here is that of privacy and therefore
anonymization, to which we return below.
247
3 Identifying and representing diversity dimensions
3.1 Assessing variety dimensions
In our opinion, a corpus can only represent some limited subsets of the language, and not the whole of
it. No corpus can contain every type of language use3. It is even true in specialized domains such as
medicine or computer science. As a matter of fact, studies of sublanguages favored until recently very few
types of textual documents (see above), “hiding” the variety of registers within each of these sublanguages.
In order to gather a corpus, one must explicitly choose the language use(s) (s)he wants to focus on.
One must identify the main underlying dimensions of diversity which are responsible for the major
contrasts within the linguistic area (s)he wants to analyze. The variety factors are twofold: external and
internal. External variety refers to the whole range of parameter settings involved in the creation of a
document: document producer(s), document user(s), context of production or usage, mode of publication,
etc. This issue is thoroughly addressed in (Sinclair, 1996) and (Biber, 1994). It is rather straightforward to
describe documents according to these lines. However, internal variety must as well be taken into account.
It follows from the range of registers corresponding to the main communicative tasks of the linguistic
community. Indeed informants in a specific domain such as medicine have intuitions about the major
relevant registers for the domain, even if they do have difficulties in establishing clear-cut borderlines.
(Wierzbicka, 1985) relies on folk names of genres (to give a talk / a paper / an address / a lecture / a
speech) as an important source of insight inside communicative characteristics of a given community. It has
even been shown as well that, while there is no well-established genre palette for Internet materials, it is
nevertheless possible, through interviewing users of Internet (students and teaching staff in computer
science), to define genres that are both reasonably consistent with what users expect and conveniently
computable using measures of stylistic variation (Dewe et al., 1998)4. This is why the very first step
consists in asking people from the domain the main communicative routines or “speech acts” they identify.
We started thus from a series of prototypic contexts, and listed the types of texts related to these starting
points: medical doctor (in hospital or in town), medical student, patient (consumer); patient care, research;
published and unpublished documents.
It is now possible to restate more precisely what we mean by variety : a domain corpus should
represent the main communicative acts of the domain and their parameter settings. This analysis leads us to
slightly change Sinclair's definition of a corpus (Sinclair, 1996, p. 4) (“a corpus is a collection of pieces of
language that are selected and ordered according to explicit linguistic criteria in order to be used as a
sample of the language”): the criteria need to be situational and sociological as well (Habert, 2000). The
definition of the dominant communicative routines of the domain and of their precise situational parameters
is a pre-requisite.
3.2 Genres of medical texts
Trying to compile a complete list of the types of medical textual documents is probably a never ending
task, since new situations may lead to the creation of new document types, and since a finer-grain
examination could always reveal finer distinctions. Our aim here is rather to identify the main kinds of
medical texts that can be found in computerized form, and to characterize each of them by specifying
values for a fixed set of orthogonal dimensions related to the external and internal factors of diversity.
We considered four main contexts of production or use of medical documents (table 1).
In the context of care, medical professionals produce information about a patient. In a hospital, this
information is registered in the form of reports. Letters are a different series of genres where the receiver is
a more targeted health care professional.
In a university context, teaching involves material produced by faculty members (lecture notes as well
as test questions for examinations) and by students (student notes).
Dissemination of knowledge much resembles what can be found in other scientific disciplines. Medical
professionals read and, for some of them, write articles in various sorts of journals and conference
proceedings. Medical professionals or scientific reporters also write articles for the general public in the
3For instance, is it possible to find real-life, that is large-scale, samples of prayers or love small-talk in any existing corpus?
4The ten following genres were distingued : Informal, Private (Personal home pages); Public, commercial (Home pages for the general
public); Interactive pages (Pages with feed-back: customer dialogue; searchable indexes); Journalistic materials (News, editorials,
reviews, popular reporting, e-zines); Reports (Scientific, legal, and public materials; formal text); Other running text; FAQs; Link
Collections; Other listings and tables; Discussions; Contributions to discussions; Usenet News material; Error Messages.
248
general press or in more specialized magazines. They also report on (often yet unpublished) conference
lectures. Students also produce memos and reports, the most prominent of which is the doctoral
dissertation. Direct computer mediated discussion and exchange of information takes place in specialized
newsgroups and electronic lists, which are as pervasive in the medical area as in other technico-scientific
domains.
Production of
information on a
patient
Reports
Letters
Discharge report
Surgery report
Examination report Endoscopy ; EKG; EEG;
Anatomopathology; Imaging ;
Functional
Request for advice; Referral; Discharge letter; Additional
prescription
Teaching Course material
(professorial)
Course material
(student)
Lecture notes; Test questions
Student notes
Dissemination of
knowledge
Periodicals
Articles
Newspaper ; Magazine ; Bulletin ; Journal
Generalist press article; Scientific article; Article abstract;
Conference report
Student memo
Electronic discussion
Dissertation
News group ; Electronic list
Knowledge
resources
Reference knowledge Book; Encyclopedia; Dictionary; Monograph
Guidelines General French medical guidelines; Consensus conference;
Recommendation; Protocol
Official Bulletin officiel; Code of deontology; Convention;
Informed consent
Coding systems Terminology (Nomenclature, Thesaurus, Classification)
Table 1: A (non-exhaustive) list of genres of medical documents.
Beyond the previous items, different kinds of knowledge resources are used in the medical practice.
Stable reference knowledge is found in dictionaries and encyclopedias or in monographs (e.g., all that
needs to be known about a given drug). More operative knowledge takes the form of guidelines and
protocols which often constitute rules that medical practitioners must follow. The “Références médicales
opposables” (RMO, http://www.upml.fr/rmo/, translated here as “general French medical guidelines”) are
national rules that state which medical acts should be avoided in certain situations. A “consensus
conference” is a conference where an authoritative group of physicians agree on a statement, e.g., about the
best treatment for a disease (e.g., www.chu-rouen.fr/ssf/recomfr.html). Protocols are precise plans of
diagnosis or treatment for specific diseases, especially in oncology. Official documents regulate the legal or
contractual aspects of medical practice: the official bulletin is a legal publication of the French government;
the Code of deontology is a regulation of the medical profession; the “Convention” is an agreement
between the medical profession and the government; and the “Informed consent” warns a patient about the
potential risks and benefits related to his or her treatment. Finally, coding systems organize different types
of terminologies used for the normalized description of medical information.
3.3 Dimensions
These document types are difficult to classify into non-overlapping groups. Therefore modeling the
corpus with descriptive dimensions is all the more useful. We studied three sets of dimensions proposed in
the literature (Sinclair, 1996; Biber, 1994; Dublin Core Metadata Initiative, 1999). Most of them are
attributes useful in medical text genres, and were kept in our final selection (table 2). The objective of these
attributes is to characterize the different types of texts and each of their instances. Here again, the set
presented here is liable to revision as more documents are added to the corpus.
249
We divide these dimensions in three groups. Bibliographic dimensions are the traditional features that
characterize the origin of the document, including its links to a larger document set (e.g., article in a journal
or chapter in a book) and its status as a partial or full document. Non-textual parts are removed from our
documents, but their descriptions can be included.
External dimension: bibliographic references
Origin : Title :
Author, Coauthor, Translator, Contributor, Editor, Text
creator:
Date of creation, of translation, etc.:
Identifier:
Localization: Page, etc.
Link: None; To a series; To another text
Extract: Full; Article; Chapter; Paragraph
Description of embedded non-textual data: e.g., Radiograph, Photograph, Table
External dimension: context of production and reception
Mode of production: Typed; Dictated; Manuscript
Mode of transmission: Oral; Electronic; Printed
Software format: Plain text, html, etc.
Producer Plurality: Individual; Association; Company;
Institution
Function: Scientist, Medical professional, Scientific
reporter, Student, Terminologist, Patient,
etc.
Receiver Plurality: Unique, Multiple
Presence: Present; Absent
Profile: Medical professional; Non-medical
professional
Objective: Record; Describe; Inform; Explain; Discuss; Persuade;
Recommend; Teach; Order
Publication status: Published; Unpublished
Frequency of publication: Periodical; Punctual
Coverage: Local; National; International
Rights: Statement of ownership, Usage restrictions
More internal dimensions
Language: French
Size of text: In words, bytes, etc.
Level of style: Low; Medium; High
Quality of presentation: Raw; Revised; Advanced
Interaction with public: Distant; Neutral; Close
Personalization: Personalize; Impersonal
Factuality: Informative factual; Intermediate; Imaginary
Technicity: Non-technical; Intermediate; Specialized
Table 2: Dimensions.
The second group contains external dimensions that characterize the context of production or reception
of the documents. The mode of production corresponds to the original authoring of the text, the mode of
transmission to the form it had before inclusion in the corpus. Only the mode of transmission is considered
in (Sinclair, 1996) and (Biber, 1994). We find it useful to make a difference between the successive forms
of a document during its life cycle. The software format applies to electronic texts and helps to document
conversion work. The producer's profile includes his or her “function”. This aspect is not mentioned in
(Sinclair, 1996), although it covers the profile of the receiver (“audience constituency”). The only
250
distinction in the receiver's profile that seems relevant up to now here is whether s/he is a medical
professional or not. The “objectives” merge those of (Biber, 1994) and of (Sinclair, 1996). The publication
status corresponds to a usual distinction. The frequency of publication was introduced as a general attribute
to help differentiate periodicals from non-periodicals. “Coverage” comes from the Dublin Core (Dublin
Core Metadata Initiative, 1999): it is meant to describe “the extent or scope of the content of the resource”.
Typically this encompasses the spatial and temporal validity of the text: e.g., the national applicability of a
law or the temporal validity of a terminology. The “rights” attribute, also from the Dublin Core, is useful to
inform the corpus user about the allowed utilization of the corpus.
The last group contains more internal dimensions which can generally be detected from the text itself.
“Language” is mentioned in (Dublin Core Metadata Initiative, 1999). We introduced “size” to control
sampling policy over the corpus. The “level of style” is a usual dimension, as well as the “quality of
presentation” (Sinclair, 1996). Some of the remaining dimensions may be related to external dimensions
(e.g., “Interaction with public” usually depends on the producer's and receiver's profiles), but we consider
that they are more reflected and verifiable in text contents.
3.4 Domains
Medicine has numerous specialized subfields, each of which entertains professional, teaching and
scientific activities, with its own societies, journals and conferences. Several lists of medical specialties can
be found, among which the official list of health care professions (“Nomenclatures des professions de
santé”), that of the US National Library of Medicine's Medical Subject headings thesaurus (MeSH,
www.nlm.nih.gov/mesh/meshhome.html) and that of the CISMeF internet directory of French medical
resources (www.cismef.org, (Darmoni et al., 2000)). We relied on the latter, which synthesizes the first
two. We made some of the categories slightly more specific by ungrouping some clusters (e.g.,
“Angéiologie & cardiologie” separated into “Angéiologie” and “cardiologie”); we also removed or
specialized a few themes that seemed too far fetched (e.g., “Anthropology, Education, Sociology and Social
Phenomena”, reduced to “Education”).
3.5 Exploiting diversity dimensions: corpus and document headers
A corpus without documentation is a (possibly huge) bag of “dead words”. For that very reason, within
the TEI standardization group, much attention has been devoted to the definition of headers (Giordano,
1995). A header is a normalized way of documenting electronic texts. The corpus header caters for the
documentation for the corpus as a whole, whereas each document header contains the meta-information for
its text.
Each document in a corpus has a header. This header describes the electronic text and its source –
fileDesc or file description in figure 1– (bibliographic information, when available), it gives the
encoding choices for the text – encodingDesc or encoding description – (editorial rationales, sampling
policy...), non-bibliographical information that characterize the text, and a history of updates and changes
(revisionDesc – revision description). In the non-bibliographical part of the header (profileDesc –
profile description), the text is “tagged” according to one or more standard classification schemes, which
can mix both free indexes and controlled ones (such as standard subject thesauri in the relevant field).
These classification schemes are thoroughly described in the corpus header. It is then possible to
extract sub-corpora following arbitrarily complex constraints stated in these classification schemes. For
instance, the interface to the BNC relies on such an approach (Dunlop, 1995) and permits to restrict queries
to sub-corpora (spoken vs written language / publication date / domain / fiction vs non-fiction ...and any
combination of these dimensions).
We followed the TEI proposals and more precisely the standard TEI XML CES model (Ide et al.,
1996). We could find a mapping into the CES header for each dimension of our model, and therefore
implemented it in the CES framework. Generally, bibliographic dimensions (and document size) fit into the
fileDesc; the definition of the other external and internal dimensions is located in the encodingDesc
section of the corpus header, and each corpus document refers to it in its profileDesc section.
An added advantage is that the CES model provides some additional documentation dimensions, e.g.,
information about the corpus construction process (text conversion, normalization, annotation, etc.). The
implemented corpus is a collection of texts, each with its own meta-information: an instantiation of the
above dimensions. On top of these texts, it provides documentation on itself: on the one hand, bibliographic
251
information of the same kind as its component texts; on the other hand, meta-information about the
documents it contains. The latter comprises the definition of the descriptive dimensions along which each
of its documents is described.
Figure 1: Overall corpus form: corpus header (<cesHeader>: upper rectangle) then documents (<cesDoc>),
each containing a document header (<cesHeader>: lower, inner rectangle) and the actual <text>.
4 Building and exploiting the corpus
4.1 Giving a shape to the corpus: document sampling
Several parameters influence the overall contents of the corpus: we focus here on the types and sizes of
documents that it will include. There is debate in the corpus linguistics community as to whether a corpus
should consist of text extracts of constant size, as has been the case of many pioneering corpora, or of
complete documents. The drawback with extracts is that textual phenomena with a larger span may not be
studied on such samples. The overall strategy of project CLEF is therefore to opt for full documents as
much as possible. In some cases however, text samples may be easier to obtain: it may be more acceptable
for a publisher, because of property rights, to give away extracts rather than full books or journals. We plan
to be pragmatic about this issue.
To initiate the construction of our corpus, we selected an initial subset of text types as target population
for the corpus. As explained above, we tried to represent the main communicative acts of the domain. The
main text types we aim to represent initially include types from all the groups of genres listed above:
hospital reports, letters (discharge), teaching material (tutorials), publications (books chapters, journal
articles, dissertations), guidelines (recommendations) and official documents (code of deontology). This
will be achieved progressively; the current status is that of a proof of concept, which we describe below.
We cautiously avoided to over-represent web documents, which could bias corpus balance because of their
immediate ease of obtention. An additional interesting family of genres would be transcribed speech; but
the cost of transcription is too high for this to be feasible.
A generic documentation for each text type was prepared. The rationale for implementation is then to
encode a document header template for each text type: this template contains the prototypical information
for texts of this type. This factorizes documentation work, so that the remaining work needed to derive a
suitable document header for an individual text is kept to a minimum. Document templates were
implemented for the text types included so far in the corpus.
4.2 Populating the corpus with document instances
The addition of documents to the corpus comprises several steps. The documents must first be
obtained. This raises issues of property. A standard contract has been established for the project with the
help of the European Language Resources Agency (ELRA), by which document providers agree with the
distribution of the texts for research purposes. For texts that describe patient data, a second issue is that of
252
privacy. We consulted the French National Council for Informatics and Liberties (CNIL). They accepted
that such texts be included provided that all proper names (persons and locations) and dates be masked.
The contents of each document are then converted from their original form (HTML, Word) to XML
format. Minimal structural markup is added: that corresponding to the TEI CES level 1 DTD. This includes
paragraphs (<p>; this is marked automatically) and optionally sections (<div>). The document header
template for the appropriate document type is then instantiated. For series of similar samples (e.g., a series
of discharge summaries), most of this instantiation can be performed automatically.
Figure 2 shows a slice of the implemented corpus.
Figure 2: A slice of the implemented corpus: extracts of the document header of document 2 and first lines
of its contents (viewed with Xerces TreeViewer).
As a proof of concept, we integrated 374 documents in the corpus: 294 anonymized patient discharge
summaries from 4 different sites and 2 different medical specialties (cardiology, from project Menelas
(Zweigenbaum & Consortium MENELAS, 1994), and haematology), 78 anonymized discharge letters, one
chapter of a handbook on coronary angiography and one consensus conference on post-operative pain. The
total adds to 143 Kwords, with an average of 385 words per document. Many colleagues have kindly
declared their intent to contribute documents, so that a few million words should be attainable.
Adding new documents to the corpus and documenting them requires a varying amount of work
depending on the type of document. Patient documents require the most attention because of
anonymization. Their actual documentation also raises an issue: a precise documentation would reintroduce
information on locations and dates, so that we must here sacrifice documentation for privacy.
A pre-specified model for document description is a need if a corpus is to be used by many different
people. The dimensions of our model, implemented as taxonomic “categories”, will probably need some
update with the introduction of the other main types of documents. We expect however that they should
quickly stabilize.
4.3 Extracting sub-corpora
Adherence to an existing standard enabled us to implement our corpus model in a principled way with
a very reasonable effort. Besides, the general move towards XML observed in recent years facilitates the
conversion of existing documents and the subsequent manipulation of the corpus, which can be
253
manipulated through standard XML tools. We ran the Xerces Java XML library of the Apache XML
project and James Clark's XT library under Linux, Solaris and HP-UX. The corpus was checked for
syntactic well-formedness (“conformance”) and adherence to the xcesDoc DTD (“validity”).
We use XSL stylesheets to produce tailored summaries of the corpus contents and to extract subcorpora.
An XSL stylesheet can specify transformations that should be applied to an input XML file, here
the whole corpus. These transformations include the selection of elements of the input file (here, individual
texts) and the construction of a new document (a sub-corpus) embedding these elements. Additional
material such as a new corpus header can be built on the fly as needed. Selection can operate on any of the
features of the texts, including their documentation, so that all the previously discussed genres, dimensions
and domains can serve as criteria for extracting corpus texts. We have written a few stylesheet to performs
specific extractions. We are currently working on a generic user interface for specifying these extractions.
An important need is to keep track of the origin of the corpus elements through successive extractions
(Illouz et al., 2000).
5 Conclusion and Perspectives
We have proposed a framework for designing a medical text corpus and a proof of concept
implementation: a set of descriptive dimensions, a standardized encoding of both meta-information
(implementing these dimensions) and content, and a small-size corpus of text samples encoded according to
these principles.
This corpus, once sufficiently extended, will be useful for testing and training NLP tools: taggers,
checkers, term extractors, parsers, encoders, information retrieval engines, information extraction suites,
etc. We plan to distribute it to NLP and Medical Informatics researchers. We believe that the availability of
such a resource may fill a gap in the current corpora and help better study the issues of genres in a
specialized domain. The corpus should also allow more methodological, differential studies on the medical
lexicon, terminology, grammar, etc.: e.g., terminological variation across genres within the same medical
specialty, or the correlation of observed variation with documented dimensions, which should teach us
more about the features of medical language.
6 Acknowledgments
We wish to thank the French Ministry for Higher Education, Research and Technology for supporting
project CLEF, D Bourigault and P Paroubek of project CLEF's management board for useful discussions,
B Séroussi and J Bouaud for help about the document genres, and the many colleagues who agreed to
contribute documents to the corpus.
Bibliography
Adda G, Mariani J, Paroubek P, Rajman M, Lecomte J 1999 Métrique et premiers résultats de
l'évaluation GRACE des étiqueteurs morpho-syntaxiques pour le français. In Amsili P (ed),
Proceedings of TALN 1999 (Traitement automatique des langues naturelles), Carg&#232;se, ATALA, pp
15–24.
Biber D 1994 Representativeness in corpus design. Linguistica Computazionale, IX-X:377–408. Current
Issues in Computational Linguistics: in honor of Don Walker.
Biber D, Finegan E 1994 Intra-textual variation within medical research articles. In Ooostdijk N, de Haan
P (eds), Corpus-based research into language, number 12 in Language and computers : studies in
practical linguistics. Amsterdam, Rodopi, pp 201–222.
Darmoni S. J, Thirion B, Leroy J. P, Douy&#232;re M, Baudic F, Piot J 2000 CISMeF: a structured health
resource guide for healthcare professionals and patients. In Proceedings of RIAO 2000: Content-
Based Multimedia Information Access, Paris, France, C.I.D.
Dewe J, Karlgren J, Bretan I 1998 Assembling a balanced corpus from the internet. In Proceedings of the
11th Nordic Conference on Computational Linguistics, Copenhagen, pp 100–107.
Dolin R, Alschuler L, Boyer S, Beebe C 2000 An update on HL7's XML-based document representation
standards. Journal of the American Medical Informatics Association, 7(suppl):190–194.
Dublin Core Metadata Inititative 1999 The Dublin Core Element Set Version 1.1. WWW page
http://purl.org/dc/documents/rec-dces-19990702.htm.
254
Dunlop D 1995 Practical considerations in the use of TEI headers in large corpora. Computers and the
Humanities, 29:85–98.
Friedman C 1997 Towards a comprehensive medical natural language processing system: Methods and
issues. Journal of the American Medical Informatics Association, 4(suppl):595–599.
Giordano R 1995 The TEI header and the documentation of electronic texts. Computers and the
Humanities, 29:75–85.
Grefenstette G 1994 Explorations in Automatic Thesaurus Discovery. Natural Language Processing and
Machine Translation. London, Kluwer Academic Publishers.
Grishman R, Kittredge R (eds) 1986 Analyzing Language in Restricted Domains. Hillsdale, New Jersey,
Lawrence Erlbaum Associates.
Habert B 2000 Des corpus représentatifs : de quoi, pour quoi, comment ? In Bilger M (ed), Linguistique
sur corpus : Études et réflexions, volume 31 of Cahiers de l'Université de Perpignan. Presses
universitaires de Perpignan, pp 11–58.
Ide N, Priest-Dorman G, Véronis J 1996 Corpus Encoding Standard. Document CES 1,
MULTEXT/EAGLES, http://www.lpl.univ-aix.fr/projects/eagles/TR/.
Illouz G 1999 Méta-étiqueteur adaptatif : vers une utilisation pragmatique des ressources linguistiques. In
Amsili P (ed), Actes de TALN'99 (Traitement Automatique des Langues Naturelles), Carg&#232;se,
ATALA, pp 185–194.
Illouz G, Habert B, Folch H, Heiden S, Fleury S, Lafon P, Prévost S 2000 TyPTex: Generic features for
text profiler. In Proceedings of RIAO 2000: Content-Based Multimedia Information Access, Paris,
France, C.I.D., pp 1526–1540.
Lehmann D, de Margerie C, Pelfr&#234;ne A 1995 Lecticiel – Rétrospective 1992–1995. Technical report,
CREDIF – ENS de Fontenay/Saint-Cloud, Saint-Cloud.
Rassinoux A.-M 1994 Extraction et Représentation de la Connaissance tirée de Textes Médicaux. Th&#232;se
de doctorat &#232;s sciences, Université de Gen&#232;ve.
Rossi Mori A, Consorti F 1999 Structures of clinical information in patient records. Journal of the
American Medical Informatics Association, 6(suppl):132–136.
Sager N, Friedman C, Lyman M. S (eds) 1987 Medical Language Processing: Computer Management of
Narrative Data. Reading, Mass., Addison Wesley.
Scherrer J.-R, Lovis C, Borst F 1996 DIOGENE 2, a distributed information system with an emphasis on
its medical information content. In van Bemmel J. H, McCray A. T (eds), Yearbook of Medical
Informatics ’95 — The Computer-based Patient Record. Stuttgart, Schattauer.
Sinclair J 1996 Preliminary recommendations on Text Typology. WWW page
http://nicolet.ilc.pi.cnr.it/EAGLES/texttyp/texttyp.html, EAGLES (Expert Advisory Group on
Language Engineering Standards).
Tuttle M, Olson N, Keck K, Cole W, Erlbaum M, Sherertz D, Chute C, Elkin P, Atkin G, Kaihoi B,
Safran C, Rind D, Law V 1998 Metaphrase: an aid to the clinical conceptualization and formalization
of patient problems in healthcare enterprises. Methods of Information in Medicine, 37(4-5):373–383.
Vihla M 1998 Medicor: A corpus of contemporary American medical texts. ICAME Journal, 22:73–80.
Wierzbicka A 1985 A semantic metalanguage for a crosscultural comparison of speech acts and speech
genres. Language in society, 14:491–514.
Zweigenbaum P, Consortium MENELAS 1994 MENELAS: an access system for medical records using
natural language. Computer Methods and Programs in Biomedicine, 45:117–120.

45
The use of the progressive in Swedish and German advanced learner English:
a corpus-based study
Margareta Westergren Axelsson* and Angela Hahn°
*Department of English, Uppsala University
°English Language and Linguistics, Chemnitz University of Technology
Margareta.Westergren_Axelsson@engelska.uu.se, Angela.hahn@phil.tu-chemnitz.de
In our paper we approach the following research questions from a contrastive perspective:
1. What are the differences, if any, in the use of the progressive in Swedish and German learner
English?
2. What functions of the progressive do learners typically choose? Are there any functions chosen
other than the purely aspectual ones?
3. What types of non-native progressives are produced by these learners?
The study is part of a cooperation project between the Chemnitz University of Technology and
Uppsala University with questions of corpus comparability as its main concern. Our research draws on
two learner corpora, namely the Uppsala Student English (USE) corpus (Axelsson 2000), and the
German component of the International Corpus of Learner English (ICLE). To facilitate comparison we
have chosen one text type only – the argumentative student essay.
We have focused on the use of the progressive, since this is a feature not present in Swedish or
German, and thus interesting from a contrastive point of view (cf. also Virtanen 1997, Hahn et al.
2000).
The method entails two types of investigations:
1. A qualitative study of about 12,000 words from each corpus (randomly chosen essays) discussing
the choice of form (simple or progressive) by advanced learners.
2. A quantitative description of the use of the progressive in a material of about 70,000 words from
each corpus.
References
Axelsson, M. W. 2000. USE – The Uppsala Student English Corpus: An instrument for needs analysis.
ICAME Journal 24: 155-157.
Hahn, A., S. Reich & J. Schmied. 2000, in press. Aspect in the Chemnitz Internet Grammar.
Proceedings of the ICAME conference 1999, Freiburg.
Virtanen, T. 1997. The progressive in NS and NNS student compositions: evidence from the
international learner corpus. In: M. Ljung (ed), Corpus-Based Studies in English. Amsterdam:
Rodopi: 299-309.

255
A reusable corpus needs syntactic annotations:
the Prague Dependency Treebank
(YD+DML&#254;RYiDQG3HWU6JDOO
Center for Computational Linguistics
Faculty of Mathematics and Physics
Charles University, Prague
e-mail: {hajicova,sgall}@ufal.mff.cuni.cz
The Prague Dependency Treebank (PDT, i.e. an annotated part of the Czech National Corpus) is
conceived as a three-layer system of tags; the individual layers can be characterized as follows: (i)
morphemic tagging capturing relatively disambiguated values of morphemic categories based on a full
morphemic analysis of Czech; (ii) syntactic tags at the so-called analytical level, capturing the
functions of individual word forms; in the analytical tree structures (ATSs), every word token and
punctuation mark has a corresponding node and is analyzed as for its POS and morphemic value, as
well as for the main syntactic functions (‘analytical functors', ‘afuns'); among the afuns, Subj, Obj,
Adv are not classified in a more subtle way; (iii) syntactic tags at the tectogrammatical level (TGTSs)
rendering the underlying (tectogrammatical) structure of the sentence, i.e., its syntactic structure proper
(with a detailed classification of underlying syntactic functions).
In the sequel we focus on a brief characterization of the TGTSs and on issues that are specific for the
PDT scenario and are crucial, especially from the linguistic point of view. These issues concern (i) the
transition from ATSs to TGTSs, (ii) the assignment of the features of the information structure of the
sentence (topic-focus articulation), and (iii) a tentative treatment of coreference relations. The TGTSs
are based on dependency syntax; the tagging at this level is guided by the following principles: (a) a
node of a TGTS represents an autosemantic (lexical) word; the correlates of synsemantic (functional,
auxiliary) words are attached to the autosemantic words to which they belong; (b) in the cases of
deletion in the surface shape of the sentence, further nodes are supplied into the TGTS to ‘recover’ a
deleted word; (c) no non-projective structures are admitted in the TGTSs (they are supposed to be
solved by movement rules between the ATS and the TGTS); (d) not only the direction of the
dependence on the governing node (dependence to the left, dependence to the right) is taken into
account, but also sister nodes are ordered (from left to right).

256
Using the BNC to produce dialectic cryptic crossword clues
David Hardcastle, Birkbeck College, University of London
1. Overview
This paper describes an attempt to generate seemingly meaningful cryptic crossword clues without
trying to analyse meaning but relying solely on word occurrence statistics. It is a continuation of a
project in which I developed an application toolkit for cryptic crossword clue compilers. The software
described here assembles simple cryptic clues using the resources developed in the earlier project
combined with the British National Corpus (BNC) Sampler.
Some pieces of the process remain problematic making it tempting to look for recourse in grammatical
and syntactic data or investigations with a high processing overhead. However, my aim is to try to
extract as much mileage as possible from data derived from the BNC that can be processed with a
limited overhead.
All of the clues are of a particular type, which I term ‘dialectic’ clues using the taxonomy of D St P
Barnard (Barnard 1963). A dialectic clue is a pair of synonyms for the answer word, appositely
combined as a single short phrase. For example, “Delicate but dainty” and “Pretty light”1 are
acceptable dialectic clues for the word “fair”. Ideally the apparent syntax of the clue should mislead the
person solving the clue by strongly suggesting a different sort of answer. Clues for “fair” such as
“Market average” and “Sound common” would fall into this category.
To assemble such clues, the software must first evaluate all possible synonym pairs2 for the clue word,
and decide which pairs are more apposite than the others. Once a list of suitable pairs has been found,
the second task is to attempt to link the pairs together in a manner which ideally is meaningful, or
failing that at least not too jarring. The principal focus of this paper is on the first of these two tasks,
namely identifying apposite pairings and ranking lists of pairings for shared meaning. I shall then
briefly address possible ways of linking the pairs together.
2. Finding and ranking apposite pairings
2.1 Retrieving the list of synonyms
This part of the process is very straightforward. I required a list of synonyms for my MSc thesis
(Hardcastle 1999), and downloaded a machine-readable version of Roget's 1912 thesaurus from the
Gutenberg Project. I then assembled the data from it into a synonym dictionary. Unfortunately many of
the synonyms are out-of-date and many form lists of co-hyponyms or loosely associated terms. As a
result, the clues are often marred by poor synonyms. The fix for this is clearly to locate a more up-todate,
machine-readable thesaurus, but for the moment the focus of my work and of this paper, is on
combining the resulting pairs rather than their quality as synonyms for the clue word.
2.2 Language and meaning in cryptic crossword clues
The aim of this project is the generation of cryptic crossword clues and not sentences. Although these
may be seen as an analogue for natural language, there are key differences. Cryptic crossword clues
usually have a simple minimal syntax which at best determines the rubric3 of the clue, and frequently
merely acts as filler between the key elements. This is particularly the case for dialectic clues, since
they have the simplest syntax of any of the groupings of clues in Barnard's taxonomy.
The key difference between cryptic clues and sentences is that of meaning. Although cryptic clues do
not have a reference in the real world, they potentially offer two other levels of meaning. The first is
the rubric of the clue, a list of instructions which can be used to solve a clue. The second gives the
reader a vague sense that the clue could refer to some situation and as such that it resembles a
meaningful English sentence.
1 All example clues given are from output generated by the software, unless stated otherwise.
2 Lists of synonyms are derived from a synonym dictionary that I constructed for an earlier project
3 By ‘rubric’ I mean a simple set of instructions through which the clue may be solved
257
For example, the clue “effeminate English to embrace the church”4, a clue for the word “epicene”, is
both a rubric and a sort of sentence. The answer means “effeminate”, and can be formed with the letter
‘E’ for English, then the word “pine” around the abbreviation “CE” for church, as such the clue is a set
of instructions. Of the many alternatives to communicate one item around another, such as “cover”,
“ensnare”, “surround”, the compiler chose the word “embrace”. Similarly the letter ‘E’ could have been
clued differently, or the whole shape of the clue could have been different. I suggest that from a wide
variety of possibilities, the human compiler settled on a particular set of words with a certain feel to
them, since they had just enough in common to suggest a reference. Although the clue is not a proper
sentence, it seems to have some sort of meaning, in that it appears to refer to something real. It is this
sense of appropriate feel that I am aiming to capture in the ranking of the pairs of synonyms for the
simpler dialectic clues.
2.3 Finding apposite pairings
This is the central focus of this paper. Given a list of pairs of words, the software should put the list
into rank order according to the extent to which each pair has something in common, rather like the
common game of word association. Tightly linked pairs such as “bus” and “stop” should come at the
top, then pairs with a tight scope of common context such as “spanner” and “mechanic”, then words
with broad shared scope such as “write” and “work” and finally words which show scant promise of
association, such as “bookcase” and “goaded”.
One way of achieving this might be to consider the definitions of the words, or to map them according
to their membership of certain thematic subsets. However I chose to consider only the extent to which
they co-occur in the BNC in order to investigate whether this data alone would be sufficient to rank the
list.
2.3.1 Getting the raw data
Since the mark-up language of the BNC specifies a clear hierarchical structure, it is possible to break it
down into many different sized chunks, from the level of a whole chapter of a book to a single sentence
or even word boundary. Pairs that share a word boundary should be good candidates for the top-scoring
set, those which share sentences or paragraphs should be good candidates for the second, and so on
down to pairs which are not even much in evidence co-habiting the largest chunks.
To examine pairs and determine the extent of their shared space in the corpus I constructed an index of
the corpus sampler. In this index, every instance of the four hierarchical section boundaries (“<div1>”
to “<div4>”) is sequentially numbered, as is each paragraph within them, each sentence in each
paragraph and each word within each sentence. The resulting index is a dictionary file which provides a
list of coded keys for each word in the corpus.
bridge / 313|-1|-1|-1|8|1|39 / 327|102|60|-1|13|3|4 / 350|164|98|-1|6|4|18
Figure 1: A sample entry from the index to the BNC Sampler
Figure 1 shows part of the entry for the word “bridge” and shows the keys for the first three
occurrences of “bridge” in the BNC Sampler. The final key (350 164 98 –1 6 4 18) states that the word
“bridge” can be found in <div1> number 350, <div2> number 164, <div3> number 98, in an area with
no <div4> context and in the 18th word of the 4th sentence of the 6th paragraph of that section. Were the
word “suspension” to have a key for example 350 164 98 –1 6 4 17 this would tell us that the phrase
“suspension bridge” is evidenced once in the BNC Sampler. Were the word “bidding” to have a key
313 –1 –1 –1 8 1 10, thus sharing the same paragraph as the first key for “bridge”, this would tell us
that the words “bidding” and “bridge” shared the same paragraph once in the BNC Sampler.
4 “Phi”, The Independent Saturday Crossword, No. 4461, 3rd February 2001
258
Using this index, the software can rapidly return the number of occurrences of any word in the BNC
Sampler and also the number of co-occurrences within each level of scope5 for any pair of words. At
present the coding does not take advantage of non-hierarchical scope data such as page breaks or clause
boundaries, nor does it differentiate between different sources. Both of these enhancements could
improve the quality of the raw data. The former would increase the depth of the data returned, and the
latter may prove a useful enhancement to the “<div1>” data, since it could give an indication of size.
While a few co-occurrences at the top hierarchical level of a novel, namely at chapter level, may not
mean a lot, in the context of the spoken corpus or of ephemera where the top hierarchical level is likely
to be much smaller it would be more significant.
2.3.2 Interpreting the raw data
The raw numbers of co-occurrences returned at the different scope levels do give some rough
indication of the level of shared context between the two input words. However, such data requires
further refinement since common words tend toward the top of the list, and rarer words toward the
bottom, regardless of how apposite the pairings.
I examined the feasibility of using the statistical test chi-squared to determine the significance of the
co-occurrences for each pair against a random distribution. Unfortunately chi-squared, and other tests
for statistical significance, lend additional weight to larger samples. While this makes perfect sense for
data such as polls, it only deepened the rift between common and rare words rather than promoting the
more apposite pairs. For example, the pairing “work” and “go” scored 17,649 in a chi-squared test,
whereas the pairing “ski” and “salopettes” scored only 4.
The hypothesis behind the process of comparison was that the closer the association between a pair of
words, the more likely it would be to find co-occurrences of the pair within small chunks of the BNC
such as sentences or paragraphs. Therefore the scoring algorithm needed to lend extra weight to cooccurrences
found within a small scope, such as a sentence. In order to determine whether the number
of co-occurrences found at each level of scope counted as likely or unlikely, I constructed a control set
of over 20,000 randomly selected pairs from a machine-readable dictionary (Mitton 1986) to measure
the average number of co-occurrences at each level of scope for each pair. To counter the advantage
toward words with a high frequency in the earlier scoring systems, I recorded the ratio of cooccurrences
to total occurrences of the pair rather than just the raw total of co-occurrences.
The scoring algorithm scores the factor of difference between the average ratio of the baseline pairs
and the recorded ratio for the pair under examination at each level of scope. The score is the total of
these factors of difference. The factoring process lends extra weight to co-occurrences at small scope
level as the baseline ratios for sentences and paragraphs are extremely small. Word boundary cooccurrences
are scored in raw quantity, and pairs which evidence such juxtapositions are promoted to
the very top of the list regardless of their overall score. To return to our earlier example, “work” and
“go” which scored 17,649 with a chi-squared test now scored 1,415 while “ski” and “salopettes” was
promoted from a chi-squared score of 4 to a score of 3,785.
There are some drawbacks to this scoring system. Firstly, the use of ratios discriminates heavily against
unbalanced pairings where one word is very common and the other very rare. It is also left open to
some bizarre results where a word shows up just a few times in the BNC Sampler and all of the
occurrences are in a particular atypical context. Finally words that do not appear in the BNC Sampler
score zero by default, thus potentially missing good pairings. These problems can all be addressed by
using a larger portion of the BNC.
In spite of these potential problems, the scoring system seems to function relatively well, although
testing the system proved a difficult task. Since the shared context, or lack of one, between any two
words is not a given it is difficult to test the scores generated against some other property of the
pairings. At some future stage I would like to compare the rankings of a set of input pairings with the
results of a survey where people rank the pairs according to how well they go together. For the timebeing
I have run two simple tests on the scoring system.
5 By ‘scope’ I mean the size of the context within which a co-occurrence has been found, such as a
sentence, a paragraph, or a chapter
259
The first involved some consideration of the distribution of test scores for a large set of randomly
generated pairings derived from the dictionary. The resulting distribution is represented in Figure 2 and
appears to be relatively promising. Firstly, it seems to be discriminatory, in that the vast majority of
pairs failed to score, and that the frequency of scores follows quite a steep curve from frequent
mediocrity to the rare sublime pairing at the top of the scale. A cursory inspection of some examples of
the ranked pairings also suggested a relatively successful scoring mechanism (see Figure 2).
Score band %age Examples
1,001 to 10,000 0.1 goodwill, contract (3,089) factory, working (2.436) jazz, waistband (1,753)
201 to 1,000 0.4 Rain, outlook (907) raid, escapee (440) fluid, stomach (330)
51 to 200 0.8 Club, sixty (187) holy, building (157) guest, tribute (101)
11 to 50 2.7 Waistcoat, guilt (26) blank, game (22) torch, wind (35)
1 to 10 3.1 Timing, accession (4) cardboard, scamper (4)
0 92.9 Haircut, garlic (0) radish, thump (0) Lutheran, shaker (0)
Figure 2: some examples of pairs in the different score bands following a test on the scoring system of
a large sample of word pairs. The percentages represent the total number of pairs in each score band as
a percentage of the total pairs under examination
Finally I tested the scoring mechanism with a set of chosen pairs using the word “market” that I felt to
have more or less in common along a relatively clear scale. The resulting rankings do at least seem to
group the pairings roughly into the correct halves, with most of the top ten being those which I picked
because I felt that they shared a common context, and most of the bottom half of the table being those
which I chose to represent a poor association (Figure 3). Four of the expected top five pairs received
the highest scores, with “fruit” unexpectedly coming in 7th place. However, the words which received
low scores did not do so just because of poor frequency; as “fruit” scored 177 with “tree” and 871 with
“veg”, while “church” and “Catholic” scored 1,703 as a pairing.
Expected top 5 scoring set Expected bottom 5 scoring set
Score Ranking Score Ranking
Average 1542 1st Church 145 5th
Sell 1000 2nd Plate 67 6th
Money 580 3rd Catholic 15 8th
Europe 560 4th Ski 15 8th
Fruit 47 7th Headlights 5 10th
Figure 3: Scores for a set of words paired with “market”
2.3.3 Refining the scoring process – ‘third party co-referents’
Although this system of scoring provided a reasonable system of ranking pairings, it did not perform
well for words with very low frequencies in the BNC Sampler. For example the words “headlights” and
“wipers” which clearly have a strong association receive an unimpressive score (9), with only 2
matches between them and those at the widest scope. Despite the strong word association, it is unlikely
that we will find many co-occurrences within the same sentence or the same paragraph. Indeed, even
with a much larger corpus we may not find a significant number of co-occurrences within small scopes.
260
To deal with pairs of words where one or both occurred relatively infrequently, I decided to examine
which other words occurred in close proximity. Reversing the index shown in Figure 1, I generated a
look-up table of all the words which co-occur in the same paragraph as the key. I will refer to the list of
words which co-occur at paragraph level with both input words as ‘third party co-referents'. Initially
the lists of co-referents were dominated by function words, pronouns and common verbs. To
determine the culprits of this ‘noise', I generated lists for a large sample of input words and identified
words which appeared in more than 80% of the outputs. These words are filtered out of all lists of coreferents
which the software produces.
Figure 4 shows the most commonly co-occurring words within paragraph scope for “headlights”,
“brake” and “donkey” filtered for noise. The words “headlights” and “brake” share two of the top
seven entries, and indeed many more of their respective full lists, while “donkey” shares none in the
top seven and few in the full list with either of the other words. Such ‘third-party co-referents’ provide
a means to promote such a pair up the rankings. An important point is that the list of paragraph-level
co-referents does not necessitate that “headlights” and “brake” co-occur in any paragraphs themselves,
instead the co-referents represent an intermediate measure of shared context to make up for the lack of
direct evidence.
Headlights Brake Donkey
Fog * Caravan Ballot
Grandma Solenoid * Votes
Bumper Portable Rifle
Vibrating Toot Exhaustive
Spotlights Slate Candidate
Bonnet LTD Battalion
Balancing Fog * Voters
Flash Fluid Camel
Solenoid * Daimler Ape
Flashing Starter Spanish
Figure 4: co-referent lists, “fog” and “solenoid” act as third party co-referents between “headlights”
and “brake”
Returning to the example of “headlights” and “wipers”, the software was now able to score a list of
third party co-referents, words which co-occurred at paragraph level with “wipers” and also with
“headlights”. Using this intermediary the pair now scored 62 rather than 9, a modest but notable
improvement. The system also provided some measure for pairs which had previously scored zero,
such as “wipers” and “brake” which rose from 0 to 40. The system occasionally threw up some rather
unusual pairing suggestions, such as “Catholic” and “sheep” which scored 610, although further
investigation always uncovered a rational explanation and an overlap of reference.
At present this system of third party co-referents remains experimental, although the results so far have
been very promising. I selected paragraph level scope since I felt that it was wide enough to provide
sufficient information to work with, while being sufficiently small that the co-reference could
reasonably said to mean something. It may be that a larger scope such as page level, or a reduction to
sentence level may produce cleaner or more informative result sets.
It is notable that at present this system has been tested predominately with nouns. Nouns, and count
nouns in particular, seem intuitively to be more apt to this type of processing as they can be more
readily ascribed to thematic subsets. Although many verbs and adjectives share this property, it is
arguable that the majority have too general a function for this process to bear fruit. Whether or not this
distinction holds is something I hope to explore as I develop this co-referencing process.
261
2.3.4 Third party co-referents and meaning lists
As I explored the first few lists of third party co-referents I observed that for many words the lists
contained many key thematic elements that to some extent defined the context of the input word.
Although an unedited list of these words would prove a crude measure of meaning, I felt that the crossreference
between this list and lists of co-hyponym sets might provide some measure of meaning.
To clarify, the idea would be to compare an input word to a set of co-hyponym lists. The crossreference
would involve neither the input word, nor the co-hyponym key, but a cross-referencing
between the list of ‘third party co-referents’ for the input word and the list of co-hyponyms under the
keyword. A large overlap might indicate that the co-hyponym keyword represents some core aspect of
the meaning of the input word.
The thesaurus I have been using contained many lists of co-hyponyms, indeed many of the clues I have
generated still suffer from a poor rubric as they contain co-hyponyms rather than synonyms. In order to
reduce the extent of this problem I removed the tagged lists of co-hyponyms from the thesaurus. Figure
5 shows the results of cross-referencing the third party co-referents from the input words with the cohyponym
lists from Roget. The percentages represent the percentage of cross-references between a coreferent
and a co-hyponym which occurred in each category. Although the lists are crude and short,
numbering less than 30 in total, the results quite clearly favour some reasonable and interesting
keywords.
Church: Social (23%) Religion (17%) Military (17%) Politics (12%)
Horse: Animals (30%) Industry (9%) Military (9%)
Doctor: Politics (18%) Social (15%) Science (9%) Medicine (7%)
Figure 5: Examples of the results of cross-referencing the co-referent list for the input word with lists
of co-hyponyms.
With the help of the full version of the BNC and a fuller set of co-hyponym lists, I hope to be able to
return a list of meaning keywords for most noun input words. As above, it is highly likely that while
this system will work well for nouns and for count nouns in particular, it will not work so well for
verbs, adjectives and adverbs since they are not so readily classified into sets.
3. Linking the pairs together
Having ranked the pairs of synonyms for the clue solution word according to their shared context using
the processes described above, the next step is to link the pairs together in as appropriate a fashion as
possible.
3.1 Juxtaposed pairs
Some of the pairs may have been found in juxtaposition within the corpus. Given that there is evidence
of how to combine them already, the best chance for an idiomatic feel is to present them as they were
found. For the most part, this approach seems to have been relatively successful producing clues such
as “Still soft” for “gentle”, “Standing order” for “condition” and “Tax band” for “press”. Where the
approach is less successful is where the pairing forms a part of a longer idiomatic phrase. While “Pretty
big” sits happily as a unit on its own, the pairing “Great big” scores the same but is clearly inferior, as a
third word is expected.
It is possible that the difference in quality between “pretty big” and “great big” could be measured with
closer examination of the surrounding tags and clause boundary markers. However, I feel that to set out
in such a direction would detract too much from the core of this project, which is to produce dialectic
cryptic clues at low processing cost, and using descriptive data from the BNC.
Thus pairs evidenced in juxtaposition are left as they are found, and the software takes its chances with
the results. Provided that the words are not function words or pronouns, unlikely in the context of
synonyms for a clue word, the result should at the very least be acceptable and seems quite frequently
to produce a nice pun. I describe the scoring of puns below (see below Section 4).
262
3.2 Pairs requiring a link word
The remaining pairs of synonyms were not found in juxtaposition, and so they cannot be presented as a
clue until a suitable way has been found to link them together. One way of achieving this is to examine
what part of speech the words could be and linking them in appropriate ways. For example, singular
noun plus “to” plus infinitive intransitive verb, infinitive transitive verb plus singular noun, adjective
plus singular noun, noun plus “or” plus noun, and so on.
While this approach generates some passable clues they do not read very idiomatically and, since such
an algorithm requires a decision on what part of speech each component is, homograph puns are likely
to be missed. Furthermore, the system does not allow for idiomatic links between pairs, since any
departure from the simplest grammatical structure is likely to generate unsightly exceptions. For
example, given the words “able” and “resolve” we could choose to interpret them as an adjective and
noun pair and produce “able resolve”. However, they could also be interpreted as an adjective and verb
pair, but the phrase “able to resolve” would be a significant risk as this structure is not guaranteed to
work with the majority of adjective-verb pairings and may produce some awful phrases such as “red to
drill” or “peaceful to dig”.
The phrase “able to resolve” is acceptable since one can be “able to” do many things. Rather than set
about listing the adjectives that could fit this pattern and build a prescriptive grammar, I decided to list
the conjunctive phrases found immediately before and immediately after all of the words in my
dictionary in the BNC. For want of a better source of conjunctive phrases I opted to use a list of
function words (Mitton 1996) and to count a phrase as one, two or three function words in a group.
Some examples of the resulting statistics are listed in Figure 6 and Figure 7. The entries for “Red” and
“Grounds” in Figure 7 exemplify problems with adjective position and framing phrases, these issues
are discussed in greater detail below
Contact: with (63), between (8), by (5), from (3)
Popped: in (44), along (11), down (11), on (11) round (11), up (11)
Ready: to (45), for (30), and (6)
Figure 6 Examples of statistics for conjunctive phrases following dictionary entries, the numbers in
brackets are percentages representing the frequency of each phrase in the total sample found
Austria: in (25), of (25), and (12), from (12), to (12), with (12)
Perspective: in (35), of (14), on (14), and (7), from the (7), of the (7), that (7)
Red: a (19), of (12), in the (6), into the (5), that (5), with (4), and the (2) …
Grounds: on the (65), in the (10), of (7), on (2), outside the (2), within the (2) …
Figure 7 Examples of statistics for conjunctive phrases preceding dictionary entries, the numbers in
brackets are percentages representing the frequency of each phrase in the total sample found
Rather than attempt to combine the pairings by determining their part of speech and looking for a
grammatical structure to accommodate them, the software looks at the function words which are
evidenced preceding and following each word in the pair. If a fit is found between the preceding
conjunctive phrases of one word and the following conjunctive phrases of another it is scored
according to the combined relative frequency of both. In this way the phrase “able to resolve” becomes
a safe bet with a score of 186 out of a maximum 200 rather than a gamble. Other outputs from this
process are listed in Figure 8.
left a pot (46), pot on the left (34), left to pot (20), pot to the left (17), left in the pot (15), left on the
pot(13) …
likely to go (158), likely that go (13), likely a go (4) …
pile of grass (82), pile in the grass (25), pile on the grass (16) …
Figure 8 Some pairs generate several similar scores, such as “left” and “pot”, others just a single high
score
263
Where a fit cannot be found the pair does not receive a score for linking and may be demoted to the
benefit of other better fitting pairs. If none of the pairs with good context scores has a good fit, then a
rough fit must be made by determining the what part of speech each member of the pair could be and
assembling them according to a very rudimentary grammar.
Although this approach produces successful results, there are still occasions when we are left with a
clue which is one word short of an idiomatic phrase. The key problems which remain in the assembly
of the pairs are:
Intervening words. Most commonly adjectives are the culprit. Not only does the intervening adjective
deprive the software of data about the conjunctive phrase and the noun, it also results in phrases such as
“in the yellow” and “under the long” seeming acceptable. Cleaning out such data without losing idioms
such as “in the red” and “in the main” will be difficult, although it may be possible to find the idioms
through testing the whole combination for frequency.
Frames. As with the previous problem, the result is phrases that feel like they are missing something,
such as “for the reason” and “on the assumption”, both of which require the word “that” and a clause to
follow. It is difficult to see how to remove these without a considerable processing overhead.
Clause boundaries. This problem arises from the indexing of the spoken section of the BNC Sampler.
Since the clause boundaries are coded differently, they are not accounted for in my index, and as a
result pairs of words crossing word boundaries are recorded as contiguous. For example the word
“hand” records “yes” as a following phrase, probably from the text “would you like a hand? Yes … “.
This could also cause unusual results in the return of supposedly juxtaposed pairings. Fortunately the
solution is not complex, only time-consuming, and will involve refreshing the index to take account of
the difference in mark-up.
4. Scoring puns
At present the resulting pairs pass through a very straightforward filter to score the puns which
examines the thesaurus entry and checks to see if the pair is a combination of words from single or
multiple entries of the thesaurus.
Homograph puns feature fairly heavily in the resulting clues, in particular in the clues derived from
juxtaposed pairs. The present system, although a crude measure, picks up the majority. A precise
measure would require a substantial overhead, especially since homograph puns are encouraged by the
software not differentiating between homographs in the processing of the synonym pairs. Indeed, for
the bulk of completed clues, the software will at no point have investigated what part of speech the
composite parts might be.
Pairs scoring as puns for the word “fair” included “market average” and “sound common”. Nonscoring
pairs included “pearly white” and “white light”. The latter of these is in fact a pun, “light”
being suggested as a noun. However to determine this the software would have to return to the
reference in the corpus and establish that the pairing “white light” which evidenced the juxtaposition
was an adjective noun pair, and this requires a substantial overhead.
5. The production of a dialectic clue
The following description provides an overview of the whole process of the generation of a dialectic
clue with examples of each processing stage, using “reign” as the input word.
The input word, “reign”, is used as a key against the thesaurus to generate a list of all the possible pairs
of synonyms for the word. Each pair is scored using the index of the BNC Sampler against the baseline
ratios for co-occurrence at various levels of scope. This process is refined by comparing the extent to
which lists of all co-occurring words at paragraph level scope, third party co-referents, overlap for each
pair. The pairs are sorted into ranked order on the co-occurrence and third party co-referent scores.
Any pairs found in direct juxtaposition are separated off.
264
Lead weight (direct juxtaposition “lead weight”)
Lead capability (direct juxtaposition “capability lead”)
Authority pressure
Importance authority
Authority control
Effect authority
Figure 9 The top pairings for “reign”. Many of the synonyms are not particularly good, but the pairings
share some common context.
Data from the BNC Sampler on conjunctive phrases occurring before and after each member of each
pair is examined to find a match. The top-scoring match is selected for each pair, and the score
recorded. By combining the linking score with the context score, pairs which have less context but link
together more idiomatically are promoted.
Pressure of authority (23), authority under pressure (22), authority of pressure (20) …
Importance of authority (64), importance of the authority (51), authority and importance (28) …
Control of authority (57), control of the authority (44), authority to control (40) …
Figure 10 The results of combining the top-scoring pairs using link words. The first two pairs (lead
weight and capability lead) are not linked as they were found in juxtaposition
A filter checks each pair to determine if it could be a homograph pun. Puns are promoted to the top of
the list. The software selects the top-scoring punning clue, or the user can select their favourite from
the head of a list of formatted clue suggestions.
Lead weight (5)
Capability lead (5)
Authority under pressure (5)
Importance of authority (5)
Authority to control (5)
Figure 11 Sample formatted clues for “reign” using varied link words
6. Evaluation
I evaluated an early version of the software by generating dialectic clues for a set of six clue words
which had been clued dialectically in broad-sheet crosswords. I selected the top five scoring clues from
each clue word, and presented them to a group of crossword enthusiasts in groups with the broadsheet
clues mixed in. The enthusiasts then scored the clues for readability without knowing the answer word.
I then averaged the scores of the broadsheet clues. For readability the clues from the broadsheets scored
3.4, meaning that on average they were placed between 3rd and 4th place against 5 clues generated by
the software, indicating stiff competition from the computer-generated clues on this front.
The software-generated clues fared less well when the answers were revealed as they frequently
contained good puns made with very poor synonyms. Some computer-generated clues were deemed to
be of good quality, even when the answer was revealed. These included “market average”, “sound
common” and “common market” for “fair”, “dictatorship of the regime” for “reign”, “tax band” for
“press” and “separate part” for “isolate”. However, many clues were deemed to be insoluble as the
synonyms were too difficult or inappropriate. For example, the clue “Lead weight (5)” was a neat pun,
but could not be solved as “weight” is too remote from “reign”. Similarly “Degree of distinction (9)”
again read well, but “distinction” for “condition” was felt to be unfair.
265
7. Summary
The result of the evaluation suggested that the project has so far been largely successful and has
generated some appealing dialectic cryptic crossword clues without human intervention. However, an
improved thesaurus is clearly much-needed if clue quality is to be raised.
The software identifies apposite pairings without recourse to definitions or tables of meaning, but using
the levels of co-occurrence in the BNC Sampler and through indirect co-occurrence by means of lists
of ‘third party co-referents'.
It identifies appropriate bridging phrases to link pairs together where they are not evidenced in
juxtaposition without recourse to syntactic or grammatical guides, but purely through frequency data
gleaned from the BNC Sampler.
References cited
Barnard D. St. P 1963 The Anatomy of the Crossword Bell
Hardcastle, D 1999 SphinX Crossword Compiler's Toolkit Unpublished MSc Thesis, Birkbeck College,
University of London
Mitton R, 1986 “A partial dictionary of English in computer-usable form” Literary and Linguistic
Computing 1(4): 214-5
Mitton, R, 1996, English Spelling and the Computer, Longman, Appendix 2

266
Comma checking in Danish
Daniel Hardt
Copenhagen Business School & Villanova University
1. Introduction
This paper describes research in using the Brill tagger (Brill 94,95) to learn to identify incorrect
commas in Danish. Trained on a part-of-speech tagged corpus of 600,000 words, the system identifies
incorrect commas with a precision of 91% and a recall of 77%. The system was developed by
randomly inserting commas in a text, which were tagged as incorrect, while the original commas were
tagged as correct. Then the tagger was trained to recognize the contexts in which incorrect commas
occur. In what follows, we first describe the corpora and tag sets used in this research, and give
background on the Brill Tagger. We then describe the methodology for learning to identify comma
errors, and then we examine some of the principles that the system learned to identify comma errors.
Finally, test results are presented, and we discuss plans for future research. The method used here is
quite general, and could be applied fairly directly to a wide range of grammar checking problems, in
Danish or other languages.
2. Background
·  Corpora and tag sets
This research uses two Danish corpora: the Parole corpus (Parole 1998) and the Bergenholtz corpus
(Bergenholtz 1988). The Brill tagger was trained on the manually tagged Parole corpus to recognize
Danish part of speech tags. The Danish Parole tag set consists of 151 distinct Tags, containing
information such as syntactic category, number, gender, case, tense and so on. As described below, we
have used a reduced version of the Danish Parole tag set for the current project.
·  Brill tagger
The Brill tagger learns by first tagging raw text with an Initial State Tagger, which tags words with
their most frequent tag. The resulting file is termed Dummy, and is compared to a file called Truth,
which has been manually tagged, and is thus assumed to be completely correct. 1 The system
Contextual-Rule-Learn searches for transformations that can be used to make Dummy more closely
resemble Truth. The system searches among transformations that instantiate the following templates:
Change tag a to tag b when:
1. The preceding (following) word is tagged z.
2. The word two before (after) is tagged z.
3. One of the two preceding (following) words is tagged z.
4. One of the three preceding (following) words is tagged z.
5. The preceding word is tagged z and the following word is tagged w.
6. The preceding (following) word is tagged z and the word two before (after) is tagged w.
Learning proceeds iteratively as follows: Contextual-Rule-Learn tries every instantiation of the
transformations templates, and finds the transformation that results in the greatest error reduction. (See
Fig. 1.) This transformation is output to the Context Rules list, and the transformation is applied to
Dummy. The process continues until no transformation results in an improvement above a preset
1 We ignore the lexical rules, which are learned in a separate phase. These are not relevant to the
present study. See Brill 94, 95 for details.
267
threshold. The tagger can then be run with rules that determine part of speech tagging for Danish,
based on the Danish Parole Corpus. We term this the Base Tagger.
Fig 1. Training the Base Tagger
3. Training the comma checker
We produced the training file by tagging 600,000 words of text from the Bergenholtz corpus, using the
Base Tagger. We converted the tags to the Reduced Parole Tag Set. This was done to facilitate the
learning of generalizations such as “no comma between a preposition and a noun”. In the original tag
set, there are 23 tags for common nouns, because of differences in number, gender, etc. In the reduced
tag set, there are just two: N (common noun), and N_GEN (genitive noun). Other categories have
similarly reduced numbers of tags.
To use this tagged file as the training corpus for developing a comma checking system, we make the
simplifying assumption that all existing commas are correct, and that no additional commas would be
correct. Thus all existing commas in the training corpus are given a new tag, GC (good comma). Next,
two copies of the training corpus are created, Truth and Dummy. In each of these, commas are inserted
at random positions (in the same positions in each file). The inserted commas are labeled BC in Truth,
and GC in Dummy file. Thus the only differences between the two files are that the randomly inserted
commas are tagged with BC in Truth and GC in Dummy.
Then, Contextual-Rule-Learn is run on these two files. The result is an ordered list of Error Context
Rules for commas. (See Fig. 2.)
Raw Text:
PAROLE Corpus
Initial State
Tagger
Dummy
Contextual-Rule-Learn POS Context Rules
Truth:
Tagged PAROLE Corpus
268
Fig 2. Learning Comma Context Rules
Thus what the system learns is contexts in which a comma's tag should be changed from GC to BC,
and in this way marked as an error. The list of such contexts is produced by the learner as an ordered
list of rules, specifying when the comma tag should be changed. It is important to note that these rules
are ordered, so that a decision specified by a rule early on the list will sometimes be reversed by a rule
later on the list.
In all, 166 Error Context Rules for commas were produced. The first 12 rules are shown below:
1. GC -> BC if one of the three following tags is End-of-sentence
2. GC -> BC if one of the two previous tags is Beginning-of-sentence
3. GC -> BC if the next tag is Preposition
4. GC -> BC if one of the two following tags is Verb(Infinitive)
5. GC -> BC if the previous tag is Conjunction
6. BC -> GC if the previous tag is Interjection
7. GC -> BC if one of the two previous tags is Subordinating Conjunction
8. GC -> BC if the previous tag is Preposition and the following tag is N
9. GC -> BC if the previous tag is Pronoun and the following tag is N
10. GC -> BC if the previous tag is Verb(past) and the following tag is Pronoun(personal)
11. BC -> GC if one of the next two tags is Subordinating Conjunction
12. GC -> BC if the previous word is er (is)
The first two rules state that a comma is marked bad (“BC”) if it is within 3 words of the end of a
sentence, or within 2 words of the beginning of the sentence. These rules were learned because there
were comparatively few correct commas in these environments in the Truth file, and a large number of
Dummy:
Errors Not
Marked
Contextual-Rule-Learn Error Context
Rules for commas
Truth: Errors
Marked
Tagged Text
(Bergenholtz Corpus)
Generate Errors:
Insert Random Commas
269
incorrect commas in these environments. However, the system soon learns that these rules are overly
general. For example, the sixth rule states that a comma is correct if preceded by an interjection. This
occurs typically near the beginning or end of a sentence, as in the following example from the training
corpus:
Naa/INTERJ ,/GC I/PRON_PERS sidder/V_PRES stadig/RGU og/CC hygger/V_PRES
Well , you sit still and enjoy
jer/PRON_PERS ./XP
yourselves.
Rule 7 doesn't permit commas between prepositions and nouns, and Rule 8 doesn't permit commas
near the beginning of a subordinate clause. This is related to the fact that a comma typically introduces
a subordinate clause in Danish. This fact is partially captured in Rule 11, which permits commas just
before subordinating conjunctions. Rule 9 disallows commas between a Pronoun and Noun. In the
Parole corpus, there is no category for Determiner, and words like the and a are tagged as pronouns.
4. The resulting system
We build a system that corrects commas in raw text, based on the rules learned above. Text is first
tagged by the Base Tagger, and then commas are all tagged GC. Next the Comma Corrector is
executed – this is the tagger with the Comma Error Rules. In the output, any incorrect commas are
tagged with BC. (See Figure 3.)
Here is a sample run of the system, with different comma positions in the (constructed) sentence Det er
godt, at du kom (It is good, that you came):
Input
Det er godt, at du kom.
Det er godt at, du kom.
Det er godt at du, kom.
Det, er godt at du kom.
Det er, godt at du kom.
Output
Det er godt , at du kom .
Det er godt at ,/BC du kom .
Det er godt at du ,/BC kom .
Det ,/BC er godt at du kom .
Det er ,/BC godt at du kom .
Of the five different comma positions, only the first is correct in Danish. 2 The system correctly labels
all the other alternatives as incorrect (BC).
2 This is in fact not entirely clear, since there are at least two distinct systems for placing commas in
Danish, and this position may not be considered correct in one of the two systems. While it is difficult
to get confident judgments, all my Danish informants agree that this example has only one possible
comma position, which is the one accepted by the comma correction system.
270
Output
Fig. 3 Comma Correction System
5. Empirical results
The system was tested with a file of distinct text from the Bergenholtz corpus, containing 14,044
words. The file contains 869 commas. 389 additional commas were introduced in random positions,
as errors. The system marked 327 commas as errors, of which 299 actually were errors. This gives a
precision of 91.4% and a recall of 76.9%.
Here is a list of the first 10 examples where the system incorrectly marked a comma as an error:
1. Hulgaard/EGEN ,/BC &#197;rhus/EGEN
2. mener/VPRES ,/BC vi/PRONPERS
3. mener/VPRES ,/BC han/PRONPERS
4. menneskemassen/VPRES ,/BC der/UNIK
5. 17-13/NUM ,/BC Norris-Paulsen/N
6. morderiske/VPRES ,/BC psykopatiske/VINF
7. S&#248;rensen/EGEN ,/BC &#197;rhus/EGEN
8. nabokommunen/N ,/BC p&#229;/SP
9. systemet/N ,/BC kan/VPRES
10. de/PRONDEMO aktive/ADJ ,/BC servicefunktionerne/N
In items 1 and 7 a line break was incorrectly placed immediately before the text in question. Items 4
and 6 involve mis-tagging:
Raw Text
Base Tagger
Tagger with POS
Context Rules
Label
Commas
Comma Corrector
Tagger with Comma
Context Rules
271
menneskemassen (“mass of people”) and morderiske (“murderous”) are both nouns, mis-tagged as
verbs. Item 10 is an interesting case De aktive, servicefunktionerne (the active, service workers). The
comma is marked as incorrect because of the following rule:
GC -> BC if the previous tag is ADJ and the next tag is N
This is normally correct; commas don't tend to appear between an ADJ and a N. Here, however, “the
active” is a complete NP, on par with, e.g., “the rich”, and “service workers” is a separate NP.
Total
Number
Commas
Incorrect
Commas
Total System
Corrections
Valid System
Corrections
Precision Recall
1258 389 327 299 91.4%
(299 / 327)
76.9%
(299 / 389)
Table 1. Results
6. Discussion and further work
The system was developed using the transformation-based learning system of the Brill tagger. This
learning system is limited in various ways: for example, only three words or tags before or after a
position are examined. It is likely that certain patterns involving commas could be learned if that
locality restriction were loosened. Furthermore, the learning system of the Brill tagger maximizes
overall success rate, using a greedy strategy. We believe precision is a more relevant measure in
grammar checking problems. Thus it would be interesting to modify the learner so that it optimizes
precision or some related measure, and we suspect that greedy learning may be problematic in this
case. We are contemplating various experiments related to these issues. It is also possible that the
precision and recall of the system would be substantially increased with a larger training corpus. Work
is proceeding on this. Finally, we plan to apply similar techniques to a wide variety of grammar
problems, both in Danish and other languages.
References
Bergenholtz, H 1988 Et korpus med dansk almensprog. Hermes.
Brill, E 1994 Some Advances in rule-based part of speech tagging. In Proceedings of the Twelfth
National Conference on Artificial Intelligence (AAAI-94), Seattle, WA.
Brill, E 1995 Transformation-based error-driven learning and natural language processing: a case study
in part of speech tagging. Computational Linguistics 21(4).
Golding, Andrew R and Schabes, Yves 1996 Combining trigram-based and feature-based methods for
context-sensitive spelling correction. In Proceedings of the 34th Annual meeting of the Association for
Computational Linguistics.
Jacobsen, Henrik Glaberg and J&#248;rgensen, Peter Stray. 1991. Politikens H&#229;ndbog i Nudansk. Politikens
Forlag.
Parole. 1998. http://coco.ihu.ku.dk/~parole/par_eng.htm

272
Tracking lexical change in present-day English
Raymond Hickey
Essen University
For several centuries English has been well-known for frequent cases of conversion (word-class change
without any formal alteration). In recent decades a further development can be observed which for the
want of a better word could be termed univerbation. By this is meant that structures consisting of
several words are reduced to one, as when a verbal phrase is compacted to a single word, e.g. we spent
the night in Vienna -> we overnighted in Vienna. My contention is that such cases illustrate a process
which is part of a long-term typological shift in English. The latter is what has been observed in the
shift from a morphologically complex to an inflectionally simplified language and is conventionally
referred to as a move from synthetic to analytic. The current process can be viewed as a later stage in
an analytic language where lexical compaction is in evidence and can thus be interpreted as part of a
typological cycle. A side-effect of this compaction is that the subcategorisation rules for existing verbs
can be altered (usually expanded to include a new type) as in This door is fitted with an alarm -> This
door is alarmed, i.e. the valency of alarm has been altered to allow for both animate and inanimate
objects. The paper will look at several cases illustrating the matters just alluded to and comment on the
theoretical ramifications for the structure of English.

273
Orality and noun phrase complexity: a corpus-based study of British and
Kenyan writing in English
Diana Hudson-Ettle*, Tore Nilsson° and Sabine Reich*
*Department of English Linguistics, Chemnitz University of Technology,
Reichenhainer Str. 39, 0917 CHEMNITZ, Germany
° Department of English, Uppsala University, Box 527, SE-751 20
UPPSALA, Sweden.
diana.hudson-ettle@t-online.de, toren@bahnhof.se, sabine.reich@epost.de
According to previous research (Biber 1988, Chafe 1982, Halliday 1989), both spoken and written texts
display evidence of orality such as lack of planning and elaboration, indicated by the relative frequency
of specific linguistic features. On the basis of this evidence, we examine selected text types to
determine their position on a cline of orality with a view to investigating whether noun phrase
complexity in terms of adjectival premodification can also be said to be sensitive to this concept of
orality.
Our first hypothesis is that noun phrases in texts showing more oral features are characterized by a
relatively lower degree of adjectival premodification. Our data consist of English texts from Britain and
Kenya.
The first step of the study focusses on three genres from the Kenyan subcorpus of ICE-EA (The East
African component of The International Corpus of English compiled at Chemnitz University of
Technology). These genres are social letters, reportage feature articles and institutional editorials. The
aim of this first step is to establish the position of each text type on the orality cline. In the second step
our attention is focussed on reportage feature articles alone.
Results obtained in the first step are seen in comparison with those gained from analyses of the same
text type taken from two corpora of British English, UPC (the Uppsala Press Corpus) and a corpus of
British Tourism and Travel Texts, both compiled at Uppsala University.
Our second hypothesis is that the texts from Kenya will contain more evidence of orality than the
British feature articles.
References
Biber, Douglas (1988) Variation across Speech and Writing. Cambridge: Cambridge University Press.
Chafe, Wallace L. (1982) “Integration and involvement in speaking, writing and oral literature”. In
Deborah Tannen (ed.), Spoken and Written Language: Exploring Orality and Literacy, 35-54.
Norwood, NJ: Ablex.
Halliday, M.A.K. (1989) Spoken and Written Language. 2nd edition. Oxford: Oxford University Press.

274
The American National Corpus:
A standardized resource for American English
Nancy Ide* and Catherine Macleod†
*Department of Computer Science
Vassar College
Poughkeepsie, NY 12604-0520 USA
ide@cs.vassar.edu
†Computer Science Department
New York University
New York, New York 10003-6806 USA
macleod@cs.nyu.edu
1 Introduction
Linguistic research has become heavily reliant on text corpora over the past ten years. Such resources
are becoming increasingly available through efforts such as the Linguistic Data Consortium (LDC) in
the US and the European Language Resources Association (ELRA) in Europe. However, in the main
the corpora that are gathered and distributed through these and other mechanisms consist of texts which
can be easily acquired and are available for re-distribution without undue problems of copyright, etc.
This practice has resulted in a vast over-representation among available corpora of certain genres, in
particular newspaper samples, which comprise the greatest percentage of texts currently available from,
for example, the LDC, and which also dominate the training data available for speech recognition
purposes. Other available corpora typically consist of technical reports, transcriptions of parliamentary
and other proceedings, short telephone conversations, and the like. The upshot of this is that corpusbased
natural language processing has relied heavily on language samples representative of usage in a
handful of limited and linguistically specialized domains.
A corpus is intended to be "a collection of naturally occurring language text, chosen to characterize a
state or variety of a language" (Sinclair, 1991). As such, very few of the so-called corpora used in
current natural language processing and speech recognition work deserve the name. For English, the
only true corpora that are widely available are the Brown Corpus (Kucera and Francis, 1967) and the
British National Corpus (BNC) (Leech, 1994). Although it has been extensively used for natural
language processing work, the million words of the Brown Corpus are not sufficient for today's largescale
applications. For example, for tasks such as word sense disambiguation, many word senses are
not represented, or they are represented so sparsely that meaningful statistics cannot be compiled.
Similarly, many syntactic structures occur too infrequently to be significant. The Brown Corpus is also
far too small to be used for computing the bigram and trigram probabilities that are necessary for
training language models used in a variety of applications such as speech recognition. Furthermore, the
Brown corpus, while balanced for different written genres, contains no spoken English data. The 100
million words of the BNC provide a large-scale resource and include spoken language data; however,
this corpus is not representative of American English and is so far available only within Europe for
purposes of research. As a result, there is no adequately large corpus of American English available to
North American researchers for use in natural language and speech recognition work.
To meet the need for a corpus of American English, a proposal was put forward at the 1998 Language
Resources and Evaluation Conference (LREC) to create a large, heterogeneous, uniformly annotated
corpus of contemporary American English comparable to the BNC (Fillmore, et al., 1998). Over the
past two and a half years the project has developed, and a consortium of supporters including
American, Japanese, and European dictionary publishers, as well as industry, has been formed to
provide initial funding for development of the American National Corpus (ANC).
At present, the creation of the ANC is underway, using texts contributed by some consortium members
and supported by membership fees. The Linguistic Data Consortium, which will manage and distribute
275
the corpus, and is contributing manpower, software, and expertise to create a first version of the corpus,
a portion of which should be ready for use by consortium members at the end of this year.
2 Why we need a corpus of American English
There is a need for a corpus of American English that cannot be met by the data in the British National
Corpus, due to the significant lexical and syntactic differences between British and American English.
Well-known variations are: "at the weekend" (Br.) vs. "on the weekend" (U.S.), "fight (or protest)
against <something>" (Br.) vs. "fight (or protest) <something>" (U.S.), "in hospital" (Br.) vs. "in the
hospital (U.S.), "Smith, aged 36,…" (Br.) vs. "Smith, age 36…" (U.S.), "Monday to Wednesday
inclusive" (Br.) vs. "Monday through Wednesday" (U.S.), "one hundred and one" (Br.) vs. "one
hundred one" (U.S.), etc. Also, in British English, collective nouns like "committee", "party", and
"police" have either singular or plural agreement of verb, pronouns, and possessives, which is not true
of American English.
British English often makes use of a to-infinitive complement where American English does not. In the
following examples from the BNC “assay”, “engage”, “omit” and “endure” appear with a to-infinitive
complement, there were no examples found in a small corpus (comprised of selections from the Brown
Corpus, the Wall Street Journal, the San Jose Mercury News, Associated Press, and the Penn Treebank)
of this construction, although the verbs themselves did appear. For the first two verbs, one can argue
that there is not an equivalent verbal meaning in American English, but, for the last two, the meaning
can be paraphrased in American English by the gerund, as shown below. (Note that the British English
examples are from the BNC and the American English examples are paraphrases.)
Verb Eng. Example sentences
assay B.E. Jerome crept to the foot of the steps, and there halted, baulked, rather, like a startled horse, drew hard breath
and ASSAYED TO MOUNT, and then suddenly threw up his arms to cover his face, fell on his knees with a
lamentable, choking cry, and bowed himself against the stone of the steps.
engage B.E. A magnate would ENGAGE TO SERVE with a specified number of men for a particular time in return for
wages which were agreed in advance and paid by the Exchequer.
omit B.E. “What did you OMIT TO TELL your priest?”
A.E. “What did you OMIT TELLING your priest?’
endure B.E. But Carteret's wife, who frequented health spas, could not ENDURE TO LIVE with him or he with her: there
were no children.
A.E. But Carteret's wife, who frequented health spas, could not ENDURE LIVING with him or he with her: there
were no children.
Verb complementation containing prepositions often differs from British English to American English
John Algeo (1988) gives a number of examples. In British English, “cater for” and “cater to” both
occur, but “cater to” has a pejorative connotation and is less frequent. In American English, only “cater
to” is used and is not considered pejorative. British English “claim for” contrasts with American
English “claim” + NP (claim for benefits vs claim benefits), and, conversely, “agree” + NP is
acceptable in British English but not in American English, which demands a preposition such as upon,
on, about, or to. Algeo's example of British English “..yet he refused to agree the draw” would be “..yet
he refused to agree to a draw” in American English Similarly, the bare infinitive after "insist",
"demand", "require", etc. (e.g., "I insist he be here by noon.") is common in American English but rare
in British English.
Adverbial usage is also different. The British English use of “immediately” in sentence initial position
is not allowed in American English For example, British English “Immediately I get home, I will attend
to that.” is incorrect in American English, in which one would say “As soon as I get home, I will attend
to that.”
Other syntactic differences include formation of questions with the main verb “have”. In British
English, one can say, “Have you a pen?” where American English speakers must use “do” (“Do you
have a pen?”). Support verbs for nominalizations also differ : for example, the British English “take a
decision” vs the American English “make a decision”.
276
There are also considerable semantic differences between the two brands of English: in addition to
well-known variations such as lorry/truck, pavement/sidewalk, tap/faucet, presently (currently)/soon,
autumn/fall, etc., there are numerous examples of more subtle distinctions, for example: "tuition" is not
used to cover tuition fees in British English; "surgery" in British English is "doctor's office" in
American English; "school" does not include higher education in British English, etc. Usage not only
differs but can be misleading, for example, British English uses "sick" for the American "nauseous",
whereas "sick" in American English is comparable to "ill" in British English; British "braces" are U.S.
"suspenders", while "suspenders" in British English refers to something else entirely. Overall, the
distribution of various semantic classes will also distort a British and an American corpus differently,
for example, names of national institutions and positions (Whitehall, Parliament, Downing Street,
Chancellor of the Exchequer, member of parliament, House of Lords, Royal Family, the queen, senate,
president, Department of Agriculture, First Family, and heavy use of the word "state", etc.) and sports
(baseball terms will be more frequent in an American corpus, whereas hockey--itself ambiguous
between British and American usage--and soccer will predominate in the BNC). Idiomatic expressions
also show wide variation between British and American English. Of course, spoken data between the
two brands of English are not comparable at all.
The above comprise only a few examples, but it should be clear that when a uniquely British corpus is
used, such examples skew the representation of lexical and syntactic phenomena. For applications that
rely on frequency and distributional information, data derived from samples of British English are
virtually unusable. The creation of a representative corpus of American English is critical for such
applications.
3 Makeup of the ANC
Our model for the contents of the ANC includes, ideally, a balanced representation of texts and
transcriptions of spoken data, as well as a large component of annotated speech data. In addition, the
ANC should include representative samples for different dialects of American English (including
Canadian English). Finally, samples of other major languages of North American, especially Spanish
and French Canadian, should also comprise a portion of the corpus and, ideally, be aligned to parallel
translations in English. However, in view of the high cost of collection and annotation of speech data,
creation of a speech component of the ANC has been put off for the near future. Similarly, we are
delaying attempts to provide comprehensive and balanced representation of regional dialects and
collection of samples of North American Spanish and French. Our goals for the next few years of the
project therefore include collection of only textual data and transcriptions of spoken data.
The ANC will contain a static component and a dynamic component. The static component will
comprise approximately 100 million words and will remain unchanged, thus providing a stable
resource for comparison of research results as well as a snapshot of American English at the end of the
millennium. This portion of the corpus will be comparable in balance to the BNC; although there is no
set definition of "balance" in a corpus, we will follow the BNC criteria in terms of domain and
medium1 to enable cross-linguistic studies between British and American English. However, unlike the
BNC, the ANC will include primarily contemporary texts (1990 onward). The selection of
contemporary texts is important for both lexicography and NLP, particularly in view of the significant
changes in language usage over the last few years (e.g., changes brought about by electronic
communication). Therefore, the ANC static corpus will overlap with only the second time period of the
BNC.
A static corpus cannot keep up with current usage, and so the ANC will also include a dynamic
component comprised of additional texts added at regular intervals. At present, we plan to add
approximately ten- percent new material every five years in a layered organization, thus enabling
access to all layers and the static core in chronological order. In this way, we hope to provide the
advantages of both a static corpus such as the BNC and a dynamic corpus (e.g., the COBUILD corpus),
while at the same time providing a resource for studies of change in American English over time.
Beyond the 100 million words comparable to the BNC, the ANC will also include additional texts from
a wide range of styles and domains that will be varied rather than balanced; i.e., it will include smaller
samples of a greater variety of texts rather than differing percentages of texts according to their
representative importance in the language. To some extent the contents of this portion of the corpus
1 See the BNC User's Reference Guide (Burnard, 1995) for details of the criteria for balance in the BNC.
277
will be dictated by availability: we hope to take advantage of the availability of large quantities of
contemporary texts such as email, rap music lyrics, etc., as well as to add historically significant novels
and other writings. Up to now, much NLP research has been focused on newspaper or newswire text,
reflecting the availability of common corpora and annotated corpora in these areas. However, other
genres are becoming not only common but also available in massive quantities, including unedited
electronic data, email, web announcements and discussion groups, technical writing in computer
manuals, help files and telegraphic reports. These genres differ in vocabulary, names and "named
entity'’ structures (e.g., formulas, addresses, currencies, etc), syntax, lexical semantics, and discourse
structure. It has been shown that adapting to genre-specific language can significantly improve analysis
performance for syntactic structures and preferences (Sekine, 1997) and for semantic or selectional
preferences. A standard multi-genre corpus can foster research on genre adaptation, where some
experiments can be conducted on raw text data and others can be effective with small amounts of
syntactically-annotated data.
4 Encoding and annotation of the ANC
An American Natural Corpus will be most useful if it is more than just a collection of words. The
corpora that have become most useful to both publishers and researchers in natural language and
speech research have been those which are annotated. The paradigm example of this is the Brown
Corpus, which has been the cornerstone of language-related research across disciplines in the United
States, indeed in psychology as much as in natural language processing. Tagging of the Brown Corpus
has played an essential role across disciplines, both in the original version (Kucera and Francis, 1967),
and in the various on-line tagged versions, such as the Penn Treebank version (Marcus et al., 1993).
For example, many modern part-of-speech taggers are trained on the Penn Treebank tagged corpus
(see, for example, Brill, 1995). Part-of-speech tagged data has been used to automatically acquire subcategorization
dictionaries (Manning, 1993), in spell checking, and for applications that require partial
parsing, etc. The syntactic parse trees that annotate the Brown Corpus in the Penn Treebank have
played a similarly fundamental role in the training and evaluation of parsing systems.
The overall plan for development of the ANC is in two broad stages. In the first stage, a "base level"
encoding (conformant to a Level 0 encoding as specified by the Corpus Encoding Standard (Ide,
1998a,b)) of the data will be provided, by automatically transducing original printer codes to XCES
markup for gross logical structure (title, paragraph, etc.). Header information regarding target audience,
text type, etc. will be inserted manually; at this stage, only minimal header information will be
provided, based on the headers used in the BNC. This will allow us to test the applicability of the basic
BNC header to our corpus at an early stage and give us the opportunity to tune it to the needs of the
ANC in the final version. The base level encoding and annotation will be performed by the Linguistic
Data Consortium at the University of Pennsylvania (Penn).
All texts in the both the base and final versions of the corpus will be marked for major structural
divisions, paragraphs, and sentence boundaries, as well as part of speech. The base corpus will be
automatically tagged, using the part-of-speech tags of the Penn TreeBank. At this stage, only spot
checking of the data will be done; the object of this step is to harmonize the data to the extent possible
using only automated means, thus avoiding the time and cost of hand-work. The resulting base level
corpus should be sufficient for many needs (in particular, those of dictionary publishers), such as
concordance generation. Software for viewing and analyzing the corpus data in this format will be
made available to consortium members along with the data, although the data will also be available
separately.
The second stage of development will be undertaken in parallel with the first, but the exact time-frame
of the work is dependent on funding. In this stage, the corpus will be produced in its "final" form, with
the goals of (1) marking as much information in the ANC as possible while providing for maximal
search and retrieval capability, and (2) providing a "gold standard" corpus, consisting of some portion
(possibly 10%) of the entire ANC, for use in natural language processing work for training, etc. In the
final corpus, annotation for linguistic phenomena (part of speech, syntactic annotation, etc.) will follow
de facto standards such as those established by EAGLES.2
2 It will be necessary to use a larger tagset than the Penn set in the final corpus. The Penn set was designed to be
used with a corpus that was parsed, not merely tagged, and hence eliminates information that is only recoverable
from a parsed corpus, such as the distinction between prepositions and subordinating conjunctions. These were
278
Encoding of the ANC will also adhere to international standards. The corpus will be encoded according
to the specifications of the eXtensible Markup Language (XML) (Bray, et al., 1998) version of the
Corpus Encoding Standard (XCES)3 (Ide, et al., 2000), part of the Guidelines developed by the Expert
Advisory Group on Language Engineering Standards (EAGLES).4 XCES was developed expressly to
serve the needs of corpus-based work in language engineering applications by providing XML
encoding conventions for various linguistic phenomena in text and speech, as well as several types of
linguistic annotation. Because XCES is an XML application, its use for encoding the ANC guarantees
that access to and use of the corpus will be supported by tools and mechanisms designed for data
delivered via the World Wide Web.
The XCES specifies a flexible document structure that provides for "layering" annotation and related
documents that may be added incrementally at later stages. The separation of linguistic annotation in
distinct documents facilitates retrieval from different annotations (including variants of the same kind
of annotation--e.g., part of speech analysis by several taggers). This strategy of “remote” or “stand-off”
markup is well-suited to the XML environment, due especially to the development within the XML
framework of the Extensible Style Language (XSL). XSL provides a powerful transformation language
(Clark, 1999) that can be used to create new XML documents from one or several others by selecting,
rearranging, modifying and adding information to it. Thus, a user of a corpus encoded following the
XCES model need not be aware of the underlying document architecture, and will see only a new
document containing all and only the information he or she is interested in, in any desired configuration
and encoding. This will enable use of the ANC for a potentially limitless set of applications, including
not only computational linguistics research but also education, etc. The XCES architecture also
provides for distribution of development and enhancement, by enabling different sites to develop
separate documents containing annotations for the primary ANC data, all of which are ultimately
linked together and retrievable as a hyper-document.
In the second stage of ANC development, at least the following tasks will be undertaken:
·  Validation and refinement of existing markup, e.g., changing paragraph markers to more precise
tags such as list, quote, etc., marking highlighted words for function (e.g., foreign word, emphasis,
etc.);
·  Provision of a full XCES-compliant header, including a full description of provenance and all
encoding formats utilized in the document. In this phase we will correct, where necessary,
categories and other header information drawn from the BNC in the first phase, and substantially
add to it;
·  Insertion of additional markup for sub-paragraph elements, such as tokens, names, dates, numbers,
etc. Identification of these elements will, to the extent possible, be done automatically;
·  Hand validation of markup for sub-paragraph elements, including sentence, token, names, dates,
etc., in the "gold standard" portion of the corpus;
·  Transduction of part-of-speech markup to XCES specifications, and possible transduction of
annotation categories to a standard scheme such as the EAGLES morpho-syntactic categories
(Monachini and Calzolari, 1996);
·  Hand-validation of part-of-speech tags in the "gold standard" portion of the corpus;
·  Implementation of the layered data architecture for annotations;
·  Adaptation and/or development of search and retrieval software, together with development of
XSLT scripts for common tasks such as concordance generation, etc.
5 The ANC consortium
Founding consortium members contribute US$21,000 over 3 years in annual installments of $7,000,
which will be used to support the development of the base level corpus. In addition, publishers and
other members are expected to provide contributions of data for inclusion in the corpus. Consortium
combined into the single tag IN in the Penn tagset, since the tree-structure of the sentence disambiguated them
(subordinating conjunctions always precede clauses, prepositions precede noun phrases or prepositional phrases).
3 See http://www.cs.vassar.edu/XCES.
4 http://www.ilc.pi.cnr.it/EAGLES/home.html
279
members who join after March 31, 2001, contribute $40,000 in two annual installments. Consortium
members receive the data as soon as it is processed and have exclusive commercial rights to it for a
period of five years after the date of the first release of data, currently anticipated to begin at the end of
this year. Current consortium members are:
·  Pearson Education
·  Random House Publishers
·  Langenscheidt Publishing Group
·  Harper Collins Publishers
·  Cambridge University Press
·  LexiQuest
·  Microsoft Corporation
·  Shogakukan,Inc.
·  Associated Liberal Creators Press
·  Taishukan Publishers
·  Oxford University Press
·  Kenkyusha Publishers
·  International Business Machines Corporation
All ANC data will be freely available to non-profit educational and research organizations from the
outset (aside from a nominal fee for licensing and distribution). There will be no restrictions on
obtaining the corpus based on geographical location; restrictions on the distribution of the BNC, which
has so far been unavailable outside the European Union, have limited large-scale and comparative
research based on the corpus. We hope to encourage comparative research by providing global access.
The Linguistic Data Consortium will obtain licenses from text providers and provide licenses to users.
In general, the license will prohibit redistribution of the corpus and the publication or similar use of
substantial portions of text drawn from the corpus without the permission of its original publisher. For
dictionary makers, who comprise a large portion of the current consortium membership, usage of short
portions of text in published dictionary examples etc. is allowed under legal definitions of "fair use".
We also plan to provide for an “open sub-corpus”, licensed to permit redistribution on the model of
open-source software. The size of this corpus will be determined by the contributors.
Development of the Level 1 corpus and the "gold standard" sub-corpus will necessarily begin later than
development of the base-level version, due to the need to secure substantial funding from external
sources to support it. In addition, this development requires time for significant planning to ensure that
the corpus is maximally usable by a broad range of potential applications and meets the needs of the
research and industrial communities. We are currently soliciting input from the research community to
feed this development. A meeting on the topic of annotation and encoding formats and data
architectures for large corpora was held at last year's ANLP/NAACL conference in Seattle in early
May5; another more comprehensive workshop on the same topics was held preceding the LREC
conference in Athens in June, 2000 (Broeder, et al., 2000). By taking into account past experience,
current and developing technologies, and user needs, we hope to be able to provide a state-of-the-art
platform for universal access to the ANC.
6 Summary
The ANC, initially proposed at the first LREC in 1998, is now well on its way to realization. Within the
year, the first data in its base level representation will be available to the NLP community and
consortium members. The final corpus in its fully marked and annotated form should be available
within three years.
A corpus of contemporary American English is a valuable resource not only for commercial
applications and research, but also for educators, students, and the general public. It is also an
important historical resource: the corpus will provide a "snapshot" of American English at the turn of
the millennium, valuable for linguistic studies in the decades to come.
5 Papers available at http://www.cs.vassar.edu/~ide/ANLP-NAACL2000.html.
280
Acknowledgments
The first ANC meeting in Berkeley, California was funded by National Science Foundation grant ISI-
9978422. We would like to thank Sue Atkins, Michael Rundell, and Rob Scriven for their support and
for providing information concerning the creation of the BNC. We would also like to acknowledge the
contribution of Wendalyn Nichols, Frank Abate, and Yukio Tono, who have been instrumental in
obtaining the support of the publishing community.
References
Algeo J 1988 British and American grammatical differences. International Journal of Lexicography,
1:1-31.
Bray T, Paoli J, Sperberg-McQueen CM (eds) 1998 Extensible markup language (XML) Version 1.0.
W3C recommendation. http://www.w3.org:TR/1998/REC-xml-19980210.
Brill E 1995 Transformation-based error-driven learning and natural language processing: A case study
in part-of-speech tagging. Computational Linguistics, 21(4):543-566.
Broeder D, Cunningham H, Ide N, Roy D, Thompson H, Wittenburg P (eds) 2000 Proceedings of the
EAGLES/ISLE Workshop on Meta-Descriptions and Annotation Schemas for Multimodal/Multimedia
Language Resources and Data Architectures and Software Support for Large Corpora. Paris,
European Language Resources Association.
Burnard L 1995 British National Corpus: User's reference guide for the British National Corpus.
Oxford, Oxford University Computing Service.
Clark J (ed) 1999 XSL transformations (XSLT). Version 1.0. W3C recommendation.
http://www.w3.org/TR/xslt
Clark J (ed) 1999 XSL transformations (XSLT). Version 1.0. W3C recommendation.
http://www.w3.org/TR/xslt.
Fillmore C, Ide N, Jurafsky D, Macleod C 1998 An American National Corpus: A proposal. In
Proceedings of the First Annual Conference on Language Resources and Evaluation, Granada, pp
965-969.
Ide N 1998a Encoding linguistic corpora. In Proceedings of the Sixth Workshop on Very Large
Corpora, Montreal, pp 9-17.
Ide N 1998b Corpus Encoding Standard: SGML guidelines for encoding linguistic corpora.
Proceedings of the First International Language Resources and Evaluation Conference, Granada, pp
463-70.
Ide N, Bonhomme P, Romary L 2000 XCES: An XML-based standard for linguistic corpora. In
Proceedings of the Second Annual Conference on Language Resources and Evaluation, Athens, pp
825-30.
Kucera H, Francis W 1967 Computational analysis of present-day American English. Providence,
Brown University Press.
Leech G, Garside R, Bryant M 1994 CLAWS4: The tagging of the British National Corpus.
Proceedings of COLING-94, Nantes, pp 622-628.
Manning C 1993 Automatic acquisition of a large subcategorization dictionary from corpora.
Proceedings of ACL, Columbus, pp 235-242.
Marcus M, Santorini B, Marcinkiewicz 1993 Building a large annotated corpus of English: The Penn
Treebank. Computational Linguistics, 19(2): 313-330.
Monachini M, Calzolari N. 1996 Synopsis and comparison of morphosyntactic phenomena encoded in
lexicons and corpora: A common proposal and applications to European languages. EAGLES report
EAG-CLWG-MORPHSYN/R. http://www.ilc.pi.cnr.it/EAGLES96/morphsyn/.
Sekine S 1997 The domain dependence of parsing. In Proceedings of the Fifth Conference on Applied
Natural Language Processing.
Sinclair J 1991 Corpus, concordance, and collocation. Oxford, Oxford University Press.


281
The positions of reporting clauses of speech presentation with special reference to
the Lancaster Speech, Thought and Writing Presentation Corpus
Reiko Ikeo, Lancaster University
(rikeo@hotmail.com)
Abstract
This paper reports the corpus findings on the positions of reporting clauses of speech presentation.
In direct speech, the preferred positions of reporting clauses vary according to text types, and the
results are compared with those of Biber et al. (1999). In indirect speech, more than 90% of reporting
clauses are found in the initial position. Although fronted reported clauses of indirect speech in fictional
texts have been treated as an intermediate form between indirect and free indirect speech, the corpus
data from news reports shows textual/contextual conditions where reported clauses tend to be fronted.
This phenomenon in the press data seems to have different motivations from the one in fiction which
often manipulates shift of viewpoints.
1. Introduction
The report of other people's discourse is at the core of narrative and journalistic discourse.
Reporting clauses in discourse presentation are one of the most explicit linguistic devices which
introduce other people's discourse in texts. The various positions of reporting clauses have both
syntactic and pragmatic importance in discourse presentation. The positions of reporting clauses can
affect the syntactic relationship between the reporting and the reported clauses. In direct speech (DS),
clear markers of the reported clauses such as quotation marks, the verb tenses and pronoun uses
distinguish the reported from reporting clauses even when reporting clauses are placed after the
reported clauses. Biber et al. (1999: 196) define the syntactic status of reporting clauses of DS as
intermediate between independent and dependent clauses, and this definition does not seem to be
affected by the varied positioning of reporting clauses. On the other hand, in the case of indirect speech
(IS), the syntactic relation between reported and reporting clauses can be affected by their relative
positions. If reported clauses are placed before reporting clauses, a ‘that'-complementiser is
compulsorily omitted. Placing reporting clauses in the medial and final positions also allows a question
form or exclamatory sentence in the reported clauses. As a result, the reported clauses may gain more
syntactic freeness than when they are subordinated to reporting clauses in the initial position.
Point of view is another issue which is deeply involved in discourse presentation. In DS, the
reported speaker(s)’ point of view is clearly distinguished from that of the reporter or other reported
speakers by quotation marks and choices of deixis and verb tense. Fronted reported clauses (i.e. before
reporting clauses) can increase the effect of immediacy. In IS, on the other hand, the narrator's point of
view intervenes more in the reported clauses and is reflected in the choice of pronouns, lexis and verb
tenses. When reported clauses are placed before reporting clauses, the reader might not notice the
reporting clause and may at first take the reported clause to be free indirect speech (FIS) or narration
before recognising the reporting clauses. This phenomenon has been an interest in the field of stylistics,
and several researchers1 have discussed it in relation to FIS in fictional texts. Leech and Short (1981)
refer to the effect of inversion in IS and locate the constructions ‘somewhere in between indirect speech
and free indirect speech’ (p.333). However, this phenomenon has hardly been examined outside literary
texts. One of the purposes of this paper is to examine the inversion of the reporting and reported
clauses of IS in journalistic texts.
Different modes of presentation of reported clauses seem to have different preferences for the
positions of reporting clauses. These differential preferences are for syntactic and pragmatic reasons.
The type of texts in which discourse presentation occurs also may affect the positions of reporting
clauses. Biber et al. (1999) report from their corpus findings that the final position of reporting clauses
of direct discourse presentation is highly favoured in both news and fiction.
The source of my data in this paper is the Lancaster Speech, Thought and Writing Presentation
Corpus (ST&WP Corpus). This corpus of 260,000 words was annotated manually for categories of
speech, thought and writing presentation using a tagset which was developed by the Lancaster research
team and based on Leech and Short's 1981 model. Biber et al. report their findings based on a sample
of 100,000 words from the Longman Spoken and Written English Corpus (the LSWE Corpus). The
ST&WP Corpus is 2.6 times larger than the LSWE sample corpus. The ST&WP Corpus has three
sections of texts: news, fiction and (auto)biographies. Each of three sections is subdivided into



Reinhart (1975), Banfield (1983) and Fludernik (1993)


282
serious/popular divisions. These divisions make comparisons between serious and popular types of
texts possible as well as between different genres.
In this paper, I concentrate on speech presentation and make the following general points:
(1) The preferred positions of reporting clauses of DS seem to vary according to text types and the
serious/popular divisions.
(2) Although more than 90 % of reporting clauses of IS are placed before the reported clauses in all
three genres, news reports seem to have particular textual/contextual patterns where the reporting
clauses of IS are placed either in the middle or at the end of the reported clauses.
2. Reporting clauses of DS
There are 2878 reporting clauses in the corpus, and about 94% of reporting clauses are attached
either to DS or IS: 53.2% are attached to DS and 40.6% are attached to IS (Table 1). About 6% of
reporting clauses accompany other types of speech presentation such as free direct speech (FDS) or
FIS.
DS (%) IS (%) DS+IS (%) Total (%)
1530 (53.2) 1169 (40.6) 2699 (93.8) 2878 (100)
Table 1 The numbers of reporting clauses of DS and IS
Table 2 shows that news reports and fiction both have about 600 reporting clauses of DS, and
(auto)biographies have 360. Thus the (auto)biography samples have less DS than news reports or
fiction. This is probably because biographers tend to lack direct access to the original speech of the
people who are involved and the nature of the non-fiction genre discourages the biographer to quote
protagonists’ speech as a faithful reproduction.
Initial (%) Final (%) Medial (%) Total (%)
News 394 (66.4) 197 (33.2) 2 (0.3) 593 (100.0)
Fiction 115 (20.0) 404 (70.2) 56 (9.8) 574 (100.0)
(Auto)biography 163 (44.9) 156 (43.0) 44 (12.1) 363 (100.0)
Table 2 The positions of reporting clauses of DS
The preferred positions of reporting clauses for DS vary in the three genres. In news reports, the
initial position is more preferred than the final or the medial position. In fiction, on the contrary, the
final position is much preferred. In (auto)biographies, reporting clauses are evenly distributed in the
initial and the final positions. Reporting clauses in the middle position are hardly found in news reports
while in fiction and (auto)biographies about 10% of the reporting clauses are inserted in the middle of
the reported clauses.
The sample corpus from the LSWE Corpus suggests different distributions of reporting clauses of
DS in the three positions.
Initial (%) Final (%) Medial (%)
News 35 50 10
Fiction 15 65 15
Table 3 The positions of reporting clauses of DS in the LSWE Corpus2
The two corpora show similar results in fiction; reporting clauses in the final position are
preferred in both corpora. On the other hand, in news reports, most reporting clauses are found in the
initial position in the ST&WP Corpus whereas half of the reporting clauses are found in the final
position in the LSWE Corpus. While reporting clauses in the medial position are scarcely found in the
ST&WP Corpus, 10% of reporting clauses are found in the medial position in the LSWE Corpus. One
possible explanation for the different results about the most preferred positions and the wide gap in the
percentages of the middle position in the news texts is that the data sources are differently comprised in
the two corpora. The news texts in the LSWE Corpus include articles on other topics than news reports
such as sports and culture whereas the ST&WP Corpus concentrates on news reports on
international/domestic matters. The articles on cultural topics could have different styles from that of
typical news reports and may have more reporting clauses in the final and medial positions. Another
difference in the news texts in the two corpora is that in the LSWE Corpus 60% of the data is from
broadsheets and 40% is from regional papers while, in the ST&WP Corpus, 50% of data is from
broadsheets and 50% is from tabloids3. The proportion of tabloid papers as a data source in the ST&WP

2 The figures are based on Table 11.1 of Biber et al. (1999: 923).
3 Broadsheets: Independent on Sunday, Guardian, Independent, Daily Telegraph, and Times. Tabloids: News of the World,

283
Corpus can be higher than in the LSWE Corpus, assuming that regional papers are either more similar
to broadsheets or in-between of broadsheets and tabloids. The higher proportion of the tabloids may be
one of the factors which raise the percentage of the reported clauses in the initial position in the
ST&WP Corpus. As the data from ST&WP Corpus shows in the next section, initial reporting clauses
are preferred in tabloid papers compared with the broadsheets.
3. Reporting clauses of DS and the serious/popular divisions
From now on, my discussion concentrates on my findings based on the ST&WP Corpus. In this
section, DS reporting clauses are examined across text types and their popular/serious divisions. As a
general tendency, the popular divisions of all three genres have more DS than the serious divisions,
reflecting the style difference which the serious/popular contrast would predict (Table 4).
Broadsheets/Serious (%) Tabloids/Popuar (%) Total (%)
News 234 (39.5) 359 (60.5) 593 (100.0)
Fiction 244 (42.5) 330 (57.5) 574 (100.0)
(Auto)biography 81 (22.3) 282 (77.7) 363 (100.0)
Table 4 DS in the popular/serious divisions
In news reports, the tabloids have roughly half as much DS again compared with the broadsheets.
Furthermore, the tabloids tend to have more reporting clauses in the initial position than the other two
positions (Table 5). In the DS mode, the reported clause can produce more immediacy than IS and
allow a wider range of registers from formal public speech to casual colloquial speech. The initial
position of reporting clauses makes the reader's processing of the speech presentation easier than other
positions. The tabloid papers may well favour these advantages of DS with reporting clauses in the
initial position. In contrast, the broadsheets show no distinct preference for either the initial or the final
position.
News Initial (%) Final (%) Medial (%)
broad tabloid broad tabloid broad tabloid Total No. of DS (%)
Broad=234 (100)
Tabloid=359 (100)
117
(50.0)
277
(77.2)
117
(50.0)
80
(22.3)
0
(0.0)
2
(0.5)
Table 5 The positions of reporting clauses of DS in News
In fiction, DS is also preferred in the popular division compared with in the serious division
(Table 6). As for the positions of reporting clauses, both popular and serious fiction prefers the final
position. Since DS in fiction is most often indented as well as accompanied by the quotation marks, the
reader is probably already aware that a character's speech is being presented at the beginning of
reported clauses even if reporting clauses are postponed until the end of the sentences. Such
graphological practice in fiction can help the reader's processing. In addition, in fictional contexts, a
range of choice of speakers is much more limited than in journalistic contexts, which often offer almost
unlimited possibilities of speakers. The reader may have less difficulty in recognising a reported
speaker with the reporting clause at the final position in fiction.
Fiction Initial (%) Final (%) Medial (%)
serious popular serious popular serious popular Total No. of DS (%)
Serious=244 (100)
Popular=330 (100)
45
(18.4)
70
(21.2)
174
(71.3)
229
(69.4)
25
(10.3)
31
(9.4)
Table 6 The positions of reporting clauses of DS in Fiction
By placing the reported clauses before the reporting clauses, the narrator's intervention is
postponed and the characters’ speech can be presented almost like a play script. Fronted DS clauses
give vivacity to characters’ speech presentation and generate dramatic effects in fictional worlds.
In (auto)biographies, there are 363 reporting clauses attached to DS and 282 reporting clauses are
found in popular (auto)biographies whereas only 81 reporting clauses are found in serious
(auto)biographies (Table 7). Popular (auto)biography does not show particular preference for either
initial or final position of reporting clauses. But it does have a higher percentage of reporting clauses in
the medial position, compared with the serious (auto)biographies or the other two genres.

Express, Mirror, Star and Sun. Intermediate: Today.


284
(Auto)biography Initial (%) Final (%) Medial (%)
serious popular serious popular serious popular Total No. of DS (%)
Serious=81 (100)
Popular=282 (100)
52
(64.2)
114
(40.4)
26
(32.1)
131
(46.5)
3
(3.7)
40
(13.1)
Table 7 The positions of reporting clauses of DS in (Auto)biographies
Out of 40 examples of reporting clauses in the medial position in the popular division, 35
examples appear in the autobiographies, and 5 in the biographies. In popular autobiographies, the
protagonists’ speech often embeds another narrative: it can be a story or a joke. The following example
is from an autobiography by the actor, Michael Caine, to whom a comedian, Eric Sykes, is talking at a
party. He is joking that Brigitte Bardot, a famous actress, ignores him because she loves him.
(1) ‘She is in love with me,’
he whispered,
‘and she can't stand it. Watch her as she goes by,’
(Michael Caine, What's It All About?)
The reporting clause comes at the clause boundary in the reported clause. The second reported
clause gives a reason the super star ignores the comedian, which renders a humorous effect. By
inserting the reporting clause before the punch line, the reader's expectation is suspended and the
punch line is made more effective. Out of 40 examples, 27 examples have reporting clauses at a clause
boundary. Such interruption suspends the onset of the latter part of the speech momentarily, making the
reader more attentive to it. The subjects of popular autobiographies are often famous actors, athletes
and comedians, whose images and tones of voice the reader is likely to be familiar with. A casual,
colloquial style which reflects the celebrity's image is commonly shared in such autobiographies, and
this tendency is especially strong in direct speech. Inserting a reporting clause in the middle of a
reported clause with a colloquial register, the reader can have processing time imagining the character's
speech with more immediacy and get ready for the following speech which often requires more
inferential work than the first part.
On the other hand, since serious (auto)biographies tend to aim for accuracy of record on past
events rather than immediacy and drama, specifying the speaker of the following speech presentation at
the beginning of the sentence is a reasonable measure in order to avoid the ambiguity over the speakers.
4. Reporting clauses of IS
In IS, the initial position of reporting clauses is a more established, syntactically-determined
pattern. More than 90% of reporting clauses are fronted in all three genres (Table 8).
Initial (%) Final (%) Medial (%) Total (%)
News 650 (92.9) 42 (6.0) 8 (1.1) 700 (100.0)
Fiction 110 (90.1) 2 (1.7) 9 (7.4) 121 (100.0)
(Auto)biography 327 (94.0) 3 (0.8) 18 (5.2) 348 (100.0)
Table 8 The positions of reporting clauses of IS
A few reporting clauses come after the reported clauses or are inserted in the middle of the
reported clauses, which, as we have already noticed, occurs much more commonly in DS. Stylistic
accounts regard such inversion of reporting and reported clauses as an intermediary form between IS
and FIS and link it with free indirect discourse in literary texts. However, in these discussions other
aspects of a character's subjectivity such as lexis, ejaculative expressions and deixis in the reported
clauses are also taken into consideration. Above all, FIS involves more characters’ points of view and
less intervention from the narrator than IS. It should be further examined whether placing a reported
clause before a reporting clause can automatically generate shifts of viewpoint. News reports offer
useful data in this respect because reported clauses in news texts tend to contain less subjective
elements than those of fiction and the inversion of the reporting and the reported clauses can be
analysed independently of other elements of FIS. If the inversion itself has an effect which suggests a
clearly speaker-oriented perspective, it will push the reported clause toward FIS.
5. Reporting clauses of IS in the final/middle position of news reports
Table 8 shows that reporting clauses of IS in the final position most frequently occur in news
reports while fiction has much less. This is quite remarkable when we remember that in DS reporting
clauses in the final position is most preferred in fiction. The phenomenon looks contradictory at first
sight because the fronted reported clause of IS can sometimes be ambiguous with narration, and news
reports are supposed to avoid any ambiguities concerning speech presentation. By examining the
examples of news reports, three major textual patterns can be found, in which the inversion of the

285
reporting and the reported clauses do not necessarily generate ambiguities or involve shifts of the
points of view.
5. 1. The first sentence of the body text
The first major textual pattern is that the sentence appears as the leading sentence of the body text
immediately after the headline. As Table 8 indicates, there are 42 examples of IS with reporting clauses
after the reported clauses. Out of these 42 examples, 13 (31%) are the first sentence of the body text.
This tendency is the same for both broadsheets and tabloids. Out of these 13 cases, in 12, the contents
of the reported clauses are previously announced by the headlines. Consider:
(2) <headline >
Britain ready to pull troops out of Bosnia
<body text>
BRITAIN could start withdrawing its troops from Bosnia within weeks if the warring factions reject the latest plans for a
settlement, Douglas Hurd, the Foreign Secretary, said yesterday.
(The Times, 5/12/94, Britain ready to pull troops out of Bosnia)
The headline gives the reader a general idea on the topic of the article; more precisely, the headline
summarises the reported clause of the first sentence of the body text. The content of the reported clause
gives more detailed information in relation to the main headline. The combination of the headline and
the reported clause of the first sentence repeats the general content of the headline and reinforces the
impact of information which the article conveys. From the perspective of ‘given/new information', the
reported clause of the first sentence is textually ‘given’ since it is previously mentioned in the headline.
On the other hand, the reporting clause, which specifies the information source, is ‘new'. The
construction of the first sentence observes the ‘end-weight’ principle of informational structure (Quirk
et al. 1985: 1365-6) although it can deviate from the normal syntactic structure of IS. It should be noted,
however, that ‘end-weight’ does not necessarily entail ‘prominence', or, in more practical terms, ‘news
value'. In some cases, the sources of information are mentioned in less specific ways. Compare:
(3) <headline>
2 BRITS AMONG DEAD IN PLANE HORROR
<body text>
A BRITISH couple were among the 109 killed in Saturday's Florida jet crash, it was revealed last night.
(The Sun,13/5/96, 2 Brits among dead in plane horror)
Such an unusual order of reporting and reported clauses in the first sentence of the body text
makes a contrast with the normal clausal order in a similar textual environment. When the reporting
clause is at the beginning of the first sentence, the importance of the syntactic subject of the reporting
clause is stressed. In the following example, the subject of the reporting clause appears in the headline,
and the full name of the speaker is again indicated in the first sentence. Compare:
(4) <headline>
Blair backs Straw over a new role for the monarchy
<body text>
TONY BLAIR confirmed yesterday that he intended to make fundamental changes to the role of the Royal Family an issue
in the next general election.
(The Times, 5/12/94, Blair backs Straw over a new role for the monarchy)
There are 23 cases where the speaker of the indirect speech appears in the fronted reporting clause as
well as in the headlines. In such cases, the source of information, the speaker, can express the ‘topic’
and the reported clause can express the ‘comment'4. In (4), the ‘topic’ is Blair, who is already known to
the reader, and the ‘comment’ is what he said. The pragmatic functions of ‘topic’ and ‘comment’
explain why famous individuals tend to be found as the syntactic subject of fronted reporting clauses.
Out of 23 cases, 18 have personal names of politicians, show business people or royals in the fronted
reporting clauses.
The inversion of the reported and reporting clauses in the first sentence of a news article seem to
have a different motivation from that of fiction. In news reports, the inversion of the two clauses are
mainly related to the shifts of focus between the speaker, i.e. the source of information, and the content



Van Dijk (1977: 116) defines a topic as ‘some function determining about which item something is being said; it often
associated with what is already known in some context, or what is presupposed.’ He refers to the ‘comment’ as what is
‘unknown’ and asserted.


286
of speech. In the above examples, the positions of reporting clauses are especially affected by the
relationship between the headline and the first sentence. On the other hand, in fiction, the issue has
more to do with changing points of view and the distance which the narrator intends to take between
the character who speaks and him/herself. As a representative example which clearly demonstrates shift
of viewpoints, I will quote a short passage by V. Woolf outside the corpus. There are three reporting
clauses in the passage, and two of them are in the final position, one in the middle.
(5) (a) ‘Do you write many letters, Mr. Tansley?’ asked Mrs. Ramsay, pitying him too, Lily supposed; for that was true of Mrs.
Ramsay - she pitied men always as if they lacked something - women never, as if they had something. (b) He wrote to his
mother; otherwise he did not suppose he wrote one letter a month, said Mr. Tansley, shortly.
(Virginia Woolf, To the Lighthouse p.93; alphabetising mine)
In (a), the first reporting clause is attached to Mrs. Ramsay's direct speech. The second reporting clause
is inserted after an adverbial phrase with a present participle ‘pitying him'. This reporting clause
suggests that the point of view is suddenly shifted from the narrator to Lily Briscoe, and she takes over
the role of telling Mrs. Ramsay's inner state when Mrs. Ramsay passes a conversational turn to Mr.
Tansley. The fronted indirect form in (b) suggests both immediacy and Mr. Tansley's reserved manner.
The final reporting clause not only specifies the speaker of the preceding speech but also signals whose
point of view is presented from now on; a new paragraph which follows the present passage
concentrates on reporting Mr. Tansley's thought.
In fiction, the positions of reporting clauses is one of the devices which can manipulate the
characters’ or the narrator's points of view. Especially in IS, even if the reported clause has no
particular signs which are more attributable to the character than the narrator such as deixis or lexis,
postposing the reporting clause after the reported clause generates some character-bound effect. This
kind of rapid viewpoint switching is not usually a feature of news reports. In news report, the positions
of the reporting clause seems to be more dependent on textual organisation and pragmatic elements
rather than shifting viewpoints.
5. 2. Continuous speech presentation by the same speaker
Another textual feature of news reports, surrounding a postponed reporting clause, is that a
fronted reported clause comes immediately after another speech presentation by the same speaker. 16
cases out of 50 with inverted reporting clauses are part of a continuous speech presentation by one
speaker. All of the 16 cases except one are found in broadsheets. In these cases, the reporters seem to
reorder reporting and reported clauses as one of the devices for re-constructing the speaker's speech. In
(6), each reporting and reported clause is alphabetised for ease of reference.
(6) (a) Mr Major warned yesterday of the dangers of Britain being left behind if a group of European Union members pushed
ahead with a single currency.
(b) Nobody could be certain,
(c) he said,
(d) of the economic impact on the UK.
(Independent on Sunday,11/12/94, Blair puts Labour troops on alert for snap election)
In (6), the reporter summarises Mr Major's speech by using the mode of IS in (a). After this summary,
the reported speaker's more specific wording is introduced in (b) and (d) in order to back up the
previous summary. It is clear that the reporter intentionally chose a particular part out of Mr Major's
speech which would most strongly support his/her previous generalisation. By fronting the reported
clause, the rhetorical connection between the summary (a) and the specification (b) and (d) are
tightened. In (7), too, the reporter seems to apply a similar strategy.
(7) (a) But Mr Lilley said
(b) Labour tactics could prompt many ordinary people who would otherwise support Labour to turn to the Conservatives.
(c) “I regret very much that they have put the future of the monarchy into the political domain,”
(d) he said on BBCl's Breakfast with Frost.
(e) “But having done so, I think that they risk losing the support of a lot of their voters.”
(f) While Labour activists were Left-wing, Labour voters were usually “very pro- monarchy, very pro-Britain” and the
Conservatives would “vigorously defend” the Queen and the Royal Family.
(g) Having abandoned its policies on the economy and education, Labour desperately needed something new to please
theLeft,
(h) he said.
(The Daily Telegraph, 5/12/94, Labour in row over Royal role)
A generalisation of Mr Lilley's speech is introduced as IS in (a) and (b). The following DS in (c) and

287
(e), FIS in (f) and the inverted reported clause of IS in (g), all not only represent Mr Lilley's specific
wording but also explain the reasons why Labour tactics could lose the potential voters against the
Conservatives. These reported clauses, however, may not be presented in the same sequence as that of
Mr Lilley's actual speech. The reporter could have rearranged the order of these clauses so that the each
segment is organised according to his logic. Fronting the reported clause (g) seems to be one of
strategic devices to link the reported clause logically with the reporter's summary of Mr Lilley's
speech.
5. 3. An introduction of a new speaker at the end
The third pattern of placing reporting clauses in the final or medial position is more
context-dependent than affected by textual organisation. Reporting clauses can ‘abruptly’ appear after
fronted reported clauses without any introduction of the speaker before. There are 11 such cases in the
corpus, out of which 10 reporting clauses are at the end. These ‘abrupt’ reporting clauses are equally
distributed across the broadsheets and tabloids. In those examples, the contents of the reported clauses
are often additional or secondary to the preceding information, which has more importance or impact.
In the following example, the reported clause has its reporting clause at the end. The content of the
reported clause (b) has less news value than the preceding information (a), and consequently, the
information source specified in (c) is even less important.
(8) (a) A powerful bomb ripped through a bus in Pakistan yesterday, killing at least 40 people. Most of the passengers were
returning home to celebrate the Muslim festival of Eid al-Adha.
(b) A second bomb was found near the wrecked bus and safely defused,
(c) the official Associated Press of Pakistan news agency said.
(The Times, 29/4/96, Bomb on crowded bus kills 40 in Pakistan)
In (8), no clues that indicate speech presentation are found in (b) after quite a long stretch of narration.
The reader will not recognise (b) as a reported clause until he/she reads the reporting clause of (c). The
possible reason for the inversion here would be that the reporting clause receives the least emphasis
because the information of the reported clause is secondary as news. Different degrees of emphasis on
the reporting and the reported clauses seem to affect the order of the two clauses. Another aspect of this
structure is that information sources in the reporting clauses are often institutions or groups rather than
individuals such as ‘the official Associated Press of Pakistan news agency’ and ‘the UN'. Out of 13
cases, 6 mention groups and organisations as information sources. Such institutional group
speakers/writers tend to be backgrounded because they have less news value. This makes a contrast
with the situation where famous individuals such as politicians and celebrities are often found in the
fronted reporting clauses in the first sentence of the body text.
6. Conclusion
This research supported our intuitive perception about the positions of reporting clauses in direct
speech and indirect speech by the quantitative data which was obtained from the ST&WP Corpus: DS
has more reporting clauses in the middle and the final positions of the reported clauses compared with
IS, while in IS the order of the reporting and the reported clauses is syntactically established and the
inversion of the two clauses occurs much less. The corpus findings also quantitatively suggested that
text types are closely related to the preferred positions of reporting clauses in DS. These results were
compared with those of the LSWE sample Corpus by Biber et al. (1999). In IS, the press data gave a
new perspective to the inversion of the reporting and the reported clauses. Although placing the
reported clause before the reporting clause can be an unusual syntactic structure, this construction tends
to be more affected by textual organization and often obeys the informational principle. Here, not only
the quantitative analysis of the corpus data but also the qualitative approach successfully revealed some
of the motivations for this construction of news reports, which seem to be different from those of
fiction.
References
Banfield A 1982 Unspeakable sentences: narration and representation in the language of fiction.
London, Routlege & Kegan Paul.
Biber D, Johansson S, Leech G, Conrad S, Finegan E 1999 Longman grammar of spoken and written
English. London, Addison Wesley Longman Dictionaries.
Fludernik M 1993 The fictions of language and the languages of fiction: the linguistic representation of
speech and consciousness. London, Routledge.
Leech G, Short M 1981 Style in fiction. London, Longman.
Quirk R, Greenbaum S, Leech G, Svartvik J 1985 A comprehensive grammar of the English language.
London, Longman.

288
Reinhart T 1975 Whose main clause?: point of view in sentences with parentheticals. Harvard Studies
in Syntax and Semantics 1: 127-71.
van Dijk T 1977 Text and context: explorations in the semantics and pragmatics of discourse. London,
Longman. 
Texts
Caine M 1992 What's it all about? London, Century.
Woolf V 1992 To the lighthouse. London, Penguin.

289
Multiple-level knowledge discovery from corpus
Wang JianDe, Chen ZhaoXiong, Huang HeYan
Institute of Computing Technology
Chinese Academic of Science
PO BOX 2704 , Beijing China
100081
wangjiande@sina.com
Multiple-Level Knowledge includes knowledge of morphology, syntax, semantics, and style. These
four levels of knowledge correspond to the levels of natural language, and thus can be of use in
language learning, language engineering, language processing and others areas related to linguistics.
Corpora contain large amounts of language data, marked up with a set of attributes; we can derive
knowledge of linguistic rules from the annotated corpus.
1) Lexical knowledge can be obtained from statistics derived from the corpus.
2) Knowledge of syntax also can be obtained from corpus data, by means of the Machine Learning
algorithm.
3) Semantic rules can be found in the classified document of different semantic type through the
statistic model and semantic learning function.
4) A style model can also be derived from these documents. According the arrangement of feature
words and syntactic structures in the documents and the learning algorithm of Style Model.
The four levels of knowledge are not independent; they interconnect, and can convert to each other in
some time.
This system is intelligent. The information of user operation and the corpus are stored as the translation
resource. With the statistical model and Machine Learning algorithm, some knowledge can be
extracted from the resource. The system combines the different methods of machine translation that
abstract the advantage of RBMT and EBMT, and use the database to store the sentence human
translated. The knowledge of translation is divided into three kinds: public, protected and private, that
can be used to different translators. So when MT uses this knowledge, the system is more and more
intelligent.
The above discovery can be automatic, but it needs much more corpus data, especially annotated data.
So we are designing the computer-aided Multiple-Level Knowledge Discovery tool.


 


	

	 
	 
 			 !"	
#$%&&'(%

 
		
  !"#    $%#%%%   	 	
	&'#& 

	  "##	#

 
	 	 	 	 	 	 	 		
	    	! "	
	#  
	 !	 	  	$ %    
		&'		%
!	(

)* +,-+,.+,/

* 0-0.0/
1* +,-0.0/
2* 0	-&.0/
3* 0-+,.0/
4* +,-+,	.0/
'* 0-&.0

		%		%$ 
&	 	    	 	
	 56 *  	 	    )2  		 
!	 	2 

7 )% 		  	 
& 7 		  	   !  	 %     	
	  &  	 !% 	! !  	 	 	 $  
	8	       	 7   
   	  
&		 		%	&7 1
% & 	 		%  6	  
5	* 	  		 	 $ 	   9  : 
	  8%    !, ;    2 	  ::
 					



 
 	
 
 
 
 
 
 
 
 
 
 
 
 	

 !
"#
$




 
)
 
< Ó  !			
 !
  Ó =%& 
  ¼   	 


 ¸ !   
			    ) &  % * !(
ÿ
þ
ý
¸

=
 
 
 

%
&
+ , ÷ 
				!		  	, 
 			 
   	!  	 	 	9	- 5*0.
0(
) 0-0.0

  

 +,-0.0
  
1 >	!-0.0
	  
2 0-0.0
  
3 	-0.0
  
4 ?
@)A: !	!  6   
  	&				 
7  %%				
;	& !"  ¸ ! !"#   %	(

)*
.
. ÿ
..
þ
ý
= +
.
. 
..










   


*
.ÿ .þ
ý
.

.

= +
	

	
	
	




   

1* 

;99 5;		9 9*  #99 5#"	9 9* 	    
#%B995B9	"%9*
  %	 #      	 	  	  
		!			!! 	
 		"	%		 76			&
 !	   $ %   &  & !  	  ;  
5; *CD?5C	&D		?	*	!:&

 


	(

;	)
$  	   :  &  	  5	   @EA  @A* 	 
  
   	   	   %     	 $ 	 " 
   	

   
 

 
  	 
   
  

 
 
 ¸  å < - 
 
  


 
 ;	%<&
%  ,  		  	 &      	  !
!	!			

 
	
  < ¸   	     	        6 < -- --   	 = + ! 
!	&
)*


 

 

 




 





 
 
 


×
= +
= 

*


 




 




 
 


= +
= 
%;	% 	&)
		&=F#5'	*	%(
)* 0G)

* %&  &  >  
1* &&  %  >  
		)E		 	&	)'		
&		( 
?&	) ?&	

D 0      
 - . )1 14 )3)
 - . ) 
) ))4
 - . 4 13 '14
 - .  
2 1)4
 - . 1E 
E 43'
 - . 4' 
 
)2
 - . )E 2' 
2
	 -" .	 4 
1) ')2
	 - . E )4
 1
 - . E )3' 
13
D 0      
 - . )1 
E E'
 - . )) 4 4E1
 - . 
2 14 ))4
 - . )1) 1 3E
 - . 2 )3 32E
 - . )2 42 2'

 - . 1 
) )E

 - . )2 44 )'
0 -" . 1) 
3 )'4
 - . 

 )E )1)
 - . )1 )4 )23
 -" . ) )1 ))


 
1
D - .	 3 )) 33
 - . 3 ) )2'
 - . )) ) )
H - . 4 )E 1
)
0 - . 4 )4 1
)
; - . )) 
 )
'
; -8 .! 3 )3 
44
D -8 .	 3 )) 33

 - . ) )
 '4
 -" .! )1 )) )
 -+ . 

4 )) )

 -" . 
2E ) ))
 
!) !

			(
)* D!)!!
(
D-".	 	-. -. D-.	
;-. -. -. H-.
0-. -8.! 	-8.	 

* 0D!
!!)(
-". -. 0-". 0-.
-. -". -. +-".!
-+.   
1*  !D!!)!
(
-. -. -. +-.
-. -. -. 
			!(    0    !

  !"
!		 	(
)* +,-+,.+,/
$% &' () *+ ,-

* 0-0.0/
./ 01 23 45 67
1* +,-0.0/
$8 &9 : ;< =>
2* +%	!-+%	!.+%	!/
?@ AB CD E? FG
3* 0-<5*.5*/
HI JK LM NO PQ
4* 
)*3*				!&	%	%
 	!	(
)* 0	-&.0/
RS TU VW XY Z"

* 0-+,.0/
[\ ]^ _` ab cd
1* +,-+,	.0/

 
2
%e fg hi Fj ,k
2* 0-&.0
lm 4n 5o 6p qr
;	!	 				,%+,-+,.
+,+	)I1		(
)* 0-0.0/

* +,-0.0/
1* 0	-&.0/
2* 0-+,.0/
3* +,-+,	.0/
4* 0-&.0
			%	!	&	%	!	8
   %    + 	!     9 	 	 !
,%%	!	 		

#"






;	

J<	8%5::
	 *   8%  5    		*   !, 
!@2A@3A 

$ %%&
	
  < A )  @ ( #  ¨ ×  ÷  !  ::  	!  	 
 ¸ Í = +   (     	 
% +  ,


= ‡”
¸
 	

 



÷ 
 	,%	!(
 	  	,%
*  5     *  5    ÷ 
*  5 ) *  ¨ )I'
*  5 )   ¨ )I'
*  5    ¨ )I'
*  5    ¨ )I'
*  5    ¨ )I'
*  5 +   ¨ )I'
	 ¨ )I)))3E 
*  5     *  5    ÷ 
*  5    ¨ )I

	 ¨ )I1)E 
!1 !2


%

&
'

     
	
#	    
	

 
3
$% 			
    	 5 	 	  , -*  	  )2
		 8%		 		 			" 
	 		!! 8
B99

' "
     	  	    	  	
 	 7  	    	
	#&	 			,%
&;	9
   	   5		* 	  	 % @)A 	   	
@
A@1A@4A@)A%	:	 5	*	
	9	!				"

( )*+"	
$		  	!
!		0	<	+"9;
        	 	 + " 	 	  CD?  K
<F	+ 	KH; ;		
	L	M				"	&
<&

,
@)A <L     	
	 - 	 
+  	     7	 	  0  
<	
'1

E

@
A <M
	'M!)'3
@1A <N$    9	         =	
 #,<	J%		)
@2A D./.0$+  #,D9)'

@3A D//	/?B	<)E1
@4A L    M C 1 - 	 +  / 
$	>
)
@'A L MC1	 
	)E
@EA L K/+	-2  +
+	7

@A L K  L          ;	
7	$	"99+599+
)*
@)A LB1	/	
	'	
/-$	>
)E
@))A M$&L1! 	 
1 +<0&	5M

 
4
*	)
@)
A <+M((7	E11E131)43
@)1A <+M123	++4
)'3

297
Corpus approaches to antonymy
Steven Jones
University of Central Lancashire
1. Introduction
The word “antonymy” was coined in 1867 by CJ Smith1 to describe word-pairs - commonly
known as “opposites” - such as hot/cold, girl/boy and buy/sell. Some linguists (e.g. Lyons (1977) and
Cruse (1986)) apply the term “antonymy” restrictively and would only identify the first of these three
pairs as being truly antonymous. However, this seems counter-intuitive in many respects, and this
paper will use “antonymy” in its broader sense to refer to all word-pairs which could reasonably be
identified as “opposites” by speakers of English.
Antonymy is often defined simply as “oppositeness of meaning” (Palmer 1976: 94). However,
the problem with an exclusively semantic definition is that it fails to explain, or even acknowledge,
the tendency for certain words to become enshrined as “opposites” in language while others do not.
For instance, rich and poor would be regarded as antonyms because they occupy opposite ends of the
same scale, namely the scale of wealth. Affluent and broke also occupy opposite ends of this scale, but,
intuitively, one would be reluctant to describe them as antonyms. Therefore, antonymy should be
defined according to lexical as well as semantic criteria. It is a phenomenon “specific to words rather
than concepts” (Justeson & Katz, 1991: 138).
This paper will examine some of the ways in which advances in corpus technology enable
antonymy to investigated afresh. Firstly, the categories to which antonymous pairs have been logically
assigned will be summarised. Secondly, a set of new, corpus-based categories will be presented.
Thirdly, a statistical analysis of co-occurrence rates among antonyms will be offered. And finally,
ways of identifying new antonyms, again using corpus data, will be explored.
2. Traditional classes of antonymy
The meanings of antonymous pairs have been logically examined by a number of linguists (e.g.
Lyons 1977, Kempson 1977, Cruse 1986, etc.) and antonyms have been classified according to their
theoretical differences, perhaps at the expense of their intuitive similarity. Using terminology favoured
by Leech (1974), each of the traditional categories of antonymy will now be outlined.
2.1. Binary Taxonomy
The name given by Leech to antonymous pairs such as man/woman, alive/dead and
married/unmarried is “binary taxonomy” (1974: 109). Other writers (see Palmer 1972, Jackson 1988,
Carter 1987) prefer to speak of “complementarity”. Kempson - whose favoured term is “simple binary
opposition” - describes examples of Binary Taxonomy as “the true antonyms” (1977: 84). However,
this description is particularly confusing in light of the unwillingness of other linguists - namely
Cruse (1986) and Lyons (1977) - to acknowledge Binary Taxonomy as a form of antonymy at all. The
criterion necessary for an opposition to be considered binary is that the application of one antonym
must logically preclude the application of the other. For instance, if X is a smoker, X cannot be also a
non-smoker; if X is baptised, X cannot be also unbaptised, and so on.
2.2. Multiple Taxonomy
Multiple Taxonomy - also known as Multiple Incompatibility (Carter 1987:19) - is a borderline
classification of antonymy that refers to pairs such as summer/winter and north/south. In some
respects, this category is akin to Binary Taxonomy. The pair male and female, for example, belong to
a two-member system, such that X can never be simultaneously more than one member; solid, liquid
and gas, by comparison, belong to a three-member system, such that X can never be simultaneously
more than one member; similarly, clubs, diamonds, hearts and spades belong to a four-member
system, such that X can never be simultaneously more than one member. And so on. Thus, Multiple
Taxonomy may be seen as Binary Taxonomy extended to three or more terms. Whether such examples
remain within the boundaries of antonymy is debatable.
1 See Muehleisen (1997) for more details or “Introductory Matter” in Webster's Dictionary of Synonyms (1951:
vii-xxxiii).
298
2.3. Polar Opposition
Polar Opposition differs from Binary Taxonomy because one antonym is not automatically
debarred by the other's application. In other words, it is possible to be neither tall nor short in a way
that it is not possible to be neither male nor female. Thus, tall/short are said to be a Polar Opposition,
as are the majority of everyday opposites (old/new, cold/hot, wet/dry, etc.). Because Polar Oppositions
are not mutually exclusive, they are readily modified (quite happy, extremely happy, fairly happy,
etc.) and can take both comparative (happier) and superlative (happiest) form. Indeed, many
commentators (e.g. Lyons 1977, Cruse 1986, Jackson 1988) prefer to label such pairs Gradable
Antonymy.
2.4. Relative Opposition
An example of Relative Opposition is tenant/landlord. The statement X is the landlord of Y
entails and is entailed by Y is the tenant of X. Therefore, landlord and tenant belong to a reciprocal
relationship, also reflected by pairs such as teach/learn, buy/sell and above/below. The majority of
semanticists label this phenomenon “converseness” and Kempson notes that if the variables X and Y
are converse verbs, the statement A X B implies B Y A and the statement A Y B implies B X A. In other
words, B precedes A implies that A follows B, and A follows B implies that B precedes A (1977: 85).
A fertile area for Relative Antonymy is the field of kinship relations. If X is the grandparent of Y,
then Y must be the grandchild of X; if X is the husband of Y, then Y must be the wife of X.
2.5. Other Categories
Leech identifies two further categories of antonymy: “hierarchy” (1974: 106) is similar to
Lyons's notion of “rank” and describes the relationship between sets of terms such as
January/February/March and one/two/three; “inverse opposition” describes pairs such as all/some
and remain/become2 which may not otherwise be regarded as “opposites”. Other types of antonymy
include “orthogonal” and “antipodal” opposition (Lyons, 1977: 286). Orthogonal - meaning
perpendicular, at right angles - describes the antonymy holding between the words man, woman, girl
and boy. Each of these four words contrasts with two of the other three. So man can be the antonym of
boy and woman, but not girl; and boy can be the antonym of girl and man but not woman. An example
of an antipodal opposition would involve the terms north, east, south and west. Here, words only
contrast in one direction. So north is an antonym of south, but not east or west; and west is an
antonym of east but not north or south.
3. New classes of antonymy
The categories outlined above are useful if we wish to look at antonymy from a logical
perspective. However, it has now become possible for antonymy to be approached from a corpus-based
angle, with new classes being created to describe not what antonyms are, but what antonyms actually
do in text.
3.1. Data and Methodology
In order to ascertain and quantify the various textual functions of antonymy, a database of 3,000
sentences was constructed. Each of those sentences were retrieved from a large corpus and features
both members of a recognisable antonymous pair. The corpus which I chose to use consists of about
280 million words from The Independent. All stories printed in the newspaper between 1 October
1988 and 31 December 1996 are included in the corpus. Journalistic corpora are suitable for studies of
this nature because they are large, genre specific and reflect a natural, modern, non-fictional use of
written language. Thus, an overview of how antonymy is used in the field of broadsheet newspaper
journalism is possible, although it should be acknowledged that antonymy might be found to function
differently in other corpora.
Selecting a representative sample of antonymous pairs is more problematic. It is difficult to
imagine a list of antonyms which would not raise a single eyebrow, either because of words included
but not considered to be “good opposites”, or because of “good opposites” which might be conspicuous
in their absence from the list.
2 Leech's reasoning is that some artists have no formal training is synonymous with not all artists have formal
training and she did not become a smoker is synonymous with she remained a non-smoker.
299
Other corpus-based investigations into antonymy, such as Justeson & Katz (1991) and Mettinger
(1994), resolved this problem by using an existing index of antonyms. Justeson & Katz chose to make
use of the 40 “historically important” (1991: 142) antonyms identified by Deese (1965). This list of
antonyms was based entirely on the results of word association tests. Deese took 278 adjectives3 and
used them to elicit responses from 100 informants. When a pair of contrast words successfully elicited
one another more than any other word, they were added to the list of antonymous pairs, which
ultimately numbered 40. Though all antonyms cited by Deese fulfilled this requirement, some passed
the test with alarmingly low scores. For example, given the stimulus together, only 6% of informants
replied alone; given alone, only 10% replied together. However, this was evidently enough to make
these answers more popular than any other, even though the fact remains that a minimum of 84% of
informants failed to give either alone as a response to together, or together as a response to alone.
Indeed, of the 278 adjectives tested, only one word succeeded in eliciting its antonym on a majority of
occasions (left, to which 51% of informants gave right). Therefore, though there remains a strong
tendency for informants to provide antonyms as responses to given stimuli in word associations tests,
it may not be wise to treat Deese's 40 antonyms as being in any sense exhaustive or definitive.
A different approach was taken by Mettinger (1997), who used Roget's Thesaurus as his source
for antonyms. Created in the middle of the nineteenth century, Roget's Thesaurus attempted to
catalogue language, not in alphabetical order, but according to “ideas”. This is of relevance to a study
of antonymy because Roget chose, where possible, to present these ideas in opposition to one another.
Thus, the thesaurus begins by listing words associated with existence, then considers words associated
with inexistence. Following next are substantiality and insubstantiality, then intrinsicality and
extrinsicality.
Neither using the Deese Antonyms nor turning to thesaural listings is ideal. Essentially, one is
still dependent on the intuitions of others to identify antonymous pairs. In the case of Roget, these
intuitions are 150 years out of date and “contain a number of lexical items that are hardly used in
contemporary English” (Mettinger, 1994: 94); in the case of the Deese antonyms, one is dependent on
the criteria for antonymy established by 1960s’ schools of psychology. However, it is impossible to
rely on anything other than intuition when it comes to a psycholinguistic phenomenon such as
antonymy. No exhaustive list of antonyms will ever be produced because the process which gives a
pair of words antonymous status is complex and dynamic. Indeed, this status could only really be
gauged by consensus, as definitions of antonymy vary not only from one linguist to the next, but also
from one mental lexicon to the next. With these limitations in mind, I decided that the best approach
would be to create a new list of antonyms, customised to meet the demands of this research and
relevant to a 21st Century investigation of antonymy:
active/passive advantage/disadvantage agree/disagree alive/dead
attack/defend bad/good badly/well begin/end
boom/recession cold/hot confirm/deny correct/incorrect
difficult/easy directly/indirectly discourage/encourage dishonest/honest
disprove/prove drunk/sober dry/wet explicitly/implicitly
fact/fiction fail/succeed failure/success false/true
fast/slow female/male feminine/masculine gay/straight
guilt/innocence happy/sad hard/soft hate/love
heavy/light high/low illegal/legal large/small
long/short lose/win major/minor married/unmarried
new/old officially/unofficially old/young optimism/pessimism
optimistic/pessimistic peace/war permanent/temporary poor/rich
private/public privately/publicly punishment/reward quickly/slowly
right/wrong rightly/wrongly rural/urban strength/weakness
Table One: Antonymous pairs selected for inclusion in the database
Any native speakers could be reasonably expected to identify the antonym of each of the above
112 words. Most core antonyms are represented and the list also features a number of lower frequency
pairs, including antonymous nouns, adverbs and verbs, which previous investigations have been
inclined to overlook.
3 Deese's list included words such as above, inside and bottom which function as adjectives less often than they
function as other parts of speech.
300
In total, 2,844 sentences were retrieved at random from the corpus, all of which features both
members of one of the above antonymous pairs. A further 156 sentences were retrieved in which a
word co-occurred with an un- version of itself. As un- is the most prolific morphological marker of
antonymy in English, this was a useful way to enrich the database with antonymous pairs which,
though often less familiar than the 56 pairs selected for analysis, still reflect opposition and are
instantly recognisable as antonyms because of their morphology.
3.2. Classifying the database
All 3,000 database sentences have been classified according to their textual function. Eight
categories are presented below, in alphabetic order only, together with three illustrative examples.
3.2.1. Ancillary Antonymy
Sentences attributed to this category contain two contrasts: that between the established pair of
antonyms (in bold) and that between a pair of words or phrases which would not usually be
interpreted contrastively (in italics). Here, it would appear that the antonyms function as lexical
signals. They serve an “ancillary” role, helping us to process another, perhaps more important,
opposition nearby.
·  ind9024: At Worcester on Wednesday, Botham - apart from bowling well - was wandering around
in a T-shirt with the message: ‘ Form is temporary, class is permanent‘.
·  ind913: Broadly speaking, the community charge was popular with Conservative voters and
unpopular with Labour voters.
·  ind891: Robin Cook, Labour's health spokesman, demanded: ‘How can it be right to limit the
hours worked by lorry drivers and airline pilots, but wrong to limit the hours of junior hospital
doctors undertaking complex medical treatment?’
3.2.2. Comparative Antonymy
This category is home to sentences in which a pair of antonyms are set up in comparison with
one another. This function of antonymy is often expressed by a lexico-syntactic framework such as
more X than Y or X is more [adjective] than Y.
·  ind891: And it is possible to accept both that Dr Higgs was a lot more right than wrong in her
diagnoses, but that it is now impossible for her to return.
·  ind923: ‘Well,’ said Cage, completely unabashed, ‘some living composers are more dead than
alive ‘.
·  ind903: Training would be based upon rewarding good behaviour, because behaviourists,
Skinner argued, had found that reward is more effective than punishment.
3.2.3. Co-ordinated Antonymy
The antonymous pair in each of the examples below is presented in a unified, co-ordinated
context. The function of such antonyms is to identify a scale, then exhaust that scale. The contrastive
power of each pair remains untapped because their purpose is to express inclusivity. Mostly, antonyms
which serve this role are conjoined by and or or.
·  ind953: He showed no disloyalty, publicly or privately, to Virginia Bottomley though it must have
irked him that she was in the Cabinet and he was not.
·  ind921: Whitehall was yesterday unable to confirm or deny other simulated devolutions.
·  ind941: Again in debates over genetic research it is significant that Christians, Muslims and
Jews have united, implicitly and explicitly, in condemning a low view of the value of embryonic
life.
4 The notation gives detail about where and when each sentence was published (newspaper; year; quarter). For
example, this sentence was published in the second quarter of 1990 by The Independent.
301
3.2.4. Distinguished Antonymy
The sentences below refer, in a metalinguistic fashion, to the semantic dissimilarity between
antonyms. The framework which houses the antonyms most frequently is n between X and Y, where n
is difference or a synonym thereof.
·  ind892: But far from that, Mortimer's father had not given him even a basic moral education, such
that today he still doesn't know the difference between right and wrong, or so he said.
·  ind931: But it made the point that the division between gay and straight is one of many rifts in
our society.
·  ind884: Mr Craxi's fresh-faced deputy, Claudio Martelli, also dissented, saying that ‘one must
distinguish between hard and soft drugs'.
3.2.5. Extreme Antonymy
Sentences classified in terms of Extreme Antonymy are similar to Co-ordinated Antonymy
examples. The difference is that here a contrast is set up, not between antonyms, but between both
ends of a semantic scale, on one hand, and the semantic space in between, on the other. Typical
frameworks show antonyms linked by or or and, and premodified by an extremity-signalling adverb
such as very or too.
·  ind892: No-one can afford to go to law except the very rich and the very poor and it can't
possibly get any worse.
·  ind903: The advantages are that the track does not need watering, and can be used when
conditions are either too dry or too wet for racing on turf.
·  ind964: Freud maintained in Civilization and Its Discontents that human beings feel a deep hate
and a deep love for civilization.
3.2.6. Idiomatic Antonymy
Many antonymous pairs co-occur as part of a familiar expression, proverb or cliché. Such
examples have been assigned to category of Idiomatic Antonymy.
·  ind944: The long and the short of it is that height counts.
·  ind893: They evidently knew they could teach this old dog a few new tricks.
·  ind892: Whoever said the female of the species was more deadly than the male hadn't met Lord
William Whitelaw.
3.2.7. Negated Antonymy
Arguably the purest function of antonymy, the sentences below each negate one antonym in
order to place additional emphasis on the other or to identify a rejected alternative. The most common
framework for this class is X not Y.
·  ind884: Well, without the combination of an arms race and a network of treaties designed for
war, not peace, it would not have started.
·  ind912: Democracy means more than the right to pursue one's own self-interest - government
must play an active, not passive, role in addressing the problems of the day.
·  ind893: However, the citizen pays for services to work well, not badly.
3.2.8. Transitional Antonymy
The function of antonyms belonging to this category is to help describe a movement from one
state to another. This transition is usually expressed by a framework such as from X to Y or hinges
around the verb to turn.
·  ind923: Her film career similarly has lurched from success to failure, with enormous periods out
of work.
302
·  ind934: The atmosphere of the negotiations was tense, discussion uneven, the mood in both
camps swung from optimism to pessimism.
·  ind923: Inflation is a tax which redistributes wealth to the sophisticated from the
unsophisticated.
3.3. Frequency of New Classes
The eight classes outlined above collectively account for 2,894 of the 3,000 database sentences.
The remaining 3.5% of contexts demonstrate that antonyms can also function in unusual, sometimes
innovative, ways about which it difficult to generalise. The table below shows the statistical
distribution of all database sentences.
ancill co-or comp distin trans negat extre idiom other total
active/passive 53 14 9 6 6 6 - - 2 96
advantage/disadvantage 15 14 2 - 4 1 - - - 36
agree/disagree 26 17 3 - - - - - 3 49
alive/dead 16 26 9 1 - 1 - - 1 54
attack/defend 10 15 3 - - 2 - - - 30
bad/good 55 47 4 4 3 1 - 2 1 117
badly/well 31 15 4 1 - 2 - - - 53
begin/end 24 23 3 - - 1 - - - 51
boom/recession 12 3 4 - 5 - - - - 24
cold/hot 21 23 - - 2 - 2 11 - 59
confirm/deny - 34 - - - - - - - 34
correct/incorrect 6 11 - 1 - - - - - 18
difficult/easy 19 5 - - - - 1 - 2 27
directly/indirectly 21 57 1 - - - - - - 79
discourage/encourage 16 8 2 - - 2 - - - 28
dishonest/honest 8 4 - - - - - - - 12
disprove/prove - 14 - - - - - - - 14
drunk/sober 8 7 - - 1 1 - - 1 18
dry/wet 11 9 3 1 3 - 4 - - 31
explicitly/implicitly 6 19 2 - - 3 - - - 30
fact/fiction 5 5 2 11 2 4 1 - 6 36
fail/succeed 30 27 5 - - 1 - - - 63
failure/success 38 20 10 12 1 6 1 - - 88
false/true 10 34 3 11 - 1 1 - 2 62
fast/slow 17 7 2 1 - - - - 1 28
female/male 23 43 1 4 1 - - 2 13 87
feminine/masculine 37 18 2 3 - - 1 - 7 68
gay/straight 3 20 7 1 1 1 - - - 33
guilt/innocence 5 27 3 5 1 2 - - 1 44
happy/sad 22 17 2 - 2 - 2 - - 45
hard/soft 17 3 2 1 3 2 3 - 1 32
hate/love 40 44 7 2 1 2 2 - 6 104
heavy/light 46 19 5 1 4 - 2 - - 77
high/low 20 3 2 3 1 1 1 1 - 32
illegal/legal 10 17 1 - 3 - - - - 31
large/small 17 23 4 2 2 - 2 - - 50
long/short 22 7 4 1 - 1 - 1 - 36
lose/win 27 25 5 - - 1 - - - 58
major/minor 11 9 - 3 4 - - - - 27
married/unmarried 4 14 8 5 - - - - - 31
new/old 81 76 21 19 10 3 1 6 37 254
officially/unofficially 14 10 1 - - - - - - 25
old/young 20 34 6 5 - 1 3 - - 69
optimistic/pessimistic 30 12 3 - 1 - 1 - - 47
optimism/pessimism 9 1 2 - 6 1 1 - 1 21
peace/war 3 5 1 1 1 2 - - 2 15
permanent/temporary 6 12 5 1 3 1 - - - 28
poor/rich 46 16 6 24 1 - 5 - 4 102
private/public 36 68 6 13 5 2 - - 4 134
privately/publicly 20 24 2 1 - - - - - 47
punishment/reward 6 5 4 - - 3 - - 1 19
quickly/slowly 16 6 2 - - - 4 - - 28
right/wrong 36 13 1 5 1 - - - 4 60
rightly/wrongly 1 43 - - - - - - - 44
rural/urban 7 13 - 2 1 - 1 - - 24
strength/weakness 11 6 6 - 4 4 - - 4 35
un-words 58 60 15 10 7 3 1 - 2 156
TOTAL: 1162 1151 205 161 90 62 40 23 106 3000
TOTAL (%): 38.7 38.4 6.8 5.4 3.0 2.1 1.3 0.8 3.5 100
Table Two: Statistical breakdown of database classes
303
Table One demonstrates that the most popular category is that of Ancillary Antonymy, to which
38.7% of all sentences have been attributed. Recording only 11 fewer sentences is Co-ordinated
Antonymy, which accounts for 38.4% of all database sentences. These two classes are significantly
larger than any others and collectively account for 77.1% of sentences. The third largest category
identified is Comparative Antonymy, but this is only a fraction the size of the two major categories.
205 sentences have been attributed to the class of Comparative Antonymy, less than 7% of the
database. Distinguished Antonymy accounts for a further 5.4% of sentences, but no other class of
antonymy is represented by more than 3% of the total sample.
Perhaps most remarkable is that the majority of pairs, regardless of their word class, follow a
similar pattern of distribution. For example, Ancillary Antonymy and Co-ordinated Antonymy are the
most commonly occurring categories, but this is not just because they are each strongly favoured by a
small number of pairs. Rather, this pattern is consistent among almost all pairs. Indeed, in the case of
44 of the 56 pairs sampled, Ancillary Antonymy and Co-ordinated Antonymy each account for more
sentences than any other category. This suggests that any given word-pair is likely to have a
predictable textual profile. However, some pairs may have unusual individual distributions. For
example, all 34 sentences retrieved which feature confirm/deny are assigned to the class of Coordinated
Antonymy. This is because refusing to confirm or deny a proposition has become a cliche
among politicians and other public figures.
4. Antonym Co-occurrence
One of the questions raised by my research is this: exactly how widespread is antonymy in
language? Gauging the true answer is very difficult. Firstly, there is the problem of defining
antonymy: the stricter the definition one uses, the less pervasive the phenomenon will appear. Then
there is the even greater problem of counting: in order to arrive at an estimate of the proportion of
sentences which feature antonyms, one would need to identify every single antonymous pair in use,
then retrieve all sentences which features both of those words. But that would not be all - one would
then need to edit all of these sentences manually (which would number over a million in my corpus) to
eliminate those in which the word-pair do not function antonymously (coincidental co-occurrence is
common among higher frequency pairs, especially those which feature a polysemous term, such as
well). Only then could one arrive at an approximation of the proportion of corpus sentences which
feature antonyms, and this approximation would still fail to account for inter-sentential antonymous
usage.
An easier way to estimate the prevalence of antonymy in text is to compare the expected cooccurrence
rate of antonyms with their observed co-occurrence rate. Therefore, I shall now examine
each of the 56 antonymous pairs in my sample to determine whether those antonyms co-occur more or
less than would be expected by chance.
Listed below are the 56 antonymous pairs selected for study in this thesis. Each pair is followed
by five columns of figures: columns one and two simply record the raw frequency of each antonym in
the corpus; column three records the number of sentences one would expect to feature both antonyms
if those words co-occurred at random; column four records the number of sentences in the corpus
which, in reality, contain both antonyms; column five records the Observed/Expected ratio, which is
generated by dividing the figure in column four by the figure in column three.
word one (W1) /
word two (W2)
raw frequency
of W1
raw frequency
of W2
Expected
Co-occurrence
Observed
Co-occurrence
Observed /
Expected
active/passive 11411 2033 1.8 172 95.6
advantage/disadvantage 21531 2483 4.2 69 16.4
agree/disagree 18196 2472 3.5 153 43.7
attack/defend 43395 9198 31.0 273 8.8
cold/hot 16466 16026 20.5 751 36.6
correct/incorrect 10529 1484 1.2 34 28.3
dead/alive 32214 11661 29.2 565 19.3
deny/confirm 7514 6595 3.9 335 85.9
difficult/easy 54244 31395 132.4 434 3.3
directly/indirectly 14172 1377 1.5 492 328.0
drunk/sober 4730 1878 0.7 56 80.0
dry/wet 10978 5109 4.4 348 79.1
encourage/discourage 12586 1614 1.6 77 48.1
end/begin 145438 19682 224.6 740 3.3
explicitly/implicitly 1320 813 0.1 32 320.0
fact/fiction 78900 7391 45.3 503 11.1
fail/succeed 10963 8258 7.0 131 18.7
304
fast/slow 22625 17374 30.6 350 11.4
feminine/masculine 1191 903 0.1 140 1400.0
good/bad 181876 47247 668.1 4804 7.2
guilt/innocence 4229 3804 1.3 162 124.6
happy/sad 28217 9420 20.7 140 6.8
hard/soft 68635 11960 63.8 526 8.2
high/low 93232 41088 297.8 2847 9.6
honest/dishonest 6922 1084 0.6 28 46.7
legal/illegal 40832 11208 35.6 302 8.5
light/heavy 36832 22898 65.6 297 4.5
long/short 131582 52119 533.2 2168 4.1
love/hate 42541 6108 20.2 511 25.3
major/minor 45452 10624 37.5 432 11.5
male/female 16930 14883 19.6 2556 130.4
married/unmarried 25581 1033 2.1 101 48.1
new/old 341832 113065 3004.9 9426 3.1
officially/unofficially 6025 394 0.2 33 165.0
old/young 113065 83247 731.8 2704 3.7
optimistic/pessimistic 7123 1984 1.1 96 87.3
optimism/pessimism 5717 1163 0.5 91 182.0
prove/disprove 20968 258 0.4 35 87.5
permanent/temporary 10413 7878 6.4 351 54.8
poor/rich 34054 20999 55.6 2027 36.5
public/private 133056 61202 633.1 6741 10.6
publicly/privately 8108 6406 4.0 282 70.5
punishment/reward 6363 6152 3.0 38 12.7
quickly/slowly 25129 8958 17.5 83 4.7
recession/boom 22707 8678 15.3 334 21.8
right/wrong 125712 42376 414.2 2677 6.5
rightly/wrongly 4558 2681 1.0 182 182.0
rural/urban 8600 7923 5.3 515 97.2
small/large 86908 69219 467.7 2928 6.3
straight/gay 21672 9734 16.4 277 16.9
strength/weakness 19866 5971 9.2 441 47.9
success/failure 47816 24438 90.8 971 10.7
true/false 35357 10245 28.2 227 8.1
war/peace 81293 38258 241.8 2586 10.7
well/badly 178431 15772 218.8 712 3.3
win/lose 76372 27771 164.9 1125 6.8
TOTAL: 2662409 955994 8441.8 55411
AVERAGE: 6.6
Table Three: Co-occurrence of Antonymous Pairs
Table Three shows that all antonymous pairs selected for study co-occur at a statistically
significant rate, at least three times more often than would be expected by chance. Some pairs record
an enormous Observed/Expected ratio, but this is often attributable to their low individual frequencies.
For example, feminine and masculine only arise on about 1,000 occasions each in the entire corpus.
Therefore, they can be expected to co-occur in only 0.1 sentence. In fact, they co-occur in 140
sentences, generating an Observed/Expected ratio of 1400. This is anomalous, but even pairs of words
with relatively high individual frequencies (female/male, cold/hot, poor/rich, etc.) are able to record
healthy co-occurrence figures. Indeed, according to this experiment, antonyms co-occur 6.6 times
more often than would be expected by chance.
When Justeson & Katz conducted a similar test on the Deese antonyms, they found that
antonyms co-occurred in the same sentence 8.6 times more often than chance would allow (1991:
142). Their results were based on a corpus much smaller than the one from which the above statistics
are drawn and this inevitably affects their reliability. For example, Justeson & Katz calculate an
Observed/Expected ratio of 19.2 for happy/sad, based on individual frequencies of 89 and 32
respectively and observed co-occurrence in just one sentence. My corpus yields an Observed/Expected
ratio of 6.8 for happy/sad, based on individual frequencies of 28,217 and 9,420 respectively and
observed co-occurrence in 140 sentences. It seems fair to conclude that statistics derived from the
latter corpus will be more trustworthy. However, despite difference in corpus size, the two average
Observed/Expected rates (6.6 and 8.6), are close enough to prove that antonyms do co-occur in text at
a relatively high rate.
5. New Antonyms
A further question which may be considered with the help of corpus data is: how and why do
certain pairs of words become enshrined as antonyms? To address this issue, the productivity of
305
frameworks associated with some of the new classes of antonymy will be investigated. Productivity
here refers to the “statistical readiness” (Renouf & Baayen, 1996) of lexico-syntactic constructions to
incorporate other related terms. In other words, if antonyms occupy certain lexical environments in
text, which other words also occupy those environment and could some of those words be seen as new,
developing antonyms?
Most new classes of antonymy were found to favour certain lexical environments in text. Three
lexico-syntactic frameworks will be used to assess the antonymous profiles of given words:
both X and Y [Co-ordinated Antonymy]
between X and Y [Distinguished Antonymy]
whether X or Y [Co-ordinated Antonymy]
The productivity of these frameworks will be tested by placing a word in the X-position,
extracting all concordances which feature that word-string from the corpus, then examining which
items occupy the Y-position. Three words will be placed in X-position for each framework. I shall
begin by investigating an antonym from my sample, good. If these frameworks are in any sense
productive, intuition demands that they should retrieve bad and, to a lesser extent, evil in Y-position
with high frequency. I shall then examine two new words (natural and style) to discover whether this
strategy can be used to identify potential antonyms for words which do not have established antonyms.
5.1. Seed Word: good
·  both good and ...
The lexical word-string both good and appears in a total of 63 corpus sentences. In 45 of those
63 sentences, it is followed immediately by bad. A further 4 of the 63 sentences recorded evil
appearing immediately after both good and. This leaves 14 occurrences of both good and which are
followed by neither bad nor evil. These are listed below, together with the noun-head they modify:
q 
both good and flawed (King Hassan's reign)
q 
both good and pathetic (years)
q 
both good and wicked (people)
q 
both good and nasty (youths)
q 
both good and hard (times)
q 
both good and not green (God)
q 
both good and true (a story)
q 
both good and lasting (friends)
q 
both good and powerful (patriotism)
q 
both good and new (a paper)
q 
both good and friendly (a service)
q 
both good and inimical to the Labour Party (Conservative belief)
q 
both good and non-sexually explicit (a novel)
q 
both good and great (wines)
The concordances above make interesting reading. Some Y-position words are very useful
contrast terms for good (flawed and pathetic are synonymous with bad; wicked and nasty are
synonymous with evil). One would not intuitively identify hard as an antonym of good, but this
contrast is perfectly valid within its given context - hard times are quite the opposite of good times.
However, the phrase both X and Y does not always reflect an obvious contrast. For example, a story is
described as being both good and true; one would not want to consider these terms as potential
antonyms. In such contexts, it would appear that the framework signals unlikely inclusiveness.
Similarly, no contrast is generated between good and either powerful, new, or friendly. Finally, both
good and great is an interesting example because a distinction is made, but that distinction is not at
the usual point on the scale on quality (i.e. between good and bad). Rather, this context distinguishes
between good and something better than good.
·  between good and ...
The framework between good and Y occurs in 140 corpus sentences, more than double the
number of both good and Y. However, the distribution of bad and evil is very different. Of the 140
examples of between good and Y, 50 feature bad in Y-position and 78 feature evil in Y-position. Only
12 sentences feature neither bad nor evil in Y-position. These contexts are listed below, together with
their corresponding noun-head, where appropriate:
q 
between good and poor (schools)
q 
between good and poor (performance)
q 
between good and lousy (comprehensives)
q 
between good and harmful (foods)
q 
between good and greed (a struggle within Lewis)
306
q 
between good and suspicious (toadstools)
q 
between good and good to soft (the going)
q 
between good and very good
q 
between good and very good
q 
between good and excellent (Melbourne's eateries)
q 
between good and really great (wine)
q 
between good and the best
Once again, some of the occurrences at the lower end of the frequency scale are valid contrast
terms and others are not. Two contexts show poor occupying Y-position in the between good and Y.
This is an excellent contrast term, as is lousy. Equally interesting are the distinctions made between
good and other, more extreme points on the scale of quality. On two occasions, very good is
contrasted with good, and excellent, really great and the best each appear in opposition on one
occasion. This is reminiscent of the both good and great word-string retrieved earlier. Although one
would expect good to contrast exclusively with negative items in language, it would seem that many
writers choose to exploit its latent contrast with “super-positive” terms instead.
·  whether good or ...
Of the three lexico-syntactic frameworks analysed, whether X or Y is the least common. In the
corpus, only 8 sentences feature the word-string whether good or Y. In 7 of those sentences, bad fills
the Y-position; in the eighth, evil fills the Y-position.
5.2. Summary of good
This analysis of good has demonstrated that it is possible to retrieve contrast words from the
corpus using productive lexico-syntactic frameworks. Collectively, the three frameworks examined
occur on 211 occasions in the corpus. On 102 of those occasions (48.3%), the given word-string is
followed by bad. This is compatible with our intuitions - one could predict that bad would be set up in
opposition against good most commonly. Indeed, one could also predict that evil would be runner-up;
evil fills the Y-position in frameworks analysed on 83 occasions (39.3%). However, the purpose of
this experiment is not to prove that bad and evil are antonymous with good; rather, it is to show that
the three frameworks identified are fertile enough to be deemed productive. This seems indisputable.
5.3. Seed Word: natural
·  both natural and ...
q 
both natural and accurate (their response to the camera)
q 
both natural and artificial (light)
q 
both natural and artificial (lighting)
q 
both natural and artificial (light)
q 
both natural and artificial (the essence of man)
q 
both natural and artificial (everything that exists)
q 
both natural and assisted (fertility)
q 
both natural and beneficial (high altitude)
q 
both natural and coloured (light)
q 
both natural and heraldic (devices)
q 
both natural and human (perturbations)
q 
both natural and inevitable (process)
q 
both natural and inevitable (that ...)
q 
both natural and lucid (her acting)
q 
both natural and man-made (components)
q 
both natural and man-made (beauty)
q 
both natural and man-made (beauty)
q 
both natural and man-made (disasters)
q 
both natural and man-made (facilities)
q 
both natural and man-made (polymers)
q 
both natural and market (forces)
q 
both natural and prudent (paying debts)
q 
both natural and safe (white sugar)
q 
both natural and sensible (idea)
q 
both natural and social (sciences)
q 
both natural and social (sciences)
q 
both natural and spiritual (creatures)
q 
both natural and superb (a history of vodka)
q 
both natural and synthetic (fibres)
q 
both natural and taboo (a child's sexuality)
q 
both natural and technical (the effect)
q 
both natural and violent (causes)
q 
both natural and vital (USA action)
The output generated by a search for both natural and comprises 33 concordances, all of which
are listed above. It can be seen that both natural and occurs less frequently in text than both good and
, but that the Y-position output is more diverse. However, this is not to say that no patterns emerge: of
307
the 33 occurrences of this lexico-syntactic framework, 6 are followed by man-made and 5 are followed
by artificial. Both of these terms make excellent contrast words for natural. Some of the words
retrieved in Y-position on one occasion only are non-contrastive, but interesting and valid oppositions
of natural include market (in terms of forces), synthetic (in terms of fibres), violent (in terms of death)
and assisted (in terms of fertility).
·  between natural and ...
q 
between natural and artificial (ozone)
q 
between natural and artificial (worlds)
q 
between natural and artificial (worlds)
q 
between natural and created (forms)
q 
between natural and cultivated (areas)
q 
between natural and juridical (persons)
q 
between natural and man-made (assets)
q 
between natural and metal (packaging)
q 
between natural and moral (evil)
q 
between natural and supernatural
Ten corpus sentences feature the phrase between natural and Y. The only word to occupy Yposition
more than once is artificial. It is interesting to note that between natural and man-made also
appears. This suggests that man-made also shares a strong contrastive profile with natural.
Significantly, it also confirms that similar words occupy the Y-position in the frameworks both
natural and Y and between natural and Y. All of the six words retrieved on one occasion reflect
contrast to a lesser degree, with supernatural perhaps being the most interesting because it ties in with
spiritual, which was picked up by both natural and Y
·  whether natural or ...
q 
whether natural or artificial (hormones)
q 
whether natural or electric (light)
q 
whether natural or imposed (punishment)
q 
whether natural or man-made (environment)
q 
whether natural or man-made (beauty)
q 
whether natural or otherwise (phenomena)
q 
whether natural or step (parents)
q 
whether natural or through external intervention (chemical changes)
Eight contexts were found to include the word-string whether X or Y. Pleasingly, both artificial
and man-made arise in Y-position. This means that all three lexico-syntactic frameworks have
successfully retrieved both of these words. One-off contrast terms again include valid and useful
examples. For example, step is not the kind of word one would intuitively identify as a potential
opposite of natural. However, within the given context of parentage, this contrast is not only
legitimate but very interesting. It is also reassuring to note the appearance of otherwise in Y-position.
Though this term is not a valid opposite of natural in itself, otherwise effectively functions as a
proform for unspecified contrast words in text.
5.4. Summary of natural
Collectively, the three frameworks examined as part of this study feature natural in X-position in
a total of 51 sentences. In 9 of those sentences, artificial occupies Y-position and, in a further 9 of
those sentences, man-made occupies Y-position. This strongly suggests that those two words are the
primary textual contrast terms of natural. Indeed, over one third of all frameworks examined feature
either artificial or man-made in opposition with natural.
This output is particularly interesting if analysed in light of the range of antonyms which
lexicographers have paired intuitively with natural. For example, Webster's Dictionary of Synonyms
(1951) lists three antonyms: artificial, adventitious and unnatural. The inclusion of the firstmentioned
of this trio is supported by this experiment, but adventitious does not occupy Y-position at
all. This is not surprising given that the word occurs only seven times in the entire corpus (or about
once per 40 million words in text).
More notable is the non-appearance of unnatural in textual opposition with natural. This could
be interpreted as a flaw in the retrieval strategy or it could be interpreted as revealing an interesting
aspect of natural: namely, that it prefers to contrast with lexical opposites rather than its
morphological opposite.
308
Collins Cobuild Dictionary5 (1985) cites six antonyms of natural, beginning with unnatural. The
other contrast words suggested are surprising (not retrieved in text), contrived (not retrieved),
artificial (retrieved 9 times), man-made (retrieved 9 times), and processed (not retrieved). Chambers
Dictionary of Synonyms and Antonyms (1989) suggests unnatural, artificial, man-made, affected and
contrived.
Therefore, some correlation emerges between intuitively identified antonyms and antonyms
identified by productive lexico-syntactic frameworks: all three dictionaries cite artificial as a good
opposite and only the oldest of the three fails to cite man-made. However, I would suggest that other
recommended antonyms (adventitious, processed and even unnatural) are not placed in textual
opposition against natural as often as may have been anticipated. Moreover, it could be argued that
such words are less valid contrast terms of natural than synthetic, supernatural, assisted and other Yposition
words which have not been cited by lexicographers, but which have been retrieved in this
experiment.
5.5. Seed Word: style
·  both style and ...
q 
both style and a demonstration of reaching speed
q 
both style and achievement
q 
both style and commercial space
q 
both style and content (x4)
q 
both style and date
q 
both style and emotion
q 
both style and fashion
q 
both style and feeling
q 
both style and heart
q 
both style and history
q 
both style and performance
q 
both style and personality
q 
both style and personnel
q 
both style and policy
q 
both style and prices
q 
both style and qualifications
q 
both style and reputation
q 
both style and standards
q 
both style and substance (x5)
The word-string both style and arises in 26 corpus sentences. Two words are set up in opposition
to style more commonly than anything else - substance appears in Y-position five times and content
appears in Y-position four times. These contexts reflect a trend for style to be seen as meaningless or
superficial, and licenses its opposition with more “weighty” terms. Other words which reflect this
trend but only occur once include performance, policy, achievement and standards. From examining
its antonymous profile, one might infer that style has developed a pejorative sense in the language.
·  between style and ...
q 
between style and content (x4)
q 
between style and disorder
q 
between style and grape
q 
between style and political ideology
q 
between style and quality (x2)
q 
between style and subject
q 
between style and substance (x5)
Four of the 15 between style and constructions are followed by content and five are followed by
substance. This means that only 6 of the 15 occurrences of this lexico-syntactic framework feature
other terms. On two occasions, this term is quality, which conforms to the underlying trend for style
to be treated negatively in text and be synonymous with emptiness or absence of quality. However, a
reminder that these frameworks are not exclusively inhabited by contrast words is provided by grape.
The sentence from which this word-string is taken actually explores the relationship between the style
of a given wine and the nature of the grape used in its production. Of course, grape could hardly be
seen as a valid or useful opposite of style, merely an unlikely instantial collocation.
5 This dictionary made use of a corpus which was smaller than my own (about 20 million words), but which was
not newspaper specific.
309
·  whether style or ...
This lexico-syntactic framework was not found in the corpus at all, probably because style
functions most commonly as a noun and nouns do not lend themselves readily to this construction.
5.6. Summary of style
The textual profile of style shows that substance is most commonly retrieved in Y-position (10
hits; 24.4%). In second place is content (8 hits; 19.5%). Between them, these two words are retrieved
in Y-position in 43.9% of frameworks. It is interesting to note that style is never seen to contrast with
concepts such as inelegance or tastelessness, as Chamber's Dictionary of Synonyms and Antonyms
(1989) suggests.
5.7. Analysis of output
The aim of this exercise was to identify where new antonyms come from. The process by which
“opposites” are created is complex, but it is reasonable to speculate that in order for a pair of words to
become enshrined as antonyms in any language, they must first receive a significant amount of
exposure. This exposure will be in contexts which are more often associated with established pairs of
antonyms. Based on this rationale, man-made and artificial have been identified as potential
antonyms of natural; and substance and content as potential antonyms of style. This output may be
seen as initial evidence that it may be possible to automatically identify embryonic antonyms in text.
Arguably the most dramatic antonymous formation of recent times involved gay and straight, which
held no obvious semantic relation in the middle of the last century, but have now achieved clear
antonymous status. The increasing attention given to different sexual preferences must have
contributed to the establishment of gay and straight as new “opposites”, though this antonymity was
surely enshrined by repeated co-occurrence in lexical environments similar to those examined here.
6. Conclusions
This paper has demonstrated that corpus-based approaches are relevant to an investigation of
antonymy. Based on evidence from broadsheet newspaper corpora, I have argued that:
·  In addition to logical distinctions, antonymous pairs are also receptive to classification according
to their textual function. Data show that the two most common text-based classes of antonymy are
Co-ordinated Antonymy (in which antonyms are joined by and or or and express exhaustiveness or
inclusiveness) and Ancillary Antonymy (in which antonyms act as a lexical signal of a further,
nearby contrast).
·  All 56 antonymous pairs examined co-occur intra-sententially at least three times more often than
chance would allow. On average, antonyms record an Observed/Expected ratio of 6.6.
·  Using lexico-syntactic frameworks associated with the co-occurrence of established antonymous
pairs, it is possible to identify new textual oppositions. Such research may shed light on the
process by which a pair of words achieve antonymous status in language and allow us to identify
new antonyms in their infancy.
Bibliography
Carter R 1987 Vocabulary. London, Allen & Unwin.
Chambers Dictionary of Synonyms and Antonyms 1989. Cambridge, Chambers.
Collins Cobuild English Dictionary 1995. London, HarperCollins.
Cruse DA 1986 Lexical Semantics. Cambridge, Cambridge University Press.
Deese J 1964 The Associative Structure of Some Common English Adjectives. Journal of Verbal
Learning and Verbal Behaviour, 3: pp 347-357.
Jackson H 1988 Words and their Meaning. Cambridge, Cambridge University Press.
Justeson JS, Katz SM 1991 Redefining Antonymy: the textual structure of a semantic relation.
Literary and Linguistic Computing 7: pp 176-184.
Kempson RU 1977 Semantic Theory. Cambridge, Cambridge University Press.
Leech G 1974 Semantics. Middlesex, Penguin.
Lyons J 1977 Semantics: Volume 2. Cambridge, Cambridge University Press.
Mettinger A 1994 Aspects of Semantic Opposition in English. Oxford, Oxford University Press.
Muehleisen V 1997 Antonymy and Semantic Range in English. Unpublished PhD dissertation,
Northwestern University.
Palmer FR 1972 Semantics. Cambridge, Cambridge University Press.
310
Renouf A, Baayen H 1996 Chronicling the Times: Productive Lexical Innovations in an English
Newspaper. Language, volume 72, number 1.
Roget's Thesaurus 1952. London, Sphere Books.
Webster's Dictionary of Synonyms 1951. Menasha, Merriam.

311
Variation across Korean text registers
Beom-mo Kang and Hung-gyu Kim
Korea University
Our paper describes the results of an adoption of Biber's (1988, 1995) multivariate statistical analysis
of text registers to the analyses of Korea text registers and styles, comparing our results with Kim's
(1990). Here, registers are conceived of as kinds or types of texts with respect to text/discourse
situations and are not distinguished from genres in a broad sense. The text corpus on which our study is
based is composed of 334 text samples of 36 registers/genres of Korean texts. Each text sample is a
computerised one and consists of about 1,000 words, the total size of the corpus being about 370,000
words. Among 13 spoken text registers, we included not only transcriptions of real conversations but
also some texts written to be spoken--transcripts of TV dramas, plays, and movies. Among 23 written
text registers, we included various kinds of informative and imaginative ones--newspaper reports,
editorials, essays, encyclopaedia, academic, informative books, etc. For the multivariate statistical
analyses, we counted the occurrences of 82 linguistics features of each text samples and performed
factor analysis, cluster analysis, and canonical discriminant analysis. Before we counted linguistic
features occurring in texts, we had to perform morphological tagging because many of the linguistic
features could only be counted from morphologically analysed texts. We used automatic morphological
analyser first, and then checked and corrected the tags manually.
By multidimensional factor analysis using SAS, we found that there are six dimensions of Korean
texts: D1. Informal interaction vs. Elaboration; D2. On-line/situated production vs. Informative
content; D3. Narrative vs. Abstract concerns; D4. Formal statement of opinions; D5. Public report; D6.
Public mention of modern things. By cluster analysis, we found that there are eight text types which are
solely based on linguistic characteristics and which are different from conventional genres or registers.
These eight text types are: T1. Abstract content; T2. Modelled informal interaction; T3. Event
description; T4. Opinion; T5. Planned presentation of information; T6. On-line production; T7.
Explanation of facts and procedures; T8. Public report. Our analysis is an application of Biber's
multivariate statistical analysis to Korean texts. The results show that this method is a fruitful one in
investigating aspects Korean text registers and styles, and reveal similar as well as different textual
characteristics of English and Korean.
References
Biber D 1988 Variation across Speech and Writing. Cambridge, CUP.
Biber D 1995 Dimensions of Register Variation: A Cross-linguistic Comparison. Cambridge, CUP.
Kim Y 1990 Register Variation in Korean: A Corpus-Based Study, Ph.D. thesis, USC.

312
Tracing idiomaticity in learner language: the case of BE
3U]HP\VáDZ.DV]XEVNL
6FKRRORI(QJOLVK$GDP0LFNLHZLF]8QLYHUVLW\3R]QD 3RODQG
1. Introduction
It is a widely known fact that language learners, especially less advanced ones, tend to rely excessively
on flexible, high-frequency, ‘core’ vocabulary items in their foreign language use1. One of such
commonly overused verbs is the primary verb lemma BE, whose multifarious nature must discourage
many corpus researchers from devoting it time. In this paper I attempt to construct a version of the
traditional tripartite scale of idiomaticity (frozen - restricted - free combinations) in order to encode
various types of occurrence of lexical BE and test the extent(s) to which particular level(s) of fixedness
are responsible for the reported overuse. The matter is vital, for core words are prone to forming
extensions of all kinds which, contrary to the simple ‘building-block’ metaphor of learner lexical
performance (cf. Kjellmer 1991: 124), indicate proficiency rather than non-proficiency. Before we
announce that learners overuse the commonest words and possibly give them make-up work, it is
useful to find out what exactly learners do with the core lemmas2.
The premises underlying the idiomatic chart proposed here rest on both the traditional,
grammatical criteria for idiomaticity (semantic opacity, lexical/syntactic fixedness, lexical/syntactic
anomaly, cf. Moon 1997: 44, Hudson 1998: 8-9) as well as on corpora-inspired views of
conventionality (viz. frequency and distribution, as reported in LDOCE3) and pragmatic specialisation
in discourse (formulae). Since many tendencies regarding EFL vocabulary production are transferrelated,
the distribution of BE's postulated idiomaticity bands will be shown in a contrastive scheme
comprising both EFL learner and control non-learner and L1-based text collections. The practical goal
of these examinations is to characterise quantitatively the use of BE by Polish advanced EFL learnerwriters.
An underlying methodological objective of the study is to demonstrate how the needs of
learner language phraseological research fail to be served by modern, robust, corpus-driven methods of
text analysis.
2. Idiomatic BE: a major challenge for corpus-driven methodology
The lemma BE poses a major challenge on corpora researchers because of its versatility and extremely
high frequency. In a phraseological study, one of the first tasks that needs resolving is, of course, the
separation of grammatical and semantic3 (here also called lexical) uses of BE. Semantic BE is generally
to be identified not only with the existential, intransitive uses of this verb but also with its linking
(copular) functions, which likewise translate lexically into other languages, a point of importance when
the impact of L1 interference on EFL language production is recalled. The two basic cases of auxiliary
use (cf. Quirk et al. 1985: 129-135) to be excluded from analyses of lexical BE are: 1) ‘central’
passives (as opposed to semi-passives and pseudo-passives in which BE functions as a copula, e.g.
‘This difficulty can be avoided in several ways’ [central passive] vs. ‘Leonard was interested in
linguistics’ [semi-passive] vs. ‘The building is already demolished’ [pseudo-passive]; cf. Quirk et al.
1985: 167-171); and 2) the use of BE as the progressive aspect auxiliary (e.g. ‘Ann is learning
1 Frequency analyses of learners’ language, such as Ringbom's (1998), Altenberg's (1997) or
Hasselgren's (1994), clearly point this way. Resorting to safe lexical items is a frequent communication
strategy not only of learners.
2 Although some corpus linguists consider idiomatic bonds to best operate between wordforms, I
follow a lemma-based approach out of conviction, after Aitchison (1994), that the lemma is the basic
lexical unit of the mental lexicon (cf. Howarth's lexemic approach, 1998).
3 Semantic uses of verbs most often correspond to the main verb function in a clause, but can also be
represented by non-finite forms (infinitives and participles) and gerunds (in non-count forms, i.e.
‘being’ but not ‘a being').
313
Spanish'). Especially the first group proves extremely difficult to tackle with contemporary textprocessing
software.
Another source of complication for disambiguation are multi-word instances of what are
called ‘verbs of intermediate function': neither entirely semantic nor grammatical (Quirk et al. 1985:
96-128, 136f). Two pertinent sub-classes of such verbs are: the modal idiom ‘BE to <do sth>', and the
more open set of semi-auxiliaries, which include ‘BE able to <do sth>’ , ‘BE about to <do sth>', ‘BE
apt to <do sth>', ‘BE bound to <do sth>', ‘BE due to <do sth>', ‘BE going to <do sth>', ‘BE likely to
<do sth>', ‘BE meant to <do sth>', ‘BE obliged to <do sth>', ‘BE supposed to <do sth>', ‘BE willing
to <do sth>', etc. In the performed analysis two other ‘verb idioms which express modal or aspectual
meaning’ (Quirk et al. 1985: 143) have been supplemented: ‘BE inclined to <do sth>’ and ‘BE allowed
to <do sth>’ (= ‘may’ or ‘have permission', as applied by some Polish users). The differentiation
between modal idioms and semi-auxiliaries is essential insofar as the latter approximate the lexical
(linking) uses of BE.
Once all the above enumerated uses of BE can be successfully identified and set aside, we can
proceed to study the remaining lexical uses, which, as with any other verb, exhibit inclinations to form
idiomatic (or frozen), phraseological (restricted / collocational), and open combinations with other
words.
First, the FROZEN idiomatic level of BE may be postulated as consisting of those phrases in
which the verb is literally ‘frozen’ both lexically and formally (as a particular wordform). Such uses are
few and typically specialised functionally, e.g. ‘that is (to say)’ (used to mark repetition), ‘to be sure’
(epistemic modality), or ‘for the time being’ (time disjunct).
Phrasal/prepositional occurrences of BE can rarely be taken as integrated semantic units, but
more like instances of phrasal/prepositional complementation, e.g. ‘BE around’ (= ‘BE available'), ‘BE
on’ (='BE working/ running/ playing'), ‘BE into <sth>’ (=be interested in sth). They will be associated
with the RESTRICTED collocational level, discussed below. One notable exception is the frozen
perfective expression ‘been around’ (=having had many and varied experiences), as in ‘a young
executive who has been around', where the meaning of ‘BE around’ acquires an extended metaphorical
meaning.
Published sources offer little help regarding collocational habits of BE. Collocation
GLFWLRQDULHVFI%HQVRQHWDO.R]áRZVND	']LHU DQRZVNDSUHVHQWYHU\PRGHVWHQWULHVIRU
lexical BE, or do not present them at all. This is because BE is an ‘upward collocate’ (Sinclair 1991) of
so many words that it makes little practical sense to list all of them. A closer look at corpus data,
however, proves that a good percentage of BE tokens and types are somehow conditioned or
conventional, i.e. that they transcend the simple slot-and-filler generative paradigm which links words
according to pre-selected syntactic choices.
As mentioned, lexical BE comes in two basic variants, copular (or linking) and intransitive,
the former usually outnumbering the latter significantly. One important fact about copular verbs is that
they require obligatory complementation, which may be of three structural types (Quirk et al. 1985:
1171-4). Two of them are simple and prototypical: 1) by an adjective phrase (‘BE <adj>'; e.g. ‘the
menace from the plant is serious'), and 2) by a noun phrase (‘BE <noun>'; e.g. ‘the movies are a form
of fiction'). The third kind of complementation involves the use of a (predication) adjunct, whose most
frequent surface manifestation is a prepositional phrase. This type of complementation may be
functionally ambiguous, since its role may be either that of an obligatory adverbial (e.g. representing
the relation ‘BE <place>’ or ‘BE <time >') or of a subject complement resembling a noun phrase or an
adjective phrase (as in the pattern ‘BE of <sth>': e.g. ‘BE of consequence/ substance/ importance'4 etc.;
Quirk et al. 1985: 732). Quite importantly, many of the prepositional phrases functioning as subject
complements of BE are multi-word units, often internally idiomatised (i.e. displaying lexical fixedness
or syntactic abnormality), with corresponding adjectival synonyms, e.g.: ‘BE out of breath’ (cf. ‘BE
breathless'), ‘BE of no importance’ (cf. ‘BE unimportant'), ‘BE not at ease’ (cf. ‘BE not relaxed'), ‘BE
in love', ‘BE in good condition'.
In contrast to their function as subject complements, prepositional phrases acting as obligatory
adverbials seem to merely describe circumstances relating to the subject's — a person's, object's or
event's — ‘being’ (i.e. presence or happening). They thus appear much less tied to the verb BE, which
assumes a decontextualised, intransitive, existential rather than typically copular function. Such a
relation is perceptible between BE and prototypical obligatory adverbials (time, space and metaphorical
space) and, perhaps less strongly, also between BE and other adjunct complements (recipient, purpose,
reason, accompaniment) (cf. Quirk et al. 1985: 731).
4 ‘BE of <noun>’ is an interestingly productive sub-type of prepositional-phrase subject complement.
314
Even less transparent/ compositional (and therefore classifiable as ‘restricted') seem to be the
cases of complementation by: 1) means adjuncts (often conventionalised and/or lexically fixed, or else
possibly replacing a passive or different predicate; e.g. ‘Transport is by ferry', ‘Entrance is by special
invitation', ‘such contracts are (= are signed) with people who...'); 2) stimulus adjuncts (rare,
stylistically marked, and greatly restricted by the subject, which controls the preposition following BE,
e.g. ‘His main interest was in sport'; 3) agent adjuncts (restricted semantically to, most typically,
artistic authorship, e.g. ‘The book was by an unknown writer'5); 4) measure adjuncts (contracting a
non-prototypical sense of BE, though obviously a frequent and salient one among non-beginner English
learners, e.g. ‘The jacket was 10 pounds'6).
From the above survey of the complementation patterns of lexical BE, a general rule can be
inferred that the verb tends to be followed by complements which either constitute idiomatic phrases,
or restrict (specialise) BE's realm of reference (by influencing its subject collocates), or which
otherwise constitute simple, ad-hoc, compositional phrases (adjectival, nominal or prepositional). I
would like to propose for the first two of these types to be joined into a complementation super-pattern
‘BE <idiom>', which will be henceforth associated with a RESTRICTED level of collocability of
lexical BE, on the grounds that: 1) copular BE, by definition, requires a complement (or adverbial), and
2) the type of complement (or adverbial) considered is itself idiomatic. Examples of restricted
collocations representing the two prototypical complementation patterns (‘BE <adj>’ and ‘BE
<noun>') will include: 1) BE + adjectival idioms or collocations (predicatively unified, often
substitutable by a single verb, e.g. ‘BE conditional upon <sth>', ‘BE worth <(doing) sth>', ‘BE alive’
(cf. ‘live'), ‘BE fraught with <sth>', ‘BE sorry for <sb>’ (cf. ‘sympathise'); 2) predicative pseudopassives
and semi-passives (e.g. ‘BE composed of <sth>', ‘BE connected with <sth>', ‘BE interested
in <sth>', ‘BE used to <(doing) sth>', ‘BE situated <somewhere>'; 3) BE + adjectival/past-participial
predicate + to-clause (e.g. ‘BE liable to <do sth>', ‘BE reluctant to <do sth>'; 4) BE + nominal idiom
(e.g. ‘BE a bitter pill (for <sb>) (to swallow)', ‘it BE high time', ‘BE the case (with <sb/sth>)'.
Following the criteria of pragmatic specialisation and frequency, another sub-category of
idiomatically RESTRICTED uses should be associated with lexicalised discourse-related formulae,
which in the case of BE are quite numerous. On account of the transparent, prototypical semantics of
BE and absence of (strong) lexical and syntactic restrictions operating on it, formulaic uses should not,
I believe, be regarded as frozen. Table I below provides a brief summary of suitable subtypes and
instances of formulae:
Table I: Restricted, discourse-conditioned phrases with lexical BE
Pattern/Subtype Example/Sub-pattern
conventional discourse formulas
and linking phrases
‘that/this BE why/ the reason why...’ etc. (often sentence initially)
‘there is every/no reason (for <sb>) to <do sth>’
‘<sth> BE that...*’ ‘the idea/problem/thing is that...’
‘<sth> BE to <do sth>**’ ‘his purpose/task/approach is to <do sth>’
idiomatic referential uses ‘BE so/otherwise’
‘<sb/sth> BE one/those that/who ...’
BE + clause: other formulae*** ‘<sth: the question etc.> BE whether ...’
‘<sth> BE how <sth> <happened>’
‘it/this BE because ...’
Other formulae ‘<sth> BE for <sb> to <do sth>’
‘<sth> BE as follows/the following’ etc.
* a prominent discourse prefacing formula
** a prominent explicational formula, also common in prefacing
*** this sub-type is arguably the least restricted (formulaic) of all the ones tabulated here; it has been added on account of
semantic analogy to other prefacing formulas
Apart from all the restricted occurrences, lexical BE also features certain independent
stylistic/rhetorical uses that are difficult to categorise within the bounds of idiomaticity. One such type
of expression are cleft and pseudo-cleft sentences, where the use of BE (in bold-type in the example
5 Many complementations of this type may be regarded as idiomatic equivalents of the long passive,
e.g. ‘The book was/had been written by an unknown writer'.
6 The pattern of such expressions may be written out as ‘BE <sth>’ but its semantic structure is totally
incongruous with the defining quality of prototypical noun complementation, captured by sentences
such as ‘The prize was 10 pounds'.
315
below) appears a result of a transformation from an underlying non-emphatic predication (underlined
below) rather than a typically lexical instance comparable to cases described earlier:
After all, it is marriage, the beginning of a family that constitutes the very basic part of every nation and society and
as such it is no longer a private affair between two people.
All his people ask for is no more war.
Another stylistically motivated feature is the use of subject-to-subject raising with an optional
infinitival phrase to be, found in copulas ‘SEEM (to be)', ‘APPEAR (to be)', ‘TURN out (to be)',
‘PROVE (to be)', or in complementations of some mental verbs, especially in the passive voice, e.g.
‘BE found/thought etc. (to be)’ 7. Some of these structures may be frequent enough (the most common
‘SEEM (to be) <adj>/<noun>’ may yield up to 100 occurrences in a 100,000-word corpus) to skew
other findings for BE, depending on whether the optional infinitive is included or excluded from global
counts. By an arbitrary decision, in the findings presented below, such optional occurrences of ‘to be’
have been added up with total scores.
Lastly, it is posited that FREE-COMBINATIONAL uses of BE should comprise all the
remaining occurrences of this verb, in particular cases of non-idiomatic complementation within the
two prototypical patterns: ‘BE <adj>’ (including -ed and -ing adjectives) and ‘BE <noun>'8. The third
major sub-category of free combinations will be made up of the de-selected instances of obligatory but
semantically ancillary adverbial complementation (adjuncts): 1) ‘BE <adjunct: time, space,
metaphorical space>’ (e.g. ‘Pure fire (the stars) is in the heavens.', ‘It was 10 years ago.'); and 2) ‘BE
<adjunct: purpose, accompaniment, measure, etc.>’ (e.g. in ‘BE with <sb>; BE for <sth> (=purpose);
‘BE about <sth>', as when used of a book, television programme etc.)
The existential use of ‘there BE’ or the use of ‘BE’ after the anticipatory ‘it’ are, in the scheme
proposed here, assumed as resulting from transformations of the basic theme-rheme informational
model (performed, e.g., to satisfy stylistic or contextual needs) and, unless lexicalised or specialised
(e.g. ‘there is every reason that ...'), such forms will be treated as free-combinational, regardless of
their various detailed functions (cf. Biber et al. 1999: 951-953).
The presented stratification of the potential occurrences of the verb BE demonstrates that,
even if only on account of its highly diversified complementation, it is not justifiable to apply one
yardstick to all instances of lexical BE present in a text. On the contrary, BE seems to feature its own
model of the idiomatic cline, whose investigation may provide useful material not only for EFL
researchers.
3. Automatic interfaces to corpus-bound phraseology
It goes without saying that the technological potential an average applied corpus linguist may have at
his/her disposal will fall short of resolving all the delicacies necessary to describe the idiomatic
distribution of BE. At the heart of the problem lies the discrepancy in the way the basic term
collocation is understood by applied linguists and the way in which it is implemented by corpora
researchers. While the traditional, applied sense of collocation associates it with co-occurrence between
items forming a syntactically interpretable unit (noun phrase, verb phrase etc.), corpus-driven methods
are usually focused on what is easily countable in electronically held text. However, as we shall see,
statistical results based on word spans or adjacent word clusters, although useful in surveying large
bodies of text, cannot fully satisfy due to both incompleteness and overgeneralisation.
7 Quirk et al. (1985: 1173) report that in both British and American English certain copular verbs
(APPEAR, LOOK, FEEL, SEEM, SOUND, REMAIN, STAY, BECOME, END UP, PROVE, TURN,
TURN OUT, WIND UP) prefer infinitive constructions before noun phrase complements. The
statistical results collated in the present study for the most frequent of the verbs, SEEM, have not
confirmed this preference in native-speaker written production, instead pointing to a generally much
more widespread adjectival complementation in which the highly more frequent (by 3-5 times) option
is the one without ‘to be'. Interestingly, this last finding showed an opposite tendency (i.e. preference
for ‘SEEM to be <adj>') among both advanced and intermediate Polish learners of English.
8 One further refinement (not pursued here) within the above group might be to isolate commentmaking
patterns beginning with ‘it’ (which touch upon discourse specialisation) in opposition to
clauses containing nominal subjects (cf. ‘it would be irresponsible to attempt...’ vs. ‘Attempting ...
would be irresponsible.')
316
Precision and recall can be improved by using POS-tagged corpora, but, unless we put in a
major effort to re-edit manually, some data will still slip through, due to systematic inaccuracy of
taggers. It is practically impossible to automatise the labelling of the central passives (as opposed to
semi-passives and pseudo-passives). The deep-tagging program tried for this project, TOSCA-ICLE
tagger (Aarts et al 1997), despite the claimed 95-6% accuracy (de Haan 1997: 218), notoriously
misinterpreted ‘BE used to <doing sth>’ as ‘BE used to <do sth>’ and likewise labelled as passive
each instance of ‘BE related to <sth>', ‘BE concerned about <sb/sth>’ or ‘BE satisfied with <sth>'.
Equally failing may be attempts at automatising the retrieval of significant collocations. One
of the crudest ways is to extract ‘recurrent word combinations’ (also called ‘word clusters', ‘word
bundles', ‘word strings’ etc.). Altenberg (1993, 1998) rightly showed that many of them can exhibit
important, pragmatic functions, particularly in spoken discourse; however, most are not, by definition,
‘idiomatic’ (Biber et al. 1999: 990), and prove difficult to interpret and sub-classify as a group. Their
significance is further undermined by the fact that many genuine collocations and multi-word
expressions are not contiguous (Kennedy 1998: 114) and do not form fixed word strings. Clusters are
certainly appealing for large-corpus research: Biber et al. devote over 30 pages to these combinations
and only about 13 to all other multi-word, idiomatic expressions (Biber et al. 1999: 990-1024).
However, they can uncover only very selective and very incomplete lexical associations, hidden
amongst results that better indicate dominant topics (cf. ‘BE allowed to adopt children’ or ‘with
Down's syndrome BE’ in Polish learner data) or stylistic mannerisms (e.g. ‘it BE obvious that', ‘and
that BE why') than reveal collocational bonding. Many clusters signify no units at all (‘it BE', ‘that
they BE') but cannot be stop-listed since the commonest words (e.g. prepositions) play important roles
in many other, meaningful clusters.
Another approach to automatising collocation extraction is to apply co-occurrence statistics.
These express arithmetically the (relative) strength of the association bond between words that tend to
appear within a specified span (window) of words. Two commonly used co-occurrence formulas (with
variants frequently experimented upon) are mutual information (MI) and the Z-score (McEnery &
Wilson 1996: 71), the latter sometimes replaced by the more accurate t-score.
The philosophy behind MI makes it difficult to apply in studies targeting phrases with highfrequency
vocabulary. MI can be applied successfully to identifying ‘idiosyncratic collocations’ (Oakes
1998: 90) and those which typify domain sublanguages because it privileges ‘rare events’ (Oakes 1998:
177). The limited helpfulness of MI in pointing at significant collocates was confirmed by the fact that
even when the collocate frequency threshold was lowered to 3 (while 5 is said to be a decent statistical
minimum), tests for lemmatised BE (carried out with WordSmith Tools) failed to produce any
significant results within the 4:4 span. This, in view of the stratified phraseological system introduced
above, is a rather questionable result. More prolific can be MI calculations performed for each attested
wordform of BE. The Polish students’ essay-writing corpus displayed connections between ‘I’ and
‘sure', ‘I’ and ‘against’ and ‘I’ and ‘afraid', mostly, however, exhibited relations with infrequent, topicinduced
nouns (‘monarchism', ‘delusion', ‘ritual', ‘centuries') and adjectives (‘doubtful',
‘conspicuous', ‘annoying'), which have less to do with genuinely significant lexicogrammatical
patterning of BE.
A measure of co-occurrence which is less ‘resistant’ to common collocates, is the Z-score.
Running Collgen (a sub-program of the freeware package TACT) on the same corpus of Polish
learners’ essay-writing reported associations (p-level 0.01; span 4:49) between the lemma BE and the
adjectivess ‘able', ‘likely', ‘supposed’ and ‘afraid'. It also pointed out the habitual co-occurrence of BE
and ‘there’ (a most likely indicator of heavy use of the existential ‘there BE’ structure, indeed popular
with learners), or BE and ‘concerned’ (indicative of learners’ frequent over-reliance on the structure ‘as
far as <sb/sth> BE concerned').
However, a problem with rated collocate lists (even those sorted by the Z-score or t-score) is
that they only indicate potentially interesting cases that require much effort and close textual analysis to
verify (e.g. the association of BE and ‘there’ may also imply the verb phrases ‘BE there'). Even access
to annotated corpora is not immediately helpful, since tagging on-the-fly may go wrong and the number
of tag combinations to be queried for a particular type of association is often not entirely predictable,
complicating search patterns and/or prolonging computer processing time.
In short, when precision in obtaining data for a pedagogically oriented study is at stake,
reliance on automatic extraction means often proves insufficient because: 1) too much ‘noise’ is
generated in the data, which, in the case of smaller corpora, may considerably slow down analysis; 2)
9 This is an approximation. Collgen actually calculates co-occurrence statistics from generated wordclusters.
In this case 2-5 word clusters (including the node word) were examined.
317
collocations can spread beyond the typically heuristic 4:4 span, in which case they will be blocked out,
while extending the span would needlessly escalate the ‘noise’ effect; 3) sometimes only grouping data
uncovers a meaningful kind of association, whereas co-occurrence extractors work with orthographic
words and easily skip over, e.g., variants of one idiomatic expression (cf. Stubbs 1998); 4) learner data
(especially at lower proficiency levels) contribute to a further lowering of the statistical ‘precision’ and
‘recall’ of automated procedures, because of grammatical, stylistic, orthographic and other mistakes
and errors. These, unless annotated or corrected in advance by hand, will often confuse taggers and/or
skew statistics10.
4. Contrastive Interlanguage Analysis and the applied corpus network
Learner corpora studies benefit strongly when a multi-corpus network with native and non-native
reference data can be applied. Such is the framework of Contrastive Interlanguage Analysis (CIA,
Granger 1996), which involves two kinds of comparisons: 1) comparison of non-native and native
varieties of the same language (e.g. to identify errors, or to trace ‘foreign-soundingness’ in patterns of
overuse and underuse); and 2) comparison of different non-native varieties of the same language (e.g.
to examine if a given IL phenomenon is bound with a given L1 background or is more universal/
developmental in nature). CIA gains further diagnostic and predictive power when connected with
classical CA, carried out on translation or, as in the corpus network outlined below, parallel corpora.
The English corpora gathered for this project fall into five pre-established proficiency categories, the
central one of which is the advanced-EFL band containing the major Polish learner corpus, IFAPICLE.
In turn, the contrastive Polish part of the network represents two proficiency levels,
expert/professional and college/secondary school learner, which mirror the native English control data.
Table II: The stratification of the corpora used in the study (word token counts: hyphens within words)
non-native English native English
‘apprentice’ corpora ‘expert’ corpora
1. Intermediate
2. Upperintermediate
3. Advanced 4. College 5. Professional
Polish
intermediate
EFL
Spanish
(upper-)
intermediate
EFL
Belgian-
French
advanced
EFL
Polish
advanced
EFL
British and
American
college
learner English
British
academic
writing
British and
American
quality press
PLLC SPAN11 FREN
IFAP(
ICLE)
LOCN(ARG) MCONC12
LOB&BROW
N
92,712
tokens
94,965
tokens
101,442
tokens
107,990
tokens
106,255
tokens
97,914
tokens
94,421
tokens
POL-STUD ‘apprentice’ corpus 4. College level Polish college compositions 103,382
POL-EXP ‘expert’ corpus
5. Professional
level
Polish academic papers +
quality-press articles
101,348
tokens
The Polish advanced EFL corpus IFA-PICLE13 belongs, alongside SPAN, FREN and LOCNARG, to
the International Corpus of Learner English (ICLE) resource, which primarily samples 500-1000-word
10 This point was worth mentioning although BE, the simplest of verbs to use and one of the first to
learn in writing, poses few problems (‘where'/‘were’ and *‘ben’ for ‘been’ were the only reported
cases).
11 The Spanish learner corpus (SPAN), although officially regarded as ‘advanced’ in the ICLE Project
structure, had to be relegated to a lower level as it contained many more grammatical mistakes and
decisively poorer vocabulary in comparison to the other advanced-level EFL data.
12 LOCNARG is a selection of argumentative essays written by English and American secondary
school and college students; the whole resource constitutes the LOCNESS corpus (=Louvain Corpus of
Native English eSSays), the primary control native corpus within the ICLE family, which, arguably, is
more comparable with the non-native learner data than are professionally written text samples.
318
argumentative essays submitted by English university students in EFL countries (Granger 1994 and
1998 ed.). Because problems were encountered finding equivalent texts (in genre and style) and
equivalent sample sizes to those in the ICLE material, the expert corpora, in particular MCONC, as
well as POL-EXP, occasionally include longer and/or incomplete extracts of text cut out of larger
publications. No topic homogeneity could be enforced, either, but efforts were made to include, in the
first place, themes typically represented in IFA-PICLE and the other ICLE learner corpora (e.g. youth
and social problems: violence, drugs, TV-addiction, etc.). PLLC is an extract from the Polish part (over
500,000 tokens) of the 10-million-word Longman Learner Corpus (LLC) including short essay
writings, some of which feature personal rather than argumentative discourse (hobbies, interests, plan
for the future, etc.). MCONC is a collection of manually extracted, jargon-free academic English texts
derived from the MicroConcord text collection B. Academic texts (1993). LOB&BROWN is a
collection of mostly quality- and popular-press extracts retrieved from the LOB and Brown corpora
(ICAME Collection of English Language Corpora 1991), exclusively from text Category B (‘Press:
Editorial') and text Category F (‘Popular Lore'), including analyses on political events, popular science
articles, columns and editorials on every-day life, etc., but excluding short press reports.
5. How Polish advanced EFL writers overuse BE: selective findings
Let us begin with a few procedural remarks. Due to the unmanageably high frequency of BE in each
corpus, most of the demanding disambiguation tasks14 involving non-frozen expressions (passives,
semi-auxiliaries, and restricted / idiomatic phrases) had to be performed on samples of random
concordance lines (500) drawn from each English corpus. When projecting the samples-based scores
onto whole corpora, approximations using the standard error (0.5-2.0%) were performed, and
confidence ranges established, assuming, for the easiest fit, a normal distribution and the minimum
95% confidence level. Some of the values presented below will consequently appear as (partially
overlapping) continua rather than as single scores. Instead of sophisticated statistical testing, which is
often dropped in applied studies of this kind (Granger 1998a), the results (comparisons of frequencies
and percentages) are assessed and commented upon impressionistically.
Secondly, due to the lack of topic homogeneity in the corpora, unexpectedly skewed and
possibly topic-induced frequencies had to be identified. This was done by taking standard deviation
measures for each recorded expression type and group across all the seven corpora and applying a
heuristically established threshold of 2 to discriminate between proportionate and skewed distributions.
The latter cases were then assessed as either instances of genuine quantitative difference or, if text
inspection confirmed consistent connections with a uniquely (over)represented topic, rejected from
further counts.
Quantitative results obtained for the first disambiguation stage (auxiliary vs. lexical BE)
showed a clear underuse of the central passives among Polish intermediate learners, possibly resulting
from a more personal and casual content of their texts. The complex semi-auxiliary structure ‘BE going
to <do sth>’ (predominantly spoken and perhaps stylistically weak, cf. Biber et al. 1999: 489) was
found a characteristic of (less proficient) learner writing that contrasted deeply with native English
expert data, especially its academic variety.
Overused informal expressions will reappear throughout this section, becoming a frequent
feature of many EFL-based findings. Amongst semi-auxiliaries where BE functions as a linking verb,
another instance of informality is a rather significant, consistent overuse of the structure ‘BE able to
<do sth>', noticeable especially in the performance of advanced-level native and non-native writers.
Generally high statistical frequency of this ‘core phrase’ (over 100 occurrences in a million words,
Biber et al. 1999: 517) spreads proportionally across various registers and text-types (conversation,
fiction, news, academic writing), implying that the expression is very familiar to most (foreign)
students of English from early stages in their education. This familiarity may be a conducive factor for
the reported overuse, since the phrase is a safe option for selection in almost any language task.
Passing on to the idiomatic uses of lexical BE, three specific expectations were developed and
tested: 1) negative correlation between rising proficiency and increasing frequencies of single-word
(non-idiomatic) uses and/or with underrepresentation of idiomatic BE; 2) prolific presence of favourite
13 IFA-PICLE is an extract of the PICLE corpus, containing over 230,000 words of running text (365
essays). Full information on PICLE can be found at: http://main.amu.edu.pl/~przemka.
14 Performed with Concord, one of WordSmith Tools.
319
expressions (‘core phrases') in EFL learner data; and 3) traceability of (at least some of) the favourite
expressions to L1 (Polish).
Table III: Lexical BE: Major summary results calculated from 500-line concordance findings
5.
Professional
4.
College
3.
Advanced
2.
Upp-Int
1.
Interm 95% confidence
intervals LOB
&BR
MCONC LOCN IFA-
PICLE
FREN SPAN PLLC
Estimated standardised frequency per 100,000 words
Frozen uses >5
<76
>24
<127
>2
<75
>27
<138
>10
<98
>25
<134
>6
<93
Restricted: BE +
idiom
>314
<525
>282
<504
>325
<552
>324
<568
>299
<529
>228
<443
>188
<391
Restricted:
formulae
>213
<397
>290
<514
>138
<306
>280
<512
>317
<551
>173
<367
>191
<396
Cleft sentences >47
<159
>49
<173
>54
<177
>82
<234
>36
<152
>8
<96
>0
<75
Free combinations >1,717
<1,990
>1,778
<2,086
>2,005
<2,290
>2,346
<2,686
>2,209
<2,528
>2,574
<2,870
>3,317
<3,611
Total: >2,552
<2,892
>2,713
<3,115
>2,775
<3,148
>3,406
<3,793
>3,176
<3,553
>3,260
<3,657
>3,968
<4,300
Estimated % of lexical BE in a corpus
Frozen uses >0.2%
<2.8%
>0.8%
<4.4%
>0.1%
<2.5%
>0.8%
<3.8%
>0.3%
<2.9%
>0.7%
<3.9%
>0. 1%
<2.3%
Restricted: BE +
idiom
>11.5%
<19.3%
>9.7%
<17.3%
>11.0%
<18.6%
>9.0%
<15.8%
>8.9%
<15.7%
>6.6%
<12.8%
>4.5%
<9.5%
Restricted:
formulae
>7.8%
<14.6%
>10.0%
<17.6%
>4.7%
<10.3%
>7.8%
<14.2%
>9.4%
<16.4%
>5.0%
<10.6%
>4.9%
<9.6%
Cleft sentences >1.7%
<5.9%
>1.7%
<5.9%
>1.8%
<6.0%
>2.3%
<6.5%
>1.1%
<4.5%
>0.2%
<2.8%
>0.0%
<1.8%
Free combinations >63.1%
<73.1%
>61.0%
<71.6%
>67.7%
<77.3%
>65.2
<74.6
>65.7%
<75.1%
>74.4%
<83.0%
>80.2%
<87.4%
Total 100% 100% 100% 100% 100% 100% 100%
The summary results presented in Table III find a good deal of agreement with the overall
proficiency-based predictions expresses in the first hypothesis. Lower-proficiency students (especially
PLLC) appear to use fewer collocational idioms (i.e. ‘BE + idiom') than the remaining groups
(corpora), and many more free combinations. In frequency terms, it is perhaps surprising to see the
restricted level best represented in the two EFL advanced corpora, which is due, perhaps surprisingly,
to a high share of formulae in these texts (comparable to the level characterising English academic
writing MCONC). At the same time, frequencies as well as percentage data show that the two
advanced-level EFL corpora and the native learner corpus share a similarly extensive predilection for
the application of free combinations. LOCNARG could have approached the position of expert English
corpora much closer were it not for the lower figures recorded for formulae, particularly the prefacing
structures of the type ‘<sth: idea, purpose etc.> BE that/to...'. However, quantitative studies of
formulae are a shaky matter when EFL data are involved, since learner language has been found to
feature many contextually unsuitable prefabs and so their pure calculation without fathoming the
context may be misleading (cf. Granger 1998b, de Cock et al. 1998).
Passing on to the frozen and restricted levels of phraseology, the data obtained point to the
presence of several expressions and collocations that are favoured by learners, and to many instances
probably attributable to the Polish L1. Thus, the second and third of the formulated hypotheses have
also found at least some confirmation in the tests.
With respect to the idiomatic levels of BE, among the scarcely represented frozen expressions
one worth noting is Polish learners’ apparent overuse of the finite clausal structure ‘what is more’ in
the function of an addition/reinforcement adverbial, e.g.:
320
It is no wonder, that then some easily influenced Poles, who have not been exposed to many such films so far, might
want to try living in a similar manner. What is more, after watching another “Rambo-like” film an average Pole may
be led to thinking that committing a crime is a part of people's existence. (IFA-PICLE)
Although a legitimate idiomatic expression (LDOCE3: 1628), ‘what is more’ (often contracted to
‘what's more') is an emphatic and rather spoken, though infrequent, ‘polyword’ (cf. Altenberg 1998:
117 or Biber et al. 1999: 1008f, 1014f). The origins of the overuse may be sought in the stylistically
rhetorical (usually written3ROLVKWUDQVLWLRQDOSKUDVHµFRZL FHM¶IUHTXHQWO\HPSOR\HGE\ZULWHUVDQG
orators to emphasise and/or extend an argument. Both native Polish corpora consulted (expert POLEXP
and learner POL-STUD) were agreed in pointing to a stable frequency of 9-10 instances per
100,000 words for this use. Although not as high as the one attested for ‘what is more’ in IFA-PICLE
and PLLC, the value is significant enough to indicate Polish-English cognateness (or possibly
transliteration since the phrase is fairly compositional) as a likely factor enhancing the detected
overuse.
Interestingly, Polish users also favour an alike finite structure ‘what is more <adj: important,
significant etc.>’ as a sentence-initial emphasising adverbial, a use more naturally rendered by singleword
adverbs like ‘importantly', ‘significantly'. This, too, can be traced to the habitual Polish
FRQQHFWRUVVXFKDVµDFRZD QHQDMZD QLHMV]HQDMLVWRWQLHMV]H¶HWFDSRLQWZKLFKUHWXUQVLQWKH
discussion of discourse formulae below.
Restricted / collocational associations are much better attested and therefore more convincing.
Within the super-pattern ‘BE + idiom', the following findings concerning Polish EFL essay writers are
worth mentioning:
·  the expression ‘BE full of <sth>’ appears overused by intermediate learners (12 occurrences in
PLLC and 10 in SPAN), and distinguishes itself also in Polish advanced learners’ writing (6); a
transfer trigger mechanism is likely as the expression is more informal in English (MCONC and
LOB&BROWN contain only 1 instance each) than in Polish (POL-STUD: 5 occurrences), where
IRUPDOSHUVXDVLYHGLVFRXUVHUHDGLO\IHDWXUHVDGLUHFWO\FRUUHVSRQGLQJµE\üSHáQ\PF]HJR !¶DVLQ
‘ ZLDWMHVWSHáHQSRNXVDQDWXUDOXG]NDVáDED¶ WKHZRUOGLVIUDXJKWZLWKWHPSWDWLRQVand the
human nature is feeble);
·  ‘BE present’ appears overused in IFA-PICLE (cf. Polish semi-IRUPDOµE\üREHFQ\P¶DVLQ
µ7HOHZL]MDMHVWREHFQDZ \FLXND GHJR]QDV¶32/678'EXWQRWLQ3//&SRVVLEO\GXHWR
genre inconsistency), so it is impossible to fully diagnose the case;
·  the semantic set ‘BE connected/associated etc. with <sb/sth>', and especially the phrase ‘BE
connected with <sb/sth>', appears a strong Polish learner's favourite, with IFA-PICLE recording
12 and PLLC 15 occurrences (as opposed to 0-1 in all the remaining corpora); transfer influence is
RQO\SDUWLDOO\MXVWLILHGVLQFHWKHWUDQVODWLRQDOHTXLYDOHQWDOWKRXJKQRWDFRJQDWHµE\ü
]ZL ]DQ\P¶UHJLVWHUVRQO\-4 times in native Polish expert and learner writing alike;
·  ‘BE concerned with <sth>’ (='deal with sth') is in prevalent use among professional English
writers (8-9), but Poles (and French Belgians) prefer for ‘BE concerned’ to operate in the linking
structure ‘as far as <sth> BE concerned', which may be the reason why they do not apply it to
other contexts; frequencies show that the latter phrase may be a favourite with many EFL
advanced learner populations;
·  ‘BE slow to <do sth>’ is arguably less transparent than most other phrases (cf. the PolLVKµRFL JDü
VL ¶DQGPD\EHDYRLGHGE\RUSHUKDSVLVXQNQRZQWR()/OHDUQHUVZKLOHLWLVUHFRUGHGLQ
moderate use among native expert writers.
With respect to discourse formulae and polywords, the following instances deserve mention:
·  ‘that/this BE why’ appears a highly typically Polish-style linker, featuring 49 occurrences in IFAPICLE
and 56 in PLLC (other corpora: FREN 23, SPAN 13, native-speaker data 1-4, including
LOCNARG). Over 30% of all the occurrences appear in the short form ‘that's why'. The
popularity may be L1-related (POL-EXP and POL-STUD both feature dozens of sentence-initial
‘Dlatego (ZáD QLH¶DVZHOODVVHYHUDOµ=WHJRSRZRGXZ]JO GX¶HWFZKLFKVHHPWUDQVODWLRQDO
pragmatic equivalents). The expression is generally not very formal and unsuitably typical of
Polish EFL learners’ written English discourse;
·  (sentence initial) ‘what is more <adj: important etc.>’ registers a few times in the IFA-PICLE
corpus, although no (heavy) overuse has been detected; what is interesting is that native-English
sources tend to employ it in clefts (e.g. ‘what is more important is that...') while adverbial uses are
typically covered by adverbs (‘importantly', ‘significantly’ etc.);
321
·  ‘BE:’ (e.g. ‘The question was: ....') is a discernible convention in native writing (especially
academic MCONC), but most EFL learners use it twice as often, which may indicate undue
overuse of this simple clause structure.
To summarise this section, the cases characterising Polish learners’ habits mainly concern
overuse and derive from their falling back on L1-inspired options and/or on common, familiar, spoken
(or universal) phraseology. Similar stylistic infelicities may also be observed in the preferred free
combinational uses of BE. For instance, existential ‘there BE', perhaps more characteristic of spoken
English, tends to typify lower-proficiency written performance (with PLLC and SPAN recording by far
the largest frequencies). Another ‘spoken habit’ of Polish users, both advanced and intermediate ones,
is their resorting to anticipatory ‘it’ clauses in preference to longer structures with full nominal subjects
that usually enhance textual cohesion.
6. Conclusion
Quantitative studies of learner phraseology, marked by heavy disambiguation of instances, require
finer, small-corpus based comparisons rather than coarse, corpus-driven, statistical methods. Results
such as those presented above would not have been possible without painful manual analysis allowing
us to reach deep into corpus data. Researchers of learner corpora should strive to capture the hard-toretrieve
covert types of ‘error': the overapplication and avoidance of words, expressions, structures etc.
Indeed, with more advanced learners, it is those unnaturally distributed rather than incorrectly applied
items that characterise ‘foreign-sounding’ style prominently. Regardless of the creative side of
language, much of what native users say and write is influenced by conventions which, at least in
statistical terms, are also expected in learners’ texts and speech, particularly at the university level. The
underlying philosophy is unavoidably prescriptive, in that it presupposes native ‘norms’ against which
learners’ performance can be assessed. Respectable voices advise caution against idealising learner
corpus evidence, especially when it is confronted with such norms (cf. Leech 1998). However, it
seems that unless we falsely hail learner corpora research as the one and only guide to native-like
competence, instead of simply naming it a contributor to more successful learning, a touch of simple,
educational prescriptivism should do little harm. Especially if we beware of obvious methodological
pitfalls, ask demanding questions, carefully prepare and scrutinise data and avoid drawing arrogant,
foregone conclusions.
The message flowing from the presented exercise is that EFL learners do tend to overapply the
simplest uses of the verb BE in comparison with native writers and that the trend is happily less marked
at the advanced level than at the intermediate level. Not only free combinations, however, add to the
overall impression of overuse. A number of popular collocational and frozen expressions with BE,
often inspired by L1 or borrowed from spoken language, also contribute strongly.
Bibliography
--- 1995 TACT (Textual Analysis Computing Tools). Version 2.1. Toronto, University of Toronto.
Aarts J, Barkema H, Oostdijk N 1997 The TOSCA-ICLE tagset. Tagging manual [accompanying the
TOSCA-ICLE tagger/lemmatiser version 1.0]. Nijmegen, University of Nijmegen.
Aitchison J 1994 Words in the mind [2nd ed.]. Oxford - Cambridge, Mass., Blackwell.
Altenberg B 1993 Recurrent verb-complement constructions in the London-Lund Corpus. In Aarts J, de
Haan P, Oostdijk N (eds), English language corpora: design, analysis and exploitation. Papers from
the 13th international conference on English language research. Amsterdam, Rodopi, pp 227-245.
Altenberg B 1998 On the phraseology of spoken English: the evidence of recurrent word combinations.
In Cowie A P (ed), Phraseology. Oxford, Clarendon Press, pp 101-122.
Benson M, Benson E, Ilson R 1997 The BBI dictionary of English word combinations. Amsterdam -
Philadelphia, John Benjamins Publishing Company.
Biber D, Johansson S, Leech G, Conrad S, Finnegan E 1999 Longman grammar of spoken and written
English. Harlow, Pearson Education Limited.
322
de Cock S, Granger S, Leech G, McEnery T 1998 An automated approach to the phrasicon of EFL
learners. In Granger S (ed), Learner English on computer. London, Addison Wesley Longman, pp 67-
79.
de Haan P 1997 An experiment in English learner data analysis. In Aarts J, de Mönnink I, Wekker H
(eds), Studies in English language research and teaching: in honour of Flor Aarts. Amsterdam -
Atlanta, Rodopi, pp 215-29.
Granger S (ed) 1998 Learner English on computer. London, Addison Wesley Longman.
Granger S 1994 The learner corpus: a revolution in applied linguistics. English Today 39 (10/3): 25-29.
Granger S 1996 From CA to CIA and back: an integrated approach to computerized bilingual and
learner corpora. In Aijmer K, Altenberg B, Johansson M (eds), Languages in contrast. Papers from a
symposium on text-based cross-linguistic studies Lund 4-5 March 1994. Lund, Lund University Press,
pp 37-51.
Granger S 1998a The computer learner corpus: a versatile new source of data for SLA research. In
Granger S (ed), Learner English on computer. London, Addison Wesley Longman, pp 3-18.
Granger S 1998b Prefabricated patterns in advanced EFL writing: collocations and formulae. In Cowie
A P (ed), Phraseology. Oxford, Clarendon Press, pp 145-60.
Howarth P A 1998 The phraseology of learners' academic writing. In Cowie A P (ed), Phraseology.
Oxford, Clarendon Press, pp 161-86.
Hudson J 1998 Perspectives on fixedness. Lund, Lund University Press.
Kennedy G 1998 An introduction to corpus linguistics. Harlow, Addison Wesley Longman.
Kjellmer G 1991 A mint of phrases. In Aijmer K, Altenberg B (eds), English corpus linguistics:
studies in honour of Jan Svartvik. London, Longman, pp 111-127.
.R]áRZVND&'']LHU DQRZVND+Selected English collocations [Revised and enlarged edition;
VWHG@:DUV]DZD3D VWZRZH:\GDZQLFWZR1DXNRZe.
Leech G 1998 Preface. In Granger S (ed), Learner English on computer. London, Addison Wesley
Longman, pp. xiv-xx.
McEnery T, Wilson A 1996 Corpus linguistics. Edinburgh, Edinburgh University Press.
Moon R 1997 Vocabulary connections: multi-word items in English. In Schmitt N, McCarthy M (eds),
Vocabulary: description, acquisition and pedagogy. Cambridge, Cambridge University Press, pp 40-
63.
Oakes M P 1998 Statistics for corpus linguistics. Edinburgh, University Press.
Quirk R, Greenbaum S, Leech G, Svartvik J 1985 A comprehensive grammar of the English language.
London, Longman.
Ringbom H 1998 Vocabulary frequencies in advanced learner English: a cross-linguistic approach. In
Granger S (ed), Learner English on computer. London, Addison Wesley Longman, pp 41-52.
Scott M 1996 WordSmith: software language tools for Windows. Oxford, Oxford University Press.
Sinclair J 1991 Corpus, concordance, collocation. Oxford, Oxford University Press.
Stubbs M 1998 A note on phraseological tendencies in the core vocabulary of English. Studia Anglica
Posnaniensia XXXIII: 399-410.
Summers D (ed) 1995 Longman dictionary of contemporary English [ 3rd Edition]. Harlow, Longman
Group Ltd. [LDOCE3]

323
Identifying parallel corpora using Latent Semantic Indexing
Yuliya Katsnelson Charles Nicholas
Highland Technologies
4831 Walden Lane
Lanham, MD 20706
UMBC
1000 Hilltop Circle
Baltimore, MD 21250
Tel: 301-306-2849 Fax: 301-306-8201 Tel: 410-455-2594 Fax: 410-455-3969
ykatsnelson@htech.com nicholas@cs.umbc.edu
Abstract
Identifying parallel corpora can be an important step in a variety of tasks related to information
retrieval. However, at present to identify parallel corpora requires human experts to examine the texts
and evaluate their respective contents. We assume in this research that texts which are translations of
each other have similarities in their semantic structure which are absent between independent
documents.
Latent Semantic Indexing (LSI) (Landauer 1989, Deerwester 1990) is a statistical technique that brings
out correlations between documents based on term co-occurrence patterns identified using the method
of Singular Value Decomposition. LSI does not involve any knowledge of the actual content of the
documents and therefore has no need for human intervention. If LSI could be used for parallel corpora
identification, it would lower the costs incurred in this task. Our purpose is to determine if it is
possible to identify a parallel corpus using the method of Latent Semantic Indexing.
We present evidence that LSI reveals similarities between parallel documents that do not exist between
non-parallel documents and is therefore useful for identifying parallel corpora.
1 Introduction
Parallel corpora identification can be considered an area of Cross-Language Information Retrieval with
applications both inside and outside of the area of IR. Recognizing the relevance of documents in
multiple languages also includes recognizing the relevance of parallel documents. Parallel documents,
in fact, make good test cases for evaluating the effectiveness of a Cross-Language IR system, since if a
document is relevant to a query, then its translation should also be relevant. Also, being able to tell the
parallel part of a corpus from the rest of the corpus allows one to reduce the amount of search, since
only one portion of the parallel subsection needs to be explored.
Parallel corpora analysis, on the other hand, relates largely to the area of document translation, human-
or computer-aided. Parallel documents are included in a corpus in order to provide the reader with a
choice of language in which to read the document. This means that the different versions of the
document need to convey the same information. This objective introduces a number of questions. Is the
corpus parallel? What part of the corpus has a corresponding parallel subset in the same corpus? How
close are the various multilingual translations? Which translation is the best among all the available
versions in the same language? In this research we aim to answer the first question, believing that the
insight that we get in the process will enable us to approach the others.
Latent Semantic Indexing (LSI) is a statistical technique that uses the method of Singular Value
Decomposition (SVD) to represent the terms and documents in the corpus as points in a multidimensional
space. The dimensions of this space represent various patterns of term co-occurrence.
Thus, documents that have similar characteristics with respect to a particular term co-occurrence
pattern have similar coordinates in the corresponding dimension of the LSI space. SVD allows a
reduction of the dimensionality of the LSI space with minimal information loss.
We report on the results of applying Latent Semantic Indexing to identifying parallel corpora. In our
research we used the unsupervised learning approach, relying on the hypothesis that the content
“signature” of the collection is going to outweigh the difference in language-related “noise” and still
324
supply meaningful results. As measures for comparing the LSI representations for parallel and nonparallel
corpora we use correlation coefficient analysis and visual inspection.
We used corpora in the English, French, Russian and Italian languages to assess the behavior of this
method in the cases of more similar (English/French) and less similar (French/Russian) language pairs.
The corpora included abstracts from “The Little Prince” by A. de Saint Exupery, short stories and the
novel “Smoke Bellew” by J. London, “The Adventures of Sherlock Holmes” by Sir A. Conan Doyle,
and “The Black Tulip” by A. Dumas. Our choices of documents were driven largely by their
availability in different languages.
We present evidence that LSI reveals relationships between parallel documents that do not exist
between independent documents and thus, provides a means of identifying parallel corpora.
The remainder of this paper is organized as follows. Section 2 lists the necessary definitions and gives
a brief description of the Vector Space model and Latent Semantic Indexing. Section 3 explains the
research hypotheses and experimental design. Section 4 discusses the results, and Section 5 gives the
conclusions and outlines future work.
2 Definitions and Background
In this section we introduce the main concepts and methods used in the course of our research. We also
briefly describe the Vector Space model (VSM) of Information Retrieval and explain in more detail the
method of Latent Semantic Indexing (LSI) and its advantages as compared with the VSM. We will also
discuss the reasons behind choosing LSI for the task of parallel corpora identification.
2.1 Definitions
Before going any further, we list the definitions of concepts that are going to be referred to throughout
the paper.
1. Parallel corpus – a collection of documents in which at least a subset of documents has
translations within the corpus.
2. Fully parallel corpus – a corpus in which each document in the corpus has a translation within the
corpus.
3. Partially parallel corpus – a corpus in which only a subset of documents has a translation within
the corpus.
4. Literal translation – translation in which the goal is to remain as close to the original choice of
words and structures as possible.
5. Literary translation – translation in which the goal is to remain as close to the original meaning
and imagery as possible.
6. Synonymy – different terms are used to describe the same concept.
7. Polysemy – the same term describes more than one concept.
8. Mixed LSI space – all documents in the corpus are used to create the LSI space for further analysis.
9. Separate LSI space – a separate LSI representation is created for each monolingual sub-collection
of the corpus under consideration.
10. Cognates – words in different languages that are spelled similarly and have the same meaning.
2.2 Vector space model
The three main models of IR are the Boolean, vector space and probabilistic models. We are only
going to discuss the Vector Space model in this paper as the most relevant to the issue in question.
Information on this and other models may be found (Baeza-Yates 1999).
In the vector space model, both documents and queries are represented as vectors. The vector
components are term frequencies, often normalized by the vector length to account for different lengths
of documents. There are a number of methods of computing similarity between these vectors. The most
popular is the cosine of the angle between the vectors.
The vector space model presents more flexibility for evaluating the degree of similarity between
documents than the Boolean model (Baeza-Yates 1999). However, it does not account for the fact that
the occurrence of one query term in the document may influence the likelihood of the occurrence of
another term. For instance, if the query is “computer networks security”, then the fact that the term
“computer” occurred in a document make it more likely that the term “networks” occurs in the same
document than, for instance, the term “knitting”.
325
2.3 Latent Semantic Indexing
Latent Semantic Indexing (LSI) can be considered an extension of the vector space model. LSI claims
that the meaning of a document is not determined by its set of terms, but rather by a latent semantic
structure that manifests itself by how each term is used with all other terms across the entire corpus.
This means that replacing one (or more) term(s) in this structure with their synonyms does not change
the meaning of the document, provided that the latent semantic structure remains intact. The goal of
LSI then is to bring forth this latent semantic structure (Landauer 1989).
The entrance point for this method is constructing the term-document matrix A for the corpus. The size
of the matrix A is [m x n] where m is the number of terms in the corpus and n is the number of
documents. Element aij is the frequency of occurrence of term i in document j.
Once the term-document matrix is constructed, the LSI method uses Singular Value Decomposition
(SVD) to create the LSI space. The result of the SVD is a 3-tuple of matrices U, S, and VT such that:
where matrices U and V are orthonormal (i.e. U*UT = V*VT = I), and matrix S is diagonal (i.e. aij = 0,
" i 1 j).
Matrix U has size [m x m]. An element of U uij contains the coordinate of term i in dimension j of the
LSI space. Matrix VT has size [n x n]. An element of VT vij contains the coordinate of document j in
dimension i of the LSI space.
The matrix S contains the singular values sorted in descending order. The size of this matrix is [m x n].
The number of non-zero diagonal elements in this matrix equals the rank of the original term-document
matrix, which can be at most min(m,n). Therefore, only min(m,n) singular values at the most are
significant for the analysis. Also, since the singular values decrease as they go down the main diagonal
of S, the dimensions corresponding to those values are of decreasing significance. Any first k
dimensions of the LSI space represent the best k-approximation of the original LSI space (Berry 1995).
However, it may not be a good practice to establish a preset cutoff point for the number of dimensions
k under analysis. In the course of our experiments we established 10 L k L 15 as a number of
dimensions that appeared to be the most useful for the purpose of parallel corpora identification.
3 Method and experimental design
The goal of the present research is to see whether it is possible to identify parallel corpora with
computational means rather than through inspection by a trained multi-lingual human professional. We
assume that given an original document A in a particular language, any adequate translation of this
document into another language is going to possess some similarity to the original document. The
hypothesis that is based on this assumption is that Latent Semantic Indexing (LSI) is a suitable method
for showing the similarities between parallel documents.
Thus, the approach taken in the course of this research is to represent both parallel and non-parallel
document collections in LSI space and measure the similarities between corresponding documents. In
this section we discuss the rationale behind applying LSI to identifying parallel corpora and describe
the particular ways LSI was applied for this purpose. We are also going to describe the experimental
strategy taken and the data that we used for the experiments.
3.1 Method
The intuitive way of parallel corpora identification is to create some kind of a dictionary, which
provides a mapping between a term in one language and its equivalents in other languages. One
problem with such a method is that it requires knowledge of what particular languages appear in the
corpus. It is also subject to synonymy and polysemy problems. A good method would allow analyzing
a corpus without needing to know the languages of the documents that comprise it.
A m
n
= U m
m
x S  m
n
VT n
n
x
326
Latent Semantic Indexing has one property that is very attractive for processing multilingual corpora:
No knowledge of the term meanings is required for the analysis, thus eliminating the need for
dictionaries. Once the term-document matrix has been constructed, the only “reality” that exists in the
analysis space is the frequencies of term occurrences with no inherent meaning associated with those
frequencies. Of course, this property can also be construed as a shortcoming of this method, since
some built-in knowledge could be useful in some situations. For instance, it is possible that there exist
two documents with completely unrelated contents, but very similar or identical lexical structure. In
that case, LSI will regard the two documents as similar, despite their semantic differences. However,
in our experience such occurrences are unlikely. The LSI method provides us with the means to
compare the co-occurrence patterns within the corpora in ways that will hopefully bring out the
similarities between parts of the corpus that belong to different languages.
3.2 Structure of LSI space
The critical question that needs to be answered when applying the Latent Semantic Indexing method is
how to construct the LSI space.
The issue of finding translations (mates) within a corpus also relates back to the work of M. Littman, S.
Dumais and T. Landauer on Cross-Language Information Retrieval using LSI (Littman 1996). Their
approach was to create a training corpus in which each document consisted of corresponding multilingual
documents pasted together. This technique allowed them to establish co-occurrence mappings
between languages. Thus, when a newly introduced query produced a pattern similar to the one in the
training corpus, the documents that had similar co-occurrence characteristics in all languages
participating in the training corpus were returned as relevant.
Our assumption that translations of the same document are similar in ways that independent documents
are not allows us to use an alternative strategy, i.e. to eliminate the training corpus, and consider only
the correlation between coordinates in LSI space of multi-lingual documents.
We used two different ways of mapping documents from a multilingual corpus into the LSI space. One
way was to process the entire multilingual corpus simultaneously and construct a single term-document
matrix. This means that every vector in matrix U of the LSI decomposition contains the values for all
the terms, and that the LSI space will be based on the term co-occurrence patterns not only across
documents, but across language boundaries as well. Such a space is called mixed LSI space throughout
this paper. The second way was to create a separate LSI space for every monolingual part of the
corpus and compare the representations of corresponding documents or document vectors in the same
way as in the case of mixed LSI space. We used both mixed LSI space and separate LSI space
approaches in this work, but an in-depth analysis of their respective merits is beyond the scope of this
paper.
This research does not rely on the existence of cognates (i.e. words in different languages that are
spelled similarly and have the same meaning). In the case of the mixed LSI space, every column of the
term-document matrix contains frequencies, raw or weighted, for every term in the corpus. If cognates
were present in significant numbers, then the actual frequency with which a given term occurred in
patterns in a particular language would be obscured. The assumption for this analysis is that no or very
few cognates exist in the corpora and they have no significant impact on its outcome.
3.3 “Mate” dimensions
Since we are interested in measuring parallelism of corpora, rather than concentrating on individual
documents, the objects of our analysis are the vectors of document coordinates within a particular
dimension, rather than the coordinates of a particular document in the LSI space.
By comparing the monolingual parts of these vectors to each other, we claim that if the corpus is,
indeed, parallel, then there is at least one pair of dimensions in the LSI space in which the
corresponding monolingual sub-patterns are similar. This claim implies that the sub-vectors of different
dimensions may be mated, i.e. the patterns in one language reflected in one dimension may be coupled
with a similar pattern in another language. Therefore, for some, if not all dimensions there exists a
“mate” which has the patterns of the multilingual sub-vectors that are similar in a quantifiable way.
This hypothesis is a “twist” on the idea of finding mates developed by Dumais et. al. (Dumais 1997).
In the case of separate LSI spaces, one can argue that, considering the inherent similarities in the subcorpora
of a parallel corpus, there would be also similarities in their LSI representations that would
327
manifest themselves despite the grammatical differences between the languages in which these subcorpora
are expressed. This might mean that the idea of mate dimensions applies also to separate LSI,
although in this case there is no common “context” that was created in the case of mixed LSI by
processing the entire corpus into a single LSI space.
A special case of this "mate dimension" hypothesis exists for languages with similar syntactic structure
(e.g. English and French). That is, the closer the grammatical structures of the languages, the more the
"mate" dimensions converge to the point where in some dimensions the sub-vectors, corresponding to
each language form a similar pattern.
3.4 Data
We constructed a parallel corpus from various digital libraries. We included literary works by Jack
London, Sir Arthur Conan Doyle, A. de Saint-Exupery, A. Dumas pére, and the United Nations corpus.
The experiments were performed on the following language combinations: English/French,
English/Russian, French/Russian, and English/Russian/Italian.
The texts were divided into abstracts of 550-600 words (about one page long). The terms used for this
research were 5-grams (Damashek 1995). An element aij of the term-document matrix A was the
frequency of occurrence of the 5-gram i in the document j. Thus, the number of terms in each
document was around 3000. The use of n-grams allowed us to use the same parser for each language,
bypassing stemming and stop-listing. N-grams have also proven useful with other LSI-based
techniques (Nicholas 1998).
The documents in the corpus were grouped together by language. In a parallel corpus consisting of n
monolingual documents and their translations, document di has document di+n/2 as its corresponding
translation. The number of dimensions in the LSI spaces constructed in the course of this research
varied from 20 to 80.
3.5 Plotting and measuring
We used the standard plotting capability provided by MATLAB to plot the values in the S*VT matrix.
The plots were all two-dimensional. The X-axis shows the sequence number of the document (and of
its corresponding translation) in the collection, and the Y-axis shows the coordinate of that document in
a particular dimension. Lines connect the document points in order to make the patterns easier to trace.
We also analyzed the correlation coefficients between all permutations of monolingual sub-vectors of
each row of the S*VT matrix. This correlation analysis produced as a result a matrix of correlation
coefficients of size [n, n]. Of this matrix only the highest values were chosen to illustrate the results of
the experiment. These results are shown as in Table 1, below. The first and the second column hold
the LSI dimension numbers of the sub-vectors participating in the correlation analysis. The third
column holds the actual correlation values. The header row indicated what were the languages of subcollections
that produced the corresponding sub-vectors. The highest absolute values were selected for
display. We also present graphs in which the two sub-vectors for each dimension are plotted for
comparison, as shown in Figure 1, for example.
Table 1. Correlation coefficient table sample.
English sub-vector dimension Russian sub-vector dimension Correlation coefficient
1 2 0.8334
3 3 0.8902
3 4 0.8610
3 5 0.7996
7 7 0.7831
7 8 0.8342
328
Fig. 1. Plotting schema.
The coordinate points for the documents are connected in the plots in order to make the pattern formed
by them more evident. Each language participating in the experiment has its own color legend: red for
English, green for French, blue for Russian, and black for Italian.
4 Results
This section is structured in the following way: we will first show the difference between the parallel
and non-parallel corpora representation; we will then show, within parallel corpora, the examples of
grammatically similar and dissimilar language combination. We will also talk about literal vs. nonliteral
translation. The majority of the research was done on mixed LSI spaces. However, the separate
LSI space experiments produced interesting results that will be mentioned briefly in this section.
4.1 Mixed LSI Space
4.1.1 Parallel vs. non-parallel corpora
The results shown in this category are very important because they confirm the main hypothesis of this
research – that the representation of a parallel corpus presents similarities between parallel parts of the
corpus that are absent in a non-parallel corpus.
The documents in this collection come from “Smoke Bellew” by Jack London, namely the parts called
“The Taste of the Meat” and “The Race for Number One”. In the case of a parallel corpus, the excerpts
from both novels were combined into a single collection with their respective translations and the LSI
space was created from them. In the case of a non-parallel corpus, the English part of the corpus was
comprised of “The Race for Number One” excerpts and the Russian part from “The Taste of the Meat”
excerpts.
The highest-value portion of the correlation coefficient matrix for the parallel corpus part of this
experiment (see Table 2) shows that the values are well above the tentative threshold of 0.75 assumed
in the course of the experiments. Also, the S*VT plot in Figure 2 shows some obvious similarities. For
instance, the sub-vectors in dimensions 5 and 7 present opposite trends, whereas dimension 6 has
similar patterns for both sub-vectors. The English sub-vector in dimension 4 is very similar to the
Russian sub-vector in dimension 5. Thus, the results of this experiment support the hypothesis.
English sub-vector dimension Russian sub-vector dimension Correlation coefficient
3 3 0.8621
4 4 0.8793
4 5 0.9318
6 6 0.7890
Table 2. Correlation coefficients for parallel corpus from "Smoke Bellew".
The analysis of the parts of this corpus separately showed that “The Taste of the Meat” was less
precisely translated into Russian than “The Race for Number One”. This might account for lower
values for the former collection. Also, the patterns formed by each sub-corpus in this experiment
remain distinct when the collections are combined.
The first 15 dimensions for this experiment involving the non-parallel collection from the same texts
did not produce nearly as high similarity estimates as the previous experiment (see Table 3). However,
there is still some similarity of behavior visible in dimensions 1 and 2, 3 and 4, and 7 and 8 of the S*VT
329
matrix plot (see Figure 3). The reason may be that the novels were written by the same author, and on
a similar topic. However, in this experiment, the similarities in the visual representation of the corpus
are accompanied by low values of correlation coefficients.
English sub-vector dimension Russian sub-vector dimension Correlation coefficient
1 3 0.6368
7 17 0.6168
7 18 0.6173
1 2 0.5439
4 3 0.3629
8 7 0.2568
Table 3. Correlation coefficients for non-parallel corpus from "Smoke Bellew".
Figure 2. Parallel corpus example, English and Russian.
Figure 3. Non-parallel corpus example, English and Russian.
We can see that the highest correlation coefficient for the non-parallel corpus is 0.64 and the lowest
correlation coefficient for the parallel corpus shown above is 0.78. We therefore set a tentative
threshold of 0.75 for the correlation coefficients in judging a corpus as parallel. The data for the nonparallel
corpora that we have tested with so far has proven consistent in not providing correlation
coefficient measures above 0.70 (usually it was much lower).
330
4.1.2 Similar vs. dissimilar languages
The texts examined in this category are the French original of the “The Little Prince” by Antoine de
Saint-Exupery with its Russian and English translations.
The analysis procedure for this category is similar to the one demonstrated in Section 4.1.1. In this
series of experiments, the range of values in grammatically similar and grammatically dissimilar
languages are not significantly different. The highest correlation coefficient for the English/French
pair is 0.99, and 0.98 for the French/Russian pair. In other words, both grammatically more and less
similar languages participating in parallel corpora produce high correlation values, and don't produce
them in a non-parallel corpus. However, depending on the style of the translation, the difference
between these values may be more significant, although the experiments done so far show that these
values still fall within the 0.75-1.0 range.
We made the following observations about the S*VT matrix. In the French/English collection of
excerpts from “The Little Prince” we saw that starting with dimension 1, the similar patterns are most
often observed within the same dimension, e.g., dimension 1, dimension 3, etc. In dimension 4 we saw
an effect of opposing patterns, where the English pattern is the same as in the previous dimension, but
the French is the opposite. Thus, the idea of “mate” dimensions converging towards the same
dimension seems to be supported.
The French/Russian representation also presents strong similarity trends, but here the “mate”
dimensions are more dispersed, e.g. similar patterns occur in dimension 1 and 2, 3 and 4, 5 and 6, 8 and
12, etc. This also seemed to confirm the “mate” dimensions suggestion. The highest correlation
coefficients for this series of experiments were between 0.99 and 0.88.
4.1.3 Literal vs. literary translation
A literary translation is intended for transmitting both the meaning and the spirit of the original work.
This means that, considering that different languages have different imagery mechanisms, code words,
and idioms, it is almost impossible to translate a literary text literally into another language and obtain
a translation that's worth reading. Thus, the only claim that can be made towards the usability of LSI
in application to literary translation is that it measures not so much the literality of translation, as its
consistency; the style of the original and of the translation needs to have similar “texture”. For
instance, if the author of the original work uses a single imagery for describing the blue sky and the
translator uses three different images for the same purpose, the consistency of the translation will
suffer.
The examples that we worked with here are the excerpts from “Smoke Bellew”, specifically, parts of
“The Race for Number One” and “The Taste of the Meat”. In this pair, “The Taste of the Meat” is the
less literal translation. The correlation coefficient values for this experiment are not very high (the
highest is 0.86), nor the dimensions in which they are high, very multiple. However, these values are
located within the assumed threshold and the dimensions that possess these high similarity values are
within the first 10-15 dimensions.
The analysis of the second element in the selected pair, “The Race for Number One” produced the
following much higher correlation coefficient values (the highest 0.9) and there is a significant number
of dimension pairs with similarity value above 0.75.
The separate LSI space approach presented similar results to those described for the mixed LSI space,
which seems to support our hypothesis. For additional information on these and our other experiments,
we refer the reader to the first author's thesis (Katsnelson 2000).
5 Conclusion
The application of the LSI method and the subsequent correlation analysis showed that parallel corpora
produce larger correlation coefficient values for rows of the S*VT matrix of the term-document matrix.
The row plots of this matrix are also visibly more similar between the parts of a parallel corpus than
between the parts of a non-parallel corpus.
We analyzed several aspects of corpora comparison, such as parallel vs. non-parallel collections, literal
vs. non-literal translations, and document collection pairs in languages with different degrees of
grammatical similarity. The proposed threshold of 0.75 held for all types of parallel corpora. The
331
experiments also showed that visual analysis is as important for the application of our method to
parallel corpora identification, as the statistical analysis.
The limitation of our experiments has been that the number of dimensions that we have worked with
was low due to the computational expense of SVD, and the limited number of documents. In the future
we are planning to address this issue by both increasing dimensionality of experimental LSI space and
considering alternative methods of dimensionality reduction that are less computationally demanding.
We also hope to extend this method into evaluating the quality of translation.
References
T. K. Landauer, M. L. Littman, "Computer information retrieval using latent semantic structure". U. S.
Patent No. 4,839,853, Jun 13, 1989.
S. Deerwester, S. T. Dumais, T.K. Landauer, G.W. Furnas, and R.A. Harshman. Indexing by latent
semantic analysis. Journal of the Society for Information Science, 41(6), 391-407, 1990.
R. Baeza-Yates and B. Ribeiro-Neto, "Modern Information Retrieval", ACM Press, New York, 1999.
M. W. Berry, S. T. Dumais, and G. W. O'Brien. Using Linear Algebra for Intelligent Information
Retrieval. SIAM Review, 37(4), 1995, pp. 573-595, 1995.
M. L. Littman, S. T. Dumais, and T. K. Landauer. Automatic Cross-Language Retrieval Using Latent
Semantic Indexing. SIGIR96 Workshop on Cross-Linguistic Information Retrieval, 1996.
S. T. Dumais, T. A. Letsche, M. L. Littman, and T. K. Landauer. "Automatic cross-language retrieval
using Latent Semantic Indexing", AAAI Spring Symposium on Cross-Language Text and Speech
Retrieval, March 1997.
M. Damashek, "Gauging similarity with n-Grams: Language-Independent Categorization of Text",
Science, February 1995, Vol. 267, pp. 843-848.
C. Nicholas and R. Dahlberg. Spotting Topics with the Singular Value Decomposition. PODDP'98, St.
Malo, pp. 82-91,March 1998.
Y. Katsnelson, "Parallel Corpora Identification Using LSI", M.S. thesis, UMBC Department of
Computer Science and Electrical Engineering, January 2000, http://www.cs.umbc.edu/~ykatsn1.

332
Exploiting large corpora:
A circular process of partial syntactic analysis, corpus query and extraction
of lexicographic information
Hannah Kermes & Stefan Evert
Institut für Maschinelle Sprachverarbeitung, University of Stuttgart, Germany.
1 Introduction
Our approach follows the work of Eckle-Kohler (1999) who used a regular grammar to extract
lexicographic information from text corpora. We employ a system that allows to improve her querybased
grammar especially with respect to recall and speed without reducing accuracy. In contrast to
Eckle-Kohler (1999), we do not attempt to parse a whole sentence or phrase at once during the
extraction process, but build complex structures incrementally. The intermediate results are annotated
in the corpus and used as input for following iterations.1 This concept enables us to accommodate new
aspects such as agreement information and the use of annotated structures together with their features
(for partial parsing as well as for extraction purposes).
Our goal is to design a tool that can be used with both small and large corpora. In addition to partial
syntactic analysis we provide queries (based on the parsing results) for an interactive use. The idea is to
build up flat annotations of (maximal) syntactic constituents (noun phrases (NP), prepositional phrases
(PP), adjectival phrases (AP), adverbial phrases (AdvP) and verbal complexes (VC)) incrementally,
using a multi-pass algorithm. The chunks/phrases allow embedding of chunks/phrases of other
categories as well as recursive embedding, but no PP attachment.
The incremental structure-building procedure enables us to analyse chunks/phrases independently of
their immediate context, i.e., even if we cannot parse the whole sentence or phrase we might still parse
part of it. Analyses of complex chunks/phrases have to be executed only once, the results being
annotated. Consequently, even when dealing with very large corpora, interactive queries and the
extraction process are relatively fast. Besides, as features specifying the character of chunks/phrases are
annotated, structures having certain characteristic can be extracted easily and quickly.
We are also able to include agreement morphology in the process. Thus, we can check the content of
chunks/phrases with respect to agreement features such as case, number, and gender. Agreement
information of chunks/phrases is disambiguated to the extent possible without guessing and added to
the structural mark-up.
2 Technical framework
Our tools are based on the IMS Corpus Workbench2 (CWB). The CWB is an environment for storage
and querying of large corpora with shallow annotations. Currently, the maximum size of a single
corpus can be approximately 300 million words, depending on number and type of annotation. The
CWB provides fast access to corpora by using a separate lexicon and a full index for each annotation
level. The data are stored in a compact proprietary format, and compressed with specialised algorithms
(Huffmann coding for the token sequence and variable-length encodings for the indices).
The CWB was initially developed for corpora annotated at token level only (typically with part-ofspeech
(PoS) and lemma values). Later, support for flat, non-overlapping structural annotation was
added (referred to as structural attributes). Since this mark-up was intended for the annotation of
document structure (e.g., source files, paragraphs, and sentences), the regions of structural attributes are
neither hierarchical nor do they allow recursion. Besides, compression algorithms are not necessary to
store the relatively small number of sentences, paragraphs, etc. in a corpus.
Queries can be specified in terms of regular expressions over tokens and their linguistic annotations,
using the Corpus Query Processor component (CQP). In contrast to most CFG-based parsers, the CQP
query language allows complex expressions at the basic token level. These include regular expression
matching of tokens and annotated strings (optionally ignoring case and/or diacritics), tests for
1 A similar approach was conducted by Steve Abney for English using a cascaded finite-state parser. (cf. Abney 1991 and 1999)
2 See Christ (1994) for an overview. More information on the IMS Corpus Workbench is available from
http://www.ims.uni-stuttgart.de/projekte/CorpusWorkbench .
333
membership in user-specified word lists, and arbitrary Boolean expressions over feature-value pairs.
Additionally, “global constraints” can be used to specify dependencies between arbitrary tokens in a
CQP query.
CQP includes a simple macro language, based on string replacement with interpolation of up to 10
arguments. The “body” of a macro (which is substituted for each macro “call” in a query) may contain
further (non-recursive) macro invocations. Thus, complex queries can be broken down into small parts,
similar to the rules of a context-free grammar. Macro definitions are loaded from text files and can be
modified at run-time.
The CWB includes support for set-valued annotations encoded as string values using a special
disjunctive notation. This allows treatment of agreement features, which is one of our improvements
over the work of Eckle-Kohler (1999). For example, all possible combinations of case, gender, and
number values a certain noun may have are stored as a single, set-valued annotation (a feature vector).
Unification of feature values (which is the basis of most complex grammar formalisms) is equivalent to
the (set-theoretic) intersection of the corresponding feature vectors. Special operators based on regular
expressions are available to test for the presence of features (e.g. a noun phrase which might have
genitive case) as well as the uniqueness of feature values (a noun phrase uniquely identified as
genitive).
We considered a number of alternative approaches for the parsing stage as well, in particular those for
which standard tools are available.
·  Complex grammars (e.g., in the LFG or HPSG framework) can model the hierarchical
structure of language, and are well-suited for handling attachment ambiguities. Drawbacks
include slow parsing speed, lack of robustness, dependence on an extensive lexicon as a
prerequisite, and the complex interactions between rules that complicate both grammar
development and the adjustment to a particular domain seriously. Furthermore, complex
grammars usually return a large number of possible analyses for each sentence, which cannot
be stored and queried efficiently for large corpora. Thus, an additional, and probably rather
unreliable disambiguation component or labour-intensive manual disambiguation would be
necessary.
·  Context-free grammars (CFGs) are modular (i.e. there is little interaction between different
rules) and allow for fast parsing. In most CFG-based systems, however, modelling agreement
and special (lexical) constructions requires large numbers of additional rules, which makes
grammars unwieldy and slows down the parsing process. For the automatic analysis of large
amounts of text, further “robustness” rules are needed. Partial or full disambiguation of
agreement information is difficult to achieve.
·  Probabilistic context-free grammars (PCFGs) extend the CFG formalism with a statistical
model of lexical information such as subcategorisation frames and collocations. In contrast to
complex grammars, probabilistic “lexicon entries” are learned from the input text and training
data without human intervention. PCFG-based parsers are slower than their CFG counterparts
and require a considerable amount of working memory for their large parameter sets. A
particular problem for PCFGs are marked constructions and other special cases, where the
parser almost inevitably prefers a more frequent unmarked alternative. In general, PCFG
parsers perform a full disambiguation of agreement features involving guesswork rather than
the partial disambiguation that we prefer.
To sum up, the advantages which led us to use the CWB as a framework for our tools are: (i) The
possibility to work with large corpora. After compression, the surface forms and lexical annotations
(lemma, etc.) require approximately 30 bits/token of disk space, whereas categorical annotations (PoS,
agreement features, etc.) require 10 bits/token or less. (ii) CQP efficiently evaluates complex queries on
large corpora. Disk files are accessed directly and do not have to be loaded into memory first. It is this
feature which makes a multi-pass algorithm (in which CQP is frequently restarted in order to re-use
intermediate results; see section 334 for details) feasible at all. (iii) The query language is modular and
allows easy treatment of special cases (using additional rules, word lists, or structural mark-up of
multiword entities). (iv) The same representation formalism and query language can be used at the
parsing stage, for interactive querying of the final results, and for the extraction of lexical information.
334
3 General concept
The general concept of our approach is to combine two usually separate processes (see also Figure 1):
(i) annotation of syntactic structures and (ii) extraction of linguistic information. Central to our
approach is a set of hierarchical query macros. These macros serve basically two purposes: (i) they can
be used by an annotator tool to analyse syntactic structure and (ii) they can be interactively used to
extract linguistic information. In the first case, the results of the queries are annotated as structural
mark-up in the corpus, incrementally building up larger and larger structures. In the latter case, the
queries are used by a human user to extract linguistic information from the corpus. For that purpose,
the queries use the structural mark-up the annotator tool constructed by means of the queries
themselves.
Queries
Annotator
Tool
Linguistic Information
Corpus
Figure 1: general concept
The results of the extraction process (in the form of linguistic or lexicographic information) may then
be used either directly by the user or they may be fed back into the queries to improve them. Useful
information for the latter are wordlists, subcategorization information, etc.. The annotator tool as well
as the human user can then use the refined queries to optimise both the annotation and the extraction
process.
4 The parser
The parser is a combination of Perl scripts and CQP queries applied in multiple passes. The results
produced by CQP are post-processed by Perl scripts. These scripts check the results with respect to
agreement and possibly other criteria rejecting inappropriate structures. The remaining structures are
then annotated in the corpus. This procedure is repeated several times in order to build more and more
complex structures. The annotated results of previous steps are taken as additional input for further
steps. In general, there are two different types of passes: (i) a primary “preparing” pass that is executed
only once, and (ii) a secondary pass that is run several times (see Figure 2).
335
Pass 2 Pass 1
Figure 2: the parsing process
In the primary pass simple non-recursive chunks/phrases are built. These chunks/phrases are mostly
syntactically and/or lexically special chunks, i.e. chunks/phrases that are either special with respect to
their distribution, their grammatical function, or their structure. Collecting and annotating these chunks
in a preliminary step enables us to model these chunks/phrases independently of other chunk/phrase
rules, which considerably improves performance and speeds up the parsing process. The queries can be
adjusted to the special characteristics (lexical or structural) of the chunks/phrases. These features are
made accessible in the form of chunk annotations. Thus, in further steps these chunks/phrases may be
either treated as regular chunks or according to their special character.
The secondary pass constitutes the main part of the parsing process. It is designed to run several times.
The number of runs is not predetermined. Currently, we run it three times, which seems to provide a
sufficient depth of embedding for most of the text in our corpora. Each iteration of the secondary pass
takes the annotated structural mark-up of all previous steps (primary or secondary) as input to build
more complex chunks/phrases.
The macro queries resemble the rules of a context free grammar (CFG), i.e., they state that
chunks/phrases may contain a specifier, certain embedded chunks/phrases (we do not distinguish
between modifiers and complements) and a head. Yet, in contrast to Eckle-Kohler (1999) and other
CFG approaches, the embedded chunks/phrases do not have to be re-analysed for every query but can
be accessed as structural mark-up (annotated in one of the previous steps). This makes it easier and
faster to query optional elements, optional coordination and the special elements/chunks described
above (see Table 1 and Table 2 for details). Recursion, in this context, is simply the inclusion of
chunks/phrases of the same category that have been built in previous steps.
Chunks/phrases built in intermediate steps need not be the most complex structure possible, but are
annotated to serve as input for further steps. Whenever a larger structure is found, the Perl scripts delete
the smaller structure in favour of the new and more complex one. If the most complex structure of a
phrase is not found, because of tagging errors or lacks in the grammar rules or the lexicon, the
intermediate structures remain, and may, nevertheless, contribute to future extraction.
5 Grammar details
In the primary step special chunks/phrases are annotated along with features specifying their
characteristics. These chunks are identified using one or more of the following criteria: (i) PoS-tags
(multiword proper nouns), (ii) certain makers in the text itself such as brackets or quotes (terminology,
multiword units), (iii) a lexicon in the form of word lists or subcategorisation information (temporal
nouns, measure nouns, complex adjectival phrases), (iv) morpho-syntax (invariant or verbally derived
adjectives). The features annotated with adjectival phrases and noun phrases are listed in Table 1 and
Table 2.
336
Table 1: annotation features of adjectival chunks/phrases
Table 2: annotation features of noun chunks/phrases
Beside of these lexical/structural features we have also annotated (partially) disambiguated agreement
features and the head lemma. As in complex grammars such as LFG or HPSG, the characteristics of
the head project to the chunk/phrase. Similarly, the characteristics of intermediate chunks project to
larger chunks/phrases with the same head. This projection of features is made possible by the use of
Perl scripts that copy annotated features from smaller to larger structures, modifying them where
necessary.
For instance, agreement disambiguation is performed in every intermediate step (where possible). The
partially disambiguated agreement features are annotated along with the chunk/phrase. Later
disambiguation steps use the annotated agreement features of relevant chunks/phrases (e.g., in the case
of noun phrases, we use the agreement morphology of the determiner, the embedded adjectival phrases,
and the noun chunk containing the head). Thus, (partial) disambiguation for a certain set of words does
not have to be repeated.
A special method was chosen to check the agreement of multiple APs embedded in a NP. Every AP is
annotated with a feature specifying its suffix (more precisely, the last letter of the adjective head, which
is extracted by a Perl script in the first pass). All APs sharing the same suffix are assumed to agree in
their morpho-syntactic features as well .3 APs whose heads are invariant adjectives are ignored.
The chunks/phrases are built using relatively simple queries. A noun phrase, e.g., consists of a noun or
year date as head and only obligatory element, which may have truncated elements (cf. Table 2). In
pre-head position there are optionally a determiner, a cardinal number and a (theoretically) unlimited
number of adjectival phrases. Cardinal numbers and adjectival phrases may be coordinated. Post-head
there are optionally an unlimited number of genitive NPs, plus a noun chunk in quotes and/or in
brackets:
das alkoholfreie Bier “Kelts”
the nonalcoholic beer “Kelts”
3 The method was adopted from the German Gramotron grammar (cf. Schulte 2000).
Annotation features of adjectival chunks/phrases
feature description examples
invar
vder
meas
norm
quot
pp
invariant adjective
verbally derived adjective
AC/AP embedding a measure noun
regular AC/AP
AC/AP in quotes
AP embedding a PP or year date
Berliner Justiz; fünfziger Jahre
Beginnender Aufschwung; überzeugende
Antworten
Hektar groß; Meter langen
´´falsche''; ´´allzu plötzlich'’
von seiner Frau geborgten; schon vor Jahren
entdecktes
Annotation features of noun chunks/phrases
feature description examples
ne
time
year
meas
news
address
tel
trunc
quot
brac
proper noun
temporal noun
year date
measure noun
news agencies
street name and number
telephone numbers
nouns with truncs
NC/NP in quotes
NC/NP in brackets
Walter Holm; Mecklenburg-Vorpommern
Mai, Feierabend
1999
Dollar Strafe; Hektar Pachtland
LONDON (afp / rtr / AP)
Musikantenweg 14; Museumsgasse 1
Tel. 23 33 25 oder 23 15 47
Schadenersatz- oder Schmerzensgeldanspruch;
Verantwortungs-, Solidaritäts-, Gerechtigkeits-, Gleichheits-
und Freiheitsdenken
´´Autonome Organisation''; ´´United Democrats'’
(LKA); (Framingham Heart Study)
337
den Namen “Werner-Herr-Haus”
the name “Werner-Herr-Haus”
das Quartett “Itchy Fingers”
the quartet “Itchy Fingers”
der Telefonnummer 602-316 (Herrn Borns)
the telephone number 602-316 (of Mister Born)
The pre-head as well as the post-head elements have to occur in the given order, yet, they do not
depend on their predecessors. Table 3 gives a graphical overview of the noun phrase structure we use.
Table 3: elements of noun chunks/phrases
An adjectival phrase also has only one obligatory element, its head, which is an attributive adjective.
Optional pre-head elements are adverbial phrases, a modifying particle (allzu [großer] (far too [big]);
zu [hohen] (too [high])), and, in the case of verbally derived adjectives or adjectives subcategorising
PPs, a prepositional phrase or a year date (see Table 4). A prepositional phrase simply consists of a
preposition as its head and an obligatory noun phrase (which may optionally be coordinated)
Table 4: elements of adjectival chunks/phrases
6 Evaluation
For evaluation purposes, we applied our parser to a 40-million-word newspaper corpus4. The corpus
was preprocessed and part-of-speech tagged with standard tools (cf. Schmid 1995 and Schiller 1996).
In order to reduce memory consumption during the parsing process, the corpus was automatically split
into slices of approximately 500,000 tokens. Each slice was encoded as a separate corpus on which the
parser was run. The structural annotations of all slices were then recombined and annotated in the
original corpus.
The parsing process5 took 13.5 hours on a standard 933 MHz Pentium III notebook with 128 MBytes of
RAM. This amounts to an average speed of 3 million words per hour. Processing speed varied across
different slices, with some slices taking almost twice as long as the “fastest” ones.
For a first quality assessment, we selected 100 sentences at random (excluding incomplete sentences
and sentences containing spelling mistakes) and manually annotated noun phrases according to the
extended chunk concept introduced in section 5. Unlike other manual annotation tasks, there is little
ambiguity in the assignment of noun phrase boundaries when PP attachment is excluded, and
agreement between the authors was easily reached.
4 The Frankfurter Rundschau (FR) corpus from the ECI Multilingual CD-ROM I.
5 Including the splitting and recombination steps, but not including preprocessing and part-of-speech tagging.
Elements of noun chunks/phrases
pre-head head post-head
optional Determiner
CARD*
adjectival phrases*
truncs*
noun
proper noun
optional genitive NC/NP
proper noun
NC in brackets
NC in quotes
year dates
Substitutive pronouns
*optional coordination
Elements of adjectival chunks/phrases
pre-head Head
optional adverbial phrase
modifying particle
Adjective
optional prepositional phrase
year date
verbally derived adjective
or
adjectives subcategorising PPs
338
We did not evaluate the agreement features assigned to the phrases because they are only partially
disambiguated and there is no guesswork on the side of the parser (any errors in case assignment etc.
are due either to the morphological analysis or to tagging errors). For the same reason, other phrase
types (such as PPs or APs) were not evaluated. PPs, for instance, can easily be identified by the parser
once the corresponding NP has been found.
In the 100 test sentences, we found 477 noun phrases. Automatic parsing yielded 487 NPs, of which
440 were true positives. This corresponds to a precision of 90% (i.e. 90% of the NPs identified by the
parser were correct) and a recall of 92% (i.e. 92% of the manually annotated NPs were also found by
parser).
Looking at the results in detail, two major factors account for the number of false positives and NPs
that were not found: tagging errors (e.g., the colon : tagged as a noun) and proper nouns that were not
correctly identified (e.g., Thomas Doll, where Doll was erroneously tagged as an adjective). If we
correct such errors (which can easily be done using lists of proper nouns etc.), the number of false
positives drops from 47 to 13, and we obtain precision and recall values of 97% and 98%, respectively.
Another four of the eight noun phrases that our parser still cannot identify contain adjectives with PP
complements. Feeding adjective subcategorisation frames extracted from the annotated corpus back
into the parsing process will thus further improve the results.
7 Extraction
As mentioned above macro queries may not only be used for annotation but also for extraction. For this
purpose they build on the results of the annotation process, searching on structural mark-up that they
have produced before. In this case, the annotated features are an important knowledge source making
morpho-syntactic information and characteristics of the chunks/phrases easily accessible. The
extraction process itself can be automatic or semi-automatic depending on the query, i.e., the results of
the queries may need manual checking before they can be used for the different purposes (e.g.,
lexicographic or linguistic).
Interesting for the extraction process are, e.g., chunks/phrases enclosed in brackets or quotes. These
“structural” markers are relatively secure signs of elements belonging together. Thus, the elements
enclosed can give hints regarding subcategorisation information of various kinds, but also with respect
to multiword units, idiomatic expressions and collocations. Multiword units, in particular multiword
proper nouns, sometimes occur in brackets or quotes, where they can be assembled securely. The same
holds for abbreviations of terms, which often occur in brackets preceded by the respective term.
“Teenage Mutant Hero Turtles”
(FC Italia Frankfurt)
„Club Marienthaler Carnevalisten“
„Rocky Horror Picture Show“
„Johann Strauß Ensemble Frankfurt“
(Sprecher Wolf Hardy Pulina)
„Arbeiter Samariter Bund“
Technische Überwachung Hessen (TÜH)
Deutscher Aktienindex (Dax)
Daimler-Benz Inter Services (Debis)
Stickstoffdioxyd (NO2)
The annotation of structural mark-up can also help to model sentence positions in which certain
elements occur without having to parse the whole sentence. Thus, the information these positions
include can be accessed, even if the parse of the whole sentence is not successful. In the first position
of German main clauses, for example, which is referred to as Vorfeld within the framework of the
topological field model (cf. Wöllstein-Leisten et al. 1997), only one constituent may occur. Adverbs
following NPs in this position can, consequently, be supposed to belong to the class of post-noun
modifiers.
Das “modernistische” Konzept hingegen lebt ...
The „modernistic“ concept however lives …
339
Er selbst berichtet ...
He himself reports ...
Die Rationalität alleine ist ...
The rationality alone is ...
Herrn Frank persönlich wünsche ...
Mister Frank personally wishes ...
Das Volk indessen lässt ...
The people meanwhile let ....
It is also possible to overgenerate certain structures, annotating them only temporarily in the corpus. If
they are not embedded in a larger construction or prove to be correct in another way, they may be
deleted again, otherwise they remain annotated in the corpus. Due to this possibility, structures can be
annotated that would need subcategorisation information not available in the lexicon. These structures
can than be queried and taken as evidence for certain subcategorisation frames. An example are
adjectives subcategorising PPs that can build complex adjectival phrases with the respective PP. In this
case, we overgenerate APs allowing all adjectives intermediately to build complex APs with preceding
PPs or year dates. The APs are deleted again, unless they are embedded in a NP after the last
annotation step, i.e., they are preceded by a cardinal number or determiner belonging to the head noun
of the NP.
Einem auf die Betreuung Aidskranker spezialisierten Sozialarbeiter
A on the care of people suffering from aids specialised social worker
“A social worker specialising in the care of people suffering from aids”
Der für chinesische Verhältnisse kleinen 20 000 Einwohner zählenden Stadt
The for Chinese standards small 20 000 inhabitant having city
“The city with 20 000 inhabitants, a small city by Chinese standards”
Die dafür erforderlichen 300 000 Mark
The for this needed 300 000 Marks
“The 300 000 Marks needed for this”
Die für eine Aufrufung des Rates notwendigen 60 Abgeordneten
The for a summoning of the council necessary 60 delegates
“The 60 delegates necessary to summon the council”
8 Acknowledgments
Part of our work was done within the framework of the DEREKO project. The DEREKO (Deutsches
Referenzkorpus) project is a joint project of the Institut für deutsche Sprache (IDS) in Mannheim, the
Institute for Natural Language Processing (IMS) in Stuttgart, and the Seminar für Sprachwissenschaft
(SfS) in Tübingen. The project is funded by the Ministry of Science, Research and the Arts of the State
of Baden-Württemberg. The goal of the project is to improve the infrastructure for text-based linguistic
research and development by making accessible a large well-balanced German text corpus. This corpus
is intended as a source for linguistic and lexicographic information. The target group for the resulting
infrastructure are lexicographers, dictionary publishers, manufactures of terminological databases and
ontologies as well as linguists.
9 References
Abney S 1991 Parsing by chunk. In Berwick R, Abney S, Tenny C (eds), Principle-based parsing.
Dordrecht, Kluwer Academic Publishers.
Abney S 1999 Partial parsing via finite-state cascades. In Proceedings of the ESSLLI ’96 Robust
Parsing Workshop.
Christ O 1994 A modular and flexible architecture for an integrated corpus query system. In Papers in
Computational Lexicography COMPLEX ’94. Budapest, Hungary, pp 22-32.
340
Christ O, Schulze B M, Hofmann A, König E 1991 Corpus Query Processor (CQP). User's Manual.
Institut für Maschinelle Sprachverarbeitung, Universität Stuttgart, Germany.
Eckle-Kohler J 1999 Linguistisches Wissen zur automatischen Lexikon-Akquisition aus deutschen
Textcorpora. Berlin, Logos Verlag.
Schulte im Walde S 2000 The German statistical grammar model: development, training and linguistic
exploitation. In Arbeitspapiere des Sonderforschungsbereichs 340 Linguistic Theory and the
Foundation of Computational Linguistics 162. Institut für Maschinelle Srpachverarbeitung, Universität
Stuttgart, Germany.
Wöllstein-Leisten A, Heilmann A, Stepan P, Vikner S 1997 Deutsche Satzstruktur. Grundlagen der
syntaktischen Analyse. Staufenberg Verlag, Tübingen.

341
A tagset for the morphosyntactic tagging of Arabic
Shereen Khoja
Computing Department,
Lancaster University,
Lancaster, LA1 4YR.
s.khoja@lancaster.ac.uk
Tel: 01524 592329
Fax: 01524 593608
Roger Garside
Computing Department,
Lancaster University,
Lancaster, LA1 4YR.
rgg@comp.lancs.ac.uk
Tel: 01524 593803
Fax: 01524 593608
Gerry Knowles
Department of Linguistics and
Modern English Language,
Lancaster University,
Lancaster, LA1 4YT.
g.knowles@lancaster.ac.uk
Fax: 01524 843085
A morphsyntactic tagging system for Arabic has been around for centuries. When analysing sentences,
Arabic grammarians give to each word a detailed morphosyntactic tag or part-of-speech. These tags
contain a large amount of information, perhaps more than what would be found in Indo-European tags.
This is because Arabic words are formed by following fixed patterns, and by knowing the tag of the
word, we know its pattern, and can thus predict various properties and meanings of the word. It is from
this tagging system that we derive our Arabic tagset.
The Arabic tagset we describe does not follow the EAGLES recommendations for the morphosyntactic
annotation of corpora, but this is to be expected since Arabic is very different from the languages for
which EAGLES was designed, and belongs to the Semitic family rather than the Indo-European one.
Following a normalised tagset and the EAGLES recommendations would not capture some of Arabic's
relevant information. However, the Arabic tagset can be mapped onto the EAGLES recommendations.
In Arabic we have three major categories, these are Noun, Verb and Particle. To handle other common
features of Arabic we have included another three categories, Residual (foreign words, abbreviations,
etc), Numerals, and Punctuation (comma, period, etc).
The EAGLES recommendations include the major categories described above (except the Particle) plus
eight other categories. These are Adjective, Pronoun/Determiner, Article, Adverb (which in Arabic are
all considered to be subcategories of the Noun), and Adpositions, Conjunctions, and Interjections
(which are subcategories of the Particle in Arabic). The Unique category in the EAGLES
recommendations is applied to categories with a unique membership, which do not follow any of the
standard part-of-speech tags.
In this paper we will show that although the Arabic tagset we have derived is very different to any
Indo-European tagset, it is appropriate for Arabic. A detailed description of the tagset, along with
examples of how the tags map onto real words will be included in this paper. We will also describe the
results of using this tagset with an Arabic part-of-speech tagger to tag an Arabic corpus.

342
Web as corpus
Adam Kilgarriff
ITRI, University of Brighton
1. Introduction
The corpus resource for the 1990s was the BNC. Conceived in the 80s, completed in the mid 90s,
it was hugely innovative and opened up myriad new research avenues for comparing different text
types, sociolinguistics, empirical NLP, language teaching and lexicography.
But now the web is with us, giving access to colossal quantities of text, of any number of
varieties, at the click of a button, for free. While the BNC and other fixed corpora remain of huge
value, it is the web that presents the most provocative questions about the nature of language. It also
presents a convenient tool for handling and examining text.
Compared to LOB, the BNC is an anarchic object, containing ‘texts’ from 25 to 250,000 words
long, screeds of painfully formulaic entries from the Dictionary of National Biography, conversations
monosyllabic and incoherent, sermons, pornography and the electronic discourse of the Leeds United
Football Club Fan Club.
Compared to the web, the BNC is an English country garden. Whatever perversities the BNC has,
the web has in spades. First, not all documents contain text, and many of those that do are not only text.
Second, it changes all the time. Third, like Borges's Library of Babel, it contains duplicates, near
duplicates, documents pointing to duplicates that may not be there, and documents that claim to be
duplicates but are not. Next, the language has to be identified (and documents may contain mixes of
language). Then comes the question of text type: to gain any perspective on the language we have at
our disposal in the web, we must classify some of the millions of web pages, and we shall never do so
manually, so corpus linguists, and also web search engines, need ways of telling what sort of text a
document contains: chat or hate-mail; learned article or bus timetable.
These may sound like arguments for not studying the web: for scientific progress, we need to fix
certain parameters so we can isolate the features we want to look at, and the web is not a good
environment for that. This is true. For the web to be useful for language study, we must address its
anarchy. If the web is a torrent and nothing more, it is not useful; for it to be useful, we must channel
off manageable quantities to irrigate the pastures of scientific and technological progress.
2. The D3CI
We are developing the D3CI (Distributed Data Distributed Collection Initiative) a framework for
distributed corpora. This will comprise a set of corpora contributed by anyone with an on-line corpus to
offer, where each corpus comes in the form of a set of URLs. The “virtual multicorpus” website will
then be a place to visit for anyone wishing to download a corpus of some known language-variety.
Corpus measures (Kilgarriff 2001) will be used to identify the homogeneity of each submitted corpus.
We shall check for duplicates (Bouyad-Agha and Kilgarriff 1999). We shall provide a program which
will go and collect a set of web pages and deliver it to a user. We shall develop links with that part of
the WWW community which is examining ways of using links between documents and other strategies
to automatically identify interesting clusters of interconnected pages (e.g. Chakrabarti 2000). Our
medium-term goal is to set up a suite of web-based corpora that can be used by linguists and language
technologists to answer questions of the form: “my theory/algorithm/program works well on the text
type I developed it for: I wonder how well it generalises to other text types.”
The use of the web addresses the hobgoblin of corpus builders: copyright. If material is on the
web, it has been published and can be downloaded without infringing copyright. If I wished to store
that material, put it on a CD and distribute that CD, I would be infringing copyright. If I merely present
a list of URLs and announce to the world that this URL set comprises a corpus (of a given text type
which I also describe) then I am clearly not infringing copyright. There are also no administrative, CDburning
or postage costs associated with web-based corpora.
To the objection that web pages die, so a corpus defined as a set of URLs would be forever
shrinking, we propose the following solution. Our virtual corpora are monitored by an agent, which
periodically checks that all URLs are still live. On discovering that one no longer is, the agent, which
has gathered a statistical profile of each of its pages, sets out to find a new page or pages to replace the
deceased. First, it submits a web search, using the terms in the deceased as search terms. This gathers in
a set of candidates. Then, using corpus similarity measures, it identifies which of the candidates do in
343
fact have the same linguistic form as the deceased. It then adds them to the corpus. The virtual corpus
will evolve.
Some may object, “but that is not suitable as use as a corpus because the texts that are there today
are not identical to those that were there yesterday, so how can we compare results?” Results can be
compared because the text type is the same. To demand more is to demand that tomorrow's
experiments on the water flow in the River Lune involve the same water molecules as yesterday's.
3. Related Work
We are not the first to note the web's usefulness for corpus research, despite its short history.
Since the mid-nineties, the net has commonly been used by summarisation researchers as a source of
documents to summarise. In this context, Radev and McKeown (1997) use internet-accessible
newswire as a knowledge source for a language generation system.
More recently, researchers have used collections of papers found on the web for a very wide range
of purposes. Grefenstette and Nioche (2000) and Jones and Ghani (2000) explore the potential of the
web as a source of language corpora for languages where electronic resources are in short supply, and
Resnik (1999), as a source for bilingual parallel corpora. Fujii and Ishikawa (2000) use the web to
generate encyclopaedia entries. Grefenstette (1999) presents prospects and experiments regarding the
web as a source of lexical information; as the web provides thousands of contextualised instances of
even fairly rare words, for many languages, it offers vast opportunities for automatic distillation of
lexical entries from empirical evidence. Varantola (2000) pursues a similar theme, showing how
translators, when confronted with a rare term, can find ample evidence of the term, its contexts, and
associated vocabulary, through the simple use of a search engine. Specialised ‘lexicographic’ search
engines have been produced (see http://www.webcorp.org.uk ) though their relative merits compared
to, e.g., google (which provides some linguistic context for each occurrence of the word, all at
breathtaking speed) remains an open question. Mihalcea and Moldovan (1999) and Agirre and
Martinez (2000) use the web as a lexical resource, and as a source of test data, for Word Sense
Disambiguation. Jacquemin and Bush (2000) use it as a source for harvesting lists of named entities.
There has recently been a Web track in the TREC Information Retrieval Competition (see
http://pastime.anu.edu.au/WAR/webtrax.html).
A field such as this, with its newness and no entry costs, is immediately appealing to students and
others, and the list above is of course incomplete. It does indicate how the use of the web as a corpus is
taking off fast.
4. Conclusion
To conclude: the BNC was one of the greatest innovations for linguistics in the 1990s. Now the
world has moved on. As corpus linguists, we are in the fortunate position of having a particular
perspective and channel of attack for examining the web --perhaps the most extraordinary phenomenon
of our time -- which also just happens to provides solutions to many of our practical problems and an
endless stream of new data. We have presented a model which uses the web as a source of data, the
web as a delivery medium, and in which the web, and its language, are objects to be explored.
The corpus of the new millennium is the web.
5. References
Agirre E and Martinez D. Exploring automatic word sense disambiguation with decision lists and the
web. In proceedings of COLING Workshop on Semantic Annotation and Intelligent Content,
Saarbruecken, Germany. August 2000.
Bouayad-Agha N and Kilgarriff A. Duplication in Corpora. In proceedings of Second Computational
Lingusitics in the UK Colloquium, Essex, 1999.
Chakrabarti S. Invited talk. Joint SIGDAT Conference on Empirical Methods in NLP and Very Large
Corpora. Hong Kong. October 2000. (http://www.cse.iitb.ernet.in/~soumen)
Fujii A and Ishikawa T. Utilizing the world wide web as an encyclopaedia: Extracting term
descriptions from semi-structured text. In proceedings of the 38th Meeting of the ACL, Hong
Kong, October 2000, pp. 488-495.
Grefenstette G. The WWW as a Resource for Example-Based MT Tasks. Invited Talk, ASLIB
‘Translating and the Computer’ conference, London. October 1999.
Grefenstette G and Nioche J. Estimation of English and non-English Language Use on the WWW. In
344
proceedings of RIAO (Recherche d'Informations Assistee par Ordinateur), Paris, 2000.
Jacquemin C and Bush C. Combining Lexical and Formatting Clues for named entity acquisition from
the web. In proceedings of Joint SIGDAT Conference on Empirical Methods in NLP and Very
Large Corpora. Hong Kong. October 2000, pp. 181-189.
Jones R and Ghani R. Automatically building a corpus for a minority language from the web. 38th
Meeting of the ACL, Proceedings of the Student Research Workshop. Hong Kong. October 2000,
pp. 29-36.
Kilgarriff A. 2001 (in press) Comparing Corpora. International Journal of Corpus Linguistics.
Mihalcea R and Moldovan D. A method for word sense disambiguation of unrestricted text. In
proceedings of the 37th Meeting of ACL. Maryalnd, USA, June 1999, pp. 152-158.
Radev D and McKeown K. Building a generation knowledge source using internet-accessible
newswire. In proceedings of the Fifth Applied Natural Language Processing conference.
Washington D. C.., April 1997, pp. 221-228.
Resnik P. Mining the web for bilingual text In proceedings of the 37th Meeting of ACL. Maryalnd,
USA, June 1999, pp. 527-534.
Varantola K. Translators and disposable corpora. In proceedings of CULT (Corpus Use and Learning
to Translate). Bertinoro, Italy. November 2000.

345
A long-standing problem in Corpus-Based Lexicography
and a proposal for a viable solution
Dimitrios Kokkinakis
Sprakdata, Göteborg University
Box 200, SE-405 30,
Sweden
Dimitrios.Kokkinakis@svenska.gu.se
Abstract
This paper describes the application of a framework for text analysis to the problem of distinguishing
unusual or non-standard usage of words in large corpora. The need to identify such novel uses, and
augment machine-readable dictionaries is a constant battle for professional lexicographers that need to
update their resources in order to keep up with the development of the dynamic and evolving aspects of
human language. Of equal importance is the need to devise automatic means upon which we can
evaluate to what extent a (defining) dictionary accounts for what we find in corpus data. A combination
of both semi-, and automatic means have been explored, and it seems that Machine Learning might be
a plausible solution towards the stated goals.
1. Introduction
Lexicography is concerned with the study of how words are used in a natural language by abstracting
away surface differences between them. This paper deals with the question of creating a methodology
that can aid professional lexicographers to distinguish novel or non-standard usage of (lexicalized)
words. The process is automatized to a large extent, using state-of-the-art implemented Natural
Language Processing (NLP) software for written Swedish, such as different kinds of annotators and
analyses tools, coupled with large repositories of static and dynamic lexical knowledge. The NLP tools
are geared towards the goal of distinguishing non-standard usage applied on very large bodies of texts.
Approximated by these tools, surface differences between words are reduced while at the same time
linguistic information is successively added in the data in order to create a suitable representation that
can be matched and measured by a Machine Learning software. The obtained results, that do not
conform to the norm according to the lexical resources and the method explored, need manual
inspection in order to make a decision regarding the actual, or not, identification of a new or extended
usage. However, the inspection process can be automatized, and accelerated, as soon as large portions
of texts have been appropriately marked and analyzed, since Machine Learning techniques can be
applied to calculate the difference or distance between previously analyzed and newly processed
material. A distance to the nearest marked instance for each test item over a certain threshold, might
indicate that an unusual or novel usage is identified, while under a threshold might indicate a
prototypical usage.
The presentation that follows is not about word sense disambiguation (WSD), sense or
semantic tagging per se, it is rather about how to use WSD, and other supporting NLP technologies, in
a practical situation. Moreover, the discussion that follows is not a polemic point of view against the
defining lexicon we chose, or any similar lexicons of that kind. We are fully aware of the limitations in
terms of space, time, resources etc. that are prohibitive for producing better coverage and richer in
content dictionaries. Dictionaries are and will always be incomplete and it is simply impractical to give
descriptions of all possible sense nuances of a word, regardless of the theoretical model incorporated in
the dictionary. Still, however, a typical scenario for many scholars, such as professional lexicographers,
is the need to have sophisticated software in their every-day work in order to cope and organize the
rapidly growing bulk of large corpora in electronic form, to automatically aid the evaluation of
(defining) dictionaries against textual data, and to automate the process of discovering non-standard
usage of words. The operational definition of non-standard usage we adopt is tightly connected to the
use of an existing dictionary. Accordingly, corpus instances can either fit into the dictionary
descriptions provided (standard usage) or not (non-standard usage).
This paper is organized as follows. We start by describing what we mean by “non-standard”
use and give some background information regarding other efforts towards the same or similar goals,
chapter (2); a brief presentation of various static and dynamic lexical resources as well as annotation
346
and analyses tools for written Swedish will be presented, chapter (3); chapter (4) describes the method
explored for the discovery of non-standard usage; while chapter (5) presents a worked example and
discusses some preliminary results obtained; finally chapter (6) end the presentation by giving some
general conclusions and directions for future research.
2. Background
2.1 What is non-standard use?
There is a justified need to distinguish new senses and novel usage of words, and accordingly augment
defining dictionaries with “fresh” material. This is regarded as a constant battle for professional
lexicographers that need to update their artifacts reliably and rapidly, in order to keep up with the
development of the dynamic and evolving aspects of human language, and its constant and rapid
change. In this paper we will use the terms: new, novel and non-standard usage or sense rather
interchangeably referring to, roughly, the same thing. Moreove, when we talk about these terms we
should always try to keep in mind that they are closely related to given dictionary senses. That is, “any
numbered section of a defining dictionary entry which supports its own definition or requires separate
treatment from the surrounding material”; (Atkins 1987).
Thus, the definition we adopt for the work presented in this paper, regarding what might
constitute a non-standard use of a particular word, is based on the lexical resources in disposal, in our
case the Gothenburg Lexical Database (GLDB). Consequently, if a word in a specific context does not
match any readings of the descriptions, provided by the lexicographers, in this specific dictionary, then
it will be worth examining it manually, in order to establish if a non-standard usage has been detected.
Manual inspection is justified by the fact that deviating readings might simply be productive
extensions1, (easily) covered by applying some kind of mapping rule or process to the available
readings, or, they might be errors produced by the complex interaction of the different tools applied on
the test data.
For similar experiments, but with manually inspecting large text instances for the purpose of
evaluating the Generative Lexicon see Kilgarriff (2000). Kilgarriff acknowledges that close reading of
definitions from a published dictionary does not provide an ideal method for distinguishing standard
from non-standard uses of words. However, the method has no fundamental flaws, he says, and there is
no better method available. A disadvantage of the work presented by Kilgarriff is that he starts by first
identifying a set of words to test whether the Generative Lexicon accounts for a novel word use.
However, this cannot always be practical due to the fact that working with multi-million corpora and
the totality of a natural language, at least as described by a dictionary, makes the suggested approach
infeasible. Therefore, what we need is better, more general means that can be applied to more than a
handfull of words. Nevertheless, I agree that manual judgement is still necessary regardless the
qualitative or quantitative view of the approach adopted.
Comparisons between established machine-readable dictionaries (MRDs) and text collections
have shown that there is a gap between the two in different dimensions. Since accurate, robust analysis
tools for large corpora and machine-readable dictionaries were rare, until recently, even for overexplored
languages, such as English, only a very coarse estimate of the coverage of the dictionaries has
been studied. This is usually taking the form of identifying ‘new’ words, words not in a ‘master
wordlist’ of ‘existing’ words, or counting how many of the words found in various text collections
were in various machine-readable resources, such as the Longman Dictionary of Contemporary English
(LDOCE) and the COLLINS dictionary; (Renouf (1993), Krovetz (1994)). Renouf (1993), for instance,
discusses the use of several analytical tools or ‘filters’ to identify ‘new’ words in a flow of data on the
basis that they are not in a list of ‘existing words'. It is easy to confuse the finding of ‘new’ words and
the identification of ‘new’ senses, since there are some similarities between the two. However, the
identification of the later requires more sophisticated and precise software tools, and is a much more
difficult task. Therefore, the lack of greater sophistication in the software tools that were available for
working with large corpora has been prohibitive for researchers to go a step further and actually try not
only to discover new words but new senses of existing, lexicalized words. Clear (1994) comments that
1 For instance, the compound word brandvägg ‘fire-wall’ is not found as an entry in GLDB. The head vägg,
however, is, with a sub-sense (1c) intended to cover metaphoric or extended usage for that specific word.
Consequently, the (metaphoric) sub-sense for brandvägg namely, “a way to protect unauthorised access to a
computer” is covered this way.
347
if one had software that could reliably categorise citations (i.e. concordance lines2) into semantic
subsets, one could find answers quickly and easily to questions such as which citations out of the set do
not appear to match any of a pre-defined set of word sense categories. Accordingly, potentially new
uses of this word could be identified. After all, in the very relevant research area of WSD the interest
has been persisently focused, until very recently, on quality WSD of a handful of target words, rather
than quantity (Yarowsky (1995); Leacock et al. (1996)). Some large-scale WSD efforts on all content
words are described in Kilgarriff & Palmer (2000) in the framework of the SENSe EVALuation
(SENSEVAL) exercise.
2.2 Previous research
The idea of employing automatic techniques dealing with the topic of discovering new or novel usage
is not new, on the contrary. In early work by Wilks (1980), within the developed “Preference
Semantic” system, methods for dealing with extensions of word-senses are discussed. These are based
on the incorporation of richer semantic structures, called pseudo-texts, and the observation of
unexpected contexts. The shortcoming of the approach presented there, however, was the inadequacy to
deal with many forms of lexical ambiguity. The elaborated mechanism of templates for all part-ofspeech
that was developed had to be both too general, for the creation of semantic representations, and
too specific, to aid disambiguation. In his experiments, however, Wilks (1980) observes that extended
or new usage is actually the norm in ordinary language use (at least for English).
Syntactic cues are used by Dorr & Jones (1996) for the derivation of semantic information and
for augmenting on-line dictionaries for novel verbal senses. The syntactic cues are divided into distinct
groupings that correlate with different word senses. For a very large number of verbs, the syntactic
signatures, or syntactic patterns, are used. These are compared with Levin's, Levin (1993), verb classes
and information from LDOCE. According to the algorithm presented by the two authors: if a verb is in
Levin's lists it is classified accordingly; if not, WordNet synsets are used (lists of synonym semantic
concepts, Miller et al. (1990)), and if the synonym is in Levin's classes they select the class that has the
closest match with canonical LDOCE codes; if there are no synonyms, or LDOCE codes, a new verb
class is created. Syntactic signatures are of the form: ‘X broke the vase to pieces’ which, according to
Dorr & Jones, becomes ‘[np, v, np, pp(to)]'.
Similarly, Wilks et al. (1996) describe a method, referring to an unpublished manuscript by
Jim Cowie at CRL-NMSU, on an effort to piggyback a dictionary from a corpus and a seed MRD. By
applying the described method, all the occurrences of a word in a corpus are classified as belonging to
sets of senses defined by a lexicographer who has examined a subset of the occurrences of the word
using concordances. The authors argue that after a sufficient number of example senses have been
marked it should be possible to classify the remaining instances of a word using different techniques.
More interestingly, it may be possible to highlight unusual usages (or different unclassified senses) by
identifying instances where the overlap occurrence is low, and subsequently it is necessary to examine
these instances manually. Cases where the overlap is high may indicate archetypical example usages.
Hanks (1996) argues that the semantics of verbs are determined by their complementation
patterns, discussing an empirical, semi-automatic approach, where it is necessary to identify typical
subjects, objects and adverbials, and then group individual lexical items into sets. In creating the
behavioural profiles of verb lemmas, such as ‘urge’ in large corpora, Hanks showed that 10% of the
uses of ‘urge’ are metaphors and figuratives, while the most common patterns account for 61% of the
occurrences “a person urging another person to do something”. Moreover, Hanks proposed, that for
unusual uses of words it is advisable to statistically sort their collocates into relevenat sets, to give a
name and note possible correlations among different sets in particular roles (subject, object), and
explain the relation by appealing to criteria of ellipsis, rhetoric, etc.
Finally, Tapanainen & Järvinen (1998) describe a tool that utilizes syntactic information and
produces dependency syntax-based concordances between lemmatised words. The tool can be used for
detecting relatively invariable phenomena, which, according to the authors, are collocations with fixed
order and strict precedence. The tool can be used for studying, for instance, long distance dependencies
by clustering sentences according to different syntactic structures. Although the authors do not directly
claim that their tool can be used for discovering new usage of words, we think that it can certainly be
an important step towards that direction as well.
2 A concordance line is a formatted version or display of all the occurrences or tokens of a particular type in a
corpus, the type is usually called the keyword or target or search item. Concordances form the main source of
information in computer assisted lexicography.
348
3. Lexical and algorithmic resources
3.1 Lexical resources
The lexical resources that are used in this work consist of the GLDB, which is the largest, most
comprehensive lexical resource for modern Swedish, upon which a number of defining dictionaries
have been produced, Malmgren (1992), and the extended content of the Swedish SIMPLE semantic
lexicon (Lenci et al. (1998)), over 25,000 entries, Kokkinakis et al. (2000). For the classification of
proper names into semantic classes, a named-entity recognizer (NE) is used, Kokkinakis (1998). The
semantic classes in the NE recognizer fall into the categories LOCATION, HUMAN, TIME and
ORGANIZATION. Proper names are both frequent and have a serious impact for the disambiguation of
the surrounding context. The NE module is also classifying personal pronouns referring to humans as
well as appositive nouns (e.g. he, she, professor) to the class HUMAN.
3.2 Algorithmic resources & machine learning
The most important NLP tools that comprise the algorithmic resources are a rule-based part-of-speech
tagger; a semantic tagger, Kokkinakis et al. (2000); a sense tagger (for content words), Kokkinakis &
Johansson Kokkinakis (1999a); and a cascaded finite-state parser, Kokkinakis & Johansson Kokkinakis
(1999b). Moreover, various finite-state based software that identify and mark idioms, multiword
expressions, phrasal verbs, and perform heuristic compound segmentation and lemmatisation are used.
The idioms consist of approx. 4,500 different ones, according to the GLDB. Compound
segmentation is based on the distributional properties of graphemes, trying to identify grapheme
combinations that are non-allowable when considering non-compound forms in the Swedish language,
and which carry information of potential token boundaries. The heuristic technique behind the
segmentation is based on producing 3-gram and 4-gram character sequences from several hundreds of
non-compound lemmas, and then generating 3-gram and 4-grams that are not part of the lists produced.
After manual adjustments and iterative refinement a list of such graphemes has been produced and used
for segmentation. Ambiguities are unavoidable, although the heuristic segmentation has been evaluated
for high precision. Finally, lemmatisation is based on the output from the part-of-speech tagger and the
rich feature representation that can be found in the part-of-speech tags. Examples of such grapheme
sequences are ‘ngs|s’ and ‘iv|b', e.g. forskningsskola ‘research school', skrivbord ‘writing desk'; ‘|’
denotes where the segmentation will take place.
Machine learning techniques, Mitchell (1997) are used in order to automate the calculation of
the overlap of the contexts between word that are candidates of defining a novel sense. More
specifically, we adopt a supervised, inductive, classification-based variant of Machine Learning called
Memory-Based Learning (MBL), and a specific implementation by Daelemans et al. (1999) called
TiMBL. Using such techniques, the contexts, or instances or analyzed, modified concordance lines, can
be sorted by calculating the distance of a new processed context with the distance to the nearest
instance (or neighbour) of each test instance already processed. Here, the distance of two contexts is
defined as the difference between the features within the instances.
Training and test instances consist of fixed-length vectors of symbolic n feature-value pairs (in
the study presented in this paper n=37 for verbs and for nouns), and a field containing the classification
of that particular feature-value vector. During classification an unseen example X, a test instance, is
presented to the system and a distance metric D between the instances of the memory Y and X is
calculated, D(X,Y). The algorithm tries to find the nearest neighbour and outputs its class, as prediction
for the class of the test instance, as well as the distance from the nearest neighbour in the training
instances.
4. Design and methodology
The methodology and design proposed in this report consists of an integrated approach to unify the
results of the software outlined previously, that manipulate the content of lexical and textual resources,
in order to aid the recognition of potential non-standard use of lexicalized words. Different tools are
adding various types of annotations on the data, such as grammatical and semantic. Such annotations
are means of giving to a text added value, in the sense that the added information can be used for a
multitude of purposes; cf. Leech (1997). The model for the interpretation of non-standrd use that is
presented is triggered by the use of a mixture of collocations and colligations, i.e. a collocation patterns
based on syntactic information rather than individual words. This type of syntactic patterning demands
349
extra information about the words in the corpus, information provided by the different software that
will be described later.
The method is inspired by the previously description of the work by Jim Cowie discussed in
Wilks et al. (1996:240) and the work by Kilgarriff (2000). In particular, I am interested in producing
different modified views of typical concordance lines (see figure 1) and then use a combination of
manual and automatic techniques for inspecting the obtained results. Moreover, when enough modified
concordance lines have been produced, inspected and marked, given a particular sense from the
available lexical resources, the overlap between old and new material can be measured, and aid a
human towards the identification of novel senses, according to the lexical resources used, here the
GLDB.
Figure 1. A more abstract representation of a concordance, using combination
of lemmata, features and labels of various types
Essentially, the following processing steps are considered:
 Gather a large number of sentences (or concordance lines or contexts) for every word to be
examined from a (newspaper) corpus
 Annotate these sentences with any possible type of information available (such as part-ofspeech,
sense and semantic information)
 Parse with a syntactic analyzer
 Normalize the information obtained by the different tools (lemmatisation, uppercase to
lowercase conversion, keeping the head of long chunks recognized during parsing, etc.)
 Decide the format for the MBL vectors
 Create the fixed-format vectors, by gathering the inform ation provided by the steps (2), (3),
and (4) above, and use them as training data
 Perform the same steps again, this time on sentences taken from another text genre and use
the result as test data
 Run MBL with the training and test data, and calculate the distance b etween the test and
training instances
 Inspect the results: zero distance? ( identical instances), small distance? (possibly
prototypical sense), large distance? (possibly, non-standard sense or processing errors)
By applying these steps, concordance lines can be transformed from raw text to a more abstract,
annotated representation, upon which MBL can calculate the overlap between the material, see the
worked example given in chapter (5).
4.1 Thresholds
Using MBL different threshold values are tested depending the provided algorithms. For instance, one
is using the normalized Information Gain, (i.e. Gain ratio), which is measured by computing the
difference in uncertainty (i.e. entropy) between the situations without and with knowledge of the value
of a feature. Another threshold is used when the algorithm tested is the nearest neighbor search (k-NN
or IB1), using weighted overlap and information gain weighting. Accordingly, the instances with the
350
highest threshold values produced by the TiMBL software are manually examined, (identical instances,
complete overlap, produce ‘0’ zero distance). Of course, the returned values are dependent on the
amount of training data, therefore, for each test performed the thresholds are adjusted in accordance
with the results.
5. A case study and results
The purpose of the experiment I conducted and describe in this chapter is twofold. First, to investigate
whether the proposed method is sufficient or not for the discovery of non-standard usage of words, and
second, to devise automatic methods for evaluating to what extent a dictionary accounts for whatever
we can find in corpora. The last point can be seen as a side-effect of the first, since both are closely
interrelated. In order to test the applicability of the proposed architecture, and methodology, I chose to
use as training material, data taken from the Swedish language bank (http://spraakbanken.gu.se/) which
consists, to a large extent, of newspaper material. As testing material I chose to use data downloaded
from the Internet, this time by searching on Swedish sites for sentences containing the words chosen
for my experiments. The reason I chose the test material from the Internet is because the corpora that
has been used for developing training data, and some of the tools that I will describe later, have also
been used by the lexicographers for the production of the static resources, particularly the GLDB.
However, there is nothing that excludes that there might be cases in the training material alone
where one can find non-standard sense or usage not used or ignored in the production and description
of the material in the lexicon. Similarly, we can speculate, and certainly anticipate, that the test data
may contain prototypical use of the words for the majority of the cases downloaded.
5.1 A worked example
In order to make the methodology described in chapter (4) clearer I will provide some examples
showing how different tools process few sample instances. The key word under investigation in the
small sample given, is the verb skyffla, which according to the GLDB has two senses, the first is similar
to ‘to shovel’ while the second is most similar to ‘to shove (away)'. All instances are taken from the
Swedish language bank, while the annotations provided are, in some cases, simplified. Four such lines,
with a very approximate interpretation, are:
(1) Socialdemokraterna försöker skyffla diskussionen om EMU under mattan.
‘The social democrats are trying to shovel the discussion about EMU under the rug.’
(2) Han försökte ocksa skyffla ansvaret för utvecklingen rörande Cypern pa EU.
‘He also tried to shove the responsibility regarding the development in Cyprus on EU.’
(3) Sakic skyfflade över pucken till Ozolinsh.
‘Sakic shovelled the puck to Ozolinsh.’
(4) Morfar skyfflade kol i fabriken i femtio ar.
‘Grand father has shovelled coal in the factory for fifty years.’
5.1.1 Tokenization, part-of-speech annotation & lemmatisation
Tokenization not only identifies graphic words (tokens) but also recognizes idioms, phrasal verbs and
multi-word expressions. The part-of-speech tags shown below are edited and simplified for the sake of
simplicity and readability. The original tagset is using a slightly modified version of the Swedish
PAROLE morphosyntactic description,
(http://spraakdata.gu.se/parole/lexikon/swedish.parole.lexikon.html).
Lemmatisation is based on the output from the previous tool, and it is implemented as a finite
state mechanism with three distinct states, one for verbs, one for nouns and one for adjectives. The
suffix of the content words, along with their rich morphosyntactic features returned by the tagger are
sufficient for establishing the base-form of each content word with very high accuracy.
(1a) Socialdemokraterna/NOUN försöker/AUX-VERB skyffla/VERB diskussionen/NOUN om/PREP
EMU/ABBREVIATION under_mattan/IDIOM ./PUNC
Socialdemokraterna[socialdemokrat] försöker[försöka] skyffla [skyffla] diskussionen[diskussion] om
EMU under_mattan .
(2a) Han/PRONOUN försökte/AUX-VERB ocksa/ADVERB skyffla/VERB ansvaret/NOUN för/PREP
utvecklingen/NOUN rörande/PREP Cypern/PROPER-NOUN pa/PREP EU/ABBREVIATION
./PUNC
351
Han försökte[försöka] ocksa skyffla[skyffla] ansvaret[ansvar] för utvecklingen[utveckling] rörande
Cypern pa EU.
(3a) Sakic/PROPER-NOUN skyfflade/VERB över/PARTICLE pucken/NOUN till/PREP
Ozolinsh/PROPER-NOUN ./PUNC
Sakic skyfflade[skyffla] över pucken[puck] till Ozolinsh .
(4a) Morfar/NOUN skyfflade/VERB kol/NOUN i/PREP fabriken/NOUN i/PREP femtio/NUMERAL
ar/NOUN ./PUNC
Morfar[morfar] skyfflade[skyffla] kol[kol] i fabriken[fabrik] i femtio ar[ar] .
5.1.2 NE-recognition and semantic annotation
The named-entity labels are obtained by the NE-recognition, these are underlined in the sample below.
The semantic annotation is following the SIMPLE model, colon ‘:’ designates the hyper-hyponym
relation in the semantic hierarchy, e.g. IDEO:HUMAN:CONCRETE means that the class IDEO “humans
identified according to an ideological criterion” is hyponym of the semantic class HUMAN, which in
turn is hyponym of the class CONCRETE. Note, that not all nouns get a semantic class annotation since
the entries in the SIMPLE lexicon (at the time this work was completed) did not account for more than
25,000 noun senses.
(1b) Socialdemokraterna/IDEO:HUMAN:CONCRETE försöker skyffla diskussionen/ABSTRACT om
EMU under_mattan .
(2b) Han/HUMAN försökte ocksa skyffla ansvaret/COGNITIVE-FACT:ENTITY: ABSTRACT för
utvecklingen rörande Cypern/LOCATION pa EU/AGENCY: ENTITY:ABSTRACT .
(3b) Sakic/HUMAN skyfflade över pucken/ARTIFACT:OBJECT:NON-LIVING:CONCRETE till
Ozolinsh/HUMAN .
(4b) Morfar/BIO:HUMAN:CONCRETE skyfflade kol/MATTER:NON-LIVING: CONCRETE I
fabriken/FUNCTIONAL-SPACE:LOCATION:NON-LIVING:CONCRETE i femtio ar/TIME .
5.1.3 Shallow parsing
Shallow parsing or chunking is based on the output from the part-of-speech tagging. During analysis,
only the lemmatised head of each chunk (noun, adverbial, and adjective phrases and verbal groups) is
preserved, as well as particles and the preposition heading a prepositional phrase. Here ‘NP’ is a noun
phrase, ‘VG’ a verbal group, ‘PP’ a prepositional phrase ‘RP’ temporal adverbial phrase. Chunking is
used for a couple of reasons; to obtain the head of the chunks, since phrases can be arbitrarily long and
complex, and to give a shorthand name to constituents lying at a certain distance on the left and right of
the word under investigation, e.g. different types of clauses.
(1c) NP[Socialdemokraterna/NOUN] VG[skyffla/VERB] NP[diskussionen/NOUN] PP[om/PREP
NP[EMU/ABBREVIATION]] IDIOM[under_mattan/IDIOM] ./PUNC
(2c) NP[Han/PRONOUN] VG[skyffla/VERB] NP[ansvaret/NOUN] PP[för/PREP
NP[utvecklingen/NOUN]] PP[rörande/PREP NP[Cypern/PROPER-NOUN]] PP[pa/PREP
NP[EU/ABBREVIATION]] ./PUNC
(3c) NP[Sakic/PROPER-NOUN] VG[skyfflade/VERB över/PARTICLE] NP[pucken /NOUN]
PP[till/PREP NP[Ozolinsh/PROPER-NOUN]] ./PUNC
(4c) NP[Morfar/NOUN] VG[skyfflade/VERB] NP[kol/NOUN] PP[i/PREP NP[fabriken/NOUN]]
RP[i/PREP NP[ar/NOUN]] ./PUNC
5.1.4 Sense annotation
Sense annotation is given for content words, nouns, main verbs and adjectives. The notation provided,
according to the GLDB, gives the lemma number followed by the lexeme or sense number. The
underlying model adopted in GLDB is the so called lemma-lexeme model described in Allén (1981).
The lemmas are grammatical paradigms, comprising formal data, e.g. technical stem, spelling
variations, part of speech, inflection(s), pronunciation(s), stress, morpheme division, compound
boundary, abbreviated form(s) and much more. The lexemes, are the numbered senses of a lemma, and
are divided into two main categories, a compulsory kernel sense and a non-compulsory set of one or
more sub-senses, called the cycles. A large number of metonymic uses of words are encoded under
separate lexemes or cycles for a lemma.
(1d) Socialdemokraterna:1/1 försöker skyffla:1/2 diskussionen:1/1 om EMU under_mattan.
(2d) Han försökte ocksa skyffla:1/2 ansvaret:1/1 för utvecklingen:1/1 rörande Cypern pa EU.
352
(3d) Sakic skyfflade:1/2 över pucken:1/1 till Ozolinsh.
(4d) Morfar:1/1 skyfflade:1/1 kol:1/2 i fabriken:1/1 i femtio ar:1/1.
5.1.5 Creating vectors
The vectors used by ML are of a fixed-format, and we have experimented with vectors of different size
and content. We think that for optimal results and for different part-of-speech categories we need to
model the content of the vectors differently. Therefore, the vectors for verbs consist of four ‘contexts’
to the left and four ‘contexts’ to the right of a verb under investigation (v1); while for nouns we model
less context, namely two chunks to the left and two to the right (v2). If the noun under investigation is
found in a prepositional phrase the preposition is also a part of the vector, finally the nearest to the
noun modifier (if any) is also used in the vector.
(v1) Key-Word Part-of-Speech Byte-Offs {Left-Context} Key-Word {Right-Context} Class
(v2) Key-Word Part-of-Speech Byte-Offs {Left-Chunk} Prep Modifier Key-Word {Right-Chunk}
Class
The left and right ‘contexts’ in (v1) and the chunks in (v2) are defined as clusters of four features of the
form:
TOKEN:MORPHOSYNTAX:SEMANTIC-TAG-or-NE:SENSE-TAG
With ‘MORPHOSYNTAX’ we mean a part of speech (if the context concerns a single token, which is
the head of a chunk, within the clause where the keyword appears) or a larger syntactic label (if the
context concern a syntactic unit outside the keyword's own clause, e.g. ‘CLAUSE'). With
‘SEMANTIC-TAG-or-NE’ is meant the result obtained by the semantic annotation and the NErecognition,
while with ‘SENSE-TAG’ is meant the result from the sense annotation (a GLDB lemma
and lexeme number). Any of the features, or even all, can be absent, in this case a missing feature is
marked with a ‘dummy’ character, an equal sign ‘='. The reason to it is that MBL requires that the
vectors are of equal size, one of the few disadvantages of the method in general.
Given the sample of the worked examples, the results are gathered in the format below, and
then converted to a fixed format of equal size for all vectors. Truncation is performed when a lot of
context is available. ‘BYTE-OFFS’ is simply the position of the key word in the discourse, a
mechanism that is inhereted by the tokenizer and helps linking the results with the original text from
where it was taken.
(1e) SKYFFLA VERB BYTE-OFFS = = = socialdemokrat:NOUN:HUMAN:1/1 skyffla:VERB:=:1/2
diskussion:NOUN:ABSTRACT:1/1 om:PREP:=:= EMU:ABBR:=:= under_mattan:IDIOM:=:=
(2e) SKYFFLA VERB BYTE-OFFS = = = Han:PRONOUN:HUMAN:= skyffla:VERB:=:1/2
ansvar:NOUN: ABSTRACT:1/1 för:PREP:=:= utveckling:NOUN:=:1/1 rörande:PREP:=:=
Cypern:PROPER-NOUN: LOCATION:= pa:PREP:=:= EU:PROPER-NOUN:AGENCY:=
(3e) SKYFFLA VERB BYTE-OFFS = = = Sakic:PROPER-NOUN:HUMAN:= skyffla: VERB:=:1/2
över:PREP:=:= puck:NOUN:ARTIFACT:1/1 till:PREP:=:= Ozolinsh:PROPER-NOUN:HUMAN:=
(4e) SKYFFLA VERB BYTE-OFFS = = = morfar:NOUN:HUMAN:1/1 skyffla:VERB:=:1/1
kol:NOUN: MATTER:1/2 i:PREP:=:= fabrik:NOUN:FUNCTIONAL-SPACE:1/1 i:PREP:=:=
ar:NOUN:TIME:1/1
5.2 Small scale evaluation
We carried out a small evaluation of the presented approach by producing vectors such as the ones
given in section 5.1.5, for a large sample of both training (newspaper articles) and testing material
(gathered from the Internet) for some verbs and a number of common nouns. More specifically, some
of the verbs we looked at were: skyffla ‘to shovel, to shove away’ (2 senses, 100 training contexts and
15 test contexts), publicera ‘to publish’ (1 sense, 300 training contexts and 20 test contexts) and the
phrasal verb hoppa in ‘to step in, to interfere’ (2 senses, 200 training contexts and 10 test contexts).
The results produced by MBL were sorted according the distance to the nearest instance in the training
353
sample, a distance calculated on the metrics gain ratio and information gain, using the IB1 algorithm.
The instances with the highest distance from the training material in every case were:
(5) ColorFusion-kortet skyfflar hela 9 MB videodata per sekund.
‘The ColorFusion-card shovels 9 megabyte video-data per second.’
(6) Framtidens taxibolag publicerar sina bilars positioner pa Internet.
‘The future's taxi companies publish their car's positions on the Internet.’
(7) Vid trafikavbrott kan den ena ringen hoppa in och ersätta den andra.
‘During interruption of traffic the one ring can interfere and replace the other.’
The characteristic for the test instances with the longest distance, in all three cases, has been the fact
that a concrete object (e.g. ‘card', ‘ring') is initiating an action usually performed by a human or
organization in the training material. While the longest distance for the instance of the verb ‘to publish’
has to do with publishing a ‘location’ or ‘position’ while the majority of the cases in the training
sample has been to publish a concrete object (e.g. ‘article', ‘report'). Over half of the test instances for
the verb ‘to publish’ had short distance to the training material, explicitly refering to prototypical
usage, while the opposite could be observed with the other two verbs.
Regarding the examined nouns, among others plattform ‘platform', we provide here a more
detailed picture of how the results look like. According to the GLDB, plattform has one main sense
(lexem) and four sub-senses (cycles). Roughly, these sub-senses are: ‘a’ platform (concrete), ‘b’ tram,
‘c’ oil-rig and ‘d’ platform (abstract); ‘d’ was the commonest sense in the training material and used as
default in all testing instances. Some of the results produced by the ML software, using 372 training
and 18 testing examples, are given below (9-12). The character and number on the right designates the
suggested sense for the key word and the distance of the example from the training instances. The
underlied features simply helps to distinguish which features belong together (see 5.1.5). Small number
(distance) designates that there are examples in the training material with many features in common
with the test. Examples (9a-12a) show the nearest instance found in the training examples. Figure (2)
shows the examples in concordance format.
Figure 2. Concordance lines before processing
(9) plattform N 00 dator N APPARATUS 1 bli V = = = = = = dominerande A = 1 PLATTFORM för S = = tillgang
N = 2 till S = = world_wide_web Y = = d d 2.306898
(9a) tag N VEHICLE 1 bli V = = = = = = = = = = PLATTFORM för S = = samtal N ABSTRACT 1 om S = =
utveckling N = =
(10) plattform N 00 = = = = = = = = i_och_med S = = öppen A = = PLATTFORM som = = = JavaCard = = = ha V
= = företag N AGENCY 1 d d 2.061991
(10a) = = = = = = = = = = = = = = = = PLATTFORM som = = = bära V = = = = = = system N = =
(11) plattform N 00 medlem N SITU = sluta_sig_samman V = 1 kring S = = politisk A = 1 PLATTFORM = = = =
= = = = = = = = = = = = d d 1.799718
(11a) folk N HUMAN 1 identifiera_sig V = 1 med S = = politisk A = 1 PLATTFORM = = = = = = = = = = = = =
= = =
(12) plattform N 00 Internet N = = vara V = = = = = = ny A = 2 PLATTFORM som = = = fa V = = support N = =
= = = = d d 1.074738
(12a) Internet N = = vara V = = = = = = ny A = 2 PLATTFORM = = = = = = = = = = = = = = = =
6. Conclusions and further research
This paper has outlined an approach to create a framework for aiding the identification of novel senses
of words in large corpora. We think that important variation of word-usage is ‘hidden’ and hard to
identify by merely looking at thousands of (non-processed) concordance lines. Therefore any means of
organising the bulk of the data is absolutely necessary for future enhancements of the dictionary
content with ‘new’ corpus-based material. Empirical preliminary results applied on a small sample of
words have shown that although a lot of noise is produced by the different modules of the system in
the form of errors in part-of-speech sense or semantic annotation sense differences between the
354
annotated concordance instances can be observed under the threshold conditions briefly outlined. The
noise caused by the reliability of some of the tools has been the main reason for the production of a
number of false positive instances. However, while the error rates encountered are high, and the
method immature for allowing fully-automatic integration of new data in the lexicon, our method can
be used as a supporting tool that can aid the lexicographers identify new usage. Actually, we never
intended to provide fully automatic means, since the manual refinement, inspection and judgement will
always be the crucial factor left for the lexicographer; that is to make a final decision regarding what
will be included and what will be left out from a dictionary.
The design presented in this paper is dictionary-dependent but the method is not specific to
Swedish, as long as there are tools that can contribute with various types of morphological, syntactic,
lexical semantic or other information to corpus data. Although our experiments are limited in
magnitude, the results showed that the coverage of GLDB is pretty good. The future direction for the
work presented will operate on a larger sample of the language and try to define more rigorous
evaluation criteria.
References
Allén S. 1981 The Lemma-Lexeme Model of the Swedish Lexical Database. In Rieger B. (ed)
Empirical Semantics. Bochum pp. 376–387
Atkins B.T. 1987 Semantic ID Tags: Corpus Evidence for Dictionary Senses. In Proceedings of the 3rd
OED. Waterloo Canada
Clear J. 1994 I Can't See the Sense in a Large Corpus. In Kiefer F. Kiss G. and Pajzs J. (eds.). Papers
in Computational Lexicography COMPLEX ’94. Budapest pp. 33–45
Daelemans W. Zavrel J. van der Sloot K. 1999 TiMBL: Tilburg Memory Based Learner version 2. ILK
Technical Report 99-01. Paper available from http://ilk.kub.nl/~ilk/papers/ilk9901.ps.gz
Dorr B. and Jones D. 1996 Role of Word Sense Disambiguation in Lexical Acquisition: Predicting
Semantics from Syntactic Cues. In Proceedings of the 16th COLING. Vol. 1. Copenhagen
Denmark pp. 322-327
Hanks P. 1996 Contextual Dependency and Lexical Sets. Journal of Corpus Linguistics. Benjamins
1(1): 75–98
Kilgarrif A. 2000 Generative Lexicon Meets Corpus Data: the Case of Non-Standard Word Uses. In
Bouillon P. Busa F. (eds) Word Meaning and Creativity. Cambridge UP
Kilgarriff A. and Palmer M. 2000 Introduction to the Special Issue on SENSEVAL. International
Journal of Computer and the Humanities. Special Issue on SENSEVAL. 00:1-13 Kluwer Academic
Publishers
Kokkinakis D. 1998 AVENTINUS GATE and Swedish Lingware. In Proceedings of the 11th
NODALIDA Conference (Nordisk Datalingvistik). Copenhagen Denmark pp. 22–33
Kokkinakis D. and Johansson-Kokkinakis S. 1999a Sense Tagging at the Cycle-Level Using GLDB. In
Proceedings of the NFL Symposium (Nordic Association of Lexicography). Gothenburg Sweden.
Paper available from: http://svenska.gu.se/~svedk/publics/nfl.pdf
Kokkinakis D. and Johansson-Kokkinakis S. 1999b A Cascaded Finite-State Parser for Syntactic
Analysis of Swedish. In Proceedings of the 9th EACL. Bergen Norway. Paper available from:
http://svenska.gu.se/~svedk/publics/eaclKokk.ps
Kokkinakis D. Toporowska-Gronostaj M. and Warmenius K. 2000 Annotating Disambiguating &
Automatically Extending the Coverage of the Swedish SIMPLE Lexicon. In Proceedings of the 2nd
LREC. Athens, Hellas (2000)
Krovetz R. 1994 Learning to Augment a Machine-Readable Dictionary. In Proceedings of the
EURALEX '94. Amsterdam Holland pp. 107–116
Leacock C. Towell G. and Voorhees E.M. 1996 Towards Buidling Contextual Representations of Word
Senses Using Statistical Models. Boguraev B. Pustejovsky J. (eds.): Corpus Processing for Lexical
Acquisition. Bradford pp. 98–113
Leech G. (1997) Introducing Corpus Annotation In Corpus Annotation. Linguistic Information from
Computer Text Corpora pp. 1-18 Longman
Lenci et al. 1998 SIMPLE WP2 Linguistic Specifications Deliverable 2.1 Pisa
Levin B. 1993 English Verb Classes and Alternations: a Preliminary Investigation. UCP
Malmgren S.G. 1992 From Svenska ordbok (‘A dictionary of Swedish') to
Nationalencyklopediensordbok (‘The Dictionary of the National Encyclopedia'). In Tommola H.
Varantola K. Salmi-Tolonen T. Schopp J. (eds). In Proceedings of the EURALEX ’92 Vol. 2.
Tampere Finland pp. 485–491
355
Miller G.A. (ed.) 1990 WordNet: An on-line Lexical Database. International Journal of Lexicography
Special Issue 3(4)
Mitchell T. M. 1997 Machine Learning. McGraw-Hill Series on Computer Science
Renouf A. 1993 A Word in Time: First Findings from the Investigation of Dynamic Text. Aarts J. de
Haan P. Oostdijk N. (eds.) English Language Corpora: Design Analysis and Exploitation. Rodopi
Tapanainen P. and Järvinen T. (1998) Dependency Concordances. Journal of Lexicography. OUP
11(3): 187–203
Wilks Y. 1980 Frames Semantics and Novelty. In Metzing D. (ed) Frame Conceptions and Text
Understanding. de Gruyter pp. 134–163
Wilks Y. Slator B. and Guthrie L. 1996 Electric Words Dictionaries Computers and Meanings. MIT
Yarowsky D. 1995 Unsupervised Word Sense Disambiguation Rivaling Supervised Methods. In
Proceedings of the 33rd ACL. Cambridge MA pp. 189–196

356
Using bilingual corpora for the construction of contrastive generation grammars:
issues and problems
Julia Lavid
Dep. English Philology I
Universidad Complutense de Madrid
28040 Madrid (Spain)
Phone and fax: +34-91-518-5799
e-mail: julavid@filol.ucm.es
This paper reports on the use of corpora for the construction of a computational grammar of Spanish,
contrastive with English, in the application context of Multilingual Natural Language Generation (MLG).
The theoretical framework for this work is Systemic Functional Linguistics (SFL) and the computational
context provided by KPML (Komet Penman Multilingual), an extensive grammar development
environment and generation engine that supports large-scale multilingual development (Bateman 1997).
The initial phenomena which are being investigated contrastively belong to three different functional
regions of the grammar, i.e., particular subareas of the grammar that are concerned with particular areas of
meanings. These regions are transitivity (ideational meaning), thematicity (textual meaning) and mood
(interpersonal meaning). The present study concentrates on textual meaning (thematicity and focus) as an
illustration.
Following what has now established itself as a standard methodology for empirically-based
Natural Language Generation (Bateman 1998a, Reiter and Dale 1997), the following steps were carried out:
first, a bilingual corpus (English-Spanish) was selected. For Spanish, a sample of spoken texts from the
MacroCorpus of the educated linguistic standard of the main cities of the Spanish-speaking world was used,
while for English a comparable sample was selected from the British National Corpus Sampler. This was
motivated by the need to provide a realistic account of the behaviour of the linguistic phenomena
investigated in unplanned and spontaneous contexts of use. The second step was to carry out a contrastive
analysis of the phenomena mentioned before. Finally, the results of the analysis were coded up as
resources/processes for generation. In the case of Spanish, these had to be created anew. In the case of
English, as the KPML already includes an English generation grammar, this last step consisted on checking
the coverage of the existing specifications and extending them when could not cover the instances found in
the corpus, and adapting them for effective MLG.
Given the nature of the NLG process, which typically converts communicative goals expressed in
some internal representation into surface forms, the kind of information that is most readily usable for NLG
are statements of mappings from functions to forms. Therefore, the corpus analysis phase for NLG usually
includes an explicit, and usually quite lengthy linguistic analysis where the analyst seeks possible
realisations of communicative functions, which restricts the size of the corpus that can be realistically
considered.
This paper describes the different steps carried out for the generation of the linguistic phenomena
mentioned above, discussing the problems encountered during the corpus analysis phase, and the
computational representation derived from it, as well as some of the decisions taken to overcome them.
357
1. Introduction
Natural Language Generation (NLG), the subfield of Artificial Intelligence and Computational Linguistics
that investigates the automated production of texts by machine, is typically a process that converts
communicative goals expressed in some internal representation into surface forms. As such, it touches upon
different linguistic areas of inquiry such as text planning and discourse organisation, lexical semantics,
grammatical and lexical choice, and the relationship between all of these. In fact, one of the main concerns
of NLG is the construction of of computational accounts of the linguistic system capable of generating texts
in one language (monolingual NLG) or in several (multilingual NLG, henceforth MLG). This theoretical
task poses unusual demands on the linguist who has to provide explicit and details accounts of how
language works and confront the results of the application of his/her theoretical claims in concrete
computational systems. A more practical concern is the creation of computational systems capable of
producing acceptable text from various sources and for different types of applications: well-known
examples include the generation of weather reports from meteorological data (Kittredge et al. 1986), the
generation of letters responding to customers (Springer et al. 1991), and other systems applied in areas such
as technical documentation and instructional texts (Not and Stock 1994, Paris et al. 1995, Lavid 1995),
patent claims (Sheremetyeva et al. 1996), information systems (Bateman and Teich 1995), computersupported
co-operative work (Levine and Mellish 1994), patient health information and education
(DiMarco et al. 1995), and medical reports (Li et al. 1986), to mention a few.1
While the first generation systems were limited to the random generation of grammatically correct
sentences, the field has experienced a very rapid growth over the past ten years, both as a research area
bringing a unique perspective on fundamental issues in artificial intelligence, cognitive science, and
human-computer interaction, and as an emerging technology capable of partially automating routine
document creation and playing an important role in human-computer interfaces. In this sense, as practical
systems became more sophisticated, it was necessary to provide a good understanding of the notion of
“textuality” and all the factors involved in the creation of different text types. In this context, it was only
natural that corpora started to be used as the empirical basis both for the theoretical investigation of textual
phenomena, and as part of the requirement analysis phase for NLG systems. As a result, the use of corpora
has now been integrated as part of the standard methodology for NLG, both in theoretically-oriented
research and in the development of concrete generation systems, to such an extent that computational tools
have been developed to support, among other functionalities, the analysis of machine-readable monolingual
or multilingual corpora for NLG (Alexa and Rostek 1997).
In this paper, we report on the use of bilingual corpora for the construction of a computational
grammar of Spanish, contrastive with English, in the application context of Multilingual Natural Language
Generation (MLG), concentrating on the textual phenomenon of thematicity and its relationship with the
related notion of focus as an illustration. Section 2 describes the two corpora selected and the criteria for
concentrating on two comparable samples as the empirical basis for computational specifications. Section
3 presents the theoretical framework selected for this study and the issues which must be explored
contrastively with respect to the textual phenomena mentioned before. Section 4 describes the corpus
analysis phase of the bilingual samples. Section 5 presents a computational specification of the results of
the analysis as resources for generation. The specification is based on the notion of functional typology as
developed in SFL and implemented in the KPML development environment. Finally, section 6 provides a
summary and discusses the implications of these results for corpus-based MLG.
2. Bilingual corpora for NLG
Two electronic corpora were initially targeted as the empirical basis for the contrastive work proposed in
1 For an extensively documented review of the field see Bateman (1998b)
358
unplanned and spontaneous speech, which are necessary to provide a realistic account of the behaviour of
the linguistic phenomena investigated in their contexts of use. In the case of NLG, it is frequent to find
computational specifications of linguistic phenomena based on the analysis of specific text types, most of
them written monologues, which, though useful for the generation of texts of a specific domain, can only
provide a partial view on the phenomena studied. As the purpose of this study is to investigate the
behaviour of textual phenomena for a multipurpose contrastive grammar of English and Spanish, the choice
of samples from unplanned, spontaneous speech offered a rich and unexplored empirical basis for the
study of the textual phenomena mentioned before.
For the study of Spanish, a sample of spoken texts from the Spanish Macrocorpus was selected.
This Macrocorpus includes the transliteration of 84 hours of recording by 168 native speakers
representative of the educated speech of twelve Hispanic cities, nine of them from SouthAmerica and three
of them from the Iberian Peninsula. The recordings are basically unstructured interviews conducted in a
conversational style where the interviewer introduces a few questions and topics to stimulate the
conversation and to establish some uniformity in the topics discussed by the speakers. In general, the
speakers were left free to talk about the topics suggested by the interviewer and to introduce any new
topics. The sample selected for this study consists of 10 interviews from the three cities of the Iberian
Peninsula contained in the Macrocorpus, i.e. from Madrid, Seville and Las Palmas de Gran Canaria.
For the study of English, a comparable sample was selected from the BNC Sampler Corpus, a
subcorpus of the British National Corpus, consisting of approximately one-fiftieth of the whole corpus, viz.
two million words. The Sampler Corpus consists of 184 different texts, comprising just under 127,000
sentences and two million words. From this sampler corpus, a subcorpus of spoken texts was chosen for the
purposes of the present study, consisting of 10 conversations.
3. The textual resources of thematicity and focus in English and Spanish
The linguistic phenomena selected for illustrating the use of corpora in NLG belong to the textual region of
the grammar, and have been the subject of several functional accounts. The theoretical framework for the
study of these phenomena is SFL, as this is the theoretical basis for the implementation of the current
English generation grammar Nigel. In SFL, the textual clause grammar is composed of two complementary
systems, the systems of Theme and Information, characterised as assigning two different kinds of textual
prominence to elements of the clause: thematic prominence, in the case of the system of Theme, and
prominence as news, in the case of the system of Information. The notion of Focus, however, has not
received a comparable attention within SFL, but has been the subject of a number of discourse-oriented
linguistic and computational studies (Lavid 2000, McCoy and Cheng, McKeown, among others), and has
been treated in Dik´s Functional Grammar (Dik 1978) as a pragmatic function which assigns more salience
to some clausal constituent with respect to the contextual (pragmatic) information between language
producers and receivers.
In view of the need to provide form-function mappings of these phenomena which can be used by
a MLG system, a corpus-based analysis for generation purposes must explore at least the following issues
contrastively:
1. Do both English and Spanish grammars have a Theme and a Focus function ?
2. How do English and Spanish realise the Theme and the Focus functions - e.g.: sequence, inflection,
adposition, intonation?
3. Are there marked and unmarked Theme and Focus selections in both languages, and to what extent do
they depend on other systems (e.g. Mood, Voice)?
4. Are there resources in English and Spanish to combine thematicity and focus?
With respect to the first question, different linguistic studies acknowledge the existence of these two
functions in different languages (see Caffarel et al. in preparation; Dik et al. 1981), so this issue will not be
further explored here. The rest of the issues, however, are central for a functional characterisation of the
textual phenomena selected for this study, and their corpus-based empirical study is the basis for the
specification of resources required for MLG. Therefore, the next section will describe the corpus analysis
carried out for this study and the problems encountered when attempting to investigate the issues
mentioned above.
359
4. Contrastive corpus analysis for MLG: problems and decisions
As in the area of discourse analysis, the type of information that NLG needs from corpus investigations is
one which basically consists of statements of mappings from functions to forms. Therefore, when
investigating specific discourse phenomena, a NLG system requires as a first step a specification of the
mappings from functions to forms with the purpose of duplicating the text analysed. In this sense, the
corpus analysis phase raises some problematic issues which must be considered by NLG practitioners. One
of these issues is the size of the corpus: if, at is usually the case, the analyst has to intervene looking for
possible realisations of communicative functions, the use of large corpora becomes impractical. Also, it is
not possible to mark-up large quantities of texts according to functional categories if they cannot be
recognised automatically on the basis of tagged texts or syntactic analysis.
In view of these problems, the following decisions had to be taken to investigate the issues
mentioned above:
1.- With respect to the first issue, i.e., the ways in which English and Spanish realise the Theme and the
Focus functions, the corpus analysis was based on the following assumptions, based on previous linguistic
studies:
a) The Theme function was recognised as realised by clause initial position in both languages, as several
linguistic studies have demonstrated (Lavid in press; Taboada 2000).
b) The Focus function was recognised on the basis of non-prosodic realisations, such as word order
patterns, focus markers and characteristic constructions, since the BNC corpus does not include prosodic
annotation. 2
2.- With respect to the second issue, i.e., the existence and realisations of marked and unmarked theme and
focus selections in both languages, the corpus analysis was based on the following assumptions:
a) marked and unmarked themes were recognised in specific mood contexts, such as declarative,
interrogative and imperative options. Absolute themes were recognised by the presence of a pause and a
comma separating them from the rest of the predication.
b) unmarked focus was assumed to coincide with the last lexical element of the clause.3
3.- With respect to the third issue, i.e., the existence and variation of resources which combine thematicity
and focus in both languages, the corpus analysis concentrated on the so-called cleft and pseudo-cleft
sentences as realisation strategies used by both English and Spanish to combine the Theme and the Focus
functions. These were semi-automatically distinguished in both corpora on the basis of their characteristic
form.
5. Towards a computational specification for MLG
The second step in what has now established itself as a standard methodology for empirically-based NLG is
the codification of the corpus analysis results as resources/processes for generation. The current
implementation of the English generation grammar contained in the KPML development environment
already includes a computational specification for the Theme function in English as a textual region of the
grammar. However, as will be shown in the following sections, it became necessary to modify the existing
specification to account for instances found in the corpus and to ensure maximal sharing of resources for
contrastive generation. For Spanish a new specification was created on the basis of the contrastive corpus
analysis. It should be noted, however, that it is not the purpose of this study to provide a full specification
of the behaviour of the textual phenomena of thematicity and focus in both languages, but rather to discuss
2 Considering that Focus in English is predominantly realised by marked prosodic prominence (Mart nez-Caro 1999), this apparent
limitation of the corpus-analysis phase must be overcome by explicitly representing this realisation in the computational specification
for English.
3 According to Halliday (1994), unmarked focus falls on the last lexical element of the tonic group, which in unmarked circumstances
coincides with the clause.
360
and illustrate with some examples some problems and issues raised by corpus-based contrastive MLG
specifications. The following sections, therefore, will illustrate some of these problems and the solutions
suggested in the context of a MLG architecture based on functional typology.
5.1. Thematic resources for English and Spanish: a functional-typological characterisation
This section presents a partial specification of the thematic resources available in English and Spanish on
the basis of the contrastive corpus-based analysis. More specifically, the area of theme markedness will be
discussed in detail, as there exist some interesting commonalities and differences which must be accounted
for when generating English-Spanish thematic variants. The approach rests on the notion of functional
typologies, as pursued in systemic-functional linguistics (cf. Halliday 1978), later developed for MLG
purposes (see Bateman et al. 1999). Functional typologies are constructed by means of classification
hierarchies called system networks (Halliday 1966), where each disjunction, or grammatical system is seen
as a point of abstract functional choice, capturing those minimal points of alternation offered by a linguistic
representation level. This representation of the so-called paradigmatic axis of linguistic description is
complemented with the corresponding syntagmatic realisation, i.e. the structural expression of the choices
made in the paradigmatic axis and associated with individual grammatical features. This task is carried out
by the so-called realisation statements which set allowable constraints on configurations of syntactic
constituents such as linear precedence, immediate dominance, or "unification" of functional constituents.
This type of approach, where the functional coherence of the paradigmatic organisation is preferred over
generalisations concerning possible structures, has been found to provide effective multilingual linguistic
descriptions and computational representations which maximise the factoring out of generalisations across
languages (cf Bateman et al. 1991, 1999).
Therefore, following this approach, and on the basis of empirical corpus-analysis, a specification
was created to account for Theme markedness in Spanish, as shown in Figure 1 below:
theme matter
marked absolute
not theme matter
predicated
Theme
Markedness non-absolute
Predication
non-predicated
unmarked (transitivity role)
Figure 1: Spanish Theme markedness systems
As the system network shows, the primary distinction in Spanish is between marked and unmarked themes,
and within marked themes between absolute or non-absolute themes. Absolute themes do not map onto any
transitivity or interpersonal functions, and are normally separated from the transitivity /interpersonal
structure of the clause by a comma in writing and by a pause in speech. They can be of two types:
representing a theme matter or not. If non-absolute, they can be predicated or non-predicated. When
predicated, they are normally realised by pseudo-cleft sentences, as will discussed in detail below. If they
are not predicated, thematic status may be assigned to any role within the transitivity structure of the clause
(participants, processes or circumstances).
With respect to English, the existing specification was found not to account for all the language
instances found in the corpus. Therefore, on the basis of the corpus-based analysis and to provide a
maximally effective specification for the purposes of MLG, the following specification was created for
English:
361
theme matter
marked absolute
not theme matter *
predicated
unmarked local
Markedness non-absolute Local theme
selection marked local
non-predicated
unmarked (transitivity role)
substitute
Theme
Substitution
non-substitute
Figure 2. English Theme markedness systems
As can be seen, an attempt has been made at capturing the comnonalities with the Spanish
specification while maintaining the English-specific integrity and difference. Thus, while English shares
with Spanish the primary distinction between marked and unmarked themes, previously existing
specifications (see Matthiessen 1995: 540) did not include a distinction between absolute and non-absolute
themes. However, in order to enforce maximal sharing of resources across languages, this system was also
included in the English network. In this sense, it should be noted that up to this point both languages are
similar, but differ as the features become more delicate: while absolute themes in Spanish may refer to a
theme matter or to any other transitivity role coreferential with a constituent within the predication, English
has a gap in this common paradigm (indicated by an asterisk in the English network) since it only has the
option of a theme matter as absolute theme. Also, English includes more delicate options within the
systems of Theme predication, as it includes the possibility of having a local theme selection, and includes
a system called Theme substitution, which is not available in the Spanish paradigm. In this new
specification, predicated themes have been considered as marked in the sense that they map the Focus
constituent, and, in many instances, the New element, onto the Theme (see Halliday 1994: 59 on predicated
Themes). Contrastive examples extracted from the corpora are discussed in detail below.
With respect to the syntagmatic realisations of the options presented in the networks above, both
languages present divergent grammaticalisations of their common paradigmatic potential. Table 1 below
illustrates the different features of the systems of both languages, together with their divergent realisations.
These are specified as sets of constraints which specify the syntactic and lexical properties of the linguistic
units being generated. For example, the feature ’theme substitute’ has the realisation statement +Pro-theme,
which means ’insert a Pro-theme’ and ’Pro-theme / Subject’ which means ’conflate or unify with the
grammatical function Subject'. The rest of the realisation specifies the ordering of constituents in relation to
one another: thus ’Rheme ^ Theme’ means order the Rheme before the Theme.
362
feature English realisation Spanish realisation
System: Markedness
marked
absolute
theme matter ’As to/as for/... + NG, + Pred. ’en cuanto a + NG', + Pred.
not theme matter Left Dislocation + Predication
non-absolute
predicated ’It + be + X..that/who’ Relative ^ Copula ^ NP
NP ^ Copula ^ Relative
Copula ^ NP ^ Relative
unmarked local
marked local ’NP it + be+ that’
non-predicated Part, Pro. or Circ. non-Subj. Part, Pro. or Circ. non-Subj.
unmarked
non-substitute
substitute +Pro-theme/ Subject ^ Rheme
^Theme
Table 1: Syntagmatic realisations of theme markedness options in English and Spanish
Some of the divergent realisations in both languages are the following:
1. Absolute Themes
Absolute themes refer to constituents which are not integrated within the transitivity/interpersonal structure
of the clause, and are, therefore, marked in speech with a pause, and with a comma in writing. As shown in
Figure 1 above, Spanish has two options when choosing absolute themes: as theme matter and as not-theme
matter. Example (1) below illustrates the choice of a theme matter in Spanish, an instance extracted from
the Spanish Macrocorpus:
(1) En cuanto a los libros que me gusta leer, t~ sabes que a m me gusta leer todo (SE-
10)
As for the books that me likes to read, you know that me likes to read everything
’As for the books that I enjoy reading, you know that I enjoy reading everything’
A contrastive example in English is illustrated by example (2) below, extracted from the BNC sampler
corpus:
(2) As for the past, I have adopted the doctrine of anamnesis (AEA)
When the absolute theme is not a theme matter, in Spanish it may be coreferential with an element of the
structure of the clause, as example (3) illustrates:
And the citizen, the Madrid Madrid (one), to him notice (3-sing) one change
’ And the citizen, the Madrid citizen, can one notice a change in him...?’
In (3), the constituent ’el ciudadano’ is not integrated in the transitivity structure of the clause, though it is
coreferential with an element functioning as beneficiary in the clause, expressed by the clitic ’le’ (italics in
the example).
English, however, lacks this option in its paradigm, and this is indicated by an asterisk in the
system network in Figure 2.
2. Predicated Themes
363
Both English and Spanish include predicated themes as a choice between presenting a Theme with a special
feature of identification or without that special feature. In fact, this choice is a textual strategy which
combines the Theme and the Focus functions into one single realisation, which, depending on the language,
may be a cleft, a pseudo-cleft, an identifying clause or a combination of the three. Thus, English typically
uses clefts with the form ('it + be + X + that ...) to realise predicated themes (italics) and to mark focus
constituents (boldface) as in examples (4) and (5) extracted from the BNC sample:
(4) It was only a relatively small Arab army that arrived in Egypt (JXL)
(5) It is not the death of the body that is important to us, it ’s the soul (NHE3)
Textually speaking, this construction serves two textual functions:
a) to identify the Theme in the transitivity role it serves in the clause: by using the identifying type
of clause we are achieving a textual distribution of meaning with the added feature of identification. For
example, in (4) ’the thing that arrived in Egypt’ is identified as ’a relatively small Arab army'. Similarly, in
(5) the ’the thing that is important to us’ is identified as not the death of the body but as the soul. According
to this, the segment ’it's not the death of the body’ would be the Theme of the clause, and ’that is important
to us’ would be the Rheme.
b) to define a specific constituent as having the Focus function, which, in many cases, is presented
as contrastive to another alternative. Example (5) is a case of what Dik would call counterpresuppositional
Focus, more specifically Substitute Focus. According to his definition, "the information presented is
opposed to other, similar information which the Speaker presupposes to be entertained by the Addressee"
(Dik 1989: 282). In (5), the speaker rejects the information which he/she presupposes to be entertained by
the Addressee (the death of the body) and corrects it with information which the speaker considers to be
correct (the soul).
English also uses pseudo-cleft and identifying clauses to combine the Theme and Focus functions.
Example (6) below illustrates a pseudo-cleft construction where the Theme is realised by a nominalising
clause introduced by ’what', and the rest of the clause serves as a Rheme. This construction also serves to
mark the focus constituent as the one following the copular verb:
(6) what we were able to do on one occasion was er to raise enough money (FYJ)
Spanish, by contrast, tends to use only pseudo-cleft clauses to combine the Theme and Focus functions.
The range of possible syntagmatic realisations in Spanish is more varied than in English, and includes three
main types:
1. Constructions where the relative clause appears in initial position, followed by the copular verb
and the nominal group functioning as an attribute. Exampe (7), extracted from the Spanish
Macrocorpus, illustrates this first type:
(7) Lo que yo necesito [...] es un poco m s del estilo rabe... (SE-01)
What I need is a bit more of the style Arabian
’What I need is a bit more of the Arabian style’
2. Constructions where the nominal group is in initial position , the verb in medial position and the
relative clause at the end. Example (8) illustrates this type:
(8) Eso es lo que pienso de la familia (SE-04)
That is what think (1-sing.) of the family
’That is what I think of the family’
3. Constructions with the verb in initial position followed by the nominal group and the relative
clause. Example (9) illustrates this type:
(9) pero ha sido precisamente el avance de la anestesia [...] la que ha podido hacer que se [...]
pudiera hacer una serie de operaciones (SE-05)
364
But was (3-sing) precisely the advance of anaesthesia that could allow [...] that could be
made certain operations
’But it was precisely the advance of anaesthesia what allowed certain operations to be done’
3. Local Theme selection
In English when theme predication is selected, and it is realised by means of a construction of the type it +
be + ...[[that ...]], that clause opens up the potential for an additional thematic contrast - a contrast local to
that clause. In this case, the Theme local to the identifying clause may be unmarked (as it is in most of the
cases) or marked. If it is marked, it will coincide with the Complement / Identifier of the identifying clause
and will be given thematic prominence in initial position of the clause. No examples have been found in the
BNC sample, probably due to the fact that it is a small one. However, as other studies have proved its
existence with empirical evidence, we include the example provided by Matthiessen as an illustration:
(10) There we fell and my leg it was that broke (Matthiessen 1995: 567)
4. Substitute Theme
In English, when the Theme is unmarked and not predicated, there is a choice whether to re-introduce the
Theme/Subject at the end of the clause so as to give it a "thematic culmination". With substitute Theme, the
Theme/Subject is a pronominal nominal group and the substitute Theme is typically a full lexical nominal
group, though it is also possible to find examples with this, one, etc. In examples (11) and (12) below, the
Theme/Subjects are realised by the demonstrative ’that', and the lexical items ’piano’ and ’day release’ are
the substitute Themes, which serve as a textual reprise of the thematic referents:
(11) That ’s good, that ’s good for you, the piano you just had there (kp8)
(12) That ’s what you want to be after, day release.
6. Summary and concluding remarks
As the contrastive examples above have illustrated, a detailed corpus analysis in search for form-function
mappings of linguistic phenomena is an indispensable step as a basis for empirically-based computational
specifications for MLG. The study of the phenomenon of thematicity and its relationship with focus in both
languages, though partial and purely illustrative, has served to show how a functional analysis of selected
corpus samples may shed light on the paradigmatic features and their syntagmatic realisations of those
phenomena, which are the basis for computational specifications for MLG architectures based on functional
typologies.
However, the requirements of NLG, similar in many respects to the goals of discourse analysis,
raise important problems for corpus analysis: the need for semantic and contextual (pragmatic) analysis of
the selected corpora, which cannot always be recognised automatically and so cannot be extracted from
large-scale corpora without an analyst's intervention, makes, in most cases, the use of large corpora and
form-based corpus tools unsuitable for NLG purposes. Given the close relation between discourse analysis
and NLG, it can be hoped that new tools and specialised environments are developed to extract more from
current electronic corpora than what can actually be obtained from form-based quantitative corpus analysis
tools.
References
Alexa, M and Rostek, L 1996 Computer-assisted corpus-based text analysis with TATOE,
Technical Report, German National Research Center for Information Technology, Institute for Integrated
Information and Publication Systems, Darmstadt, Germany.
Bateman, J 1997 Enabling technology for multilingual natural language generation: the KPML
development environment. Journal of Natural Language Engineering 3 (1): 1-41.
365
Bateman, J 1998b Automated discourse generation. In Kent A and Hall, C.M (eds), Encyclopedia
of Library and Information Science, Vol 62, New York, Marcel Dekker, pp 1-54.
Bateman J, Teich E 1995 Selective information presentation in an integrated publication system:
an application of genre-driven text generation. Information Processing and Management 31(5): 753-767.
Bateman J, Matthiessen C, Nanri K , Zeng L 1991 The Re-use of linguistic resources across
languages in multilingual generation components. In Proceedings of the 12th International Joint
Conference on Artificial Intelligence, Sydney, pp 966-971.
Bateman J, Matthiessen C, and Zeng, L 1999 Multilingual Natural Language Generation for
Multilingual Software: a Functional Linguistic Approach. Applied Artificial Intelligence 13: 607-639.
Caffarel A, Martin J R, and Matthiessen C in preparation Language typology: a functional
perspective.
Dik S C 1978 Functional Grammar. Dordrecht, Foris.
Dik S C, Hoffmann M, de Jong JR, Djiang S, Stroomer H, and de Vries L 1981 On the typology of
focus phenomena. In Hoekstra et al. (eds) Perspectives on functional grammar. Dordrecht, Foris.
DiMarco C, Hirst G, Wanner L, Wilkinson J 1995 Healthdoc: customizing patient information and
health education by medical condition and personal characteristics. In Proceedings of the Workshop on
Patient Education, Glasgow.
Halliday M A K 1966 Some notes on "deep grammar". Journal of Linguistics 2 (1): 57-67.
Halliday M A K 1978 Language as social semiotic. London, Edward Arnold.
Halliday M A K 1994 An introduction to functional grammar. London, Edward Arnold.
Kittredge R, Polguere A, Goldberg E 1986 Synthesizing weather reports from formatted data. In
Proceedings of the 11th International Conference on Computational Linguistics, Bonn, Germany, pp 563-
565.
Lavid J 1995 From interpersonal option to thematic realisation in multilingual instructions. In
Kittredge R (ed) Working notes of the IJCAI-95 workshop on text generation. Montreal.
Lavid J 2000 Theme, focus, given and other dangerous things: linguistic and computational
approaches to information in discourse. Revista Canaria de Estudios Ingleses :
Lavid J in press La noci~n gramatical de tema en un contexto multilingüe: una perspectiva
Levine J, Mellish C 1994 Corect: Combining CSCW with natural language generation for
collaborative requirements capture. In Proceedings of International Joint Conference on Artificial
Intelligence, Montreal, Canada, pp 1398-1404.
Li P, Evens M, Hier D 1986 Generating medical case reports with the linguistic string parser. In
Proceedings of the 5th National Conference on Artificial Intelligence, Philadelphia, pp 1069-1073.
McCoy K, Cheng J 1990. Focus of attention: constraining what can be said next. In Paris et al.
(eds) Natural language generation in artificial intelligence and computational linguistics. Kluwer,
Dordrecht, 1990.
McKeown, K 1995 Text generation: Using discourse strategies and focus constraints to generate
natural language text. Cambridge, Cambridge University Press.
Matthiessen C 1995 Lexicogrammatical cartography: english systems. Tokyo, International
Language Sciences Publishers.
Not E, Stock, O 1994 Automatic generation of instructions for citizens in a multilingual
community. In Proceedings of the European Language Engineering Convention, Paris.
Paris C, Van der Linden K, Fisher M, Hartley A, Pemberton L, Power R, Scott D 1995 A support
tool for writing multilingual instructions. In Proceedings of the International Joint Conference on Artificial
Intelligence, Montreal, pp1398-1404.
Reiter E, Dale R 1997 Building applied natural language generation. Journal of Natural Language
Engineering 3(1): 57-87.
Sheremetyeva S, Nirenburg S, Nirenburg I 1996 Generating patent claims from interactive input.
In Proceedings of the 8th International Workshop on Natural Language Generation, Herstmonceaux,
England, pp 61-70.
366
Springer S, Buta P, Wolf T 1991 Automatic letter composition for customer service. In Smith R,
Scott C (eds) Innovative applications of artificial intelligence 3. Menlo Park, Ca AAAI Press.
Taboada M 2000 Collaborating through talk: the interactive construction of task-oriented
dialogue in English and Spanish. Unpublished PhD thesis, Universidad Complutense de Madrid.

356
Using bilingual corpora for the construction of contrastive generation grammars:
issues and problems
Julia Lavid
Dep. English Philology I
Universidad Complutense de Madrid
28040 Madrid (Spain)
Phone and fax: +34-91-518-5799
e-mail: julavid@filol.ucm.es
This paper reports on the use of corpora for the construction of a computational grammar of Spanish,
contrastive with English, in the application context of Multilingual Natural Language Generation (MLG).
The theoretical framework for this work is Systemic Functional Linguistics (SFL) and the computational
context provided by KPML (Komet Penman Multilingual), an extensive grammar development
environment and generation engine that supports large-scale multilingual development (Bateman 1997).
The initial phenomena which are being investigated contrastively belong to three different functional
regions of the grammar, i.e., particular subareas of the grammar that are concerned with particular areas of
meanings. These regions are transitivity (ideational meaning), thematicity (textual meaning) and mood
(interpersonal meaning). The present study concentrates on textual meaning (thematicity and focus) as an
illustration.
Following what has now established itself as a standard methodology for empirically-based
Natural Language Generation (Bateman 1998a, Reiter and Dale 1997), the following steps were carried out:
first, a bilingual corpus (English-Spanish) was selected. For Spanish, a sample of spoken texts from the
MacroCorpus of the educated linguistic standard of the main cities of the Spanish-speaking world was used,
while for English a comparable sample was selected from the British National Corpus Sampler. This was
motivated by the need to provide a realistic account of the behaviour of the linguistic phenomena
investigated in unplanned and spontaneous contexts of use. The second step was to carry out a contrastive
analysis of the phenomena mentioned before. Finally, the results of the analysis were coded up as
resources/processes for generation. In the case of Spanish, these had to be created anew. In the case of
English, as the KPML already includes an English generation grammar, this last step consisted on checking
the coverage of the existing specifications and extending them when could not cover the instances found in
the corpus, and adapting them for effective MLG.
Given the nature of the NLG process, which typically converts communicative goals expressed in
some internal representation into surface forms, the kind of information that is most readily usable for NLG
are statements of mappings from functions to forms. Therefore, the corpus analysis phase for NLG usually
includes an explicit, and usually quite lengthy linguistic analysis where the analyst seeks possible
realisations of communicative functions, which restricts the size of the corpus that can be realistically
considered.
This paper describes the different steps carried out for the generation of the linguistic phenomena
mentioned above, discussing the problems encountered during the corpus analysis phase, and the
computational representation derived from it, as well as some of the decisions taken to overcome them.
357
1. Introduction
Natural Language Generation (NLG), the subfield of Artificial Intelligence and Computational Linguistics
that investigates the automated production of texts by machine, is typically a process that converts
communicative goals expressed in some internal representation into surface forms. As such, it touches upon
different linguistic areas of inquiry such as text planning and discourse organisation, lexical semantics,
grammatical and lexical choice, and the relationship between all of these. In fact, one of the main concerns
of NLG is the construction of of computational accounts of the linguistic system capable of generating texts
in one language (monolingual NLG) or in several (multilingual NLG, henceforth MLG). This theoretical
task poses unusual demands on the linguist who has to provide explicit and details accounts of how
language works and confront the results of the application of his/her theoretical claims in concrete
computational systems. A more practical concern is the creation of computational systems capable of
producing acceptable text from various sources and for different types of applications: well-known
examples include the generation of weather reports from meteorological data (Kittredge et al. 1986), the
generation of letters responding to customers (Springer et al. 1991), and other systems applied in areas such
as technical documentation and instructional texts (Not and Stock 1994, Paris et al. 1995, Lavid 1995),
patent claims (Sheremetyeva et al. 1996), information systems (Bateman and Teich 1995), computersupported
co-operative work (Levine and Mellish 1994), patient health information and education
(DiMarco et al. 1995), and medical reports (Li et al. 1986), to mention a few.1
While the first generation systems were limited to the random generation of grammatically correct
sentences, the field has experienced a very rapid growth over the past ten years, both as a research area
bringing a unique perspective on fundamental issues in artificial intelligence, cognitive science, and
human-computer interaction, and as an emerging technology capable of partially automating routine
document creation and playing an important role in human-computer interfaces. In this sense, as practical
systems became more sophisticated, it was necessary to provide a good understanding of the notion of
“textuality” and all the factors involved in the creation of different text types. In this context, it was only
natural that corpora started to be used as the empirical basis both for the theoretical investigation of textual
phenomena, and as part of the requirement analysis phase for NLG systems. As a result, the use of corpora
has now been integrated as part of the standard methodology for NLG, both in theoretically-oriented
research and in the development of concrete generation systems, to such an extent that computational tools
have been developed to support, among other functionalities, the analysis of machine-readable monolingual
or multilingual corpora for NLG (Alexa and Rostek 1997).
In this paper, we report on the use of bilingual corpora for the construction of a computational
grammar of Spanish, contrastive with English, in the application context of Multilingual Natural Language
Generation (MLG), concentrating on the textual phenomenon of thematicity and its relationship with the
related notion of focus as an illustration. Section 2 describes the two corpora selected and the criteria for
concentrating on two comparable samples as the empirical basis for computational specifications. Section
3 presents the theoretical framework selected for this study and the issues which must be explored
contrastively with respect to the textual phenomena mentioned before. Section 4 describes the corpus
analysis phase of the bilingual samples. Section 5 presents a computational specification of the results of
the analysis as resources for generation. The specification is based on the notion of functional typology as
developed in SFL and implemented in the KPML development environment. Finally, section 6 provides a
summary and discusses the implications of these results for corpus-based MLG.
2. Bilingual corpora for NLG
Two electronic corpora were initially targeted as the empirical basis for the contrastive work proposed in
this paper. These were the Macrocorpus de la norma lingüística culta de las principales ciudades del
mundo hispánico (Samper et al. 1998), a corpus of the educated linguistic standard of the main cities of the
Spanish-speaking world, and the British National Corpus. These two corpora were initially selected for two
main reasons: they both describe the educated speech of both Spanish and English and contain samples of
1 For an extensively documented review of the field see Bateman (1998b)
358
unplanned and spontaneous speech, which are necessary to provide a realistic account of the behaviour of
the linguistic phenomena investigated in their contexts of use. In the case of NLG, it is frequent to find
computational specifications of linguistic phenomena based on the analysis of specific text types, most of
them written monologues, which, though useful for the generation of texts of a specific domain, can only
provide a partial view on the phenomena studied. As the purpose of this study is to investigate the
behaviour of textual phenomena for a multipurpose contrastive grammar of English and Spanish, the choice
of samples from unplanned, spontaneous speech offered a rich and unexplored empirical basis for the
study of the textual phenomena mentioned before.
For the study of Spanish, a sample of spoken texts from the Spanish Macrocorpus was selected.
This Macrocorpus includes the transliteration of 84 hours of recording by 168 native speakers
representative of the educated speech of twelve Hispanic cities, nine of them from SouthAmerica and three
of them from the Iberian Peninsula. The recordings are basically unstructured interviews conducted in a
conversational style where the interviewer introduces a few questions and topics to stimulate the
conversation and to establish some uniformity in the topics discussed by the speakers. In general, the
speakers were left free to talk about the topics suggested by the interviewer and to introduce any new
topics. The sample selected for this study consists of 10 interviews from the three cities of the Iberian
Peninsula contained in the Macrocorpus, i.e. from Madrid, Seville and Las Palmas de Gran Canaria.
For the study of English, a comparable sample was selected from the BNC Sampler Corpus, a
subcorpus of the British National Corpus, consisting of approximately one-fiftieth of the whole corpus, viz.
two million words. The Sampler Corpus consists of 184 different texts, comprising just under 127,000
sentences and two million words. From this sampler corpus, a subcorpus of spoken texts was chosen for the
purposes of the present study, consisting of 10 conversations.
3. The textual resources of thematicity and focus in English and Spanish
The linguistic phenomena selected for illustrating the use of corpora in NLG belong to the textual region of
the grammar, and have been the subject of several functional accounts. The theoretical framework for the
study of these phenomena is SFL, as this is the theoretical basis for the implementation of the current
English generation grammar Nigel. In SFL, the textual clause grammar is composed of two complementary
systems, the systems of Theme and Information, characterised as assigning two different kinds of textual
prominence to elements of the clause: thematic prominence, in the case of the system of Theme, and
prominence as news, in the case of the system of Information. The notion of Focus, however, has not
received a comparable attention within SFL, but has been the subject of a number of discourse-oriented
linguistic and computational studies (Lavid 2000, McCoy and Cheng, McKeown, among others), and has
been treated in Dik´s Functional Grammar (Dik 1978) as a pragmatic function which assigns more salience
to some clausal constituent with respect to the contextual (pragmatic) information between language
producers and receivers.
In view of the need to provide form-function mappings of these phenomena which can be used by
a MLG system, a corpus-based analysis for generation purposes must explore at least the following issues
contrastively:
1. Do both English and Spanish grammars have a Theme and a Focus function ?
2. How do English and Spanish realise the Theme and the Focus functions - e.g.: sequence, inflection,
adposition, intonation?
3. Are there marked and unmarked Theme and Focus selections in both languages, and to what extent do
they depend on other systems (e.g. Mood, Voice)?
4. Are there resources in English and Spanish to combine thematicity and focus?
With respect to the first question, different linguistic studies acknowledge the existence of these two
functions in different languages (see Caffarel et al. in preparation; Dik et al. 1981), so this issue will not be
further explored here. The rest of the issues, however, are central for a functional characterisation of the
textual phenomena selected for this study, and their corpus-based empirical study is the basis for the
specification of resources required for MLG. Therefore, the next section will describe the corpus analysis
carried out for this study and the problems encountered when attempting to investigate the issues
mentioned above.
359
4. Contrastive corpus analysis for MLG: problems and decisions
As in the area of discourse analysis, the type of information that NLG needs from corpus investigations is
one which basically consists of statements of mappings from functions to forms. Therefore, when
investigating specific discourse phenomena, a NLG system requires as a first step a specification of the
mappings from functions to forms with the purpose of duplicating the text analysed. In this sense, the
corpus analysis phase raises some problematic issues which must be considered by NLG practitioners. One
of these issues is the size of the corpus: if, at is usually the case, the analyst has to intervene looking for
possible realisations of communicative functions, the use of large corpora becomes impractical. Also, it is
not possible to mark-up large quantities of texts according to functional categories if they cannot be
recognised automatically on the basis of tagged texts or syntactic analysis.
In view of these problems, the following decisions had to be taken to investigate the issues
mentioned above:
1.- With respect to the first issue, i.e., the ways in which English and Spanish realise the Theme and the
Focus functions, the corpus analysis was based on the following assumptions, based on previous linguistic
studies:
a) The Theme function was recognised as realised by clause initial position in both languages, as several
linguistic studies have demonstrated (Lavid in press; Taboada 2000).
b) The Focus function was recognised on the basis of non-prosodic realisations, such as word order
patterns, focus markers and characteristic constructions, since the BNC corpus does not include prosodic
annotation. 2
2.- With respect to the second issue, i.e., the existence and realisations of marked and unmarked theme and
focus selections in both languages, the corpus analysis was based on the following assumptions:
a) marked and unmarked themes were recognised in specific mood contexts, such as declarative,
interrogative and imperative options. Absolute themes were recognised by the presence of a pause and a
comma separating them from the rest of the predication.
b) unmarked focus was assumed to coincide with the last lexical element of the clause.3
3.- With respect to the third issue, i.e., the existence and variation of resources which combine thematicity
and focus in both languages, the corpus analysis concentrated on the so-called cleft and pseudo-cleft
sentences as realisation strategies used by both English and Spanish to combine the Theme and the Focus
functions. These were semi-automatically distinguished in both corpora on the basis of their characteristic
form.
5. Towards a computational specification for MLG
The second step in what has now established itself as a standard methodology for empirically-based NLG is
the codification of the corpus analysis results as resources/processes for generation. The current
implementation of the English generation grammar contained in the KPML development environment
already includes a computational specification for the Theme function in English as a textual region of the
grammar. However, as will be shown in the following sections, it became necessary to modify the existing
specification to account for instances found in the corpus and to ensure maximal sharing of resources for
contrastive generation. For Spanish a new specification was created on the basis of the contrastive corpus
analysis. It should be noted, however, that it is not the purpose of this study to provide a full specification
2 Considering that Focus in English is predominantly realised by marked prosodic prominence (Martínez-Caro 1999), this apparent
limitation of the corpus-analysis phase must be overcome by explicitly representing this realisation in the computational specification
for English.
3 According to Halliday (1994), unmarked focus falls on the last lexical element of the tonic group, which in unmarked circumstances
coincides with the clause.
360
of the behaviour of the textual phenomena of thematicity and focus in both languages, but rather to discuss
and illustrate with some examples some problems and issues raised by corpus-based contrastive MLG
specifications. The following sections, therefore, will illustrate some of these problems and the solutions
suggested in the context of a MLG architecture based on functional typology.
5.1. Thematic resources for English and Spanish: a functional-typological characterisation
This section presents a partial specification of the thematic resources available in English and Spanish on
the basis of the contrastive corpus-based analysis. More specifically, the area of theme markedness will be
discussed in detail, as there exist some interesting commonalities and differences which must be accounted
for when generating English-Spanish thematic variants. The approach rests on the notion of functional
typologies, as pursued in systemic-functional linguistics (cf. Halliday 1978), later developed for MLG
purposes (see Bateman et al. 1999). Functional typologies are constructed by means of classification
hierarchies called system networks (Halliday 1966), where each disjunction, or grammatical system is seen
as a point of abstract functional choice, capturing those minimal points of alternation offered by a linguistic
representation level. This representation of the so-called paradigmatic axis of linguistic description is
complemented with the corresponding syntagmatic realisation, i.e. the structural expression of the choices
made in the paradigmatic axis and associated with individual grammatical features. This task is carried out
by the so-called realisation statements which set allowable constraints on configurations of syntactic
constituents such as linear precedence, immediate dominance, or "unification" of functional constituents.
This type of approach, where the functional coherence of the paradigmatic organisation is preferred over
generalisations concerning possible structures, has been found to provide effective multilingual linguistic
descriptions and computational representations which maximise the factoring out of generalisations across
languages (cf Bateman et al. 1991, 1999).
Therefore, following this approach, and on the basis of empirical corpus-analysis, a specification
was created to account for Theme markedness in Spanish, as shown in Figure 1 below:
theme matter
marked absolute
not theme matter
predicated
Theme
Markedness non-absolute
Predication
non-predicated
unmarked (transitivity role)
Figure 1: Spanish Theme markedness systems
As the system network shows, the primary distinction in Spanish is between marked and unmarked themes,
and within marked themes between absolute or non-absolute themes. Absolute themes do not map onto any
transitivity or interpersonal functions, and are normally separated from the transitivity /interpersonal
structure of the clause by a comma in writing and by a pause in speech. They can be of two types:
representing a theme matter or not. If non-absolute, they can be predicated or non-predicated. When
predicated, they are normally realised by pseudo-cleft sentences, as will discussed in detail below. If they
are not predicated, thematic status may be assigned to any role within the transitivity structure of the clause
(participants, processes or circumstances).
With respect to English, the existing specification was found not to account for all the language
instances found in the corpus. Therefore, on the basis of the corpus-based analysis and to provide a
maximally effective specification for the purposes of MLG, the following specification was created for
English:
361
theme matter
marked absolute
not theme matter *
predicated
unmarked local
Markedness non-absolute Local theme
selection marked local
non-predicated
unmarked (transitivity role)
substitute
Theme
Substitution
non-substitute
Figure 2. English Theme markedness systems
As can be seen, an attempt has been made at capturing the comnonalities with the Spanish
specification while maintaining the English-specific integrity and difference. Thus, while English shares
with Spanish the primary distinction between marked and unmarked themes, previously existing
specifications (see Matthiessen 1995: 540) did not include a distinction between absolute and non-absolute
themes. However, in order to enforce maximal sharing of resources across languages, this system was also
included in the English network. In this sense, it should be noted that up to this point both languages are
similar, but differ as the features become more delicate: while absolute themes in Spanish may refer to a
theme matter or to any other transitivity role coreferential with a constituent within the predication, English
has a gap in this common paradigm (indicated by an asterisk in the English network) since it only has the
option of a theme matter as absolute theme. Also, English includes more delicate options within the
systems of Theme predication, as it includes the possibility of having a local theme selection, and includes
a system called Theme substitution, which is not available in the Spanish paradigm. In this new
specification, predicated themes have been considered as marked in the sense that they map the Focus
constituent, and, in many instances, the New element, onto the Theme (see Halliday 1994: 59 on predicated
Themes). Contrastive examples extracted from the corpora are discussed in detail below.
With respect to the syntagmatic realisations of the options presented in the networks above, both
languages present divergent grammaticalisations of their common paradigmatic potential. Table 1 below
illustrates the different features of the systems of both languages, together with their divergent realisations.
These are specified as sets of constraints which specify the syntactic and lexical properties of the linguistic
units being generated. For example, the feature ’theme substitute’ has the realisation statement +Pro-theme,
which means ’insert a Pro-theme’ and ’Pro-theme / Subject’ which means ’conflate or unify with the
grammatical function Subject'. The rest of the realisation specifies the ordering of constituents in relation to
one another: thus ’Rheme ^ Theme’ means order the Rheme before the Theme.
362
feature English realisation Spanish realisation
System: Markedness
marked
absolute
theme matter ’As to/as for/... + NG, + Pred. ’en cuanto a + NG', + Pred.
not theme matter Left Dislocation + Predication
non-absolute
predicated ’It + be + X..that/who’ Relative ^ Copula ^ NP
NP ^ Copula ^ Relative
Copula ^ NP ^ Relative
unmarked local
marked local ’NP it + be+ that’
non-predicated Part, Pro. or Circ. non-Subj. Part, Pro. or Circ. non-Subj.
unmarked
non-substitute
substitute +Pro-theme/ Subject ^ Rheme
^Theme
Table 1: Syntagmatic realisations of theme markedness options in English and Spanish
Some of the divergent realisations in both languages are the following:
1. Absolute Themes
Absolute themes refer to constituents which are not integrated within the transitivity/interpersonal structure
of the clause, and are, therefore, marked in speech with a pause, and with a comma in writing. As shown in
Figure 1 above, Spanish has two options when choosing absolute themes: as theme matter and as not-theme
matter. Example (1) below illustrates the choice of a theme matter in Spanish, an instance extracted from
the Spanish Macrocorpus:
(1) En cuanto a los libros que me gusta leer, tú sabes que a mí me gusta leer todo (SE-
10)
As for the books that me likes to read, you know that me likes to read everything
’As for the books that I enjoy reading, you know that I enjoy reading everything’
A contrastive example in English is illustrated by example (2) below, extracted from the BNC sampler
corpus:
(2) As for the past, I have adopted the doctrine of anamnesis (AEA)
When the absolute theme is not a theme matter, in Spanish it may be coreferential with an element of the
structure of the clause, as example (3) illustrates:
(3) .Y el ciudadano, el madrileno madrileno, se le nota un cambio.... (MA-10)
And the citizen, the Madrid Madrid (one), to him notice (3-sing) one change
’ And the citizen, the Madrid citizen, can one notice a change in him...?’
In (3), the constituent ’el ciudadano’ is not integrated in the transitivity structure of the clause, though it is
coreferential with an element functioning as beneficiary in the clause, expressed by the clitic ’le’ (italics in
the example).
English, however, lacks this option in its paradigm, and this is indicated by an asterisk in the
system network in Figure 2.
2. Predicated Themes
363
Both English and Spanish include predicated themes as a choice between presenting a Theme with a special
feature of identification or without that special feature. In fact, this choice is a textual strategy which
combines the Theme and the Focus functions into one single realisation, which, depending on the language,
may be a cleft, a pseudo-cleft, an identifying clause or a combination of the three. Thus, English typically
uses clefts with the form ('it + be + X + that ...) to realise predicated themes (italics) and to mark focus
constituents (boldface) as in examples (4) and (5) extracted from the BNC sample:
(4) It was only a relatively small Arab army that arrived in Egypt (JXL)
(5) It is not the death of the body that is important to us, it ’s the soul (NHE3)
Textually speaking, this construction serves two textual functions:
a) to identify the Theme in the transitivity role it serves in the clause: by using the identifying type
of clause we are achieving a textual distribution of meaning with the added feature of identification. For
example, in (4) ’the thing that arrived in Egypt’ is identified as ’a relatively small Arab army'. Similarly, in
(5) the ’the thing that is important to us’ is identified as not the death of the body but as the soul. According
to this, the segment ’it's not the death of the body’ would be the Theme of the clause, and ’that is important
to us’ would be the Rheme.
b) to define a specific constituent as having the Focus function, which, in many cases, is presented
as contrastive to another alternative. Example (5) is a case of what Dik would call counterpresuppositional
Focus, more specifically Substitute Focus. According to his definition, "the information presented is
opposed to other, similar information which the Speaker presupposes to be entertained by the Addressee"
(Dik 1989: 282). In (5), the speaker rejects the information which he/she presupposes to be entertained by
the Addressee (the death of the body) and corrects it with information which the speaker considers to be
correct (the soul).
English also uses pseudo-cleft and identifying clauses to combine the Theme and Focus functions.
Example (6) below illustrates a pseudo-cleft construction where the Theme is realised by a nominalising
clause introduced by ’what', and the rest of the clause serves as a Rheme. This construction also serves to
mark the focus constituent as the one following the copular verb:
(6) what we were able to do on one occasion was er to raise enough money (FYJ)
Spanish, by contrast, tends to use only pseudo-cleft clauses to combine the Theme and Focus functions.
The range of possible syntagmatic realisations in Spanish is more varied than in English, and includes three
main types:
1. Constructions where the relative clause appears in initial position, followed by the copular verb
and the nominal group functioning as an attribute. Exampe (7), extracted from the Spanish
Macrocorpus, illustrates this first type:
(7) Lo que yo necesito [...] es un poco más del estilo árabe... (SE-01)
What I need is a bit more of the style Arabian
’What I need is a bit more of the Arabian style’
2. Constructions where the nominal group is in initial position , the verb in medial position and the
relative clause at the end. Example (8) illustrates this type:
(8) Eso es lo que pienso de la familia (SE-04)
That is what think (1-sing.) of the family
’That is what I think of the family’
3. Constructions with the verb in initial position followed by the nominal group and the relative
clause. Example (9) illustrates this type:
(9) pero ha sido precisamente el avance de la anestesia [...] la que ha podido hacer que se [...]
pudiera hacer una serie de operaciones (SE-05)
364
But was (3-sing) precisely the advance of anaesthesia that could allow [...] that could be
made certain operations
’But it was precisely the advance of anaesthesia what allowed certain operations to be done’
3. Local Theme selection
In English when theme predication is selected, and it is realised by means of a construction of the type it +
be + ...[[that ...]], that clause opens up the potential for an additional thematic contrast - a contrast local to
that clause. In this case, the Theme local to the identifying clause may be unmarked (as it is in most of the
cases) or marked. If it is marked, it will coincide with the Complement / Identifier of the identifying clause
and will be given thematic prominence in initial position of the clause. No examples have been found in the
BNC sample, probably due to the fact that it is a small one. However, as other studies have proved its
existence with empirical evidence, we include the example provided by Matthiessen as an illustration:
(10) There we fell and my leg it was that broke (Matthiessen 1995: 567)
4. Substitute Theme
In English, when the Theme is unmarked and not predicated, there is a choice whether to re-introduce the
Theme/Subject at the end of the clause so as to give it a "thematic culmination". With substitute Theme, the
Theme/Subject is a pronominal nominal group and the substitute Theme is typically a full lexical nominal
group, though it is also possible to find examples with this, one, etc. In examples (11) and (12) below, the
Theme/Subjects are realised by the demonstrative ’that', and the lexical items ’piano’ and ’day release’ are
the substitute Themes, which serve as a textual reprise of the thematic referents:
(11) That ’s good, that ’s good for you, the piano you just had there (kp8)
(12) That ’s what you want to be after, day release.
6. Summary and concluding remarks
As the contrastive examples above have illustrated, a detailed corpus analysis in search for form-function
mappings of linguistic phenomena is an indispensable step as a basis for empirically-based computational
specifications for MLG. The study of the phenomenon of thematicity and its relationship with focus in both
languages, though partial and purely illustrative, has served to show how a functional analysis of selected
corpus samples may shed light on the paradigmatic features and their syntagmatic realisations of those
phenomena, which are the basis for computational specifications for MLG architectures based on functional
typologies.
However, the requirements of NLG, similar in many respects to the goals of discourse analysis,
raise important problems for corpus analysis: the need for semantic and contextual (pragmatic) analysis of
the selected corpora, which cannot always be recognised automatically and so cannot be extracted from
large-scale corpora without an analyst's intervention, makes, in most cases, the use of large corpora and
form-based corpus tools unsuitable for NLG purposes. Given the close relation between discourse analysis
and NLG, it can be hoped that new tools and specialised environments are developed to extract more from
current electronic corpora than what can actually be obtained from form-based quantitative corpus analysis
tools.
References
Alexa, M and Rostek, L 1996 Computer-assisted corpus-based text analysis with TATOE,
Technical Report, German National Research Center for Information Technology, Institute for Integrated
Information and Publication Systems, Darmstadt, Germany.
Bateman, J 1997 Enabling technology for multilingual natural language generation: the KPML
development environment. Journal of Natural Language Engineering 3 (1): 1-41.
Bateman, J 1998a Using corpora for uncovering text organization. IV-V Jornades de corpus
lingüístics. Barcelona, Institut Universitari de Lingüística Aplicada, Universidad Pompeu Fabra.
365
Bateman, J 1998b Automated discourse generation. In Kent A and Hall, C.M (eds), Encyclopedia
of Library and Information Science, Vol 62, New York, Marcel Dekker, pp 1-54.
Bateman J, Teich E 1995 Selective information presentation in an integrated publication system:
an application of genre-driven text generation. Information Processing and Management 31(5): 753-767.
Bateman J, Matthiessen C, Nanri K , Zeng L 1991 The Re-use of linguistic resources across
languages in multilingual generation components. In Proceedings of the 12th International Joint
Conference on Artificial Intelligence, Sydney, pp 966-971.
Bateman J, Matthiessen C, and Zeng, L 1999 Multilingual Natural Language Generation for
Multilingual Software: a Functional Linguistic Approach. Applied Artificial Intelligence 13: 607-639.
Caffarel A, Martin J R, and Matthiessen C in preparation Language typology: a functional
perspective.
Dik S C 1978 Functional Grammar. Dordrecht, Foris.
Dik S C, Hoffmann M, de Jong JR, Djiang S, Stroomer H, and de Vries L 1981 On the typology of
focus phenomena. In Hoekstra et al. (eds) Perspectives on functional grammar. Dordrecht, Foris.
DiMarco C, Hirst G, Wanner L, Wilkinson J 1995 Healthdoc: customizing patient information and
health education by medical condition and personal characteristics. In Proceedings of the Workshop on
Patient Education, Glasgow.
Halliday M A K 1966 Some notes on "deep grammar". Journal of Linguistics 2 (1): 57-67.
Halliday M A K 1978 Language as social semiotic. London, Edward Arnold.
Halliday M A K 1994 An introduction to functional grammar. London, Edward Arnold.
Kittredge R, Polguere A, Goldberg E 1986 Synthesizing weather reports from formatted data. In
Proceedings of the 11th International Conference on Computational Linguistics, Bonn, Germany, pp 563-
565.
Lavid J 1995 From interpersonal option to thematic realisation in multilingual instructions. In
Kittredge R (ed) Working notes of the IJCAI-95 workshop on text generation. Montreal.
Lavid J 2000 Theme, focus, given and other dangerous things: linguistic and computational
approaches to information in discourse. Revista Canaria de Estudios Ingleses :
Lavid J in press La noción gramatical de tema en un contexto multilingüe: una perspectiva
funcional-tipológica. In Treinta anos de la Sociedad Espanola de Lingüística. Madrid, Gredos.
Levine J, Mellish C 1994 Corect: Combining CSCW with natural language generation for
collaborative requirements capture. In Proceedings of International Joint Conference on Artificial
Intelligence, Montreal, Canada, pp 1398-1404.
Li P, Evens M, Hier D 1986 Generating medical case reports with the linguistic string parser. In
Proceedings of the 5th National Conference on Artificial Intelligence, Philadelphia, pp 1069-1073.
Martínez-Caro E 1999 Gramática del discurso: foco y énfasis en inglés y en espanol. Barcelona,
Promociones y Publicaciones Universitarias.
McCoy K, Cheng J 1990. Focus of attention: constraining what can be said next. In Paris et al.
(eds) Natural language generation in artificial intelligence and computational linguistics. Kluwer,
Dordrecht, 1990.
McKeown, K 1995 Text generation: Using discourse strategies and focus constraints to generate
natural language text. Cambridge, Cambridge University Press.
Matthiessen C 1995 Lexicogrammatical cartography: english systems. Tokyo, International
Language Sciences Publishers.
Not E, Stock, O 1994 Automatic generation of instructions for citizens in a multilingual
community. In Proceedings of the European Language Engineering Convention, Paris.
Paris C, Van der Linden K, Fisher M, Hartley A, Pemberton L, Power R, Scott D 1995 A support
tool for writing multilingual instructions. In Proceedings of the International Joint Conference on Artificial
Intelligence, Montreal, pp1398-1404.
Reiter E, Dale R 1997 Building applied natural language generation. Journal of Natural Language
Engineering 3(1): 57-87.
Samper JA, Hernández Cabrera CA, Troya Déniz M 1998 Macrocorpus de la norma lingüística
culta de las principales ciudades del mundo hispánico. Universidad de las Palmas de Gran Canaria.
Sheremetyeva S, Nirenburg S, Nirenburg I 1996 Generating patent claims from interactive input.
In Proceedings of the 8th International Workshop on Natural Language Generation, Herstmonceaux,
England, pp 61-70.
366
Springer S, Buta P, Wolf T 1991 Automatic letter composition for customer service. In Smith R,
Scott C (eds) Innovative applications of artificial intelligence 3. Menlo Park, Ca AAAI Press.
Taboada M 2000 Collaborating through talk: the interactive construction of task-oriented
dialogue in English and Spanish. Unpublished PhD thesis, Universidad Complutense de Madrid.

367
Tracing referent location in oral picture descriptions
Maarten Lemmens
Université de Lille Katholieke Universiteit Leuven
Dept. of English I.L.T. (Swedish)
BP 149 Dekenstraat 6
59653 Villeneuve d'Ascq Cedex, France 3000 Leuven
research grant: URM 8528 SILEX du CNRS Belgium
Against the background of the typological distinction between verb-framed and satellite-framed
languages, the present paper will show that the opposition between verbs of POSITION (e.g. sit, lie,
stand, etc.) and verbs of EXISTENCE (e.g. be, be found, etc.) can also be regarded as a parameter in the
typological distinction. At the same time, some of the claims made in the literature on manner of
motion verbs will have to be nuanced when applied to verbs of POSITION/EXISTENCE. The empirical
basis of this contrastive research project (initially focussing on Germanic languages) consists of
standard text corpora as well as elicited oral picture descriptions.

369
A corpus study of impersonalisation strategies in newspaper discourse in English
and Spanish1
Juana Marín-Arrese, Elena Martínez-Caro, Soledad Pérez de Ayala Becerril
Universidad Complutense de Madrid
1. Introduction
Strategies of impersonalisation in English and Spanish include the use of passives, resultatives,
anticausatives, impersonal constructions, nominalizations, and various forms of lexical
underspecification (Nedjalkov 1988, Shibatani 1988, Gómez Torrego 1992, Fox and Hopper 1994,
Mendikoetxea 1999). The use of these strategies in newspaper discourse reporting political events
reflect issues in language and ideology, such as the intentional mystification of agency and vagueness
of responsability in discourse (Fairclough 1989, Fowler 1991, Gruber 1993, Curran and Seaton 1997,
van Dijk 1998).
This paper reports on on-going research as part of a major project on the use of these strategies
in British and Spanish newspaper articles on political issues. In the collection of texts, we have
established a gradient, from those news reports where neither Britain nor Spain is alluded to or
implicated, to a situation where each country is both mentioned and implicated in the event.
The purpose of this study is twofold: (1) the identification of qualitative and quantitative
differences in the use of impersonalisation strategies between the two languages2, i.e. whether the
same type of strategies are used in the two languages and the extent to which they are used; and (2)
the correlation between the use of strategies and the degree of implication which the newspaper
articles reflect in both languages.
2. Impersonalisation Strategies
2.1. Impersonalisation
We have used the term ‘impersonalisation strategies’ to refer to a variety of linguistic means
which allow for mystification of the role of agency. In our study we have focused on the following:
agentless passives, ed-participles, resultatives, impersonals, anticausatives, impersonal pronouns,
infinitive clauses, nominalizations, existentials, and a set of miscellaneous occurrences of lexical
underspecification, including metonymy and others.
In terms of recoverability of the identity of the agent, the examples chosen represent a gradient
in implicitnes of agency. In some cases, the underlying agent is recoverable from the preceding or
following co-text. In others, it may be inferred on the basis of shared knowledge, or shared event or
context models, which allows us to predict the type of agent characteristically involved in the event,
though the identity of the agent is not recoverable to the point where unique reference can be
established. Finally, there are cases where deagentivization is absolute.
The reasons for omission of the agent in the passive may be based on relevance criteria (Sperber
and Wilson 1986). As Biber et al. (1999:477) note, in news reports the agent “may be easy to infer,
uninteresting, or already mentioned”. But this exclusion may also be the result of “possible
ideologically motivated obfuscation of agency, causality and responsibility”, as Fairclough (1989:124)
puts it. In this way, the passive allows not only for mystification of agency but also for claiming
ignorance about the identity of the agent, thus obscuring responsibility for negative action.
Impersonal constructions or impersonal use of pronouns such as people, they, someone, no one
typically exclude both Speaker/Writer and Addressee/Reader from the action, exonerating them from
responsibility and implication. The use of we, you, one, on the other hand, allows for the inclusion of
the Speaker/Writer (and the Addressee/Reader), in this way creating an expectation of implication and
responsibility in the action. Very often, however, they also reflect the distinction established between
ingroups and outgroups of various kinds (van Dijk 1998).
In the inchoative or anticausative3 construction, intentional actions are presented as ‘events’
1 This paper is based on work supported by the Universidad Complutense de Madrid under research project n
(Project Director: Juana Marín-Arrese).
2 In this paper for the proceedings we only present and discuss the data for Spanish, due to limitations of space.
3 There is considerable variation with regard to the terminology used in the literature. The terms ergative and pseudo-passive have
also been found.
370
occurring spontaneously. Spontaneous events are typically cases of autonomous or absolute construal.
We may distinguish between ‘intrinsically spontaneous events’ (Kemmer 1993) or ‘internally-caused’
events (Levin and Rappaport 1995), that is, those viewed as occurring without the direct initiation of
an external cause, and ‘non-intrinsically spontaneous events', involving some abstract and schematic
cause, the evocation of the notion of ‘external causation’ (Langacker 1991), which may be agent,
instrument, natural force or circumstance. In the case of non-intrinsically spontaneous events, the
expression of the causative event is realized by means of an unmarked lexical causative construction;
the same verb form is used in both causative and inchoative in English, the so-called labile
alternation. Spanish presents a far more complex situation, though, in general terms, nonintrinsically
spontaneous events tend to be coded by means of the marked anticausative construction
with se, while intrinsically spontaneous events are coded by an unmarked intransitive (unaccusative)
construction (Marín-Arrese 2000).
Nominalizations represent a step further in impersonalisation. The examples chosen were those
where the agent is not mentioned and the patient participant is defocused. The actional component is
obscured, and the event is presented as ‘fact'.
Referential-underspecification is found in various forms of lexicalization used to identify the
participants. Agents may be described in general or abstract terms, or may be referred to through the
use of metonymic expressions. Other such devices include ethnonyms, role description, attribute
description, etc. Choices in the level of specificity in describing the actions may also result in the use
of abstract nouns denoting events, rather than intentional actions.
In view of the above mentioned strategies, we may posit a continuum in ‘agency', ranging from
implicit reference to the agent to some abstract and schematic notion of causation, and a parallel
continuum in ‘actionality', with actions at one polar end, and facts at the other (adapted from Marín-
Arrese, in press).
IMPLICIT <------ NON-RECOVERABLE --------> SCHEMATIC
Passive Nominalization Anticausative Abstract nominals
ACTION <------ EVENT -------> FACT
Passive Anticausative Nominalization Abstract nominals
These notions are related to the more general conceptual dimension ‘relative elaboration of
event'', which, as Kemmer (1994:211) suggests, “can be thought of as the degree to which different
schematic aspects of a situation are separated out and viewed as distinct by the speaker”. This
dimension subsumes the semantic parameter relative ‘distinguishability of participants'. In passive
events, for example, the Initiator or Agent participant is defocused and thus the degree of
distinguishability of participants is lower than in active causative events. Similarly, in spontaneous
events, the single participant coded is construed as the Initiator and also as the Endpoint, since it
undergoes some change of state as well.
Langacker (1991:372) observes that “since transitivity depends on the conception of distinct,
well-differentiated participants, it is potentially influenced by the extent of their differentiation along
not only the objective but also the subjective axis”. This participant distinctiveness thus involves not
only more objective features, such as the conceptual distinction of entities into separate participants
(agent vs. patient), or the relative salience of these entities with respect to each other and from their
background, but also a subjective component, comprising parameters such as “the precision and detail
of its type specification” (basic level vs. superordinate category), “the degree of definiteness, which
pertains to whether the speaker and hearer have succeeded in making mental contact with a particular
instance of the type in question”, and finally “the profile's extension”, that is whether the entity is
presented as compact (vs. extended/diffuse) in terms of the oppositions: “spatially compact vs.
spatially extended; participant vs. setting; singular vs. plural; count vs. mass; concrete vs. abstract;
and restricted portion of a reference mass vs. the mass as a whole”.
2.2. Towards a Taxonomy of Impersonalisation Strategies in Spanish
Impersonalisation strategies in Spanish include the periphrastic passive with ser and the
resultative with estar, as well as the ed-participle and a variety of motion and result verbs in nonprototypical
and resultative passives.
371
(1) a. Una persona fue detenida por resistir a la autoridad (CSp01)
one person BE.PAST.3SG arrested
‘One person was arrested for resisting ... ‘
b. , una vez vencido el terrorismo, (CSp10)
once defeat.ED the terrorism
‘Once terrorism is defeated’
c. .Que por qué estamos amenazados? (CSp02)
that why BE.1PL threatened
You mean, why are we threatened?
d. , la propuesta va dirigida contra ... (CSv04)
the proposal GO.3SG directed against
‘the proposal is directed against ...’
Spanish makes use of a reflexive element se for both the foregrounding and backgrounding, or
promotional and non-promotional, reflexive passive4 (Foley and van Valin 1984; Givon 1990), as well
as for the impersonal reflexive. The expression of non-intrinsically spontaneous events in Spanish
also involves the use of se in the anticausative construction (Marín-Arrese 1992, 2000).
(2) a. Aunque el informe se elaborará más tarde, (CSv05)
although the report SE prepare.FUT.3SG later
‘Although the report will be prepared later, ...’
b. que se importaran harinas cárnicas , (CSa02)
that SE import.SUBJ.3PL flour.PL meat.PL
‘that compound feed (made from meat) should be imported,
c. no se ha detenido a nadie (CSp01)
not SE have.3SG arrested to.ACC nobody
‘Nobody has been arrested’
d. que se les equipare a los funcionarios
(CSp04)
that SE them.ACC put on a level.SUBJ.3SG to.DAT the civil servants
‘that they should be put on a level with civil servants’
e. no se les reconoce derechos fundamentales de las personas
(CSp09)
not SE them.DAT recognize.3SG rights.PL fundamental of the persons
‘They do not recognize their fundamental rights as people/their fundamental rights
are not recognized’
f. se cree que podría ser decisivo ... (CSa04)
SE believe.3SG that might.3SG be decisive
‘It is believed/They believe that it might be decisive ...
g. mientras no se avance. (CSp03)
while not SE advance.SUBJ.3SG
‘while there is no advance’
h. ‘sorprendentemente’ empezaron a producirse los reiterados incendios de
... (CSp07)
surprisingly begin.PAST.3PL to produce.SE the repeated fires of
‘surprisingly, the repeated fires (of ...) started to take place’
Intentional actions may also be portrayed as intrinsically spontaneous events by means of an
intransitive (unaccusative) construction (Martínez-Caro 1999).
(3) a. porque la lucha empieza ahora. (CSp05)
because the fight begin.3SG now
‘because the fight begins now’
4 We have established a distinction between the promotional reflexive passive, where the Endpoint nominal acquires features of
subjecthood, and the non-promotional passive, where there is no verbal agreement. In both cases, the expression of the agent is
typically not allowed. Finally, we have included within the impersonal category all those instances where there is no Endpoint
nominal element (Marín-Arrese 1992)
372
b. estas propuestas han surgido ...(CSp04)
these proposals have.3PL arisen
‘these proposals have arisen ...’
As in the case of English, other impersonalization strategies in Spanish include impersonal use
of pronouns, infinitive clauses, existentials, and nominalizations.
(4) a. “En el sur nos explotáis, en el norte nos expulsáis”
(CSv05)
in the south us.ACC exploit.2PL, in the north us.ACC expel.2PL
‘In the South you exploit us, in the North you expel us’
b. donde le informaron que ... (CSv03)
where him inform.3PL that
‘where they informed him that ...’
c. para luchar contra el terrorismo (CSa01)
to fight.INF against the terrorism
‘to fight against terrorism’
d. hay que acabar con el tabú de ... (CSp10)
have.3SG that finish.INF with the taboo of
‘we have to put an end to the taboo of ...’
e. urge poner en común una política de ... (BSv02)
urge.3SG put.INF in common a policy of ...
‘it is urgent to negotiate an immigration policy...’
f. No hubo torturas. (BSv03)
not have.PAST.3SG torture.PL
‘There was no torture/He was not tortured’
g. la diseminación de modelos culturales (ASa03)
the dissemination of models cultural
‘the dissemination of cultural models’
Finally, we find miscellaneous occurrences of referential underspecification and lexical
vagueness, where the actors are described at various levels of generality, or by means of metonymy,
metaphor, etc., and the actions are presented as abstract nominals.
(5) a. si la gente altera el balance de la creación (ASa03)
if the people alter.3SG the balance of the creation ...
‘if people alter the balance of creation ...’
b. Los extranjeros desconfían de la buena voluntad de ... (CSa05)
the foreigners mistrust.3PL of the good will of
‘Foreigners mistrust the good will of ...’
c. mataderos que incumplen la ley. (CSp01)
slaughterhouses that break.3PL the law
‘slaughterhouses that break the law’
d. esta reliquia criminal le ha arrancado a companeros como
...(CSp02)
this relic criminal him.DAT have.3SG torn off to.ACC comrades like ...
‘this criminal relic has taken away from him comrades such as...’
e. los sucesos en la comarca de Almería (CSv05)
the events in the region of Almería
‘the events in the region of Almería’
f. que las responsabilidades puedan llegar al Ministerio de Agricultura
(CSp07)
that the responsibilities may.3PL arrive at.the Ministry of Agriculture
‘that the responsibilities may reach the Ministry of Agriculture’
3. Text Collection and Data
3.1. The Texts
373
The corpus used for this paper consists of a total of 60 texts extracted from three Spanish
newspapers: 30 from El País, 15 from ABC and 15 from La Vanguardia (approx. 45.000 words). ABC
and La Vanguardia are rather conservative, and therefore represent ideologies that are close to the
present Government in Spain. El País is more radical, and thus more critical with the present
governmental policy. ABC and El País are national newspapers, while La Vanguardia is issued in
Cataluna, though there is also an edition printed in Madrid.
The texts collected are news reports of a political nature, chosen from the National and
International sections in the papers. The texts vary in degree of ‘potential’ implication: Type A
includes international news reports, with no national implication, i.e. the news do not affect Spain in
an immediate and direct way, and the country is not even mentioned (e.g. the recent U.S. presidential
election); Type B is a selection of international news reports which affect Spain in a direct way (e.g.
the conflict concerning the presence of the nuclear submarine Tireless in Gibraltar); Type C includes
national news reports, which obviously affect the country in a direct way.
The codification system that has been used for examples of each text is the following:
AS: International [- implication], SPANISH
BS: International [+ implication], SPANISH
CS: National, SPANISH.
p: El País,
a: ABC,
v: La Vanguardia
01, 02...: text number.
For example, CSp01 refers to a National item of news taken from the Spanish newspaper El
País, text number 1.
3.2. Categories of Impersonalisation Strategies
The types of impersonalisation strategies found in the Spanish texts have been organized and
numbered in the following way:
(i) Agentless Passive
(ii) –Ed participle (-agent)
(iii) Resultative estar
(iv) Non-prototypical passive/resultative (-agent)
(v) Passive se (foregrounding or promotional)
(vi) Passive se (backgrounding or non-promotional)
(vii) Impersonal se
(viii) Anticausative se, Unmarked intransitive (spontaneous events)
(ix) Impersonal use of pronouns (they, you, one, we, ...)
(x) Infinitive clauses (-agent)
(xi) Modalised impersonal expressions (hay que, urge, ...)
(xii) Existential
(xiii) Nominalisations
(xiv) Miscellaneous lexical strategies
4. Discussion of results
The following table (Table 1) shows the number of instances of each strategy found in the three
groups of texts: international without implication (A); international with implication (B); and national
(C). Within each type, differences between conservative (ABC and La Vanguardia) and liberal
newspapers (El País) are shown. The third column of each group of texts contains the global
percentages of use for each strategy in comparison to the other groups of texts. For example, we have
found a total amount of 58 cases of strategy I (Agentless periphrastic passive); of this amount, 23.3%
represent instances found in A texts (both conservative and liberal), 34.4% in B texts and 36.2% in C
texts. The last line contains the total number of examples of all the strategies in each type of text and
the percentages that these represent. The bottom line contains the total number of cases for the three
groups of texts analysed, adding up to a total of 859 instances.
374
The last column of the table includes the total amount of instances of each strategy followed by
the percentages each category represents with regard to the overall number of cases. These figures
give us an idea of the relative distribution of impersonalisation strategies in discourse.
A texts (15,696 words) B texts (15,234 words) C texts (14,088 words) Linguistic
strategies Cons. Liberal % Cons. Liberal % Cons. Liberal %
TOTAL
I 7 10 23.3 12 8 34.4 2 19 36.2 58 6.7%
II 6 14 25.3 6 5 13.9 21 27 60.7 79 9.1%
III 1 1 13.3 2 4 40 5 2 46.6 15 1.7%
IV 2 3 27.7 1 4 27.7 5 3 44.4 18 2%
V 14 18 19.8 16 26 26 38 49 54 161 18.7%
VI 0 1 12.5 1 3 50 0 3 37.5 8 0.9%
VII 8 5 30.9 6 5 26.1 6 12 42.8 42 4.8%
VIII 4 2 11.7 2 11 25.4 16 16 62.7 51 5.9%
IX 6 8 36.8 2 5 18.4 13 4 44.7 38 4.4%
X 7 3 18.8 7 4 20.7 18 14 60.3 53 6.1%
XI 2 3 21.7 8 2 43.4 5 3 34.7 23 2.6%
XII 0 1 10 0 3 30 2 4 60 10 1.1%
XIII 25 23 27.7 20 13 19 43 49 53.1 173 20.1%
XIV 10 22 24.6 7 10 13 31 50 62.3 130 15.1%
TOTAL
92
114
23.9
90
103
22.4
205
255
53.5
859
Table 1: Number of instances of impersonalisation strategies and percentages within the three groups
of texts, A, B and C and total
In this table we may observe a number of interesting features. Going from the more general to
the more specific, one striking aspect is that both the A and the B texts have similar total figures and
percentages, while the total figures for the C texts are more than double those of A and B. This, we
think, may reflect the higher degree of implication of the writers in the national news reports (C
texts), who also resort to a higher number of impersonalisation strategies than the writers of
international news. Although the B texts represent news in which Spain is also implicated, the data
seem to evince that the degree of implication is always more reduced, and writers appear to use
impersonalisation strategies in a similar way to writers of international news with no implication (A
texts).
Among the more specific results, we would like to highlight the following aspects:
(a) With respect to the differences in the use of the various strategies, the impersonalisation device
most used in Spanish seems to be nominalisation (strategy XIII), followed by the use of the se
passive (V), and somehow less frequently, -ed participles (II). The miscellaneous category (XIV)
also shows high figures, but the results are not as significant since it is a mixed category
comprising various lexical forms of underspecification.
(b) The figure for the ordinary or periphrastic passive (strategy I) is remarkably low, especially if
compared with the use of the passive in English. Biber et al. (1999:476) find that passives are
quite common in news, “occurring about 12,000 times per million words”. The ratio in English
would thus be 0.012, while the ratio of use of the agentless passive in the Spanish texts is 0.0012,
that is, one tenth of the frequency in English (of both agented and agentless passives). It is also
interesting to compare this figure to that of the se passive (V), which shows that Spanish writers
tend to use se passive constructions twice as often as the ordinary passive construction.
Nonetheless, if we add up the results for passive in Spanish (strategies I, IV, V, VI), the ratio of
use , 0.005, is comparable to that found for agentless passives in English, 0.006, (Marín-Arrese
1997).
(c) If we now compare the percentages for texts A, B and C, as we already mentioned, the figures
are in general much higher in the C texts. This is especially the case with respect to the use of the
-ed participle (II), the se passive (V), anticausatives and intransitives (VIII), impersonal infinitive
clauses (X), existential haber, existir (XII) and nominalisations (XIII).
375
With respect to the use of each strategy, it is worth considering the following:
(a) The cases of agentless periphrastic passive can be divided into two groups, from the point of view
of the writer's intention to hide the identity of the agent. On the one hand, as mentioned above,
there are many examples where the agent is not mentioned simply because it is unimportant or
unknown. On the other hand, and less frequently, there are also examples which seem to show
that the writer is using this syntactic device to intentionally make the agent disappear. Compare,
in this respect, examples 6 and 7, which respectively show the two tendencies mentioned:
(6) (Pinochet instructs the military auditor to hide the fact that Mr. Eugenio Ruiz-Tagle was
tortured in the following way:) “El senor Eugenio Ruiz-Tagle O. fue ejecutado en razón
a los graves cargos que existían contra él. No hubo torturas.” (BSv03) (‘Mr. Eugenio
Ruiz-Tagle was executed due to the grave accusations that existed against him. There
was no torture/He was not tortured')
(7) ... miembros de la mafia que residen en Espana y que están pendientes de ser juzgados o
han sido condenados por graves delitos en su país. (BSv01) (‘... members of the mafia
living in Spain and who still have to be tried or have been sentenced for serious crimes
in their country').
(b) In some of the strategies it is interesting to note that there is a tendency for most examples to
come from direct quotations. This is typically the case of the impersonal use of pronouns (IX) and
the impersonal se (VII), and is also found in strategies such as the non-prototypical passive (IV).
See the following :
(8) “En este país todos sabemos qué es lo que tenemos que hacer para ...” (CSp02) (‘In this
country we all know what we have to do to ... ‘)
(c) Most examples of impersonal se occur with verbal processes of the type of decir, asegurar,
confirmar:
(9) En medios diplomáticos y políticos se aseguraba ayer que si fracasa la operación Peres...
(ASp08) (‘In diplomatic and political circles they assured that ...')
(d) It often happens that writers use more than one category, or the same category more than once, in
the same stretch of text, thus creating a cumulative effect of mystification of agency. See for
example:
(10) “Nosotros (los jueces) no criticamos, sólo observamos y se resuelve sobre lo que se
observa. Y mientras aquí se siga creyendo que un juez toma partido, seremos
subdesarrollados mentalmente”, remarcó el magistrado chileno. (BSa01) (‘We (the
judges) do not criticize, we only observe, and rule according to what is observed. And
while people here continue to believe that a judge takes sides, we will be mentally
underdeveloped')
(11) Los querellantes aseguran que se ha permitido al Tireless navegar por aguas espanolas,
no ha sido inspeccionado por técnicos espanoles, no se ha adecuado plan de
emergencia alguno en previsión de incidentes en la reparación del submarino, se han
ocultado datos relevantes, se ha mentido por el presidente del Gobierno (sic) respecto a
la existencia de dictámenes del Consejo de Seguridad Nuclear y se ha puesto en riesgo a
los habitantes del Campo de Gibraltar. (BSp04) (‘...the submarine Tireless has been
permitted ..., it has not been inspected by ..., no emergency plan has been adopted ...,
relevant facts have been concealed ..., the President of the Government has lied with
respect to ... and the inhabitants of Gibraltar have been endangered ...')
Example (11) is particularly interesting, because the cumulative effect of mystification of agency
(the various members of the Government) is suddenly broken by the ‘allegedly ungrammatical’
inclusion of the adjunct “por el presidente del Gobierno”, expressing the identity of the agent of
the action denoted by an impersonal passive clause. The effect is such that this agent is
376
immediately attributed direct responsibility for all the previous events denoted.
(e) It has been observed that many cases of mystification of agency may have to do with the use of
linguistic politeness, that is, writers use impersonalisation strategies to avoid direct facethreatening
acts. Brown and Levinson (1987) include the phenomenon of impersonalisation as
negative politeness strategy number 7 (see also Pérez de Ayala 2001). In our corpus we have
found the following:
(12) “Hay actitudes y comportamientos que sobrepasan los límites más elementales del
sentido común”, comentó (Josep Piqué) (BSp04) (‘There are attitudes and behaviours
that excede the most elemental limits of common sense.')
It should be taken into account that the results here presented are rather tentative, since the
corpus used is relatively limited.
5. Conclusion
The present paper has investigated the use of impersonalisation strategies in Spanish in a corpus
of newspaper articles on political issues. The texts represent various degrees of ‘potential’ implication
on the part of the writer with regard to the topic of the news item.
The results found seem to point to a direct relationship between degree of implication and more
frequent use of the various strategies of impersonalisation. As regards the relative use of each of the
strategies, Spanish newspaper discourse seems to favour the use of passive se and that of
nominalisations.
Although the factor of relevance is undoubtledly crucial in the omission of the agent, we may
also surmise that the varied use of all these strategies cognitively contributes to construct, in van
Dijk's (1998) words, ‘preferred models’ of a situation, and, socio-politically, to hide institutional or
elite group responsibility.
References
Biber, D., Johansson, S. Leech, G., Conrad, S. and E. Finegan 1999 Longman Grammar of Spoken
and Written English. London, Longman.
Brown, P. and S. Levinson 1987 Politeness: Some universals in language usage. Cambridge,
Cambridge University Press.
Curran, J. and J. Seaton 1997 Power without Responsibility. The Press and Broadcasting in Britain.
London, Routledge.
van Dijk, T. 1998 Ideology: A Multidisciplinary Approach. London, Sage.
Fairclough, N. 1989 Language and Power. London, Longman.
Foley, W.A. and R.D. van Valin 1984 Functional Syntax and Universal Grammar. Cambridge,
Cambridge University Press.
Fowler, R. 1991 Language in the News: Discourse and Ideology in the Press. London, Routledge.
Fox, B. and P. Hopper (eds.) 1994 Voice: Form and Function. Amsterdam, John Benjamins.
Givon, T. 1990 Syntax: A Functional-Typological Introduction, Vol.II. Amsterdam, John Benjamins.
Gómez Torrego, L. 1992 La Impersonalidad Gramatical: Descripción y Norma. Madrid, Arco libros.
Gruber, H. 1993 Political language and textual vagueness. Pragmatics 3(1): 1-28.
Kemmer, S. 1993 The Middle Voice. Amsterdam, John Benjamins.
Kemmer, S. 1994 Middle voice, transitivity and the elaboration of events. In: B. Fox and P. Hopper
(eds.), pp. 179-230.
Langacker, R. 1991 Foundations of Cognitive Grammar. Vol. II. Stanford, CA, Stanford University
Press.
Levin, B. and M. Rappaport Hovav 1995 Unaccusativity: at the Syntax-Lexical Semantics Interface.
Cambridge, MA, MIT Press.
Marín Arrese, J. 1992 La Pasiva en Inglés: Un Estudio Funcional-Tipológico. PhD dissertation,
published 1993. Madrid, Editorial Complutense.
Marín-Arrese, J. 1997 Cognitive and discourse-pragmatic factors in passivisation. ATLANTIS XIX (1):
203-218.
Marín-Arrese, J. 2000 On ‘thematic subject’ constructions in English and Spanish. Paper presented at
the International Conference on ‘Cognitive Typology', 12-14 April 2000, University of
377
Antwerp, Antwerp.
Marín-Arrese, J. in press The linguistic coding of spontaneous and facilitative events in English: A
cognitive perspective. To appear in: New Waves: 21st Century English Grammar. Huelva,
Universidad de Huelva.
Martínez-Caro, E. 1999 Gramática del Discurso: Foco y Énfasis en Inglés y en Espanol. Barcelona,
PPU.
Mendikoetxea, A. 1999 Construcciones con se: medias, pasivas e impersonales. In: Bosque, I. and V.
Demonte (eds.,1999) Gramática Descriptiva de la Lengua Espanola. Madrid, Espasa Calpe.
Myhill, J. 1997 Toward a functional typology of agent defocusing. Linguistics 35: 799-844.
Nedjalkov, V. (ed.) 1988 Typology of Resultative Constructions. Amsterdam, John Benjamins.
Pérez de Ayala Becerril, S. 2001 FTAs and Erskine May: Conflicting needs? Politeness in Question
Time. Journal of Pragmatics 33: 143-169.
Shibatani, M. (ed. ) 1988 Passive and Voice. Amsterdam, John Benjamins.
Sperber, D. and D. Wilson 1986 Relevance: Communication and Cognition. Oxford, Basil Blackwell.

378
Is there such a thing as a translator's style?
Mikhail Mikhailov and Miia Villikka
Department of Translation Studies, University of Tampere, Finland
{Mihail.Mihailov, Miia.Villikka}@uta.fi.
Like women, translations should be either beautiful or faithful (cf. Mounin 1994)
1. General
It would be naive to think of a literary translation as an exact copy of the original, its simple
replica in another language. Even a situation in which two translators are given the same source text
and instructed to translate it as faithfully to the original as possible would result in two clearly
different translations.
In translation studies the question of loyalty towards the original is one of the most discussed
ones. It has been widely argued that literary translations should maintain the style and structure, the
“spirit” of the original as intact as possible (see e.g. Chesterman 1997, Nida & Taber 1974). After all,
it is the original author's name that is printed on the very cover of the translation, i.e. even the
translation is considered to be a work of the author, not of the translator.1
However, it must be borne in mind that translators are also individuals and it is impossible for
them to totally set aside their own personality and “get under the original author's skin”. There are
several factors influencing the translation process and, consequently, the final product, the translation.
One important factor – the key focus area of this research - is the fact that many translation
solutions must be decided upon independently. Translating is not just a simple decoding-recoding
action performed merely on the level of words – and even if it were, even the best of dictionaries could
not by far offer equivalents and examples to every possible context of a certain word or expression.
Also, there are often more than just one suitable equivalent. Usage of these variants is in most cases
solely up to the translator's choice.
Further, relations between two different language (grammar) systems are no more stable. Rather
seldom is there only one possible way to translate a certain structure. Naturally, every translator aims
at producing as fluent a text in the target language as possible, but even opinions on fluency, let alone
the means of achieving it, vary greatly between different individuals.
2. “Stylistic fingerprints”
The ‘stylistic fingerprint’ problem is nowadays widely discussed in applied linguistics. Most
scholars agree that every author has a unique and identifiable style. However, there is no shared
opinion on the criteria which can be used for authorship attribution. For example, proportion of nouns
or adjectives, and so called “marker words” (e.g. while - whilst, upon - on) are frequently used. New
methods like principle components analysis of the most common words, measuring vocabulary
richness and even letter frequency analysis are being developed (Holmes & Forsyth 1995; Tweedie &
Opas-Hänninen 2000).
However, the existence of translator's stylistic fingerprints is less self-evident. It is inevitable
that the translator is always under strong influence of the original s/he is translating. Still, is there
something personal the translator adds to the target text? Is it possible to define this something and to
use it to identify the translator?
The aim of the research carried out at the Department of Translation Studies of the University of
Tampere is to find out whether translators also have ‘stylistic fingerprints'. The research is based on a
parallel corpus of Russian fiction texts and their translations into Finnish. We compared, on the one
hand, original Russian texts written by the same author and by different authors and, on the other
1 Recent theories of translation (and literary theories, as well) have strongly questioned the authority
of the original author (e.g. Oittinen 1995). However, the translations analysed in this research have
mostly been produced around mid-20th century and thus we find it appropriate to look at the
translations against the theoretical framework and background of their own time.
379
hand, analysed Finnish translations of different texts performed by the same translator and, in one
case, translations of the same text performed by different translators.
3. Vocabulary richness
One of the widely used methods of authorship attribution is vocabulary richness measures. D.
Holmes and R. Forsyth (1995) used the following quotients for their analysis of the Federalist Papers:
)
V
( 1
100
) 1 (
1
V
LogN
R
-
=
2
1
2 4 ) ( 10
) 2 (
N
N V i
K i i aY
= -
= ,
a V N W
- = ) 3 (
where N is the text length, V the total number of different words used in the text, V1, V2, Vi the
number of words used 1, 2, i times, a = 0.172.
The higher the number of words which were used only once (hapax legomena), the higher is the
R quotient. The more high-frequency words in the text, the higher is the K quotient. The more
different words there are in the text, the higher is the W quotient.
We used these quotients in our research. The R, K, and W values were calculated for different
original Russian texts and for Finnish translations.
Table 1. Vocabulary richness. Russian texts2
Title R K W
R1 1107.778 58.414 8.307
R2 1087.733 59.892 8.941
R3 1100.168 60.117 8.183
S1 1026.101 50.405 9.003
S2 1073.588 47.996 8.425
S3 1094.196 47.124 7.834
S4 1023.956 50.285 8.839
S5 1078.889 42.718 8.058
T1 1067.322 46.582 8.191
T2 998.055 49.245 8.638
T3 982.655 43.654 8.835
Table 1 shows that the values of R, K, and W are not identical for the same author (however, in
most cases they are quite close, e.g. cf. R1 vs. R2, S1 vs. S4). Still, texts by different authors might
sometimes have pretty close values of vocabulary richness measures, e.g. Juri Trifonov's novel Dom
na naberezhnoj (The building on the Embankment, T1) is closer to Arkadi and Boris Strugatski's
Piknik na obochine (Roadside Picnic, S2) than to other works by Trifonov (T2, T3). The Strugatskis’
works fall into two groups, S1, S4 and S1, S3, S5, which differ notably from one another (one possible
explanation might be a different degree of participation of the two co-authors). Only Rasputin's works
really demonstrate close vocabulary richness. Thus, it can be stated that the vocabulary richness of a
certain author is not a stable factor; variation between early and later works may be explained with the
fact that, quite naturally, the author's style changes all the time along with his / her taste, prejudices,
habits, and so on.
2 Texts are referred to with the initial letter of their author's (for translations also the translator's)
name (see list of texts below).
380
The same method was then used to compare translations. Most of the texts were translated by the
same person, Esa Adrian; for one of the texts — Dostoyevski's Zapiski iz podpolja (Notes from the
Underground) — we have two translations, by E. Adrian and by V. Kallama; and one text —
Lermontov's Geroj nashego vremeni (Hero of our time) — was translated by U.-L. Heino.
As demonstrated in Table 2 below, the texts translated by E. Adrian (DA, OA, R1A, SA) have
quite different R, K, and W values, which seems to indicate that the vocabulary of a translation is to a
large degree dependent on that of the original. This assumption is supported by the observation that
the vocabulary richness values of the original R1 (see Table 1) and its translation R1A are quite close
to each other (substantial deviation is found in the K-factor only, which can be attributed to the
difference between the two language systems). Another important point is that the R, K, and W values
for the two Finnish translations of Dostoyevski's story (Adrian, DA, and Kallama, DK) are almost
identical.
Table 2. Vocabulary richness. Finnish translations from Russian
Title R K W
DK 1038.76 40.03 8.54
DA 1034.74 40.94 8.48
OA 1021.40 30.17 7.91
R1A 1105.85 44.70 8.09
SA 1092.77 32.70 8.17
LH 1059.37 32.88 8.06
4. Most frequent words
Another method which could help to distinguish different authors could be a study of most
frequent words in the texts in question. Our method is based on a comparison of lemmatised word lists
(this is because people use lexemes rather than word forms). Two texts are compared by selecting the
40 most frequent words from their word lists. Then the F-index is calculated: 3 points are added for
each word with close relative frequencies, 2 points for each word with different relative frequencies, 1
point for each word with quite different frequencies, and 1 point is deduced for each word absent in
the other list. The results for the original texts were as follows:
Table 3. Most frequent words. Russian texts
Texts Indexes
R1 — R2 40
R1 — R3 53
R2 — R3 43
R1 — S 34
R2 — S 35
R3 — S 34
O — R1 31
S — O 33
It is obvious that if the texts were written by the same author, the frequent words’ lists overlap
and many words have close frequencies. One could assume that the same thing might happen if the
topic of the texts is close enough, but it doesn't. For instance, Rasputin's novels have the same topic
as Shukshin's short stories (country life); moreover, these authors belong to the same “school” of
country prosaists. However, their F-index (35 or 34) doesn't differ much from that of Shukshin vs.
Olesha (33) or Olesha vs. Rasputin (31). The table clearly indicates that if the F-index is less than 40,
the texts in question are quite likely written by different authors.
The results of the study of the translations (presented in Table 4 below) were entirely different.
381
Table 4. Most frequent words. Finnish translations
Translations F-Index
DK — DA 63
SA — R1A 44
LH — DK 32
LH — DA 29
DA — R1A 28
SA — OA 28
LH — OA 26
OA — R1A 25
It is evident that the F-index for the texts translated by the same person is high only if the topic
is close enough (Shukshin vs. Rasputin, SA — R1A). In other cases the F-index for texts translated by
the same person doesn't significantly differ from those translated by different persons. Confirming
obvious expectations, the F-index has its highest value when the two different translations of the same
text are compared.
5. Favourite words
Every person speaks his/her own idiolect, which means that a certain, more or less unique list of
favourite words can be compiled from everybody's vocabulary. Although variation is possible, no
dramatic changes can be expected. In this respect, we carried out the following experiment: word lists
of the two texts were compared against the data of a large text corpus and two lists of words with
frequencies much higher than in the corpus were generated. Then these two lists were compared and
the number of coincidences (FW-index) was calculated. The higher this FW-index is, the closer is the
language of the texts and the more probable it is that the texts were written by the same author.
Table 5. Favourite words. Russian texts
Texts FW-index
R1 — R2 385
R1 — R3 577
R2 — R3 426
R2 — S 242
R2 — O 148
O — S 124
The comparison of the translations, again, shows that the language of different translations of
the same text performed by different people is closer than that of the different translations by the same
translator.
Table 6. Favourite words. Finnish translations
Translations FW-Index
DK — DA 360
R1A — SA 74
R1A — OA 71
LH— DA 45
R1A — DA 31
R1A — DK 21
6. What is specific?
However, although the translator is a ‘chameleon’ and the language, style, and core vocabulary
of the translation depend on the author's style, we still believe that translator's style is indeed an
382
existing phenomenon. Despite the strong dependence on the original, all translators have favourite
equivalents and patterns of language usage. The analysis of Finnish equivalents for Russian modal
markers shows how different translators use different equivalents in analogous situations. Further, the
analysis also reveals some patterns of equivalent usage, i.e. certain translators being more fond of
certain words than others. This tendency is clearly presented in the different translations of the same
text, as well.
This analysis is a good example of the inadequacy of dictionaries as all-embracing guidelines for
translators and the inevitability of the translator's own choices. As indicated in Figure 1 below,
Finnish equivalents used for the Russian word kazhetsja (‘it seems (to be)') are tuntuu, taitaa, tuskin,
ehkä, ilmeisesti, luuultavasti, tietääkseni, kai, kaipa, mielestäni. The most widely recognised Russian-
Finnish dictionary gives the following equivalents: 1. näyttää; 2. tuntua; 3. kai, taitaa. It is worth
noting that the first equivalent offered in the dictionary was not used in the translations at all and, on
the other hand, the most widely used translation, ehkä, is not mentioned in the dictionary at all.
Figure 1. Finnish equivalents for kazhetsja in different translations
0,000
0,200
0,400
0,600
0,800
1,000
1,200
1,400
WX
Q
WX
X
WD
LWD
D
WX
V
N
LQ
ehkä
LOP
H
LV
H
V
WL
OX
X
OWD
Y
D
V
WL
tietääkseni
N
D
L
N
D
LS
D
mielestäni
SA
R1A
OA
DA
DK
LH
Based on this data, it can roughly be concluded that translating kazhetsja as taitaa is typical of
E. Adrian; this translation is not used by the other translators at all. Compared to the other translators,
V. Kallama seems to prefer the word mielestäni and U.-L. Heino appears to be especially fond of the
word ilmeisesti. An especially noteworthy point is that the frequencies of these words in the two
translations of the same novel (see DA and DK) are quite different; thus, their usage doesn't seem to
entirely depend on the original.
Similar conclusions can be drawn from the data on the equivalents for vse-taki (presented in
Figure 2 below). It appears that E. Adrian does not like the expression kaikesta huolimatta at all and
that kuitenkin is mostly typical of U.-L. Heino. It can also be claimed that V. Kallama's repertoire of
equivalents is the broadest: he is the only one of these translators who has used all the equivalents
analysed.
383
Figure 2. Finnish equivalents for vse-taki in different Finnish translations
0,000
0,200
0,400
0,600
0,800
1,000
1,200
VLWWH
Q
N
LQ
sittenkään
N
X
LWH
Q
N
LQ
N
X
LWH
Q
N
D
D
Q
VLOWL
N
X
P
P
LQ
N
LQ
WR
N
L
N
D
LN
H
VWD
K
X
R
OLP
D
WWD
MR
N
D
WD
S
D
X
N
VH
VVD
sentään
SA
R1A
OA
DA
DK
LH
Interesting results can be discovered by comparing word, sentence, and paragraph counts of
originals and translations. The number of words in original / number of words in translation ratio (Wquotient),
number of sentences in original / number of sentences in translation ratio (S-quotient) as
well as the ratio of number of paragraphs in original / number of paragraphs in translation (Pquotient)
are in fact stable values and depend on the pair of languages (Mikhailov 2001). However,
table 7 shows that the values of these three quotients are closer for the texts translated by the same
person. It is also evident that the two translations of Dostoyevski (DA and DK) differ in this respect.
This might be explained by the translator's attitude to the structure of the original: some translators
try to generate a text of the same length and the same structure as the source text, some believe that
good style and best possible readability in the target language is more important than fidelity to the
original.
Table 7. Words', Sentences', and Paragraphs’ ratios for Finnish translations of Russian texts.
Text W-quotient S-quotient P-quotient
R1A 1,010 0,929 0,975
OA 1,074 0,979 1,001
DA 1,113 0,860 0,975
S4A 1,069 0,953 1,084
S1A 1,073 0,947 1,081
LH 1,033 0,668 0,879
DK 1,069 0,956 0,979
7. Conclusions
As was argued in the beginning of this article, it is inevitable that a translator makes a great deal
of independent decisions during the translation process. However, when translations were analysed
with some widely used authorship attribution methods (e.g. vocabulary richness, frequent words), it
appeared as if translators didn't have a language and a style of their own. Still, every translator has a
personal set of instruments and stylistic devices. Therefore, in search of the translator's identity
(personal features), the most important indicators could be the use of modal words, particles,
conjunctions, grammar forms, etc., as well as splitting or joining sentences and paragraphs and
expanding or shortening the text.
384
List of texts
Russian original texts3
O: Olesha Ju. Zavist’ (‘Envy')
R1: Rasputin V. Zhivi i pomni (Live and remember)
R2: Rasputin V. Poslednij srok (‘The deadline')
R3: Rasputin V. Proshchanie s Materoj (Farewell to Matyora)
S: Shukshin V. Short stories.
S1: Strugatski A. & B. Paren’ iz preispodnej (‘The guy from Hell')
S2: Strugatski A. & B. Piknik na obochine (Roadside Picnic)
S3: Strugatski A. & B. Ponedel'nik nachinaetsja v subbotu (‘Monday begins on Saturday')
S4: Strugatski A. & B. Popytka k begstvu (Escape Attempt)
S5: Strugatski A. & B. Trudno byt’ bogom (Hard to be a God)
T1: Trifonov Ju. Dom na naberezhnoj (‘The Building on the Embankment')
T2: Trifonov Ju. Predvaritel'nyje itogi (‘Preliminary Results')
T3: Trifonov Ju. Obmen (‘Exchange')
Finnish translations
DA: Dostoyevski F. Zapiski iz podpolja (Notes from the Underground). Finnish title: Kirjoituksia
kellarista. Translator: E. Adrian.
DK: Dostoyevski F. Zapiski iz podpolja (Notes from the Underground). Finnish title: Kellariloukko.
Translator: V. Kallama.
LH: Lermontov M. Geroj nashego vremeni (Hero of our time). Finnish title: Aikamme sankari.
Translator: U.-L. Heino.
OA: Olesha Ju. Zavist’ (‘Envy'). Finnish title: Kateus. Translator: E. Adrian.
R1A: Rasputin V. Zhivi i pomni (Live and remember). Finnish title: Elä ja muista. Translator:
E. Adrian.
SA: Shukshin's short stories translated by E. Adrian.
S1A: Strugatski A. & B. Paren’ iz preispodnej (‘The guy from Hell'). Finnish title: Poika helvetistä.
Translator: E. Adrian.
S4A: Strugatski A. & B. Popytka k begstvu (Escape Attempt). Finnish title: Pakoyritys. Translator:
E. Adrian.
References
Baayen R, Tweedie FJ, Neijt A, Krebbers L 2000 Back to the Cave of Shadows: Stylistic Fingerprints
in Authorship Attribution. In The 12th Joint International Conference of the Association for
Literary and Linguistic Computing and the Association for Computers and the Humanities.
University of Glasgow, 21-25 July, 2000, pp. 156-158.
Chesterman A 1997. Memes of translation: the spread of ideas in translation theory. Amsterdam
Benjamins cop.
Holmes DI, Forsyth RS 1995 The Federalist revisited: new directions in authorship attribution.
Literary and Linguistic Computing 10(2): 111-129.
Mikhailov M 2001. Two Approaches to Automated Text Aligning of Parallel Texts in Fiction. Across
Languages and Cultures. (forthcoming).
Mounin G 1994. Les belles infideles. Presses universitaires de Lille.
Nida E, Taber CR 1974. The theory and practice of translation. Leiden United Bible Societies.
Oittinen R 1995. Kääntäjän karnevaali. Tampere, Tampere University Press.
Tweedie FJ, Opas-Hänninen L 2000 A comparison of methods for the attribution of authorship of
popular fiction. In The 12th Joint International Conference of the Association for Literary and
3 If no official English translation of the title was found, our own translation is used in inverted
commas.
385
Linguistic Computing and the Association for Computers and the Humanities. University of
Glasgow, 21-25 July, 2000, pp. 105-107.

386
Corpus analysis and results visualisation using self-organizing maps
Dr. H. Moisl and Dr. J. Beal
Centre for Research in Linguistics, University of Newcastle
{Hermann.Moisl, Joan.Beal}@ncl.ac.uk
This paper addresses the related issues of statistical analysis of text corpora and of intuitivelyaccessible
representation of the results of such analysis, with reference to the Newcastle-Poitiers
Electronic Corpus of Tyneside English (NPECTE) project. It proposes topographic mapping as a tool
for analysis and visualization of the NPECTE corpus, and is in three main parts. The first gives a brief
account of the NPECTE project, the second explains the nature of topographic mapping and the
motivation for its use, and the third gives an example of how topographic mapping can be implemented
using the Self-Organizing Map artificial neural network architecture.
1. The NPECTE project
The NPECTE project is based on two separate corpora of recorded speech:
(i) The earlier of the two corpora was gathered during the Tyneside Linguistic Survey (TLS) (Strang
1968, Pellowe 1972) in the late 1960s, and consists of 86 loosely-structured 30-minute interviews.
The informants were drawn from a stratified random sample of Gateshead in North-East England,
and were equally divided among various social class groupings of male and female speakers, with
young, middle, and old-aged cohorts. Some transcription and analysis was done on this material at
the time, but little of it was published, and work on it languished until 1995, when Joan Beal of
DELLS secured funding from the Catherine Cookson Foundation to salvage the original reel-toreel
tapes to audio cassette format and to catalogue and archive the cassettes. This material is now
housed in the Catherine Cookson Archive of Tyneside and Northumbrian Dialect in the
Department of English Literary and Linguistic Studies (DELLS), University of Newcastle upon
Tyne
(ii) The more recent corpus was collected in the Tyneside area in 1994 for an ESRC-funded project
‘Phonological Variation and Change in Contemporary Spoken English’ (PVC). This data is in the
form of 18 DAT tapes, each of which averages 60 minutes in length. Dyads of friends or relatives
were encouraged to converse freely with minimal interference from the fieldworker, and
informants were again equally divided between various social class groupings of male and female
speakers in young, middle, and old-age cohorts. This material is housed in the Department of
Speech, University of Newcastle upon Tyne;
Recently, an AHRB grant was awarded under the Resource Enhancement Scheme to combine the
TLS and PVC collections into a single corpus and to make it available to the research community in a
variety of formats: digitised sound, phonetic transcription, standard orthographic transcription, and
various levels of tagged text, all aligned.
2. Topographic mapping and its application to NPECTE
a) Topographic mapping
The aim of topographic mapping is to represent relationships among data items of arbitrary
dimensionality n as relative distance in some m-dimensional space, where m < n. In practice, it is used
in applications where there is a large number of high-dimensional data items, and the interrelationships
of the dimensions are not obvious: the data items are typically represented as a set of length-n realvalued
vectors V = {v1, v2…vk), and these vectors are mapped to points on a 2-dimensional surface
such that the degree of similarity among the vi is represented as relative distance among points on the
surface.
b) Motivation and application to NPECTE
Corpus analysis is often concerned to discover regularities in the interrelationships of certain
features of interest in the data – correlations of phonetic or graphemic features, for example, or of such
things as social class, age, gender and geography with aspects of linguistic usage. Cluster analysis
(Everitt 1993) has been widely and successfully used for this purpose (Manning and Schütze 1999),
and topographic mapping is in fact a variety of cluster analysis. Its chief advantage over standard
387
cluster analysis techniques is the intuitive accessibility with which analytical results can be displayed:
projection of a large, high-dimensional data set onto a two-dimensional surface gives an easilyinterpretable
spatial map of the data's structure.
With regard to the application of topographic mapping to the NCEPTE corpus in particular, the
project's aim is not only to create an electronic resource, but to make that resource the basis of
analytical research projects. We are therefore developing software tools to supplement those generally
used in corpus analysis, and topographic mapping is the first of these.
3. Implementation of topographic mapping using the SOM architecture
There are several ways of implementing topographic mapping, that is, of forming twodimensional
projections of data distributions in high-dimensional spaces: principal component analysis
(Jolliffe 1986, Everitt 1993), multidimensional scaling (Borg and Groenen 1997, Everitt 1993), and
self-organizing maps (SOM) (Kohonen 1995). This paper adopts the last of these because SOMs have
been successfully used in natural language corpus processing, and the relevant work provides a good
basis for development of the applications required for the NPECTE. This section briefly describes the
SOM architecture, then gives pointers to current applications of SOM in processing of textual corpora,
and finally presents an example of how a SOM can be used in analysis of corpora like the NPECTE.
a) SOM
The self-organizing map, also known as the Kohonen net after its inventor, is a k-dimensional
surface of processing units, where k is usually 2. Associated with each unit is a set of connections from
an input buffer such that, for a buffer of length n, there are n connections per unit (for clarity, only
sample connections are shown in Figure 1):
Figure 1: A self-organizing map.
Given a set V of input vectors of length n, such a net can, using the SOM training algorithm, learn
to approximate the similarity relations among the vi Î V in n-dimensional space on the twodimensional
surface of processing units. After training is complete, each of the vi is associated with a
specific unit uj in the sense that it activates uj more strongly than any other; when the activations for all
the vi are plotted on the unit surface, the distances among activated units represent the similarity
relations in the input vector space. Details of the network training algorithm can be found in most
textbooks on artificial neural networks (for example Haykin 1999, Rojas 1996); the standard reference
is Kohonen 1995.
b) SOM and corpus analysis
SOMs have found application in a wide range of disciplines (Kohonen 1995, chapter 7). In natural
language processing (Kohonen 1995 pp 237-249, 301; Honkela 1997), the main application to date has
been in the classification of texts in large document collections. In particular, Kohonen and his research
group have developed WEBSOM (Kohonen et al 2000, Kaski et al 1998, Lagus et al 1999), a system
that has successfully classified over one million web documents on the basis of their lexical content.
WEBSOM underlies development of the NPECTE-specific analytical tool being described here.
c) Example
Assume the existence of a phonetically-transcribed spoken corpus, like NPECTE, consisting of a
fairly large number of interviews, each labelled for region, age, gender, and social class. One is
interested in the, say, region and age distribution of phonetic segments in two environments, that is, of
segments that occur between two specific (phonetic prefix - phonetic suffix) pairs. To carry out the
analysis, the transcribed corpus is scanned for the relevant prefix-segment-suffix sequences, labelling
388
each such sequence with the regional and age information associated with the interview from which it
came. The aim is to show how a SOM can generate and display a topographic map of the structure of
such a data set. To show this data with a known structure is required; for clarity of exposition, a small,
artificially constructed data set D1 will be used.
The phonetic segment of interest comprises 5 variants, V1 - V5, distributed as shown in Figure 2:
Figure 2: The structure of the example data set D1
·  In Region 1 all age categories use variant 1 in environment 1, and variant 2 in environment 2
·  In Region 2 age categories 1 and 2 use variant 3 in environment 1 and variant 4 in environment 2,
but age category 3 uses variant 4 in both environments
·  In region 3 all age categories use variant 5 in all environments
The first step is to encode the environmental prefixes and suffixes and the segment variants for
processing by a SOM. This means some form of vector encoding. Again for clarity, a binary encoding
is adopted, where 1 and 0 represent the presence and absence respectively of a phonetic feature;
prefixes and suffixes are encoded as 3-bit and segment variants as 6-bit binary vectors:
E1 prefix: 010
E1 suffix: 101
E2 prefix: 011
E2 suffix: 110
V1: 100011
V2: 100111
V3: 010011
V4: 010111
V5:001111
The encodings are arbitrary, and are not intended to be interpretable as specific phonetic features.
The data set corresponding to the structure in Figure 2 is thus:
1. R1 A1 E1 0 1 0 1 0 0 0 1 1 1 0 1
2. R1 A1 E2 0 1 1 1 0 0 1 1 1 1 1 0
3. R1 A2 E1 0 1 0 1 0 0 0 1 1 1 0 1
4, R1 A2 E2 0 1 1 1 0 0 1 1 1 1 1 0
5. R1 A3 E1 0 1 0 1 0 0 0 1 1 1 0 1
6. R1 A3 E2 0 1 1 1 0 0 1 1 1 1 1 0
7. R2 A1 E1 0 1 0 0 1 0 0 1 1 1 0 1
8. R2 A1 E2 0 1 1 0 1 0 1 1 1 1 1 0
9. R2 A2 E1 0 1 0 0 1 0 0 1 1 1 0 1
10. R2 A2 E2 0 1 1 0 1 0 1 1 1 1 1 0
11. R2 A3 E1 0 1 0 0 1 0 1 1 1 1 0 1
12. R2 A3 E2 0 1 1 0 1 0 1 1 1 1 1 0
13. R3 A1 E1 0 1 0 0 0 0 1 1 1 1 0 1
14. R3 A1 E2 0 1 1 0 0 0 1 1 1 1 1 0
15. R3 A2 E1 0 1 0 0 0 0 1 1 1 1 0 1
16. R3 A2 E2 0 1 1 0 0 0 1 1 1 1 1 0
17. R3 A3 E1 0 1 0 0 0 0 1 1 1 1 0 1
18. R3 A3 E2 0 1 1 0 0 0 1 1 1 1 1 0
Hierarchical cluster analysis (squared Euclidean distance, average linkage) reveals the structure of
this data (Figure 3):
389
Figure 3: A hierarchical cluster analysis of D1
The two main clusters (1-3) and (4-6) correspond to environments E1 and E3, and within both of
these there is subclustering first by region and then by age. This is the structure which the SOM is
expected to discover from the data.
A SOM was trained on S1 with the following parameters:
Map axis: 9 (that is, a 9 x 9 unit layer)
Initial learning rate: 0.9
Learning rate decrement: 0.01
Learning rate decrement interval: 10 iterations
Initial neighbourhood: 9
Neighbourhood decrement interval: 40 iterations
Number of training iterations: 10000
After training the S1 vector set was presented to the net, with the following result (Figure 4):
Figure 4: The SOM's analysis of D1
The main E1 and E2 clusters are clearly separated from one another on the left and right sides of
the map. The groups in the E2 region are equidistant from one another, corresponding to the E2 subtree
in Figure 2; the distance match with the hierarchical cluster tree, where the distances among 4, 5, and 6
are slightly asymmetrical, is not exact, but the map approximates this as closely as possible given its
coarse granularity. The EI region also closely reflects the EI cluster subtree since, in both, 2 and 3 are
closer to one another than they are to 1. In addition, the cluster tree shows that the structure of group 2
is unlike that of the other groups 1 and 3-6 in that R2A3E1 differs substantially from the other two
members of the group; the map shows a corresponding distance relation.
It can, therefore, be said that the SOM gives a good 2-dimensional spatial representation of the
vector similarity relations in the data set, which itself encodes regional, age, and phonetic environment
variation in our hypothetical corpus.
390
Now, the hierarchical cluster tree is easily as clear about the structure of the data as the SOM.
What, therefore, is the advantage of SOMs over established cluster analysis methods like hierarchical
analysis in corpus work? The answer is that, as data sets grow larger and their structure more complex,
existing hierarchical methods become increasingly difficult to interpret, whereas SOMs remain clear.
Consider, for example, another artificial data set D2 of 1000 length-24 real-valued vectors. These were
generated by a process which subdivided them into 5 main groups, numbered (0-200), (201-500), (501-
800), (801-950), and (951-1000). Each was then given some subsidiary structure, and finally noise was
injected into the whole set by randomly selecting two components of each vector and incrementing the
values found there by a small random amount. Figures 5 and 6 show the results of hierarchical cluster
analysis (Euclidean distance, single linkage) and SOM analysis respectively.
Figure 5: A hierarchical cluster analysis of D2
Figure 6: A SOM analysis of D2
Both are clear with respect to the main structure of the data, but the subsidiary structure within the
main clusters is much clearer in Figure 6. Both hide some structural information. In Figure 5 it is
almost entirely opaque and difficult if not impossible to comprehend. In Figure 6 not all 1000 vectors
391
are present on the map: many vectors are mapped to a single unit and, since only one vector label can
be represented at any given location, only one vector location can be displayed. Thus, the upper righthand
corner of the SOM represents vectors 1-200, but only a minority of these is visible. The solution
is both cases is to implement a graphical interface that permits interactive browsing of the structure
display. For Figure 5 this would reveal ever more detailed subtrees, but, as data sets grow larger, such
zooming-in soon make it difficult to see the selected subtree region in relation to the structure tree as a
whole. For Figure 6 there are at least two possibilities. On the one hand, selection of a given unit could
display a list of vectors associated with that unit in a way that allows the analyst to maintain a clear
view its place in the overall structure map, as in Figure 6a. On the other, and more interestingly, one
could use a hierarchical feature map (Merkl 2000) in which each unit of the main SOM has its own
SOM associated with it, allowing the structure of the vectors mapped to the node of interest to be
displayed, as in Figure 6b.
Conclusion
Topographic mapping is a nonhierarchical clustering technique that can project high-dimensional
data sets onto low, usually two-dimensional surfaces such that the similarity relations of the data are
represented as spatial distribution of points on the surface. In relation to text corpus analysis, its main
advantage over standard hierarchical cluster analysis methods for the purpose is in the intuitive clarity
of results visualization as a spatial structure map, and the scope of that visualization for interactive
exploration of the map. The aim is to develop a SOM-based implementation of a topographic mapping
tool for analysis of the NPECTE corpus.
References
Borg I, Groenen P 1997 Modern Multidimensional Scaling - Theory and Applications. Springer.
Everitt B 1993 Cluster Analysis, 3rd ed. E. Arnold.
Haykin S 1999 Neural Networks. A Comprehensive Foundation. Prentice Hall International.
Honkela T 1997 Self-Organizing Maps in Natural Language Processing. PhD thesis, Helsinki
University of Technology, Espoo, Finland.
Jolliffe I 1986 Principal Component Analysis. Springer.
Kaski S, Honkela T, Lagus K, Kohonen T 1998 WEBSOM--self-organizing maps of document
collections. Neurocomputing 21: 101-117.
Kohonen T 1995 Self-Organizing Maps, 2nd ed. Springer.
Kohonen T, Kaski S, Lagus K, Salojärvi J, Paatero V, Saarela A 2000 Self Organization of a Massive
Document Collection. IEEE Transactions on Neural Networks 11(3): 574-585.
Lagus K, Honkela T, Kaski S, Kohonen T 1999 WEBSOM for textual data mining. Artificial
Intelligence Review 13(5/6): 345-364.
Manning C, Schütze H 1999 Foundations of Statistical Natural Language Processing. MIT Press.
Merkl, D Text data mining. In Dale R, Moisl H, Somers H (eds), Handbook of Natural Language
Processing, Dekker: 889-903
Pellowe J et al 1972 A dynamic modeling of linguistic variation: the urban (Tyneside) linguistic
survey. Lingua 30: 1-30.
Rojas R 1996 Neural Networks. A Systematic Introduction. Springer.
Strang B 1968 The Tyneside Linguistic Survey, Zeitschrift für Mundartforschung, Neue Folge 4: 788-
94.

392
Pragmatic and discursive aspects of German modal particles:
a corpus-based approach
Martina Möllering
Department of European Languages, Macquarie University, Sydney
Modal particles fulfil important pragmatic and discursive functions in German. Their meaning is
complex and highly dependent on linguistic as well as situational features of the context. Following the
premise that German modal particles occur with greater frequency in the spoken language, the paper
reports on an analysis which is based on corpora representing spoken German. The concept of ‘spoken
language’ is discussed critically with regard to the corpora chosen for analysis and narrowed down
concerning the use of modal particles.
The analysis is based on the following corpora: Freiburger Korpus, Dialogstrukturenkorpus and
Pfeffer-Korpus, which are all kept at the “Institute for the German Language” (Institut für deutsche
Sprache) and can be accessed via the institute's on-line system COSMAS. In addition, a collection of
telephone conversations (Brons-Albert 1984) was scanned into computer readable files and analysed
with MicroConcord (Scott and Johns 1993). A quantitative analysis was carried out on all corpora; the
qualitative analysis was limited to the telephone conversations.
With regard to these analyses, the paper discusses:
·  the realization of the concept of ‘spoken language’ in the corpora under discussion
·  the limitations of computer-based analysis for the language feature investigated
·  collocational patterns which help to identify pragmatic and discursive functions of modal particles
References
Brons-Albert R 1984 Gesprochenes Standarddeutsch: Telefondialoge. Tübingen:Günter Narr
Institut für deutsche Sprache 1999 COSMAS. http://www.ids-mannheim.de/kt/cosmas.html
Scott M, Johns T 1993 MicroConcord. Oxford: Oxford University Press.

393
Evidence of Australian cultural identity through the analysis of Australian and
British corpora
Rachel Muntz
Department of Linguistics, University of Wales, Bangor
1. Abstract
This paper reports on findings of a keywords analysis comparing the ACE corpus of written
Australian English (AusE) and the Flob corpus of written British English (BrE). The main aims of the
study are to provide evidence of the cultural representativeness of ACE and to identify significant
features of lexical frequency associated with written AusE which are distinct from written BrE. This
paper shows how the physical environment has helped shape the Australian lexicon. It also
demonstrates clear differences in the use of colour and personal reference terms. Although this study
has discovered many lexical features which confirm that Australia's identity is very separate from that
of Britain's, it is demonstrated that the British influence on Australian English and culture is still
pervasive.
2. Background
2.1 Setting the context of this study: Australia's cultural and linguistic identity
On November 6, 1999 the people of Australia voted narrowly against creating an Australian
republic. They voted to retain Britain's queen as its head of state, and not to replace her with an
Australian citizen. However, the referendum was highly politically charged and the no vote was carried
more due to factors such as fears that the proposed constitutional changes might induce political and
economic instability, and rather less due to Australians feeling culturally tied to Britain (Stephens,
1999). The referendum was a defining point in several decades of debate on Australia's cultural
identity. During the late 1980s and early 1990s (the time period ACE and Flob represent) former prime
minister Paul Keating vigorously promoted such debate, encouraging Australians to reflect on the
question “What does it mean to be Australian?” (Snow, 1999). Elements of this coming-of-age
discourse included the growing awareness of Australia's economic role and geographical position
within the Asia-Pacific region; greater acknowledgement of the breadth of the population's ethnicity
due to worldwide immigration; and increased recognition of indigenous communities. In discussing
who and what it was, Australia was also coming to terms with who and what it wasn't anymore:
Britain.
Obviously, one of the greatest, enduring influences the British had on Australia was linguistic
(Gorlach, 1991). In the 200 years plus since English was transported to Australia, first with the convicts
and administration, and then with the free settlers, AusE and BrE have developed in their own unique
directions. Of course, the most striking difference now is phonological (Trudgill & Hannah, 1985). An
Australian accent, indeed any accent, is a marker not only of linguistic identity, but of cultural identity
too. It marks the speaker as a member of a speech community. That is, we can tell from this accent that
the person shares common ground in the form of knowledge, beliefs and assumptions with the other
members of that community - the community here being at the level of the nation (Clark, 1996). But
what about subtler linguistic features, at syntactic and lexical levels? Do they also provide clear signals
as to the cultural identity of the speaker? This paper seeks to answer this question by analysing a very
specific characteristic at the lexical level: frequency of word use in written AusE, as compared to
written BrE. Such research is valuable, not only because it adds breadth and depth to the literature on
linguistics and corpus linguistics, but also because it can contribute to a young country's search for
self-knowledge and sense of identity.
2.2 Previous studies and theories of culture and linguistics
For many years now linguists have been investigating links between culture and language. Any
discussion of this subject inevitably refers to Benjamin Lee Whorf and the theories of linguistic
relativity and linguistic determinism (Gumperz & Levinson, 1996). Although these theories are focused
on the binary factor of the presence versus absence of a word for a particular concept in different
languages, here they are applied to information on the frequency of use of a word within different
varieties of the same language. The basic premise of linguistic relativity is that our thoughts and points
of view are influenced by the language we speak due to the way the language is syntactically and
semantically structured. Linguistic determinism is a more radical and less popular theory which claims
that the language we speak dictates the way we think about our physical and social world (Clark,
1996). Clark expands the linguistic relativity theory to include not merely major language
394
communities, but “any cultural community that corresponds to people's social identities” (1996: 353).
So applying the theory of linguistic relativity to a cross-cultural study of two varieties of English which
exist in very different parts of the physical world seems a good way to test this theory. Could the
English that the convicts and settlers took with them to Australia cope with the demands of a new
physical environment? Did it need to change as some words became less useful and others more
useful? Undoubtedly it did, and this paper seeks to uncover the nature of these changes.
On the subject of Australian culture, work by Wierzbicka (1992) has provided interesting
linguistic insights into its nature by examining Australians’ fondness for using diminutive forms of
nouns and proper nouns. Of course, this is a not just an academic observation. The Lonely Planet guide
to Australia recommends that “if you want to pass for a native try speaking slightly nasally, shortening
any word of more than two syllables and then adding a vowel to the end of it, making anything you can
into a diminutive” (1998: 56). Seriously, though, introspection and consultation of Wierzbicka's work
and cultural guidelines found in travel guides provided a starting point for identifying semantic
categories which might be present in ACE, such as struggle/adversity and mateship.
However, the true beginning point for this paper was a study by Leech and Fallon entitled
Computer corpora – What do they tell us about culture? (1992). This paper was the first systematic
attempt to use computer corpora as a source of cultural information. They investigated significant
differences in word frequencies between American and British English, based on Hofland and
Johanssen's (1982) published frequency tables, and attempted to identify which differences were
attributable to cultural difference. A particular strength of their paper was its frank discussion of the
limitations of corpus research and clear description of the goals of their study, which excluded
analysing what they termed ‘linguistic contrasts’ which included differences in the two language
varieties due to spelling conventions or ‘lexical’ differences where the two varieties simply, through
convention, use different lexemes to denote the same thing. Examples of such a lexical linguistic
difference between Australian and BrE would be the British crisps and lorry whose Australian
counterpart terms are chips and truck. Whilst interesting to note, and whist unique to a particular
community's lexicon, these are superficial differences in convention not indications of one
community's need to refer to a something more often than another's. “Words evolve in a community in
direct response to their usefulness and usability in that community” (Clark, 1996: 341). And corpus
frequency data can tell us whether the same word is more useable in a certain community and culture.
These are Fallon and Leech's ‘non-linguistic contrasts', those which cannot be explained by linguistic
code or variety. Excluding proper nouns, Leech and Fallon grouped their results into the following 15
domain categories which occurred naturally within their dataset: sport; transport and travel;
administration and politics; social hierarchy; military and violence; law and crime; business; mass
media; science and technology; education; arts; religion; personal reference; abstract concepts; and ifs,
buts and modality. Their generalised conclusion on the cultural differences evidenced in American and
British English were that American culture was…
…masculine to the point of machismo, militaristic, dynamic and actuated by high ideals, driven by
technology, activity and enterprise – contrasting with one of British culture as more given to
temporizing and talking, to benefiting from wealth rather than creating it, and to family and emotional
life, less actuated by matters of substance than by considerations of outward status. (Leech & Fallon,
1992: 44-45).
2.3 The present study: cultural differences in AusE and BrE
To a certain extent this paper is a replication of Leech and Fallon's study, but with the following
important differences:
1. This study contrasts different language varieties - British and Australian English
2. This study analyses language from a different time period - the late 1980s/early 1990s
3. The minimum significance level used in this study was p=0.01, meaning its results are of a higher
statistical significance than those of Leech and Fallon's study which used p=0.05
4. This study has developed its own more suitable domain categories
5. This study includes proper nouns referring to place names, as it was felt they could provide
information on a country's place within the world, which, in turn, contributes to its identity
6. This study includes spelling conventions as they provide evidence of cultural choices and language
variation in progress
7. Increased effort has been made in the present study to verify concordances of results, making it
less likely that polysemous words and biased dispersion have compromised results
8. This study is more asymmetrical– it focuses on observations about the Australian variety of
English as compared with BrE, rather than the other way round.
395
2.4 About the corpora
The study is a near-synchronic study of contemporary Australian and British Englishes. The
instruments used are the Australian Corpus of (written) English (ACE), and the Freiburg Lancaster-
Oslo/Bergen (Flob) corpus of British written English. The two corpora are comparable in size, date,
composition and parentage (Peters & Smith, online; Hundt et al, online). Both contain around 1 million
words from 500 texts (around 2000 words per text). ACE consists of texts published in 1986, whilst
Flob's texts come from 1991. Both were modelled on the LOB (Lancaster-Oslo/Bergen) corpus which
represented the BrE of 1961. The compilers of both Flob and ACE used random selection methods to
select their component texts. These commonalties make the two corpora highly comparable and this
contributes to the robustness of results. However, perfect replication of content was not possible due to
shortages of Australian publications in certain fiction genres, namely mystery/detective fiction
(category L), science fiction (category M), romance (category P) and western and adventure fiction
(category N). The shortfall in these areas was compensated for by the inclusion of human relationship
and fantasy genres in the romance category, bush fiction in the western and adventure fiction genre and
through the addition of two additional sections: historical fiction (S) and women's fiction (W), for
which there are no corresponding categories in Flob. It is noted throughout this paper where it is felt
these differences may have skewed the study's results. The categories and corresponding number of
texts of each corpus are described in the table below.
Table 1: Text composition of ACE and Flob
Number of texts
Category ACE Flob
A: Press: Reportage 44 44
B: Press: Editorial 27 27
C: Press: Review 17 17
D: Religion 17 17
E: Skills, trades and hobbies 38 38
F: Popular Lore 44 44
G: Belles lettres, biographies, essays 77 77
H: Miscellaneous (e.g. government
and industry reports)
30 30
J: Science 80 80
K: General Fiction 29 29
L: Mystery and detective fiction 15 24
M: Science fiction 7 6
N: Adventure and western 8
(including bush fiction)
29
P: Romance and love story 15
(including fantasy and human
relationship fiction)
29
R: Humour 15 9
S. Historical fiction 22 -
W: Women's fiction 15 -
Total texts: 500 500
3. Objectives
The main objectives of this study of frequency data are:
1. To assess ACE's representativeness of AusE and Australian culture
2. To identify some key semantic domains and lexemes within these domains which occur more in
Australian written English than in British written English
3. To provide evidence that Australia's cultural comparator is still Britain
4. Method
Using the computer software program Wordsmith, a keywords or keyness analysis was performed
by comparing the entire frequency wordlists of both the ACE and Flob corpora. A log-likelihood test
was applied to the data, using a p-value of L 0.01. The default on the number of results returned by the
program was suspended (minimum frequency = 1; database minimum frequency = 1; associate
minimum frequency =5). This yielded a total number of 5323 results (2486 results for the Flob corpus;
2837 results for the ACE corpus). These were the words which occurred significantly more often in one
396
corpus than in the other. The fact that more results occurred in the ACE corpus is partly due to the fact
that due to different encoding of the two corpora - parts of hyphenated words were counted in the ACE
corpus as separate entries. Although this could have been avoided by changing the Wordsmith settings
at the outset, these results were filtered out manually in the next stage of analysis. This was a mistake
in the methodology and may have compromised a few of the results, but in the overall analysis, it is not
expected to have a significant impact.
The next stage of the analysis was to break down the long list of results into categories. These
categories were at first based on those that Fallon and Leech found useful in their analysis. However, it
soon became apparent that a list more appropriate to this data was necessary. Some categories were
retained, some were dropped and some new categories were added. The 11 categories focused on in
this study are as follows:
1. place names: proper nouns referring to other countries
2. origin/ethnicity adjectives*
3. multiculturalism, prestige forms and borrowings into English*
4. American spelling conventions*
5. colour terms*
6. geography *
7. housing and communities*
8. fauna and flora*
9. weather and clothing *
10. personal references *
11. abstract concepts: challenge and adversity
More categories were created in the analysis, but as the focus of the project narrowed, only these
categories were chosen for further analysis. For each of the categories marked with an asterisk,
concordances and dispersion patterns were checked. In instances where the word occurred more than 3
times in the same text, only 3 occurrences were attributed. This prevented skewing of results by sifting
out data which was not normally distributed. Unfortunately, due to time constraints, it was not possible
to check the distribution of the other categories and, as such, caution should be taken in interpreting
these results. At this stage of the analysis each entry is checked for instances of polysemy. Any
meanings which fall outside the target semantic domain were also discounted. This type of analysis is
more rigid (and more time consuming!) than that used by Fallon and Leech and ensures more robust
results. By eliminating unsuitable results, the numerical data given by the computer in the first instance
on the keyness of each word was altered and recalculated manually using Johannsen and Hofman's
frequency tables as a guide to determine the significance of the new result.
5. Results
5.1 Place names: proper nouns referring to other countries
ACE and Flob mention certain countries significantly more often than the other. Each writes
about itself the most and then about its near geographical neighbours. ACE mentions Asia-Pacific
countries such as New Zealand, Papua New Guinea, Timor, Malaya and Vietnam and Flob mentions
European countries such as Ireland, France, Portugal, and Germany. Political events at the times the
corpora were compiled have impacted on these results. For example, Iran, Iraq and Bosnia are due to
this.
5.2 Origin/ethnicity adjectives
This trend also holds true for adjectives such as British and Australian. Aboriginal people are
represented in ACE's results. Also present is the diminutive Aussie, evidence that the custom of
abbreviating words is so entrenched that it is strong enough to occur in the written language, not just
the spoken language. Interestingly, like the proper nouns results, Flob contains many more
origin/ethnicity adjectives than ACE. These observations on place names and origin, while fairly
straightforward, provide evidence of the representativeness of the two corpora - results are as expected,
with each country writing about itself more than all others.
5.3 Multiculturalism, prestige forms and borrowings into English
Other related terms which occurred significantly more often in ACE included multicultural,
migrants, migration. These seem relevant to Australia's increasing awareness, in the media at least, of
itself as a multicultural country made up of migrants. However, as the media input into ACE is
substantial, there is potential overrepresentation of the word multiculturalism in particular, as it has
been something of a buzzword since the Keating years (late 80s, early 90s). Evidence in ACE of
multiculturalism's influence on AusE is negligible. There is no trend of the Italian, Greek or Chinese
397
languages, which came to Australia with large numbers of speakers, making any inroads into written
English. This is in contrast to the British corpus, which shows the influence of the French language on
its English. Several French words are significant in the Flob corpus: le, dans, qui, tout, elle. But it
appears that this tendency to drop in quelques mots du français is a habit that Australians do not
practice. The reason for their inclusion in BrE is probably due to French words being considered
prestige forms in Britain. If this is not the case in Australia, then what are Australia's equivalent
prestige forms? It is important to note that French is not an immigrant language in Britain. Following
this reasoning, perhaps it can be predicted that Italian, Greek and Chinese are not likely to be sources of
prestige borrowing in AusE in the future.
5.4 American spelling conventions
One form of borrowing which Australians seem to be incorporating into their writing is American
spelling of words that the British spell with containing ‘ou', such as color, humor, harbor, behavior,
neighborhood . The ACE results contain 11 such lexemes, whilst Flob yielded one American spelling
of a different kind: center. Whilst the majority of Australians prefer the British spelling, it will be
interesting to observe if this is in fact evidence of language variation in progress and whether in 50
years time the old-world ’u’ in such words will be obsolete in AusE, as has happened in the new-world
English of America.
5.5 Colour terms
One of the most unexpected findings of this study was the discovery of the significance of colour
terms in ACE. In descending order of frequency, the following colours occurred significantly more
often in ACE: white, red, blue, green, yellow, brown, pink, orange, beige. Intriguingly, there were no
colours of particular British significance. This list from ACE includes eight of Berlin and Kay's (1991)
11 basic colour groups, plus a supplementary colour – beige. The absence of the other three groups
(black, purple and grey) means these are used with broadly equal frequency in both Australian written
English and British written English. In addition to this, the word colours itself was a significant result
in ACE. How can these convincing results be explained? Berlin and Kay's work was used to disprove
the strong form of the linguistic relativity theory and to support the existence of linguistic universals.
But they were only concerned with the existence, not the frequency of terms. Here we have clear
evidence of one linguistic community finding use for colour terms far more often than another. From
this data, we can say with some certainty that Australians feel the need to classify things in terms of
colour to a much greater extent than their British counterparts. One possible explanation is geophysical,
due to the difference in the quantity or quality of natural light in Australia. But this is only a theory
based on introspection and a similar theory by Van Wijk (cited in Berlin & Kay, 1991) was rejected by
Berlin and Kay. Further analysis of the concordances of these colour terms would reveal the reasons
behind this result.
5.6 Geography
This section centres around physical and geographical features. Obviously, there are great
differences between Australia and Britain in this respect and this is borne out in the data. However, the
results show that Australians write about these features much more, evidence that such things are more
central to the Australian psyche, e.g. land, landscape, rocks, hill(s), mountains, soil, parks, ground.
This compares to Flob's few entries such as meadow, woodlands, moor, terms, which, in any case,
border on being linguistic rather than non-linguistic contrasts. Here it must be noted that the differences
in the component texts of the two corpora may have had an effect on the results. The inclusion of bush
fiction in ACE must be taken into account, but equally, Flob contains western fiction which ACE does
not. Words pertaining to the coast were present in roughly equal numbers in both corpora, though
ACE's results were related to the beach and surf, whilst Flob's included items of a maritime theme:
buoy, moorings. In a largely arid country such as Australia, the importance of sources of water is also
evident. Creek, drainage, swamp, waterhole, lagoon and dam are all present in the ACE results, whilst
Flob yields only reservoir. An interesting avenue to investigate would be to see if there is any link
between ACE's plethora of geographical terms to the colour terms which were so prevalent.
5.7 Housing and communities
We are also able to see how the two countries arrange their territory. Whereas Australia has
blocks of land, with the ubiquitous shed in the backyard, in Britain there exist estates (both of the
housing estate and country estate variety), manors, castles and mews. People live in villages in Britain,
whereas in Australia there are towns and remote townships. However, in both countries, most people
live in urban areas and this is a big theme with cities, suburb, suburbs and suburban all present in
398
ACE. Difference in housing design is also evident, with Australian dwellings tending to have a veranda
(h).
5.8 Fauna and flora
In a similar vein to the geography category, large categories for flora and fauna appeared in ACE,
but not in Flob. Surprisingly, there was only one entry in the Flob column (dolphin) against 59 in ACE.
The ACE results consisted mainly of native animals and plants, such as dingo, kangaroo, gum, wattle,
which of course would not be expected to be found in Flob. But Britain has its own native animals and
plants which are not found in Australia, e.g. squirrel, yet these are absent from the results. Both
countries have many kinds of animals in common, such as foxes, rats, the bird and cat. But these are
more significant to AusE and there are no corresponding entries for Flob. Generic terms such as tree,
leaf, plants, ferns, grass are all significant in ACE, with no counterparts in Flob. Another interesting
result is croc which, as an abbreviated form, shows the common Australian practice of shortening
words. As with the term Aussie, the fact that croc appears in the written form means it is firmly
entrenched in the Australian lexicon.
5.9 Weather and clothing
Not surprisingly, these domains show a trend of rain and cold weather words in Flob e.g.
precipitation, freezing, against Australia's heat and high winds, e.g. sun, hot, cyclone, wind.
Corresponding clothing for each kind of weather was also noted: shorts and bikini against suits, boot,
and cloak.
5.10 Personal reference
This category yielded some interesting results. A preference for third person singular personal
pronouns and possessives is evident in Flob with he, him, himself, his, she, her, herself, it, itself all
more significant. In ACE, first person singular and first and third person plural pronouns are present: I,
my, myself, us, our and they. A complementary result was found for personal reference nouns. It seems
where BrE prefers the third person pronoun, Australians choose from a number of nouns: men,
bloke(s), boys, father(s), husband, or the idiomatic mate(s). In the same category, Flob yields just
gentlemen and sir. Nouns referring to women in ACE include woman, women, women's, girls, mum,
daughters. Feminists and wimmin are certainly due to the presence of women's fiction texts in ACE
and it is also a possibility that the results referring to men may be influenced by this, as well as those
for children which include kid(s), teenagers, and toddlers. Reflecting the phonological changes in
AusE, Australians have begun to write meself and ya, mainly in reported speech.
5.11 Abstract concepts: challenge and adversity
One small, uniquely Australian category emerged from the data - that of struggle and adversity. If
you come from the wrong side of the tracks, you “grew up in Struggletown”. The figure of the “Aussie
battler” still looms large in the national psyche. Even if you come from the right side of the tracks,
“life's not meant to be easy”. Struggles, tough, battler, conflict, sweat, feat, overcoming are results
form ACE which reflect this.
6. Discussion and conclusion
6.1 Representativeness of ACE
This study has found that ACE is highly representative of Australian written English, as
anticipated results, such as certain proper nouns, were present. The fact that the text make-up of ACE is
different to that of Flob needed to be borne in mind, but was not found to have a great influence on
results overall.
6.2 Significant semantic domains
Overall then, what lexical frequency evidence have these corpora provided us with which prove
Australia's cultural identity is significantly different from Britain's? Results from 11 significant
semantic domains were identified and discussed. Because of the focus of this paper, most of these
domains deal with lexemes which are more significant to AusE than to BrE. Several of the domains are
related to geography or the physical environment (categories, 1, 2, 6, 7, 8 & 9) and it seems that
members of the Australian speech community have more use for such terms than the members of the
British speech community. This shows how the language, as part of the people, has adapted to a new
physical environment. Evidence of multiculturalism was found in ACE, although for a country which
talks up this aspect of its identity, there are not the corresponding number of origin/ethnicity adjectives
(category 2) which would support this image. Category 1 confirmed that Australia is very well aware of
399
its location in the Asia Pacific region. Certain BrE prestige forms were shown not to have any AusE
equivalents – potential verification of what is known as the ‘tall poppy syndrome’ in Australia, where
anyone acting above their station is quickly mown down. This notion goes hand in hand with the ethos
of struggle and adversity. The best finds of this study, though, are those relating to the colour and
personal reference terms. Clear trends were found in the different cultural uses for these terms and
further study of concordances would reveal the reasons behind these trends.
6.3 Cultural comparitor
Despite all this linguistic evidence of Australia's independent identity, this study provides
evidence that Britain may still be Australia's everyday cultural comparator. The results closely reflect
some of the most commonly observed facts and stereotypes about Australia and its culture: the size of
its coastline, barren interior, native animals and hot weather. Whilst this assures us of the validity of the
corpora, it also tells us that Australians have inherited a way of thinking about such things from Britain.
Australia does have a massive coastline, but only in comparison with Britain, not in comparison with
the United States. It is a dry country in comparison with Britain, but not compared with Middle Eastern
countries. Australians and Britons think of it as a hot country, but the Aborigines probably didn't think
in such terms, etc. If the ACE corpus had been compared with its American counterpart, Frown, for
instance, then presumably, another set of significant words would have been yielded. But I suspect that
these results would not have matched so closely with observations by social commentators about
characteristics of Australian culture as the Flob results, partly because Australia has shared physical
similarities (size of country, melting pot population, indigenous inhabitants). So although AusE has
many of its own lexical frequency characteristics, at a deeper level – in the way Australians think –
perhaps they are not so different from the British. In Whorfian terms then, perhaps the fact that
Australians speak a type of English dictates the way Australians think about their physical
environment, if not the frequency with which they use those words to describe it.
7. References
Berlin B, Kay P 1991 Basic color terms: their universality and evolution. Berkeley, University of
California Press.
Clark HH 1996 Communities, commonalities and communication. In Gumperz JJ, Levinson SC (eds),
Rethinking linguistic relativity. Cambridge, Cambridge University Press: 324-355.
Gorlach M 1991 Englishes: studies in varieties of English 1984 -1988. Amsterdam, John Benjamins.
Gumperz JJ, Levinson SC (eds) 1996 Rethinking linguistic relativity. Cambridge, Cambridge
University Press.
Hofland K, Johansson S 1982 Word frequencies in British and American English. Harlow, Longman.
Hundt M, Sand A, Siemund R Manual of information to accompany the Freiburg - LOB corpus of
British English (‘FLOB'). Accessed 25/1/01 at:
http://khnt.hit.uib.no/icame/manuals/flob/INDEX.HTM
Lonely Planet 1998 Australia. Melbourne, Lonely Planet Publications.
Leech G, Fallon R 1992 Computer corpora – what do they tell us about culture? ICAME Journal 16:
29-50.
Peters P, Smith A Manual of information to accompany the Australian corpus of English (ACE).
Accessed 25/1/01 at: http://www.hd.uib.no/icame/ace/aceman.htm
Snow, D 13 November 1999 When no means yes. Sydney Morning Herald Accessed 20/01/01 online
at: http://www.smh.com.au/news/review/9911/13/review4.html
Stephens, T 13 November 1999 The great divide. Sydney Morning Herald Accessed 20/01/01 online at:
http://www.smh.com.au/news/review/9911/13/review6.html
Trudgill P, Hannah J 1985 International English: a guide to varieties of standard English (second
edition) London, Edward Arnold.
Wierzbicka A 1992 Semantics, culture, and cognition: universal human concepts in culture-specific
configuration. Oxford, Oxford University Press.

400
Investigating characteristic lexical distributions and grammatical patterning
in Swedish texts translated from English
P-O Nilsson
English Department
Göteborg University
Box 200, SE 405 30 Göteborg, Sweden.
Tel. +46(0)31-773 5134. Fax +46(0)31 773 4726
per-ola.nilsson@eng.gu.se
This paper is a work-in-progress report about a study within the field of corpus-based descriptive translation
studies. It outlines a contrastive investigation in progress using the English-Swedish Parallel Corpus (ESPC), a
combined comparable and parallel corpus of English and Swedish original and translated fiction and non-fiction
texts. The paper discusses some initial results of corpus investigations.
The investigation involves studying typical distributions of lexical items and patterns in original and translated
texts in the corpus. An example of typical lexical distribution in the fiction part of the ESPC is represented by the
very frequent Swedish grammatical word av (‘of', ‘by’ etc), which is one and a half times as common in
Swedish texts translated from English as in original Swedish texts.
In the investigation, this characteristic lexical distribution is used as a starting point for the search for patterns
with specific distribution in translated texts: The collocates of the over-represented word are used to give a
picture of the constructions of which the word is a part, with the aim of describing possible patterns with specific
distribution in translated Swedish texts.
The fact that the ESPC is not only a comparable corpus but also an aligned parallel corpus makes it possible to
then track the sources of target text constructions back to the original texts. The final step of the investigation is
to describe the paradigm of correspondences between constructions involving characteristically distributed
constructions in translations and their actual sources in the original texts.

401
Using feature structures as a unifying representation format for corpora
exploration
Julien Nioche
LexiQuest & Université Paris X
Nanterre
julien.nioche@lexiquest.fr
Benoît Habert
LIMSI-CNRS & Université Paris X
Nanterre
habert@limsi.fr
Abstract
In this paper we report on the use of feature structures to represent the linguistic information
of a corpus. This approach has been adopted in TyPTex, a project which aims at providing a
generic architecture for corpora profiling. After a brief overview of the Typtex project, we
show that corpora exploration requires manipulating linguistic features in order to obtain a
required level of linguistic information or changing the set of features to get a new point of
view on the data. We show that feature structures formalism can help the building and
management of linguistic features with Meta-Rules based on unification. Finally, we provide
an example of marking which uses a mixed approach between projection of information
from a static lexicon and contextual marking via Meta-Rules. Results tend to show that the
use of feature structures can improve the coverage and reliability of the marking.
1 Introduction
Huge tagged or parsed corpora are used in a broad number of language-related studies (McEnery
and Wilson, 1996), with very different goals, such as lexical acquisition, speech processing, language
learning or discourse analysis. The common point of these works is the annotation of these corpora
according to linguistic features. Those features may be of various kind, going from simple
morphological or lexical information to more complex grammatical, semantic, functional or phonetic
features. Global information (like the number of words, the average word length, etc.) can be also used
for corpora processing. Some kinds of features can be easily identified in texts, such as morphological
information which can be obtained with simple n-gram computing, or lexical features which requires
only word segmentation. On the other hand, more complex tools or resources may be necessary to get
the awaited level of information. This is the case of syntactic analysis, obtained with a parser, and of
semantic features, which require dictionaries, or anaphora resolution.
Depending on the goal of the study, the set of features observed in corpora changes. For example,
readability measures consider the length of words and their average frequency compared to frequencies
from a reference corpus, while stylometry studies, the stylistic analysis of texts for the purpose of
author attribution, require other kind of linguistic information, like lexical features, syntactic
constructions or textual organization. Therefore the choice of features is task dependent. Some corpusbased
studies need to combine different kind of features at the same time. This is typically the case of
distinguishing among language registers (Biber 1993, 1995) and style (Tambouratzis et al., 2000). For
example, Tambouratzis et al. (2000) combine morphological, lexical, grammatical and structural
features to bring out style differences within a Greek corpus. Sets of heterogeneous features are thus
put together in order to compare subparts of a text collection. This kind of studies concerns corpora
exploration, since the choice of a feature set changes the results obtained and reveals different aspects
of the data. Thus linguistic features must be manipulated easily, which raises the issue of their
representation.
A lot of formalisms are used to handle linguistic annotation1. They are nowadays mostly based on
manipulation of XML/SGML entities. Tools are also provided in order to create, search or browse
corpora within these formalisms. The lack of standards to represent and manipulate linguistic
information is a problem for Natural Language Processing, since processing corpora out of their
foreseen use or in combination with textual resources in a different format requires an extensive work
to build conversion and/or manipulation tools. Attempts are made to solve this problem. The
AMALGAM (Atwell et al., 1994) project aimed at developing methods of automatically mapping
between the annotation schemes of the most widely known corpora, for both POS tag sets and phrase
structure grammar schemes, to improve the reusability of the data. More recently, Bird et al. (2001)
propose a formal framework for linguistic annotation, in the context of Speech Processing. This
1 See the Linguistic Annotation Page at www.ldc.upenn.edu./annotation.
402
framework aims at providing a core representation format, which regroups the common features of
existing annotation schemes. In the domain of morphosyntactic annotation, the EAGLES project
recommends a common formalism2, where information is represented by the position of characters in a
tag. There is an obligatory position for the POS features, which has a closed set of possible values.
Other linguistic features can be freely encoded with a symbol within the tag.
This paper explores the benefits that feature structures offer for the representation and
manipulation of linguistic information in corpora. Experience on corpora profiling in the TyPTex
project shows that the use of this format helps to handle information in a clean way, to manipulate and
modify sets of linguistic features, and improves reusability of both data and experiences. Using feature
structures can also improve the marking of linguistic phenomena.
2 Overview of the TyPTex project
The goal of the TyPTex project is to provide a generic architecture for corpora profiling. This
project is financed by ELRA (European Language Resources Association) and is carried out jointly at
LIMSI and UMR 85033. Work within this project has been previously described by Illouz et al. (2000)
and Folch et al. (2000).
2.1 Background
The underlying idea is that the reliability of the knowledge acquired form a corpus depends on the
homogeneity of its data and is decreased by its heterogeneity. In the domain of morphosyntactic
tagging, Biber (1993: 223) used the LOB (Lancaster-Oslo-Bergen) corpus to show that the probability
of occurrence of a morphosyntactic category depends on the domain of the text. The same is true with
sequences of morphosyntactic categories, which frequencies vary according to the domain. Sekine
(1997) compared the performances of a probabilistic syntactic parser with different configurations for
training and testing, using 8 domains of the Brown corpus. This work proved that the quality of the
parsing in terms of precision and recall falls as the domains of the texts used for training and testing
differs. Ruch and Gaudinat (2000) compared the lexical ambiguity between medical and general texts
and underlined the necessity to build domain-adaptable tools for Natural Language Processing. These
studies lead to the conclusion that the use of important corpora requires profiling tools in order to get
indications about lexical and morphosyntactic uses of their subparts and thus determine their
homogeneity or heterogeneity. Corpora profiling and tuning can globally improve the performances of
NLP tools, as shown in Illouz, (2000).
2.2 Previous works
The approach in TyPTex consists in developing a typology of texts through inductive methods. It
means that the text types are defined in terms of sets of correlated linguistic features obtained through
multivariate statistical techniques from annotated corpora. This approach is based on Biber's (1988,
1995) work. Biber uses 67 features corresponding to 16 different categories (verb tense and aspect
markers, interrogatives, passives, etc.). He examines their distribution in the first 1.000 words of 4.814
contemporary English texts from reference corpora. The identification of the 67 features in the corpus
is done automatically on the basis of a preliminary morphosyntactic tagging. The accuracy of the
tagging is checked by a linguist. The sets of correlated features (the dimensions) are obtained through a
multivariate statistical technique (factor analysis). Each dimension consists of two complementary
groups of features which can be interpreted as positive and negative poles. In other words, when one
group of features occurs in a text, the other group is avoided. Statistical methods are then used to group
texts into clusters according to their use of the dimensions. These clusters correspond directly neither to
text “genres” nor to language style or registers.
2.3 Data, tools and methods
The corpus used in TyPTex to test and tune the system represents 5 million words and is a part of
the corpus gathered by G. Vignaud (INALF – Institut National de la Langue Française) and B. Habert
within the European project PAROLE4. The texts are tagged according to the TEI (Text Encoding
Initiative) recommendations. Queries are then performed to extract a subset of texts which are relevant
for a determined study or application. The next step is to achieve a morphosyntactic tagging which
2 Available at http://www.ilc.pi.cnr.it/EAGLES/annotate/annotate.html.
3 See http://www.limsi.fr and http://www.ens-lsh.fr.
4 See http://www.elda.fr/Fr/cata/doc/parole.html.
403
associates each lexical item (or polylexical item) with a given lemma, a part of speech and other
morphosyntactic information. The tagger used currently is Sylex-Base. It is based on the work of
P. Constant (Ingenia, 1995), and proved to be robust during the tagger evaluation program GRACE
(Adda et al., 1998).
The second step is typological marking. It consists of replacing the information generated by the
morphosyntactic tagger with higher-level linguistic features. These new features are obtained on top of
the morphosyntactic tags and vary according to the oppositions the user wishes to bring out. Section 3
will explain more in detail why and how such manipulations of linguistic features are effected. From
the resulting marked corpus several matrices are generated, in particular the matrix containing the
frequencies of each feature in each text of the corpus under study. The resulting matrix is then analysed
by statistical software programs. The analysis of the matrix aims at, on the one hand, identifying
features that reveal a certain kind of opposition among the subparts of the corpus, and on the other
hand, making an inductive classification of texts.
3 Corpora exploration and manipulation of linguistic features
A lower level tagging used in TyPTex includes shifters, modals, presentatives (“il y a” and
“c'est”), tense use, passives, certain classes of adverbs (negation, grading), determiners, etc. From the
features tagged initially (around 300 available with Sylex and 170 with Cordial (Synapse, 1998)), about
40 were kept and divided into 2 subsets. The first subset comprises functional elements which role is
the organization of discourse and sentence. The second subset comprises open categories like nouns,
adjectives or verb tense.
The features available with the initial POS tagging may not be sufficient for a given study. There
is often a gap between what one gets at the output of a tagger and what is aimed at. In TyPTex, we call
typological marking the set of features that is presumed useful to bring out different types of texts.
Features has to be manipulated in order to get this awaited level of typological marking. However a set
of linguistic features can not be settled once and for all. Typological marking requires a lot of
explorations : one needs to test a set of linguistic features by analysing the distinctions it brings within
a corpus.
Sometimes features can be too fine-grained and lead to a scattering of occurrences which makes
contrasts imperceptible. This was the case in one of the pilot studies (Illouz et al., 1999) for the TyPTex
project with the verb category, which was divided into some 50 features (due to the morphology of
French). Most of those features had a only a few occurrences and were not statistically significant. The
other problem with this splitting of the features was that it offered no indication about the use of the
verb in general. Thus it was impossible to check whether a under-use of Nouns in a subpart of the
corpus was related to an over-use of Verbs in the same subpart. This is why some elementary features
had to be regrouped inside “super features”, covering larger categories which were not available with
the initial tagging.
Some features can also be too rough and hide real oppositions. For example, the same tag can be
used by a tagger to cover indifferently quantity indicators, as well as dates. We can presume that
splitting that general “Cardinal Number” feature into two sub-features (quantity and dates) would
create finer distinctions among the corpus. For instance, it can be necessary to gather tags
corresponding to the same function but belonging to different morphosyntactic categories, such as
some punctuation marks and some conjunctions, which can be regrouped as textual markers. For
example, this could be the case of punctuation symbols marking an incision in the discourse, such as
quotes, parenthesis or long dashes.
A good illustration of this point is the use of the distinction between qualitative and relational
adjectives made by Habert and Salem (1995). The aim of their study was to reveal differences of
language use in a sociologic corpus of open answers. A first feature set, using “traditional” POS
features (Verb, Adjective, Noun, etc.), revealed a difference in the use of verbs (overused by lesseducated
persons) and nouns (overused by educated persons) between two different groups of answers.
At this stage, adjective was considered as an “atomic” feature. A modified set of features introduced an
opposition between qualitative and relational uses inside the adjective category. Relational adjectives
are sometimes called “pertainyms”, since they mean something like “of, relating/pertaining to, or
associated with” some noun and play a role similar to that of a modifying noun (Fellbaum, 1990). For
example, geographical in “a geographical map” refers to geography, as presidential in “presidential
election” is linked to president. Adjectives that are not relational are considered to have a “qualitative”
function (which modifies the quality of a noun), such as “nice” in “a nice child” or “good” in “a good
404
practice”. This notion of relational use of adjectives is used in WordNet5 (Miller, 1998), where
relational adjectives are linked to their corresponding name, the other adjectives being mainly studied
through antonymy and contrast. This distinction enabled to refine the cluster of “educated persons”
between non-graduate and graduate persons. In that case, splitting the adjective category into two finer
categories provided a better description of the corpus.
Feature manipulation is required to get the awaited level of linguistic information, by completing
and modifying the feature set available with the initial POS tagging or changing the features retained
for typological marking. It is necessary therefore to be able to group features for one contrast, to divide
others, and at times even to start afresh tagging and marking. Corpus exploration requires flexibility in
the manipulation of the feature set which in turn introduces constraints on the representation formalism.
In the Typtex project, feature structures are used to represent linguistic features of the words in the
corpora.
4 Using feature structures as representation format
4.1 From tags to features
A morphosyntactic tagger associates a given piece of information with a lexical token by the way
of a tag. According to the software used, the content of this information may vary, and its form as well.
action action Ncfs (CORDIAL)
“action” nom : féminin singulier (SYLEX)
Figure 1 Examples of different POS tagger outputs
Figure 1 presents the output of two taggers CORDIAL and SYLEX for the word “action”, where
the linguistic information is the same (except for the lemma, which is not present here with Sylex) but
changes in form. In fact, there's often a confusion between the graphical forms that programs
manipulate (tags) and the linguistic features used by the linguist. Tags may be different but represent
the same linguistic feature. In the TyPTex project, an intermediate format is used to represent linguistic
information, independently of the tags provided by the preceding process (POS tagging, for example).
4.2 Representation
Feature structures such as those employed in unification grammars are used in order to represent
the linguistic information contained in a corpus. The format used in TyPTex is inspired from the PATR
formalism (Shieber, 1986). A feature structure associates values with a set of features, and can be
represented by an equation, where the feature is written between < > and the value is placed after a
symbol equals (=). Feature structures can be atomic (one feature associated with a value) or complex (a
value is itself a feature structure, in a recursive way). The example of tagging provided above will give
the following feature structure :
<form> = action
<lemma> = action
<category> = noun
<type> = common
<agreement gender> = feminine
<agreement number> = singular
.
Figure 2 Equation of a feature structure
In this example, some features have an atomic value (<form> for instance), whereas another as a
complex one (<agreement>). Feature structures can also have a graphical representation with a DAG
(Directed Acyclic Graph), as on figure 3.
5 See http://www.cogsci.princeton.edu/~wn/.
405
Figure 3 DAG representation of a feature structure
4.3 Modifiers
Modifying operators can also be used in the equation. This is the case of negation, marked by a
tilde ~, which implies that a feature shall not have a given value, as in : <agreement gender> =
~neutral. Disjunction, is an other aspect. It is the possibility to associate multiple values with a feature,
by putting these values between braces { }6. At last, two or more structures can share the same value
(co-indexation) when it is placed between parenthesis. This enables, for example, to specify an
agreement in number and gender between an adjective and a noun. In that case, the agreement features
of the adjective and the noun are not only equals, but share the same value.
4.4 Use in the TyPTex project
Each word of the corpus used in TyPTex is represented by a feature structure. Thus, it is possible
to modelise more precisely the information resulting from marking, in the style of Gazdar et al. (1988).
This kind of representation format allows one to manipulate linguistic features directly instead of tags.
A mapping (comparable to the mapping done in Atwell et al., 1994) is effected between the output of
the morphosyntactic tagger used upward and the corresponding linguistic feature structures, the
downward processing gaining in independence from the format of the tagger used. It is possible to use
an other program for the POS tagging-task and keep the same level of information thanks to a mapping
into feature structures. This enables to compare the results provided by different taggers and see their
influence on the quality of the marking.
An other advantage is that feature structures can encode any type of linguistic features, at any
level, morphological, syntactic, semantic or functional. Compared to a tag, a feature structure has the
following qualities :
1. Linguistic information is named explicitly and does not depend on the position of a character
inside a linear string, like in a tag. A feature structure (ex: <category>=noun,
<type>=common, <gender>=feminine, <number>=singular) is easier to read than a tag and
less ambiguous (ex: Ncfs).
2. Linguistic information is structured and hierarchized. A structure such as (<category>=noun,
<type>=common, <agreement gender>=feminine, <agreement number>=singular), is more
logical than (<category>=noun, <type>=common, <gender>=feminine,
<number>=singular). In this example, the sub-features <gender> and <number> can be
manipulated through the whole complex feature <agreement>.
However, one of the most remarkable aspects of the feature structure formalism is that it provides
useful mechanisms for feature manipulation, which is strongly required for corpora exploration.
5 Manipulation of data with Meta-Rules based on unification
Since some linguistic features one wants to obtain for typological marking may not be available at
the output of a morphosyntactic tagger, it becomes necessary to add, retrieve or modify some features
to get the awaited level of information for typological marking. On top of that, corpora exploration
requires to test several feature sets in order to bring out new contrasts inside the corpus, which also
implies the use of tools for features manipulation.
6 This is a good way to represent the output of a non-deterministic tagger, which can give different
possible tags for a single token.
406
5.1 Unification of feature structures
Feature structure are based on the concept of unification, which can be defined by the following :
“Unification of two feature structures A and B is the minimal structure containing both A and B”.
Unification checks the compatibility between two feature structures (a feature must not have two
different values) and when possible produces a structure containing all the information of both
structures.
For example, the unification of the structure :
<category> = noun
<type> = common
<agreement gender> = feminine
with the structure
<category> = noun
<type> = common
< agreement number> = singular
yields:
<category> = noun
<type> = common
<agreement gender> = feminine
< agreement number> = singular
5.2 Meta-Rules
Meta-Rules (as in Jacquemin 1994a, 1994b and 1997) based on unification are used to modify
feature structures. Basically, a Meta-Rule is a feature structure consisting in two parts : a source part
and a target part (in an equation, the source is separated from the target by the symbol “=>”). A Meta-
Rule is applied on the feature structures representing a word (or a sequence of words) of the corpus : if
unification is possible between the source part of the rule and the feature structure of the corpus, then
the last is replaced by the target part of this rule. Thus Meta-Rules do not only add information to a
given feature structure of the corpus but totally rewrite it.
The role of unification is to check the compatibility between the source part of the Meta-Rule and
a word of the corpus. However the unification process used here slightly differs from usual unification.
It could be called constrained unification, because it stipulates that the feature structure representing
the word must contain at least all the information of the source part of the Meta-Rule (same features
with compatible values). Thus, the source part of the Meta-Rule subsumes the feature structures of the
word. This condition (unification + subsumption) constraints the writing of the rules and ensures that
they are used in the correct cases.
For example, using normal unification with the Meta-Rule :
<tense>=present
=>
<tense>=present
<deictic>=yes
changes any Noun represented minimally by the feature structure :
<category> = noun
<type> = common
< agreement number> = singular
into :
<tense>=present
<deictic>=yes
407
The use of constrained unification prevent such cases.
5.3 Positional information
One important aspect of using feature structures is that it can help to modify the linguistic
information of a corpus in a contextual way. In this case word tokens are not submitted to Meta-Rules
separately, but inside sequences (corresponding to a paragraph, for example). Thus, the scope of a
Meta-Rule can be larger than one slot in the sequence and covers several words. One tries to apply each
Meta-Rule starting from each position inside the sequence (by checking whether the source-part of the
Meta-Rule subsumes and can be unified with a part of the sequence of feature structures). All Meta-
Rules are tested on position 1 in the sequence, then on position 2, and so on until the end of the
sequence. In case of success, the slots inside the sequence covered by the source part are replaced by
the target-part of the Meta-Rule.
Positional information is added to the Meta-Rule in order to specify the distance between the
words in the sequence. This information is present in the feature structures of the rule as a feature by
itself. The notation depends on the type of the distance, which can be fixed (noted by a “p” + number
of the token in the sequence, ex: p4), free (noted by a “*” + number of the token, ex: *4), or limited
(noted by a “*” + number of the token + “+”+ maximal distance, ex: *4+3). The position 1 corresponds
to the current position of the processing inside the sequence.
<p1 lemma>="
<p1 category >=punctuation
<*2+5 lemma>="
<*2+5 category >=punctuation
=>
<p1 lemma>="
<p1 category >=punctuation
<p1 type>=start_short_citation
<*2+5 lemma>="
<*2+5 category >=punctuation
<*2+5 type>= end_short_citation
.
Figure 4 Distances in Meta-Rule
Figure 4 shows a Meta-Rule, which adds a feature <type> to the quotes of a text, with a value
start_short_citation or end_short_citation, if there is a distance of up to 5 words between the positions
of the quotes. In that example, the distance is limited to 5. This Meta-Rule could be used to
differentiate cases of reported discourse or to distinguish citations marking a phenomenon of
distanciation form the speaker.
5.4 Meta-rules in TyPTex
Meta-Rules are used in order to manipulate linguistic features by regrouping them into larger
categories, or in the opposite splitting into finer features. This is a convenient tool for managing the
feature sets used in a study and for comparing the results that they provide. It also helps to add features
not available at the output of a tagger and thus get the level of information awaited for typological
marking. One of the most powerful aspect of the Meta-Rules is that they are contextual : one can
manipulate the content of a feature structure (basically a word) dependently of the context of that
structure (other words in the same sentence or paragraph). In the rest of that paper, we will show that
the use of Meta-Rules can improve the marking of such a subtle distinction as the one between
qualitative and relational adjectives.
6 An application case: distinguishing between qualitative and relational adjectives in a corpus
In the Typtex project, an opposition is projected in the corpora between relational and qualitative
adjectives (see section 3). The description of relational adjectives provided by Habert and Salem (1995)
is followed :
1. they are equivalent to a sequence of nouns : presidential election / election of the president
2. they are never gradable : *a very geographical map
3. they cannot have a predicative function : *response to the virus was immune.
408
We realized that the addition of a distinction between qualitative and relational adjectives
improved the description of a corpus in Habert and Salem (1995) by refining the groupings of texts
obtained after a multivariate analysis. But although this opposition seems to be useful for corpora
description, its use is far from being obvious. As noticed by Bartning and Noailly (1993), a lot of
adjectives can be analysed either as relational or qualitative, depending on the context. This is the case
with the French adjective économique, which has a relational function in “la politique économique”
(related to economics), but a qualitative one in “une formule économique”( which is not expensive).
One can even assume that any relational adjective can take a qualitative function in a given context.
That is why the distinction between these two aspects is somehow difficult to process automatically
without information about the context of use.
The solution adopted in Typtex was to combine two approaches to distinguish between qualitative
and relational adjectives : on the one hand a list of potentially non-ambiguous adjectives was used to
annotate the corpus (information is projected on the texts) while on the other hand a set of Meta-Rules
was intended to disambiguate the adjectives in context (information is then extracted from the texts).
Here we compare these two approaches.
6.1 An empirically-build static list
A list of relational and qualitative adjectives has been constituted manually using press articles of
the French newspaper Le Monde, and taken from the PAROLE corpus. The 14 million words subpart
Press of the corpus has been built by random selection of full issues of Le Monde and gathers issues
from 1987, 1989, 1991, 1993 and 1995. To build the list we extracted the thousand most frequent
adjectives of the whole corpus Le Monde and analysed them manually, to check whether they have a
priori a relational or qualitative function out of context.
Relational
annuel
automobile
bancaire
budgétaire
cardinal
constitutionnel
exécutif
…
Qualitative
absolu
actuel
ambitieux
ami
ancien
beau
bien
…
Ambiguous
commercial
français
historique
humanitaire
idéologique
judiciaire
logique
…
Figure 5 A sample of the adjective list
In its final state the list contains 264 non-ambiguous adjectives with 244 qualitative and 20
relational. All the other adjectives studied were judged too dependent on the context and thus
ambiguous. Figure 5 shows a sample of this list.
6.2 Description of the Meta-Rules
Afterwards we created a set of Meta-Rules using feature structures for disambiguation. These
rules are the following :
An ambiguous adjective is considered as qualitative if :
1. it is directly preceded by a grading adverb
2. it is directly preceded by a stative verb
3. it is directly preceding a noun, with the same value of number and gender7
4. it is directly preceded by a qualitative adjective and a conjunction
5. it is directly preceding a conjunction and a qualitative adjective
6. it stands alone between two double quotes
An ambiguous adjective is considered as relational if :
7. it is directly preceding a conjunction and a relational adjective
8. it is directly preceded by a relational adjective and a conjunction
7 This is at least true in French, where the only position of the relational adjective is after the noun.
409
Rules 4, 5, 7 and 8 are based on the hypothesis that coordination is only possible between
adjectives sharing the same function as whether qualitative or relational (e.g., compare young and nice
with *beautiful and geographical). Hatzivassiloglou and Wiebe (2000: 300) report on similar property
of conjunctions for assigning semantic orientation to adjectives (e.g., compare corrupt and brutal with
* corrupt but brutal).
Figure 6 shows the equation corresponding to rule 1.
<p1 form>=(1)
<p1 lemma>=(2)
<p1 category>=adverb
<p1 type>=grading
<p2 form>=(3)
<p2 lemma>=(4)
<p2 category>=adjective
=>
<p1 form>=(1)
<p1 lemma>=(2)
<p1 category>=adverb
<p1 type>=grading
<p2 form>=(3)
<p2 lemma>=(4)
<p2 category>=adjective
<p2 qualitative>=true
<p2 relational>=false.
Figure 6 Example of Meta-Rule used for disambiguation of adjectives
This rule indicates that any adjectives directly preceded by a grading adverb has a qualitative
function. If the source part of the Meta-Rule subsumes and can be unified with feature structures
representing a sequence of words, these structures will be replaced by the target part of the Meta-Rule.
In this example, features <qualitative>=true and <relational>=false are added to the adjective.
This Meta-Rule will correctly assign a qualitative function to the adjective tendu in the sequence
“un climat de plus en plus tendu” (a situation more and more tense), witch can be minimally
represented by the following sequence of feature structures :
<form> = un
<lemma> = un
<category> = determiner
<type> = particle
<defined> = false.
<form> = climat
<lemma> = climat
<category> = noun
<type> = common.
<form> = de_plus_en_plus
<lemma> = de_plus_en_plus
<category> = adverb
<type> = grading.
<form> = tendu
<lemma> = tendu
<category> = adjective
<qualitative>= true
<relational>= true.
In this case, the value of the feature <relational> will be changed to false by the Meta-Rule.
However this rule is not aimed at recognizing the qualitative function of the adjective dangereux in “un
climat dangereux a tous égards ” (a dangerous situation in all respect).
<form> = un
<lemma> = un
<category> = determiner
<type> = particle
410
<defined> = false.
<form> = climat
<lemma> = climat
<category> = noun
<type> = common.
<form> = dangereux
<lemma> = dangereux
<category> = adjective
<qualitative>= true
<relational>= true.
<form>= a_tous_égards
<lemma>= a_tous_égards
<category>= adverb
<type>= general .
This sequence of feature structures will remain unchanged by the Meta-Rule (and the function of
the adjective ambiguous) because of the mismatch between the postposition of the adverb and the value
of its feature <type> which value is not equal to grading.
6.3 Building of a reference corpus
For this comparison between a fixed list approach and the rule-based approach, we used a sample
of 13 papers from Le Monde taken from the PAROLE corpus. These texts has been extracted from the
Economy section of the newspaper and represent around 10.000 words. The corpus was first tagged
using the CORDIAL8 tagger and then converted into feature structures. A refinement of the original tag
set has been provided by adding an information via Meta-Rules about grading adverbs for 129 frequent
adverbs. Thus, adverbs such has “tres” (very), “plus” (more) or “extremement” (extremely) gained a
new feature <type>= grading which was not present after the original tagging made by CORDIAL and
its conversion into feature structures. The next step was a manual categorization of adjectives between
relational and qualitative. No adjectives have been left ambiguous. A rectification of the data was
necessary in order to correct the errors made by the POS tagger. This is commonly the case of verbs
erroneously analysed as adjectives or a bad tokenisation (“Ministre de l'Intérieur” (Minister of the
Interior) recognized as a single token but “Premier Ministre” (Prime Minister) identified as an
Adjective followed by a Noun). After this correction, the corpus contained 507 adjective occurrences
with 378 qualitative for 129 relational uses.
6.4 Results and discussion
This corrected corpus serves as reference to evaluate and compare the two approaches for
adjective categorization. At the beginning of the experience a version of the corpus was created, where
all adjectives were ambiguous (the values of their features <qualitative> and <relational> were both
true). The goal of the evaluation is to measure the recall and precision provided by the different
methods for adjective marking. By recall, we mean the ratio of adjectives which have been
disambiguated ((#total - #ambiguous)/ #total * 100), correctly or not, while precision indicates the ratio
of well-tagged adjectives after disambiguation (#well-tagged / (#total - #ambiguous) * 100). Three tests
have been carried out using respectively the plain adjective list described above, the set of Meta-Rules
and a mix of list and rules on the ambiguous version of the corpus. Results are compared against the
reference corpus in order to determine which of these approaches is the most efficient to distinguish
between qualitative and relational uses of the adjectives. Figure 7 shows the results obtained.
8See http://www.synapse-fr.com.
411
List Rules List +
Rules
Correct 222 178 290
Wrong 6 0 6
Ambiguous 278 328 210
Total 506
Recall (%) 45.05 35.17 58.49
Precision (%) 97.36 100 97.97
Figure 7 Compared results for relational / qualitative tagging
The values in the row Correct indicate how many adjectives has been correctly categorized as
whether qualitative or relational according to the different methods. Wrong gives the number of wrong
marked adjectives, while the numbers in Ambiguous refer to the occurrences of adjectives not covered
neither by the list nor by the Meta-Rules.
Using only the contextual Meta-Rules improves the precision of the adjective categorization
compared to the use of the fixed list but provides a loss of recall that reaches 10%. The best solution
seems to be a mixed use of the list and the rules, which improves the recall with a relatively equal rate
of precision. However this gain in recall is relatively moderate. It could be explained by a partial
overlap of coverage between the two approaches (the adjectives recognized are often the same).
This example of linguistic feature marking (a new feature is added to the description of a corpus)
illustrates the use of feature structures as a representation format. The combined use of fixed
information and contextual rules improved the realization of such a subtle opposition as the one
between relational and qualitative for adjectives, compared to the projection of a lexicon alone. This
method enables the characterization of individual word occurrences, rather than word types, without
requiring an important learning phase (Hatzivassiloglou, 2000). Another aspect is the fact that this
mixed method allows non-usual cases to be marked correctly, like adjectives which have in context a
different function than their most probable one (ex: “a very Parisian atmosphere”). However the
results of this experience could be surely improved by refining the content of the fixed list and of the
rules. Some semantic information would be interesting to solve the ambiguity between relational and
qualitative functions of adjectives (this is required to disambiguate the example with “économique”
provided at the beginning of this section), as well as regular expression operators in Meta-Rules in
order to use morphological information (word endings).
7 Conclusions
In this paper we report on the use of feature structures to represent the linguistic information of a
corpus. Experience of corpora profiling in the TyPTex project shows that this approach helps to
represent any kind of linguistic information, independently of the tools used for tagging. Feature
structures is an unifying format which can be used to map from an annotation scheme to an other, in the
spirit of Atwell et al. (1994).
Feature structures formalism also helps to handle a set of features with Meta-Rules based on
unification. We showed that corpora exploration requires to modify the linguistic features in order to
obtain new results and thus to change the point of view on the data. Another aspect is that the features
available at the output of a POS tagger may not be sufficient for a given experimentation, one needs to
add some information to get the awaited level of marking. By defining Meta-Rules to operate on the
feature structures representing a corpus, one can modify the information in a contextual way. An
example of a mixed approach between projection of information from a static list and contextual
marking via Meta-Rules showed that feature structures can improve the reliability and coverage of the
marking.
Acknowledgements
The authors wish to thank Marianne Dabbadie and Lee Humphrey (LexiQuest) for their help
during the writing of this paper.
References
Adda G, Mariani J, Lecomte J, Paroubek P, Rajman M 1998 The GRACE French Part-Of-Speech
Tagging Evaluation Task. In Proceedings of LREC'98 (1st International Conference on Language
Resources and Evaluation), Granada, Spain, pp 2433-2441.
412
Atwell E, Hughes J, Souter C 1994 AMALGAM: Automatic Mapping Among Lexico-Grammatical
Annotation Models. In Klavans J, Resnik P (eds), The Balancing Act : Combining Symbolic and
Statistical Approaches to Language. Las Cruces, Association for Computational Linguistics, pp
11-21.
Bartning I, Noailly M 1993 Du relationnel au qualificatif: flux et reflux. In L'information grammaticale
58(1): 27-32.
Biber D 1988 Variation across speech and writing. Cambridge, Cambridge University Press.
Biber D 1993 Using register-diversified corpora for general language studies. Computational
Linguistics, 19(2): 243-258.
Biber D 1995 Dimensions of register variation : a cross-linguistic comparison. Cambridge, Cambridge
University Press.
Bird S, Liberman M 2001 A formal framework for linguistic annotation. Speech Communication 33(1):
23-60.
Fellbaum C, Gross D, Miller K 1990 Adjectives in WordNet. International Journal of Lexicography
3(4): 265-277.
Folch H, Heiden S, Habert B, Fleury S, Illouz G, Lafon P, Nioche J, Prévost S 2000 TyPTex: Inductive
typological text classification analysis for NLP systems tuning/evaluation. In Proceedings of the
Second International Conference on Language Resources and Evaluation, Athens, pp 141-148.
Gazdar G, Pullum G, Carpenter R, Klein E, Hukari T E, Levine R D 1988 Category structures.
Computational Linguistics, 14(1): 1-19.
Habert B, Salem A 1995 L'utilisation de catégorisations multiples pour l'analyse quantitative de
données textuelles. TAL, 36(1-2): 249-276.
Hatzivassiloglou V, Wiebe J 2000 Effects of adjective orientation and gradability on sentence
subjectivity. In Proceedings of the 18th International Conference on Computational Linguistics
(COLING-2000), Saarbrücken, pp 299-305.
Illouz G, Habert B, Fleury S, Folch H, Heiden S, Lafon P 1999 Maîtriser les déluges de données
hétérogenes. In Condamines A, Fabre C, Péry-Woodley M-P (eds), Corpus et traitement
automatique des langues : pour une réflexion méthodologique, Cargese, pp 37-46.
Illouz G, Habert B, Folch H, Fleury S, Heiden S, Lafon P, Prévost S 2000 TyPTex: Generic features for
Text Profiler. In Content-Based Multimedia Information Access (RIAO'00), Paris, pp 1526-1540.
Illouz G 2000, Typage de données textuelles et adaptation des traitements linguistiques. Doctorat
d'informatique, Université Paris-Sud.
Ingenia 1995 Manuel de développement Sylex-Base. Paris, Ingenia – Langage naturel.
Jacquemin C 1994a FASTR: A unification-based front-end to automatic indexing. In Proceedings of
Intelligent Multimedia Information Retrieval Systems and Management (RIAO'94), New York, pp
34-47.
Jacquemin C 1994b FASTR: A unification grammar and a parser for terminology extraction from large
corpora. In Proceedings of Journées IA'94, Paris, pp 155-164.
Jacquemin C 1997 Variation terminologique : Reconnaissance et acquisition automatiques de termes et
de leurs variantes en corpus. Mémoire d'Habilitation a Diriger des Recherches en informatique
fondamentale, Université de Nantes.
McEnery A, Wilson A 1996 Corpus linguistics. Edinburgh, Edinburgh University Press.
Miller K 1998 Modifiers in WordNet. In Fellbaum C (ed), WordNet: an electronic lexical database.
Cambridge, MIT Press, pp 47-67.
Ruch P, Gaudinat A 2000 Comparing corpora and lexical ambiguity. In Proceedings of the Comparing
Corpora Workshop (38th Annual Meeting of the Association for Computational Linguistics),
HongKong, pp 14-20.
Sekine S 1998 The domain dependence of parsing. In Proceeding of the Fifth Conference on Applied
Natural Language Processing (Association for Computational Linguistics), Washington, pp 96-
102.
Shieber S 1985 An introduction to unification-Based Approaches to Grammar. Stanford, CSLI Lecture
Notes 4 (Center for the Study of Language and Information).
Tambouratzis G, Markantonatou S, Hairetakis N, Vassilliou M, Tambouratzis D, Carayannis G 2000
Discriminating the registers and styles in the Modern Greek language. In Proceedings of the
Comparing Corpora Workshop (38th Annual Meeting of the Association for Computational
Linguistics), HongKong, pp 35-43.

368
Annotated corpora for assistance with English-Polish translation.
Barbara Lewandowska-Tomaszczyk, University of Lodz, Poland,
Michael P. Oakes, University of Sunderland, England,
Paul Rayson, University of Lancaster, England.
Alignment of a large bilingual corpus of original material and an acceptable translation facilitates a
number of automated and partially automated approaches to translation. (Kay and Roescheisen 1993, p
121). The approach to the automatic alignment of Polish and English texts taken in this paper is that of
Gale and Church (1993). Once the texts have been aligned, they can then be displayed to the translator
as required using Scott's (1996) “WordSmith” concordancing tool. Using WordSmith, sentences and
their translations can be retrieved and shown to the translator if they contain specified words, phrases
or word fragments. The power of the bilingual concordancing tool can be enhanced by using annotated
corpora for alignment.
The two types of annotation we have employed are a) Part of Speech tagging, provided by the CLAWS
tagger (Garside and Smith, 1997), and b) semantic tagging, provided by the ACASD suite of computer
programs (Thomas & Wilson, 1996). With part of speech tagging, every word in the corpus is
automatically assigned its most likely grammatical class, e.g. the notation “book_NN1” shows that a
particular instance of the word “book” is most probably a singular common noun. If each word in the
aligned text has its own part of speech category, we can use the search term “book” to retrieve every
aligned region containing the word “book”, irrespective of whether it occurs as a noun, verb, or any
other grammatical category. On the other hand, the notation “book_NN1” will only retrieve aligned
regions in which the word “book” occurs as a singular common noun, enabling us to see the various
translations in Polish of the English word “book” when it occurs as a singular common noun.
Semantic tagging involves assigning each word in the corpus with a label showing the semantic
category (akin to a thesaurus category such as “colour”, “power” or “education”) to which each word
belongs. For example, the notation “clinical_B2” would retrieve only aligned regions containing
“clinical” as a medical term, while “clinical_E1-“ would retrieve only those regions containing the
word “clinical” where it means “lacking emotion”. The notation “clinical” allows the retrieval of
aligned regions containing the word “clinical” irrespective of its meaning, “_B2” will allow the
retrieval of aligned regions containing any medical term, and “_E1-” will allow us to view all the
synonyms in the English corpus meaning “without emotion” alongside their Polish equivalents.
At present only the English text has been fully annotated, since a Polish semantic tagger is currently
unavailable. We will discuss our approach whereby a machine readable Polish-English dictionary and
an alignment of semantically tagged English text with the equivalent unannotated Polish text might be
used to semantically tag certain words in the Polish part of the text.
References
Gale W A, Church K W 1993 A program for aligning sentences in bilingual corpora, Computational
Linguistics 19(1): 75-102.
Garside R, and Smith N 1997 A hybrid grammatical tagger: CLAWS4, in Garside R, Leech G, and
McEnery A. (eds.) Corpus Annotation: Linguistic Information from Computer Text Corpora.
Longman, London, pp. 102-121.
Kay M, Roescheisen M 1993 Text-translation alignment, Computational Linguistics 19(1):121-142..
Scott M 1996 WordSmith Tools Manual, Oxford University Press.
Thomas J, Wilson A 1996 Methodologies for studying doctor-patient interaction, in Thomas J, Short
M (eds) Using corpora for language research, London & New York, Longman.

413
OpenText.org: the problems and prospects of working with ancient discourse
Matthew Brook O'Donnell, Stanley E. Porter and Jeffrey T. Reed
University of Surrey Roehampton and OpenText.org
1. Introduction
The vast majority of studies in corpus linguistics have focused upon contemporary usage of
modern languages. However, although there have been a number of studies of the earlier periods of
some of these languages, such as Old English and Old French, they have tended to adopt the methods
developed for modern languages. In our theoretical paper (Porter and O'Donnell 2001), we have
explored the methodological challenges and questions posed by the study of an epigraphic language,
such as Hellenistic Greek.1 In particular, we have found the need for closer attention to the criteria
used in the compilation of a corpus, the integration of the levels of annotation applied to a corpus,
and maintaining a focus upon both traditional referential access and narrative or sequential textanalysis.
2 As a result, our approach is more than simply computer-aided text analysis, but in some
ways fulfils the goals of the originators of corpus-based linguistics. It does this by performing a full
analysis of the language, utilizing a structured corpus, rather than analyzing just a small portion of a
much larger corpus. The textual orientation of classical and New Testament scholarship is compatible
with and, in fact, requires a micro-pattern analysis, and the reading and analysis of full texts. Rather
than placing it outside the scope of corpus linguistics, we argue that this perspective offers new
avenues for the discipline, ones that we are exploring in OpenText.org.
OpenText.org is a web-based initiative dedicated to creating resources for the linguistic analysis
of Hellenistic Greek, and especially the Greek of the New Testament, in collaboration with interested
scholars around the world. OpenText.org aims to make use of the insights and methods of corpus
linguistics, specifically in terms of building a representative corpus of Hellenistic Greek, richly
annotated according to a functional discourse model. This bottom-up discourse model relies upon the
notion of levels of formal analysis. We parse all forms, beginning with morphology, but categorize
from the word group up. We have found that the word group constitutes the smallest meaningful unit
for discourse analysis. The increasingly higher levels are those of the clause, paragraph and
discourse. One of the major principles of our annotation scheme is to mark features at the level of
discourse at which they function. This has required the development of level-reflective and levelsensitive
notational categories, as certain elements such as a conjunction may operate either at the
word group, clause or paragraph level. These notational categories serve in a horizontal dimension
both to specify the function of the individual element and to indicate its relationship to other
elements within the structural unit. Each structural unit then constitutes a minimal unit at the next
highest level of discourse. Each level of analysis builds upon the previous level, and thus analysis at
a particular level can reach down to include features from lower levels. This vertical dimension of
analysis is also segregated according to the components of register, field, tenor and mode. As a result,
various features at a given level will serve different register meta-functions. This schematization
provides a means for moving from the elements of text to the context of situation. The intended result
is a complex calculus of features and functions that enables the analysis of the discourse. At the heart
of this method, therefore, is annotation. More than that, we have found that the process of annotation
itself constitutes a major part of the analytical process, raising questions as to the function of
components of discourse within their respective units and the nature of text itself as these elements
constitute the discourse (DeRose et al. 1990; Renear et al. 1996; Leech 1994).
This paper presents two examples utilizing this methodology, drawn from the Greek New
Testament. The first demonstrates how the OpenText.org annotation model can facilitate a full
discourse analysis of the letter to Philemon. We have consciously selected this text for a number of
reasons. The letter to Philemon is a small, complete text that encapsulates an interesting moment in
discourse. It allows for us to present a synchronic view of this text, while still facilitating analysis of
individual textual components. The result is, we think, a presentation of most of the major features of
discourse as they are contained and displayed in our discourse model. We unfortunately will not be
able to present all of these dimensions here. Though the shortest Pauline letter, Philemon has
1We are defining Hellenistic Greek as that Greek written by native and non-native Greek speakers throughout the Hellenistic and
Roman worlds from approximately the fourth century B.C. to the fourth century A.D.
2The third element aligns corpus linguistics with discourse analysis, sharing a common concern for the analysis of real language
usage, the observation of patterns of linguistic usage, and the (quantitative) filtering of large amounts of data. On referential and
narrative methods of access to corpus data, see McCarty 1996.
414
received considerable discussion in scholarly circles; however, the major focus of this discussion has
been the attempt to reconstruct the probable context of situation that gave rise to the discourse (see
Pearson 1999 for an overview of major positions). Few of these attempts are predicated upon a
thorough analysis of the text as discourse, at least as we are defining and using the concept here. We
think that insight can be gained into the context of situation of this provocative and intriguing New
Testament text through close analysis of the elements of discourse, particularly the questions of the
relationships among the major participants (Paul and Philemon) and of the function that Onesimus
plays in these relationships. We think that our discourse model pushes interpretation forward by
pointing out textual relationships that previous interpreters have often overlooked. In that sense,
questions of the tenor of discourse, among others, are specifically addressed in this example. The
second example illustrates the use of semantic-domain annotation of the book of Revelation to
explore structure, specifically through the identification of cohesive units. In this example, rather
than examining a single discourse from a variety of perspectives, we are examining one particular
component of register over a larger and potentially more complex text. The structure of the book of
Revelation suggests a set of complex semantic relations often linked with shifts in text-type. Our
analysis suggests that through a variety of textual means the author has structured discourse themes
to progress the thematic content of the book. This thematic content is introduced through lexis.
Analysis of the semantic-domain patterns over the length of the discourse shows how the thematic
material is not only introduced and treated in individual units, but how it is used to cohede the
individual paragraph units and the entire discourse. Whereas the instance from Philemon examines a
number of levels in order to discuss the tenor of discourse, this examination treats broad patterns of
semantic-domain structure to say something about field and mode of discourse.
2. Philemon and participant structure
As we mentioned above, our method of analysis begins with the word group, and here we
present three word groups from the book of Philemon. We present these because each illustrates
important elements, before moving to analysis of a single set of clauses that constitute part of a
paragraph. Under the field component of register the semantic relationships between words in a word
group are annotated. We have defined four forms of modification, and one type of conjunction. The
four modifiers are: specifier (sp), including articles and prepositions; definer (df), adjectives and
appositional words; qualifier (ql), genitive and dative modifiers; and preposition (pr), a prepositional
phrase that modifies a substantive. The conjunction or connection (cn) relationship is used to join two
words within the group. It is helpful to visualize these semantic relationships through a series of
nested boxes. Each box represents a word with slots for each of the relationships below the word. The
boxes for modifiers (and their associated modifiers) are drawn within the relative slot of the word
they modify. Thus word groups can be represented through the application of a recursive process.
The first is word group 1 (see fig. 1) in the book of Philemon, part of what is traditionally called
the epistolary salutation, ‘Paul, prisoner of Christ Jesus'. This word group consists of four words,
with the head term being 3DXOR in the nominative, or what might be called the subject case, but
this implies analysis beyond the word group. Syntagmatically, the three words follow in sequential
order the word 3DXOR (syntagmatic order is indicated by the word identifier, e.g., w10), but there
are two types of semantic relations indicated. The first is that of definer, in which GHYVPLR (prisoner)
defines who Paul is. This relationship often describes what is traditionally called an epexegetical
relationship that restates the head term. The relationship of ¨,KVRX (Jesus) and &ULVWRX (Christ) is
also one of definition. However, the relation between &ULVWRX (Christ) and GHYVPLR (prisoner) is
one of qualification. This is not a relationship that defines but qualifies who the servant is—he
belongs to Christ (modifiers in the genitive case in Greek, as this one is, often have qualifier
relations). This is a fairly straightforward example of a word group. This first word group is
annotated in XML in the following manner and visualized in figure 1.
<wg:group id="wg1" head="w1">
<w id="w1">3DXOR</w>
<w id="w2" modify="w1" rel="define">GHYVPLR</w>
<w id="w3" modify="w2" rel="qualify">&ULVWRX</w>
<w id="w4" modify="w3" rel="define">¨,KVRX</w>
</wg:group>
415
w1 3DXOR
sp df ql pr
w2 GHYVPLR
sp df ql pr
w3 &ULVWRX
sp df ql pr
w4 ¨,KVRX
Fig. 1. Word group 1 ‘Paul, prisoner of Christ Jesus’
The second word group is number 3 (see fig. 2; appendix fig. 8 for annotation), ‘Philemon, the
beloved and fellow worker of us'. It is also part of the salutation of the book of Philemon, except that
here the head term, )LOKYPRQL (Philemon), is in the dative case, or what might be called a
complement case, that is, a case in which a complement can occur.3 In word group 3 we have a
slightly different kind of semantic relationship among the elements, one that is not linear as word
group 1 is. Here, the head term is defined by two separate modifiers, both adjectives, DMJDSKWZ
(beloved) and VXQHUJZ (fellow worker). The first of these is preceded by a specifier, the article
(WZ), and the second is followed by a qualifier, the pronoun K-PZQ (our).
w9 )LOKYPRQL
sp df ql pr cn
w11 DMJDSKWZ
sp df ql pr
w10 WZ
w12 NDL
w13 VXQHUJZ
sp df ql pr
w14 K-PZQ
Fig. 2. Word group 3 ‘Philemon, the beloved and fellow worker of us’
These two modifiers are connected in their modifying function by a conjunction (NDLY),
operating at the word-group level.4 Our word-group analysis is economical in so far as distribution of
elements is concerned, so we only analyze the article as occurring as a specifier of the first and the
pronoun as a qualifier of the second adjective. The annotation principle behind this is to indicate each
modifying word as connected to the one other word in the group to which it is most closely attached
in terms of grammatical function.
So far, from the structure of the word groups, one might well think that these groups are
somewhat similarly structured, and probably perform similar functions. Word group 54, the third
example to consider (see fig. 3; appendix fig. 9 for annotation), consists of a significantly larger
number of words than in either word group 1 or 3, ‘Onesimus, the then useless to you, but now useful
to you and to me'. In some ways, the structure here is similar to word group 3. There is a head term,
¨2QKYVLPRQ (Onesimus), in the accusative case, another of what might be called the complement
cases (again extending analysis beyond the word group). There are also two definers, both adjectives.
The first is D>FUKVWRQ (useless) and the second HX>FUKVWRQ (useful). Each of these, however, has a
number of further modifiers. D>FUKVWRQ has a specifier, the article WRYQ, and two qualifiers, SRWHY
(then) and VRL (you), the first an adverb and the second a pronoun. The second definer, HX>FUKVWRQ,
has three qualifiers, the adverb QXQLY (now), and the pronouns VRLY (you) and HMPRLY (me). There are a
3 However, at the level of the word group this is not significant, since here we are analyzing relations of the words within the
word group, not its relations outside of it.
4 The word VXQHUJRY can be used as a substantive, in which case the phrase could be analyzed as consisting of two separate
word groups, ‘Philemon the beloved’ and ‘fellow worker of us', joined at the clause level by the conjunction NDLY. The head term
of the first word group, )LOKYPRQL, has a single definer, DMJDSKWZ, and the second, VXQHUJZ, a qualifier K-PZQ. The annotation
scheme can allow for alternative analyses where necessary.
416
number of instances of parallelism here worth noting. First is the structural parallelism with the two
definers; the second is the temporal alternation between past and present; the third is the use of the
common root of the adjectives; and the fourth is the repetition and expansion of scope indicated by
the pronouns. Some of this parallelism emerges more clearly because of the use of the word group
conjunctions. The first, GHY (but), joins the two defining phrases, while the second, NDLY (and), joins
two qualifiers. The parallelism, as well as the sheer bulk of the construction, seems to give significant
weight, at least at the level of the word group, to this particular construction.
w145 ¨2QKYVLPRQ
sp df ql pr cn
w149 D>FUKVWRQ
sp df ql pr
w146 WRYQ w147 SRWHY
w148 VRL
w151 GH
w156 HX>FUKVWRQ
sp df ql pr cn
w150 QXQL
w153 VRL
w154 NDL
w155 HMPRL
Fig. 3. Word group 54 ‘Onesimus, the then useless to you, but now useful to you and to me’
These three word groups are important for a number of reasons. The first is that these are the
word groups in which the major participants of the discourse of the book of Philemon are introduced,
Paul, Philemon and (according to most scholars) Onesimus, a point we will come back to below.
There are a number of other participants also mentioned in Philemon (e.g. Christ Jesus, who is,
theology aside, in the position of a qualifier of a definer in relation to the head term, Paul; Timothy;
Apphia; Archippus; and ‘the church in your house'; etc.; see fig. 6), but, as we would see if we
analyzed more of the discourse, these are not given prominence as the others are. Each of the major
participants is introduced in a word group in which there is significant modification, which is not
found in the word groups for the other participants (see word groups 2, 4 and 5, where the
modification is noticeably less). For each major participant, there is a grammaticalized reference by
name, and then appropriate modification to indicate their role and relation. Subsequent participant
reference in the discourse is made by a combination of grammaticalized, reduced and implied
reference. For word groups 3 and 54, there is also internal modification that references Paul (and in
word group 54 Philemon as well), thus further indicating participant relations at the word-group
level. On the basis of this evidence, it appears that Onesimus is at least as important, at least at the
level of the word group, as any of the other major participants, and certainly more important than the
minor ones. That is consistent with traditional examination of Onesimus (word group 54, see fig. 3)
within this Pauline letter by the vast majority of New Testament scholars. Though our treatment of
annotation at the word-group level has focused upon the interpretative and discourse significance of
individual word groups, it should be clear how this detail of annotation applied across a larger corpus
of texts could be utilized using both the traditional referential retrieval paradigm (i.e. searching for
particular words, participants or grammatical features in a certain modification position) and more
discourse oriented narrative approaches.5
The next level of analysis is that of the clause. Within the book of Philemon, there are 47
clausal units, arranged within 5 paragraphs. Paragraph one consists of two clauses. The first has six
word groups, two of which are presented above. They form the two major groups that constitute the
structure of clause one. In paragraph three, however, there are 16 clauses. We wish here to analyze
5 The annotation of semantic relationships (specifier, definer, qualifer and preposition) within a word group is just one element
of the field of discourse. Other features marked at the word-group level include semantic domains (field), part of speech and
lexical information (mode) and participant reference (tenor). For detailed specification of the OpenText.org annotation model for
each of the levels of discourse and the associated XML DTDs, see http://www.opentext.org/specifications.
417
clauses 15-24, since it is within this complex of clauses that the word group that introduces Onesimus
into the discourse is found. Clausal structure, in our analysis, consists of subject, predicate,
complement and adjunct. The predicate is the major unit of the Greek clause, but whether the subject
is grammaticalized, reduced or implied in relation to that subject, and its placement in first or later
position in the clause are very important for the information structure. We analyze this information
structure in terms of prime and subsequent elements in the clause. Once the clauses have been
analyzed, we examine the relations of the clauses at the paragraph level. A primary clause usually
has a finite verb form or, in the absence of a finite verb, a word group functioning similarly. These
clauses are used to convey the main thread or backbone of the discourse. Secondary clauses are
usually relative, participle and infinitive clauses. These clauses are used to add further specification
at the present point in the discourse, and do not progress the discourse in the horizontal plane. The
paragraph level, under the mode of discourse, is the appropriate point in the model to analyze the
connection and relationship between clauses. These connections determine the level of a clause. For
example, a clause with a finite verb form that might be classified as primary, but is clearly connected
to a secondary clause (e.g. a relative clause), is classified as functioning at the secondary level. The
appendix contains the clause and paragraph level analysis of Philemon 10-14. Figure 4 presents a
visual representation of this annotation (see appendix, fig. 10 for XML annotation of clauses).
Clauses are represented by boxes. The single primary clause (c15), SDUDNDOZVHSHULWRXHMPRX
WHYNQRX, is placed on the left-hand side of the diagram. All secondary level clauses connected to this
clause are joined to the right-hand of this main clause. Other clauses not connected directly to c15
but to another of the secondary clauses are placed to the right of the clause to which they connect,
unless they are embedded within the clause (e.g. clause c23 SRLKVDL is a complement of clause c22)
where they are positioned within the box of the clause in which they are embedded.
c16 R`QHMJHYQQKVDHMQWRLGHVPRL©2QKYVLPRQWRYQSRWHY
VRLD>FUKVWRQQXQLGHVRLNDLHMPRLHX>FUKVWRQ
c17 R`QDMQHYSHP\DYVRL
DXMWRYQ
c18 WRXWRH>VWLQWDHMPD
VSODYJFQD
c19 R`QHMJZHMERXORYPKQ
c20 SURHMPDXWRQ
NDWHYFHLQ
c21 L^QDX-SHUVRXPRL
GLDNRQKHMQWRLGHVPRL
WRXHXMDJJHOLYRX
c22 FZULGHWKVK
JQZYPK
RXMGHQKMTHYOKVD
c15 SDUDNDOZVH
SHULWRXHMPRX
WHYNQRX
 c23 SRLKVDL 
c24 L^QDPKZ-NDWDDMQDYJNKQ
WRDMJDTRYQVRXK?DMOOD
NDWDH-NRXYVLRQ
Fig. 5. Display of clause level and connections for Philemon 10-14
What is important for this paper is the analysis of word group 54 within such a paragraph
analysis. Clause 15 is the primary clause of the unit that consists of clauses 15-24, with a series of
four secondary clauses related to it, three of them linked by relative pronouns (clauses 16, 17 and 19).
Clause 15, ‘I urge you concerning my child', refers to the three participants that we mentioned
before: Paul, this time implied through the use of the first-person singular verb; Philemon, referred to
by use of a reduced pronoun; and a figure cited in a prepositional phrase (adjunct) as ‘my child', not
yet named. The author then defines this child in the three relative secondary clauses. The first one
has complement-predicate-adjunct-complement structure, with a split complement because of the
relative pronoun (R^Q). Again, Paul is the implied subject of the clause, but it is here that Onesimus is
introduced as the complement of this secondary clause. A number of further statements are made
about this ‘child', Onesimus, though without using his name, in the further secondary relative clauses
all with Paul as the implied subject (clauses 17 and 19) and the further secondary clauses (clauses 18,
20, 21).
In other words, even though Onesimus has often been analyzed as a major participant in the
book of Philemon, with much secondary discussion concerned with who he is and how he relates to
Paul and Philemon, the discourse structure does not confirm this analysis. Paul especially, but also
Philemon, are the major participants, and their relationship is in fact the major element of the tenor
component. This is confirmed by how many times Paul and Philemon, whether in grammaticalized,
418
reduced or implied form, are the subjects or complements of the primary clauses, but also secondary
clauses.
Participant Primary clause Secondary
clause
Total
1 Paul 26 14 40
2 Jesus Christ 9 3 12
3 Timothy 2 0 2
4 Philemon 21 13 34
5 Us 4 0 4
6 Apphia 2 0 2
7 Archippus 2 0 2
8 You (plural) 4 0 4
9 God 3 0 3
10 The Saints 1 1 2
11 Onesimus 4 9 13
Fig.6 Summary of participant reference according to clause level in Philemon
Figure 6 shows a summary of the number of references to each of the participants annotated in
the discourse. Onesimus is confined for the most part to peripheral status as a complement of
secondary clauses, often in reduced or implied form (9 of the 13 references to Onesimus occur in
secondary clauses and 8 of these references occur in the complement slot of the clause). He is not
even introduced by name the first time he is referred to, as are the other major participants, but is
referred to in a reduced form by means of a noun, ‘child', that fulfils a discourse function of
characterizing Onesimus in relation to the major participants. The implications of this analysis for
interpretation of the discourse are significant. The focus of future analysis will need to be upon the
major participants as supported by the discourse itself, Paul and Philemon, recognizing that, as
important as Onesimus may be in terms of a catalytic function in their interpersonal relations, his
function is secondary to the primary relation. Further, instead of the status of Onesimus as a slave or
a runaway, and whether or not he took property that was not his, the major issue seems instead to be
how it is that Paul as apostle relates to Philemon as significant figure in the church to which Paul
writes. References to Paul in the letter according to clause component are: subject 8x, predicate 16x,
complement 12x and adjunct 6x. For Philemon, these figures are: subject 1x, predicate 9x,
complement 17x and adjunct 7x. These basic figures require filtering according to the causality
(voice of verbal forms, indicating actor or patient status) and position within a word group (see
above). Some have noted that Paul uses a number of discourse techniques in his communication with
the church there, although wanting to hold back from characterizing these techniques as at all
manipulative or forcefully persuasive (Wilson 1992; Fitzmyer 2000; cf. Porter 1999). However, this
notion seems to be much closer to what the discourse supports rather than examining this letter as an
exposition of slave and master relations in the ancient world, in which Paul has an incidental interest.
3. Revelation, semantic-domains and cohesion
The book of Revelation is much larger than Philemon. Philemon has a total of about 335 words
compared to the approximately 9850 in Revelation. This extra length provides enough discourse
scope for development of a number of different patterns. The way several of these patterns relate to
each other is what we would like to examine in this second example.
As discussed in the introduction, a common problem for both discourse analysis and traditional
referential corpus linguistics is the analysis and summarization of large amounts of data in order to
carry out a full analysis of even a single feature of discourse, such as lexical cohesion, over even a
medium sized discourse, such as the book of Revelation. Considerable amounts of information must
be tracked. The standard reference for the study of cohesion in discourse is the work of Halliday and
Hasan (1976). This has served as the basis for a number of recent discourse annotation schemes
(Wilson and Thomas 1997) and theoretical studies (Hoey 1991). In addition, a number of more
computationally orientated studies (Morris and Hirst 1991; Kozima and Furugori 1994) have
developed algorithmic approaches to the study of lexical cohesion. One of the ways the study of
Hellenistic Greek, particularly the Greek of the New Testament, is more advanced than comparable
study of English is in the availability of a semantic-domain lexicon (Louw and Nida 1988; Nida and
419
Louw 1993). This lexicon distinguishes 93 broad semantic domains, covering all of the semantic
fields of the Greek found in the New Testament. Within these 93 domains are a large number of subdomains.
Words are categorized within these domains and sub-domains, with some words being
found within as many as four or five domains, except for such function words as prepositions, which
are categorized in as many as six domains. It is not appropriate to discuss the shortcomings of this
important work, although one of the results of our study is ideally to construct a more principled set
of domains that are more closely linked to patterns of New Testament usage. However, in the
meantime, this tool has proved invaluable in the study of lexis and cohesion in the New Testament.
For our study of lexical cohesion in the book of Revelation, we focus upon content words only
(verbs, nouns, adjectives, and adverbs). Each of these word types in the text was annotated with its
major domain number from the lexicon, with no attempt to disambiguate multiple classifications. We
then made use of a simple algorithmic technique to identify the number of semantic chains found
within a fifty word window measured at intervals of ten words throughout the text. The first window
begins at word 1 and extends to word 50, the second window at word 11 extending to word 60, etc.
At each interval, the number of domains exhibiting chains of five or more words in the subsequent
window are noted. For example, the first window (words 1-50) has two chains, one from domain 33,
communication, and one from domain 93, names of persons and places. The following window
(words 11-60) has only a single chain from domain 33. In a number of ways this fits the pattern that
one might expect at the beginning of a book. We have found that domain 33 is probably the most
common domain in New Testament books, especially the epistles, and indicates what one might
expect in a theological text, that is, the desire to communicate is activated, and often this chain is
maintained through much of the discourse. It is also worth noting, though not necessarily of great
significance, that the beginning introduces at least some of the people or places involved in the
action. This is a relatively low-level use of vocabulary. Figure 7 contains a line-plot with the position
in the discourse in terms of word number on the x-axis and the number of significant semantic chains
(domains with five or more words) at each point indicated on the y-axis. This graph, through the
alternation of peaks and troughs of semantic chain usage, provides a helpful macro-view of lexically
cohesive units within the book of Revelation (and perhaps insights into thematization), and serves as
a tool to highlight sections for closer micro-level analysis. The genre of the book of Revelation is a
complex one, since it has a mix of text-types. The peaks and troughs of the semantic chains are
coordinated with the major structural divisions of the text-types. For example, within a new text-type,
such as the letter section, a recurring pattern is the use of a semantic chain that peaks and then
recedes. This seems to be a consistent pattern, in which the new text-type activates a set of related
terms. The text-type continues, even though the semantic chains may not continue, before a new texttype
activates a new set of domains.
Fig. 7 Lexical cohesion at 10 word intervals across the book of Revelation
Two micro-level analyses serve to illustrate the kinds of observations that emerge from this
close study. The first is from Revelation 4 and the other Revelation 16. The window beginning at
word 1611 (corresponding to Rev. 3.22, the last verse of chapter 3) seems to mark a significant shift
in the discourse. The preceding sample windows have had only one or two semantic chains. One of
these, domain 33 (communication), is relatively low-level in significance, but the other chains
anticipate the next structural unit, by introducing words from semantic domain 12 (supernatural
beings and powers) and domain 37 (control, rule). This confirms a similar finding concerning the
way in which grammatical features such as tense-form and voice anticipate and mark transitions in
discourse (Biber, Conrad, and Reppen 1998; Porter 1994). Beginning with word 1611, we find a long
chain of domain 12 that extends over 200 words. A number of additional but shorter chains are also
activated at this point at the opening of the unit. These include a continuation of semantic domain 37,
and the introduction of domains 85 (existence in space), 6 (artefacts), 60 (number), and 14 (physical
events and states). Shorter chains dispersed throughout this paragraph (extending from words 1611 to
1931) include semantic domains 2 (natural substances), 4 (animals), 67 (time), 57 (possess, transfer,
exchange), 41 (behavior and related states). These shorter chains are usually between two and four
420
windows long, a maximum of 80 words. This is a semantic-chain based description of the unit. The
scene itself in the book of Revelation marks a transition from the epistolary sections (the so-called
letters to the seven churches) to the apocalyptic vision that constitutes much of the rest of the book.
Most commentators would see these as the two major sections of the text. The first scene of this
apocalyptic section is the dramatic appearance of the heavenly throne room. In this room, there is a
majestic throne surrounded by other thrones with heavenly beings seated upon them. These beings
have crowns and jewels and golden lamps. These two different descriptions are complimentary. The
latter is an aesthetically based account of the various elements depicted. The semantic-chain
description is an attempt to quantify and even explain the aesthetic account by classifying the data
that make up the account.
The second section is in many ways similar to the first. Here we are concerned with the unit
from word 6621 to word 6751, which is preceded by a 200 word section with virtually no significant
semantic chains. In contrast to the section discussed above, there is no preparatory transitional
material that anticipates this unit. The semantic chains in this section come from semantic domains
15 (linear movement), 13 (be, become, exist, happen), 14 (physical events), 41 (behavior and related
states), 57 (possess, transfer, exchange), 78 (degree), 79 (features of objects), 85 (existence in space),
and 91 (discourse markers). The semantic domains in the Louw-Nida lexicon are arranged along a
continuum, with contiguous domains having overlapping semantic features. Thus a given word may
be classified within several contiguous domains. That is the case in this particular episode, as
demonstrated by the domains activated. The result is that the significance of these chains may be less
than these data first appear, since a number of the chains activated may contain the same word. In
other words, there are probably 4 or 5 major semantic areas activated here, rather than the 7 or 9
noted above. Also in contrast with the example above, this intense confluence of semantic chains
here is relatively short. Attention to the text reveals that the section marks the climax of a unit
beginning around word 6300 (Rev. 16.1), which begins: ‘I heard a loud voice from the temple saying
to the seven angels, “Go pour out the seven bowls of wrath upon the earth”'. Then follows the
account of each of the seven angels distributing the contents of their bowls over the earth. This
section describes the action and result of the seventh angel, completed with a voice from the throne
saying that ‘It is done’ (Rev. 16.17). This is followed by dramatic environmental disruption,
including lightening, thunder, and earthquakes. Commentators have noted the climactic nature of this
event in the discourse, which can be confirmed by the pronounced accumulation of semantic chains.
We introduced this part of our paper by talking about how such semantic chain patterning
applied to textual cohesion. Cohesion is a concept that seems to be useful on at least two levels. One
is in terms of the macro-patterns of usage that unite an entire discourse. We have seen that there are a
number of semantic chains that are activated throughout the book of Revelation, such as domain 33
(communication). There are a number of domains that function at this low level to activate a number
of basic concepts that unite the discourse. Cohesion also seems to function in terms of the
relationship between various units or paragraphs in the discourse. Part of the cohesive function of
semantic chains is to activate accumulations of chains at appropriate times. These serve the function
of delineating the units of the discourse by closing and opening units, and marking transitions. The
boundary here between cohesion and prominence therefore is not a firm one, since those semantic
chains that are marked as prominent because of their relatively infrequent but high-level activation
only become prominent when they are seen in relation to the semantic chains that cohede the
discourse.
4. Conclusion
In this paper, we have tried to show some of the practical differences that working with an
epigraphic language such as Hellenistic Greek makes for corpus linguistics. The kind of microanalysis
that we have briefly introduced in the above two examples seems to be required if one is to
utilize the limited corpus size from the ancient world. Micro-analysis allows for close scrutiny of a
finite set of elements, but each is seen to function within a variety of levels of discourse. The
definition and classification of these levels enables each element to provide maximal data for
interpretation. Such micro-analysis can only take place, however, if the corpus of texts is richly
annotated to provide the largest amount of information as possible. In the first example, that of
Onesimus and his relation to Paul and Philemon, we saw that at one level—that of the word group—
Onesimus seems to be grammaticalized in similar fashion to the other two major participants.
However, when the word group in which he is introduced is placed in the larger frame of being a
component of paragraph structure, his prominence fades, and he is seen to be relegated to a
peripheral participant role. In the examples of Revelation, we have examined a more narrowly
circumscribed set of features in terms of the entire discourse. Here we have noted that by using an
intensive semantic-domain study, and correlating this with paragraph boundaries, one can observe
421
how semantic chain shifts are coordinated from paragraph to paragraph. The result is a clearer
demarcation of how the major subject matter of each paragraph unit is shifted and developed, but
also how cohesion is both created within the individual paragraph units and extended over the entire
discourse. One of the facts of corpus-based studies as usually conceived is that they can generate
huge quantities of data for analysis, and this is thought to be a desirable feature, allowing more
precise generalizations to be reached on the basis of a larger sample surveyed. One of the complaints
about discourse analysis is that it generates too much data to study within the confines of a single
analysis. These two are not necessarily reconcilable, especially when a non-finite corpus is involved.
Even for an epigraphic language such as Hellenistic Greek, an abundance of data can be generated
for analysis. In the light of the finite corpus size, however, these data are to be desired, and to be
maximized for their use in corpus-based discourse studies.
References
DeRose S J, Durand D G, Mylonas E, Renear A H 1990 What is text, really? Journal of Computing in
Higher Education 1(2): 3-26.
Fitzmyer J A 2000 The letter to Philemon. New York, Doubleday.
Halliday M A K, Hasan R 1976 Cohesion in English. London, Longman.
Hoey M 1991 Patterns of lexis in text. Oxford, Oxford University Press.
Kozima H, Furugori T 1994 Segmenting narrative text into coherent scenes. Literary and Linguistic
Computing 9(1): 13-19.
Leech G 1994 Corpus annotation schemes. Literary and Linguistic Computing 8(4): 275-281.
Louw J P, Nida E A 1988 Greek–English lexicon of the New Testament based on semantic domains. 2
vols.; New York, United Bible Societies.
McCarty W L 1996 Peering through the skylight: towards an electronic edition of Ovid's
Metamorphoses. In Hockey S, Ide N (eds), Research in humanities computing 4: selected papers
from the ALLC/ACH conference, Christ Church, Oxford, April 1992. Oxford, Clarendon Press,
pp 240-262.
Morris J, Hirst G 1991 Lexical cohesion computed by thesaural relations as an indicator of the
structure of text. Computational Linguistics 17: 21-48.
Nida E A, Louw J P 1993 Lexical semantics of the Greek New Testament. Atlanta, Scholars Press.
O'Donnell M B 1999 The use of annotated corpora for New Testament discourse analysis: a survey of
current practice and future prospects. In Porter S E, Reed J T (eds), Discourse analysis and the
New Testament: results and applications. Sheffield, Sheffield Academic Press, pp 71-116.
Pearson B W R 1999 Assumptions in the criticism and translation of Philemon. In Porter S E, Hess R
S (eds), Translating the Bible: problems and prospects. Sheffield, Sheffield Academic Press, pp
253-280.
Porter S E 1994 Idioms of the Greek New Testament. Sheffield, Sheffield Academic Press.
Porter S E 1999 Is Critical Discourse Analysis Critical? An Evaluation Using Philemon as a Test
Case. In Porter S E, Reed J T (eds), Discourse analysis and the New Testament: results and
applications. Sheffield, Sheffield Academic Press, pp 47-70.
Porter S E, O'Donnell M B 2001 Theoretical issues for corpus linguistics raised by the study of
ancient languages. In Proceedings of Corpus Linguistics 2001.
Renear, A H, Mylonas E, Durand D 1996 Refining our notion of what text really is: the problem of
overlapping hierarchies. In Hockey S, Ide N (eds), Research in humanities computing 4: selected
papers from the ALLC/ACH conference, Christ Church, Oxford, April 1992. Oxford, Clarendon
Press, pp 263-280.
Wilson A 1992 The pragmatics of politeness and Pauline epistolography: a case study of the letter of
Philemon. JSNT 48: 107-19.
Wilson A, Thomas J 1997 Semantic annotation. In Garside R, Leech G, McEnery A (eds), Corpus
annotation: linguistic information from computer text corpora. London, Longman, pp 53-65.
Appendix: annotation examples
<wg:group id="wg3" head="w9">
<w id="w9">)LOKYPRQL</w>
<w id="w10" modify="w11" rel="specify">WZ</w>
<w id="w11" modify="w9" rel="define">DMJDSKWZ</w>
<w id="w12" join=”w11” to=”w13” rel="connect">NDL</w>
<w id="w13" modify="w9" rel="define">VXQHUJZ</w>
422
<w id="w14" modify="w13" rel="qualify">K-PZQ</w>
</wg:group>
Fig. 8. Annotation of word group 3 (Philemon 1b)
<wg:group id="wg54" head="w145">
<w id="w145">¨2QKYVLPRQ</w>
<w id="w146" modify="w149" rel="specify">WRYQ</w>
<w id="w147" modify="w149" rel="qualify">SRWHY</w>
<w id="w148" modify="w149" rel="qualify">VRL</w>
<w id="w149" modify="w145" rel="define">D>FUKVWRQ</w>
<w id="w150" modify="w156" rel="qualify">QXQL</w>
<w id="w151" join=”w149” to=”w156” rel="connect">GH</w>
<w id="w153" modify="w156" rel="qualify">VRL</w>
<w id="w154" join=”w153” to=”w155” rel="connect">NDL</w>
<w id="w155" modify="w156" rel="qualify">HMPRL</w>
<w id="w156" modify="w145" rel="define">HX>FUKVWRQ</w>
</wg:group>
Fig. 9. Annotation of word group 54 (Philemon 10b-11)
<cl:clause id="c15" level="primary">
SDUDNDOZVHSHULWRXHMPRXWHYNQRX
</cl:clause>
<cl:clause id="c16" level="secondary" connect="c15">
R`QHMJHYQQKVDHMQWRLGHVPRL©2QKYVLPRQWRYQSRWHYVRLD>FUKVWRQQXQLGHVRLNDL
HMPRLHX>FUKVWRQ
</cl:clause>
<cl:clause id="c17" level="secondary" connect="c15">
R`QDMQHYSHP\DYVRLDXMWRYQ
</cl:clause>
<cl:clause id="c18" level="secondary" connect="c17">
WRXWRH>VWLQWDHMPDVSODYJFQD
</cl:clause>
<cl:clause id="c19" level="secondary" connect="c15">
R`QHMJZHMERXORYPKQ
<cl:clause id="c20" level="secondary" connect="c19">
SURHMPDXWRQNDWHYFHLQ
</cl:clause>
</cl:clause>
<cl:clause id="c21" level="secondary" connect="c19">
L^QDX-SHUVRXPRLGLDNRQKHMQWRLGHVPRLWRXHXMDJJHOLYRX
</cl:clause>
<cl:clause id="c22" level="secondary" connect="c15">
FZULGHWKVKJQZYPKRXMGHQKMTHYOKVD
<cl:clause id="c23" level="secondary" connect="c22">
SRLKVDL
</cl:clause>
</cl:clause>
<cl:clause id="c24" level="secondary" connect="c22">
L^QDPKZ-NDWDDMQDYJNKQWRDMJDTRYQVRXK?DMOODNDWDH-NRXYVLRQ
</cl:clause>
Fig. 10. Annotation of clause level and connection (Philemon 10-14)

423
Spelling out the optionals in translation: a corpus study
Maeve Olohan
Centre for Translation and Intercultural Studies, UMIST
PO Box 88, Manchester M60 1QD
maeve.olohan@umist.ac.uk
Abstract
While the use of translations in parallel corpora, mostly for the purposes of contrastive linguistic
analysis, is relatively well established, the analysis of translated language as an object of study in its
own right has only fairly recently been made possible through the development of corpus resources
designed specifically for this purpose. The Translational English Corpus (TEC) at UMIST was the first
corpus consisting exclusively of translations, in English, from a variety of source languages and text
types. Much of the research carried out thus far using TEC (e.g. Laviosa-Braithwaite 1996, Kenny
1999 and 2000) has been interested in identifying and confirming features of translated language such
as explicitation, normalisation, simplification and levelling out (Baker 1996). This kind of research is
based on the assumption that, by retrieving and analysing data from TEC and a comparable corpus (e.g.
the British National Corpus), it is possible to pinpoint consistent differences in syntactic or lexical
patterning between translated English and original English. Some of these may arise from deliberate
translation strategies on the part of the translator who wishes to make his/her text more explicit, to
normalise or simplify etc. However, TEC can also be used as a means of identifying linguistic
patterning which translators will not have been aware of producing, but which occurs as a result of the
complex nature of the translation activity itself.
Against this background, this paper presents an investigation of explicitation in translation.
Preliminary studies using TEC and a subcorpus of the BNC (Burnett 1999, Olohan and Baker 2000)
have shown that patterns of use of the optional that with reporting verbs are rather different in
translated English than in original English, with translated English very much favouring the use of that,
even in contexts which do not warrant it, e.g. for purposes of disambiguation or for the signalling of
more formal style. This paper will present further analysis of optional syntactic features in English and
their occurrence in TEC and the BNC, test the hypothesis that translated English displays a higher
incidence of a range of optional syntactic features than is observed in a comparable corpus of original
English, and that this is direct evidence of subconscious processes of explicitation in translation.
1. Corpus-based translation studies
Corpus-based translation studies is a relatively new area of research within translation studies,
motivated by an interest in the study of translated texts as instances of language use in their own right.
This is in contrast to the not uncommon perception of translations as ‘deviant’ language use, a view
which has generally led to the exclusion of translated texts from most ‘standard’ or ‘national’ corpora
(Baker 1999). While translations have been seen as useful in parallel bilingual or multilingual corpora,
this has usually been for contrastive linguistic analysis which has studied the relationship between
source and target language systems or usage. Parallel corpora are naturally also of interest to the
translation scholar as they facilitate investigation of the relationship between a translation and its
source. Recent work using corpora in translation studies has, however, been more concerned with
building corpora of translations so that the use of language in translations may be studied. The first
corpus of this nature was the Translational English Corpus at UMIST (described below) which, since
its inception, has provided the impetus and inspiration for a number of similar projects for other
languages, including Italian, German, Spanish, Finnish, Catalan and Brazilian Portuguese.
One of the fundamental concepts in corpus-based translation studies has been the notion of
comparable corpus, defined by Baker (1995: 234) as ‘two separate collections of texts in the same
language: one corpus consists of original texts in the language in question and the other consists of
translations in that language from a give source language or languages…both corpora should cover a
similar domain, variety of language and time span, and be of comparable length'. Baker's initial
groundbreaking work posited a number of features of translation, or ‘translation universals', which
could be investigated using comparable corpora (Baker 1996). While the term universal in this context
is somewhat controversial, not least because of the practical difficulties involved in testing whether
something holds true across diverse languages (for many of which corpora of translations and/or
original writing do not exist), it has been suggested, for example, that translations tend to be more
explicit on a number of levels than original texts, and that they simplify and normalise or standardise in
a number of ways. Much of the corpus-based work carried out to date has focused on syntactic or
424
lexical features of translated and original texts which may provide evidence of these processes of
explicitation, simplification or normalisation. It should be stressed that, while translators may at times
consciously strive to produce translations which are more explicit or simplified or normalised in some
way, the use of comparable corpora also allows us to investigate aspects of translators’ use of language
which are not the result of deliberate, controlled processes and of which translators may not be aware.
2. Corpus data
The Translational English Corpus is a corpus of translated English held at the Centre for Translation
and Intercultural Studies at UMIST. It was designed specifically for the purpose of studying translated
texts and it consists of contemporary written translations into English of texts from a range of source
texts and languages. At the time of writing, it has over 6.4 million words. TEC consists of four text
types – fiction, in-flight magazines, biography and newspaper articles – with fiction representing 82%,
and biography and fiction together making up 96% of the corpus. The translations were published from
1983 onwards and were produced by translators, male and female, with English as their native
language or language of habitual use.
The corpus of original English put together for this particular study is a subset of the BNC made up
of texts from the imaginary domain. It is thus comparable in terms of genre and publication dates (from
1981 onwards). The texts have been produced by native speakers of English, both male and female. A
minor difference between the two corpora which is not significant for current investigations is that TEC
consists of full running texts whereas some of the BNC texts are extracts (some as long as 40,000
words). There is a little variation in size between the two corpora with TEC now slightly bigger than
the BNC corpus. As TEC continues to grow, new texts will be added to the BNC subcorpus so that the
corpora remain comparable in all respects.
The data discussed here was extracted from these two untagged corpora using Wordsmith Tools
V.3.0.
3. Explicitation
The analyses reported on here arose from an interest in studying processes of explicitation in
translation, where explicitation refers to the spelling out in target text of information which is only
implicit in a source text. This has long been considered a feature of translation and has been
investigated by a number of scholars (e.g. Vanderauwera 1985, Blum-Kulka 1986; Laviosa-Braithwaite
1996; Laviosa 1998; Baker 1995, 1996) who have identified different means or techniques by which
translators make information explicit, e.g. using supplementary explanatory phrases, resolving source
text ambiguities, making greater use of repetitions and other cohesive devices. This current research
focuses, in so far as this is possible, on subconscious processes of explicitation and their realisation in
linguistic forms in translated texts. Since the starting point is the linguistic form, we have concentrated
on optional syntactic features, hypothesising that, if explicitation is genuinely an inherent feature of
translation, translated text might manifest a higher frequency of the use of optional syntactic elements
than original writing in the same language, i.e. translations may render grammatical relations more
explicit more often – and perhaps in linguistic environments where there is no obvious justification for
doing so – than original writers.
4. Analysis of optional syntactic features in English
Linguists may present the optional syntactic features of English in different ways, but we opted to
base this study on Dixon's (1991: 68-71) omission conventions for English, presented in summary
form as follows:
A. Omission of subject NP
B. Omission of complementiser that
C. Omission of relative pronoun wh-/that
D. Omission of to be from complement clause
E. Omission of predicate
F. Omission of modal should from a THAT complement
G. Omission of preposition before complementisers that, for and to
H. Omission of complementiser to
I. Omission of after/while in (after) having and (while) *ing
J. Omission of in order
425
These features span a range of linguistic phenomena, from frequently occurring relative pronouns to
much less common constructions (e.g. to be in complement clause), and that they do not focus
exclusively on optionality of omission. As will be obvious from the discussion below, they also vary
considerably in terms of their identification and quantifiability in a corpus which is neither tagged nor
parsed. In some instances, as can be seen in 4.3, 4.4, 4.9 and 4.10, omission is difficult to measure but
occurrence, i.e. inclusion, can be traced and compared across corpora to give an indication of
differences in usage of the longer surface form between corpora.
4.1. Omission of subject NP
This refers to omission of a subject NP in a number of circumstances, e.g. under coordination, in
subordinate time clauses, from an ING complement clause or from a modal (FOR) TO complement
clause. There is no obvious way of finding instances of these in a corpus which is not tagged for parts
of speech.
4.2. Omission of complementiser that
Dixon states that ‘the initial that may often be omitted from a complement clause when it
immediately follows the main clause predicate (or predicate-plus-object-NP where the predicate head is
promise or threaten’ (1991: 70). An extensive analysis of the use of that/zero-connective with reporting
verbs SAY and TELL, with reference to TEC and BNC, is presented in Olohan and Baker (2000). The
results are summarised in Tables 1 and 2 below, which present both the absolute values (i.e.
occurrences) and the percentages for each form:
Form say
(TEC)
say
(BNC)
said
(TEC)
said
(BNC)
says
(TEC)
says
(BNC)
saying
(TEC)
saying
(BNC)
that
316
55.5%
323
26.5%
267
46.5%
183
19.2%
116
40.4%
64
12.8%
76
67.3%
142
43.0%
zero
253
44.5%
895
73.5%
307
53.5%
771
80.8%
171
59.6%
435
87.2%
37
32.7%
188
57.0%
Table 1: SAY + that/zero in BNC and TEC
Form tell
(TEC)
tell
(BNC)
told
(TEC)
told
(BNC)
tells
(TEC)
tells
(BNC)
telling
(TEC)
telling
(BNC)
that
247
62.8%
300
38.2%
353
60%
584
43.6%
55
68.7%
28
37.5%
64
73.6%
85
42.3%
zero
146
37.2%
486
61.8%
233
40%
755
56.4%
25
31.3%
52
62.5%
23
26.4%
115
57.7%
Table 2: TELL + that/zero in BNC and TEC
It is immediately clear that the that-connective is far more frequent in TEC than in BNC. With the
exception of said and says, that occurs more often than zero for all forms of SAY and TELL in TEC. By
contrast, the zero-connective is more frequent for all forms of both verbs in the BNC corpus. These
differences have been proven to be statistically significant. Furthermore, the results of the SAY and
TELL study were consistent with findings by Burnett (1999) who reviewed use of the verbs SUGGEST,
ADMIT, CLAIM, THINK, BELIEVE, HOPE and KNOW in TEC and BNC . While that study did not include all
forms of these verbs, the data available shows that the that-connective is far more common than the
zero-connective in translated than in original English for forms of all seven of the verbs investigated.
The hypothesis that the optional that in reporting constructions occurs proportionately more frequently
in translated texts than in original English texts is thus supported. Although Olohan and Baker (2000)
highlight the relative vagueness with which omission and inclusion are accounted for in the linguistics
literature, and the lack of guidance on this in reference works for users of English, there are clear
patterns of usage in contemporary English writing as evidenced in the BNC corpus, and there is an
equally clear contrast between these patterns and those perceived in translated English.
A brief analysis of one of the verbs suggested by Dixon, namely PROMISE, serves as further
illustration and corroboration. Table 3 and Figure 1 below show that, although the number of instances
of promise + that/zero were almost identical in the two corpora (135 in BNC and 131 in TEC), the
relationship between that and zero in TEC (that = 67.9%, zero = 32.1%) is almost directly inverse to
that in BNC (that = 34.1%, zero = 67.9%).
426
That/zero Total
zero that
Corpus BNC Count 89 46 135
% within Corpus 65.9% 34.1% 100.0%
% within That/zero 67.9% 34.1% 50.8%
% of Total 33.5% 17.3% 50.8%
Corpus TEC Count 42 89 131
% within Corpus 32.1% 67.9% 100.0%
% within That/zero 32.1% 65.9% 49.2%
% of Total 15.8% 33.5% 49.2%
Table 3: PROMISE + that/zero in BNC and TEC
Figure 1: occurrences of PROMISE + that/zero in BNC and TEC
A breakdown of each lexical item (Table 4 and Figure 2) shows that this holds true for all forms of
the verb, although some have low occurrences in general (e.g. promises + that/zero occurs only twice
in TEC and not at all in BNC).
Figure 2: All forms of PROMISE + that/zero in BNC and TEC
427
That/zero
Form zero that Total
Count 38 19 57
% within Corpus 66.7% 33.3% 100.0%
% within That/zero 64.4% 41.3% 54.3%
BNC
% of Total 36.2% 18.1% 54.3%
Count 21 27 48
% within Corpus 43.8% 56.3% 100.0%
% within That/zero 35.6% 58.7% 45.7%
promise Corpus
TEC
% of Total 20.0% 25.7% 45.7%
Count 1 1 2
% within Corpus 50.0% 50.0% 100.0%
% within That/zero 100.0% 100.0% 100.0%
promises Corpus TEC
% of Total 50.0% 50.0% 100.0%
Count 46 20 66
% within Corpus 69.7% 30.3% 100.0%
% within That/zero 69.7% 27.8% 47.8%
BNC
% of Total 33.3% 14.5% 47.8%
Count 20 52 72
% within Corpus 27.8% 72.2% 100.0%
% within That/zero 30.3% 72.2% 52.2%
promised Corpus
TEC
% of Total 14.5% 37.7% 52.2%
Count 5 7 12
% within Corpus 41.7% 58.3% 100.0%
% within That/zero 100.0% 43.8% 57.1%
BNC
% of Total 23.8% 33.3% 57.1%
Count 9 9
% within Corpus 100.0% 100.0%
% within That/zero 56.3% 42.9%
promising Corpus
TEC
% of Total 42.9% 42.9%
Table 4: All forms of PROMISE + that/zero in BNC and TEC
4.3. Omission of relative pronoun wh-/that
This frequently occurring construction is difficult to measure in an untagged corpus. Thus far, only
total counts of occurrence of which have been taken, with 11,201 in BNC and 23,607 in TEC. A first
step in discarding irrelevant instances was to identify sentence-initial and sentence-final/clause-final
which. Their removal leaves 10,457 concordance lines in BNC and 22,483 in TEC, indicating
considerably higher usage of which in TEC. Obviously further detailed analysis of these instances is
required to identify the occurrences in relative clauses where the coreferential NP is not in subject
function in the relative clause, i.e. where omission could have taken place.
4.4. Omission of to be from complement clause
From a very frequent feature above, we come to a very infrequent structure. Dixon is referring here
to the omission of to be with ‘some verbs taking a Judgement TO complement clause, whose VP begins
with be’ (1991: 70), with an example of thought + to be + modifier. Both THINK + to be and FIND + to
be were investigated in the corpora (see Table 5). The most common occurrence in both corpora was
for the past tense forms (thought and found), and TEC exhibits a greater tendency overall to include to
be, but the number of occurrences overall was very small in both corpora.
Form BNC TEC
THINK (+ *)(+ *) to be 2 6
FIND (+ *)(+ *) to be 4 7
Table 5: think + to be and find + to be in BNC and TEC
4.5. Omission of predicate
The omission of the predicate in coordinated clauses is difficult to capture in an untagged corpus
and this has therefore not yet been investigated.
4.6. Omission of modal should from a THAT complement
This refers to the omission of modal should from a THAT complement with examples of verbs
ORDER and SUGGEST. Neither is particularly common, and both occur predominantly in the past tense
form (ordered and suggested). A greater proportion of omission is seen in TEC (see Table 6).
428
Form BNC TEC
ORDER + that + should 1 6
ORDER + that + zero 2 7
SUGGEST + that + should 19 19
SUGGEST + that + zero 43 58
Table 6: ORDER and SUGGEST + that + should/zero in BNC and TEC
4.7. Omission of preposition before complementisers that, for and to
Some transitive verbs with a preposition as last element in their lexical form which may take a
complement clause in object function will omit the preposition before that, for and to, e.g. he confessed
to the crime, he confessed to strangling her, but he confessed that he had strangled her. This is not an
optional omission and is therefore not of interest in this study.
4.8. Omission of complementiser to
According to Dixon, the complementiser to is optional following HELP or KNOW. The form help was
analysed, first discarding all uses of help as noun, as reflexive verb, verb + ING complement and verb +
preposition, and then looking at occurrences of help (*) (*) to in detail (Table 7).
Form BNC TEC
Total
occurrences
Relevant
occurrences
Total
occurrences
Relevant
occurrences
Occurrences of help 2374 300 1792 365
help + to 62 26 72 38
help + * + to 67 50 98 80
help +* + * + to 19 3 35 19
Total help (+*) (+*) + to 79 137
help (+*) (+*) + zero 229 228
Table 7: help (+*) (+*) + to in BNC and TEC
This data tells us that although the word form help is more frequent in TEC, its verbal use in both
corpora is quite similar with help (+*) (+*) + to/zero occurring slightly more often in TEC than in
BNC, of which the complementiser to is used in 37.5% of TEC instances, compared with 26% of the
BNC occurrences.
4.9. Omission of after/while in (after) having + participle and (while) *ing
As in 4.3 and 4.4 above and 4.10 below, we can more readily measure occurrence of these features
rather than omission. Concordances of while *ing are pruned, discarding constructions such as all the
while *ing, after/in/for a while *ing, worth your while *ing. The while *ing construction is much more
frequent in TEC overall and in relation to the gerundial use (Table 8).
Form BNC TEC
Total while *ing concordances 150 360
Relevant concordances 138 330
Table 8: while *ing in BNC and TEC
A count of after *ing *ed (which obviously does not take irregularly formed past participles into
account) also shows a tendency for TEC to use this construction more frequently than BNC (Table 9).
Form BNC TEC
after *ing *ed 11 65
Table 9: after *ing *ed in BNC and TEC
4.10. Omission of in order
According to Dixon, in order is usually omitted before to and may occasionally be omitted before
for or that. While the investigation of every instance of the items to, that and for to see whether an in
order has been omitted is not practical, we can easily measure usage of in order to, in order for and in
order that and compare results from the two corpora. This investigation yields the following (Table
10):
429
Form BNC TEC
in order to 250 1225
in order for 1 14
in order that 12 18
Total 263 1257
Table 10: in order to/for/that in BNC and TEC
This does not conclusively prove that in order has been omitted more often in BNC but certainly
indicates that the longer forms of the conjunctions appear with markedly higher frequency in TEC.
5. Correlations, contractions, co-occurrences
To return to the notion of explicitation then, it could be claimed, on the basis of these measures of
inclusion and/or omission of optional syntactic elements above, that the language of TEC makes
explicit grammatical and lexical relations which are less likely to be made explicit in original English.
Furthermore, this tendency not to omit optional syntactic elements may be considered subliminal or
subconscious rather than a result of deliberate decision-making of which the translator is aware – most
translators do not have a conscious strategy for dealing with optional that, for example. It can be
argued that it is the nature of the process of translation and the cognitive processing which it requires
which produces the kind of patterning seen here. However, inclusion or omission of syntactic features
do not reveal the whole story. Olohan and Baker (2000) pointed out that the optional that data
discussed in that study revealed potentially different patterns in other features, such as use of modifiers,
pronominal forms, modal constructions etc. in TEC compared with the BNC. Thus, although a specific
syntactic or lexical structure can be investigated in terms of overall occurrence and of its usage within
the narrow context of a concordance line, the wider issue of co-occurrence and interdependency of
features must be considered. Research of this kind on the language of translation still has a long way to
go; however a small example can be used to illustrate the possible significance of interdependencies
and how they might be investigated further. We can take the data referred to earlier in relation to
promise and re-examine it in relation to a number of linguists’ suggestions that that is more likely to be
omitted in informal usage (for example Storms 1966; Elsness 1984; Dixon 1991). If we also accept that
the use of contracted forms constitutes evidence of informal style, then a search for contracted forms,
within the promise concordance line only, reveals the following (Table 11):
Form BNC TEC
promise total 57 48
promise with contracted forms in
concordance line
41
(72%)
21
(43.75%)
promise + that with contracted forms in
concordance line
7
(17%)
4
(19%)
promise + zero with contracted forms
in concordance line
34
(83%)
17
(81%)
promise with no contracted forms in
concordance line
16
(28%)
27
(56.25%)
promise + that with no contracted
forms in concordance line
12
(75%)
23
(85%)
promise + zero with no contracted
forms in concordance line
4
(25%)
4
(15%)
Table 11: Co-occurrence of promise +that/zero and contracted forms in BNC and TEC
From this we can see that, although that occurs with much higher frequency in TEC than in BNC,
promise co-occurs with contracted forms to a much higher degree in BNC than in TEC, and that, when
the that/zero usage is correlated with contracted forms and then compared across corpora, there is
actually little difference between the two corpora. Using contracted forms as a measure of informality,
this would indicate, firstly, that there is a correlation between inclusion of that and level of formality,
and, secondly, that the language of TEC may thus be judged more formal. A large-scale study of
contracted forms based on production and pruning of word lists for both corpora yielded the following
Table 12 and Figure 3):
430
Form BNC
forms
BNC
totals
TEC
forms
TEC
totals
apostrophe 5,851 5,269
*'s 4,818 4,623
*'ll 212 9,651 43 4,799
*'d 111 10,645 29 5,349
*'t 48 40,782 30 20,316
*'ve 53 7,768 17 4,068
*'re 12 7,344 8 4,250
it's 9,554 5,046
that's 4,650 2,640
there's 2,655 1,424
he's 2,628 1,951
she's 2,266 1,154
what's 1,601 1,021
let's 913 654
who's 396 334
where's 241 117
how's 146 36
here's 132 89
e's 102 0
I'm 8,773 4,256
d’ = do 3 418 3 84
t’ = the 99 126 0 0
y’ = you 22 53 7 7
Table 12: Contracted forms in BNC and TEC
Figure 3: Contracted forms in BNC and TEC as percentage of total occurrence across corpora
The most frequent form with apostrophe is *'s, which in the vast majority of cases is a possessive
marker rather than a contraction of is or was; many of the *'s occurrences are with names, and many
occur only once or a couple of times in the corpus. For this reason, individual occurrences of *'s have
not been counted, apart from the most common *'s contractions in BNC (it's, that's, there's, he's,
she's, what's let's, who's, where's, how's, here's, e's). Without looking at data for individual
occurrences for *'s forms, we can see from the figures above that the total number of *'s forms is
similar for both corpora. This is in stark contrast with all other categories, which represent true
contractions rather than grammatical markers. For all other contracted forms counted, a very clear and
consistent pattern emerges; they are much more frequently used in BNC than in TEC.
As mentioned above, one of the conclusions of the linguistics literature in relation to the optional
that is that omission is more likely in informal usage. This may also be the case for omission of the
relative pronoun that or which and the other optional features discussed above. The only exception is
perhaps the modal should following verbs such as suggest and order; if the modal is omitted, the
subjunctive is used, which arguably constitutes more formal style than the should construction.
Interestingly this is the only feature above for which TEC seems to favour omission rather than
inclusion. On all other optional forms, TEC is considerably more likely to use the optional item and
longer surface form.
According to the co-occurrence patterns which Biber (1988) and Biber et al. (1998) suggest as
underlying the five major dimensions of English, that-deletion and contractions are in the top three
431
features at the positive end of one scale (Dimension 1); this is indicative of their tendency to co-occur
in texts of shared function. These and the other features grouped with them are associated with
‘involved, non-informational focus, related to a primarily interactive or affective purpose and on-line
production circumstances’ (Biber et al. 1998: 149). Biber et al. continue to describe certain of these
positive features, including the two we have dealt with here – that-deletions and contractions – as
constituting a reduced surface form which results in a ‘more generalized, less explicit content’ (ibid.).
They talk of two separate communicative parameters, i.e. purpose of the writer (informational vs.
involved) and production circumstances (allowing careful editing vs. constraints of real-time
production). Dimension 1 is therefore labelled ‘involved versus informational production’ (ibid.).
Relating this to the findings above, it would appear than the BNC writing is more involved, more
generalised, less explicit, less edited than the writing in TEC; the original writer's purpose is more
involved, the translator's less so. The translator's surface form is not reduced to the same extent as the
original writer's, the translator is thus more explicit, less generalised in both form and content. The
translation is perhaps more carefully edited; are original writers more concerned with the creative
content and translators with explicitation of linguistic relations?
6. Conclusion
In terms of concrete findings of this kind in corpus-based translation studies, there is considerable
scope for further studies, particularly in the area of co-occurrence. Mauranen's (2000) study of research
on comparison of co-selectional restrictions in Finnish translation and original English is one of the few
which tackles collocational and colligational patterning in translated language using comparable
corpora, and much more research of this nature needs to be done. In addition, the other co-occurrence
features proposed by Biber for this and the other four dimensions of English could be investigated and
compared across the corpora. Ongoing work in Saarbrücken using a tagged version of TEC is likely to
yield interesting results in this respect, and research is continuing at UMIST to identify and investigate
relationships between linguistic features of translated language and the cognitive and social factors
which may give rise to them.
References
Baker M 1995 Corpora in translation studies: an overview and some suggestions for future research.
Target 7(2): 223-243.
Baker M 1996 Corpus-based translation studies: the challenges that lie ahead. In Somers H (ed)
Terminology, LSP and translation: studies in language engineering, in honour of Juan C. Sager,
Amsterdam and Philadelphia, John Benjamins, pp 175-186.
Baker M 1999 The role of corpora in investigating the linguistic behaviour of translators. International
Journal of Corpus Linguistics 4(2): 281-298.
Biber D 1988 Variation across speech and writing. Cambridge, CUP.
Biber D, Conrad S and Reppen R 1998 Corpus linguistics: investigating language structure and use.
Cambridge, CUP.
Blum-Kulka S 1986 Shifts of cohesion and coherence in translation. In House J and Blum-Kulka S
(eds) Interlingual and intercultural communication: discourse and cognition in translation and
second language acquisition studies. Tübingen, Gunter Narr. pp 17-35.
Burnett S 1999 A corpus-based study of translational English, Unpublished MSc dissertation, UMIST.
Dixon R M W 1991 A new approach to English grammar, on semantic principles. Oxford, Clarendon
Press.
Elsness J 1984 That or Zero? A Look at the Choice of Object Clause Connective in a Corpus of
American English. English Studies 65: 519-533.
Kenny D 1999 Norms and creativity: lexis in translated text. Unpublished PhD thesis, UMIST.
Kenny D 2000 Lexical hide-and-seek: looking for creativity in a parallel corpus. In Olohan M (ed)
Intercultural faultlines: research models in translation studies 1, Manchester, St. Jerome. pp 93-
104.
Laviosa-Braithwaite S 1996 The English Comparable Corpus (ECC): a resource and a methodology
for the empirical study of translation. Unpublished PhD thesis, UMIST.
Laviosa S 1998 The English Comparable Corpus: a resource and a methodology. In Bowker L, Cronin
M, Kenny D. & Pearson J. (eds) Unity in diversity: current trends in translation studies.
Manchester, St. Jerome Publishing. pp 101-112.
Olohan M, Baker M 2000 Reporting that in translated English: evidence for subconscious processes of
explicitation? Across Languages and Cultures 1(2): 141-158.
432
Mauranen A 2000 Strange strings in translated language: a study on corpora. In Olohan M (ed)
Intercultural faultlines: research models in translation studies 1, Manchester, St. Jerome. pp. 119-

Storms G 1966 That-clauses in Modern English. English Studies 47: 249-270.
Vanderauwera R 1985 Dutch Novels Translated into English: The Transformation of a ’Minority’
Literature. Amsterdam, Rodopi.

433
Patterns in scientific abstracts
Constantin Orasan
School of Humanities, Languages and Social Sciences
University of Wolverhampton
C.Orasan@wlv.ac.uk
http://www.wlv.ac.uk/~in6093
1. Introduction
In today's world, large amounts of information have to be dealt with, regardless of the field
involved. Most of this information comes in written format. Computers seem the right choice for
making life easier, by processing text automatically, but in many cases at least partial understanding (if
not full understanding) is necessary in order to automate a process. Luhn (1958) proposed a method for
producing abstracts which works regardless of the type of the document. Although further research
carried out in this area is based on his work, it has become apparent that general methods are not a
solution. Instead, more and more research has been done into restricted domains, where certain
particularities are used for “understanding”. A well known case is that of DeJong (1982) where the
structure of newspaper articles was used in order to get the gist of the articles and then generate
summaries.
However, all these methods, due to their inherent specificity, require prior knowledge about the
characteristics of the genre to which they are applied. Whether they refer to the distribution of words
throughout a document or to the overall structure of the document, such textual features have to be
identified in a corpus, by applying methodologies from corpus linguistics. For example, Biber (1998)
investigated four different genres: conversation, public speeches, news reports and academic prose, at
lexical, grammatical and discourse level. The findings showed that each genre had its own
characteristics which distinguish it from the other genres. One can use these findings for the automatic
identification of a genre, or for improving the results of an automatic method for one of these genres.
In this paper, the characteristics of a very narrow genre, that of scientific abstracts, are explored
on three different levels: lexical, syntactic and discourse. The hypothesis is that it is possible to find
patterns, which could be used at a later stage to find similarities between abstracts. It is hoped that these
patterns will be useful for improving the results of automatic summarisation methods when applied to
scientific texts.
The patterns identified in this paper are not only useful for automatic abstracting or computational
linguistics in general, but they can also be used in order to teach students how to write abstracts. As is
explained in the next section, both reading and writing an abstract are not a trivial task, and many
students experience difficulties. Those students who are learning English as a second language have
even greater problems with such tasks. The patterns which are identified in this paper could help them
to write abstracts.
2. What is an abstract and why is it useful?
The notion of an abstract is part of everyday language, but there is more than one definition
accepted for it. According to Cleveland (1983: 104) “an abstract summarises the essential contents of a
particular knowledge record and is a true surrogate of the document”. A similar definition is given by
Graetz (1985): “the abstract is a time saving device that can be used to find particular parts of the
article without reading it; … knowing the structure in advance will help the reader to get into the
article; … if comprehensive enough, it might replace the article” These two definitions emphasise the
most important function of an abstract (i.e. its role as a replacement for an entire document). However,
these definitions refer to ideal abstracts produced by professional summarisers. This paper argues that it
is very unlikely that an abstract produced by the author(s) of a paper (as in the case for most of the
abstracts in the corpus used for this paper) is to be used as a replacement for the whole document.
Therefore, a simpler definition of an abstract is considered more appropriate in this context: “a concise
representation of a document's contents to enable the reader to determine its relevance to a specific
information” (Johnson, 1995). So, the abstract is no longer a “mirror” of the document, but instead
draws attention to the most important information contained within the document. Moreover, the main
purpose of this definition (i.e. to highlight what is important in the document) can be applied to
automatic abstracting.
Regardless of the differences between the definitions mentioned above, they all highlight the use
of abstracts as filtering devices. In the present days, people are constantly being bombarded by large
434
amounts of information. Scientists and academics use abstracts to filter the existing literature when
conducting research into a certain topic or when trying to keep up-to-date with the latest advances in
their field of interest.1 On the basis of the abstract, they can decide if an entire document is worth
reading or not.
Swales (1990) considers the process of writing an abstract to be a “rite de passage” for gaining
entry into the scientific community via a demonstration of increasing mastery of the academic dialect.
This is true, writing an abstract is not a trivial task given that it does not allow redundancies and forces
the writers to use a lot of compound words. As Halliday (1993) points out, in scientific texts, which
include scientific abstracts, lexical density is very high, which makes it difficult to both read and write
such texts.
There are several ways of classifying abstracts. One way is to divide them, according to their
usage. The categories used in this case are: indicative, informative and critical (Lancaster, 1991). An
indicative abstract provides a brief description to help the reader understand the general nature and
scope of the original document without going into a detailed step by step account of what the source
text is about. An informative abstract is more substantial than an indicative abstract and is often used as
an alternative to the original document, following the main ideas presented in the document. Usually,
these two categories are combined, an abstract performing both an indicative and informative function.
A critical abstract gives not only a description of the contents, but also a critical evaluation of the
original document. Usually, in this case, it is not the author of the paper who writes the abstract.
After analysing the corpus of abstracts used in this paper, it became evident that the vast majority
of the abstracts are indicative. Some of them also have an informative function, but cannot be classified
as informative because they cannot replace the original document. No critical abstract was included in
the corpus.
There are also other ways of classifying an abstract: by the way they are used, and by their author,
but these classifications are not relevant to this paper. Most of the abstracts in the corpus present the
disadvantages of abstracts which are written by the authors of the papers, instead of trained
summarisers. As is shown is section 6, most of the abstracts do not always follow the Problem-
Solution-Evaluation-Conclusion structure (Swales, 1990). One reason for this could be because the
authors do not consider the abstract to be particularly important, in many cases it is written just before
the paper is submitted. Moreover, in some cases the content of the abstract does not necessarily reflect
the content of the paper, as Cleveland (1983: 110) has remarked “authors as abstractors have been
known to use their abstracts to promote the paper; this can create a misleading abstract and is unfair to
the user”.2
3. The corpus
In order to analyse the structure of scientific abstracts, a corpus of abstracts has been built. It
consists of 917 abstracts with 146,489 words. For research in corpus linguistics this may seem a very
small corpus, but building a corpus of abstracts which has a large number of words is a tremendous
task, given the small size of one abstract. Moreover, as Sinclair (2000) has pointed out, small corpora
are not necessarily bad; in some cases a small corpus is the right choice. The research presented in this
paper required a lot of human input, and therefore its size had to be kept down to make the analysis
possible. However, whenever possible, automatic processing was used.
Two sources were used for building the corpus. The first one was the Journal of Artificial
Intelligence Research (thereafter JAIR), from which 141 abstracts, with 24,509 words, were extracted.
As the name suggests, this journal publishes articles in the field of artificial intelligence. Due to the fact
that the size of this corpus was too small and the author wanted to compare abstracts from different
areas, the INSPEC database was used as a second source of abstracts. The INSPEC database contains
abstracts of papers from more than 4,200 journals and 1,000 conferences. Six topics have been selected
and the first few abstracts have been included in the corpus. Table 1 presents some details about each
topic.
1 Given the purpose of this paper, the use of news abstracts for reporters, politicians or businesspersons is not
considered. However, it can be argued that, in such a context, summaries of a single document are not so useful,
digests (multidocument summaries) being more appropriate
2 An example is the following sentence from an abstract of an article from conference proceedings: “By mastering
the fundamental issues discussed in this paper, you will increase the return of your organisation's investment in
data warehouses”
435
Topic No. of words No. of files In proceedings In journals
Artificial intelligence 82,141 512 230 282
Computer science 21,467 137 117 20
Biology 16,081 100 50 50
Linguistics 6413 50 26 24
Chemistry 12,096 68 43 25
Anthropology 7717 50 24 26
Total 146,489 917 490 427
Table 1: The characteristics of the corpus
Several remarks have to be made regarding the corpus. In this research, articles from artificial
intelligence are of particular interest; therefore, most of the abstracts are from this field and other
related areas (machine learning, information retrieval, etc.). However, other areas, like anthropology,
chemistry and biology have been included for making comparisons between different genres. When
computer searches were carried out in order to obtain abstracts which could be analysed in this paper,
those abstracts found were from both journal articles and conference proceedings. In one case, that of
the biology abstracts, the first fifty abstracts returned by the search came from conference proceedings.
As a result, a second search has been performed in this area, this time restricting the search to abstracts
of documents published in journals. These two categories of abstracts allow the author to see if there is
a difference between the abstracts of papers published in conference proceedings and those published
in journal articles. In some cases, an abstract can belong to more that one category. For example, it has
been noticed that some abstracts considered as belonging to the field of linguistics, are concerned with
some computational aspect of linguistics, and therefore could also be considered as belonging to the
field of artificial intelligence. For each topic, the number of abstracts published in conference
proceedings and journals is shown in Table 1.
No conditions were imposed on abstracts’ place of publication or the author's(s') mother tongue.
Therefore, not all of them are written in perfect English. However, it was considered that this would
better reflect the use of English in the research community.
4. Length
The length of abstracts was considered to be the first way of comparing them. Given that usually a
conference paper is shorter than one published in a journal, the author expected to find that the
abstracts of journal articles were longer than those of conference papers. Also, in the case of the
former, the editors impose a strict control on the quality of the article, and subsequently on their
abstracts. The statistics showed that abstracts of the journal articles are noticeably longer, both in terms
of sentences and words, than the ones belonging to the conference papers (Table 2). The shortest
abstracts are the ones belonging to humanist disciplines (linguistics and anthropology). However, due
to the small number of abstracts taken from these disciplines it is not possible to conclude that all the
abstracts from the humanities are short.
Journal Proceedings Total
Topic
Sent/Abs Words/Abs Sent/Abs Words/Abs Sent/Abs Words/Abs
Artificial Intelligence 8.20 165.67 6.05 152.44 7.24 159.74
Computer Science 9.58 232 5.94 163.30 6.40 159.32
Biology 7.9 196.18 5.65 130.02 6.78 163.43
Linguistics 5.78 149.52 5.92 108.65 5.85 127.83
Chemistry 8.58 215.08 6.34 163.30 7.14 181.85
Anthropology 6.23 157.88 6.08 154.43 6.16 156.26
Total 7.39 174.56 5.96 147.94 6.61 160.31
Table 2: The length of abstracts in sentences and words
The length of an abstract ranges from 1 sentence to 21 sentences in the case of abstracts taken
from journal articles, and 1 to 16 sentences in the case of abstracts in conference proceedings. In both
cases it is possible to find abstracts which have only one sentence, but usually this sentence just
enumerates the topic covered by the article. An interesting result was obtained when the length in
words was computed. The longest abstract has been published in conference proceedings. As a result of
this, it is possible to conclude that overall, the abstracts published in journals are longer than the ones
published in conference proceedings, both in terms of words and sentences. However, there is no rule
which states this.
5. Lexical level
There are several ways of analysing a corpus. The most basic form is by displaying and analysing
lists of characteristics. The analyses can involve very simple wordlists or be more sophisticated,
436
including the classic concordance format (Kennedy, 1998). In this paper the author starts by analysing
word frequency lists and lists of n-grams. These lists are also compared with the same lists generated
from a general purpose corpus, the BNC (Burnard, 1995). In the next section, grammatical features of
the texts are considered by analysing subject-predicate pair lists. Whenever it was necessary, specially
designed programs were used to display the context.
5.1 Word frequency lists
Given the small size of the corpus it was thought unwise to make significant generalisations, but
even by analysing the lists interesting features can be noticed. For this analysis and the ones which
follow, the corpus was tagged using the FDG tagger (Tapanainen, 1997). Using this tagged version of
the corpus, word frequency and lemma frequency lists were produced. For each case, two different lists
were generated, with and without considering the part-of-speech of each word. These lists were
compared with similar lists produced from BNC. It should be pointed out that the results of the
comparison have to be treated with caution. The first reason is the huge difference in the size of the two
corpora. Therefore, the decision was made not to draw a comparison between the two corpora in terms
of frequencies, but using their position in the list instead. Of course the frequencies could have been
normalised, but given the small size of the texts in the corpus, it was thought that the results would still
be unreliable.
The second problem with this comparison comes from the language. BNC is a corpus of British
English, whereas the corpus of abstracts was not filtered on the basis of the variety of English used.
Moreover, it can be argued that most of the English used in the scientific domain is written in
American English. For example, the word summarisation and its derived forms (e.g. summarising,
summarise, etc.) appears 137 times. In 114 of the cases the American English spelling (AE) was used,
whereas the British English spelling (BE) was used in only 23 cases.3 The same problem was found
with the word generalise (out of 83 occurrences, 78 used AE and only 5 BE) and characterise (56
occurrences, 46 used AE and 10 BE). However, it can be argued that there is no word with a high
frequency of occurrence, which has different spellings in AE and BE.
Word freq. list
from abs
7132 the
5908 of
4156 and
3082 a
2925 to
2485 in
1953 is
1575 for
1310 The
1204 that
1093 are
910 on
886 with
790 by
717 an
698 be
607 this
585 as
528 system
521 which
Word freq. list with
tags from abs
7132 the DET
5908 of PREP
4156 and CC
3069 a DET
2466 in PREP
1953 is V
1747 to TO
1569 for PREP
1310 The DET
1178 to PREP
1093 are V
894 on PREP
886 with PREP
789 by PREP
717 an DET
698 be V
597 that CS
575 that PRON
563 this DET
528 system N
Lemma freq. list
from abs.
8442 the
5913 of
4543 be
4162 and
3293 a
3010 to
2890 in
1644 for
1398 this
1277 that
1165 we
940 use
933 on
909 with
906 system
823 by
781 an
735 have
694 it
636 as
Lemma freq. list with
tags from abs.
8442 the DET
5913 of PREP
4179 be V
4162 and CC
3270 a DET
2870 in PREP
1823 to TO
1635 for PREP
1269 this DET
1187 to PREP
1165 we PRON
917 on PREP
909 with PREP
906 system N
822 by PREP
781 an DET
718 have V
694 it PRON
615 that PRON
597 that CS
Word freq. list
from BNC
5538939 the
3086807 of
2631593 to
2574912 and
2091285 a
1824289 in
1088658 that
983593 is
917103 was
897690 I
849027 for
847109 it
802227 ‘s
712502 on
661109 be
657574 with
631554 The
619043 as
596588 you
501209 at
Figure 1 Different frequency lists
Figure 1 shows the different frequency lists for the first 20 entries in the lists. It is evident that the
first 6 entries of the word frequency list, from the corpus of abstracts and from BNC, are almost
identical. However, further down in the list, differences appear. The word was, which in BNC occupies
the 9th position, in the corpus of abstracts appears in the 51st position. This can be explained by the
small number verbs which appear in the past tense. In the frequency list of the corpus the most frequent
noun is system, appearing in the 19th position, but if the lemmatised version of the list is taken into
consideration instead, it is in the 15th position (even higher if the different types of systems are taken
into consideration e.g. eco-system, geo-system). After this position, the nouns are quite frequent; paper
occupies the 21st position (24th in lemmatised list), data 23rd (25th), information 27th (26th). In BNC, the
first noun on the list is time, which occupies the 71st position, and there are not many nouns in the first
3 However, it should be pointed out that in BNC both spellings of the word summarise can be found, although the
British spelling is more frequent that the American one (1220 BE, 751 AE)
437
200 words (e.g. people 94th, years 120th, etc.). It is also evident that the types of nouns are completely
different.
Also, a quick check on the lists reveals that many of the most frequent words from BNC, belong
to more than one grammatical category. This is not true with the most frequent words in the corpus of
abstracts, especially the ones which are nouns. This may indicate that the abstracts focus more on
abstract states, objects and processes. A similar result was obtained by (Biber, 1998) studying
nominalization in scientific texts.
5.2 N-gram lists
N-grams are groups of consecutive N words in the corpus. Punctuation marks were not considered
as being part of an n-gram, therefore all of those containing punctuation marks were removed. When
sorted by their frequency, n-grams uncover frequent patterns in a corpus. N-grams (with N from 2 to 9)
have been generated. Initially the idea was to compare them with the ones produced from BNC, but it
became apparent that there is not much of a link between the two, except for the very frequent patterns
of the, in the, which are not very useful. However, this is not surprising, given that the n-grams are an
indicator of a document's contents. Figure 2 shows the first 29 entries of 2-grams, 3-grams and 4-grams
lists. The lists with a higher number of words had much lower frequencies and for space reasons they
are not displayed here.
2-grams
1276 of the
640 in the
360 this paper
320 on the
319 of a
311 and the
306 to the
273 have be
258 in this
256 for the
250 can be
242 base on
215 in a
204 it be
201 be a
198 be use
196 of this
192 with the
174 to be
166 that the
163 show that
144 be the
143 use to
142 the system
141 number of
139 by the
138 as a
123 artificial
intelligence
121 such as
3-grams
143 in this paper
115 be use to
72 the use of
61 base on the
58 be base on
53 a set of
50 show that the
48 we show that
47 the problem of
47 the development of
46 the number of
44 this paper present
43 one of the
43 be apply to
42 we present a
42 this paper we
41 a number of
39 this paper describe
39 can be use
37 a variety of
35 in term of
35 be able to
34 of the system
33 the performance of
33 base on a
32 we propose a
31 with respect to
31 the result of
28 some of the
4-grams
41 in this paper we
26 can be use to
20 this paper present a
20 in the context of
17 the world wide web
17 it be show that
17 be one of the
16 the size of the
16 a wide range of
15 one of the much
15 be base on a
14 this paper we present
14 on the other hand
14 in the form of
14 be base on the
13 this paper describe the
13 of this paper be
13 in the field of
12 the performance of the
12 on the basis of
12 in the size of
11 with respect to the
11 this paper describe a
11 this paper be to
11 the use of a
11 can be apply to
10 the development of a
10 of a set of
10 in the presence of
Figure 2: The lists of 2,3,4-grams from the lemmatised version of the corpus
It seems that the lists are not seriously influenced by the type of abstract. A comparison of the lists
of n-grams produced from abstracts published in journals and the ones from proceedings did not reveal
many differences. When the 3-grams are considered, in both cases the first element on the list is in this
paper, followed by be use to. However, the third element from the first list, we show that, appears in
the 67th position in the second list. In the list of 2-grams, show that, appears in the 14th position in the
first category and only 48th on the second category. Given that the size of the two subcorpora is almost
the same, such a result is unexpected. However, it could be explained by the fact that in many cases,
the conference papers present work in progress and, therefore the conclusion is not necessarily the
strongest point of a paper and therefore no reference is made to the conclusion in the abstract. As is
argued in section 6, there are cases when the abstracts do not have an evaluation section, but this
happens less frequently with the abstracts belonging to journal papers.
The n-gram lists also uncover terms for specific domains (e.g. information retrieval, neural
networks, world wide web, etc.). Although this is a possible use of them, this paper does not intend to
investigate this aspect.
438
5.3 The case of the noun paper
The analysis of word lists and n-gram lists represent a very easy and powerful way to find patterns
in texts. For example, if the word paper is taken, it is appears 499 times, in 473 abstracts, which means
that more than half of the abstracts use it. In one abstract, it is used four times, its authors introducing
each move using the following constructions: this paper investigates, this paper introduces, this paper
describes, this paper ends. There is another abstract in which the word paper is used 3 times. In 24
abstracts it is used twice, although in the rest of the abstracts it appears only once. In addition to this the
word study is used as a noun 170 times, research 154 times and work 111 times. Even though the
nouns study, research and work are not always synonymous with the word paper, these three words
together with paper strongly indicate that most abstracts make a reference to the paper from which they
are derived using constructions like: in this paper, in this study etc.
The n-gram lists strengthen this aforementioned conclusion. The word paper usually appears in
constructions like this paper (360 times) or the paper (115). Constructions such as this study (18), this
research (14), this work (25), this article (27) are also found. In many cases, the word paper is used as
the subject of verbs like: present (62 times), describe (50), be (45), introduce (15). Clearer patterns
appear when more words are considered: in this paper (143 times), this paper presents (44), this paper
we (42), this paper describes (39). By increasing the number of contextual words, the patterns become
less frequent: in this paper we appears 41 times, this paper presents a (20 times), this paper we present
(14 times), this paper describes the (13 times). Even when 5 words are considered, the first element on
the list is in this paper we present (14 times).
As a result of this analysis, it is reasonable to conclude that the patterns found with regards to the
word paper, are not accidental and cannot merely be explained by a high number of occurrences of the
word paper. Instead, they represent patterns specific for abstracts. If the most frequent noun (i.e.
system) is considered, such patterns do not appear.
6. Grammatical level
Analyses of the word frequency lists and n-gram lists proved to be a very useful way of
discovering patterns. However, they can only reveal patterns between words which are adjacent. As a
result of this, it is possible that many patterns were missed due to some modifiers or adverbs. In this
section, grammatical structure is used for uncovering patterns in the abstracts. All the abstracts were
tagged using the FDG tagger. This tagger, in addition to assigning part-of-speech tags to each word,
also provides partial dependency relations between words. These dependency relations were then used
for finding common noun-verb pairs. Consequently, two lists were generated. The first list represents
subject-predicate pairs and the second one contains pairs of nouns, which are not the subject of the
sentence (e.g. objects), and verbs. It should be pointed out that the process of generating these lists was
completely automatic; therefore they contain some errors. However, the number of these errors is
relatively low, and they do not influence the validity of the results. Figure 3 presents the first 20 entries
from the two lists.
120 it be
106 we present
88 we show
86 there be
84 that be
66 we propose
63 paper present
58 which be
56 we describe
50 paper describe
35 we discuss
31 result show
30 it show
28 we introduce
26 they be
25 approach be
24 we use
24 we develop
24 this be
65 be system
53 be problem
38 be information
37 present paper
37 be data
36 present system
36 be knowledge
33 present approach
29 be it
28 be model
27 be agent
25 be research
25 be first
24 play role
24 be number
24 be method
23 be science
23 be process
23 solve problem
Figure 3: Pairs of subject-predicate and verb-noun
As expected, the patterns found using n-grams also appear in these lists, but with an increased
frequency. This is normal given the fact that intervening adverbs do not affect the patterns. For
example, the pair we-present is found 106 times, an increase of 5 from the list of 2-grams. This is
because of groups like we also present. The increase is greater in the case of we-show, from 71 to 88.
439
In the list of subject-predicate pairs some interesting pairs can be noticed. The first pair in the list
is it-be. After manually checking all the appearances in the corpus, it was noticed that in only 8 cases it
was used as an anaphoric pronoun. This finding is not surprising given that it has been shown by
previous research that the pronoun it is frequently used in the scientific domain as non-anaphoric. In
addition to the 112 pairs of it-be, where it is used non-anaphorically, there are 86 appearances of
existential there as subject for the verb be. All these cases suggest that existential sentences are
frequently used in abstracts.
The subject we appears with a closed set of verbs (e.g. present, show, describe, discuss, etc.). This
set includes verbal processes, in Halliday's terms (Halliday 1994), and presentational processes; they
can be used to determine the different types of moves (as is shown in the next section). The subject is
usually the one which realises an action. Eight out of the first 20 pairs contain the subject we in them,
which suggests that the author is present in the abstract as the one who presents, shows, etc.
The large number of be predicates in the second list (14 times in the first 20 most frequent pairs)
reiterates the fact that existential sentences are quite frequent. The pair present-paper appears usually
because passive voice is used (e.g. …is presented in this paper), and therefore it is an instance of the
subject-predicate pair paper-present. The other pairs suggest that systems and approaches are presented
and problems are solved.
7. The structure of abstracts
The most distinctive feature of abstracts is their rhetorical structure. Gopnik (1972) has identified
three basic types of scientific paper: the ‘controlled experiment', the ‘hypothesis testing’ and the
‘technique description'. Each type has its own structure, but according to Hutchins (1977) they can be
reduced, either by degradation or by amelioration, to a problem-solution structure. However, this
structure is too general for the purposes of this paper. A more detailed organisation can be identified in
the scientific papers: background information about the domain, the problem, the solution to the
problem, evaluation of the solution and conclusion. Sometimes, they are referred to as moves. Graetz
(1985) and Swales (1990) claim that an abstract should have the following structure: problem-solutionresults-
conclusion. However, Salanger-Meyer (1990a) analysed a corpus of 77 abstracts from the
medical domain for this structure and found that only 52% of the abstracts followed the structure.
In each abstract, the moves have been manually identified. Given the large number of abstracts in
the corpus and the difficulty of annotation, only 67 abstracts have been selected and annotated.
Therefore the results presented in this section are just preliminary, in the future a semiautomatic
procedure will be used. Out of the 67 annotated abstracts, 35 were published in journals and 32 in
conference proceedings. The patterns found in each move are similar to the ones reported in Salanger-
Meyer (1990b) for abstracts from the medical domain.
During the annotation, five moves have been considered: Introduction, Problem, Solution,
Evaluation, Conclusion. Ideally, an abstract should contain all five moves, but an abstract with only
Problem, Solution, and Evaluation moves, has been considered to be perfectly acceptable and correct.
It should be pointed out that the annotation process is difficult and in many cases highly subjective. At
present, the corpus has been annotated by only one person, the author, but in the future at least some of
the abstracts have to be annotated by another person in order to compute the reliability of the
annotation using interannotator agreement measures. Out of 67 abstracts, it was found that only 39 of
them (58%) could be considered to follow the expected structure. In the rest of the cases, either an
important move was missing or the moves were in logical order. In quite a few cases, it was noticed
that the evaluation was given before the method was presented. The abstracts of journal papers proved
to be slightly better than the ones of conference papers in terms of organisation (21 abstracts from
journal papers and 18 abstracts from conference papers), but more data have to be investigated.
7.1 The Introduction section
The introduction section is meant to provide the reader with some background information, to
explain what has been done in the field, etc. Given the constraints on the size of an abstract, this move
is not compulsory. Moreover, abstracts are written for relatively informed readers and therefore they
should not provide too many background details. However, by analysing the abstracts in the corpus, the
author has noticed that there are cases when the introduction is quite long, in some cases almost half of
the abstract. Usually this section makes references to previous work using expressions like existing
approaches, prior work, previous work. The sentences in this section contain general truths (e.g.
Storing and accessing texts … has a number of advantages over traditional document retrieval methods
or Discourse analysis plays an important role in natural language understanding) usually expressed
through the present simple tense. In addition to the present simple tense, the present perfect is used
quite often for stating generic truths, but is usually used for emphasising the weaknesses of the
440
previous work (e.g. It is usually expected … but … or The standard approach … has been to … Such an
approach becomes problematic).
The introduction also gives hints about the problem which is going to be solved, highlighting in
an appropriate way the weaknesses of previous approaches. In some cases it was quite difficult to make
a clear distinction between the Introduction and the Problem.
7.2. The Problem section
The introduction section, prepares the reader for the problem section. In this section, the problem
with which the article deals, is expressed. There are cases when the problem is not clearly stated, but
the reader can usually infer it from the introduction and the solution sections. Even though the reader
can guess the problem, it is not desirable to have an abstract which does not state the problem
explicitly.
Given the fact that there are often some very frequent patterns, this section can be identified
relatively easily. Usually it is explicitly signalled through phrases like: we describe, we present, a
formalism is presented, we outline, etc. The preferred tense for stating the problem is the present
simple. In some cases comparison with previous work is used for stating the problem: because of
…existing methods can erase …we propose …. In these cases the Problem section also serves also as
introduction.
7.3. The Method section
In this section of the abstract, the author(s) should explain how the problem is resolved. This
section is very important for the reader because it enables him/her to understand the kind of approach
to solve the problem that was used. Some sentences from this section are marked overtly using phrases
like: an alternative solution, the approach described here uses …. Some of the patterns used for stating
the problem, also appear in the method section (e.g. This paper reports on a Japanese information
extraction system that merges information using a pattern matcher and discourse processor) In this
example, as well as in the other cases where a pattern which would be usually found in the Problem
section appears, the phrase has a double role. On one hand, it reiterates the problem, or state the
problem if it has not been stated already, and on the other hand, it explains the method.
The verb tense seems to be the present tense, although Biber (1990) found that the past test is
more frequently used in this move. Normally, this move should describe each step taken in the research
for solving the problem, and therefore the past simple should be preferred. However, in the abstracts of
the corpus used for this paper, the method is described in general terms, and not as a sequence of steps.
7.4 The Evaluation section
Another important section is the one which summarises the evaluation. An abstract without an
evaluation section is not considered to be correct. This is because, the evaluation proves the validity of
the method proposed. If a researcher is trying to find a solutions for a problem similar to the one
discussed in the abstract, he or she needs this section in order to judge the usefulness of the solution. If
the evaluation suggests that the solution is appropriate for the given problem, one can read the whole
article in order to obtain a full explanation of the method and a detailed evaluation.
It has been noticed that whenever the verb show is used, it appears in the evaluation move in
phrases like: we show, it is shown. In addition to the verb show, other phrases are also used for
explicitly marking this move (e.g. this work provides, reveal, yield, investigate, find etc.) Besides these
phrases, words which either refer to measures or ways of measuring are used (e.g. limited, compare,
quantify, are tuned, shortcomings, this allows us to measure etc.)
There are cases when the evaluation is not presented. Instead, the authors indicate that an
evaluation was performed (e.g. we briefly describe some experiments, we present three case studies,
etc.) On the basis of such an abstract, one cannot decide if the article is relevant or not, and therefore it
is not really useful.
The preferred verb tense is the present simple tense, but a large number of verbs in conditional
tense have been noticed. This could indicate that the authors do not want to overestimate their results
(e.g. can be adapted, can use, etc.) However, in addition to being part of the evaluation, these sentences
can be also used as a conclusion. Connectors, which hardly appear in the other moves, are quite
frequent in this one (e.g. however, in addition, etc.), being used for justifying the evaluation.
7.5 The Conclusion section
Many research papers have a conclusion section in which the results of the method are placed in a
broader context. However, it is not absolutely necessary to have this section separate in the abstracts,
but its presence makes an abstract more valuable in a broader context. Therefore, an abstract that does
not contain such a section is not considered to be incorrectly structured. In many cases the evaluation
section also plays the role of conclusion.
441
In the majority of cases, this move contains an explicit reference to the abstract (e.g. this work
provides, these observations suggest, etc.) In addition, phrases like this paper concludes, as a
conclusion, or adverbs (e.g. therefore, as a result, etc.) are used to mark the beginning of the
conclusion move.
8. Conclusion and future work
In this paper it has been shown that regardless of the level of analysis, lexical, syntactic or
discourse, it is possible to extract patterns from scientific abstracts. The simplest way of analysing, the
lexical level, uncovered groups of words which usually go together. As an example the word paper has
been analysed, noticing that it appears in frequent patterns. By using dependency relations between
subject and predicate, pairs were generated. Many of these pairs were also found in the lists of 2-grams,
but given that in this case the intervening adverbs do not matter anymore, the frequency of the pairs is
higher, so giving more reliable figures.
By analysing the discoursal structure of abstracts, it became evident that the scientific abstracts,
written by the authors of the papers, do not necessarily follow the structure which the literature
predicts. In each abstract, the moves were manually annotated and those groups of words which signal
the type of a move were identified. Some of the words seem to indicate reliably a certain type of move
(e.g. the verb show appears usually in the evaluation section).
The question which arises at this point is how these patterns can be used in computational
linguistics and especially for automatic summarisation. For the beginning it has been noticed that the
word paper appears usually only once in the abstract, in constructions like in this paper we. These
constructions are very similar to the findings reported in Paice (1981), where common patterns from
full-length papers, called indicating phrases, were identified in scientific papers and used for producing
a summary. Therefore, a sentence from a document which contains a pattern similar with one
previously identified, is more likely to be important, and consequently worth including in the abstract.
However the usefulness of each pattern has to be assessed in a corpus of scientific papers.
The patterns, which have been identified, are not only frequent in the abstracts, but in many cases,
they also indicate a certain move. This suggests that it could be possible to design an automatic
procedure for identifying each move in the text. However, it has been shown that more than one pattern
is used to introduce a move, therefore for each move it is possible to find more that one way of
introducing it. This suggests that it is possible to find general templates for each move. Such an
approach would not be new, Paice and Jones (1993) proposing a similar method for building abstracts.
However, in their case the templates are very specific for a certain domain.
Patterns in abstracts are useful not only for automatic abstracting. They are also useful for helping
researchers to produce their own abstracts. Narita (2000) proposed a system for Japanese, which can be
help writing abstracts by displaying sentences and collocations from a corpus of annotated abstracts.
Acknowledgements
The author of this paper would like to thank Ramesh Krishnamurthy, Dr. Andrew Caink and
Richard Evans for their comments provided at different stages of this paper, and IEE and Journal of
Artificial Intelligence Research for the permission to use their abstracts for this research
References
Biber D, Conrad S and Rippen R 1998 Corpus Linguistics: Investigating Language Structure and Use,
in Cambridge Approaches to Linguistics Series, Cambridge University Press
Burnard L 1995 Users Reference Guide: British National Corpus Version 1.0, Oxford University
Computing Services, UK.
Cleveland DB 1983 Introduction to Indexing and Abstracting, Libraries Unlimited Inc.
DeJong G 1982 An overview of the FRUMP system. In W. G. Lehnert and M. H. Ringle (eds),
Strategies for natural language processing. Hillsdale, NJ: Lawrence Erlbaum, pp. 149 – 176
Graetz N 1985 Teaching EFL students to extract structural information from abstracts. In Ulign J. M
and Pugh A. K. (eds) Reading for Professional Purposes: Methods and Materials in Teaching
Languages, Leuven: Acco, pp. 123 – 135
Halliday M.A.K and Martin JR 1993 Writing Science: Literacy and Discursive Power, The Falmer
Press
Hutchins WJ 1977 On the structure of scientific texts, UEA Papers in Linguistics, 5(3) pp. 18 – 39
Johnson F 1995 Automatic abstracting research, Library review 44(8)
Kennedy G 1998 An Introduction to Corpus Linguistics, Longman
Lancaster FW 1991 Indexing and abstracting in theory and practice, Library Association Publishing
Ltd.
442
Luhn H P 1958 The automatic creation of literature abstracts. IBM Journal of research and
development, 2(2): 159 – 165
Narita M 2000 Constructing a Tagged E-J Parallel Corpus for Assisting Japanese Software Engineers
in Writing English Abstracts, in Proceedings of the Second International Conference on Language
Resources and Evaluation (LREC'2000), Athens, Greece
Paice, CD 1981 The automatic generation of literature abstracts: an approach based on the
identification of self-indicating phrases. In Oddy, R. N., Rijsbergen, C. J. and Williams, P.W.
(eds.) Information Retrieval Research, London: Butterworths, pp. 172 – 191
Paice, CD and Jones PA 1993 The identification of important concepts in highly structured technical
papers. In Proceedings of ACM-SIGIR'93, pp. 123 – 135
Salanger-Meyer F 1990a Discoursal flaws in Medical English abstracts: A genre analysis per research-
and text-type, Text, 10(4), pp. 365 – 384
Salanger-Meyer F 1990b Discoursal movements in medical English abstracts and their linguistic
exponents: a genre analysis study, INTERFACE: Journal of Applied Linguistics 4(2) pp. 107 – 124
Sinclair JM 2001 Preface. In Ghadessy, M., Henry, A. and Roseberry, R. L. (eds) Small Corpus Studies
and ELT: Theory and Practice, John Benjamins
Swales JM 1990 Genre Analysis: English in academic and research settings, Cambridge University
Press
Tapanaine P and Jarvinen P 1997 A Non-Projective Dependency Parser. In Proceedings of the 5th
Conference of Applied Natural Language, pp. 64 – 71

443
An attempt to develop a lemmatiser for the Historical Corpus of Hungarian
Gabriella Kiss and Júlia Pajzs
Department of Lexicography and Lexicology
Research Institute for Linguistics, Budapest
{gkiss, pajzs}@nytud.hu
For the project of the Historical Dictionary of Hungarian a carefully selected representative corpus was
collected (24.5 million running words). The texts were chosen from three centuries. A morphological
analyser programme was successfully run on the modern texts, but the analysis of the earlier texts was
problematic. In our paper we will describe a method for the conversion and analysis of archaic texts,
without losing the original word forms. The Historical Dictionary project itself, and the analyser
developed for the modern text, will also be reported briefly.
1. The Historical Dictionary project and its corpus
The project first started in the late 19th century with the collection of old fashioned dictionary
slips. The idea was to compile an OED-like dictionary which covers the vocabulary of the Hungarian
language from 1772 up to the time of collection. (There exists an historical dictionary for the former
period : Czuczor G & Fogarasi J 1862). The collection of the slips continued until about 1960. From
time to time there was an attempt to compile some draft entries based on this collection, but somehow
or other these experiments always happened to fail. Possibly these repeated failures were partly due to
the lack of an adequate personality in charge of a project at this scale (Hutás, 1974). Without a
dedicated and convinced chief editor not even a one volume dictionary can ever be completed, not to
mention an OED-like several volume reliable historical dictionary.
During the years 1950-1960 a fairly valuable seven-volume monolingual dictionary was prepared
using the slips collected for the historical dictionary (Bárczi & Országh 1959-1962). It contains some
illustrative quotations, but not necessarily to each meaning. Bibliographic information is not supplied
with the quotations (only the author's name in abbreviated form is given in some cases). It was not
published as a historical dictionary, but as an explanatory one, and it still continues to be the largest
existing Hungarian explanatory dictionary (Országh 1960). In the late 60's a one-volume abridged
version was produced on its basis. It is still in print, and is now being revised to be published in 2002.
This revised version is also based on newly but traditionally collected data, not on a corpus.
In 1985 the Hungarian Academy of Sciences decided to start the historical dictionary project all
over again, based on a computerised corpus to be compiled first. The collection method seems rather
naive these days: it was a combination of old fashioned collection and corpus building. At that time
nobody in Hungary had any experience about efficient corpus collection methods, or about realistic
expectations from a corpus of a given size or type. At the outset we planned to collect a corpus of 10
million running words. To establish the source material for a sound historical dictionary, several
small excerpts were carefully chosen by the literary historians of each period. Since the short sample
texts usually contained only some book pages we found that optical character recognition was not
really efficient for this task. Therefore the selected text parts were keyboarded manually, which, of
course, took a very long time (keyboarding was finished at the very end of 2000), and, despite
repeated controlling, many keyboarding and other kinds of errors still exist.
Currently the corpus consists of 24.5 million running words, with a majority from the 20th
century (16 million words), 6.8 million words from the 19th century and 1.7 million from the late
18th century. The size of the samples is varied: since every poem and each part of a book is
considered a separate sample, the number of different samples is quite large: over 21,000. The average
sample length is around 1200, but there are also extremely short samples (e.g. a very short poem of
two words), and there are surprisingly large ones: a text of 34,812 words is the current maximum.
Although the literary experts tried to be as objective as possible and the selection was reviewed by
different experts, the extremely overrepresented works obviously happened to be their favourite ones,
and some very well known authors and/or works were simply left out for some reason. Most of the
texts are different kinds of prosaic texts (prosaic fiction: 31%, other kinds of prose: 51%, poetry:
8,5%, drama: 5,7%). In order to supply exact philological data for the planned historical dictionary,
bibliographic data and the estimated (or sometimes exact) year of production were recorded for each
444
sample text. This information along with the data of keyboarding, updating etc. are stored in the
header of each sample file in SGML format. When we started keyboarding SGML did not exist, so for
a long time special codes were used to mark paragraph ends, stanzas, quotations, etc. As soon as we
learnt about SGML (in 1987) we started to convert the earlier material into this format, and after a
while keyboarding contitued in it, there are, however, still too many formal errors in the texts
keyboarded previously. (This is a very serious point to consider for projects starting these days: one
should try to rely on XML and TEI or other standard recommendations as strictly as possible, and
should not think that it is always possible to correct the errors easily by a good conversion
programme.)
The first draft dictionary entries based on the corpus were prepared seven or eight years ago,
when the corpus was only about half as large as it is today. When we first realised that we can not
make reliable historical dictionary entries based solely on the corpus, we hoped to be able to fill in the
obvious gaps either by the traditional slips or by the enlargement of the corpus. Recently, we had to
cope with the fact that, although it is theoretically possible to compile a more or less historical-like
dictionary from the available sources, this would require a despairingly long time and/or several times
larger staff and facilities than we can hope for. (Just to give you a hint of our possibilities: right now
nine full time compilers are working on the project, but only three of them have some experience in
writing a dictionary, and none of them have a formal education in lexicography. And this is about the
largest team we can expect for the future, as well.) Faced with this situation, we are about to redesign
the whole project. At this turning point there are three alternative plans: one is to further improve the
historical corpus, and forget about dictionary writing, at least for the near future. The other is a less
maximalistic historical dictionary than the one planned originally, but it is sill a very traditional
historical approach, which wishes to build a dictionary that includes only the chronologically first
quotation to each known sense of the words (either from the slips or the corpus), chiefly due to space
considerations (this plan imagines an eight-volume dictionary). Our personal view, which is shared by
most in our team, is to produce a good, up-to-date one volume dictionary, at least as a first step, with
an electronic version including many additional facilities. This, in many respects, would rather
resemble the modern corpus-based monolingual English dictionaries (COBUILD, CIDE, LDOCE, or
even the new OALD) more than the OED. We would like to make clear-cut entries with easily
understandable definitions, synonyms, antonyms, hyponyms, etc. The main specificity of this
dictionary compared with the above mentioned modern English ones would be the illustrative
quotations not only from the current part of the corpus, but from the earlier periods, as well. In the
electronic version the exact philological reference of the citations would be supplied, as it is done in
traditional historical dictionaries. The electronic version could also contain many more examples from
the corpus, plus several additional properties — like the generation of the inflected forms of the words
(which is a much more difficult task and therefore even more necessary for Hungarian than for
English). We believe that with a brand new and up-to-date concept a really good modern dictionary
could be compiled, combining the advantages of modern corpus based dictionaries and traditional
historical ones. We are convinced that it could be produced in a realistic period of time even with a
relatively small team of compilers, and this product would find its market.
2. The morphological analysis of the corpus
2.1. The analyser programme
The HUMOR analyser programme was developed by the MorphoLogic Ltd in co-operation with
our department in the late 80's (Pajzs 1991, Prószéky 1996, Prószéky & Kis 1999). The first version
was based on the entries of the above mentioned seven-volume dictionary, which were supplied with
morphological codes. Encoding was further improved during the development of the programme. Now
it contains a complex classification including information about the types of suffixes that can follow
the given entry and the matching suffix variant. Hungarian morphology is extremely complex: it is
mainly agglutinative, several suffixes may follow each other, at least in theory. Based on the large
analysed corpus we have evidence that the highly complex combinations practically never occur (Pajzs
& Papp 1998). The combination of two suffixes is quite frequent, but due to vowel harmony, if the
first suffix is a back variant, the next suffix must also be back, therefore the actual number of real
suffix combinations is not that large. The information on the possible forms of the suffixes was stored
in the databases used by the analyser. The databases of the entries and the suffixes are disjunct, and
the programme only checks whether the elements actually found in the text can be matched according
to the information given in the databases. It uses the unification method for choosing the correct
445
solution(s). When the analyser finds a possible match, it can also identify the root of the word, even
when the actual form of the root is different from the entry (e.g. the original root ló ’horse’ becomes
lov in front of the suffixes, and its analysed version contains both the original lemma in front of the
„=” sign and the actual root after it: ló[FN]=lov+aink[PSt1i]+nak[DAT] ‘for our horses').
This analyser programme was applied on the corpus in several steps. First it was only tested on
the contemporary part of the corpus, later on the 19th century part, but this test already raised
numerous problems. Hungarian orthography was standardised only in the late 1930's, therefore the
earlier texts contain many alternative orthographic possibilities. Since the analyser is also used as the
engine of a Hungarian spell-checker, the old (currently unacceptable) alternatives could not be
included in its databases. Vowels in some words which now should be spelt with long accents used to
be spelt either with short or with long accents (even within the same text). Several compounds which
are now written in one word used to be written in separate words or with a hyphen. So when we
applied the analyser on the texts from the 19th century, only 90 per cent of the words were recognised
by the programme, while in the 20th century part 95 per cent were recognised. Naturally, the
recognition of the analyser is not always correct, sometimes the words are analysed as non existing
compounds or as strange derivates. When the programme finds several possible analyses it outputs
each variant. For the disambiguation of these alternatives a local rule-based programme was tested
and run on the whole 19th-20th century corpus (Pajzs 1997, Pajzs 1998). Although the result of this
attempt was far from the expected correctness rate, because the texts were very varied, its outcome was
a usable lemma-oriented corpus, where one could directly search the entries without enumerating
every possible form of the word or having too much surplus data. (So for example if you try to search
the word ad ’give’ in the non-analysed corpus, you will get every word which happens to start with
the same character string, while if you search it in the analysed version, you can explicitly define that
you only wish to search the verb ad.)
When using the analysed corpus (of 17 million running words at that time) we realised some of
its drawbacks. In the meantime the keyboarding of the late 18th century texts also started which
meant a new range of problems.
The orthography of that time hardly resembles that of modern texts. Some typical problems:
· In the earlier prints there were several characters which are not used any more. For example,
instead of the short and long vowels ö,o or ü,u, standard nowadays, there used to be several
different accents in between, which represent either short or long vowels. As the historical
linguists considered these specialities important, we were bound to keep this information
somehow when keyboarding the texts. As these characters were used sometimes instead of the
short, sometimes instead of the long variant, they could not be converted directly to their
current form.
· There were several suffixes, old root forms and words that are not used anymore.
· Some of the still existing words used to be spelt in completely different ways: words which are
now written in two words sometimes (not always!) used to be written in one word, others the
other way round.
· Since there was no standardised orthography at all, the very same consonant phonemes were
sometimes spelt differently even in various occurrences of the same word within the same text,
as e.g. lly or lyly or jj or lj for [j]. The s letter was often represented by something like a  or
 , and the letter z also had an archaic version. To make things even more complicated, the
phoneme now spelt as zs was spelt as a single s or its old forms or with the combination of old
s with old z or with any other combinations of these.
All this information was kept during keyboarding. The special characters were represented by a
combination of letters and digits. This way we could keep all the required information in an easily
convertible format (so the representation of the characters did not depend on any operation system or
the facilities of any given word processor). Not only the archaic characters, but the current accented
characters are also represented in this way, chiefly for the sake of portability. (The letters of the
English alphabet and the digits are always the same in every code table.) When, however, we intend to
retrieve the old words together with the new corpus, we either leave the problem of searching of the
different possible forms for the lexicographers who should be able to retrieve the concordances as fast
as possible, or we must find a way to standardise the old words, while keeping the original old form.
For this aim we designed a special format:
Keyboarded form
446
Honnat-is nem kis bos43zs43zusa1g-te1tellel bu20no20s43u20lnek azok, kik tellyes43se1ggel
Current ASCII form
Honnan is nem kis bosszúságtétellel bunösülnek azok, kik teljességgel
‘where from those persons will be punished by a not small annoyance, who are completely...’
Lemmatized form
<w><o>Honnat</o> <t>honnan</t> <a>honnan[HA]</a></w>
-
<w> <t>is</t> <a>is[KOT]</a> </w>
<w> <t>nem</t> <a>nem[MOD]</a> </w>
<w> <t>kis</t> <a>kis[MN]</a></w>
<w><o>bos43zs43zusa1g</o> <t>bosszu1sa1g</t> <a>bosszu1sa1g[FN]</a></w>
-
<w> <t>te1tellel</t> <a>te1tel[FN][INS]</a></w>
<w><o>bu20no20s43u20lnek<o> <t>bu3no2su2lnek</t>
<a>bu3no2su2l[IGE][t3]</a></w>
<w> <t>azok</t> <a>az[NM][PL]</a></w>
,
<w> <t>kik</t> <a>ki[NM][PL]</a></w>
<w><o>tellyes43se1ggel</o> <t>teljesse1ggel</t> <a>teljesse1g[FN][INS]</a></w>
In the field tagged by <o> the original, old version is kept if it is different form the current
spelling. In the field tagged by <t> either the converted version of the original old form, or the token
as it was found in the text if it does not differ from the current norms, is kept. If there are more than
one possible and analysable conversions each one is kept separated by a ’|'. If the analyser is able to
recognise the token, the analysed version is kept in the field marked by <a>. If it could not find the
correct analysis a special tag NE ’not analysed’ is given. If there are more than one possible analyses
each one is given. The analysed (or rather tagged) version consists of the recognised lemma, the part
of speech code and the suffix codes. The superfield word is marked by the <w> tag.
The main advantage of this format is that we can keep both the original archaic form as it occurs
in the text, the normalised tokens and the lemmatised version. For the time being we do not intend to
disambiguate the analysed version. The retrieval interface takes care of finding the lemma in the
analysed field first. If the searched word is not present in this field, then the token field or even the
original field can be searched. After the search the result is displayed either from the original field
(when there is one) or from the token field if there was no original version. Of course this storing
format is very redundant, it requires plenty of disk space, but nowadays hard disks are becoming less
and less expensive. We have been using the Open Text SGML text retrieval software, but we are
considering to switch to a more modern and efficient software tool. From this tagged version we can
easily convert the text to any other format preferred by different tools.
2.2. The conversion of the archaic forms
The regularly occurring orthographic and grammatical alterations were aimed to be converted at
this phase.
2.2.1. Old characters and variant orthography
The variant spellings of the letter s , were keyboarded as s41 and s43, respectively, and then
those can be converted to s.
There are several strange accented forms for the letter u and o, which were keyboarded as u20,
o20, u23, o23, u24, o24 etc. and can be converted to ü ö, u and o, respectively, according to current
orthography. If there are many of these old characters in one word, each of the possible combinations
must be generated. The HUMOR programme tries to analyse each version and outputs every seemingly
correct analysis to the analysed field, and from the converted tokens only those having a
corresponding analysis will be kept.
Other archaic character combinations:
The phoneme currently represented by the letters cs used to be written by ts
The phoneme currently represented by the letter c used to be written by tz or cz
The long versions of the digraphs are now spelt by repeating only the first character of the
digraph, earlier it was variable. (E.g. a suffixed form of asszony ’woman’ is spelt asszonnyal ’with a
woman', while in the archaic texts it could either be aszszonynyal, asszonynyal or asszonnyal, and, of
course, any of the s-es or z-s could have been old ones.)
447
2.2.2. Phonological rules regularly appearing in the texts
Consonants often lengthened in intervocalic position. Those which are represented by digraphs
had to be handled separately.
old: segitto (keyboarded as: s43egitto24), new: segíto
old: gyilkossa (keyboarded as: gyilkos34s43a), new: gyilkosa
old: tallyigába new: talyigába
old: fénnyiben, new: fényében
Spelling according to pronunciation
old: akarattyán, new: akaratján
old: tilcsa or tiltsa, new: tiltja
old: tanúji, new: tanúi
old: eladgyák, new: eladják
Vocals lengthened before l, n
old: mozdúlásra new: mozdulásra
old: múnkái, new: munkái
old: óldva, new: oldva
The phoneme now spelt zs sometimes used to be spelt s
old: strásának, new: strázsának
old: désa, new: dézsa
2.2.3. Morphological variants
The third person plural possessive suffix was often spelt -jok, which is now -juk.
old: búzájok new: búzájuk
old: hazájok new: hazájuk
The third person singular possessive suffix also had a variant form -ok, which is now -uk.
The verbal causative derivational suffix -ít sometimes used to be written as -it.
The use of long and short variants of the same vowels was much less regular than it is today.
Some words had variant root forms, which are not used anymore.
2.3. The process of corpus analysis
The programme which made the above described conversions and tried to analyse the converted
tokens was only one module of the process.
2.3.1. The first module is a PERL programme which picked the running words from the corpus. Its
output was ordered by the ‘sort’ and ‘uniq’ unix commands. The result is a file containing 1,467,230
different tokens.
2.3.2. The HUMOR analyser was run on this list. (The number of analysed words after this phase was
896,153). After the analysis the unanalysed words were separated from the output. The number of
unanalysed tokens was roughly half a million at this phase.
2.3.3. On the unanalysed list a converter programme was run, which contains a series of PERL
regular expressions based on the above scetched grammatical rules. It converts the possible variant
forms of the non- recognised words, and then tries to analyse them with the help of the HUMOR
programme. If it finds at least one analysable version, the corresponding token and the analysed
version is outputted in the format described in 2.1. If there is no analysis even after the conversion,
one possible converted form is still kept in the token field, and in the analysed field the code „NE”
marks the missing analysis.
2.3.4. The output of 2.3.2 is merged with the output of 2.3.3. The result is put into an ACCESS
database, which contains the original running word as it was found in the corpus in the first field and
the analysed version in the format described in 2.1 in the second field. The database is indexed for the
first field.
2.3.5. A programme runs on the whole corpus that reads and copies the header of the text files to the
analysed text files, then reads the running words one by one, searches them in the first field of the
database created in 2.3.4, and outputs the result from its second field. The rest of the information
448
found in the text (SGML tags, punctuation etc.) is copied to the analysed version without any change.
In some sample texts there are notes from the original texts. They are kept separately at the end of
each sample, and are copied into the end of the analysed file.
2.4. Evaluation of the lemmatisation method
With the software package described in 2.3 the whole corpus has been analysed. Here are some
examples for the problems of the conversion and identification from 18th-century samples.
2.4.1. Some examples of successful analysis after the conversion
<w><o>kapkodgyon</o><t>kapkodjon</t><a>kapkod[IGE][Pe3]</a></w>
The current form is kapkodjon ‘imperative of ‘snatch''. The written from, based on the
pronounced assimilated word, contained the letters dgy instead of the current dj. Since it is a regular
deviation, it was correctly converted and analysed.
<w><o>eggy</o><t>egy</t><a>egy[DET]|egy[SZN][NOM]</a></w>
The current form is egy ’one'. Again it is a regular deviation, so it was handled correctly.
<w><o>melly</o><t>mely</t><a>mely[NM][NOM]</a></w>
The current form is mely ’which'. Regular, correct.
<w><o>alats43ony</o><t>alacsony</t><a>alacsony[MN][NOM]</a></w>
<w><o>s43zaba1su1</o><t>szaba1su1</t><a>szaba1su1[MN][NOM]</a></w>
<w><o>ts43ats43ogni</o><t>csacsogni</t><a>csacsog[IGE][INF]</a></w>
<w><o>kits43al</o><t>kicsal</t><a>kicsal[IGE][e3]</a><t>kicsal</t><a>kicsal[IGE][e3]</a></w>
<w><o>kapts43olatok</o><t>kapcsolatok</t><a>kapcsolat[FN][PL]</a><t>kapcsolatok</t><a>kapcsolat[FN][PL]</
a></w>
<w><o>vis43zontag</o><t>viszontag</t><a>viszontag[HA]</a></w>
In these words only the characters had to be converted (s43 to s, ts to cs). After the conversion
the analysis was correct.
<w><o>tis43zta1talansa1gokat</o><t>tiszta1talansa1gokat</t>
<a>tiszta1talansa1g[FN][PL][ACC]|tisztatalansa1g[FN][PL][ACC]</a></
w>
After the character conversion the analysis is only partially correct: the lemma tisztátalanság
’impurity’ is correctly identified, but the suffix -ok used here is an old form of the possessive suffix
-uk, and not the current plural suffix -ok. The second analysis is a misinterpretation: the supposed
lemma is tisztatalanság, which is a supposed derivation of the word tiszta+talan+ság
’clean'+privative+adjective-to-noun nominal suffix, but an actually non-existing word. The accusative
is correctly identified at the end of the word. Although this analysis is not quite correct, the main
point is to give the good lemma, and it is also given there.
Usually the conversion was the most successful when only one or two old characters had to be
converted and the root or the suffixes were same as the current form.
2.4.2. Some examples for unsuccessful analysis after the conversion .
<w><o>o2s43zvekapts43olva</o><t>o2szvekaptsolva</t><a>NE</a></w>
The current form of this word is: összekapcsolva ’connected'. In order to be able to recognise
this, the variant lemma összve ’together’ must be added to the database of the entries. Then the
conversion of s43 to s, and that of ts to cs would be sufficient for the recognition of this word.
<w><o>eggyu2gyu2</o><t>egyju3gyu3</t><a>NE</a></w>
The current from of this word is együgyu ‘foolish'. The attempt to convert the ggy to gyj was not
successful, and also only one of the short ü-s should have been replaced by the long u.
<w><o>s43zo2me1rmetes</o><t>szo2me1rmetes</t><a>NE</a></w>
This is an archaic version of an obsolete word szemérmetes ’coy, prudent', nowadays only used
ironically as a common saying originating from a well known mid-19th century ironic epic poem.
<w><o>gyu24lekezo24tt</o><t>gyu2lekezo3t</t><a>gyu2lekezo3[FN][ACC]|gyu2lekezo3[MN][ACC]</a></w>
The current word would either be gyülekezet ’assembly (noun)', or gyülekezett ’past tense of
‘gather’ (verb)’ or gyülekezot ’present participle of ‘gather’ (verb) with an accusative suffix'. In the
current example the second case would have been the correct choice, but it was not among the
converted forms, because the formerly frequently occurring -ött variant form of the -ett current suffix
is not included in the suffix database. In the ancient texts many ö-s occurred where today e-s are
written, because this used the be a frequent pronunciation and orthographic variant. Nowadays it is
rather just a regional alternative, and appears in written form only occasionally.
<w><o>o24rizko2dgy</o><t>o3rizko2dj</t><a>NE</a></w>
449
The current form is orizkedj ’the imperative of ‘refrain from''. The conversion was partially
correct (dgy to dj), again the alternative -öd form of the -ed suffix should have been included in the
suffix database.
<w><t>mennyen</t><a>menny[FN][SUP]</a></w>
The current form is menjen ’the imperative of ‘go''. The given analysis is mistaken for the
superessive of the noun menny ’heaven', and seems to be analysed in the first analyser phase (2.3.2),
so it does not appear among the words to be converted.
<w><o>Tragyo24dia1nak</o><t>tragyo3dia1nak</t><a>NE</a></w>
The current form is tragédiának ’dative of ‘tragedy''. The letter g is not often replaced by gy in
the old text, and é is also rarely replaced by ö/o. So there was no conversion rule for this word.
<w><o>Tana1ts43be1li</o><t>tana1csbe1li</t><a>tana1csbe1l[FN][IKEP][NOM]</a></w>
The current form is tanácsbeli ‘belonging to the council'. Although the conversion was partially
correct (s43 to s, ts to cs), the suffix -béli was not converted to -beli, and the converted token was
mistakenly recognised by the analyser as tanács+bél+i ’council'+'intestine'+noun-to-adjective
derivative suffix.
2.4.3. An analysed example sentence
Old keyboarded form:
Hallom s43ivednek foha1s43zkoda1s43ait, e1rtem azon bu1tsu1t, melly ne1ked a Kira1lyne1to3l adatott, de
erro2l ma1skor les43z s43zo1llanunk, s43okkal nyomos43s43abb gondok foglalnak el lelku2nket.
Current ortographic ASCII form:
Hallom szívednek fohászkodásait, értem azon búcsút, mely néked a Királynétól adatott, de errol máskor lesz
szólanunk, sokkal nyomósabb gondok foglalnak el lelkünket.
A sentence with a similar meaning in current Hungarian:
Értem, mennyire fáj a szíved amiatt, hogy búcsut kell venned a királynétól, de errol majd máskor beszéljünk,
most sokkal fontosabb gondjaink vannak.
‘ I sympatise with your feelings on saying farewell to the queen, but we have to talk about it later, because we
have much more urgent problems to solve at the moment.’
+ <w> <t>hallom</t>
<a>hallik[IGE][Te1]|hall[IGE][Te1]|hall[FN][PSe1]|hallom[FN]</a></w>
- <w><o>s43zivednek</o> <t>szivednek</t> <a>NE</a></w>
+ <w><o>foha1s43zkoda1s43ait</o><t>foha1szkoda1sait</t><a>foha1szkoda1s[FN][PSe3i][ACC]</a></w>
,
+ <w> <t>e1rtem</t>
<a>e1n[NM][CAU][]|e1rt[IGE][Te1]|e1r[IGE][Me1]|e1r[IGE][TMe1]
|e1rt[MN][PSe1]|e1rik[IGE][Me1]|e1rik[IGE][TMe1]</a></w>
+ <w> <t>azon</t> <a>azon[NM]|az[NM][SUP]</a></w>
+ <w><o>bu1tsu1t</o> <t>bu1csu1t</t> <a>bu1csu1[FN][ACC]</a></w>
,
+ <w><o>melly</o> <t>mely</t> <a>mely[NM]</a></w>
+ <w> <t>ne1ked</t> <a>te[NM][DAT]</a></w>
+ <w> <t>a</t> <a>a[DET]</a></w>
'
- <w><o>Kira1lyne1to2l</o> <t>kira1lne1to3l</t> <a>NE</a></w>
+ <w>
<t>adatott</t><a>adatott[MN]|adat[IGE][Me3]|adatik[IGE][Me3]|ad[IGE][MUV][Me3]</a></w>
,
+ <w> <t>de</t> <a>de[HA]|de[KOT]</a></w>
+ <w><o>erro2l</o> <t>erro3l</t> <a>ez[NM][DEL]</a></w>
+ <w> <t>ma1skor</t> <a>ma1skor[HA]|ma1skor[FN]|ma1s[FN][TEM]|
ma1s[NM][TEM]|ma1s[MN][TEM]</a></w>
+ <w><o>les43z</o> <t>lesz</t> <a>lesz[IGE][e3]</a></w>
- <w><o>s43zo1llanunk</o> <t>szollanunk</t> <a>NE</a></w>
,
+ <w><o>s43okkal</o> <t>sokkal</t> <a>sokk[FN][INS]|sok[SZN][INS]</a></w>
+ - <w><o>nyomos43s43abb</o> <t>nyomosabb</t> <a>nyomos[MN][FOK]</a></w>
+ <w> <t>gondok</t> <a>gond[FN][PL]</a></w>
+ <w> <t>foglalnak</t> <a>foglal[IGE][t3]</a></w>
-
+ <w> <t>el</t> <a>el[IK]</a></w>
+ <w> <t>lelku2nket</t> <a>le1lek[FN][PSt1][ACC]</a></w>
450
The sentence contained 22 running words. Those of them which are marked by a ‘+’ sign at the
beginning of the column are analysed correctly, or rather, at least one of the given analyses is correct.
Those which are marked with ‘-‘ are not analysed correctly. There is one with a ‘+-‘ sign, the analysis
is nearly correct, but not completely (the current lemma would be nyomós strong, powerful’ not
nyomos ‘trail, trace+noun-to-adjective derivative suffix', which is just a supposed but non existing
form).
Those words which were recognised as archaic versions had a first field marked with the <o>
</o> tags. There were 10 words like this in the above sentence. Out of these 6 were correctly analysed
after the conversion. Altogether 18 words out of the 22 were analysed correctly in this example.
2.5. Statistical evaluation of the results
Cent Running
words
A
Analysed
words
after 2.3.2
B
Analysed
words after
2.3.5
C
Archaic
forms
D
Analysed
archaic
forms
E
Not
analysed
words
F
Percentage
of not
analysed
words
18th 1,689,735 1,433,100 1,727,697 379,898 294,597 231,319 13.6%
19th 6,839,688 6,666,168 6,873,935 339,326 207,767 684,328 10%
20th 1,615,5007 16,117,793 16,159,742 169,404 41,949 798,562 4.9%
Note that the number of analysed words is sometimes larger than that of the running words,
because in many cases the words have more than one supposedly correct analysis. The ratio of
unanalysable words in the texts from the 18th century was 22.48 per cent (1-[(A-D)/A]) before using
the conversion programme. After the use of the above described conversion rules most of the formerly
unrecognised words yielded an analysis (E/D=0.775). Although there still remained a much larger
number of unanalysed words than either in the 19th or 20th century texts, it is clear, that this algorithm
had the most successful effect on the targeted part of the texts: while the ratio of recognised words was
raised by 22.49 per cent (E/[A-D]) in the 18th century part of the corpus, in the 19th century part the
improvement was 3.1 per cent, while in the 20th century part 0.2 per cent. It might be surprising that
the above described rules could have been used in the 20th century texts at all, but remember that
Hungarian orthography became standardised only in the 1930s. Naturally, in many cases the
attempted conversions are misleading, so the resulting analysis sometimes has nothing to do with the
correct recognition of the lemma.
We have also examined the number of rules employed during the conversion, we have given an
identification number to each group of rules and this number was put into the resulting database to an
additional field. Using this field we could investigate the effectiveness of the rules. In most archaic
words only one conversion rule was employed (42.57 per cent), in 12 per cent two rules were
employed, three rules were used only in 2.39 per cent. The largest number of different rules employed
for the same word was 7, but it was done for 6 different running words only (0.001 per cent). We are
planning to study this database further.
3. Conclusion and further research
The attempt to be able to search lemmata in historical texts has proved to be promising. Our
purpose at this stage was to handle the difficulties raised by a diachronic corpus in a relatively
straightforward way. Instead of preparing a completely different analyser programme for the archaic
texts we have been trying to find regular alternations and convert the archaic words to forms as
similar to the recent ones as possible. Although plenty of problematic cases remain unsolved, we still
believe that we are on the right track. From the preliminary results described above it is possible to
draw new conclusions: whenever we look at the current output, we can find some new regularities
which can be added to the programme, thus improving the ratio of correct analysis. The often
occurring irregularities can also be added to one of the databases used by HUMOR, called the inclusion
database. This is a very simple database where you can add a running word in the first field and put
its correct analysis in the next column (so for example the frequent archaic word forms like mennyen,
451
vala can be added here). HUMOR also uses a similar list for excluding some incorrect analyses, so
whenever we find an error among the analyses given by the programme, we can just eliminate them
by adding them to the exclusion database. The further development of the conversion rules combined
with the careful use of HUMOR's inclusion/exclusion databases can make the lemmatisation process of
the archaic texts much more accurate.
References
Czuczor G, Fogarasi J 1862 A magyar nyelv szótára I.-VI. ’The dictionary of Hungarian’ Pest, Emich
Gusztáv Magyar Akadémiai Nyomdász
Benko L (ed) 1991–1992 A magyar nyelv történeti nyelvtana I–II., ’The historical grammar of
Hungarian’ Budapest, Akadémiai Kiadó.
Bárczi G, Országh et all. (eds.) 1959-1962 A magyar nyelv értelmezo szótára I.-VII. ’The explanatory
dictionary of Hungarian’ Budapest, Akadémiai Kiadó
Elekfi L 1994 Magyar ragozási szótár – ’Dictionary of Hungarian Inflections', Budapest, Research
Institute for Linguistics
Kiefer F (ed) 2000 Strukturális magyar nyelvtan 3. Morfológia, ’A Structural Grammar of Hungarian
3. Morpology'. Akadémiai Kiadó, Budapest
Olsson, Magnus 1992 Hungarian Phonology and Morphology, Lund, Lund University Press,
Országh L 1960 Problems and Principles of the New Dictionary of the Hungarian Language. Acta
Linguistica X/3-4. Budapest, Research Institute for Linguistics of the Hungarian Academy of
Sciences, pp 211-273.
Pajzs J 1991 The Use of a Lemmatized Corpus for Compiling the Dictionary of Hungarian Using
Corpora Proceedings of the 7th Annual Conference of the OUP & Centre for the New OED and
Text Research. Waterloo, University of Waterloo, pp 129-136.
Pajzs J 1997 Synthesis of results about analysis of corpora in Hungarian. Linguistica Investigationes
XXI-2 John Benjamins, Amsterdam pp 349-365
Pajzs J, Papp F 1998 Statistical Examination of the Hungarian Noun Paradigm Proceedings of
ALLC/ACH Debrecen, Lajos Kossuth University pp 89-93.
Papp I Leíró magyar hangtan, ’Hungarian descriptive phonology’ Budapest, Tankönyvkiadó, 1966.
Prószéky G, Tihanyi L 1992 A Fast Morphological Analyser for Lemmatizing Corpora of
Agglutinative Languages. Proceedings of COMPLEX '92. Budapest, Research Institute for
Linguistics, pp 275-278.
Prószéky G (1996). HUMOR - A Morphological System for Corpus Analysis. Proceedings of the first
TELRI Seminar in Tihany. Budapest, Research Institute for Linguistics pp 149-158.
Prószéky G, Kis B 1999 Agglutinative and Other (Highly) Inflectional Languages. Proceedings of the
37th Annual Meeting of the Association for Computational Linguistics College Park, Maryland,
USA pp 261–268
R. Hutás M 1974 Az Akadémiai Nagyszótár történetének vázlata (1898-1952) ‘The brief history of the
Unabridged Dictionary of Hungarian’ Nyelvtudományi Közlemények. LXXV. Budapest,
Akadémiai Kiadó, pp 447-465.

452
Korean grammatical collocation of predicates and arguments
Byungsun Park, Beom-mo Kang
Korea University
1. Introduction
This paper investigates the grammatical collocation of predicates and arguments in Korean in an
objective way through statistical methods. We investigate the statistical meaning of the co-occurrence
of the two words and the semantic relationship between them. 1
In the light of Korean grammatical structure, we study the collocation of the argument and the
predicate using statistical measures. The definition of the collocation in Korean is clear; however, on
the basis of previous studies, we can say that it is words with adjacency and high co-occurrence
relations.
Collocation has been an active field of study in English, and the results of these studies have been
used to do part-of-speech tagging and analyse the grammatical structure of large corpora.
Collocation studies of western languages generally focus on words close to each other. However,
in studying Korean, on the other hand, more meaningful results are produced by examining the cooccurrence
of words in a grammatical relationship within a sentence, such as subject and object.
We first investigate in how far the correlation between the occurrences of two words is
statistically meaningful, and then analyse the semantic relationships between them. The Korean
language has many auxiliary words used for case marking. For example, the subjective case is marked
by the auxiliary words ‘-i', ‘-ka'. Korean has developed its case mark system in this way. Therefore,
Korean language order is relatively flexible. In this view, rather than the relationship between keyword
and adjacency word, the relation between keywords and words that are grammatically related to the
keyword is more meaningful for collocation. This paper focuses especially on predicates as keywords.
In this paper, we focus on a methodology for analysing data, so we just introduce Korean
grammatical collocation by one word ‘boda –see ’ as an example. The verb ‘boda’ is extracted from the
Korea-1 corpus (H. Kim & B. Kang, 1996) that was balanced on a 10 million word scale.
2. Data analysing by statistics
2.1 Data processing
At first, the data in this paper was selected from the 100 most frequent verbs in the Korea-1
corpus. Then, concordances of these verbs were extract by a concordance package, and the argument of
every sentence was marked by hand. An example is given below.
S seongyosa-neun amu maleobsi i O chobeol-eul [bo-goman]
isseul ppun …
S missionary-sub without saying anything this O punishment-obj [see(stem)–only(ending)]
stand just …
“ The missionary, without saying anything, stood there, seeing this punishment…”
(S: subject marker, O: object marker)
In this way, the grammatical markers for every concordance sentence were marked, and the
frequency of the marked word for each argument extracted.
2.2 Statistical approach
Up to now, a large part of linguistic research has focused on determining whether the expressions
in a language are appropriate or not through intuition. This type of research is usually based on the
intuition of specifically native-speakers, rather than on objective data. In our research, however, we are
interested in finding out what are meaningful collocations not through intuition, but through actual
language use. Since the amount of language data used in corpus linguistics is usually large, a statistical
1 This paper is part of the research project in which the authors collaborated with Jong-sun Hong, and
Ho-cheol Choe of Korea University.
453
approach is essential. Moreover, a statistical approach reveals important information for natural
language processing and specific data for use in a general theoretical approach.
When two words that have a grammatical relation to each other co-occur and also when the
frequency of co-occurrence is higher than we can usually expect, we can say that the two words have a
collocation. From this point of view, statistical methods are used in this research. How exactly the
statistical measures are used will be described in the following section.
2.2.1 t-score
The t-score shows the difference between expected frequency of co-occurrence in the population
and observed frequency of those in the sample. In this paper, the population is the Korea-1 corpus. The
bigger the difference, the higher the degree of collocation. If a word occurs x times in the population,
and the number of observed words in the text is y, we can calculate the probability of x/y. For example,
a word, A, occurs 5000 times in the 10 million corpus, and the size of the observed text is 1000 words,
we can expect that a word, A, can occurs 0.5 times in the observed text. If the observed frequency of A
is larger than 0.5, it can be said that A is in collocation to the node. However, we consider not only the
difference between observed frequency and expected frequency but also the statistical significance. In
the character of t-score, the larger the observed frequency compared to the expected frequency, the
more exaggerated the t-score is expressed. Consequently, the t-score is not simply comparing the
difference between A and B. For example, if the t-score of A is 6.22 and the t-score of B is 3.11, it does
not mean that A is two times larger than B in collocation. In this view, statistical significance is
important.
The t-score formula is shown below:2
O
E O
t - =
Where:
O = the observed frequency of occurrence of the word within the span
E = the expected frequency of occurrence of the same word
And also, E is calculated like this:
E = the frequency of occurrence of the population X
size population
size span
_
_
2.2.2 MI-score (Mutual Information score)
The MI-score is different from the t-score when expressing collocation. The t-score is the
measured collocation by fixed node and co-occurring word. This means that the MI-score does not
express the difference between observed and expected frequencies in terms of standard deviation, but
represents the amount of information to which each of the two words, the node and its collocate, are
related. So, the MI-score provides information about the words in relation to each other by comparing
the observed probability of their co-occurrence with the expected probability, assuming that they are
distributed randomly. In the MI-score, it is assumed that the amount of information that each of the two
words contains is same. However, in this research, the MI-score measure shows that Korean is very
different from English.
Especially collocation studies of western languages generally focus on the words close to each
other. we believe, on the other hand, that more meaningful results are produced by examining the cooccurrence
of words in a grammatical relationship within a sentence. we therefore attempted to test the
MI-score in Korean collocation research, but the result was not appropriate to Korean collocation
research.
The MI-score formula is shown below:
I = log2
3. The collocation of the main argument
In this paper, the collocation is calculated by the statistical measure as previously stated. The
population is 10 million words of the Korea-1 corpus. The sample, the span, is calculated based on the
2 This formula and the explanation were described in Barnbrook (1996).
E
O
454
fact that one sentence in Korean usually consists of 12 words on average (B. Kang 1999, B. Park1997),
so the span is 12 * the frequency of concordance.
Even if the score is a negative number, it is also meaningful because negative numbers mean the
two words usually do not co-occur. Especially when a negative number also has statistical significance,
it gives a meaningful result.
3.1 The collocation of ‘boda (see)’ arguments
The verb ‘boda’ needs a subject and an object as arguments. The meanings of the verb ‘boda’ are
‘to see’ and ‘to induce or deduct'. The verb ‘boda,’ which means to see, only needs a subject and an
object, but when it means to ‘induce or deduct,’ it needs a subject (nominative), an object, and also an
adverbial expression. The result of the statistical measure of this is shown in the table below. Statistical
significance depends on the individual research propositions, so, in this paper, we assume the level of
significance of the t-score and MI-score to be over 1.96.
3.1.1 The main arguments
Table 1: t-score of the verb ‘boda’ 3
Argument Mark Word form Frequency t-score
O moseub(‘shape')-eul(Obj) 39 5.187
O geol(‘thing') 28 4.462
O na(‘I')-leul(Obj) 29 4.045
Oko issda(‘be-past')-ko(Q) 47 3.914
O yeonghoa(‘film')-leul(Obj) 20 3.838
O eolgul(‘face')-eul(Obj) 18 3.655
S Taisu(P) 13 3.602
O nunchi(‘sense')-leul(Obj) 12 3.288
S na(‘I')-neun(Sbj) 71 3.076
O sigieoi(‘watch')-leul(Obj) 8 2.702
S jeo(‘Ip')-neun(Sbj) 13 2.688
O Taisu(P)-leul(Obj) 7 2.644
O ccog(‘side')-eul(Obj) 8 2.570
O bakk(‘outside')-eul(Obj) 7 2.519
O sonhai(‘loss')-leul(Obj) 7 2.444
O kkol(‘side')-eul(Obj) 6 2.315
O pihai(‘damage')-leul(Obj) 8 2.194
S jeoi(Ip)-ga(Sbj) 9 2.049
O byeol(‘star')-eul(Obj) 5 2.037
Oko eobda(‘nothing')-go(Q) 14 2.036
S nu(‘who')-ga(Sbj) 12 2.032
O Soohai(P) 4 1.996
O duismoseub(‘back')-eul(Obj) 4 1.956
As table1 shows, the objectives are mostly over 1.96, which is statistically significant, and some
subjectives (nominatives) are also included in the table. But the other argument has not appeared. Some
proper nouns in subject places are not significant because this depends on the specific text. Therefore
they just appear without any collocational meaning.
Table 2: MI-score of the verb ‘boda'4
Argument Mark Word form Frequency MIscore
O Taisu(P)-leul(Obj) 7 10.219
S Taisu(P) 13 10.113
S Byeonghoa(P)-neun(T) 2 9.412
S Yeongjin(P) 2 9.412
3 Suj: subject case word, Obj: object case word, P: proper noun, Q: quotation mark word.
4 C: adverbial case word.
455
S juinajeossi(‘owner')-neun(T) 2 9.412
O Suhai(P)-leul(Obj) 4 8.827
Cr silyeon(‘loss of love')-eulo(C) 1 8.412
Cr2 yongdo(‘usaage')-lona (C) 1 8.412
Cx bangjeung(‘corroboration')-eulo(C) 1 8.412
Cx sayong(‘using')-ilagoman(C) 1 8.412
Cx junggandangieoi(‘middle step')-lo(C) 1 8.412
The MI-score, unlike the t-score, shows that the co-occurring words with low frequencies are
usually beyond the statistically significant score, 1.96. So the words with high MI-score are usually low
frequency items. This is very different from high t-score word list. But in English collocation research,
the high score t-score words and the high score MI-score words have a similar word list. In this aspect,
the collocation in Korean is very different from collocation in English. This paper just represents the
sample of the statistically significant words. We can conclude from this result that the degree of
dependency of the two words is very different in Korean. The assumption behind the MI-score measure
is that the dependency between two words is same. But Korean is somehow different in this respect, so
that we need other statistical measures for the mutually dependent approach. Since this analysis of the
MI-score is not needed, it will not be mentioned here.
3.1.2 The argument in subject
Table 3: the t-score of the subject place
Argument Mark Word form Frequency t-score
S Taisu(P) 13 3.602
S na(‘I')-neun(Sbj) 71 3.076
S jeo(IP')-neun(Sbj) 13 2.688
S jeo(IP)-ga(Sbj) 9 2.049
S nu(‘who')-ga(Sbj) 12 2.032
S namdeul(‘others')-i(Sbj) 4 1.818
S sonyeon(‘boy')-eun(T) 4 1.792
S nai(‘I')-ga(Sbj) 35 1.734
S Byeonghoa(P)-neun(T) 2 1.412
There are 5 words that have statistical significance in the subject place of ‘boda (see)'. The word
with the highest t-score is a proper noun, but, as previously stated, this just depends on the specific text
of the corpus. Due to its dependency on the text, this is not so meaningful from a semantic point of
view. Except for this, the word with the highest t-score word is the first person pronoun. The words, ‘na
(I)', ‘jeo (I-modest expression)', and ‘jieo (I-modest expression), are Korean first person pronouns.
From this perspective, the grammatical collocation of the verb ‘boda (see)’ is related to the cognition of
visual sense and the induction or deduction. We can also conclude from this that the first person
pronoun is the main subject of a cognition and an induction or deduction. The interrogative word ‘nuga
(who)’ is also related to a person, so we can conclude in the same way.
Table 4: the t-score of the object place
Argument Mark Word form Frequency t-score
O moseub(‘shape')-eul(Obj) 39 5.187
O geol(‘thing') 28 4.462
O na(‘I')-leul(Obj) 29 4.045
Oko issda(‘be-past')-go(Q) 47 3.914
O yeonghoa(‘film')-leul(Obj) 20 3.838
O eolgul(‘face')-eul(Obj) 18 3.655
O nunchi(‘sense')-leul(Obj) 12 3.288
O sigieoi(‘watch')-leul(Obj) 8 2.702
O Taisu(P)-leul(Obj) 7 2.644
O ccog(‘side')-eul(Obj) 8 2.570
O bakk(‘outside')-eul(Obj) 7 2.519
O sonhai(‘loss')-leul(Obj) 7 2.444
O kkol(‘side')-eul(Obj) 6 2.315
O pihai(‘damage')-leul(Obj) 8 2.194
456
O byeol(‘star')-eul(Obj) 5 2.037
Oko eobda(‘nothing')-go(Q) 14 2.036
O Suhai(P) 4 1.996
O duissmoseub(‘back')-eul(Obj) 4 1.956
In this table, the mark ‘Oko’ is an object quotation clause. This is an argument of the verb ‘boda
(see)’ as an opinion and an induction or deduction. It is 19 word forms that have statistical significance
in the object argument. Among them, it is 2 word forms that appear in the ‘Oko’ place. The other word
forms are usually related to the main meaning of ‘boda’ that is the cognition of sight. The words,
‘nunchi (sign)', ‘pihai (damage)', and ‘habui (consent),’ are the object arguments of ‘boda (see)', and it
means that the meaning of ‘boda’ is expanded to the cognition of abstraction. In the dictionary, this
expression is written as an idiom. This is an important piece of information for translation and Korean
teaching as a second language.
Table 5: the t-score of the other argument place
Argument Mark Word form Frequency t-score
C nun(‘eye')-eulo(C) 5 1.568
C bigoanjeog(‘pessimistic')-eulo(C) 2 1.379
C eolgul(‘face')-lo(C) 2 1.049
C geungjeongjeog(‘optimistic')-eulo(C) 2 1.047
C silyeon(‘loss of love')-eulo(C) 1 0.997
C yongdo(‘usaage')-lona (C) 1 0.997
C bangjeung(‘corroboration')-eulo(C) 1 0.997
C sayong(‘using')-ilagoman(C) 1 0.997
C junggandangieo(‘middle step')-lo(C) 1 0.997
The other argument does not appear with statistical significance. We can suppose that this other
argument helps the meaning of the object. These meanings are usually an instrument of the verb ‘boda
(see)'.
4. The negative score
As previously described, if the t-score is negative but has statistical significance, then we should
consider the word form as statistically significant. Because a negative t-score means that the two words
have a tendency not to co-occur. There are 90 word forms that have statistical significance with
negative scores. Even if we take a stricter approach to conformity with statistical significance, which is
3.96, then 71 word forms are in this standard. So we can predict that the argument of the verb ‘boda’
appears in various word forms. The table below shows a sample:
Table 6: the t-score of the statistical significant negative5
Argument Mark Word form Frequency t-score
Sp geos(‘thing')-eun(T) 1 -86.080
Op geos(‘thing')-i(C) 1 -81.200
O uli(‘we') 1 -64.038
Op geos(‘thing')-eun(T) 2 -60.160
Op geos(‘thing')-eul(Obj) 1 -52.169
Oko geos(‘thing')-elo(C) 1 -46.019
S uri(‘we') 2 -44.575
Sp na(‘I')-neun(T) 1 -44.084
O geo(‘he')-neun(T) 1 -33.096
Sp nai(‘I')-ga(Sbj) 1 -23.739
S nai(‘I') 1 -23.032
O geos(‘thing')-do(T) 1 -20.903
Op geos(‘thing')-do(T) 1 -20.903
O deung(‘etc')-eul(Obj) 1 -18.857
Op Hangug(‘Korea') 1 -17.727
5 Sp: a subject that follows a predicate, Ob: a object that follows a predicate.
457
O geos(‘thing') 4 -16.872
O geogeos(‘that')-eun(T) 2 -16.586
Sp geu(‘he')-nuen(T) 4 -15.048
As this table shows, the frequencies of most word forms are low, but the absolute value of t-score
is very high. In this table, however, it is very noticeable that the first person pronoun ‘Na (I)’ has a
negative t-score. This is contrary to the previous description that a first person pronoun is the main
subject of cognition and induction or deduction. The reason for this difference is the location in the
sentence. This means that the collocation depends on the location whether is in front predicate or not.
5. Conclusion
To sum up, this research focused on a grammatical collocation approach using the verb ‘boda
(see)’ as an example. As a result, the verb ‘boda (see)’ has many statistically significant collocations in
the object argument. In the subject argument, there were 5 word forms that had statistical significance.
These word forms are usually first person pronouns, and this is related to the meaning of the verb ‘boda
(see)'. This type of research is expected to be very useful for natural language processing and Korean
teaching as a second language.
References
(K) - Korean
B. Kang. 1999 Frequencies and Language descriptions Research for language information –1. Seoul:
Yonsei University, Center for Language and Information Development.(K)
B. Kang 1999 The text genre and language character of Korean. Seoul, Korea University Press. (K)
B. Park. 1997 The character of word use in Korean spoken language. MA thesis, Korea University (K)
D. Lee. 1998 The semantic research of Korean collocation. Seoul, Korea University MA thesis, Korea
University (K)
G. Barnbrook. 1996 Language and Computers. Edinburgh University press
H. Kim & B. Kang 1996 Korea-1 Corpus: the design and the organization Korean Linguistics 3. Seoul,
The association for Korean linguistics. (K)
J. Hong et al. 1998 The dictionary of modern Korean verb structure. Seoul, Doosandonga Press. (K)
J. Hong, B. Kang, & H. Choe 2000 The research of applied analyzing of Korean collocation
information, Korean Linguistics 11. Seoul, The association for Korean linguistics. (K)
J. Kim. 2000 Korean collocation research. PhD thesis, Kyunghee University (K)
J. Sinclair. 1991 Corpus, Concordance, Collocation. Oxford and New York, Oxford University Press
J. Yun. 1997 Korean structure analyzing by co-occurrence based word relation., PhD thesis, Yonsei
University. (K)
U. Paik. 1996 The introduction of statistics. Seoul, Jayuacademi Press. (K)

458
Corpus-based terminology extraction applied to information access
Anselmo Penas, Felisa Verdejo and Julio Gonzalo
{anselmo,felisa,julio}@lsi.uned.es
Dpto. Lenguajes y Sistemas Informáticos, UNED, Spain
Abstract
This paper presents an application of corpus-based terminology extraction in interactive
information retrieval. In this approach, the terminology obtained in an automatic extraction procedure
is used, without any manual revision, to provide retrieval indexes and a “browsing by phrases” facility
for document accessing in an interactive retrieval search interface. We argue that the combination of
automatic terminology extraction and interactive search provides an optimal balance between
controlled-vocabulary document retrieval (where thesauri are costly to acquire and maintain) and free
text retrieval (where complex terms associated to domain specific concepts are largely overseen).
1 Introduction
Although thesauri are widely used in Information Retrieval, their development requires laborintensive
processes with a high manual cost. On the other hand, new domains with specific conventions
and new terminology are continuously appearing. The development of terminology lists is a previous
step in thesaurus building that allows the use of automatic techniques in order to facilitate
documentalists’ labor.
Terminology lists can be seen as an intermediate point between free and controlled access to
information. They are used as indexes for document and resources access. In the context of education,
they are used in schools, libraries and documentation centers for database access (ERIC). Although
there are several available educational thesauri, the domain of new technologies in education is more
specific and requires the addition of new terms.
This work has been developed inside the European Treasury Browser (ETB) project which is aimed
to build the needed structures to organize and retrieve educational resources in a centralized web site
server. One of the main resources which is being developed inside ETB, is a multilingual thesaurus
whose terms will be used for describing the educational resources.
We describe the corpus-based method for the terminology extraction procedure used to extract and
suggest to documentalists the Spanish terms in the domain of new technologies, primary and secondary
education. The method is based on the comparison of two corpora extracted from the web: the first one,
an appropriate corpus in the domain and, the second, a corpus in a different and more general domain
(international news in our case). The comparison of terms in both corpora facilitates the detection of
specific terms of our domain. Section 2 will describe the methodology followed for the Terminology
Extraction (TE) procedure based on corpora and morphosyntactic analysis. Exploration and evaluation
of the results will be given in section 3.
Thesauri and controlled vocabularies are widely used in Information Retrieval (IR). We argue that
the cost of thesaurus construction can be skipped for IR purposes if constrains over terminology are
relaxed. This is possible in a framework of interactive text retrieval. TE then, becomes an appropriate
tool for providing richer indexing terms to IR indexes. Section 4 analyses the differences between IR
and TE which allows the relaxing of TE process, and shows a first prototype where Corpus-Based TE
is applied to interactive IR.
2 Corpus-based Terminology Extraction
Terminology Extraction (TE) tasks deal with the identification of terms which are frequently used
to refer to the concepts in a specific domain. Typically, automatic terminology extraction (TE, Term
Extraction) (ATR, Automatic Terminology Recognition) is divided in three steps (Bourigault, 1992)
(Frantzi et al., 1999):
1. Term extraction via morphological analysis, part of speech tagging and shallow parsing.
2. Term weighting with statistical information. The weight is a measure of the term relevance in
the domain.
3. Term selection, ranking and truncation of terminological lists by thresholds of weight.
These steps need a previous one in which corpora are obtained and prepared for the TE task. We
will distinguish between one word terms (mono-lexical terms) and multi-word terms (poly-lexical
459
terms), extracted with different techniques. The following subsections explain all the TE process
performed for the domain of interest: multimedia educative resources for primary and secondary school
in Spanish.
2.1 Construction of the corpora
The corpora have been constructed from web pages harvested with crawlers. These web pages need
preprocessing because they contain information which can disturb the term extraction process:
1. Treatment of html tags.
2. Deletion of pages in other languages than Spanish. For this task a language recognizer has been
used.
3. Deletion of repeated pages and chunks. Due to the continuous update of web site contents,
pages with different names but the same content are very frequent. This becomes a problem
since identical sequences of words in different documents gives a positive evidence of
terminological expressions. Repeated pages and chunks produce noise in statistical measures.
As mentioned above, the automatic terminology Extraction method (Manning and Schütze, 1999)
used in this work is based in the use of two corpora:
Educative Resources Corpus
The first corpus is related to the domain and contains useful terminology in order to classify,
organize and retrieve multimedia resources for secondary school. The pages of the two following web
sites have been collected with a crawler:
·  Programa de Nuevas Tecnologías:
http://www.pntic.mec.es/main_recursos.html
·  Aldea Global:
http://sauce.pntic.mec.es/~alglobal
This corpus has 1,075 documents and 670,646 words.
International News Corpus
With the aim of discarding frequent terms which are not domain specific, a second corpus has been
collected from the web. This corpus is composed by 7,364 international news from an electronic
newspaper (http://www.elpais.com), and has a size of 2.9 million words.
As explained below, the comparison of term frequencies in both corpora gives a relevance measure
for domain terminology.
2.2 Term detection
Texts were first tokenized. Abbreviations, erroneous strings and words from other languages than
Spanish were ignored. In order to obtain mono-lexical terms, texts were tagged on their part of speech
using (Márquez et al., 1997; Rodríguez et al., 1998) and only nouns [N], verbs [V] and adjectives [A]
were extracted. Different forms of the same word are counted by considering their lemma.
In order to detect poly-lexical terms, syntactic pattern recognition has been applied to the collection.
Complex noun phrases can be splitted into simpler ones, but all of them must be considered for
terminology extraction. The selection of the appropriate, even correct ones will be decided in
subsequent steps. For example, “distance education teachers” is a noun phrase which contains a simpler
one: “distance education”. Term detection phase requires the extraction of all candidate phrases.
The use of syntactic patterns is adequate for this task. Patterns enable to find all the phrases that
match them in all the documents of the collection. Patterns are defined as morphosyntactic tag
sequences. If the text contains a word sequence whose tags match the pattern, then a new phrase has
been recognized. The patterns do not attempt to cover all the possible constructions of noun phrases,
but only the ones that appear more frequently in terminological expressions. They were obtained
empirically after an iterative refinement of prototypes. The patterns used are listed in figure 1. Along
the pattern recognition process, a record of phrase occurrences and the documents in which they appear
is built.
460
Patterns recognized 72.453 candidate phrases in the “Educational Resources Corpus”. 75% of them
appear only once in the whole corpus and largely consist of wrong expressions or irrelevant terms for
the domain. As discussed later, it is not necessary to discard wrong expressions for Information
Retrieval purposes, but for term extraction, the correctness of the identified expressions is preferable
even if some relevant expressions are lost. In other words, precision is more important than recall.
Therefore, for the Terminology Extraction task a threshold for the number of phrase occurrences has
been defined, and all the expressions that appear only once in the educational corpus have been
discarded.
The results of the term detection phase are two lists:
1. a list of lemmas (mono-lexical terms), and
2. a list of terminological phrases (poly-lexical terms).
Every term is associated with its frequency and the number of different documents in which it
appears. Such statistics are obtained both for the educational corpus and for the newspaper corpus.
2.3 Term weighting
Term weighting gives a relevance value to every detected term in order to select the most relevant
terms in the domain. We have defined a weighting formula that satisfies the following constraints:
1. Less frequent terms in the domain corpus should have less relevance.
2. Highly frequent terms in the domain corpus should have higher relevance, unless they are also
very frequent in the comparison corpus or they appear in a very small fraction of the
documents in the domain corpus.
The formula considers:
1. Term frequency in the collection.
2. Document frequency of terms in the collection.
3. Term frequency in a more general domain.
where
Ft,sc: relative frequency of the term t in the specific corpus sc
Ft,gc: relative frequency of the term t in the generic corpus gc
Dt,sc: relative number of documents in sc where t appears.
Although a majority of documents in the educational corpus are related to the domain under study,
some documents contain very specific terms belonging to different domains (e.g. tales for children
about witches). If this kind of documents are long enough they can give high frequencies for non
relevant terms. To solve this problem, the measure considers document frequency. One term in the
domain must appear in several documents to be considered relevant.
2.4 Term selection
Three criteria have been used in order to reduce the number of candidate terms:
1. N N
2. N A
3. N [A] Prep N [A]
4. N [A] Prep Art N [A]
5. N [A] Prep V [N [A]]
Figure 1. Syntactic patterns for Spanish.
1
Relevance (t, sc, gc) = 1 –
Ft,sc · Dt,sc
log2 2 +
Ft,gc
461
1. Removal of unfrequent terms in the educational corpus (threshold=10). Terms not frequent in the
corpus have a low probability of being representative in the domain.
2. Removal of very frequent terms in the newspaper Corpus (threshold=1000). Terms which appear
very frequently in other domains have a low probability of being specific to our domain.
3. Selection of the first n (n=2000) terms ranked according to the relevance measure.
The thresholds used for the previous list truncation depend on the number of terms that will be
handled in the following phases. As we want to evaluate manually the precision of the automatic term
extraction process, the thresholds were adjusted in order to obtain between 2000 and 3000 candidate
terms.
Poly-lexical term frequencies do not behave as mono-lexical frequencies. Poly-lexical terms are
much less frequent than mono-lexical terms, but a couple of occurrences of a poly-lexical term give
high evidence of lexicalised expressions. For this reason, further criteria were needed to add polylexical
terms to the selection list attending to the relevance of their components. Through the
exploration of poly-lexical terms we corroborate that very few compounds without relevant
components were indeed relevant. Those terms were ignored and all the poly-lexical terms with
relevant components were added to the term list for manual revision.
3 Evaluation of the term extraction procedure
3.1 Visual exploration of results
Visual exploration of results is needed during the whole process:
·  to help in the decisions of prototype development and refinement,
·  to evaluate measures and techniques, and suggest modifications and improvements,
·  to give documentalists the possibility of exploring data in order to assist final decisions in
thesaurus construction.
We needed a simple, intuitive and comfortable way for data exploration. Thanks to hyperlinking,
the use of html pages has resulted appropriate for this task. The html pages containing the extracted
data were automatically generated in each iteration of the prototype. Figure 2 shows the pages with the
statistical data for mono-lexical relevance measure computation. In this case, terms are ordered by
relevance weight. The columns contain the following data:
1. Term frequency in the educational corpus
2. Document frequency in the educational corpus.
3. Term frequency in the newspaper corpus.
4. Number of compounds containing the term.
5. Relevance weight.
6. Whether the term is contained in a electronic dictionary or not.
7. Hyperlink to the page with all the contexts where the term appears in any inflected form.
8. Hyperlink to the page with all the compounds which contain the term in any inflected form.
The pages for poly-lexical terms contain, again, statistical information for each term, and hyperlinks
to keyword in context (KWIC) exploration pages. The KWIC pages have hyperlinks to the documents
the context belongs to in two versions, text and part of speech tagged files. These links allow a deeper
analysis of term contexts and better discrimination of the term senses in the collection.
462
3.2 Evaluation
The final list of candidate terms contained 2,856 mono and poly-lexical. The terms were manually
revised and classified to test the accuracy of the extraction process. Terms were classified as:
·  Incorrect, when the term is not acceptable in the language (Spanish in the example).
·  Non lexicalised, when it is correct but it does not have a specific meaning further than the
combination of meanings of its components.
·  Not in the domain, when the term is lexicalised but does not belong to the domain.
·  Adequate.
·  Specific domain, when it should be part of a microthesaurus inside the domain.
·  Computers domain.
·  Variant, when the term has already been considered in some other flexive form.
Tables 1 and 2 show the classification of the candidate terms.
Adequate Specific
domain
Computers Variants Total of
terms
1235
43.24%
513
17.96%
59
2.07%
78
2.73%
2856
100%
Table 1. Correct terms.
Incorrect Not
lexicalised
Not domain Total of
terms
151
5.29%
515
18.03%
305
10.68%
2856
100%
Table 2.Incorrect and not adequate terms.
The appropriate terms with the manual classification were used as the input to documentalists to
produce the thesauri. They were 66% of the terms automatically selected, an indication that the
automatic procedure can be useful in detecting phrases for Information Retrieval purposes, as it is
discussed in the next sections.
Figure 2. Visual exploration of terms
463
4 Terminology based Information Retrieval
Traditional Information Retrieval only uses mono-lexical terms for collection indexing. Arbitrary
consideration of N word sequences (n-grams) generate indexes too large to be useful. An intermediate
approach is to add only terminological phrases to the collection index. In this way, the term extraction
procedure described above has been applied to indexing tasks for Information Retrieval.
However, besides the addition of useful terms to document indexes, the use of terminology gives
another possibility: instead of navigating through the collection documents, it's possible to navigate
through the collection terminology and access the documents from the relevant terms (Anick et al.,
1999; Jones et al., 1999). Furthermore, the consideration of two areas, one for document ranking and a
second for term browsing, opens an interesting way for interactive information retrieval.
Along these lines, a first monolingual prototype has been developed in order to explore term
browsing possibilities for accessing information. Indexing and retrieval are described in the following
subsections.
4.1 Indexing
In the terminology extraction task, the goal is to decide which terms are relevant in a particular
domain. In the Information Retrieval task, on the other hand, users decide the relevant terms according
to their information needs. This implies that, for IR purposes, precision at indexing time should be
sacrificed in favor of a higher recall, delaying the determination of which phrases are relevant until the
user poses a particular query. For this reason, the terminology extraction procedure described above has
been used here ignoring one step: term list truncation.
The term extraction process has been adapted to keep all indexable phrases, ranked but without a
cutting threshold. The ranking is used to structure and organize the relevant phrases for a given user's
query.
In the indexing phase, the lemmas of nouns and adjectives are linked to the phrases in which they
participate. The phrases, in turn, are linked to the documents in which they appear. Indexing levels are
shown in figure 3.
The first level of indexing provides access to phrases from the query's isolated terms. The second
level supplies access to documents from the selected phrases.
4.2 Retrieval
From the above indexing schema, the retrieval process follows this steps:
1. Through the interface shown in Figure 4, the user provides to the system the searching terms
(‘Look for’ text area) . Terms do not need to have any syntactic structure as phrases or
sentences.
2. The system obtains the mono-lexical terms related to the query through their lemmatization and
categorization.
3. Poly-lexical terms are retrieved from mono-lexical ones through the corresponding index.
4. From poly-lexical terms, documents which contain them are retrieved.
5. According to their relevance weight, terms are organized and shown to users (rightmost area in
Figure 4) to allow document access directly from terms.
6. Documents are also ranked and shown in a different area (leftmost area in Figure 4). Ranking
criteria are based on the number of identified terms contained in documents.#
Lemma
Phrase
Document
Figure 3. Indexing Levels
464
Both areas, term area and document area have two kind of links:
1. Links for exploring the selected documents.
2. Links for exploring the term contexts in the collection (Figure 5). From these contexts user can
select and access the relevant documents.
Figure 4. Website Term Browser interface.
Figure 5. Term contexts with links to documents.
465
5 Conclusions and future work
The work has been developed in the context of ETB project with two objectives. First, to provide to
documentalists the Spanish terminology for a thesaurus building in the domain of new technologies,
primary and secondary education. This thesaurus will provide multilingual structure for resources
organization and retrieval. This paper has shown the methodology used for the automatic extraction of
Spanish terms.
Second, the extracted terminology has been used in a first prototype for information access. The
developed search engine gives an intermediate way for information retrieval between free searching
and thesaurus-guided searching in an interactive framework. In this prototype, documents are
accessible from the terminological phrases suggested by the system after user's query. As users usually
don't make use of the same phrases contained in the collection, this approach bridges the distance
between the terms used in queries and the terminology used in the collection.
Our present interest is focused in extending this work to cross-language information access, where
the use of phrases provides not only a way for document accessing but also an excellent way for
reducing ambiguity in query translation (Ballesteros et al., 1998). We plan to extend this way of
disambiguation not only for query translation but also for query expansion and term variation. For
query expansion and translation we plan to use the synonymy, hyper/hyponymy and meronymy
relations of EuroWordNet (Vossen 1998), developing a complete multilingual framework for
interactive information access.
Acknowledgments
This work has been partially supported by the European Commission, ETB project IST-1999-
11781.
References
Anick P G and Tipirneni S 1999 The Paraphrase Search Assistant: Terminological Feedback for
Iterative Information Seeking. Proceedings of 22nd ACM SIGIR Conference Research and
Development in Information Retrieval. 153-159.
Ballesteros L and Croft W B 1998 Resolving Ambiguity for Cross-Language Information Retrieval.
Proceedings of the 21st Annual International ACM SIGIR Conference on Research and
Development in Information Retrieval. 64-71.
Bourigault D 1992 Surface grammatical analysis for the extraction of terminological noun phrases.
Proceedings of 14th International Conference on Computational Linguistics, COLING'92. 977-
981.
ERIC: http://ericae.net
Frantzi K T, Ananiadou S 1999 The C-value/NC-value domain independent method for multiword term
extraction. Journal of Natural Language Processing. 6(3):145-180.
Jones S, Staveley M S 1999Phrasier: a System for Interactive Document Retrieval Using Keyphrases.
Proceedings of the 22nd ACM SIGIR Conference on Research and Development in Information
Retrieval. 160-167.
Manning C, Schütze H 1999 Foundations of Statistical Natural Language Processing. MIT Press.
Cambridge, MA.
Márquez L, Padró L 1997 A flexible POS tagger using an automatically acquired language model.
Proceedings of ACL/EACL'97.
Rodríguez H, Taulé M, Turmo J 1998An environment for morphosyntactic processing of unrestricted
Spanish text. Proceedings of LREC'98.
Vossen P 1998 Introduction to EuroWordNet. Computers and the Humanities, Special Issue on
EuroWordNet.

466
Multi-word Unit Alignment in English-Chinese Parallel Corpora
1. Introduction
Multi-word unit (MWU) alignment in bilingual/multilingual parallel corpora is an important goal for
natural language engineering. An efficient algorithm for aligning MWUs in different languages could
be of use in several practical applications, including machine translation, lexicon construction and
cross-language information retrieval. A number of algorithms have been proposed and tested for this
purpose, including collocation and association-strength testing statistics (Dagan et al., 1994; Smadja et
al., 1996), n-gram, approximate string matching techniques (ASMT), finite state automata, (McEnery
et al., 1997), bilingual parsing matching (Wu, 1997), and a hybrid connectionist framework (Wermter
et al., 1997). Despite the work undertaken to date, however, reliable and robust MWU alignment
remains an elusive goal.
In this paper, we describe an algorithm which combines the n-gram approach, linguistic filters and cooccurrence
statistical metrics to extract and align English and Chinese nominal MWUs in English-
Chinese parallel corpora. We decided to focus on nominal MWUs as they are stable and form the
largest group of MWUs in natural language1. This algorithm has been evaluated on a sentence aligned
English-Chinese Parallel corpus (Piao, 2000). It obtained precision rates of 92.17% and 87.37% for
English and Chinese nominal MWU extraction respectively while it obtained a precision rate of
80.63% for English-Chinese MWU alignment.
2. Related works
As noted, a number of techniques have already been applied to the problem of MWU extraction and
alignment, including Smadja et al.’ (1996) collocation translation system, McEnery et al.'s (1997)
approximate string matching techniques (ASMT) and finite state automata, and Wu's (1997) stochastic
inversion transduction grammars used to align Chinese-English phrases.
Dagan and Church (1994) also describe a semi-automatic tool, Termight, which extracts technical
terms and translations from parallel corpus data. This tool first extracts candidate multi-word noun
phrases and single words from a POS tagged corpus using a syntactic pattern filter. Then it groups
terms by head words, and sorts terms within each group in reverse word order. Finally, concordance
lines are produced so that human experts can distinguish true terms from the candidate terms. Then, the
Termight aligns bilingual MWUs using aligned words. For each source term, the tool identifies a
candidate translation by selecting a sequence of target words whose first and last word are aligned with
any of the words in the source term. Bilingual concordances are produced for selected term pairs to
allow terminologists to verify true translations. Dagan et al, (1994) tested Termight on 192 terms found
in English and German versions of a technical manual. They found that in 40% and 7% of the cases the
first and second target language candidates respectively were the correct translations. In other cases, the
correct translation was always somewhere in the concordances.
The Champollion system of Smadja et al.’ (1996) can produce translations of the source collocations in
the target language. It is based on a MWU extraction system, Xtract, which was also developed by
Smadja et al. (1993). They first extracted source language (English) collocations with Xtract. After
that, for each source collocation, they extracted its translationd in the target language (French) by
testing Dice-score and collocation lists. Champollion was tested on the Hansard corpus and an accuracy
of between 65%-78% was reported.
1 Biber et al. (1999: 231) observe: “In the news report, nominal elements make up about 80 per cent of
the text (measured in terms of words). The corresponding figures for the other text samples are
approximately: academic prose 75 per cent, fiction 70 percent, and conversation 55 per cent. In other
words, nominal elements make up between a half and four-fifth of the text.”
Scott Songlin Piao
Department of Computer Science
University of Sheffield
Email: s.piao@dcs.shef.ac.uk
Tony McEnery
Department of Linguistics and MEL
Lancaster University
Email: mcenery@comp.lancs.ac.uk
467
McEnery et al. (1997) tested extracting multi-word cognates from parallel corpora using ASMT and
finite state automata. In this algorithm, all n-grams from source text are compared against all potential
m-grams in an aligned region of the parallel text. Dice's similarity coefficient is calculated for each pair
(n, m). The (n, m) pair which gains the highest score is selected as the best potential MWU cognate.
For 3,142 windows matched in a million-words English-Spanish parallel corpus, an overall precision of
96.5% was reported. They also developed finite state automata for typical English and Spanish
compound noun constructions to extract alignment candidates, then filtered the candidates with a
similarity score. This effectiveness of the technique was found to be directly linked to the strength of
the similarity scores computed for candidate terms.
Wu (1997) approached Chinese-English phrase alignment by bilingual parsing with stochastic
inversion transduction grammars. Wu reports that his approach produced phrase alignments as long as
12 English tokens and 15 Chinese characters. After pruning, a precision rate of 81.5% (based on
random samples drawn from about 2,800 phrasal alignments) was reported.
3. A hybrid algorithm for aligning English-Chinese nominal MWUs
In this paper, we describe an algorithm of nominal MWU alignment in which a number of techniques
outlined in the previous section are combined. Our approach uses n-grams, POS filters and cooccurrence
metrics in concert. The n-gram approach is used to extract candidate MWUs. POS filters
are then used to extract a shortlist of candidate nominal MWUs. After this, co-occurrence metrics are
used to extract true nominal MWUs and their alignments (explained in detail later in this paper). To
show the effectiveness of this approach, an English-Chinese parallel corpus is used as a testbed. This
corpus contains 61,534 English words and 98,537 Chinese characters. The English data in the corpus
was POS tagged with the Lancaster CLAWS tagger (Garside et. al. 1987) and the Chinese data was
tagged with Zhang et al's (2000) Chinese tagger. Additionally, the corpora have been sentence aligned
using Piao's (2000) program.
One assumption underlying the approach taken to nominal MWU alignment in this paper is that
nominal MWUs in the source language are generally translated into a nominal MWU (hereafter, MWU
in this paper refers to nominal MWU) in the target language, and hence their occurrences in the parallel
translation texts are correlated. Based on this assumption, we propose that nominal MWU alignment
can be approached through a) first extracting significant MWUs from each language in the parallel
corpus, then b) aligning them based on their co-occurrence affinity. An algorithm was designed to
implement this approach. Two main stages are involved in this algorithm: a) English and Chinese
MWU extraction, b) MWU alignment.
3.1. Candidate Nominal MWU extraction
The first stage of our algorithm is to extract candidate nominal English and Chinese MWUs from the
English-Chinese parallel corpus. We assume that most MWUs are stable continuous strings of words.
Therefore we adopted an n-gram approach to extracting candidate MWUs from the corpus texts. In
order to remove irrelevant candidates from the process, simple POS filters are used to filter out n-grams
whose POS structures are unlikely to constitute nominal MUWs.
Firstly, candidate MWUs are extracted from the corpus with the following algorithm:
(1) Extract English and Chinese n-grams (2 <= n <= 6)2 from the English and Chinese section of the
corpus respectively. Considering the unreliability of statistical scores for low frequency items,
those n-grams whose frequency is lower than three were ignored.
(2) An English and Chinese POS filter is used to filter out those n-grams whose POS patterns are
unlikely nominal MWUs .
In the first step of this algorithm, any one of the n-grams extracted, from bi-grams to 6-grams, can be
candidate MWUs. However, as noted, many of these n-grams have POS sequences that make them
2 Because only twelve 6-grams with frequencies equal or greater than three were found after POS filtering in the
tesbed corpus, we assumed that the nominal MWUs longer than six words are non-existent in this corpus.
468
unlikely candidate nominal MWUs. These false MWUs cause “noise” for the algorithm, and hence it is
desirable to filter them out before proceeding.
In filtering the candidate n-grams, the POS sequence associated with each n-gram is matched against
simple POS filters. The English POS filter3 is a simple rule system that excludes any of the candidates
if the any of the following conditions are met:
1) For the initial word of a n-gram, if
a) the initial code of its POS tag is any one of “A”, ‘C', ‘D', ‘G', ‘I', ‘M',
‘P', ‘R’ or ‘T', or
b) the initial code of its POS tag is ‘V’ and the last code is not ‘G’ or ‘N', or
c) it has the POS tag of “AT”.
2) For the last word of a n-gram, if its initial POS code is not ‘N'.
Table 1 shows the English POS categories denoted by the POS tags or tag-initials used in the English
filter (for English C7 tagset see appendixIII in ).
Code Category
* Any codes
A* Possessive pronouns and articles
C* Conjunction
D* Determiner
G* Genitive markers ‘ and ‘s
I* Preposition
M* Numeral
N* Noun/Proper noun
P* Pronoun
R* Adverbs
T* Infinitive marker “to”
V* Verb
V*G -ing verb
V*N Past participle
Table 1: English POS categories denoted by the codes in the English POS filter
The Chinese POS filter4 works by excluding candidate Chinese n-grams if any of the following
conditions are met:
1) For the initial word,
a) the initial code of its POS tag is any of ‘C', ‘D', ‘M', ‘P’ and ‘V',
c) its POS tag is “AUX”, “AV0” or “NMW”.
2) For the last word, the initial code of tag is not ‘N'.
Table 2 shows the Chinese POS categories denoted by the codes in the Chinese filter.
Code Category
AUX Auxiliary word
AV0 Adverb
C* Conjunction
D* Markers “ ”, “ ” and “ ”
M* Numeral
P* Pronoun and determiner
V* Verb
3 For C7 tagset, see appendix III in Roger Garside, Geoffrey Leech, Anthony McEnery (eds) (1997) Corpus
annotation, London & New York, Longman.
4 The Chinese tagset used by Zhang et al's tagger was modified. For details, see appendix.
469
Table 2: Chinese POS categories denoted by the codes in the Chinese POS filter
A notable feature of the filter is that it does not thoroughly examine n-grams in the sense that it only
considers the initial and last words of the n-grams. In spite of the simplicity of this filtering mechanism,
it generally performs well, particularly on bi-grams and tri-grams. For example, in an experiment, these
filters filtered out 6,009/4853 irrelevant English/Chinese bigrams from 6767/5599 English/Chinese bigrams.
Thus while the filter is effective, it does not seem to exclude good candidates while
demonstrating a fair degree of success at filtering out bad candidates.
3.2. Seed-gram extraction
After the POS filter has eliminated a proportion of the bad candidate n-grams, a further filtering
process, to identify the true nominal MWUs from the candidate list, is necessary. In order to achieve
this, we identify seed-grams. Seed-grams are short MWUs, including bi-grams and tri-grams, in which
the element-tokens are strongly associated. Such seed-grams are identified by testing their cooccurrence
associations (for bi-seed-grams) and some specific POS patterns (for tri-seed-grams). It is
assumed that a good nominal MWU must contain one or more seed-grams. Therefore, seed grams can
be used to find longer significant nominal MWUs, i.e. a seed-gram can “grow” longer.
Bi-seed-grams are extracted as follows. For each bi-gram, the mutual information (MI)5 and t-score is
calculated. These scores reflect the co-occurrence affinity between the two tokens of the bi-gram.
These two scores are calculated by the following formulas:
Mutual information
t-score
where, a, b, c and d are elements of a contingency table. For example, given a bi-gram containing
tokens x and y,
a = number of bi-grams in which both x and y occur;
b = number of bi-grams in which only x occurs;
c = number of bi-grams in which only y occurs;
d = number of bi-grams in which neither x nor y occurs.
For the purposes of seed-extraction, thresholds of 1.65 for MI and –3 for t-score are used6. If a given
bi-gram produces both an MI and t-score greater than the thresholds, it is accepted as a seed-gram. This
process does not vary in the sense that the same algorithm is applied to both English and Chinese.
When applied to the testbed corpus, this algorithm extracted 414 English seed-grams and 315 Chinese
seed-grams out of the 758 and 746 English and Chinese candidates respectively. All of the extracted
seed-grams were found meaningful and accepted to be true seed-grams, giving the technique a
precision of 100%. However, the recall for the technique for both English and Chinese seedgram
extraction is significantly lower at 52.74% and 42.22% respectively. Figures 1 and 2 show examples of
English and Chinese seed grams.
5 In formula 1, numerator a is squared, for previous study shows that this modified formula performs better than
the original one in which the exponent of a is one (see Piao, 2000).
6 These parameters were established empirically for the corpus we tested the algorithm on. We accept that these
parameters may vary depending upon the corpus being exploited.
.
) (
) )( (
) , (
1
) ( ) ( ) , (
d c b a a
c a b a
a
W W prob
M
W prob W prob W W prob
t
b a
b a b a
+ + +
+ + - = - =
,
) )( (
log
2
2
2
c a b a
a
MI
+ +
=
470
MI t-score f Seed gram POS pattern
--------------------------------------------------------------------------------------
4.717 6.527 43 Education Commission NN1 NNJ
4.373 4.990 25 Nobel Prize NP1 NN1
4.334 4.683 22 Lianhe Zaobao NP1 NP1
4.272 5.087 26 hanyu pinyin NN1 NN1
4.164 4.682 22 Hong Kong NP1 NP1
4.005 6.864 48 higher learning JJR NN1
3.816 4.236 18 United States NP1 NP1
3.700 3.603 13 Chen Qing-shan NP1 NP1
Figure 1: A sample of top English seed grams
MI t-score f Seed gram POS pattern
------------------------------------------------------------------------------------------
4.431 9.497 98 AJ0 NN0
4.130 5.627 32 NN0 NN0
4.104 7.539 59 AJ0 NN0
3.286 3.990 16 NN0 NN0
3.225 8.569 85 AJ0 NN0
3.126 5.919 36 NN0 NN0
2.831 3.311 11 NN0 NN0
2.645 5.405 30 NN0 NN0
Figure 2: A sample of top Chinese seed grams
For trigrams, POS matching patterns are used for extracting seed-grams, as shown below. Trigrams are
matched against POS patterns and deemed to be seed grams if they match the following patterns for
English and Chinese:
(1) English POS patterns:
[Noun/Proper_noun + of/genitive_marker + Noun/Proper_noun]
(2) Chinese POS patterns:
[Noun/Adjective/Proper_noun + de( ) + Noun/Proper_noun]
These heuristic patterns were developed on the basis of the grammatical properties of English and
Chinese noun phrases. This proved to be an effective technique, as when tested on the corpus, all of the
tri-grams extracted with these POS patterns proved to be significant nominal MWUs. Fig. 1 shows
sample tri-seed-grams extracted in this way.
f Eng. tri-seed-gram POS pattern f Chi. tri-seed-gram POS pattern
------------------------------------------------------------------------------------------------------------------
33 People ’s Republic NN GE NN1 6 NP0 DE1 NN0
30 Republic of China NN1 IO NP1 5 NN0 DE1 NN0
8 institutions of China NN2 IO NP1 5 NP0 DE1 NN0
6 China ’s education NP1 GE NN1 5 NP0 DE1 NN0
6 Ministry of Education NN1 IO NN1 4 AJ0 DE1 NN0
6 number of people NN1 IO NN 4 AJ0 DE1 NN0
6 number of women NN1 IO NN2 4 NN0 DE1 NN0
6 women ’s education NN2 GE NN1 4 AJ0 DE1 NN0
Fig. 3: A sample of English and Chinese seed tri-grams
As shown previously, the n-gram approach, simple POS filters and co-occurrence metrics combined
provide an efficient algorithm for extracting significant short MWUs, or seed-grams. The process of
471
‘growing’ these seed-grams to identify nominal MWUs of length greater than 3 is described in the
following section.
3.3. Extracting longer MWUs based on2 and 3 length seed-grams
In natural languages, a nominal MWU can clearly be longer than three words. As discussed previously,
our supposition was that if an MWU is significant, it is likely to contain one or more seed-grams;
conversely, if an MWU contains one or more seed-grams, it is likely to be significant. Based on this
assumption, we used seed-grams to identify true MWUs of a length greater than 3.
To determine the usefulness of this approach to extracting nominal MWUs, we applied the the
hypothesis to n-grams of length 3 – 67. The POS filters used to filter out candidate nominal MWUs
work well on short n-grams, but work less efficiently on longer n-grams. Hence we applied a further
filter to candidate n-grams (3 <= n <=6) as follows:
a) English n-grams containing tags “VV*” (verbs) or “APPGE” (pre-nominal possessive pronoun)
are filtered,
b) Chinese n-grams containing “VV0” (verbs), “VM” (modal verbs) are filtered,
where the asterisk denotes any letter(s).
The candidates survived the pruning are taken as candidate nominal MWUs. Each of them is matched
against the seed-grams. Those which contain one or more seed-grams are accepted as nominal MWUs.
In the experiment, This approach extracted 626 English nominal MWUs and 467 Chinese nominal
MWUs from the corpus. Figures 4 and 5 show samples of the extracted English and Chinese noun
MWUs respectively.
f MWU POS
---------------------------------------------------------------------
6 Goh Chok Tong NP1 NP1 NP1
3 Gross Domestic Product JJ JJ NN1
7 HIV infection NP1 NN1
3 Hanyu pinyin NN1 NN1
5 Harvard University NP1 NN1
3 Health Information NN1 NN1
3 Health Publications NN1 NN2
3 Health Publications Unit NN1 NN2 NN1
4 Health Service NN1 NN1
3 Heywood Stores NP1 NN2
Figure 4: A sample of automatically extracted English nominal MWUs
f Nominal MWU POS Tags
-------------------------------------------------------------------
4 NN0 NMW PND NN0
30 AJ0 NN0
3 AJ0 NN0 NN0
98 AJ0 NN0
3 AJ0 NN0 CJ0 NN0 NN0
10 AJ0 NN0 NN0
14 AJ0 NN0 AJ0 NN0
3 NP0 NN0
6 NN0 NN0
3 AJ0 DE1 NN0
Figure 5: A sample of automatically extracted Chinese nominal MWUs
7 Note we still consider n-grams of length 3 at this stage, as these may be missed by the POS pattern matcher, but
contain significant seed-grams of length 2.
472
A manual examination of the results showed precision rates of 92.17% and 87.37% for English and
Chinese respectively. In a further analysis, it was found that 59 Chinese bad MWUs, or 20.34% of the
mistakes, were caused by errors in the Chinese POS tagging. This means that, if a more accurate
Chinese POS tagger is available, a higher success rate could reasonably be expected for Chinese. Given
the considerably high precision yielded throughout the various stages of the technique developed for
monolingual nominal MWU extraction, we use the output from the English and Chinese algorithms as
the basis upon which alignment of MWUs between the two languages is attempted.
3.4. Aligning English and Chinese MWUs
With the English and Chinese MWUs extracted the next step is to align them. As assumed previously,
the English and Chinese translation equivalents are generally expected to co-occur in corresponding
sentence translations. Since the test-bed corpus is aligned at sentence level, it is possible to test cooccurrence
affinity between the candidate MWUs at sentence level.
Again, the mutual information (MI) and t-score (see formulae 1 and 2) are used for testing cooccurrence
correlation. For each English MWU xi (i = 1, 2, …, m) every Chinese MWU yj (j = 1, 2, …,
n) is considered as a potential translation (n and m denote the numbers of English and Chinese MWUs).
For a given xi, a contingency table is extracted against every yj. The contingency table contains
elements, a, b, c and d which are defined as follows:
(1) a denotes the number of aligned English-Chinese sentence pairs in which both xi and yj occur;
(2) b denotes the number of aligned English-Chinese sentence pairs in which only yj occurs;
(3) c denotes the number of aligned English-Chinese sentence pairs in which only xi occurs;
(4) d denotes the number of aligned English-Chinese sentence pairs in which none of xi and yj occur.
It was found that letter case distinctions in English caused considerable “noise” by dispersing
frequencies. For instance, “GREAT BRITAIN” and “Great Britain” are indexed as different items
despite the fact that they are variants of a single MWU, dividing their common frequency between
them. In order to avoid this problem, all lowercase was taken as the canonical form of English MWUs.
The MWU translations are identified as follows:
1) For each English MWU, all of the Chinese candidate MWUs are collected;
2) The Chinese candidates with t-scores lower than 1.658 are filtered out;
3) The candidates with MI lower than –0.2 are removed;
4) Finally, the surviving candidates are sorted by MI into descendent order.
5) The top one accepted as true Chinese translation of the given English MWU.
6) If no Chinese candidate survives the filtering, the given English MWU is ignored.
In the experiment, out of the 626 English nominal MWUs and 467 Chinese nominal MWUs extracted
by the monolingual extraction process, the alignment algorithm extracted 191 potential English-
Chinese MWU alignments. Figure 6 shows a sample of aligned MWU pairs. In the sample, the first set
of square brackets encloses the frequency of the English MWU and the second set of square brackets
contains the MI-score, co-occurrence frequency and frequency of the Chinese MWU.
1) [48] chinese_JJ culture_NN1:
(1) [2.3339 ; 22; 44] _NP0 _NN0
-----------------------------
2) [8] chinese_JJ intellectual_NN1:
(1) [0.9475 ; 6; 14] _NN0 _NN0
(2) [-0.5670 ; 3; 5] _NN0 _NN0 _NN0
-----------------------------
3) [7] chinese_JJ intellectual_NN1 and_CC cultural_JJ elite_NN1:
(1) [1.1402 ; 6; 14] _NN0 _NN0
(2) [-0.3744 ; 3; 5] _NN0 _NN0 _NN0
-----------------------------
4) [10] chinese_JJ language_NN1 teachers_NN2:
(1) [0.3455 ; 6; 17] _NN0 _NN0
8 A t-score threshold of 1.65 indicates the significance level of MI is above 95%.
473
-----------------------------
5) [16] chinese_JJ singaporeans_NN2:
(1) [0.9475 ; 6; 7] _NN0 _NP0 _NN0
(2) [-1.5850 ; 4; 12] _NP0 _NN0
(3) [-1.7370 ; 6; 45] _NP0 _NN0
(4) [-1.8301 ; 3; 6] _NP0 _NN0
-----------------------------
6) [12] chinese_JJ studies_NN2:
(1) [3.3339 ; 11; 11] _NN0 _NN0
(2) [0.3808 ; 5; 8] _NP0 _NN0
(3) [0.3808 ; 5; 8] _NP0 _NN0 _NN0
(4) [0.0589 ; 5; 10] _NN0 _NN0
-----------------------------
7) [4] cigarette_NN1 rolling_JJ tobacco_NN1 box_NN1:
(1) [-0.5850 ; 2; 3] _NN0 _NN0
-----------------------------
8) [3] coffee_NN1 shops_NN2:
(1) [0.8480 ; 3; 5] _NN0 _NN0
(2) [-0.1699 ; 2; 3] _NN0 _NN0
-----------------------------
9) [8] cold_JJ weather_NN1:
(1) [0.1699 ; 3; 3] _AV0 _AJ0 _DE1 _NN0
(2) [0.1699 ; 3; 3] _AJ0 _DE1 _NN0
(3) [-1.5850 ; 2; 3] _AJ0 _NN0
-----------------------------
Figure 6: A sample of aligned English-Chinese nominal MWUs
As shown in figure 6, for most of the English MWUs more than one candidate survives the filtering.
But three of them, “Chinese culture”, “Chinese language teachers” and “cigarette rolling tobacco box”
are precisely matched with unique candidates “ ” “ ” and “ ”9.
Due to multiple translations, some English MWUs may have more than one true translation in Chinese.
For example in figure 6, “Chinese Singaporean” is translated as both “ ” and “
”. In this particular case, the English MWU and the two Chinese translations occurred for 16 , 7,
and 12 times in the corpus. Of the Chinese candidates, “ ” co-occurred with the English
MWU 6 times (in the same aligned English-Chinese sentence pairs) while “ ” co-occurred
with the English MWU only four times. This shows that, in the corpus “Chinese Singaporean” is
translated as “ ” more often than “ ”. Their MI-scores, 0.9475 and -1.5850
reflect the situation accurately.
Considering that for a given nominal MWU more than one true alignment may exist, the process of
identifying true and false alignments is somewhat complex. To represent this complexity, in the
evaluation of the MWU alignment algorithm, two categories are used for ‘correct’ alignments, precise
match and partial match. Precise match refers to an exact match between the source MWU (English in
this case) and the top target MWU (Chinese in this case) in the candidate list. Partial match refers to
cases in which alignments are approximate matches or where the correct translation is the second
ranked item from the candidate list. For example in figure 6, Pair (1) is judged to be a pair of precise
matches while pairs (2), (3) and (8) are judged to be partial matches. A manual examination revealed
99 precise matches, 55 partial matches and 37 mismatches among the total 191 potential MWU
alignments. If precise matches only are taken into account, the precision of the technique is 51.83%; if
both precise and partial matches are considered, the precision score increases to 80.63%. In both cases
recall is significantly lower; recall is calculated by dividing the number of English MWUs with the
number of resultant MWU alignments, giving a recall (100% ´ 191/626=) 30.51%.
4. Conclusion
Bilingual/multilingual MWU alignment is a challenging but worthwhile task. An efficient and effective
system for MWU alignment will be of use to a number of areas including bilingual/multilingual
9 All of these MWUs reflect major topics in the corpus.
474
contrastive studies, machine translation and multilingual lexicon building. Although much effort has
been made in this area, no satisfactory solution has been found yet.
In this paper, we described a hybrid algorithm of English-Chinese MWU alignment which combines
the n-gram approach, POS filters and co-occurrence coefficients. Given a sentence aligned English-
Chinese parallel corpus, this algorithm automatically identifies and aligns nominal English and Chinese
MWUs with high precision, but relatively low recall. As the result shows, it is a practical algorithm for
extracting MWU alignments. So while a limited success has been achieved, it provides an inexpensive
but practical tool for aligning nominal English-Chinese MWUs with a high degree of precision. Also,
although not tested, the possibility exists that this algorithm could be ported to other language pairs by
modifying the POS filters.
References:
Garside, Roger, Leech, Geoffrey and Sampson, Geoffrey 1987 The Computational Analysis of English,
London, Longman.
McEnery Tony, Langé Jean-Marc, Oakes Michael, Véronis Jean 1997 The exploitation of multilingual
annotated corpora for term extraction. In Garside Roger, Leech Geoffrey, McEnery Anthony
(eds), Corpus annotation --- linguistic information from computer text corpora, London & New
York, Longman, pp 220-230.
Piao Scott Songlin 2000 Sentence and word alignment between Chinese and English. Ph.D. thesis,
Lancaster University.
Smadja Frank 1993 Retrieving collocations from text: Xtract. In Computational Linguistics 19(1): 143-
177.
Smadja Frank, McKeown Kathleen R., Hatzivassiloglou Vasileios 1996 Translating collocations for
bilingual lexicons: a statistical approach. In Computational Linguistics 22(1): 1-38.
Wermter Stefan, Joseph Chen 1997 Cautious steps towards hybrid connectionist bilingual phrase
alignment. In Proceedings of International Conference Recent Advances in Natural Language
Processing, Tzigov Chark, Bulgaria, pp 364-368.
Wu Dekai 1997 Stochastic inversion transduction grammars and bilingual parsing of parallel corpora.
In Computational Linguistics 23(3): 377-401.
Zhang Min, Li Sheng 1997 Tagging Chinese corpus using statistics techniques and rule techniques. In
Proceedings of 1997 International Conference on Computer Processing of Oriental Languages
(ICCPOL'97), Hong Kong, pp 503-506.
475
Appendix: Modified Zhang et al.'s Chinese tagset
No. Part-of-Speech
Classification (Eng)
Part-of-Speech
Classification (Chi)
Original
Tags
Modified
Tags
1 Punctuation mark w XX0
2 Idiom I IDM
3 Positional Noun s NNC
4 Pronoun r PN0
5 Verb v VV0
6 Directional f NND
7 Non-linguistic Code x ZZ0
8 Adverb d AV0
9 Suffix k SUF
10 Acronym j ACN
11 Preposition p PRP
12 Conjunction c CJ0
13 Classifier q NMW
14 Noun n NN0
15 Prefix h PRF
16 Determiner b PND
17 Temporal Noun t TIM
18 Numeral m MC
19 Exclamatory word e ITJ
20 Onomatopoeic word o OP0
21 Idiomatic Expression l FIX
22 Adjective a AJ0
23 Auxiliary of Mood y AS0
24 Morpheme g ELM
25 State word * U z AJ0
26 Auxiliary Word u AUX
Added POS Categories
27 Proper Noun NP0
28 Modal Verb VM
29 Attribute Marker de “ ” DE1
30 Adverbial Marker de “ ” DE2
31 Complement Marker de “ ” DE3
32 Auxiliary suo “ ” SUO

476
Syntactic change in Abidjanee French
Katja Ploog (ERSS a Bordeaux)
Abidjanee popular French has acquired some notoriety in the scientific world during the seventies.
But in spite of its very well-known sociolinguistic emergence context (a highly heterogeneous contact
in urban area, French becoming mother-tongue), we still do not know the factors that condition its
actual structures. Furthermore, the research about variation and specially about this non-standard
variety is generally not interested in syntactical considerations. In fact, its approach poses serious
methodological problems because of a high variability. The aim of my communication is to show how
it is possible to initiate a description of the syntactical change of Abidjanee French.
The analysis is based on about 20 hours of spontaneous speech produced by young Abidjanees ; the
corpus has been integrally retranscribed in a data-base of 10000 propositional units, systematically
annotated (morphologic form and syntactic constituents) and analysed. The description is based on
derivational framework, which conceives the structure elaboration from a deep predicative structure.
After a short inventory of the structure types gathered (concerning null-arguments and co-occurrence
of clitics) my work sheds some light on the internal articulation oft the Abidjanee system by focussing
on the pragmatic elaboration of the Abidjanee verb and the structure of its actants.
My communication tackles in details the structural hypothesis of change. We notice that some of the
reorganizations (for example the limiting of the preverbal clitic constituents) also exist in other
spoken varieties of French and in French Creole languages. Even if its quite impossible to prove this
hypothesis, in this case we can imagine that the reasons of change are inherent to the goal language.
Some other structural particularities may derive from the various linguistic or cultural substrata,
specially the subcategorisation of nouns, which is very close to West African languages. Furthermore,
it may be asked whether a part of the change is due to universal cognitive and 1 or general discursive
constraints, like the reanalysis of subject-verb agreement.

561
HPSG-based syntactic treebank of Bulgarian
(BulTreeBank)
Kiril Simov, Gergana Popova1, Petya Osenova2
The BulTreeBank Project
Linguistic Modelling Laboratory - CLPPI, Bulgarian Academy of Sciences
Acad. G.Bonchev Str. 25A, 1113 Sofia, Bulgaria
Tel: (+3592) 979 28 25, Fax: (+3592) 70 72 73
kivs@bgcict.acad.bg, gdpopo@essex.ac.uk, osenova@slav.uni-sofia.bg
Our paper will be mostly dedicated to a project about to start at the LML, BAS. Its main objective is to
create a high quality set of syntactic structures of Bulgarian sentences within the framework of HPSG.
We will discuss the methodology of the project with emphasis on the new aspects of the adopted
approach, as well as its expected results and their applications.
Methodology. An annotation scheme usually has to be theory-independent in order to allow different
interpretations of the tagged texts in different linguistic frameworks. We think, however, that on a
certain level of granularity (and linguistic descriptions in the BulTreeBank will be very detailed in
order to demonstrate the information flow in the syntactic structure) we will have to exploit some
linguistic descriptions that are theory dependent. We choose HPSG for the following reasons: (1)
HPSG is one of the major linguistic theories based on rigorous formal grounds; (2) HPSG allows for a
consistent description of linguistic facts on every linguistic level: phonetic and phonological,
morphological, syntactic, even the level of discourse. Thus, it will ensure the easy incorporation of
linguistic information which does not belong to the level of syntax if such is needed for the correct
analysis of a given phenomenon; (3) HPSG allows for both integration and modularisation of
descriptions and will therefore enable different experts to work on different parts or levels of analysis.
(4) The formal basis of HPSG allows easy translation to other formalisms. We not only choose HPSG
to be the linguistic theory within which we will explicate the syntactic structures, but make a step
further and choose the actual logical formalism that we will use in the annotation process: namely, SRL
for HPSG. For the annotation we will use descriptions called feature graphs. Such detailed descriptions
will be extremely useful in the future exploitation of the Tree-Bank, but they might be difficult to use
in the annotation process. Here we hope to use the (special) inference mechanisms of the logic and
some of the HPSG principles in order to allow the annotator to provide only part of the needed
information with the rest of it being inferred automatically. In order to minimise the necessary human
intervention, we will exploit all possibilities to provide an automatic partial analysis of the input string
before the actual annotation starts. We would also use the partial information entered by the annotator
in order to predict or constrain the possible analyses in other parts of the whole description of the
element. In this way we will exploit all the constraints available from pre-encoded grammars.
Expected results. At the end of the project we expect to have a set of Bulgarian sentences marked-up
with detailed syntactic information. These sentences will be mainly extracted from authentic Bulgarian
texts. They will be chosen with two criteria in mind. First, they will have to cover the variety of
syntactic structures of Bulgarian. Second, they should reflect the statistical distribution of these
phenomena in real texts. A core set of sentences will be extracted to serve as a test-suite for software
applications incorporating syntactic processing Bulgarian texts. The project should result also in a
reliable partial grammar for automatic parsing of phrases in Bulgarian. This grammar will be
extensively tested and used during the creation of the TreeBank. It will be used as a module separate
from the TreeBank in tasks which require only partial parsing of natural language texts such as
information retrieval, information extraction, data mining from texts and etc. Work on the TreeBank
will require the creation of software modules for compiling, manipulating and exploring the data. This
software will support both the creation of the TreeBank, and its use for different purposes such as
automatic extraction of grammars for Bulgarian.
1 PhD student, Department of Language and Linguistics, University of Essex.
2 Also at the Bulgarian Language Division, Faculty of Slavonic Languages, St. Kl. Ohridsky
University, Sofia, Bulgaria

477
Linguistic expressions as a tool to extract information1
Sylvie Porhiel – LaTTICe, LIMSI
When searching for multiple words in a corpus database, we all have encountered the problem of
not coming up with ‘the right thing'. Indeed, very often the results produced by the computer do not
meet the analysis we would have produced as humans. In a text, in order to understand it correctly, we
rely not only on the semantic content of the words but also on more grammatical means, which provide
reading instructions. Those are the ones we will be dealing with here and we will use French
prepositions (pour ce qui est de, en ce qui concerne, a propos de, au sujet de, etc.) which have the
syntactic properties of being either detached and generally in an initial position or, of being dependent
on another morphosyntactic constituent. This syntactic behaviour is of importance in the way in which
information is processed and corresponds to two discoursal functions. When detached, these
prepositions introduce a thematic (Charolles 1997) and mark an author's pragmatic intentions. When
introducing a thematic they can integrate one or more propositions. When they depend on another
constituent they simply focalise and do not have that integrative property. The ultimate aim of this
ongoing analysis is to engineer a tool using linguistic expressions to extract thematic information.
The purpose of this paper is three fold: to detail how linguistic information concerning the
thematic introducers is captured; to show how they help in segmenting texts; and to give a technical
overview of the system currently being developed by the LaLIC team, utilising this linguistic
information.
1. Capture of linguistic knowledge about thematic introducers
This section analyses the linguistic markers and captures the linguistic information related to
them. The data collection goes through different steps.
1.1. Description of linguistic markers
The first step consists of describing the linguistic markers identified as potential thematic
introducers (from now on T). Four criteria have to be taken into account:
·  Morphological variations of the linguistic pattern which can be broken into:
-  number variations for the few that undertake them: au chapitre (de), aux chapitres (de), sur
le chapitre (de), sur les chapitres (de) ‘in the matter of';
-  aspectual variations: among those, only the variation used are declared, i.e.: en ce qui
concerne ‘as far as... is concerned', en ce qui concernait, en ce qui concernera but not *en
ce qui a concerné.2
·  Case variation: all T can start a sentence and consequently start with an upper case but can also be
after a word or a group of words, a comma or a semicolon, etc., in which case they start with a
lower case.
·  The unbroken or broken structure of the T: written as follows a propos de, the constituents of
which are only separated by spaces is an unbroken linguistic pattern, whereas a propos+de, the
constituents of which are linked by the metacharacter ‘+’ is a broken linguistic pattern which
matches: a propos notamment de, a propos tout particulierement de, etc. The elements which fit
between the basic constituents of the T are called ‘insertions'. An analysis of the linguistic markers
shows that there is before the base a single point of insertion in which case it is an adjective, and
after the base there are two points of insertion, an adverb (a propos de) or an adjective and an
adverb (sur le sujet de, sur le chapitre de). Pour ce qui est de accepts insertions such as: pour ce
qui est bien sur de X ‘as far as of course X is concerned', pour ce qui est d'abord de X ‘as far as
first X is concerned', pour ce qui est par exemple de X ‘as far as for example X is concerned'.
1 This research takes place within the project ‘Modele d'exploration sémantique de textes guidé par les
points de vue du lecteur’ under the direction of G. Sabah. The LIMSI, the LATTICE, the CEA and the
LaLIC of the CAMS contribute to it. I am indebted to J.L Minel and T. Samuels who reviewed this
paper.
2 Unlike the English as far as sth is/are concerned, en ce qui concerne does not undertake number
variation.
478
· Paradigmatic variations, which concern the prepositions before and after the base and the
determiners before the noun-base. Firstly, when chapitre is a constituent of a T it can be preceded
by two prepositions a or sur: au chapitre, sur le chapitre; and concerne by either en, pour or pour
tout: en ce qui concerne, pour (tout) ce qui concerne. Indeed, the more paradigmatic variations
possible, the more linguistic patterns will have to be captured. Pour ce qui est de varying
aspectually and paradigmatically, we currently have: pour/pour tout + ce qui + est/était/sera. Only
a few T present preposition and determiner variations after the base: en ce qui touche (a) is an
example of this. Secondly, determiners can vary and be definite, indefinite, and demonstrative.
This raises two questions: a) should we consider all determiners for each T and b) should we mix
the demonstratives with the others? Because of insertions, the definite and the indefinite article
have three forms, respectively le3, l', les and un, des, d': sur les sujets délicats de X et Y; sur des
sujets aussi délicats que ceux de X et de Y, sur un sujet aussi délicat que celui de Y, etc. As for
demonstratives such as in sur ce chapitre and sur ce chapitre du chômage, they play a resumptive
role in texts. This particular role is of importance in text segmentation, which explains why
demonstratives are not part of the paradigm of the determiners.
1.2. Pointing towards the correct prepositional phrases
Software can be said to have located the proper prepositional phrase, i.e. a T, when the located
phrase possesses the following characteristics: it is syntactically initially placed and prefixed; it sorts
information and places it in ‘boxes'; and it integrates one or more propositions, even a whole paragraph
(Charolles 1997). This section deals with the recognition of the proper prepositional phrases and gives
four scenarios which highlight the difficulties of locating the correct ones. The first three occur within
the syntactic boundaries of the sentence and the fourth one exceeds this boundary.
First, a system might encounter difficulties as far as syntactic segmentation is concerned. In (1)
and (2), the phrase a propos de is placed initially and prefixed, meeting the syntactic properties of T:
(1) A propos de la grenouille, qui va s'en occuper ? ‘Concerning the frog, who is going to look after it ?’
(2) A propos, de la rouge ou de la bleue, laquelle préferes-tu ? ‘By the way, between the red or the blue, which is your
favourite ?’
Only the phrase in (1) is a T and the difference between the two phrases lies in the presence of a
comma after propos. To be extracted the T have to be described correctly so that it accepts insertions
but no comma.
Second, the system might have to overcome lexical ambiguities resulting from the polysemy of
the base of the T. Let's consider (3)-(5):
(3) Au chapitre des insectes, il est incollable ‘On the matter of insects, he is impossible to catch out’
(4) Au chapitre des insectes, on sent que le scientifique est vraiment dans son élément. ‘In the chapter about insects,
you feel that the scientist is in his element’
(5) Au chapitre des dépenses, les institutions répertorient (...) ‘In the section on expenditure, the institutions list (...)’
All au chapitre de are syntactically detached, followed by the same plural determiner and a noun,
though only (3) contains a T. In French, chapitre has several meanings which happen to combine in the
same syntactic environment and to be positioned syntactically alike. Nonetheless, their semantic
compatibilities differ: chapitre when referring to a book can be followed by the topic and the author;
chapitre, related to a budget is also followed by specific nouns (recette, dépenses, etc.) and may also
have the name of an administration. As a result, to optimise the chance for a system to locate the right
phrase, au chapitre de must not be found in combination with an author's name or a specific noun.
Third, the positional characteristic has to be reviewed and the hypothesis of the initial position
rephrased. In (6), though not initially, a propos de and au chapitre de are T as (1) and (3):
(6) Dis donc, a propos de la grenouille, qui va s'en occuper ? ‘Hey, concerning the frog, who is going to look after
it ?’
(7) Pourtant, au chapitre (des) insectes, il est incollable. ‘And yet, on the matter of insects, he is impossible to catch
out’
Indeed, in a text, it is fairly common for T to be preceded by 1 to 3 groups of elements such as: spatial
expressions, temporal expressions, articulative conjuncts, etc. Currently 24 groups which combine in
different ways have been identified: Par exemple, au chapitre de X (...), Comme, par exemple, en ce qui
concerne X (...), Ainsi, par exemple, en matiere de X (...), Il en va ainsi, par exemple, pour ce qui
touche a X (...).
Lastly, the fourth difficulty encountered by a system lies in the recognition of thematic utterances
from truncated ones, when the sentence prototypically consists of the marker plus its complement. Two
cases can be considered: first, the T is placed initially; second, it appears after another group of
elements. (8)-(9) illustrate the configuration in which the T, strictly followed by their complements, are
placed initially or are preceded by other elements (10):
3 All the noun base are masculine.
479
(8) Il me semble qu'il m'a raconté une anecdote. A propos de Staline. ‘It seems to me that he told an anecdote. About
Stalin.’
(9) [§] A propos de démocratie: Jabotinski se définissait comme un libéral et défendait avec fermeté le systeme
parlementaire. (...). ‘About democracy: Jabotinski defined himself as a liberal and battled in favour of a
parliamentary system. (...)’
(10) J'aurais besoin de votre aide demain. Notamment en ce qui concerne cette note de service. ‘I will require your help
tomorrow. ‘Particularly as far as this memo is concerned.’
Generally speaking, the prepositional phrase in truncated sentences and thematic ones share
characteristics: it has no verb, it is not followed by a proposition, it can be preceded by a group of
elements and ends either with [.], [:] or [...]. It also has differences:
-  the intonation goes down in truncated sentences (8) (10) and rises in thematic ones (9);
-  the referent is not present in the interdiscourse in truncated sentences whereas it is in thematic ones
(9);
-  they tend to occupy different syntactic positions: in the middle of a paragraph with truncated
sentences (8) (10), at the beginning of one with thematic sentences (9);
-  truncated sentences require the help of the linguistic context and depend on a morphosyntactic
constituent in the preceding sentence (8) (10), and can introduce an answer; whereas thematic ones
do not.
-  some groups of elements as well as punctuation tend to favour the interpretation of sentences as
truncated or thematic ones: interjections favour a thematic reading, selective markers a truncated
one (10); the configuration in which a T is followed by [...] favours a thematic reading: (Dis donc),
a propos de femme... ‘(Hey) concerning women...’
-  truncated sentences do not have an integrative property, thematic sentences do.
2. Thematic introducers and text segmentation
This section is about the segmentation of texts in large corpora (Le Monde Diplomatique,
Frantext) and deals with frame openers and closers; examples are used to illustrate these.
2.1. T as frame openers and text segmenters
To use T as signals for text segmentation, it is necessary to consider their place in the text and
their combination with other markers found in the cotext.
As with time and locative adverbials (Virtanen 1992), a series of initial T can either appear within
paragraphs (11) or start paragraphs (12). As such, they create text strategy continuity (Virtanen 1992):
(11) [§]4 Marie s'est résolue a quitter sa maison et a aller vivre en appartement. Pour ce qui est de ses meubles, elle laisse
son fauteuil a bascule a sa femme de ménage. Le secrétaire sera pour Ludovic. La salle a manger revient a sa niece.
Enfin, elle gardera la télé. Concernant sa voiture, elle la donne a son petit fils.(...), etc. Quant a son chien, le voisin s'en
occupera. ‘Marie brought herself to leave her house and to move into an apartment. As regards her furniture, she leaves
her rocking chair to her cleaner. The secretaire is for Ludovic. The dining room suite goes to her niece. Lastly, she will
keep the TV. As far as her car is concerned, she gives it to her grandchild (...) etc. As for his dog, the neighbour will take
care of him.’
(12) [...] Cette situation a des conséquences décisives sur trois variables: X, Y et Z. ‘This situation has decisive
consequences upon three variables: X, Y and Z:
[§] En ce qui concerne X, (...) ‘As far as X is concerned, (...)’
[§] Quant a Y, (...) ‘Regarding Y (...)’
[§] Enfin, pour ce qui est de Z, (...) ‘Lastly, concerning Z (...)’
On the one hand, (11) starts with a sentence, which does not tell how the information will be
broken down: the referents are implicit. As each referent is introduced by a T, pour ce qui est de,
concernant, quant a, a frame opens while closing the previous one. The last T, quant a, signals the last
item in the list. The frame instantiated by pour ce qui est de exemplifies the integrative property of T:
the referent meubles ‘furniture’ is a hyperonym encompassing the different pieces of furniture listed.
Enfin ‘lastly’ heralds firstly the end of the furniture list, and secondly the end of the scope of the T. (12)
provides an example of a derived thematic progression: the paragraph finishes with an introducing
sentence explicitly specifying that three variables X, Y and Z, will be reused and developed in the
following paragraphs. Each three T, en ce qui concerne, quant a, pour ce qui est de, prefixed and at the
beginning of a paragraph or following a connector (enfin), signals a shift from one variable to the next.
The paragraph structure is of importance as the idented line tells the reader that s/he has just dealt with
a sense unit and that s/he is going to start on a new one. However, if the notion is to be of use as a
segmentation indicator, it does not have to be overstressed: first because typographical paragraphs and
thematic ones do not always coincide and second, because stylistic and balancing parameters enter the
matter (Bessonnat 1988, Virtanen 1992). In both these examples T open a series of thematic frames
which are examples of thematic strategic continuity. But each example is different as to the level of
4 [§] indicates the beginning of a paragraph.
480
structuration. In (11) a structuration indicates a shift from one particular point to another one within the
paragraph in accordance with the first sentence; and in (12) the structuration is running in more than
one paragraph5, indicating a thematic break each time.
As different groups of elements are likely to precede T, it will be necessary to find out which ones
play a part in the structuration of text. Here, the role of enfin ‘lastly’ has to be outlined. In (11), it
signals the end of a list as well as the end of the integration under pour ce qui est de. In (12), it signals,
with redundancy, the shift into the last variable; it also opens the last frame with the T pour ce qui est
de. The scope of a connector such as enfin is different when within a paragraph or placed initially in a
paragraph (Bessonnat 1988).
To conclude, T are linguistic features which open new frames and consequently close the already
instantiated one; they mark textual boundaries; they segment texts on different levels (within a
paragraph with a linear thematic succession or across paragraphs with a hierarchical thematic one);
they can be preceded by groups of elements, the place of which brings scope variation; and in addition
the place of T indicates a textual organisation on the part of the text producers, which makes them
reliable indicators for text summarisation.
2.2. Frame closers
The opening boundary is relatively easy to identify as it overlaps with the presence of the T in a
text. Things are different, though, as far as the closing boundary is concerned. Indeed, even if a T is at
the beginning of a paragraph, it does not necessarily means that it integrates all the propositions of that
paragraph. Markers likely to induce the closing of a thematic frame are for example: a) T themselves
among which some tend to indicate the end of a list, quant a (11) (12), b) logical connectors (11) (13)
(16), c) spatial and temporal adverbials (14), resumptive anaphors (15) (17a), discourse markers (17c),
sentence adjuncts (17b), aspectual changes, typographical means, etc.
(13) [...] Cette situation a des conséquences décisives sur trois variables: X, Y et Z. ‘This situation has decisive
consequences upon three variables: X, Y and Z:
[§] En ce qui concerne X, (...) ‘As far as X is concerned, (...)’
[§] Quant a Y, (...) ‘Regarding Y (...)’
[§] Enfin, Z, (...) ‘Lastly, Z (...)
(14) [§] En ce qui concerne l'allocation des ressources, en Côte d'Ivoire, l'excédent (...). En Corée du Sud (...) [§] ‘[§]
As far as the resource allowance is concerned, in Ivory Coast, the excess (...). In South Korea (...) [§]’
(15) [..] En ce qui concerne les déchets toxiques, les autorités affirment (...). Mais cet argument ne tient pas (...) ‘[§] As
far as toxic waste is concerned, authorities claim (...) But such an argument does not hold (...)’
(16) A propos des sous-marins, les auteurs (...) A leurs yeux, la marine (...) ‘Concerning submarines, the authors (...)
According to them, the Navy (...)’
(17) [§] S'agissant de la coopération, le président s'est félicité (...) ‘As regards co-operation, the president was pleased
(...)’
a) Cette co-operation (...) ‘This co-operation (...)’
b) Personnellement, nous aurions (...) ‘Personally, we would have (...)’
c) Mais revenons a X (...) ‘But to return to my point (...)’
These frame closers, all text segmenters interacting with one another and taking part in discourse
management, have a double property: while indicating a break they also signal textual continuity, then
coherence. In (13) the connector enfin, not followed by a T, closes the preceding frame and opens the
last one in the derived thematic progression. (14) illustrates the case when more than one frame appears
at the beginning of a sentence, being subordinate to another. In this instance, en ce qui concerne
l'allocation des ressources gives the general topic of the paragraph while the locative adverbials, en
Côte d'Ivoire and en Corée du Sud, are subordinate and deal with particular points related to the topic.
In (15)-(17) the presence of the writer is noticeable through resumptive anaphors preceded (or not) by a
logical connector (15) (17a), sentence adjuncts indicating an attitude (17b), discourse markers (17c)
which signal the reintroduction of a previous topic of the discourse.
There are also cases where no frame closers appear on the textual surface. The linguistic tools
which have been listed so far are then of little use. A solution can be found in looking at changes in the
text vocabulary. Such experiments have been conducted by Ferret, Grau and Masson (1998). They
worked out two methods, the outcome of which varies according to the type of texts. Ideally, the
combination of properly recognised linguistic tools (T and connectors) and a statistic method should
optimise the results produced in the segmentation of text, as well as improve the thematic coherence of
the summary.
3. Technical overview
This section describes the architecture of the ContextO software and gives a technical overview of
how texts are processed.
5 Unless with spatial or temporal adverbials it is rare to find a text in which T run from the beginning
till the end.
481
3.1. The architecture
The ContextO software runs under the Filtext platform (figure 1) (Minel et al., forthcoming) both
of which have been developed by the LaLIC team in Paris IV. It identifies specific semantic
information in texts and extracts relevant sentences meeting applied criteria. To achieve this, the
system uses linguistic data declared in a database and contextual rules written according to the
contextual exploration method (Desclés 1997). In ContextO, the database is used to store relevant
linguistic data in order to locate T. This linguistic knowledge is grouped into classes and individual
classes can make compound classes in order to better describe markers. Here are examples of classes
and sets of items belonging to them :
&marqueurs_liaison = {cependant, Cependant, enfin, Enfin}
&prep_de = {de, d', du, des}
&ponctuation = {,}
&a_propos_min = {a propos, en fait}
&introducteurs_thématiques = {&avant_motbase_pdg_en_maj+ce qui +&vb_etre,
&avant_motbase_pdg_en_maj+ce qui +&vb_concerne}6
Contextual rules are contained in Java methods and use the class information to process texts. The
notion of context is a contextual one : a context is determined by indicators (here T) and constituted of
sentences not necessarily adjacent. It is in that delimited space that constraints are applied. For example
to locate the T a propos de et en ce qui concerne in a text, the rules R1 and R2 have to be written :
R1 extracts sentences in which no comma appears after a propos but still allows for insertions
between a propos and de:
R1: Items belonging to &a_propos_maj = T
IF placed initially
AND IF &a_propos_maj is not followed by an item of &ponctuation
AND IF &prep_de appears within 5 words
R2 extracts sentences in which &introducteurs_thématiques are preceded by enfin, cependant
R2: Items belonging to &introducteurs_thématiques = a T
IF in the left context there is an item belonging to &marqueurs_liaison
T ex tu a l u n its
T ask
C lu es ,
C o n te x tu a l R u le s
C o n tex tu a l
ex p lo ra t io n
e n g in e
S p e c ia lis ed
a g en t
S p e c ia lis ed
a g en t
S p e c ia lis ed
a g e n t
L in g u istic
k n ow le d g e
d a ta b a se
Figure 1. FilText Architecture
3.2. Processing overview
The following sentences used as a corpus will show how a text is actually processed by the
ContextO software when the specific task ‘recognition of thematic frames’ has been chosen: A propos
de la grenouille, qui va s'en occuper ? , A propos, de la rouge ou de la bleue, laquelle préferes-tu ?, A
propos notamment des chemins de fer, l'État a décidé (...), Cependant, en ce qui concernait ce
probleme particulier, on aurait pu (...). The text is first segmented into its component sentences. Each
sentence is then analysed by applying the individual rules. The results of applying our example rules
R1 and R2 to the corpus will be as followed:
6 This implies that the mentioned classes as well as the items pertaining to them have been declared
beforehand: &vb_concerne = {concerne, concernait, concernera}, &vb_etre : {est, était, sera},
&avant_motbase_pdg_En_maj = {En, Pour, Pour tout}. Only the relevant tenses of the verbs are
declared
482
Sentences R1 R2
A propos de la grenouille, qui va s'en occuper ? True False
A propos, de la rouge ou de la bleue, laquelle préferes-tu ? False False
A propos notamment des chemins de fer, l'État a décidé (...) True False
Cependant, en ce qui concernait ce probleme particulier, on aurait pu (...) False True
Sentences where either rules R1 or R2 are true will receive a ‘frame’ tag and thus be extracted.
The approach presented is based on linguistic knowledge in order to analyse texts and more
specifically to automatically segment texts without relying on a field of application. We first detailed
the analytical steps required for a system to identify T and not a phrase of matching characters. The
second part, focused on frame openers and closers. T can be used to open text segments which can be
closed, for example, by other linguistic tools. As linguistic tools are not always available on the textual
surface it is suggested that combining the use of T and statistic methods would lead to more promising
results and should improve the thematic coherence of summarised texts. The last part gave a technical
overview of the implementation of the captured linguistic knowledge in the ContextO software
developed by the LaLIC team. The next step of this research will consist in increasing the list of T, in
refining the contextual rules and in testing these rules in full texts.
References
Bessonnat D 1988 Le découpage en paragraphes et ses fonctions. Pratiques 57: 81-105.
Charolles M 1997 L'encadrement du discours - Univers, champs, domaines et espace. Cahier de
recherche linguistique 6.
Desclés JP 1997 Systeme d'exploration contextuelle. Co-texte et calcul du sens, Presses Universitaires
de Caen, pp. 215-232.
Ferret O, Grau B, Masson N 1999 Thematic segmentation of texts: two methods for two kinds of texts.
In Proceedings of the ALC-COLING'98, Montréal, pp. 392-396.
Minel JL, Descles J.P. 2000 Résumé automatique et filtrage des textes. In Pierrel J.M. (ed), Ingénierie
des langues. Paris, Hermes, pp. 253-270.
Minel et al. (forthcoming) Résumé automatique par filtrage sémantique d'informations dans des textes.
Présentation de la plate-forme FilText. Technique et science informatique 3.
Virtanen T. 1992 Discourse Functions of Adverbial Placement in English. Abo, Abo Akademi
University Press.

483
Theoretical Issues for Corpus Linguistics Raised by the Study of Ancient
Languages
Stanley E. Porter and Matthew Brook O'Donnell
University of Surrey Roehampton and OpenText.org
Corpus linguistics has focused largely upon the analysis of modern languages, usefully compiling large
corpora for linguistic analysis. In the course of our study of ancient Greek from a corpus-based
perspective, a number of issues regarding corpus linguistics have come to the fore. In this paper we
wish to highlight and discuss three of these issues.
(1) Corpus size and compilation criteria The potential size of a corpus of a modern language, such as
English, is virtually infinite, limited perhaps only by storage capacity and compiler effort. These
limitations are less and less significant due to developments in technology and automated corpus
building. However, the situation is very different for the study of a language such as ancient Greek,
where the corpus size has been limited by historical accident. The number of texts is restricted to those
that have for some reason been preserved from the ancient world. Early studies in corpus linguistics,
faced with technological limitations, were forced to address issues of representativeness and sampling,
similar to those facing compilers of a corpus of ancient texts. More recent corpus studies have tended to
emphasize the size of the corpus over the contours of its composition. However, significant linguistic
results, such as those from Svartvik's study of the English voice system, can result from a carefully
compiled corpus of limited size.
(2) Annotation and levels of analysis In the light of the limited size of the available corpus of ancient
Greek, it is necessary to include greater detail and levels of linguistic annotation in the corpus than is
common in large modern language corpora. A limited corpus demands that as much information as
possible be garnered from the available data. This involves a much more detailed analysis than the
general lexical and morphological patterning that takes place in large-scale corpus studies. In order to
facilitate this study, we have had to develop a consistent system of annotations of higher linguistic
levels. This has not only generated much useful data, but forced us to examine a number of issues
previously unaddressed by corpus annotators.
(3) Analysis of texts One of the strengths of corpus linguistics has been the observation of linguistic
patterns across large samples of text. In contrast, the textual orientation of Classical and New
Testament scholarship requires close attention to the linguistic features and function of individual
documents as well as overall patterns in the language. This requires sensitivity to and the analysis of
contextual and co-textual features and patterns. This micro-analysis is more suitable to this kind of
limited corpus, and has provided useful data for studying various text-types and created new
possibilities for the use of discourse analysis in corpus-based studies.

UNIVERSITY CENTRE FOR COMPUTER
CORPUS RESEARCH ON LANGUAGE
Technical Papers
Volume 13 - Special issue.
Proceedings of the Corpus Linguistics 2001
conference
edited by
Paul Rayson,
Andrew Wilson,
Tony McEnery,
Andrew Hardie
and
Shereen Khoja.
ISBN 1 86220 107 2.
Lancaster University (UK), 29 March - 2 April 2001
ii
Table of contents
Preface vi
Bas Aarts, Evelien Keizer, Mariangela Spinillo & Sean Wallis: Which or what? A study of
interrogative determiners in present-day English
1
Anne Abeillé, Lionel Clément, Alexandra Kinyon & François Toussenel: The TALANA annotated
corpus for French : some experimental results
2
Annelie Ädel: On the search for metadiscourse units 3
Karin Aijmer: Discourse particles in contrast 13
Takanobu Akiyama: John is a man of (good) vision: enrichment with evaluative meanings 14
Jean-Yves Antoine & Jérôme Goulian: Word order variations and spoken man-machine dialogue in
French : a corpus analysis on the ATIS domain
22
Dawn Archer and Jonathan Culpeper: Sociopragmatic annotation: New directions and possibilities
in Historical Corpus Linguistics
30
Eric Atwell & John Elliott: A Corpus for Interstellar Communication 31
Manuel Barbera: From EAGLES to CT tagging: a case for re-usability of resources 40
Margareta Westergren Axelsson & Angela Hahn: The use of the progressive in Swedish and
German advanced learner English: a corpus-based study
45
Anja Belz: Optimisation of corpus-derived probabilistic grammars 46
Ylva Berglund & Oliver Mason: "But this formula doesn't mean anything!" 58
Roumiana Blagoeva: Comparing cohesive devices: a corpus based analysis of conjunctions in writen
and spoken learner discourse
59
Hans Boas: Frame Semantics as a framework for describing polysemy and syntactic structures of
English and German motion verbs in contrastive computational lexicography 64
Rhonwen Bowen: Nouns and Their Prepositional Phrase Complements in English 74
Ted Briscoe: From dictionary to corpus to self organising dictionary: Learning valency associations in
the face of variation and change
79
Estelle Campione & Jean Véronis: Semi-automatic tagging of intonation in French spoken corpora 90
Pascual Cantos-Gomez: An attempt to improve current collocation analysis 100
Roldano Cattoni, Morena Danieli, Andrea Panizza, Vanessa Sandrini & Claudia Soria: Building
a corpus of annotated dialogues: the ADAM experience
109
Frantisek Cermák, Jana Klimová, Karel Pala, Vladimir Petkevic: The Design of Czech Lexical
Database
119
Ngoni Chipere, David Malvern, Brian Richards & Pilar Duran: Using a corpus of school children's
writing to investigate the development of vocabulary diversity
126
Claudia Claridge: Approaching Irony in Corpora 134
Niladri Sekhar Dash and Bidyut Baran Chaudhuri: Corpus based Empirical Analysis of Form,
Function and Frequency of Characters used in Bangla
144
Liesbeth Degand & Henk Pander Maat: Contrasting causal connectives on the Speaker Involvement
Scale
158
Anne Le Draoulec & Marie-Paule Péry-Woodley: Corpus based identification of temporal
organisation in discourse
159
Stefan Evert & Anke Lüdeling: Measuring morphological productivity: Is automatic preprocessing
sufficient?
167
iii
Cécile Fabre & Didier Bourigault: Linguistic clues for corpus-based acquisition of lexical
dependencies
176
Richard Foley: Going out in style? Shall in EU legal English 185
Anna-Lena Fredriksson: Translating passives in English and Swedish: a text-linguistic perspective 196
Cécile Frérot, Géraldine Rigou & Annik Lacombe: Phraseological approach to automatic
terminology extraction from a bilingual aligned scientific corpus 204
Robert Gaizauskas, Jonathan Foster, Yorick Wilks, John Arundel, Paul Clough, Scott Piao: The
METER Corpus: A corpus for analysing journalistic text reuse
214
Hatem Ghorbel, Afzal Ballim, & Giovanni Coray: ROSETTA: Rhetorical and semantic
environment for text alignment
224
Solveig Granath: Is that a fact? A corpus study of the syntax and semantics of the fact that 234
Benoît Habert, Natalia Grabar, Pierre Jacquemart, Pierre Zweigenbaum: Building a text corpus
for representing the variety of medical language
245
Eva Hajicova & Petr Sgall: A reusable corpus needs syntactic annotations: Prague Dependency
Treebank
255
David Hardcastle: Using the BNC to produce dialectic cryptic crossword clues 256
Daniel Hardt: Comma Checking in Danish 266
Raymond Hickey: Tracking lexical change in present-day English 272
Diana Hudson-Ettle, Tore Nilsson & Sabine Reich: Orality and noun phrase complexity: a corpusbased
study of British and Kenyan writing in English
273
Nancy Ide & Catherine Macleod: The American National Corpus: A Standardized Resource for
American English
274
Reiko Ikeo: The Positions of the Reporting Clauses of Speech Presentation with Special Reference to
the Lancaster Speech, Thought and Writing Presentation Corpus
281
Wang JianDe, Chen ZhaoXiong & Huang HeYan: Multiple-Level Knowledge Discovery from
Corpus
289
Yu Jiangsheng & Duan Huiming: POS Estimation of Undefined Chinese Words 290
Steven Jones: Corpus Approaches to Antonymy 297
Beom-mo Kang & Hung-gyu Kim: Variation across Korean Text Registers 311
Przemyslaw Kaszubski: Tracing idiomaticity in learner language - the case of BE. 312
Yuliya Katsnelson & Charles Nicholas: Identifying Parallel Corpora Using Latent Semantic Indexing 323
Hannah Kermes & Stefan Evert: Exploiting large corpora: A circular process of partial syntactic
analysis, corpus query and extraction of lexicographic information
332
Shereen Khoja, Roger Garside & Gerry Knowles: A Tagset for the Morphosyntactic Tagging of
Arabic
341
Adam Kilgarriff: Web as corpus 342
Dimitrios Kokkinakis: A Long-Standing Problem in Corpus-Based Lexicography and a Proposal for a
Viable Solution
345
Julia Lavid: Using bilingual corpora for the construction of contrastive generation grammars: issues
and problems
356
Maarten Lemmens: Tracing referent location in oral picture descriptions 367
Barbara Lewandowska-Tomaszczyk, Michael Oakes & Paul Rayson: Annotated Corpora for
Assistance with English-Polish Translation
368
Juana Marín-Arrese, Elena Martinez-Caro & Soledad Pérez de Ayala Becerril: A corpus study of
impersonalization strategies in newspaper discourse in English & Spanish
369
iv
Mikhail Mikhailov & Miia Villikka: Is there such a thing as a translator's style? 378
Hermann Moisl & Joan Beal: Corpus analysis and results visualisation using self-organizing maps 386
Martina Möllering: Pragmatic and discursive aspects of German modal particles: a corpus-based
approach
392
Rachel Muntz: Evidence of Australian cultural identity through the analysis of Australian and British
corpora 393
P-O Nilsson: Investigating characteristic lexical distributions and grammatical patterning in Swedish
texts translated from English
400
Julien Nioche & Benoît Habert: Using feature structures as a unifying representation format for
corpora exploration
401
Matthew Brook O'Donnell, Stanley E. Porter & Jeffrey T. Reed: OpenText.org: the problems and
prospects of working with ancient discourse
413
Maeve Olohan: Spelling out the Optionals in Translation: A Corpus Study 423
Constantin Orasan: Patterns in scientific abstracts 433
Gabriella Kiss and Júlia Pajzs: An attempt to develop a lemmatiser for the Historical Corpus of
Hungarian
443
Byungsun Park & Beom-mo Kang: Korean grammatical collocation of predicates and arguments 452
Anselmo Penas, Felisa Verdejo & Julio Gonzalo: Corpus-Based Terminology Extraction Applied to
Information Access
458
Scott Songlin Piao & Tony McEnery: Multi-word unit alignment in English-Chinese parallel corpora 466
Katja Ploog: Syntactic change in abidjanee French 476
Sylvie Porhiel: Linguistic expressions as a tool to extract information 477
Stanley E. Porter & Matthew Brook O'Donnell: Theoretical Issues for Corpus Linguistics Raised by
the Study of Ancient Languages
483
Helena Raumolin-Brunberg: Temporal aspects of language change: what can we learn from the
CEEC?
484
Andrea Reményi: Use logbooks and find the original meaning of representativeness 485
Antoinette Renouf: The Web as a Source of Linguistic Information 492
Jérôme Richalot: The influence of the passive on text cohesion and technical terminology 493
Rema Rossini Favretti, Fabio Tamburini & Cristiana De Santis : CORIS/CODIS: A corpus of
written Italian based on a defined and a dynamic model
512
Hans-Jörg Schmid: Do women and men really live in different cultures? Evidence from the BNC 513
Josef Schmied: Exploring the Chemnitz Internet Grammar: Examples of student use 514
Mark Sebba & Susan Dray: Is it Creole, is it English, is it valid? Developing and using a corpus of
unstandardised written language
522
Mirjam Sepesy Maucec & Zdravko Kacic: Language Model Adaptation For Highly-Inflected
Slovenian Language In Comparison To English Language
523
Noëlle Serpollet: The mandative subjunctive in British English seems to be alive and kicking… Is this
due to the influence of American English?
531
Serge Sharoff: Through the looking glass of parallel texts 543
Kiril Simov, Zdravko Peev, Milen Kouylekov, Alexander Simov, Marin Dimitrov, Atanas
Kiryakov: CLaRK - an XML based system for corpora development
553
Kiril Simov, Gergana Popova & Petya Osenova: HPSG-based syntactic TreeBank of Bulgarian
(BulTreeBank)
561
Simon Smith & Martin Russell: Determining query types for information access 562
v
Josef Szakos & Amy Wang: Not last, even if least: Endangered Formosan aboriginal languages and
the corpus revolution
571
Elke Teich & Silvia Hansen: Methods and techniques for a multi-level analysis of multilingual
corpora
572
Dan Tufis & Ana-Maria Barbu: Accurate automatic extraction of translation equivalents from
parallel corpora 581
Tamás Váradi: The Linguistic Relevance of Corpus Linguistics 587
Serge Verlinde & Thierry Selva: Corpus-based vs intuition-based lexicography. Defining a word list
for a French learner's dictionary
594
Jean Véronis: Sense tagging: does it make sense ? 599
Adriana Vlad, Adrian Mitrea, & Mihai Mitrea: A Corpus-Based Analysis of How Accurately
Printed Romanian Obeys to Some Universal Laws
600
Martin Volk: Exploiting the WWW as a corpus to resolve PP attachment ambiguities 601
Martin Weisser: A corpus-based methodology for comparing and evaluating different accents 607
Anne Wichmann & Richard Cauldwell: Wh-Questions and attitude: the effect of context 614
Kay Wikberg: His breath a thin winter-whistle in his throat: English metaphors and their translation
into Scandinavian languages
615
Karen Wu Rongquan: Public Discourse as the mirror of ideological change: A keyword study of
editorials in People's Daily
616
Richard Xiao Zhonghua: A Corpus-Based Study of Interaction Between Chinese Perfective -le and
Situation Types
625
vi
Preface
All the papers in this collection are based upon talks or poster presentations given at
the Corpus Linguistics (CL2001) conference, held at Lancaster University between
29th March and 2nd April 2001 organised by members of UCREL, from the
Departments of Linguistics & Modern English Language and Computing.
The conference attracted over 100 participants from the language engineering and
corpus linguistics communities in over 20 countries world-wide. The presentations
represented a truly impressive rainbow of languages in corpus research, ranging from
ancient languages, eastern and western European languages, to Semitic and Asian
languages.
One of the aims of CL2001 was to celebrate the works of Geoffrey Leech who
reached 65 in 2001. Geoffrey, a pioneer in the construction and exploitation of
machine-readable corpora, has had considerable influence in many areas of linguistics
over the years. A selection of papers from the conference will appear in an edited
collection to be published in honour of Geoffrey Leech.
Paul Rayson
Andrew Wilson
Tony McEnery
Andrew Hardie
Shereen Khoja
Lancaster University, March 2001.

484
Temporal aspects of language change: what can we learn from the CEEC?
Helena Raumolin-Brunberg, University of Helsinki
My paper discusses the time course of a number of morphosyntactic changes in Renaissance English.
It is surprisingly seldom that linguistic changes have been attributed a more accurate timing than a
‘full’ period, such as Late Middle English or Early Modern English. My study will give a more
precise timing to the diffusion of the changes under scrutiny among the population of England. Both
the macro- and micro-level will be taken into account and changes presented as S-curves.
In addition to timing, the following questions will be raised on the macro-level. How should we define
the beginning and end of a change? At what rate do changes proceed? What are the factors that play a
role in the progression of a change?
On the micro-level, individual speakers are in focus. How do individuals behave in relation to ongoing
changes? Do they change their language during their lifetimes? If they do, where shall we look
for reasons?
The data are retrieved from the Corpus of Early English Correspondence (CEEC), compiled at the
University of Helsinki by the project ‘Sociolinguistics and Language History'. The CEEC contains
around 6000 letters from 1417-1681, forming a corpus of 2.7 million words especially designed for
studies in historical sociolinguistics.
The following changes will be dealt with: replacement of subject YE by YOU, third person singular
suffix -TH versus -S, loss of multiple negation, object of gerund constructions, possessives MINE and
THINE versus MY and THY, and introduction of possessive ITS.
References
Nevalainen, Terttu & Helena Raumolin-Brunberg (eds.) 1996. Sociolinguistics and Language History.
Studies Based on the Corpus of Early English Correspondence. Amsterdam/Atlanta GA:
Rodopi.
Nevalainen, Terttu & Helena Raumolin-Brunberg (in preparation). Historical Sociolinguistics.
Longman.

485
Use logbooks and find the original meaning of “representativeness”
Andrea Ágnes Reményi
Research Institute for Linguistics,
Hungarian Academy of Sciences
Budapest, Hungary
(remenyi@nytud.hu)
Debates about overall problems in general synchronic corpus design seem to have settled. Yet,
solutions whether and how to pre-structure one's target population and corpus are still based either on
practical considerations of comparability or on intuitive proportions, partly due to an exclusively
textual view of representativeness, and partly to the fact that designers deny the possibility to estimate
the relative distribution of texts among media, genres, registers, etc. of a language. In this paper first
general problems are tackled again: what do ‘representativeness', ‘balance’ and ‘influentialness’
mean? In the second part a relatively simple and cost-efficient method to estimate those distributions
is described, and results of a two-step pilot study are analysed. Finally I will suggest how both textual
and demographic representativeness can be controlled in a modular corpus.
General synchronic corpora wish to grasp the totality of a language in some sense. Electronic
corpora support reliable quantitative studies only if the sample selected from this totality represents
the totality as fully as possible. Leech (1991) states that “a corpus is ‘representative’ in the sense that
findings based on an analysis of it can be generalised to the language as a whole or a specified part of
it” (as cited by Kennedy 1998: 62). In my view a sample can be called ‘representative’ only if it aims
to reproduce the statistical variance of the population to the highest possible degree – in terms of the
distribution of text types and of linguistic features (cf. Biber 1993: 243), but not excluding
demographic factors.
1 Theoretical considerations
The requirement of representativeness poses a major difficulty in corpus design, as designers are
supposed to find valid bases to delimit the concept of “every text in the given language”, i.e. the target
population, on the one hand, and to find the most reliable sampling methods to select from that, on
the other.
1.1 Statistical sampling.
Branches of social science applying statistical sampling and inference most often employ either
random sampling or stratified random sampling methods. In the former case every member of the
target population has an equal chance to appear in the sample. When the latter method is applied,
researchers previously establish a few basic categories considered by the researchers/the research
community/the research paradigm to be structuring the target population in some essential way (for
example sex, age, schooling, location; medium, genre, domain, etc.), and random sampling is
achieved only within these ‘strata'.
1.2 Stratified random sampling.
When defining the target population of a general synchronic corpus, designers must also start by
deciding which sampling method to apply, that is, whether or not they should pre-structure the
population by a classification of text types or language users, a structure that is maintained in the
sample. Johansson (1980:26), for example, supports a textual stratification over simple random
sampling, stating that “the true ‘representativeness’ of the LOB Corpus arises from the deliberate
attempt to include relevant categories and subcategories of texts rather than blind statistical choice.
Random sampling simply ensured that, within the stated guidelines, the selection of individual texts
was free of the conscious or unconscious influence of personal taste or preference.”
Note, however, that this is not only a methodological, but also a conceptual problem, as ‘strata’
of the population are pre-defined by the researchers, necessarily reflecting what they consider to be
the most essential structuring factors of that population: textual, demographic or other factors.1
1 Apart from this, the most obvious advantage of stratified random sampling over simple random
sampling is that it requires a smaller sample.
486
1.3 What are the units of observation?
The basis of this meta-theoretical issue is the double nature of the units of observation in corpus
linguistics. While in sociology the units of observation are mostly unambiguous (individual human
beings), this is not so in corpus design. Should language users (text producers and receivers) or texts
(the products of language use) be chosen as the units of observation? While census or similar
statistical data about the totality of language users of a country are easily accessible in most countries,
no similar methods have been developed to estimate either the totality of the target population of texts
or the distribution of text types within it. Thus in text-based sampling the basis of the composition and
proportioning of ‘strata', i.e. text types, is statistically unjustifiable. To my knowledge, most
megacorpora (Brown, LOB, Cobuild, BNC, ICE, Longman, etc.) are structured according to medium,
register, genre, domain, discourse function and/or subject in a way that proportions of these text types
are determined by the practical consideration of balance, that is, by including comparable amounts of
texts within subcategories.
On the other hand, corpora organised by demographic proportions would not support the
criterion of ‘sample variability matching population variability’ as far as text types are concerned.
Biber is right in stating that “a corpus with this design might contain roughly 90% conversation and
3% letters and notes, with the remaining 7% divided among registers such as press reportage, popular
magazines, academic prose, fiction, lectures, news broadcasts, and unpublished writing. [...] Such a
corpus would permit summary descriptive statistics for the entire language represented by the corpus.
These kinds of generalisations, however, are typically not of interest for linguistic research” (1993:
247). Some linguistic research, however, e.g. sociolinguistics or L2-lexicography and textbook
methodology, may find interest in a demographically structured corpus.
To sum up, the problems of ‘representativeness’ are mostly due to the double nature of the unit
of observation in corpus design: either the diversity of language users, or that of text types is eclipsed.
1.4 A combination of the two types of units of observation.
Among the above mentioned corpora, the spoken component of the British National Corpus
(BNC) tried to solve this dilemma, pairing text-based and demographic sampling methods. In the
text-based (‘context-governed') part of the corpus both major and minor a priori categories were
proportioned to be balanced (four equal-sized contextually based categories were established, each
divided into 40 per cent monologue and 60 percent dialogue) (Burnard 1995: 23), consonant with the
proportioning of the written component. In the demographically sampled part of the BNC's spoken
component 124 individuals were recruited based on random location sampling, asked to carry a tape
recorder and to record all their conversations for 2-7 days. “Recruits were chosen in such a way as to
make sure there were equal numbers of men and women, approximately equal numbers from each age
group, and equal numbers from each social grouping.” (BNC Online 1997.) Note that stratified
sampling was performed only in terms of location, but not in terms of the other basic demographic
variables, e.g. sex, age, and socio-economic group,2 i.e. the proportions of age groups or socioeconomic
groups did not follow those in the UK population. Thus, while the BNC has a complex
structure of ‘strata', stratified sampling according to these basic demographic variables within none of
them is achieved.
The conversation subcorpus of the Longman Spoken and Written English Corpus was also
sampled on demographic bases: “a set of informants was identified to represent the range of English
speakers in the country (UK or USA) across age, sex, social group, and regional spread” (Biber et al.
1999: 29), but the informants may not have been randomly selected for that corpus, either.
1.5 Is it possible to estimate the distribution?
Designers usually dismiss the possibility to take the daily distribution of spoken and written
medium, domain, genre, register, discourse function or subject variation into account (e.g. Biber
1993:247, Burnard 1995: 20, Kennedy 1998: 63). There are direct methods developed to obtain the
production and indirect methods to estimate the reception of published written and internet texts
(book and periodical publication lists; best seller, library lending and periodical circulation statistics;
internet search engines and click-on per internet page measures, respectively). Direct quantitative
methods to assess the reception of published written texts, and the production and reception of
unpublished written texts have not been found, and neither are there objective measures to define the
2 Tamás Váradi directed my attention to this point.
487
target population of the spoken medium available. Kennedy writes: “No one knows what proportion of
the words produced in a language on any given day are spoken or written. Individually, speech makes
up a greater proportion than does writing of the language most of us receive or produce on a typical
day. However, […] a broadcast conversation on radio or television will reach many more ears than a
commercial encounter involving just a customer and a salesperson. Within a written corpus, balance is
equally intractable. […] How to get a balance between the few writers and speakers who are
prestigious and the great majority of text producer and speakers who have no special claim to fame is
not simple” (1998: 63). Biber also stresses the factor of influentialness: “proportional samples are
representative only in that they accurately reflect the relative numerical frequencies of registers in a
language — they provide no representation of relative importance that is not numerical” (Biber
1993:247-8; the author shifts the problem to culture studies, similarly to Sinclair 1991: 13). These
authors seem to be mixing up the concept of ‘representativeness’ with the concept of an external
factor, that of ‘influentialness'.
1.6 ‘Influentialness’
As I have stated, a general corpus should be as diverse as possible to fulfil the statistical axiom
that every effort is to be made to reproduce the variance of the population in the sample. But the noncontrolled
early introduction of influentialness rules out the possibility of the study of variable
influentialness of texts later, in the analysing stage (as, for example, in a sociolinguistic investigation
about the factors of certain sayings becoming proverbs, while others not).
An example from sociology may illustrate my point. Network analysts can undoubtedly suppose
that certain members of the population are more influential, and their sampling methods support the
study of the effect of this influentialness as a variable. It would be a mistake to collect data only of the
‘influential’ members of society the same way as it would have been a mistaken move by early
demographers and sociologists to include only those individuals (i.e. well-educated upper-/middleclass
men over, say, 40 years of age) in their samples, but, fortunately, the tradition never developed
in that discipline.
2 The logbook method
How can we estimate the daily distribution of medium, domain, genre, register, discourse
function and/or subject variation in the population of a given language satisfactorily? Data collection
based on tabular format logbooks filled in by a demographically representative sample of (adult)
speakers yields a reliable picture of these distributions. The logbook consists of sheets of tables
informants are asked to carry along their daily activities, and to fill in rows of cells whenever they use
language, excluding self-talk and meta-notes (‘filling in the logbook'). Information is to be collected
about:
·  the duration of the activity (in minutes or seconds)
·  whether the informant is the producer, a receiver or a third party (role-changes indicated in the
same row)
·  the approximate number of participants (if known)
·  brief demographic details of other participants (if assessable)
·  the medium (spoken or written)
·  the setting (home, office, street, shop, church, etc.)
·  genre (based on a list of possible genres, but extendible).
Other factors (e.g. the domain, or the subject or aim of language use) can be added, controlled
the same way by extendible lists given in the short guide that is fastened to the booklet of the fill-in
tables. A more detailed manual must be given to each informant, who must also be shortly trained.
Data-collection per informant must take two to seven days, at least two days when the
informants’ activities are characteristically different, e.g. a weekday and a weekend day.
To develop the logbook method, I have conducted two pilot studies.
2.1 Pilot study 1
First a small pilot study was carried out to figure out if the logbook method with the structure
described above was at all feasible. Two individuals took notes while travelling on public transport for
a few days. The procedure was executable, and, apart from notes on conversations with acquaintances
and strangers, greetings, chance remarks to strangers (e.g. ‘Sorry!'), newspaper and book reading, the
488
study yielded the yet unacknowledged genres of browsing mega-posters, eavesdropping on others’
conversation, reading shopping lists, etc.
2.2 Pilot study 2
The seven informants participating in this data collection (five females and two males) filled in
the logbook tables for a full day whenever they used Hungarian, starting either early morning and
finishing when going to sleep, or starting at any time on one day, and finishing at the same time the
following day. Data was produced about three weekdays and three weekend days (the seventh
informant started on Sunday, and finished on Monday).
The logbook-booklet included several sheets of A4 tables (see the format below), fastened
together with a short guide. (For the guide and a list of abbreviations, see the Appendix.) Informants
were also verbally instructed how to fill in the tables, what to watch for, and were informed about the
aim of the study.
Informant:................................. Date: ....................................... Started at (hour, minute): ........................
duration You (P/R/3) No. of Rs3 the others wr/sp setting genre (subject) (aim)
A new row in the table was to be filled in every time when either the number of participants, the
medium, the setting or the genre changed, when, for example, a new participant joined or left the
interaction. As the subject was not of primary research importance, subjects within a type of linguistic
activity were allowed to be listed within one cell. When two types of language use were either
alternating or happening parallelly, the informants were asked to connect the two rows describing
them with brackets. E.g. if the informant was reading and at the same time listening to other people
talking; or if spontaneous conversation was regularly interrupted by interesting news on TV.
When I collected the logbooks, I asked the informants to comment on it or the task: they clarified
ambiguous details, and gave plenty of comments, both about the pitfalls of the logbook and their
experiences. They were all surprised to realise that so much time was spent on so many different types
of linguistic activities. One of them even called it consciousness-raising task, because now she realised
that her whole life was being spent almost on nothing else but giving and receiving verbal, visual and
other signals.4
2.3 Results
Table 1 shows some quantitative results: the overall duration5 of linguistic activity by the seven
informants in minutes (mean: 622.86 minutes, standard dev.: 240.2), also broken down by medium.
As it can be expected, a majority of all these people's language use was in the spoken medium, though
with a highly varying proportion.
2.3.1 Spoken activities
The proportion of longer spontaneous face-to-face conversations (LSFC) with family,
acquaintances or strangers were calculated for each informant (see Table 1). The remaining time
within the spoken medium was spent on telephone conversations, fleeting interactions (greetings,
saying Sorry!, paying in a shop), watching movie films or TV-programmes, listening to radio
programmes or to others’ conversations and loudspeaker announcements while travelling on public
transport, mixed classroom activities, etc.
3 “Number of participants” would have been a better label.
4 All my informants were extremely helpful, for which I am indebted.
5 In the case of parallel or alternating linguistic activities, the duration spent on each activity was
calculated separately.
489
age sex time of week sum (min.) written (min.)* spoken (min.)* LSFC**
40 female weekend day 739 95 13% 644 87% 503 78%
39 female weekday 1058 509 48% 549 52% 451 82%
24 male weekend day 736 175 24% 561 76% 540 96%
46 female weekend day 653 196 30% 457 70% 455 99%
12 female weekend-weekday 502 114 23% 388 77% 125 32%
31 female weekday 4006 192 48% 208 52% 141 68%
36 male weekday 2727 32 12% 240 82% 227 95%
Table 1. Duration of Hungarian linguistic activity by informant in pilot study 2 (in minutes);
* = percentage given in proportion to sum; ** = percentage of longer spontaneous face-to-face
conversations (LSFC) given in proportion to spoken activity
2.3.2 Written activities
Informants’ activities included reading newspapers (various types and columns), reading books
(non-fiction: popular science, professional, school textbook; fiction), browsing megaposters, reading a
map, writing and reading e-mails, taking notes while reading, checking one's own paper manuscript,
browsing a library catalogue, leafing through books and magazines in a library or bookshop, reading
posters, billboard advertisements, tourist signs, bus signs, streetname plates and sign-boards in the
street, reading out a bedtime story to a child, entering data into a mobile phone, writing a school-test
and homework, filling in a library request card, using a Hungarian-language word-processor while
correcting an English text. The bedtime tale, a regular activity for parents with young children, is a
genre that was difficult to classify, because it is a mixture of a written text read aloud and spontaneous
conversation.
2.4 Informant sensitivisation
A disadvantage of the logbook method may be that is not robust enough to counterbalance
differences in informant sensitivity. To raise the method's reliability level, informants should be
carefully trained to be able to consciously check all their activities. The pilot study indicated that
certain communicative activity types were not acknowledged, or were not broken down in sufficient
detail, by some informants. For example, classroom interaction is a multi-genre linguistic activity.
Similarly, several genres mix in a morning TV or radio programme: news-reading (the informant is a
recipient of a text written down to be read aloud), interviews (the informant is a third party of others’
spontaneous conversation) and commercials (the informant can be either or both). The detailed
logbook manual must also include informative definitions of setting, genre, subject, etc. with plenty of
supporting examples.
2.5 Advantages
This method, while not yielding language-use data, provides a statistically reliable picture of the
daily distribution of text types in the demographic population, which can be exploited in the
compilation of a demographically oriented corpus. While methods to determine the target population
of published written and internet texts are available, the logbook method offers a solution to assess the
target population of spoken and unpublished written texts. It also makes a comparison with an
existing general corpus based on genre, register, or other classification possible. Moreover, new
genres can be found, and the connection between the number of recipients and ‘relative importance’
or ‘influentialness’ of texts can be studied. The logbook method seems simple and relatively costefficient,
thus it can be based on a large sample of speakers.
2.6 Modular corpora
One may ask: “Do we need representative corpora at all? Monitor corpora are collected with a
primary emphasis on the quantity of texts, and if the texts are well-documented, the user can decide
on the proportions.” The problem is that most users (lexicographers, syntacticians, etc.) are not
informed about problems of corpus design, and have no tools to assess populations but their own
6 She reported having spent most of her day writing a paper in English.
7 He reported having been writing an English language computer programme all day.
490
intuitions. It is the corpus designer's and the sociolinguist's task to assess the textual and
demographic population not only for reference, but also for monitor corpora.
If texts in a monitor corpus are well-documented as far as the relevant categories (‘strata') are
concerned, a user-interface picking texts according to given criteria can produce any composition of
texts, with the possible control over other criteria. Thus, with the help of such an interface either a
balanced build-up of texts can be composed (e.g. for a multi-genre comparative analysis of a syntactic
feature), or a demographically proportional one — based on the results of a survey using the logbook
method (e.g. for a sociolinguistic study on ‘influentialness', or for another study on the spread pattern
of politically correct phrases). The end-user's preferred composition could also be set (e.g. including
only the coded spoken texts for a prosodic analysis, or texts produced in a given year for a study on
new coinages), or the totality of the modular corpus's texts could be used, as well. When the interface
has randomly picked the texts according to the given criteria, it must give summary statistics about
the composition of texts, so that the analyst could change the criteria if the composed corpus is too
small. Misbalance in the original monitor corpus (causing also end-user corpora to be too small) can
be adjusted by cyclical fine-tuning (Biber 1993, Váradi 1998) by including missing text types. Such an
interface may help to exploit the possibilities of monitor corpora.
References
Biber D 1993 Representativeness in corpus design. Literary and Linguistic Computing 8 (4): 243-257.
Biber D, Johansson S, Leech G, Conrad S, Finegan E 1999 Longman grammar of spoken and written
English. Harlow, Longman.
BNC Online (1997) http://info.ox.ac.uk/bnc/what/spok_design.html
Burnard L 1995 The BNC Handbook. Oxford, Oxford University Press.
Johansson S 1980 The LOB Corpus of British English texts: Presentation and comments. ALLC
Journal 1 (1): 25-36.
Kennedy G 1998 Introduction to corpus linguistics. London-New York, Longman.
Leech G 1991 The state of the art in corpus linguistics. In Aijmer K, Altenberg B (eds), English
corpus linguistics: Studies in honour of Jan Svartvik. London, Longman, pp 8-29.
Sinclair J 1991 Corpus, concordance, collocation. Oxford, Oxford University Press.
Váradi T 1998 Nyelv és korpusz: a reprezentativitás a korpusznyelvészetben (Language and corpus:
representativeness in corpus linguistics). Manuscript. Budapest, Research Institute for
Linguistics.
491
Appendix: The logbook guide for pilot study 2.
Carry the booklet with this guide for a full day wherever you go,
and fill in the lines every time you use the Hungarian language.
(Except when thinking/talking to yourself or when filling in the tables.)
duration: of the linguistic activity (how many minutes/seconds did it take?)
You (P/R/3): were you a Producer/Receiver/third party?
No. of Rs: an approximate number of Receivers
others: brief demographic details of other participants — if known
(sex, age, profession, residence: Budapest/town/village)
wr/sp: written/spoken (or other: e.g. written text read aloud)
setting: e.g. shopping, cinema, conference, religious service, etc.
genre: OTHER GENRES CAN BE NAMED, TOO!
·  two- or several-party personal conversation
(e.g. with family, acquaintances, strangers in the street, in a shop, in the doctor's office,
etc., or listening to others’ conversation as a third party)
·  two- or several-party personal conversation
(e.g. with family, acquaintances, strangers)
·  fleeting interaction with acquaintance, stranger (e.g. only greetings, saying sorry, asking
for a journal at the news-stand, etc.)
·  giving presentation in front of a present audience/listening to one
·  listening to a loudspeaker announcement
·  work meeting
·  radio/TV: news, interview, debate, sports commentary
·  radio/TV/tape recorder/CD: music with text (if you listen to text)
·  radio/TV/movie film, theatre play
·  radio/TV: commercial
·  reading: newspaper, magazine, internet
(column: news report, editorial, fiction, advertisement, etc.)
·  reading: book (fiction, science, law, etc.)
·  reading: short message (letter, shopping list, etc.)
·  reading: megaposter, poster, announcement, brand names on buildings or clothes, sign-board
·  writing:: short message (letter, shopping list, etc.)
·  writing: creation of longer text (diary, paper)
Optional:
subject
aim: e.g. playing, giving or receiving information/orders, amusement, killing time, etc.
If two types of language use are either alternating or happening parallelly, connect the two rows
describing them with brackets.
Thanks a lot for your effort!

492
The Web as a Source of Linguistic Information
Antoinette Renouf
Research and Development Unit for English Studies, University of Liverpool.
However large and up-to-date the available electronic text corpora are, there will always be aspects of
the language which are too rare or too new to be evidenced in them. In fact, the WWW is the largest
existing repository of texts across a range of textual domains. It is not surprising that individual corpus
linguists have increasingly hit upon the idea of querying the standard web search engines in order to
retrieve the more recondite or newly-minted instances of language use. Whilst this strategy can yield
useful linguistic results, the standard engines are not designed for the purpose, and the procedure is
prohibitively slow and the output requires extensive post-editing. Last year, the Research and
Development Unit for English Studies at Liverpool moved on from being such users, taking on board
the needs of the community and beginning to develop ‘WebCorp', an Internet search system which
allows on-line access to web texts as linguistic rather than information sources. A demonstration tool is
available at: http://www.webcorp.org.uk. This paper will report on the research initiative and highlight
some the issues involved.

493
The influence of the passive on text cohesion and technical terminology
A corpus-based study of research article abstracts from the domain of electrical
engineering.
Jérôme Richalot
Institut National des Sciences Appliquées de Lyon (INSA)
jerome.richalot@insa-lyon.fr
Abstract
The passive is used extensively in research article abstracts. It is often agreed that the passive helps
focusing on the actual content of the article rather than on its author(s). The abundance of passive
sentences is often frowned upon and regarded as unnecessarily burdening scientific discourse. The aim
of this paper is to go beyond this assumption and to show that the use of the passive serves a clear
purpose: underlining argumentation and therefore strengthening text cohesion.
A study of passive sentences in 83 research article abstracts from IEEE Transactions on
Ultrasonics, Feroelectrivity and Frequency Control (UFFC) is conducted. Passive sentences are
extracted and classified. The first criterion for classification is the part of the abstract the sentence
belongs to (IMRaD). A study of verbs, their semantics and informative content is then carried out.
Subsequently, the grammatical subjects of passive sentences are examined. The appearance of technical
terms from the domain of electrical engineering and their prospective or retrospective orientation
receive particular attention. An attempt is made at characterising subject noun groups.
The corpus-based approach helps underlining the blandness of verb semantics as opposed to the
richness of subject noun groups in sentences that often appear unbalanced. It is thereby established that
the use of the passive in the 83 abstracts studied only partially matches the description of the passive
based on general English. Indeed, in the abstracts that make up the corpus, passives do not seem to
either characterise the grammatical subject of the sentence or put any particular stress on the result of
the process as referred to by the verb.
It is suggested that, by receiving so much emphasis, the subject noun group becomes the centre of
predication for the sentence. The “castaway” verb group only carries little information and can
sometimes even take on a purely metalinguistic role. This topicalisation process is seen as particularly
efficient in helping authors structure their abstracts and therefore helps strengthen text cohesion.
Technical terminology seems to benefit greatly from this use of the passive. Terms appear in a cotext
helping to situate them in the author's conceptual framework thus contributing to a better
understanding of the specialised field. Both the corpus approach and the results yielded could be seen
as valuable contributions to terminology processing and TEFL.
1. Introduction1
Many books have been dedicated to “English for science” aimed at specialists of other areas than
English. The ones I am referring to in this introduction all deal with abstracts with more or less details.
In the instructions to contributors of IEEE Transactions on Ultrasonics Feroelectricity and
Frequency Control (UFFC), it is clearly specified that “Each contribution must contain an abstract (not
more than 200 words for papers and 50 words for correspondences and Letters).” This is the only
instruction given to contributors on the abstract itself. However, in the confidential proof-reading form
given to each author, under the heading Summary of evaluation, one of the first criteria regarding
presentation is “Is the summary an adequate digest of the work reported in the paper?” This remark
underlines the importance of the abstract as well as its conventional aspect. Robert A. Day (1989: 28)
goes as far as considering the abstract as a “miniversion of the paper.”
The aim of the abstract is to help the distribution of the article it refers to hence this statement by
Day (1989: 28) “The abstract should provide a brief summary of each of the main sections of the
paper.” Lobban et Schefter (1992: 47) add another characteristic to the abstract which they see as a
“self-contained synopsis of the report”. Sites (1992: 113) also endorses this remark stating that
“abstracts are self-sufficient”. These elements of definition however concise they may seem underline
the secondary aspect of the abstract. Day (1989: 28) uses the term “secondary publication” with
reference to secondary services in which an abstract can be published independently from the article it
sums up (Biochemical Abstracts, Chemical Abstract, etc.) It therefore seems appropriate to consider the
abstract as a metatext. It accompanies research articles but usually has its own layout. In the same way
1 Quotations from French references have been translated and marked [translation]. The French is
given in footnotes when necessary.
494
as literary criticism it is based on another text. The subject of the abstract is the article. According to
Lobban and Schefter (1992: 47) “The Emphasis in an abstract is on the results and conclusions. It
should have only the objectives from the Introduction, and only a brief reference to the Materials and
Methods (unless the experiments focused on methods).” They explicitly refer to the Introduction,
Methods and Materials, Results and Conclusion parts of the article. In many cases the structure of the
abstract does indeed reflect the IMRaD structure of the article. Day (1989: 28) clearly lays out the
following principles: “The Abstract should (i) state the principal objectives and scope of the
investigation, (ii) describe the methodology employed, (iii) summarize the results, and (iv) state the
principal conclusion.”
Sites (1992: 113) makes a difference between informative abstracts and descriptive abstracts
qualifying the latter of “little more than prose table of contents”. In the study of passive conducted here
I shall try to show that but for the negative connotation Sites’ remark is very appropriate.
The very functional elements of definition given above are usually followed by a series of rules and
regulations for a “good abstract”. Here are just a few examples. For Day (1989: 159) “Most of the
abstract should be in the past tense because you are referring to your own present results.” For Vernon
Booth (1993: 13), “The passive voice, commonly used to describe results, sometimes makes clumsy
constructions. Turn a passive voice to direct style when you can.” These instructions seem difficult to
apply and sometimes even to understand. More pragmatic advice such as Lobban and Schefter's (1992:
57) is certainly preferable: “However, there are divergent and often strong opinions about whether use
of the passive is good or bad. [Ask what your professor wants !]. One of our colleagues aggressively
promotes the passive. (“The report is about the experiment, not about you ! she exclaims.) Another
colleague equally hotly denounces the use of the passive as “Victorian prudery” that leads to
“committee writing [style], ponderous discussions, and avoidance of responsibility.” ”
In the literature for specialists of other areas than English there seem to be as many “recipes” for a
good abstract as books published on the topic. Still however awkward some of the instructions might
sound, they show that abstracts do have their own style. If the authors quoted above seem to find it
difficult to clearly define this style — Lobban and Schefter (1992: 57) even resort to a so-called
“scientific etiquette” — they at least fully recognise its existence.
2. A linguistic approach
Since linguistic studies are one of the only research areas in which the object being studied is also
the one used to report on those studies, a clear metalanguage should be defined. I shall therefore first
set out as clearly as possible the definitions for the terms used throughout this study. In the case of the
passive, this seems all the more important as the metalanguage gives us a first hindsight into the actual
operations at work when building a passive sentence.
As Henri Adamczewski (1993: 180) points out the very terms of passive verb or passive form do
not seem appropriate. They tend to restrict the passive to a phenomenon mainly concerned with the
verb group. The verb group is indeed on the forefront of morphological and syntactic changes and no
research on the passive can possibly be conducted without a thorough study of the verbs being used in
the passive, however this study focuses on sentences (or utterances2) and I shall therefore, depending
on the context, use either passive sentences or passive voice in which voice refers to subject-verb-object
ordering. Although it seems awkward to do without the very term passive, particularly in language
teaching, linguistic grammars have made it difficult to actually justify its use. Jean-Rémi Lapaire and
Wilfried Rotgé (1991: 364) propose the following etymology. Passive as an adjective comes from the
Latin pati for “suffer” or “undergo”. Passive would therefore be motivated by the appearance, in a
passive sentence, of a subject undergoing the process described by the verb. It seems obvious that this
feature greatly depends on the semantics of the verb used. In (1), John does indeed undergo a process.
(1) John was hit by Paul.
But the situation is altogether different in (2).
(2) John was greeted by Paul on the doorstep.
Furthermore, the emergence of a subject undergoing the process described by the verb is not, in
itself, a prerequisite for a passive sentence as in (3).
(3) John suffered the same fate as his brother.
2 Marie-Line Groussier and Claude Riviere define the utterance as a basic unit of study. “An utterance
is produced throughout a unique uttering act by an utterer for whom it makes up a whole, which has
consequences on prosody.” [translation] Utterances are therefore not exclusively oral. Groussier and
Riviere point out that utterances and sentences often correspond. But featuring a verb is not a
prerequisite and they consider the French “Moi, un voleur !” as an utterance.
495
The inaccuracy these three basic examples underline most probably arises from the confusion
between linguistic and extralinguistic, or in other words between a universe of experience and one of
representation. André Joly's and Dairine O'Kelly's remarks (1990: 152 [translation]) offer significant
help: “[...] subject and object are grammatical functions, they refer to imaginary behaviours existing as
thoughts and more particularly thoughts related to language. On the contrary, the impressions which
define the notions of agent or patient do not correspond to functions – that is to roles played by a noun
or its substitute in a sentence –, but to situations, states originating in the world of sensations.”
This distinction enables me to clearly disconnect the active voice from the passive voice. The
active/passive couple is an inheritance from descriptive grammars in which a sentence with a transitive
verb “in the active voice” had its equivalent “in the passive voice”. Whenever “John was eating an icecream”
that very ice-cream was systematically “being eaten by John.” The trend was most certainly
further accentuated by generative grammars. In Syntactic Structures (1957: 80), Chomsky does wonder
about the possibility of considering passive sentences as part of the kernel of English, but rejects the
idea faced with the complexity of the transformations then required: “When we actually try to set up
for English, the simplest grammar that contains a phrase structure and transformational part, we find
that the kernel consists of simple, declarative, active sentences (in fact, probably a finite number of
these), and that all other sentences can be described more simply as transforms.”
Lucien Tesniere (1959: 242) opened new perspectives considering the passive in the wider scheme
of transitivity. He defined four diatheses3:
§ the active diathesis. “Alfred hit Bernard.”
§ the passive diathesis. “Bernard was hit by Alfred.”
§ the reflective diathesis. “Alfred killed himself.”
§ the reciprocal diathesis. “Alfred and Bernard killed each other.”
Each diathesis is a virtual realisation of transitivity. Adamczewski (1993: 185) as well as Joly and
O'Kelly (1990: 151) praised the coherence of Tesniere's model but also underlined its limits. In
Tesniere's model, the passive diathesis remains closely linked to the active diathesis.
Nowadays, the so-called subordination of the passive to the active is widely recognised as irrelevant
and a few simple examples effectively stress the point. Adamczewski (1993: 186 [translation]) sees the
passive as “a phenomenon inscribed in the workings of the human language.” He presents the
following verb classification4:
§ Verbs with an active or a passive orientation (also known as notional passives)
(4) They grow tomatoes.
(4') The tomatoes won't grow this year.
(5) She reads the play well.
(5') The play reads well.
§ Exclusively passive verbs
(6) He was addicted to heavy drinking.
(7) She is bound to find it if you're not careful.
§ Non-passivable verbs
- subject oriented and non reversible (middle verbs)
(8) The young soldier weighed six stones.
(8') * Six stones were weighed by the young soldier.
- with a cognate object
(9) She shrugged her shoulders.
(9') * Her shoulders were shrugged by her.
- reflective
(10) John praised himself.
(10') Himself was praised by John.
Still, I do not intend to reject as a whole previous studies on the passive and will not deny that the
active voice often sheds light on the operations being carried out when a passive sentence is produced.
The metalanguage used in the studies I quote as references is often reminiscent of the traditions of
descriptive grammars. When Adamczewski (1991: 187 [translation]) proposes the following model for
the passive voice:
N2 {be} + past participle {by} N1
the very labels N2 and N1 are reminiscent of the N1-Vt-N2 model for the active voice.
3 The examples are a translation of Tesniere's.
4 The examples are Adamczewski's.
496
causation effection
operation
(event)
Joly and O'Kelly (1990: 155) do propose the term resultative voice5 which is however first
introduced in a parallel with their operative voice.
It therefore seems appropriate for Lapaire and Rotgé (1991: 365) to point out an ambiguous
dependence of the passive on the active. And it should come as no surprise that Adamczewski (1993:
181 [translation]) himself refers to the active voice as a structure “perceived as primary though with a
certain unease.”
However, Adamczewski goes on explaining the reasons for this uneasiness. He sets out the minimal
ternary model N1 Vt N2 as corresponding to the traditional SVO model of transitivity. He argues the
SVO model of transitivity can be considered as primary being “as close as possible to the
extralinguistic universe: in the SVO model, the autonomous subject uses the semantics of the verb to
somehow modify or affect the object of the verb. Transitivity can be nothing else than this movement
or dynamics, which in itself is the language parallel of man's action on the material world.” (1993:
186-187 [translation])
Joly and O'Kelly (1990: 154 [translation]) quote Moignet (1974 and 1981) adding that “any event
expressed by a verb has a causation: an initiating reference. It also has an effection6: upon completion
of the operation represented by an event, a result is necessarily attained.” The model they propose
(figure 1) features the same “movement” or “dynamics” referred to by Adamczewski:
Figure 1
The passive therefore does appear as a secondary construction, the reversing not of the constituents
of an hypothetical active sentence but of a predicative movement. The “affected” – for which we might
venture the term “effected” - becomes the origin of the sentence, the second term of the equation being
the verbal predicate. The newly founded relation between the effected (N2) and the verbal predicate
(Vt) is the very raison d'etre of the passive. Adamczewski (1993: 187 [translation]) puts it as
straightforwardly as possible “given an abstract clause: N1 Vt N2, N2 has been chosen as the origin of
the utterance and what the utterer is aiming at is the N2-Vt relation, N1 being absent.” The passive can
clearly be considered as an operation pertaining to thematisation.
The syntactic components of a passive sentence are most interesting with regards to the very nature
of this phenomenon. As seen above (3) having a subject undergoing the process described by the verb
is not enough for a sentence to be considered passive. Lapaire and Rotgé (1991: 348 [translation])
consider the use of be as a “capital phenomenon” through which “the predicate [is] formally
“separated” from the grammatical subject [and] is placed under the control of the utterer.” They
propose (1991: 347-348 [translation]), a very interesting brief diachronic study of be which should
certainly be considered here. According to them the original meaning of be was “to occupy a place” but
it soon became “to have a place in the universe, thus to exist.” They therefore define be as “a marker or
instigator of existence, either absolutely (I think therefore I AM), or relatively enabling the utterer to
assign the grammatical subject a permanent or temporary characteristic (e.g. I AM tall, she IS angry.)”
They do see in be “an actuating power – in the philosophical meaning of the term “give a reality to...”,
“state to the world the reality of...” – and a characterising power: the utterer takes a defining point of
view with regards to the grammatical subject by assigning it one or several traits he deems essential or
at least relevant (at least in a given situation).”
This hints at the double effect of be. In terms of position in the sentence, between the grammatical
subject and the verb, it breaks up the direct predicative link and freezes the grammatical subject.
Lapaire and Rotgé (1991: 368 [translation]) quite efficiently use the term “freeze frame”. They remind
us that this impression is further accentuated by the fact that “in terms of semantics, the subject of a
passive construction does not control the lexical verb.” Let us add that most passive sentences are in
fact agentless, which might have a role to play in freezing the grammatical subject.
The second effect of be is a highlighting one. As can be understood through the previous paragraphs,
be is an attributive auxiliary, it looks towards the grammatical subject of the sentence and actualises or
characterises it. The model Joly and O'Kelly (1990: 161) propose (figure 2) underlines this:
(11) The door is locked.
5 “voix résultative” in French as used by Joly and O'Kelly.
6 Causation and effection in French as used by Joly and O'Kelly.
497
Figure 2
Trying to replace be with another auxiliary offers interesting contrasts. Statistically get is the most
likely candidate but also possible are become, remain, seem, appear, feel, look – with an effect close to
that of get. Let us consider the following examples:
(12) He got run over by the whole pack.
(13) He became irritated and soon lost his temper.
(14) He seemed affected by the sad news.
(15) He appeared relieved when she left.
(16) He feels overwhelmed.
(17) He looked disappointed.
Although these sentences remain subject-oriented they tend to let us see the process described by
the verb (Vt-EN) as unfolding. Joly and O'Kelly (1990: 161 [translation]) do remind us of the existence
of the auxiliary weor.an in old English meaning become the use of which, in passive sentences, was
close to the one I describe here. Adamczewski (1993: 189 [translation]) points out that “in German
werden (a metalinguistic operator derived from werden = become) is used and not sein (be).”
The semantics, or semic programme7, applied to the grammatical subject is of course that of the
transitive verb (Vt-en). The very form of the verb, the past participle, is revealing. We might
instinctively recognise Vt-EN and V-ED (preterit) as quite close, however the two forms are quite
different. According to Lapaire and Rotgé (1991: 446 [translation]) “thanks to -EN, the predicate is
linked to a past period of time (cf. past participle).” Joly and O'Kelly (1990: 159 [translation]) offer a
slightly different perspective. “[...] The form of the past participle, e.g. broken, marks the verb as
having reached the end of its tension: it is its effection form. The verb is then reduced to the result of
the operation called operatively by break.” The past participle form of the verb therefore signals the
operation of the verb as “effective”; from this point on, what is being considered is the result of this
operation.
The syntax of a passive sentence enables us to look back on the operations carried out by the
speaker. Off course, it supports the potential of the passive as described above. With the help of be the
speaker enters the predicative relation and makes it his own. From this point on, he handles discourse
objects and gets further away from the extralinguistic universe.
Lexis as defined by Groussier and Riviere (1996: 112-113 [translation]) could help summing up
what I have developed so far. Groussier and Riviere make a difference between predicable lexis and
predicated lexis8. The predicable lexis is the notional model of a process in terms of actors and it is up
to the speaker to assign a value to each of the actors described in the model. The label given to the first
actor “x0” which becomes “a” or “C0” once assigned a value, symbolises its position as origin of the
message. The predicated lexis is therefore the result of predication applied to the predicable lexis.”
Passive predication is “the choice as first argument, not of the actor in the x0 position of the lexis
model (the source of the primitive relation), but of the actor in the x1 position (aim of the primitive
relation).”
So far I have tried to show that using the passive is in fact setting up a predicative model the aim of
which is to promote the secondary notion in the predicable lexis to a primary status in the predicated
lexis. The operation means defining the starting point of the sentence, its theme. In this study of highly
specialised but also very controlled texts (see definition and characterisation of the abstract above),
aimed at a well defined discourse community, thematisation takes on an even greater importance. For
the authors of the abstracts which make up the corpus I am studying choosing the theme of a sentence
means deciding which information should be considered as given, as determined enough to be shared
with their discourse community, with no further explanation. I therefore expect these theme segments
to reveal some of the characteristics of the discourse community the abstracts are aimed at. This could
be particularly interesting with regards to technical terminology shared by the discourse community.
The theme segments should feature a high density of technical terms related to the specialised field
being considered.
7 Semic programme refers to the semantic features or semes of the verbs. These traits represent the full
semantic potential of the verb only part of which realises at predication.
8 “Lexis prédicable “ and “lexis prédiquée” as used in French by Groussier and Riviere.
The door
is
locked
498
3. A corpus-based study
The purpose of the following corpus-based analysis is to test my hypothesis on the passive. No indepth
study of a corpus of research articles is in fact necessary to underline the frequency of passives.
A mere count of the occurrences of be (be/being/is/are/was/were) over several articles will actually
yield very little noise. I decided to conduct a study of passive sentences in research article abstracts. I
selected 83 abstracts from IEEE Transactions on Ultrasonics, Feroelectrivity and Frequency Control
(UFFC)9 that is the abstracts from the three issues of the first semester of 1999. Thanks to Mike Scott's
Wordsmith tools suite I extracted the passive sentences from the corpus. Using the Concord tool,
passive sentences can be extracted automatically concordancing for be or any of its derived forms
being, is, are, was, were and *ed or any of the infamous “irregular past participles” to its right. The
concordance should exclude past participles left of be and look for the past participle two to three
words right of be. Minor post editing is necessary.
Using this set up, I extracted 358 passive sentences from ABCORP. Throughout the study of these
358 sentences I shall use and adapt the terminology first used by Adamczewski. I shall therefore refer
to the grammatical subject of the sentence and to the verbal predicate (a transitive verb) using
respectively N2
10
and Vt-en11.
I shall first try to classify the Vt-en's used in the passive sentences in ABCORP. Then I shall study
their N2's to try to get a better understanding of the Vt/N2 relation which I defined above as being the
raison d'etre of the passive.
4. Categorisation of transitive verbs in passive sentences
The first data that can be used upon processing of the corpus are the raw results given by Concord.
From the 358 passive sentences I have extracted 368 occurrences of Vt-en's. The difference is due to
sentences in which two or more Vt-en's are co-ordinated by and or or without be being repeated, as in
the two examples below.
(18) Frequency spectra and modes are computed and examined.
(19) These effects can be reduced or eliminated by using narrow-band experiments.
The 368 occurrences correspond to 145 different verbs. Their frequency is interesting. On the one
hand, 30% of those Vt-en's appear only either once (81)12 or twice (23). On the other hand the four
most frequent Vt-en's represent 20% of the 368 occurrences. These four verbs are in fact somehow
emblematic of the IMRaD structure. They are use (29) for the Method section, present (20) for the
Introduction section, obtain (16) for the Result section, and propose (13) for the Discussion section.
However this mere frequency count should not lure us into hasty conclusion.
Table 1 below maps those verbs over the IMRaD part they appear in. It also adds other less frequent
verbs with a semantic content close to them. The methodology I am using here is somewhat similar to
the one suggested by Biber (1998: 123). Biber describes the register of research articles in experimental
science as “one of the few English registers that clearly distinguishes among internal purpose-shifts.
[…] Because each of [the IMRaD sections] is overtly marked in the text and has distinct
communicative functions […]”
This table leaves out some of the verbs likely to be classified at this stage but which only occurred
once throughout the corpus. I consider these verbs of no statistical relevance.
The verbs in Table 1 all have a very large semantic content, i.e. a low number of semes or semantic
features. All are widely used outside the scientific register and their meaning does not vary
significantly whether within or without this register. None of them can be said to belong exclusively to
a specialised scientific domain. I shall refer to these verbs as category 1 verbs.
9 The abstracts are available under HTML format from IEEE's web site at
http:\\uffc.brl.uiuc.edu\Trans\ I shall henceforth refer to this collection of abstracts as ABCORP.
10 N2 can refer to a simple noun as well as a complex noun phrase.
11 In metaoperational linguistic grammars such as Adamczewski's the use of Vt-en to refer to the past
participle of a transitive verb underlines the regularity of the system. In the system, the verb (V) can be
V-ing, V-ed or V-en.
12 The figure in brackets indicates the number of occurrence of form being referred to.
499
Part of the abstract Vt-en
Introduction achieve (1/4) apply (2/7)
base (2/7) consider (3/5)
describe (4/7) determine (1/4)
discuss (3/7) find (2/7)
improve (1/5) investigate (3/6)
know (2/3) obtain (2/16)
present (6/20) propose (7/13)
study (2/5) use (6/29)
Method and Materials achieve (1/4) apply (4/11)
base (5/6) consider (1/5)
demonstrate (1/3) describe (3/7)
determine (3/4) discuss (2/7)
give (1/6) identify (3/3)
investigate (2/6) observe (1/5)
obtain (3/16) perform (6)
present (5/20) produce (1/4)
propose (5/13) show (1/10)
study (3/5) suggest (1/2)
use (20/29)
Results achieve (2/4) apply (1/11)
consider (1/5) demonstrate (2/3)
find (5/7) give (5/6)
investigate (1/6) observe (2/5)
obtain (11/16) present (8/20)
produce (2/4) show (9/10)
suggest (1/2) use (1/29)
Discussion discuss (2/7) observe (2/5)
present (1/20) produce (1/4)
propose (1/13) use (2/29)
Table 113
The second category of verbs extracted from the passive sentences of the corpus gathers verbs with
a more restricted semantic content. Those verbs describe processes central to research in science or
technology. Table 2 below shows they are used more frequently in a scientific context than in “general
English”. To compare the frequency of these verbs in ABCORP and in “general English”, I have
chosen the British National Corpus14 as an element of comparison. The size of the BNC, 100.1 million
words, and its diversity, some 4124 different files, make it the corpus most representative of “general
English” available today. It could be argued that the BNC represents British English whereas ABCORP
is composed of abstract published in an American Journal. However, apart from minor spelling
differences (i.e. analyse vs. analyze) I do not think that the abstracts are overtly marked by their
“Americanness” and I am convinced that the comparison made with the BNC yields reliable results.
Because I do not think they can be of any statistical relevance for this study, I have excluded from
the comparison and the resulting categorisation the verbs that could have been taken into consideration
at this point but which only occurred once throughout the corpus even once lemmatised (e.g. adjust in
(20)). I have kept the verbs describing processes that I considered central to research in science or
technology even though they occurred only once if they somehow participated in a specialised term of
the domain of ferroelectricity, ultrasonics and frequency control (e.g. enlarge in (21) was kept because
asymmetrical bandwidth enlargement belongs to the terminology of UFFC and occurs in ABCORP). I
have also kept the verbs likely to be categorised here but which only appeared once in the corpus if a
word of the same family appeared in at least one file other than the one the passive had been isolated in
(e.g. tune in (22) because cavity-tuned hydrogen masers appears in a file different from the one (22)
appears in).
13 The first figure in brackets refers to the number of occurrences of the verb in the part of the abstract,
the second figure refers to the total number of occurrences of the verb in the corpus.
14 Data cited herein has been extracted from the British National Corpus Online service, managed by
Oxford University Computing Services on behalf of the BNC Consortium. All rights in the texts cited
are reserved.
500
(20) The time interval between two consecutive samples can be continuously adjusted to
avoid undesirable sample volumes.
(21) At higher pressure levels, it even may happen that, in a steady, unidirectional flow
(which should generate only positive Doppler frequencies), the Doppler spectrum is
enlarged up to the point that negative Doppler shifts also are produced.
(22) As an example of this new class of voltage-tunable chip SAW devices, a voltagecontrolled
oscillator (VCO) is presented in which the output frequency can be tuned by
an applied gate voltage.
For the purpose of this comparison, I have used one of the frequency lists provided by Kilgariff15
(RF2). The list is a lemmatised frequency list. It features the words occurring more than 800 times in
the BNC and gives their part of speech. I have lemmatised the verbs describing processes which can be
considered as scientific or technological and which appear in passive sentences in ABCORP and
calculated their raw frequency (RF1). I have then calculated a mean frequency (MF1) over 1,000 words
for each of them. In a similar way, I have calculated a mean frequency (MF2) for the same verbs in the
BNC. I have subtracted each verb's MF1 to its MF2. A positive result means that on average the verb
occurs more frequently in ABCORP than in the BNC; a negative result means that on average the verb
appears more frequently in the BNC, or in “general English” than in ABCORP.
Table 2 shows the ten most and least frequent verbs from this second category. The top ten
undoubtedly describe a process central to research in science or technology. I have decided to keep the
verbs with an MF1-MF2 close to zero or negative arguing that those verbs have a rather large semantic
content which realises somewhat differently in “general English” and in ABCORP. The category
gathers 78 verbs which I shall refer to as category 2 verbs.
Verb infinitive ABCORP BNC
RF MF1 RF2 MF2 MF1-MF2
measure 16 0,8676 6683 0,0668 0,8009
compare 16 0,8676 12591 0,1258 0,7418
detect 13 0,7050 3231 0,0323 0,6727
calculate 12 0,6507 3922 0,0392 0,6115
provide 19 1,0303 47923 0,4788 0,5516
analyze 9 0,4880 4106 0,0410 0,4470
simulate 8 0,4338 <=80016 0,0080 0,4258
operate 8 0,4338 <=800 0,0080 0,4258
design 10 0,5423 11810 0,1180 0,4243
develop 12 0,6507 24205 0,2418 0,4089
…
realize 2 0,1085 9726 0,0972 0,0113
fix 1 0,0542 5282 0,0528 0,0015
drive 3 0,1627 16477 0,1646 -0,0019
train 2 0,1085 11907 0,1190 -0,0105
prefer 1 0,0542 6854 0,0685 -0,0142
point 2 0,1085 12844 0,1283 -0,0199
receive 4 0,2169 24111 0,2409 -0,0240
expect 1 0,0542 27221 0,2719 -0,2177
carry 1 0,0542 31258 0,3123 -0,2580
set 1 0,0542 40381 0,4034 -0,3492
Table 2
Table 3 maps these verbs over the IMRaD part of the abstracts they appear in.
Part of the abstract Vt-en
Introduction align (1/2) analyze (1/4) compare (1/7)
15 Kilgariff's frequency lists are available from http://www.itri.bton.ac.uk/~Adam.Kilgarriff/bncreadme.
html or via ftp from ftp.itri.bton.ac.uk/bnc.
16 Kilgariff does not list lemmatised forms with a frequency smaller than 800. For the purpose of my
calculations, I have consider the RF2 of verbs not listed by Kilgariff as 800.
501
construct (1/2) design (1/2) detect (1/5)
develop (5/8) establish (1/2) expect (1/1)
fabricate (1/4) fix (1/1) monitor (1/1)
operate (1/2) relate (1/3)
Method and
Materials
align (1/2) analyze (3/4) assess (1/3)
average (1/2) calculate (6/6) calibrate (2/2)
choose (1/1) compare (5/7) compute (3/5)
control (1/1) convert (1/1) correct (1/1)
decrease (1/1) degrade (1/1) derive (3/3)
detect (2/5) develop (3/8) dominate (1/1)
drive (1/1) establish (1/2) evaluate (1/2)
fabricate (3/4) formulate (1/1) insert (2/2)
introduce (1/1) locate (1/1) manufacture (1/1)
map (1/1) measure (2/2) model (1/1)
modify (2/2) observe (1/5) operate (1/2)
position (1/1) predict (1/1) propagate (1/1)
provide (1/1) realize (1/1) record (1/1)
scan (3/3) select (1/1) separate (2/2)
set (1/1) substitute (2/2) test (1/1)
train (1/1) transform (3/3) utilize (1/1)
validate (1/1)
Results affect (1/1) assess (2/3) attenuate (1/1)
attribute (1/1) average (1/2) carry (1/1)
compare (1/7) compute (2/5) confirm (1/1)
construct (1/2) detect (2/5) enlarge (1/1)
evaluate (1/2) examine (1/1) illustrate (1/1)
increase (1/1) indicate (1/1) model (1/1)
observe (2/5) point (1/1) rate (1/1)
receive (1/1) reconstruct (1/1) reduce (1/1)
relate (2/3) report (1/1) tune (1/1)
verify (1/1)
Discussion characterize (2/2) design (1/2) implement (1/1)
observe (2/5) prefer (1/1) quantify (1/1)
reconstruct (1/1) require (1/1) simulate (1/1)
solve (2/2)
Table 3
The third category of verbs extracted from the passive sentences of ABCORP gathers verbs that
would only occur in a scientific register17 and quite specifically in the domain of electrical engineering.
This category is made up of only two verbs: pole (2) and repole (1). The two verbs appear in the same
abstract; pole occurs twice in the Discussion part whereas repole occurs in the Method and Materials
part.
As stated earlier in this paper, I have decided to study the passive as a sentence-level operation. I
therefore have not calculated a mean frequency count18 for passives by section as proposed by Biber
(1998: 124-125) since his count is word-based. The example given by Biber, a study conducted over 19
research articles from either the New England Journal of Medicine or the Scottish Medical journal
published in 1985, points out to Methodology sections being “marked by their extremely frequent use
of agentless passives”. Unlike Biber's, this study is based on a corpus of research article abstracts and
not full-length articles and so far, it has made no difference between agentless passives and passives
with an agent. However, the three categories that I have defined hint at similar results in terms of the
frequency of passives over the IMRaD sections of abstracts. Table 4 below shows the breakdown of
passive sentences by IMRaD section.
17 Pole as a verb only occurs three times in the BNC with a meaning similar to the one it has in the
following example. “This is how we pole a raft and just because a white man is watching through his
funny machine we aren't going to do it any differently.” (Barnes J 1990 A history of the world in 101
Chapters London, Picador). Repole does not occur in the BNC.
18 For an explanation of the mean frequency count, see Biber's methodology box 8, the unit of analysis
in corpus-based studies (1998: 269).
502
Part of the abstract First category Second category Third category Total
Introduction 26% (53) 13% (18) 20% (71)
Method and Materials 42% (84) 55% (80) 33% (1) 47% (165)
Results 28% (56) 23% (33) 26% (89)
Discussion 4% (9) 9% (13) 66% (2) 7% (24)
(202) (144) (3) (349)19
Table 4
Biber's brief study is given as an example for the study of discourse characteristics and it concludes
that agentless passives contribute to “presenting events impersonally, with no acknowledged agent.”
The following aims at explaining that this effect, although sensed by most readers, is probably a
secondary effect for the targeted discourse community.
5. Categorisation of subject noun-groups in passive sentences
Building on the above categorisation, a study of the verbs’ grammatical subjects (N2's) is
particularly revealing. I shall focus on noun determination.
N2's first appear as very diverse as shown by the examples below:
(23) The effects of the array parameters on the array performance, such as the selectivity
of Lamb modes and effectiveness of Lamb wave generation are investigated.
(24) A new procedure for preparing lead zirconate titanate (PZT)/poly(vinylidene
fluoride-trifluoroethylene) (P(VDF-TrFE)) 1-3 composites with both phases
piezoelectrically active is described.
(25) A Electromechanical coupling mechanisms in piezoelectric bending actuators are
discussed.
(26) These findings are shown to be in excellent agreement with previously reported
theoretical predictions by the authors.
(27) Three specimens are used in the study: a block of Plexiglas that has a linear
attenuation, a layer of a special rubber compound with an attenuation proportional to
f1.38, and a phantom made of castor oil that has an attenuation proportional to f1.67.
(28) Some important factors are studied for the bimodal ultrasonic motor design.
(29) The composite disks have been fabricated into transducers with air-backing and with
no front face matching layer, and their performance characteristics have been evaluated
in water.
(30) It was found that the ~680 µm spot size of the experimental zone-plate did not vary
appreciably with changing frequency, whereas the focal length increased markedly with
increasing frequency (from ~5 mm at 450kHz up to ~15 mm at 900kHz).
(31) Tissue-characterization parameters, which have been used successfully by other
authors, were calculated for each segment.
Examples (23) to (31) illustrate the nine different criteria I used for N2 classification. Sentences
(23), (24) and (25) are examples of the use of the, a, and A. (26) features a deictic (these). The N2's in
(27) and (28) are both determined by quantifiers either numerical or not. In (29), the N2 is determined
by a possessive adjective. (30) and (31) both have a pronoun as an N2.
Beyond this simple labelling, a further classification is necessary. Each of the above determiners
(and for the purpose of this study, I will consider pronouns as having an built-in determination)
contributes to a thematic or rhematic environment. N2 determination therefore participates in text
cohesion and in the use of technical terminology.
In this part I shall study N2 determination with relation to the abstract's IMRaD parts but also to Vten
categorisation as shown above.
Table5 provides an overview of determination patterns in the introductions of abstracts in
ABCORP.
Introduction
A 20% (14)
a/an 35% (25)
19 The restrictions in the categorisation of verbs resulted in the exclusion of 33 verbs from the study.
503
The 38% (27)
Others 7% (5)
(71)
Table 5
Zero-determination (marked A) is seen by Adamczewski (1993: 210) as a direct reference to the
extralinguistic universe20 with no thematisation. It occurs in 14 of the 71 passive sentences in the
Introduction sentences in ABCORP. Eleven sentences feature verbs from the first category defined
above (i.e. with a broad semantic content). Example (32) below is typical of zero-determination.
(32) Quasi-monolthical integration of thin GaAs/InGaAs/AlGaAs-quantum well
structures on LiNbO3 SAW devices is achieved using the epitaxial lift-off (ELO)
technique.
By giving the chemical composition of the components, the speaker can hardly be any closer to the
extralinguistic universe. Other N2's in this category are technical terms which are absolutely central to
UFFC such as deformation, acoustic fields, electromechanical coupling mechanisms, piezoelectric
bending actuators, piezo helical springs, curved unimorphs, or piezo springs or concepts derived from
such terms such as Improvement of sensitivity in ultrasonic fields of piezocomposite transducers.
The information is apparently not proposed as given but as new which may seem incompatible with
a passive sentence in which the grammatical subject is proposed as given. I shall address this apparent
conflict of interests further below.
Although only three sentences in the Introduction section of abstracts in ABCORP feature a “A N2”
sequence and a category 2 verb (i.e. a verb describing a scientific or technical process) it is interesting
to notice that the observation made above does not seem to apply to these three sentences. Their N2's
are small cracks, known defects, previous studies which are indeed technical terms but not in the
domain of UFFC.
In ABCORP, 25 sentences featuring a passive as part of the introduction of an abstract also feature
a/an as a determiner for the N2 of the sentence. Using a/an enables the speaker to refer to a specific
occurrence of a notion therefore limiting its scope. A/an also conveys minimal classification.
Adamczewski (1993: 213) gives the following example:
- Let's not talk about all this. It was only a dream.
The speaker recognises the reference of this as being “an occurrence of a dream” and classifies it as
such. Using a/an is a twofold operation which first entails acknowledging the existence of a reference
and then classifying it albeit minimally. The operation sets the noun group in Adamczewski's phase 121
i.e. a rhematic phase. The classifying aspect of the operation is of course particularly interesting for
anyone concerned with the technical terminology of a given area of specialisation. A/an therefore
introduces the speaker's conception of the noun or noun group it determines. He refers to the
prototypical occurrence of a notion to acknowledge the existence of his own occurrence of the notion.
This appears quite clearly in this subsection of ABCORP in which six of the 25 N2's feature the
adjectives new or novel as does example (33) below.
(33) A new numerical model of a short-term stability measuring system of quartz crystal
resonators is presented.
However, the main characteristic of the N2's extracted from this subsection of ABCORP is their
extreme categorisation as shown in the following three examples
(34) A resonant liquid capillary wave theory which extends Taylor's dispersion relation
to include the sheltering effect of liquid surface inclination caused by air flow is
presented.
(35) A condition monitoring nondestructive evaluation (NDE) system, combining the
generation of ultrasonic Lamb waves in thin composite plates and their subsequent
detection using an embedded optical fiber system is described.
(36) In this paper a novel ultrasound tomography imaging system is presented.
20 The absence of any visible or audible morpheme for this determination is symbolic of the lack of
intervention of the speaker on the extralinguistic universe.
21 Adamczewski has defined a system of 2 phases for sentences. Phase 1 is rhematic or nonpresupposing
whereas phase 2 is considered thematic or presupposing. Typically, a sentence in the
simple present or simple past will be in phase 1 whereas one featuring the be + V-ing aspect will be in
phase 2.
504
(34) and (35) show several levels of embedded clauses and (36) exemplifies heavy noun
premodification. All three N2's thereby reach a level of classification which seems hardly compatible
with the rhematic operation described above. I shall try to reconcile these two aspects further below.
Eight sentences from this subset feature verbs from the second category however, only four
different verbs are used (develop, fabricate, construct, and design). Their N2's seem to have
characteristics similar to those of category 1 verbs.
Having said that the grammatical subject of a passive sentence is chosen as the starting point or
theme of the passive sentence and is therefore considered as given information as opposed to new
information, it should come as no surprise that a majority of the N2's in the passive sentences in
ABCORP are determined by the. Adamczewski's views on determination using the are particularly
helpful. “The indicates that the noun group is in phase 2. The relation is presented as thematic by the
utterer.” (1993: 215 [translation]) He gives the following example:
- Mother, did I ever tell you? I am lucky !
- “ No, you never did “, said the mother.
The first occurrence of “mother” is a direct reference to the person the sentence is directed to.
However, the second occurrence , “does not refer to the partner in the conversation but to the mother
already mentioned.” This shows the discourse has moved a step further away from the extralinguistic
universe. It underlines the role of the speaker who filters the extralinguistic universe. Adamczewski
refers to “distanciation from phase 1”. He further defines the notion when he deals with the ± generic
effect of the the operator (1993: 218 [translation]):”The signals that the utterer somehow works on the
notion's semic programme. We no longer have a truthful image of reality but a filtered one; there is a
discrepancy with the extralinguistic universe. The relative alteration of the notion's semantics is
modulated by the context and the situation.” These few lines are of an even greater relevance when
studying scientific discourse. The specific operation of determining with the, and therefore of filtering
the extralinguistic universe, is in total agreement with passivisation as described earlier. The speaker
distanciates himself from the extralinguistic universe, he handles discourse objects to distribute
information. Adamczewski (1993: 218 [translation]) offers the following summary: “The is related to
phase 2 and enables the utterer to play on notion extension depending on his intentions and the
requirements of the context.”
Just under 40% of N2's in the Introduction section of abstracts in ABCORP feature the as their
determiner. This represents 27 sentences out of 71 of which 24 feature category 1 verbs. Here are two
examples.
(37) The time-frequency distribution (TFD) of Doppler blood flow signals is usually
obtained using the spectrogram, which requires signal stationarity and is known to
produce large estimation variance.
(38) The optically pumped cesium beam clock named Cs~IV is operated with a new
short Ramsey cavity satisfying strict requirements on the microwave leakage level.
The starting point of those sentences has been selected and is presented as such by the speaker.
Through the use of the passive, he has chosen as N2 a noun group which he reckons is understood by
the discourse community he is targeting. He has made of the understanding of the N2 a defining
characteristic of his discourse community. In (37), the speaker's intervention is further underlined by
the use of the abbreviated form in brackets stressing his endorsing of the terminology. In (38), the same
intervention is made even more explicit through the embedded “named Cs~IV” thereby fixing Cs~IV as
a term in a highly definitory context. In the 27 sentences, abbreviated forms appear seven times.
In ABCORP, this set-up highlights such terms as longitudinal leaky surface waves (LLSW), crosscorrelation
method (CCM), coherent population trapping (CPT), Doppler angle, direct digital
frequency synthesizers (DDFS), digital-to-analog (DAC) output, surface acoustic waves (SAW),
piezoelectric leaky surface waves, among others.
The operation is further emphasised by the category the verb belongs to. By having a very broad
semantic content, the Vt-en actually reveals very little of his N2.
In ABCORP The, a/an and A account for more than 90% of N2 determination in passive sentences
from Introduction sections. The remaining ten percent (five sentences) are occurrences of deictics,
quantifiers, relative pronouns or possessive adjectives. They do not seem to have any statistical
relevance at this point of the study.
Although I have clearly stated above that the passive contributes to promote the grammatical
subject of the sentence to the position of theme, of the three operators studied so far, the is the only one
to apparently follow suit.
505
In fact, A/an and A seem to do quite the opposite — that is put forward new information. Still they
are heavily used throughout the introduction of abstracts in ABCORP. Example (39) may offer clues as
to the actual value of this combination.
(39) A new numerical model of a short-term stability measuring system of quartz crystal
resonators is presented.
This is the first sentence of an abstract entitled “Modeling of a short-term stability measuring
system of quartz crystal resonators.” In itself, this sentence does not bring any more information than
already present in the title of the abstract. Its purely informative content is null. I believe the role of
such a sentence is therefore not linguistic per se but metalinguistic. What the passive promotes is the
operation carried out on the N2 and not the N2 itself. In (39) the author of the abstract acknowledges the
existence of an already extremely complex notion, that of numerical model of a short-term stability
measuring system of quartz crystal resonators and of an occurrence of this notion. His
acknowledgement of the existence of such a notion is a trace of the categorisation operated by the
speaker i.e. one of the building blocks of the experience he is trying to recreate through his text. He is
setting the conceptual boundaries of his discourse.
This metalinguistic feature can be implemented by other means than the passive. Examples (40) and
(41) are also the first sentences of abstracts. They define the conceptual boundaries of the speaker's
experience more explicitly.
(40) Recent research has shown that, for a rotating phantom, the speckle pattern may not
replicate the phantom motion, rather it may show a large lateral translation component in
addition to rotation.
(41) Recent papers have shown that focused ultrasound therapy may be feasible in brain
through an intact human skull by using phased arrays to correct the phase distortion
induced by the skull bone.
I am assuming that the same operation occurs when the N2 of a passive sentence in the Introduction
section of an abstract is determined by A or the. What is promoted is the determination of the N2 not
the N2 itself. Because of the very nature of A, the operation leads to a definition of wider conceptual
boundaries hence, in ABCORP, the occurrence of such central terms to UFFC as deformation, acoustic
fields, Electromechanical coupling mechanisms etc. (see list above). Inversely, the conceptual
boundaries set through the use of the are only central to the research article itself. By being promoted
N2's they endorse the role of key concept hence the high density of technical terms occurring as N2 of a
Vt-en in the introduction section of abstracts in ABCORP.
However, the above description of determination is very close to what it would be in a context other
than passive. I have emphasised that the passive is a well-thought reorganisation of an initial pattern for
a purpose. It clearly underlines the intent of the speaker that is to relate the scientific process he has
been through, to recreate it for the benefit his reader. It seems that the passive contributes to bringing
forward the two-way dimension of communication. It does not just enable the speaker to reorganise
and, as shown above, reconceptualise his own experience of the process he wants to relate but it also
calls for the reader's (co-utterrer's) acknowledgement and acceptance of the delimitation/definitions
operated over the extralinguistic universe.
As already mentioned above, the Method sections of abstracts in ABCORP contain more passives
than any other parts. Whereas introductions contain 71 passive sentences, Method sections contain 165.
The breakdown by verb category is also quite different: 84 of the Vt-en's are from category 1, 79 from
category 2 and 1 from category 3. This breakdown confirms that the “quantum of information”, as it is
referred to by Haliday (1994: 34), is not evenly spread over the clause. It shows that, even in a section
in which one expects accounts of procedures and therefore a sizeable amount of material processes, in
passive sentences, a majority of verbs, though a short one, still convey very little of those processes
most of the information being conveyed by their N2's.
A comparison of N2 determination in this subsection of ABCORP and in the one previously studied
also shows interesting results. Those results are summed up in table 6.
Introduction Method
A 20% (14) 24% (40)
a/an 35% (25) 15% (25)
506
The 38% (27) 46% (75)
Others 7% (5) 15% (25)
(71) (165)
Table 6.
When considering zero-determination in passive sentences, the comparison between the
Introduction section and the Method section of abstracts in ABCORP highlights a similar frequency. Of
the 40 N2's determined by A in the Method section of abstracts in ABCORP 21 are subjects of
category 1 verbs and 19 of category 2 verbs. However, the N2's determined by A are quite different
from the ones undergoing similar determination in introductions. In the Method section, when the verb
belongs to the first category, a majority of N2's refers to such concepts as computations, method,
technique, measurements, simulations, or experiments.
When premodified, those N2's take on a more restrictive semantic content than in introductions as
can be seen in examples (42) and (43)
(42) Optical interferometric measurements of pellicle displacement at discrete
frequencies in tone-burst fields are converted to acoustic pressure, and the hydrophone
for calibration is substituted at the same point, allowing sensitivity in volts per pascal to
be obtained directly.
(43) Moreover, to guarantee the convergence of identification and tracking errors,
analytical methods based on a discrete-type Lyapunov function are proposed to
determine the varied learning rates of the FNNI and the optimal learning rate of the
adaptive controller.
As in introductions, the other N2's are also essential concepts to UFFC or more precisely to the
subject of the study being carried out. In (44) the concepts of location and extent are central to the
study describing an application to detect prostatic carcinoma.
(44) For these patients, location and extent of the carcinoma were known from
histological findings after radical prostatectomy.
Quite expectedly, there are fewer occurrences of determination using a/an in the Method sections of
abstracts in ABCORP than in Introduction sections. In general, categorisation seems to have already
been operated. The boundaries the author has set for his research are well in place once he gets down to
explaining his methodology. This is emblematically highlighted by the one and only occurrence of the
sequence a/an new N whereas the same sequence was concordanced in 24% of the a/an determined
N2's of passive sentences introductions.
Still, categorisation does occur in 15% of all passive sentences in the Method sections of abstracts
in ABCORP. It seems however less marked than in introductions. Examples (45) and (46) are
particularly relevant at this point
(45) A line-of-sight optical projection through a test object is identified from an
amplitude null and a sharp phase transition produced by diffusive waves originating
from two in-phase (initial phase 0o) and out-of-phase (initial phase 180o) light emitting
diode sources.
(46) A second air-coupled capacitance detector (apertured to 200 µm) was scanned in
the field of the zone-plate source in order to image the generated ultrasonic field at
various frequencies of operation.
In (45), the author of the abstract only makes use of one prepositional group for postmodification
whereas in examples (34) and (35) three whole embedded clauses are used. In (46) extra information is
provided in brackets whereas it could perfectly have been embedded as a clause.
When in need of detailed categorisation, authors seem to resort to premodification. Example (47) takes
this to great length.
(47) A 0.91Pb(Zn1/3Nb2/3)O3-00.09PbTiO3(PZN-PT) single crystal with high
electromechanical coupling factor (k33)>90% has been used to fabricate a 40-channel
phased array ultrasonic probe with greater sensitivity and broader bandwidth than
concentional probes.
Premodification offers a better integration of the modifier but it also blurs the relation between the
modifier and the head noun making it implicit. The resulting noun group is therefore “denser” and
more is demanded of the reader to understand it correctly. The same is true of abbreviation as used in
example (48).
507
(48) Then, because the dynamic characteristics of the USM are complicated and the
motor parameters are time varying, an AFNNC is proposed to control the rotor position
of the USM.
As already hinted at in the study of a/an determination of N2's in introductions, a “semantically
rich” N2 seems to call for a “semantically poor” Vt-en (category 1 verb). In keeping with the
observations made when studying N2 determination in introductions, it come as no surprise that 15 out
of the 25 Vt-en's extracted in this subsection of ABCORP belong to category 1. Once again, the
quantum of information is very unevenly spread over the sentence. On the one hand the speaker sets
out and acknowledges the existence of highly specialised notions, on the other hand he tells us very
little about them and the actual role they might play in his research. The fact that the sentence is in the
passive emphasises the effect. It makes is clear for the reader that he is expected to fully apprehend the
whole semantic content of the N2 with very little co-textual help. His understanding requires appealing
to his very own experience of the specialised field. By using the passive, the speaker gives the
impression of setting out the N2's he has selected as the “compulsory checkpoints” in his research.
Whoever would not go through those checkpoints would not understand his research. As mentioned in
the introduction of this paper when dealing with the actual purpose of the research paper abstract, it
appears quite clearly at this stage that through the abstract, the speaker carefully signposts22 the way for
his reader. In this respect, the role of the abstract is very close to that of a table of contents. The
function of the passive in this case seems to be a metalinguistic one pertaining to emphasising
discourse23 cohesion.
This metalinguistic function of the passive is even clearer when N2's are determined by the. In this
case the modifiers and the head noun are sealed into a concept and labelled as a term as in the
Lagrange multiplier method, the dynamic range (DR) of the new TIAOC, the backpropagation
algorithm, which all occur as N2's in the Method section of abstracts in ABCORP. As could be
expected, N2's are less premodified than when determined by a/an and off course by A.
In the Method section of abstracts, the author takes his readers through the successive steps of the
method he implemented. The fact that this section features the highest frequency of passive sentences
but also of N2 determination by the shows that a particular emphasis is put on signposting this section
more than any other. This is confirmed by the very high frequency of category 1 Vt-en's. Although
some of the research described in ABCORP involves extremely precise processes, category 1 Vt-en's
still account for almost half of the Vt-en's in the Method section. For example, such a process as
“sintering”, which is central to piezoelectric ceramics production, does occur in ABCORP but never as
a verb and although “poling” and “prepoling” are essential they only rarely occur as verbs.
The occurrence of N2's as intrinsically “weak” as the model or the experimental data with
category 2 Vt-en's is marginal and can hardly be considered a trend. Example (49), which features a
rather weak N2, would prove the opposite but it being the only occurrence of a category 3 verb in this
subsection of ABCORP, drawing any conclusion would be hazardous to say the least.
(49) Sintered PZT rods are inserted into a prepoled copolymer matrix, and the composite
is repoled under a lower electric field.
A closer observation of the category labelled “Others” in table 6 proves an efficient conclusion to
this part in which I have tried to bring forward the role of the passive as a means for discourse
cohesion. “Others” represent 15% (25 occurrences) of N2 determination in the Method section of
abstracts in ABCORP. Among these 15%, a majority of determiners are deictics (9 occurrences which
represents just above 5% of the total for Method sections) all of which are occurrences of this or these.
With this or these as a determiner, the N2 is introduced as rhematic with a reference to the
extralinguistic universe. Example (50) features the only case of a premodified head noun in an N2 from
this subsection of ABCORP.
(50) By using an extension of the slowly varying functions method, this differential
equation is transformed into a nonlinear differential system with perturbation terms as
the right-hand side.
The other N2's are this method, these methods, these parameters, this analysis, these data, these
devices, this error, this line-of-sight. All trigger sentences developing right of the verb group and
strengthen discourse cohesion.
22 “signposting” is a term used by Salager-Meyer (1990) and quoted by Gledhill (2000: 42)
23 At this stage in the study, through the use of discourse rather than text as first announced in the title
of the paper, I intend to emphasise the role of the speaker in the production of the abstract. This goes
against the idea that the abstract would result from the mere selection of key points (usually on
methods and results) by the author.
508
In the above paragraphs, I have tried to show that the operation carried out by the passive in the
Method section of abstracts in ABCORP is intrinsically identical to the operation carried out by the
passive in the Introduction section of abstracts in ABCORP. However, there being a major “purposeshift”
between the introduction of an abstract and its Method section and the two having distinct
“communicative functions” the effect of passivisation can be quite different. In the Method sections
most passives seem to highlight the key points of the research conducted. These points have been
selected by the author to signpost his abstract.
It is usually made more or less explicit, in the instructions for contributors, that abstracts should
state at least the method used for the study and the results achieved. Still the Result section of abstracts
in ABCORP is not as overtly marked by passives as the Method section as table 7 shows.
Introduction Method Results
A 20% (14) 24% (40) 31% (28)
a/an 35% (25) 15% (25) 17% (14)
The 38% (27) 46% (75) 31% (28)
Others 7% (5) 15% (25) 21% (19)
(71) (165) (89)
Table 7
Because of yet another “purpose-shift” from the Method section to the Results section, operations
carried out by the speaker, although similar, take on a different interpretation.
Zero-determination occurs for 31% of N2's in the Results section of abstracts in ABCORP, which is
the highest frequency for this type of determination. However, the N2's determined by A are quite
different from the ones in the previous subsections. Indeed, this subsection of ABCORP features ten
occurrences of results, data to which can be added occurrences of such N2's as measurements,
comparison, or performance. (51) and (52) are two examples:
(51) Data are given showing the results of using the linear quadratic Gaussian (LQG)
technique to steer remote hydrogen masers to Coordinated Universal Time (UTC) as
given by the United States Naval Observatory (USNO) via two-way satellite time
transfer and the Global Positioning System (GPS).
(52) Results are presented demonstrating that the new method has both satisfactory
tracking performance and the potential for practical real-time implementation.
These two examples show the cataphoric function of the N2 and the sentence developing right of the
verb group. The N2's play a role identical to the one played by deictics in the previous part. They do
take on a metalinguistic function and strengthen discourse cohesion. It is no coincidence to find nine
cases of determination by deictics and four occurrences of a cataphoric it as an N2, also in this
subsection of ABCORP. In this perspective, the role of the N2 and hence of the passive in (51), (52)
and (53) is certainly similar.
(53) It was found that the ~680 µm spot size of the experimental zone-plate did not vary
appreciably with changing frequency, whereas the focal length increased markedly with
increasing frequency (from ~5 mm at 450kHz up to ~15 mm at 900kHz).
The structure seems even more efficient with category 1 verbs, which is the case in 21 of the 28
passive sentences of this subsection.
The technical terms determined by A in the Results section of abstracts in ABCORP are usually
more specific to the study than to UFFC in general.
The proportion of N2's determined by a/an in this section is identical to the previous section and
quite similarly the Results section does not seem to be the place to either categorise or introduce new
notions.
There are 28 N2's determined by the in this subsection of ABCORP ; 15 are grammatical subjects of
category 1 verbs and 13 of category 2 verbs. What differs at this point of the study is the nature of the
N2's. Only 13 of them are actual specialised terms and most of the N2's refer directly to their co-text.
As examples (54) and (55) show, specialised terms can take part in the N2 but are no longer the head
nouns of the N2.
(54) In addition, the effectiveness of the adaptive fuzzy-neural-network (AFNN)
controlled USM drive system is demonstrated by some experimental results.
(55) The resulting improvement in radial and lateral blind deconvolution is demonstrated
on six short ultrasound image sequences recorded in vitro or in-vivo.
509
From the Method section of abstracts in ABCORP and the Results section the “purpose-shift” does
not seem to be as marked as it is between the Introduction section and the Method section. Therefore
the effects of the passive in Method sections and in Results sections bear similarities and emphasis is
put on text cohesion. Few specialised terms need to be introduced at this late stage of the study. They
tend to be put in relation with another noun as part of the N2 rather than head the N2.
As Table 8 shows, only 24 passive sentences occur in the Discussion section of abstracts in
ABCORP, which barely represents 7% of passive sentences in the whole corpus. Very little conclusive
evidence can actually be drawn from such a small sample I shall therefore only underline that by
definition, the Discussion section of an abstract is the part of the study the reader cannot anticipate on.
The bulk of the informative content is provided by the author. The effects conveyed by the passive do
not seem to match the communicative function of this section.
Introduction Method Results Discussion
A 20% (14) 24% (40) 31% (28) 21% (5)
a/an 35% (25) 15% (25) 17% (14) 12% (3)
The 38% (27) 46% (75) 31% (28) 50% (12)
Others 7% (5) 15% (25) 21% (19) 17% (4)
(71) (165) (89) (24)
Table 8
6. Conclusion
As underlined in the introduction to this paper, the abstract is often considered as a “miniversion” of
the research paper and should be “self-contained.” The IMRaD sections of the research paper also
characterise the abstract and each section participates in a rhetorical development. Throughout this
study I have tried to show that far from betraying stylistic shortcomings, the use of the passive in
research article abstracts participates in this rhetoric. I have first established that the passive is not a
mere mechanical transformation of a hypothetical primary active sentence but a clear and well-thought
reversing of an original pattern (predicable lexis). I have then underlined two effects of the use of the
passive. The passive first enables the author to set out the terminology he intends to use. In this respect
and with regards to the consistency of use of the passive, I have tried to bring forward the passive as
another example of “the way phraseology helps to shape a specific view of transitivity at the same time
as framing terms stereotypically” as underlined by Gledhill (2000, 167). The passive as staging
technical terminology is particularly present in the Introduction section of abstracts.
In abstracts, the passive also contributes to signposting. It clearly marks the compulsory
“checkpoints” the reader has to go through in order to understand the abstract. In this role, the passive
strengthens discourse cohesion., which is particularly noticeable in the Method and Results sections of
abstracts in ABCORP. Salager-Meyer (1990: 37824) has underlined the lack of cohesive devices in
“unsuccessful abstracts” and the remark certainly applies to ABCORP. However, when lexical
cohesive devices occur, they tend to do so in active sentences or to co-occur with “weak” N2's. Quite
logically, Also is the link-word occurring most frequently in passive sentences; it introduces a rather
neutral link unlike more marked link-words such as thus or therefore. Table 9 shows the results of a
concordance of also over ABCORP.
Also
In active
sentences
In passive
sentences
With a semantically With a semantically
24 As quoted in Gledhill (2000: 42)
510
weak subject strong subject
Number of
occurrences
13 7 5
Type of N2
§ Experimental results also
are presented
§ […] the same data also can
be averaged
§ Data also are shown […]
§ It is also substantiated […]
§ This improvement also can
be observed […]
§ The numerical results also
are compared to […]
§ Also, it was observed that
[…]
§ A dispersive perfectly
matched layer (DPML)
boundary condition, which is
suitable for boundary
matching to such a dispersive
media whole space, is also
proposed […]
§ Modern beam forming
techniques such as
apodization, dynamic aperture,
elevational focusing, multiple
transmit focusing, and
dynamic receiving focusing
also can be simulated.
§ […] the free and soft
planar baffle also can be
considered.
§ Good image
reconstructions based on
simulations and real objects
also are provided […]
§ […] negative Doppler
shifts also are produced.
A general overview of the
theory behind the LQG
technique also is given.
Table 9
In this study, I have categorised verbs occurring in passive sentences. The categories I have
proposed are based on the semantic content of the verbs. They somehow differ from the categories set
out by Gledhill in his study of the phraseology of cancer research papers. Gledhill (2000: 213) defines
four main process types: research, empirical, clinical and biomedical. He points out that “these four
dimensions form a continuum in which they represent the relative involvement of the author in
scientific activity (either in experiment or writing up).” I have tried to show that the passive is one of
the clues of this involvement, my category 1 largely overlapping Gledhill's research and empirical
ones.
The categorisation of N2's I have proposed, based on their determination, has underlined the lexical
cohesion of discourse through labelling (Francis’ term 1994). N2's become grammatical metaphors
contributing to, in Gledhill's words (2000: 204) “the distribution of thematic roles within the clause and
at the same time [being] a key mechanism in the construction of new meanings.”
In keeping with Gledhill's (2000: 166) and Nwogu and Bloor's (199125) findings, the metalinguistic
role of the passive contributing to strengthen discourse cohesion confirms that “abstracts tend to
employ simple thematic progression, linearly converting rheme to theme.”
Authors are often unaware of whether they “should” use the passive or not. This study has tried to
show that the passive is used with great consistency in research paper abstracts through which
“grammatical collocations” (Gledhill's term) have emerged. This would favour the existence of what
Gledhill terms (2000: 167) a scientific “voice”.
Acknowledgements
The author wishes to thank Professor M. Clay from Lyon III University for his suggestions. This
study was supported by INSA-Lyon's ESCHIL, Centre for Humanities, and Department of Electrical
Engineering.
References
Adamczewski H 1993 Grammaire Linguistique de l'Anglais Paris: Armand Colin
25 As quoted in Gledhill (2000: 166)
511
Biber D, Conrad S, Reppen R 1998 Corpus linguistics. Cambridge, Cambridge University Press.
Booth V 1993 Communicating in science. Cambridge, Cambridge University Press.
Brown G, Yule G 1983 Discoursed Analysis. Cambridge, Cambridge University Press.
Couper-Kuhlen E 1979 The prepositional passive in English. Tübingen, Niemeyer.
Day R 1989 How to write and publish a scientific paper. Cambridge, Cambridge University Press.
Francis G 1994 Labelling discourse: An aspect of Nominal Group Cohesion in Coulthard (ed),
Advances in Written Text Analysis. London, Routledge, pp 83-101.
Gledhill C 2000 Collocations in Science Writing. Tübingen, Gunter Narr Verlag.
Granger S 1983 The be + past participle construction in spoken English. Amsterdam, North Holland.
Groussier ML & Riviere C 1996 Les mots de la linguistique, lexique de linguistique énonciative. Paris,
Ophrys.
Halliday MAK 1994 An Introduction to Functional Grammar. London, Edward Arnold.
Joly A & O'Kelly D 1990 Grammaire Systématique de l'Anglais Paris: Nathan
Lapaire R. & Rotgé W. 1991 Linguistique et Grammaire de l'Anglais. Toulouse, Presses Universitaires
du Mirail.
Lobban C & Schefter M 1992 Successful lab reports. Cambridge, Cambridge University Press.
Nwogu KN & Bloor T 1991 Thematic progression in professional and popular medical texts. In
Ventola (ed), Functional and systemic linguistics: approaches and uses. Den Haag, Mouton de
Gruyter, pp 369-384.
Salager-Meyer F 1990 Discoursal Flaws in medical English abstracts. Text 10(4): 365-384.
Sides C 1991 How to write and present technical information. Cambridge, Cambridge University
Press.
Svartvik J 1966 On voice in the English verb. Den Haag, Mouton de Gruyter.
Tesniere L 1959 Eléments de syntaxe structurale. Paris, Klincksieck.

513
Do women and men really live in different cultures?
Evidence from the BNC
Hans-Jörg Schmid, University of Bayreuth, Germany
In her bestseller You just don't understand Deborah Tannen claimed that “talk between women and
men is cross-cultural communication” (1990: 18). Two years later, Geoffrey Leech and Roger Fallon
(1992) published a paper with the title “Computer corpora – what do they tell us about culture?”. They
showed how the frequencies of words in the Brown and the LOB corpora mirror the importance of
certain concepts in American and British culture. In a note they expressed their hope that “by the year
2000, it will be possible to make use of these corpora [i.e. BNC and COBUILD] for cross-cultural
studies on a much larger scale than is now possible on the limited basis of the Brown and LOB
corpora” (1992: 47). To a large extent thanks to Geoffrey Leech's own contribution to corpus
linguistics, their hopes were not in vain.
If we combine Tannen's claim with Leech and Fallon's method, we arrive at an obvious question: Can
the BNC tell us whether Tannen is right? The paper addresses this question by comparing the
frequencies of words and collocations as used by women and men in the spoken part of the BNC.
Words from the following domains have been investigated:
· personal references (personal pronouns, male and female proper names)
· family
· personal relationships
· home
· food and drink
· body and health
· clothing
· car and traffic
· computing
· sport
· public affairs
· abstract notions
· alleged “women's” and “men's” words
· swearwords
· hesitators, fillers, backchannel behaviour
· linguistic politeness markers
· linguistic markers of uncertainty and tentativeness
· linguistic markers of conversational cooperation and support
The data indicate that Tannen's claim is indeed true, since most of the words and collocations
investigated exhibit significant differences which seem to be gender-determined. On the whole, there is
strong converging evidence that women's speech style tends to be marked by proximity and
involvement, and men's by distance and detachment. This can indeed be interpreted as reflecting some
kind of cultural difference. Nevertheless, not all the distribution patterns are in conformity with what is
suggested in the gender-linguistic literature. The expletive bloody, for example, is much more often
used by women than by men, particularly frequently in fact by women in the 45 to 59 years age band.
(The factors age, social class and education are also taken into consideration but not focussed on.)
Depending on the time/space available, a selection of this data will be discussed in detail. The
rest of the material will be summarised in tables in order to leave room for an adequate interpretation of
the findings.

514
Exploring the Chemnitz Internet Grammar: examples of student use
Josef Schmied
English Language & Linguistics, Chemnitz University of Technology, Germany
1. Research context and design
1.1. Context
The Chemnitz Internet Grammar (CING) is a research tool and a teaching aid at the same time.
The aim of the research project1 is to induce guiding principles for the development of interactive,
learner-specific information retrieval (grammar) programs for the internet and to produce a program
based on aspects of English grammar which applies these principles. The target group of users consists
of advanced learners of English, mainly with German as a mother tongue; thus it concentrates on those
areas of English grammar where substantial differences occur between English and German. The
research results on learners’ behaviour and preferences are always directly re-implemented in the
grammar program, which serves as a learning tool. In contrast to other internet grammars, ours is
essentially a double reference work: a database that enables inductive language learning using authentic
examples, and a description of the grammar that enables deductive language learning using the “rules”
contained within it. In many ways our grammar is based on the complex system of pedagogic grammar
[PG] proposed by Corder as early as 1973, illustrated in his circular figure on PG in teaching materials
with its interdependent four key elements inductive exercises, data and examples, explanations and
descriptions and testing exercises. Through the related exercises and links, the reference work can be
used for university-level language teaching and in-service teacher training.
Generally, our approach can be characterized by the following key-words: learner-centred,
contrastive in the deductive component and data-based in the inductive component, interactive and
learner-adaptive, e.g. providing immediate feedback and correction, in the exercise component.
1.2. Comparison with other grammars in hypertext and bookform
The Chemnitz Internet Grammar can of course be compared with other grammars that were
published recently. The only other empirical and corpus-based internet grammar is the London Internet
Grammar (1996-98), which is however neither contrastive nor has it an inductive component. The last
point also goes for the other grammar books published recently: The Longman Grammar of Spoken
and Written English (Biber/Johansson/Leech/Conrad/Finegan 1999) and Mindt (1999) are also corpusbased
but they are not directly EFL-oriented. Our grammar is written explicity for the foreign language
learner like Celece-Murcia/Larsen-Freeman 1999, which however, in contrast to all the others
mentioned, is not corpus-based. By combining corpus linguistic approaches with an EFL rule-based
grammar we can also make it lear to the advanced learner that most language rules are not absolute but
rather relative and there is a wide (acceptable) range between prototypical and creative constructions.
An important aspect of our deductive grammar (although the grammar corpus can be used to
investigate all grammatical question inductively is that it is not a complete grammar (like Biber et al.
1999 or Celece-Murcia/Larsen-Freeman 1999) but it concentrates on exemplary grammar areas that
make English special in many ways. In the German - English contrastive perspective, it covers three
different areas so far, verbal, nominal and clausal:
· In the tense/aspect/modality section it emphasizes for instance the progressive aspect, which is not
grammaticalized in verbal form at all in German, the present perfect, where the distribution in
German is completely different, and modulation, where the preference is more towards the
adverbial rather than the modal auxiliary construction (Schmied/Schäffler 1996).
· From the noun phrases our grammar discusses prepositions, as complements as well as adverbials,
again since there are interesting differences between the two typologically closely related
languages German and English.
1 The project has been financed by the German Research Association (DFG) since 1998 as part of the
New Media research group at Chemnitz University of Technology (cf. Schmied 1999 for conceptual
and Gorlow et al. 2001 for technical aspects). It also serves as a basis for other e-learning projects. I
wish to thank my collaborators Naomi Hallan, Diana Hudson Ettle, Christoph Haase, Angela Hahn and
Sabine Reich for many interesting and thought-provoking discussions.
515
· The clausal level is represented by relative constructions, where the basic forms are parallel to
their German equivalents but again the distribution is different, and conditional clauses, where the
sequence of tenses is unusually strict in English.
The following presentation concentrates on the specific research aspects of our grammar, i.e.
analysing learner behaviour. It illustrates the sociobiographic questionnaire, the tracking mechanisms
and the first results of the experiments on student behaviour in the CING. It uses the verb phrase as an
example and compares the deductive explorations of ‘grammar rules', the inductive searches for corpus
samples and the work on the exercise component. Many of the results are preliminary, which seems
obvious since the CING is an on-going research project, which will keep us occupied for another four
years.
2. Basic guidelines and assumptions
2.1. Tracking user behaviour
One of the basic assumptions of the CING research project is that different user types use
different learning strategies, depending on age, language skills, exposure, computer literacy other
variables that might have an influence on learning style. That is why our questionnaires cover a wide
range of variables under three lists, general, language-related and computer-related. The influence of
these variable on learning strategies is then compared in a statistical analysis where the sociobiographic
data are compared with the recorded usage data. Of course, one of the central variables for us is the
proportion of pages used in the explanations and the discovery sections. Fig. 1 thus shows that for
many learners grammar is mainly a deductive exercise, only few venture into the inductive section. On
this basis we can explore, for instance, which type of learner uses a relatively high proportion of the
discovery section (which has less than half as many pages as the explanations section).
Figure 1: User diagram
The first results of our tests confirm the tentative conclusion by Yan-Ping (1991: 272) for Chinese
learners of English:
With respect to the more complex properties such as the semantic meanings of the present
perfect, explicit instruction does not show any superiority to implicit instruction. ... A
tentative conclusion can be drawn that explicit instruction is effective with simple rules but
not with complex rules.
Of course, whether learners can actually apply later on in tests what they have “learnt” by looking
at on the screen depends on sociobiographic variables (like computer experience and language skills) as
well as on presentation variables. In this context, the different perspectives on “learning” by specialists
in linguistics, artificial intelligence and psychology have become obvious.
516
2.2. Hypertext advantages and disadvantages
2.2.1. Advantage: user-specific presentation of information
A usually infamous feature of internet presentations is that they are so “evasive”, i.e. they can be
changed quickly and adapted to new conditions or knowledge - or disappear. For some research
purposes it is a great advantage if we can experiment with various types of presentations and thus
measure the “explanatory value” of presentations. Explanatory value here does not mean whether a
linguistic concept or theory can explain all or the majority of the cases but whether a presentation
applies more or less to an advanced language learner and (thus) a better effect on tested practical
language skills (more than theoretical knowledge). For the English tense/aspect complex, for instance,
we have several options: Leech and Svartvik (21994: 150f) have developed a comprehensive overview
of tense and aspect, which can be used as a summary of the topic. The little diagrams there are however
not directly related to the diagrams we used (cf. Fig. 2) because we thought they would be palatable for
advanced learners of English (cf. Hahn/Reich/Schmied fc.). We used Reichenbach's concepts as a
starting point and presented speech time versus reference time as central to verbal time relations. In the
students section we used prototypical examples and even animations to bring the basic ideas across, in
the specialist section we used non-prototypical examples and problem cases (cf. also Schmied 1998).
Figure 2: Speech and reference time
By changing the concepts, in some cases even the (syntax) theory displayed, we can determine the
explanatory value in this sense. We can for instance demonstrate clines and genre distributions as in
Biber et al. (1999) and Mindt (1999) and see whether advanced learners can use them to produce more
prototypical sentences - far beyond the usual right - wrong dichotomy of “rules”.
Mindt (1999: 249) for instance just lists 9 meanings of progressive: incompletion, temporariness,
iteration/habit, highlighting/prominence, emotion, politeness/downtowning, prediction,
volition/intention and matter-of-course (and exemplifies them with prototypical examples), but
afterwards he singles out the first three as progressive meaning in contrast to non-progressive (ibid:
250). This seems a good compromise for showing a cline from more to less central meanings. Although
many students appreciate this type of guidelines, for a corpus-based grammar this approach may be
surprising, since even Swan (1980) emphasises the collocational aspect when discussing the usage the
progressive after certain verb classes and adverbs (s.v. progressive).
517
2.2.2. Advantage: crossreferencing
One of the basic advantages of an internet grammar, apart from the possiblities of recording user
behaviour, is that information can be presented in a hypertext format, where more than one direction of
thought and argumentation can be offered to the reader. Thus cross-referencing from progressive forms
to collocations like stative verbs, prepositions and adverbs, which support or contradict the
“progressive” interpretation of meaning, can show the complex network of grammar more impressively
than any linear description. We had seen it as a main advantage to be able to link the main “branches”
of our internet grammar not only to the “stem” but also to other “twigs” pointing in the same direction.
This also implies “understanding difficult contrasts in tense – aspect combinations”, as they are called
by Celce-Murcia/Larsen-Freeman (1999: 124-128). The idea of the grammar as a system, “the tense
and aspect system” (ibid: chapter title), is essential to foreign learners of English whose first languages
do not have aspect grammaticised in the verb phrase, since the contrast to “alternative” constructions
has to be referred to constantly.
2.2.3. Disadvantage: orientation and navigation
One of the well-known problems of the internet is that its users tend to “get lost in cyberspace”.
Our grammar is no exception to this rule, although references outside of the (so far) five hundred pages
are few. Although we had clear frame signals which section the learner was in, explanations/discovery
or student/advanced (cf. Fig. 2 above), although we provided a bookmark option and an index, our
students obviously found orientation and navigation difficult in the hypertext maize we had created,
where “one could not even quote page numbers”. So we had to put in a user history for learners who
wanted to trace back visually what they had read during their current session, a tree diagram to show
the structure of the section and a complex reference system on every page for those who wanted to
quote certain pages.
2.2.4. Disadvantage. self-contained pages
One of the major claims of the Internet Grammar had been that it facilitates cross-referencing, but
the non-linear structure also has clear disadvantages, since web pages have to be more or less selfcontained.
Thus a headline like “Why is there a choice?” is, of course, ambiguous if one does not get
there directly from a page that has engraved it deeply into the reader's mind that we are talking about
the contrast between simple and continuous tenses. In the index a simple headline like this could refer
to many “crossroads” in a grammar.
3. Research results and their implications
3.1. Writer - user interaction
One of the basic drawbacks of grammar writers has always been that it has been very difficult for
the grammar writer to anticipate the needs and learner strategies of the user. This is a special case of
expert - layman interaction although, of course, at least in an EFL context a certain grammatical
awareness (in a sense of familiarity with linguistic thinking and terminology) can be assumed in a
foreign-language context (in contrast to British schools where it has been deplored for a long time).
In our internet grammar we can not only discuss with our students to gain qualitative information
from actual users and distribute questionnaires to collect more quantitative data - both measurements
are relatively subjective and have to take into consideration that even if students actually knew what
they are doing their answers might be adapted by the need to make a good impression with the
grammar teachers. With the help of computers two more objective ways of measuring how students use
grammars can be applied:
· In off-line experiments we can use an eye tracker to record how students read texts and how they
gather information (in this case on grammar) from a webpage. Here hesitation phenomena may
indicate problems of terminology, a negative interpretation, as well as inspiring thoughts, a
positive interpretation.
· For on-line recording we have developed a special procedure which records all the URLs that
were called up, the reading time, the user input and the user ID. The recorded data in their raw
format are not easy to interpret, thus we wrote a special software programme to make the figures
of the individual sessions more palatable for the interpreter. Through a different little programme
the data of individual users are transferred into an SPSS file so that a comparative quantitative
analysis can be undertaken.
518
3.2. Terminology: boring stuff - catchy language?
One of the great prejudices against grammar is that it is boring. In writing our grammar we
therefore experimented with more “adapted”, user- or age-group-specific language, such as “Is it
simple: be progressive, use the continuous” (Fig. 3), which we thought would be appropriate for the
students. This headline indicates at least three ideas.
· Although there is an important difference between English and German in the use of the
progressive it is not difficult.
· The use of the progressive or continuous forms is a modern feature which is part of grammatical
change and progress in British English.
· Progressive and continuous are two terms that indicate more the function and the form
respectively but ultimately refer to the same thing.
Most of these thoughts or even innuendoes were lost on our students. They considered grammar
writing a serious business and were not trapped by the “young” language at all. They rather considered
some of it inappropriate and distracting. This indicates that witty grammar writing at least on advanced
topics is only something for the very advanced user. Thus, catchy language seems to be more
appropriate for the expert to expert communication than the expert to layman communication.
A further problem with this style is consistency. Thus have not been able to come up consistently
with witty headlines or direct reader-specific questions and relapsed back into the traditional headlines
like “Verbs of state and mental states”, “Verbs of bodily perception”, etc. - and this tradition was not
too bad sometimes after all.
It has to be remembered that consistency in grammatical terminology seems very desirable from
the learners’ point of view. Whereas applied linguists often play with overlapping concepts, students
struggling with concrete problems only find it confusing when “similar” ideas can sometimes be found
under the heading “continuous” so other time under “progressive” (thus most grammars rightly use
only one term as central).
Figure 3: Be progressive
519
3.3. A bilingual approach?
One result of our bilingual basis and contrastive analysis was that we also used the translation
database to exemplify meaning in the explanations section and in the exercises (Fig. 4)2.
Unfortunately, German students very often looked at the German equivalents first; that means that
the old notion of the monolingual teaching tradition is challenged, when the students suggested that - in
particular in such cases where there is a marked contrast between the first language German and the
second language English there would be very useful to look at the translation equivalents. Maybe,
however, the students should not be given the easy option of the German equivalents? In an internet
grammar one can of course change the presentation for a certain group of users and investigate whether
the test results or the reactions are significantly different.
4. Outlook: cultural and tutorial systems
In the second phase the CING will be expanded into a more diversified and comprehensive
learning tool (and thus research instrument). In this context more diversified means that learners in the
grammar will be given more choices or that different learner groups will presented with different, more
appropriate versions. More comprehensive means that learners will not only be given grammar sections
to read but complete authentic texts, so that they can also practise and we can also measure the
comprehension of lexemes, idioms and cultural conventions. This will be achieved by introducing more
“distanced” texts in the learning tool: texts from non-European English-using cultures like East Africa,
where English is used as a second language and has to be adapted to the specific environment and
sociocultural conventions.
Figure 4: MC-EXERCISE
We will not only present new texts but also new learning aides such as a culture-specific lexicon
that enables the non-initiated user to acquire not only grammar, but also “culture”, as it is encoded for
instance in such contrastive concepts like askari (Kiswahili for watchman/policeman) or jui kali (for
“under the hot sun”, i.e. informal) sector. It should be mentioned here that the verb phrase also include
2 This figure also serves as an illustration of our (tradtional) multiple choice questions and the reactions
generated automatically by our system. After completing the exercise students can receive a summary
of their results and recommendations for further study.
520
culture-specific variation. The expansion of the continuous forms to stative verbs in cases like it is
costing a lot is very common feature in second-language varieties of English in Africa and Asia and
has mainly intralinguistic reasons. The reduced usage of modal verbs can be seens as culture-specific,
most second-language users would consider it exaggerated to use too many Would you be so kind
to/Could you please ... -forms.
With the expansion of the internet anyone in Europe for instance has access to East African
newspapers on the internet even before the direct readership in East Africa can hold them in their
hands. This raises the more general question how cultural outsiders cope with culture-specific
expressions and our research could make a contribution to fast-growing area of intercultural
communication. In order to measure that we also intend to develop new test methods for the internet,
such as
A second new development could be user-specific tutorial systems. Obviously many users have,
initially at least, found it difficult to use the system efficiently, as we can see from the tracking records,
their help calls and their interview comments. Thus the option of a “guided tour” for newcomers seems
appropriate. Although this offers new research perspectives, we have to bear in mind that this also
skews our original research, which concentrated on monitoring unrestricted user choice according to
individual learning strategies and initial “guidance” influences personal learning styles. For many
learning systems however tutorials are a decisive factor, therefore gathering information on their usage
and influence only adds a modern dimension to the old problem how grammars are used. There are of
course different tutorial systems possible for the Internet Grammar:
· A straightforward walk-through simply illustrates what is written in the existing guidelines
anyway and animation is generally more appreciated by internet users then running texts. The
linear sequence of slides like the well-known PowerPoint presentations can be implemented
without much effort.
· A more interactive tutorial would be more in line with the original principles of the CING. Such a
tutorial uses learner input either immediately or from earlier sessions to select appropriate
versions of grammar sections. The effect of a machine recommending “You need to practice more
PROGESSIVE TENSES” suggests objectivity and urgent need to many students.
· A fully adapted tutorial includes much more: learner-specific information from the
sociobiographical questionnaire and results from placement tests etc. Here even inconsistent input
can be identified and learnertype-specific information , such as successful patterns of learning
strategies can be included so that a very diversified and modular system evolves. This is a major
step towards fully autonomous learning systems without a personal tutor.
This brief outlook shows that the learning system developed under to name CING can make a
more general contribution to the development of our society, which is said to be moving towards a new
internet and knowledge society. Whether modern human beings that use an internet-based system like
the CING really appreciate this development is not clear, but we can at least indicate in a very limited
area where usergroup-specific preferences tend to lie.
References
Biber D, Johansson S, Leech G, Conrad S, Finegan E 1999 Longman grammar of spoken and written
English. London, Longman.
Celce-Murca M, Larsen-Freeman D 21999 The grammar book. An ESL/EFL teacher's course. Heinle
& Heinle.
Corder S P 1973 Introducing applied linguistics. Harmondsworth, Penguin.
Gorlow E, Haase C, Hallan N, Hudson-Ettle D, Schmied J 2001 Internet Grammar. Technical
Documentation. http://www.tu-chemnitz.de/phil/InternetGrammar/manual.html
Hahn A, Reich S, Schmied J fc “Aspect in the Chemnitz Internet Grammar”. ICAME Proceedings
Freiburg 1999. Amsterdam, Rodopi.
Leech J, Svartvik J 21994 A communicative grammar of English. London, Longman.
London Internet Grammar 1996-98 http://www.ucl.ac.uk/internet-grammar/home.htm
McDonough, Steven H. 1999 Learner strategies. Language Teaching 32: 1-18.
Mindt D 2000 An empirical grammar of the English verb system. Berlin, Cornelsen.
Schmied J 1998 To choose or not to choose the prototypical equivalent. In: Schulze, Rainer (ed.).
Making Meaningful Choices in English. On Dimensions, Perspectives, Methodology, and Evidence.
Tübingen, Narr, pp 207-222.
Schmied, J 1999 Applying contrastive corpora in modern contrastive grammars: the Chemnitz Internet
Grammar of English. Hasselgard H, Oksefjell S (eds), Out of corpora. Studies in honour of Stig
Johansson. Amsterdam, Rodopi, pp 21-30.
521
Schmied J, Schäffler H 1996 Approaching translationese through parallel and translation corpora. In:
Synchronic Corpus Linguistics. Papers from the 16th international conference on English language
research on computerized corpora (ICAME 16), Toronto 1995. Amsterdam, Rodopi, pp. 41-56.
Swan M 1980. Practical English usage. Oxford, Oxford U.P.
Yan-Ping Z 1991 The effect of explicit instruction on the aquisition of English grammatical structures
by Chinese learners. James C, Garrett P (eds). Language awareness in the classroom. London,
Longman, pp 254-277.

522
Is it Creole, is it English, is it valid? Developing and using a corpus of
unstandardised written language.
Mark Sebba and Susan Dray
Department of Linguistics and Modern English Language, Lancaster University,
This paper will present our experiences of developing and using two computer corpora of written
Creole. By ‘Creole’ here we mean English-lexicon creoles of Caribbean origin, which are also used in
Britain. Creole, both written and spoken, is unstandardised and subject to a high degree of variability at
all levels – especially in grammar, phonology and orthography - because of its relationship with
Standard English.
The corpora we will discuss are part of an on-going project at Lancaster University, investigating the
practices of writers using Creole. They are:
· the Corpus of Written British Creole (CWBC), a collection of texts written wholly or partly in
Creole by West Indians whose formative years have been spent in Britain
· a Corpus of Written Jamaican Creole (CWJC), a collection of texts of diverse types written by
Jamaicans in Jamaica
We will briefly discuss issues which arose when setting up the corpora, in particular the practical issue
of the identification of texts for inclusion, and the specific problems that this raises with respect to
English-lexicon Creoles. Can a text containing Creole features be included in the corpus if the writer's
target appears to be Standard English? Where Creole occurs together with Standard English, to what
extent is it necessary to include the Standard English (SE) parts in the corpus – and where is it
legitimate to make a ‘cut'?
We then focus on some potentially complex linguistic questions which arose during the annotation
procedure, e.g. to what extent do graphological features, such as punctuation, layout, and the use of
upper and lower case letters in a text, form part of a ‘naturally’ developing orthography? Is it
important for these features to be indicated in the corpus? To what extent do an individual's literacy
skills affect what is written, and what are the implications of this for the representativeness of the
corpora?
We will end with some remarks on the potential of these corpora as a research tool in order to illustrate
the application of the annotation method. Linguistic items and their orthographic representations can be
compared and contrasted with variables such as demographic information (about writers or voices
within texts), country, text type, and social context. This could provide information, for example, not
only on orthographic variations within or across texts, but also on how writing practices may vary
according to social, cultural, generational and educational factors as well as over time.

523
Language model adaptation for highly-inflected Slovenian language in comparison
to English language
0LUMDP6HSHV\0DX.HFDQG=GUDYNR.D.L.
University of Maribor, Faculty of Electrical Engineering and Computer Science
Smetanova 17, SI-2000 Maribor, Slovenia
mirjam.sepesy@uni-mb.si
Abstract
The language model in this article is meant as an information source of a speech recogniser. The
environment to which the recogniser will be put is topic-specific. The idea is to try to adapt the model of
general language to the target domain of discourse. We concentrate on feature extraction process devoted to
highly-inflected languages. Results of experiments on English and Slovenian corpora are reported.
1. Introduction
A language model aims to provide a representation of language. We concentrate on statistical aspects
of modelling and not on grammatical ones. Statistical models rely on the assumption that the future use of a
language will follow similar linguistic patterns to those used in the past.
Few years ago the problem of statistical language model was defined as the sparse data problem.
Almost all language model research has adopted “bigger is better” approach where enormous volumes of
training text are analysed in order to derive more reliable statistics. However, improvements with size did
not yield much better language models for speech recognition. Training corpus should be used in more
advanced way.
2. Basic language modelling
The task of a language model is to assign the probability ) (W P to every conceivable word string W .
N-gram language models are most widely used (Jelinek, 1998). An N-gram is a model that uses the last N-1
words of the history as its sole information source. Although they are very simple, the experiments have
shown that they are surprisingly difficult to improve on. We use trigram models which restrict the history
to two immediately preceding words
O O
=
- - -
=
» =
n
i
i i i i i
n
i
w w w P w w w P W P
1
1 2 1 1
1
) ( ) ... ( ) ( (1)
We should point out that the word refers to a word form defined by its spelling. Two differently
spelled inflections or derivations of the same stem are considered different words. This fact don't lead to a
problem in modelling English language but in modelling highly-inflected languages. Great number of
different word forms derived from one lemma cause the enormous vocabulary growth. This problem can be
solved by choosing another basic unit instead of word (for example, morpheme) (Byrne et al., 2000). In this
article we were not concerned with the basic language modelling. Deriving trigram and bigram
probabilities is always a sparse estimation problem, probability smoothing was performed by Katz backingoff
algorithm (Katz, 1987).
3. Topic adaptation
N-gram techniques seem to capture well short-term dependencies. They lack any ability to exploit the
linguistic nuances between domains. The environment to which the recogniser will be put is topic-specific.
If we could effectively identify the domain of discourse, a model appropriate for the current domain could
be used. We do not assume the target domain to be equivalent to one predefined topic. Target domain can
be seen as a combination of several elemental topics.
524
The goal of the adaptation is to lower the language model perplexity by providing a higher probability
of words and word-sequences, which are characteristic of the domain of discourse. Adopted models were
built on three semantic levels:
·  general language model (G ). It was built by using all available training text.
·  topic model (T ). It was built by using only the text of a predefined topic most similar to the target
domain.
·  general topic model ( 10 T ). It was built by using the text of 10 predefined topics most similar to the
target domain.
We built two interpolated models:
·  combined model (C )
) ( ) 1 ( ) ( ) ( w P w P w P G C l l - + = . (2)
·  novel model ( N )
) ( ) ( ) ( ) ( 3 2 1 10
w P w P w P w P G T T N l l l + + = (3)
Figure 1 shows the adaptation scheme.
Figure 1: The adaptation scheme.
The adaptation scheme consists of the following three steps:
·  corpus organisation. With the growing availability of textual data in electronic form large
topically diverse corpora are constructed. Stories that share similar topics are gathered together
into a set of clusters.
·  topic classification. A classifier is used to find the clusters that are most similar in topic to the
sample from the target environment.
·  language model building. Language models at different semantic levels are built. The models are
interpolated at the word level.
The first step is corpus organisation. We need the corpus organisation, which enable us to extract
target-topic-similar parts of the whole collection and treat them as more representative. Given a corpus with
keywords assigned to each story, topic clusters are simply created by defining each keyword as a label for a
525
cluster. Unfortunately for minority languages such corpora are often not available. Automatic generation of
clusters of documents need to be used. The success of automatic clustering is conditioned with the quality
of document and topic cluster representation. Before introducing the right representation we will describe
main characteristics of Slovenian.
4. Inflectional morphology of Slovenian
Slovenian is a South Slavic language with a speech area, wedging into the Croatian, Italian, German,
and Hungarian linguistic territories. The morphological complexity of Slovenian in comparison to English
will be described. It is Slovenian inflectional morphology which formally distinguishes both languages.
Using a highly simplified notion, the word formation in Slovenian does not differ much from those
languages where new word forms are created using a stem with the addition of derivational suffixes.
Slovenian morphology introduces three main concepts: word classes, inflection, and grammatical
categories. The main feature of word classes is their division into
·  inflectional classes: substantive words, adjective words, and verb, and
·  non-inflectional classes: adverb, predicate, preposition, conjunction, copula, and interjection.
The basic grammatical categories of Slovenian are: gender, number, case, degree, person, tense, mood
and aspect. Slovenian shares its grammatical categories with other Slavic languages, except the category of
dual in addition to singular and plural, and the Slovenian nominal system does not possess a category to
express an appeal. A match of the grammatical categories and word classes results in different inflectional
patterns. We will describe each grammatical category with examples.
4.1 The category of gender
Slovenian distinguishes three genders: the masculine, the feminine and the neuter. In the majority of
the Slavic languages, gender is inherent in substantives, inflected in adjectives, and not expressed in
pronouns. Slovenian has extended gender to personal pronouns, and marginally to verbal inflections. Table
1 shows some examples.
Masculine Feminine Neuter
Slovenian English Slovenian English Slovenian English
Noun brat brother sestra sister dete baby
Adjective lep pretty lep-a pretty lep-o pretty
Verb dela-l worked dela-la worked dela-lo worked
Pronoun moj my moj-a my moj-e my
Table 1: The examples of the application of gender.
4.2 The category of number
Slovenian morphology uses besides the singular and plural also the dual, when referring to two
persons or objects. The category of number is applied to nouns, adjectives and pronouns. Table 2 shows
some examples.
Singular Dual Plural
Slovenian English Slovenian English Slovenian English
en-a miz-a one table dv-e mizi two tables tr-i miz-e three tables
en-o mest-o a town dv-e mest-i two towns tr-i mesta three towns
lep-a pretty lep-i pretty lep-e pretty
on he onadva they two oni they
Table 2: The examples of the applications of number.
526
4.3 The category of case
The most striking differences between Slovenian and English morphology is the use of six cases in
Slovenian, which denotes the relationship of individual words in sentence. In Slovenian we use different
patterns for the following word classes: nouns, adjectives, and pronouns. The sentences in Table 3 illustrate
the use of cases of the word mesto (Eng. “the town”) and shows the main differences between Slovenian
and English.
Case Slovenian English
Nominative To je mest-o. This is a town.
Genitive Ne vidim nobenega mest-a. I can't see any town.
Dative 3ULEOLaXMHPVHPHVW-u. I'm walking towards this town.
Accusative Kako bi opisal to mest-o? How would you describe this town?
Locative .GRaLYLYWHPPHVW-u? Who lives in this town?
Instrumental Pod tem mest-RPWH.HUHND There is a river beneath the town.
Table 3: The examples of the use of cases.
4.4 The category of degree
The gradation of adjectives and adverbs in Slovenian is quite similar to that in English. As in English,
there are three degrees of comparison in Slovenian. Table 4 shows some examples.
Positive Comparative Superlative
Slovenian English Slovenian English Slovenian English
bel white bolj bel more white najbolj bel most white
star old star-ejši older naj-star-ejši the oldest
Table 4: The examples of the use of gradation.
4.5 The category of person
Verbal forms are related to the three types of the category of person. Table 5 shows the example of the
conjugation of the verb “to work”.
Singular Dual Plural
Slovenian English Slovenian Slovenian English
1st person dela-m I work dela-va dela-mo we work
2nd person dela-š you work dela-ta dela-te you work
3rd person Dela he work dela-ta dela-jo they work
Table 5: The examples of the applications of category of person.
4.6 The category of tense
There are four tenses in Slovenian language. Table 6 shows the use of them.
Tense Slovenian English
Present del-am I work
Past dela-l sem I worked
Future dela-l bom I shall work
Plusperfect dela-l sem bil I had worked
Table 6: The examples of the use of tenses.
4.7 The category of mood
There are three moods in Slovenian: indicative, imperative and conditional. Table 7 shows their use.
527
Slovenian English
Indicative dala-m I work
Imperative del-aj work
Conditional dela-l bi I would work
Table 7: The examples of the use of mood.
4.8 The category of aspect
Every verb obligatorily belongs to one of two classes of aspect: perfective or imperfective. The
contrast between them is expressed not only by different suffixes, but also by a radical alternation of the
stem. Table 8 gives some examples.
Perfective Imperfective
Slovenian English Slovenian English
dvig-n-iti to lift dvig-a-ti to be lifting
se-.-i to reach se-ga-ti to be reaching
pri-ti to come pri-h-ajati to be coming
Table 8: The examples of the use of aspect.
4.9 Morphemic alternations
There is one additional feature of Slovenian. Besides the extensive set of suffixes, words are also
subject to a process of alternation. Two types of alternation are relevant to the written form of Slovenian:
vocalic and consonantal. See Table 9.
Alternation in Slovenian Meaning in English
vet-e-r vetra wind
bolez-e-n bolezni illnes
jo-k-ati jo-.-em to cry
zgu-b-iti zgu-blj-en lost
Table 9: The examples of morphemic alternation.
On the basis of the above description of the morphological structure of Slovenian in comparison to
English, two main points can be emphasized:
·  Slovenian displays features of the extremely rich inflectional morphology;
·  Slovenian is characterized by various types of morphemic alternations in both stems and suffixes
during inflection.
5. Document and topic cluster representation
Given a corpus with keywords assigned to each story, topic clusters are simply created by defining
each keyword as a label for a cluster. Unfortunately for Slovenian such a corpora is not yet available.
Automated generation of clusters of documents based on some similarity measure need to be used.
First we have to find the suitable representation. Documents and clusters are represented as a set of
features. In most applications words are used as features. It has been argued that maximum performance is
often not achieved by using all available features, but using a good subset of those only. Having features
which do not help to discriminate between topics add noise. We want to show that it makes sense to group
features into clusters, at least for languages with rich morphology. We want to group all words with the
same meaning, but different grammatical form, in one cluster and represent them as one feature. We
propose a novel approach for feature extraction based on soft comparison of words.
528
To avoid the use of an additional knowledge source like lexicon we define a set of membership
functions. Each cluster defines its own membership function. The membership function associates to each
word from the vocabulary a number representing the grade of membership of this word in that cluster.
Membership function c~ m of cluster c is defined as
( ) { }. , ~ ~ V w w c c Î = m (4)
c~ denotes a fuzzy set of cluster c . Cluster membership functions are based on fuzzy comparison
function. Each word defines its own fuzzy set
( ) { }. , ~ ~ V w w w w Î = m (5)
The function sees the word as a sequence of characters. It returns value 1 if compared words are the
same and 0 for extremely different words. In other cases it returns the value between 0 and 1. The
comparison function is created by using fuzzy rules, which provide a natural way of dealing with partial
matching. We define three sets of rules:
·  language independent rules,
·  rules describing English language and
·  rules describing Slovenian language.
The rules are expressed as fuzzy implications, which use linguistic variables to express the grade of
similarity (for example: not very similar, quite similar).
To get the impression of language independent rules we present two examples. a denotes the word
with n characters and b denotes the word with mcharacters. The fuzzy implication
similar very not are words different are words of characters . (6)
is transformed into the predicate
î
í
i = $
=
otherwise
b(j) a(i) j:
b a i p
1
0
) , , ( 1 (7)
.
) , , ( ) , , (
) , (
1 1
1 1
1 a a
= = +
+
+
=
n
i
m
i m n
a b i p
m n
b a i p
b a e (8)
The fuzzy implication
similar quite are words same the are words of sequences character two . (9)
is transformed into the predicate
î
í
i + = + U = $
=
otherwise
) b(j ) a(i b(j) a(i) j:
b a i p
0
1 1 1
) , , ( 2 (10)
a a
= = - +
+
- +
=
n
i
m
i m n
a b i p
m n
b a i p
b a e
1 1
2 2
2 .
2
) , , (
2
) , , (
) , ( (11)
The predicates are scaled by linguistic variables. Their values are empirically chosen. The final value
of comparison function is computed using scaling
.
)) , ( max( )) , ( max(
)) , ( max(
) ( ~
b a e b a e
b a e
b
NotSimilar Similar
Similar
a +
= m (12)
Similar e denotes the set of predicates which describe similarity and NotSimilar e denotes the set of predicate
which denote the distinction.
Language dependent rules for English are taken from the suffix stripping set of rules provided by
(Porter, 1980). Language dependent rules for Slovenian describe in simplified form the inflectional
PRUSKRORJ\GHVFULEHGLQ6HFWLRQ3RSRYL.
The membership function of word i w in cluster j c is computed by using a modified single link
agglomerative clustering (Voorhees, 1986). Similarity values of word pairs can be represented as a
weighted, undirected graph where nodes represent words and weights represent the similarity of words
connected by the edge. To save space, we keep only edges with weights greater then a prespecified
threshold. The result of the single link hierarchy are locally coherent clusters. To avoid a chaining effect
and consequently elongated clusters, we modify the merging criterion. A word is added to the cluster if its
average similarity with all words in the cluster is the largest among all the words not yet clustered. Clusters
529
are made one at the time. We start building a new cluster as soon as the largest similarity value does not
exceed a prespecified threshold. Each cluster defines one feature. The number of clusters represents a
feature vector length.
6. Topic detection
Once we have training documents, topic clusters and test samples represented as feature vectors, we
use topic detection to determine the similarity between two feature vectors. Topic detection is performed by
the use of TFIDF classifier (Joachims, 1996). It is used to determine the similarity between two documents
or clusters.
7. Experiments
In our experiments we were using the broadcast news corpus (1996 CSR Hub-4 Language Model) for
English DQGQHZVSDSHUQHZVFRUSXV9H.HUIRU6ORYHQLDQGXHWRWKHLUVHPDQWLFULFKQHVV
The English broadcast news corpus contains 100 mio words. It was organised into topic-specific
clusters of documents based on manually-assigned keywords. We were experimenting with topic clusters
that have at least 300 articles. 244 clusters satisfy this constrain. Language model adaptation was performed
on 20 randomly chosen topics. 80% of each topic cluster text was used for language model training, 10% of
text for interpolation parameter estimation and 10% of text was used as test sample.
All words from the corpus were used for feature extraction. Before word clustering was performed,
words from stop word list were removed. Using language independent word clustering feature vector size
was reduced from 170,000 to 36,000. A sample of clusters is shown in Table 10. It shows that also
misspelled words are correctly clustered.
aadmirable admirable admirably admira admirally admire admired admirer admires admir admirers
bbecause becau becaue
chinasports sports sport sporto sporty sported spotrer sportin sporting sportscar sportsman sportsmen
sportcoat sportless
cilton clnton cinton
conferenced conferences conferencing teleconference teleconferenced teleconferences videoconferences
Table 10: Sample of English clusters.
For each test sample, we want to model, all topic clusters were ranked by the similarity value. If we
used language dependent rules in feature extraction process the top 10 topics didn't change. Four types of
language models were built: general language model (G ), topic model (T ), combined model (C ) and
novel model ( N ). All language models were trigram models with the vocabulary of 64,000 most frequent
words. Results of 5 topics are shown in Table 11. Averaging over all 20 experimental topics the perplexity
of adopted language model was reduced by 15%.
Topic G PP T PP C PP N PP
Automobiles 58 247 55 53
Middle East 55 145 27 27
Clinton, Bill 48 90 46 41
Holidays 62 331 49 48
Simpson, O. J. 45 59 25 23
Table 11: Test set perplexities for English language.
Unfortunately, the Slovenian newspaper corpus of 20 mio words is not yet annotated with keywords.
Clustering was done automatically. Documents were merged into 100 clusters iteratively by the use of
agglomerative clustering(Voorhees, 1986), TFIDF classifier and feature vectors built in feature extraction
process. Using language independent word clustering the feature vector size was reduced from 200,000 to
21,000. A sample of clusters in shown in Table 12. Words in italic are from semantic point of view not
correct clustered. Adding language dependent rules feature vector size was reduced to 18,000.
530
afer afera aferah afere aferi afero aferami fer
EDQ.QDEDQ.QHEDQ.QHPEDQ.QLEDQ.QREDQ.QLKEDQ.QLNEDQ.QLPEDQ.QLNRPEDQ.QLNRYEDQ.QLaNL
cestnemu mestnemu mestnem mestne mestnega mestna mestni mestno
nihanje nihanj nihanja nihanju ihan
dobojevati izbojevati izbojeval
Table 12: Sample of Slovenian clusters.
Three test samples were manually created. All of them consist of 5 documents similar in topic.
Language models were built in the same way as in the previous experiment with English corpus. There was
only one difference. All words from test samples were added to the vocabulary to avoid the problem of outof-
vocabulary words. By using language independent feature extraction we have got 14% perplexity
reduction. By adding language dependent rules the perplexity was reduced up to 30%. Results are given in
Table 13.
Language independent feature extraction
Topic G PP T PP C PP N PP
Sport 200 621 199 197
Weather forecast 154 201 153 141
Politics 220 598 210 215
Language dependent feature extraction
Topic T PP C PP N PP
Sport 455 183 171
Weather forecast 168 150 115
Politics 598 201 189
Table 13: Test set perplexities for Slovenian language.
8. Conclusion
In our experiments we have shown that topic adaptation does result in a decrease in perplexity. To
train a language model it does not make sense to use only a small portion of topic specialised text.
The results have shown that word clustering delivers significant topic detection improvement for highlyinflected
languages and almost no improvement for English language.
The main drawbacks of experiments on Slovenian corpus were corpus size and absence of keyword labels.
9. References
Jelinek F 1998 Statistical methods for speech recognition .Cambridge, MIT Press.
%\UQH : +DML. - ,UFLQJ 3 .UEHF 3 3VXWND -  0RUSKHPH EDVHG ODQJXDJH PRGHOV IRU VSHHFK
recognition of Czech. In Proceedings of the Third International Workshop: Text, Speech and
Dialogue, pp 211-216.
Katz M S 1987 Estimation of probabilities from sparse data for the language model component of a speech
recognizer. IEEE Transaction on Acoustics, Speech, and Signal Processing 35(3):400-401.
Porter M F 1980 An algorithm for suffix stripping. Program 14(3).130-137.
Voorhees E M 1986 Implementing agglomerative hierarchic clustering algorithms for use in document
retrieval. Unpublished TR 86-765, Cornell University.
Joachims T 1996 A probabilistic analysis of the Rocchio algorithm with TFIDF for text categorization.
Unpublished CMU-CS-96-118, Carnegie Mellon University.
3RSRYL.0Implementation of a Slovene language-based free-text retrieval system. Unpublished PhD
thesis, University of Sheffield.

531
The mandative subjunctive in British English seems to be alive and kicking… Is
this due to the influence of American English1?
Noëlle Serpollet
Lancaster University
1. Introduction
Johansson & Norheim (1988: 34) wrote at the end of their 1988 paper: “results from elicitation tests
[…] suggest that the mandative subjunctive may be on the increase in British English. To study such
changes, we need two new comparable British and American corpora”. With the completion of the onemillion-
word corpora FLOB (Freiburg-LOB, 1991) and Frown (Freiburg-Brown, 1992), the two 1990s
counterparts of the 1960s LOB and Brown, it is now possible to study these changes. These four
corpora being available, the objective of this paper is to study the variation of a grammatical category
of modern British English (henceforth BrE) such as the mandative constructions (modal should and the
mandative subjunctive) over a thirty-year period. In order to carry out this analysis, I will first analyse
two general categories – Press and Learned Prose – of two corpora of BrE: the Lancaster-Oslo/Bergen
(LOB) and Freiburg-LOB corpora (FLOB).
(1) […] but it is also very important that they should be fair. (LOB Press, B)
(2) […] nor to obtain an order that the child be accommodated by them […] (FLOB Learned
Prose, H)
(3) […] usually by recommending that politicians or administrators introduce incentive […].
(FLOB Learned Prose, J)
Then in order to know if the tendency observed in my findings is only intrinsic and limited to
the two text categories studied or is indeed verified throughout the other genres (Fiction and General
Prose), I will carry out a thorough analysis of the whole LOB and FLOB corpora. I have applied a
grammatical approach to corpus data, using Xkwic, a fast concordance programme to carry out an
analysis which involved developing complex queries on grammatically tagged, comparable and
computerized corpora.
My results confirmed previous studies (Övergaard, 1995 - Hundt, 1998) and showed that the
use of the mandative subjunctive is increasing, whereas the use of mandative should is declining.
So far, the description of two complementary phenomena has been provided, but we are still
in need of a possible explanation for this “revival” of the subjunctive suspected by Johansson and
proven by other linguists’ observations. If the mandative subjunctive is indeed ‘alive and kicking', is its
health sustained by American English (henceforth AmE)? Is the increase of the subjunctive in BrE
therefore due to the Americanization of the British language? The final part of this paper will attempt
to provide an answer to this question by analysing two corpora of American English: Brown and its
1992 counterpart Frown.
2. Background
2.1. Previous studies
The grammarians of the 1950s and 1960s considered the extinction of the subjunctive either imminent
or already accomplished. They stated that the subjunctive was gradually dying out of the language, that
it was fossilised, that its death throes could be observed in literary English or that living English had no
subjunctive at all. Harsh (1968 : 11) reports the fact that Sir Ernest Gowers, in his 1965 revision of
Fowler's A Dictionary of Modern English Usage, decided to leave intact what Fowler had written in
1926, “writing off” the subjunctive in the following statement: “it is moribund except in a few easily
specified uses” (1965: 595). According to Harsh (1968:12), “the inflected subjunctive, though hardly in
a state of robust good health, has been taking a long time to die. But that it is still dying, as Fowler
noted, can hardly be denied”.
Johansson and Norheim (1988: 27) state nonetheless and without any doubt that “English
verbs have distinctive forms under certain circumstances which differ from the normal indicative forms
and convey the meaning of ‘non-fact', which is characteristic of the subjunctive in other languages”.
1 Acknowledgement: the research reported here was supported by an award from the Economic and Social Research Council
(UK).
532
Quirk et al. (1985: 1012-1013) define the subjunctive occurring “in that-clauses after verbs,
adjectives, or nouns that express a necessity, plan, or intention for the future” as the mandative
subjunctive and this is this use of the subjunctive that “[this] corpus-based investigation of language
change in progress” is focusing on.
I borrowed here the terms in inverted commas above, to Mair & Hundt (1995) who use this expression
as the subtitle of their article presenting a pioneering effort on that subject.
The analysis by Johansson and Norheim (1988) which is the starting point of this study aimed
at verifying if the subjunctive was more frequently used in AmE than in BrE, examining for that
purpose the two comparable Brown and LOB corpora. The results showed that the mandative
subjunctive was favoured in the American corpus while its number was very low in the British one, and
that the construction with should was preferred to the subjunctive in the British material. This confirms
the observations of Quirk et al. (1985: 1012-1013) who emphasise the fact that the mandative
subjunctive is especially used in American English, whereas in BrE, mandative should with the
infinitive is more common.
While a synchronic study of mandative constructions in two types of English had been carried
out, what was needed in 1988 was a diachronic analysis of these constructions to observe their
evolution and to check if the following statements by Quirk et al. (1985) were still applicable to the
English of the 1990s:
- the present subjunctive occurs more frequently in AmE than in BrE
- its use in that-clauses seems to be increasing in BrE.
Several studies of “language change in progress” have been undertaken since, and analyses focusing on
the evolution of different grammatical features have been conducted on parallel corpora and more
specifically on LOB / Brown and on the new comparable corpora FLOB / Frown2. Indeed, recent
studies using new corpus resources (Asahara, 1994; Övergaard, 1995; Hundt, 1998) have analysed the
diachronic evolution of mandative constructions in BrE and AmE. They have presented findings which
indicate a remarkable increase of the use of the mandative subjunctive in British English, especially in
late 20th century. Apparently, this use of the subjunctive, although not very frequent, is far from
becoming extinct. In sections 4. and 5., more will be said about previous work on the mandative
constructions.
With four carefully matched corpora now available, an exhaustive corpus-based study of language
change in progress over a thirty-year period can be conducted. This study can analyse and compare
synchronic corpora to examine for example the possible influence of American English on British
English with FLOB/Frown or diachronic corpora to study the evolution of linguistic features with
LOB/FLOB in BrE.
2.2. The “mandative constructions” (non-inflected or morphological subjunctive & periphrastic
construction with the modal should): a definition
In this section, I will concentrate on the “mandative constructions” that I will describe, I will set out
the criteria to recognise the subjunctive and clarify the terminology used.
Etymologically, “mandative” is derived from the verb ‘mandate', itself coming from the Latin
manda-re: to enjoin, command. The term “mandative expressions” is used in reference to verbs, nouns
and adjectives (that I also call triggers) which express a demand, request, intention, proposal,
suggestion, recommendation, etc. This expression is borrowed from Algeo (1992: 599) who himself
adapted it from the term “mandative subjunctive” used by Quirk et al. [1985:156] for one of the three
verb forms in the that-clause that follows certain expressions of resolution, intention, etc. By extension,
I use the term “mandative constructions” for the different verb forms which can follow mandative
expressions. Therefore instead of using the expression periphrastic construction with the modal should
(Övergaard, 1995; Hundt, 1998) to designate the construction with the modal which is one variant of
the mandative subjunctive, I use the term “mandative SHOULD”.
(4) I insisted that he should take part in the concert, Alan said. (LOB Fiction, P)
(5) During the stand-up confrontation, which took place shortly after the new year at Highgrove,
the Prince of Wales's Gloucestershire home, Charles insisted that his son have a more
conventional celebration in the newly refurbished Orchard Room in the house. (The Sunday
Times (6th February 2000), “Charles and William in nightclub row”)
Different verb forms can follow the mandative expressions: we can have a mandative
subjunctive as in the example (6), or the non-distinctive form (7) that I will define below, or the modal
2 These four corpora will be described in detail in section 3.
533
auxiliary should followed by an infinitive (8) or the indicative (9). The last construction, namely the
indicative, will not be considered further in this paper.
(6) She insisted that he leave early.
(7) He suggests that we leave early.
(8) Her wish was that he should leave early.
(9) She was eager that he left early.
(examples from Algeo (1992 : 599) apart from the non-distinctive form in (7))
The subjunctive is difficult to identify because it is identical to the base form of the verb.
According to Asahara (1994: 2) “the present subjunctive refers to a grammatical form that takes only
the base form of the verb regardless of tense contrast, person and number concord”. Therefore, with a
plural subject, there is no difference between the indicative and subjunctive forms. The non-inflected or
morphological subjunctive is distinguishable from the indicative (through morphological criteria) in the
following cases:
- in the 3rd person singular present tense (no –s) (10)
- in past contexts (no sequence of tenses) (11)
- in finite forms of be (base form for all persons and no tense marker) (12), (13)
- in negated clauses (no do-periphrasis and not is placed before the verb) (14)
(10) […] he proposes to Isabella that she join his plan to frame Mariana […]. (FLOB General
Prose, G)
(11) Russia insisted that the Western powers take immediate measures to put an end to the
unlawful and provocative actions of the Federal German Republic in West Berlin. (LOB
Press, A)
(12) Hence it is important that the process be carried out accurately. (FLOB Learned Prose, H)
(13) Conditions have dictated that operations be scaled down enabling overheads to be reduced
[…]. (LOB Press, A)
(14) Moreover, it requires that the concepts F(x) and G(x) not themselves contain any
quantification […]. (FLOB Learned Prose, J)
I included in my counts, as part of the mandative subjunctive forms, not only the distinctive/genuine
subjunctive forms but also the non-distinctive forms which are indistinguishable from the indicative, as
in (15a).
(15a) I will guard your house for you on condition that you bake me an apple pie every day. (LOB
General Prose, F)
In that case, we can perform a substitution test by putting a third person singular subject in the place of
“you” and we can see that we obtain a distinctive subjunctive form:
(15b) I will guard his house for him on condition that he bake me an apple pie every day.
3. Material used and purpose of this paper
3.1. Data
The four carefully matched corpora that have been analysed are the following:
¨ Two well-known and widely used corpora representing the language of the 1960s:
· The Brown corpus, compiled at Brown University consists of one million words made up of
500 texts of American English from 1961 and spread over 15 categories3.
· The Lancaster-Oslo/Bergen Corpus (LOB) has been compiled, computerized and word-tagged
by research teams at Lancaster, Oslo and Bergen. It consists of 500 British English texts of
about 2,000 words each, printed in 1961, divided into 15 different genre categories and
contains one million words. It is synchronically parallel to the Brown corpus.
¨ Two parallel one-million-word-corpora matching the original LOB and Brown corpora, developed
at Freiburg University to enable linguists to study language change in progress:
· The Freiburg-LOB Corpus (FLOB, 1991) has been modelled on LOB; it is constituted of one
million words of British English texts printed in 1991.
· The Freiburg-Brown Corpus (Frown, 1992) modelled on Brown is synchronically parallel to
FLOB and diachronically parallel to Brown.
3 They are listed in the notes to table 1.
534
3.2. Objectives
As I have indicated in section 3.1. above, this study only undertakes the analysis of written data. It is
intended to be an observation and a description of linguistic change in contemporary English. On that
aspect, I refer to the study of grammatical change in present-day English by Mair (1997) and Hundt
(1997) and I quote Holmes (1994:37):
The prospect of using corpora data to infer language change over time is an exciting one. It is clearly possible to
make suggestive and interesting comparisons between the frequencies of items in corpora of similar size and
composition which have been constructed at different points in time.
The Press and Learned Prose genre (which amount to about 2/5 of each corpus) are represented in the
figure below by two rectangles with upward (LOB) and downward (FLOB) diagonal patterns.
1961 1991
1961 1992
Figure 1: British English 30 years on, a description - AmE, a possible explanation to the
changes in BrE?
The paper will develop a descriptive analysis based on a thorough and exhaustive observation of
the data. Therefore, in order to describe what has been happening over a thirty-year period, I have
carried out a comparative study of two parallel and computerised corpora LOB and FLOB to find out if
there is an increase in the use of the mandative subjunctive form and concurrently a decline in the use
of the mandative should over the years. My objective is also to examine in what text genres this has
taken or is taking place.
· Thus I will first analyse the Press and Learned Prose categories to see if two different genres
follow the same trend, i.e. have experienced the same evolution regarding the mandative
constructions.
· Then I will verify if the trend is found in the rest of the two corpora. Is it general to BrE or genre
specific?
· Finally the analysis of American data will enable me to see if the ongoing change in BrE is
dependent of diachronic developments in AmE or of the synchronic influence of AmE.
A detailed qualitative analysis of the data is beyond the scope of this paper. Nonetheless, I hope that
the last part of this study will shed some light on the evolution of mandative constructions and provide
the beginning of a possible explanation.
4. METHODS AND PILOT RESULTS
4.1. Method used
I have applied both a grammatical approach to corpus data, and a corpus linguistics methodology, using
computer tools and retrieving software such as Xkwic. First, I carried out a concordance of should in
WordSmith Tools4 (Version 3.00.00), an integrated suite of programs used to investigate how words
behave in texts, in order to retrieve the total number of occurrences of the modal.
4 More detailed information can be found at http://www.liv.ac.uk/~ms2928/ and in Scott (1996).
LOB
Brown
FLOB
Frown
535
Table 1: Frequency of should in the LOB and FLOB corpora
SHOULD
CATEGORIES LOB FLOB Difference
(abs)
Difference
(%)
PRESS5 (A-B-C) 285 185 - 100 - 35.1
FICTION6 (K-LM-
N-P-R)
214 250 + 36 + 16.8
GENERAL
PROSE7 (D-E-FG)
472 330 - 142 - 30.1
LEARNED
PROSE8 (H-J)
330 382 + 52 + 15.8
TOTAL 1301 1147 - 154 - 11.8
We are shown with these results that the overall number of occurrences of should has decreased
between the 1960s and the 1990s. However, this trend is not generalised to all genres, as we can note a
decrease in the press and general prose categories and yet an increase in fiction and in learned prose.
Identifying the mandative uses of the modal on which this present study focuses required a
semantic analysis leading to a classification of each of the retrieved occurrences. This process was
laborious and time consuming and I limited the analysis to the Press and Learned Prose categories of
LOB and FLOB. Then, I used a concordancing tool to find the mandative constructions without having
to go through a detailed manual analysis. I started with WordSmith and as I progressed with my search
queries, I realised that the package was not adapted to the type of complex queries I needed to create in
order to retrieve only the mandative constructions. This tool could not support at the same time wildcards
(*), windows {n}, interval operators {n, m} nor tags. Therefore I had to switch to Xkwic, which
could handle much more complex queries, as it has a more powerful search language, and therefore I
could limit the number of hits I was getting.
Xkwic is a software part of the ISM Corpus Workbench9, and a motif-based user interface to
the Corpus Query Processor (CQP). CQP, the concordance engine itself receives an input from a file,
under the form of a query entered in Xkwic, and returns the result back to Xkwic once it has been
computed. The analysis reported thereafter involved developing complex queries to retrieve only the
relevant instances of both the modal and the subjunctive – which is not part-of-speech tagged. I used
four totally comparable, grammatically tagged and computerized corpora of British English and
therefore, I could run exactly the same retrieving queries in all corpora (LOB / FLOB and Brown /
Frown). This is where the originality of my research comes from: in this very use of the same search
patterns which provided me with comparable findings.
As I have already mentioned in section 1., some of the previous studies of mandative
constructions presented results that were somehow incomplete. Asahara (1994) did not use
computerised corpus data, and relied on a rather small number of examples; nonetheless, her results are
very interesting. A similar situation is found with Övergaard (1995) who did not use truly parallel
corpora: she analysed the Brown and LOB corpora for the 1960s, but worked with four other corpora
that were not computerised for the more recent period. The instances in these corpora are random
examples that she recorded when she encountered them, hence the non-reliability of one part of her
research. Regarding Hundt's analysis (1998: 162), it appears as very valuable but incomplete as she
used her own findings on FLOB and Frown (which was yet complete at the time) as well as results
from Johansson and Norheim (1988) for Brown and LOB. Therefore, the range of governing
expressions (triggers) was limited to only 17 verbs and their related nouns. Nonetheless these previous
studies indicate a trend in the evolution of mandative constructions that I intend to verify in the two
sets of parallel and comparable corpora.
The occurrences of mandative constructions have been retrieved using the concordance
programme Xkwic and an appropriate search query. I will present below some issues involved with the
5 A = reportage, B = editorial, C = reviews [88 texts a 176,000 words]
6 K = general fiction, L = mystery & detective fiction, M = science fiction, N = adventure & western fiction, P = romance & love
story, R = humour [126 texts a 252,000 words]
7 D = religion, E = skills, trades & hobbies, F = popular lore, G = Belles Lettres, bibliography, essays [176 texts a 352,000
words]
8 H = miscellaneous, mainly government documents, J = learned & scientific writings [110 texts a 220,000 words]
9 The IMS Corpus Toolbox has been developed at the University of Stuttgart. Further details can be obtained from the following
web page: http://www.ims.uni-stuttgart.de/projekte/CorpusWorkbench/ and in Chist (1994).
536
search criteria. When a triggering expression was followed by more than one that-clause, only the first
clause was included in the final results. The decision could have been different if the data had been
only analysed by hand; however, Xkwic stops at the first clause and does not account for any following
that-clause triggered by the same expression. In the case of several verbs appearing in a that-clause, I
took the decision to include in my counts only the first one, as the concordancer only accounts for the
first verb following that. Regarding that issue, I have taken the same approach as Johansson and
Norheim (1988). I shall indicate as well that I accounted for the that-deleted clauses in my results (even
although they are very rare) and as I mentioned in 1.2., I also included the non-distinctive forms in my
counts (details are given in the tables). I limited my research to a set of trigger, i.e. a finite set of
suasive expressions: 64 suasive verbs, 52 corresponding nouns and 40 adjectives. A few words now
about precision and recall:
- precision error: the search output given by Xkwic contained instances that were not mandative
constructions (they did indeed contain a trigger and the modal should or a base form, but the
mandative construction had a non-mandative meaning). These were false instances or noise that
needed to be removed manually. Even with an automatic computer search the manual intervention
of the analyst is still needed and indeed necessary in order to provide accurate results.
- Ratio of error: about 4 hits out of 10 had to be discarded. Therefore, the recall rate generally varied
between 57 and 67%, i.e. an average of 63%.
- Recall error: having retrieved all the occurrences of mandative should in the Press category of
LOB and FLOB (using WordSmith), I tested my queries in Xkwic in order to see if I was
retrieving all the occurrences of the modal. In fact, the concordancer failed to retrieve genuine
occurrences of mandative constructions because they fell out of the search criteria set up in the
Xkwic query (it would appear that the syntax is more complex in Learned Prose and that some of
the mandative forms fall out of the limitations of the search queries).
Table 2: Retrieval rate of Xkwic for mandative should (comparison of manual analysis and automatic
retrieval with that-deleted clauses included)
Retrieval rate (should)
LOB FLOB
Press 97% 91%
Learned
Prose
90% 75%
A possible remedy would be to extend the search of the search scope. Unfortunately, such an extension
revealed unmanageable as too much noise was encountered and a practical choice had to be made.
However, the instances that have not been retrieved are in a limited number. Moreover, the occurrences
retrieved still form a large subset of the total number of possible occurrences contained in the corpus.
4.2. Results in two genres of British English
This section presents the results of the queries carried out in only two genres of the two corpora of BrE:
· Press (A, reportage; B, editorial and C, reviews), ca. 176,000 words
· Learned Prose (H, miscellaneous, mainly government documents and J, learned & scientific
writings), ca. 220,000 words
It would seem that the Press would be more prone to evolution as it tends to reflect the changes
occurring in modern written language. We might therefore expect to witness in this genre a decline of
the mandative subjunctive which belongs to a formal, legalistic style such as the one found in the
Learned Prose category considered as more formal and conservative.
537
Table 3: Frequency of mandative should in LOB and FLOB [A-B-C] with verbs, nouns and adjectives
as triggers (that-deleted clauses included)
Should in LOB (Press) Should in FLOB (Press)
VERBS NOUNS ADJ. TOTAL VERBS NOUNS ADJ. TOTAL
2110 8 7 36 1211 6 1 19
We are shown that mandative should has decreased in the Press category from the 1960s to the 1990s
by 47.2% after the three types of triggers, which is the general tendency of all the uses of should in
Press. It has decreased from 36 to 19 occurrences, which means going from 12.6% of the total number
of occurrences of should to 10.3%.
(16) They will accompany Mr. Heath next month when he goes to Brussels, headquarters of the
Common Market Commission, or wherever the Six decide negotiations should be held. (LOB
Press, A)
(17) The suggestion that Royton should be demolished for the delight of the yuppie mugwumps of
Oldham will alarm many Roytoners. (FLOB Press, B)
Table 4: Frequency of genuine mandative subjunctives and non-distinctive forms (ND) in LOB and
FLOB [Press] with verbs, nouns and adjectives as triggers (that-deleted clauses included)
Mandative subjunctives and ND
in LOB (Press)
Mandative subjunctives and ND
in FLOB (Press)
VERBS NOUNS ADJECT. TOTAL VERBS NOUNS ADJECT. TOTAL
Subj. 2 2 0 4 3 1 0 4
Non-dist. 3 0 1 4 4 0 0 4
TOTAL 5 2 1 8 7 1 0 8
The results of the concordances on the base form carried out in the Press category of the two corpora
show that the total number of mandative subjunctives and non-distinctive forms is identical in the two
corpora with four subjunctives and four non-distinctive forms. In this genre, although the total number
of subjunctive forms is constant from the 1960s to the 1990s, the variation in the number of triggers
does not follow one particular direction. There are more occurrences triggered by verbs in FLOB and
fewer instances triggered by nouns and adjectives.
(18) Since Peking realises just how much Britain needs the deal, it demanded that Mr Major - and
his kudos as world leader – come in person to sign it. (FLOB Press, A)
Table 5: Frequency of mandative should in LOB and FLOB [H-J] with verbs, nouns and adjectives as
triggers (no that-deleted clauses)
Should in LOB (Learned Prose) Should in FLOB (Learned Prose)
VERBS NOUNS ADJ. TOTAL VERBS NOUNS ADJ. TOTAL
24 6 14 44 20 6 1 27
In Learned Prose we can note that mandative should has decreased by 36.8%, but mainly after the
triggering adjectives where we can notice a sheer drop in numbers, whereas should as a whole had
increased in this genre. The construction which used to represent 13.3% of the total number of
occurrences of should represents now only 7% of this total.
(19) It is essential that the landscaping should be designed for ease of maintenance […]. (LOB
Learned Prose, H)
10 This count of 21 occurrences includes two occurrences of SHOULD in two that-deleted clause triggered by the verbs decide and
agree.
11 This count of 12 occurrences includes one occurrence of SHOULD in a that-deleted clause triggered by the verb propose.
538
(20) We recommend that the Department should re-appraise and update such calculations at
frequent intervals […]. (FLOB Learned Prose, H)
Table 6: Frequency of genuine mandative subjunctives and non-distinctive forms (ND) in LOB and
FLOB [Learned Prose] with verbs, nouns and adjectives as triggers (that-deleted clauses included)
Mandative subjunctives and ND
in LOB (Learned Prose)
Mandative subjunctives and ND
in FLOB (Learned Prose)
VERBS NOUNS ADJECT. TOTAL VERBS NOUNS ADJECT. TOTAL
Subj. 1 0 0 1 9 3 1 13
Non-dist. 4 0 1 5 4 0 1 5
TOTAL 5 0 1 6 13 3 2 18
In the Learned Prose category, the mandative subjunctive is on the increase by 1211% [sic] if we
consider the genuine subjunctive forms on their own. If we include the non-distinctive forms which
have remained stable at five occurrences in each corpora, we still have a very important increase of
198% which is encountered mainly after trigger-verbs. In that case, there is no doubt that we are
witnessing a rise in the mandative subjunctive over a thirty-year period and although the numbers are
very small, this would tend to prove that the subjunctive is not dying in Learned Prose in BrE.
(21) The petitioner would then request that the house overrule the injunction or, alternatively,
make a clear determination on where the suit ought best to be tried. (FLOB Learned Prose, J)
So far, the findings from two genres confirm the trend presented by previous studies, i.e. that
the use of the modal should as a periphrastic alternant to the non-inflected subjunctive is declining.
Regarding the remarks that I made at the beginning of this section, a closer examination of the results
leads me to say that in the Press category the counts of the subjunctive are inconclusive as no variation
has be noted, i.e. the form is stable. However a “revival” of the subjunctive can be noted in Learned
Prose. Hundt (1997: 167) indicates that “it is hardly surprising that a genre [Academic prose, category
J] which is resisting the trend towards a more colloquial written style should be the vanguard of a
change that is reviving a formal syntactic option”.
5. Full results in all genres
The tables that follow present the results obtained after the analysis of the two corpora of written BrE
in their entirety. This will enable me to have a full picture (nonetheless limited to one million words
and four general genres) of some specific grammatical changes that happened between 1961 and 1991
and to see if the trend previously identified is verified in all genres or specific to some.
Table 7: Raw frequency and proportion of genuine mandative subjunctives and non-distinctive forms
in LOB and FLOB
Mandative
subjunctive forms
LOB FLOB
Genuine
subjunctive
14
(48.3%)
33
(58.9%)
Non-distinctive 16
(51.7%)
14
(43.1%)
Total 29 58
From LOB to FLOB, the number of mandative subjunctives has doubled and the proportion of nondistinctive
forms has decreased by 10.6%. This means that in FLOB, 58.9% of the subjunctive forms
are genuine subjunctive forms. Hence, even if the non-distinctive forms are not included in the
quantitative analysis, the direction of the evolution is not skewed; the increase is even more remarkable
(from 100% to 135.7%).
In the tables 8 and 9 below, both the raw frequencies and the frequencies normalized per 100
texts are presented12. The latter count is due to the fact that the different categories are not balanced.
The general genres, which regroup several text categories, each contain a different number of texts (see
table 1). The normalized frequencies provide a better idea of the repartition of the mandative forms in
12 A raw frequency of five subjunctives within 176 texts in General Prose is normalized to 2.8 occurrences within 100 texts.
539
each genre and give us an insight into the stylistic distribution. Looking at the raw frequencies for
FLOB in table 8, one would think that there were more genuine subjunctives in General Prose, then in
Learned Prose, in Fiction and finally in Press. In reality, the normalised frequencies show us that it is
the Learned Prose category which ranks first with the highest number of mandative subjunctives forms,
then General Prose, Press and Fiction. This classification does not really come as a surprise as the two
“Prose” genres tend to use a more formal style with legalistic writing (category H), academic prose
(category J) on the one hand and religious writings (category D), Belles Lettres, bibliographies and
essays (category G) on the other hand.
Table 8: Frequency of genuine mandative subjunctives and non-distinctive forms across genres in LOB
and FLOB
LOB FLOB
Genre Subj. Non-dist. Total Subj. Non-dist. Total
Frequency
(n.)
(88 texts)
4 4 8 4 4 8
Press
Normalized
per
100 texts
4.5 4.5 9 4.5 4.5 9
n. (126) 4 2 6 7 4 11 Fiction
% 3.2 1.6 4.8 5.6 3.2 8.8
n. (176) 5 4 9 9 12 21 General
Prose % 2.8 2.3 5.1 5.1 6.8 11.9
n. (110) 1 5 6 13 5 18 Learned
Prose % 0.9 4.6 5.5 11.8 4.6 16.4
The number of mandative subjunctives stayed constant in the Press category, whereas in Learned
Prose, the number of occurrences increased by 198%. Table 10 summarises the increases per genre of
all the mandative constructions.
Table 9: Frequency of genuine mandative subjunctives and non-distinctive forms across genres in LOB
and FLOB
LOB FLOB
Genre Should Should
Frequency
(n.)
36 19
Press
Normalized
per
100 texts
40.9 21.6
n. 19 9
Fiction % 15 7.1
n. 56 28 General
Prose % 31.8 15.9
n. 44 27 Learned
Prose % 40 24.5
The use of the periphrastic construction with should has decreased in all four genres between 38.6%
and 100%.
To be able to check easily the evolution of all the mandative constructions studied, the two
tables above are summarised below. I have indicated in parenthesis the rank of each category from the
highest rise or decline to the lowest.
540
Table 10: Summary of the evolution of the mandative constructions from LOB to FLOB
Categories Genuine subjunctive Subj. with nondistinctive
forms
Should
Press (Same number) (Same number) (4) - 47.2% (3)
Fiction + 75% + 83% (3) - 52.6% (2)
General Prose + 82% + 133% (2) - 100% (1)
Learned Prose + 1211% + 198% (1) - 38.6% (4)
We find almost the same ranking between the categories containing the highest number of mandative
subjunctive forms and the categories having experienced the highest increase, the only difference being
between Press and Fiction which have exchanged their third and fourth places. Regarding mandative
should, its use has decreased between 38.6% in Learned Prose and 100% in General Prose. And
although there does not seem to be a correspondence within genres between the fall of one construction
and the rise of the other, Hundt (1998:163) notes about results13 on the whole LOB and FLOB corpora
that “[f]or the British corpora, a chi-square test proves that the increase in mandative subjunctives and
the concomitant decrease of the periphrastic construction is highly significant (p  0.001)”. The only
feature of interest is the fact that while Learned Prose is the category where the most important rise of
subjunctive has taken place, it is also the very category where mandative should has decreased the
least. This tends to reconfirm our observation that this genre is formal and conservative and that the
only change experienced is to revive a formal syntactic option.
An observation of the data and description of the results that confirmed previous analyses
have been provided. They showed that amongst stylistic variations from one genre to the next, general
and real grammatical changes concerning mandative constructions are taking place in BrE. Now we
need a possible explanation of the evolution witnessed in BrE. Where does this “revival” of the
mandative subjunctive and decline of mandative should come from? Is this due to the americanization
of the British language? This possibility is worth investigating as many observations carried out on
American vs. British English have suggested the influence of the former on the latter. The American
influence is referred to an ‘accelerator of change’ within BrE (Barber, 1964: 141). Hence would this be
the main reason for these linguistic changes?
The final section of this paper will thus analyse the two corpora of American English: Brown and
Frown in an attempt to provide an answer to these changes.
6. Analysis of two corpora of American English: a possible explanation?
The following examples are extracted from Brown and Frown:
(22) It was essential that he should restore his formidable reputation as a rip-roaring, ruthless gunslinger
and this was the time-honored Wild West method of doing it. (Brown Fiction, N)
(23) The panel recommended that public affairs preparations should be included in the planning for
future military operations […]. (Frown General Prose, E)
(24) The doctors had suggested Scotty remain most of every afternoon in bed until he was
stronger. (Brown Fiction, K)
(25) Lattimore's attention to these events was distracted by a request from the United Nations that
he head a technical assistance mission to Afghanistan […]. (Brown General Prose, G)
Table 11: Repartition of the mandative subjunctive forms per genre in the four corpora14
Genres LOB FLOB Brown Frown
Press 8 8 (0) 29 23 (- 20.7%)
Fiction 6 11 (+ 83%) 21 35 (+ 66.7%)
General Prose 9 21 (+ 133%) 42 27 (- 35.7%)
Learned Prose 6 18 (+ 198%) 15 22 (+ 46.7%)
Total 29 58 (+100%) 107 107 (0)
13 Her results display frequencies (expressed in percentages) which are not dissimilar to mine – see table 12.
14 The results for the two American corpora are still provisional.
541
We can see that the status of the mandative subjunctives in the Press genre in AmE is even worse than
in BrE (decrease of 20.7%). This would tend to indicate that the status quo of the subjunctive observed
in FLOB might only be the preliminary to a future decline, as it is the case in AmE. The increase of the
subjunctives in Fiction is marked more in FLOB than in Frown although the number of instances in
FLOB (11) is inferior to the one in Brown (21) and far from the number reached in Frown (35). In
General Prose, a rise can be observed in FLOB contrary to a decline in Frown. Number wise, BrE has
not yet but almost caught up with AmE (21 compared to 27 occurrences) and the proximity in numbers
seems to be due to the fact that the subjunctive is decreasing in this category of AmE. Here the results
are inconclusive. What variety of English is leading the changes? Finally, Learned Prose shows an
increase in the two 1991 corpora and BrE is catching up with AmE even if the number of instances in
FLOB (18) is just in between Brown (15) and Frown (22). Here BrE seems to be following the trend
set up by AmE.
The findings on mandative should (see table 12) present a decrease of this construction
throughout the genres (- 23% from the 1960s - with 26 occurrences, to the 1990s - with 20 instances);
the number are very small (between 3 and 7 for each genre) and little variation is shown.
If AmE is indeed leading the way on the path of linguistic change (as is indicated in the
results), BrE appears to be lagging behind but is doing its best to catch up.
Table 12: Frequency of mandative should vs. mandative subjunctive forms in LOB/FLOB and
Brown/Frown
Brown Frown LOB FLOB
Should Subj. Nondist.
Should Subj. Nondist
Should Subj. Nondist
Should Subj. Nondist
26 91 16 20 78 29 155 14 15 83 33 25
19.6% 68.4% 12.0% 15.8% 61.4% 22.8% 84.2% 7.6% 8.2% 58.9% 23.4% 17.7%
=80.4% =84.2% =15.8% =41.1%
The proportion of mandative subjunctive forms represents around 80% of mandative constructions in
AmE and has only risen by 3.8%. Would there be a slowing down of the evolution of the mandative
constructions on the way, leading to a future stabilisation? In BrE the proportion went from 15.8% to
41.1% with a rise of 25.3%. The subjunctive seems to be the preferred option in AmE (we can note
than should has not disappeared yet) whereas the periphrastic construction with should seems to still be
favoured by British people although the trend could be reversing in the future.
The trend observed in BrE regarding the mandative should (-23%) is mirrored in AmE; it is
also decreasing in all genres by 46.5%. The hypothesis presented by Johansson (1988) is verified, the
mandative subjunctive forms are on the increase in BrE by 100% whereas it has stabilised in AmE with
a great disparity between text categories. Hence the spread and rise in the subjunctive, observed in the
first half of the 20th century must have slowed down in this variety of English.
7. Conclusion
This descriptive and exploratory study has identified specific trends in the evolution of mandative
constructions which are both specific to genres and varieties of English. These linguistic changes
observed are much more drastic in BrE than in AmE where the language change in progress seems to
be slowing down.
Three hypotheses for the evolution of BrE can be presented:
· Americanization
Several studies have presented the hypothesis (confirmed by their analyses) that AmE is more
innovative than BrE in ongoing morphological and syntactical changes. Therefore, it would influence
BrE and would lead to an americanization of the language.
· Grammaticalization
· Colloquialization
Mair (1997) states that very few genuine and significant instances of grammatical change can be
observed.
The changes “are not due to the fact that the grammar of the language itself has changed. Rather, these
developments [increased frequency of the progressive and the going-to future, greater use of contracted forms]
show that informal options which have been available for a long time are chosen more frequently today than
would have been the case thirty years ago” (1997: 203).
542
Therefore, he prefers talking about “colloquialization” of the norms of written English rather than
about “grammaticalization15” of the language when a syntactic structure is being replaced by an older
grammatical form. But could this be argued with the rise in the use of the mandative subjunctive which
used to be reserved to formal genres? This hypothesis could be verified in the Press section with either
the status quo or decrease in the mandative constructions. However, this possible explanation needs to
be rejected in the other genres: in Learned Prose, a very feeble decrease of should and an important
increase of the subjunctive has been noted, hence the formality of this genre is not loosing any ground.
The same can be said about Fiction with less extreme variations.
Our work presents some limitations and an even more detailed study by type of triggers and/or
by single text category might provide more explanations. We also need to be extremely cautious with
the conclusions drawn from the results, as the corpora analysed are rather small (only one million
words). It would be worth carrying out the same study on a bigger corpus such as the British National
Corpus containing 90 million of BrE written texts. If the trends observed in BrE were confirmed in the
BNC data, this would show that the changes noted in this present analysis are not due to chance and/or
to the sampling and size of the data.
References
Algeo J 1992 British and American mandative constructions. In Banks C (ed), Language and
Civilisation: A Concerted Profusion of Essays and Studies in Honour of Otto Hietsch, Vol. 2,
Frankfurt, Peter Lang, pp 599-617.
Asahara K 1994 English Present Subjunctive in Subordinate That-Clauses. Kasumigaoka Review 1: 1-
30.
Christ O 1994 A modular and flexible architecture for an integrated corpus query system.
COMPLEX'94, Budapest.
Fowler H W 1965 A dictionary of modern English usage. Rev. ed. Gowers E, 2nd ed. Oxford,
Clarendon Press.
Harsh W 1968 The subjunctive in English. University of Alabama, University of Alabama Press.
Holmes J 1994 Inferring language change from computer corpora: Some methodological problems.
ICAME Journal 18: 27-40.
Hundt M 1997 Has BrE been catching up with AmE over the past thirty years? In Ljung, M (ed),
Corpus-based studies in English, Papers from the seventeenth international conference on English
language research on computerized corpora (ICAME 17). Amsterdam, Rodopi, pp135-151.
Hundt M 1998 It is important that this study (should) be based on the analysis of parallel corpora: On
the use of mandative subjunctive in four major varieties of English. In Lindquist H et al. (eds), The
major Varieties of English, Papers from MAVEN 97, Växjö University, pp 159-175.
Johansson S, Norheim E H 1988 The subjunctive in British and American English. ICAME Journal 12:
27-36.
Mair C, Hundt M 1995 Why is the progressive becoming more frequent in English? – A corpus based
investigation of language change in progress. Zeitschrift für Anglistik und Amerikanistik 43: 123-132.
Mair C 1997 Parallel corpora: a real-time approach to the study of language change in progress. In
Ljung, M (ed), Corpus-based studies in English, Papers from the seventeenth international conference
on English language research on computerized corpora (ICAME 17). Amsterdam, Rodopi, pp195-209.
Övergaard G 1995 The Mandative Subjunctive in American and British English in the 20th Century.
Stockholm, Almqvist & Wiksell International, Acta Universitatis Upsaliensis, Studia Anglistica
Upsaliensia, Vol. 94.
Quirk R, Greenbaum S, Leech G N, Svartvik J 1985 A Comprehensive Grammar of the English
Language. London, Longman.
Scott M 1996 WordSmith (Computer program). Oxford, Oxford University Press.
15 Grammaticalization: study of how grammatical morphemes are produced from the lexicon.

543
Through the looking glass of parallel texts
Serge Sharoff (sharoff@aha.ru)
Russian Research Institute for Artificial Intelligence /
Alexander von Humboldt Fellow at the University of Bielefeld, LiLi Faculty
Postfach 10 01 31, D-33501 Bielefeld, Germany
1. Introduction
The project reported in the paper was aimed at the comparative analysis of meanings of linguistic
expressions taken in their context, in particular, how meanings intended by the speaker/writer are
realized via lexical items. The goal of the project was to assess the influence of context, on the one
hand, and different languages, on the other hand onto meanings delivered by the same lexical item.
Since no access to enumeration of original meanings is possible, the method for analysis was based on
the investigation of corpora. Several types of lexical items (verbs of motion, names of plants and
animals and size adjectives) were chosen from the English or Russian corpora. Then their translations
into respectively Russian or English were checked.
The corpora used in the project include two technical texts:
· Microsoft Word'97 User Manual and its translation into Russian;
· excerpts from the AutoCAD v.13 User's Manual (Chapter 2: Drawing objects) and its
translation into Russian1;
and three literary texts:
· Lewis Carroll's Alice's Adventures in the Wonderland and three of its translations into
Russian by Demurova, Nabokov and Zakhoder;
· Vladimir Nabokov's The Vane Sisters and two of its Russian translations made by Ilyin and
Barabtarlo;
· Vladimir Nabokov's Lolita and its Russian translation made by Nabokov himself.
All the texts were originally written in English and translated into Russian. The corpora were
aligned at the sentence level in order to compare expressions of the same state of affairs both between
languages and between translators. The corpus (it comprises about 250 thousand words in total) is
relatively small in comparison to parallel corpora developed in other projects, for example, the
Chemnitz English-German/German-English corpus (Schmied, Schäffler, 1996). However, its size
allowed to check the most occurrences of lexical items under investigation manually within the
reasonable amount of time (about 3200 instances of verbs of motion and 1100 size adjectives were
consulted in total).
The paper starts with the description of the XML representation for multilingual concordances and
the Perl-based tools developed for creating, maintaining and consulting them. Then, Section 3
discusses results of the annotation using lexical semantic features taken from the dictionaries. Finally,
Section 4 presents a network of oppositions, which are important for modeling lexical semantics of
verbs of motion and size adjectives in English and Russian. Here, lexical semantics is represented by
means of the systemic network, which is used in multilingual generation applications within the
KPML environment (Bateman, et al, 1999).
2. Software for working parallel concordance
A Perl-based software has been developed within the project to create and maintain parallel
concordances, which are represented in XML. The concordance structure encodes:
1. the general information about each text;
2. the alignment between sentences in parallel texts;
3. morphosyntactic and lexical-semantic properties of words of each text.
A concordance consists of three types of entities: a concordance text, sentence and a word. The
general description of a document to be concordanced includes the following attributes:
1. docid—the identifier of the document (typically its file name);
2. lang—the language, in which the document is written (actually, the lang attribute sets the
default value for intepretation of all words in the concordance. The default value may be
1 The texts were prepared within AGILE, the EU project aimed at multilingual generation of
CAD/CAM manuals, cf. (Bateman, et al, 2000).
544
redefined for a separate sentence or a word, for example, when a foreign language citation is
found);
3. author—the author of the document.
The sentence description includes the following attributes:
1. sentenceid—the identifier of a sentence (it is composed of the document identifier plus the
sentence index within the document);
2. correspondencelist—the list of identifiers of respective sentences in other parallel texts (if a
sentence corresponds to several sentences in one text, this is also reflected in the list).
The word description includes the following attributes:
1. wordid—the identifier of a word instance (it is composed of the sentence identifier plus the
word index within the sentence);
2. lemma;
3. POS—the part of speech, e.g. POS="verb";
4. morphfeatures—morphological features of the word instance, e.g. morphfeatures="3pers,
sing";
5. lexfeatures—lexical features of the word instance, e.g. lexfeatures="wordnet5, motion,
physical, away, source-fg".
The concordance structure is quite flexible for describing parallel texts and is suitable to be the basis
for research in contrastive semantics. Also, it allows to use several languages and several translations
of the same text, in particular, the German translation of Alice is planned to be added to the
concordance. The structure mostly conforms to the EAGLES guidelines (EAGLES, 1996), in
particular, the corpus annotation is separated from the corpus itself. The most important difference
from the EAGLES scheme concerns keeping the parts of speech (major categories in EAGLES terms)
in one attribute (POS) and other values (like person and number) in another one (morphfeatures).
This simplification is suitable for the current purposes of the concordance development, i.e.
consultation of uses of lexical items. For instance, a concordance query, which consults uses of the
verb leave in the corpus, can easily discard the noun leaves and the adjective left, which are irrelevant
for the query, by means of specifying the following condition POS="verb" in the query.
As the result of XML conversion, the concordance is significantly larger than the original text
(approximately by the factor of ten), however, the XML representation of attributes keeps the size of
corpora smaller than the position-based encoding, which is used, for example, in the Susanne corpus
(Sampson, 1995). Alignment of the text pairs has been performed by means of the Marc Alister
software package (Paskaleva, Mihov, 1997), which implements the Gale-Church algorithm for
language-independent alignment (Gale, Church, 1993). Since the language-independent alignment
algorithm frequently fails to detect translation equivalents, the alignment results should be corrected
manually. The alignment of “Alice in the Wonderland” and respective texts with its Russian
translations has been corrected to the possible extent; the alignment of other texts has been mostly left
unchanged. However, the quality of automatic alignment of other texts was much better, because the
Gale-Church algorithm is noise-sensitive. The “noise” in Alice is related to the reported speech,
which is coded in the original English and Russian texts quite differently. By this reason, sentence
boundaries and their alignment cannot be determined automatically. Anyway, alignment errors that
remain in the corpora do not cause significant problems when searching for translation equivalents,
because the consultation function (see below) may present a wide context for each occurrence of a
lexical item. A Perl script allows an incremental annotation of lexical items that conform to a
condition. Initially, the lexical features were based on the set of senses of these lexical items under
investigation from WordNet (Miller, 1990) for English and ECD and (Apresjan, 2000) for Russian. In
the course of the project development, the set of features for annotation was extended to include
oppositions discussed in Section 4.
A concordance helps in development of an instance-based vocabulary, which provides another layer
of representation of semantic properties of lexical items. A lexical item is an XML entity, which has
the following attributes:
1. lexid--the identifier of the lexical item (typically it is the same as lemma, but may have
indices for representing homonyms);
2. lemma;
3. lang--language;
4. lexfeatures--the lexical features of the lexical item; they correspond to the set of features,
which are pertinent to the lexical item in all or most typical cases of its usage;
5. comments.
545
Also, the XML entity for a lexical item has the following multiple-value attributes:
1. use--its values refer to the identifiers of sentences, in which the lexical item occurs;
2. synonym--in addition to the sentence identifier, it specifies the lexical item that is
considered as synonymous in the context of the sentence;
3. translation-- in addition to the sentence identifier, it specifies the identifier and the language
of the lexical item that is considered as a translational equivalent in the context of the
sentence.
The lexid, lemma, lang, and use attributes are filled automatically, when the vocabulary is
constructed from the concordance, while lexfeatures, synonym and translation are filled manually
in the process of corpus analysis. The vocabulary is shared across all concordances and their
languages.
The consultation function produces an HTML file with a KWIC (KeyWords In Context) list, which
items are sentences that conform to the set of search criteria specified as a dedicated Perl function.
The search criteria are applied to one sentence and may include all the information stored in the
concordance about this sentence, e.g. co-occurrence of words within the sentence, specific
morphological or lexical features of words, as well as their synonyms or translation equivalents. The
search criteria can restrict the length of the context, which is extracted from found sentences and
presented to the user. Also, the output can be sorted with respect to the left or right context of the
found items or according to any other condition defined by a Perl function. As for translation
equivalents, the respective parallel sentence or a hyperlink to it may be also included in the output.
Each sentence in the output list can be explored with respect to its wider context, since it is
hyperlinked to its position in the original text. The screen shot in Figure 1 shows the selection of
English and Russian sentences, in which a size adjective (such as little or small) is accompanied with
a noun denoting a child (girl, boy, child or children).
Figure 1 A concordance query result
The developed software allows an incremental research for lexical semantics of specific groups of
lexical items. For example, the dictionary provides all lexical items, from which a list of, say, verbs of
motion that appear in the corpus can be extracted. The concordance query uses the list in order to
classify contexts in which such verbs can be used. As the result, specific contexts can be marked in the
concordance (the information is stored in the lexfeatures attribute) and can be used in further
concordance queries.
3. Two methodologies in lexical semantic representation
Approaches to lexical meanings fall roughly into two groups, which can be (superficially) labelled
546
as logic- and communication-centered paradigms2. The first paradigm assumes that lexical meanings
are concepts that belong to an ontology, which represents real-world objects and their properties.
Lexical items may refer to one or several concepts and by virtue of this reference they are endowed
with a meaning. This relationship between words and meanings is primary with respect to
communication, since the ontology exists and the mapping from words to concepts is defined
independently from any possible act of communication. Within computational approaches to lexical
semantics, good representatives of this paradigm are WordNet (Miller, 1990) and the Explanatory
Combinatorial Dictionary, ECD, (Mel'chuk, 1988).
The second paradigm assumes the primacy of communication: human languages are not aimed at
the correct representation of the world, but at the communication of experience. In this view, language
is a tool for acting in the world, and words are hints, which refer to meanings intended by the speaker.
This view can be defined as the meaning-as-use position. It is also shared by a wide community of
philosophers of language and linguists, for example, Wittgenstein, Harman, Halliday. If the meaning
of a word depends on its contribution to the ongoing exchange between the speaker (S) and the hearer
(H), it should be analysed in terms of its occurrence in an utterance taken in its context. The two
paradigms seems to be contradictory. At the same time, they are complementary both in terms of their
purposes and their results. Thus, they can mutually benefit from their interaction. On the one hand,
the proposed description utilizes lexicographical resources such as WordNet, ECD, and (Apresjan,
2000), as well as the analysis of verbs of motion from (Levin, 1993). On the other hand, it is based on
the communication-centred systemic-functional linguistics, SFL (Halliday, Matthiessen, 1999).
The communication-oriented approach exercised in the project contrasts to the representation of
meanings of lexical items as definitions in a dictionary. A dictionary entry lists senses as concepts that
can be referred to by means of the respective lexical item. Each concept in the list of senses is
considered as a separate item, which may be related to other concepts, which are other senses of the
same word, but this is not specified explicitly in its definition. When a word is used in an utterance,
its meaning is an element which is selected from the list of senses. A use of a word is considered
ambiguous, when it refers to more than one element. The length of the list of senses of a word
depends on the word and the lexicographer, however, the list is typically long. For example, for the
verb leave it contains 17 senses in WordNet, 15 senses (without idioms) in the Random House
Webster's, and 31 senses (also without idioms) in the Oxford English Dictionary3. Analogously,
(Apresjan, 2000) analyzes 19 senses of vyjti, which is the most typical translation equivalent of leave
in Russian.
When a formal lexicographic description (of the type found in WordNet or ECD) is applied to
corpora, two problems arise. On the one hand, some examples of real and felicitous usage do not fit
into the fixed list of senses in the definition of a lexical item. On the other hand, some examples fall
simultaneously into several senses. This is not related to the disambiguation, since all the senses are
relevant for a human judge, who also does not consider the usage as ambiguous. Another problem
concerns translated texts: the piece of reality described in parallel texts (or parallel translations of the
same text) is assumed to be essentially constant, but its lexicogrammatical realization varies
significantly and is not constrained to choosing different synonyms or translation equivalents.
The concordance software and the parallel aligned corpus (reported above) provided the possibility
to analyse English and Russian corpora with respect to:
1. the usage of verbs referring to motion, e.g. go, leave, vyjti;
2. the usage of nouns referring to natural-kind objects, such as names of trees and animals, e.g.
camomile, rabbit;
3. the usage of adjectives referring to the size.
The annotation of lexical features was based on their senses from WordNet for English and ECD
and (Apresjan, 2000) for Russian. The experiment shows that, in the case of verbs of motion, 6% uses
of English verbs in the corpus do not fit into any sense from WordNet, while about 35% are
ambiguous, i.e. more than one sense can be used for the annotation, in spite of the fact that their use is
not ambiguous. For example, WordNet contains two subsets of senses of leave, which refer,
respectively, to leaving a place and a person. The first case contains two senses: go away and move
2According to (Matthiessen, Bateman, 1992: 54-55) the origins of both logical- and communication-
(rhetorical) centered paradigms dates back to the very beginning of thinking about language in the
Western tradition. Nowadays, the two perspectives are partly mirrored in the opposition of formal and
functional linguistics.
3 The senses for the noun and phrasal verbs were not counted.
547
out of. In many utterances the distinction between them cannot be drawn, so the use should be
considered as ambiguous. The second case requires the death or the divorce, so no sense from
WordNet is applicable to:
(1) But her sister sat still just as she left her
The same situation is true for size adjective. Properties of objects with respect to their size are
particularly important for the story of ‘Alice', but the usage of such words as large, little, small does
not always follow their typical definitions, for example, of large as ‘greater than average size,
quantity, or degree'. Each time, when such properties are mentioned, they refer to more than just
physical characteristics of an object in comparison to its average size, because the size of an object is
mentioned only if the reference to it is appropriate for some purpose of the author, often in
comparison to the size of Alice:
(2) a small passage, not much larger than a rat-hole (so that Alice could not pass through it with
the size she had at that moment)
(3) [she went on crying] until there was a large pool all round her.
As the result, some references to the size are not translated into Russian at all or are expressed in a
different way. For example, in Nabokov's translation of ‘Alice', (3) is rendered as:
(4) … posredine zaly obrazovalos’ glubokoe ozero
in-center-of hall appeared-3sg deep lake.
Large vs. glubokij (deep, in this sense) and pool vs. ozero (lake) cannot be considered as translation
equivalents in a normal dictionary. From the logical viewpoint, “a large pool” is in no way
synonymous to “a deep lake”. Yet, basically the same meaning is delivered in the translation. As for
annotated occurrences, 11% uses of English size adjectives are not covered in WordNet and about
65% are ambiguous. This is partly related to the unclear design of lexical entries for size adjectives,
for example, WordNet has the following senses of little:
1. limited or below average in number or quantity or magnitude or extent;
5. of little importance or influence or power; of minor status;
6. (informal terms) small and of little importance;
8. contemptibly narrow in outlook, e.g. "a little mind consumed with trivia"; "petty little
comments";
11. used of persons or behavior; characterized by or indicative of lack of generosity.
It is unclear which sense is implied in:
(5) - Oh, you wicked little thing! - cried Alice, catching up the kitten, and giving it a little kiss to
make it understand that it was in disgrace.
Both occurrences of little in the example correspond simultaneously to several senses from the set:
little thing means 1, 5, 6 and 8 (also, little4--young), while a little kiss means 1, 6 and 11.
4. Words as Resources for Communication
The description reported in the previous section is mostly negative: the real usage of words does not
necessarily conform to their definitions in a dictionary. However, the concept and the format of
modern monolingual dictionaries depend on the history of their development4 and on the function
they serve in the society, namely, to be an authoritative source, which helps in understanding a
specific use of a word or in checking/ensuring the correctness of its use. As the result, a dictionary
entry is designed as a list of senses that denote to events, objects or their properties. In other words, it
statically describes the result of the dynamic process, when words are used by S to refer to events,
objects, and their properties5. Also, S uses words not only for the purposes of referring, but also for
acting on H by means of referring.
The model, which is proposed in the paper, is aimed at the description of how words are used in
purposeful communication. The model is based on Systemic-Functional Linguistics (SFL), cf. analysis
of its traditions for representing lexical meanings in (Wanner, 1997). In spite of all the differences in
4 Origins of the tradition of sense enumeration in a monolingual dictionary are related to the
development of printed discourse, particularly the new periodicals, in England in the eighteenth
century. This brought about a reevaluation
of the nature of meaning, cf. (Kilgariff, 1997).
5 The Collins COBUILD English Dictionary is an exception in this respect; it presents senses
according to the communicative potential of lexical items.
548
approaches, the computational formalism for describing lexicogrammatical meanings in SFL is the
systemic network, which represents choices between interrelated oppositions. For instance,
classification of the English mood starts with the features ‘indicative’ vs. ‘imperative'. Semantically,
it corresponds to the opposition of speech acts referring to exchange of information vs. issuing
commands. Thus, the grammatical choice is controlled by an inquiry to relevant parameters, which
are beyond the grammar, cf. (Matthiessen, Bateman, 1992).
The proposed communication-oriented description of lexical items separates the potential of
possibilities, which defines possible usage of lexical items and is represented by oppositions in the
systemic network, from instantiation of the potential, which is developed for their instances in the
context of an utterance. The set of features, which are fired for a lexical item as the result of its use in
the discourse, may correspond to its sense in a dictionary. However, specific communicative goals of S
or S's idiolect may also result in a different set of features. The meaning of a lexical item in the
context depends not only on the set of features chosen for itself, but (a) on its contribution to the
lexicogrammatical structure of the utterance (the principle of compositionality) and (b) on inquiries,
which are responsible for their choice, i.e. how features correspond to extralinguistic concepts,
traditions in the register and to S's communicative goals. SFL also distinguishes between
paradigmatic classifications, which are represented by the feature inheritance network, and
syntagmatic realizations, which are implied by the choice of features. In the case of lexical semantics,
realization statements constrain the choice of lexical items.
The proposed description also addresses the issue of multilingual differences between senses. The
systemic network is based on the notion of (multiple) inheritance of features. The least delicate, more
general, choices tend to be shared across languages, while more delicate choices tend to be languagespecific
(Bateman, et al, 1999). For example, many languages have the type of motion verbs and also
distinguish between the motion towards or away from the reference object, as its subtypes, e.g. enter
vs. leave. However, in Russian there are further subtypes, which are lexicalized using prefixes, e.g.
the away-from motion subtype can be designated by such verbs as vykhoditj, otkhoditj, uhoditj (all of
them are typically translated as leave), but they denote different configurations of source and
destination properties: vyhoditj implies motion from within the source, otkhoditj implies motion from
the vicinity of the source, ukhoditj implies motion towards a remote destination. Such subtypes are
language-dependent choices. Thus, the systemic network represents both commonalities and
differences between languages.
The following subsections contain systemic network fragments that are required for representing
lexical semantics of verbs of motion and size adjectives. The description is oriented towards automatic
generation of expressions for denoting respective events and properties.
4.1 A fragment of the systemic network for verbs of motions
The most designations of motion processes in English and Russian are based on the following
oppositions6:
The Type of Motion: physical vs. imaginary
(6) He left the room vs. He left the job.
Most often, the reference to physical or imaginary motion is not an inherent feature of a lexical
item, since either interpretation is possible depending on context: to go through a lot of trouble; to
advance the claim, to jump to the conclusions. Within the physical motion there may be a difference
in the manner of motion: run, crawl, fly, or the manner is left unspecified. The lexical item may also
express the cause for the motion. Two clear cases are (a) external circumstances, e.g. to quit the job
(compare to to leave the job); and (b) habits or legal requirements, e.g. withdraw (Apresjan, 2000).
6 There are other oppositions that influence lexical meanings of verbs of motion, e.g. activity vs.
achievement (Levin, 1993). However, they are not special for verbs of motion.
549
Within the imaginary motion subtype, the opposition is drawn between those processes that retain
some kind of metaphorical motion and those that use verbs of motion to designate an activity without
an overt relation to spatial properties, e.g. The boy left school in the middle of the year (when an
institution is considered as a space to be left) vs. He had left off writing on the slate. In Russian, the
number of options for the imaginary motion includes, among others, processes for changing one's
status with respect to a social institution (cf. Figure 2), in particular:
(7) vyjti, vypisat'sja iz bol'nicy (a patient leaves a hospital);
(8) vyjti, osvobodit'sja iz tjur'my (a prisoner leaves a prison);
(9) vyjti na rabotu, vyjti na pensiju, vyjti v otstavku
(to start one's duties, to become a pensioner, to resign);
(10) vyjti iz otpuska (to end vacation).
The Direction: unspecified vs. towards vs. away
(11) She ran for two hours vs. She entered the room.
A lexical item may either designate an inherently directed motion with respect to a reference point
(arrive, depart, descend, return) or be indifferent to it (roll, slide, walk). In the case of inherently
directed, it can be treated as away from a reference point or towards it (to enter a room vs. to leave a
room). This opposition is also applicable to the imaginary motion.
The Focal Point: source- vs. destination-foregrounded
(12) He came in with a teacup vs.
(13) A torrent of Italian music came from an open window.
This opposition differs from the direction of motion with respect to the reference point, since the
motion may be directed towards it, but the source of motion is foregrounded, cf. (12) and (13). The
opposition does not necessarily concerns the semantics of the prepositional phrase. The meaning of
Figure 2: The systemic network for Russian verbs of motion (the “away” subtype).
550
the focal point of an utterance can be incorporated into the verb as well, cf. ausgehen (to go out for an
entertainment) in German or vyjti v svet (a publication is issued) in Russian.
Thus, the example (1) can be represented by the following sequence of features: motion-process,
physical, nonspecific, directed-motion, away, destination-fg.
4.2 A fragment of the systemic network for size adjectives
As with verbs of motion, the investigation of size adjectives goes beyond the domain of pure size
descriptions. The following oppositions are considered (they are shared in English and Russian):
The type of measurement: spatiotemporal vs. amount
Firstly, the majority of adjectives which refer to size are not specific for this purpose. They may also
refer to the description of quantities, e.g. little door vs. little hope vs. little pattering of feet. From the
cognitive viewpoint, a reference to size implies a mapping from a qualitative state into a onedimensional
space, which represents the measure for the state. So many words that designate the
mapping are naturally applicable both to sizes and quantities. However, the opposite is not true,
because some adjectives are applicable only to amount, e.g. a lot of, insignificant. Secondly, many
words that refer to size can also be used for a reference to time. In language (at least, in Indo-
European languages) temporal qualities are often expressed by the same means as spatial ones, e.g.
little stay, long duration (as well as from April to June). In the case of spatiotemporal measurements,
the properties denoted by an adjective may be either directional (deep, early) or non-directional
(small). It is not easy to classify adjectives along measured directions, an example is the adjective
deep, which has the set of senses referring to all possible directions: a deep well (vertical), a deep
shelf (horizontal), a deep border (broad). However, all the senses of deep are related to the presence of
a remote surface, cf. also metaphorical uses deep space, deep thoughts, deep sleep. The described set
of senses of the English adjective deep corresponds to its Russian translation equivalent glubokij.
The reference of measurement: small vs. big
This opposition is simultaneous with the distinctions in the type of measurement. Both references to
spatiotemporal and amount properties, as well as metaphorical expressions can be classified in this
respect, cf. deep sorrow, shallow thoughts; large breakthrough, small problem. The options for
realization of the opposition in Russian are often based on diminutive suffixes to render the smaller
side of the scale. In the translation of large pool in (4), the contextually important features, such as
the large physical measurement and the water surface (to swim in), are preserved, thus making the
translation contextually appropriate.
The interpersonal dimension: sympathy vs. neutral vs. antipathy
A reference to a property of an object is often aimed at providing a rhetorical impact onto H. The
interpersonal dimension of the way how properties are denoted is responsible for what Sinclair (1991)
calls semantic prosody. The reference to the size of an object/person may be mentioned to justify the
need in taking care of it or being afraid of it. Often, the small scale of size is associated with
sympathy, cf. the example (5). However, the sympathy vs. antipathy opposition can be detected even
in roughly synonymous lexical items. Let's consider examples from the corpus, in which the adjectives
little and small are used to refer to a child, and check their translations7:
(14) ignorant little girl
(D) strashnaja nevezhda (awful ignorant),
(N, Z) durochka (fool-femin-diminut)
(15) I'm a little girl
(D) Ja malen'kaja devochka (I'm a small girl)
(Z) Ja devochka (I'm a girl)
(16) very few little girls of her age
(D) nemnogo najdjotsja devochek ejo vozrasta (few find-refl girls of her age),
(17) A small boy and a begrimed, bowlegged toddler lurked behind them. Malen'kij mal'chik i
zamyzgannyj, kolchenogij mladenec zanajachili gde-to za nimi. (Lolita)
7 Translations with insignificant differences are omitted. D stands for Demurova, N for Nabokov, Z
for Zakhoder.
551
(18) there came a blinding flash, and beaming Dr. Braddock, two orchid-ornamentalized matrons,
the small girl in white, and presumably the bared teeth of Humbert Humbert were immortalized
The selection provides two observations. Firstly, the reference to the age may be omitted from the
English original without a loss in the representation of a person. Thus, the reference is used mostly for
adding a specific semantic flavour. The reference to the age is omitted in some Russian translations
and preserved in others. Secondly, in spite of the fact that the most vocabularies, including WordNet,
Random House Webster's and OED treat little and small as synonymous for denoting young children,
each time, when small is used, the context is unfavourable, (though, this observation may be related to
the limited size of the corpus).
5. Conclusions
The project reported in the paper presents an attempt of the contrastive semantic study on an
empirical basis of the corpus-driven description. The study of verbs of motion and size adjective in
terms of their use in texts illustrates that even though the lexicographic research aimed at the
enumeration of senses of a lexical item provides an invaluable resource for lexical semantic analysis
of texts, a dictionary cannot account for all the possible felicitous uses of words. When the list of
senses gets more elaborate, more uses become ambiguous, since often they start to refer to more senses
from the dictionary. The communication-centred perspective onto lexical semantics shifts the
attention from the meaning of a lexical item to possible uses of groups of lexical items. The question
of the lexical semantic analysis in this case is not “What is the meaning of a lexical item?”, but “How
things are meant by lexical items?” This also better corresponds to the social foundations of linguistic
interaction: the end of using language is in acting on others. The future direction of the reported
research consists in the development of the large-scale corpus-based comparative description which
models the usage of motion verbs in English, German and Russian and its implementation for
automatic generation within the KPML environment. Another outcome of the reported research
consists in the development of a parallel English-Russian corpus, which is aligned at the sentence
level. No corpus of this type existed at the beginning of the project. The project also led to the
development of the special-purpose software to create multilingual concordances and interact with
them. Fortunately, Perl as the programming language and XML as the underlying representation
allowed for rapid prototyping.
Acknowledgments
The research reported in this paper has been funded by the research support grant RSS321/1999
from the Open Society Institute, and by the Fellowship awarded by the Alexander von Humboldt
Foundation, Germany.
References
Apresjan, J.D. (2000) Systematic lexicography. Oxford: Oxford University Press.
Bateman J.A., Matthiessen C.M.I.M., & Zeng L.. (1999) Multilingual natural language generation for
multilingual software: a functional linguistic approach. Applied Artificial Intelligence, 13, 607-
639.
Bateman J.A., Teich E, Kruijff G-J, Kruijff-Korbayova I., Sharoff S., & Skoumalova H. (2000).
Resources for multilingual text generation in three Slavic languages. In Proc. Languages
Resources and Evaluation Conference LREC2000, Athens, Greece, May 30-June 2, 2000, pp.
1763-1767.
EAGLES (1996) Recommendations for the morphosyntactic annotation of corpora. EAG-TCWGMAC/
R. Available from ftp://ftp.ilc.pi.cnr.it/pub/eagles/corpora/annotate.ps.gz
Gale W, & Church K., (1993) A Program of Aligning Sentences in Bilingual Corpora. Computational
Linguistics, 19(1):75102.
Halliday, M.A.K., & Matthiessen C.M.I.M. (1999) Construing experience through meaning: a
language-based approach to cognition. London: Cassell.
Kilgariff, A. (1997) "I don't believe in word senses". Computers and the Humanities 31 (2), 91-113.
Levin, B. (1993) Towards a lexical organization of English verbs. Chicago: University of Chicago
Press.
Matthiessen C.M.I.M., & Bateman J.A. (1992). Text generation and systemic functional linguistics:
experiences from English and Japanese. London: Pinter Publishers.
552
Mel'chuk I.A. (1988). Semantic description of lexical units in an Explanatory Combinatorial
Dictionary: basic principles and heuristic criteria. International Journal of Lexicography 1, 165-
188.
Miller G., ed. (1990). WordNet: an online lexical database. International Journal of Lexicography 3
(the special issue).
Paskaleva E, Mihov S, (1997) Second Language Acquisition from Aligned Corpora. In Proceedings
of the International Conference “Language Technology and Language Teaching”, Groningen.
Sampson G. (1995) English for the computer: the SUSANNE corpus and analytic scheme. Oxford:
Clarendon Press 1995.
Sinclair J. (1991) Corpus, Concordance, Collocation. Oxford: Oxford University Press.
Schmied J., Schäffler H. (1996). Approaching Translationese through Parallel and Translation
Corpora. Percy, C.E., C.F. Meyer & I. Lancaster (Eds), Synchronic Corpus Linguistics.
Amsterdam: Rodopi.
Wanner L. (1997) Exploring lexical resources for text generation in a systemic functional language
model. PhD Thesis, University of Saarbrücken.

553
CLaRK - an XML-based System for Corpora Development1
Kiril Simov, Zdravko Peev, Milen Kouylekov, Alexander Simov, Marin Dimitrov2,
Atanas Kiryakov3
The CLaRK Programme
Linguistic Modelling Laboratory - CLPPI, Bulgarian Academy of Sciences
kivs@bgcict.acad.bg, zpeev@bgcict.acad.bg, mkouylekov@dir.bg, adis_78@dir.bg,
marin@sirma.bg, naso@sirma.bg
1 Introduction
In this paper we describe the architecture and the intended applications of the CLaRK system.
The development of the CLaRK system started under the Tübingen-Sofia International Graduate
Programme in Computational Linguistics and Represented Knowledge (CLaRK). The main aim
behind the design of the system is the minimization of the human work during creation of corpora.
Creation of corpora is still important task for majority of languages like Bulgarian where the invested
effort in such development is very modest in comparison with more intensively studied languages like
English, German and French. We consider the corpora creation task as editing, manipulation,
searching and transforming documents. Some of these tasks will be done for single document or a set
of documents, others will be done on a part of a document. Besides efficiency of the corresponding
processing in each state of the work, the most important investment is the human work. Thus, in our
view, the design of the system has to be directed to minimization of the human work.
For document management, storing and querying we chose XML technology because of its
popularity and its ease for understanding. Very soon XML technology will be part of our lives and it
will be the predominant language for data description and exchange on the Internet. Moreover a lot of
already developed standards for corpus description, such as CES (Corpus Encoding Standard 2001)
and TEI (Text Encoding Initiative 1997) are already adapted to XML requirements. The core of the
CLaRK system is an XML Editor which is the main interface to the system. With the help of the
editor the user can create, edit or browse XML documents. To facilitate corpus management we
enlarge the XML inventory with facilities that support linguistic work. We added the following basic
language processing modules: a tokenizer with a module that supports a hierarchy of token types, a
finite-state engine that supports the writing of cascade finite-state grammars and facilities that search
for finite-state patterns, the XPath query language which is able to support navigation over the whole
set of mark-up of a document, mechanisms for imposing constraints over XML documents which are
applicable in the context of some events, database over the XML mark-up and tokenized content of
the documents.
We envisage several uses for our system: 1) Corpus markup. Here users work with the XML
tools of the system in order to mark-up texts with respect to an XML DTD. This task usually requires
an enormous human effort and comprises both the mark-up itself and its validation afterwards. Using
the available grammar resources such as morphological analyzers or partial parsing, the system can
state local constraints reflecting the characteristics of a particular kind of texts or mark-up. One
example of such constraints can be as follows: a PP according to a DTD can have as parent an NP or
VP, but if the left sister is a VP then the only possible parent is VP. The system can use such kind of
constraints in order to support the user and minimize his work. 2) Dictionary compilation for human
users. The system will support the creation of the actual lexical entries whose structure will be defined
via an appropriate DTD. The XML tools will be used also for corpus investigation that provides
appropriate examples of the word usage in the available corpora. The constraints incorporated in the
system will be used for writing a grammar of the sublanguages of the definitions of the lexical items,
for imposing constraints over elements of lexical entries and the dictionary as a whole.
1 The work on the system is currently supported by BulTreeBank project funded by the Volkswagen-
Stiftung, Federal Republic of Germany under the programme “Cooperation with Natural and
Engineering Scientists in Central and Eastern Europe” contract I/76 887.
2 Currently at the OntoText Lab, Sirma AI LTD, Sofia
3 Currently at the OntoText Lab, Sirma AI LTD, Sofia
554
The structure of the paper is as follows: in the next section we give a short introduction to the
main notions of XML language and XPath querying language, the third section describes the main
components of the CLaRK system and their functionality, the last section outlines some directions for
future development.
2 XML technology
XML stands for eXtendable Markup Language (see XML 2000) and it emerged as a new
generation language for data description and exchange for Internet use. The language is more
powerful than HTML and easier to implement than SGML. Starting as a markup language, XML
evolved into a technology for structured data representation, exchange, manipulation, transformation,
and querying. The popularity of XML and its ease for learning and use made it a natural choice for
the basis of CLaRK system. This section presents in informal way some of the most important notions
of the XML technology. For more rigorous and full presentation the reader is directed to the
corresponding literature on the following address: http://www.w3c.org/
XML defines the notion of structured document in terms of sequences and inclusions of elements
in the structure of the document. The whole document is considered as an element which contains the
rest of the elements. The elements of the structure of a document are marked-up by means of tags.
Tags can surround the content of an element or tags can mark some points in the document. In the
first case the beginning of an element is marked-up by the so called open tag written as <tagname>
and the end is marked-up by a closing tag written as </tagname>. For example, TEI documents
include on top level the following two elements:
<TEI.2>
<TeiHeader> ... content of the TEI header element ... </TeiHeader>
<text>
... Content of the text element ...
</text>
</TEI.2>
The tags of the second kind, so called empty elements, are represented as <tag/>. For example,
a line break within a sentence can be represented in the following way:
<s> ... first line of text ... <lb/> ... second line of text ... </s>
Each element can be connected with a set of attributes and their values. The currently assigned
set of attributes of an element is recorded within the open tag of the element or before the closing
slash in an empty element. Some of the attribute-value pairs are by default assigned to some tags and
thus it is not obligatory to list them.
One important requirement for an XML document is that elements having common content must
strictly include one into another. This means that overlap of the elements is forbidden. Such
documents are called well-formed. For example, the following document is not well-formed and thus it
is not an acceptable XML document:
<doc><el1> ... <el2> ... </el1> ... </el2></doc>
XML technology defines a set of mechanisms for imposing constraints over XML documents.
Such kind of a very basic mechanism is the so called DTD (Document Type Definition). A DTD
defines the inclusion of elements and the possible sequences of elements within the content of an
element. These definitions are given as ELEMENT statements in the DTD. Each ELEMENT
statement has the following format:
<!ELEMENT tagname content_definition>
where tagname is the name of the element and content_definition is a definition of the content of
this kind of elements. Besides some reserved words as EMPTY and ANY, the content definition is
represented as a regular expression over tag names. This regular expression determines the tags and
their order in the content of the enclosing element. Additionally, the DTD can contain definitions of
555
the allowed attributes for the elements, entities declaration and others. For more details the interested
reader is directed to the literature on the corresponding notions – see the above address.
An XML document containing elements whose content obeys the restrictions stated in a DTD is
said to be valid with respect to this DTD.
Another important language defined within XML world and used within the CLaRK system is
XPath language. XPath is a powerful language for selecting elements from an XML document. The
XPath engine considers each XML document as a tree where the nodes of the tree represent the
elements of the document, the document itself is the root of the tree and the children of a node
represent the content of the corresponding element. Attributes and their values of each element are
represented as an addition to the tree. Each expression in XPath language is evaluated with respect to
a chosen node in the tree, called context node, and consists minimally of two parts: axis and node test.
Optionally one can impose an additional predicate expression. Axis part determines the direction with
respect to the context node in which the expression will be evaluated. Node test determines the type of
the nodes that we are interested in. This can be text, tag names and so on. Among all nodes of the
required type in the specified direction there can be nodes that we want to choose. These nodes are
further filtered by a predicate expression which can be evaluated as true or false over a node. The
XPath syntax allows for recursive XPath expressions and also union of XPath expressions. The
complete definition of the XPath language can be found in XPath (1999). Here we give a very simple
example, the following expression
/descendant-or-self::gram[contains(string(child::text()),"V")]
will be evaluated to a list of all <gram> nodes such that their textual value contains the letter
"V".
3 CLaRK System
At the heart of the CLaRK system is the XML technology as a set of utilitiess for structuring,
manipulation and management of data. We started with basic facilities for creation, editing, storing
and querying of XML documents and developed further this inventory towards a powerful system for
processing not only of single XML documents but of an integrated set of documents and constraints
over them. The main goal of this development is to allow the user to add to the XML documents a
desirable semantics reflecting the user's goals. Inside the system, the core structure of the
representation of the XML documents follows the DOM Level1 specification (DOM 1998). When an
XML document is imported (or created) in the system it is stored in this internal representation and in
this way the user has access to it only via the facilities of the system. This restriction allows us to
support the consistency of the data represented in the system. We plan to exploit this feature of the
system even further in future for automatic support of construction of XML documents that reflect the
content of a corpus or a set of documents.
The CLaRK system includes the following components: XML Engine, XML Editor, Database,
Document Transformation, Tokenizer, Constraints Engine, XPath Engine and FSA Engine. In this
section we describe each of these components in turn. Some of these components are not directly
accessible by the user and they are used in the other components to support the corresponding
functionality. The most important components of this type are the XPath Engine and the FSA Engine.
The first is a module which evaluates XPath expressions over a document and the second is a module
dealing with compilation of regular expressions into finite-state automata, determinization and
minimization of the compiled automata.
The following screen shot gives an overview of the main interface to the system. On the left side
of the screen the tree view of the current document is displayed, under it is the attribute value table
shows the attributes and their values of the selected element. On the right side of the screen the textual
representation of two documents is given. The message window is at the bottom of the screen.
556
3.1 XML Engine
XML Engine offers a full set of facilities for processing XML documents. This includes DTD
compiler which compiles the element, entities and attribute definitions in a DTD and represents them
in an internal format. For the elements the internal format is a set of finite-state automata
corresponding to the content definition of the elements. These automata are determined and
minimized during the compilation. Attributes and entities are stored as hash-tables. The second
element of the XML engine is the XML parser. This parser transforms an XML document into
internal for the system DOM representation. During the parsing process the parser checks the wellformedness
of the document and reports the corresponding errors. The third component is the
Validator. This module checks the validity of the document with respect to a DTD. Each document
which is loaded in the system has to be attached to a DTD. Once a document is parsed to the internal
representation of the system, it can be saved in this internal representation and the next time when it
is used it will not be necessary to be parsed again. The Validator is active for the currently loaded
document in the editor and when the user is modifies the document the Validator reports the changes
in the validity of the document with pointers to the corresponding wrong elements of the document.
3.2 XML Editor
Access to the system is via a structure-driven editor which allows the user to edit and manipulate
XML documents. Each loaded into the editor document is presented to the user in two or more views.
One of these views reflects the tree structure of the document as described in the previous section. The
other views of the document are textual. Each textual view shows the tags and the text content of the
document. The tags in the textual view are separate elements from the rest of the text and can not be
edited. The user has the possibility to attach to each textual view a filter which determines the tags
and the content of which elements to be displayed in the view. This option allows the user to hide
some of the information in the document and to concentrate on the rest of the information. With
different textual views of the same document the user can attach different filters.
The editor supports a full set of editing operations, such as copy, cut, paste and so on. These
operations are consistent with the XML structure of the document. Thus the user can copy or delete a
557
whole subtree of the document. Some of these operations as search and replace are defined in terms of
XPath expressions. This allows the user to search not only in textual content of the document but also
with respect to the XML mark-up. The most powerful operation here is the XPath replace. This
operation is used for various commands for restructuring the document. Generally, the scenario is the
following: (1) a list of nodes (subtrees, text elements) is chosen by the Source XPath expression. In
this way the elements which will be copied or moved in the document are defined; (2) a list of nodes is
chosen by the Target XPath expression. In this way the place(s) where the source elements will be
copied or moved are defined; (3) the elements from source list are attached to the elements of the
target list. There are several options defining the way of performing of the above action. These
concern such things as whether the elements of the source are copied or cut from the document before
they are attached to the target, the mapping between the source and target elements - it is possible for
the source elements to be attached to each element of the target, or each element of the source to be
attached to the corresponding element of the target. Via different options this operation becomes very
powerful means for document modification or entering new information in case the source is given
not as an XPath expression but as a fragment of XML document or text. In future we plan to allow an
evaluation of the source and the target expressions over different documents and thus to allow their
merging.
The editor allows editing of the document textual content or its structure. The editing of the
structure is supported by the attached to the document DTD. When the cursor is located at some point
in the document structure, the user can enter a child, a sibling or a parent of the pointed element. In
both cases the DTD is consulted and the list of the allowed for this position tags is offered to the user.
3.3 Document Transformation
The system offers a general mechanism for the transformation of some XML documents into
other XML documents. This is done by implementing XSLT language (XSLT 1999). The
transformations can be applied in two modes: globally and locally. When a transformation is applied
globally it is applied to the whole document. In future we plan some transformation to be applied to a
set of documents. When the user wants to apply a transformation locally he/she first selects an
appropriate fragment of the document and then the transformation is applied only to this fragment.
This last option provides a mechanism for construction of a set of transformations which the user
applies depending on the current task and thus avoiding the necessity to write very specific conditions
on the applicability of the transformation.
Some transformations, corresponding to small changes in the DTD of documents (such as
reordering of elements), are generated automatically.
3.4 Tokenizer
XML considers the content of each text element as a whole string that is unacceptable for corpus
processing where one usually requires to distinguish wordforms, punctuation and other tokens in the
text. In order to cope with this problem the CLaRK system supports a user-defined hierarchy of
tokenizers. At the very basic level the user can define a tokenizer in terms of a set of token types. In
this basic tokenizer each token type is defined by a set of UNICODE symbols. Above this basic level
tokenizers the user can define other tokenizers for which the token types are defined as regular
expressions over the tokens of some other tokenizer, so called parent tokenizer. In the system tokens
are used in different processing modules. For each tokenizer an alphabetical order over the token types
is defined. This order is used for operations as comparing two tokens, sorting and similar.
Sometimes in different parts of one document the user will want to apply different tokenizers.
For instance in a multilingual corpus the sentences in different languages will need to be tokenized by
different tokenizers. In order to allow this functionality, the system allows for attaching tokenizers to
the documents via the DTD of the document. To each DTD the user can attach a tokenizer which will
be used for tokenization of all textual elements of the documents corresponding to the DTD.
Additionally the user can overwrite the DTD tokenizer for some of the elements attaching to them
other tokenizers.
3.5 Constraints
General syntax of the constraints in the CLaRK system is the following:
558
(Selector, Condition, Event, Action)
where the selector defines in which node(s) in the document the constraint which is applicable;
the condition defines the state of the document when the constraint is applied. The condition is stated
as an XPath expression which is evaluated with respect to each node selected by the selector. If the
evaluation of the condition is a non-empty list of nodes then the constraints are applied; the event
defines some conditions of the system when this constraint is checked for application. Such events can
be: the selection of a menu item, the pressing of key shortcut, some editing command as enter a child
or a parent and similar; the action defines the way of the actual application of the constraint.
At the moment the following constraints are implemented in the system:
FSA constraints
In this kind of constraints the action is defined as a regular expression which is evaluated over
the content of each element selected by the selector. If the word formed by the content of element can
be recognized as belonging to the language of the regular expression then the constraint is evaluated
as true. Otherwise it is evaluated as false and an appropriate message is given. Because the content of
the elements can contain text and tags, one problem here is how to determine the word which
corresponds to the content of the element. For example, if all wordforms in a sentence are surrounded
by <w> tag then the content of a sentence element will be a list of <w> tags which is obviously not
acceptable. In order to overcome this problem we allow the “letters” used in the definition of the
regular expressions to be of three kinds: tag value, token type, and token value.
Tag value is a string which is the result from the evaluation of an XPath expression over an
element. The appropriate XPath expression for each tag is attached to the DTD. When the content of
an element is converted into a word to be checked by the FSA constraint, for each non-textual element
of the content the corresponding XPath expression is evaluated and the first node in the returned list
is considered as a string. If this first node is a text node then the first token is taken as a value. If the
node is an attribute then the first token of the attribute value is taken. Otherwise the tag of the node is
taken as value.
Token type and token value “letters” correspond to the tokenized textual content of the element.
When the constraint is applied to an element, the element's immediate children are first
processed, tokenizing any textual data, evaluating all tag values and then sent to the FSA representing
the regular expression. Token's string value has a higher priority over token's category. Since the
evaluation of the FSA is linear, the FSA may reject some sequences that are valid. For example
consider the category LAT to be the category for all Latin words. Then having the regular expression
(LAT,<a>)|("to",<b>)
and evaluating it on the element
<el>to<a/></el>
the FSA will answer that the content of el is not valid.
Number Constraints
This kind of constraints are defined in terms of an XPath expression, which is evaluated to a list
of nodes, and MIN and MAX values where MIN and MAX are natural numbers. The constraint is
satisfied (evaluated as true) if the length of the list returned by the XPath expression is between MIN
and MAX.
Value Constraints
These constraints determine the possible children or the parent of an element in a document.
These constraints apply when the user enters a new child or a new parent of an element. In both cases
a list of possible children or parents are determined by the DTD, but depending on the context in the
document an additional reduction of these lists is possible. In case the only possible child of an
559
element is a text then these constraints determine the possible text values for the element. Let us take
as an example the following definitions in a DTD:
<!ELEMENT np ((np, pp) | ...) >
<!ELEMENT vp ((vp, pp) | ...) >
which in part define that a PP can be attached to a NP or a VP. Then let us take the partially
marked-up sentence:
<s>
<np>The man</np><v>saw</v><np>the boy</np><pp>in the garden</pp>
</s>
For the PP "in the garden" there are still two possibilities for a parent - a NP or a VP. But if the
user enters a new information than "saw the boy" is a VP then for the PP "in the garden" there is only
one possible parent - a VP. This information can be encoded in the system as a value constraint for the
parent of PP elements. In future versions of the system we envisage such kind of constraints to be
compiled from grammar represented in a grammar development environment.
3.6 Cascade Finite State Automata Grammars
Having a finite state automata facilities implemented in the system it is relatively simple to use
them for regular grammar development. The CLaRK system incorporates mechanisms for writing
cascade finite state grammars as defined in Abney (1996). The evaluation of the regular expressions
follow the longest match strategy. Again the regular expression can be defined over tag values, token
types and token values. The new category for each recognized word can be presented in the document
as any kind of mark-up, but usually this is done by surrounding tag with appropriate attributes.
3.7 Database
A relational database over the content of the documents imported into the system is established
in order to support an evaluation of limited XPath expressions over a set of documents. In the
database information about the tags, the attributes and the tokens of the documents is stored. The
documents are stored separately as files in internal format. The evaluation of a query with respect to
the database returns a list of documents satisfying the query. These documents are further processed if
this is necessary.
3.8 Other facilities
Here we will describe two more functions of the system which are useful in the process of
corpora development. The first is concerned with sorting elements of a document according to some
keys defined over these elements. The sorting is defined in terms of two XPath expressions. The first
expressions determine which elements will be sorted. This expression is evaluated with respect to the
root of the document as a context node. The second XPath expression defines the key for each element
and it is evaluated for each node returned by the first XPath expression. The list of nodes returned by
the first expression is sorted according to the keys of the nodes. Afterwards the nodes are returned in
the document in the new order.
A concordance tool is implemented on the bases of the XPath engine and the sorting module.
The first step in a concordance construction is to extract the relevant information from the current
document. This is done by an XPath expression which is evaluated and the returns list of nodes is
stored as a separate document. The extracted elements are ordered in an appropriate way with the help
of the sorting module. For example, in case of appropriately marked-up corpus one can extract all
verbs and order them with respect to the first noun on the right side of the verb (not necessarily the
first word on the right side).
The system supports definition of different keyboards for supporting of different languages. At
the moment we support the standard American keyboard and Bulgarian keyboard for entering Cyrillic
letters.
560
4 Future developments
The ClaRk system will be very intensively used within the BulTreeBank Project which just has
started at the Linguistic Modelling Laboratory – see Simov, Popova, Osenova (2001). We plan to
extend the system in the following directions:
External programs. A mechanism for calling external programs which receive as input fragments of
an XML document and returns also fragments of XML document. We envisage the actual
communication to the external programs to be implemented via transformations of fragments of
documents to and from special interface XML documents. In this way an external program will be
declared within the system only once and the user will be able to use the program with XML
documents with different structure.
Schemes of dependencies between elements in several documents. This is in connection with
databases. We can consider each XML DTD as a conceptual scheme over XML documents. Then we
can use a set of DTDs to describe interconnected schemes. We plan to implement support for such
schemes. Because the task can prove to be very hard we will start with one basic DTD and auxiliary
DTDs defining interconnections in table format.
We plan to extend the set of events and actions available to the user for defining the constraints.
The target here will be a macro language for definitions of actions. Also we plan to make the
constraints more active and as an activating event will be used the result from evaluation of some
other contraint. In this way we will have mechanisms for propagation of information from one
constraint to others.
We also plan to add statistical facility for evaluating the quantity characteristic of the documents. The
result of this facility will be a table of the relative frequency of some mark-up to some other mark-up
in the document. Also we plan to add some other views over documents that are not naturally
represented in textual or tree view of XML document such as graph view reflecting the ID references
inside a document or other interpretations of the content of a document.
References
Abney St 1996 Partial Parsing via Finite-State Cascades. In: Proceedings of the ESSLLI'96 Robust
Parsing Workshop. Prague, Czech Republic.
Corpus Encoding Standard 2001 XCES: Corpus Encoding Standard for XML. Vassar College, New
York, USA. http://www.cs.vassar.edu/XCES/
DOM 1998 Document Object Model (DOM) Level 1. Specification Version 1.0. W3C
Recommendation. http://www.w3.org/TR/1998/REC-DOM-Level-1-19981001
Simov K, Popova G, Osenova P 2001 HPSG-based syntactic treebank of Bulgarian (BulTreeBank).
In: Proceedings of Corpus linguistics 2001, Lancaster, UK.
Text Encoding Initiative 1997 Guidelines for Electronic Text Encoding and Interchange. Sperberg-
McQueen C.M., Burnard L (eds).
XML 2000 Extensible Markup Language (XML) 1.0 (Second Edition). W3C Recommendation.
http://www.w3.org/TR/REC-xml
XPath 1999 XML Path Lamguage (XPath) version 1.0. W3C Recommendation.
http://www.w3.org/TR/xpath
XSLT 1999 XSL Transformations (XSLT) version 1.0. W3C Recommendation.
http://www.w3.org/TR/xslt

562
Determining query types for information access
Simon Smith and Martin Russell
School of Electronic and Electrical Engineering, University of Birmingham
smithsgj@eee.bham.ac.uk
Abstract
A body of research exists on the statistical characterization of speech acts, drawing on the features
of the utterance at various linguistic strata. If queries to a multi-functional information access system
could be similarly analysed, it might be possible to determine, without user intervention, the module to
which control should be passed for a given task. Query determination is taken to be a special case of
the general multi-class problem, also exemplified by topic spotting. Therefore, after introducing some
previous work on the problem and providing motivation for the present research, the paper describes a
preliminary topic spotting experiment, along with the results. Future directions for the query typing
work are then set out.
1. Introduction
Most of those involved in information research are actively engaged in designing Information
Access tools. Information retrieval, information extraction, text mining, data mining, spoken document
retrieval, named entity extraction: all are concerned, in the end, with the mapping of some user query
on to some set of relevant data, and passing that data back to the user.
What we wish to undertake is probably less ambitious. We assume that there exists a
comprehensive information access system: discrete components of the system can retrieve documents,
extract information from documents, retrieve data from fixed format files (for example the on-board
address book or calendar manager), and perform certain user requested actions, such as connecting a
phone call. The system would probably be mounted on a PDA-type device, and incorporate a voice
input facility. We further assume that the system, for all its capabilities, is entirely piecemeal: what is
lacking is a central control or shell mechanism which can determine, from the form of the user query,
which of the four components is right for the task at hand.
What we shall be attempting to do, therefore, is to examine the linguistic evidence present in a
query, and on the basis of that evidence determine the query type. This task is similar in many respects
to the assignment of speech act type, utterance type, illocutionary force, or dialogue act type to an
utterance in Discourse Analysis; an important strand of the work, therefore, will be to investigate the
applicability of traditional linguistics insights to a very modern problem.
Research on speech act classification, as well as the strategy proposed for query typing, is closely
paralleled by work on language identification (LID). If it is assumed that word boundaries are known,
as is the case for text LID, “the algorithm used can be derived from basic statistical principles [and] is
not based on hand-coded linguistic knowledge, but can learn from training data”, according to Dunning
(1994). Interestingly, Dunning also notes that the same algorithm has been implemented for species
identification in the domain of biochemistry; moreover, his keyword-counting approach bears
comparison to Garner's (1997) dialogue act classification.
Such techniques can be generalized to a means of solving other multi-class problems: Gorin (1995)
worked on a method of determining the topic of a telephone call, so that it could automatically be
routed correctly. This is an appropriate application for LID, too: for example, in a multilingual culture,
it would be helpful if calls could be directed to an operator fluent in the language of the caller (Lazzari,
Frederking and Minker 1999). Now, imagine a machine translation system which consists of
components for various language pairs. If the source language were not specified by the user, it might
be determined by a keyword-based language spotter, so that control could be passed to the requisite
MT component.
In both these cases, then, we have something akin to a topic spotter acting as a command or shell
function, determining the correct module for the task itself. The same idea can be extended to
information access: a personal digital assistant might have a limited number of command and query
strategies at its disposal, each employing a distinct technology. The choice of strategy would, in
principle, be governed by the user's expectation of the results of the query or command, so one
approach would be to require him or her to select a query type before entering the query itself. Far
better, though, and more in accordance with one's expectations of a personalisable device, if the right
module could be determined from features of the query itself.
563
As well as addressing the practical matter of an approach to a real world problem, the work is also
an exercise in statistical language analysis. We plan to consolidate the extension of topic-spotting to
multi-class problems in general, and to the determination of query types in particular; then build from
training data a set of convenient query features at various significant linguistic strata.
2. A query typology
What, then, are the possible query types, and what applications may they be associated with?
Possible types might include command action, open exploration, factual enquiry, how-to-do enquiry,
buying enquiry. Many of these, it turns out, can actually be decomposed into what might be termed
query primitives: a buying enquiry, for example, could be taken to consist of open exploration, factual
enquiry and command action phases.
Open exploration corresponds to what is technically known as information retrieval. As with a web
search, it is not answers to questions that are returned, but details of (or hypertext links to) documents
in which the user might find the answers. IR retrieves pointers to information, not the information
itself.
Factual enquiry seeks dates, names or other data in direct response to the query. Two subtypes can
be distinguished: the first exploits the relatively new technologies question answering and information
extraction, where the contents of texts and other documents are analysed and data returned. Far more
straightforward is data retrieval, or database enquiry: this would apply to local files of known formats,
such as the address book or calendar manager found on all PDA and similar systems.
Command action needs no elaboration, beyond stating that it is not strictly a “query” type, and the
task application is one of synthesis rather than analysis.
Example queries of each type are now presented.
(1) IR we need a map of/information on Manchester Airport.
(2) IE What terminal does Singapore Airlines use at Manchester Airport?
(3) DR What's the name of the guy I'm meeting at the airport on Tuesday?
(4) CA Call him, please.
It is immediately clear that there is little in the way of an intuitive connection between the query
type and the vocabulary used. It is true, perhaps, that one might associate information with the IR type,
wh- question words with factual enquiries, and call or please with command actions; but it is easy to
think of counter-examples, and most real queries will not contain any of these keywords. In any case,
the examples are not real queries but have been invented by us.
The goal, however, is not to handcraft rules of association, but to observe whether, over a
reasonably large training set, any patterns do emerge; whether, and to what extent the query type can be
predicted on the basis of query keywords.
3. Lexical speech act determination
Samuel et al (1998) used a transformation-based learning (TBL) heuristic algorithm to assign
speech act type to texts. They derived a set of ordered rules and applied them in turn, so that each
utterance was first labelled with a default speech act tag. The application of subsequent rules led to
utterances containing known keywords (dialogue act cues) being re-tagged. In their data, over 71% of
utterances were correctly tagged by this means. Samuel et al noted limitations in their TBL model,
namely that the rule-templates had to be hand-crafted, and that the algorithm could suggest speech act
tags but had no means of asserting its confidence in those tags; they identify a possible solution for the
first problem and a workaround for the second.
Lager and Zinovjeva (1999) applied the TBL algorithm to the Edinburgh Map Task, a corpus which
incorporates dialogue act mark-up, and attained an accuracy of 62.1%. The corpus consists of a series
of conversations about the contents and features of a map, and each utterance is marked up for dialogue
move: Go to the left is an INSTRUCT move, What's behind the church? is classified as QUERY-W, Right
and OK as ACKNOWLEDGE, and so on. In all, there are 12 different dialogue moves.
Garner (1997) undertook the same task, based on the same corpus, using a maximum likelihood
model. Garner reports a classification success rate of 47%, which does not sound particularly promising
until it is recognized that his model treats both moves and keywords as independent, that is to say no
account is taken of contextual information. Furthermore, the approach is based purely on lexical
frequency, and no appeal is made to other linguistic strata.
The annotation of the Map Task corpus is reported on by Carletta et al (1998).
564
There is a clear parallel in Garner's experiments with the query typing work here proposed.
Furthermore, since we propose only four distinct query types, it is reasonable to expect a higher
classification accuracy.
4. Experiment – a topic spotter
As an initial experiment in multi-class problem solving, we built a baseline topic-spotting system,
implementing algorithms of (and effectively replicating the work of) Garner (1997) and Wright et al
(1995). The approach is rather different from that of standard IR systems or web search engines: it
attempts to find the single best solution to a multi-class problem from a very constrained set of
candidates, or alternatively to rank each member of that set. IR, on the other hand, offers in general a
ranked list of possible solutions from a virtually unlimited set of possibilities. In principle, however, the
topic-spotter could be extended to handle a very much larger topic set.
Our aim was to build a system which would correctly identify the topic of an unseen news text, by
examining the frequencies of significant keywords used. To do this, we first decided on ten appropriate
topics: domestic politics, foreign politics, health, finance, crime, sport, media, technology, environment
and education. Two sets of training data were assigned to each category, one set consisting of eight
news stories, the other — a subset of the first — of three stories. All the texts were taken from BBC
Online News, where they are arranged in categories not dissimilar to ours; the fact that multiple
category membership of texts is commonplace in the BBC scheme is what led us to develop our own.
Often, though, the topic of a text seemed to motivate its assignment to more than one class (a story
about Elizabeth Taylor and her AIDS-related work belonged intuitively to both media and health
categories, for example). In such cases, we simply excluded the text from training data.
Once training data had been prepared, the probable topic of an unseen text was computed by
maximizing probability (5) (Garner 1997) over all keywords and all topics:
(5) P m P w m P w m P w m i i i K i ( | , ) ( | , ) ( | , ) ( | , ) x D D D D = 1 2 K
Here, P(x|mi,D) represents the probability that an observation x, instantiated as the unseen text, will
occur in the training data D associated with a topic m. Whenever a word encountered in the unseen text
is found in the training data for a particular topic, its probability of occurrence (P(wj|mi,D)) is computed
by dividing the number of tokens of that word in D by the total number of words in D; if D does not
attest the word in question, the probability defaults to an arbitrarily small value. As (2) shows, these
probabilities are then multiplied together to yield P(x|mi,D). Normally, P(x|mi,D) would in turn be
multiplied by the topic prior probability before maximization, but that step is skipped in this
implementation, as it is assumed that all topics are equally likely.
When the small (three stories per topic) training set was used, 30 out of 50 stories were assigned to
the right topic, while 42 stories were correctly matched by means of the eight-story training set.
4.1 Data pruning
One of the basic principles on which this work is founded is the use of information-theoretic
measures, such as usefulness (Garner 1997) or salience (Gorin 1995) to identify structure which is, in
some sense, optimal for discriminating between the different classes in question. These techniques are
most commonly applied at the word level, either to text, verbatim transcriptions of speech, or the
output of a speech recognition system. However, in principle they are equally applicable to any
symbolic representation of data. For example Gorin et al (1999) have applied similar techniques to the
output of an automatic phone recognizer. Of course, first-order statistics describing the occurrence of
individual phones are unlikely to provide strong evidence for the classification of an utterance.
However, by extending these techniques to phone sequences, it may be possible to detect,
automatically, phone sequences which describe discriminative lexical or even syntactic or semantic
structure. Potentially useful sequences will be characterized by their frequent occurrence in different
contexts and can be detected automatically using methods such as the Context-Adaptive Phone (CAP)
analysis described in Moore et al (1994). Variations in the instantiation of the same underlying
sequence, due to deletion, insertion or substitution of symbols can be accommodated using various
schemes based on dynamic programming (Sankoff and Kruskal 1983), and this could be formalized, for
example, by representing each sequence as a statistical model, such as a hidden Markov model (Jelinek
1999). Similar techniques could be applied to sequences of primitive acoustic symbols, produced by a
data-driven cluster analysis of the output of a speech signal processing system, or, as in the case of the
research proposed here, to more symbolic representations such as those used to describe prosodic
information, such as that described by Shriberg et al (1998).
565
Whilst the experiments conducted so far have been confined to lexical analysis, the ultimate goal is
to process input from an unsegmented speech stream. Thus, it is intended to build models based on subword
units, probably phones.
4.1.1 Usefulness thresholding
Our program then applied (6) (Wright et al 1995) to determine a usefulness score for each
vocabulary item in the training data.
(6) U P w
P w
P w k k
k
k
= ( |
( |
( |
T)log
T)
T)
A word w is thereby said to be useful when it is frequent in training texts of topic T, and occurs
relatively rarely in other texts; the usefulness score describes the discriminatory contribution of
keywords to the topic of the text. Thresholding or pruning is then carried out so that only the n most
useful keywords are searched for when determining the topic of an unseen text.
Table 1 shows, for each news topic, what were computed by the usefulness algorithm to be the top
ten keywords. The lists probably correspond to most people's intuitions of words that would epitomize
each topic.
Table 1
finance computers crime domestic
politics
education environ
ment
foreign
politics
health media sport
banks internet plane party students environ
ment
eritrea cancer film boxing
yen web victim tories schools masts lazio tamoxifen olsen sydney
merger websites suharto kennedy college gm ethiopia breast laurel olympic
bank engines bomb labour oxford fuel eritrean parodi travolta ham
jp lottery victims mp university crops speight fruit films spurs
shares computer cheng ira pupils we monitors everson magazine garcia
sega information police donaldson curriculum oil kosovo boots hardy talent
debenhams sites trial ulster comprehensive pioneer fiji krishnamurthy hurley robson
chase neurons musharraf trimble state environ
mental
ethiopian removed comedy mcgrath
banking identity li romsey i prince electoral pacemaker book edwards
However, when topics were assigned to unseen (test) news stories, greater accuracy was on the
whole achieved when all training data was considered than when thresholding was applied. Table 2
shows how many of 50 stories were correctly identified when usefulness thresholding was applied at
various levels n: that is to say, when only the n most useful words in the training corpus for each topic
were taken into account.
566
Table 2
threshold usefulness modified usefulness salience
10 10 30 10
20 13 31 10
30 12 35 15
40 12 35 19
50 13 37 18
60 12 39 24
70 11 38 28
80 12 35 30
90 13 34 30
100 13 34 29
150 12 35 31
200 16 36 34
249 16 37 35
299 16 37 36
349 16 38 36
399 16 42 42
449 16 39 42
499 19 41 43
549 22 41 42
599 23 41 40
no threshold 42 42 42
Table 2 also shows the results of thresholding under a reformulation of the usefulness calculation
which we term modified usefulness, also suggested by Garner (1997). There is a marked improvement
here, although the best performance is still achieved when no thresholding is carried out. The
reformulation, shown at (7), differs only from (6) in that the absolute probability of occurrence of a
word is ignored.
(7) U
P w
P w k
k
k
= log
T)
T)
( |
( |
The salience results will be discussed presently.
Part of the discrepancy between the results from the two formulations of usefulness arises because
the original form tends to rank function words unduly highly, which leads the application to treat these
items as keywords. Table 3 shows the ranking of the determiner the for each training corpus, under
both formulations of usefulness.
567
Table 3
corpus modified original
computer 931 1032
crime 1026 1132
domestic 896 32
education 946 146
environment 1101 172
finance 912 1012
foreign 928 29
health 1038 518
media 951 1046
sport 1003 957
4.1.2 Salience thresholding
Gorin (1995) postulates salience as “an information-theoretic measure of how meaningful a word is
for a particular device [i.e. task]”. Gorin's work focused on unconstrained speech driven routing of
telephone calls to one of a number of human operators, each dealing with one task, such as reversecharge
calls, credit card calls and directory enquiries. Salience of a word to a class T is computed by
(8).
(8) sal( T )log
T )
T)
w P w
P w
P
) ( |
( |
(
=
P(T) is ignored for our purposes, as all topics are equally likely. We do not know the probability of
the class given the word, but it can be derived from Bayes’ Law, as shown at (9).
(9) P w
P w P
P w
( |
( | (
(
T ) =
T) T)
)
P(T) is again ignored; P(w) is the number of tokens of a particular word in the whole of the training
data (in principle divided by the total number of tokens of any word, but this is a constant); P(w|T) is
the number of tokens of the given word divided by the total word count for the topic, as per the
usefulness calculation.
It will be seen that whereas usefulness compares the likelihood of a token in on-topic and off-topic
texts, salience relates incidence in on-topic texts and all texts, whether on- or off-topic. As Table 3
shows, at low pruning thresholds, the algorithm performs as badly as the original usefulness
formulation, but the performance accelerates towards that of the modified usefulness.
We are not unduly discouraged by the poor performance of the data pruning algorithms. They were
implemented by Gorin and Garner with short utterances, rather than news stories, in mind; we will in
due course be adapting these models to handle queries, which are by their nature short.
What we have attempted at this stage of the research is a baseline maximum-likelihood classifying
program, and we are satisfied that it is working correctly.
5. Speech act determination from non-lexical features
We plan, next, to begin building other query features into the model. Of the knowledge sources
described above, we expect discourse context, the likelihood of the next query type given the current
selection, to prove the least difficult to integrate. This is because it involves the recycling of
information that the program was designed to provide - the value assigned to the first utterance is
simply passed as a parameter in the computation of the next, and, unlike the prosodic input, does not
have to be determined externally.
5.1 Evidence from discourse context
Nagata and Morimoto (1993) trained a corpus of conference-booking dialogues using a trigram
model of utterances classified by speech act type, and attempted to predict, using mutual information,
the following utterance type given the current one. Their approach is entirely probabilistic, and ignores
any linguistic or lexical considerations: they report a classification accuracy of 61%, amongst the nine
speech act types catered for.
568
One would expect, certainly, patterns of speech act type to emerge at the discourse level: given a
question, it is reasonable (if facile) to suppose that the next utterance will be an answer. An example of
a sequence of query types was noted above with reference to the “buying enquiry”, and (1) to (4) were
intended to convey some sort of logical progression in an imaginary user's query trail. Whatever
“answer” the query may generate is not part of the process here, of course, but there is no reason to
suppose that monologue cannot be just as effectively modelled, statistically, as dialogue.
A couple of drawbacks would apply to this approach to query typing. First, in an isolated query,
there is no “next” or “previous” utterance; and if there is, how does one decide whether
contemporaneous queries are in fact related? And how recent does the previous query have to be for it
to count?
Jurafsky et al (1997) labelled a portion of the Switchboard telephone corpus with 42 different
utterance types. They combined discourse grammar (estimation of type based on adjacent utterances, as
with Nagata and Morimoto) with keyword-based classification, and claimed accuracy of 64.6%
(compared to 42.8% for their own keyword-only classification) in assigning utterance types to a test
set.
6. Evidence from prosodic features
Jurafsky et al took into account information from another linguistic stratum in their classification:
namely, prosodic features of the speech stream. They set up several dozen utterance-wide prosodic
feature types, including f0_max_utt (the maximum fundamental frequency reached), rel_nrg_diff (ratio
of RMS energy of final and penultimate phrasing region), and mean_enr_utt (mean speaking rate
value); then they trained CART-style decision trees (Breiman et al 1984) whose task was to distinguish
between two utterance types. The path through the tree, and eventual classification, was achieved
through comparison of the feature values to constants stipulated at decision points. In the proposed
work, we would initially adopt Jurafsky et al's prosodic features.
The CART tree-building technique employs binary recursive partitioning. Given training data
(speech segments, the dialogue act types they represent and the prosodic feature values associated with
them), configurations of parent nodes with two children are hypothesized. Each bifurcation represents a
decision point on a particular feature; the algorithm examines all possible values for that feature, and
attempts to find a splitting rule that maximizes the discriminatory power of the feature. Thus, while the
choice of features is made by the experimenters, the trees are derived by data-intensive means.
Although the incorporation of prosodic information secured only a marginal improvement in
performance, it is fairly well motivated, as there is a significant literature in the Discourse Analysis
domain of theoretical linguistics on the relationship between intonation and speech act type (for
example Crystal (1969), and Brazil (1985) for the Birmingham discourse intonation perspective).
The chief success of Jurafsky et al, in recruiting a prosodic contribution, seems to spring from the
observation that, in English, a yes/no question tends to end with rising intonation. Queries to
information access systems probably only rarely take the form of a yes/no question, so one might justly
be sceptical about the application of prosodic techniques here: it is reiterated, however, that the aim of
the work we propose is to establish what patterns of usage, if any, do emerge, with respect to the query
types described.
There is a body of related work exploiting prosodic features in speech recognition, including that of
King (1998), who integrates language model, dialogue context and intonational information to improve
recognition of spontaneous dialogue speech. Jensen et al (1994) present a scheme for phrase-level
recognition of intonation contours, and show how it can help compute the perceived pitch of voiceless
utterance segments (where no fundamental frequency measurement is available). Hirschberg et al
(1999) found that prosodic features can be used to predict recognition errors: their work was based on
dialogues from the TOOT train information corpus. Carey et al (1996), working on speaker
identification, established that prosodic features are less sensitive to noise distortion than cepstral
coefficients.
7. Conclusions and future work
The next logical step is the extension of the topic spotter to determine dialogue moves in the Map
Task corpus, following Garner, and this work is now under way. The extraction of annotated utterances
to move-specific training corpora, using XML tools, is complete; some preliminary tests with held-out
data indicate that usefulness has more of a role to play in dialogue move assignment than with the
topic-spotter. Experiments using bigrams, rather than single words, will in due course be conducted.
569
The Map Task is not, however, a collection of queries, and identification of suitable query corpora
is a high priority. The ATIS (Airline Travel Information Service) is a possible candidate, as are query
logs from IR systems, some of which are publicly available on the web.
Once an appropriate corpus has been located, we will almost certainly have to do substantial
annotation work. We plan to annotate using XML tags, for two reasons: first, we are currently working
with the XML version of the Map Task, so fewer modifications to the dialogue move program will be
needed. Secondly, the use of what is likely to become the standard annotation of choice would mean
our work was more likely to be of use to other researchers, in the future.
It may be found more practical to use more than one corpus, although there could be
methodological difficulties here, particularly if each corpus held only one query type (at the very least,
the discourse context approach to query typing would thus be ruled out). Crucially, though, the corpora
must represent true queries to either an on-line or Wizard of Oz system, rather than likely-sounding
utterances coined for the purpose.
It may prove necessary to filter out very short queries: it might be difficult to distill any features at
all from a query which consists of just one keyword, where the discourse context, prosodic and
syntactic analyses are apparently unavailable. On the other hand, we might find that query length has a
direct bearing on query type.
In the longer term, we shall be collecting data on the discourse context and prosody of spoken
queries, to supplement the key word/phone information. Ultimately, we hope to employ data fusion
techniques to develop a model of query type determination based on all these linguistic sources.
Once this has been completed to our satisfaction, we should like to consider exploiting a further
source of information present in the query term which we have not discussed here in any detail.
Intuitively, it seems plausible that the syntactic structure of a query may reveal information about its
nature: a command action may be more commonly associated with the use of the imperative form of
the verb, for instance, than other types, and one might expect a factual enquiry to invoke a more
complex grammatical structure, perhaps including long distance dependencies.
Gorniak (1998) used a syntax based feature extraction algorithm to determine the topic of email
messages. It is likely, too, that other topic-spotting techniques may make implicit use of syntactic
information, perhaps through n-tuple based methods or statistical grammars. This possibility will be
investigated further if it is decided to proceed with the syntactic work.
However, we know of no work so far that has attempted to evaluate or use the syntax of short texts
or, specifically, queries. Such work would be very interesting and challenging. It is true that parsers and
part of speech taggers are available and freely downloadable, and assertions about syntactic structure
can be made with more confidence than is usual for prosodic structure, but the difficulty lies in
establishing which features are to be taken into account, and what weighting is to be assigned to them.
The work we have embarked on is of potential importance to a number of disciplines. Information
Access will benefit from the work, although it is not in itself an IA application; the prosody findings
may be of general application in Speech Recognition; ultimately, it is envisaged, the work could form
part of a production PDA device. It is, we believe, a novel and interesting approach, and can reasonably
be expected to make a central contribution to the state of the art.
Acknowledgements
Thanks are due to Graham Tattersall of Snape Signals Research for stimulating discussion on
queries and query typology; to Harvey Lloyd-Thomas of Ensigma Technologies for help with BBC
Online News stories; and to Steve Isard of Edinburgh University for supplying the annotated MapTask,
and for useful advice.
References
Breiman L, Friedman R, Olshen R, Stone C 1984 Classification and regression trees. Pacific Grove
CA, Wadsworth.
Brazil, D 1985 The communicative value of intonation in English. Birmingham, University of
Birmingham English Language Research.
Carey M, Parris E, Lloyd-Thomas H, Bennett S 1996 Robust prosodic features for speaker
identification. In Proceedings of the 4th International Conference on Spoken Language
Processing, Philadelphia, pp 1800-1803.
Carletta J, Isard A, Isard S, Kowtko J, Doherty-Sneddon G, Anderson A 1998 The reliability of a
dialogue structure coding scheme. http://www.cogsci.ed.ac.uk/~jeanc/maptask-codinghtml/
rerevised.html.
Crystal D 1969 Prosodic systems and intonation in English. Cambridge, Cambridge University Press.
570
Dunning T 1994 Statistical identification of language. Technical report CRL MCCS-94-273,
Computing Research Lab, New Mexico State University.
http://www.comp.lancs.ac.uk/computing/users/paul/ucrel/papers/lingdet.ps.
Garner P 1997 On topic identification and dialogue move recognition. Computer Speech and Language
11(4): 275-306
Gorin A 1995 On automated language acquisition. Journal of the Acoustical Society of America 97:
3441-3461
Gorin A, Petrovska-Delacrétaz D, Riccardi G, Wright J 1999 Learning spoken language without
transcriptions. In Proceedings of the Workshop on Automatic Speech Recognition and
Understanding, Keystone, Colorado.
Gorniak P 1998 Sorting email messages by topic, University of British Columbia Computer Science
Dept. http://www.cs.ubc.ca/spider/pgorniak/um/bucfe.html
Hirschberg J, Litman D, Swerts M 1999 Prosodic cues to recognition errors. In Proceedings of the
Workshop on Automatic Speech Recognition and Understanding, Keystone, Colorado.
Jelinek F 1999 Statistical methods for speech recognition. Cambridge MA, MIT Press.
Jensen U, Moore R, Dalsgaard P, B Lindberg 1994 Modelling intonation contours at the phrase level
using continuous density hidden Markov models. Computer Speech and Language 8(3): 247-260.
Jurafsky D, Bates R, Coccaro N, Martin R, Meteer M, Ries K, Shriberg E, Stolcke A, Taylor P, Van
Ess-Dykema C 1997 Automatic detection of discourse structure for speech recognition and
understanding. In Proceedings of the Workshop on Automatic Speech Recognition and
Understanding, Santa Barbara, pp 88-95.
King S 1998 Using information above the word level for automatic speech recognition. Unpublished
PhD thesis, Edinburgh University.
Lager T, Zinovjeva N 1999 Training a dialogue act tagger with the µ-TBL system. In Proceedings of
the Third Swedish Symposium on Multimodal Communication, Linköping.
Lazzari G, Frederking R, Minker W 1999 Speaker-language identification and speech translation,
Multilingual information management: current levels and future abilities, Pittsburgh, Carnegie
Mellon University Computer Science Dept.
Moore R, Russell M, Nowell P, Downey S, Browning S 1994 A comparison of phoneme decision tree
(PDT) and context adaptive phone (CAP) based approaches to vocabulary-independent speech
recognition. In Proceedings of the International Conference On Acoustics, Speech, And Signal
Processing, Adelaide, I: 541-545.
Nagata M, Morimoto T 1993 An experimental statistical dialogue model to predict the speech act type
of the next utterance. In Proceedings of the International Symposium on Spoken Dialogue,
Tokyo, pp 83-86.
Samuel K, Carberry S, Vijay-Shanker K 1998 Dialogue act tagging with transformation-based learning.
Proceedings of the 17th International Conference on Computational Linguistics and the 36th
Annual Meeting of the Association for Computational Linguistics, Montreal, pp 1150-1156.
Sankoff D, Kruskal J 1983 Time warps, string edits, and macromolecules: the theory and practice of
sequence comparison. London, Addison-Wesley
Shriberg E, Bates R, Taylor P, Stolcke A, Ries K, Jurafsky D, Coccaro N, Martin R, Meteer M, Van
Ess-Dykema C 1998 Can prosody aid the automatic classification of dialog acts in conversational
speech? Language and Speech: 41, 443-492
Wright J, Carey M, Parris E 1995 Improved topic spotting through statistical modelling of keyword
dependencies. In Proceedings of the International Conference On Acoustics, Speech, And Signal
Processing, Detroit, pp 313-317.

109
Building a corpus of annotated dialogues: the ADAM experience
Roldano Cattoni, ITC-irst, Trento, Italy, <cattoni@itc.it>
Morena Danieli, Loquendo, Torino, Italy, <Morena.Danieli@loquendo.it>
Andrea Panizza, Universita del Piemonte Orientale “A. Avogadro”,Vercelli, Italy,
<Andrea.Panizza@CSELT.IT>
Vanessa Sandrini, ITC-irst, Trento, Italy, <sandrini@itc.it>
Claudia Soria, ILC-CNR, Pisa, Italy, <soria@ilc.pi.cnr.it>
1. Introduction
ADAM1 is a corpus of annotated spoken dialogues currently being developed as part of the Italian
national project SI-TAL2. Each dialogue is annotated at five levels of linguistic information: prosody,
morphosyntax, syntax, semantics and pragmatics. The five levels were chosen for both practical (their
interest for real applications) and scientific reasons (the possibility to investigate inter-level
phenomena). For each level a corresponding annotation scheme has been defined that provides
annotation instructions, examples and criteria. The result of each annotation is an XML file that
encodes the content of a dialogue with respect to a particular level according to the annotation scheme
of that level. The aim of this paper is therefore to present the ADAM corpus and the experience gained
in defining and building such multi-level corpus. Section 2 describes the ADAM spoken corpus that
includes both human-human and human-machine dialogues in the semantic domain of tourism and
railways transportation. Section 3 provides a detailed introduction to the transcription format and to the
five annotation schemes, one for each level of linguistic information. Section 4 focuses on the
architectural issues of the ADAM corpus: essential requirements that drove the design process – like
corpus reusability – are presented and discussed.
2. Corpus description
The ADAM spoken corpus is a collection of 450 vocal dialogues: they are both human-human
(200 dialogues) and human-machine (250 dialogues). All the dialogues are recordings and
transcriptions of telephone conversations in the semantic domain of tourism and railway transportation.
The format of the audio files is the standard format for telephone signal data recommended by the
SPEECHDAT project directions.
The human-human dialogues are simulated telephone conversations between two experimental
subjects, playing the roles of a travel agent and of a caller, respectively. They had to perform predefined
scenarios, including the request for train or flight timetables, the request for hotel
arrangements, the reservation of both transportation and hotel, and the communication of credit card
information. Each of the two acoustic signals (agent and caller) has been captured by two microphones
(one directional and one “close-talk”), recorded on a digital tape as signed linear PCM 16bit at 16kHz.
The human-human dialogues are longer than the human-machine ones, since the amount of information
exchanged is greater. The total amount of recorded speech is more than 7 hours, for a total number of
58377 words for the human-human dialogues, while the length of the human-machine dialogues
amounts to around 1250 utterances.
The human-machine dialogues were collected on the field: they are interactions between the
automatic telephone information service of the Italian railway company and callers, recorded during an
experimental phase of that service3. The callers called the system during night hours (from nine p.m. to
eight p.m.). The calls came from all Italy: each call is from a different caller and several varieties of
standard Italian are represented in the speech data. The system was designed to provide callers with
information about the Italian train timetable. The dialogue manager allowed the user to enter several
task parameters in a single turn; in case of recognition errors, the dialog manager exploited a set of
repair strategies for recovering from the misunderstanding and being able to access the timetable
database with the correct task parameters. The transaction success rate of the automatic service was
around 85%, so 15% out of the dialogues present miscommunication phenomena. The speech signal
was recorded at 8kHz and stored according the PCM-Ulaw 8 bit protocol.
1 The acronym ADAM stands for “Architecture for Dialogue Annotation on Multiple Levels”, see Soria, Cattoni and Danieli
(2000).
2 The ADAM Corpus will be released by the end of 2001. A pilot set of 30 dialogues is currently available upon request.
3 The automatic service is based on CSELT speech and dialogue technologies (see Baggia, Castagneri, and Danieli 2000 for some
details on service architecture and system performance).
110
3. Transcription and annotation practices
Each dialogue in the ADAM corpus is represented by an orthographic transcription (physically an
XML file), which in turn is linked to an audio file containing the corresponding recording. In addition,
the transcription of each dialogue is associated to five XML annotation files, according to five different
levels or layers of linguistic information, namely prosody, morphosyntax, syntax, semantics and
pragmatics. The five levels of annotation were mainly chosen in consideration of their interest for
practical applications of the annotated material. In spite of the number of levels considered, and their
sometimes conflicting requirements, we tried to develop a coherent, unitary approach to design and
application of annotation schemes. In particular, in developing the different annotation schemes for the
five levels envisaged, attention was paid to be consistent with criteria of robustness, wide coverage and
compliance with existing standards and previous efforts in annotation of spoken dialogues. As a
general criterion, however, attention has been paid so as to design the various annotation schemes and
transcription conventions to be general enough to accommodate as many different annotation practices
as possible.
3.1 Transcription
The representation of dialogues in terms of orthographic transcription implies making several
choices about which aspects and phenomena of dialogues to represent. In accordance with the
EAGLES recommendations for the representation of spoken language (see Gibbon 1999), the
following information is made explicit:
- turns and speakers: according to common practice, a turn is identified with a stretch of talk
produced by a single speaker. Each turn is numbered and the corresponding speaker is identified
with a conventional label
- words: the transcription conventions adopted represent each recognisable word in orthographic
form; non-recognisable words and word fragments are represented through approximated
orthographic rendering of the perceived sound
- pauses are represented by means of specific symbols; length of pauses is also specified
- hesitation signals or fillers are given a specific status and classified as a separate category
- non-linguistic phenomena such as coughs, laughs or noises from surrounding context are classified
and signalled via dedicated symbols.
For an illustration, see the following example:
t_002B: {a} buongiorno mi scusi mi chiamo annamaria degasperi [ breath ] senta io avrei bisogno di prenotare un
treno [ ... ] [ breath ] che parte da roma lunedi` magari verso le otto di mattina non piu` tardi [ ... ] [ breath ] {e} [ puff
] per verona pero` per cortesia mi basterebbe un posto solo [ puff ] c’ e` qualcosa [ puff ]
As a general criterion, we follow the practice of recording only actually pronounced words,
without making assumptions about the nature and type of words in cases of word truncation and
disfluency phenomena. The same holds for non-standard forms such as dialectal expressions or
mispronounced words.
In these cases, the actually pronounced word or word fragment is represented. Optionally, the
annotator can further specify whether the form is to be considered as a non standard form or an
interrupted form. It is also possible, if needed, to specify the target form possibly intended by the
speaker. This range of information can be expressed via a set of dedicated attributes that come as
optional extensions of the basic annotation weaponry.
3.2 Prosodic annotation
In the ADAM corpus, the Prosodic Annotation (PA) is used to represent the prosodic structure of
utterances at the suprasegmental level. This structure is represented by providing a subset of the ToBI
scheme (Tones and Break Index, see Silverman at al. 1992).
ToBI is a widely used system since its five different levels of break indexes are able to capture a
variety of intonation phenomena. The ToBI subset used in the ADAM project concerns the annotation
of prosodic phrasing.
The break indexes used are five, from 0 to 4. They are applied on the basis the following criteria:
· index 0: used in clitic groups, and every time that a word is atonic and is leaned on following
words (for example: I [0] am; a [0] car);
111
· index 1: used every time in which there is not a word separation (word boundary stronger than
clitic but without intonational cut, for example: I [0] saw [1] him [1] yesterday, where between the
words “saw” and “him”, and between “him” and “yesterday”, there are neither separation nor
pauses);
· index 2: used to mark anomalous intermediate boundaries (mostly when a break occurs in an
unexpected place, for example I [0] bought [2] a [0] new [1] ca,. where between the words
“bought” and “a”, there is an unexpected pause);
· index 3: used to mark intermediate intonation boundaries.
In the ADAM corpus, utterances of spontaneous speech makes it difficult to distinguish those
cases where it is better to use index 3 from those where index 4 would be more appropriate. Therefore,
index 4 is used according to the following criteria:
· index 4: used to mark complete disjunctions (an obvious case is at the end of turns) and used to
mark intermediate disjunctions too, if and only if they are conclusive, for example in the cases of
clear logic separations and intonation patterns changing (Ok, I booked you a seat. Do you need
more information?...)
In this example, the two parts of the utterance are clearly separated and there is a change in the
intonation pattern (from assertive to interrogative).
In addition to these five break indexes, we can use the following symbols for PA:
· symbol [ p ]: combined with the break index symbol (ex. 2p, 3p) and used every time the boundary
has a feature of hesitation;
· symbol [ - ]: used to signal an uncertainty in attributing levels;
· symbol [u ]: used to signal uncertainty combined with hesitation.
3.3 Morpho-syntactic annotation
The ADAM proposal for morphosyntactic and syntactic annotation is a two-layer annotation
structure, containing respectively information on word category and morphosyntactic features (pos
tagging), and non recursive phrasal nuclei (called chunks). The morphosyntactic annotation level
encodes the following information: a) identification of morphological words and linking to their
corresponding orthographic counterparts; b) annotation of their pos-category; c) annotation of
morphosyntactic features (such as number, gender, person, tense, etc.); d) annotation of their
corresponding lemma. The particular tag set, though adapted to representation of Italian, is compliant
with EAGLES recommendations (Gibbon 1999).
An example is given below:
<mw id=“mw_004” lemma=“BUONGIORNO” pos=“I” mfeats=“X”>buongiorno</mw>
<mw id=“mw_005” lemma=“ESSERE” pos=“V” mfeats=“S1IP”>sono</mw>
<mw id=“mw_006” lemma=“ANNAMARIA” pos=“SP” mfeats=“FS”>annamaria</mw>
<mw id=“mw_007” lemma=“DEGASPERI” pos=“SP” mfeats=“NN”>degasperi</mw>
<mw id=“mw_008” lemma=“VOLERE” pos=“V” mfeats=“S1DP”>vorrei</mw>
<mw id=“mw_009” lemma=“PRENOTARE” pos=“V” mfeats=“F”>prenotare</mw>
<mw id=“mw_010” lemma=“UN” pos=“RI” mfeats=“MS”>un</mw>
<mw id=“mw_011” lemma=“VIAGGIO” pos=“S” mfeats=“MS”>viaggio</mw>
In addition, the tag set is structured into a core scheme, supplying basic means for annotating
morphological information, and a periphery tag set, which serves the purpose of making provision for
further linguistic annotation to be added to obligatory information.
This is the case, for instance, of a set of optional tags devised in order to annotate the so-called
“discourse marker” class of words, i.e. a range of words belonging to several traditional grammatical
categories and characterising themselves as a compact class as to their discursive or dialogic function.
In this case, additional features are provided as an orthogonal dimension to recommended
classification, so as to make it possible to express the fact that a given word is functioning as a
discourse marker:
<mw id=“mw_104” lemma=“LE” pos=“PQ” mfeats=“FS3”>le</mw>
<mw id=“mw_105” lemma=“ANDARE” pos=“V” mfeats=“S3IP”>va</mw>
<mw id=“mw_106” lemma=“BENE” pos=“B” mfeats=“X”>bene</mw>
<mw id=“mw_107” lemma=“SI'“ pos=“B” mfeats=“X” dfeats=“PF”>si'</mw>
<mw id=“mw_108” lemma=“QUINDI” pos=“C” mfeats=“X” dfeats=“MD”>quindi</mw>
<mw id=“mw_109” lemma=“OTTO” pos=“N” mfeats=“NN”>otto</mw>
<mw id=“mw_110” lemma=“E” pos=“CC” mfeats=“X”>e</mw>
<mw id=“mw_111” lemma=“UN” pos=“RI” mfeats=“MS”>un</mw>
112
Robustness and coverage were a crucial aspect in the development of the two schemes, in
particular for what concerns i) syntactic constructions specific of spoken dialogues (ellipses,
anacolutha, non verbal predicative sentences etc.), and ii) disfluencies (repetitions, false starts, trailing
off etc.). The syntactic annotation level is built on top of the previous one and consists in identification
of non-recursive phrasal nuclei (called chunks) and annotation of their category4, as well as of their
internal structure. The preference given to shallow parsing over, e.g., phrase structure trees is chiefly
motivated by the locality of the analysis offered by this approach, a useful feature if one wants to
prevent a local parsing failure from backfiring and causing the entire parse of an utterance to fail. This
is particularly desirable when dealing with particularly noisy and fragmented input such as spoken
dialogue transcripts. For an illustration of syntactic annotation, see the examples below:
<pn id=“pn_004” type=“INT” href=“dial_002_mor.xml#id(mw_004)”>
buongiorno
<h id=“h_004” href=“dial_002_mor.xml#id(mw_004)”/> </pn>
<pn id=“pn_005” type=“FV” href=“dial_002_mor.xml#id(mw_005)”>
sono
<h id=“h_005” href=“dial_002_mor.xml#id(mw_005)”/> </pn>
<pn id=“pn_006” type=“N” href=“dial_002_mor.xml#id(mw_006)”>
annamaria
<h id=“h_006” href=“dial_002_mor.xml#id(mw_006)”/> </pn>
<pn id=“pn_007” type=“FV” href=“dial_002_mor.xml#id(mw_007)”>
degasperi
<h id=“h_007” href=“dial_002_mor.xml#id(mw_007)”/> </pn>
<pn id=“pn_008” type=“FV”
href=“dial_002_mor.xml#id(mw_008)..id(mw_009)”>vorrei prenotare
<d id=“d_001” type=“modal” href=“dial_002_mor.xml#id(mw_008)”/>
<h id=“h_008” href=“dial_002_mor.xml#id(mw_009)”/> </pn>
<pn id=“pn_009” type=“N”
href=“dial_002_mor.xml#id(mw_010)..id(mw_011)”>un viaggio
<h id=“h_009” href=“dial_002_mor.xml#id(mw_011)”/> </pn>
3.4 Semantic annotation
The framework developed for ADAM allows the annotation of the semantic information through
concepts. A concept is a typed structure that, using an ontology (e.g. a set of symbols that encode the apriori
information), represents the semantic information in a synthetic and non-ambiguous form. The
result of the conceptual annotation of a dialogue is therefore an XML file in which a (possibly empty)
collection of concepts is associated to each turn.
For the design of the annotation scheme of the conceptual level we have identified the following
requirements: (1) soundness: the scheme should refer to well studied and formally sound
representational approaches; (2) expressiveness: the scheme should allow the representation of the
content of complex dialogues; (3) minimality: each turn should be annotated in a unique way (this
requirement is rather strong since it is difficult to identify the “best” abstract level for the semantic
content; therefore the requirement actually means that the annotation scheme should provide practical
rules and criteria for this problem); (4) simplicity: the syntactic complexity of the language describing
the concepts is to be minimised; (5) locality: each turn is independent of the previous turns and, in
general, of the dialogue history; (6) portability: the annotation scheme should be domain-independent.
The ADAM annotation scheme takes inspiration from the so called “Frame-based Description
Languages” (Cattoni and Franconi 1990) a framework developed in the field of the Knowledge
Representation. In our scheme a concept is encoded like a “frame”, a typed structure with “slots”. Slots
represent the properties of the concept and its relations with other concepts. Slots are encoded with the
couple <slot-name, slot-value>: the former contains the name of a property, the latter either a simple
value or a reference to another concept. This recursion allows the encoding of complex and structured
semantics information. There are different types of concepts according to the content to be represented
(e.g. “time”, “trip”, “room”).
An example is needed at this point: given the simple sentence “The train leaves from Rome”, the
corresponding semantic annotation is:
<concept id=“c_001” ctype=“trip”>
<slot sname=“transportation-type” svalue=“train”/>
<slot sname=“origin” svalue=“rome”/>
</concept>
4 Syntactic annotation in ADAM is done automatically with manual check.
113
In this case only a concept (of type “trip”) is used, with two slots (properties): “transportationtype”
with value “train” and “origin” with value “rome”.
Complex concepts can be encoded using reference to simpler ones. For example the sentence
“The train leaves from Rome at eight on Saturday fifteen” is annotated with :
<concept id=“c_001” ctype=“trip”>
<slot sname=“transportation-type” svalue=“train”/>
<slot sname=“origin” svalue=“rome”/>
<slot sname=“departure-time” svalue=“*c_002”/>
</concept>
<concept id=“c_002” ctype=“time”>
<slot sname=“hour” svalue=“8:00”/>
<slot sname=“week-day” svalue=“saturday”/>
<slot sname=“month-day” svalue=“15”/>
</concept>
The simpler concept of type “time” (with identifier “c_002”) encodes a specific point in time. The
more complex concept of type “trip” refers to such time to identify the value of the property
“departure-time” - to do this the star ‘*’ character followed by the identifier of the referenced concept
is used.
As far as the ontology is concerned, three categories of symbols may be distinguished: (1)
symbols that identify the type of concept (the value of the “ctype” attribute of <concept>), (2) symbols
that identify the name of concepts’ property (the value of the “sname” attribute of <slot>), (3) symbols
that identify the value of concepts’ property (the value of the “svalue” attribute of <slot>). The user is
free to adopt his/her conventions to encode the three categories of symbols of the ontology;
nevertheless a good reference are the rules and symbols adopted by the C-STAR Consortium (Waibel
1996) for the inter-lingua: they have been developed on the basis of the experience on six different
(Asiatic and European) languages and this appears to guarantee a good inter-lingua portability. For the
semantic annotation of the ADAM corpus we actually adopted the C-STAR conventions: i) all symbols
are English words, ii) complex symbols of categories (1) and (2) are obtained by means of the dash ‘-’
character (e.g. week-day, interval-time), iii) complex symbols of categories (3) are obtained by
means of the underscore ‘_’ character (e.g. new_york).
It is important to emphasise here that the scheme is domain-independent so that the annotation is
portable. In fact even if the domain changes or the ontology is enriched with new symbols, the
annotation scheme and the corresponding representation in XML doesn't change. For example, let us
change the domain from the Transport Information context to that of Hotel Reservation: given the
sentence “I would like a single room in Venezia for Saturday fifteen”, the annotation scheme doesn't
change even if the types of concepts, the name and values of their properties change:
<concept id=“c_001” ctype=“room”>
<slot sname=“quantity” svalue=“1”/>
<slot sname=“type” svalue=“single”/>
</concept>
<concept id=“c_002” ctype=“location”>
<slot sname=“value” svalue=“venice”/>
</concept>
<concept id=“c_003” ctype=“time”>
<slot sname=“week-day” svalue=“saturday”/>
<slot sname=“month-day” svalue=“15”/>
</concept>
Although most of the concepts encode strictly domain-dependent information, some domainindependent
(or cross-domain) concepts do exist, like the temporal expressions. An user is clearly free
to annotate temporal expressions as he/she likes; nevertheless for ADAM we have defined a set of
predefined concepts to represent temporal expressions, taking inspiration from the Verbmobil TEL
(Temporal Expression Language) (Reithinger 1999).
3.5 Pragmatic annotation
In several recent works on dialogue, there is an underlying assumption about the explicative
power of the dialogue act notion for characterising discourse attitudes in human-human and humanmachine
dialogue (for example, Isard and Carletta 1995 and Di Eugenio et al. 1998), and some authors
argue for the use of dialogue act tagging schemes for dialogue system evaluation. The ADAM metascheme5
for pragmatic annotation is based on that widely held assumption, i.e. the pragmatic dimension
5 For an explanation of the concept of “meta-scheme” see Section 4.
114
of the dialogues is characterised in terms of the “linguistic acts and the contexts in which they are
performed” (Stalnaker 1970). The goal of the pragmatic annotation level is the attribution of one (or
more than one) dialogue act tags to each utterance of the dialogue.
The scheme is a modified version of the tagging schemes DASML (Core and Allen 1997) and
SWITCHBOARD-DAMSL (Jurafsky et al. 1997). In particular, it shares with DAMSL the features
used to capture the communicative dimension of a dialogue turn. The ADAM annotation meta-scheme
allows a three-layer pragmatic annotation practice: at the first layer, each dialogue turn is characterised
with respect to its communicative level; at the second layer, the annotation captures the illocutionary
dimension of the utterance(s) included in the turn; at the third layer, the discourse relationships among
different utterances are characterised.
3.5.1 The communicative dimension
Table 1 reports the tags, and some examples provided by the ADAM pragmatic scheme for
annotating the communicative status of the dialogue turns. The annotation of this dimension is largely
inspired by Core and Allen (1997). The four tags may be used by the annotators to characterise the
following aspects:
1. TASK: the turn provides a contribution to the fulfilling of the goal of the conversation
2. TASK-MANAGEMENT: the turn addresses some specific features of the problem-solving process related to
the task;
3. COMMUNICATION-MANAGEMENT: the turn addresses phenomena connected with the maintenance of the
communication channel.
4. OTHER-LEVEL: the communication content of the turn cannot be characterised with any of the previous
three tags (for example, jokes, word-plays, cliché, etc...)
TAG EXAMPLES
TASK “Do you want to
reserve on that
flight?”
TASK-MANAGEMENT “I'm taking note of
your requiremnts for
the flight
reservation. “
COMMUNICATION-MANAGEMENT “Please, hold on!”
OTHER-LEVEL “Better late than
never”
Table 1
3.5.2 The dialog act dimension
The dialog act labels provided by the ADAM pragmatic annotation scheme are reported in Table
2. The pragmatic annotation meta-scheme allows to characterise each utterance of the dialog with one
or more dialog act tags on the basis of the role(s) of the utterance in the discourse. At the beginning of
the annotation practice, we tried to apply to the annotation of this linguistic level the same minimality
principle stated for the conceptual level, i.e. tagging each utterance one and only one dialogue act label.
However, we soon realised that this simplification was not applicable at the pragmatic level, even in a
task-oriented domain and for “simple” (question answering) dialogues. For example, indirect speech
acts were hard to capture. At present, the pragmatic meta-scheme allows to attribute a primary label
(characterising the direct speech act) and one, or more, secondary ones (for coding the indirect act).
The secondary label is optional6.
The dialogue act tags cover a large set of illocutionary functions in a task-oriented domain, and
we believe that the scheme may be re-used, at some extent, for different domains. However, as for the
other levels of the ADAM corpus, the problem of re-usability has been approached in terms of the
pragmatic meta-scheme and its formal realisation (see below, section 4). Under this aspect, the
distinction between the communicative dimension of a turn and the illocutionary content(s) of its
utterance(s) is likely to be re-used in several annotation tasks.
6 Under this respect the pragmatic annotation scheme presented here and applied in the ADAM corpus differs from the one
described in Soria, Cattoni and Danieli (2000).
115
LABEL EXAMPLE
Statement I'm leaving today
Request I'd need a double room
Accept The flight leaving at ten is nice for me
Accept-Part Yes, but I'd need an extra bed for my child
Open-Option Do you want me to reserve the return flight?
Action-Directive Please, reserve two seats on the BA3476
Repeat-Rephrase Oh, you said BA3476, the one leaving at 10 pm
Collaborative-Completion …and I want to leave from NY next Sunday
Conventional-Opening Hello, this is the Tourist Information Desk
Conventional-Closing Good-bye
Backchannel/Acknowledge Yes, of course
Backchannel/Question Is that ok?
Summarize/Reformulation So, you want to leave around 8 p.m.
Or-Question Do you prefer a room with view on the garden or on the street?
Apology Excuse me
Thanking Thank you for calling
Offer-Commit I've to check if there is a reduced fare available
Yes/No Question Do you want to reserve the return flight on
Thursday?
Open-Question Which company do you prefer to travel?
Reject No, I don't like to travel with this air company
Yes-Answer Yes
No-Answer No
Response-Acknowledgement I agree
Dispreferred-Answers No, I'd prefer to have a smoking room
Opinion I believe this is the best solution
Appreciation I enjoyed very much to work with you
Abandoned/Uninterpretable I thin…
Suggestion Perhaps we could try with another travel agent
Signal-Non-Understanding Pardon?
Signal-Understanding I see
3rd-Party-Conversation Fido, stop barking, I can't hear a word!
Other You know, I'd need to take a week off
Table 2: Dialogue-act tag set
4. Architectural framework
The ADAM approach is mainly driven by the need of making the corpus widely reusable across
different research and application purposes. In short, the concept of corpus reusability could be
rephrased as the sum of the two concepts of “cross disciplinary acceptability” and “wide circulability”.
The two concepts refer, respectively, to the fact that a corpus a) either express a consensual or standard
view about the type of information encoded (the content or semantics of annotation) and b) express a
consensual view for what concerns the way information is physically encoded (the encoding syntax or
markup language). The latter goal seems to have been reached through adoption of XML as a de-facto
standard. On the other hand, the former point is more tricky and represents the motivation for the many
well-known standardisation efforts. Adoption of common or standard annotation schemes, however,
seems to be at least only partially practicable, for the very simple reason that establishing what the
needs of future users might be is hardly feasible. An alternative to the search for standards is thus
desirable.
A way out from the standardisation stricture is represented by concentrating our efforts in corpus
design on maximisation of corpus flexibility. We claim that the degree of flexibility of a corpus
depends on the extent to which the annotation is easily and quickly modifiable at a moderately low cost
by subsequent users of the corpus.
To clarify a little, we can think of at least two possible scenarios where a user might need to
customise the annotation provided with a corpus. First, it might be the case that a user wishes to reuse a
corpus which is annotated for several types of linguistic information, but lacks of a particular
annotation type; the potential user could nevertheless be interested in the existing annotations, and
would like to supplement them with a new one. On the other hand, it might be the case that a user is
interested in some annotation only (e.g., pos-tagging or syntactic structure) and s/he might want to
leave aside other annotation types. Reusability of an annotated corpus can thus be thought of as a
function of the extent to which new levels of linguistic information can be added, or uninteresting ones
can be removed. This is what we call the vertical dimension of customisation in annotated corpora.
Second, for each level of linguistic analysis, an annotated corpus is likely to be reused depending on
the extent to which existing annotation can be changed, so as to accommodate different annotation
practices. It is often the case that a corpus which is annotated with a given annotation scheme “hard-
116
wires” the annotation so as it is impossible to replace the annotation without reverting to the raw text
and rebuilding the annotation from scratch, which is enormously expensive. This is what we call the
horizontal dimension of customisation of an annotated corpus.
The extent to which an annotated corpus can be flexible enough to be compliant with these two
requirements clearly depends on the particular choices made at the design level about the organisation
and structuring of annotation.
If, for instance, all types of annotation are flattened onto a single representation level, it is clear
that the customising operations above become hardly feasible. In ADAM we aim at maximising corpus
flexibility by appealing to the two related notions of modularity of annotation and use of annotation
meta-schemes.
The notion of modularity of annotation refers to data architecture (see MATE, Dybkjaer et al.
1998). In an annotated corpus, several different types of annotation or linguistic information may be
present in relation to the same input data. These types of information can be thought of as independent,
yet related, levels or dimensions of linguistic description. We thus can think of a level of prosodic
analysis, another of pos-tagging, another of semantic analysis, etc. By annotation modularity we mean
that the different layers of annotation are to be kept independent one of another. In the ADAM Corpus
synchronisation among the different analyses and between these and the speech signal is ensured by the
different annotations (stored as separate files) making reference to the same input file. This file,
containing the transcription of the dialogue, is in turn linked to the audio file in PCM (a-low or u-low)
format. Support for this structure is provided by the use of XML as mark-up language. By adopting this
structure, annotation layers are linguistically heterogeneous and mutually orthogonal, so that changing
one of them affects others only to a limited extent; layers are nevertheless indirectly related through a)
their hinging on a common reference file (the “raw” text represented by the transcription file); b) the
indirect correlation of the linguistic information they convey. This vertical modularity of the ADAM
approach has interesting consequences for the purposes of reusability.
A potential user of the ADAM Corpus is left free to select, among the proposed levels of
annotation, those which best reflect his/her theoretical and practical interests. (S)he can also feel the
need for adding a new layer of information, not contemplated in today's ADAM realisation. By the
way, level modularity is also of theoretical interest, since most annotation schemes we know differ
mainly in the way pieces of linguistic information categorised, rather than in the intrinsic nature of
these levels.
Moreover, level modularity seems to have a useful impact on our theoretical understanding of the
linguistic phenomena at stake, since it is capable of expressing correlation relationships between layers,
and ultimately between dimensions of linguistic analysis.
Horizontal customisation in annotated corpora can be enhanced by implementing the concept of
annotation meta-schemes. According to our view, an annotation meta-scheme is a general descriptive
framework in which different annotation schemes can be accommodated. In many cases the same unit
of linguistic information can be annotated in different, arguably mutually incompatible ways, which are
nonetheless all compatible with the recommended vertical modularity described above: so it is better to
provide the potential user with the possibility of adopting any arbitrary annotation scheme without
being forced to re-build the annotation from scratch or to forcefully comply with some other annotation
scheme, no matter how standardised. To do so, it is necessary to have a representation format for the
annotation that is general enough for competing schemes to be mutually substitutable. In other words, it
is necessary to make the representation of annotation schemes as much scheme-independent as
possible. It should be noted how the ADAM different annotation schemes do not, in fact, merely
amount to another set of ready-made annotation schemes, but actually are represented in their XML
annotation format in such a way that, for each annotation level, those features that are common to
several competing schemes become slots or descriptive element tags to be associated with linguistic
elements; the values of these attributes can be any arbitrary set of tags. Let us consider, for instance, the
case of pragmatic annotation. The main difference between annotation schemes for this level of
analysis lies in the particular types of dialogue act chosen rather than in the notion of dialogue act
itself, which appears to be uncontroversial. If, however, we adopt a scheme where the basic descriptive
element of any arbitrarily long set of words is the general tag <dialogue act>, further described by an
attribute “type”, different schemes can be applied to the same corpus without totally discarding the
existing annotation: a substitution in the set of values will be enough. Conversion from one annotation
scheme into another is easily done through XSLT transformations. It is our belief that enforcing this
practice in the design of annotation schemes will bring us to more effective corpora exchange and
reuse7.
7 In addition, the meta-scheme can be seen as a tool for effective comparison of alternative annotation schemes.
117
Finally, it must be noted that actual corpus reusability crucially depends also on the physical
format or mark-up language used for corpus encoding8. As already stated throughout the paper, the
mark-up language used for the encoding of the ADAM Corpus is XML. XML proved to be the ideal
candidate for a number of reasons, all related to corpus reusability. First, it is an emerging and
widespread standard, which ensures a good degree of corpus reusability in the times to come. Second,
because of its platform-independence it enhances the potential for wide circulation of the annotated
material, together with a considerable flexibility of use. More crucially, however, XML proved
essential for implementation of the architectural choices described above. Annotation modularity is
supported via extensive use of Xlink elements (DeRose et al. 2000). Each XML element in the
annotation files is actually an hypertextual link which refers to an element (or set of elements) in the
transcription file. All annotations for each dialogue are thus connected to the same input reference
source (the transcription), thus ensuring synchronisation of the different annotations and still preserving
their independence. On the other hand, the concept of annotation meta-scheme is easily implemented
in XML, thanks to translation of the different annotation schemes content-independent. In other words,
a general preference was given towards representing the different annotation tags as values of generic,
scheme-independent attributes of XML elements. In this way the different annotation schemes
(represented as different DTDs) are represented in a generic enough way, so that a future user of the
corpus will only need to change the values of the different attributes for the entire annotation scheme to
be changed. We believe that this approach represents a further value of the ADAM Corpus.
5. Conclusions and future work
In this paper we have described the methodological assumptions, the annotation practice and
general architectural framework underlying the ADAM Corpus, which is a corpus of annotated spoken
dialogues currently being developed as part of the Italian national project SI-TAL.
As far as annotation is concerned, our next step is to proceed to a validation phase, where the
annotations performed by several annotators will be evaluated according to the metrics of (Isard and
Carletta 1995). In addition to provide a concrete annotation experience, we have introduced what we
believe to be an essential aspect to bear in mind in corpus design, namely the requirement of
reusability. We have claimed that, for effective circulation and re-use of corpora, it is essential to make
provision for as many practices of dialogue annotation as possible, as well as approaches to annotation
at different levels, instead of providing fixed levels and schemes of analysis, no matter how
standardised. Corpora will have a chance to be reused as far as it will be easy and relatively
inexpensive to adapt them to different needs and application purposes. Use of XML as mark-up
language is a further step toward this end.
6. References
Baggia P, Castagneri G, Danieli M 2000 Field trials of the Italian ARISE train timetable system.
Speech Communication 31: 355-367.
Cattoni R, Franconi E 1990 Walking through the semantics of frame-based description languages: a
case study. In Proceedings of the Fifth International Symposium ISMIS ‘90, Knoxville, TN, pp
234-241.
Core M, Allen J 1997 Coding dialogues with the DAMSL annotation scheme. In Working Notes of the
AAAI Fall Symposium on Communicative Actions in Humans and Machines, Cambridge, MA, pp
28-35.
DeRose S, Maler E, Orchard D, Trafford B 2000 XML linking language (Xlink). W3C Working Draft,
21 February 2000. http://www.w3.org/TR/xlink/.
Di Eugenio B, Jordan P, Moore J D, Thomason, R 1998 An empirical investigation of proposals in
collaborative dialogues. In Proceedings of the 17th International Conference on Computational
Linguistics and the 36th Annual Meeting of the Association for Computational Linguistics,
Montréal, Canada, pp 325-329.
Dybkjaer L, Bernsen N O, Dybkjaer H, McKelvie D, Mengel A 1998 The MATE markup framework.
MATE Deliverable 1.2. http://mate.nis.sdu.dk.
Gibbon D (ed) 1999 Handbook of standards and resources for spoken language systems. First
supplement, EAGLES LE3-4244, Spoken Language Working Group.
8 For a similar view see Ide and Brew (2000).
118
Ide N, Brew C 2000 Requirements, tools, and architectures for annotated corpora. In Proceedings of
the Workshop on Data Architecture and Software Support for Large Corpora, LREC 2000,
Athens, Greece, pp 1-5.
Isard A, Carletta J 1995 Replicability of transaction and action coding in the Map Task corpus. In
Walker M, Moore J D (eds), AAAI Spring Symposium: Empirical Methods in Discourse
Interpretation and Generation. Stanford, CA, pp 60-66.
Jurafsky D, Shriberg E, Briasca D 1997 Switchboard DAMSL labeling project coder's manual.
Technical Report 97-02, University of Colorado, Institute of Cognitive Science, Boulder,
Reithinger N 1999 Robust information extraction in a speech translation system. In Proceedings of
Eurospeech ‘99, Budapest, pp 2427-2430.
Silverman K, Beckman M, Pitrelli J, Ostendorf M, Wightam C, Price P, Pierrehumbert J, Hirschberg J
1992 TOBI: A standard for labeling English prosody. In Proceedings of ICSLP 1992, pp 867-870.
Soria C, Cattoni R, Danieli M 2000 ADAM: An architecture for xml-based dialogue annotation on
multiple levels In Dybkjaer L, Hasida K, Traum D. (eds), Proceedings of the First SIGDial
Workshop on Discourse and Dialogue, Association for Computational Linguistics, Hong Kong,
pp 9-18.
Stalnaker, R C 1970 Pragmatics. Synthese 22, pp 272-289.
Waibel A 1996 Interactive translation of conversational speech. Computer 29(7): 41-48.

571
Not last, even if least: endangered Formosan aboriginal languages and the
corpus revolution
Dr. Josef Szakos and Amy Wang
English Department, Providence University, Taichung, Taiwan
In our paper we intend to report on the present stages and uses of corpus creation for the Austronesian
languages of Taiwan. Our main attention goes to the Tsouic tribes, including Northern Tsou,
Kanakanavu and Sa’ arua. The latter two can be regarded as highly endangered as there are only dozens
of old speakers left. Our corpus serves a three-fold purpose: Authentic documentation of the oldest
remnants of early Austronesian languages, making them analysable for the linguistic community
around the Globe, and enabling a revitalisation and stabilisation of language use in the aboriginal
communities. As these languages have lacked writing systems until now, the concomitant task of
alphabetisation has to be solved, too. The dozens of hours of transcribed speech data are searchable by
any kwic programs and concordancers. The innovation of our approach is the linking of sound files (the
length of intonational units) with the corpus, and consequently with the output of searches. This makes
not only the instruction of speech patterns for young people easier, but it also provides the researchers
with the phonetic context. The quick comparison of intonation patterns can give additional information
for semantic subdistinctions. The corpus is constantly growing, providing a series of CD-s, while we
also intend to combine it with web-availability. For the young learners of their mother-tongue, we have
created a sympathetic user interface including a combined vocabulary. Remaining problems are the
possible automatization or partial automatization of the recording and segmentation procedure, search
for topics, themes and balancing the corpus by including the speech of more persons of the last
speakers. Since the users, the mother-tongue learners and researchers need an interface in Chinese or
English, we still have to work out better solutions for that (like parallel corpus arrangement). We hope
that the overview and introduction of some problems may raise the attention of some experts who could
solve the further problems.

572
Methods and techniques for a multi-level analysis of multilingual
corpora
Elke Teich Silvia Hansen
Institute for Applied Linguistics, Institute for Applied Linguistics,
Translation and Interpreting (FR 4.6) Translation and Interpreting (FR 4.6)
University of Saarland, Germany; University of Saarland, Germany
& S.Hansen@mx.uni-saarland.de
Department of Linguistics
University of Sydney, Australia
E.Teich@mx.uni-saarland.de
1 Introduction
The present paper discusses the application of a set of computational corpus analysis techniques
for the analysis of the linguistic features of translations. The analysis task is complex in a number of
respects. First, a multi-level analysis (clause, phrases, words) has to be carried out; second, among the
linguistic features selected for analysis are some rather abstract ones, ranging from functionalgrammatical
features, e.g., Subject, Adverbial of Time, etc, to semantic features, e.g., semantic roles,
such as Agent, Goal, Locative, etc.; third, monolingual and contrastive analyses are involved. This
places certain requirements on the computational techniques to be employed both regarding corpus
annotation and information extraction. We show how a combination of commonly available techniques
can fulfil these requirements to a large degree and point out their limitations for application to the
research questions raised.
The paper is organized as follows. Section 2 describes the concrete analysis scenario at hand,
including the corpus design and the kinds of linguistic features we are interested in extracting from the
corpus. Section 3 discusses the application of a range of computational tools in different stages of
corpus analysis, ranging from encoding and alignment in the corpus preparation stage over part-ofspeech
tagging and grammatical and semantic annotation in the linguistic annotation stage to
concordance programs in the information extraction stage. Section 4 concludes the paper with a
summary.
2 The analysis of a multilingual corpus
One of the primary goals of the kind of corpus analysis described here is to test some hypotheses
about the specific features of translations compared to their source language (SL) originals and to
comparable original texts in the target language (TL). The language pair we are interested in primarily
is English-German. Among the hypotheses tested are Toury's law of growing standardization (Toury
1995)/Baker's normalization (Baker 1995) and Toury's law of interference (Toury 1995). The first says
essentially that translations are even more typical of the target language than are original texts in the
same language, exaggerating the typical features of the TL (TL normalization). The second says that
what makes translations a particular kind of text is that the source language always shines through in
one way or another (SL shining-through).
Given the kinds of relations that are referred to in these hypotheses, we need a corpus that consists
of SL originals and their translations as well as comparable original texts in the TL (see Figure 1).
Figure 1: Multilingual corpus and relations between sub-corpora
We make a couple of starting assumptions relating to the kind of analysis that needs to be carried
out to test the hypotheses. First, the difference of translations to SL original texts and comparable
original TL texts is one of degree, and can thus be measured on a quantitative basis. That is, we can
TL original texts
(German)
TL translations
(German)
SL original texts
(English)
573
analyze the relations between sub-corpora in terms of frequencies of occurrence of particular linguistic
features and compare their distributions across sub-corpora. Second, while a text in a language l1 and
its translation into a language l2 are comparable simply because they are in a translation relation, for
two original texts in a language l1 and a language l2 and for two original texts in the same language,
we need to make sure that they are comparable using some other criterion. For a definition of the
notion of ‘comparable', we draw on the concept of register, i.e., linguistic variation according to
function in situational context (cf. Quirk et al. 1985; Halliday 1978).
The features selected for analysis are taken from contrastive and monolingual register analysis
(Biber 1995; Halliday 1998; Beneš 1981; Fluck 1997). They range from syntactic features such as verb
complementation patterns and voice, over semantic features, such as agency, to textual features, such as
theme-rheme structure. As will be seen in Section 3, some of these (e.g., passive) can be extracted on
the basis of sequences of parts-of-speech, but others (such as agentive) have to be extracted on the basis
of manually coded text.
The relation of these features to the testing of the two hypotheses is the following. For example,
for the register of scientific writing it is commonly known that English texts from this register are
characterized by the frequent use of passives. In German, passive also constitutes a register feature of
scientific texts; however, the grammar of German offers other possibilities with similar functions which
are ”quasi passives” (examples (1) and (2)), so that the core passive occurs less frequently in German
original texts than in English original texts of the given register.
(1) Somit lassen sich auch bei diesen Spielen verschiedene Strategien gegenüberstellen.
thus let themselves also with these games different strategies oppose
”For these games, too, it would be possible to compare different strategies.”
(2) Dabei ist eine sehr bemerkenswerte Verlagerung der Schwerpunkte zu verzeichnen.
thereby is a very remarkable shift of emphases to note
”There has also been a remarkable shift in emphasis.”
Comparing the frequency of core passives in German translations to their frequency in English
originals, two things may happen: Either, there is a significant difference, or there is no significant
difference. If there is a significant difference, the frequency of core passives may either be closer to the
one in the corpus of SL originals or it may be closer to the one in comparable TL originals. The first
would be an indication of SL shining-through, the second an indication of TL normalization.1
Syntactic features, such as passive, can be extracted more or less straightforwardly on the basis of
text annotated with parts-of-speech. With more abstract features, this is not possible any more. Agency,
an attribute of the clause with two values agentive (Agent involved; see example (3)) and non-agentive
(no Agent involved; see example (4)) is a case in point. Features such as agency are tested in a similar
way as described above for passive, i.e., distributions are compared across corpora and interpreted as
SL shining-through or TL normalization.
(3) She was moving the horses.
(4) The horses were moving.
Finally, if we want to analyze features such as passive or agency according to different registers,
we would like to be able to formulate queries such as ”Search for all agentive clauses in passive voice
in the register of popular-scientific writing”. This would require that more than one level of annotation
can be referred to at the same time. We will see in Section 3 that this is not trivial with the tools
available to date.
Thus, the following requirements are placed on the tools to be used for the kinds of analysis we
carry out:
· Syntactic features need to be extracted. Searches on raw text, which can well be successful if one
is interested in lexical material, are therefore pointless in the present context. The corpus has to be
annotated at least with part-of-speech information so as to enable the extraction of instances of
particular syntactic constructions.
· Semantic features need to be extracted. Since semantic analysis cannot be carried out
automatically, we need a mechanism for manual annotation, on the one hand, and a query
mechanism that is responsive to that annotation.
1 For details of the methodology and analysis results see (Teich in progress).
574
· Contrastive data need to be extracted. Since we need to carry out contrastive analyses, we need the
tools to be employed to be applicable for more than one language, on the one hand, and we need
querying facilities that are responsive to more than one language at a time.
· Multiply-annotated data needs to be referred to (cf. above).
In the following section we discuss the computational techniques we have employed in corpus
preparation, linguistic annotation and for information extraction. The benefits and limitations of each
technique for the present analysis requirements are assessed.
3 Computational techniques
3.1 Corpus preparation
Since part of the analysis relates to translations and their SL originals, the parallel corpus needs to
be aligned. For this purpose we use the alignment program Déja Vu.2 See Figure 2 showing an SL and
a TL text aligned with this tool.
Figure 2: Multilingual corpus alignment
Déja Vu aligns a text and its translation, storing the aligned texts in one file or in two separate
files depending on the requirements of the information extraction tool used in later stages of analysis.
Files can be exported to translation workbenches and to Microsoft Excel and Access. Figure 3 shows a
Déja Vu output in a TSV (tab separated vector) format.
"17 One such is hydrogen chloride, HCl, and textbooks often write this process
as HCl H+ + Cl-." "17a Salzsaeure, HCl, ist eine solche Verbindung.
17b In Lehrbuechern wird dieser Prozess oft durch die Gleichung HCl H+ + Cldargestellt."
"18 But H+ is a ""bare"" proton, and as it has an overwhelming attraction to
any electron pair in its vicinity it cannot exist apart from a molecule."
"18 Da aber H+ ein ""nacktes"" Proton ist und jedes in seiner Naehe
befindliche Elektronenpaar anzieht, kann es nicht wirklich ausserhalb eines
Molekuels existieren."
Figure 3: Déja Vu alignment format
Also, we encode each text of the corpus in terms of a header that provides some meta-information
(including title, author, publication, translator, etc) as well as register information (field, tenor, mode).
2 http://www.atril.com/
575
Each file is encoded in XML using a modified version of TEI3 (illustrated in Figure 4) and employing a
standard XML editor (here: XML Spy4). The text body is annotated for headings, sentences,
paragraphs, etc.
<tei.2>
<teiHeader>
<fileDesc>
<filename>code_tl_e.txt</filename>
<subcorpus>popular-scientific (trans_en)</subcorpus>
<language>English</language>
<titleStmt>
<title>Code breaking</title>
<author>
<name>Ewald Osers</name>
</author>
</titleStmt>
<publicationStmt>
<publisher>The Overlook Press</publisher>
<pubPlace>Woodstock</pubPlace>
<date>1999</date>
</publicationStmt>
<translation>
<direction>German-English</direction>
</translation>
<sourceText>
<title>Verschlüsselte Botschaften</title>
<language>German</language>
<author>
<name>Rudolf Kippenhahn</name>
</author>
</sourceText>
<registerAnalysis>
<register>popular-scientific</register>
<field>
<experientialDomain>types of code and their function</experientialDomain>
<goalOrientation>exposition</goalOrientation>
<socialActivity>communication</socialActivity>
</field>
<tenor>
<agentiveRole>expert to educated layperson</agentiveRole>
<socialRole>unequal</socialRole>
<socialDistance>maximal</socialDistance>
</tenor>
<mode>
<languageRole>constitutive</languageRole>
<channel>graphic</channel>
<medium>written</medium>
</mode>
</registerAnalysis>
</fileDesc>
<encodingDesc>Modified TEI</encodingDesc>
</teiHeader>
<text>
<body>
…
</body>
</text>
</tei.2>
Figure 4: XML corpus encoding
3.2 Linguistic annotation
Depending on how abstract the linguistic features to be analyzed are, the linguistic corpus
annotation can be done automatically (using morphological analysis tools, part-of-speech taggers or
parsers) or it can be computer-aided (using corpus annotation tools).
A fairly reliable method of syntactic annotation is part-of-speech tagging. The tagger we employ is the
TnT tagger, a statistical part-of-speech tagger that analyzes trigrams, incorporating several methods of
smoothing and of handling unknown words (Brants 1999). The system is trainable on different
languages and comes with the Susanne tagset5 for English and the Stuttgart-Tübingen tagset6 for
German. It includes a tool for tokenization, which is a preparatory step in the tagging process. In the
basic mode, the tagger not only adds a part-of-speech tag to each token, but it omits alternative tags,
3 http://www.tei-c.org/index.html
4 http://www.xml-spy.com
5 http://www.cogs.susx.ac.uk/users/geoffs/RSue.html
6 http://www.sfs.nphil.uni-tuebingen.de/Elwis/stts/stts.html
576
together with a probability distribution. It analyzes between 30,000 and 60,000 tokens per second and
has an accuracy of about 97 per cent. Figure 5 shows a sample output of TnT, which is in a TSV
format.
Figure 5: TnT sample output
When more abstract features are to be coded, annotation has to be carried out manually. Tools
supporting such annotation allow the definition of annotation schemes and support manual coding by
graphical user interfaces (GUI's). One such tool is Coder (O'Donnell 1995). Coder has five
functionalities: chunking-up of texts into units for coding, definition of coding schemes, annotation
of texts with a coding scheme, calculating of basic descriptive statistics for a coded corpus and
outputting concordances. The chunking mechanism chunks up the text into sentences and paragraphs.
If units smaller than sentences need to be annotated, chunking must be done manually. The definition
of a coding scheme is supported by a GUI, additions and changes to schemes are straightforward.
Coding is supported by another GUI that highlights the unit currently being coded and presents the
coding options. The system keeps a record of the codings, on the basis of which a simple descriptive
statistics can be calculated and exported. Finally, there is the reviewing function, which is a
concordance function operating on the coded features. Figure 6 displays a coding scheme used for
coding agency.
Figure 6: Coder's interface for annotation scheme definition
The annotated texts are written out in an XML/SGML-like format. See Figure 7 for an example.
Let VV0
us PPIO2
consider VV0
, YC
by II
a AT1
simple JJ
illustration NN1
, YC
the AT
problem NN1
of IO
depositing VVG
a AT1
coding VVG
with IW
sender NN1
and CC
receiver NN1
. YF
Coding VVG
keys NN2
are VBR
much DA1
like VV0
the AT
keys NN2
we PPIS2
use VV0
in II
our APPG
daily JB
lives NN2
. YF
577
<codings>
<header>
<scheme root=agency>
<system name="AGENCY-TYPE" ec="agency" features="agentive nonagentive">
</scheme>
</header>
<body>
<segment features="agency nonagentive" comment="" ignore=0>The story was about a boy
</segment>
<segment features="agency nonagentive" comment="" ignore=0> who had a bucket a net and a dog
</segment>
<segment features="agency agentive" comment="" ignore=0> and the dog took the bucket
</segment>
<segment features="agency agentive" comment="" ignore=0> and the boy took the net
</segment>
<segment features="agency nonagentive" comment="" ignore=0> and they walked over to go to the
pond to catch a frog
</segment>
<segment features="agency nonagentive" comment="" ignore=0> but when they went
</segment>
<segment features="agency nonagentive" comment="" ignore=0> they looked all over almost all day
</segment>
<segment features="agency agentive" comment="" ignore=0> but they couldn't find the frog.
</segment>
</body>
</codings>
Figure 7: Coder output format
3.3 Information extraction
In order to extract particular kinds of linguistic information from the corpus annotated in the ways
described above, tools for querying the corpus in terms of the features annotated are needed.
For extracting syntactic information, we use the IMS Corpus Workbench (Christ 1994). The IMS
Corpus Workbench is a concordance tool with which it is possible to query for words and/or part-ofspeech
tags on the basis of regular expressions. Moreover, it allows queries on parallel corpora (aligned
translation corpora). The IMS Corpus Workbench consists of two modules: the Corpus Query
Processor (CQP) and the user interface (Xkwic). Figure 8 shows Xkwic with a query for passive
extraction.
Figure 8: User Interface of the IMS Corpus Workbench
578
The query is based on the part-of-speech tags VB.* (forms of the verb ‘be') followed by VVN.*
(past participle) and zero to three words in between. The results are displayed in the KWIC (keyword in
context) list indicating the number of matches as well.
For the extraction of text coded for more abstract features that have been annotated with Coder,
the review and statistics functions of Coder can be used for further processing. With the statistics
function it is possible to create a descriptive statistics of the analysis; the review function is a
concordance function with which it is possible to extract text instances that have been annotated with a
particular feature. See Figure 9, which presents an example of extraction of text annotated with the
feature agentive.
Figure 9: Review function of Coder
It is not possible to investigate multilingual corpora with Coder.
4 Summary and conclusions
The analysis task we are faced with in the corpus-based investigation of the linguistic properties
of translations places a number of requirements on the computational tools to be used in corpus
analysis (cf. Section 2). We have described the application of a set of techniques for the analysis of
multilingual corpora ranging from alignment and encoding over linguistic annotation to information
extraction (cf. Section 3). Taken together, the tools we have discussed can support the kinds of analysis
we carry out, but there are some remaining problems. One has to do with the input and output
representations and formats of the individual tools, the other has to do with information extraction.
Each of the tools used for encoding, annotation and extraction employs different input formats that do
not necessarily match straightforwardly. While for encoding we have employed XML, the IMS corpus
workbench requires as input a tokenized text with syntactic annotations in a TSV format, and Coder
requires as input raw text (segmentation is done within Coder). Also, the outputs that are generated
from coding are again different across tools: TnT produces a TSV format, Coder produces an
XML/SGML-like format. While part of this problem can be dealt with simply by format
transformations (e.g., a TSV format can be straightforwardly transformed into an XML format by a
Perl script, and an XML-like format can be straightforwardly transformed into XML with the help of
XSLT (W3C-XSLT 2000; see Figure 10 for a PoS-tagged text in a simple XML notation), there are
some more principled questions involved here to do with the fact that we do multi-level annotation.
<head_id_h1> <AT1>A</AT1> <JJ>short</JJ> <NN1>lesson</NN1> <II>on</II> <NN2>keys</NN2>
</head_id_h1>
<p_id_p1> <s_id_s1> <VV0>Let</VV0> <PPIO2>us</PPIO2> <VV0>consider</VV0> <YC>,</YC> <II>by</II>
<AT1>a</AT1> <JJ>simple</JJ> <NN1>illustration</NN1> <YC>,</YC> <AT>the</AT> <NN1>problem</NN1>
<IO>of</IO> <VVG>depositing</VVG> <AT1>a</AT1> <VVG>coding</VVG> <IW>with</IW>
<NN1>sender</NN1> <CC>and</CC> <NN1>receiver</NN1> <YF>.</YF> </s_id_s1> <s_id_s2>
<VVG>Coding</VVG> <NN2>keys</NN2> <VBR>are</VBR> <DA1>much</DA1> <VV0>like</VV0> <AT>the</AT>
<NN2>keys</NN2> <PPIS2>we</PPIS2> <VV0>use</VV0> <II>in</II> <APPG>our</APPG> <JB>daily</JB>
<NN2>lives</NN2> <YF>.</YF> </s_id_s2> <s_id_s3> <VVG>Encrypting</VVG> <VBZ>is</VBZ>
<JJ>similar</JJ> <II>to</II> <VVG>hiding</VVG> AT1>a</AT1> <NN1>message</NN1> <II>in</II>
<AT1>a</AT1> <VVN>locked</VVN> <NN1>box</NN1> <YF>.</YF> </s_id_s3> </p_id_p1>
Figure 10: PoS-tagged text in XML format
579
If, for instance, clause annotations as we have done them using Coder are to be integrated with
PoS annotations like the ones given in Figure 10 into one uniform representation, different units of
annotation have to be merged. Again, this would be feasible simply operating on the different formats,
but in a more principled treatment, the units of annotation would have to be defined explicitly in the
first place. This is a typical task of document type definition, as handled by, for instance, XML.
A possible document type definition (DTD) for our annotation purposes could look as displayed
in Figure 11.
<!ELEMENT Sentence (Clause+)>
<!ELEMENT Clause (Phrase+)>
<!ELEMENT Phrase (Token+)>
<!ELEMENT Token (#PCDATA)>
<!ATTLIST Token
Pos NMTOKEN #REQUIRED>
Figure 11: XML DTD for multi-level annotation
This defines a formal grammar for annotation specifying the units of annotation (sentence, clause,
phrase, token) and their attributes (exemplified here with the unit token and the attribute PoS (partof-
speech). ‘+’ denotes ‘one or more occurrences of'.
In information extraction, problems arise for similar reasons. Unless format transformations are carried
out (where possible), different tools have to be employed for information extraction and the corpus can
only be queried with respect to one level of annotation at a time. It therefore seems to be desirable
after all, to have a uniform representation that is built on first principles (see Figure 12 for illustration).
Again, this would require employing a document encoding standard in which document types can be
properly defined.
Perl Perl XSLT
TSV XML TSV XML/SGML-like
TnT XML Spy Déja Vu Coder
Figure 12: Integrated representation
A recent development in such a direction is the MATE system (Mengel 1999; Mengel and Lezius
2000), which allows for multi-level annotation in a uniform representation, using XML. However,
MATE would still need to be tested in a multilingual application of the kind we are involved in here.
Finally, the query mechanisms available in the tools we have tested are either simple Boolean
searches on strings (as in the case of Coder) or they are based on regular expressions (as in the case of
the IMS workbench). This limits the possibilities of corpus querying. In particular, since the queries in
the IMS workbench have to be formulated on sequences of PoS tags, the queries can become quite
complex. In our immediate future work, we are going to test more expressive query systems, such as
the one implemented in G-Search (Keller et al. 1999), which allows searching with context-free
grammars.
To conclude, with a complex corpus analysis task as the one discussed in this paper, we cannot
expect to find the one ideal tool that can deal with all aspects of annotation and fulfil our particular
requirements on information extraction. The concrete tools we have discussed here are exemplars of
standard techniques used in corpus linguistics and we would thus expect other linguists with similar
analysis requirements to run into the same kinds of problems. Finally, we have formulated the
desideratum of a uniform representation of corpus annotation, so that on that basis searches on multiple
levels of annotation can be carried out. This currently remains an unsolved problem in our analysis
scenario.
information
extraction
integrated
representation
(XML)
580
Acknowledgments
Thanks to Peter Fankhauser and Frank Klein for advice on XML.
References
Baker M 1995 Corpora in translation studies: An overview and some suggestions for future research.
Target 7(2):223-245.
Beneš E 1981 Die formale Struktur der wissenschaftlichen Fachsprachen aus syntaktischer Hinsicht. In
Bungarten T (ed) Wissenschaftssprache. München, Fink, pp 185-212.
Biber D 1995 Dimensions of register variation: A cross-linguistic comparison. Cambridge, Cambridge
University Press.
Brants T 1999 TnT - A Statistical Part-of-Speech Tagger (User manual). Department of Computational
Linguistics, Universität des Saarlandes, Saarbrücken, Germany (http://www.coli.unisb.
de/~thorsten/tnt/).
Christ O 1994 A modular and flexible architecture for an integrated corpus query system. In
Proceedings of COMPLEX 94, 3rd Conference on Computational Lexicography and Text
research, Budapest, pp 23–32 (http://www.ims.uni-stuttgart.de/projekte/CorpusWorkbench/).
Fluck H R 1997 Fachdeutsch in Naturwissenschaft und Technik: Einführung in die Fachsprachen und
die Didaktik/Methodik des fachorientierten Fremdsprachenunterrichts. Heidelberg, Groos.
Halliday MAK 1978 Language as social semiotic. Arnold, London.
Halliday MAK 1998 Things and relations: Regrammaticising experience as technical knowledge. In
Martin J, Veel R (eds) Reading Science. Critical and functional perspectives on discourses of
science. London, Routledge.
Quirk R, Greenbaum S, Leech G, Svartvik J 1985 A comprehensive grammar of the English language.
London, Longman.
Keller F, Corley M, Corley S, Crocker M, Trewin S 1999: Gsearch: A Tool for Syntactic Investigation
of Unparsed Corpora. In Uszkoreit H, Brants T, Krenn B (eds) Proceedings of the EACL
Workshop on Linguistically Interpreted Corpora, Bergen, pp 56-63.
Mengel A 1999 Die integrierte Repräsentation linguistischer Daten In: Gippert J (ed) Multilinguale
Corpora. Codierung, Strukturierung und Analyse (11. Jahrestagung der Gesellschaft für
Linguistische Datenverarbeitung). Prag, enigma corporation, pp 115-121.
Mengel A, Lezius W 2000 An XML-based representation format for syntactically annotated corpora. In
Proceedings of LREC 2000, Athens, pp. 121-126 (http://mate.mip.ou.dk).
O'Donnell M 1995 From Corpus to Codings: Semi-Automating the Acquisition of Linguistic Features.
In Proceedings of the AAAI Spring Symposium on Empirical Methods in Discourse Interpretation
and Generation, Stanford University, California, pp 120-124
(http://cirrus.dai.ed.ac.uk:8000/Coder/index. html).
Teich E in progress Contrast and commonality between English and German in system and text. A
methodology for investigating the contrastive-linguistic properties of translations and
multilingual texts. Department of Applied Linguistics, Translating and Interpreting, Universität
des Saarlandes, Saarbrücken, Germany.
Toury G 1995 Descriptive Translation Studies and beyond. Amsterdam, John Benjamins.
W3C-XSLT 2000 XSL Transformations (XSLT), Version 1.0 (http://www.w3c.org/TR/xslt).

581
Accurate automatic extraction of translation equivalents from parallel corpora
Dan Tufiú and Ana-Maria Barbu
RACAI-Romanian Academy Center for Artificial Intelligence
13, “13 Septembrie”, RO-74311, Bucharest, 5, Romania
{tufis, abarbu}@racai.ro
Abstract
The paper describes a simple but very effective approach to bilingual lexicons extraction
from parallel corpora. After briefly describing the method we present the evaluation for six
pairs of languages in terms of precision and recall and processing time. We conclude by
discussing the merits and the drawbacks of out method in comparison with other works and
comment on further developments.
1. Introduction
The vast and continuously growing amount of information available nowadays on the WEB poses
numerous and challenging problems which strongly influenced current approaches to natural language
processing. One of the most obvious tendencies in NLP and related technologies, is a distinct preference for
shallow processing. The term usually implies not only a partial coverage of the difficult problems in
computational linguistics but also, accepting a limited imprecision or inaccuracy in automatic decisions
made in order to achieve a useful degree of language analysis or generation. The motivation for this trend,
to a large extent imposed by the Internet industry and market, is given by the requirement to process in real
time very large quantity of texts. It has been noticed that many useful tasks (document indexing, document
classification, information retrieval, etc) may be achieved when using even a quite superficial processing of
natural language texts. Increasing the amount of linguistic processing in more and more web-based
applications is a constant preoccupation of many R&D professionals and companies. The development of
large linguistic resources (thesauri, ontologies, translation memories, multilingual dictionaries, annotated
corpora etc.) became a very active area both for academic research and industry. As one would expect,
printed dictionaries and lexicons were primary sources for constructing lexical databases. However, relying
on human-dedicated lexical sources was not as successful as expected for computer programs due to various
reasons among which one could mention variation of lexical stock and lexical gaps, shifts of meaning, sense
granularity to fine for automatic discrimination, etc. The costs involved in turning the human-oriented
linguistic knowledge sources into useful computational resources are also high (copyright fees, man-power
costs) so the idea of using computers to extract and organise linguistic information according to specific
needs is natural and significant research is carried along these lines. Combining supervised and
unsupervised linguistic knowledge acquisition is a trade-off between time and costs versus quality and
completeness. As basic language resources (meant for training /learning) are cleaner and larger, the
acquisition of more structured information and higher levels of linguistic knowledge is possible with less
human supervision and with simpler computational means. We showed (Tufiú) for instance that the
accuracy of tagging new texts can go higher than 99% when very clean training corpora are used and
language modelling is adequate with respect to the underlying technology. For this example (based on
HMMs), we argued (TufiúHt. all 2000) that preserving in the language model distinctions that cannot be
reliably made by a specific distributional analysis model is source of noise and the cause of performance
deterioration. The tagging technology is more or less the same in many projects but the accuracy of the
results varies (even for the same language) and the explanations can be found both in the quality of the
training data and (mainly) in the quality of the tagsets.
Aligning parallel texts is a very good example where simple techniques and limited linguistic
knowledge (if any at all) can ensure surprisingly good results for the problem of interest. Since charAlign
(Gale, Church 1993) was published, many variants refinements and improvements of this program were
implemented, but the basic underlying ideas remained extremely simple and thus easy to implement and
fast in operation.
Extracting bilingual dictionaries from corpora can be seen as a very fine-grained alignment process,
were the aligned units are not paragraphs or sentences but words and phrases. Most approaches to this
problem rely on statistical means to build translation lexica from bilingual texts, roughly falling into two
categories: the hypotheses testing approach such as (Gale, Church 1991), (Smadja et all 1996) etc. and the
estimating approach (Brown et all 1993), (Kupiec 1993), (Hiemstra 1997) etc. The first approach involves a
generative device that produces a list of translation equivalence candidates (TECs), each of them being
subject to an independence statistical test. The TECs that pass the test are assumed to be translation-
582
equivalence pairs (TEPs). The second approach assumes building from data a statistical model the
parameters of which are to be estimated according to a given set of assumptions. There are pros and cons for
each type of approach, some of them discussed in (Hiemstra 1997).
Our method hardly fit into one of these two categories, but is closer in spirit to hypotheses testing
approach, by first generating a list of translation equivalent candidates and then iteratively extracting the
most likely translation-equivalence pairs. The candidate list is constructed from the translation/alignment
units (TU). That is to say that the translation of an item in a source language sentence is looked for only in
the alignment corresponding sentence(s) of the target language.
2. Format of the input data
The translation equivalents extraction process does not assume a pre-existing bilingual lexicon for the
considered languages. If such a lexicon exists, it might be used for the validation purposes, but also for
some kind of “land-marking” for increasing the certainty in building the TECs.
Three files physically represent the input data:
·  the source language file, containing one half of the aligned bi-text TS
·  the target language file, containing the other half of the aligned bi-text TT
·  the alignment index for the sentences in TS and TL.
Each part of the bi-text is represented in the same tabular format with one tagged and lemmatised token
per line and sentence mark-up explicitly shown. Table 1 shows excerpts from the contents of the three files.
The first column in the Source/Target file represents the type of the item (lexical token, left/right split,
punctuation etc), the second column contains the word-form while the last column contains the lemma, the
morpho-syntactic description of the word-form and the POS only tag used in the alignment (separated by a
“\”).The SGML alignment file is a cesAlign type of document and specifies the sentence pairing in the two
languages by means of sentence identifiers in the first two files. An alignment unit defines a translation unit
(TU) as composed by the sentences identified by the IDs.
Source file Target file Alignment file
<S FROM=Oen.1.1.1.1> <S FROM=“Oro.1.2.2.1”> <!doctype cesAlign PUBLIC”-//CES//DTD cesAlign//EN”[]>
TOK It it\Pp3ns\P LSPLIT Într- =\Spsay\S <linkList id=“oroen”>
TOK was be\Vmis3s\AUX TOK o un\Tifsr\T <linkGrp id=“oroen.1” type=“body” targtype=“s” domains=“oro
oen”>
TOK a a\Di\D TOK zi =\Ncfsrn\N <link xtargets=“ Oro.1.2.2.1 ; Oen.1.1.1.1 Oen.1.1.1.2 “>
TOK bright bright\Af\A 72.VHQLQ VHQLQ\Afpfsrn\A <link xtargets=“ Oro.1.2.3.1 ; Oen.1.1.2.1 “>
… … …
Table 1: The format of the input data
For instance, in Table 1, the first TU is made of the sentence identified by the id “Oro1.2.2.1” (Într-o zi
VHQLQ «) and the two sentences identified by the ID “Oen.1.1.1.1” (It was a bright…) and “Oen.1.1.1.2”
(Winston Smith, his chin…).
The format of the input data is conformant with different conventions adopted within the MULTEXTEAST
(MTE) project (which developed the “1984” multilingual corpus we worked with). Various preprocessing
steps (tokenisation, alignment and tagging) were initially achieved by using tools developed
within the MULTEXT project, for which MTE was a follow up. Because of interest in this corpus, it was
extended with new languages within the TELRI project and further cleaned up within the CONCEDE
project. We provide more details on the corpus in the “Experiments and results” section.
3. The baseline and the iterative algorithm
Based on the alignment, the first step is to compute a list of translation equivalent candidates (TECL).
This list contains several sub-lists, one for each POS considered in the extraction procedure. Obviously if
the parallel text contains alignment and tagging errors, several real translation equivalents would not be
found because they will not be member of the corresponding TECpos. Each POS-specific sub-list contains
several pairs of tokens <tokenLANG1:tokenLANG2> of the corresponding POS that appeared in the same TUs.
These pairs (translation equivalents candidates-TECs) are generated by a Cartesian product of the set of
tokens (of the given POS) in one half of the TU with the set of tokens (of the same POS) in the other half.
Each pair has attached the number of occurrences of the respective association throughout all the TUs.
The baseline algorithm is represented by a retaining from the list of TECs only those pairs which cannot
be considered to occur just by chance. This hypothesis can be tested by different statistical tests (we used S.
Banerjee's and T. Pedersen's Bigram Statistics Package, which includes chi-square, dice, mutual
information and log-likelihood statistical measures). For instance, when chi-square test is used, the
coefficients (computed for each TEC) given by the formula
583
) ( * ) ( * ) ( * ) (
) * * (
22 21 22 21 21 11 12 11
2
21 12 22 11 * * 2
n n n n n n n n
n n n n n
+ + + +
- = c may be used to select the most likely candidates as
TEPs. For a 99.9% confidence level, the threshold condition for rejecting the null hypothesis (TS and TT cooccurred
by chance) would be l2> 10.83. One could use also a minimal number of occurrences for <TS TT>
(usually this is 3). This baseline algorithm may be enhanced in many ways (using a dictionary of already
extracted TEPs for eliminating generation of spurious TECs, stop-word lists, considering token string
similarity etc.). An algorithm with such extensions (plus a few more) is described in (Gale, Church 1991).
In spite of being extremely simple, this algorithm was reported to provide impressive results (Canadian
Hansard, precision about 98% and recall about 50%). However the response time is not among its assets and
it is not clear how or whether different translations of the same item are extracted.
The iterative algorithm we propose is also very simple but significantly faster than the baseline. It can
be enhanced in many ways (including those discussed above). The algorithm gets as input the aligned
parallel corpus and the maximum number of iterations. At each iteration step, the pairs that pass the
selection (see below) will be removed from TECL so that this list is shortened after each step and
eventually may be emptied. Based on TECL, for each POS is constructed a contingency table (TBLk) as
shown in Table 2:
TT1 … TTn
TS1 n11 … n1n n1*
… …
…
…
…
TSm nm1 … nmn nm*
n*1 … n*n n**
Table 2: TBLk - Contingency table with counts for TECs at step K
The rows of the table are indexed by the distinct source tokens and the columns are indexed by the
distinct target tokens (of the same POS). Each cell (i,j) contains the number of occurrences in TECL of the
<TSi, TTj> TEC: nij = occ(TSi,TTj); ni* = a
=
n
j 1
ij n ; n*j= a
=
m
1 i
ij n ; and n** = ) n (
n
1 j
m
1 i
ij a a
= =
. The selection condition
is expressed by the equation: (1) )} n (n ) n (n q p, | T ; T { TP pj ij iq ij Tj Si
k 3 U 3 " > < = .
This is the key idea of the extraction algorithm and it expresses the requirement that in order to select a
TEC <TSi, TTj> as a translation equivalence pair, at step k, the number of associations of TSi with TTj must be
higher than (or at least equal to) any other TTp (p1j) that are represented in the TBLk. The same holds for
the other way around. If TSi is translated in more than one way, the rest of translations will be found in
subsequent steps (if frequent enough). The most used translation of a token TSi will be found first.
4. Experiments and results
We conducted experiments on one of the few publicly available multilingual aligned corpora, namely
the “1984” multilingual corpus (Dimitrova et al 1998) containing 6 translations of the English original. This
corpus was developed within the Multext-East project, published on a CD-ROM (Erjavec et al 1998) and
recently improved within the CONCEDE project (to be soon released to the research community
CONCEDE's homepage: www.itri.brighton.ac.uk/projects/concede/).
Each monolingual part of the corpus (Bulgarian, Czech, Estonian, Hungarian, Romanian and Slovene)
was tokenised, lemmatised, tagged and sentence aligned to the English hub.
Language Bulgarian Czech English Estonian Hungarian Romanian Slovene
No. of wordforms 15093 17659 9192 16811 19250 14023 16402
No. of lemmas 8225 8677 6871 8403 9729 6626 7157
No.of >2-occ lemmas* 3350 3329 2916 2876 3294 3052 3189
Table 3:The lemmatised monolingual “1984” overview
* the number of lemmas does not include interjections, particles, residuals)
The number of lemmas in each monolingual part of the multilingual corpus as well as the number of
lemmas that occurred more than twice are shown in Table 3.
584
For validation purposes we set the step limit of the algorithm to 4. The evaluation was fully done for
Estonian, Hungarian and Romanian and partially for Slovene (the first step was fully evaluated while from
the rest were evaluated randomly selected pairs).
The evaluators judged a translation pair as correct, partially correct or incorrect. Pairs that were judged
as partially correct (all cases appeared as expected in Hungarian and Estonian) corresponded to English
multiword equivalents of the source words. For instance the Estonian word “armastusministeerium” means
“ministry of love”, but our algorithm found as translation equivalents (armastusministeerium = ministry)
and (armastusministeerium = love). For all such cases we used the BSP package to test whether the correct
multiword equivalent pass a collocation test and if so, we included the partially correct TEP into the class of
correct pairs. Such a decision might be motivated on the grounds that a preliminary collocations analysis
and recognition would have raised this issue.
The figures in Table 4 show the results and the evaluation. In the following the notion of correctness of
a pair is taken in the above-mentioned interpretation.
The extracted bilingual lexicons are available at http://www.racai.ro/bi-lex/. The precision (Prec) was
computed as the number of correct TEPs divided by the total number of extracted TEPs. The recall (Rec*)
was computed as the number of correct TEPs divided by the number of lemmas in the source language with
more than 3 occurrences. When the (usual) threshold of minimal 3 occurrences is considered, the algorithm
provides a high precision and a good recall.
As one can see from the figures in Table 4, the precision is higher than 98% for Romanian and Slovene
almost 97% for Hungarian and more than 96% for Estonian. The recall (our defined Rec*) ranges from
50.92% (Slovene) to 63.90% (Estonian). We run the extractor for the Ro-En bitext without imposing a step
limit. The program stopped after 25 steps with a number of 2765 extracted pairs, out of which 113 were
wrong. The precision decreased to 95,91%, but the recall significantly improved: 86,89%.
We should mention that Rec*, as we compute it, is slightly different from the usual recall, because Rec*
is reporting the percentage of the number of correct translations found for the lemmas (occurring more than
the threshold) in the source language. Let us assume that in the source language there are N different
lemmas occurring more than the preset threshold and the program found M correct translation equivalents.
Further, let us assume that each lemma is used, on average, with S different senses each one occurring more
than the set threshold. Then, our Rec* will be M/N when it should return M/N*S. As we specified before,
different translations of the same lemma are found, usually, in subsequent steps. Since we set the number of
iteration steps to 4, only for a few words (those very frequent) there would be found multiple valid
translations. That is to say, that Rec » Rec*. However, when the number of iteration steps is increased, Rec*
becomes an overestimation of Rec.
In an initial version of this algorithm we used a chi-square test (as in the baseline algorithm) to check the
selected TEPs. Since the selection condition (EQ1) is very powerful, the vast majority of the selected TEPs
passed the chi-square test while many pairs that used to pass the chi-square threshold did not pass the
condition (EQ1) and therefore we eliminated the supplementary and time consuming statistical tests. This is
certainly one of the reasons for the speed (see next section) of our extraction algorithm.
If one source word has different translations in the target language (either lexicalisations of different
senses of a polysemous source word or different synonyms for the target word), in general they are found, if
frequent enough, in different iteration steps. For instance, when processing the RO-EN bitext of “1984”
parallel corpus, there were extracted 10 correct TEPs for “mare” (big, great, large, vast, sea, long, main,
thick, general, important) but none of them would have been found unless each pair appeared in TECL more
than twice.
Language
Bg-En
Prec/Rec*
Cz-En
Prec/Rec*
Et-En
Prec/Rec*
Hu-En
Prec/Rec*
Ro-En
Prec/Rec*
Sl-En
Prec/Rec*
Step 1 1336
NA/NA
1399
NA/NA
1216
99.50/42.07
1299
98.61/38.88
1394
99.71/42.74
1177
99.91/36.87
Step 2 1741
NA/NA
1886
NA/NA
1617
97.89/55.04
1737
97.63/51.48
1867
99.30/52.23
1489
99.52/46.47
Step 3 1896
NA/NA
2085
NA/NA
1807
96.63/60.84
1863
96.99/54.85
2067
99.03/54.84
1589
99.06/49.63
Step 4 1986
NA/NA
2188
NA/NA
1911
96.18/63.90
1935
96.89/56.92
2182
98.57/56.36
1646
98.66/50.92
Table 4: The results after 4 iteration steps and partial evaluation
From the results shown in the Table 4 one can notice that most part of bilingual lexicons is extracted in
the first step (between 63% and 71%). We tried to extract translation equivalents even for words that
appeared in the source language only twice. Because of validation reasons we did this experiment only for
585
the Ro-En bitext. The experiment considered an extra-iteration step, after the last one, with the occurrence
threshold set to two. The extracted number of pairs (1311) was more than half of the pairs extracted in the
previous steps altogether, but also the number of erroneous pairs (297) was almost 10 times higher than the
previous number of errors (31). However, when lowering the occurrence threshold from the first step, the
number of correct pairs was almost the same (3174 versus 3165) but the number of errors was significantly
higher (417 versus 328) with a global error rate increase from 9,38% to 11,61%.
5. Implementation
The extraction program is written in Perl and runs under practically any platform (Perl implementations
exists not only for UNIX/LINUX but also for Windows and MACOS). The Table 5 shows the running time
for each bitext in the “1984” parallel corpus. The program was run under LINUX on a Pentium III/600Mhz
with 96 MB RAM.
Ro-En Language Bg-En Cz-En Et-En Hu-En
4 steps 25 steps
Si-En
Time (sec) 181 148 139 220 183 415 157
Table 5 :Extraction time (in seconds) for each of the bilingual lexicons
A quite similar approach to ours (also implemented in Perl) is presented in (Ahrenberg et all 1998) and
(Ahrenberg et all 2000). The languages considered by their experiments are English and Swedish. For a
novel of about half the length of Orwell's “1984” their algorithm needed 55 minutes on a Ultrasparc1
Workstation with 320 MB RAM and the best results reported are 96.7% precision and 54.6% recall. For a
computer manual containing about 45% more token than our corpus, their algorithm needed 4.5 hours with
the best results being 85,6% precision and 67,1% recall. Unlike us, they don't rely on tagging and
lemmatisation, although they use a “morphology” module that achieves some kind of stemming and
grouping of inflectional variants. This strategy makes it possible to link low-frequency source expressions
belonging to the same suffix paradigm. The obvious advantage of not using POS categorisation is that their
approach would be able to identify TEPs where the POS of the source token is different from the POS of the
target token.
An explanation of the much better response time in our case, besides not using statistical tests (they use
t-test), is that the search space in our case is probably several orders of magnitude smaller.
6. Conclusions and further work
We presented a simple but very effective algorithm for extracting bilingual lexicons, based on a 1:1
mapping hypothesis. We showed that in case a language specific tokeniser able to recognise and “pack” the
compounds is responsible for pre-processing the input to the extractor the 1:1 mapping approach is not a
limitation anymore. The MULTEXT tokeniser, for instance allows for recognition of generic multiword
expressions (dates, literally expressed numbers) or specific ones based on external resources containing lists
of compounds, proper names, idiomatic expressions. If the compounds cannot be dealt with in the
segmentation pre-processing phase one may consider either extending the bilingual lexicon extractor's
model to an N:M paradigm or consider using a monolingual tool as a pre-processor for recognising the
compounds. We are currently considering both options. For the first one we started the implementation of a
new tokeniser including collocation recognition. As most of the multiword tokens (found by the collocation
extraction module) are not expected to be in the lexicon, a post-tagging module will check (based on very
simple chunking grammars) whether the assigned tag is compatible with the tags recorded in the lexicon for
the constituent (known) words. For the second option, we are carrying out some preliminary experiments
with a slightly modified version of the program presented in this paper. Conceptually, the modified version
of the program can be seen as receiving the same text as source and target input file with all the sentence
alignments being 1:1. Two additional modifications are:
·  the TECL must not include pairs made of identical strings.; this condition is necessary for limiting the
search space to the only potential collocations
·  the POS condition is removed; this restriction is not necessary anymore since most sequences of words
that should be translated as one unit are not characterised by the same POS.
A new and customisable version (implemented in C++) of the algorithm described in this paper,
incorporating BSP, is under construction.
586
Acknowledgements
The research reported here started as an AUPELF/UREF co-operation project, coordinated by Patrick
Paroubeck of LIMSI/CNRS (CADBFR).
Special thanks are due to Heiki Haalep, Csaba Oravecs, and Tomaz Erjavec for the validation of the Et-En,
Hu-En and Si-En extracted dictionaries.
References
Ahrenberg L, Andersson M, Merkel M 1998 A simple Hybrid Aligner for Generating Lexical
Correspondences in Parallel Texts. In Proceedings of COLING'98, Montreal, pp 29-35.
Ahrenberg L, Andersson M, Merkel M 2000 A knowledge-lite approach to word alignment, in Véronis J
(ed), Parallel Text Processing. Text, Speech and Language Technology Series, Kluwer Academic
Publishers, pp 97-116.
Brown P, Della Pietra S, Della Pietra V, Mercer R 1993 The mathematics of statistical machine translation:
parameter estimation. Computational Linguistics 19 (2): 263-311.
'LPLWURYD/(UMDYHF7,GH1.DDOHS+3HWNHYLF97XILú'1998 Multext-East: Parallel and Comparable
Corpora and Lexicons for Six Central and East European Languages in Proceedings of the 36th Annual
Meeting of the ACL and 17th COLING International Conference, Montreal, pp 315-319.
Gale W, Church K 1991 Identifying word correspondences in parallel texts. In Fourth DARPA Workshop on
Speech and Natural Language, pp 152-157.
Gale W, Church K 1993 A program for aligning sentences in bilingual corpus. Computational Linguistics
19(1): 75-102.
Erjavec T, Lawson A, Romary L 1998 East Meet West: A Compendium of Multilingual Resources. TELRIMULTEXT
EAST CD-ROM, 1998, ISBN: 3-922641-46-6.
Hiemstra D 1997 Deriving a bilingual lexicon for cross language information retrieval. In Proceedings of
Gronics 21-26
Kupiec J 1993 An algorithm for finding noun phrase correspondences in bilingual corpora. In Proceedings
of the 31st Annual Meeting of the Association of Computational Linguistics, 17:22
Smadja F, McKeown K, Hatzivassiloglou V 1996 Translating collocations for bilingual lexicons: A
statistical approach. Computational Linguistics, 22 (1): 1-38.
7XILú '  8VLQJ D /DUJH 6HW RI (DJOHV-compliant Morpho-Syntactic Descriptors as a Tagset for
Probabilistic Tagging. In Proceedings of the Second Conference on Language Resources and Evaluation,
Athens, pp.1105-1112.
7XILú '  'LHQHV 3 2UDYHF] & 9iUDGL T 2000 Principled Hidden Tagset Design for Tiered Tagging of
Hungarian. In Proceedings of the Second Conference on Language Resources and Evaluation, Athens,
1421-1426.

587
The linguistic relevance of Corpus Linguistics
Tamás Váradi
Department of CL
Research Institute for Linguistics
Hungarian Academy of Sciences
varadi@nytud.hu
1. Introduction
The present paper is intended as a review of some basic principles of corpus linguistics (CL) from a
linguistic point of view. In particular, it examines the aims and methods of CL in terms of how it stands up
to some basic linguistic tenets regarding the nature and use of data for linguistic analysis. Central to the
discussion will be the problem of representativeness, which will be discussed with reference to an
influential paper by Douglas Biber (1993).
The perspective from which CL is examined in this paper is its utility and relevance for linguistic
description. This is just one of the contexts in which CL is used today and indeed it may not be the bulk of
its applications. It can hardly be doubted that CL has proved its utility in areas of human language
technology too numerous to mention. Within fields more close to linguistics, its role in lexicography is
nothing short of revolutionizing the whole practice of the discipline, introducing a new technology that
produced a new generation of dictionaries. These are undeniable achievements that has established CL as a
thriving and dynamically growing field.
Amidst this burgeoning activity it is perhaps opportune to take this occasion to reflect on some
underlying assumptions and methodological principles characterizing current practice in CL.
2. The concept of language and corpus linguistics
Fundamentally, CL undertakes an empirical study of language. As Leech (2000: 685) rightly points
out, this means that CL is ‘patently’ concerned with documenting and analysing performance, or the
Saussurean notion of parole. The notion performance grammar is somewhat misleading in that CL is
obviously concerned with the actual product of language use as against processes involved in speech
production. The main theoretical claim that CL can make as a contribution to linguistics is that through the
compilation and analysis of masses of data corpus linguistics can provide a solid objective empirical
foundation on which to build a grammar of a language and indeed the technological facility to sift through
large amounts of data leads to new insight into the structure of the language. One key aspect of language
use that CL is particularly well suited to reveal is its quantitative characteristics.
It is to be noted, though, that the very concept of language that CL aims to capture is most forcefully
rejected by Chomsky. Earlier, he consistently defined language as a set of sentences (Chomsky 1957,
1965), which, at least on the face of it, seemed to be congruous with the concerns of CL in the sense that,
typically, CL also approaches language in terms a set of sentences. However, in later works he came to
regard the traditional concept of language, which he now prefers to call Externalized-language, as “an
epiphenomenon at best” (Chomsky 1986: 25) a notion so laden with “complex and obscure sociopolitical,
historical and normative-teleological elements (Chomsky 1991: 31) that he is doubtful if the concept can
be given a coherent interpretation at all. The primary object of linguistic investigation in Chomsky's view
is the Internalized-language, the mental grammar that each native speaker has internalized and which they
draw upon in their linguistic communication.
Even if one does not share Chomsky's dismissive opinion on the relative standing of E-language visa-
vis I-language, the dichotomy is one which CL also has to face. The issue boils down to the need to
account for I-language from the facts of E-language as displayed in a corpus. It seems reasonable to take
the view that language primarily exists in the minds of the speakers. Hence the starting point and focus of
the linguistic enquiry should be the individual speakers with their I-language, their communicative
competence and the actual products of their language use. Focusing on the actual products of actual
language use by individual speakers of a language should, on the face of it, be quite amenable to the
methods of CL.
588
Even at the level of the idiolect, the I-language of individual speakers, the research program raises
some inherent theoretical problems stemming from the need to extrapolate from finite data to an infinite
system. The difficulties are compounded, however, by the attempt to deal with the linguistic output of a
group of speakers, let alone that of a whole language community. The hypothesis of an ideal hearerspeaker
of a homogeneous speech community (Chomsky 1965) does save the theoretical linguist from a
host of complications but CL cannot operate from this perspective.
The ambition of CL was from the very beginning to provide an empirical record of a language at the
level of the whole speech community. As an interesting historical aside, we should note that this ambition
was absent from what McEnery and Wilson (1996) term early phase of CL i.e. prior to Chomsky. Without
the technological support ensuring a semblance of feasibility, one could not even entertain the idea of
capturing and processing enough data for the whole language community. Nor was this the intent. Harris
(1951:12), advocates a method of corpus compilation that proceeded in close cooperation with informants.
This piecemeal, interactive procedure was driven by the expectations of the field workers, based on
experience, about the completeness and consistency of the grammar that they were seeking to build.
3. E-language as a sampling problem
As noted above CL deals with the realm of actual language use. The primary raw data it encounters is
a set of utterances produced by the language community. At first sight, this seems to be an infinite set.
However, the number of words produced either in speech or writing are limited by obvious human
physiological limits. Biber et al. (1999: 27) quote the figure of around 7000 words per hour as the average
speech rate observed in the conversational part of the Longman Spoken and Written English Corpus.
Therefore, once the inevitable fuzziness in the geographical, social and temporal boundaries of the notion
of the language community is somehow resolved, one can put a number to the totality of language produced
by any set of speakers over a given period of time.
Nevertheless, even if, as the above thought experiment suggested, the set of utterances produced by a
language community in a given time interval is finite in size, it is unrealistic to expect that the totality of
language production could ever be captured on electronic media. The bottleneck is not necessarily storage
or technological capacity. While the incredible rate of advancement of computerization and the exponential
spread of the Internet may eventually make the bulk of written language output at least accessible,
capturing spoken output in a corresponding manner is not only unfeasible and even imponderable. This fact
leaves us to conclude that any corpus, however large, is and will necessarily remain a mere sample of the
totality of language output. That is how the issue of representativeness in the design of the corpus assumes
key importance.
From the very beginning the aim of CL was to compile a corpus that was representative of a language.
In terms of the concepts introduced above, this means nothing less than to design a corpus that models the
totality of language use of a speech community. This is certainly a tall order, given the complexity and the
scope of the phenomena that it undertakes to cover. In practice, though, the task was attempted from the
outset with some reasonable limitations in the temporal and geographic dimensions of the data. The
pioneering Brown corpus (Kueera and Francis 1967) set out to capture the written language of the United
States of the year 1963. It was intended to be a general purpose, balanced corpus of American English of
the period. 4.2 discusses how representativeness was achieved and Table 1 displays the corresponding
figures.
Another claim that Corpus Linguistics makes is that it shows up ‘language as is spoken', real language
in its rawness and richness. This intention in obviously inherent in the whole corpus linguistic enterprise of
capturing vast amount of actual data. Apart from marketing purposes, CL only needed emphasizing this in
contraposition to the ruling generative linguistic school, which tended to base its findings on introspective
evidence.
4. Basic design issues
It is clear that the key issue for Corpus Linguistics to make good its promises lies in the scope and
composition of the data that it provides. This will be the focus of our attention for the rest of the paper. It is
widely agreed that a corpus is not simply an archive of texts but rather a principled collection of texts. One
of the first and most important principles referred here concern the selection of texts to go into the corpus.
589
The first question that arises in examining this issue is whether we should care too much about the
composition of the corpus. Accordingly, there developed two kinds of schools of thought supporting two
kinds of corpora: the so-called opportunistic and the balanced corpora.
4.1 Monitor corpus vs. balanced corpus
It is fairly easy to deal with the opportunistic kind as it denies that there is any principled way to
balance a corpus and it makes recourse to the law of large numbers. Perhaps size will automatically sort out
all questions of ‘balance’ in the structure of the data. This approach is vigorously represented by Sinclair
(1991 pp. 23-24) who proposes instead the idea of a monitor corpus – a very large corpus, which after
reaching some sort of a saturation point will undergo a partial self-recycling: the new material flowing in
will be subjected to an automatic monitoring process which will only retain those parts of the incoming
data which show some significantly different features than the stable part of the data.
Once it is decided that some sort of scheme will be set up to compile a corpus in some principled way,
the question that confronts us is whose job is it to do so. Sinclair (op. cit.: 13) holds that it is a task that
should belong to the students of culture rather than corpus linguists. They should only undertake it as a
matter of necessity. The use of language, Sinclair seems to argue, should be studied in the wider cultural
context, which goes beyond the competence of the corpus linguist.
4.2 Units of sampling
Another important sampling issue to decide is the units of the overall population in terms of which the
sample will be compiled. Should the sample be compiled in terms of the speakers or language? If the latter
is chosen, as it was originally done, (without apparently considering any alternative) what are to be the
linguistic units in terms of which the population is sampled: words, sentences, texts, speech situations etc.?
In the first generation of balanced corpora, the Brown and the LOB corpus, this issue was decided by
a panel of experts who designed a scheme where different varieties of language, called genres, are
represented in specific proportions. Table 1 shows how the 1 million word corpus is divided into 15 genres
and how many texts of 2000 word length each are allocated into each category. Note how despite the
professed intention to develop a replica of the pioneering American corpus for British English, the internal
composition of the LOB corpus was slightly changed in categories E,F and G). These subtle changes were
introduced so as to accommodate the structure of the corpus to the peculiarities of British culture. As for
the selection of the particular texts, apparently, a great deal of effort was spent into making sure that the
texts within each category were chosen at random but I am not aware of any public arguments offered in
justification for the particular ratios used between the categories.
4.3 Methods of sampling
Choosing things at random suggests itself as a safe procedure to eliminate any bias or skewing in the
result. However, purely random sampling works against the selection of items that are relatively rare in the
population, out of which the sample is made. An important principle that a sample should meet in order to
be representative of the population is that the sample should show the same ratios between elements within
the sample as they have in the population. Samples are, as it were, severely scaled down versions of the
population. The more frequent an item is in the population, the better chance it stands of being selected at
random. Therefore, it may easily happen that items which occur pretty rarely in the population, will not be
selected by the random process at all. Alternatively, if for some reason or other, we would like to see the
rare items included in the sample, we would have to increase the size of the sample, perhaps out of all
manageable proportions.
590
Genres No of texts
Brown LOB
A Press:report 44 44
B Press:editorial 27 27
C Press:reviews 17 17
D Religion 17 17
E Trades, hobby, leisure 36 38
F General lore 48 44
G Belles lettres, biography, essays 75 77
H Misc. government. documents, public
reports, university catalogues
30 30
J Scientific journals 80 80
K General fiction 29 29
L Crime fiction 24 24
M Science fiction 6 6
N Adventure and Western 29 29
P Romance 29 29
R Humour 9 9
~ ~ 500 500
Table 1 Composition of the BROWN and the LOB corpus
One solution that is devised to overcome the above difficulty is to use stratified random sampling.
Under this procedure the population is first divided into a number of categories (strata) and random
sampling is only applied to fill up the chosen categories with items selected at random. The question of
how many categories to set up into which the population is arranged and how much data should be
collected for each category is decided beforehand. (These are indeed the figures shown in Table 1 for the
BROWN and the LOB corpus.) The taxonomy of the categories is established independently of statistical
considerations. Yet, it has a direct bearing on the quantitative results as well. Once a category is
established, it is bound to be represented in the sample. For example, if we have a general category for
reviews, chance will decide whether the random sampling will select any articles on reviews of early
twentieth century travel books. (Chance will be helped by the number of such articles in the whole
population in that the more there are the higher the chances that a purely random method will select them.)
If on the other hand a special category is adopted to cover travel books, this is taken as a target to be met
and the selection procedure is considered incomplete until data is selected for that category as well. Hence,
the granularity of the classification scheme will effect the structure of the sample as well.
An even more direct intervention in the workings of chance is the setting of target figures for the
amount of data to be collected within each category (i.e. the figures against the categories in Table 1
representing the number of texts, each about 2000 words long). In order for a sample to be representative of
the population for the set of categories in terms of which the sample is compiled, the sample should
conform to the principle of proportionality. This requires the various categories in the sample to be
represented in the same ratio as they are in the total population. For the BROWN corpus to qualify as a
representative sample of the totality of written American English for 1963 for humorous writing, it would
have to be established that humorous writings did make up 1.8 % of all written texts created within that
year in the US. This single requirement serves to illustrate the enormous difficulty if not impossibility of
the task. Surely, it is simply not feasible to put a figure on the amount of text within the various genres in
the totality of texts produced by a speech community. Yet, this is what the statistical concept of a
representative sample calls for.
Note that the difficulty is not necessarily that of dealing with an infinite set. It is, rather, inherently a
logical one. If sampling is done in terms of text type, a representative sample would require knowledge
about the whole population that is simply not available. If it were, we would hardly need a sample, and in
order to find out about proportions obtaining in the population, one would obviously like to examine a
sample of it.
4.4 Demographic vs. context-based sampling
How can we break this vicious circle? One lesson obviously is that one can only provide a
representative sample of the population in terms of features about which one has reliable knowledge from
some independent source. One such source of outside knowledge is indeed available in data about the
591
speakers. One could consult National Census figures to find out about chief characteristics of speakers such
as age, gender, schooling, type of settlement they live in etc. It is then feasible to compile a representative
sample of speakers for such selected features. This type of demographic sample of informants is a wellestablished
procedure in opinion poll surveys, psychological or socio-linguistic research. For corpus
linguistics, the total output of such representative group of speakers would ipso facto amount to a
representative corpus of the speech population.
This procedure was indeed used by the spoken component of the British National Corpus (cf. Burnard
1995: 20-25). 124 adults were selected so that, as far as practical limitations allowed, they would be
represented in equal numbers in terms of sex, age (divided into six age groups) and social class (defined in
four main categories). The recruited informants were asked to record their speech conversations,
unobtrusively whenever possible, for a period of up to a week. Approximately four million words were
collected in this manner, a little under half of the spoken component of the BNC, which in turn, for obvious
practical constraints, made up one tenth of the 100,000 word corpus. The rest of the spoken component,
termed the context-governed part, was selected by “a priori linguistically motivated categories” defined in
terms of a hierarchy of categories with the four context categories educational, business, public/institutional
and leisure at the top and three regional and two interaction type categories providing further subdivisions.
It should be noted that the demographical sample used by the BNC cannot be considered
representative in the sense of the sample being proportional to the population. Curiously, for reasons not
disclosed in the manual, the BNC did not make use of actual demographic figures about the relative
proportions of sex, age and social class obtaining in the UK population. Instead, “the intention was, as far
as possible, to recruit equal numbers of men and women, equal numbers from each of the six age groups,
and equal numbers from each of four social classes” [emphasis added] (Burnard op.cit.: 21). This
methodological laxness in the almost single area where the required information to compile a representative
sample was, in fact, available is hardly compensated by the care to use “established random location
sampling procedures” to select individual members within the groups.
Despite the undeniable practical difficulties of implementing it, the demographic sampling technique
was applied in a limited way on purpose. The Reference Guide notes that ‘many types of spoken text are
produced only rarely in comparison with the total output of all “speech producers”: for example, broadcast
interviews, lectures, legal proceedings and other texts produced in situations where – broadly speaking –
there are few producers and many receivers. A corpus constituted solely on the demographic model would
thus omit important spoken text types'. (Burnard op. cit.: 20)
5. Biber's notion of representativeness
The issues reviewed so far are certainly nothing new to practitioners of the field. With predictable
regularity a discussion flares up on the Corpora List around the notion of the balanced corpus. Newcomers
to the discussion are often referred to Douglas Biber's article “Representativeness in Corpus Design”
(Biber 1993), which is indeed one of the most comprehensive discussions of the topic available in print1.
The rest of the paper will concentrate on this as a canonical text, particularly as it reflects views that are
upheld by Biber in essentially the same form in more recent works (Biber 1998, Biber et al 1999).
Biber distinguishes three possible approaches to corpus design depending on whether they are aimed
at covering text production, text reception and texts as products. The first two are basically different from
the third in that they both define the population in terms of the agents (i.e. speaker/hearer) of language use,
while the third covers it in terms of the output i.e. language. Accordingly, the first two approaches would
call for a demographic sample. However, Biber also rejects demographic samples on the grounds that “they
would not represent the range of text types in a language, since many kinds of language are rarely used,
even though they are important on other grounds.’ […] It would thus be difficult to stratify a demographic
corpus in such a way that it would insure representativeness of the range of text categories. Many of these
categories are very important, however, in defining a culture” [emphasis added] (op. cit.: 245).
This revealing passage spells out some assumptions that may be difficult to reconcile with some basic
assumptions about the role of corpus linguistics. One of the fundamental aims of Corpus linguistics as I
understand it is to show up language as is actually attested in real life use. However, Biber seems to argue
1 There is one unwritten item that comes to mind: there was a live debate held in Oxford between prominent advocates
of the two corpus design philosophies Quirk aided by Leech speaking up for the balanced corpus vs. Sinclair and Meijs
arguing for the open-ended monitor corpus. Oral tradition has it that the debate was decided by the audience in favour
of the Sinclair team.
592
that in designing a corpus one should apply a notion of importance that is derived from a definition of
culture. For lack of any means of operationalizing this criterion of relative importance in culture, this
throws the door wide open to subjective judgment in the compilation of the body of data that is expected to
provide solid empirical evidence for language use.
Biber seems to think very little of the value of a corpus assembled on demographic criteria. “Such a
corpus would permit summary descriptive statistics for the entire language represented by the corpus.
These kinds of generalizations, however, are not typically of interest for linguistic research”, “… it is not
necessary to have a corpus to find out that 90% of the texts in a language are linguistically similar (because
they are all conversations)”; rather, we want to analyse the linguistic characteristics of the other 10% of the
texts since they represent the large majority of the kinds of registers and linguistic distributions in a
language” (op. cit.: 248).
Biber concedes that there is no a priori way to establish the relative proportions of the different genres
obtaining in the population hence a representative sample would have to be demographic by definition.
This impasse leads Biber to conclude that the notion of representativeness as we know it from statistics do
not apply in corpus linguistics. What lies at the root of the problems to implement representativeness is the
principle of proportionality that has been discussed above. Biber not only considers proportional sampling
difficult or unfeasible to implement in any other way than the demographic approach but also goes as far as
to simply reject the notion of proportional sample as an appropriate concept. In justifying his position he
makes the following curious argument: “proportional samples are representative only (sic!) in that they
accurately reflect the relative numerical frequencies of registers in a language – they provide no
representation of relative importance that is not numerical. Registers, such as books, newspapers, and news
broadcasts are much more influential than their relative frequencies indicate.” [emphasis added] (op. cit. :
248)
First, it is disingenuous to find fault with proportional sampling for something it is not intended for
i.e. to reflect this non-numerical relative importance. Second, there is no suggestion how this kind of
importance can be established, let alone quantified in any objective manner. No attempt is made to show
how to measure and accommodate the extent of the influence of the above registers. Earlier, we already
noted the potential methodological danger for arbitrary decisions creeping in the corpus design principles.
One cannot avoid feeling that once recourse is made to non numerical factors such as importance in
compiling the corpus, this makes the whole enterprise of corpus design so vulnerable to subjective value
judgments that any amount of methodological rigour applied in the random selection of the items for
categories looks like the farcical effort of searching for the lost key where there is light.
Rejecting the traditional notion of representative sampling based on the principle of proportionality,
Biber blandly declares that “language corpora require a different notion of representativeness”, “researchers
require language samples that are representative in the sense that they include the full range of linguistic
variation existing in a language.” (op. cit.: 247) First of all, one must voice serious misgivings about any
attempt to divest such a key term of its well-established meaning, which has a clear interpretation to
statisticians and the general public alike. Of course, any self-respecting corpus would like to advertise itself
as a representative corpus. There is such a strong and unanimous expectation from the public and scholars
alike for corpora to be representative that it is an assumption that is virtually taken for granted. However, to
meet this demand by the semantic exercise of redefining the content of the term is a move that hardly does
credit to the field.
6. Conclusions
My aim with this brief overview of the issues in corpus design has been to highlight the linguistic
implications of the choices that are made. By highlighting on the uncertainties, inconsistencies and
methodological fudges currently employed in corpus linguistics, my intention was to show up where further
effort is needed. The picture that emerges helps to dispel the unintended disparity in scientific rigour: in
order to live up to its expectations corpus linguistics must put its methodology on more solid footing and
users of corpus linguistics would do well to be aware of the linguistic issues at stake and the extent to
which they can expect ready solutions.
593
References
Biber D 1993 Representativeness in corpus design. Literary and Linguistic Computing 8(4):243-257.
Biber D, Conrad S, Reppen R, 1998 Corpus Linguistics. Investigating Language Structure and Use.
Cambrdige, Cambridge University Press.
Biber D, Johansson S, Leech G, Conrad S, Finegan E 1999 Longman Grammar of Spoken and Written
English. London, Longman.
Burnard L (ed) 1995 British National Corpus. Users Reference Guide for the British National Corpus.
Oxford, University Computing Service.
Chomsky N 1965 Syntactic Structures. Cambridge Mass., MIT Press.
Chomsky N 1986 Knowledge of Language Its Nature, Origin and Use. New York, Westport, London,
Praeger.
Chomsky N 1991 Linguistics and cognitive science: Problems and mysteries. In Kasher A (ed) The
Chomskyan Turn. Oxford, Blackwell, pp. 26-53.
Kueera H, Francis W 1967 Computational Analysis of Present-Day American English. Providence RI,
Brown University Press.
Leech G N 2000 Grammars of Spoken English: New Outcomes of Corpus-Oriented Research Language
Learning 50(4):675-724.
McEnery T, Wilson A 1996 Corpus Linguistics. Edinburgh, Edinburgh University Press.
Harris Z S 1951 Methods in Structural Linguistics. Chicago, University of Chicago Press.
Johansson S, Leech G N, Goodluck H 1978 Manual of information to accompany the Lancaster-Oslo-
Bergen Corpus of British English, for use with digital computers. Department of English, University of
Oslo.
Sinclair J 1991 Corpus, Concordance, Collocation. Oxford, Oxford University Press.

594
Corpus-based versus intuition-based lexicography:
defining a word list for a French learners’ dictionary.
Serge Verlinde and Thierry Selva
Modern Language Institute, K.U. Leuven (Belgium)
1. Introduction
Although French lexicographers were among the first to integrate corpus-analysis into the
dictionary-making process, with the Trésor de la langue française project in the early seventies and
its corpus of 170 million words, corpus-based lexicography is certainly not a common practice in
contemporary lexicography in France. There are a few exceptions, however, e.g. the Oxford-Hachette
English-French/French-English translation dictionary (Corréard 1997) and the Dictionnaire
d'apprentissage du français des affaires (DAFA - Binon, Verlinde, Van Dyck, Bertels 2000), but
mainstream lexicography is undoubtedly intuition-based.
As far as we know, no comparative studies have been made of the results of the two
lexicographic approaches. The aim of this paper is to present such a comparative study on a selection
of words to be described in a learners’ dictionary of French. On the one hand, we have a recent
learners’ dictionary, which has been published by one of the leading French dictionary publishers
(Dictionnaires Le Robert), where the selection of the entries is intuition-based (Dictionnaire du
français - DF, Rey-Debove 1999). On the other hand, we have the DAFLES (Dictionnaire
d'apprentissage du français langue étrangere ou seconde, an (electronic) French learners’ dictionary
we are presently working on.
We try to use an objective frequency criterion to select the words and multiword units described
in our dictionary. Therefore we use an automated statistical analysis of a 50 million word corpus of
newspaper texts, taken from the 1998 issues of Le Monde (France) and Le Soir (French speaking part
of Belgium).
In the two first sections of this paper, we will present this corpus and the analyses made on it. In
the third section, we will compare the two lists of words in order to reveal the most important
differences between these two lists. In the last section, we will make another comparison, namely with
the only large frequency list existing for French, which has been published along the Trésor de la
langue française (TLF), the Dictionnaire des fréquences (Imbs 1971).
2. The corpus
Two important questions arise when building a corpus: its representativeness and its size. For the
French language there is currently no project like the British National Corpus (BNC 2000) or the
Bank of English (BOE 2000); therefore we must rely on the texts that are freely accessible (see
Verlinde, Selva (forthcoming) for an overview of available corpora for French). The choice is limited
to literature in the public domain and newspaper texts published on archive CD-ROMs. In order to
cover actual language, we have chosen the 1998 issues of two newspapers: Le Monde (France) and Le
Soir (French speaking part of Belgium). Both CD-ROMs permit the texts of all articles to be exported.
In the case of Le Soir, exporting can be done by date or by newspaper section. In the case of Le
Monde, there is no clear-cut classification of the articles. Therefore, only exporting by date makes it
possible to export all articles. The corpus has a total size of 54 260 926 words, both subcorpora having
approximately the same size.
We cannot say that our corpus is perfectly balanced, but it is made up of the kind of texts that the
potential users of our dictionary will have to deal with.
At the first stage, we cut all documentary information about the articles from the corpus.
Indications about source, date and page have been coded separately in the format of the text analysis
software we use (see below). The whole corpus was then tagged and lemmatised with the Cordialsoftware,
splitting up all multi-word units like chemin de fer and pomme de terre and removing
proper nouns. The result of this analysis was processed in order to restore the aspect of the original
texts. We submitted the entire lemmatised corpus (51 845 143 words) to Wordcruncher, a well-known
text analysis tool. As Wordcruncher was not able to merge both subcorpora, we have merged the two
separate frequency lists of both subcorpora to create a frequency list for the whole corpus. This
frequency list has been corrected on some minor points. For example, frequent words written with a
595
hyphen that were split up during the lemmatisation process have been extracted from the original
corpus and added to the list. Some errors of lemmatisation have also been corrected.
Our corpus is smaller than the two big English corpora but it seems to be large enough, taking
into account our objective of writing a learners’ dictionary with a selection of the most common
vocabulary, collocations and grammatical structures of the current language.
3. Frequency list and dictionary word list
For the lexicographer, it is particularly difficult to define the importance of a dictionary word
list. We see that in many cases, the number of entries (macrostructure of the dictionary) is much more
important than the content of each entry (microstructure of the dictionary). Thus, the accent lies on
single words more than on word combinations (collocations), for instance. This is paradoxical
because, for productive purposes, learners need this information on collocations much more (Bogaards
1996, 1998) than an impressive number of isolated words, many of which will never be looked up. We
decided provisionally to limit the word list to 12 156 words, selecting all lemmas that appear at least
100 times in our corpus. These 12 156 lemmas represent approximately 93.14% of all the words of the
corpus, proper nouns not included. Extending this word list to 22 000 words, as in the DF, would only
increase the coverage of the texts by 1%.
It is surprising to see that this limited list contains a large number of words that are very
common in spoken language: maman, papa, job, sympa, bosser for example. There are also a lot of
words that should perhaps not appear in a learners’ dictionary because they are immediately linked to
current affairs (bosniaque, kosovar) or to the local pages of the newspapers (brabançon, brainois,
borain).
Another aspect of vocabulary use that can be easily studied with the frequency list is the
generalization of English words in French. It is well known that French authorities follow a policy of
“defending” the French language by suggesting, quite systematically, French equivalents for English
words. We might suppose that newspapers, mainly the French ones, wouldtry to reinforce this policy,
but this seems not to be the case in light of the frequency of some English words (table 1).
frequency
Le Monde
frequency
Le Soir
business 446 471
coach 83 1424
cool 108 169
design 305 312
fast-food 42 88
goal 69 108
holding 573 505
joint(-)venture 84 95
leasing 29 69
lobbying 137 114
marketing 847 767
team 76 590
trader 48 35
Web/web 1057 514
Table 1: frequency of some English words in the Le Monde and Le Soir corpora
Even for those words that have an accepted and well-know French equivalent (affaires for
business, but for goal, équipe for team), the English word seems to be used quite regularly. In some
cases, the French equivalent is not used at all (mercatique for marketing).
The fact that we are working with corpora from two different language communities makes it
also possible to compare the vocabulary used in both communities and to extract words that are
specific to one of these communities by comparing the relative frequency of their occurrences in both
corpora. Table 2 shows an extract of the list of typical French and Belgian words and abbreviations.
596
typical French words typical Belgian words
ballottage Échevin
préfectoral mai eur/mayeur
départemental Communal
baccalauréat Deputation
minitel Tram
préfet Subside
lycéen play-off
intéressement Coach
cantonal Voirie
interministériel Urbanistique
typical French abbreviations typical Belgian abbreviations
mdc Asbl
insee Prl
cfdt Cpas
smic Psc
cgt Rtbf
Table 2: list of typical French and Belgian words and abbreviations
Such information on geographical variants is rarely mentioned in the essentially France-oriented
French dictionaries.
3. Corpus frequency list and word list of the DF
The DF is in fact the first learners’ dictionary of French for twenty years. The objective,
presenting the words of both everyday conversation and the press (Rey-Debove 1999: VII), is very
close to our objective, and to the objective of every learners’ dictionary in general. As the authors do
not say that they integrated a corpus analysis, it is possible to make a comparison between a corpusbased
approach and an intuition-based approach, at least for the word list of the dictionary. Similar
comparisons could be made for the collocations and the syntactic structures that are described in the
dictionary. Table 3 shows to what extent the word list of the DF matches the words of the corpus
frequency list.
corpus frequency
ranges
number of words not
mentioned in the DF
percent cumulative
frequency
cumulative
percent
0-500 0 0 0 0
501-1000 2 0,4 2 0.2
1001-1500 3 0,6 5 0.3
1501-2000 1 0,2 6 0.3
2001-2500 10 2 16 0.6
2501-3000 16 3,2 32 1.1
3001-3500 18 3,6 50 1.4
3501-4000 28 5,6 78 2
4001-4500 40 8 118 2.6
4501-5000 45 9 163 3.3
5001-5500 48 9,6 211 3.8
5501-6000 61 12,2 272 4.5
6001-6500 58 11,6 330 5.1
6501-7000 67 13,4 397 5.7
7001-7500 39 7,8 436 5.8
7501-8000 87 17,4 523 6.5
8001-8500 80 16 603 7.1
8501-9000 102 20,4 705 7.8
9001-9500 120 24 825 8.7
9501-10000 115 23 940 9.4
597
10001-10500 110 22 1050 10
10501-11000 129 25,8 1179 10.7
11001-11500 154 30,8 1333 11.6
11501-12000 124 24,8 1457 12.1
Table 3: corpus frequency ranges and DF word list
The conclusion that can be drawn from this table is that 12.1% of the 12 000 most frequent
words of our corpus do not appear in the DF. The differences in coverage are limited up to frequency
4000 with a difference less than 10%. From frequency 4 000 on, and mainly from frequency 8 500 on,
the differences in coverage increase seriously (up to 20% and more).
In the list of ‘forgotten’ words and abbreviations, we find investisseur, budgétaire, entité,
concertation, restructuration, infrastructure, forum, info, privatisation, amendement for example.
These words need to be mentioned in a general purpose dictionary.
When we have a look at the words mentioned in the DF that do not appear in our frequency list,
we notice that these words can not really be considered as current words (table 4, excerpt from the
beginning of the letter A).
a fortiori abetissant abreuvoir accessoiriste
a gogo abjurer abricotier accotement
a jeun ablution abrutir accouder (s')
a.z.t. aboiement abrutissant accoudoir
abasourdi abois (aux) abscisse accoutrement
abat-jour abominablement absenter (s') accoutrer
abats abortif abyssin accroupir (s')
abattant aboutissants acadien accumulateur
abattis abracadabrant acariâtre accus
abetir abrasif accablement achalandé
Table 4: DF words not appearing in the corpus frequency list
The authors of the DF identify the frequent and important words by marking them with a blue
triangle. In our learners’ dictionary we classify the words into six frequency ranges (table 5).
frequency range Range occurrences text coverage
1 <= 427 >= 11 183 66 %
2 <= 990 >= 5 273 75 %
3 <= 1 926 >= 2 482 82 %
4 <= 3 920 >= 854 88 %
5 <= 12 156 >= 100 93 %
6 < 100 100 %
Table 5: DAFLES frequency ranges
Both frequency indications can be linked and compared. Once again, the intuitive approach
seems to be less rigorous than a corpus-based approach: words like acajou, adipeux, ablation, affairé
and affublé are “frequent and important” according to the authors of the DF but not a, année,
américain, allemand, afin de/que.
Looking into detail at the whole list of “frequent and important” words for the letter A of the DF
reveals however also some weaknesses of our corpus-based approach. Some everyday life words as
s'absenter do not appear into our frequency list. They need certainly to be added to a learners’
dictionary word list.
4. Comparing two corpus-based frequency lists: literature and newspapers
As it was mentioned above, the Dictionnaire des fréquences (Imbs 1971) is a frequency list
published along the TLF. It is the only frequency list based on a large corpus (170 million words) for
598
French with literary texts from the beginning of the nineteenth century to the sixties. We selected the
first 12 174 lemmas of this list in order to compare them to our frequency list.
It is not surprising that a lot of current words as régional, match, euro, championnat, football,
culturel, télévision and festival, for example, do not appear in the frequency list of the Dictionnaire
des fréquences. Words that do not appear in our list mainly characterise personal feelings (sottise,
fâché, gémir, tressaillir) and things that do not exist anymore (pardessus, sou, écu).
In addition to everyday life words mentioned above, words expressing feelings in general form a
second important group of words to be added to the word list of our dictionary.
5. Conclusion
From the comparison of both lexicographic approaches (corpus-based and intuition-based), we
can conclude that corpus-based lexicography gives a strong and necessary empirical evidence to the
lexicographer's personal intuition, even if this personal intuition remains helpful in filling the gaps in
our corpus.
These gaps are undoubtedly due to the fact that the corpus is unbalanced. Taking into account
this observation, there is a strong need to design and construct for French, and for other languages as
well, a carefully selected corpus with a large variety of texts, in order to improve the quality of
(learners') dictionaries, and vocabulary learning and teaching in general.
References
Binon J, Verlinde S, Van Dyck, J, Bertels A 2000 Dictionnaire d'apprentissage du fraçais des
affaires. Paris, Didier. (an electronic version of this dictionary can be found at
http://www.projetdafa.net).
Bogaards P 1996 Dictionaries for learners of English. International Journal of Lexicography 9(4):
277-320.
Bogaards P 1998 Des dictionnaires au service de l'apprentissage du français langue étrangere.
Cahiers de lexicologie 72(1): 127-167.
Corréard M-H (ed.) 19972 The Oxford-Hachette French Dictionary. Paris-Oxford, Hachette-OUP.
Imbs P 1971 Dictionnaire des fréquences. Vocabulaire littéraire des XIXe et XXe siecles, I - Table
alphabétique, II - Table des fréquences décroissantes. Nancy-Paris, CNRS-Didier.
Rey-Debove J (ed.) 1999 Dictionnaire du français. Référence. Apprentissage. Paris, CLE
International-Dictionnaires le Robert.
TLF. Imbs P 1971-1994 Trésor de la langue française. Paris, CNRS-Gallimard.
Verlinde S, Selva Th forthcoming Nomenclature de dictionnaire et analyse de corpus. Cahiers de
lexicologie.
Websites
BNC 2000: http://info.ox.ac.uk/bnc/
BOE 2000: http://titania.cobuild.collins.co.uk/boe_info.html

599
Sense tagging: does it make sense?
Jean Véronis
Université de Provence
29, Avenue Robert Schuman, 13100 Aix-en-Provence (France)
Jean.Veronis@up.univ-mrs.fr
Sense tagging is probably one of the challenges that corpus linguists have to face in the near future. So
far, computerisation of this task has yielded very modest results despite numerous efforts, and sense
tagging is turning out to be a touchy task. Difficulties stem from various sources, extracting
disambiguating information from the context. However, one of the main problems that lies upstream of
the disambiguating process is the sense inventory itself. Most tagging efforts rely on traditional
dictionaries to supply the reference senses, or on computer-oriented resources such as WordNet, which
do not differ significantly from traditional dictionaries in terms of sense division. The present paper
shows that human taggers perform very poorly when given a traditional dictionary as the reference, and
that machines should therefore not be expected to perform any better if the same kind of resource is
used. A detailed analysis reveals the lack of distributional criteria in dictionary entries: traditional
dictionaries are chiefly concerned with meaning definition, and not with the surface clues (syntactic,
collocational, etc.) that are required to match a given sense with a given corpus occurrence. It is argued
that no fundamental progress can be made until large-scale lexical resources have been built that
incorporate extensive distributional information, and that, until that time, any massive sense tagging
efforts based on traditional dictionaries or computer-oriented resources such as WordNet would not
only be premature but also questionable in terms of resource management.
Keywords: sense tagging, polysemy judgements, interannotator agreement, dictionaries, distributional
information, word sense disambiguation.

600
A corpus – based analysis of how accurately printed Romanian obeys to some
universal laws
Adriana Vlad, Adrian Mitrea, and Mihai Mitrea
“POLITEHNICA” University of Bucharest
Faculty of Electronics and Telecommunications
1-3 Iuliu Maniu Bvd., Bucharest, Romania, vadriana@vala.elia.pub.ro
A main objective of the paper is how accurately printed Romanian complies with the stationarity
hypothesis. A statistical approach to NL stationarity, based on the mgram structure is presented. The
statistical inferences are: estimation theory with multiple confidence intervals, test of the hypothesis
that probability belongs to an interval and test of the equality between two probabilities. The b size of
the type II statistical error plays a special role in the designing of a corpus for mathematical purposes.
The stationarity investigation was also used to investigate how accurately printed Romanian complies
with two frequency–rank laws.
Key words: natural language stationarity, frequency–rank laws, multiple confidence intervals for
probability.

601
Exploiting the WWW as a corpus to resolve PP attachment ambiguities
Martin Volk
University of Zurich
Department of Computer Science, Computational Linguistics Group
Winterthurerstr. 190, CH-8057 Zurich
volk@ifi.unizh.ch
1. Introduction
Finding the correct attachment site for prepositional phrases (PPs) is one of the hardest problems when
parsing natural languages. An English sentence consisting of a subject, a verb, and a nominal object
followed by a prepositional phrase is a priori ambiguous. The PP in sentence 1 is a noun attribute and
needs to be attached to the noun, but the PP in 2 is an adverbial and thus part of the verb phrase.
(1) Peter reads a book about computers.
(2) Peter reads a book in the subway.
If the subcategorisation requirements of the verb or the competing noun are known the ambiguity
can sometimes be resolved. But many times there are no clear requirements. Therefore, there has been
a growing interest in using statistical methods that reflect attachment tendencies.
This new line of research was kicked off by Hindle and Rooth (1993). They tackled the PPattachment
ambiguity problem (for English) by computing lexical association scores over a partially
parsed corpus. If a sentence contains the sequence V+NP+PP the triple V+N+P is observed with N
being the head noun of the NP and P being the head of the PP. The probabilities are estimated from cooccurrence
counts of V+N and of N+P. They evaluated their method on manually disambiguated verbnoun-
preposition triples. It resulted in 80% correct attachments.
In the meantime the method has been improved and extended. The best reported results are from
Stetina and Nagao (1997: up to 88% correct attachment). They use a supervised learning approach
(they train the disambiguator over the Penn-Treebank) and a semantic dictionary to cluster the words.
We applied unsupervised statistical methods to German. Since there is no large German treebank
available we first worked with a partially parsed corpus. The gathering of co-occurrence data is more
complicated for German because of its variable constituent ordering. In Langer et al. (1997) we show
that we can achieve around 76% attachment accuracy for the decidable cases. But many cases cannot
be decided because of sparse data.
Therefore we have experimented with using the WWW, a corpus that is orders of magnitude larger
than our locally accessible corpora. With the help of a WWW search engine we obtain frequency
values (“number of pages found”). In querying a search engine we lose some precision compared to
corpus analysis. Our hypothesis is that the size of the WWW will compensate our rough queries.
Our method for determining co-occurrence values is based on a simple formula. We use the
frequency of a word co-occurring with a given preposition against the overall frequency of this word.
For example, if some noun N occurs 100 times in a corpus and this noun co-occurs with the preposition
P 60 times then the co-occurrence value of N+P will be 60/100 = 0.6. The general formula is (where X
can be either a noun N or a verb V):
freq(X,P) / freq(X) = cooc(X,P)
In Volk (2000) we have explored this formula in detail. We have shown that the WWW frequencies
can be used for the resolution of PP attachment ambiguities if the difference between the competing cooccurrence
values is above a certain threshold. In this way the co-occurrence values served to decide
58% of our test cases with an attachment accuracy of 75%.
In the more successful experiments for PP attachment in English (Stetina and Nagao 1997, Collins
and Brooks 1995) the co-occurrence statistics included the noun within the PP. The motivation behind
this becomes immediately clear if we compare the PPs in the example sentences 3 and 4. Since both
PPs start with the same preposition only the noun within the PP helps to find the correct attachment.
(3) Peter saw the thief with his own eyes.
(4) Peter saw the thief with the red coat.
602
In a new round of experiments we have included the head noun of the PP into the queries. This
means we are now working with the extended formula:
freq(X, P, N2) / freq(X) = cooc(X, P, N2)
Let us look at an example sentence from our corpus:
(5) Unisource hat die Voraussetzungen für die Gründung eines Betriebsrates geschaffen.
Unisource has set up the prerequisites for the foundation of a work council.
freq(X,P,N2) freq(X) cooc(X,P,N2)
X=N1 freq(Voraussetz.,für,Gründung)
274
freq(Voraussetzungen)
255'010
cooc(Voraussetz.,für,Gründung)
0.001074
X=V freq(geschaffen,für,Gründung)
139
freq(geschaffen)
172'499
cooc(geschaffen,für,Gründung)
0.000805
The co-occurrence value cooc(N1,P,N2) is higher than cooc(V,P,N2), and thus the model correctly
predicts noun attachment for the PP.
2. Preparation of the test corpus
We manually compiled a treebank as a test suite for the evaluation of our method. We semiautomatically
disambiguated and annotated 3000 sentences. In order to be compatible with the German
NEGRA treebank we used the same annotation scheme as Skut et al. (1997).
We selected our evaluation sentences from the 1996 volume of the ComputerZeitung, a weekly
computer magazine that is available on CD-ROM (Konradin-Verlag 1998). We tagged the text and
selected 3000 sentences that contained
1. at least one full verb and
2. at least one sequence of a noun followed by a preposition.
With these conditions we restricted the sentence set to those sentences that contain a prepositional
phrase in an ambiguous position.
Manually assigning a complete syntax tree to a sentence is a labour-intensive task. This task can be
facilitated if the most obvious phrases are automatically parsed. We used our chunk parser for NPs and
PPs to speed up the manual annotation. We also used the NEGRA Annotate-Tool (Brants et al. 1997)
to semi-automatically assign syntax trees to all (preparsed) sentences. This tool comes with a built-in
parser that can suggest categories over selected nodes. The sentence structures were judged by two
linguists to minimize errors. Finally, completeness and consistency checks were applied to ensure that
every constituent was included into the sentence structure.
We then used a Prolog program to build the nested structure and to recursively work through the
annotations in order to obtain sextuples with the relevant information for the PP classification task:
1. the full verb (a separated verbal prefix is reattached),
2. the real head noun N1 (the noun which the PP is attached to),
3. the possible head noun N1 (the noun that immediately precedes the PP; this noun leads to the
attachment ambiguity),
4. the preposition of the PP,
5. the core noun of the PP (called N2), and
6. the attachment decision (as given by the human annotators).
Let us illustrate this with some example sentences.
(6) Das Dorfmuseum gewährt nicht nur einen Einblick in den häuslichen Alltag vom Herd bis zum
gemachten Bett.
The village museum allows not only insights into the everyday life from the oven to the bed.
(7) ... nachdem dieses wichtige Feld seit 1985 brachlag.
... since this important field lay idle since 1985.
(8) Das trifft auf alle Waren mit dem berüchtigten "Grünen Punkt'' zu.
This holds true for all goods with the ill-famed "Green Dot''.
603
These corpus sentences will lead to the following sextuples:
verb real N1 possible N1 prep. N2 (in PP) function of the PP
gewährt Einblick Einblick in Alltag postnominal modifier
gewährt Alltag Alltag vom Herd postnominal modifier
gewährt Alltag Herd bis Bett postnominal modifier
brachlag / Feld seit 1985 verb modifier
zutrifft Waren Waren mit Punkt postnominal modifier
Each sextuple represents a PP with the preposition occurring in a position where it can be attached
either to the noun or to the verb. Note that the PP auf alle Waren in 8 is not in such an ambiguous
position and thus does not appear in the test cases.
In sentence 6 we observe the difference between the real head noun and the possible head noun.
The PP bis zum gemachten Bett is not attached to the possible head noun Herd but to the preceding
noun Alltag. Obviously, there is no real head noun if the PP attaches to the verb (as in 7). In the
following tests we use the real reference noun N1 if it is present else the possible reference noun N1.
Our test corpus consists of 4383 test cases, out of which 63% are noun attachments and 37% verb
attachments.
3. Disambiguating with WWW frequencies
We queried AltaVista in order to obtain the frequency data for our co-occurrence values. For all queries
we use AltaVista advanced search restricted to German documents. For co-occurrence frequencies we
use the NEAR operator.
· For nouns and verbs we query for the word form by itself.
· For co-occurrence frequencies we query for Verb NEAR preposition NEAR N2 and N1
NEAR preposition NEAR N2 again using the verb forms and noun forms as they appear
in the corpus. The NEAR operator in AltaVista restricts the search to documents in which its
argument words co-occur within 10 words.
We then compute the co-occurrence values for all cases in which both the word form frequency and
the co-occurrence frequency are above zero. We evaluate these co-occurrence values against our test
corpus using the following disambiguation algorithm.
if (cooc(N1,P,N2) && cooc(V,P,N2)) then
if (cooc(N1,P,N2) > cooc(V,P,N2)) then
noun attachment
else
verb attachment
else
noun attachment
If both co-occurrence values exist, the attachment decision is based on the higher value. If one or
both co-occurrence values are missing we decide in favour of noun attachment since 63% of our test
cases are noun attachment cases. The disambiguation result is summarized in table 1.
correct incorrect accuracy
noun attachment 2553 1129 69.34%
verb attachment 495 206 70.61%
total 1800 1335 69.54%
Table 1: Attachment accuracy for the complete test corpus
604
The attachment accuracy is improved by 6.5% compared to pure guessing. But it is way below the
accuracy that we computed for the decidable cases in earlier experiments. Even in the WWW many of
our test triples do not occur. Only 2422 (55%) of the 4383 test cases can be decided by using both cooccurrence
values. The attachment accuracy for these test cases is 74.32% and thus about 5% higher
than when forcing a decision on all cases (cf. table 2)
correct incorrect accuracy
noun attachment 1305 416 75.83%
verb attachment 495 206 70.61%
total 1800 622 74.32%
Table 2: Attachment accuracy when requiring both cooc(N1,P,N2) and cooc(V,P,N2)
3.1. Using the co-occurrence values against a threshold
A way of tackling the sparse data problem lies in using partial information. Instead of insisting on
both cooc(N1,P,N2) and cooc(V,P,N2) values, we can back off to either value for those cases with only
one value available. Comparing this value against a given threshold we decide on the attachment. If, for
instance, cooc(N1,P,N2) is available (but no cooc(V,P,N2) value), and if this value is above the
threshold then we decide on noun attachment. If cooc(N1,P,N2) is below the threshold we take no
decision. Thus we extend the disambiguation algorithm as follows:
if (cooc(N1,P,N2) && cooc(V,P,N2)) then
if (cooc(N1,P,N2) > cooc(V,P,N2)) then
noun attachment
else
verb attachment
elseif (cooc(N1,P,N2) > threshold) then
noun attachment
elseif (cooc(V,P,N2) > threshold) then
verb attachment
Now the problem arises on how to set the thresholds. It is obvious that the attachment decision gets
more reliable the higher we set the thresholds. At the same time the number of cases that are decidable
decreases. We suggest to set the threshold in such a way that using this partial information is not worse
than using both the cooc(N1,P,N2) and cooc(V,P,N2) values. That means that we set the threshold so
that we keep the overall attachment accuracy at around 75%.
correct incorrect accuracy
noun attachment 1448 446 76.45%
verb attachment 629 245 71.97%
total 2077 691 75.04%
Table 3: Attachment accuracy when requiring either cooc(N1,P,N2) or cooc(V,P,N2)
We thus set the threshold to 0.001 and obtain the result in table 3. The attachment rate (the number
of decidable cases) has risen from 55% to 63%; 2768 out of 4383 cases can be decided based on either
both co-occurrence values or on the comparison of one co-occurrence value against the threshold.
Noun attachment is still better than verb attachment.
3.2. Using the co-occurrence values of word forms and base forms
The above frequencies were based on word form counts. But German is a highly inflecting language
for verbs, nouns and adjectives. If a rare verb form (e.g. a conjunctive verb form) or a rare noun form
605
(e.g. a new compound form) appears in the test corpus it often results in a zero frequency for the triple.
We may safely assume that the co-occurrence tendency is constant over the different verb forms. We
may therefore substitute the rare verb form with a more frequent form of this verb. We decided to
query with the given verb form and with the corresponding verb lemma (the infinitive form).
For nouns we also query for the lemma. As a special case we reduce compound nouns to the last
compound element and we compute the lemma for the last element (e.g. Informationssystemen ®
System). We do the same for hyphenated compounds (e.g. GI-Kongresses ® Kongress). We also
reduce company names ending in GmbH or Systemhaus to these keywords and use them in place for
the lemma (e.g. CSD Software GmbH ® GmbH).
The co-occurrence value is thus computed as (X is the verb V or the reference noun N1):
freq(Xform,P,N2) + freq(Xlemma,P,N2)
freq(Xform) + freq(Xlemma)
=cooc(X,P,N2)
The disambiguation algorithm is the same as above and we use the same threshold of 0.001. As
table 4 shows, the attachment accuracy stays at around 75% but the attachment rate increases from 63%
to 71% (3109 out of 4379 test cases can be decided).
correct incorrect accuracy
noun attachment 1615 459 77.87%
verb attachment 735 300 71.01%
total 2350 759 75.59%
Table 4: Attachment accuracy including threshold and lemmas
In order to complete the picture we evaluate without using the threshold. We get an attachment
accuracy of 74.72% at an attachment rate of 65%. This a 10% increase to the result we computed for
word forms (cf. table 2). If, in addition, we use any single co-occurrence value (i.e. we set the threshold
to 0), the attachment accuracy slightly decreases to 74.23% at an attachment rate of 85%. This means
that for 85% of our test cases we have at least one co-occurrence value from the WWW frequencies. If
we default the remaining cases to noun attachment we end up with an accuracy of 73.08% which is
significantly higher than our initial result of 69.54% (reported in table 1).
3.3. Conclusion
The most important lesson from these experiments is that triples (X,P,N2) are much more reliable than
tuples (X,P) for deciding the PP attachment site. Using a large corpus such as the WWW helps to
obtain frequency values for many triples and thus provides co-occurrence values for most cases.
Furthermore, we have shown that querying for word form and lemma substantially increases the set
of decidable cases and thus the attachment rate without any loss in the attachment accuracy. The
accuracy is 74% for all decidable test cases and 73% for all test cases. We can further enhance the cooccurrence
frequencies by querying for all word forms, as long as the WWW search engines index
every word form separately.
If we are interested only in highly reliable disambiguation cases (80% accuracy) we may lower the
number of decidable cases by increasing the threshold (or by requiring a minimal distance between
cooc(V,P,N2) and cooc(N1,P,N2) as we have shown for tuples in Volk, 2000).
When using frequencies from the WWW the number of decidable cases should be higher for
English since the number of English documents in the WWW by far exceeds the number of German
documents. Still the problem remains that querying for co-occurrence frequencies with WWW search
engines using the NEAR operator allows only for very rough queries. For instance, the query P NEAR
N2 does not guarantee that the preposition and the noun co-occur within the same PP. It matches even
if the noun precedes the preposition. There are various possibilities for improved queries.
1. X NEAR "P DET N2" with an appropriate determiner DET will query for the sequence "P DET
N2'' and thus for P and N2 co-occurring in a standard PP.
606
2. X NEAR (P NEXT 3 N2) will query for N2 as one of the three tokens following P. The NEXT
operator is often available in information retrieval systems but not in the WWW search engines
that we are aware of. This query is more flexible than querying for a standard PP.
3. "N1 P" NEXT 3 N2 will query for noun N1 and preposition P immediately following each other
as is most often the case if the PP is attached to N1.
4. V SAME_SENTENCE (P NEXT 3 N2) will query for the verb V co-occurring within the same
sentence as the PP. From a linguistic point of view this is the minimum requirement for the PP
being attached to the verb. In fact, to be linguistically precise we must require the verb to co-occur
within the same clause as the PP. But none of these operators is available in current search engines.
Obviously, any of these constraints will reduce the frequency counts and may thus lead to sparse
data. We will therefore have to counterbalance this with querying for words that behave similarly with
respect to PP attachment, for instance, words from the same semantic class.
Acknowledgement
We thank Charlotte Merz for comments and corrections on earlier versions of this paper.
References
Brants T, Skut W, Krenn B 1997 Tagging grammatical functions. In Proc. of EMNLP-2, Providence,
RI.
Collins M, Brooks J 1995 Prepositional phrase attachment through a backed-off model. In Proc. of the
Third Workshop on very large corpora.
Hindle D, Rooth M 1993 Structural ambiguity and lexical relations. Computational Linguistics, 19(1):
103-120.
Konradin-Verlag 1998 Computer Zeitung auf CD-Rom. Volltextrecherche aller Artikel der Jahrgänge
1993 bis 1998. Leinfelden-Echterdingen, Konradin-Verlag.
Langer H, Mehl S, Volk M 1997 Hybride NLP-Systeme und das Problem der PP-Anbindung. In
Wermter S, Busemann S, Harbusch K (eds), Berichtsband des Workshops "Hybride
konnektionistische, statistische und symbolische Ansätze zur Verarbeitung natürlicher Sprache" auf
der 21. Deutschen Jahrestagung für Künstliche Intelligenz, KI-97 (auch erschienen als DFKIDocument
D-98-03), Freiburg.
Skut W, Krenn B, Brants T, Uszkoreit, H 1997 An annotation scheme for free word order languages. In
Proceedings of the 5th Conference on Applied Natural Language Processing, Washington, DC, pp
88-95.
Stetina J, Nagao M 1997 Corpus based pp attachment ambiguity resolution with a semantic dictionary.
In Zhou J, Church K (eds), Proc. of the 5th Workshop on very large corpora, Beijing and Hongkong,
pp 66-80.
Volk M 2000 Scaling up. Using the WWW to resolve PP attachment ambiguities. In Proc. of Konvens-
2000. Sprachkommunikation, Ilmenau, VDE Verlag, pp 151-156.

607
A corpus-based methodology for comparing and evaluating different accents
Martin Weisser
Department of Linguistics and Modern English Language
Lancaster University
1. Learner corpora and their uses
The past few years have seen a growing interest in the development of learner corpora (cf. Granger,
1998). However, so far the main emphasis for this kind of corpus has been on collecting and marking
up written texts produced by native and non-native speakers. This was mainly done in order to be able
to determine where the most common problems and differences in the usage of written language, both
as far as grammar and lexis are concerned, lie for the learners. Learner corpora of spoken language are
still relatively rare and even if they do exist, often consist of written orthographic transcriptions, possibly
minimally enriched with symbols to indicate lengthening, etc. only. Attempts at producing learner
data relating to pronunciation have so far, to my knowledge, been restricted to setting up a flatfile
database documenting the realisations of school children, as in the Austrian Learner's Database
(Wieden/Nemser, 1991b), and to parts of the ISLE Project1.
1.1. Corpora vs. LE speech databases
As pointed out above, learner corpora are so far mainly being used to investigate learner behaviour
with regard to grammar and lexis, but very little attention has been paid to issues of pronunciation.
Phonetic transcriptions of spoken language on a corpus scale are generally the domain of Language
Engineering (LE), where the purpose has mainly been to collect speech data that may be used for applications
such as speech recognition, speech synthesis, telebanking applications or spoken dialogue
systems. However, the material recorded for these purposes is in many cases severely limited with respect
to the particular requirements of the application it is being collected for, e.g. series of numbers or
certain keywords needed for communicating with the application.
The first notable exception to this kind of corpus is the aforementioned Austrian Learner Database,
which documents the pronunciation of Austrian schoolchildren of varying age-groups and from different
regions. The purpose of this database, amongst others, was to establish which particular problems
in pronunciation for theses learners may be introduced by their L1. However, this database had
some serious drawbacks and flaws due of its design and implementation, some of which will be discussed
with the relevant topics further below. The second exception is the material collected for the
EU-funded ISLE project, which aims to provide an architecture for incorporating speech recognition
technology into language-learning software products.
1.2. Transcription, encoding and fonts
Due to the relative high use of UNIX-based workstations in LE, transcriptions in LE speech databases
are often based on ASCII representation schemes of the IPA character sets, such as SAMPA
(Wells et al., 1992) or use other mapping algorithms, which work well for the computer, but often make
it difficult for the human reader to understand the transcriptions, especially if the latter contain many
diacritics. The transcription and encoding system used in the Austrian database is even more complex
since it not only uses numerical codes to represent the transcriptions themselves, but also includes
codes that represent information about the relative closeness of the pronunciations to RP and any
deviations from the implicit norms underlying the analysis. For example, one of the representations of
the word rubber given in Wieden/Nemser, 1991b (p. 354) is “00004540000200001000000050020000”.
Coding schemes such as this may make it easy to conduct relatively exact statistical comparisons
containing minute details of pronunciation, but are extremely difficult to handle and may lead to results
biased more towards the quantitative than qualitative side. To facilitate research of a more qualitative
nature, the use of standard IPA (True-type) fonts is thus preferable.
1 http://nats-www.informatik.uni-hamburg.de/~isle/speech_text.html
608
2. Spoken language models applied to learner English
2.1. RP vs. native speaker corpus-based reference models
When learners of English are assessed or tested, this is usually done against implicit reference
models such as RP (Received Pronunciation) for British English or GenAm (General American) for
American English because these represent the standardised varieties the learners have supposedly been
taught. Furthermore, learners are often expected to enunciate far more clearly and possess a far higher
rhetorical skill than native speakers of different varieties of English. Not only is this unrealistic and an
unfair practise towards the learners, but it is also, at least as far as British English is concerned, an
unrepresentative model of the language as RP is only spoken by about 3 % of the overall population
(Hughes/Trudgill, 1997: 3), and even then in different forms (Wells, 1982: 279-301). However, so far
no attempts at creating more realistic models of native speakers English have been made that could
serve as an adequate basis for comparison between native and non-native speakers.
3. Creating speaker/learner population models
The methodology I propose here represents a synthesis of ideas from corpus linguistics – especially
learner corpora – and LE. Creating a kind of reference model from the native speaker realisations in the
first instance makes it possible to compare native and non-native speaker data by establishing an
adequate basis for comparison, rather than relying on the more abstract established teaching models,
such as RP. Such a model would highlight tendencies and patterns in native speaker variation against
which one can then evaluate the performance of non-native speakers realistically, and in turn identify
problem areas that suggest possible changes in teaching methodology and practice. Apart from
comparing native and non-native speakers in this way, there are also other implications and usages of
such a methodology, which are discussed towards the end of this paper.
3.1. What needs to be stored?
In order to create suitable models for analysing the spoken language of different speaker populations
a number of different types of information need to be stored. Just like in corpora of written language, it
is important to store orthographic representations of the spoken material and as far as possible enrich
these by at least incorporating morphosyntactic information in the form of grammatical tags2. In my
implementation, the orthographic material, i.e. a dialogue created and read by 7 native and 10 nonnative
speakers, is stored word-by-word in a table and information about the word-classes, i.e.
grammatical tags, in another.
More than for purposes of handling written language, this type of information needs to be
complemented by detailed information about the speakers, such as age, sex, place of birth, etc. because
these factors may have a strong influence on the speech behaviour. For non-native speakers it is also
important to record details about their exposure to the target language, in order to be able to establish
‘proficiency’ levels.
The part that makes a spoken corpus essentially most different from a written one is that not only
does a spoken corpus/database need to incorporate the transcriptions themselves, but also needs to give
the user access to the original recordings in form of sound files, so that the original transcriptions can
easily be verified and potentially corrected. This is why simple annotated text files generally do not
represent a suitable storage format for spoken data. More often than not, genuine spoken corpora
therefore tend to take on the form of applications that allow linking in and controlling analysis and
playback tools (cf. Deutsch et al., 1998).
The methodology I describe in this paper is based around an MS Access 2000 database application
that already allows for some of these features, while some others are yet to be implemented. It is, for
example, already possible to start a speech analysis program with a specified soundfile and at a specific
offset, in order to do transcriptions and to control this program to some extent from a form via Visual
Basic for Applications (VBA).
2 For more information on different types of annotation, see Leech et al. (1998).
609
3.2. Transcription issues
As already mentioned above, the use of fonts may present some difficulties, especially if maximum
compatibility between different operating system platforms is required. However, with the increasing
importance of Unicode3 and at least some Linux implementations now supporting the use of True-type
fonts, the use of IPA fonts has become less of an issue. Working on a Windows-based system, the
obvious choice for my implementation is to use an IPA font whose character representations are saved
in the transcription table(s). Since MS Access unfortunately does not allow for multiple fonts within
the same table and the font I use only contains a limited amount of numbers, which would therefore
make it unsuitable for representing large IDs for linking pronunciations to words, transcriptions and
comparisons in the database are made accessible via MS Access forms that can display different fields
of a table using different fonts. Figure 1 and Figure 2 below demonstrate the difference between the
two forms of representation.
Figure 1 – A snapshot of the Realisations table.
Figure 2 – The Realisations form, open for speaker E03.
As Figure 2 shows, the Realisations form also provides buttons that make it easier to input some of
the characters and diacritics that would normally have to be input via character codes as they are not
mapped onto the keyboard. Something else that can be seen from the illustration above is that in order
to be able to conduct precise phonetic and phonemic analyses, a great amount of detail should be
included in the initial transcription. This is necessary because in empirical analyses like mine, and
especially when comparing speakers from different countries, it is extremely difficult to predict which
3 Although existing Unicode fonts still often lack in typographic quality as far as phonetic characters are concerned.
610
features may turn out to be relevant. For example, after having listened to some of my data, I had
initially assumed that one of the major differences between the native and non-native speakers may be
the amount of creak in the realisations of the latter, but it later turned out that both speaker populations
use creaky voice, while this feature only becomes a distinguishing marker in certain contexts.
A further issue in the transcription spoken data is consistency. In a large scale implementation of my
methodology, it would be extremely important to have a sufficient amount of transcriptions verified by
different transcribers since this is the only way in which any internal consistency could be guaranteed –
and thus also any statistical validity.
3.3. Phonetic categories
Unlike in a written corpus, where words are generally separated by whitespace or punctuation marks,
in transcriptions of spoken data it is important to look at the context in which each word appears.
Transitions between words thus represent an important and relatively easily categorisable feature of
pronunciation and often serve a distinctive markers between different speaker populations. It is
therefore important to keep a record of the phenomena occurring between words alongside the general
transcription. Figure 2 above shows a list-box on the right-hand side from which the different
transitions can be selected. These transitions contain both simple types, such as assimilation, elision,
etc., but also information about pauses and other complex types, such as assimilation + elision,
assimilation before a short pause, assimilation before a long pause, etc.
Information about stress patterns can easily be incorporated by including primary and secondary
stress marks in the transcriptions, complemented by information about possible multi-word-units, such
as compounds, etc. Furthermore, other suprasegmental information, such as the length of the realisation
of different objects in the text, i.e. the length of the dialogue (text) itself, of individual sentences,
phrases, and down to the level of individual words if necessary. At present, my database(s) only
contains information down to the level of individual sentences.
3.4. ‘Phonetics’ vs. ‘phonology’
As previously pointed out, for comparing different speaker populations often a great level of phonetic
detail may be required in order to detect relevant distinguishing features. However, this wealth of detail
may sometimes make it difficult to arrive at high-level observations. For this reason, my application
contains a VBA routine that creates a copy of the original data that can be ‘filtered’ by stripping out
any diacritics deemed irrelevant for any particular part of the analysis, in order to make the remaining
data more ‘legible'. For example, if it is unimportant for analysis purposes whether initial plosives are
aspirated or not, all the aspiration can simply be filtered out in the comparison. In this way, we can
move step by step from a phonetic to a more phonemic representation of the data. Figure 3 below
shows an ‘unfiltered’ comparison of the realisations and transitions associated with the word town and
Figure 4 shows a ‘filtered’ representation of the same word with ‘normal’ aspiration removed4. The
‘filtering’ routine is started by clicking on the “Filter Occurrences” button at the top of the form
displayed in Figure 3, which then prompts the user to input a series of characters to be stripped out,
separated by spaces.
4 Note that the stronger, more unusual aspiration represented by an [s] is still shown.
611
Figure 3 – Realisations + Transitions (‘unfiltered')
Figure 4 – Realisation Comparisons (‘filtered')
4. Data Storage
Apart from what is being recorded in the kind of corpus under discussion, it is also important to
understand how the different types of information are stored and how they can be related to one another
in a meaningful way. As far as data storage is concerned, we end up with two fundamentally different
types of data, the digitised soundfiles, which are stored as 16 bit, 16kHz .wav files, which can be
played back by many player applications both on Windows and Linux systems, and the transcriptions
and annotations, which are stored in the relational MS Access database(s).
4.1. Relational model
The choice of using a relational, rather than a flatfile database model was an easy one because a
relational database allows to store different types of information together and relate them to each other
when needed. It therefore provides a highly flexible and easily expandable architecture to which other
types of information can be added if and when necessary.
612
4.2. Levels of information and their coordination
The most basic levels of information in the database are represented by the transcriptions of
individual word tokens, their transitions and the word tokens they are related to in the dialogue.
Through this, it is not only possible to combine and display all the different realisations for a particular
word in a specific context, but also generate a kind of extended pronouncing dictionary with
information about the different possibilities of pronunciation for a specific word in different contexts.
At the next level, we can incorporate grammatical tags in order to see whether specific realisation
phenomena, such as for example the deletion of complete words, are related or even restricted to
specific word classes for individual speaker populations. We can also investigate whether specific
groups of word classes, such as deictica (e.g. pronouns or locatives) are often realised with final
release, indicating a potentially unusual degree of emphasis.
Next, we can incorporate punctuation to determine speaker behaviour at phrase, sentence or turn
boundaries and see how it relates to the realisation of pauses or final lengthening indicated in the word
transition categories. This could then, as a further step, be complemented by prosodic information to
investigate the proportion of long or short pauses in relation to the use of prosodic indicators of textual
cohesion.
In those cases where we may find particularly unusual realisations that do not fit any speaker
population patterns, we can make recourse to the information stored about individual speakers in order
to verify whether anything in their backgrounds may have triggered such particular speech behaviour,
such as e.g. long periods of time spent abroad, etc. In a similar way, we may also investigate variation
within particular populations based upon the age of the speakers.
5. Comparison strategies
Comparisons between different speaker populations can take on various forms. As an initial step, all
the realisations (potentially also including the transitions) for each particular population have to be
combined into one table. This can be achieved by running SQL union queries against the database.
Once the query results exist, additional queries can be run against them and aggregate functions, such
as counts, averages or standard deviations can be used to establish frequency information for particular
realisations, transitions, etc. Just like with concordance programs for analysing written corpora,
wildcard searches in these queries help to focus in on particular parts of the data.
If all that is needed for an analysis is to establish the deviances of one population from another, one
can even write programs that analyse the results of the union queries and write these out to a table.
However, according to my experience this may not necessarily help the researcher to understand all the
relevant differences between the populations. Therefore, contextualised side-by side comparisons, such
as illustrated in Figures 3 and 4 above are a much more preferable way of analysing the data, since this
way, tendencies for both populations can relatively easily be spotted and compared. Once a particular
model for a native speakers population has been established, it is of course also possible to evaluate the
performance of individual non-native speakers against this model and to identify in which areas there is
scope for improvement.
6. Applications and implications
The methodology I have developed for my PhD thesis provides a way of using relational databases in
order to store transcriptions of native and non-native speaker data and for capturing the differences
between them statistically. This is a rather different approach from the one taken by most other studies
investigating the performance of non-native speakers, who simply assume that a convenient basis for
comparison exists in form of a standard.
This kind of methodology has implications for both language teaching and also testing, as it may
provide a more realistic account of the way that native speakers speak and therefore also what should
potentially be taught to foreign learners in order to enable them to communicate efficiently with the
former. As far as language testing is concerned, the implications are that
a) it is possible to assess the speech of learners in a relatively objective way, rather than having
to relay on the largely impressionistic marking schemes that are still currently in use (cf.
Heaton, 1975: 100 and Weir, 1993: 43/44), and
b) that this provides an eminently better and fairer way of assessing the speech of foreign
learners in comparison to native speaker performance.
613
Apart from its use for the evaluation of non-native speaker accents, the methodology can also easily
be applied to the study of different native speaker accents, not only for purely linguistic research
purposes, but also potentially in order to establish criteria that may be used to improve speech
recognition and other language engineering technology, such as dialogue systems (cf. Leech and
Weisser, 2001).
References
Deutsch W, Vollman R, Noll A, Moosmüller S 1998 An Open Systems Approach for an Acoustic-
Phonetic Continuous Speech Database: The S_Tools Database-Management System (STDBMS).
In: Nerbonne J (ed.) 1998. Linguistic Databases. Center for the Study of Language and
Information: Stanford, California. pp 77-92.
Chollet G, Cochard J-L, Constantinescu A, Jaboulet C, Langlais P 1998 Swiss French PolyPhone and
PolyVar: Telephone Speech Databases to Model Inter- and Intra-speaker Variability. In: Nerbonne J
(ed.) 1998. Linguistic Databases. Center for the Study of Language and Information: Stanford,
California. pp 117-135.
Granger S (ed.) 1998 Learner English on Computer. London: Longman.
Heaton JB 1975 Writing English Language Tests. London: Longman.
Hughes A., Trudgill P 1987 English Accents and Dialects. London Edward Arnold.
Leech G, Weisser M forthcoming 2001 Pragmatics and Dialogue in: Mitkov R (ed.). The Oxford Handbook
of Computational Linguistics. Oxford: OUP.
Leech G, Weisser M, Wilson A, Grice M. 1998 Survey and Guidelines for the Representation and Annotation
of Dialogue. In: Gibbon D, Mertins I, Moore R (eds.) 2000 Handbook of Multimodal and
Spoken Dialogue Systems. Dordrecht: Kluwer Academic Publishers.
Nerbonne J (ed) 1998 Linguistic Databases. Center for the Study of Language and Information: Stanford,
California.
Wieden W, Nemser W 1991a The Pronunciation of English in Austria. Tübingen: Gunter Narr.
Wieden W, Nemser W 1991b Compiling a database on regional features in Austrian-German English.
In: Wieden W, Nemser W 1991 The Pronunciation of English in Austria. Tübingen: Gunter Narr.
pp. 350-363.
Wells, J. 1982. Accents of English. Cambridge: CUP. (Vol. 2)
Wells J, Barry W, Grice M, Fourcin A, Gibbon D 1992 Standard Computer-compatible transcription.
Esprit project 2589 (SAM), Doc. no. SAM-UCL-037. London: Phonetics and Linguistics Dept.,
UCL.
Weir C 1993 Understanding and Developing Language tests. Hemel Hempstead: Prentice Hall
International (UK).

614
Wh-questions and attitude: the effect of context
Anne Wichmann and Richard Cauldwell
University of Central Lancashire and University of Birmingham
One of the most commonly cited functions of intonation is its ‘attitudinal’ function, and yet this
remains its most elusive aspect. The profusion of meanings frequently ascribed to one and the same
contour serves to show that the contour itself means none of them. However, we know intuitively that
such meanings are generated, and recently there have been some attempts to investigate what these
meanings are and how they arise.
A persistent problem in the investigation of prosodically generated ‘attitudes’ is the tendency by many
to conflate many different kinds of affective meanings, in particular emotion and attitude. There have
been some attempts to identify and categorise different affective states (see Wichmann 2000, Scherer
forthcoming). These studies have been motivated mainly by an upsurge in interest in the direct effects
of emotion on the voice, and, for this reason, the indirect, context dependent affective meanings which
can be observed in interaction have so far been neglected.
Cauldwell (2000) observed that the attitudinal meaning conveyed by an utterance (a WH-question) in
isolation was absent when the utterance was heard in its original conversational context. In this paper I
also focus on WH-questions, extracted from the ICE GB corpus, and observe the various affective (or
neutral) meanings both in isolation and in context.
These perceived meanings include those which reflect the emotional state of speaker, the affectively
coloured beliefs or predispositions of the speaker to a person or proposition, or an interpersonal stance
(an affective stance taken toward another person in a specific interaction) (after Scherer forthcoming).
I attempt to explain the contribution of intonation to the presence or absence of perceived affective
colouring on the basis of auditory prosodic analysis.
References
Wichmann A 2000 The attitudinal effects of prosody, and how they relate to emotion. In Proceedings
of ISCA workshop on Speech and Emotion, Belfast.
Cauldwell RT 2000 Where did the anger go? The role of context in interpreting emotion in speech. In
Proceedings of ISCA workshop on Speech and Emotion, Belfast.
Scherer KR (forthcoming) Psychological models of emotion. To appear in J.Borod (ed) The
neuropsychology of emotion. New York, Oxford University Press

615
His breath a thin winter-whistle in his throat: English metaphors and their
translation into Scandinavian languages
Kay Wikberg
Department of British and American Studies
University of Oslo
k.b.wikberg@iba.uio.no
Fax: 47 22 85 68 04
In a previous study (Wikberg, forthcoming) based on 11 fiction extracts from the ENPC corpus I show
that the translation of innovative metaphors from English into Swedish and Finnish are mostly based on
equivalent images and that most of the changes that occur in the rendering of the metaphors are simply
due to the constraints of the target languages. Another finding was that the overwhelming majority of
the metaphors have other than plot-advancing functions.
In this study I will examine the translation of metaphors and idioms with a metaphorical meaning in a
wider selection of texts in the ENPC corpus. My aim is further to examine the functions these nonliteral
expressions play in discourse and to investigate the relation between metaphorical structure and
function (cp Cacciari 1998). Finally, I will address the question of how metaphors can be handled in a
discourse model (cp Werth 1999).
References
Cacciari, C. 1998. Why do we speak metaphorically? Reflections on the functions of metaphor in
discourse and reasoning. In Albert N. Katz, Cristiana Cacciari, Raymond W. Gibbs & Mark
Turner, Figurative Language and Thought, New York & Oxford: Oxford University Press, 119-
157.
Werth, P. 1999. Text Worlds: Representing Conceptual Space in Discourse. Longman: London.
Wikberg, K. (forthcoming) Studying the translation of metaphors in a multilingual corpus (English into
Swedish and Finnish). In K.M. Jaszczolt & K. Turner (eds.), Proceedings from the Second
International Conference in Contrastive Semantics and Pragmatics, Cambridge, 11-13 September
2000. Elsevier Science.

616
Public discourse as the mirror of ideological change:
a keyword study of editorials in People's Daily
Karen Wu Rongquan
English and Communication Department, City University of Hong Kong
enkarenw@cityu.edu.hk
1. Overview
China, in its 50 years of communist rule since liberation, has experienced cultural and ideological
change on a massive scale. The editorials from the People's Daily, the typical public discourse
representing the Communist ideology of China, encoded these ideological changes in its formulations.
This study will present the meaning changes of some political-social keywords in editorials over the
years and try to interpret the changing content of ideology by virtue of these semantic findings.
While many communication studies have investigated the change reflected in news and media,
noting its functions in Communist party-state (Liu, 1971, 1981; Bishop, 1989; He and Chen, 1998;
Chang, Wang, and Chen, 1994; Lee, 2000; Zhao, 1998; He, 2000), or have concentrated on the politics
of information flow from the party-state center to the mass media (Wu, 1994), few have devoted
themselves to the semantic study of typical public discourse regarding China’ s cultural and ideological
transformation over its history.
This study will examine editorials from the People’ s Daily (“PD” hereafter) since 1949 – the year
the Chinese nation was officially “stood up” by the Communist Party, to 1995, the year in which the
Party celebrated its 46 years of leadership over the state.
Taking the notion that discourse is language in use (Fairclough and Wodak, 1997), the current
study will apply and develop the social theories concerning discourse and social change. Analysis will
be conducted at two levels. At the lexical level, it will draw on contextual meaning theory to examine
the meanings of keywords by observing the collocations occurring in 9-word span. At the textual level,
a text analysis will be introduced to describe the meaning of keywords under particular practical
discursive environment. At both levels, meaning analysis will be processed regarding the words
immediate context (particular editorial text in which particular keyword occurs) and the far-reaching
context (the changing social process viewed from an ideological perspective).
This keywords study will put emphasis on the semantic field of ideology study. The ideological
implications of the keywords concerned have undergone great transformation in Chinese official media
discourse along the years. Focusing on the ideological significance that discourse and the lexicon
encoded, this analysis aims to be a bridge linking our understanding of particular words and the
ideological changes that China has experienced along its social development.
This study will argue that, for the aim of critical discourse analysis, instead of merely taking
social ideological change for granted, detailed analysis on keywords is helpful in understanding the
content of the changing ideology.
2. Literature Review
2.1. The concept of ideology
Among numerous scholars (de Tracy, [1796]; Marx and Engels, 1974[1845]; Lenin, 1970[1905];
Lukacs, 1971; Mannheim, 1960[1933]) contributing to ideology studies, Marx and Engels stand out
significantly for they tried to construct a complete conceptual system of ideology. Their concept of
ideology is pejorative in essence and different from previous studies (e.g., Napoleon’ s negative
concept of ideology) in that they input Hegel's dialectics into their philosophy as well their theory of
ideology. Marx and Engels constructed ideological theory from the perspective of economic
stratification together with class domination and defined the term “ideology” as systemized ideas
originating from interests of the dominant class.
Ideology, as the old-fashioned, controversial term, is caught within even more debates in
contemporary studies (Geertz, 1964; Althusser, 1965; Shils, 1968; Foucault, 1972; Pecheux, 1982;
Harbermas, 1988; Bourdieu, 1977, 1991; Gee, 1990). Among these debates, the Marxian twodimensional
stance in understanding ideology has been a prominent influential one upon recent
language-related ideological studies (van Dijk, 1998; Hodge and Kress, 1993). Hodge and Kress’ s
(1993) work is an example of some linguistic studies trying to avoid one-dimensional surface
understanding of ideology and attempting to investigate ideology in a dialectic way.
With Marx and Engels’ ideology theory as the basic framework, some of the contemporary studies
offer this old-fashioned, controversial term some reasonable definitions (Hodge and Kress, 1993;
617
Fairclough, 1995b; van Dijk, 1998). But due to the more or less different perspectives and goals, it is
still difficult to say one is more correct than others. Unlike those studies which thoroughly adopted
Marx’ s ideological theory as their basis, this study will add its own “ strategy” in defining ideology:
to define the concept by revealing its relations with discourse.
2.2. Discourse and ideology
The necessity to define the concept of ideology in relation to discourse resides in the knowledge
that the relation between signifiers and their meaning cannot be easily interpreted and controlled
without considering the relation between discourse and ideology. As stated by Stalin, language is not in
itself ideological, discourse is (Stalin, 1951).
On this issue, the scope and sophistication of the thought of contemporary French thinkers, like
Althusser (1970), Bourdieu (1980) and Foucault (1979, 1980a, 1980b), are imposing.
Drawing on Malrieu’ s (1999) definition of discourse, and adopting Marx and Engels’ two
dimensional stance in defining ideology, this study defines ideology as a set of values and ideas
advocated by the social dominant groups that guide actions and regulate the relationship of power and
are expressed in a conventional discourse. In other words, ideology is a system that consists of three
essential components: (1) a set of values and ideas advocated by the dominators according to their
interests; (2) the guiding function of those values and ideas on actions and the structure of power they
endorse; and (3) a discursive convention that expresses, perpetuates and determines the contents of the
ideology.
2.3. Keyword studies
Some scholars have tried to interpret culture through keywords. Researchers whose names should
not be unmentioned in the literature of keywords study include Firth (1935), Williams (1961, 1983),
Said (1978), Fairclough (1990, 1992), Wierzbicka (1992, 1997) and Stubbs (1996).
3. Hypothesis and research questions
This study treats the past of PRC as a history composed of three social periods, i.e. 1949-1965;
1966-1978 and 1979-1995. This is called the “ three-fold-division method”. The “ three-fold-division
method” is based on the following assumptions: 1966 is the year in which the Cultural Revolution was
initiated. The abnormal social situation in China caused by the Cultural Revolution didn’ t finish until
early 1979, when the reform strategies indicated in the Eleventh Party Congress of the CCP (held in
1978,18/12-22/12) began to be implemented in reality. The period between 1949-1965 represents the
time during which Communist China built it’ s national integration (Liu, 1971), and 1979-1995 is the
period in which China resurrected its economy by practising a market-oriented open-door policy. The
three periods are comparable in terms of the time span and how each represents a radical change.
In the study, the following research questions will be addressed:
a) Are there any semantic changes in particular keywords in editorials?
b) What explanations can be offered for the meaning change or stability of particular keywords?
c) How can we understand the changing content of ideology through comparing the meanings of
keywords in each period?
4. Methodology
4.1. Data collection and analysis
The data corpus was built up with editorials from the People’ s Daily since January 1,1949 to
December 31, 1995. The total number of PD editorials in this period was 6812. After eliminating those
outside the fields of politics and economics, articles under research will number 6064.
Analysis with editorials as target data offers a reasonable approach in understanding the function
of apparatus discourse in the changing context of ideology. As a typical kind of “discourse of
apparatus” , the editorial from People's Daily represents to a large degree what He (2000) labels as
“discursive conventions” . Besides, as argued by van Dijk (1988) and Hodge and Kress (1993), the
editorial, as the most preferred formulation place for a newspaper to express its political and
ideological attitudes, most clearly reflects the real standpoint and perspective of the opinion of the
newspaper holders. It is therefore reasonable for the current study to take a semantic approach to
interpret the ideological sense encoded in particular keywords.
I will argue that, in terms of the transmission medium and the audience, basically, there are two
kinds of public ideological discourse in China, i.e. theoretico-ideological discourse and practicoideological
discourse (or pure ideological discourse and applied ideological discourse). Ideology is
expressed explicitly in the former kind, in the form of official party documents of Communist China;
and is strategically manipulated to convince people in the form of the latter kind, which are those
618
appearing in media forms. Ideological keywords should be located within theoretico-ideological
discourse and be understood in their practico-ideological discursive context.
In this study, the meanings of some 30 ideological keywords, which are drawn from the political
reports of Communist China Party Congresses (hereafter CCPC), will be investigated under their
practical discursive environment, i.e. the editorials from the “PD”. By observing the collocations
occurring in 9-word span, meaning analysis will be processed regarding the words immediate context
(particular editorial text in which particular keyword occurs) and the far-reaching context (the changing
social process viewed from an ideological perspective). Discourse analysis here aims to be the bridge
linking our understanding of particular word usage and the social progress.
Analysis will consist of four stages:
Stage 1: Locating Chinese basic ideological keywords on the basis of theoretico-ideological
discourse: the political reports of Chinese Communist Party Congress (CCPC).
Scholars have set out certain academic definitions for keywords (Williams, 1961; Wierzbicka, 1997),
which will be taken as the basic criteria for locating keyword. With somehow different perspectives,
there are various categories of keywords which are all revealing in ideology studies, for example: the
keywords appeared in all social periods, the ideologically expressive words once prevalent for certain
period but later disappeared, and words coined as new ideological expression for new social periods.
This study will focus on the basic ideological keywords, i.e., the ideological keywords being used in all
political reports of CCPC (from the 8th CCPC in 1956 to the 15th CCPC in 1997) during the social
periods concerned (Murata, 1998, see Table.1 for examples of basic ideological keywords).
Stage 2: Locating keywords in the practico-ideological discourse, i.e., the editorials.
Based on the results of stage 1, some 30 ideological keywords will be chosen for further analysis.
These keywords will be located and investigated considering their usage in the editorials from the “PD”.
Stage 3: Observing the collocation lists of the keywords, determining the meaning components of
the keywords according to their collocation lists.1
To identify the meaning of particular keywords, rather than merely depending on the intuition of
native speakers, the keywords will be investigated in its context. By context, we mean 9-word
collocation span (Firth, 1935, Stubbs, 1996), 80-charater context (Scott, 1991) as well as the text as a
whole. One need to decide the scale of the context considering the purpose of the particular study. This
study will observe the keywords in a context adopted in Firth’ s (1935) and Stubbs’ (1996) key words
studies, that is the 9-word collocation span (four words to left and right).
In this study, data of the three social periods, which will be called sub-corpora for the convenience
of describing, will be processed respectively (for the basis of division, see section 4.). In each subcorpus,
collocations of certain keywords will be listed and categorized according to the fields (what is
going on, what is that collocation for?) of the collocations. The fields list thus developed is the meaning
components list of the keywords. For example, in the collocations list of keyword “ revolution” , while
some collocations such as “worker” “Chairman Mao” “proletarian” are used accompanying revolution
to illustrate the participants of revolutionary activity, some other collocations like “improve”
“maintain” are used to describe the expectation that the editorials producers hold towards revolution. In
that case, “participant” and “expectation” are the fields that the collocations stand for, and therefore are
two of the meaning components of the word “revolution” (see Table S-1 in Sample Analysis below).
With computer-aided work, collocations under each meaning components will be sorted according
to the frequency of occurrences. Different collocations with the same or co-relative meaning will be
grouped together and considered as a unit while sorting the frequencies of collocations of particular
meaning. Collocations (or collocation units) ranked higher in the list will be regarded as the closer
contextual meanings of the words in that particular social period.
Stage 4: Analysing the collocations (with focus on frequency, reference) and interpreting the use
of key words through the observation on collocations of each components. Comparing the meaning
components of keywords across different social periods on the basis of the results from collocation
analysis. Interpreting the keywords meaning variation regarding the social-context change (See section
of findings and discussions in Sample Analysis attached).
Through analysis of the collocation, the main uses of the keywords, under each meaning
components, can be listed in a meaning table (see Table S-2 in the Sample Analysis attached).
4.2. Quantitative analysis
In order to study keywords collocations across large corpus, this research requires a method to
summarize collocation data and to calculate the frequency and the likelihood of association between
1 During the three stages of analysis, I referred to the Modern Chinese-English Dictionary (Foreign Language
Teaching and Research Press, 1991) when translating the keywords and collocations from Chinese to English.
619
collocations. Work with quantitative method includes:
a) Identifying in the corpus all occurrences of keywords and their frequency;
b) Keeping a record of collocations of the keywords with occur in a window of defined size (e.g. four
words to left and right);
c) Counting the frequency of each collocation in the whole corpus and in each sub-corpus;
d) Sorting the collocations (and the collocation units, in which the collocations share the same or corelative
meaning) under each meaning components according to the frequencies in each sub-corpus.
4.3. Qualitative analysis
Qualitative method will be employed to interpret the results drawn from quantitative research on
collocations regarding the social and ideological context of China.
Sample analysis:
The change in the meaning of “ ” (ge ming, revolution) in China from 1976 to 1980
1. Overview
This analysis examines the meaning change of the keyword “ ” (ge ming, revolution)
appeared in editorials of “PD” during the period 1978-1980. Findings for this purpose are presented in
form of collocations lists according to the meaning components of “ ” (ge ming, revolution).
Another aim of this study is to interpret the meaning change regarding to the ideological changes that
China has experienced during this 4 years.
The main theory referred is the contextual meaning theory, which believes that words meaning is
embodied in the usage, and the semantic study on words should take context into careful consideration.
The main concept is that words occur in characteristic collocations, which show the associations and
connotations they have, and as well the assumptions that they embody. It is also assumed that
collocations of particular word encode the “meaning components” of that word (Wierzbicka, 1984). By
identifying the collocations of the word “ ” in different periods of history, one can trace the
meaning change occurring in this word over years.
2. Data collection and procession
Considering the historical significance of the year 1978, in which China released the reform
strategies (in the Eleventh Party Congress of the CCP, held in 1978, 18/12- 22/12), this analysis will
take data two years to the forth and after 1978, i.e. from Jan 1, 1976 to Jan 1, 1980. Data will be
processed concerning the two “ turning points” of China in the 4 years: the downfall of The Gang of
Four (in Oct, 1976) and the official appeal of “ focusing on economy” (at the end of 1978).
Corresponding editorials for the two points are respectively on Jul 10, 1976, and Jan 01, 1979. The 44
editorials therefore will be categorized to three groups according to the two points, and the comparison
of the collocations will be held within this three-division frame.
3. Findings
Table S-1 attached shows part of the collocation table of “ ” ge ming, revolution in
editorials of the category “national politics” (Jan. 1, 1976-Jan. 1, 1980.). Categories (except the “ fixed
phrases”) in the left-most column are defined on the basis of the functions or referents of collocations
in contributing to the usage of “ ” , and therefore can be regarded as meaning components of this
word (definitions are given in the brackets following each category). Figures in square brackets show
the collocation frequency of each component in different social periods. While figures in brackets of
first row show the number of editorials observed for each social periods, those in brackets following
each collocation show the total times of that occurrence in that particular data scope.
We will analyze the collocations regarding each meaning component.
Fields:
a) During the Cultural Revolution, main fields of revolutionary activity are social institutions, such as
education, public health, art and science and technology. Revolution in these institutions means to
politically change the minds of the people engaged in these careers and to destroy the traditional
working principles of these institutions. The “institutions” meaning occurred in 22.9 per cent
(11/48) of fields in Cultural Revolution.
b) After the Cultural Revolution and before the late 1978, the “ institutions” meaning occurrence
dropped to 11.3 percent (5/44). The main field became the construction in social economy, including
socialist construction (17/44: 38.6%), ownership (2), economy (1), and manufacture industry (1),
which together occurred in 47.7 per cent (21/44) of collocations.
c) Since late 1978, the “institutions” meaning disappeared from the fields of revolution. Main field is
620
about the ownership (5), which ranked 1st in the collocations list. Other collocations (e.g. the
attribute (2), rule (2), theory (2), cognition(1), general truth (1) reason (1) indicate that the CCP
began to reflect on the revolution theoretically. Revolution became a object requiring study, rather
than a unquestionable spirit or claim. Besides, the juxtaposition of revolution and construction
indicates that revolution is considered as a process, which aim is to liberate the productivity, in order
to clear the way for social construction. Social construction now began to take place of revolution in
terms of the important role it plays in Chinese society.
Participants and Opposites:
a) Categorization: During the Cultural Revolution, subjects are categorized from the perspective of
social classes, as we can see from the distinction between proletarian (the people, cadre, mass),
capitalist. After the Cultural Revolution, the categorization turns to take basis on social groups, e.g.
the worker, cadre, mass, peasantry, PLA, criminals. A fact needs to be noticed that the opposites of
revolution in this period were criminals, which were opposite to the social orders and security, but
no longer the Capitalist during the Cultural Revolution, according to the classic points of view.
b) Alliance: Rather than make distinctions among the participants, discourse after the Cultural
Revolution began to introduce the notion of alliance into the participants of revolution. This explains
why words like (worker-peasantry alliance), (foundation),
(nation) and (main force) enter into collocations of the second period. Editorials producers
try to unite people from various social resources in order to strengthen the forces for social
construction.
c) Mao Zedong was no longer mentioned as (Chairman Mao) after the Cultural
Revolution. Since late 1978, producers use (Comrade Mao) instead. This is not
only a result of abandoning the “personal cult” , but also, by rejecting the symbol of traditional
revolutionary time, an indicator of detaching oneself from the old revolutionary thoughts appealed in
Mao” s Era.
d) Solidarity and power semantics: as a pronoun indicating ‘others’ , the term “ ” (they) was
used to refer to the opposites of the revolution during and after the cultural revolution (in 01/01/76 -
19/10/76, one for anti-revolutionist; in 25/10/76 - 18/11/78, six referring to anti-revolutionary, one
referring to members of Left Wing or people engaging political mistakes ). This is significantly
contrary to the situation in the Cultural Revolution, in which (they) also has ‘ participants
of revolution ‘ as its reference for three times.
e) After the Cultural Revolution, (they) was no longer used to refer to the people or the
mass. Instead of (they), editorial producers began to use (we) (one and one for
each period) to refer the readers (audiences) and the people as well. Also, with the pronoun
(we), the readers and the producers were mentioned together, which are both regarded as the
participants of revolution. This semantic change indicated that the top leaders of China try to
establish the solidarity rather than power relationship between the CCP and the mass. They try to
make up the distance between the leaders and the mass, and to have the power less explicit.
f) After the Cultural Revolution, the word (main forces) was used metaphorically to refer
to the massive people participating in the process of social construction of China. The ‘ main
forces’ , which was historically portrayed as brave, patriotic, was considered as prototype of
revolutionary spirit in China. Chinese leaders try to use a revolutionary discourse to match the
values of Chinese mass, in order to encourage their devotion to the social construction.
Internal relationship between participants:
Verbs encoding meaning of ‘leading’ ‘following’ , e.g. (carry out), (along
with), (cause), (according to), (direct), (implement),
(guide), (encourage), (follow), (instruct), (learn), occurred with
rapid decrease over the three periods. During the Cultural Revolution, words with “leading” meaning
occurred in 82.3 per cent (14/17) of relationship verbs. These verbs imply the relationship between
participants: the producers, as representative of the top leaders of china, is higher, brighter than mass
people, the people need to be educated to know how to justify and participate in the revolution.
Considering the semantic choices of pronouns when referring to the mass, there is also an undermined
indication: at that time, although the top leader often claimed that people are the owners of the country,
they are actually considered as “others”. With semantic choice of pronouns like (they), the
producers make a clear distinction between the mass people and themselves, i.e. the members
representing the power centre of China politics. Mass people are positioned at a ‘lower’ or ‘remote’
place, and actually excluded from the power centre. They have no power of speaking, judging, and
deciding. The only approved and encouraged participation in this revolution process is very simple: to
621
follow the leaders.
Expectation:
a) During the Cultural Revolution, the use of collocation verbs like (require), (do),
(engage) , indicate that “revolution” is mainly considered as stative situation. Collocations with
the meaning of ‘maintaining', ‘consolidating’ occurred in 43.9 per cent (29/66) in this period. The
figure kept nearly at the same level (42 percent by 11/26) during period 25/10/76 - 18/11/78. However,
since late 1978, collocations with meaning of “ maintaining” disappeared. Instead, collocates
describing revolution as dynamic process appeared (shift, resurrect, speed up). These collocates, which
indicate the meaning of “ transformation” , occurred totally in 61 per cent (8/13),
The changes indicate that people began to view revolution through an empirical and dynamic
perspective. Contrary to the expectation during the Cultural Revolution, in which revolution was
considered a mature cause to be maintained, peoples require more transformation and innovation on it.
Revolution is no longer regarded as a pre-determined concept or an undoubted slogan but rather a
process, people began to add something substantial in it and hope they can really get or gain something
from it. “ Revolution”, in this period actually became a metaphor to indicate the reformation that
China will take in all social areas, which includes politics, thoughts and economy.
Attitude:
For all periods, the semantic prosody of revolution is very positive. The word
(revolution) was used in connection with positive words like continue, inherit, develop, go ahead,
deepen, enhance and so on. This shows the positive attitude towards this word. The difference across
different periods is, after the Cultural Revolution, collocations indicating this positive attitude
decreased. In particular for period since late 1978, only one collocation of that use occurred. One
possible explanation for this is that people’ s use of the word “ revolution” decreased. Also, they
gradually lose interest in expressing their attitudes towards “ revolution” .
Requirement:
The valued characters of revolution have changed over the years. Contrary to that in the Cultural
Revolution, in which people are required to have characters like consciousness (desire to do something
from the drive inside), and to follow the directions from the top, since late 1978, people are encouraged
to do something creative, i.e. to (dare to think), (dare to say).
4. Discussion
1976 and 1978 were significant years for China. Top leaders, including Premier Zhou Enlai, Mao
Zedong and Zhu De died consequently in 1976. The Gang of Four was smashed in the October. The
whole nation was left not only with no clearly recognized leader but also with no clear ideological and
policy direction. People began to be doubted about the belief they have been insisted on during the
Cultural Revolution.
The vacuum between the downfall of The Gang of Four and the release of reform policy (1978)
was a period in which people reflect on the horrors and violence of that decade. Also in this period
CCP leaders like Deng Xiaoping, who was reinstated in late 1978, kept trying to set a new ideology for
the whole nation. It is very clear to Deng and his reform-allies that the ideological influence of Mao
Zedong would not allow for radical social reforms. But the task of uniting the whole nation on the basis
of the new ideology in order to make social shift from class struggle to economic reform is urgent. To
solve this contradiction, one of Deng's strategies is to gradually change the public political discourse
for the sake of social reform in productivity and social relationship. The change reflected in the use of
the keyword (revolution) is a typical evidence of such manipulation on discourse as well as
ideological perception. To certain degree, the these changes can be generalized as:
a) The frequency of revolution decreased along the four years. People gradually lose interest in
choosing the word. This indicates that the need for the use of this word decreased.
b) The use of FP increased along the years. FP was used to serve the aim to change the traditional
notions of revolution, to encourage people’ s acceptance of the new notions of (revolution).
Because (revolution), as an old key word in Chinese political discourse, cannot simply be
“kicked out” of people's habitual lexicon, the CCP try to change the meaning of this word by means
of the frequent usage of FP.
c) Viewed from the meaning components, the meaning changes can be represented as Table S-2.
d) The word (revolution) continues to be used in Chinese public discourse after the Cultural
Revolution while the meanings have changed a lot. Besides, this word is used mainly as an integral
part of fixed phrases. As for those used separately outside the FP, Deng took advantage of this term
to introduce his own strategy of reform and socialist construction. Rather than simply rejecting this
term, whose meaning was deeply rooted in the political context of Mao's era, Deng infused new
622
notions relating to social reform into the old key word, and appealed new social values in the process
of referring back to and reflecting on the traditional values of China.
Considering the relatively small amount of work, the coding work in this analysis was done by
hand. As for the main corpus, this coding work will be processed with the help of “Winmax” software –
software specifically designed for text coding and organizing.
Table. 1: Examples of basic ideological keywords
Category Examples
Stable frequency
Socialism, people, politics, ideology, line, mass, central, locality, unite, liberate,
consolidate, worker, Mao Zedong, Marx, Marxism-Leninism, Communism, army,
Taiwan, advanced, peaceful coexistence.
Frequency
dropping
obviously during
Cultural revolution
economy, development, construct, democracy, nationality cadre, improvement,
cooperate, peace, unify, science, technology, benefit, productivity, common,
independence, stability, culture, theory,
Frequency
dropping
obviously since
reform and opendoor
policy:
class, revolution, struggle, property, capital, dictatorship, Lenin, destroy, class
struggle, revolutionary, war.
Table S-1: Collocations of “ ” (ge ming, revolution) in editorials of “ PD” during 1976-1980
01/01/76 - 19/10/76 (7) 25/10/76 - 18/11/78 (15) 01/01/79 - 01/01/80 (22)
Fields
( the fields
that
revolution
is
introduced,
or what is
referred
when
talking
about
revolution)
[48/318: 0.150]
(8), (4),
(4), (3)
(3) (3) (3),
(3), (2), (2),
(2), (2), (1)
(1), (1),
(1), (1), ,
(1)
The proletarian dictatorship
(8), production (4),
practice(4), public
health(3), art(3), education
(3), object(4), guiding
principle(3), nature(3),
task(2), future(2), , socialist
construction(1), theory(1) ,
party(1), work(1),
policy(1), tradition (1), the
great debates in Qinghua
University(1) …
[44/292: 0.150]
(17)
(2), (1),
(1), (1),
(1) (1), (1),
(1), (1), (1),
(1), (1),
(1), (1),
(1) (1)
(1) (1),
(1)
Socialist construction (17),
ideological policy (2),
politics (1), thought (1),
economy (1), diplomacy
(1), theory (1), practice (1),
education (1), art (1),
public health (1),
technology (1), class
struggle (1), production
competition (1), scientific
experience (1), privacy
ownership (1), public
ownership (1),
manufacturing industry (1),
standard (1), principle
(1)…
[21/160:0.131]
(3) (2),
(2) (2),
(1) (1)
(1) (1)
(1), (1)
(1), (1), (1)
(1) (1)
(1)
Ownership (3), attribution
(2), rule(2), theory(2),
modernization construction
(1), privacy ownership (1),
public ownership (1),
machine building (1)
recognition (1), general
truth (1), movement (1),
reason (1), aspect (1),
center (1), production (1)
relationship (1), material
condition (1)…
623
Table S-2: The meaning change of ge ming, revolution since 1976-1980
01/01/76 - 19/10/76 25/10/76 - 18/11/78 01/01/79 - 01/01/80
Fields (Destroy the orders
and principles of)
social Institution,
As a weapon
Social construction,
As a weapon
Production ownership, economic
construction
As a theory
Participants and
Opposites
Classes,
Distinction,
opposite
Social groups,
Alliance
Social groups
Alliance
Internal
relationship
between
participants
Power semantic Solidarity semantic Solidarity semantic
Relationship
between
participants and
opposites
Hostile Hostile Hostile
Expectation Stative situation, to
be maintained and
consolidated
As a concept or
slogan
Stative situation, to be
maintained
As a concept and
process
Dynamic process, to be
transformed
As a substantial process
Attitudes Positive Positive Indifferent
Evaluation Positive, eulogistic Positive, eulogistic Positive
Requirement Consciousness,
drive from inside
Zeal, bravery Creativity
References
Althusser L 1970 Idéologie et appareils Idéologiques d'État (notes pour une recherche). La Pensée, 151:
3-38.
Althusser L 1965. Pour Marx. Paris: Maspero.
Bishop R L 1989 Qi Lai! Mobilizing One Billion Chinese: the Chinese Communication System. Ames:
Iowa State University Press.
Bourdieu P 1977 Outline of a Theory of Practice. Cambridge. Cambridge University Press.
Bourdieu P 1980 Le mort saisit le vif. In Actes de la Recherche en Sciences Sociales, 33: 3-14.
Bourdieu P 1991 Language and Symbolic Power. Cambridge: Harvard University Press.
Chang T K, Wang J, Chen CH 1994. News as social knowledge in China: The changing worldview of
Chinese national media. In Journal of Communication, 4(3), 52-69.
Fairclough, N 1990 What might we mean by “ enterprise discourse” ? In R. Keat and N. Abercrombie
(Eds). Enterprise Culture. London: Routledge, 38-57.
Fairclough NL 1992 Discourse and Social Change. Cambridge: Polity Press.
Fairclough NL 1995a Media Discourse. London: Edward Arnold.
Fairclough NL 1995b Critical Discourse Analysis: The Critical Study of Language. London: Longman.
Fairclough NL, Wodak R 1997 Critical discourse analysis. In T. A. Van Dijk (Ed.), Discourse studies: A
Multidisciplinary Introduction, Vol.2, 258-284. London: Sage.
Firth JR 1935 The technique of semantics. Transactions of the Philological Society, 36-72.
Foucault M 1972 The Archaeology of Knowledge. London. Tavistock.
Foucault M 1979 Discipline and Punish: the Birth of the Prison. New York: Vintage Books.
Foucault M 1980a The History of Sexuality, Vol. I: An Introduction. New York: Vintage Books.
Foucault M 1980b Power/Knowledge: Selected Interviews and Other Writings, 1972-1977. Pantheon
Books.
Gee J P 1990 Social linguistics and literacies: Ideology in Discourses. London: Falmer Press.
Geertz, C. 1964. Ideology as a cultural system. In D. Apter (Ed.). Ideology and Discontent. New York:
The free Press.
He, Z 2000, forthcoming. Working with a dying ideology: dissonance and its reduction in Chinese
journalism.
624
He Z, Chen H L 1998. The Chinese Media: A New Perspective. Hong Kong: Pacific Century Press.
Hodge R, Kress G 1993. Language as Ideology. London: Routledge.
Lee C C (ed.) 2000 Money, Power and Media: Communication Patterns in cultural China. Chicago:
Northwest University Press.
Lenin V. I. 1970 [1905]. What is to Be Done? London: Panther.
Liu, Alan PL 1971 Communications and National Integration in Communist China. Berkeley:
University of California Press.
Liu, Alan PL 1981 Mass campaigns in the People’ s Republic o China during the Mao era. In R. E. Rice
and C. K. Atkin (Eds.), Public Communication Campaigns. New York: Oceana.
Lukacs G 1971 History and Class Consciousness: studies in Marxism. Dialectics. London: Merlin
Press.
Malrieu J P 1999. Evaluative semantics: Cognition, Language and Ideology. London: Routledge.
Mannheim K 1960 [1933]. Ideology and Utopia: an Introduction to the Sociology of Knowledge.
London: Routledge and Kegan Paul.
Marx K, Engels F 1974 [1845]. The German Ideology. London: Lawrence and Wishart.
Moon R 1994. The analysis of fixed expressions in text. In M. Coulthard (ed.), Advances in Written
Text Analysis. London: Routledge, 117-35.
Moore T E (ed.) 1973 Cognitive Development and the Acquisition of Language. New York and London:
Academic Press.
Murata T 1998 A study on Ideological Changes through language in Political reports of Chinese
Communist Party’ s Plenary Sessions. In B. K. T’ sou, T. B. Lai, S. W. K. Chan and S. Y. W. Wang
(eds.), Quantitative and Computational Studies on the Chinese Language (pp. 209-234). Hong
Kong: City University of Hong Kong, Language Information Sciences Research Center.
Pecheux M 1982 Language, Semantics, and ideology: Stating the Obvious. London: Macmillan.
Said E. 1978 Orientalism. London: Routledge and Kegan Paul.
Scott M 1991 Demystifying the Jabberwocky: A research narrative. PhD dissertation. University of
Lancaster.
Shils E 1968 The concept and function of ideology. International Encyclopedia of the Social Sciences,
vol. 7, pp.66-76. New York: The Macmillan Company and Free Press.
Stalin I V 1951 On Marxism in Linguistics. In The Soviet Linguistic Controversy (translation from the
Soviet Press by J V Murra, R M Hankin, F Holling). New York: King’ s Crown Press, pp. 70-76.
Stubbs M 1996 Text and Corpus Analysis. Oxford: Blackwell.
Van Dijk T A 1988 News Analysis. New Jersey: Lawrence Erlbaum Associates.
Van Dijk T A 1998 Ideology : a Multidisciplinary Approach. London: Sage.
Wierzbicka A 1984 Lexicography and Conceptual Analysis. Ann Arbor: Karoma Publishers.
Wierzbicka A 1992 Semantics. Culture and Cognition: Universal Human Concepts in culture-specific
Configurations. Oxford: Oxford University Press.
Wierzbicka A 1997 Understanding Cultures through Their keywords: English, Russian, Polish, German
and Japanese. New York: Oxford University Press.
Williams R 1961 Culture and Society 1780-1950. Harmondsworth: Penguin.
Williams R 1983 Keyowrds: A Vocabulary of Culture and Society. London: Fontana Press.
Wu G G 1994 Command communication: the politics of editorial formulation in the People’ s Daily.
The China Quarterly, 137, 194-211.
Zhao Y Z 1998 Media, Market, and Democracy in China: Between the Party Line and the Bottom Line.
Urbana and Chicago: University of Illinois Press.

625
A corpus-based study of interaction
between Chinese perfective -le and situation types
Zhonghua Xiao, Lancaster University
Mandarin Chinese as an aspect language (Norman, 1988:163) has a rich inventory of aspect markers,
including perfective -le1, -guo and imperfective -zhe, zai. Of these -le is the most studied marker
because of its mysterious behaviors. For example, is it necessary to differentiate between the perfective
-le and the COS le? Does -le indicate completion or termination? Are there any constraints on the
interaction of -le with various situation types? All of these issues have aroused much controversy
because different authors have invented different “acceptable” examples to support their arguments.
Many of these examples, however, are rarely found in real language, though they are good for the
purpose of argumentation.
In this study, I will take another approach to address these issues and find evidence from authentic
language data. An L1 Chinese corpus of 124,164 Hanzi (Chinese characters) was compiled for this
purpose. The corpus was first automatically segmented and POS-tagged, and post-editing was
conducted by hand for the tagging of the marker LE to ensure consistency and accuracy. Then all the
clauses containing -le and le are extracted into two databases and the situation type of each instance is
judged on the basis of human decision. A total of 1,208 occurrences of LE are found in our data, of
which 1019 are the perfective -le and 166 are the COS le. In the other 23 instances where LE appears
in the sentence-final position, the morpheme has the dual function indicating both perfectivity and
change of state. Other functions of LE, e.g., as a full verb, as a modal particle and as a bounded
morpheme, are also found in the corpus. Because they are irrelevant to our study here, these functions
are not counted. The high frequency of LE and its rich functions justify this corpus as a good basis for
the case study of this morpheme, albeit the small corpus size.
This paper is concerned with the three questions raised at the beginning and is organised as follows:
Section 1 discusses the one-morpheme approach vs. the two-morpheme approach; Section 2 considers
the type of closure indicated by the perfective -le; Section 3 examines the interaction between the
perfective -le and situation types and Section 4 concludes.
1. Verbal -le vs. sentential le
There is an unanimous agreement that -le is a perfective aspect marker (e.g., Chao, 1968; Henne &
Rongen & Hansen, 1977; Smith, 1991, 1997; Zhang, 1995, Dai, 1997). Yet much controversy arises
when it comes to whether the perfective -le and the COS le have the same functions. While the twomorpheme
approach focuses on their differences in terms of syntactic distributions, semantic
functions, and etymological sources, the one-morpheme approach focuses on their semantic
similarities. Zhang (1995: 120), for example, supports the unified treatment of LE and describes its
major functions as denoting a change of state by termination and establishing a boundary between two
different situations. However, despite her explicit favour for the one-morpheme approach, she has to
turn to the two-morpheme approach to explain the interchangeability of -le and -guo (see Zhang,
1995:217-219).
I argue in favour of the two-morpheme approach. As suggested above, the perfective -le and the COS
le differ in terms of syntactic distributions, semantic functions, and etymological sources. The terms
“verb-final suffix” -le and “sentence-final particle” le (Li & Thompson, 1981) best illustrate their
difference in syntactic distributions. Syntactically, the perfective -le occurs post-verbally while the
COS le appears in post-sentential position. However, when an intransitive verb2 takes the sentence
final position, we have to take into account the different semantic functions of the two morphemes to
determine which LE we have in front of us. The perfective -le focuses the actualisation of a situation
and presents it as a whole; the COS le, on the other hand, mainly indicates a change into a new
situation and signals its current relevance3. There are three possibilities for LE taking the sentence
1 In this study, the morpheme LE in the verb-final position indicating perfectivity is glossed as -le while that in the sentence-final
position indicating change of state (COS) is glossed as le. The capitalised LE refers to either.
2 As transitive verbs are always followed by their objects, they cannot possibly appear in the sentence final position.
3 “Current” should be interpreted in relation to the reference time rather to the speech time.
626
final position. It can be either the COS le if the sentence only allows a change-of-state reading, or the
perfective -le if the sentence only has a perfective reading, or it has dual function if the sentence has
both the change-of-state and perfective readings (c.f. Li & Thompson, 1981:296). In this last case, the
additional COS le is absorbed into the first perfective -le, as Chinese “always avoids a repetition of the
same syllable by way of haplology: -le le”(Chao, 1968:247)4.
Historically, the perfective -le and the COS le developed at different stages of evolution. The COS le
is derived from the verb liao “to finish, to come to an end” (the same syllable with a different
pronunciation), as in siliao “to settle out of court ”. When its sentence-final function was well
established, it also developed a use in which it appears directly after the main verb (whether or not it
is sentence-final) functioning to signal perfectivity. Therefore diachronically, the COS le developed
earlier and gave rise to the perfective -le (c.f. Bybee, 1993:84-85). The evolution of these two
morphemes also furnishes evidence in favour of the two-morpheme approach: if they are the same and
one morpheme can function adequately, why is it necessary for the other to be derived?
The differentiation between the perfective -le and the COS le is also supported by the quantitative data
in our corpus. Of a total of 1,208 occurrences of LE, 1,019 (84.36%) are the perfective -le, 166
(13.74%) are the COS le, and in 23 instances (1.9%) the morpheme denotes both COS and
perfectivity. The ratio of the perfective -le over the COS le is 6.139. The higher frequency of the
former over the latter is predicated because our corpus mainly contain narrative discourses, of which
the perfective aspect is a prominent syntactic feature. Our finding here is in conformity with
Christensen (1994), who finds a ratio of 6.818 for the written narratives, as shown in Table 1 below:
Table 1: Frequency data for LE:
data total -le le dual -le/le
Our corpus 1208 1019 84.36% 166 13.74% 23 1.90% 6.139
C's written 86 75 87.21% 11 12.79% 0 6.818
The table shows that the perfective -le is more productive than the sentential le and dual-function LE.
Christensen's written data show a higher frequency of -le because his data are purely narrative.
It can be seen from the discussions above that the perfective -le is different from the COS le in many
respects: (i) syntactically, -le appears in the verb-final position whereas le in sentence-final position;
(ii) semantically, -le signals perfectivity whereas le indicates change of state; (iii) etymologically, -le
is derived later than le, and (iv) empirically, -le is more productive than le. All of these argue strongly
for the two-morpheme approach.
2. Completion vs. termination
Another issue which is as controversial as the one discussed above is the type of closure signalled by
the perfective -le. Traditionally, the perfective -le is considered to indicate completion of the action
denoted by the verb. Chao (1968:247), for example, argues that the verbal -le has the class meaning of
“completed action”. Following Chao, Henne & Rongen & Hansen (1977:117) claim -le indicates “the
completed action of the verb to which it is attached”. Similar views can also be found in Zhu (1981),
Lü (1981:314-321) and Tiee (1986:96). But the traditional view cannot account for the puzzle in (1)
below:
(1a) zhe-ben xiaoshuo wo kan-le san-tian
this-CL novel I read-le three-day
I read the novel in three days (I finished reading it).
(1b) zhe-ben xiaoshuo wo kan-le san-tian le
this-CL book I read-le three-day le
I have been reading this book for three days (I haven't finished reading it).
4 According to Chao (1968), in certain dialects such as Cantonese and the Wu dialects, there are separate morphemes to indicate
actuality and change-of-state which can co-occur contiguously. Haplology of -le le only occurs in the Mandarin Chinese (putonghua
“the common language”).
627
Clearly (1a) and (1b) have different aspectual meanings. While the former indicates the completion of
the reading event, the latter gives no such indication. If LE indicates completion, “why a completed
reading is derived when one LE is used, but it is not allowed when an additional LE is used?”5
Interestingly, all of the scholars quoted above relate completion to the action of a verb rather than a
situation. Their approach is clearly incompatible with the definition of the aspect6. In fact, the
compositional nature of aspect is widely observed in the literature (e.g., Verkuyl, 1972; 1993; Smith,
1991; 1997; Brinton, 1988). Therefore the aspectual value of a situation is contributed to by the
semantic features of all sentential elements, though a verb plays an important role.
More recent studies, however, realize that the perfective -le does not necessarily indicate completion.
While perfectives with a resultative verb complement (RVC) unequivocally indicate completion,
“simple perfectives” (Smith, 1997:264) 3 sentences with -le alone but without RVCs 3 only present
situations without indication of the closure type7 (Smith, 1988:216,218; Tai, 1984:291-292; Chu,
1976: 48). This view has won growing popularity in the literature (e.g., Li & Thompson, 1981:215-
216; Zhang, 1995: 115-116; Christensen, 1994; Smith , 1997: 264-265; Dai, 1997: 21).
While I agree to this recent view in principle, I argue that the type of closure indicated by -le is not so
arbitrary as Smith (1988:228) claims: “Semantically, sentences without completive RVCs do not
present a completed event; but pragmatically, they often do just that.” Such arbitrariness has led to
much confusion in her own studies. Let us consider Smith's examples (1988:218-219):
(2a) wo zuotian xie-le yi-feng xin
I yesterday write-le one-CL letter
I wrote a letter yesterday.
(2b) *wo zuotian xie-le yi-feng xin, keshi mei xie-wan
I yesterday write-le one-CL letter, but not write-finish
Lit.: I wrote a letter yesterday, but didn't finish it.
(2c) wo zuotian xie-wan-le yi-feng xin
I yesterday write-finish-le one-CL letter
I finished writing a letter yesterday.
Smith (1988:218) argues that sentences like (2a) “present events as terminated but not necessarily
completed”, but in Smith (1997) she contradicts her own assertion by admitting that in fact, the most
natural interpretation of (2a) would be that the letter was finished, though in order to remedy the selfcontradiction,
she adds immediately that “the completive interpretation is conversationally only: it can
be cancelled by other information” (1997:265), as shown in (2b). But I argue that (2b) in fact sounds
unacceptable semantically, if not grammatically, to a native ear (c.f. also Teng, 1986). And like (2c)
in which -le co-occurs with the RVC -wan “to finish”, (2a) also indicates the completion of the
writing event, i.e., the letter was finished. If we followed Smith's assumption that completive readings
denoted by “simple perfectives” can be cancelled, we would have the following absurd situation:
(3) *shanggeyue ta sheng-le yi-ge nanhai, keshi mei sheng-wan
last month she give:birth-le one-CL boy but not give:birth-finish
Lit.: Last month, she gave birth to a baby boy, but did not finish it.
It is true that “simple perfectives” may indicate either completion or termination, but the type of
closure depends on the type of situation. That is, telic situations8 are presented as completed whereas
atelic situations are presented as terminated. When a telic situation is presented perfectively as a
single unanalysable whole, its inherent final endpoint is naturally included, thus resulting in a
completive reading. On the other hand, an atelic situation does not have an inherent end point, so
when it is presented perfectively, only an arbitrary final endpoint is included, and thus a terminated
reading is appropriate. (2b) above will become acceptable if the quantified direct object is replaced by
a bare noun9, as shown in (4):
5 Translated from Dai's (1997:21) quotation of Lü (1961) “The current task for researchers of Mandarin Chinese.”
6 In our model, we follow Smith's (1997:1) and define aspect as “the semantic domain of the temporal structure of situations and their
presentation.”
7 Chu (1976), in his study of action verbs, also finds that the structure of “action verb+-le” only indicates active attempt and actual
performance rather than attainment of goal, while the structure of “action verb+RVC” indicates all of the three.
8 Telic/atelic distinction is an important distinguishing feature for aspectual classification. A situation is telic if it has an inherent
spatial final endpoint.
9 As there is no articles in Chinese, and the plural suffix -men is syntactically optional, bare nouns in Chinese can be regarded as bare
plurals in English.
628
(4) wo zuotian xie-le xin, keshi mei xie-wan
I yesterday write-le letters but not write-finish
I wrote letters yesterday, but I didn't finish them.
The acceptability of (4) can be explained as follows. In this sentence, the object xin “letters” is a bare
noun, which is at best ambiguous between specific and non-specific readings. When it interacts with
the accomplishment verb xie “to write”, the resulting situation can be naturally understood as atelic.
Thus the situation conveyed by the first clause in (4) has a terminated reading and further assertion
can be made that the letters were not finished.
The above analysis suggests that the type of closure indicated by the perfective -le is related to
situation types. Smith (1988: 218) also realizes this point when she claims:
The choice between termination and completion arises only with telic events, of course. Atelic events have no
other possibility besides termination.10
While agreeing to the second part of this claim, I argue that no choice is open to telic situations either.
That is, for telic situations, only completive readings are possible. Let us examine the three
examples11 Smith (1988) uses to support her claim.
(5a) Zhangsan xue-le Fawen, keshi mei xue-hui
Zhangsan study-le French but not learn-know
Zhangsan studied French, but he still didn't know it.
(5b) *wo mai-le san-ben shu, keshi mei mai-dao
I buy-le three-CL book but not buy-succeed
Lit.: I bought three books, but I didn't buy them.
(5c) Zhangsan zhao-le ta de shoubiao, keshi mei zhao-dao
Zhangsan look:for-le he DE watch but not look:for-succeed
Zhangsan looked for his watch, but he didn't find it.
The first point to be noted here is that Smith asserts that completive readings in (5a)-(5c) are
cancelled by the conjuncts (Smith, 1988:288). On a closer examination, however, we find xue Fawen
“study French” and zhao ta de shoubiao “look for his watch” are both atelic events, because only xuehui
Fawen “to learn French” and zhao-dao tade shoubiao “to find his watch” are telic (c.f. Smith,
1988:220, 234; Tai, 1988: 290). If in (5a) and (5c) -le did signal completive readings which were
cancelled by the conjuncts, Smith would be contradicting her own claim quoted above that only
termination is possible for atelic situations. Secondly, while Smith is right in saying that the first
clauses in (5a) and (5c) do not have completive readings, she is wrong in the case of (5b). For the
same reason discussed in the analysis for (2) above, mai san-ben shu “buy three books” in (5b) is a
telic event, and thus its completive reading cannot be cancelled. Therefore, this is an invalid example
to serve her purpose. Smith is on the right track when she realizes that “because telic events involve
completion, they may be used to implicate completion” (Smith, 1988:228). But regrettably, she
attributes the final decisive role of the closure type to pragmatics.
Tai (1984: 291-292) also observes that “Vendler's examples of accomplishment expressions such as
‘to paint a picture’ and ‘to write a letter’ may or may not imply attainment of goal in Chinese”
(ibid:291). Tai's observation is true to the fact, but the reason he provides for this 3 “depending on
the particular context which a native speaker happens to be in” (ibid:291) 3 is not. It is argued here
that the closure types of these situations depend on how we translate these phrases. If we translate “to
paint a picture” as huahua and “to write a letter” as xiexin, then they are atelic. When they are
presented perfectively with the verbal -le, only terminated readings are possible; but if we translate “to
paint a picture” as hua yi-fu hua and “to write a letter” as xie yi-feng xin, then they are telic situations
and only allow completive readings when presented perfectively12.
Tai (1984:291) argues that sentences like (2a) may imply the attainment of goal “for many native
speakers”, but sentences like (2b) “suffice to show the implication is not absolute.” Tai's argument is
10 But regrettably, even this claim is negated later by herself: “But in Chinese perfectives termination and completion are expressed
separately for all situation types” (1997:73), which in turn is contradicted by her own assertion that accomplishments may be either
terminated or completed with simple perfective viewpoint (1997:264).
11 This pair of examples are taken from (Smith 1988:220). But the English translations of (a) and (c) are modified, because in (a),
according to Tai (1984:290-291), xue and xue-hui in Chinese can find equivalents in English: “study” for xue and “learn” for xuehui.
While xue and “study” are atelic, xue-hui and “learn” are telic. In (c), the same applies: while zhao and “look for” are atelic,
zhao-dao and “find” are telic.
12 These two translations are both possible because Chinese has no articles.
629
even less convincing than Smith's claim that “the completive interpretation is conversational only”
(1997:265). One problem with Tai's argument is its unreliable theoretical basis. “For many native
speakers” is a rather vague concept: how many? what percentage? Unfortunately Tai has not made a
demographic survey. Another problem is the acceptability of his counter-examples. As noted above,
sentences like (2b) are in fact unacceptable semantically. If Tai had followed the convention of
treating Chinese as a “non-article” language and had not translated these two phrases so literally, he
might have come to the point.
Our argument for the correlation between closure type and situation type13 does not go far away from
Smith (1988:218), because she also agrees that atelic events14 can only be interpreted as terminated.
We differ in our treatment of telic events. Smith's accomplishments are of two types: one is the simple
form like xie yi-feng xin “to write a letter”, the other is the RVC form like xie-wan yi-feng xin “to
write-finish a letter”. Her second type of accomplishment falls within the category of achievements in
our model15. As an achievement encodes result in itself and is punctual by nature, it is expected that
once such a situation is realised, it is completed. This prediction is supported by the empirical
evidence. Of the 510 achievements taking the perfective -le found in our corpus, all have completive
readings without exception. Here are some examples:
(6a) na jiahuo shao-cheng-le hui, wo ye neng ren-chulai (File 9558601)
that guy burn-become-le ash I too can recognise
Even if that guy was burnt into ashes, I would recognize him.
(6b) ta...zhidao yu-shang-le gaoshou (File 9560501)
he...know encounter-le master-hand
He knew that he had encountered a master-hand.
Our difference with Smith in this respect revolves around the closure type of accomplishments (her
simple form accomplishments) when they take the perfective -le. My argument is that
accomplishments can only be interpreted as completed, whereas Smith assumes that this type of
situation may have a choice between termination and completion. This assumption, however, is
ungrounded, because the counter-examples she uses for the contradiction test, e.g., (2b) and (5b), are
semantically unacceptable. Smith's assumption also lacks empirical evidence. Even if her intuition is
correct when she invents such examples, these utterances are not supposed to be found in real
language. In our corpus data, all of the 326 accomplishments taking the perfective -le can only allow
completive readings. Let us consider an example cited from the corpus:
(7a) wo jimang yi gaojia zhu-le yiliang Beijing jipuche, zhishi Wangzhuang (File
9560601)
I hurriedly with high:price hire-le one-CL Beijing jeep direct:drive Wangzhuang
I hurriedly hired a “Beijing” jeep at a high price, and headed direct for Wangzhuang.
(7b) *wo jimang yi gaojia zhu-le yiliang Beijing jipuche, keshi mei zhu-dao
I hurriedly with high:price hire-le one-CL Beijing jeep but not hire-succeed
Lit.: I hurriedly hired a “Beijing” jeep at a high price, but didn't succeed hiring it.
The situation “I hired a Beijing jeep” in (7a) is an accomplishment presented perfectively. (7b) shows
that even if a conjoined second clause could cancel its completive reading, the second clause would
clash with some other sentential element, i.e., “at a high price”. We normally assume that when the
price is settled, the deal is done. Furthermore, if the completive reading of the actualised
accomplishment could be cancelled, there would be no subsequent event “headed for Wangzhuang”.
Therefore, our argument for the positive relation between telicity value of a situation and its closure
type is supported by both theoretical analysis and empirical evidence.
3. Interaction between -le and situation types
Before we go on to examine the interaction between the perfective -le and situation types, it is
necessary to make a brief introduction to our aspect model. Following Smith (1991; 1997), aspect is
taken to have two components, namely, situation aspect and viewpoint aspect. The former is
concerned with the inherent temporal features of a situation while the latter provides a perspective to
view the situation. Aspect is the synthetic result of these two components.
13 Pan (1993) also observes that “different situation types influence the interpretation of perfective”, with an accomplishment, -le
indicates that the event started and finished later; with an activity, -le indicates it started and terminated later.
14 Smith argues that the perfective does not interact with statives in Chinese, which, according to our data, is not true (see Section 3).
15 In our model, all verbs that encode result are classified as achievements (See Section 3).
630
In our model, situation aspect is concerned with both the lexical and the sentential levels. This twolevel
approach is different from Vendler (1967) and Smith (1991, 1997). At the lexical level, verbs are
grouped into six classes based on five distinguishing features16, as shown in Table 2:
Table 2: Feature matrix system of verb classes:
classes [±dynamic] [±durative] [±bounded] [±telic] [±result]
activities + + - - -
semelfactives + - ± - -
accomplishments + + + + -
achievements + - + + +
individual-level states - + - - -
stage-level states ± + - - -
These verb classes interact with their arguments and adjuncts at three different levels according to the
following rules:
A: Lexical level:
Rule 1: Verb[-telic/±bounded] + RVCs . Derived Verb[+result/+telic]
Rule 2: Verb[ -telic/ ±bounded] + reduplicant. Derived Verb[+bounded]
B: Core sentence level:
Rule 3: NP+Verb[+telic] + NP[ acount] . Situation[ atelic]
17
Rule 4: NP[ acount] + Verb[+telic] (+ NP) . Situation[ atelic]
Rule 5: NP+Verb[–telic]+PP[Goal] . Situation[+telic]
C: Full sentence level:
Rule 6: Core-sentence[ -bounded]+for-PP/from...to . Full-sentence[+bounded]
Rule 7: Core-sentence[+telic]+for-PP/from...to . Full-sentence[ -telic]
Rule 8: Core-sentence[ ±bounded]+Quantity NPs . Full-sentence[+bounded]
Rule 9: Core-sentence[+telic]+Progressive . Full-sentence[ -telic]
Rule 10: Core-sentence[ -result] + ba/bei-construction . Full-sentence[+result]
The interaction at these levels result in six basic situation types and five derived types as shown
below:
Table 3: Feature matrix of situation types:
Situation Types [±dynamic] [±durative] [±bounded] [±telic] [±result]
ILS basic - + - - -
derived - + + - -
SLS basic ± + - - -
derived ± ± + - -
ACC + + + + -
ACT basic + + - - -
derived + + ±18 - -
SEM basic + - ± - -
derived + + ± - -
ACH basic + - + + +
derived + + + + +
16 In addition to the three traditional ones, two new features, [±bounded] and [±result], are introduced to separate verb classes from
situation types. Both telicity and boundedness are related to final endpoint, but the former is spatially defined while the latter is
temporally defined. The feature of [±result] refers to whether or not a verb encodes result in itself.
17 a is a variable with the value of either plus or minus. +Count] NPs should be understood as singular or specific plural countable
NPs or “quantised” arguments in Krifka's (1987, 1989) terms, while [-count] NPs include mass nouns and bare plurals. The
[±count] distinction is similar to Smith's count/mass opposition or Verkuyl's (1993) [±SQA].
18 Derived activities have the value of [±bounded] because they represent a complicated categories. When basic activities are
delimited by specific time frame, they are [+bounded]; when accomplishment verbs take [-count] NPs or the progressive, or when
semelfactives allow indefinite multiple event readings, they are derived activities with the value of [-bounded].
631
It should be noted that situations types discussed here are the final result of composition processes at
the full-sentence level. When basic states and activities are temporally bounded by delimiting
mechanisms, bounded states and bounded activities come as a result. Derived activities can also be
obtained from accomplishments taking the progressive, semelfactives occurring the progressive or
temporal adverbials indicating indefinite time frame, and achievements taking the progressive or
[-count] NPs. Accomplishments do not have derived situation type19. The basic semelfactives have
single-event reading; when they occur with quantity NPs or temporal adverbials indicating definite
time frame, they become derived semelfactives. When basic achievements take [+count] NPs, derived
achievements come as a result. Having discussed the temporal features of situation types, we are now
in a position to examine the interaction between the perfective -le and situation types.
More recently, this topic has attracted much interest. Smith (1997:70,264) and Pan (1993), for
example, assert that the perfective -le is not available to states. Pan (1998) becomes aware of the
distinction between stage-level and individual-level predicates and corrects his generalization as
“perfective marker -le can be used only with stage-level predicates which include some of the
statives”. “Some of the statives” here refer to stage-level states (SLS) like ta bing-le san-tian “He was
ill for three days”. Smith and Pan's assertions suggest that the perfective -le is sensitive to the feature
of dynamicity.
On the other hand, Li (1999) argues that the perfective -le only appears in telic situations20 like
accomplishments and achievements, but not in atelic situations like states and activities21. Yang
(1995) is aware of the different natures of spatial and temporal endpoints. She argues that all
situations with a spatial final endpoint (i.e., telic situations) can be presented with the perfective
viewpoint marked by -le. In addition, atelic situations (including states), when they are temporally
bounded by delimiting mechanisms, can also take the perfective -le. But without such delimiting
devices providing a temporal boundary, atelic situations cannot felicitously co-occur with -le. The
arguments made by these two authors suggest that the perfective -le is sensitive to spatial or temporal
endpoint.
Yang's observations appear to be closer to the fact, but her categorical statement that no [-bounded]
situation can take -le (ibid:115) is arguable, because our data does not allow for a clear-cut distinction.
Based on our corpus data, I argue the perfective -le is more sensitive to the feature of telicity and
boundedness than to dynamicity as Smith and Pan suggest. But the sensitivity is rather a matter of
degree. As can be seen in Table 3, activities and two types of states are inherently [-bounded] and
[-telic], while accomplishments and achievements are intrinsically [+bounded] and [+telic].
Semelfactives are [-telic] but shift between [+bounded] and [-bounded]. Therefore, we expect the
perfective -le to be more likely to co-occur with accomplishments and achievements. This prediction
is in fact borne out of the corpus data. A breakdown of the situations taking -le in the corpus is given
as follows:
Table 4: A breakdown of situations taking the verbal -le:
ILS SLS ACT SEM ACC ACH Total
29 19 109 26 326 510 1019
2.85% 1.86% 10.70% 2.55% 31.99% 50.05% 100%
From these figures, it is clear that more than 80% of the total are telic situations. Furthermore, of the
atelic situations (accounting for around 18% of the total), more than half involve a temporal boundary
provided by delimiting devices. Specifically, 82 out of 109 activities, 16 out of 26 semelfactives, and 2
out of 19 SLS are temporally bounded, taking up 9.81% of the total. When these [+telic] and
[+bounded] situations are taken together, they account for more than 90% of the situations taking the
perfective -le in our data. The chi-square test shows that our result is highly significant. This indicates
19 Because the derived situation types of accomplishments have exactly the same feature values as their basic types, these two are
combined into one.
20 Although Li (1999) also uses the term “bounded”, she actually intends the term to mean “telic”, because in her model,
“boundedness” actually refers to “the natural final point signaling change of state”.
21 Li (1999) does not differentiate between achievements and semelfactives, nor is she aware of the distinction between SLS and ILS.
632
a strong tendency for -le to occur with situations with spatial or temporal endpoints. In what follows,
we'll discuss the interaction of the perfective -le with various situation types.
States may hold for an indefinite interval and are therefore intrinsically open-ended. This feature
explains their relatively low co-occurrence frequency with the perfective -le. ILS verbs are predicated
of the more permanent dispositions or “properties” of an individual. Because -le only functions to
present a situation in its entirety but does not provide any endpoint, the mere addition of -le to ILS
normally does not result in grammatical sentences unless a temporal boundary is explicitly provided
by an extra delimiting device. But this requirement is not absolute. Here are some corpus examples, in
which ILS verbs are italicised.
(8a) Yindu he Bajisitan ye you-le he nengli (File 9558801)
India and Pakistan also have-le nuclear capacity
India and Pakistan also had nuclear capacities.
(8b) Yang Qinxian jiu jubei-le zhe-lei renwu de quanbu tezheng (File 9559901)
Yang Qinxian then possess-le this-type people DE all characteristics
Yang Qinxian bears all of the characteristics of a dangerous person.
In both sentences, ILS situations are not bounded. The perfective -le indicates that these situations are
presented as a single whole. But it should be noted that -le in these sentences can be omitted without
significant change in meanings. This shows that ILS behave quite differently from other situation
types in respect of aspectual marking: while the latter have to be marked aspectually, either overtly or
covertly, to have a specific closed reading, the former do not have this requirement (c.f. also Yang,
1995:108; Moens, 1987). In this respect, SLS are more “event-like” because they also have to be
marked aspectually. Compare the acceptability of the following:
(9a) yi-ge laotaipo chulai, jian shi ji-ge jingcha, dunshi huang-le shen (File
9560701)
one-CL old woman come:out see be some-CL police at:once scare-le spirit
An old woman came out. She was scared out of her wits when she found the visitors were
some policemen.
(9b) shuo dao zher, Zhang Dandan shiran-le (File 9561301)
say reach here Zhang Dandan at:ease-le
Having said these, Zhang Dandan felt at ease.
These two sentences denote SLS. If the perfective -le was removed, they would become
ungrammatical. In this sense, SLS are more akin to non-statives than to ILS.
Activities are intrinsically neither telic nor bounded unless there is an extra delimiting device
providing them with a temporal boundary. Because the perfective -le is sensitive to endpoint, we
predict activities taking -le are more likely to be temporally bounded. This prediction is in fact
supported by our empirical data. Out of the 109 activities taking the perfective -le found in our corpus,
82 have a temporal endpoint provided by some delimiting mechanism, accounting for more than three
quarters of the total. This piece of evidence also tells against the claim made by some scholars (e.g.,
Yang, 1995:116 and Li, 1999:216) that atelic or unbounded situations can never take -le. Rather, our
data show that the compatibility is merely a matter of tendency. Consider the following corpus
examples:
(10a) ta pai-le wushu-ge huaqian-yuexia de baima-wangzi (File 9560301)
he act-le countless-CL romantic DE white knight
He has acted countless romantic white knights.
(10b) yi-ge xiao nühai... beishang de ku-le qilai22 (File 9560701)
one-CL little girl sadly DE cry-le start
A little girl began to cry sadly.
The situations described in (10) are both unbounded activities, but it is not hard to find them in real
language. The verb pai “to play the part, act” in (10a) is an accomplishment verb, but its interaction
with a [-count] object NP (modified by wushu-ge “countless”) results in an atelic situation; ku “to cry”
in (10b) is also an activity with no endpoint. In these cases, the perfective -le simply focuses on the
realisation of these situations and gathers them in their entirety. In comparison, bounded activities
take the perfective -le more easily. Our data register a ratio of 3.04:1 between bounded and unbounded
22 The suffix -qilai is an imperfective aspect marker indicating inceptiveness.
633
activities. As activities are inherently unbounded, their temporal final endpoint is normally provided
by an extra delimiting mechanism. Consider the following examples:
(11a) xingxun yanxu-le san-ge xiaoshi (File 9556901)
inquisition by torture last-le three-CL hour
The inquisition by torture lasted as long as three hours.
(11b) na hanzi zuoyou xunshi-le yi-fan, disheng dao... (File 9557601)
that man left:right look-le one-CL low:voice say
The man cast his eyes around, and said in a low voice...
(11c) wo huitou wang-le wang zhe-ge popo-lanlan de jia (File 9560701)
I turn:around look-le look this-CL worn out DE home
I turned around and took a brief look at this run-down home.
The activities denoted in the above sentences are bounded respectively by a temporal NP (11a), a
quantity NP (11b) and a verb reduplicant (11c). It is clear that the aspect marker -le does not provide
any endpoint information, rather it only indicates the occurrence or realisation of a situation.
Because their inherent temporal boundary can be easily overridden when they shift from the singleevent
reading to the multiple-event reading, semelfactives pattern with activities. But semelfactives
differ from activities in that they may have the feature of [+bounded] even without an extra delimiting
mechanism. Therefore we predict that semelfactives can take the perfective -le more freely. This
prediction is supported by our data. Of the 26 occurrences of semelfactives taking -le, 16 are bounded
by extra delimiting mechanisms, with a ratio of 1.6:1, lower than the ratio for activities 3.04:1. Our
observations on the behavior of semelfactives also run against Yang (1995:118), who assumes that
“delimiting mechanisms have to be employed to provide specific closed readings out of
semelfactives.” Here is a corpus example of semelfactives without an extra delimiting device:
(12) Fu Yiwei de xiao guzi da-le Chen Hua (File 9559301)
Fu Yiwei de younger sister-in law beat-le Chen Hua
Fu Yiwei's younger sister-in-law beat Chen Hua.
When a semelfactive needs to be bounded, the same three delimiting devices also apply, as shown in
the following examples:
(13a) (tamen) da-le ni ji-tian? (File 9556901)
they beat-le you how many-days (Temporal NP)
For how many days did they beat you?
(13b) Yang Qinxian zhui-shang-le ta, ju dao lian chi-le liu-xia (File 9559701)
Yang Qinxian chase-up-le him raise knife successively stab-le six-CL (Quantity NP)
Yang Qinxian caught up with him and stabbed him six times with his knife.
(13c) laoren xiao-zhe dou-le dou shou (File 9560501)
old man smile-DUR shake-le shake hand (Reduplicant)
The old man shook his hand with a smile.
While the interaction of the perfective -le with all other situation types is an issue that has aroused hot
debate, there is an unanimous agreement that accomplishments and achievements can take -le without
any trouble (e.g., Smith, 1997; Pan, 1998; Yang, 1995; Li, 1999). Accomplishments and
achievements are both telic situations, this means that they have both spatial final endpoint and
temporal boundary even without the help of an extra delimiting mechanism. As such, these two
situation types interact with the perfective -le most naturally. From Table 4 above, we see that
accomplishments and achievements combined account for more than 80% of the total number of
situations taking the perfective -le found in our corpus data. This furnishes empirical evidence in
favour of our assumption that the perfective -le is sensitive to endpoint, but the sensitivity is merely a
matter of degree. In the following examples, situations in (14) are accomplishments and those in (14)
are achievements.
(14a) women you kaifa-le yixilie xin chanpin (File 9561401)
we also develop-le a:series:of new product
We also developed a series of new products.
(14b) qunian shiyue, Yang Bingming xie-le liang-feng xin (File 9560401)
last year October Yang Bingming write-le two-CL letter
Last October, Yang Bingming wrote two letters.
(15a) (tamen) di'er tian shangwu shi dian jiu dida-le mudidi (File 9558001)
they 2nd day morning 10 o'clock already reach-le destination
634
They arrived at their destination at 10 o'clock the next morning.
(15b) wo haishi kan-chu-le pozhan (File 9557301)
wo still see-out-le weak:point
But I still spotted his weakness.
It should be noted that although accomplishments and achievements have both spatial and temporal
endpoints, these endpoints are either encoded in basic or derived verbs themselves (achievements) or
provided by their arguments or adjuncts (accomplishments). In other words, -le interacting with these
two situation types only present them as an unanalysable whole. As with all other situation types, -le
does not provide any endpoint.
Summing up, it is clear that (1) the perfective -le interacts with all situation types in Chinese; (2)
there is a strong tendency for -le to co-occur with spatially or temporally bounded situations; (3) as a
perfective aspect marker, -le only focuses on the totality of a situation but does not provide any
endpoint.
4. Conclusion
In this study, we have cleared away some confusion over the perfective -le with empirical evidence
from a Chinese corpus. Based on the discussions above, our answers to the three questions raised at the
beginning are clear enough. First, as a perfective aspect marker, -le is different from the COS le. Their
differences in respect of syntactic distribution, semantic function, etymological source, and productivity
in the natural language all evidence that this is an unarguable linguistic fact. Second, the perfective
viewpoint marked by -le can presents a situation either as completed or as terminated. The perfective -
le only gathers a situation as a whole but does not provide any endpoint, so the closure type depends
upon situation types. That is, telic situations are presented as completed whereas atelic situations are
presented as terminated. Third, the perfective -le can interact with all situation types, but it
demonstrates a strong tendency to co-occur with spatially or temporally situations.
References
Brinton L 1988 The development of English aspectual system. Cambridge, Cambridge University
Press.
Bybee J 1993 The evolution of grammar: tense, aspect and modality in the languages of the world.
Chicago, University of Chicago Press.
Chao Y 1968 A grammar of spoken Chinese. Berkeley, University of California Press.
Christensen, M 1994 Variation in spoken and written Mandarin narrative discourse. Unpublished
PhD thesis, Ohio State University.
Chu C 1976 Some semantic aspects of action verbs. In Lingua 40: 43-45
Dai Y 1997 Xiandai hanyu shiti xitong yanjiu (A study of aspect in modern Chinese). Hangzhou,
Zhejiang Educational Press.
Henne H, Rongen O, Hansen L 1977 A handbook on Chinese language structure. Oslo,
Universitetsforlaget.
Li C, Thompson S 1981 Mandarin Chinese. Berkeley, University of California Press.
Li M 1999 Negation in Chinese. Unpublished PhD thesis, Manchester University.
Lü S 1981 Xiandai hanyu babai ci (800 words in Modern Chinese). Beijing, Commercial Publishing
House.
Moens M 1987 Tense, aspect and temporal reference. Unpublished PhD thesis, Edinburgh University.
Norman J 1988 Chinese. Cambridge, Cambridge University Press.
Pan H 1993 Interaction between adverbial quantification and perfective aspect. In Stevan L,
Bloomington (eds) Indiana University Linguistic Club Publications. 1993:188-204.
Pan H 1998 Adverbs of quantification and perfective aspects in Mandarin Chinese.
http://ctlhpan.cityu.edu.hk/haihuapan/pan/pan-publication.htm
Smith C 1988 Event Types in Mandarin. In Chan M, Ernst T (eds.) Proceedings of the 3rd Ohio State
University Conference on Chinese Linguistics. Ohio, pp215-243.
Smith C 1991 The parameter of aspect (1st Ed.). Kluwer Academic Publishers.
Smith C 1997 The parameter of aspect (2nd Ed.). Kluwer Academic Publishers.
Tai J 1984 Verbs and times in Chinese: Vendler's four categories. In Papers from Parasession on
Lexical Semantics (CLS) 27-28: 289-296.
635
Teng S 1986 Hanyu dongci de shijian jiegou (The temporal structure of Chinese verbs). In
Proceedings of 1st International Symposium on Chinese Teaching. Beijing, Beijing Languages
Institute Press.
Tiee H 1986 A Reference grammar of Chinese sentences. Tucson, the University of Arizona Press.
Verkuyl H 1972 On the compositional natural of aspects. Dordrecht-Holland, D. Reidel.
Verkuyl H 1993 A theory of aspectuality. Cambridge, Cambridge University Press.
Vendler Z 1967 Linguistics in Philosophy. Cornell, Cornell University Press.
Yang, S 1995 The aspectual system in Mandarin Chinese. Unpublished PhD thesis, Victoria
University.
Zhang L 1995 A Contrastive study of aspectuality in German, English & Chinese. Peter Lang.
Zhu D 1981 Yufa jiangyi (Teacher's guide to Grammar). Beijing, Commercial Publishing House.

A Corpus-based Contrastive Analysis of Spoken and Written Learner Corpora: 
The Case of Japanese-speaking Learners of English 
Mariko Abe (Sophia University) 
1. Introduction 
The purpose of this research is to investigate the variability of interlanguage that has been claimed in previous study by means of a corpus-based quantitative analysis. It aims to observe the style shifting of various grammatical features and word formation errors by tagging errors in 297 learners’ data. Various studies have been undertaken to describe and explain the process of second language (L2) acquisition. Tarone (1983), for example, claims that interlanguage capability continuum of learners diverges with respect to the degree of attention to language form, so-called ‘careful’ style to ‘vernacular’ style. Additionally Tarone (1985) showed that learner's performance varies depending on the dissimilarity of task, whether a written grammar test, oral interview and oral narrative. Ellis (1987) has confirmed the style shifting in the L2 learners’ use of the past tense. The variability of accuracy in past tense morphemes was observed in his study, when different amounts of planning time were set for a single narrative discourse task. Through his examination Ellis (1987) concluded that “so-called ‘natural’ order may not be a stable phenomenon” (p.1). In this study I compared the features of variability in L2 written and spoken corpus data based on the hypothesis that processing mode of learners affects their performance in L2. This research focussed on the analysis of the same task that used different production mode. The similarities and differences of learners’ errors in each mode were examined mainly from the perspective of grammatical and word formation features. In addition, English proficiency level of learners is another factor added to this study. Although this addition was only possible for the spoken corpus data, we can still observe the performance of learners at different proficiency levels. 
2. Corpus selection 
The spoken data were extracted from the Standard Speaking Test (SST) Corpus (Tono et al. 2002). This test has 9 different levels to assess the speaking proficiency of learning English. Spoken data came from 100 examinees belong to SST level 2 to 9. Although there are 5 stages in this speaking test, only one of the stages, single picture description stage, was used. A single picture is chosen from 5 different pictures, and the examinees were asked to describe it in 2 or 3 minutes. The written data were all collected by the author using a similar type of picture description task. The 197 Examinees were all university students who have been studying English for 6 years. In addition, 31 out of 197 examinees have also taken the simple version of SST, and these results were used to create a L2 spoken subcorpus. 
3. Data processing 
All of the hand-written manuscripts for the written corpus were transcribed on a word processor by the researcher, and two sets of data were error-tagged according to The TAO Speech Corpus of Japanese Learner English Error Tagging Manual Ver.1.0. (Isahara, Saiga, and Izumi 2002). This error-tag set is divided into three main levels. The first level consists of three criteria: Word Formation Errors (WF), Grammatical Errors (G), Lexico-Grammatical Errors (LG), Lexical Errors (LXC), and Others (O). The second level is divided into part of speech and the other categories. The final level is the category for the errors as follows: inflection (inf), number (num), Japanese English (je), genitive (gen), agreement (ag), form (f), tense (tns), voice (vo), finite/infinite (fin), negation (ng), question (qst), modal (mo), quantifier (qnt), inflection (inf), position (pst), countability (cnt), complement (cmp), dependent preposition (dprp), word redundancy (rdd), omission (oms), misordering (odr), ambiguity (amb), and unnaturalness (unl). Spelling errors and word division errors were not normalized but manually tagged and included in the section of word formation <wf_o_o> error. Since the total corpus size of written and spoken data was almost the same therefore no normalization has been done in this research. 
4. Data analysis: general error type 
In this section, spoken and written errors are sorted by general error types. The following Table1 shows that grammar is the category with the most errors for Japanese learners of English. Consequently, it might be meaningful to inspect the effect of mode of production on grammatical errors. According to Ellis (1987), previous studies have not mainly investigated the interlanguage variability from the aspect of grammatical structure. 
1
Table 1: Frequency of general error types 
WR 
Freq. % SP 
Freq.% Grammatical errors 1,136 50.02 473 58.40 Others 421 18.54 18522.84 Lexical errors 291 12.81 11113.70 Lexico-grammatical errors 53 2.33 334.07 Word formation errors 370 16.29 80.99 Total 2,271 810 
Figure 1: Frequency of general error types 
010203040506070GOLXCLGWF(%)SP WR
Except for the category of word formation error (WF), there is no difference in frequency rank order between the spoken and written corpora. Almost all the category indicate similar percentage in overall errors, however, interestingly spoken data have higher error frequency than that of written, excluding the category of word formation. Since the criteria of G (Grammatical errors), LG (Lexico-grammatical error) and WF (Word formation) are subcategorised in part of speech, further examination will be provided in following sections. 
5. Data analysis: part of speech 
When we sort each category by part of speech, another interesting feature can be observed. Articles occupy the highest error rate in both production modes, followed by verbs, nouns, pronouns, adjectives, adverbs, and prepositions. Error frequency rate as well as rank order is almost identical in both modes, whereas spoken data have higher density of error in noun and pronoun. In addition to the low frequency of preposition, the error rate of adjective, and adverb is low. This may not mean that the grammatical rules associated with these parts of speech are mastered, but it also indicates that learners are unconsciously avoiding these rules. Therefore, considering this rank order, learners may underuse nouns and overuse pronouns, since according to Granger & Rayson (1998) the rank order of word category in non-native data is as follows: nouns, verbs, prepositions, articles, adjectives, conjunctions, adverbs, determiners and pronouns. 
The following Table2 specifies the error distribution pattern in terms of part of speech, but it does not illustrate the accuracy rate. The analysis does not depend on the accuracy rate of learners, but on the frequency rate of the learners’ error, so that we cannot generalise the degree of learner's avoidance and acquisition of certain grammar points. However, we can at least compare the error rate between that of written and spoken mode in various grammar categories of each part of speech. From this standpoint, detailed examinations are presented in the subsequent sections through the subcategories of nouns (N) and verbs (V) respectively. 
2
Table 2: Error distribution pattern in part of speech 
WR 
Freq. % SP 
Freq. % AT 822 36.2299 36.91 V 312 13.74141 17.41 N 41 1.8151 6.30 PN 21 0.9215 1.85 AJ 1 0.044 0.49 AV 0 -1 0.12 PRP 0 -1 0.12 
AT=article; V=verb; N=noun; PN=pronoun; AJ=adjective; AV=adverb; PRP=preposition 
Figure 2: Error distribution pattern in part of speech 
0510152025303540ATVNPNAJAVPRP(%)WRSP 
5.1. Noun 
Singular common nouns are most frequently used in both spoken and written mode by native speakers of English (Leech et al. 2001), but this rule cannot be applied in this study. There are 5 subcategories in the criteria of noun and its error rate in different production modes is shown in Table3. In this section we will concentrate on the most striking errors, and on the items that have a striking dissimilarity between modes. 
Table 3: Error frequency rate of subcategories in noun 
WR 
Freq.% SP 
Freq. % g_n_num 2765.853772.55lg_n_cnt 512.2059.80wf_n_inf 0-59.80Countability (32)(78.05)(47)(92.16)g_n_gen 921.9523.92lg_n_dprp 0-23.92Noun total 41 51 
3
Figure 3: Error frequency rate of subcategories in noun 
01020304050607080g_n_numlg_n_cntwf_n_infg_n_genlg_n_dprp(%)WRSP 
5.1.1. Countability 
The following tags, <g_n_num> (many *book), <lg_n_cnt> (listening to a *music) and <wf_n_inf> (*childs) can be included under the category of countability. When Tarone (1985) examined the accuracy rate of plural markers ‘s', only form shift of morpheme was tested, and no variability was observed due to the different task. In this section, I would like to examine the error rate of the plural marker. Before comparing the error rate of <g_n_num> between written and spoken mode, errors such as “one of the *girl”, “I like *dog” should be excluded so that I only count the error rate of plural marker. The result is that the error frequency for writing mode is 14 out of 27 (51.85%) and that of speaking mode is 18 out of 37 (48.65%). There seems to be no significant variability in two different modes as might be expected. However, there is one prominent finding when we sort the error of spoken mode in learners’ proficiency level as shown in Table4. 
Table 4: Error frequency in different SST level (SP) 
SST 23456789g_n_num 0436995137Plural marker (0)(0)(0)(4)(5)(3)(5)(1)(18)Other (0)(4)(3)(2)(4)(6)(0)(0)(19)lg_n_cnt 011002105wf_n_inf 002210005g_n_gen 010001002lg_n_dprp 000002002Total 066810146151 
Figure 4: Error frequency in different SST level (SP) 
012345678910g_n_numlg_n_cntwf_n_infg_n_genlg_n_dprpSST2SST3SST4SST5SST6SST7SST8SST9 
4
As can be seen from the dispersion of the frequency in Table4, a morpheme change of plural marker is an error that often occurs in the intermediate levels but not in the novice level, whereas other grammar rules concerning countability occurs more often in the novice level than the intermediate level. As a result, we can hypothesise that when the necessity of attention to the form and grammar is low, intermediate learners tend to make errors; and when the necessity of attention is high they can avoid making errors. In addition to this, when the necessity of attention is low, because the rule is simple, novice learners make fewer errors; and when the necessity is high, because the rule is complicated, they have a tendency to make errors. By extracting Krashen's monitor model Ellis (1987) explains that easy rules can be monitored consciously causing the differences in style shifting. But this leads to another conclusion that intermediate learners may have an inclination to make much more errors over easily-learned rules than well-acquired rules and novice learners’ error rate increases as the load of attention to the rules increases. 
5.1.2. Variability in easily learned rules 
Regarding the error over gender <g_n_gen> (*woman hair is black) and inflection <wf_n_inf> (*childs), both has variability between the modes. Also both can be categorised as easily learned rules, nevertheless unexpectedly, these errors have the opposite error frequency rate in production modes. While there are only 2 examples in spoken corpus, larger corpus with adequate learners’ proficiency level information is needed to understand the cause of these differences. 
5.2. Verb 
The subcategories for verbs and their corresponding error rates in different production modes are shown in Table5. In this section we will mainly focus on the most striking errors, and on the items that have dissimilarity between modes. 
Table 5: Error frequency rate of subcategories in verb 
WR 
Freq. % SP 
Freq. % g_v_agr 16151.60 7653.90 g_v_fml 4514.42 96.38 g_v_tns 4113.14 2215.60 g_v_vo 41.28 21.42 g_v_fin 30.96 74.96 g_v_ng 10.32 0-g_v_qst 10.32 0-g_v_mo 0-0-lg_v_dprp 4414.10 1510.64 lg_v_cmp 41.28 96.38 wf_v_inf 82.56 10.71 Total 312141 
Figure 5: Error frequency rate of subcategories in verb 
0102030405060g_v_agrg_v_fmlg_v_tnsg_v_vog_v_fing_v_ngg_v_qstlg_v_dprplg_v_cmpwf_v_inf(%)WRSP 
5
5.2.1. Agreement <g_v_agr> (there *are the lady) (cat *sleep in her bed) 
In Tarone (1985) the accuracy level of third singular verb correction was examined, and variability according to tasks was also investigated. In this study the error frequency rate in different production modes does not diverge markedly. However, the error tag-set used in this research involves not only the third singular verb agreement errors but also every type of verb agreement errors. I excluded the agreement error for be-verbs and modal verbs from the data. The result is that the error frequency for writing mode is 95 out of 161 (59.00%) and speaking mode is 48 out of 76 (63.16%). Still a significant difference cannot be seen according to the production mode, but we have to remember the fact that Arabic learners of English were also included in Tarone's study. When we only focus on Japanese learners of English in her study, there is no apparent accuracy rate difference on different tasks. Although variability was found in neither different tasks nor modes, there is a striking result on error frequency in different SST level (see Tabel6). 
Table 6: Error frequency in different SST level (SP) 
SST 23456789G_v_agr 3281615461376person sing. verb (2)(21)(10)(12)(0)(3)(0)(0)(48)Be & modal verb (1)(7)(6)(3)(4)(3)(1)(3)(28)G_v_fml 102212019G_v_tns 1557112022G_v_fin 012301007Lg_v_dprp 0430341015Lg_v_cmp 030212109Total 5412829101654138 
Figure 6: Error frequency in different SST level (SP) 
051015202530g_v_agrg_v_fmlg_v_tnsg_v_finlg_v_dprplg_v_cmpSST2SST3SST4SST5SST6SST7SST8SST9 
Novice learners overwhelmingly tend to make errors over whole verb agreement, compared with higher-level learners; but when we focus on the third singular verbs and the other verbs separately, another distribution is found. If we follow the hypothesis concluded in section 5.1.1., ‘error rate is in inverse proportion to the degree of attention to the rule in intermediate learners, and error rate is in direct proportion to the degree of attention in novice learners,’ the agreement rule for third person singular verbs may be much more difficult than for be-verbs and modal verbs for Japanese learners of English. 
5.2.2. Form <g_v_fml> (one boy is *listen to music and drinking something) 
Interestingly, form error is much more frequently seen in the written mode, when learners can pay more attention to the form than spoken mode. Regarding the distribution of proficiency level in spoken 
6
3rd 
data, errors can be observed in almost every level in almost same number. When I sorted the errors into present particle errors and the others, there are 42 out of 45 (93.33%) in written mode and 5 out of 9 (55.56%) in spoken mode. In spoken data there is one example in each SST proficiency level 2, 4, 5, 7, and 9. How can we explain the fact that most of the errors are occur with present particle error, which is presumably easy learned, and with written mode? We need more adequate data and precise study to draw a conclusion. However, if learners do not have a high accuracy in simple mechanical verb form changing, we can assume that it is not fully acquired by the learners so that they need ample practice in verb form changing. 
5.2.3. Complement <lg_v_cmp> (this person is teaching *how to them) 
While Tarone (1985) found the variability in the aspect of third person singular direct object pronoun (D.O. Pro ’It') in different task, it was difficult to discover the variability in different mode. This is because not all the examples in both modes cannot be categorised as errors in D.O. Pro ’It'. In the written mode 3 out of 4 (75%) examples are of this type: “A woman wears *to it”, “I will characterize *of her”, and “A girl crosses *with her legs.” Whereas in spoken mode only one example out of 9 (11.11%) can be found: “I don't know how to say *in English.” Moreover, the error related with <lg_v_cmp> does not seem to be connected to learners’ level. Considering the fact that there are not sufficient examples in both written and spoken mode, complement errors over the verb may have deeper connection with the misunderstanding of usage of verbs as far as Japanese learners are concerned. Consequently, analysis must be more focussed on the verbs themselves. 
6. Data analysis: others 
In previous sections we were able to observe the variability and consistency in some of the categories. In this section, we examine the general tendency of errors that have not been mentioned so far. 
Table 7: The size of four corpora 
WR SP WR-subSP-subPicture 1 1-5 11SST level --- 2-9 2-62-6File 197 100 3128Tokens 17,863 17,222 3,2213,908Types 951 1,314 422451Type/Token/Ratio 5.32 7.63 13.111.54Ave. Word Length 4.56 3.37 4.513.31Sentences 1,507 986 265252Sent.length 10.5 16.51 10.4214.71sd. Sent. Length 5.45 14.7 6.2613.66 
7
Table 8: Rank frequency of error in each corpus 
WR SP WR-sub SP-sub 17,863 17,222 3,221 3,908 Rank Freq.% Freq.% Freq. % Freq. % 1 g_at 82236.20 g_at 29936.91 g_at 13034.30 g_at 95 46.12 2 wf_o_o 36215.94 lxc 11113.70 lxc 5213.72 g_v_agr 29 14.08 3 lxc 29112.81 o_oms 10212.59 wf_o_o 5514.51 o_oms 28 13.59 4 o_oms 27812.24 g_v_agr 769.38 o_oms 4411.61 lxc 22 10.68 5 g_v_agr 1617.09 o_rdd 516.30 g_v_agr 3910.29 g_v_tns 5 2.43 6 o_odr 703.08 g_n_num 374.57 g_v_tns 123.17 g_pn 4 1.94 7 g_v_fml 451.98 g_v_tns 222.72 lg_v_dprp112.90 lg_v_dprp 4 1.94 8 lg_v_dprp 441.94 g_pn 151.85 g_v_fml 71.85 g_n_num 3 1.46 9 o_amb 421.85 lg_v_dprp 151.85 o_amb 61.58 g_v_fin 3 1.46 10 g_v_tns 411.81 o_amb 131.60 o_odr 61.58 o_odr 3 1.46 11 o_rdd 311.37 o_odr 101.23 g_n_num41.06 lg_v_cmp 2 0.97 12 g_n_num 271.19 g_v_fml 91.11 g_pn 30.79 g_av_pst 1 0.49 13 g_pn 210.92 lg_v_cmp 91.11 o_rdd 30.79 g_v_fml 1 0.49 14 g_n_gen 90.40 o_unl 91.11 g_n_gen 10.26 lg_aj_dprp 1 0.49 15 wf_v_inf 80.35 g_v_fin 70.86 g_v_ng 10.26 o_amb 1 0.49 16 lg_n_cnt 50.22 wf_n_inf 50.62 g_v_qst 10.26 o_unl 1 0.49 17 g_v_vo 40.18 lg_n_cnt 50.62 g_v_vo 10.26 wf_o_o 1 0.49 18 lg_v_cmp 40.18 g_av_pst 40.49 lg_n_cnt 10.26 wf_v_inf 1 0.49 19 g_v_fin 30.13 g_n_gen 20.25 lg_v_cmp10.26 lg_n_cnt 1 0.49 20 g_v_ng 10.04 g_v_vo 20.25 wf_v_inf10.26 21 g_v_qst 10.04 lg_n_dprp 20.25 22 g_aj_qnt 10.04 wf_v_inf 10.12 23 wf_o_je 10.12 24 wf_o_o 10.12 25 lg_aj_dprp 10.12 26 lg_prp_cmp10.12 Total error 2271 810 379 206 
Errors due to the omission of one or more necessary words <o_oms> have relatively high frequency, but there is no significant difference in the error rate for the two modes. Regarding errors over word order <o_odr>, only the written corpus has a high frequency, but again there is not so much difference in error rate. This high error frequency in written corpus can be explained by novice low-level learners’ consistent errors patterns. It is caused by the total misunderstanding of English structure such as “*Open door” (“door is open”), and this was counted as an error related word order. The ranking of error concerning ambiguity <o_amb> is almost equal but except for the subcorpus of speaking data. Errors that were difficult to categorise in any of the criteria were included in this group. Another criterion is errors related to redundancy <o_rdd>, and the error rate is dissimilar in each corpus. The most striking finding is that there is no error of this type in spoken subcorpus, while error rate in spoken corpus is fairy high. One possible explanation for this difference is that the proficiency level of learners in the spoken corpus is higher than that of the sub-corpus, consisting of learners from SST level 2 to 6. Therefore, we can presume that higher-level learners have a tendency to make redundancy errors. Lastly, the study did not show variability in the category of “other”, but the comparison between different proficiency level corpora will be useful in further studies. 
7. Conclusion 
Through the detailed analysis on subcategories of nouns and verbs, we can observe the error rate difference in error over noun gender and verb form. Another finding that is noteworthy is that the error category, which has a high error rate, also has a large distribution among the learners’ proficiency level, as can be seen from the example of errors related to countability and agreement. Also we were 
8
able to acknowledge that the error rate is in inverse proportion to the degree of attention to rules for intermediate learners, and error rate is in direct proportion to the degree of attention to rules for novice learners. Granger and Rayson (1998) have shown in their research the resemblances of written and spoken production of learners, and they conclude that communicative approach is one of the factors that have an influence on “speech-like nature of learner writing” (p.130). Since the rise of this ELT methodology we may come to emphasise fluency but not accuracy, however, this study suggests that it is also necessary to take notice on learners’ errors through instruction and feedback in the classroom. 
Since not all the examinees of written data were able to take the SST test in this study, it was unfortunately impossible to investigate the correlation of written and spoken modes in terms of learner's proficiency level. More detailed data on learner's proficiency level and much larger corpora will be needed in future studies. Another drawback was that all the analysis comprised of the error rate, but not of the accuracy rate. Much more impartial examination could be done, if it were possible to determine whether learners are avoiding the certain usage or not. The last point is that subcategorised tag-sets that accord with learners’ error tendency will be necessary for further study. Tag-sets for relative pronoun and conjunction, for example, were eliminated in this study. By analysing the similarities and differences between the two modes of learner corpora, I have arrived to identify the features of interlanguage variability in a more objective way, which will shed some light on the nature of the interlanguage development and possible implications for EFL pedagogy. 
References 
Ellis R 1987 Interlanguage variability in narrative discourse: Style-shifting in the use of the past tense. Studies in Second Language Acquisition 9: 1-20. 
Granger S, Rayson P 1998 Automatic profiling of learner texts. In Granger S (ed), Learner English on computer. London, Longman, pp119-131. 
Isahara H, Saiga T, and Izumi E 2002 The TAO Speech Corpus of Japanese Learner English Error Tagging Manual Ver.1.0. 
Leech G, Rayson P, Wilson A 2001 Word Frequencies in Written and Spoken Language. London, Longman. 
Tarone E 1983 On the Variability of Interlanguage Systems. Applied linguistics 4(2):143-163. 
Tarone E 1985 Variability in interlanguage use: a study of style-shifting in morphology and syntax. Language learning 35: 373-404. 
Tono Y, Kaneko T, Isahara H, Saiga T, Izumi E 2002 The Standard Speaking Test Corpus. Studies in Lexicography 11(2): 7-18.339 
9

Methodology and steps towards the construction of EPEC, a corpus of written Basque tagged at morphological and syntactic levels for the automatic processing 
Aduriz I.*, Aranzabe M.J., Arriola J.M., Atutxa A., 
Díaz de Ilarraza A., Ezeiza N., Gojenola K., Oronoz M., Soroa A., & Urizar R. 
Department of Computer Languages and Systems 
Computer Science Faculty 
University of the Basque Country 
P.O. box 649, E-20080 Donostia 
jibaregj@si.ehu.es *Department of Linguistics 
Faculty of Philology 
University of Barcelona. 
E-08007 
EPEC, the Reference Corpus for the Processing of Basque, is a corpus of standard written Basque that has been manually tagged at different levels (morphology, surface syntax, phrases) and is currently being hand tagged at deep syntax level. It is aimed to be a "reference" corpus for the development and improvement of several NLP tools for Basque. 
Although small (50,000 words), EPEC is a strategic resource for the processing of Basque and has already been used for the development and improvement of some tools. Half of this collection was obtained from the Statistical Corpus of 20th Century Basque, a reference corpus of Basque including 4.7 million word-forms. The other half was extracted from Euskaldunon Egunkaria, the only daily newspaper written entirely in standard Basque. 
When defining a general framework for the automatic processing of agglutinative languages like Basque, a morphological analyser of words is an indispensable basic tool. However, previous to the completion of the morphological analyser MORFEUS, the design of the tagset and a lexical database had to be accomplished. 
Choosing an appropriate tagset is a crucial task since the usefulness and ambiguity-rate of the analyser depend on it. For the morphosyntactic treatment of Basque texts, the tag system we developed is a four level system, ranging from the simplest part-of-speech tagging scheme up to the full morphosyntactic information. In addition to these four levels, further tags are added to mark verb chains, noun phrases, and postpositional phrases. Nowadays, we are involved in the syntactic tagging of the corpus, following the Dependency Structure-based Scheme in order to build a treebank. 
The Lexical Database for Basque (EDBL) is a general-purpose lexical database used in several text-processing tools for Basque. This large repository of lexical knowledge is the basis in many different NLP tasks, and provides lexical information for several language tools including, obviously, the morphological analyser. At present, it consists of nearly 80,000 entries. 
Morfeus is a robust morphological analyser for Basque. It is a basic tool for current and future work on NLP of Basque. It is based on the two-level formalism proposed by Koskenniemi (1983). Morfeus consists of three main modules: (i) the standard analyser, capable of analysing and generating standard word-forms, (ii) the analyser of linguistic variants (dialect uses and competence errors), and (iii) the guesser or analyser of words without lemmas in the lexicon. 
The manual disambiguation of the corpus was performed on the output of Morfeus. Thus, the whole corpus was morphosyntactically analysed giving to each word-form every possible analysis, without taking into account the context in which it appeared. Once each word-form in the corpus was analysed, we carried out the manual disambiguation process. Two linguists marked independently the correct syntactic tag to each word in the corpus, applying the “double blind” method described in Voutilainen & Järvinen (1995). Both linguists' answers were compared and, when differences occurred, they agreed a single tag. 
This manually disambiguated corpus was used both to improve a Constraint Grammar disambiguator and to develop a stochastic tagger. 
After disambiguating the morphological tags in the corpus, the next step was to assign the corresponding syntactic tag to each word-form. Syntactic function tags follow the philosophy of the Constraint Grammar (CG). By adopting the CG formalism, we express the syntactic functions of words and the interdependencies that exist among them rather than deep structural relations. So, the syntactic tags at this level refer to shallow syntactic functions, i.e. they may provide information about the surface structure of verb chains, noun phrases, or postpositional phrases. 
Once each word-form in the corpus was given at least one syntactic tag, we carried out the manual disambiguation process again. The method used was similar to the one used for the morphological disambiguation in the previous step. 
At this stage we have the corpus manually tagged with surface syntactic tags following the CG syntax. No phrase units are marked yet, although based on this representation, the identification of various kinds of phrase units, such as verb chains, noun phrases, and postpositional phrases is reasonably straightforward. 
In order to detect verb, noun, and postpositional phrases, we use different function tags as well as some particles (such as negative or modal particles). At present, a linguist is checking the tags that the first set of mapping rules marked up in the corpus. Whenever necessary, she adds, removes, or changes the tags automatically assigned. Once this work is finished, the first set of mapping rules developed will be tested on the corpus and the results will be used to improve the rules iteratively as well as to develop new ones. 
Nowadays, we are also involved in the syntactic tagging of the corpus following the Dependency Structure-based Scheme to tag syntactically the corpus in order to build a treebank. 
10
During the last three years, a great effort has been done in our research group (Artola et al., 2002) to integrate the NLP tools for Basque described in previous sections. Due to the complexity of the information to be exchanged among the tools, Feature Structures (FSs) are used to represent it. Feature structures are coded following the TEI's DTD for FSs, and Feature Structure Definition descriptions (FSD) have been thoroughly defined. The documents used as input and output of the different tools, contain TEI-P3-conformant feature structures (FS) coded in SGML. 
In the future, we also intend to extend the corpus annotation to word sense tagging and anaphora annotation. 
11

The mood of the (financial) markets: In a corpus of words and of pictures1 
Khurshid Ahmad, Pensiri Manomaisupat, David Cheng, Tugba Taskaya, Saif Ahmad, Lee Gillam and Andrew Hippisley 
Department of Computing, University of Surrey, Guildford, Surrey. 
Methods of corpus linguistics are typically used to study language either synchronically or diachronically. Much of the work currently being carried out under the rubric of information extraction benefits directly or indirectly from work in corpus linguistics: the extraction of the so-called named entities, template elements, template relations, and scenario templates, all relating to meaning bearing units of language within a text, relies on various statistical tests that have to be carried out over a corpus of texts. Some brave folk attempt to carry out these tests over a corpus of multilingual texts. One of the important developments in information extraction relates to event modelling. Here linguists look at the lexico-grammatical properties of texts and attempt to derive information about events that are supposed to be reported in the texts. We have found that event modelling requires a good understanding of the modes used in communicating the events, including natural language, graphs and images. A case study of financial market movement, where a corpus of news wires and graphical information, or a financial time series, were correlated, is described. These are preliminary results of an EU 5th Framework Project –GIDA (No. IST 2000-31123). 
News streams provided by organisations like Reuters or Bloomberg comprise a range of keywords and indexical names that may change from one news item to the next; an event modeller will need to filter the news from such a diverse information resource. Specialist information providers deliver not only news texts but also supply, for example, time series of changes in value of stocks, shares, currencies, bonds and other financial instruments. We have a narrower focus than other authors in information extraction (see for example Maybury et al, 1995) in that we are looking for changes in key financial instruments that are reported in financial news-wires. The news coverage of these instruments is of two types: first, there is a daily report about changes in the value (numerical) of the instruments for instance, one can see time series comprising historic data about the changes in values of currencies; second, the manner in which the value of the instruments changes depends on the reports relating, directly or indirectly, to the instrument. The reports, for example, about war or economic uplift/downturn, affect the value of the instruments. Some authors claim that there is a correlation between ‘good’ or ‘bad’ news relating to the instrument and its potential numerical value. 
Our work focuses on primary movements within a market which lasts from few months to many years and represents the broad trend within a market. We report on some initial work that attempts to changes in an index, FTSE100, with changes in ‘market sentiment’ as expressed in news reports about the UK economy specifically and reports about the Wall Street indices. The later has substantial influence on the UK economy. Financial analysts use sophisticated political, economic and psychological analysis to determine the reaction of market operatives and to predict the possible trading decisions of the operatives. Reports related to the sentiment use a range of metaphors to express the state of a market and its possible movements. Francis Knowles has written about the use of health metaphors used in the financial news reports: markets are full of vigour and are strong or the markets are anaemic or are weak (1996); most newspapers also use animal metaphors – there are bull markets and bear markets, the former refer to expansion, and indirectly to fertility, and the later to shy, retiring and grizzly behaviour much like that reported about bears in popular press and in literature for children. Indeed, there are fairly literal words that express the sentiment, as reported in the news wires, about the markets: financial instruments rise, fall, markets boom, go bust, and there are gains, losses within the markets, economies slowdown, suffer downturns, whole industry sectors maybe hardpressed. We created a corpus of 1,539 English financial texts from one source (Reuters) on the World Wide Web, published during a 3-month period (Oct 2001-January 2002) comprising over 310,000 tokens. The corpus comprised a blend of both short news stories and financial reports. Most of the news is business news from Britain with thirty percent of the news is from Europe and from the United Stages. We automatically extract the sentiment words and key terms from the text corpus diachronically. The correlation between the metaphorical words and the FTSE looks good at first sight: 
1 Based on papers presented at two workshops: LREC Event Modelling Workshop (Spain 2002) and Financial News Analysis Workshop, 11th International Terminology and Knowledge Engineering Congress (France 2002). 
12
00.20.40.60.811.21256789121314151619202122232627282930DateRatioGood wordsBad wordsFTSE100 
Recent work has looked beyond the frequency distributions of positive and negative sentiment keywords to the actual relevance of a keyword's contribution to the analysis through its grammatical properties. Specifically we look at the tense / aspect features of verbal keywords to determine their relevance to the immediate situation. Some of the categories and their relevance measures are shown below: 
Greater relevance Less relevance Present Continuous 
e.g. stocks are rising Present Simple 
e.g. stocks rise Perfect 
e.g. stocks have risen Past 
e.g. stocks rose 
The claim is that a positive verb such as rise which is marked as Present Continuous will suggest greater immediacy and therefore greater relevance as an indicator of current market sentiment. In this paper we will describe our attempts to look at a much larger corpus of texts (c. 1 million token financial news stream) and perform information extraction by making greater use of the linguistic properties of the sentimental tokens thus identified. 
Maybury (1995) Generating Summaries from Event Data. Information Processing and Management. 31(5) 733-751. 
Knowles, F. (1996) Lexicographical Aspects of Health Metaphors in Financial Texts. In (Eds.)Martin Gellerstam et al. Euralex'96 Proceedings (Part II). Göteborg, Sweden: Göteborg University, pp 789-796. 
13

The Lacio-Web Project: overview and issues 
in Brazilian Portuguese corpora creation 
Sandra M. Aluísio*#, Gisele M. Pinheiro#, Marcelo Finger†, 
Maria das Graças V. Nunes*#, Stella E. O. Tagnin‡ 
*ICMC – DCCE, University of Sao Paulo, CP 668, 13560-970 Sao Carlos, SP, Brazil 
#Núcleo Interinstitucional de Lingüística Computacional (NILC), ICMC-USP, CP 668, 13560-970 Sao Carlos, SP, Brazil 
†IME – DCC, University of Sao Paulo, Rua do Matao, 05508-090 Sao Paulo – SP, Brazil 
‡FFLCH – DLM, University of Sao Paulo, Av. Prof. Luciano Gualberto, 403, 05508-900 - Sao Paulo – SP, Brazil 
sandra@icmc.usp.br, gisele@nilc.icmc.usp.br, mfinge@ime.usp.br, gracan@icmc.usp.br, seotagni@usp.br 
1. Introduction 
The pioneering balanced Brown Corpus launched in 1964, annotated reference corpora, such as Suzanne and the Penn Treebank and the balanced mega British National Corpus (BNC)1, to cite only a few, have helped both the development of English computational linguistic tools and English corpus linguistics. Portuguese, on the other hand, still requires a lot of work for building the basic resources to develop linguistic research based on and driven by corpus. Portuguese is the mother tongue of approximately 200 million people (Brazil, Portugal, Angola, Cabo Verde, Guiné Bissau, Mozambique and S. Tomé e Príncipe) and is the sixth most spoken language in the world today. The two language variants with the greatest number of users – European Portuguese (EP) and Brazilian Portuguese (BP) – differ in phonological, lexical, morphological and syntactical levels (Wittmann et al. 95) suggesting a real need to build corpora considering both of them. 
For European Portuguese (EP), the project AC/DC2 has compiled several corpora of non-literary (e.g. journalism) and literary texts (poetry, prose, plays) mainly from 16th to 19th centuries. These corpora are available both in raw format and annotated with lemma, POS and associated attributes, and syntactic tags, being the main purpose of this compilation to raise the quality of Portuguese NLP. Another example for EP is the Corpus de Referencia do Portugues Contemporâneo (CRPC)3, which has been under construction since 1988, and contains excerpts of several types of written discourse (literary, journalism, technical, scientific, didactic, economy, legal, parliamentary, etc.) and oral discourse from the main variants of Portuguese. Its main goal is to establish an on-line representative sample collection of general usage contemporary Portuguese accessible to anyone interested in engaging in theoretical and practical studies or applications. Both projects include texts from the Brazilian Portuguese and are valuable resources although there is a lot of work to be done with regard to text classification based on genre and text typology to better balance the variants included. On the other hand, Brazilian Portuguese (BP) corpora to date have mainly addressed spoken language; some written language corpora are only partially available. These corpora were used by specific projects, particularly for the production of dictionaries, and some are not publicly available due to copyright restrictions. As for their application, they have generally been used for specific linguistic studies such as sociolinguistic and phonetic-phonological research as well as historical linguistic studies (e.g NURC-RJ4 and NURC-SP5, PHPB6 and VARSUL7 projects); lexicography (e.g. the written corpus “Usos do Portugues” from the State University of Sao Paulo, at Araraquara, which gave birth to three dictionaries: a 
1 info.ox.ac.uk/bnc/ 
2 www.linguateca.pt/ 
3 www.clul.ul.pt/sectores/projecto_crpc.html 
4 www.letras.ufrj.br/nurc-rj/ 
5 www.fflch.usp.br/dlcv/nurc/ 
6 www.letras.ufrj.br/phpb-rj/ 
7 www.cce.ufsc.br/~varsul 
14
dictionary of frequency (Biderman, 2001), a dictionary of verbs (Borba, 2001) and one of contemporary Brazilian Portuguese usage (Borba, 2002); and a grammar of Brazilian Portuguese usage (Neves 2000)) and literary studies (e.g. the NUPILL project8, which made available classical Brazilian literary books for teaching and literary studies). There are also small specialized written language corpora compiled by researchers or research groups in order to assess the performance of NLP systems. Such corpora are generally not available either and their results are not reproducible in principle. 
The Lacio-Web project, a two-year project launched in early 2002, tries to fill this gap as it aims at compiling corpora which are freely accessible for both non-expert users interested in the Brazilian Portuguese language and expert users who pursue theoretical and practical linguistic studies and develop computational linguistics tools (e.g. taggers, parsers, sentence and word aligners, automatic term extraction tools, and automatic summarizers) and applications such as computer systems for natural language information retrieval, machine translation and grammar checking. Lacio-Web (LW) is being developed at the University of Sao Paulo (USP) under the auspices of the governmental agency Conselho Nacional de Desenvolvimento Científico e Tecnológico do Brasil (CNPq). 
The LW project comprises six corpora: 1) a reference corpus called Lacio-Ref; 2) Mac-Morpho9, a gold standard portion from Lacio-Ref, comprising 1,1 million words, which was manually-validated for morpho-syntactical tags; 3) an automatically-annotated portion of the Lacio-Ref with lemmas, POS and syntactic tags which are used by the parser Curupira developed at NILC10; 4) a deviation corpus composed of non-revised texts (Lacio-Dev); 5) and parallel and 6) comparable Portuguese-English corpora called, respectively, Par-C and Comp_C. The corpora will be available on the WWW for download. We will also develop a web-based interface for access to the corpora to meet several users´ needs. For this purpose we will consider the Project of Korpus 2000 (Andersen et al., 2002) as its corpus interface was designed with non-expert user needs in mind. 
It is worth mentioning that the design of the Lacio-Ref corpus and its text typology have been based on corpus linguistics principles (Sinclair and Ball, 1996) and on important corpora projects, e.g. American National Corpus (ANC) (Filmore et al. 1998; Ide & Macleod, 2001; Ide et al. 2002), BNC and Czech National Corpus (CNC)11. We have also tried to overcome some flaws in the typologies used in the written parts of the latter two (BNC and CNC) which would prevent us from broadening the set of potential users of the LW corpora. The construction of LW corpora builds upon the previous experience at NILC in the ad hoc compilation of a 35 million-token corpus named Corpus Nilc (CN). This paper details the corpora being created (Section 3), presents the rationale for LW corpora (Section 4) as, in Brazil, there is an urgent need for corpora (both annotated and raw) constructed according to corpus linguistics principles. It also compares and contrasts the development of both corpora (LW and CN) emphasizing the lessons learned in the process (Section 2). 
2. Lessons learned from developing and critically analyzing CN 
The CN corpus was built to support the development of a grammar checker for Brazilian Portuguese named ReGra (Martins et al., 1998). Specifically, CN was designed to inform linguistic studies for the development of ReGra and to provide data for performance testing. The construction of ReGra started in 1993 and it has been improved since then under the auspices of both a Brazilian governmental agency and a Brazilian software company. CN is an opportunist corpus built in an ad hoc manner: its text selection was carried out on demand and its text classification was based on a very particular typology suitable for testing the grammar checker. 
At the beginning of the LW Project, we counted on being able to feed a major part of the CN into the Lacio-Web, thus, reducing the cost of its construction. However, after a detailed analysis of the CN subcorpora we found major problems regarding: classification, the number of texts in certain subcorpora, sample size (the main criterion in CN was full texts), grouping and formatting, documentation and copyright. 
8 www.cce.ufsc.br/~nupill 
9 Details about its tagset, the annotation process, including the results of the inter-annotator agreement evaluation and linguistic problems faced in developing Mac-Morpho can be found at www.nilc.icmc.usp.br/nilc/projects/lacio-web.htm 
10 In a follow up project, we intend to do a manual revision of it. The resulting Treebank will serve, for example, to improve the parser itself and to train statistical parsers. Details about Curupira can be found at www.nilc.icmc.usp.br/nilc/tools/curupira.html 
11 ucnk.ff.cuni.cz/english/index.html 
15
These items will be detailed below together with an indication of the cost involved to correctly include the CN texts in LW. 
2.1 Text classification 
As CN was built on demand, its text classification became problematic. The texts were divided into three classes, driven by the purpose of the corpus, namely: a) corrected texts, b) uncorrected texts and c) semi-corrected texts, i.e. texts published in books or journals e.g., unrevised texts, and text revised by advisors e.g., respectively. Inside these classes, the texts were grouped in an ad hoc fashion, either by domain (or subject), or genre or textual type. This irregular text classification prevents a user from recovering all texts on “sports” because only some of them were classified according to domain. One reason for this is that the texts in CN were not prepared to be automatically retrieved by a user. On the other hand, as we intend to design powerful and customized search tools for LW´s users, the first task after defining LW corpus typology was to define its text typology which has a 4-orthogonal category typology to classify texts (genre, textual type, medium and subject) and information about authorship and the publication of texts. Another benefit a sound text typology (driven by linguistic or commonly accepted external criteria) brings along is the possibility to better control the insertion of material into a corpus in order to achieve better representativeness in corpus design. 
2.2 Number of texts 
Some CN subcorpora are under-represented, i.e. they consist of a small number of texts. For example, the “Technical and Scientific” subcorpus has only a few samples of theses and incomplete dissertations, most of them from the Computer Science domain. We can say that CN benefits terminology studies in Computer Science but it is not representative regarding other technical and scientific domains. On the other hand, the design of the LW corpora is based on a 4-orthogonal category typology to organize the texts and we will endeavour to complete each category in each corpus (see Section 4 for details on this procedure). This set-up will enable the development of several types of tools and linguistic researches. Automatic text categorization, for example, is a type of research which would benefit both the advance of the research area itself and the organization of the texts in a typology. Moreover, in the near future we will be able to perform linguistic analysis to evaluate the representativeness of our newborn corpora following Biber´s recommendation to proceed in a cyclical fashion of analysis, design, compilation and analysis again (Biber, 1993). 
2.3 Sample size 
Some text samples deviated from the standard followed by CN which was to include only full texts. For example, some only have a few chapters of a book, other have only excerpts from the beginning, middle and end part of a whole document. In the LW Project, we are following the criterion of inclusion of sequential12 full texts to allow for research on text structure and summarization, for example. However, if copyright issues demand otherwise we will have this fact annotated in the text header. This may happen with textbooks, for example. 
2.4 Grouping and formatting 
Issues regarding ad hoc grouping such as, to group several small texts from a same class in just one text causes several problems for the compilation of the header. It would be difficult to import these groups of texts to LW since some information on them has been lost and we intend to edit a header for each and every text from the corpora. This grouping has occurred in encyclopedia entries and small articles from newspapers. Also, CN has not keep text formatting e.g. sentence and paragraph marks for many texts. As we consider this annotation important it would be costly to recover this information. 
12 The newspaper corpora in AC/DC Projects had to have their texts scrambled, which prevents several types of research of being conducted. 
16
2.5 Documentation and copyright 
Annotating a text with a header which provides internal and external information on the texts was not taken seriously in the compilation of CN. Some texts have the traditional information on authorship and publication details but nothing is said about its domain (subject) or genre and text type; others do not have any header at all. The LW will try to correct this flaw and will invest large amounts of effort to edit and encode a header following largely accepted standards (see Section 4). With regard to copyright, there was no effort to secure rights to include material in the CN as it was designed to support the development of an application. LW, however, has other design criteria as it will be freely available on the WWW. One of the most difficult problems in building a corpus is surely to get permission to include texts from copyright holders as there is a lot of correspondence involved before we manage to get copyright clearance. We are using similar contact and permission letters provided by BNC and ANC to include texts in LW corpora. Although we do not have a consortium of commercial members to feed our corpora we have managed to include relevant newspaper, magazines and books in our corpora. 
In summary, CN was of great value to the development of ReGra and with regard to LW, CN is ready to contribute with the legislation (110.571 tokens), journalism (25,167.436 tokens) and literary texts (1,761.373 tokens). 
3. The corpora of the Lacio-WEB Project 
A common concern in recent studies is to provide precise guidelines concerning the typology of a corpus. Atkins, Clear & Ostler (1992) discussed contrast-based parameters that allow identification of different types of corpora: synchronic vs. diachronic corpus; closed vs. open-ended corpus; full text vs. sample vs. monitor corpus; general vs. terminological; single vs. parallel-2 vs. parallel-3 vs. …, etc. Some criteria for classifying corpora have also been provided by Cathy Ball, in a tutorial about concordances and corpora13, on top of the parameters mentioned above: balanced vs. opportunist vs. pyramidal corpus, and plain vs. annotated corpus. A similar work on such parameters was carried out by Berber Sardinha (2000), who distinguished corpora according to their purpose: reference vs. study vs. training and testing. 
The corpora comprised by this project have a varied composition and can be classified according to the 8 parameters cited above. Regarding purpose, the Lacio-Ref corpus is a reference corpus; the Lacio-Dev – a deviation corpus composed of unrevised texts – was meant for training and testing applications, such as grammar checkers; and the Mac-Morpho corpus is closed and serves as a training and testing corpus for NLP tools, such as POS taggers. As far as design is concerned, the Mac-Morpho corpus and the automatically-annotated portion of Lacio-Ref are of annotated type, while the Lacio-Ref is a plain corpus. For the parameter single vs. parallel, the Par-C is of the parallel-2 type; and for the general vs. terminological parameter, the Comp_C is of the terminological type. 
All the corpora are synchronic, presenting BP language from 1900 onwards. Lacio-Dev, Par-C and Comp_C are opportunistic corpora. The specificities of each of these corpora are discussed below, together with their status in terms of completion and encoding. We distinguish two types of encoding: i) header and sentence markup and ii) annotation for linguistic phenomena such as lemmatization, POS and syntactic tags. Our header editor includes an XML compliant header, according to the international standard adopted in the Translational English Corpus (TEC)14 from University of Manchester's Institute of Science and Technology (UMIST). A tool named SPLITTER, developed at NILC, will be used to split the texts into sentences. 
1) Lacio-Ref Corpus: contains texts from various genres (e.g., literary and its subdivisions, factual, informative, scientific, law), textual types (e.g. article, manual, research project, letter, biography), subjects (e.g., politics, environment, life style, sports, arts, religion etc.); and medium of distribution (e.g., books, internet texts, cd-rom material, newspapers and magazines, etc.). The four-category typology cited above (genre, textual type, medium and subject) will be used to allow for specific searches on the corpus. Also, the texts may be searched by authorship and other publication details. 
13 www.georgetown.edu/faculty/ballc/corpora/tutorial.html 
14 www2.umist.ac.uk/ctis/research/TEC/tec_home_page.htm 
17
All pieces of text are authentic with identified sources (see Section 4) and are, preferentially, full texts. Tools will also be available to users for obtaining a statistical description of the texts, in terms of the text size (in number of words, pages or kbytes). Status: the corpus is being compiled (it already contains material contributed from the CN); the edition of the headers has not been started. 
2) Lacio-Dev Corpus: most texts (in a total of 516.840 tokens) will be imported from the CN which contains a subcorpus comprised of unrevised texts of varied subjects produced by undergraduate students and prospective students attempting to enter the University. This Corpus can be used to assess the performance of grammar checkers for BP, as there is a need to ensure that the checkers are able to detect linguistic inadequacies during the tests. Status: edition of the headers has not been started. 
3) Mac-Morpho Corpus: this 1.1 million-word Corpus is composed of a collection of randomly selected texts from several issues of Folha de Sao Paulo (1994)15, a major Brazilian newspaper, which ensures high quality contemporary Brazilian Portuguese from different authors and domain. The manual validation and correction process was carried out on the morphosyntactic tagging of the texts performed by the parser PALAVRAS16. The corpus contains structural markers for sentences. Besides being annotated by the XML-compliant format proposed by the Advisory Group on Languages Engineering Standards EAGLES (see Section 4) it will be available in annotators’ format (one word per line followed by its tag) which is appropriate for training and evaluating POS tagging methods. Status: the compilation and manual annotation have been completed; both the header edition and the corpus encoding have not been started. 
4) a portion of Lacio-Ref automatically-annotated with lemmas, POS and syntactical tags: as opposed to Mac-Morpho, this corpus will consist of a varied selection of text genres. It will be annotated by the parser Curupira and in the near future we hope to carry out a manual revision of it. The resulting Treebank will have several uses: to improve the parser itself, to train statistical parsers, to perform more accurate searches, etc. Status: compilation and header edition have not been started. 
5) Par-C Corpus: because of its opportunistic behavior, this corpus will be enlarged from time to time. In this initial phase of the LW project Par-C is composed of 65 pairs of authentic academic parallel texts (abstracts) in Computer Science contributed from Project PESA17 which aims at evaluating sentence alignment methods. The corpus was divided into two groups: one comprising 65 pairs of authentic (non-revised) texts; another with the same 65 pairs revised by a human translator (pre-edited corpus). They were named CAT and CPT, respectively. CAT has 416 BP sentences and 439 English sentences. CPT has 418 BP sentences and 431 English sentences. Status: compilation and alignment completed; edition of headers has not been started. 
6) Comp-C Corpus: it is also an opportunistic corpus which can be used to evaluate term extraction methods as well as for other linguistic researches. In this initial phase it comprises English-Portuguese comparable texts contributed by the Project COMET (Tagnin 2001, 2002). These texts are technical, scientific and marketing-related, amounting to 300,000 words in each language. The corpus was compiled by students in the Diploma in Translation course at FFCLH/USP in order to build glossaries18. Status: compilation completed; edition of the headers to be started. 
4. Issues in corpus creation tackled in the Lacio-Web Project 
4.1 Accessibility of the corpora 
As mentioned before, one of our main goals in constructing the LacioWeb corpora is to create a benchmark for Computational Linguistics in Brazilian Portuguese for tasks such as POS tagging, parsing, text alignment, and term extraction. For that purpose, texts must be provided with several sets of hand-validated linguistic annotation e.g. lemmas, POS, and syntactic tags. Additionally, in order to maximize its use, a corpus should be encoded according to largely agreed standards and must be available to each and every researcher in 
15 http://www1.folha.uol.com.br/fsp/ 
16 http://visl.hum.sdu.dk/visl/ 
17 www.nilc.icmc.usp.br/nilc/projects/pesa.htm 
18 http://www.fflch.usp.br/citrat 
18
general. Therefore, the texts of the LacioWeb corpora project will be made freely available on the Internet, followings the initiative of several recently created corpora such as: the Tycho Brahe Corpus of Historical Portuguese19, which also provides POS- and parsing-annotated texts and the COMPARA parallel corpus for Brazilian and European Portuguese texts20. With regard to providing standardized encoding for the resources we are following the design of the ANC Corpus21 which will use the specifications of an XML-compliant format of the Corpus Encoding Standard XCES (Ide et al., 2000). In our case, the texts will be available for download from the project's home page22. It will be redesigned to allow powerful searches and friendly access to non-expert users who may use LW corpora as an extension of dictionary consultation to obtain authentic examples of real word usage in BP. 
4.2 Exhaustivity versus selectivity 
One of the basic tenets of Science is the reproducibility of results. In the field of Computational Linguistics, when an algorithm is tested against a non-public corpus the result is not reproducible in principle. It may well be the case that other researchers, applying the same algorithms to other corpora, may find distinct results. Without applying the test to a common, public corpus, these discrepancies may never be reconciled. From this point of view, any method, algorithm or measurement performed over a non-public corpus cannot be considered definitively tested. Moreover, no comparison of two distinct computational linguistic methods which perform the same task can be made unless they are tested against the same set of data. The reason is clear. 
One method may be better suited for one kind of text, while the other may be better suited for some other kind of data. Two researchers may test both methods on their private data and arrive at different conclusions. Again, the only way to solve this dispute is to apply both methods to the same piece of public data. And if different pieces of data yield different results, this is an important piece of information that one will not have unless the data is public. This fact has guided us in the construction of the Lacio-Ref, in particular in the guidelines for text gathering and classification23. This leads us to the next topic: the exhaustivity of the texts to be included, mainly, in Lacio-Ref. By exhaustivity we mean the broad and varied coverage of the most common categories used to classify texts such as: authorship, text size, and the four-category typology, namely genre (literary ¯ subcategorized into poetry, drama, prose ¯ and non-literary ¯ subcategorized into factual, informative, academic and legislation), textual type (e.g. article, manual, research project, letter, biography), medium (e.g. newspapers, books, ads, journals, magazines) and subject (or domain). The genre typology instances can be further subcategorized. This is a differential regarding reference corpus design as they are generally built in an ad hoc manner. Linguistic aspects of texts are usually only provided for corpora which are designed to be balanced, but in the hope of broadening the set of potential users of the Lacio-Ref such information will be provided for its texts. 
Each text classification according to these categories and others related with text publication will be documented in the text's header via XML-annotations in a format inspired by the TEC corpus header annotation. Choosing to be exhaustive instead of selective is another way to broaden the set of potential users of the corpus and the possibilities of linguistic analysis. We are not limited to including only a specific genre, e.g. informative with newspaper data, as followed by the CRPC project, cited in the Introduction, with regard to the Brazilian Portuguese variant. We also follow a much more detailed genre typology than those used in the BNC written corpus and the CNC SYN2000. Additionally we use a 4-orthogonal category typology (genre, type of text, medium and domain) as commented before while these other corpora focus mainly on domain. For example, BNC classifies its texts by medium, domain and time, reducing its genre typology to two classes: informative writings which are subcategorized by domain and imaginative writings which are subcategorized by literary and creative works. SYN2000 follows the same genre typology as the BNC but subdivides imaginative texts into poetry, drama, fiction, other and transitional types and the informative into journalism and technical and specialized texts. The latter is again subcategorized by domain as in the BNC. This does not mean that we are unaware of the difficulties to classify the texts following a more detailed taxonomy but this enterprise is worth trying as it can provide a better search tool for the user. 
19 www.ime.usp.br/~tycho 
20 www.linguateca.pt/COMPARA/ 
21 http://americannationalcorpus.org/ 
22 www.nilc.icmc.usp.br/nilc/projects/lacio-web.htm 
23 We tried to adhere as much as possible to the EAGLES recommendations on text typology. 
19
4.3 Representativeness and balance 
Availability and exhaustivity were two criteria that superseded representativeness and text balancing in the selection of texts for the Lacio-Ref corpus. Problems of copyright would prevent us from obtaining a balanced corpus within the two-year duration of the project for, unlike other corpora created by a consortium of publishers like the ANC, we started with no repository of texts of our own24. However, we have so far been very successful in obtaining authorization from newspaper and magazine publishers as well as donations of electronic versions of public domain books, which will enable us, in the near future, to start the design of a balanced corpus of modern Brazilian Portuguese which will be included in the set of corpora of the Lacio-Web project. After the initial text gathering and classification we will be in a position to follow Biber's classical recommendation (1993, abstract), “The actual construction of a corpus would then proceed in cycles: the original design based on theoretical and pilot-study analyses, followed by collection of texts, followed by further empirical investigations of linguistic variation and revision of the design.” 
5. Conclusion and future work 
In Brazil, there is an urgent need for corpora constructed according to corpus linguistics principles regarding its text typology, encoded according to largely accepted standards, and freely available. In this paper we presented the Lacio-Web project which aims at compiling freely accessible corpora for both non-expert users interested in the Brazilian Portuguese language and expert users who pursue theoretical and practical linguistic studies and develop computational linguistics tools. Its purpose is not only to raise the quality of Portuguese NLP but also popularize the use of corpora for layman who may be interested in using them as an extension of dictionary consultation to obtain authentic examples of real word usage in BP. After the first year of the project we have already corrected the morphosyntactic annotation of the 1,1 million-word corpus MAC-Morpho, gathered several texts for the Lacio-Ref, critically analyzed the CN corpus which can also provide material for the Lacio-Ref. We are almost ready to release the first version of the bilingual corpus Par-C and the Comp-C. It is worth mentioning that the header must be included in the entire corpora before the release. In a new project, we intend to pursue the balancing of Lacio-Ref to make it more useful for research. 
References 
Andersen, M.S., Asmussen, H. & Asmussen, J. 2002 The project of Korpus 2000 going public. In Proceedings of Euralex 2002, pp 291-299. 
Atkins, S., Clear, J. & Ostler, N. 1992 Corpus design criteria. Literary and Linguistic Computing 7: 1-16. 
Biber, D. 1993 Representativeness in corpus design. Literary and Linguistic Computing 8: 1-15. 
Biderman, M.T.C. 2001 Dicionário de freqüencias do portugues brasileiro contemporâneo. In: Martins Fontes (ed), Teoria Lingüística. Sao Paulo, pp 335-348. 
Borba, F. S. 1991 Dicionário Gramatical de Verbos. Sao Paulo, UNESP. 
Borba, F. S. 2002 Dicionário de usos do Portugues do Brasil. Sao Paulo, Editora Ática. 
Fillmore, C., Ide, N., Jurafsky, D., and Macleod, C. 1998 An American National Corpus: A Proposal. In Proceedings of the First International Language Resources and Evaluation Conference, Granada, Spain, pp 965-70. 
24 It is important to note that although we have the CN its texts don´t have copyright clearance, therefore they are not ready for inclusion in the Lacio-Ref. 
20
Ide, N., Reppen, R., Suderman, K. 2002 The American National Corpus: More Than the Web Can Provide. In Proceedings of the Third Language Resources and Evaluation Conference (LREC), Las Palmas, Canary Islands, Spain, pp 839-844. 
Ide, N., Macleod, C. 2001 The American National Corpus: A Standardized Resource of American English. In Proceedings of Corpus Linguistics 2001, Lancaster UK. Available in: 
www.cs.vassar.edu/faculty/ide/pubs.html 
Ide, N., Bonhomme, P., Romary, L. 2000 XCES: An XML-based Standard for Linguistic Corpora. In Proceedings of the Second Language Resources and Evaluation Conference (LREC), Athens, Greece, pp 825-30. 
Martins, R.T.; Hasegawa, R.; Nunes, M.G.V.; Montilha, G.; Oliveira Jr., O.N. 1998 Linguistic issues in the development of ReGra: a Grammar Checker for Brazilian Portuguese. Natural Language Engineering 4(4): 287-307. 
Neves, M. H. M. 2000 Gramática de Usos do Portugues. Sao Paulo, UNESP. 
Sardinha, T.B. 2000 Lingüística de Corpus: Histórico e Problemática (Corpus Linguistics: History nd Problematization), DELTA 16(2): 323-367. 
Sinclair, J. and Ball, J. 1996 Preliminary Recommendations on Text Typology. EAG-TCWG-TTYP/P, June 1996. Available in: www.ilc.pi.cnr.it/EAGLES/texttyp/texttyp.html 
Tagnin S.E.O. 2002 Corpora and the Innocent Translator: how can they help him. In Lewandowska-Tomaszczyk, Barbara & Marcel Thelen (eds.) Translation and Meaning - Part 6 - Proceedings of the Lodz Session of the 3rd International Maastricht-Lodz Duo Colloquium on 'Translation and Meaning", held in Lodz, Poland, 22-24 September, 2000, Maastricht: Universitaire Pers Maastricht, pp 489-496 
Tagnin S.E.O. 2001 COMET – A Multilingual Corpus for Teaching and Translation. In: PALC ‘01 – International Conference on Practical Applications in Language Corpora, Lodz, Polônia, To appear in the Proceedings. 
Wittmann, L. Pego,T. & Santos, D. 1995 Portugues do Brasil e de Portugal: alguns contrastes. In Actas do XI Encontro da Associaçao Portuguesa de Lingüística, Lisboa, Portugal, pp 465-487. 
21

A corpus of seventeenth-century English news reportage: 
construction, encoding and applications 
Dawn Archer, Andrew Hardie, Tony McEnery & Scott Piao 
Dept. Linguistics, Lancaster University 
This poster describes a 750 thousand-word corpus of news reportage from the English Civil War, currently under construction at Lancaster, using texts drawn from the Thomason Tracts. This collection (held at the British Library) is unique in containing the greater part of what was published in London in the period 1640-1661. 
We are in the process of transcribing the periodical newsbooks published between December 1653 and May 1654 to an SGML-based format. The fairly light markup initially used by the transcribers is similar to HTML, allowing the texts subsequently to be mapped automatically to both a TEI-compliant SGML/XML format and a web-compatible HTML format, facilitating the widest possible potential re-use of the corpus. 
Simultaneous with the development of the corpus, a number of linguistic/historical issues have been investigated using the transcribed newsbooks. These include an examination of the complicated nature of text re-use in the press at this period, and an inquiry into the presentation of women in different newsbooks. 
32

A database system for storing second language learner corpora 
Bertol Arrieta(1), Arantza Díaz de Ilarraza, Koldo Gojenola, Montse Maritxalar, Maite Oronoz 
Affiliation: IXA Group (http://ixa.si.ehu.es) 
University of the Basque Country (UPV/EHU) 
Postal address: Faculty of Computer Science 
649 p.k., 20080 Donostia (The Basque Country) 
Tel.: +34 943 015 061 
Fax: +34 943 219 306 
E-mail (1): bertol@si.ehu.es 
Abstract 
With the aim of storing learner corpora as well as information about the Basque language students who wrote the texts, two different but complementary databases were created: ERREUS and IRAKAZI. Linguistic and technical information (error description, error category, tools for detection/correction…) will be stored in ERREUS, while IRAKAZI will be filled in with psycholinguistic information (error diagnosis, characteristics of the writer, grammatical competence…). These two databases will be the basis for constructing i) a robust Basque grammar corrector and, ii) a computer-assisted language-learning environment for advising on the use of Basque syntax. 
1. Introduction 
The IXA research group has been working in Natural Language Processing during the last 14 years. At the same time, we have worked on Intelligent Computer Assistant Language Learning (ICALL) environments. The work we present in this paper has a wide background in these fields: NLP tools, error detection and ICALL environments using adapted NLP tools. 
Background in NLP tools 
In order to work on error detection, a very important background in NLP tools is needed. In this sense, these are the tools implemented in our group: 
a. EDBL, a lexical database, which at the moment contains more than 80,000 entries (Aduriz et al., 1998). 
b. A tokeniser that identifies tokens from the input text. 
c. Morpheus, a wide-coverage morphosyntactic analyser for Basque (Alegria et al., 2002) that includes a segmentiser, a morphosyntactic analyser and a recogniser of multiword lexical units (MWLUs). 
d. EusLem, a general-purpose tagger/lemmatiser. (Ezeiza et al., 1998). 
e. A shallow syntactic analyser that identifies noun phrases and verbal chains. 
Background in error detection 
As we have developed most of the tools in the linguistic analysis chain (morphology, morphosyntax, surface syntax, phrases, etc.), we started working on error detection. Thus, a robust spelling corrector, called Xuxen (Aduriz et al., 1997), was developed some years ago. With the aim of following with this work, a syntactic approach was planned. This way, some work in syntax error detection has been done in the last years, using different approaches: 
a. We have combined a robust partial parser which obtains the main components of the sentence (implemented in PATR-II), and a finite-state parser used for the description of syntactic error patterns (Xerox Finite State Tool, XFST, (Karttunen et al., 1997)) to detect errors in dates (Gojenola K. & Oronoz M., 2000). We defined six different types of errors and its combinations. 
33
b. The Constraint Grammar (Karlsson, 1995) formalism has been used to analyse 25 types of errors about postpositions and other 10 different types of grammar errors. 
c. The relaxation of syntactic constraints (Douglas & Dale, 1992) has been used for the detection of agreement errors between the verb and the subject, object or indirect object (Gojenola, 2000). This grammar-based method allows the analysis of sentences that do not fulfil some of the constraints of the language by identifying a rule that might have been violated, determining whether its relaxation might lead to a successful parse. 
Background in ICALL environments 
The main work in this field done in our group is an environment for studying the learning process of language learners, called MUGARRI (Maritxalar, 1999). In this environment, we find three systems: IRAKAZI, IDAZKIDE and HITES. IRAKAZI helps the teacher in gathering psycholinguistic information about the students and the texts they write; IDAZKIDE is a student oriented ICALL environment for second language learning; and HITES is a system for modelling the interlanguage of particular learners and the common interlanguage of learners at the same language level. IRAKAZI interacts with the teacher, IDAZKIDE with the student, and HITES with the psycholinguist. 
ERREUS and its connection with IRAKAZI 
With this background, we realised that gathering error corpora is a very important task in order to i) have a basis for deciding which type of linguistic phenomena are important to treat, ii) have a corpora for tool-testing and evaluating. That is why we began thinking about a system that would store information about the errors of the corpora. The ERREUS database was born with this aim. ERREUS has the purpose of storing technical and linguistic information about any type of error, and it was designed for being, in some sense, a repository of error corpora. On the other hand, IRAKAZI is used to store all the information about the student (mainly, relative to his/her learning process) and the deviant structures he/she has used. 
Working in the design of the ERREUS database, we realized that ERREUS is complementary with IRAKAZI. In ERREUS we are going to store any kind of error made by language learners and native speakers, and in IRAKAZI, only the deviations made by language learners. It must be pointed out that in the ICALL environment, we will speak about deviant structures instead of errors. The word “error” is directly joined to correction, and it has a negative sense. That is why we have decided to use the word deviation when speaking about the learning process, following some psycholinguistic trends (Maritxalar et al, 1996). So, all deviations made by students are going to be referenced in both the IRAKAZI and the ERREUS databases, while the errors found in corpora that were not written by students are only going to be stored in the ERREUS database, as we can see in Figure 1. 
Due to the fact that both databases provide different points of view about the same matter, we saw the need of joining the two databases. Thus, for each error-containing-text, we would have its technical-linguistic information in ERREUS, as well as its corresponding psycholinguistic information in IRAKAZI. 
Taking into account the information stored in each database (see example in figure 2), we note that the information about the text that contains the error and its category appears in both. However, there is a difference when representing the linguistic category. In the case of IRAKAZI, we store the concrete category of the deviation (AGREEMENT_SUBJ_VERB), while in ERREUS we use a hierarchical classification of linguistic errors (Morphosyntactic   Agreement   Agreement between subject and 

Figure 1: Errors vs. deviations 

Deviations in the IRAKAZI database 

Errors in the ERREUS database 
34
verb). Being the case that the final category in ERREUS matches with the category in IRAKAZI, it is viable to join both databases. Deviation-Sentence: *Hura igeriketamaitedu(*He love swimming) -Category: AGREEMENT_SUBJ_VERB-Deep reason: Generalization of a ruleText-Number of words: 245-Type of text: exercise-Reference: 122Student-Name: AnaBerazadi-Age: 25-School:Ilazki-Language knowledge level: +Spanish (speak,understand,write, read): 5, 5, 5, 5+French (speak,understand,write, read): 3, 3, 3, 4-Mother language: Spanish-Learning historyError containing text-Sentence: *Hura igeriketa maitedu(*He love swimming)-Correction: Hark igeriketamaitedu(He loves swimming)-Text reference: 122Error-Description: loose of the letter ‘s’ in third singular person, in present tense-IsDetected? Yes-DetectionTool: Constraint Grammar-IsCorrected? No-CorrectionTool: -Category and subcategory levels-FirstCategory: Morphosyntactic-SecondCategory: Agreement-ThirdCategory: Agreement between subject and verbIRAKAZI databaseERREUS database 
2. The ERREUS database 
The ERREUS database will be the basis for constructing a robust Basque grammar corrector. That is the reason why we have designed a database that stores linguistic and technical information of errors found in the corpora. 
Designing the database, we have followed several steps for assuring a good design and development, taking into account that a) it is important to access the database via Internet, b) many non-specialized users would access it, c) we want to store a very large range of linguistic errors, and d) we want to link ERREUS to IRAKAZI. 
Next, we will briefly explain the steps we followed to build the database. Firstly, we made a complete classification of errors based on bibliographic research and hand-made studies of real corpora (step 1). Secondly, this classification was complemented with the results of a questionnaire made to some proofreaders and Basque language teachers (step 2). And, finally, this classification was used as a basis for designing and constructing the ERREUS database (step 3) and its corresponding ZOPE based interface (step 4). 
Step 1: Classifying the errors 
As mentioned before, in order to make a thorough classification of the errors we could find in any corpora, we used as a basis a set of Basque grammars, our previous experience in error classification (Maritxalar, 1999), and the advice of the linguists in our research group. Besides, we contrasted our classification with other works on error typology (Becker et al., 1999) made in other languages. 

Figure 2: A view of the main information stored in ERREUS and in IRAKAZI (these boxes are not the entities of neither ERREUS nor IRAKAZI) 
35
This way, we obtained a classification in which all errors were divided into five main categories: 
- Spelling errors 
- Morphological, syntactic or morphosyntactic errors 
- Semantic errors 
- Punctuation errors and style suggestions 
- Errors due to the standardisation process of Basque 
Each category was subcategorised so as to make a classification as detailed as possible. 
The relevance of this error classification relies on guiding the user through the interface into the appropriate category/subcategory. This procedure will let us organise in the database all the error occurrences according to the mentioned classification. 
Step 2: The questionnaire and its results 
The standardisation of Basque has not been yet completed. The Basque Language Academy (http://www.euskaltzaindia.net) publishes periodically rules for the standardisation of the language, but they do not cover all its aspects. For this reason, sometimes it is difficult to decide whether a given structure may be considered standard or not. 
All these characteristics made more difficult to create a proper error classification in Basque. Therefore, we prepared a questionnaire in order to contrast our classification. As we assumed that learners of Basque and natives do not make the same kind of errors and with the same frequency, we asked both, experienced Basque teachers and proofreaders, about two different aspects. We gave them our first draft of the error classification, and asked whether they knew any error category that was not included in such classification and whether all the errors we considered were actually errors. If this was the case, we also wanted to know, which was the frequency of occurrence of each error in the kind of texts they usually work with. Using this data, we completed our error classification. In the near future, we are going to intend to continue implementing rules for the detection of errors, starting with those ranked with the highest frequency in the questionnaire. Our objective is to use these rules to detect automatically such errors in real corpora. 
Step 3: The design of the database 
We carried out the design of the database with the objectives of being open and flexible enough to allow the addition of new information. We designed a simple, standard database to collect errors of the different types mentioned in the classification. The system will also allow restricted users to update the database via Internet. 
The database is composed of these main entities: error, linguistic categories, text and correction. 
In the entity named ‘error', we store, among other things, the following technical information: whether the error is automatically detectable/rectifiable, and in such case, which is the most appropriate NLP tool to detect/correct it. We also specify the origin of the error (e.g. influence of Spanish) and the possible cause of it. 
We have used four tables, each one for each level of the hierarchy in the classification. For example, in the first table (FirstLevelCategory), we have the general category of the error (orthographic, morphosyntactic, semantic, punctuation, style and errors due to the lack of standardisation of the language). Besides, each general category is divided into second level subcategories using the table (SecondLevelCategory), and so on (see figure 3). 
36
The entity named ‘text’ stores, for each error occurrence, the sentence that contains the error. Besides, we have an attribute (with a value ranging from 0 to 5) to indicate to which extent we are sure that it is really an error in the context where it appears. A given word or structure might be always considered an error or it might be considered an error just in some given contexts (e.g. “The bread ate John” might be correct in a literary context). -Spelling errors -Morphological, syntactic or morphosyntacticerrors -Semantic errors -Punctuation errors -Errors due to the lack of standardisation of BasqueFirstLevelCategory-Loose of the “h” letter -…-Derivation of words-Dates -Postpositions-Bad use of adverbs-Fixed phrases-Word order-Agreement-…--…--…SecondLevelCategory…ThirdLevelCategory…ForthLevelCategory
In the ‘correction’ entity, we store the correction of each error occurrence. In this sense, it is important to remark that if we have more than one error in a concrete sentence, we will have one different text occurrence for each error, in order to i) have the proper reference to each kind of error, and ii) have one correction for each kind of error. 
Step 4: The ZOPE based interface 
As the interface will be used by people that are not specialised in computers, and, therefore, it has to be an easy-to-use tool, we designed a simple and user-friendly interface based in ZOPE technology (Latteier & Pelleitier M., 2001). 
This way, we built an interface to guide the user into the error classification in order to choose one concrete category/subcategory. The user has the possibility of making different operations: 
a. Consulting operations as to find real examples of errors in the corpora for the chosen category. For example, if the category/subcategory “Morphosyntactic / Agreement / AgreementBetweenSubjectVerb” is chosen, the system will show all the texts with this error, e.g. “Hura igeriketa maite du” (“He love swimming”), “Bera ingelesa daki” (“She cans speak English”) and so on. 
b. Inserting operations as to insert an error-containing-text into the chosen category, with its own correction. For example, let us suppose that the linguists find the next sentence “That's not very apropriate”. The steps to follow should be: 
i. Find the proper category/subcategory for that error (Spelling error) 
ii. Check if the error has already been inserted (one “p” instead of the double “p”) 
iii. If not, then, insert the error and its technical characteristics. 
iv. Check if the sentence has already been inserted. 
v. If not, insert the sentence and its correction. 

Figure 3: Classification hierarchy 
37
c. Updating operations, related to error information as well as text information. an TranslationDevelopmentSummaryBasoarenbarrena joan zirenThey went across the forestBasoarenbarrena joan ziren perretxikoen bilaThey went across the forest to pick up mushroomsBasobarrena joan ziren perretxikoen bilaThey went across the forest to pick up mushroomsSAVE A TEXTTo which extend we are sure that it is really an error in this context.
3. The IRAKAZI database 
In the introduction, we have done a distinction between the information stored in ERREUS (linguistic/technical) and the information stored in IRAKAZI (psycholinguistic). In the same way, as we mentioned before, we distinguish errors (ERREUS) from deviant structures (IRAKAZI). So, when speaking about IRAKAZI, we will refer to students’ deviations. 
IRAKAZI is responsible of storing the knowledge about the student, given by the teacher. The main goal is twofold: 
- To get information about the student, relative to his/her learning process. 
- To work on the diagnosis of the deviant structures of the learner. 
In the future, all this information will be used in the development of the diagnosis module of IDAZKIDE (Diaz de Ilarraza et al, 1999), a student oriented ICALL system for second language learning. 
IRAKAZI is composed of an interface to interact with the teacher and a knowledge base (field work). This knowledge base contains information about the learners, the type of exercises they do and the deviations found in texts written by them (see figures 5, 6 and 7). 

Figure 4: Inserting a text in ERREUS 

Figure 5 First screen in IRAKAZI. 
38
The interface of IRAKAZI helps the teacher to provide the necessary information to keep in the knowledge base. Such information is composed of: 
- Specific features of the student such as age, language level, mother tongue, other languages, studies, frequency of use of Basque, environment of use (home, business, tourism...) and so on (see figure 6). 
- Information about texts written by the student (including a list of deviations written in the texts). Each deviation will be classified from three different points of view: i) a superficial point of view (e.g. omission of a letter), ii) a linguistic/metalinguistic point of view (e.g. agreement between subject and verb, creation of new words by means of loans…), and iii) a deep perspective (e.g. transfer from mother tongue…). The last one includes the reasons why the deviations were committed (Maritxalar & Díaz de Ilarraza, 1993). 
- Information about the exercises proposed and their objective. For example, some texts can be the result of guessing a story or talking about something heard before, etc. 
At this moment, the implementation of IRAKAZI is done in Access, but a new improved version implemented in Zope will be available in Internet in few months. In this new version where we are working on, we will add information about the necessary strategies that should be followed when helping the student in improving the knowledge related to his/her deviant structures. In order to do that, we will collect information about the most adequate types of exercises for the treatment of the deviant structures in each case, that is, in the case of each particular learner (see figure 7). 
Some years ago, when IRAKAZI and MUGARRI were designed and implemented, a field work was done collecting Basque students texts from some schools specialised in the teaching of the language (Diaz de Ilarraza et al., 1998). These text corpora are a very interesting source of data, and they will be described in the next section. 
4. Text corpora 
Annotated corpora of errors for Basque is a very important resource, not only for deriving an empirically based error classification, but also as a basis for the development of error detecting tools. In our case, ERREUS and IRAKAZI will be used as repositories of errors that will be annotated from linguistic/technical and psycholinguistic points of view, respectively. Text corpora provide the necessary information to both databases. 
In text corpora, each kind of error occurs with very low frequency and, therefore, big corpora are needed for testing. The task of collecting corpora is not easy and it turns very difficult when error corpora have to be collected. Even if such corpora were available, the task of recognising error instances for evaluation is a hard task, as there are no syntactically annotated treebanks in Basque with error marks. So, if we want to obtain naturally occurring test data, hundreds of texts have to be automatically and manually examined and marked. 
This work of collecting Basque students texts was done following some criteria: we collected written material from different language schools (IRALE1, ILAZKI, AEK) and grouped this material depending on some features of the texts as i) the kind of exercise proposed by the teacher (e.g. abstract, 39 

Figure 7: Texts and deviations in the texts 

Figure 6: Specific features of the student 
1 IRALE, ILAZKI and AEK: schools specialised in the teaching of Basque 

article about a subject, letter…) and ii) the student who wrote the text. These were students who attended classes regularly, and with different characteristics and motivations for learning Basque (e.g. different learning rates, different knowledge about other languages, mother tongue…). The corpus is made up of 350 texts written from 1990 to 1995. We codified the texts of the corpora following a prefixed notation (e.g. il10as) showing the language school (e.g. “il”, ILAZKI), the language level (e.g. “10”, Level 10), the learner's code (e.g. “a”, first letter of the name Ainhoa), and the type of exercise proposed (e.g. “s”, summary). Information related to this corpus is stored in ERREUS and IRAKAZI. 
In addition to these texts, an archive of 1600 e-mail messages from the mailing list “EuskaraZ”, the first workgroup in Basque, has been collected. This list was created in 1996 with the purpose of exchanging information about everything related to Basque. This corpus has the advantage of being easily accessible, electronically available and contains linguistic errors. On the other hand, it has the disadvantage of being a corpus written in an informal language, sometimes with incomplete words and abbreviations, so it is not easy to analyse it. 
Apart from this text corpus, we will use grammars with error examples (Zubiri, 1994) as a source of errors and texts for filling in ERREUS. 
5. Conclusions and Future Work 
We have implemented two databases that gather information on linguistic errors analysed from different points of view. An interdisciplinary approach has been followed when analysing written errors in Basque texts. IRAKAZI contains the information related to the learning process of the student (diagnosis, writer characteristics, grammatical competence…), and ERREUS is provided with a vast linguistic and technical description of the errors (classification of the error, description, occurrences in texts, possible tools used for detection/correction…). Both databases have a reference to previously built and encoded learner corpora. So, we can link the two databases and create a complete system that will take into account different points of view about errors/deviations. 
In the near future, we have two projects in mind: 
a. A robust grammar corrector of Basque. 
b. A system for syntax teaching that will improve IDAZKIDE. 
The information contained in ERREUS is essential in the development of both projects, while IRAKAZI is a very important source of psycholinguistic information that will be used in IDAZKIDE. 
HITES is a system for modelling the interlanguage of particular learners and the common interlanguage of learners at the same language level. Using the information obtained from HITES, IDAZKIDE will be able of giving linguistic advice to the students considering their level. For that purpose, the tools previously constructed in our NLP research group (the spelling corrector, the electronic dictionaries, the tool for shallow parsing…) could be adapted taking into account the knowledge level of the student. 
In the future, we will construct mechanisms in the form of linguistic rules, grammars or statistical methods for the detection of, basically, grammar errors and deviations. 
Acknowledgements 
This research is supported by the University of the Basque Country (9/UPV00141.226-14601/2002) and, the Ministry of Industry of the Basque Government (XUXENG project, OD02UN52)). Thanks to Eli Pociello for her help writing the final version of the paper. 
References 
Aduriz I., Aldezabal I., Ansa O., Artola X., Díaz de Ilarraza A., Insausti J. M. 1998 EDBL: a Multi-Purposed Lexical Support for the Treatment of Basque. In Proceedings of the First Int. Conf. on Language Resources and Evaluation, vol II, 821-826. Granada (Spain). 
Aduriz I., Alegria I., Artola X., Ezeiza N., Sarasola K., Urkia M. 1997 A spelling corrector for Basque based on morphology. Literary & Linguistic Computing, Vol. 12, No. 1. Oxford University Press. Oxford. 1997. 
40
Alegria I., Aranzabe M., Ezeiza A., Ezeiza N., Urizar R. 2002 Robustness and customisation in an analyser/lemmatiser for Basque. In proceedings of the LREC-2002 Customizing knowledge in NLP applications workshop. 
Becker M., Bredenkamp A., Crysmann B., Klein J. 1999 Annotation of Error Types for German News Corpus. In Proceedings of theATALA workshop on Treebanks, Paris. 
Díaz de Ilarraza A., Maritxalar A., Maritxalar M., Oronoz M. 1999 IDAZKIDE: an intelligent CALL environment for second language acquisition. In Proceedings of a one-day conference "Natural Language Processing in Computer-Assisted Language Learning" organised by the Centre for Computational Linguistics , UMIST, in association with EUROCALL, a special ReCALL publication, 12-19. UK. 
Díaz de Ilarraza A., Maritxalar M. Integration of natural language techniques in the ICALL systems field: the treatment of incorrect knowledge UPV/EHU-LSI TR 9-93. 
Díaz de Ilarraza A., Maritxalar M., Oronoz M. 1998 An Implemented Interlanguage Model for Learners of Basque. Language Teaching and Language Technology. Swets and Zeitlinger (Publisher). Sake Jager, John Nerbonne and Arthur van Essen editors. Lisse. pp 149-166. 
Douglas, S., Dale R. 1992. Towards Robust PATR. In proceedings of COLING'92, Nantes. 
Ezeiza N., Aduriz I., Alegria I., Arriola J.M., Urizar R. 1998 Combining Stochastic and Rule-Based Methods for Disambiguation in Agglutinative Languages. In Proc. COLING-ACL'98, 10-14. Montreal (Canada). 
Gojenola K. & Oronoz M. 2000. Corpus-Based Syntactic Error Detection Using Syntactic Patterns. In proceedings of NAACL-ANLP00,Student Research Workshop . Seattle. 
Gojenola, K. 2000 EUSKARAREN SINTAXI KONPUTAZIONALERANTZ. Oinarrizko baliabideak eta beren aplikazioa aditzen azpikategorizazio-informazioaren erauzketan eta erroreen tratamenduan. Unpublished PhD thesis, University of the Basque Country. 
Karlsson F., Voutilainen A., Heikkilä J., Anttila A. 1995 Constraint Grammar: A Language-independent System for Parsing Unrestricted Text. Mouton de Gruyter. 
Karttunen L., Chanod J-P., Grefenstette G., Schiller A. 1997 Regular Expressions For Language Engineering. Journal of Natural Language Engineering. 
Latteier A. & Pelleitier M. 2001. The Zope Book. New Riders. 
Maritxalar M., Díaz de Ilarraza A., Alegria I., Ezeiza N. 1996 Modelización de la competencia gramatical en la interlingua basada en el análisis de corpus. Procesamiento del Lenguaje Natural (SEPLN), 19: 166-178. 
Maritxalar, M. 1999 Mugarri: Bigarren Hizkuntzako ikasleen hizkuntza ezagutza eskuratzeko sistema anitzeko ingurunea. Unpublished PhD thesis, University of the Basque Country. 
Zubiri I. 1994 Gramática didáctica del euskera Didaktiker, S.A. 
41

Towards a methodology for corpus-based studies of linguistic change 
Contrastive observations and their possible diachronic interpretations 
in the Korpus 2000 and Korpus 90 General Corpora of Danish 
Jorg Asmussen 
Society for Danish Language and Literature 
Det Danske Sprog- og Litteraturselskab 
DSL 
The Korpus 2000 Project, www.dsl.dk/korpus2000 
Christians Brygge 1, DK-1219 Copenhagen K 
ja@dsl.dk 
Abstract 
Corpora serve as a widely accepted base for synchronic descriptions of language. Yet easily accessible general corpora that enable diachronic descriptions of language are still quite rare, the Korpus 90 and Korpus 2000 Corpora of Danish being one exception. 
Korpus 90 and Korpus 2000 were both designed and compiled at the Society for Danish Language and Literature (DSL). Korpus 90 comprises text material from the period 1983-1992 and was compiled in the early 1990s. Korpus 2000 is a recently compiled corpus holding text material from the period 1998-2002. The joint web-based query interface of the two corpora enables immediate comparative studies. 
This paper first gives a very brief introduction to the background of the two corpora before focusing on examples of contrastive observations and their possible diachronic interpretations - and misinterpretations. The examples cover frequencies (new words, vanishing words), the inflectional and collocational behaviour of certain words, and their connotations. An example illustrating syntactical differences is also briefly sketched. The paper then discusses whether these observable differences reflect real changes in the Danish language, or whether they reflect the probable fact of differently compiled - and thus perhaps incomparable - corpora. 
Finally, the paper proposes some prerequisites for a methodology of comparative corpus investigation and the determination of diachronic corpus similarity. In this context, the concept of invariant textual features will be introduced. 
42

A New Machine Learning Algorithm for Neoposy: coining new Parts of Speech 
Eric Atwell, 
School of Computing, University of Leeds 
eric@comp.leeds.ac.uk 
http://www.comp.leeds.ac.uk/eric 
1. Introduction: Unsupervised Natural Language Learning 
According to the Collins English Dictionary, “neology” is: a newly coined word, or a phrase or familiar word used in a new sense; or the practice of using or introducing neologies 
. 
We propose “neoposy” as a neology meaning “a newly coined classification of words into Parts of Speech; or the practice of introducing or using neoposies”. 
Unsupervised Natural Language Learning, the use of machine learning algorithms to extract linguistic patterns from raw, un-annotated text, is a growing research subfield; for examples, see Proceedings of annual conferences of CoNLL: Computational Natural Language Learning, or the membership list of ACL-SIGNLL, the Association for Computational Linguistics – Special Interest Group in Natural Language Learning. Corpora, especially tagged and parsed corpora, can be used to train `machine learning’ or computational language learning models of complex sequence data. (Jurafsky and Martin 2000) divide Machine Learning systems into Supervised and Unsupervised approaches (p118): 
“… The task of a machine learning system is to automatically induce a model for some domain, given some data from the domain and, sometimes, other information as well… A supervised algorithm is one which is given the correct answers for some of this data, using those answers to induce a model which can generalize to new data it hasn't seen before… An unsupervised algorithm does this purely from the data. While unsupervised algorithms don't get to see the correct labels for the classifications, they can be given hints about the nature of the rules or models they should be forming… Such hints are called a learning bias.” 
Hence, in Corpus-based Computational Language Learning, a supervised algorithm is one trained using an annotated corpus; for example, a supervised ML parser such as (Atwell 1988; 1993) is trained with a Treebank of example sentences annotated with their parses. An unsupervised algorithm has to devise an analysis from raw, un-analysed corpus data; for example, an unsupervised ML parser such as (van Zaanen 2002) is trained with raw text sentences and has to propose phrase-structure analyses “by itself”. 
2. Clustering words into word-classes 
A first stage in Unsupervised Natural Language Learning (UNLL) is the partitioning or grouping of words into word-classes. A range of approaches to clustering words into classes have been investigated (eg Atwell 1983, Atwell and Drakos 1983, Hughes and Atwell 1994, Finch and Chater 1993, …, Roberts 2002). In general these researchers have tried to cluster word-types whose representative tokens in a Corpus appeared in similar contexts, but varied what counts as “context” (eg all immediate neighbour words; neighbouring function-words; wider contextual templates), and varied the similarity metric and clustering algorithm. 
This approach ultimately stems from linguists’ attempts to define the concept of word-class in term of syntactic interchangeability; the Collins English Dictionary explains “part of speech” as: a class of words sharing important syntactic or semantic features; a group of words in a language that may occur in similar positions or fulfil similar functions in a sentence. For example, the previous sentence includes the word-sequences a class of and a group of ; this suggests class and group belong to the same word-class as they occur in similar contexts. 
43
Clustering algorithms are not specific to UNLL: a range of generic clustering algorithms for Machine Learning can be found in the literature (eg Witten and Frank 2000). These generic clustering systems require the user to formalise the problem in terms of a feature-space: every instance or object to be clustered must be characterised by a set of feature-values, so that instances with same or similar feature-values can be lumped together. Generally clustering systems assume each instance is independent; whereas when clustering words in a text, it may be helpful to allow the “contextual features” to either be words or be replaced by wordclass-labels as clustering proceeds (as in Atwell 1983). 
3. Ambiguity in natural language word-classification 
A common flaw, from a linguist's perspective, is that these clustering algorithms assume all tokens of a given word belong to one cluster: a word-type can belong to one and only one word-class. This results in neoposy which passes a linguist's “looks good to me” evaluation (Hughes and Atwell 1994, Jurafsky and Martin 2000) for some small word-clusters corresponding to closed-class function-word categories (articles, prepositions, personal pronouns): the author can claim that at least some of the machine-learnt word groupings “look good” because they appear to correspond to linguistic intuitions about word-classes. However, the basic assumption that every word belongs to one and only one class does not allow existing word-clustering systems to cope adequately with words which linguists and lexicographers perceive as syntactically ambiguous. This is particularly problematic for isolating languages, that is, languages where words are generally not inflected for grammatical function and may serve more than one grammatical function; for example, in English many nouns can be used as verbs, and vice versa. 
The root of the problem is the general assumption that the word-type is the atomic unit to be clustered, using the set of word-token contexts for a word-type as the feature-vector to use in measuring similarity between word-types, applying standard statistical clustering techniques. For example, (Atwell 1983) assumes that a word-type can be characterised by its set of word-types and contexts in a corpus, where the context is just the immediately preceding word: two word-types are merged into a joint word-class if the corresponding word-tokens in the training corpus show that similar sets of words tend to precede them. Subsequent researchers have tried varying clustering parameters such as the context window, the order of merging, and the similarity metric; but this does not allow a word to belong to more than one class. 
4. Classifying word types or word tokens? 
One answer may be to try clustering word tokens rather than word types. In the earlier example, we can say that the specific word-tokens class and group in the given sentence share similar contexts and hence share word-class, BUT we need not generalise this to all other occurrences of class or group in a larger corpus, only to occurrences which share similar context. To illustrate, a simple Prolog implementation of this approach, which assumes “relevant context” is just the preceding word, produces the following: 
?- neoposy([the,cat,sat,on,the,mat],Tagged). 
Tagged = [[the,T1], [cat,T2], [sat,T3], [on,T4], [the,T5], [mat, T2]] 
The Prolog variable Tagged is instantiated to a list of [word, Tag] pairs, where Tag has an arbitrary name generated by the program, letter T follwed by an integer. These integers increment starting from T1, unless a word has a “context” seen earlier, in which case it repeats the earlier tag. In the above example, word “mat” has the same context (preceding word “the”) as earlier word “cat”, so it gets the sameT2 tag instead of a new tag T6. 
We see that the two tokens the have distinct tags T1 and T5 since they have different contexts; but the token mat is assigned the same tag as token cat because they have the same context (preceding word-type). This also illustrates an interesting contrast with word-type clustering: word-type clustering works best with high-frequency words for which there are plenty of example tokens; whereas word-token clustering, if it can be achieved, offers a way to assign low-frequency words and even hapax legomena to word-classes, as 
44
long as they appear in a context which can be recognised as characteristic of a known word-class. In effect we are clustering or grouping together word-contexts rather than the words themselves. 
5. How many token-classes will be learnt? 
However, this prototype demonstrator also illustrates some problems with clustering on tokens. If we no longer assume all tokens of one type belong to one class, do we allow as many classes as there are tokens? 
Presumably not, as there would then be no point in clustering; in fact it would be hard to justify even calling this clustering. In the above example sentence there are 6 words ad 5 Tags: at least two words share a cluster, because they share a context. This means there are as many clusters as there are “contexts”; in the above case, a context is the preceding word-TYPE, so this implies we will get as many clusters as there are word-types. A million-word corpus such as the Lancaster-Oslo/Bergen (LOB) corpus yields about 50,000 word-types, so our simple token-clustered would yield about 50,000 word-classes. This is a lot less than a million (one per token), but arguably still too many to be useful. 
A “learning hint” to guide clustering may be a constraint on the number of word-classes for a word-type, for example to say all tokens of a word-type must partition into at most 5 word-classes; but how is this to be achieved? 
6 Clustering by constraint-based reasoning 
In supervised Part-of-Speech tagging research, statistical approaches (eg Constituent Likelihood Automatic Word-tagging System CLAWS, Leech et al 1983) have been challenged (and arguably surpassed) by constraint-based taggers, exemplified by Transformation-Based Learning taggers (Brill 1995), and the ENGTWOL English Constraint Grammar tagger, (Voutilainen 1995). So, a constraint-based approach to neoposy may be worth considering. However, the constraints in Brill and Voutilainen taggers were based on surrounding PoS-tags, not words. If there are no “given” (supervising) PoS-tags, all constraints in our neoposy system would have to framed in terms of word-contexts. This introduces significant computational complexity: Voutilainen's tagger had over 1000 constraints based on parts of speech within a window context, but if each PoS-context had to be replaced with specific word-contexts the number of constraints could easily grow to unmanageable proportions. 
For the neoposy task, we would like a word-type to be allowed to belong to more than one PoS-class. This means the set of tokens (and their contexts) for each word-type needs to be partitioned, into a small number of subsets, one for each PoS the word can have. We would like this partitioning to yield similar context-subsets for pairs of words which share a PoS. We can try to formalise this: 
Each word-type Wi is represented in a training corpus by word-tokens {wi1,wi2,…win} and their corresponding contexts {ci1,ci2,…cin} 
We wish to partition the set of contexts for every word-type Wi in a way which maximises the similarity of context-subsets between words which have the same PoS: 
We want to find partitions of context-sets for Wi, Wj, Wk,… of the form: 
{ {ci1,ci2,…cia}, {cia+1,...cib}, {cib+1,…},…, {…, cin} }, 
{ {cj1,cj2,…cja}, {cja+1,...cjb}, {cjb+1,…},…, {…, cjn} }, 
{ {ck1,ck2,…cka}, {cka+1,...ckb}, {ckb+1,…},…, {…, ckn} }, … 
in a way that maximises “reuse” of similar context-subsets between words: 
{ci1,ci2,…cia} ¡Ö {cj1,cj2,…cja} ¡Ö {ck1,ck2,…cka} ¡Ö … , 
{cic,…cid} ¡Ö {cje,…cjf} ¡Ö {ckg,…ckh} ¡Ö … , 
… 
45
This is a horrendously large constraint-satisfaction problem, potentially more challenging than traditional constraint-satisfaction applications such as scheduling, e.g. see (Atwell and Lajos 1993). A million-word training corpus such as LOB contains 1000K word-tokens and about 50K word-types, yielding an average of about 20 word-tokens, and hence word-token-contexts, per word-type. Cross-comparing all full context-sets, on the assumption that each word can belong to only one word-class, is already a heavy computational task. A set of 20 contexts could be partitioned in a very large number of ways, so it would take an impossibly long time to cross-compare every possible partitioning of every word with every other partitioning of every other word. 
7. Semi-supervised clustering via a language discovery toolkit 
A general observation in machine learning is that completely unsupervised learning is very hard, but even a little guidance can yield much more plausible results. The speculation above suggests that wholly unsupervised machine learning of token-based clustering may be unachievable, but perhaps some hybrid of token-based and type-based clustering, combined with limited sensible “learning hints” may be more manageable. To explore the range of possible components and parameters which might make up a successful hybrid solution, we need a “toolkit” of compatible “language discovery” software modules, to try putting together in various combinations. 
8 Further research aims 
This vague concept needs to be explored further, to pin down the sort of model and hints which could work; we need a programme of research combining Corpus Linguistics resources and Machine Learning in a Language Discovery Toolkit. (Atwell 2003) outlines a programme of further research: 
1) to explore theories and models from Machine Learning and Corpus Linguistics, to synthesise generic frameworks and models; 
2) to fit past and current research projects and software into a coherent generic framework and architecture; 
3) to collate and develop a general-purpose software toolkit for experiments with a wide range of algorithms for Corpus-based Computational Language Learning: discovery of language characteristics, patterns and structures from linguistic training data; 
4) to explore applications of this toolkit for Language Education, and for language-oriented knowledge-mining, in discovery of language characteristics, patterns and structures in a range of datasets from bioinformatics, astronomy, and multimedia datasets; 
5) to disseminate the Language Discovery Toolkit to a wide range of potential users and beneficiaries, to uncover new unforeseen applications, by providing an internet-based showcase demonstrator for the wider research and education community. 
References 
E Atwell, 1983 “Constituent-Likelihood Grammar” in ICAME Journal Vol.7 
E Atwell, 1988 “Transforming a Parsed Corpus into a Corpus Parser” in Kyto, M, Ihalainen, O & Risanen, M (eds), “Corpus Linguistics, Hard and Soft: Proceedings of the ICAME 8th International Conference on English Language Research on Computerised Corpora”, pp61-70, Amsterdam, Rodopi 
E Atwell, 1993 ”Corpus-based statistical modelling of English grammar” in S Souter and E Atwell (eds), “Corpus-based computational linguistics: Proc 12th ICAME”,pp195-214, Amsterdam, Rodopi 
E Atwell, 2003. Combining Corpus Linguistics resources and machine Learning in a Language Discovery Toolkit. Internal research proposal, School of Computing, University of Leeds 
46
E Atwell, N Drakos, 1987 “Pattern Recognition Applied to the Acquisition of a Grammatical Classification System from Unrestricted English Text” in B Maegaard (ed), “Proceedings of EACL'87: the Third Conference of European Chapter of the Association for Computational Linguistics”, Copenhagen, ACL 
E Atwell, G Lajos, 1993 ”Knowledge and Constraint Management: Large Scale Applications” in E Atwell(ed), “Knowledge at Work in Universities: Proc 2nd HEFCs-KBSI” pp21-25, Leeds, Leeds University Press 
S Finch, N Chater,1992 “Bootstrapping Syntactic Categories Using Statistical Methods” in “Proceedings 1st SHOE Workshop”, Tilburg University, The Netherlands 
J Hughes, E Atwell, 1994 ”The automated evaluation of inferred word classifications” in A Cohn (ed), “Proc 11th European Conference on Artificial Intelligence”, pp535-539, Chichester, John Wiley 
D Jurafsky, J Martin, 2000 “Speech and language processing: an introduction to natural language processing, computational linguistics, and speech recognition”, Prentice-Hall, New Jersey 
F Karlsson, A Voutilainen, J Heikkila, A Anttila (eds), 1995 ”Constraint Grammar”, Mouton de Gruyter, Berlin 
A Roberts, 2002. Automatic acquisition of word classification using distributional analysis of content words with respect to function words, Technical Report, School of Computing, University of Leeds 
M. van Zaanen, 2002. Bootstrapping Structure into Language: Alignment-Based Learning, PhD 
Thesis, School of Computing, University of Leeds. 
47

Detecting student copying in a corpus of science laboratory reports: simple and smart approaches 
Eric Atwell, Paul Gent, Julia Medori, Clive Souter, 
University of Leeds, Leeds LS2 9JT, England 
0. Introduction 
This case study is an evaluation of generic, general-purpose plagiarism detection systems applied to a specific domain and task: detecting intra-class student copying in a corpus of Biomedical Science laboratory reports. From the outset, our project had the practical, pragmatic aim to find a workable solution to a specific problem. Biomedical Science undergraduates learn experimental methods by working through a series of laboratory experiments and reporting on their results. These laboratory reports are “peer-reviewed” in large classes, following a prescribed marking scheme; as the reports are effectively marked by other students rather than by a single lecturer, there is an opportunity for an unscrupulous student to avoid having to carry out and report on an experiment, by simply copying another student's report. To reduce this temptation, the Biomedical Science director of teaching, Paul Gent, approached Eric Atwell of the School of Computing and Clive Souter of the Centre for Joint Honours in Science, to look at ways to compare laboratory reports automatically, and flag candidates with signs of copying. We were joined by Julia Medori, forensic linguist from Trinity College Dublin, who developed and evaluated a range of possible solutions. 
1. Detailed requirements analysis 
We examined example student reports in the specific genre, to identify measurable characteristic features (e.g. use of identical diagrams and graphs may be indicative of copying but generic checkers like Turnitin assume text-only essays). We also interviewed Biomedical Science teaching staff to elicit/confirm significant diagnostic features which can identify copying, and identify the level of copying considered unacceptable (as some overlap in lab reports is expected). 
2. Survey of what is available and how well this matches the requirements specification 
Unlike other surveys of generic plagiarism detection systems, we aimed to evaluate systems against the detailed specific requirements identified above. A number of candidate systems are available, for example: 
- (CheatChecker 2002) - an n-gram based tool; 
- (SIM 2002) and (YAP 2002) based on longest common sub-sequence approach; 
- detecting ‘unusual’ similarities between a closed set of essays (or programs) is called collusion and is the basis for (Copycatch 2002); 
- (Copyfind 2002) is an example of a system aimed at scientific rather than humanities documents; 
- (Turnitin 2002) is probably the most widely-known system, but the website gives scant details of the underlying methods; 
- (Clough 2002) uses more sophisticated techniques from Natural Language Processing. 
We decided to develop two ‘simple’ copying-detection solutions to assess; and compared these against ‘smarter’ commercial-strength systems. 
48
3. Collating a test corpus of Biomedical Science student reports 
Evaluation using a test corpus is a well-established methodology in Corpus Linguistics research, and is appropriate to this task: in our case, the aim is to test candidate systems and compare detection (and ‘red-herring') rates. 
3.1 sources: first and second year coursework exercises 
The initial test corpus used in our study was composed of 103 Biomedical Science first-year student reports. At the outset we did not know whether this included real cases of copying, or if so how many; none had been detected by teaching staff. We found 2 pairs of reports were almost identical, and called these Albert1, Albert2 and Barbara1, Barbara2 (all filenames are randomly generated and are not the real names of the students). 
As there turned out to be few examples of `real' plagiarism in our initial corpus, we extended it by adding 94 reports written by second year students, covering 3 more challenging, less constrained tasks; during subsequent tests, we discovered 4 pairs which appeared to involve copying: Geraldine2, Fred2; Jane2, Naomi2; Colin2, David2; and Jane3, Naomi3. There appeared to be no other clear cases of copying. 
To ensure our corpus included samples which we knew were definitely copied, we also added an example of first-year plagiarism that had been detected previously by Biomedical Science staff, which we called Scanfile1, Scanfile2; we were given the hardcopy of two reports to scan using an Optical Character Recognition tool. 
3.2 adding artificial TestFiles by merging parts of existing reports 
To check the sensitivity of each plagiarism detection system tested, we created 19 new TestFiles: each composed of one file from which 5%(then 10%,..., 95%) was removed and replaced by a portion of another file. We also created 2 extra TestFiles: TestFileAB made up of half SourceA and half SourceB; and TestFileABC, made up of TestFileAB with added text from SourceC from a different genre (en email message). 
This gave a final corpus of 220 student laboratory reports including artificial TestFiles (which were still “authentic corpus material” as they were composed of parts of real reports). 
3.3 Format and content of the laboratory reports 
The first year assignment described the experiment and gave instructions on how to use a computer simulation of the laboratory apparatus to generate results; then the students had to produce tables and graphs and answer a few questions to analyse the results. 
It seemed likely that any plagiarism we would find would be in the text of the answers. Unlike the first year student reports, the second year students did not have to answer specific questions; so the text should be more `free' and show more originality. However, they still follow the same generic structure of all science laboratory reports: Introduction, Methods, Results, Discussion. 
A characteristic of this genre is that a lot of the vocabulary from the specific science domain will be shared between all the reports. 
The answers were quite short (generally about 25 words, occasionally up to 100 words or more), so there is not much scope for originality from each student. As most of the plagiarism checkers available at the moment appear to be developed to detect plagiarism either in humanities essays (high level of originality) or in programming (with limited vocabulary), there is no existing copying checker for the specific problem of scientific reports: low level of originality but not a limited vocabulary. 
49
4. Testing and Evaluation of a range of candidate copying-detection systems 
We used our Corpus to test contrasting approaches to detection. We developed two "simple" programs: Zipping, based on a standard file-compression tool, and Bigrams, a basic comparison of frequent bigrams; and we tested "smart" methods using commercial-strength plagiarism-checking systems Turnitin, Copycatch, and Copyfind. Our findings are reported in detail in (Medori et al 2002). 
4.1 Zipping, a simple compression-based approach 
As a baseline of our study, we tested the idea that using file compressors or zippers, we would be able to measure a `distance' between files according to their similarities. 
The algorithm of common zippers (like gzip) finds duplicated strings and marks up the matching strings with a pointer; which means that, by appending a questioned text to different corpora (e.g. of different authors or languages) and measuring the difficulty of zipping these files, we can measure the level of similarity between two files. The smaller the zipped file of the concatenation, the more common elements the questioned text shares with the corpus. 
4.2 Another 'simple' approach: Bigrams 
This method is another `home-grown' program based on the idea that when 2 files are similar, they should share a high number of their bigrams. The programs were all implemented in Perl. 
We tested this method first with character bigrams and then, word bigrams. In the first place, we created a model for each file in our corpus by counting all the occurrences of each character bigrams in the file. To compare two files, we simply checked whether the most frequent bigrams were the same in both files. We compared the results for a model containing 10, 20, or 30 most frequent bigrams. Note that this comparison did not take account of the frequencies of the top 10/20/30 bigrams, it just compared members of the two lists. 
The results even for "top 30" character bigrams were disappointing: this method will flag copying of an entire document by highlighting documents sharing the same most frequent 30 letter-pairs, but below a total match, most files share a large proportion of character bigrams. This may be at least in part because most laboratory reports will have similar style and content, so need a somewhat more sophisticated discriminator to go beyond finding near-identical documents. 
We tried the same experiment with word-bigrams, and got better results: word-bigrams are better discriminators. 
4.3 Turnitin: a commercial WWW-based system 
The first commercial system we used was Turnitin.com, a web-based plagiarism detection service first designed to detect material cut and pasted from the Internet but also detects similarities between submitted papers; Leeds University has subscribed to the service. 
The submission of papers is done by means of laborious cut-and-paste of the text from each document. 
After about 24 hours, an “Originality Report” is sent onto the user's account for each submitted paper. 
We found that TurnItIn gives quite good results. However, the fact that it gives an index of similarity between the file and all papers found on the web and in the database leads it to miss a few cases of plagiarism, especially in the case of the pair `Colin-David'. 
50
Turnitin gives as a result the level of similarity between the submitted paper and all the papers in the database and the data found on the web; it seems a waste of time checking on the web and other papers than the ones in the class. 
A more significant shortcoming is the laborious input method. 
On the positive side, Turnitin underlines the similarities, making them easier to visualise; and it makes it easier to detect partial plagiarism. 
4.4 CopyCatch: a commercial PC-based system 
These conclusions led to the next step of our study, to test CopyCatch, a software developed by Woolls from CFL Software Development. It seemed that it might be more appropriate to our specific needs as it gives instant results, sorting them by percentage of similarity between 2 files. 
The main distinguishing characteristic of this software was that it was much easier to use, especially the submission process: it allows the user to browse and select the files to check. The results are almost immediate. It outputs a list of pairs of files sorted by percentage of match between them. 
It is then possible to have a list of the vocabulary or phrases shared between the 2 files and it can also mark up the files highlighting the similarities between them. It allows the user to check Word documents as well as text, rtf and HTML files: no previous conversion is needed. 
CopyCatch found all our `real' cases of plagiarism. It even found the pair `Colin-David', which was not an obvious case of plagiarism and which we missed using TurnItIn. 
The selection of our files in a few clicks is a very important practical advantage. 
4.5 CopyFind: a university-developed freeware alternative 
As another alternative, we decided to test another CopyFind, available for Linux as well as for Windows. This software requires that the Copyfind executable and all the files to be checked are in the same folder. The user has to add a list of all the files that have to be checked. These can be Word documents. The user can define a threshold of similarity. The system will then output a file containing all the pairs of files and the number of matched words for each of them. 
An HTML report will then be created for every pair which are above the threshold, with underlined similarities. 
We ran the program with standard parameter values; the results were quite good, but it didn't detect the pair `TestFile-TestFile2'. To find this pair in the results, we had to lower the threshold by changing the minimum number of matching words to report from 500 (value suggested) to 300. The modification of this threshold didn't change any of our other results. 
CopyFind is a program that works quite well and gives a clear output underlining the similarities between the files. The results are immediate, but its submission method is cumbersome. 
5. Discussion 
We found little difference in detection rates between "simple" and smart" approaches, and in practice the main performance discriminator was ease of use rather than accuracy. We found this surprising, given the naivety of the "simple" models: Zipping found copying as a side-effect of file compression; and Bigrams did not compare frequencies, each document is simply characterised by a vector of its 50 most-used word-pairs. These minimal models were enough to flag all known real cases of plagiarism in our corpus. 
51
Turnitin gave interesting results too, as it was the only system to flag the overlap between our two artificially-constructed test files (TestFileAB, TestFileABC) and their sources (SourceA, SourceB). However, it didn't successfully detect the pair Colin, David. As this plagiarism is more representative of how students really plagiarise, this failure outweighs the success with artificial TestFiles. 
In the tests, all approaches successfully flagged examples of ``blatant'' copying, but results were less clear-cut when reports had small amounts of overlapping text. The other top results were mainly pairs of small but unrelated files. This could be because of the nature of the assignment: the students often rewrite the objectives of the experiment, copy the questions, or answer them rephrasing the question in a negative or affirmative sentence. This could explain why the reports where the students did not develop the answers are found first. 
This also suggests that we could try to take account of knowledge specific to our domain. In Biomedical Science laboratory reports, students must always write four sections: Introduction, Method, Results, Discussion. In fact this is standard practice even for research journal papers; and this format is a mandatory part of the course work specification. Biomedical Science lecturers know that certain parts of the text can legitimately be copied (the given specification of Method, and specific questions set by the lecturer), whereas originality is more important in other parts (notably the Discussion section). Perhaps a copying-checker should include some ``intelligence'', to make it ignore overlap in unimportant sections, and focus on the Discussion section; overlap in Discussion is a much surer indicator of copying. 
On the other hand, perhaps we do not need such "intelligence". Research into students' reasons for copying suggests often the reason is either last-minute panic to meet a submission deadline, or else laziness or unwillingness to devote the time needed to do the work. In these circumstances, students generally make blatant copies rather than intelligently disguised copying through time-consuming detailed modifications. Our "simple" approaches suffice for these cases. 
Anecdotal evidence from Biomedical Science teaching staff suggests that if cheaters modify the copied document to try to avoid detection, they tend to change just the first page (on the grounds that this is all that will be seen in a cursory skim through 200+ reports for blatant copying). A "simple" answer could be to check for blatant copying but ignore the first page; this would incidentally upgrade the significance of the Discussion section as it comes last. 
6. Conclusions 
We have analysed the requirements of our specific problem, through discussion with Biomedical Science teaching staff. We have surveyed a range of candidate systems, and evaluated five systems (Turnitin, Copycatch, Copyfind, and two "home-grown" systems, Zipping and Bigrams) in experiments with a Test Corpus of Biomedical Science student laboratory reports. We concluded that none of the systems stood clearly above the rest in their basic detection ability, but that Copycatch seemed marginally more successful, and more significantly it had the most appropriate user interface for our needs as it ran on a local PC and simplified the analysis of a whole directory of lab reports. 
Following personal contact with Copycatch developers, we agreed to be the first beta-test site for the enhanced Java version. We have installed Copycatch on Biomedical Science computing network for future use by staff. 
A side-effect of availability and use copy- checking software is the need to migrate from paper-based report submission and processing to electronic submission, probably via student pigeon-holes in the “Nathan Bodington” virtual building, an e-learning environment set up by the Flexible Learning Development Unit for general teaching use at Leeds University. The Flexible Learning Development Unit has accepted our recommendations (Atwell et al 2002). The School of Biomedical Sciences has decided to require electronic submission of all laboratory reports in future, to facilitate use of copying-detection software. 
During tests, we found evidence to support suspected cases of copying, and discovered at least one new (previously unknown) case of student copying, which was reported to the appropriate authority. 
52
A growing array of tools have appeared for detection of plagiarism and copying; some authorities such JISC in Britain advocate standardised adoption of a generic centralised web-based system. We conclude that our task is an exception meriting a specialised corpus-based study and tailored solution; and that such language analysis tools can be a catalyst for reform in assessment practices. 

7. References 
Atwell, Eric; Gent, Paul; Souter, Clive; Medori, Julia. 2002. Final Report of the C&IT in the 
Curriculum project: Customising a copying-identifier for Biomedical Science student reports, 
School of Computing, University of Leeds. 
http://www.comp.leeds.ac.uk/eric/citFinalReport.txt, 
or http://www.comp.leeds.ac.uk/eric/citFinalReport.doc 
CheatChecker. 2002. http://www.cse.ucsc.edu/~elm/Software/CheatChecker/ 
Clough, P. 2002. http://www.dcs.shef.ac.uk/~clough 

CopyCatch. 2002. http://www.copycatch.freeserve.co.uk 
CopyFind. 2002. http://plagiarism.phys.virginia.edu/software.html 
Medori, Julia; Atwell, Eric; Gent, Paul; Souter, Clive. 2002. Customising a copying-identifier 
for biomedical science student reports: comparing simple and smart analyses in: O'Neill, M, 
Sutcliffe, R, Ryan, C, Eaton, M, & Griffith, N (editors) Artificial Intelligence and Cognitive 
Science, Proceedings of AICS02, pp. 228-233 Springer-Verlag. 
SIM. 2002. http://www.few.vu.nl/~dick/sim.html 
Turnitin. 2002. http://www.turnitin.com/ 
YAP. 2002. http://www.cs.su.oz.au/~michaelw/YAP.html 
53

A Corpus of Sworn Translations – for linguistic and historical research 
Francis Henrik Aubert 
Stella E. O. Tagnin 
University of Sao Paulo 
1. Introduction 
In Brazil, all and any documents and papers, when in a foreign language, need to be translated by public sworn translators if they are to be used for any official purposes. Such documents will normally range from school papers needed for a student's transfer from one country to another, birth, marriage or death certificates for naturalization, marriage or inheritance purposes, up to contracts, powers-of-attorney, promissory notes and articles of incorporation or other commercial documents for international transactions. In principle however, any text may be submitted to a “sworn” translation procedure if such text, for any reason, is to be processed by government authorities, at any level, or by the Courts. Thus, for instance, a love letter may well have to be translated by a public sworn translator if such letter is used as evidence in a divorce suit. A translation of a play may give rise to a suit for copyright infringement, and, as a consequence, even a literary text may have to be dealt with by a sworn translator, back-translating it into the initial source language, as documentary evidence in the records of the suit. (See also Aubert 1996). 
Public translators must register these translations (i.e., true and unabridged copies of each translation delivered to the clients) in a special ‘book of records', conforming to a certain number of rules. Upon their retirement or death, these books are turned in to the Board of Trade of the state in and for which they are qualified (the Junta Comercial). 
The Board of Trade of the State of Sao Paulo (JUCESP) has approached the University of Sao Paulo, where a large corpus for teaching and translation purposes is being built – the COMET – offering this material for inclusion in the corpus. It covers a period of over one hundred years and is written in more than 20 languages. However, this offer is not without problems: a large part of the material will require a long and strenuous work of restoration, which, in turn will demand substantial fundings. 
To bring the project off the ground in a relatively short period of time, it has been decided to first concentrate our efforts on the books of the last thirty years with translations into and out of Portuguese, English, German and Spanish, the languages presently addressed by the COMET. 
This material will not only allow for research in linguistic and translation matters, such as, contrastive stylistics, lexicography, legal terminology, translation norms and translationese, but also in studies of a more historical nature, especially Brazil's immigration waves, periods in which there is a high demand for public translations to comply with the legal aspects of immigration. We can envisage historians and sociologists as interested researchers. 
The article will discuss the relevance of this unique material, the design and structure of the corpus, its population, the audience it is aimed at, the problems expected in the preparation of the texts, the header, and possible research areas. 
2. The COMET project 
This project grew out of two experiments in teaching English into Portuguese translation to students at the Specialization in Translation course at the University of Sao Paulo (Tagnin 2002a; 2003b to 
54
appear). The students built small bilingual corpora ranging between 100,000 and 200,000 words in each language in several technical areas1 over a period of two years, 2000-2001. This material was put together on a CD-ROM and made available for their terminological research. The resulting glossaries are available at http://www.fflch.usp.br/citrat As other corpora were being built to inform masters and doctoral studies, it was decided to bring all this material together under a common project: COMET – A Multilingual Corpus for Teaching and Translation. The COMET consists of three subcorpora: a Technical Corpus, a Learner Corpus and a Translation Corpus. The Technical Corpus favours mainly three areas in which a significant lack of terminological sources has been identified by professional translators: Commercial Law, Computing and Orthodontics. This means that regular work is being carried on to enlarge these corpora systematically. Nevertheless, all technical corpora produced by student work or otherwise at the University of Sao Paulo (USP) will eventually be hosted here.2 
As corpus work became more evident at USP, we joined the Br-Icle project, which was being conducted by Tony Berber Sardinha at the Catholic University of Sao Paulo. This was brought to the attention of other scholars interested in language teaching/acquisition and it was subsequently decided to construct a multilingual learner corpus at USP as the Department of Modern Languages is comprised of five different areas: English, French, German, Italian and Spanish. At the moment of writing, English, German and Spanish have teamed up to pursue this project (for more details see Tagnin 2003a). 
The Translation Corpus initially consisted of student literary translations: 9 American short stories and 20 Canadian short stories. The latter have been published in book format, along with a bio-bibliography of each author (Tagnin 2002b). 
To these will be added parallel (original/translation) literary texts collected in the course of several theses and dissertations in translatoloty, including short stories by Edgar Allan Poe (for which several renderings into Brazilian and European Portuguese are available), the translation into English of the classic “Os Sertoes” (“Rebellion in the Backlands”), by Euclides da Cunha, to mention but a few. 
Inclusion of uncorrected student production has been considered so as to form a Translation Learner Corpus to function as a source for research in the pedagogy and practice of translation. 
As mentioned above, JUCESP has approached USP to donate the books of records of deceased and retired sworn translators to be included in COMET's Translation subcorpus. The sections below will provide the details of the project. 
3. The JUCESP subproject 
The corpus is meant for research primarily in the fields of legal translation, lexicology and terminology. It will initially be fed with JUCESP texts translated from English, German and Spanish into Portuguese or out of Portuguese into any of these foreign languages. 
The audience envisaged is both translation students and teachers, lexicographers and terminologists. However, as all texts will be included in their full form they will also make valuable material for researchers interested in discourse analysis. 
1 The areas in which these corpora were built are: Biotechnology: transgenic foods; Cooking: Spices; Computing: Security; Fashion: Clothes; Veterinary: Bovine diseases; Ecology: Biodiversity; Dentistry: Orthodontics; Automation: Safety Locks; Business: Brazilian Financial Market; Tourism: Ecotourism; Genetic Engineering: Genoma. 
2 Part of this corpus will be fed into the Lácio-Web Project (Aluísio et al., 2003). In exchange, all corpus tools developed within that project are being made available to the COMET Project. 
55
As explained below, parallel texts (originals and their translations) will be rare so that the corpus will be above all a comparable one. It will also be an open-ended corpus as new material may be inserted when new material is processed or more books of records are made available due to the death or retirement of other sworn translators. 
A relevant feature is that the material is of public domain so that no copyright restrictions apply. 
The sworn translations on file at the Board of Trade are organized in ‘books of records', identified by the name of the sworn translator and by the foreign language from or into which the translation was carried out. Thus, e.g., a sworn translator qualified for English, French and Spanish, will have three series of volumes (of up to 400 pages each), one for each foreign language. If a given original contains textual material in more than one foreign language, the translation will normally be recorded in a volume corresponding to the prevailing language in the original text, or, alternatively, in the volume corresponding to the official language, if any, of the country in which the original document was issued.3 
The major foreign languages represented (as source or as target languages) in the material are English, Spanish, French, Italian and German (roughly in this order). A certain amount of material is also to be found for Arabic, Dutch, Greek, Hebrew, Hungarian, Japanese, Korean, Latin, Norwegian and Russian, though for these languages the actual volume has yet to be assessed. 
3.1 Conversion of the texts into electronic format 
One initial major difficulty will be the conversion of the material into electronic format. Up to the early 50's, most translators copied the translations into the book of records by hand. From the mid 50's and up to the early 70's, the copies found are mostly carbon copies. Photostatic reproduction became common only in the 1980's, and, even here, the quality of the copies is not always such that it will permit electronic scanning to be performed without involving an unreasonable amount of revision. It is therefore expected that, to a very large extent, the material will have to be transcribed, an operation which – as one knows from the times of Medieval copyists – is fraught with the risks of errors, slips of the finger, lapses and the conscious or subconscious desire to “improve”. The fact that, over the relevant period (1902-2002), Brazilian Portuguese has undergone two major orthographical reforms, does not render the task any easier. Nevertheless, the variety of the material and potential rewards for research are such that the effort involved (including strict supervision of the electronic transcripts) is felt to be well worth the while. 
Due to the volume of material donated, it will be processed in batches: 
1. texts covering the last 30 years: 1972-2002; 
2. texts covering the period 1935-1971; 
3. texts covering the period 1902-1934. 
3 The linguistically hybrid texts are indeed fairly common. A daughter company of a US holding, organized in the Grand-Duchy of Luxemburg, will have its Articles of Association drawn up in English, followed by a French version, and with its notarization also in French, unless one of the signatories is resident in Italy, in which case one of the notary public acknowledgments may be worded in Italian. In a more extreme case, a bill of lading was found to have been printed in English, its different boxes filled out in a blend of Portuguese and Spanish (in all likelihood, in a not very successful attempt to produce Portuguese); the rubber stamp of the carrier was worded in German, the address of the shipping company appeared in Norwegian, but the notarization had been conducted in the Canton of Ticino, Switzerland, and was thus formulated in Italian. Since the basic text (the starting point) was in English, this translation was inserted in the book of records for the English language of the relevant translator. 
56
As soon as a substantial amount of the first batch has been processed, it will be made available on the Web as a pilot corpus. It will then be updated regularly as new texts are prepared and ready to be read electronically. 
3.2 Text types 
Although any text can, in a given situation, be subjected to a “sworn translation” procedure (see Introduction), the major part of the translations available in the material can be divided into four main groups: (a) personal documents (identity documents, birth, marriage, divorce and death certificates, school documents, and the like); (b) corporate documents (articles of association or incorporation, corporate deliberations, minutes of shareholder meetings, secretary's certifications, etc.); (c) financial documents (bills of lading, agreements in general – purchase and sale, rental, licensing of trademarks, technology transfers –, promissory notes and other securities); and (d) legal documents (petitions, letters rogatory, court decisions). Obviously, groups (b), (c) and (d) intersect, to a large extent, in terminology and phraseology, although their specific purposes and the actors involved in the respective communicative processes are somewhat different. Texts dealing with matters directly related to industrial technology are relatively rare, save for patent registrations and exhibits to technology transfer agreements, but even these contain, to a greater or lesser extent, terms and phrasings which pertain to the legal and commercial specialty languages. 
3.3 The structure of the corpus 
Because of the comparative and contrastive studies envisaged, structuring of the material by language will take precedence. The next level will be by date and then by text type. For instance: 
English 
2002 
Personal Documents 
Birth, Death, Marriage and Divorce Certificates 
Transcripts 
Financial Documents 
Balance Sheets 
Bills of Lading 
Promissory Notes 
Corporate Documents 
Articles of Incorporation/Association 
Minutes of Shareholder and Board Meetings 
Legal Documents 
Petitions, pleas and similar acts of bringing suit 
Powers of Attorney etc 
German 
2002 
Personal Documents 
Birth, Death, Marriage and Divorce Certificates 
Transcripts 
Financial Documents 
Balance Sheets 
Bills of Lading 
Promissory Notes 
Corporate Documents 
Articles of Incorporation/Association 
Minutes of Shareholder and Board Meetings 
Legal Documents 
Petitions, pleas and similar acts of bringing suit 
Powers of Attorney etc 
57
3.4 Header 
Each text will be identified by a code indicating language direction, date, text type and text number. Thus GP902BC0001 means: German into Portuguese, 1902, Birth Certificate, text nr. 00001 in that category. This will allow up to 99,999 texts in each category. The potential total will probably never be reached for most categories, but if we are to plan ahead and hope to process most of the material, we may well come close to that number for at least a few categories, especially in English, which is, by far, the most prevalent source language. 
A list will be drawn up of all text types and their corresponding abbreviations, which will be part of the identifying code of each text (see next section). 
A header, based on the international standards devised for the Translational English Corpus (TEC)4, developed at the UMIST (University of Manchester Institute of Science and Technology), will provide more detailed information as to the direction of translation (into mother tongue, into foreign language), language of source text, language of target text, the date of the translation, the name of the translator, his/her sex and nationality (if available), the text type, the extent of the text, and the subject. 
This procedure requires careful analysis of the material so that the header is completed correctly as the data therein will serve as pointers for selecting the texts to be submitted to research with the aid of electronic search tools, i.e., the texts will be searchable by language, by text type, by date, by translator, by subject etc. 
3.5 A brief analysis of the JUCESP texts 
Legally speaking, sworn translations are not “independent” texts. A sworn translation is not used officially in Brazil in lieu of the original; rather, it ensures that the original text5 may be put to official use, and the original and the translation are therefore jointly submitted to the public office, agency, court or other institution (schools and academies, banks, insurance companies, traffic departments, etc.) to which they are destined. This, in principle, suggests that sworn translations will tend to adhere more closely to the original texts, that, in a sense, they will be more “literal” than common unofficial translations. This, indeed, is – supposedly – a defining feature of sworn translations in general: the intent of such translations is to assist the recipient in understanding the text within its source cultural setting, and not to propose solutions which would have been appropriate if the text had been originally produced in the target language setting. In Venuti's (1995) terms, sworn translations adopt – or ought to adopt – translation strategies or procedures which are “foreignizing” rather than domesticating. The material to be included in the corpus will, it is expected, afford the possibility of testing this hypothesis, and, more specifically, assist in identifying if and to which extent other factors beyond the “sworn translation mode” as such tend to stimulate or check the “foreignization rule” (e.g. text typology, translational direction – from/to Brazilian Portuguese –, subject matter, etc.). 
But here the material available presents a specific problem. Although the sworn translators must retain a full copy of each translation they have produced in their official capacity, there is no equivalent requirement concerning copies of the original texts. Thus, the books of records which will be fed into the COMET project will not provide a strict parallel corpus, and the original texts can only be inferred from the translations 
This limitation will, at times, pose a problem in that it will not allow a comparison with the original. An odd passage in the translation might, of course, indicate an error or mistake of the translator. 
4 www2.umist.ac.uk/ctis/research/TEC/tec_home_page.htm 
5 or a certified copy thereof, but never a non-certified copy or other reproduction (facsimile, electronic printout, etc.). 
58
Alternatively, however, it could be indicative of an attempt to reproduce an oddity existing in the original, given that a sworn translation is supposed to “mirror” the original, not to improve on it (although, not infrequently, the original texts would probably have benefited from such improvements) (see Aubert, 1996, op.cit). 
3.6 Research possibilities 
Despite the limitations referred to in the preceding paragraphs, the material is expected to provide a wide range of information relevant to linguistic and historical research. 
A cursory review of the material seems to indicate that approx. 3 of the translations have been made into Portuguese, the remaining 1 corresponding to translations from Portuguese and into the relevant foreign languages. Despite the marked difference in distribution, the sheer quantity of the material available (some 3,000 volumes – i.e., approx. 1.200.000 pages – covering the entire 20th century) is expected to provide sufficient textual samples for a thorough linguistic, stylistic, translational, intercultural and historical investigation in both translation directions (from/into Portuguese), and afford relevant comparisons. 
For terminological purposes, the corpus of sworn translations is expected to provide a vast range of real-life translational situations, disclosing both the underlying strategies and the actual solutions provided. The fact that different source and target languages are involved will also afford a reasonable degree of comparison, and the possibility of verifying to what extent the specific source or target language exerts, as such, any influence on the terminological options made by the translators. 
The terminological studies afforded by the corpus will not only derive from the actual translated texts. Some sworn translators have, in the course of their careers, set up their own personal glossaries, and one such glossary (containing close to 2,000 terms) has already been made available to the Centre for Translation and Terminology at USP, by the heirs of a recently deceased sworn translator for the German language. The textual material to be included in the COMET will serve the purpose of validating, reviewing and/or varying the solutions proposed, by comparing the glossary with the actual usages and corresponding contexts found in the said translator's recorded sworn translations. Under a subsequent stage, the terminological solutions validated for the production of this specific translator can be compared to those found in the records of other translators as well as those proposed/recommended by Chambers of Commerce, monolingual glossaries in German and Brazilian Portuguese, and so on. 
Much the same expectations are relevant to phraseology, although, since phraseology is intimately related to stylistics and to the idiomatic features of each language, one might expect that the subcorpus of translations into the foreign languages will afford a safer ground for comparative analysis than translations into Brazilian Portuguese from several different foreign languages. 
Consider, for instance, the more or less set phrase which usually closes the preamble to a standard Brazilian contract: “As partes qualificadas supra tem entre si justo e acordado o que segue” (discursively equivalent to “Now therefore, in view of the promises and mutual undertakings and obligations contained herein, the parties agree to be bound by the following terms and conditions”). Although the actual original texts will not be available, the predictability of this set phrase is such that it can be readily inferred in the several translations to be analysed. It can be expected to reappear in a number of different translations, into a variety of languages, with varying solutions, and will thereby provide a basis for a typical “stylistique comparée” investigation, much as originally conceived by Vinay and Darbelnet (1958). 
59
In the absence of the corresponding original texts, a direct observation of the translation procedures involved cannot be conducted in any systematic fashion6. Yet, the observation of the lexical and syntactical structures and their frequencies in the translated material, as compared to their corresponding frequencies in authentic original texts and in common (i.e. “not-sworn”) translations is expected to unveil the degree of structural “contamination” (or literal shift) of sworn translations from and into Brazilian Portuguese. The from/to distinction is here of a certain relevance. In fact, one of the initial hypotheses is that the translational strategies will not be the same, but will vary according to the translational direction.7 Also of a certain interest is the observation of translational solutions which become standard “translationese” for translations from Brazilian Portuguese into foreign languages, as markers of Brazilian cultural, legal and institutional specificities, e.g. the widespread use of “quotaholder” as a translation for “quotista” (a reference to the shareholder of a Brazilian limited liability company, legally termed “sociedade por quotas de responsabilidade limitada”) or of the official name of the country as “Federative Republic of Brazil” (derived from “República Federativa do Brasil”, although “Federal Republic of Brazil” would probably be a more idiomatically adequate solution in English).8 
As already mentioned, the first stage of the corpus will cover a period corresponding to the last thirty years (1972/2002). At later stages, the books of records of sworn translators from earlier times will also be included, covering a full century. Even the initial stage, however, will provide sufficient elements for diachronic investigations; probably not in terms of linguistic structure, but most certainly in terminology and, very possibly, in translation procedures and strategies. Here, a number of relevant extralinguistic factors are likely to have exerted marked influences: the redemocratization process, after a long period of military rule, culminating with the 1988 Federal Constitution (and the new institutions and legal concepts arising therefrom); the first free presidential elections in a generation, in 1989; the opening of Brazilian economy as from 1990; the intensification of the commercial and cultural exchanges with other Latin American countries, specially within the framework of the Mercosur; the growing assertion of intellectual property rights by actions in court or otherwise (which, in turn, tend to require that translations be more closely tied to the original texts, so as to avoid the risk of infringing on such property rights by the intromission of a more explicit – and uncalled for – co-authorship); and, obviously, the translators’ move from the typewriter to the personal computer as a tool for writing, editing and proofreading, for terminological research, and for exploiting new strategies, including intersemiotic translation. If these factors are indeed relevant to the production of public sworn translators, one might reasonably expect to observe their reflections and refractions on the translated material, by comparing the translations produced in the mid-70's with those produced in the second half of the 90's. 
The preceding considerations point to other research possibilities beyond the realm of language studies as such. The close connection between public sworn translation and the political, institutional, 
6 Occasionally, sworn translation clients request that the original text be set up together with the actual translation, in two parallel columns. Also, a significant volume of sworn translations involve standardized original texts (e.g. passports, driving licenses), in which the wording is basically the same, save only for the personal data of the actual bearer. Here, it will suffice to have access to one such original standardized text in each language in order to conduct direct observation of the translation procedures applied by the different translators. 
7 In her doctoral dissertation, Sonia T. Gehring (1998), working with a different text and translation typology (social sciences), provides statistical evidence that translations from English into Brazilian Portuguese are in a sense more “literal” (or “foreignizing”), whilst equivalent texts translated from Brazilian Portuguese into English tend to be “freer” (or more “domesticated”). 
8 Evidently, the same concern holds good for the opposite translational direction. It is interesting to observe that a US “county” is usually translated as “condado”, although the Brazilian institutional system has a reasonably close correspondent in “comarca”. And, for reasons unknown and which would bear further investigation, the Brazilian consulates abroad, when in the US or Canada, tend to identify “notary publics” as “notários” but, elsewhere (including the UK), as “tabeliaes”. 
60
economic and legal spheres suggests that material covering a full century will also bear marks of the historical processes that the target community of these translations has undergone: the initial republican regime, after the abolition of monarchy in 1889 and the subsequent dictatorships, alternating with civilian rule; the two world wars and Brazil's participation therein; the several waves of immigration, specially from Spain, Portugal, Italy, Lebanon, Japan and, more recently, from Korea; the budding industrialization of the country and the shift from a rural to an urban society, accelerated as from the mid-1950's; the ups and downs of the economy. At this point, the project branches out to a promising inter- and transdisciplinary co-operation with historians, sociologists and anthropologists. 
4. Conclusion 
This paper has reported on the creation of a Multilingual Corpus of Sworn Translations (MCST) at the University of Sao Paulo, Brazil. 
The MCST will initially consist of sworn translations into Portuguese and out of English, German and Spanish, as well as translations out of Portuguese into these foreign languages, covering a 30-year period (1972-2002), extracted from the complete works of deceased and retired sworn translators in the state of Sao Paulo over a period of one hundred years (1902-2002). 
Due to the unique character of this material it is believed that even the “small” part currently under consideration will offer enough material for relevant studies in the areas of translational, lexical, syntactic, terminological, discursive and even historical and sociological research. 
References 
Aluísio S M, Pinheiro G M, Finger M, Nunes M G V, Tagnin S E O 2003 The Lácio-Web Project: overview and issues in Brazilian Portuguese corpora creation. In Proceedings of Corpus Linguistics 2003, Lancaster, UK. 
Aubert F H 1996 Translation Typology: the Case of 'Sworn Translations'. In Coulthard M, De Baubeta P A O (org) Theoretical Issues and Practical Cases in Portuguese-English Translations, Edwin Mellen Press. 
Gehring S T 1998 As modalidades de traduçao ingles/portugues: correlaçoes bidirecionais. Unpublished Doctoral Dissertation. University of Sao Paulo. 
Tagnin S E O (ed.) 2002a Lá do Canadá, Sao Paulo: Olavobrás. 
Tagnin S E O 2002b Corpora and the Innocent Translator: How can they help him. In Thelen M (ed.) Translation and Meaning, Part 6, Proceedings of the Lodz Session of the 3rd Maastricht-Lodz Duo Colloquium on “Translation on Meaning”, Lodz, Poland, September 22-24, 2000, Maastricht: Universitaire Pers Maastricht, 489-496. 
Tagnin S E O 2003a A multilingual learner corpus in Brazil. In the Proceedings of the Workshop on Learner Corpora at Corpus Linguistics 2003, Lancaster. 
Tagnin S E O 2003b Os Corpora: instrumentos de auto-ajuda para o tradutor. To appear in the special issue Translation and Corpora, Cadernos IX - 2002/1, University of Santa Catarina. 
Venuti L. 1995 The translator's invisibility. London, Routledge. 
Vinay J, Darbelnet J P 1958 Stylistique comparée du français et de l'anglais. Paris, Didier. 
61

Constructing Corpora of South Asian Languages 
Paul Baker*, Andrew Hardie*, Tony McEnery*, and Sri B.D. Jayaram° 
* Department of Linguistics, Lancaster University 
° Central Institute of Indian Languages, Mysore 
{j.p.baker, a.hardie, a.mcenery}@lancaster.ac.uk, jayaram@ciil.stpmy.soft.net 
Abstract 
The EMILLE Project (Enabling Minority Language Engineering) was established to construct a 67 million word corpus of South Asian languages. In addition, the project has had to address a number of issues related to establishing a language engineering (LE) environment for South Asian language processing, such as translating 8-bit language data into Unicode and producing a number of basic LE tools. This paper will focus on the corpus construction undertaken on the project and will outline the rationale behind data collection. In doing so a number of issues for South Asian corpus building will be highlighted. 
1 Introduction 
The EMILLE project1 has three main goals: to build corpora of South Asian languages, to extend the GATE LE architecture2 and to develop basic LE tools. The architecture, tools and corpora should be of particular importance to the development of translation systems and translation tools. These systems and tools will, in turn, be of direct use to translators dealing with languages such as Bengali, Hindi and Punjabi both in the UK and internationally (McEnery, Baker and Burnard, 2000). 
This paper discusses progress made towards the first of these goals and considers to a lesser extent the third goal of the project. Readers interested in the second goal of the project are referred to Tablan et al (2002). 
2 Development of the corpora 
This section describes our progress in collecting and annotating the different types of corpora covered by EMILLE. EMILLE was established with the goal of developing written language corpora of at least 9,000,000 words for Bengali, Gujarati, Hindi, Punjabi, Sinhalese, Tamil and Urdu. In addition, for those languages with a UK community large enough to sustain spoken corpus collection (Bengali, Gujarati, Hindi, Punjabi and Urdu), the project aimed to produce spoken corpora of at least 500,000 words per language and 200,000 words of parallel corpus data for each language based on translations from English. At the outset we decided to produce our data in Unicode and annotate the data according to the Corpus Encoding Standard (CES) guidelines. As the project has developed, the initial goals of EMILLE have been refined. In the following subsections we describe the current state of the EMILLE corpora and outline the motives behind the various refinements that have been made to EMILLE's goals. 
2.1 Monolingual written corpora 
The first major challenge facing any corpus builder is the identification of suitable sources of corpus data. Design criteria for large scale written corpora are of little use if no repositories of electronic text can be found with which to economically construct the corpus. This causes problems in corpus building for the languages of South Asia as the availability of electronic texts for these languages is limited. This availability does vary by language, but even at its best it cannot compare with the availability of electronic texts in English or other major European languages. 
1 Funded by the UK EPSRC, project reference GR/N19106. The project commenced in July 2000 and is due to end in September 2003. 
2 Funded by the UK EPSRC, project references GR/K25267 and GR/M31699. 
71
We realised that much of the data which, in principle, we would have liked to include in the corpus existed in paper form only. On EMILLE, it would have been too expensive to pay typists to produce electronic versions of the 63 million words of monolingual written corpus (MWC) data. Even if the initial typing had been affordable, checking the data for errors would have added a further cost, particularly since tools for error correction, such as spell checkers, do not exist for many of the languages studied on EMILLE (Somers, 1998, McEnery and Ostler, 2000). 
Scanning in the text using an optical character recognition (OCR) program is a viable alternative to typing in printed text for languages printed in the Roman alphabet. However, OCR programs for South Asian scripts are still in their infancy (for an example of some early work see Pal and Chaudhuri, 1995) and were not considered stable and robust enough for this project to use gainfully.3 
As part of a pilot project to EMILLE4, we ran a workshop that examined potential sources of electronic data for Indian languages. The workshop identified the Internet as one of the most likely sources of data5. This prediction proved accurate, and we have gathered our MWC corpus from the web on the basis of four, largely pragmatic, criteria: 
1. Data should only be gathered from sources which agreed to the public distribution of the data gathered for research purposes; 
2. Text must be machine readable: we could not afford to manually input tens of millions of words of corpus data; 
3. Each web-site used should be able to yield significant quantities of data: to focus our efforts we excluded small and/or infrequently updated websites from our collection effort; 
4. Text should be gathered in as few encoding formats as possible: as we map all data to Unicode, we wished to limit the development of mapping software needed to achieve this. 
While the first three criteria are somewhat easy to understand and have been discussed elsewhere (Baker et al, 2002) the fourth criteria merits some discussion. Ideally, we would have liked to include texts that already existed in Unicode format in our corpus. However, when we first started to collect data, we were unable to locate documents in the relevant languages in Unicode format6. We found that creators of such documents on the internet typically rely on five methods for publishing texts online: 
• They use online images, usually in GIF or JPEG format. Such texts would need to be keyed in again, making the data of no more use to us than a paper version; 
• They publish the text as a PDF file. Again, this made it almost impossible to acquire the original text in electronic format. We were sometimes able to acquire ASCII text from these documents, but were not able to access the fonts that had been used to render the South Asian scripts. Additionally, the formatting meant that words in texts would often appear in a jumbled order, especially when acquired from PDF documents that contained tables, graphics or two or more columns; 
• They use a specific piece of software in conjunction with a web browser. This was most common with Urdu texts, where a separate program, such as Urdu 98, is often used to handle the display of right-to-left text and the complex rendering of the nasta'liq style of Perso-Arabic script; 
• They use a single downloadable True Type (TTF) 8-bit font. While the text would still need to be converted into Unicode, this form of text was easily collected; 
• They use an embedded font. For reasons of security and user-convenience, some site-developers have started to use OpenType (eot) or TrueDoc (pfr) font technology with their web pages. As with PDF documents, these fonts no longer require users to download a font and save it to his or her PC. However, gaining access to the font is still necessary for conversion to Unicode. Yet gathering such fonts is difficult as they are often protected. We found that owners of websites that used 
3 We wished to produce the corpora in the original scripts and hence avoided Romanised texts altogether. 
4 This project, Minority Language Engineering (MILLE), was funded by the UK EPSRC (Grant number GR/L96400). 
5 While we also considered publishers of books, religious texts, newspapers and magazines as a possible data source, the prevalence of old-fashioned hot-metal printing on the subcontinent made us realise early on that such sources were not likely providers of electronic data. Indeed, a number of publishers expressed an interest in helping us, but none could provide electronic versions of their texts. 
6 To date, the only site we have found that uses Unicode for Indic languages is the BBC's; see for example www.bbc.co.uk/urdu or www.bbc.co.uk/hindi. 
72
embedded fonts were typically unwilling to give those fonts up. Consequently using data from such sites proved to be virtually impossible. 
There are a number of possible reasons for the bewildering variety of formats and fonts needed to view South Asian scripts on the web. For example, many news companies who publish web pages in these scripts use in-house fonts or other unique rendering systems, possibly to protect their data from being used elsewhere, or sometimes to provide additional characters or logos that are not part of ISCII. However, the obvious explanation for the lack of Unicode data is that, to date, there have been few Unicode-compliant word-processors available. Similarly, until the advent of Windows 2000, operating systems capable of successfully rendering Unicode text in the relevant scripts were not in widespread use. Even where a producer of data had access to a Unicode word-processing/web-authoring system they would have been unwise to use it, as the readers on the web were unlikely to be using a web browser which could successfully read Unicode and render the scripts. 
Given the complexities of collecting this data, we chose to collect text from South Asian language websites that offered a single downloadable 8-bit TTF font. Unlike fonts that encode English, such as Times New Roman as opposed to Courier, fonts for South Asian languages are not merely repositories of a particular style of character rendering. They represent a range of incompatible glyph encodings. In different English fonts, the hexadecimal code 42 is always used to represent the character “B”. However, in various fonts which allow one to write in Devanagari script (used for Hindi among other languages), the hexadecimal code 42 could represent a number of possible characters and/or glyphs. While ISCII (Bureau of Indian Standards, 1991) has tried to impose a level of standardisation on 8-bit electronic encodings of Indian writing systems, almost all of the TTF 8-bit fonts have incompatible glyph encodings (McEnery and Ostler, 2000). ISCII is ignored by South Asian TTF font developers and is hence largely absent from the web. To complicate matters further, the various 8-bit encodings have different ways of rendering diacritics, conjunct forms and half-form characters. For example, the Hindi font used for the online newspaper Ranchi Express tends only to encode half-forms of Devanagari, and a full character is created by combining two of these forms together. For example, to produce he (Unicode character U+092A) in this font, two keystrokes would need to be entered (h + e). However, other fonts use a single keystroke to produce he. 
We were mindful that for every additional source of data using a new encoding that we wished to include in our corpus, an additional conversion code page would have to be written in order to convert that corpus data to the Unicode standard. This issue, combined with the scarcity of electronic texts, meant that we didn't use as many sources of data as we would have initially liked. Thus we had to focus almost exclusively on newspaper material7. However, as noted in the following paragraph, as a consequence of the collaboration between Lancaster University and the Central Institute of Indian Languages (CIIL), the eventual corpus will now contain a wider range of genres. 
Web data gathered on the basis of these four criteria would have allowed us to fulfil our original MWC project goals. However, the MWC collection goals of the project have altered significantly. Thanks to a series of grants from the UK EPSRC8 the EMILLE project has been able to establish a dialogue with a number of centres of corpus building and language engineering research in South Asia. As a consequence, the EMILLE team has joined with the CIIL in Mysore, India, to produce a wider range of monolingual written corpora than originally envisaged on the EMILLE project. One effect of this change is that the uniform word counts of the monolingual written corpora will be lost.9 Each language will now be provided with varying amounts of data, though no language will be furnished with less than two million words. However, there is a further important effect of this collaboration: the corpus will now be able to cover a much wider range of languages (14 rather than 7) and a wider range of genres. By a process of serendipity, the corpus data being provided by CIIL covers a number of genres, but not newspaper material.10 As the material gathered at Lancaster focuses almost exclusively on 
7 One important exception to this is the incorporation of the Sikh holy text, the Adi Granth or Guru Granth Sahib, into the Punjabi corpus. 
8 Grants GR/M70735, GR/N28542 and GR/R42429/01. 
9 This change was also necessitated by the varying availability of suitable newspaper websites for the different languages. For Hindi and Tamil, for example, plenty of data is available to be gathered; for Punjabi and Bengali, somewhat less; for Urdu, almost none. 
10 The data provided by CIIL to the project covers a number of genres, including Ayurvedic medicine, novels and scientific writing. 
73
newspapers, the CIIL and Lancaster data is complementary. Table 1 shows the state of the EMILLE/CIIL monolingual written corpora at present, and the revised target corpus size. 
Language Target word count (millions) Current word count (millions) Assamese 2.6 2.6 Bengali 9.0 5.5 Gujarati 10.6 10.6 Hindi 12.0 11.2 Kannada 2.2 2.2 Kashmiri 2.3 2.3 Malayalam 2.3 2.3 Marathi 2.2 2.2 Oriya 2.7 2.7 Punjabi 9.0 4.5 Sinhalese 9.0 6.0 Tamil 15.0 13.9 Telegu 4.0 4.0 Urdu 3.0 1.6 Total 85.9 72.1 
Table 1: Word counts for each language in the EMILLE/CIIL Corpus as of January 2003 
2.1.1 Encoding of the monolingual written corpora 
The decision had been made early on to use CES encoding for the EMILLE corpora. However, for CES documents there are a variety of different levels of conformance. Level 2 conformance requires the use of SGML entities to replace special characters such as emdashes, currency signs, and so on; however, since the corpus was to be encoded as 16-bit rather than 7-bit text this was not a matter of importance, as there is provision in the Unicode standard for all these “special” characters. Level 3 conformance involves adding tags for abbreviations, numbers, names and foreign words and phrases. Given the size of the corpus, this could only be practicable if accomplished automatically; however, given the large range of writing systems we were dealing with, a suitable algorithm would have been too time-consuming to implement. 
The corpus texts therefore extend only to level 1 CES conformance. For this it is necessary for documents to validate against the cesDoc DTD. The level 1 recommendation that italics and similar textual information that might conceivably indicate a linguistically relevant element be retained has not been implemented, again due to the unwieldiness of attempting to do so for the large number of scripts involved. 
In practice, this means that for monolingual written texts the main textual elements are <p>, <s>, and <head>. These elements could be deduced automatically from the HTML code of the original web pages from which the data was gathered. A full CES header has been used, however; see the Appendix to this paper for an example. 
2.1.2 Mapping the corpus texts to Unicode 
The task of mapping the data in the monolingual written corpus is, as has been indicated above, a fairly difficult one. Whilst it is fairly simple to write a program that will map every character in a given font to one or more given Unicode characters, this basic algorithm will not handle the more problematic fonts. The formats we had to deal with fell into three broad groups. 
74
• Texts in Urdu or western Punjabi required one-to-one or one-to-many character mapping. This was due to the nature of the alphabet11 in which in they were written, which does not contain conjunct consonants as the Indian alphabets do. 
• Texts in ISCII required one-to-one character mapping. These texts, primarily those from the data provided by CIIL, could be mapped very simply because the Unicode standard for Indian alphabets is actually based on an early version of the ISCII layout. 
• Texts in specially-designed TTF fonts as discussed above required the most complex mapping. They typically contain four types of characters. The first type need to be mapped to a string of one or more Unicode characters as with ISCII and the Perso-Arabic script. The second type have two or more potential mappings, conditional on the surrounding characters. Some of these conditional mappings could be handled by generalised rules; others operated according to character-specific rules. The third type of characters required the insertion of one or more characters into the text stream prior to the point at which the character occurred12. The fourth type, conversely, required characters to be inserted into the text stream after the current point (in effect, into a Unicode stream which does not yet exist)13. In neither of the latter two types was it simply a case of going “one character forwards” or “one character back”; the insertion point is context-sensitive. 
The third type of text in particular could not be dealt with using simple mapping tables – each font required a unique conversion algorithm. The task of developing and coding these algorithms was split between a partner in India and the University of Lancaster. The “Unicodify” software suite developed at Lancaster is currently capable of mapping three fonts for three separate languages, and has been successfully used to encode the monolingual data published in the beta release of the corpus (see section 4.0 below). 
Unicodify is also capable of re-interpreting the HTML elements created by the “save as Web Page” function of Microsoft Office programs and mapping them to the CES elements of <p>, <head> and <s>, and of generating an appropriate header. 
At the time of writing, the collection phase for the EMILLE/CIIL MWC data is nearly complete. Only around 13 million words of data remain to be collected (as shown in Table 1 above). Good progress is also being made on mapping this data to Unicode. Consequently, the focus of the project is now falling increasingly on parallel and spoken data. 
2.2 Parallel corpora 
The problems we encountered in collecting MWC data were also encountered when we started to collect parallel data. However, the relatively modest size of the parallel corpus we wished to collect (200,000 words in six languages) meant that we were able to contemplate paying typists to produce electronic versions of printed parallel texts. We eventually decided to do this as we had an excellent source of parallel texts which covered all of the languages we wished to look at: UK government advice leaflets. This was a good source of data for us, as we wished to collect data relevant to the translation of South Asian languages in UK in a genre that was term rich. 
The leaflets we were able to gather were mostly in PDF or print-only format. Typing these texts became a necessity when the UK government gave us permission to use the texts, but the company that produced the electronic versions of the texts refused to give us the electronic originals. We found it 
11 We have come to refer to this alphabet as “Indo-Perso-Arabic”, although it is more widely known simply as the “Urdu alphabet”, or in the case of the various forms of western Punjabi that use it, “Shahmukhi”. This name is designed to capture the fact that the Perso-Arabic script as used for Indo-Aryan languages has certain shared features not found in Arabic, Persian, etc. – for instance, characters for retroflex consonants, or the use of the nasta'liq style of calligraphy. 
12 This is primarily the case for those Indian alphabets which allow conjunct consonants whose first component is the letter “ra”. When this letter is the first half of a conjunct, it takes the form of a diacritic which appears after the second half of the conjunct. In Unicode, the text stream contains the logical order of the characters, but in the TTF fonts, the graphical order is almost always the order that is held in the computer's memory. 
13 This is primarily the case for certain vowel diacritics which indicate vowels that follow the consonant but which appear before the consonant. Again, Unicode follows the logical order, whereas TTF fonts almost always follow the graphical order of the glyphs. 
75
economic to pay typists to produce Unicode versions of the texts using Global Writer, a Unicode word-processor.14 
The research value of the British government data is very high in our view. The UK government produces a large number of documents in a wide range of languages. All are focused in areas which are term-rich, e.g. personal/public health, social security and housing. To build the parallel corpus we collected 72 documents from the Departments of Health, Social Services, Education and Skills, and Transport, Local Government and the Regions.15 
Other than the need to type the data from paper copies, the parallel corpus also presented one other significant challenge: while most of the data is translated into all of the languages we need, there are a few instances of a document not being available in one of the languages. Our solution is to employ translators to produce versions of the documents in the appropriate language. While far from ideal, this is not unprecedented as the English Norwegian Parallel Corpus project also commissioned translations (see Oksefjell, 1999). All such texts are identified as non-official translations in their header. 
The parallel corpus is now complete, and we are beginning the process of sentence aligning the texts using the algorithm of Piao (2000). 
2.2.1 Annotation of the parallel corpora 
The annotation of the parallel texts was essentially the same as that applied to the monolingual texts. The principal difference was that the parallel texts had to be keyboarded manually by native speakers of the relevant languages, and thus rather than the automated insertion of SGML elements which characterises the monolingual written corpora, it was necessary to formulate guidelines that would allow transcribers with no knowledge of SGML to accurately mark up the text. In the event, the ultimate content of the transcription guidelines was dictated by certain problems relating to the recruitment of transcribers. 
Our initial strategy was to recruit typists from among the student body of Lancaster University, which includes native speakers of all the languages in question (Hindi-Urdu, Bengali, Punjabi, and Gujarati). However, we were only able to find two or three reliable transcribers in this way. The majority of students recruited were unable to commit themselves to working on the project for the necessary extended period. This problem was particularly acute because we had to recruit outside the Department of Linguistics; while there were potential academic benefits for Linguistics students which would make the task more worthwhile, this was not the case for students of other subjects. 
Therefore, we have formed an arrangement with a data-processing firm in India, Winfocus PVT, who have been able to take on the transcription of a significant proportion of the parallel corpus (as well as nearly all the spoken data – see also below). However, because Winfocus deployed a wide range of staff members on the project, it was necessary that the transcription guidelines be as simple as possible, as typing would otherwise be slowed to an unacceptable speed. 
The decision had already been made for the monolingual corpora to adhere to only the most basic level of CES-compliance (see above). This meant straightaway that the guidelines could be fairly simple. The SGML elements included in all versions of the guidelines were <s>, <head>, <p>16, and <foreign> – these were also used in the monolingual corpus. Unique to the parallel corpus was the tag <corr> for a transcriber's correction of a typographical error in the original printed text (the SGML of course retains the original “incorrect” version). 
The instructions for the <gap> element (replacing pictures and other non-transcribable graphical elements) have likewise always been part of the guidelines. It would seem that these instructions have 
14 When the project began, Global Writer was one of the few word-processors able to handle the rendering of Indic languages in Unicode. Since then, Microsoft have made Word 2000 Unicode-compliant. However, unless running on a Windows 2000 machine the Unicode compliance of Word 2000 is not apparent. 
15 We also collected a smaller number of texts from the Home Office, the Scottish Parliament, the Office of Fair Trading, and various local government bodies (e.g. Manchester City Council). 
16 Lists of bulleted-pointed items – which are very common in the UK government information leaflets – are encoded as separate <s> elements within a single <p>, as are table cells. 
76
been ignored by transcribers, however, who have just passed silently over the illustrations17. We intend to harmonise the use of the <gap> element in the data at the end of the project. 
In the initial version of the guidelines, instructions were included for an SGML encoding of footnotes using <ptr> elements to anchor the notes and <note> elements at the end of the file containing the footnote text. However, in practice transcribers seem to have ignored these guidelines as well, encoding footnotes using normal <p> and <s> elements wherever they physically occurred in the text. Therefore <ptr> and <note> were excised from later versions of the guidelines, as were the instructions for the <bibl> reference for bibliographic references – which in the event was never needed anyway. 
Initially, we also asked the transcribers to add details of title, publication date/place, etc. from the printed text to the header of the file as they typed it. However, due to regular inconsistencies and errors in the headers thus created, we later moved over to a system in which the header was added subsequently by a member of the EMILLE research team using information from a central database. Doing this drastically reduced the bulk of the transcribers’ guidelines, greatly facilitating the training of new transcribers. 
Similarly, it is a design feature of the EMILLE corpora that the filename of each text embodies the hierarchical structure of the corpus as a whole. For the parallel corpus, this means that each filename includes information on the language, medium and category of the text, as well as a descriptive name which is that text's unique identifier. For example, the Gujarati version of the text “The Health of the Nation and You”, an NHS information booklet published by the Department of Health, has the filename guj-w-health-nation.txt18. The guidelines initially gave step-by-step instructions for composing these filenames, but transcribers uniformly got it wrong, so this task too was excised from later versions of the guidelines. 
2.3 Spoken corpora 
For the collection of spoken data we have pursued two strategies. Firstly we explored the possibility of following the BNC (British National Corpus) model of spoken corpus collection (see Crowdy, 1995). We piloted this approach by inviting members of South Asian minority communities in the UK to record their everyday conversations. In spite of the generous assistance of radio stations broadcasting to the South Asian community in the UK, notably BBC Radio Lancashire and the BBC Asian Network, the uptake on our offer was dismal. One local religious group taped some meetings conducted in Gujarati for us, and a small number of the people involved in transcription work on the project agreed to record conversations with their family and friends. The feedback from this trial was decisive – members of the South Asian minority communities in Britain were uneasy with having their everyday conversations included in a corpus, even when the data was fully anonymised. The trial ended with only 50,000 words of spoken Bengali and 40,000 words of Hindi collected in this way. 
Consequently we pursued our second strategy and decided to focus on Asian radio programmes broadcast in the UK on the BBC Asian Network as our main source of spoken data.19 The BBC Asian Network readily agreed to allow us to record their programmes and use them in our corpus. The five languages of the EMILLE spoken corpora (Bengali, Gujarati, Hindi-Urdu, and Punjabi) are all covered by programmes on the BBC Asian Network. At least four and a half hours in each language (and more in the case of Hindi-Urdu) are broadcast weekly. The programmes play Indian music (the lyrics of which have not been transcribed) as well as featuring news, reviews, interviews and phone-ins. As such the data allows a range of speakers to be represented in the corpus, and some minimal encoding of demographic features for speakers is often possible as at least the sex of the speaker on the programmes is apparent. 
The recordings of the radio programmes are currently being digitised and edited, to remove songs and other such material. The recordings will be made available in conjunction with the transcriptions. However, the transcriptions and recordings will not be time aligned. An obvious future enhancement of 
17 It should however be noted that one transcriber went the opposite way, being extremely “trigger happy” with his use of the <gap> element and filling the text with non-requisite information which had to stripped later. 
18 Spoken texts are similarly titled; for instance the file guj-s-cg-asiannet-02-11-23.txt is a context-governed spoken text containing a transcription of the BBC Asian Network Gujarati programme transmitted on the 23rd November 2002. 
19 Programmes broadcast in Bengali and Urdu on BBC Radio Lancashire make up the remainder of the spoken corpus. 
77
this corpus data would be to work on techniques, already well established for English, to time align the transcriptions. 
The recording and transcription of the broadcasts is ongoing and to date we have completed the transcription of 265,000 words of Bengali, 109,000 words of Gujarati, 41,000 words of Hindi, and 119,000 words of Urdu. 
2.3.1 Annotation of the spoken corpus 
The transcription of the spoken texts was undertaken by the same group of typists who worked on the parallel corpus. In a similar way, the annotation guidelines, which began by embracing the full range of possible CES encoding elements, were of necessity simplified as the project progressed. 
As with the parallel transcriptions, the requirement to generate an appropriate header and filename were dropped early on. Information required for the header – on participants sex/age/profession, their relationships, and the setting of the conversation – was noted in plain English at the top of the file by the transcribers, and then converted to a proper SGML header later on. 
Within the text itself, the elements that were implemented were <u> for utterance – which also records the speaker and a unique ID for each utterance – and also <foreign>, <unclear>, <omit>, <vocal>, and <event>, which relate to either the content of the speech or other noises on the tape. The codes for noting overlapping speech, the <pause> code, and the large set of <shift> codes to indicate changes in stress, seem to have been almost entirely neglected by the transcribers, so although they were not removed from the guidelines their importance was de-emphasised in later versions to avoid distracting attention from the greater importance of the <u>, <unclear> and <omit> elements. 
3 Analytic annotation of the corpora 
We aimed from the outset to explore morphosyntactic annotation of the Urdu corpus. For a description of the work undertaken on this aspect of the project, see Hardie (2003). 
The corpus annotation research of EMILLE has recently expanded to cover another form of annotation – the annotation of demonstratives – in Hindi. The work on Hindi is at an early stage, with an annotation scheme originally designed to annotate demonstratives in English (Botley and McEnery, 2001) being used to annotate Hindi. The annotation is currently underway and the goal is to annotate the demonstratives in 100,000 words of Hindi news material by the end of the project. 
4 Accessing the corpus 
A beta release of the EMILLE/CIIL corpus will be available, free of charge, for users from April 2003. The beta release of the corpus will contain a sample of MWC, parallel and spoken data for the core EMILLE languages. In order to register for access to the beta release, users should contact Andrew Hardie. 
5 Conclusion 
The EMILLE project has adapted and changed over the course of the past two years. With regard to the EMILLE corpora, this has in large part been due to the project team engaging in a dialogue with the growing community of researchers working on South Asian languages. As a result of this dialogue the EMILLE team has made some major changes to the original design of the EMILLE corpora. However, as with all large-scale corpus-building projects, other changes have occurred on the project which have been responses to unexpected factors, such as the reluctance of members of the minority communities to engage in the recording of everyday spontaneous speech, and the lack of compatible 8-bit font encoding standards used by the different producers of electronic texts in the relevant languages. Devising methodologies to convert the numerous disparate 8-bit based texts to Unicode has been one of the most complex and time-consuming tasks of the project. 
The area of South Asian corpus building is growing. As well as work in the UK and India, a new centre for South Asian language resources has been established in the US20. As the centres cooperate and integrate their research, there is little doubt that further work on the construction and annotation of South Asian corpora will grow. As this work grows, we believe that corpus builders should not loose 
20 See http://ccat.sas.upenn.edu/~haroldfs/pedagog/salarc/overallplan.html 
78
sight of two important truths. Firstly, that collaboration is better than competition – the corpus produced by Lancaster/CIIL will be larger and better because we have accepted this. The construction of large scale language resources needs the acceptance of this truth if it is to be effective. Secondly, that while many South Asian languages are entering the growing family of languages for which corpus data is available, there are still languages spoken in South Asia and the world for which corpus data is not available. While we must celebrate the creation of corpora of South Asian languages, we should also think of the work yet to be done in creating corpora for those languages not yet corpus enabled. 
References 
Baker, JP, Burnard, L, McEnery, AM and Wilson, A 1998 Techniques for the Evaluation of Language Corpora: a report from the front. In Proceedings of the First International Conference on Language Resources and Evaluation (LREC), Granada. 
Baker, JP, Hardie, A, McEnery, AM, Cunningham, H, and Gaizauskas, R 2002 EMILLE, A 67-Million Word Corpus of Indic Languages: Data Collection, Mark-up and Harmonisation. In: González Rodíguez, M and Paz Suarez Araujo, C (eds.) Proceedings of 3rd Language Resources and Evaluation Conference(LREC). Las Palmas de Gran Canaria. 
Botley, S.P. and McEnery, A 2001 Demonstratives in English: a Corpus-Based Study. Journal of English Linguistics, 29: 7-33. 
Bureau of Indian Standards 1991 Indian Standard Code for Information Interchange, IS13194. 
Crowdy, S 1995 The BNC spoken corpus. In: Leech, G, Myers, G and Thomas, J (eds.), Spoken English on computer: transcription, mark-up and application. Longman: London. 
Hardie, A 2003 Developing a model for automated part-of-speech tagging in Urdu. Paper presented to CL2003 conference. 
McEnery, A, Baker, JP and Burnard, L 2000 Corpus Resources and Minority Language Engineering. In M. Gavrilidou, G. Carayannis, S. Markantontou, S. Piperidis and G. Stainhauoer (eds) Proceedings of the Second International Conference on Language Resources and Evaluation (LREC): Athens. 
McEnery, AM and Ostler, N 2000 A New Agenda for Corpus Linguistics – Working With All of the World's Languages. In Literary and Linguistic Computing, 15: 401-418. 
Oksefjell, S 1999 A Description of the English-Norwegian Parallel Corpus: Compilation and Further Developments. In International Journal of Corpus Linguistics, 4:197-219. 
Pal, U and Chaudhuri, BB (1995) Computer recognition of printed Bengali script. In International Journal of System Science, 26: 2107-2123. 
Piao, SS 2000 Sentence and Word Alignment Between Chinese and English. Ph.D. thesis, Department. of Linguistics and Modern English Language, Lancaster University, UK. 
Somers, H 1998 Language Resources and Minority Languages. Language Today 5. 
Tablan, V, Ursu, C, Bontcheva, K, Cunningham, H, Maynard, D, Hamza, O and Leisher, M 2002 A Unicode-based Environment for Creation and Use of Language Resources. In: González Rodíguez, M and Paz Suarez Araujo, C (eds.) Proceedings of 3rd Language Resources and Evaluation Conference(LREC). Las Palmas de Gran Canaria. 
79
Appendix 
Example of the CES header for a monolingual corpus text. Note that throughout the corpus the date format yy-mm-dd is employed. 
<cesDoc id="guj-w-samachar-news-01-05-23" lang="guj"> 
<cesHeader type="text"> 
<fileDesc> 
<titleStmt> 
<h.title>guj-w-samachar-news-01-05-23.txt</h.title> 
<respStmt> 
<respType>Electronic file created by</respType> 
<respName>Department of Linguistics, Lancaster University</respName> 
<respType>text collected by</respType> 
<respName>Andrew Hardie</respName> 
<respType>transferred into Unicode by</respType> 
<respName>"Unicodify" software by Andrew Hardie</respName> 
</respStmt> 
</titleStmt> 
<publicationStmt> 
<distributor>UCREL</distributor> 
<pubAddress>Department of Linguistics, Lancaster University, Lancaster, LA1 4YT, UK</pubAddress> 
<availability region="WORLD"></availability> 
<pubDate>02-12-18</pubDate> 
</publicationStmt> 
<sourceDesc> 
<biblStruct> 
<monogr> 
<h.title>"Gujarat Samachar" internet news (www.gujaratsamachar.com), news stories collected on 01-05-23</h.title> 
<h.author>Gujarat Samachar</h.author> 
<imprint> 
<pubPlace>Gujarat, India</pubPlace> 
<publisher>Gujarat Samachar</publisher> 
<pubDate>01-05-23</pubDate> 
</imprint> 
</monogr> 
</biblStruct> 
</sourceDesc> 
</fileDesc> 
<encodingDesc> 
<projectDesc>Text collected for use in the EMILLE project.</projectDesc> 
<samplingDesc>Simple written text only has been transcribed. Diagrams, pictures and tables have been omitted and their place marked with a gap element. 
</samplingDesc> 
<editorialDecl> 
<conformance level="1"></conformance> 
</editorialDecl> 
</encodingDesc> 
<profileDesc> 
<creation> 
<date>02-12-18</date> 
</creation> 
<langUsage>Gujarati</langUsage> 
<wsdUsage> 
<writingSystem id="ISO/IEC 10646">Universal Multiple-Octet Coded Character Set (UCS).</writingSystem> 
</wsdUsage> 
<textClass> 
<channel mode="w">print</channel> 
<constitution type="composite"></constitution> 
<domain type="public"></domain> 
<factuality type="fact"></factuality> 
</textClass> 
<translations></translations> 
</profileDesc> 
<revisionDesc></revisionDesc> 
</cesHeader> 
80

The “new” quotatives in American English: A cross-register comparison 
Federica Barbieri 
Northern Arizona University 
Department of English 
Liberal Arts Building 
Flagstaff, AZ 86011-6032 
USA 
Tel.: (928) 523 6845 
Fax: (928) 523 7074 
E-mail: Federica.Barbieri@NAU.EDU 
In the past decade, a growing number of studies has been concerned with the use of like as direct speech introducer, mostly in American English (Blyth et al. 1990, Romaine & Lange 1991, Ferrara & Bell 1995), but also in other varieties of English, such as Scottish English (Macaulay 2001), British and Canadian English (Tagliamonte & Hudson 1999), and African American Vernacular English (Cukor-Avila 2002). While they provide important insights in the use of this new quotative, as well as cumulative evidence of its expansion, these studies are based on small samples of conversational narrative, and their findings are thus hardly generalisable. In addition, very little attention has been devoted to other new direct quotation introducers, such as be all and go. To date, there are indeed no studies of the use of like and other new quotatives, in principled, diverse corpora of spoken English. 
This corpus-based study investigates the frequency of use and the discourse-pragmatic function of the innovative quotatives be like, go, be all, compared with the traditional quotative say, in contemporary spoken American English. The analysis is based on four corpora of spoken American English representing different registers of spoken interaction: casual conversation, campus-related service encounters, academic office hours consultations, and students’ study groups. 
The study includes quantitative analyses of frequency of occurrence of the quotatives, frequency in association with grammatical person, and comparison of frequency across corpora; and qualitative analysis of the discourse-pragmatic function of the quotatives. Simple present and simple past tense forms of the quotatives were analysed quantitatively and qualitatively in four small corpora of approximately 500,000 words. 
The findings showed that type and frequency of occurrence of the quotatives vary across different registers of spoken language, thus suggesting that the use of direct quotation and the way it is introduced are sensitive to context and level of formality. The new quotatives be like and go were found to be frequent quotatives which successfully compete with the traditional say in casual registers such as conversation and service encounters. However, these quotatives are infrequent in consultative registers such as office hours. The new quotative be all was found to be rather infrequent in all registers. The study also revealed that there are clear patterns of association between frequency of occurrence of the quotatives and grammatical person, as well as between grammatical person and discourse-pragmatic function of the quotative. 
81

A preliminary analysis of collocational differences 
in monolingual comparable corpora 
Marco Baroni and Silvia Bernardini, University of Bologna at Forli 
1. Introduction 
The notion of collocation has enjoyed mixed fortunes in the 50 odd years of its existence. Claimed to be obscure (Lyons, 1977: 612), counterproductive (Langendoen 1968: 63ff) and generally useless (Lehrer 1974)1 by its detractors, the idea that part of the meaning of a word is somehow related to its “word accompaniment, the other word material in which [it is] most commonly or most characteristically embedded” has also had its supporters. These have suggested that words have no meaning out of context, and meaning itself is not contained anywhere, but rather dispersed as the “light of mixed wave-lengths into a spectrum” (Firth 1957(1951): 192). 
The intuitive appeal of this view is evident if one thinks of the difficulty a compositional approach to meaning has (even allowing for subcategorisation frames) in explaining the patterned quality of language performance, as found in a corpus, and ultimately the speaker's or writer's effortless routine handling of co(n)textual restrictions. The hypothesis that “everything we say may be in some degree idiomatic – that […] there are affinities among words that continue to reflect the attachments the words had when we learned them, within larger groups” (Bolinger, 1976:102) provides a powerful argument in favour of the empirical study of collocations, with implications for theoretical, descriptive and applied branches of linguistics. In recent years, notwithstanding the vagueness of the notion and consequent methodological problems in investigating it empirically, the study of collocations has indeed defied difficulties and criticism and sparked renewed interest in a number of areas ranging from computational and corpus linguistics to lexicography, language pedagogy, and crucially for our purposes, translation studies. 
The hypotheses that “everything we say may be in some degree idiomatic” (Bolinger, above), and that “actual usage plays a very minor role in one's consciousness of language” (Sinclair 1991:39) raise a number of interesting questions for translation research. Is there any evidence that translators be aware of collocational restrictions in the source and target languages? Do they show sensitivity to phraseological (a)typicality and restrictedness? These are very complex issues, that can hardly be resolved in one fell swoop. For a start, theoretical as well as methodological problems remain as to what collocations are in the first place,2 and how best they can be retrieved from corpora and compared (see e.g. Krenn 2000a). Secondly, different types of corpora for the study of translation exist, providing different perspectives on the translation process. 
In this paper, we limit our investigation to monolingual comparable corpora (MCC) and present a number of attempts at selecting and comparing collocations across original and translated texts. This study is novel in at least two ways: to the best of our knowledge, no previous investigation of the behaviour of translators through MCC has focused on collocational restrictions, and no study of collocational restrictions in translated texts has attempted to select candidate bigrams automatically. 
The paper is organised as follows. Section 2 provides a general background on monolingual comparable corpora and collocation extraction. In section 3 we present a brief study conducted on a small corpus of EU reports in translated and original English. Section 4 describes a more substantial study of a corpus of translated and original world affairs articles in Italian. In section 5 we briefly discuss directions for further work. 
2. Background 
2.1 Monolingual comparable corpora in translation research 
Monolingual comparable corpora are collections of original and translated texts in the same language, assembled according to comparability criteria such as “a similar domain, variety of language and time span […]” (Baker 1995: 234). 
According to Frawley (1984: 168-169) 
translation […] is essentially a third code which arises out of the bilateral consideration of the matrix and target codes […] since [it] has a dual lineage, it emerges as a code in its own right, setting its own standards and structural presuppositions and entailments […]. 
1 “The main criticism against the lexical approach to co-occurrence is that it does not explain anything. The lexical item is found to collocate with a second item and not with a third, but no explanation is given. Collocations and sets are treated as if combinatorial processes of a language were arbitrary” (Lehrer 1974: 176). 
2 A recent book on corpus-based lexical semantics gives the following rather general definition of collocation: “‘collocation’ is frequent co-occurrence” (Stubbs 2001: 29). 
82
Following Baker (e.g. 1995), work on MCC has adopted a corpus-based research methodology to unveil universal features of translation seen not as an individual act of interlinguistic transfer, but as a mediated communicative event, with its own “third code”. 3 Thus, rather than focusing on differences between single originals and their translations (i.e. parallel texts), MCC allow the analyst to compare collections of originals and translations in the same language, unrelated to each other but chosen so as to be broadly comparable. 
There is no doubt that MCC provide an innovative research environment in which translation norms (Toury 1995) and strategies (Löscher 1991) can be observed against the backdrop of target language use. Yet they raise substantial methodological problems that risk invalidating the results obtained. The main problem one is confronted with relates to the comparability of the corpus components. 
According to one of its compilers (Laviosa 1998a), the two components of the English Comparable Corpus (ECC) are comparable with regard to the relative proportion of biography and fiction (i.e. genre), time span, distribution of female and male authors, distribution of single and team authorship, and overall size of each component. Furthermore, continues Laviosa, “the target audience of both collections can be characterised as literate, intellectual adults of both sexes”. 
The compilers have clearly thought out their design criteria. Yet doubts about the comparability of the two corpora remain. The criteria just mentioned would appear to be derived from monolingual corpus building heuristics. Yet your average translated novel, say, has a much more complex history than your average original novel. First, it derives directly from an existing text (its source text). Disregarding the latter may have somewhat worrying consequences: for example, the source text could have been written much earlier than its translation. In this case, it would not seem unlikely for the latter to display features typical of a diachronically different state of the target language (for further discussion and a concrete example see Bernardini and Conrad 2002). 
In more general terms, translation involves two cultures-languages, at times distant from each other, and a set of decisions – often involving substantial investments of time and funds - that have to be taken in order for a work to migrate between the two. It has been suggested that these migrations are subject to socio-cultural norms (Toury 1995). For instance, not everything that is published in the Netherlands is translated into English, far from it; instead, 
Dutch fiction is chosen for translation either in the function of assumed target taste or in that of the status the work has acquired at the source pole, often as a combination of the two. (Vanderauwera 1985: 132) 
Ignoring such socio-cultural factors when setting up a monolingual comparable corpus may result in reduced comparability and doubtful interpretations of the data obtained. 
Given the complexity of the translation situation – we have only scratched the surface of the problem here, see Bernardini and Zanettin (forthcoming) for a more detailed discussion –, and the fact that corpus comparability is in itself an untrivial concept (Kilgarriff 2001), it would seem wise to give serious thought to the composition of a MCC, at least at these early stages, and to refrain from the assumption of comparability based on crude situational criteria inherited from monolingual research. 
In an attempt to limit the proliferation of variables, and consequent difficulty in interpreting results, the corpora on which the experiments we present are based were selected so as to be maximally comparable. 
The first corpus (EU) is a collection of official reports submitted by different EU countries to the EU Commission, describing progress made in the implementation of European guidelines in the area of employment policies.4 Two versions are typically available, the original in the country's language, and a translation, normally into English. Though the originals were not included in the corpus, they were consulted to make sure that the English texts were indeed translations, not independent texts nor source texts for the pair. 
From these texts we constructed a small corpus of originals from Ireland and the United Kingdom and translations from Finland, Italy, Portugal and Sweden.5 The corpus contains 72,966 words in the original section and 145,932 words in the translation section. 
3 Among the universals proposed are simplification, explicitation/explicitness, normalization, levelling out, disambiguation and standardization (e.g. Baker 1995, 1996; Schmied and Schäffler 1996, Laviosa 1998a, 1998b; Olohan and Baker 2000; Olohan 2001). 
4 The reports are freely available on the Web: http://europa.eu.int/comm/employment_social/news/2001/may/naps2001_en.html 
5 The direction of translation was confirmed by bilingual speakers of English and each of the four languages in question. 
83
This corpus has its advantages and its disadvantages. On the one hand, translation is carried out into international English, thus limiting the effect of preliminary norms deciding what to translate when (Toury 1995, above). The texts are very homogeneous, in terms of topic, date, register etc. Copyright clearance and text preparation are straightforward, and this is no trivial advantage in a exploratory study like this one, in which the possibility that the material be inadequate for the purposes of the research has to be investigated empirically. On the other hand, these are rather boring texts, translated into a highly conventionalised variety of English – sometimes called EUese; thus, doubts may be raised about the generalisability of any results to different translation settings. Furthermore, the corpus is rather small. 
The second corpus (LIMES) is a slightly less homogeneous but far larger collection of articles published in the Italian Quarterly Limes - Rivista italiana di geopolitica [Italian geopolitics journal]. 6The complete collection of articles published between 1993 and 1999 is included. The total size of the corpus is approximately 3 million words, with translations accounting for slightly less than one third of the total. These have been carried out by approximately 40 different translators, about half male and half female. The total number of authors exceeds 700. Each volume is centred around a theme (“War in Europe”; “Divisions within Islam”, “What use is NATO?”, “The global bomb”), and normally contains both originals and translations, thus ensuring thematic consistency across the original and translated components. Judging from the topic matter and names of authors and translators, it would appear that source languages for these include, among others, Albanian, Arabic, Chinese, English, French, German, Serbo-Croatian, Russian. 
These characteristics of homogeneity (originals and translations address the same audience, deal with virtually the same topics, conform to the same editorial policies) would appear to guarantee an acceptable level of comparability between the subcorpora, such as is rare in MCC. The likely variety of source languages should limit specific source language effects. Similarly, the variety of authors, translators and topics covered should limit the effect of idiosyncracies. 
2.2 Collocational restrictions and translation 
In order to compare the level of patterned-ness in translated versus original language, it is first of all necessary to retrieve candidate patterns. A common approach to this problem has been the structural one: patterns are defined in terms of sequences of parts of speech, which are then searched either manually, or automatically with the help of a tagged corpus. For example, Gitsaki (1996) adopts this approach in her study of collocations in ESL student written production, deriving candidate structures from a dictionary (Benson et al 1986). Krenn and Evert (see e.g. Krenn 2000b, Krenn and Evert 2001) adopt statistical measures to rank potential collocations that match certain syntactic templates, on the basis of a tagged and partially parsed corpus. Heid (1996: 121) describes “discovery procedures for collocations […] based on a detailed description of the targeted collocations”. The starting point in this case is a classification of lexical functions as defined by Mel'èuk (e.g. 1996). 
Within TS, two recent works have attempted to analyse collocations in original vs translated language. They both rely on bilingual (parallel) corpora, adopting a different viewpoint from the one adopted here. Yet the method used for retrieving relevant units of analysis is equally relevant. Kenny's study of “sanitisation” in translation adopts various techniques for spotting creativity in originals before checking how it is rendered in translation. One “node” is identified – the German word Auge [eye] - as an impressionistically viable choice, i.e. a word that is frequent enough, enters into fixed expressions, and has been found in previous studies to be the object of creative manipulation by other writers. Concordances, tables of collocates and lists of clusters are then retrieved for this word. This method is appropriate to investigate whether translators tend to normalise creative collocations, but would appear to be hardly adaptable to the aims of the present study. More relevant to our concerns is Danielsson's (2001) attempt at designing an automatic process whereby UMs (units of meaning) for English and Swedish can be identified in a parallel corpus and compared. She takes a sample of about 200 “interesting” words occurring 200 times or more in her corpora, and automatically works through their downward and upper collocates (Sinclair 1991). This method yields units of varying length containing some of the most frequent words in the corpora. A third method, the longest-linear method, is used to retrieve “units of structure” of the type “the X of the new Y” (ibid: 149). What would appear to be left out by the joint application of these methods is the potentially large set of collocations containing less frequent words, whose significance is not necessarily small. 
The collocation extraction methods discussed in this section, all relatively knowledge-intensive, have led to cleaner, easier-to-interpret results than the ones we report below. It might be 
6 We would like to thank Limes for granting permission to use their CD-ROM for this research 
84
argued, however, that they place strong interpretative grids over the data (e.g. through intuitive classifications, POS tagging and so forth) that are better avoided at these early stages of research. 
We prefer the simple knowledge-free method we describe below (4.2) since we do not know, a priori, which collocational structures are most typical of the original and translated languages we are studying, and thus we would not want to bias the results in the direction of a specific subset of collocational templates. Indeed, as we will briefly discuss in 4.5, it appears that some interesting differences between translated and original texts concern frequent syntactic structures that would not traditionally be considered collocations. 
No less important, we are interested to see whether it is possible to obtain meaningful results with a method that would be applicable to any language or sub-language, independently of the NLP resources available. 
3. A pilot study with the EU corpus 
We conducted a pilot study with the EU data in which, following a method similar to those proposed in Kilgariff (2001), we verified that the lists of collocations extracted from subcorpora constructed from parts of the same corpus are more strongly correlated than those extracted from different corpora. This result was obtained with all the collocation extraction and scoring methods we describe below. Thus, the pilot experiment indicates that our measures are sensitive to systematic similarities among corpora more than to the random similarities and differences that we expect to exist in any set of textual data. 
However, a more thorough investigation of the EU data along the lines of the one we report below for the LIMES data failed to reveal systematic differences between original and translated documents. 
While it is tempting to attribute this failure to the “scripted” nature of the EU report genre, that would tend to reduce the differences between originals and translations, we feel that, because of the small size of the corpus, it is premature to draw any conclusions from this failure. We plan to collect a larger corpus of EU reports, in order to obtain more robust results. 
4. Analysis of the LIMES corpus 
4.1 Corpus pre-processing 
The text extracted from the LIMES database was split into a corpus of articles originally written in Italian and a corpus of articles translated from other languages into Italian using an automated procedure. Interviews and roundtables were discarded, since we were not sure about their status. 
The output of the automated procedure was checked and corrected by hand. We also performed other minor clean-ups in a semi-automated fashion. 
The original text corpus (O) and the translated text corpus (T) were tokenized in an extremely rudimentary way, removing all non-alphabetic symbols except the apostrophe, that, following the conventions of Italian orthography, was tokenized as part of the preceding word. 
After tokenization, the O corpus contained 2,132,060 words and the T corpus contained 895,820 words. 
O and T were further subdivided into subcorpora as described in sections 4.3 and 4.4. 
4.2 Collocation extraction and ranking 
In order to find collocations, we first collected for each subcorpus of interest (see discussion in 4.3 and 4.4 below) candidate bigrams that had the following characteristics: 1) they were made of words that occurred at least twice in all the subcorpora to be compared; 2) they occurred at least 3 times in the relevant subcorpus. 
The rationale for the first of these conditions is that we are not interested in differences in collocations that are due to differences in the topics covered by the various articles. We expect that words that are relatively frequent in all the subcorpora being compared are words that are not strongly linked to any particular topic. 
The second condition guarantees that we have a list of manageable size, and it is unlikely to exclude any “true” collocation, since collocations are, by definition, frequent ngrams. 7 
We used three association measures to rank the lists of bigrams: raw frequency, (point-wise) mutual information (Church and Hanks 1990) and (-2*) log-likelihood ratio (Dunning 1993). Unlike raw frequency, the other two measures take the unigram frequency of the words composing the bigram 
7 We ran some preliminary experiments in which the bigram collection procedure ignored words from an automatically constructed list of likely function words. We are still in the process of analyzing the results obtained in this way, but see note 10 below for some short remark on them. 
85
into account, and favor word combinations whose frequency is higher than what we would expect under assumptions of independence. 
Mutual information and log-likelihood ratio are calculated using the formulas given in Manning and Schütze (1999 ch. 5). For discussion of these and other measures of collocativity see also Evert (2001). 
We chose not to pick one specific measure as the “right” one given the exploratory nature of this study. Moreover, recent work (e.g. Inkpen and Hirst 2002, Baroni et al. 2002) suggests that measures such as mutual information and log-likelihood ratio should be used in combination, as they tend to discover different types of related words. 
As a consequence of the collocation extraction and evaluation methods we used, the results reported below are based on a rather generous and vague notion of what counts as a collocation: essentially, any pair of adjacent words that has a high frequency, and/or a higher frequency than what we would expect by chance, is treated as a collocation. 
Furthermore, by working with lists of ranked bigrams, we are implicitely assuming that collocativity is gradient, rather than binary. 
4.3 Comparing the number of collocations in translated and original text 
The first question we were interested in answering was the following: Do translators have a greater tendency to use fixed expressions than original authors? In principle, there could be an effect in either direction: On the one hand, translators could have a tendency to use a simplified language characterized, among other things, by the frequent repetition of the same expressions (a possible effect of the tendency towards explicitness, on which see Schmied and Schäffler 1996). On the other, faithfulness to the source language text, coupled with the fact that many fixed expressions are often not translatable from a language to the other, could lead the translators to use fewer collocations than the creators of original texts. 
In order to study this issue, we split the T corpus into 5 subcorpora containing 179,164 words each, and we randomly selected 5 chunks of 179,164 words from the larger O corpus. 
From each of the 10 subcorpora created in this way, we extracted candidate bigrams and computed frequency, mutual information and log-likelihood as described above. 
First of all, we compared the number of candidate bigrams in the T-subcorpora to the number of candidate bigrams in the O-subcorpora. For the T-subcorpora, the average number of bigrams is 8,094.2 (median: 8,149); for the O-subcorpora the average is 8,044.8 (median: 8,128). According to the results of a two-tailed Mann-Whitney U test (see Siegel 1956, ch. 6), the difference between the two sets is not significant at the á = 0.05 level. 
In the subsequent analyses, rather than considering simply the number of bigrams extracted from each subcorpus, we looked at the association scores that were assigned to these bigrams. In particular, for each measure m and for each cutoff point c from a set of cutoff points across the distribution of m, we computed the percentage of bigrams in each subcorpus that had an m-score equal or greater than c. We then compared the percentage of bigrams at or above the various cutoff points in the T- vs O-subcorpora. 
In Table 1, we report the results of this type of analysis performed at 3 cutoff points for each of the measures (for frequency and log-likelihood ratio, the cutoffs are expressed as logarithms of the actual values). We chose this particular set of cutoff values since they seem to represent well the range of patterns encountered. 
For each measure and representative cutoff point, the table reports the average percentage of bigrams with scores at least as high as the cutoff point in the T- and O-subcorpora, the medians, and whether a two-tailed Mann Whitney test comparing the T- and O-subcorpus sets with respect to the relevant percentages was significant at the á = 0.05 level. 
measure cutoff T-avg O-avg T-med O-med MW test fq 2 25.8 24.87 25.84 24.73 significant fq 3 6.45 5.8 6.42 5.73 significant fq 4 1.43 1.09 1.39 1.05 significant mi 5 25.53 25.07 25.55 25.09 significant mi 8 4.58 4.75 4.56 4.70 not sig mi 10 1.05 1.15 1.04 1.16 not sig llr 3 41.39 40.07 41.56 39.92 significant llr 4 12.42 11.65 12.35 11.36 significant llr 5 3.11 2.69 3.02 2.83 significant 
Table 1 Proportion of bigrams ¡Ý cutoff value 
86
As the table shows, there is a small but clear tendency for the translated texts to contain a larger number of bigrams with stronger association scores (this is also in line with the results on the absolute number of bigrams we presented above). 
The only data that do not go in this direction are those for the “middle'” and “high” mutual information cutoffs. It is interesting that these are also the only levels at which the difference between the groups is not statistically significant, i.e., what we have here is not a reversal of the effect, but a lack of significant effects of the translated/original distinction. 
An informal comparison of the top bigrams according to mutual information to the top bigrams according to frequency and log likelihood ratio suggests that the latter two measures (the first more than the second) tend to pick up bigrams where at least one component is a function word, whereas mutual information tends to pick up bigrams that are closer to our intuitive idea of what a collocation is (frequent/lexicalized N+Adj or V+N structures). 
Thus, the difference between translated and original texts detected with frequency, log-likelihood ratio and the lowest mutual information cutoff seems to be due to frequent bigrams that would not normally be treated as collocations. We will come back to this topic in 4.5 below. 
4.4. Collocation overlap among translated and original texts 
The data reported above provide some (weak) evidence that there are systematic differences between translated and original texts in terms of collocational patterns, but they do not tell us whether such differences are due to a general tendency for translators to use more fixed expressions, or whether there are specific fixed expressions that tend to be favored by translators (or by original writers). 
In order to test this second possibility, we conducted another set of experiments in which we measured the degree of overlap and correlation among the collocations found in original and translated texts. 
This time, we split the T corpus into 10 subcorpora containing 89,582 words each, and we randomly selected 10 chunks of 89,582 words from the larger O corpus. 
We then merged 5 randomly selected T-subcorpora into a 447,910 word “reference” T corpus and 5 randomly selected O-subcorpora into a 447,910 word “reference” O corpus. 
The idea, then, was to compare the bigrams found in the unmerged T- and O-subcorpora to the bigrams found in reference T and reference O. If there is a tendency to use similar bigrams in texts of the same type, we should find that the bigrams in the T-subcorpora tend to be closer to those in reference T than to those in reference O, and/or that the bigrams in the O-subcorpora tend to be closer to those in reference O than to those in reference T.8 
First of all, we looked at the number of candidate bigrams (in the sense of 4.2 above) that the T-subcorpora and the O-subcorpora shared with the reference corpora. To control for the effect of the absolute size of the bigram lists we computed the percentage of shared bigrams over the total number of distinct bigrams in the two lists being compared. 
The average percentage of bigrams shared by the T-subcorpora with reference T was 21.28 (median: 21.36); the average percentage shared by the T-subcorpora with reference O was 21.32 (median: 21.11). The average percentage of bigrams shared by the O-subcorpora with reference O was 21.16 (median: 21.20); the average percentage of bigrams shared by the O-subcorpora with reference T was 20.23 (median: 20.25). 
These data suggest that there is no strong trend in either direction as far as simple ovelap of the candidate bigram lists goes. This was confirmed by the statistical analysis. We ran Wilcoxon two-tailed matched-pairs signed-rank tests (Siegel 1956, Ch. 5) comparing the T-subcorpora percentage overlap with reference T vs their overlap with reference O, and comparing the O-subcorpora percentage overlap with each of the reference corpora. Neither test gave significant results at á = .05. 
We then computed, for each of the association measures, the Spearman rank correlation coefficients (Siegel 1956, ch. 9) between each T- or O-subcorpus and the reference corpora. The correlations were computed by considering only those bigrams that occured both in the list extracted from the relevant subcorpus and in the list extracted from the relevant reference corpus.9 
8 We use the reference corpus strategy rather than directly comparing all subcorpora to each other since the latter strategy would yield results that are difficult to interpret, as the samples would not be independent from each other (e.g. the degree of overlap between, say, subcorpora T1 and T2, that between T1 and T3 and that between T2 and T3 would all have counted as instances of T-to-T comparisons). 
9 In general, the percentage overlap between the bigrams in a subcorpus and the bigrams in a reference corpus is around 21%, as we have just seen. Including the 79% of bigrams that are not shared by the compared corpora into the correlation analyses would have been problematic both from a statistical point of view, because of the massive 
87
tie problem due to the 0-scores, and from an empirical point of view, since, in the best case, the analyses would have essentially been a replica of the overlap analyses we just presented. 
The results of these analyses are summarized in Table 2. The first column reports the association measure; the second column reports the subcorpus set; the third column reports the average (in parenthesis: median) of the Spearman coefficients of the correlations between each of the relevant subcorpora and reference T; the fourth column reports the same data for the correlations with reference O; the fifth column reports whether a Wilcoxon test for the corresponding data (correlation coefficients of the subcorpora with reference T vs reference O) gave significant results at á = .05. 
measure subcorpora avg (med) r with T avg (med) r with O W test fq T .63 (.63) .59 (.60) not sig fq O .61 (.60) .62 (.62) not sig mi T .91 (.91) .91 (.91) not sig mi O .91 (.91) .91 (.91) not sig llr T .74 (.73) .72 (.72) not sig llr O .72 (.72) .73 (.73) not sig 
Table 2 Correlations between subcorpora and reference corpora 
First of all, notice that in general the correlation coefficients between corpora are quite high, in the case of mutual information so high that the uniform results could be due to a ceiling effect. 
The results with frequency and log-likelihood ratios go in the expected direction (each set of subcorpora is correlated more strongly with the corresponding reference corpus). However, the differences are very small and they are not statistically significant. 
Given the small size of the subcorpora (< 100,000 tokens) and their limited number (5 per set), it seems that an obvious next step with respect to the overlap/correlation analysis would be to test whether the weak trends we have detected are confirmed by an analysis based on a larger data set. 
4.5 Qualitative analysis 
In order to collect a smaller data set for a preliminary qualitative analysis, we extracted the collocations that appear to be most typical of translated texts and the collocations that appear to be most typical of original texts using the following method. 
We first computed the average log-likelihood ratio for each bigram in the O-subcorpora and in the T-subcorpora described in 4.3. Then, we computed the log ratio of these two values for each bigram. We put the bigrams with a positive value of this measure equal to or greater than 12 in the list of bigrams typical of original text, and the bigrams with a negative value equal to or greater than 12 in the list of bigrams typical of translated text. The ±12 cut-off point was arbitrarily chosen to limit the data to a manageable amount. 
Based on a subjective evaluation of meaningfulness and wellformedness, each set was further divided into two (sub-)sets. Set A contains sequences that appear to be meaningful and well-formed, while set B contains less likely collocation candidates, i.e. incomplete sequences resulting in syntactically ill-formed structures (termine geopolitica; iniziativa centro; veda nota); fully-predictable sequences (suo figlio [“his/her son”]; noi europei [“we Europeans”]; sara possibile [“it will be possible”]) and, somewhat more controversially, content words preceded or followed by function words (usually articles or prepositions), such as sull'isola [“on the island”]; delle riserve [“of the reserves”]; proveniente dal [“coming from the”]. Since this is no attempt at proposing a classification of collocations, we have not provided intepretative labels for these groupings. Table 3 shows the number of bigrams assigned to each set, and their proportion out of the total number of bigrams selected for analysis. 
Tot Set A%Set B%original 166 83508350translated 203 7436.4512963.54 
Table 3 A tentative classification of bigrams 
Although the initial number or bigrams selected is substantially larger in the case of translations (203 vs 166 in originals), when less meaningful and wellformed sequences are removed only 74 bigrams are 
88
left (vs 83 in the originals).10 These can be futher analysed in terms of their topic-dependency, and grouped along the cline “technical-general” into the fuzzy categories strongly topic-dependent (containing a geographical term), weakly topic-dependent and topic independent (general language). The top 5 bigrams from each category are reproduced in Table 4: 
ORIGINALS meaning approximately TRANSLATIONS meaning approximately strongly 
topic-dependent lega nord Northern League 
(It pol. party) fratelli musulmani Muslim brothers lingua russa Russian language marco tedesco German mark minoranza italiana Italian minority mar rosso Red Sea Alto adriatico High Adriatic chiesa russa Russian church centro europea Central-European vicino oriente Near East weakly 
topic-dependent guerra giusta right war governo federale Federal government minoranze etniche ethnic minorities nucleo centrale central nucleus opinioni pubbliche public opinions sistema monetario monetary system spazio vitale vital space istituto orientale Oriental institute prodotti industriali industrial products autorita federali Federal authorities topic-independent basti pensare suffice it to think terza fase third phase breve periodo short period porre fine put a stop chi scrive the writer [lit. s/he who writes] stessa cosa same thing occorre realizzare it is necessary to set up reso noto made public scorso anno last year far si make possible 
Table 4 Examples of bigrams grouped according to topic-dependency 
Table 5, based on a manual count, shows that topic-independent typical sequences are twice as common in originals as in translations, whilst the opposite is true of strongly topic-dependent sequences: 
topic-independent % strongly topic-dependent%original 21 (25,3%) 12 (14,4%)translated 10 (13,5%) 29 (39,1%) 
Table 5 Distribution of topic independent and strongly topic-dependent bigrams 
The initial impression of a more substantial incidence of repeated patterns in translated vs original language, supported by the number of patterns retrieved, is mitigated by observation of actual instances. It does seem that translated language is repetitive, possibly more repetitive than original language. Yet the two differ in what they tend to repeat: translations show a tendency to repeat structural patterns and strongly topic-dependent sequences, whereas originals show a higher incidence of topic-independent sequences, i.e. the more usual lexicalised collocations in the language. The latter may be viewed as instances of those “target-specific features” that according to Mauranen (forthcoming), who analyses Finnish data, tend to be underrepresented in translations with respect to comparable originals. 
Closer observation of the bigrams excluded from this analysis (B set) reveals further interesting patterns. For example, the sequences considerato come [considered.MASC.SING as] and considerata come [considered.FEM.SING as], appear in the list of typically translational expressions. A search for all the variants of the adjective/past participle (masculine singular and plural, feminine singular and plural) retrieves 619 occurrences from the original corpus, and 333 occurrences from the translation corpus. These figures are in accordance with the relative sizes of the two corpora, approximately 2:1. However, if we look at the frequency of the collocate come in the first position to the right of the keyword, the proportions change dramatically (table 6): 
10 This agrees with what we observed in 4.3 concerning the results obtained with mutual information vs frequency and log-likelihood ratio, i.e. that the former tends to pick up the most plausible collocations whilst failing to detect differences between originals and translations. The same point emerges from a preliminary analysis of the bigrams extracted after removing the function words (see footnote 7 above). Again, in this kind of data the differences between original and translated language nearly disappeared. 
89
considerato/considerata 
considerati/considerate Total tokens%+ come (R1) % Original 6192,096,1912.969 9.8 Translated 333922,9463.661 20.7 
Table 6 Frequencies of considerato and considerato come 
It might be hypothesised that translators show a preference for using optional come [“as”] in this structure. This would be in line with Olohan's findings concerning overuse of optional elements in (English) translation (Olohan 2001). 
Clearly, caution must be exercised in drawing conclusions, especially in an exploratory study like this one. However, the current findings are promising, hinting at some systematic differences in the use of collocations in closely comparable translated and non-translated texts. 
5. Conclusions 
We believe that this study has shown that monolingual (closely) comparable corpora are promising resources for the study of collocational restrictions in translated vs non-translated language. Simple data-exploration methods coupled with qualitative analyses would appear to be adequate in providing at least preliminary insights in this area. 
At the same time, we were only able to detect weak trends in the LIMES corpus, and no effects in the EU corpus. We plan to improve on this via two strategies. On the one hand, we hope that by simply enlarging both corpora, we will be able to identify more robust statistical trends. On the other, we can bootstrap from the data-driven insights presented in this exploratory study to devise knowledge-richer collocation extraction methods. For example, the observations in 4.5 suggest that analysis of frequent bigrams including function words might reveal itself to be particularly relevant in telling translated language apart from original language. This is a strategy we would have not considered had we not performed this preliminary data-driven investigation. 
References 
Baker M 1995 Corpora in translation studies: an overview and some suggestions for future research Target 7(2): 223-243. 
Baker M 1996 Corpus-based translation studies: the challenges that lie ahead. In Somers H L (ed), Terminology, LSP and translation, Amsterdam, Benjamins, 175-186. 
Baroni M, Matiasek J, Trost H 2002 Using textual association measures and minimum edit distance to discover morphological relations. Paper presented at the International Workshop on Computational Approaches to Collocations. Online: http://sslmit.unibo.it/~baroni/ 
Benson M, Benson, E, Ilson R 1986 (1997) The BBI dictionary of English word combinations. Amsterdam, Benjamins. 
Bernardini S, Conrad S 2002 “Multidimensional Analysis and translation”. Paper presented at the international conference Corpora and Discourse, Camerino 27-29 September 2002. 
Bernardini S and Zanettin F forthcoming. When is a universal not a universal? Some limits of current corpus-based methodologies for the investigation of translation universals. In Proceedings of Translation universals, do they exist? Savonlinna, 19-20 October 2001. 
Bolinger D 1976 Meaning and memory. Forum Linguisticum 1(1): 1-10. 
Church K, Hanks P 1990 Word association norms, mutual information, and lexicography. Computational Linguistics 16: 22-29. 
Danielsson P 2001 The automatic identification of meaningful units in language. Unpublished PhD Thesis. Göteborg University. 
Dunning T 1993 Accurate methods for the statistics of surprise and coincidence. Computational Linguistics 19(1): 61-74. 
Evert S 2001 On lexical association measures. Online: http://www.collocations.de/EK/am-html/ 
Firth J R 1957 (1951) Modes of meaning. Papers in linguistics. London, OUP, 190-215. 
Firth JR 1964 (1930). Speech. London, OUP. 
Frawley W 1984 Prolegomenon to a theory of translation. In Frawley W (ed), Translation: literary, linguistic and philosophical perspectives. London and Toronto, Associated University Presses, pp159-175. 
Gitsaki C 1996 The development of ESL collocational knowledge. Unpublished PhD Thesis. University of Queensland. 
90
Heid U 1996 Using lexical functions for the extraction of collocations from dictionaries and corpora. In Wanner L (ed), Lexical functions in lexicography and Natural Language Processing. Amsterdam, Benjamins, pp115-146. 
Inkpen D Z, Hirst G 2002. Acquiring collocations for lexical choice between near-synonyms. SIGLEX Workshop on Unsupervised Lexical Acquisition, 40th meeting of the Association for Computational Linguistics, Philadelphia, June 2002. 
Kenny D 2001 Lexis and creativity in translation. Manchester, St. Jerome. 
Kilgarriff A 2001 Comparing corpora International Journal of Corpus Linguistics 6(1): 1-37. 
Krenn B 2000a The usual suspects: data-oriented models for identification and representation of lexical collocations. Saarbrücken, Saarland University. 
Krenn B 2000b Collocation mining: exploiting corpora for collocation, identification and representation. KONVENS , pp209-214. 
Krenn B, Evert S 2001 Can we do better than frequency? A case study on extracting PP-verb collocations. Proceedings of the ACL Workshop on Collocations, Toulouse. 
Langendoen T 1968 The London School of linguistics: a study of the linguistic theory of B. Malinowski and J.R. Firth.Cambridge (MA), MIT Press. 
Laviosa S 1998a “Core patterns of lexical use in a comparable corpus of English lexical prose. Meta 43(4): 557-570. online: http://www.erudit.org/revue/meta/1998/v43/n4/. 
Laviosa S 1998b Universals of translation In Baker M (ed), The Routledge encyclopedia of translation studies. London, Routledge, pp288-291. 
Lehrer A 1974 Semantic fields and lexical structure. Amsterdam and London, North Holland. 
Löscher W 1991 Translation performance, translation process, and translation strategies. Tübingen, Gunter Narr. 
Lyons J 1977 Semantics - Volume I. Cambridge, Cambridge University Press. 
Manning C, Schütze H 1999 Foundations of statistical natural language processing. Cambridge (MA), MIT Press. 
Mauranen A forthcoming “Where's cultural adaptation?” InTRAlinea. 
Olohan M 2001 Spelling out the optionals in translation: a corpus study. In Rayson P, Wilson A, McEnery T, Hardie A and Khoja S (eds.) Proceedings of Corpus Linguistics 2001, Lancaster, pp423-432. 
Olohan M, Baker M 2000 Reporting that in translated English. Evidence for subconscious processes of explicitation? Across languages and cultures 1(2) 141-158. 
Schmied J, Schäffler H 1996 Explicitness as a universal feature of translation. In Ljung M (ed), Corpus-based studies in English: papers from the seventeenth International Conference on English Language Research on Computerized Corpora (ICAME 17). Amsterdam, Rodopi, pp21-36. 
Siegel S 1956 Nonparametric statistics for the behavioral sciences. New York, McGraw-Hill. 
Sinclair J 1991 Corpus, concordance, collocation. Oxford, Oxford University Press. 
Stubbs M 2001 Words and phrases. Corpus studies of lexical semantics. Oxford, Blackwell. 
Toury G 1995 Descriptive translation studies and beyond. Amstedam, Benjamins. 
Vanderauwera R 1985 Dutch novels translated into English. The transformation of a "minority" literature. Amsterdam, Rodopi. 
91

Investigating cross-linguistic constraints on adjectival and adverbial premodification. A corpus-based study of English and German. 
Dr. Sabine Bartsch, Darmstadt University of Technology, Department of Linguistics and Literature, Hochschulstrasse 1, 64289 Darmstadt, Germany, eMail: bartsch@linglit.tu-darmstadt.de, http://www.linglit.tu-darmstadt.de/bartsch/ 
Combinatorial constraints are commonly assumed in linguistics to be either based on the grammatical system of a language or to be idiosyncratic constraints on the combinatorial properties of individual lexical items and not extensible in any systematic way to larger subsets of the vocabulary. Thus, the required complements of verbs are an example of a constraint of the former type, while the latter type of constraint is instantiated by lexical collocations which are commonly assumed to be individual and idiosyncratic co-selections of lexical items and are generally treated as a usage phenomenon. There are, however, subsets of the lexicon that display striking combinatorial constraints or, rather, combinatorial requirements, over and above singular lexical combinatorial preferences which are not modelled by the grammatical rules of the language. Such constraints can be shown to hold across larger subsets of the lexicon but are neither explicable as co-selection preferences of individual lexical items nor as constraints imposed by the grammatical rules of a language. Such constraints are found to govern, for instance premodification requirements of a subset of attributive adjectival past participle (APP) - noun combinations. It has thus been observed that for reasons yet to specify, the following noun phrases – comprised of an adjectival element plus head noun – are felt to be unacceptable unless further qualified by an appropriate premodifier: 
(1) (a) ?a built house (a') a newly built house (b) ?ein gebautes Haus (b') ein neu gebautes Haus (c) ?a born child (c') a recently born child (d) ?ein geborenes Kind (d') ein kürzlich geborenes Kind (e) ?a footed dancer (e') a light footed dancer (f) ?ein füßiger Tänzer (f') ein leichtfüßiger Tänzer (g) ?a prone routine (g') an accident prone routine (h) ?ein trächtiger Verlauf (h') eine unfallträchtiger Verlauf 
In a paper on adjectival passives, Levin, Rappaport (1986: 634) observe that „some APPs sound peculiar unless qualified, for reasons that are not entirely clear”. The unacceptability and unnaturalness of the non-premodified examples in the left column is, indeed, neither explicable based on grammatical rules nor is it attributable to singular idiosyncrasies of individual items. What makes this phenomenon so interesting is precisely the fact that it must be explained in terms different from other lexical or grammatical phenomena. This paper attempts to pursue this issue in terms of the information content of these phrases and their constituents. It also addresses the question what influence the information structure of utterances has on linguistic constraints, on the one hand, and discusses the language user's sensitivity to constraints on the amount of information comprised in an utterance on the other. Another aspect that is of interest in the context of this phenomenon is whether the constraints observed in the English data can also be shown to hold cross-linguistically in other languages which would allow statements about the universal validity of the constraint across languages. Surprisingly, relatively few studies have addressed this issue so far (e.g. Grimshaw & Vikner, 1993; Ackerman & Goldberg 1996), and none, to my knowledge, have made use of corpus evidence, especially in more than one language. This paper reports findings on this phenomenon based primarily on examples from two corpora of English and German, the British National Corpus (BNC) and the tagged subsection of the COSMAS corpus collection of contemporary German at the IDS (Institut für Deutsche Sprache) in Mannheim, Germany. The paper investigates premodification constraints based on a corpus study of English and German. It addresses the following questions: 
1) what properties of this type of structure and its constituents lead to this (obligatory) premodification requirement? 
2) does this type of constraint allow for generalisations across a wider range of data beyond adjectival past participle plus noun phrases? 
3) what does this type of constraint tell us about the interplay of language structure and world knowledge? 
4) can this constraint be shown to hold cross-linguistically in other languages e.g. German? 
The paper traces the premodification requirement in the examples above to the information content of the constituents involved and hypothesizes that the language user's world knowledge about the 
92
properties of the head noun referent governs the amount and type of information that can be meaningfully contributed by an attributive adjective. This reasoning is in line with Ackerman & Goldberg's (1996: 28) conclusion that “APPs can only occur if they are construable as predicating an informative state of the head noun referent.” 
Based on a corpus-study of the phenomenon just outlined, the paper investigates the mechanisms responsible for the premodification requirement by a comparison with similar structures that do not require obligatory premodification. The paper also attempts to extend the findings to related structures besides those containing an adjectival past participle. A comparison of English and German examples of this type of phrase seeks to establish the cross-linguistic validity of the postulated premodification constraint. 
The paper postulates a pragmatic motivation for the above mentioned constraints on premodification. This postulate is based on the observation that the provision of a surplus of information by the attributive adjective violates pragmatic constraints such as conversational maxims (Grice 1975) on the informativeness of an utterance. More precisely, it is postulated that it is an overlap of information comprised in the adjectival and the noun constituent which results in a level of redundancy that renders the examples above unacceptable unless further qualified by an appropriate premodifier. Redundancy, which is defined here as a surplus of information, is standardly assumed to be recovered by a reduction of the information provided. Paradoxically, in the structure under study here, the redundancy is recovered by providing additional information by means of a premodifier that further qualifies the information provided by the noun phrase in order to establish an informative state. The paper raises the question how our knowledge of the world leads to or result in constraints on what can be or has to be expressed explicitly, which types of information are perceived to be redundant and how avoidance of redundancy is reflected in constraints on or requirements of premodifying elements in the above mentioned types of structures. 
1. The nature of the pattern under study 
The paper is specifically concerned with the premodification requirements of a subset of English and German phrases of the basic pattern: 
obligatory premodifier (adverb, adjective, noun) + adjective + noun 
As it became apparent in the course of the study that the constraints investigated originally for APP plus noun phrases are also found in at least one other type of phrase of the above mentioned basic structure, namely obligatorily premodified desubstantival adjectives, it was decided that the following patterns should be analysed: 
I. premodifier + adjectival past participle (APP) + noun: (2) (a) a newly built house (b) ein neu gebautes Haus 
II. premodifier + desubstantival adjective (DesubstAdj) + noun (3) (a) a light footed dancer (b) ein leichtfüßiger Tänzer 
The following sets of examples illustrate the phenomenon for pattern I. (examples (4) and (6)) and II. (examples (5) and (7)) in German and English respectively: 
(4) (a) a nobly born orphan HH1 (‘a) ?a born orphan (b) a British born woman EA1 (‘b) ?a born woman (c) freshly baked bread C8S (‘c) ?baked bread (d) specially designed dashboard J1T (‘d) ?a designed dashboard (e) a lightly built rugby player CL2 (‘e) ?a built rugby player (f) one of the most injury-prone cricketers in the country CBG (‘d) ?a prone cricketer (g) the guilt ridden middle classes CE6 (‘d) ?the ridden middle classes (h) the traffic ridden part of Argyle Street GWL (‘d) ?the ridden part of Argyle St. (i) a naked, angst ridden, shorn haired young man ART (‘d) ?a ridden, haired young man 
(5) (a) to arrive empty handed (a') *to arrive handed (b) a hard headed woman (b') *a headed woman (c) a scatter-brained person (c') *a brained person (d) a thick leafed plant (d') *a leafed plant (e) a hot-blooded character (e') *a blooded character 
93
(6) (a) ein neugeborenes Baby (a') *ein geborenes Baby (b) frisch gebackenes Brot (b') *gebackenes Brot (c) ein schlank gebauter Spieler (c') *ein gebauter Spieler (d) ein verletzungsanfälliger Spieler (d') ein anfälliger Spieler (e) ein kurzhaariger Mann (e') ein haariger Mann 
(7) (a) eine starrköpfige Frau (a') *eine köpfige Frau (b) eine leichtsinnige Person (b') *eine sinnige Person (c) eine dickblättrige Pflanze (c') ?eine blättrige Pflanze (d) eine dickstämmige Birke (d') ?eine stämmige Birke (e) ein heißblütiger Charakter (e') *ein blütiger Charakter 
While the phrases in the first column of example sets (4) - (7) are well-formed and perceived to be natural, acceptable and grammatical, the examples in the second column are mostly perceived as non-acceptable, unnatural and deviant in some unspecified way. The only difference between the phrases in the first column and those in the second column lies in the presence or absence of a premodifying element. It is thus reasonable to seek for an explanation of the perceived unacceptability of the non-premodified phrases in the role of the premodifying element and to ask why these phrases should require premodification in order to be rendered acceptable. After all, as the following examples show, other phrases of the same structure, APP/DesubstAdj plus noun, exist which are perfectly acceptable without premodification: 
(8) (a) It is now a declared policy of many governments and international agencies that the only vehicle for such preparation is ‘education, education, education', … (b) A balanced diet includes a variety of foods from all 5 food groups. (c) […] that grew out of the post-World War I settlement of the broken Ottoman Empire. (d) The born loser. (Comic title) (e) A married father of three. (f) He scored his first headed goal. (g) headed notepaper (h) a hung parliament 
(9) (a) …, denn ab 11:30 gibt es gebackenen Fisch auf dem Fest. (b) Die getötete Person ist circa 1,80 m groß. (c) gehackte Kräuter (d) Ein verheirateter Vater dreier Kinder. (e) Es ist der erklärte Wunsch der Parteien …. (f) eine ausgewogene Diät. 
Contrasting these sets of examples in (8) and (9) with the previous sets of examples raises the issue what exactly the difference is between those adjectival past participle/desubstantival adjective plus head noun phrases that do require premodification to be rendered acceptable, and those that do not require premodification. Two previous studies offering explanations of this type of structure and premodification requirement are discussed in the next section. 
2. Previous studies: an event structure account versus a non-redundancy account 
In a paper on English adjectival past participles, Ackerman & Goldberg (1996: 17-30) postulate that it is the amount of information provided by an expression rather than the event structure of the verb underlying the APP that gives the relevant clue to the unacceptability of the above mentioned phrases unless they are premodified. Ackerman & Goldberg conclude that “adjectival past participles can only occur if they are construable as predicating an informative state of the head noun referent” (ibid.: 28). Another previous study (Grimshaw & Vikner 1986) of adjectival past participles which is critically discussed in Ackerman & Goldberg (ibid.: 19-20) traces the premodification requirement of adjectival past participles based on creation verbs to the event structure of the verbs underlying the adjectival past participles. This event structure account argues that the underlying creation verbs display an event structure with two subevents, a process and a state and that as the APP only serves to “identify” the state subevent, the process subevent must be specified by the adverb premodifier. As Ackerman & Goldberg (ibid.) correctly observe, this event structure based account would have to be assumed to hold for all verbs with this event structure, i.e. for all accomplishment verbs, not just for creation verbs. Yet, many change of state verbs such as broil, cool etc. can be used without obligatory premodification without being rendered unacceptable. 
In this light, Ackerman & Goldberg's account based on the observed redundancy of the non-premodified phrases appears more plausible and also has the advantage of promising to cover a wider 
94
range of data. Building on the notion of redundancy introduced by Ackerman & Goldberg (ibid.), this paper explores the impact of the information content of the constituents in the phrases under study on the level of redundancy 
It will therefore be argued in this paper that the requirement of an obligatory premodifier in a subset of the data can be attributed to the information content of the expressions under study and the knowledge of the semantic frameworks of the constituents that the language user (has). It will be demonstrated, furthermore, that while on the one hand redundancy is avoided in language and information processing and is usually recovered by omission of redundant information, we are dealing here with an example in which the redundancy of the adjective plus noun phrase is recovered by an added premodifier. This premodifier, in turn, is rendered obligatory by the fact that the plain combination of adjectival past participle / desubstantival adjective plus noun expresses a non-informative and therefore non-permissible state. 
3. The semantic nature of the structures under study 
As was already indicated, the motivation for the required premodification of the adjective plus noun phrases above lies in the semantics of the adjective and the noun, and specifically in the knowledge of the language user about the inherent properties of the head noun referent and the information contributed by the attributive adjectival modifier. It will thus be necessary to give a characterisation of the semantic content of the noun in relation to the information provided by the adjectival constituent. 
APPs are best characterised as denoting resultativity, i.e. a state resulting from an action or event described by the verb they are derived from. Syntactically, they function as adjectival elements modifying nouns or pronouns or as predicative adjectives. Desubstantival adjectives are in so far related to adjectival past participles and parallel in this context as they fulfil the same syntactic function. They likewise denote a state or property of their referent. 
The head nouns of these phrases denote either a referent or object that is, in a sense, the end-point or result of the property introduced by the APP (a newly born baby ¨ ‘having been born is an inherent property of a baby'), or that has the property introduced by the adjective as an inherent property, or a referent, object or artefact (a hard-headed woman ¨ having a head is an inherent property of a woman, hence, we cannot speak informatively of a headed woman). There thus appears to be an overlap of information between the adjectival past participle / desubstantival adjective and the head noun, i.e. the adjectival modifier provides information about its head noun referent which is already tacitly understood by the language user. The non-premodified adjectival constituent thus provides a surplus of information that is deemed uninformative and therefore perceived to be redundant and unacceptable by the language user. Based on this observation a pragmatic motivation for the perceived unacceptability of the non-premodified and the requirement of a premodifier in a subset of these structures is postulated. 
The next section introduces a classification of the types of relations between adjectival constituent and head noun which will serve as a basis for the following discussion of the semantic and pragmatic reasons for the obligatoriness of the premodification in these phrases. 
4. Classification of the data 
In general terms, it may be said that premodification of the adjectival constituent, APP or DesubstAdj, in the examples quoted here seems to be the rule rather than the exception even though some of the following adjectival past participle/desubstantival adjective plus noun phrases will occasionally be found without a premodifying element. These cases are, however, heavily dependent on some kind of contrastive context. It also must be stressed that dependence of the acceptability of many of the phrases below on the premodifying element is often a matter of degree rather than a matter of absolute decisions. In this section, a set of phrases from the BNC and the Cosmas corpus is classified according to the role of the adjectival constituent in relation to the head noun it modifies. The classification is based on Bartsch (forthcoming, 2003). 
1) Adjectival element denotes an inherent property of the modified head noun 
In the first type of phrase in this classification, the adjectival element denotes an inherent property of the modified head noun. In these examples, an adjective which is obligatorily premodified denotes an inherent property of the noun it modifies. For example, in the phrase 
(10) (a) fearlessly built industries 
95
the adjectival past participle built denotes the fact that industries as a man-made artefact must, inherently, be built in order to come into existence. Even in the example 
(11) (a) the thickly carpeted mountainside (b) (…) statt üppiger Vegetation nur spärlich bewachsenen Untergrund 
the DesubstAdj carpeted metaphorically denotes the fact that a mountainside is assumed to have some kind of layered natural covering (e.g. moss, grass etc.). In this case, the particular selection of adjective is a metaphorical extension indicating the quality of the covering. In the German example, the information provided by the adverbial premodifier that the ground is sparsely covered makes the phrase sound more natural, even though it has to be said that the German phrase is also acceptable without the premodifier, arguably at the expense of its naturalness. 
A number of subtypes of this structure are distinguishable in the corpus data: 
1.1) Construction / shape / configuration: 
In this subtype, the adjective refers to the construction, shape or configuration of the head noun referent. 
(12) (a) […] I guess that the slightly askew likes of Strangelove should be welcomed with open charms. CHB 
(b) Working as I do for a technically based industry, I hope I may be forgiven for believing that almost anything can be achieved through technology, […]. EA8 
(c) […] that when a lightly built eight-stone person runs around on a hard surface, his/her knees suffer a momentary weight equivalent to three-quarters of a ton with each step ASD 
(d) Matchsticks almost fell out stepping back awkwardly like a badly constructed puppet. CEC 
(e) After 1970 the newly created Department of the Environment, […]. G05 
(f) […] a surprisingly hard-edged collection of songs. C9K 
(g) These fossil seeds are preserved in a relatively coarse-grained sandstone. AMM 
(h) […] in fact a very fine-particled clay, Montmorillionite. C97 
(i) I was astonished when we began walking down the now weed-strewn path to feel a familiar feeling of fear and expectation. CE9 
(13) (a) Das erste privat gebaute Gefängnis Deutschlands steht in […]. 
(b) […] dort den köstlichen Duft frisch gebackener Plätzchen schnuppern, […]. 
(c) […] der auch Inhaber des neu geschaffenen Neurochirurgie-Lehrstuhls ist […]. 
1.2) States of development / creation / preparation 
The following phrases denote states of natural development and man-made creation or preparation. These are felt to be intrinsic to the object denoted by the head noun and can therefore only be addressed explicitly when this information is sanctioned by further modification and specification of the nature of the properties. Thus in the first two examples, (a) and (b), it is obvious that a son or an animal would have been born or bred in order to come into existence as a living being. The information that a son, or indeed a living being, is born or an animal bred is, by itself, redundant and therefore not permissible in this construction. Yet, as soon as qualification is added in the form of a specification of the circumstances of the birth of the son or the breeding of the animal, the expression is sanctioned as providing new and relevant information. 
(14) (a) Here two men, a ship's officer and a gently born younger son sent from England to make his way in the world, are involved in certain events […] EC8 
(b) […] in most sheep rearing countries (although not Australasia or Japan); transmissible mink encephalopathy, occurring in rare outbreaks in captive bred animals on mink farms, mainly in North America; chronic wasting disease of captive mule deer and elk, seen in North America; […] EC7 
(c) […] preventing certain grievances from developing into full fledged issues. G1G 
(e) Shostakovich's First Violin Concerto is a more rough-hewn affair. A5E 
(f) […] Ruth privately thought – there were freshly baked rolls, split and filled with ox-tongue. CB5 
(g) Ockleton pointed in turn at a broadly built man smiling conventionally towards the camera […] H8T 
The German examples are slightly less clear-cut. Example (a) below is also permissible without the premodifier, example (b) must be judged differently because it is a metaphorically extended meaning, but example (c) corresponds to the mechanisms already explained for the English examples above; we can neither speak of a schultrig gebauter Mann (‘shouldered built man') nor of a gebauter Mann (‘built man') without further specification. 
(15) (a) […] das von seiner Frau nichtehelich geborene Kind zu adoptieren. 
(b) in relativ altbackener Kleidung 
(c) der breitschultrig gebaute Mann 
96
2) inalienably possessed parts: 
This set of examples comprises adjectival elements referring to inherent and inalienably possessed parts of – mostly – living beings, i.e. humans, animals, plants. The part denoted as a property or part of the living being is perceived to be an intrinsic part of this living being. So much so, in fact, that mentioning this part explicitly requires the provision of further information in order to be sanctioned as new and relevant information. This type of the structure is one of the most frequent and reliable instantiations of obligatory premodification. It rests on the knowledge of language users regarding parts of living beings which is part of language users’ world knowledge. Such world knowledge is much more inseparable from purely linguistic knowledge than has been suggested at times in linguistic research. As will be shown below, this is an important factor in accounting for the obligatoriness of the premodification of the adjectival elements in these examples. 
2.1) inalienably possessed body parts 
In these examples, the adjectival element functions as a modifier to a noun denoting an inseparable part of a living being (human, animal, plant) that is inherent or inalienable to its owner. In the example: 
(16) (a) a harshly-faced person 
the adjective faced denotes an inherent and, in fact, inalienable part of the human being, i.e. a person inherently has a face which is an inalienable part of his or her physiology. When the adjective faced is used in the context of a human being, it is tacitly understood that every human being has a face, in fact, so much so that this detail must not be mentioned explicitly unless there is good reason to do so. Good reason for mentioning such attributes or properties is the provision of additional information about the inherent part which goes beyond the mere statement of the existence of this part. Thus, stating about a human being that he or she has a face is only permissible if this information is further specified beyond what is already known as part of the language user's knowledge of the properties of living beings. In each of the examples below, the adjectival constituent must be further premodified because the combination of the information provided by the desubstantival adjective plus the head noun is, by itself, redundant. The premodifier contributes the information that recovers the unacceptable and unnatural redundancy of the non-premodified expressions. 
(16) (b) And it reached her; in a totally unwelcome manner she seemed trapped in a web spun by golden eyes, a harshly boned face, a sensual mouth that often hid its humour. HA7 
(c) The house was very mucky and rotting food spilled in the side alley next to it which attracted the most bleary-eyed flea-ridden dogs. H9G 
(d) […] Mrs Beattie and Mrs Friar could hardly be classed as neighbours and even the most cock-eyed optimist would never even hope that they might even begin to be pleasant to each other. ATE 
(e) […] spend the rest of my life playing it, thickened with doleful dirges, vainly trying to lay the trauma, my only satisfaction the ashen faced, staring eyed audiences staggering out at the end of performances, primed, and ready to carry on the good work. A6C 
(f) Nuggett is the hot headed one. K20 
but: But in the 86th minute of a frantic Roman derby, he produced a stunning headed goal as the game ended 1–1. CEP Ballpoint pen and carbon-paper suffice for ‘one-off’ letters written on donated headed notepaper.HHP 
The two last phrases under (16)(f) show that the phrase can also be used without premodification under the condition, as instantiated in these examples, that the adjectival past participle be used in a sense that does not overlap with tacit information language users already have about the head noun referent. In this case the phrase headed goal is used not in the sense ‘goal with a head’ parallel to ‘person with a head', but in the sense ‘goal scored with or by means of the head'. The expression is thus non-redundant and does therefore not require an obligatory premodifier. The same is true of the phrase headed notepaper in which the adjective headed contributes an informative state about the notepaper, namely that it has a printed header. This is a property that is not inherent to or prototypically assumed of notepaper; thus, the information provided by the adjectival constituent of the phrase is not redundant and does therefore not require premodification in order to be rendered informative. 
(17) (a) die unterschiedlichen Pigmente blauäugiger, hellhäutiger Menschen 
(b) ein bleichgesichtiger Mensch 
(c) Barfüßig und im Schlafrock parodiert …. 
(d) ein fahlgesichtiger junger Mann 
(e) ein kurzhaariger Mann 
2.2) inalienably possessed plant parts 
These examples are analogous to the examples of inalienably possessed body parts in 2.1) above. They refer to parts of plants such as branches, leaves and the trunk that are perceived to be inalienable, 
97
characteristic, prototypical parts of plants. Because a tree is assumed to have these parts, mentioning them explicitly as parts of a tree is redundant to the proficient language user unless this information is sanctioned by further specification. 
(18) (a) "A slight sudden puff of breeze lifted a thickly leafed branch in front of her and she saw two figures, framed by the foliage at the instant their lips met […].” H9H 
(b) double-, five-, many-branched (OED 2nd: branched ppl. a.) 
(c) Sprinting across it, he reached a particularly thick-trunked tree at the farther edge. HJD 
(19) (a) Gelbrandige Spielarten bleiben buntblättrig, wenn man den grünen Mittel […] 
(b) […] vierblättrige Kleeblätter aber zerstörte … 
(c) Und daß neben dem Waldvögelchen die breitblättrige Stendelwurz schon einen g[…] 
3) cognition, perception, communication: 
This category comprises adjectival elements referring to abilities and properties of the head noun referent that are associated with cognitive skills, with perception and communication. In some cases, e.g. spoken, voiced, it is difficult to decide whether speech or the voice is a part of the human body (in which case these examples would have had to be classified under 2.1) above or whether they belong to the more abstract category of cognition, perception and communication. This latter interpretation was given preference because the voice or the capacity for speech is not a part of the human body in the same way that e.g. the limbs and head are. 
3.1) cognitive and perceptive skills and properties 
This first set of examples comprises phrases with adjectival constituents that refer explicitly to cognitive abilities (e.g. minded) or perceptive skills (e.g. sighted; -sichtig). Most examples found in the BNC refer to distinctly human skills and properties. 
(20) (a) He was humble minded and never forced his opinions down other people 's throats. EVH 
(b) In view of the deeply felt resentment at the reign of Bayezid I on part of at least the more pious elements of the state, […]. H7S 
(c) One was the detestation by the liberally oriented of religious paternalism, a mild form of anti-clericalism. A07 
(d) Ernest Gowers, a very far-sighted man, had no hesitation in suggesting a separate council for Scotland and this is what happened; […] K5M 
but: the sighted UFO 
(21) (a) […] nur die weitsichtige Entscheidung der Regierung kann schlimmere Folgen abwenden 
3.2) communicative faculties 
A subtype of the type this type of pattern refers to the human faculties in the context of verbal communication. Again, it is interesting to note that a person would be said to be just ‘spoken’ in the sense of having the faculty of speech because speech is obviously perceived as an inherent human faculty. 
(22) (a) (…) and this time an impeccably spoken man replied, albeit with another question. CRE 
(b) An initially kind-voiced woman took down my details (…) CAG 
(23) (a) […] der vielstimmige Chor der Waldvögel. 
4) affected states 
This last set of examples involving obligatory premodification of adjectival past participles denote affected states or physical conditions. 
(24) (a) The most seriously-affected areas are Scotland, Wales and the Pennines. J2V 
(c) […] the reasons for the supposedly violence-prone nature of the UMW, […] G1H 
(d) This problem was to make it the most accident-prone routine in Tiller history. B34 
(e) Bed ridden patients at St Mary 's A St Mary Abbots Hospital […]. KCN 
(g) […] a large number of the rural population […] are poverty stricken. AN3 
(25) (a) die von der Hochwasserkatastrophe am härtesten betroffenen Gegenden 
98
In some of these examples, e.g. (24) and (25) (a) and (b), it is not entirely clear that the premodifier is obligatory. The following modification of sentence (a) without the premodifier seems to be permissible without sounding unnatural or providing redundant information: 
(a') The most affected areas are Scotland, Wales and the Pennines. J2V 
A brief note on the level of establishment of this premodification pattern in both English and German might not be out of place: Looking at the structural types of the phrase pattern under study, it is interesting to note that the constraint on premodification is deeply entrenched in the language, indeed. So much so, in fact, that many of the premodifier plus adjectival constituent combinations are spelled not as two separate words anymore, but are hyphenated or even spelled as one word. German has a speciality here in linking some of the premodifiers to the adjectival past participles by means of a so-called Fugen-s (e.g. verletzungsanfällig – injury-prone), but as this feature is not specific to the pattern under study, the phenomenon will not be discussed any further here. It is interesting, however, that there is a strong indication that many of these phrases are becoming lexicalised as so-called complex participles which are spelt with a hyphen or as one word and many of which have an adjectival constituent that cannot be directly derived from an underlying verb. Here are some examples: 
(26) (a) rough-hewn, accident-prone, weed-strewn 
(b) altbacken 
It should be mentioned explicitly that both English and German also allow different types of premodification such as attributively premodifying temporal or circumstantial clauses providing the same type of information as attributive premodifiers, however, these alternative structures are not the concern of this study. 
The examples discussed under 1.) - 4.) in this section as instantiations of an obligatory premodification requirement are only some of the most prominent ones discovered. More research might still lead to subtler classifications, but the classification undertaken here serves to illustrate the different areas of mostly tacit conceptual knowledge that the language user brings to the interpretation of such expressions. It is this knowledge of the proficient – an obviously knowledgeable – language user that leads to the perceived redundancy of the adjectival past participle / desubstantival adjective plus noun patterns when used without premodification. The language user's tacit knowledge about prototypical and inalienable properties of a human body or other living beings or the processes by which living creatures and artefacts come into existence render redundant information unnatural and unacceptable. Only by adding premodification can phrases such as the ones discussed above be recovered to an informative and therefore acceptable state. Language users have strong intuitions of the amount and types of information they will accept as informative and permissible and reject such expressions that violate this requirement. The examples discussed in this paper have given but a small glimpse of the extent to which conceptual knowledge influences and shapes linguistic combinatorial requirements. Conceptual knowledge brings about a combinatorial requirement that is neither covered by the grammar not by the collocational co-selection requirements of the language. An interesting question is thus, how allegedly extralinguistic information on this type of constraint can be incorporated in linguistic theory. The next section proposes a solution. 
5. Recovering redundancy: pragmatic constraints on attributive modification 
As was already indicated, pragmatic constraints are postulated to play an important role in the establishment of collocations alongside lexical constraints. But why should the human language user be bothered by being provided a surplus of information? After all, is it not helpful and does it not ensure successful communication and information processing if as much information as possible is provided? The answer is: yes and no. On the one hand, it is, of course, true that communication can only succeed if the utterance made is sufficiently informative to be decodeable; on the other hand, communication can fail if too much, i.e. superfluous, information is provided. Sufficient information thus must be interpreted as: enough information, but not too much information. Linguistic communication is not normally maximally redundant and language systems can be shown to avoid redundancy, and what is true of elements in the language system (e.g. completely synonymous words) is also true of units of information and communication. This observation lies at the heart of Grice's (1975) conversational maxims. Grice formulated a set of maxims that can be applied to the constraints discussed in this paper. 
The redundancy which results from talking about *built houses, *born babies, *surfaced tables, *faced persons, *trunked trees, *gebaute Häuser, *geborene Babies, *blättrige Bäume, *haarige Männer etc. 
99
violates at least the conversational maxims of quantity and relation of Grice's Cooperative Principle (CP) (Grice 1975: 45-46) which entails – among other things – the following requirements: 
QUANTITY: 1. Make your contribution as informative as is required (for the current purposes of the exchange). 2. Do not make your contribution more informative than is required. 
RELATION: Be relevant. 
By explicitly speaking of *a built house and *a born child, of *a surfaced table and *a faced person information is provided that exceeds the information required for the exchange. It makes the expression redundant and therefore non-informative. At the surface, it actually makes the utterance more informative than is required because the information provided about the head noun by the APP or the desubstantival adjective can be assumed to be known by every competent member of the linguistic community. As the Cooperative Principle rests on the prerequisite that every participant in a linguistic exchange observes this principle and its maxims, it can be expected by every participant in an exchange that by modifying a noun by means of an adjective, new and potentially interesting information is provided. Stating the obvious without further qualification violates the maxim that a contribution should not be more informative than required. The information supplied by the adjectival past participle or the desubstantival adjective in the expressions discussed above is not only more informative than required, stating the obvious tacitly insinuates that the speaker either does not know the rules of conversation or violates them intentionally in the assumption that the person he/she communicates with does not know things that are obvious. 
These examples are strong evidence for the postulate that in the case of this type of expression pragmatic constraints have an influence on the type and amount of information that can be introduced felicitously into an utterance. 
6. Cross-linguistic validity of the constraint 
An issue raised at the beginning of this paper is concerned with the question of the cross-linguistic validity of this type of constraint on premodification. In other words, does this constraint on premodification hold for parallel expressions in other languages? The corpus study of English and German has been able to show that the constraint does indeed hold in both languages and can thus be assumed to be valid in both languages. However, there are also differences which have be addressed here. From the small set of examples discussed here, it looks like the English and German data generally adhere to the same principles, however, there are, as yet unsystematic differences which might indicate that different linguistic communities perceive of different types of information differently. There are, first of all, cases such as in the German examples (6)(d) and (e) above in which German permits for the non-premodified use of the following examples: 
(6) (d) ein verletzungsanfälliger Spieler (d') ein anfälliger Spieler an injury-prone player *a prone player (e) ein kurzhaariger Mann (e') ein haariger Mann a short-haired man *a haired man 
In the first example, German can speak of a prone player (anfälliger Spieler) which covers unspecifically all sorts of injuries, illnesses and other afflictions that can affect a player. In the second example, the lexical collision of *haired with the already established adjective hairy prevents the use of the adjectival past participle form per se, the form –haired is permissible in combination with an appropriate premodifier. These examples are not permissible in English without premodification. 
In a small number of cases, native speaker informants of both languages were asked to pass judgement over certain phrases which yielded interesting responses mostly corroborating the corpus findings and attempting telling explanations of the fact that a phrase was or was not perceived acceptable. An interesting intercultural difference was for example the fact that none of the English speakers would permit in any context the phrase *a killed person, whereas German speakers did not have any problems motivating contexts in which the phrase eine getötete Person could be used. There were a few other examples in which phrases were likewise found acceptable without premodification in German, but never in English. The premodified counterexamples were deemed acceptable by both group of speakers, but as the group of native speaker informants was small and by no means representative, further research with native speaker informants will have to be carried out before definite statements can be made about potential intercultural differences concerning the phrases under study in the two languages. 
100
7. Conclusion 
This paper has attempted to show in how far constraints, or requirements on premodification in a set of phrases of the basic structure premodifier plus adjectival past participle / desubstantival adjective plus head noun are shaped by the tacit conceptual knowledge that the language user brings to the processing of language. It could be shown that such constraints shape obligatory structural requirements that are clearly not covered by either the grammar of the language nor the collocational co-selection requirements of the individual lexical item. It has furthermore become clear that an explanation of the type of premodification requirement exemplified in this paper is best attempted on the basis of the information content of the phrase constituents and the ensuing redundancy if the adjectival past participle or desubstantival adjective is not premodified. It has been possible to show that findings on the premodification requirements of adjectival past participles can also be extended to other, similar types of qualifying adjectives for which similar models of explanation can be employed. As postulated, the same mechanisms based on conceptual knowledge and avoidance of redundancy govern premodification requirements both in English and German such that the question of the cross-linguistic validity of the constraint can be answered affirmatively for the two languages studied comparatively. Only some of the aspects underlying this phenomenon could be discussed here. More comparative studies are needed to investigate the alternative ways of resolving redundancy in these and similar structures. Further research must be extended to other languages and other patterns in order to establish the impact of conceptual knowledge on linguistic structures on a broader cross-linguistic basis. A question that remains unresolved and will continue to pose a challenge to linguistics is the question how this type of constraint which is obviously based on factors that have been deemed extra-linguistic by many approaches to language is to be incorporated in linguistic theory. 
References: 
Books 
Ackerman, F, Goldberg, AE 1996 Constraints on adjectival past participles. In: Goldberg, AE (ed) 1996 Conceptual Structure, Discourse and Language. Standford, Cal., CSLI. 17-30. 
Bartsch, S forthcoming, 2003 Structural and functional properties of collocations in English. A corpus study of lexical and pragmatic constraints on lexical co-occurrence. Tübingen, Verlag Gunter Narr. 
Cole, P, Morgan, JL (eds) Syntax and Semantics 3: Speech Acts. New York, London, Academic Press. 
Goldberg, AE (ed) 1996 Conceptual Structure, Discourse and Language. Standford, Cal., CSLI. 
Grice, HP 1975 “Logic and conversation.” In: Cole, P, Morgan, JL (eds) Syntax and Semantics 3: Speech Acts. New York, London: Academic Press. 41-58. 
Levin, B 1993 English Verb Classes and Alternations. A Preliminary Investigation. Chicago, London: The University of Chicago Press. 
Levin, B, Rappaport Hovav, M 1995 Unaccusativity: at the syntax-lexical semantics interface. Cambridge, Mass., MIT Press. 
Levin B, Rappaport M 1986 The formation of adjectival passives. Linguistic Inquiry 17, 623-661. 
Levinson, S 1983 Pragmatics. Cambridge, CUP. 
Nuyts, J, Pederson E (eds) 1997 Language and conceptualisation. Language culture and cognition 1. Cambridge, Cambridge University Press. 
Corpora: 
British National Corpus: http://www.hcu.ox.ac.uk/BNC/ 
The corpus comprises 100 mio. running words of contemporary British English. 
The three letter code accompanying each BNC example in the text refers to the source text from which the example was taken. 
The Cosmas Corpus Collection at the IDS (Institut für Deutsche Sprache) in Mannheim, Germany (http://www.ids-mannheim.de/kt/corpora.shtml): The corpus used for the German examples in this study is the tagged subsection of the Cosmas Corpus at the IDS (Institut für Deutsche Sprache), Mannheim, a morpho-syntactically annotated subcorpus of contemporary German. This morpho-syntactically annotated subcorpus consists of a public section of the LIMAS corpus plus four years of newsprint of the German newspaper Mannheimer Morgen (years 91 + 94 + 95 + 96). The corpus comprises 18.22 Mio. running words. 
101

Synchronic and diachronic variation: 
the how and why of sociolinguistic corpora. 
Dr. Kate Beeching, Senior Lecturer, 
Linguistics and French, Faculty of Humanities, Languages and Social Sciences, 
University of the West of England, 
Bristol, Frenchay Campus, Coldharbour Lane, 
Bristol BS16 1QY. 
Tel: 0117-344-2385; 
Fax: 0117-344-2820; 
E-mail: Kate.Beeching@uwe.ac.uk 
Abstract 
Tape-recordings have come of age. The spoken word has been captured, stored on tape and (in as yet regrettably restricted amounts) transcribed for a number of decades now and this is a situation which can only improve. Where good accompanying demographic data are available, it is beginning to be possible to trace linguistic change and to provide empirical evidence of the way that synchronic and diachronic variation may be linked. In the past, it has only been possible to guess at long-gone spoken manifestations through written, mainly literary, evidence. This paper aims to describe and illustrate the potential of (spoken) sociolinguistic corpora for research studies in both synchronic and diachronic variation, with reference to French, and to suggest ways in which useful research corpora may be established for future generations of scholars. 
The Etude Sociolinguistique d'Orléans, currently available from the University of Leuven, Belgium, was conducted by a group of university teachers of French at Reading University from 1966-70. The resulting electronic corpus comprises approximately 109 hours of spoken French, 902,755 words of which have been orthographically transcribed. with a further 13 hours of phonetic transcription. These transcriptions are available at : http://bach.arts.kuleuven.ac.be/elicop. Although French researchers, notably Claire Blanche-Benveniste and the GARS team in Aix and Mireille Bilger in Perpignan, have collected and transcribed spoken French, their corpora are not (yet?) available on line. Beeching's (1980-90) “Bristol” Corpus , which comprises 17.5 hours of orthographically transcribed speech, or 155,000 words, involving 95 speakers, aged 7 to 90 years with a balanced range of educational backgrounds, though small in scale, may be accessed at http://www.uwe.ac.uk/facults/les/staff/kb/CORPUS.pdf. 
Beeching (forthcoming b) suggests that pragmaticalisation is a process of language change during which words, used in everyday social interaction, shift in meaning or accrue a certain social semiotic, become habituated in that usage and are propagated because of a new fashion or prestige which is attached to them. Spoken corpora and corpus tools are an excellent heuristic in charting distributional frequencies or probabilistic factors. Andersen (2000) suggests that the upsurge of innit and like in the COLT Corpus of adolescent English may be more than age-grading. The present paper will present broadbrush preliminary evidence with respect to the evolution of selected pragmatic particles in French. 
102

The MEANING Italian Corpus 
Luisa Bentivogli, Christian Girardi, Emanuele Pianta 
ITC-irst 
Via Sommarive 18 - 38050 Povo (Trento) – Italy 
E-mail: {bentivo, cgirardi, pianta}@itc.it 
Abstract 
The MEANING Italian Corpus (MIC) is a large size corpus of written contemporary Italian, which is being created at ITC-irst, in the framework of the EU-funded MEANING project. Its novelty consists in the fact that domain-representativeness has been chosen as the fundamental criterion for the selection of the texts to be included in the corpus. A core set of 42 basic domains, broadly representative of all the branches of knowledge, has been chosen to be represented in the corpus. The MEANING Italian corpus will be encoded using XML and taking into account, whenever possible according to the requirements of our NLP applications, the XML version of the Corpus Encoding Standard (XCES) and the new standard ISO/TC 37/SC 4 for language resources. A multi-level annotation is planned in order to encode seven different kinds of information: orthographic features, the structure of the text, morphosyntactic information, multiwords, syntactic information, named entities, and word senses. 
1. Introduction 
A domain-based corpus can be a useful resource in different research areas. It is well known that domain-specific sublanguages exhibit specific features at various linguistic levels (Grishman and Kittredge, 1986). Linguistic analyses carried out on a multi-domain corpus can uncover differences in the lexicon and morphology, in names and named entity structures, and in lexical semantics, syntactic and discourse structure. The NLP community can find in a domain-based corpus a fundamental resource for several tasks such as, for example, parsing (Sekine, 1997), domain-dependent lexical acquisition, Word Sense Disambiguation (WSD), etc. In WSD, domain lexical information has proved to be very useful in the development of high precision algorithms (Magnini et al., 2003). 
Domain (also called topic, or subject matter) is one of the criteria for text selection and/or classification in many existing corpora. For instance, in the written component of the British National Corpus two main text selection criteria were used: “medium” and “domain”. More specifically, the BNC uses 9 knowledge domains (arts, social sciences, world affairs, etc.). Similar selection criteria (medium and domain) have been adopted also in the design of the American National Corpus (Ide and Macleod, 2001). The Brown and LOB corpora classify texts in 15 different text categories but such categories are a mix between genre labels (bibliography, popular lore) and domain labels (religion, “skill, trade and hobbies”). The NERC report (Calzolari et al., 1995), offers a summary of the classification systems used by major corpus projects in Europe, showing that domains are generally used in the classification of the texts. The same holds for the most important corpora created for the Italian language. In the SI-TAL Italian Treebank (Montemagni et al., 2000) texts have been “selected to cover a good variety of topics”. The reference corpus CORIS/CODI (Rossini Favretti et al., 2001) is structured in subsections, some of which can be compared to domains. 
However, in all the mentioned corpora, a complete representation of domains is not pursued in a systematic way. On the contrary, domain is the fundamental selection criterion of texts to be included in the MEANING Italian corpus (MIC). The MIC is being developed with the aim of supporting domain-based WSD in the framework of the MEANING project (Rigau et al., 2002). MEANING is an EU funded project which aims at enriching existing wordnets (for English, Spanish, Catalan, Basque, and Italian) by acquiring new lexical information from corpora. MEANING tries to exploit the inter-dependency between Word Sense Disambiguation (WSD) and knowledge acquisition by applying the following steps: 1. Train accurate WSD systems and apply them to very large corpora; 2. Use the partly disambiguated data in conjunction with shallow parsing techniques and domain information to extract new linguistic knowledge to be incorporated into wordnets; 3. Re-train WSD systems and re-tag the corpus, exploiting the information acquired in the second step. The result of this cycle is twofold: the enrichment of the lexical resources with information acquired from the corpus and a multi-level linguistic annotation of the corpus itself. 
The rest if this paper is structured as follows. In Section 2 the structure of the MIC is described in detail. Section 3 deals with the encoding of the corpus while in Section 4 the multi-level linguistic annotation of the corpus is illustrated with annotation scheme examples. Section 5 summarizes what has been done up to now and what are the tasks still to be undertaken. 
103
2. Corpus design 
The MIC is being created with the aim of representing the domains used in WORDNET DOMAINS (Magnini and Cavaglia, 2000), an extension of WordNet 1.6 where each synset has been annotated with at least one domain label, selected from a set of 164 labels hierarchically organized. The WORDNET DOMAINS hierarchy was created starting from the subject field codes used by current dictionaries, and the Dewey Decimal Classification system (DDC), a general knowledge organization tool that is continuously revised to keep pace with knowledge development. The DDC is the most widely used library classification system in the world and provides a very large and complete set of hierarchically structured domain labels (see DDC 1996). A core set of 42 basic domains (the second level of the WORDNET DOMAINS hierarchy) has been chosen to be represented in the MIC. A study carried out by (Magnini and Gliozzo, 2002) shows that these 42 domains have a domain-coverage equivalent to the domain-coverage of the DDC system. 
In the MIC, texts are assigned to a topic category on the basis of an existing, text-external, list of domain labels. The value of this kind of classification is one of the central controversial areas of text typology, as pointed in the EAGLES preliminary recommendations on text typology (Sinclair and Ball, 1996). This report argues that it is not possible to classify the texts produced in the world on the basis of a limited list of topics, chosen on a text-external, a priori basis; there are too many possible methods for identifying the topic of a text. Also, the boundaries between topics are blurred, and texts usually cover a variety of topics. On the contrary, the topic(s) of a text should be identified on the basis of text-internal evidence such as vocabulary clustering. 
However, while claiming that internal evidence should be the primary criterion for the identification of a text topic, the EAGLES report admits the possibility of a defensible use of topic categories based on few external criteria. These are the sectionalisation of newspapers, some topic-related classifications institutionalized in a society (in particular lists of recognised professions and educational courses), and, when existing, the self-classification of the text. 
We recognize the problems mentioned above and we agree with the position that there is no objective, scientific means of assigning topics. However, a commonly accepted topic classification scheme based on internal criteria has not been developed yet. Moreover, some practical consideration must be made. In the current corpus practice text-external criteria are widely used to assign topics to texts. As it is shown in the introduction, the topic categories given in the NERC report have a common ground in many or most of the corpora studied. The MIC is in line with the trend in corpus practice as most of the commonly used topics reported in that document correspond to our basic domains. Moreover, as we will see below, in the construction of the corpus we exploit all the acceptable external criteria mentioned in the EAGLES report. 
Coming back to the model designed for the creation of the MIC, we were faced with two requirements. First, we needed completeness, i.e. we wanted all of the 42 domains to be represented. Second, we wanted the corpus to reflect the fact that different domains do not have the same relevance in the language. To meet the completeness requirement we are creating a micro-balanced corpus composed of 42 subcorpora, each representing a basic domain. On the other hand we are creating a macro-balanced corpus, i.e. a homogeneous corpus of the contemporary Italian language created without taking into account the domain criterion but in which we know that most domains are represented. This corpus will allow us to verify in an independent way the relevance of the different domains in the generic language. Figure 1 shows the overall structure of the MIC. 
As regards the other corpus building criteria, the MIC represents only the written (electronic) mode, relying on written texts already available in electronic form. The media used are essentially three: newspapers, press agency news, and web documents. The genre is mainly that of informative, “factual” prose. 
An important characteristic of the corpus is that a part of it is bilingual. It includes 5 million words of aligned parallel English/Italian news and the first version of MultiSemCor (Bentivogli and Pianta, 2000), which is a bilingual aligned parallel corpus semantically tagged with a shared inventory of senses. Up to now MultiSemCor consists of 30 English texts of the SemCor corpus (a subsection of the Brown corpus semantically tagged with WordNet senses) along with their Italian translations, for a total of about 120,000 words. 
104
Figure 1 Corpus Composition 
2.1 The micro-balanced component 
The micro-balanced section of the MIC will be composed of 42 subcorpora representing the 42 basic domains selected from WORDNET DOMAINS (reported in Table 1). To create the subcorpora, we take into account the whole hierarchy of WORDNET DOMAINS. This means that for each subcorpus we look for texts belonging not only to the corresponding basic-level domain, but also to the more specific domains related to it in the hierarchy. It is important to underline that the micro-balanced corpus is not composed of specialized texts as we do not aim at creating specialized corpora but a general language corpus in which all the domains are covered. Given the fact that the 42 basic domains seem to have a different absolute relevance, we distinguish major domains (e.g. Economy and Sport) and minor domains (e.g. Linguistics and Astronomy). Each major domain subcorpus will include 2 million words while the minor subcorpora will be composed of 1 million words each. 
The texts to be included in the micro-balanced corpus come from three main sources: press agency news, newspaper weekly special supplements, and web documents. Each domain subcorpus should be balanced with respect to the three media; however for some subcorpora most of the texts will be web documents as it is unlikely that we will be able to find enough news or supplements belonging to those domains (see for instance Mathematics, Pedagogy, Archeology). 
2.1.1 Press agency news and special supplements 
The press agency news were collected through the Excite (http://www.excite.it) and Virgilio (http://www.virgilio.it) portals. They come from the following press agencies: Reuters, ANSA, ASCA, DataSport, and ADNKRONOS (parallel Italian/English). Supplements come from a wide circulation newspaper called “La Stampa”, which contains weekly special supplements dealing with science (“Tuttoscienze”), books (“Tuttolibri”), finance (“Tuttosoldi”), television (“TV”), cars and motorbikes (“Speciale motori”), agriculture (“Speciale agricoltura”), Italian elections (“Speciale elezioni”) and local events in the town of Turin (“Speciale citta” and “Torinosette”). 
To speed up the creation of the micro-balanced corpus, we explored the possibility of developing a methodology for the (semi-)automatic classification of news and special supplements to be assigned to the various subcorpora. This is made easier by the fact that both press agency news and special 
Administration Artisanship Computer science Law Philosophy Sexuality Agriculture Astrology Earth Linguistics Physics Sociology Alimentation Astronomy Engineering Literature Play Sports Anthropology Biology Economy Mathematics Politics Telecommunications Archaeology Body care Fashion Medicine Psychology Tourism Architecture Chemistry History Military Publishing Transport Art Commerce Industry Pedagogy Religion Veterinary 
Table 1 Basic domains in WordNet Domains - Version 1.1 

Newspapers 

News 

Web documents 

Newspapers 

Web documents 

News 

Web documents 

Newspapers 

News 

Newspaper 

Newspaper 

… 42 

DOMAIN-2 

DOMAIN-3 

DOMAIN-1 

MICRO-BALANCED 
CORPUS 

MACRO-BALANCED 
CORPUS 

MEANING CORPUS 
105
supplements are already classified by the publishers with two different kinds of information: broad topic and keywords. News are divided into 9 broad topics, namely economy, politics, cars and motorbikes, artistic performances, sports, science and technology, generic news, foreign news, local news. Also the 9 special supplements can be considered self-classified in 9 broad topics given in their title (science, finance, etc.). Moreover, one or more keywords, often corresponding to domain labels, are always associated to each piece of news and supplement article. 
We studied how to develop a procedure able to exploit this information to (semi-)automatically assign news and supplements to the appropriate domain. To develop and evaluate the procedure, a development and a test set have been created for both news and supplements. The development set is composed of all the 20,399 news collected from the Excite portal in five months, from April to August 2002. As the news constitute an always growing open set, it is important to verify the productivity of the procedure when applied to news belonging to a different period of time. For this reason, the time-span covered by the test set was kept different from that of the development set. The test set was created by selecting 500 out of 15,014 news, collected in five months from September 2002 to January 2003. The news in the test set were chosen randomly, keeping temporal distribution and the proportions of the broad topics with which they were classified by Excite. 
As regards supplements, we had at our disposal newspapers special supplements covering a time-span of 10 years (from 1992 to 2001), for a total of 66,927 articles. As these supplements represent a closed set, both the development and test set can be selected from the same period of time. We selected 30,000 articles for the development set and 500 for the test set, randomly chosen but homogeneously distributed over the 10 years and keeping the proportions of each broad topic. 
The news composing the two test sets were read and manually assigned to the appropriate domain. Note that the texts have been assigned to the most specific domain available among the WORDNET DOMAINS. 
As a first step, we tried a simple algorithm that can be considered as a baseline for our experiment. We associated to each domain a set of Italian words currently used to refer to that domain. This set of words (domain word set) has been manually created and contains the lemma of the domain, possible morphological variants, and possible synonyms. As an example, the following is the domain word set associated to the pharmacy domain: 
Domain: PHARMACY 
Domain word set: farmacia (pharmacy), farmaceutica (pharmaceutics), farmacologia (pharmacology) 
Then, for each domain a procedure looks for a matching between words in the domain word set and keywords associated to the texts. This procedure assigns a text (piece of news or article) to a domain if at least one of the words contained in the domain word set corresponds to one of the keywords associated to the text. The procedure exploits information about keywords as their granularity is similar to that of domains. The 9 broad topics are too generic to be useful for the baseline algorithm.This procedure relies entirely on a priori information, as it does not require any kind of analysis of the development set 
The results of the application of the baseline procedure to the test sets of the news and the special supplements are shown in Table 2. In the evaluation, the domain assigned by the procedure and the manually assigned domain are considered to match if they are equal or if they have a common ancestor that is a basic domain. 
Since these results show -especially for the news- a very law recall, a second procedure has been developed, based on a number of rules manually written on the basis of the study of the development set. These rules exploit wider information than the baseline algorithm: 
• keywords that are not in the domain word set, but are somehow related to the domain 
• the broad domains 
• the words in the title 
Table 3 shows three sample rules, which apply to the pharmacy and the computer science domains. The first rule only considers information about the text keyword(s). The second rule looks at both keywords and words in the title. The last one considers both keywords and broad topic. 
Precision Recall Coverage News 0.72 0.15 0.20 Special supplements 0.54 0.56 0.70 
Table 2 Performances of the baseline procedure on news and supplements 
106
PHARMACY if KEYWORD = farmacia (pharmacy) or farmaceutica (pharmaceutics) or farmacologia (pharmacology) or farmaco (medicine) or vaccine (vaccine) PHARMACY if 
and KEYWORD = epatite (hepatitis) or morbillo (measles) or meningite (meningitis) or antipolio (polio) or virus (virus) 
TITLE = vaccine (vaccine) COMPUTER SCIENCE if 
and KEYWORD = internet (internet) 
BROAD TOPIC = tecnologia e scienza (science and technology) 
Table 3 Examples of rules 
This second procedure has been developed only for the press agency news and gives the results shown in Table 4 below. The precision decreases but both recall and coverage improve significantly. Unfortunately, despite these improvements, the precision of both algorithms is still unsufficient to avoid manual intervention. Thus, we plan to use the results of the application of the second algorithm for supporting humans in creating the 42 subcorpora. Manual work will be speeded up as corpus builders will have to check if the assignment of the text to a domain is correct or not, a task which is much simpler than assigning a text to one of the 42 subcorpora. 
In order to test the applicability of the hand-written rules -developed for the press agency news- also to other texts, we applied them also to the articles of the special supplements. The application of the second procedure to the supplements does not change significantly the results obtained with the baseline algorithm: precision goes from 0.54 to 0.53, and recall from 0.56 to 0.57. These results demonstrate that the rules created on the basis of the press agency news are specific to the news themselves and cannot be reused for different kinds of texts. 
2.1.2 Web documents 
The third main source of texts to be included in the micro-balanced corpus is the web. The web gives access to colossal quantities of texts of any type and more and more linguists and language technologists rely on it as a huge source of corpus materials (see Kilgarriff, 2001). The MEANING project itself treats the web as a corpus to learn information from it, with the final aim of opening the way for a concept-based access to the Multilingual Web. 
Despite web's usefulness for corpus research, when trying to collect web documents we have to face several problems: the web contain duplicates or very similar documents, not all documents contain enough text, they may contain mixes of languages, and so on. As it is impossible to visit, download and manually classify some of the millions of web pages, we are at the moment studying how to devise automatic methods to draw materials from the web for inclusion in the corpus. 
Precision Recall Coverage News 0.64 0.44 0.55 
Table 4 Performance of the second procedure on the news 
2.2 The macro-balanced component 
The macro-balanced corpus is being created in order to evaluate in an independent way the relevance of the domains in a generic corpus. This corpus is not intended to be a reference corpus for the Italian language, as it is not balanced with respect to different literary genres, media, modes, and styles. It is a homogeneous corpus composed of two general high circulation newspapers (“La Repubblica” and “La Stampa”) in which we expect most domains to be represented. The macro-balanced corpus contains about 90 million running words covering a time-span of 4 years (1998-2001). This time-span has been chosen in order to keep the corpus comparable with the other corpora of the MEANING consortium. 
We assume that in the selected material the most common topics dealt with in periodical are represented, giving us a picture of the distribution and proportions of the topics within the corpus. This will allow us to verify the relevance of the different domains in the current language. Table 5 summarizes the data about the texts we included in the corpus. 
Size (tokens) Time-span La Repubblica 38 millions 2000-2001 La Stampa 48 millions 1998-1999 
Table 5 Structure of the macro-balanced component 
107
3. Corpus encoding 
The corpus will be encoded using XML as a common data format. We will take into account, whenever possible according to the requirements of our NLP applications, the Corpus Encoding Standard for XML (XCES) guidelines and the new standard ISO/TC 37/SC 4 for language resources (Ide and Romary, 2002). We chose full text as type of sample for the corpus, that is the complete newspaper article, piece of news, or other document is taken as the minimum size of the text. Each text is stored in a separate file. 
CES distinguishes three broad categories of information which are of direct relevance for the encoding of corpora for use in NLP applications: 
• Documentation, which includes global information about the text, its content, and its encoding. 
• Primary data, which consist of the text marked up with information regarding both gross structure (paragraphs, chapters, titles, footnotes, etc.; features of typography and layout; non-textual information, such as graphics, etc.) and sub-paragraph structures (sentences, highlighted words, dates, abbreviations, etc.) 
• Linguistic annotation, i.e. information added to the primary data as a result of some linguistic analysis 
In the MIC, documentation about each text will be included in the form of a separate XCES-conformant header. All the original texts are stored in the legacy corpus, which is kept as a backup corpus. Then, to obtain the encoded version of the corpus, the legacy texts undergo a series of transformations. To this extent, a number of normalization scripts have been implemented. 
In the CES guidelines primary data (i.e. the text itself marked up with information about its structure) form the so-called base or hub text. The hub text does not include linguistic annotations which are stored in separate documents and linked to the hub text or other linguistic annotation documents. In the encoding of the MIC, we follow CES guidelines in retaining linguistic annotation in separate documents. However we differ from CES in the way we treat primary data. In fact, we prefer our hub corpus to be completely plain, i.e. pure text without any type of markup (apart from carriage returns). Thus the encoding of the primary data is not kept together with the text itself: primary level information is coded in the same way as linguistic information and is stored in different files separated from the hub text. 
4. Corpus annotation 
A multi-level annotation of the corpus is planned in order to encode seven different kinds of information: orthographic features, the structure of the text (primary data, level 1 and 2), morphosyntactic information, multiwords, syntactic information, named entities (primary data, level 3), and word senses. 
All annotations are performed automatically, using linguistic tools developed at ITC-irst. Information about each level of annotation is stored in separate documents. Following the CES recommendations (see Section 3), all annotation documents are linked to the hub corpus or other annotation documents using one-way links. Two different means can be used to specify locations, namely reference to a unique identifier (ID) and reference to the position of the characters in the text We use the character position locators to link the orthographic annotation to the hub corpus. ID locators are used to link all the other linguistic annotation documents. 
In the next sections, information about the different kinds of annotation are given. All the examples reported refer to parts of the same sentence: “Il Ministero della Sanita dice che coi superalcolici bisogna andarci veramente piano” (Eng. “The Department of Health and Human Services says that people must take it really easy with liquors”). 
4.1 Orthographic annotation 
The corpus is automatically tokenized and each token is annotated with: 
• token ID 
• Location in the hub corpus 
• case (upper, lower, capitalized) 
108
Example: “Il Ministero…” (The Department…) 
<struct type="ortho"> 
<struct type="t-level" id="t_1"> 
<feat type="token">Il</feat> 
<feat type="case">capitalized</feat> 
<seg startsAt="0" endsAt="1"></seg> 
</struct> 
<struct type="t-level" id="t_2"> 
<feat type="token">Ministero</feat> 
<feat type="case">capitalized</feat> 
<seg startsAt="3" endsAt="11"></seg> 
</struct> 
… 
</struct> 
4.2 Structure annotation 
At this level of annotation, primary data (level 1 and 2) are encoded. As said before, this information is stored in a document separeted from the hub file, which contains only the pure text without any tags. The following information is recorded: 
• text divisions, paragraphs, sentences, rendition information, etc.(i.e. all structural information) 
• ID for text divisions, paragraphs, and sentences 
• link to token IDs in the orthographic annotation file 
Example: <p> Il Ministero della Sanita dice che coi superalcolici bisogna andarci veramente piano.<br> Negli ultimi anni, infatti, il numero di cirrosi epatiche e in continuo aumento. <p> ... 
<struct type="structure" xml:base="../ortho/ministero-ort.xml"> 
<struct type="p-level" id="p_1" 
xlink:href="#xpointer(id('t_1')/range-to(id('t_29')))"> 
<struct type="s-level" id="s_1" 
xlink:href="#xpointer(id('t_1')/range-to(id('t_13')))"></struct> 
<struct type="s-level" id="s_2" 
xlink:href="#xpointer(id('t_14')/range-to(id('t_29')))"></struct> 
</struct> 
<struct type="p-level" id="p_2" 
xlink:href="#xpointer(id('t_30')/range-to(id('t_32')))"> 
<struct type="s-level" id="s_3" 
xlink:href="#xpointer(id('t_30')/range-to(id('t_...')))"></struct> 
</struct> 
</struct> 
4.3 Morphosyntactic annotation 
After PoS tagging and lemmatization each token in the corpus is annotated with its morphosyntactic information, that is: 
• word ID 
• link to token ID in the orthographic annotation file 
• lemma, stem, PoS, form (when necessary), morphological features (gender, number, mood, tense, person) 
Moreover, if the word belongs to a multiword: 
• link to the multiword ID in the multiwords annotation file 
• function of the word in the multiword (head, satellite) 
As regards POS tags, the tagset applied is a subset of the tagset specified in the EAGLES Guidelines for morphosyntactic annotation. 
Example: “andarci veramente piano” (Eng. “take it really easy”) 
<struc type="morpho" xml:base="../ortho/ministero-ort.xml"> 
… 
<struc type="w-level" id="w_12" xlink:href="#xpointer(id('t_10'))"> 
<feat type="lemma">andare</feat> 
<feat type="stem">and</feat> 
<feat type="form">andar</feat> 
<feat type="pos">v</feat> 
<feat type="elra-tag">VF</feat> 
<feat type="mood">inf</feat> 
<feat type="tense">pres</feat> 
<feat type="mwd-element" 
109
xlink:href="../multiwords/ministero-mwd.xml#xpointer(id('mwd_2'))"> 
head</feat> 
</struc> 
<struc type="w-level" 
id="w_13" 
xlink:href="#xpointer(id('t_10'))"> 
<feat type="lemma">ci</feat> 
<feat type="pos">pron</feat> 
<feat type="elra-tag">+E</feat> 
<feat type="mwd-element" 
xlink:href="../multiwords/ministero-mwd.xml#xpointer(id('mwd_2'))"> 
satellite</feat> 
</struc> 
<struc type="w-level" id="w_14" xlink:href="#xpointer(id('t_11'))"> 
<feat type="lemma">veramente</feat> 
<feat type="pos">avv</feat> 
<feat type="elra-tag">B</feat> 
</struc> 
<struc type="w-level" id="w_15" xlink:href="#xpointer(id('t_12'))"> 
<feat type="lemma">piano</feat> 
<feat type="pos">avv</feat> 
<feat type="elra-tag">B</feat> 
<feat type="mwd-element" 
xlink:href="../multiwords/ministero-mwd.xml#xpointer(id('mwd_2'))"> 
satellite</feat> 
</struc> 
</struc> 
4.4 Multiwords annotation 
All expressions in the corpus which are multiwords are coded with the following information: 
• multiword ID 
• PoS, lemma 
• link to the word ID of the components in the morphosyntactic annotation file 
• function of the components words (head, satellite) 
Example: “andarci piano” (Eng. take it easy) 
<struct type="multiwords" xml:base="../morpho/ministero-mph.xml"> 
… 
<struct type="w-level" id="mwd_2"> 
<feat type="lemma">andarci_piano</feat> 
<feat type="pos">v</feat> 
<struct type="mwd-element" xlink:href="#xpointer(id('w_12'))"> 
<feat type="function">head</feat> 
</struct> 
<struct type="mwd-element" xlink:href="#xpointer(id('w_13'))"> 
<feat type="function">satellite</feat> 
</struct> 
<struct type="mwd-element" xlink:href="#xpointer(id('w_15'))"> 
<feat type="function">satellite</feat> 
</struct> 
</struct> 
</struct> 
If the multiword is present in our reference lexicon MultiWordNet1 (Pianta et al. 2002), PoS and lemma are those of MultiWordNet. 
4.5 Named Entities annotation 
All named entities in the corpus are recognized and coded as such with the following information: 
• named entity ID 
• type of named entity 
• link to the word ID or multiword ID in the respective annotation files 
Example: Ministero della Sanita (Departement of Health and Human Services) 
<struct type="namedentities" xml:base="../morpho/ministero-mph.xml"> 
<struct type="ent-level" id="e_1" 
1 MultiWordNet is a multilingual lexical database, developed at ITC-irst, in which the Italian wordnet is strictly aligned with Princeton WordNet (version 1.6) (Fellbaum, 1998) 
110
xlink:href="../multiwords/ministero-mwd.xml#xpointer(id('mwd_1'))"> 
<feat type="enamex">organization</feat> 
</struct> 
</struct> 
The tagset applied to annotate named entities is the one adopted in the framework of the DARPA/NIST HUB4 evaluation exercise. 
4.6 Word sense annotation 
Content words and multiwords in the corpus which are present in MultiWordNet are disambiguated according to MultiWordNet synsets. The annotation includes: 
• link to the word ID or multiword ID in the respective annotation files 
• MultiWordNet lemma, PoS, and synset ID 
Example: “bisogna andarci piano” (Eng. (people) should take it easy) 
<struct type="semantic" xml:base="../morpho/ministero-mph.xml"> 
<struct type="sem-level" 
xlink:href="#xpointer(id('w_11'))"> 
<feat type="MWN-lemma">bisognare</feat> 
<feat type="MWN-pos">v</feat> 
<feat type="MWN-sense">v#3990811</feat> 
</struct> 
<struct type="sem-level" 
xlink:href="../multiwords/ministero-mwd.xml#xpointer(id('mwd_2'))"> 
<feat type="MWN-lemma">andarci_piano</feat> 
<feat type="MWN-pos">v</feat> 
<feat type="MWN-sense">v#03437782</feat> 
</struct> 
</struct> 
4.7 Syntactic annotation 
Syntactic annotation will be carried out only in the last phase of the creation of the MIC. The precise encoding of the syntactic annotation has not been decided yet. However we plan to automatically annotate at least the main phrases of the sentence by using shallow parsing (phrase chunking) techniques. 
5. Summary and conclusions 
The MEANING Italian corpus has been presented in this paper. MIC is being developed in the framework of the MEANING project with the aim of supporting word sense disambiguation, however a domain-based corpus can be a very useful resource not only for natural language processing applications but also for different kinds of linguistic analyses. 
The corpus is in its way to realization. All its overall structure has been designed and the multi-level annotation scheme has been developed. The macro-balanced component has been created, normalized and linguistically annotated up to level of morphosyntactic annotation. XCES-conformant headers for each texts have been automatically created. As regards the micro-balanced component, we are collecting materials from different sources and we are devising semi-automatic procedures to speed up its construction. Our work will go on until the corpus will be entirely created and all the levels of linguistic annotation will be performed. 
Acknowledgments 
We would like to thank Pamela Forner for her huge and precious work on the creation of the corpus and Claudio Giuliano, Nancy Ide, and Tomaz Erjavec who offered helpful comments for the development of the corpus annotation scheme. 
References 
Bentivogli L, Pianta E 2002 Opportunistic semantic tagging. In Proceedings of the Third International Conference on Language Resources and Evaluation (LREC 2002), Las Palmas, Canary Islands - Spain. 
Calzolari N, Baker M, Kruyt T (eds) 1995 Towards a network of european reference corpora. Report of the NERC Consortium Feasibility Study, coordinated by Antonio Zampolli. Linguistica Computazionale XI-XII. Pisa, Giardini. 
111
Dewey Decimal Classification and Relative Index 1996, Ed. 21, edited by J.S. Mitchell, Forest Press, Albany. 
Fellbaum C (eds) 1998 WordNet: An Electronic Lexical Database. Cambridge(Mass.), The MIT Press. 
Grishman R, Kittredge R (eds) 1986 Analyzing Language in Restricted Domains. Lawrence Erlbaum. 
Ide N, Romary L 2002 Standards for Language Resources. In Proceedings of the Third International Conference on Language Resources and Evaluation (LREC 2002), Las Palmas, Canary Islands – Spain. 
Ide N, Macleod C 2001 The American National Corpus: A Standardized Resource of American English. In Proceedings of Corpus Linguistics, Lancaster, UK. 
Kilgarriff A 2001 Web as corpus. In Proceedings of Corpus Linguistics 2001 Conference, Lancaster,UK. 
Magnini B, Strapparava C, Pezzulo G, Ghiozzo A 2003 The Role of Domain Information in Word Sense Disambiguation. Journal of Natural Language Engineering (special issue on Senseval-2) 9(1). 
Magnini B, Cavaglia G 2000 Integrating Subject Field Codes into WordNet. In Proceedings of the Third International Conference on Language Resources and Evaluation (LREC 2000), Athens, Greece. 
Magnini B, Gliozzo A 2002 Mapping WordNet Domains to the Dewey Decimal Classification. ITC-irst Technical Report. 
Montemagni S, Barsotti F, Calzolai N, Corazzari O, Zampolli A, Fanciulli F, Massetani M, Raffaelli R, Basili R, Pazienza M T, Saracino D, Zanzotto F, Mana N, Pianesi F, Del Monte R 2000 Building the Italian Syntactic-Semantic Treebank. In Building and using syntactically annotated corpora. Kluwer, Dortrecht. 
Pianta E, Bentivogli L, Girardi C 2002 MultiWordNet: developing an aligned multilingual database. In Proceedings of the First Global WordNet Conference, Mysore, India. 
Rigau G, Magnini B, Agirre E, Vossen P, Carroll J 2002 MEANING: A Roadmap to Knowledge Technologies. In Proceedings of COLING Workshop "A Roadmap for Computational Linguistics". Taipei, Taiwan. 
Rossini Favretti R, Tamburini F, De Sanctis C 2001 A corpus of written Italian: a defined and dynamic model. In Proceedings of Corpus Linguistics 2001 Conference, Lancaster,UK. 
Sekine S 1997 The Domain Dependence of Parsing In Proceedings of the Fifth Conference on Applied Natural Language Processing, Washington D.C., USA. 
Sinclair J, Ball J 1996 EAGLES Preliminary Recommendations onText Typology. (http://www.ilc.pi.cnr.it/EAGLES96/texttyp/texttyp.html) 
112

Discovering Regularities in Non-Native Speech 
Julie Carson-Berndsen 1, Ulrike Gut 2 & Robert Kelly 1 
1 Department of Computer Science, 2 Faculty of Linguistics and Literary Studies, 
University College Dublin, Ireland, University of Bielefeld, Germany 
{Julie.Berndsen, Robert.Kelly}@ucd.ie gut@spectrum.uni-bielefeld.de 
Fax: +353 1 2697262 Fax: +49 521 106 6008 
This paper presents ongoing collaborative research which focuses on the application of computational linguistic techniques to the analysis of a corpus of native and non-native speech. The aim of this research is to use computational tools for phonological acquisition and representation to identify regularities and sub-regularities between different speaker groups. The corpus is being collected and annotated at different levels as part of ongoing research into the acquisition of prosody by non-native 
speakers at the University of Bielefeld (see Milde & Gut, 2001). The computational tools have been designed and implemented at University College Dublin as part of a suite of tools aimed at providing a development environment for modeling, testing and evaluating phonotactic descriptions of lesser studies languages (Carson-Berndsen, 2002). The two hitherto separate research directions have now come together to apply computational linguistic tools to a corpus-based investigation of non-native speaker phonotactics. The term phonotactics refers to the permissible combination of sounds in a language. There are various ways of acquiring representations of phonotactic constraints. One approach is to manually construct a set of rules based on the linguistic intuitions of a native speaker. Another approach is to learn such constraints from a data set. The latter is the approach taken in this paper. 
Currently the corpus consists of 253 annotated recordings of between 2 and 30 minutes length by 88 different speakers with 21 different native languages. The corpus is annotated at a number of linguistic levels such as the level of the intonational phrase, the word, the syllable, and the skeletal (CV) structure, and comprises annotations of the prosodic structures of intonation and pitch range. Each level of annotation is viewed as a tier, analogous to the representations of autosegmental phonology (Goldsmith 1990). Analysis can take place either with respect to individual tiers or with respect to an associated set of tiers. In the latter case, one tier is chosen as the primary tier and the others are associated with it in terms of overlap and precedence relations between the units as suggested in Carson-Berndsen (1998: 60). Using the computational linguistic tools, finite state automaton and finite state transducer representations of the tiers are extracted automatically from the annotated corpus. Regularities in the data are then identified either with respect to a single tier or with respect to an associated set of tiers. 
The majority of previous studies on the acquisition of phonotactic constraints are based on small numbers of participants, which reflects the time-consuming nature of a manual analysis of this kind of data. Results on an initial subset of the corpus for Italian and Polish speakers of German demonstrate that the task of identifying phonotactic regularities and sub-regularities in the corpus data can be performed elegantly and efficiently using the computational linguistic tools. Distinct differences between the phonotactic violations produced by the Italian and the Polish speakers have been found and will be documented in detail in the full paper. Further experiments are now underway on a larger corpus. 
113

NLP model and tools for detecting and interpreting metaphors in domain-specific corpora 
P. BEUST, S. FERRARI, V. PERLERIN 
GREYC, Computer Science Laboratory, University of Caen - bd Maréchal Juin F14032 Caen Cedex - France 
Abstract 
The aim of this paper is to present how a user-centred lexical representation model, based on the theory of Interpretative Semantics, can be used for detecting and interpreting metaphors in domain specific corpora. We present here several tools useful for such tasks and discussing the results of an experiment. 
Introduction 
In this paper, we present NLP (Natural Language Processing) project addressing the interpretation process. This project, called “ISOMETA1”, focuses on computer-assisted metaphor interpretation following a user-centred point of view. We propose a model for lexical representation as well as tools for validation on corpora. 
In the first section, we give an overview of some previous approaches related to metaphor detection and interpretation in order to highlight the main concepts we deal with. We also introduce the theoretical background for knowledge representation and text interpretation sustaining our approach. 
In the second section, we argue for user-centred lexical representations and we present our model for this purpose (called Anadia) as well as practical examples. This model enables automatic computing of customized help for interpretation by means of the isotopy concept. We detail how to produce such help when dealing with conventional metaphors. 
In the third section, we present some of the tools implementing our main propositions. AnadiaBuilder is a user-friendly interface to build structured lexical representations. Complementary tools have been developed for corpus analysis, producing graphical representations for easy browsing through the results and customized help for interpretation. 
In the last section, we present the results of an experiment on a domain-specific corpus. We study examples of a specific conventional metaphor: the stock market domain expressed with meteorological terms. 
Finally, we discuss how to carry out an evaluation of our work. We also propose other applications of our model and tools. We conclude by pointing the main directions for further developments and the next steps for the “ISOMETA” project. 
1 Framework 
1.1 Metaphors in NLP 
It is generally agreed that a metaphor involves two concepts: a source concept, related to the words used metaphorically, also called the vehicle of the metaphor, and a target concept, which is what the metaphor is used for and tries to describe, also called the tenor of the metaphor. If we consider the following example, first proposed by Wilks (1978), and still studied by Fass (1997): 
(1) “My car drinks gasoline”, 
the source of the metaphor is the action of drinking, and the target may be described as the use of gasoline by a car. 
The different NLP approaches for metaphor interpretation mainly depend on how the relation between the source and the target is viewed: as an analogy, as a novelty, or as an anomaly. In (Gentner, 1983; Falkenhainer et al., 1989), this relation is mostly viewed as an analogy. Thus, interpreting a metaphor requires deeply structured knowledge representations in order to trace back and describe the analogy between concepts. In (Indurkhya, 1992; Gineste et al, 1997), the relation between the source and the target is viewed as a novelty: it is not a pre-existing similarity but one created by the existence of the metaphor. Thus, interpreting it requires the dynamic selection and transfer of knowledge from the source domain to the target domain. 
1 “ISOMETA” stands for ISOtopy and METAphor. 
114
Metaphor may also be viewed as a semantic anomaly. In example (1), there is an anomaly if one considers that “drinking” does not normally apply to physical objects such as cars. As shown by Martin (1992), metaphors are not always anomalies, and anomalies are not always metaphors. For instance, in: 
(2) “McEnroe killed Connors” (ibid), 
there is no anomaly, nonetheless “killed” may be viewed as metaphoric. Only contextual information can help for disambiguating the whole sentence. Fass (1997) proposes a method for discriminating semantic relations, which makes a clear distinction between metaphors and anomalies. This method makes it possible for multiple interpretations to coexist, as in example (2). 
It is not necessary to focus on the relation between the source and the target to interpret metaphors. Kintsch (2000) shows how the meaning of a metaphor can be interpreted and represented by a multi-dimensional vector, exactly like other meanings in the Latent Semantic Analysis approach. 
We also consider that metaphors require the same interpretation process as other meanings. We do not focus on the relation between the source and the target either. But in our approach, we use a symbolic representation in order to provide a novice user with easily understandable tools. 
Lakoff and Johnson (1980) introduced the notion of conventional conceptual metaphor, based on the observation that, for some semantic domains, multiple terms from a common source domain may be used to describe metaphorically multiple corresponding concepts from a common target domain. In (Ferrari, 1997), such conventional metaphors are studied in the scope of domain specific corpora. For instance, he observed that stock market events are often described by meteorological terms in newspaper articles related to economics. 
In our work, we look at conventional metaphors in order to use the pre-existent knowledge that the target domain may be partly structured as the source domain. We focus on the previous example, which we call “economics is meteorology”. 
Using limited and user-centred resources, we try to track down the analogy and the novelty points of view. In the next section, we present the linguistic basis of our approach. 
1.2 Knowledge representation and text interpretation 
The lexical representation and the analysis process we use are mainly inspired by continental structural linguistics (Greimas, 1966; Pottier, 1987) and especially by the linguistic theory developed by F. Rastier (1987): Interpretative Semantics. In this theory, the interpretation is considered as a description of semantic units located both in a linguistic unit (corpus, text, sentence...) and a situation. Interpretation involves an interpreter, along with his knowledge, his goals and his social relation2 to these given linguistic units. Thus, the meaning of a word, for instance, is not a definition of this word, as could be found in a dictionary, but rather an explanation of its role in a given linguistic unit. 
A lexical content is described in terms of meaning components, themselves described in terms of semantic features called semes. For example, the lexical item “depression” can be related to a ‘meteorological phenomenon’ or a ‘mental state', and the meaning component ‘meteorological phenomenon’ can be represented with the following semes: /area/, /low pressure/, /bad weather/… Such a description is called a componential representation. 
Semes depend both on the user and on the task. They are potential meaning features, relevant only in specific contexts. The notion of isotopy, introduced by Greimas (1966), characterizes these contexts. An isotopy is the recurrence of one seme in a linguistic unit. For instance, in this paper, one may notice at least two main isotopies related to ‘computer science’ and ‘linguistics', supported by many different lexical items. 
In our work, we focus on lexical items from two domains, meteorology and stock market, in order to describe the underlying conventional metaphor. In the next section, we present Anadia, the model we have previously developed for such lexical representations, and show how to use it for metaphor processing. 
2 A model for lexical representation 
2.1 Main principles 
The main principles of our model have been described in details by Beust (1998) and Nicolle et al. (2002). Anadia is a model of lexical categorization based on both componential and 
2 We are talking about the relation to linguistic units through social role. For instance, a juridical text is differently interpreted by a lawyer and by common people. 
115
differential representation. The differential paradigm states that a lexical content can be described by opposing it to others through structural relations, following the notion of “linguistic value” proposed by Saussure (1915). The Anadia model allows a user to produce descriptions of meaning components by the way of semes, which are the componential part of the representation. Rather than classical componential representations, semes are represented by a set of opposite features. This is the basis of the differential part of the representation. For example, “depression” can be described as the combination of the semes [Zone] and [Pressure] respectively corresponding to the opposite features “area vs. line” and “low vs. high”. The activated features for “depression” are area and low. These semes also allow a semantic representation of the lexical item “anticyclone” described by the activated features area and high. 
Lexical items representations are therefore made from the combination of semes. In this way, our model allows its user to build tables where lexical items can be described in terms of differences and common points, as shown in Figure 1. 
Figure 1. Example of an Anadia table describing some pressure zones3. 
In Figure 1, the combination of the semes [Zone] and [Pressure] gives rise to four table rows in which lexical items can take place. When there are several lexical items in the same row, it implies that their semantic representations are not considered as different in this table (in another one, they could be differentiated). It is the case for “tropical wave” and “easterly wave” in the example. A row can stay empty if we do not know any lexical item corresponding to a certain combination of features. It is the case for the combination of ‘line’ and ‘high’ in the example. A row can also be filled in later if we find a corresponding lexical item (for instance, by the way of a corpus study). 
Several tables can be used to describe a specific semantic domain. In such a set of tables, a table can be linked to a row in another table by a subcategorization relation (Figure 2). 
Figure 2. Extract from a set of tables for the stock market domain. The second row of the Domain objects table is linked to the Stock indices table by a relation of subcategorization. 
For many reasons (choice of semes, content of rows, subcategorization relations) tables represent the points of view of the user for a given task. Anadia is a user-centred model and the lexical representations built with the model are not supposed to be either universal or exhaustive. Tables can be modified and updated at any time, depending on the results obtained from the analysis process. 
Anadia tables allow proposing an analysis process based on the concept of isotopy. As shown by Tanguy (1997), isotopy can be seen as an easy and understandable way of expressing themes in linguistic units. Therefore, the interpretation process consists in finding isotopies in linguistic units. 
(3) During the three days immediately proceeding depression formation, anomalous moisture transforms from a pattern associated with a tropical wave transversing the open Atlantic Ocean ... (http://ams.confex.com/ams/25HURR/25HURR/abstracts/35268.htm) 
3 The examples have been translated for this paper. 
116
In example (3), using the representation of Figure 1, we notice that “tropical wave” and “depression” are described with the same semes: [Zone] and [Pressure]. These two recurring semes involve two isotopies that contribute to the meaning to the sentence. The recurring features also show that the sentence deals with pressure zones of different type : one corresponding to a ‘line’ of ‘low’ pressure and one to a ‘area’ of ‘'low’ pressure. 
2.2 Using the model for metaphor processing 
The Anadia model was not originally designed for metaphor processing. The latter is just a specific task for which the model can be used. In order to study how the model can effectively be applied to metaphor processing, and what adjustments are to be made, we focus on the specific conventional metaphor: “economics is meteorology”. 
The model enables us to represent our lexical knowledge concerning the source and the target domains involved in this specific metaphor. Let us work on the assumption that one set of tables, set S, describes the lexical items of the source domain, meteorology, and a second one, set T, is dedicated to the target domain, stock market. 
At this point, the Anadia model enables us to use a single lexical item in multiple sets of tables. For instance, it is possible to represent “barometer” both in set S and in set T. In set S because it is a common term of meteorology, and in set T because we have noticed in newspaper articles that it is sometimes used in phrases such as “stock market barometer”, suggesting some economical tools for measures or predictions. 
This possibility becomes a problem when dealing with metaphors. If we want to use the model to detect the metaphorical use of “barometer” in phrases such as “stock market barometer”, we must not represent it in set T. Moreover, lexical items of set T must not be formed with words that can be considered as lexical items of set S. This is a first adjustment, or constraint, added to the Anadia original model: when building sets of tables for metaphor processing, it is necessary not to use words from a source domain in a set of tables for a target domain. 
Following this rule, “barometer” is now banished from the lexical items of set T. The reason for this is that when computing isotopies, the source semes are required to spot a metaphorical use. If “barometer” were in the two sets, S and T, its metaphorical use in “stock market barometer” would be ignored because an isotopy of words from set T would only hide the existence of semes from the source domain. It is important to notice that such a representation must not be considered as “wrong” and would not lead to misinterpretation. It would simply reflect the conventional aspect of the metaphor, which itself would be part of the knowledge of the user who would include “barometer” in the lexicon related to “stock market”. 
Assuming that S and T are now built according to that constraint, let us see how it is possible to spot a metaphor, and to what extent the lexical representation can produce guidance for its interpretation. The whole point is to detect an isotopy involving words from both the source and the target domain. On the one hand, with the Anadia model, isotopies are based on semes shared by lexical items involved in a single linguistic unit. On the other hand, previous works on conceptual metaphors have shown the existence of underlying structure analogies between the source and the target domains. It then stands to reason that the solution is to use some semes which are shared by lexical items from the two sets of tables, and which represent the structure analogy between the two domains. For example, if we use the seme [Role = studying, analysing vs playing a part] to describe “barometer” from the meteorology domain and “stock exchange” from the stock market domain, it then becomes possible to spot and produce guidance for interpreting the following metaphor : 
(4) a- “the Dow Jones is a stock exchange barometer”. 
The seme [Role] is here shared by two lexical items: “barometer” from the source domain and “Dow Jones” from the target domain. The fact that the lexical items involved belong to different domains is characteristic of a metaphorical use. The shared seme, creating an isotopy, is a first step for guiding the interpretation process. We shall discuss these points further in the following sections. 
At the moment, we can consider the use of shared semes as a second adjustment or constraint added to the model when processing metaphors. If sets S and T are built according to the two constraints presented in this section, it is not only possible to spot metaphors involving the lexical items initially used to organize the two sets, but also to process some of their extensions. Actually, when building the set of tables concerning meteorology, the user will probably consider lexical items such as “thermometer”, “mercury”, and propose to use the same seme [Role] to describe them. It will then be possible to process the following examples: 
(4) b- “the Dow Jones is a stock exchange thermometer” 
117
(4) c- “the Dow Jones is the New York stock exchange mercury” 
even though the sets of tables were not originally designed for these specific metaphors. 
The next section presents tools developed in order to validate our model on corpus. 
3 Tools 
The tools we created for our experiments are freely available for research purposes. They have been implemented with platform-independent languages (Java, XML and XSL). They can be used for different kinds of tasks including figurative language analysis (as shown in this paper) or for instance document retrieval (as shown in (Perlerin, 2001)). 
3.1 AnadiaBuilder: a tool for building Anadia lexical representations 
AnadiaBuilder is software enabling to build lexical representations following the Anadia model (Nicolle et Al., 2002). The created data is stored in XML format. Via a user-friendly graphical interface, the user can build sets of tables according to the current task. The interface contains five main interactive panels: 
(A) (B) (C) (D) (E) The first one enables the 
user to create the semes he finds relevant for the representation. The user chooses the related sets of opposed features and an explicit name for each seme. 
The second one makes it possible to create tables made from the combination of semes (Figure 3). The user chooses the semes and the machine computes the combinations and automatically builds the table. The user fills in the cells (on the left-hand part of the table) corresponding to a given set of features from different semes with relevant lexical items. 
The third one displays a graphical representation of a table (called “topique” in French) showing the differences and the semantic proximity between lexical items by means of annotated links (Figure 3). 
The fourth one creates the relations between tables. It also makes it possible to see the whole set of tables through a schematic representation where only table names are displayed (Figure 4). In this panel, the user can allocate a colour to each table, which is useful for further corpus analysis. 
The last one is linked to the MAHTLEX lexical database, developed at the University of Toulouse4. For each lexical item, the computer proposes a set of inflections or enables the user to build the corresponding set of inflections by himself. Inflections will be used to match occurrences of lexical items in texts. 
At step (B), when building a table, if the user estimates that he can fill in several cells with the same lexical item, he must correct his proposals. This fact can happen because of two reasons. The chosen semes are not mutually exclusive, or the features of at least one seme are not mutually exclusive. The building contraints of the Anadia model are discussed by Beust (1998). Perlerin et Beust (2002) have undertaken an experiment with novice users. The results have shown that building a set of tables following the Anadia constraints is accessible to novice users. Such results may have to be moderated when dealing with a linguistic phenomenon such as metaphor. 
4http://www.irit.fr/ACTIVITES/EQ_IHMPT/ress_ling.v1/accueil01.php 
118
Figure 3. AnadiaBuilder: tables building panel and corresponding “topique” from the “topique” panel (extract of the screenshot). 
Figure 4. AnadiaBuilder: set of tables representation related to the stock market domain. 
Each set of semes, each set of tables or inflections dictionary can be saved independently and reused in different experiments. In particular, the sets of tables can be used for corpus analysis. Results are then produced as an annotated version of the corpus. Several tools help us to browse through the resulting corpus, mainly by the use of colours and charts. 
119
3.2. Corpus analysis tools 
During the automatic part of the corpus analysis, all the possible occurrences of lexical items from the sets of tables are located in the texts. A first tool builds a graphical representation of each text in the corpus5, as shown in Figure 5. For one text, each table is represented by one bar inheriting its colour. Each bar is proportional to the number of matched lexical items from the table. In our experiment on the metaphor “economics is meteorology”, the purpose of this graphical representation is to provide the user with a quick way to track down articles where the source domain is evoked. A single HTML page contains all the charts along with hyperlinks to the related texts (Figure 5). 
Figure 5. Graphical representations of the outputs: moving the mouse over a bar shows the corresponding table name and matches the number of lexical items. 
A second tool transforms the XML version of each text into an HTML version, as shown in Figure 6. In the HTML version, the matched lexical items are in the same colour as the corresponding table. This provides the user with an easy means to find the precise location of the lexical items he is interested in. 
Figure 6. A coloured article. Moving the mouse over a coloured lexical item shows the corresponding table name and the corresponding set of semes/features. 
5 In our experiments, the article appeared to be a relevant unit to build the charts. The level of this linguitic unit can be changed. 
120
The next session presents some results of an experiment realised on a journalistic corpus. 
4 First results 
Our work has been validated through a corpus experiment. The corpus is constituted of about 600 articles from the French newspaper “Le Monde”, addressing economics and stock market (around 450,000 words) between 1987 and 1989. This corpus, already studied by Ferrari (1997), contains numerous examples of the conventional metaphor “economics is meteorology”. It also contains lexical items from the meteorology domain that are not used in a figurative way. 
For our experiment, the sets of tables have been designed with nine shared semes. These semes reflect our own view of the conceptual metaphor. Specialists of any of the two domains would probably have designed the sets of tables in a much different way. Our point of view reflects our knowledge of the underlying analogy between the two domains. In the following, we discuss two different examples in order to show how the analogy and novelty points of view can be retrieved with our proposals. 
(5) Le Dow Jones par exemple, le thermometre de la Bourse de New York, qui avait chuté de 508 points …6 - Article n°126 – Paragraph 1 
In example (5), three lexical items from the sets of tables were matched (therefore coloured) by the analysis process. “Dow Jones” appears in the “Stock Indices” table of the stock market domain (see Figure 9). “thermometre” (thermometer) appears in the “Measuring Instruments” table of the meteorology domain (see Figure 8). 
Figure 8. Extract of the meteorology Anadia set of tables. 
Figure 9. Extract of the stock market Anadia set of tables. 
Following these representations of the two domains, an isotopy involves the shared inherited seme [Role] and the value ‘studying, analysing’ can be found thanks to the first two coloured lexical items. One can then conclude in favour of a metaphorical use and propose the following interpretation: “thermometre” (thermometer) is used in the same way as “graphics”, “ratio”... i.e. to suggest an object for analysis and study in the stock market domain. The lexical item could be replaced (more or less efficiently) by others from the “Measuring Instruments” table. 
(6) Ce krach était du (…) a la chute vertigineuse et incontrôlée du dollar, signe que la tempete affecte dorénavant les marchés financiers.7 - Article n°153 – Paragraph 3 
In example (6), the lexical items “krach” (crash) and “tempete” (storm) appear in the following tables (Figure 10 and Figure 11). 
6 Literal translation: The Dow Jones, for instance, the thermometer of Wall Street, which had fallen 508 points … 
7 Literal translation: This crash was due (...) to the vertiginous and uncontrolled fall of the dollar, sign that the storm will henceforth affect the financial markets. 
121
Figure 10. Extract from the stock market Anadia set of tables (the table has been truncated). 
Figure 11. Extract from the meteorology Anadia set of tables (the tables have been truncated). 
The isotopy found in this sentence (example 6) is based on two different semes. The first seme involved is [Connotation] (inherited for “storm”) with the same activated value ‘bad'. The second one is [Direction] with two different activated values: ‘down’ for “krach” and ‘up’ for “tempete”. Example (3) makes it possible to conclude in favour of a metaphorical use. First, due to the activated values, the seme [Direction] is less relevant than the other one, [Connotation]. Moreover the seme [Axis] is exclusively used in the meteorological domain and is not involved in any isotopy. We propose therefore to consider it as “irrelevant” in the context. The seme [Strength] does not take part in an isotopy either; but, unlike [Axis], it can be shared between several lexical domains. It seems to us that we can therefore consider it as relevant in this context. This illustrates how novelty is dealt with in our approach. Finally, we propose the following help for interpretation: “tempete” (storm) is used to evoke a not only bad but also violent dynamic phenomenon in the stock market domain. 
Numerous examples of sentences where the sets of tables enable to conclude in favour of metaphorical uses have been discovered in the corpus thanks to our tools. The two sets of tables have been modified several times depending on the results obtained from the analysis process. Those results are the fisrt step of the “ISOMETA” project validating our approach and our tools. 
Conclusion and further works 
This paper has presented a user-centred lexical representation model and its use to produce help for metaphor interpretation. There is no need to be an expert in a given domain to describe it by means of this user-centred model. Nevertheless, metaphor interpretation is a linguistic task. Thus, a description for a study on a conceptual metaphor, such as the one we have presented in this paper, requires a certain familiarity with linguistic sciences. The user must indeed be able to describe how he appreciates the analogy between the source domain and the target domain by the use of shared semes. 
Though we have presented the use of the Anadia model for a very specific task, we have already argued for its use in many applications, such as domain-specific corpus browsing or document retrieval, as shown in (Nicolle et al. 2002). We hope the same applies to the tools developed for the “ISOMETA” project. 
An experiment on domain-specific corpus has validated our method. Actually, producing customized help for metaphor interpretation appears to be possible. However, this result must be evaluated, both quantitatively and qualitatively. Nevertheless, such an evaluation is not easy to carry out. On the one hand, the user-centred aspect of the model implies that the evaluation process should be user-centred too. On the other hand, this evaluation requires an annotated corpus. Such a reference corpus does not exist yet and seems difficult to produce. 
In order to start the evaluation, our further works will concern other examples of conceptual metaphors, as well as other domain-specific corpora for their study and the automatic processing of isotopies. We also plan to use our model for metaphor and paraphrase in automatic text generation. 
122
References 
Beust P. 1998 Contribution a un modele interactionniste du sens. Computer Sciences PhD Thesis of the University of Caen, France. 
Falkenhainer B., Forbus K.D. and Gentner D. 1989 The Structure-Mapping Engine : Algorithm and Examples. Artificial Intelligence, 41/1, pp.1-63. 
Fass, D. 1997 Processing metaphor and metonymy. Greenwich, Connecticut: Ablex Publishing Corporation. 
Ferrari, S. 1997 Méthode et outils informatiques pour le traitement des métaphores dans les documents écrits. Computer Sciences PhD Thesis of the University of Paris XI, France. 
Gentner D. 1983 Structure-Mapping: A Theoretical Framework for Analogy. Cognitive Science, 7, pp. 155-170. 
Gineste, M.-D., Indurkhya, B. and Scart-Lhomme, V. 1997 Mental representations in understanding metaphors. Technical report, 97/2, Groupe Cognition Humaine, LIMSI-CNRS, Orsay. 
Greimas A.J. 1966 Sémantique structurale. Paris: Larousse. 
Indurkhya, B. 1992 Metaphor and Cognition. Dordrecht, The Netherlands: Kluwer Academic Publishers. 
Kintsch, W. 2000 Metaphor comprehension: A computational theory. Psychonomic Bulletin & Review, pp. 257-266. 
Lakoff G. and Johnshon M. 1980 Metaphors we live by. University of Chicago Press, Chicago, U.S.A. 
Martin J.H. 1992 Computer Understanding of Conventional Metaphoric Language. Cognitive Science (16), pp.233-270. 
Nicolle A., Beust P. and Perlerin V. 2002 Un analogue de la mémoire pour un agent logiciel interactif. In Cognito, 21, pp. 37-66. 
Perlerin V., 2001 La recherche documentaire, une activité langagiere. In proceedings of TALN2001, Tours. 
Perlerin V. and Beust P. 2002 Pour une instrumentation informatique du sens. Proceedings of the CNRS/ARCO Summer School in Tatihou, to be published. 
Pottier B. 1987 Théories et analyse en linguistique. Hachette: Paris , p. 224. 
Rastier F. 1987 Sémantique interprétative. Presses Universitaires de France : Paris. 
Saussure F. de 1915 Cours de Linguistique Générale. Mauro-Payot: Paris (1986). 
Tanguy L. 1997 Computer-Aided Language Processing: Using Interpretation to Redefine Man-Machines Relations. Proceedings of the 2nd International on Cognitive Technology (CT'97), Humanizing the Information Age, Aizu Wakamatsu City, Japan, August 25-28. 
Wilks Y. 1978 Making Preferences More Active. Artificial Intelligence, 11/3, pp.197-223. 
123

A corpus-based technique for grammar development 
Philippe Blache, Marie-Laure Guénot & Tristan van Rullen 
LPL-CNRS, Université de Provence 
29 avenue Robert Schuman 
13621 Aix-en-Provence, France 
{pb,mlg,tristan}@lpl.univ-aix.fr 
Abstract 
We argue in this paper in favor of a fully constraint-based approach in the perspective of grammar development. Representing syntactic information by means of constraints makes it possible to include several parsers at the core of the development process. In this approach, any constraint (then any syntactic property) can be evaluated separately. This aspect, on top of being highly useful in a dynamic grammar development, also allows to quantify the impact of a constraint over a grammar. We describe in this paper a general architecture for developing a grammar in this perspective and different tools that can be used in this schema. 
1. Introduction 
Modern applications, especially in the context of human-machine communication, need to treat unrestricted linguistic material (including spoken languages) with a fine granularity. This means that NLP applications have to be at the same time robust and precise. The first step towards this perspective consists in developing adequate resources, in particular broad-coverage grammars. However, developing grammars remains an important problem and this work is usually done only empirically. 
We present in this paper an experiment based on a constraint-based linguistic formalism consisting in developing a broad-coverage grammar by means of corpus-based techniques. More precisely, the approach consists in using different tools, namely a POS-tagger, a shallow parser and a deep parser, for developing an electronic grammar for French taking into account various phenomena and different uses including spoken language syntactic turns. Different parsers are used in this perspective at different stages both in a development and an evaluation perspective. The general idea consists, starting from a core grammar, in parsing previously tagged and disambiguated corpus by means of a deep non-deterministic parser. The results are interpreted in order to identify syntactic phenomena beyond the scope of the grammar. The grammar is then completed and the modifications are experimented over the corpus. The new result is interpreted again in order to examine the consequences of the modifications: new parses, new structures, elimination of unexpected parses, but also false results, spurious ambiguities, etc. This work is then completed with a more systematic evaluation by means of a shallow parser which makes use of the entire grammar, but applies some heuristics in order to control the parse. This parser presents the advantage of being robust and efficient and can be used for parsing large corpora. In our experiment, this parser is used in order to evaluate the different versions of the grammar over different corpora. It is then very easy to compare automatically the efficiency of the grammar, even without any treebank. 
2. The formalism of Property Grammars 
We think that using a fully constraint-based formalism for representing syntactic information offers several advantages that can have deep consequences in grammar development. More precisely, provided that 
124
information is represented only by means of constraints, it is possible to conceive a grammar development architecture starting from zero and taking advantage of evaluation tools. 
In this section, we present the “Property Grammars” formalism (hereafter PG, see [Blache00]) in which linguistic information corresponds to different kinds of constraints (also called properties). In this approach, parsing comes to constraint satisfaction and, more interestingly in the perspective of grammar development, the result of a parse corresponds to the state of the constraint system. Such a result, called “characterization”, is available whatever the form of the input, being it grammatical or not. This makes this approach well adapted to the treatment of unrestricted texts and even spoken language corpora. We will give in the following some examples taken from different corpora, from newspapers to dialogs. In the end, we want to show that in such a perspective, different parsing tools can be also used as tools for developing but also evaluating grammars. 
In the remaining of this section, we present the PG formalism, illustrating the fact that constraints constitute a radically different approach in the perspective of parsing unrestricted texts. In PG, different kind of syntactic information corresponds to different kind of constraints. Basically, we use the set of following constraints: linearity (linear precedence), dependency (semantic relations between units), obligation (set of possible heads), exclusion (cooccurrence restriction), requirement (mandatory cooccurrence) and uniqueness: 
1. Obligation (noted Oblig): Specifies the possible heads of a phrase. One of these categories (and only one) has to be realized. 
Example: Oblig(NP) = {N, Pro} 
2. Uniqueness (noted Uniq): Set of categories that cannot be repeated in a phrase. 
Example: Uniq(NP) = {Det, N, AP, PP, Sup, Pro} 
3. Requirement (noted Ë): Cooccurrence between sets of categories. 
Example: N[com] Ë Det 
4. Exclusion (noted ‚): Cooccurrence restriction between sets of categories. 
Example: AP ‚ Sup (in a NP, a superlative cannot co-occur with an AP) 
5. Linearity (noted <): Linear precedence constraints. 
Example (in a NP): Det < N 
6. Dependency (noted ¨): Dependency relations between categories. 
Example (in a NP): Det ¨ N 
In this approach, all constraints are defined as relations between categories, and all categories are described using a set of properties. Moreover, even if we preserve a hierarchical representation of syntactic information (distinguishing lexical and phrasal levels), the notion of constituency doesn't constitute an explicit information. In other words, categories are only described by a constraint graph in which nodes represent categories involved in the description and edges are relations (or constraints; cf. fig.1). The fact that the set of properties describing a category forms a graph is a side effect. It indicates certain cohesion of the description coming from the fact that the description of the category is specified, at the difference of classical generative approaches, not in terms of relations between some constituents and a projection, but only in terms of relations between these constituents. A grammar is then formed by a set of such constraint graphs. The important point, especially in the perspective of grammar engineering, is that all constraints are at that the same level and then independent from each other. This means that it is possible to evaluate them separately. It is also possible to evaluate only a subpart of the constraint system. 
125
figure 1 : Constraint graph of the Prepositional Phrase (PP) 
The categories mentioned in the properties of the PP are represented by nodes; each relation (or property) is represented with edges, eventually looping on one node (obligation, uniqueness) or connecting two nodes (requirement, exclusion, dependence, linearity). 
Let's describe more precisely the parsing architecture. In GP, describing a category consists in finding all the relevant constraints in the grammar and evaluating them. In a data-driven perspective, this comes more precisely to identify a set of categories and verify their properties. At the beginning of the process, the initial set of categories corresponds to the set of lexical categories. The parsing process simply consists in identifying the set of constraints that can be evaluated for the categories belonging to this set. Any constraint belonging to a constraint graph, the evaluation process comes to activate one (or several) constraint graphs. Activating a constraint graph leads to instantiate new categories (which are the categories described by the corresponding graphs). 
Activating a graph is then simply the consequence of the evaluation of the constraint system. This means that after evaluation, for each graph, we obtain the set of constraints that are satisfied plus eventually the set of constraints that are violated. These two sets form what we call the characterization of a category. We can then describe any kind of input, grammatical (i.e. whose set of violated constraints will be empty), or not. 
figure 2: Characterization graph 
Characterization graph of the phrase “The air line that the doctor Fergusson intended to follow”. The nodes correspond to the parsed categories, arrows correspond to the satisfied properties. 
126
Such an architecture shows in what sense this approach differs from generative ones. In this last case, the general process consists in first building a structure (typically a tree) and then verifying its properties. In our approach, we only use constraint evaluation, without needing any derivation relation nor other extra devices in order to build the syntactic description of an input. 
3. The role of constraints in Grammar Engineering 
One of the most important aspects of PG is that all information is represented by constraints and all constraints are independent. The information comes from constraint interaction, but different kinds of information are represented separately. As explained above, it is then possible to evaluate separately all constraints. We will describe in the next section how it can be possible to take advantage of this aspect in our shallow parsing technique. As for grammar development itself, this characteristic also has important consequences. 
Let's first come back on an important point. One of the main differences between a fully constraint-based approach such as PG and other classical generative techniques lies in the fact that there is no need to build a structure before being able to verify its properties. More precisely, generative method consists in expressing relations in terms of structures where PG uses only relations between objects. This means that a grammar has to be coherent, consistent and complete when using generative formalisms. At least, it should allow to build a structure (a tree) covering an input. Moreover, it is not possible in such approaches to deal easily with partial information. Using PG avoids these problems. 
First, even when starting from zero the development of the grammar, it is possible to evaluate any subset of properties. This aspect has several consequences on the utility of automatic devices during the development. On one hand, it is possible at any stage of the development of the grammar, to use a parser in order to verify the consequences of new properties, for example in parsing the same corpus and comparing the results of different versions of the grammar. On the other hand, the same devices can also be used in order to evaluate the organization of the grammar itself. All properties being assessable separately, it is possible to evaluate the respective consequences, for example in terms of filtering, for each constraint. This means a possibility of evaluating empirically the power of each constraint as well as specifying this power for a given type of property with respect to the others. 
Second, especially when developing a broad coverage grammar, unrestricted texts have to be taken into account. It is then necessary to deal with partial information and partial structures. The constraint-based approach proposed in PG relies, as shown before, on graphs which is well adapted to the representation of such information. 
Finally, dealing with unrestricted texts leads to treat even ungrammatical inputs. In a classical approach, such data are problematic when using automatic devices: the input can be either rejected (which is not interesting for example when treating spoken languages) or accepted (in this case, a wrong construction is added to the grammar). In our approach, it is possible to introduce a gradient in the grammaticality relying on the possibility of quantifying constraint evaluation. For a given construction, one can give the number and the type of constraints that are violated. Such information is in our opinion as important as the set of satisfied constraints, reason why our characterization contains both information. Reciprocally, for a given constraint, it is possible to specify its satisfiability degree in a corpus. And even more generally, it could be possible to specify what kind of constraint is preferably violated. At the difference with the optimality theory (see [Prince93]), we have here a systematic tool for evaluating the respective importance of constraints, generally and locally. 
The approach proposed here presents many advantages. It is in particular well adapted for the elaboration of grammars taking into account many different sources (including spoken languages one). Moreover, this approach is highly flexible in the sense that any syntactic information can be evaluated separately by means of tools making it possible to quantify their roles. This approach also presents a very practical interest: it only requires a simple constraint editor (which consists in an interface between an XML file and its representation). It can easily be adapted to the evolution of category representation. 
127
4. Step by step semi automated grammar development 
The experimental process of grammar development described in this section is shown hereafter on figure 3 and can be schematised as a versioning step by step semi-automated sequence of experiments. 
Figure 3: step by step semi-automated sequence of experiments 
The development of a broad-coverage grammar for French relies, within our framework, on several points: a basic corpus, a core grammar, and some tools (several parsers). In order to carry out the tests on our grammar, we use a three million words corpus of journalistic texts (hereafter “corpus-test”), annotated and disambiguated (the treebank corpus of the LLF, University of Paris 7). 
A non-deterministic deep parser can be used as a real grammar development platform. It makes it possible to obtain various results for any input, while entrusting to him either the totality of grammar or one of its subpart, and then being able for example to focus on a subset of properties (to precisely observe their operation, their weight and their incidence on the parse), or on a subset of categories (to focus on the results obtained, the number of non relevant parses, to observe if one can deduce a recurring error pattern and try to avoid it, or on the contrary a recurring satisfactory pattern), without taking into account the rest of the grammar. We can make this type of handling without modifying the structure of neither grammar nor the parser, and that only by neutralizing the non-desired data. The non-determinism of the parser allows to get a mean to evaluate at a first level the efficiency of grammar according to the number and quality of the proposed parses. 
The development process can be alternatively carried out in three different ways. First, from a descriptive point of view: in this case, we first isolate from the corpus various occurrences of a given construction (e.g. clefts, coordination). We then build its fine empirical linguistic description, based on observation of 

… … … 

… … … 

releasing 

completion 

Large tests stage 
on whole grammar releases 

Versioning stage 
on parts of a grammar 

Interpretation of modifications with the indications of statistics 

Multiplexed output and 
statistics about results 

Multiplexer 

parse 
result 
Rn+1 

Property 
Gramma 
r Ver n+1 

parse 
result 
Rn 

Property 
Grammar 
Ver n 

Shallow 
parser 

Different Large tagged 

  Analysis   

  Versioning   

Syntactic phenomena 

parse 
result 
Rn 

Property 
Grammar 
Ver n 

parse 
result 
R1 

Property 
Grammar 
Ver 1 

non-determinist Deep parser 

tagged 
Corpus 

… … … 

… … … 
128
attested productions, in order to integrate to our basic grammar the properties corresponding to our theoretical conclusions. Then we use a deep parser to analyse our test-corpus and compare the quality of the new results with the ones obtained from the precedent version of the grammar. Such evaluation takes into consideration the evolution of the quantity of parsing proposals according to the modifications of the grammar (i.e. conclusions about the consequences of introductions, suppressions and/or corrections, and in which proportions), the evolution of the quality of these proposals (number of relevant parses), the emergence of new suggested structure patterns, etc. 
The second development prospect concerns the coverage of the grammar. In this case the first stage consists in isolating, among the results obtained with the first version of the grammar, various occurrences of a particular construction (e.g. unbounded dependencies). We can then study in details the errors caused by the grammar (omission, imprecision, etc.). Once corrected, we run the parser again with the new version of the grammar to compare the quality of the results. 
Thirdly, in order to evaluate the set of properties itself and its semantics, we can isolate a type of property and observe its behaviour. This use of the deep parser allows to observe directly the impact of a type of property on the results. This can lead to modify the set of constraints itself, i.e. their operating mechanism and/or their presence. For example we can compare the quality of one parse result obtained with only one type of property on the one hand, and another one obtained with the other properties on the other hand. Then, while referring to the results obtained with the entire grammar, we can deduce the weight of a type of property on the parsing, either by its presence (for example if the majority of relevant analyses are found in the results obtained with only that one), or by its absence (for example if the majority of relevant analyses are found in the results obtained without this property, and if the number of false results is decreased by its absence). This can lead to an in-depth redefinition of a type of property, for example a modification of its semantics (i.e. its satisfaction mechanism), and even its suppression from the grammar (if it appears to be useless, and even more a source of recurrent errors). 
Let us consider the parsing results for the following phrase (coming from a dialogue corpus): “Alors on vous demandera aussi heu pourquoi c'est le meilleur” (“so we'll ask you too hum why it's the best”). One parsing proposition with the first version of the grammar is: 
figure 4: A parsing proposition with the first version of the grammar 
Following the approach explained above, we have done a lot of changes in the grammar using alternatively one of the three methods described here, and get another parsing proposition with a later version of the grammar: 
figure 5:A parsing proposition with a later version of the grammar 
129
Each modification of grammar, whatever the method, can have side effects on the rest of the parse (i.e. on the elements that we put aside during our different focusings). For example a modification of the constraint graph describing a category, aiming at allowing a particular structure treatment can lead to the proliferation of non-relevant parses for the rest of the corpus (e.g. this structure can then be proposed more often, which can cause as a consequence a more significant number of erroneous proposals). This is the reason why we have, for each modification of the grammar, to launch the parse again on the entire test-corpus. If necessary, a new handling of the grammar is to be planned so as to control the effects of the described process. This checking can be done with our deep parser, or with a shallow parser if we want to widen significantly the size of the corpus to parse. 
Each new grammar version is subsequently tested over large corpora by means of a shallow parser. The evaluation relies on the analysis of outputs and consists in comparing phrase boundaries of each sentence and in studying local statistics about the width, the nature an the count of parsed phrases. The Property Grammars parser used for these tests is deterministic and implements some heuristics in order to control the process. Each sentence is parsed once. The result is built with a stack of categories obtained by means of a classic left-corner parsing, and with a dynamic satisfaction algorithm of the constraint set given by the grammar. The main interest of this parser is that it provides linguistic information about chunks, with hierarchical features, while keeping fastness and robustness as with other shallow parsers. 
Thus it is necessary within this grammar development framework to use a tool for comparing parsing results which is fast, complete and efficient. This is the reason why we must handle a test-corpus (so that the provided results can be comparable in a reliable way) and an automatic tool capable of providing quickly and intelligibly some information about common points and differences between results. In this way we can preserve throughout a detailed, step-by-step elaboration, a general point of view on the efficiency of the grammar, which is of primary importance so that the provided results remain homogeneous. 
The results of these tests are interesting for grammar development process as for parser improvement. Such a goal is reached with a parameterised automatic evaluation strategy (cf. [Blache02a]) which uses a multiplexer. The technique operates on common boundaries. Its output is the intersection or the difference of the inputs and statistics are given about the relative importance of the categories detected for each border. Set operators such as union, difference and intersection are applied on input borders as well as on categories of compared phrases. This multiplexing method is interesting for an evaluation process without needing a treebank. 
The first experiment consists in testing different versions of the grammar over the “treebank corpus”, so that the shallow parser results can be compared for two grammars in order to allow a manual correction of the grammar. The second experiment uses several corpora with different characteristic features such as spoken language, literary style etc. Its aim is to show efficiency of grammars on unrestricted texts. Results improve our knowledge on side effects, limits and the relative importance of properties. Both experiments give statistics about each kind of phrase, their width, their depth, their position. The multiplexed evaluation gives a precise comparison of grammars and information about their main differences. With this tool, we can observe differences and likelihood between two compared grammars in quantitative terms. 
The sample results below, obtained on the test corpora, show two grammars letting NPs unchanged, and giving 25% of different VPs and 15% of different PPs. 
130
Figure 6: Some sample results of a multiplexing process for two grammar releases on a same corpus 
5. Conclusion 
This paper describes a technique for empirical corpus-based grammar elaboration. Our approach relies on a fully constraint-based formalism that makes it possible to evaluate precisely each constraint. This is one of the main advantages of such a technique in comparison with classical derivational methods. It allows in particular to evaluate separately any constraint as well as any kind of constraint. This method constitutes then an efficient platform both for developing grammars as well as experimenting and evaluating theoretical results. 
The architecture proposed here makes it possible to implement an incremental and step-by-step grammar development environment in which several parsers are used both for developing and evaluating the grammars. 
References 
[Abeillé03] Abeillé A. 2003, Une grammaire électronique du français, CNRS Editions. 
[Blache00] Blache P. 2000, “Constraints, Linguistic Theories and Natural Language Processing”, in Natural Language Processing, D. Christodoulakis (ed), LNAI 1835, Springer-Verlag 
[Blache02a] Blache P, Van Rullen T. 2002, “An evaluation of different symbolic shallow parsing techniques”, in procs of LREC-02. 
[Blache02b] Blache P, Guénot M.-L 2002, “Flexible Corpus Annotation with Property Grammars”, in procs of the International Workshop on Treebanks and Linguistic Theories. 
[Butt99] Butt M., T.H. King, M.-E. Nino, & F. Segond 1999, A Grammar Writer's Cookbook, CSLI. 
[Copestake01] Copestake A. 2001, Implementing Typed Feature Structure Grammars, CSLI. 
[Kaplan02] Kaplan R., T. King, J. Maxwell III 2002, “Adapting Existing Grammars: The XLE Experience”, in proceedings of Workshop on Grammar Engineering and Evaluation (COLING-02). 
[Kinyon02] Kinyon A. & C. Prolo 2002, “A Classification of Grammar Development Strategies”, in proceedings of Workshop on Grammar Engineering and Evaluation (COLING-02). 
[Prince93] Prince A. & P. Smolensky 1993, “Optimality Theory: Constraint Interaction in Generative Grammars”, Technical Report RUCCS TR-2, Rutgers Center for Cognitive Science. 
[Simov02] Simov K. & al. 2002, “Building a Linguistically Interpreted Corpus of Bulgarian: the BulTreeBank”, in proceedings of LREC-02. 

Grammar GP1 GP2 Phrases / sentence19.0418.97 Words / phrase 1.50 1.50 

NP StatsGP1 GP2 GP1 100%100%GP2 100% 

VP Stats GP1 GP2 GP1 100% 75% GP2 100% 

PP StatsGP1 GP2 GP1 100% 85% GP2 100% 
131

Towards an integrated model of service encounters 
Birte Bös, Rostock, Germany 
1. Introduction 
Service encounters are everyday social contacts, in which certain goods and/or services are provided. Despite booming Internet and shopping channels most of these encounters are still carried out face-to-face as they have always been, e.g. at department stores, travel agencies and restaurants. These institutionalised interactions are relatively clearly structured, and they generally have a fixed role allocation, involving (at least) a customer and a server. Service encounters are usually realised in a complex interplay of cognitive processes, dyadic verbal components, non-verbal elements and physical actions; and it has been stressed that these aspects should be considered in discourse analysis (cf. Ehlich 1993: 124). 
Looking at various existing models of service interactions one gains the impression that most of them have certain deficiencies. For example, the diverse phase models, which are particularly popular in practical sales guides, but also suggested in some linguistic publications, mostly represent a rather unrealistic picture. Besides postulating fixed sequences of phases, they often focus on the server's perspective, neglecting the interactive character of the event. Another disadvantage of most of the approaches examined is that they do not capture the actual realisation of the interaction. This is why this study strives for an integrated model, which allows for a comprehensive representation of service encounters (cf. Bös forth.). 
It is obvious that such a model cannot be developed without authentic data. Therefore, a corpus of 100 service encounters recorded in London bookstores was established as an essential prerequisite. As it is one of the main concerns to go beyond the standard focus on the verbal elements, the scope of this study was extended to include selected facial expressions, gestures and physical actions documented during the recordings at the actual point of sale as well. 
Based on a close analysis of this material and the existing representations, an explanatory model was created, which is shown (in extracts) and described in section 2. In section 3, the proposed form of representation is applied to a sample interaction from the corpus, illustrating that the model cannot only cope with idealised service encounters, but is also capable of reflecting the characteristics of specific service interactions. 
2. The integrated model 
An integrated model of service encounters has to meet various requirements. One of the key objectives is to develop a general pattern which can serve as the basis for the representation of individual service encounters. This general form of the integrated model should provide all possible components of service encounters in bookstores as well as different alternatives of their realisation. It should be flexible enough to be adapted to specific service encounters. The model should include both the customer's and the server's perspectives, whose contributions are considered as equally important for the progress of the interaction. Besides, it should capture communicative as well as physical and cognitive aspects and illustrate their interplay. For the diverse activities involved in service encounters, a unified representation appears necessary to allow for a better comparability of the interactions. 
132
Global structure of service encounters 
One of the first steps in the development of the integrated model was the investigation of the global structure of service encounters in bookstores. The corpus analysis has produced a set of 10 possible structural elements, which, in the individual interactions, occur in various combinations. A full list of these components is given in Fig. 1. 
• RUN UP (RU) 
• ESTABLISHING CONTACT (EC) 
• GREETING (GR) 
• SERVICE BID (SB) 
• NEED PRESENTATION (NP) 
• SERVICE (SE) 
• PURCHASE DECISION (PD) 
• PURCHASE REALISATION (PR) 
• FAREWELL (FA) 
• BREAKING UP CONTACT (BC) 
Fig. 1 Global structural elements of service encounters 
Although the RUN UP precedes the actual service encounter and is, in this respect, not part of the interaction proper, it is included in the model, because it essentially influences the progress of the interaction. In shops with self-service like bookstores, customers may browse, select products, and even make their purchase decision in the RUN UP, before they turn to the salesperson for help, for further information or simply to pay for the goods chosen. 
The remaining global structural elements differ in their degrees of obligation. The only obligatory elements observed in the corpus are ESTABLISHING CONTACT, BREAKING UP CONTACT and the NEED PRESENTATION. More specifically, ESTABLISHING and BREAKING UP CONTACT are physical elements which are necessary to create the face-to-face situation of the interaction. In the investigated data it is usually the customer who approaches or leaves the counter, where the server is placed. The most central obligatory element is the NEED PRESENTATION. Since it is the primary goal of every service encounter to satisfy certain customers’ needs, inevitably, these needs have to be expressed at some point in the interaction either verbally or non-verbally. 
SERVICE, PURCHASE DECISION and PURCHASE REALISATION are considered to be obligatory alternatives in the course of the service interaction. As the corpus analysis has proved, a considerable number of service encounters does not involve the transfer of goods but the supply of information or other services, which are provided in the SERVICE element. This means, a service encounter does not necessarily include a (positive or negative) PURCHASE DECISION of the customer. This cognitive process is only realised when certain products are involved. If there is a PURCHASE DECISION, and it is positive, the actual exchange of money and goods takes place in the PURCHASE REALISATION, which usually consists of a routine sequence of typical physical actions and may be supplemented by certain communicative acts. 
Components like GREETING, SERVICE BID and FAREWELL are communicative elements which are truly optional, i.e. all of them can be omitted, and the data show that often they are indeed not realised. The element GREETING occurs in only 56% and the SERVICE BID even less often, namely in 14% of the service encounters in the corpus; the FAREWELL is documented more frequently in 85% of the cases. 
133
Local structure of service encounters – primitive acts 
Another aim of the integrated model is to provide a detailed, but comparable description of the various activities performed during service encounters. Searching for a way to represent the communicative, physical and mental components of the interactions, the concept of primitive acts, first suggested by Schank (1972), proves helpful. As their application in the cognitive script model by Schank/Abelson (1977) has shown, primitive acts allow for a unified representation of actions. Therefore, they can help to clarify certain regularities of service encounters independently from their individual realisation. For its use in the integrated model, the set of primitive actions was adjusted and now comprises the acts summarised in Fig. 2. Apart from modifying the meaning of acts such as MTRANS (which, in Schank's classification, includes the transfer of information between persons), and placing more emphasis on the communicative function of SPEAK, the new act GEST was introduced to account for the non-verbal aspects of communication. Additionally, the primitive act ATTEND is used to document the non-verbal aspect gaze (‘ATTEND eyes'). Another new act is OPERATE, which allows for the description of certain activities such as writing, using the telephone, computer etc. 
PHYSICAL PTRANS Change of the local position of a person or object; or transfer of an abstract relationship, e.g. possession GRASP Griping of an object by a person INGEST Taking in of an object by a person EXPEL Expulsion of an object from the body of a person into the physical world OPERATE Performing of an action involving the use of a certain instrument, e.g. ‘OPERATE computer', ‘OPERATE pen’ MENTAL MTRANS Transfer of information within a person MBUILD Construction of new information based on old information ATTEND Focussing of a sense organ to a certain stimulus COMMUNICATIVE SPEAK Transfer of information between persons by producing sounds GEST Transfer of information between persons by moving parts of the body or the face, e.g. ‘GEST lips ª’ (smile) 
Fig. 2 Modified set of primitive acts (based on Schank/Abelson 1977: 12ff) 
This limited set of primitive acts presupposes a basic level of description. It does not include the instrumental acts which are necessary to perform the suggested acts. For example, GRASP could be split up into various subordinate acts, e.g. moving the arm, bending the finger etc. On the other hand, the act GRASP itself can operate as an instrumental act, e.g. in the realisation of a PTRANS of an object. However, in this function it is, like all other instrumental acts, neglected in the model proposed. In this way, the modified set tries to avoid ‘endless conceptualisations’ which have been criticised as one of the major problems of Schank's concept (cf. Dresher/Hornstein 1976: 368). 
Graphic representation of service encounters 
Fig. 3 shows an extract from the general form of the integrated model illustrating the graphic arrangement which is used to capture the structure of service encounters. It demonstrates how the two perspectives of the major participants and various alternatives in the realisation of a service interaction are integrated in one form of representation. As mentioned before, both interactants are considered to have an equal share in the interaction, therefore they are assigned an equal space in the model, the left half of the diagram being reserved for the customer's, the right for the server's contributions. 
134
CUSTOMER (C) SERVER (S) RUN UP 
ESTABLISHING CONTACT 
GREETING 
SERVICE BID RUN UP 
ESTABLISHING CONTACT 
GREETING 
SERVICE BID NON-PERCEPTIBLE PERCEPTIBLE NON-PERCEPTIBLE NEED PRESENTATION C MTRANS 
Present need? 
C MBUILD yes 
no 
C MTRANS 
Continue SE? 
C MBUILD yes 
no C MTRANS 
Specification necessary? 
C MBUILD yes 
no C SPEAK / C GEST 
or: C PTRANS P to S 
C SPEAK / C GEST S SPEAK / S GEST 
S SPEAK / S GEST S MTRANS 
Need been 
presented? 
S MTRANS no S MBUILD 
Does C have 
need? yes 
yes S MBUILD 
no 
S MTRANS 
Specification necessary? 
yes S MBUILD 
no NEED PRESENTATION SERVICE C MTRANS 
Service necessary? 
C MBUILD yes C MTRANS 
Support service? 
no C MBUILD yes 
no 
C MTRANS 
All service needs 
supplied? 
no C MBUILD 
yes C SPEAK / C GEST 
(+ others) S SPEAK / S GEST 
(+ others) S MTRANS 
Service necessary? 
yes S MBUILD 
no 
S MTRANS 
All service needs 
supplied? 
S MBUILD no 
yes SERVICE PURCHASE DECISION 
PURCHASE REALISATION 
FAREWELL 
BREAKING UP CONTACT PURCHASE DECISION 
PURCHASE REALISATION 
FAREWELL 
BREAKING UP CONTACT 
Fig. 3 General integrated model for the representation of service interactions (extract) P = product 
135
The extract contains the SERVICE BID, NEED PRESENTATION and SERVICE, i.e. three out of the 10 components of a service interaction, which the analysis of the corpus has yielded (cf. Fig. 1). These global structural elements are given in the left and the right margin of the figure. They are indicated separately for customer and salesperson, because the data show that asymmetries can occur (cf. section 3). In the general model, the structural elements are arranged in the idealised sequence presented in Fig. 1. However, this order is not fixed, but can be rearranged to represent specific service encounters. 
The various activities in the realisation of service encounters are described with the help of the primitive acts assembled in Fig. 2, and can be classified as perceptible or non-perceptible. This distinction is shown in Fig. 3 by their graphic separation. The perceptible segment contains the actual observable interaction. These communicative (i.e. verbal and non-verbal) and physical activities of the participants are arranged in the middle of the diagram (where they are again differentiated into customer's and server's part). The general form of the model can only include the primitive acts which are typically used in the realisation of the various global structural elements. However, as Fig. 4 will illustrate, this segment extends immensely in detail when the model is applied to a specific interaction. 
What should always be taken into account is that the observable interaction is determined by the underlying mental processes of the interactants. An explanatory model aiming at a comprehensive picture of a service encounter should therefore also include cognitive aspects. These non-perceptible elements are taken down separately for the customer and the server on the left and right of the figure respectively (shaded grey). As the interactants are not just “passive vehicles in the unfolding of generic structure”, as George (1988: 316) puts it, but “knowledgeable participants”, their decisions presuppose a background knowledge of continually changing experiences and expectations. 
Generally, assumptions about cognitive processes are, of course, of a hypothetical nature and involve interpretative problems. Their representation in the model has to remain restricted to standardised points of decision. The principle of incorporating yes/no choices is also used, for example, in flowchart models (e.g. Ventola 1987), and makes it possible to include alternatives for the progress of the interaction which are indicated in Fig. 3 by the arrows. 
3. Application of the model 
The general integrated model introduced in section 2 can flexibly be adapted to represent specific service interactions by adjusting the number and sequence of global structural elements and adding the relevant primitive acts as well as the actual wording of the interaction. 
The analysis of the corpus has revealed that the complete, idealised sequence of structural elements presented in Fig. 1 does, in fact, never occur. Instead, there are certain typical combinations depending on whether the interaction is focussed on the provision of goods and/or different kinds of services. As this determines the choice and realisation of the obligatory alternatives (SERVICE, PURCHASE DECISION, PURCHASE REALISATION, s.a.), eight subtypes of service encounters in bookstores can be distinguished according to their global structure. It does not come as a surprise that in a self-service environment like the bookstore the most frequent type of interaction is the simple payment, which does not include a special SERVICE element. However, there are, for example, also many cases in which the customer seeks only the advice of the salesperson and, thus, no PURCHASE DECISION and REALISATION are required. As the global structure of the integrated model is not considered to be fixed, the model can cope with the different subtypes. 
The flexible form of representation also allows for a visualisation of the three major structural peculiarities which could be observed in the corpus: loops, embedded elements, and asymmetries. As 41% of the service encounters display one of these divergences or a combination of them, approaches which postulate linear sequences of phases are disproved. Loops are caused by the repetition of structural elements. For example, customers often present more than one need in the course of the 
136
service encounter. As a result, NEED PRESENTATIONS and, of course, also other elements, e.g. the ones in which the needs are dealt with, can occur repeatedly. Embedded elements are performed while another element is in progress. A customer might, for instance, ask for further information while s/he is paying for some goods. In such cases, the (second) NEED PRESENTATION is embedded in the PURCHASE REALISATION. Loops and embedded elements cannot only occur separately or in combination with each other, they can also be linked with asymmetries, i.e. asynchronous sequences, in which the interactants are engaged in different structural elements. A sample of these phenomena is given in Fig. 4. 
For the description of the local structures of a particular service interaction, the activities displayed in the non-perceptible and the perceptible segments of the diagram have to be specified. The non-perceptible, cognitive processes, which involve a variety of alternatives in the general form of the model, can now be geared to the actual service interaction. Consequently, the representation will be restricted to those points of decision and their resolutions which can be derived from the progression of the specific interaction. The perceptible segment of the model, on the other hand, which contains only a few typical primitive acts in the general form, can be enlarged with the details of the concrete verbal, non-verbal and physical realisation of the encounter. For the inclusion of the actual verbal utterances, a new feature is added to the figure. The dialogue is shown in a column in the middle of the diagram, thus symbolising the centrality of this part in the interactive process. 
Fig. 4 illustrates the way the necessary adaptations are accomplished and the potential such a form of representation offers. The extract shown is taken from a service encounter of the corpus which does not involve any goods, but only the advice of the salesperson. The customer, in her first NEED PRESENTATION, has asked the server for books on a certain subject (‘erm I'm looking for something on – marriage as opposed to weddings – the state of marriage‘). In the subsequent SERVICE section (SE1), the server starts checking on the computer, but due to the fact, that the search word is rather vague, she has difficulties in finding suitable publications (cf. Fig. 4, ‘...I'm not coming across anything in general'). 
The global structure of the example demonstrates the peculiarities discussed above. It contains a loop, as well as cases of embedded elements and asymmetries. For example, the customer accepts the negative reply of the server (‘okay') and then comes up with a new NEED PRESENTATION (NP2) (‘alright I'll just have a look on my own do you have a philosophy section?). It appears that for the customer the SERVICE (SE1) for her first need is finished (though unsuccessfully), and, as the next issue is brought up (NP2), a loop occurs in the structure of the encounter. However, the server still tries to deal with the first problem, even after the customer has signalled that her needs have changed. While the salesperson provides the information asked for (SE 2), she still continues her search for the first request. Consequently, from her perspective, NP2 and SE2 are embedded elements, performed while SE1 is in progress. As this is not the case for the customer, an asymmetry arises, which continues up to the end of the interaction. At that point, the server is still busy operating the computer, searching for suitable books, while the customer impatiently tries to signal that the interaction is finished by applying the discourse marker ‘okay’ and an expression of thanks (FA). Even as she leaves the counter, the server is still involved in SE1. 
137
C NON-PERCEPTIBLE PERCEPTIBLENON-PERCEPTIBLE S SE 1 --- etc. --- 
I've just come up with so many different ones but I'm not coming across anything in general S SPEAK 
S ATTEND eyes to screen SE 1 NP 2 C MTRANS 
Present need? 
C MBUILD 
yes C SPEAK 
C SPEAK 
C SPEAK okay 
erm the thing is all the books are in the region of about thirty five to fifty plus pounds 
hm hm 
so that's all sort of 
alright I'll just have a look on my own do you have a philosophy section? S SPEAK 
S SPEAK SE 1 C MTRANS 
Service necessary? text ones erm S SPEAK 
+ S ATTEND eyes to C S MTRANS 
Need been presented? 
S MBUILD 
yes NP 2 SE 
1 SE 2 C MBUILD 
yes 
C MTRANS 
Support service? 
C MBUILD 
no yeah that's downstairs S SPEAK S MTRANS 
Service necessary? 
S MBUILD 
yes SE 2 SE 
1 I mean I've got things on parenting . erm about . perspectives on the family 
(13 sec) 
there's nothing that's really – general what you're looking for S ATTEND eyes to screen 
+ S SPEAK 
S OPERATE computer 
S SPEAK SE 1 FA C MTRANS 
Terminating greetings necessary? 
C MBUILD 
yes C SPEAK okay . thanks very much S OPERATE computer 
Fig. 4 Representation of a sample (extract) 
138
As emphasised before, the cognitive, verbal, non-verbal and physical elements of the interaction are “not discrete activities but aspects of an ongoing stream of behavior” (Arndt/Janney 1987: 4). Fig. 4 exemplifies how this interplay can be illustrated in the integrated model. The perceptible part of the extract contains mainly verbal activities of the participants, represented by the primitive act SPEAK. Additionally, we find instances of non-verbal behaviour, e.g. visual contact (‘S ATTEND eyes to C'), and physical actions like the salesperson's use of the computer (‘S OPERATE computer'). As pointed out above, the underlying cognitive processes are restricted to selected points of decision. The chronological order of the activities is represented by their position in the diagram, their causal relations by the arrows connecting the entries. Simultaneous actions are placed on one horizontal line, as, for instance, demonstrated in the last line of the extract, where the customer is speaking and the server is still involved the computer search at the same time. Needless to say, the non-perceptible activities (which are frequently realised in a split second) are usually running in parallel with the perceptible ones. The sequences of events can be derived from their vertical arrangement. 
Often, it is this interplay of features which clarifies apparently abnormal phenomena in the plain text versions. For example, the inclusion of physical activities can explain the occurrence of pauses in the verbal part. In the corpus, long pauses (of 5 seconds up to several minutes) are usually observed when physical tasks are performed. Our sample interaction contains five instances of long pauses caused by the server operating the computer and waiting for the search results. They vary in length from 13 seconds (as shown in the extract) to up to 25 seconds. The neglect of physical and non-verbal aspects in the analysis of service encounters might lead to the impression that certain global structural elements are missing. The obligatory NEED PRESENTATION, for instance, is realised non-verbally in almost 30 % of the cases, usually by handing over a product for payment (‘C PTRANS P to S'). These instances would, of course, escape the analyst of the plain text versions. 
Thus, for a comprehensive analysis of service encounters the concentration on the verbal parts of the interaction proves insufficient. With the integrated model discussed in this paper, however, a detailed picture of service interactions in bookstores can be achieved. As this model offers the chance to represent complex processes in a relatively clear graphic form, it could also be applied (with the necessary adjustments) to other types of service encounters. Besides, it might be interesting to test the potential of the graphic arrangement for the representation of other types of dialogues. 
References 
Arndt H, R W Janney 1987 InterGrammar. Toward an Integrative Model of Verbal, Prosodic and Kinesic Choices in Speech. Berlin, de Gruyter. 
Bös B forth. Integrative Darstellung englischer Serviceinteraktionen: verbale, nonverbale, physische und kognitive Aspekte. Dissertation. 
Ehlich K 1993 HIAT: A Transcription System for Discourse Data. In Edwards J, Lampert M D (ed), Talking Data: Transcription and Coding in Discourse Research. Hillsdale, NJ, Erlbaum, 123-147. 
George S 1988 Postscript: discourse descriptions and pedagogic proposals. In Aston G (ed), Negotiating Service. Studies in the discourse of bookshop encounters. Bologna, Editrice CLUEB, 303-332. 
Schank R C 1972, Conceptual Dependency: A Theory of Natural Language Understanding. Cognitive Psychology 3, 552-631. 
Schank R C, Abelson R P 1977 Scripts, Plans, Goals and Understanding. Hillsdale, NJ, Erlbaum. 
Ventola E 1987 The Structure of Social Interaction. A Systematic Approach to the Semiotics of Service Encounters. London, Frances Pinter. 
139

A Statistical Analysis of the Source Origin of Maltese – Abstract 
Roderick Bovingdon 
99 Chetwynd Road 
Merrylands NSW 2160 
Australia 
roderick_bovingdon@hotmail.com Angelo Dalli 
NLP Research Group1 
University of Sheffield 
United Kingdom 
angelo@dcs.shef.ac.uk 
The most recent theories relating to the original source of the Maltese language point to a direct Sicilian-Arabic connection (Agius, 1990; Agius, 1993; Agius, 1996; Brincat, 1994). This paper presents the results of the first ever large-scale statistical analysis of Maltese using the newly formed Maltilex Corpus (Rosner et al., 1999; Rosner et al., 2000). Traditional etymological and categorical analyses were supplemented with data mining techniques to provide accurate results confirming traditional subjective notions. 
The Maltilex Corpus is made up of a representative mixture of newspaper articles, local and foreign news coverage, sports articles, political discussions, government publications, radio show transcripts and some novels. As of the time of writing, the corpus had over 1.8 million words and almost 70,000 different word forms, making it the largest digital corpus of Maltese in existence. 
A random sample of 1000 unique word forms was selected from the corpus and the etymology and category class noted down for every word in the sample. Words falling under multiple category classes were duplicated and a single category was entered for every word to ensure unique etymology/word class pairs. Weights were added to every entry representing the number of category classes associated with a particular word to maintain accurate statistics. 
A data matrix with 1034 entries was thus obtained, representing all possible etymology/word class pairs for the sample word forms. The data matrix was analysed using a custom written data mining tool to extract statistics about the relationship between etymology and word classes in Maltese. Overall statistics about the source language origins of Maltese, together with the most commonly occurring word classes were also extracted. The use of a data mining tool enabled us to analyse the data from two different perspectives – word class distribution for every etymological class and vice-versa. 
Maltese grammar and morphology remain to this day largely Arabic, but with distinct Romance and English morphological accretions (Aquilina, 1973; Aquilina, 1979). Traditionally the Romance element in Maltese was thought to commence with the coming of the Normans in Medieval ages and intensified during the long period of rule under the Knights of Malta. Pre-Norman Arabic content appears to have been heavily Sicilian based (Brincat, 1995). 
The greatest linguistic inroads from the Italian mainland occurred during the early days of the British rule when large numbers of political refugees sought and were granted asylum during the unification of Italy (Friggieri, 1979). The most recent linguistic influence on Maltese is English. English has steadily and increasingly affected Maltese, adding another language facet to the overall structure of Maltese (Mifsud, 1995). 
Despite the relatively very recent accretions from English and the vastly different morphological structure of the two languages, the assimilation of English lexemes into a Maltese mould occurs with the least possible disturbance to Maltese morphology, especially with English verbs. Interestingly, contemporary Arabic is adopting similar assimilative patterns as Maltese in its borrowings from the English-American lexis. 
This study clearly shows that, in addition to other aspects, Italian lexical influence upon present day Maltese has exceed the Arabic content in a quantitative sense. Such development has also enriched Maltese from a purely root based morphology, with the additional productive Romance feature of catenation (Schweiger, 1994). 
1 With support from the Maltilex Project and the Department of Computer Science and Artificial Intelligence at the University of Malta. 
140
References 
Agius, Dionisius A. 1990. Il-Miklem Malti: a contribution to Arabic lexical dialectology, British Society for Middle Eastern Studies. 
Agius, Dionisius A. 1993. Reconstructing the Medieval Arabic of Sicily. Languages of the Mediterranean, Brincat, Joseph M. Msida, University of Malta. 119-129. 
Agius, Dionisius A. 1996. Siculo Arabic. Library of Arabic linguistics (Kegan Paul International) 12. 
Aquilina, Joseph. 1973. The structure of Maltese. Msida, University of Malta. 
Aquilina, Joseph. 1979. Maltese-Arabic Comparative Grammar. Msida, University of Malta. 
Brincat, Joseph. 1994. Gli albori della lingua maltese: il problema del sostrato alla luce delle notizie storiche di al-Himyari sul periodo arabo a Malta. Languages of the Mediterranean, Msida, University of Malta. 
Brincat, Joseph. 1995. 870-1054: Al-Himyari's Account and its Linguistics Implications. Msida, University of Malta 
Friggieri, Oliver. 1979. Storja tal-Letteratura Maltija. Msida, University of Malta. 
Mifsud, Manwel. 1995. Loan verbs in Maltese a descriptive and comparative study. Studies in Semitic 
languages and linguistics. Leiden: Brill. 
Rosner, Michael et. al. 1999. Linguistic and Computational Aspects of Maltilex. Proc. of the ATLAS 
Symposium, Tunis. 
Rosner, Mike, Ray Fabri, Joe Caruana, M Lougraieb, Matthew Montebello, David Galea, and G. Mangion. 1999. Maltilex Project, University of Malta. 
Rosner, Mike, Ray Fabri, and Joe Caruana. 2000. Maltilex: A Computational Lexicon for Maltese, Msida, University of Malta. 
Schweiger, Fritz. 1994. To what extent is Maltese a Semitic Language?. Languages of the Mediterranean, Msida, University of Malta. 
141

Xara: an XML aware tool for corpus searching 
Lou Burnard and Tony Dodd 
Research Technologies Service, Oxford University Computing Services 13 Banbury Road, Oxford OX2 6NN lou.burnard@oucs.ox.ac.uk tony.dodd@btinternet.com tel: +44 1865 273285 fax: +44 1865 273275 
From SARA to Xara 
Xara is the working name for a new version of SARA, the `SGML aware retrieval application' originally developed for use with the British National Corpus (BNC) in 1994. The system has been completely rewritten as a general purpose tool for searching large XML corpora, with a particular focus on the needs of corpus linguists, with close attention to new XML-based encoding standards, and with the benefit of hindsight derived from a decade of feedback from hundreds of SARA-users world wide. 
The Xara system combines the following components: (1) an indexer, which creates inverted file style indexes to a large collection of discrete XML documents; (2) a server, which handles all interaction between the client programs and the data files; (3) a Windows client, which handles interaction between the server and the user. The modularity of this architecture has several advantages, permitting, for example, the development of multiple specialized client programs for different applications or styles of usage. 
In addition, an index building utility, called Indextools, is supplied with Xara, which simplifies the process of constructing a Xara database. Its chief function is to collect information about the corpus to be supplied additional to that present in any pre-existing corpus header, and to produce a validated and extended form of the corpus header. It can also be used to run the indexer and test its output. 
Rather than review extensively its history and design philosophy in this short presentation, we give here some general comments covering the following aspects of the system: 
• XML support 
• Corpus related features 
• Ease of use 
XML support 
Xara will process any XML encoded corpus. The more detail present in the tagging, the more facilities are available to the client but the minimal requirement is only that the text be well-formed XML. If the corpus to be processed specifies a document type definition (DTD), then Xara will validate it at indexing time, and will not proceed if any validity errors are discovered. Unlike earlier versions of the program, any DTD may be used for this purpose, though Xara was (naturally) designed with the likely needs of such widely used DTDs as the TEI or CES in mind. Also unlike earlier versions of the program, as previously noted, no DTD at all need be supplied. 
Xara can thus do something useful with the full range of digital material one might wish to build into a corpus. At one extreme, we will demonstrate how it can be used to provide basic searching facilities for a collection of Project Gutenberg style texts, innocent of any explicit descriptive markup at all; at the other, we will show how it can also take full advantage of the rich annotation present in a multilingual corpus produced in full conformance with the XCES Guidelines, containing detailed feature structure analysis, POS-tagging, and explicit lemmatization. Oddly enough, the most problematic material is likely to be texts which have been marked up in loose (syntactically incorrect) HTML, such as that 
142
generated by automatic conversion from Microsoft Word; fortunately, utilities such as Dave Ragget's Tidy are readily available to generate well-formed XML from such conversions. 
With XML support, comes Unicode support. Xara uses Unicode internally to represent all character data: it can thus handle text in any language, and any combination of languages. To take full advantage of this, the user of the system needs convenient methods both for displaying and for inserting Unicode characters at their workstation. 
Good Unicode fonts are now available for display of texts in several different scripts (we have so far tested Chinese, Eastern European, Medieval, and Ancient Greek scripts) and the number continues to grow. Xara does not assume the existence of any particular font, however, and allows the user to select the display font at run time. In common with other XML systems, the system will correctly process character entity references found in the data, such references being retained untranslated in the underlying corpus index, but rendered as the appropriate Unicode code point when displayed in non-XML modes. 
To enter characters not found on the keyboard, for example to search for words containing them, however, an appropriately configurable input system is necessary. Dynamic keyboard redefinition is built into more recent operating systems, but is still a little user-unfriendly in most cases; the Xara client therefore includes its own keyboard redefinition facilities, which allow selection of specific characters from a Unicode table by point and click, temporary mapping of keyboard keys, or complete redefinition of a new keyboard map, which can be loaded as needed. 
Features for Corpus Linguists 
Any XML or SGML aware search engine has the ability to locate specific tagged components and to carry out searches within the context of such components. Most also have the ability to reorganize and display search results in a variety of forms. SARA extended these facilities with some additional, more lexically-motivated, abilities. These include: 
• implicit or explicit tokenization of element content 
• implicit or explicit lemmatization of element content 
• multiple keys for index searching 
• expandable automatic collocation search 
Xara supports the full range of facilities originally provided by Sara, but with several modifications and simplifications to the interface. A number of new facilities have also been added. 
Xara inherits from SARA a rich range of query facilities. The user can search for substrings or regexp-style patterns, words, phrases, or the tags which delimit XML elements and their descriptive attributes, either simply checking for their presence or absence in the lexical index maintained for the corpus (and inspecting their frequency or collocative patterns therein), or retrieving and displaying actual instances of the words etc. sought for. 
Searches can be made using additional keys such as part of speech, or root form (lemma) of a token, specified either explicitly in the tagging of the texts, or implicitly by means of a named algorithm. Xara also supports a variety of scoped queries, searching for combinations of words etc. in particular contexts, which may be defined as XML elements, or as combinations of other identifiable entities, or as stretches of text. Such searches may be order-sensitive or insensitive. 
Different kinds of search can be combined to form complex queries of various kinds, using either a simple graphical interface, or a (rather esoteric) `Corpus Query Language' (CQL). This language defines the full capability of the query interface and forms a major part of the protocol by means of which client and server modules communicate. In Xara, it has been re-expressed using an XML syntax, in line with the desire to leverage XML standards wherever possible. The client also supports a simple ECMAscript-like scripting language which makes direct calls on an API defined in very similar terms to the CQL. 
143
Like its predecessor, Xara displays the results of searches either one at a time or within a traditional KWIC style window which can be sorted, thinned, expanded etc. The context of hits located in this way can be explored by expanding it, up to the full text level if necessary. Unlike its predecessor, Xara allows user-definition of a range of formatting properties for KWIC and single hit displays, using a subset of the formatting properties defined by the W3C standard Cascading Stylesheets (CSS); it also allows the user to export results in a simple XML format which can then be reprocessed, either by Xara, or by any other XML-competent application, such as a word processor or XSLT engine. 
Corpora can be reorganized or partitioned in a user-defined way, using the results of any query, the values of specified element/attribute combinations, or a manual classification. Searches carried out across partitioned corpora can be analysed by partition: so the client can display the relative frequencies of a given lexical phenomenon in texts of different categories identified in a corpus. 
Using Xara 
For existing users of SARA, the main new feature of Xara will probably be the facilities it offers for indexing of new corpora. As noted above, all the parameters required by the indexer and server are now gathered and stored transparently within a TEI-conformant XML header file, rather than being supplied at runtime by various obscure control files, command line options, or hard-coded declarations. A new Windows utility has been included to facilitate creation and modification of this data: it allows the user to control the behaviour of the indexer by selecting available options from a series of dialogues, and saves the results of these decisions in an appropriate part of the TEI/XCES header. The same utility can be used to check validity of the supplied corpus files and to run the indexing utility, and provides an interface for testing that the system index files have been correctly generated. Its use is not however essential: other XML-aware software can be used to define a corpus header which Xara will use, and the indexer utility can be run independently. 
The indexer needs to be provided with the following information: 
• how PCDATA (element content) is to be tokenized 
• how tokens are to be mapped to index terms (for example, by lemmatization or by the inclusion of additional keys) 
• how indexed terms are to be referenced in terms of the document structure 
• how and whether XML tags and attributes are to be indexed 
Much, perhaps most, of this information is implicit in the XML structure for a richly tagged corpus: one could imagine a corpus in which every index term was explicitly tagged, with lemma, part of speech, representation etc. In practice however, such richly tagged corpora remain imaginary, and software performs the largely automatic task of adding value to a document marked up with only a basic XML structure. The new indextools utility specifies how the task is to be performed in a standardized way, using the existing structure of the TEI header (see http://www.tei-c.org/Guidelines/HD.htm) to hold the specification. Because our original design goal was to avoid any need for extension or modification of the existing TEI DTD; we have used very general constructions of the type<list type="foo"/> rather than defining more `application focussed' tags such as <foo>. We have however done this is a fairly consistent and easily validated form, so that when TEI support for external namespaces is available, it will be easy to make our generic tags more specific. 
The Xara system is currently under beta-test and will be demonstrated at the conference. We hope to make an initial general release available by the summer of 2003, but participants wishing to experiment with it earlier are very welcome to participate in the beta test. The first release will operate in any modern Microsoft Windows environment; it is intended, however, to port both indexer and server to Unix environments as soon as possible; it is also intended to licence open source versions of at least those components of the system at that time (open source licensing of the Microsoft-dependent components is rather less likely, for obvious reasons) 
144

Expressions and structures of the delexical verb ÊÁÍ [“MAKE” / “DO”] 
in Modern Greek language: A corpus-based approach to newspaper articles1. 
Marianna N. Christou 
Postgraduate research student in Greek language and lexicography 
at the University of Birmingham (UK) 
1. Introduction 
This is perhaps the first lesson to be learned from corpus study. 
Language cannot be invented; it can only be captured. 
(Sinclair 1997: 31) 
In this study I intend to explore the usage of the Modern Greek verb ÊÁÍ2 by means of a sub-corpus extracted from the 30-million-word written Hellenic National Corpus (HNC) developed by the Institute for Language and Speech Processing (ILSP) in Greece. In the belief that there is a cline of semantic differentiation between fixed expressions with figurative meaning on the one hand, and ‘simple’ collocations of ÊÁÍ with nouns on the other, I stress the need for applying this cline to language research. In addition, I present arguments for the explanation of the syntactic distribution of such phrases, which would contribute to the understanding of delexical structures. 
For the purposes of this study, I start with some methodological issues. Next, I adopt the term cline of idiomaticity for the development of a theoretical framework that supports the generalised structure of ÊÁÍ + noun, and divides all the instances of my sub-corpus into five categories. Subsequently, I refer to the distribution of the verb and its complement within either the same or different clauses. Finally, I discuss the significance of adopting the proposed cline of idiomaticity in dictionary-making, since this proposes a shift of the lexical load that a lexicographer needs to clarify and describe. 
2. Methodology 
This study, as has already been mentioned, is corpus-based, since it was built on a sample (see Appendix I) extracted from the HNC (on-line access: http://hnc.ilsp.gr). 
2.1. The data extracted from the corpus and their processing 
The whole corpus of my study was compiled by means of a lemma query. As the question posed initially was the examination of the role that the (delexical) verb ÊÁÍ plays in Modern Greek language, I expected that real data – even limited in number – would yield some fruitful results to this end. Thus, I restricted the lemma query of the verb ÊÁÍ to a particular medium (two of the most popular Greek newspapers, Åëåõèåñïôõðßá and Ôï ..µá, see Hatzigeorgiu et al. 2000: 1737), genre (informative texts) and topic (related to society). It has to be noted, though, that I made no further selection (e.g. according to more specific genre or topic), with the intention of achieving at least a representativeness “for certain high frequency linguistic features” (McEnery and Wilson 2001: 78). Therefore, my sub-corpus was at the same time small – compared with the whole 30-million-word HNC – and sufficient for the needs of the present research. 
To be more precise about the identity of the corpus used, this consisted of selected articles written between 1993 and 1997. The results showed 6,200 texts (4,851 from Åëåõèåñïôõðßá and 1,349 from Ôï ..µá), within which 4,236 instances (tokens) of the lemma KAN were automatically extracted. Then, all examples were processed by use of WordSmith Tools (henceforth WordSmith). 
1 I would like to thank Mrs. Maria Gavriilidou, Head of the Electronic Lexicography Department of the Institute for Language and Speech Processing (ILSP) in Athens, and Mr. Nikos Hatzigeorgiu, Head of the ILSP branch in Xanthi, for the invaluable information they provided me with concerning the Hellenic National Corpus (HNC). I am also grateful to Professor Emeritus Geoffrey N. Leech, Dr. Andrew Wilson, and Mr. Philip King for their help and useful comments on this study. 
2 Greek verbs are capitalised, when they are used as lemmas. 
145
WordSmith, developed by Mike Scott3, proved to be an invaluable tool for the processing even of a language with an alphabet other than the roman. It provided the concordance for the instances extracted from the sub-corpus (see Appendix II), since the equivalent Greek tool can only be applied to the whole HNC for the moment. Nevertheless, WordSmith was by no means the most perfect and accurate solution to my problem. This became more obvious when I saved the HNC concordance output (4,236 instances) to file, and then ran Wordsmith on it. By virtue of some partial incompatibility between the two programmes, I had to remove some incomplete, misread or unrecognisable sentences manually. After that, 4,059 tokens of the lemma ÊÁÍ underwent a closer examination on an advanced stage of research. Subsequently, I sorted out the instances that I would later use by adding extra information in the set column of the concordance (Categories A, B, C and D, see section 3.2. ff.). Next, I both re-sorted the whole sub-corpus and weeded out the sentences that would be of no interest to me (Category E, see section 3.2.5.). Thus, I ended up with 2,139 instances. 
Since the annotation of the corpus used was considered an essential prerequisite for this research, I added simple tags4 manually. It is important to acknowledge here that this (non-machine-aided) procedure may not be one hundred percent accurate and reliable; however, the aims of the present study were accomplished, since all problems encountered were solved to a great extent. 
2.2. Problems encountered and their solutions 
The table below illustrates some of the problems encountered, although not always anticipated throughout the manipulation of the data. Moreover, it is also hereby shown how I temporarily (i.e. for the purposes of the present study) solved them, along with how I perceive their potential amelioration. 
Table 2.2.: Problems encountered and their (temporary and future) solutions 
Problems encountered Temporary solutions Future solutions 
  no reliable parsers / taggers for the Modern Greek language available for the moment 
  manual annotation of the verb and the noun only across all 2,139 instances 
  improvement of the existing tools (of the ILSP) and / or development of new ones 
  consequence of the previous problem: Greek text untagged – no possibility of identifying the systematic co-occurrences 
  related to the preceding problem: practice on a combination of WordSmith, Excel and Notepad to find collocations 
  part of the above problem: amelioration of the concordance tool, so that it can carry out more complex commands 
  partial incompatibility between the number of results of the HNC (4,236) and their insertion to the WordSmith programme (4,059) 
  manual elimination of the (few) incomplete, misread or unrecognisable examples 
  improvement of both the WordSmith programme and the Greek tool for concordance (the latter does not accept a query on a particular sub-corpus) 
  large amount of instances (4,236) needed for a representative (insofar as possible) sample of newspaper articles 
  reduced number of instances (2,139) taken into account – a theory based on their closer reading 
  incorporation of all instances found could have a further impact on this theory, i.e. either consolidate it or modify it 
  significant imperfections of the HNC (e.g. words not disambiguated yet) 
  reliance placed on the trustworthiness of the results 
  enhancement of the morphological lexicon and machine-aided disambiguation 
3 For more information on WSmith see McEnery and Wilson (ibid. 211). 
4 Leech and Smith (1999: 24) have demonstrated that annotation is useful for ‘inputting’ information, whereas a concordance programme helps in ‘outputting’ information from a corpus. 
146
Problems encountered (cont.) Temporary solutions (cont.) Future solutions (cont.) 
  only one sentence of the corpus available, the one containing the node word (need for tracing back for more context, which was impossible using the WordSmith concordance) 
  work simultaneously at three levels: results and context (HNC), concordance (WordSmith) and manual annotation (Notepad) 
  expansion of the number of sentences that the HNC allows on its web interface (e.g. by allowing users to define by themselves the amount of context needed) 
  certain limits of space and time 
  a general theory introduced 
  evaluation and review of the work done 
3. Data analysis 
3.1. Different approaches to the notion of ‘delexical structures’ 
Until recently, there has been no attempt to standardise the terminology through which verbonominal structures (Stein 1991: 4, Nakas 2000: 125 ff.)5 are defined. That is, several of the terms suggested refer either to the verb itself or to the noun. More precisely, the idea of [semantically] ‘empty’ or ‘light verbs’ (Jespersen 1942: 117 ff.), on the one hand, has led modern theory to extremes, i.e. this has been both rejected (by Stein 1991: 15) and adopted (by Biber et al. 1999: 428). On the other hand, the labels of ‘eventive object’ and ‘deverbal noun’ (ibid. 128 and 428, Quirk et al. 1985: 750 ff.)6 are, among other labels, attributed to the noun that collocates with verbs of this kind. 
In this study, I shall use the term ‘delexical verbs’ as defined by Sinclair et al. (1998: 147): 
[t]here are a number of very common verbs which are used with nouns as their object to indicate simply that someone performs an action, not that someone affects or creates something. These verbs have very little meaning when they are used in this way. 
for two reasons: first, in order to benefit from corpus evidence to support the cline of semantic shift for the delexical structures, and second, in order to complement the definition in Babiniotis’ dictionary (2002: 246), in which it is stated that the meaning of a verb sometimes takes the form of a periphrasis instead of being represented by a cognate simple verb/lexeme7. As we shall see later, even though there is not always such a possibility of substituting the periphrasis for a simple verb deriving from the noun's stem, we still define the verb as delexical. In this sense, we accept that the ‘lexical load’ is carried by the second part of the phrase (Live 1973: 31). 
As regards the present study, the results of the concordance of the verb ÊÁÍ revealed a variety of word-classes (e.g. noun, adjective, article, adverb, pronoun, preposition, conjunction etc.) that are commonly combined with it. However, the major issue of my concern will be the collocations of this verb with its nominal complements (nouns / noun phrases). 
3.2. Collocations of the delexical verb ÊÁÍ + (noun / noun phrase): the cline of idiomaticity 
Adopting Sinclair's terminology (1991: 115), ÊÁÍ could be regarded as a ‘node', and its complement – any noun / noun phrase, in this case – could be considered its ‘collocate’ in that 
[c]ollocation is the occurrence of two or more words within a short space of each other in a text … Collocations can be dramatic and interesting because unexpected, or they can be important in the lexical structure of the language because of being frequently repeated (ibid. 170). 
5 In his article (1968), Nickel introduces the equivalent concept of “complex verbal structures”, while Live (1973: 32) preferably accepts a ‘phrasal form’ of the ‘light verbs'. 
6 Stein (1991: 2) and Allan (1998: 2-3) provide a more general overview of the terminology used in the past. 
7 Using the term simple (= single-word) verb I have translated the Greek µïíïëåêôéêü ..µá (e.g. äçëþíù [“state”] instead of ÊÁÍ äÞëùóç [“make a statement”]) which a) constitutes one word, b) has an equivalent meaning with the periphrasis and c) should be contrasted to (English) phrasal verbs (two or more words). 
147
On the one hand, before looking into the corpus, I surmised that the core meaning of the verb would be expanded as well as restricted to some extent, since ÊÁÍ is among the most common verbs in Modern Greek (cf. “make” / “do”8 (English), “faire” (French), “machen” / “tun” (German), “hacer” (Spanish), etc.). A closer examination of the corpus, on the other hand, allowed me to provide concrete examples in support of the theory that I will develop next. Having named this cline of idiomaticity, on the basis of a proposal by Biber et al. (1999: 1026), I shall further suggest five distinct Categories. 
3.2.1. Category A: ÊÁÍ + noun = fixed expression with figurative meaning (ÊÁÍ öôåñÜ) 
The first Category comprises instances of fixed (idiomatic) expressions with the verb ÊÁÍ, which have figurative meaning. The term ‘fixed expressions’ denotes that, whereas the verb conjugates regularly, the collocate remains uninflected, e.g. ÊÁÍ öôåñÜ [“vanish”], ÊÁÍ èñáýóç [“be popular”], ÊÁÍ <êÜôé> öýëëï ... ..... [“search sth. thoroughly”]. Furthermore, it has to be made clear that adjectives modifying the nominal complement of the verb ÊÁÍ do not frequently intervene, except in cases where they form part of the expression, e.g. ÊÁÍ ôá ....ß. µÜôéá [“turn a blind eye to sth.”] (but not * ÊÁÍ (ôá) µÜôéá), ÊÁÍ ÷ñõóÝò ........ [“earn a lot of money”] (but, ÊÁÍ äïõëåéÝò has a totally different (literal) meaning). Similarly, an indirect object of the verb is sometimes essential for the syntactic structure to be considered as grammatical, as in ÊÁÍ <óå .......> ôï ....... [“prepare and invite sb. for a meal”] and ÊÁÍ <óå .......> ôïí ß.. .ß.... [“make life unbearable for sb.”]. 
Finally, examples of rather informal or colloquial set phrases mentioned in the newspaper articles have been incorporated in the same Category, given that they constitute collocations: ÊÁÍ êÝöé [“feel like doing sth.”], (different from ÊÁÍ <êÜðïéïí> êÝöé [“like one's company”]), ÊÁÍ ðáé÷íßäé [“take the initiative”], ÊÁÍ êïõµÜíôï [“be in control” / “be the boss”]. 
3.2.2. Category B: ÊÁÍ + noun = semi-fixed expression with figurative meaning (ÊÁÍ (+ adj.) âÞµá) 
Category B includes set phrases that have figurative meaning, since they do not originate directly from the literal content of the words in question, which is similar to the previous case. The difference, though, lies in that in the second Category the (idiomatic) expressions are semi-fixed, i.e. allow adjectives, pronouns, articles, etc. to intervene and modify the noun, e.g. ÊÁÍ Ýíá ............ ß.µá [“take a decisive step”], ÊÁÍ åíôõðùóéáêÞ ...... [“take an impressive change in direction”]. Moreover, it should be clarified that some of the adjacent groups of this kind can be used both in the singular and plural. Here are some examples: ÊÁÍ µéá ............. ...... [“do a masterstroke”] and ÊÁÍ ôéò ........... ........ [“act as is necessary”], ÊÁÍ ðñïóåêôéêü .....µá <ðñïò .......> [“try carefully to approach sb. / sth.”] and ÊÁÍ êÜðïéá .....µáôá [“try to approach sb. / sth. somehow”]. Similarly, ÊÁÍ ðáñáôÞñçóç and ÊÁÍ ðáñáôçñÞóåéò [“reprimand sb. for doing / saying sth.”] appear in both numbers. However, the findings of the corpus analysis underpinned the fact that there are also some set phrases, which are commonly applicable to either number, e.g. ÊÁÍ Ýñùôá [“make love”] (but not * ÊÁÍ Ýñùôåò), ÊÁÍ èõóßåò [“make sacrifices”] (occurring only in plural in my corpus, although it has a (less frequent) singular, as well). 
In this Category, account is taken of standardised Greek expressions in the singular only, such as ÊÁÍ ÷ñÞóç [only in the sense of “take drugs” / “drink alcohol” etc.], ÊÁÍ ôï .... (ôïý ...µïõ) [for “disseminating information around the world”] and ÊÁÍ áðåñãßá ...... [“go on a hunger strike”], as well. 
3.2.3. Category C: ÊÁÍ + noun = main delexical structure with literal meaning (ÊÁÍ äÞëùóç) 
While figurative meaning was reflected in the previous two Categories, where the verb ÊÁÍ had an idiomatic and not a delexical function, Category C provides instances of what I have called main delexical structures with literal meaning. These are: main, in contrast with the subordinate ones (as explained below), because they can easily be substituted for a simple verb which shares the same meaning and stem with the collocate (noun, in most cases); delexical structures, as their most meaningful item is the noun; and literal, since they are meant in the noun's original sense. 
8 Altenberg and Granger (2001: 173-175) place the English verbs ‘make’ and ‘do’ among high frequency verbs, which are interesting from a cross-linguistic perspective. 
148
It is not surprising that the most frequent main delexical structure found in articles from the Greek press is by far the phrase ÊÁÍ äÞëùóç [“make a statement”]. A similarly high rate of the cognate verb ÇËÍ was also anticipated and eventually found (see Appendix III). These significant occurrences can be explained by the actual fact that eminent persons of public life, such as Prime Ministers, Ministers, VIPs, etc., make official statements, which the reporters note down. This also alludes to the question of why half of the most popular structures in this Category are closely related to speech acts (cf. ÊÁÍ áíáöïñÜ [“make reference to sb. / sth.”], ÊÁÍ ðñüôáóç [“make a suggestion”], ÊÁÍ ðáñݵâáóç [“intervene verbally”], ÊÁÍ áíáêïßíùóç [“make an announcement”]. 
It could be further argued here that the combination of verb and noun makes the foregrounding of information much easier in a language such as Greek, where the order of the sentence constituents could be characterised as either loose or rather variable in terms of focalisation. Following this argument, we can shed light on cases, such as ðñïóðÜèåéá ..... [“he made an attempt”], Ýëåã÷ï ...... [“they checked”] and Ýñåõíá ...... [“they searched”], where emphasis is placed on the noun (for the syntactic distribution of the verb ÊÁÍ, see also section 3.3. below). 
A special emphasis is also placed on collocates that not only precede the node, but are modified as well, e.g. áñ÷çãéêÞ / åðéèåôéêÞ .µöÜíéóç ..... <ï X.> [“X. appeared as leader / having an aggressive attitude”]. Lastly, as regards the delexical ÊÁÍ ÷ñÞóç [“make use of sth.”] (in its literal sense, instead of “use”), the concordance of the corpus showed that this is most commonly combined with another noun in the genitive, nothing intervening between ÊÁÍ and ÷ñÞóç in most cases. 
3.2.4. Category D: ÊÁÍ + noun = subordinate delexical structure with literal meaning (ÊÁÍ ëüãï) 
The fundamental difference between this and the previously discussed group of expressions lies in that the structures which are brought together in Category D are still delexical (so as to complement Babiniotis’ definition (2002: 246), see 3.1.), but subordinate, in this case. I shall call subordinate those patterns that cannot be substituted for a simple verb, purely because no such verb derives from the noun's stem in Greek, for instance ÊÁÍ êáêü [“do harm”] and its opposite, ÊÁÍ êáëü [“do good to sb.”], ÊÁÍ ðáñÝá [“keep sb. company”], ÊÁÍ öáóáñßá [“make noise”]. 
In some similar cases, even though the cognate simple verb may exist, it can derive from an older tradition of Greek, such as Ancient Greek or even êáèáñåýïõóá (katharevousa), and therefore may nowadays have neither the same meaning nor the same use. For example, ÊÁÍ ëÜèïò = óöÜëëù [“make a mistake”] is rather distinct from ëáíèÜíù [“be concealed”], even though they share the same stem; ÊÁÍ äéÜëïãï [“converse with sb.”] is currently much more preferable than the ‘antiquated’ äéáëÝãïµáé having the same meaning; and ÊÁÍ Ýêêëçóç [“make a plea”] cannot be replaced by åêêáëþ [“make an appeal”], since the latter is restricted to juridical terminology. 
The following three collocations are subordinate delexical structures, as well, for their basic meaning is close to the literal one proposed by the noun: ÊÁÍ ëüãï [“refer to sth.”] (cf. ëÝãù [“say”]), ÊÁÍ µíåßá [“make reference to sb. / sth.”] (cf. µíçµïíåýù [“mention”]), ÊÁÍ ôçí ....... [“make a strong impression”] (cf. åêðëÞôôù [“surprise”]). The first structure is considerably the most common of this Category (cf. also ÊÁÍ äÞëùóç in Category C), whereas the collocate of the second is usually modified by the adjective éäéáßôåñç [“special”] (ÊÁÍ éäéáßôåñç µíåßá). In the third phrase the article ôçí is essential and adds sense to the meaning of åíôõðùóéÜæù; it can be thus dissociated from the main delexical structure ÊÁÍ Ýêðëçîç = åêðëÞôôù [“surprise”]. 
3.2.5. Category E: ÊÁÍ + noun = lexical structure – both items carry a literal lexical load (ÊÁÍ Ýíá .....) 
The last Category of this cline of idiomaticity is composed of combinations of ÊÁÍ with any noun, so that both the verb and its complement carry a literal lexical load. In fact, these are neither idiomatic nor delexical, but still constitute an extreme of the cline, since they “retain the core meaning” of the verb (Biber et al. 1999: 1027). In this way, the core meaning of ÊÁÍ is expanded to multiple semantic fields, thus revealing the polysemy of this word (e.g. ÊÁÍ bearing the meaning of “arrange an event” (ÊÁÍ Ýíá .....), “broadcast” (ÊÁÍ µéá ....µðÞ), “construct”, “produce”, “create”, “accomplish”, “commit”, etc.). However, due to the fact that these lexical structures are too many to be identified even in such a small corpus and, additionally, require a lot of space and time for their analysis, they are not discussed here. 
149
To sum up, the following diagram illustrates the cline of idiomaticity, as described in section 3.2: 
Category A: 
ÊÁÍ + noun = fixed expression with figurative meaning 
(ÊÁÍ öôåñÜ) Category B: 
ÊÁÍ + noun = semi-fixed expression with figurative meaning 
(ÊÁÍ (+ adj.) âÞµá) Category C: 
ÊÁÍ + noun = main delexical structure with literal meaning 
(ÊÁÍ äÞëùóç) Category D: 
ÊÁÍ + noun = subordinate delexical structure with literal meaning (ÊÁÍ ëüãï) Category E: 
ÊÁÍ + noun = lexical structure 
(both items carry a literal lexical load) 
Diagram 3.2.: The cline of idiomaticity of the verb ÊÁÍ 
3.3. Syntactic distribution of the node ÊÁÍ and its (noun) collocates 
It has already been mentioned (in 3.2.3.) that the (noun) collocates can either precede or follow the node (verb). To demonstrate this, the table below charts the frequency of occurrence of nouns before and after the verb, as well as before the subordinate clause (there were no findings in relation to nouns following the subordinate clause) in numbers. (NB: freq. indicates absolute frequency, i.e. frequency of occurrences out of 2,139 instances): 
Table 3.3.: The frequency of syntactic distribution of the node ÊÁÍ and its (noun) collocates (Categories A-D) 
Category noun following verb (same clause) noun preceding verb (same clause) noun: main clause verb: subordinate clause Total hits freq. (%) hits freq. (%) hits freq. (%) A. 44 2.06 3 0.14 2 0.09 49 B. 173 8.09 11 0.51 17 0.79 201 C. 928 43.38 199 9.30 243 11.36 1,370 D. 491 22.95 17 0.79 11 0.51 519 Total 1,636 230 273 2,139 
3.3.1. Distribution within the same clause 
According to the above results, the noun is most likely to follow the verb, when they are both found in the same clause (main or subordinate). This is not surprising, since it is commonplace for the complement (object) to come after the predicate in Modern Greek, in a ‘natural’ order of sentence constituents (Subject-Verb-Object / Verb-Subject-Object, see also Clairis and Babiniotis 1999: 298 ff.). 
3.3.2. Distribution within different clauses: noun first 
More interesting, however, is the case of collocate preceding the node, since this offers flexibility to the syntax of the sentence and causes the foregrounding of information. By placing the noun – i.e. the item that carries the lexical load chiefly in Categories C and D – first, emphasis is given to the complement, while the reader's attention is drawn to the ‘unexpected’ order of the constituents. As Table 2 indicates, in approximately one third of the cases the noun comes before the verb, which is highly significant for the focalisation of the main information provided. As a final point, I should clarify that the subordinate clauses were almost exclusively relative clauses that facilitated the structure ‘Object-Verb-Subject,’ thus highlighting simultaneously the first and the last position of the constituents’ occurrence, again because of the disturbed ‘natural’ order, as described earlier. 
150
4. Implementing the cline of idiomaticity in relation to dictionary-making 
Having analysed the theoretical framework of the cline of idiomaticity, it would be of supreme importance to show a way of utilising it during the compilation of a dictionary. The contribution of each separate Category could be summarised as follows: 
  Category A provides all fixed expressions that have figurative meaning and usually constitute an obstacle for the foreign language learner. Furthermore, phrases of this Category are commonly used by native speakers. 
  Category B provides the semi-fixed expressions with figurative meaning, which are quite helpful for the understanding of metaphor in language. In addition, the Category in question designates frequent collocations of nouns being modified by adjectives and other word-classes. 
  Category C offers the analysis of a simple verb into its main delexical structure, which has literal meaning and can be used alternatively, in accordance with the speaker's / writer's intentions. The delexical structure has the advantage of the noun being both broadly modified and extensively preferred in a focussed (i.e. preceding the verb) position. 
  Category D supports the subordinate delexical structure that cannot be substituted for a verb deriving from the noun's stem (the substitution is possible in the previous case). Nevertheless, structures of this Category are potentially replaced by a synonymous, simple verb with literal meaning, and synonyms belong to the lexicographer's field of research. 
  Category E is equally essential for the dictionary's needs, since it reveals a thesaurus of multiple lexical meanings of a word. Moreover, polysemy can be examined to a significant extent through real examples extracted from a corpus. 
5. Synopsis of the results and conclusion 
For the needs of the present study, I used a sub-corpus of the more extensive HNC developed by the ILSP in Greece. This sub-corpus consisted of articles from two popular Greek newspapers (medium) and was representative of informative texts (genre) with social content (topic). My main concern was to look into the usage of the common verb ÊÁÍ [“make” / “do”] and for this reason I adopted and put forward a theory on a cline of idiomaticity. 
With the help of the WordSmith software and by adding tags to the nodes and collocates that I would later use, I worked out the concordance of the corpus and focussed on the colligation of ÊÁÍ + noun. I then divided the latter into five main Categories, according to the semantic load that the verb carried within the phrase. Thus, I suggested that there is a cline for ÊÁÍ + noun, which ranges from fixed (idiomatic) expressions with figurative meaning (Category A) to lexical structures, where both items carry a literal meaning (Category E). In the middle, there exists what I called ‘semi-fixed expressions with figurative meaning’ (Category B), as well as the two Categories of (main and subordinate) delexical structures (Categories C and D, respectively). 
Having looked through the grammatical, lexical and semantic structure, I brought up the issue of the syntactic distribution of the phrases in question. My results attempted to make clear that the collocate commonly follows the node in these cases, while the noun often precedes the verb, when there is a relative clause following. 
Finally, I tried to highlight the usefulness of the cline of idiomaticity for lexicographic purposes. Given that most expressions and set phrases constitute a problematic field for learners, they should be adequately and appropriately described in the dictionary. Moreover, the delexical structures offer a wide range of possibilities for noun modification, foregrounding or focus, whereas a variety of collocations can be explained through the polysemy of the verb. However, these are matters that remain to be further analysed, since they go beyond the scope of the present research. 
151
References 
Allan Q 1998 Delexical Verbs and Degrees of Desemanticization. Word 49(1): 1-17. 
Altenberg B, Granger S 2001 The Grammatical and Lexical Patterning of MAKE in Native and Non-native Student Writing. Applied Linguistics 22(2): 173-194. 
Babiniotis G 2002 Ëåîéêü ... .... ......... ....... (µå ...... ... .. ..... ..... ... ......) (äåýôåñç ......). [Dictionary of Modern Greek Language (with Comments on the Correct Usage of Words) (second edition)]. Athens, Lexicology Centre (in Greek). 
Biber D, Johansson S, Leech G, Conrad S, Finegan E 1999 Longman Grammar of Spoken and Written English. London, Longman. 
Clairis Chr, Babiniotis G 1999 ÃñáµµáôéêÞ ... .... ......... (äïµïëåéôïõñãéêÞ – åðéêïéíùíéáêÞ) ÉÉ. Ôï ..µá: Ç ........ ... µçíýµáôïò. [Modern Greek grammar (structural-functional – communicative) II. The verb: The organization of the message]. Athens, Greek Letters (in Greek). 
Hatzigeorgiu N, Gavrilidou M, Piperidis S, Carayannis G, Papakostopoulou A, Spiliotopoulou A, Vacalopoulou A, Labropoulou P, Mantzari E, Papageorgiou H, Demiros I 2000 Design and Implementation of the Online ILSP Greek Corpus. LREC 2000 Proceedings 3: 1737-1742. 
Jespersen O 1942 A Modern English Grammar on Historical Principles, part 6: Morphology. Copenhagen, Ejnar Munksgaard. 
Leech G, Smith N 1999 The Use of Tagging. In van Halteren H (ed), Syntactic Wordclass Tagging, vol. 9, Text, Speech and Language Technology. Dordrecht, Kluwer Academic Publishers, pp 23-36. 
Live A H 1973 The Take-Have Phrasal in English. Linguistics: An International Review 95: 31-50. 
McEnery T, Wilson A 2001 Corpus Linguistics (second edition). Edinburgh, Edinburgh University Press. 
Nakas Th 2000 ÃëùóóïöéëïëïãéêÜ B´: ÌåëåôÞµáôá ... .. ...... ... .. .......... (Ýêôç ......). [Studies on language and literarure, B´ (sixth edition)]. Athens, Parousia (in Greek). 
Nickel G 1968 Complex Verbal Structures in English. International Review of Applied Linguistics in Language Teaching (IRAL) 6: 1-21. 
Quirk R, Greenbaum S, Leech G, Svartvik J 1985 A Comprehensive Grammar of the English Language. London, Longman. 
Sinclair J 1991 Corpus, Concordance, Collocation. Oxford, Oxford University Press. 
Sinclair J M 1997 Corpus Evidence in Language Description. In Wichmann A, Fligelstone S, McEnery T, Knowles G (eds) Teaching and Language Corpora. London, Longman, pp 27-39. 
Sinclair J M et al. (eds) 1998 Collins COBUILD English Grammar. London, Collins. 
Stein G 1991 The Phrasal Verb Type ‘to Have a Look’ in Modern English. International Review of Applied Linguistics in Language Teaching (IRAL) 29: 1-29. 
152
Appendices 
5.1. Appendix I: A sample of the results from the HNC 
5.2. Appendix II: A sample from the WordSmith Tools concordance 
153
5.3. Appendix III: The frequency of main delexical verbs (Category C) and their cognate simple ones 
Delexical verb Hits Hits Lexical verb ÊÁÍ äÞëùóç [“make a statement”] 243 5,000+ ÇËÍ ÊÁÍ áíáöïñÜ [“make reference <to sb. / sth.>”] 78 3,428 ÁÍÁÖÅÑ ÊÁÍ ðñüôáóç [“make a suggestion”] 72 935 ÐÑÏÔÅÉÍ ÊÁÍ ÷ñÞóç [“make use <of sth.>”] 54 972 ×ÑÇÓÉÌÏÐÏÉ ÊÁÍ ðñïóðÜèåéá [“make an attempt”] 50 1,006 ÐÑÏÓÐÁÈ ÊÁÍ (ôçí) åµöÜíéóç (µïõ) [“appear”] 34 897 ÅÌÖÁÍÉÆ ÊÁÍ Ýëåã÷ï [“control” / “check”] 33 451 ÅËÅÃ× ÊÁÍ ðáñݵâáóç [“interfere”] 28 153 ÐÁÑÅÌÂÁÉÍ ÊÁÍ Ýñåõíá [“search” / “carry out research”] 24 181 ÅÑÅÕÍ ÊÁÍ áíáêïßíùóç [“make an announcement”] 19 1,655 ÁÍÁÊÏÉÍÍ 
154

Using natural language processing tools to assist semiotic analysis of information systems 
Ken Cosh and Pete Sawyer 
Computing Department 
Lancaster University 
UK LA1 3EU 
[k.cosh, sawyer]@comp.lancs.ac.uk 
1. Abstract 
Semiotic Analysis has been used to aid understanding of information or communication systems, providing information that can be used during requirements engineering. The MEASUR approach begins by analysing short, natural language problem statements and manually extracting the key themes involved. As the process is scaled up and applied to longer problem statements, as found in many real life circumstances, the manual effort required increases. When the starting point for Semiotic Analysis is a large document describing the information system, such as an ethnographic report, assistance in the analytical process is necessary. This paper investigates how statistical Natural Language Processing Tools can aid this analysis. 
Natural Language Processing Tools can assist the analyst by directing them to the central themes in the document. Comparing a frequency list of the document with a frequency list from a large corpus of text such as the British National Corpus reveals the key words in the document. Collocation analysis of these keywords enables the creation of a lexical network and then closer investigation of the collocates in context allows the analyst to add semantic information to the model. 
2. Keywords 
Natural Language Processing, Semiotic Analysis, MEASUR, Requirements Engineering, Organisational Semiotics 
3. Introduction to semiotic analysis 
Semiotics is the study of ‘signs', and how they are used to communicate information between people. Organisational Semiotics is merely the study of how these signs are used within organisations(Stamper 2000). As a sign can be anything that conveys information, understanding the properties and meanings of these signs can be a useful aid to understanding the workings of the organisation. One application of semiotics is the semiotic analysis of information systems. This can be used to aid requirement engineering. 
Requirement Engineering is used to analyse a problem domain, prior to designing a solution to the problem. It is a crucial part of any software engineering process, as it is vital to have a thorough understanding of the problem before attempting to solve it. Ambiguity and misunderstandings between the user and the developer are a big cause of costly rework. Analysing a problem using semiotics can reduce ambiguity by creating a common understanding of the semantics involved in a problem domain. 
This paper reviews the MEASUR approach to Semantic Analysis, developed by Stamper (Stamper 1994), and looks at the similarities between steps in MEASUR and some statistical Natural Language Processing techniques. It looks at some problems that currently exist with this Semantic Analysis approach and discusses how some statistical Natural Language Processing tools can be used as a solution to these problems. 
Several authors have pointed out the apparent links between Organisational Semiotics and Natural Language Processing (NLP). (Charrel 2002)(Connolly 2000). Whenever Semiotic Analysis is used to model any problem or domain, then natural language, whether spoken or documented, is studied. The purpose of this research is to demonstrate how probabilistic NLP techniques can be applied to Semantic Analysis, as described in the next section. 
4. The MEASUR approach to semantic analysis 
There are 4 major phases to Semantic Analysis as proposed by MEASUR (Lui 2000): 
155
Problem Definition 
| 
Candidate Affordance Generation 
| 
Candidate Grouping 
| 
Ontology Charting 
Figure 1, Stages in semantic analysis 
The first step, as shown, is problem definition. MEASUR begins by formulating a concise, well-articulated problem statement, which includes all the relevant parts of a problem. 
Working from this problem description, semantic analysis takes over, with the goal of creating a model of the problem. This begins with candidate affordance generation, where all the agents, objects, actions, etc. are identified from the text. Once these candidates have been identified, Candidate Grouping takes over as the first step towards creating an ontology chart. The two key types of entity that need to be identified during candidate affordance generation, to be used as construction blocks for the Ontology Chart, are agents and affordances. An agent is a type of object, a performer or processor, the initiator of an event. An agent can be a human, a device or a program, whatever has responsibility for the action. The behaviour of an agent is directed by its knowledge of, and constrained by the nature of, the environment. 
For instance an agent could swim assuming it has the knowledge of how to swim, and the environment affords it an area of liquid to swim in. A swimming pool could therefore be seen as an affordance within the environment enabling the agent to swim. Affordances can be seen as properties of a situation, not necessarily objects such as swimming pools. Other examples of affordances are budgets, projects, time and even agents, as sometimes agents can become affordances of another agent, depending on context. (Gibson 1979)(Stamper 1996) 
Agents are relatively easy to identify since they typically appear as nouns in the problem statement. Nouns may also represent affordances or roles (roles are explained in the conference organisation example below). Unlike agents (and roles), affordances do not map neatly onto a single part of speech. They can be nouns, but equally could be verbs or several other parts of speech. Hence, while a list of candidate agents may be generated mechanistically, the identification of affordances requires analysis of the semantics by considering the relationships between the various syntactic elements of the problem statement. Normally elements will depend on other elements for their existence. For instance, the affordance swim depends upon the existence of a swimming pool, in which to swim, and also a swimmer (the role of a person agent), to actually swim. This is depicted in the fragment of an ontology chart in figure 2. Here, agents are shown as ellipses, and affordances are depicted in rectangles. 
Figure 2. A fragment of an ontology chart 
The ontology chart is one of the goals of this process; it is a graphical representation of the relationships between these agents and affordances. The process to create an ontology chart is better demonstrated in the example below. The eventual chart is comprised from grouping the affordances together. Once the ontology chart has been created, the next step is to assemble a set of norms, which govern the standard behaviour of the model. Norms describe how the model is expected to work. They describe the normal behaviour of agents within the problem domain and their use of affordances. When the norms are attached to an ontology chart a thorough understanding of the problem is completed. This paper however concentrates upon the steps involved in creating the ontology chart. 

swimmer 

Swimmingpool 

Person 

Swim 
156
5. Conference organisation example 
The following minimal problem statement permits the identification of the principal entities of the problem domain listed in table 1.: 
A member of an organisation can organise a conference, which they invite people to attend. Participants can submit papers to be reviewed by the conference organiser. 
Member Role of person in organization Organisation Agent Organise Affordance of conference and conference organizer Conference Affordance of organization Invite Affordance of conference organiser and person People Agent Attend Affordance of participant Participant Role of person who accepts invitation Submit Affordance of Participant and Paper Papers Affordance Review Affordance of conference organiser and contributed paper Conference Organiser Role of member who organises conference 
Table 1. The principal entities in a conference organisation problem 
The above list of entities represents the set of candidate nodes in the ontology chart. Once the candidates have been identified, the next task as also illustrated in table 1, is to categorise these as agents, roles and affordances and assemble the ontology chart to explicate the relationships between them. In this example, Organisation is an agent (agents aren't necessarily human) while Member is a role of a person within the organisation. In order to make the relationship understandable, the affordance Membership has to be added. Membership depends upon an Organisation, which can allow Membership, and a Person, to take advantage of Membership, whereupon they become a Member (Figure 3). 
Figure 3. Modelling the membership of an organisation 
In the model in figure 3, organisation and person are agents, while membership is an affordance. Member is a role and so is placed between person and membership. The important entities of the problem domain can be grouped to begin to create an ontology chart. Once candidates have been grouped a structure for the chart appears. 
Member has been identified as a role, not just an agent, as the problem statement identifies People, Participants and Members. An experienced analyst will recognise that Participant and Member are (in O-O terms) specialisations of Person. In an O-O notation such as UML class diagrams, the analyst would have the option of modelling these using either sub-classing or, if instances of a single class could play multiple roles, using roles at the termination points of association relationships. In an ontology chart, roles are the modelling mechanism used. 
Roles only exist in circumstances where a dependent affordance is taken advantage of, so they are depicted along the arc between the antecedent and the dependent. For instance, in figure 4 which shows the complete ontology chart for the conference organisation problem, a person only becomes a participant when they accept the invitation. Roles in an ontology chart are not merely labels, they can form nodes. This is illustrated in the conference organiser role which is itself derived as a special type of the role member when organising the conference. Note that roles needn't be restricted to the agents in the model. For 

membership 

organisation 

person 

member 
157
example, contribution is a specific type of the affordance paper which has been submitted by the role participant. 
In the same way in which membership was added to the ontology chart to explain the relationship between a person and an organisation, other terms can be added. Authorship and accept are added to the ontology chart despite not being in the original problem statement. Authorship explains the relationship between a person and a paper. Accept has to be added to enable the existence of a participant as a person only becomes a participant when they have accepted the invitation to the conference. 
Figure 4. Complete ontology chart of conference organiser example 
An ontology chart should be read from left to right, as objects to the right are dependent on the items they are connected to by arcs on their left (Barjis & Chong 2002). For example, the affordance invite can only exist when there is a conference organiser (to perform the inviting) and a person (to be invited). Similarly the affordance accept relies on a person (to perform the accepting) and the existence of the invitation (to be accepted). 
The construction of an ontology chart, even from a concise problem description, isn't mechanistic. It requires considerable experience and skill in the application of MEASUR on the part of the analyst and this acts as an inhibitor to the application of MEASUR in systems development. 
6. Statistical natural language processing and organisational semiotics. 
In all the documented case-studies and examples used in the organisational semiotics literature, the problem scope is small, and the problem statement is a concise description with no redundant description or ambiguity. For many real life cases it isn't possible to neatly and perfectly summarise the problem in a brief problem statement, so the starting point for semantic analysis may not be a carefully worded problem statement. Instead it could be a set of long descriptive reports. Examples could include ethnographic study reports, legal documents, codes of practice, meeting transcripts, and any other documentation which could add knowledge to the problem definition. Many problems are just too complex and subtle to be neatly encapsulated by a short problem statement. To be practical it should be possible to apply MEASUR to problems where the sources of information are less clearly bounded, diffuse, scattered and poorly structured. 
The problem with this is that to read through and analyse long documents, using the approach discussed above, becomes increasingly difficult, particularly if we intend to “generate candidate affordances”, by selecting every noun, noun-phrase, verb etc. While in a short precisely written problem statement the analyst will only find the necessary semantic units in order to create the ontology chart, in a longer larger document, the analyst might find many nouns which only occur once throughout the document and don't actually add anything to the model. This irrelevant information needs to be filtered out. 

contribution 

participant 

Conference 
organiser 

member 

person 

organisation 

conference 

membership 

organise 

review 

invite 

accept 

authorship 

submit 

paper 
158
Many of the terms will be used several times, in different contexts, so they become easier to define accurately. Having selected the key candidates using the frequency tests, we can then look at these words in context, singling out the word and the passage in which the word occurs. Looking at the key word in context (KWIC), as well as aiding definition of terms, can also be used for grouping the keywords together, as instances where keywords are used in conjunction with each other can be isolated. 
As large amounts of natural language has to be analysed, it is logical to use some NLP tools to assist with the automation of the approach. The following section discusses how some NLP techniques can be used to aid with Candidate Affordance Generation and Candidate Grouping. 
7. Applying natural language processing techniques 
For the purposes of this example, information collected in an ethnographic report into the Air Traffic Control (ATC) domain is used. This has previously been looked at in the REVERE project (Rayson, Garside & Sawyer 2000). The document is 66 pages long with over 40,000 words in it, so it clearly isn't the concise carefully written problem statement alluded to previously. 
Using this as the ‘problem statement', the next step is to generate a list of candidate affordances. Using the statistical NLP tools this can be done by firstly compiling a frequency list – counting the number of occurrences for each word. This frequency list can then be compared to a Corpus frequency list. A Corpus is a large body of text. The example used here is the British National Corpus (BNC), which is a giant frequency list consisting of many words from the English language, and how frequently they can be expected to occur. 
The first step in this process, is to create a frequency list for each word in our document. Initially the most frequently occurring words are likely to be words such as ‘the', ‘of', ‘and’ and ‘to'. These words aren't a particularly helpful indication of what the document is about, as they are the words which occur most commonly in written natural language. The most interesting and helpful words are those which occur most significantly more often than we would expect them to within the document, and we can detect these by comparing the document with the BNC. 
The first step in calculating the most significantly overused words is to calculate how frequently we would expect a word to occur, given the size of the document. This can be done by first creating the following contingency table; 
Text to be 
analysedBNC Total Frequency of word a b A + b Frequency of other wordsc-a d-b C + d – a - b Total c d C + d 
Figure 5. Contingency table. (Rayson & Garside 2000) 
With the information in this table, the expected frequency for any word in the text to be analysed can be calculated using the following formula; 
E1 = c * (a + b) / (c + d) 
And the expected frequency, given out text to be analysed, for any word in the BNC can be calculated; 
E2 = d * (a + b) / (c + d) 
Once the two expected frequencies have been calculated, we can calculate the significance in the difference between these two scores, using a Log Likelihood test. The following formula will give a significance score showing how significant it is that the word occurs as frequently as it does; 
Sig = 2 * ( ( a * ln ( a / E1) ) + ( b * ln ( b / E2 ) ) ) 
(Rayson & Garside 2000) 
The higher the result of this test, the more significant it is that the word has occurred more often than it should have. After this comparison using a log likelihood test between the BNC frequency list, and 
159
the ATC frequency list, a new frequency list is created with the most significantly overused words prominent. These words become the keywords of the problem domain. Using the example of the air traffic control ethnographic report, the words that are most significantly overused within the document are, as could be expected, words like, controller, radar, flight etc. (fig 6). Further information about this technique can be found in (Rayson, Garside & Sawyer 1999). 
Word Frequency Log Likelihood 1 controller 217 1386.84 2 strips 227 1372.39 3 radar 168 1060.39 4 strip 173 966.805 5 flight 113 576.605 6 controllers 73 458.849 7 chief 114 451.441 8 sector 82 353.931 9 pole 75 334.929 10 traffic 74 307.131 11 of 671 294.06 12 planes 56 293.969 13 pending 48 281.001 14 aircraft 58 279.465 15 hill 75 273.275 16 level 106 255.248 17 the 1848 254.816 18 airspace 41 246.31 19 ph 40 239.917 20 im 38 239.149 
Figure 6. Significantly overused words in the ATC example. 
This list still includes words like “the” and “of” which are significantly overused within the document, but don't add to the model. These words can be removed by refining the list so we only have words that we are interested in. 
Clearly with a large document to attempt to draw out the candidate affordances manually would be very time consuming. Using NLP tools, the first steps of the semiotic analysis method can be completed more speedily, and with less manual input. The third stage of the process is to group candidate affordances. Once a list of the terms that should occur within our ontology chart has been generated, they can be grouped. Once again there are NLP tools to assist during this stage. 
By looking at each keyword in more detail, it is possible to discover what it means and how it relates to the other keywords more precisely. The overused keyword ‘controller’ is chosen, it is a role, played by a person agent. 
Collocation analysis is a statistical test, which produces a Z score that tells us how likely it is for two words to have “co-occurred”. This works by first predicting the number of times that the second word should occur within a specified range, bearing in mind the frequency of the word within the entire document. Given the expected co-occurrence frequency and the actual co-occurrence frequency, the probability can be calculated. With this probability, the significance of the co-occurrence can be tested using Berry Rogghe's z-score calculation (Oakes 1998). 
To calculate the significance of a collocation, the following information is needed; 
Z – total words in text 
A – number of times keyword occurs in text 
B – number of times potential collocate occurs in text 
K – number of times the keyword and the collocate co-occur within span 
S – Span – number of words on either side of the keyword to be considered 
160
The first step is to calculate the number of times the collocate should co-occur near the keyword if the two words were randomly distributed and then compare this with the actual number of co-occurrences. To calculate the expected number of co-occurrences, we first need the probability of the collocate occurring where the keyword does not occur; 
P = B / ( Z – A) 
Then, the expected number of co-occurrences is given by; 
E = P * A * S 
The statistical test to see how significant the collocation is, is determined by calculating a z score, as follows; 
____________ 
z = ( K – E ) / ã (E * ( 1 – p ) ) 
Once again, the higher the z score, the more significant the collocation. 
Setting the span to 10 words either side of our keyword, ‘controller', the collocation significance of each other word in the document can be calculated. Once we have the z score, as given by Berry Rogghe's calculation, we can compare this to a percentage significance level. Here, words have been split into 1%, 5% and 10% significance,1% being the most significant. Words are significant at the 1% level with a z score greater than 2.33, at the 5% level when greater than 1.65 and at 10% level when greater than 1.3. The most significant co-occurring words (collocates) for ‘controller’ are; 
Word Frequency # Co-occurancesExpected Co-occurancesZ Score Roger 14 14 1.54 10.03 midland 17 14 1.87 8.86 Ph 40 19 4.41 6.96 Upper 4 5 0.44 6.87 131 2 3 0.22 5.92 Fault 2 3 0.22 5.92 Asks 2 3 0.22 5.92 Wit 1 2 0.11 5.69 32 1 2 0.11 5.69 looming 1 2 0.11 5.69 105 1 2 0.11 5.69 arrived 1 2 0.11 5.69 searching 1 2 0.11 5.69 charters 1 2 0.11 5.69 Fallen 1 2 0.11 5.69 Leans 4 4 0.44 5.36 10 17 9 1.87 5.21 Who 48 17 5.29 5.1 assistant 9 6 0.99 5.03 Pilot 35 13 3.85 4.66 
Figure 7. Significant collocates at 1% level 
As can be seen several of the keywords only occur once within the entire document, so it isn't that interesting that they co-occur with ‘controller'. As they only occur once within the document, they add little to the eventual model, so we can filter the list and investigate further the interesting co-occurrences. 
161
Word Frequency # Co-occurances Expected Co-occurances Z Score Roger 14 14 1.54 10.03 midland 17 14 1.87 8.86 Ph 40 19 4.41 6.96 Upper 4 5 0.44 6.87 Leans 4 4 0.44 5.36 Who 48 17 5.29 5.1 assistant 9 6 0.99 5.03 Pilot 35 13 3.85 4.66 Red 5 4 0.55 4.65 marks 3 3 0.33 4.64 express 3 3 0.33 4.64 Him 78 22 8.59 4.58 climbing 24 10 2.64 4.53 
Figure 8. Significant collocates of ‘Controller’ 
Given the list of significant collocates for each keyword a lexical network could be created automatically. This involves simply connecting every word to those which occur near them. However, as this network contains little semantic content it isn't very useful in understanding the problem domain. Further investigation into each collocate produces much more valuable insight into the nature of the problem. 
Each of these significant collocates can be looked at in context, firstly so the meaning of the second word can be understood, and secondly so the relationship between the two words can be analysed. In this example, “roger” is the most significant collocate, and further investigation of the pair in context reveals that the controller communicates to a pilot via radio, and regularly uses the radio term “roger”. From this information it is possible to begin to group possible candidate affordances, such as that radio is dependent upon pilot and controller for it's existence. 
Figure 9. Grouping of controller, radio and pilot. 
Looking at keywords in context with other keywords assists the analyst in understanding the semantics of them. By isolating the controller keyword it is possible to find sentences such as; 
It is the job of the radar or sector controller To coordinate this traffic through his/her sectors. 
Which amongst others provides the information that the controller is a role performed by a human agent. This information can be added to the growing ontology chart; 
Figure 10. Fragment of ontology chart demonstrating the role controller. 

radio 

control 

controller 

person 

pilot 

radio 

controller 

pilot 
162
It is also possible to learn further information about the controller role. Firstly from the above sentence alone, a function of the controller role is to co-ordinate traffic through his or her sector. There is clearly a relationship between the controller and ‘sector'. Using NLP to look at keywords in context with each other, it is possible to isolate every piece of text which contains both ‘controller’ and ‘sector'. This provides the information needed to discover that it is a sector which the controllers control, or in other words, the affordance control can only exist when there is a controller (to perform the controlling) and a sector (to be controlled). This information can be added to the ontology chart. 
Figure 11. Fragment of ontology chart demonstrating addition of sector. 
Gradually by grouping more information, as various parts of the text are investigated using the keywords in context, the ontology chart can be added to until it provides a thorough analysis of the Air Traffic Control domain. 
8. Conclusions 
Semiotic analysis of an information system can aid requirement engineers and other analysts to fully understand it. When conducting semiotic analysis of an information system a rich source of information could be an ethnographic report, or another natural language document describing the domain. Current approaches to semiotic analysis are designed for use with concise, carefully constructed problem statements. This paper has investigated how NLP can be used to scale up the approach so larger, more information rich documents can be analysed. 
The NLP tools that have been included in this research are statistical frequency tests and collocation analysis, aimed at guiding an analyst to the important areas of the document being analysed. Further work in this area could be useful to further aid the analyst in identifying agents and affordances automatically, either by refining Part of Speech tagging or semantic tagging. 
Using collocation analysis it is possible to automate the creation of a lexical network, connecting related words based on them occurring near one another in a document. Further human input is necessary to add semantic information to the lexical network. Whilst viewing the keyword in context alongside other keywords is an aid for the analyst, further tool support her could aid the process further. 
9. References 
Barjis & Chong 2002 Integrating Organisational Semiotic Approach with the Temporal Aspects of Petri Nets for Business Process Modeling, in J.Filipe, Sharp, B. and Miranda, P. (Eds.), Enterprise Information Systems III. Kluwer Academic Publishers, Dordrecht, The Netherlands 
Charrel 2002 Viewpoints for knowledge management in system design. In Proceedings 5th Annual International Workshop on Organisational Semiotics 
Connolly 2000 Accomodating Natural Language Within The Organisational Semiotic Framework, Third International Workshop in Organisational Semiotics WOS3 
Gibson 1979 The Ecological Approach to Visual Perception. Boston, Houghton Mifflin company. 
Liu 2000 Semiotics in Information Systems Engineering, Cambridge University Press 

sector 

control 

controller 

person 

pilot 

radio 
163
Oakes 1998 Statistics for Corpus Linguistics, Michael P. Oakes. Edinburgh textbooks in empirical linguistics. 
Rayson, Garside & Sawyer 1999 Language Engineering for the recovery of requirements from legacy documents. REVERE project report, Lancaster University, May 1999 
Rayson & Garside 2000 Comparing corpora using frequency profiling. In proceedings of the workshop on Comparing Corpora, held in conjunction with the 38th annual meeting of the Association for Computational Linguistics (ACL 2000). 1-8 October 2000, Hong Kong, pp. 1 - 6. 
Rayson, Garside & Sawyer 2000 Assisting requirements engineering with semantic document analysis. In Proceedings of Content-based multimedia information access RIAO 2000 (Recherche d'Informations Assistie par Ordinateur, Computer-Assisted Information Retrieval) International Conference, College de France, Paris, France, April 12-14, 2000. C.I.D., Paris, pp. 1363 - 1371. ISBN 2-905450-07-X 
Stamper 1994 Signs. Information, Norms and Systems In: B. Holmqvist, P. B. Andersen, H. Klein and R. Posner (Eds), Signs of Work: Semiosis and Information Processing in Organisations, Walter de Gruyter & Co, Berlin, pp.349-397. 
Stamper 1996 Ontological Dependency In Proceedings of the workshop on “Ontological Engineering” Aug 1996 
Stamper 2000 – Organisational Semiotics Informatics without the Computer? In Information, Organisation and Technology - Studies in Organisational Semiotics By Kecheng Liu, Rodney J. Clarke, Peter Bogh Andersen, Ronald K. Stamper (eds), Kluwer Academic Publishers, Boston, 2001. 
164

Language engineering tools for collaborative corpus annotation 
H. Cunningham*, V. Tablan*, K. Bontcheva*, M. Dimitrov** 
* Department of Computer Science, University of Sheffield, 
Regent Court, 211 Portobello Street, Sheffield S1 4DP 
{H.Cunningham, V.Tablan, K.Bontcheva}@dcs.shef.ac.uk 
**Ontotext Lab, Sirma AI Ltd. 
38 A Hristo Botev Blvd. Sofia 1000, Bulgaria 
marin.dimitrov@sirma.bg 
1. Introduction 
A vital step towards the creation of distributed corpora is the provision of tools for collaborative corpus annotation, in order to enable researchers to collaborate on annotating corpora regardless of their physical location. This problem can be decomposed in two major tasks: (i) provide users with access to distributed corpora; and (ii) provide visualisation and editing tools that require no installation effort and are easy to use. In this paper we will present the new collaborative corpus annotation facilities, recently developed as part of the GATE language engineering tools and infrastructure. These facilities have been used to build OLLIE – a client-server application that allows users to use the collaborative corpus annotation facilities in their own Web browser. 
This paper is structured as follows. An overview of GATE is provided in Section 2. Section 3 describes the support for distributed language resources implemented using relational database servers. The client-server architecture of OLLIE is introduced in Section 4 and details of its collaborative corpus annotation facilities are provided in Section 5. Section 7 concludes the paper and outlines future work. 
2. GATE overview 
GATE (Cunningham et al 2002) is a open-source language engineering infrastructure1 which, among other things, is being used for the creation and annotation of a number of corpora in many languages, e.g., the American National Corpus and EMILLE - a 63 million word corpus of Indic languages (Baker et al 02). It is written in Java and exploits component-based software development, object orientation and mobile code. One advantage of GATE from a corpus processing perspective is that it uses Unicode throughout (Tablan et al. 2002), and has been tested on a variety of Slavic, Germanic, Romance, and Indic languages (Gamback and Olsson 2000), (Pastra et al 2002). 
Another advantage is its support for a wide range of input and output document formats – XML, HTML, SGML, email, RTF, and plain text – and GATE can also be extended easily to handle other formats, in a manner transparent to the rest of the system. When a document is loaded in GATE, its format is analysed and converted into a GATE document, which consists of content and one or more layers of annotation. The annotation format, a modified form of the TIPSTER format (Grishman 1997), is largely isomorphic with the Atlas format (Bird and Liberman 1999) and successfully supports I/O to/from XCES and TEI (Ide et al. 2000). An annotation has a type, a pair of nodes pointing to positions inside the document content, and a set of attribute-values, encoding further linguistic information. Attributes are strings; values can be any Java object. An annotation layer is organised as a Directed Acyclic Graph on which the nodes are particular locations in the document content and the arcs are made out of annotations. All mark-up contained in the text used to create the document content is automatically extracted into a special annotation layer and can be used for processing or for exporting the document back to its original format. 
1 GATE is available for download from http://gate.ac.uk and is distributed freely as an open-source system under the GNU library license. 
165
In addition to corpus annotation facilities, GATE provides a set of language processing components, e.g., tokeniser, part-of-speech tagger, named entity recogniser; services for persistent storage of language resources; and extensive visualisation and evaluation tools aimed at facilitating the development and deployment of language processing applications. 
The new distributed corpus annotation facilities discussed here were built to resemble GATE's easy-to-use and extendable facilities for text annotation of locally stored documents. These facilities allow both manual and semi-automatic annotation of corpora, based on GATE's language processing components. The text is annotated in a point-and-click fashion, where the user first highlights the text to be annotated and then clicks on the desired annotation type (e.g., ENAMEX, POS). 
The rest of the paper will discuss how GATE has been extended to support distributed language resources and how these facilities were used to build OLLIE – a client-server application for collaborative corpus annotation. 
3. Distributed language resources 
Big corpora, such as the British National Corpus (BNC) are best stored on server machines with large amounts of disk space, so they are installed once and shared among many researchers. GATE offers support for such distributed language resources by storing them in relational databases, which run on the servers and accessed by the GATE visual environment that acts as a client. Currently Oracle and PostgreSQL are supported, of which the latter is freely available. 
Currently the database servers support storage of corpora and documents and their associated annotations. These GATE objects are stored in about 20 relational tables, grouped in the following categories2: 
1) Language Resource-related tables: contain information common for all language resources, e.g., their name and type. 
2) Corpus-related tables: contain information about the stored corpora, such as which documents belong to them. 
3) Document-related tables: contain information about the stored documents, e.g., their encoding, content, and which annotation sets belong to them. 
4) Annotation-related tables: contain information about the annotations, the sets they belong to, and the feature-value pairs associated with them. 
5) Security related tables: contain information about users and groups. 
The GATE users are completely isolated from the concrete ways in which LRs are stored in the relational tables. When choosing to store an LR in a distributed database, users just have to choose which of the available servers they want to use and provide their user name and password. Then GATE transparently converts all LRs in the corresponding database objects and stores them via JDBC. In this way, all GATE users and applications are completely isolated from the technicalities of using a database server for distributed LR storage - remote RDBMS datastores are accessed in the same way as local file system based datastores. 
An important aspect of supporting distributed LRs is security and access control. In GATE we have implemented role-based access control and every language resource is associated with security properties specifying the actions that certain users and groups may perform with this resource. When users create LRs into the database datastore, they specify access rights for users, the group of the user and other users. For example, LRs created by one user/group can be made read-only to others, so they can use the data, but not modify it. The access modes supported by GATE are detailed in Table 1. 
If needed, ownership can be transferred from one user to another. Users, groups and LR permissions are administered in a special administration tool, by a privileged user, i.e., the database administrator. 
As already mentioned, the RDBMS persistence is fully JDBC compliant but in order to improve the performance, specific features of the two databases were used when possible. While this approach induces certain development overhead, since small parts of the implementation are database specific 
2 For a detailed description of GATE's database storage mechanism and the relational tables used see http://gate.ac.uk/gate/doc/persistence.pdf and GATE's User's Guide (http://gate.ac.uk/sale/tao/). 
166
(such as properties of the database objects, stored procedures, SQL optimizer hints, etc.) it ensures that the optimal performance is achieved for each database. 
Mode Owner Owner's 
(Read/Write) group 
(Read/Write) Other users 
(Read/Write) World Read / Group Write +/+ +/+ +/- Group Read / Group Write +/+ +/+ -/- Group Read / Owner Write +/+ +/- -/- Owner Read / Owner Write +/+ -/- -/- 
Table 1 Database access rights. 
Another performance related feature of the database-based persistence is the load-on-demand implementation of certain GATE resources. Since it is inefficient to load huge amounts of data from a remote datastore, all language resources in GATE employ a load-on-demand behaviour when loaded from database datastore. In such cases the corpus documents, document content and annotation sets will be loaded from the datastore only when first accessed from the user (or the application). In a similar manner a complex tracking system ensures that only the parts of a resource that were changed are propagated back to the database for updates. This approach significantly reduces the amount of data transferred and the round trips to the server. 
Read-only resources allow shared use of existing corpora such as the British National Corpus (BNC), which only need to be saved in the database once and then become available through GATE to all users, who do not need to install and store them locally. In the future we will allow users to extend such resources by creating a new 'composite' LR, which references inside the original read-only LR. New data is added to the modifiable part of the composite LR, while users can access data from both parts. In this way, different groups can share a read-only copy of e.g. the BNC or WordNet, which is referenced by their separate customised LRs which store the new information, and made accessible to other groups/users as required. 
Key benefits of the approach include: 
1) The database infrastructure takes care of multi-user access, locking and so on without any intervention by the user. 
2) LRs can in this way be easily distributed across networks, and need not be installed locally on each machine used by their developers and users. 
3) GATE's graphical display tools can be used to visualise and edit the data in the resources. 
4) GATE's framework can be used for embedding the resources in diverse applications. 
The database infrastructure offered by GATE is employed in a number of projects and production systems. Some real world statistics from OntoText Lab for a project that uses GATE to annotate news articles look like: 
• Database size: 3.5 GB (growing by 20MB per day) 
• Number of documents: 170,000 (1000 new per day) 
• Number of annotations: 3,500,000 
• Number of annotation features: 10,500,000 
• Total number of rows in the database: 23,500,00 
167
4. The OLLIE architecture for collaborative annotation 
OLLIE is a client-server application providing support for collaborative corpus annotation by using any Java-enabled Web browser. The three main functions of the OLLIE client are: 
1) support user authentication and profiles; 
2) support collaborative corpus annotation; 
3) provide language processing tools to (partially) automate this task. 
OLLIE also supports training and running different Machine Learning methods but these aspects will not be discussed here due to space limitations. 
The OLLIE server on one hand communicates with the OLLIE client and on the other uses GATE and its distributed database facilities (described in Section 3) to store the user data and the language resources being edited (see Figure 0 for an illustration). Every user has a username and a password, used to retrieve their profiles and determine which documents they can access (according to the database access policies discussed above). The profiles contain information provided by the user, specifying the types of information that they are annotating in the corpora. For example, the profile for basic named entity annotation will specify Person, Organization, and Location. These tags will then be provided in the document annotation pages in the client (see Figure 0). 
The OLLIE server is implemented as a set of Java Server Pages (JSPs) along with some supporting Java classes. The entire application is then packaged for deployment into a servlet container such as 

Figure 0 The OLLIE corpus and document browser 
168
Apache Tomcat. All the configuration data (e.g. URLs to the databases holding the security data or the GATE distributed database) is handled through the configuration of the servlet container (which in the case of Tomcat consists of a set of XML files). The server's efficiency is improved by using connection pooling for efficient database access. 
The client-server communication during on-line document annotation is carried over Java Remote Method Invocation (RMI), which supports data sending and exchange of messages between the client and the server. The continuous communication comes with the added benefit of data security in case of network failure. The data on the server always reflects the latest version of the document, so no data loss can occur. 
5. The Web-based collaborative corpus annotation client 
OLLIE supports collaborative annotation of documents and corpora by allowing their shared, remote use and by making updates made by one client immediately available on the OLLIE server. In this way users can share the annotation task with other users. For example, one user can annotate a text with organisations, and then another annotates it with locations. The documents reside on the shared server which means that one user can see errors or questionable mark-up introduced by another user and initiate a discussion. 
The OLLIE client provides facilities for loading documents/corpora for annotation from a URL, uploaded from a file, or created from text pasted in a form. A variety of formats including XML, HTML, email and plain text are supported. As part of this process, the user also specifies the access rights to the document/corpus, which determine whether it can be shared for viewing and collaborative annotation (see Section 3). Figure 0 shows the OLLIE client window listing all corpora and documents accessible to the user and available for annotation through the Edit button. The Admin button is for specifying the access rights and there is also a button for deleting the LR from the server. 

Figure 0 The OLLIE client-server architecture 

Figure 0 Remote document editing in OLLIE 
169
The document editor is used to annotate the text (see Figure 0). The right-hand side shows the classes of annotations (as specified in the user profile) and the user selects the text to be annotated (e.g., ``McCarthy") and clicks on the desired class (e.g., Person). The new annotation is added to the document and the server is updated immediately. The client also provides facilities for deleting wrong annotations, which are then propagated to the server, in a similar way. 
The annotation facilities in the OLLIE client are designed to work exactly in the same way as those in GATE's visual environment, so that users familiar with one can use the other without a major overhead. The advantages of using the OLLIE client is that it does not require any installation and runs in any Java-enabled Web browser, which makes it ideal for collaborative corpus annotation, especially if users tend to change their locations and machines. On the other hand, the GATE visual environment provides more extensive facilities, e.g., specialised viewers/editors for complex structures such as syntax trees, while still supporting distributed LRs via the Oracle/PostgreSQL databases, but needs to be installed locally on the machine. In addition, due to its more generic nature, as a development environment for language processing applications, it offers many additional features which make its user interface too generic, whereas OLLIE's client is specifically designed for collaborative corpus annotation. 
The users do not need to manually annotate all data, instead they can use GATE's language processing tools to bootstrap the annotations by running them on the corpus. The actual processing occurs on the OLLIE server where the corpus resides and can also be scheduled to take place overnight. While some other infrastructures, e.g., AGTK (Ma et al 2002) offer corpus sharing in a manner similar to GATE, GATE has the advantage of offering the bootstrapping support discussed here. 
OLLIE supports bootstrapping with a wide range of linguistic information - part-of-speech, sentence boundaries, gazetteer lists, and named entity class. This information, together with tokenisation information (kind, orthography, and token length) is provided by the language processing components available within GATE, as part of the ANNIE system (Cunningham et al 2002). The user chooses which of this information is needed and the respective language processing components are run on the OLLIE server, the document is updated with new annotations, and they are propagated immediately to the OLLIE client and shown to the user. The user can then correct these annotations by deleting the wrong ones and adding new ones. However, such bootstrapping would help only if the language processing tools have acceptable performance. Therefore, if the automatic results for some annotation types (e.g., Organizations) have proven unreliable, the OLLIE client also offers the facility to delete all annotations of a given type by selecting this type in the right-hand side pane and pressing the Delete key. 
6. Web Services based collaboration 
Web services are a new breed of web applications that are based on the service-oriented architecture (SOA). Web Services are self-contained, modular applications that can be published, located and invoked across the Internet. They ease the inter-enterprise application integration, collaboration and process automation, reduce complexity associated with development and deployment and make it possible to easily build loosely-coupled systems that allow dynamic binding and just-in-time integration. 
At present Web Services functionality is being added to GATE. This will allow clients to use remotely the GATE functionality and use services (processing resources) that are otherwise not available locally. Making GATE Web Services aware will lead to: 
• increased collaboration between disparate research sites 
• easier integration of the HLT functionality in GATE with other applications 
• reduced cost and time of the development of GATE based applications 
• easy integration with light-weight and non-java components that require HLT functionality 
7. Conclusion 
In this paper we described a set of language engineering tools for collaborative corpus annotation, which allow users to share work on language resources distributed over the network and also to automate the annotation process by running relevant language processing tools. These tools form part of GATE, the General Architecture for Text Engineering. Recently we have also provided machine 
170
learning tools as part of GATE, in order to enable users to have trainable NLP tools they can use for corpus annotation. 
In future work we plan to enhance the GATE support for collaborative corpus annotation by providing access to the GATE corpora and tools via Web services. As shown by some recent work on using Web services and SOAP for distributed language resources (Dalli 2002), this model combines very well with efficient database storage mechanisms, like the ones used in GATE. 
8. Acknowledgements 
Work on GATE has been supported by the Engineering and Physical Sciences Research Council (EPSRC) under grants GR/K25267 and GR/M31699, and by several smaller grants. The first author is currently supported by the EPSRC-funded AKT project (http://www.aktors.org) grant GR/N15764/01 
. 
171

Figure 0 Logical model of the GATE relational schema. 
172
9. References 
Baker P, Hardie A, McEnery A, Cunningham H, Gaizauskas R. 2002. EMILLE, A 67-Million Word Corpus of Indic Languages: Data Collection, Mark-up and Harmonisation. In Proceedings of 3rd Language Resources and Evaluation Conference (LREC'2002), pages 819--825. 
Bird S, Liberman M 1999. A Formal Framework for Linguistic Annotation. Technical Report MS-CIS-99-01, Department of Computer and Information Science, University of Pennsylvania. http://xxx.lanl.gov/\-abs/cs.CL/9903003. 
Cunningham H, Maynard D, Bontcheva K, Tablan V 2002. GATE: A framework and graphical development environment for robust NLP tools and applications. In Proceedings of the 40th Anniversary Meeting of the Association for Computational Linguistics (ACL'02). Philadelphia, US. 
Dalli, A 2002. Creation and Evaluation of Extensible Language Resources for 
Maltese. In: Proceedings of 3rd Language Resources and Evaluation Conference 
(LREC'2002). Gran Canaria, Spain. 
Gamback B, Olsson F 2000. Experiences of Language Engineering Algorithm Reuse. In Proceedings of the Second International Conference on Language Resources and Evaluation (LREC), pages 155--160, Athens, Greece. 
Grishman R 1997. TIPSTER Architecture Design Document Version 2.3. Technical report, DARPA. http://www.itl.nist.gov/div894/894.02/related_projects/tipster/. 
Ide N, Bonhomme P, Romary L 2000. XCES: An XML-based Standard for Linguistic Corpora. In Proceedings of the Second International Language Resources and Evaluation Conference (LREC), pages 825--830, Athens, Greece. 
Ma X, Lee H, Bird S, Maeda K 2002. Models and tools for collaborative annotation. In Proceedings of 3rd Language Resources and Evaluation Conference (LREC'2002), Gran Canaria, Spain. 
Pastra K, Maynard D, Hamza O, Cunningham H, Wilks Y 2002. How feasible is the reuse of grammars for Named Entity Recognition? In Proceedings of 3rd Language Resources and Evaluation Conference (LREC'02), Gran Canaria, Spain. 
Tablan V, Ursu C, Bontcheva K, Cunningham H, Maynard D, Hamza O, McEnery A, Baker P, Leisher M 2002. A Unicode-based environment for creation and use of language resources. In Proceedings of 3rd Language Resources and Evaluation Conference (LREC'02), Gran Canaria, Spain. 
(Oracle) Oracle database documentation from http://technet.oracle.com 
(Postgres) Postgres database documentation from http://www.postgresql.com/docs 
173

Annotation without lexicons: an alternative to the standard bootstrapping approach 
Mark Davies 
Illinois State University 
1. Introduction 
A fundamental problem facing the creators of corpora for less-common languages – or the older stages of an established language – is the lack of suitable lexicons to annotate the corpus. Typically, the annotation of these languages involves “bootstrapping”, which refers to the process of starting with the most frequent forms (or some of the morphologically most predictable forms, or both) and progressively annotating the corpus as one also constructs the lexicon. As the annotation process continues, one can deal with less common or less regular forms, because of the predictive capacity of the increasingly robust lexicon and the increasingly rich, annotated context in the corpus itself . (For papers dealing with recent approaches to bootstrapping for a wide range of languages, see Rocio et al 1999, van Eynde et al 2000, Simov et al 2002, Cucerzan et al 2002, Ghani et al 2002, Moreno et al 2003). 
A standard approach to corpus annotation is to recursively traverse the entire textual corpus itself, searching for textual patterns, and then adding annotation to the corpus. This can be done either by large-scale pattern matching and replacement (using regular expressions or a similar schema), or by processing sequential chunks of text in a buffer (where only a small window of text is used to disambiguate part of speech (POS) and lemma). The important point is that the annotation typically takes place directly in the textual corpus itself. 
In this paper, we will outline an alternative schema that was used to annotate the Corpus del Espanol (www.corpusdelespanol.org) – a 100 million word corpus of Spanish texts from the 1200s-1900s. The annotation of this corpus was done without directly dealing with or seeing the actual textual corpus itself. Rather, the annotation was done on tables containing all of the distinct 1, 2, 3, and 4-word sequences (n-grams) in the entire corpus, along with the frequency of each of these n-grams in each historical period and modern register of Spanish. As we will see, this non-traditional approach affords a number of important advantages, both in terms of the flexibility and speed of the search engine for the corpus. 
For the purposes of this paper, we will focus primarily on the method of annotating the older stages of the language – the texts from the 1200s through the 1500-1600s. The lexicon that we created for Modern Spanish was able to annotate the texts from the 1800s-1900s quite well, but was progressively less useful for older stages of the language. For example, only 40% of the types from the 1700s appear in the Modern Spanish lexicon, and this decreases to 33% for the 1500s and 16% for the 1200s. In other words, most of the types from older historical periods were from a different “language”, as far as the lexicon was concerned. We will see, however, that by using an approach based on n-grams tables in relational databases, we were still able to annotate tens of thousands of distinct word forms from the oldest stages of the language in a matter of just a few hours. 
2. Corpus architecture and design 
Before discussing in detail the way in which the annotation is carried out with the aid of large relational databases of n-grams, let us briefly consider the overall organization of the Corpus del Espanol. The actual textual corpus for the 100 million word corpus is stored as 1000-2000 word chunks of text in a Microsoft SQL Server 7.0 database. This textual corpus itself is not annotated in any way, apart from a code that indicates the source of each block of text. However, it is indexed with SQL Server “Full-Text Indexing”, which is similar to the standard Microsoft Search engine. This indexing scheme allows exact words and phrases to be found fairly quickly – usually less than one second to query the entire corpus and return the relevant examples. The important limitation, however, is that the Full-Text search engine for SQL Server only works well with exact words and phrases. Even wildcard searches are problematic, and certainly there is no capability for customized annotation of any sort. 
The annotation for the Corpus del Espanol resides in relational databases, which are completely separate from the textual corpus itself. These databases are composed of different tables for all of the 1, 2, 3, and 4-grams in the corpus. These tables also include the frequency for each of these n-grams in each of the centuries from the 1200s-1900s, as well as the different registers of Modern Spanish. The data for these tables was generated from the textual corpus itself using the WordList function of WordSmith. This program was run separately for several three to four million word blocks of text, and then merged together in the SQL Server tables. 
174
As might be imagined, the tables are rather large, since they include all of the distinct 1, 2, 3, and 4-grams in the entire corpus. There are nearly one million distinct 1-grams (i.e types), eleven million distinct 2-grams, forty million distinct 3-grams, and 65 million distinct 4-grams. An example of one of the forty million 3-grams from the corpus is the following: 
Table 1. N-grams/frequency table 
w1 w2 w3 x12 x13 x14 x15 x16 x17 x18 x19 19-Lit 19-Oral 19-Misc son las cosas 38 16 77 67 16 19 33 68 24 40 14 
The columns w1, w2, w3 refer to each of the “slots” in the 3-gram; the columns x12-x19 refer to the frequency of this 3-gram in the 1200s-1900s; and 19-Lit, 19-Oral, and 19-Misc refer to the frequency in these three registers from the 1900s. Each of these relational database tables is indexed, including some clustered indices (to be discussed in more detail later on), all of which leads to very fast retrieval. 
3. Queries without annotation 
As this level, however, there is still no annotation per se – only the textual corpus and the n-grams tables. Even at this level, however, there are some useful queries that can be run against the database. For example, a user can input any of the following queries into the web-based form: 
Table 2. Queries without annotation 
QUERY SORT BY LIMITS tan * como 1900s *ización 1900s 1900s>5 –1800s quer* lo/la/los/las *r 1200s +1200s -1500s 
The first query will search the 3-grams table for all cases where the [w1] column is tan “as/so” and the [w3] column is como “as”, and order the results by the frequency in the [x19] column. This will return strings like tan bueno como “as good as”, tan rápido como “as fast as”, etc. The second query will search the 1-grams table for all of the records where the word in the [w1] column ends in [-ización], the value in the [x19] column is more than 5, and the value in the [x18] column is 0 (meaning that the word appears for the first time in the 1900s). This will return strings like privatización, globalización, and urbanización. The final example will return strings like queriendo lo fazer “wanting to do it” and queremos las dezir “we want to say them”, which represent cases of a form of querer “to want” followed by a direct object pronoun, followed by an infinitive. This query will search the 3-grams table for all records where the [w1] column is has the pattern [quer-], the w2 column is one of the following (lo, la, los, las), the word in the [w3] column ends in [-r], the [x12] column is greater than 0, and the x15 column is 0 (meaning that the phrase is found primarily in Old Spanish). 
The simple search syntax of the third query in the table above is transformed via web-based scripting into the following SQL statement, which is then run against the database, and which returns the results in less than one half of a second. 
select top 300 * from x3 where 
w1 like (‘quer%') and 
w2 in ('lo', 'la', 'los', 'las') and 
w3 like ('%r') and 
x12>0 and x15= 0 
Because Spanish has morphology that is both strong and fairly regular, users can employ simple lists of words and word patterns to search for even relatively complex syntactic constructions, as in Table 2 above. However, at some point it will obviously be necessary to have more complete annotation, including annotation for those lemma that are not morphologically regular (e.g. quis* for preterite forms of querer), as well as parts of speech that are not predictable in terms of forms (such as nouns and adjectives in Spanish). The major focus of this paper, then, is the way in which this can be done using collocational and frequency information from the n-grams tables themselves, and the challenge that this presents for languages (or stages of a language) for which we do not have a lexicon. 
4. Annotating the corpus: inheriting information from related lexicons 
Before turning to the basic question of how to enable “bootstrapping” via n-grams tables in relational databases, however, let us first consider a related and somewhat less difficult scenario. Imagine that there is a lexicon for the modern stage of a particular language, but that the need exists to annotate an older stage of the same language. Obviously, some of the forms from the modern language will be applicable to older stages, but this lexicon will become progressively less useful the farther back one 
175
goes. For example, as we have previously mentioned, 40% of the types from the 1700s in the Corpus del Espanol appear in the Modern Spanish lexicon, and this decreases to 33% for the 1500s and 16% for the 1200s. To the degree that there is similarity in types between the older and newer stages, however, perhaps the best strategy for annotating the corpus is simply to use the database to “inherit” features from related forms in the modern language. 
Let us briefly consider how this “inheritance” of annotation features has been carried out with the Corpus del Espanol. First, we used the frequency information in the n-grams tables to identify the highest frequency forms from older stages of the language, for which lemma and/or POS has not been applied from the Modern Spanish lexicon. For example, a simple SQL query like the following would produce a rank-ordered list of the 300 most common words from the 1500s (x15 column) in the x1 table (single words), where the word (w1) is the same as a word in the lemma table (x_l), which does not have a lemma assigned (column x1) (a similar query could be run for other centuries, as well as for the POS table [x_c]): 
select top 100 x1.x15,x1.w1 
from x1,x_l 
where x_l.x1 is null and x_l.w1 = x1.w1 
order by x1.x15 desc 
This list of the most common unannotated forms for a particular historical period can be INSERTed into another table called “unannotated”. The corpus creator would then manually go through this list and type into an adjacent column the modern forms that correspond to the older, unannotated forms, in those cases where there is such a correspondence. For example, in this query from the Corpus del Espanol, some of the unannotated forms that appear are començó, cavallos, and hazían, which correspond to the modern forms comenzó, caballos, and hacían “3SG-started, horses, 3PL-made”. Once the modern forms are entered into the “unannotated” table along with the modern form, we use a simple SQL UPDATE command to “copy” the POS and lemma values for the modern forms and apply these to the older forms. 
In addition to looking at the unannotated forms with the highest frequency, we can also search for forms according to regularized phonetic or morphological changes between the modern language and older stages of the language. For example, there was a regularized shift from [-zi-] to [-ci-] in Spanish, and the following query will find the 1000 most common, unannotated forms from the 1500s that have the pattern [-zi-], and which correspond to an annotated [-ci-] word in the modern lexicon, such as the older forms haziendo, juizio, and vezinos. 
select top 1000 x1.x15,x1.w1 
from x1,x_l 
where x_l.x1 is null and x_l.w1 = x1.w1 
and x1.w1 like '%zi%' and patindex(x1.w1,'%zi%') in 
(select patindex(w1,'%ci%') from x1 
where w1 like '%ci%') 
order by x1.x15 desc 
Once these older forms are INSERTed into an “unannotated” table, a simple REPLACE command can be used to place the modern form into another column, and a subsequent UPDATE query would copy the modern Spanish POS and lemma values to the older forms. In this way, with a knowledge of some of the basic phonetic, morphological, and orthographic changes in the language, it is possible to annotate thousands of forms from older stages of the language in a matter of a few hours. 
5. POS annotation with n-grams/frequency information and pattern matching 
In the previous section we assumed a more optimistic scenario, in which there is some type of related lexicon that can be applied to our corpus. Let us now turn to the more pessimistic scenario, in which there is no lexicon at all and we are simply “working from scratch”. The only assumption here is that we have the n-grams/frequency tables and the relational database structure that we have previously discussed, but there are no other annotation tools available to us. 
In order to assign POS, at the most basic level we will simply use SQL commands to select the most common word forms that have a certain morphological pattern. For example, the following SQL query selects those forms that end in [-ADO/-ADA/-ADOS/-ADAS], which is the typical marker of the past participle: 
176
select top 100 x12, w1 
from x1 
where w1 like '%do' or w1 like '%da' or w1 like '%dos' or w1 like '%das' 
order by x12 desc 
The one problem with such queries, however, is that they also retrieve many items that are morphologically similar, but which do not in fact belong to the desired grammatical category. For example, the preceding query would retrieve (-DO) quando, grado, mando; (-DA) espada, nada, cada, (-DOS) todos, dos; and (-DAS) espadas, todas – none of which are past participles. 
Fortunately, we can use SQL sub-queries to limit forms that have overly-generalized patterns (e.g. -DO, -DA, etc), by comparing them to other forms with which they share a predictable morphological relationship. For example, the past participle of [-AR] verbs is (nearly always) formed by removing the [-AR] of the infinitive, and replacing it with [-ADO]. We can therefore include in the SQL query a sub-query that checks to see whether the “root” of an –ADO form (i.e. remove the –ADO) can also be found as the root for an infinitive (i.e. remove the –AR): 
select top 100 x12, w1 
from x1 
where w1 like '%ado' and len(w1) >=3 and 
left(w1,len(w1)-3) in 
( 
select left(w1,len(w1)-2) 
from x1 where 
w1 like '%ar' and len(w1) >=2 
and x12 > 10 
) 
order by x12 desc 
Using this sub-query, we eliminate spurious [–ADO] forms like grado, obispado, and prelado, because there are no corresponding infinitives with the (hypothetical) form *grar, *obispar, and *prelar. 
In addition, we can use sub-queries to find derived forms of grammatical categories. For example, in searching for past participles ending in [–ADA] (fem sg), we can have the sub-query check to see whether the corresponding [–ADO] (m sg) form also exists (and at a certain frequency, such as 10 occurrences in the 1200s): 
select top 100 x12, w1 
from x1 
where w1 like '%ada' and 
left(w1,len(w1)-1) in 
( 
select left(w1,len(w1)-1) 
from x1 where 
w1 like '%ado' and 
x12 > 10 
) 
order by x12 desc 
This would eliminate spurious “[-DA] past participles” like cada “each” and espada “sword” because there are no corresponding [–DO] past participles like the hypothetical forms *cado and *espado. 
6. POS annotation with n-grams and collocation information 
In the preceding examples we have seen how POS annotation can be done by looking for morphological patterns (prefixes, suffixes, etc), and how this can be enhanced through the use of sub-queries, which test for the existence of related forms. In all of these cases, however, we have been dealing with queries that look at individual word forms. The real strength of the n-grams approach comes when we begin to limit queries based on the occurrence of given words within a certain n-gram sequence. 
For example, suppose that we want to identify the 1000 most common infinitives in the Spanish of the 1200s. We could do a simple search on the 1-gram table in which we search for words (w1) ending in [-R] (the marker of the infinitive), and produce a rank-ordered list. The following query would select these forms, but would also produce false entries like por, quier, mayor, muger, etc. 
177
select top 1000 x12, w1 
from x1 
where w1 like ‘%r’ 
order by x12 desc 
An alternative strategy – and one that relies on the inherent strengths of an n-grams approach – would be to search the 2-grams table for words that end in [-AR / ER / IR / YR], but which also occur after some of the most common forms of one of the three auxiliary verbs poder, querer, and deber. 
select top 1000 w2,sum(x12) 
from x2 where 
(w1 like 'quer%' or w1 like 'quis%' or w1 like 'quier%' or w1 like 'pod%' or w1 like 'pued%' or w1 like 'dev%' or w1 like 'deb%' or w1 like 'deu%') and 
w2 like '%r' 
group by w2 
having sum(x12) > 2 
order by sum(x12) desc 
This query is much more accurate than the 1-gram query shown previously (producing only one false hit in the top 200 word forms), and takes less than one second to run. 
The following are some additional searches that show how the n-grams tables can be used to annotate parts of speech. In each case, we want to identify a two or three word syntactic environment in which nearly all of the words in one of the “slots” belong to a particular syntactic category. For example, suppose that we want to identify the 2000 most common nouns in the 1200s. A syntactic environment in which these would occur is [indef art] + __ + [que] (un omne que “a man that”, vna casa que “a house that”, etc). The following query – which takes less than one second to run – selects the 2000 most frequent words in slot 2 (w2) in the 3-grams table (x3), which occur more than two times in that slot and which are preceded by a word (w1) that is (una, una, vn, vna) and which are are followed by que “that” in the third (w3) slot: 
select top 2000 w2,sum(x12) 
from x3 where 
w1 in ('un', 'una','vn', 'vna') and 
w3 = 'que' 
group by w2 
having sum(x12) > 2 
order by sum(x12) desc 
Likewise, the following query – which can be run against the entire 100 million word corpus in less than one second – is an attempt to define a syntactic environment for adjectives. It selects all words in the third slot (w3) of the 3-grams table (x3), in which the first slot (w1) is one of several high-frequency forms of ser “to be”, and the second slot (w2) is a form of muy “very” or tan “so”: 
select w3,sum(x12) 
from x3 where 
w1 in ('es','era','será','sera','fue') and 
w2 in ('muy','mui','muy'','tan') 
group by w3 
having sum(x12) > 2 
order by sum(x12) desc 
Finally, let us return to the category of past participle, which we considered in the previous section. Recall that in the case of pattern matching with individual 1-grams, it was fairly difficult to morphologically define a past participle (words that typically end in –ADO /-ADOS / -ADA / –ADAS), because of the number of false hits like cada, dos, and espada. By using the 2-grams or 3-grams table, however, we can constrain the search much better, by selecting only those forms that occur after one of the auxiliary verbs haber “to have”, or estar or ser “to be”, which is the real evidence for a form being a past participle In the following query, we search the 2-grams table (x2) to select those forms (w2) that end in [–ADO /-ADOS / -ADA / –ADAS], which are preceded (w1) by some of the highest frequency patterns matching forms of haber, ser and estar. 
178
select top 1000 w2 
from x2 where 
(w1 in (‘ha','has','han') or w1 like ‘ha__a%’ or like 'esta_a%' or w1 in ('fue','fueron','era','eran')) and 
(w2 like '%ado' or w1 like '%ada' or w1 like '%ados' or w1 like '%adas') 
group by w2 
having sum(x12) >= 2 
order by sum(x12) desc 
As a final note regarding POS annotation, we should mention the progressive nature of the process. As with other methods of bootstrapping, our queries will become progressively more refined and powerful. For example, once we have accurately defined the 500 or 1000 most frequent nouns, we can then use the category [noun] to look for the members of other categories. In other words, it is not necessary for us to continually be doing queries “from scratch”, as in the examples given here. 
7. Lemmatization with n-grams, frequency, collocates, and pattern matching 
In addition to POS tagging, the n-grams/frequency tables can also be used to help with lemmatization. Probably the most obvious application is to search for word forms that have a certain morphological pattern. For example, the following query will list the sixty most frequent words (w1) in the 1200s, where the word pattern is ‘quer*’ (querya, querremos), ‘qu_er*’ (quiere, qujeren) or ‘quis*’ (quisyere, quise), which are all Old Spanish forms of querer “to want”: 
select top 100 x12,w1 from x1 
where w1 like 'quer%' or w1 like 'qu_er%' or w1 like 'quis%' 
order by x12 desc 
We may determine, however, that a number of the morphologically similar word forms actually belong to another lemma. In the example above, for example, a number of the matching forms belong to the Old Spanish verb querellar “to quarrel”. To further limit the forms, we may wish to provide a syntactic context in which only the forms of querer “to want” will be found, such as those forms that precede an infinitive. The following query does this by using a sub-query to limit the [que-] forms to just those that occur immediately before an infinitive (w2; [-AR/-ER/-IR/-YR] for Old Spanish) in the table of 2-grams (x2), which is a syntactic environment in which querellar “to fight” would likely not occur: 
select top 100 x12,w1 
from x1 
where (w1 like 'quer%' or w1 like 'qu_er%' or w1 like 'quis%') 
and w1 in 
( 
select w1 from x2 
where (w1 like 'quer%' or w1 like 'qu_er%' or w1 like 'quis%') and 
(w2 like '%ar' or w2 like '%er' or w2 like '%ir' or w2 like '%yr') 
) 
order by x12 desc 
By combining pattern matching (e.g. ‘qu_er*'), collocational information for multi-word n-grams tables, and frequency information for each of these n-grams, it is possible to lemmatize hundreds or thousands of words in just a few hours, even when there is no lexicon available. 
Finally, we should once again keep in mind the progressive and cumulative nature of the annotation process. The more we have defined a particular grammatical category or a particular lemma, the easier it will be to find members of additional categories or lemma. For example, in attempting to find the members of the lemma [querer] “to want”, we have looked at n-grams in which the word in the following “slot” ends in [-AR / -ER / -IR / -YR] (in Old Spanish). Yet at certain point we will have refined the [V_INF] category to the point that we can use it directly in the queries. Likewise, once we have identified the forms of a particular lemma, we can then use that lemma directly in subsequent queries. For example, rather than using ‘quer*’ and ‘qu_er*’ to define [querer], at a certain point we can refer to the lemma [querer] directly as we search for other lemma and grammatical categories. This is of course similar to more traditional methods of bootstrapping, which perform progressively more refined annotation on the textual corpus itself. 
179
8. The advantage of n-grams databases vs. traditional approaches 
In the preceding sections, we have shown how forms can be annotated, based on information from n-grams and frequency in relational databases. This is a much less orthodox approach than the standard approach, which is to use regular expressions (or other pattern matching schemes) to search for strings and patterns in the textual corpus itself, and then insert the annotation into the textual corpus. However, there are certain clear advantages to the relational database / n-gram approach, as we discuss in this section. 
First, our approach is probably much more economical and efficient than the standard approach. As has been mentioned several times, even the most complex queries on tens of millions of records in the n-grams tables take only one or two seconds to run. In the standard approach, each query may involve a new traversal of the entire textual corpus. This is much less of a problem for small corpora (one million words or less), but may quickly become prohibitive for corpora containing tens of hundreds of millions of word of text. 
Second, the relational database approach is able to take advantage of sub-queries, which allows it to perform complex multi-stage tests on the data. For example, in order to search for a 1SG form of the present tense (fablo “I speak”), the query could have a number of sub-queries: check to see if the form ends in [-O], check whether the verbal root is also the root for a relatively high frequency infinitival-like form (i.e. with the [-AR/-ER/-IR/-YR detached), and check whether the form occurs in a multi-word sequence (e.g. a 2- or 3-grams table) where (1SG) present tense verbs would likely occur. Although the database actually performs these searches in sequence, the intermediate result of each sub-query is stored in a temporary table and is automatically processed by the database program. 
In the standard approach, it would likely be much more complicated. After each traversal of the entire corpus (to look for high frequency, morphologically-related forms, or word sequences), the program would have to store the temporary results and then re-use these as part of subsequent traversals of the corpus. Depending on the efficiency of the script used to process the data, this could quickly become too complicated or prohibitive, in terms of speed and performance. 
Third, the n-grams approach naturally lends itself to advanced collocational analysis of the data. For example, in examining the top one hundred forms of suspected adjectives, it might be useful to see the fifteen or twenty most frequent 3-grams for each word form, in which the word form occupies the middle slot. With the relational database approach, the sorted results could be produced (using sub-queries) in two or three seconds. While it would certainly be possible to produce the same listing using the standard approach, it is likely that it would be somewhat more complicated than the five or six lines of code in the relational database SQL command. 
Fourth, the relational database approach allows one to easily select and deselect word forms that are potential members of a syntactic category or particular lemma. For example, the SQL command can INSERT 100 or 1000 probable forms into a “temporary” table, along with the highest frequency preceding and following words, if so desired. These forms can then be quickly reviewed and the value for an “action” column can be quickly set to a particular value for those word forms that belong to the grammatical category or lemma. Another UPDATE command can then go back and assign the particular POS or lemma to all of the matching forms in the actual n-grams databases, for those items whose “action column” has been set to a certain value. In the standard approach, it would likely not be as intuitive or natural to review a list of suspected forms, select a subset of these, and then annotate the corpus itself based on which items have been selected. 
The fifth and final advantage of the relational database / n-grams approach is also perhaps the most important one. In this approach, the n-grams/frequency tables are simply one part of the overall database, albeit the most important part. But because they are in a relational database, they can very easily be joined to other tables within the same database, and there is no limit to the amount of annotation that can be applied to the corpus. For example, in the Corpus del Espanol, there is a table containing the synonyms for 30,000 words, and users can access this information as part of their search. An example of this is the following query, which searches for all forms of all lemma that are synonyms of mandar “to order, command”, followed by the subjunctive marker que “that”, followed by a past subjunctive (mandé que fueran “I made them leave”; hicieron que dijera “they made her say”): 
!mandar.* que *.v_subj_ra 
Likewise, users of the Corpus del Espanol can create customized lists of words that are morphologically, syntactically, or semantically related, such as adjectives describing emotions, words ending in [-azo] that denote a blow or strike, or a list of temporal adverbs. These are stored in tables, where they can later be used as part of the query syntax. For example, suppose that a user [Jones] creates a list called [emotions], which contains a list of verbs of emotion (e.g. gustar, alegrar, 
180
sorprender “to please, make happy, surprise”). The user can then use this list as part of the following syntactic construction to return phrases like me gusta que haya “it please me that there is”, le sorprende que tengan “it depresses him/her that they have”. The following is the query as it is entered into the search form, along with a description of the queryL 
me/nos/te/os/le/les [Jones:emotions].* que *.v_subj_pres 
[one of the indirect object pronouns me, nos, te, os, le, les + any form of any of the words in the [emotions] list created by Jones + que + present subjunctive] 
The web script then translates this into the following SQL command, which is passed to the database: 
select top 300 * from x4 where 
w1 in ('me', 'nos', 'te', 'os', 'le', 'les') and 
w2 in (select w1 from x_l where x1 in ('gustar', 'sorprender', 'agradar', 'alegrar')) and 
w3 in ('que') and 
w4 in (select w1 from x_c where x1 in ('v_subj_pres')) 
The important point is that there can be an unlimited number of levels of annotation on the corpus – whether parts of speech, or lemma, or synonyms, or translations between languages, or etymologies, or customized lists, and these can all be linked together with simple SQL JOIN commands. It is not apparent how this degree of flexibility or power would be an inherent property of the standard scheme, in which the annotation is based within the textual corpus itself. 
9. The design and architecture of annotation in the relational database 
As explained previously, there are two approaches to the placement of annotation in the relational databases. One approach would be to include it as additional columns in the n-grams/frequency databases themselves. For example, in the following case there is annotation (POS and lemma) within the table for each of the three words in the 3-gram son las cosas “are the things” 
Table 3: N-grams/frequency table with integrated POS and lemma annotation 
W1 L1 C1 W2 L2 C2W3 L3 C3 x12x13x14x15x16x17x18x1919-Lit19-Oral 19-Misc son ser vp las lo adef cosas cosa N 381677671619336824 40 14 
In this scenario, for example, a user who is searching for cases of a form of poder + estar + a present participle (e.g. puede estar pensando “3SG-can be thinking”) would enter the following in the web-based form: 
poder.* estar *.v_ndo 
This is then translated into the following SQL command, which queries only the 3-grams table (x3), since it has all of the POS and lemma information included within that one table: 
select top 300 * from x3 where 
L1 = ‘querer’ and 
W2 = ‘estar’ and 
C3 = ‘v_ndo’ 
The alternative is to have the n-grams/frequency information in one table (x3), and the POS and lemma information in two additional tables (x_c and x_L, respectively): 
Table 4: Separate n-grams/frequency, POS, and lemma tables 
w1 w2 w3 x12 x13 x14 x15 x16x17x18x19 19-Lit19-Oral 19-Misc x3 son las cosas 38 16 77 67 16 19 33 68 24 40 14 
w1 x1 pos w1 x1 lemma es v_pres (x_c) es ser (x_L) será v_fut cosa cosa sido v_pp cosas cosa 
In this scenario, there would be simple SQL JOIN queries to link the two databases. The same query shown above would be translated into the following SQL command. In this case, the database first sub-queries the POS (x_c) and lemma (x_L) tables, and then feeds the output from these tables into a query of the main n-gram/frequency table (x3): 
181
select top 300 * from x3 where 
w1 in (select w1 from x_L where x1 in ('poder')) and 
w2 in ('estar') and 
w3 in (select w1 from x_c where x1 in ('v_ndo')) 
So which of the two approaches produces the best results? The advantage of having the annotation within the n-grams / frequency table itself is that the database creator can use contextual information to resolve ambiguity. For example, in the 3-grams table two of the 40+ million records are for the strings el poder de “the power of” and para poder “in order to be able”: 
Table 5. Contextual disambiguation, based on n-grams 
W1 L1 C1 W2 L2 C2 W3 L3 C3 x12x13x14x15x16x17x18x1919-Lit 19-Oral 19-Miscel el a_d poder de de prep19257542001359336025567 36 152 para para prep poder ser ser v_inf0 0 2 115 3 13376 17 14 
In Spanish, poder can either be a noun (el poder “the power”) or an infinitive (poder saber “to be able to know”). Therefore, it is not immediately clear what the POS or the lemma should be for the [poder] in [w2] slots. 
Using simple SQL updates, however, we can look at the contextual information to fill in or alter the POS and lemma values, based on the values in the other word slots. For example, the following UPDATE query will assign the value of [noun] to [poder] when it occurs before a preposition (as in the first row) and it will assign a value of [v_inf] when it is followed by another infinitive (the second row). The second case is shown in the following example: 
update x3 
set c2 = ‘v_inf’ 
where w2 = ‘poder’ and c3 = ‘v_inf’ 
Judging from languages like English – which have such a high degree of polysemy – it may seem unwise to have to run a large amount of SQL updates – such as the one just shown – to improve the annotation. Yet it turns out that because Spanish is such a morphologically strong language, there are relatively few forms (as with poder) that have a high frequency as two different parts of speech or two different lemma. 
Because of this very low level of polysemy, we might take a more radical step and completely remove the annotation from the n-grams/frequency tables, as shown in Table 4 above. In this case there would be two entries for poder in both the POS and lemma tables, and in most cases, the queried phrase will naturally disambiguate itself. For example, if a user searches for [de *.v_inf *.v_inf], then [de poder saber], [de poder estar] and others will be retrieved, because poder is listed as a [v_inf] in the POS table. And it turns out that all of these examples of the potentially polysemous poder really are in its use as an infinitive. Likewise, if a user searches for [el *.n de], then [el poder de] “the power of” will be retrieved, and again virtually all of these cases will be with poder as a noun (because of the following preposition). There will be problems only in those relatively few cases in which 1) a word is polysemous and 2) both meanings are highly frequent and 3) the context is not sufficiently rich to disambiguate the multiple meanings. 
Assuming that it is the case that ambiguity in POS and lemma assignment is much less of a problem in a morphologically complex language like Spanish (as opposed to a morphologically weak language like English), then we can take the further step of simply removing the annotation from the n-grams/frequency tables altogether (where they might be useful for contextual disambiguation) and place them in separate POS and lemma tables. In fact this is precisely what we have done in the Corpus del Espanol, and more than a year's worth of complex searching on the corpus suggests that it very rarely leads to any problems. 
One of the major advantages of placing the annotation in tables that are separate from their context deals with redundancy. In the database architecture in which the POS and lemma information is part of the n-grams tables, then redundant annotation occurs every time a word occurs in any slot of any n-gram table. Thus a word like es “is” would be annotated as [POS=v_pres; lemma = ser] in each of the 209,160 rows of the 3-grams table where it appears in the [w2] slot – as well as hundreds of thousands of other rows for the other slots of the 3-grams table and the other n-grams tables. By placing the annotation in a separate table, there is exactly one entry for [es]. 
There are also secondary benefits from placing the annotation in separate tables, both of which are related to the issue of redundancy. First, because the annotation only occurs in one or two rows of the POS or lemma tables, hundreds or thousands can be updated in a matter of one or two seconds. If a 
182
form can occur in hundreds of thousands of rows, on the other hand, then updating the annotation for the large amount of forms becomes more difficult. 
A second issue is somewhat more technical in nature, and deals with the physical architecture of database tables. In most databases, only one column in each table can have a “clustered” index, which means that the rows in the table are physically arranged on the hard drive according the contents of that column. (In a non-clustered index, on the other hand, there are pointers to the data, but the data itself may be spread over the entire hard drive.) For our case, the important point is that if the clustered index is placed on the [w1] column (the first word in the n-gram), then any indices on the annotation columns in the n-grams table cannot be clustered, and queries dealing with POS or lemma will be relatively slow. By placing POS and lemma annotation in their own tables, each of these tables can contain a clustered index, and text retrieval will be much faster. This is why queries of the 100 million word Corpus del Espanol typically take just one or two seconds – for even more the most complex queries, involving POS, lemma, word patterns, synonyms, and user-defined lists of words. 
References 
Cucerzan S, Yarowsky D 2002 Bootstrapping a multilingual part-of-speech tagger in one person-day. CoNLL-2002 Sixth Conference on Natural Language Learning. Taipei, Taiwan, pp. 132-38. 
Ghani R, Jones R 2002 A comparison of efficacy and assumptions of bootstrapping algorithms for training information extraction systems. In Proceedings of LREC 2002 Workshop on "Linguistic Knowledge Acquisition and Representation: Bootstrapping Annotated Language Data". Spain: Las Palmas, pp. 61-72. 
Moreno, A., López S, Sánchez F 2003 Developing a syntactic annotation scheme and tools for a Spanish treebank. In Abeillé A (ed), Building and using syntactically annotated corpora. Dordrecht: Kluwer, pp. 149-65. 
Rocio V, Pereira G 1999 Análise sintáctica parcial em cascata". In: Marrafa P, Mota M (eds), Linguística Computacional: Investigaçao Fundamental e Aplicaçoes. Lisboa: Ediçoes Colibrí, pp. 235-251. 
Simov K, Kouylekov M, Simov A 2002 Incremental specialization of an HPSG-based annotation scheme. In Proceedings of LREC 2002 Workshop on "Linguistic Knowledge Acquisition and Representation: Bootstrapping Annotated Language Data". Spain: Las Palmas, pp. 16-23. 
van Eynde F, Zavrel J, Daelemans W 2000 Lemmatisation and morphosyntactic annotation for the spoken Dutch corpus. In M. Gavrilidou et al. (eds), Proceedings of the Second International Conference on Language Resources and Evaluation. Paris: ELRA, pp. 1427-1433. 
183

Consonant variation within words 
Joost van de Weijer 
Department of Linguistics and Phonetics 
Lund University 
Helgonabacken 12 
22362 Lund, Sweden 
email: vdweijer@ling.lu.se 
1. Introduction 
The aim of this paper is to show that the consonants that a simple (monomorphemic) word is constructed with usually are different from each other. To illustrate this, consider words like glass, drink, butter, or tray in which no consonant occurs more than once (it is the pronunciation I am concerned with, not the spelling). Changing one consonant so that it becomes identical to another one would result in rather odd-sounding new words, e.g., glagg, krink, butteb or bubber. 
Of course I realize that there are a great deal of words in which the same consonant occurs more than once, as there are clock, deed, paper, text. However, I believe that these words are in a minority, and of relatively low frequency so that they do not show up often in everyday speech. 
I do not think that the phenomenon is restricted to any language in particular, but instead applies to many languages, although I would not dare to claim all. I am not aware of any study in which this specific claim was questioned, but there are a number of findings that point in the direction of the idea that identical consonants repulse rather than attract one another. 
First of all, it is common that suffixes are deleted when the adjacent consonant in the stem is homophonous, a phenomenon that has been labeled morphological haplology (Stemberger 1981). In English, for example, the genitive –s is not realized in plural forms, unless the plural does not end on –s (cf. the *boys's books with the children's books). Similarly, in Swedish, the present tense markers –er or –ar are not realized when the verb stem ends on –r, e.g., jag kommer, (I come), but jag hör — instead of *jag hörer (I hear). 
Second, identical nearby consonants sometimes undergo change during a process of borrowing between languages or as a result of historical sound change, a process known as phonological dissimilation (Hock and Joseph 1996). Examples of such a change in loan words are the French word marbre which became marble when adopted into English, or the Italian word tartuffeli (truffle) which became Kartoffel (potato) in German. An example of an historical sound change is the Swedish word nyckel (key) which once was lyckel. 
A third observation is that whereas relatively many languages show vowel harmony (a phonological process according to which vowels within a word become more similar), there are only few examples of languages with consonant harmony. A major exception being child language in which patterns of consonant harmony frequently occur (Smith 1973; Vihman 1978). 
Finally, a well-known phonological constraint, the obligatory contour principle (OCP), states that 'adjacent identical segments are prohibited' (Clements and Hume 1995). Originally this constraint was applied to explain the distribution of lexical tones, but at a later stage it was also used to explain regularities in speech segments. McCarthy (1986), for instance, used the OCP to explain patterns in Arabic trilateral and quadriliteral roots that lack identical consonants. 
In sum, it appears that there is a tendency for nearby consonants to be different. The aim of the present study is to take this claim one step further by stating that most morphologically simplex words are constructed of only different consonants. This means that not only nearby consonants are different, but also consonants further apart, as long as they are within the same morpheme. If this is true, then the distribution of consonants can be a valuable source of information for automatic morphological 
184
decomposition, since there is an increased likeliness that there is a morphological boundary between two identical consonants. A second, related, application is that the distribution of consonants may help the listener or a speech recognizer in identifying word boundaries. Since there are no reliable markers of word boundaries in spoken language as there are in written language (Cole and Jakimik 1980), many researchers have searched for cues that may aid in the recognition of spoken words (e.g., Cutler and Norris 1988; McQueen 1998). The presence of identical consonants, then, may be a simple cue for the listener to hypothesize that there is a word boundary between them. 
As a starting point for investigating these research claims, I analyzed the distribution of identical consonants in a selection of Swedish words. Swedish is a language for which I believe that the hypothesis is true. It has a phonological structure that is in a number of ways similar to that of English. Its syllable structure, for example, is comparable to that of English, with consonant clusters consisting of up to three consonants in syllable-initial position and clusters up to three consonants in syllable-final position, or even five if suffixes are included (Sigurd 1965). The phonemic inventory contains 18 different consonants (Elert 1989), but additional consonants occur in loanwords, mainly from English or French. 
I restrict my analysis to morphologically simple words, so that compounds, derivations and inflections are not included. I do not know of any phonological process (other than morphological haplology which only applies when the adjacent consonant is homophonous) according to which consonants in affixes change because the same consonant also occurs in the stem that the affix is attached to. The suffix –s for third person singular in English, for instance, is the same for verb stems containing an s (as in he sits) or any other consonant (as in he hits). 
The remainder of the paper is organized as follows. In the next section I describe the characteristics of the material that was used, and how it was analyzed. Thereafter, I will address the following issues: How many words contain two or more identical consonants? Are certain consonants more likely to occur more than once within a word than others? Are words with identical consonants relatively common or rather rare? 
2. Method 
Material 
The material was selected from a word list that was used for the production of a current Swedish pronunciation dictionary (Hedelin 1997). The complete list is composed of 116,362 entries. An entry consisted of the orthographic word form, a word-class code (noun, adjective, adverb, pronoun, verb, count noun, proper noun, preposition, article, interjection, abbreviation, conjunction, infinitive marker, prefix, word group), the word's pronunciation, and whether the word was a compound, a derivation or an inflected form. 
Selection of the material 
The aim of the selection was to obtain a word list of different monomorphemic words. Clearly, the dictionary contained a large number of entries that did not meet this requirement. Furthermore, a proportion of the dictionary was redundant for the purpose of the present analysis, since many entries were repetitions of the same word root (e.g., words that could be a noun as well as a verb). Luckily, a large proportion could be excluded on the basis of the information already provided in each entry. A total of 90,463 entries coded as compounds, e.g., bordduk (table cloth), derivations, e.g., alkoholist (alcoholic – noun), alkoholism (alcoholism), alkoholisk (alcoholic – adjective), etc., and inflected forms (e.g., present tense verbs, genitive forms of pronouns, etc.) were excluded automatically. 
Obviously, it is debatable where the line between what counts as a morphologically complex word and a morphologically simple word should be drawn (see, e.g., Sproat 1992 for further discussion on this issue). From the list that resulted after the exclusion, I manually excluded an additional 1,164 of entries that I considered morphologically complex as well. The entries that were excluded contained words with a meaning that could easily be derived from the individual meanings of the components. These words included for instance a number of change of state verbs which in Swedish are constructed of an adjective plus the suffix –na (e.g., the verb mjukna (to become soft) consists of the adjective mjuk 
185
(soft) followed by the suffix na). When the meaning could not be derived from the word's individual components – for instance the meaning of the word protest (protest) – the word was not excluded. 
Although this strategy worked reasonably well for bisyllabic words, it became more difficult for words of three syllables or longer. For that reason, I decided to restrict the analysis to words of one and two syllables only, and excluded all words that were longer than two syllables. 
Finally, proper names (e.g., Alfa Romeo, Charlotte, Amsterdam), abbreviations and acronyms, (e.g., SAS, NATO), interjections, (e.g., oj (oops), usch (ugh)), orthographic variants (e.g., mej (me) instead of mig, or sebra (zebra) instead of zebra), and repeated entries of the same word root (e.g., bank in the meaning of 'sofa' as an alternative to bank in the meaning of 'bank') were also excluded. The final selection then consisted of 8,887 words. The number of consonants in each word varied from zero to eight. 
Word Frequencies 
Word frequency information was obtained from a language corpus collected at the University of Gothenburg. The research group there provided me with a list of approximately 100,000 word types (100,998) that had a frequency of 20 or more in a large corpus consisting mainly of newspaper articles. The sum of the frequencies of these words was somewhat below 57 million tokens (56,916,383). In this list, there were a total of 5,388 word types that also occurred in the selection of morphologically simple words. Together these had a total token frequency of little below 29 million (28,845,117). 
Analysis 
The decision whether a word contained two or more identical consonants was made on the basis of the transcripts provided in the dictionary, with the following two exceptions. 
In the dictionary a distinction is made between long and short consonants. This difference is primarily context dependent, i.e., long consonants follow short vowels, as in the word tack /tAk&#162;/ (thanks), and short consonants follow long vowels, as in the word tak /tA&#162;k/ (roof). In addition, the difference in pronunciation is not realized in many Swedish dialects (Elert 1989). For these reasons, long and short variants of the same consonant were considered to be identical. 
Second, in some dialects of Swedish, the /r/ has an effect on the place of articulation of one or more subsequent alveolar consonants. In these cases, the consonants adopt a retroflex or supradental place of articulation, and the /r/ disappears. The word barn (child), for example, is pronounced as /bA ~/ (Elert 1989). The assimilated forms are given in the dictionary, but these were replaced in the analysis by the unassimilated form (e.g., /bA rn/). 
Notice that these two criteria make the analysis more conservative, since the likeliness of finding two or more identical consonants within a word increases in both cases. 
3. Results 
During the analysis, it became clear that no consonant occurred twice within a syllabic onset or syllabic coda. For instance, there were no syllables that ended on –tst or –ksk or any other combination of three consonants. Although this is an intuitively plausible finding it is not a trivial observation since the presence of two identical consonants within a syllable onset or coda is in principle not prohibited by phonotactic constraints. In English, for instance, any noun ending on –st combined with a genitive –s, or any verb ending on –sp combined with third person singular –s results in words with codas with two identical consonants, e.g., guest's or gasps, etc. 
A second thing that became clear during the analysis, was that no two consonants directly adjacent to each other ever were the same. This could have been possible, for instance, at the boundary between the first and the second syllable in a bisyllabic word. However, there were no such examples. 
These two findings were the starting point for dividing the words according to their syllable structure as shown in Table 1. The left column of the table shows the structure of the words, starting with words 
186
that consisted of a vowel only. The O's and the C's in the following rows stand for Onset and Coda, where O1 and C1 refer to the onset and the coda of the first syllable, and O2 and C2 refer to the onset and the coda of the second syllable. C1 in the second row then, refers to all the words that had one or more consonants in the coda of the first syllable, but no consonants in the onset, or in the second syllable if there was any. The second column shows the numbers of word types with the structure listed in the first column. The third and the fourth columns show the absolute and the relative numbers of hits, i.e., the words that contained at least two identical consonants. The last column shows two examples of words with the structure given in column 1. The examples are, wherever possible, words with at least two identical consonants. 
The table starts with five categories that could not have identical consonants. For the sake of simplicity, bisyllabic words with only one consonant in C1 and one consonant in O2 (for example the words inte and syssla in the table) are classified as if these two consonants both belonged to the onset of the second syllable. Together the first five categories consisted of 438 types, corresponding to 4.93% of the total. 
The total number of words with at least two identical consonants was 1,001 (11.26%). Most of them contained no more than two consonants that were the same, a few common examples of which were the pronouns nagon (someone) and annan, (other), the adverb nästan (almost), the conjunction trots (although), and the adjective/noun svensk (Swede, Swedish). However, there were 54 word types in which two consonants occurred twice, including the words status (status), porträtt (portrait), struktur (structure), and taktik (tactics). In addition, there were ten words in which one consonant occurred three times, including skepsis (skepticism), substans (substance), traktat (treaty), census (census). There were no words that contained more than three identical consonants or more than two consonants that occurred twice. 
Table 1: Type count. C stands for coda; O stands for onset. 
structure types hits (%) examples - 3 0 (0.000) i (in), ö (island) C1 113 0 (0.000) upp (up), älv (river) C2 3 0 (0.000) oas (oasis), eon (aeon) O1 153 0 (0.000) trä (wood), vra (corner) O2 166 0 (0.000) idé (idea), inte (not) C1O2 29 0 (0.000) önska (wish), ändra (change) O1O2 2,373 154 (6.490) syssla (work), kruka (pot) O1C1 2,432 185 (7.607) klipsk (shrewd), klok (wise) O2C2 424 45 (10.613) annan (other), essens (essence) O1C1O2 159 20 (12.579) virvla (whirl), dundra (thunder) O1C2 66 12 (18.182) nyans (nuance), neon (neon) O1O2C2 2,799 527 (18.828) banan (banana), staket (fence) C1O2C2 39 11 (28.205) emblem (badge), anstalt (institution) O1C1O2C2 128 47 (36.719) substans (substance), distrikt (district) totals 8,887 1,001 (11.264) 
Among the words that contained identical consonants, there were four relatively small groups of words that suggested that identical consonants in some cases serve a specific purpose. These four groups contained words that are typically used by children, onomatopoetic verbs and nouns, a set of nouns and adjectives that all had a pejorative or distasteful connotation, and finally a set of bisyllabic mimetic ('flip-flop') words that consist of two syllables that are exact or nearly exact replications of each other (see Table 2 for examples). 
187
Table 2: Specific functions of identical consonants. 
child words onomatopoetic phonestemes 'flip-flop' words mamma (mother) 
pappa (dad) 
baby (baby) 
pippi (dickybird) 
jojo (yoyo) 
bebis (baby) 
vovve (dog) 
dada (nanny) pipa (chirp, squeak) 
bubbla (bubble) 
mummel (murmur) 
tuta (hoot) 
nynna (hum) 
babbla (blather) knark (drugs) 
skurk (villain) 
strunt (rubbish, trash) 
snusk (uncleanness) 
skolk (truancy) 
skunk (skunk) 
stursk (insolent, impudent) 
smisk (smack) 
slisk (cloyingly sweet) 
slusk (shabby person) 
smask (smacking noise) 
slask (wet garbage) 
stursk (stubborn) tiptop (tip-top) 
vigvam (wigwam) 
pingpong (ping-pong) 
picknick (picnic) 
virrvarr (crisscross) 
sicksack (zigzag) 
mischmasch (mishmash) 
gonggong (gong) 
snicksnack (chatter) 
Which consonants were most likely to occur more than once in a word? Table 3 shows how often each consonant occurred overall and how often it occurred as identical consonant. Comparing the two frequencies it shows that /t/, /k/ and /s/ were relatively likely to occur more than once, since their overall relative frequencies were much lower than their frequency as identical consonants. On the contrary, /l/ was not likely to likely to occur more than once since there was an almost 7% difference in how often it occurred overall and how often it occurred as identical consonant. The difference between the two frequencies was not more than 3% for all the other consonants. 
Table 3: Consonant frequencies (%). 
consonant total relative frequency frequency as identical consonant d 5.022 2.395 f 2.998 0.958 g 3.417 2.011 h 1.499 0.000 j 2.929 0.383 k 9.068 14.464 l 10.644 3.831 m 5.276 4.789 n 7.787 7.184 p 4.832 4.693 r 12.831 13.027 s 11.928 20.881 t 10.444 19.253 v 3.413 2.490 Ó 1.008 0.000 &#178; 0.568 0.192 N 2.180 0.479 S 0.389 0.192 b 3.766 2.778 
Table 4 shows the results of the token count. Comparing the values listed in Tables 1 and 4 reveals that the total percentage of words with identical consonants has decreased from 11.26 to 1.57. A major cause of this decrease is that the numbers of tokens in the first five rows are much higher in Table 4 than in Table 1. Included in these categories are a small number of very highly frequent function words, such as och (and), att (to), en (a). However, all the percentages in the other rows of Table 4 are 
188
lower than the corresponding percentages in Table 1 as well, indicating that the words with identical consonants were relatively rare. 
Table 4: Token count. O stands for Onset, C stands for Coda. 

structure types tokens targets (%) - 3 1,738,874 0 (0.000) C1 92 6,866,180 0 (0.000) C2 1 241 0 (0.000) O1 134 3,171,934 0 (0.000) O2 123 928,972 0 (0.000) C1O2 21 127,133 0 (0.000) O1C1 1,796 11,079,424 164,988 (1.489) O1O2 1,397 2,244,855 37,974 (1.692) O1C2 28 15,063 295 (1.958) O2C2 243 1,067,550 44,522 (4.170) C1O2C2 20 47,768 2,079 (4.352) O1C1O2 71 54,653 4,013 (7.343) O1O2C2 1,402 1,475,306 194,242 (13.166) O1C1O2C2 57 27,164 5,021 (18.484) totals 5,388 28,845,117 453,134 (1.571) 
4. Conclusions 
The hypothesis of the present investigation was that monomorphemic words that contain identical consonants are rare. The results showed that 11.26 percent of Swedish monosyllabic and bisyllabic words contained at least two identical consonants. However, these words seem to be relatively uncommon and, consequently, will not be heard or read often. The estimate that I found based on a selection of more than 28 million word tokens, I estimated that only 1.57 percent contained identical consonants. 
Taken together, the results show that identical consonants within words indeed constitute a dispreferred pattern in Swedish. Based on the result of the token count, it would be expect to find a word with two or more identical consonants only one in every 65 words. The first question that is open for future research is whether this is also the case in other languages. I expect to find similar percentages in languages with a phonological structure that is comparable to that of Swedish, for instance the other Germanic languages. More interesting comparisons will be with languages that have deviant phonological structures, e.g., different syllable structure or fewer consonants. 
A second question that needs to be answered is whether listeners are aware that identical consonants within words are relatively rare, and whether they use the presence of identical consonants for morphological decomposition. If this is true, then it is predictable that listeners will identify non-words with two identical consonants more quickly than non-words with only different consonants. A second prediction is that listeners will decompose morphologically complex words that contain identical consonants (e.g., sits) more quickly than words that do not (e.g., hits). I intend to test these predictions in the near future. 
Finally, one of the reasons for doing the present study was that identical consonants might be useful for morphological decomposition. As a preliminary attempt to answer the question how useful identical consonants are I counted the number of morpheme boundaries that were marked by identical consonants in a short Swedish text. The following text was selected from the introduction of a writing style guide (Svenska Skrivregler, Utgivna av Svenska Spraknämnden, 1991): 
Var-för behöv-er man skriv-regl-er. Tal är ett sprak för öra-t, skrift är ett sprak för öga-t. I tal-et ha-r vi fler-a sätt att signal-era hur det vi säg-er skall upp-fatta-s: ton-fall, paus-er, beton-ing, röst-styrka, och dess-utom kan vi an-vänd-a gest-er och min-er. I skrift maste vi an-vänd-a andr-a signal-er, som stycke-in-del-ning, stor eller lit-en bok-stav, skilj-e-tecken. De o-lik-a signal-er-na i tal-sprak lär vi oss 
189
huvud-sak-lig-en spontan-t de är natur-lig-t fram-vuxn-a, men för skrift-sprak-et behöv-s viss-a bestäm-d-a regl-er för den yttre ut-form-ning-en. 
In this text, there are 56 morpheme boundaries (marked by dashes) and 79 word boundaries (marked by spaces). Out of these 135 boundaries, 26 (19.3%) were marked by identical consonants. On the contrary, there was only one place were two identical consonants did not mark a boundary, namely within the word spontan-t (spontaneously). This suggests that the presence of two identical consonants is a limited but rather reliable source of information that is useful for the identification of morphological boundaries. 
5. References 
Clements G, Hume E 1995 The internal organization of speech sounds. In Goldsmith J (ed), The handbook of phonological theory. Oxford, Basil Blackwell Ltd. pp 245–306. 
Cutler A, Norris D 1988 The role of strong syllables in segmentation for lexical access. Journal of Experimental Psychology: Human Perception and Performance 14: 113–121. 
Elert C 1989 Allmän och svensk fonetik [General and Swedish phonetics]. Stockholm, Norstedts Förlag AB. 
Hedelin P 1997 Norstedts svenska uttalslexikon [Norstedts Swedish pronunciation dictionary]. Svenska Förlag: Norstedts Ordbok. 
Hock H, Joseph B 1996 Language history, language change, and language relationship. New York, Mouton de Gruyter. 
McCarthy J 1986 OCP effects: gemination and antigemination. Linguistic Inquiry 17: 207–263. 
McQueen J 1998 Segmentation of continuous speech using phonotactics. Journal of Memory and Language 39: 21–46. 
Sigurd B 1965 Phonotactic structures in Swedish. Lund, Berlingska Boktryckeriet. 
Smith N 1973 The acquisition of phonology. Cambridge, Cambridge University Press. 
Sproat R 1992 Morphology and computation. Cambridge MA, The MIT Press. 
Stemberger J 1981 Morphological haplology. Language 57: 791-817. 
Vihman M 1978 Consonant harmony: its scope and function in child language. In Greenberg J (ed) Universals of language, vol. 2, phonology. Stanford CA, Stanford University Press, pp 281–334. 
190

Rationale for a multilingual corpus for machine translation evaluation 
Debbie Elliott Anthony Hartley Eric Atwell debe@comp.leeds.ac.uk a.hartley@leeds.ac.uk eric@comp.leeds.ac.uk 
School of Computing and Centre for Translation Studies, 
University of Leeds, Leeds LS2 9JT. England. 
1. Introduction 
An overview of research to date in human and automated machine translation (MT) evaluation (Elliott 2002) points to a growing interest in the investigation of new automated methods, allowing for the quick and inexpensive evaluation of MT output. It is clear, however, that corpora designed for this purpose are lacking. Our own research in automated evaluation methods will require not only a corpus of source texts with machine translations that represent actual MT use, but also the detailed scores for these translations given by human evaluators. These scores will allow us to test the reliability of new automated evaluation methods. It is our intention, therefore, to compile a multilingual corpus specifically for MT evaluation, to meet not only our own research requirements, but the needs of the MT community at large. 
2. Machine translation evaluation 
The evaluation of machine translation output has played a crucial role in the development of MT systems since their emergence over five decades ago. Evaluations are required both by developers, before and after system modifications, and by end-users who wish to compare different systems before making a purchase. However, evaluating the quality of any translated text is complex. Unlike the evaluation of part-of-speech taggers, parsers or speech recognisers (Atwell et al. 2000) it is not simply a matter of comparing MT output to some “gold standard” human translation, since translation is legitimately subject to stylistic and other variation. Instead, MT evaluation relies on either the objective scoring of very specific linguistic phenomena using test suites, or the somewhat subjective quality judgements made by evaluators, who are trained to score individual sentences or text segments using a chosen metric. The problem of subjectivity can, however, be reduced by obtaining scores from several evaluators for each sentence and by calculating a mean score. The reliability of results can also be increased by using a large number of texts. 
Designing and conducting reliable human MT evaluations has proven to be costly and time-consuming. As a result, more recent research has involved the investigation and application of automated methods, including IBM's BLEU (BiLingual Evaluation Understudy) method (Papineni et al. 2001) and work by Rajman and Hartley (2001, 2002). Successful automated evaluation methods will allow both developers, who need to conduct frequent evaluations after system modifications, and end-users to evaluate systems more quickly and cheaply. 
3. Corpora or test suites? 
The evaluation of MT output involves the use of either a collection of texts, which in few cases seem large enough to be classified as corpora, or test suites. A corpus designed for this purpose has typically comprised texts in the chosen source language(s), machine translations produced by the systems for evaluation and one or more expert human translations of each text. Bilingual evaluators might then rate the fidelity (preservation of original content) of each machine-translated sentence or marked segment by comparing it to the source text and assigning a score using a particular scale. Alternatively, monolingual native speakers of the target language would perform the same kind of evaluation using the expert human translations for comparison. Scoring the fluency of each sentence, on the other hand, requires access only to the machine translations from the corpus, as no reference to the source text is needed when evaluating this attribute in isolation. 
Whereas corpora are widely used for “black box” MT evaluations by end-users, test suites are more often devised and used by researchers and developers, who need to pinpoint the handling of specific linguistic phenomena to guide system modifications (a “glass box” approach). Test suites for MT evaluation typically comprise many short annotated test items in the source language, with correct target translations, which are referenced according to specific linguistic categories. They allow for the systematic 
191
and objective evaluation of carefully selected linguistic phenomena, complete control over every test point (which may be tested in isolation or in combination with other features) and the opportunity to include negative data to determine how a system deals with input errors. However, as test suites are normally designed to evaluate the handling of grammatical phenomena, the vocabulary is intentionally limited, making them less suitable than corpora for the evaluation of MT system glossaries. Furthermore, test suites for natural language processing applications “normally list items on a par without rating them in terms of frequency or even relevance with respect to an application” (Oepen et al. 1997: 25). Corpora, on the other hand, represent naturally occurring data and can be designed to include texts that reflect user needs. This factor is particularly important for end-users who wish to select an MT system to translate specific text types. It is clear, therefore, that the use of test suites and corpora are not competing evaluation methods, but complementary, insofar as they serve different purposes. Our own research interests lie in the evaluation of MT systems for end-users. We require, therefore, a corpus that represents current user needs. 
4. A need for multilingual corpora for MT evaluation 
Previous research in MT evaluation has involved the use of either sentences or fairly small numbers of texts. Papineni et al. (2001), for instance, rely on a very small corpus that includes human reference translations. Other research (see Table 1) has made use of the much larger DARPA (Defense Advanced Research Projects Agency) corpus, along with results from the largest DARPA human MT evaluation, carried out in 1994. Researchers have used the DARPA corpus and evaluation results to validate (or not, as the case may be) experimental automated evaluation methods, by seeking correlations between the human DARPA scores and those from new methods. Table 1 details texts or corpora used in a sample of published MT evaluation projects, listed chronologically. 
Table 1: The use of corpora and test sentences in previous MT evaluation projects 
Author(s) and/or project name Evaluation 
type Attributes 
tested No. of source items used for evaluation = N No. of human translations of N No. of machine translations of N Carroll 
(Pierce 1966) Human Intelligibility 
Fidelity 144 sentences 
Scientific Russian 3 English 3 English Nagao et al. 
(1985) Human Intelligibility Accuracy 1,682 sentences 
Scientific Japanese 0 1 English Shiwen 
(1993) Human and automated 6 test points: 
words, idioms, morphology, elementary, moderate, advanced grammar 3,200 random sentences 
English 1 Chinese 1 Chinese DARPA 
1994 series 
(White 1997, 2001, forthcoming) Human Adequacy 
Fluency 
Informativeness 100 texts French 
100 texts Spanish 
100 texts Japanese 
(news articles of approx. 300-400 words or 800 Japanese characters) 2 English 5 English 
(Human and machine translation scores available for research) JEIDA 
(Isahara 1995) Human Linguistic test sets 770 sentences English 1 Japanese 8 Japanese Author(s) and/or project name Evaluation 
type Attributes 
tested No. of source items used for evaluation = N No. of human translations of N No. of machine translations of N 
192
IBM BLEU 
(Papineni et al. 2001) Human and automated Number of n-gram matches between MT output and human translations (with penalties) Approx. 500 sentences Chinese (all from news articles) Up to 4 English 3 English White and Forner 
(2001) Test: potential automated method Noun-compound handling 33 texts French 
33 texts Spanish 
(DARPA corpus) 0 5 English 
(DARPA corpus with scores) Reeder et al. 
(2001) Test: potential automated method Named-entity handling 0 1 English of 1 Spanish text 
(DARPA corpus) 5 English of 1 Spanish text 
(DARPA corpus with scores) Miller and Vanni 
(2001) 
Vanni and Miller (2001, 2002) Test: 
potential automated methods Coherence, clarity, syntax, morphology, dictionary update, names, terminology 0 1 English of 2 Spanish texts 
1 English of 1 Japanese text (DARPA corpus) 3 English of 2 Spanish texts 
3 English of 1 Japanese text (DARPA corpus) Rajman and Hartley 
(2001, 2002) Human and automated Grammaticality, preservation of semantic content 20 French (DARPA corpus) 1 English 5 English of 100 French texts (DARPA corpus with scores) 
1 English of 20 French texts (DARPA) by an additional MT system 
The largest known corpus for MT evaluation, the DARPA corpus, makes available the associated evaluation scores, which has proved invaluable to the MT community. However, this corpus does have its limitations; it comprises only newspaper articles, representing only a small part of MT use, the source texts are in only three languages and all target texts are in American English. It is also clear from the above information that most projects and, therefore, corpora for MT evaluation are concerned with English as a target language. 
In response to these findings, it is our intention to compile a multilingual corpus specifically for MT evaluation. This will not only be used for our own work, but will also be made available for research within the MT community. Before text collection begins however, decisions must be made regarding corpus content, size, language pairs and text types for inclusion. 
5. Corpus content 
We intend to provide a balanced corpus in terms of the number of words and text types for each language pair. Texts and language pairs will be selected to reflect the actual use of MT systems and our decisions will be guided by a survey of MT users. The corpus will comprise source texts with at least one human translation and a number of machine translations of each one, along with our own detailed human evaluation scores. 
The corpus will be made available online, allowing users to browse the contents of each language pair, displayed in the form of a list of text types and topic areas. Users will be able to view each source text along with its human and machine translations, and analyse our human evaluation scores, which will be regularly updated as soon as they become available. The source texts will be of use to anyone wishing to evaluate their own system(s), and the human reference translations will provide material for comparison when scoring the MT output. Furthermore, our evaluation results, in addition to those from the DARPA series, will allow for the testing of experimental automated metrics. 
193
6. Corpus size 
Constraints in terms of research time and cost mean that informed decisions must be made with respect to corpus size. Using a very large corpus would be unsuitable for human MT evaluation projects for practical reasons: the greater the number of texts, the more time-consuming and expensive the evaluation. Furthermore, the provision of expert human translations of thousands of texts is costly and unnecessary if valid evaluation results can be obtained from a smaller corpus. On the other hand, proven automated evaluation methods might benefit from a larger corpus, which would allow for the generation of more scores at no greater cost than if a smaller number of texts were used. 
This begs the question: at what point does a larger number of texts cease to give us more reliable evaluation results? How many texts do we need to obtain valid scores for system comparison? Our first attempt to answer this question has involved analysing DARPA scores with varying numbers of evaluated texts. We used the three scores (adequacy, fluency and informativeness) for the five machine translations and one human translation of each of the 100 French source texts (of approximately 300-400 words) to calculate a mean score for each number of texts evaluated. Figures 2, 3 and 4 show the mean scores for each of the three attributes for every number of texts evaluated (ie. from one text to one hundred texts). Figure 4 shows the overall mean scores. Figure 2: Comparison of adequacy scores: DARPA 1994 (French-English)00.10.20.30.40.50.60.70.80.911112131415161718191Number of texts evaluatedMean adequacy score
194
CandideGlobalinkMetalSystranHumanXLT Figure 3: Comparison of fluency scores: DARPA 1994 (French-English)00.10.20.30.40.50.60.70.80.911112131415161718191Number of texts evaluatedMean fluency scoreFigure 4: Comparison of informativeness scores: DARPA 1994 (French-English)00.10.20.30.40.50.60.70.80.911112131415161718191Number of texts evaluatedMean informativeness score
195
CandideGlobalinkMetalSystranHumanXLT
CandideGlobalinkMetalSystranHumanXLT Figure 5: Comparison of overall scores: DARPA 94 (French-English)00.10.20.30.40.50.60.70.80.911112131415161718191Number of texts evaluatedOverall mean score
Results show that scores from a very small number of texts (perhaps a sample of ten, amounting to around 3,500 words) can allow us to determine the highest and lowest ranking systems, in terms of individual attributes and overall scores. However, the highest scoring “system” here was the human, whom we would normally expect to perform better than the MT systems. It must also be noted that some MT evaluation projects do not involve the evaluation of human translations, but focus on the comparison of MT systems alone. Even then, we are able to determine that Systran performs better than the other MT systems by using scores from as few as ten texts. The only anomaly here is the informativeness score, where Systran and Globalink compete. 
A clearer picture of how all five MT systems compare can be obtained after the evaluation of approximately 40 texts (around 14,000 words) for each attribute, and further sampling serves only to confirm this. After around 30 samples, we see that scores begin to remain consistent within a relatively small variance fluctuation, although we do find instances of pairs of systems constantly switching position as more texts are evaluated (Systran/Globalink for informativeness, Globalink/Metal for fluency and adequacy and Metal/Candide for the overall score). In these cases, any number of samples may never see the situation resolved, and the systems that continue to compete according to the number of texts evaluated, can be considered “equal” in terms of particular attributes. It would then be up to the potential user to decide which attribute was more important for their translation needs. For example, a high adequacy score and low fluency score would be more acceptable to someone wishing to use an MT system for gisting or information extraction. 
Having obtained these results, our second step was to conduct the same statistical analysis using the Spanish-English and Japanese-English DARPA scores. Results for both language pairs confirmed that reliable scores can be obtained from the evaluation of around 40 texts. We now intend to use texts from our new corpus to conduct human evaluations and to carry out the same analysis. Our initial sample will comprise 35-40,000 words, equal in size to one language pair in the DARPA corpus. This will allow us to compare the number of words required for valid results when evaluating both newspaper articles and different text types, which better represent MT user needs. Our findings will then guide us in terms of the initial number of words required per language pair. 
196
CandideGlobalinkMetalSystranHumanXLT
7. Language pairs 
In January 2003 we carried out a survey of MT users in order to obtain guidelines for corpus content. In the survey, sent as an email to a number of MT and translation-related mailing lists, we asked which language pairs and text types users regularly translate with the aid of fully automatic MT systems. The 25 replies received to date (16 from large translation providers or international corporations/organisations, 9 from single users) have provided valuable information on both issues. Of the 25 responses, 21 were used for this research, as 4 reported only on their use of translation memory tools. The survey is ongoing and will shortly be available online. 
Texts in a number of different language pairs will be needed for our own research, when we investigate new approaches to automated MT evaluation. Furthermore, the availability of texts and translations in several languages will make the corpus more useful for other research projects. It is important to evaluate texts translated from and into more than one language, including languages that are typologically different from one another, to explore the portability of new evaluation methods. Additionally, translation providers often use MT to translate more than one language pair and may need to test systems for several languages. 
Figure 6 shows the language pairs (in which the source or target language is English) translated by respondents using MT systems. A very small number of respondents also use systems to translate language pairs that do not involve English. Figure 6: Language pairs translated by MT users024681012English-SpanishEnglish-FrenchEnglish-GermanEnglish-PortugueseEnglish-ItalianEnglish-JapaneseEnglish-DanishEnglish-DutchEnglish-GreekEnglish-ChineseEnglish-FinnishEnglish-NorwegianEnglish-RussianEnglish-SwedishEnglish-VietnameseGerman-EnglishFrench-EnglishSpanish-EnglishItalian-EnglishJapanese-EnglishChinese-EnglishPortuguese-EnglishFinnish-EnglishVietnamese-EnglishLanguage pairNumber of respondents 
197
The number of language pairs that MT systems are now able to handle is constantly increasing. The IAMT (International Association for Machine Translation) Compendium of Translation Software (Hutchins and Hartman 2002) lists an enormous number of MT systems translating many more languages than those shown above. As a starting point, therefore, we plan to collect source texts (with human and machine translations in English) in French, German, Spanish and Italian, along with texts in some typologically different languages, such as Chinese and Japanese to begin with. These will allow us to carry out our initial evaluations of systems translating into English. However, in a second phase we will add translations out of English, which will allow us to test how well existing MT evaluation methods transfer to other language pairs and to develop new machine learnt metrics, which generalise across languages. The target languages for inclusion will be the subject of further research. 
8. Text types 
Since expectations of MT systems have become more realistic, a greater number of uses have been found for imperfect raw MT output. Consequently, a variety of text types, genres and subject matter are now machine-translated for different text-handling tasks, including filtering, gisting, categorising, information gathering and post-editing (White 2000). It is crucial, therefore, to represent this variety of texts, ranging from emails to technical reports, in our corpus, allowing for the evaluation of texts that represent real MT use. 
The main purpose of our survey was to gather information on the kinds of texts and topics most frequently translated using MT systems. Information obtained from this part of the survey is providing useful guidelines on the types of texts to include in our corpus, but there are several problems involving the analysis of data. Firstly, results are based on respondents’ own interpretations of the “text types” suggested in the survey and these inevitably overlap in terms of content and grammatical structures. For example, technical material can be found in several separate categories in our questionnaire: internal company documents, technical documents, user manuals, instruction booklets, academic papers and web pages. This must be taken into account when we select our texts. Secondly, some respondents did not specify the subject matter of the material they machine translate, and many were unable to provide details on the number of texts. Finally, it is difficult to equate the comparatively small number of words translated by single users with the millions of words translated by international companies every year. In response to this last problem, we present two sets of results at this stage. Figure 7 shows the number of companies and Figure 8, the number of single users who use MT to translate particular text types. 
Responses to date show that single users and companies use MT systems to translate different types of documents. Five of the international companies/organisations who responded did give information about the number of texts they translate. Of these five respondents, all use MT systems to obtain a first draft of either user manuals, instruction booklets, technical documents or internal company documents, or a combination of these. Their total monthly word count is estimated at 3.5 million words. It is crucial, therefore, to represent these documents in our corpus. However, the single user market is likely to grow, as systems become cheaper, so it is important to reflect the needs of such users also. 
Findings so far tell us that we must represent all of the above text types to reflect MT use. Documents in our corpus will be categorised, enabling anyone wishing to compare MT systems to easily select source texts for evaluation according to their own needs. The subject matter of these texts will inevitably overlap, as it does in the real world. We are still receiving replies to our survey and updated results will shortly be available online. 
198Figure 7: Number of companies who use MT to translate particular text types024681012web pagesacademic papersnewspaper articlestechnical docsemailstourist/travel infoscientific docsmedical docslegal docsinternal company docsbusiness letterspatentscalls for tenderuser manualssoftware stringsmemosinstruction bookletsfinancial docsText typeNumber of respondents 
Figure 8: Number of single users who use MT to translate particular text types01234web pagesacademic papersnewspaper articlestechnical docsemailstourist/travel infoscientific docsmedical docslegal docsinternal company docsbusiness letterspatentscalls for tenderuser manualssoftware stringsmemosinstruction bookletsfinancial docsText typeNumber of respondents 199 9. Conclusion Our findings to date have provided valuable guidelines for the size and content of our corpus. Analysis of the existing DARPA scores indicates that a small sample of texts (amounting to around 14,000 words) is sufficient to rank a range of MT systems in terms of individual attributes and overall scores. However, our user survey indicates that we need to cover a much wider range of genres, beyond newspaper articles, so there is still a need for a larger corpus. We intend to compile a dynamic corpus, which will be updated to reflect changing trends in the MT user market. New source texts and translations will be added to reflect language change and the introduction of new terminology, and additional MT systems will be added to our evaluations over time. The key feature of our corpus, however, will be the detailed scores from our human evaluations, which will be made available to aid research in automated MT evaluation. Acknowledgements We wish to thank everyone who has responded to our MT user survey. References Atwell E, Demetriou G, Hughes J, Schiffrin A, Souter C, Wilcock S 2000 A comparative evaluation of modern English corpus grammatical annotation schemes. ICAME Journal 24: 7-23. Elliott, D 2002 Machine Translation Evaluation: Past, Present and Future. Unpublished MA dissertation, University of Leeds. Hutchins J, Hartmann W 2002. IAMT Compendium of Translation Software 1.5. http://www.eamt.org/compendium.html Isahara H 1995 JEIDA's Test-Sets for Quality Evaluation of MT Systems – Technical Evaluation from the Developer's Point of View. In Proceedings of Machine Translation Summit V, Luxembourg. Miller K, Vanni M 2001 Scaling the ISLE Taxonomy: Development of Metrics for the Multi-Dimensional Characterisation of Machine Translation Quality. In Proceedings of Machine Translation Summit VIII, Santiago de Compostela, Spain. Nagao M, Tsujii J, Nakamura J 1985 The Japanese government project for machine translation. Computational Linguistics 11: 91-109. epen S, Netter K, Klein J 1997 TSNLP – Test Suites for Natural Language Processing. Linguistic Databases. CSLI Lecture Notes, CSLI Stanford. Papineni K, Roukos S, Ward T, Zhu W 2001 BLEU: a Method for Automatic Evaluation of Machine Translation. IBM Research Report RC22176. Yorktown Heights, NY:IBM. Pierce J (Chair) 1966 Language and Machines: computers in Translation and Linguistics. Report by the Automatic Language Processing Advisory Committee (ALPAC). Publication 1416. National Academy of Sciences National Research Council. Rajman M, Hartley A 2001 Automatically predicting MT systems rankings compatible with Fluency, Adequacy and Informativeness scores. In Proceedings of the 4th ISLE Workshop on MT Evaluation, Machine Translation Summit VIII, Santiago de Compostela, Spain. Rajman M, Hartley A 2002. Automatic ranking of MT systems. In Proceedings of the Third International Conference on Language Resources and Evaluation, Las Palmas de Gran Canaria, Spain, pp 1247-1253. Reeder F, Miller K, Doyon J, White J 2001 The Naming of Things and the Confusion of Tongues. In Proceedings of the 4th ISLE Workshop on MT Evaluation, Machine Translation Summit VIII, Santiago de Compostela, Spain. Shiwen Y 1993 Automatic evaluation of output quality for machine translation systems. Machine Translation 8: 117-126. Vanni M, Miller K 2001 Scaling the ISLE Framework: Validating Tests of Machine Translation Quality for Multi-Dimensional Measurement. In Proceedings of the 4th ISLE Workshop on MT Evaluation, Machine Translation Summit VIII, Santiago de Compostela, Spain. Vanni M, Miller K 2002 Scaling the ISLE Framework: Use of Existing Corpus Resources for Validation of MT Evaluation Metrics across Languages. In Proceedings of the Third International Conference on Language Resources and Evaluation, Las Palmas de Gran Canaria, Spain. White J 1997 MT Evaluation: Old, New and Recycled Methods. Tutorial slides, Machine Translation Summit VI, San Diego. White J 2000 Toward an Automated, Task-Based MT Evaluation Strategy. In Proceedings of the Workshop on the Evaluation of Machine Translation, Third International Conference on Language Resources and Evaluation, Athens, Greece. White J, Forner M 2001 Predicting MT fidelity from noun-compound handling. In Proceedings of the 4th ISLE Workshop on MT Evaluation, Machine Translation Summit VIII, Santiago de Compostela, Spain. White J (Forthcoming) How to evaluate Machine Translation. In Somers H (ed), Machine translation: a handbook for translators. Amsterdam, Benjamins. O 200

Detecting gender-preferential patterns of linguistic features 
in face-to-face communication 
Jens Fauth and Hans-Jörg Schmid, University of Bayreuth, Germany 
Gender-preferential language is a research area with a long tradition. Most of the previous investigations, however, included linguistic features that are not suited for large-scale automatic frequency counts in computerized corpora, because they are semantic or pragmatic in nature rather than grammatical, and/or difficult to operationalise (e.g. references to emotions). The present paper sets out to discover previously unknown gender-related linguistic features that are retrievable automatically from corpora. 
Although the present study chooses a grammatical approach similar to Biber's (1988), it differs in that it does not only look for features that have already been suspected to bear a significant relation to gender but also for features that were not determined a priori in this respect. All in all 45 linguistic features from various grammatical domains were taken into consideration. Their occurrences were retrieved from the subcorpus direct conversation of ICE-GB and counted. Only conversations between same-sex participants were investigated because this constellation is known to enhance gender differences (for a complementary study on mixed-sex conversations see the abstract for this conference by Schmid and Fauth). In a second step the Pearson Correlation Coefficient was used to determine significant correlations between gender and linguistic features. The results were then used to predict the gender of the speakers of a conversation by computing a score for each text based on the correlating features. Basic descriptive statistics such as mean, standard deviation and z-scores were used for this purpose. 
The results show that four linguistic features are correlated with female gender: third person personal pronouns, indefinite pronouns, predicative adjectives and intensive adverbs. Male gender is correlated with three different features: the definite article, nominalizations and NP postmodifications realized by of-PPs. These features are interpreted with regard to their communicative function and the extent to which their use reflects gender-specific topic choices. The methodology allows us to predict the gender of speakers in same-sex conversations with a probability of 88.10% for female texts and 85.80% for male texts. 
Jens Fauth Professor Hans-Jörg Schmid 
Chair of English Linguistics Chair of English Linguistics 
D-95440 Bayreuth D-95440 Bayreuth 
E-mail: jot.fauth@gmx.net Hans-Joerg.Schmid@uni-bayreuth.de 
Fax: +49 921 553627 Fax: +49 921 553627 
211

Analysis of the rhetorical structure of computer science abstracts in Portuguese 
Valéria D. Feltrim 
Sandra M. Aluísio 
Maria das Graças V. Nunes 
NILC – Computational Linguistics Group 
ICMC – University of Sao Paulo 
C.P.668 Sao Carlos, SP, Brazil 
{vfeltrim|sandra|gracan}@icmc.usp.br 
1. Introduction 
It is widely acknowledged that academic writing is a complex task even for native speakers, since it involves the complexities of the writing process as well as those specific to the academic genre1 (Sharples and Pemberton 1992). To make it worse, especially to novice writers, often the demands of the academic genre are not clear enough. A number of writing tools have been described in the literature whose ultimate goal is to improve the quality of academic English texts produced by novice and/or non-native writers, e.g. WE (Smith and Lansman 1988), Writer's Assistant (Sharples et al. 1994), Composer (Pemberton et al. 1996), Academic Writer (Broady and Shurville 2000), Abstract Helper (Narita 2000), and Amadeus (Aluísio et al. 2001). These tools provide help during different stages of the writing process, from the generation and organization of ideas to post-processing tasks. For Portuguese, on the other hand, one can only find post-processing tools, such as spell-checkers, grammar checkers and online dictionaries and thesauri. 
With the ultimate goal of aiding academic writing, the project SciPo, currently being developed at NILC2, aims at analyzing the rhetoric structure of Portuguese academic texts . in terms of schematic structure, rhetorical strategies and lexical patterns . to derive models for supporting the creation and evaluation of computational writing tools. To make the project feasible, the analysis has focused on specific sections of theses in Computer Science . abstract, introduction and conclusion, which are the most studied in the literature (Swales 1990, Weissberg 1990, Liddy 1991, Santos 1996) and also have been pointed out as the most difficult ones in a questionnaire applied to graduated students. The choice for this kind of text was made mainly for three reasons: theses having to be written in Portuguese, unlike research articles, which are preferably written in English; the high standardization of Computer Science texts, as in other scientific research areas; and SciPo's developers’ familiarity with the Computer Science domain. 
A corpus of fifty-two (52) Computer Science theses was compiled and has been used both in the analysis of writing patterns specific to the focused research community and in the identification of their main writing problems. The whole corpus analysis is being carried out in a set of stages, the first of which has been accomplished, namely the analysis of the abstracts. This paper presents the results of this first stage and discusses the features of abstracts at a macro level of textual organization as well as comments on its annotation process. 
The next sections present the methodology used for the annotation and analysis of the abstracts (Section 2) as well as the results obtained (Section 3). Finally, features for a writing tool to assist novice academic writers are put forth based on what has been observed in the corpus (Section 4). 
2. Corpus annotation 
The annotation of the corpus was carried out manually in each of the 52 abstracts, consisting of three levels of analysis: identification of structural components, identification of rhetorical strategies used for the realization of each component, and identification of lexical patterns (similar to the “formulaic expressions” used by Teufel et al. 1999) that can be used as lexical clues of the argumentative role of a sentence. These three levels were focused because they reflect the levels of help that we intend to give in a writing tool, namely: how to organize the text (structural components), how to realize each part of 
1 We call “academic genre” the one employed in published academic works (papers, theses, technical reports, etc) of a research community. 
2 NILC – a Brazilian Computational Linguistics Research Group (www.nilc.icmc.usp.br) 
212
the text (rhetorical strategies) and what are the common expressions used in each part of the text according to the selected strategies (lexical patterns). 
In order to render the annotation more reliable, we had four different human annotators annotate our whole corpus of abstracts. Given a scheme of 6 structural components, with 3 rhetorical strategies each, the annotators were asked to identify text fragments corresponding to those components and strategies, not leaving unclassified fragments. We actually had to perform the annotation of our corpus several times, since the first experiments showed low agreement between annotators. This may have been caused mainly by four factors: (1) the initial inadequacy of the annotation scheme, (2) the lack of familiarity of the annotators with the annotation scheme, (3) the nature of our corpus and (4) the subjective nature of this kind of annotation. 
Although the first versions of the annotation scheme had been based on models accepted in the literature, like Swales's model, the corpus presented some peculiarities that did not fit in the model. Additionally, the actual meaning of its elements was not thoroughly stated, making room for misinterpretation. Finally, as the annotators were initially not very familiar with the annotation scheme, disagreement between them followed naturally. Another factor of difficulty was the nature of our corpus. Although it is composed of academic theses, the texts probably were not submitted to such rigorous revision as published articles usually are. Many abstracts even presented passages that could not be classified within any category. This made the annotation process still more difficult and also required a critical view from the annotators. In addition, working with Portuguese brought the difficulty of dealing with long sentences. The average number of words per sentence in our corpus is 26.8 while in a corpus of English abstracts3 it is 23. So, it is common to find different components of the annotation scheme in one sentence, sometimes mixed. The mixed components found in the corpus will be discussed in Section 3, and the major writing problems observed will be described in Section 4. 
1 Setting 
S1 Arguing about the topic prominence 
S2 Familiarizing terms, objects, or processes 
S3 Introducing the research topic from the research area 
2 Gap 
G1 Citing problems/difficulties 
G2 Citing needs/requirements 
G3 Citing the absence of previous research 
3 Purpose 
P1 Indicating the main purpose 
P2 Specifying the purpose 
P3 Introducing more purposes 
4 Methodology 
M1 Listing criteria or conditions 
M2 Citing/Describing materials and methods 
M3 Justifying choices for methods and materials 
5 Results 
R1 Describing the artefact 
R2 Presenting/Indicating results 
R3 Commenting/discussing on the results 
6 Conclusion 
C1 Presenting conclusions 
C2 Presenting contributions/value of research 
C3 Presenting recommendation 
Figure 1. Overview of the designed annotation scheme 
As we could not change the nature of our corpus, we tried to minimize the subjectivity of the task by focusing on better defining the annotation scheme. Since the annotators took part in this process, they automatically became more familiar with the scheme and likely to master it and agree on its usage. After these adjustments, we managed to reach a stable scheme and higher inter-annotator consistency. As a starting point, the annotation scheme was derived from three models: Swales’ CARS (1990) and those by Weissberg (1990) and by Aluísio and Oliveira Jr (1996), the latter modeling introductions of experimental research papers written in English. Although these works deal with 
3 In order to calculate this average, we took 54 abstracts from the Computation and Language (cmp-lg) corpus. The cmp-lg corpus is composed by scientific papers which appeared in Association for Computational Linguistics (ACL) sponsored conferences. 
213
introduction sections, the basic structure of their models could also be applied to abstracts. So, during the first annotation experiment, the scheme was modified to accommodate all the argumentative roles found in the corpus. The components and rhetorical strategies which compose our annotation scheme are similar to the ones presented by those authors, especially to Aluísio and Oliveira Jr's. The final version of our annotation scheme is presented in Figure 1. 
Such similarity had been expected and shows that, despite the heterogeneity of the corpus, it exhibits predictable rhetorical patterns of scientific argumentation. The major deviation from the traditional structure pattern was found in sentences reporting results. Many of them focus on the resulting product (mainly computational systems) instead of on the corroboration of the initial hypotheses of the underlying research issue. To accommodate this particularity, we included the new rhetorical strategy Describing the artifact. We believe that this “non-standard” strategy to report results stems from the technological nature of Computer Science, in which it is common to emphasize the artifact (i.e. a piece of software or a method) developed during the research. 
3. What has been found in the corpus 
The texts in our corpus were collected from online theses repositories and date from 1994 to 2000, comprising 49 MSc theses and three PhD theses, most of them written by students from our Computer Science Department. Only 2 texts were written by students from other Brazilian universities. Texts in the corpus span several Computer Science sub-areas. So, we divided them into 7 research topics: database systems, computational intelligence, software engineering, hypermedia, digital systems, distributed systems, and graphical computation and image processing. Figure 2 presents the number of abstracts classified in each research topic as well as the total and average number of words per abstract. 
Research Topic Abstracts Number of words Total Average Database Systems 3 1,006 335.3 Computational Intelligence 7 1,452 207.4 Software Engineering 16 3,202 200.1 Hypermedia 12 2,097 174.7 Digital Systems 1 171 171 Distributed Systems 12 1,962 163.5 Graphical Computational and Image Processing 1 94 94 Total 52 9,984 192 
Figure 2. Corpus distribution across Computer Science research topics 
Considering the components of the annotation scheme presented in the previous section (Figure 1), 96.2% of the abstracts are classified as strict subsets of that scheme, with some repetitions of components. Only 3.8% of the abstracts contain all the six components, in one of the following sequences: [S G P M C R C]4 or [S G P C M R], neither being in the order recommended by the scheme. Considering the number of different components observed in each abstract (Figure 3), 50% of the abstracts present 5-4 components, 44.3% present 3-2 components and 1.9% present only one component (Purpose). As mentioned earlier, repetition of components is very common in the corpus and some patterns5 have been identified. The most frequent pattern is the repetition of Setting followed by Gap, i.e. (SG)+, present in 25% of the abstracts. Others repetition patterns have been observed, such as Methodology followed by Result, i.e. (MR)+, and Result followed by Conclusion, i.e. (RC)+, but with much lower frequency. 
Number of components Frequency 6 3.8% 5-4 50% 3-2 44.3% 1 1.9% 
Figure 3. Numbers of components per abstract 
4 In this section, the letters S, G, P, M, R, C stand for the components of the annotation scheme: Setting, Gap, Purpose, Methodology, Result and Conclusion. 
5 We use regular expressions to represent repetition and structure patterns. 
214
Regarding ordering of components, we also observed some patterns, especially involving the components at the beginning of the abstracts. The pattern Setting followed by Gap, with repetition or not, followed by Purpose, i.e. ((SG)+|(GS))P, appears in 30.7% of the corpus. Instances6 of this pattern are [S G S G P R] and [G S P R M]. Another frequent pattern is Setting followed by Purpose, followed or not by Methodology or Result, i.e. SP[M|R], which is observed in 21.1% of the corpus. Instances are [S P], [S P R C] and [S P M R]. The pattern Purpose followed by Result, with repetition or not, followed or not by Methodology or Conclusion, i.e. (PR)+[M|C], and the pattern Purpose followed by Methodology, followed or not by other Purpose or Result or Setting PM[P|R|S] appear, respectively, in 19.2% and 13.4% of the abstracts. Other combinations of components appear in the corpus with a very low frequency, so they were not considered here. A summary of the most frequent patterns is presented in Figure 4. 
Pattern Frequency ((SG)+|(GS))P 30.7% SP(M|R)* 21.1% (PR)+[M|C] 19.2% PM[P|R|S] 13.4% 
Figure 4. Observed patterns frequency 
The ordering patterns identified show that the order adopted in our annotation scheme is reasonable, since 51.8% of the corpus start with components Setting or Gap followed by Purpose, despite order varying greatly from middle to final components. In fact, we observed a lack of clarity when analysing these components, especially Methodology and Result. They usually occur mixed with each other or, even more often, with Purpose. The frequency of these cases is shown in Figure 5. The most frequent case of mixing occurs between Result and Purpose, which appears in 38.5% of the corpus. This means that, in 38.5% of the abstracts, the passage annotated as Indicating the main purpose also includes traces of Result. Also, in 9.6% of the corpus, Purpose presents traces of both Result and Methodology. The high frequency of Result mixed with Purpose may be viewed as another evidence of the importance given to the resulting product of the research – the artifact – as previously noticed in Section 2. We believe that such importance is the reason why writers focus on the produced artifact inside Purpose, since it is a central component with high relevance for the abstract. Still considering the observed patterns, we noticed that many texts present not only similar structure but even the same lexical patterns, mainly when written by students who had the same advisor. Such similarity was also observed in texts written by students from the same research group. This suggests that students tend to use texts produced inside their work group as models when writing their own texts and also shows the great influence of the advisor on the student text. 
Mixed Components Frequency Result mixed with Purpose 38.5% Methodology mixed with Purpose 19.2% Result and Methodology with Purpose 9.6% Methodology mixed with Result 7.7% 
Figure 5. Frequency of attached components 
The frequency of each component of the annotation scheme in the corpus is presented in Figure 6. The numbers show lower frequencies for Setting, Gap and Conclusion in contrast with components Purpose, Methodology and Result. This is in agreement with the results reported by Santos (1996) and Motta-Roth and Hendes (1998). These authors have also observed the optional character of using initial and final components in abstracts and a tendency to a more frequent use of middle components, especially Purpose, which appears in all the analysed abstracts. This strongly suggests that writers see Purpose as the most important kind of information that an abstract should provide to the reader, followed by Result and Methodology. Although, the frequency of Setting in our corpus also suggests writers’ concern with contextualizing their research, which was not observed by Motta-Roth and Hendes. We believe that this difference is due to the nature of the two corpora. As our corpus is composed by thesis abstracts, the writers were not faced with the constraint of a limit of words, which generally hold for journal abstracts, and so they are free to make extensive contextualization. 
6 All the presented structure examples are taken from the corpus. 
215
Component Frequency Setting 55.7% Gap 42.3% Purpose 100% Methodology 63.4% Result 67.3% Conclusion 30.7% 
Figure 6. Distribution of components 
Another aspect related to the nature of our corpus is the high frequency of the rhetorical strategy Presenting contributions/value of research. As can be viewed in Figure 6, component Conclusion occurs in 30.7% of the corpus. Of these occurrences, 68.8% use the referred rhetorical strategy (or 21.2% of the corpus occurrences). Considering that is very important in MSc/PhD theses to emphasize the contributions to a research area, it seems natural for writers to use that conclusion strategy in the abstracts with some frequency. 
4. Observed writing problems 
In this paper, when we refer to writing problems we mean deviations from the traditional structure model that can compromise the communicative objective of the abstract. As was previously commented in Section 2, the texts in our corpus probably have not been through such rigorous revision, even though the texts are MSc and PhD theses. So, it is not surprising that the abstracts presented some problems, not only general, like grammatical mistakes, but also specific to the academic genre. Moreover, as the major part of the corpus is MSc theses, we can also attribute the observed problems to the inexperience of the writers. We will not comment on the superficial problems found in the corpus, since these are not our object of study in this paper. Thus, we will only focus on problems that are specific to the academic genre. 
The major problems observed in the abstracts were misuse of lexical patterns and verbal tenses, inefficient organization and inappropriate emphasis on some specific components. We regard as misuse of lexical patterns those cases in which the writer uses expressions in a component which are specific to other components. An example is the use of however, which is a lexical clue to component Gap, in a Setting sentence and not with its correct contrasting value. Other examples are expressions of the type “A and B are also presented.”, which can be seen as an indication of the rhetorical strategy Introducing more purposes, used in sentences reporting results in an indicative way. Furthermore, the writers sometimes do not use the appropriate verbal tense, especially in sentences reporting results. It is recommended by the literature (e.g. Weissberg 1990) to use the past tense when reporting findings. This gives the reader the notion that the results were obtained by the reported research and not in a previous one. However, it is common to find sentences in our corpus reporting findings in the present tense. Due to this, sometimes we could not determine quite whether the writer was reporting his own results or commenting on someone else's results (as in literature review). Misuse of lexical patterns and verbal tenses surely demanded great effort from the annotators when it came to interpreting the texts to identify their components. 
Regarding inefficient organization, we observed some abstracts in which the writer mixed components in such a way as to confuse readers. An example of this problem can be observed in the structure [P M S G P]. The writer initiated by indicating the main purpose (first P) and then described the methodology used to accomplish that purpose (M). After that, a more natural move would be to present results; however, the writer used a Setting component, followed by a Gap (SG), in order to lead the reader into the further detailing of the previously stated purpose and the introduction of yet other purposes. The presence of Setting and Gap in the middle of the abstract, separating the main purpose from its detailing, confuses the reader, who may lose track of the main purpose of the related research. Also, the sequence Methodology-Setting disrupts the cohesion of the text, causing the reader to feel that “something is missing”. 
A different case of inefficient organization is inappropriate emphasis on some components. In our corpus such emphasis was observed mainly in contextualization (Setting + Gap). One such example is an abstract that has the structure [S G S G P] and 160 words. A Purpose component, which also has traces of Result and Methodology, corresponds to 16.9% of the abstract. The rest of it (83.1%) is dedicated to the contextualization of the problem, (SG)+. Taking into account that these writers had no limit of words to write their abstracts, we can consider such an abstract little informative. It would be 
216
desirable to find more information about the reported research in addition to such a relatively extensive contextualization. 
Though less frequent, three further problems were found in the corpus, namely: (1) passages that could not be classified as a component of the annotation scheme. This also took the annotators some time, as previously commented in Section 2; (2) passages indicating obvious information, like “this work also presents a literature review about the focused subject”. It is quite sure that an MSc/PhD thesis will have a literature review on the main focused subjects, fairly obviating that piece of information; (3) exactly one abstract presenting the outline of the theses. This kind of information is common in introductions, but not in abstracts. 
4. Conclusions 
In this paper we have reported on the annotation and analysis of a corpus of thesis abstracts in Computer Science, based on an annotation scheme designed specially for this project. We have argued that this scheme is stable and reasonable on the basis of the results of the corpus analysis. Furthermore, we have discussed structuring patterns and some particularities of the corpus, such as repetition of components and levels of relevance given to each component, and identified writing problems, which are deviations from what is prescribed as characteristic of a good academic abstract. 
As we mentioned in the introduction, the presented corpus analysis is part of a project which aims at deriving models for computational writing tools specific to the academic genre in Portuguese. Based on what was observed in the corpus, including its problems, we are developing computational tools for aiding especially novice writers. These tools are being implemented as Web-based applications featuring (1) prescriptive guidelines related to the academic genre and (2) repositories of good and bad examples of structure, writing strategies and lexical patterns. The example repository will have abstracts, introductions and conclusions. Another tool is meant to be a critiquing system capable of giving advice/criticism on structure organization, verbal tenses and level of emphasis given to each component. As a result, we expect to help writers overcome their difficulties related to the academic genre. As future work, we intend to extend our analysis to introductions and conclusions, implement stable prototypes of these tools, and evaluate them with real users, e.g. graduate Computer Science students. 
6. References 
Aluísio S M, Barcelos I, Sampaio J, Oliveira Jr O 2001 How to learn the many unwritten "Rules of the Game" of the Academic Discourse: A hybrid Approach based on Critiques and Cases. In Proceedings of the IEEE International Conference on Advanced Learning Technologies, Madison/Wisconsin, pp 257-260. 
Aluisio S M, Oliveira Jr. O N 1996 A Detailed Schematic Structure of Research Papers Introductions: An Application in Support-Writing Tools. Revista de la Sociedad Espanyola para el Procesamiento del Lenguaje Natural, 19: 141-147. Also available in <http://www.cica.es/sepln96/sepln96.html> 
Broady E, Shurville S 2000 Developing Academic Writer: Designing a Writing Environment for Novice Academic Writers. In E. Broady (ed.), Second Language Writing in a Computer Environment, CILT, London, pp 131-151. 
Liddy E D 1991 The Discourse-Level Structure of Empirical Abstracts: An Exploratory Study. Information Processing & Management, 27(1): 55-81. 
Motta-Roth D, Hendges G 1998 Uma Análise Transdiciplinar do Genero Abstract. Revista Intercâmbio, VII: 125-134. 
Narita M 2000 Corpus-based English Language Assistant to Japanese Software Engineers. In Proceedings of MT-2000 Machine Translation and Multilingual Applications in the New Millennium. pp 24-1 – 24-8. 
Pemberton L, Shurville S, Hartley A 1996 Motivating the Design of a Computer Assisted Environment for Writers in a Second Language. In Proceedings of CALICE'96, pp 141-148. 
Santos M B 1996 The textual organization of research paper abstracts. Text 16(4): 481-99. 
Sharples M, Pemberton L 1992 Representing writing: external representations and the writing process. In P.O. Holt and N. Williams (eds.) Computers and Writing: State of the Art. Intellect, Oxford, pp 319-336. 
217
Sharples M, Goodlet J, Clutterbuck A 1994 A comparison of algorithms for hypertext notes network linearization. International Journal of Human-Computer Studies 40(4): 727-752. 
Smith J B, Lansman M 1988 A Cognitive Basis for a Computer Writing Environment. Technical Report, n.87-032, Chapel Hill. 
Swales J M 1990 Genre Analysis: English in Academic and Research Settings. Cambridge applied linguistics series. 
Teufel S, Carletta J, Moens M 1999 An annotation scheme for discourse-level argumentation in research articles. In Proceedings of EACL 1999. 
Weissberg R, Buker S 1990 Writing up Research: Experimental Research Report Writing for Students of English. Prentice Hall. 
218

Updating LSP dictionaries with collocational information 
Katerina T. Frantzi 
Department of Mediterranean Studies, 
University of the Aegean 
Dimokratias 1, 85100, Rhodes, Greece 
frantzi@rhodes.aegean.gr 
Abstract 
Despite the big amount of general language dictionaries in electronic form, those coming from specialised areas are still “under construction”. There are two main reasons for this: firstly, the need for these dictionaries was/is less essential than the need for general language dictionaries, since these were/are aiming mainly to specialists, and secondly many specialised areas are changing over time, resulting to dictionaries that need continuing updating. Due to this, techniques that improve the automatic or semi-automatic construction and updating of specialised dictionaries are and will always be welcome. 
In this work we are concerned with the updating of dictionaries for Languages for Special Purposes (LSPs) with information coming from collocations. The collocations to be used are extracted from LSP corpora of not necessarily big size. 
1 Introduction – collocations 
The big number of applications for collocations (dictionary construction, translation, language learning, etc.), makes them an interesting area to work on. The availability of corpora in electronic form has given a great deal of help to this kind of research since we are now able to work with real data. English is not any more the only language with electronic corpora, though it owns the greatest deal. Also, although most electronic corpora describe the general language, corpora of languages for special purposes (LSPs) become more and more available. 
Firth, (Palmer 1968), introduced the meaning of a collocation when discussing about senses. He suggested that part of the sense of a word depends on its neighbour words in texts: “You shall know a word by the company it keeps”, (Palmer 1968:179). This “company” is what he named collocation, and kept it very important for understanding words. 
It is quite some time now that linguists have shown interest in collocations (Jones and Sinclair 1974), and various definitions have been given. Some allow collocations to only consist of two words, while others of much more. Some care about what information collocations can give us on semantics, others on syntax or grammar. Some accept common words, others not. Some allow collocations to cross a comma, others not. Regarding interrupted collocations, there are differences as for the size of the gap(s) among the collocates. Despite all the differences, collocations are arbitrary, recurrent and cohesive lexical clusters, and depend on the language (Smadja 1993). We adopt the collocation definition given by Sinclair and Carter agreeing for a collocation to be the occurrence of two or more words within a short space of each other in a text (Sinclair and Carter 1991). 
As mentioned above, collocations depend on the language and sublanguage they are found. They actually play an important role in sublanguages (Frawley 1988; Ananiadou and McNaught 1995). The study of collocations in general language needs large corpora since phenomena in general language are sparse: in the Brown Corpus we only have two instances of “cups of coffee”, five of “for good” and seven of “as always” (Kjellmer 1994). However, when we talk about LSPs, things are easier as for the size of the corpus which can be a lot smaller since information there is dense. 
Early work on collocation extraction was determinant. Choueka et al. were among the first to use frequency of occurrence for recognising collocations (Choueka et al 1983). The work of Nagao and Mori was also 
219
based on frequency of occurrence but they also considered the length of collocations to be extracted, giving priority to longer ones (Nagao and Mori 1994). Church and Hanks were the first to use association ratio (Church and Hanks 1990), a measure based on mutual information first expressed by Fano (Fano 1961). They cared about the semantic relations of the word-pairs they recognised, which could be interrupted by other words. On mutual information is based the work of Kim and Cho (Kim and Cho 1993), which extent it to three words, but in a different way than that originally defined by Fano. Collocation extraction is still an interesting issue for researchers (Kilgarriff and Tugwell 2001; Kim et al. 2001). 
Collocations can be divided to those that do not appear as part of other longer collocations and those that they do. The latter we call nested collocations. For example, in Computational Linguistics, “Natural Language” is a collocation itself, but is also part of the longer collocation “Natural Language Processing”. Three important works that mention the problem of nested collocations are those of Smadja, Kita et al., and Ikehara et al. Xtract, based on frequency of occurrence, recognised as collocations only those expressions of the greatest length (Smadja 1993). It did not extract collocations that were part of others. The work of Ikehara et al., which was based on Nagao and Mori's work, only accepted those that were found with satisfying frequency as not-nested (Ikehara et al. 1995). The problem of nested collocations was a big a concern for Kita et al. These accepted a nested collocation when it also appeared as not-nested with satisfying frequency (Kita et al. 1994). 
2 Updating the dictionary 
We deal with the updating of LSP dictionaries for the Greek language. We use nested collocations to get the information in a way easier than looking directly into the corpus, which can be very time consuming. C-value is used for the extraction of collocations from LSP corpora. C-value has been initially constructed and used for the extraction of English collocations (Frantzi et al. 2000). It has been also applied to Japanese language (Mima et al. 2001). In this work we will be using it for Greek collocations and the updating of Greek dictionaries. 
Let us remind that C-value pays particular attention to nested collocations. When applied to the “Artillery Firing Military Rule Book” (“Óôñáôéùôéêüò .......µüò ....ß........ .....”, the corpus we will be using) one of the collocations it extracts is the “éïñèþóåéò .. .... .. ...µµÞ ß....”. So it does to “ãñáµµÞ ß....” which is a nested to the previous one collocation, but also stands as a collocation by itself. We need such a method since we will use nested collocations to get the information for updating the dictionary. 
When C-value is applied to an expression, it considers the following parameters: 
1. The length of the expression (in terms of number of words). The longer the expression, the more important. 
2. The frequency of occurrence of the expression in the corpus. The bigger the frequency the more important the expression. 
3. Whether the expression appears as nested, and if yes the number of the different longer collocations that contain it. The number of times it is found in these longer collocations is also a considered parameter. 
Let us remind that C-value is evaluated as follows (a is the expression we examine): 
1. C-(value)(a)=0 
if the expression is part of one longer collocation and its frequency of occurrence is the same as this longer collocation's frequency. In this case the examined expression is not a collocation by itself. 
2. C-value(a)= (|a| - 1)n(a) 
if the expression is not part of any longer collocations. 
|a| is the size of the expression a in terms of number of words, 
220
n(a) is the frequency of occurrence of the expression a in the corpus. 
3. C-value(a)= (|a| - 1)(n (a) – t(a)/c(a)) 
if the expression is part of longer (more than one) collocations. 
c(a) is the number of these longer collocations that include the expression a, 
t(a) is the total frequency of the expression a as part of these longer collocations. 
After extracting the collocations we group them and choose a group to start with. Attention should be taken when grouping the collocations. If for example we only group them alphabetically based on the first word, then we could miss out members of the group and as a result possibly useful information. Which group of collocations to start with is up to the application. A group of collocations that would be used to update the dictionary could be the following: 
ðáñÜããåëµá ß.... 
...... ........µá ß.... 
...... ........µá ß.... µïßñáò 
...... ........µá ß.... ....ß........ 
...... ........µá .µåóçò ß.... 
...... .......... ........µáôïò ß.... 
........ ....... ........µáôïò ß.... 
...... ........µá ß.... ...... ....µïý 
...... ........µá ß.... ......... ....µïý 
...... ........µá ß.... ......... ....µïý 
...... ........µá .µåóçò ß.... ....ß........ 
...... ........µá .µåóçò ß.... ...... ....µïý 
...... ........µá .µåóçò ß.... ......... ....µïý 
...... ........µá .µåóçò ß.... ......... ....µïý 
The algorithm for updating the dictionary is the following: 
L: existing LSP dictionary; 
entry_L(.): an entry in L; 
Extract collocations from the LSP corpus using C-value; 
Group collocations creating collocation_groups; 
for each collocation_group cg from collocation_groups 
for each collocation c in cg 
length(c)= number of words of c 
max_length = max(length(c)) where collocation c in cg 
new_c = collocation c in cg with length(c)=min(length(.)) 
if entry_L(new_c) = 0 
create entry_L(new_c) 
info_length = length(new_c) 
while info_length < max_length + 1 
for each collocation c from cg with info_length = length(new_c)+1 
check c for new information 
update entry_L(new_c) 
end_for 
221
info_length = info_length +1 
end_while 
end_for 
The choice of C-value as the method for extracting the collocations is critical since it deals with nested collocations, the type of collocations we need for getting the information. Let us now assume the following imaginative group of collocations from the collocation list: 
a b 
a b c 
a b d 
a b f e 
a b c g 
a b f g h 
where a b c d e f g h words. 
We take the collocation of the smallest length. In our example the “a b”. If the collocation “a b” does not yet exist in the lexicon a new entry is created. Now we consider the collocations of length the next smallest, (in terms of number of words). In our case the “a b c” and the “a b d”. We can start with “a b c” considering the word “c” in terms of the information it can give us on grammar, syntax or semantics (depending on the type of dictionary we want to update). We continue with “a b d” and the grammatical, syntactical or semantical information that the word “d” gives for collocation “a b”. Then we move to collocations (of the same group always) of the next smallest length, that is the “a b f e” and the “a b c g”. We do the work we did before, so we consider “f e” as for the information it can give for the collocation “a b”. For the collocation “a b c g” we consider the fact that “a b c” is a nested collocation we have already checked and add the information given by word “g”, and of course any new information acquired by the word combination “c g”. We finish with the collocation “a b f g h”, where we take information from the word combination “f g h”. 
When a collocation group is over we can move to the next collocation group. 
The method is semi-automatic since the machine, the domain expert and lexicographer need to cooperate. The human factor is necessary for the evaluation of information coming from the collocation under consideration. It is the domain expert and the lexicographer to judge which information is useful to be used and which not. 
3 Application 
The method is applied to the “Artillery Firing Military Rule Book” (“Óôñáôéùôéêüò .......µüò ....ß........ .....”) of about 35,000 words. Since we are working with an LSP corpus we can use a small corpus. With a general language corpus things would be a lot harder in terms of its size since phenomena in that case are sparse. No tagging has been applied on the corpus. The implementation was done in Linux. Table 1 shows a sample of it. 
At first, collocation extraction is taking place using C-value. In this application we extract expressions of 2 to 7 words. This is a variable and changes according to application. The extracted collocations are ordered according to their C-value. A threshold can be applied to only allow those expressions above a value to be extracted and therefore proceed to the next stage. A threshold could have also been applied to the frequency of occurrence of the candidate expressions. 
Table 2 shows a sample of the list with the extracted collocations. The first column gives the C-value for the expression shown on fifth column. The fourth column gives the frequency of occurrence of the expression. The third column gives the number of (longer) expressions that contain the current expression while the second the total frequency of the expression in these longer ones. Expressions on Table 2 have 
222
been chosen such that differences between C-value and frequency of occurrence can be noticed. We can see for example that long expressions despite their low frequency are valued high by C-value, e.g. “óå .......... .. .... .. ...µµÞ ß....”, and “ôï ..µéóµá ..... ... µå .. ß........”. Those expressions are domain-dependent, and for that they are (correctly) valued high. On the contrary expressions such “êáé ...” (“and to”), “ôç .....” (“the angle”), and “áõôü .....” (“this is”), are valued more by pure frequency of occurrence. 
ÁÐÏÔÅËÅÓÌÁÔÉÊÏÔÇÔÁ ... ........... ..... 
1. Óõíåñãáóßá ... ... ........ ..... ....ß....... 
Ôï ...ß..µá ... ............ ... ..... µéáò ....... ....µïý ......... µå ... ........µÝíåò ........... ... .........., ôïõ ....... éåõèýíóåùò ..... (¥ÊÐ) êáé ... ....ß........ ..... 
(Ó÷. 1). Ôá .... .... .µÞµáôá ... ....ß......, ðñÝðåé .. ..... .......µÝíá µå ....... ...... ............. Ôï ...... ...µá ....... .. ........ µå ........ ... .. ....ß...... ....... ........... µåéþóåùò ... .......µåíïõ ......, ãéá ... ........µáôéêÞ ........ µéáò ......... ß..... 
á. ÐáñáôçñçôÞò. 
Ï ........... ..... «ôá µÜôéá» ôïõ ....ß...... ...... ÁíáæçôÜ ... ............ .. .... .......... ... .. ....ß..... ......, µÝóá ... .... ............ .... Ãéá .. ....ß..... ... ....., äéáâéâÜæåé ... ...... ß.... ... .... .......... ....... .......µü ... ß..... Åðéôçñåß .. .... ... ... ....... ........ ... ¥ÊÐ. 
â. ÊÝíôñï éåõèýíóåùò ...... 
Ôï ¥ÊÐ ........ ... «åãêÝöáëï» ôïõ ....ß....... ËáµâÜíåé ... ...... ß.... ... .........., ðñïóäéïñßæåé ........ ß.... ... .. µåôáôñÝðåé .. ........µáôá ß...., ôá ..... ...ß.ß.... ... ....ß.... Åêôåëåß ...... ... ....... ......... ... ...... Ëüãù ... µåãÜëùí .......... µåôáîý ... µïíÜäùí ..... (ðõñïâïëáñ÷éþí) êáé ... .......... ... ... ...... ...... ..... ............, ç ....... ......... ... ..... .......... ....... ... ¥ÊÐ ... ....ß......... Ôï ¥ÊÐ ...... ....... ....... ......... ... ..... (ôñüðï ....ß.... ... ......) êáé ............ ... .. ...... ß..... ÅðéðëÝïí ß.... .. ¥ÊÐ ... ....ß........ .... ....... ......... ... ....., ðáñÝ÷ïíôáò .. .... ........ ß.... ... .. ...... ..... ... .......... ... ........ ¥ÊÐ, üôáí .......... 
Ó÷Þµá 1. Óõíåñãáóßá ... ... ........ ..... ....ß....... 
ã. Ðõñïâïëáñ÷ßá ...... 
Table 1 Sample of the corpus. 
Table 3 shows how the method behaves with nested expressions. We can see that, if instead of C-value we used frequency of occurrence, and in order to give a value to a candidate expression we were subtracting from its frequency the summation of its frequency when part of longer expressions, we would underestimate quite a few important expressions. 
The extracted list is expected to contain “useless” expressions, like “êáé ...” (“and to”) or “áõôü .....” (“this is”). However according to Kjellmer no extracted expression can easily -if at all- be characterised “useless” (Kjellmer 1994). His dictionary of English collocations incorporates everything that has been extracted with no characterisation as “correct” or “wrong”. However, we could use a part-of-speech tagger to only allow expressions of a particular form. This way we would eliminate some expressions we do not want but could also lose some we do. What we do depends on the application. 
223
C-value(a) t(a) c(a) f(a) extracted collocation 
8.42206 0 0 3 ôï ..µéóµá ..... ... µå .. ß........ 
8.42206 0 0 3 óå .......... .. .... .. ...µµÞ ß.... 
8.42206 0 0 3 ï ........... ............ .. .... ... ...... 
8.42206 0 0 3 ï ........... ß..... .. ........ ... ß..µáôïò 
8.42206 0 0 3 µå .. ....... ... µåôñÜ .. ....... 
8.42206 0 0 3 Ì10/Ì17 ç ........ .. .... .. ...µµÞ 
8.42206 0 0 3 ßóï µå .. ß........ .. ........ µÝôñùí 
8.42206 0 0 3 ç ........ .. .... .. ...µµÞ ....ß... 
8.42206 0 0 3 ç ..... ....... ..µåßïõ ... ...... ..... 
8.42206 0 0 3 ãéá ... ........ ..' åõèåßáò ......... ß.... 
8.42206 0 0 3 ãݵéóµá ..... ... µå .. ß........ .. 
8.42206 0 0 3 áðü .. .ß.... .10/Ì17 ç ........ .. 
8.42206 0 0 3 áâÜêéï .10/Ì17 ç ........ .. .... .. 
8.33333 8 3 11 ôç ..... 
8.33333 8 3 11 êáé ... 
7.92481 12 2 11 ï ........... ...... 
7.66667 7 3 10 áõôü ..... 
7.66667 10 3 11 óôç .... 
7.66667 10 3 11 ðáñáôçñçôÞ .. 
6.96578 0 0 3 ï ......... ... µïíÜäáò ....µïý 
6.96578 0 0 3 µå ......... .... ...... ..µåßï 
6.96578 0 0 3 ãéá ... ....ß... ... ...... 
6.96578 0 0 3 ãéá ... ........ ... ......... 
5.61471 0 0 2 õðïëïãéóµüò .......... .... µÞêïò ... ...µµÞò 
............ 
5.61471 0 0 2 ôïí ....ß. ..........µü ... ...... ... ...... 
Table 2 Sample of the list with the extracted collocations. 
Let us now see an example from our corpus, on how we get the information for updating the dictionary. We have already extracted the collocation list. Assume the collocation group we work with, is the following: 
Length: 2 
óôïé÷åßá ß.... 
Length: 3 
õðïëïãéóµüò ......... ß.... 
......... ......... ß.... 
Length: 4 
óôïé÷åßá ß.... ....ß.... ...... 
µÝèïäïò ........µïý ......... ß.... 
Length:5 
óôïé÷åßá ß.... ... .....µáíóç ....ß.... 
Length: 6 
õðïëïãéóµüò ......... ß.... µå ..... PC32F 
õðïëïãéóµüò ......... ß.... µå ..... TI59 
óôïé÷åßá ß.... µå ..... µåôåùñïëïãéêþí ......... 
........µüò ......... ß.... µå ..... laser 
õðïëïãéóµüò ......... ß.... ... ... .......... 
Length: 7 
áíáãùãÞ ......... ß.... .... ...... ............ .......... 
....... ......... ß.... µå ..... .ß..... .17 
Length: 8 
÷ñÞóç ... ......... ... ... ........µü ......... ß.... 
224
The collocation we are dealing with is the “óôïé÷åßá ß....”. Length is taken in terms of number of words. The collocation is met in the corpus under two forms: “óôïé÷åßá ß....” and “óôïé÷åßùí ß....”. We take these two as the same collocation and then we consider the two collocations of Length=3. The domain expert and the lexicographer need to evaluate the information taken from each of the two words “õðïëïãéóµüò” and “êáôáãñáöÞ”. The domain expert has to decide whether the expression “õðïëïãéóµüò ......... ß....” is a collocation or not. If it is, and the information of the word “õðïëïãéóµüò” regarding the “óôïé÷åßùí ß....” does not exist in the dictionary then the dictionary has to be updating by the lexicographer with the information given by the domain expert. The same happens for the other collocation of length=3, the “êáôáãñáöÞ ......... ß....”. When we finish with collocations of Length=3, we move to those of Length=4, in our example the “óôïé÷åßá ß.... ....ß.... ......” and the “µÝèïäïò ........µïý ......... ß....”. The first collocation “óôïé÷åßá ß.... ....ß.... ......” does not directly relate to any of the two collocations of length=3, and so the “ðñïóâïëÞò ......” will be treated by the domain expert and the lexicographer as the words “õðïëïãéóµüò” and “êáôáãñáöÞ” of the previous stage. The collocation “µÝèïäïò ........µïý ......... ß....” will have to update the information of the previously checked collocation “õðïëïãéóµüò ......... ß....”, so the latter will be taken under consideration. The method continues with the same simple way until we reach and use the collocation with the greatest length in the group, in our example the “÷ñÞóç ... ......... ... ... ........µü ......... ß....” with Length=8. 
C-value(a) t(a) c(a) f(a) Extracted collocation 3 4 2 5 ãùíéïµåôñéêü ...... 8.8 11 5 11 äéüñèùóç ß......... 42.7 33 10 46 äñáóôéêÞ ß... 10.2857 12 7 12 åêñçêôéêü ß..µá 10.8 11 5 13 åðéóÞµáíóç ....ß.... 11.8872 10 4 0 åõèåßá ......... ß.... 23 26 13 25 æþíç ............ 49.5 50 20 52 êáôÜ ......... 8.75 10 8 10 ðñþôç ........ 18.7143 23 7 22 ðñþôï ß..µá 11.4118 10 17 12 ðõñïâüëá ...... 18.8571 15 7 21 óçµåßï .......µïý 16.8571 15 7 19 óôïé÷åßá ß.... 17.2222 16 9 19 ýøïõò ......... 5.3333 8 3 8 ÷áñáêôçñéóôéêÜ ..µåßá 
Table 3 Collocations that have been also found as nested. 
The method is quite simple in the way it works. It is semi-automatic in the sense that it needs the domain-expert and the lexicographer. We believe that this is necessary in order to provide accuracy and completeness to a high degree. However the domain expert and the lexicographer do not have to look (unless really needed) on the corpus itself to obtain the information for the LSP dictionary updating, which of course is a considerable gain on time. We have yet not applied an evaluation measure to judge the results as for the correctness and completeness of information gained. This is a subject still to be done. Another matter is the stemmer. It is not easy to decide whether to use one or not. If yes, words having the same thema would count as one (as they should be in most cases). However there are cases where this should not happen, like with the words “ðáñáôçñçôÞò” (“observer”) and “ðáñáôçñçôÝò” (“observers”). These two words in many cases need to stay as they are found, since they are often used to indicate different meanings in different collocations. 
4 Summary 
In this paper we presented the incorporation of the C-value method for the extraction of collocations to dictionary updating. C-value offers to this since it focuses on nested collocations, the type of collocations we look at in order to obtain our dictionary new information. The method makes the process faster since we actually look at the extracted collocation list instead of the whole corpus. It is semi-automatic since the 
225
final decision on which information should update the dictionary is taken by the domain expert and the lexicographer. 
Regarding future work we first are to apply the method to other languages starting with English, but Turkish, Arabic and Hebrew as well. Should things be working as expected, we will move to the application of the method to multilingual corpora (including parallel) for the updating of multilingual dictionaries. 
References 
Ananiadou S, McNaught J 1995 Terms are not alone: term choice and choice terms. Journal of Aslib Proceedings 47(2): 47-60. 
Choueka Y, Klein T Neuwitz E 1983 Automatic retrieval of frequent idiomatic and collocational expressions in a large corpus. Journal of Literary and Linguistic Computing 4:34-38. 
Church K W, Hanks P 1990 Word Association Norms, Mutual Information, and Lexicography. Computational Linguistics 16:16-29. 
Fano R M 1961. In Transmission of Information: a statistical theory of communications, New York, M.I.T. Press. 
Frantzi K T, Ananiadou S, Mima H 2000 Automatic Recognition of Multi-Word Terms. International Journal on Digital Libraries 3(2): 115-130. 
Frawley W 1988 Relational models and metascience. In Evens M (ed) Relational models of the lexicon. Cambridge, Cambridge University Press, pp 335-372. 
Ikehara S, Shirai S, Kawaoka T 1995 Automatic Extraction of Collocations from Very Large Japanese Corpora using N-grams Statistics. Transactions of Information Processing Society of Japan 11: 2584-2596 
Jones S, Sinclair J 1974 English Lexical Collocations: A Study in Computational Linguistics. Cahiers de Lexicologie, 24(1): 15-61. 
Kilgarriff A, Tugwell D 2001 WORD SKETCH: Extraction and Display of Significant Collocations for Lexicography. In Proceedings of Collocation Workshop, ACL 2001, pp 32-38. 
Kim P K, Cho Y K 1993 Indexing Compound Words from Korean Texts using Mutual Information. In Proceedings of Natural Language Pacific Rim Symposium, pp 85-92. 
Kim Y, Zhang  T, Kim Y T 2001 Collocation Dictionary Optimization Using WordNet and k-Nearest Neighdor Learning. Journal of Machine Translation 16: 89-108. 
Kita K, Kato Y, Omoto T, Yano Y 1994 A comparative Study of Automatic Extraction of Collocations from Corpora: Mutual Information vs. Cost Criteria. Journal of Natural Language Processing 1: 21-33. 
Kjellmer G 1994 A dictionary of English Collocations, Oxford: Clarendon Press. 
Mima H, Ananiadou S, Neradic G 2001 The ATRACT Workbench: Automatic Term Recognition and Clustering for Terms. In Lecture Notes in Computer Science, LNAI 2166, Springer-Verlag, pp 126-133. 
Nagao M, and Mori S 1994 A new Method of N-grams Statistics for Lange Number of n and Automatic Extraction of Words and Phrases from Large Text Data of Japanese. In Proceedings of the 14th International Conference on Computational Linguistics, pp 611-615. 
Palmer F R (ed) 1968 Selected papers of J.R. Firth. Harlow: Longman. 
Sinclair J, Carter R (eds) 1991 Corpus, Concordance, Collocation. Oxford, England, Oxford University Press. 
Smadja F 1993 Retrieving Collocations from Text: Xtract. Computational Linguistics, 19:143-177. 
226

Using the XARA XML-Aware Corpus Query Tool 
to Investigate the METER Corpus 
Robert Gaizauskasa, Lou Burnardb, Paul Cloughc and Scott Piaod 
Department of Computer Sciencea 
University of Sheffield, Sheffield, S1 4DP 
R.Gaizauskas@dcs.shef.ac.uk Research Technologies Servicetb 
University of Oxford, Oxford, OX1 2JD 
lou.burnard@oucs.ox.ac.uk Department of Information Studiesc 
University of Sheffield, Sheffield, S1 4DP 
p.d.clough@sheffield.ac.uk Department of Linguistics and Modern English Languaged 
Lancaster University, Lancaster,LA1 4YT 
s.piao@lancaster.ac.uk 
Abstract 
The METER (MEasuring TExt Reuse) corpus is a corpus designed to support the study and analysis of journalistic text reuse. It consists of a set of news stories written by the Press Association (PA), the major UK news agency, and a set of stories about the same news events, as published in various British newspapers, some of which were derived from the PA version and some of which were written independently. The corpus has been annotated in accordance with the TEI Guidelines. The annotations include both descriptive metadata, such as the title, source, and date of publication, and human judgements about text reuse. To exploit the value of the annotations, for searching and querying the corpus, requires a tool which is designed to “understand” XML, or even, ideally, TEI. Such a tool is XARA, which has been built specifically to support corpus investigations over XML-annotated corpora. This paper reports lessons learned in using XARA to explore the METER corpus, particularly the importance of designing the annotation scheme with an understanding of the capabilities and limitations of the retrieval application. 
1. Introduction 
The METER (MEasuring TExt Reuse) corpus is a resource designed to support the study and analysis of journalistic text reuse. It consists of a set of news stories written by the Press Association (PA), the major UK news agency, and a set of stories about the same news events, as published in various British newspapers, some of which were derived from the PA version and some of which were written independently. The corpus has been annotated in accordance with the TEI guidelines. The annotations include both descriptive metadata, such as the title, source, and date of publication, and human judgements about text reuse. These judgements were made at two levels. Firstly, each newspaper story, as a whole, is labeled as wholly derived, partially derived, or not derived from a PA source. Secondly, for certain selected stories, every word in the text is labelled as either reused verbatim from the PA, rewritten from the PA, or new. 
The corpus and the associated annotation scheme have been described elsewhere (Clough et al., 2002; Gaizauskas et al., 2001), and this description will not be repeated, beyond bare essentials, below. The current paper focuses on work we are carrying out to explore the corpus using the newest version of the SARA (SGML Aware Retrieval Application) corpus query tool, called XARA, which was originally designed to support exploration of the British National Corpus. This work has two main purposes: 
(1) To verify and validate the TEI encoding scheme we have adopted for the METER corpus; i.e. have we encoded the correct things and have we encoded them correctly? 
(2) To verify and validate the XARA query tool; i.e. does XARA allow us to retrieve the things we want to retrieve and does it retrieve these things correctly? 
Specifically, we wanted to know whether XARA would allow us to answer queries such as: Show me the titles of Telegraph stories wholly derived from the PA; or, Extract a sub-corpus consisting of murder stories with the catchline ``Axe''; or, Extract all sentences from the Sun stories which were rewritten from the PA; or Which words occur most frequently in the verbatim/rewritten/new portions of texts? 
227
More generally this work addresses the utility of adopting TEI encoding standards for corpus linguistic work. It also provides lessons into how to more effectively annotate texts for subsequent investigation and how to extend the capabilities of the XARA querying tool to better support corpus linguists. 
2. The METER project 
2.1. Aims and objectives of the project 
The METER project1 investigated the reuse of news agency (also known as newswire) sources by journalists and editors in the production of newspaper articles. While the reuse of others’ text without acknowledgement is, in academic life, a cardinal sin, in the newspaper industry this is not only an accepted behaviour, but is in fact a standard business practice. In the newspaper industry, journalists depend upon “pre-fabricated” agency press releases as the source for many of their own stories: “most of what journalists actually write is a re-processing of what already exists” (Bell, 1991). Because newspapers subscribe to newswire services, they are at liberty to reuse source texts in any way they want (even verbatim), with or without attribution to the original author: the news agency. 
Aims of the METER project included the following: 
(1) To investigate and provide examples of text reuse by journalists writing for the British Press. 
(2) To define a framework in which text reuse between the newswire-newspaper could be assessed. 
(3) To build a corpus, a collection of carefully selected source and possibly derived texts for further study and evaluation of automatic methods of detection. 
(4) To select and evaluate a range of algorithms to measure text reuse based on existing methods of computing similarity between natural language texts. 
The METER project was supported by one of the largest UK news agencies, the Press Association, who allowed us access to the full newswire service as supplied to journalists. Being a primary supplier, the news issued by the PA is widely reused, either directly or indirectly, in British newspapers. 
The study of text reuse has, aside from its intrinsic academic interest, a number of potential applications. Like most newswire agencies, the PA does not monitor the uptake or dissemination of copy they release because tools, technologies, and even the appropriate conceptual framework for measuring text reuse are unavailable. For news agencies like the PA who release vast amounts of news material everyday (on average the PA release over 1,500 stories each day), manually analyzing stories for text reuse is not only impractical, but with limited resources is also infeasible. For the PA, potential applications of being able to automatically measure reuse of their text accurately include: (1) monitoring of source take-up to identify unused or little used stories; (2) identifying the most reused stories within the British media; (3) determining customer dependencies on PA copy, and (4) new methods for charging customers based upon the amount of copy reused. This could create a fairer and more competitive pricing policy for the PA2. 
2.2. The METER corpus 
To support our investigation of text reuse, we created a corpus of newswire and newspaper texts called the METER corpus. This corpus, the first of its kind as far as we are aware, provides over 700 examples of text reuse between Press Association source texts and subsequent articles published by nine newspapers in the British Press who subscribe to the PA news service. The corpus consists of 1,716 texts selected from the period between 12th July 1999 and 21st June 2000 from two staple and recurring domains in contemporary British Press: (1) law and court reporting, and (2) show business and entertainment. Newspaper and newswire texts collected for the METER corpus vary around a number of parameters which were taken into account during the corpus construction. These include the following: (1) domain, (2) source, (3) time period, (4) newspaper register, (5) length of newspaper story, (6) coverage of the news topic, (7) degree of reuse, and (8) the number of newspapers reporting the same story. Texts were selected to represent these parameters and their influence on text reuse was taken into account during analysis. 
1 METER was a 2-year UK EPSRC-funded research project – for details see: http://www.dcs.shef.ac.uk/nlp/meter. 
2For more information on the PA and public access to a portion of their newswire, see: http://www.ananova.com. 
228
As well as selecting source texts for the corpus, professional journalists also analysed the PA and newspaper text pairs and recorded the derivational relationship believed to exist between them according to a two-level, three-degree scheme for classification. At the document level, all 944 newspaper articles were classified as either: (1) wholly derived, (2) partially derived, or (3) non-derived, reflecting their degree of dependency on the PA source text for the provision of news material. A more fine-grained scheme, at the lexical level, aimed to capture the reuse of text within the newspaper text itself and classified word sequences as either: (1) verbatim, (2) rewritten, or (3) new. At this level of reuse, 445 texts only were analysed and annotated due to limitations of time and resource. These derivation relationships were captured in the corpus as pragmatic annotation – interpretive information added to the base text to capture a professional journalist's view of text reuse between a PA source text and corresponding newspaper article. 
2.3. Annotating the corpus in TEI/XML 
To explicitly capture characteristics of the newspaper texts, e.g. their headline, author, date of publication, catchline, domain and source, as well as the derivation relationships existing between the newswire-newspaper counterpart texts, we originally created an SGML markup scheme in a beta release of the corpus. However, we later transformed the entire collection into XML conforming to the Text Encoding Initiative (TEI) standard with the goal of making the resource compatible with international standards for corpus encoding and hence facilitating exchange with other members of the corpus linguistics community. 
The corpus is stored physically in 27 separate files, one for each day on which stories were sampled. A global corpus header file contains information about the corpus as a whole, including publication information and the definition of attributes specific to the METER project (e.g. the document and lexical level annotation schemes). Within each day file, material is organized into catchlines – a group of articles from different sources all dealing with the same story, which is identified by a PA-assigned tag known as a catchline (e.g. “axe” for a story about an axe murderer). Within catchline are the individual articles as published by the PA or newspapers. 
Figure 1 shows how we have captured this structure using TEI tags and attributes. The TEI scheme allows for the renaming of tags, but we chose to use standard names for simplicity; in some cases, therefore, the tag and attributes names do not necessarily provide an intuitive abbreviation of what they are intended to represent. Each file consists of a header (encapsulated within the <teiHeader> tag), followed by a collection of catchlines for both PA and newspaper texts within a <text> and <group> tag. Each catchline is defined by a <text> and <body> tag. The <text> tags are not shown in Figure 1 to simplify the diagram. Within <body> tags the <div> tag is used to indicate individual newspaper articles or pages of PA text for this catchline. At the most detailed level of annotation, each text consists of paragraphs and sentences, denoted by <p> and <s> tags. Within sentence the <seg> tag is used to indicate portions of verbatim, rewritten and new text. More information about the structure of the XML/TEI corpus can be found in Clough et al. (2002). 
Figure 1: The TEI markup of individual days in the METER corpus 
229
Table 1 lists, in order of their hierarchical structure, the key TEI tags and attributes which are used to encode the METER corpus, as shown schematically in Figure 1. 
TEI tag Attributes Values Comment <text> id 
n e.g. “M01032000-1” 
e.g. “Burstein”, “mother”, “wagstaff” Unique catchline ID 
Name of catchine <div> id 
n 
type 
ana e.g. “A636” 
e.g. “pa-01032000-15” 
“courts” or “showbiz” 
“src”, “wd”, “pd”, or “nd” Unique ID (A=PA, M=newspaper) 
Reference for each news story 
Domain for this catchline 
Document-level classification <head> Type e.g. “pa”, “sun”, “mail”, “times”, “guardian” The source of the news article <s> N 1..N Sentence number in the text <seg> Ana “verbatim”, “rewritten” or “new” Lexical-level classification 
Table 1: Some of the TEI tags and attributes used in the METER corpus 
3. XARA – the XML Aware Retrieval Application 
3.1. Background 
SARA ("SGML-Aware Retrieval Application") was originally developed with funding from the British Academy and the British Library to meet the need for a simple searching interface to the British National Corpus (BNC), when this was first distributed in 1994. The original design brief was to survey freely-available text retrieval software libraries and to build on these a simple user interface which could also be distributed without additional licensing costs. At this time the amount of freely available SGML-aware software was small, and it was rapidly decided that it would be quicker and more cost-effective to develop our own within the BNC project. 
SARA was accordingly developed with the very specific needs of the BNC in mind, incorporating a number of unusual design features unique to that resource (such as the specifics of how words are tagged, and speakers in its spoken texts are identified), while excluding a number of more generic but expensive features (such as indexing of POS tags). The system was also designed with a particular software environment in mind: one in which researchers used desktop machines to access local server machines, probably running a dialect of Unix, rather than standalone desktop systems; and in which data were counted in megabytes rather than gigabytes. Because it was intended for use with only one specific data set, little thought went into generalizing (or even distributing) the component which created the index files used by the server to optimize access to the corpus. For similar reasons, little effort was put into optimizing the structure of those indexes, or extending them to cope with user requirements beyond those already identified within the project. 
Some consequences of these design decisions are readily apparent: it is designed to operate in a network environment, with the bulk of data storage and much of the processing carried out on a central server. The overall functionality of the system is constrained by the expressivity of the protocol by which clients communicate with the server. This is probably a good thing, since the availability of that protocol makes it easier to develop new clients in new environments; constraints consequent on the need to support only the BNC are less good however, since the fixed index format made it difficult or impossible to support additional requirements such as efficient identification of collocates or POS searches. 
3.2. New directions 
In redesigning the tool, the primary objective was to take advantage wherever possible of the availability of XML and of XML encoded data, reflected in a change of name – from SARA to XARA. This decision conveniently allows us also to leverage a number of powerful technologies: most significant of these is probably Unicode, which not only enables us to support corpora in any of the world's current languages, but also allows us to rely upon standard methods of tokenization and character classing. XARA will operate on any well-formed XML document, without the need for any DTD or detailed tagging; it will also make use of whatever DTD it is supplied with to enrich the functionality of the supplied data. 
230
After nearly a decade of use, we were much better aware of the problems reported by users of the BNC and SARA, and felt confident that we now knew which of the many possible additional enhancements we might add would be of most general usefulness to our target user community: some of the new features are listed below. As well as relying on Unicode and XML, XARA uses other recognised standards for its communication with other systems and with the outside world. The metadata required by the indexing system is all recorded using the TEI header (though there is no requirement that the rest of the corpus follows this particular XML application); the scripting language offered for application development is based on Javascript; the formatting language is a simple subset of CSS, the W3C's Cascading Stylesheet Language; and we are also considering how best to re-express the query protocol itself in an existing XML vocabulary. 
3.3. Technical Overview 
The XARA system combines the following components: (1) an indexer, which creates inverted file style indexes to a large collection of discrete XML documents; (2) a server, which handles all interaction between the client programs and the data files; (3) a Windows client, which handles interaction between the server and the user. 
In addition, an index building utility, called Indextools, is supplied with XARA, which simplifies the process of constructing a XARA database. Its chief function is to collect information about the corpus to be supplied additional to that present in any pre-existing corpus header, and to produce a validated and extended form of the corpus header. It can also be used to run the indexer and test its output. 
3.3.1. Functionality for indexing 
In order to process a large amount of textual data marked up in XML, XARA uses a pre-built index to optimize access to the original source texts for particular (largely lexically-motivated) kinds of query. To construct this index, the software needs information about: 
• how PCDATA (element content) is to be tokenized 
• how tokens are to be mapped to index terms (for example, by lemmatization or by the inclusion of additional keys) 
• how index terms are to be referenced in terms of the document structure 
• how and whether XML tags and attributes are to be indexed 
Much, perhaps most, of this information is implicit in the XML structure for a richly tagged corpus: one could imagine a corpus in which every index term was explicitly tagged, with lemma, part of speech, representation etc. In practice however, such richly tagged corpora remain imaginary, and software performs the largely automatic task of adding value to a document marked up with only a basic XML structure. XARA allows for the specification of how that task is to be performed in a standardized way, using the existing structure of the TEI header (see http://www.tei-c.org/Guidelines/HD.htm) as the vehicle for its specification. 
3.3.2. Functionality for searching 
XARA inherits from SARA a rich range of query facilities, but adds to them considerably. The system allows the user to search for substrings, words, phrases, or the tags which delimit XML elements; it also supports a variety of scoped queries, searching for combinations of words etc. in particular contexts, which may be defined as XML elements, or as combinations of other identifiable entities, as further discussed elsewhere in this paper. Searches can be made using additional keys such as part of speech, or root form of a token, specified either explicitly in the tagging of the texts, or implicitly by means of a named algorithm. 
Outputs from a search usually take the form of a traditional KWIC concordance, which can be displayed or organized in several different ways, under user control, or can be exported as an XML file for use with other XML-aware systems such as a word processor. 
Corpora can be reorganized or partitioned in a user-defined way, using the results of any query, the values of specified element/attribute combinations, or by means of manual classification. Searches carried out 
231
across partitioned corpora can be analysed by partition: so the client can display the relative frequencies of a given lexical phenomenon in texts of all different categories identified in a corpus. 
4. Using XARA to explore the METER corpus 
4.1. Indexing the METER corpus 
To index the METER corpus, we used the XARA Indextools utility. Given a TEI corpus header file and a group of XML/TEI texts, the indexer tool creates a searchable index from the texts, which can then be explored using XARA. XML/TEI validation is carried out during indexing. 
One aspect of XARA which has important implications for subsequent searching, is the requirement for the user to specify a notional document and unit during indexing. The document can be any XML element. Its significance is that partitions of the corpus, or sub-corpora, can only be defined in terms of sets of whatever element is chosen as document. The unit can also be any XML element; its function is to identify the lower level context within which hits are identified. We chose <div> to be our notional document, i.e. the individual newspaper or PA article, and <s>, or sentence, to be our notional unit. Thus we can create partitions, or sub-corpora, of articles matching various search criteria and lexical search results are displayed in terms of the sentences in which they occur. 
4.2. Searching the METER corpus 
Once indexed, the corpus can be searched. XARA offers a number of query options including: word, phrase, part-of-speech, pattern and XML-based searching. There are at least four specific features of XARA which are useful for querying the METER corpus: (1) the Query Builder tool which enables users to create complex queries using a visual interface, (2) the ability to define a partition of the corpus, dividing it into sections according to a number of different criteria, (3) the way in which search results may be presented to the user and (4) how search results may be saved for subsequent exploitation. We describe these capabilities in this section, then present a set of example queries in the next section. More detailed information about searching with SARA can be found in Aston and Burnard (1998). 
Query Builder 
To create complex queries the Query Builder tool can be used, the visual interface for which is shown in Figure 2. The left node is called the scope node and defines the scope or context in which a complex query is executed. For example, if the scope node is defined as the <div> annotation, the search will take place on text and annotations only within the scope of this annotation – a news story in the case of the METER corpus. To fully specify the query, one or more right-hand nodes must be specified using any of the query options, e.g. pattern, word, phrase or XML annotation. Nodes can be added both vertically and horizontally. Query nodes added vertically are logically ANDed together to narrow the context in which the search is executed. Nodes added horizontally represent alternatives, i.e. are logically ORed. For vertical connections, the link type between nodes can be specified as either next (i.e. adjacent), not next, one-way, or two-way, all indicating the order and proximity in which matching search terms must be found. 
Figure 2: Ordering of nodes can affect the query: left-hand query produces 
no solutions; right-hand one produces 21,209 solutions 
Searches based on annotations and one-way link types are sensitive to the hierarchical structure and order of the corpus annotations. For example, consider the problem of listing all sentences from news stories. Two versions of the query are shown in Figure 2. The query on the left returns no solutions, whereas the 
232
query on the right returns all 21,209 sentences in the METER corpus. The difference is the ordering of nodes in the query. The left-hand query does not work because the ordering requires matching sentences first and then the headline, but sentences always come after the headline. 
Defining partitions 
XARA allows the user to create sub-corpora, or partitions, each representing a different grouping of the documents in the whole corpus. By this means the user can select a portion of the corpus which is of interest and perform further analysis on this portion only. Partitions can be created by either selecting texts from the user interface or by using a query to partition the corpus into documents that match the query and those that do not. Once a partition has been created, it can be activated and queries are addressed to the activated partition only. By partitioning the METER corpus, queries aimed at one particular subset of the corpus, e.g. a particular newspaper or only articles derived from the PA, can be easily created, thereby reducing their complexity. This capability highlights the importance of (1) ensuring that the textual units over which one wants to aggregate results are defined as XML elements in the building the corpus, and (2) selecting the appropriate XML/TEI element as the notional “document” in indexing the corpus with XARA. 
Viewing the search results 
After executing a query in XARA, the results are displayed as either a listing or each result is displayed individually. The listing usually takes the form of a KWIC concordance in which results can be viewed as plain text, in XML showing the annotations, or in a custom-format using a user-defined Cascading Style Sheet (CSS). The context of the listing will depend on the search itself, but can be broadened and narrowed by the user. The listing can be exported as an XML file which can be viewed or edited with other XML-aware applications. 
Saving search results 
XARA enables the user to save various outputs including word frequency lists produced during a word query and the results of query analysis, in XML or comma/tab separated plain text formats. The XML results can be parsed by and viewed in any XML-aware tool. 
4.3. Sample Searches 
In this section we present a number of example questions that one might like to ask of the METER corpus, and discuss whether and how XARA can support these requests. These fall into three classes: those addressing the lexical content of the corpus only, those addressing the XML annotations alone, and those combining lexical content and annotations. 
4.3.1. Lexical searches based on the content of the news stories: free-text searching 
(1) Find me which texts contain a given word or phrase 
Suppose we begin with a simple word or phrase search, for example: “find me all the texts which contain the word ‘Ipswich'”. This can be performed using the “word query” option which matches 30 occurrences of “Ipswich”, and 1 occurrence of “Ipswich's”. At this point, the word frequency table can be saved in XML, or displayed as a KWIC concordance. As well as exact match searches, a regular expression can also be defined to search for a pattern, e.g. “town*” finds the words “town”, “town's”, “townhill”, “townhouse” and “townend”. Phrases can also be searched for, e.g. “Ipswich town”, although this option does not permit pattern searches. 
(2) Find me which words co-occur with a given word 
XARA can be used to find collocations, a common task in corpus linguistics. A Z-score or Mutual Information (MI) score indicates the “strength” of collocate and can be used to rank the results. Collocates of “Ipswich” include “CID”, “borough”, “town” and “crown”. Again, this listing can be viewed and/or saved. 
233
(3) Find me which words are most frequent in the METER corpus 
To create a list of word frequencies, the word query option is used and the search term defined as a pattern, e.g. [a-zA-Z-]*. This calculates word frequency based on the entire METER corpus and the results can be saved to a file in the form <word><frequency> (e.g. comma-separated or XML tagged). It would be useful to be able to filter certain words, e.g. function words, which are invariably most frequent, from these lists, either at search or index time: XARA does not support this directly, though it is possible to order lists by frequency and specify a cut-off point. 
When a partition is active, word frequencies are provided for each component of the partition, allowing one to compare the relative frequencies of the given search term across the partition. 
(4) More complex lexical searches 
As with most search engines, XARA can be used to generate complex queries using Boolean operators, such as “find me which texts contain the words X and Y”, or “find me which texts contain the words X or Y”. The Query Builder tool can be used to perform this search task, but also supports more sophisticated searching by enabling the user to limit the scope and order with which the query words match. For example, the search “Find me all texts containing Ipswich and council” can be defined in Query Builder as a search for two vertically-placed word nodes within the scope of a <div>. With the link type defined as next, the query requires the word “council” to immediately follow the word “Ipswich” (equivalent to a phrase search), a 1-way link means that “council” must follow “Ipswich” within the scope of <div> (i.e. ordered), and 2-way link implies the words can occur in any order, but still within the same scope. 
4.3.2. Searching on the XML annotations 
Using the XML markup, we are able to exploit the interpretative information added to the METER corpus to answer queries, which cannot be answered from the lexical content. As with searching over lexical content, XARA supports complex searching over annotations too, enabling us to answer questions such as “Find me all texts from the Sun newspaper derived from the PA”. 
What you can do with XARA depends crucially on the annotation scheme; therefore, careful design of this scheme is paramount. For example, in the initial TEI markup of the METER corpus, we did not use the “type” attribute of the <head> tag to identify the article source, but only to indicate whether the source was a newspaper or a PA text. The specific source could only be extracted from a substring of the “n” attribute on the <div> tag (e.g. “sun-01032000-3”). The version of XARA we used did not support pattern searches on the XML attribute value strings: our solution was to extract the source name from the <div> annotation, using a Perl script, and change the “n” attribute to reflect this, i.e. to modify the annotation scheme. 
(1) Show me how many newspaper stories are in the METER corpus 
When constructing an XML query, selecting the <head> tag and “type” attribute displays all possible attribute values and their counts, giving the number of articles from each newspaper in the corpus. This is true for all attributes, e.g. selecting the <div> tag and “ana” attribute enables the user to determine there are 205 non-derived texts, 438 partially-derived and 301 wholly derived. 
(2) Extract all titles from the Sun which are wholly derived from the PA 
This can be performed using Query Builder to create the query shown schematically below. Scope is defined to be within <div>, and the additional query constraints are that the “ana” attribute has be “wd” (wholly-derived) and the <head>'s “type” attribute must be “sun”. The result is a list of 19 headlines. 234 
Scope node: 

<element name=”div”/> 

Query term nodes: 
<element name=”div”><attribute name=”ana”>wd</attribute></element> 
CONTAINING 
<element name=”head”><attribute name=”type”>sun</attribute></element> 

As a further example, to select all wholly-derived texts from the Sun for the courts domain, the preceding query need only be modified by adding a further constraint on the scope node (“type” equals “courts”). 
(3) Extract all sentences from the Sun which were taken verbatim from the PA 
To extract verbatim sentences a query can be defined to match sentences containing a <seg> tag with the “ana” attribute value equal “verbatim. If during indexing the unit is specified to be <s>, as we have assumed, the results will be displayed within the context of the sentence as required. Searches such as “Show me all verbatim segments from the Sun”, or “Show me texts containing verbatim segments from the Sun” require re-indexing the corpus, defining the unit to be either <seg> or <div> respectively. 
4.3.3. Combining searches 
Using the Query Builder tool, search requests which involve combining both information from the annotations and free-text can be constructed, enabling us to answer questions such as the following. 
(1) Show me all titles from the Sun with the word “Axe” in the headline 
We define the scope node to be <head> with the attribute “type” set to match “sun”. By adding a word query node, as shown below, to match the word “axe”, two titles are returned: “Sex taunt made axe monster murder 3 in love triangle” and “Street to axe Ravi”. Note that in the first, “axe” is used to modify the noun “monster”, the context being a murder, whereas in the second “axe” is a verb which in this story is used to headline a story about the actor Ravi being removed from the British soap opera, Coronation Street. This reveals some interesting characteristics of the kind of newspaper language used by the popular press. 
s 
(2) Extract a sub-corpus consisting of all murder stories with the catchline “Axe” 
Start by defining a query whose scope node is <div> and whose “type” attribute is “courts” (this category defines murder stories). Then, add a word query node of “axe”. This results in 45 stories from both newspapers and the PA (this could also be filtered to allow, e.g., just newspaper stories). A new partition is defined based on this query, saved and the partition activated to limit further queries to just these texts. 
(3) Which words occur most frequently in the verbatim, rewritten and new portions of the texts? 
This search is more complex than the preceding because it involves creating a new XML file based on the results of a query, and then re-indexing this file to compute word frequencies. First, we index the METER corpus with <div> as the document, and <seg> as the unit to create a listing with a context of just <seg>. This listing is saved in XML and used as the basis for a new index. This index is then loaded into XARA and word frequencies can again be computed. 
5. Conclusions 
The METER corpus was encoded in TEI/XML format to enable the data to be shared with other members of the TEI community, and to enable generic XML and TEI parsers and search tools to be used, avoiding the need to build custom applications. The goals of the work described in this paper were (1) to verify and validate the TEI encoding scheme we have adopted for the METER corpus and (2) to verify and validate the XARA query tool. We pursued these goals by examining a variety of questions related to the annotations and lexical content of the METER corpus, questions which we hoped the corpus annotation and XARA would enable us to answer, and used XARA to try to answer them. 
Our first observation is that the TEI encoding of METER is in fact valid – by indexing the corpus using XARA we have been able to verify that the texts are encoded in valid XML. Secondly, the encoding was verified to the extent that it allowed us to answer all of the questions that we set by ourselves. However, a deeper question is whether the TEI encoding was really necessary. Certainly, there can be no doubt that an XML encoding of some sort is highly useful, enabling searches over the structured annotations, as well as 

Query term nodes: 
<or><lemma>axe</lemma></or> 

Scope node: 
<element name=”head”> 
<attribute name=”type”>sun</attribute> 
</element> 
235
combined searches over lexical content as well as the structured metadata. However, it is not clear that for the METER corpus annotating the corpus in TEI provides any additional benefit beyond encoding the corpus in XML. For those who are not familiar with TEI – in this case the creators of the corpus when we set out to build it – TEI involves a steep learning curve. Using TEI offers the corpus encoder a choice of predefined tags and attributes, at the expense of sometimes difficult decisions about how exactly to map the structural units of the corpus into the available TEI elements and their associated attributes. XARA works perfectly well with any XML files, so the benefits it offers could be derived without adopting TEI. While for many applications a ready-made tag and attribute set of the sort TEI offers may be entirely adequate and promote sensible annotation, for others it may be the case that carefully designing a bespoke annotation hierarchy and attributes from the first principles is a better use of time than shoe-horning a problem into the framework that TEI provides. 
One specific problem we encountered illustrates this issue and the interaction between the annotation scheme and the retrieval tool. When encoding the METER corpus in TEI, we annotated sentences and then annotated verbatim, rewritten or new word sequences using the TEI <seg> within sentences. Since XML does not permit overlapping elements, the result was that verbatim, rewritten or new word sequences that spanned sentences were split into multiple shorter sequences that fit within sentences. However, our research into detecting text reuse found that the length of verbatim matches was a key indicator of reuse. The TEI encoding scheme we had adopted precluded determining whether adjacent verbatim word sequences in different sentences were part of a single longer verbatim sequence. One solution to this difficulty is to use a link attribute to chain together multiple connected segments. Another solution would be to do without sentence annotations since they are not essential to the METER aims. However, this is not entirely satisfactory because not all texts in the corpus were annotated to the same degree of granularity, i.e. some are annotated to the level of word sequence and some only to the document level. Certain degree of markup more fine-grained than the whole text, e.g. the sentence, is required to enable the results of search requests in XARA to be of practical benefit. 
The second aim of this paper was to verify and validate the XARA query tool as a possible search tool for release with the METER corpus. XARA has been able to support all of the initial questions we set out to answer, and has so far as we are aware, correctly answered all of them. This included queries concerned with lexical content only, those concerned with structured annotations only, and combinations of the two. The only limitations we discovered in the course of this study were that (1) partitions cannot be saved in a XML format that can be read by other applications, (2) XARA does not support pattern match searching over XML attribute value strings, which can be useful when values assigned to attributes are non-atomic. This facility has subsequently been added to the system. 
A final lesson for would-be corpus creators is this: before committing to an annotation scheme, learn about the capabilities of a XML query tool such as XARA, identify key queries you would like the scheme to support and assure yourself that the query tool and annotation scheme together support the queries. 
References: 
Aston, G. and L. Burnard. 1998. The BNC Handbook: Exploring the British National Corpus with SARA. Edinburgh Textbooks in Empirical Linguistics, Edinburgh University Press, Cambridge, UK. 
Bell, A. 1991. The Language of News Media. Blackwell, Oxford, UK. 
Clough, P, R. Gaizauskas, and S. L. Piao. 2002. Building and annotating a corpus for the study of journalistic text reuse. In Proceedings of the 3rd International Conference on Language Resources and Evaluation (LREC02), pp. 1678-1691. 
Gaizauskas, R., J. Foster, Y. Wilks, J. Arundel, P. Clough, and S. Piao. 2001. The METER corpus: A corpus for analysing journalistic text reuse. In Proceedings of the Corpus Linguistics 2001 Conference, pp. 214-223. 
Sperberg-McQueen, C. and L. Burnard, editors. 1999. Guidelines for Electronic Text Encoding and Interchange. TEI P3 Text Encoding Initiative, Oxford, UK, revised reprint edition. 
236

Repetition and young learners´ initiations in the L2: a corpus-driven analysis 
Ana Llinares García 
Universidad Autónoma de Madrid 
Abstract 
The present study is based on the analysis of classroom interactions between children and with their teacher in different types of foreign language contexts. The subjects of our study are five-year-old children and their teachers from different types of schools, with different degrees of immersion in the L2. The data comes from the UAM-corpus, which is collecting spoken data from different types of EFL contexts.1 
The source of our analysis is based on Halliday´s and Painter´s classification of the communicative functions that children can convey in their mother tongue at the pre-school level (Halliday 1975; Painter 1999). In this paper, we will provide a corpus-driven classification of the different types of repetitions performed by both teachers and children. We believe that teachers should encourage certain types of repetitions in the children´s discourse. The data shows that when children are encouraged to repeat certain utterances with specific discourse functions, even children in low immersion contexts will end up initiating the interaction with no help of the teacher. This is especially important in the case of foreign language learners, in order to avoid their functional language production being limited to responses to their teacher´s initiations. 
1. Introduction 
The present study is part of a larger study that focuses on the functional analysis of a spoken corpus of English as a foreign language. The corpus has been collected from different types of EFL contexts with different degrees of immersion (from complete to low-immersion contexts). The first part of this corpus, which will be used in this study, contains data from five-year-old children and their teachers. One of the reasons for our choice of this type of subjects is the growing interest in Spain, in the last few years, in introducing the teaching of English at the pre-school level. Moreover, there has been very little research on how a foreign language develops at this age. 
Halliday´s (1975) and Painter´s (1999) analysis of the language of their children focused on their observation that children develop their language because they need to do things with it. Following this idea, the point of departure of our study was that, at the pre-school level, the teacher should encourage learners to use the L2 to convey different types of functions (to give information, to ask for information, to order an action, etc...). At this age and level, in our opinion, the priority should be that the children perceive the learning of a foreign language as a process where they can achieve things with it, where the L2 has a functional purpose, in the same way as their mother tongue. 
In this particular paper, we focus on the use of repetition, both by teachers and children, in both a high-immersion and a low-immersion classroom context, and the different functions conveyed. 
2. Repetition in the language of teachers and pupils 
In the 1970s began a great interest in the analysis of the language used with the children, which was called baby talk or motherese (Snow and Ferguson 1977). This kind of research focused on the importance of interaction and found that parents used well-structured sentences, repetitions and reformulations in order to guarantee communication. However, some authors, such as Aitchison (1998), questioned the validity of the input theory and argued that parents´corrections and reformulations had no positive effects on the child´s linguistic development. Aitchison referred here to the grammatical complexity. However, as we will try to show in this paper, repetitions and expansions have a lot of influence in other types of linguistic developments, such as the pragmatic ones. In our opinion, input and 
1 The UAM-Corpus has been collected since 1998 and contains longitudinal data from children and teachers in EFL classroom contexts with different degrees of immersion in the L2. At the moment we have recorded data from the same children from the age of five to nine (Project financed by the Comunidad Autónoma de Madrid, 06/0027/2001) 
237
interaction are fundamental aspects in the functional development of a foreign language, especially at the pre-school level. 
Repetitions are not only a characteristic of the language of the parents. The results of studies on communication in childhood show that the children tend to reinitiate communication when they don´t receive any response. Garvey and Beringer (1981) find out that children use repetition to reinitiate their discourse 1/3 of the times. Repetition is also a common strategy used by foreign language learners. In her classification of learner strategies, Oxford (1990) considers “repetition” a cognitive strategy within the group of “direct strategies”. 
As far as teacher repetitions are concerned, Pica (1994) shows that when sentences are repeated or reformulated to enhance comprehension, the learners have more opportunities to become aware of characteristics of the L2. In the analysis of the L2 teacher´s language, Richards and Lockhart (1994) find that the instructions are repetitive in order to facilitate their pupils´comprehension. Classroom discourse is not always spontaneous and it has some elements common to other type of interactions in everyday life and some elements that are specific of the classroom context. At the pre-school level, the classroom discourse tends to be very similar to the type of interaction that is carried out at home between parents and children. Therefore, the use of repetitions by both teachers/parents and learners/children is common in both contexts. Parents tend not to correct the form of their children´s utterances. They seem to be more worried about communication and they tend to repeat the correct form if something is not very clear (Kess 1992). With these parameters children´s language develops. Therefore, the teacher´s discourse should be similar in second language contexts at an early age. 
However, there is a type of interaction that is characteristic and specific of the L2 classroom: teacher initiation-learner response-teacher feedback (Sinclair & Coulthard 1975). Usually, after correcting the learners, the teacher expects them to repeat the correct utterance. This is a type of repetition encouraged by the teacher that is very common in classroom contexts. We believe it is very important for teachers to ask learners to produce this kind of repetitions in EFL contexts at initial learning levels, as they will lead, in the end, to the learners´ initiation of interactions, as we will see in the present paper. 
As far as bilingual contexts are concerned, Genesee (1994) suggests the teacher repetitions of important words as one of the ways of facilitating comprehension. He suggests the importance that the teacher offers opportunities for language use and interaction, promoting rich activities that give the children the opportunity to initiate a conversation. He also suggests the importance of encouraging children´s repetitions of the teacher´s model in order to stimulate specific linguistic aspects. 
3. The function of repetition in the corpus: qualitative analysis 
The function of repeating is included in many of the taxonomies on pragmatic functions both outside and inside the classroom context. In the analysis of the language of the child, it is important to mention taxonomies such as the one by Vila (1987) and by Ninio et al. (1994), which include the function of imitating through repetition as very characteristic of the language of the child. Vila (1987), for example, describes the function of imitating as characteristic of the dialogue and defines it as the function of repeating the adult´s utterance without providing any information. In the analysis of children´s production in second language classroom contexts it is important to mention Cathcart´s taxonomy (1986). She identifies another type of repetition, which consists of children repeating their own utterances in order to reinforce a message (in this case it is self-repetition, as opposed to imitation of the other´s utterances, which is the function identified in Vila´s and Ninio et al.´s taxonomies). 
In a broader analysis, we classified the different pragmatic functions realised by teachers and pupils in our corpus based on Halliday´s and Painter´s classification of the functions of language that children realise in their mother tongue: personal, informative, heuristic, interactional, regulatory and instrumental. We made a corpus-driven subclassification within each function and we added some functions that we found specific of the foreign language classroom context. We found that it was important to consider a function specific of the teacher, called teacher feedback, and another function that we called secondary functions, whose aim was reinforcing a specific message. The different types of repetitions were classified as follows2: 
2 Not all the functions presented below were always realised through repetitions. 
238
Teacher repetitions: 
I./ TEACHER FEEDBACK (other-repetition/ voluntary) 
I.a./ Repetition of the message given by the learner in order to encourage them to say more or as a simple acknowledgement 
Example: 
TCH: And how do you tear it off? CH: This cuts it 
TCH: Oh, that cuts it... 
CH: Like this 
I.b./ Repetition of the message given by the learner in order to show positive assessment. 
Example: 
TCH: You want to cut the bread. Where does that come from? Where does th- the bread come from? 
CH: From wheat. 
TCH: From wheat. 
In example Ib, the teacher asks the pupil a display question. Therefore, we have considered the teacher´s repetition of the child´s response as having a positive evaluative function. On the other hand, in example 1a the teacher asks a referential question, and the repetition of the child´s response does not have an evaluative function in this case, but an interactional function.3 
Children repetitions: 
II./ RESPONSE TO THE TEACHER´S PEDAGOGIC REGULATORY FUNCTION (other-repetition/ non-voluntary) The teacher asks a pupil to imitate or repeat with pedagogic purposes, so that he/she learns an utterance better or as reinforced input for their classmates. 
II.a./ The children are made to imitate the teacher´s utterance 
Example:4 
TCH: An elephant? CH: An elephant? TCH: In my house? No way CH: No way 
II.b./ The children are made to repeat their own utterance 
Example: 
CH: How many? 
3 Display questions are those where the answer is known for the speaker and referential questions are those where the speaker does not know the answer. The first type is characteristic of the language of the teacher, in order to test their pupils’ knowledge. 
4 
This extract belongs to an activity where several pupils participate in a role-play about a mother who does not allow her child to have certain animals at home. The dialogue is, in many cases, repetitive. However, this repetition seems to be necessary for the learners to complete the dialogue at this initial stage (we must remember that the pupils are five-year-old EFL learners). 
239
TCH: Fernando, sh, sh, can you repeat, David? 
CH: How many? 
One of the most characteristic types of interaction in a second language context is that where the teacher gives a model that the child has to imitate (Prator 1969). Among first language acquisition studies, some within a generative approach have suggested the low impact of children´s imitation for their language development (Ervin 1964). Aitchison (1998) points out that repetition is necessary but is not enough: What is being said is that practice alone cannot account for language acquisition: children do not learn merely by constant repetition of items (Aitchison 1998:75). In our opinion, it is important to encourage children to imitate in foreign language classroom contexts at an early age, even though it does not create a natural interaction in the L2 classroom. 
III./ CHILDREN´S SPONTANEOUS REPETITIONS OF THE TEACHER´S OR OTHER CHILDREN´S UTTERANCES (other-repetition/voluntary) 
Example: 
TCH: Sit down ((TO FERNANDO)) OK. And now tell me, what's this? (( SHOWS A PICTURE )) 
CH: Boy. 
CH: Boy 
One of the most characteristic types of interaction between the child and the adult includes the repetition that the child makes of the language of the adult (Montfort et al. 1996). In second language learning contexts, Genesee (1994) observes that the children have the tendency to repeat, only with the aim of practising something new for them. 
Teacher and children repetitions: 
IV./ RESPONSE TO A REQUEST FOR CLARIFICATION (self-repetition/non-voluntary) 
Example: 
a) 
CH: Red. 
TCH: Sorry? 
CH: Red 
b) 
CH:You know? My daddy has . three or four brothers. CH: What? 
CH: My daddy has three or four brothers 
In examples a) the teacher demands the repetition for clarification and in b) it´s a child that demands it from one of his classmates. 
V./ SECONDARY FUNCTION (self-repetition/ voluntary) Self-repetition of a message to reinforce it 
Example: 
CH: and then .. he do it wrong TCH: He did it wrong. Why did he do it wrong? How was it wrong? 
In the acquisition of the mother tongue, self-repetition is one of the most common strategies used to teach and learn. In the acquisition of a second language, Tomlin (1994) argues that repetition is a social 
240
act with cognitive consequences. With repetition, the teacher helps the pupil to understand the sentence produced, and this has the cognitive consequence of helping the learner to transform “input” in the L2 into “intake”. 
The secondary functions could be classified into what Halliday calls “textual meaning”, as they are used to clarify what has already been said. We have followed Sinclair and Coulthard (1975) when they classify some of the moves as “subordinate”, when they are not followed by a response and depend on the central move. In our taxonomy, we have considered different types of secondary functions. We have included here the one that involves repetition of the previous utterance. We have codified an utterance with this function when there is a repetition of the content of the message, even though it is repeated with other words, as we can see in the example above. However, when the children use this function, reinforcing their message, they tend to use the exact words of the central move, as we can see in the following example: 
CH1: I'm hungry. ((NOISE)) I'm hungry. CH2: This is television. This is television. TCH: OK. Television. 
In this example, we have a case of repetition of an utterance by a pupil to reinforce his need: he wants something to eat. The other type of repetition made by another child has the purpose of showing that he knows the answer to the teacher´s question. 
In our opinion, this is an important function both in the discourse of the teacher and the pupils. The fact that the teacher reinforces an utterance can have important implications on the pupils´ comprehension and future production in the L2. In the case of the realisation of this function by the pupils, it helps them to achieve their communicative goals. It is very common that children reinitiate the communication when they don´t obtain any response. Garvey and Beringer (1981) observed that children between 2;10 and 5;7 tended to reinitiate their discourse through repetition one third of the times. 
4. Methodology and Quantitative analysis 
The data analysed for this study comes from an English school, where children were exposed to the L2 for the whole school day, and from a low-immersion bilingual school, where children were exposed to the L2 for one hour everyday. In both cases, the children were five years old. We recorded five sessions from two groups in the English school and one group in the low-immersion school. After transcribing and tagging each individual utterance in both the language of teachers and pupils, we identified 67 different functions (Llinares García 2002). In this particular study, we will focus on the functions realised through repetition.5 
4.1. Self-repetition 
Out of 67 functions codified in the corpus we show below how the function of self-repeating or self-reinforcing an utterance is one of the three most frequent functions in the language of both teachers and children in both groups, as shown in the tables below: 
5 The Wordsmith tools were used to analyse the data, once it was tagged 
241
Table 1: Three most common functions realised by the teachers in both the high and the low immersion contexts 
Teacher/high-imm./Group A Teacher/high-imm./Group B Teacher/low-imm. Marker of initiation of an utterance (S1) 
11,88% Open display questions (H3.bo) 
10,71% Self-repetition or reinforcement of an utterance (S2) 
13,16% Call somebody (R1) 
11,10% Self-repetition or reinforcement of an utterance (S2) 
9,43% Positive evaluation (TF2.a) 
13,16% Self-repetition or reinforcement of an utterance (S2) 
10,01% Ask somebody to perform an action (R3) 
9,11% Call somebody (R1) 
10,32% 
Table 2: Three most common functions realised by the pupils in the L2 in both the high and the low immersion contexts 
Pupils/high-imm./Group A (L2) Pupils/high-imm./Group B (L2)Pupils/low-imm. (L2) Call somebody (R1) 
10,90% Response to open display questions (H3.re.bo) 
7,07% Completion of the teacher´s request (RP2.re) 
16,82% Self-repetition or reinforcement of an utterance (S2) 
8,79% Call somebody (R1) 
6,42% Identify or describe people or things in the present (Inf1) 
12,78% Identify or describe (Inf1) 
6,08% Self-repetition or reinforcement of an utterance (S2) 
6,06% Call somebody (R1) 
6,05% 
Table 1 above shows that the function of self-repetition is among the three most common in the language of the three teachers, as we can expect from a classroom context, where teachers need to reinforce their message to make sure that the pupils understand. This function is even more necessary in the language of the teachers in EFL contexts. In our data, the teacher in the low-immersion context seems to perceive even more the need to reinforce the message, as it is the most common function. This function is also one of the most common in the language of the pupils in the high-immersion context. 
4.2. The children´s response to the teacher´s feedback in the high-immersion context 
An interesting result in the high-immersion context was to observe that one of the teachers used the function of feedback very frequently, encouraging the pupils to continue with their discourse, whereas the other teacher used it much less frequently. The teacher in group A uses this function 25 times against twice in the case of the teacher in group B. As we can see in the example below, children go on with their discourse when they are encouraged to do so 
TCH: He does? Who kills him? 
CH: a bird, a bird kills him 
TCH: A bird kills him 
CH: It´s funny at the end because the grasshopper said, is that one another one of your tricks? 
The example below is one of the two that correspond to group B, and we can see that this function is not only realised through repetition: 
242
CH1: ((native)) I I I saw a grasshopper in my garden 
TCH: Did you? 
CH2: Me too. I saw it when I was little 
CH3: I saw one in the old school 
A native pupil initiates this interaction, but once the teacher encourages going on, other non-native children participate. 
4.3. The pupil´s imitation of the teacher´s utterances in the low-immersion context 
In the case of the low-immersion context, we identified a type of repetition that was important to encourage the pupils´ discourse initiation: children´s imitation or repetition after the teacher´s request to do so. Llinares García (2002) classified the function of asking for repetition and asking for imitation within a group of three functions that were called teacher pedagogic regulatory functions. These included asking for imitation, asking for repetition and asking for completion. In this paper, we are only focusing on the first two, which are the ones that involve repetition. All these functions were much more frequently used by the teacher in the low-immersion EFL context, as we can see in figure 1 below. In figure 2, we can observe that the functions of asking to repeat and asking to imitate are not used at all in the high- immersion classes: 
Figure 1: Use of the teacher pedagogic regulatory functions in the high and low-immersion contexts 
020406080100120140160180high-immers. Ahigh-immers.Blow-immers 
Figure 2: Use of each individual pedagogic regulatory function in the high and low-immersion contexts 
020406080100120140160TRP1TRP2TRP3high-imm Ahigh-imm Blow-immers 
The example below shows the importance of this function for the language development of children in low-immersion contexts: 
TCH: Who said yellow? Okay. Is it yellow? Santiago, ask Luis. 
CH1: Is it yellow? 
CH2: No 
243
TCH: Good boy.. No. It isn´t yellow <x......x> No. 
CH2: No. 
TCH: It isn´t yellow. 
CH2: It isn´t yellow. 
TCH: No. Good boy. Table three. Who´s going to ask a question in table three. Ínigo. Ask if it´s dangerous... Is it dangerous? 
CH3: Is is dangeorus? 
CH2: Is dangerous. 
CH3: Can I go to the bathroom, please? 
TCH:No. <x......x> Ínigo. W- wait <x till after x> class. Who wants to ask a question. Cristina, ask a question. What question would you like to ask? 
CH4: Is it fast? 
CH2: N o. 
In the example above, we can observe how the teacher asks the pupils to repeat a question to find out the animal. At the end, we can see how one of the pupils asks the question herself without having to repeat the teacher´s question. 
5. Conclusions 
There are three main conclusions that we can raise in relation to the use of repetition in the EFL classroom contexts in our corpus: 
- Self-repetition is a very common function in the language of the teacher, and it is the most common in the low-immersion EFL context. This surely implies that the teacher feels the need to reinforce the message to make sure that the pupils understand. 
- In the high-immersion EFL context, pupils have the L2 knowledge to be able to express their messages in the L2. However, they need to be encouraged by their teacher. Therefore, we can see that, although both groups are exposed to the same hours of English, in one the pupils participate more in a spontaneous interaction, because the teacher motivates them to do so. Maybe we can conclude that the time of exposure to a language is a key point in child L2 acquisition, but it is also important the type of interaction promoted by the teacher. 
- In low-immersion contexts, on the other hand, it is not enough to motivate the pupils to talk, as they don´t have the language level. Here, it is important for the teacher to carry out interactions based on repetitions and imitations, that will later make it easier for the children to initiate the discourse on their own and to express things with more autonomy. 
6. Bibliography 
Aitchison J 1998 The articulate mammal. London, Routledge. 
Cathcart R 1986 Situational differences and the sampling of young children´s school language. In Day R. (ed.) Talking to learn: conversation in second language acquisition. Rowley, Mass, Newbury House. 
Ervin, SM 1964 Imitation and structural change in children´s language. In Lenneberg, E.H. (ed.) New directions in the study of language. Cambridge, MA, MIT Press. 
Garvey C, Beringer G 1981 Timing and turn-taking in children´s conversations. Discourse Processes 4: 27-57. 
244
Genesee F 1994 Educating second language children. Cambridge, Cambridge University Press. 
Halliday MAK 1975 Learning how to mean: explorations in the functions of language. London, Edward Arnold. 
Kess JF 1992 Psycholinguistics. Amsterdam, John Benjamins. 
Llinares García A 2002 La interacción lingüística en el aula de segundas lenguas en edades tempranas: análisis de un corpus desde una perspectiva funcional. Unpublished doctoral dissertation. 
Monfort M, Juárez Sánchez A 1996 El nino que habla. Madrid, Colección Educación Preescolar. 
Ninio A, Snow C, Pan B, Rollins, P 1994 Classifying communicative acts in children´s interactions. Journal of Communicative Disorders 27: 157-188. 
Oxford R. 1990 Language learning strategies: what every teacher should know. New York, Newbury House. 
Painter C. 1999 Learning through language in early childhood. London, Cassell. 
Pica T. 1994 Research on negotiation: what does it reveal about second-language learning conditions, processes, and outcomes? Language Learning 44 (3): 493-527. 
Prator CH 1969 Adding a second language. TESOL Quarterly 3 (2): 95-104. 
Richards JC, Lockhart, C 1994 Reflective teaching in second language classrooms. Cambridge, Cambridge University Press. 
Sinclair J, Coulthard, M 1975 Towards an analysis of discourse: the English used by teachers and pupils. Oxford, Oxford University Press. 
Snow C 1977 Mothers´speech research: from input to interaction. In Snow C, Ferguson, C (eds) Talking to children. Cambridge, Cambridge University Press. 
Tomlin RS 1994 Repetition in second language acquisition. In Johnstone B (ed.) Repetition in discourse interdisciplinary perspectives. Norwwod, NJ., Ablex. 
Vila I 1987 Adquisición y desarrollo del lenguaje. Barcelona, Grao. 
245

Learner Corpora: Design, Development and Applications
Development of NLP tools for CALL based on learner corpora
(German as a foreign language)
Sandrine Garnier, Youhanizou Tall, Sisay Fissaha, Johann Haller
Institut für angewandte Informationsforschung
Institute of Applied Information Sciences (IAI)
Martin-Luther-Str. 14, D-66111 Saarbrücken
E-Mail: info@iai.uni-sb.de, http://www.iai.uni-sb.de
Telefon: +49 (0)681-38951-0 Fax: +49 (0)681-38951-40
Introduction
The project unideutsch.de is carried out by the DAF Institute in Munich (German as foreign language) in collaboration with IAI which is responsible for the parts described in this paper.
IAI was founded as a non-profit-making Research & Development (R&D) organisation in 1985 in Saarbrücken, Germany, to carry out the European EUROTRA machine translation project for the German language. Meanwhile IAI is an internationally acknowledged R&D institute in the field of multilingual information processing covering all aspects of natural language processing, computer-aided translation, and information management and knowledge management in advanced information technology environments. This is mainly done through strategic alliances with industry and national and European authorities.
At present, the institute's major R&D activity is in the area of multilingual language technology, in particular machine translation technology with special emphasis on domain-specific terminology and concept-based methodologies, multilingual information retrieval and information filtering, as well as general knowledge management. 1
Project uni-deutsch.de: description
Currently IAI is working on learning projects such as uni-deutsch.de, which combines our experience of NLP and the results of the research in foreign languages learning. The use of the technical advanced NLP tools provides another means of learning.
The main objective of the project is to devise an autonomous, long-distance, language learning system for advanced learners. The project is aimed further at:
  improving the language performance of advanced learners.
  boosting the learning process by giving vocabulary in certain specified domains such as science or education.
  checking automatically a text produced by a foreign language learner allowing them to learn by an interactive process which combines a human corrector and a machine corrector.
The machine corrector has the purpose of correcting spelling and grammar errors whereas the human corrector has to answer learner´s questions concerning errors found by the machine and to comment on other semantic errors.
The combination of these both methods allows for accelerated language acquisition through error based learning.
246
2 3 Terminology Computer based learning of vocabulary
In order to achieve the above mentioned aims, IAI has developed a programme, LiLa, - Linguistisch intelligente Lehrwerksanalyse – the task of which is to analyse textbooks with linguistic intelligence (texts, exercices and vocabulary listing).1
The programme is a combination of morphosyntactic taggers and parsers based on partial parsing techniques (Constraint Grammar) with developed linguistic resources in German. The different programmes break down the text into sentences and words.
This programme allows students to have a list of new vocabulary which is the result of comparing the existing vocabulary of a lexicon containing a list of basic words (level B1: Zertifikat Deutsch) with the vocabulary in a text.
The new entries are produced with grammatical information. LiLa indicates new nouns together with their gender information and plural form, and for verbs infinitive, present, imperfect and past participle.
Other words such as conjunctions are associated with their grammatical categories. All this information is essential for learners in order to improve their use of vocabulary. Consequently the student will then see from the list what they have still got to learn and what they should already know. LiLa provides a second lexicon which is the addition of the first lexicon to the other new words. This second lexicon can be then compared with another text.
Example of an automatically produced vocabulary listing:
abkühlen, kühlt ab, kühlte ab, abgekühlt
Abkühlung, die, Abkühlungen
absinken, sinkt ab, sank ab, abgesunken
adiabatisch (Adjektiv)
aerob (Adjektiv)
aggressiv (Adjektiv)
Alge, die, Algen
altern, altert, alterte, gealtert
Ammoniak, das
andauernd (Adjektiv)
anhand (Präposition)
anpassen, passt an, passte an, angepasst
Anreicherung, die, Anreicherungen
Argon, das
The listing can also be shown as one text where new words appear in color. Other words can be tagged, e.g. a spelling error, a proper noun etc.
Another programme which isn´t part of the project uni-deutsch.de, but which could also be useful for learning vocabulary is the programme AUTOTERM. This programme, based on statistics, automatically gives the terminology from specialised texts.
This could be useful when students are working in specialized fields in English, German and French.
1 http://www.spz.tu-darmstadt.de/projekt_ejournal/jg_07_1/beitrag/haller1.htm
247
Example of a text from the online newspaper Wissenschaft-aktuell.de (http://www.wissenschaft-aktuell.de/):
Eintrittspforten in der Hülle von Nervenzellen nachgewiesen [Neurowissenschaften]
Durham (USA) - Bisher glaubte man, dass die äußere Zellmembran gleichförmig aufgebaut ist, so dass an jeder Stelle der Zelloberfläche Moleküle in eine Zelle eindringen können. Jetzt haben Wissenschaftler der Duke University bei Nervenzellen ganz definierte Eintrittsstellen nachgewiesen. Die Regulation des Stofftransports in diesen Membranzonen ist für die Funktion der Nervenzelle wichtig. Weitere Untersuchungen der neu entdeckten Strukturen könnten daher helfen, Wirkstoffe gegen verschiedene Nervenerkrankungen zu entwickeln, schreiben die Forscher im Fachblatt "Neuron".
Nervenzellen geben über so genannte Synapsen Signale in Form von Botenstoffen an Nachbarzellen weiter. Diese Neurotransmitter binden an Rezeptorproteine an der Außenseite der Empfängerzelle. Die Rezeptoren werden ständig durch neue ersetzt, also in beiden Richtungen durch die Membran transportiert. Indem die Zelle diesen Austausch reguliert, kontrolliert sie die Zahl der Rezeptoren und damit die Empfindlichkeit, mit der sie auf die Neurotransmitter reagiert. Wirkstoffe, die diesen Transport hemmen oder beschleunigen, könnten als Medikamente zur Behandlung von Depressionen, Epilepsie oder Suchterkrankungen eingesetzt werden.
Die Arbeitsgruppe von Michael Ehlers wollte verfolgen, wie die außen sitzenden Rezeptormoleküle wieder in die Zelle gelangen. Dazu markierten die Wissenschaftler das Protein Clathrin mit einem Fluoreszenzfarbstoff und gaben es zu Kulturen von Nervenzellen. Clathrin lagert sich an die Stellen der Membran, an denen sie sich einstülpt, um Proteine nach innen zu transportieren. Die mikroskopische Auswertung ergab, dass sich das Clathrin nicht gleichmäßig verteilt sondern nur in bestimmten Zonen auf der Membran ablagerte und damit definierte Eintrittsstellen markierte. "Wir haben herausgefunden, dass die Nervenzelle mit einem Zimmer vergleichbar ist, in das nur ganz bestimmte Türen führen", sagt Ehlers. Bisher habe man die Zellmembran eher für einen an vielen Stellen durchlässigen Vorhang gehalten.
Während sich die Verteilung der Clathrin-markierten Eintrittsstellen bei jungen Nervenzellen häufig veränderte, blieb sie in reifen Neuronen weitgehend konstant. Möglicherweise sei das ein Ausdruck für die im Alter nachlassende Anpassungsfähigkeit des Gehirns, vermutet Ehlers. Die jetzt nachgewiesene Membranzone könnte nur die erste von weiteren noch unbekannten speziellen Membranstrukturen sein, glauben die Forscher.
Author: Joachim Czichos
Source: EurekAlert
Extract of an automatically produced terminology listing:
Anpassungsfähigkeit
Anpassungsfähigkeit des Gehirns
Ausdruck
Austausch
Außenseite
Außenseite der Empfängerzelle
Behandlung
Behandlung von Depressionen
Behandlung von Epilepsie
Botenstoff
Clathrin
Clathrin-markierte Eintrittsstellen
bestimmte Tür
bestimmte Zone
definierte Eintrittsstelle
248
Example of the dialoge interface showing the nouns and noun groups found in the text:
4
Error-based self learning
According to some authors (Prof. H. J. Heringer) students make the same or similar errors which are repeated. If the student knows what type of errors he constantly makes, he could then accelerate the process of language acquisition. This idea is the basis of the development of specialized programmes for foreign language learners.
Several programmes are connected together in order to give the structure of a German sentence. The different information which is given by the programmes is finally processed by an additional programme, KURD, which has the task of finding syntactically or morphologically incorrect structures in the German sentence. KURD is "a formalism for shallow post morphological processing" and uses "the input from the morphological analyser MPRO".2 It works on grammar checking and style control (German & English).
This programme uses several operators such as unification and deletion to add and delete information in the linguistic representation of the sentence in order to mark one or several errors in the sentence. Formalism rules have been developed to add information in the analysis of the most common structural mistakes made by learners of German. The new information is then processed by another programme which tags the errors found in the text.
2 http://www.iai.uni-sb.de/iaide/en/kurd.htm
249
Example: The position of the verb in a subordinate clause in German, where the verb must be at the end of the sentence. The error message is "Subordinate clause: verb at the end".
4.1 4.2
Error messages
This objective aims at developing the necessary tools for the system to evaluate the task done by the learners and give them feedback on their performance. Qualitative methods (morphosyntactic tagger and syntactic parser) allows a linguistically based analysis of the texts produced by learners.
Error messages can be more precise due to linguistic analysis. This method replaces the straightforward right/wrong answer or the comparison of pattern structures and instead provides a detailed message to help the students correct themselves. The type of error message can be adapted to the linguistic knowledge of the user.
Example 1: the genitive form in German.
Sentence: “Der Stil des schweizerischen Autors des langen Textes ist langweilig.“
The error message is the following: “too many genitive forms.”
Example 2: wrong participle.
Sentence: “Ich habe das gebrach.”
The error message is the following: “Wrong participle”.
Work basis: learner corpora
In order to develop rules adapted to foreign learners we had to collect errors made by speakers of other languages. The sentences collected were written by Socrates DAF students and students from Burkina Faso, Vietnam and Russia.
250
We have also used material from Professor Heringer. He has listed different incorrect types of sentences in the book Fehlerlexikon and on his web-site.3 From these sources we have classified different error types in order to help us formalize them.
A lot of errors made can be automatically found:
  spelling errors
  grammatical errors
  some valence errors.
What we can´t find at the moment in most cases are the valence. Work on this is currently in progress and we hope to resolve it during the coming year.
Classification: structure and some examples
H stands for examples from Prof. Heringer, I for IAI, S for Socrates students and K for correct structures. The sign + means that IAI is able to detect the wrong structure and gives a corresponding error message, whereas the sign – means that we are not able at the moment to detect the error.
2.4. Wortbildung
2.4.1. Partizip von starken Verben
+H: Ich habe Münzen zu den Leuten geworft.
+H: Danach hat er ein Rezept geschreiben.
+I: Ich habe das gebrach.
2.4.2. Partizip von schwachen Verben
+H: Am Sonntag bin ich um 10 Uhr aufgewach.
+H: Sofort habe ich eine Wohnung gemiet.
-V: Die Stadt wurde überschwommen.
2.4.3. Kein ge- bei nicht trennbaren Präfixverben und Verben auf -ieren
+H: Er hat ein Zimmer für seinen Vater gebestellt.
+H: Er hat dort Medizin gestudiert.
2.4.4. Falsche Konjugation
+S: Man verglicht.
+U: Mit Sicherheit geb es auch künftig große Kraftfahrzeuge, aber die Besitzer von kleinen Autos würden lächeln, weil Sie durchkommen viel schneller durch den Straßenchaos.
2.4.5. Komposition
+K: Beschwerdeführer
+I: Beschwerdenführer
-I: Beispielssätze
-U: Statt eines 20-Minuten-Taktes sei in Basel ein 6-Minute-Takt eingeführt werden.
The result of the analysis can be seen in unix as well as in winword. The userfriendly interface shows the tagged wrong structure and gives a message which helps learners to correct themselves. The programme can also give a correct example or several options of correction.
You can find the site at: http://www.uni-deutsch.de.
The internet address of the DAF Institute is the following: http://werkstadt.daf.uni-muenchen.de/ home2.htm.
3 http://www.philhist.uni-augsburg.de/Faecher/GERMANIS/daf/Forschung/fehler/analyse.html
251
Conclusion The NLP tools will be a powerful additional means for the acquisition of a foreign language. Computer-assisted-language-learning seems to be a good solution today to the individual e-learning of languages and helps the human corrector, who can then be more aware of semantic and style problems. The CALL must be seen as another option and be added to the other traditional means of acquiring a foreign language. What we have described in this paper, is just a beginning. Special grammar rules for specific grammar or vocabulary exercises, integration and comparison with predefined answers into grammar chapters and dictionary could be the next task in order to customize the learning process. Books Garnier S, Tall Youhanizou, 2003 Uni-deutsch.de Report, forthcoming, IAI Heringer H. J. 2000 Fehlerlexikon Deutsch als Fremdsprache, Berlin, Cornelsen Verlag. Internet pages Haller J 2002 Linguistisch intelligente Lehrwerksanalyse. In Zeitschrift für Interkulturellen Fremdsprachenunterricht, 7(1), 6 pp. http://www.spz.tu-darmstadt.de/projekt_ejournal/jg_07_1/beitrag/haller1.htm (stand 02.13.2003) Michael C 2001 KURD, 10 pp . http://www.iai.uni-sb.de/iaide/en/kurd.htm (stand 02.13.2003) Heringer Hans Jürgen, 1999, Aus Fehlern lernen (Multimediales Computerprogramm zur Fehleranalyse, mit Grammatikteil), Augsburg http://www.philhist.uni-augsburg.de/Faecher/GERMANIS/daf/Forschung/fehler/analyse.html (stand 02.13.2003) 252

The company women and men keep: what collocations can reveal about culture 
Sara Gesuato 
Dept. of Anglo-Germanic and Slavic languages and literatures, 
University of Padua, Italy 
1. Introduction 
Habitual ways of talking about given phenomena show people's attitudes to and conceptualizations of them: the meanings associated with certain events or situations determine the cultural significance of given topics within a group of speakers (cf Fairclough 1992, Hodge and Kress 1993). In particular, a language's keywords give insight into the culture of its speakers (e.g. Krishnamurty 1996, Partington 1998, Wierzbicka 1999); but actually all the words repeatedly circulating in a community form constellations of repeated meanings conveyed in typical patterns that produce conventional expressions and opinions. Therefore, any word recurrently used in interaction may carry social implications, even if these may go unnoticed among the interactants (see Tognini-Bonelli 2000: 125). To understand the social salience and semantic nature of the terms exchanged in verbal interactions, it is useful to look at their recurrent patterns of usage, namely their occurrences in co-text (see Stubbs 2001). 
In this paper I examine collocations of WOMAN, MAN, GIRL, and BOY (from the Usbooks, Ukbooks, Time and Today components of the Cobuild on-line corpus) to determine if their denotational symmetry (identification of adult and non-adult human beings of the female and male sexes) is paralleled by similar patterns of usage (association with the same semantic fields and embodiment of the same semantic roles in given syntactic environments), i.e. if they belong to similar discourse domains. 
2. Most significant collocates 
The lexemes under examination do not share all of their most representative collocates (i.e. words they co-occur with in a statistically significant way), which reveals that they are part of different domains of discourse. 
The top 50 significant collocates of WOMAN include terms that fall within the discourse domains of physical attractiveness (beautiful, pretty, attractive), age (young, old, older, elderly, aged), physical appearance (body, dressed, looking, white, black), family and personal relations (married, pregnant, singles), women's liberation (rights, movement), religion (pontiff, ordination), (groups of) people (woman, women, girls, man, men). Other collocates comprise said, drowned, listening, named, division, percent, working, golf, title, doubles, lives, and won. 
The wordforms man and men co-occur with words exemplifying some of the semantic fields identified for WOMAN: age (old, older, aged, young), physical appearance (white, black, tall, big, fat), family and personal relationships (married, singles), and (groups of) people (woman, women, men). Other collocates are relevant to distinct domains of discourse: negative states (blind, dead, poor), the military (officers, enlisted, squad), sex (gay, sexual), negatively connoted physical force or action (violence, arrested), non-physical attractiveness (rich, kind, nice, good, wise). Among the remaining collocates, some are common to WOMAN as well (division, percent, working; labeling: title, named), while others are peculiar only to MAN (burned, made, isle, saw, matc). 
The data show that both WOMAN and MAN are described in terms of such notions as age, physical appearance, family and personal relationships. On the other hand, WOMAN and MAN partly occupy different semantic space. First of all, MAN is relevant to a wider range of semantic fields, which points to a more versatile use of its wordforms. In addition, even when both lexemes are relevant to the same domains, these may be highlighted in different ways. For example, in the domain of physical appearance the collocates of MAN stress the notion of ‘size’ while those of WOMAN that of ‘overall qualitative effect', as if to suggest that a person's visual impact on others is more relevant to women than men. Moreover, the euphemistic term referring to old age (elderly) is reserved to WOMAN, which signals a mild taboo in the co-occurrence of the notion of ‘femaleness’ and ‘age'. Furthermore, WOMAN co-occurs with a greater variety of terms denoting people, indicating that relationships or interaction are more often discussed with reference to women. Finally, WOMAN and MAN occur in partially complementary distribution; the former lexeme is associated with the domains of physical attractiveness, religion, civil rights, and involuntary actions (drowned, won), the latter with those of non-physical attractiveness, the military, violence, and voluntary actions (burned, made), which shows that distinct areas of activity appear to be salient to the two gender groups. 
The terms occurring with GIRL and BOY reproduce patterns very much like those outlined above. The collocates of GIRL are representative of the semantic fields of age (little, year, young, teenage, baby, aged, old, adolescent), personal relationships (boy, girl, father, married, lover, boys, women, girls, men), physical attractiveness (lovely, pretty, beautiful, beauty), negative state (poor, 
253
raped, died, dead, killed, murdered, screaming), positive state (nice, golden, jolly, pregnant), labeling (named, called), and physical appearance (black, white, haired). The collocates for BOY refer to the domains of age (little, old, year, young, baby, aged, teenage, older), size (small, big), relationships (girl, girls, boy, boys, men, brigade, club, posse, scouts, scout, mother, father), negative state (died, poor, dead, bad), positive state (golden, good), activities (school, game, play, jobs), labeling (named, called), and outdoor events or fun activities (beach, pet, shop, do). The collocates reveal that GIRL and BOY show up in partially different discourse domains. The former lexeme is relevant to the topics of physical appearance and attractiveness, which suggests a young woman's outward characteristics are paid attention to. GIRL is also often associated with participles which denote dangerous or hopeless conditions and convey ideas of helplessness and victimization. Finally, GIRL may occur in contexts of talk focusing on adult relationships (see married, lover, pregnant), which indicates that such notions are considered appropriate even to young females and/or that GIRL is also used in partial contradiction with its denotational meaning, that is with reference to not very young women. The lexeme BOY, on the other hand, is more relevant to the notions of behavior (good, bad), bonding (brigade, club, possse), active involvement (game, play, jobs) and associated with outdoor contexts (scout, beach). This lexeme, therefore, occurs in contexts focusing on the idea of direct participation in concrete activities. 
In general, the collocates of WOMAN, GIRL, MAN, and BOY show that these lexemes share certain contexts of use, but also that there are discourse domains pertaining only to WOMAN and GIRL, and others only to MAN and BOY. The distinct sets of collocates suggest that WOMAN and GIRL are more frequently associated with notions of passivity and physicality, and MAN and BOY with those of activity and general behavior. 
3. Descriptive labels 
The descriptive labels attached to a noun specify the characteristics of its referent and actually narrow it down by identifying a subset of it. A language's typical descriptive labels are attributive adjectives, but a similar categorizing function is also played by post-modifiers like relative clauses and of-headed prepositional phrases. 
3.1. Attributes 
Adjective+noun combinations represent a type of colligation or association between grammatical categories. When examined in high numbers, they can be semantically revealing because they show what properties (as encoded in the pre-nominal adjective or other modifier) are frequently or by default ascribed to what entities (encoded as nouns). The following findings are based on a sample of the relevant Cobuild data (100 concordances per each wordform woman, man, girl and boy), systematically selected through the software running the corpus (one every a given number of occurrences determined by the program). 
A comparison of representative samples of attribute+BOY vs. +GIRL combinations reveals that most of the attributes preceding both lexemes are of three types: they mention physical characteristics (little, grown, blind), refer to behavioral traits (good, brave) or identify the group membership of the relevant referent (Jewish, factory). But despite this basic similarity, the collocates of GIRL and BOY also reveal differences in the usage of those terms: GIRL is associated with the notion of beauty, encoded through various terms (beautiful, lovely, pretty), four times as frequently as BOY, which co-occurs with pretty only once; in addition, only GIRL is occasionally associated with attributes that signal its referent is thought of as an adult engaged or involvable in heterosexual relationships (call-girl, flirtatious girl, and had a steady girl, is going to find a nice girl and marry her). On the other hand, more varied terms are available for a description of the behavior and personality of the referents of BOY (bully, charming, different, dull, fine, funny, nasty, problem, suitable, wonder) than those of GIRL (dangerous, hard, material, naughty, sympathetic). Furthermore, if the attributes referring to physical attractiveness are excluded, it turns out that those referring to boys are also more evenly distributed between the categories of positive and negative connotations. Finally, BOY is accompanied by attributes referring to intellectual abilities (clever, dull, sharp) three times as frequently as GIRL (smart). 
The attributes preceding WOMAN and MAN are more varied than those of GIRL and BOY, and identify, e.g., states or conditions experienced by their referents (pregnant, dead, forgotten), their area of origin (Palestinian, Yale), permanent properties (Catholic, religious), traits of behavior (liberated, gambling), physical and emotional characteristics (tall, big, broken, loud, naked), and group or category membership (neighbor woman, company man). However, although the general patterns of usage are similar, the specifics of these descriptive labels point out subtle differences in the ways people talk about women and men. 
Attributes of WOMAN often signal negative or demeaning characteristics of the lexeme's referents . as revealed in other contexts as well (see sections 4.1. and 5.2.) . namely, women's past or 
254
present role as victims (liberated, helpless), mental or emotional instability (deranged, vengeful), lack of control on their behavior (loud, broken), limited intellectual ability (stupid), socially marginalized love life (divorced, spurned) or low-qualified professional position (cleaning woman); the salience of their physical attributes (beautiful, fair, fine, full) and the non-prototypicality of their playing certain social roles (woman boss, neighbor woman) are also highlighted in the concordances. Neutral or positive attributes are infrequent (black, tall, rich, real, sensible). 
Attributes of MAN signal neutral, positive, and only occasionally negative characteristics of its referents, which indicates that people's discourse practices provide a more balanced picture of men's personalities and behavior. Pre-modifiers of MAN emphasize men's importance and power (big, powerful, wealthy), appropriate conduct (cautious, devout, helpful, proper, religious, spiritual, suitable), general likeability (great, fantastic, decent, lovely, handsome, dishy), education (cultured), cleverness (witty), independence (bold, free [cf liberated of WOMAN]), modernity and authenticity (modern, plain man of the people), satisfaction (chuffed), and emotional success (ladies man). Negative attributes may refer to conditions the referent of MAN is not responsible for (blighted, harassed, dying), denote reprimandable behavior (marked, violent, strange, gambling) or point out lack of socially desirable qualities like out-goingness (timid, shy). Neutral attributes are those identifying types or groups of men performing specific functions in given contexts (ball man, in the context of badminton; best man, in the context of wedding, radio man, in the context of communication) or temporary or permanent physical characteristics (naked, hooded, white). 
In general, while a variety of attributes are available for describing both females and males, those associated with the former are more frequently negatively connoted, restricted to fewer domains of discourse or focused on the decorative function of women. On the other hand, the attributive collocates of MAN are typically more numerous and varied, equally reveal positive and negative qualities of their referents, and highlight either socially important roles played by men or their intellectual values. 
3.2. Of-headed prepositional phrases 
More of-headed prepositional phrases modify BOY (100) and MAN (425) than GIRL (82) and WOMAN (218). Most can be grouped under the same headings, i.e. they classify their referents in similar ways. Those attached to boy and girl mostly refer to age (When I was a boy of 13; A girl of four saved her family yesterday); only 3 of the concordances of girl and 1 of those of boy refer to other characteristics of their referents (Alex is a girl of very high morals; A Girl of the Streets; he was a boy of mercurial nature). The phrases describing boys and girls are more varied and some are shared: they include reference to age (Dr. Stuart Horner said girls of 12 could be donors; at Middlesex University found that boys of 13 or 14 who played computer), place of origin (Fox suggests that the girls of Jerusalem speak the first part; compared to the big boys of Hollywood), and group membership (and said the girls of the horde learned their walk; but even more the boys of his own class). In addition, the concordances of girls exemplify only one other kind of descriptive phrase, which indicates what girls may possess (a description of bombed houses in The Girls of Slender Means). The other concordances of boys, on the other hand, specify a few more types of characteristics of their referents: character (The right type of boys of a type - to appreciate her), job (the smart boys of the press), epoch (all the other little boys of his generation), condition of being somebody's “property” (They have four boys of their own), identification with reference to a topic (the seven Hard Flint boys of the Navajo myth; one of the boys of Old Woman Running). 
The concordances of both woman and man reveal the same types of classifications: both sets describe their referents in terms of their age (anyone who thinks that a woman of 40 plus is undesirable; Just as a man of 70 probably can't lift as much), good qualities and bad traits (she was a woman of considerable literary ability; a woman of dubious loyalty; he was not a man of achievement, but a man of promise; a man of disgusting morals), physical features (You're a woman of medium height; if it is possible for a man of his size to be peripheral), group membership (a woman of the horde; a man of the tribe called the Westfolk), possession (the title of her book A Woman of Substance; would be a difficult matter to find a man of any property in the country), character (a woman of her word; a man of constant surprises), origin (a woman of England called Agnes; a man of the north), importance (a woman of rank; might well have become a man of destiny); only woman, however, has concordances describing its referent as a man's “property” (He's the sort that needs a woman of his own), and only man has concordances mentioning its referent's work (curator of the museum had to be a man of the pen and of the book; We might have had Msgr Montini a man of letters) and epoch (Only a man of Colonel Sartoris’ generation). In general, phrases that describe non-physical characteristics make up about 60% of the concordances of man and 42% of those of woman. Furthermore, of the concordances mentioning positive qualities, only 5 (i.e. about 8%) of those relevant to woman include intensifiers (a woman of singular attainments; A woman of strong convictions!), while 72 (i.e. about 45%) of those relevant to 
255
men are pre-modified in this way (a man of charismatic personality; a man of considerable intelligence; A man of evident refinement; A man of formidable intellectual gifts; A man of great courage and determination; a man of immense patience; a man of international calibre). 
The concordances of women and men reveal the same patterns as those identified above for woman and man: their embedded prepositional phrases describe the referents of those wordforms along very similar lines, but to different degrees. Age is mentioned 26 times (i.e. about 17%) with reference to women and 11 times (i.e. about 4%) with reference to men (can be used on women of all ages; four units a day by men of all ages); physical characteristics show up 4 times with reference to women (i.e. about 2.5%) and 3 times (i.e. about 1.1%) in the case of men (This chapter is meant for women of every size; place the onus on men of Cleese proportions); the time or generation a person belongs to is indicated more often in the vicinity of women (13 times, i.e. about 8%) than men (8 times, i.e. about 3%; the women of that 1960s generation; men of an old generation); the geographical, national and/or family origin of women is referred to more frequently than that of men (58 times, i.e. about 37% vs 57 times, i.e. about 21%, respectively; All the women of the valleys had found some pride; labourers assumed that men of Anglo-American origins); group membership, on the other hand, appears to be more salient to male than female referents, the frequence of occurrence being 31 times (i.e. 20%) for the former and 83 times (i.e. about 31%) for the latter (that was unheard of among women of the aristocratic classes; to the men of the B company it felt as if); the specification of people's non-physical qualities is seemingly equally relevant to women and men, since it is to be found in 19 concordances of women (i.e. about 12%) and in 44 concordances of men (i.e. about 16.5%; it stresses the importance to women of good qualifications; was led by men of enduring stature); however, there are also prepositional phrases accompanying men that mention other non-physical characteristics of their referents, like profession (13 times, i.e. about 5%; we men of business), type, personality and/or behavior (27 times, i.e. about 10%; For men of the stamp of Scheer), level of importance (9 times, i.e. about 35%; You find it a lot with men of power); overall, therefore, men are described in more varied detail than women through of-headed prepositional phrases. Finally, the positive characteristics of women are enhanced by intensifying adjectives slightly less frequently than those of men, namely 10 times (i.e. about 6%) vs 15 times (i.e. about 5%), respectively (women of strong will; as free men of definite and sincere worth). 
In conclusion, the of-headed prepositional phrases post-modifying GIRL, BOY, MAN, and WOMAN present comparable, but not identical, kinds of specifications; for one thing, some of these do not occur with the same frequency across the concordances, that is physical attributes appear to be more salient to female than male referents, while non-physical characteristics are more tightly associated with male than female referents; in addition, certain types of prepositional phrases are not always available to describe females, for instance those mentioning jobs; finally, the positivity of certain attributes tends to be more strongly emphasized (through adjectives) when the words encoding them collocate with terms marked for maleness. In brief, GIRL, BOY, WOMAN, and MAN are not equally salient to the same types of concepts used to describe their referents. 
4. Roles and situations 
A clause is the verbal representation of an event, that is the linguistic rendering of a phenomenon in the world. It pivots around a predicate, which represents what is going on, and may include nominals, phrases, and adverbials that refer to entities and circumstances that contribute to making up that event or situation. It is often possible to verbally represent the “same” event in various ways; the specific lexico-syntactic wording chosen will highlight certain aspects of it and obscure others (see Halliday 1994: ch. 5). Examining clauses in which WOMAN, MAN, GIRL, and BOY occur will help identify what events and situations they are most typically associated with and what roles their referents are represented as embodying. 
4.1. Relative clauses 
A relative clause represents an event or situation in which the referent of the relative pronoun is a main participant. In the corpus, I looked for examples of who-headed relative clauses, in which, that is, the referent of the pronoun would most often be the subject or direct object of the clause, and thus likely to play the semantic role of agent or patient, respectively. From the output of each of my queries, I had the system select a random representative sample of 100 concordances. 
The result of the girl+who search presents three interesting findings: (1) 34 of the concordances describe negative situations that the referent of the relative pronoun is affected by or, occasionally, responsible for (strongly enough to want to murder the girl who'd informed them; the five-year-old girl who escaped the massacre; a troubled girl who felt pressures at home; I have a neurotic girl who is panic-stricken; a severely disabled girl who is unable to speak; I had a girl who was a Down's Syndrome's baby; the subject of the aggressive little girl who had had a ‘perfect birth'; a young 
256
Monk shelters an Albanian girl who is being hunted for the murder). (2) Moreover, 7 concordances show that girls may be talked about with reference to their bodies, especially their physical attractiveness, occasionally seen as a source of problems (period: puberty. What happens to a girl who begins to menstruate at 10; She was really pretty like that. A girl who feels pretty has a tendency; a girl with breasts and a girl who's skinny; in the case of a teenage girl who is ashamed of too little breast), and 9 more associate girls with the notions of romantic and/or sexual involvement, that is in (a not always happy) relation to some man (We found a Romanian girl who is married to a local man; for a little girl who loved him; a girl who fails to marry her suitor; a Mexican girl who eventually jilted him). (3) Finally, 79 concordances reveal that the subject of the relative clause plays the role of an experiencer of situations, events or emotions, or the patient of others’ actions, which may even be unpleasant (a Colorado girl who becomes an opera singer; A girl who lived with her parents; She was a vivacious, outgoing girl who was always enthusiastic; a girl who will never forget a friend; a girl who shares his interest in sport; A teenage girl who collapsed; The girl who died was her friend; the handcuffed girl who was being pushed; A girl who was raped and strangled); on the other hand, when playing the role of an agent, the situation the referent of girl is in charge of is often negative, vilifying or restricted to the field of love-relationships (A heart-swap girl who battled against all the odds; did not bode well for the girl who had invited Kurt to lunch; Except that the girl who told the tale had been expelled; the black girl who'd killed herself; bordello in Lima, Noriega asked a girl who'd just finished making love; Paula shot to fame as the girl who ditched all her lover's gifts). 
The 100 concordances of girls+who reveal very similar patterns: 32 depict negative events or situations (it may be that girls who are weak academically; his daughter was attacked by girls who are fellow pupils; to throw stones at the Chinese girls who broke world records; girls who develop eating disorders fail; This was very hard on the girls who had paid for their machines; the poor and unfortunate girls who walk the streets; child labor and raw immigrant girls who would work for next to nothing; nervous girls who never laugh); 17 concordances mention girls in relation to sex, the emotions, or careers in the show business (while girls who are sexually active are loose; more than half of the girls who become pregnant before age 18; and happily-ever-after endings, girls who buy into this script; suspicious of the girls who want commitment; one of the other girls who crossed his bachelor path; scores of local girls who ended up marrying; girls who are cheerleaders; where modeling is at now, for girls who could once only dream of owing; Hollywood's pick of the girls who look set to become tomorrow's); 80 lines of concordance portray girls as experiencers or patients, sometimes in negative contexts, and not always in control of themselves (girls who are average in development; girls who have good communication about; girls who had sat in that chair; three girls who arrived rather apprehensively; girls who looked pale and unhealthy; delicate girls who enjoy shopping; those girls who most fully accept; memories of the girls who'd drowned; girls who were adopted in the 1960s; girls who were sacked; quite young girls who must have been the victims; fair young girls who trembled; the three girls who were panicking); when presented as agents, girls do not appear to always cut a good show (girls who drop out most often do so; He caught two other girls who were cheating; they were middle-class girls who, to revolt against; there are far more girls who say they would rather be boys; ran a network of call girls who serviced French high society). 
Of the 200 collocations of the combinations woman+who and women+who, 49 present women as facing or, occasionally, causing difficult circumstances (frustrating it must be for women who couldn't do what they wanted; I know there are many women who endure similar treatment; women who have abnormal Pap smears; only against illegal practitioners, women who obtained abortions; talking to the other women who shared some of their concerns; author of Women Who Love Too Much; mortally wounded a young woman who was watching the crowds; an unmarried, unemployed 24-year-old woman who is struggling to feed her four; warned against women who are always fawning; in fact, it was the woman who instigated the affair; a woman who is forever scaring; the woman who made his life a misery); 157 concordances depict women as being passively involved, as experiencers or patients, in situations determined by external circumstances or run by more active participants; the verbs most commonly accompanying the wordforms woman and women are indeed be and have (women who are infected with HIV; women who have had their ovaries removed; battered women who explode into loss of control; the number of women who are encouraged to do so; a young woman who disappears after meeting him; woman who had been infected); however, when presented as agents, only occasionally are women presented as involved in negative situations (the woman who'd rented Samelu his brickworks; the woman who had administered the test; the woman who tried to kidnap April; for two young women who have booked to go abroad; the older women who supported Mother Marie; Only one in ten women who runs a business works fewer); 51 occurrences show women in association with the domains of emotions, sex, and physical appearance (a young Roman woman who begins a passionate affair; was one of the women who adored their husbands; a woman who, along with sexual aversion; a fair number 
257
of women who have never had an orgasm; lovely succulent woman who should be eaten as whole; a raven-haired, proud-breasted young woman who has risen from a humble; the woman who brings back the elegance; particularly for women who may be watching their weight). Also, the notions of negativity and passivity often co-occur and the semantic fields that are specially “reserved” to women are negatively connoted (women who are incapacitated; unmarried women who aren't prepared to occupy; women who have been widowed; women who have bulimia; a woman who discovers a breast lump; single woman who was attacked; woman who is afraid of flying). 
The 200 concordances of the wordforms boy and boys also include reference to negative contexts, but less frequently than those of girl and girls, namely on 30 occasions overall (angry, needy and frightened boy who looks to women to mother him; Gregory was a loner, an Afrikaans farm boy who had been neglected; eight-year-old retarded quadriplegic boy who was murdered; motivate pupils, especially boys, who fail at school; prison sentences imposed on the two boys who killed James Bulger; neighbours also joined boys, who had set fire to curtains); interestingly, these negative contexts occasionally involve females more directly than males (it is girls rather than boys who are at greater risk; a boy who saw his grandmother killed). In addition, only 10 are the concordances which somehow include reference to sex, physical appearance or feelings (many teenage boys who feel homoerotic urges; for high school boys who are too skinny to play; A 16-year-old boy who's never had a girlfriend). Finally, more numerous in these sets of data, namely 99, are the concordances that include representations of events in which boys are actively participating (incredibly wiry boy, who could climb the highest pines; from a boy who first lifted a tennis racket; said the boy who had brought the coffee; I had trained a boy who had won a scholarship; punched and kicked by a boy who had knocked on her door; crowds are made up of boys who play sports; were moistened by two small boys who pressed sodden cloths against; and the boys who take the field will be very); 15 of these portray boys as involved in or causing problems (an eleven-year-old who has stolen some dried figs; a wild boy who poached rabbits; the boy who took the most drugs; care for boys who had committed offences; pension book stolen by boys who tricked their way into his East). 
The wordforms man and men display collocational patterns similar to those of boy and boys: 45 of the 200 concordances considered refer to negative situations (A man who appeared to be drunk; What rung for the man who has been unemployed; Here we are in a room with a man who's been charged with murder; about the same frequency in men who are insuline-dependent; one of the families of the SAS men who died on the mission; in black balaclavas attacked two Asian men who were lying on the floor). Reference to the domains of love or sex life and beauty are limited to 17 concordances (he's the man who sleeps his way to the top; I am the type of man who can give to more than one woman; Clearly, a man who feels sexually attracted to; can't compete for women with men with flourishing forelocks; a troupe called The Hollywood Men, who dance in skimpy briefs; twice as likely to get divorced as the men who married later). Furthermore, 97 are the concordances showing men taking an active role in the events being represented, and only a dozen of these refer to negative circumstances (the man who brought you The Sound of Music; why I'm the man who helped fill the missing link; lost their lives trying to save a man who jumped in the water to rescue; a chapel man who later spoiled himself by drink; men who ally themselves with England; were written by men who had fought in it; half of the men who immigrated to Australia; prosecute the men who sexually harass me at work); only 4 are the instances of passive clauses (the man who had been certified dead; two men who had been killed in the quarry). Finally, quite frequent, i.e. 40, are the concordances that show men in a positive light, whether they are presented as determining or simply experiencing the positive circumstances relevant to their lives (This is a young man who believes in the future; the son of a man who had risen from beginnings; paid their last respects to the man who made Jaguar a world leader; and so is the man who now seeks supreme power; this is the man who won over the public; He was a loving man, a man who would do anything for anybody; the universal regard for men who had risked their lives to defend; Toward the men who held national power; eloquent men who stand above the crowd; two men who turned their dream into reality; the only two men who could win the title). 
The above data suggest that while WOMAN, GIRL, MAN, and BOY share similar contexts of usage, they do not occur in those contexts with the same frequency: it is more often the referents of WOMAN and GIRL than those of MAN and BOY that appear to be involved in difficult situations, affected by external circumstances or other agents’ interventions, and associated with discourse domains that draw attention to a female's physical attractiveness and her role in relation to a male. 
5. Semantic preferences 
The notions people exchange in interaction when talking about females and males can be revealed in part by examining their collocates in syntactic environments likely to introduce topics for discussion. Two such constructions are the genitive and for-headed prepositional phrases. 
258
5.1. Genitive 
The semantic relation holding between ideas expressed by nouns can be of various types. When these nouns are grammatically linked by the genitive, they can express such notions as possession (e.g. my neighbor's house), personal relation (e.g. Frank's son-in-law), classification (e.g. a children's story), origin (e.g. John's telegram), authorship (e.g. Wilbur Smith's latest bestseller), agency (or subiect-verb relation, e.g. Mary's consent), passive experience of an event (e.g. the town's destruction), predication of a state relative to an entity (or subject-complement relation, e.g. Paul's happiness), and more. 
Of all the types of phrases examined in this study, the genitive is the one that reveals the least striking categorial differences between the collocates of the pairs of gender-marked lexemes. The concordances obtained for wordforms girls (92) and boys (86), and representative samples of 120 concordances for girl, boy, woman, women, man, and men show that the words following the lexemes in question (almost exclusively nouns or noun phrases) convey information about the group membership, physical characteristics, thoughts or attitudes, feelings, states, qualities of the possessee, and also about the objects possessed or actions performed by them, the events, experiences or abstract notions relevant to them or finally their relationships with others. 
Neither the concordances of boy nor those of girl exemplify the notion of group membership. All the other concepts are represented, although to varying degrees within each set. For girl and boy, respectively, the frequencies of occurrence of the various notions, given in percentage, are as follows: event (about 3% vs 6%; Girl's 24-hour van rape ordeal; Boy's 130mph joyride), relationships (about 13% vs 7%; Last night the girl's grandmother said; sent the boy's aunt Marlene in her place), physical characteristics (about 42% vs 26%; the desert area where the little girl's body had been found; He crossed to Mack and touched the boy's arm), actions (about 14% vs 10%; to detect a euphemism for the girl's ‘welcoming’ her lover into her; for his appointed visit. The little boy's coming back), feelings (about 3% vs 0.8%; the girl's love is said to be better; the latter start fighting for the boy's affections), thoughts (about 4% vs 6%; because of a girl's mistrust of one's courtiers; one boy's desire to become an actor), states (about 2.5% vs 4%; to ensure the girl's safety; she said the boy's presence made teaching the class), qualities (about 0.8% vs 4%; a fraction of a 13-year-old girl's energy and concern; If you can tell a boy's character from the company he), objects (about 9% vs 13%; Leaphorn checked the girl's duffel bag; The boy's bicycle was found in a pigsty), and abstract notions (about 3% vs 9%; only partially to do with a girl's psychological make-up; it became the boy's duty). It appears from the above data that certain types of notions cluster more frequently around girl (actions, feelings, relationships, and especially physical characteristics), while others more typically accompany boy (events, conditions, thoughts, qualities and abstract notions, and especially possession of objects). In addition, more frequently associated with girl than with boy (i.e. about 26% vs 3%) is the reference to all of the following domains taken together: love (may be rivals for the girl's affection; men's and boy's [sic] dominance in sexual relationships), sexual attributes (spending the night between the girl's breasts; keep secret a boy's description of his genitals), clothing (a girl's dress is displayed like an exhibit). Finally, differently from other sets of concordances, reference to negative situations is only slightly higher in the case of girl than boy (i.e. about 26% vs 22%; Moors murder girl's grave wrecked by ‘sick’ gang; Four charged after ecstasy boy's death). 
The frequency of occurrence of the semantic categories identified above is markedly different among the collocates of both girls and boys. The genitives of girls do not include reference to the notions of qualities or thoughts, and mostly concentrate on those of events (about 16%; choir concert, night out, reunion, race, party) and “possession” of objects (about 36%; bedroom walls, cabin, dressing room, home, independent school, tent, toy). On the other hand, the concordances of boys do not represent the category of state, while they focus on those of group membership (about 25%; Brigade, Choir, club, network, soccer team) and possession of objects (about 37%; boarding school, camp, dormitory, house, locker room, old baseball caps, test room). Also, the concordances of girls and boys represent negative situations much less frequently than those of girl and boy (both about 8%; I saw the girls’ tent had collapsed; too familiar with the terrors of the boys’ locker room). Finally, the association of the referents of the above wordforms with the notions of beauty, love, and sex is reduced to about 10% in the case of girls (he knows that the girls’ beauty will burn out a lot quicker; all Catholic girls dream of falling in love with; Girls’ interest in sexuality) and not attested at all in relation to boys. 
The data about WOMAN and MAN reveal that the wordforms of these lexemes share only some semantic preferences. The concordances of both woman and women do not refer to the notions of feelings or states, but while the former focus on actions (about 12.5%; actions, annual consumption, answer, becoming pregnant, behaviour, knitting, laugh, leading a caravan) and especially physical appearance (about 56%; acne, age, appearance, body, dark hand, face, lap, mouth), the latter concentrate on the discourse domains of abstract concepts (about 8%; health, issues, politics, rights, 
259
roles), group membership (25%; City Club Members, Health Book Collective, Health Education Network, liberation movement, political organization, Political Caucus, teams), and especially sports events (about 37%; championship, lives, national indoors singles, polo tournament, World Cup downhill races). On a parallel with the concordances of GIRL, those of WOMAN too show that representation of negative contexts and reference to physical appearance and sexuality is more frequent in association with the wordform woman (about 34% and 23%, respectively) than with the plural women (about 10% and 4%, respectively). Examples include: the concentrated toxins in a woman's blood; women's abuse, the great waste of women's potential; skin care products improve a woman's appearance; sexual parts of a woman's body parts, possibly “vulva”; women's lingerie. 
The data about MAN show that its singular wordform disregards the notion of group membership, while it prefers those of action (about 9%; abuse, activity, advances, bluff, historic quest), abstract concepts (10%; destiny, dignity, ego), possession (about 19%; books, country, house, land, name-tag, poster), and especially physical attributes (about 28%; abdomen, breath, cool eyes, face, hair, legs, shirt, shoulder, tonsured head), only a few of which, however, suggest attractiveness. On the other hand, the concordances of the plural men do not refer to qualities or relationships, but draw attention to abstract concepts (about 8%; title, greater access, sexual problems, studies), groups (about 16%; amateur squad, teams, Bible Study group, division, group), actions (about 27%; abuse, bidding, further domination, practice, projection and externalization, violence), events, especially related to sports (about 27%; 400m hurdles, doubles, game, lives, singles). These data thus reveal that both man and men show a preference for the categories of actions and abstract concepts, although only for the latter to a comparable degree. Another similarity has to do with the frequency of occurrence of negatively marked contexts, which is about 16% for man (there was the problem of the man's face, distinctive even in death) and about 17% for men (within the complex relationship of men's sexual abuse and men's violence), and which in most cases refer to problems caused, rather than experienced by, men. On the other hand, reference to men's attractiveness or sexuality is less frequent in the case of man (about 7%; blocking the man's ability to get an erection; a means of bridling man's unbridled lust) than in that of men (about 15%; In the ballet world men's bodies are stunning, sinewy; an intro for some men's mag sex in the kitchen). 
In conclusion, in these sets of data it is still the referents of WOMAN and GIRL who are more frequently talked about in terms of sexuality, physical attractiveness, and romantic involvement; however, interesting correlations can also be found between their singular and plural wordforms and those of MAN and BOY. For instance, in the singular both GIRL and BOY are more often associated with the notions of physicality, sexuality, attractiveness, negativity, possession, and action. Similarly, both WOMAN and MAN show a preference for the notions of physicality, action, and negativity in the singular, and for group membership and events (mostly sports events) in the plural. Some conceptual categories turn out to be favored across wordforms; for example, that of possession is a “favorite” with girl, girls, boy, boys, women, and man; that of action with woman, man, men, girl, and boy; physicality with girl, woman, man, and boy; that of abstractness with man, men, and boy; and finally, that of group membership with women, men, and boys. Therefore it appears that in the concordances relative to the genitive both gender and number are relevant grammatical categories for the identification of similarities and differences in the patterns of usage of WOMAN, MAN, GIRL, and BOY. 
5.2. For-headed prepositional phrases 
The preposition for potentially encodes a number of semantic relationships between two concepts; for instance, it may indicate that something is intended to be given a specific person (e.g. Let's save some cake for Phil), that someone is in favor of a person (e.g. Who are you going to vote for?), that a characteristic is surprising when considering what a given person is like (e.g. That was fast for you), that something is an intended action (e.g. The plan is for us to leave early). For-headed prepositional phrases, therefore, may provide a convenient window to the range of semantic preferences linkable to the referent of the object of that preposition. 
The Cobuild corpus reveals that for-headed prepositional phrases embedding the lexemes GIRL or WOMAN tend to be accompanied by words that can be grouped under a few broad notions: appropriateness of conduct, adequacy of conditions, problematicity of circumstances, unusuality of achievement, (cause of) disruption of relationships, and feelings. Such phrases are often preceded by adjectives or expressions conveying judgments about what a woman could, couldn't, should or shouldn't do (advisable, indispensable, fitting role, acceptable norm, offence, unbecoming, unthinkable, risky), expressing opinions about what is acceptable or condemnable in her behavior (absurd, believable goal, lunatic) or indicating restrictions on her actions (unusual job, no place, only sphere, wasn't a job, what else is there, ain't the right kind of life, not the kind of garment for a girl to wear in a lonely search, too much). Alternatively, such phrases are preceded by evaluative descriptions of the contexts in 
260
which women act and of what they represent to them; often, these are negatively connoted (torture [reference is to age], obstacle, unusual, urgency, important, nostalgic event). Several of the circumstances mentioned in relation to such prepositional phrases are explicitly negatively marked (cause of misery [reference is to age], to have her face smacked, complicated training, lonely life, ultimate time possible, problem, too small, dissatisfaction, concern, chances … are remote, fears), which reveals that problematic situations are frequently mentioned in association with women, and more frequently than neutral or positive ones (conditions of life, pleasurable activity, splendid effect). Moreover, when positive qualities are mentioned, these are presented as exceptionally related to women (she could run well, professional, not bad, good question, convincingly). A woman is also not infrequently mentioned as the “indirect” cause of the disruption of a relationship, thus being cast in the ambiguous role of a new beneficiary of the loving care that someone else is being deprived of (left me, left his wife, leaves his male lover, left home). Finally, when playing the role of a beneficiary, the referent of GIRL and WOMAN appears to be the object of affection rather than other kinds of attention (affection, ideal mate, desire, love, hopeless love). 
Comparable prepositional phrases for MAN and BOY are more numerous and less negatively marked. For one thing, more frequent are the phrases in which the referents of those lexemes play the role of beneficiaries, depicted as profiting from situations in which they are not specifically or necessarily the object of someone's affection, but of various forms of attention (I would do this for a man; they had voted for a man who had promised much; very little a woman won't do for a man who makes her look half her age; asking for immunity for a man who slaughtered; public affirmations of support for a man who was directly responsible). Secondly, more numerous are the phrases that introduce or are part of mini-descriptions of sets of conditions in which the referents of MAN or BOY may find themselves acting; these are not meant to express value judgments, but rather to simply report on the state of their circumstances or to mention their salient characteristics: most of them are neutral or downright positively connoted (for a man of Haig's background; for a man like you; For a man running a campaign; For a man as careful as De Gaule; for a man in his position; For a man who has had little or no trouble; For a man who potentially had another life; For a man who could have taught Japanese; For a man with such power at his fingertips; for a man on his stag night; forty was a terrific age for a man). Indications of the appropriate conduct for a man are scanty (What greater shame was there for a man than to wald the streets cringing; You consider it right for a man of my years) as is the reference to negative experiences or problematic circumstances (The problems will be worse for a man if the disfigurement is on the left; an impossible position for a man of deeply skeptical inclinations; was the inevitable result for a boy whose disaffection from his family; scandal represented tradegy for a man). More frequent are instead representations of positive situations applying to men: advantageous circustances, personal achievements, solutions to possible problems (wonderfully releasing experience for a man who, as a child, felt dominated; it will make it easier for a man to find and keep a partner; ease and comfort; it was a dream job, no mean achievement for a man of 50; Not bad for a man with an average voice, didn't do badly at all - for a man hospitalized). More generally, such for-headed prepositional phrases are set in text segments presenting positive or neutral descriptions or evaluations of the circumstances of events or situations regarding men (which was normal for a man of his age; more creditable for a man to capture an animal alive; It is usual, I imagine, for a man to look for perfection; may be particularly intense for a man; a move to … is a gamble for a man who has spent his whole career; jail would be another kick for a man who is down); sometimes they mention untypical situations (this is strange for a man to be turned into a woman; it's somehow abnormal for a man with a partner; It seemed a queer thing for a man to do; Most unusual for a man to offer another man a seat; how out of the common it is, for a man to say something like that); finally, although concordances occasionally re-present some of the semantic contexts already identified for WOMAN and GIRL (cause of disruption of relationship: dumps her for a man; object of affection: would fall for a man serving a life sentence; yearning of a man; problematic conditions: it's dangerous for a man knowing and feeling; not an easy assignment for a man; appropriateness of conduct: it is high treason for a man to violate the wife of the Sovereign; it's unnatural for a man to show such inertia), these do not constitute the focus of the various patterns of discourse characterizing men and boys. 
6. Conclusion 
An examination of the most significant collocates of WOMAN, MAN, GIRL, and BOY in four components of the Cobuild on-line corpus has revealed that these lexemes both share some contexts of use (e.g. age, physical appearance, personal relationships, positive and negative states), but not always to the same degree, and also that they are relevant to complementary discourse domains (e.g. religion, physical attractiveness, danger, and women's liberation for females, and the military, non-physical attractiveness, activities, and physical force for males). A consideration of the concordances of the 
261
above lexemes in selected syntactic environments has also shown that females and males may be associated with the same types of topics and notions, but also that the frequency of their co-occurrence is often dissimilar. Thus, for example, attributes describing females are more often negative or focused on the notion of beauty, while those modifying males tend to be more evenly distributed between positive and negative qualities or to stress men's social roles and intellectual qualities. Similarly, of-headed prepositional phrases preferably describe physical attributes of females but non-physical characteristics of males. In relative clauses, it is more often females rather than males who are represented as having to face difficult situations or in relation to their feelings, while men are portrayed in their professional roles. For-headed prepositional phrases typically refer to appropriateness of conduct with regard to females, and to advantageous circumstances or personal achievements in the case of males. On the other hand, concordances of genitive constructions have shown that comparable wordforms of different lexemes may occasionally share similar patterns of co-occurrence with certain semantic fields; thus, the singular of GIRL and BOY are both frequently associated with the notions of physical attractiveness, physicality, and negativity, while both WOMAN and MAN in their plural forms are similarly associated with the notions of group membership and sports events. 
The findings indicate that in the discourse practices of English speakers the world of females is frequently associated with notions of passivity, negativity, and physicality, while that of males with notions of activity, positivity, and cognitivity. This suggests that socio-culturally salient concepts encoded in potentially symmetric terms may develop different ideological significance when they recurrently keep company with different sets of words. On the other hand, more data need to be collected to confirm or disprove these conclusions especially considering that different wordforms of the same lexeme may not occur in the same contexts of use (see Stubbs 2001: 27-28), and that collocations differ considerably in different text types and thus in different corpora components (see Nakamura and Sinclair 1995: 108-109). To this end, further pairs of gender-marked terms need to be examined (e.g. sister, brother, wife, husband, female, male) in more varied syntactic contexts (e.g. in-headed prepositional phrases, complements, passive constructions) and in additional corpora components (e.g. relevant to the spoken medium and to specific genres). 
References 
Fairclough N 1992 Discourse and social change. Cambridge, UK, Polity. 
Halliday M.A.K. 1994 An Introduction to Functional Grammar. London, New York, Sydney, Auckland, Arnold. 
Hodge R, Kress G 1993 Language as ideology. New York, Routledge. 
Krishnamurty R 1996 Ethnic, racial and tribal: the language of racism? In Caldas-Coulthard R, Coulthard M (eds) Texts and practices: readings in critical discourse analysis. London, Routledge, pp. 129-149. 
Nakamura J, Sinclair J 1995 The World of Woman in the Bank of English: Internal Criteria for the Classification of Corpora. Literary and Linguistic Computing 10(2): 99-110. 
Partington A 1998 Patterns and meanings. Amsterdam, Benjamins. 
Stubbs M 2001 Words and phrases. Corpus studies of lexical semantics. Oxford, UK, Blackwell. 
Tognini-Bonelli E 2000 Corpus Linguistics at Work. Birmingham, UK, TWC. 
Wierzbicka A 1999 Emotions across languages and cultures: diversity and universals. Cambridge, CUP. 
262

Tracking Lexical Changes in the Reference Corpus of Slovene Texts 
Vojko Gorjanc 
University of Ljubljana, Faculty of Arts, Aškerèeva 2, 1000 Ljubljana, Slovenia 
vojko.gorjanc@guest.arnes.si 
Keywords: corpus linguistics, corpus-based lexicology, lexical semantics, lexical changes, Slovene language 
Introduction 
With the help of a corpus, we can track lexical changes quickly and reliably, and also observe the response of a selected language to new lexical items introduced into it from other languages, e.g. English, or some other language with which the selected language has direct contact, in the case of Slovene, these are Italian, German, Hungarian or Croatian. The present paper focuses on lexical items introduced into English in the last decade of the 20th century. The starting point for the comparison with the state in the corpus of the Slovene language is John Ayto's list of lexical items. We will try to determine how the Slovene language reacts to lexical items introduced into Slovene from English as the language of global society. Using the corpus, we can track the new lexical items through the last decade, and observe their characteristics in the corpus. 
The Corpus 
The Corpus of the Slovene Language, called FIDA, is a reference corpus of Slovene. It is composed of contemporary Slovene texts, the majority of which were published in the 1990's. The corpus contains just over 100 million words, encompassing a broad variety of language variants and registers. It is composed of written texts and texts originally produced as written-to-be-spoken; the transcripts of Slovene parliamentary proceedings are the only spoken component of the corpus. 
Methodology 
Using a FIDA wordlist, we will obtain information on the lexical items from Ayto's list which are relevant to the Slovene language. By means of corpus analysis, we will determine when a word occurs in the Slovene language and how it establishes itself in the language, and by means of statistical analysis, we will determine the possible collocations of the selected word and their changes from the first occurrence of the word until the end of the decade. Since pairs of synonyms or strings often occur with new expressions, we will try to determine how they occur in the corpus and how they disappear. With the help of markers of semantic relations already identified for the Slovene language by corpus analysis, we will identify pairs of synonyms and strings within the corpus. We will focus on the context of synonyms, their distribution within the corpus regarding the time of their occurrence and the genre, and on when and why one of the synonyms becomes dominant in the language while other variants disappear. For extracting collocations, the MI3 value with information on the probability of a word pair occurring together or separately will be used. The MI3 value has turned out to be sound information for content words in Slovene. On the other hand, with this value it is hard to determine function words as part of collocations; in particular, in the case of collocators of verbs and nouns with prepositional phrases. For instance, to detect propositional words, raw statistics provide more valuable information. After detection, a noun + propositional word pair, for example, the MI3 value for the whole pair is calculated to extract the string of collocators. 
Conclusion 
In its observation of the process of accepting new lexical items into Slovene, corpus analysis reveals the great creativity of Slovene language speakers; in addition to loan words, original Slovene expressions occur almost invariably. Corpus data shows a great deal of variability, linked above all to the desire for original expressions, and, after a few years, the data begin to reveal the prevailing variant within the entire corpus or a selected genre. The question of which variant is eventually fully accepted, however, remains open. Full acceptance of a variant is conditioned by a series of linguistic as well as non-linguistic factors; in order for a variant to be fully accepted in Slovene today, it must be also sexy and cool. 
263

A corpus-based approach to informality: the case of Internet chat 
Stefan Grondelaers – Dirk Speelman – Dirk Geeraerts 
RU Quantitative Lexicology and Variational Linguistics 
Dept. of Linguistics – University of Leuven 
Blijde-Inkomststraat 21 
B-3000 Leuven, Belgium 
fax: +32 16 324767 
e-mail: stefan.grondelaers@arts.kuleuven.ac.be 
The language of IRC – Internet Relay Chat – is in many respects an example of “spoken language in written form”: although produced in a written medium, it shares with spoken language a dialogical immediacy that ordinary written text usually lacks, as a result of which, it tends to appear highly informal, even to the untrained observer. 
Linguists should not, however, be misguided by this global impression of informality: the language of IRC is informal in at least four different respects. In addition to its dialogical character (a situational form of informality which manifests itself in a.o. the relatively higher frequency of 2nd person pronouns and vocatives), IRC is characterized by an abundance of abbreviation and ellipsis, which are parameters of the production demands of a written medium whose users try to imitate spoken language (e.g. Hentschel 1998). A third source of informality in IRC is speaker-related: being for the most part pre-adult males, chatters tend to indulge significantly more often in tabooed topics and territorial behavior (Hentschel 1998, De Gryse 2000). A fourth group of informality-inducing factors is register-related: no matter how colloquial the situation and/or medium, chatters may choose to sound “vernacular”, or to maintain a more formal standard, to “look after” their language. 
Our paper has a methodological and an empirical aim. We will demonstrate, first, that whereas stylometric approaches based on the calculation of isolated variables (e.g. Biber 1994) can be used to identify the first three types of informality (by counting & comparing over corpora the relative frequency of, for instance, 2nd person pronouns, vocatives, abbreviations, ellipses, taboo words and maledicta), the register-related source of informality cannot be effectively identified unless the alternate surface forms of 2nd person pronouns, vocatives, taboo words, etc. are taken into account. While current stylometric approaches, more particularly, correctly observe that 2nd person pronouns and maledicta occur much more frequently in IRC than in other written mediums, the proportion of, for instance, dialectical and vernacular variants of 2nd person references must be determined to identify register variation. 
In order to tackle the latter type of variability more adequately, an operational measure of linguistic overlap was introduced in Geeraerts, Grondelaers & Speelman (1999), which builds on the notions of onomasiological profile - the set of denotationally equivalent designations of a concept/function and their respective relative frequencies in a corpus - and uniformity, i.e. the quantified overlap between onomasiological profiles. This methodology was subsequently used to compare lexical, morphological, syntactic & phonological preferences in Belgian IRC logs and other modes of written communication (viz. UseNet, regional & national popular newspapers, and quality newspapers). What these calculations reveal is that the four types of informality do not coincide in Belgian Dutch IRC: no matter how informal chat is situation-, medium-, and speakerwise, from the register point of view, chatters do not manifest colloquial or non-standard linguistic behavior to the degree one might anticipate. This leads us to the conclusion that the linguistic specificity of chat is not, or not in the first place, determined by register choices, but by the production demands (specifically speed, and turn-taking efficiency) of spoken conversation. 
References 
Biber, D. 1994. An analytical framework for register studies. In D. Biber & E. Finegan (eds.), Sociolinguistic perspectives on register, 31-56. OUP. 
De Gryse, P. 2000. Vloeken en verwensingen op IRC. M.A. Thesis University of Leuven. 
Geeraerts, D., S. Grondelaers & D. Speelman 1999. Convergentie en divergentie in de Nederlandse woordenschat. Een onderzoek naar kleding- en voetbaltermen. Amsterdam: Meerteninstituut. 
Hentschel, E. 1998. “Communication on IRC”. Online publication on http://viadrina.euv-frankfurt-o.de/~wjournal/irc.html 
264 

Between Verbs and Nouns and Between the Base Form and the Other Forms of Verbs – A Contrastive Study into COLEC and LOCNESS 
Xiaotian Guo 
Birmingham University 
Introduction 
One of my previous contrastive studies into a learner corpus COLEC1 and a native speaker corpus LOCNESS2 has shown that these learners overuse some of the verbs. In this article, I am going to apply the theories of Halliday and Biber et al to find out some possible reasons. There are two questions I am going to answer throughout this paper: Do COLEC writers use verbs more than LOCNESS writers where there are noun equivalents? And do COLEC writers use the same amount of base form as LOCNESS writers within the verbs? In the first part, a study of twenty-five verbs and their related nouns is carried out into the tendencies of the corpus writers of the two groups in choosing nouns in their writings. In the second part, a study of fifty verbs and then four verbs in particular is made focusing on the distribution of the frequencies and the behaviour of the base form and the other forms of the verbs. 
I. Verbs or Nouns: A Study of Tendency 
According to Halliday (1985: 72-75), it is a characteristic of written English that “lexical meaning is largely carried in the nouns”. He attributed two reasons to this phenomenon: one is the structure of the nominal group and the other is the structure of clause. In analyzing the structure of the nominal group, Halliday believes that “there are a lot of things that can only be said in nominal constructions: especially in registers that have to do with the world of science and technology, where things, and the ideas behind them, are multiplying and proliferating all the time”. He compares the structure of verb groups with that of nominal groups and points out that in verbal groups there is only one lexical element: the verb itself. Even though “other lexical material may be expressed in adverbial groups”, “there are very limited in scope”. In the structure of the clause, Halliday also finds out the internal requirement for nominal groups in the grammar of modern English. Based on corpora investigations, Biber et al (1999:65) finds out that the lexical word classes vary greatly both in overall frequency and across registers. In overall frequency nouns are the most frequent word class and across registers nouns are most common in news and academic prose but least in conversation. 
As has been found in my previous study of COLEC, learner English is characterized by spoken-like language rather than an academic style which is supposed to be the right one. Supported by this finding and enlightened by Halliday and Biber et al's theories, it can be predicted that native speakers will tend to use nouns more frequently than learners where there are noun equivalents of verbs. 
Now I will choose some verbs and their noun equivalents to find out whether it is true that native 
1 COLEC is a corpus of learner English mainly composed of Chinese university students’ essays in national tests. Its size is 503,799 words. 
2 LOCNESS is a corpus of native speakers. It is composed of four components, namely, British Essays of A-Level students, British essays of university students, American argumentative essays and American literary-mixed essays. Its size is 324,161 words. 
274
speakers tend to use nouns more frequently than learners where there are noun equivalents of verbs. If this is true, it means the tendency to choose noun equivalents is stronger for native speakers than for learners. These verbs and their noun equivalents are as follows: 
accept (acceptance), apply (application), argue (argument), assume (assumption), believe (belief), choose (choice), commit (commitment), communicate (communication), compare (comparison), complete (completion), create (creation), enter (entry), examine (examination), express (expression), include (inclusion), indicate (indication), introduce (introduction), involve (involvement), manage (management), occur (occurrence), produce (production), realise (realization), realize (realization), refuse (refusal) and survive (survival). 
Table 1 shows the frequencies of these verbs and their equivalent nouns. Each verb in the table is referred as lemma including all the forms of the verb: the base form, the third singular form, the “-ing” form, the past form and the past participle. Each noun frequency includes both the singular form and the plural form. 
Table 1: The frequencies of some verbs and frequencies of their equivalent nouns 
COLEC COLEC LOCNESS LOCNESS VERB NOUN VERB NOUN ACCEPT 41 0 182 33 APPLY 65 2 60 11 ARGUE 1 3 167 339 ASSUME 3 0 40 13 BELIEVE 295 14 373 125 CHOOSE 121 31 140 129 COMMIT 8 0 90 12 COMMUNICATE 24 30 24 27 COMPARE 52 2 49 15 COMPLETE 35 0 42 5 CREATE 18 1 182 38 ENTER 84 1 56 10 EXAMINE 17 60 28 5 EXPRESS 25 9 56 12 INCLUDE 67 0 111 2 INDICATE 21 0 10 6 INTRODUCE 12 2 61 44 INVOLVE 12 0 159 9 MANAGE 29 8 27 12 OCCURE 25 0 96 5 PRODUCE 239 65 89 38 REALISE 9 0 98 16 REALIZE 196 3 122 14 
275
REFUSE 27 0 64 13 SURVIVE 34 4 47 16 Total 1460 235 2373 949 N Total3 28.98 4.66 73.2 29.28 
If we look at the total use of these fifty verbs and their equivalent nouns, there are 1460 cases of verb use and 235 cases of noun use in COLEC and there are 2373 cases of verb use and 949 cases of noun use in LOCNESS. If we normalize these figures by the total frequency of the two corpus respectively, we get the following comparison: in every 10,000 words, there are 28.98 cases of verb use and 4.66 cases of noun use in COLEC and there are 73.2 cases of verb use and 29.28 cases of noun use in LOCNESS. If we compare the noun use with the verb use across the two corpora based on normalized total frequency, we will get the following ratio: NOUN:VERB in COLEC is 0.16 and NOUN:VERB in LOCNESS is 0.4. This means LOCNESS writers use two and half times more than COLEC writers in noun use. 
If we look at the individual frequency of these verbs and their equivalent nouns, we may find that most nouns are less in number than verbs in both corpora. But there are also a few exceptions. For example, examination appears much more frequently than that in LOCNESS because COLEC writers are using a lot the sense of test for examination rather than the sense of investigation as can be found in LOCNESS: an exhaustive examination of the broadcast networks’ programming. Since tests are overwhelmingly a major concern of university students, the overuse of this sense of examination is understandable. Another point I have noticed is the frequent occurrences of production, which is the largest use of nouns within these nouns in COLEC. This is caused by the topic about the production of fake commodities in a majority of the essays in COLEC. If we may ignore these two cases, however, we will find a more striking underuse of nouns in COLEC. 
Table 2: The total frequencies and the normalized total frequencies of the selected verbs and those of their related nouns in COLEC and LOCNESS 
COLEC COLEC LOCNESS LOCNESS VERB NOUN VERB NOUN Total 1204 110 2256 906 N Total 23.9 2.18 67 27.95 
If we compare the ratios between the cases of noun use and the cases of verb use across the two corpora, we may see the difference of COLEC writers underuse of nouns is getting extremely larger: NOUN:VERB in COELC is 0.09 and NOUN:VERB in LOCNESS is 0.4. LOCNESS writers are using nouns 4.4 times more than COLEC writers. To put it in another way, the tendency to choose nouns is even milder in COLEC if we may discard the most often used ones due to the influence of topics. 
After it is certain that verbs are more often chosen by COLEC writers in language use, it would help us to understand better of learner English if we can know how verbs are used between the different forms. 
3 “N Total” stands for normalized total. It means the comparable total normalized by the total frequency of the corpus. 
276
II. The Favoured Form of Verbs: the Base Form 
As has been found in my previous studies, there is a large discrepancy between the base form and other forms of a verb in COLEC than in LOCNESS. This part will examine the detailed differences of them from the use of the different forms of 50 verbs and then four verbs in particular to see how properly the base form is used in COLEC. These verbs are chosen first and then selected and then re-chosen to make sure that they are verbs only and any individual form of any individual verb does not overlap in spelling with its noun or other lexical form. It has been a difficult process to choose the fifty verbs. For example, KNOW was chosen first but was found to be overwhelmingly used by learners. Thus, it was deleted from the list in the end not to let it deform the whole picture of frequencies and affect the soundness of judgment. Table 3 shows the distribution of these 50 verbs in their base form (V), present participle form (V-ing), third singular form (V-s), past form (V-ed(1)) and past participle form (V-ed(2)). The corpus of COLEC is represented by C and LOCNESS by L. 
Table 3: The distribution of all the forms of 50 verbs in COLEC and LOCNESS in alphabetical order 
V V V-ing V-s V-ed(1) V-ed(2) C L C L C L C L C L ACCEPT 28 92 5 21 0 14 8 55 AFFECT 32 29 0 14 7 23 13 37 AFFORD 16 38 0 0 1 0 1 2 ALLOW 5 83 1 43 0 47 5 100 APPLY 51 27 1 6 4 11 9 16 ARGUE 1 81 0 13 0 29 0 44 ASSUME 3 21 0 7 0 3 0 9 BEGIN 108 52 99 64 8 39 52 54 2 22 BELIEVE 269 226 1 13 6 76 19 58 BRING 290 80 4 18 34 35 37 81 CARRY 39 48 3 21 2 10 17 40 CEASE 1 15 0 3 0 0 1 1 CHOOSE 99 57 5 20 1 23 13 27 3 13 COMMIT 7 36 0 20 0 10 1 24 COMMUNICATE 18 18 4 4 0 1 2 1 COMPARE 11 12 11 1 1 4 29 32 COMPLETE 32 34 0 1 0 0 3 7 CONSIDER 88 60 7 26 2 9 26 81 CREATE 11 70 1 28 0 19 6 65 EAT 127 44 29 50 3 1 13 1 4 8 ENTER 60 31 8 7 1 8 15 10 EXAMINE 12 13 2 5 1 2 2 8 EXPRESS 22 24 1 7 1 6 1 19 GIVE 297 164 10 62 31 52 23 41 42 147 GO 684 212 169 148 63 92 115 35 22 37 
277
HAPPEN 69 66 22 25 13 35 76 30 HURT 60 23 0 7 3 4 INCLUDE 16 38 42 39 7 17 2 17 INDICATE 5 4 1 3 12 2 3 1 INTRODUCE 5 13 2 8 0 2 5 38 INVOLVE 5 17 1 16 0 24 6 102 LEAD 138 130 10 30 33 49 25 74 MANAGE 22 4 2 0 0 10 5 13 OCCUR 13 47 2 9 5 24 5 16 PRODUCE 145 48 46 12 12 10 36 19 REALISE 9 33 0 7 0 38 0 20 REALIZE 137 72 8 15 1 21 50 14 REFUSE 19 12 1 10 0 23 7 19 RUN 126 84 44 52 5 12 30 7 SAVE 199 40 32 29 5 8 11 10 SEEM 18 130 0 2 48 140 22 24 SEND 20 13 16 3 0 3 19 20 SPEAK 161 35 134 25 3 4 3 4 33 4 SPREAD 6 19 3 4 2 1 SURVIVE 34 31 0 6 0 3 0 7 TAKE 909 293 99 115 51 77 63 59 117 136 TELL 151 41 9 11 41 41 77 57 THROW 17 13 5 7 1 4 1 1 6 22 WANT 1154 217 5 17 94 106 99 90 WRITE 228 19 105 25 0 30 7 23 17 33 Total 5977 3009 950 1079 502 1202 963 1513 246 422 
If we add the frequencies of all the forms of the 50 verbs in COLEC (5977+950+502+963+246), the figure amounts to 8638 and if we add those in LOCNESS (3009+1079+1202+1513+422), the figure is 7225. To get an overall percentage of the use of the fifty verbs (including all the forms of each lemma) in the two corpora, we may divide the two figures above by the total frequencies of the two corpora respectively. For COLEC it is 8638/503799=0.017 and for LOCNESS it is 7225/324161=0.022. There is not a dramatic difference between the two groups of corpus writers in the total percentages of verbal uses because most of the verbs are commonly used in both of the corpora. However, if we compare the percentage of the use of the base form and that of the other forms of the verbs in the corpora, the picture will look different (see Table 4). The base form takes up 69.4 percent of the total use of the lemma in COLEC and 41.6 percent in LOCNESS. Contrary to the overuse of the base form of verbs, COLEC writers are underusing all the other forms of verbs. 
Table 4. Percentages of all the forms of the 50 verbs in the two corpora 
Form COLEC LOCNESS V 69.4 41.6 
278
V-ing 10.9 14.9 V-s 5.8 16.6 V-ed (1) 11.1 20.9 V-ed (2) 2.8 5.8 Total 100 99.8 
There may be several reasons for the overuse of the base form of verbs. One possible reason is that COLEC writers are misusing the base form for other forms that native speakers use. This assumption is based upon one of the features of the native language of the learner corpus writers, namely, Chinese is a non-inflected language. COLEC writers are not properly aware of the requirement of the English language in the change of word forms in different contextual situations. In the following part, this assumption will be tested. To do this, four verbs are chosen to see how properly the base forms are used in COLEC: APPLY, HAPPEN, CHOOSE, and BEGIN. 
An investigation into all the concordance lines of the base form of these verbs shows several types of misuses of the base form. These misuses can be mainly classified into the following categories: 
1. the misuse of the base form as the third singular form. Some examples are: 
"Practice Makes Perfect " also apply to other subjects, 
The phenomenon often happen in people's everyday life, 
That often happen to everyone student. 
Whatever happen, wherever we are, we must keep our mind clear. 
So, when he choose a job he must … 
2. the misuse of the base form as the “-ing” form. Some examples are: 
Because failure is the thing which is always happen 
everything has a process from happen to developing 
We can know what happen in our city from what we talk about. 
“OH, my dear, what's happen? Asked the father. 
A good begin is half done. 
3. the misuse of the base form as the past participle form. Some examples are: 
we shall know what has happen outside 
We can learn what have happen in society. 
Since we have choose the professinl field, 
Once I have choose a kind of job. 
Well begin is half done. 
4. the misuse of the verb as a noun. Some examples are: 
I have written the apply form to join the Party. 
Therefore, choose a good job is very important. 
279
Now we have no choose, we must … 
…from begin to end … 
There are 6 misuses in APPLY, 16 in HAPPEN, 6 in CHOOSE and 12 in BEGIN. All these misuses take up 11.7 percent of all the lemma of the four verbs. Due to the misuse of the base form for other forms of verbs, the number of the base form use has been improperly presented. Till now, the assumption has been proved to be valid after the investigation into the use of the base form of HAPPEN, CHOOSE and BEGIN, which means COLEC writers do have problems in choosing the proper form of the verb when it is needed. By over-applying the wrong form of verbs, the frequency of the base form has accordingly increased. But the over-applications of the wrong form of verbs vary from verb to verb. 
III. Some miscellaneous findings 
Apart from the misuse of the verbal forms, there are quite a few cases of misuse of the verb where another verb or even word class should be used. Examples are: 
The factory can produce all kinds of producs which apply good servece for the people. (provide) 
the hospital apply the people with morden servece. (provide) 
factorys require the adequent water to apply the contemporary production (meet the need of ) 
The numourous wells can also apply lots of fresh water (supply) 
We should create many conditions to cause the fire to apply to its advantage (exert) 
more and more people want to live comfortable and happen (happy) 
In most of the examples above, the verb APPLY is used to mean SUPPLY. The reason for this kind of misuse is simply due to the similarity in the spelling. Even though the misuse of a verb as another verb does not make any difference for the analysis of the reason of the overuse of the base form of verbs, it is a kind of by-product of this study. It shows a sort of progress of language acquisition and the necessity to distinguish one word from another when they share some similarity in form. 
Another striking point of the examination is that a couple of words have been found to be underused dramatically by COLEC writers. For example, the lemma ARGUE has appeared 167 times in LOCNESS whereas it has appeared only once in COLEC. Another example is the lemma INVOLVE which has been used 159 times by LOCNESS writers is used only 12 times by COLEC writers. These two verbs rank 14 and 15 respectively in LOCNESS which are near the top of the list but 50 and 44 which are at the bottom of the list in COLEC. Furthermore, if we refer to Table 1 to see whether COLEC writers are using the related nouns of the two verbs, they still remain underused. Therefore, I would speculate that these two verbs are not adequately acquired by this group of learners compared with other verbs in the list. 
Another interesting finding is that COLEC writers tend to choose the British spelling for the word REALIZE instead of the American spelling REALISE (REALIZE 196 in total and REALISE 9) in total. This must be the influence of the text books used by COLEC writers and their teachers. 
Conclusion 
280
This chapter has revealed learners tendency to choose verbs where native speakers could prefer to choose nouns for the writing purpose. The ignorance is explicit of the difference of language styles is obvious in the choice of nouns and verbs by COLEC writers. Another important finding of learner English style is that COLEC writers overuse the base form of verbs as a whole but underuse the V-ing form, the third singular form, the past form and the past participle form of verbs. Among the overuses of the base form, a large frequency of misuses of the base form has been found where another form should have been used. COELC writers sometimes choose a wrong word to mean something that should be expressed by another word even though this is not a dominant feature of the misuse. Attention should be to these features of learner English in the field of English learning and teaching. 
Reference 
1. Biber. D., Johansson S., Leech G., Conrad S. and Finegan E. 1999 Longman Grammar of Spoken and Written English. London, Longman. 
2. Halliday, M.A.K. 1985 Spoken and Written Language. Oxford, Oxford University Press. 
281

A method for word segmentation in Vietnamese 
Le An Ha 
Research Group in Computational Linguistics 
School of Humanities, Languages and Social Sciences 
University of Wolverhampton 
Stafford Street, Wolverhampton, 
WV1 1 SB, UK 
L.A.Ha@wlv.ac.uk 
Abstract 
Word segmentation is the very first step in natural language processing for languages such as Vietnamese. Given the fact that un-annotated corpora are the only widely available resources, we propose a method of word segmentation for Vietnamese, which only uses n-gram information. We calculate the probabilities of different combinations of n-grams in a chunk, and choose the one that produces maximum probability. In order to calculate these probabilities, we build a 10M-word corpus of two year newspaper article. The results, while not very impressive, show that the method works. 
1. Introduction 
It has long been known that word segmentation is a vital problem in natural language processing for certain Asian languages such as Chinese, Japanese, and Vietnamese. Without knowing the boundaries of words in a sentence, we cannot move anywhere further. The problem is that in these languages, each sound unit, when written, is separated by a space, and there is nothing to be used to identify word boundaries. Of courses, native speakers do not have any problem using their language, they can still communicate efficiently and precisely in both spoken and written form without explicitly identifying word boundaries. But a computer is more confused by word boundary in these languages. But “what is a word?” itself is also a problem. This question cannot be answered in a way to satisfy everybody, so they continue to have their own definitions. The evidence is that there were efforts in the past, which tried to group sound units into “words” in written form, such as “ky-thuat”, or “kythuat” instead of “ky thuat” (technology), and failed, because not everybody were happy with this kind of notation, and they are also still confusing. Thus the problem of word boundary continues to exist. A lot of methods have been introduced to solve the segmentation problem in Japanese and Chinese, but none for Vietnamese. Another problem is that these methods tend to use lexical resources, which is still underdeveloped for Vietnamese. These issues lead to an attempt to develop a method of word segmentation in Vietnamese, using only pure statistics. In order to do that, we build a corpus of 10M tokens, and use it to calculate different scores needed for the method. In this paper, different issues arising when we carry out the experiment will be also discussed. 
2. About Vietnamese language and the problem of word boundary 
Vietnamese, although still debatable, is a monosyllabic language, and belongs to the southeast Asian language family. It has different phonetical, grammatical and semantic features from Indo-European languages, which make it difficult for the Vietnamese, not only to learn European languages, but also to develop techniques for natural language processing. The options are either try to make the Vietnamese language fit into the framework of other well-investigated European languages, or to develop from scratch a new framework. Both are proved to be very painful, where the former is not very successful, given the fact that Vietnamese is significantly different from Indo-European languages, and the latter needs a lot of resources, both human and materials, which may not be available for a developing country such as Vietnam. 
Vietnamese, traditionally, is a spoken language, thus to the native speaker, the problem of identifying word boundary is not a very serious problem. They know what it is, and use it naturally, even if there is some difference of opinion between them on the word boundary issue, it does not make the communication process more difficult, because they all agree on what is a sound, the more basic unit of Vietnamese. When written forms of the language were developed either using Chinese characters or Latin characters, these written forms are only the extensions of the spoken form, where each sound was represented by a (sequence of) character(s), and then separated by a space. This also causes no difficulty for native speakers to understand each other and the problem of word boundary did not arise 
282
yet. At the beginning and middle of the 20th century, when Vietnamese scholars were introduced to Western grammar schools, some changes in written form of Vietnamese had been proposed, to make it more “word oriented”, using different marks for the word boundary to be more explicit, and the language look more like European ones. These changes included the elimination of space between syllables that may form “word”, and using the hyphen marks. These attempts were unsuccessful, maybe because of the nature of Vietnamese language, whereas the identification of words may not be that important at all. The discussion about what is a word in Vietnamese goes on, and, up to now, there is still no general agreement on the issue. 
The development of computers, in general, and computational linguistics, in particular, does not permit researchers to avoid the problem of word boundary anymore. Unlike native speakers, computers (at least until now), cannot easily identify word boundaries in electronic texts. And it is the bottleneck of natural language processing for Vietnamese, because without knowing word boundaries, the computer cannot do anything else (part-of-speech tagging, parsing etc.), all it can do is n-grams counting. 
In fact, a way to process texts without knowing about word boundary may exist after all. But given the reality that a lot of developed methods of automatic natural language processing are for word-oriented languages, the time of developing word segmentation techniques, and after that, applying other available methods of natural language processing, maybe a lot less than the time spent on developing whole new theories and techniques for Vietnamese language processing. 
Why identifying word boundary in Vietnamese is so difficult? 
As discussed in the previous section, Vietnamese, at first, is a spoken language. In a spoken language, the most important unit, is the syllable, not the word. The word boundary can vary from person to person, and still does not affect the communication process. Another reason is that the combination of different syllable units is the only way to construct new lexical units to describe new concepts in Vietnamese. There is no prefix, suffix in Vietnamese, just syllable, and it makes everything look confusing. The fact that the part-of-speech system of Vietnamese, like that of Chinese or Japanese, is not very well-defined leads to differences among dictionaries, thus contributes its part in the difficulty of word boundary identification. Just another problem of word boundary is that a large part of vocabulary of Vietnamese come from (ancient) Chinese, and these units seem to bound together more strongly than pure Vietnamese syllable units, for example, “cong nhan” (worker), “thuong nhan” (businessman). (“nhan” roughly means “person” in Chinese). Shall we treat these units differently or the same? Note that their word order are different from natural Vietnamese word order (modifier head in comparison to head modifier as in pure Vietnamese (nguoi lao dong, nguoi buon ban). 
Yet there is another type of lexical units in Vietnamese which is problematic. It is reduplication, usually two syllables, in which only one, or even none, has some meanings, the other one is just a variant of the sound of the other. This type of units is very popular, especially for adjectives, or in fact, almost every adjectives have their reduplication forms. It can be explained, for as a spoken language, the sound is very important. The question is that how we will treat this kind of unit in word boundary identification. 
3. The construction of the corpus and some of its properties. 
With so many problems, the only solution seems to be the use of corpora to discover the nature of word boundary in Vietnamese. Until recently, due to different reasons, it is very difficult to construct a Vietnamese corpus. It is either because the lack of human or materials resources, where there are other more important things to do in a developing country. But the rapid development of the Internet and World-wide web in Vietnam makes it possible to build an inexpensive corpus. News, stories, books, etc. become more and more available on the web. Finding resources to build a corpus is not a difficult task any more. Of course, if we consider the task of building a well-balanced corpus, maybe the web is still not a very good resource, but if our task is just to evaluate some techniques in their early days, the web is the right place. And for Vietnamese, methods of natural language processing are in their early days, and researchers cannot wait until a standard corpus is built, to try their techniques. For these reasons, we choose to harvest news from online sources to construct our corpus. In particular, two years news articles from http://vnexpress.net are collected. These articles came from different newspaper sources, thus somehow balanced in styles and genres. These articles are classified into different topics: society, world, business, health, science, and way-of-life. Different perl scripts are 
283
used to get these articles, and put them together in a corpus. The size of the final corpus is difficult to define, depending on what we will use to measure the size of the corpus. The Vietnamese characters are encoded using the unicode UTF-8 standard, and stored in a html-friendly format (in order to display the content easily on different applications). If we measure the size of the corpus, using every thing between two spaces (and other punctuation marks) as a token, (except punctuation marks) the number of tokens in the corpus is about 10M. Of courses, one can argue that this is a rather small size corpus, (where different segmentation techniques in Chinese use 40M to 60M corpus), but we think that this is enough for our experiment. The information harvested from the web site allows us to mark the author (sources) of the articles, a short summary, and the date of the articles. These will not be used in our project, but maybe useful for future development, thus they are recorded in the corpus using sgml tags. Paragraphs are also marked according to the html tags. 
Statistical figures of the corpus. 
There are quite a few problems counting n-gram in Vietnamese. As discussed above, natural language processing in Vietnamese is still in its early days, and the definition for an n-gram is not very well discussed. For example, personal names in Vietnamese, like in Chinese, have some meanings, and unlike Chinese, they are in upper case, thus distinguishable from other syllable. This leads to a problem: shall we convert everything into lower-case before counting n-gram? Of course, there are certain possible solutions for the problem, such as a name-entity recognition module, but we do not have it. In the end, we decide that we will count n-grams in both cases: original, and lower case converted ones. Observing the different between two approaches will give us certain ideas of the effects of lower and upper case in n-gram frequency in Vietnamese. 
Like other languages, sentence limit is also a problem in processing Vietnamese, and a simple approach has been used, whereas everything between two periods is a sentence. Hopefully this will not make a lot of errors. 
Observing the unigram list gives us the idea of how different Vietnamese is from English. Some of the highest frequency words are given in table 1. Comparing the unigram list when upper case is taken into account and not taken into account shows that it is better to ignore case in this particular application. 
When looking at the number of unique unigrams, we find out that it is 60637, which is a very high number, and almost half of them only appear once in the corpus. The explanation is that the corpus contains a lot of foreign words, which will be another issue in NLP applications in Vietnamese. 
va (and) 88000 các 84380 c.a (of) 83915 có 81828 m.t (one) 65332 la (to be) 59392 trong (in) 58964 cho 56378 ngu.i 55388 ðu.c 54722 ða 53700 không (negative) 48899 theo 44716nay 43040nh.ng 41430v.i 38816. 38767công 37175s. (will) 34160ra 31654b. 31093khi 29587th. 29308ông 28955 ð. 28936tren (on) 27762 
nu.c (water, nation) 26209vao (into) 25539t. (from) 25109hi.n 24467nha 24439ð.n 24050v 24015ðó 23843t.i 23172cng 22980 s. 22936 nãm (year) 22902 nhân 22726 ð.ng 22471 vi.c 22446 thanh 20947 ph.i 20721 nhu 20711 nhiu 20173 chính 19952 
Table 1: unigram frequency, and their meaning in English (where available). 
4. Methods used for word segmentations in different languages. 
Word segmentation is a common problem for Asian languages, such as Chinese, Japanese, Korean, etc., although the root of the problem maybe different. Numerous methods have been introduced to solve the problem, which can also be divided into two sub problems: 1)disambiguating word boundary, whereas a lexicon database is available, and 2) identifying word boundary for new lexical items. In 
284
order to solve the first problem, dictionaries and statistical methods have been used. Usually, the longest possible string will be matched, using a dictionary, and statistical scores will be calculated to disambiguate when ambiguities arise. These approaches have some problems. Firstly, when based on a dictionary, it implies that the dictionary is a reliable source for natural language processing, in the sense that it is consistent and complete. But in reality, such a dictionary is very hard to find, and different dictionaries seem to be inconsistent, especially with languages which have not been extensively investigated. Furthermore, these days, vocabulary of a developing country language is also developing, and dictionary cannot be up-to-date any more. In particular, the problem with Vietnamese dictionaries is that they are often built on different standards, and have different sets of entries. The grammar is also partly to be blamed, where there is still no concrete grammar theory for Vietnamese, and every attempt to apply Indo-European grammars for Vietnamese have failed, making the dictionary compiler confused about which system of category (s)he should classify her/his words into. Nouns are generally agreed between linguists, but verbs, adjectives and adverbs are arguable. In the past, these problems were tolerant, because, as discussed previously, it does not affect the quality of communication much, but we cannot avoid them any more, if we want to use computer to process natural language in Vietnamese. The computer is not human, and without explicitly showing it the exact word boundary, it will refuse to do anything else. 
The main approach for identifying word boundary of new lexical units is the statistical one, where different statistical scores have been calculated, to determine the level of “bound” between different smaller units (sound). The assumption lies behind this approach is that certain units appearing together in the text frequently form a bigger, steadier unit that we call “word”. The problem of this approach is to find a good score that really reflects the phenomenon of “word” in the subject language. This problem seems to be a difficult one, where a solid statistical measure cannot guarantee the success of the segmentation process. Mutual information and t-score are examples. Although these scores have solid statistical theories behind them, the results using them are not that impressive. In the t-score case, it maybe because the assumption of normal distribution has failed, or natural occurring texts are not very suitable for information theory, in the case of mutual information. There are still a lot of works to do in the field to establish the relation between the phenomenon of “word” in monosyllabic languages, and statistics, or even disprove the phenomenon. But it does not mean that we should not do anything. Different methods should be tried to solve smaller problems, thus may give hints to the development of a more robust theory of “word”. 
Following is a short review of methods used for word segmentation. 
Maosong el. al. use mutual information and t-scores to identify word boundary in Chinese, and the reported results, although challenged by other authors, are very high (>90%). The problem with this method is the use of t-score implies the normal distribution. Wong and Chan employ a lexicon of 80.000 entries and a corpus of 63M character for word segmentation based on maximum matching and word binding force. This algorithm relies heavily on the lexicon. Sproat et. al. introduce a stochastic finite-state word-segmentation algorithm, which also relies on lexicon resources. Sornlertlamvanich el. al. use C4.5 learning algorithm for word segmentation in Thai. Information used for the algorithm includes string length, mutual information, frequency, entropy. 
5. The proposed method 
Given the situation that, for Vietnamese, there is a lack of large lexicographic resources, and annotated corpora are also rare, we want to develop a method that will not rely on these resources, and will use only raw corpus. From a raw corpus, frequency is the only statistical score that can be calculated reliably. Our approach begins with other units that native speaker will agree on, which are sentences, and/or chunks separated by punctuation marks (commas, hyphens, quotes, etc.). These units are less ambiguous than words, in both spoken and written forms. We then try to maximise the probability of the chunk, using different segmentations. The final segmentation is the one that gives the chunk the maximum probability. Of course, we are not talking about real probability of an n-gram, or a chunk. The used “probability” of an n-gram is its maximum likelihood estimation, and the probability of a chunk is the product of its n-gram “probabilities”. The implicit idea behind this calculation is that given a chunk, it is most likely that it is combined by the segmentation that gives the maximum probability. But in this method of calculation, we face certain problems. The first one is combinatorial explosion, where the number of ways of segmentation is an exponential function of the length of the chunk. Other problems include the difficulty of calculating probability, and sparse data. In order to solve the problem 
285
of combinatorial explosion, dynamic programming is employed, whereas the maximum probability of a smaller chunk is only calculated once and then reused in other calculations. By using dynamic programming, the complexity of the problem is reduced to O(n3). The detailed calculation is given below. 
P(i,j): maximised probability of a segment begin at i, end at i+j. 
p(i,j): probability of the n-gram begin at i, and end at i+j. 
The calculation of maximum probability: 
P(i,0)=p(i,0); 
P(i,j)=max {P(i,0)* P(i+1,j-1), P(j+i,0)*P(i,j-1), P(i,1)*P(i+2,j-2), P(j+i-1,1)*P(i,j-2),..., p(i,j)} 
The actual calculation using dynamic programming method is following: 
for(i=0;i<chunk_length;i++){ 
P(i,0)=p(i,0); 
BackTrack(i,0)=”this n-gram”; 
}; 
for(i=1;i<chunk_length;i++){ 
for(j=0;j<chunk_length-i;j++){ 
max=p(i,j); 
backtrack=”this n-gram”; 
k=0; 
while(1){ 
if((i-1-k)<k){ 
last; 
}; 
if(max< P(k,j)*P(i-1-k,j+1+k)){ 
max= P(k,j)*P(i-1-k,j+1+k); 
backtrack=”(k,j):(i-1-k,j+1+k)”; 
}; 
if((i-1-k)<=k){ 
last; 
}; 
if(max< P(k,j+i-k)*P(i-1-k,j)){ 
max= P(k,j+i-k)*P(i-1-k,j); 
backtrack=”(k,j+i-k):(i-1-k,j)” 
}; 
} 
k++; 
}; 
P(i,j)=max; 
BackTrack(i,j)=backtrack; 
}; 
At each step, the combination that maximises the probability will be recorded for backtracking. In the end, these backtracking information will be used to reconstruct the combination of n-grams that give the chunk the maximum probability. In order to avoid problem of multiplying probability, we use logarithm and convert multiply operations into add operations. 
In the experiment, we will stop at the tri-gram level. The reason is that, our corpus is still a small size one, and if we go beyond this tri-gram level, the probability will become biased toward the whole n-gram rather than the combination of smaller n-grams, making the results less reliable. As we can see, the complexity of the algorithm is only O(n3) (where n is the length of the chunk). 
Chunks separated by punctuation marks will be extracted from the corpus and the above calculation will be performed. The results will be given to a native speaker, who will judge whether or not the segmentations make sense. 
286
6. Evaluation: results and some issues 
As discussed in various papers, evaluation for word segmentation is very problematic. We think that the root of the problem is that there is no concrete definition for “word” in languages such as Vietnamese. This lead to the phenomenon that even native speakers can not agree with each other about word boundary (Sproat and Chang). And it makes all evaluations of word segmentation, either based on native speakers or on available lexicon, only have relative meanings. Word segmentation is only the first step of natural language processing, further steps should be made before we can look back and say whether or not a word segmentation method is reliable. 
In our evaluation scheme, we ask a native speaker to look at the results from the method, and say 1) whether or not he will identify word boundaries like the system, 2) whether or not he thinks the segmentation produced by the system is reasonable. The two questions asked are to make sure that we have to accept the differences in word segmentation, as far as it is reasonable. Of course, one can argue that “reasonable” is rather vague, but so is “word”. An ideal evaluation for word segmentation should be an extrinsic one, where word segmentation is incorporated in an NLP application, but when such an evaluation still cannot be carried out, the above evaluation scheme still gives us a fair idea how the method perform. 
We extract randomly 100 chunks from the corpus. From these chunks, the system identify 614 “words”, of which, the evaluator agrees on 315 (51%), and thinks that 402 (65%) are reasonable. The percentage seems to be low, but when taking into account that the method neither uses lexicon nor annotated corpus, we think that the result is not very disappointing. The reason why we do not extend our evaluation is that the we feel with this method alone, we cannot get a higher number of successful. Hopefully, when combining with other methods of word segmentation, the results will become and more acceptable. 
Conclusion and future work 
The introduced method has yield a promising results, when we take into account its advantage, which is only using un-annotated corpus for word segmentation. But we also raise a few problems of word segmentation, not only for Vietnamese, but also for other languages. A lot of discussions are still needed, including 1) definition of what is a “word”; 2) definition concrete evaluation schemes for word segmentation methods, including extrinsic evaluation. And for Vietnamese, a lot of works also have to be done to improve the performance of word segmentation, including the use of other available resources and/or the development of resources, and the combination of different methods for word segmentation. 
Reference 
Cao, X H. 1985. Ve cuong vi ngon ngu hoc cua Tieng. Ngon Ngu 1. 25-53. 
Cao, X H. 1990. Some Preliminaries to the Syntactic Analysis of the Vietnamese Sentence. In Proceeding of the Prague Congress of 1990. 
Constantine P. Papageorgiou. 1994. Japanese word segmentation by hidden markov model. In Proceedings of the Human Language Technology Workshop. 283-288. 
Doan, T. T. Dong gop vao viec gioi dinh tu da tiet bang tieu chi trong am trong tieng Viet. 1965. Thong bao khoa hoc 2. 124-125. 
Hoang, V. H. Ve hien tuong lay trong tieng Viet. Ngon ngu 2. 5-15. 
Li Hai-Zhou and Yuan Bao-Sheng 1998. Chinese word segmentation. In Proceedings of the 12th Pacific Asia Conference on Language, Information and Computation, PACLIC-12, 1998. 212-217. 
Maosong Sun, Dayang Shen, and Benjamin K. Tsou. 1998. Chinese word segmentation without using lexicon and hand-crafted training data. In Proceeding. of COLING-ACL 98. 1265-1271. 
Meknavin, S., Charoenpornsawat,P. and Kijsirikul, B. 1997. Feature-based Thai Word Segmentation. In Proceedings of the Natural Language Processing Pacific Rim Symposium (NLPRS'97). 41-46. 
Palmer, David. 1997. A Trainable Rule-based Algorithm for Word Segmentation. In Proceedings of ACL. Madrid. 321-328. 
Sornlertlamvanich, V., T. Potipiti and T. Charoenporn. 2000. Automatic Corpus-Based Thai Word Extraction with the C4.5 Learning Algorithm. In Proceedings of COLING 2000. 
Sproat, R., Shih, C., Gale, W. and Chang, N. 1996. A stochastic finite-state word-segmentation algorithm for Chinese. Computational Linguistics, 22 (3), 377-404. 
287

Linguistic enrichment and exploitation of the 
Translational English Corpus 
Silvia Hansen-Schirra 
Institute for Computational Linguistics 
Saarland University, Germany 
hansen@coli.uni-sb.de 
1 Introduction 
The aim of the present paper is the investigation of the nature of translated text. This kind of analysis is based on the assumption that translations differ from their source language texts and from comparable texts in the target language, in the sense that they have specific properties which cannot be found in non-translated text. The present research focuses on the comparison of English translations from the register of narrative writing to English originals from the same register. Within this context, the analysis of the relevant register features shows whether or not the translations conform to the norms of the given register, i.e. whether or not normalization can be found in the translation corpus (see section 2). The corpus under investigation is composed of the fiction part of the Translational English Corpus and the fiction part of the British National Corpus (see section 3). This monolingually comparable corpus comprises about 10 million words in total. To cope with the this large amount of words included in the comparable corpus, computational methods which support corpus enrichment and exploitation are employed. Along these lines, the automatic annotation of the corpus and its representational format are presented (see section 4). Furthermore, the querying techniques as well as the significance tests are discussed (see section 5). Hereby, the comparability of the two sub-corpora (i.e. translated vs. original text) plays an important role for the methodology chosen to investigate the specific properties of translations. The analysis results in a profile of the nature of translated text, which clearly shows to what extent the translations differ from originally produced texts. Furthermore, possible explanations are discussed on the basis of examples taken from the translation corpus. The paper concludes with a summary and an outlook on related research perspectives (see section 6). 
2 The nature of translated text 
In translation studies, the role of corpora was traditionally restricted to their use to the applied branch of this discipline. In particular, it has been used in the fields of terminology, translation aids (e.g., to develop translation memories or machine translation programs), translation criticism and translation training (to improve the final product with the help of corpus-based contrastive analysis and the study of translationese). Corpus linguistics has only very rarely been considered in terms of its importance to the theoretical and descriptive branches of translation studies. Researchers even tried to ban translations from corpora because translated text was regarded as inferior compared to originals and it was not considered worth investigating because it is generally constrained by the presence of a fully articulated text in another language. Sager (1984) was one of the first researchers who saw the need to examine translations as a special kind of text production and to look into their special characteristics. Nevertheless, he thought that the value of a translation is dependent on that of its original text (cf. Sager 1994). In contrast to Sager, Baker (1995) goes a step further by trying to exclude the influence of the source language on a translation in order to analyze characteristic patterns of translations independent of the source language. Within this context, Baker (1996) developed the following hypotheses on the universal features of English translations: 
• Explicitation. Explicitation means that translators tend to render explicit implicit contents of the source language text in their translations. As a result, translated texts tend to contain less ambiguities than originally produced texts. Evidence of explicitation may, for example, be found in the text length (number of words of the individual texts), since, in many cases, translations are longer than texts produced originally in the target language or in the source language. This kind of analysis requires a comparison of source language texts and their translations on a text-by-text basis. Explicitation can also be analyzed in view of lexis and syntax, using a monolingual corpus of translated texts and a comparable corpus of original texts produced in the same language. Translations tend to use more explanatory vocabulary (e.g., ”therefore“, ”consequently“) and optional subordinators (e.g., ”that“) than in originally produced texts, thereby rendering implicit contents more explicit. 
288
• Simplification. Simplification describes the tendency of translators to (consciously or unconsciously) simplify texts in order to improve the readability of their translations. Evidence of this tendency may, for instance, be found in the average sentence length: the mean sentence length of translated texts tends to be lower, as translators often break up long and complex sentences into two or more sentences in their translations in an effort to make the texts easier to read. Some linguistic features indicating simplification (e.g., the use of finite structures in English translations as opposed to non-finite structures in English originals) may also, at the same time, be a sign of explicitation. Another linguistic feature which reflects simplification is punctuation. Translators often change the punctuation from a weaker to a stronger mark, often using semicolons or periods instead of commas and periods instead of semicolons. This can be seen as an attempt to make texts easier and structure them more clearly by strengthening the punctuation. Another piece of evidence of simplification consists in the lexical density of a corpus, lexical density being the ratio of lexical vs. grammatical words. It is calculated by first subtracting the number of function words from the total number of words. The number of lexical words thus obtained is divided by the total number of words and then multiplied by 100. In translations, the lexical density tends to be lower than the lexical density of originals. This means that translations contain more function words and fewer lexical words than originals and are thus easier to read. A further method to test simplification is the type-token ratio, that is the ratio of different tokens vs. running words. This percentage is determined by dividing the number of lemmata (types) by the total number of words (tokens) and then multiplying the result by 100. Translators tend to use fewer types in translations than authors do in originals, and thus the type-token ratio of translations is lower than the type-token ratio of originals. 
• Normalization. Normalization (or conservatism) means that translators tend to conform to the typical patterns of the target language or even to exaggerate their use. If, however, the status of the source language has an influence on the language use of the target language (like the influence of the English language on other languages in the area of software), normalization in translations is weakened, or even counteracted by a contrary tendency. If this is the case, the typical patterns of the source language are still visible in the translations. This universal feature also includes the tendency to normalize marked and ungrammatical structures. This often occurs in simultaneous or consecutive interpreting, where interpreters tend to finish unfinished sentences and to grammaticalize ungrammatical structures. 
• Levelling out. In a corpus which consists of a sub-corpus of translations and a sub-corpus of texts originally produced in the target language, translations are more alike in terms of features such as lexical density, type-token ratio and average sentence length than the individual texts in the comparable corpus of source language and target language originals. This means that translators tend to use centered linguistic features, moving translations away from extremes. 
Features like these translation universals constitute the specific properties of translations and thus nature of translated text. 
3 Analysis scenario and corpus design 
This section deals with the investigation of the relation between English translations and English originals, i.e. the investigation of translated text as special kind of text type or register. This analysis is based on Baker's normalization hypothesis (cf. Baker 1996), as the notion of register is used to substantiate the definition of what is ”normal“ (cf. Teich 2001). With this in view, Biber's lexico-grammatical register features (cf. Biber 1995) are taken into account. Since the corpus consists of English fiction texts, the following functional dimensions are of relevance: Dimension 2 (narrative vs. non-narrative discourse), Dimension 3 (situation-dependent vs. elaborated reference), Dimension 5 (abstract vs. non-abstract style) and Dimension 6 (on-line informational elaboration marking stance). Taking together all the sub-registers of fiction (general fiction, mystery fiction, science fiction, adventure fiction and romantic fiction), fiction can be characterized as narrative (i.e. the positive features of Dimension 2 are typical, while the negative ones are untypical), situation-dependent (i.e. the positive features of Dimension 3 are typical, whereas the negative ones are untypical), non-abstract (i.e. the positive features of Dimension 5 are typical, whereas the negative ones are untypical) and edited (i.e. the positive features of Dimension 6 are untypical, whereas the negative ones are typical). The combination of the functional dimensions relevant to fiction results in the following list of typical and untypical fiction features: 
289
• typical features: past tense verbs, third person pronoun, perfect aspect, public verbs, synthetic negation, present participle clauses, time adverbials, place adverbials, adverbs, phrasal coordination; 
• untypical features: present tense verbs, attributive adjectives, wh-relative clauses, pied piping, phrasal coordination, nominalizations, conjuncts, agentless passives, by-passives, past participle clauses, subordinators, that-clauses, demonstratives, final prepositions, existential there. 
According to this list of lexico-grammatical features, a similar use (or overuse) of typical features as well as similar use (or underuse) of untypical features in English translated fiction texts (compared to English originals) would support Baker's normalization hypothesis since the target language texts would conform to (or even exaggerate) the norm of the target language. However, the notion of normalization has to be extended for this kind of analysis since the contrary tendency is investigated as well. The overuse of untypical features as well as the underuse of typical features in English translated fiction texts would therefore be seen as an indicator of anti-normalization, the contrary tendency of normalization, since the translated texts would not conform to the norms of the target language. 
Since Baker's normalization hypothesis serves as basis for the investigation in this section, her criteria concerning corpus design are also taken into account (cf. Baker 1996). Thus, the corpus under investigation in this section consists of an English comparable corpus comprising of a sub-corpus of English translations and a sub-corpus of English originals. The sub-corpus of English translations is taken from the fiction part of the Translational English Corpus (TEC), whereas the sub-corpus of English originals is extracted from the fiction part of the British National Corpus (BNC). Thus, both sub-corpora belong to the register of fiction, as mentioned above. The translational sub-corpus is made up of translations from several languages into English (translated by professional translators who are English native speakers). The translational sub-corpus consists of 4,843,763 words and the original sub-corpus includes 4,741,500 words (9,585,263 words in total). From this it can be seen that the corpora are compiled in such a way as to make them as comparable as possible, both in terms of register and in terms of size. 
4 Corpus enrichment 
Since the corpus under investigation in this section is quite large (approximately 10 million words) and the features to be analyzed are rather shallow, linguistic annotation can be carried out automatically. Part-of-speech tagging is a fairly reliable method of annotation, either using a rule-based or a statistical approach. Recently, however, statistical approaches have become more popular. For this reason, the tagger which has been employed, the TnT tagger, is a statistical part-of-speech tagger that analyzes trigrams, incorporating several methods of smoothing and of handling unknown words (cf. Brants 1999). The system can be trained to deal with different languages and comes with the Susanne tagset for English (cf. Sampson 1995) and the Stuttgart-Tübingen tagset (STTS) for German (cf. Schiller et al. 1999). It includes a tool for tokenization, which is a preparatory step in the tagging process. In basic mode, not only does the tagger provide each token with a part-of-speech tag, but it omits alternative tags and also performs probability calculations. It analyzes between 30,000 and 60,000 tokens per second and has an accuracy of about 97 %. 
A TEC sample output of TnT in tab separated vector (TSV) format is shown in figure 1. The part-of-speech tags used for tagging TEC and BNC are based on the Susanne tagset (cf. Sampson 1995). 
Figure 1: TnT sample output of TEC 
290
There are two TEI-conformant headers for the files in TEC: a header for single volumes and a header for collected works. The header for single volumes can be found in the following figure: 
<Header> 
<title> 
<filename></filename> 
<subcorpus></subcorpus> 
<collection></collection> 
<editor></editor> 
</title> 
<translator> 
<name></name> 
<gender></gender> 
<sexualOrientation></sexualOrientation> 
<Nationality></Nationality> 
<employment></employment> 
<status></status> 
</translator> 
<translation> 
<mode></mode> 
<extent></extent> 
<publisher></publisher> 
<pubPlace></pubPlace> 
<date></date> 
<copyright></copyright> 
<sponsor></sponsor> 
<reviews></reviews> 
<comments></comments> 
</translation> 
<translationProcess> 
<direction></direction> 
<mode></mode> 
<type></type> 
</translationProcess> 
<author> 
<name></name> 
<gender></gender> 
<sexualOrientation></sexualOrientation> 
<Nationality></Nationality> 
</author> 
<sourceText> 
<language></language> 
<mode></mode> 
<status></status> 
<publisher></publisher> 
<pubPlace></pubPlace> 
<date></date> 
<comments></comments> 
</sourceText> 
</Header> 
Figure 2: Header for single volumes in TEC 
Each element in the headers has a start tag and an end tag. A start tag at the beginning of an element is represented by a balanced pair of angle brackets containing annotation strings, while a slash preceding the annotation strings indicates an end tag. The TEC header for single volumes includes the following information: title of the book, translator (where status refers to the question whether the translators have a full-time or part-time job and whether they work on a free-lance or in-house basis), translation (where extent means the number of words), translation process (where direction is relevant to whether or not the source language text is translated into the translator's mother tongue and type means a full translation in contrast to a summary, gist or excerpt), author of the original, source language text (where status refers to the problem whether the text is an original or a translation). The TEC header for collected works comprises an additional element called section. The section is repeated for each article or story contained in the collection. It includes information on the translator, the translation, the translation process, the author and the source language text of each article, story or paper in the collection. 
In addition to the header, the body of each file contains meta-information on the text structure represented in a modified version of the Standard Generalized Mark-Up Language (SGML) (see figure 3 for the most important tags). 
291
<title></title> 
<head></head> 
<sbhead></sbhead> 
<chapter n="..."> 
<p> 
<frontmatter></frontmatter> 
<backmatter></backmatter> 
<footnote></footnote> 
<endnote></endnote> 
<caption></caption> 
Figure 3: Textual mark-up in TEC 
For example, the following mark-up is included in each file of TEC: information on the (sub-)headings of the chapters or sections, the chapter number, the frontmatter (including introduction, preface etc.), the backmatter (including afterword, bibliography etc.), footnotes, endnotes, captions (e.g., for pictures or tables) and paragraphs1. 
The TEI-conformant header of the BNC includes the main elements displayed in figure 4: 
<teiHeader> 
<fileDesc> 
<titleStmt></titleStmt> 
<editionStmt></editionStmt> 
<extent></extent> 
<publicationStmt></publicationStmt> 
<sourceDesc></sourceDesc> 
</fileDesc> 
<encodingDesc> 
<projectDesc></projectDesc> 
<samplingDecl></samplingDecl> 
<editorialDecl></editorialDecl> 
<tagsDecl></tagsDecl> 
<refsDecl></refsDecl> 
<classDecl></classDecl> 
</encodingDesc> 
<profileDesc> 
<creation></creation> 
<langUsage></langUsage> 
<particDesc></particDesc> 
<settingDesc></settingDesc> 
<textClass></textClass> 
</profileDesc> 
<revisionDesc> 
<change> 
<date></date> 
<respStmt></respStmt> 
<para></para> 
</change> 
</revisionDesc> 
</teiHeader> 
Figure 4: Header in BNC 
The BNC header consists of a file description, an encoding description, a profile description and a revision description. The file description contains information on the title, the edition, the extent (i.e. size), the publication and the bibliographic source. The purpose of the coding project, the sampling criteria, the editorial principles, the linguistic annotation, the structure of the canonical references and the classification codes used for the texts within the corpus are spelled out in the encoding description. The profile description provides insight into the creation of the text, the language usage, the participants as well as their interaction, the settings of the communicative situation and the classification scheme with which the texts are categorized. The revision description explains the major changes which have taken place during the revision process. 
The mark-up scheme of the BNC is an standardized SGML application (ISO 8879). The main elements of the textual mark-up describe the use of headings, segments, words, punctuation, texts, spoken texts and paragraphs. There are, however, other elements and attributes which describe the header and the markup of the BNC in greater detail2. 
In the BNC, all special characters are presented by SGML entity references, which take the form of an 
1 For further information on TEC refer to the following URL: http://ceylon.ccl.umist.ac.uk/. 
2 More information on the BNC can be found under the following URL: http://www.hcu.ox.ac.uk/BNC/. 
292
ampersand followed by a mnemonic for the character and terminated by a semicolon (e.g., the representation &eacute;t&eacute; for the word été). In TEC, the transcription of these characters had to be carried out manually to make the data processable for annotation and querying tools. The use of international standards for the specification of application-independent document grammars (such as TEI for headers or SGML for mark-up) makes the corpora processable for computers as well as exchangeable and usable for researchers. 
Since TEC had not been tagged before, the part-of-speech tags produced by TnT were added to the corpus. The BNC was tagged according to the CLAWS tagging scheme (cf. Garside 1987). However, in order to make the corpora as comparable as possible in terms of their linguistic interpretation, the CLAWS tags were removed and the BNC was tagged once again using TnT. For the representation of the part-of-speech tags in the comparable corpus, the vertical TSV output format was transformed into a horizontal format and added to the text, as can be seen from figure 5. 
<body> 
<w AT> The </w> <w NP1> Sheikh </w> 
<w NN1> ' </w> <w FO> s </w> <w NN1> moustache </w> 
<w VBDZ> was </w> <w ICS> like </w> <w AT1> a </w> 
<w NN1> thistle </w> <w II> in </w> <w APPG> his </w> 
<w NN1> bed </w> <w YC> , </w> <w AT> no </w> 
<w NN1> matter </w> <w RRQ> how </w> 
<w PPHS1> he </w> <w VVD> tossed </w> 
<w CC> and </w> <w VVD> turned </w> <w YC> , </w> 
<w PPHS1> he </w> <w RR> only </w> 
<w VVD> rolled </w> <w AT> the </w> <w DAR> more </w> 
<w II> on </w> <w II> to </w> <w PPH1> it </w> 
<w YF> . </w> <w RR> Yet </w> <w PPHS1> he </w> 
<w VBDZ> was </w> <w II> at </w> <w AT> the </w> 
<w NN1> height </w> <w IO> of </w> <w APPG> his </w> 
<w NN1> power </w> <w YF> . </w> 
</body> 
Figure 5: Part-of-speech tagging in TEC 
In the horizontal tagging format, the part-of-speech tags are encoded as attributes of tokens. 
5 Corpus exploitation 
In order to find particular kinds of linguistic information in the corpus annotated in the ways described above, tools for querying the corpus for the features annotated are needed. For this purpose, the IMS Corpus Workbench (cf. Christ 1994) can be used. This concordance tool, with which it is possible to query for words and / or part-of-speech tags on the basis of regular expressions, consists of two modules: the corpus query processor (CQP) and the user interface (Xkwic). 
Importing TnT output to the workbench is a straightforward step since the preparatory steps were all carried out by the part-of-speech tagger TnT (see section 4). These steps include character set normalization, tokenization and sentence boundary detection, such that the input format of the tagged corpus is TSV. The corpus is then encoded in such a way that it can be queried by the system. After the encoding of the corpus, all attributes (words, part-of-speech tags etc.) are declared in a registry file, which is a crucial element for all operations of the corpus maintenance. 
The required information can then be queried using CQP, which implements a query language on the basis of the following regular expressions: concatenation, disjunction, negation, Kleene star, the plus and the interval operator. The results of the query are displayed using Xkwic. Figure 6 shows Xkwic with a query for passive. The query is based on the part-of-speech tags VB.* (forms of the verb be) followed by VVN.* (past participle) and zero to one words in between. The results are displayed in the KWIC (keyword in context) list indicating the number of matches. Xkwic offers the usual functionalities of a concordance program: concordances can, for example, be saved (all at once or in different sub-corpora), deleted, sorted and printed. The query history can be viewed, saved and loaded, and sub-corpora can be saved as well. Furthermore, an extended view on the KWIC concordances, a window for messages and an alignment window for parallel concordances can be displayed, too. Xkwic also calculates a frequency distribution for the first word or part-of-speech tag of the matches. The information thus provided makes it possible to analyze, for example, the use of aspect within passive constructions. 
293
Figure 6: Passive query with Xkwic 
As to the query of instances of complex syntactic constructions, typically several different queries need to be made to obtain all (satisfactory recall) and only the relevant matches (satisfactory precision). In some cases irrelevant matches have to be removed from the list manually. Thus, the most useful queries for the typical and untypical fiction features had to be found. There are three groups of queries: queries based on words, queries based on part-of-speech tags and queries based on both words and part-of speech tags which makes the queries quite complex. 
Since the two comparable fiction parts in TEC and BNC analyzed in this paper are both in English, the queries can be used on both sub-corpora. This procedure ensures consistent querying even in cases where the values for precision and recall do not attain 100 %. 
In order to attain comparable results, all frequencies of occurring features were normed using the following formula: 
frequency of feature occurrences x basis of norming 
= normed frequency 
word count 
The basis of norming is 5 million for TEC and for BNC, this being the approximate size of the two sub-corpora. 
As only the statistically significant results are relevant, all the frequencies of occurring features have to be subjected to a statistical test. The chi square test (cf. Oakes 1998) was employed in this paper. This test is a non-parametric statistical procedure for testing whether or not the distribution of feature occurrences is accidental. The first step of the test is to define the null hypothesis, according to which there is no statistically significant difference between the frequencies of feature occurrences found in the sub-corpora. If so, all the frequencies of occurrences would be the same as the sum of all frequencies divided by the number of categories. This value referred to as the expected value (E) is determined using the following formula: 
raw total x column total 
= E 
grand total of items 
In contrast to the expected value, the actual frequency is called the observed value (O). With these two values the chi square value can be calculated as follows: 
Ó (O-E)2 : E = X2 
294
For the significance tests carried out in this paper, the two-tailed/non-directional chi square test was employed and, additionally, Yates's correction was calculated given that the amount of data under investigation is rather large. For the same reason, the level of significance was set to 0.001, which means that the results have to reach 99.9 % to be statistically significant. 
6 The nature of TEC 
The frequent use of typical register features is an indicator of normalization in translated texts. A comparison with English originals makes it possible to determine whether or not the use of fiction features in English translations obeys the usage norms. Of course the same holds true for the untypical fiction features, so that, if the English translations show instances of normalization, the untypical fiction features are expected to be underused. If the English translations do not obey the English usage norms, the typical features are underused, whereas the untypical features occur more frequently in the English translations than in the English originals. 
In figure 7, the differences between the frequencies found in TEC and BNC are calculated and normed (as percentages). Thus, the positive percentages refer to the degree of normalization in connection with the typical features, whereas the negative percentages relate to the degree of anti-normalization for the typical features. The degrees of anti-normalization and normalization for the untypical features are summarized in figure 8. 
Figure 7: Degree of anti-/normalization for typical features 
Figure 8: Degree of anti-/normalization for untypical features 
As can be seen in figure 7, the higher frequency of typical fiction features to be found in TEC, as compared to the BNC, means that the translations show a trend towards normalization for typical fiction features. Untypical fiction features occur more frequently in TEC than in BNC, that is the translations show a trend towards anti-normalization for the untypical fiction features. 
Thus, it can be observed that the typical fiction features are normalized and the untypical fiction features are anti-normalized in English translated fiction compared to English original fiction. This means that English translated fiction is more narrative and situation-dependent than English original fiction texts are (since an overuse of the typical features can be attested). At the same time, English translated fiction is more abstract and less edited than original fiction (since an overuse of the untypical features can be attested). This leads to the conclusion that translators tend to conform to the typical 
295
patterns of the target language in terms of typical fiction features, the use thereof being exaggerated, whereas they tend to use the untypical fiction features less often. These observations show that a register shift between English original narrative texts and English translated narrative texts takes place in the sense that, owing to the extensive overuse of typical fiction features, English translated narrative texts are even more typical of their register. However, they point towards a more neutral register through the extensive overuse of untypical fiction features. 
In the following, a sample concordance taken from TEC, the normalized use of the typical fiction feature place adverbial is exemplified: 
  Marden is about to enter the house when he turns around: „You must leave the gun, naturally; 
  there wasn't enough air to breathe. After he went back outside, on the road, he noticed that 
  The bad weather will return. It was there, near the magazine (he can see it clearly from 
Here, the place adverbials ”around“, ”back“ and ”there“ are instances of normalization, showing the overuse of this typical fiction feature. In the following, another sample concordance from TEC is displayed which illustrates the anti-normalized use of the untypical fiction feature by-passive: 
  asked when it was all going to start, he was pacified by the chairman of the discussion: 
  a black man's head horribly shrunken. Rebecca was told all this and more by Nell about a week after her arrival at Broom House. 
  because I could hear voices in the corridor and I was distracted by the noise of the lift. It didn't 
In this example, the by-passives ”was pacified by“, ”was told all this and more by“ and ”was distracted by“ contribute to anti-normalization, meaning that this untypical fiction feature is overused. 
Reasons for normalization or anti-normalization regarding the typical and untypical fiction features are discussed in the following on the basis of examples from the German and French source language texts and their translations taken from TEC: 
  German: Während seiner Wanderung durch den Wald dachte er an 
  English: he walked through the forest and thought about 
This example shows that the German nominalization ”Wanderung“ is translated into the English verb ”walked“. Nominalization being a feature untypical of English fiction, it was transferred into the typical fiction feature past tense verbs. In this case, register-specific language use causes an instance of normalization. A similar phenomenon can be found in the following example: 
  German: Die Wahrheit über das dunkle Geheimnis fand man damals in ihrem Tagebuch. 
  English: At that time, her diary explained the dark secret 
Here, the typical fiction feature past tense verb is used both in the English and the German sentence, the semantic roles, however, are distributed differently as the German typology allows the frequent use of constructions like the impersonal passive alternative ”man“ (here: in combination with the PP ”in ihrem Tagebuch“), whereas the English typology provides other lexico-grammatical means of expressing the same meaning. In this case, the non-agentive NP subject ”her diary“ is used. Therefore, typological differences are responsible the translational choices. 
Those properties of translated text which are not register- or typology-related are inherent in the translation process itself. Evidence of this is contained in the following example: 
  German: nicht mehr die einzige Frau, die im Ort gefeiert wurde. 
  English: was no longer the only celebrity in town. 
In this example, the German relative clause ”die im Ort gefeiert wurde“ is rendered into the English NP ”celebrity in town“. The English translation is less explicit that the source language equivalent because the fact that a woman is the only celebrity in town (and not a man) is implied by the context and not spelled out in the German text. This example clearly entails a loss of explicitation. In contrast to this, the following example illustrates the co-occurrence of simplification and explicitation: 
  French: Au début, on crut a une bouderie d'enfant. 
  English: When they started, it was thought to be a fit of childish 
In this case, the French PP ”Au début“ is translated with the English subordinate clause ”When they started“. The use of the subordinate clause makes the English translation easier to read and more explicit, spelling out the meaning of the French PP. 
The examples discussed in this section show that the different properties of translation intervene with each other. Furthermore, they can be in a causal relation towards each other with, for example, explicitation or simplification causing normalization. 
7 Summary and outlook 
The present paper described the linguistic enrichment and exploitation of the Translational English Corpus. This approach was based on Baker's prototypical hypotheses (section 2 and 3), but profited from linguistically enriched data (section 4). This means that lexico-grammatical features such as Biber's register features could be exploited automatically (section 5). The results of the investigation, i.e. the nature of the Translational English Corpus were presented in section 6. 
Since normalization of the typical fiction features and anti-normalization of the untypical fiction 
296
features could be found in English translated fiction, as compared to English original fiction, the following can be attested: English translated fiction is more narrative and situation-dependent than English original fiction texts (the typical features being overused). At the same time, English translated fiction is more abstract and less edited than original texts (the untypical features being overused). Thus, English translated narrative texts are more typical than English original narrative texts as regards the use of typical features, whereas they point towards a more neutral register owing to the extensive overuse of untypical fiction features. 
On the basis of bilingual examples it could be shown that the sources of normalization and anti-normalization can partially be found in register- and typology-specific language use. The translation process, which triggers not only normalization and anti-normalization, but also other translation properties such as explicitation, simplification etc., can be seen as a further explanation. 
The empirical analysis which was carried out in the framework of this paper has to be extended by taking into consideration the source language texts of the translations. This would allow the analysis of the influence of the source language on a broader basis and with empirical methods. Additionally, a sub-corpus of comparable texts in the target language could be added to the cross-linguistic approach. As a result, the translations could be compared to originals in the target language on the basis of a more profound linguistic analysis. 
References 
Baker M 1995 Corpora in translation studies: An overview and some suggestions for future research. Target 7(2): 223-245. 
Baker M 1996 Corpus-based translation studies: The challenges that lie ahead. In Sommers H (ed), Terminology, LSP and Translation: Studies in Language Engineering in Honour of Juan C. Sager. Amsterdam, John Benjamins, pp 175-186. 
Biber D 1995 Dimensions of register variation: A cross-linguistic comparison. Cambridge, Cambridge University Press. 
Brants T 1999 TnT - A Statistical Part-of-Speech Tagger (User manual). Department of Computational Linguistics, Universität des Saarlandes, Saarbrücken, Germany. 
Christ O 1994 A modular and flexible architecture for an integrated corpus query system. In Proceedings of COMPLEX 94, 3rd Conference on Computational Lexicography and Text research, Budapest, pp 23-32. 
Garside R 1987 The CLAWS Word-tagging System. In Garside R, Leech G, Sampson G (eds), The Computational Analysis of English: A Corpus-based Approach. London, Longman. 
Oakes M P 1998 Statistics for Corpus Linguistics. Edinburgh, Edinburgh University Press. 
Sager J C 1984 Reflections on the didactic implications of an extended theory of translation. In Wilss W, Thome G (eds), Translation theory and its implementation in the teaching of translating and interpreting. Tübingen, Gunter Narr, pp 333-343. 
Sager J C 1994 Language engineering and translation: consequences of automation. Amsterdam, John Benjamins. 
Sampson G 1995 English for the Computer. Oxford, Oxford University Press. 
Schiller A, Teufel S, Stöckert C 1999 Guidelines für das Tagging deutscher Textcorpora mit STTS (Technical report). Stuttgart and Tübingen, University of Stuttgart and University of Tübingen. 
Teich E 2001 English-German contrast and commonality in system and text. A methodology for the investigation of parallel and multilingually comparable texts (Habilitationsschrift). Saarbrücken, Saarland University. 
297

Developing a tagset for automated part-of-speech tagging in Urdu 
Andrew Hardie 
Department of Linguistics and Modern English Language, University of Lancaster 
a.hardie@lancaster.ac.uk 
1. Abstract 
While part-of-speech tagging is an established technology for Western European languages such as English or Spanish, extending the technique to Urdu presents a range of interesting issues. There are some problems associated with the writing system, e.g. the problems of locating token boundaries in the Urdu version of the Arabic script. However, there are also linguistic issues. 
Little work has hitherto been done in the area of tagset creation for Urdu. The tagset discussed here was created in accordance with the EAGLES guidelines for morphosyntactic annotation of corpora. Although these guidelines were written to cover the languages of the European Union, they can be applied fairly easily to Urdu, which, coming as it does from another branch of the Indo-European family, is structurally quite similar. They can also be extended to deal with the idiosyncrasies presented by Urdu grammar. 
This paper will look at the process of creating one of the necessary resources for the development of a POS tagging system for Urdu, that of a suitable tagset, considering some of the problems encountered along the way. 
2. Introduction 
As part of the EMILLE project, it was decided to develop a POS tagger for one of the languages of South Asia covered by the project. Urdu was chosen as the language in question for a number of reasons. Firstly, it is widely spoken in the UK, both as a first and second language, and native speakers were available to be consulted at Lancaster where this part of the EMILLE project is taking place. Secondly, as the lingua franca of a multilingual community (that of South Asian Muslims) and the official language of Pakistan, Urdu has considerable political and cultural importance. Thirdly, there are a number of factors that we anticipated would make tagging Urdu more complicated than tagging any other EMILLE language. For example, the right-to-left directionality of the Indo-Perso-Arabic script in which Urdu is written and the presence of grammatical forms borrowed from Arabic and Persian, which are structurally quite distinct from Indo-Aryan forms, mean that Urdu represents a unique challenge within the EMILLE corpora. It seemed the best course of action to confront these problems by choosing Urdu as the language for which to develop POS tagging. 
The first stage of the work was to develop a tagset for use in Urdu texts and corpora, an area which has not been research extensively heretofore2. The next stage, now underway, is to test the tagset's usability in manual tagging, and build up a set of tagged texts to serve as training data for the final phase of this part of the project. This will be to automate the tagging and subsequently tag the whole of the EMILLE Urdu corpus. In this paper, the first, completed stage of this process is discussed: the devising of a tagset for Urdu based on the Urdu grammar of Schmidt (1999). 
3. Some background on the Urdu language 
Urdu is an Indo-European languages of the Indo-Aryan branch of the family. It is spoken in India and Pakistan (where it is the main official language) and also throughout the world 
Urdu is more closely related to Hindi than either is to any other language. Indeed, their high level of similarity has led some to consider them dialects of the same language (as reported by Bhatia and Koul 2000: ix-x). Masica (1991: 27) goes to so far as to suggest that by one definition of a dialect, Urdu and Hindi “are different literary styles based on the same linguistically defined subdialect”. Both originate from the dialect of the Delhi region and share their phonology, morphology and syntax in all but the smallest details. However, Urdu has borrowed a great deal of vocabulary (and its writing system) from Persian and Arabic, whilst Hindi has borrowed much vocabulary from Sanskrit. 
The most notable features of Urdu grammar are as follows (see Schmidt 1999 for further detail). Its word order is principally SXOV, with some flexibility in the order of these elements; subject pronouns are frequently dropped. It possesses postpositions rather than prepositions. Inflection on 
1 A project to develop language engineering resources for the languages of South Asia undertaken at the Universities of Lancaster and Sheffield: see Baker et al. (2003). 
2 We are only currently aware of one other study into this area, undertaken by the Department of Electronics of the Indian government (personal communication, Dr. I. Hasnain). We did not become aware of this study until a late stage in the research and so it is not discussed further in this paper. 
298
verbs, nouns and adjectives takes the form of fusional affixes, many of which are homophonous with one another. Nouns are inflected for case and number (singular/plural); the suffixes also indicate their gender (masculine/feminine). Gender agreement is marked by suffixes on verbs and adjectives; verbs show agreement either with the subject or with the direct object, although not both at once. Urdu verbs have one simple finite verb form (the subjunctive), two simple forms that may be finite or non-finite (the perfective and imperfective participles), and two further non-finite simple forms (the root and the infinitive). Tense and aspect, however, are mostly expressed through the use of irregular auxiliary elements within the verb phrase; there are also a number of frequently-used semi-auxiliary elements which confer semantic shading. 
The history of linguistic investigation into Urdu (and Hindi3) is described by Bhatia (1987). The current standard grammar is that of Schmidt (1999), although a great many pedagogical books have also been published (e.g. Bailey et al. 1956). There has also been a certain amount written on the language within the field of theoretical linguistics (e.g. Butt 1995). However, there remains some contention about certain points of its grammar. 
There is for instance some disagreement as to whether Urdu possesses three cases, or a much larger number. Urdu nouns generally display two clear cases, oblique (most commonly before postpositions) and nominative (elsewhere). A third case, the vocative, is identical with the oblique except in the plural. However, because of the structure of the Urdu noun phrase, postpositions always occur directly after the head noun of the noun phrase which they govern – in stark contrast to English, for instance, where a variety of elements may come between a preposition and the noun it governs. This phenomenon has led some (e.g. Kellogg 1875) to conclude that Urdu postpositions are actually “case suffixes” and that Urdu thus has a much larger variety of cases, possibly including accusative, dative, genitive, ergative, and so on. In this context it may be noted that the case system for other parts of speech (e.g. adjectives) displays the two-way nominative/oblique split that we would predict if the postpositions were not nominal affixes. 
Another contentious issue relates to whether the language should be considered to display split-ergativity or not. In past and perfective clauses in Urdu, the subject is marked with the postposition ne, which has no other use than to indicate such a subject, whereas the object remains in the nominative case. This can be treated as evidence of split ergativity in the language. However, it has been argued (e.g. by Butt 1995) that ne, rather than marking an ergative case, is a semantic marker of agentivity or volitionality. 
These differences of opinion on matters of Urdu grammar are not insignificant for the task of designing a set of morphosyntactic categories. Ideally one would wish to compose a markup scheme that does not commit the user to a particular theoretical analysis (as suggested, for example, by Leech 1997), thus making the categories equally acceptable and useful to researchers on either side of the two debates mentioned above. 
4. A model for categorisation of the Urdu language 
To create the linguistic categories of a tagset, it is necessary to have a model of the language to categorise. An ideal approach would be to derive this model from empirical data – however, this cannot be done prior to the creation of a tagset. A native speaker of a language could use their own intuitions about the language as a model, but as none of the researchers on the EMILLE project are native speakers of Urdu, this is not an option. The only remaining option is to make use of a published description of Urdu grammar as a model of the language. 
The decision was taken to rely on the current standard grammar of Urdu by Schmidt (1999) to furnish a model of the language. It would probably have been preferable to rely on a synthesis of a range of published descriptions; however, in practice this was impossible. Most other recent works fall into two categories: pedagogical manuals, and works in theoretical linguistics that look at Urdu (or “Hindi-Urdu”). It might be assumed that the latter group of studies could be used to compile, in conjunction with Schmidt (1999), a synthesised model of the language on which to base the tagset categories. However, this is not so. Works in theoretical linguistics which concentrate on Hindi-Urdu tend to focus on one aspect of the language to the exclusion of the rest4. Thus they are of little use in developing a complete model of the language. Most of the studies listed in Masica's comprehensive bibliography (1991: 493-497, 510) are of this kind. Another type of linguistic study is the language survey, some of which have been published covering Urdu (e.g. Kachru 1990). These, while they cover the whole language, are not sufficiently detailed to constitute adequate models. Likewise, pedagogical 
3 Until the early 20th Century, the distinction between Hindi and Urdu was not as clearly defined as it later became. 
4 For example, Butt (1995) looks in detail at complex verb phrases, touching in cursory fashion or not at all on other aspects of Urdu grammar. 
299
works5 – aimed at language learners and covering mostly no more than what it is anticipated a learner would need to know first – are generally too partial to contribute a large amount to a composite model. 
For these reasons, Schmidt (1999) was used as the model of Urdu grammar for the definition of the tagset. This necessitated the assumption that the model of Urdu presented by Schmidt is identical to the actual language, which may well be unwarranted. However, there was no alternative to this assumption. 
5. Some background on the EAGLES guidelines 
The Urdu tagset described in this paper was created in accordance with the EAGLES guidelines on morphosyntactic annotation (Leech and Wilson 1999). These guidelines were designed to help standardise tagsets for what were then the official languages of the European Union6. 
The EAGLES guidelines outline a set of features for tagsets, some recommended, some optional. Simultaneously, a scheme of encoding all these features into an “intermediate tagset” is given. This is an encoding using numerical values for the assorted EAGLES attributes. The choice of how the features are encoded within a given EAGLES-compliant tagset is left to the user, as long as the categories thus created can also be expressed using the intermediate tagset. The purpose of the intermediate encoding is to allow mapping between any two tagsets created in compliance with the EAGLES guidelines, thus ensuring their compatibility. EAGLES tags are defined as sets of morphosyntactic attribute-value pairs (e.g. Gender is an attribute that can have the values Masculine, Feminine or Neuter). In the intermediate tagset, these attribute-value pairs are arranged in a hierarchical structure. 
The obligatory feature of an EAGLES-compatible tagset is a set of “major word categories”; the EAGLES guidelines suggest thirteen of these, such as noun, verb, adjective etc. The recommended and optional attributes are then organised by these major word categories, and do not necessarily correspond across word classes. For example, the first recommended attribute is Type (Common/Proper) for nouns but Person (First/Second/Third) for verbs and Degree (Positive/Comparative/Superlative) for adverbs. The recommended attributes also cover number, gender, case, finiteness, tense, voice, and other important features which one would anticipate being of relevance to a range of languages. The optional part of the recommendations consists of similar attributes of more narrow applicability, and some additional values – mainly specific to one language or a small group of languages7 – for the recommended attributes. 
The EAGLES guidelines provide a flexible framework that in theory encompasses all the things which one would wish to mark up, without restricting the freedom of the tagset designer. It promotes consistency and reusability of linguistic resources for different languages and discourages “reinvention of the wheel”. However, as Leech and Wilson point out, “[i]t remains to be seen how far these guidelines can be extended, without substantial revision, to other languages” (1999: 58). 
6. Extending the EAGLES scheme to Urdu 
While Urdu did not fall under EAGLES’ EU remit, it was decided to work with this international standard in order to ensure the maximum utility of the final tagged corpus. Furthermore, from a typological perspective it is not unreasonable to expect that the EAGLES guidelines would prove compatible with Urdu on the grounds that both Urdu and the original EAGLES languages are all from the Indo-European family. Genetically speaking, Urdu is no more distant from EAGLES languages such as German and Italian than they are from one another (although of course this does not take into account areal features which would make Urdu more divergent due to its geographical distance from the original EAGLES languages). Furthermore, impressionistically, the vast majority of the inflectional system of Urdu is very reminiscent of EAGLES languages such as French, German, and so on. 
Indeed, it transpired that most of design features of the attribute-value system outlined in the EAGLES guidelines were suitable for application in the design of the Urdu tagset. The major categories in Urdu – nouns, pronouns, verbs, adjectives, adverbs, postpositions and conjunctions – are virtually identical to the equivalent categories as defined in EAGLES. Note that this is not the case for all languages. Others who have compiled tagsets for genetically unrelated languages such as Arabic (Khoja et al. 2001) and Korean (e.g. Chae and Choi 2000) have of necessity employed categorisation strategies alien to the system laid out in the EAGLES guidelines. 
It was therefore possible to link the Urdu tagset (see the Appendix) directly to the EAGLES guidelines, using the EAGLES intermediate tagset as described by Leech and Wilson (1999), with only 
5 For instance, Bhatia and Koul (2000), Barz (1977), Bailey et al. (1956) 
6 English, Dutch, German, Danish, French, Spanish, Portuguese, Italian and Greek 
7 For example, the fairly language-specific attribute aux-function, which relates to the difference in English between the primary auxiliaries (do/be/have) and the modals (can/may/will etc.) 
300
some minor modifications. For instance, although the EAGLES guidelines deal very well with the gender, case and number system of Urdu (as described above), since there was no value for “oblique case” in the EAGLES system, the value for “dative case” was used instead, on the grounds that the usage of the Urdu oblique corresponds quite closely to that of the dative in some EU languages, such as German. Most other general attributes in the EAGLES system were similarly amenable to being extended to Urdu. The verbal system proved a little more problematic, in the sense that mapping the mood, tense and finiteness features outlined in the EAGLES attribute-value system onto those found in the Urdu language was less straightforward. 
However, the greatest difficulty arose in dealing with the minor, idiosyncratic features of Urdu – whilst the idiosyncratic features of the EU languages are covered by the EAGLES guidelines this is obviously not the case for Urdu. Some of these difficulties are discussed in the following section. However, on the whole, the EAGLES guidelines have proved a robust and useful framework within which to approach Urdu POS tagging. 
7. Points of difficulty in devising an EAGLES-compliant tagset for Urdu 
7.1. Minor idiosyncratic features of the language 
Some idiosyncratic features of Urdu corresponded to nothing built into the EAGLES guidelines. These features include: the appearance of case on some verbal elements8; the distinction between ‘marked’ and ‘unmarked’ nouns; the Urdu honorific pronoun ap, which does not fit easily into any of the EAGLES categories for pronouns; the borrowed Persian enclitic called izafat; However, none of these problems were insurmountable. For instance, the EAGLES guidelines include a “Unique” category for words that effectively form a class on their own; this category was used for some idiosyncratic features of Urdu which resisted classification elsewhere (e.g. izafat, or the use of a marker-word for yes/no questions). Others could be handled by means of minor extensions to the EAGLES system, e.g. to handle case on verbs a case attribute was added to the end of the EAGLES intermediate tags for verbs9. 
However, some idiosyncrasies could not be resolved this easily and uncontroversially. An example is the definite article. Urdu does not really have a definite article, but the Arabic definite article al– does occur in loans from that language. The question arises as to whether it is really appropriate to equate this via the intermediate tagset with the definite articles of the EAGLES languages, given that the distribution of al– in Urdu is much more restricted than, say, that of the in English or le/la/les in French. 
Another example of this type relates to the distinction between relative and interrogative pronouns. In the EAGLES guidelines this distinction is made only at the optional level (attribute Wh-type), whereas the distinction between demonstrative and interrogative/relative is made by the recommended-level attribute Pronoun-type. This maps easily onto sets of words such as this~that~what in English and similar EAGLES languages. In Urdu, however, there is a four-way distinction between proximal demonstratives, distal demonstratives, interrogatives and relatives (for example, the pronouns yah, vah, kya, jo respectively). This distinction is maintained throughout a system of pronouns, determiners and adverbs. Thus, in Urdu the distinction between interrogatives and relatives, which is only made by the EAGLES guidelines at the secondary optional level, appears a priori to be as significant as that between demonstrative and relative. To categorise this in the tagset, it was necessary for a high-level distinction in the Urdu tags (PY… PV… PK… PJ…) to map to distinctions made at varying levels in the intermediate tagset – an unfortunately inelegant solution. 
7.2. Token division in Urdu 
Urdu is written in Indo-Perso-Arabic, a form of Perso-Arabic which adds letters for some characteristically Indo-Aryan sounds (e.g. the retroflex consonants) and uses some original Arabic 
8 The participles and the infinitive can all display case. 
9 A similar approach was taken to solve problem of encoding the marked/unmarked distinction found in Urdu nouns, and the marking of case, gender and number on one Urdu preposition. In all these cases, entire attributes were added to the end of the EAGLES intermediate tags as described by Leech and Wilson (1999), rather than adding more values to the attributes that already existed. This was to make it less problematic for an EAGLES-compliant computer application to ignore the extra elements added to handle Urdu (simply by passing over the extra attributes), while retaining in the intermediate tagset all the information in the full Urdu tagset. A concrete example is the intermediate tag V10212101000000 (first person plural subjunctive lexical verb) which has an added “0” on the end compared to what would be the equivalent tag in Leech and Wilson's description of the intermediate tagset: the extra attribute allows the tagset to encode case on those verb forms that display it. 
301
letters with different phonetic values10. One aspect of this writing system and its application to Urdu is that many things described in the literature on Urdu grammar as suffixes are actually written as independent words (for example, the verbal auxiliary element indicating future tense: see Schmidt 1999: 106, Bhatia and Koul 2000: 331-332). 
For consistency, the (essentially arbitrary) decision was taken to treat every orthographic space as a word break even if it occurs within a lexical word11. However, this meant that the tagset had to contain some means of tagging elements which did not constitute entire words. 
For example, the word zimmah dar (“responsible”) consists of a root plus a derivational affix, with an orthographic space between them. Although it would be attractive to describe this as a phrase for the purposes of tagging, this cannot be supported linguistically: the same suffix appears without a word break in other contexts (e.g. samajhdar, “sensible”), and moreover other derivational suffixes can be added (e.g. zimmah dari, “responsibility”). A number of other derivational suffixes behave in the same way, as do some simple lexemes, for example Teli fon, “telephone”. The phenomenon appears to be common in borrowed vocabulary (dar derives from Persian, Teli fon from English). 
This created a problem12: how to tag two tokens which make up a single morphological word? After considering various solutions to this problem, it was decided to incorporate a tag into the categorisation system for a non-grammatical lexical element, i.e. a token which has no effect on the syntax of the clause and is dependent for its grammar on the subsequent token. This tag is LL, and it would be used thus13: 
samajhdar_JJU 
zimmah_LL dar_JJU 
7.3. Loan words and inflections 
As mentioned above, Urdu has borrowed a great deal of vocabulary form Arabic and Persian (and also, more recently, from English). Some of the words thus borrowed are inflected forms (Schmidt 1999: 253, 259-264). However, it is not currently clear a) how extensive these are in Urdu, and b) whether they are used as lexical roots just like any other loan word or whether they are used as actual inflected forms. The latter would pose a problem, as some special tagging might be necessitated for grammatical forms with no equivalent in the “native” Urdu vocabulary. 
While this is evidently an important issue, there is not currently enough knowledge of all aspects of the “loan-word” phenomenon in Urdu to incorporate a comprehensive means of handling it into the tagset. Therefore it is our intention at a later date to study this phenomenon in a corpus of Urdu text tagged according to the current tagset, on the basis of which changes to subsequent tagsets may be necessary. 
8. Conclusion 
Experience in developing an annotation scheme for Urdu has demonstrated that the scope of the EAGLES guidelines may productively be extended beyond the languages for which they were originally devised, to genetically and typologically similar languages such as those of the Indo-Aryan group. Furthermore, although certain modifications to the scheme were required, these were only minor and did not involve any disruption to the large-scale organisation of the EAGLES categories. Although some features of the Indo-Perso-Arabic alphabet create problems for tokenisation and the assignation of tags to some tokens, these difficulties are not insuperable. As a result of this process, a tagset for use with Urdu texts and corpora has been developed (see Appendix). 
It should also be noted that, while a tagset is the most obvious prerequisite resource for tagging, there are other significant resources which must be in place prior to the development of an 
10 Because this form of the Arabic writing system is used by other Indo-Aryan languages (e.g. some forms of Western Punjabi), I refer to it as Indo-Perso-Arabic; it is popularly referred to simply as the “Urdu alphabet” or “Urdu script” (e.g. by Nakanishi 1980: 36). 
11 Word breaks are also introduced in some places where there is no orthographic space, e.g. where clitics precede/follow another word without a break. 
12 This is only partially analogous to the problem of multi-word idioms in English and similar languages that leads, for example, to phrases such as “given that” being tagged as the two parts of a single subordinating conjunction. In these cases, there is also an analysable internal syntactic structure (in this case, verbal past participle followed by conjunction). In the Urdu case, it would be very difficult to assign any internal structure to Teli fon, and the internal structure of zimmah dar would of necessity be morphological rather than morphosyntactic – both undesirable analyses. 
13 The underscore character is used here for visual clarity: the system under development actually uses a columnar format for communication between different modules and outputs SGML compliant word tags. 
302
automated tagger. These include a set of guidelines for application of the tags, and a lexicon (or else a reliable means of acquiring such automatically from tagged text). The guidelines are of course needed for the generation of a manually tagged training/test corpus, a sine qua non of automated tagger design14. Development of these resources is currently underway. 
9. References 
Bailey, TG, ed. by Firth, JR and Harley, AH 1956 Teach Yourself Urdu. New York: David McKay Company. 
Baker, JP, Hardie, A, McEnery, A and Jayaram, BD 2003 Corpus data for South Asian language processing. Paper given at the Corpus Linguistics 2003 conference, Lancaster. 
Barz, RK 1977 An Introduction to Hindi and Urdu. Canberra: Australian National University Press. 
Bhatia, TK and Koul, A 2000 Colloquial Urdu. London: Routledge. 
Brill, E and Pop, M 1999 Unsupervised learning of disambiguation rules for part of speech tagging. In: Armstrong, S, Church, K, Isabelle, P, Manzi, S, Tzoukermann, E and Yarowsky, D (eds.) Natural language processing using very large corpora. Dordrecht: Kluwer Academic Publishers. 
Butt, M 1995 The Structure of Complex Predicates in Urdu. Stanford, California: CSLI Publications. 
Chae, Y-S and Choi, K-S 2000 Introduction of KIBS (Korean Information Base System) Project. In: Gavrilidou, M, et al. (eds.) Second International Conference on Language Resources and Evaluation: Proceedings. 3 volumes. Athens: European Language Resources Association. Vol. 3, 1731-1735. 
Kachru, Y (1990) Hindi-Urdu. In: Comrie, B (ed.) 1990 The Major Languages of South Asia, the Middle East and Africa. London: Routledge. 
Kellogg, SH 1875 A grammar of the Hindí language, in which are treated the High Hindí, Braj, and the eastern Hindí of the Rámáyan of Tulsí Dás. (Reprinted 1965.) London: Routledge and Kegan Paul. 
Khoja, S, Garside, R, and Knowles, G 2001 A tagset for the morphosyntactic tagging of Arabic. Paper given at the Corpus Linguistics 2001 conference, Lancaster. 
Leech, G 1997 Grammatical tagging. In: Garside, R, Leech, G and McEnery A (eds.) (1997) Corpus annotation: linguistic information from computer text corpora. London: Longman. 
Leech, G and Wilson, A. 1999 Standards for tagsets. (Edited version of EAGLES Recommendations for the Morphosyntactic Annotation of Corpora 1996: available on the internet at http://www.ilc.pi.cnr.it/EAGLES96/annotate/annotate.html .) In: van Halteren, H (ed.) (1999) Syntactic wordclass tagging. Dordrecht: Kluwer Academic Publishers. 
Masica, CP 1991 The Indo-Aryan languages. Cambridge: Cambridge University Press. 
Merialdo, B 1994 Tagging English text with a probabilistic model. In: Computational Linguistics, 20 (2): 155-171. 
Nakanishi, A. 1980 Writing systems of the world. Tokyo: Charles E. Tuttle Company. 
Platts, JT. 1884 A Dictionary of Urdu, Classical Hindi, and English. Oxford: Oxford University Press (reprinted 1965). 
Schmidt, RL 1999 Urdu: an essential grammar. London: Routledge. 
Appendix: the U1 tagset 
Tag Description AL Arabic definite article AU Interjection CC Coordinating conjunction CCC Correlative coordinating conjunction CS Subordinating conjunction FF Foreign word FX Non-Perso-Arabic string FO Formula (e.g. mathematical) FZ Letter of the alphabet FS Other symbol 
14 Although techniques have been developed for training a tagger based on untagged data (see for instance Merialdo 1994, Brill and Pop 1999), all such techniques appear to depend on the availability of a reliably accurate lexicon in which words are linked to all their possible tags. In the case of a language that has not previously been tagged automatically, such a lexicon must also be derived from manually tagged data – or else written item by item using a linguist's intuition, a task which may well be as onerous as manual tagging in any case. Thus there is no way to avoid the effort associated with manual tagging of tens of thousands of words of text. 
303
FA Acronym FB Abbreviation FU Other unclassifiable non-Urdu element IB Preposition II Unmarked postposition IIC Clitic postposition e, e~, he~ IIM1N Marked masculine singular nominative postposition ka IIM1O Marked masculine singular oblique postposition ke IIM2N Marked masculine plural nominative postposition ke IIM2O Marked masculine plural oblique postposition ke IIF1N Marked feminine singular nominative postposition ki IIF1O Marked feminine singular oblique postposition ki IIF2N Marked feminine plural nominative postposition ki IIF2O Marked feminine plural oblique postposition ki JJM1N Marked masculine singular nominative adjective JJM1O Marked masculine singular oblique adjective JJM2N Marked masculine plural nominative adjective JJM2O Marked masculine plural oblique adjective JJF1N Marked feminine singular nominative adjective JJF1O Marked feminine singular oblique adjective JJF2N Marked feminine plural nominative adjective JJF2O Marked feminine plural oblique adjective JJU Unmarked adjective JD Indefinite determiner JDNU Cardinal number JDNUO Oblique cardinal number JDNUC Pre-multiplicative clitic cardinal number du–, ti-, cau– JDNM1N Masculine singular nominative ordinal number JDNM1O Masculine singular oblique ordinal number JDNM2N Masculine plural nominative ordinal number JDNM2O Masculine plural oblique ordinal number JDNF1N Feminine singular nominative ordinal number JDNF1O Feminine singular oblique ordinal number JDNF2N Feminine plural nominative ordinal number JDNF2O Feminine plural oblique ordinal number JDFU Unmarked fraction JDFM1N Masculine singular nominative fraction JDFM1O Masculine singular oblique fraction JDFM2N Masculine plural nominative fraction JDFM2O Masculine plural oblique fraction JDFF1N Feminine singular nominative fraction JDFF1O Feminine singular oblique fraction JDFF2N Feminine plural nominative fraction JDFF2O Feminine plural oblique fraction JDYM1N Masculine singular nominative proximal demonstrative adjective (itna, aisa) JDYM1O Masculine singular oblique proximal demonstrative adjective (itne, aise) JDYM2N Masculine plural nominative proximal demonstrative adjective (itne, aise) JDYM2O Masculine plural oblique proximal demonstrative adjective (itne, aise) JDYF1N Feminine singular nominative proximal demonstrative adjective (itni, aisi) JDYF1O Feminine singular oblique proximal demonstrative adjective (itni, aisi) JDYF2N Feminine plural nominative proximal demonstrative adjective (itni, aisi) JDYF2O Feminine plural oblique proximal demonstrative adjective (itni, aisi) JDVM1N Masculine singular nominative distal demonstrative adjective (utna, vaisa) JDVM1O Masculine singular oblique distal demonstrative adjective (utne, vaise) JDVM2N Masculine plural nominative distal demonstrative adjective (utne, vaise) JDVM2O Masculine plural oblique distal demonstrative adjective (utne, vaise) JDVF1N Feminine singular nominative distal demonstrative adjective (utni, vaisi) JDVF1O Feminine singular oblique distal demonstrative adjective (utni, vaisi) JDVF2N Feminine plural nominative distal demonstrative adjective (utni, vaisi) JDVF2O Feminine plural oblique distal demonstrative adjective (utni, vaisi) JDKM1N Masculine singular nominative interrogative adjective (kitna, kaisa) JDKM1O Masculine singular oblique interrogative adjective (kitne, kaise) JDKM2N Masculine plural nominative interrogative adjective (kitne, kaise) JDKM2O Masculine plural oblique interrogative adjective (kitne, kaise) JDKF1N Feminine singular nominative interrogative adjective (kitni, kaisi) JDKF1O Feminine singular oblique interrogative adjective (kitni, kaisi) JDKF2N Feminine plural nominative interrogative adjective (kitni, kaisi) JDKF2O Feminine plural oblique interrogative adjective (kitni, kaisi) JDJM1N Masculine singular nominative relative adjective (jitna, jaisa) JDJM1O Masculine singular oblique relative adjective (jitne, jaise) JDJM2N Masculine plural nominative relative adjective (jitne, jaise) 
304
JDJM2O Masculine plural oblique relative adjective (jitne, jaise) JDJF1N Feminine singular nominative relative adjective (jitni, jaisi) JDJF1O Feminine singular oblique relative adjective (jitni, jaisi) JDJF2N Feminine plural nominative relative adjective (jitni, jaisi) JDJF2O Feminine plural oblique relative adjective (jitni, jaisi) JXGM1N Masculine singular nominative multiplicative marker guna JXGM1O Masculine singular oblique multiplicative marker gune JXGM2N Masculine plural nominative multiplicative marker gune JXGM2O Masculine plural oblique multiplicative marker gune JXGF1N Feminine singular nominative multiplicative marker guni JXGF1O Feminine singular oblique multiplicative marker guni JXGF2N Feminine plural nominative multiplicative marker guni JXGF2O Feminine plural oblique multiplicative marker guni JXSM1N Masculine singular nominative adjectival particle sa JXSM1O Masculine singular oblique adjectival particle se JXSM2N Masculine plural nominative adjectival particle se JXSM2O Masculine plural oblique adjectival particle se JXSF1N Feminine singular nominative adjectival particle si JXSF1O Feminine singular oblique adjectival particle si JXSF2N Feminine plural nominative adjectival particle si JXSF2O Feminine plural oblique adjectival particle si JXVM1N Masculine singular nominative adjectival / occupational particle vala JXVM1O Masculine singular oblique adjectival / occupational particle vale JXVM2N Masculine plural nominative adjectival / occupational particle vale JXVM2O Masculine plural oblique adjectival / occupational particle vale JXVF1N Feminine singular nominative adjectival / occupational particle vali JXVF1O Feminine singular oblique adjectival / occupational particle vali JXVF2N Feminine plural nominative adjectival / occupational particle vali JXVF2O Feminine plural oblique adjectival / occupational particle vali LL Nongrammatical lexical element NNMM1N Common marked masculine singular nominative noun NNMM1O Common marked masculine singular oblique noun NNMM1V Common marked masculine singular vocative noun NNMM2N Common marked masculine plural nominative noun NNMM2O Common marked masculine plural oblique noun NNMM2V Common marked masculine plural vocative noun NNMF1N Common marked feminine singular nominative noun NNMF1O Common marked feminine singular oblique noun NNMF1V Common marked feminine singular vocative noun NNMF2N Common marked feminine plural nominative noun NNMF2O Common marked feminine plural oblique noun NNMF2V Common marked feminine plural vocative noun NNUM1N Common unmarked masculine singular nominative noun NNUM1O Common unmarked masculine singular oblique noun NNUM1V Common unmarked masculine singular vocative noun NNUM2N Common unmarked masculine plural nominative noun NNUM2O Common unmarked masculine plural oblique noun NNUM2V Common unmarked masculine plural vocative noun NNUF1N Common unmarked feminine singular nominative noun NNUF1O Common unmarked feminine singular oblique noun NNUF1V Common unmarked feminine singular vocative noun NNUF2N Common unmarked feminine plural nominative noun NNUF2O Common unmarked feminine plural oblique noun NNUF2V Common unmarked feminine plural vocative noun (Tags for proper nouns are as the tags for common nouns, except that they begin NP instead of NN) OO Persian compound-forming conjunction o PPM1N First person singular nominative personal pronoun (mai~) PPM1O First person singular oblique personal pronoun (mujh) PPM2N First person plural nominative personal pronoun (ham) PPM2O First person plural oblique personal pronoun (ham) PPT1N Second person singular nominative personal pronoun (tu) PPT1O Second person singular oblique personal pronoun (tujh) PPT2N Second person plural nominative personal pronoun (tum) PPT2O Second person plural oblique personal pronoun (tum) PGM1M1N First person singular masculine singular nominative possessive adjective (mera) PGM1M1O First person singular masculine singular oblique possessive adjective (mere) PGM1M2N First person singular masculine plural nominative possessive adjective (mere) PGM1M2O First person singular masculine plural oblique possessive adjective (mere) PGM1F1N First person singular feminine singular nominative possessive adjective (meri) PGM1F1O First person singular feminine singular oblique possessive adjective (meri) PGM1F2N First person singular feminine plural nominative possessive adjective (meri) 
305
PGM1F2O First person singular feminine plural oblique possessive adjective (meri) PGM2M1N First person plural masculine singular nominative possessive adjective (hamara) PGM2M1O First person singular masculine singular oblique possessive adjective (hamare) PGM2M2N First person singular masculine plural nominative possessive adjective (hamare) PGM2M2O First person singular masculine plural oblique possessive adjective (hamare) PGM2F1N First person singular feminine singular nominative possessive adjective (hamari) PGM2F1O First person singular feminine singular oblique possessive adjective (hamari) PGM2F2N First person singular feminine plural nominative possessive adjective (hamari) PGM2F2O First person singular feminine plural oblique possessive adjective (hamari) PGT1M1N Second person singular masculine singular nominative possessive adjective (tera) PGT1M1O Second person singular masculine singular oblique possessive adjective (tere) PGT1M2N Second person singular masculine plural nominative possessive adjective (tere) PGT1M2O Second person singular masculine plural oblique possessive adjective (tere) PGT1F1N Second person singular feminine singular nominative possessive adjective (teri) PGT1F1O Second person singular feminine singular oblique possessive adjective (teri) PGT1F2N Second person singular feminine plural nominative possessive adjective (teri) PGT1F2O Second person singular feminine plural oblique possessive adjective (teri) PGT2M1N Second person plural masculine singular nominative possessive adjective (tumhara) PGT2M1O Second person singular masculine singular oblique possessive adjective (tumhare) PGT2M2N Second person singular masculine plural nominative possessive adjective (tumhare) PGT2M2O Second person singular masculine plural oblique possessive adjective (tumhare) PGT2F1N Second person singular feminine singular nominative possessive adjective (tumhari) PGT2F1O Second person singular feminine singular oblique possessive adjective (tumhari) PGT2F2N Second person singular feminine plural nominative possessive adjective (tumhari) PGT2F2O Second person singular feminine plural oblique possessive adjective (tumhari) PY1N Singular nominative proximal demonstrative pronoun (yah) PY1O Singular oblique proximal demonstrative pronoun (is) PY2N Plural nominative proximal demonstrative pronoun (yah) PY2O Plural oblique proximal demonstrative pronoun (in) PY2E Plural oblique proximal demonstrative pronoun before ne (inho~) PV1N Singular nominative distal demonstrative pronoun (vah) PV1O Singular oblique distal demonstrative pronoun (us) PV2N Plural nominative distal demonstrative pronoun (vah) PV2O Plural oblique distal demonstrative pronoun (un) PV2E Plural oblique distal demonstrative pronoun before ne (unho~) PK1N Singular nominative interrogative pronoun (kya, kaun) PK1O Singular oblique interrogative pronoun (kis) PK2N Plural nominative interrogative pronoun (kya, kaun) PK2O Plural oblique interrogative pronoun (kin) PK2E Plural oblique interrogative pronoun before ne (kinho~) PJ1N Singular nominative relative pronoun (jo) PJ1O Singular oblique relative pronoun (jis) PJ2N Plural nominative relative pronoun (jo) PJ2O Plural oblique relative pronoun (jin) PJ2E Plural oblique relative pronoun before ne (jinho~) PRF Reflexive pronoun (ap, xud) PRC Reciprocal pronoun (apas) PGRM1N Masculine singular nominative reflexive possessive adjective (apna) PGRM1O Masculine singular oblique reflexive possessive adjective (apne) PGRM2N Masculine plural nominative reflexive possessive adjective (apne) PGRM2O Masculine plural oblique reflexive possessive adjective (apne) PGRF1N Feminine singular nominative reflexive possessive adjective (apni) PGRF1O Feminine singular oblique reflexive possessive adjective (apni) PGRF2N Feminine plural nominative reflexive possessive adjective (apni) PGRF2O Feminine plural oblique reflexive possessive adjective (apni) PNN Nominative indefinite pronoun (koi, kuch, sab) PNO Oblique indefinite pronoun (kisi, kuch, sabho~) PA Honorific pronoun (ap) QQ Question marker kya RR General adverb RRJ General adverb derived from adjective RD Degree adverb RM Modal adverb RMN Negative modal adverb (nahi~, nah, mat) RY Proximal demonstrative adverb (ab, yaha~, idhar, yu~) RYJ Proximal demonstrative adverb derived from adjective (aise) RV Distal demonstrative adverb (tab, vaha~, udhar, tyu~) RVJ Distal demonstrative adverb derived from adjective (vaise) RK Interrogative adverb (kab, kaha~, kidhar, kyo~) RKJ Interrogative adverb derived from adjective (kaise) RJ Relative adverb (jab, jaha~, jidhar, ju~) 
306
RJJ Relative adverb derived from adjective (jaise) TT Sentence tag-word VV0 Root form lexical verb VVNM1N Infinitive lexical verb, masculine singular nominative VVNM1O Infinitive lexical verb, masculine singular oblique VVNM2 Infinitive lexical verb, masculine plural nominative VVNF1 Infinitive lexical verb, feminine singular nominative VVNF2 Infinitive lexical verb, feminine plural nominative VVTM1N Masculine singular (nominative) imperfective participle lexical verb VVTM1O Masculine singular oblique imperfective participle lexical verb VVTM2N Masculine plural (nominative) imperfective participle lexical verb VVTM2O Masculine plural oblique imperfective participle lexical verb VVTF1N Feminine singular (nominative) imperfective participle lexical verb VVTF1O Feminine singular oblique imperfective participle lexical verb VVTF2N Feminine plural (nominative) imperfective participle lexical verb VVTF2O Feminine plural oblique imperfective participle lexical verb VVYM1N Masculine singular (nominative) perfective participle lexical verb VVYM1O Masculine singular oblique perfective participle lexical verb VVYM2N Masculine plural (nominative) perfective participle lexical verb VVYM2O Masculine plural oblique perfective participle lexical verb VVYF1N Feminine singular (nominative) perfective participle lexical verb VVYF1O Feminine singular oblique perfective participle lexical verb VVYF2N Feminine plural (nominative) perfective participle lexical verb VVYF2O Feminine plural oblique perfective participle lexical verb VVSM1 First person singular subjunctive lexical verb VVSM2 First person plural subjunctive lexical verb VVST1 Second person singular subjunctive lexical verb VVST2 Second person plural subjunctive lexical verb VVSV1 Third person singular subjunctive lexical verb VVSV2 Third person plural subjunctive lexical verb VVIT1 Second person singular imperative lexical verb VVIT2 Second person singular imperative lexical verb VVIA Second person honorific imperative lexical verb (Tags for general auxiliary verbs are as the tags for lexical verbs, except that they begin VX instead of VV; there is a third set of parallel tags, beginning VH, for forms of the auxiliary verb hona, “be” – see also the tags for the irregular past and present tenses of hona below) VGM1 Masculine singular future auxiliary ga VGM2 Masculine plural future auxiliary ge VGF1 Feminine singular future auxiliary gi VGF2 Feminine plural future auxiliary gi VRM1 Masculine singular durative auxiliary raha VRM2 Masculine plural durative auxiliary rahe VRF1 Feminine singular durative auxiliary rahi VRF2 Feminine plural durative auxiliary rahi VC1 Singular cahie-type auxiliary VC2 Plural cahie-type auxiliary VHHM1 First person singular indicative present hu~ VHHM2 First person plural indicative present hai~ VHHT1 Second person singular indicative present hai VHHT2 Second person plural indicative present ho VHHV1 Third person singular indicative present hai VHHV2 Third person plural indicative present hai~ VHPM1 Masculine singular indicative past tha VHPM2 Masculine plural indicative past the VHPF1 Feminine singular indicative past thi VHPF2 Feminine plural indicative past thi~ XT Contrastive emphatic particle to XH Exclusive emphatic particle hi XHC Clitic exclusive emphatic particle i, i~, hi~ XB Inclusive emphatic particle bhi ZZ Izafat Punctuation marks are tagged as themselves. 
307

Personal pronouns & academic writing: a multidisciplinary corpus-based critical pragmatic approach to EAP 
Nigel Harwood, Canterbury Christ Church University College 
Dept of Language Studies, Canterbury Christ Church University College, Canterbury CT1 1QU. 
Email nh19@cant.ac.uk Fax 01227 781 558 
Although EAP textbooks and style manuals tend to have little to say about the role of personal pronouns in academic discourse, I draw upon a multidisciplinary corpus of research articles to show pronouns are in fact multifunctional and worthy of attention. For instance, they can act as a discourse guide to the reader ('I will discuss x, then I go on to argue that y'); they can clarify terms or methodological assumptions ('For clarity, we use the term x'); they can recount the research process ('I collected data from x…Once themes began to emerge from the interviews, I validated them by y'); and they can help the writer underscore the innovation and novelty of their research ('No one has ever done x. In this paper I aim to do just that'; 'We have extended the ideas of Smith & Jones in this paper'). 
In addition, EAP course books give students the idea that academic discourse is homogeneous, that there is an easily defined model of 'good writing' that is equally valid for every discipline from Physics to Philosophy. Yet my corpus shows that different disciplines tend to use different pronouns for different reasons. In the light of these findings I will present EAP pedagogical materials taking what I call a multidisciplinary corpus-based critical pragmatic approach. The multidisciplinary, corpus-based elements of my materials give postgraduate learners information from corpus studies about these disciplinary preferences, showing, for instance, that in the case of single-authored papers, Physicists prefer we, while Business & Management authors prefer I. The pragmatic aspect of the materials comes from the fact that learners are being given Anglo-American norms of pronoun use which they can model their own writing on, while the critical aspect of the materials asks the class to discuss the potential sanctions of flouting, or indeed, conforming to, the dominant conventions. 
308

Building better corpora for summarisation 
Laura Hasler, Constantin Orasan and Ruslan Mitkov 
Research Group in Computational Linguistics 
School of Humanities, Languages and Social Sciences 
University of Wolverhampton 
{L.Hasler, C.Orasan, R.Mitkov}@wlv.ac.uk 
1. Introduction 
Annotated corpora have proved essential in many areas of NLP, and over the years have been successfully exploited for a wide range of tasks in Computational Linguistics, including part-of-speech tagging, parsing and information extraction. One field in which they have been particularly useful is automatic summarisation. Within this field, annotated corpora are mainly used for machine learning, to learn patterns for the extraction of important (and other) information from texts, as well as for the more complex task of evaluation of summarisation methods (Edmundson, 1969; Kupiec, Pederson and Chen, 1995; Zechner, 1996; Marcu, 1997). When annotating corpora, one (accurate) method is to employ humans to indicate those parts of text to be annotated with whatever information necessary. These human-selected units of text can then be used as a gold standard by which to measure the performance of a system, as well as for discerning which types of units are chosen or discarded by humans during the summarisation process. There are semi-automatic (Orasan, 2002) and automatic (Jing and McKeown, 1999; Marcu, 1999) ways to annotate corpora, but given that we are investigating new types of information to be marked, manual annotation is most appropriate here. Despite the fact that they are vital to the field, corpora annotated for summarisation are relatively sparse, and those resources which do exist do not contain as much information as they could. 
This paper presents an enhanced annotated corpus which differs from the majority of available resources in that it contains more information. In addition to containing information about the importance of the sentences, we also indicate parts which can be removed from sentences marked as essential or important, and provide a different label for those sentences which are not significant enough to be marked as important in their own right, but which have to be considered as they contain information essential for the understanding of the content of other sentences marked as essential / important. These last two types of information can give us an insight into the conciseness and coherence of summaries, respectively. Our corpus also contains annotations for linked sentences, where both sentences are considered important or essential, but one relies on the other to be completely understood. The building of the corpus is part of the “CAST – A Computer-Aided Summarisation Tool” project, which develops a tool to aid humans in creating summaries. 
The rest of the paper is structured as follows: Section 2 details previously annotated corpora for summarisation. The next section discusses our corpus. The annotation guidelines are described in Section 4. Section 5 comprises our results and discussion, followed by our conclusions and future work. 
2. Previous attempts at annotating corpora for summarisation 
Edmundson (1969) was the first to use corpora for summarisation. Human judges annotated the most important sentences from a corpus consisting of 200 documents in the areas of physics, life science, information science and humanities. Guidelines were used to ensure consistency, and the annotators were advised to minimise redundancy and maximise coherence. 
Kupiec, Pederson and Chen (1995) took a different approach in order to make the task of annotation slightly easier. Still using human annotators and providing guidelines for the annotation, sentences from summaries were aligned with sentences from the corresponding full texts. 79% of sentences appearing in the summaries could be matched with sentences in the full texts in their corpus of 188 scientific and technical documents. Teufel and Moens (1997) took a similar approach but achieved a much lower percentage of alignment (31.7%), using a corpus of 202 Computational Linguistics articles. This highlights the effect of text type, for both the alignment method and for annotation with regards to summarisation in general. 
Although work has been done using clauses or elementary discourse units (Marcu, 1997) as the standard unit of extraction, as well as allowing the human judge to determine the most appropriate 
309
units (Tsou, Lin and Lan, 1997), the corpora described above are all concerned with sentences, and sentences may not be the best textual units on which to solely base a summarisation system. We decided to select sentences due to the simplicity of defining the unit, but we also removed parts from these sentences so that only the most important information is present in the extract. 
3. The corpus 
The first step in building any corpus is to select the texts for inclusion in it. A description of these texts and the properties of the corpus we built can be found in Section 3.1. To encode the added information present in our corpus, a suitable annotation scheme was developed which accounts for all the types of information that it was necessary to include. This annotation scheme is presented in Section 3.2. Given the complexity of this scheme, the annotation cannot be applied using a simple text processor, but a specially developed tool has to be used instead. The tool is briefly described in Section 3.3. 
3.1 Texts used in the corpus 
One purpose of the CAST project is to develop summarisation methods for newswire texts. In order to tune and evaluate these methods, a corpus containing these types of texts had to be built. The texts included in our corpus were taken from the Reuters Corpus (Rose et. al., 2002). In addition to these, we also included a few popular science texts from the BNC (Burnard, 1995). We incorporated this latter type of text because it contains features from both the newswire and scientific genres, and the next step in our project is to extend our corpus to include scientific documents. By annotating these texts now, we hope to identify potential problems which could arise during the annotation of scientific documents later in our project. 
Newswire Popular science Total No. of texts selected 14716163No. of texts annotated by at least 2 annotators 311243No. of texts annotated by at least 3 annotators 7-7Total words annotated 11737828095145473Total sentences annotated 521413706584 
Table 1: Corpus statistics 
Table 1 summarises the statistics of our corpus. The texts with multiple annotations can be used to measure the interannotator agreement, and as a direct result of this, the difficulty of the task. The results of the comparison are presented in Section 5.3. 
For the annotation process we used four annotators; three graduate students and the first author of this paper. Three of the annotators were native English speakers, and the fourth had advanced knowledge of English. Before starting the annotation process, the annotation guidelines were explained to the annotators (see Section 4.2 for the guidelines). 
During the annotation, it became apparent that a selection of the texts were concerned with the same story, but they were written from a different perspective. We annotated some of these texts to give us an insight into whether the angle/order/authorship of a news text has any bearing on those sentences which are considered important. Some preliminary findings are presented in Section 5.7. 
The fact that the Reuters Corpus is freely available for research makes it possible for us to apply the same policy to the part of that corpus annotated by us. The newswire part of the corpus can be freely downloaded from the web page of the CAST project (http://clg.wlv.ac.uk/projects/CAST/). Given that the science texts are taken from the BNC it is not possible to distribute this part of the corpus. 
When we started developing the corpus, we wanted to produce a resource from which the whole research community could benefit. In light of this, we decided to use XML, an encoding widely used by researchers in Computational Linguistics. Given that an XML encoded file can be quite difficult to read and annotate, we had to use a special tool (see Section 3.3). 
3.2 Annotation scheme 
Section 1 states that our corpus is annotated with more information than the just the importance of the sentences. For this reason, it was not enough to indicate the importance of each sentence in a simplistic way (e.g. a character/symbol before each sentence); we had to develop a more complex annotation scheme. In this section we present the annotation scheme used in our corpus. 
310
We marked three types of information: the importance of the sentence (essential or important), links between sentences, and sections which can be removed from the marked sentences. The importance of the sentence was marked using the <EXTRACT ID="XXX"> tag, where the ID attribute identifies the tag uniquely. This tag has another attribute, IMP, which indicates the importance of the sentence and can take the following values: ESSENTIAL for sentences which are considered essential by the annotators, and IMPORTANT for those deemed important. In addition there is a third value for the IMP attribute, REFERRED, which indicates that a sentence is neither essential nor important, but required for the comprehension of a marked sentence. If a sentence is not marked in any way it is considered to be unimportant. The links between the sentences are marked by an empty tag <LINK REFERRED="XXX"/> which indicates that the EXTRACT tag which contains it is linked to the EXTRACT tag with the ID indicated by the attribute REFERRED. The sections from the selected sentences which are redundant or irrelevant are indicated by the <REMOVE> tag. All these tags have an optional attribute COMMENT, where the annotators can provide comments on the annotation process. 
Figure 1 presents a short extract from our corpus. 
3.3 Annotation tool 
In order to achieve the annotation described in the previous section, a multi-purpose annotation tool was used. In addition to helping with the marking process, the tool indicates throughout the annotation process what proportion of the file has been selected, and registers the time taken to annotate a file. 
The tool used to perform the annotation, PALinkA, offers a user-friendly graphical interface with different colours for the different types of information, which makes it easier to distinguish between them, as well as speeding up the annotation process. PALinkA is easy to use, even for non-computer experts. To mark a unit of text, the annotator uses the mouse to indicate the boundaries of the unit, the tag assigned to the unit, and whatever attributes are required by the tag. To avoid errors, some attributes such as unique IDs and references are determined automatically by the tool. 
PALinkA is a multi-purpose annotation tool which can be used to annotate a variety of phenomena. The set of tags which can be used in the annotation is specified by a preferences file loaded in the tool before annotation starts. For our project, a preferences file containing the annotation scheme described above was used.1 
4. Annotation guidelines 
Guidelines are essential for consistent and reliable annotation of texts. Annotation guidelines are a set of rules that indicate to the annotator which parts of a text should and should not marked, and for what types of information. They contain examples to aid the annotator in this process, and should be simple to follow but include enough detailed information to ensure strict adherence to them. Without these guidelines, the human annotator would have the freedom to indicate whatever parts of the text they wanted, regardless of their actual relevance to the task in hand. By developing guidelines to which annotators can adhere, we can reduce the amount of discrepancies between annotators who work on the 311 

<P ID="P3"><S ID="S3"><EXTRACT COMMENT="" ID="0" IMP="ESSENTIAL"><W ID="W19">Impl ied</W><W ID="W20">volatilities</W><W ID="W21">for</W><W ID="W22">U.S.</W><W ID=" W23">interest</W><W ID="W24">rate</W><W ID="W25">options</W><W ID="W26">were</W>< 
W ID="W27">steady</W><W ID="W28">to</W><W ID="W29">higher</W><W ID="W30">in</W><W ID="W31">quiet </W><W ID="W32">dealings</W><PUNCT>,</PUNCT><W ID="W33">with</W><W ID="W34">the</W><W ID="W35">underlying</W><W ID="W36">futures</W><W ID="W37">post ing</W><W ID="W38">mild</W><W ID="W39">declines</W><W ID="W40">on</W><W ID="W41"> the</W><W ID="W42">day</W><PUNCT>.</PUNCT></EXTRACT></S></P> <P ID="P4"><S ID="S4 "><EXTRACT COMMENT="" ID="1" IMP="ESSENTIAL"><W ID="W43">Volatilities</W><W ID="W 44">at</W><W ID="W45">the</W><W ID="W46">short</W><W ID="W47">end</W><W ID="W48"> of</W><W ID="W49">the</W><W ID="W50">yield</W><W ID="W51">curve</W><W ID="W52">we re</W><W ID="W53">bid</W><W ID="W54">for</W><W ID="W55">a</W><W ID="W56">second</ W><W ID="W57">day</W><PUNCT>,</PUNCT><W ID="W58">as</W><W ID="W59">the</W><W ID=" W60">market</W><W ID="W61">continues</W><W ID="W62">to</W><W ID="W63">adjust</W>< 
W ID="W64">for</W><W ID="W65">a</W><W ID="W66">higher</W><W ID="W67">interest</W> <W ID="W68">rate</W><W ID="W69">environment</W><W ID="W70">after</W><W ID="W71">a </W><W ID="W72">prolonged</W><W ID="W73">period</W><W ID="W74">of</W><W ID="W75"> stability</W><PUNCT>,</PUNCT><REMOVE COMMENT="" ID="6"><W ID="W76">Eurodollar</W> <W ID="W77">analysts</W><WID="W78">said</W><PUNCT>.</PUNCT></REMOVE></EXTRACT></S ></P> 

Figure 1: An extract from our corpus 
1 The tool and the preferences file can be downloaded from the project's web page. 

same text, as well as across a range of texts annotated by the same person, therefore ensuring consistent and valid annotation. In the field of summarisation these guidelines are especially important, given the notorious subjectivity of what is considered “important” enough to be included in a summary of a text. 
4.1 Other examples of guidelines 
Mitkov et al. (2000) suggest a “master” strategy for annotation of coreference and anaphora, which comes from a combination of different annotators’ approaches, and this can be adapted to summarisation as it is a fairly general strategy (changing those parts pertaining to the particular field) that could be extended to any area. These guidelines again seem to be based on general common sense, and provide a sound base for any type of annotation. These general guidelines, adapted for summarisation, are: 
- Prior to annotation, read the whole text to familiarise yourself with it and get a feel for what the text is “about” (one or maybe two main topics). 
- Ensure that the annotation is done in one intensive period (relatively easy in our case, as the texts are news articles and not too long), as sporadically annotating a file can lead to the annotator having to re-read the document for familiarisation several times. 
- Comment on troublesome cases of annotation and then discuss them with other annotators to decide upon the best solution to tackle them in future. 
Cremmins (1982) provides a number of pointers for writing abstracts. Although these are meant for human-written abstracts, they can also be valuable when we need a human annotator to mark those relevant sections in a text which contain information that could be used in a summary. Amongst other things, Cremmins advocates that the abstractor scan the whole text before deciding on important information (similar to the first “master” strategy point above), as well as comparing different versions of the same information to ensure that there is no over-emphasis of relevant information at the expense of other equally (or possibly more) relevant information through careless or inadequate reading of the text. As is evident from examining the guidelines below, this last point regarding non-repetition of information is central to our approach. 
Mitkov, Le Roux and Descles (1994) propose a set of rejection rules for knowledge-based automatic abstracting in the sub-language of elementary geometry. Some of these rules (such as elimination of text in brackets and text in quotation marks) may be appropriate for our corpus, but they need expanding on (see Section 4.2). Other rules they present which are similar to ours here concern the elimination of subordinate clauses, although we argue that not all subordinate clauses need to be removed, and text containing examples. 
4.2 Our guidelines for annotation 
In addition to the general strategies mentioned in Section 4.1, more specific guidelines were formulated based on an analysis of the type of texts we wanted to annotate for our corpus. As we work with news texts, the guidelines are slightly different to guidelines for texts in other areas, due to the features of newswire as a genre. It is important to remember that there may always be cases which prove to be the exception to the rule, which is why we provide room for comments and explanations as part of the annotation tool. A length restriction of 30% was imposed on the amount of sentences that could be marked; 15% to be marked as essential and 15% as important. The outline of our annotation guidelines, and the motivations behind them are described below. 
The annotators were instructed to identify the main topic of the text and mark sentences (using the <EXTRACT IMP=”ESSENTIAL”> tag in PALinkA) which gave essential information about the topic, keeping as close to 15% of the full text as possible. A newspaper text is usually “about” one main topic, that is, it tends to concentrate on one main focus throughout the length of the text. This means that the information which is considered suitable for inclusion in a summary (i.e. marked as essential / important; Section 5.1 elaborates more on the need for this distinction) should generally relate to this topic. 
Sub-headings generally summarise the text which follows and are subsequently useful to mark as important. Unlike titles, they are not automatically included in the list of extracted sentences, so they 
312
need to be marked explicitly. The annotators were instructed to mark sub-headings as long as they are relevant to the main topic of the text. 
Due to referential expressions there are sentences which rely on others for full understanding. An example of such a pair of sentences is: 
(S1) For film lovers the Festival's the place to be in September. 
(S2) It grows from strength to strength each year. 
In our annotation process, if S2 is marked, the sentence containing the referent for “it”, should be marked in some way. If S1 is important enough, it is already in the list of selected sentences, otherwise, it has to be highlighted with the <EXTRACT IMP=”REFERRED”> tag. The links between sentences need to be indicated by the <LINK> tag. 
Tables, figures and examples (including constructions starting with “e.g.”, “for example”, “such as”, “like”, “for instance” etc.) are not necessary in a summary, and therefore should not be marked, along with sentences concerning sub-topics unless they directly influence the main topic (or present new, essential information on it) and do not repeat information. Reported speech should not be included unless it presents vital and new information concerning the main topic of the text which is not presented elsewhere. Reported speech in news texts tends to provide opinions or statements to emphasise points already made in the article and does not usually warrant marking. However, it is important to distinguish between this speech and other text in quotation marks which may be important, for example: 
(S3) But its “killer app” is reinventing how the software industry works. 
Sentences which contain the same information as others which are marked should not be included, and each sentence should be considered carefully before selection. It is important to select the sentence with the most appropriate information, and not to include similar sentences as this will increase redundancy and take up valuable space. The most appropriate sentence is not necessarily the longest or most descriptive one, but that which most succinctly expresses the essential information. To compare two sentences similar in information content, we can look at the following pair, where (S4) is preferable (in most cases) to (S5): 
(S4) Inflow of export proceeds picking up: $300 million likely by February 15. 
(S5) The inflow of stuck-up export proceeds has picked up pace and at least $300 million are expected before the dead-line of February 15, say banking sources. 
Within these marked sentences, there may be parts which are not relevant to the overall importance. It is better to remove these parts of the sentences to minimise redundancy and maximise relevant information in the space available. Having marked the essential sentences in the text, the annotators were instructed to indicate segments (not single words) of these which were not vital to the understanding of the main topic. This was to be done using the <REMOVE> tag in PALinkA. So within sentences already marked as essential, irrelevant subordinate clauses should be marked as remove. The “which” clause could be removed from the following sentence if it is not important within the context: 
(S6) Customer interest is high for the whole product line, which underlines the strong fundamentals of the new period of growth. 
Text in brackets and text occurring between dashes (-…-) should be removed unless central to the main topic. For example, bracketed text would be considered important if it is an abbreviation which will replace a noun phrase later in the text, as in: 
(S7) A Poverty Reduction and Growth Facility (PRGF)… 
(S8) It is intended that PRGF-supported programmes… 
Adjuncts (not single words) which specify dates, times, places etc. should be marked for removal, unless they are vital to the main topic, likewise examples (see above) should be removed, as should phrases such as “in addition to…”, due to…” and “compared to…” which elaborate on information, and constructions like “a spokesman said”, “it was claimed”, “she told The Guardian” and “he explains”. 
Once the annotators had completed this selection process of essential and remove, they were advised that if the total amount of marked text (the <REMOVE> tag subtracts that part of the sentence marked 
313
as remove from the whole percentage of the marked text) is substantially below 15% of the full text, they should try to add more units which they considered essential to increase the percentage. Having completed the annotation for the essential classification, they had to repeat the process (using the same guidelines) for units of text they considered important, again keeping as close to 15% as possible. The annotators were also asked to comment on the annotation process noting any problems or indecisions, both as the annotation proceeded as well as more general comments at the end of the document. 
5. Results and discussion 
This section details the main findings resulting from an analysis of our corpus. Here we discuss the reason for maintaining a distinction between essential and important tags to mark sentences, correlations between the overall text length and the time taken to annotate that text, interannotator agreement for sentence classification, the distribution of marked sentences, a comparison of removed parts of sentences, sentences marked as linked or referred, the annotation of similar texts and an assessment of our annotation guidelines. 
5.1 Distinction between tags 
We felt it was important to distinguish between those sentences which were essential to the general understanding of a text, and those which were important. This distinction enables us to produce two types of summary – a shorter one (essential sentences only) and a longer one comprising essential and important sentences. It has been shown (Marcu, 1997) that human judges agree more consistently on those units in a text which they consider to be either very important or unimportant than when the same judges are asked to identify textual units which they consider to be important. As we keep this distinction in our annotation, we will also be able to test for ourselves whether this is the case in our corpus, and to pave the way for further investigation of the subjectivity of the notion of importance within summarisation. Marcu (1997) asks judges to assign a label to each unit in the text, whereas we are concerned with the selection of those units considered essential and important within a certain compression rate from a text. 
5.2 Text length / time 
During the annotation process, we recorded the time take to annotate the text, and measured the text length in words. Our hypothesis was that we would be able to identify a link between the length of the text and the time taken to annotate it. The time measurement was especially accurate as the annotator could stop the clock on the tool whenever they paused during the annotation. We computed the Pearson correlation between text length and time and found a strong correlation. For three out of the four annotators, the correlation was found to be significant at the 0.01 level. For the fourth annotator, the correlation was slightly lower, being significant at the 0.05 level. This correlation indicates that the texts annotated were of similar complexity, as the processing time depended on the length. 
5.3 Interannotator agreement for classifying the sentences 
As shown in Section 4.2, the annotators had to identify those sentences which contained information worth including in a summary. They were asked to label 15% of the whole text as essential, and another 15% as important. In order to assess the quality of the annotation we computed the interannotator agreement using the Kappa statistic (Siegel and Castellan, 1988). Kappa indicates whether there is any agreement among annotators, and takes into account not only the classification provided by the annotators, but also the possibility of their agreeing by chance. Kappa takes values between 0 and 1, and it is considered that a value over 0.8 indicates high agreement between annotators, whereas values between 0.68 and 0.8 indicate moderate agreement. Usually values below 0.68 are taken to indicate little or no agreement. 
Part of our corpus was annotated by two or three annotators, so it was possible to compute their agreement on these files. In order to be able to compute the Kappa statistic, we considered the annotation process as a 3-class taxonomy of essential, important and unimportant sentences.2 We computed the agreement for each text, as well as on the whole set of texts annotated by two or three annotators. The agreement between both two and three annotators was very low, and varies 
2 The annotators were not asked to identify the unimportant sentences, a sentence was considered unimportant if it was not marked in any way. 
314
considerably from one text to another, suggesting that there is little agreement among our annotators. The overall agreement for two annotators was 0.24, and for three annotators 0.23. 
We analysed the files in an attempt to discover the reason for the low agreement and noticed that sometimes the annotators focused on different sub-topics which they saw as relevant to the main topic. More importantly, we could see that a high proportion of the disagreement was because the annotators did not agree, in a large number of cases, as to whether a sentence was important or essential. The same sentences were marked as to be included, but with a different tag by different annotators. For this reason, we collapsed the important and essential classes into a single class and recomputed the Kappa statistic. In this case the value increased to 0.35 for texts marked by both two and three annotators, but the agreement is still very low. 
One possible reason for this low agreement is the fact that the annotators were not experts or professionals in summarisation, and so may not have been as accurate or confident in their annotations as they could have been otherwise. Another explanation is that some of the annotators found parts of the guidelines not explicit enough to ensure consistent annotation (see Section 5.8 for an assessment of the annotation guidelines). Section 6 discusses future avenues for investigating and improving these issues. 
Further investigation, which considered the extracted sentences as summaries, revealed that the annotators selected different sentences, but covered more or less the same information. In order to see how much the annotators agreed on the information (as opposed to the sentences) they selected, we computed the similarity between the sentences marked by the annotators. The similarity was computed using the cosine distance (Salton and McGill, 1983). The values for this distance indicate higher agreement between annotators, the average for essential and important sentences was 0.73. We also computed the similarity for essential sentences only, given that there are two possible extracts from each text. When we did this, the agreement decreased to 0.57. These figures prove that it may not be the sentences which are the most important element of the text to mark, but the actual information that one would want to include in a summary. The similarity measure relies on words, not on their senses. In future, we plan to improve its accuracy by taking into account synonymy relations between words. As a result of this change we expect an increase in the similarity. 
5.4 Distribution of marked sentences in texts 
It is a view widely held that the important sentences in news texts are located at the beginning of the text. An analysis of the distribution of marked sentences in our corpus proved this not to be the case. We divided the text into three equal parts (beginning third, middle third and end third) to see where the majority of sentences were chosen from. In the whole corpus, the highest proportion of sentences selected came from the first two parts of the text, with a slightly higher proportion coming from the middle third (38.7%) than the beginning third (35.4%). A lower proportion of sentences were marked in the end third of the texts (25.9%), and although this was not as high as the other two thirds, it was still high considering the general claim that such sentences can usually be found at the start of a text. Although the distribution of our annotators’ sentences does not match with the generally held view, they do match up with each other. The texts marked by all four annotators strongly displayed these patterns, despite the fact that annotation was carried out independently. 
5.5 Comparison of removed sentence segments 
Given the low agreement between the sentences marked for extraction and the fact that we did not provide a strict definition for the units to be removed, we could not apply a similarity measure like the one used in Section 5.3. Instead, we tried to identify recurring patterns manually. An analysis of the removed sentence segments demonstrates definite patterns with regards to the types of constructions that are likely to contain irrelevant information within important sentences and from which we can learn what kind of information we would not want to include in a summary. The types of information removed fall into several well-defined categories, each of which contain certain types of constructions. Many of the removed segments serve to qualify previous information given in a text. For this reason, we chose not to include qualifiers in general as a category as this would subsume other important groups. 
The first category of removed sentence segments is that containing information referring to time and location. This information is characterised by constructions such as prepositional phrases (“in…”, “at…”, “on…”, “by…”) and adverbial phrases (“during…”) acting as temporal adjuncts. Phrases introduced by “elsewhere” and “meanwhile” also appeared with this kind of information. The second 
315
category contains removed information concerning the coverage of events via reported speech. Whilst the actual speech or the gist of what was said may be important, who reported this was not usually important. Verbs of reporting, such as “said”, “writes”, “revealed”, “reported”, “pointed out”, “alleged”, “added that”, “according to” either preceded or followed by a nominal group (generally person, organisation, type of report) are characteristic of this information. 
General information irrelevant to the main topic of the text was mainly contained in subordinate clauses. This provided us with another group of removed sentence segments. These clauses contained all kinds of different information, but were generally exemplified by starting with “where”, “when”, “with”, “as”, “after”, “due to”. Interestingly, there were a number of co-ordinate clauses starting with “and” and “but” which were also removed. This occurred when the second co-ordinate clause elaborated on information in the first and was therefore not needed as it increased redundancy, or when this clause was not pertinent to the main topic of the text. 
Relative clauses which qualified nominal phrases made up a fourth category. The main types of relative clause began with “which”, “who” and “that”, as well as the verb “including”, and served to give further information about an entity in the text, not needed in a summary. Examples typically started with “usually”, “such as”, “like”, “especially”, and gave us a fifth group of removed text. Another group was text in between or following dashes, which again qualified preceding information or entities. The seventh and final category of removed segments of sentences comprised other ways of elaborating textual units, such as apposition of nominal phrases, and the use of the prepositions “to”, “from”, “by” and “for” to give more information about a change that had occurred. Words introducing counter-information like “though”, “although”, “even though”, “versus”, “instead” started a number of the segments. Constructions following the “in a bid + infinitive” structure were also removed quite frequently. 
5.6 Linked and referred sentences 
As explained in Section 4.2, the reasoning behind marking linked and referred sentences in our corpus is that they will give us an insight into how we can improve the coherence of summaries by examining the types of links that occur between sentences. 
Preliminary results show that the number of sentences which need to be extracted only because they contain entities referred to in other sentences is very low. In our corpus, we identify 42 such sentences, 0.64% of the total number of sentences. This suggests that in general the set of important sentences contains enough information to be understandable on its own. One explanation for this low number could have been that there were no links between sentences. After counting the number of links marked between tagged sentences, such an explanation was quickly ruled out as we found that over 18% of the selected sentences contained links to one or more other sentences. As expected, most of the links were needed because of pronouns and noun phrases, a much lower number were due to anaphoric verbs. Given that the annotators were not asked to mark the relations explicitly, it is not possible to obtain these statistics automatically. In the future, we intend to detail these relations to improve the investigation. 
5.7 Annotation of similar texts 
As mentioned in Section 3.1, some of the texts in our corpus were based on the same sequence of events, but were slightly different versions of these events. One annotator marked important sentences in 5 versions of one text, 4 of another, and 2 of a third text in order to assess the impact of different versions of the same story on marking important information. To do this, we computed the similarity between the different versions of the texts using the cosine distance, as we did for the other texts (see Section 5.3). 
Preliminary results show that when the same annotator is instructed to mark the important information in different versions of the same texts, the similarity between them is not very high. When this is compared to the average similarity for the same text annotated by different people, we can see that it is considerably less: 0.44 compared to 0.73. This shows that differences in texts primarily “about” the same thing have a considerable effect on the information that is considered important. This is due to the different locations of certain information within the texts, how much space is devoted to that information, and what subsidiary information is introduced to flesh out the main points of the story. The title is a good indicator as to the main focus of each text, including both the structure and the content. 
316
5.8 Assessment of guidelines 
A discussion with the annotators generated several points with regards to the suitability of the guidelines developed in order to help them in their annotation. As we can see, although the guidelines seemed generally helpful and the comments positive, we recognise that there are ways in which they could be improved to ensure more consistent annotation in the future. 
In general, the annotation guidelines were helpful for the task. They are useful in the way that they indicate what should and should not be marked, and give appropriate examples to illustrate instructions. They cover most of the different situations and alternatives that may be faced when annotating and are concise and clear. They are strict with regards to the amount of sentences which could be extracted, however, this may have been slightly too long considering a summary was meant to be made from it. According to the guidelines for summarisation the length restriction is 15% essential sentences and 15% important sentences. In some cases, it was difficult to adhere to the 15% restriction as it was not easy to distinguish between the important and essential sentences in a text, and more than 15% could have been considered essential or important. 
An analysis of the corpus showed that there were many exceptions to the rules stated by the guidelines; this had, however, been pre-empted and was the reason for the inclusion of “unless vital to the main topic” on the end of many of the instructions. Dates and times were important in some contexts, such as articles where time was an important factor, as was reported speech, and sometimes even the speaker, especially where there were conflicting arguments from two different organisations or people, and examples were occasionally necessary to emphasise an important point. Sub-headings were not very relevant when they did occur, which was much more rarely than predicted, as they tended to be subjective “asides” on a particular situation and made no sense without the whole of the original text. 
One obvious problem was a general misunderstanding of the term “figures”, from the instruction not to include tables and figures. Some of the annotators took this to mean numerical figures, whereas the meaning in the guidelines was illustrations such as graphs. In future, instructions such as this should be explained more clearly. 
6. Conclusions and future work 
In this paper, we discussed issues involved in building a corpus for summarisation. We believe that our corpus is better than existing corpora for summarisation because it includes features which are not accounted for in currently available resources. In addition to the possibility of computing the performance of summarisation methods in terms of the information extracted, our corpus can be used to account for phenomena such as coherence. The former is obtained by comparing the information extracted by an automatic method against the information selected by a human annotator as important. The latter can be computed by identifying sentences in the automatic extract which were marked by the human annotators as being linked to one or several sentences. As well as this, the corpus can be exploited in little investigated fields such as the identification of sub-sentential units for removal in order to produce concise summaries. 
So that we could assess the interannotator agreement, some texts were annotated by more than one annotator. The results of this comparison showed that there is little agreement between the sentences selected for extraction, but this is similar to the findings of other researchers (Lin and Hovy, 2002). When comparing the information content of the extracts, we noticed that the agreement was much higher, which proves that it may not be sentences which are the most important elements of the text to mark, but the actual information that one considers important. 
The guidelines provided for the annotators also have an influence on the interannotator agreement. A discussion of the guidelines with the annotators revealed certain directions for future improvements. Many of the annotators’ comments were concerned with the fact that they found it difficult to identify the main topics in fields they were not familiar with. In the future, we are considering employing professional summarisers, because they have more experience in dealing with unfamiliar topics. 
Acknowledgements 
This paper was written as part of an AHRB-funded project, “CAST – A Computer-Aided Summarisation Tool”. 
317
References 
Burnard, L. 1995 Users Reference Guide: British National Corpus Version 1.0. Oxford: Oxford University Computing Services. 
Cremmins, E.T. 1982 The Art of Abstracting. Philadelphia: ISI Press. 
Edmundson, H.P. 1969 New methods in automatic abstracting. Journal of the Association for Computing Machinery, 16 (2): 264-285. 
Jing, H. and McKeown, K. 1999 The Decomposition of Human-Written Summary Sentences. In Proceedings of the 22nd International Conference on Research and Development in Information Retrieval (SIGIR'99), Berkeley, pp129-136. 
Kupiec, J., Pederson, J. and Chen, F. 1995 A trainable document summarizer. In Proceedings of the 18th ACM/SIGIR Annual Conference on Research and Development in Information Retrieval, Seattle, pp 68-73. 
Lin, C-Y. and Hovy, E. 2002 Manual and Automatic Evaluation of Summaries. In Proceedings of the ACL 2002 Workshop on Text Summarization, Pennsylvania, pp 45-51. 
Marcu, D. 1997 From discourse structures to text summaries. In Proceedings of the ACL/EACL '97 Workshop on Intelligent Scalable Text Summarization, Madrid, pp 82-88. 
Marcu, D. 1999 The automatic construction of large-scale corpora for summarization research. In Proceedings of the 22nd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR'99), Berkeley, pp 137-144. 
Mitkov, R., Le Roux, D. and Descles, J.P. 1994 Knowledge-based automatic abstracting: Experiments in the sublanguage of elementary geometry. In Martin-Vide, C. (ed.) Mathematical Linguistics. The Netherlands: North-Holland, pp 415-421. 
Mitkov, R., Evans, R., Orasan, C., Barbu, C., Jones, L. and Sotirova, V. 2000 Coreference and anaphora: developing annotating tools, annotated resources and annotation strategies. In Proceedings of the Discourse Anaphora and Anaphora Resolution Colloquium (DAARC'2000), Lancaster, pp 49-58. 
Orasan, C. 2002 Building annotated resources for automatic text summarisation. In Proceedings of the Third International Conference on Language Resources and Evaluation (LREC-2002), Las Palmas de Gran Canaria, pp 1780-1786. 
Rose, T.G., Stevenson, M. and Whitehead, M. 2002 The Reuters Corpus Volume 1 - from Yesterday's News to Tomorrow's Language Resources. In Proceedings of the Third International Conference on Language Resources and Evaluation (LREC-2002), Las Palmas de Gran Canaria, pp 827-833. 
Salton, G. and McGill, M.J. 1983 Introduction to Modern Information Retrieval. New York: McGraw-Hill. 
Siegel, S. and Castellan, N.J. 1988 Nonparametric Statistics for the Behavioral Sciences. 2nd Edition New York: McGraw-Hill. 
Teufel, S. and Moens, M. 1997 Sentence extraction as a classification task. In Proceedings of the ACL'97/EACL'97 Workshop on Intelligent Scallable Text Summarization, Madrid, pp 58-59. 
Tsou, B.K., Lin, H-L. and Lan, T.B.Y. 1997 A Comparative Study of Human Efforts in Textual Summarization. In Proceedings of the 3rd Pacific Association for Computational Linguistics Conference (PACLING '97), Tokyo. 
Zechner, K. 1996 Fast Generation of Abstracts from General Domain Text Corpora by Extracting Relevant Sentences. In Proceedings of COLING - 96, The International Conference on Computational Linguistics, Copenhagen, pp 986-989. 
318

Not KWIC but Quick: KeyWords in Court 
Dr Chris Heffer 
Department of English and Media Studies 
The Nottingham Trent University 
Burton Street 
Nottingham NG1 4BU 
chris.heffer@ntu.ac.uk 
Tel. 0115 848 3399 
As represented by the KeyWords tool in Mike Scott's WordSmith suite of corpus tools, a ‘keyword’ is a word which occurs in a text or corpus with a frequency which is unusually high in comparison with a norm indicated by a reference corpus such as the BNC. This paper argues that keyword analysis, which has been used for stylistic analysis, can also tell us a great deal about courtroom discourse. Drawing on a set of corpora of Crown Court transcripts compiled by the author, the paper shows how keyword analysis can reveal: differences in barrister strategy between examination-in-chief and cross-examination; the distribution of lexical judgement through jury trial; and modal preferences in the various parts of the trial. While keyword analysis is not a replacement for careful study of KWIC concordances, it can provide a clear and quick first impression of a particular genre or style, particularly in a genre complex such as trial by jury. 
319

A corpus-based analysis of word order variation: 
The order of verb arguments in the German middle field 
Kris Heylen1 and Dirk Speelman 
University of Leuven 
Abstract 
A perennial problem in German syntax is the order of verb arguments in the Mittelfeld. The Mittelfeld is the section of the clause between the two parts of the discontinuous verbal group. In it, all verb arguments can be realized simultaneously, however, not always in the same order. There is a long-standing debate about a number of factors that possibly govern this variation, yet, what their actual influence is, remains unclear. This study takes a quantitative corpus-based approach to the problem and looks specifically at a type of variation that has scarcely been dealt with up to now, viz. the relative order of a pronominal object and a nominal subject in the Mittelfeld. Clauses that show this kind of variation have been collected from the NEGRA-corpus consisting of German newspaper material and they have been annotated for 5 factors frequently mentioned in the literature: grammatical function of the arguments, given/new status and animacy of the arguments' referents, difference in length between the arguments, and their occurrence in either a main or a subordinate clause. The effect of these factors has been statistically checked and modelled in a logistic regression model. The results of the statistical analysis show an effect of all factors except for grammatical. The effect of main versus subordinate clause is especially strong, contradicting earlier hypotheses that this factor is only an epiphenomenon of length difference 
1 Introduction 
German clause topology is characterized by the so-called Klammer construction. Within the clause, verbal elements have two fixed positions that seem to hold the clause together like two braces (German: Klammer). Using this representation scheme, other constituents can be positioned relative to the two Klammers: the site before the first Klammer is called Vorfeld (front field), the one between the two klammers is referred to as the Mittelfeld (middle field), and the position after the second Klammer is known as the Nachfeld (back field). Table 1 shows that in the main clause, the first Klammer is occupied by a finite verb, in this case hat, whereas the second Klammer is filled by infinite verbal elements, here the past participle geschenkt. The Vorfeld has room for one constituent (Gestern) and the remaining constituents normally occupy the Mittelfeld. Very long constituents or, like in this case, subordinate clauses are placed in the Nachfeld. The subordinate clause has a somewhat different structure because all of the verbal elements, both finite and infinite, now occupy the second Klammer, whereas the first Klammer is filled by the complementizer (dass) or is left empty. The front field is empty too so that all of the clause's constituents land in the Mittelfeld. 
Vorfeld 1st Klammer Mittelfeld Klammer Nachfeld main clause Gestern Hat der Vater dem Sohn einen Ball geschenkt, weil er Geburtstag hatte Yesterday, the father has given the son a ball because it was his birthday subordinate clause (Er sagte,) dass der Vater dem Sohn gestern einen Ball geschenkt hat, weil er Geburtstag hatte (He said) that the father has given the son a ball yesterday because it was his birthday Table 1 
The order of constituents in the Mittelfeld is not fixed. Especially the order of the verb arguments is subject to considerable variation. Nominative subject, dative object and accusative object can occur simultaneously in the Mittelfeld. Although the default order is the one shown in Table 1 with 
1 Kris Heylen is Research Assistant of the Fund for Scientific Research - Flanders 
320
2nd 
nominative < dative < accusative, other sequences are possible as well. The question which principles determine this word order has led to an animated debate within the German linguistics community, starting with Lenerz's (1977) seminal study and reaching a climax in the mid eighties (see Reis 1987 for an overview). In this discussion functionally oriented linguists opposed advocates of an autonomous syntax. The former stressed the importance of pragmatic factors, whereas the latter insisted on purely grammatical principles. As both camps kept on coming up with examples that contradicted previous examples from the opposite side, the debate was never settled. Since then, a number of attempts have been made to examine the variation more empirically. On the one hand there have been psycholinguistic studies looking at the influence on processing time of reordering constituents with different grammatical functions (Pechmann et al. 1996, Poncin 2001). On the other hand there have been small-scale corpus studies that following Hawkins (1994) have looked at syntactic complexity, i.e. length, as a possible fundamental principle regulating word order (Primus 1994, Kurz 2000). Yet, despite of all this research the factors determining the word order variation in the Mittelfeld remain unclear. 
The most important reason for this lack of insight is certainly the variation's complexity. So many different factors are involved that Reis (1987: 139) speaks about a "jungle of factors" and Eisenberg (1994: 419) refers to a "welter of criteria". However, the research up to now has had shortcomings of its own that prevented an adequate description of this complexity. First of all, the different studies used disparate methods like grammaticality judgements, psycholinguistic experiments or corpus studies, which make the results very difficult to compare. Apart from that, the different methods had weaknesses of their own. The bulk of the studies from the eighties but also some more recent ones use introspection and grammaticality judgements of constructed examples as research data. Apart from the precariousness of using constructed examples, it is also problematic that the factors rarely have a categorical effect that leads to clear ungrammaticality. This often makes the assessment of a factor's effect vere subjective. The psycholinguistic experiments on the other hand can rely on clear factual evidence. However, their measurements in terms of latencies cannot always be easily linked to the different linguistic variables that play a role in the complex variation. This caused Pechmann et al. (1996) and Poncin (2001) to restrict themselves to only one factor, which they could control and gather enough data for. Finally, the corpus studies carried out by Primus (1994) and Kunz (2000) did investigate 2 or 3 factors but their datasets were rather small and most of all, they lacked the statistic apparatus to analyse multivariate data. The study presented here tries to learn from the weaknesses of previous studies by opting for a quantititave corpus study, based on the statistical analysis of actual usage data. 
The current study will focus on a specific type of variation in the Mittelfeld that up to now has been scarcely taken into account. When the Mittelfeld of a clause contains both a subject realized as a full NP and an object realized as a personal pronoun, the pronominal object most frequently precedes the nominal subject. In a substantial number of cases however, the pronominal object follows the nominal subject. Moreover, the two orders seem freely interchangeable without any obvious meaning difference between them. In the literature pronominality is often mentioned without any further consideration as a factor that stimulates pre-posing of constituents. Somewhat more in-depth studies are Lenerz (1994) who looks at this type of variation from a Government & Binding perspective, only to find that this must be some form of scrambling, and Shannon (2000), who uses corpus data, but without further analysis for the synchronic variation, since he is mainly interested in the diachronic analysis. In one respect, the word order variation of nominal subjects and pronominal objects is even harder to get at because it does not show the differences of meaning or grammaticality, that can still be observed up to a certain degree with full-NP subjects and objects. That is probably the reason why this type of variation is scarcely studied and often dismissed as free variation. Yet in another respect, controlling for pronominality and restricting the variation to pronominal objects and nominal subjects reduces the complexity of the problem considerably. Pronouns vary less in length and given/new status, two factors that have been central in the debate. This means we can concentrate on the subject as the most prominent verb argument. The purpose of this study is therefore twofold: On the one hand it tries to gain some insight into a less studied type of variation, but on the other hand it uses this type of variation as a starting point to study the word order variation in the Mittelfeld in general. By initially reducing the complexity and making the problem more amenable to examination, we hope to get an easier grip on the more complex types of variation afterwards. 
In practice, this study will look at 5 factors that are frequently mentioned as having an influence on the order of verb arguments in the Mittelfeld: the grammatical function of the arguments, their length, their occurrence in a main versus a subordinate clause (clause type), their given/new status, and the animacy of their referents. We will examine what the actual effect of these factors on the word order is; what 
321
their combined effect is; whether there is any interaction between factors; whether some factors are more important than others; and finally, how much of the variation is explained by these factors. 
We will start off with a short description of the corpus material. Next, we discuss the individual effect of the factors and some two-factor interactions. Then, we use a logistic regression model to look at the factors' combined effect. Finally, we conclude with a round-up of the results and give some suggestions for future research. 
2 Corpus material 
In the choice of a corpus two issues play an important role: the representativeness of the corpus and its accessibility. Representativeness relates both to what kind of language use has been included in the corpus and to the number of instances of the construction under investigation. That number should be large enough to base any reliable conclusions on. Accessibility on the other hand does not only pertain to the obvious fact that (machine-readable) corpora should be made available to the academic community. It also relates to the corpus's mark-up and annotation. An extensive annotation makes it easier to trace more complex syntactic structures. After comparing a number of electronic German corpora, the NEGRA-corpus seemed to offer the best compromise for our study between these two concerns. The NEGRA-corpus2 was compiled at the university of Saarbrücken (Germany) and is made up of newspaper material from a local newspaper called Frankfurter Rundschau, year 1991-1992. This means that, strictly speaking, any conclusions we reach can only be valid for this type of language use. Yet, newspapers do function actively in a language community so that they might be considered appropriate to base a first study on. The NEGRA-corpus contains some 355,015 tokens or 20,602 sentences. That is not an awful lot, but the construction we are looking at will appear to be common enough to yield a sufficient number of instances for analysis. The main advantage of the NEGRA-corpus over the other corpora is its annotation. It has been annotated and manually checked for Part-of-speech values and syntactic structure. This allows for an almost automatic detection of the Klammer construction, which is central to German word order description. 
A first step in preparing the corpus material for further study, was to extract the observations containing the relevant construction. To do so, PERL-scripts were written that exploited the NEGRA corpus's morpho-syntactic annotation. Relevant observations were defined as all clauses that contain a nominal subject and a pronominal object in their Mittelfeld. A nominal subject had to be either a full NP, containing a common or a proper noun, or a demonstrative or indefinite, which are known from the literature (Zifonun 1997: 1510) to have the same topological behaviour as nouns. Pronominal objects had to be either accusative or dative objects. Clauses with both an accusative and dative object were so rare that they could not be taken into account for statistic analysis. Furthermore, pronominal objects had to be realized as personal pronouns, including reflexives. As just said, demonstrative and indefinite pronouns have a noun-like topological behaviour. The output of the scripts was manually checked for spurious hits, but they were rare (4%). Part of the corpus was controlled manually to see whether the scripts had missed relevant observations, but that was not the case. In the end, 995 observations were retained, which means that almost 5% of all sentences in the corpus exhibited the construction under investigation. 
In a second step, we annotated each of the observations by recording how each of the 5 factors was realized for that observation. An annotation scheme was set up with for each factor the possible values it could take. The actual annotation was done semi-automatically using the Abundantia Verborum annotation tool (Speelman 1997). The annotation procedure took the form of an annotation loop: When an observation did not fit into the initial annotation scheme, the scheme was revised to accommodate for that observation. Then, all previously annotated observations were checked to see whether they were still in compliance with the new annotation scheme. This reiteration went on until all observations were annotated adequately. Finally, the outcome of the annotation was turned into a data matrix with one line per observation stating the values observed for each factor. 
3 Statistical analysis of individual factors 
The ultimate purpose of this study is to determine the influence of the 5 factors, individually and combined, on the word order variation of pronominal objects and nominal subjects in the Mittelfeld. This is done through statistical analysis. Statistical analysis offers two main advantages: it generalizes over a lot of examples and it tells you how sure you can be of your results. Generalizing over examples 
2 More information about the corpus is available under http://www.coli.uni-sb.de/sfb378/negra-corpus/ 
322
is important to detect effects that are not apparent at first sight. In the case of word order variation in the Mittelfeld, none of the factors has a categorical effect leading to a clear difference in meaning or grammaticality. The influence these factors have is a matter of tendency, which is very hard to get at with the traditional heuristic methods like grammaticality judgements used in many of the earlier studies. By counting and generalizing over instances, these tendencies do become clear. The other advantage of proper statistical analysis is that it can verify the significance of the results obtained from a sample for the phenomenon in general. In this study, we will check whether the factor influences we observe in the corpus can safely be generalized for this type of language use. 
The factors we study are so-called categorical variables. Their values are not measured on a continuous numerical scale but are discrete categories. Categorical data analysis requires specific statistical techniques. The statistics3 used here can be split up in three categories. In this section we will first carry out a univariate analysis, looking at one isolated variable's distribution. This will be the variation in word order itself, the variable whose behaviour we are ultimately interested in, also called the response variable. Confidence interval will be calculated for its distribution. The remainder of this section is reserved for bi- and trivariate analysis. Here we investigate each time the relation between the response variable and one or two of the factors that we suspect have an influence on the response variable. They are referred to as the explanatory variables. Whether there is association between an explanatory variable and the response variable will be tested with the Pearson chi-square statistic. How strong that association is will be expressed by the odds ratio. For the ordinal variables length and given/new status, we will test for linear association using the Mantel-Haenszel chi-square statistic. Strength and direction of this linear association is conveyed by the gamma coefficient. In section 4 then, we will look at a Multivariate analysis. There we will try to examine what the combined effect of the explanatory variables is on the response variable, using a logistic regression model4. 
3.1 Response variable: word order 
Let us first look at the response variable, viz. the variation in word order of nominal subjects and pronominal objects in the Mittelfeld of the German clause. The response variable can take two values: either the pronominal object precedes the nominal subject (object first as in1) or the pronominal object follows the nominal subject (subject first as in 2). 
(1) Ein paar Tage später nahm <ihn (OBJ)> <der SED-Chef der Uni (SUBJ)> beiseite (NEGRA, 1618)5 
A few days later the university's SED-chief took him aside 
(2) Später , als <die Kommission (SUBJ)> <ihn (OBJ> entlassen hat , sagt er ,.... (NEGRA, 1665) 
Later, when the commission has dismissed him, he says.... 
If we look at the distribution between object first and subject first as presented in Table 2, we see that object first is much more common. This confirms the status of object first as the default word order, which is often mentioned in the literature. To check how representative this distribution in the NEGRA-corpus is for newspaper language in general, we can look at the 95% confidence interval. For the probability of having object first this interval lies between 87.1% and 91.0%6. The confidence interval tells us with 95% certainty that the share of object first cases in this type of newspaper language in general will lie between 87.1% and 91.0%, which again confirms object first as the default order. 
Object first Subject first 889/995 
(89,0%) 106/995 
(11,0%) Table 2 
Yet, the distribution is much more askew then previous exploratory studies suggested. Hoberg's (1981) descriptive figures for her mixed corpus show a ratio of 75% to 25% based on 358 observations. A chi-square test on a cross tabulation of both datasets shows that this difference is very significant (p<0.01). Shannon's (2000) study of literary texts has a 62% to 38% ratio for object first versus subject first based on 786 observations and that difference is very significant too(p<0.01). Moreover, the difference between Shannon's and Hoberg's corpus distribution is significant as well (p<00.1). It is not immediately clear what causes these differences in distribution. However, it is notable that Shannon's 
3 For an overview of the statistics for categorical data analysis used here, see Agresti (1996) 
4 For the analyses, the statistical software pack, SAS version 8 was used. 
5 Examples from the NEGRA corpus are referred to by "NEGRA" followed by the number of the sentence they appear in. 
6 For the calculation of the confidence interval for extreme proportions, see Agresti (1996: 11) 
323
corpus mainly contains literary material, Hoberg's is a mix of all kind of written material, and ours is newspaper material. Therefore, these differences might be due to an influence of text sort. 
3.2 Grammatical function 
Grammatical function refers to the case role assigned to an argument, which is (mostly) morphologically marked in German. Because case has always been a central feature of syntactic descriptions, it is probably the oldest factor to be acknowledged as having an influence on the order of verb arguments in the Mittelfeld. Indeed, it is the factor used in grammars to tell the verb arguments apart and to assign ordering preferences to them. In this study, we use case as a basic descriptor as well. We defined the relevant construction as the one having a nominal, nominative subject and a pronominal accusative or dative object. Because the presence of a nominative subject is held constant, the values of the factor grammatical function pertain to the case of the pronominal object: this is either an accusative object or a dative object. Looking at the contingency table in Table 3, it is clear that accusative objects are more frequent than dative objects. This is not a surprise as there are many more transitive verbs than ditransitive ones. What is surprising though, is that the case of the object does not seem to have any influence at all on the distribution of the word order (chi-square, p=0.94). For both dative and accusative case object first has a proportion of about 89%. Although the literature (Lenerz 1977, Zifonun 1997) suggests for nominal objects, that the dative object has a greater tendency to precede the subject than the accusative object, this is not borne out for pronominal objects here. In general, grammatical function is seen as the most important factor of all for word order (Reis 1987, Primus 1994). Again, this does not seem to hold for the order of nominal subjects and pronominal objects. 
Object first Subject first /total Accusative 724/810 
(89.4%) 86/810 
(10.6%) 810/995 
(81,4%) Dative 165/185 
(89.2%) 20/185 
(10.8%) 185/995 
(18,6%) Table 3 
3.3 Length 
Length refers to the difference in length between the pronominal object and the nominal subject as measured in syllables7. Difference of length was already mentioned by Behaghel (1932) as his "law of growing constituents" saying that longer constituents follow shorter ones. More recently, Hawkins (1994) stated in his EIC-theory that length difference is the main principle governing word order universally, and also in German. More specifically the theory8 states that word order is determined by the attempt to minimize the time a listener needs to determine how many constituents a clause has. This can be done by putting longer constituents further behind in the clause A first word of a constituent, e.g. a preposition, is often enough to determine the nature of constituent (e.g. a prepositional phrase). If shorter constituents were to follow longer ones, the listener would have to go through all the words of that long constituent before he or she could determine the presence of the shorter constituent at the end. By putting the shorter ones first, a listener already knows about all the other constituents by the time he reaches the first word of the last constituent and from that word alone he can now determine the total number of constituents without having to go through all the words of the long constituent. 
In Table 4 the length difference has been split up into 6 categories starting from no length difference and going up to larger differences in number of syllables. As pronouns in German are only 1 or 2 syllables long, increasing length difference is caused by longer subject NP's. Table 4 bears out the expected influence of length difference: The larger the difference between the pronominal object and the nominal subject gets, the more often the object precedes the subject. Statistically we can check this directional relation using the Mantel-Haenszel chi-square test, which confirms that there is linear association (p<0.01). The strength and direction of this association is given by the gamma coefficient, which has a scale from –1 to 1 with –1 being perfect negative association, 0 no association and 1 perfect positive association. Here a gamma coefficient of –0.4 (ASE=0.065) indicates moderate negative association, which means that higher length differences tend to correspond relatively more with object first and smaller length differences tend to correspond relatively more with subject first. 
7 Number of syllables was chosen over number of words as a measure because German has very long compound nouns, to which a number of words measure is not sensitive. 
8 The theory is presented somewhat simplified here. For an in-depth treatment see Hawkins (1994). 
324
Object first Subject first 0 105/137 (76.6%) 32/137 (23.4%) 1-3 194/225 (86.2%) 31/225 (13.8%) 4-6 212/233 (91.0%) 21/233 (9.0%) 7-10 153/163 (93.9%) 10/163 (6.1%) 11-20 155/164 (94.5%) 9/164 (5.5%) 20-65 70/73 (95.9%) 3/73 (4.1%) Table 4 
3.4 Clause type: main versus subordinate clause 
The factor of clause type, i.e. the occurence of the construction in a main versus a subordinate clause, has also been put forward in the context of Hawkins' (1994) EIC-theory explained above. According to Hawkins, the topological characteristics of German subordinate clauses diminishes the effect length difference has on word order. As explained in the introduction, the position of the finite verb in German is different in main clauses from subordinate clauses. In main clauses the finite verb is in second position, whereas in subordinate clauses, the finite verb is always near the end of the clause. In Hawkins's terms, this means a listener in subordinate clauses always has to wait until the end to be able to ascertain the total number of constituents, no matter whether the speaker puts longer constituents before shorter ones or not. That is why, according to Hawkins, the speaker will put relatively more shorter constituents before longer ones in main clauses than in subordinate clauses. This results in more object first cases in main clauses than in subordinate ones. Note that in this hypothesis the effect of clause type is subsidiary to length: clause type has only an influence because of length's initial effect. 
Object first Subject first main clause 646/674 
(95.9%) 28/674 
(4.1%) subordinate clause 243/321 
(75.7%) 78/321 
(24.3%) Table 5 
If we look at the results in Table 5, we can indeed see that object first is significantly more common in main clauses than in subordinate clauses (chi-square p<0.01). An odds ratio of 7.4 (with confidence interval 4.7 to 11.7) tells us that the odds of object first versus subject first are 7.4 times higher in main clauses than in subordinate clauses, which indicates a very strong association. Considering the fact that the pronominal objects are generally shorter than the subjects, this domination of object first in main clauses would seem to confirm Hawkins' hypothesis. If we perform the analysis for length we did in the previous section again, but now separately for main clauses and subordinate clauses, we see that length has a significant influence in both clause types (chi-square p<0.01), but that, as Hawkins predicts, the effect is much stronger in main clauses (gamma = -0.57) than in subordinate clauses (gamma = -0.37). Yet, there are reasons to doubt Hawkins' assumption that clause type has only a subsidiary influence on the word order, i.e. dependent on length. For length we only measured a moderate association with word order (gamma = -0.4), whereas for clause type the association with word order was very strong (odds ratio of 7.4). It would be counterintuitive for a subsidiary effect to be stronger than the one it depends on. One can argue that the two measures of association are difficult to compare, but there is more. Table 6 shows the cross tabulation of word order and clause type for observations where there is no difference in length between the object and the subject. Here too, there is a strong effect of object first being more common in main clauses than in subordinate clauses (chi square p<0.01, odds ratio = 4.6 with confidence interval of 2.0 to 10.6). Yet, length difference cannot come into play here as the underlying explanation. This indicates that clause type is not just a subsidiary effect to length, but has a significant influence of its own. We will present more evidence for this independent effect of clause type in section 4 discussing the logistic regression model. 
325
Object first Subject first main clause 77/89 
(86.5%) 12/89 
(13.5%) subordinate clause 28/48 
(58.3%) 20/48 
(41.7%) Length difference=0 Table 6 
3.5 Given/new status 
Given/new status was initially introduced as a factor influencing word order by linguists of the Prague School in the 1930's. Since then, it has been commonly assumed that constituents referring to contextually known information precede constituents referring to new information. Lenerz (1977) considers given/new status also for German as central to the word order. However, it is not always clear what counts as given information and what as new. To make the given/new distinction operational, we have used the taxonomy designed by Grondelaers (2000), which estimates the givenness of referents by how mentally accessible they are to the listener. Table 7 demonstrates the scale of the given/new taxonomy ranging from an entity that has to be newly created in the mental representation of the listener (1) to an entity immediately accessible in context (8). Because referents of pronouns are almost always accessible in the near context, they were not taken into account. This allows us to concentrate on the given/new status of the subject's referent. Table 8 shows the distribution of word order over the different degrees of the given/new taxonomy. The percentage of subject first cases is not monotonic increasing as the givenness of the subject referent rises. The Mantel-Haenszel chi-square statistic indicates however, that there is a significant linear association (p<0.01). The gamma-coefficient of 0.28 (ASE 0.067) points at a moderate, positive linear association. This means that the more given a subject referent is, the more likely it is for the subject to precede the pronominal object. This confirms the general view that given information tends to precede new information. 
Given /new taxonomy Object first Subject first 1 to be created 1162/169(95.9%)7/169 (4.1%) 2 to be created but constrained by context 248/55(87.3%)7/55 (12.7%) 3 to be created, constrained by an anchor entity in context 324/24(100.0%)0/24 (0.0%) 4 accessible through encyclopaedic knowledge 4103/117(88.0%)14 (12.0%) 5 inferable from anchor entity in context 5101/106(95.3%)5/106 (4.7%) 6 accessible in wider linguistic context 6255/291(87.6%)36/291 (12.4%) 7 inferable from near linguistic context 7 140/159(88.1%)19/159 (11.9%) 8 accessible in near context 8 56/74(75.7%)18/74 (24.3%) Table 7 Table 8 
3.6 Animacy 
Animacy is the central principle determining word order within the Mittelfeld according to the functionally inspired reference grammar edited by Zifonun et al. (1997). Constituents with animate referents are said to have a tendency to stand at the front of the middle field. Because the reflexives among the pronominal objects "copy" the animacy of their subjects, results for the object's animacy would be somewhat misleading. Therefore, we concentrate on the animacy of the subject. Table 9 presents the distribution of word order for animate subjects versus inanimate ones. The results confirm that there is a significant association between animacy and word order (chi-square p<0.01). The odds ratio of 0.43 (confidence interval 0.27 to 0.70) tells us that the odds of object first versus subject first 
326
amongst animate subjects is only half of what the odds are amongst inanimate subjects. In other words, animate subjects do get relatively more fronted than inanimate subjects. 
Object first Subject first animate subject 532/614 
86.6% 82/614 13.4% inanimate subject 357/381 93.7% 24/381 6.3% Table 9 
4 Multivariate analysis: logistic regression 
After having verified the influence of each of the 5 factors separately, we will now look at their combined effect. This means that we will try to determine which factor is most important, what kind of interactions between factors show up and how good our model based on 5 factors is. This kind of multivariate analysis can be done using a logistic regression model, in the sociolinguistic literature also known as VARBRUL. This technique models the probability for one of the values of the response variable as a weighted linear function of the explanatory variables as in the formula: 
logit(probality of response value1) = constant + weight1*factor1 + weight2*factor2 + weight3*factor3 
Because probabilities can only lie between 0 and 1 and the function can take in principle any value, the logit (the odds ratio's natural log) of the probability is modelled instead. For our study, the logistic regression model will model the probability of having object first using the 5 factors discussed above as explanatory variables 
We want the model to use only those explanatory variables that really explain some of the variation in the data. Statistical software can select these relevant explanatory variables using a stepwise selection procedure. The software will successively add more explanatory variables to the model by checking whether the model improves significantly compared to a model with only a constant called the intercept.. The software adds the variables that explain most of the variation first and stops adding variables when the model cannot be improved significantly any more.. 
Summary of Stepwise Selection Step Effect DF Score Chi-Square p-value Entered 1 Clause type 1 92.7053 <.0001 2 Given/new 1 17.0988 <.0001 3 Animacy 1 13.7755 0.0002 4 Length 1 10.5681 0.0012 5 Length*Animacy 1 24.9775 <.0001 Table 10 
Table 10 shows the outcome for the stepwise selection procedure for all of the 5 factors and all of their two-by-two interactions. The final model contains 4 explanatory variables and 1 interaction term. The chi-square statistic shows for each step that a significant improvement was made. The model-of-fit statistics indicate that the model fits the data reasonably well (Pearson p=0.12, Hosmer & Lemeshow p=0.74). It appears that clause type has the biggest effect on the choice of word order, followed by the given/new distinction, animacy and length difference. As we could already expect from the bivariate analysis above, grammatical function did not enter the model. Apparently it has no significant influence on the variation. The stepwise procedure also confirms what we had noted in section 3.4, viz. that clause type seems to be far more important than length difference. This gives an additional indication that clause type is not a subsidiary effect of length difference but that it has an effect of its own. Even more so, it has the most important effect of all. Moreover, the interaction between length difference and clause type did not make it into the model, which would have been expected, had clause type been a subsidiary effect. Somewhat surprisingly, the interaction between length difference and animacy did enter the model. Apparently, the effect of length is bigger at a certain level of animacy or vice versa, animacy has a bigger effect for certain length differences. The nature of this effect will become clearer looking at the next table. 
327
Table 11 shows the estimates for the weights each explanatory variable gets in the final model for predicting the probability of object first. The intercept represents the default result value. "Value" gives the value of the factor that was taken into account. The estimates indicate how strong the effect of a factor having a specific value is. Negative estimates point at a decrease in probability for object first, whereas positive estimates are a sign of increasing probability. The Wald chi-square and its p value designate whether a factor's effect is significant (p<0.01). The odds ratios give the effect the modelled value of a factor has on the odds of choosing object first over subject first. From these results we learn that the default value (the intercept) favours object first. Subordinate clause, given subjects and animate subjects disfavour object first, whereas a bigger length difference leads to more object first cases. Now the meaning of the interaction between length and animacy becomes clearer as well: bigger length differences combined with an animate subject lead to more object first cases. 
Parameter Value DF Estimate SE Wald chi-square p Odds ratio Intercept 1 2.8903 0.4174 47.9386 <.0001 Clause type subordinate 1 -1.0597 0.1233 73.8087 <.0001 0.347 Given new 1 unit increase 1 -0.1682 0.0559 9.0487 0.0026 0.845 Animacy animate 1 -0.9974 0.1812 30.3050 <.0001 0.369 Length 1 unit increase 1 0.1014 0.0253 16.0799 <.0001 1.107 Length*animacy animate 1 0.1096 0.0252 18.8982 <.0001 1.116 Table 11 
Finally, we come to the question, how good our model is. One way to assess the model is by looking at the explained variation. The Akaike Information Criterion gives a first measure for the unexplained variation in the model with only the intercept and a second measure for the unexplained variation in the model with the variables. The more measure decreases, the better. For our model we go from 677 to 551, which is a significant decrease. A second way of assessing the model is by looking at its predictive power. To measure this, we let the model predict word order in the original dataset and we look at how many cases the model predicted correctly. This we compare with the number of correct predictions an uninformed model with only a default parameter (the intercept) can make. The bigger the difference between these numbers, the better our model. The uninformed model predicted 89.1% of the cases correctly. The model with factors predicted 89.9% of the cases correctly at a cut-off point of 0.4. This tiny increase of 0.8% is of course somewhat disappointing, but not unexpected. The distribution of the response variable is so much askew (89% object first versus 11% subject first) that there is not much to it to have a good score on the prediction. You simply have to choose the most frequent value all the time and you get 89% correct. For the model with variables, it is very difficult to do much better. For a better assessment of the model we will have to use more data and other statistical techniques that are better suited for skewed data. 
5 Conclusion 
In this study we have looked at a long-standing problem in German syntax, viz. the word order variation of verb arguments in the Mittelfeld. We have analysed some of the problems previous studies encountered in studying this variation. We have proposed ways in which to deal with this very complex variation. On the one hand, we have chosen to restrict ourselves to one specific type of word order variation in the Mittelfeld, viz. the variation in the relative order of a nominal subject and a pronominal object. In this type of variation the complexity is somewhat reduced and therefore it provides a good starting point to study word order variation in the Mittelfeld in general. On the other hand we have argued that the influence different factors have is so subtle that a quantitative analysis on a firm empirical basis is called for, using extensive corpus data and the appropriate statistical apparatus. Next, we went on to present as a case study a first corpus analysis dealing with the word order variation of pronominal objects and nominal subjects in the Mittelfeld. First, we have been able to confirm that in general pronominal objects precede nominal subjects. Then we looked at 5 factors that according to the literature have an influence on the variation. Analysing their individual effects, we were able to show, that grammatical function is not as important as previously thought. Furthermore, length difference and clause type were both shown to have an influence, but clause type is not an epiphenomenon of length 
328
difference, as previously thought. The given/new distinction and animacy did have the effect described in the literature. In the multivariate analysis we were able to show using a logistic regression model that clause type is the most important of the 5 factors in determining the relative order of nominal subjects and pronominal objects. Additional confirmation was given to contradict the hypothesis that clause type is a subsidiary factor to length difference. We have shown that our model does reduce the amount of unexplained variation. Predictive power however, is difficult to assess due to the skewness of the response variable 
Some results from the analysis offered interesting suggestions for future research. We noted that in comparison with studies of literary text material the word order in our study of newspaper material is significantly more biased towards the default order. This might indicate an effect of text sort that has to be further looked into. The insignificance of grammatical function for the variation was very surprising as it is the one factor about which there is general agreement that it does matter. Further examination is needed to determine why its effect is not apparent here. Now we have shown that clause type has a strong effect, independent of length difference, the next step is to look for an alternative motivation behind it. Finally, an interesting interaction between length difference and animacy appeared in the model. What shape this interaction takes precisely, has yet to be determined. 
References 
Agresti A 1996 An Introduction to Categorical Data Analysis. New York, John Wiley & Sons. 
Behaghel O 1932 Deutsche Syntax: eine geschichtliche Darstellung. 4: Wortstellung, Periodenbau. Heidelberg, Winter. 
Eisenberg P 1994 Grundriß der deutschen Grammatik. 3., überarbeitete Auflage. Stuttgart, Metzler 
Grondelaers S 2000 De distributie van niet-anaforisch er buiten de eerste zinsplaats. Sociolexicologische, functionele en psycholinguistische aspecten van er's status als presentatief signaal. Unpublised PhD thesis, K.U. Leuven. 
Hawkins J 1994 A performance theory of order and constituency. Cambridge, CUP. 
Hoberg U 1981 Die Wortstellung in der geschriebenen deutschen Gegenwartssprache. München, Max Hueber Verlag. 
Kurz D 2000 Wortstellungsphänomene im Deutschen. Unpublished Master thesis, Universität Saarbrücken. 
Lenerz J 1977 Zur Abfolge nominaler Satzglieder im Deutschen. Tübingen, Niemeyer. 
Lenerz J 1994 Pronomenprobleme. In Haftka B (ed), Was determiniert Wortstellungsvariation? Studien zu einem Interaktionsfeld von Grammatik, Pragmatik und Sprachtypologie. Opladen, Westdeutscher Verlag, pp 161-173. 
Pechmann T, Uszkoreit H, Engelkamp J, Zerbst D 1996 Wortstellung im deutschen Mittelfeld. Linguistische Theorie und psycholinguistische Evidenz. In Habel C, Kanngießer S, Rickheit G (eds), Perspektiven der kognitive Linguistik. Modelle und Methoden. Opladen, Westdeutscher Verlag, pp 258-299. 
Poncin K 2001 Präferierte Satzgliedfolge im Deutschen: Modell und experimentelle Evaluation. Linguistische Berichte 186:175-203. 
Primus B 1994 Grammatik und Performanz: Faktoren der Wortstellungsvariation im Mittelfeld. Sprache und Pragmatik 32: 39-86. 
Reis M 1987 Die Stellung der Verbargumente im Deutschen. Stilübungen zum Grammatik:Pragmatik-Verhältnis. In Rosengren I (ed), Sprache und Pragmatik: Lunder Symposium 1986. Stockholm, Almqvist & Wiksell International, pp 139-177. 
Shannon T 2000 On the order of (pro)nominal arguments in Dutch and German. In Shannon T, Snapper J (eds), The Berkeley Conference on Dutch Linguistics 1997. Lanham (MD), University Press of America, pp 145–196. 
Speelman D 1997 Abundantia Verborum. A computer tool for carrying out corpus-based linguistic case studies. Unpublished PhD thesis, K.U.Leuven. 
Zifonun G, Hoffmann L, Strecker B 1997. Grammatik der deutschen Sprache. Berlin / New York, de Gruyter. 
329

A web-based concordance system for spoken language corpora
Knut Hofland HIT Centre University of Bergen Allegt. 27 N-5007 Bergen Norway Telephone: +47 5558 9463 Fax: +47 5558 9470 E-mail: Knut.Hofland@hit.uib.no URL: http://www.hit.uib.no/hit/hofland.htm Keywords: spoken corpus, transcripts, Web concordance, digitized sound
1. Summary
At the University of Bergen we have developed a web-based environment for the study of spoken language corpora. This system has been used in several projects both with English and Norwegian material. The recordings (done with different kinds of recorders) are digitized and orthograpically transcribed. The transcripts and sound files are aligned. From a web concordance it is possible to play the part of the sound file that corresponds to the text in the concordance line. The concordance lines can then be classified by entering codes on the concordance line. These are saved and can be used as a basis for statistical analysis (cross tabulation). A link to a figure of the system is given in the reference section. The paper/poster will describe this system in more detail.
2. Digitizing and transcription
In some projects the transcription has been done from the original tapes (COLT). In other projects the recordings have first been digitized (and also improved) and the transcription has then been done by playing the sound files on the computer. Most of the time a standard player and word processor have been used. In some cases we have used the special player developed by the University of Santa Barbara (VoiceWalker). In some recent projects we have used the program system Praat, developed at the University of Amsterdam.
Most of the recordings were digitized by using a PC with a sound card and a sound editor like CoolEdit. The recordings were saved in WAV format on CD-R (before the availability of large and cheap hard disks) and later on a large disk system. For some recordings the digital signal was sent directly from the recorder (DAT or MiniDisc) to the sound card.
For each informant a set of demographic parameters, such as gender, age, and occupation were also registered.
3. Alignment of text and sound
To be able to play the sound from the concordance we need to know the time stamp for each word in the text. This was done differently for the different projects. The COLT files were sent to a company in England (SoftSound) for automatic text and sound alignment. Due to the quality of the recordings (they were done by equipping the informants with portable Walkmans) and a lot of concurrent speech, there were a great deal of errors in the time stamps. We have manually gone through the material and corrected many of these errors.
For the Norwegian recordings we have mostly done the alignment manually. In the transcripts we put a time mark for every ten seconds. For the words spoken between these marks we did an interpolation (divided the ten seconds by the number of words). For recordings containing few long pauses this was sufficient, but in some cases we also had to mark the time at other places in the transcript.
In some projects we used Praat. This program keeps track of the time codes for the beginning and end of each segment which is transcribed. The text and the time codes can be read by other programs and are converted to the main format used for searching (we had to interpolate and also re-sort the different "tiers").
330
Some corpora are distributed with text and sound alignment. An example is the Santa Barbara Corpus of Spoken American English. Each line has a start and end time code. In Santa Barbara they used their own program, SoundWriter, to add these codes to the transcripts in an interactive process (using a suggestion based on the rate of speech). In general, 10-15 hours of work are required in order to transcribe and manually time align one hour of speech. 4. Tagging Most of the spoken corpora were also tagged with part of speech tags. The COLT Corpus were tagged with the CLAWS tagger in Lancaster. The Norwegian material was tagged by the new Norwegian tagger which was originally developed in Oslo and has been further developed in Bergen. In Norway we have two written standards for what linguistically is considered as one language. The transcripts were done in the standard most similar to the dialect of the informant. But often the informants used a mixture of the two standards and this is a challenge for the tagger (which originally works on one of the standards at a time). In one project the user manually tagged a set of words with her own SGML-tags. 5. Searching We used the program Corpus WorkBench from the University of Stuttgart for searching. This program uses a format of structural attributes (as COCOA mark-up) for information on the text and positional attributes (divided by tab characters) for each word. Words and punctuation marks are placed on separate lines. Each line has a column for time code (we only keep the reference in whole seconds) and additional columns for lemma, part of speech and information about each informant, such as gender, age group and home place. We have made CGI scripts for accessing the Corpus Query Program through the web. The user fills in a form in his web browser, giving information about words and additional information about the informant. The scripts return a concordance and each line has a link to an URL which extracts a sound fragment and returns this sound fragment either as a WAV or a MP3 file (the MP3 file is compressed with a factor of at least 10). On the server a special program that takes three arguments (a file, a start time and a duration) is used to extract the sound fragment. The user can save the sound file for processing in special purpose programs like Praat. The user may filter away some of the concordance lines, classify the rest and then store the lines on the web server. Files can be merged and then basic cross tabulation may be done. The data can also be exported to be used in other programs. 6. Conclusion The system has been used by researchers and students at the universities of Bergen and Oslo. They have got a valuable tool for the study of spoken language material. The Norwegian texts processed so far will be included in a future national spoken corpus. 7. References Figure: http://www.hit.uib.no/knut/mons/tale-fig.gif Example of search form: http://helmer.hit.uib.no/knut/sbcsae-test.html Example of concordance: http://helmer.hit.uib.no/knut/sbcsae-snd.html Praat program: http://www.fon.hum.uva.nl/praat/ SoftSound: http://www.softsound.com/ CWB: http://www.ims.uni-stuttgart.de/projekte/CorpusWorkbench/ COLT: http://helmer.hit.uib.no/colt/ Norwegian Spoken Corpus (pilot, in Norwegian): http://gandalf.hit.uib.no/talemaalskorpus/Hovedside.htm 331

The Corpora of Mandarin Chinese and German Animal Expressions 
Shelley Ching-yu Hsieh, Jan Jucker 
Southern Taiwan University of Technology 
1. Introduction 
Animal expression (AE) develops as a part of language along the time with the society. Everyday usage may disappear as Zheng (1993:6) says: the amount of basic lexemes of a language grows by 20% per 1000 years. So are the “modern” AEs in Spalding's work (1969:120) Gossenmolch (gutter-newt – policeman), lascher Bock (limp-buck – wimp) and Eule (owl – girl) no more used today. On the other hand, new AEs are created as a result of the technical development, e.g. Maus (mouse – the computer equipment mouse) and Elchtest (elk-test – dynamic test of the car stability). The present corpora contain 2980 Mandarin Chinese (MC) and 2630 German (G) written and spoken AEs. The included metaphorical vehicles are: 
Mammals: monkey, bear/panda, badger, squirrel, elk, elephant, donkey, fox, gazelle, hare/rabbit, deer, dog, hyena, hedgehog, jaguar, camel, cat/tomcat, leopard, lion, lynx, mouse/rat, horse/steed/zebra, cattle/ox/bull/buffalo/ cow/calf, sheep/buck/lamb/goat/ram/kid, jackal, pig/sow, tiger, whale, weasel, wolf, sable. 
Birds: eagle, magpie, duck/mandarin duck, owl, goose/wild goose, vulture, cock/chicken, Zhen,1 owl, crane, cuckoo, nightingale, parrot, peacock, oriole, turkey, crow, roc, reed bunting, swallow, swan/swan goose, sparrow, woodpecker, sparrow hawk, stork, pigeon/turtledove, bird, ostrich. 
Amphibians: frog, toad, salamander. 
Reptiles: dinosaur, crocodile, turtle/soft tortoise, snake. 
Molluscs: oyster, shell/clam, snail, squid/calamari. 
Annelid: ant, bee/wasp/drone/bumblebee, fly/dayfly, flea, praying mantis, cricket, chafer, crab, louse, maggot, moth, mosquito, caterpillar/silkworm, butterfly, spider, tick, cicada. 
Worms: worm. 
Fish: eel, fish, shark, pike, carp, manta. 
Mythical creatures: dragon, Phoenix, Qilin.2 
The metaphorical vehicles in G and MC are not concurrent with each other. Whereas the mandarin duck and the hyena appear to be as notable as the wolf in MC, there are neither mandarin duck nor hyena metaphors in G frequent. On the other hand, there are variations of vehicles in G with respect to a specific animal species, e.g., cattle, bull, buffalo, calf, cow and ox, while only the macro term - cattle ‘niu ‹�’ is used in MC. This study deals with all the important vehicles in each of the languages, even though they do not appear in the other one. 
Since half of the collected metaphors are seldom used nowadays, language change and language development in MC and G are observed by means of the application of the corpus, such as the reason for the disappearing of an AE and the newly generated metaphorical tenors. Also the concepts of the animal names are investigated. As human beings live close to animals since beginning of the history, studies based on this corpus about how people observe, describe animals and using animal names in their languages may help to understand or reveal culture backgrounds and cultural differences. 
1.1 Definition of the terms 
Animal expressions in the present corpora include: metaphors, similes, proverbs, sayings, frozen collocations, grammatically ill-formed collocations and routine formulae (Alexander 1978, Carter 1987, Moon 1998) as long as they are fixed expressions, not ad-hoc terms or freely generated phrases and contain at least one animal name that has metaphorical meaning (tenor). 
German refers to High German, say the official language in Germany. Mandarin Chinese refers 
1 Zhen  is a Chinese legendary bird with poisonous feathers. 
2 Qilin is a Chinese legendary animal, it is sometimes refered as Chinese unicorn. 
332
to Mandarin Chinese, the official language in Taiwan.3 
1.2 Methodology 
The written data are collected from literature and dictionaries (Chang 1992, Liang 1992, Riegler 1907, Büchmann 1972, Röhrich 1991 and elsewhere). Spoken data are gathered from mass media, daily conversation and questionnaires. The data are categorized by the animal names in alphabetical order in EXCEL. Different kinds of data relating to individual AEs were recorded in up to 12 separate fields, including phonetic transcription of the MC, word-to-word translation, metaphorical tenor of the AE, the origin, the underlying conceit, semantic feature of the metaphorical vehicle, frequency, etc. This paper reports mainly the field of the semantic feature and its interaction with other fields. The semantic features are abstracted as follows: 
1. compound A: e.g. shui3she2yao1 …ŽÖ�(water-snake-waist – a slender waist), here “snake” carries the semantic features [+slender, +curvaceous]. The features are heavily bound with other elements in the AE. 
2. compound B: in many cases the sole metaphoric sense of the vehicle is the intensifier of a word, as e.g. in Affenhitze (monkey-heat – scorching and sizzling heat) the monkey stands for [+intensifier]. 
3. phrase: An example for this category is null Bock (nil-ram – it doesn't really grab me to it) here the “ram” contains the semantic features [+mood, +desire, +lust]. 
4. sentence: shan1zhong1wu2lao3hu3, hou2zi cheng1da4wang2 ŽR'†–³VŒÕàËŽqâi‘剤(mountain-center-no-tiger monkey-call-big-king – the weak can king and dominate without the existence of the strong), here the tiger conveys the sense of the whole sentence [+head, +powerful, +significant] 
Lakoff & Johnson (1980) indicate that the Americans very often integrate the concepts such as “love is madness” and “time is money” into their daily used language and actions. Such conceptual metaphor is a use of imagery based on concepts which is heavily culture-related. The present AE corpora also demonstrate that our adopting animal names as metaphorical vehicles in our languages is closely related to situation or to culture. Low (1988:133) points out that AEs refer to undesirable traits, reflecting human views of animals as lower forms of life. The present corpora give a different view. Though animal metaphors are often applied as abuse, some of them are used positively (e.g. terms of endearment, benediction, praise, jest…) and neutrally (e.g. objects, euphemisms, exclamatory expressions…). 
3. Semantic functions of animal names in languages 
Only the most productive Vehicles will be briefly discussed below because of the space of this paper. 
3.1 In Mandarin Chinese 
The followings will discuss only most productive metaphorical vehicles: domestic animals occupy 35.2%, wild animals 20.7% and mystical animals 9.5% of the collected MC corpus. 
3.1.1 Names of domestic animals 
Table 1 lists the semantic functions of the most productive vehicles, where with percentages indicate the more salient features. Horse belonged to important transportation means and played an important role in the war. These roles are still active in languages, though not in society. Bing1ma3 •º”n(troop-horse) first stands for military forces, now also for workforce or personnel. Qian1jun1wan4ma3 çŒRäÝ”n(thousand-soldier-ten thousand-horse) means not only a huge army, but also an imposing scenery. The metaphorical vehicle ma3 ”n(horse) records the Chinese history – the transportation, war, social activities, old utensil, clothing and philosophy, as shown in table 1. 
In modern usage, as languages contact extensively, when translating the phonetic combination 
3 The official language in People's Republic of China is also MC. Here I concentrate on the Mandarin in Taiwan as it is my mother tongue. There are linguistic differences between both regions, though a large overlapping of the language variants. 
333
/ma/, the morphological representation ma3 ”nis adopted, such as in ma3sai4ke4 ”næÎ�Ž(mask; censorship), ma3la1song1 ”n�f�¼(marathon), etc. The reasons for this are: ma3 ”nis one of the 5000 most frequent Chinese morphological representations, it has a neutral or positive connotation as shown in table 2, and further, /ma/ is a common phonetic combination in alphabetical languages. Ma3 ”nis here a semantically arbitrary application. 
Ji1 (chicken) has a similar development. Ji1po2 ”k(chicken-aunt) is a phonetic translation from the Taiwanese gebe4 (someone who likes to take care of something which is not his business), a frequent AE in newspapers. The linguists believe that jipo “”k” should be written as jiapo “‰Æ”k”, but the mass media persist the usage. The Chinese character ji is probably used because the semantic feature [+talkative] of the chicken-AE does fit to the sense “smart-alecky, saucy, brassy”. Long2 —´(dragon) has the similar and even more stronger usage to ma3 ”n(horse) and ji1 (chicken) that will be further discussed in 3.1.3. 
Yang2 —r(sheep) is not productive in MC. One of the salient features of it is [+profit] that is also one for niu2 ‹�(cattle). Niu2 is similar to ma3 ”n(horse), both introduce the old and new society, such as lao3niu2che1 V‹�ŽÔ(old-cow-car – a very slow car) and niu2rou4chang2 ‹�“÷�ê(cattle-meat-field – striptease). 
Table 1. Semantic functions of domestic animal names in Mandarin Chinese 
Vehicles Semantic Functions Cat Weakness 27.3%, shrillness, falseness, lechery, gluttony Cattle big size 11.1%, strength 11.1%, stupidity 9.3%, sluggishness 5.6%, stubbornness 5.6%, intensifier 5.6%, slowness, profit, Chicken inferiority 23.4%, announce the daybreak 10.6%, talkative, Dog inferiority 23.7%, serf 10.5%, cheat 6.6%, stupidity 5.3% Horse strength 14.9%, war 12.3%, transportation means 9.6%, power 7.9%, good people 3.5%, woman 2.6%, toil, agile, stupidity, labour, diligence, wild Mandarin duck wedded bliss 66.7%, Pig stupidity 29.4%, laziness 17.6%, dirt 17.6% Sheep profit, event, 
3.1.2 Names of wild animals 
The semantic features are highlighted through the blend of other lexical items in the AE, e.g. tiger in MC can combine with wolf, and its [+greed, +hunger, +cruelty] are stressed, as in lang2tun1hu3yan4T ŒÕš‹(wolf-swallow-tiger-gobble –wolf down; gobble up; devour ravenously) and ru2lang2si4hu3 ”@TŽ—ŒÕ(like-wolf-like-tiger – as ferocious as wolves and tigers; like cruel beasts of prey). When sheep and tiger come together, tiger's [+cruelty, +violence, +danger] is emphasized, as in yang2ru4hu3kou3 —r“üŒÕŒû(sheep-in-tiger-mouth – in a perilous position). There are 11.78% AEs with two or more vehicles in MC, but only 1.93% in G. This reveals the syntactic and semantic nature of MC. 
The MC hu3 ŒÕ(tiger) occupies the domain ‘big, strong', the other features such as [+courageous, +vitality, +proud, +significant, +energetic] are included in this domain; big and strong can be ‘powerful’ or conversely ‘dangerous’ depending on the need. This makes tiger a “wonderful” semantic contributor for the subject ‘man', we therefore have a good number of tiger-AEs in MC denoting ‘man', e.g. biao1xing2da4han4 •VŒ`‘势(young tiger-form-big-man – husky fellow) and hu3jiang4 ŒÕ›'(tiger-general – brave general), etc. This domain is bordered by gender. It is kept for males in MC and cannot be applied to females. When it is, the sense is shifted to ‘a terrible woman', as in mu3lao3hu3•êVŒÕ(female tiger – tigress; fractious women), hu3gu1po2ŒÕŒÆ”k(tiger-aunt – evil 
4 Taiwanese is a dialect that 73.3% of speakers speaking in Taiwan (Huang 1993:21). The stranscription of Taiwanese is based on IPA (Interantional Phonetic Alphabet). 
334
woman)5 and bai2hu3xing1”'ŒÕ¯(white-tiger-star – jinx; women who bring hard luck), showing only negative tiger semantic features. Cultural background is required for understanding such expressions. 
AEs demonstrate sense extension involved with metonymy, e.g. e2mei2 ‰é”û(moth-eyebrow – delicate eyebrows; beautiful woman). A part-for-whole coding strategy amply exploited in language. 
Although some domestic animals are also food products, yu2 ‹›(fish) and niao3 ’¹(bird) are semantically representative for [+profit, +gain], such as in fang4chang2xian4diao4da4yu2 •ú'·ü'Þ‘å‹›(cast-long-line-catch-big-fish – to wait patiently for one's chance; good bait catches good fish) and yi4shi2liang2niao3 êΙ_'¹(one-stone-two-bird – to kill two birds by one stone) and so on. 
The animal pairs yuanyang ‰•éñ(Mandarin ducks), hudie åš'±(butterflies), fenghuang –P™€(Phoenixes), longfeng —´–P(dragon-Phoenix) und jiandie úYéÕ(the legendary bird-flatfish) represent also in pairs in MC no matter they exist in nature or in fantasy. 
Table 2. Semantic functions of wild animal names in Mandarin Chinese 
Vehicles Semantic Functions Bird gain 10.3% Crane isolation 30%, freedom 30% Deer profit 28%, Fish profit 17.1% Fox malevolence 41.7%, Snake malevolence 46.7%, dread 33.3% Tiger power 24.4%, danger 22.1%, malevolence 15.1%, cruelness 9.3%, courage 4.7%, greed 3.5%, big, great, swallowing, jumping, vitality, proud, significant, valuable, energetic, robust, awfully, auspicious, superstitious Wolf malevolence 26.9%, cruelness 15.4% Worm damage 64%, laziness 28%, inferiority 24% 
3.1.3 Names of mystical animals 
The above discussion supports Lakoff and Turner's (1989: 170ff in Moon 1998:197) proposal: “animals form the second highest level, and are seen in terms of ‘instinctual attributes and behaviour.'” However, another kind of AE, formed by mystical animal names, is used with over 10% in MC AE (only 0.67% in G) and should be taken into consideration. MC mystical AE convey [best people, strong man, power]. They can even be [holy, auspicious] as shown in table 3. 
From the viewpoint of derivation, dragon lexemes unlike other animal lexemes that favor to describe the appearance, e.g., ying1gou1bi2 ‘éçê•@(hawk-hook-nose – aquiline nose) and yu2du4bai2 ‹›ãê”'(fish-belly-white – the whitish color of a fish's belly; gray dawn), etc. or the character of the referents, e.g., gou3ji2tiao4qiang2 ‹ç‹}'µà­(dog-rush-jump-wall – a cornered beast will do something desperate) and ji2ru2re4guo1shang4de'ma3yi3 ‹}”@”M“ç�ã“IΛ‹a(hot-pot-on-of-ant – as restless as ants on a hot pan). Dragon lexemes only mention the body parts but have no description (long2yan3 —´Šádragon-eye – longan) or only draw attention to the dragon's movement but still leave the space and imagination to the language users, as in long2xing2hu3bu4 —´�sŒÕ  (dragon-walk-tiger-pace – a great warrior's firm strides are like the dragon's and the tiger's) and long2fei1feng4wu3 —´”ò–P•‘(dragon-fly-phoenix-dance – like dragons flying and phoenixes dancing; lively and vigorous flourishes in calligraphy). A mystical creature has no determined appearance to be described, thus, also it offers the speaker and listener a space for imagination and a blank to fill in on their own accord. 
The same as the modern usage of above mentioned ji1 (chicken) and ma3 ”n(horse), long2 —´(dragon) has even become a morphological root in transliteration, e.g. meaning textile goods: šú—´(Orlon), ’él'é“Á—´(Teijen Tetoron), “Á‘½—´(Tetoron), ¤ãN—´(Exlan) and–Ñ—í—´(Vonnel) (Yao 1992:343). Homonyms play a key role as the words are loaned. Long2 not only is a semantic 
5 Hu3gu1po2  (tiger-aunt –evil woman) is originally a fairy tale figure who transformed into a loving old grandma, but actually just wanted to devour children. 
335
marker denoting [+positive, +super], but it also serves as a popular phonetic representation for the phonological unit [+liquids] + [-front vowels] + [+nasal C.]. This is a robust tendency for many Chinese characters when loaning words from other languages: semantic marker > phonetic element. 
There are many meteorological and astronomical terms which have adopted long2, since long2 carries the semantic concept of nature – sky, earth, water and fire. As a bound morpheme used to describe the head of the lexeme, dragon represents this group of adjectives “remarkable, valuable, important, strong, powerful, super”, all positive, e.g. long2zi3 —´Žq(dragon-son – descendents of emperors in ancient times) and long2zhong3 —´Ží(dragon-race – Chinese people). It reaches all dimensions big, deep, wide, high, far. The female counterpart of long2 (dragon) is feng4 –P(phoenix) which also carries the same positive sense, e.g. long2feng4pei4 —´–P”z(dragon-phoenix-match – union of a dragon and a phoenix). Feng4 –Pis male when compounds with huang2 ™€or luan2êasuch as in feng4qiu2huang2 –P‹™€(male phoenix-ask-female phoenix – the male chasing after the female) and luan2feng4he2ming2 êa–Pa–Â(female phoenix-male phoenix-harmony-sound – be blessed with conjugal felicity; be a happy couple). 
Table 3. Semantic functions of mystical animal names in Mandarin Chinese 
Vehicles Semantic Functions Dragon best people 32.8%, auspiciousness 32.8%, strong man 16.4%, holiness 9.8%, power 9.8% phoenix auspiciousness 60%, beauty 30% Qilin auspiciousness 66.7% 
3.2 In German 
The followings will discuss only most productive metaphorical vehicles in G corpus: domestic animals 37% and wild animals 10.1% of the collected AEs. 
3.2.1 Names of domestic animals 
Null Bock haben (nil-ram-have – it doesn't really grab me to it) and auf etwas Bock haben (to-something-ram-have – feel like doing something) may have it's origin in the late 70ties and became popular among teenagers (Röhrich 1991: 227). The adjective “stier” is a homonym of Stier (bull). Drosdowski (1997:713) says this is a conversion of the low German and of the Dutch word stur being influenced by the animal name Stier. Even around 1600 the Dutch stuur (stern,gruff) is used in sense of “staring like a bull in a wild and threatening way”. 
The distinct responsibilities of semantic domains is demonstrated by AEs. Take vehicle Katze (cat) as an example, [weak, false, small, unimportant, flattering, quick, shrill] are conceptualized in a German speaker's mind and a good number of Katze expressions connote these semantic features. Therefore Katze is a “perfect” semantic contributor for the subject ‘woman'. Many German Katze expressions denote this meaning, such as Kätzchen (little cat – an endearment for a woman), falsch wie eine Katze ((a woman who is) as false as a cat) and Schmusekatze (flattering cat – an endearment for a woman). This domain is occupied by Katze, and should not be invaded. It is bordered by gender,6 i.e., the semantic features of Katze [+small, weak, false, moody] don't function for males, otherwise the meaning will shift to [+intoxicate, loaded, hangover], the behavior when a man is drunk, as in Katzenjammer (cat's complaint – the complaint after a hangover). 
Some semantic features of the metaphorical vehicles seem awkward and have puzzled linguists, driving them to trace folk-etymology for answers (Riegler 1907, Storfer 1935, Büchmann 1972, Röhrich 1991:818). However, they can be understood when applied semantic interconnection and interaction, e.g., one of the semantic feature of German cat is [+nothing], as in Das ist für die Katz (this is for the cat – This is worth nothing, I don't know why I bothered) or Dann gehört er der Katz (this belongs to the cat – This is in vain, I don't know why I bothered). The Germans nevertheless can understand the meaning of Das ist für die Katz (this is for the cat – this is worth nothing) from the other semantic features of Katze: [+small] (as in Katzenschwanz7 and Katzentisch8) and [+unknown 
6 The Katze (cat) can also be applied to children, but not to men. 
7 Meaning: A cat tail – a trivial. 
8 Meaning: A cat's table – a little table for dining. 
336
matter] (as in die Katze im Sack kaufen9 and die Katze aus dem Sack lassen10). The interconnection and interaction of semantic features explains. The senses of the vehicles in question are intimately related since they are developed from the same animal and denote the same cultural background. Salient elements will surface and occupy syntactic position available with understood elements backgrounded. 
According to Braun (1993:495) the meaning of Hahnrei (cuckold – a deceived husband) may have been originally Kapaun (capon), as being used as late as 16/17th century. This may provide an explanation for the origin of the saying jmdm. Hörner aufsetzen (someone-horn-put on – to cuckold someone). There was a custom to cut the spurs of the capon and to implant them into it's crest where they continued growing like horns. 
Table 4. Semantic functions of domestic animal names in German 
Vehicles Semantic Functions Horse strength 12.2%, stupidity 12.2%, arrogant 10.2%, transportation means 8.2%, values 8.2%, robustness 6.1%, heavy workload 6.1%, common people Cow stupidity 27.8% Calf silly behaviour 53.8%, stupidity 30.8%, money 23% Ox stupidity 41.2%, heavy labour 23.5% Bull stupidity 17.6%, aggressiveness 17.6%, intensifier 17.6% Sow intensifier 26.3%, dirt 15.8%, obscenity 15.8%, happiness 13.2%, mess 10.5% , insidious 7.9%, Chicken stupidity 16.7%, woman 10%, craziness 10%, mess 10%, talkative, proud Rooster man 18.2%, weather vane 18.2% Cat Insignificance 14%, superficiality 8%, flattering behaviour 8%, small amount 6%, falseness, quickness, shrillness, ill-temper, pursuance, cleanliness, misfortune, cowardice, absurdity, intolerableness Tomcat hangover 55.6%, depression, craziness, misfortune Dog poverty 13%, wickedness 9.8%, intensifier 7.6%, worthlessness 6.5%, baseness 5.4%, insignificance 5.4%, malevolence 4.3%, someone 3.3%, labour Pig dirt 37.5%, mess 25% 
3.2.2 Names of wild animals 
The notable semantic functions of Tiger (tiger) [+quickness, +jealousy], Vogel (bird) [+symbol of freedom],11 Wurm (worm) [+wriggling forward] and Fisch (fish) [+no intelligence, +uncertainty, +unreliability] as discussed in Riegler (1901) are fading, some of them are lost during the course of last century. 
Many endearments in terms of animal names are observed in G. They reveal the traditional gender roles in German society: The endearments that are applied to women are derived either from a domestic animal (lamb: Lamm, Lämmchen), pets (cat: Schmusekatze, Kätzchen, rabbit: Hase, Hasi, Häschen), a culture follower12 (mouse: Mäuschen, Mausi) or small and light birds (swallow: Spatz, Spätzchen, dove: Täubchen), whereas, those for men are derived from a wild animal, the bear: Bärchen, Knuddelbär, Brummbär. This linguistic fact first shows the human nature that men are physically stronger than women. Secondly, traditionally women were responsible for the household while men were considered to be the breadwinners in the society. As the notion fades in modern society, language continues to file it. 
Originally the meaning of Elefantenhaut (elephant skin) was merely the skin of an elephant. By sense extension the expression gained an additional meaning. The semantic feature [+thick/fat] plays here the decisive role. The outer physical size transformed into a behaviour feature. The extension of 
9 Meaning: To buy a cat in a sack – to buy something without knowing what that is. 
10 Meaning: To let cat go out of the sack – to tell the covered truth; let the cat out of the bag. 
11 The Vogel (bird) is still the symbol of freedom in German society, but not shown in modern German language. Vogelfrei (bird-free – outlawed) is a closed one. 
12 Culture followers are animals that live in close proximity to humans. 
337
the meaning of Elefantenhaut (to have a thick skin; to be insensitive, inured, indifferent, apathetic) goes from the skin of the elephant to insensitive people. This tendency of changing from concrete concept to abstract concept echoes Aitchison's (2001:130) universal laws: Humans “using everyday external bodily behaviour to describe internal events”. 
Sometimes both Chinese and German peoples share the same points of view regarding some animals. But the differences in the interpretation and in the opinion lead to the development of different AEs. That a snail carries a shell is observed by both peoples and AEs are produced accordingly. They are wu2ke2gua1niu2 –³Ÿvå—‹�(no-shell-snail – people who are not capable of purchasing houses) and gua1niu2zu2 å—‹�‘°(snail-tribe – people who do not possess real estate) in MC, where the basic need - housing, is concerned, and sich in sein Schneckenhaus zurückziehen (self-in-one's-snail shell- withdraw – to go into one's shell) und jemanden zur Schnecke machen (someone-to-snail-make – to come down on someone like a ton of bricks) in G, it concentrates more on the behaviour and emotional phases. The underlying conceits vary in these snail-expressions and the idiom schemas are recognizable. 
Table 5. Semantic functions of wild animal names in German 
Vehicles Semantic Functions Bird comic 11.1%, symbol of freedom Fish fish-fishing 26.4%, fish-water 14.7%, profit 11.8%, event 11.8%, fish-swimming 11.8%, no intelligence, uncertainty, unreliability Fox cleverness 28.6%, shrewd 19%, anger 19% Rabbit fear 26.7%, woman 20.7% event 17.2% Snake poison 15.8%, cunning 15.8% Tiger power 66.7%, courage, hunt, jealousy, protector, quickness, gasoline Wolf cruelness 27.8%, destruction 22.2%, hunger 16.7%, greed 16.7%, malevolence 16.7% Worm parasite 25%, small size 20%, defenselessness 20%, defect 15%, danger 15%, wriggling forward 
4. Conclusion 
4.1 The concepts of animal names 
Fixed expressions are culturally marked lexical items and they are “established as pithy ways of expressing and referring to concepts” (Moon 1998:43), so are AEs. The vehicles of AEs can be domestic animals, wild animals or mystical creatures. There are more AEs generated from domestic animals than from wild ones. Since human beings and domestic animals live closely together and come into everyday contact, the concepts of these expressions relate to [normal, insignificant]. Because those animals depend on human beings, these AEs also convey the concepts of the [good-hearted, weak]. On the other hand, the concepts of wild-animal expressions tend toward [big, crude] or [great, dangerous], since wild animals are difficult to tame. The semantic functions that generated more than 2.6% expressions of the respective vehicles are listed in table 6. Some vehicles produce only a limited number of AEs, nevertheless concepts are found among them. This goes back to specific ideas and concepts that people associate with these vehicles, e.g. Ameise (ant): busty, Esel (donkey): stupidity, yuan1yang1 ‰•éñ(mandarin ducks): wedded bliss, and lu4 Ž­(deer): profit. 
While there are many newly invented AEs, such as those related to computer (Maus mouse – the computer equipment mouse, qian1xi1chong2 çâUå³thousand-year-worm – millennium bug), the tendency of fading AEs are also invasive. The reasons for the disappearing or replacement of a semantic function can be traced from the development of the technology and society, e.g. people wear yan4wei3fu2 ‰�”ö•ž(swallow-tail – swallow-tailed coat) instead of chang2pao2ma3gua4 ’·åÚ”nŠ|(long-robe horse-gown – a ceremonial wide-sleeved jacket of a mandarin) in a ceremonial occasion. When electronic mail are popular, the retronym snail mail is produced. Neither zwei Finger Adler-System (two-finger-eagle-system – the beginner's method to type using two fingers) nor Elchtest (elk-test – dynamic test of the car stability) were needed some decades ago. Language contact changed the structure of a language, such as the enlargement of roots (long2 —´dragon stands for textile) and the development of semantic markers (such as ma3 ”nhorse) to phonetic representations. 
338
Table 6. Concepts of the Animal Expressions in Mandarin Chinese and German 
Vehicles Mandarin Chinese German ant tiny Bustle bear rudeness, power, men, intensifier beast cruelty, fright Malice bee diligence, attractiveness, women bird profit, gain comic effect cuckoo: devil goose: stupidity nightingale: beautiful singing peacock: vanity raven: blackness sparrow: tiny stork: delivering kids swan: presentiment vulture: greed, devil crane calmness, isolation Mandarin-ducks wedded bliss sheep profit, event stupidity, profit, gain, need of protection, property lamb: Patience, innocence, love, gentleness, defenselessness buck: stubbornness, lust, men cat weakness something small, insignificance, superficial, flattering behaviour tomcat: results of alcohol consumption, hangover cattle big size, height, strength, stupidity, laziness, stubbornness, intensifier, big size, sluggishness, intensifier Worthlessness, immoral person bull: stupidity, aggressiveness, intensifier calf: stupidity, silly behaviour, money, foolishness cow: stupidity ox: stupidity, heavy labour chicken inferiority, announce the daybreak women, craze, stupidity, mess rooster: men, weather vane cricket curiousness, mood deer profit, dog inferiority, deceit, stupidity, serf, cheat misery, poverty, wickedness, meanness, insignificance, worthlessness, malice, anybody, intensifier, in connection with weather, baseness, malevolence poodle: intensifier donkey stupidity stupidity, stubbornness dragon holiness, power, best people, strong, auspiciousness Wickedness fish profit, gain, goal profit, gain, event, things, fish-fishing, fish-water, fish-swimming pike: nimbleness flea tiny, restlessness, intensifier fly tiny, insignificance fox wickedness red, clever, crafty, troubles hare fear, women, event hedgehog protective posture horse means of transport, war, forces, strength, means of transport, values, strength, robustness, hard work, miner 
339
women, good people, power standards, stupidity, arrogance, common people lion strength, king, power, center, malice, danger louse annoyance, brazenness monkey foolishness, intensifier mouse villain something small, cute appearance, money rat: enthusiasm phoenix beauty, propitiation, auspiciousness pig stupidity, laze, dirt, filth dirt, filth, luck, mess sow: dirt, filth, luck, indecency, ambush, intensifier, obscenity, happiness, mess, insidious qilin propitiation, auspiciousness chicken inferiority, announces daybreak snail slow snake malice, fright poison, trick tiger strength, courage, greed, danger, malice, cruelty, power, malevolence strength, power weasel quickness wolf wickedness, cruelty, malevolence hunger, greed, malice, cruelty, destruction, malevolence worm inferiority, decomposition, laze, damage tiny, defenselessness, parasite, disturbance, defect, danger maggot: spoilt, disparage 
4.2 Other possible applications of the corpora 
Other possible applications of the corpora are as follows: From the field of “underlying conceit” we can observe how Germans and Chinese observe, perceive animals and produce AEs. It seems that Chinese tend to generate more expressions from animal's appearances, whereas the Germans more from animal's habitats, such as the above mentioned snail-expressions. When examining in particular those AEs of the same metaphorical meanings but different vehicles, cultural difference can be exposed. 
As many vehicles serve solely as intensifier in German AEs, while there are a good number of AEs with two or more vehicles in MC (11.78%). To reveal syntactic and semantic nature of the languages in question could be a topic for further study. 
Many AEs are proper names (e.g. Maus mouse – the computer equipment mouse) or technical terms (e.g. Elchtest elk-test – dynamic test of the car stability) but they are common for language users. We can analyze their syntactic structure or metaphoricality to see the principles of analogy (see Aitchison 1992:40ff). 
Based on the field of “metaphorical tenor”, we can analyze the life philosophies, aesthetic values, modes of thinking of the peoples. Moon (1998:163) recognizes the connoted evaluation of fixed expressions. In the course of present study, it became clear that AEs are vocabulary of peoples’ values, because about 80% of them are used to scorn or warn people, i.e., they convey values from different societies, for example, [stupid], [worthless], [fast], [pretty] are most salient animal concepts in those expressions, a dai1tou2e2 •ð“ªê@(stupid-head-goose – a man who is not aware of a woman's flirt) is considered as “stupid” in a Chinese speaking society, while albernes Huhn (silly-chicken – a giggly woman) stands for “stupid woman” in Germany. Ren2sheng1bu4du2shu1 huo2zhebu4ru2zhu1 l¶•sæ¤�‘Š'•s”@æµ(people-life-not-read-book live-not-as-pig – people living in the world would be ignorant if they do not study) is thought to be “worthless” in a MC speaking society. Further more, together with the field “frequency”, we can compare newly generated AEs and old-society AEs to observe the change of social structures. 
References 
Aitchison J 1985 Language change: progress or decay. New York, Universe Books. 
Aitchison J 1992 The articulate mammal (3rd edition). London, Routledge. 
Alexander R J 1978 Fixed expressions in English: a linguistic, psycholinguistic, sociolinguistic and didactic study (part 1), Anglistik und Englischunterricht 6:171-88. 
Anttila R 1989 Historical and Comparative Linguistics. New York, John Benjamins. 
340
Braun, Wilhelm, Ginschel, Gunhild, Hagen, Gustav 1993 Etymologisches wörterbuch des deutschen. Berlin, Akademie. 
Büchmann G 1972 Geflügelte worte – der zitatenschatz des deutschen volkes. Berlin, Haude & Spenerschen. 
Carter R 1987 Vocabulary. London, Allen and Unwin. 
Chang F C ('£–Fž^) 1992 Far East Chinese-English dictionary. Taipei, Far East. 
Chen G H C ('ÂŒdŽì) 1987 Study of Chinese lucky words and taboos. Unpublished M.A. thesis, Fu Jen Catholic University. 
Drosdowski G (ed) 1997 Duden (Band 7) etymologie: herkunftswörterbuch der deutschen sprache. Mannheim, Dudenverlag. 
Hsieh, S C Y 2001 Teirmetaphern im modernen Chinesischen und Deutschen: eine vergleichende semantische und soziolinguistische studie. Unpublished PhD thesis, Tübingen, Tübingen University. 
Hsieh, S C Y 2002 Betrachtungspunkte zu tieren und interpretationen der tiermetaphern: kulturunterschiede zwischen Chinesen und Deutschen. In A collection of speeches & papers delivered at the Symposium: Combining Multi-culture with English Language Teaching in Vocational Universities and Senior High School, Taiwan, pp 117-143. 
Hsieh, S C Y 2003 Vogel metaphern und Taiwan heute: sprache und was dahinter steht. Taiwanese culture and literature news. Germany, Ruhr University Bochum. (paper accepted) 
Huang S 1993 Yuyian, shehei yu zuqun yishi - Taiwan yuyian sheheixue de yanjiu. (Language, society and population conscious - a sociolinguistic study in Taiwan). Taipei, Crane. 
Lakoff, Turner M 1989 More than cool reason: a field guide to poetic metaphor. Chicago, University of Chicago Press. 
Lakoff G, Johnson, M 1980 Metaphors we live by. Chicago, University of Chicago Press. 
Liang S C (â뛉�H) 1992 A new practical Chinese-English dictionary. Taipei, Far East. 
Low G D 1988 On teaching metaphor. Applied linguistics 9(2): 125-47. 
Moon R 1998 Fixed expressions and idioms in English. Oxford, Clarendon Press. 
Riegler R 1907 Das tier im spiegel der sprache. Dresden, C A Kochs. 
Röhrich L 1991 Lexikon der sprichwörtlichen redensarten. Freiburg, Herder. 
Spalding K 1969 Some aspects of taboo and up-grading in contemporary German. In Essays in German language, culture and society. Siegbert S P, Thomas R H (eds.) London, Univeristy of London. 
Storfer A J 1935 Wörter und ihre schicksale, Berlin, Atlantis. 
Yao R S (›LžÄ�¼) 1992 The problem of current loanwords in Taiwan. Bulletin of National Taiwan Normal University (6): 37. 
Zheng Y F (“A–ç•v) 1993 Liyu zhouci guanqiang heihua (Greetings, curse, official jaron, argot). Beijing, Guangming ribao chubanshe. 
341

Frame, phrase or function: a comparison of frame semantics and local grammars 
Susan Hunston, University Of Birmingham 
1. Introduction 
In this paper I wish to address some issues that arise when considering the integration of lexis and grammar, and the mapping of semantic roles on to regularly occurring sequences of English. More specifically, I wish to compare the ‘frame semantics’ approach that underlies work done by Fillmore and his colleagues at Berkeley and the ‘local grammar’ approach underlying work carried out at Birmingham. My intention is not to denigrate either approach but to see how both might benefit from a recognition of their similarities and differences. 
Since the development of large corpora for language research, it has become a commonplace to note that sense and syntax are connected in some way. For example, a number of writers draw attention to variation in the behaviour of individual lexical items that have several meanings. Sinclair (1991: 53-65) notes that the various senses of YIELD are differentiated by their syntactic patterning. For example, most instances of yield meaning ‘give way’ are intransitive, while most instances of yield meaning ‘produce’ are transitive. Roland et al (2000) identify verbs and their behaviour as transitive or intransitive in different corpora. Where there are marked differences, this is because the verbs have slightly different meanings in the two corpora. For example: The ball floated downstream (Intransitive; Brown corpus); ..float its big paper and British retailing businesses via share issues to existing holders (Transitive; Wall Street Journal corpus). Hunston (2002: 46-48) notes of the noun initiative that as a count noun (with any determiner) it means ‘something that someone starts to try to solve a problem', while take/seize/lose the initiative (with the only) means ‘(fail to) start something and so (fail to) gain an advantage over a competitor', and initiative with a possessive or with no determiner (an uncount noun) means ‘the quality of being able to do things without being told'. 
Further observations are that words with similar meanings have similar grammatical behaviour. Levin (1993), Baker and Ruppenhofer (forthcoming) and Dang et al (1998) all classify verbs based on their semantics and their syntactic behaviour. For example, Levin (1993: 177) describes a class of verbs whose members are alter, change, convert, metamorphose, transform, transmute and turn. These verbs all share what Levin calls ‘properties'; that is, they all occur in structures such as ‘The witch turned him into a frog', ‘He turned into a frog', ‘The witch turned him (from a prince) into a frog’ but not ‘The witch turned him from a prince'. Francis et al (1996, 1998) propose ‘meaning groups’ composed of words that share what is called ‘pattern’ and also aspects of meaning. For example, nouns complemented by prepositional phrases beginning with among include a group that refer to situations of conflict or competition: competition, conflict, controversy, differences, disagreement, dispute, dissension, divisions, fighting, infighting, rivalry, split, struggle, violence, war. 
Sinclair and Levin account for these groupings in similar but converse ways. Levin (1993: 1) argues that ‘… the behaviour of a verb, particularly with respect to the expression and interpretation of its arguments, is to a large extent determined by its meaning.’ Talking of discoveries inspired by corpus-driven lexicography, Sinclair (1991: 7) says ‘Soon it was realized that form could actually be a determiner of meaning'. Although Sinclair ultimately rejects any directional causality between meaning and pattern, the opposition between ‘meaning causes behaviour’ and ‘behaviour causes meaning’ is an important one to this paper. 
The association of sense and syntax has a number of important consequences. Firstly, a word may be used in an unusual way but its meaning will be recognisable from the pattern it occurs in. Hunston and Francis 1999 (100-107) give a number of examples such as 
He couldn't apologise his way out of this. (Pattern: V way prep) 
There is something almost American about the minister's informality. (Pattern: something ADJ about n) 
…as if by doing so I could debate him into loving me. (Pattern: V n into –ing) 
These patterns, which are very productive (i.e. the lexical items they are used with are relatively unrestricted), are prime candidates for using words in unusual ways. Secondly, through diachronic change, words may take on new meanings with the patterns associated with those meanings. E.g. 
342
It was big of you to take the risk. (H&F 1999: 105) (= good of you) 
A Labour MP leaked him a copy of the committee report. (= gave) 
The Governor ‘remains comfortable that no innocent person has been executed'. (= confident) 
Thirdly, genuine ambiguity is relatively rare (Sinclair 1991: 104-5; also Mason and Hunston 2001) and occurs when, in Sinclair's terms, a series of open choices leads by chance to a sequence that would exist also by virtue of the idiom principle. The rareness is exploited in jokes (see Hunston and Francis 1999: 23; Hunston 2002: 149; and Hoey 2002). Where ambiguity occurs ‘by accident', it is raw material for the ‘humorous cuttings’ pages that many newspapers invite their readers to contribute to. An example spotted recently is this from the Guardian cuttings column: 
Volunteers at a Worcester charity shop have walked out after their manager was allegedly suspended over a poster hanging in the window. 
The humour here depends on the apparent ambiguity of suspended, which can mean both ‘hanging from a point’ and ‘temporarily removed from work'. Concordance lines for ‘suspended over’ (below) show that both meanings of suspended are represented. Thus the sequence ‘suspended over', without more co-text, is indeed ambiguous. The two meanings are distinguished by what follows over: a physical location or object (often water) in sense 1; a claim, allegation or crime in sense 2. They are also distinguished by subject: an inanimate object, or a person identified by name in sense 1; a person identified by role in sense 2. In the twenty (randomly selected) concordance lines displayed there are no exceptions to this. In the Guardian cutting, however, the subject is a person identified by role and the prepositional phrase indicates something that appears to be an object. This leads to the ambiguity and hence to the humour. (The synonymy of suspended and hanging is another factor, of course.) 
of greens. Boat-shaped planters are suspended over an emerald pool and, 
his deputy, Volodymyr Dyakov, were suspended over the crash of a Russian 
they consist of a shoe frame suspended over a 6&inch; high plastic 
high school students have been suspended over a schoolyard bashing at a 
Channing woke up in her bedroom suspended over the water, the center of 
since the bank's operations were suspended over allegations of fraud. Mr 
Stoke Mandeville Hospital have been suspended over a waiting list scandal. 
insisted Major Ingram will not be suspended over the insurance investigati 
Ministry staff. Five workers were suspended over the photos - including 
Last week, a 16-year-old pupil was suspended over alleged cannabis dealing 
members of his force have been suspended over the death. <hl> Horse and 
A red paper accordion bell is <p> suspended over the table. Nate bumps it 
low in such a way that the dough is suspended over the coals. The bannock is 
wax into a double boiler or a basin suspended over a pan of hot water and st 
where two ward parties have been suspended over claims that members worke 
of Rune Hauge, the agent currently suspended over his role in the George 
Morgan. <pg> 6 </pg> <hl> Officer suspended over shooting </hl> <date> 
<pg> 2 </pg> <hl> Police chief suspended over fatal shooting </hl> 
crash-landed into its facade and suspended over the streets. Blurring the 
melt the white chocolate in a bowl suspended over simmering water. Once 
The writers cited above, while they observe similar phenomena, are not united in their interpretation of them. Comparisons have been made, for example, between Fillmore's FrameNet and Levin's verb classes (Baker and Ruppenhofer forthcoming), and between Levin's classes and Francis et al's meaning groups (Hunston and Francis 1999: 142-146). Stubbs (2001, 2002) notes that a range of apparently overlapping work exists and appeals for a more focused discussion of the similarities and differences. In this paper I want to investigate one pair of research outcomes which, at least at first glance, seem remarkably similar. These outcomes are: frame constituent annotations as produced by researchers at the University of California Berkeley; and local grammar annotations as produced by researchers at the University of Birmingham. To observe their similarity, consider the following: 
FrameNet website: ‘difficulty’ 
ACTIVITY DEGREE ‘NODE’ EXPERIENCER Longer communications with the computer are extremely difficult for novice users. 
‘NODE’ ACTIVITY It Was difficult getting motivated on a night like this. 
343
Hunston and Francis 1999: ‘evaluation’ 
EVALUATED ENTITY EVALUATIVE CATEGORY AFFECTED ENTITY Life Is difficult for males with a family in tow. 
EVALUTIVE CATEGORY EVALUATED ENTITY It Was pretty difficult reading into a man's mind. 
There are differences in terminology here, but the basic concept appears to be the same. In each case, the sentence is parsed using a set of ‘frame elements’ (Baker et al 1998) or ‘meaning elements’ (Hunston and Francis 1999). Those elements are not identical, but they are by no means incompatible. In each case, furthermore, the sentences are among those that have been chosen to represent the behaviour of the word difficult as it appears in a large corpus. Is this a case of two minds with a single thought? Of two versions of the wheel being invented on either side of the Atlantic? This paper seeks to answer these questions. 
2. Methods and assumptions 1: frame or function? 
Accounts of the FrameNet method and its assumptions appear in a number of papers, notably Lowe et al (1997) and Fillmore and Atkins (1992), as well as in the book published on the FrameNet website (www.icsi.berkeley.edu/~framenet). This method begins with individual lexical items (‘words'), and argues that every word brings with it a set of associated concepts that might be related to a ‘script’ (as in schema theory and other models of representing real-world knowledge computationally). For example, ‘restaurant’ brings with it a frame that includes a menu, waiters, choosing, eating and paying for food and so on. The FrameNet project involves: identifying frames; identifying the elements associated with each frame; identifying the lexical items associated with each frame (different senses of words might be associated with different frames); and mapping the frame elements on to example sentences containing the target lexical items, obtained from several large corpora. To quote from Lowe et al (1997: 2-3): 
‘In frame semantics we take the view that word meanings are best understood in reference to the conceptual structures which support and motivate them. We believe, therefore, that any description of word meanings must begin by identifying such underlying conceptual structures. Frames have many properties of stereotyped scenarios – situations in which speakers expect certain events to occur and states to obtain. In general, frames encode a certain amount of ‘real-world knowledge’ in schematised form.’ 
Some fairly simple examples include the commercial transaction frame (Fillmore and Atkins 1992: 79). Elements from this frame, such as BUYER, SELLER, GOODS and MONEY map on to the typical uses of verbs such as buy, sell, charge and spend; for example: 
BUYER buys GOODS from SELLER for MONEY 
SELLER sells GOODS to BUYER for MONEY 
SELLER charges BUYER MONEY for GOODS 
BUYER spends MONEY on GOODS 
Another simple example is the health frame. Elements are mapped on to sentences as follows (Lowe et al 1997: 7): 
HEALER BODYPART TREATMENT The doctor treated my knee with heat. 
HEALER DISORDER The doctor Cured my disease. 
and so on. 
A more complex example is the risk frame. Although for many frames, intuition and examination of examples will be sufficient to derive the frame elements, for the risk frame Fillmore and Atkins (1992, 
344
1994) use categories developed in decision analysis. This enables them to identify in a very clear way the three uses of risk followed by a noun (risk your life; risk the climb; risk an accident), in which different frame elements are realised by the noun: a ‘valued possession’ (your life), a ‘decision’ (the climb), or a ‘harm’ (an accident). 
I shall now turn to an approach to semantic parsing that is very different: Barnbrook's local grammar of dictionary definitions (Barnbrook 1995, 2002; Barnbrook and Sinclair 1995, 2001). 
Barnbrook's corpus was the set of definitions in the Collins COBUILD Students Dictionary. These are, on the whole, one-sentence definitions that use the COBUILD ‘full-sentence’ defining style. COBUILD definitions are notable because they encode, albeit somewhat cryptically, a lot of information about how a word is used as well as what it means. In some cases it encodes what the user of a word means instead of what the word means. Barnbrook (2002: 7) cites these examples: 
If you call someone a prat, you mean that they are very stupid or foolish. 
If someone calls someone else a bastard, they are referring to them or addressing them in an insulting way. 
Barnbrook notes that both these definitions treat the words prat and bastard as terms of judgement rather than of description. Both use if rather than when, and include the metalinguistic you mean that or they are referring to them. In other words, the words prat and bastard are not described as having an inherent meaning at all. In addition, as Barnbrook notes, the word prat is described as a word that ‘you’ (the reader of the dictionary) might conceivably use, whereas bastard is reserved for the enigmatic and much more foul-mouthed ‘someone'. Barnbrook's program parses such definitions into elements which have names that are specific to the act of defining. For example, a parse of the first sentence above might be: 
Hinge Projection subject Projection verb Projection complement Article Headword Projection subject Projection structure explanation If you call someone A Prat you mean that they are very stupid or foolish. 
What is noticeable here, in contrast to, say, FrameNet, is that the motivation for parsing this sentence in this way is based on the function of the sentence, not the words it has in it. Compare FrameNet: 
He had called her a princess (from the entry for ‘call') 
SPEAKER NODE ENTITY? NAME He had called Her a princess 
[Note: on my screen, the colour given for her had no code, but ‘entity’ (a different colour) seems the most consistent labelling. ] 
The frame here is ‘name bearing', and this sense of the verb CALL is assigned to that frame. There is no similarity between the labels ‘NAME’ and ‘headword', of course, because the function of If you call someone a prat is quite different from that of He called her a princess. The distinction is between what in systemics (Martin 1992) is called the Modes of the two sentences. The first one constitutes the speech act of defining/naming, whereas the second one reflects the speech act of naming/defining. This may be illustrated further by FrameNet's analysis of DEFINE, in a sentence which reflects on the action of defining: 
…we have defined early mortality as appearing at less than thirty days. 
COGNIZER NODE ITEM CATEGORY We have defined early mortality as appearing at less than thirty days. 
Compare this with a parsed, constitutive, definition (after Barnbrook): 
The mortality in a particular place or situation is the number of people who die. 
345
Article Headword Qualifier Hinge Article Superordinate Discriminator The mortality in a particular place or situation Is The number of people who die. 
Following the constitutive/reflective distinction further, here is FrameNet's analysis of a sentence including the word DEMAND, which reflects on the activity of ‘demanding': 
Tom Dale … demanded a rethink at yesterday's council meeting. 
Tom Dale = SPEAKER 
a rethink = MESSAGE 
It is difficult in an unannotated corpus to find sentences that encode constitutive demands,1 but here are some based on the word rethink, that is, lines that might be reported as ‘demanded a rethink':2 
adverts. Perhaps there should be a rethink there with pictures of tourists 
t help thinking that with a bit of a rethink, extra resources, a promotional 
can return to first principles and rethink the role of the state. Discharg 
said. `We have to have a complete rethink otherwise the whole thing will 
task. <p> And yet if you do rethink your company's business design, 
be that we've got to fundamentally rethink how we face these kinds of 
as Cairns points out, we must `rethink the meaning of societal 
are not the answer. Managers must rethink their roles and relationships f 
Page 22 <hl> We need a radical rethink on schools and hospitals; Opini 
Growth Pact. Yet without a radical rethink by policy makers of their fisca 
occupation like that--should really rethink its attitude toward its people. 
mankind must do nothing less than `rethink our way of life." Mr. Lovejoy 
of abusers. We must, therefore, rethink our strategies for punishing or 
and cinemas get smaller. Unless they rethink their role, at some point early 
gun control laws and to a thorough rethink of all our legislation relating 
unsupervised. Society needs to rethink how to provide fresh opportunit 
bogus. Fine. Mr Howard might need to rethink those referendum questions. May 
must act, and it will have to rethink its support for the motor car. 
bits? Quite. But Boots will have to rethink their men's shaving ranges as, 
school of thought, you m ay want to rethink your purchasing habits. Today 
hair for years, this is the time to rethink your shade. `Many women start 
from your diary, perhaps you need to rethink your mania. For the majority 
Using Barnbrook's terminology, there are a number of types of demand here, which might be listed as follows: 
‘Modal’ + verb 
adverts. Perhaps there should be a rethink there with pictures of tourists 
can return to first principles and rethink the role of the state. Dischargin 
gus. Fine. Mr Howard might need to rethink those referendum questions. Maybe 
its? Quite. But Boots will have to rethink their men's shaving ranges as, 
school of thought, you may want to rethink your purchasing habits. Today, 
‘Modal’ + support verb + noun 
said. `We have to have a complete rethink otherwise the whole thing will 
If/unless + verb + outcome 
And yet if you do rethink your company's business design, the result is a liberating experience. 
Unless they rethink their role, at some point early in the 21st century cinemas will find themselves duplicating a domestic experience… 
I can't help thinking + with + noun + outcome 
I can't help thinking that with a bit of a rethink…it would double its sale. 
This is the time to + verb 
If you've been lightening your hair for years, this is the time to rethink your shade. 
1 The same problem exists for finding constitutive definitions. Barnbrook, in using a dictionary database, effectively has an annotated corpus. For the identification and parsing of definitions in an unannotated corpus, see Pearson (1998). 
2 Concordance lines for the item rethink were selected and edited to remove all except those which represented a directive speech act. 
346
For these examples we might propose the following frame/semantic elements: Hinge; Modal; Actor; Action; Outcome (positive or negative). 
Parsed examples would look like this: 
ACTOR MODAL ACTION OUTCOME We have to have a complete rethink otherwise the whole thing will slide into total oblivion Mr Howard might need to rethink those referendum questions. This is the time to rethink your shade. there should be a rethink there 
HINGE ACTOR ACTION OUTCOME If you do rethink your company's business design the result is a liberating experience. Unless they rethink their role cinemas will find themselves duplicating a domestic experience. I can't help thinking that 
with a bit of a rethink it would double its sale. 
Of course a complete local grammar of ‘demand’ would necessitate a much more detailed study, probably restricted to a single register. The point to be made here, though, is that treating the sentences as constitutive or reflective gives quite different results; one leads to an analysis based on the meaning of a word, the other leads to an analysis based on the function of the sentence. Reversing this is also true: an analysis based on the meaning of a word treats the sentence as reflective; an analysis based on the function of the sentence treats the sentence as constitutive. 
3. Methods and assumptions 2: frame or phrase? 
I would like now to turn to the work which awoke my own interest in local grammars. As with FrameNet, the starting point for this work was individual words, and the entries for them in dictionaries. The initial purpose of the work was not, however, to produce descriptions within semantic frames. The steps from dictionary entry to local grammar are set out below: 
COBUILD: from lexical item to local grammar. 
1. In line with the observation that words occur in sequences, and that meanings are distinguished by those sequences, words are given a grammatical coding. This is very ‘surface’ grammar and is intended for transparency, comprehensiveness and ease of use by learners. The codings are used in CCED. 
2. Building on key observations by Gill Francis (1993) that most grammatical contexts have lexical restrictions, the words that share patterns are collected, in two books (Francis et al 1996, 1998). Different senses, as given in CCED, are distinguished. The lists are divided into ‘meaning groups'. These are not motivated by semantic theory, as such, though some distinctions do owe something to other researchers. For example, verbs with the pattern V that are divided into ‘saying’ and ‘thinking', reflecting Halliday's distinction between verbal and mental processes. In the large V n category, distinctions are made on the basis of how participant roles map on to clause elements e.g. ask a question; ask permission and ask your mother are in separate meaning groups, further grouped under ‘communication'. In some places, information is given about alternations, although these do not form the basis of classification. For example, in the pattern V n with n, verbs which also occur with V n prep (e.g. daub the walls with mud; daub mud across the walls) are mentioned and listed, but they occur in a group with other verbs that do not show this alternation e.g. engrave, furnish, stencil and trim. 
3. To help teachers find their way around the verbs book, we put in what we called a ‘meaning finder'. This collected together similar meanings as they occurred through the book, and so in part compensated for the fact that many words and groups occurred several times in different sections. Again, the categories were not theory-driven but were simply a common sense grouping of 
347
frequently-mentioned meanings. Some of the categories are: ‘Attacking and doing harm'; ‘Beginning, continuing and ending'; ‘Bodily functions and movements'; ‘Changing'; ‘Changing something or changing its state'; ‘Fighting and competing'; ‘Giving, getting and paying for things'; ‘Communication'; ‘Directing something at or towards someone or something'; ‘Eating drinking and smoking'. (Unfortunately there was no time to complete a similar ‘meaning finder’ for the nouns and adjectives book.) One of the things that this activity brought to our attention was that that meanings we were identifying were reflective rather than constitutive. Comparing our work with that of Wilkins (one of the pioneers of the notional approach to syllabus design for language teaching) we noted (Hunston and Francis 1999: 118-123) that Wilkins focuses on language in constitutive mode whereas our concern with lexical items, and specifically with verbs at that point, leads us to focus on reflective mode. For example, whereas a notional/functional syllabus might include an item such as ‘giving instructions', our meaning finder has a category of ‘telling someone to do something', but this in fact means ‘talking about telling someone to do something’ – examples would include She told him to go home rather than You must go home. Our lexical item-based categories were leading us towards a FrameNet model rather than a Barnbrook one. 
4. From the meaning-finder it was a short step to the proposal to map elements of meaning on to pattern elements. When all the relevant patterns of all the relevant lexical items had been parsed in this way, this was to be called a local grammar. (We used Barnbrook's term without at the time recognising the essential difference between constitutive and reflective.) Because the relation between meaning element and pattern element remains constant, it was hoped that this approach could lead to automatic meaning extraction. However, the mapping of meaning element on to pattern element depends on both the pattern and lexical item that is the node word. For example, the meaning elements relating to a give word may be arranged differently depending on the pattern: 
Word: nuisance (from Hunston and Sinclair 2000: 99) 
Evaluation carrier Evaluative category Thing evaluated it v-link n to-inf It was a damn nuisance to have to put on new clothes and go out. 
Thing evaluated Evaluative category Person affected v-link N for n They turned out to be a nuisance for match anglers. 
Alternatively, a single pattern may have a different mapping of meaning elements depending on the node word: 
Pattern: V n to n (Francis et al 1996: 427) 
Reason for approval Object of approval Person who approves V N to n These qualities recommend Him to the electorate 
Verbs: commend, endear, recommend 
Person who approves Object of approval V N to n The company wants to Attract Investors to something new 
Verbs: attract, draw 
Thus, an approach that started with grammatical patterning and came to meaning second has led us to much the same place as an approach that began with meaning and with frame elements. If nothing else, the two approaches confirm each other. It is time now, however, to look at some of the conceptual differences between the two approaches, before embarking on a more detailed comparison of how the two approaches deal with the particular topic of evaluation. 
348
4. Theories of meaning 
The theory of meaning used by the Framenet researchers is that meaning belongs to a word: they refer to themselves as ‘lexical semanticists’ (Fillmore and Atkins 1992: 76). They specifically set themselves in opposition to those embracing semantic field theory and comment that ‘a word's meaning can be understood only with reference to a structured background of experience, beliefs, or practices, constituting a kind of conceptual prerequisite for understanding the meaning’ (ibid: 76-77). Another approach, espoused by Sinclair, Teubert and others, however, claims that meaning belongs not to the word but to the phrase.3 A corollary of this is that a word does not inherently possess a meaning, but that meaning is contingent upon immediate context. 
To illustrate this I would like to take the example developed by Stubbs (2001) from a comment by Fillmore – a ripe old age – and extend it a little further. Here are a number of sets of concordance lines, each one illustrating one use of the word age: of age, at poss age, a ripe old age and the ripe old age of. The phrase of age turns out to occur predominantly in two sequences: COME of age and months/years of age, and these are shown separately in sets 1a and 1b below. 
Set 1a: come of age 
People said that the industry "came of age" in 1983 when the nominees includ 
boomed because the adults who came of age in the '50s and '60s, which was t 
May on the day Sunday racing came of age in Britain, looks set to complete 
us," Walsh said. `We need to come of age in these sessions." That's puttin 
seen diamond film technology come of age. By the mid-1980s, a few research 
</b> <p> Cyberplonk has come of age. Supermarkets are selling wine ov 
<p> Kinnock generation comes of age <p> Charles Clarke's elevation t 
new life to the phrase `a coming-of-age drama". Who knows what mainstre 
long-held myths about the coming of age of a young hero who must go thr 
based on the novel. It's the coming-of-age story of a young lesbian reared 
a contract promptly upon coming of age, or it will be enforceable agai 
of coming out and coming of age. A beautifully written and, in 
Set 1b: years of age 
babies show at 4 or 6 months of age is correlated positively with IQ 
back line averaged only 20 years of age, but one which eases the pressure 
for children aged under three years of age, it is generally safer for them t 
enough for a man nearing 65 years of age. <p> I've sold out to the young 
East on boys seven to ten years of age to produce not only harem eunuchs 
of children under ten years of age was 1.63 for Italian children, we 
take part. Persons under 18 years of age are ineligible to play. No purcha 
an ex-serviceman at nineteen years of age <M01> <tc text=tuts> <M02> 
WAIS-R&symbequal;91.8, 55-64 years of age: WB&symbequal; 87.6, WAIS&symbequ 
t until about seven to eight years of age that most children can do laces u 
until your child reaches 4 years of age. They have passed the safety 
Set 2: at poss age 
worrying about being mugged. At my age it's always reassuring to know th 
he grows older. <p> But perhaps at his age your son is too young to manage 
her many previous boy-friends. At her age it would be impossible not to hav 
and The Cardigans came on. At his age he should be wearing cardigans no 
But why go for all that stress at his age? He might be popular now, but hal 
But Parsons admitted: `Even at my age this competition is special. Three 
Oh, no. No. I don't think--at my age, what does it matter? And I'm--I'm 
that possibility, particularly at his age. But when I hinted at some person 
was watching with interest. At her age, mid-twenties, we must have looke 
sometimes feels depressed, at your age. Look, see you Saturday - the Do 
to defend my homeland, even at my age." <p> Strangely, Mr Matveyev is le 
t want to get married yet, and at his age I believe that to be very wise. W 
Set 3: a ripe old age 
a breakthrough cure to see him into a ripe old age. His predicament invites 
living, have managed to live to a ripe old age. IN the Smart Money segme 
find out what makes people live to a ripe old age. There are quite a lot in 
we'd pick ones who lived to a ripe old age." 20 MAYBE QUT engineerin 
object. Whoever he was, he lived to a ripe old age," he said. `Ninety eight! 
agents. If Jesus had lived to a ripe old age and died happily, the 
a reasonable chance of living to a ripe old age. Today, at least in 
and people have been living to a ripe old age, and I just don't think 
Nor does cider inhibit progress to a ripe old age. Uncle Tom's Cabin has a 
life's pleasures but survived to a ripe old age; at least he can take 
was between 14 and 16 years old, a ripe old age for the species. It would 
3 A third approach, adopted by Halliday, would suggest that meaning belongs to the choice between items, not to the items themselves. 
349
Set 4: the ripe old age of 
Our eldest son, James, has, at the ripe old age of 41, been given his ver 
of guacamole and yoghurt. Even at the ripe old age of 26, when I'm driving o 
Saint-Saens died in 1921, at the ripe old age of 86, crowned with all t 
of so much official patronage at the ripe old age of 67. He has become the 
emerged from a career slump at the ripe old age of 18 to take out another 
years and he is still working at the ripe old age of 76. `I don't look at t 
from Alois Klein. In 1979, at the ripe old age of 21, he opened his own 
hall generation. But now, at the ripe old age of 67, Lonnie is back. He 
debut as a teenager. <p> Now, at the ripe old age of 23, he can speak with 
was going to be a bike star -at the ripe old age of four. <p> That was whe 
become a big-time winner -- at the ripe old age of 71. Jack, a retired Po 
lead Swindon's attack again -- at the ripe old age of 40. The Robins' boss c 
until his death 35 years later at the ripe old age of 87. <p> Nikko never 
having passed away in 1992 at the ripe old age of 96. Nevertheless, his 
matter much now that Firmage, at the ripe old age of 28, has discovered The 
8.95), who died recently at the ripe old age of 87, believed that wine 
started playing back in 64 at the ripe old age of ten, and it shows 
The ‘meanings’ of these phrases seem to be: 
Set 1a: come of age means broadly ‘achieve adulthood or maturity'. It has three more specific meanings: (of people) attaining legal majority; (of people) achieving emotional maturity; (of institutions) coming to be established in its social context. 
Set 1b: years of age indicates how old someone is. 
Set 2: at poss age is used to set a situation in the context of what is expected of people at certain stages of their life. For example, someone is expected to react in a certain way because they have lived a certain number of years, or someone is too young or too old to be required to do something. 
Set 3: a ripe old age indicates that someone (or an animal) has lived longer than might be expected, and that this is a good thing. 
Set 4: the ripe old age of is used either with a high number to indicate the same as set 3, or with a low number to comment on how young someone is, or in more general terms to draw attention to and comment on someone's age. 
I take it that any theory of meaning would have to treat come of age as a single lexical item. Leaving that aside, if we ask ‘what does age mean', it would be possible to argue that it has two meanings: a neutral, measurement-of-life meaning, as in four years of age, and an affective, comparative meaning of something like ‘number of years compared with a norm', as in at your age, live to a ripe old age, at the ripe old age of 24. An alternative view is that age, as a single lexical item, really has no meaning, but that x years of age, at your age, live to a ripe old age etc do have meaning, some of those meanings being comparative and affective. 
The relevance of this for identifying and annotating frame elements might be illustrated with a couple of examples. The first is an example quoted at the beginning of this paper: 
The Governor ‘remains comfortable that no innocent person has been executed.’ 
It was said there that in this case comfortable takes on the meaning of ‘confident', by analogy with confident that. It would be possible to say (consistent with FrameNet) that comfortable evokes two frames, one of which is the ‘certainty’ frame that confident also evokes. But it is also possible to argue that what evokes the ‘certainty’ frame is not a word but a sequence – ‘v-link ADJ that’ – with certain adjectives filling the ADJ slot. The sequence might be expressed as: 
link verb certain 
comfortable 
confident 
convinced 
doubtful 
dubious 
persuaded 
positive 
satisfied 
sceptical 
sure 
uncertain 
unconvinced that-clause 
350
Outside of this sequence, comfortable does not evoke the ‘certainty’ frame. My proposal is, then, that the unit of meaning identification is not the word, nor the pattern, but the word-pattern combination. 
To explain Fillmore and Atkins’ own response to cases such as this I take again the word that they present in such impressive detail: risk. In their 1992 paper, Fillmore and Atkins list the various frequently-occurring syntactic environments occurring with the noun or verb risk – in Hunston and Francis's terms, they list the patterns of risk – and they specify which frame elements occur where in each such environment. They then introduce some other usages which are less easily accommodated within the frame because, in those, risk takes on the meaning of other words and no longer ‘really means’ risk. To quote Fillmore and Atkins (1992: 96): 
‘On developing the preceding categorization of the segments of sentences built around the verb RISK, we found a number of examples that did not lend themselves to a direct interpretation in terms of what we took to be the RISK schema. These words, in these contexts, seemed to mean more than just RISK.’ 
They go on to suggest that a word can overlap in meaning with, and can ‘inherit’ grammatical properties from, another word. This argument is rather similar to the one made at the beginning of this paper, where it was suggested that some words take on the meanings of other words when they occur in the same patterns as those words. These examples were given: 
It was big of you to take the risk. (H&F 1999: 105) (= good of you) 
A Labour MP leaked him a copy of the committee report. (= gave him a copy) 
Fillmore and Atkin's examples with risk include: 
He feared to risk his two precious flattops to enemy submarine attacks. 
Cf Dr James Smith… exposed his only daughter to smallpox in order to prove the effectiveness of Jenner's remedy.4 
In pattern terms, both risk and exposed have the pattern V n to n, with the noun in the prepositional phrase indicating ‘the threat against which something is unprotected’ (Fillmore and Atkins 1992: 97). 
Roosevelt risked more than $50,000 of his patrimony in ranch lands in Dakota Territory. 
Cf Clive Sullivan…invested L2.5million in the club. 
The noun following in is an ‘invested-in object'. 
He's likely to risk a week's salary on a horse. 
Cf Would Hoberman encourage poor pensioners to bet their savings on his company? 
The noun following on is ‘what is bet on'. 
Thus Fillmore and Atkins overcome their problem of a word being used with elements that do not ‘belong’ to it from the point of view of its frame, but the problem arises only because they assume that a word such as risk has an inherent ‘real’ meaning which is then extended metonymically. In Francis et al 1996, 1998, by contrast, the word risk (noun or verb) is treated as belonging to different meaning groups depending on the pattern it is used with. In other words, we regard the meaning as being triggered by the word-in-pattern rather than by the word itself. Examples are: 
‘How likely or unlikely something is’ 
Noun that-clause chance 
danger 
guarantee 
impossibility 
likelihood 
odds 
off-chance 
possibility 
probability 
question 
risk 
4 I have added the second example in each case, using the words (such as expose) suggested by Fillmore and Atkins. 
351
Noun of n/-ing chance 
danger 
hope 
likelihood 
possibility 
prospect 
risk 
threat 
‘Indicating danger and its source’ 
Noun from n/-ing danger 
hazard 
risk 
threat 
‘Indicating danger and the possible victim of harm’ 
Noun to n/-ing danger 
hazard 
menace 
risk 
threat 
A similar argument is used by Hunston and Francis (1999) in discussing the differences between their approach and Levin's. They take issue, as do Baker and Ruppenhofer (forthcoming), with the evidence for some of Levin's categories, but then argue that this is more than a matter of detail. They suggest that ‘allowing all patterns of a verb to be considered, instead of just a few, might militate against the consistent formation of semantic groups’ (H&F 1999: 145). For example, although the verb bite usually belongs in a group with eat and so on, when it is in the pattern V into n it belongs with bore, dig and drill. That is, although it still has the meaning of ‘break off food with the teeth preparatory to ingestion', a different aspect of the activity is being focused on. In other words, if it is not the word but the sequence that has meaning, and if therefore a word has a slightly different meaning depending on the sequence, the semantic groupings of words will be different for each of the sequences or patterns which the words are part of. 
5. A local grammar of evaluation – again 
In this part of the paper I wish to consider the topic of evaluation, and how this kind of meaning may be coded. In my own early work on evaluation in academic discourse I regarded evaluation as comprising three discourse functions: the status function (what kind of epistemic entity is being evaluated); the value function (how good or bad it is); and the relevance function (the discoursal significance). Of these, the one that corresponds to what most people think of as ‘evaluation’ is ‘value', and that is the phenomenon that I will give the name of ‘evaluation’ in this paper. 
Evaluation is a prime candidate for frame analysis because it is essentially a semantic rather than a grammatical resource, as Martin (2000) points out in his model of APPRAISAL. Martin proposes that APPRAISAL can be realised using three sets of semantic resource: feelings and emotions (AFFECT); social (dis)approval (JUDGEMENT) and social value (APPRECIATION). These correspond very roughly to the following FrameNet frames: Emotion_active or Emotion directed (AFFECT); Morality evaluation, Social behaviour evaluation and Social interaction evaluation (JUDGEMENT); Quality evaluation (APPRECIATION). The correspondences are not exact, however. For example, the adjective splendid belongs to the Quality evaluation frame, and normally realises APPRECIATION. The sentence Amazon ants are splendid at enslaving other ants is included by FrameNet in the Quality evaluation frame – as it has to be, because frame membership is entirely dependent on lexical item – but is coded as JUDGEMENT (sub-set: capacity) in Martin's model.5 
5 I am grateful to Peter White for confirming this interpretation of Martin. 
352
Once again, then, there seem to be problems that arise when a word alone, rather than a word-pattern combination, is seen as determining the frame. For example, in the frame ‘Social interaction evaluation’ there is a frame element of Grounds which occurs in a prepositional phrase beginning with about. It is described in the FrameNet website as something to which the evaluee's behaviour responds. Examples are given with the adjectives nice and kind (e.g. He was nice about the accident; She was kind about my work.) Francis et al (1998: 418, 420) suggest two relevant meaning groups of adjectives which are sometimes followed by a prepositional phrase beginning with about. The first is glossed as ‘someone talks about a person or thing in a way that is complimentary to them or insulting to them'. Adjectives include abusive, acerbic, affectionate, bitchy, blunt, catty, caustic, censorious, charitable, cheeky, churlish, complementary, critical and cruel, as well as kind. Note that many of these, such as affectionate and cruel do not mean ‘talk in a particular way’ in most contexts. The second is glossed as ‘someone reacts to a situation in a way that is judged to be good or bad’ and includes adult, beastly, brave, brilliant, cool, excellent, fine, foolish and funny, as well as kind. This group notably includes adjectives in their less frequent senses e.g. adult, mature, heavy and sweet. It also includes adjectives which FrameNet lists under Quality evaluation (e.g. marvellous) as well as those such as kind which FrameNet includes in Social interaction evaluation. The point is that words are not fixed in their meaning. In the case of evaluative words, the precise referent for the evaluation is determined by the pattern used more than by the adjective used. 
At this point, then, I am going to combine some of the observations of Martin and of FrameNet, with some of my own observations. From Martin I am going to take part of the category of semantic resources which he calls AFFECT. In the FrameNet tradition, however, I am going to treat (part of) AFFECT as a frame, with frame elements. Following my own tradition I am going to take word-pattern combinations as the starting point of frame description, and am going to deal here only with adjective patterns. 
I shall also propose that the distinction between reflective and constitutive expression of AFFECT is recognised. One of the complexities of this semantic area is that emotional response may be indicated as a quality of the responder, as in Everyone in the school is distressed to hear of this tragedy or as a quality of the thing evaluated, as in …after the distressing events of 1887…. The first of these is reflective evaluation (it attributes evaluation to everyone in the school) while the second is constitutive (it avers an evaluation of the events).6 Martin treats both of these as equivalent, whereas the FrameNet system places the first under Emotion_active and the second under a different frame: Subject_stimulus. The problem, for me, is that when AFFECT is constitutive it tends to come very close in meaning to other forms of evaluation. To illustrate this, consider these examples which are given different codings by Martin, reflecting the different meanings of the adjectives concerned (emotional reaction, social sanction, or social value): 
It's annoying to have people clicking their fingers at you to get your attention. (AFFECT) 
I thought it would be selfish to marry if I were going to be killed. (JUDGEMENT) 
It is pathetic to see such stupidity on show. (APPRECIATION) 
To me, these differences are secondary to the fact that all these examples constitute an evaluation of the situation indicated by the to-infinitive clause, and all are different on a primary level from a description of emotion as in Everyone in the school is distressed… In what follows, therefore, I shall deal with reflective AFFECT only. 
In the following tables, the first column gives the adjective pattern (identified from Francis et al 1998) and the second column gives a list of meaning groups in Francis et al (1998) whose constituent adjectives in the pattern identified realise reflective AFFECT. The third column proposes the frame or meaning elements (in each case including ‘Experiencer’ and ‘Emotion') expressed in ‘pattern’ terms, and the fourth column gives examples. Where the third column gives alternative elements (usually Cause/Target) the fourth column gives an example of each. 
6 For the averral-attribution distinction, see Hunston (2000). 
353
The first table comprises patterns with prepositions. 
Pattern Meaning groups Elements Examples ADJ about n ‘passionate & cool’ 
‘happy’ 
‘unhappy’ 
‘philosophical’ 
‘nervous’ 
‘angry’ 
‘curious’ 
‘cynical & serious’ Experiencer is Emotion about Cause/Target She is increasingly nervous about her future… 
She did seem curious about why the dogs were wet… ADJ as to wh ‘curious’ 
‘worried’ Experiencer is Emotion as to Cause/Target I…was worried as to how my death would affect them 
We were curious as to why our father's family had black curly hair… ADJ at n ‘nervous’ 
‘angry’ Experiencer is Emotion at Cause/Target Some people looked quite horrified at the idea of reading a new book 
Paul is very angry at the way he has been treated ADJ by n ‘astonished’ Experiencer is Emotion by Cause …the British are exasperated by rising crime ADJ for n ‘eager’ 
‘guilty’ Experiencer is Emotion for Cause/Target I felt guilty for disturbing his solitude 
…I was really desperate for money ‘afraid’ Experiencer 
‘happy’ is Emotion for 3rd party He's had a great career and I'm happy for him ADJ in n ‘safe’ Experiencer is Emotion in Cause He was happy in his Apache life ADJ of n ‘fond & critical’ 
‘afraid’ 
‘tired’ 
‘desirous’ Experiencer is Emotion of Cause/Target We are proud of our achievements 
I was envious of their anonymity ADJ on n ‘keen’ 
‘optimistic’ Experiencer is Emotion on Target I'm not that big on religion ADJ over n ‘angry’ 
‘despondent’ 
‘worried’ 
‘enthusiastic’ 
‘jealous’ 
‘go mad’ Experiencer is Emotion over Cause We like an artist who is enthusiastic over talent in others ADJ to n ‘partial’ Experiencer is Emotion to Target As General Haig was wedded to his profession… ADJ towards n ‘sympathetic’ 
‘aggressive’ 
‘ambivalent’ Experiencer is Emotion towards Target I've always felt very affectionate towards Karen because… ADJ with n ‘angry’ Experiencer is Emotion with Target I thought I was angry with them but… ‘breathless’ Experiencer is Emotion with Emotion … was frozen with fear 
354
The next table comprises patterns with clauses. 
Pattern Meaning groups Elements Example ADJ that ‘surprised’ 
‘angry’ 
‘horrified’ 
‘glad’ 
‘anxious’ Experiencer is Emotion that Cause/Target He was angry that she had spoken to people… 
The people are terrified that … they might be killed ADJ wh ‘afraid’ Experiencer is Emotion Target They are afraid what their neighbours and children will think ADJ to-inf ‘astonished’ 
‘sorry’ 
‘delighted’ Experiencer is Emotion to Action/Phenomenon You've got to be very thankful to win once 
They were puzzled to find the kitchen door locked ADJ –ing ‘comfortable’ Experiencer is Emotion Action I felt good seeing Gideon…again 
The final table comprises patterns with it. 
Pattern Meaning groups Elements Example It v n ADJ that (only one group in the pattern) It makes Experiencer Emotion that Phenomenon It makes me sick that anybody should doubt my commitment It v n ADJ to-inf ‘angry’ 
‘sad’ 
‘sick & nervous’ 
‘happy’ It makes Experiencer Emotion to experience Phenomenon It makes me sad to see all the god work we have done devalued in this way 
As noted above, a number of examples and patterns have been omitted from these tables because they do not meet the criterion of including both Experiencer and Emotion: that is, they constitute rather than reflect evaluation. Some examples are given below, though there is not enough space here to develop this analysis further. In each case, emotion is expressed as a quality of the object rather than of the experiencer (the match is boring rather than I am bored): 
ADJ for n e.g. This is very distressing for Carol 
ADJ to n e.g. There is nothing more infuriating to an author than… 
v it ADJ that e.g. I find it offensive that I am being accused of giving advice in return for political favours. 
v it ADJ to-inf e.g. You might find it interesting to inquire about how your children get on… 
it v-link ADJ that e.g. It's interesting that she's never asked what he looks like. 
it v-link ADJ wh e.g. It's inexplicable why a teenage girl had careered onto the road like a toddler. 
it v-link ADJ what/how e.g. …it is appalling how much litter, bottles and food scraps are scattered over…our streets. 
it v-link ADJ when/if e.g. It's frustrating when people try to do things and are held up with red tape. 
it v-link ADJ to-inf e.g. It's annoying to have people clicking their fingers at you to get your attention. 
it v-link ADJ –ing e.g. …it must be frightening starting a downhill race at the top of a mountain. 
ADJ to-inf e.g. Such matches are boring to watch. 
6. The synergy of approaches 
This paper has represented a journey of discovery through some different approaches to the coding of meaning in discourse, comparing mainly the Berkeley and the Birmingham approaches to frames and local grammar and in part reflecting on the contribution of Martin's approach to APPRAISAL to the coding of evaluation. My attempts to grapple with similarities and differences has led me to an 
355
appreciation of the strong points of all the approaches mentioned here, and to a realisation of the importance of three factors: the role of frame elements in delimiting frames, Mode of discourse, and word-pattern, as opposed to word or pattern alone, as the site of meaning. 
As a final illustration of the possible synergy between approaches, I would like to reflect in detail on one of the FrameNet lexical entries: distinguish and to compare the corpus analysis that this entry represents with the entry for the same word in the Collins Cobuild English Dictionary.7 The FrameNet entry for distinguish appears to give more comprehensive information than CCED. CCED lists these patterns for the verb: 
V n from n distinguish item from item 
V between pl-n distinguish between two items / between item and item 
V pl-n distinguish two items / item and item 
V n distinguish something 
The first three patterns are used with two meanings of the verb (‘someone distinguishes one thing from another’ and ‘a quality distinguishes one thing from another') while the fourth is used with another meaning (‘someone could distinguish something from a background'). 
FrameNet, on the other hand, gives a large number of items which are found to follow distinguish. Expressing these in CCED terminology, they are: 
V n amongst n V n with n 
V n by n and passive equivalent (be V-ed by n) V n prep 
V n for n V from n 
V n in n V in n 
V n on n V on n 
There is insufficient space here to give a complete account of the corpus searches inspired by these sequences; I shall simply summarise some of the more interesting findings. 
A ‘new’ meaning of distinguish 
FrameNet mentions the sequence ‘distinguish … amongst…'. This was very rare in the BoE (2 lines), but the search for this revealed another pattern V among n, almost exclusively a US usage, as in 
…no guidelines to help the consumer distinguish among astringents… 
The sense of this is not ‘distinguish one thing from another’ but ‘separate one thing from the rest'. This sense, and its patterns, is not specifically given in CCED and is not recognised as a separate sense of distinguish in FrameNet. The ‘new’ sense also occurs with the pattern be V-ed by n, as in 
Kathak is a classical dance from the north of India, which is distinguished by intricate foot rhythms…[part of a list of dance styles] 
Thus the FrameNet information has helped to identify a new sense of the word. 
Adding another pattern to distinguish 
FrameNet identifies the sequence by n, which is not given as a pattern in CCED, but which probably should have been. CCED rarely codes passive patterns, on the grounds that all verb+object clauses are open to passivisation and that therefore the occurrence of passive patterns does not distinguish between lexical items. This may, however, lead to a neglect of passives in the compiling of local grammars. The passive version of the pattern V n by n has been mentioned above as a realisation of a ‘new’ meaning of distinguish. 
Recognising ‘non-pattern’ prepositional phrases 
Many of the sequences identified in FrameNet are not patterns in the CCED sense (there is no reason why they should be). I would suggest, however, that distinguishing non-pattern from pattern is useful in identifying frame elements. 
The prepositional phrase by n, as well as being a potential pattern element, can also co-occur with other patterns, as in Boot is easily distinguished from other cartoons by the garishly coloured faces of the 
7 This entry was chosen because I compared it with a study carried out by Woodward (2002) on expressions of difference. There is not enough space here to report on the comparison in detail. Unfortunately, the FrameNet entry for distinguish is not in its final form. 
356
main characters. It is in a sense a ‘free-floating’ phrase that needs to be labelled as a local grammar or frame element wherever it occurs. 
Other non-pattern prepositional phrases that realise additional semantic elements and can co-occur with other patterns include: 
with + noun (Keen-sighted people can distinguish them both with the naked eye.) A pattern parser should parse as V pl-n (with n) 
in + noun (He distinguishes in the book between an error, an illusion and a delusion.) Parse as V between pl-n (in n). 
on + noun (…difficulties in distinguishing on stylistic grounds alone between late Malaga and early Valencia lustre pottery.) Parse as V between pl-n (on n). 
Other instances of prepositional phrases mentioned in FrameNet are simply part of the relevant noun phrase and should not be parsed separately by a pattern parser e.g. 
in + noun (…we can distinguish three main classes in contemporary capitalist society) Parse as V pl-n. 
on + noun (Tedrow distinguishes arctic brown soils on well-drained sites, and tundra soils on wetter areas.) Parse as V pl-n. 
Identifying ‘non-canonical forms’ of patterns 
As Francis et al (1996: 611-615) point out, patterns can be altered by their grammatical context. FrameNet identifies a sequence V from n, but all examples of these in the BoE are non-canonical forms of V n from n, e.g. 
The ensuing symptoms are often difficult to distinguish from those of an acute attack of asthma. Parse as V n from n. 
There are a number of conclusions to be drawn from this. Firstly, I would suggest that the mass of information about context that emerges from concordance lines can be ‘tidied up’ by a consideration of pattern, so that not every possible sequence has to be accounted for by a separate parsing. Secondly, the detailed FrameNet work has suggested, but not accounted for, a sense of distinguish that is not found in the CCED but which does occur quite frequently in the current BoE (which has increased its US coverage). This has diagnostic patterns (notably V among n) and a slightly different co-occurrence of frame elements, in that only one Phenomenon (in FrameNet terms) is required. 
Finally, and most importantly, the study suggests that, as might be expected from corpus work, new insights are most usefully gained when observation is not hampered by assumption. Because the FrameNet approach does not make assumptions about what a pattern is (and is not), it includes information which a pattern approach would omit, which the pattern approach can then be used to organise. Because the pattern approach does not assume that the word has an inherent meaning, it allows a ‘new’ meaning to emerge when the patterns suggest it. 
References 
Baker CF, Fillmore CJ and Lowe JB 1998 The Berkeley FrameNet Project. Proceedings of the COLING-ACL, Montreal, Canada. 
Baker CF and Ruppenhofer J forthcoming. FrameNet's frames vs. Levin's verb classes. Proceedings of the 28th Annual Meeting of the Berkeley Linguistics Society. 
Barnbrook G 1995 The Language of Definition. Unpublished PhD thesis, University of Birmingham. 
Barnbrook G 2002 Defining Language: a local grammar of definition sentences. Amsterdam, Benjamins. 
Barnbrook G and Sinclair JM 1995 Parsing Cobuild entries. In Sinclair, JM, Hoelter M and Peters C (eds), Studies in Machine Translation and Natural Language Processing. Luxembourg, European Commission, pp 13-58. 
Barnbrook G and Sinclair JM 2001 Specialised corpus, local and functional grammar. In Ghadessy M et al (eds), Small Corpus Studies and ELT: theory and practice. Amsterdam, Benjamins, pp 237-276. 
357
Dang HT, Kipper K, Palmer M and Rosenzeig J 1998 Investigating regular sense extensions based on intersective Levin classes. 38th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics. 
Fillmore CJ and Atkins BT 1992 Toward a frame-based lexicon: the semantics of RISK and its neighbors. In Lehrer A and Kittay EF (eds), Frames, Fields, and Contrasts: new essays in semantic and lexical organisation. Hillsdale N.J., Lawrence Erlbaum, pp 75-102. 
Fillmore CJ and Atkins BT 1994 Starting where the dictionaries stop: the challenge of corpus lexicography. In Atkins BT and Zampolli A (eds), Computational Approaches to the Lexicon. Oxford, Oxford University Press, pp 349-393. 
Francis G 1993 A corpus-driven approach to grammar - principles, methods and examples. In Baker M et al (eds), Text and Technology: in honour of John Sinclair. Amsterdam, Benjamins, 137-156. 
Francis G, Hunston S and Manning E 1996 Collins COBUILD Grammar Patterns 1: Verbs. London, HarperCollins. 
Francis G, Hunston S and Manning E 1998 Collins COBUILD Grammar Patterns 2: Nouns and adjectives. London, HarperCollins. 
Hoey M 2002 Language as choice: what is chosen? 29th International Systemic Functional Linguistic Congress, University of Liverpool, July 2002. 
Hunston S 2000 Evaluation and the planes of discourse: status and value in persuasive texts. In Hunston and Thompson (eds), 177-206. 
Hunston S 2002 Corpora in Applied Linguistics. Cambridge, Cambridge University Press. 
Hunston S and Francis G 1999 Pattern Grammar: a corpus-driven approach to the lexical grammar of English. Amsterdam, Benjamins. 
Hunston S and Sinclair JM 2000 A local grammar of evaluation. In Hunston and Thompson (eds), 75-101. 
Hunston S and Thompson G (eds), 2000 Evaluation in Text: authorial stance and the construction of discourse. Oxford, OUP. 
Levin B 1995 English Verb Classes and Alternations: a preliminary investigation. Chicago, The University of Chicago Press. 
Lowe JB, Baker CF, Fillmore CJ 1997 A frame-semantic approach to semantic annotation. Proceedings of the SIGLEX workshop ‘Tagging Text with Lexical Semantics: Why, What and How?’ April 1997, Washington DC. 
Martin JR 1992 English Text: system and structure. Amsterdam, Benjamins. 
Martin JR 2000 Beyond exchange: APPRAISAL systems in English. In Hunston and Thompson (eds), 143-175. 
Mason O and Hunston S 2001 The automatic recognition of verb patterns: a feasibility study. COMPLEX conference, Birmingham, England. 
Pearson J 1998 Terms in Context. Amsterdam, Benjamins. 
Roland D, Jurafsky D, Menn L, Gahl S, Elder E and Riddoch C 2000 Verb subcategorization frequency differences between business-news and balanced corpora: the role of verb sense. Proceedings of the Workshop on Comparing Corpora. Held in conjunction with the 38th Annual Meeting of the Association for Computational Linguistics. 
Sinclair JM 1991 Corpus Concordance Collocation. Oxford, OUP. 
Stubbs M 2001 On inference theories and code theories: corpus evidence for semantic schemas. Text 21: 437-465. 
Stubbs M 2002 Corpus analysis: the state of the art and three types of unanswered questions. 29th International Systemic Functional Linguistic Congress, University of Liverpool, July 2002. 
Woodward R 2002 And now for something completely different: a corpus-driven local grammar of ‘difference'. Unpublished MA dissertation, University Of Birmingham. 
358

The development of the spoken corpus of Japanese learner English and the applications in collaboration with NLP techniques 
Emi Izumi1,3 Toyomi Saiga1,2 Thepchai Supnithi1,2,4 
Kiyotaka Uchimoto1 Hitoshi Isahara1,2,3 
Communications Research Laboratory, Japan1 
Telecommunications Advancement Organization of Japan2 
Kobe University, Japan3 
Information Technology R&D Division, NECTEC, Thailand4 
1. Introduction 
To keep up with the information-driven society, it must be one of the most important things to acquire foreign languages, especially English for international communications. In order to develop a computer-assisted language teaching and learning environment, we have been compiling a large-scale speech corpus of Japanese learner English, which provides a lot of useful information to construct a model of the developmental stages of Japanese learners’ speaking ability. In this paper, first, we will present the overview of this project by introducing the activities done so far such as its data collection procedure, annotation schemes including error tagging, and the development of the original software tool which makes data collection process easier. Secondly, we will explain how this corpus can be exploited for second language acquisition research or system development by introducing several experiments we have done so far, such as a linguistic analysis on how the English article system is acquired by Japanese learners of different proficiency levels and the attempts to automatic processing of learners’ language in collaboration with Natural Language Processing (NLP) techniques. 
2. SST Corpus 
First, we would like to give some details about our corpus named “SST Corpus”. We will do this first by explaining what the Standard Speaking Test (SST), which is the main speech resource of this corpus, is like and showing the data collection procedure such as transcribing, and tagging, then by introducing our original software tool, named the “TagEditor”, which was developed for transcribing and tagging this corpus. 
2.1 The characteristics of SST corpus 
The main characteristics of the TAO corpus are: 
  The corpus data is entirely based upon the audio-recorded data of the interview test, the “Standard Speaking Test”. 
  The interviewees are Japanese learners of English. 
  The data is divided into nine groups depending on the learner's proficiency levels based on the SST evaluation scheme. 
  Compared with other learners’ speech corpora, the scale of the corpus is relatively large (300 hours of data, totaling one million words) 
2.2 The Standard Speaking Test (SST) 
Now, we would like to give a general description of the SST. This 15-minute interview test was jointly developed for Japanese learners by the American Council on the Teaching of Foreign Languages (ACTFL) and a famous Japanese ELT publishers, ALC Press. This test is based on the ACTFL's Oral Proficiency Interview (OPI) exam. The SST is a face-to-face interview between one examiner and a 
359
test-taker. In most cases, the examiner is a native speaker of Japanese who is officially certified as a SST examiner. There might be some criticism to the effect that the interview data is biased because a non-native English speaker conducts it, but we consider that sometimes it must be good for making interviewees relaxed. The SST consists of five stages as shown in Figure 1 below. Stage1Warming up3-4 min.Stage2Picture Description2-3 min.Stage3Role playing1-4 min.Stage4Story telling2-3 min.Stage5Winding down1-2 min. 
The interview starts with an informal chat on general topics such as the interviewee's job, hobbies, family, and so on. From stage 2 to stage 4, the interviewee is asked to do three task-based activities, namely picture description, role-playing, and story telling. Each stage consists of two parts; a task part and a follow-up part. After the actual task part, the interviewer asks some questions which have to do with the task in the follow-up part. The interview ends with, again, an informal chat. All the interviews are audio-recorded, and judged by two or three raters based on the SST evaluation scheme (SST level 1 to 9). 
We consider the SST to be a very useful spoken resource, because most existing learner corpora consist of written language only. Another benefit of choosing the SST as the main resource of the corpus is that each file of data has specific information on the examinee's oral proficiency level, as assessed by the professional examiner. There are some developmental learners’ corpora available, but in most cases, they determine the learners’ proficiency level based on external factors such as school years. A comparison between sub-corpora based on school years sometimes is not reliable because it ignores other parts of the background of students, how they have been learning English and from what kind of teachers. On the other hand, the SST data contains more reliable information on learners’ proficiency levels, which will help to make comparative research based on proficiency subsections of the corpus more valid. (Tono, 2001) 
2.3 Transcribing and tagging 
There are some general rules for transcribing the speech data. Even though a word is mispronounced, it is transcribed with a correct spelling as long as the transcribers can understand which word the speaker produced. If acronyms are pronounced as sequences of letters, they must be transcribed as a series of upper case letters, which are separated by spaces. It is not allowed to use either Roman or Arabic numerals. All numbers must be transliterated as words. The transcribers are allowed to insert the phrase and sentence boundaries with commas and full stops on their own decision. Some information on non-verbal behaviors or concurrent events such as remarkable noises is also inserted. 
There are two kinds of tags used in this corpus; the basic tags for such as filled pauses or repetitions, and the error tags for the analysis of the learners’ errors. In this section, we would like to explain about the basic tags. The error tags will be explained in the section 2.4. 
The tags in this corpus are based on the XML syntax although there is one exception that crossings of different tags are allowed. There are many advantages of using the tags based on XML syntax. XML can clearly identify the structure of the text and it is quite beneficial when the corpus data is exploited for the web-based pedagogical tools or database as a hypertext. 
There are more than 30 basic tags and they are divided into four categories: tags for representing the structure of the entire transcription file, tags for the interviewee's profile, which is attached as a header of the file, tags for speaker turns, and tags for representing several phenomena in an utterance 

Figure 1. Five stages in SST 
360
such as fillers and repetitions. 
We have developed the original software tool for SST transcription, the “TagEditor”. We believe that it must be good to use an editor designed especially for transcribing and tagging because manual transcribing and tagging are time-consuming, and it is inevitable that the transcribers will make a lot of mistakes if they work on just a simple text editor. Another reason for developing the TagEditor is that since we use a non-standard XML style tagset, it would not be a good idea to use the existing format checkers for standard XML texts. We believe that the TagEditor improves the efficiency of transcribing. The system can be easily updated whenever the tagset is revised. 
Although the TagEditor is still under development, it is already accommodating a lot of functions. The transcribers can insert tags just by clicking on buttons. If the transcribers highlight the letters, then click on the button indicating a certain tag, the selected letters are wrapped with a tag. Even people who are not good at working on computers can easily do this. For those who are advanced computer users, it might be easier for you to input by keyboard than to click on a mouse. Tags can also be inserted with short-cut keys. Users can assign a key to each tag as they like. 
2.4 Error tagging 
The most remarkable difference between corpora of native and non-native (learners) speakers would be that a lot of errors are contained in learners’ corpora. It has been said that analyzing errors produced by learners is quite efficient for finding out the learners’ developmental stages and decide what is the best teaching method for them. In this project, we decided to analyze errors mainly by error tagging to construct a model of Japanese learners’ English across different proficiency levels. We are aware that it is quite difficult to design a consistent error tagset because the learners’ errors extend across various linguistic levels including grammar, lexis and phonetics. To do this, it is necessary to have a robust error typology. 
We designed our original error tagset for learners’ grammatical and lexical errors which are relatively easy to categorize, compared with other types of errors such as discourse errors or the ones related to more communicative aspects of learners’ language. Our error tags contain three pieces of information; part of speech, a grammatical/lexical system and a corrected form. For some errors which cannot be categorized to any word class, such as the misordering of words, we prepare special tags. Our error tagset currently consists of 45 tags. The structure of the error tags is as follows. (Figure 3) 

Figure 2. Screen dump of the TagEditor 
361
I will show several examples of error-tagged data below. 
*She has been waiting here for many <n_num crr=”hours”>hour</n_num>. 
(The wrong number of a noun) 
*The car looks <aj_inf crr=”better”>more good</aj_inf> now that it has been waxed. 
(The wrong inflection pattern of an adjective) 
*There <v_agr crr=”is”>are</v_agr> a computer. 
(The wrong subject-verb agreement) 
*If you <v_tns crr=”go”>will go</v_tns> to Vancouver, be sure to take a day trip to Victoria. 
(the wrong tense of a verb) 
2.5 Sub corpora 
We have also compiled sub-corpora for comparison. For example, if we have the native speakers’ speech data, it would be useful for comparing the utterance of native speakers and Japanese learners. We could do this by collecting the speech data of native speakers’ taking the similar kind of interviews as SST. We have also compiled a back-translation corpus. It can be compiled mainly by guessing what the learners intended to say in the interview, and translate it into correct Japanese. We believe it’ not so difficult to do this for Japanese native speakers or people who are familiar with English that Japanese learners speak. With the back-translation corpus, we could study how L1 (Japanese) transfer interferes the second language acquisition, or what kind of things are difficult for Japanese learners to express in English. As stated above, we perform error tagging only for grammatical and lexical errors. These sub corpora may cover what we cannot examine only by error tagging. 
3. Linguistic analysis 
In this chapter, we would like to show an example of how this corpus can be exploited to examine the learners’ language. 
3.1 Analysis on Japanese learners’ acquisition of English article system 
The English article system is one of the most difficult item for Japanese learners. We examined the article errors made by Japanese learners by using this corpus. For this analysis, we used the error-tagged part of the SST Corpus. We wanted to compare the tendencies of each SST level, but since we have not got enough data of certain levels (L1 and L9), we decided to divide them into three groups; novice (L1-3), intermediate (L4-6), and advanced (L7-9), and make a comparison between them. 

<n_cnt> … </n_cnt> 

part of speech (ex. noun) 

grammatical system (ex. countability) 

learner's error 

Figure 3. The structure of an error tag 
362The number ofthe interviewsThe number ofarticle errorsThe number ofnouns in learners'utterancesThe number ofarticle errorsper 100 nounsovice (L1-3)37569315818ntermediate (L4-6)35817481816vanced (L7-9)3054759279 
NI 
Ad 
As seen in Figure 4, the number of the article errors per 100 nouns is getting lower and lower as the level gets higher. By examining all these article errors, we found that there were three types of errors: overuse, underuse, and replacement of articles. We compared these types across different proficiency levels. (Figure 5) 0.0%10.0%20.0%30.0%40.0%50.0%60.0%70.0%80.0%underuse75.9%63.0%61.2%overuse9.8%19.4%19.8%relace14.3%17.6%19.0%Novice(L1-3)Intermediate(L4-6)Advanced(L7-9)
We found that the most frequent errors were the underuse errors in all proficiency levels. The reason for this might be that Japanese language does not have an article system. However, the frequency of the underuse errors decreases the level goes up, and the overuse and the replacement errors increase. Most of the underuse errors are related to the syntactic rules, on the other hand, the overuse and the replacement errors are related to the semantic and pragmatic rules in most cases. It is often said that the semantic and pragmatic rules are more strongly influenced by the L1 transfer, which means they are more difficult to understand properly for learners than the syntactic rules. (Mizuno, 2000) 
59%48%Intermediate(L4-6)Advanced(L7-9)We also found an interesting tendency in the replacement errors. In all proficiency levels, the replacement of “a/an” with “the” is the most frequent. This might be because if the learners cannot decide which is better, “a/an” or “the”, they tend to choose “the”, which can be put to a singular, plural and other abstract nouns. The learners often use this kind of communication and learning strategy of simplifying a rule to make communication carry on even if they do not understand the rule properly. "a/an" replaced with "the""the" replaced with a/an"an" replaced with "a"articles replaced withother determiners52%15%6%15%19%7%15%29%13%10%0%20%40%60%80%100%Novice(L1-3) Figure 6. Four types of the replace errors 

Figure 4. The number of article errors in error-tagged data of SST Corpus (Izumi, 2003) 

Figure 5. Three types of the article errors 
363
In the overuse errors, we found that the overuse of “the” was more frequent in the intermediate and the advanced levels than in the novice level. This result might imply that understanding the meaning of zero articles is more difficult than understanding how to distinguish “a/an” from “the”. Although there is not an article system in Japanese, “no articles” in Japanese is not equal to zero article in Engish. 58%42%77%23%75%25%0%50%100%Novice(L1-3)Intermediate(L4-6)Advanced(L7-9)overuse of "the"overuse of "a/an" 
60%40%Advanced(L7-9)In the underuse errors, the underuse of “a/an” was more frequent than the underuse of “the”. This might be occurred because L2 learners tend to produce an L2 output by translating the sentence in L1. An English article “the” is usually learned as the couterpart of a Japanese pronoun “sono”, but Japanese teachers often tell their students not to translate “a/an” into Japanese. Therefore, the learners tend to forget to put “a/an” when they translate a Japanese sentence into English. It can be said that this might be a type of errors which has been induced by the way of teaching. underuse of "a/an"underuse of "the"Novice(L1-3)Intermediate(L4-6)62%38%55%45%0%20%40%60%80%100%
There has been much research on the learners’ acquisition order of various linguistics phenomena, such as negation, tense and aspect, and so on. Most of them are examined based on the data that was collected on purpose for inducing the learners’ errors on an intended grammatical rule. This must be a good way to get results that the researchers have intended, from a small amount of data. However, it can be said that the results extracted from the more spontaneous and large-scale data are more reliable, although it would sometimes be rather difficult to formulate the extracted results. 
4. Applications in collaboration with NLP 
In this chapter, we would like to introduce an experiment we have done to examine how this corpus can be applied to develop the system in collaboration with NLP. 
4.1 Automatic level check system 
We examined the efficiency of judging learners’ proficiency levels automatically, and what kind of information in learners’ data is the most effective for it. We thought the following information is useful for the automatic level check based on SST evaluation scheme. (Figure 9) 

Figure 7. Two types of the overuse errors 

Figure 8. Two types of the underuse errors 
364
The automatic level check can be considered as the similar task to the automatic text categorization. For automatic text categorization, the machine learning method called SVM (Support Vector Machines) is often used. We used this method for the level check. SVM is defined by Cristianini and Shawe-Talor (2000) as follows: “Support Vector are learning systems that use a hypothesis space of linear functions in a high dimensional feature space, trained with a learning algorithm from optimization theory. This learning strategy … is a principled and very powerful method that in the few years since its introduction has already outperformed most other systems in a wide variety of applications.” 
In the next step, we tried to decide which features in Figure 9 are the most effective as the features for the automatic level check based on SVM. We did this by the depth-first search. We first did the level check by using all the information (the full set of all the features) in Figure 9. We repeated the same thing by using the set of 8)-17) and one feature picked up from 1)-7). Then, we found the set of 8)-17) and 7) has the highest accuracy rate. Then we decided to use those features as a root of searching for the level check. From the searching result, we found that the features 7), 8), 9), 10), 11), 13), and 15) are the most effective information. 
Figure 10 shows the result of our experiment for the automatic level check. 

Figure 9. The features which can be used for the automatic level check (Supnithi, 2003) 

Vocabulary 
1) The number of each vocabulary 
2) The number of 1) which appears more than once 
3) The number of 1) which is a content word 
4) The number of 3) which appears more than once 
5) The number of the sequences of two words (Bi-gram) 
6) The number of 5) which appears more than once 
7) The number of the words produced only by learners (excludes thewords also produced by interviewers) 
8) The frequency of each level when assigning the words to the twelvelevels of ALC Press's Standard Vocabulary List (SVL) 
Grammar 
9) The frequency of each part of speech 
Fluency 
10) The number of fillers, repetitions, self-corrections and pauses 
11) Duration per word 
12) The total number of words 
13) Average sentence length 
14) The number of sentences 
15) Duration of an entire interview 
Communicative skills 
16) The volume of interviewers’ vocabularyand the frequency of eachword 
17) The number of speaker turns 
365
As seen in Figure 10, overall accuracy rate using our most effective information (65.57% for exact match, and 96.52% for allowing one-level gap) is not bad. However, the accuracy rates are still relatively low in the levels of which we have not got the enough data. Therefore, we are planning to improve our framework by collecting more data of those levels, and using other kinds of information such as learners’ errors. 
5. Conclusion 
In this paper, we have presented an overview of the SST Corpus by explaining the data collection procedure such as transcribing and tagging including error tagging for error analysis. We also showed to what extent this corpus can be exploited first by showing our analysis on learners’ acquisition of the English article system, which might be useful for constructing a model of the learners’ language, and by introducing the framework for the automatic level check. 
We are planning to make this corpus publicly available so that teachers and researchers in many fields can use it for their own research interests, such as second language acquisition research, syllabus and material design, or the development of computerized pedagogical tools by combining it with NLP technology. 
Acknowledgements 
We have benefited greatly from cooperation and discussions with Assistant Prof. Yukio Tono of Meikai University. 
References 
Cristianini, N & Shawe-Talor, J 2000 An Introduction to Support Vector Machines and other karnel-based learning methods. Cambridge University Press, pp.7 
Izumi, E 2003 Error tag tsuki nihonnjinn eigogakushusha hatsuwa corpus wo mochiita gakushusha no kanshi shuutoku keiko no bunseki. (In Japanese) In Proceedings of the ninth annual meeting of the Association for Natural Language Processing, Japan 
Mizuno, M 2000 Chukan gengo bunseki, (In Japanese) Kaitaku-sha pp.74-75 
Supnithi, T 2003 Eigogakushusha hatsuwa corpus wo mochiita shuujukudohanntei. (In Japanese) In Proceedings of the ninth annual meeting of the Association for Natural Language Processing, Japan 
Tono, Y 2001 The Standard Speaking Test (SST) Corpus: A 1 million-word spoken corpus of Japanese learners of English and its implications for L2 lexicography. In Proceedings of ASIALEX2001, Korea, pp.257-262 

Figure 10. The result of automatic level checking (Supnithi, 2003) SST LevelThe number ofinterviewsAccuracy rate(0)Accuracy rate (±1)1 3 0%100%2 29 65.52%100%3 198 68.69%99.50%4 452 83.23%99.78%5 226 44.69%98.67%6 127 47.24%92.13%7 52 26.92%82.69%8 24 12.50%70.83%9 9 0%11.11%All1,121 65.57%96.52% 
366

Automatic feeding of translation memory tools 

Inés Jacob 
Facultad de Ingeniería 
ines@eside.deusto.es 
fax: 34(9) 44 139101Joseba Abaitua, Josu Gómez Facultad de Filosofía y Letras abaitua@fil.deusto.es fax: 34(9) 44 139087 

Universidad de Deusto 
Avda. de las Universidades, 24 
48007 Bilbao 
Abstract 
We present a method for the automatic extraction of aligned segments as a way to speed-up the process of translation-memory development. Parallel documents are captured from bilingual sites in the Web and processed in two steps. First, relevant information is extracted from downloaded files and transformed into a simplified TEI format. In a second step, parallel TEI files are converted into single files in TMX, which contain large collections of translation segments in a format that allows for the automatic feeding of translation memory tools. 
1. Introduction 
The development of methods and tools that accelerate the translation of multilingual digital contents is a matter of great importance in the information society. Normally translators work with documents that are related in one way or another with other documents that they have previously translated. Translators often recognize fragments that they would like to consult or even reuse, but which they fail to locate in the different memory devices of their computers. 
Translation Memory (TM) systems are tools that help translators carrying out this task in a manner that is respectful with the way they normally work. These systems make it possible to store and retrieve bilingual fragments, which normally correspond to previously translated paragraphs or sentences. TMX (Translation Memory eXchange format) is the standard method that most system use to exchange translation data among tools. 
The DELi research group at the University of Deusto specializes in the mechanization of the different stages that are involved in the process of generating translation memories in TMX format (Abaitua et al. 2002. Jacob et al. 2002). 
Recent projects cover the following issues: 
• analysis, selection, and integration of different software-tools that support import and export of multilingual contents in TMX and TBX; 
• work-flow management tools for multilingual contents based on TMX and XLIFF; and 
• multilingual web update and maintenance. 
As a result, procedures for the exchange of translation and localization resources have been defined with application in content-management for multilingual web sites. 
2. File-capture from the Web 
As a way to benchmark-test these techniques, were extracted from the Web documents belonging to several years of the official bulletins of our local institutions in the Basque Country. Texts from these bulletins contain a particular type of language, with a predominance of legal and administrative terminology, a distinctive class of registers, and a highly structured distribution of contents. The 
367
documents published in Internet normally contain monolingual texts, with the versions in Spanish and Basque in different separated files. 
In some cases, the original file format was plain text, with no tags, in others we were able to get files in HTML. (There were also big collections of files in PDF format, which we did not process.) This diversity of formats in origin is unavoidable, and requires the development of particular program modules for each particular source. 
3. Information Extraction and TEI tagging 
TEI markup has been adopted because it is a standard format for electronic text interchange, but also because it permits a precise cataloguing method. TEI-headers provide an effective way to manage the corpus. Furthermore, since TEI markup can be expressed in XML, the corpus can be browsed by means of common web-exploring facilities. Our particular corpus has been described through a simplified tag-set (TEI-Bi) inspired in TEI-Lite. 
Each document has been saved separately and converted into a XML file with the basic structure shown in Figure 1. 
<teiHeader>...</teiHeader> 
<text> 
<front> [Information from the front-page, up to the last title] </front> 
<body> [The text properly, including place and date] </body> 
<back> [place, date, signature, and control number] </back> 
</text> 
Figure 1. Document structure 
The header (<teiHeader>) assigned to each documents contains the elements of Figure 2. 
<teiHeader> 
<fileDesc> 
<titleStmt> 
<title> 
<author> 
<principal> 
<publicationStmt> 
<publisher> 
<sourceDesc> 
<bibl> 
<encodingDesc> 
<projectDesc> 
<classDecl> 
<taxonomy> 
<bibl> 
<profileDesc> 
<creation> 
<langUsage> 
<textClass> 
<classCode> 
Figure 2. Structure of the document header 
In order to meet the requirements of the DTD defined in TEI-Bi, it is necessary to extract from the texts a series of relevant data, such as the values of tags as title, author, editor, language, etc. The uniform structure of downloaded documents facilitates this task. In Figure 3 we show some of the automatically identified elements which will feed the teiHeader. 
368
N Elemento de información Ejemplo 1 Título del Boletín Boletín Oficial de Gipuzkoa 2 Número Número 23 3 Fecha Fecha 01-02-2001 4 Página Página 1843 5 ------------------------------------------ -------------------------------------------- 6 Num y Administración 7 Administración Municipal 7 Nombre Departamento Ayuntamiento de Errenteria 8 Descripción de la Orden Aprobación e información pública... 9 ------------------------------------------ -------------------------------------------- 10 Departamento Superior (puede omitirse o ser doble) Ayuntamiento de Errenteria 11 Departamento Inferior (puede omitirse) Rentas y Exacciones 12 Tipo o Título de la Orden Anuncio 13 Texto TEXTO 14 Lugar Errenteria, 15 Fecha a 19 de enero de 2001 16 Firma o Firmas El Delegado del Area de Hacienda... 17 Control (392)(963) Tabla (existe a veces) 
Figure 3. Automatically identified elements 
The text is marked-up in accordance with the following template. The numbers in parentheses identify elements as shown in Figure 3. 
<front> 
(1) <head type="main"> 
(2) <head type="num"> 
(3) <docDate type="main"> 
(4) <head type="page"> 
(6) <head type="Admin"> 
(7) <head type="Dep"> 
(8) <head type="law"> 
(10) <head type="DepSup"> 
(11) <head type="DepInf"> 
(12) <head type="abbreviated"> 
</front> 
<body> 
(13) <p> 
<p><table></p> 
<list type="ordered">, <list type="bulleted"> 
</body> 
<back> 
(14) <head type="place"> 
(15) <docDate type="sub"> 
(16) <docAuthor> 
(17) <head type="control"> 
</back> 
For each document, the elements <body> (<p> and <list>) are ordered by means of an id attribute whose value will be 1, 2, etc.. The element <table> will always appear inside a <p>, and will be assigned no number. 
4. Alignment and TMX markup 
Once documents are assigned a header and a basic set of TEI tags, original files and their translations are paired. These paired files are automatically processed so as to recognize and annotate every parallel 
369
segment. The output of this process are two files with aligned segments explicitly marked-up. Alignment takes place at the level of well-defined textual units, mainly paragraphs and headings. If for two paired files the number of segments does not match, and alarm is activated and the output set aside for manual revision. 
The final format chosen for the aligned corpus is TMX because it facilitates a quick reutilization of the material. Once alignment of TEI documents has been carried out, conversion into TMX is carried out through a series of filters. 
TMX files also contain a header and a body. The header provides information regarding the file's references (author, title, language, dates, publisher, etc.). The body is made up of equivalent fragments that are called units of translation, marked-up as <UT>. These translation units correspond to the segments already aligned in the TEI files. 
The architecture of the system is as follows: 
5. Results 
The output of the process is an automatically obtained translation memory in TMX. It is important to take into account that each new source of bilingual information will have specific characteristics with regard to the structure of the documents, to the storage-format, and to the filing structure. For this reason, the tools that render TEI-Bi formats will have to adapt to each particular information-source. However, from that stage onwards, the acquisition TMX memories can be carried out with any further adaptation. 

original 
files 

TEI-Bi 
files 

information extraction and TEI tagging 

file- 
pairing 

unpaired TEI-Bi files 

TEI-Bi 
(eu-es) file pairs 

alignment 

unaligned 
TEI-Bi 
paired files 

aligned TEI-Bi 
paired files 

TMX 
conversion 

TMX 
corpus 
370
6. XLIFF for multilingual content managment 
The definition of the XML Localisation Interchange File Format (XLIFF) started in 2000 by a group of localization suppliers and tools vendors, including: Oracle, Novell, IBM/Lotus, Sun MicroSystems. This format is based on the principles defined as OpenTag, and adopts many of the ideas later developed for TMX, with innovations that facilitate the exchange of the information that must be translated. The first draft of XLIFF, version 1.0, was published in May 2001. The development of the format was moved under the aegis of OASIS in December 2001, and the XLIFF version 1.0 became an OASIS Committee Specification at the beginning of 2002. The version 1.1 is being worked on currently. 
We are currently studying the adoption of XLIFF as a way to facilitate the work-flow management of multilingual contents in the different stages of the translation and location processes. 
7. References 
Joseba Abaitua, JosuKa Díaz, Josu Gómez, Inés Jacob, Garikoitz Araolaza, Luistxo Fernández. 2002. SARE-Bi: Gestor de documentación multilingüe sobre XML/TEI. Procesamiento de Lenguaje Natural 29: 313-314. 
Inés Jacob, Josu Gómez, Joseba Abaitua. 2002. Obtención de corpus bilingües para la alimentación automática de gestores de memorias de traducción. II Congreso Internacional de Traducción Especializada: La traducción científica. Universitat Pompeu Fabra, Barcelona 
TEI: http://www.tei-c.org 
TMX: http://www.lisa.org/tmx 
XLIFF: http://www.opentag.com/xliff.htm 
371

An Analysis of Lexical Text Coverage in Contemporary German 
Randall L. Jones 
Department of Germanic & Slavic Languages 
Brigham Young University 
One of the many practical applications of corpus studies is the generation of word frequency information, which can provide useful data to language teaching professionals as well as lexicographers. It makes sense that for the teaching of vocabulary in a second language, lexical frequency should play a significant role in the selection of vocabulary to be included in pedagogical materials. Likewise, a dictionary intended for language learning should be based at least in part on words that occur most frequently in the language being studied. As Aston has stated, "Insofar as frequency data from corpora can indicate whether a particular feature is likely to be worth learning, this fact makes it relevant to teachers and learners as well as to syllabus and material designers" (P. 8). 
In this arena of corpus-based frequency studies for pedagogical applications, Nation and others have been successful in determining what has come to be known as "text coverage", or the relationship between a given number of high frequency words in a text and the percentage of the text that these words account for. As Nation points out, "... studies of native speakers' vocabulary growth see all words as being of equal value to the learner. Frequency based studies show very strikingly that this is not so, and that some words are more useful than others" (P. 9). He goes on to show that in a specific text of English fiction the most frequent 1000 words in the text accounted for 82.3% of the total text. The second 1000 most frequent words increase the coverage by only 5.1%, i.e. a total of 87.4%. To achieve coverage more than 90% requires 2000-3000 additional words. 
This paper will report on the results of a similar study for contemporary German. The corpus from which the frequency information is generated is the BYU/Leipzig Corpus of Contemporary German and contains more than 4 million words, including a spoken component (ca. 1,000,000 words) and a written component (ca. 3,300,000 words). The spoken corpus is based on 400 interviews conducted in 60 communities in Germany, Austria and Switzerland. The written corpus consists of four genres: fiction, journalism, special purpose texts (e.g. operating instructions, advertisements, etc.), and academic prose. Using the RANGE program available from the School of Linguistics and Applied Language Studies at Victoria University of Wellington, New Zealand, calculations are being made on subsets of the BYU/Leipzig corpus. Information will be given for the total corpus, the spoken corpus as well as for each of the four genres of the written corpus. The paper will also discuss some of the difficulties of German frequency studies, e.g. complex inflectional and derivational morphology, polysemy and ambiguity. Applications to the generation of a word frequency dictionary for German will also be discussed. 
REFERENCES 
Aston, Guy (Ed.) . 2001. Learning with Corpora. Houston: Athelstan. 
Nation, I.S.P. 2001. Learning Vocabulary in Another Language. Cambridge: 
University Press. 
373

Antonymy in Childhood: a corpus-based approach to acquisition 
Steven Jones (University of Central Lancashire) 
M. Lynne Murphy (University of Sussex) 
sjones@uclan.ac.uk lynnem@cogs.susx.ac.uk 
Antonyms are pairs of words that express semantic opposition (hate/love, short/tall, over/under, etc.) and many studies have made use of corpora to shed light on how antonyms are learnt, stored and used (eg. Fellbaum 1995, Jones 2002). These studies update, and occasionally challenge, traditional research into antonymy (Lyons 1977, Cruse 1986) which has tended towards an intuitive analysis of the phenomenon. 
Antonymy is widespread in adult language, with as many as one corpus sentence in 50 being found to contain both members of an antonymous pair (Jones 2002). Antonymy is also a familiar concept to children. For example, Kreezer & Dallenbach (1929) found that 90% of children aged 7;6 are familiar with the notion of lexical opposition, and those who are not familiar with it easily learn it. Still, little research has been undertaken to determine whether adults and children use antonyms in the same way. We don't know which ‘opposites’ are favoured by children, at what age they are learnt, and how they are used in context. 
This paper uses corpora from CHILDES (Child Language Data Exchange System) to address how children make use of antonyms. The corpus consists of data from longitudinal studies of American English-speaking children recorded in family situations at regular intervals between two and five years of age. This data is suitable for antonym analysis, as shown by Murphy (1998). In that case, the corpus data demonstrated that the children used a wide range of antonyms, and were often leaders, rather than followers, in antonym use in conversation with adults. 
In this paper we turn our attention to the discourse functions of antonyms in child speech. Jones (2002) demonstrated that antonyms in context could be categorised according to their discourse function (e.g. Negated Antonymy: it's public money, not private; or Coordinated Antonymy: she couldn't confirm or deny the rumours). But this classificatory system was based on a corpus of written, adult language. Would these categories be appropriate to a corpus of spoken, children's language? And, if so, how would the distribution of data among these categories compare? 
Because the CHILDES corpus contains child-directed language as well as child-produced language, it is possible to identify which antonymous pairs and functions are experienced most frequently by children and whether this affects the ways in which they use antonyms in their own speech. Other issues that this paper seeks to address include whether children creatively produce their own ‘opposites’ using frameworks understood by them to signal contrast, and whether adults use antonyms differently when speaking to children than when speaking to other adults. 
Hypotheses of antonym acquisition (e.g. Charles and Miller 1989, Justeson and Katz 1991) have used evidence from adult written corpora to argue that ‘opposites’ are learnt through repeated exposure within particular frameworks. However, it is only by examining the antonymous output of children (and the antonymous input they receive) that an accurate assessment of this key childhood learning process can emerge. 
References: 
Charles, W.G., & G.A. Miller 1989 Contexts of antonymous adjectives. Applied Psycholinguistics 10:357-75. 
Cruse, D.A. 1986 Lexical Semantics, Cambridge: Cambridge University Press. 
Fellbaum, C. 1995 ‘Co-occurrence and Antonymy', International Journal of Lexicography, 8.4: 281-303. 
Jones, S. 2002 Antonymy: a corpus-based approach. London: Routledge. 
Justeson, J.S., & S.M. Katz 1991 Co-occurrences of antonymous adjectives and their contexts. Computational Linguistics 17:1–19. 
Kreezer, G., & K.M. Dallenbach 1929 Learning the relation of opposition. American Journal of Philosophy 41:432–41. 
Lyons, J. 1977 Semantics: Volume 1, Cambridge: Cambridge University Press. 
Murphy, M.L. 1998 Acquisition of antonymy: evidence from child input and output. LSA conference, NY, 8–11 Jan. 
372

Domain specific corpus building and lemma selection in a computational lexicon 
Stig W. Jorgensen, Carsten Hansen, Jette Drost 
Dept. of Computational Linguistics, Copenhagen Business School, 
Bernhard Bangs Allé 17B, DK-2000, Denmark. 
{swj.id, ch.id, jd.fra}@cbs.dk 
Dorte Haltrup, Anna Braasch, Sussi Olsen 
Centre for Language Technology, Njalsgade 80, DK-2300, Denmark. 
{dorte, anna, sussi}@cst.dk 
1. Introduction 
The Danish STO project, SprogTeknologisk Ordbase (i.e. Lexical database for language technology) is a national project with the aim of creating a large size Danish lexicon for natural language processing. This work greatly benefits from the Danish lexicon material developed within the framework of the multinational PAROLE project (see LE-PAROLE 1998). 
Most of the researchers working on the project are computational linguists, but the group also includes general linguists, lexicographers and people with a background in language for specific purposes (LSP). The work reported in this paper is very much a result of this combination of various expertises. 
The STO lexicon is planned to contain a minimum of 50,000 lemmas, of which approximately 15,000 will come from six different domains of language for specific purposes. The lemmas are provided with detailed morphological and syntactic information while the semantic information in this first version for a large part of the vocabulary is reduced to domain information. The planned number has already been reached for the encoding of morphological descriptions; 24,000 entries have also been encoded with syntactic descriptions. 
The first version of the lexicon should be complete by the end of 2003, but extension of the linguistic information especially at the semantic level, addition of pronunciation information as well as an extension of the number of domains will be possible at a later date, depending on funding. For further information on the STO project, see Braasch (Braasch et al. 1998, Braasch 2002) and the STO homepage: www.cst.dk/sto/uk. 
2. The domains and corresponding corpora 
The selection of specific domains is based on the potential future applications. At present, work has been done on the domains of IT (information technology), environment and health, and is just starting on the domain of public administration. The subjects of the two remaining domains have not been decided. 
Existing lemma lists or dictionaries from the different domains are not directly suitable for our purpose, for various reasons. 
Firstly, the delimitation of a domain varies a lot, e.g. the domain IT in some lemma lists includes lemmas from domains like commerce and marketing, and some health domain lemma lists 
374
have a large proportion of lemmas relating to nutrition, bodily exercise and alternative medicine, while other lemma lists have a more narrow definition merely including technical or scientific terms. 
Secondly, lemma lists and domain specific dictionaries are made for different purposes and address different user profiles. Mostly they are highly specialised term lists, while some cover a broader vocabulary like the one appearing in newspapers or words from general language having another meaning in the language of the domain in question. The STO database is not supposed to cover the most specialised terms of the different domains but rather the vocabulary that laymen might encounter in various contexts. Specialised termlists can later be added by future users. 
Thirdly, Danish is a less widely spoken language and up-to-date lemma lists or dictionaries are not available for all domains. 
For these reasons, the lemma selection for the domain specific vocabulary of STO is primarily based on text corpora. As a matter of fact this goes for the STO project as such. STO is supposed to be corpus based, which means that not only the lemma selection for both general language and domain specific language but also the morphological, syntactic and semantic encoding of each lemma depends on what is found in the relevant corpora. 
The selection of specific domains and the lemma selection from these was one of the first tasks to be initiated in the project (previously reported in Olsen 2002), but the procedure for establishing corpora and selecting lemmas is being continuously refined. We are aware that the vocabulary of the domains changes very fast and that the vocabulary of the STO database will be out of date after a short time. But since we have established a method, and the process of collecting texts for the corpora and the editing of the lemma candidate lists is to a large degree automatic, it will be rather unproblematic to extend the vocabulary of each domain in the future and thus keep it up-to-date. 
At this moment we have built three corpora, IT, environment and health. The health corpus was built from scratch according to the methods described below, whereas the other two corpora were initiated at an earlier stage of the project and have subsequently been revised to conform to the standards of the renewed methodology. From the IT domain we have selected and encoded – morphologically and syntactically – about 2000 words, and a similar number from the environment domain, whereas the health domain turned out to be richer with approximately 2500 lemmas. 
3. The building of domain specific corpora 
The main source for texts in the corpora is the web, since WWW texts are easily available and can be collected with a high degree of automatisation. Using the web as a resource for lexicography poses certain problems, including the fact that is is often a “dirty” source with no guaranteed level of correctness with respect to spelling and grammaticality (see Grefenstette 2002 for a discussion of webbased lexicography). The most important problem, however, is the selection of the texts to be included in a given corpus. 
There is a danger of circularity in the process of establishing a domain corpus. The purpose of the corpus is to discover which words are typical and frequent in a given domain, but to establish a representative corpus we need a notion of the delimitation and composition of the domain. If the corpus is to be built from available web pages, relevant search words have to be used to identify useable 
375
websites. In a sense one needs to know some of the terms in advance, even if the end product of the task is to discover these lemmas! It is not possible to avoid this basic problem altogether, but it is possible to develop a strategy which neither results in a purely arbitrary composition of the corpus, nor falls into the opposite trap by giving it such a fine-grained structure that the result is more or less determined in advance. 
Our strategy is an onomasiological approach1 where what we call a onomasiological structure, or OS, serves as definition and delimitation of the domain, and our main goal is a corpus with sufficient coverage of the domain. 
The corpus building process consists of four main tasks, as sketched in table 1: 
Task 1 Construction of OS for the domain Task 2 Identification of relevant websites Task 3 Downloading and sorting of text Task 4 Editing text into a corpus 
Table 1 
Though there is a natural progression from task 1 to task 4, these should not simply be read as steps in the process: Once a first version of the OS is established, tasks 1-3 are performed more or less simultaneously, with mutual feedback. 
Task 1: On the basis of existing thesauri and available literature, including major encyclopedias, we construct an onomasiological structure – the OS – a hierarchically structured list of topics and key words relating to a domain. That means that the OS is constructed very much like a thesaurus, but it does not (necessarily) fulfil the formal requirements of a thesaurus (or a concept system, for that matter, in spite of the fact that the OS is often presented graphically like a concept system). A thesaurus is “a structured list of expressions intended to represent in unambiguous fashion the conceptual content of the documents in a documentary system and of the queries addressed to that system” (Eurovoc homepage 2002). The OS serves a different purpose: to establish the collection of documents. 
Task 2: The items from the OS are used as search terms on the web to identify relevant texts. The search is performed using standard search engines, and with the appropriate amount of creativity usually employed by experienced web users (substituting search words for related words, trying various spelling variants etc.). Relevant texts are those that cover one, or preferably more, topics of the domain and confirm to the general standards of the project, most importantly: 
• Language: Danish. (A trivial demand, but often a practical problem, since many Danish websites also include text in English) 
• Time: Modern. We want the corpora to be as up-to-date as possible. 
• Communication: Preferably expert to layman, not expert to expert. The lemmas we wish to extract “should not be highly specialized terms of the domain, but rather words that laymen 
1 The onomasiological approach is a principle of the so-called Vienna school of terminology. Terminology studies concepts before terms. In this tradition, unlike in lexicology, onomasiology “does not refer to the content aspect of the sign but rather to the concept seen as part of the world outside language” (Temmerman 2000: 5). 
376
have to read and understand as part of their everyday life” (Braasch 2002). This also corresponds to the fact that we use a newspaper corpus as a main resource for the general language part of the lexicon. 
If the seach reveals new relevant topics not covered by the OS, or if the search yields insufficient or inappropriate results, the searcher reports to the OS constructor who improves or extends the structure. Relevant web addresses are passed on to task 3. 
Task 3: Texts are downloaded from the web pages. HTML codes are deleted, and foreign (primarily English) documents and identical documents are removed. These processes are performed automatically and can pose certain technical problems. New and constantly changing web technology (advanced Java scripts etc.) may render the extraction of text difficult with the available tools, and the removal of documents containing foreign text is often less than perfect when done automatically, but very time-consuming if done manually. If the net result (in terms of size of the resulting text file) is poor, we cannot guarantee that the body of text covers the topics it is supposed to cover. We do not engage in the time-consuming labour of reading through the text, but report back to task 2 that the web page in question does not ensure coverage, that is, we need more relevant web addresses on a given topic. 
Task 4: Once coverage is achieved, the texts are assembled into a corpus with a minimum of editing. In rare cases, a downloaded text is reduced in size before entering into the corpus. This is only done in extreme cases where a website has been included to ensure coverage of a minor and relatively unimportant topic, but the resulting text file is inappropriately huge compared to the rest of the texts and would distort the statistics of the lemma selection list that is to be derived from the corpus. Thus, reduction is only used with great caution, in accordance with our pragmatic goal of ensuring coverage, but not weighting. 
An important feature of this approach is that we have not predefined a size for the corpus we are building. Rather, the corpus is of sufficient size when it is of sufficient coverage! This leaves room for individual differences depending on the specific domain. It is our experience that the approach results in corpora in the range between 1 and 2 million tokens, which has proved appropriate to our purposes and the lemma selection methods. 
A major problem with the approach is that the actual composition of a given corpus depends very much on our tools. This has to do with the technical difficulties mentioned in task 3. If we could simply download all the text we wanted, the process would be much simpler with less feedback between the different tasks. As it is now, we are locked in an arms race with the web designers, and either we have to spend more time to acquire or produce new tools that can extract text from more advanced web pages, or we have to spend the time, as we do now, to find websites that our tools can handle. 
4. The selection of domain specific lemmas 
The overall method for the lemma selection is sketched in table 2: 
377
Step 1 Tokenisation (and POS-tagging of corpus) Step 2 Lemmatisation Step 3 Generation of lemma candidate list Step 4 Manual examination of lemma candidates Step 5 Quality evaluation 
Table 2 
The method has been changed recently, mainly by the introduction of a new lemmatiser. What we describe here is the new method (which was applied in the health domain), with a few remarks on the methods that were used previously. 
Step 1: Firstly, the domain specific corpus is tokenised, involving separation of punctuation marks from word forms and identification of abbreviations and common multiword units. The tokens are part-of-speech tagged, using the Brill tagger. 
Step 2: Previously, normalisation of the word forms was performed in a separate step, that is, capitalisation and other special symbols like the special Danish characters were substituted by lower-case letters or other characters respectively. As the next step, the normalised words forms were truncated according to manually constructed rules (based on a list of the 35 most common Danish word endings and their contexts), and the truncated forms were grouped together as candidate lemmas. This truncation and grouping of word forms was used as a temporary step since we did not have access to a lemmatiser for Danish, capable of treating unknown words. Now the truncation has been discarded, and normalisation is performed in combination with lemmatisation by a lemmatiser that is currently being developed at the Centre for Language Technology. 
The lemmatiser consists of the following linguistic components: i) a lexicon containing 450.000 word forms with their corresponding 60.000 lemmas and morpho-syntactic categories, ii) 43.000 rules generated automatically from a list of word forms and corresponding lemmas, and iii) frequencies of word forms and lemmas from other corpora, used in the disambiguation process of homographs. In the present case, the input to the lemmatiser is the list of part-of-speech tagged word forms from the collected corpus. Each token is looked up in the lexicon, according to the following principles: 
IF it is unambiguous then the lemma is selected, 
IF it is ambiguous then the input part-of-speech tag is used to disambiguate, but 
IF the homographs are within the same word class 
then frequencies of lemmas and word forms are used to disambiguate, or 
IF the part-of-speech tag and the morpho-syntactic category in the lexicon do not agree, 
then the lexicon overrules 
IF the token is unknown to the lexicon then rules are applied to generate the appropriate lemma from the word form. 
The lemmatiser has different possible output formats. For the lemma selection the output is a list of: lemmas, lemma frequencies, frequencies of the word forms in the input, part-of-speech tags, and 
378
marking of word forms not found in the lexicon and of word forms found with another part-of-speech tag in the lexicon. 
The lemmatiser has a performance rate of 97.8% correct lemmas for lemmatisation with dictionary and part-of-speech tagged input. However, this result depends on the part-of-speech tagging being correct; the actual outcome in the project is less perfect, since the Brill tagger introduces some errors, mostly relating to proper names. The performance rate for lemmatisation with untagged input is 94.5% correct lemmas. We therefore consider doing lemmatisation without the part-of-speech tagging. 
Step 3: The dictionary used during the lemmatisation consists of the words already encoded in the STO database. New lemmas not found in the dictionary are labelled as such, which enables us to put them on a separate list. The list is then sorted by frequency. This list of lemmas not previously encoded is what we call the lemma candidate list. 
Step 4: The lemma candidate list is examined manually, ignoring lemmas with a frequency of 1 (i.e., only a single occurrence in the domain corpus). The main purpose of the manual examination is to select the actual domain lemmas from the candidates (see section 5 below). Of course, the examination also includes correcting errors of lemmatisation and part-of-speech tagging. Previously, the part-of-speech assignment was made manually during this step, and we may return to this if we decide to do lemmatisation without part-of-speech tagging, but we also experiment with having only nouns, verbs and adjectives on the candidate list. 
A couple of figures to illustrate the outcome of the process: The health corpus consisted of approximately 2 million tokens. This resulted in a candidate list containing approximately 17,000 lemmas, of which 7000 had a frequency higher than 1 (3000 had a frequency higher than 3). From the 7000 candidates with more than one occurrence, 2200 domain lemmas were selected. (An additional 600 were put on a list of general language words – see section 5 below). 
Step 5: Finally, a quality evaluation is performed on the resulting vocabulary. One way of doing this is by comparing it to existing dictionaries or lemma lists of the domain. Though we do not find such dictionaries suitable as a primary source for our vocabulary, we can use them to check whether some central words of the domain accidentally do not figure in the selected vocabulary. The selection from the environment domain was compared to the Miljoordbog, an environmental dictionary (Heise and Kaalund 2001), and included every single item of this term list. In relation to the health domain, we are performing an experiment that will also serve as a quality evaluation of our corpus building method. We have acquired a body of health texts from the Danish National Encyclopedia. We will use these texts as an additional health corpus, treat it to the same methods, and see if the resulting lemma list is significantly different from the one we obtained from our webbased health corpus. In general, we are also experimenting with varying the corpus size (that is, including more than our method prescribes) and/or including more low-frequency lemmas to see how this changes the final result. 
5. Principles of the manual selection 
The manual selection of lemmas from the lemma candidate list is performed simply by deleting inadequate candidates. The following are deleted from the list: 
379
• Proper names. All names of persons are deleted, with no respect to their frequency. Names of important organisations and institutions are preserved if sufficiently frequent. In the domain of environment, for instance, a name like Greenpeace is included, but the name of the Danish Environment Secretary is not. 
• Expert terms. Terminology more appropriate of expert-to-expert communication than the layman-oriented vocabulary we wish to cover. For instance, formulas and long chemical names are deleted. 
• Long and/or unusual compounds. In Danish orthography, like in German, but unlike English, compounds are written as single words, which accentuates the problem. It is viewed as more important to cover all relevant simplex words from which the compounds can be generated. Furthermore, unusual compounds are often instances of what we call expert terms above. 
• Overrepresentation. Since our corpora are constructed on the basis of coverage, not weight, it is possible that lemmas from one or more less important subdomains become too dominant on the list. These are deleted if the overrepresentation is obvious from reading through the list. Of course, some basic steps have been taken to prevent distortion: We delete identical documents from the corpus, and reduce the size of documents when necessary (see section 3, tasks 3 and 4, above). Furthermore, we plan to differentiate the frequencies given in the lemma candidate list, so that the figures will not just reflect the frequency of a given lemma in the corpus as a whole, but also show how the occurences are distributed among the different text sources. 
• Errors. Typos etc., or errors due to the automatic lemmatisation. Orthographical errors relating to compounding (Danish compounds are often, mistakenly, written as more than one word, which can lead to unlikely lemma candidates). Non-Danish words may occur due to foreign language quotes in the corpus texts. One should, however, hesitate to delete foreign words since a word from a foreign language may be used as a technical term in Danish. In some domains, a “foreign” spelling (e.g. “ch” for “k” in names of chemicals) is usual, even if it is unauthorized. 
Of course, the list will not just include domain specific words, but also words belonging to general vocabulary that are not in the database already. Words that seem common are not deleted, but put on a separate list to be included among the general language lemmas. 
6. Some specific problems 
In this section we will discuss two very different, specific problems we have encountered in our work, to illustrate the kind of issues that can be raised by our methods and procedures. The first problem is one of automatic lemma selection. The procedure is not perfect, and there is a need for additional methods, for instance for the selection of collocations and multi-word units. Here, we will address another kind of problem, namely how to detect those words that appear both in the general language vocabulary and in a specific domain. The second problem is of a more theoretical nature and has to do with our corpus building methods. We identify WWW texts that deal with specific topics, but “text on the web” is not a homogenous genre, and the type of text very much depends on the domain. Does that create too large a typological difference between two given domain corpora? 
380
6.1. Lemmas occurring in both general language and domain specific language 
Words that appear in the general language vocabulary of STO will automatically be excluded from a candidate list of domain specific lemmas according to step 3 of the lemma selection described above. But a word might belong to the general language and at the same time be part of the language of a specific domain, so we need a method to detect words that have already been encoded as general language words but are also found with another meaning and perhaps another syntactic and/or morphological behaviour in a specific domain. Table 3 shows some examples. 
Semantic difference Word Domain Meaning ‘bus’ general language a passenger vehicle IT a data channel ‘port’ general language a large door or gate IT an external computer connection Morphological difference Word Domain Plural inflexion ‘indeks’ (Eng. ’index') general language ‘indekser’ IT and mathematics ‘indekser', ‘indices’ 
Table 3 
In order not to lose track of these lemmas, in the STO database we mark all entry words with source reference indicating in which corpus a lemma appears. Thus, the lemma ‘bus’ will be source-marked both for general language and IT since it appears in both corpora. 
This means that a single word can have source reference to general language as well as to all specific domains. This will be the case for the most common general language words. Words from the general language with low frequency will (hopefully) only appear in the general language corpus and will not be object to further treatment, but words that are marked with source reference from a general language corpus and from one or two domain specific corpora have to be picked out for special treatment. 
These lemmas will be subject to a special encoding process. For each lemma it has to be decided whether the linguistic behaviour of this lemma in each domain in which it occurs, differs from the existing encoding of the general language lemma at all the three linguistic levels. Any linguistic behaviour – morphological, syntactic or semantic – that differs from the general language encoding demands an encoding that reflects this behaviour and makes it apparent that the encoding is specific for a certain domain. Thus the ‘indeks’ example in table 3 will have two morphological units containing the inflectional patterns connected to it, one of which will be marked as valid for the IT domain only. 
We have not yet started the process of encoding the lemmas with a particular, linguistic behaviour when appearing in a domain specific context. The process will be started when all domain corpora have been established. 
381
6.2. Domain-dependant text type variations 
In section 3 above we noted that our domain corpora should consist of texts that are in Danish, as up to date as possible, and preferably written by an expert for an audience of laymen. Without going into the details of text and corpus typology (Atkins et al. 1992), we can state that demands to the text define the corpus we are building, including of course the demand that the texts should fall within a given domain. But what about the decision to use the WWW as a source? A main reason for using the web is the pragmatic one already mentioned: WWW texts are easily available and can be collected with a high degree of automatisation – and they are up to date in the sense of being available now. One could however add a less pragmatic reason for using the web as a resource. Some of the potential applications that the STO lexicon could be used for, will deal with the web and web texts, so for this reason the web itself could be a suitable choice of resource. Our attitude towards the corpora we establish depends on whether we view the choice of resource as purely pragmatic or partly dictated by the nature of potential users and applications. 
In comparing the environment corpus and the health domain corpus, we noted that most of the environment texts turned out to be texts originally produced in print (papers, reports, book chapters etc.) and subsequently made available on the web, whereas most of the health texts were produced for the web without prior publication (FAQs and online medical information services). Whether we should accept this result, depends on our attitude to WWW as a resource. If we maintain that we primarily gather our texts from the web to make life easier, we have failed in the sense that we have overlooked a parameter that is necessary to obtain uniform corpora – or, more precisely, that our criterion of “expert-to-layman communication” is too general to ensure uniformity. In that case, we should seek out other resources than the web, and add new kinds of material to the corpora when necessary. If, however, we want the corpora to be representative of the texts available on the WWW within given domains, the difference between the environment corpus and the health corpus does not have to be a problem. The composition of the corpora simply reflects the fact that web texts on environment typically take the form of papers and reports, whereas web texts on health do not, and web-based applications should benefit from this being reflected in our vocabulary. 
Since the corpora used for the general language vocabulary are not web-based, there are good reasons for maintaining that the use of WWW is mainly pragmatic. Our present attitude is that we should simply pay more attention to this aspect of text collecting – and seek to improve our quality evaluation methods. As it is, we have no conclusive evidence of how the difference in text type actually affects the vocabulary. Our experiments with comparing lemma lists from web-based and non-web-based corpora within the same domain, may yield results that shed light on this question. 
7. Concluding remarks 
In this paper we have presented an approach to domain specific corpus building and lemma selection; an approach that tries to bring a certain amount of structure into the text selection process without restricting the outcome, and tries to apply a high degree of automatisation while still leaving some key decisions to human agents. The methodology is still being developed. We have been generally satisfied 
382
with our results, but we need to establish more rigorous procedures for quality evaluation, and we still await the outcome of the various experiments that have been suggested in this paper. 
References 
Atkins S, Clark J, Ostler N 1992 Corpus Design Criteria. Literary and Linguistic Computing 7(1): 1-16. 
Braasch A 2002 Current developments of STO – the Danish Lexicon Project for NLP and HLT applications. In Proceedings from the Third International Conference on Language Resources and Evaluation, Las Palmas, pp 986-992. 
Braasch A, Buhr-Christensen A, Olsen S, Pedersen B S 1998 A Large-Scale Lexicon for Danish in the Information Society. In Proceedings from the First International Conference on Language Resources and Evaluation, Granada, pp 249-255. 
Eurovoc 2002 http://europa.eu.int/celex/eurovoc/ The European Communities. 
Grefenstette G 2002 The WWW as a resource for lexicography. In Marie-Hélene Corréard (ed.) Lexicography and Natural Language Processing. A Festschrift in Honour of B.T.S. Atkins. Göteborg, EURALEX, pp 199-215. 
Heise P, Kaalund L 2001 Miljoordbog. Copenhagen, Amtsradsforeningen. Electronic version: www.miljoeordbog.dk 
LE-PAROLE 1998 Danish Lexicon Documentation. Report. Copenhagen, Center for Sprogteknologi. 
Olsen S 2002 Lemma selection in domain specific computational lexica – some specific problems. 
In Proceedings from the Third International Conference on Language Resources and Evaluation, Las Palmas, pp 1904-1908. 
Temmerman R 2000 Towards new ways of terminology description. Amsterdam/Philadelphia, John Benjamins Publishing. 
383

How non-native speakers express anger, surprise, anxiety and grief: 
a corpus-based comparative study 
Tomoko Kaneko 
Showa Women's University 
1. Introduction 
Recently, there have been several studies on learner grammar and lexicon using learner corpus data (Granger 1998). Corpus has also opened an exciting door to the investigation of various pragmatic functions and the discourse of genuine language. By only examining small amounts of data, we fail to overlook some interesting rules pertaining to language use, which may be acquired by examining large amounts of data. Corpus analysis is especially useful to compensate for this weakness in the study of language. In this paper, we try to explore some features of English emotional expressions in the Japanese, Chinese and French learner corpora. Although emotional expression has been studied from the perspective of many different disciplines, the study of the use of learners’ emotional expressions in SLA has not yet been fully explored. In this study, we will also compare the learners’ use of emotional expression against those of native speakers based on the London Lund Corpus (LLC) and Wellington Spoken Corpus (WSC), which hopefully will give us a wider view of the use of English emotional expressions. 
2. Studies on the linguistic expression of emotions 
Human emotion has been studied in various disciplines; however, there is very little research available on the linguistic expression of emotions. Studies on the receptive ability of emotional expressions are more pervasive than that of productive ability. Rintell (1984) studied Spanish speakers’ ability to understand the underlying emotional content of English conversation. The range of emotions classified were pleasure, anger, depression, anxiety, guilt and disgust. She demonstrated that for all the learners, the task of choosing the correct emotion by listening to the conversation was difficult. She also argued that the emotional expression was interesting not only as a core of the study of human behavior but also as a pragmatic function. She explains that emotional expression is considered to be one of the illocutionary acts, because expression of emotion is a speech act in which the listener recognizes the speaker's intention to perform that act. Thus, the utterances that express emotion can simultaneously perform a variety of social and psychological functions. For example, the expression of anger can serve the function of complaining. In other cases, the expression of pleasure may serve the goal of communicating satisfaction, but could be contrived as boasting. Since the expression of emotion is one of the pragmatic functions, Rintell points out that those conventions constrain us from directly expressing emotion. She explains that politeness is the key issue in the appropriate use of the language, while in the case of emotional expressions, the range of strategies to adjust indirectness in the expression is very important for effective communication. Indirectness in emotional expression, particularly when people convey negative emotion through language, can be adjusted by minimizing the expression of those feelings. Thus, it is an important task for speakers to manipulate language so 
384
as to control the level of directness with which emotion is expressed. 
A recent study on the receptive ability of emotional expression was administered by Graham et al. (2001). They examined how well Japanese and Spanish native speakers learning English understood emotions expressed in utterances. The subjects listened to narrations recorded by actresses and actors. They were to judge which type of emotion was expressed among the eight choices. There was a significant difference between the native speakers’ and non-native speakers’ ability to judge speaker's emotion. They found that native Japanese speakers confused the paralinguistic cues of certain emotions like “anger”, “hate”, and “nervousness” in ways that native speakers of English rarely did. Another interesting finding was that there was no correlation between proficiency level and the ability to judge emotions. Graham et al. (2001) explained that this was because the acquisition of the ability to judge emotions is a slow process. Additionally, the researchers presumed that because emotional expressions are not taught in most classrooms, the learners would not notice them even if they were exposed to them outside of the classroom. Thus the learners could not utilize even those rare chances to notice the genuine input of emotional expressions. This is why the teaching of emotional expressions in the classroom is strongly advocated by the researchers. 
The focus of Rintell's second paper (1989) was the production of emotional expressions in narratives by Arabic speakers learning English. She explored the following: 1) what are some of the discourse features and patterns found in language in which the emotion is conveyed or described? 2) how can the emotional expression of native speakers and learners be characterized and compared? The emotion categories studied in the study were anger, fear, fright, anxiety, depression, and guilt. Rintell concluded that while native speakers of English used various strategies to allow the listener to participate actively in the conversation and thus allow the listener to “read between the lines” and empathize with the speaker, the learners lacked these strategies. Although linguistic strategies such as direct, minimized, and repeated statements of emotions were shared by the two groups, only native speakers used figurative language, reported speech, epithets and/or depersonalization. 
The most recent study on the production of emotional expressions was reported by Dewaele and Pavlenko (2002). Their study focused on the acquisition of emotion vocabulary by bilingual speakers. They were interested in the study of emotion vocabulary because “psycholinguistic studies also suggest that emotion words may be distinct from other abstract words on a number of characteristics and should be treated as a category separate from both concrete and abstract words” (265). Thus, they presumed that emotion vocabulary might be subject to different constraints and usage from the rest of the lexicon in L2 learning. In addition, they believed, it was plausible that the use of emotion words was related not only to socio-cultural factors, but also to individual experiences. They demonstrated that, contrary to what Graham et al. (2001) found, the use of emotion words in interlanguage was linked to the learners’ proficiency levels, type of linguistic material available, demeanor, and in some cases, the gender of the interlanguage speakers. 
The above mentioned studies on receptive and productive ability for emotion vocabulary raise some interesting questions about the use of emotional expressions by second language learners. 
385
However, before we start our investigation, we need to define the emotion vocabulary which will be studied in the present study. 
3. Emotion vocabulary and the limit of the present study 
Emotion is defined as “any of the strong feelings of the human spirit” in the Longman Dictionary of Contemporary English (1978). Thorndike Barnhart Beginning Dictionary (1972) also explains that “joy, grief, fear, hate, love, anger and excitement are emotions.” 
A limitation of the present study was that we focused only on negative emotional expressions. As has been stated in Rintell (1989), negative emotional expressions need more pragmatic consideration than positive emotional expressions. Negative ones are inevitably face-threatening and thus speakers need more pragmatic considerations and linguistic form devices to adjust the level of directness. Since we plan to use the Louvain International Database of Spoken English Interlanguage (LINDSEI) corpus data for the present study, it will not be possible for us to study phonological or super-segmental features which are not detected in the language forms. Thus in this paper, we would like to focus on the second language learner's negative emotional expressions, whose special features are easier to detect by using only transcribed data. 
Review of the earlier studies on emotion vocabulary generated the following three research questions: 
1) How do Japanese second-language learners use English negative emotion vocabulary? Is their usage different from that of other non-native speakers and native speakers of English? 
2) What are the strategies employed by second-language learners and native speakers when these emotions are expressed? Is there any difference in the usage of the strategies among Japanese students, other non-native speakers of English, and native speakers of English? 
3) Is there a correlation between the Japanese learners’ English proficiency level and the frequency of negative emotion vocabulary or frequency of strategies to express negative emotions? 
4. The study 
4.1. Data & Subjects 
The data for the present study consists of the following five corpora. 
Chart 1: Data for the present study 
Kinds of Corpus Tokens Types Type/Token Ratio 
LINSDEI Japanese sub-corpus (non-native turns) 38,767 2,702 6.97 
LINDSEI Chinese sub-corpus (non-native turns) 62,811 3,484 5.55 
LINDSEI French sub-corpus (non-native turns) 88,517 4,432 5.01 
London Lund Corpus 1,789,044 24,807 1.38 
Wellington Spoken Corpus (New Zealand) 1,101,190 25,711 2.33 
386
In 1995, a complementary project of the International Corpus of Learner English (ICLE) project was launched at the University of Louvain in Belgium to compile a corpus of spoken learner language, the LINDSEI. The subjects are all advanced level (third/fourth year) university students. In the first part of the interview, the subjects freely chose a topic among the available four choices. The topics were: 1) an experience the subject had which taught her/him an important lesson, 2) a country the subject visited which impressed him/her, and 3) a film/play the subject watched which s/he believed was particularly good/bad. In the second half of the interview, the subjects were asked to make a story based on a sequence of pictures. The theme of the story revolves around a painter that was asked to paint a portrait of a lady, but the lady wasn't satisfied with the result and asked the painter to repaint it. In the end the lady was happy with her painting and proudly shared her painting with her friends. 
4.2. Measure and Analysis 
WordSmith tools were used to analyze the corpora. First all the vocabulary for negative emotions was retrieved from the three LINDSEI sub-corpora. The result showed that except for the words used for the emotion categories of anger, surprise, anxiety, and grief, only a few words were used in the corpus. For example, “hate” was the only word used to express the emotion of hate and was used just twice as a noun and three times as a verb in the Japanese sub-corpus. Considering the huge amount of total tokens in the data, we decided to check the frequency of each vocabulary word only when the total frequency of the emotional expressions of the target emotion category counted over 0.02 % of the total tokens in the present study. In this manner, ten words in four categories expressing negative emotions were chosen from the three LINDSEI sub-corpora as shown in Chart 2. Since all the words are adjectives, the emergence of each word was also categorized according to whether it was in predicative use or in attributive use. 
Chart 2: A list of Vocabulary of Emotions Studied 
Emotion Category Vocabulary 
Anger angry 
awful 
terrible 
Surprise shocked 
surprised 
surprising 
Anxiety nervous 
worried 
afraid 
Grief sad 
387
Next, only the vocabulary listed in the above chart was retrieved from LLC and WSC. Thus it is possible that in the present study, the holistic view of the use of emotion vocabulary and strategies in the LLC and WSC can not be observed. Since the focus of the present study is to look at how emotions are expressed by non-native speakers of English, the holistic feature of emotion vocabulary of native speakers of English will be studied at another time. 
The strategies listed in Chart 3 were based on Rintell (1989). As for the strategies employed in emotional expressions, an overview of the data of the three different groups of students demonstrated that only strategies No. 2 and No. 3 in the Chart were utilized. The focus here was to find out what types of strategies were utilized by non-native speakers and whether there were any differences in their usage between native speakers of English and English learners. Strategy No. 1 was added to the list only after we confirmed the concordance lines of the ten negative emotion words in the LLC and WSC. Strategy No. 1, the sentence structure strategy, was used only in those native speakers’ corpora. The purpose of including strategy No. 1 is to provide points of contrast between the strategies used by non-native speakers and native speakers. 
Chart 3: Categories of Emotional Expression Strategies Found in the LINDSEI, LLC, and WSC 
Categories Details 
1. Sentence Structure Exclamatory Sentence 
ex. oh how awful 
2. Word Modifier Intensifier 
ex. the woman is really angry 
Minimizer 
ex. she is a little bit nervous 
3. Additional Word(s) Repetition of the Emotion Word 
ex. he is terrible terrible 
Addition of “oh”, “oh no” 
ex. oh it's surprising 
Addition of “God”, “oh my God” 
ex. oh my God it was awful 
Although the amount of data was not sufficient for statistical purposes, as we only had TOEFL scores for twenty-four Japanese subjects out of fifty-three in the corpus, we checked to confirm whether there was a correlation between the TOEFL scores and the range of emotional expressions or in the frequency of strategies used by the target subjects. The two different views on the relationship between the learners’ ability of emotional expressions and TOEFL scores led us to confirm this. Regarding this point, Grahan et al. (2001) concluded that the learners’ ability to understand emotions did not improve as their target language proficiency improved, in contrast to Dewaele et al. (2002), who stated that the use of emotion words in interlanguage was linked to proficiency level as well as the type of research data, extroversion vs. introversion personality trait, and so forth. 
388
4.3 Results 
The following chart shows the summary of the use of emotional expressions. 
Chart 4: Frequency of Negative Emotion Vocabulary in Five Corpora 
Japanese Chinese French 
38767 tokens 62811 tokens 88517 
tokens LLC 
1789044 
tokens WSC 
1101190 
tokens anger angry 15 35 12 13 35 awful 2 3 38 27 111 terrible 3 5 3 18 99 Sub-total 20(5.16) 43(6.85) 53(5.99) 58(0.32) 245(2.22) surprise shocked 3 1 3 1 10 surprised 14 6 10 7 42 surprising 0 1 4 1 13 Sub-total 17(4.39) 8(1.27) 17(1.92) 9(0.05) 65(0.59) anxiety nervous 9 9 2 2 26 worried 3 4 1 17 84 afraid 2 8 23 5 28 Sub-total 14(3.61) 21(3.34) 26(2.94) 24(0.13) 138(1.25) grief sad 1 13 24 12 64 Sub-total 1(0.26) 13(2.07) 24(2.71) 12(0.07) 64(0.58) Grand-Total 52(13.41) 85(13.53) 120(13.56) 103(0.58) 512(4.65) 
Note) The numbers in the ( ) show the frequency per 10,000 words in the corpora. 
Chart 5 is a summary of strategies in utilizing emotional expressions used in the five corpora. 
Chart 5: A Summary of Strategy Use in the Five Corpora 
Japanese Chinese French LLC WSC Sentence 
Structure Exclamatory Sentence 6 22 Modifiers Intensifier absolutely 1 awfully 1 bloody 1 extremely 1 1 god 1 intensely 1 jolly 1 just 1 5 (very) much 1 1 1 overly 1 pretty 10 quite 1 11 24 really 1 5 13 4 25 terribly 1 so 4 4 4 4 13 too 2 3 0 4 unduly 1 very 11 34 11 8 15 Minimizer a bit 4 6 12 
389
a little 1 3 1 rather 1 fairly 1 a little bit 1 1 kind of 1 bits 1 Additional Repetition 3 3 Words “Oh” “Oh no” 7 5 31 “God” “Oh my God” 2 1 Frequency of strategy use 19 52 60 42 171 Frequency of the target emotion 52 85 1207 103 512 Rate of strategy use 36.5% 61.2% 50.0% 41.1% 33.4% 
Finally, Chart 6 shows the Pearson Product Moment Correlation coefficients between the TOEFL scores of the twenty-four Japanese subjects and their use of emotional expressions. 
Chart 6: Correlations Between TOEFL Scores and the Frequency of Emotional 
Expressions and Strategies 
token type strategy r with TOEFL -0.23 -0.11 -0.02 
N=24; p<0.01 
5. Discussion 
It was a surprise to see that the Japanese subjects were required to learn only a minimum of twelve emotion words while attending junior high school by the Education Ministry. We screened the emotion vocabulary listed in the seven most popular English textbooks used by the subjects during junior high school. This method allowed us to make lists of 32 adjectives, 15 set phrases, and 11 verbs that express emotions the subjects could have learned through the end of junior high school and beyond. Doubt concerning how much they could communicate, even if they could fully utilize those twelve words, was our first reaction when we checked the textbooks they had used. Nevertheless, considering the fact that the minimum total number of words required to be learned in junior high school at that time was 507 and that the minimum number of emotion vocabulary required was twelve (the ratio of emotion words to the entire targeted vocabulary was 2.37%), we can't argue that the range of emotion words taught at junior high school was extremely poor. On the contrary, the actual use of emotion words by Japanese subjects in the corpus (although we only focused on the four negative emotion categories in the present study) represents 0.13% of the token, while those of native speakers was between 0.01 to 0.05%. Although it is true that the context of the utterances greatly affects the range and frequency of the usage of emotion vocabulary, since the corpus is meant to reflect the natural distribution of various language forms, the two native speakers’ corpora at least suggests that the rate of emotion words against the total number of words taught in Japanese junior high schools does not 
390
seem to be too low. 
First, we would like to answer the first research question: how do Japanese learners of English use emotion vocabulary in their second language? The three LINDSEI corpora collected using the same procedure demonstrate an interesting comparison. Japanese students used “surprise” more often than Chinese and French students. On the other hand, the latter group of students used more “anger” and “grief” expressions. Japanese students did not use the word “surprising” at all, while they used the word “surprised” very often. As for the emotion of anxiety, Japanese students preferred “nervous” than “afraid”, while the French students showed exactly the reversed tendency, Chinese students in between. 
The big difference between the corpora by learners and by native speakers was that learners used only a few attributive uses of adjectives. Among the ten 254 samples of negative emotion words studied in the present study, there were in total seventeen examples (6.7%) in the three learners’ corpora; five examples of “sad” in Chinese corpus, and four examples of “awful” in French corpus and so forth, while in the native data, there were 125 examples (20.4%). Including this phenomena, the range of expressions used by native speakers seems to be much wider than those of non-native speakers. In particular, the New Zealand teenager corpus shows various combinations of verbs and emotive adjectives in predicative use. 
The second research question concerns the strategy employed in expressing emotions. The frequency of strategy usage among the Chinese students was the highest among the three learner corpora. In twenty-three out of thirty-four cases, the Chinese students utilized the intensifier “very” with the phrase “be angry.” Only the French students seemed to prefer the term “really” to “very.” In addition, the distribution of the strategies among French students is quite different from the other two groups. Only the French group used the strategy of adding extra interjections (e.g., Oh no, God, and so forth) to intensify the meaning. This result may support the stereotypical observation that French students are more talkative than taciturn Asian students. Interestingly enough, on the whole, the rate of strategy usage against the total frequency of emotional expressions by Chinese and French students eclipse those of Japanese students. The rate of strategy usage by the Japanese students was roughly the same with that of the two groups of native speakers. As Rintell (1989) suggested, native speakers often used minimizing strategies with negative emotional expressions. The LLC confirms a minimizer frequency rate of 14.3% while this rate was only 8.2%, 9.6%, and 8.9% in the WSC, Chinese corpora, and French corpora respectively. The fact that the frequency rate among the Japanese students at 15.8% which is nearly identical to the LLC's rate while the frequency rate among all the other non-native students was below 10% is an interesting finding. If our presupposition that the English proficiency of the three groups of students is roughly equal is correct, the explanation for the Japanese students’ overuse of minimizers seems to be linked to their preference for less face-threatening speech acts due to their cultural background. Again, on the whole, the three groups of students used less variety of strategies than native speakers. 
Our answer to the last research question is no. Although the amount of data was not sufficient, neither tokens nor types of emotional expressions, nor those of supporting strategies demonstrate a 
391
correlation with the TOEFL scores of the twenty-four Japanese students. This supports the finding by Graham et al. (2001). Although their study focused on the learners’ receptive ability of emotional expressions, it is feasible that we can predict the same tendency in their productive ability because when the students did not understand certain emotional expressions, it was impossible for them to know how to express them. On the other hand, Dewaele and Pavlenko (2002) studied bilingual speakers’ productive ability of emotional expressions. Naturally the proficiency level of the subjects must have been much higher than Graham et al.'s (2001) and the present study. Most of the Japanese students seemed to employ communication strategies, which were used to compensate for their lack of linguistic ability, rather than expressive strategies to intensify or minimize the emotional expressions. We definitely need a bigger base of subjects with a wider range of proficiency levels to make any conclusions about the correlation between these two factors, but we presume that the use of emotional expressions by learners must also be influenced by other factors, for example, the learners’ target language proficiency, learner characteristics, culture, and so forth. We have to wait for additional research to answer these questions. 
6. Conclusion 
Based on this comparative study using the three non-native and two native speaker corpora on English negative emotional expressions, we reached the following conclusions: 
1) The use of negative emotion vocabulary differed according to the learners’ language background. Japanese students preferred less face-threatening expressions than the other non-native groups. The non-native students’ usage of negative emotion vocabulary on the whole also differed from that of native-speakers'. The most noticeable point was that native speakers exploit varieties of expression including attributive uses of negative emotive adjectives. 
2) Strategies used in expressing negative emotions also differed according to the learners’ linguistic background. The Chinese and French students used different strategies more frequently than the Japanese students and native speakers, but out of the five groups the Japanese students used minimizers most frequently. The choice of strategy and the choice of actual wording in the strategies also varied according to the learners’ linguistic background. 
3) English proficiency level and the frequency of the usage of negative emotional expressions and supporting strategies do not seem to correlate in the present study. However, this finding was not conclusive as the amount of data available in the present study was limited. 
Noticeably among the three non-native data sets, the Japanese data set included the narrowest range of negative emotional expressions and supporting strategies. Generally, Japanese culture avoids face-threatening expressions, and this fact may have limited the range of expressions utilized in accordance with their limited English proficiency. However, the results shown here have proven that Japanese students as well as the other non-native students do try to express emotions, in fact more frequently than native speakers of English. The difference seems to lie in how well they can use a variety of emotional expressions and their supporting strategies. Providing students with sufficient 
392
exposure to emotional expressions and their supporting strategies should be one of the most important instructional goals to implement. Issues of the learners’ cultural background and idiosyncrasies only comes after the learners have been immersed with an abundant array of input. 
In expanding the present study, role of teaching emotional expressions in language acquisition and the ways those expressions are used in different modes will be good research topics in the future. 
References 
Davitz J, Davitz L 1959 The communication of feelings by content-free speech. Journal of 
Communication 9: 6-13. 
Dewaele J, Pavlenko A 2002 Emotion vocabulary in interlanguage. Language Learning 52 (2): 
263-322. 
Granger S 1998 The computer learner corpus: a versatile new source of data for SLA research. In 
Granger S (ed.), Learner English on computer. London, Longman, pp3-18. 
Graham C, Hamblin A, Feldstein S 2001 Recognition of emotion in English voices by speakers of 
Japanese, Spanish and English. International Review of Applied Linguistics in Language Teaching 
39:19-37. 
Greenbaum S, Svartvik J The London-Lund Corpus of Spoken English. 15 Dec. 2002. <http://khnt. 
hit.uib.no/icame/manuals/LONDLUND/INDEX.HTM> 
Hadley A 2001 Teaching for cultural understanding. In Hadley A (ed.), Teaching language in 
context (Edition). Singaporre, Heinle & Heinle Thomson Learning, pp345-89. 
Hofland K, Lindebjerg A, Thunestvedt J 1999 ICAME Collection of English Language Corpora 
(second edition). Bergen, The HIT Centre University of Bergen. 
Hughes R 2002 Teaching and Researching Speaking. London, Pearson Education. 
Rintell E 1984 But did you feel about that?: the learner's perception of emotion in speech. 
Applied Linguistics 5 (3): 255-64. 
Rintell E 1989 That reminds me of a story: the use of language to express emotion by second- 
language learners and native speakers. In Eisenstein M (ed.), The dynamic interlanguage: 
empirical studies in second language variation. New York, Plenum Press, pp 237-57. 
Tannen D 1989 Repetition in conversation: toward a poetics of talk. Language 63 (1): 574-605. 
393
3rd 

Patterns of elaboration and inter-language development: an exploratory corpus analysis of college student essays 
Sachie Karasawa, Ph. D. 
Community College of Southern Nevada 
Abstract 
Corpus linguistic analysis has been increasingly applied to the studies of variation in surface linguistic features of second language writing. Previous research found that nonnative English speaking (NNS) writers at higher levels of proficiency produced larger quantity (measured by the number of words) and greater quality (measured by the number of varieties of lexical items used in the texts) than those at lower levels of proficiency (e.g. Ferris, 1994). Karasawa (2001), however, found large quantity did not always mean judged higher quality of NNS essays. The present study compared the patterns of elaboration found in NNS student essays in high, intermediate, and low holistic score ranges. The specific target linguistic features examined were lexical items used for modification of nouns and other clause elements, namely, adjectives, adverbs, past/present participles, to-infinitives, prepositional phrases, and that- and wh- clauses. These three NNS sub-groups were further compared with two sub-groups of essays written by native English speaking (NS) students. The results showed some differences in the combinations of linguistic items used for elaboration among the NNS essays in different score ranges while the NS essays did not show any clear trends. This paper concludes that the different patterns of elaboration found in the NNS essays may reflect different stages of inter-language development. 
1. Introduction 
Corpus linguistic analysis is becoming a valuable tool for examining learner language. Learner corpora may not necessarily share the same characteristics found in corpora of published work produced by expert writers, and observations and descriptions of learner language use through corpus analysis could contribute to a better understanding of linguistic features used by second language (L2) learners. Variations are also likely to be found among L2 learners in different stages of language acquisition, which calls for a comparative analysis of corpora produced by learners at different proficiency levels. 
The present study analyzes the types and the numbers of linguistic features used for elaboration in the essays written by nonnative English speaking (NNS) college students who were classified into three different proficiency levels. The main purpose of the study is to examine whether there are different elaboration patterns found among the NNS writers at each proficiency level. This study is exploratory in nature and examines relatively small corpora. The results reported in this paper are primarily descriptive although some inferences are also made from them. 
The next section briefly reviews the previous research and gives a rationale for the present study. The third section describes the research method of the present study. The fourth section presents and discusses the results. The final section draws a tentative conclusion, discuss the limitation of the present study and offers some suggestions for future research. 
2. Background to the study 
The length of an essay, measured by the total number of words, has been widely used as a quantitative fluency variable in second language (L2) writing research (Reid, 1990). The use of this variable for measuring fluency has validity in that higher linguistic competence, which includes lexical and syntactic knowledge, should be necessary for producing essays in greater length. 
Greater length of individual sentences, however, may not always guarantee greater quality. Lengthy sentences may contain repetitive use of the same lexical items and may consequently become excessively redundant. Lengthy sentences may, on the one hand, express highly complex ideas, but they may, on the other hand, obscure the main point and may not necessarily convey meanings in the most efficient and comprehensible manner. 
394
Ferris (1994), responding to Biber's call for more multi-dimensional approaches to written discourse (Biber, 1985), examined 28 lexical and syntactic variables in essays of 60 low and 100 high proficiency level ESL students. The study found that the differences in the numbers of 18 of the 28 variables were statistically significant, and the total number of words was one of them. 
Karasawa (2001) examined the introduction of argumentative essays written by NNS college students, and found greater numbers of instances of unnecessary types of redundancy in the essays in intermediate score ranges. The study was initially inspired by Blum-Kulka & Olshtain's research on NNS subjects’ speech act realization patterns (Blum-Kulka & Olshtain 1986). They applied Grice's theory of conversation, quantity and manner maxims in particular (Grice 1975), and found relative verbosity observed among L2 speakers at intermediate proficiency levels. They argued that L2 learners needed more words to accomplish the same task than first language (L1) speakers, which could result in a pragmatic failure (Thomas 1983). They identified three types of verbosity: (1) “verbosity per se” – using too many words, which may be caused by “overindulgence in words,” (2) “overinformedness” – trying to explain contextual backgrounds that are irrelevant, and (3) “double messages” – using several contradictory expressions (Blum-Kulka & Olshtain 1986, 175-76). They further argued that verbosity was prevalent in intermediate level speakers because L2 speakers at lower proficiency levels would lack linguistic competence for being verbose while advanced level speakers would gradually approximate native norms. 
If an intermediate proficiency level is where verbosity is most prevalent, examining only high and low proficiency levels may mask the phenomenon because some of the intermediate level learners would be classified at a high proficiency level while others would be at a low proficiency level. It is, therefore, of crucial importance to examine at least three levels of proficiency. 
More instances of occurrences of unnecessary redundancy found in intermediate level essays in Karasawa (2001), to some extent, support Blum-Kulka & Olshtain's (1986) findings. Karasawa (2001) only examined redundancy present in the introduction of student essays, but the study also analyzed overall linguistic features of entire texts of sample essays drawn from the same data. The analysis found textual variations among NNS essays in high, intermediate, and low score ranges, but the statistical method used in the study could not locate specific linguistic items that accounted for the variations. 
The present study selected a few lexical items used for elaboration as target linguistic features of analysis on the grounds that different patterns of elaboration may result in wordiness or terseness in written discourse. 
3. The present study 
The corpora analyzed in the present study were collected from 53 NNS students who were enrolled in freshman composition courses at a large four-year university in the U.S. Another set of data was also collected from 54 NS students in the same composition courses as a comparison group. The data collection was fully integrated into the courses to elicit natural performance, and the data were collected as one of the essay assignments for the courses. 
Two essay assignments from the composition courses, one of which provided the data for this study, were assigned scores by two independent raters, using a six-point holistic scale. The inter-rater reliability was r=0.90. Although the holistic scoring used for this study is not direct measurement of the writer's linguistic competence, Karasawa (2001) found a fairly high correlation between the score and TOEFL, and therefore, the assigned scores were used for classifying NNS writers into three different proficiency levels. The main target population of the present study is intermediate level writers, and to increase the validity of the research, only the essays written by the subjects who received intermediate scores (i.e. 3-4) on both of the essays were selected for the analysis. The rationale for the selection was that receiving intermediate scores on both essays indicated consistent performance of intermediate level learners. Due to the limited numbers of high score and low score essays (5-6 and 1-2 respectively), the same selection method could not be used for high and low proficiency levels. In the NS data, none of the essays received low scores. Thus, only high and intermediate score essays were used for comparison. The numbers of the NNS and NS essays are listed below in Table 1: 
395
Table 1: Numbers of NNS and NS essays used for analysis 
NNS NS 
High 7 12 
Intermediate 15 19 
Low 8 N/A 
First, the total numbers of (1) adjectives, (2) simple adverbs, (3) high frequency prepositions at, in, of, on, and to; (4) that- and wh- clauses and (5) present and past participles used as a noun modifier were counted. Word List, one of the WordSmith Tools (version 3.0, Mike Scott & Oxford University Press, 1998), was used for identifying and counting the numbers of these lexical items; and Concord, another tool of the same program, was used for putting them into appropriate word classes. Then the following sub-categories of linguistic items were sorted from the classified lexical items for making a specific observation of elaboration patterns used for modification of nouns and other clause elements: (1) adjectives that are in attributive use, (2) adverbs as a premodifier, (3) prepositional phrases used as either (a) adverbial adjuncts or (b) a postmodifier to a noun, (4) the preposition of used as a postmodifier in noun phrases (5) to-infinitives used as a postmodifier, (5) that- and wh- relative clauses used as a postmodifier, and (6) present and past participles as a noun modifier. The classification followed that in Comprehensive grammar of the English language (Quirk, Greenbaum, Leech & Svartvik 1991). The total numbers of the two sets of lexical items listed above were compared within the NNS and the NS sub-groups, and across the NNS and the NS sub-groups. 
Next, the number of each of the linguistic items in each essay was converted to the standardized numbers, which indicated the predicted numbers of occurrences of these items in every 330 words (i.e. overall average essay length). Then a correlation study was conducted to examine possible relationships among different linguistic items. Finally, a set of regression analyses was conducted to find out whether any of the target linguistic items could be a strong predictor variable for the essay score. 
4. Results and discussion 
The descriptive statistics of the NNS and NS corpora examined in this study are shown below in Table 2. 
Table 2: Descriptive statistics of the NNS and the NS corpora 
Bytes Token Type T/T Ratio # of Sentences Ave. Sent.Length 
Sub-groups 
NNS high (n=7) 19978 3407 961 28.21 168 20.28 
NNS intermed. (n=15) 19766 3426 828 24.17 165 20.76 
NNS low (n=8) 9777 1689 511 30.15 95 17.78 
NS high (n=12) 23929 4164 982 23.58 211 18.55 
NS intermed. (n=19) 21994 3978 879 22.10 205 19.40 
Type/token ratio is sensitive to text length, which makes the corpus of low score essays incomparable to the other corpora because it is less than half the size of the others. Both the NNS and the NS pairs of corpora of high and intermediate score essays, however, are of similar lengths, and the NNS high score essays show a clearly higher type/token ratio than the NNS intermediate essays. 
As the numbers of the essays and the total numbers of the tokens show, both the NNS and the NS high score essays were generally longer than other sub-groups. This supports the notion that greater length indicates higher quality. The average sentence length, on the other hand, show a slight decline from the intermediate to high score essays in both the NNS and the NS data. 
Table 3 below lists the total number of the first set of lexical items presented above in the previous section. The figures in the parentheses are the percentage of the particular item in the corpus. 
Table 3: Total numbers of lexical items 
396
Sub-groups NNS high NNS intermed NNS low NS high NS intermed. 
(n=7) (n=15) (n=8) (n=12) (n=19) 
Adjectives 179 (5.25%) 158 (4.61%) 81 (4.89%) 161 (3.87%) 193 (4.85%) 
Adverbs 63 (1.85%) 58 (1.69%) 21 (1.24%) 53 (1.27%) 65 (1.63%) 
Prepositions 324 (9.51%) 270 (7.88%) 136 (8.05%) 376 (9.03%) 314 (7.89%) 
Th-/Wh- Clauses 84 (2.47%) 81 (2.36%) 35 (2.07%) 90 (1.27%) 99 (2.49%) 
Participles 9 (0.26%) 7 (0.2%) 2 (0.12%) 6 (0.14%) 9 (0.27%) 
The numbers of the lexical items almost consistently show some increase from the low to the high score essays in the NNS corpora, whereas the NS corpora show more mixed results. 
The numbers of specific target linguistic items used for elaboration of nouns and other clause elements, however, indicate slightly different trends. They are listed in Table 4 below. 
Table 4: Total numbers of target linguistic items used for noun/clause element modification 
Sub-groups NNS high NNS intermed NNS low NS high NS intermed. 
(n=7) (n=15) (n=8) (n=12) (n=19) 
Adjectives 133 (3.90%) 144 (4.20%) 39 (2.31%) 117 (2.81%) 118 (2.97%) 
Adverbs 10 (0.29%) 16 (0.47%) 7 (0.41%) 7 (0.17%) 13 (0.33%) 
Prep. Phrases 94 (2.80%) 69 (2.01%) 34 (2.01%) 118 (2.83%) 99 (2.49%) 
Modifier of 50 (1.47%) 31 (0.90%) 5 (0.30%) 40 (0.96 %) 27 (0.68%) 
To-infinitives 18 (0.51%) 32 (0.93%) 16 (0.94%) 27 (0.65%) 29 (0.73%) 
That Clauses 17 (0.50%) 21 (0.61%) 12 (0.71%) 29 (0.70%) 37 (0.93%) 
Wh- Clauses 27 (0.79%) 29 (0.85%) 23 (1.36%) 26 (0.62%) 26 (0.65%) 
Participles 9 (0.26%) 7 (0.20%) 2 (0.12%) 6 (0.14%) 9 (0.27%) 
The numbers of linguistic items except for prepositional phrases, the modifier of, and participles are higher in the NNS intermediate score essays than the NNS high score essays. Once again, the NS portion shows more mixed results although prepositional phrases and the modifier of seem to be the types of elaboration patterns favored in both the NNS and the NS high score essays. 
A comparison of Tables 3 and 4 reveals that the percentage of adjectives used in an attributive function was the highest in the NNS intermediate essays (91.1%). Less than half of the adjectives in the NNS low score essays, on the other hand, were used in this way (48.15%). This may indicate that a predicative use of adjectives as subject complement is easier for the low proficiency level learners to use. 
All levels of the NNS essays show a higher use of wh- relative clauses than the NS essays. This could be interpreted as transfer of learning from formal L2 writing instruction where use of this linguistic item is often explicitly taught. That- clauses are more highly favored than wh- clauses in the NS essays. 
One additional observation, which is not shown in the table, is that the NNS high score essays contained a slightly higher variety of adjectives (i.e. 80 types in 133 tokens) as compared to the 82 types in 144 tokens used in the NNS intermediate score essays. 
Tables 5.1 and 5.2 below show a part of the results of the correlation studies. The tables list only the items that showed positive or negative correlations of at least 0.250. The complete results are listed in Appendix A. 
Table 5.1: Results of correlation study of linguistic items in NNS essays 
Adj. Participles Adv. That Prep. P. Of 
To-inf. -0.271 
That 0.516 
Prep. P. 0.483 -0.362 -0.250 
Of -0.262 
Wh-Cl 0.675 0.324 -0.272 0.286 
397
Table 5.2: Results of correlation study of linguistic items in NS essays 
Adj. Participles Adv. 
Adv. 0.362 
That 0.256 -0.252 -0.330 
Prep. P. -0.360 
Of 0.279 0.298 
In the NNS corpora, the use of adjectives and prepositional phrases show a high relationship, which allows an interpretation that the combination of these two linguistic features is a common elaboration pattern. The use of prepositional phrases shows a moderate negative correlation with the use of wh- clauses. The use of wh- clauses also shows negative correlation with the use of the modifier of. The use of that- clauses, on the other hand, shows a moderate negative correlation with to-infinitives. These negative relationships could indicate a contrast between clausal versus phrasal elaboration patterns. The writers may have tendencies to choose one type of elaboration over the other. The attributive use of adjectives also shows a moderate negative correlation with the use of the modifier of. Again, the relationship of these two items may indicate writers’ preferences between premodification versus postmodification. 
The results of the correlation study of the NS essays are harder to interpret. No specific choices of elaboration patterns can be inferred from the relationships. 
As the final portion of the analysis, a set of linear regression and multiple regression analyses was conducted to examine whether any of the target linguistic features could make a prediction for the essay score. Table 6 below lists the results of linear regression analysis for the NNS data that were found statistically significant. None of the results of the NS portion of the analysis reached a statistically significant level. 
Table 6: Results of linear regression analysis for NNS data 
Predictor variable Adjusted r2 F p 
Prepositional Phrase 0.147 0.116* 4.82 0.037 
Wh- Clause 0.193 0.164* 6.68 0.015 
Adjective 0.198 0.169* 6.92 0.014 
*P<0.05 
The results indicate that Wh- Clause and Adjective are two stronger predictor variables of the three. As the next step, a multiple regression analysis was conducted to find out whether the overall prediction would improve. Table 7 below shows the results. 
Table 7: Results of multiple regression analysis for NNS data 
t-ratio R2 Adjusted R2 F p 
Step 1 
Wh- Clause 2.20* 
Adjective 2.25* 
0.320 0.269** 6.34 0.006 
Step 2 
Prepositional Phrase 0.76 
Wh- Clause 1.97 
Adjective 1.65 
0.335 0.258* 4.36 0.013 
*p<0.05 
**p<0.01 
The t-ratios of Step 1 indicate that both of the variables individually contributed to R2, and the combination of these two variables improved the prediction of variance in the essay scores to 26.9%. The t-ratios of 
398
r2 
Step 2, on the other hand, indicate that none of these three variables individually contributed to R2, and when Prepositional Phrase variable was added to the equation, the prediction power slightly decreased to 25.8%. Thus the combination of the two variables in Step 1 was found to be the strongest model. 
5. Conclusion 
The sample sizes of this study were very small, and no strong claims nor generalizations could be made from the results. Nevertheless, there were some noticeable tendencies observed in the NNS corpora which are worth interpretations. Also, the fact that the tendencies stated above in the previous section and summarized below in this section were found only in the NNS corpora allows an interpretation that they may, to some extent, characterize different stages of inter-language development. 
First of all, the low score essays were clearly reductive, compared with the high and the intermediate score essays. The numbers of linguistic items used for noun modification were very low except that the percentage of the use of that- and wh- relative clauses was higher than the other two sub-groups. Most noticeably, the percentage of an attributive use of adjectives was very low (48.15% of the entire adjectives in the corpora). The low proficiency level writers used adjectives more often in a predicative function, which showed a strong contrast against the high use of adjectives in an attributive function found especially in the intermediate score essays (91.1%). The percentage dropped to 74.3% for the high score essays. As was pointed out in the previous section, a higher use of wh- relative clauses found in the NNS essays at all three levels may indicate an influence of formal writing instruction. 
In a comparison of the NNS high and intermediate score essays, the high score essays showed (1) greater length indicated by the number of tokens per essay and (2) higher linguistic complexity indicated by a higher type/token ratio and higher numbers of the first set of lexical items examined (Tables 2 and 3). When it comes to the average length of each sentence and the numbers of the target linguistic items used for modification, however, the intermediate score essays showed slightly larger quantity (Tables 2 and 4). One possible interpretation is that the intermediate proficiency level writers elaborate more within each sentence than the high proficiency level writers. 
The slightly larger total number and the slightly fewer variety of adjectives found in the intermediate score essays indicate that the writers at the level used the same adjectives more repeatedly than those at the higher score level. The higher percentage of the modifier of and the lower percentage of adjectives in the high score essay indicates a preference for phrasal postmodification of nouns in the sub-group. 
As was discussed in the previous section, the NNS correlation studies indicated tendencies of preferences of phrasal versus clausal modification, and premodification versus postmodification. Since this portion of analysis was conducted on the NNS essays as a whole group, these tendencies found in the correlation studies may also reflect individual writers’ stylistic choices. 
The interpretation of the results of the regression analyses is not simple. The easiest to interpret is the results that involve Adjective variable. Judging from the results of the other parts of this study, it is understandable that Adjective is a strong predictor variable. Its use was very low in the low score essays and high in the other two sub-groups. It is also understandable that Prepositional Phrase as a single variable is a fairly strong predictor variable because its use is favored in the high score essays. The results that involve the Wh- Clauses, however, may have been slightly skewed because of its prevalent use in all levels of essays and the small number of low score essays. 
L2 writing involves a number of complex linguistic, cognitive, social and cultural factors which include the writer's level of linguistic competence, L1 background, L1 writing skills, preference in topic types, and former educational background. This study focused on the analysis of linguistic features of L2 writing for examining patterns of elaborations. It used small corpora, and the next step is to replicate the study with larger corpora. As has been analyzed extensively in the field of contrastive rhetoric (Kaplan 1966, Connor 1996), elaboration patterns may also be influenced by other factors stated above, such as L1 background because the writer may transfer her knowledge of L1 rhetorical tradition to her L2 writing. The future research should include analyzable sizes of different L1 sub-groups so that a possible L1 influence could also be examined. 
399
Appendix A 
The results of correlation studies of NNS and NS essays 
NNS Essays 
Adj. Participle Adv. That To-inf. Prep. P. Of Wh- Clause 
Adj. 
Participle -0.171 
Adv. 0.166 0.218 
That 0.126 -0.028 0.516 
To-inf. 0.024 -0.065 -0.207 -0.271 
Prep. P. 0.483 -0.362 -0.250 0.004 -0.103 
Of -0.262 -0.095 -0.140 -0.241 -0.104 -0.099 
Wh-Clause 0.222 0.072 0.675 0.324 -0.272 0.286 -0.106 
NS Essays 
Adj. Participle Adv. That To-inf. Prep. P. Of Wh- Clause 
Adj. 
Participle 0.026 
Adv. 0.174 0.362 
That 0.256 -0.252 -0.330 
To-inf. -0.054 0.133 -0.046 -0.075 
Prep. P. 0.223 -0.121 -0.360 0.135 -0.053 
Of 0.279 0.075 0.298 -0.212 0.206 0.053 
Wh- Clause 0.149 0.195 0.022 -0.078 0.164 0.017 -0.063 
References 
Biber D 1885 Investigating macroscopic textual variation through multifeature/mutidimentional analyses. Linguistics 23: 337-360. 
Blum-Kulka S, Olshtain E 1986 Too many words: length of utterance and pragmatic failure. Studies in Second Language Acquisition, 8: 165-180. 
Connor U (ed) 1996 Contrastive Rhetoric: cross-cultural aspects of second-language writing. Cambridge, Cambridge University Press. 
Ferris D R 1994 Lexical and syntactic features of ESL writing by students at different levels of L2 proficiency. TESOL Quarterly 28: 414-420. 
Grice H P 1975 Logic and conversation. In Cole P, Morgan J (eds), Syntax and semantics 3: speech acts. New York, Academic Press, pp 41-58. 
Kaplan R B 1966 Cultural thought patterns in intercultural education. Language Learning, 16: 1-20. 
Karasawa S 2001 Relevance theory and redundancy phenomena in second language learners’ written English discourse: an interlanguage pragmatics perspective. Doctoral Dissertation, University of Arizona. 
400
Quirk R, Greenbaum S, Leech G, Svartvik J 1991. A comprehensive grammar of the English language. New York, Longman. 
Reid J 1990 Responding to different topic types: a quantitative analysis from a contrastive rhetoric perspective. In Kroll B (ed) Second language writing: research insights for the classroom. Cambridge, Cambridge University Press, pp 191-210. 
Thomas J 1983 Cross-cultural pragmatic failure. Applied Linguistics, 4(2): 91-112. 
401

Linguistic Search Engine 
Adam Kilgarriff 
ITRI, University of Brighton, Lewes Road, Brighton, BN2 4GJ 
E: adam@itri.brighton.ac.uk T: 01273 642919 F: 01273 642908 
We propose to build a linguistic search engine, similar in overall design to Google or AltaVista but meeting the specifications and requirements of researchers into language. 
Language scientists and technologists are increasingly turning to the web as a source of language data, because other resources are not large enough, because they do not contain the types of language the researcher is interested in, or simply because it is free and instantly available. The default means of access to the web is through a search engine such as Google. While the web search engines are dazzlingly efficient pieces of technology and excellent at the task they set themselves, for the linguist they are frustrating. The search engine results do not present enough instances (Google sets a limit of 2000) or enough context for each instance (Google generally provides a ca 10-word fragment), they are selected according to criteria which are, from a linguistic perspective, distorting, and they do not allow searches to be specified according to linguistic functions such as lemmatization and word class. 
Each of these goals could straightforwardly be resolved, but approaches to Google to date have gone unanswered. However this suggests a better solution: rather than depend upon existing search engines, it would be possible to set up a linguistic search engine, dedicated to linguists’ interests. Then the kinds of processing and querying would be designed explicitly to meet linguists’ desiderata, without any conflict of interest or ‘poor relation’ role. Once this is set up, large numbers of possibilities open out. All those processes of linguistic enrichment and `linguistic data mining’ which have been applied with impressive effect to smaller corpora could be applied to the web so that web searches could be specified in terms of linguistically interesting units such as lemmas, word classes, and constituents (e.g. noun phrase) rather than strings. Thesauruses and lexicons could be developed directly from the web. The way would be open for further anatomizing of web text types and domains, both a topic of interest in itself and one where strategies would be needed so that web-based lexical resources could be developed for specific text types or domains, or so that the biases of the web could be countered to provide ‘general languages’ resources from the web. All of this can potentially be done for all of the many languages for which there is ample data on the web. 
The web, teeming as it is with language data, of all manner of varieties and languages, in vast quantity and freely available, is potentially a fabulous linguists’ playground. The Linguistic Search Engine will bring that dream closer to reality. 
412

A methodology for inducing a chronology of the Pä li Canon 
Paul Kingsbury 
University of Pennsylvania 
1. Introduction. 
The use of statistics to describe language change has already enjoyed notable success in its relatively brief life. While the use of lexicostatistics and glottochronology met with considerable skepticism, more recent studies have returned more unambiguous results. Foremost among these studies have been the ones exploring the so-called Constant Rate Effect (Kroch 1989 and others). This hypothesis seeks to explain the S-shaped curve which best describes how any particular aspect of a grammatical system changes. In this type of curve, change starts slowly, then accelerates, then slows again, as seen in the lines in figure 1. It was previously thought that this curve reflected the underlying mechanism of linguistic change, that the rate of language change itself changed across time. The Constant Rate Effect does not challenge the description, but instead changes the explanation behind the curve. Instead of the rate of change changing, it is the frequency of the contexts in which a change is found which changes. As a context becomes more frequent, a linguistic change found in or dependent upon that context will also become more frequent. It is also the case that sometime the same change occurs in multiple contexts. Previously it was thought that the rate of change was independent in each context. The Constant Rate Effect shows that this view is false and that the rate of change is constant across all contexts--it is actually the frequency of the contexts which varies. This is one part of the "constant" in the Constant Rate Effect. 
The other important part of the CRE which remains constant is the rate of change itself. While it is undeniable that the curves show an apparent change in steepness, under the proper mathematical viewpoint this change disappears. The mathematical construct necessary here is borrowed from biology and called the logistic, described by the equation 
where p is the rate (probability) of attestation, t is the historical time, s is the (constant) slope of the curve, and e and k are constants.1 One the right-hand side of the equation, the only factor changing is t. When t is small, at the beginning of the linguistic change, the denominator of the fraction will dominate and p will be small, reflecting a low rate of attestation. When t is very large, at the end of the linguistic change, the difference between the numerator and the denominator will be very small and p will be close to 1--meaning a nearly 100% probability of attestation. In between the two extremes, the ratio between the numerator and denominator changes very quickly, reflecting the high "steepness" of the middle of the curve. In this way the logistic equation describes the sigmoid curve. p ek st1 ek st 
The methodology for demonstrating the CRE depends on having a relatively good-sized corpus from which many examples of some change can be extracted, both in the positive (the change having taken place) and the negative (the change having not taken place). The relationship between these numbers gives the 
1E is the mathematical constant, the base of the natural logarithm and approximately 2.718. K describes where the curve intersects the y-axis. As such, it is necessary for graphing the curve but otherwise uninteresting to the linguist. 
413
probability of attestation for that change in the text in question. In order to determine a good time dimension, it is necessary that the texts of the corpus extend across some reasonable length of time, usually on the order of two or three centuries. With these methodological constraints in place, the CRE has been demonstrated for Old English (Kroch 1989), Yiddish (Santorini 1993), Ancient Greek (Taylor 1994), and others. 
Difficulties arise, however, when one or more of the criteria for the CRE methodology is absent or faulty. Such is the case for Pä li. The Pä li literature includes nearly 4 million words in the canonical literature alone, spanning four centuries at the very least.2 Pä li was used at a time of great linguistic change in India, as the elaborate grammatical systems of Old Indic (eg Vedic and Sanskrit) were collapsing and the seeds of the vastly typologically different Modern Indic (eg Hindi) systems were just germinating. For example, between Old and Modern Indic the verb system lost the medial voice altogether, lost several modes such as the subjunctive and optative, and reconstructed the non-present system from single inflected forms to periphrastic constructions. The noun system lost the dual number and basically the entire case system, moving from a nominative-accusative system to an ergative. Any of these changes are countable and therefore could serve as the basis of a diachronic study. What is missing is the clear division into historical strata. That is, while there are a lot of texts spread across a lengthy period of time, the correlation between any given text and any particular period is usually unknown. This study will attempt to show how a rough chronology can be induced based on a known historical change, focussing on the shift from multiple formations for finite past tenses ("preterites" or "aorists") to only a single formation. 
2. Grammatical Background. 
Old Indic, as attested in Vedic and Sanskrit, had a wealth of finite past tenses, including a perfect, an aorist, and an imperfect, each based upon a different stem formation. By the time of even the earliest Middle Indic, however, the perfect was largely gone except for a few fossilized remnants, and the imperfect had been folded into the aorists. The aorists throughout this time were the most productive system, and the system itself showed multiple formations. These are traditionally divided into four types. Of these, two are characterized by the addition of a morpheme -s-, while the other two are not. The root and thematic aorists, collectively known as asigmatic aorists, are generally considered highly archaic and unproductive. In contrast, the sigmatic aorists, formed by either -s- or -is- or phonological variants thereon, are highly productive. This can be seen partly by the fact that every verb which attests an aorist attests one of the sigmatic forms, while the same is not true for the root and thematic formations. Also, looking at later Middle Indic shows that the nonsigmatic aorists have disappeared altogether. Therefore, there is a clear competition between the sigmatic and nonsigmatic formations, a competition which the sigmatic eventually wins. Since the two types of formations are semantically identical, they appear in (mostly) the same contexts, and thus a simple two-way contrast can be counted: does a past-tense sentence attest a sigmatic or a nonsigmatic aorist? For any given text the proportion of sigmatic to nonsigmatic aorists can be determined and this proportion used as the basis of the induced chronology. 
3. Methodology. 
As mentioned before, the Pä li Canon comprises some four million words. There are a number of traditional divisions of these texts, either by style into nine types or into the Tipiöaka ("Three Baskets") and their subdivisions, reflecting a more semantic or functional division. The latter is more commonly used. The Pä li Canon was transmitted orally for several centuries and experienced considerable modification during that time, as divisions and even individual texts were expanded and abbreviated. Therefore, this study has taken the traditional divisions and further refined them into units which have a reasonable chance of stemming from a single historical period. This results in somewhat over 10 000 texts to be dealt with, ranging in size from many two-liners3 to one text of over 20 000 words.4 Of these texts, slightly over 5000 attest at least one aorist of any formation, providing a good source of data for this study. 
2Including later works such as commentaries and histories vastly increases both the size of the corpus and its chronological range, but these works are omitted from the current study. 
3Many of the metrical texts are poems of only a single verse, and the traditional scholarship recognizes that many of 
414
these poems were written by distinct authors. 
Unfortunately, most of these texts cannot be used because they do not attest aorists robustly enough. That is, while the proportion between (eg) 15 sigmatic and 5 nonsigmatic aorists is significant, the difference between one sigmatic and no nonsigmatics is not trustworthy. For example, consider a text consisting solely of two lines of verse, attesting only a single sigmatic aorist. Was that form used because it was the only formation available at the time the poem was written, or because it fit the metre while the corresponding nonsigmatic did not? Such a short text does not allow that question to be answered with any confidence. Therefore, any text which attested fewer than 10 aorists was discarded from the study.5 This left 685 texts,attesting a total of 27145 tokens or an average of 40 aorists per text.6 For each text the number of sigmatic and nonsigmatic aorists was counted, and the proportion of sigmatic aorists to total aorists was calculated. The average proportion was somewhat over 0.76, indicating that about three out of every four aorists in the set of texts were sigmatic. This is perfectly consistent with the idea that the sigmatic aorists are the productive and winning formation in Pä li. 
A working assumption was made that this calculated proportion was a perfect correlate to historical age, and that any text which showed a low proportion was necessarily earlier than any text with a higher proportion. This is at best only approximately true but allows the rest of the analysis to proceed. The texts are ordered by proportion and plotted as Figure 2. 
4The Yamaka, an exhaustive treatise on dualism. 
5Why 10? Why not? 
6This number is slightly skewed by two texts which attested over 1000 aorists each, but since the average number of aorists per text is unimportant to the study as a whole, this is acceptable. Removing these two texts from the calculation still leaves an average of almost 31 aorists per text. 
415
The first thing that is remarkable about this plot is that it does not look much like the expected sigmoid. Specifically, the beginning of the curve rises quickly, rather than the middle as would be expected. This indicates that the time dimension as plotted is inaccurate, and some portions need to be stretched and others compressed. In terms of the chronology of the texts, the assumption had been made with this plot that texts were written at a constant rate (say, one per year). This is clearly false. It is therefore necessary to adjust the assumed date of composition to better fit the expect curve. 
Before this is done, however, it is useful to consider the effect of context. Part of the Constant Rate Effect is the assumption that the context of a particular grammatic feature governs the rate of change. A useful division of contexts is the difference between metrical text (verse) and straight prose. Both these types are well attested in Pä li and in the selected texts of this study. Somewhat over half of the texts used are just prose and about 13% are verse. The remaining third are texts where the verse and the prose is intermixed. It is expected that these three types will show different curves, reflecting different usage of the innovative sigmatic aorists in the different contexts. 
In order to determine an improved chronology, the constants from the logistic equation must be determined. This is done relatively easily by assuming the plotted curve is a "noisy" or partially inaccurate reflection of the true curve, which can be found with a logistic regression, a standard statistical function7. Because of the expected difference between verse and prose, three separate logistic regressions are run, resulting in the three curves seen in Figure 3. 
As expected, the three types of text show different regression curves, indicating that there is a real difference in the usage of sigmatic aorists. (If there had been no difference, the smooth curves above would have been identical.) The difference is not as robust as might have been expected, though. As 416 7My thanks to Sasha Popescul for his help with this part of the 
statistics. 

expected, furthermore, the `mixed' curve is squarely in between the `prose' and `verse' curves, reflecting correctly that mixed texts pattern partly like prose texts and partly like metrical texts. 
With the curves in place and the coefficients of the logistic equation known, it is now possible to reverse the process and solve for the time factor of the equation. That is, the logistic equation from above has an equivalent equation call the logit, from which it is arithmetically trivial to solve for t: 
logit <--> logistic <--> lnpp1 p1 p k stln k tp ek st1 ek sts 
For each text, the proportion of sigmatic aorists to total aorists is fed into this equation. The resulting time values differ quite dramatically from the originals, for two reasons. First, many proportion values were either one or zero, indicating the total absence of either nonsigmatic or sigmatic aorists, respectively. Either of these values produces a mathematical impossibility, for 1/(1-1) is undefinable, often taken as infinite, and ln(0/1) is also undefinable, sometimes taken to be negative infinity. Thus, the ends of the graph are pushed out to infinite values. While this might seem to be a mistake or a byproduct of the statistics, it actually reflects the linguistic reality quite well. There is always a small chance that a nonsigmatic aorist could be used, even at a very late date, indicating that the end point of the grammatical change never actually arrives.8 The inverse case is harder to understand intuitively; it suggests that there is always a possibility of producing a sigmatic aorist even before the innovation begins.9 Fortunately there are not many of these cases--only 3, one from each of the text types. The remaining (sensical) time numbers are scaled and recentered to conform to the old time numbers, and plotted into Figure 4. 
8In terms of a learning paradigm of linguistic change, there is always a chance that a child learner will hear a nonsigmatic aorist, incorporate it into her own language, and pass it on to the next generation. This chance becomes vanishingly small, but never completely disappears. 
9This means effectively that it is (slightly) possible to create a sigmatic aorist, even before that becomes a grammatic formation in the language! 
417
We now have the classic sigmoid curve. Note, however, how the graph is densest somewhat to the right and above the halfway point--timestamps 200 to 400, approximately, or with proportion values between 0.6 and 0.9. This is to be expected. The movement towards the sigmatic aorist had been ongoing by the time of the Pali Canon, having begun even back in Old Indic. What is happening in Pä li is the logical conclusion of this change. The differences between the three types of text are emphasized as well. 
The open question at this point is how this relates to other received wisdom about the chronology of the texts under consideration. There has been a fair amount of research (eg, Winternitz 1956, Pande 1974, Law 1974, Warder 1980, Norman 1983, von Hinüber 1996) concerning the development of the Pä li Canon, although these works are hindered either by their lack of methodological rigor or their narrow scope. These scholars have offered chronologies based on other aspects of Pali linguistics, textual criticism, theology, and internal evidence such as quotations and citations within the Canon. The coordination of these criteria is no small matter. Nevertheless, they do offer a guide to the possible chronology of the texts. On the basis of these studies I have partitioned the Canon into three strata, "old" "middle" and "late", with an additional layer of "mixed" for those texts which are probable combinations of material from multiple strata. Details on this stratification can be found in Kingsbury (2002). Naturally, there is a sizable residue of texts which cannot be assigned to any stratum and are labelled as "unknown." The texts under consideration in this study fall into the five strata in roughly the expected proportions. It is the "old" "late" and "unknown" texts which are the most interesting. 
418
As seen in Figure 5, the "unknown" texts dominate the field, comprising 206 of the 695 texts under consideration. The vertical lines indicate the "centers of gravity" for all texts and the texts classified as "old" or "late". Note that the "late" texts are appearing, on average, slightly later than the "old" texts, seen by the fact that the median of the "old" texts is to the left of the median of the "late" texts.10 This difference can be exploited for reclassifying the "unknown" texts. The idea is simple: once again, the assumption is made that the proportion of sigmatic aorists is an indicator of age. The "unknown" texts with a clearly low proportion, then, can be reclassified as belonging to the "old" stratum and similarly with high proportions and the "late" stratum. The only problem is the definition of "clearly," and that is easily solved by selecting only the outying quarters of texts as defined by the induced timestamps. These divisions are indicated in Figure 5 by the black vertical lines. It might seem more intuitive to use thirds rather than quarters, since we are trying to divide the texts into three strata rather than four. This would be the wrong approach, however, since it is not clear that texts placed near the (eg) one-third mark should necessarily belong to one stratum or the other. There is a margin of error, in other words, and using the one-third and two-thirds divisions would fall within that margin. Dividing by quarters, on the other hand, avoids this margin of error and the hypothesized stratification is more trustworthy. The same applies for texts in the middle--only the middlemost quarter may be trusted. With these caveats in place, however, most of the "unknown" texts can be reclassified, resulting in the distribution seen in Figure 6. 419 10The "middle" texts fall in place between the "old" and "
late", but for the sake of clarity that line is omitted. 

This process allows 141 of the 180 "unknown" texts to be assigned to a stratum. It is important to remember that this is only a tentative chronology for these texts. The single criterion of the aorists, while more compelling than the more subjective tests, cannot be taken as the last word and must be combined with other judgements. The texts above are thus presented as a list of good candidates for further examination. 
4. Accuracy 
In order to make any estimates about the accuracy of this methodology a little sleight-of-hand needs to take place. That is, in an ideal world the proper stratification for each of the "unknown" texts would be known and thus that stratification could be compared to the one reached by this method. But of course, it is the very fact that the texts are of unknown date that necessitates this method in the first place! Instead, a variant of `held-out estimation' can be used. In this strategy, a number of texts which have been assigned to some stratum are temporarily reassigned to the "unknown" stratum and the procedure run as before. These `held-out' texts will then be assigned to a stratum just like any other "unknown" stratum text, and this assignment can be compared to the `correct' stratification previously given. Repeating this several times, with different texts held out for testing, allows for a good estimation of the accuracy of the procedure on the truly unknown texts. 
Results, while not staggering, are encouraging. On average, texts are assigned by to the same stratum by both this methodology and by the subjective criteria 24% of the time. Since choosing randomly would produce 20% accuracy, this is not terribly impressive. However, two of the strata used by the subjective method are "mixed" and "unknown", which this method does not attempt to assign texts to. It is thus incorrect to grade this method on those texts. Using only the "old" "middle" and "late" strata raises the accuracy to about 58%, compared to a 33% (1 in 3) baseline of random selection. It must be remembered, also, that the target stratification is not without error itself. Many of the texts which were supposedly misclassified by this methodology have chronologies which are suspicious at best. For example, the texts 
420
in the Niddesas are all assigned to the "old" stratum on the basis of their being commentaries on the Suttanipä ta, a text which is unanimously agreed to be very old. However, being a commentary on an old text does not require that the Niddesas themselves be old, and indeed many of the Niddesas were restratified into the "middle" or "late" strata. Similarly, the texts in the Suttavibhai ga, being combinations of old rules and later explanations, are a difficult target and should realistically be assigned to a "mixed" category rather than "old." On the converse, texts which clearly belong to one stratum or another are usually correctly assigned to that stratum. For example, the texts of the Suttanipä ta are almost always assigned to the "old" stratum, rarely the "middle" stratum, and never to the "late," while the Apadä nas are usually assigned to the "late" stratum, with the remainder falling in the "middle" stratum. These 'misclassified' texts, then, could be interpreted not as errors but rather as anomalies worthy of closer attention. 
5. Conclusion. 
A little simple mathematics and a single assumption allows for a large number of previously unclassified texts to be assigned to a chronological stratum, by taking a small difference and exaggerating it. In theory, it is possible to use the same methodology to change all of the stratification, but it must be remembered that this study used only one grammatical change out of many, and that the chronology used was based only a probabilistic distribution. That is, it is probable that a text with a low proportion of sigmatic aorists comes from an earlier stratum, but is not necessary for that to be the case. A certain degree of variation is expected at all points in the chronology. While the effects of this variation cannot be compensated for in a study using a single linguistic variable, by combining multiple independent variables should lessen the effect of variation in any single variable. Since, however, language is in constant change, there is a wealth of appropriate variables to measure and induce chronologies from. The comparison and combination of these independent chronologies will result in a single chronology for the Pä li Canon which is both wide in scope and methodologically rigorous. 
6. References 
Bloch, Jules. 1877/1965. Indo-Aryan: From the Vedas to Modern Times. English edition, trans. Alfred Master. Paris: Librairie d'Amerique et d'Orient. 
Fahs, Achim. 1985. Grammatik des Pä li. Leipzig: VEB Verlag Enzyklopädie. 
Geiger, Wilhelm. 1956. Pä li Literature and Language. Second Edition, trans Batakrishna Ghosh. Calcutta: University of Calcutta Press. 
Hinüber, Oskar von. 1977. Notes on the e-preterite in Middle Indo-Aryan. Münchner Studien zur Sprachwissenschaft 36: 39-48. 
Hinüber, Oskar von. 1994. Selected Papers on Pä li Studies. Oxford: Pali Text Society. 
Hinüber, Oskar von. 1996. A Handbook of Pä li Literature. Berlin: Walter de Gruyter & Co. 
Kingsbury, Paul. 2002. The Chronology of the Pä li Canon: The case of the aorists. Unpublished PhD dissertation, Department of Linguistics, University of Pennsylvania. 
Kroch, Anthony. 1989. Reflexes of grammar in patterns of language change. Language Variation and Change 1: 199-244. 
Law, B.C. 1974. History of Pä li Literature. Varanasi: Bhartiya Publishing House. 
Norman, K.R. 1983. Pä li Literature. Wiesbaden: Otto Harrassowitz. 
Pande, Govind Chandra. 1974. Studies in the Origins of Buddhism. Second Edition. Delhi: Motilal Banarsidass. 
Santorini, Beatrice. 1993. Phrase structure change in Yiddish. Language Variation and Change 5: 257-283. 
Taylor, Ann. 1994. The change from SOV to SVO in Ancient Greek. Language Variation and Change 6: 1-37. 
Warder, A.K. 1980. Indian Buddhism. Second edition. Delhi: Motilal Banarsidass. 
Winternitz. 1972. A History of Indian Literature. Vol II, second edition. New Delhi: Oriental Books Reprint Corporation. 
421

Tagging a corpus of Malay texts, and coping with ‘syntactic drift’ Gerry Knowles, Lancaster University, UK Zuraidah Mohd Don, University of Malaya. 
The structure of Malay presents the corpus linguist with an extremely interesting problem. At high syntactic levels, the language is familiar enough, and one can talk of direct objects in transitive constructions, and even of agentless passives. The dominant sentence order is SVO. Parsing at this level is therefore relatively straightforward. The problem is at lower levels, where Malay patterns quite differently from Indo-European languages. If the linguist tries to process Malay using categories and techniques designed for Indo-European, then it comes across as at best confusing and at worst in a state of chaos. Malay is neither confusing nor in chaos; but it does need to be analysed using techniques which are sensitive to its own patterns. 
Conventional tagging is based on the assumption that grammatical class is static, allowing for some ‘ambiguities’ such as telephone as a noun or as a verb. In Malay, grammatical class is dynamic: adjectives can occur as verbs or adverbs and some verbs as adjectives; some verbs can pattern as nouns and others can take on the role of function words. In order to cope with this, we have to make a rigorous distinction between lexical class and the syntactic slots which words fill. Words are given a single class label in the lexicon, and the parser then has to identify cases in which words have drifted away from their default slot. 
Tagging and parsing can be carried out for English as separate if independent processes. , for Malay they have to be treated as at least complementary. 
1. Introduction 
This paper arose out of research supported by Dewan Bahasa dan Pustaka in Kuala Lumpur into the automatic grammatical tagging of a corpus of Malay texts. It became clear from the beginning and for several reasons that tagsets of the kind developed for Indo-European languages would not be suitable for Malay. In the first place, there is no simple way of matching Indo-European grammatical classes with those of Malay. There are words in Malay that seem to correspond to “verbs”, but they are also likely to appear as “nouns”; and similarly “adjectives” appear in the guise of “verbs” or even “adverbs”. “Nouns” can even appear as “prepositions”. In this way the simplest text of Malay gives linguists ample opportunity to disagree over the class of the words. Until we have ascertained the major classes, there is no possibility of adding the kind of refinement required for a modern tagset, which essentially adds more detail to the traditional ‘parts-of-speech'. 
Tagging a European language also has the advantage that the categories behind the tags are familiar and immediately comprehensible to other linguists. Another researcher can use the tags independently as input to a parser, so that tagging and parsing can in practice be tackled as independent operations. The word classes of Malay, by contrast, are unfamiliar both in their membership and in their mode of operation in texts; as a result they require extensive explanation, and there is no reason for another linguist to accept their validity without proof. In practice, therefore, a tagset has to be justified by a working parser that correctly identifies high-level syntactic structures using the information given in the tags. For Malay, tagging and parsing have to be treated as complementary operations, and they have to be formally integrated into a syntactic model of Malay. Our task in this paper is to show how the slippery parts-of-speech of Malay can be linked to categories which can be recognised by a parser. 
422
2. The Malay corpus 
The texts we have studied are taken from the DBP corpus, a collection of Malay texts amounting to some 80 million words, and held by the Dewan Bahasa dan Pustaka in Kuala Lumpur. We have tagged an arbitrary sample of 120,000 words containing literary texts, actually four modern novels. We have made no attempt to take a representative sample, because at this early stage any sample is as good as any other. 
To tag the texts, we have constructed an annotated lexicon of some 15,000 words. This contains all the words found in the texts, and also the simpler forms of complex words. The equivalent for English would be to encounter writings in a text, but also include writing and write in the lexicon. In this way, the lexicon includes a full morphological analysis of each word of the text, and provides the information needed for a future detailed morphological study of Malay. For each word the lexicon contains just one grammatical tag. Given the nature of syntactic drift in Malay, corresponding to what in English tagging is known as “ambiguity”, it might seem perverse to give just one tag per word. On the other hand, it enforces an integrated approach to the design of the tagger and the parser, and an approach to parsing that obviates the need for multiple tags. 
We are developing a rudimentary parser to check the tagging system. Although at this stage it only deals with straight-forward grammatical constructions, it seems to be working not just well, but suspiciously well. In general, Malay uses much less formal grammatical coding than is expected in western languages, and correspondingly relies more heavily on inferencing at the pragmatic level. Our parser makes use of formal grammatical coding, and its success may point to an interesting bias in our literary texts. Malay has long been in contact with languages such as Sanskrit, Arabic and English, and this may have led to a re-modelling of the syntax of literary texts on western lines. 
3. ‘Parts of Speech’ 
European ‘parts-of-speech’ are the accepted point of departure for considering grammatical class in Malay (see Asmah, 1993; Sneddon, 1996). In the initial stages of research, these lead to quick results, but in the longer term they inevitably generate difficulties. It would be easy to jump to the conclusion that the Malay language is in a state of chaos, and that anything goes. In order to identify valid grammatical classes for Malay, it is essential to take a rigorous data-driven approach and identify classes based on the direct evidence of Malay texts. But an exclusively Malay tagset would create its own problems, for it would present Malay as a language sui generis, when in reality a comparison of Malay syntax with that of English reveals a number of remarkable similarities. It is important to avoid imposing English categories on to Malay, but it is also important to recognise the similarities that really exist. 
The ‘parts-of-speech’ approach is not the only way of dealing with grammatical class. Arab grammarians devised a three way classification of ism ‘noun', fi9l ‘verb’ and harf ‘particle', backed up by a morphology based on jithr ‘root’ and wazn ‘frame, template'. These make an efficient analysis of a language with a structure very different from that of English. Abdullah (1974) also has three classes in the morphology, which is sufficient for his morphological analysis, but this approach needs to be complemented by a more detailed model to handle the syntax of Malay. 
All these classifications need far more detail to handle syntactic relations, and tagsets are much larger than traditional classification systems. Our tagset currently contains 119 tags in 19 different major classes. Some of our class labels look like traditional parts of speech, but the underlying definitions are entirely different. 
4. Ambiguity 
In designing a tagset for English and other western languages, the parts of speech are used as a primary classification system. However, many words do not fit neatly into just one category, and may belong to more than one part of speech. This lack of fit is handled by an extension of the notion of homonymy. A good example of homonymy is found in the case of the two unrelated words of the form can, one a 
423
modal verb and the other a noun. By extension, given a word like telephone, we can say that there are two homonyms, i.e. a noun telephone and a verb telephone. An alternative is to say that telephone is used sometimes as a noun and sometimes as a verb. In either case, the homonymy is treated as a source of “ambiguity” which has to be resolved by the grammatical tagger. 
In order to resolve the ambiguity, we have to examine the syntax. After the, telephone is likely to be a noun, and after will it is more likely to be a verb. In other words we use parsing to ascertain the tag of a word in context. At a later stage, a parser uses the tag to make parse. At the very least, there is some overlap here between tagging and parsing. An alternative approach is to make a clear distinction between properties of the lexicon and properties of the text. When we see a word like telephone out of context, we know it usually fits contexts where either a noun or a verb is expected, and which one it is is the property of a particular text. This is not an isolated case, and English has had large numbers of similar examples for hundreds of years. Another set contains words like after, which we might think of as essentially prepositions, but which can also be used as adverbs or subordinating conjunctions. 
This discussion of English is relevant because Malay operates in a very similar way, only much more so. The language simply does not work in the manner presupposed by the part-of-speech classification. It is preferable to study words as entities in their own right, independently of preconceived grammatical classes, and to allow the natural grammatical classes of the language to emerge. 
5. Syntactic drift in Bahasa Melayu 
The class of many words in European languages is made unambiguous by their morphology, and to a lesser extent this is true of Malay. However, in view of the lack of any inflectional morphology, Malay has a large number of simplex forms which belong to no clearly defined class, and appear to ‘drift’ from one class into another. For example, masuk is the normal word for ‘enter', which makes it a kind of verb; but it is used in such a way on buildings and in carparks that it could also be taken to be a noun ‘entrance'. This indeterminacy is also found in words in syntactic contexts, and has to be recognised as an important characteristic of the language. What we here call syntactic drift is so widespread and indeed systematic in Malay that the homonymy explanation is unrealistic. We need a grammatical model that allows us to trace words as they cross from one class to another. In the case of telephone, there are many similar nouns that also occur as verbs, to the extent that English can be said to have a class of “noun-or-verb”, and the tagger and parser together have to decide on a more precise class in any given case. Consider now the behaviour of what in Malay might be called an “adjective”. In buah keras itu ‘that hard fruit', keras behaves as an attributive “adjective”. In buah itu sudah keras sekarang, ‘that fruit has become hard now', where it follows the time marker sudah, it is a predicator and behaves more like a “verb”. In kami bekerja keras ‘we work hard', it is more like an “adverb”. 
We have invented these examples to illustrate different uses of (what we now consider) the same word. For automatic processing we enter a single form in the lexicon with a single tag, in this case kata sifat “adjective”. The parser then has to recognise “adjectives” in different syntactic positions, and identify their differing syntactic roles. For example, in bulan samar ‘dim moon', the “adjective” samar behaves as expected and follows the noun as a modifier. In Seman terlalu gembira ‘Seman was extremely happy', the “adjective” gembira follows the intensifier terlalu. The English translation makes it still look like an adjective, but the structure is one of a large set relating to the verbal group, and our parser treats gembira as a kind of “verb”. In ibu bapanya membangkang keras ‘his mother and father disagree strongly’ the parser treats the “adjective” keras as an “adverb” after the “verb” membangkang . 
To a large extent, we can identify sets of words that drift in the same way, and to that extent drift is a property of the grammar. In the parser, rules normally refer to tags, and words with the same tag function in the same way in different syntactic contexts. There remains a residue of words that drift in an entirely idiosyncratic manner. This situation can be illustrated from examples in English. The drift of telephone from noun to verb applies equally to e-mail and even table and chair. This is part of the grammar of English. Idiosyncratic drift is illustrated by a work like provided, which is a past participle that can be used as a subordinating conjunction. This is a property specific to the word provided, and not a general property of past participles. A Malay example is menarik which is tagged in the lexicon as a “verb” and which can occur in texts as an “adjective”. (It is part of the grammar of Malay that “adjectives” can drift to “verbs”, but the reverse is not generally true.) There are a number of important words referring to past time – including sudah, lepas and lalu – which are important for an 
424
understanding of Malay syntax, and which drift in highly idiosyncratic ways. To handle words like this in the parser, we need rules that operate not on tags, but on individual word forms. 
When individual words are referred to by syntactic rules, it is of course irrelevant what tag we give them initially. For example, o'clock is only found in expressions of the time of day, and no other word patterns anything like it. However we choose to tag o'clock, it is only found after a numeral, as in three o'clock. This is a very useful fact, because it means we can sometimes tag a word according to its normal use, and disregard a specialised use. For example, the Malay expression corresponding to o'clock is a “verb” pukul normally meaning ‘to hit, strike (e.g. a gong)'. To handle an expression such as pukul tiga ‘three o'clock', the parser has to test for the specific word pukul before a numeral, and so the fact that pukul is tagged in the lexicon as a “verb” causes no problem at all. 
Behind the conventional tagging of European languages is the assumption that words belong in principle to one class. In practice they do not, and “ambiguity” is seen as a problem of exceptions that we have to solve. In view of syntactic drift in Malay, this is not a good starting point. Grammatical class in Malay is not a static property, as we assume for European languages, but a property that is by nature variable. 
6. Lexical tags and syntactic slots 
We have so far used scare quotes to refer to classes we might want to think of as “adjectives” or “verbs”. This usage masks a logical inconsistency that is commonly found in linguistic discussion. When we say that big is an “adjective”, we are referring to its lexical class, and we can predict morphologically related forms such as bigger and biggest. We expect to find it labelled as an “adjective” in the dictionary. When, on the other hand, we say that “adjectives” come before “nouns” in English, we are referring to positions in syntactic structure. It so happens that Indo-European languages are in general organised in such a way that there is a highly predictable relationship between lexical classes and syntactic positions, with the result that ambiguity in the use of terms such as “adjective” (and indeed labelling words as adjectives in the dictionary) does not lead to confusion. 
For Malay, these distinctions are essential and must be maintained rigorously and consistently. First we can separate out the lemma, the concept behind the dictionary headword. It is usually defined as a set of words related by inflectional morphology and consequently of the same “part of speech”, so that dictionary headwords can be given a single part-of-speech field. For example, {sing, sings, sang, sung, singing} are all grouped under the headword SING and classed as a verb. This is relatively unproblematical for languages with an inflectional morphology, as long as one does not ask about the status of words such as song and songs. 
For a language like Malay, with only derivational morphology, this is not a sensible thing to do at all. For example, under the headword besar ‘big’ in a dictionary we might find not only terbesar ‘biggest’ but also membesarkan ‘enlarge’ and kebesaran ‘size'. Lemmatising is an important step in processing a Malay text, but it does not and cannot have a fixed link to parts of speech. The lemma BACA ‘read', for instance, has to be defined in such a way as to include “nouns” such as pembaca ‘reader’ and bacaan ‘reading'. Since members of Malay lemmas belong to different grammatical classes, it is meaningless to ask the part of speech of a Malay lemma. Malay lemmas can be classified, but on a quite different basis. Malay dictionary entries do not have a part-of-speech field, and the relationship between words in dictionaries and words in texts is different in Malay and Indo-European languages. 
The distinction between adjective as a syntactic position and a lexical class is essentially that between a container and its contents. In algebra we distinguish the name of a variable from its value, in programming we distinguish a memory location from the information stored in it, and in social life we distinguish a house from its inhabitants. A distinction is sometimes made in linguistics between “slots” and “fillers”, but in grammatical tagging it is not always made clear whether the tags are slots or fillers. In fact we logically need two sets of terms, one for the slots and the other for the fillers. In our work on Malay, we use the term “tag” to label a lexical class, and “slot” to refer to a position in syntactic structure. We maintain the distinction consistently by giving lexical classes Malay labels, and syntactic slots and constructions English labels. 
425
A given slot can be filled by words of different lexical classes. A head noun slot, for example, can be followed by a modifier slot, and the modifier can be a kata sifat ‘adjective', e.g. pintu hijau ‘green door', or a kata nama ‘noun', e.g. pintu rumah ‘house door'. More surprisingly, it can be filled by the simplex form of a kata kerja ‘verb', e.g. pintu masuk ‘entrance door'. By distinguishing lexical classes and slots, we arrive at a straight-forward statement of the distribution of simplex kata kerja forms, without having to resort to a metaphysical process by which a “verb” transforms itself into a “noun” or “adjective”. Nor is there anything remotely ambiguous in a phrase such as pintu masuk. (In a surreal world, pintu masuk could conceivably be taken to mean ‘a door enters'; but such bizarre possibilities can safely be ignored in the practical processing of normal texts.) 
Illustration: kata sifat 
Kata sifat or “adjectives” are a relatively straight-forward class by the standards of Malay, but nevertheless appear in unexpected places, and drift into “verbs” and “adverbs”. They can even follow certain verbs as “adjectives” without becoming “adverbs”. In the phrase berbaju merah ‘wear a red shirt', the “verb” berbaju ‘wear a shirt’ is formed from the “noun” baju ‘shirt', and this noun can still be followed by an “adjective” such as merah ‘red'. This situation is actually an artefact created by the Malay writing system in writing ber- solid with baju: the structure is not strictly ((berbaju)(merah)) but (ber(baju merah)). 
Of the 120,000 words of text which we have tagged so far, we have made a more detailed study of one corpus text of 21635 words, using a prototype parser. Of these words, 1610 (7.4%) were tagged as kata sifat. These appeared in a number of different positions: 
1 immediately after a noun 541 2 as a predicator 505 3 after the verb 215 4 berbaju merah type 25 5 after dengan 9 6 residue 315 
Table 1: Distribution of kata sifat 
The most frequent position is after a noun, where the kata sifat plays the same role as an attributive adjective in English. The second most frequent group corresponds to English predicative adjectives, but in Malay they are much more like “verbs” and this is one of several situations in which it is difficult to make a clear distinction between “adjective” and “verb”. Our parser analyses them successfully by treating them as “verbs”, the main difference being that they do not take the full set of verbal markers. 
A surprisingly large proportion of kata sifat immediately follow the verb. From an English point of view the “adjective” functions like an “adverb”, but one can equally argue that the “verbs” are really like “nouns” in being followed by “adjectives”. There is unlimited scope here for confused argument over the presence or absence of “adverbs” in Malay. Whatever we want to call them, there is a slot immediately after the verb which is commonly filled by a kata sifat. The point is that both head “nouns” and “verbs” can be followed by modifers, and kata sifat can fill the modifier slot. Originally our figures for this structure included formations of the berbaju merah type, but these have now been separated out in Table 1 above. There are just 9 cases in which a kata sifat follows dengan to form an adverbial expression. There remains a residue of cases which our parser is unable to process satisfactorily at this stage, or for which we are unable to extract meaningful information from our database. 
We have so far related a lexical class to syntax. We can also look at these patterns from the point of view of the syntactic slot. The most common nominal modifier is actually another kata nama or “noun”, as in pintu rumah ‘house door', and these are more common than kata sifat. It is often claimed in grammars of Malay that adjectives are for preference linked to nouns not directly, but by using the relative particle yang, so that pintu yang hijau ‘door that is green’ is more natural than the simple pintu 
426
hijau ‘green door'. This is the reverse of what we found in our admittedly very small sample, as seen in Table 2. 
kata nama + kata sifat 388 yang + kata sifat 114 
Table 2: “Adjectival” expressions 
Nor does yang plus kata sifat seem to be a particularly common use of the yang-clause. Of the 655 yang-clauses counted, only 114 had yang immediately followed by a kata sifat, and only another seven were formed with the negative particle tidak ‘not', thus yang+tidak+kata sifat. Note that after yang, the kata sifat is actually in the predicator slot. If it were true that kata sifat are normally used with yang, the most frequent use of kata sifat overall would be as predicators rather than as nominal modifiers. 
While there is much debate over whether Malay has adverbs or not, it is quite clear that, like any other language, it has adverbial expressions. Descriptions of Malay point to an unusual construction in which dengan ‘with’ is followed by a kata sifat, thus dengan betul ‘with correct, correctly', and to noun phrases using the noun secara ‘manner', thus secara betul ‘in a correct manner, correctly'. These two types proved to be rare in our data, occurring only 9 and 4 times respectively, as shown in Table 3. 
(verb +) kata sifat only 215 (verb+) dengan + kata sifat 9 (verb +) secara + kata sifat 4 
Table 3: “Adverbial” expressions 
The dominant type of manner adverbial was formed by using a kata sifat as a verbal modifier. 
It would not be unreasonable to regard the most frequent use, namely as a nominal modifier, as the prototype, so that pintu hijau represents the prototypical use of a kata sifat. The nominal modifier slot can thus be regarded as the ‘home slot’ for kata sifat. But kata sifat drift from the home slot into the predicator slot, the verbal modifier slot, and can follow dengan to form an adverbial expression. As this example shows, drift is not random, but follows clearly defined directions. Kata sifat contrast in the direction of drift, for example, with kata kerja, or “verbs”. A word such as berikut ‘follow’ is a good example of a “verb”, and its home slot is the predicator slot. However, in an expression such as tren berikut ‘following train’ it drifts into the nominal modifier slot. 
7. Conclusion 
Malay is of considerable interest to the corpus linguist because it defies analysis using assumptions taken for granted in work on Indo-European languages. Conventional notions of “parts of speech”, in particular, really belong to Indo-European and can be highly confusing when imposed on Asian languages such as Malay. There are two basic approaches. The first, and alas probably the more common, is to proceed in a Procrustean manner, forcing the words of a Malay text into categories designed for European languages, and ignoring the obvious fact that they do not fit. The alternative is a genuine data-driven approach, identifying the categories that actually appear in the texts. This second approach recognises that grammatical class is organised on different bases in Malay and Indo-European languages, and that grammatical class sequences found in texts are consequently very different. To some extent, Malay grammatical classes are a language-specific property of Malay. Ironically, this point was well known to grammarians earlier in the twentieth century. Lewis (1947: xvii) was quite explicit: ‘Malay words change their function according to context. Be prepared for this, and do not attempt to force the language into a set mould. It will escape'. On this see also the valuable review by Azhar (1988:46-53). Perhaps there is an important insight here that has been lost, and which needs to be rediscovered. 
The slots which grammatical classes fill are by contrast not generally specific to Malay at all, and are in many cases recognisably similar to those of Indo-European and other languages, e.g. noun + adjective, 
427
negative particle + verb, number + noun. As we move to higher syntactic levels, Malay structures increasingly correspond to those found in many other languages. General linguistic terminology can safely be used, e.g. ‘noun phrase', ‘prepositional phrase', ‘agentless passive', or ‘direct object of transitive verb'. If we analyse invented sentences such as saya membaca buku itu ‘I read that book', the high level syntax and the lack of low-level detail give the false impression that the structure of Malay is remarkably similar to English. The syntax of real corpus texts, with their wealth of low-level syntactic detail, offers a real challenge to the corpus linguist, a challenge that any linguist who tries to impose low-level English categories is bound to fail. 
We are still at a very early stage in this research, and our target for 2003 is to complete the processing of our first million words. Our understanding of the basic categories is still growing, and our tagset is still evolving. However, it is already clear that the foundation of a well-defined set of categories for Malay is the clear logical distinction between lemmas, lexical classes and syntactic slots, and the integration of tagger and parser. Setting up word classes and labelling them with tags is complementary to using those tags in a working parser. 
References 
Abdullah Hasan 1974 The Morphology of Malay. Kuala Lumpur: Dewan Bahasa dan Pustaka 
Asmah Haji Omar 1993 Nahu Melayu Mutakhir. Kuala Lumpur: Dewan Bahasa dan Pustaka. 
Azhar M. Simin 1988 Discourse-Syntax of “Yang” in Malay (Bahasa Malaysia). Kuala Lumpur: Dewan Bahasa dan Pustaka. 
Lewis, M.B. 1947 Teach Yourself Malay. London: English Universities Press. 
Sneddon, J.M. 1996 Indonesian: a comprehensive grammar. London: Routledge. 
428

A Study on Abridgement for Spoken Word Titles 
Toshihiko Kubota 
Meiji University, Tokyo 
tk@ff.iij4u.or.jp 
This paper is a part of the survey aiming to show the characteristics unique to the spoken word (=audio book, talking book) medium. The whole survey including this paper claims that titles in the spoken word format are independent enough of the book they are based on so that they should be treated as such. This 
particular paper investigates its textual aspect, especially abridgement. 
The comparison is made among the original text (=book format) and abridged texts for the spoken word titles. The latter text has been prepared by the author listening to the recordings. 
First, the simple survey shows how short common abridged CD titles are. Then next survey investigates how this shortening is done with some hypotheses including the four below. 
(1) If the abridged text is a scaled model of the original, it would be seen in the POS distribution. 
(2) Common modifiers, adjectives and adverbs, are likely to be omitted in abridged texts. 
(3) Adverbials modifying reporting verbs are likely to be omitted in abridged texts. 
(4) POS distribution changes as the story develops, and this should be observable in both texts. 
As a result the survey shows 
(1) statistical information on shortening for spoken words 
(2) editors' concept of text identity and their method to preserving it 
(3) advantages of properly abridged texts 
439

Spoken Academic Lexicogrammar and Discourse Patterns 
David YW Lee 
English Language Institute, University of Michigan 
dvdlee@umich.edu 
While there is a considerable body of research on written academic English, much less is known about spoken discourse in academia and how the various spoken genres impact on and shape the experiences of students and faculty. In particular, learning and speaking in the academy go beyond lectures and tutorials to include study groups, conferences and colloquia, office hour exchanges and dissertation defences. In this paper, I present MICASE, the Michigan Corpus of Academic Spoken English, and the results of some research I have conducted on this specialised corpus at the lexicogrammatical and discoursal levels, showing how the availability of this corpus has now made possible many empirical insights, which have thus far eluded researchers. The theoretical and pedagogical implications of this and previous research on the corpus will then be discussed, and a brief outline of a planned book on the grammar of academic speech by the MICASE team will be given. Future directions for the MICASE project will also be mentioned, with the emphasis on how the enhanced corpus may be used for pedagogical and language testing purposes. 
440

Generic speech act annotation for task-oriented dialogues 
Geoffrey Leech and Martin Weisser 
University of Lancaster and Friedrich-Alexander-Universität Erlangen-Nürnberg 
1. Introduction 
The annotation of dialogues in terms of speech acts (or dialogue acts) has typically followed one of two paths. One path has been to tailor the speech act categories to a specific task or domain, as in the Edinburgh Map Task annotation scheme (Carletta et al. 1995). Another has been to aim at a more general coverage of dialogue, as in the DAMSL (Dialogue Act Mark-up in Several Layers) annotation scheme (Allen and Core 1997). We report here on a pilot project (supported by EPSRC grant GR/R37142/01) which has attempted to achieve a middle ground between the aims of genericity and specificity in data coverage. The SPAAC (SPeech Act Annotated Corpus) scheme has been developed to annotate a range of different kinds of dialogue, but within the general scope of telephone task-oriented dialogue between two people, as contrasted, for example, with general conversation. To try out the generic potential of the scheme, we have annotated different kinds of dialogue, especially British Telecom and The Trainline service-providing dialogues.1 A set of 41 speech-act categories has been applied to these sets of corpus data. 
As an efficient environment for annotation processing of the dialogues, an XML tool (SPAACy) has been developed by Martin Weisser (further details will be found in Weisser forthcoming 2003 and Weisser forthcoming). Although the 15-month duration of this pilot project did not allow time for a full-scale software development programme, SPAACy enabled us to develop a semi-automatic XML annotation procedure whereby the dialogues were assigned speech-act tags with gradually improving accuracy, before the dialogues were manually post-edited to achieve consistency with practices laid down in a speech-act annotation scheme manual.2 The result was that 1219 dialogues were annotated, amounting to over 182,300 words.3 
In addition to an initial assignment of speech-act values, the annotation tool also assigned four supplementary values for each speech act: 
(a) A form category: declarative, yes-no question, wh-question, imperative, fragment, yes, no. 
(b) A polarity category: positive, negative 
(c) A topic category: e.g. (relating to train journeys) location, name, day, date, time, railcard 
(d) A mode category: e.g. semantic categories such as deixis, probability, reason. 
1 We are grateful to both British Telecom and The Trainline for making this data available for the pilot project. The OASIS and 150 corpora were supplied by British Telecom in a transcribed form. The transcription of The Trainline dialogues was undertaken, using a transcription scheme modelled on that of the BT Corpora, by Paul Baker at Lancaster University. The initial segmentation and manual post-editing of the dialogues were undertaken at Lancaster by Jeremy Bateman, Jean Forrest, Costas Gabrielatos, David Hooper and Ursula Weinberger. Most dialogues went through a further manual post-editing by Geoffrey Leech, to improve consistency. We acknowledge the valuable advice of Jean Carletta, who with Amy Isard had previously undertaken a separate XML speech act annotation of part of BT's OASIS corpus (consisting of 325 dialogues). The intention is to combine the more domain-specific Edinburgh annotations with the generic Lancaster annotations, and to make this part of the corpus available as a separate entity. 
2 The Speech Act Annotation Scheme document is available on the UCREL website: http://www.comp.lancs.ac.uk/ucrel/ projects.html#spaac. In more detail, the main procedures performed or enabled by SPAACy were: 
(1) Automatic conversion of the text files containing the transcription to XML mark-up. 
(2) Interactive segmentation of the dialogue into utterance-units (here called C-units). 
(3) Automatic assignment of speech-act categories, together with other categories giving information on polarity, topic and mode. 
(4) Manual post-editing and correction of speech-act tags. 
3 This set of dialogues was composed as follows: 
BT operator calls: 643 dialogues, c. 85,840 words 
BT customer services calls: 541 dialogues, c. 72,220 words 
The Trainline dialogues: 35 dialogues, c. 24,246 words 
The word estimates are based on a sample count. 
441
These values supplied syntactic, semantic and pragmatic information which helped the tool to ‘home in’ on the correct speech act. However, with a few exceptions, these values have not been post-edited after the automatic assignment process has run, so that many errors and information gaps remain. It is likely that most of these supplementary values will be deleted before the corpus is made available to the research community. 
2. The rationale for the project 
Over the last decade there has been an intense interest in the development of spoken dialogue systems (Grice et al. 2000, Leech and Weisser 2003) with the objective of enabling humans to interact with a computer using the medium most natural and congenial to them: human speech. While existing working systems fall short of natural dialogue in many ways, it may be assumed that one of the means to the goal of human spoken interaction with machines will be achieved through the close study and modelling of real human-to-human dialogues. This is more likely to succeed to the extent that the dialogue is relatively constrained, task-oriented and transactional. A first step in this direction is the collection of corpora of such dialogues, e.g. calls made to a call centre or a telephone service provider by a member of the public. The dialogues we have annotated in the present project are of this kind. The BT OASIS and 150 corpora consist of dialogues from two rather different kinds of telephone calls: ‘100’ calls to the operator, and ‘150’ calls to BT residential customer services. The Trainline dialogues are of calls to a call centre providing railway timetable information, ticketing and seat reservation services. These three types of dialogue are sufficiently diverse to test the genericity of the annotation scheme to a reasonable extent, but we still intend to apply the scheme experimentally to a wider range of dialogue data: a task which has so far been only partially accomplished. 
Once a suitable corpus has been collected and annotated, it can be used as a training corpus and/or test corpus for the modelling of dialogue behaviour using either statistical, rule/structure-based or hybrid language models. (On statistical dialogue modelling see for example Kita et al. 1996, Nagata and Morimoto 1994, and Reitlinger and Klesen 1997.) Some annotation schemes (e.g. Edinburgh HCRC Map Task scheme) already build into an annotated corpus a higher-level analysis of structure, analogous to the phrase markers of grammatical structure, or to the Sinclair and Coulthard (1975) framework for analysing classroom discourse. However, our pilot project confined itself to the more limited goal of speech-act annotation on a single level of sequential dialogue acts, leaving to future research the formulation of grammars adding hierarchical levels of structure. 
The paucity of existing corpus data for the training and testing of dialogue models was one of the motivations for our speech-act annotation project. Although call centres have burgeoned in recent years and carrying out service transactions by telephone dialogue is one of the growth industries of the present day, the fact remains that such datasets have been difficult to come by for research, particularly in the U.K. This is partly because of problems relating to confidentiality and the Data Protection Act. Another reason is that firms investing in dialogue system research and development do not particularly want to share their data with others. No doubt this data logjam will be overcome, but for the present, we are fortunate that British Telecom and The Trainline have agreed that their dialogue data may be made available to the research community.4 It seems important to give priority to the annotation of sufficient dialogue data for the development of robust dialogue models, including statistical models. In fact the SPAAC annotations have been applied to a large number of dialogues: 1219. This number compares very favourably with all but two of previous speech act annotation projects.5 
While the development of computational models of dialogue was the primary purpose behind the project, we would emphasise the potential of the data for linguistic research in general – for example, to mention one topic, the study of politeness phenomena in these corpora has an obvious appeal. Other topics waiting to be investigated are grounding and dysfluency. Plans are now being made for the release of this data to researchers, and it is intended that these arrangements will be announced in the near future via the UCREL website (http://www.comp.lancs.ac.uk/ucrel/projects.html). The dialogues have been subjected to anonymization routines ensuring that no confidential information is preserved in 
4 BT makes this data available to the community only under contract, to ensure regulatory requirements are met, and to cover data protection issues. 
5 The European MATE (Multilevel Annotation Tools Engineering) project supports 16 dialogue act coding schemes, of which the majority had been tested on less than 100 dialogues. The two major exceptions in this regard are SWBD-DAMSL and VERBMOBIL, which had been applied respectively to 1155 and 1172 dialogues (Klein et al. 1998). 
442
the final form of the corpora. It should be pointed out, however, that no sound files accompany the transcribed record of the dialogues. 
3. Exemplification of the corpus data 
To illustrate the corpus data, mark-up and annotation, we reproduce the first two turns of one of the The Trainline dialogues. For clarity, in this sample, the speech act values and the actual words spoken are printed in italic: 
<turn id="1" speaker="A"> 
<utt id="1"> 
<frag sp-act="greet" polarity="positive" topic="opening" mode="greet"> 
good afternoon 
</frag> 
</utt> 
<utt id="2"> 
<frag sp-act="identifySelf" polarity="positive" topic="intro" mode="greet"> 
Virgin trainlines Sandra speaking 
</frag> 
</utt> 
<utt id="3"> 
<q-wh sp-act="reqDirect" polarity="positive" topic="journey-preference" mode="open"> 
for which journey do you wish to purchase a ticket 
</q-wh> 
</utt> 
</turn> 
<turn id="2" speaker="B"> 
<utt id="4"> 
<decl sp-act="direct" polarity="positive" topic="to-location-from" mode="closure"> 
it's from London Euston to Birmingham International 
</decl> 
</utt> 
</turn> 
Within the structure of the overall dialogue, there is a hierarchy of three tags, one embedded in another. The outermost is the “turn” tag, showing the alternation between speakers A (the service provider) and B (the caller). Within this is the “utterance” tag. An utterance may contain a single speech-act unit, or more than one (if one or more of them are discourse-marker or yes or no tags, which may be considered peripheral to the utterance). The speech acts themselves are handled as attribute values of the C-unit, a grammatical unit which corresponds to a speech act in extent, and which is labelled by one of the syntactic class labels “decl” for declarative, “q-wh” for wh-question, etc. 
Although this example may suggest a highly ritualised and predictable style of question-and-answer sequence which could be handled by a simple template-filling device, the following section from another of The Trainline dialogues illustrates the greater complexity which arises in a negotiating context with mixed initiatives, competing goals, and frequent dysfluencies. The mark-up has been stripped away from this example, to enable the key elements for this presentation (turns, words spoken, and speech acts,) to show up clearly. [Note: # indicates a pause, and #<xs> indicates a pause of x seconds.] 
A1: … for which journey do you wish to purchase a ticket [reqDirect] 
B2: hi [greet] 
i wanna buy em a ticket for Edinburgh [direct] 
to leave em going on the ninth of October [refer] 
A3: travelling to [reqInfo] 
A4: Edinburgh [answ] 
B5: from [reqInfo] 
A6: well {#} [init] 
em is there a train from Liverpool [reqInfo] 
B7: now [init] 
do you hold a current credit or debit card [reqInfo] 
B8: yes [answ] 
A9: and is it just for one person [reqInfo] 
B10: it is [answ] 
yes [answElab] 
A11: so it's Friday the ninth of October (yes) from Liverpool to Edinburgh # 
[confirm] 
do you have a railcard at all [reqInfo] 
B12: i don't [answ] 
no [answElab] 
A13: what time around would you like to depart [reqDirect] 
443
B14: well [init] 
saying actually i've got a couple of queries first # [raise-issue] 
me friends are going up and they're leaving from Wigan at 12 28 to Edinburgh 
[inform] 
but they said there wasn't # there was only one a day from Liverpool [inform] 
so that's why I wanna know what time the one from Liverpool is [inform] 
A15: well [init] 
you'll be able to get the train from Liverpool Lime Street at 11 35 (yes) 
[inform] 
you have to change at Wigan Northwestern anyway (right) to arrive at 12 22 
[inform] 
then it's the 12 28 from Wigan to arrive in Edinburgh at 15 30 [inform] 
B16: oh right [ackn] 
so [init] 
the train from Liverpool goes to Wigan anyway [confirm] 
yeah [init] 
A17: yeah [ackn] 
B18: alright [ackn] 
so [init] 
i may as well get on with them in Wigan then [express-possibility] 
how much is the ticket [reqInfo] 
can you tell me [reqInfo] 
A19: you want a ticket from Wigan now [confirm] 
B20: yeah [answ] 
on that train 12 28 [answElab] 
A21: #<5s> i'll just check that for you [inform-intent] 
B22: cheers thanks [thank] 
A23: #<14> there's not any advance purchase tickets left for the ninth now # 
[inform] 
the cheapest fare that's going to be available is the saver return [inform] 
and that's 65 pounds 30 [inform] 
4. The speech act categories 
Previously, a variety of annotation schemes have been devised for different genres of dialogue data, varying from a wide range of non-task oriented dialogue (for example, the SWBD-DAMSL scheme – Jurafsky et al 1997), to a single specialized task, such as the HCRC Map Task (Carletta et al 1995). The varying order of complexity of these schemes is a reflection of the functional spread of dialogues they are meant to capture. For example, the SWBD-DAMSL scheme, intended for relatively unconstrained conversational dialogue, began with a set of 200 speech-act categories (subsequently narrowed down to 42 –Jurafsky et al 1997) whereas a much smaller set of categories (12) was devised for the ‘laboratory’ dialogue task represented by the HCRC Map Task Corpus. For a scheme designed to be generically applicable to task-oriented dialogue, we devised after experimentation a category set almost of the same size as the SWBD-DAMSL reduced set, of 41 tags: sufficiently rich to capture a wide range of dialogue types, yet sufficiently parsimonious to engender controllable and consistent annotation practices. Table 1 summarises the set of 41 categories: 
Table 1 – SPAAC Classified List of Speech Acts [O = the other speaker; S = self] 
label superordinate class broadly characterised as 
accept mainly responding responding in an active positive way 
ackn(owledge) mainly responding signalling decoding, understanding 
answ(er) mainly responding answering a question 
answ(er)Elab(orate) mainly responding elaborating the answer to a question 
appreciate mainly responding expressing appreciation 
bye interpersonal management saying farewell; closing a dialogue 
complete dialogue control completing O's move 
confirm responding / initiating repeating what O has said/implied 
correct dialogue control correcting what O has said 
correct-self dialogue control correcting one's own utterance 
direct mainly initiating eliciting O's non-verbal response 
directElab(orate) mainly initiating elaborating a “directive” move 
echo dialogue control repeating O's words for verification 
exclaim expressive expressing emotion 
expressOpinion expressive expressing an opinion/evaluation 
expressPossibility expressive expressing a possibility 
expressRegret expressive expressing regret; apologizing 
expressWish expressive expressing a wish or desire 
greet interpersonal management greeting: hello; hi; good morning etc. 
444
hold dialogue control signalling to O to hold the line 
identifySelf dialogue control identifying S's name/institution 
inform mainly initiating conveying information/awareness 
informIntent mainly initiating indicating S's intention 
informIntent-hold dialogue control (a combination of both moves) 
init(ialize) dialogue control initiating a new phase of the dialog 
negate mainly responding responding negatively 
offer mainly initiating offering a service to benefit O 
pardon dialogue control signalling the need for O to repeat 
raiseIssue mainly initiating raising an issue (non-informative) 
refer mainly initiating indicating a reference 
refuse mainly responding responding negatively to an offer, etc 
req(uest)Direct mainly initiating requesting a directive 
req(uest)Info(rm) mainly initiating requesting verbal information 
req(uest)Modal mainly initiating requesting permission, advice, etc 
selfTalk external to dialogue goals speaking to oneself (S) 
suggest mainly initiating proposing action by O (or O and S) 
thank interpersonal management thanking 
thank-bye interpersonal management thanking and saying farewell 
thirdParty (talk) external to dialogue goals speaking to s.o. who is not S or O 
unclassifiable (unspecified) (an unclassifiable move – e.g. a joke) 
uninterpretable (unspecified) (because incomplete or incoherent) 
[Note: Parentheses enclose what has been omitted from the abbreviated labels represented in the annotation schme.] 
The second column of the table assigns the 41 moves to a set of 5 superordinate categories, including metadiscoursal categories containing speech acts concerned with the control of the dialogue and the management of interpersonal relations. 
This set of categories owes a great deal to preceding proposed sets of speech act categories, particularly those of DAMSL (Allen and Core 1997) and VERBMOBIL (Alexandersson et al. 1997). The distinction between ‘mainly initiating’ and ‘mainly responding’ particularly applies to speech acts concerned with the goals of the dialogue itself, and recalls a similar classification in the DAMSL and HCRC annotation schemes. However, it is less absolute: for example, the apparently anomalous “confirm” speech act type is janus-faced, i.e. both initiating and responding. It confirms what has previously been said, and at the same time seeks an opportunity for further confirmation or acknowledgement from the other speaker. Another case is that of discourse markers such as right, alright, and okay, which are frequently used to demarcate the end of the main business of the phone call: in such a context, they appear both to respond (signalling the end of previous discussion) and to initiate (pointing forward to a new beginning). They could have been assigned both ‘ackn’ and ‘init’ labels, but in practice we decided to mark them as ‘init': their demarcation function appears to apply to the previous transaction as a whole, rather than to a preceding individual turn. 
5. Conclusion 
This has been a pilot project, and therefore lacks conclusiveness: there are improvements and further research tasks to undertake. For example, should more than one speech act label be assigned to the same piece of dialogue? We avoided this except in very clear cases. Also, we did not deal conclusively with the problem of indirect speech acts, where again one could argue that more than one label should be assigned to the same speech unit, one for the surface or ‘literal’ interpretation, and one for the underlying pragmatic intent (as in Stiles's annotation scheme – Stiles 1992) . Although there is more to be done on the development of the SPAACy tool, it is hoped that these can be added incrementally, either by ourselves or by other researchers working on the data. Meanwhile, these annotated corpora will already be a useful resource for research on task-oriented human-human dialogue. 
References 
Allen, J and Core M 1997 Draft of DAMSL: Dialog Act Markup in Several Layers http://www.cs.rochester.edu/research/cisd/resources/damsl/ 
Alexandersson J, Buschbeck-Wolf B, Fujinami T, Maier E, Reithinger N, Schmitz B and Siegel M 1997 Dialogue Acts in VERBMOBIL-2. VM-Report 204, DFKI GmbH, University of Saarbrücken. 
Carletta J C, Isard A, Isard S, Kowtko J, Doherty-Sneddon G, Anderson A. (1995). The coding of dialogue structure in a corpus. In J. A. Andernach, S. P. van de Burgt, & G. F. van der Hoeven (Eds.), 
445
Proceedings of the Twente Workshop on Language Technology: corpus-based approaches to dialogue modelling . Universiteit Twente, Enschede, The Netherlands. 
Gibbon D, Moore R, Winski R (eds) 1997 Handbook of standards and resources for spoken language systems. Berlin, Mouton de Gruyter. 
Grice M, Leech G, Weisser M, Wilson A 2000 Representation and annotation of dialogue. In Gibbon D, Mertins I, Moore R (eds) Handbook of multimodal and spoken dialogue systems Boston, Kluwer: 1-101. 
Jurafsky D, Shriberg L, Biasca D 1997 Switchboard SWBD-DAMSL shallow-discourse-function annotation: coders manual. http://www.colorado.edu/ling/jurafsky/manual.august1.html. 
Kita K, Yoshikanzu F, Masaki N, Tsuyoshi M 1996 Automatic acquisition of probabilistic dialogue models, ICSLP-96, Philadelphia pp196-199. 
Klein M, Bernsen N O, Davies S, Dybkjaer L, Garrido J, Kasch H, Mengel A, Pirelli V, Poesio M, Quazza S, and Soria C 1998 Supported coding schemes. MATE Deliverable D1.1, LE Telematics Project LE4-8370. 
Nagata M, Morimoto T 1994 First steps toward statistical modeling of dialogue to predict the speech act type of the next utterance Speech Communication, 15: 193-203. 
Reitlinger N, Klesen M 1997 Dialogue act classification using language models. In Proceedings of the 5th European Conference on speech communication and technology (Eurospeech ’97), Rhodes, Greece. 
Sinclair J, Coulthard M 1975 Towards an analysis of discourse: the English used by teachers and pupils. London: Oxford University Press. 
Stiles, W. B. 1992 Describing talk: a taxonomy of verbal response modes. Beverly Hills, Sage. 
Weisser M forthcoming 2003 SPAACy – a semi-automated tool for annotating dialogue acts. International Journal of Corpus Linguistics 8(1). 
Weisser M forthcoming Determining generic elements in dialogue. 
446

The curse and the blessing of mobile phones – 
a corpus-based study into Polish and American rhetoric strategies 
Agnieszka Leñko-Szymañska 
Department of English Language 
University of £ódŸ 
lenko@krysia.uni.lodz.pl 
The study applies the content-analysis methodology to identify the differences in the rhetoric strategies applied by Polish and American apprentice writers in their argumentative essays. As part of the project aiming at the compilation a Polish corpus of learner English, the data was collected from 79 Polish first-year students of English at the Institute of English Studies, University of £ódŸ, and their 80 American counterparts, freshman non-English majors at South West University in Marshall, Minnesota. Both groups of students were asked to write an essay in almost identical conditions and on the same topic: “The mobile phone – the curse or the blessing of the end of the 20th century”. Since the Polish and the American essays differed greatly in their treatment of the topic, the decision was made to pursue this observation in a more rigorous way. 
The simplest method was chosen to analyse the differences between the two groups of essays: the comparison of wordlists also known as the key-word analysis. According to Scott (2000) key words are a good indicator of the ‘aboutness’ of a text, thus the procedure seemed the appropriate first step to pinpoint the discrepancies between the samples. A list of 321 key words was produced in Wmatrix (the corpus-analysis environment developed at Lancaster University). The items were further sorted into those overused by the American students (145 words and phrases) and those overused by the Polish students (176 items). Both lists were examined in search for the key words which could point to the re-occurring themes in the samples (and thus the ‘aboutness’ of the texts) as well as to other dissimilarities between the two groups of essays. The examination was supported by the scrutiny of the concordance lines of the key items. The analysis led to interesting findings concerning the differences in the choice of arguments and their organisation. Some of these differences are not surprising and can easily be explained, but some reveal unexpected facts about the rhetoric strategies applied by both groups of students. 
The lists demonstrate the expected variation between the groups in the choice of arguments, which is attributable to the cultural differences between Poland and the U.S., as well as the differences in the life style of the two groups of students. Thus, the theme of the low cost comes up as one of the advantages of mobile phones in America, especially for making long-distance calls, whereas such argument does not surface in the Polish data, since mobile phone calls are expensive in Poland. The differences in life styles are reflected by the fact that in the American essays the most frequent theme is driving (the experience that most Polish 19-year-olds do not have yet, at least not on regular basis), whereas the Polish list contains the word mountains associated with the theme of emergency situations (trekking in the mountains is a popular holiday activity among Polish students). 
Although the findings mentioned above specify the differences among the two groups of essays, they hardly reveal any interesting facts about the samples, since such differences could only be expected. However, further scrutiny of the key word lists brings into light a deeper and more intriguing variation between the groups of essays. The American list contains many items pointing to several readily identifiable themes which can be easily explained by cultural differences and differences in life style, at the same time the Polish list contains very few of these identifiable and explainable themes. With the exception of the theme of using texting as a convenient way of contacting people, there are few items in the Polish list whose keyness can be attributed to the cultural differences. On the other hand, the most re-occurring theme in the Polish data is the concept of heath represented by items such as health, harmful, and diseases. Another frequent theme in the Polish essays is labelled general and is composed of items such as inventions, human, and development. In fact, this seems to be the most striking difference between the Polish and the American data. While the American students answered the essay question by making reference to their own life experience such as driving, or problems with service, the Polish learners talked about civilisation and health hazards. 
This discrepancy is further supported by other key words in the samples. The American list comprises such items as the pronouns I and my, whereas the Polish list includes the pronouns we, our, and us. Moreover, the American list incorporates names of family members (father, dad and mom), and other items pointing to the familiarity such as house, home, and Minnesota, but there are no similar themes in the Polish list. Finally, the Polish list abounds in linking expressions, as opposed to none in the American list. 
All these findings point to the fact that the Polish students approach the topic on a more general level, and tend to structure their arguments better, whereas American students relate to their own experience in tackling the 
447
problem and do not pay much attention of the argumentation structure. One explanation of such variation could be that the Polish learners did not own mobile phones at the time of writing. Unfortunately, no data is available on this issue (since the essays were collected with an entirely different purpose); yet this explanation seems highly unlikely, because the mobile phone is a commonplace device among Polish teenagers. 
The study has demonstrated that adopting the text rather than the language approach to the analysis of corpus data (Scott 2000) can bring revealing insights concerning the variation in the argumentation structure of Polish and American essays. This variation is not a result of ‘nativeness’ and ‘non-nativeness’ in language use but represents deeper differences in the rhetoric strategies applied by Polish and American students. 
References: 
Scott, M. 2000 'Focusing on the Text and Its Key Words', in L. Burnard & T. McEnery (eds.), Rethinking Language Pedagogy from a Corpus Perspective, Frankfurt: Peter Lang., pp. 103-122. 
448

Porting an English semantic tagger to the Finnish language 
Laura Löfberg1, Dawn Archer2, Scott Piao2, Paul Rayson2, 
Tony McEnery2, Krista Varantola1, Jukka-Pekka Juntunen3 
1University of Tampere, Finland 
2University of Lancaster, UK 
3Kielikone Oy, Finland 
Abstract 
Semantic annotation is an important and challenging issue in corpus linguistics and language engineering. While such a tool is available for English in Lancaster (Wilson and Rayson 1993), few such tools have been reported for other languages. In a joint Benedict project funded by the European Community under the ‘Information Society Technologies Programme', we have been working towards developing a Finnish semantic tagger that will parallel the existing English semantic tagger. The intention is to avoid building a completely new system but to bootstrap using the existing software and the largely hand-constructed English lexical resources. In this paper, we report on our work to date, which includes (i) a comparative study of the grammar of English and Finnish, (ii) the tagging of an English-Finnish parallel corpus, and (iii) the building of a Finnish lexicon using existing lexicons and software such as a Finnish-English-Finnish machine-translation system, Finnish dependency parser and morphological analyser, etc. This paper also discusses some challenging issues that have arisen during the construction of the parallel semantic tagging system between English and Finnish, namely, the complications caused by the widely different grammatical systems of the two languages. We believe that our work will provide a valuable experience for the community working on cross-language annotation schemes. 
1. Introduction 
In the past decades, numerous tools have been developed for annotating linguistic information in corpora (i.e. part-of-speech taggers, syntactic parsers, etc.). Fewer tools have been developed to undertake semantic annotation, that is, the automatic assignment of semantic categories to words in a running text.1 One such tool is the English Semantic Tagger (henceforth EST) developed at Lancaster University (Wilson and Rayson 1993). 
The EST employs a set of semantic tags loosely based on Tom McArthur's (1981) Longman Lexicon of Contemporary English. That said, the tagset has been considerably revised in the light of practical tagging problems met in the course of previous research. The revised tagset is arranged in a hierarchy with 21 major discourse fields expanding into 232 category labels. The following table shows the 21 labels at the top level of the hierarchy.2 
Table 1: The top level of the USAS system 
A 
general and abstract terms B 
the body and the individual C 
arts and crafts E 
emotion F 
Food and farming G 
government and public H 
architecture, housing and the home I 
money and commerce in industry K 
entertainment, sports and games L 
life and living things M 
movement, location, travel and transport N 
numbers and measurement O 
substances, materials, objects and equipment P 
Education Q 
language and communication S 
social actions, states and processes T 
Time W 
World and environment X 
psychological actions, states and processes Y 
science and technology Z 
Names and grammar 
Being a hybrid system, the EST combines lexicons and various disambiguation template rules. The entries of the lexicons provide possible semantic categories for words or multi-word units (MWU) – mostly idioms and fixed expressions. Given a key word or MWU, the disambiguation algorithm matches near contexts against template rules in 
1 Such tools are important for corpus linguistics and language engineering, and can potentially be applied to a wide range of corpus-based studies and practical NLP tasks. 
2 For the full tagset, see http://www.comp.lancs.ac.uk/ucrel/usas/. 
457
order to determine correct semantic categories for the given key word/MWU. The EST is reported to have obtained 92% accuracy on general English texts (Rayson and Wilson 1996). 
We have been working on improving the EST and constructing an equivalent Finnish Semantic Tagger (henceforth FST) as part of the Benedict project (the general aim of which is to promote the use of language tools in practical application).3 In order to avoid re-inventing the wheel, the architecture of the FST is being designed so that it will parallel that of the EST. In other words, we envisage that a single program package (of parallel lexicons and rule sets) will be able to process both English and Finnish (by switching linguistic components). 
We cannot construct a Finnish lexicon for the FST without first investigating the similarities and differences between the lexical structure of Finnish and the lexical structure of English, of course. A focus of this paper, then, is to highlight some of the grammatical features of Finnish in comparison to English. In addition, we discuss the proposed design of the Finnish lexicon, in particular, our intention of using existing lexicons and software to build the Finnish lexicon, before highlighting some of the practical issues that have arisen during the construction of the lexicon. 
2. Distinct lexical features of Finnish language 
Finnish is a Finno-Ugric language belonging to the Uralian language family, and is spoken mainly in Finland and by the people of Finnish origin living in Sweden and other countries. It uses the Latin alphabet set similar to the English alphabet with three exceptions: A, Ä and Ö (the phonetic values of the letters are those of the International Phonetic Alphabet). 4 However, unlike English spelling, Finnish spelling is mostly phonetic, i.e. in principle, Finnish words are pronounced as they are written. 
Finish is also very distinct from English in terms of its grammatical features. Indeed, English is by large an analytic language whereas Finnish is an agglutinative, synthetic language. Since we aim to build the FST lexicon to parallel that of the EST, the following sections compare some of the lexical features of the two languages in more detail. 
2.1. Heavy morphological affixation 
Due to its agglutinative nature, the Finnish language relies heavily on morphology. Generally, what is expressed in English through syntactic structure is expressed via morphological affixation in Finnish. For example, prepositions are used to express relations between words in English. In Finnish, case endings are used to express the same relations (the case endings are attached to the base forms of the words). Finnish also uses morphemes to express plural and possessive relations (the morphemes are added to the stems of nouns as suffixes), and to denote morpho-syntactic concepts pertaining to verbs, including tense, subject-conformant verb inflection, verb nominalization, interrogative form and some pragmatic verb variants. Such inflectional/derivational changes can also convey information about mood (indicative, conditional, potential, imperative) and voice (active, passive). 
The flexible inflectional/derivational morphological changes and the high number of morphemes enable Finnish nouns, verbs and adjectives to carry a very high information load. Indeed, as the transliteration of the following example illustrates, a single noun can carry a meaning equivalent to an English phrase (in this case, also in my plants). 
kasve/i/ssa/ni/kin [= base nominative form + plural marker + inessive case ending + possessive affix + clitic affix] 
plant- s-in-my-also[= also in my plants] 
To a non-Finnish speaker it means that Finnish words often look very long and unanalysable. This impression is further enhanced by the Finnish tradition of spelling N+N+ (Nn) compounds as single words. However, Finnish is not as incomprehensible as it seems, not least because it is extremely systematic and regular. Indeed, the morphology and sound changes of the Finnish language are governed by clear rules which can be operationalized. Automatic tools for analysing Finnish morphological units are therefore highly accurate and reliable. 
2.2. Finnish in a nutshell 
(1) Articles 
While articles are one of the most frequent word classes in English, they are not used in Finnish. This is because determinedness can be expressed via case variation. It is worth noting that pronouns such as se ('it'), joku ('some'), eräs ('one') and the numeral yksi ('one') occasionally behave in an article-like manner. Unlike its English equivalent, the third person pronoun hän does not distinguish between genders (cf. ’he’ and ’she'). 
3 The partners of the project include language technology provider Kielikone, the Universities of Lancaster and Tampere, the publishing houses HarperCollins and Gummerus, and Nokia. For more information, see http://mot.kielikone.fi/benedict/. 
4 The first letter A corresponds to the Finnish O sound and does not appear in Finnish words, but it occurs in proper nouns of Swedish origin which are common in Finland. 
458
(2) Negating the verb 
When the predicate verb is negated, the negation ei ('not') takes on the conjugation form that indicates person. The verb form is identical with the indicative form, e.g. minä ajan -minä en aja ('I drive' - 'I don't drive'), me ajamme - me emme aja ('We drive' - 'we don't drive'). 
(3) Prepositions and postpositions 
As mentioned previously, Finnish uses case endings where English uses prepositions. Unlike English, Finnish also uses postpositions. Typically, they express relational aspects, i.e. jälkeen ('after') and päällä ('on'). 
(4) Case 
In theory, Finnish nouns, adjectives and numerals may appear in 15 different cases. However, not all of them are likely to be found in real-life use, even though they are grammatically valid. Table 1 shows examples of uses of different cases in Finnish and their approximate equivalents in English (Karlsson 1982). 
Table 2: Examples of cases of Finnish language 
Finnish singular numeral, adjective and noun Case Approximate English equivalents yksi punainen omena nominative 'one red apple' yhde/n punaise/n omena/n genitive 'one red apple's punainen omena/ 
yhd/en punais/en omena/n accusative (the object form which is identical with the nominative or the genitive form) (I can see) 'a red apple' 
(I ate) 'the red apple' yhte/nä punaise/na omena/na essive 'as one red apple' punais/ta omena/a partitive (object form) (I was eating) 'a red apple' punaiseksi omenaksi translative (It will turn) 'into a red apple' yhde/ssä punaise/ssa omena/ssa inessive 'in one red apple' yhde/stä punaise/sta omena/sta elative 'from one red apple' yhte/en punaise/en omena/an illative 'in/into one red apple' yhde/llä punaise/lla omena/lla adessive 'on one red apple' yhde/ltä punaise/lta omena/lta ablative 'from one red apple' yhde/lle punaise/lle omena/lle allative 'onto one red apple' 
(5) Compound words 
Compounds are stems consisting of more than one root, such as 'car-wash', 'black market', etc (Jackson and Amvela 2000: 79). Some English compounds have spaces between element roots. However, in this particular case, we use the term compound to refer to those formed by concatenating two or more words without a blank space between them. In Finnish, compounding is a very productive means of word-formation. Compounds are formed mostly of nouns, but words of other parts of speech can also appear in compounds. In theory, it is possible to combine any number of N+N+ (Nn) to form a compound. In reality, nevertheless, most compounds consist of an N+N combination in which the modifying N is either in the nominative form or in the genitive form. 
Lexicalised compounds may have meanings which differ from sum of the meanings of the element words. Such compounds are usually included in dictionaries with their own definitions. On the other hand, the compounds of ad hoc formation have meanings that can be deduced from that of the element words. In this sense it is very similar to the English the N of N structure with the difference that in ad hoc N+N compounds the semantic relation between the modifier and the head is not explicit but has to be deduced with extra-linguistic knowledge, e.g. saunakahvit (sauna coffees) - coffee and cakes and/or savouries offered after bathing in the sauna has ended (Karlsson 1982). 
459
Table 3: Examples of Finnish compounds 
Finnish compound Constituents English equivalents osa/päivä/työ osa 
päivä 
työ 'part' 
'day' 
'work' 'part-time job' sähkö/paimen sähkö 
paimen 'electricity' 
'shepherd' 'electric cattle fence' auringon/keltainen aurinko+ the genitive-forming morpheme 
keltainen 'sun' 
'yellow' 'yellow like the sun' 
(6) Flexible word order 
As in any agglutinative language, the word order in Finnish is relatively flexible, but not random (Vilkuna 2000: 32). This is because the information about part-of-speech and syntactic function of a word is usually embedded in its inflectional/derivational pattern in text context. It is therefore possible to change word order without changing the core meaning of the sentence. Nonetheless, changing word order inevitably affects the thematic structure of the clause, resulting into new emphases and nuances. 
In Finnish new information is usually given at the end of the sentence, but it can be topicalized by moving it to the beginning of the sentence. For instance, the words in the sentence: Nina rakastaa oliiveja (‘Nina loves olives') can be put in six different orders: 
Nina rakastaa oliiveja ('Nina loves olives'); 
Oliiveja Nina rakastaa ('Olives Nina loves'); 
Rakastaa oliiveja Nina ('loves olives Nina'); 
Nina oliiveja rakastaa ('Nina olives loves'); 
Rakastaa Nina oliiveja ('loves Nina olives'); 
Oliiveja rakastaa Nina ('olives loves Nina'). 
All of the above sentence variants are grammatically correct, but only the first one is in the unmarked, natural word order. 
As shown in our previous contrastive study, the Finnish language has a number of distinct grammatical features widely different from English. Such distinction presents a challenge to us in porting the EST framework into the Finnish language. In the following section, we report our current progress in this work and discuss related challenging issues. 
3. Building Finnish lexicons for the FST 
Note that the purpose of our work is to build a Finnish semantic tagger that makes use of the existing architecture of the EST. It would be ideal if we could simply switch the English linguistic database of the EST with Finnish counterparts. Unfortunately, things are not that straightforward. Indeed, our research to date suggests that changes to the EST framework are necessary if we are to deal with the distinct grammatical features of Finnish. The changes needed range from simple tasks such as dealing with three unique letters of Finnish (A, Ä and Ö) to complex tasks such as adjusting tagsets to reflect dictinct syntactic features of the Finnish language. In the following sections, we will discuss the problems that need be solved for developing the FST. 
3.1. Tackling the rich Finnish morphology 
In the semantic tagger, part-of-speech information provides a basis for determining the semantic category of a word. In the Lancaster EST, the C7 tagset of the CLAWS tagger is used for this purpose. In order to develop the FST, we need a Finnish counterpart POS tagset and tagger. The C7 tagset or other English POS tagsets are not appropriate for the Finnish language without modification. 
Moreover, considering the complex morphological structure of Finnish words, a specially designed morphological analyser is needed to identify stems and affixes embedded in Finnish words. Furthermore, in order to provide the syntactic tags, the Finnish texts need to be syntactically processed with a Finnish parser. For example, Ajoimmeko? (='Did we drive?') comprises a full sentence by itself. This word is derived from the base form ajaa ('to drive'). In order to analyse its morphological structure, we need several tags to annotate attributes. By way of illustration: 
the category of the word is verb, 
460
it is a finite form, 
it is in the active voice (as opposed to the passive voice), 
it is in the indicative mood (as opposed to the conditional, potential mood, etc.), 
it is in the past tense, 
it is in the 1st person plural form (we), 
it ends with the clitic -ko which implies a question. 
As previously highlighted, a single Finnish word form carries the same information as an English clause. It is worth noting that a single Finnish verb may have thousands of different forms, many of which are semantically quite different from each other. We therefore need some kind of a method to separate these words syntactically, i.e. a different 'tag' for all. Fortunately, software modules designed for this purpose are available. 
3.2. Morfo, TextMorfo and Dependency Parser 
A software package designed by the Finnish language technology company, Kielikone Ltd, perfectly serves the purpose of analysing POS and morphological structure of Finnish words. This package includes three tools capable of analysing Finnish text in various aspects (Juntunen 2002). 
The first tool, called Morfo, analyses the morphological structure of Finnish words. Given a word or compound, Morfo extracts all morpho-syntactic information from it and returns the candidate base forms with syntactic categories. It also splits the compounds into component elements. 
The second tool, called DCParser, is a full dependency parser of the Finnish language. Given a raw Finnish sentence, it returns a dependency tree representing the dependency structure of the input sentence. The parser is deterministic and linear in time behaviour (Arnola 1998). 
The last tool, called TextMorfo, works on the output of the DCParser to disambiguate word forms. That is, based on the candidate interpretations of the input word produced by the Morfo tool, it selects the correct interpretation in the given context. TextMorfo also extracts nominal and other constituent phrases. For example, given the word Ajoimmeko, the TextMorfo returns: 
Ajaa (Ajoimmeko), Verb, Imp Act Ind P 1P ko 
When necessary, this entry can be converted into an input format of the EST-core engines. 
The TextMorfo is an efficient tool that can potentially be used as a Finnish equivalent to the English POS tagger and lemmatiser. This kind of software makes it possible and feasible to build a Finnish semantic tagger. 
3.3. Tackling the compounds 
There are a large number of compounds in the Finnish language, as is the case for most languages. As mentioned earlier, lexicalised compounds carry meanings which are different from the sum of the meanings of their elements, and they function syntactically as single words. Although there is an argument for including such compounds in the lexicon, we have found that it may not be feasible to compile an extensive compound lexicon. The main obstacle to such a lexicon is the unlimited number of potential compounds. For example, hätäpuhelin ('emergency phone'), yleisöpuhelin ('public phone'), käsipuhelin ('portable phone') etc. are all types of phones and should automatically get the semantic tag of a phone. Such new words are being coined frequently; therefore it is doubtful that anyone can ever collect these compounds exhaustively. 
We suggest that a practical and reasonable solution to this problem is to break the Finnish word into element words. Each word will be assigned its own tag. At the final stage, multiple membership tags will be formed automatically by combining the tags of the compound parts. For example, N+N+ (Nn) compounds may receive two, three or even more tags, as the sample output shows below: 
suodatusaika_NNT1_T1/A1.1.1 ('filtering_A1.1.1 time_T1') 
kasvukorkeus_N3.7/N3.2+/A2.1 ('growing_N3.2/A2.1 height_N3.7') 
In case of a lexicalised compound which has its own entry in a dictionary, the definition provided by the dictionary will be used instead of going through the disintegrating/integrating procedure. 
Besides the types of compounds discussed so far, there is another type of compound whose element words cannot be separated. The reason is that when the element words are separated, they have different meanings from their combined form. A typical example of such a compound is the word jälki- which is often translated as the prefix 'post-' in English. However, when used on its own, the word jälki often means 'imprint'. 'track', 'trace' or 'mark'. In Finnish, this is a more general phenomenon than in English, though. 
Considering the complicated structure of Finnish words, we suggest that a specially designed algorithm is needed to ensure that the program is able to pick the correct semantic tag for each part of a compound. For example, a 
461
compound could be split into its elements, then each of the element could be given a special syntactic attribute like 'CompPart' before processing the lexicon. In this way, the rule 
jälki_CompPart XXXX 
would fire only when the word jälki occurred in a compound. In the future, a set of generic rules in the MWU lexicon will be needed for analysing parts of compounds. 
3.4. Tackling the flexible word order 
Like English, Finnish has plenty of fixed phrases and idioms, such as 
mennä kalaan ('to go fishing' or literally 'go into a fish') 
olla marjassa ('to be berry picking', literally 'be in a berry') 
However, the word order in Finnish is very flexible. For example, one could say: 
Marjaan mentyäni huomasin, että olin jättänyt korin kotiin. 
('After having gone berry-picking I realized that I had left the basket at home.') 
As in English, there might be several words between the actual idiom components. For example, 
Olisi kiva mennä joskus sinunkin kanssasi marjaan. 
('It would be nice to go some day with you berry-picking') 
In order to identify such non-consecutive components of single idioms, we need to design a special algorithm. A brute force solution would be to list every idiom in every possible formation, but it would require an excessive amount of time and human-labour. One of the ultimate solutions for this problem would be to include processing dependency trees instead of linear surface form text streams. In the dependency trees approach, the tree topology and attributes of the nodes specify the grammatical relationships between words. 
In addition to the main lexicon, we will also consider attributes-based rules such as: 
*_Base:ajaa_* *_Base:auto_Case:{ Nom/Gen/Acc/Part/Iness/Adess} M3 
where the items separated by '/' indicate possible cases. This rule would be able to tag all different ways to drive a car with semantic tag M3. However, it would exclude, for example, the act of driving someone out. 
3.5. Construction of Finnish lexicon 
The construction of a Finnish lexicon is probably the most laborious part of the porting project of the semantic tagger. To alleviate the problem, automatic methods/existing software will be used whenever possible. For instance, the Kielikone lexicons will be used for a rough translation of the existing EST lexicon. This technique has proved fruitful in the past where bilingual English-Polish dictionaries were used to assign semantic tags to Polish texts (Lewandowska-Tomaszczyk et al, forthcoming). Nevertheless, a time-consuming and meticulous manual post-editing phase is still needed. 
3.5.1. Test corpus 
Initial pilot tests include the construction of a parallel and comparable English and Finnish corpus. The Finnish corpus was compiled from texts found at http://www.kahvilasi.net/ - a Finnish web page for aficionados of good coffee. The collected Finnish texts were slightly amended (i.e. some grammatical and typographical errors were corrected), and then machine-translated into English and post-edited. Both the Finnish and English corpus were then compared and, where necessary, amended so that their content was as close as possible (in semantic terms). The Finnish corpus contains 2,063 words and the English corpus contains 3,473 words. 
3.5.2. Initial lexicon experiments 
In our initial experiment, we aimed to test the possibility of using a translated EST lexicon to tag Finnish text, as well as to construct a hand-tagged parallel corpus. The Finnish corpus was parsed with Kielikone's DC Parser to find all the base forms of the words. Altogether 1,489 base forms were extracted. They were arranged in alphabetical order, and English-Finnish translation lexicons were used to find the English equivalents for them. Finally, the resulting list of the English equivalents was tagged with the EST lexicon. 
462
As can be expected, this method produced some noise to the lexicon: some Finnish words received more semantic tags than they should have. Extra tags were manually removed while missing tags were added to obtain a clean test lexicon for the subfield of coffee. 
This lexicon was then applied back to the sample corpus. Afterwards, the corpus was manually checked to make sure that each word had only one correct semantic tag. This sample corpus will be used further for developing a statistical algorithm for the FST. 
3.5.3. Future development of the lexicon 
After having conducted initial experiments with different approaches, we conclude that existing lexicons and resources can help our work in porting the EST into the Finnish language. In terms of semantic categories, we found it is wisest to start with the most concrete, unambiguous semantic categories such as food, drink, animals, plants, anatomy and physiology, numbers and measurements etc. More abstract categories can be added later on. With abstract categories, automatic methods seem more a burden than help. 
For those words which have a one-to-one translation entry in the Finnish-English lexicon, a fully automatic approach can be used, i.e. for a given Finnish word its semantic tag can be found in the EST lexicon via its English translation equivalent. Also, automatic methods can be used in subdomains where there are semantically tagged lexicons available, such as scientific names of animals and plants and other things. Finally, in order to guarantee a wide coverage of the lexicon, the translated and post-edited lexicon is compared against a Finnish-English dictionary or Finnish monolingual dictionary to find missing Finnish words. 
Building idiom templates seems to be the most challenging part of our work. Little help from automatic tool can be expected. We have found it is unfeasible to translate all of the rules of the existing EST idiom list due to the regular expressions used within it. Possible help may come from the lexicons of the Kielikone's existing Finnish-English machine translation system TranSmart. This system has a quite extensive set of context-specific rules for translating Finnish to English and, to some extent, it does some semantic analysis, too. Unfortunately, the rules of TranSmart are in a format that is not compatible with the EST. However, efforts are being made to make use of this tool wherever possible. 
4. Conclusion 
In this paper, we described our research work on building a Finnish lexicon in the efforts of building a Finnish semantic tagger via porting the English semantic tagger architecture to the Finnish language. We examined distinct grammatical features of Finnish in comparison to English, discussed a list of challenging issues that have arisen in the course of our work, and proposed solutions to cope with these problems. 
Our experiments have shown that different grammatical features between different languages may present tough challenges in building a system that is compatible with multiple languages. In our particular case, the distinct agglutinative and synthetic nature of Finnish and the analytic nature of English require different algorithms and tools for annotating the same type of linguistic information. In particular, the complex and flexible morphological structure of Finnish words entails a drastically different approach to extracting POS information. Our experiments to date have also shown that the semantic categories developed for the EST are mostly compatible with the semantic categorizations of objects and phenomena in Finnish and can thus be applied cross-linguistically in prototypical cases. Certainly, some minor deviations, such as the lack of gender markers in Finnish, will cause discrepancy between related categories of these two languages. 
The unresolved issues mentioned above are both interesting and challenging to us. As our work progresses, we will seek solutions to these issues, which will be beneficiary to the corpus research community in general. 
Acknowledgements 
The work described in this paper was carried out within the Benedict Project funded by the European Community under the 'Information Society Technologies' Programme (project reference IST-2001-34237). The partners of the project are language technology provider Kielikone, the Universities of Lancaster and Tampere, the publishing houses HarperCollins and Gummerus, and Nokia. For more information, see the web site http://mot.kielikone.fi/benedict/. 
References 
Arnola, Harri 1998 On Parsing Binary Dependency Structures Deterministically in Linear Time. COLING 98. Montreal 
Hakulinen, Auli; Karlsson, Fred 1979 Nyky-suomen lauseoppia ('Syntax of Contemporary Finnish'). Suomalaisen kirjallisuuden seura. 
Jackson, Howard and Etienne Zé Amvela 2000 Words, Meaning and Vocabulary - An introduction to modern English lexicology, London: Casell. 
Juntunen, Jukka-Pekka, Kielikone Ltd 2002 Features and API of Kielikone Language Processes for Automatic Processing of Finnish 
Karlsson, Fred 1982. Suomen peruskielioppi ('Finnish Grammar'). Suomalaisen kirjallisuuden seura. 
463
Lewandowska-Tomaszczyk B, Oakes M and Rayson P (forthcoming). Annotated Corpora for Assistance with English-Polish Translation. In Wilson A, Rayson P and McEnery T (eds.) Corpora by the Lune: a festschrift for Geoffrey Leech. Peter Lang, Frankfurt. 
McArthur Tom 1981 Longman Lexicon of Contemporary English. Longman London. 
Rayson Paul and Wilson Andrew 1996 The ACAMRIT semantic tagging system: progress report. In Evett L J and Rose T G (eds.) Language Engineering for Document Analysis and Recognition, LEDAR, AISB96 Workshop proceedings, pp. 13-20. Brighton, England. Faculty of Engineering and Computing, Nottingham Trent University, UK. 
Vilkuna, Maria 2000. Suomen lauseopin perusteet ('The basics of Finnish Syntax'). Kotimaisten kielten tutkimuskeskus. Edita. 
Wilson Andrew and Rayson Paul 1993 Automatic Content Analysis of Spoken Discourse. In: Souter C and Atwell E. (eds.), Corpus Based Computational Linguistics. Amsterdam: Rodopi. pp. 215-226 
464

Signalling well-written academic articles 
in an English corpus by text mining techniques 
Nadine Lucas*— Bruno Crémilleux*— Leny Turmel ** 
GREYC, CNRS UMR 6072,Caen University 
Campus Côte de Nacre 
F- 14032 Caen Cedex France 
Tel 33 (0)2 31 56 73 36 fax 33 (0)2 31 56 73 30 
*{bruno, nadine}@info.unicaen.fr 
** lturmel@etu.info.unicaen.fr 
KEYWORDS: text mining, textual linguistics, corpus characterisation, rules characterising classes, academic style, detection of English mistakes 
1. Introduction 
In the last decades, most of the information supports have been transformed into electronic form. The value of these data has been fully recognised only recently: their potential use may greatly exceed the uses for which the data were actually collected and stored. Data mining and text mining address the interactive and iterative knowledge discovery processes from large databases and collections of texts. This research domain has been quite active in the last decade due to the pressure of corporate data owners who need added value from the huge volume of data they easily collect. 
We meet such a challenge in classifying academic articles in English prior to copy editing. We have to classify a large corpus to discriminate well-written versus poorly-written academic articles in an English corpus. The aim is to inform computer-aided copy editing by focussing attention on poorly-written articles. We also wish to skip correct (i. e. with no mistake) segments and highlight incorrect segments in these articles in order to save time. The training corpus includes forty academic articles written by authors who use English as first or second language with varying degrees of competence. We use pairs of articles allowing comparison between two successive versions, before and after human copy editing. 
The task is challenging, because correctness in style is difficult to define. This corpus of great interest requires an original text-mining method discovering rules from the linguistic corpus on correct and incorrect segments. The main feature presented here is the use of several different text segments, or textual measures, where stylistic mistakes or improper use can be detected. Each text is thus divided into parts, into paragraphs and into sentences, in order to circumscribe textual semantic units. 
We claim that results stemmed from cross-fertilisation between novel text-mining and textual linguistics approaches. The level-specification of forms and the top-down positional inheritance (see Section 3) are essential to extract associations concluding on correct or incorrect segments. It would be hopeless to rely on words alone. Furthermore, owing to the large number of segments, efficient data mining tools are required. We use MVMiner, a prototype that is able to extract all associations in a data set above a frequency threshold. Associations are given in a special condensed representation (see Section 4) allowing relevant rules characterising classes to be exhibited. 
The paper is organised as follows. First we present the problem and second introduce the linguistic approach, based on position and form. The main notions of level-specification and positional inheritance are explained. Third, the text mining method is presented as well as the techniques to extract associations and rules characterising classes. Broadly speaking, the text mining method captures reliable similarities and rules embedded in the articles. Then we examine results. Experiments show that extracted rules highlight features that do characterise classes. Last, we discuss the usefulness of this method. 
2. Problem description 
First, let us describe the task. The ultimate objective is to ensure "readability" or "better understanding" of academic articles. Corrections have to do with style mainly and sometimes with grammar, they have nothing to do with word-spelling. Thus words cannot be used as descriptors. Two classes are 
465
established, well-written and poorly-written articles. Descriptors characterising one of the classes (well-written articles) are established: they are designed for characterising correctness in academic articles in English. The aim is to extract association rules from the pairs of texts, and then to apply them to discriminate well-written from poorly-written texts in a new un-revised collection of academic articles. New texts can then be labelled automatically. 
2.1. Segments 
Current approaches in text mining equate descriptors with words. But since most corrective changes concern word position rather than words themselves, checking words would be hopeless. Comparison of the pairs of text would return no significant results. To overcome this difficulty, we established a hierarchy of textual measures that correspond to selection as described by Hockett (1958) or Harris (1952) and involve the linguistic notion of span for textual markers. We emphasise the relative positions of textual markers inside these observation windows. Thus, instead of handling simple units (text and words), we now consider correct and incorrect segments. 
2.2 Main features 
Our approach may be characterised by two concepts, called top-down positional inheritance and level-specification for forms. Level-specification for forms is devised to represent span. Top-down inheritance amounts to keep information obtained at the higher levels of hierarchy down to the smallest measures of text. This amounts to memorising information on the tree-like hierarchy provided by the layout, since any word can be related to its including comma-punctuated unit. A comma-unit found, say, in an introduction, is related to the sentence where it is located, to the paragraph including it and to the part including this paragraph (see Figure1). 
3. Linguistic contextual approach 
Descriptors are textual markers that come from linguistic knowledge on discourse structure. They fall in two main categories, forms and positions. Forms may be associated with words as a first approach (Péry-Woodley 1993). Words are here considered as small segments in the chain of characters actually handled by computers. Words are not informative by themselves; they belong to a context. 
3. 1 Textual measures 
Our postulate is that the context can be approached through textual measures, as expressed by the material layout of the document. Texts are automatically divided into text-zones, and the text-body is subdivided in parts, sections, paragraphs, sentences and comma-punctuated units. When checking modifications in the original and revised versions of articles, these units are compared at each level of the tree-like hierarchy provided by the layout. For the two examples produced below, the clause span will be associated with a text chain delimited by comma, while text segments characterised by contrasted use of voice will be best approximated by sections or parts. These text measures are observation windows. They play the same role as selection, the difference being that we consider all available segments in the text hierarchy. 
Two examples can be used to illustrate this point. One common mistake is the misplacement of adverbs; this is checked within the span of the clause and inside the verb group. It concerns word position in a narrow context, the verb phrase, itself belonging to a wider context, which is a period or comma-delimited unit roughly encompassing a clause. 
Example 1 a (original text) 
They concur to identify an area, in a certain way homogeneous, where, besides the beauty of places, the Cultural Heritage represents an extraordinary richness worldwide recognized, which animates a strong need of safeguard and valorisation. 
Example 1 b (revised text) 
They concur to identify an area, in a certain way homogeneous, where, besides the beauty of places, the cultural heritage represents an extraordinary richness recognized worldwide, which animates a strong need of safeguard and valorisation. 
In this example, note the last position in the fifth comma-punctuated unit is filled with the form recognized in the original text and with the form worldwide in the revised text. 
Words are obviously not the only type of segments useful for the current study. As will be explained below, most stylistic mistakes involve lack of coherence in paragraphs and parts. Example 2 shows an incorrect use of infinitive mode. It appears in the last but one sentence, in a context where 
466
indicative mode and passive voice are required. The narrow context is the paragraph, and the larger one is the part (given only for a) below). Letting aside the confusion between infinitive and imperative, we would say that parts may be marked by mode, but not at the particular position filled by this paragraph. Here, sentences have been shrunk and each is on a line. 
Example 2 a (original text) 
§1 Cancer risk in mutation carriers has been evaluated by a …by 60 years of age. 
The age-specific penetrance was higher in women than in men …of breast cancer [14]. 
In another hospital-based series…, the risk was estimated to be …at the age of 45. 
Lifetime risk was estimated to be 73% in males…, …by breast cancer [30]. 
To keep in mind that individuals with LFS …, …in childhood [80]. 
It could be firmly stated that penetrance is high, but incomplete particularly in males. 
§2 However, cancer incidence …did not seem… of ADCC in childhood [34; 81]. Paradoxically, …with a wide spectrum of mutations [27; 29; 35; 56; 63; 66; 82]. 
§3 Mutations outside the core …could be associated…in the core DNA-binding domain [73]. 
Some cases … have been described…, they could be not so rare [30; 83; 84]. 
This is not unexpected …of mutation carriers. 
§4 One case of mosaicism was reported [85], it must be kept in mind for genetic counseling. 
Example 2 b (revised text) 
Cancer risk in mutation carriers has been evaluated by a …by 60 years of age. 
The age-specific penetrance was higher in women than in men …of breast cancer [14]. 
In another hospital-based series…, the risk was estimated to be …at the age of 45. 
Lifetime risk was estimated to be 73% in males…, …by breast cancer [30]. 
It should be kept in mind that individuals with LFS …, …in childhood [80]. 
It could be firmly stated that penetrance is high, but incomplete particularly in males. 
Representing span is not obvious. The next sections detail the main concepts of text observation windows or textual measures and their descriptors. 
3.2 Descriptors 
3.2.1 Forms 
Knowledge on the academic genre as such was acquired prior to this experiment (Lucas 1991) and special attention was given to academic English focussing on English as second language (Grabe 1987, Péry-Woodley 1989). Specific words or features are detected, as having functional value (Péry-Woodley 1993). Such forms are discourse connectors in a very broad meaning. Our postulate was to use different descriptors for each measure of text, according to its level, in an effort to represent span. Special effort was made to select textual markers for higher levels. This is called level-specification of descriptors. Note that a full syntactic analysis could not be run prior to our experiment, because tags would interfere with text-mining. Moreover, classic syntactic parsing does not provide the kind of information we needed. Hence, only marks that could be found by a very shallow parsing through regular expressions were selected. 
Characteristic word endings such as -ly or words of few letters are used as descriptors for comma-punctuated units and sentences, pairs of words for paragraphs, groups of three words are used to describe sections or parts. Whenever possible, words or contiguous words are used, for instance co-ordination marks and, as well, and as well as are descriptors respectively for sentence, section, and part level. Features such as passive voice were checked at the pr-processing stage through very rough criteria including be auxiliary and -ed. Such features type paragraphs, sections or parts when they are present in all sentences in the considered measure. However, gaps inside the measure are allowed, that is to say that beginning and end are considered remarkable positions as explained below. Special punctuation marks are also used as descriptors. All these textual markers were selected on the basis of a careful examination of the corpus. 
In the same way, different forms belonging to the same grammatical or rhetorical category were grouped into one class as descriptor for higher levels, for instance the, this, these, that, those, which stand for themselves at sentence level, were merged as anaphoric determiners at paragraph level. Some examples are given and classes are shown between {} in Table 1. Classes were fused or rearranged as a single class as descriptor for part level, for instance anaphoric determiners and anaphoric adverbs were merged in an anaphoric general class. These level-specific descriptors are also matched with first and last positions clues. 
467
3.2.2 Positions 
Specific position in a given measure is a descriptor, independent from forms. Special positions refer to beginning or end of any measure of text, e.g. beginning of a sentence or end of a paragraph. The beginning of a comma-punctuated unit is the first word, the beginning of a sentence is the first comma-punctuated unit, if any, otherwise the first word. Similarly the beginning of a paragraph is the first sentence, the beginning of a section is the first paragraph, and the beginning of a part is the first section, if any, otherwise, the first paragraph, while the beginning of the text-body is the first part. The same principle applies for end. 
Table 1. Examples of descriptors 
Measures Descriptors morphological features Comma-units Word endings (-ed, -ing, -ly) 
Connectors 1 (Despite, Indeed, Because) 
Co-ordination 1 (and) 
Adjective anaphoric (Its, Their, Such…) 
Typography ()[]"" sentences Same as above paragraphs Connectors 2 (There is, In fact…) 
Co-ordination 2 As well, in addition 
Adverb {-ly words, + despite, indeed…} 
Preposition 
Pronoun 
Determiner definite {this, these, those} 
Determiner indefinite {one, some…} 
Subordination Conjunctions … sections Ordinal 
Cardinal 
Voice 
Aspect 
Major Conjunctions {If, although, when, because, …} parts Connectors 3 (in spite of, for this reason…) 
Co-ordination 3 as well as 
Tense Future, Past 
Aspect Perfect 
Voice Passive 
Adverbial Connectors {however, nonetheless…} 
Anaphoric words {this, these…thus} 
Wh conjunction {which, what, when, why…} 
Other conjunctions {if, although, because…} 
Thus, we carefully describe where textual marks may occur, instead of using just words as in classical text mining approaches. Examples of remarkable co-occurrence of descriptors, as related to specific position and specific forms, is for instance the high number of so-called anaphoric pronouns at the beginning of sentences themselves located at the end of paragraphs. 
A special position descriptor describes repetition of the same textual mark at the beginning and end of a text measure. It is called "isoperiphery" and signals a repetition e.g. at the beginning and end of a paragraph there is a sentence bearing the same feature, for example, a "definition" sentence containing the word is. We also call this parallelism of forms. 
468
Figure 1. Overview of the system 

Part Section Paragraph Sentence Comma-delimited unit CLAUSE related MARKS SENTENCE related MARKS PARAGRAPH related MARKS SECTION related MARKS PART related MARKS MISTAKE And or but s Verb negation a an the this ( ) [ ] Impersonal definition(Itis)Adverb Adverb-type Conjunction Preposition Pronoun Aspect Conjunction Isoperiphery Adverb-type Connector Future tense Past tense 0 1 1 1 0 1 1 1 0 1 11 0 0 1 1 11 0 1 0101 1 1 0 1 1 1 1 1 0 10 1 0 0 1 01 1 1 0110 0 0 1 0 1 1 1 0 0 11 0 0 1 1 11 0 1 0111 0 0 0 0 1 1 1 0 0 11 0 0 1 1 11 0 1 0100 0 0 0 0 1 1 1 0 0 11 0 0 1 1 11 0 1 0101 0 1 0 1 1 1 1 1 0 01 0 0 0 1 11 0 1 0101 1 0 1 0 1 1 1 0 0 11 0 0 1 1 11 0 1 0101 0 0 1 0 1 1 1 0 0 11 0 0 1 1 11 0 1 0111 0 0 1 0 1 1 1 0 0 11 0 0 1 1 11 0 1 011… … … … … … … … … … …… … … … … …… … … ……… 

Correct 
class 

Incorrect 
class 

characterisation rules 

Extraction of rules for class characterisation 

textual marks 

segmentation 

revised article 

Corpus 

Un-revised article 
469
4. Associations characterising English correctness 
Broadly speaking, the text mining method captures reliable similarities and rules embedded in the articles. In our experiments, articles are translated into patterns based on the textual markers. We get between 728 and 22,041 patterns according to the window covered by the textual markers. Rules are built on these patterns and are evaluated by the classical measures of frequency and confidence. As the task is the discovery of all rules, the frequency and confidence of which are user-specified thresholds, there are a huge number of candidate rules and efficient algorithms are required. We use recent results on condensed representations based on ä-free patterns to mine ä-strong rules characterising classes. 
4.1 Data mining and text mining 
There are numerous text and data mining techniques that have often been developed on core techniques from statistics or machine learning. Data mining and text mining share common methods and algorithms (e.g., the search of frequent associations of patterns with a level-wise approach) (Mannila and Toivonen, 1997). Text mining has been introduced in order to offer tools for utilising text resources in data mining driven decision support. Text mining has some specificity: documents can be more or less structured, the sequence of words has to be taken into account, the pre-processing stage has a great role and it is recognised that linguistic and natural language resources are required. Many text mining methods include text operations such as pre-lexical analysis (e.g., treatment of digits, hyphens, punctuation marks), lexical analysis such as elimination of stop-words, stemming and morphology, selection of index terms, syntactic analysis (e.g., determination of noun phrases), semantic constructions including linguistic aspects as well as domain-specific components. Sequential patterns have been used for discovering trends among patents (Lent, Agrawal. and Srikant, 1997) and efficient algorithms to find co-occurring of text phrases have been developed (Ahonen-Myka et al. 1999). One popular data mining technique concerns knowledge discovery from frequent association rules. This kind of process has been thoroughly studied since the definition of the mining task in (Agrawal, Imielinski, Swami, 1993). Association rules can tell something like "It is frequent that when descriptors A1 and A2 are true within an example, then descriptor A3 tends to be true'' where A1, A2 and A3 are for instance descriptors of texts (Feldman, abd Dagan and Klösgen, 1996). In the following sections, we define association rules more precisely and ä-strong rules characterising classes which are the simplest rules characterising classes with respect to their left-hand sides. 
4.2. Association Rules 
Let us provide a simple formalisation of association rules mining task. In the following, definitions are relating to a set of examples (e.g. paragraphs, sentences, comma-punctuated units). 
[item, itemset] 
Let D = {D1,…, Dn} a set of descriptors. An element of D is called item and a subset of D itemset. 
For instance, an item corresponds to the presence of mark -ly at the end of a comma-punctuated unit. From a technical point of view, algorithms require binary items to be efficient but a set of initial non-binary descriptors can always be translated into a set of items. Hence, in our experiments, the whole set of descriptors (see Table 1 for a subset) corresponds to 270 items. 
[association rule] 
An association rule is an expression X => B, where X belongs to D (i.e. X is an itemset) and B ¸ D\ X. 
For instance, parenthesis the an that => incorrect is an association rule meaning that when the items parenthesis the an that are true (present), then the segment is incorrect. X is also called the left-hand side of the rule. The fact that B must not belong to X is only to avoid producing trivial rules. Note that it is easy to generalise this definition to allow rules with several items in their right-hand side (i.e. conclusion) (Agrawal, Imielinski, Swami, 93). The classical measures of frequency and confidence capture the semantics of the "representativity" and the "strength" of the rule (Agrawal, Imielinski, Swami, 93). 
[frequency, confidence] 
Given X belonging to D, F (X) (or frequency of X) is the number of examples for which each item of X is true. The frequency of X=>B is defined as F (X ¾ B) and its confidence is F (X ¾ {B})/ F (X). Frequency is also called support. 
The standard association rule mining task concerns the discovery of all rules the frequency and confidence of which are greater than a user-specified threshold. In other words, one wants rules that are 
470
frequent "enough" and valid. The main algorithmic issue concerns the computation of every frequent itemset. 
[frequent itemset] 
Let ã be a frequency threshold lower or equal to the number of examples. An itemset X is said frequent if F (X) < ã. 
The complexity of frequent itemset mining is exponential with the number of descriptors. Many research works (e.g. Pasquier et al. 1999, Boulicaut Bykowski Rigotti 2000) concern the contexts for which such a discovery remains tractable, even though a trade-off is needed with the exact knowledge of the frequencies and/or the completeness of the extractions. 
From a technical point of view, the search of association rules can be divided in two parts: first, the extraction of all frequent itemsets and then the generation of rules. Itemsets correspond to associations between descriptors. In this work, we use the discovery of frequent itemsets to validate hypothesis set out in Section 3. Furthermore, we use a special kind of association rules (ä-strong rules characterising classes) that we have developed (Crémilleux and Boulicaut 2002) to infer rules about the presence or not of English mistakes in segments. 
4.3 ä-strong rules characterising classes 
Let us consider a classification task with k class values. Assuming C1, … ,Ck are the k items that denote class values. Here, we have k = 2 (two classes, correct and incorrect). A ä-strong rule characterising classes concludes on a class value with a rather high confidence. 
[ä-strong rule characterising classes] 
A ä-strong rule characterising classes is an association rule with a minimal left-hand side that allows at most Ý exceptions and that concludes on one class value (i.e., Ci). 
In other words, the confidence of a ä-strong rule characterising classes is at least equal to 1-(ä/ .). It is out the scope of this paper to explain precisely how such rules are built (details are in Crémilleux and Boulicaut 2002). Let us simply say that we use recent results on condensed representations based on ä-free patterns (Boulicaut Bykowski Rigotti 2000, 2003). ä-strong rules are built from ä-free patterns that constitute their left-hand sides (Boulicaut, Bykowski and Rigotti, 2000). The property of freeness checked by ä-free patterns enables to give a safe pruning criterion in the search of frequent itemsets. It means that we are able to design effective algorithm even in the case of huge, dense and/or highly correlated learning data sets where usual approaches do not fit (Crémilleux and Boulicaut 2002). Furthermore, a ä-free pattern is a minimal conjunction of items to know the frequencies of a set of items. It means that we are able to extract the simplest rules (i.e. with the minimal left-hand sides) to conclude on an item, the uncertainty being controlled by ä. We argue that this property of minimal left-hand side is a fundamental issue for class characterisation. Not only it prevents from over-fitting (i.e. over-specified rules acquired on a learning set and leading to miss-classified new examples) but also it makes the characterisation of an example easier to explain. It provides a feedback on the application domain expertise that can be reused for further analysis. Moreover, if the extraction of patterns is done under a sensible sufficient condition, important classification conflicts (identical body conflicts, included bodies conflicts) are avoided, which is useful to classify new examples (Crémilleux and Boulicaut 2002). 
Experiments are performed with our prototype MVMiner, which has been implemented by F. Rioult at GREYC laboratory. Given ä and ã, MVMiner extracts all ä-free patterns and rules characterising classes. The following section shows examples of such rules. 
5. Results 
Experiments show that extracted rules highlight features that do characterise classes (correct and incorrect segments). These features enable to classify new segments. 
First let us note that the use of a top-down inherited context allows extracting rules even from a relatively small number of articles because it enables to define a sufficiently large number of patterns. This could not be obtained from data ignoring context, as is shown in Table 2. The last line shows a comparison with an experiment run without top-down inheritance. It is clear that context is essential to extract frequent itemsets. 
Second, in order to test the appropriateness of the rules to qualify well-written versus poorly-written articles, we ran an experiment in which articles were tested irrespective of their primitive labelling (before and after copy-editing), on the whole corpus. Results showed that we succeeded in classifying segments in articles belonging to the revised corpus as correct. 
471
Table 2. Frequent itemsets extracted with and without context 
Comma-units SentencesParagraphs Parts Nb. of measures 22041106433520728 Frequency 60805050 ä 101000 Number of frequents itemsets with top-down inherited context 4850781595329 Nb. of frequents itemsets without top-down inherited context 58140 
ä-strong rules characterising classes provide a list of correlated textual markers, which are considered as being representative of a well-written text. This validates the established model for automatic editing and helps to define sound contexts for triggering correction rules. This is especially true to capture large contexts through large span patterns. It should be noted that disappointedly few rules were extracted from high level segments, none for parts and fewer than expected for paragraphs. 
Positional inheritance balanced these results. It is particularly useful, because it is linked with high order coherence of text. Rules extracted on comma-units but exhibiting a large span of inherited features do characterise well-written articles, as shown in Table 3. The notation is somewhat difficult to read and needs to be translated, for example the first line reads "if a comma-unit belongs to a (complex) sentence including where and belongs to a part including a complex co-ordination syntagm as well as, then it probably is a correct segment". Here we can see that level-specification for marks also plays a role in characterising correctness. Example 3 illustrates such a case (see last sentences of the paragraphs for the cited marks). 
Example 3a (original) 
3.2 The MutL-related complexes 
The existence of four MutL homologs, MLH1-3 and PMS1 have been reported in yeast (Table 2) [72, 73]. In human, in addition to the four MutL homologs, namely hMLH1, hMLH3, hPMS1 and hPMS2, a cluster of hPMS2-like genes have been localized on chromosome 7 [74, 75]. […] Because MLH1 is a common subunit to all three complexes, its deficiency leads to a severe phenotype, similar to that of MSH2-deficient cells. The MSI phenotype resulting from MLH1-deficiency is characterized by a tremendous increase in base-base mismatches, as well as frameshift mutations resulting from unrepaired IDL. 
3.3 The discrimination of the newly-synthesized strand and the processing steps of MMR 
Once bound to mismatches, MutLá complex is able to interact with numerous factors, consistent with the assembly of a higher-order complex that is involved in the excision of a large fragment of the newly-synthesized DNA strand containing the mismatch (rev. in [8, 11, 85]). In organisms other than E. coli, the signal that allows to discriminate the newly-synthesized strand from the template is still doubtful, but does not involve DNA methylation [86]. The proliferating cell nuclear antigen (PCNA) which is essential for DNA replication, where it acts as a processivity factor has also been implicated in MMR before and during the DNA synthesis step and could be involved in this process. […] 
Example 3b (revised) 
3.2. The MutL-related complexes 
The existence of four MutL homologs, MLH1–MLH3 and PMS1 has been reported in yeast (Table 2) [72,73]. In human, in addition to the four MutL homologs, namely hMLH1, hMLH3, hPMS1 and hPMS2, a cluster of hPMS2-like genes have been localized on chromosome 7 [74,75]. […] Because MLH1 is a common subunit to all three complexes, its deficiency leads to a severe phenotype, similar to that of MSH2-deficient cells. The MSI phenotype resulting from MLH1 deficiency is characterized by a tremendous increase in base–base mismatches, as well as frameshift mutations resulting from unrepaired IDL. 
3.3. The discrimination of the newly synthesized strand and the processing steps of MMR 
Once bound to mismatches, the MutLá complex is able to interact with numerous factors, consistent with the assembly of a higher-order complex that is involved in the excision of a large fragment of the newly synthesized DNA strand containing the mismatch (reviewed in [8,11,85]). In organisms other than E. coli, the signal that allows us to discriminate the newly synthesized strand from the template is still doubtful, but does not involve DNA methylation [86]. The proliferating cell nuclear antigen (PCNA) which is essential for DNA replication, 
472
where it acts as a processivity factor has also been implicated in MMR before and during the DNA synthesis step and could be involved in this process. […] 
There are a fair number of rules involving repetition or parallelism, at the paragraph level: the label "isoperiph" in Table 3 signals a paragraph beginning and ending with sentences bearing the same textual marker. This stylistic feature appears with a fairly high confidence. Line two reads "if a comma-unit belongs to a sentence showing the presence of by, and belongs to a paragraph that can be characterised by parallelism, namely "definition" sentences containing the mark is at the beginning and end, and also belongs to a part well-characterised by special features, i.e. by the passive voice and the presence of impersonal pronouns, then there is a good chance that it is a correct segment". All these marks indeed describe coherent "passive" parts, and these parts occur in well-written articles. In the same way, line 3 reads "if a comma-unit belongs to a sentence marked by the special punctuation semicolon, and belongs to a paragraph that can be typed by parallelism of "argumentative" sentences marked by a conjunction, and also belongs to a part that is characterised by the presence of a strong adverbial connector, then its probably is a correct segment". All these features concur to characterise a well-marked "argumentative" part and such parts characterise well-written texts. 
It is particularly interesting to note that comma-units are not characterised by descriptors sui generi in well-written texts, but because they belong to a well-marked context. 
In Table 3, the textual measures where features appear and from which the comma-unit inherits are noted P for Part, § for paragraph, S for sentence, and CU for comma-punctuated unit. 
Table 3. Examples of class characterisation rules concerning comma-delimited units (22,041 comma-units in the data) 
Characterisation rules Class Frequency Confidence P As well as S where CORRECT 229 1 P PASSIVE IMPERSO § ISOPERIPH S by CORRECT 837 0,96 P ADV § CONJ § ISOPERIPH S semicolon CORRECT 295 0,89 P FUTURE § ISOPERIPH S THERE CORRECT 499 0,93 S PARENTHESIS CU an CU the CU that INCORRECT 47 0,37 S PARENTHESIS CU an CU the CU with INCORRECT 36 0,37 S PARENTHESIS CU the CU for CU and INCORRECT 27 0,36 S PARENTHESIS CU the CU to CU if INCORRECT 31 0,30 S PARENTHESIS CU the CU to CU [] INCORRECT 40 0,30 S PARENTHESIS CU the CU with CU and INCORRECT 22 0,26 S PARENTHESIS CU the CU with CU or INCORRECT 22 0,23 S PARENTHESIS CU the CU it CU [] INCORRECT 57 0,26 S PARENTHESIS CU the CU or CU [] INCORRECT 33 0,26 
By contrast, the produced rules exhibit correlation of textual markers used in poorly written papers. The main result is that in poorly-written texts, there is little or no correlation between parts and paragraphs and sentences. Lack of coherence is the contrary of clearly contrasted text measures that characterised well-written articles. Rules show correlation only in a very narrow window, sentences and comma-punctuated units. Mined rules are interesting although no strong tendency emerges to type families of correlated mistakes. Our sample contains articles from various geographic origins, and articles written with varying degrees of competence. The confidence threshold is set at a low level in order to catch some regularities, for instance the fact that in a sentence with parenthesis, the co-occurrence of the and an signals awkwardness. Such information obtained from patterns is useful because the potential mistakes or mismatches in academic English are extremely numerous and we cannot even figure them out. Instead of asking a professional corrector to list frequent mistakes by hand, we retrieve information both on mistakes and on their context from the corpus itself. 
Interestingly, a fair number of characterisation rules highlight the role of special punctuation marks, such as parenthesis. This is linked with the conventional use of bibliographical references and other deictic devices, which play a very important role in structuring academic articles (Lucas 2003). Misplacement of references is a common mistake, and human correctors frequently check them. 
This work is an example of cross-fertilisation between textual linguistics and text mining techniques and call for further exploration on a much larger sample of texts. Future work is planned on high-level descriptors involving more elaborate pre-processing for a better liability. A larger number of forms should also be used to catch regularities in poorly-written articles. 
473
At this stage, results prove the soundness of our linguistic description of well-written academic articles. Using segments also enables to produce a finer description of our corpus. Incorrect segments can be highlighted in any text, which proves to be a valuable practical help for computer-assisted copy-editing. 
Bibliographical references 
Ahonen-Myka H, Heinonen O, Klemettinen M, Verkamo A I 1999 Finding co-occurring text phrases by combining sequence and frequent set discovery. In Proceedings of the workshop on text mining: foundations, techniques and applications, pp. 1-9. 
Agrawal R, Imielinski T, Swami A 1993 Mining Association Rules between Sets of Items in Large Database. In Proceedings of ACM SIGMOD 93, pp. 207-216. 
Boulicaut JF, Bykowski A, Rigotti C 2000 Approximation of frequency queries by means of free-sets. In Proceedings of the Fourth European Conference on Principles and Practice of Knowledge Discovery in Databases, PKDD'00, Lyon, Springer, pp. 75-85. 
Boulicaut JF, Bykowski A, Rigotti C 2003 Free-sets: a condensed representation of boolean data for the approximation of frequency queries. Data Mining and Knowledge Discovery journal, 7(1):5-22. 
Crémilleux B, Boulicaut JF 2002 Simplest Rules Characterizing Classes Generated by Ý-Free Sets, In Proceedings of 22nd Annual International Conference of the British Computer Society's Specialist Group on Artificial Intelligence (ES 02), Cambridge, Springer, pp. 33-46. 
Feldman R, abd Dagan I, Klösgen W 1996 Efficient algorithms for mining and manipulating associations in texts, In Thirteenth European Meeting on Cybernetics and Systems Research, Cybernetics and Systems, Vienna, Volume II. 
Grabe W 1987 Contrastive Rhetoric and Text-type Research. In Connor, Kaplan (Eds) Writing across languages: Analysis of L2 Text. Reading, MA, Addison-Wesley, pp. 115-137. 
Harris Z 1952 Discourse analysis. Language (28): 1-30. 
Hockett C 1958. A Course in Modern Linguistics. New York. 
Lent B, Agrawal R, Srikant R 1997 Discovering trends in text databases. In Proceedings of Third International Conference on Knowledge Discovery and Data Mining KDD 97, Newport Beach, California, AAAI Press, pp. 227-230. 
Lucas N 1991 Syntaxe discursive du japonais scientifique: a propos du theme. In Centro de estudos japoneses da Universidade de Sao Paulo (Ed.), II encontro nacional de professores universitários de língua, literatura e cultura japonesa.Sao Paulo, Sao Paulo University, pp. 77-98. 
Lucas N 2003 La citation et l'appel a référence bibliographique dans les articles académiques. In Marnette, Rosier, López Munoz (eds) Le discours rapporté dans tous ses états Question de frontieres. Bruxelles, Duculot (in press). 
Mannila H, Toivonen H, 1997 Levelwise search and borders of theories in knowledge discovery. Data Mining and Knowledge Discovery, 1(3), pp. 241-258, 
Pasquier N, Bastide Y, Taouil R, Lakhal L 1999 Efficient Mining of Association Rules Using Closed Itemset Lattices. Information Systems, 24(1):25-46. 
Péry-Woodley MP 1989 Textual designs: signalling coherence in first and second language academic writing. Lancaster University thesis, Notes et documents LIMSI 91-1. 
Péry-Woodley MP 1993 Une pragmatique a fleur de texte: marques superficielles des opérations de mise en texte. In Moirand. et al. (eds), Parcours linguistiques de discours spécialisés, Bern, Peter Lang, pp. 337-348. 
474

High frequency nouns in English: aspects of a grammatical description 
Michaela Mahlberg 
Dipartimento di Lingue Letterature e Tradizioni Culturali Anglo-Germaniche, Universita degli Studi di Bari 
m.a.mahlberg@web.de, michaela@ccl.bham.ac.uk 
Abstract 
The term 'noun' belongs to the basic terminology used both in theoretical descriptions of the English language as well as in school textbooks. Generally, nouns are characterised as referring to people, things, and abstract ideas; they are inflected for number, and they typically occur as the head of noun phrases, which in turn can fulfil a number of functions in the clause, such as subject, object, complement etc. Focusing on the relation between the noun and the action or situation described by the verb, semantic interpretations of clause elements often stress the association of nouns and noun phrases with semantic roles such as agent, affected, and recipient (e.g. Biber et al. 1999: 63). While such a characterisation might serve as an overview, we cannot expect that the behaviour of individual nouns or subgroups of nouns is sufficiently accounted for in this way. This becomes particularly obvious when we look at high-frequency nouns and their occurrences in natural language texts. 
Among the ten most frequent noun forms, both in the Bank of English and the BNC, we find time, year, years, and day. The frequency of these nouns is to some extent related to their tendency to occur in time adverbials, for instance in 'frames' like at _____ time, for ______ year(s). Such expressions provide circumstantial information and often take a peripheral role in the structure of clauses. Grammatical descriptions, however, seem to focus on more central functions of nouns in syntactic constructions. Thus it is easy to overlook, for example, the productivity of the singular forms year and day in combinations such as 13-year-old, five-year warranty, three-day event, where one of the features of these nouns is their modifying function. Furthermore, corpus data provides examples of phrases such as years of smoking/suppression. Here the form years is regarded as the head of the noun phrase if we take a purely formal point of view, while a description of its meaning in the given context reveals the emphasing function of years and identifies the following noun (smoking/suppression) as the notional head (cf. Sinclair 1991). These are some examples showing that high-frequency nouns are not easily accounted for by general grammatical descriptions. 
With the focus on high-frequency nouns expressing time the present paper explores various aspects of grammatical descriptions of nouns and makes suggestions for descriptive categories that attempt to combine lexical, structural and textual criteria. Based on evidence from two large corpora, the Bank of English and the BNC, the theoretical approach that is taken stresses the inseparability of meaning, pattern, and function. 
References 
Biber, Douglas; Stig Johansson; Geoffrey Leech; Susan Conrad; Edward Finegan (1999): Longman 
Grammar of Spoken and Written English, Harlow: Longman. 
Sinclair, John (1991): Corpus, Concordance, Collocation, Oxford: OUP. 
484

'Constructing comparable and parallel corpora for terminology extraction – work in progress' 
Belinda Maia 
Faculdade de Letras da Universidade do Porto 
Via Panorâmica s/n, 
4150-564 Porto, 
Portugal. 
bmaia@mail.telepac.pt 
This paper will focus on the theory and practice of creating bi- or multilingual parallel and comparable corpora for research into specialized language. Such corpora pose specific problems because – apart from the usual problems of copyright and accessibility of texts - the texts need to be evaluated on the basis of domain, register and style. One also needs to consider the relative advantages and disadvantages of parallel and comparable corpora and the type of research for which they are required. 
Parallel corpora are understood to be originals texts aligned with their translations. Aligned texts can be made into the translation memories that assist translators with their work, and they can be used by researchers to analyze a wide variety of linguistic phenomena related to translation, particularly at the lexical, syntactic and semantic levels. Comparable corpora are original texts in two or more language that are similar in subject matter, or domain, register and style. The theoretical side of compiling such corpora is complex, because of the differing opinions on the comparable aspects of texts in different languages. 
There is considerable interest in corpora of this kind for areas of research that range from terminology management, through information retrieval to knowledge engineering. We shall begin by suggesting ways in which texts can be selected, and how the cooperation of domain specialists can be maximized. We shall analyze the possibilities and problems of using such corpora to extract terminology and examine the methodology being developed from this for information retrieval. We shall also look at other ways in which such corpora can be used for research, and special attention will be paid to the distinction between general and specific language. The practical aspects of such corpora building will be discussed, with special reference to our experience of working with corpora and terminology with projects and dissertations at postgraduate level. 
The work undertaken requires cooperation between the linguist and experts in information technology and the specialized domains, and we shall explain why such interdisciplinary work is essential to progress in a field which at present suffers from fragmentation and lack of communication. The paper will describe the efforts of the Porto branch of the Linguateca (see: http://www.linguateca.pt ) to develop a faculty site for corpora as well as bi-lingual specialized corpora of comparable and parallel texts in Portuguese and English. 
485

Learner corpus building and a ‘living’ university foreign language curriculum 
Kevin Mark 
Meiji University 
1. An educational slant on learner corpus research 
In this paper I would like to outline an educational approach to learner corpus research that can be contrasted with what can be called an applied linguistic approach. Briefly, I see the prime aim of an applied linguistic approach to learner corpus work as this: to be able to provide descriptions of learner language in relation to the target language that would not otherwise easily be accessible, if at all. An educational approach, on the other hand, sees a concern with quality of linguistic description as being subordinate to the quality of learner experience: particularly in terms of self-confidence, self-esteem, enthusiasm and engagement. I am deeply interested in the linguistic insights that can be gained from learner corpus work, and by the potential for learner corpus research to enhance materials and test development, but I am a teacher before I am a researcher: I feel comfortable saying, as some anonymous teacher is credited with having said, “I don't teach English. I teach people.” What I would like to briefly discuss here is how this attitude is finding expression in the new phase of corpus building that I am about to embark on. 
2. The TRIO learner corpus project 
For some eight years or so years now I have been working on the TRIO project, a parallel corpus of Japanese university learner English, which now consists of roughly 160,000 words of learner English. It is a 3-way parallel corpus, because the learners’ English has been reformulated by me, and a Japanese version. has been subsequently added. Sentences or—in some cases—sub-sentential word groups are aligned directly with each other within the Filemaker database designed for this purpose, and it is possible to toggle from an individual line to seeing it within the passage from which it was taken. In the process of exploring or analyzing the data, a typical search will involve searching for all lines in which a given item is present in the reformulated version but not in the learner version. I have written a number of papers on the methodology and long-term potential of the project, as well as some fairly detailed—if not rigorous or exhaustive—descriptions of various contrastive aspects of the native English and learner versions. Some of these papers are listed in the references (Mark 1997, 1998, 2000, 2001, 2002, 2003). 
3. An educational current in corpus work: data-driven learning 
Although it is not conventional to frame a discussion of corpus research—or even learner corpus 
496
research—in terms of general educational principles, there is a clear educational current in what we call "data-driven learning"(DDL), now well-established as a way for learners to become engaged with the language by exploring it as they find it in a corpus. Aston (2001, p19 ) summarizes the processes of data-driven learning as a form of ‘discovery learning', a term derived from general education rather than language teaching, and which is associated with the work of Jerome Bruner. Bruner's depiction below of the notion of 'discovery' (in relation to ideas) could easily be modified so that it relates to language rather than ideas. It clearly resonates with the benefits claimed for DDL: that giving language learners opportunities to explore the language through a corpus can encourage them to formulate questions and hypotheses that they can test them for themselves; and that being able to do so can also be powerfully self-affirming. 
Just as a physicist has certain attitudes about the ultimate orderliness of nature and a conviction that order can be discovered, so a young physics student needs some working version of these attitudes if he is to organize his learning in such a way as to make what he learns usable and meaningful in his thinking. To instill such attitudes by teaching requires something more than the mere presentation of fundamental ideas. Just what it takes to bring off such teaching is something on which a great deal of research is needed, but it would seem that an important ingredient is a sense of excitement about discovery—discovery of regularities of previously unrecognized relations and similarities between ideas, with a resulting sense of self-confidence in one's abilities. (Bruner 1963, p20) 
4. From a long period of gestation and experimentation: a new kind of course 
My department, the School of Political Science and Economics, is about to open a new course known as the Advanced Communicative English (ACE) program. Before I go on, I should say that the word “Advanced” is a little pretentious for what we are actually doing: this appellation has been selected, to some extent, for diplomatic and promotional reasons rather than for the reality of the English with which we are dealing. In fact we have a wide band of abilities within the class; and we also go out of our way to encourage students to enter the course if they feel drawn towards the activities it offers, even if we sense that their English is fairly weak. We do this providing they seem to be sufficiently motivated to persevere. Since there is a strong self-access element, it is possible to cater for different levels, and the meaning of “advanced” should therefore be taken to be this: by the time a student completes the 2-year program his or her English should, if all has gone well, have significantly advanced. 
The course, which will be starting in April this year, represents an expansion of an existing program that has been going for twelve years. This course has been small enough in terms of student numbers to allow ideas and projects to germinate and grow. As a prototype, it is now sufficiently tested as to be ‘safe’ with a larger group of students. One of the most pleasing aspects of this course is 
497
that in recent years the level of student enthusiasm has been steadily gathering momentum. The positive atmosphere seems to derive primarily from the students’ understanding that the course does not represent a static body of material which they have to master, but rather that it is something dynamic and living, and that their input is crucial to its success as a whole. It is, in other words, a ‘living’ curriculum. With this positive student energy as a base, we are about to enter a new phase of corpus building. 
5. Corpus building within a ‘living’ curriculum: a diary writing project 
The curriculum that began to crystallized last year, that will form the basis for curriculum development in ACE, can be described as self-renewing, ‘living’ and ‘owned’ by students in the sense that they themselves generate, through the activities in which they take part, the content of the materials which they, their peers and their juniors use for study. It is thus possible for students within a particular group to study with materials that have emerged from their own or their peers’ activities, or for students in a group to work with materials produced by their seniors. 
Figure 1 shows both how this process of materials generation is also combined with corpus building. In order to give a more concrete sense of how this ‘living curriculum’ works, I will supplement the diagram with the description of a diary writing project that went on over the course of the April to July semester in 2002. The success of this experience was one of the main factors that led to the planning of the much more ambitious and comprehensive teaching / research project in the 2003-4 academic year (which is about to begin this April). 
The class last year had 15 students, and over a period of roughly 8 weeks, two students each week were asked to send to me by e-mail at least 100 words about their daily experiences. Although nominally ‘diary entries, the writing tended to go well beyond simple narratives, and included a good deal of reflection, soul-searching and humour. Many were much longer than the minimum requirement. Once I received these diaries, they were rewritten by me so as to sound native-like, almost invariably within a day or two. I usually gave no comment about the language itself, but would send back to the student a personal comment of some kind. 
498499
The very positive response from the students to this activity seemed to be primarily due to these factors: 
a) They knew that their writing would be read with interest, both by their peers and by myself. 
b) The rewriting was done very promptly. 
c) The students got to know each other much better through the activity. 
d) The activity was reasonably challenging without being over-demanding. 
e) Since the writing which each person produced was read by fellow students only in a teacher-corrected version, there was no embarrassment or pride associated with the quality of the English itself. 
When this process was completed, I asked each student to look at both my version and at their own, and to produce as natural-sounding a Japanese version as possible. Once this was done, I was able to enter the three versions of the writing into TRIO, thereby increasing the size of the corpus by about 10,000 learner words. In the process of producing the Japanese version, several students commented that they noticed many intriguing differences in the native and learner versions that they had not noticed when they originally read my rewritten version. 
While the rewriting was going on, I had been developing some cloze software with which the students could review and practice with my version of their own and each others’ writing. This software allows for quite a sophisticated range of over 20 types of cloze and other exercises, including multi-word items of up to five words, as well as for dictation practice. I was able to give the students several hours of practice with it in class time. At the time, although it worked tolerably well in the classroom situation, I judged that taking it home might result in glitches that could be time-consuming to deal with. The bugs are now ironed out, and the software is ready to be used by students at home during the new academic year. It incorporates a further dimension of corpus building, in that it allows for the students’ input to be sent on to the teacher by e-mail, in an attached database file. This means that special parallel corpora of cloze items, as well as a parallel corpus of entire dictation sentences, will in future be generated. 
One of the pleasing aspects of this experience has been that several if not all of the students not only enjoyed the process, but also developed an understanding of the methodology behind the course. Knowing what a course is setting out to achieve, how the processes fit together and understanding that one is responsible as a student for the success of one's own course is a powerful educational experience in itself, independent of language learning. 
500
6. Sub-corpora based on other types of activity. 
Similar cycles, with different activities, were started last year and although rewriting is not yet complete, they were carried out with sufficient success to make it seem worthwhile to have them incorporated into the new phase of development. It is therefore planned that there will be several sub-corpora incorporated into the TRIO project, each of which will be directly applicable to instruction that prepares students for specific activities. 
6.1 Simulation language 
One of these activities is a 45-minute management simulation. Last year one such session was carried out, with relatively little student preparation. Three groups of five students each took part, separately, in the same simulation. Two groups spoke in English, while one group spoke only in Japanese. A contrived native English version of this spoken language, which mirrors each of these spoken language mini corpora, is being prepared. New groups of students will perform the same simulation next year, but will first have the opportunity to prepare for it by using worksheets and computer materials designed to focus their attention on language missing from English of the two original groups, and on English, dictation and cloze practice based on the mini TRIO corpus generated by this activity. It is anticipated that a number of simulations will be treated in this way, allowing for the building up of a corpus-based ‘simulation course'. 
6.2 A ‘questionnaire corpus’ 
A limited amount of work has so far been done to develop a ‘questionnaire corpus'. Experience has shown that students tend to find it interesting to devise questionnaires, conducting a survey and reporting back on their findings. The question writing for these can be seen as a kind of pseudo-spoken language, since many of the questions which students ask are ones which they would want to be able to use in spoken conversation. 
6.3 Video production 
One of the most impressive achievements of this particular group of students was a 30-minute video story, an entirely student-produced murder mystery, with an accompanying live trial scene on stage, in which the audience participates by listening to the defence and prosecution and then voting as a jury. At the time of writing key scenes from this production are being prepared for a computer-based multimedia study unit that will be used with first-year students as a way of introducing them both to the use of computers for self-paced study and to the creative possibilities of the course. 
6.4 Web site 
Students have created a bilingual web site including a wide range of information related to the course, opportunities for study abroad, travel and also short reports on pleasure reading and video watching. All of the different types of material that are found in the web site are valid input for the main TRIO corpus, but there is 
501
one genre that seems to be worthy of being given the status of a TRIO sub-corpus: that of writing a report on a book or film that one has enjoyed. 
6.5 Building a parallel corpus of test items 
In addition to the 3-way sub-corpora outlined above, the corpus-building / teaching system outlined in this paper also allows (Figure 2) for the rapid development of a database (generated by the cloze and dictation practice) of items correctly or wrongly matched by students. It is anticipated that, as the ACE programme develops, the corpus and the teaching materials will feed into a large pool of potential test items which will form the basis for adding a further dimension to the project. 
FIGURE 2TeachingMaterialsTRIOparallellearnercorpusTestsDatabaseofTest Items&Responses toTestQuestions 
7. Envoi 
This has been a very brief sketch, and there are inevitably many aspects of the TRIO project that are left untouched. The reader is likely to feel there are many unanswered questions. In particular, the concept of reformulating large amounts of learner written and spoken language, the rights and wrongs of so doing and the practical problems it entails have not been touched upon at all. It can be said, however, as a final note, that students do not seem to have found this ‘hypercorrection’ to be intrusive, but rather seem to find it very valid in the context of the activities in which it is used. When one wishes to make a presentation or give a performance, one naturally wants to put as much care into it as one can. 
Let me refer in conclusion to another classic of education which captures one aspect of the ACE curriculum, although I risk sounding patronizing of students since the quotation refers to children. I believe it captures well the spirit of the way the ACE programme combines corpus building and teaching: 
All children like to express themselves through the medium of form and color. If you simply indulge this interest by letting the child go on indefinitely, there is no growth that is more than accidental. But let the child first express his impulse, and then through criticism, question, and suggestion bring him to consciousness of what he has done, and what he needs to do, and the result is quite different. (Dewey 1956 ) 
References 
Aston G, 2001 Learning with corpora: an overview. In Aston, G (ed), Learning with corpora. Houston, 
502
Athelstan, pp 7-45. 
Bruner J, 1963 The process of education. New York, Vintage. 
Dewey, J, 1956 The School and Society. Chicago, The University of Chicago Press 
Mark, K, 1997 Curriculum development issues and the need for a learner corpus. In Berry, V, Adamson, B., Littlewood, W.(eds), Applying Linguistics—Selected Proceedings of the 1996 International Conference of the Institute of Language in Education The English Centre, University of Hong Kong pp 139-151 
Mark, K. 1998, The significance of learner corpus data in relation to the problems of language teaching, Meiji University Bulletin of Arts and Sciences No. 312, September pp. 77-90 
Mark, K. 2000, Analysis of a corpus of Japanese learner English: applying insights to the development of ‘frequent error’ reference materials , Meiji University Bulletin of the Institute for Research in the Humanities No. 47, pp 233-254 
Mark, K. 2001 A Parallel Learner Corpus: Using Computers in a Humanistic Approach to Language Teaching and Research, The Japanese Journal of Language in Society, Vol.4 No. 1 pp 5-16 
Mark, K. 2002 The TRIO Project: an Integrative Approach to teaching and Learner Corpus Building, Meiji University Bulletin of Arts and Sciences No. 360 (September) pp 69-89 
Mark, K. 2003 A Learner Corpus-Based Approach to Lexical Syllabus Specification and Curriculum Design: Examples with the Verb BE, Meiji University Bulletin of Arts and Sciences No. 361 (January) 
503

Fuck revisited 
Tony McEnery & Zhonghua Xiao, Lancaster University. 
1. Introduction 
This paper is a follow up to the investigation of McEnery, Baker and Hardie (2000) into the use of the word fuck in spoken British English. Both that paper and this are based on the British National Corpus. However, at the time of writing in 2000, the analysis of fuck in the written BNC had not been completed, hence the 2000 paper focussed on spoken English alone. In doing so, it explored the way fuck varied with respect to a range of meta-data encoded in the spoken BNC, principally age, sex and social class. We have now explored the written section of the BNC, and have explored the distribution of fuck with respect to a subset of the metadata encoded in the written BNC, namely domain, author gender, author age, audience gender, audience age, audience level, reception status, medium of text and date of creation. As some of these features have clear analogues in the spoken BNC (most clearly age and sex) comparisons between the work presented here and the earlier work on spoken English will be presented wherever possible. Throughout, unless otherwise stated, references to the frequency of usage of features in spoken language are taken from McEnery, Baker and Hardie (ibid). 
2. Domain 
This section examines the distribution of fuck in written language. Table 1 compares 9 written domains encoded in the BNC. 
Table 1: Domains of written section of the BNC 
Domain Words RF1 NF2 LL ratio Sig. level Imaginative 19664309 1485 75.52 2827.945 <0.001 Arts 7014792 208 29.65 Leisure 8991740 98 10.9 World affairs 15243340 73 4.79 Commerce/business 6668357 29 4.35 Social science 12186378 45 3.69 Applied science 7341375 21 2.86 Belief/thought 3035896 3 0.99 Nat./pure science 3746901 0 0 
Clearly, the distribution of fuck is statistically significant by domain in written English. Forms of fuck are used most frequently in imaginative writing, probably because texts of this category are primarily fiction which contain a lot of reports of speech and are more akin to spoken language. This is followed by the domains of arts and leisure. In contrast, fuck occurs rarely in the domains of belief/thought and is non-existent in natural/pure science. This distribution pattern also applies to the individual word forms of fuck. 
3. Gender of author 
One might imagine that author gender would have a similar effect on the pattern of uses of fuck to that of speaker gender. In Baker, McEnery and Hardie (2000), males were found to use the word fuck much more frequently than females. This tendency does indeed seem to translate into writing also. As can be seen from Table 2, male authors use fuck more than twice as frequently as female authors. This difference is significant at the level of p<0.001 (LL=162.124, 1 df). The difference between the two genders is also quantitatively significant for each word form, though the significance level may vary, with fucking demonstrating the greatest contrast. In terms of word forms, while female authors appear to prefer fuck to fucking 
1 RF denotes the observed frequency of the feature in the BNC. 
2 NF denotes the normalised frequency per million words of the feature in the BNC. 
504
more than male authors (see Table 4), the difference is not statistically significant (LL=0.439, 1 df). The proportion and rank of word forms show a very similar distribution pattern across author gender (Table 3). The fluctuation in the normalized frequencies can be discarded (LL=1.162, 3 df). 
Table 2: Gender of author: 
Form Gender Words RF NF LL ratio Sig. level fuck Male 31586324 486 15.39 28.625 <0.001 Female 15497994 147 9.49 fucked Male 31586324 78 2.47 7.549 0.007 Female 15497994 20 1.29 fucks Male 31586324 14 0.44 6.503 0.029 Female 15497994 1 0.06 fucking Male 31586324 709 22.45 128.474 <0.001 Female 15497994 132 8.52 fucker(s) Male 31586324 35 1.11 7.142 0.012 Female 15497994 6 0.39 All forms Male 31586324 1322 41.85 162.124 <0.001 Female 15497994 306 19.74 
Table 3: Proportion and rank of word forms by male and female authors 
Gender Form Proportion (%) Rank Male fucking 53.63 1 fuck 36.76 2 fucked 5.90 3 fucker(s) 2.65 4 fucks 1.06 5 Female fucking 43.14 2 fuck 48.04 1 fucked 6.54 3 fucker(s) 1.96 4 fucks 0.33 5 
Table 4: Comparison the normalized frequencies of word forms across gender 
Form Male Female LL ratio Sig. level LL ratio Sig. level fucking 22.45 8.52 0.439 0.570 1.162 0.867 fuck 15.39 9.49 fucked 2.47 1.29 0.680 1.000 fucker(s) 1.11 0.39 fucks 0.44 0.06 
4. Age of author 
Author age in written language is a sociolinguistic variable comparable to speaker age in spoken language and may, therefore, influence the distribution of fuck. Table 5 compares age groups of authors in the BNC written section by word form. 
As can be seen, the differences in the frequencies of fuck between authors of different age groups are statistically significant when all word forms are taken as a whole. A comparison by word form shows that except for the two very infrequent words fucks (3 instances) and fucker(s) (9 instances), all of the other word forms demonstrate a significant variation between age groups. 
While young people also use fuck a lot in written language as they do in spoken language, the pattern of using fuck in written language appears to be different from that in spoken language in spite of some similarities, as shown in Table 6. In written English, the age group 60+ uses fuck least frequently. However, authors aged 25-24 use fuck most frequently, followed by the age group 45-59. While authors aged 45-59 use fuck slightly more often than those aged 34-45, the difference is not statistically significant (LL=1.721, p=0.217). Like speakers under 15, authors of the same age group use fuck are amongst the most frequent users of the word fuck, though the frequency of usage is not as high as in the spoken language. 
505
Surprisingly, people aged 15-24 use fuck less frequently than expected in written English, though this age group is the most frequent user of fuck in spoken English. 
Table 5: Age of speaker 
Form Age Words RF NF LL ratio Sig. level fuck 0-14 581962 3 5.15 178.234 <0.001 15-24 437149 3 6.86 25-34 1325516 97 73.18 35-44 2813226 32 11.37 45-59 2847335 36 12.64 60+ 2451519 14 5.71 fucked 0-14 581962 0 0 46.263 <0.001 15-24 437149 0 0 25-34 1325516 20 15.09 35-44 2813226 5 1.78 45-59 2847335 11 3.86 60+ 2451519 0 0 fucks 0-14 581962 0 0 3.286 0.778 15-24 437149 0 0 25-34 1325516 1 0.75 35-44 2813226 1 0.36 45-59 2847335 1 0.35 60+ 2451519 0 0 fucking 0-14 581962 12 20.62 121.236 <0.001 15-24 437149 5 11.44 25-34 1325516 87 65.63 35-44 2813226 36 12.8 45-59 2847335 41 14.4 60+ 2451519 21 8.57 fucker(s) 0-14 581962 2 3.44 7.216 0.129 15-24 437149 0 0 25-34 1325516 3 2.66 35-44 2813226 1 0.36 45-59 2847335 4 1.4 60+ 2451519 1 0.41 All forms 0-14 581962 17 29.21 336.394 <0.001 15-24 437149 8 18.3 25-34 1325516 208 156.92 35-44 2813226 75 26.66 45-59 2847335 93 32.66 60+ 2451519 36 14.68 
Table 6: Comparison of spoken and written languages 
Age group Spoken Written NF Rank NF Rank 0-14 851.01 2 29.21 3 15-24 1549.26 1 18.3 5 25-34 618.65 3 156.92 1 35-44 74.99 5 26.66 4 45-59 138.86 4 32.66 2 60+ 18.71 6 14.68 6 
5. Gender of audience 
The BNC classifies the gender of the intended audience of writing contained in the corpus into four types: male, female, mixed and unknown. In this section, we will only consider the first three categories. Table 7 compares the use of different word forms across gender. 
506
Table 7: Gender of audience 
Form Gender Words RF NF LL ratio Sig. level fuck Male 2451934 21 8.56 0.521 0.471 Female 6235502 44 7.06 Mixed 54289029 591 10.89 --- --- fucked Male 2451934 17 6.93 28.091 <0.001 Female 6235502 3 0.48 Mixed 54289029 90 1.66 --- --- fucks Male 2451934 0 0 --- --- Female 6235502 0 0 Mixed 54289029 14 0.26 -- -- fucking Male 2451934 24 9.79 1.405 0.236 Female 6235502 45 7.22 Mixed 54289029 701 12.91 --- --- fucker(s) Male 2451934 0 0 --- --- Female 6235502 0 0 Mixed 54289029 43 0.79 --- --- All forms Male 2451934 62 25.29 10.270 0.001 Female 6235502 92 14.75 Mixed 54289029 1439 26.51 --- --- 
As can be seen from the table, when word forms are considered together, the difference between audience genders is statistically significant. However, fucked is the only word form which, in itself, shows a significant difference of distribution across writing intended for males and writing intended for females. Fucked is frequently used as the past form of the word with its literal meaning. Writing with an intended female audience contains significantly less occurrences of fucked than writings for an intended male audience. Other word forms (especially fuck and fucking) used for emphasis do not show a significant contrast. 
Interestingly, writing intended for a mixed audience is quite similar to writing intended for a male audience in terms of distribution patterns of fuck (LL=0.134, df=1, p=0.714) when all word forms are taken together. The difference in distributions of fuck in writing intended for females and that for a mixed audience is statistically significant at the level p<0.001 (LL=35.363, 1 df). With respect to individual word forms, the difference between writing with an intended male audience and writing intended for a mixed audience is not statistically significant while the difference between writing with an intended female audience and writing intended for a mixed audience is significant for fuck and fucking. For fucked, the difference of writing for the three types of audience is significant, though writing intended for a mixed audience is more akin to writing with an intended female audience. 
6. Age of audience 
This section examines the possible influence of audience age on the pattern of uses of fuck in written English. There are four age groups for audience: adults, teenagers, children and unknown. We will consider the first three categories. Table 8 gives the frequencies of fuck across these age groups. 
As can be seen from the table, writing for adults contains nearly twice as many uses of fuck as writing for teenagers. Fuck occurs in writing for adults over 7 times as frequently as in writing for children. This difference is significant at the level of p<0.001. In terms of word forms, the greatest contrast is in fucking, followed by fuck while fucked, fucks and fuckers do not show a significant contrast because of the low overall frequencies of these word forms.3 This finding is in line with the social convention that writing for children should avoid swearwords in an attempt to influence their linguistic behaviour, i.e. to encourage them not to use the words.4 
3 There are only 2.73, 0.22 and 1.76 instances of fucked, fucks and fucker(s) per million words. 
4 The desire to shield children from swearwords is apparent in other aspects of British public life, such as the decision not to broadcast such words on the television before 9 p.m. Such a decision has clearly had little influence on the linguistic habits of children, as shown in table 6, and teenagers themselves, it would appear, are not in favour of being shielded from such language - a recent web-based poll amongst children below the age of 15 showed that nearly half of them (46.7%) believed that there should be 
507
swearwords in young adult novels because normal teenagers swear and to avoid using swearwords would be unnatural. (http://www.dream-tools.com/tools/polls.mv?view+youngadultspeech, accessed on 9th December 2002). 
Table 8: Age of audience 
Form Age Words RF NF LL ratio Sig. level fuck Adult 82335639 784 9.52 14.482 0.001 Teenager 1697721 10 5.89 Child 969382 1 1.03 fucked Adult 82335639 128 1.55 0.755 0.712 Teenager 1697721 2 1.18 Child 969382 0 0 fucks Adult 82335639 18 0.22 0.110 1.000 Teenager 1697721 0 0 Child 969382 0 0 fucking Adult 82335639 960 11.66 22.217 <0.001 Teenager 1697721 7 4.12 Child 969382 2 2.06 fucker(s) Adult 82335639 48 0.58 1.412 0.347 Teenager 1697721 2 1.18 Child 969382 0 0 All forms Adult 82335639 1938 23.54 37.603 <0.001 Teenager 1697721 21 12.37 Child 969382 3 3.09 
7. Level of audience 
The BNC annotation scheme includes information pertaining to the levels of intended readership for a document, thus enabling us to explore the pattern of uses of fuck in this dimension. Table 9 compares the distribution of fuck in writings for different levels of audience. 
Table 9: Level of audience 
Form Level Words RF NF LL ratio Sig. level LL ratio Sig. level fuck Low 17126603 229 13.37 7.998 0.005 118.407 <0.001 Medium 43837214 465 10.61 High 23967568 101 4.21 --- --- fucked Low 17126603 32 1.87 0.086 0.660 10.527 0.005 Medium 43837214 77 1.76 High 23967568 21 0.88 --- --- fucks Low 17126603 5 0.29 0.384 0.826 0.853 0.671 Medium 43837214 9 0.21 High 23967568 4 0.17 --- --- fucking Low 17126603 243 14.19 2.73 0.098 52.212 <0.001 Medium 43837214 547 12.48 High 23967568 179 7.47 --- --- fucker(s) Low 17126603 13 0.76 0.001 0.980 12.749 0.002 Medium 43837214 33 0.75 High 23967568 4 0.17 --- --- All forms Low 17126603 522 30.48 9.711 0.002 178.857 <0.001 Medium 43837214 1131 25.8 High 23967568 309 12.89 --- --- 
It can be seen that the rate of usage of fuck declines with a higher audience level. As far as word forms are concerned, the difference between audience levels is statistically significant for all words expect fucks, which occurs rarely. The greatest contrast is found for fuck (LL=118.407). It is also interesting to note that the medium level is closer to the low level than to the high level. Except for fuck, the difference between different audience levels is not 
508
statistically significant. While it is not clear why fuck shows a significant contrast, we speculate that the contrast is due to its high overall frequency. When all word forms are taken as a whole, the difference between medium and low levels is significant (LL=9.711, 1 df). But this significance level is weighted by the marked contrast for fuck. 
8. Reception status 
In this section, we will examine the potential relationship between reception status and the pattern of usage of fuck. The BNC classifies written texts into four types in terms of their reception status: high, medium, low and unknown. We will discard cases where reception status is unknown. As can been seen from Table 10, whether we consider the word forms of fuck separately or together, the difference in the distribution of fuck across reception status is statistically significant. In this case, medium reception status appears to be closer to high than low status. In terms of word forms, the difference between high and medium reception statuses is only significant for fucks and fucking. 
Table 10: Reception status 
Form Level Words RF NF LL ratio Sig. level LL ratio Sig. level fuck High 24138350 278 11.52 1.353 0.245 73.179 <0.001 Medium 31885282 402 12.61 Low 16488041 83 5.03 --- --- fucked High 24138350 40 1.66 0.776 0.381 8.456 0.015 Medium 31885282 63 1.98 Low 16488041 15 0.91 --- --- fucks High 24138350 11 0.46 7.357 0.007 7.077 0.025 Medium 31885282 3 0.09 Low 16488041 4 0.24 --- --- fucking High 24138350 402 16.65 6.252 0.012 179.914 <0.001 Medium 31885282 447 14.02 Low 16488041 60 3.64 --- --- fucker(s) High 24138350 13 0.54 3.006 0.083 9.681 0.008 Medium 31885282 30 0.94 Low 16488041 4 0.24 --- --- All forms High 24138350 744 30.82 0.639 0.424 245.785 <0.001 Medium 31885282 945 29.64 Low 16488041 166 10.07 --- --- 
We can get a vague picture of the pattern of usage of fuck across reception status by sorting by normalized frequencies, as shown in Table 11. The table by itself does not show a pattern of fuck usage. However, if we combine Tables 10and 11 and take statistical significance into consideration, we are able to see clearly the pattern of usage for fuck across reception status. 
Table 11: Distribution pattern of fuck by reception status 
Row Form High Medium Low 1 Fuck 2 1 3 2 fucked 2 1 3 3 Fucks 1 3 2 4 fucking 1 2 3 5 fucker(s) 2 1 3 6 All forms 1 2 3 
Table 10 shows that the difference between high and medium reception statuses is not statistically significant for fuck (p=0.245), fucked (p=0.381) and fucker (p=0.083), hence High and Medium in rows 1, 2 and 5 in Table 11 can be swapped, i.e. High (1), Medium (2) and Low (3). Note, however, that the ranks of High and Medium cannot be inverted, because the inverted order cannot explain the statistical significance as shown by fucks (p=0.007) and fucking (p=0.012). As the difference between high and medium reception statuses is significant for fucks and fucking, High and Medium cannot be swapped in rows 3 and 4. 
509
However, in row 3, Medium and Low can be swapped (i.e. High (1), Medium (2) and Low (3)) because the difference between these two categories is not statistically significant (LL=1.551, 1 df, p=0.213). These rearrangements clearly present the pattern of usage of fuck across reception status: High>Medium>Low. This format is in harmony with the pattern observed when all word forms are taken as a whole, as shown in row 6 in Table 11. This finding is unusual but true. The explanation for this phenomenon, however, is beyond the corpus-based approach and would require, at the very least, substantial sociological study to explain. 
9. Medium of text 
Five basic types of medium of text are annotated in the BNC corpus. This section will use this information to examine the effect of publication medium on the distribution pattern of fuck. Table 12 compares the rate of usage of fuck across medium. 
It is clear that for all of the word forms, the contrast between types of medium is statistically significant. While miscellaneous unpublished ranks before book for four out of five word forms (fuck, fucked, fucks and fucker(s)), the difference in the frequencies between the two media is not statistically significant. Hence, for these word forms, book and miscellaneous unpublished can be re-ordered. Book ranks before miscellaneous only for fucking. Yet the difference in its frequencies between the two types of medium is significant, therefore book and miscellaneous unpublished cannot be re-ordered. Fuck is most frequently used in book, followed by miscellaneous unpublished, periodical, miscellaneous published and written-to-be-spoken. As can be seen from the table, fuck occurs nearly 5 times as frequently in books as in periodicals, and over 12 times as frequently as in miscellaneous published works. No use of fuck is found in written-to-be-spoken scripts. 
Table 12: Medium of text 
Form Medium Words RF NF LL ratio Sig. level LL ratio Sig. level fuck Book 52574506 667 12.69 0.198 0.657 265.830 <0.001 Mis. unpub. 3461953 47 13.58 Periodical 23978695 80 3.34 --- --- Mis. pub. 3922977 1 0.25 --- --- To-be-spoken 861592 0 0 --- --- fucked Book 52574506 100 1.9 0.740 0.390 22.373 <0.001 Mis. unpub. 3461953 9 2.6 Periodical 23978695 19 0.79 --- --- Mis. pub. 3922977 1 0.25 --- --- To-be-spoken 861592 0 0 --- --- fucks Book 52574506 16 0.3 0.619 1.000 11.720 0.014 Mis. unpub. 3461953 2 0.58 Periodical 23978695 0 0 --- --- Mis. pub. 3922977 0 0 --- --- To-be-spoken 861592 0 0 --- --- fucking Book 52574506 875 16.64 22.333 <0.001 430/306 <0.001 Mis. unpub. 3461953 25 7.22 Periodical 23978695 59 2.46 --- --- Mis. pub. 3922977 8 2.04 --- --- To-be-spoken 861592 0 0 --- --- fucker(s) Book 52574506 41 0.78 0.030 1.000 11.007 0.018 Mis. unpub. 3461953 3 0.87 Periodical 23978695 6 0.25 --- --- Mis. pub. 3922977 0 0 --- --- To-be-spoken 861592 0 0 --- --- All forms Book 52574506 1699 32.32 6.137 0.013 709.749 <0.001 Mis. unpub. 3461953 86 24.84 Periodical 23978695 164 6.84 --- --- Mis. pub. 3922977 10 2.55 --- --- To-be-spoken 861592 0 0 --- --- 
510
10. Date of creation 
In this section, we will compare written English in the periods 1960-1974 and 1975-1993 to see whether language change has influenced the pattern of uses of fuck in British English. As date of creation is encoded for the written section of the BNC alone, it was not possible to examine changes in the distribution pattern of fuck in spoken English in McEnery et al (2000). As there is no ready made analogue of the spoken BNC available for an earlier period, the exploration of diachronic change in spoken English is, in effect, impossible using the corpus-based methodology. 
Table 13: Date of creation 
Form Date Words RF NF LL ratio Sig. level fuck 1975-1993 75501632 762 10.09 5.241 0.022 1960-1974 2036939 11 5.4 fucked 1975-1993 75501632 128 1.7 6.815 0.009 1960-1974 2036939 0 0 fucks 1975-1993 75501632 18 0.24 0.958 1.000 1960-1974 2036939 0 0 fucking 1975-1993 75501632 937 12.41 0.020 0.888 1960-1974 2036939 26 12.76 fucker(s) 1975-1993 75501632 47 0.62 1.642 0.200 1960-1974 2036939 3 1.47 All forms 1975-1993 75501632 1892 25.06 2.520 0.112 1960-1974 2036939 40 19.64 
As can be seen in Table 13, when all word forms are taken together, there is no significant difference in the frequency of fuck between the two periods under consideration, in spite of a 5% increase in 1975-1993. In terms of word forms, however, there are some remarkable changes. While fucking was used at almost exactly the same rate in the two periods, the frequency of fuck doubled in the latter period. The difference in the frequencies of fucker(s) is not significant, but the use of the word was reduced by half in 1975-1993. It is also interesting to note that, fucked and fucks appear to be a new development in 1975-1993, because the two-million-word texts sampled for 1960-1975 does not contain a single instance of the two words 
11. Conclusion 
Fuck in written English generally acts as we would expect it to, given how it acts in spoken English. It is correlated with writing for a lower level of audience, as it is associated with speech from the lower classes. It is a marker of male readership/authorship as it is a marker of male speakers. Also, it is a word used more frequently by younger writers, just as it is a word more often spoken by younger speakers. 
However, the written BNC also allows us to explore the influence of context of use on the word more clearly than we can using the spoken BNC. As one would expect, the word is associated with more informal sorts of writing, being totally absent from writing in the natural and pure sciences, but most frequent in imaginative writing. The issue of hearer, which is vexatious to explore with the spoken BNC (see McEnery, Baker, Hardie 2000) is translated into intended reader in the BNC. This research question is more tractable, and we can see that texts intended for females shun the word relative to texts intended for males. The effect of this avoidance of fuck is even noticeable in texts intended for mixed audiences, which are more akin to texts for female audiences than male with regard to the frequency of their usage of the word fuck. Yet running contrary to these findings is the finding related to reception status, where high reception status is linked to an elevated level of fuck usage. 
While the investigation presented in this paper is only possible with appropriate corpus resources, we feel that the corpus-based approach is not all-powerful (cf. McEnery, Baker & Hardie 2000: 47) and often the process of explaining corpus findings leads one to methodologies other than the corpus method (McEnery, 2003). Corpora are useful in formulating and testing linguistic hypotheses, but they cannot readily provide explanations for questions such as why people from higher social class use fuck frequently and why writings of the highest reception status contain the most frequent use of fuck. Nevertheless, the corpus 
511
methodology, in combination with other methodologies, is undoubtedly of use in providing descriptions that any purported explanations must account for. 
References 
McEnery, A. 2003. Swearing in English. London: Routledge. 
McEnery, A, Baker, P & Hardie, A. 2000. Swearing and abuse in modern British English. In Lewandowska-Tomaszczyk, B & Melia, P (eds.) PALC'99: Practical Applications in Language Corpora. : Europäischer Verlag der Wissenschaften: Peter Lang. 
512

The Construction of a Corpus to Investigate the Presentation of Speech, Thought and Writing in Written and Spoken British English1 
Dan McIntyre, Carol Bellard-Thomson2, John Heywood, 
Tony McEnery, Elena Semino and Mick Short, 
Lancaster University, UK 
Abstract 
In this paper we describe the Lancaster Speech, Thought and Writing Presentation (ST&WP) Spoken Corpus. We have constructed this corpus to investigate the ways in which speakers present speech, thought and writing in contemporary spoken British English, with the associated aim of comparing our findings with the patterns revealed by the previous Lancaster corpus-based investigation of ST&WP in written texts. We describe the structure of the corpus, the archives from which its composite texts are taken, the decisions that we made concerning the selection of suitable extracts from the archives, and the problems associated with the original archived transcripts. We then move on to consider issues surrounding the mark-up of our data with TEI-conformant SGML, and explain the tagging format we adopted in annotating our data for ST&WP. 
1. Introduction 
The presentation of speech and thought has long been of interest to a range of scholars. Recent research in this area has been done by philosophers (Clark and Gerrig 1990), applied linguists (Buttny 1997; Thompson 1996; Baynham and Slembrouck 1999; Myers 1999), conversation analysts (Holt 1999) and psychologists (Ravotas and Berkenkotter 1998). In stylistics, there is a long tradition focussing on speech and thought presentation in written fiction (see, for example, Banfield 1973; McHale 1978; Leech and Short 1981 and Fludernik 1993). One of the most widely accepted frameworks for the description of the phenomenon in this tradition is Leech and Short's (1981) model. Leech and Short proposed parallel scales of speech and thought presentation categories for the novel, arranged on a cline of different degrees of apparent narratorial interference (see fig. 1). 
NRA NRSA IS FIS DS FDS 
NRA NRTA IT FIT DT FDT 
Fig. 1 The cline of speech and thought presentation categories in Leech and Short (1981) 
As one moves across the cline from left to right, the categories reflect an increasing lack of apparent narrator ‘control’ of the report. This results at the extreme right of the scale in the categories of ‘free direct’ speech or thought, the effect of which is to suggest that what we have in these instances are the words and thoughts of the characters themselves, with no narratorial intervention at all. (The categories themselves are defined below in section 5). 
Descriptions of speech and thought presentation such as the Leech and Short model have generally been based on a combination of intuition and wide reading experience and have been established and illustrated with carefully selected textual examples, chosen to best illustrate particular phenomena. As a result, existing frameworks have remained untested systematically on large quantities of data. In order to address this issue, in 1994 Short, Semino, Culpeper and Wynne embarked on a corpus-based investigation of speech and thought presentation in written literary and non-literary texts (see Short et al. 1996, Semino et al. 1997, Wynne et al. 1998, Short et al. 1999, Short et al. 2002, Short forthcoming and Semino and Short forthcoming). The aim of this initial project was to test the model of speech and thought presentation described in Leech and Short (1981) against a specially constructed quarter-of-a-million word data-set of fictional and non-fictional narratives to see how robust the framework was and how far it would stand up to exposure to corpus data. Among other things, this project introduced an additional scale, parallel to the speech and thought scales, to take account of writing presentation. In this paper we describe the latest phase of this project, which is to further test and refine the model by investigating the nature of speech, thought and writing presentation (henceforth ST&WP) in spoken, as opposed to written, data. To this end we have constructed the 
1 The research presented in this paper was supported by a grant from the Arts and Humanities Research Board (B/B/RG/AN2314/APN12482). We are also grateful to Andrew Hardie and Scott Piao for their help with some technical difficulties. 
2 Carol Bellard-Thomson now works in the School of Languages at the University of Kent, UK. 
513
Lancaster Speech, Thought and Writing Presentation Spoken Corpus. Below we outline in more detail the background to the earlier written project, before going on to describe the spoken corpus and its construction, issues involved in annotation, and the outcomes of some preliminary analyses. 
2. A corpus-based approach to ST&WP 
2.1 The Lancaster Speech, Thought and Writing Presentation Written Corpus 
The Lancaster Speech, Thought and Writing Presentation Written Corpus was built to investigate the nature of ST&WP in written narrative texts. The ST&WP Written Corpus project extended the boundaries of investigation beyond the focus on literary texts in Leech & Short (1981) by including non-literary texts within its remit (see Short et al. 1996). Developed between 1994 and 1997, the corpus is now approximately 260,000 words in size. The relatively small size of this in comparison to most modern electronic corpora is due to the fact that the whole corpus needed to be hand-annotated. It is divided into three narrative genres: (1) prose fiction, (2) newspaper news reports; and (3) (auto)biography. These three genres are then sub-divided into ‘serious’ and ‘popular’ sections. The analysis of the corpus texts resulted in some adjustments to Leech and Short's earlier model and also revealed the necessity of the parallel scale referred to above to take account of the report of writing (see Semino et al. 1999, Short et al. 1999 and Wynne et al. 1998 for more details). 
2.2 The need for a Speech, Thought and Writing Presentation Spoken Corpus 
Work on the ST&WP Written corpus raised the question of the extent to which the quantitative and qualitative results that were arrived at would apply to spoken as opposed to written language. The work that has been done on ST&WP in speech has tended to concentrate purely on direct speech, or has analysed qualitatively small amounts of data gathered from very specific contexts (e.g. Hall et al. 1999; Holt 1999). We have attempted to address this issue by constructing a small, balanced corpus of contemporary spoken British English in order to analyse the presentation of speech, thought and writing in spoken data systematically. Our aim is to further test the model of ST&WP originally proposed in Leech and Short (1981) and expanded in the work of Short, Semino and Wynne (e.g. Wynne et al. 1998), in order to arrive at a systematic and comprehensive framework developed through exhaustive analysis of both written and spoken data. For this reason, in building the corpus we decided to explore both elicited and spontaneous speech. 
3. Selecting the corpus data 
The texts that form our corpus are drawn from two sources: (1) the spoken demographic section of the BNC (World edition); and (2) oral history archives in the Centre for North West Regional Studies (CNWRS) at Lancaster University. Whereas the texts for the written corpus were randomly selected, we deliberately chose spoken texts that appeared to be rich in ST&WP in order to ensure that we had a substantial amount of data to work with (hence we cannot claim that our spoken corpus is representative in terms of the overall amount of ST&WP it contains). 
The Spoken Corpus is approximately 260,000 words in order to make it comparable in size with the existing ST&WP corpus. The CNWRS archives and the BNC obviously provide a far larger body of data than we required, and so we opted to select 120 ‘chunks’ (60 from the BNC and 60 from the CNWRS archives) of approximately 2,000 words each (as this was the size of the texts in the written corpus), providing 240,000 words in total. We also decided that the chunks would not be stopped at exactly 2,000 words, but would be allowed to run on a little, to allow each chunk to represent a coherent stretch of conversation, a decision parallel to that made when constructing the written corpus. This gave us the remaining 20,000 words needed to make our corpus approximately 260,000 words in size. 
The CNWRS data is drawn from two archives. The ‘Family and Social Life’ archive was compiled from data collected in the 1970s and 1980s by Elizabeth Roberts3 and Lucinda Beier4, and consists of 250 hours of interviews, stored on audiocassettes and reel to reel tapes, with accompanying transcripts. We used the transcripts to identify sections rich in ST&WP. The interviewees recall what life was like in Lancaster, Preston or Barrow between the periods 1890–1940 or 1940–1970. The data in the ‘Childhood and Schooling’ archive was collected in the 1980s by Penny Summerfield5, and consists of approximately 200 hours of interviews on audiocassette, with accompanying transcripts. Again, the interviews are one-to-one, with the interviewees recalling their years spent in education 
3 Emeritus Reader in History, Lancaster University. 
4 Professor in the departments of History and Political Science at Illinois State University. 
5 Professor of Modern History at the University of Manchester. 
514
between 1920 and 1950 in Lancaster and Morecambe, Preston, Blackburn, Burnley and Clitheroe. We aimed to balance for male and female interviewees in this data set. 
With regard to the BNC texts, we decided to use only material from the spoken demographic section of the corpus, as this would allow us to contrast spontaneous dialogue with the elicited monologues of the CNWRS archives. Since the BNC data was collected in the early 1990s and the CNWRS data in the 1970s and 1980s, we also left open the possibility of studying diachronic developments in speech. We chose texts from the BNC that cover all age ranges, with an equal division between male and female respondents. We also concentrated solely on face-to-face interaction – we did not use transcripts of radio phone-ins, for example - and we used only those texts which constitute spontaneous, unscripted data. 
After the initial selection of the transcriptions on demographic grounds, we examined each transcript for long turns, on the basis that these were more likely to be narrative turns that would provide a higher density of the kind of features we were interested in. This meant excluding those files which were of a brief question-answer format, or which contained numerous short turns. In addition to this, with the BNC texts we used the BNC Web query facility to search for common discourse reporting verbs. Where the query returned favourable results, we then examined that area of the text in question manually to see if it was likely to yield numerous examples of ST&WP. So, in addition to the reporting verbs picked up in the electronic search, we also looked for further examples of ST&WP in close proximity to these. As with the CNWRS data, each member of the project team then read the texts in order to identify suitable extracts for inclusion within the corpus. 
4. Constructing the corpus 
The transcripts from the CNWRS archives were initially the most problematic as these had originally been transcribed for an oral history research project, without regard for linguistic transcription conventions. In some cases, then, we had to newly transcribe stretches of interaction that had been omitted or simply summarised. We removed anomalous punctuations and corrected misspellings. 
In addition to producing electronic copies of the CNWRS transcriptions, we also made copies of their corresponding sound files. We digitised the cassettes using the CoolEdit software package, which allowed us to convert the original tapes to wav files. We recorded in mono, at 16-bit resolution, in order that the resulting wav files should be in a form suitable for later time-alignment with the transcripts. 
4.1 Mark-up of the corpus 
The 120 files in our corpus are all marked up using TEI- (Text Encoding Initiative) conformant SGML (Sperberg-McQueen and Burnard 2001) in order to create a shareable archive, compatible with other corpora and concordancing packages. The SGML mark-up allows the corpus to be searched using concordancing programs such as Wordsmith Tools and SARA. For each file in the corpus we have generated a header containing bibliographical information about the computer file itself (with which it is possible to catalogue the file in a library archive), information about the types of tags that are used in the file and how the encoders resolved any problems that arose during tagging, classificatory and contextual information about the text, and a history of changes made in the development of the electronic version. We have also generated an overall corpus header and a document-type declaration for the corpus files. 
5. Annotating the corpus for ST&WP 
Having described the structure and composition of our corpus, in this section we explain the system of annotation that we used to tag the files for speech, thought and writing presentation. To enable us to compare our findings from the Spoken Corpus with those of the Written Corpus project (see Semino et al. 1997), we make use of the system of annotation outlined in Wynne et al. (1998), though with some modifications to take account of the differences between written and spoken data. Before describing the category set and outlining the tagging format that we used in annotating our data, it is useful to summarise briefly the categories of ST&WP that we used in analysis. We begin by presenting the category sets we used in both the Written and the Spoken Corpora and consider the changes that we made to our tag-set as a result of working with spoken data. 
5.1 ST&WP categories in the Written and Spoken Corpus projects 
Table 1 details the acronyms used to mark instances of ST&WP in the Written Corpus project and their equivalents in the Spoken Corpus 
515
Categories outside the discourse presentation clines Written Corpus Spoken Corpus Category Definition Category Definition N Narration A Anything other than ST&WP 
(narrative and non-narrative) RU Report of Language Use NRS Narrator's Report of Speech RS Report of Speech NRT Narrator's Report of Thought RT Report of Thought NRW Narrator's Report of Writing RW Report of Writing Discourse Presentation Categories Written Corpus Spoken Corpus Category Definition Category Definition NV Narrator's Representation of Voice RV Representation of Voice NI Narrator's Representation of Internal States RI Representation of Internal State NW Narrator's Representation of Writing RN Representation of Writing NRSA Narrator's Representation of Speech Act RSA Representation of Speech Act NRTA Narrator's Representation of Thought Act RTA Representation of Thought Act NRWA Narrator's Representation of Writing Act RWA Representation of Writing Act NRSAp Narrator's Representation of Speech Act with Topic RSAp Representation of Speech Act with Topic NRTAp Narrator's Representation of Thought Act with Topic RTAp Representation of Thought Act with Topic NRWAp Narrator's Representation of Writing Act with Topic RWAp Representation of Writing Act with Topic IS Indirect Speech IS Indirect Speech IT Indirect Thought IT Indirect Thought IW Indirect Writing IW Indirect Writing FIS Free Indirect Speech FIS Free Indirect Speech FIT Free Indirect Thought FIT Free Indirect Thought FIW Free Indirect Writing FIW Free Indirect Writing DS Direct Speech DS Direct Speech DT Direct Thought DT Direct Thought DW Direct Writing DW Direct Writing FDS Free Direct Thought FDS Free Direct Thought FDT Free Direct Thought FDT Free Direct Thought FDW Free Direct Writing FDW Free Direct Writing 
Table 1 Categories in the ST&WP Written corpus and their equivalents in the Spoken Corpus6 
NRS/T/W and RS/T/W are reporting signals (prototypically reporting clauses), and are not a part of the discourse being presented. They are therefore placed outside the discourse presentational clines. The convention we use is that ST&WP category labels are written in upper-case letters. The ST&WP Written project also developed a set of four additional features that categories might have. These are marked in lower-case to distinguish definitional labels from more minor associated features. The four features are discussed below in Section 5.2.2 as part of the expanded set developed for the Spoken Project. 
5.2 ST&WP categories in the Spoken Corpus project 
For the spoken project we began with the tag set in the left half of Table 1 (see Semino et al. 1997 for a discussion of these categories). However, in the course of annotating the spoken data we made various alterations and additions to our categories and their corresponding acronyms, as shown in the right half. The main changes were as follows: 
• Leech and Short (1981) initially used the term ‘report’ in the description of NRSA and NRTA. This term was replaced by ‘representation’ in some of the later publications describing the written corpus. Short and Semino (forthcoming: Chapter 1, Section 1.1) now argue for the term ‘presentation'. The arguments for and against these alternative terms are too complex to go into here. However, it will be helpful if we point out that we have retained ‘R’ in our various category acronyms in order to preserve as much annotational continuity as possible and we continue to gloss it as ‘representation’ in order to avoid possible confusion for our various readerships. 
6 For a full definition of the linguistic criteria of each category, see Wynne et al. (1998) and Semino and Short (forthcoming). 
516
• We have dispensed with the N constituent of the categories as a consequence of tagging oral texts. N previously stood for ‘narration’ or ‘narrator's', and is not always applicable to non-narrative written data or to spoken data. Hence, what in the written corpus would have been NRSA is in the spoken corpus simply RSA. Likewise, the single N attribute value, which was used in the Written corpus to mark anything not annotated as ST&WP, is replaced in the Spoken corpus by A, which, simply standing for ‘[A]nything other than ST&WP', comprises both narrative and non-narrative text. 
• Dispensing with the N constituent had the knock-on effect of leaving us with the same acronym - RW - to refer both to a reporting clause (or non-clausal equivalent) of writing presentation preceding either the direct or indirect presentation of writing, and the minimal presentation of writing (e.g. ‘I wrote to Eileen'). We therefore needed a different acronym in order to distinguish between the two phenomena. We chose to use RN to refer to the latter, N being the only remaining consonant in the word ‘writing’ that is not used elsewhere in the tag-set. 
• We have introduced a new tag, RU, to refer to ‘report of language use'. This is used to tag instances where speakers refer to words or expressions, often idiosyncratic, that were habitually used either by groups of people or individuals to refer to particular things. A prototypical example would be ‘So we had a box of [RU] what we called wet day stockings'. Instances of RU are most common (220 out of 247 instances) in the CNWRS texts in our corpus where people are talking about their past lives. 
• We have chosen to mark the grammatical structure of instances of ST&WP in an effort to provide more information about the forms of ST&WP in our corpus. We assume the default grammatical structure of a stretch of ST&WP to be declarative and this is not tagged. Imperatives are tagged with ‘p’ and interrogatives with ‘v'. Confusion with the lower case p for ‘topic’ is avoided by their being placed in different positions. 
• We expanded the number of additional features.7 
Below, we explain the acronyms used to refer to the main categories of speech, thought and writing presentation, via some examples from our corpus. We then describe the acronyms for additional feature constituents. All new additions to our tag set were to cope with particular phenomena we encountered in the spoken data. For ease of interpretation, the tags are represented here in simplified form. The full format is presented at the end. 
We now present an explanation of the main categories of ST&WP in 5.2.1, and of possible additional feature constituents in 5.2.2. This is followed by the tagging format we use in 5.3. Our descriptions of the scope of each category and our examples aim to account primarily for central cases, since we do not have the space in this paper to discuss complex and borderline cases. 
5.2.1 Main categories of ST&WP 
As the three scales are in parallel, to bring out what they have in common we combine their definitions as far as possible. In general, speech and writing presentation categories share many formal features and function. Thought presentation categories, however, often display different functional properties. We therefore group speech and writing together and place thought last in the following lists. 
The Direct Categories (DS, DW and DT) 
The direct categories consist of independent clause/s or phrase/s which convey the illocutionary force of speech or writing acts, their propositional content, and which include the deictic features appropriate to the anterior speech, thought or writing event that is being presented. Prototypically, the ‘direct’ categories usually claim to represent the ‘actual words’ used, or to exemplify the kinds of words and expressions typically used. Although Direct Thought is formally similar to Direct Speech and Direct Writing, aspects of the definition of the latter two, such as illocutionary force and ‘actual words', do not sensibly extend to DT. The following are examples: 
Direct Speech (DS) 
1. [A] He looked round [RS] and said to all the lot of us lads he said, he said [DS] I bet you buggers like your fish and chips. 
7 In the early publications arising from the Written Corpus we use a capital ‘P’ in the acronyms NR{S/T/W}AP to indicate a speech, thought or writing act with an extended topic. We now prefer to use a lowercase ‘p’ since NR{S/T/W}AP is not a category in its own right, but simply a category variant. 
517
Direct Writing (DW) 
2. [RWA] he wrote me this letter [RW] saying erm saying [DW] I, I realise that there's been something on your mind recently 
Direct Thought (DT) 
3. [RT] I thought [DT] well I might as well come 
The Free Direct Categories (FDS, FDW and FDT) 
As for DS, DW and DT but without an accompanying RS, RW or RT.8 
Free Direct Speech (FDS) 
4. And I remember al always our Leonard taking me next door and knocking at the door and the [FDS] I've come to show you our Peggy's new frock 
Free Direct Writing (FDW) 
5. Look, they've stuck a sticker in the back [FDW] cars kill trees 
Free Direct Thought (FDT) 
6. I went into the loo [FDT] it stinks of smoke in here 
The Free Indirect Categories (FIS, FIW, FIT) 
The Free Indirect categories are characterised by a mixture of deictic, syntactic and lexical features, some appropriate to current speaker, others to the producer of the anterior speech, writing or thought event that is being presented. They are prototypically realised by an independent clause, but an accompanying RS, RW or RT is sometimes possible. 
Free Indirect Speech (FIS) 
7. [RS] Father said [DS] can my girls come? [FIS] No they couldn't come 
Free Indirect Writing (FIW) 
8. [RW] Dennis, who had been my boyfriend wrote from Italy where he was stationed, [FIW] when he came home at Christmas, could we be engaged? 
Free Indirect Thought (FIT) 
9. [A] I persisted in getting dressed and immediately went home. I was quite <unclear> by that time, but [FIT] I wasn't putting up with this garbage I was going home, that was it 
The Indirect Categories (IS, IW and IT) 
The Indirect categories consist of a reported clause which is grammatically subordinated to an RS, RW or RT. All deictic features are appropriate to the speaker in the posterior, discourse presenting, situation. Prototypically, the propositional content of the original speech, thought or writing act is specified, but no claim is made to present the words and structures originally used to utter that proposition. 
Indirect Speech (IS) 
10. [RS] he said [IS] it made him happy 
Indirect Writing (IW) 
11. [RWr] it was put down on in a book [IWr] that you'd taken a pair of stockings home 
Indirect Thought (IT) 
12. [RT] he thought [IT] there was nowhere else 
Representation of Speech/Writing/Thought Act (RSA/RWA/RTA) 
RSAs and RWAs present the illocutionary force of an utterance or text (part) with an optional noun or prepositional phrase indicating the topic, but do not claim to represent the propositional content or the 
8 The analysis of the written corpus provided support for Short's (1988) proposal that FDS should be seen as a variant of DS, and also suggested that the same applies to FDT and FDW (see Semino et al. 1997 and Semino and Short (forthcoming Chapter 6, Section 6.5.2). 
518
original wording of that content. RTAs are formal equivalents, but the notion of a ‘thought act’ seems likely to have a much more restricted range than speech or writing acts. More specifically, the notion of ‘illocutionary’ force in relation to thought acts is problematic, while that of perlocutionary effect associated with speech and writing acts is inapplicable. 
Representation of Speech Act (RSA) 
13. [RSA] I just threatened them. 
Representation of Writing Act (RWA) 
14. [RWA] Vivian voted Conservative 
Representation of Thought Act (RTA) 
15. [A] I just move some of this stuff out the way, I know, [RTA] I've had a good idea, a smart idea 
Representation of Voice/Internal State/Writing (RV/RI/RN) 
Representation of Voice (RV) 
RV captures minimal references to speech with no indication of the illocutionary force, let alone the propositional content or form of the utterance (part). RVs can present either individual instances of talk or whole Speech Events. As with the RSA category, a reference to a topic may be attached. 
16. I was sitting there [RV] talking [A] and they had a drop of wine 
Representation of Writing (RN) 
RN captures minimal references to writing or writing events or to the writing of an instance of a text-type with possibly a minimal reference to topic, but with no indication of the illocutionary force or of the propositional content or linguistic form of the portion of text. RNs can present either individual instances or a series of writing events, or group participation in them. 
17. they had slates [RNr] and they used to write with a piece of slate 
Representation of Internal State (RI) 
RI captures references to cognitive or emotional states or processes that do not amount to specific thoughts. 
18. [RI] I was frightened to death of him I was really I was frightened to death of him 
Other categories 
Reporting signals (RS, RW and RT) 
RS, RW and RT are prototypically represented by a reporting clause associated with a stretch of direct, indirect, and in some cases, free indirect, speech, thought or writing. As we pointed out in our discussion of Table 2, RS/T/W, as reporting signals, are not a part of the discourse being presented. The RS/RW/RT function is sometimes performed by a noun, adjectival, adverbial or prepositional phrase. 
Report of speech (RS) 
19. [RS] Mrs Hall said [DS] I don't know how you find time to go to your church every morning like this. 
Report of writing (RW) 
20. [RW] across the certificate he wrote [DW] this man should be in bed 
Report of thought (RT) 
21. [RT] I decided [IT] I'd like to be an engineer 
Report of Use (RU) 
RU captures meta-linguistic mentions of language use, such as the words or expressions habitually used to refer to things, or the ways words were spelled or pronounced. 
22. and then you see [RU] what they called the tacklers were over the weavers 
519
Anything other than ST&WP (A) 
The A tag was applied to all those stretches of text which do not contain any references to speech, thought or writing presentation. 
23. [A] Well Mother Monica Mother Mary Monica was the headmistress. 
5.2.2 ST&WP category features 
Of the symbols below, the definitions given here for p (indicating topic), e, h, i, q, and # are those initially developed for the Written Corpus. While we found that e, h, i and # could be applied to the spoken data straightforwardly, the extent to which p and q could appropriately be applied raised theoretical issues that are currently being investigated. The other symbols were adopted during the annotation of the Spoken Corpus. 
p ( = topic) 
The p suffix marks an extended topic, most commonly of a speech, thought or writing act. 
24. [A] Erm I don't ever remember [RSAp] my mother expressing any interest or desire or wish to have a job 
# ( = problematic) 
The symbol # was used to signal ‘problematic’ tags that needed further investigation. 
25. at ten o'clock at night and <pause> pub was packed. [A-RV#] People singing with the the group 
e ( = embedded) 
The suffix e marks instances of discoursal embedding where one ST&WP category is embedded discoursally, but not necessarily syntactically, in another. 
26. [RV] Joan rang last night [RS] to say [IS] that Reg [RSe] had asked us [ISe] to go to to see the daffodils. 
g ( = negative) 
The suffix g marks a grammatical negative. 
27. [A] And, um, well, I suppose I can't I shouldn't say [RSApg] but my father would never allow you to go to dances 
a ( = absence) 
The suffix a signals the marked absence of performance of a speech, thought or writing act. 
28. And I never heard once heard my family turn round [RSa] and say, [DSa] That's my son. 
h ( = hypothetical) 
The suffix h marks an instance of ST&WP that does not present an anterior discourse but “refers” to an event that has not (or not yet) taken place. 
29. [RTh] Well if she wants if she wants [ITh] to get rid of it [RShp] ask her [ISh] how much [RTAehv] she wants for it 
i ( = inferred) 
The suffix i signals instances of thought presentation where the reporter did not have direct access to the relevant thoughts. 
30. and then, and erm, [RTi] this woman, receptionist, whatever, obviously thought [DTi] oh well, [RIei] he knows the guy 
q ( = quotation phenomenon) 
The suffix q marks the presence of a direct quotation which is enclosed within a non-direct category of ST&WP and which does not count as a straightforward example of direct speech. 
31. [A] I think er I agree with er Tennyson on that. I think [RWApq] he spoke of Virgil as wielder of the stateliest measure ever moulded by the lips of man. 
520
r ( = reiterated) 
The suffix r marks an iterated instance of ST&WP. 
32. we appear to be the most consistent pub in the area, with er customers and what have you. They all come in and [RSr] tell us [ISr] we're the busiest [RSr] and I say [DSr] well if we're the busiest, God help those that're the quietest. 
v ( = interrogative) 
The suffix v marks a grammatical interrogative. 
33. [RSv] Did they ever say [ISv] why they did it, why they went to view the body and took children 
p (= imperative) 
The suffix p marks a grammatical imperative. Note that in the tagging format below, p for imperative is differentiated from p for topic by the fact that it appears in a different attribute value slot. 
34. [RS] He said er [DS] [RSep] Tell your mammy [DSep] it'll be alright [A] and we turned back home [NWRS 177] 
u (= unfinished) 
The suffix u signals that the relevant ST&WP category is unfinished. 
35. [RS] and I said [DSu] well that was stra 
1/2/3 etc 
In the Written Corpus, numerals indicate the number of levels of discoursal embedding. In the Spoken Corpus, they are also used to record the number of repeated adjacent categories represented by one label. The different functions are distinguished by the field in which the numeral occurs (see 5.3). 
Level of Embedding 
36. [RT] I felt [IT] I ought [RWAe] to write to him [RT] because I thought [DT] we're both getting old, [RIe] I'd like [RWAe2] to write [RWeh3] and ask him [IWeh3] [RIeh4] if he remembers his father 
Repeated categories 
37. [A] He looked round [RS3] and said to all the lot of us lads he said, he said [DS] I bet you buggers like your fish and chips. 
5.3 The tagging format 
We use the element <sptag> to mark instances of ST&WP. Each constituent of the ST&WP categories are marked within one of fifteen <sptag> attributes. We use ‘x’ as a placeholder for those slots that are not filled for a particular ST&WP category. This is done for ease of concordancing. We use an end tag (</sptag>) to mark the end of a particular stretch of ST&WP. Below is an example of an ST&WP tag. This particular example would be used to mark a stretch of hypothetical Free Indirect Speech: 
<sptag one="F" two="I" three="S" four="x" five="x" six="x" seven="x" eight="x" nine="h"> 
Table 2, below, details the allowable values for each of the fifteen attributes: 
Attribute Allowable values Definitions One x A F Anything other than ST&WP; Free Two x R I D # Representation; Indirect; Direct, # interesting example of A Three x S T W V I N U Speech; Thought; Writing; Voice; Internal state; Writing; Use Four x A Act Five x p topic Six x # 1 2 3 4 # = odd/interesting cases; numerals = repeated adjacent categories Seven x e embedded Eight x g a grammatical negative; marked absence of ST&WP Nine x h hypothetical Ten x i inferred Eleven x q quotation phenomenon Twelve x r iterative Thirteen x v p interrogative; imperative Fourteen x u unfinished Fifteen x 1 2 3 4 numerals = level of embedding 
Table 2 Allowable values for each of the fifteen <sptag> attributes 
521
6. Conclusion 
Our project, so far, has demonstrated that the model of speech, thought (and later) writing presentation suggested by Leech and Short (1981) and developed by Short, Semino and Wynne in their work on the Written Corpus, is applicable to spoken data, with few modifications. Our work on the Spoken Corpus would seem to confirm the robustness of the model of ST&WP that we are using. 
We are currently carrying out quantitative analyses of the various different ST&WP categories in the Spoken Corpus in order to determine the distribution and frequency of these. We aim to compare our quantitative findings with those for the Written Corpus, in order to consider any differences in ST&WP between the spoken and written data. In addition we are also carrying out qualitative analyses of the corpus data in order to try and explain more fully our statistical findings. We will report on these issues in future publications. 
References 
Banfield, A 1973 Narrative style and the grammar of direct and indirect speech. Foundations of Language 10: 1-39. 
Baynham, M, Slembrouck, S 1999 Speech representation and institutional discourse. Text 194: 439-57. 
Buttny, R 1997 Reported speech in talking race on campus. Human Communication Research 234: 477-506. 
Clark, H H, Gerrig, R J 1990 Quotations as demonstrations. Language 66: 764-805. 
Fludernik, M 1993 The Fictions of Language and the Languages of Fiction: The Linguistic Representation of Speech and Consciousness. London, Routledge. 
Hall, C, Sarangi, S, Slembrouck, S 1999 Speech representation and the categorization of the client in social work discourse. Text 194: 539-70. 
Holt, E 1999 Just gassing: an analysis of direct reported speech in a conversation between employees of a gas supply company. Text 194: 505-37. 
Leech, G N, Short, M H 1981 Style in Fiction. London, Longman. 
McHale, B 1978 Free indirect discourse: a survey of recent accounts. Poetics and Theory of Literature 3: 235-87. 
Myers, G 1999 Unspoken speech: hypothetical reported discourse and the rhetoric of everyday talk. Text 194: 571-90. 
Ravotas, D, Berkenkotter, C 1998 Voices in the text: the uses of reported speech in a psychotherapists notes and initial assessments. Text 182: 211-39. 
Semino, E, Short, M forthcoming Corpus Stylistics: A Corpus-based Study of Speech, Thought and Writing Presentation in Fictional and Non-fictional Narratives provisional title. London, Routledge. 
Semino, E, Short, M, Culpeper, J 1997 Using a corpus to test and refine a model of speech and thought presentation. Poetics 25: 17-43. 
Semino, E, Short, M, Wynne, M 1999 Hypothetical words and thoughts in contemporary British narratives. Narrative 73: 307-34. 
Short, M 1988 Speech presentation, the novel and the press. In van Peer, W (ed), The Taming of the Text. London/New York, Routledge, pp 61-81. 
Short, M forthcoming A corpus-based approach to speech, thought and writing presentation. in Wilson, A, Rayson, P, McEnery, A (eds), Corpus Linguistics by the Lune: A Festschrift for Geoffrey Leech. Frankfurt/Main, Peter Lang. 
Short, M, Semino, E, Culpeper, J 1996 Using a corpus for stylistics research: speech and thought presentation. In Short, M, Thomas, J (eds), Using Corpora in Language Research. London, Longman, pp 110-31. 
Short, M, Semino, E, Wynne, M 2002 Revisiting the notion of faithfulness in discourse presentation using a corpus approach. Language and Literature 114: 325-55. 
Short, M, Wynne, M, Semino, E 1999 Reading reports: discourse presentation in a corpus of narratives, with special reference to news reports. In Diller H J, Gert Stratmann, E O (eds), English via Various Media. Heidelberg, Universitatsverlag C Winter, pp 39-66. 
Sperberg-McQueen, C M, Burnard, L (eds) 2001 TEI P4: Guidelines for Electronic Text Encoding and Interchange. Oxford-Providence-Charlottesville-Bergen, The TEI Consortium. 
Thompson, G 1996 Voices in the text: discourse perspectives on language reports. Applied Linguistics 174: 501-30. 
Wynne, M, Short, M, Semino, E 1998 A corpus-based investigation of speech, thought and writing presentation in English narrative texts. In Renouf, A (ed), Explorations in Corpus Linguistics. Amsterdam, Rodopi, pp 231-45. 
522

Seeing the wood and the trees: 
Reconciling findings from discourse and lexical analysis. 
John McKenny 
Northumbria University 
In this paper, I describe an exploration of a sub-corpus of learner English which views the data from widely different perspectives: at the macro or discourse level and the word or phrase level. The corpus under scrutiny is PORTICLE, the Portuguese sub-corpus of ICLE, the International Corpus of Learner English. This sub-corpus is a collection of argumentative essays written by Portuguese advanced learners of English and currently contains 126,000 running words. The data could be classed as quasi-naturalistic given that although the students did not know the purpose of the investigation, they had every right to suspect that anything they wrote might be used against them (Scholfield 1995:47). 
I am fortunate to be a part of the ICLE community. This worldwide ring of investigators, with its hub in Louvain, is responsible for a growing body of research on the various sub-corpora of ICLE which are being explored individually and compared to each other and to native speaker argumentative writing. As the investigations continue, relative newcomers like myself can draw on the findings that have been published (Granger 1998) and continue to be published about established and burgeoning sub-corpora (Granger S, Hung J, Petch-Tyson S (eds) 2003) . For example, those aspects of learner English which can be ascribed to mother tongue transfer are distinguished from those features which are shared by all ICLE sub-corpora. Interestingly, there are two Portuguese sub-corpora growing up, PORTICLE in Portugal and a sub-corpus being compiled in Brazil at the Catholic University of Sao Paolo. The co-ordinator of the Brazilian sub-corpus, baptized BRICLE, is Tony Berber Sardinha. Each national team of compilers can compare their sub-corpus with the others to see which features it shares and where and how it differs. 
For comparative purposes I analyse a comparable number words from another essay corpus, LOCNESS, which consists of essays written by British and American undergraduates. Incidentally, access to this corpus compiled by the Centre for English Corpus Linguistics at the Catholic University of Louvain is another perk for those belonging to the ICLE research community. The section I use is written exclusively by American undergraduate students. 
I am aware that we all tend to specialize in one or several areas of linguistics, pure or applied, and can become less interested in areas more removed from our present concerns. My Ph.D. research is on the phraseology of English for Academic Purposes. When I recently engaged in discussions with colleagues about their work in Conversational Analysis and Language Ecology I initially felt less enthusiasm than I should have done. I soon discovered that openness to other areas of linguistics (and indeed to other social and pure sciences) can pay dividends in terms of new ideas and ways of seeing things. This was brought home to me at a recent seminar on the ecology of language where a colleague, Mark Garner (2003), came up with his analysis of data collected to investigate what native speakers really talk about and do with language when they meet for the first time, as opposed to what language textbooks imagine they do . I was initially surprised by the convergence between his findings about spoken language and my own on reading into my PORTICLE essays. Here is Garner's list of functions so far: 
1) Establishing common ground 
2) Self-disclosing (reminiscing, expressing feelings, opinions, hopes and plans) 
3) Prompting other's self-disclosure 
4) Keeping the conversation going 
With the possible exception of 3) these functions of recently acquainted conversants are similar to those performed by writers of argumentative prose. I would suggest that 3) is never far away: the writer's inveigling the readers to admit that they know they agree with the writer . The similarity should not surprise us: the essay writer is addressing his reader as if for the first time (even Montaigne in his later essays would have to anticipate new, virgin readers). As we study the argumentation of the essays we must not lose sight of the written equivalent of number 4) above, keeping the conversation going or in writing, maintaining the readers’ interest. This seems to provide a more writer-oriented model than Hoey's (1983) more reader-oriented Situation-Problem-Solution-Evaluation model of text macrostructure and I hope to apply it to the PORTICLE sub-corpus in the months ahead. 
523
The number of essays in my learner corpus and the control corpus of native-speaker writers, approximately 240,000 words in all, means that sitting down and reading them and making notes on salient features was out of the question. Even to read a novel of comparable length, say Jane Austen's Emma retaining consistent linguistic and literary observations would prove a mammoth task. To get down into lexical and phraseological details while reading ‘manually’ would prove forbiddingly labyrinthine. Computer assistance seems to be called for. 
I decided to choose a small sample of NNS and NS essays to examine at the level of discourse. The purpose of this real reading rather than electronic scanning was to seek salient features of the NS and NNS essays and to get a feel for the writers’ approach to argumentation. This approach is really the key to all responsible use of learner corpora. Before setting off on bouts of computerized quantification, the researcher needs to continually re-discover the textuality and discursive flow of the data by reading it with the attention it deserves. This continual toing and froing from quantitative and qualitative analysis, with the two different approaches feeding naturally into each other, seems to be the most adequate methodology for interrogating learner corpora. This dual approach has been recommended within ICLE from the outset (Granger 1989:16). 
Obviously, there are some discourse features which lend themselves to computer-assisted corpus investigation. The use of discourse markers such as ‘first’ or ‘firstly', ‘second’ or ‘secondly’ and then ‘finally’ or ‘in conclusion’ can be measured and plotted through all the texts. A list of such text-structuring words and phrases can be built up iteratively and run through the corpus. Key argumentative words such as ‘but', ‘so', ‘therefore', ‘thus', ‘then', ‘since’ and ‘suppose’ can be concordanced and tracked through the texts. Important though such anatomical takes on the texts might be, there are many discourse features which slip completely through such a net. We are dealing with a huge deficiency in most quantitative approaches. The purpose of an argumentative essay is to persuade and until such days as a Leibnizian ‘universal character’ has been invented, when we can translate all the essays into strings of predicate calculus symbols and say ‘calculemus’ and decide exactly how cogent and persuasive each essay is, the only way such persuasiveness can be evaluated is through the engagement of an interested reader with each text. The use of the common topoi of logic, e.g. modus ponens and modus tollens which are a closed set of argumentative schemata, is another aspect of the essays which |I intend to examine in the next stage of my investigation drawing on the work of McElholm (2002). 
We can compare Wordlists and N-gram lists from the two corpora and find over- and under-used words and phrases. We can compare lexical density and suggest that the writing in one corpus tends to be more towards the spoken end of the spoken-written continuum. Let's look at the two compared wordlists of PORTICLE and LOCNESS and see what we can glean. What we find are tendencies about which we can make hypotheses which we can then go back and test on the essays. We must be wary of only testing those characteristics which lend themselves to quantitative investigation. Using a stoplist of 172 structure or grammatical words which I generated by extracting all words tagged with DT, CC, PRP, PRP$, TO, WDT, WP$, WRB, WP in the Brown corpus (method recommended by Diego Molla Aliod in personal communication) , I calculated the number of lexical words in each essay i.e. those words which were not filtered out from the wordlists and given as types. Developing a suggestion from Halliday (1985), the lexical density of the essays in each corpus was calculated as the ratio of the lexical words in relation to the total number of words in each essay., expressed as a percentage. This Hallidayan lexical density had the same value as the Type/Token ratio in Wordsmith Tools when a stoplist of grammatical words is used. Figures of 37.9% lexical items in LOCNESS compared with 29.5% for Portuguese students show us the not too surprising fact that the native speakers have a richer lexis at their command. But it should be remembered that ICLE scholars found that NS writers can display lower lexical density than NNS writers and that in NNS essays lexical richness does not correlate highly with quality of writing. Perhaps the number of different words used is less important than what is done with the words. 
When we use the Wordsmith Tools Wordlist function which compares two wordlists we are shown those words that are significantly overused or underused by the two target groups of students. 
Portuguese students use 'we' 1,791 times while the American NS subjects used it 273 times, i.e. well over six times more. When we add this startling fact to the 1,470 Portuguese uses of 'I' as opposed to 383 American uses of the first person singular pronoun, and look at 'our' (770 v206), 'us' (346v65), 'my' (428v129) we can see that there is a very different deployment of the pronoun system between the two groups. This overuse of ‘I’ is much greater than any other language group in ICLE (571 occurrences in SPICLE, the Spanish sub-corpus and 394 occurrences in the Brazilian sub-corpus ( my version kindly furnished by Tony Berber Sardinha was a smaller sub-corpus of 74,617 words). Although Fanny Meunier 
524
(Granger 1989: 32) warns against comparing corpora of different sizes this BRICLE figure for the occurrence of ‘I’ could be crudely converted to a percentage of the whole text (0.5%) compared to 1.4% for PORTICLE. This is startling as two groups with the same mother tongue display such a difference. 
If I return to the discourse level and to my role as a reader of student argumentative essays, I can point out that Portuguese students are not content to aver a proposition but seem to feel the need to put their whole being behind the asseveration. I am thinking primarily of Hunston's (2002) concept of the status of a proposition within a discourse. She describes how a writer gives a status to each clause in their essay and clauses are typically averred or attributed. 
Here are a few examples from 157 examples of constructions using I followed by believe 
I really believe that this is not a question of theory. 
I do believe that the fantasy world is really, 
I strongly believe that we must create a better world 
Being so I do not truly believe that this solution could be applied. 
From my point of view I believe that the years that we, nowadays, 
I personally believe that it´s important to help others 
I often had the impression that these Portuguese writers in English were writing letters of reference for the point of view being defended and were putting themselves forward as personal advocates. "You will believe what I'm telling you because I'm a good person and you, I hope, are a good person.". There are 90 examples of in my opinion in PORTICLE but LOCNESS has only four exemplifications. I'm sure looks as if it is being used comparably often with three and four uses in LOCNESS and PORTICLE respectively but when we add the complete form I am sure, the combined totals for the contracted and un-contracted forms rise to five and nine. Another striking contrast is the 117 uses of I think that used in PORTICLE compared with a mere six instantiations found in LOCNESS. We must be careful not to get carried away with the greater frequency of this prefabricated phrase used often for averral and occasionally for hedging. We find a hundred more occurrences of this prefab in PORTICLE but it might be the case that writers in LOCNESS are carrying out a comparable number of averrals using different strategies. It must be remembered that the simplest and often the most effective way of averring something is to state it in a declarative sentence. Such a bald statement of something usually carries with it the commitment of the writer to its truth or verisimilitude. (or acceptability). Aarts and Granger (1998:137) observed ‘striking differences in the way learners and native speakers begin their sentences’ through their study of sentence-initial trigrams. They found that they tended to begin their sentences with something other than the subject. I think the averral systems of the two groups of students (NS an NNS) is worth further study and the Hunston model would provide a rich frame of reference for examining how the writers assemble their sentences and the value they give to each proposition they add to their prose. The work of Petch-Tyson (1998) uses different concepts (writer/reader visibility) but comes to similar conclusions about EFL written discourse. The advanced learner of English is more concerned with interpersonal involvement in their writing at the expense information content. 
The differences between the two groups of arguers might be at a different level. The Portuguese writers might be thinking more of influencing their readers feelings and also maintaining a good relationship with them (including writing in an enjoyable way, sometimes they use humour, other times moral indignation or poetic imagery). The LOCNESS writers seem to pursue a more argumentative Aristotelian approach. Note the use of argument 194 times by LOCNESS writers and only four times by PORTICLE writers. If we lemmatize and include arguments and argue the difference in use of what could be viewed as key meta-terms is even more striking: 300 v19 uses. 
One methodology for text analysis which seemed to hold out hopes for bridging the gap between discourse and the lexical level was the software and formulae evolved by Youmans. I came to Youmans's work through Michael Stubbs’ (2002) analysis of the stages of James Joyce's story “Eveline” using the Vocabulary Management Profile. Although it had thus far been put only to uses within literary criticism, I thought that here might be a way to delineate more precisely the moves talked of within genre analysis (e.g. Swales 1990). I began to upload my student texts to Youmans's most accommodating website in Missouri and got back wonderful Vocabulary Management Profiles (VMPs). Below is the VMP for the first essay in PORTICLE. I include an edited version of the VMP statistics for the same essay in the appendix. As my main interest is in prefabs or formulaic sequences, I looked in each of the VMP statistics for each essay expecting to find more prefabs in the valley section of the essay profile but was surprised to 
525
notice that peaks or their surrounding text seem to contain more prefabs than did the environs of the valleys. 
Here is Youmans's explanation of why he chose VMP 2.2 and how he computes it: 
VMP2.2 computes ratios wrap-around style, for the second pass through a text. Hence, the first occurrence of a word such as "the" (near the beginning of a text) occurs shortly after its last occurrence (near the end of the text); hence, its ratio is nearer to 0.0 than to 1.0. The same is true for all other repeated words; their first occurrences are assigned ratios greater than 0.0 and less than 1.0. Words that appear only once in the text are assigned ratios = 1.0. Unlike VMP2.1, VMP2.2 shows no rapid downtrend at the beginning of a text. VMP2.2s mirror our second readings of texts, when the beginnings are as familiar to us as the ends. Because we normally associate rhetorical structure with second (and subsequent) readings rather than first readings, VMP2.2 is the default program selected for this web site. 
Youmans (2001:1) 
I began to study the valleys and the peaks. The valley, where the type/token ratio falls to a low, is where the writer is reiterating and consolidating and recycling already used lexis. Youmans suggests that this often corresponds to the end of a paragraph or section of a story or essay. According to Youmans, less-recently-used vocabulary at the ends of moving intervals tends to correlate with new topics, whereas more-recently-used vocabulary tends to correlate with a continuation of the same topic. Hence VMP 2.2s are surprisingly sensitive indicators of the ebb and flow of new topics in discourse'. I found there to be in most essays 3 or 4 main valleys followed by a similar number of peaks. All but the final valley was also followed by a peak. (As I carried on studying the VMP, I decided to use trough as my metaphor for the lowest points on the graphs as peak and trough seem to collocate more frequently in writing and talk about graphs). Interestingly a small group of writers (approx. 20%) did not end with a downturn but with a rise in their introduction of new or earlier words. These were the essayists who became inspired towards the end and began to do what most teachers proscribe: introduce new ideas within the conclusion. 
I noticed that Youmans's formula was no respecter of formulaic expectancies. The reader can look at words 100-102 in the appendix and see that in the case of and so…on the so is given the type-token ratio over the 35 word interval of 0.5240471, and the extremely predictable on gets the value 0.5466959: a rise is registered where a human assessor would suggest that on was completely predictable and deserved a drop or at least a plateau on the graph. In the Appendix, I only highlight some of the prefabs contained in the essay: those occurring near a peak or a trough. Although I am very impressed by the graphs ands statistics, I have not managed to find a generalizeable way to apply them to large collections of essays. I did find that all my essays had a peak roughly midway through at the 250-300 word point. Usually this was also the point in the essay with the highest type-token ratio and so was where the student writers marshalled their word-hoard to greatest effect. Sometimes in longer essays this climax comes later but still at the half-way stage. I haven't therefore found a good use for the VMP but will continue to look for potential applications of it to EAP. I am confident that such a good idea will not go to waste and if I fail to discover a use for it someone else will. This indeed was a challenge made by Stubbs (2002) to his readers. 
Here is the essay whose VMP graph appears below: 
<ICLE-PT-ESE-0001.1> 
I agree with this statement because I believe that nowadays it's given a lot of value to materialism. Our modern world, which is ruled by science, technology and industrialisation is creating caps between people. We don't have the human contact that we had many years ago, nowadays we depend more of technology. It's is true that this has given to us a lot of development, but and the relations between man? 
What about dreaming? What about imagination? I believe that these are the most important things that we have. I'm going to give an example. Nowadays children don't have free time, if they aren't at school they are having classes of music, painting, ballet and so on. Parents don't have time because they are very ocupied at work, so they put their children at extra-curricular classes. I believe that it's very important for children and for us to have free time just to do nothing. It's when we aren't being useful, this is, when we aren't doing nothing, that we dream, that we appeal to our imagination. Nowadays we don't have this, there is always something do. At extra-curricular classes children do apeal to their imagination, however it's an guided imagination. 
We live in a globalized world, everything is near us, everybody is next to us (due to the net), but, and what about the physical contact, the look eyes to eyes, the hugs? This is so important, even if we have the webcam, it isn't the same thing! 
Nowadays we don't dream, we are more concerned with world problems, which we created, than with imagination. Many times I think that my imagination is getting sofucated with all the information that I receive through t.v, net, radio etc. Our world is starting to be extremely industrialised, work... work... work..., that's all many of us think. 
526527
The only time that I appeal to my imagination is when I'm reading a book or when I look to a picture, when I stop a bit and I "turn of" from the reality. The rest of the time I'm to concern in my work to stop and dream. I don't like the way world is, I would have loved to live in those times when family got together and ask to the story teller to tell a story. This is an image that I have constantly in my head, nowadays we don't give value to all the old god values. We are very materialistics, we don't give value to simple things like look into the sky and try to guess what the clouds are telling us. 
Technology is everywhere, science is discovering everything that can be discovered, but should all the secrets be discovered? Shouldn't we leave anything in secret? I'm afraid that with this we destroy our world more than already is. 
I believe that sometimes we must stop and think a little in what we want, what we whish, what we are doing to nature. We must continue dreaming, we must take care of our world, because it is it the world that gives everything to us! 
When I examine this essay as a piece of argumentative writing I find it quite persuasive. Prefabs (marked in blue) seem to make up roughly 10% of the text. I have deliberately hedged the previous estimate as one person's prefab is another person's freshly coined phrase. The writer's phraseology is heavily influenced by Portuguese (given a lot of value; depend more of technology; but and…; think a little in what we want) and there are numerous “comma splices”, one of the deadly sins for many EAP writers. Many of these splices would be acceptable punctuation in Portuguese. 
The writer builds up a picture of how all the benefits that accrue to us from science and technology might come to nought if we lose touch with our own imagination and that of our culture. It reminds me of the gospel passage where Jesus asks the rhetorical question: what shall it profit a man if he gains the whole world and loses his soul? (Matthew 16). There are poetic images of families sitting down together to listen to a specially commissioned storyteller (as still happens in Donegal and Marrakesh) or of people looking at the clouds to see what they are telling us. This is not, by the way, a translated Portuguese idiom. 
The techniques of discourse analysis consider the whole text and with the insights we glean from this kind of holistic approach we can delve down again atomistically looking for confirmation or disconfirmation of our appraisal. An important lesson that we can learn from engaging with an essay as discourse is the inter-subject variability in any corpus. There are so many ways to persuade, parody, reasoning, examples, quoting authority, irony, parable-telling, satire or creating affective bonds with the reader to name just a few. The essay I have used as a discourse and processed with Youmans's software would probably be considered fairly weak from the point of view of spelling, punctuation, collocation, the use of prepositions and phraseology. As an argumentative essay and as a piece of persuasive writing it is fairly successful (I would claim). But I would welcome some corroboration from other readers. 
References 
Aarts J, Granger S 1998 Tag sequences in learner corpora: a key to interlanguage grammar and discourse. In Granger S (ed) Learner English in corpora. London, Addison Wesley, Longman. 
Garner M (2003) “Developing culturally appropriate language teaching”. Paper delivered at Viseu Polytechnic, February 14, 2003 reporting ongoing research funded by La Trobe Univeresity, Australia. 
Granger S (ed) 1998 Learner English in corpora. London, Addison Wesley, Longman. 
Granger S, Hung J, Petch-Tyson S (eds) 2003 Computer learner corpora, second language acquisition and foreign teaching. Amsterdam, John Benjamins. 
Halliday M 1985 Spoken and written language. Oxford, Oxford University Press. 
Hoey M 1983 On the surface of discourse. London, Allen & Unwin 
Hunston S and Thompson G 2002 Evaluation in Text Oxford, Oxford University Press. 
McElholm D 2002 Text and Argumentation in English for Science and Technology Unpublished Ph.D. Thesis, University of Hanover. 
Meunier F 1998 Computer tools for the analysis of learner corpora. In Granger S (ed) Learner English in corpora. London, Addison Wesley, Longman. 
Petch-Tyson S 1989 Writer/reader visibility in EFL written discourse. In Granger S (ed) Learner English in corpora. London, Addison Wesley, Longman. 
Scholfield P 1995 Quantifying language. Clevedon, Multilingual Matters. 
Stubbs M 2002 Words and Phrases: Corpus Studies of Lexical Semantics. Oxford,Blackwell. 
Swales J (1990) Genre Analysis. Cambridge, Cambridge University Press. 
Youmans G 1991 "A New Tool for Discourse Analysis: the Vocabulary-Management Profile." 
Language 67.4:763-789 
Youmans G 2001 "The Hierarchical Structure of Discourse: A New, Improved Vocabulary-Management Profile." Manuscript: 
528
http://www.missouri.edu/~youmansc/vmp/help/vmp22.html 

APPENDIX 
VMP2.2: "PTES1001.txt" Interval: 35 TotalTypes=216 TotalTokens=522 Types/Tokens=0.4138 AvgR = the ratio of Types / Tokens over the moving interval. Mean avgR = 0.41362 Standard Deviation = 0.07575 ~Midpoint, AvgR, Last word in interval 1, 0.3685221, value , 18 2, 0.3693995, to , 19 3, 0.3693995, materialism , 20 4, 0.3425829, our , 21 5, 0.3623252, modern , 22 , PEAK 1 11, 0.4020291, science 12, 0.4047162, technology 13, 0.4081163, and 14, 0.4346038, industrialisation, 15, 0.4063614, is , 32 16, 0.4308747, creating , 33 17, 0.4582945, caps , 34 18, 0.4802852, between , 35 19, 0.4802852, people , 36 20, 0.4546202, we , 37 21, 0.4597751, don't , 38 22, 0.4401974, have , 39 23, 0.4394297, the , 40 24, 0.4647655, human , 41 25, 0.4647655, contact , 42 26, 0.4655333, that , 43 27, 0.4655882, we , 44 28, 0.4912531, had , 45 29, 0.5047436, many , 46 30, 0.5260762, years , 47 31, 0.5364409, ago , 48 32, 0.5124212, nowadays , 49 33, 0.5099534, we , 50 34, 0.5126405, depend , 51 35, 0.5165341, more , 52 36, 0.5119276, of , 53 37, 0.5122018, technology , 54 38, 0.4858788, it's , 55 39, 0.4853852, is , 56 40, 0.4853852, true , 57 41, 0.4846723, that , 58 42, 0.4721689, this , 59 43, 0.4989855, has , 60 44, 0.4729367, given , 61 45, 0.4466685, to , 62 46, 0.4442007, us , 63 47, 0.4407458, a , 64 48, 0.4397039, lot , 65 49, 0.4117905, of , 66 50, 0.4400329, development , 67 51, 0.4190842, but , 68 52, 0.3925967, and , 69 53, 0.3676446, the , 70 54, 0.3676446, relations , 71 55, 0.3667124, between , 72 56, 0.3875514, man , 73 57, 0.3840417, what 58, 0.4017549, about 59, 0.3782835, dreaming TROUGH 1 60, 0.3498218, what , 77 61, 0.3482314, about , 78 62, 0.3633123, imagination , 79 81, 0.3523992, nowadays , 98 82, 0.3730738, children , 99 83, 0.3737867, don't , 100 84, 0.3736221, have , 101 85, 0.3708802, free , 102 86, 0.3783384, time , 103 87, 0.3971483, if , 104 88, 0.4226488, they , 105 89, 0.4191939, aren't , 106 90, 0.4411297, at , 107 91, 0.4411297, school , 108 92, 0.4358103, they , 109 93, 0.4170003, are , 110 94, 0.4404716, having , 111 95, 0.4644365, classes , 112 96, 0.4668495, of , 113 97, 0.4800110, music , 114 98, 0.5047436, painting , 115 99, 0.5294763, ballet , 116 100, 0.5307924, and , 117 101, 0.5240471, so , 118 102, 0.5466959, on , 119 PEAK 2 103, 0.5744996, parents , 120 104, 0.5470250, don't , 121 105, 0.5280504, have , 122 106, 0.5188922, time , 123 107, 0.5248697, because , 124 108, 0.5235536, they , 125 109, 0.5215794, are , 126 110, 0.5261859, very , 127 111, 0.5261859, ocupied , 128 112, 0.5256375, at , 129 113, 0.5310667, work , 130 114, 0.5193858, so , 131 115, 0.4911434, they , 132 116, 0.5170825, put , 133 117, 0.5187826, their , 134 118, 0.5173568, children , 135 119, 0.5171922, at , 136 120, 0.5168632, extra-curricular, 137 121, 0.5031533, classes , 138 122, 0.4854401, i , 139 123, 0.4615300, believe , 140 124, 0.4392103, that , 141 125, 0.4200164, it's , 142 126, 0.3922676, very , 143 127, 0.3951741, important , 144 128, 0.4222100, for , 145 129, 0.3941870, children , 146 130, 0.3717028, and , 147 131, 0.3692898, for , 148 132, 0.3453798, us , 149 133, 0.3198245, to , 150 134, 0.2927886, have , 151 135, 0.2928983, free , 152 136, 0.2726625, time , 153 
529
137, 0.2726625, just , 154 TROUGH 2 138, 0.2443104, to , 155 139, 0.2695915, do , 156 140, 0.2970661, any , 157 141, 0.3193858, thing , 158 142, 0.3139567, it's , 159 143, 0.3298602, when , 160 144, 0.3328763, we , 161 145, 0.3231149, aren't , 162 146, 0.3231149, being , 163 147, 0.3505347, useful , 164 148, 0.3400603, this , 165 149, 0.3453798, is , 166 150, 0.3453798, when , 167 151, 0.3171374, we , 168 152, 0.2924595, aren't , 169 153, 0.3011242, doing , 170 154, 0.3293666, nothing , 171 155, 0.3055114, that , 172 156, 0.3043597, we , 173 157, 0.3196051, dream , 174 158, 0.3165341, that , 175 159, 0.3138470, we , 176 160, 0.3300795, appeal , 177 161, 0.3304634, to , 178 162, 0.3360022, our , 179 163, 0.3097889, i , 180 164, 0.3378119, agination , 181 263, 0.4050452, sofucated , 280 264, 0.4023581, with , 281 265, 0.4211133, all , 282 266, 0.3942418, the , 283 267, 0.4088292, information , 284 268, 0.3807513, that , 285 269, 0.3812449, i , 286 270, 0.3812449, receive , 287 271, 0.4046614, through , 288 272, 0.4293392, t , 289 273, 0.4574170, v , 290 274, 0.4571429, net , 291 275, 0.4812174, radio , 292 276, 0.5096792, etc , 293 277, 0.5086921, our , 294 278, 0.4990403, world , 295 279, 0.4714012, is , 296 280, 0.4859336, starting , 297 281, 0.4865369, to , 298 282, 0.4779819, be , 299 PEAK 3 283, 0.4933918, extremely , 300 284, 0.5215794, industrialised , 301 285, 0.5023855, work , 302 286, 0.4851111, work , 303 287, 0.4847820, work , 304 288, 0.5098437, that's , 305 289, 0.4988210, all , 306 290, 0.4942693, many , 307 291, 0.4998629, of , 308 292, 0.4876885, us , 309 293, 0.4841788, think , 310 294, 0.4638333, the , 311 295, 0.4920757, only , 312 296, 0.4922402, time , 313 297, 0.4652043, that , 314 298, 0.4381683, i , 315 299, 0.4451330, appeal , 316 300, 0.4268714, to , 317 301, 0.4274198, my , 318 302, 0.4010968, imagination , 319 303, 0.4018646, is , 320 304, 0.4095969, when , 321 305, 0.3935838, i'm , 322 306, 0.3935838, reading , 323 307, 0.3712640, a , 324 308, 0.3712640, book , 325 309, 0.3962161, or , 326 310, 0.3679188, when , 327 311, 0.3400055, i , 328 312, 0.3389635, look , 329 313, 0.3379216, to , 330 314, 0.3373183, a , 331 315, 0.3373183, picture , 332 316, 0.3341925, when , 333 317, 0.3144502, i , 334 318, 0.3063340, stop , 335 319, 0.2779819, a , 336 TROUGH 3 320, 0.2971758, bi t , 337 321, 0.3032629, and , 338 322, 0.3034823, i , 339 323, 0.3034823, turn , 340 324, 0.3039759, of , 341 325, 0.3306279, from , 342 326, 0.3216891, the , 343 327, 0.3454346, reality , 344 328, 0.3435701, the , 345 329, 0.3706608, rest , 346 330, 0.3423636, of , 347 331, 0.3402797, the , 348 332, 0.3406636, time , 349 333, 0.3406087, i'm , 350 334, 0.3341376, too , 351 335, 0.3617220, concern , 352 336, 0.3673704, in , 353 337, 0.3670414, my , 354 338, 0.3685221, work , 355 339, 0.3603510, to , 356 340, 0.3489443, stop , 357 341, 0.3214149, and , 358 342, 0.3207020, dream , 359 343, 0.2932273, i , 360 TROUGH 4 344, 0.2703592, don't , 361 345, 0.2952564, like , 362 346, 0.2953660, the , 363 347, 0.3187277, way , 364 348, 0.3218536, world , 365 349, 0.3239923, is , 366 350, 0.2957499, i , 367 351, 0.3240471, would , 368 352, 0.3304634, have , 369 353, 0.3385797, loved , 370 354, 0.3391280, to , 371 355, 0.3195503, live , 372 356, 0.3145051, in , 373 357, 0.3428572, those , 374 358, 0.3428572, times , 375 359, 0.3434055, when , 376 360, 0.3434055, family , 377 
530
361, 0.3702769, got , 378 362, 0.3702769, together , 379 363, 0.3713737, and , 380 364, 0.3713737, ask , 381 365, 0.3716479, to , 382 366, 0.3725802, the , 383 367, 0.3992322, story , 384 368, 0.4263230, teller , 385 369, 0.4253907, to , 386 370, 0.4253907, tell , 387 371, 0.4202906, a , 388 372, 0.4469427, tory , 389 373, 0.4524266, this , 390 374, 0.4535235, is , 391 375, 0.4626817, an , 392 376, 0.4902111, image , 393 377, 0.4890047, that , 394 378, 0.4893885, i , 395 379, 0.4851111, have , 396 380, 0.4885111, constantly , 397 381, 0.4890595, in , 398 382, 0.4629010, my , 399 383, 0.4876885, head , 400 384, 0.4932273, nowadays , 401 385, 0.5003016, we , 402 386, 0.4739786, don't , 403 387, 0.4841788, give , 404 388, 0.4767754, value , 405 389, 0.4770496, to , 406 390, 0.4735399, all , 407 391, 0.4738141, the , 408 392, 0.4738141, old , 409 393, 0.4738141, god , 410 394, 0.5000823, values , 411 395, 0.4720044, we , 412 396, 0.4447491, a , 413 397, 0.4399781, e , 414 398, 0.4536880, very , 415 399, 0.4536880, materialistics , 416 400, 0.4533589, we , 417 401, 0.4530847, don't , 418 402, 0.4252810, give , 419 403, 0.3974774, value , 420 404, 0.3980806, to , 421 405, 0.3980806, simple , 422 406, 0.4136002, things , 423 407, 0.3883740, like , 424 408, 0.3853578, look , 425 409, 0.4126131, into , 426 410, 0.4032904, the , 427 411, 0.4032904, sky , 428 412, 0.4015903, and , 429 413, 0.4286811, try , 430 414, 0.4277488, to , 431 415, 0.4277488, guess , 432 416, 0.4376748, what , 433 417, 0.4355909, the , 434 418, 0.4355909, clouds , 435 419, 0.4561557, ar , 436 420, 0.4773238, telling , 437 421, 0.4820949, us , 438 422, 0.4862627, technology , 439 423, 0.4677269, is , 440 424, 0.4952564, everywhere , 441 425, 0.5124212, science , 442 426, 0.5112147, is , 443 427, 0.5112147, discovering , 444 428, 0.4953661, everything , 445 429, 0.4695914, that , 446 430, 0.4976693, can , 447 431, 0.5044695, be , 448 432, 0.5092404, discovered , 449 433, 0.5066630, but , 450 434, 0.5066630, should , 451 435, 0.5088566, all , 452 436, 0.5090759, the , 453 437, 0.5368796, secrets , 454 438, 0.5364409, be , 455 439, 0.5642446, disc , 456 440, 0.5642446, vered , 457 441, 0.5744996, shouldn't , 458 442, 0.5734028, we , 459 443, 0.5967644, leave , 460 444, 0.5967644, anything , 461 445, 0.5992323, in , 462 446, 0.5992323, secret , 463 PEAK 4 447, 0.6027968, i'm , 464 448, 0.6027968, afraid , 465 449, 0.6033452, that , 466 450, 0.5849191, with , 467 451, 0.5778996, this , 468 452, 0.5780641, we , 469 453, 0.5780641, destroy , 470 454, 0.5591445, our , 471 455, 0.5363861, world , 472 456, 0.5409926, more , 473 457, 0.5311763, than , 474 458, 0.5571154, already , 475 459, 0.5302989, is , 476 460, 0.5120921, i , 477 461, 0.5304634, believe , 478 462, 0.5025501, that , 479 463, 0.5183987, sometimes , 480 464, 0.5162051, we , 481 465, 0.5148889, must , 482 466, 0.5136276, stop , 483 467, 0.4880176, and , 484 468, 0.4852756, think , 485 469, 0.4606526, a , 486 470, 0.4868111, little , 487 471, 0.4871950, in , 488 472, 0.4616397, what , 489 473, 0.4617494, we , 490 474, 0.4617494, want , 491 475, 0.4332876, what , 492 476, 0.4048259, we , 493 477, 0.4311489, whish , 494 478, 0.4026871, what , 495 
531532

Interactive Word Alignment for Corpus Linguistics 
Magnus Merkel, Michael Petterstedt & Lars Ahrenberg 
Department of Computer and Information Science 
Linköpings universitet, S-581 83 Linköping, SWEDEN 
Tel. +46 13 28 19 64. Fax: +46 13 28 44 99 
{mme,g-micpe,lah}@ida.liu.se 

Abstract 
In this paper an incremental method and an interactive tool to improve the performance of word alignment are presented. The most important factor in our proposal is to put a human in the alignment loop. This is achieved by using interactive word alignment, i.e., a word alignment system and a human that are collaborating in order to make word alignment as efficient and accurate as possible. The aim is full coverage alignments with high accuracy, as the quality of word alignments is crucial in many applications within corpus linguistics, for example, in lexicography, contrastive linguistics and translation studies. 1 Introduction 
Automatic word alignment systems are used in various language and NLP tasks, such as bilingual lexicon extraction for lexicography, bilingual terminology and machine translation. Although word alignment has improved, (precision figures are often reported ranging from 80 to 95 per cent), recall is still too low for some more advanced applications. Furthermore, relevant word correspondences may not be discovered for rare words since most automatic systems rely on co-occurrence measures. 
As in most areas, alignment errors can have discouraging effects for applications in corpus-based translation studies or any form of corpus-based linguistics. For corpus-based translation studies it is important to be able to identify lexical additions and deletions in the target text, but current automatic word alignment systems lack the capacity to distinguish such operations from cases when the systems simply fail to identify likely source and target correspondences. In other words, a deletion in translation cannot be distinguished from the situation where the system says “sorry, couldn't find a probable alignment”. If the alignment of a bitext is complete, i.e. all instances of the phenomena of interest have been found and classified; we would have better justification for generalisations and observations concerning bilingual data. At present there is no automatic word alignment system near achieving complete recall, but by adding a human annotator this would indeed be possible, if appropriate tools were available. 
Concordancing tools have been around a long time, primarily for monolingual text, but in later years also for bilingual corpora. One problem with the concordancing tools have been that although they assist the corpus linguist in locating sentences or paragraphs containing the lexical items of interest, there is still the task of locating the exact lexical equivalence pair in the source and target segment. If a lexicographer wants to find out how a source lexical item is rendered in the target text, the concordancing tool will guide her to the sentence/paragraph pairs, highlight the source item, but then she has to pinpoint the target correspondence manually. Given bitexts with correspondences annotated below the sentence level, i.e., correspondences for clauses, phrases and words, a whole new range of possibilities for the corpus linguist would be opened up. Recent corpus applications for monolingual corpora, such as Word Sketch (Kilgarriff & Tugwell 2001) and FrameNet (Fillmore et al. 2002), have shown what can be done given richer linguistic annotation coupled with search and analysis tools. However, to our knowledge this kind of tools has so far only been created for monolingual purposes. One aim of our research is to act as a starting point for creating better bilingual tools for corpus linguistic applications. 
In this paper we propose an incremental method to improve the performance of word alignment. The most important factor in our proposal is to put a human in the alignment loop. This is achieved by using interactive word alignment, i.e., a word alignment system and a human that are collaborating in order to make word alignment as efficient and accurate as possible. In Ahrenberg et al. (2003), the advantages of interactive word alignment for machine translation systems are discussed in more detail. 
533
An interactive approach to word alignment requires an efficient interface in order to manipulate correspondences between bilingual segments. Such an environment will provide language engineers and corpus linguists with a tool that can help them to quickly produce reference data (gold standards) that can be used to evaluate the performance of their applications. Furthermore, valid reference data will improve the evaluation process of applications for fully automatic word alignment, bilingual lexicon extraction, detection of omissions in translations, etc. 
The major innovation of the tool in focus, I*Link, is that there is real interaction between the alignment system and the user. I*Link will propose correspondence candidates, based on information from bilingual resources and built-in heuristics. The user can accept, revise or reject these proposals on the fly. Furthermore, I*Link stores the strategies inherent in the user's choices and adapts its way of suggesting as more and more word alignments are made. This has the effect that the accuracy of the proposed word links is continuously improved during and across word alignment sessions, which in turn means increased efficiency. One important observation regarding interactive word alignment is to acknowledge that there could be several objectives for word alignment. A lexicographer would need correspondence data of a different form than a language engineer involved in developing or tuning a data driven MT system. 
In the paper previous approaches to automatic and manual word alignment are described along with some research problems. The system, I*Link, is then described in more detail, including the actual alignment process, the resources used as well as the built-in search and inspection tools. This is followed by a section describing applications within corpus linguistics for parallel corpora and word alignment. The paper ends with a discussion of how an interactive word alignment system could be combined with an automatic system and how such a combined system could be applied within corpus linguistics. 2 
Accurate full-coverage word alignment 
Let us say that a bitext where all sentence pairs have been assigned an alignment has full coverage. Accurate full coverage of a bitext is clearly beyond the capabilities of current automatic word alignment systems. While the methods used for generating probabilistic lexicons and translation probabilities at the word level are very useful, we could clearly generate data with less noise from an accurate full-coverage bitext. 
Accuracy in this context must be understood as relative to some set of assumptions and guidelines. Often, as in the two corresponding headings below, there may be differences in opinion as to what an accurate alignment is. 
ENGLISH: They watched the moths in the tobacco flowers. 
SWEDISH: De följde med blicken fjärilarna nere bland tobaksblommorna. 
(Lit. They followed with the_gaze the_moths down among the_tobacco_flowers.) 
For example, one strategy could be to use a construction-oriented approach and align the English article the as in the noun phrase alignments of “the moths – fjärilarna” and “the tobacco flowers – tobaksblommorna”. Another strategy could instead leave out the definite articles in the previous example and just couple “moths – fjärilarna” and “tobacco flowers – tobaksblommorna”, if the goal is to compile lexicon-like entries. It could also be discussed whether the verb phrase “följde med blicken” should be regarded as an alignment for the English “watched” and what to do with the unusual translation of “in” to the Swedish“nere bland”. 
Even in such simple cases like an English complex verb form is working and the corresponding Swedish arbetar, where the facts are hardly disputable, one has (at least) two different options: the English copula could be regarded as being deleted or as part of a periphrastic form that is aligned as a unit with the single-form Swedish verb. 
If both the source text and the target text of the bitext have been parsed and assigned a linguistic annotation of some sort, the data that can be extracted become even richer (though more complex). In addition, as we show below, the enriched linguistic analysis will make it possible to improve word alignment. In our on-going project, working with English-Swedish parallel texts we use Connexor's FDG 
534
parsers (Functional Dependency Grammar) for parsing (Tapanainen & Järvinen, 1997)1. These parsers provide data of the following kinds for each word token: 
• Base form 
• Part-of-speech and morphological features 
• Syntactic function 
• Dependency relation 
• Head of dependency relation 
The FDG parsers have been developed primarily with monolingual parsing in mind. Thus, their tag sets are not in perfect harmony, but the extra linguistic knowledge provided furthers the actual alignment stage as well as serves as a rich resource when further analysis of the bitext is being performed. 
Returning to the previously mentioned objective of full-coverage accurate word alignment there are clear advantages for applications where the word-level correspondences are applied, as in translation studies, contrastive linguistics and lexicography: 
• Derived lexical data have a higher quality, and are therefore more reliable 
• With a parsed bitext more general and abstract data can be derived at later stages of analysis. 
In the following we present the system and its method to improve the performance of word alignment. 3 3.1 I*Link – an interactive word 
alignment system 
It has been pointed out before that alignment bears resemblance to translation and, as with translation, systems could improve by learning from human decisions. Martin Kay's argument for the role of humans in translation holds for alignment too; i.e., we should “expect better performance of a system that allows human intervention as opposed to one that will brook no interference until all the damage has been done” (Kay 1997, p 22). 
In order to review, modify and create alignments with human assistance we need an efficient interface. Anybody who has tried to manually create tables of word correspondences in a word processor or in a spreadsheet, or to draw lines between word tokens in bitext printouts would agree, we think. The main features of our system are that it proposes alignments to the user on the basis of its combined linguistic resources and that it is able to improve on its performance by learning from the user sentence-by-sentence. Furthermore, any errors made can be easily corrected, both on the fly and in post-alignment sessions. 
Previous work 
Most word alignment systems to-date have been automatic, exploring the co-occurrences of terms in large parallel corpora to generate translational equivalences among word types. In addition to co-occurrence data, some systems utilise linguistic knowledge on different levels of sophistication (Melamed 2001, Ahrenberg et al., 2000b, Gaussier et al. 2000, Tufiº 2002). However, the idea of improving the outcome of an automatic system, though quite common with sentence aligners and the creation of tree-banks (Marcus et al., 1993), seems not to have been applied systematically to word alignment. Isahara and Haruno (2000) present a post-editing tool for sentence alignment that has been extended with functions for alignment of phrases and proper nouns. In the Cairo system (Smith and Jahr, 2000) a user can examine visualizations of the word alignments produced by a word aligner, but is not allowed to make changes to them. 
1 These parsers are now marketed as Machinese Syntax. See http://www.connexor.com/ for further information. 
535
In Ahrenberg et al (2002) an earlier version of the interactive linker was presented. This version had a more primitive interface and also lacked several of the resources and learning capabilities included in the current version. 
The current version of I*Link supports the following tasks: 
• Manual word alignment, 
• Automatic proposals of token alignments, 
• Reviewing and editing alignment proposals from the system in an orderly fashion, 
• Configuring the resources to be used by the system in a work session, 
• Compiling reports and statistics from aligned files. 
The system has a graphical interface that allows direct manipulation and interaction with static and dynamic resources. 3.2 
Graphical modules 
The graphical interface is divided into four windows: the Link Panel, the Link Table Panel, the Resource Panel and the Settings Panel. In the Link Panel, where the alignments of the current sentence pair are presented, the user can manually select correspondences, or interact with the automatic proposals from the system which can be accepted or rejected according to the user's preferences. 
All alignments that are confirmed by the annotator will be marked in corresponding colours in the Link Panel. Furthermore, the alignments are also visualized in a table representation in the Link Table Panel. The Link Panel is shown in Figure 1 and the Link Table in Figure 2. 
Figure 1. The Link Panel from I*Link. The buttons in the lower right corner allows the user to accept or reject alignments proposed from the system. Alignments are shown in corresponding colours. Additions and deletions are visualized through strike-through lines in the target and source texts. 
Apart from annotating straight-forward correspondences, it is possible to represent deletions and additions. The linguistic information for lexical items in focus is visualised in the interface, both in pop-up boxes and directly in the lower part of the Link Panel. 
536
Figure 2. The Link Table showing the same alignments as the Link Panel in a table format. 3.3 
Input and Resources 
The input to I*Link consists of parallel source and target files which have been aligned on the sentence level in advance. Input files may be line numbered text files or annotated files in XML format. The annotation records linguistic information on four levels: word form, base form, part-of-speech with morphosyntactic features and dependency relations, such as subject, object, and attribute, etc. 
The Resource Panel displays the configuration of active resources for an alignment project. There are basically three types of resources available in the current version of I*Link, namely, static resources, dynamic resources and patterns. All types of resources could in principle be used on the four different levels of abstraction supported by the system. Static resources are set up at the start of the alignment project and do not change during the session. Typical examples of static data are bilingual term lists and core lexicons. Recurring POS correspondences can also be used. 
Dynamic resources change during the session. When the annotator accepts or rejects a proposal from the system, or defines an alignment manually, the action is recorded in the dynamic resources (for example, both as a word form and a base word correspondence, as well as data on the parts-of-speech and syntactic function correspondences). The dynamic resources will therefore contain both positive (accepted and manually defined) data and negative data (rejected proposals). The third type of resource used in I*Link are pattern resources, which define correspondences for tokens such as cognates, numbers and punctuation characters. All resources have the capacity to store positive and negative resources. An example is shown in Figure 3. 
… formerly#tidig#1#0 for-example#exempelvis#1#0 for-example#till-exempel#6#0 from#fran#14#0 from#i#3#1 from#pa#1#0 from#via#1#0 from#<NULL_LINK>#2#0 future#framtid#1#0 gain#<NULL_LINK>#1#0 generate#läsa av#0#1 … 
Figure 3. Example of a dynamic base form resource. The first field (before "#") holds the source item; the second field contains the target item, and the third number of observed positive instances (acceptances). Finally the number of negative observations (rejection of system proposals) is given. In the given sample the pair “from –i” has been accepted three times and rejected once whereas “generate – läsa av” has no positive observations, only one rejection. 
The Resource Panel used for controlling the resources in an alignment project is shown in Figure 4 below. 
537
Figure 4. The Resource Panel. On the active word form level six resources are active; one dynamic resource, two static resources and three pattern resources. The figures on the right hand side display the number of current matches in the active bitext segment as well as the total number of acceptances and rejections made by the annotator. 3.4 3.5 3.6 3.7 Performance Heuristics and 
settings 
I*Link supports different strategies for how to select and present alignment proposals to the annotator. For example, the annotator can decide that alignments should be presented from left to right based on the source sentence, or that proposals should be given in the order that reflects the over-all ranking, made by I*Link. 
Interactive alignment and learning 
The learning approach taken in I*Link is based on the fact that the dynamic resources are updated incrementally during the manual revision stage. Each time the user confirms a proposed link the information inherent in the link is stored in the different dynamic resources. The inflected word forms will be added to the word form resources and the base forms to their dynamic resources. Also, new information on correspondences for parts-of-speech and dependency relations will be put in the dynamic resources. This also applies when the user adds new alignments manually by selecting items in the Link Panel, making such alignments to be stored as positive data. However, if a user rejects a proposal this information is stored as negative data in the dynamic resources on all applicable levels. The updating of the dynamic resources is made incrementally which means that the new information is available immediately for I*Link and can be applied when new proposals are made in the next sentence pair. In our own tests, the improvements from the learning strategies are clearly observable even after a rather limited number of sentence revisions. 
Analysis and reports 
To be able to analyse the alignment data, I*Link contains some additional tools. One such tool is the Link Inspector, which functions like a fine-grained bilingual concordance program in that it is possible to define search criteria on all combinations of representation levels, word form, base form, POS and function. This means, for example, that one could identify all the links where a subject noun corresponds to an object noun, an adjectival construction corresponds to a verb construction, etc. 
There are also inspectors for viewing, searching and editing the static and dynamic resources and a Link Reporter that can summarize and configure the information in the database, including compiling fine-grained concordances according to the user's preferences. Examples of how these tools can be used are shown in section 4. 
In an experimental session we measured the speed and consistency of four subjects. A small set of guidelines was used. All subjects were familiar with the system, but only two of them, A and C, with the guidelines. The guidelines were explained and discussed in a prior session lasting for twenty minutes. 
538
Each user aligned 97 sentence pairs from the help files of Microsoft Access 2000 for XP. The results are shown in Tables 1 and 2. 
Table 1 shows results for individuals and speed. The difference between subject B and the other subjects can on the whole be attributed to the use of a different computer, where there is no delay in going from one sentence pair to the next. On the other machines this delay varies between 1-10 seconds depending on the length of the sentences in the pair. 
Table 1. Working speed with I*Link for four different users familiar with the system. 
Id No. Sent. 
Pairs Total No. 
Links Time (min.) Links 
per min. Min. 
per sent. A 97 1564 121 12.9 1.25 B 97 1542 92 16.7 0.95 C 97 1551 111 14.0 1.14 D 97 1542 108 14.3 1.11 Mean 97 1550 108 14.4 1.11 
Table 2 shows agreement in word alignment between subjects. All subjects agreed on 83.4% of the links produced, and if null links were removed, where alignment strategies were the most varied, the percentage of agreement is as high as 86.5%. Subject D generally was less consistent with the others. If a training session is included so that subjects can discuss disagreements, and guidelines be more detailed, it is likely that these figures can be raised substantially. 
Table 2. Agreement in word alignment between subjects. 
Common Links (%) ABCD ABC BC AD Incl. Null links 83,4 88,7 91,6 85,4 Excl. null links 86,5 90,4 93,1 88,9 
The method that the system used for alignment proposals in this experiment was “one-by-one”, i.e., the subjects were given one suggestion at a time, and was asked to give a judgement of this as “accept”, “reject” or ask for a new proposal. An alternative way of presentation would be to ask the system to show all the alignment proposals for a sentence pair immediately and let the user modify the ones that are erroneous. When sentence pairs are short and the proposed alignments have high accuracy, this may speed up the interaction substantially. We have not tested the system with the latter method yet, but we plan to do so. 4 
Corpus Linguistics applications for interactive word alignment 
As mentioned in the introduction there are interesting applications given full coverage word alignment with high accuracy. For example, for commercial lexicographical work, Atkins (2002, p. 13) makes the following remark concerning bilingual concordancing software: ”A smarter program which would tailor the output of the concordancing program to our needs might persuade reference publishers to change their minds about the use of parallel corpora /…/ Lexicographers need some bilingual form of the Word Sketch tool to help them use parallel corpus data within the time constraints of commercial dictionary production.” Although the inspection and analysis tools included in I*Link are rather limited, they are still very powerful for producing overviews of lexical correspondences in parallel corpora. Consider the brief examples shown in Table 3 and 4 below. Here searches have been made for two kinds of functional shifts in an English-Swedish bitext, namely when source objects have been rendered as target subjects and vice versa. 
539
Table 3. Output from Link Reporter on object-to-subject shifts 
Source base Target base Source example Target example Count (S:2) mweta mweta ... one stage , for getting [Mweta] banished to the far Western ... ... sin tid styrt om att [Mweta] hade förvisats till en avkrok ... 1 you du "They expect [you] back , " she said ... "De räknar med att [du] ska komma tillbaka " , ... 1 
Table 4. . Output from Link Reporter on subject-to-object shifts 
Source Target base Source example Target example Count (S:10) decision beslut ... of practical matters by which [decision] is broken up into reality ... ... de praktiska problem som förvandlar [beslut] till verklighet . 1 he han ... of dill ; " There [he] is , " she said ... ... . " Där har vi [honom] " , sade hon . 1 it den ... code so deeply accepted that [it] had never been discussed .. ... ... att de aldrig hade diskuterat [den]: man stod till förfogande ... 1 she hon Because [she] was suddenly realizing that it ... Ty det slog [henne] plötsligt att sa hade det ... 1 that som ... bought , filled with possessions [that] had been stored all the ... ... köpt och fyllt med tillhörigheter [som] de haft magasinerade under alla ... 1 that som ... true sense of after all [that] had gone before ) an ... ... ( bokstavligen , trots allt [som] livet fört med sig ) ... 1 /…/ 
If the lexicographer wants to find examples of when single verbs have been translated with more complex verb phrases, it is possible to express this in the search window and end up with something similar to the examples in Table 5. Here single verbs in the English source text have Swedish correspondences that consist of more complex verb phrases in the form of verb – preposition – noun. 
Table 5. Output from the Link Reporter showing single verbs (POS=V) in the source text that correspond to complex verb phrases (V PREP N) in the target. 
Source base Target base Source POS Target POS Source example Target example Count (S:3) garden pata i träd-gard V V PREP N He and Olivia [gardened] on summer evenings , not ... Han och Olivia brukade [pata i trädgarden] om sommarkvällarna , inte pa ... 1 make göra i ordning V V PREP N She used to [make] packages of sandwiches for Mweta ... Hon brukade [göra i ordning] smörgaspaket at Mweta att ta ... 1 settle komma i ordning EN V PREP N ... - you 'll be more-or-less [settled] by the time she arrives ... ... fall har du mer-eller-mindre hunnit [komma i ordning] när hon kommer ner . ... 1 
Another application for the high-quality word alignment that we are aiming for is to try to derive wordnet relations such as synonymy and hyponymy from translation corpora using the concept of semantic mirrors suggested by Dyvik (2002). The idea is to use word correspondences from a parallel corpus to build synonymy and hyponymy sets via the word alignments created by a tool like I*Link. This approach requires data with high accuracy and can therefore be suited to the interactive word alignment approach suggested here. The resulting data sets covering synonymy, hyponymy and semantic nearness could also serve as resources in future versions of I*Link, as a kind of extended semi-semantic lexical resources which could increase the performance of word alignment. 
5405 6 7 
Integrating automatic and interactive alignment 
The current version of I*Link is a stand-alone word alignment tool that proposes token alignments and interacts with the user. We are currently adding a fully automatic mode to the system where I*Link aligns the bitext without interaction with the annotator. The output from the automatic alignment is then reviewed by the user (by accepting, rejecting, adding and modifying word links). The user will go through a subset of the automatically generated links (for example the first 50 sentence pairs) and then the automatic component will take over again and re-align everything that has not been verified by the user, with the aid of the new information stored in the dynamic resources. 
The automatic alignment component is still at a preliminary stage, but the goal is to make use of the resources available to I*Link, that is, information on word forms, base forms, parts-of-speech and syntactic functions. In addition to these, the automatic component utilises information on co-occurrence statistics for lexical correspondences along the lines of our previous automatic aligner, Linköping Word Aligner, LWA, (Ahrenberg et al. 2000). The main difference between LWA and the current automatic aligner lies in the possibility of exploiting more linguistic information and the approach that the token alignments are in focus, that is, all possible word alignments in one sentence pair is aligned at a time. In evaluations of LWA we found that it had problems with over-generalizations of certain alignments (mostly homographs, such as att which can function both as a subjunction and infintive marker in Swedish). Furthermore, multi-word units (MWUs) had a much higher error rate than single word units which was due to the built-in approach in LWA to generalize type alignments to token alignments. In the new approach, the dynamic and static resources are used as global knowledge bases for the whole alignment process, but contextual factors present in specific sentence pairs function jointly with the global resources in a way that steers the alignment to better performance. 
Future work 
As a tool for creating accurate full-coverage word alignment, the current version of I*Link has certain limitations that we intend to overcome in future versions. These extensions will include 
• handling of discontinuous multi-word units, 
• integration between the interactive mode of alignment and a fully automatic component, 
• support for optional user annotations of alignments, 
• database connections, 
• integration of various support tools for sentence alignment that adhere to the input XML formats, 
• improved search and analysis tools, 
• support for more syntactic annotations, i.e., other taggers. 
Concluding remarks 
In this paper we have argued that interactive systems is the best way to improve word alignment for corpus linguistics applications and have presented the I*Link system as a first step forward in this direction. 
Information about I*Link is continuously being updated at http://www.ida.liu.se/~nlplab/ILink/. Versions for academic use can be downloaded from this site. 
541
References 
Ahrenberg, L Andersson M, & M. Merkel, 2000. A knowledge-lite approach to word alignment.. In J. Véronis (ed.) 2000: 97-116. 
Ahrenberg, L., M. Merkel, & M. Andersson. 2002. A System for Incremental and Interactive Word Linking. In Third International Conference on Language Resources and Evaluation, Las Palmas, 485-490. 
Ahrenberg, L., M. Merkel & M. Petterstedt, 2003. Interactive word alignment for language engineering. To be published in the Proceedings of EACL-2003, Budapest. 
Atkins, Sue. 2002. Then and Now: Competence and Performance in 35 Years of Lexicography. In Proceedings of the Tenth EURALEX International Congress, Volume 1: 1-28, Copenhagen. 
Dyvik, H. 2002. Translations as semantic mirrors: from parallel corpora to wordnet. Paper presented at The 23rd International Conference on English Language Research on Computerized Corpora of Modern and Medieval English (ICAME 23), Gothenburg, 22-26 May, 2002. http://www.hf.uib.no/i/LiLi/SLF/ans/Dyvik/ICAMEpaper.pdf. 
Fillmore, C.J., C.F. Baker, and H. Sato. 2002: The FrameNet Database and Software Tools. In Proceedings of the Third International Conference on Language Resources and Evaluation (LREC). Las Palmas: 1157-1160. 
Gaussier, E, D. Hull, & S. Ait-Mokhtar. 2000. Term alignment in use - Machine-aided human translation. In J. Véronis (ed.) 2000: 253-276. 
Isahara, H. and M. Haruno. 2000. Japanese-English aligned bilingual corpora. In Véronis, J. (ed.) 2000, 313-334. 
Kay, M. 1997. The Proper Place of Man and Machine in Language Translation. In Machine Translation Volume 12, Nos. 1-2, 1997, 3-23 (reprint from 1980). 
Kilgarriff A, & D. Tugwell. 2001. Word Sketch: Extraction and Display of Significant Collocations for Lexicography. ACL Workshop on Collocation, Toulouse, July 7, 2001. 
Marcus, M., B. Santorini and M. A. Marcinkiewicz, 1993. Building a Large Annotated Corpus for English: The Penn Treebank. Computational Linguistics, 19(2): 313-330. 
Melamed, I. D., 2001. Empirical Methods for Exploiting Parallel Texts. Cambridge, MA, The MIT Press. 
Smith, N. A. and M. E. Jahr, 2000. Cairo: An Alignment Visualization Tool. Second Conference on Language Resources and Evaluation, Athens, 2000. Vol I: 549-551. 
Tapanainen, P. and T. Järvinen, 1997. A non-projective dependency parser. Proceedings 5th Conference on Applied Natural Language Processing (ANLP'97): 64-71. 
Tufiº, Dan, and A.-M. Barbu. 2002. Lexical token alignment: experiments, results and applications. In Third International Conference on Language Resources and Evaluation (LREC 2002), Las Palmas: 458-465. 
Véronis, J. (ed.). 2000. Parallel Text Processing: Alignment and Use of Translation Corpora. Dordrecht, Kluwer Academic Publishers. 
542

Relating Lexical Items to Sociolinguistic Features in a Spontaneous Speech Corpus of Spanish 
José María Guirao Miras 
Departamento de Lenguajes y Sistemas Informáticos, Escuela Técnica Superior de Ingeniería Informática, Universidad de Granada 
c/ Daniel Saucedo Aranda s/n, 18071 Granada, Spain 
jmguirao@ugr.es 
Ana González Ledesma 
Guillermo de la Madrid Heitzmann 
Manuel Alcántara Plá 
Antonio Moreno Sandoval 
Laboratorio de Lingüística Informática, Departamento de Lingüística, Universidad Autónoma de Madrid 
Carretera de Colmenar Viejo Km.15, Cantoblanco, 28049 Madrid, Spain 
ana.ledesma@adi.uam.es, guille@maria.lllf.uam.es, manuel@maria.lllf.uam.es, sandoval@maria.lllf.uam.es 
This paper shows the application of statistical tests to a spontaneous speech corpus of Spanish. Our goal is to find representative differences between different parts of the corpus. To this end, we tagged n-grams in the corpus with features related to the speaker (age, gender, etc), or the context (dialogue, monologue, media, etc), and applied the log-likelihood test (Dunning 1993) in order to find the most representative lexical items for each specific feature. 
The paper is divided in two sections. In the first part, the characteristics of the spoken corpus are shown together with the explanation of the computational tool. In the second part, a first rough estimate of the results obtained is given, as well as possible applications of the model. 
We work with the Spanish part of the multilingual corpus C-ORAL-ROM, which is made up of 300,000 words. These have been orthographically transcribed. The domain distribution is diversified in order to allow cross-linguistic comparison. The corpus is organized using three variables: situation (formal/informal), channel (media, telephone, natural context) and number of speakers (dialogue, monologue, conversation). Each text contains the transcription and a header where the information about the gender, age, education, occupation and geographical origin of the speakers is shown. 
We have developed a tool that permits the clustering of those words or multi-words that appear three or more times and sorts them according to their frequency of use. The corpus is XML-tagged, which allows to automatically relate these results to the sociolinguistic variables (age, gender, etc) and to the text typology (formal, informal, media...). The access to the data is possible through two ways. The first one provides the sorted list of words and clusters, and from each item one can reach the corresponding sociolinguistic and text-typologic information. The second one works the other way around: from a list of sociolinguistic and typological items one obtains the sorted list of words and clusters for that field (men, formal, media...). 
The second part of the paper is devoted to the analysis and explanation of these results. Here we show how the frequency of words and clusters varies noticeably according to the different parameters. For instance, the discourse marker "o sea" (that is) is much more frequent in informal contexts, and the frequency of use of the expression "no sé" (I don't know) in feminine speakers is double for male speakers. 
This paper ends by stating the importance of this procedure as an empirical method for the validation of sociolinguistic hypotheses in spoken language. 
543

ANESTTE: a writer's assistant for a specific purpose language 
Juan M. Montero 
Departamento de Ingeniería Electrónica 
ETSI Telecomunicación 
Universidad Politécnica de Madrid 
Ciudad Universitaria S/N 
28040 Madrid Spain 
juancho@die.upm.es M. Mar Duque 
Departamento de Filología Inglesa aplicada a la Ciencia y a la Tecnología 
ETSI Telecomunicación 
Universidad Politécnica de Madrid 
Ciudad Universitaria S/n 
28040 Madrid Spain 
marduque@etsit.upm.es 

Abstract 
This paper presents a new automatic tool for assessing the linguistic quality of scientific papers written in English. A set of complex lexical and syntactic surface-level rules compute more than 80 style-related variables. Their combination defines the score of a text in the four dimensions of style assessment for scientific papers: clarity, variety, conciseness and conviction. The software has been tested on 60 published articles and incorporates an animated agent that acts as a personal assistant and explains the results to the writer and how to improve them. 1 
Introduction 
After the emergence of English as a global language, it has become the predominant language in science and technology publications. Proficiency in written English is a necessary skill for current scientists all over the world, as they must publish their research results in international conferences and journals. Similar remarks can be applied to other areas of activity such as business or education: to succeed, people need to express their written ideas effectively and in English. As this is a difficult task for non-native English speakers, it has provoked a rapid increase in the interest on studying English for Specific Purposes. 
One of the best ways to improve the quality of texts would be the use of good automatic computer-assisted text analyzers and style checkers. These software tools could aid in the creation of accurate English-language documents. They could be especially useful when adapted to the rules and conventions of a specific genre, such as technical or scientific articles. The top performance would be achieved when these programs are trained to verify a simplified or a controlled language, making the translation easier. 
Nowadays spell checkers (software tools that analyze the lexical component of a text, and search for the use of misspelled words or mechanical errors), are usually part of any state-of-the-art word processing suite. These computer programs also incorporate some kind of grammar checker (number agreement errors, the use of unacceptable syntactic structures…) that outputs one or several simple readability measures such as Flesch Reading Ease index (Flesch 1948) or Flesch-Kincaid Grade Level. Among the complementary linguistic information that is taken into account by these programs, we can find: the length of noun clusters, the balance between the use of active and passive voice, the structure of the clauses and sentences (simple vs. coordinate or subordinate), etc. Nevertheless, thorough computer-based style analyzers are still a matter of NLP research and there is much room for improvement. 
Authoring and proofreading tools can also incorporate some standardization of the typographical style of the documents (fonts, margins…), or can include facilities for sharing documents or checking consistency in a collaborative writing environment (Glover 1996). Finally, they are usually augmented with additional help documentation on grammar basics or examples of use (although in most cases they are designed with native speakers of English as the target users). 
In this context of writer's assistants, we have developed a new software tool called ANESTTE (ANalyzEr of Style for Technical Texts in English). Our work has been focused on reviewing the text 

style for scientific and technical writers, especially when they are non-native and need to write research articles for international journals in English. We have not developed any tool for spelling or grammar checking, because our interest is in a higher-level concept of style as described below. 
By addressing this genre or this specific language, we aim at providing a deeper and more complex analysis. Although including rules and formulas developed for a specific purpose language, the tool is fully configurable and could be easily customize to any other register or genre. 
Another important feature in our system is that incorporates an HTML guide on style excellence and an animated agent. This agent explains the results of the analysis to the users by means of speech synthesis and speech recognition (Montero 2000), and explains how to achieve a better overall quality in their writing style. The critiquing agent comments which concrete linguistic variables must be increased or decreased to improve the style of the text, and it can display the HTML pages that explain the meaning of the variables in the native language of the writer (our system is mainly aimed at Spanish technical or scientific users). 2 
Style analysis 
Modern stylistic studies comprise two areas. One is called Literary Stylistics, and is related to the description of the use of lexical and grammar variations in literary works. The other area is Linguistic Stylistics, that describes the formal characteristics of a text in terms of linguistic variation, by means of qualitative and quantitative analysis. 
2.1 Linguistic Style Analysis 
The reference frame for Linguistic Style Analysis can be a text genre (for instance, the scientific or technical research article), a register or even a specific author. 
A statistical approach in Stylistics (e.g. to compute the mean length of words or the percentage of passive-voice sentences) can be very useful for the development of automatic applications such as forensic ones: authorship attribution that determines the identity of the author of a given text. It could be also helpful in writer's assistants, for the determination of the estimated reading level needed to comprehend a written text, or the adequacy of the text to the publishing context: journal, conference… (Sharples 1992). Another important area of application is the automatic detection of genre: to assign electronic documents to a taxonomy of pre-defined subject categories (Kessler 1997). 
For this style evaluation process, a certain norm must be defined. If we are studying the appropriateness of one particular paper, we need to define a set of measures to determine the distance between our sample paper and the abstract good-quality technical paper. Mathematically, this distance (based on measurable characteristics) is the norm of the style for this genre (probably part of this norm is general and, therefore, applicable to other genres or registers). 
Previous approaches made use of several natural language strategies that now we can combine: 
  If-then-else rules: they establish conditions on linguistic variables, such as the mean number of words per sentence or the maximum number of words per sentence (McGowan 1992). These rules should depend on the genre of the text, the audience of the text… In our case (scientific and technical papers), the context is homogeneous, and can be modeled with only one set of rules. 
  Parametric formulas: as in Critique (Jensen 1993), that uses non-Boolean acceptability limits and weighting factors (for instance, applied to the assessment of the distance between subject and verb). Thus, we can provide a more detailed analysis that allows relaxed or fuzzy conditions. 
  Lexical analysis: as included in most proofing tools. It is used to reject the use of informal words, clichés or contractions in a formal style, to detect long words that make a text more difficult to read, etc. 
2.2 Technical writing style features 
The language of scientific and technical papers as published in international journal and proceedings, constitutes a Language for a Specific Purpose. It must provide an efficient means of communication between research workers and must stimulate the development of future research activities (Lundberg 1994). 

Although the contents of the articles are the central point for achieving these objectives (guided by experimentation, innovation and the scientific method), the linguistic aspects of the paper also play an important role for providing efficiency and attractiveness to the text. 
Four sub areas of linguistic analysis can be distinguished (Duque 2000): 
  Clarity: the meaning must be transparently conveyed in an effortless manner. Among the enemies of clarity, we can list: 
• long noun or preposition clusters, 
• long sentences, 
• an excessive use of passive voice, 
• the abundance of abstract nouns or static verbs, 
• the lack of connecting elements, that bring cohesion to the discourse. 
  Conciseness: the use of the right words and just the right words. Against this virtue we can find: 
• an excess of nominalizations, 
• the use of redundant or unnecessary expressions. 
  Variety: it brings attractiveness to the communication process. Monotonous features that impoverish a text can be: 
• the repetition of the same sentence structures, 
• an excessively impersonal tone, 
• the lack of richness in terms of sentence and paragraph lengths, or in terms of types of verbs. 
  Conviction: it expresses the confidence of the author in the information that the text conveys. The abuse of certain modal verbs and expressions of doubt (or probability) can decrease the conviction of a paper, transmitting the feeling that the author lacks security in his/her assertions. 
In addition to this, semantic constraints should be imposed, constraints that would check for textual coherence or textual progression. As the state-of-the-art automatic semantic analysis is not reliable yet, our system will not model the meaning of the words or the logical flow, but the syntactical structure of the sentences. ControlFileAccept.FileCount andauxiliaryvariablesAccept.variablesFilePartialscoresGlobalscore
ANESTTE system 
Our system follows a knowledge-based approach, using a set of specific linguistic rules to count the occurrence of most of the features that are relevant for the analysis of style. Upon these variables, a set 

Figure 1. Process of analysis 3 

ANESTTEText
of acceptability formulas is applied. Thus, using a two-phased approach (Figure 1), the system arrives at one general style score of the text, and four particular scores in the sub-areas of analysis, as mentioned above. 
These linguistic context-sensitive rules perform a robust parsing and disambiguation of unrestricted text (we are not dealing with a controlled language, but with the complexities of technical texts). In this analysis, the local grammars used by the rule engine, concentrate on the most common surface patterns of scientific English. 
3.1 Counting Phase 
This step of the analysis process is guided by the general control script file. The file has already been preprocessed: detection of phrasal verbs, contractions and abbreviations, using a special configurable preprocessing file. The control file contains a sequence of commands that will be processed by the analyzer in a sequential way. The current version of the ANESTTE control file contains up to 80 style-related variables. All the files admit comment lines. 
Two types of commands are available: 
  Counting commands: they specify the grammar rule file that must be used to compute a certain linguistic primary variable: 
• Percentage of abstract nouns. 
• Percentage of… 
  Computing commands: they allow to define new auxiliary or secondary variables, and to apply arithmetical, logical and comparison operations on pre-computed counting variables. These operations are: addition, subtraction, multiplication or division, conditional assignment, percentage, etc. 
In addition to these commands and user-defined variables, up to 14 predefined system variables can be used (ranging from number of words, sentences or paragraphs, to Gunning or Flesch indices). 
An example of computation is: 
GlobalScore= ConvictionScore + VarietyScore + 
ClarityScore + 
((ConcisionScore>7) @ (ConcisionScore-7)) 
In this equation, the variable GlobalScore is the sum of the other scores (with the exception of ConcisionScore, that needs to be scaled down when it is greater than seven). 
3.2 Grammar Rule Files 
The core of the grammatical system is the set of contextual hand-coded rule files (Duque 2000). They specify the conditions to increment the count of the associated variables. 
The text is iterated on a word-by-word basis and each rule in each file is checked against the stream of input words. Each line of the rule files defines a condition for incrementing the counter of a variable. Every time a line matches the input text at current position, the counter is incremented. A line contains a sequence of elements such as: 
  Words or punctuation marks, that should be present or should not be present (the matching can be carried out in a case-sensitive or in a case-insensitive way). 
  Endings of words: we can specify conditions that force the presence or the absence of a certain ending in the input words. 
  Syntactical tags: according to a dictionary search. 
  Another rule file: rule files are hierarchical and one rule line can contain a reference to another whole rule file, allowing a reuse of rules in several files, complex recursive patterns, etcetera. A rule file element is satisfied when at least one of its rule lines matches the text at current position. 
  A logical combination of conditions: we can impose the presence (or absence) of a word (or a word ending or a syntactical tag) and, at the same time, the satisfaction of another rule file. 

We can ask for the positive or negative satisfaction of two rule files, or for the positive matching of one rule file and the negative matching of another one. 
Although the language of rule files is mainly declarative, some procedural elements are necessary for the detection of certain structures: 
  A sequential search in the text for the matching of a file (it usually contains conditions for the determination of the end of a sentence). While the system is going forward, it must also count the occurrence of the elements from another rule file. For the satisfaction of the complex rule at the end of the search, the counter must have a certain configurable value (it must be zero, or greater than a minimum value…). This rule is not applied in a word-by-word basis. Each search moves forward the current analysis pointer. 
  The same rule as above, but with word-by-word processing if the element is not satisfied. 
  A modifier for an element that checks for the presence of this element, but does not change the analysis pointer. 
More than 80 counting and computation rules have been written. They reference 324 carefully designed rule files. 
Some examples of rule lines (that describe infinitive phrases) are: 
  [Infi-0] to +infinitive 
  [Infi-0] +infinitive 
  [Infi-0] [Object] +infinitive 
  [Infi-0] [Object] to +infinitive 
  [Infi-0] for [Object] to +infinitive 
The Infi-0 rule file contains a list of verbs that can be followed by to, followed by infinitive. The file called Object describes typical grammatical-object structures, and it contains lines such as: 
  +Pronoun 
  [Det1]+adjective_<Adject-0> +noun_<Noun-0> 
The element +Infinitive specifies that the last word in the sequence has to be tagged as a verb in the dictionary. 
Emboldened words are words that must be literally present in the text at the specified relative location (after the first infinitive, before the last one, etc.) 
The rule +adjective_<Adject-0> matches adjectives that are not included in the Adject-0 rule file. 
For writing and checking all these rule files, we have created a development environment with an integrated editor and a rule file parser (that checks the syntax of the file), the analyzer software and final dialog windows that show the detailed results of the analysis (Figure 2). 
3.3 Acceptability Phase 
Using the primary and secondary variables computed at the first phase, the acceptability files contain a set of formulas that define the Boolean features that characterize a good writing style. These new variables will be true when a range-condition formula is satisfied. 
These formulas specify the minimum and maximum acceptable values for the appropriate combinations of the first-phase variables. These acceptability formulas can be grouped in four main categories of analysis: 
  Clarity formulas: they check which clarity related variables are within the acceptable range: 
• less than 30% of passive voice sentences, 
• less than 50% of abstract nouns. 
• … 
  Conciseness formulas: that check which Conciseness-related variables are within the acceptable range: 

• less than 1.3 compound prepositions, 
• average sentence length between 20 and 28 words. 
• … 
  Conviction formulas: that check which conviction related variables are within the acceptable range: 
• predominance of strong modal verbs, 
• predominance of conviction adverbs over doubt adverbs. 
• … 
  Variety formulas: they check which variety related variables are within the acceptable range: 
• A balance between short, medium and long sentences or paragraphs, 
• a balance between static, dynamic and modal verbs. 
• … 
The number (or a range) of satisfied features determines the local score of the text in each category, and the combination of these four scores is the general evaluation of the text. 4 5 
Evaluation of the analyzer 
There have been two types of evaluation processes: 
  Objective evaluation: using the debugging version of the software, we have checked the correctness of the linguistic analysis. This debugging version applies the rules to a text and outputs the rule files, the specific rules lines and the fragments of texts that match each rule. This output allows for an easy revision of the precision of the analysis (we look for maximizing precision, not recall). 
  Evaluation with published papers: we analyzed 60 papers that were previously published in international journals (average number of pages: 12). The test confirms that mean global scores and their distribution for native and non-native papers are not significantly different under ANOVA and chi-square tests with 0.95 confidence level (all the papers were refereed and revised). More than 90% of the papers receive more 5 points in a 7-point scale, confirming the quality of the analyzer on published papers. 
The dialogue component 
Although the general and sub-area results from the ANESTTE analysis can be very useful for a technical writer, an assistant must provide an interactive help system. 
The system can access up to 116 HTML connected pages that have been developed for explaining the elements of style and the acceptability rules that define a good style (Figure 2). 
An animated agent (Gustafson 1999) explains the scores at the sub areas, both the good ones and the bad ones (Figure 3). A score is considered good (or remarkable) when it is above the global score (and above a certain minimum score: we should not praise a sub-area that is below this value, because we would be praising a really bad style). Using this strategy, most users receive a certain praise (some of the scores will be above the mean and, at least, are more positive than the others). 
A score is considered not good (or bad) when it is below the mean (or below a certain minimum score). For these scores (some of them can be good in absolute values), the agent asks the user whether she/he wants more details. If the answer is positive, the agent establishes a new objective for the less positive sub-areas (not more than two: we cannot overflow the user with many faults) and suggests a set of variables for the improvement in the worst sub-area. The number of suggested variables depends on the specific score. For each variable, the agent shows the corresponding HTML help file (although a general-purpose search engine is also included in the environment). 
As the rest of the system, the dialogue component has been designed in a fully configurable way (San-Segundo 2001). It receives a database of abstract results of analysis, and a set of reference values. A script file describes the sequence of multimodal interactions with the user, and specifies the 

operations on the input database that determine the parameters of the dialogue (the good and the bad sub-areas, the degree of praise, an objective for improvement…). 6 
Conclusions 
We have designed, implemented and tested a new tool for helping technical writers improve the quality of their papers in terms of linguistic style. The program performs syntactical and lexical analyses to check a paper for clarity and conciseness of linguistic expression, and variety and conviction in writing style. 
The tool has been tested on a corpus of published papers, verifying the correctness of the analysis and confirming the hypothesis that published papers from non-native speakers are stylistically very close to native-speakers papers. 
The new writer's assistant includes a set of web pages that explain the relevant linguistic elements to non-native users, describing the acceptability rule that must guide the way they write. An animated agent makes comments on the results and suggests some weak points that should be improved. 
Acknowledgement 
This work has been partially funded by Universidad Politécnica de Madrid through its A9904 project. Special thanks to José Manuel Pardo, for his comments; to Georgina Cuadrado, for her linguistic help; to Rogelio Vargas, Pilar Santamaría and Jesús Heras, who have helped in the programming tasks, and to all people that work with us at ETSIT-UPM. 
References 
Duque M.M. 2000. Manual de Estilo. El arte de escribir en inglés científico-técnico. Madrid, Paraninfo. 
Flesch R. 1948. A new readability yardstick, Journal of Applied Psychology, 32: 221-233. 
Glover A., Hirst G. 1996. Detecting stylistic inconsistencies in collaborative writing. In Sharples M., van der Geest T. (Eds.) The new writing environment: Writers at work in a world of technology. London: Springer-Verlag, pp. 147-168 
Jensen K., Heidorn G. E., Richardson S. 1993. Natural Language Processing: the PLNLP approach. Boston: Kluwer. 
Kessler B., Numberg R., Schütze, H. 1997. Automatic detection of text genre. In Proc. of the 35th Annual Meeting of the ACL, pp. 32-38. 
Lundberg G. 1994. The Functions of Refereed Scientific Journals. In Weeks R., Kinser, D. (Eds.) Editing the Scientific Refereed Journal. Practical, Political and Ethical Issues. IEEE Press, pp. 1-6. 
McGowan S. 1992. Ruskin to McRuskin: Degrees of interaction. In Computers and Writing: State of the Art, P. O'Brian and N. Williams (Eds.), Oxford: Kluwer, pp. 297-318. 
Montero J.M., Córdoba R., Vallejo J.A., Gutiérrez-Arriola J., Enríquez E., Pardo J.M. 2000 Restricted-Domain Female-Voice Synthesis in Spanish: from Database Design to ANN Prosodic Modeling. In Proc. of the International Conference on Spoken Language Processing, Beijing. 
San-Segundo R., Montero J.M., Colás J., Gutiérrez- Arriola J., Ramos J.M., Pardo J.M. 2001 Methodology for Dialogue Design in Telephone-Based Spoken Dialogue Systems: a Spanish Train Information System. In Procceedings of EUROSPEECH'01. 
Sharples M., Goodlet J., Pemberton L. 1992 Developing a Writer's Assistant. In J. Hartley (ed.) Technology and Writing: Readings in the Psychology of Written Communication. London: Jessica Kingsley, pp. 209-220. 
Figure 3. An animated agent explains the analysis results to the user. Figure 2. The result of the analysis and an HTML page that instructs on the use of the passive voice. 

552
The Student Engineering Corpus: Analysing word frequency
Olga Moudraia
Lancaster University, UK
Abstract
This paper presents the findings of a computer-aided research that aimed to establish a frequency-based
corpus of student engineering lexis. The Student Engineering Corpus reported here represents the
engineering lexis encountered in the English-language textbooks in basic engineering disciplines that
are compulsory for all engineering students regardless of their fields of specialization.
The Student Engineering Corpus contains nearly 2,000,000 tokens and over 18,000 types. On its
basis, a word list of the most frequent engineering lexis was developed which is organised by word
families consisting of over 1,200 word families that comprise about 9,000 words. The word-familybased
approach to word list organization may help broaden the EFL / ESL learners’ lexical base and,
more importantly, raise their awareness of the lexical nature of language. The Student Engineering
Word List can also serve as a lexical syllabus foundation of English for Engineering.
In addition, frequency and other statistical information may help teachers and learners make
judgements about the importance of frequent usage and core meaning of individual vocabulary items.
This paper shows the results of the content analysis and word frequency analysis of the Student
Engineering data, in comparison with the COBUILD Bank of English Corpus and the written part of
the British National Corpus (BNC). The analysis was concerned with the most frequent word forms in
all three corpora, including the most frequent closed-class (grammatical) and open-class (content) word
forms, as well as keywords and key concepts.
1. Introduction
Corpus-based studies of language data have become a norm in linguistics. Larger and larger language
corpora are being developed, and a corpus containing less than a million words will be considered
small now. Various professional corpora are coming into existence. One such project, the Corpus of
Professional English (in collaboration between Professional English Research Consortium, Japan, and
Lancaster University, UK) is underway, for example, and, when finished, it will consist of a 100-
million-word database of English used by professionals in science, engineering, technology and other
fields (http://www.perc21.org/cpe_project/index. html). Another one, a monitor engineering corpus of
several million words representing the English used by engineers in over 355 professional engineering
organizations, has been growing at the University of Aizu in Japan (Orr and Takahashi 2002).
However, parallel to the rapid development of large corpus studies, an interest in the analysis of
small corpora has arisen (Chadessy and Roseberry 2001), especially in the area of language teaching
where smaller corpora can be more useful than, for example, large, evergrowing, constantly changing
monitor corpora. Particularly, smaller corpora are designed to represent the specific part of the
language under investigation and are tailored to address the aspects of the language relevant to the
needs of the learner. Importantly, well-designed specialist corpora provide practical examples of
particular language uses that are of specific inerest / relevance to the language learner. Furthermore,
they are more manageable allowing easier and faster access to language data. The examples of the
smaller technical corpora designed for language learners are the Guangzhou Petroleum English Corpus
of about 400,000 tokens (Qi-bo 1989) and the Hong Kong University of Science and Technology
(HKUST) Computer Science Corpus of 1,000,000 tokens (James et al 1994).
The Student Engineering Corpus reported in this paper contains nearly 2,000,000 tokens and is
similar in design to the HKUST Computer Science Corpus. It was built in order to establish a corpus of
Student Engineering English that represents engineering lexis encountered in compulsory textbooks for
engineering students regardless of their fields of specialization, and is quite unique in this respect. I
also aimed to provide teachers and learners with a word list that could serve as the lexical syllabus
foundation for Engineering English.
This paper presents the findings of my research establishing a frequency-based corpus of student
engineering lexis that led to the development of a word list of the most frequent engineering lexis
which is organised by word families consisting of over 1,200 word families that comprise about 9,000
words. The paper will also discuss the results of the content analysis and word frequency analysis of
the Student Engineering data, in comparison with the COBUILD Bank of English Corpus and the
written part of the British National Corpus (BNC).
553
2. The Student Engineering Corpus
2.1 Rationale behind the study
This project's goal was to develop a reliable lexical syllabus for the engineering students in order to
meet the objectives of English teaching for Engineering at Walailak University in Thailand1 where I
had worked for nearly seven years. One of those objectives was to provide students with a solid basis
for further study, or for entering careers which require use of English. During their studies, students are
supposed to acquire a working knowledge of English - a practical skill valued in many of today's
professions. This is a demanding task, and to cope with it, dependable teaching materials are needed.
At Walailak University English teachers create teaching materials to suit the particular needs of their
students.
We were also in a situation quite common in Southeast Asia: lectures in most subjects were
delivered in a local language (Thai, in this case) while textbooks were in English. That is why, in order
to build a representative corpus of Student Engineering English, I selected the English-language
textbooks in basic engineering disciplines, such as Engineering Mechanics, Engineering Materials,
Mechanics of Materials, Mechanics of Fluids, Thermodynamics, Electrical Engineering, Engineering
Drawing, Manufacturing Process and Computer Programming that were compulsory for all engineering
students at Walailak University regardless of their fields of specialization2. The major criterion for the
selection was that those textbooks were recommended for the engineering students who had to read
them in English.
2.2 Procedures
The stages in the project included gathering a text corpus, processing it onto computer, conducting the
computer analysis of the material and building the word list. The processing of the text corpus onto
computer ready for analysis required scanning and verifying the texts with the help of Optical
Character Recognition (OCR) software3. The most laborious stage, however, was the analysis of the
engineering lexis that involved lexical computing and frequency count.
The material was analysed with the help of WordSmith Tools 2.0 software – an integrated suite of
programs for examining words’ behaviour in texts. The WordList tool was employed to generate lists of
all the words or word-clusters in the corpus, set out in alphabetical and frequency order, while the
KeyWords tool was applied to identify keywords and make a database of keyword lists enabling
identification of key keywords – words that are most frequent over a number of files in the database.
The Concord tool that produces concordances and finds collocates of the search word was used to
differentiate between parts of speech (i.e. use as a noun and a verb), homonyms (i.e. light = heavy /
light = dark), and different senses of the same word (i.e. impress meaning a) press hard into a soft
surface leaving a mark or b) have a favourable effect on somebody). This material gave a corpus of
about 2 million tokens and over 18,000 types (see Fig. 1).
***********************************************************************************
Corpus
tokens - 1,986,595
types - 18,203
token / type ratio - 109.14
type / token ratio - 0.0092
Word Families
entries in the word list - 1,260
comprise types - 8,850
minimum frequency - 0.005 %
Figure 1 Statistics on the Student Engineering Corpus
***********************************************************************************
1 The research on which this article is based was supported by a small grant # 970112 from the
Walailak University Research Council.
2 I would like to thank the Walailak University lecturers of Engineering and especially Asst. Prof. Dr.
Rugchart Owasith for help with selecting textbooks.
3 This step also had to deal with obtaining permission from the publishers for the electronic use of their
texts. My acknowledgements go to McGraw-Hill Australia (permission issued on October 12, 1998),
McGraw-Hill Companies, Inc. (December 1, 1998), Brooks / Cole Publishing Company (grant No. G-
09857, November 17, 1998) and Addison Wesley Longman Limited (ref. AP/2743, November 25, 1998)
for their understanding of the nature of the project and kind permission to store their texts in an
electronic format in order to create a word list.
554
2.3 The structure of the corpus
The Student Engineering Corpus is composed of thirteen text files as presented in Figure 2.
***********************************************************************************
N Text File Bytes Tokens Types Type/
Token Ratio
Standardised
Type/Token Ratio
Ave.
Word Length
Overall 11,694,812 1,986,595 18,203 0.92 9.85 4.64
1 Manufact.txt 1,764,178 290,782 10,082 3.47 13.72 4.86
2 Material.txt 1,444,793 232,743 7,056 3.03 10.49 4.98
3 Fluidmech.txt 1,307,973 220,666 5,333 2.42 9.15 4.67
4 Mechmat.txt 1,177,429 202,513 4,125 2.04 7.79 4.48
5 Elec.txt 983,672 167,394 5,626 3.36 10.27 4.62
6 Intofluidmech.txt 860,281 147,028 4,666 3.17 9.54 4.61
7 Dynamics.txt 795,910 142,446 3,205 2.25 7.07 4.38
8 Statics,meriam.txt 710,854 127,623 4,129 3.24 9.57 4.39
9 Statics,beer.txt 668,896 121,696 2,919 2.40 6.94 4.28
10 Chemi.txt 653,622 110,812 4,299 3.88 9.60 4.62
11 Graph.txt 486,152 80,804 5,034 6.23 11.55 4.80
12 Pascal.txt 466,756 77,242 3,124 4.04 8.54 4.73
13 Draw.txt 374,296 64,846 3,030 4.67 9.05 4.52
Figure 2 The structure of the Student Engineering Corpus
***********************************************************************************
2.4 Annotation and tagging
The Student Engineering Corpus has not been annotated yet. The part-of-speech and semantic tagging
of the corpus is being carried out at the University Centre for Computer Corpus Research on Language
(UCREL) at Lancaster using CLAWS (the Constituent Likelihood Automatic Word-Tagging System)
for POS tagging and the SEMTAG program for semantic tagging.
Semantic tagging is a form of corpus annotation that assigns semantic field tags to the words in a
corpus. For this task, we will use the SEMTAG program, part of the UCREL Semantic Analysis
System (USAS), which was designed and used across a number of research projects over the last ten
years at Lancaster University that led to the initial design and implementation of the tools for semantic
analysis and their application in the domain of software engineering documentation with a web front
end called Wmatrix (for details, see Rayson, Garside, and Sawyer 1999) that we will use for the content
analysis of the Student Engineering Corpus.
However, prior to the actual semantic tagging of the Student Engineering Corpus, the current
SEMTAG Lexicon will need to be enhanced with the specific lexis occurring in the Student
Engineering Corpus that is not currently recognised by the software because of either missing in the
Lexicon or having a special meaning in the Student Engineering Corpus which is different from the one
in the Lexicon. These unrecognised words will be assigned proper semantic tags according to the
established set of semantic fields and subsequently added to the Lexicon. At the current stage, over 6%
of the lexis in the Student Engineering Corpus is not recognised.
2.5 Word list organization
The entries in the resulting Student Engineering Word List were lemmatized according to word
families. The lemmatization process reduced the number of entries to about 7,700 that were treated
according to the cumulative frequency of occurrence of the members of the word families, and the most
frequent word families (with the sum total of 100, or 0.005%) were selected. As a result, over 1,200
most frequent word families comprising nearly 9,000 words were included in the Student Engineering
Word List. Incidentally, the most frequent word family in engineering textbooks is use4 (see Fig. 3).
As can be seen from Figure 3, the word family here is interpreted in the the most broad sense - in
accordance with the Bauer's and Nation's (1993) level 7 of generalization which includes derived and
inflected forms as well as compound words. Thus, the word entry use lists not only use, uses, using,
used but also useful, usable, user, reuse, unused, misuse, abuse, multiuse and their derivatives, giving
details on the ‘sub-families’ within a family, i.e. misuse within use. In most modern dictionaries these
are separate entries, and an EFL / ESL learner may not necessarily notice the connection between them.
4 Interestingly, used is the most frequent content word form in Kuo's (1999: 10) Corpus of Scientific
Journal Articles of a similar size followed quite closely by using - the ninth most frequent content word
form in his corpus.
555
***********************************************************************************
# Head
word
Freq. % Words Joined
ABC order -
1186
Freq. order -
1
Use 10,313 0.52 use (2784: n – 961, v – 1823), uses (262: n – 48, v - 214), using (2100), used (4538);
useful (341), usefully (1), usefulness (7); useless (6);
usable (22), useable (2);
user (149), users (24), user's (2); usage (39);
reuse (4: n – 3, v - 1), re-use (3: n – 1, v - 2), reused (5),
reusable (7);
unused (5 - adj), unusable (5);
misuse (1 – n), misusing (1), misused (1);
abuse (2: v – 1, attrib – 1);
multiuse (1 – attrib), multi-user (1 – attrib)
Figure 3 Use - the most frequent word family in the Student Engineering Corpus
***********************************************************************************
Moreover, the Student Engineering Word List provides some additional information that allows the
learner to find out which word forms of a particular word are used more frequently than the others,
what part of speech represented by a particular word form is encountered more commonly (i.e. use as a
noun or use as a verb), and even compare different spelling of the same word (i.e. usable and useable
or reuse and re-use). Besides, frequency and other statistical information may help teachers and
learners make judgements about the importance of frequent usage and core meaning of individual
vocabulary items. Figure 4 presents the one hundred most frequent entries listed by headwords – the
base word or the most frequent word in the family.
***********************************************************************************
N Headword Freq. % N Headword Freq. %
1 use 10,313 0.52 51 area 2,827 0.14
2 force 9,247 0.46 52 plane 2,820 0.14
3 form 7,075 0.35 53 direction 2,784 0.14
4 flow 7,045 0.35 54 result 2,763 0.14
5 pressure 7,016 0.35 55 move / remove 2,751 0.14
6 show (v) 7,002 0.35 56 all 2,741 0.14
7 determine 6,896 0.34 57 follow 2,731 0.14
8 figure / configure 6,650 0.33 58 constant 2,719 0.14
9 section 6,404 0.32 59 unit 2,661 0.13
10 line 5,812 0.29 60 view 2,647 0.13
11 equation 5,771 0.29 61 fluid 2,639 0.13
12 point 5,236 0.26 62 know 2,609 0.13
13 angle 4,923 0.25 63 draw 2,603 0.13
14 act / react / interact / transact / counteract 4,666 0.23 64 operation 2,601 0.13
15 velocity 4,614 0.23 65 component 2,560 0.13
16 system 4,540 0.23 66 expression 2,528 0.13
17 value 4,484 0.23 67 beam 2,513 0.13
18 apply 4,327 0.22 68 end 2,484 0.12
19 problem 4,278 0.21 69 pipe 2,476 0.12
20 work 4,198 0.21 70 make 2,467 0.12
21 give 4,103 0.21 71 steel 2,429 0.12
22 axis 4,053 0.20 72 assume 2,424 0.12
23 stress 4,033 0.20 73 shear 2,409 0.12
24 material 4,014 0.20 74 case (= state) 2,351 0.12
25 center 3,992 0.20 75 find 2,343 0.12
26 length / long 3,890 0.19 76 diameter 2,341 0.12
27 part 3,867 0.19 77 obtain 2,341 0.12
28 surface 3,821 0.19 78 mass 2,337 0.12
29 solution (of a problem) 3,776 0.19 79 air / aero- 2,315 0.12
30 type 3,606 0.18 80 define 2,276 0.11
31 produce 3,582 0.18 81 also 2,267 0.11
32 metal 3,457 0.17 82 calculate 2,266 0.11
33 example 3,447 0.17 83 water 2,262 0.11
34 load 3,406 0.17 84 cut 2,258 0.11
556
35 other / another 3,371 0.16 85 element 2,254 0.11
36 time 3,299 0.16 86 rotate 2,250 0.11
37 high 3,252 0.16 87 maximum 2,246 0.11
38 energy 3,245 0.16 88 different 2,235 0.11
39 vary 3,232 0.16 89 change 2,205 0.11
40 number 3,216 0.16 90 equilibrium 2,183 0.11
41 temperature 3,119 0.16 91 structure 2,183 0.11
42 body 3,101 0.16 92 position 2,177 0.11
43 process 3,048 0.15 93 base / basic 2,172 0.11
44 chapter 3,016 0.15 94 write 2,167 0.11
45 moment 2,989 0.15 95 consider 2,154 0.11
46 machine 2,979 0.15 96 design 2,125 0.11
47 dimension 2,938 0.15 97 free 2,087 0.10
48 put 2,889 0.14 98 friction 2,086 0.10
49 placement 2,840 0.14 99 low 2,083 0.10
50 require 2,828 0.14 100 method 2,070 0.10
Figure 4 The one hundred most frequent word families in the Student Engineering Corpus
***********************************************************************************
In organising the Student Engineering Word List by word families, I accord with J. Nattinger
(Nattinger 1988: 69) who has suggested that words built about a particular root are gathered so that the
associations among them can be seen, and even though the meanings of these words may be slightly
different, clustering them will aid students in remembering their general meaning. P. Nation (Nation
1990: 17) has also pointed out that grouping words under headwords is an attempt to increase the
coverage of high-frequency vocabulary; the implication is that learning a word involves learning its
derived and inflected forms as well. Nation's statement echoes Carter and McCarthy's (Carter and
McCarthy 1988: 44) who have noted that among other things, ‘knowing a word’ means knowing its
underlying forms and derivations. I also believe that this approach may help broaden the EFL / ESL
learners’ lexical base and, more importantly, raise their awareness of the lexical nature of language5.
As it was pointed out elsewhere (Nation 1990; Nation & Waring 1997), word lists can be useful in
a number of ways – course designers may refer to them when considering the vocabulary component of
a language course; teachers may use them to judge whether a particular word deserves attention or not,
and whether a text is suitable for a class; and learners may use them as a checklist or even as a goal.
The Student Engineering Word List can help English instructors in selecting the vocabulary component
of a course for the engineering students or deciding whether a particular text is useful for a class,
serving as a lexical syllabus foundation of English for Engineering.
3. Word frequency analysis
3.1 Comparison with the COBUILD and the BNC Written
The word frequency analysis of the Student Engineering data was carried out in comparison with the
COBUILD Bank of English Corpus and the written part of the British National Corpus (BNC). The
analysis (Fig. 5-7) was concerned with the most frequent word forms in all three corpora, including the
most frequent closed-class (grammatical) and open-class (content) word forms. It has revealed, firstly,
that the most frequent word forms in all three corpora – being mainly function words – concur (see Fig.
5). The Spearman's rank order correlation between the fifty most frequent closed-class word forms in the
Student Engineering Corpus and the COBUILD Bank of English is .778 while in the Student Engineering
Corpus and the BNC Written is .802, with both figures significant at the .01 level.
***********************************************************************************
Student Engineering Corpus COBUILD BNC Written
N Word % N Word % N Word %
1 the 8.50 1 the 5.58 1 the 6.43
2 of 4.19 2 of 2.60 2 of 3.11
3 a 2.84 3 to 2.51 3 and 2.70
4 and 2.72 4 and 2.37 4 to 2.60
5 is 2.43 5 a 2.21 5 a 2.18
6 in 2.07 6 in 1.83 6 in 1.95
7 to 2.06 7 that 1.04 7 is 0.99
8 for 1.08 8 is 0.93 8 that 0.99
9 are 0.88 9 it 0.92 9 was 0.94
5 For a detailed overview of the lexical approach to foreign language teaching, see Moudraia 2001.
557
10 be 0.83 10 for 0.87 10 it 0.93
11 that 0.80 11 i 0.78 11 for 0.88
12 at 0.76 12 was 0.76 12 on 0.72
13 as 0.75 13 on 0.70 13 with 0.67
14 by 0.71 14 he 0.65 14 he 0.67
15 with 0.57 15 with 0.64 15 be 0.67
16 on 0.50 16 as 0.57 16 i 0.66
17 from 0.48 17 you 0.54 17 by 0.55
18 an 0.47 18 be 0.53 18 as 0.55
19 this 0.47 19 at 0.52 19 at 0.49
20 or 0.46 20 by 0.50 20 you 0.47
21 we 0.42 21 but 0.47 21 are 0.47
22 which 0.42 22 have 0.46 22 his 0.47
23 it 0.38 23 are 0.44 23 had 0.46
24 if 0.32 24 his 0.43 24 not 0.46
25 figure 0.31 25 from 0.43 25 this 0.45
26 flow 0.31 26 they 0.43 26 have 0.44
27 can 0.28 27 this 0.39 27 from 0.44
28 determine 0.27 28 not 0.38 28 but 0.43
29 force 0.27 29 had 0.35 29 which 0.39
30 two 0.26 30 has 0.34 30 she 0.38
31 shown 0.25 31 an 0.32 31 they 0.37
32 will 0.25 32 we 0.32 32 or 0.37
33 used 0.23 33 or 0.29 33 an 0.36
34 may 0.22 34 said 0.28 34 her 0.35
35 velocity 0.22 35 one 0.28 35 were 0.33
36 pressure 0.22 36 there 0.27 36 there 0.28
37 its 0.20 37 will 0.27 37 we 0.28
38 when 0.20 38 their 0.27 38 their 0.28
39 have 0.20 39 which 0.27 39 been 0.28
40 has 0.19 40 she 0.26 40 has 0.27
41 equation 0.19 41 were 0.26 41 will 0.26
42 not 0.19 42 all 0.25 42 one 0.26
43 one 0.18 43 been 0.25 43 all 0.25
44 each 0.18 44 who 0.25 44 would 0.25
45 point 0.18 45 her 0.24 45 can 0.22
46 where 0.18 46 would 0.23 46 if 0.21
47 system 0.17 47 up 0.22 47 who 0.21
48 forces 0.17 48 if 0.22 48 more 0.21
49 these 0.16 49 more 0.22 49 when 0.21
50 between 0.16 50 when 0.22 50 said 0.20
Figure 5 The fifty most frequent word forms in the Student Engineering Corpus, the COBUILD Bank of
English Corpus and the BNC Written
***********************************************************************************
Secondly, the comparison of the fifty most frequent open-class (content) word forms (Fig. 6) has
indicated that the content word forms in the Student Engineering Corpus are predominantly from the
scientific register while the most frequent content word forms in COBUILD and BNC Written are of a
general nature.
***********************************************************************************
Student Engineering Corpus COBUILD BNC Written
N Rank Word % N Rank Word % N Rank Word %
1. 5 Is 2.43 1. 8 is 0.93 1. 7 is 0.99
2. 9 are 0.88 2. 12 was 0.76 2. 9 was 0.94
3. 10 Be 0.83 3. 18 be 0.53 3. 10 that 0.99
4. 25 figure 0.31 4. 22 have 0.46 4. 15 be 0.67
5. 26 flow 0.31 5. 23 are 0.44 5. 21 are 0.47
6. 27 can 0.28 6. 29 had 0.35 6. 23 had 0.46
7. 28 determine 0.27 7. 30 has 0.34 7. 26 have 0.44
8. 29 force 0.27 8. 34 said 0.28 8. 35 were 0.33
9. 30 two 0.26 9. 35 one 0.28 9. 39 been 0.28
10. 31 shown 0.25 10. 37 will 0.27 10. 40 has 0.27
11. 32 will 0.25 11. 41 were 0.26 11. 41 will 0.26
12. 33 used 0.23 12. 43 been 0.25 12. 42 one 0.26
13. 34 may 0.22 13. 46 would 0.23 13. 44 would 0.25
14. 35 velocity 0.22 14. 55 can 0.20 14. 45 can 0.22
15. 36 pressure 0.22 15. 58 new 0.16 15. 50 said 0.20
16. 39 have 0.20 16. 59 do 0.16 16. 51 do 0.20
558
17. 40 has 0.19 17. 60 two 0.16 17. 61 could 0.16
18. 41 equation 0.19 18. 62 time 0.15 18. 64 time 0.15
19. 43 one 0.18 19. 63 people 0.15 19. 67 two 0.14
20. 45 point 0.18 20. 64 like 0.15 20. 70 may 0.14
21. 47 system 0.17 21. 68 now 0.15 21. 73 new 0.13
22. 48 forces 0.17 22. 71 year 0.14 22. 74 like 0.13
23. 51 surface 0.16 23. 75 first 0.13 23. 78 first 0.12
24. 52 energy 0.16 24. 76 could 0.13 24. 80 did 0.12
25. 53 stress 0.16 25. 81 last 0.12 25. 81 now 0.12
26. 54 section 0.15 26. 83 well 0.12 26. 83 people 0.11
27. 55 example 0.15 27. 85 years 0.11 27. 85 should 0.11
28. 57 line 0.14 28. 86 know 0.11 28. 86 very 0.11
29. 58 chapter 0.14 29. 89 very 0.10 29. 88 see 0.10
30. 60 use 0.14 30. 91 pound 0.10 30. 91 made 0.10
31. 63 temperature 0.13 31. 92 back 0.10 31. 93 back 0.10
32. 64 problem 0.13 32. 94 get 0.10 32. 94 way 0.09
33. 65 must 0.13 33. 95 may 0.10 33. 96 years 0.09
34. 66 given 0.13 34. 97 think 0.09 34. 97 being 0.09
35. 67 time 0.13 35. 98 even 0.09 35. 100 work 0.09
36. 68 body 0.12 36. 100 way 0.09 36. 107 make 0.08
37. 72 area 0.12 37. 101 right 0.09 37. 108 even 0.07
38. 73 constant 0.12 38. 102 three 0.09 38. 111 must 0.07
39. 75 value 0.12 39. 104 don't 0.09 39. 112 own 0.07
40. 77 number 0.12 40. 106 world 0.09 40. 113 know 0.07
41. 78 solution 0.12 41. 110 being 0.09 41. 115 year 0.07
42. 79 fluid 0.12 42. 111 says 0.09 42. 116 good 0.07
43. 80 shear 0.12 43. 112 government 0.09 43. 119 last 0.07
44. 81 length 0.12 44. 114 dollar 0.08 44. 120 get 0.07
45. 82 moment 0.11 45. 115 should 0.08 45. 121 three 0.07
46. 84 mass 0.11 46. 116 made 0.08 46. 122 well 0.07
47. 85 axis 0.11 47. 117 good 0.08 47. 123 take 0.07
48. 86 maximum 0.11 48. 119 see 0.08 48. 125 go 0.07
49. 87 thus 0.11 49. 120 go 0.08 49. 126 government 0.07
50. 88 work 0.11 50. 121 did 0.08 50. 129 man 0.06
Figure 6 The fifty most frequent open-class (content) word forms in the Student Engineering Corpus,
the COBUILD Bank of English Corpus and the BNC Written
***********************************************************************************
However, most frequently encountered words in the Student Engineering Corpus appear to be subtechnical,
i.e. words with non-technical as well as technical senses, common in most kinds of technical
writing. Words in general non-technical sense appear to be more frequently used in the Student
Engineering Corpus than the specialist terms, as can be seen in Figure 7. It seems to me an interesting
finding worthy of notice. It also concurs with Lynn's (Lynn 1973) observation about the absence of
technical terms in his resulting specialized word list for commercial students.
***********************************************************************************
Rank
ABC order Freq. order
Headword Freq. %
1032 29 solution (of a problem) 3,776 0.19
1033 242 solution (liquid) 1,025 0.05
Figure 7 Technical vs. non-technical senses
***********************************************************************************
3.2 Keyword analysis
The keyword analysis of the Student Engineering data was carried out by using the WordSmith Tools
2.0 software. To locate and identify keywords in a given text, the KeyWords tool compares the words in
the text with a reference set of words taken from a large corpus of text that acts as a norm.
Characteristically, keywords are not the most frequent words but the words which are most unusually
frequent in a given body of text against the reference corpus. Another interesting feature provided by
the KeyWords tool is the key-keyword analysis that allows us to see the most frequent keywords over a
number of files in the database (and not just, for example, in one or two texts only) ensuring even
dispersion. The key-keyword analysis of the Student Engineering data against the written part of the
BNC Sampler has shown some interesting information on the key verbs in the Student Engineering
Corpus – they appear to be predominantly from the academic register. The key verbs in the Student
Engineering Corpus are as follows: be, show, determine, use, require, obtain, apply, assume, calculate,
559
correspond to, define, give, act, illustrate, occur, become, consider, exert, indicate, locate, sketch,
solve, and substitute (see Fig. 8).
***********************************************************************************
N Word of 13 texts as % of 13 texts
1 is / are 13 / 10 100.00 / 76.92
2 shown 12 92.31
3 determine 12 92.31
4 used / using / use 11 / 9 / 5 84.62 / 69.23 / 38.46
5 required 10 76.92
6 obtained / obtain 9 / 6 69.23 / 46.15
7 applied 7 53.85
8 assume 7 53.85
9 calculate 7 53.85
10 corresponding (to) 7 53.85
11 defined 7 53.85
12 given 7 53.85
13 acting 6 46.15
14 illustrates 6 46.15
15 occurs 6 46.15
16 becomes 5 38.46
17 consider 5 38.46
18 exerted 5 38.46
19 indicated 5 38.46
20 located 5 38.46
21 sketch 5 38.46
22 solve / solving 5 / 5 38.46 / 38.46
23 substituting 5 38.46
Figure 8 Key-key verbs in the Student Engineering Corpus compared against the BNC Sampler Written
**********************************************************************************
This finding has an important implication for teaching Engineering English – more attention in the
ESP classrooms should be given to non-technical than technical vocabulary. It agrees with Hutchinson
and Waters’ (Hutchinson & Waters 1981: 66) opinion who have challenged the assumption that
technical vocabulary is the most useful for ESP students and arrived at the conclusion that students of
ESP require not a corpus of technical language but the ability to mobilize the resources of general
English in the solving of technical problems, and that technical English is only a development or
extended application of general English while a wide-ranging knowledge of everyday vocabulary and
the ability to mobilize this knowledge in the interpretation of technical discourse are important aids to
comprehension and memory.
4. The semantic profile of the Student Engineering Corpus
As I have mentioned above, the semantic tagging of the Student Engineering Corpus is being carried
out with the help of the SEMTAG program, part of the UCREL Semantic Analysis System (USAS),
which was developed at Lancaster University, and the content analysis of the Student Engineering
Corpus is being performed using a web front end called Wmatrix.
The semantic tagset employed by SEMTAG was loosely based on the major categories in
McArthur's Longman Lexicon of Contemporary English (McArthur 1981) and refined by Andrew
Wilson (available online: http://www.comp.lancs.ac.uk/ucrel/usas/semtags.txt). It has a multi-tier
structure with 21 major discourse fields, subdivided, and with the possibility of further fine-grained
subdivision in certain cases, for example:
T1 Time
T1.1 Time: General
T1.1.1 Time: General: Past
T1.1.2 Time: General: Present; simultaneous
T1.1.3 Time: General: Future
T1.2 Time: Momentary
T1.3 Time: Period
T2 Time: Beginning and ending
T3 Time: Old, new and young; age
T4 Time: Early/late
Wmatrix allows us to see significant concepts in the corpus and the words related to those
concepts in frequency order. The Wmatrix semantic frequency list has a very useful option ‘compare to
normative BNC IT’ which compares the concept frequencies against a subcorpus of the BNC that has
560
been semantically tagged. The BNC IT corpus contains 135 files selected from the pure and applied
science section of the BNC which are related to Information Technology (IT). Collectively, the files
form a corpus of 1.7 million words, of which about 60% are news stories relating to IT.
The results of the key concept comparison against BNC IT are displayed in Wmatrix with the most
significant key items towards the top of the list since the results are sorted on the LL (log-likelihood)
field which shows how significant the difference is. The items with a ‘+’ code signify overuse in the
given text as compared to the standard English corpora. To be statistically significant, the LL value
should be over 6.63 which is the cut-off for 99% confidence of significance (p<0.01).
Figure 9 illustrates the most statistically significant data on the overused semantic categories in the
Student Engineering Corpus produced in comparison between the Student Engineering semantic
frequency list and BNC IT. However, these results are not extremely accurate as 127,870 lexical items
(i.e. 6.4% of the lexis) are not currently recognised by SEMTAG. The current SEMTAG Lexicon will
need to be enhanced with this specific lexis occurring in the Student Engineering Corpus to be able to
assign proper semantic tags to the as yet unmatched lexis according to the established set of semantic
fields.
***********************************************************************************
Item Frequency LL Semantic category
N1 74870 +12844.38 Numbers
O4.4 21627 +10704.81 Shape
N3.5 7165 + 7287.83 Measurement: Weight
Z5 679652 + 7254.53 Grammatical bin
N2 11061 + 7152.46 Mathematics
O2 53221 + 6810.75 Objects generally
O1.2 6883 + 6703.00 Substances and materials: Liquid
O1 8707 + 6294.31 Substances and materials generally
N3.7 7675 + 4440.78 Measurement: Length & height
O4.6 3259 + 3805.63 Temperature
O1.3 3667 + 3679.58 Substances and materials: Gas
E3- 5407 + 3102.38 Calm/Violent/Angry
E6- 4666 + 2762.14 Worry, concern, confident
X6+ 6847 + 2715.36 Deciding
B1 8780 + 2664.81 Anatomy and physiology
M6 23444 + 2541.75 Location and direction
O4.6+ 2975 + 2318.45 Temperature
M5 3671 + 1880.51 Aircraft and flying
N3.2 3734 + 1758.18 Measurement: Size
O3 7581 + 1739.22 Electricity and electrical equipment
Z99 127870 + 1737.24 Unmatched
G3 3934 + 1600.35 Warfare, defence and the army; weapons
A1.7+ 4324 + 1387.76 Constraint
T3 3171 + 1371.41 Time: Old, new and young; age
A6.1+++ 4155 + 1352.22 Comparing:- Similar/different
N3.3+ 2642 + 1237.55 Measurement: Distance
T1.2 4270 + 1206.39 Time: Momentary
M8 1867 + 1008.59 Remaining/stationary
N5.1- 7570 + 917.77 Entirety; maximum
N3.8 2039 + 854.25 Measurement: Speed
A6.3+ 4361 + 835.41 Comparing: Variety
X5.2+ 4037 + 814.79 Interest/boredom/excited/energetic
B5 2923 + 800.28 Clothes and personal belongings
O4.6- 960 + 740.58 Temperature
A10+ 13370 + 739.95 Open/closed; Hiding/Hidden; Finding; Showing
F2 1324 + 678.71 Drinks
N3.1 2767 + 657.03 Measurement: General
A3+ 44725 + 640.90 Being
O4.5 2919 + 626.17 Texture
A12- 5182 + 598.44 Easy/difficult
A1.2+++ 760 + 591.31 Suitability
O4.3 3695 + 517.15 Colour and colour patterns
N6 1375 + 492.99 Frequency etc.
N5 22374 + 489.55 Quantities
A1.6 1060 + 484.44 Physical/mental
B4 921 + 476.94 Cleaning and personal care
A4.1 16529 + 475.69 Generally kinds, groups, examples
A2.2 13157 + 453.21 Affect: Cause/Connected
S1.2.5+ 1307 + 443.30 Toughness; strong/weak
W3 3929 + 442.86 Geographical terms
561
N5+ 11644 + 439.62 Quantities
B2- 1345 + 428.03 Health and disease
A6.2+ 6947 + 399.49 Comparing: Usual/unusual
N3 338 + 398.00 Measurement
Figure 9 Comparison between the Student Engineering semantic frequency list and BNC IT
***********************************************************************************
5. Further development
This paper has reported on the creation of a Student Engineering Corpus that was originally developed
at Walailak University, Thailand. The project had three primary aims: a) to establish a representative
corpus of Student Engineering lexis regardless of the fields of specialization; b) to provide teachers and
learners with a word list that could serve as the lexical syllabus foundation of English for Engineering;
and c) to explore the data for the linguistic analysis of the syntactical, morphological, lexical, and
discursive features of Engineering English. The first two aims have been accomplished although the
corpus still needs to be annotated, and proper part-of-speech and semantic tagging has to be carried out.
The third aim, however, is a long-term. The empirical evidence from the part-of-speech and
semantically tagged corpus of Student Engineering English will provide the basis for research into
syntax, morphology, vocabulary, and discourse of Engineering English. Tagged data will be beneficial
for the studies of semantic fields and grammatical categories. Subsequently, the material is expected to
produce valuable information relevant to wide-ranging linguistic analysis.
References
Bauer L, Nation P 1993 Word families. International Journal of Lexicography 6(3): 1-27.
Carter R, McCarthy M 1988 Vocabulary and language teaching. Harlow, Longman.
Chadessy M, Henry A, Roseberry R L (eds) 2001 Small corpus studies and ELT: theory and practice.
Amsterdam / Philadelphia, John Benjamins Publishing Co.
Hutchinson T, Waters A 1981 Performance and competence in English for specific purposes. Applied
Linguistics 2(1): 56-69.
James G, Davidson R, Heung-yeung A C, Deerwester S 1994 English in Computer Science: a corpusbased
lexical analysis. The Hong Kong University of Science and Technology, Longman Asia Ltd.
Kuo C H 1999 Can numbers talk? Basic data management of a corpus. RELC Journal 30(1): 1-17.
Lynn R W 1973 Preparing word lists: a suggested method. RELC Journal 4(1): 25-32.
McArthur T 1981 Longman lexicon of contemporary English. London, Longman.
Moudraia O June 2001 Lexical approach to second language teaching. Eric Digest EDO-FL-01-02.
Washington, DC, ERIC Clearinghouse on Languages and Linguistics
(http://www.cal.org/ericcll/digest/0102lexical.html).
Nation I S P 1990 Teaching and learning vocabulary. Boston, Heinle and Heinle Publishers.
Nation P, Waring J 1997 Vocabulary size, text coverage and word lists. In Schmitt N, McCarthy M
(eds), Vocabulary: description, acquisition and pedagogy. Cambridge, Cambridge University Press, pp.
6-19.
Nattinger J 1988 Some current trends in vocabulary teaching. In Carter R, McCarthy M, Vocabulary
and language teaching. Harlow, Longman, pp. 62-82.
Orr T, Takahashi A 2002 Constructing a corpus of fundamental engineering English for nonnative
speakers. In Williams J(ed), Proceedings of the IEEE International Professional Communication
Conference. Oregon, USA, pp. 403-409.
Qi-bo Z. April 1989 A quantitative look at the Guangzhou Petroleum English Corpus. ICAME Journal
13: 28-38.
Rayson P, Garside R, Sawyer P. May 1999 Language engineering for the recovery of requirements
from legacy documents. REVERE project report, Lancaster University.

Formulating Writer Stance: A Contrastive Study of EFL Learner Corpora 
JoAnne Neff, Francisco Ballesteros, Emma Dafouz, 
Francisco Martínez, Juan-Pedro Rica: 
Universidad Complutense de Madrid; Mercedes Díez, Universidad de Alcalá; 
Rosa Prieto: Escuela Oficial de Idiomas, Madrid 
1. Introduction 
In previous studies, the SPICLE1 writing research team at the University of Madrid studied various major factors influencing L2 writing ( Neff, et al. 1994, 2002). The results of these studies show that two influential factors in the construction of texts are language proficiency in L2 (including knowledge of the linguistic code and the rhetorical conventions of the L2) and transfer factors from L1 (including transfer of the linguistic code and rhetorical/literacy skills). A cross-sectional study of university EFL texts written by 1st and 4th year English Philology students, from the MAD Corpus (Neff, Dafouz, Díez and Prieto 1992), revealed that, as students become more proficient, they eliminate language errors but still experience major problems in constructing texts, such as formulating writer stance, choosing adequate information structure techniques and over- under- or mis-using metadiscourse connectors. 
This paper, part of the on-going research project funded by the Spanish Ministry of Education, presents a comparison of the devices used to construct writer stance in argumentative texts written by advanced EFL students (Belgian-French speaking, Dutch, Italian and Peninsular-Spanish EFL writers, from the ICLE corpus), with those used by native speakers. The main aim of the present study is the analysis of the EFL students' use of the rhetorical conventions of the L2 discourse community as compared to the reference groups, American university students (from the Locness corpus, Louvain) and professional newspaper writers (from the ESCC, Madrid2). 
As other EFL writing researchers have noted (Lorenz 1998, Petch-Tyson, 1998), advanced learners, even when they have mastered the basic rules of syntax and morphology, still experience difficulty in the adept use of rhetorical skills, especially the construction of writer stance, a lack of expertise which they share with novice native writers. Expressions which manifest the writer's voice (‘it is certain/clear/obvious/true that…’ or ‘I am sure/certain that…') are not very problematical for EFL students although not without difficulty. However, more intricate strategies, such as the construction of a reader-in-the-text (Thompson, 2001), pose difficulties for both native and non-native novice writers. By reader-in-the-text strategies, we mean constructions such as ‘It might be thought that ...', 'Most people believe that...' or ‘This is now well-understood'. That is, the writer constructs a reader-in-the text with sets of shared attitudes and knowledge so that it does not appear that the writer is presupposing ideas which might be unacceptable for the real reader. This strategy allows the writer to construct a balanced discourse (Quirk et al. 1985: 1436) ‘so that the text seems to anticipate objections and crosscurrents raised in the mind of the reader/hearer.’ 
The results of the study, based on both corpus and qualitative analysis, reveal significant differences between native and non-native texts, but also between signed editorial texts and American student texts. All of the student texts, both native and non-native, show a lack of awareness of the appropriate degrees of directness and indirectness in constructing writer stance. This lack of sophistication is compounded by the use of amateurish interactional strategies, such as the use of but when the student writer has not established a real antithesis. Since these interactional strategies are common to all the student texts, they appear to be a characteristic of novice writing, rather than a result of the influence of L1 writing conventions. Although language errors, particularly the incorrect use of lexical phrases, and some transfer of L1 writing conventions (Neff, et al. in press) affect the reader's understanding and acceptance of advanced EFL texts, writing expertise appears to be just as decisive a factor. 
1 The SPICLE research group is the Spanish team contributing to the International Corpus of Learner English (ICLE), Université Catholique de Louvain-la-Neuve. 562 
2 The ESCC is the English-Spanish Contrastive Corpus, held at the Department of English Philology, Universidad Complutense, Madrid (Project BFF-2000-0699-C02-01 and 02, researchers, JoAnne Neff and Juana Marín) 

2. Interactive and interactional strategies 
Although interaction with the reader has long been a focus in the teaching of L1 writing in English (Britton et al. 1975; Shaunessy 1977; Bazerman 1981), it has a more recent history in ESL and EFL writing instruction. In an assessment of writing expertise and second-language proficiency on three composition tasks, Cumming (1989: 81) found that second-language proficiency was ‘an additive factor, enhancing the overall quality of the writing produced', but that, in the argumentative task, the quality of the writing was also influenced by discourse organization and content, by problem-solving behaviours, and by control strategies. He found that exceptionally high correlation ratings between the level of writing expertise and raters’ quality scores occurred only for the professionally experienced ESL writers. This finding partially coincides with Neff et al's (2002) study of advanced EFL writers included in the ICLE corpus. This research showed that the transfer of writing conventions from the EFL student writers’ L1 affected writer-reader interactional patterns in English. However, some of the unsophisticated argumentation strategies, particularly regarding the formulation of writer stance, occurred both in native and non-native student texts. This seems to point to novice writer characteristics as an important factor affecting the quality of EFL student writing, in addition to proficiency level. Thus, the training of novice EFL academic writers should focus not only language proficiency, but also on the development of writers’ interactional resources. 
According to Thompson (2001: 59), there are two main types of interaction in written texts. The first, called ‘interactive', attends to the management of information; the second, called ‘interactional’ attends to the readers’ involvement in the ethos of the argumentation. In the latter strategies, writers interact with readers ‘by appearing in the text to comment on and evaluate the content through the use of modality and evaluation and by assigning speech roles to themselves and to the readers’ (Thompson 2001: 63-64). This reader-in-the-text strategy is expressed through lexico-grammatical patterns which signal a hypothetical position (as in ‘it might be argued’ or ‘it may seem that') put forth by the writer, which is then followed by a statement of the ‘real', or frequently, a concessive relation (such as, ‘it is true/clear/inevitable that ...., but'), as in this text (1), which discusses the advantages and disadvantages of a pan-European market for share dealing. 
(1) There has been progress of a sort on the political front. The EU's political leadership has accepted the broad thrust of the recommendations of a committee of wise men aimed at harmonising Europe's regulatory regime. Critics, however, argue that the plan does not go far enough and lines of demarcation between the different authorities are unclear. Much work still remains to be done. 
The development of Europe's stock markets is not only an issue for companies and institutional investors. Private investors need to keep a close eye on developments, too. Long regarded as the ugly ducklings of the investment sector, private investors are suddenly sexy again. For one thing, as the chart shows, they are becoming richer, very much richer. 
In addition, as Europe's governments have embraced privatisation they have had to spread the gospel of wider share ownership in order to win both political and financial support for the new ideology. Latterly demographics have pressured governments into trying to persuade their citizens to make their own provision for retirement rather than rely entirely on state pension systems which are creaking under the strain. 
Inevitably much of that saving will be invested through Europe's stock market, either indirectly through managed funds or directly as investors make increasing use of the ease of access offered by the internet. But without a true single market Europe's savers and the companies in which they invest will remain at a disadvantage to their counterparts in the US. (Mark Milner, The Guardian, June 25, 2001) 
The hypothetical-real and concessive patterns (such as critics, however, argue and Inevitable ... But), along with the interactive signals (such as in addition) are strategies that allow the skillful writer to consider various hypothetical reader arguments and then strongly put forward his/her argument, in this case, signaled by the antithetical conjunction but. Sophisticated argumentation involves not only the use of both interactive and interactional markers over long stretches of text but also the presentation of propositions which support the major claims, as in the paragraph beginning with In addition. Without these further statements, which provide intermediate justification (Toulman 1958) for the final statement of position, stretches of text would appear to be marked by a profusion of signals with no claims, as frequently occurs in the student texts analyzed in this paper. 
In regards to the concessive patterns (those involving it is + (adverb) adjective + that), it must be noted that the choice of adjective puts strong constraints on what type of claim may be put forth in the following stretch of text. For instance, it is more likely that a writer will refute a previous argument expressed with it is true that X with but Y than when the initial phrase is it is noticeable that. Of the adjectives frequently used in this construction, there seem to be three semantic patterns: 1) those indicating probability (it is likely/probable that); 2) those signalling normativity, appropriateness and obviousness (i.e., those that are most often followed by a rebuttal or a partial qualification, such as it is 
563
clear that); and, 3) those expressing expectability, desirability and significance (i.e., those that are most often followed by additional proof, such as it is important that). 
3. Methods 
3.1. Description of corpora 
The argumentative texts used in this study came from three different corpora: 1) the essays written by advanced EFL university students whose first languages are Dutch (237, 631 words), Belgian-French (287, 683 words), Italian (226, 988 words), and peninsular Spanish (194, 845 words), all subcorpora from the ICLE Corpus, Louvain; 2) texts written by their American university counterparts (149, 790 words), a subcorpus from the Locness Corpus, Louvain; and, 3) texts written by professional newspaper writers, from a subcorpus of the Spanish-English Contrastive Corpus, held in Madrid (Marín and Neff 2000). From the latter corpus, we used only the signed editorials (103,367 words) as the reference group, as they are the editorials most similar to the student argumentative essays in regard to overt signals of writer stance. 
3.2. Analysis 
In order to search for the various constructions and to identify the most frequent lexico-grammatical patterns used by the EFL writers, the American university writers and the editorials, we used the Oxford University Press Wordsmith Tools to find the most frequent clusters and collocates for two types of patterns. One pattern was it is + (adverb) adjective + that, including agentless passive constructions, such as it is + (adverb) said/thought + that. The specifically the adjectives searched for were true, clear, obvious, undeniable, apparent, essential, important, understandable, un/likely, possible, probable, and sure. The second pattern involved the use of the conjunctions however, yet, nevertheless and but. 
For the evaluative adjective pattern, we initially used Lemke's seven evaluative orientations (1999), but because of the different rhetorical strategies involved in the selection of the adjectives, we decided to reduce the categories from seven to only three, one of which includes a different syntactic pattern, the agentless passive formula, such as it is well known that, as follows: 
1) Patterns that signal probability (i.e., those that express epistemic possibility, such as it is likely that) 
2) Patterns which express normativity, appropriateness and obviousness (i.e., those that are most often followed by a rebuttal or a partial qualification, such as it is clear that or it is well known that) 
3) Evaluative attributes which express expectability, desirability and significance (i.e., those that are most often followed by additional proof, such as it is important that) 
For the conjunctions, it was necessary to eliminate from the concordance lines all of the cases in which but, yet and however did not connect two clauses, as in difficult but necessary and all of the cases in which but was used as a preposition, as in nothing but trouble, or an adverb, as in decide but quickly, and yet used as an adverb, as in they have not yet decided. 
The statistical calculations were made by first norming the figures for each pattern by 10,000 words. We then compared the results to those of the reference group, the signed editorial texts, by using the Wordsmith Keyword tool in order to calculate the chi-square test of significance with Yates correction for a 2 x 2 table. 
After this quantitative analysis, there followed a long process of qualitative examination of the most significant quantitative results. This involved examining stretches of text surrounding the concordance lines to study how each group of writers used the interactive and interactional strategies with the propositions they put forth to the reader. 
4. Results 
4.1. Results for patterns expressing probability, normativity, appropriateness, obviousness, expectability, desirability, and significance 
The patterns used by each group to express probability, normativity, appropriateness, obviousness, expectability, desirability, and significance are displayed in Figure 1. When the two types 
564
2.325.946.488.244.773.81PRO/SIGNEDAMERICANSPANISHFRENCHDUTCHITALIAN 
Figure 1. The total figures for the use of it is + (adverb) adjective + that and agentless passive 
of patterns – the adjective with and without a premodifying adverb and the agentless passive – were counted together, all of the EFL writers showed an overused for the adjectival and the agentless passive constructions. Only the American university writers showed a use similar to the professional editorialists. An examination of the adjectival patterns showed that all the EFL groups overused the pattern with no premodifying adverb. For the pattern with a premodifying adverb, only the Dutch, the French and the Italian groups displayed a significant overuse, as compared to the professional editorialists. 
Regarding the findings for the two other it is + adjective patterns, none of the differences between groups were significant. 
4.2. Results for conjunctions however, yet, nevertheless and but 
Figure 2 shows the results for the total usage of the four conjunctions per group. All of the groups, except for the Spanish university writers, show a significant overuse of the conjunctions, as compared to the professional writers. This does not mean, however that the Spanish EFL writers are 
01020304050607080ProfAmericanDutchFrenchItalianSpanish 
Figure 2. Total tokens for conjunctions (however, yet, nevertheless and but) used per group 
the most similar to the editorialists, since in the total figures some of the large differences between the novice writers and the professionals cancel themselves out. The examination of the figures for each individual conjunction reveals a very different usage per group. Figure 3 shows that all the novice writers underuse yet, in comparison with the professional writers. 
565
0102030405060Prof.AmericanDutchFrenchItalianSpanish 
Figure 3. Number of tokens of the conjunction yet per group 
The professional writers are those who make most use of yet, followed by the American students, the French-speaking Belgians, the Dutch, the Italians and finally, the Spanish. All of the novice writers show a significant underuse, and for the Spanish writers, this conjunction is almost non-existent – only 2 tokens in almost 200,000 words. 
Equally interesting are the figures for however, displayed in Figure 4. All of the novice 
020406080100120Prof.AmericanDutchFrenchItalianSpanish 
Figure 4. Number of tokens of the conjunction however per group 
writers show a significant overuse of however, in comparison with the professional writers. The American university writers are those that most use this conjunction, followed by the Dutch, the 
010203040506070Prof.AmericanDutchFrenchItalianSpanish 
Figure 5. Number of uses of the conjunction but per group 
French-speaking Belgians, the Italians, and the Spanish. These results should be interpreted in the light of the novice writers’ underuse of but, in comparison to the editorialists, as displayed in Figure 5. 
566
The results for nevertheless also present interesting comparisons between the reference group and the novice writers. The difference in use between the professional writers and the American university writers was not significant, but this finding should also be compared with the latter's use of 
0510152025SignedAmericanDutchFrenchItalianSpanish 
Figure 6. Number of tokens of nevertheless per group 
the other three conjunctions. All of the EFL student writers showed a significant overuse of this conjunction, and again, this result is most probably related to their significant lack of other, more subtle conjunctions, such as yet. 
5. Discussion 
One of the characteristics of expert writing is the elaboration and extension of concepts over large stretches of text. Expert writers use reader-in-the-text strategies to anticipate diverse arguments which their readers might put forth. For these purposes, they make use –sparingly -- of lexico-grammatical patterns such as those examined in this paper, but, by themselves, these devices cannot provide a balanced argumentation. As these devices are highly visible, student writers seem to rely on them to structure their arguments, which are often single-clause explanations rather than extended inductive, deductive or causal chains of reasoning. The result is a profusion of devices, but weak argumentation. 
All of the novice writers, including the American university students, showed divergence from the professional writers in the use the two types of patterns examined here: the it is + Adj + that or agentless passive formula, and the conjunctions however, yet, nevertheless and but. As noted above, the types of adjective used in the adjectival construction may constrain the kinds of comments a writer may make in the following clauses. An examination of the types of patterns in each corpus revealed that not only did the professional writers use many fewer of each of these patterns, but they also used each of the patterns almost an equal number of times. The pattern they least used was the agentless passive (e.g., it is well-known that). All of the EFL student corpora reflected a significant overuse of this pattern. An examination of the students’ texts shows that they often use an agentless passive formula such as it is well-known that in order to introduce propositions which, in fact, cannot be taken for granted, as in example (2) from the Italian corpus. In cases such as these, the propositions which should establish a common ground may actually offend readers. 
(2) For example in America it is commonly believed that a 16-years-old teenager should be able to live alone. 
Some of the EFL students, particularly the Spanish, use agentless passive formulae such as it is said that to introduce a new topic or sub-topic, as in this discussion of the merits of military service in example (3). 
(3) Nowadays one of the great debates in the Spanish society is about the period of military training. It is a big problem for many families with sons who have to do the period of military training when they leave school. It has advantages and disadvantages. First of all it is necessary in case of war of the own country against an enemy country. Although at the present time the Great Powers are demilitarizing, there can be conflicts between two countries. In that case the military service would be essential. I also think that it should be compulsory just in case of emergency or if you had to call 
567
on people when not enough people are in the army. But I also think that people should make their own decision on what kind of wars to fight. Moreover it is said that many young people mature and shape their character when they do the military service. In contrast they have to be far from their families for a year. Furthermore they waste time trying to learn something that they will forget in the future. (Spm 07002) 
It is said that appears to introduce the new topic about military service bringing about increased maturity, but because it is preceded by moreover, an adverbial connective indicating addition, it looses its effectiveness as a reader-in-the-text device. The claim ‘many young people mature and shape their character when they do the military service’ cannot logically be an extension of the previous sentence ‘But I also think that people should make their own decision on what kind of wars to fight'. Furthermore, there is no following justification for why some people might think that military service makes the young mature and thus this claim appears to be a shallow proposition, as if the student writer were not really willing to give the idea careful consideration. The effect of lack of expertise caused by the profusion of connectors – each of the last four sentences has one in thematic position -- corroborates the findings of other EFL researchers (Lintermann-Rygh 1985), who have found that the percentage of connectors in relation to essay length does not correlate with the skill level of the writers. In fact, good argumentation relies more on the balancing of sound claims than on the connection of ideas through lexico-grammatical devices, as can be deduced from the lack of such devices in the editorial texts. 
Another pattern that was significantly overused by the EFL writers, but not the American student writers, was the one involving evaluative attributes which express normativity, appropriateness and obviousness, such as it is clear that. In the professional texts, the most frequent construction involved the adjective true. These propositions are almost always followed by a rebuttal or a partial qualification, which may also be placed in an initial concessive clause. That is, they represent a true reader-in-the-text strategy. The EFL student texts there were many cases in which the adjectival pattern is not supported by further statements of proof, as in example (4) from the Italian corpus. 
(4) On the other hand, someone can say that it is not possible made gun ownership illegal, because guns, if well used, are important for our protection. At these times we need protection, and a gun could be important in this occasion. Moreover, it is always true that if guns are keep illegal, the illegal market will increase, too. And in Italy, for example, there will be new earnings for Mafia. In conclusion, the solution for a so important problem, like crime, is in changing our society, in cleaning our television programmes and movies from violence, and in making everyday more difficult to keep a license. (It tto 2035) 
As can be observed in both example (3) and (4), the argumentation includes very few supporting statements. This most probably forces the student writer to continue ahead to the next sub-topic and therefore, to another search for a topic-presenting device. 
The results for the total number of the four adversative conjunctions reveal that all the novice writers, except for the Spanish, overused them in comparison with the professional writers. This does not indicate, however, that the Spanish writers use these conjunctions skillfully. The conjunction which most shows the divide between the professional writers and the novice ones is yet. Less forceful than but, especially in thematic position, yet has a concessive import which allows the skillful writer to make an additional adversative assertion. The concessive clause, apart from signaling contrast, also implies that there are slightly unexpected claims to be added (Quirk et al. 1985: 1088). It signals a careful second consideration of a additional reasoning, as in this argumentation chain found in one of the professional texts. 
(5) Maastricht went far to destroying the European parties of the Right by making them seem, or actually to be, the enemies of national prosperity. Yet even without Maastricht, Europe in the 1990s would have been failing the test of competitiveness. In terms of growth, it has fallen behind Asia. In terms of export competitiveness and financial strength, it has fallen behind Japan. In terms of technology, Europe has fallen behind the United States. It is a disastrous record. (William Rees-Mogg, The Observer, June 5 1997) 
The second sentence in this text makes the additional claim that European competitiveness was already failing before the Maastricht agreements, which is the first claim. The three following sentences provide justification in parallel structures, ‘in terms of growth', ‘in terms of export competitiveness and 
568
financial strength’ and ‘in terms of technology'. Then the last, short sentence offers the only conclusion a reasonable reader could deduce from such conditions. 
The novice writers’ use of yet is very scarce and not always appropriate, as this passage from the American essays. In example (6), yet is not followed by clauses which offer the reader additional and unexpected claims. The co-occurrence of but and yet is not only stylistically undesirable (Quirk et al. 643) but the following clause is so obvious in nature as to not constitute a claim at all. 
(6) Children are not educated enough below the age of twelve to experiment everything they see on television. The variety of television programs that children have to choose from is not large at all. But yet children have more time and use their time to watch television more than adults do. There are even some cartoon programs that are on early in the morning and early in the evening that are suitable for children,to watch. The parents do not pay attention to the cartoon programs to see what the content is. They just assume that because it is a cartoon that it is suitable for their child to watch. (ICLE-US-SCU-0010.2) 
However, the American students do frequently show a more sophisticated use of yet which might match the use the professionals make of this conjunction. In example (7), the student writer introduces the topics of ‘drugs’ or ‘crime’ as major social problem only to discount these in favor of his/her topic, 
(7) When one thinks of the worst problem facing the young people today, 
things such as drug abuse and crime are the first things to come to mind, yet 
arguably, the worst problem concerning are young people today is adolescent 
suicide. Adolescent suicide is a major problem in our society that gets less 
attention than the other big problems in our society such as guns and drugs. 
(ICLE-US-PRB-0034.2) 
adolescent suicide. In this case, the conjunction yet does signal a following and unexpected topic presentation and is accompanied by arguably, an adverbial hedge which has the same semantic import as ‘some might argue that'. 
In regards to the use of but, the American students writers underuse this conjunction while the Italian and French EFL students overuse it. The results of the American students suggest a teaching effect, since native speakers are often told not to begin sentence with this conjunction. However is the most frequent suggestion for sentence initial position. These native-speaker results coincide with those found by Altenberg and Tapper (1998: 86). Their comparison of Swedish EFL writers with British university writers showed that the native students relied strongly on however as a contrastive connector; in fact every fourth conjunct in the native essays was represented by however. 
Learners of English may also be influenced by the idea that but should not be used in initial position, in spite of the fact that this conjunction is frequently used in professional texts. For example, the Collins Cobuild English Usage manual (1992:112) gives the following advice to learners: ‘You do not normally put but at the beginning of a sentence, but you can do so when you are replying to someone, or writing in conversational style.’ These admonitions may account for some of the learners’ propensity to use however instead of but. In all the students texts, however is overused in comparison with the professional texts, while the editorials show a moderate use of this conjunct but a much more prominent use of the subtle yet. 
Some of the student texts tend towards conversational style in their use of but, as in this French EFL essay, where but is combined with conjoining lexical phrases that are stylistically aberrant. 
(8) During this period, they would live in special rehabilitation centres, which would form a kind of bridge between prison and normal life. The third and last stage would be the total reinsertion in society. As I explained it, the rehabilitation process would thus be a long and progressive one but according to me, it is necessary in order to ensure the protection of society. On the other hand, this process could not fit any case. Indeed, authorities must not take the risk of reinserting a dangerous murderer in society. That is why each case should be examined separately by a special court. Anyway, I am perfectly aware that it is much easier to imagine new systems than to put them into practise. 
(UC rul 2024) 
This pattern also occurs in the Italian EFL data. Once again, this EFL student writer's essay shows a profusion of connectors, some formal, some informal, where perhaps the content should be left to speak for itself. In other cases involving the uses of but, this conjunction appears to be a topic introducer rather than an strong adversative to a previous argument, as in example (9) from the French EFL essays. 
This is also the case with Martin on the rock trying to ensure his life with a minimum of comfort. In this has becom the main preocupation. Man sees his richies as a reward by God for his work. Man seems to be guided by hatred, egoism, ambition and not by love ... This is a terrifed picture of life but 
569
very realistic. There are many other passages illustrating this satire but we must go over with another important feature of Postmodernism illustra ted also by means of the Tralfamadore namely : the rationalism. Postmodernist writers seem to doubt rationalism. We can give some exemples which show what is meant by this. The modern man always asks questions, he must always know ... (Fr- uc 2053) 
Nevertheless is used so infrequently by the professional and American student writers and so frequently overused by the EFL that it appears to be, as well, the effect of not having been taught about the semantic differences between connectors. In Spanish, nevertheless corresponds to sin embargo, which is also the translation of however. This may lead Spanish EFL students to believe that these conjunctions are interchangeable, although this is not so. Nevertheless, that is in spite of all these facts, signals that many or very strong reasons have previously been given for arguing in favor of a particular position, while however does not indicate the same strong claim. The American writers seem to know this, as their usage is more in accord with the professional editorialists (i.e., no significant difference). But the fact that all of the EFL student writers show a significant overuse of nevertheless seems to point to a teaching effect. Perhaps textbooks, such as Jordan's Academic Writing Course (19 ) lump all these conjunctions together, although, in fairness, it must be said that Jordan at least mentions the concessive nature of yet and nevertheless and adds that this type of connector indicates the surprising or unexpected nature of what is being stated in view of previous statements. 
One aspect which merits attention, although not directly connected with argumentation strategies is the number of mistaken lexical or grammatical choices in the use of a formula merits. Formulae used by EFL writers, such as highly possible, instead of highly probable, or it is scientifically demonstrated, instead of it has been scientifically demonstrated, show not only that students need more practice with common expressions, but also that they rely just as much as native speakers do on lexical and grammatical patterns. 
6. Conclusion 
In this paper, we have examined writer's discourse strategies. Some of these present the writer's voice clearly, as in it is true/ clear that although these devices might be used to establish a previous context for a posterior disclaimer, as in the professional texts. The major difference between the professional writers and the student writers appears to be, on the one hand, an unsophisticated use of connectors (that is, not distinguishing between the adversative and concessive values) and on the other hand, a lack of supporting statements which would justify the previous claims. The latter seems to be due to novice writing techniques rather than to EFL language difficulties. That is, while the American university writers resembled the professional editorialists more than the EFL writers, they still resembled the EFL writers in quite a few aspects of novice writing. 
This consideration brings us to a methodological reflection. In previous studies, we have stressed the importance of using professional texts as a reference for the analysis of EFL texts instead of those written by native student writers. We have argued that student writers, even if proficient in English, are not necessarily proficient in writing skills. In this study, and after a comparison of the characteristics of signed and unsigned professional editorials (Dafouz 2000), we decided to use the signed editorials as the reference for all the university writers’ texts, both native and non-native. Signed editorials come closer to the type of task required of the university writers with regard to the explicit and implicit expression of writer stance and the use of boosters and hedges. 
We believe that studies of this type will lead not only to better EFL writing but more precise indications for native speakers of English as well. 
References 
Altenberg B and Tapper M 1998 The use of adverbial connectors in advanced Swedish learners’ written English. In Granger, S (ed), Learner English on computer. London, Longman, pp 80-93. 
Bazerman, C 1981 The informed writer: using sources in the disciplines. Boston: Houghton Mifflin. 
Britton, J, Burgess, T, Martin, N, McLeod, A, Rosen H 1975 The development of writing abilities. London, MacMillan. 
Collins Cobuild 1992 Collins Cobuild English Usage. London, Harper Collins. 
Cumming, A 1989 Writing expertise and second-language proficiency. Language Learning 39(1): 81-141. 
Dafouz, E , 2000 El metadiscurso como estrategia retórica en un corpus de textos periodísticos: estudio contrastivo de lengua inglesa y lengua espanola. Unpublished doctoral thesis, Universidad Complutense de Madrid. 
570
Jordan, R R 1980 Academic Writing Course. London, Collins. 
Lemke, J 1999 Resources for attitudinal meaning: evaluative orientations in text semantics. Functions of Language 5(1): 33-56. 
Lintermann-Rygh, I 1985 Connector density – an indicator of essay quality? Text 5(4): 347-357. 
Lorenz, G 1998 Overstatement in advanced learners’ writing: styloistic aspects of adjective intensification. In Granger, S (ed), Learner English on computer. London, Longman, pp 53-66. 
Marín J and Neff J 2000 Spanish-English Contrastive corpus. UCM, Madrid. 
Milner, M 2001 Time to take stock, The Guardian, June 25 (on-line edition). 
Neff, et al. in press Contrasting learner corpora: the use of modal and reporting verbs in the expression of writer stance. In Granger S, Petch-Tyson, S (eds), Extending the scope of corpus-based research: new applications, new challenges. Amsterdam, Rodopi. 
Neff, J, Dafouz, E, Díez, M, Martínez, F, Rica, J P, Sancho, C 2002 Subjective and objective modalization of evidentiality in native and non-native argumentative texts. In Facchinetti, R (ed), English modality in perspective. Genre analysis and contrastive studies. Verona, Centro Arti Grafiche Universita di Verona. 
Neff, J, Prieto, R l994 First language influence on Spanish EFL university writing development, in ERIC (ERIC Document Reproduction Service No. ED 385 144). Washington, D.C., Educational Resource Information Center. 
Neff, J, Dafouz, E, Díez, M, and Prieto, R 1992 The MAD corpus: Spanish EFL university writers. Madrid, Universidad Complutense de Madrid. 
Petch-Tyson, S 1998 Writer/reader visibility in EFL written discourse. In Granger, S (ed), Learner English on computer. London, Longman, pp 107-118. 
Quirk, R, Greenbaum, S, Leech, G, Svartvik, J 1985 A comprehensive grammar of the English Language. London, Longman. 
Rees-Mogg, W 1997 William Hague's impending success tells us much about British culture, The Observer, June 5 (on-line edition). 
Shaunessy, M P 1977 Errors and expectations: A guide for the teacher of basic writing. New York, Oxford University Press. 
Thompson, Geoff. 2001. Interaction in academic writing: Learning to argue with the reader. Applied Linguistics 22, 1: 58-78. 
Toulman, S 1958 The uses of argument. Cambridge, Cambridge University Press. 
571

The Cambridge Learner Corpus - error coding and analysis for lexicography 
and ELT 
Diane Nicholls 
Cambridge University Press (freelance) 
diane.nicholls3@btopenworld.com 
Abstract 
The Cambridge Learner Corpus is a 16 million-word corpus of Learner English collected by Cambridge University Press in collaboration with the University of Cambridge Local Examinations Syndicate (now Cambridge ESOL). It comprises English examination scripts, transcribed retaining all errors, written by learners of English with 86 different mother tongues. The scripts range across 8 EFL examinations and cover both general and business English. A 6 million-word component of the corpus has been error coded to date, using an error-coding system devised at CUP specifically for the Cambridge Learner Corpus. The majority of codes are based on a two-letter system in which the first letter represents the general type of error (e.g. wrong form, omission), while the second letter identifies the word class of the required word. There are 88 possible codes in all. 
This paper will describe the coding system and the corpus tools used for analysis of the coded corpus, and will demonstrate the benefits which this coding and analysis provides for both lexicographers and writers of other ELT books at CUP. 
1. Description of the corpus1 
Since 1993, Cambridge University Press, in collaboration with the University of Cambridge Local Examinations Syndicate (now Cambridge ESOL), has compiled a 16 million-word corpus of Learner English. Students' examination essays are carefully transcribed, reproducing all errors, checked for inputter-generated errors, and stored in the corpus, along with candidate details and examination scores. 
The corpus is growing all the time. At present, the complete corpus contains more than 16 million words. 86 mother tongues are represented in the corpus, with more than 350,000 words for more than 15 of the mother tongues. The error-coded component of the corpus currently contains 6 million words. 
A profile of each candidate is given for each examination script. This includes information on the first language, age, sex, education history and years of English study of each student. This information can be used to specify the parameters for the creation of subcorpora. For example, it is possible to isolate for analysis the English of very young learners or a particular examination level, mother tongue or language group. A combination of any of these details can be used to create a subcorpus. 
2. The coded corpus 
6 million words of the Cambridge Learner Corpus have been error coded to date. While error coding is certainly a laborious and time-consuming task, its benefits for our purposes, which I discuss below, have proved to far outweigh the difficulties. The system of error codes and software have been designed in such a way as to overcome, as far as possible, problems with the indeterminacy of some error types. The corpus has also been manually coded by just two coders, with one coder overseeing the work of the second, thus keeping to a minimum any problems with consistency of tagging. 
Our aim in coding the corpus is not to create a systematic taxonomy of learner errors but, where possible, to capture under one heading all errors (both of omission and commission) of a particular type so that they can easily be extracted and analyzed and the information gained passed on to examining bodies, teachers, lexicographers, researchers and ELT authors for use in developing tools for learners of English. The codes are not an end in themselves, but rather, act as bookmarks to the contexts in which an error repeatedly occurs. These bookmarks can then be referred to for further analysis. 
1 For further information about the Cambridge Learner Corpus, please refer to links at our website: http://www.cambridge.org/elt/corpus 
572
Also of vital importance is the additional feature of a 'correct' version being added, wherever possible, alongside the error. Care is taken not to 'interpret' or paraphrase errors and only to add a corrected version where there is relative certainty and only one clear replacement possible. This measure vastly increases the search potential of the corpus, allowing access to data which would otherwise be present but only indirectly accessible. 
The main advantages of a coded (and 'corrected') corpus over a raw corpus (without error tags) are as follows: 
• We do not have to look at everything and scan through both correct and incorrect uses. Correct uses (which are automatically tagged NE (no error)) can be deselected and the remaining cites sorted according to error type for easy look-up and analysis. This also allows the possibility of comparing what learners get right (an often neglected area in ELT) with what they get wrong. 
• Using the statistical tools built into our corpus software, we can simply establish the frequency, level of student or mother tongue for a given error (or correct use). 
• Perhaps the greatest advantage over an uncoded corpus is that we can search for errors of omission as well as commission. After searching through a concordanced search on 'at', for example, in an uncoded corpus, it is possible to locate errors such as the unnecessary use of the preposition (e.g. *watching at the television), and the erroneous use of the preposition (e.g. *she invited me at her birthday party). However, it is not possible easily to find: 
i) Instances of failure to use the preposition (coded <#MT>) where it is required (e.g. *we looked each other) 
ii) Instances of where 'at' should have been the chosen preposition, but a wrong preposition was chosen instead (coded <#RT> (e.g. *we arrived to our destination). This would not be possible without the addition of and ability to search on a corrected version. 
• An additional feature which is unavailable in a raw corpus, is that we can also search on the error tags themselves, collecting together all errors of a particular type and establishing statistics for their comparative frequencies of occurrence and obtaining information on which students are most likely to make a particular error and at which examination levels. For example, we can look at all noun countability or noun inflection errors in the corpus and establish which are the most common and for which students. This important advantage of the error coded corpus is demonstrated in section 5 below. 
• We can also search on clusters of errors. For, example, we can look at all noun-related or adjective-related errors en masse. 
3. The system of error codes 
Learner errors are tagged using the following convention: 
<#CODE>wrong word|corrected word</#CODE> 
The majority of the error codes are based on a two-letter coding system in which the first letter represents the general type of error (e.g. wrong form, omission), while the second letter identifies the word class of the required word. 
General types of error (first letter) 
F wrong Form used 
M something Missing 
573
R word or phrase needs Replacing 
U word or phrase is Unnecessary (i.e. redundant) 
D word is wrongly Derived 
The codes M, R, and U can occur alone where no more specific information can be given. 
Word classes (second letter) 
A Pronoun (Anaphoric) 
C Conjunction (linking word) 
D Determiner 
J AdJective 
N Noun 
Q Quantifier 
T Preposition 
V Verb (includes modals) 
Y Adverb (-lY) 
Punctuation errors (Error type + P) 
Punctuation errors are coded with P as the second letter, and one of the error types M, R, U as the first letter. 
MP punctuation Missing 
RP punctuation needs Replacing 
UP Unnecessary punctuation 
When punctuation errors are corrected by the coder, any change in capitalization is also shown within the error coding. For example, the sentence *He died we buried him the next day omits a full stop after he died and therefore requires the <#MP>|.</#MP> coding. However, the new sentence break requires the w of we to be capitalized. This is shown by the coding He died <#MP>we|.We</#MP> buried him the next day. This covers both the punctuation error and the required changes to the capitalization. 
Countability errors (C + word class) 
CN countability of Noun error 
CQ wrong Quantifier because of noun countability 
CD wrong Determiner because of noun countability 
CN means that the student has used a form which is not available in the intended sense of the noun (e.g. *the country's natural beauties, *two transports). Where a noun could be either count or uncount but the wrong form has been used, the error is FN, wrong form of noun (e.g. vacation/vacations). 
False friend errors (FF + any word class) 
All false friend errors are tagged with FF. The word class of the required word is specified by adding a word class code (A, C, D, J, N, Q, T, V, Y) to the FF code. This code is only used when the coder is certain he/she is dealing with a documented False Friend. Otherwise, it is treated as a replace (R) error. 
Agreement errors (AG + word class) 
AGA Anaphoric (pronoun) agreement error 
AGD Determiner agreement error 
AGN Noun agreement error 
AGV Verb agreement error 
574
Additional error codes 
AS incorrect Argument Structure 
CE Compound Error 
CL CoLlocation error 
ID IDiom error 
IN Incorrect formation of Noun plural 
IV Incorrect Verb inflection 
L inappropriate register (Label) 
S Spelling error 
SA American Spelling 
SX Spelling confusion error 
TV wrong Tense of Verb 
W incorrect Word order 
X incorrect formation of negative 
AS (argument structure error) covers errors in argument structures which cannot be coded as MT (missing preposition, e.g. *he explained me) or UT (unnecessary preposition, e.g. *he told to me). AS is particularly used for double object verbs, e.g. *it caused trouble to me is coded <#AS>it caused trouble to me|it caused me trouble</#AS> to circumvent the need for multiple codes to correct what is, in fact, a single error. 
CE (complex error) is a catch-all code to cover multiple errors and groups of words the intended sense of which cannot be established. By using this code, we factor out of the equation strings which can yield little useful information on learner errors. 
SA (American Spelling) is not a true error code since it is not always possible to know whether a learner has made a mistake or is deliberately using US spelling, but was introduced in view of the fact that learners are usually very clear about which variety of English they wish to learn and ELT and dictionary publishers still need to highlight the differences between the varieties of English in their products. The SX code (spelling confusion error) covers spelling confusables such as to/too, their/there and weather/whether. It is possible to override the distinctions made by these codes by searching on the general-spelling group code <#SPELL>. 
4. Some practical issues 
4.1. Avoiding over-coding and 'creating' errors 
We are not attempting to rewrite the scripts into perfect English or to interpret the text. Often, things could be expressed better by paraphrase - this is not our task. We are only correcting and documenting errors. Equally, the coder must resist the temptation to make moral judgements about a student's intended meaning. If the language used is 'correct', the idea behind it is not brought into question. 
4.2. Embedding errors 
In cases where a word is both wrongly spelled and the wrong word, for example, one code can be embedded within another. 
e.g. I like to <#S>hospitelize|hospitalise</#S> my relatives - the student means ‘put them up’ or ‘have them to stay’ (show them hospitality). This should be an <#RV> error, not a spelling error and the spelling correction should be embedded inside the more significant RV code. Thus: <#RV><#S>hospitelize|hospitalise</#S>|put up</#RV>. 
4.3. Choosing error codes in ambiguous cases 
Coders are occasionally faced with a decision between two different codes. This is the case in, for example: *He said me that ... The coder must decide whether to correct this by replacing the verb (RV) with a synonym (told) which works in the given argument structure (He told me that), or to add the missing preposition (MT) to make the argument structure of the student's chosen verb correct (He said to me that). 
We want to keep as close as possible to the student's original text, so it is helpful to think of the error codes as having a loose hierarchy. In this case, changing a verb's argument structure is a less severe change than changing the verb itself and puts right what the student said wrongly, rather than starting again 
575
from scratch. So, ‘He said <#MT>|to</#MT> me that’ is more helpful, than, ‘He <#RV>said|told</#RV> me that'. It is more helpful to teach students to use a verb correctly than to teach them to avoid using a verb which they use incorrectly. Similarly, where it is impossible to know what the student intended in a Replace error, no correct version is inserted, as a coder's ‘wild guess’ can be of little help and will distort the data. 
5. Using the data 
At CUP, the data gathered from the coded corpus is used by lexicographers and researchers working on dictionary projects to identify those words and constructions which are particularly problematic for learners. Based on the corpus information gathered, decisions are made about how best to direct the learner to the correct use of such words or constructions. Information about the language groups affected is also relevant. Attention is also paid to the examination levels at which particular errors occur. This search option is particularly useful as it enables us to establish which errors are typically elementary-level errors, for example, as well as to identify those more 'intransigent' errors which still persist, even at Proficiency level. 
Corpus data is also used by course and reference book authors to inform their work and by Cambridge ESOL to inform course designing, analyze marking schemes and to support their on-going examination research. 
5.1 Searching on individual words 
Figure 1 shows a screen from a search on the word 'Hardly': 
Figure 1 
The search is randomly sorted at this stage. The user can scroll up and down the search. As the user scrolls through the search, the blue line (here, in the centre of the screen) indicates the script which the user is currently looking at. The information to the left shows the mother tongue and examination level of the candidate. More candidate information for the relevant script appears in the grey field at the top left hand 
576
corner of the screen and is important for verifying whether an error is idiosyncratic to one individual student or occurs with a number of different students. There is the option to select an entire script for further analysis by moving the cursor to the required script and simply pressing Return. 
We can then get the statistics for the error. The statistics for error tags for the search word 'Hardly' are shown in figure 2 below: 
Figure 2 
This tells us that the frequency of correct use (NE) is 68.9% (416 citations); that the most frequent error is one of word order (W) (8.6%); the next most frequent error is one of wrong adverb use (RY) (5.4%) and the next, bad negative formation (X) (2.3%). From the statistics box, we can select the error that most interests us and display the citations containing that error. The statistics are not viewed as concrete evidence in themselves and can, admittedly, be distorted by many factors. Rather, they act as a clear signpost to a significantly recurring error and can be used as a path to viewing the errors themselves in context. The final judgment on the importance of an error and how to treat it lies with the analyst/lexicographer. 
Figure 3 shows results obtained when selecting the word order error (W): 
577
Figure 3 
Our search is also not restricted to the search word itself. Using the Position field in the Statistics box, we can select 0 to get statistics for the error tags of the search word. By putting '1' in the Position field, we get the error tags associated with words following the search word. This is very useful for establishing the following environments of a verb, for example. If we search on a verb (e.g. enjoy) and insert '1' in the position field, we find a high percentage of verb form errors (FV). If we select this statistic and View Cites, we find that learners frequently use the infinitive form of a following verb (*I enjoy to swim) rather than the -ing form (I enjoy swimming) with enjoy. It is then a short step to finding which mother tongues make this error most and at which examination levels. To examine preceding environment errors, a '-1' is inserted in the position field. 
5.2 Searching on error tags 
As discussed in section 3 above, a coded corpus gives the important advantage of being able to search on particular error types or groups of errors in exactly the same way as we search on individual words. For example, we can search for all noun countability errors (CN) in the corpus. Countable nouns are nouns which can take a plural or an indefinite article (e.g. tree), while uncountable nouns cannot take an indefinite article and are never used in the plural (e.g. information). Noun countability is an important area of confusion for learners of English, not least because a noun which is countable in their mother tongue is often inexplicably uncountable in English. 
Figure 4 shows a screen from a randomly sorted search on noun countability errors (CN): 
578
Figure 4 
This search can then be sorted for ease of reference and the statistics box used to obtain a list of the words most frequently involved in countability errors: 
579
Figure 5 
We can also look at the countability errors made by individual mother tongues2: 
Figure 6 
2 A forthcoming version of the software will be balanced for corpus frequency, enabling us more easily to establish which mother tongues most frequently encounter problems with noun countability or other errors. 
580
We can also obtain information on the examination levels at which these errors occur. These are shown in figure 7 below. It is encouraging to note that the incidence of countability errors gradually decreases as the students progress through examination levels to Proficiency level. However, for teachers and ELT authors alike, the fact that the incidence of error is still quite high (28.2% of these errors are made at Proficiency level) is a clear indication that more needs to be done to alert learners to this major pitfall in English language learning. 
Figure 7 
6. Conclusion 
Formerly, publishing houses engaged in producing ELT reference and course books were dependent primarily on the intuitions of highly skilled and experienced lexicographers to anticipate learners' difficulties with English. With a learner corpus up and running, these vital human skills are powerfully reinforced and supplemented by a database of 'living' errors and, most importantly, the contexts in which they commonly occur. 
The addition of error coding makes it possible to follow a structured, step-by-step procedure to arrive at the required data quickly, efficiently and informatively. 
A search on an uncoded corpus must, necessarily, depend largely on the searcher's expectations of what he/she is likely to find, and can only be as fruitful as the searcher is inventive in his/her searching strategies. In addition, with an uncoded corpus, a searcher can only search for what is there and can have only indirect and, to a large extent, 'accidental', access to what is not there - to errors of omission and corrected versions. 
The Cambridge Learner Corpus, with the error coding and corpus tools developed for its exploitation, is providing lexicographers, researchers, ELT authors and examiners with easy, direct access to a fund of information which they can interpret and use for widely varying applications. 
581

Using a dedicated corpus to identify features of professional English usage: 
What do “we” do in science journal articles? 
Judy Noguchi Thomas Orr Yukio Tono 
Mukogawa Women's University of Aizu Meikai University 
Nishinomiya, Japan Aizuwakamatsu, Japan Urayasu, Japan 
Abstract 
This paper addresses the problem of inadequate educational materials for the effective training of non-native speakers in the professional English of the scientific community. The paper claims that one cause of this inadequacy is lack of proper linguistic research based on suitable linguistic research tools. The development of a major international corpus of professional English in the sciences and other fields is described as a significant resource for solving this problem, illustrated with a specific research project examining the use of we in general English from the Corpus of Professional English. Results reveal the value of well-designed dedicated corpora for addressing the specific English instructional needs of non-native speakers of English in the professions. 
582

Using the HTK speech recogniser to analyse prosody 
in a corpus of German spoken learners’ English 
Toshifumi Oba (tosh@comp.leeds.ac.uk ) 
and Eric Atwell ( eric@comp.leeds.ac.uk ) 
School of Computing, University of Leeds, Leeds LS2 9JT, England 
1. Introduction: speech research, prosody and learners’ English 
Intonation is important in human communication to help the listener to understand the meaning and attitude of the speaker (Brown, 1977; O'Connor, 1970). Language students and teachers see intonation as a part of the structure of the language (Tench, 1996). However, acquisition of proper intonation is difficult for non-native speakers and requires repeated practice (O'Connor, 1970). For example, the Interactive Spoken Language Education (ISLE) project found that intonation was the biggest problem for German learners of English, but the project did not tackle intonation (Atwell et al, 1999). 
Intonation is also important for speech recognition to retrieve correct semantic and syntactic information and to successfully identify words (Rodman, 1999: Werner and Keller, 1994). It not only influences acoustic models, but can also contribute for language models; Taylor et al (1997; 1998) focused on the relationship of intonation and ‘move types’ of the dialogue to constrain the number of possible language models. However, intonation is generally limited to be that of native-speakers in speech recognition research. 
Speech recognition research has been done for non-native speakers too. The works can be categorized into two groups: multilingual speech recognition; and use of speech recogniser for foreign language learning. Uebler (1998) investigated bilingual (Italian and German) and multi-dialectical speech recognition by assuming the two languages being one. Stemmer et al (2001) developed acoustic models of foreign words which appeared in German dialog, such as English words in film titles. In the FLUENCY project, Eskenazi (1996) used a speech recogniser to detect foreign speakers’ pronunciation errors for second language training. This work also involved prosody looking at a correlation of pronunciation and prosody errors. However, little research has been undertaken on speech recognition targeted on non-native speakers’ intonation: the majority of speech recognition research focuses on pronunciation of native speakers. While there have been relatively few studies on speech recognition dealing with prosody or non-native speakers, those for non-native speakers’ prosody have received even less attention. 
The use of Speech recognition has been investigated in Computer-Assisted Language Learning (CALL) such as in Eskenazi (1996) and Witt and Young (1997). However, the ISLE project reported that detection of errors and providing feedback to the learners were not well developed in existing language learning software (Atwell et al, 1999). For example, a well-known software, ‘Tell Me More’ of Auralog, improved the detection and the feedback for pronunciation practice by pointing out error phonemes and showing a 3D animation to visualize the ‘model’ articulation. However, its technology for intonation practice is still poor. Eskenazi (1999) mentions that a visual display is more effective than oral instructions for intonation practice and ‘Tell Me More’ displays a waveform and pitch curve, which traces the amplitude and frequency variations, respectively, of both user's voice and a ‘model’ utterance. However, it neither points out the placement of the intonation errors, nor provides suggestions to improve intonation, leaving the comparison tasks to the users. Investigation of Germans speakers’ English prosody should help to improve these technologies, by pointing out their common weakness in English intonation, which should be included in exercises. 
Table 1 shows a variety of speech recognition research and which fields are involved in this research. Research fields to be considered are pronunciation, prosody, native speaker, non-native speaker, monolingual, multilingual and the HTK. ‘Y’ represents a field which a research deals with: otherwise ‘N’ is given. More precisely, ‘Y’ is marked in each column as follows: 
591
Non/Mul: when speech recogniser are for non-native speakers’ utterances or for multi-languages; 
German: when speech recogniser deals with German speakers’ English; 
Inton: when speech recogniser considers intonation features; 
G/B: when the research deals with ‘goodness’ and ‘badness’ of intonation; 
HTK: when the research exploits the HTK based-speech recogniser. 
Table 1: Features of Various Speech Recognition Research 
Research Reference Non-N GermanInton G/P HTK (Taylor, 1998) N N Y N Y (Uebler, 1998) Y N N N N (Stemmer, 2001) Y Y N N N (Teixeira, 1996) Y Y N N Y (Hansen, 1995) Y Y Y Y N (Yan and Vaseghi, 2002) N N Y N Y (Jurafsky et al, 1994) Y Y N N N (Berkling et al,1998) Y N N N Y (Oba and Atwell, 2003) Y Y Y Y Y 
The HTK (Hidden Markov Model Toolkit) is a free and portable toolkit for building and manipulating Hidden Markov Models (HMMs) primarily for speech recognition research, although it has been widely used for other topics such as speech synthesis, character recognition and DNA sequencing (HTK3, 2000; Young et al, 2001). Taylor et al (1998) exploited the HTK-based speech recogniser to provide word hypotheses constrained by ‘move types’ correlated with intonation. However, direct analysis of prosody using the HTK has not been the focus of any past research. 
The ISLE project exploited the IHAPI HMM-based speech recogniser to improve the performance of computer-based English learning systems, such as providing clear feedback by specifying error words and phones (Atwell et al, 1999; 2003). The project collected a corpus of audio recordings of 23 Italian, 23 German Spoken Learners’ and 2 native speakers’ English, in which subjects read aloud samples of English text and dialogue selected from typical second language learning exercises, such as pronunciation and stress placement training using minimal pairs and polysyllabic words. (Atwell et al, 2000b; 2003; Menzel et al, 2000). The audio files were time-aligned to graphemic and phonetic transcriptions, and speaker errors were annotated at the word- and the phone-level, to highlight pronunciation errors such as phone realisation problems and misplaced word stress assignments. We re-used the first three blocks of the corpus, Block A through C, which contained 82 sentences edited from ‘The Ascent of Everest’ (Hunt, 1996). As the rest of the corpus generally consisted of shorter sentences or just words without a uniform topic, the first three blocks were the best for prosodic analysis using a speech recogniser. 
Our research analysed German spoken learners’ English prosody re-using the ISLE speech corpus by using the HTK-based speech recogniser. There were three main stages to the research: prosodic annotation of the English text in the corpus, following a model devised for speech synthesis; native speakers’ assessments of the intonation abilities of the 23 German speakers; and speech recognition experiments using the HTK. 
2. Prosodic annotation 
Prosodic annotation was done following the set of instructions or informal algorithm in (Knowles, 1996), to predict 'model’ intonation patterns for written English text, to be passed to a speech synthesiser. The annotation was done to all 27 sentences of Block A of the ISLE corpus. All the tags were added by hand using Windows Excel. The process consisted of four steps: prosodic parsing to divide the text into blocks; assembling blocks to form a potential tone group; adapting lightening rules to merge the groups into an actual tone group; deciding the type of the tone group. The annotation was generally achieved by simple mappings of each step such as from grammatical tags (Atwell et al 2000a) and transition markers to 
592
assembling of tone units. The marked-up prosody was compared with the native speakers’ recordings and some of the patterns were modified. 
The 27 sentences consisted of 429 words and were divided into 84 tone groups: 1 ‘low rise', 3 ‘high rise', 52 ‘fall-rise’ and 28 ‘fall’ patterns. The prosodic annotation produced many ‘fall-rise’ patterns as predicted in the instructions. Tone types of the first 10 sentences, which were used for evaluating German speakers’ intonation abilities, were modified by comparing against 2 native speakers’ recordings. The 10 sentences consisted 157 words retained 15 ‘fall-rise’ and 10 ‘fall’ patterns after canceling 1 ‘low rise', 2 ‘high rise’ and 4 ‘fall-rise’ patterns. This modification was undertaken as we tended to create more tone groups than found in actual native speakers’ utterances. This is due to an ambiguity of the annotation instructions; when it was not clear from the instructions if the two successive blocks should be merged to form a single tone group, we left them as individual tone groups in most such cases. However, the instructions generally required simple mappings such as grammatical tags to the degrees of accentual types, which could be handled by non-linguists. 
We hoped to use the modified prosodic patterns as a 'native speaker target', against which to compare learners’ actual prosodic patterns, so we investigated automated methods to extract prosodic features from the learners’ speech-files. Unfortunately, we were unable to automatically predict markup equivalent to the synthesiser cues, so could not directly compare the learners against this model. 
3. Human evaluation of intonation abilities and grouping of German speakers 
Instead, we turned to expert human evaluation: a computational linguistics researcher and an English language-teaching (ELT) researcher subjectively assessed the intonation of the recorded utterances from German learners of English, by listening to the recorded utterances, comparing against the 'model' marked-up script, and counting perceived intonation errors. The judgments were used to partition the speakers: the speakers were divided into two groups by assigning the upper half of them, who made fewer intonation errors, to a ‘good’ intonation group; and the rest to a ‘poor’ intonation group. Three different groupings were done: two groupings based on each of the two evaluators; and the third based on agreement of the two evaluators. Speakers with exceptionally poor pronunciation as indicated by the ISLE corpus pronunciation markup were excluded in this grouping so that results of the following experiments would be independent from pronunciation ability. 
The two evaluators, one computational linguistics researcher (Evaluator I) and one ELT researcher (Evaluator II), listened to the 10 utterances from each speaker, and compared their prosodic patterns against ‘model’ tone types from the annotation. If the evaluator perceived that all the tone types of each utterance were the same as the model patterns, the utterance was marked as ‘correct'; otherwise ‘error'. We separately counted the number of ‘errors’ for every speaker marked by each evaluator. 
The agreement of two evaluators’ judgments was at about 63 %. Evaluator II's judgment was stricter; this evaluator marked 109 errors out of 230 judgments (10 utterances from 23 speakers), while Evaluator I marked 78 errors. This was probably due to the difference of their judgments norms. Evaluator II mentioned that German speakers tended not to have a clear ‘fall’ in a ‘fall-rise’ pattern. This should result in marking more errors due to ‘fall-rise'; however it can not be seen from the score sheet, as one judgment was made for one utterance. 
As evaluators’ judgments did not agree in some cases, three different groupings were done to 23 German speakers: Grouping I based on Evaluator I, Grouping II based on Evaluator II and Grouping III based on the agreement of the two evaluators. Before the groupings, 3 exceptionally ‘poor’ speakers were eliminated, so that the following HTK experiments should not be affected by pronunciation factors. In Grouping I and II, top 8 and bottom 8 speakers were categorized into ‘good’ and ‘poor’ intonation ability groups leaving 4 intermediate speakers each time. 5 ‘good’ and 7 ‘poor’ speakers were in the same groups in both groupings. In Grouping III, 7 ‘good’ and 7 ‘poor’ speakers were grouped by adding 2 speakers, who were categorized as ‘good’ by one evaluator and ‘intermediate’ by the other, into the 5 agreed ‘good’ speakers. Despite the high rate of disagreement from two evaluators, many speakers were categorized into 
593
the same intonation ability groups. Therefore, it can be said that human evaluation was successful enough. 
4. The HTK speech recognition experiments for analyzing prosody 
Finally, the speech recognition experiments were undertaken using the HTK. Before starting the main experiments for prosodic analysis, we made two preparation experiments: for investigating best training models and several parameters for recognition tests; and for checking the influence of reducing training speakers to the recognition accuracy. In the main experiments, the HTK was used to train monophone and triphone Hidden Markov Models for a 'poor' intonation group and a 'good' intonation group separately. In each case, the training set excluded test speakers and each model was tested with the test speakers from both ‘good’ and ‘poor’ intonation group. This was achieved via cross-validation, repeating the experiment, taking out a different test-subset each time, and averaging the results. The whole process was repeated three times taking a different grouping. Results reveal that recognition accuracy becomes higher when models are trained with a group of same intonation ability as test speakers. Cross-merging experiments confirm that these results are generally consistent. 
Before the main HTK speech recognition experiments for analysing German speakers’ English prosody, the following HTK parameter settings were decided by preparatory HTK experiments: 
• Monophone and 3-mixture word internal triphone HMMs would be trained. 
• No language models would be used for recognition tests. 
• Recognition accuracy was still reasonable, even when the number of training speakers were reduced from 17 to 12 and 6. 
• WIP (Word Insertion Penalty) would be set to -60.0. 
Three main experiments were undertaken using one of the three groupings: Experiment I through III taking Grouping I through III, respectively. The HMMs were trained with 6 ‘good’ and 6 ‘poor’ speakers separately, and each model was tested by the rest of speakers from both groups. This was repeated by taking different sets of test speakers each time. 
In all of the three experiments, recognition accuracy was generally higher except ‘poor’ test speakers against monophone models when training and test speakers’ intonation abilities were the same. Experiment II showed the most significant improvement, whose grouping was based on Evaluator II, who had a stricter norm on ‘fall-rise’ patterns. Average improvement of recognition accuracies by the agreement in triphone cases (with monophone cases) were: 
• Experiment I: 10.13 % (13.20 %) among ‘good’ test speakers 
6.76 % (-2.85 %) among ‘poor’ test speakers 
• Experiment II: 19.20 % (15.43 %) among ‘good’ test speakers 
15.50 % (1.67 %) among ‘poor’ test speakers 
• Experiment III: 17.11 % (16.41 %) among ‘good’ test speakers 
13.14 % (-0.48 %) among ‘poor’ test speakers 
The improvement was lower from ‘poor’ test speakers. This was because of one speaker, SESS0189, who was categorized into a ‘poor’ intonation group by both evaluators, but always had much higher recognition accuracy against models trained by ‘good’ intonation speakers. This must be because this speaker had different intonation error types from the other ‘poor’ speakers, while the rest of ‘poor’ speakers created similar intonation errors. 
Although exceptionally ‘poor’ pronunciation speakers were excluded from the groupings, the following two support experiments gave supporting evidence that the above results were obtained by intonation abilities. These two experiments were done taking Grouping II, which showed the most significant improvement in previous experiments. 
594
We counted correctly recognised ‘keywords’ for tone types. White (2002) found that the locus of accentual lengthening was shown to be the word, with the greatest lengthening tending to be at word edges. We called the word containing the last accented syllable of each tone group a ‘keyword'. Improvement of recognition accuracy among the ‘keywords', especially for ‘fall-rise’ patterns, was higher than that among all the words. This result showed that trained models were clearly distinguished by prosodic features, and ‘poor’ intonation speakers tended to show the difficulties at ‘fall-rise’ patterns as perceived by Evaluator II. 
The other experiment was done taking two ‘worst’ and ‘best’ pronunciation speakers from ‘good’ and ‘poor’ intonation groups, as the former group tended to have slightly better scores on ‘pronunciation’ abilities. This result also showed the improvement when training and test speakers’ intonation abilities agreed. This confirmed the result is not relevant to pronunciation factors. 
Overall, we can conclude that the HTK was able to train clearly different HMMs according to training speakers’ intonation abilities. We found that it was better to use models trained by speakers with the same intonation ability as the test speakers in order to achieve higher recognition accuracy, and that German speakers who showed ‘poor’ English intonation abilities, generally had similar errors. 
5. Conclusions 
Our research focussed on analysing non-native speakers’ prosody using the HTK speech recogniser. As mentioned in the Introduction, the main focus of speech recognition researchers is generally on pronunciation factors, or if prosody is taken into account, they tend to deal with native speakers', that is, this research was unique, and important; we showed that a test German speaker should choose a model trained by other German speakers with the same intonation abilities as the test speaker, in order to obtain higher recognition accuracy. Therefore, it is worth considering intonation abilities for speech recognisers for non-native speakers. 
There has not been previous research which used the HTK speech recogniser to directly analyse prosody. Our work proved that the HTK was able to deal with prosodic factors. The HTK trained two clearly different HMMs: those trained with similar prosodic patterns to native speakers’ ‘model’ patterns; and those trained with common intonation error patterns among German speakers. Speech recognition researchers can deal with prosody using the HTK. 
Our research suggests that foreign language learning software should be able to detect learners’ intonation abilities unlike any existing educational software. The learning tool should contain different models separately trained by ‘good’ and ‘poor’ intonation speakers. By comparing recognition accuracies of ‘keywords’ for prosody against the two models, it should be possible to detect the accuracy of the learner's intonation and to point out intonation patterns where the learner especially showed weakness. 
Fox (1984) and Grabe (1998) compared English and German intonations, and revealed that German language rarely had a similar ‘fall-rise’ pattern to that of English. One of our experiments implied that German speakers with ‘poor’ English intonation tended to have errors at ‘fall-rise’ patterns. German speakers of learning English require intensive practice of the ‘fall-rise’ pattern. 
This research successfully showed that the agreement of training and test speakers’ intonation abilities, ‘good’ or ‘poor', brought about higher recognition accuracy. The intonation abilities were judged at only ‘fall-rise’ and ‘fall’ patterns; however, there are also other tone types, such as ‘rise’ and ‘level'. This suggests that further investigations are required: 
Whether the same grouping would be given when all the tone types were taken into account in human evaluation of German speakers’ English intonation abilities; 
If not, whether the different grouping would also shows the improvement of recognition accuracy by the agreement of intonation abilities. 
595
We also need to consider the diversity of intonation errors. In the HTK experiments, one ‘poor’ intonation speaker showed the opposite result; recognition accuracy was better when models trained by ‘good’ intonation speakers were tested against this speaker. This was probably because this speaker had different types of intonation errors from the rest of ‘poor’ intonation speakers. Speech recognition should deal with this kind of exceptional speaker when it takes account of intonation abilities. 
In this research, only German speakers were considered. It is worth investigating whether the same results would be obtained from other nationalities, and the possibility of adjusting the idea into multilingual speech recognition, in which there should be diversity even within the same intonation group because of influences from different mother languages. 
A significant challenge is to use these results in real language-teaching systems. A lesson from the ISLE project is that theoretical results and practical of these results are quite different achievements! 
We believe the analysis contributes to speech recognition research and foreign language learning technology. Our HTK experiments found better training models for German speakers with ‘good’ and ‘poor’ intonation speakers separately. Its results should help to find an effective way to train English speech recognition systems for German speakers with various English intonation abilities. 
References 
Atwell, E., Dementriou, G., Hughes, J., Schiffrin, A., Souter, C., and Wilcock, S. 2000a A comparative evaluation of modern English corpus grammatical annotation schemes. In: International Computer Archive of Modern and Medieval English (ICAME) Journal, vol.24, pp.7-23. 
Atwell, E., Herron, D., Howarth, P., Morton, R. and Wick, H. 1999 Pronunciation Training: requirements and solutions, Project Report, Interactive Spoken Language Education Project LE4-8353, Deliverable D1.4. Cambridge: Entropic. 
Atwell, E., Howarth, P., and Souter, C. 2003 The ISLE Corpus: Italian and German Spoken Learners’ English In: International Computer Archive of Modern and Medieval English (ICAME) Journal, vol.27. 
Atwell, E., Howarth, P., Souter, C., Baldo, P., Bisiani, R., Pezzotta, D., Bonaventura, P., Menzel, W., Herron, D., Morton, R., and Schmidt, J. 2000b User-Guided System Development in Interactive Spoken Language Education. In: Natural Language Engineering journal, Special Issue on Best Practice in Spoken Language Dialogue Systems Engineering, vol.6 (3-4), pp.229-241. 
BEEP dictionary 1996 [Online]. Available from World Wide Web: 
<http://www-svr.eng.cam.ac.uk/comp.speech/Section1/Lexical/beep.html>. 
Berkling, K., Zissman, M., Vonwiller, J., and Cleirigh, C. 1998 Improving Accent Identification through Knowledge of English Syllable Structure In: Proceedings of the 5th International Conference on Spoken Language Processing (ICSLP-1998), vol.2, 30 November-4 December 1998, Sydney. pp.89-92. 
Brown, G. 1977 Listening to Spoken English. London: Longman. 
Cruttenden, A. 1997 Intonation, 2nd edition. Cambridge: Cambridge University Press. 
Eskenazi, M. 1996 Detection of foreign speakers’ pronunciation errors for second language training In: Proceedings of the 4th International Conference on Spoken Language Processing (ICSLP-1996), vol.3, 3-6 October 1996, Philadelphia. pp.1465-1468. 
Eskenazi, M. 1999 Using Automatic Speech Processing for Foreign Language Pronunciation Tutoring: Some Issues and a Prototype In: Language & Technology (LLT) Journal, vol.2 (2), pp.62-76. 
596
Evermann, G. 2002 HTK History [Online]. Available from World Wide Web: <http://htk.eng.cam.ac.uk/history.shtml>. 
Fox, A. 1984 German Intonation. Oxford: Clarendon Press. 
Grabe, E. 1998 Comparative Intonational Phonology: English and German, PhD thesis. Max-Planck-Institute for Psycholinguistic and University of Nijmegen. 
Hansen, J.H.L. and Arslan, L.M. 1995 Foreign Accent Classification Using Source Generator Based Prosodic Features In: Proceedings of the 1995 International Conference on Acoustic, Speech, and Signal Processing (ICASSP-1995), vol.1, 9-12 May 1995, Detroit. pp.836-839. 
HTK 2000 [Online]. Available from World Wide Web: <http://htk.eng.cam.ac.uk/index.shtml>. 
Hunt, J. 1996 The Ascent of Everest. Stuttgart: Ernst Klett Verlag, English Readers Series. 
Interactive Spoken Language Education Non-Native Speech Data 1999 [CD-ROM]. Cambridge: Entropic. 
Jurafsky, D and Martin, J.H. 2000 Speech and Language Processing. New Jersey: Pearson Higher Education. 
Jurafsky, D., Wooters, C., Tajchman, G., Segel, J., Stolcke, A., Folser, E., and Morgan, N. 1994 The Berkeley Restaurant Project In: Proceedings of the 3nd International Conference on Spoken Language rocessing (ICSLP-1994), September 1994, Yokohama. pp.2139-2142. 
Knowles, G. 1996 From text structure to prosodic structure. In: Knowles, G., Wichman, A. and Alderson, P., (editors). Working with Speech. Harlow: Addison Wesley Longman Limited. pp. 146-167. 
Menzel, W., Atwell, E., Bonaventura, P., Herron, D., Howarth, P., Morton, R., and Souter, C. 2000 The ISLE Corpus of non-native spoken English. In: Gavrilidou, M., Carayannis, G., Markantionatou, S., Piperidis, S., Stainhaouer, G. (editors). Proceedings of 2nd International Conference on Language Resources and Evaluation (LREC 2000), vol.2, 31 May-2 June 2000, Athens. pp.957-964. 
Morton, R. 1999 Recognition of Learner Speech, Project Report, Interactive Spoken Language Education Project LE4-8353, Deliverable D3.3. Cambridge: Entropic. 
Oba, T. and Atwell, E. 2003 Using the HTK Speech Recogniser to analyse prosody in a Corpus of German Spoken Learners’ English. In: Proceedings of the 2003 International Conference on Corpus Linguistics (CL- 2003), 28-31 March 2003, Lancaster. 
O'Connor, J.D. and Arnold, G.F. 1970 Intonation of Colloquial English, 7th edition. London: Longman. 
Rodman, R. D. 1999 Computer Speech Technology. Norwood: Artech House Inc. 
Souter, C., Howarth, P., and Atwell, E. 1999 Speech Data Collection and Annotation, Project Report, Interactive Spoken Language Education Project LE4-8353, Deliverable D3.1. Cambridge: Entropic. 
Stemmer, G., Nöth, E., and Niemann, H. 2001 Acoustic Modeling of Foreign Words in a German Speech Recognition System In: Dalsgaard, P., Lindberg, B., and Benner, H., (editors). Proceedings of the 7th European Conference on Speech Communication and Technology (Eurospeech-2001), vol.4, 3-7 September 2001, Aalborg. pp.2745-2748. 
Taylor, P., King, S., Isard, S. and Wright, H. 1998 Intonation and Dialog Context as Constraints for Speech Recognition. In: Language and Speech, vol.41 (3-4), pp.493-512. 
597
Taylor, P., King, S., Isard, S., Wright, H., and Kowtko, J. 1997 Using Intonation to Constrain Language Models in Speech Recognition. In: Proceedings of the 5th European Conference on Speech Communication and Technology (Eurospeech-1997), vol.5, 22-25 September 1997, Rhodes. pp.2763-2766. 
Teixeira, C., Trancoso, I., and Sarralheiro, A. 1996 Accent Identification. In: Proceedings of the 4th International Conference on Spoken Language Processing (ICSLP-1996), vol.3, 3-6 October 1996, Philadelphia. pp.577-580. 
Tench, P. 1996 The Intonation Systems of English. London: Cassell. 
Thambiratnam, D. 2001 [HTK-Users] Problem with HERest [Online]. 
Available from World Wide Web: 
<http://htk.eng.cam.ac.uk/pipermail/htk-users/2001-August/001145.html>. 
Uebler, U., Schüßler, M., and Niemann, H. 1998 Bilingual and Dialectal Adaptation and Retraining In: Proceedings of the 5th International Conference on Spoken Language Processing (ICSLP-1998), vol.15, 30 November-4 December 1998, Sydney. pp.1815-1818. 
Werner, S and Keller, E. 1994 Prosodic Aspects of Speech. In Keller, E. (editor) Fundamentals of Speech Synthesis and Speech Recognition. Chichester: John Wiley & Sons Ltd. pp.23-40. 
White, L. 2000 English speech timing: a domain and locus approach, PhD thesis. University of Edinburgh. 
Witt, S. and Young, S. 1997 Language Learning Based on Non-Native Speech Recognition In: Proceedings of the 5th European Conference on Speech Communication and Technology (Eurospeech-1997), vol.2, 22-25 September 1997, Rhodes. pp.633-636. 
Woodland, P. 2000 HTK History [Online]. Available from World Wide Web: <http://htk.eng.cam.ac.uk/history.shtml>. 
Yan, Q. and Vaseghi, S. 2002 A Comparative Analysis of UK and US English Accents in Recognition and Synthesis In: Proceedings of the 2002 International Conference on Acoustic, Speech, and Signal Processing (ICASSP-2002), vol.1, 13-17 May 2002, Florida. pp.413-416. 
Young, S., Everman, G., Kershaw, D., Moore, G., Odell, J., Ollason, D., Valtchev, V., and Woodland P. 2001 The HTK Book 3.1. Cambridge: Entropic. 
598

THE METACOMMUNICATIVE SETTING OF PHRASEOLOGICAL UNITS 
AND THEIR MODIFICATIONS – EVIDENCE FROM 
THE BRITISH NATIONAL CORPUS 
Marija Omaziæ, MA. 
Senior language instructor 
Josip Juraj Strossmayer University 
Pedagoški fakultet, Lorenza Jägera 9, 31000 Osijek, Croatia 
momazic@pedos.hr 
Summary 
The study reported in this paper examines the metacommunicative setting of phraseological units and PU modifications in particular, using the corpus of naturally-occurring data in support of statements (BNC). The corpus data indicate that it is not only PUs that are signalled in the context by means of metacommunicative introductions or comments, it also applies to PU modifications. However, the metacommunicative setting used to embed PU modification is shown to have a far more complex function, as it not only serves as an introduction, but pragmatically as an alert to users that they need to make more of an effort to interpret the modified PU. 
1. Introduction 
The study reported here is a part of a more extensive study of the modifications of phraseological units (PUs) in English based on the evidence from the British National Corpus. The searches for the instances of PU modifications in the BNC uncovered an interesting feature of both PUs and PU modifications, i.e. their tendency to occur in discourse in different kinds of metacommunicative settings. This metacommunicative ‘embedding’ is the focal point of the present study. 
2. On the corpus 
All the examples used in this study come from the British National Corpus (100 million words). The version of the BNC used in this study is the British National Corpus World Edition, released in December 2000.1 The software tool used to search the BNC is SARA, version 0.98 (short for SGML Aware Retrieval Application). 
3. Search procedures 
Identifying PUs, and particularly PU modifications, in the corpus as large as the BNC is a rather complex task. The first step was to differentiate between grammatical transformations, phraseological variations, and PU modifications. To clearly define what we mean by ‘modifications‘, we first need to distinguish between phraseological variations, which I believe to be institutionalised (to be as white as a sheet/snow); transformations, which I understand to be grammatical (to sleep/slept like a log/logs); and phraseological modifications (The cloud in this particular silver lining is progestogen, a synthetic form of the hormone progesterone…) which I define as deliberate, creative, innovative, ephemeral, and often idiosyncratic, ad hoc changes of the canonical PU structure and/or meaning that produce different semantic, stylistic or pragmatic effects. Several criteria were used to differentiate between variations and modifications, first of all the frequency of occurrence. If a changed expression is attested only once in the corpus, chances are that it is a case of modification. With multiple hits for a certain changed expression, it is more likely that it is a variation. In unclear cases, phraseological dictionaries were consulted to see if the expression is recorded, and thus institutionalised. Understandingly, there arises the question of canonical form, i.e., how it is decided whether it is a base form or a varied form. In those cases, phraseological dictionary entries were taken as base forms. 
The inventory of different patterns metacommunicative settings was identified during searches for PUs and PU modifications. Those were mostly key-word searches, i.e. searches for particular lexemes that are part of the canonical form of a PU. When a particular pattern was identified, further specific searches for more examples of such patterns were conducted. Examples containing patterns of metacommunicative settings were recorded in a separate database, and later analysed. 
1 For more information, go to: <http://www.hcu.ox.ac.uk/BNC> 
599
4. Corpus findings 
The metacommunicative settings identified in the corpus appear to be of different kinds. They can be in the form of simple metacommunicative lexemes added or inserted into PUs or their modifications (such as proverbial, proverbially, proverbials, idiomatic, idiomatically), metalinguistic introductions preceding PUs or their modifications (remember the old proverb…, as the old saying goes…), or metalinguistic comments following PUs or their modifications (…ran a popular proverb; the usual tired cliché). The functions of the three types of metacommunicative settings are of necessity different, with further differences noted between their use with original and modified PUs. 
4.1. Metacommunicative lexemes 
Let us first look at the instances of what we call extension by metacommunicative lexemes. The most frequently used metacommunicative lexeme is proverbial and its derivatives, followed by idiomatic. 161 instances of proverbial were found in the corpus, of which 97 either preceded or were inserted into a PU or a PU modification. In the following examples proverbial is used to extend the canonical form of a PU: 
  I should also say that these are already selling like the proverbial hot cakes, so maybe I should move swiftly on to the ME-10… 
  He took to it like the proverbial fish to water. 
  Debt has been proverbially frowned on: He that borrows must pay again with shame or loss, He that goes a-borrowing goes a-sorrowing, Out of debt, out of danger; Better go to bed supperless than rise in debt; Let him that sleeps too sound borrow the debtor's pillow; and, but only grudgingly, Debt is better than death. 
In such cases proverbial and proverbially have the function of making a comment on the truth-value of the expression, making it a statement recognized as well-known, often used, or believed by many people. Proverbial thus reinforces the phraseological quality of the expression, marking it as fixed, conventionalized phrase. 
Let us now compare this function with the function of proverbial when used with PU modifications. In all of the following examples, proverbial signals the modified use of a PU. It is used to attract the attention of readers/listeners to the modified phraseological expression, almost as if the authors wanted to make sure that the audience will recognize the expression alluded to, by making it stand out in the context. Arguably, its function is not to reinforce the fixedness of form or meaning as in the examples discussed above, or other phraseological characteristics of a PU for that matter, but rather the absence of them, or departure from them. Evidently, some of the following examples illustrate drastic departures from phraseological convention. 
  An astute businessman and virtual workaholic, he has his finger in more proverbial puddings than Little Jack Horner. 
  The Mitry peasants (like the proverbial cat) could look at a king, and they spoke the same legal language of customary rights and fairness as Charles and his faithful men used in their capitularies. 
  This is not the proverbial free lunch. 
  Uneasily, Montgomery felt that his hand was hovering over the proverbial can of worms; remove the lid and there they would be, writhing in a tortuous, slimy, unholy mass. 
  Taken somewhat aback by the speed and scope of their success, and worried about the proverbial `bursting bubble', they planned no future releases until well into the new year. 
Another plausible explanation for the use of proverbial may be that speakers sometimes feel the need to apologize for using fixed expressions, as noted by Tannen (1982), instead of novel ways of expression. Tannen noted that many Americans, when uttering formulas, make apologies like ‘I know this is a cliché, but…’ or ‘Everyone must say this, but…', or otherwise mark their expressions with verbal or nonverbal equivalents of quotation marks. Presumably, what Tannen referred to are mostly proverbs, sayings, familiar quotations, and some idioms. The corpus provided ample evidence of Tannen's observation: 
  I know it's a cliché but we really do have to take it from game to game it's the only way. 
  But as the cliché goes, it was too little, too late. 
  And, when some other retired persons say in my presence ‘I don't know how I found time for work', I no longer inwardly cringe with scorn at the well worn cliché as I used to do but instead I enthusiastically and of course sagely nod my head in hearty agreement. 
600
A further confirmation of the tendency to embed or precede formulaic language by some sort of a metalinguistic comment comes from Dobrovol'skij and Lubimova's (1993) study of German, only their preceding comments are not apologetic, but introductory in tone. Tannen's examples are more straightforward apologies than just using proverbial, but especially in cases where proverbial is used with the canonical form of PUs, we could say that it functions as an implicit apology for resorting to formulaic language. 
A further distinct use of proverbial noted in the corpus is its occurrence in contexts heavily marked as impolite, slang, offensive, or even plain rude. Proverbial is used to soften the offensive tone of the expression, here clearly functioning as something of an apology. 
  So when the proverbial shit had hit the fan in Whitehall, Connors, or somebody like him, would have drawled at the emergency conference. 
  Getting hold of tickets for the game is like getting hold of the proverbial rocking horse s**t i.e. a bit difficult. 
  The pain in the proverbial ass that hampers an otherwise good idea is Mr Gameplay. 
The next three examples illustrate the very specific use of proverbial and proverbials. Here, they became substitutes functioning as euphemisms for the missing original lexeme that is perceived by users as offensive. The resulting expression is clearly a modification of the original PU created by lexical substitution. It stands in a synonymous relationship with the original PU, but its expressive quality is much more toned down. We could say that proverbial and its derivatives in the following examples also have a signalling function, but such cases it is definitely not their primary one. Moreover, the effect achieved is a humorous one, which is not the case with other uses of proverbial. 
  One word out of line, and the proverbial hits the fan. 
  God, it's cold enough to freeze the proverbials off a brass monkey! 
  Speed (10 credits): If you want a player to move like the proverbial off a shovel, this is the power-up for you. 
The analysis of all examples including proverbial indicates that it tends to be used mostly with idioms of comparison and clipped proverbs and sayings. 
As for proverbially, it occurs less frequently (only 19 hits), and usually with verbal idioms: 
  Personally, I wouldn't trust you as far as I could proverbially throw you! 
  Oh if only the ground would proverbially open and swallow me. 
4.2. Metacommunicative introductions 
Let us now look at more elaborate metacommunicative settings. Our corpus data clearly confirm that it is not only PUs that are signalled in the context by means of metacommunicative introductions and comments, it also applies to PU modifications. The following examples illustrate the practice of introducing PUs, with the introductory statement making a point that the expression that follows is used by others, thus stressing its conventionality and fixedness, as if the speaker is somewhat distancing itself from it. 
  Oh, well, they say every cloud has a silver lining. 
  There is a saying about there being more than one way to skin a cat: in the case of toxic wastes they could be contained in an indefinitely leakproof box. 
The distancing may also occur with metacommunicative introductions that bring in PU modifications. Here, however, introductions have a preparatory function, sensitising users for the modified use, rather than reinforcing the phraseological status of the expression. 
  But that did mean she was also between me and Prentice, who was hovering trying to cut off my retreat as well, so maybe it's true what they say about every silver lining having a cloud. 
  You know what they say – the spice of life, and all that… 
They may also ‘buy time’ for the speaker who aims to modify the canonical form of a PU, i.e. function as slot-fillers. 
  But it's certainly, how can I put it, the silver lining. 
601
In the following examples, metacommunicative introductions unambiguously signal the modified use. In addition, they mark the attitude of the speaker towards the expression he/she is about to modify. Presumably, the very attitude may serve as a reason why the user decides to resort to modifying the expression. 
  However, in order to sway you over to their way of thinking, Peterson might paraphrase the old adage and say: Look after the cents and the dollars will take care of themselves. 
  To paraphrase an old saying, the water has to be there for the horse to be able to drink. 
4.3. Metacommunicative comments 
It is not unusual to find metacommunicative comments following the PU or a PU modification. They may be of two kinds: comments on either the phraseological form, or content: 
  Don't look a gift horse in the mouth, that's what they say innit? 
  What I said was…something quite different… Utterly different… entirely different… As different as chalk from cheese. Although when you come to think of it, cheese isn't all that different from chalk! 
5. Conclusion 
The metacommunicative setting of PUs and PU modifications in discourse is achieved using metacommunicative lexemes, metacommunicative introductions, and metacommunicative comments. The metacommunicative devices identified in this study used with the canonical form of a given PU were shown to perform the following set of functions: 
  they signal the phraseological use; 
  they provide information on the speaker's attitude towards the truth-validity and status of a given PU; 
  they soften the tone of offensive, slang and impolite PUs; 
  they function as implicit or explicit apologies for using formulaic language; 
  offer interpretations of the meaning of used PUs. 
On the other hand, the metacommunicative devices used with the modified PUs, though their forms and functions partly overlap with the ones listed above, have a rather distinct set of functions: 
  they implicitly or explicitly signal the modified phraseological use; 
  they activate the phraseo-lexicon of the user necessary for the decoding; 
  they functions as slot-fillers, buying time for users preparing to modify a PU; 
  metacommunicative lexemes may even function as substitutes for PU constituents perceived as rude or offensive; 
  they provide information on the speaker's attitude towards the truth-validity and status of a PU they are about to or have already modified. 
Bibliography: 
Dobrovol'skij, D., Lubimova, N. 1993 ‘Wie man so schön sagt, kommt das gar nicht in die Tüte.’ Zur metakommunikativen Umrahmung von Idiomen.’ Deutsch als Fremdsprache, 30/3, 151-156. 
Omaziæ, M. 2002 Modifications of phraseological units in English – presentation of a project. In E. Petroviæ (ed.) 25 Years of English Studies in Osijek. Sveuèilište Josipa Jurja Strossmayera, Osijek, 114-148. 
Omaziæ, M. (unpublished dissertation) Modifications of phraseological units in 
English. Zagreb University. 
Tannen, D. 1982 The Oral/Literate Continuum in Discourse. In D. Tannen (ed.) Spoken and Written Language: Exploring Orality and Literacy. Advances in Discourse Processes, vol IX, 1-17. 
602

Corpus linguistics meets language technology: deep syntactic parsing for question answering 
Nelleke Oostdijk 
Department of Language and Speech 
University of Nijmegen, The Netherlands 
n.oostdijk@let.kun.nl 
Abstract 
To the extent that NLP is used by QA systems, it is mostly limited to tokenization, named entity recognition, stemming, POS tagging, and shallow parsing. More sophisticated NLP such as (deep) syntactic parsing is hardly ever used. In the present paper I investigate why this should be the case and try to establish how deep syntactic parsing as developed in the field of corpus linguistics might contribute to improve the overall performance of such systems. 
Keywords: deep syntactic parsing, question answering, corpus linguistics 
1. Introduction 
The amount of information on the Internet is vast and continues to grow steadily. For users to successfully explore the Internet it is essential that they can gain effective access to the information. With the present search engines, it is possible – in response to a user's query – to retrieve a set of documents that are likely to contain relevant information. Although users have learned to make do with the present systems, they are quite some distance removed from what users actually want. Ideally, it should be possible to formulate questions or some other forms of requests for information in the same fashion as one does in human communication, and either retrieve a set of documents or obtain a direct answer, whichever is preferred. 
The organizers of the Text REtrieval Conferences (TREC) in 1999 set up a Question Answering (QA) Track to encourage research in this field. Research groups were invited to compete on the following task: given a set of fact-based questions and a collection of newspaper/newswire texts, return a ranked list of [document, answer string] pairs. Answer strings were limited to 50 or 250 bytes and the document had to support the answer (cf. Voorhees 2000). Systems that participated all had more or less the same architecture (cf. de Boni 2001): they created a query from the user's question, used information retrieval to locate (segments of) documents likely to contain an answer, and then pinpointed the most likely answer passage within the candidate documents (cf. Hovy et al. 2001: 339). 
Since the first QA track was organized (TREC-8), it has taken on the form of a yearly event which attracts more participants each year. Over the years the task has become more realistic. Whereas the early QA tracks included a single task, in TREC-10 three separate tasks were defined: the main task, the list task and the context task (cf. Voorhees 2002). The main task was essentially the same task as it was set previously, the difference being that it was no longer guaranteed that the text collection contained the answer to a given question. In last year's TREC, the difficulty of the task was further increased as questions now required an exact answer (a noun or a noun phrase) instead of a text snippet containing the answer. It is expected that future TREC QA tracks will increase the kinds and difficulty of the questions that systems are expected to handle. This is in line with the goal of the ARDA/AQUAINT programme which “intends to address a scenario in which multiple inter-related questions are asked in a focused topic area by a skilled, professional information analyst who is attempting to respond to larger, more complex information needs or requirements.” (http://www.ic-arda.org/ InfoExploit/aquaint/) 
Throughout the various TRECs the basic strategy of most systems has remained the same. However, as Voorhees (2002) observes in her overview of the TREC2001 QA track, “there was much less agreement on the best approaches to realizing that strategy. ” She summarizes the approaches taken by the participants in TREC-10 as follows, identifying two areas in which they divert: 
Many groups continued to build systems that attempt a full understanding of the question, but increasingly many groups took a more shallow, data-driven approach. The data-driven approaches rely on simpler pattern matching methods using very large corpora (frequently the web) rather than sophisticated language processing. The idea 
603
exploited in the massive data approach is that in a large enough data source a correct answer will usually be repeated often enough to distinguish it from the noise that happens to occasionally match simple patterns. 
A second area in which there is no consensus as to the best approach is classification schemes for answer types. Some systems use a few very broad classes of answer types, while others use many specialized classes. The difference is a standard trade-off between coverage and accuracy. With many specialized answer types, finding the actual answer once an answer type is correctly classified is much easier because of the specificity of the class. However, deciding which class is correct, and ensuring there is a class for all questions, is more difficult with many specialized classes. 
In the present paper I investigate the role of natural language processing (NLP) in QA systems and try to establish how deep syntactic parsing as developed in the field of corpus linguistics might contribute to improve the overall performance of such systems. 
2. NLP in QA systems 
Although an evaluation of the respective contributions made by NLP components to the relative performance of systems is impossible – in the published reports and papers generally no detailed information is given about how exactly systems work and in-depth analyses of the performance are lacking (cf. De Boni 2002) – an inventory of the types of NLP components used in the various systems participating in the TREC QA track brings to light that to the extent that NLP is used by most systems it is mostly limited to tokenization, named entity recognition, stemming, POS tagging, and shallow parsing. So far, more advanced NLP such as (deep) syntactic parsing is hardly ever used. Various explanations have been given why this should be the case. Apart from the argument put forward by Voorhees viz. “that in a large enough data source a correct answer will usually be repeated often enough to distinguish it from the noise that happens to occasionally match simple patterns”, other arguments have been raised against the use of syntactic parsing in the context of QA. Thus Jurafsky and Martin (2000: 573) observe that analyses obtained through syntactic parsing are “often not particularly well-suited for the task of compositional semantic analysis”. The structures yielded by conventional parsers suffer from the fact that (1) key semantic elements are often widely distributed across parse trees, thus complicating the composition of the meaning representation; (2) many syntactically motivated constituents play essentially no role in semantic processing; and (3) the general nature of many syntactic constituents results in semantic attachments that create nearly vacuous meaning representations (cf. Jurafsky and Martin 2000: 573-574). While in this interpretation the mismatch between syntactic structure and semantic analysis must essentially be attributed to the nature of syntactic analysis, it fails to recognize that this is not an inherent quality of syntax, but rather the consequence of an unfortunate choice of descriptive framework and possibly the inadequacy of the representation of the analysis. Another factor that has played a role in the limited use of syntactic parsing is the fact that the coverage of parsers generally has been insufficient and parsers on the whole have appeared to be not robust enough to deal with text in an unrestricted domain. This suggests that for syntax to play a more than trivial role, it is essential that the parser's coverage and robustness are extended. 
A review of various question answering systems that have participated in the TREC QA track shows that there are two obvious points where it is advantageous to have access to syntactic information, viz. in the analysis of the question and in pinpointing the answer in the ranked texts/segments. “Finding the expected answer type of a natural language question by relying only on the semantics of the question stem (i.e. What, How), and some bag-of-words approaches is not always possible, since stems are associated with many different types of answers and shallow syntax (e.g. phrasal parsers) fails to find the most discriminating concept in the question. The syntactic dependencies between the question phrases help solve this ambiguity: the answer type is indicated by the question phrase most connected to other concepts.” (Harabagiu et al. 2001). Syntactic parsing proves even more valuable as requests for information are not just expressed in the terms of wh-questions, but may take on a range of syntactic forms (incl. declarative and imperative sentences). 
As for pinpointing the answer, a syntactic analysis of texts or segments likely to contain the answer would help overcome some of the limitations inherent in methods relying solely on word level information. So far many systems employ some form of window-based scoring method for determining which answer string is most salient. A 50 or 250 byte window is moved across the texts/segments holding the candidate answers. At each position a score is computed on the basis of the presence of desirable words. The window at the position giving the highest total score is returned as most likely to contain the expected answer (cf. Hovy et al. 2001). One of the drawbacks of this method is that – because of the sole reliance on information at the word level – it is not possible to recognize 
604
information of the desired type (such as Person or Location). Given the question What does the Peugeot company manufacture? the XEROX system (Hull 2000) on the basis of the sentence Mr Longuet said Peugeot's decision to stop manufacturing bodies for its 504 utility vehicle from the end of November left the fate of the company in Renault's hands. returned the following set of possible answers: {Mr Longuet, decision, bodies, 504 utility vehicle, end, November, fate, Renault, hands}. Had the system had access to syntactic information such as that the wh-element in the question refers to the direct object of manufacture, the return of the answer bodies for its 504 utility vehicle would have been straightforward. Another problem with the window-based scoring method is that it cannot pinpoint the answer boundaries precisely. This explains the awkward truncation of the answer strings returned. For example, a typical answer returned by one of the systems to the question What is the name of the rare neurological disorder with symptoms such as: involuntary movements (tics), swearing, and incoherent vocalizations (grunts, shouts, etc.)? is as follows: who said she has both Tourette's Syndrome and. Syntactic parsing would be helpful in identifying Tourette's Syndrome as the exact answer. 
In the light of what was said and argued above, there is no reason to believe that syntactic parsing cannot play a role in question answering. It can, not as a substitute for a lexical probabilistic analysis, but rather in aid of it. The relevant question then is: What information exactly should syntactic parsing provide? and, related to this question, What requirements does this impose on the parser? In the next sections I investigate to what extent a wide-coverage parser such as the current TOSCA parser that was originally developed in the field of corpus linguistics can be of use or what adaptations are necessary. 
3. The TOSCA parser 
The current TOSCA parser is the latest in a line of parsers that I developed previously for the linguistic annotation of corpora and which were used in a number of projects including the TOSCA (Tools for Syntactic Corpus Analysis; Oostdijk 1991) and the ICE (International Corpus of English) projects (Greenbaum 1992; Oostdijk 2000). The role of the parser in these projects was two-fold: (1) it served to test the linguistic hypotheses in the underlying formal grammar against the language data in the corpus; and (2) it was instrumental in annotating the corpus. The annotated corpus in its turn was used as input for research in English descriptive linguistics. In view of its role and the intended use of the annotated corpus, the parser was designed to yield for each input string minimally the correct parse in a given context. Obtaining the correct parse at the risk of over generating thus prevailed over minimizing the amount of ambiguity. The parser was one of the key components of the TOSCA analysis system (Aarts et al. 1998), where it would take as input the (manually corrected) result of the part-of-speech tagging. Through manual intervention the one correct parse was selected from possibly multiple parses. 
The parser can best be characterized as a linguistically motivated, rule-based deep syntactic parser that has been developed for use in an unrestricted domain. The descriptive framework that is used in the underlying grammar is based on immediate constituency and the rank hierarchy, and in the analyses constituents are labelled for both the functional role and the categorical realization. See, for example, Figure 1 which shows the analysis that is obtained for the utterance Mr Longuet said Peugeot's decision to stop manufacturing bodies for its 504 utility vehicle from the end of November left the fate of the company in Renault's hands. 1 
The present implementation does not require (corrected) part-of-speech tagged input, but in principle can handle raw text input. The rate of ambiguity has been sharply reduced mostly through the incorporation of additional linguistic information that was acquired over the years; developments in the field of AGFL (Affix Grammars over Finite Lattices; Nederhof and Koster 1993) have increased overall efficiency. Moreover, the current implementation of the AGFL formalism makes it possible to include a number of heuristics that can be used for handling unknown lexical items and structures. 
Application of this parser in the field of QA throws up questions such as ‘What syntactic relations are relevant for retrieving information content?', ‘To what extent is parsing accuracy required for the task?', ‘To what extent should the parser be robust?, ‘Can relations and/or attachments be left unspecified/underspecified?’ and ‘To what extent is a constituency analysis adequate for the present task? I will turn to each of these questions below. 
1 For an explanation of the abbreviations used, see the Appendix. 
605
-:UTT(utt_cat_prop, prop_cat_s) 
PROP:S(s_type_declarative, complementation_motr, finiteness_past, mood_indic, voice_active) 
SU:NP(number_sing, person_third, nppo_cat_none) 
NPHD:N(n_type_proper, n_class_other, number_sing) "Mr Longuet" 
V:VP(cat_operator_no_cat_op, complementation_motr, finiteness_past, mood_indic, voice_active, number_sing, person_third) 
MVB:LV(complementation_motr, finiteness_past, mood_indic, number_sing, person_third) "said" 
OD:CL(cl_type_zero_subordinate, complementation_motr, finiteness_past, mood_indic, voice_active) 
SU:NP(number_sing, person_third, nppo_cat_none) 
DT:DTP 
DTCE:NP(NUMBER, person_third, nppo_cat_none) 
NPHD:N(n_type_proper, n_class_other, number_sing, case_gen) "Peugeot's" 
NPHD:N(n_type_common, n_class_other, number_sing) "decision" 
NPPO:CL(cl_type_zero, complementation_motr, finiteness_infin, mood_indic, voice_active) 
TO:PRTCL(prtcl_type_to) "to" 
V:VP(cat_operator_no_cat_op, complementation_motr, finiteness_infin, mood_indic, voice_active, NUMBER, PERSON) 
MVB:LV(complementation_motr, finiteness_infin, mood_indic, NUMBER, PERSON) "stop" 
OD:CL(cl_type_zero_subordinate, complementation_motr, finiteness_prespart, mood_indic, voice_active) 
V:VP(cat_operator_no_cat_op, complementation_motr, finiteness_prespart, mood_indic, voice_active, NUMBER, PERSON) 
MVB:LV(complementation_motr, finiteness_prespart, mood_indic, NUMBER, PERSON) "manufacturing" 
OD:NP(number_plu, person_third, nppo_cat_none) 
NPHD:N(n_type_common, n_class_other, number_plu) "bodies" 
NPPO:PP(prep_wrd_other) 
P:PREP(prep_wrd_other) "for" 
PC:NP(number_sing, person_third, nppo_cat_none) 
DT:DTP 
DTCE:PN(pn_type_possessive, function_pn_dt, NUMBER, person_third) "its" 
NPHD:N(n_type_common, n_class_other, number_sing) "504 utility vehicle" 
A:PP(prep_wrd_from) 
P:PREP(prep_wrd_from) "from" 
PC:NP(number_sing, person_third, nppo_cat_ajp) 
DT:DTP 
DTCE:ART(NUMBER, definiteness_def) "the" 
NPHD:N(n_type_common, n_class_other, number_sing) "end" 
NPPO:PP(prep_wrd_of) 
P:PREP(prep_wrd_of) "of" 
PC:NP(number_sing, person_third, nppo_cat_none) 
NPHD:N(n_type_proper, n_class_time, number_sing) "November" 
V:VP(cat_operator_no_cat_op, complementation_motr, finiteness_past, mood_indic, voice_active, number_sing, person_third) 
MVB:LV(complementation_motr, finiteness_past, mood_indic, number_sing, person_third) "left" 
OD:NP(number_sing, person_third, nppo_cat_pp) 
DT:DTP 
DTCE:ART(NUMBER, definiteness_def) "the" 
NPHD:N(n_type_common, n_class_other, number_sing) "fate" 
NPPO:PP(prep_wrd_of) 
P:PREP(prep_wrd_of) "of" 
PC:NP(number_sing, person_third, nppo_cat_none) 
DT:DTP 
DTCE:ART(NUMBER, definiteness_def) "the" 
NPHD:N(n_type_common, n_class_other, number_sing) "company" 
A:PP(prep_wrd_other) 
P:PREP(prep_wrd_other) "in" 
PC:NP(number_sing, person_third, nppo_cat_none) 
DT:DTP 
DTCE:NP(NUMBER, person_third, nppo_cat_none) 
NPHD:N(n_type_proper, n_class_other, number_sing, case_gen) "Renault's" 
NPHD:N(n_type_common, n_class_other, number_plu) "hands" 
PUNC:PM(punc_type_per) "." 
Figure 1. Analysis for Mr Longuet said Peugeot's decision to stop manufacturing bodies for its 504 utility vehicle from the end of November left the fate of the company in Renault's hands . 
606
4. Deep syntactic parsing for QA 
One of the strong points in the syntactic analyses produced by the TOSCA parser is that a descriptive framework is used which is based on immediate constituency and the rank hierarchy. Where a shallow analysis essentially identifies simple phrases in the input and does nothing to structure the analysis result, the TOSCA type of analysis structures the information in such a fashion that it is immediately apparent what information belongs together and should be considered at any one time. In this respect the distinction between clauses and phrases and between phrases and words appears to be essential. Compare Figures 2 and 3. Figure 2 exemplifies part of a shallow analysis for the sentence Mr Longuet said Peugeot's decision to stop manufacturing bodies for its 504 utility vehicle from the end of November left the fate of the company in Renault's hands. In this type of analysis typically there is no embedding and no attachment of the prepositional phrases. 
(Part of) a deep syntactic analysis for the same sentence is exemplified in Figure 3 (cf. Figure 1). Although a mapping of the syntactic functions onto semantic roles is not entirely trivial, it is feasible to implement an algorithm by means of which this type of syntactic structure can be mapped onto a semantic representation. Syntactic function labels for constituents such as subject, object and adverbial in combination with additional information that is available regarding the type of clause (e.g. active declarative sentence with unmarked word order) and the type of verb (e.g. action verb, event verb) are useful in identifying the agent, recipient, location, etc. 
While clausal functions play a crucial role in the mapping of the syntactic structure onto a semantic representation, phrasal functions such as head and modifier are useful since they help identify the key element in a phrase (cf. Figure 4). 

Figure 3. Hierarchical structure showing a (clausal) analysis based on immediate constituency and the rank hierarchy 

Figure 2. Shallow analysis showing flat/linear structure 

NP 

VP 

NP 

NP 

PP 

VP 

VP 

PP 

P 

UTT:CL 

SU:NP 

V:VP 

OD:CL 

SU:NP 

V:VP 

OD:NP 

A:PP 

SU:NP 

DT:DTP 

NPHD:N 

NPPO:CL 

Figure 4. Phrasal structure 
607
5. Adaptation of the TOSCA parser for use in QA 
Having argued above that there is a role for syntactic analysis in QA and deep syntactic parsing can contribute to a better performance of QA systems, this section more specifically addresses the question What adaptations are necessary in order to make the TOSCA parser optimally suited for use in QA? The discussion focuses on the following points: coverage, accuracy, ambiguity, and robustness. 
Coverage 
Since the parser will be used to parse the questions and the ranked paragraphs/segments containing candidate answer strings, wide coverage is required. The TOSCA grammar already covers most syntactic constructs, including structures that are highly marked and/or relatively infrequent, so that in this respect few extentions are necessary. There are two areas, however, where adaptation of the parser in the light of the QA task is definitely required. Thus, additional rules are necessary for the adequate analysis of non-sentential constructions (incl. headlines and enumerations) that are characteristic of newspaper/newswire text. At present coverage is also poor in the area of named entities. The parser lexicon contains only a limited set of proper names and holds no information as to which items denote for example currencies and weights. In addition, the grammar does very little in the way of incorporating rules that describe the patterns displayed in names, dates, addresses, amounts and such like items. 
Accuracy and ambiguity 
Irrespective of whether the parser is used for the annotation of corpora with an eye to the advancement of linguistic description or the parser is used in an application such as a QA system, the accuracy should be as high as possible. While I agree with Hovy et al. (2001:341) that “parsing accuracy is particularly important for question parsing, because for only one question there may be several answers in a large document collection”, I also think that parsing accuracy is quite important too for parsing the segments containing the candidate answer strings, especially when exact answers are required instead of text snippets containing the answer. In so far as the TOSCA parser at present yields ambiguous analyses for certain input strings, it is desirable to incorporate additional information in the parser lexicon from lexical resources such as WordNet (Fellbaum 1998) and FrameNet (http://www.icsi.berkeley.edu/~framenet). This includes semantic-syntactic information about collocations and subcategorization which will help to steer the parsing process and further reduce the degree of ambiguity. For those inputs where the parse result still remains ambiguous experiments are necessary to investigate the effect of using whatever parse the system generates first (instead of the one correct parse) and also the effect of collapsing the ambiguous parse trees into one while leaving the constituent boundaries and/or labels under specified. 
Robustness 
More than in the context of the linguistic annotation of corpora, a parser that is used with a QA system must be robust. When used for the annotation of corpora the performance of the parser is usually monitored by the linguist and failure to analyse a given input is taken as an indication that the input must be closely studied and possibly the lexicon and/or grammar adapted. For QA systems parse failures are unacceptable as they tend to block the (automatic) retrieval process. 
6. Conclusion 
In the present paper I have addressed the question as to whether and, if so, how deep syntactic parsing as developed for the linguistic annotation of corpora might contribute to improve the overall performance of question answering systems. I have argued that a wide-coverage deep linguistic parser such as the TOSCA parser can contribute, especially when it comes to analyzing the question and pinpointing the answer. Adaptations that are necessary so as to make the parser better suited for use in QA must be aimed at improving the recognition of named entities and non-sentential constructions, and increasing robustness. Further research is necessary to determine how best to strike a balance between granularity and ambiguity in the analyses. 
Acknowledgement 
Thanks are due to Lou Boves for his comments on an earlier version of this paper. 
608
References 
Aarts J, van Halteren H, Oostdijk N 1998 The linguistic annotation of corpora: The TOSCA Analysis System. International Journal of Corpus Linguistics, 3(2): 189-210. 
ARDA/AQUAINT http://www.ic-arda.org/InfoExploit/aquaint/ 
Atwell E 1996 Comparative evaluation of grammatical annotation models. In Sutcliffe R, Koch H, McElligott A (eds), Industrial parsing of software manuals. Amsterdam, Rodopi, pp 13-23. 
de Boni M 2001 Question answering system architecture. 
http://www-users.cs.york.ac.uk/~mdeboni/research/general/general_qa_system.html 
de Boni M 2002 QA task Overview. http://www-users.cs.york.ac.uk/~mdeboni/research/general/ trec2002.html 
Greenbaum S 1992 A new corpus of English: ICE. In Svartvik J (ed.), Directions in corpus linguistics. Proceedings of Nobel Symposium 82 Stockholm, 4-8 August 1991. Berlin, New York, Mouton de Gruyter, pp 171-179. 
Fellbaum C (ed.) 1998 WordNet: An electronic lexical database. Cambridge, Mass., MIT Press. 
FrameNet. http://www.icsi.berkeley.edu/~framenet. 
Harabagiu S, Moldovan D, Paºca M, Mihalcea R, Surdeanu M, Bunescu R, Gîrju R, Rus V, Morãrescu P 2001 Falcon: boosting knowlegde for answer engines. 
http://trec.nist.gov/pubs/trec9/t9_proceedings.html 
Hovy E, Gerber L, Hermjakob U, Lin C, Ravichandran D 2001 Towards semantics-based answer pinpointing. In Proceedings of HLT 2001. First International Conference on Human Language Technology Research. San Francisco, Morgan Kaufmann, pp 339-345. 
Hull D 2000 XEROX TREC-8 Question Answering Track Report. 
http://trec.nist.gov/pubs/trec8/t8_proceedings.html 
Jurafsky D, Martin J (eds.) 2000 Speech and language processing. An introduction to natural language processing, computational linguistics, and speech recognition. Upper Saddle River, NJ, Prentice Hall, Pearson. 
Lin D 1996 Dependency-based parser evaluation. A study with a software manual corpus. In Sutcliffe R, Koch H, McElligott A (eds.), Industrial parsing of software manuals. Amsterdam, Rodopi, pp 25-45. 
Nederhof M, Koster C 1993 A customized grammar workbench. In Aarts J, de Haan P, Oostdijk N (eds.), English language corpora: design, analysis and exploitation. Amsterdam, Rodopi, pp 145-161. 
Oostdijk N 1991 Corpus linguistics and the automatic analysis of English. Amsterdam: Rodopi. 
Oostdijk N 1996 Using the TOSCA analysis system to analyse a software manual corpus. In Sutcliffe R, Koch H, McElligott A (eds.), Industrial parsing of software manuals. Amsterdam, Rodopi, pp 179-206. 
Oostdijk N 2000 Corpus-based English linguistics at a crossroads. English Studies 81(2): 127-141. 
TREC – Text REtrieval Conferences. http://trec.nist.gov/ 
Voorhees E 2000 The TREC-8 Question Answering Track Report. 
http://trec.nist.gov/pubs/trec8/t8_proceedings.html 
Voorhees E 2002 The TREC-10 Question Answering Track Report. 
http://trec.nist.gov/pubs/trec10/t10_proceedings.html 
Appendix 
In the example analyses abbreviations were used to denote function and categories. They are explained below. 
Function labels: Category labels: 
A adverbial ART article 
DT determiner CL clause 
DTCE central determiner DTP determiner phrase 
MVB main verb LV lexical verb 
NPHD head of noun phrase N noun 
NPPO noun phrase postmodifier NP noun phrase 
OD direct object PM punctuation mark 
P preposition PN pronoun 
PC prepositional complement PP prepositional phrase 
PROP proposition PREP preposition 
609
PUNC punctuation PRTCL particle 
SU subject S sentence 
TO to infinitive marker UTT utterance 
V verb VP verb phrase 
610

Extending computer-assisted text analysis techniques to the detection of source code plagiarism and collusion: assisting manual inspection 
Maeve Paris 
School of Computing and Intelligent Systems 
University of Ulster 
1.0 Introduction 
While there have always been opportunities for collusion among students, the Internet explosion has facilitated the ease with which material can be copied and pasted. Fortunately, the academic sleuth has access to a wide range of publicly-available detection software and systems. Unfortunately, these products vary greatly in terms of functionality, usability and effectiveness. 
Traditionally, a distinction has been drawn between software and services to detect text-based plagiarism or collusion, and products to detect such practices in computer programs. The Joint Information Services Committee (JISC) even commissioned two separate studies, the Technical Review of Plagiarism Detection Report (Bull et al, 2001) which focused on text-based assignments, and Source Code Plagiarism in UK HE Computing Schools, Issues, Attitudes and Tools (Culwin et al, 2001), which concentrated on source code plagiarism. The text-based survey evaluated the performance of five products with mixed results, while the source code survey evaluated two products, with different strengths and weaknesses. This separation of concerns is not spelled out explicitly in either report: both seem to be based on an assumption that the practice of plagiarism and collusion differs between text-based documents and examples of computer programs written in a particular programming language. 
This paper argues for a change of focus: a computer programming language should be treated in similar fashion to any natural language. While the syntax and semantics of programming languages are more formal and restricted than those of natural languages, it is still possible to distinguish a programming (and hence a programmer's) style, much in the same way as one can attempt to distinguish a writing style in a written text or corpus. It could follow that source code samples might be suitable for the application of computer-assisted text analysis techniques. Concordances, the use of KWIC indexes, and other statistical methods from the domain of computational linguistics can all be employed to assist the academic in a computing school in identifying instances of plagiarism or collusion among samples of students’ source code. 
This paper will define source code with examples from different programmers, with a view to illustrating the existence of programming styles, and identify shortcomings in existing detection techniques. It will indicate how computer-assisted text analysis might assist the academic in manual inspection of students’ source code, by augmenting human expertise. An analysis based on authentic examples from the Java programming language will test the suitability of a concordance programme in assisting an academic. The study will attempt to ascertain whether the use of a concordance programme will accelerate the process of plagiarism and collusion detection, and possibly even make it more accurate. 
2. 0 Plagiarism and collusion 
For teaching and assessment purposes, a distinction can be drawn between the offences of plagiarism and collusion. Plagiarism is where a student submits work for assessment which has been borrowed from another writer but fails to indicate (whether by the use of quotation marks or explicit referencing) which sections or phrases have been borrowed. Collusion is usually the manifestation of a joint effort between students where the intention is to mislead the assessor in identifying the person responsible for writing the material. Whether it is a case of reproducing someone's work or producing joint work under different names, the omission of any acknowledgement is of most concern. 
While the aim of this study is to consider methods for detecting plagiarism and collusion, this does not extend to the domain of authorship attribution (determining the author of a piece of work). This is distinct from plagiarism and collusion detection: ‘plagiarism detection attempts to detect the similarity between two substantially different pieces of work but is unable to determine if they were produced by the same author’ (deVel et al, 2001). However, these fields overlap, since both authorship analysis and plagiarism detection are concerned with similarity detection, which ‘calculates the degree of similarity between two or more pieces of work without necessarily identifying the authors’ (deVel et al, 2001). 
611
3.0 Source code 
Source code refers to the set of programming statements written by a programmer in a particular programming language. This code is then compiled into object code: ‘source code and object code refer to the “before” and “after” versions of a computer program that is compiled before it is ready to run in a computer’ (SearchWebServices.com, 2001). 
A Java programmer, for example, types a set of statements in the Java programming language into a visual development environment (or a simple text editor). These statements are then saved into a file, the source file. In the case of Java, the source file is then compiled into instructions that the Java Virtual Machine (Java VM) can understand, in the shape of a bytecode file. It is this program which can be run in the computer. 
Figure 1 is a sample of Java source code which most learners encounter. This program outputs the phrase ‘Hello World!’ to the screen. 
/** 
* The HelloWorldApp class implements an application that 
* displays "Hello World!" to the standard output. 
*/ 
public class HelloWorldApp { 
public static void main(String[] args) { 
// Display "Hello World!" 
System.out.println("Hello World!"); 
} 
} 
Figure 1: Hello World! 
Even this trivial example highlights some characteristics of source code which are of interest to this study: different ways to comment on code (using //… or /*…*/), indentation (through the placing of { and }, for example), naming schemes (the class is named HelloWorldApp), and system output (seen through the print instruction). 
3.1 Source code as a means of expression 
Programming languages can be categorised according to generation (from first generation machine language to fourth generation SQL), or type (such as procedural or declarative), but all languages typically consist of a vocabulary and syntax. While these may be restricted in nature, they can be treated as a form of language from a linguistic perspective. 
To illustrate this claim, consider the examples in Figures 2, 3 and 4 below. All are suggestions as to how to write a function that prints ‘Hello World!’ 100 times: 
for(int j=0; j<100; j++) { system.out.println(printFunction()); } //this function print Hello World 100 times without using for loop and recursion... public string void printFunction() { return("Hello World"); } 
Figure 2: Java program 
public class foo { int i = 0; public void bar() { System.out.println("Hello World"); if(++i < 100)baz(); 
612
} public void baz() { System.out.println("Hello World"); if(++i < 100)bar(); } } 
Figure 3: Java program 
String[] strArray = new String[100]; java.util.Arrays.fill(strArray, "Hello World"); String str = java.util.Arrays.asList(strArray).toString(); str = str.substring(1, str.length()-1); str = str.replace(',', '\n'); System.out.println(str); 
Figure 4: Java program 
All of these programs have the same functionality; they all have the same outputs, but all solve the problem differently. Different algorithms are used, and different styles are used. The first example uses a for loop, the second uses an if loop, the third tries to avoid loops and recursion altogether. Only the first uses comments, while only the second uses indentation to enhance readability. The second also makes greater use of white space to separate off lines of code. There are different variables used and different method/ class names. From this small example, it is possible to see how the same functionality can be implemented differently by different programmers in the same language. 
These statements provide instructions for the computer to carry out a particular function, broken down into different sections and sequences, some of which are conditional. While the programming language provides a limited vocabulary and syntax, the individual programmer has a lot of flexibility in determining how the functionality will be achieved, from choice of algorithm (how the function is achieved), to naming of variables and classes, to layout on the screen (use of spacing, indentation, whitespace, and so on). There is considerable flexibility in the choice of variables, expressions, statements and blocks, and control flow statements. 
Sallis et al (1996) identified this flexibility in the context of authorship attribution: ‘the stylistic influence of an individual on algorithm implementation within the constraints of a given programming language is limited but can be identified to some extent as traits or tendencies in the expression of logic constructs, data structure definition, variable and constant names and calls to fixed and temporary data sets.’ (Sallis et al, 1996). More recently, de Vel et al (2001) asserted that ‘it is possible to identify the author of a section of program code in a similar way that linguistic evidence can be used for categorising the authors of free text’ (deVel et al, 2001). 
The issue of source code as a means of expression has also come to the fore in the United States through the Digital Millennium Copyright Act (DCMA), which was intended to protect publishers from electronic piracy. One major court case has raised the question whether code can be defined as speech. In January 2000, the Motion Picture Association of America filed a lawsuit against the magazine 2600 based on the charge that 2600 had violated the DCMA by publishing the DeCSS decryption routine, a code for decrypting DVDs. A computer scientist at Carnegie Mellon, David Touretsky, gave expert evidence arguing that computer code has expressive content which can convey ideas, like other forms of speech. Judge Kaplan agreed that ‘both source and object code have expressive content, and thus deserve First Amendment protection. Code really is speech’ (Touretsky, 2001). 
4.0 Existing detection systems 
Source code plagiarism detection systems are generally based on metrics and aim to produce a measurement which quantifies the closeness of two programs. Metrics (both software metrics and linguistic metrics) can be gathered on a range of items, including the number of each type of data 
613
structure, the cyclomatic complexity of the control flow of the program, the quantity and quality of comments, the types of variable names chosen, and the use of layout conventions. All of these can assist in building a profile of a particular authoring style (Gray et al, 1997). The JISC report (Culwin et al, 2001) distinguished between early systems based on attribute counting, and those based on structure metrics, on which most modern systems are based and which produce more effective results. 
Jones (2000) characterised plagiarism detection as a pattern analysis problem, where plagiarising transformations have been applied to a source file. He identified the following transformations: 
• Verbatim copying 
• Changing comments 
• Changing white space and formatting 
• Renaming identifiers 
• Reordering code blocks 
• Reordering statements within code blocks 
• Changing the order of operands/operators in expressions. 
• Changing data types. 
• Adding redundant statements or variables. 
• Replacing control structures with equivalent structures. 
(Jones, 2000) 
Most contemporary detection systems adopt a lexical-structural approach to identify these transformations: source programs are tokenised, and profiles are created and compared. While some academic institutions have developed their own in-house detections systems, such as Big Brother (Irving, 2002), there are also services available through a Web interface. The main players in this field are sim (Software Similarity Tester), YAP (Yet Another Plague), MOSS (Measure of Software Similarity) and JPLAG. The sim system (Gitchell and Tran, 1999) tokenises source programs and compares strings using pattern-matching algorithms based on work from the human genome project. 
The YAP (Wise, 1996) approach also tokenises source programs but only retains those tokens which are concerned with the structure of the program. This is based on a lexicon which is created specifically for each programming language. The output is a numeric profile which computes the closeness between two programs. This closeness between programs is partly a function of the programming language chosen and the type of task undertaken (for instance, the COBOL programming language is by nature a highly structured and verbose language, and can lead to very similar programs, likewise with Visual Basic). 
MOSS (MOSS, 2002) can be applied to a range of programming languages. Registered instructors can submit batches of programs to the MOSS server, and results are returned to a website. Little information is available on how the tool works (presumably because if this were known, it would be possible to evade detection), but it is based on the syntax or structure of a program, rather than the algorithms which drive the program (Stutz, 1998). The MOSS database stores an internal representation of programs, and then looks for similarities between them. 
JPLAG, on the other hand, compares submitted programs in pairs, and is based on the assumption that plagiarists may vary the names of variables or classes, but they are least likely to change the control structure of a program. The algorithm employed is available as a technical report (Prechelt et al, 2000). 
The performance of both MOSS and JPLAG were evaluated in the JISC report (Culwin et al, 2001). The survey concluded that there was not much consensus between both engines in terms of identifying instances of plagiarism, that JPLAG was easier to use but supported fewer languages than MOSS and could not deal with programs which do not parse; ‘the results returned from each system are apparently widely different’ (Culwin et al, 2001). As students often submit files which do not parse, such a limitation would mean that many files would be not be under consideration. A performance comparison of JPLAG, MOSS, and Sherlock (Warwick University's detection system) produced by the LTSN (LTSN, 2002) also revealed patchy performance and inconsistencies between the tools on items such as changing comments and replacing expressions by equivalents. In addition, neither JPLAG nor MOSS is easy to use: JPLAG requires that the user have Netscape 4 or use an applet, while MOSS requires configuration of perl files on a UNIX account. 
614
However, the JISC report also noted that ‘the tools are not intended as tests for plagiarism. They supply an ordered list of apparent similarities that allow a tutor to more efficiently locate the parts that should be examined to determine if they require further investigation’ (Culwin et al, 2001). 
Another report (Stutz, 1998) noted that tools such as JPLAG which rely on control structure metrics to detect pairwise similarities are problematic: ‘these primitive constructs – the IF, THEN and ELSE statements- are used in about the same ratio in just about every program. The end result is that plagiarism detection software that uses this scheme is prone to generate false positives’ (Stutz, 1998). 
5.0 Computer-assisted text analysis techniques 
Given the patchy performance of existing tools (Culwin et al 2001, LTSN 2002), perhaps the separation of concerns into source code and text is of limited use to the practising academic, and it may be that techniques from CATA are of greater use in detecting instances of plagiarism. Some commentators have speculated on potential of this area, although sometimes from the opposite perspective: Clough (2000) identified similarities between methods used for source code and text plagiarism detection, including ‘replacement of synonyms, re-ordering of sentences, insertion and deletion of text, change of author style, etc.’ He concluded that ‘methods used for software plagiarism detection may well work for text also’ It should also be argued that if there are similarities between detection of plagiarism in both areas, the methods used for text plagiarism detection may well work for software plagiarism detection also. 
Sallis et al (1996) observed that work in the domain of computational linguistics relating to the issue of authorship attribution based on text corpora has parallels for source code, and they advocated ‘a combination of techniques from conventional software metrics and computational linguistics’ (Sallis et al, 1996). 
Spelling and grammar measurements may be useful in identifying similarity between programs, and tools developed for computer-assisted text analysis would be able to assist in their identification: ‘many programmers have difficulty writing correct prose. Misspelled variable names (e.g. TransactoingReciept) and words inside comments may be quite telling if the misspelling is consistent. Likewise, small grammatical mistakes inside comments or print statements, such as misuse or overuse of em-dashes and semicolons might provide a small additional point of similarity between two programs’ (Spafford & Weaver, 1993). 
6.0 Analysis 
The intention of this analysis was to investigate the extent to which a concordance program such as Concordance (Watt, 2002) could assist in the detection of similarities in source code programs. The role of the tool is to assist the experienced academic, and was also compared with a manual inspection of the same files. The Concordance program offers a user interface to the analysis and also offers a web-based version of the concordance produced. 
In order to accomplish the analysis, a corpus of Java source code programs was obtained from first year BSc Computer Science students. The corpus was assembled from a set of files which were created in response to the following section of an assignment: 
(a) Develop a static method which takes a percentage value as a parameter and returns a letter grade according to the following criteria: 
• 80% or greater = ‘A’ 
• 60% - 79% ‘B’ 
• 40% - 59% = ‘C’ 
• less than 40% = ‘D’ 
(b) Write a program which: 
• prompts for and reads a student's name and percentage gained 
• uses the method in (a) to determine the grade obtained 
• prints to the screen the student's name and the grade achieved 
(Sayers, 2002) 
A corpus of eleven files were used for this analysis, and three approaches were taken: manual inspection of the files, use of JPLAG, and use of the Concordance program. Two of the files, StudentGrade.java and GradeWork.java were similar, apart from a difference in class name and 
615
substitution of ‘got’ for ‘achieved’ in the output. Prac7.java and Percentage.java had an identical static method to convert percentages to grades, which was also unusually back-to-front in its approach so would be expected to attract the attention of an assessor. The other files had trivial similarities, and some would not compile successfully in order to run the program. 
6.1 Manual Inspection 
The files were presented as they are submitted in printed form to an assessor who identified the two very similar files in five minutes and took a further ten minutes to establish that there were similarities in Prac7.java and Percentage.java. The assessor felt that this method would be too time-consuming for a larger group of files, especially as papers had to be shuffled around, and the variations in indentation meant that it was not easy to keep track of algorithms. The assessor noted that she was particularly looking for similar variable or class names, similar loops, programs with comments removed, or unusual data types or naming schemes. 
6.2 JPLAG 
The eleven files were submitted to the JPLAG server through a downloadable Java applet. This was relatively easy to set up, and results were returned to a webpage. JPLAG unfortunately rejected all but three of the files on the grounds that they did not parse. This meant that JPLAG failed to consider the majority of files submitted. It generated the following results: 
prac7Q2.java->prac7.java (82.3%) question4.java (61.5%) 
question4.java->prac7.java(46.1%) 
This had not been revealed by the manual inspection. A consideration of the 82.3% similarity between the two files revealed the following screen: 
Figure 5: JPLAG results window 
Closer examination by the assessor revealed that all that these sections shared was the control structure, but the way the structure was used was different in each case. The assessor felt that these could not be considered as similar for the purposes of plagiarism or collusion detection. JPLAG failed to identify the other clearer example of similarity, as the program would not parse. These are two drawbacks of the JPLAG approach. 
6.3 Concordance 
The eleven programs were input to the Concordance (Watt, 2002) tool, which can be easily adapted to use for computer programs. Some changes need to made before requesting a concordance: the existing stop list was modified to take into account the most frequent words which are to be found in Java programs (public, static void, main, and so on), and changes were made to definitions of what constitutes a word. Unlike other concordance applications, the Concordance tool had no problem dealing with java files and batch converted them to text files while retaining file and line information. The interface was simple and compact (Figure 6). 
616
Figure 6: selecting files 
To speed the analysis it is possible to select only those words which occur more than once, as those appearing once only have no relevance to this study. The results are displayed, sorted by descending frequency, as in Figure 7 below. 
Figure 7: Concordance window 
The assessor could select suspect headwords on the left, and the keywords appeared in context on the right. To investigate the program further, a simple click on the line brought up the program concerned in a window. As the files are referenced it is easy to see where problems arise. In the above example, similarities are revealed between Prac7.java and Percentage.java, and in the example below, Figure 8, there are clear points of comparison between StudentGrade.java and GradeWork.java. These are the results which the manual inspection also revealed. 
617
Figure 8: Detecting similarities 
The assessor had no experience of a concordance application and took some time to learn how to use it, but was pleased with the result, especially as it was quick and easy to identify real instances of similarity. The Concordance program was judged to have assisted the assessor in locating instances of similarity quickly and accurately, and enabled her to identify the files where there was collusion or plagiarism. One drawback was having to scroll through the list of headwords in order to make judgements concerning the selection of words to check; in addition, the program needed to be configured to handle computer files, and this took some time. 
7.0 Conclusions 
Traditionally there has been a separation of concerns into text and source code plagiarism detection systems, but perhaps it would be more effective to merge both concerns, by using software metrics to inform computational linguistics (and vice versa). If one accepts that source code is a language (albeit one with a restricted lexicon and syntax), this preliminary study indicates that the detection of similarity in programs might benefit from techniques from the world of computer-assisted text analysis. It may be that our student plagiarists tend to make amendments to the very items which source code detection systems ignore: they tend to leave control structures largely intact, but make changes to variables, or class names. Many student assignments are relatively short, and one would expect a preponderance of certain types of control structures, so any judgements made on these alone would not be accurate as the JPLAG example showed. This analysis will lead to a larger study with more files under consideration and longer programs, with a longer-term view to the development of a tool which would combine features of a concordance with metric-based profiles. 
Existing tools for detecting similarities in source code are useful but do not necessarily yield the expected results. Some of this is due to the fact that the practising academic has to deal with many examples of programs, some of which may not be capable of being compiled, but are just as susceptible to being examples of collusion or plagiarism. Some of this is also due to the algorithms which support these tools and are often based on control structures. In the end, of course, it is not the role of tools such as JPLAG or Concordance to judge plagiarism: they merely indicate its possibility, and so they are tools for the user to assist in its detection. ‘Although attempts have been made at automatically detecting plagiarism, it will still require the intervention of a human to prove the plagiarism and provide the final verdict’ (Clough 2000). It is up to the user to decide if plagiarism or collusion has taken place, but some combination of software metrics and computer-assisted text analysis might prove more effective. 
References: 
Bull J, Collins C, Coughlin E, Sharp D 2001Technical Review of Plagiarism Detection Report JISC. 
Clough P 2000 Plagiarism in natural and programming languages: an overview of current tools and technologies. [online] University of Sheffield. Available at: http://www.dcs.shef.ac.uk/~cloughie/plagiarism/HTML_Version/ [Accessed 18 December 2003]. 
618
Culwin F, MacLeod A, Lancaster T 2001 Source Code Plagiarism in UK HE Computing Schools, Issues, Attitudes and Tools JISC. 
DeVel O, Anderson A, Corney M, Mohay GM 2001 Mining Email Content for Author Identification Forensics. SIGMOD RECORD 30(4): 55-64. 
Gitchell D, Tran N 1999 Sim: a utility for detecting similarity in computer programs. Proceedings of the thirtieth SIGCSE technical symposium on computer science education New Orleans, Louisiana, pp 266-270. 
Gray A, Sallis P, MacDonell S 1997 Software Forensics: Extending Authorship Analysis Techniques to Computer Programs. Proceedings of the 3rd Biannual Conference of the International Association of Forensic Linguists pp 1 – 8. 
Irving R 2000 Plagiarism Detection: Experiences and Issues. JISC Fifth Information Strategies Conference, Focus on Access and Security, London. 
Jones E 2000 Plagiarism monitoring and detection – towards an open discussion. Proceedings of 7th annual CCSC Central Plains Conference, Branson, Missouri, April 6-7. 
LTSN 2002 Tools to assist detecting plagiarism [online]. Available at: http://www.dcs.warwick.ac.uk/ltsn-ics/resources/plagiarism/tools/comparison_performance.html [Accessed 1 February 2003]. 
MOSS 2002 Available at: http://www.cs.berkeley.edu/~aiken/moss.html 
Prechelt L, Malpohl G, Philippsen M 2000 JPlag: Finding plagiarisms among a set of programs, Universität Karlsruhe, Institut für Programmstrukturen und Datenorganisation. 
Sallis, P, Aakjaer A, MacDonell 1996 Software Forensics: old methods for a new science. Proceedings of Software Engineering: Education and Practice IEEE Computer Society Press, pp 481-485. 
Sayers H 2002 Coursework for Com 158M1 University of Ulster. 
SearchWebServices.com 2001 Source code [online] Available at: http://searchwebservices.techtarget.com/sDefinition/0,,sid26_gci213030,00.html [Accessed 15 January 2003]. 
Stutz M 1998 Catching Computer Science Cheaters. Wired News [online] Available at: http://www.wired.com/news/technology/0,1282,10464,00.html [Accessed 18 January 2003]. 
Touretsky DS 2001 Free Speech Rights for Programmers. Communications of the ACM, 44(8). 
Watt RJC 2002 Concordance. [online] Available at: http://www.rjcw.freeserve.co.uk/ . 
Weeber SA, Spafford EH 1993 Software forensics: can we track code to its authors? Computers & Security 12(6): 585-595. 
Wise M 1996 YAP3: improved detection of similarities in computer programs and other texts. SIGCSE'96, Philadelphia, Feb 16 – 17. 
619

Lexicalising a robust parser grammar using the WWW 
Núria Gala Pavia Salah Ait-Mokhtar 
XRCE and LIMSI-CNRS XRCE 
Bt. 508 Université de Paris-Sud 6 chemin de Maupertuis 
91403 Orsay Cedex 38240 Meylan 
gala@limsi.fr ait-mokhtar@xrce.xerox.com 
Abstract 
This paper describes an unsupervised method to lexicalise a robust parser grammar for French in order to improve prepositional phrase (PP) attachment. The ambiguous attachments produced by the parser after a first analysis of an input text are transformed and used in queries to find and download documents from the Web, where the involved words occur. The collected corpus is parsed and, from the parsing results, we acquire statistical information on PP-attachment configurations, hence building a weighted subcategorisation lexicon. This automatically acquired subcategorisation information is used in a second analysis of the input text in order to improve the disambiguation of multiple PP-attachment. 
1. Introduction 
Many robust parsing systems that can process real-world texts were developed in the nineties (Abney 1996, Grefenstette 1996, Collins 1996, Ait-Mokhtar and Chanod 1997, Charniak 2000). These parsers assign linguistic structures to text sentences and are used with some success in various applications, such as information extraction, question answering systems, word sense disambiguation, etc. Most of them are reported to have good accuracy, sometimes above 90% precision in the recognition of some syntactic structure elements (noun and prepositional phrases, subject-verb and object-verb relations, etc.). 
However, the correct handling of attachment ambiguities, especially PP-attachment, is still an issue. The use of traditional subcategorisation lexicons is not of much help. Large-scale lexical subcategorisation information for verbs, nouns and adjectives is available for few languages. When available, it is very often not informative enough to resolve such ambiguities. In particular, in ambiguous situations where the involved headwords share the same subcategorisation properties, it is not possible to induce attachment preferences from traditional subcategorisation lexicons. 
In this context, probability-based techniques, be they supervised or unsupervised, have been designed to capture and measure subcategorisation preferences, and have been applied to PP-attachment. 
Unsupervised methods exploit regularities in raw or automatically annotated corpora and calculate frequencies of the presence of a word next to another. One of the classic approaches is that of lexical association scores (Hindle and Rooth 1993). This method, which obtains about 80% precision, estimates the probability of association of a preposition with a noun or a verb from their distributions in corpora (previously annotated by a parser). Other unsupervised approaches, such as that of Volk (Volk 2000, Volk 2001), obtain 74% accuracy for all decidable test cases for German using triples (X,P,N2). 
Supervised methods such as transformation-based learning (Brill and Resnik 1994), memory-based learning (Zavrel et al. 1997), etc. use manually annotated training corpora. The average accuracy results on PP-attachment ambiguities, restricted to the [Verb NP PP] configuration, are reported to be around 80-84% for English. The use of semantic classes in the supervised method of (Stetina and Nagao 1997) allows them to reach a precision of 88% for the same configuration. 
As for more ambiguous attachments, with multiple alternatives, (Merlo et al. 1997) propose a generalized back-off method for English, dealing with structures such as [Verb NP PP1 PP2]. The reported precision rate is 69,6% for PP2 and 43,6% for PP3. According to the authors, the increasing number of configurations to take into account, and a problem of sparse data may justify these figures. 
620
For PP-attachment disambiguation in French, (Gaussier and Cancedda 2001) propose a statistical model that integrates different resources (subcategorisation frames, semantic information). The results reported are around 85% for [Verb NP PP] but only 74% for other structures such as [NP PP PP]. 
To summarize this brief overview on existing methods to resolve PP-attachment, (a) most of the proposed methods are supervised, because it is more efficient to compare the results with corpora created or verified by humans and (b) precision rates are around 84% for the [Verb NP PP] configuration but significantly less for more ambiguous configurations. 
2. Overview of our approach 
Our method for lexicalising the core dependency grammar for PP-attachment resolution combines the results of a first analysis of the parser with statistical information extracted from the Web. We applied an unsupervised method to improve PP-attachment in an existing rule-based dependency system for French based on the XIP parser (Ait-Mokhtar et. al. 2002). 
2. 1 The parser 
XIP is an incremental parsing framework that allows the extraction of deep dependency relations at sentence and inter-sentential level. A dependency is, in this approach, a syntactic relation holding between the headwords of two chunks, e.g. a noun and a verb, or in the case of PP-attachment, between two headwords linked by a preposition. We represent the PP-attachment dependency with A(X,Prep,N) where X is a noun, a verb or an adjective, Prep a preposition and N a noun, for instance A(impact,sur,comptes)(impact on accounts). 
Several grammars have been designed and implemented for chunking and dependency extraction in a two-tier approach (Gala 2001). Figure 1 gives an idea of the overall architecture of the system. 
Figure 1: Architecture of our parsing approach. 
The grammars (a core grammar and some specialized modules) are applied depending on the linguistic and structural characteristics of each input sentence (Gala 2003). For chunking, the core grammar identifies main chunks (NPs, PPs, etc.) whereas specialized chunking modules mark particular constructions such as lists, enumerations, etc. For dependency extraction, specialized modules calculate dependencies in those constructions. Core dependency grammars extract main syntactic functions (subject, object, etc.) as well as PP-attachment. 
The output of this model is a chunked sentence with a list of dependency relations. Given the French sentence “La méthode d'amortissement choisie aura un impact significatif sur les comptes.” (“The 

Core dependency grammars 

input 

Lexical database 

Specialized chunking and dependency grammars 

Core grammar 

output 

W W W 

Automatic 
Learning 

Second tier 

First tier 
621
selected depreciation method will have a significant impact on the accounts.”), the parser gives the following output1: 
SUBJ(aura,méthode) 
OBJ(aura,impact) 
NADJ(impact,significatif) 
NADJ(méthode,choisie) 
A(méthode,de,amortisement) 
A(a,sur,comptes) 
A(impact,sur,comptes) 
A(significatif,sur,comptes) 
0>MAX{NP{La méthode} PP{d' NP{amortissement}} AP{choisie} FV{aura} 
NP{un impact} AP{significatif} PP{sur NP{les comptes}} .} 
2. 2 Automatic learning of subcategorisation 
To improve PP-attachment some of the dependencies extracted after a first analysis are transformed into subcategorisation frames and a measure of frequency for each frame (a weight) is calculated using a big collection of documents. These weights are integrated into the system and later used to disambiguate controversial attachments in a second analysis. 
Unlike Volk's approach (Volk 2001), the subcategorisation weights here are syntactic co-occurrence probabilities, as opposed to textual co-occurrence probabilities, based on the frequencies of words occurring in the same text within a maximal linear distance, i.e. with some maximal number of words in between. Instead, in our approach syntactic co-occurrence probabilities are measured from the frequencies of words co-occurring in the same syntactic dependency relations (i.e. attachments already yielded by the parser in a first analysis). The approach is unsupervised since the dependency relations are automatically produced by the existing parser itself, which may suggest for each PP one or several possible attachments. 
Our hypothesis is that PP-attachments produced by the parser are more reliable than rough textual co-occurrences, because the parser disregards all the cases where the words co-occurring in text clearly cannot be syntactically linked. Such situations include words occurring in distinct sentences, in distinct finite clauses, etc. 
3. Initial and baseline grammars of the parser 
The grammar that produces the dependencies for PP-attachment (the baseline grammar that we call G2) is made of two initial grammars. The first one (G0) gives priority to recall (the rules present few constraints in order to extract as many dependencies as possible) while the other grammar (G1) produces fewer dependencies but with a high degree of precision. 
The corpus used for the experiments is a heterogeneous corpus from different domains: scientific (46,8% of the corpus), economic (41%), and legal (12,2%). It contains 2.493 words (89 sentences) with an average of 28 words per sentence. The evaluation of PP-attachment using G0 and G1 grammars gives the following results: 
Grammar Number of rules Precision Recall F Number of dependencies G0 16 41,88 % 94,28 % 58,00 % 2.438 G1 11 87,49 % 77,04 % 81,93 % 964 
Table 1: Results of the initial grammars. 
1 SUBJ is a subject relation, OBJ a direct object, NADJ an adjective modifying a noun, A a PP attachment. MAX is the maximal group, NP a noun phrase, PP a prepositional phrase, AP an adjective phrase, FV a finite verb phrase. 
622
The baseline G2 grammar is an optimisation of the two initial grammars. It is made of “reliable” G1 rules (those rules that, individually, have more than 93% precision2) and the remaining G0 rules, only applied after verifying that a PP has not been attached by a reliable G1 rule. The overall results of G2 are the following: 
Grammar Number of rules Precision Recall F Number of dependencies G2 21 71,34 % 92,10 % 80,40 % 1.413 
Table 2: Results of the baseline grammar. 
While the precision rate decreases compared to G1, the recall rate increases significantly. The F-score (average measure) remains a little bit higher than 80%. 
The output provided by G2 gives information on the reliability of a PP-attachment (we use the features MF1 and MF2, MF stands for mesure de fiabilité –reliability measure). For the previous example3: 
A_MF1(méthode,de,amortissement) 
A_MF2(avoir,sur,compte) 
A_MF2(impact,sur,compte) 
A_MF2(significatif,sur,compte) 
The dependency relations with a “less reliable” mark (MF2) are used to build a database of subcategorisation patterns enriched with statistical measures obtained from the Web. 
4. Building a subcategorisation database with the WWW 
The use of the World Wide Web as a huge database of examples for different tasks related to NLP is quite a recent idea: for translating compound nouns (Grefenstette 1999), for acquiring named entities (Jacquemin and Bush 2000), for PP-attachment resolution (Volk 2001). In our approach, less reliable PP-attachments produced by the baseline grammar on a given document are used to query the Web in order to obtain a huge collection of documents. Our hypothesis is that occurrences of the attachments to validate should be present in this corpus (or not at all if they are wrong). 
4.1 The queries 
After a first analysis of the initial corpus with G2, the 869 dependencies having the feature MF2 have been transformed into queries for the Web. Each query contains the three words involved in the PP-attachment dependency A(X,Prep,N), where X is a verb, a noun or an adjective. The two first tokens are what we call the subcategorisation frame; the third one is the word to attach. 
We have chosen Altavista (www.Altavista.com) with the advanced search option, which permits the use of Boolean operators such as NEAR (allowing a distance of at most ten words between two given words). The queries are of the form (X Prep) NEAR N. For the dependency A_MF2(impact,sur,comptes) the query is the following one. 
(impact sur) NEAR comptes 
This query may select documents containing the following configurations: 
Document 1 … impact sur les comptes … 
Document 2 … Impact sur l'analyse des comptes … 
Document 3 … impact sur la fermeture de vos comptes … 
2 An evaluation of the 11 rules of G1 has shown that 5 of them have a precision rate going from 100% to 93,72% (while the others from 89,79% to 41,67%). These 5 rules where P > 93% are called “reliable”. 
3 The dependencies can also be displayed with the tokens and their word number: 
A_MF2(<a^avoir:5>,<sur^sur:9>,<comptes^compte:11>) . 
623
Some of the selected documents may be inappropriate, that is, the occurrences present in the document are not syntactic co-occurrences. In the previous example, the occurrence in document 3 is not appropriate because the right attachment is “fermeture de comptes” and not “impact sur comptes”. Before computing the frequencies of subcategorisation patterns, we parse the documents and only those occurrences that yield a PP-attachment candidate are considered. We think that the huge size of the WWW should prevent any sparse data problem that such restrictions might otherwise yield. 
4.2 The corpus for the database 
Each of the 869 queries (created from the 869 MF2 dependencies) retrieves 20 URLs. To perform this task we have used a set of perl scripts combined with the Unix command wget. The result of this process is a collection of about 17.000 documents (the sizes of the corpora obtained for one single query can be very different; it is also possible to harvest fewer than 20 URLS for a given query). 
After removing all the HTML tags from the corpora, we have a new corpus (we call it the Web corpus) of 38.242.073 words and 1.368.903 sentences. To create the database, we have parsed this corpus and extracted the overall dependencies for PP-attachment. Here, we were interested in the quantity of the dependencies (about 4 million) and not in the quality (produced by a reliable or less reliable rule). Therefore no difference has been made between MF1 and MF2 dependencies. 
4.3 The database 
The dependencies A(X,Prep,N) from the above mentioned corpus have been automatically transformed into subcategorisation frames(X Prep)using the lemmas. As in (Volk 2001) a measure of co-occurrence (MC) has been automatically calculated. This measure is determined by the frequency of a word (X) and the frequency of the co-occurrence(X Prep): 
MC(X,Prep) = freq(X Prep) / freq(X) 
As a result, we obtain a database that has entries of the following form: 
0.0008 avoir | sans 
0.0005 avoir | sous 
0.0118 avoir | sur 
… 
0.0005 impact | selon 
0.0626 impact | sur 
… 
0.0323 significatif | a 
This weighted subcategorisation lexicon contains about 100.000 subcategorisation frames obtained from the Web corpus, each one having a co-occurrence value. The database is incorporated as a resource in the parser and used to resolve PP-attachment ambiguities. 
5. Disambiguation of PP-attachment dependency relations 
For PP-attachment resolution, we have applied a disambiguation algorithm to all “less reliable” (MF2) dependencies from the initial corpus. This algorithm aims at finding the correct attachment for a given headword when the parser has yielded multiple attachments after the first analysis step. 
Each PP-attachment candidate is compared to the patterns in the weighted subcategorisation lexicon. Among the set of attachment candidates for a given PP (Prep N), only the one that corresponds to the best-weighted subcategorisation pattern is selected. For the previous example, the dependency A(impact,sur,compte) is kept while A(avoir,sur,compte) is wiped away (see the scores above). Whenever the scores are equal, as the algorithm does not have any other information to resolve the ambiguity, both dependencies are kept. 
Otherwise, if a frame is not in the database, we suppose its score to be null. In this case, the comparison to an existing pattern does not take place and the existing subcategorisation frame is kept. In the example, this means eliminating A(significatif,sur,compte). When none of the patterns exists in the database, both dependencies are kept. 
624
The final result for the example above is: 
A(méthode,de,amortissement) 
A(impact,sur,compte) 
0>MAX{NP{La méthode} PP{d' NP{amortissement}} AP{choisie} FV{aura} 
NP{un impact} AP{significatif} PP{sur NP{les comptes}} .} 
Note that the first PP-attachment is yielded after a first analysis by a reliable rule while the second one is produced in a second round, after applying the disambiguation algorithm using the information on the lexical database. 
The overall results on the initial corpus are the following: 
Grammar Precision Recall F Number of dependencies G2 71,37 % 92,10 % 80,40 % 1.413 Lexicalised G2 83,21 % 85,12 % 84,16 % 1.120 
Table 3: Results of the baseline grammar before and after lexicalisation. 
As shown in the table, we get an attachment precision of 83,21%, which means an 11,84% increase compared to the results obtained only with the rule-based baseline grammar. The F-score also increases (+3,76%). Unlike most of the evaluations reported on section 1, these figures concern all kinds of configurations for PP-attachment in French [V NP+ AP+ PP1 PP2 … PPn]. 
6. Conclusions 
This paper presents an unsupervised method to improve PP-attachment resolution for French in an existing rule-based dependency parser. The initial system, made of different grammars, is enriched with a weighted lexical database containing subcategorisation frames and weights (syntactic co-occurrence probabilities) automatically acquired from the Web. 
To construct the lexical database, a measure of reliability yielded by the parser after a first analysis permits the identification of some of the syntactic dependencies that will be used to query the Web. 
Compared to the initial rule-based grammar, the results obtained prove that the use of both lexical and statistical information significantly increases the precision rate of the PP-attachment dependencies. 
Further refinements can be foreseen to improve the overall results. One suggestion is the use of the three elements of a dependency (X Prep N) to calculate the co-occurrence weight. This solution would allow more precise configurations to be used during the disambiguation process. Another point would be the generalisation of the method using semantic classes encoded as features: the concordance of two classes would be given priority during the PP-attachment resolution. 
Finally, this approach could be applied to other problems of structural ambiguity such as coordination or complex named entity recognition. 
References 
Abney, S. P. 1996 Partial Parsing Via Finite-State Cascades. In ESSLLI'96 Workshop on Robust Parsing, Prague, Czech Republic. 
Ait-Mokhtar, S. and Chanod, J. P. and Roux, C. 2002 Robustness beyond shallowness: incremental deep parsing. In Natural Language Engineering, 8(2/3), pp. 121-144, Cambridge University Press, UK. 
Ait-Mokhtar, S. and Chanod, J.-P. 1997 Incremental Finite-State Parsing. In Proceedings of the 5th Conference on Applied Natural Language Processing (ANLP-97), pages 72-79, Washington, DC, USA. 
Bourigault, D. and Fabre, C. 2000 Approche linguistique pour l'analyse syntaxique de corpus. Cahiers de Grammaire (25): 139-151. 
625
Brill, E. and Resnik, P. 1994 A rule-based approach to prepositional phrase attachment disambiguation. In Proceedings of the 15th International Conference on Computational Linguistics (COLING-94), Kyoto, Japan. 
Charniak, E. 2000 A maximum entropy inspired parser. In Proceedings of the 1st Meeting of the North American Chapter of the Association for Computational Linguistics, pp. 132-139, Seattle, Washington. 
Collins, Michael J. 1996 A new statistical parser based on bi-gram lexical dependencies. In Proceedings of the 34th Annual Meeting of the Association for Computational Linguistics (ACL), pp. 184-191, Santa Cruz, USA. 
Gala Pavia, N. 2003 Un modele d'analyseur syntaxique robuste basé sur la modularité et la lexicalisation de ses grammaires. PhD thesis. Université de Paris-Sud (forthcoming). 
Gala Pavia, N. 2001 A two-tier corpus-based approach to robust syntactic annotation of unrestricted corpora, TAL 42 (2): 381-411. 
Gaussier E. and Cancedda, N. 2001 Probabilistic models for PP-attachment resolution and NP-analysis. In Proceedings of ACL-2001, Computational Natural Language Learning Workshop, CoNLL-2001, pp. 45-52, Toulouse. 
Grefenstette, G. 1999 The World Wide Web as a resource for example-based machine translation tasks. In Proceedings of Aslib Conference on Translation and the Computer 21, London. 
Grefenstette, G. 1996 Light Parsing as Finite-State Filtering. In Proceedings of the ECAI'96 workshop on extended finite state models of language, Budapest. 
Hindle, D. and Rooth, M. 1993. Structural ambiguity and lexical relations. In Computational Linguistics, 19(1), pp.103-120, the MIT Press. 
Jacquemin, C. and Bush, C. 2000 Combining lexical and formatting cues for named entity acquisition from the Web. In Schutze H. editor, Proceedings of joint Sigdat Conference on Empirical Methods in Natural Language Processing and Very Large Corpora, EMNLP-VLC-2000, Hong Kong. 
Merlo P., Crocker M. W. and Berthouzoz C. 1997 Attaching multiple prepositional phrases: Generalized Backed-off Estimation. In Cardie C. and Weischedel, R. editors, Proceedings of the second conference on Empirical Methods in Natural Language Processing, EMNLP-97, pp. 149-155, Providence, R.I. 
Stetina, J. and Nagao, M. 1997 Corpus Based PP-Attachment Ambiguity Resolution with a Semantic Dictionary. In Proceedings of the Fifth Workshop on Very Large Corpora, pp. 66-80, Beijing and Hong Kong. 
Volk, M. 2000 Scaling up. Using the WWW to resolve PP attachment ambiguities. In Proceedings of Konvens-2000, pp. 151-155, Ilmenau, Germany. 
Volk, M. 2001 Exploiting the WWW as a corpus to resolve PP attachment ambiguities. In Proceedings of Corpus Linguistics 2001, pp. 601-606, Lancaster, UK. 
Zavrel, J., Daelemans, W. and Veenstra, J. 1997 Resolving PP Attachment Ambiguities with Memory-Based Learning. In Proceedings of the Conference on Computational Natural Language Learning (CoNLL-97), Madrid. 
626

On the combination of corpus-based and experimental methodologies in the study of causal, contrastive and metadiscourse connectives in L1 and L2 text comprehension and production 
Julien Perrez & Liesbeth Degand 
Université catholique de Louvain (UCL) ; Belgium 
1. Introduction 
The purpose of this paper is to give an overview of work in progress, with an emphasis on methodological matters. In our study, our aim is to tackle two fields of (foreign) language learning, viz. text comprehension and text production focussing on the impact of causal, contrastive and metadiscourse connectives on L21 text comprehension and their usage in L2 text production. The learner group we aim at consists of learners with a L1 French background learning Dutch as a foreign language. For the comprehension studies, we will follow the methodology used by Degand and Sanders (2002), which is exclusively experimental. Corpora will be topical in the production studies, where it is our intention to use a Dutch learner corpus. However, since there is not any official corpus of that kind available at the moment, one of the first steps of our project will be to collect one. In the following we will present the methods we mean to adopt to gather that learner corpus. The results we will arrive at when using the corpus will subsequently be complemented by an experimental study whose aim will be to compare learners’ text productions in their second language to text productions in their mother tongue in order to determine the exact role played by their L1 when writing in their L2 (see Kubota 1998). We hope that this last step will allow us to draw more accurate conclusions as to connective use by French-speaking L2 learners of Dutch. In combining learners’ L2 text comprehension and production we hope to be able to detect some degree of interaction between those two aspects of language learning. 
2. L2 Text comprehension methodology 
In their study, Degand and Sanders (2002) underline the utmost importance of using a successful methodology when studying the impact of connectives on text comprehension. The variety of different methodologies used in past studies has led to a multitude of different conclusions, from which it can be derived that a lack of methodological accuracy can lead to biased conclusions (compare Millis, Graesser, Haberlandt (1993) to Degand, Lefevre, Bestgen (1999) and Degand and Sanders (2002)). There is therefore at the moment an urgent need for a unified method when studying the impact of given connectives on text comprehension. That is the reason why we intend to replicate the methodology used in Degand and Sanders (2002) and Degand et al (1999), which has shown convincing and homogeneous results. 
This methodology is exclusively experimental and can briefly be described as follows: the participants have to read an expository text, e.g. a purely informative text which does not require any particular background knowledge on the part of the reader to understand it, and to fulfill a recall task consisting of four questions about that text. The texts are manipulated with respect to the presence of explicit relational markers and are scaled in terms of difficulty. Two of the questions refer to the coherence relation marked by the connective (or not, if the coherence relation is left implicit) and the other two tap general ideas from the text. The time to read the text and answer the questions is controlled. After having read the text and before answering the comprehension questions, participants perform a distraction task, aiming to empty their short term memory about the text. This process is then repeated as long as there are texts to be read. This is roughly the method we plan to employ to measure the off-line impact of connectives on text comprehension, since it has already led to encouraging results, indicating a positive impact of causal connectives on text comprehension both in L1 and L2 (Degand and Sanders 2002). 
In our project, we will nevertheless broaden the scope of these studies in trying to confirm the observed results for causal connectives and to come to first tendencies regarding the impact of other types of 
1 The abbreviation ‘L2’ is used in this essay to refer to foreign language learning , not to second language learning. 
627
connectives such as contrastive and metadiscourse connectives. In keeping with Oversteegen (1997) we refer to ‘contrastive connectives’ as to linguistic devices which establish a link between two utterances and express either a ‘semantic opposition', a ‘denial of expectation’ or a ‘concession'. Connectives such as ‘maar’ (‘but'), ‘terwijl’ (‘while', ‘whereas') and ‘hoewel’ (‘although', ‘however') belong to this category. Our definition of ‘metadiscourse connectives’ has been borrowed from Halliday (1978) who described them as “linguistic elements that refer to the organization of the discourse itself and to aspects of the relationship that develop between the author and the reader of the texts.” As a matter of fact, this category comprises a huge number of linguistic expressions. As exemplified by Jackiewicz (2002), it basically includes connectives expressing enumeration, such as ten eerste, ten tweede (‘firstly, secondly), ten slotte (‘finally'),… or denoting spatial organization (aan de ene kant, aan de andere kant –‘on the one hand, on the other hand', in de eerste plaats, in de tweede plaats – ‘in the first place, in the second place'…) or temporal organization (dan-‘then'-, vervolgens-‘further', daarna –‘subsequently'-, tot slot –in conclusion'-,…) and expressions like wat mij betreft (‘as far as I am concerned'). It is not our aim to study the impact of all these different forms of metadiscourse (see also, Hyland, 1998; Crissmore et al., 1993). Rather we will concentrate on text organizing markers, also called advance organizers (Corkill et al., 1988). 
Another aspect of research we wish to go into concerns the general linguistic and/or proficiency level which is required for connectives to have an impact on L2 text comprehension. Degand and Sanders (2002) correlated the variable ‘presence of connectives’ with the variable ‘language proficiency’ (measured by a L2 proficiency test) and showed that all the participants benefited from the presence of explicit causal markers both in their L1 and in their L2, which made them conclude that the “L2 proficiency level [of the participants] was high enough to understand the general functions of the signals and their usage conditions.” They backed up their conclusion by means of the inter-dependence hypothesis (Cummins 1984) and consequently claimed that 
“as soon as readers master an efficient reading strategy in their mother tongue, including the ability to utilize and infer coherence relations in discourse, they can transfer this skill to another language, provided they have also developed a sufficiently high L2 competence level.” (Degand and Sanders 2002: 754) 
In our study, we will try to determine that minimum L2 proficiency level by focussing on different groups of learners. We will first distinguish the different participants on the basis of their level of instruction and then on the basis of the score they obtain on a language proficiency test. Two groups of participants of different level of instruction will be selected (six formers2 versus university undergraduates3) and each will subsequently be divided up into three levels of L2 proficiency, on the basis of their results at the test. This subdivision makes it possible to cope with the individual level differences in the selected groups and will allow us to observe the correlation between the impact of the connectives and the L2 proficiency level of the participants more accurately. 
3. L2 Text production methodologies 
3.1. Collecting a Dutch learner corpus 
As already mentioned above, a Dutch learner corpus does not exist at the moment. In order to get round that problem, we have begun collecting a corpus, and this process is still ongoing. Considering our relative inexperience in the matter, we have been inspired by the work of our colleagues of the English Department of the University of Louvain, who have conducted the ICLE-project (Granger 1994, see also http://www.fltr.ucl.ac.be/fltr/germ/etan/cecl/Cecl-Projects/Icle/icle.htm). 
Having to compile a Dutch learner corpus in the framework of our project, we have been looking for fellow researchers in order to extend our data. We have consequently established a collaboration with researchers from the University of Namur (FUNDP) and from the University of Liege (ULg). Since our project is still in its infancy, we are still in the early design process. In line with the ICLE project, we have established a certain number of variables to be taken into account in the realization of the corpus. Figure 1 gives an 
2 Pupils from the last year of secondary school having studied Dutch for at least six years. 
3 year at the university 
628
2nd 
overview of these variables. Among those variables, we find for example the mother tongue of the participants (French), but also the medium (written language). We intend to focus in the first place on French-speaking learners of Dutch, because they constitute the major unified group of learners of Dutch. Since more than 70% of the French-speaking Belgians learn Dutch as a Foreign Language at one time or another in their school curriculum, research results and pedagogical implications are most likely to be of interest to a high number of didacticians. Our interest for written language is essentially practical, considering that it would be somewhat premature to try to insert spoken language data in a corpus which does not have any written basis, and that it is far more complicated to obtain spoken language data in a relatively short period. The variable ‘genre’ will be left open since it is our intention to collect different kinds of material ranging from essays to written proficiency exercises and letters. 
Shared features 
Learner variables Task 
Figure 1: Corpus of Learner Dutch: Learner and task variables (based on ICLE) 
In our own research on connective usage by learners of Dutch, we won't be dealing with the whole set of variables presented above, but we will combine some of them to create homogeneous subsets (see Figure 2). In this regard, besides the fixed features, we intend to control the variables ‘genre’ (essay-writing), ‘region’ (French-speaking part of Belgium) and to limit the considerable variation resulting from variables such as ‘task setting', ‘L2 exposure’ and the influence of other foreign languages. In addition, we will carefully manipulate the features ‘level of instruction’ and ‘L2 proficiency’ in order to detect any differences between diverse groups of learners. This will in turn allow us to relate these variables in the ‘production part’ of our study to the same variables in the ‘comprehension part’ (see above), which will possibly point to a correlation between the impact of certain connectives on L2 text comprehension and their use in L2 text production (see below). 
So far, we have been collecting data from different sources. The data we have at our disposal for the moment consist of writing skill exercises from six-formers (essays, reports, description tasks), essays from first year undergraduates and essays from learners ranging from first year undergraduate to last year undergraduate, written within the framework of an essay competition, organized by the ANBF (Association des Néerlandistes de Belgique Francophone)4 and essays from the CNaVT (Certificaat Nederlands als Vreemde Taal)5. We won't begin working and derive any frequencies or tendencies before we have reached the level of 300.000 words, which seems to be the minimum limit at which a number of linguists are working (see for example Osborne 1999). 
629 

Corpus of Learner Dutch 

Variable features 

variables 

Task variables 

Learner variables 

• Mother tongue 

• Medium 

• Gender 
• region 
• Level of instruction 
• Other FL 
• L2 exposure 
• L2 proficiency 

• Field 
• Genre 
• Topic 
• Task setting 
- timing 
- exam 
-reference tools 
4 “Association of the Dutch specialists from the French-speaking part of Belgium” 
5 Dutch counterpart of the Cambridge certificate. 

Shared features • • 
Learner variables Task Learner variables Task variables 
Figure 2: Distribution of the variables in the study on connective usage by French-speaking learners of Dutch . 
3.2. Working with the corpus 
The contribution of (learner) corpus analyses to this type of research is twofold: firstly, corpora will allow us to work with a large amount of natural language data (Granger 1998), and with specific computerized tools which make it possible to easily approach those data from the desired point-of-view (Granger 1998, Meunier 1998). Secondly, corpus research provides the textual context in which a word or an expression, in this case a connective, appears, which can lead to semantic assumptions concerning connective usage by L2 learners of Dutch. This will indirectly also point to phenomena such as avoidance, a strategy regularly used by learners to avoid a particular L2-specific difficulty (Liu and Shaw 2001). We could for instance imagine that French-speaking learners of Dutch would use the connective ‘want'(‘because/for') more frequently than the connective ‘omdat'(‘because'), both making a causal relation explicit, considering that the syntactic environment required by the former is more French-like, and therefore often regarded as easier to use, than the syntactic environment required by the latter. Corpus analyses will besides point to the usage frequency of the different connectives under investigation by the group of learners we focus on and will in turn, when compared to L1 data, lead to the emergence of interesting tendencies, concerning the (over-, under- or mis-) use of those linguistic devices by our group of L2 learners (Granger & Tyson 1996, Osborne 1998, Liu and Shaw 2001). 
3.3. Shortcomings of corpus-based studies 
Whereas corpora offer incredible quantitative resources, we prefer to turn to more experimental approaches for the qualitative analysis of the results of the corpus analysis. The reason for this is based on one shortcoming of- and one criticism concerning corpus studies, namely their inability to deal with a whole range of different variables and the frequent overgeneralizations of the observed results. 
As it has often been claimed, different methodologies can be responsible for result variations. This can be illustrated by contrasting the respective studies of Granger and Tyson (1996) and Osborne (1998), both studying connective use by French-speaking learners of English when writing in their L2, the former using argumentative texts and the latter focussing on more ‘neutral’ texts. The two studies arrived at different observations about the use of connectives by L2 learners of English: while Granger and Tyson detected an overuse of certain types of connectives, which they related to a case of L1-L2 transfer, Osborne (1998) didn't find any. This clearly illustrates that the variable ‘type of text’ seems to be a source of result variation. Other variables are likely to influence the results as well: Jarvis (2000) compiled a list of variables, which, according to him, “should ideally be controlled in an investigation of L1 influence”. The list in question includes the following variables: 

Corpus of Learner Dutch 

Variable features 

variables 

• Gender 
• Other FL 
• L2 exposure 

• Topic 

• Medium 
• Genre 
• Field 
• Task setting 

• Mother tongue 
• Region 

Investigated features 

Level of instruction 
L2 proficiency 
630
(1) age 
(2) personality, motivation and language aptitude 
(3) social, educational, and cultural background 
(4) language background 
(5) type and amount of target language exposure 
(6) target language proficiency 
(7) language distance between the L1 and target language 
(8) task type and area of language use 
(9) prototypicality and markedness of the linguistic feature 
(Jarvis 2000: 260-261) 
Among these variables, some are highly subjective and therefore hard to measure and almost uncontrollable (such as ‘personality’ and ‘motivation'), others can be manipulated in the framework of corpus collection, among which the study-dependent variables such as ‘task type', ‘area of language use’ or ‘prototypicality and markedness of the linguistic feature', but still others can only be accurately measured by an experimental study. One of these variables deserves special attention in this regard, namely the ‘target language proficiency’ variable. The different levels of language proficiency of the participants is a variable which has generally been too vaguely dealt with in most corpus studies. The labels ‘intermediate', ‘upper intermediate’ or ‘advanced’ have been recurrently used and are most of the time just synonymous with years of instruction. Those generic terms are exceedingly subjective and strongly inclined to personal, group and even geographical variation, or to instruction antecedents. We want to cope with this variable by having a language proficiency test taken by the different participants in our experiment, and by dividing up these participants into well-defined groups of L2 language proficiency (this is basically the same idea as the one exposed in section 2, concerning the distribution of the participants among different levels of language proficiency for the comprehension experiments). Another variable which does not clearly emerge from Jarvis’ list but which might deeply influence the results of learner corpus research on L2 text production is the rhetorical ability of the learners in their mother tongue (Kubota 1998; see below). These examples highlight the importance of being highly consistent with the adjustment of the settings of a study and that relying solely on corpus data could lead to some unexpected and undesired variation. 
The second shortcoming of corpus research I would like to tackle lies in the almost recurring overgeneralizations or hasty conclusions made by some authors of corpus studies: these often tend to infer a lot from what the observed frequencies point to, and to generalize the obtained results, which often contributes to their drawing of very intuitive conclusions. A common case of generalization in learner corpus researches is the claim that the weird-soundingness of L2 written productions has to be related to a case of L1-L2 negative transfer (Granger & Tyson 1996, Osborne 1999). But this conclusion is not as easy proven as it sounds. The following passage fairly illustrates the phenomenon: 
“Another discrepancy between the MCC [Micro Concord Academic Corpus Collection] and LC [Learner Corpus] was in the grammatical patterning of because. In the LC, the pattern It is because…, which did not occur in the MC, accounted for 8 out of the 35 instances of because. […] There were also several instances of double connectors being used, e.g. Because…so that…which is a case of L1 transfer” (Flowerdew 1998: 332). 
This conclusion relies upon the own intuition of the author and has further not been verified by further observations. The point is that transfer goes beyond the scope of intuitions and is more than what Jarvis (2000) ironically calls a “you-know-it-when-you-see-it-phenomenon”. The literature about transfer of L1 skills or linguistic devices to L2 is huge and disparate. Fortunately, Jarvis (2000) provides a complete state-of-the art of this literature, representing the different trends and trying to combine them to come to a ‘unified framework'. His justification for the need for that unified framework within the study of transfer is motivated by his assumption that confusion reigns over the theoretical transfer patterns. He argues that there are “conflicting claims [in the literature] about the nature of L1 influence and its interaction with other factors” and highlights the “lack of consensus concerning what L1 influence is and how it should be investigated”, pointing to the difficulty of appropriately defining what the concept ‘L1 transfer’ stands for. He defines it as an underlying phenomenon which shows itself in three different effects, namely “intra-L1-group homogeneity in learners’ IL performance”6, “inter-L1-group heterogeneity in learners’ IL 
6 “when learners who speak the same L1 behave in an uniform manner when using the L2” (Jarvis 2000: 254). 
631
performance”7 and “intra-L1-group congruity between learners’ L1 and IL performance”8. Jarvis consequently states that an ideal analysis of transfer should consider these three effects to come to reliable conclusions: 
“All three should be examined before one is justified in making claims concerning when, where, in what form, and to what extent L1 influence emerges in learner data” (Jarvis 2000: 259). 
An illustration of the application of this transfer verification method is indirectly found in Granger and Tyson (1996). They observed an over- and misuse of connectors belonging to what they call the ‘corroborative category’ (‘in fact', ‘indeed') in the L2 production of French-speaking learners (this can be related to Jarvis’ first effect: ‘intra-L1-group homogeneity'). In order to test whether they were confronted to a case of transfer, they “compared the French figures with figures from a comparable corpus of German learner writing” and found that French-speaking learners clearly overused the connector ‘indeed’ (which corresponds to Jarvis’ second effect: ‘inter-L1-group heterogeneity'). They finally related those observations to the conclusion that “it is likely that the significant overuse of ‘indeed’ is transfer-related, especially since it is often viewed by French speakers as the translational equivalent of ‘en-effet', a very common connector in written French” (this matches Jarvis’ third effect: ‘intra-L1-group congruity'). Even if their last assumption does not rely upon any mentioned sources, their conclusion has been consistently verified and seems therefore to hold water. 
Besides the fact that transfer cannot be assimilated to an intuition-based concept, one has to consider that (negative) transfer from L1 to L2 is not the solely factor accounting for specific learners’ interlanguage features. As Kubota (1998) puts it: 
“non-nativeness in ESL essays can be a reflection of various factors other than L1-specific rhetoric […] what influences L2 texts is not only L1 rhetoric, but also various factors such as previous English instruction, strategies specific to L2 writing, L2 proficiency, and L1 writing ability” (Kubota 1998: 75). 
This point-of-view is supported by Jarvis (2000) as well, as he claims that “multiple factors may combine to influence a learner's use of the L2 at any given moment and at all stages of development.” He consequently goes into a full consideration of the relation between L1 influence and other variables (see the list above) and tries to determine which combine and which are the most influential in the case of L2 lexical choices. Although he comes to the conclusion that L1 is the predominant factor for influencing L2 lexical choices, he also observes that “many outside variables collude rather than compete”, which equally supports the argument presented here. 
Among those factors which might exert an influence on L2 text production, L2 proficiency and L1 writing skills deserve special attention. The choice between a whole set of possibilities to express oneself in a foreign language is likely to be conditioned (or restricted) by the lexical knowledge (which is part of the proficiency) one has at one's disposal in the target language. Lack of vocabulary or semantic inconsistencies are often put forward in second language acquisition research to clarify the erroneous use of certain linguistic devices by some learners in their L2 production. This can be illustrated in the light of the following statement made by Liu and Shaw (2001): 
“Learners may not be aware of the gaps in their vocabulary knowledge (Wong 1983; Laufer 1989), especially in cases where there are direct translations between L1 and L2. They tend to perceive the target L2 items as easy to learn, ignoring the fact that each of the translational equivalents has other properties of its own which is distinct from those of its counterpart. This obscures many difficulties and learners are easily trapped by the deceptively transparent target items” (Liu and Shaw 2001: 188). 
More specific connective-based studies (Granger and Tyson 1996; Osborne 1998) have led to similar observations, stating for example that “learners seem not to recognize that connectors such as ‘in fact’ and 
7 “when comparable learners of a common L2 who speak different L1s diverge in their IL performance” (Jarvis 2000: 254). 
8 “is found where learners’ use of some L2 features can be shown to parallel their use of a corresponding L1 feature” (Jarvis 2000: 
255). 
632
‘indeed’ lead the reader to expect some new information” (Granger and Tyson 1996: 22). These examples provide evidence for the relevance of considering the factor lexical/semantic knowledge as a possible explanation for the non-native appearance of L2 writing. 
3.4. Within experimental design 
Although Jarvis’ method to approach transfer appears to be valid and reliable, an even more accurate way to confirm first language influence in second language production is to concurrently compare learners’ productions in their native and in their second language. In his study, starting from the assumption that “it seems premature to confirm transfer based on a similarity between a group writing in L1 and another writing in L2”, Kubota (1998) tries to find out whether there is a link between L2 text production patterns and L1 writing skills by “investigating whether individual students actually use similar rhetorical structures in L1 and L2 writing”, which, according to him, “could provide insightful data that might confirm or challenge the legitimacy of the transfer hypothesis.” In comparing the L2 writing of Japanese learners to both productions of native speakers, acting as the control group, and to productions of the same learners in their mother tongue (Japanese), Kubota comes to the conclusion that “good Japanese writers are able to produce well organized ESL essays” while “poor Japanese writers are unlikely to write effectively in English” and therefore that 
“the poor organizational quality often identified in ESL writing may not be so much the result of using ‘cultural conventions’ as it is a manifestation of the lack of ability to organize a coherent text in L1.” (Kubota 1998: 88) 
One has to recognize that the former conclusion could never have been drawn from the sole results of a corpus analysis, considering that these two opposed observations would have nullified each other. This points to the extreme care needed when analysing corpus results since the general patterns of a group of L2 learners of some given L1 observed in a learner corpus study, do not reflect the sum of the pattern(s) used by each individual. This highlights the relevance of considering the results from a ‘within experimental design’ point-of-view when trying to answer to question why discrepancies emerge between native and learners’ written production, as exemplified by the work of Kubota (1998). 
Even though Kubota's conclusions are appealing, we have nevertheless to keep in mind that Kubota (1998) has been focusing on general rhetorical features such as argumentation or text-orientation, thus working at the macrostructure of language, whereas others, such as Jarvis (2000) or Liu and Shaw (2001) have been dealing with lexical items, which are located at the micro level of language. It is therefore possible that Kubota's methodology would be less relevant in the study of lexical, phonological or morphological aspects of foreign language learning. In this regard, connectives take an intermediate position. While metadiscourse connectives occur at a more global level of language use, and can be seen as being part of the macrostructure of language, causal and contrastive connectives are located at a more local level of language use, and therefore belong to an intermediate structure of language. Considering connectives as rhetorical items will then allow us to lean on Kubota's methodology and to rightly hope it will show significant results. 
4. Interaction between text comprehension and text production 
One of the major interests of our study, while studying both L2 text comprehension and production, is to arrive at some conclusions concerning the possible correlation between the two. In the literature, a clear link has been established between reading and writing (Shanahan and Lomax 1986, Carson et al 1990). Shanahan and Lomax (1986) concluded for instance that a theoretical interactive model of the relationship between reading and writing, “in which reading knowledge could be used in writing and writing knowledge could be used in reading, provided the best description of the data”, which points to the interactive nature of both skills. Carson et al (1990) tried to put this theoretical model into practice and focused on the relation between reading and writing abilities across L1 and L2, and more precisely on the relations between first and second language reading and writing abilities, and between the reading and writing in the learners’ native and second language. Their findings point to a (significant) correlation between L1 and L2 reading and a (weak) correlation between L1 and L2 writing, whereas the relation between L1 reading and L1 writing and between L2 reading and L2 writing is highly dependent on language proficiency. Leaning upon 
633
Cummins (1981) they further argue that “transfer of capability emerges only after individuals attain a threshold level of L2 proficiency to permit cognately demanding language use.” This idea of threshold level of L2 proficiency matches the observation made by Degand and Sanders (2002) in their study of the impact of causal connectives on L1 and L2 text comprehension (see section 2). 
Transferring those observations to our specific study of connectives, we may expect a correlation between the impact of connectives on L2 text comprehension and connective usage in L2 text production. The question whether it occurs in both directions (interaction) or only from the one to the other has nevertheless to be left unanswered at this stage. One can imagine that learners benefiting from the presence of connectives in L2 text comprehension will use them more adequately in L2 text production, but reversing this statement, and claiming that learners who correctly use connectives in L2 text production will benefit more from their presence in text comprehension, does not give rise to an erroneous assumption either. Osborne (1998), by referring to the work of Crew (1990): “c'est souvent au moyen d'une multiplication des connecteurs que le scripteur inexpérimenté tente de donner un semblant de cohérence a des idées désordonnées9” and further arguing that “les étudiants eux-memes, quand ils sont lecteurs, ne sont pas dupes de cette stratégie10” even suggests that a significant overuse of connectives in foreign language production does not imply an absence of their effect in L2 text comprehension. This emphasizes that the relation between the impact of connectives on L2 comprehension and their usage in L2 production can go into completely different ways. It will be our work to determine which direction(s) it can take. 
5. Conclusion 
Our study of causal, contrastive and metadiscourse connectives focuses on two constituents of second language learning, notably L2 text comprehension and L2 text production. For the comprehension part studies, we will replicate the methodology used in Degand and Sanders (2002) since it has shown conclusive results. For the production studies, after the compilation an adequate Dutch learner corpus, we will try to make profit from the advantages of both corpus and experimental research. In keeping with Osborne (1998) who stated that 
“une étude purement quantitative des mots de liaison dans les productions d'étudiants peut attirer l'attention sur certaines différences entre l'utilisation qui est faite par les natifs et les non-natifs, mais les résultats obtenus sont tres sensibles a la nature et a la taille des corpus. Un examen plus détaillé et plus qualitatif est nécessaire pour comprendre pourquoi certains connecteurs paraissent mal utilisés, meme lorsqu'il ne sont pas franchement sur-utilisés” (Osborne 1998: 11)11, 
we will adopt a corpus-based method to come to the quantitative observations we could never arrive at without a corpus, but then, when focussing on the question why discrepancies between native and learner language occur, we will turn to a more experimental method which will help overcoming the shortcomings presented by corpus methods, for instance in offering a way to control more accurately a wide range of different variables. Among the variables we wish to manipulate, we find the rhetoric abilities of the participants in their mother tongue, which we will analyse by working with a 'within experimental design', comparing learners’ L2 productions with their L1 production, which will eventually allow us to account for individual differences and to be experimentally more accurate (Kubota 1998). 
The last step of our study will consist of a comparison between the impact of the connectives under investigation on L2 text comprehension and their usage in L2 text production to see if there exists an interaction between both aspects of second language learning. 
9 “Inexperienced learners often try to give their written productions in L2 some semblance of coherence in 
considerably increasing connective usage.” 
10 “Learners themselves, when reading, are not fooled by that strategy.” 
11 “A purely quantitative study of connective usage by learners may point to certain discrepancies between natives and learners use, 
but the observed results are sensitive to the corpus’ size and nature. A more detailed and qualitative consideration is needed to 
understand why some connectives seem to be misused, even though they are not really overused.” 
634
Ellis (1994) once claimed that “good research is research that makes use of multiple sources of data”. We can easily broaden the scope of this definition and apply it to a larger methodological framework by stating that good research is research that makes use of multiple sources of methods. 
6. Bibliography 
Carson JE, Carrel PL, Silberstein S, Kroll B, Kuehn PA 1990 Reading-writing relationships in first and second language. TESOL Quaterly 24(2): 245-266. 
Corkill A, Bruning R, Glover J, Krug D 1988 Advance organizers: retrieval context hypotheses. Journal of Educational Psychology 80: 304-311. 
Cummins J 1981 The role of primary language development in promoting educational success for language minority students. In Schooling and language minority students: a theoretical framework. Los Angeles: evaluation, dissemination and assessment center, California State University, pp 3-49. 
Cummins J 1984 Bilingualism and special education: issues in assessment and pedagogy. Clevedon, Avon: Multilingual Matters. 
Degand L, Lefevre N, Bestgen Y 1999 The impact of connectives and anaphoric expressions on expository discourse comprehension. Document Design 1(1): 39-51. 
Degand L, Sanders T 2002 The impact of relational markers on expository text comprehension in L1 and L2. Reading and Writing: An Interdisciplinary Journal 15(7-8): 739-758. 
Ellis 1994 The study of second language acquisition. Oxford, Oxford University Press. 
Granger 1994 The learner corpus: a revolution in applied linguistics. English Today 39(10): 25-32. 
Granger S, Tyson S 1996 Connector usage in the English essay-writing of native and non-native EFL speakers of English. World Englishes 15(1): 17-27. 
Granger S 1998 The computer learner corpus: a versatile new source of data for SLA research. In Granger S (ed.), Learner English on Computer, New-York, Addison Wesley Longman, pp 3-18. 
Halliday MAK 1978 Language as social semiotic. London, University Park Press. 
Hyland K 1998 Persuasion and context: the pragmatics of academic metadiscourse. Journal of Pragmatics 30: 437-488. 
Jackiewicz A 2002 Repérage et délimitation des cadres organisationnels pour la segmentation automatique des textes. In Proceedings of the CIFT'02: Colloque International sur la Fouille de Textes, Hammamet, pp 95-105. 
Jarvis S 2000 Methodological rigor in the study of transfer: identifying L1 influence in the interlanguage lexicon. Language Learning 50(2): 245-309. 
Kubota R 1998 An investigation of L1-L2 transfer in writing among Japanese university students: implications for contrastive rhetoric. Journal of Second Language Writing 7(1): 69-100. 
Liu ETK, Shaw PM 2001 Investigating learner vocabulary: a possible approach to looking at EFL/ESL learners’ qualitative knowledge of the word. IRAL 39: 171-194. 
Meunier F 1998 Computer tools for the analysis of learner corpora. In Granger S (ed.), Learner English on computer, New-York, Addison Wesley Longman, pp 19-37. 
635
Millis K, Graesser A, Haberlandt K 1993 The impact o connectives on the memory for expository texts. Applied Cognitive Psychology 7: 317-339. 
Osborne J 1999 L'utilisation des marqueurs de liaison en Anglais: étude inter-culturelle des productions d'étudiants. In Actes des journées d'étude du GEPED, Université de Paris VII, pp 97-109. 
Oversteegen LE 1997 On the pragmatic nature of causal and contrastive connectives. Discourse Processes 24: 51-85. 
Shanahan T, Lomax RG 1986 An analysis and comparison of theoretical models of the reading-writing relationship. Journal of Educational Psychology 78: 116-123. 
636

UNIVERSITY CENTRE FOR COMPUTER 
CORPUS RESEARCH ON LANGUAGE 
Technical Papers 
Volume 16 - Special issue. 
Proceedings of the 
Corpus Linguistics 2003 conference 
edited by 
Dawn Archer, 
Paul Rayson, 
Andrew Wilson, 
and 
Tony McEnery. 
ISBN 1 86220 131 5. 
Lancaster University (UK), 28 - 31 March 2003 

Table of contents 
Preface viiiMariko Abe : A Corpus-based Contrastive Analysis of Spoken and Written Learner Corpora: The Case of Japanese-speaking Learners of English 1Aduriz I., Aranzabe M.J., Arriola J.M., Atutxa A., Díaz de Ilarraza A., Ezeiza N., Gojenola K., Oronoz M., Soroa A., and Urizar R.: Methodology and steps towards the construction of EPEC, a corpus of written Basque tagged at morphological and syntactic levels for the automatic processing 10Khurshid Ahmad, Pensiri Manomaisupat, David Cheng, Tugba Taskaya, Saif Ahmad, Lee Gillam, Andrew Hippisley: The mood of the (financial) markets: In a corpus of words and of pictures 12Sandra M. Aluísio, Gisele M. Pinheiro, Marcelo Finger, Maria das Graças V. Nunes, Stella E. O. Tagnin: The Lacio-Web Project: overview and issues in Brazilian Portuguese corpora creation 14Dawn Archer, Tony McEnery, Paul Rayson, Andrew Hardie: Developing an automated semantic analysis system for Early Modern English 22Dawn Archer, Andrew Hardie, Tony McEnery, Scott Piao: A corpus of seventeenth-century English news reportage: construction, encoding and applications 32Bertol Arrieta, Arantza Díaz de Ilarraza, Koldo Gojenola, Montse Maritxalar, Maite Oronoz: A database system for storing second language learner corpora 33Jorg Asmussen: Towards a methodology for corpus-based studies of linguistic change: Contrastive observations and their possible diachronic interpretations in the Korpus 2000 and Korpus 90 General Corpora of Danish 42Eric Atwell: A New Machine Learning Algorithm for Neoposy: coining new Parts of Speech 43Eric Atwell, Paul Gent, Julia Medori, Clive Souter: Detecting student copying in a corpus of science laboratory reports: simple and smart approaches 48Francis Henrik Aubert, Stella E. O. Tagnin: A Corpus of Sworn Translations – for linguistic and historical research 54Bogdan Babych, Anthony Hartley, Eric Atwell: Statistical modelling of MT output corpora for Information Extraction 62Paul Baker, Andrew Hardie, Tony McEnery, and Sri B.D. Jayaram: Constructing Corpora of South Asian Languages 71Federica Barbieri: The “new” quotatives in American English: A cross-register comparison 81Marco Baroni and Silvia Bernardini: A preliminary analysis of collocational differences in monolingual comparable corpora 82Sabine Bartsch: Investigating cross-linguistic constraints on the premodification of adjectival past participles and desubstantival adjectives. A corpus-based study of English and German 92Kate Beeching: Synchronic and diachronic variation: the how and why of sociolinguistic corpora. 102Luisa Bentivogli, Christian Girardi, Emanuele Pianta: The MEANING Italian Corpus 103Julie Carson-Berndsen, Ulrike Gut and Robert Kelly: Discovering regularities in non-native speech 113 
ii
P. Beust, S. Ferrari, V. Perlerin: NLP model and tools for detecting and interpreting metaphors in domain-specific corpora 114Philippe Blache, Marie-Laure Guénot and Tristan van Rullen: A corpus-based technique for grammar development 124Birte Bös: Towards an integrated model of service encounters 132Roderick Bovingdon and Angelo Dalli: Statistical analysis of the source origin of Maltese 140Lou Burnard, Tony Dodd: Xara: an XML aware tool for corpus searching 142Marianna N. Christou: Expressions and structures of the delexical verb KAN [“MAKE” / “DO”] in Modern Greek language: A corpus-based approach to newspaper articles 145Ken Cosh and Pete Sawyer: Using natural language processing tools to assist semiotic analysis of information systems 155H. Cunningham, V. Tablan, K. Bontcheva, M. Dimitrov: Language engineering tools for collaborative corpus annotation 165Mark Davies: Annotation without lexicons: an alternative to the standard bootstrapping approach 174Joost van de Weijer: Consonant variation within words 184Debbie Elliott, Anthony Hartley and Eric Atwell: Rationale for a multilingual corpus for machine translation evaluation 191John Elliott and Debbie Elliott: The Human Language Chorus Corpus (HULCC) 201Jens Fauth, Hans-Jörg Schmid: Detecting gender-preferential patterns of linguistic features in face-to-face communication 211Valéria D. Feltrim, Sandra M. Aluísio, Maria das Graças V. Nunes: Analysis of the rhetorical structure of computer science abstracts in Portuguese 212Katerina T. Frantzi: Updating LSP dictionaries with collocational information 219Robert Gaizauskas, Lou Burnard, Paul Clough and Scott Piao: Using the XARA XML-Aware Corpus Query Tool to Investigate the METER Corpus 227Ana Llinares García: Repetition and young learners´ initiations in the L2: a corpus-driven analysis 237Sandrine Garnier, Youhanizou Tall, Sisay Fissaha, Johann Haller: Learner Corpora: Design, Development and Applications - Development of NLP tools for CALL based on learner corpora (German as a foreign language) 246Sara Gesuato: The company women and men keep: what collocations can reveal about culture 253Vojko Gorjanc: Tracking lexical changes in the reference corpus of Slovene texts 263Stefan Grondelaers, Dirk Speelman, Dirk Geeraerts: A corpus-based approach to informality: the case of Internet chat 264Leif Grönqvist and Magnus Gunnarsson : A method for finding word clusters in spoken language 265Xiaotian Guo: Between Verbs and Nouns and Between the Base Form and the Other Forms of Verbs – A Contrastive Study into COLEC and LOCNESS 274Le An Ha: A method for word segmentation in Vietnamese 282Silvia Hansen-Schirra: Linguistic enrichment and exploitation of the Translational English Corpus 288 
iii
Andrew Hardie: Developing a tagset for automated part-of-speech tagging in Urdu 298Nigel Harwood: Personal pronouns and academic writing: a multidisciplinary corpus-based critical pragmatic approach to EAP 308Laura Hasler, Constantin Orasan and Ruslan Mitkov: Building better corpora for summarisation 309Chris Heffer: Not KWIC but Quick: KeyWords in Court 319Kris Heylen and Dirk Speelman: A corpus-based analysis of word order variation: The order of verb arguments in the German middle field 320Knut Hofland: A web-based concordance system for spoken language corpora 330Shelley Ching-yu Hsieh: The Corpus of Mandarin Chinese and German Animal Expressions 332Susan Hunston: Frame, phrase or function: a comparison of frame semantics and local grammars 342Emi Izumi, Toyomi Saiga, Thepchai Supnithi, Kiyotaka Uchimoto, Hitoshi Isahara: The development of the spoken corpus of Japanese learner English and the applications in collaboration with NLP techniques 359Inés Jacob, Joseba Abaitua, Josu Gómez: Automatic feeding of translation memory tools 367Steven Jones, M. Lynne Murphy: Antonymy in Childhood: a corpus-based approach to acquisition 372Randall L. Jones: An Analysis of Lexical Text Coverage in Contemporary German 373Stig W. Jorgensen, Carsten Hansen, Jette Drost, Dorte Haltrup, Anna Braasch, Sussi Olsen: Domain specific corpus building and lemma selection in a computational lexicon 374Tomoko Kaneko: How non-native speakers express anger, surprise, anxiety and grief: a corpus-based comparative study 384Sachie Karasawa: Patterns of elaboration and interlanguage development: an exploratory corpus analysis of college student essays 394Hannah Kermes, Stefan Evert: Text analysis meets corpus linguistics 402Adam Kilgarriff: Linguistic Search Engine 412Paul Kingsbury: A methodology for inducing a chronology of the Pä li Canon 413Gerry Knowles, Zuraidah Mohd Don: Tagging a corpus of Malay texts, and coping with 'syntactic drift' 422Natalie Kübler and Cécile Frérot: Verbs in specialised corpora: from manual corpus-based description to automatic extraction in an English-French parallel corpus 429Toshihiko Kubota: A Study on Abridgement for Spoken Word Titles 439David YW Lee: Spoken Academic Lexicogrammar and Discourse Patterns 440Geoffrey Leech, Martin Weisser: Generic speech act annotation for task-oriented dialogues 441Agnieszka Lenko-Szymanska: The curse and the blessing of mobile phones - a corpus-based study into Polish and American rhetoric strategies 447Robert Liebscher and David Groppe: Rethinking context availability for concrete and abstract words: a corpus study 449Laura Löfberg, Dawn Archer, Scott Piao, Paul Rayson, Tony McEnery, Krista Varantola, Jukka-Pekka Juntunen: Porting an English semantic tagger to the Finnish language 457 
iv
Nadine Lucas, Bruno Crémilleux, Leny Turmel : Signalling well-written academic articles in an English corpus by text mining techniques 465Anke Lüdeling and Stefan Evert: Linguistic experience and productivity: corpus evidence for fine-grained distinctions 475Michaela Mahlberg: High frequency nouns in English: aspects of a grammatical description 484Belinda Maia: Constructing comparable and parallel corpora for terminology extraction - work in progress 485Manolis Maragoudakis, Katia Kermanidis and Nikos Fakotakis: Towards a Bayesian Stochastic Part-Of-Speech and Case Tagger of Natural Language Corpora 486Kevin Mark: Learner corpus building and a ‘living’ university foreign language curriculum 496Tony McEnery, Zhonghua Xiao: Fuck revisited 504Dan McIntyre, Carol Bellard-Thomson, John Heywood, Tony McEnery, Elena Semino and Mick Short: The Construction of a Corpus to Investigate the Presentation of Speech, Thought and Writing in Written and Spoken British English 513John McKenny: Seeing the wood and the trees: Reconciling findings from discourse and lexical analysis 523Magnus Merkel, Michael Petterstedt and Lars Ahrenberg: Interactive Word Alignment for Corpus Linguistics 533José María Guirao Miras Ana González Ledesma, Guillermo de la Madrid Heitzmann, Manuel Alcántara Plá, Antonio Moreno Sandoval: Relating lexical items to sociolinguistic features in a spontaneous speech corpus of Spanish 543Juan M. Montero and M. Mar Duque: ANESTTE: a writer's assistant for a specific purpose language 544Olga Moudraia: The Student Engineering Corpus: Analysing Word Frequency 552JoAnne Neff, Francisco Ballesteros, Emma Dafouz, Francisco Martínez, Juan-Pedro Rica: Formulating Writer Stance: A Contrastive Study of EFL Learner Corpora 562Diane Nicholls: The Cambridge Learner Corpus - error coding and analysis for lexicography and ELT 572Judy Noguchi, Thomas Orr, Yukio Tono: Using a dedicated corpus to identify features of professional English usage: What do “we” do in science journal articles? 582Attila Novák, Viktor Nagy, Csaba Oravecz: Corpus assisted development of a Hungarian morphological analyser and guesser 583Toshifumi Oba and Eric Atwell: Using the HTK speech recogniser to analyse prosody in a corpus of German spoken learners’ English 591Marija Omazic: THE METACOMMUNICATIVE SETTING OF PHRASEOLOGICAL UNITS AND THEIR MODIFICATIONS – EVIDENCE FROM THE BRITISH NATIONAL CORPUS 599Nelleke Oostdijk: Corpus linguistics meets language technology: deep syntactic parsing for question answering 603Maeve Paris: Extending computer-assisted text analysis techniques to the detection of source code plagiarism and collusion: assisting manual inspection 611Núria Gala Pavia, Salah Ait-Mokhtar: Lexicalising a robust parser grammar using the WWW 620 
v
Julien Perrez and Liesbeth Degand: On the combination of corpus-based and experimental methodologies in the study of causal, contrastive and metadiscourse connectives in L1 and L2 text comprehension and production 627Scott S.L. Piao and Tony McEnery: A Tool for Text Comparison 637James Pustejovsky, Patrick Hanks, Roser Saurí, Andrew See, Robert Gaizauskas, Andrea Setzer, Dragomir Radev, Beth Sundheim, David Day, Lisa Ferro and Marcia Lazo: The TIMEBANK Corpus 647Andrew Roberts and Eric Atwell: The use of corpora for automatic evaluation of grammar inference systems 657Juhani Rudanko: More on horror aequi: evidence from large corpora 662Sarah Rule, Emma Marsden, Florence Myles, Rosamond Mitchell: Constructing a database of French interlanguage oral corpora 669Geoffrey Sampson: Are we nearly there yet, Mum? 678Hans-Jörg Schmid, Jens Fauth: Women's and men's style: fact or fiction? New grammatical evidence 679Serge Sharoff: Methods and tools for development of the Russian Reference Corpus 680Bayan Abu Shawar and Eric Atwell : Using dialogue corpora to train a chatbot 681Gerardo Sierra, Alfonso Medina, Rodrigo Alarcón, César A. Aguilar: Towards the Extraction of Conceptual Information from Corpora 691Kiril Simov, Alexander Simov, Milen Kouylekov: Constraints for corpora development and validation 698Milena Slavcheva: Corpus shallow parsing: meeting point between paradigmatic knowledge encoding 706Nicholas Smith: A quirky progressive? A corpus-based exploration of the will + be + -ing construction in recent and present day British English. 714Harold Somers: Some Issues in the Mark-up of Handwriting in a Learner Corpus 724Dirk Speelman, Stefan Grondelaers, Dirk Geeraerts: A profile-based calculation of region and register variation: the synchronic and diachronic status of the national variants of Dutch 733Somayajulu G. Sripada and Ehud Reiter and Jim Hunter and Jin Yu: Exploiting a parallel TEXT - DATA corpus 734Asa M. Stepak: A proposed mathematical theory explaining the sequence of grammatical categories 744Petra Storjohann: The lexicographic use of corpora and computational tools for disambiguation 754Jozsef Szakos: Cultures and Corpora: Extracting Anthropological Information from Corpora of Formosan Endangered Languages 763Jun Arata Takahashi : Do we talk (or write?) differently over the Net?- A lexical enquiry into ‘a’ Net-EN - 764Kaoru Takahashi: A Study of Text Types and Register Variation in the British National Corpus 773Yuri Tambovtsev: The Structure of the Consonant Patterns in the Spanish Speech Sound Chain as a Clue of Typological Closeness 774 
vi
Yuri Tambovtsev: Phonological similarity between Basque and other world languages based on the frequency of occurrence of certain typological consonantal features 775Tess Yu-Shan Ke, Liang-Feng Chen, Chien-Chung Chen: Investigation on the uses of temporal subordinators by NS and NNS in academic spoken English 780Carole Tiberius, Dunstan Brown, Greville Corbett: Ambiguity in Russian Morphology 790Juhani Toivanen, Tapio Seppänen, Eero Väyrynen: Creation and utilisation of the MediaTeam Emotional Speech Corpus 791Yukio Tono: Learner corpora: design, development and applications 800Montserrat Civit Torruella, Ma Antonia Martí Antonín, Lluís Padró Cirera : Using hybrid probabilistic-linguistic knowledge to improve pos-tagging performance 810Patrick Tschorn, Anke Lüdeling: Morphological knowledge and alignment of English-German parallel corpora 818Francesca Vaghi, Marco Venuti: The Economist and The Financial Times. A study of movement metaphors 828Bertus van Rooy and Lande Schäfer: An evaluation of three POS taggers for the tagging of the Tswana Learner English Corpus 835Tamás Váradi: Shallow parsing of Hungarian business news 845Isabel Verdaguer and Anna Poch: Collocational and colligational patterns in lexical sets: A corpus-based study 852Maria Verde: Shedding light on SHED, CAST and THROW as nodes of extended lexical units 859Shih-Ping Wang: Mutual information and corpus-based approaches to reduplicative fixed expressions 869Julie Weeds and David Weir: Finding and evaluating sets of nearest neighbours 879David Wible, Ping-Yu Huang: Using learner corpora to examine L2 acquisition of tense-aspect markings 889Sandra Williams and Ehud Reiter: A corpus analysis of discourse relations for Natural Language Generation 899Andrew Wilson, Celia Worth: Building and annotating corpora of spoken Welsh and Gaelic 909Andrew Wilson, Celia Worth: Conceptual Glossaries of the Latin Vulgate Bible 918Andrew Wilson, Olga Moudraia: Quantitative or Qualitative Content Analysis? Experiences from a cross-cultural comparison of female students' attitudes to shoe fashions in Germany, Poland and Russia 919Martin Wynne, Rowan Wilson, Ylva Berglund: Virtual Corpora at the Oxford Text Archive 920Yang Xiaojun: Survey and Prospect of China's Corpus-Based Researches 930Debra Ziegeler, Sarah Lee: Analysing a Corpus-based Semantic Investigation of English Dialects 931Heike Zinsmeister, Ulrich Heid: Identifying predicatively used adverbs by means of a statistical grammar model 932 
vii
Preface 
The papers in this collection are based upon talks or poster presentations given at the Corpus Linguistics (CL2003) conference, held at Lancaster University between 28th and 31st March 2003 organised by members of UCREL, from the Departments of Linguistics & Modern English Language and Computing. The proceedings also includes those papers presented at the pre-conference workshop “Learner Corpora: Design, Development and Applications” organised by Yukio Tono and Fanny Meunier. The three other workshops produced separate proceedings. 
The conference attracted over 150 participants from the language engineering and corpus linguistics communities from many countries. The previous conference in the series, CL2001, was also held at Lancaster University in March 2001. 
Dawn Archer 
Paul Rayson 
Andrew Wilson 
Tony McEnery 
Lancaster University, March 2003. 
viii

The TIMEBANK Corpus 
James Pustejovsky 
Patrick Hanks 
Roser Saurí 
Andrew See 
Dept. of Computer Science 
Brandeis University 
jamesp@brandeis.edu 
patrick.hanks@rcn.com 
roser@cs.brandeis.edu 
aragorn@cs.brandeis.edu Robert Gaizauskas 
Andrea Setzer 
Dept. of Computer Science 
University of Sheffield 
R.Gaizauskas@dcs.shef.ac.uk 
A.Setzer@dcs.shef.ac.uk 
Fax: +44 (0) 114 22 21810 
Dragomir Radev 
University of Michigan 
radev@umich.edu Beth Sundheim 
SPAWAR Systems Center 
beth.sundheim@navy.mil 
David Day 
Lisa Ferro 
Marcia Lazo 
The MITRE Corporation 
day@mitre.org 
lferro@mitre.org 
mlazo@mitre.org 
Abstract 
While the mechanisms for conveying temporal information in language have been have been extensively studied by linguists, very little of this work has been done in the tradition of corpus linguistics. In this paper we discuss the outcomes of a research effort to build a corpus, called TIMEBANK, which is richly annotated to indicate events, times, and temporal relations. We describe the annotation scheme, the corpus sources and tools used in the annotation process, and then report some preliminary figures about the distribution of various phenomena across the corpus. TIMEBANK represents the most fine-grained and extensively temporally annotated corpus to date, and will be a valuable resource both for corpus linguists interested in time and language, and for language engineers interested in applications such as question answering and information extraction for which accurate knowledge of the position and ordering of events in time is of key importance. 
1. Introduction 
During 2002 an extended workshop, called TERQAS1, was held to address the problem of how to answer temporally based questions about the events and entities in text, especially news articles. It was motivated by the fact that questions like the ones below are not supported in a unified way by existing question answering (QA) systems: 
1. Is Gates currently CEO of Microsoft? 
2. When did Iraq finally pull out of Kuwait during the war in the 1990s? 
3. Did the Enron merger with Dynegy take place? 
Such questions cannot be answered without paying attention to the temporal properties, modalities and ordering of events. 
There has recently been a renewed interest in temporal and event-based reasoning in language, in particular when applied to information extraction and reasoning tasks (cf. ACL Workshop on Temporal and Spatial Reasoning, 2001; LREC workshop Annotation Standards for Temporal Information in Natural Language, 2002). One of the major differences between this work and earlier work in AI and computational linguistics on temporal reasoning and temporal semantics has been a focus on the identification of the real linguistic phenomena that are used to convey temporal information in text. This has led to a growing movement to build corpora annotated with temporal information – not simply times and dates, but events and temporal relations between events. Clearly an understanding of these phenomena, as they occur in real texts, is necessary to be able to design systems to answer questions such as those above. 
1 The TERQAS (Time and Event Recognition for Question Answering Systems) workshop was held at MITRE Bedford and Brandeis University. It was funded through the US National Regional Research Centre (NRRC) and fully sponsored by the US Advanced Research and Development Agency (ARDA). 
647
To advance this understanding, the TERQAS workshop had the following major objectives: 
1. To define and design a common meta-standard for the mark-up of events, their temporal anchoring, and how they are related to each other in news articles (to be called TimeML, for Time Markup Language). 
2. To create a gold-standard human-annotated corpus marked up for temporal expressions, events, and temporal relations, based on the TimeML specification (TIMEBANK). 
In this paper, we give a brief overview of the TimeML annotation scheme and its development (section 2), but we will focus mainly on the description of the TIMEBANK corpus (section 3). TIMEBANK contains 300 newswire articles with careful, detailed annotations of terms denoting events, temporal expressions, and temporal signals, and, most importantly, of links between them denoting temporal relations. This collection, the largest temporal-event annotated corpus to date, provides a solid empirical basis for future research into the way texts actually express and connect series of events. It will support research into areas as diverse as the semantics of tense and aspect, the explicit versus implicit communication of temporal relational information, and the variation in typical event structure across narrative domains (e.g. striking differences were observed by annotators between the typical event structure of business reports and the typical event structure of political, social, and military events). Finally, it will support research efforts to construct practical QA and information extraction systems which are sensitive to temporal information. 
2. Annotating the Corpus 
Unlike most previous attempts at event and temporal specification, TimeML separates the representation of event and temporal expressions from the anchoring or ordering dependencies that may exist in a given text. There are four major data structures that are specified in TimeML (Ingria and Pustejovsky, 2002, Pustejovsky et al., 2003): EVENT, TIMEX3, SIGNAL, and LINK. These are described in some detail below. The features distinguishing TimeML from most previous attempts at event and time annotation are summarized below: 
1. Builds on the TIMEX2 guidelines for temporal expressions (Ferro et al., 2001) but introduces, for example, temporal functions to allow intensionally specified expressions: three years ago, last month (TIMEX2 annotates these as well, but requires the annotator to supply an ISO value). 
2. Identifies signals determining interpretation of temporal expressions: 
a. Temporal Prepositions: for, during, on, at; 
b. Temporal Connectives: before, after, while. 
3. Identifies all classes of event expressions: 
a. Tensed verbs: has left, was captured, will resign; 
b. Stative adjectives: sunken, stalled, on board; 
c. Event nominals: merger, Military Operation, Gulf War. 
4. Creates dependencies between events and times: 
a. Anchoring: John left on Monday. 
b. Orderings: The party happened after midnight. 
c. Embedding: John said Mary left. 
2.1. The Conceptual and Linguistic Basis 
Before the annotation scheme is described in detail, it is necessary to make clear what kind of temporal entities and relations we suppose exist. The main entities are similar to the ones in Setzer (2001), on which the development of TimeML was based. They are events, times, and relations holding between events and events and times. 
Events are taken to be situations that occur or happen. They can be punctual (John reached the summit) or last for a period of time (John walked up a mountain). Events are generally expressed by the means of tensed or untensed verbs, nominalisations, adjectives, predicative clauses, or prepositional phrases. 
648
Predicates describing states or circumstances in which something obtains or holds true are also considered as events. However, only those stative predicates are marked up that participate in an opposition structure in a given text. For example, one can imagine that the state described in the expression New York is on the east coast holds true for a time span that is considerably longer than the time span containing the events of focal interest in a typical newswire text. This expression would not be annotated. An expression like the people on board the airbus, appearing in a newswire text about a crash of that airbus, can be seen as describing a state that would change within the time span covered by the text – and would thus be annotated. 
Times may be either points, intervals, or durations, and may be referred to by fully specified temporal expressions like June 11, 1989, underspecified temporal expressions such as Monday, intensionally specified expressions like last week, and duration expressions like three months or two years. 
Relations can hold between events and events and times. In order to correctly identify the temporal order of events, it is necessary to identify more than temporal relations in a text. Subordinate relations, for example, can indicate that an event is hypothetical and thus should receive a different temporal interpretation. There are also aspectual relations (John started to run), or relations representing polarity (John did not run) and temporal quantification (John ran three times). 
There are three types of relations included in TimeML: temporal relations, subordinate relations, and aspectual relations. The full set is detailed in section 2.3. 
2.2. The Annotation Scheme 
With the conceptual framework in place, we can now proceed to present the XML-based TimeML annotation scheme. This is a brief overview of the annotation scheme; for more detailed information please see Pustejovsky et. al. (2003). 
Annotating Events 
Events are marked up by annotating a representative of the event expressions, usually the head of the verb phrase. Note that generics like Use of corporate jets for business travel is legal are not tagged as events. The attributes of events are a unique identifier, the event class, tense, and aspect. The following fully annotated example for an event expression illustrates the even annotation. 
All 75 passengers <EVENT eid="1" class="OCCURRENCE" tense="past" aspect="NONE"> died </EVENT>. 
Events are subcategorised into the following classes: occurrence, perception, reporting, aspectual, state, intensional state, intensional action, and modal. 
Annotating Times 
The annotation of times was designed to be as compatible with the TIMEX2 time expression annotation guidelines (Ferro et al., 2001) as possible. The XML tag for time expressions within TimeML is called TIMEX3 to distinguish it from both the tags in Setzer (2001) and TIMEX2. TIMEX3 has the following attributes: 
• A unique identifier: tid, 
• The attribute functionInDocument indicates the function of the TIMEX3 in providing a temporal anchor for other temporal expressions. For example, the date of a newspaper article is often used to anchor underspecified temporal referring expressions. 
• A TIMEX3 object is of a certain type, either a date, a time, or a duration. 
649
• Time expressions like last week are analysed by using temporal function and the attribute temporalFunction indicates that the TIMEX3 is of this kind. Which temporal function is to be used, is registered in the attribute TemporalFunctionID. 
• The actual value of a temporal expression, for example 1998, is stored in the value attribute. 
• Temporal expressions often have to be analysed with respect to another time or an event. The ID of these anchors can be found in anchorTimeID and anchorEventID. 
Straightforward time expressions like July 1966 are annotated as follows: 
<TIMEX3 tid="1" type="DATE" value="1966-07" temporalFunction="false"> July 1966 </TIMEX3> 
This kind of annotation is well known from earlier annotation schemes adopted in TIDES (Ferro et. al., 2001) and the MUC competitions (Message Understanding Conferences, see MUC6 (1995) and MUC7 (1998)). One of the major contributions of TimeML is the treatment of temporal functions, which allows time-value dependent algorithms to delay the computation of the actual (ISO) value of the expression. The following informal paraphrase of some examples illustrates this point, where DCT is the Document Creation Time of the article. 
• last week = (predecessor (week DCT)) 
That is, we start with a temporal anchor, in this case, the DCT, coerce it to a week, than find the week preceding it. 
• the week before last = (predecessor (predecessor (week DCT))) 
Similar to the previous expression, except that here we go back two weeks. 
Annotating Signals 
The SIGNAL tag is used to annotate sections of text, typically function words, that indicate how temporal objects are to be related to each other. The functionality of the SIGNAL tag was introduced by Setzer (2001). In TimeML it is also used to mark polarity indicators such as not, no, none, etc., as well as indicators of temporal quantification such as twice, three times, and so forth. Signals have only one attribute, a unique identifier. The following annotated example illustrates the annotation of a signal: 
Two days <SIGNAL sid="1" before </SIGNAL> the attack … 
Annotating Temporal and other Relations 
To annotate the different types of relations that can hold between events and events and times, the LINK tag has been introduced. This marks a significant departure from Setzer (2001). There are three types of link tags. 
1. A TLINK or Temporal Link represents the temporal relationship holding between events or between an event and a time, and establishes a link between the involved entities making explicit if they are: before, after, includes, is included, holds, simultaneous, immediately after, immediately before, identity, begins, ends, begun by, ended by. 
2. An SLINK or Subordination Link is used for contexts introducing relations between two events, or an event and a signal. SLINKs are of one of the following sorts: Modal, Factive, Counter-factive, Evidential, Negative evidential, Negative. 
3. An ALINK or Aspectual Link represents the relationship between an aspectual event and its argument event. Examples of the aspectual relations to be encoded are: initiation, culmination, termination, continuation. 
A TLINK records the relation between two events, or an event and a time. Thus, the attributes include the IDs of the source and the target entity, the ID of the signal representing the type of relation, and the relation type. The attributes for an SLINK include the IDs of the subordinating and the subordinated events, the ID 
650
the signal indicating the type of subordination, and the type itself. The attributes of an ALINK include the ID of the aspectual event, the ID of the argument event, and the ID of a signal indicating the link. 
Examples of LINK annotations are: 
• The expression John taught on Monday states that there is an event taught, a time Monday and that the temporal relation holding between these two entities is that taught is temporally included in Monday – as signalled by the signal on. This is captured by the annoation <TLINK eventInstanceID=”2” relatedToTime=”4” signalID=”4” relType=”IS_INCLUDED”/>2. 
• The expression John said he taught includes a said event, and a taught event, and a evidential subordinate relation holds between them. This is annotated as follow: <SLINK eventInstanceID=”3” subordinatedEvent=”4” relType=”EVIDENTIAL”/>. 
• In the expression John started to read, the started event is of an aspectual nature and indicates that the read event has started (but not necessarily finished). This can be annotated like this: <ALINK eventInstanceID=”5” relatedToEvent=”6” relType=”INITIATES”/>. 
A full worked example can be seen in figure 1. 
Figure 1: A full worked example of TimeML annotation 
3. The Corpus 
3.1. The Text Sources 
The texts in the TIMEBANK corpus were chosen to cover a wide variety of media sources from the news domain and come from a variety of sources: 
• DUC (TIPSTER) texts from the Document Understanding Conference corpus cover areas like biography, single and multiple events (for example dealing with news about earthquakes and Iraq). This covers 12% of the corpus, or 35 texts; 
• Texts from the Automatic Content Extraction (ACE) program come from transcribed broadcast news (ABC, CNN, PRI, VOA) and newswire (AP, NYT). These are 17% (50) texts and 16% (49) texts of the corpus, respectively. One of the reasons for chosing the ACE data is that they were already tagged according to the TIDES time expression guidelines, which was helpful in developing the TIMEX3 tag. 651 

There was <SIGNAL sid="8"> no </SIGNAL> <EVENT EID="57" CLASS="STATE"TENSE="PAST" ASPECT="NONE"> hint of trouble</EVENT> <SIGNAL id="11">in </SIGNAL> the last<EVENT class="OCCURRENCE" aspect="NONE"eid="10" tense="NONE"> conversation </EVENT> between controllers andTWA pilot Steven Snyder. But <TIMEX3 TID="58" val="PT1M30S"type="DURATION" temporalFunction="false"> a minute and a half</TIMEX3> <SIGNAL SID="59"> later </SIGNAL>, a pilot from a nearbyflight <EVENT aspect="NONE" eid="18" tense="PRESENT"LASS="REPORTING"> calls </EVENT> in. C 
<SLINK eventInstanceID=”eiid57” signalID=”8” relType=”NEGATIVE”/> 
<TLINK eventInstanceID=”eiid57” relatedToEvent=”eiid10” signalID=”s11” relType=”IS_INCLUDED”/> 
<TLINK eventInstanceID=”eiid18” relatedToEvent=”eiid10” signalID=”s59” relType=”AFTER” magnitude=”t58”/> 
2 Note that the event instance ID is automatically created by the annotation tool used. 

• Propbank (Treebank2) texts are Wall Street Journal newswire texts, covering 55% or 166 texts of the corpus. 
3.2. The Annotation Effort 
The annotation of TIMEBANK has been carried out in two separate stages: in the first stage, 70% of the corpus was annotated (210 documents), and in the second stage 30% of the documents were annotated (90 documents). The initial stage was carried out by 5 annotators of remarkably different profiles with regards to their linguistic background. All of them however had participated in the development of the TimeML annotation scheme. The group of annotators for the second stage comprised 45 computer science undergraduate and graduate students, from a course on Commonsense and Temporal Reasoning at Brandeis University. 
The annotation in the initial stage was carried out during several annotation-intensive weeks, which were preceded by some sessions of plenary discussion in order to attain a maximum level of agreement among annotators. As for the second stage, the annotation was developed by students who had no prior familiarization with TimeML, and thus some training by the previous annotators was required. In addition, each of the documents annotated at this stage was reviewed to guarantee the quality of the annotation. 
The annotation of each document involved a pre-processing step in which some of the events and temporal, modal and negative signals were tagged. When possible, the information concerning event class, tense and aspect of events was also introduced at that point. 
The output resulting from the pre-processing was checked during the following human annotation step, and completed with the introduction of other signals and events, time expressions, and the appropriate links among them. The average time to annotate a document of 500 words by a trained annotator is 1 hour. 
3.3 The Annotation Tool 
The TimeML data annotated within the TERQAS Workshop were annotated using a variety of annotation and pre-processing tools3. There were two major pre-processing steps. One step involved performing temporal expression tagging that annotated simple temporal expressions according to the TimeML guidelines. 
This was performed to reduce the amount of manual labour required of the annotator and to help call their attention to relevant temporal expressions. A second pre-processing step involved running a modified version of the Alembic NLP system (Vilain and Day 1996) to generate likely event anchors, including verb phrases and their associated TimeML modalities (based on tense and aspect), as well as a modest number of nominalised event references. This data was also pre-processed running software developed to look inside the verb chunks and label them for tense, aspect and voice. 
At this point the data could be loaded into a modified version of the Alembic Workbench annotation tool. The "relation" tagging table provided in the Workbench was used to edit and create new event tags, create SLINK tags where appropriate, and apply a first pass of temporal relation tags to the textual data. When a text is loaded into the tool, the text itself is shown in one window, which shows the results of the pre-processing via coloured tags. These tags can be edited or deleted, and new tags can be introduced. Links, which are shown in a second window, can be created by selecting tags in the text window and inserting these into the link window. Figures 2 and 3 show screenshots of the tool. 
3 The authors would like to acknowledge Jose Castano, Marc Verhagen, and Luc Bélanger for their work on developing the tools described in this section. 
652
Figure 2: Screenshot 1 of the annotation tool 
Figure 3: Screenshot 2 of the annotation tool 
One aim of the annotation exercise is to allow the temporal relation between every pair of events or of events and times to be determined, insofar as this information is in the text. Because of the large number of possible temporal relation pairs that can be derived from even relatively short texts, the data entered in the steps described above was then passed on to a customized annotation tool, the semi-graphical temporal constraint tagger. This tool was arranged so that for any given anchoring event, all the other events are arrayed in lists to the left, right, above, or below, depending on the tools ability to establish their temporal relations. As the annotator assigns distinct temporal relations to particular event pairs, the tool applies a temporal closure algorithm to compute all the other temporal relations entailed from this new constraint, 
653
thereby reducing the actual number of pairwise constraints that must be directly specified by the annotator (see Setzer (2001) for more details, on which this tool is based). 
At the end of this process all of the TimeML information is represented in XML tags embedded directly in the original source document. A separate utility was able to render this XML into a web-based graphical representation. This graphical rendering allows the user to make subsequent graphical modifications to the visualization – but not to edit the underlying temporal or event information. A new fully graphical annotation tool that will support all of these distinct annotation tasks/steps within a single interface is under construction. 
3.4 Corpus Statistics 
The following figures give an idea of the density of the annotation and the distribution of the different tags throughout the corpus. The data presented below was obtained from 60% of the corpus comprising only texts annotated by the 5 initial annotators. We would expect this sample to be representative for the whole corpus. Table 1 shows the total number of Event, Timex3, and Signal tags, together and individually, and the proportion of words that fall into these classes. 
Count Proportion of words Words 68555 100% Tags (all 3 kinds) 11206 16.3% Events 7571 11.0% Timex3 1423 2.1% Signal 2212 3.2% 
Table 1: Proportion of tags 
The distribution of the different kinds of events and time expressions (Timex3 tags) can be observed in Tables 2 and 3, respectively. There are 1966 signals in the part of TIMEBANK which was analysed, but they are not shown here in a separate table because TimeML does not explicitly distinguish signals into different classes. They are, however, implicitly subcategorised by the links in which they occur. For example, signals being referred to by TLINKS (or temporal links) can be classed as temporal signals. This subcategorisation of signals has not been done yet. 
Event Class Count Frequency Occurrence 3933 51.90% State 1151 15.20% Reporting 954 12.60% i-action 636 8.50% i-state 562 7.40% Aspectual 256 3.40% Perception 40 0.50% missing_class 39 0.50% total 7571 100.00% 
Table 2: Distribution of event classes 
Timex3 Type Count Frequency Date 975 68.50% Duration 314 22.10% Time 80 5.60% missing_type 54 3.80% total 1423 100.00% 
Table 3: Distribution of TIMEX3 types 
654
Finally, Table 4 gives the total count and relative frequency for each kind of link: 
Link type Count Relative Frequency Link 8242 100.00% Tlink 5132 62.2% Tlink:Before 1183 14.3% Tlink:IsIncluded 973 11.8% Tlink:After 701 8.5% Tlink:Simultaneous 633 7.7% Tlink:Identical 624 7.6% Tlink:Included 568 6.9% Tlink:Holds 282 3.4% Tlink:Ends 67 0.8% Tlink:Begins 41 0.5% Tlink:I-After 32 0.4% Tlink:I-Before 28 0.3% Slink 2857 34.7% Slink:Modal 1437 17.5% Slink:Evidential 1072 13.0% Slink:Negative 271 3.3% Slink:Counter-Fact 50 0.6% Slink:Neg-Evidential 27 0.3% ALink 253 3.1% 
Table 4: Distribution for LINK types 
3.5 The Question Corpus 
One of the research areas to which TimeML aims to contribute is Question Answering (QA) – temporal question answering in particular. Temporal questions can be broadly classified into two categories: 
1. Questions that ask for a temporal expression as an answer, like When was Clinton president of the United States? When was Lord of the Rings: The Two Towers released? At what time does the summit start? How often did Elvis perform live on stage? We call this type explicit. 
2. Questions that either use temporal expression to ask for a non-temporal answer or that ask about the relations holding between events. Examples are Who was president of the United States in 1990? Did world steel output increase during the 1990s? Did sugar prices rise during WWI? Who was CEO of IBM when it bid to take over Apple? We call this type of temporal question implicit. 
To evaluate the usefulness of TimeML for (temporal) QA, a question corpus of 50 questions has been created, which was annotated according to a specially developed annotation scheme. This scheme allows features such as the type of the expected answer and the volatility of the answer (i.e. how often it changes) to be annotated. The questions contained in the corpus cover both types mentioned above. Example questions are: 
• When did the war between Iran and Iraq end? 
• When did John Sununu travel to a fundraiser for John Ashcroft? 
• How many Tutsis were killed by Hutus in Rwanda in 1994? 
• Who was Secretary of Defense during the Gulf War? 
• What was the largest U.S. military operation since Vietnam? 
655
4. Conclusion and Future Work 
In the foregoing we have briefly described the TimeML annotation scheme for annotating times, events and temporal relations in text and we have presented the TIMEBANK corpus, a 300 text corpus annotated according to the TimeML scheme. A first version of TIMEBANK will be released later in 2003 for general use via the Linguistic Data Consortium (LDC). 
Since the corpus creation effort is just coming to a close, there has as yet been no time to analyse the corpus; the statistics given in section 3.4 do not represent such an analysis, but only a very preliminary scoping of the corpus. However, we anticipate that the corpus will allow a whole new range of questions, both theoretical and practical, to be pursued. For example, it will become possible to study to what extent temporal ordering of events is conveyed explicitly through signals, such as temporal subordinating conjunctions, and to what extent implicitly through the lexical semantics of the verbs or nominalizations expressing the events. From a practical computational perspective, it will become possible to consider training and evaluating algorithms which determine event ordering and time-stamping, and to explore their utility in question answering. 
We are under no illusions that TimeML is as yet fully comprehensive and without anomalies or that TIMEBANK has been completely and correctly annotated. Further revision of the annotation scheme and of the corpus is inevitable. However, we believe that together TimeML and TIMEBANK form a significant contribution to the study of how temporal information is expressed in natural language, and a significant step for corpus linguistics in the direction of the semantic study language. 
5. References 
Ferro L, Mani I, Sundheim B, and Wilson G 2001 TIDES Temporal Annotation Guidelines, Version 1.0.2 MITRE Technical Report, MTR 01W0000041. 
Ingria R and Pustejovsky J 2002 TimeML Specification 1.0 available at http://www.time2002.org. 
MUC6 1995 Proceedings of the Sixth Message Understanding Conference (MUC-6), Morgan Kaufman. 
MUC7 1998 Proceedings of the Seventh Message Understanding Conference (MUC-7), available at http://www.itl.nist.gov/iaui/894.02/related_projects/muc/proceedings/muc_7_toc.html. 
Pustejovsky P, Castano, J, Ingria R, Saurí R, Gaizauskas R, Setzer A, and Katz G TimeML: Robust Specification of Event and Temporal Expressions in Text. In: Proceedings of the IWCS-5 Fifth International Workshop on Computational Semantics, 2003. 
Setzer, A. 2001 Temporal Information in Newswire Articles: an Annotation Scheme and Corpus Study, Unpublished PhD thesis, University of Sheffield. 
Vilain M and Day D 1996 Finite-State Parsing by Rule Sequences International Conference on Computational Linguistics (COLING-96). Copenhagen, Denmark. 
656

The use of corpora for automatic evaluation of grammar inference systems 

Andrew Roberts 
andyr@comp.leeds.ac.uk Eric Atwell 
eric@comp.leeds.ac.uk 
School of Computing 
University of Leeds 
Leeds 
LS2 9JT 
United Kingdom 
Abstract 
The evaluation of grammar inference systems is clearly a non-trivial task, as it is possible to have more than one correct grammar for a given language. The ‘looks good to me’ approach, carried out by computational linguists analysing their own grammar inference system results, has prevailed for many years. This paper explores why this method has been so popular, in terms of its strengths, and also why it is no longer adequate as a reliable means to measuring performance. Corpus based methods, that can be performed automatically, are investigated to see how they can meet the needs of this difficult problem. 
1 Introduction 
In the past few years, the Natural Language Learning community have produced systems targeted at the complex task of Grammar Inference (GI): automatically inferring or learning grammatical descriptions of a language from a corpus of language examples. A low-level GI task is to group or cluster words into tentative categories according to their distributional behaviour, e.g., Atwell and Drakos (1987), Hughes and Atwell (1994) and Roberts (2002). A more ambitious aim of GI is to propose grammatical phrase structure; a number of mature solutions have emerged, for example, GraSp (Henrichsen 2002), CLL (Watkinson and Manandhar 2001a), ABL (van Zaanen 2001) and EMILE (Adriaans 1992). All systems mentioned aim to be unsupervised, and thus can only rely on raw (unlabelled) text for their learning process. Results from these systems are promising – a variety of linguistic phenomena are induced. 
Despite the advances in this field, the issue of thorough evaluation has been poorly address. Evaluation within Natural Language Processing tasks in general is problematic, not least because there is no obvious single correct output to measure against. For example, for PoS-tagging and parsing, different linguists advocate different tagsets and parsing schemes, making it difficult to compare accuracy metrics (Atwell 1996; Atwell et al. 2000). Ambiguity poses a similar threat in GI: the basic problem is given a training corpus, there is no single correct grammar that represents it. In the majority of systems, a ‘looks good to me’ approach has been used: success is illustrated by presenting some grammar classifications or structures proposed by the system which appeal to the linguist's intuition. 
There is a need to develop methods for consistent evaluation of GI systems. Without a reliable way of determining the performance of such systems, it will become increasingly difficult to assess how competently they are doing the task they are supposed to do. Nor will it be trivial to compare two or more systems, which would be very valuable in deciding which techniques and algorithms work best for Natural Language Learning. 
This paper investigates the feasibility of harnessing corpora that can be used to develop a standard mechanism (which aims to be reliable and accurate) for evaluating GI systems in the future. The next section explores the ‘looks good to me’ approach mentioned earlier to understand why it is so widely used, its merits and any shortcomings. Section three introduces various approaches for evaluating GI. A discussion takes place in section four to highlight important issues from the approaches reviewed in this paper. Section five provides brings the paper to a close with a conclusion. 
2 Looks good to me 
657
The success of this approach is essentially its apparent simplicity. A system performs its GI procedures on a piece of unstructured text, and its resulting grammar can be analysed by a computational linguist, generally the computational linguist who built the Grammar Inference system under scrutiny. A qualitative evaluation takes place based on the linguistic intuitions of the evaluator, by highlighting features or structures in the learner output which look “good”, or reminiscent of structures in a recognised linguistic theory. Of course, the apparent simplicity is due to the fact that a linguist possesses the required skills and experience to easily deduce whether the grammar in question contains a plausible structure. Researchers who came into the field of GI from a computing/AI/machine learning background are less likely to be as proficient at evaluating grammars using this approach. 
‘Looks good to me’ is an important method of evaluation, and certainly should not be discredited just due to its lack of automation. Verification of a system carried out independently by one or more linguists would provide – in most cases – a more reliable measure of performance. It is also arguably the most resource efficient, since it can be evaluated on different languages without the need of structured corpora (van Zaanen 2001). Examples of this method in use can be found in. (Finch and Chater 1992; Losee 1996; Vervoort 2000; Henrichsen 2002). 
However, the method has many disadvantages. Evaluation of this nature is mainly conducted by the developer of the system rather than someone independent to the work. Thus, there is a high chance of bias whereby the systems successes are highlighted and shortcomings are glossed over, making it almost impossible to gain an accurate picture of system performance. Even without any bias, the process is time consuming and would rely on the subjectivity of the expert(s), and may also be prone to unknown external factors that can affect humans. And finally, it does not offer the facility for comparing different systems which would be of great benefit. 
3 Automatic evaluation 
Ideally, the process of evaluation should be performed automatically which will save on time and on the necessity of an expert linguist in the chosen language. This section focuses on methods that harness corpora for this purpose. 
3.1 ‘Gold standard’ treebank 
This method is currently the most common to be adopted. It works by extracting the original natural language sentences from an existing treebank (the ‘gold standard'), and using it as input to a given GI system. The structured sentences produced by the GI system are then compared to the structure found in the original treebank. Common metrics include recall (measures the completeness of the learned grammar) and precision (measures the correctness of the learned grammar) that can be calculated to give an objective measure of system performance. This method has been used in (Brill 1993; Déjean 2000; van Zaanen 2001) 
Although on the surface, this method appears sound, and simple to implement, there are in fact many issues that cause it to be insufficient for it to be reliable and accurate. Treebanks are expensive (both in time and money) to create, and so they are not in plentiful supply. The material within also tends to be exclusive to a particular domain, e.g., the ATIS treebank consists of sentences on questions and imperatives on air traffic, or the Penn Treebank is taken mostly from articles in the Wall Street Journal. If a GI system is not designed to learn from raw text of the same genre or subject,, then it is not an ideal candidate for a ‘gold standard'. The way in which the treebank has been structured poses a problem. Just because it has been labelled the ‘gold standard’ does not necessarily mean that anything that differs is wrong – it may simply be different, but equally correct. However, such differences will clearly affect the measure of the systems’ performance. It does also beg the question about whether certain treebanks are credible enough to be used for the purposes of evaluation. One must be confident that the quality and validity of an annotated is sufficiently high, otherwise, it is pointless evaluating systems with corpora that contain errors or are not well structured. 
On a slightly more technical level, a large step to overcome is the likely discrepancy between the actual annotation schemes of the GI system to be evaluated and the ‘gold standard'. For example, an 
658
increasingly common grammar being employed within GI is the categorial grammar (Ajdukiewicz 1935). However, there is no corresponding treebank whose annotation formalism uses CG, which means the only useful measurements that can be acquired from this mismatch are metrics such as crossing-brackets, but so much information is wasted. Therefore, a system of translation is required to either convert the original treebank to the annotation scheme used by the GI system, or the other way around, in order for a more precise comparison to commence.. This was the tactic taken by Watkinson and Manandhar (2001b) in their evaluation of CLL, whereby they set about establishing a ‘gold standard’ by translating the Penn Treebank annotation scheme into one using categorial grammar markup . 
3.2 Multi-annotated corpora 
To overcome some of the main disadvantages of the ‘gold standard’ approach, the next logical step is to develop a multi-annotated corpus, as exemplified in Atwell et al. (2000). The premise is that the same corpus is parsed by a variety of systems, rather than merely one as all common treebanks are. Providing all parses are trusted to be correct, then upon evaluation of a GI system, it is less likely to be penalised just because it produces different but equally correct parses, because it is being compared to a variety of such parses. 
This step should make the evaluation fairer, however, it does add a new set of complexities. Not least due to the aforementioned annotation translation problem, which multiplies for each unique scheme used by the various parsers. Ideally, the same annotation would be used for all parses within the treebank – the ‘gold standard’ annotation scheme. But this too is fraught with difficulties, especially as it would be much simpler to accomplish a multi-annotated corpus by using off-the-shelf parsers. 
Another issue is how to compare a given parsed sentence to be evaluated with a set of known correct parses. The best case is that a perfect match will be found. The worse case is no match. However, for many cases, it will be somewhere in between, where a number of partial matches are likely to exist. A best match approach seems logical, i.e., of the candidate parses, the one that is the most similar is the one that is then considered for the actual metric calculations. 
3.3 Multi-corpora 
In a similar vein to the above approach, the idea for using more than one corpus is primarily to avoid the domain specific issue that was addressed earlier (see section 3.1). Unlike other corpora, such as the Brown or LOB, which span many subjects in different contexts (newspaper articles, novels etc), well known treebanks fail to compare in size and variety. Therefore, this method aims to take advantage of smaller individual treebanks and combine into a single, large one. This would certainly make it fairer for general purpose learning systems, although perhaps there is less of a need for more focused systems. 
Having said that, an alternative tactic is to treat each of the contributing corpora as individual and each as with the normal ‘gold standard’ method. You would then have a more detailed type of benchmark style scoring system, where not only is there an overall performance measure against all the corpora on test, it can also be seen on which type of corpora the GI system works best (or worse) on. 
Of course, the most thorough approach would be if the smaller treebanks were also multi-annotated as described in the previous section. A ‘multi-multi-annotated corpus’ – if you like – would be a magnificent resource. This subtly shifts the aim of evaluation: the result or output is not a single overall “accuracy score”, but an exploration of a range of parsing schemes and genres to highlight similarities and differences between GI output and a variety of accepted linguistic analysis schemes. Ideally such an analysis could be facilitated by a parsed-corpus exploration toolkit: a tool to compare parses in GI output and linguist-annotated multi-multi-corpus, and automatically highlight differences. 
4 Discussion 
There does not seem to be a perfect solution to the problem of reliable evaluation, whether it be manual or automatic. GI and parsing is very subjective, and it is unlikely that a method will be created that 
659
will satisfy everyone. However, what should be unanimous is that a system of evaluation needs to exist. It may not be perfect, but at least a fair and consistent scheme can be created. 
Unfortunately, it is clear that the most thorough automated evaluation approaches could be such a burden to developers to implement that they will opt for a simpler approach. Which is why it may be worthwhile to outsource the task of creating the ultimate ‘gold standard’ into a research project within its own right. The creation of a collection of multi-annotated corpora as the central resource, as well as a variety of translation interfaces for commonly used annotation schemes, that allow easy comparison between output of a GI system and the ‘gold standard’ corpus. It could be packaged like an evaluation toolkit – a black box that is publicly available, easily accessible, and simple to add on as the final phase within the pipeline of the GI system. 
Alternatively, a multi-multi-annotated corpus could be achieved by getting the GI community – or at least a group of GI researchers – to work together, each contributing their preferered “target” analysis schemes. Sutcliffe et al (1996) describes an analogous evaluation exercise for a range of (non-ML) parsers: each researcher was asked to “bring along” their parser analyses of an agreed set of test sentences to a joint workshop, and then highlight and discuss successes (and failings) of their systems. 
Perhaps in a future GI workshop, GI researchers could be invited to bring along their preferred evaluation treebanks to challenge each other, and to each present both “looks good to me” and quantitative evaluations of systems for comparison. 
5 Conclusion 
The ‘looks good to me’ approach, despite the critical slant within this paper, is not an inferior one. However, it will not meet the demands for robust NLL evaluation. Which is why corpus based approaches have been presented as the most feasible and reliable way of essentially trying to emulate ‘looks good to me’ whilst eliminating bias and providing consistency. 
There is a great need for researchers within the field to begin addressing accurate evaluation techniques. The current ‘gold standard’ method (as described in section 3.1) is too basic and will simply favour GI systems that behave similar to the way the ‘gold standard’ treebank was parsed. Greater flexibility is required, and a method like the multi-multi-annotated corpus (see section 3.3) will be a beneficial innovation to do just that. Instead of replacing “looks good to me”, the two approaches should be combined, so that evaluation is no longer a simple quest for a single “score”, but instead an exploration of strengths (and weaknesses) of GI systems. 
References 
Adriaans P W 1992 Language learning from a categorial perspective. PhD thesis, Unversiteit van Amsterdam. 
Ajdukiewicz K 1935 Die syntaktische Konnexiät. Studia Philosophica 1:1-27. 
Atwell E 1996 Comparative evaluation of grammatical annotation models. In Sutcliffe R, Koch H, McElligott (eds), Industrial parsing of software manuals. Amsterdam: Rodopi, pp 25-46. 
Atwell E, Demetriou G, Hughes J, Schiffrin A, Souter C, Wilcock S 2000 A comparative evaluation of modern English corpus grammatical schemes. ICAME Journal, 24:7-23. 
Atwell E, Drakos N 1987 Pattern Recognition applied to the acquisition of a grammatical classification system from unrestricted English text. In Proceedings of EACL: Third conference of the European chapter of the association for computational linguistics, New Jersey. 
Brill E 1993 Automatic grammar induction and parsing free text: a transformation-based approach. In Proceedings of the 31st annual meeting of the Association for Computational Linguistics (ACL), Columbus, Ohio, USA, pp 259-265. 
660
Déjean H 2000 ALLiS: a symbolic learning system for natural language learning. In Cardie C, Daelemans W, Nédellec C, Tjong Kim Sang E (eds), Proceedings of the fourth conference on computational natural language learning and of the second learning language in logic workshop. Lisbon, Portugal, pp 95-98. 
Finch S, Chater N 1992 Bootstrapping syntactic categories using statistical methods. In Daelemans W, Powers D (eds), Backgrounds and experiments in machine learning and natural language: Proceedings first SHOE workshop. Institute for Language Technology and AI, Tilburg University, pp 230-235. 
Henrichsen P J 2002 GraSp: Grammar learning from unlabelled speech corpora. In Roth D, van den Bosch A (eds), Proceedings of CoNLL-2002. Taipei, Taiwan, pp 22-28. 
Hughes J, Atwell E 1994 The automated evaluation of inferred word classifications. In Cohn A (ed), Proceedings of ECAI ’94: 11th European conference on artificial intelligence, John Wiley, Chichester, pp 535-540. 
Losee R M 1996 Learning syntactic rules and tags with genetic algorithm for information retrieval and filtering: An empirical basis for grammatical rules. Information processing & management, 32(2):185-197. 
Roberts A 2002 Automatic acquisition of word classification using distributional analysis of content words with respect to function words. Technical report, School of Computing, University of Leeds. 
Sutcliffe R, Koch H, McElligott (eds). 1996 Industrial parsing of software manuals. Amsterdam: Rodopi 
van Zaanen M 2001 Bootstrapping structure into language: alignment-based learning. PhD thesis, School of Computing, University of Leeds. 
Vervoort M R 2000 Games, walks and grammars. PhD thesis, Unversiteit van Amsterdam. 
Watkinson S, Manandhar S 2001a A psychologically plausible and computationally effective approach to learning syntax. In CoNLL ’01: the workshop on computational natural language learning, ACL/EACL. 
Watkinson S, Manandhar S 2001b Translating treebank annotation for evaluation. In Proceedings of the workshop on evaluation methodologies for language and dialogue systems, ACL/EACL 
661

More on horror aequi: evidence from large corpora 
Juhani Rudanko 
1. Introduction 
I have argued that there has been a major change affecting factors bearing on complement selection in recent centuries: 
... it appears that there has been a general process of change in principles affecting complement selection over the last three centuries. As has been shown repeatedly, in the eighteenth century the form of the complement often depended on the grammatical or even the morphological environment of the complement. ... To facilitate discussion, the kind of complement selection in question might be called “context-dependent.” ... 
By contrast with eighteenth century English, in present-day English there is less emphasis on the environment of a complement clause and matrix verbs themselves tend to select their complements or alternatively, particular senses of matrix verbs can be linked to particular types of complements. This means that matrix verbs, or particular senses of matrix verbs, are subcategorized for particular types of syntactic complement. ... This kind of complement selection, where the form of a complement clause is directly linked to the matrix verb or its specific sense, might be termed “inherent” or “context-independent” or “lexically driven.” (Rudanko (2000, 141 f.)) 
Adopting the terms suggested, the change of complement selection principles is from context-dependent selection to lexical selection. The change is gradual in nature, and even in present-day English there are some traces of context-dependent selection. 
The hypothesis outlined regarding general principles of complement selection is sweeping and subject to being challenged. It may turn out to be wrong. The purpose of this paper is to illustrate and examine the operation of one particular principle that bears on the issue, in that it is typical of context-dependent selection, the horror aequi principle. This principle is simple. It “concerns the universal tendency to avoid the (near-)adjacency of identical grammatical structures” (Rohdenburg (2001)). 
A first example may clarify the principle. Consider the verb evade, which has the prototypical sense ‘to contrive to avoid (doing something); to get out of performing (a duty), making (a payment) etc.’ (OED, sense 3a). 
This verb selects sentential complements in present-day English, and as noted in the OED, they are of the -ing type, as in this example from the same source: 
1. The sheykh evaded traslating them (OED, 1867, Lady Herbert, Cradle L.) 
In the eighteenth century the verb likewise selected sentential complements and they were similarly of the -ing type for the most part, as in (2): 
2. ... the ladies smiled on each other, but evaded answering my question, which only encreased my curiosity. (1762, Sarah Scott, Millennium Hall, 22)) 
However, the following example is worth noting: 
3. She is so penetrating, that there is no possibility of evading to give her satisfaction. (1778-1788, Francis Burney, Evelina, 3, 98)) 
In (3) the complement of evade is not an -ing clause. The sense of the verb appears to be the same as that quoted above from the OED for sentence (1). Instead it is a to infinitival clause. What is interesting about the example is that the to infinitival complement occurs in a context where the matrix verb is in the form evading. That is, an -ing complement here would be a violation of the horror aequi principle. This suggests itself as evidence for context-dependent selection. 
The discussion in this paper concerns the verb neglect and its complement selection properties in the eighteenth, nineteenth and twentieh centuries. The data come from three large electronic corpora. For the eighteenth century, the source is the Chadwyck-Healey Corpus of Eighteenth-Century Fiction, which is a 
662 
corpus of some 12 million words. For the nineteenth century the material comes from the Chadwyck-Healey Corpus of Nineteenth Century Fiction. For the twentieth century the Bank of English Corpus is used. It is used either in its impressive entirety, comprising some 500 million words, or else the search is limited to the British Books segment of the corpus. The focus is thus on British English to some extent, but sometimes it is of interest to relax the focus, in the interests of getting sufficient amounts of data. 
As regards earlier work on the horror aequi principle, Rudanko (2000) and Rudanko (2002), in considering a number of matrix verbs, draw on the Chadwyck-Healey Corpus of Eighteenth-Century Fiction and on the BNC. The present study thus contributes a new angle in also drawing on the Chadwyck-Healey Corpus of Nineteenth Century Fiction, and on the large Bank of English Corpus. The verb neglect was not among the verbs systematically studied in relation to the horror aequi principle in Rudanko (2002), and the present investigation is designed to redress this omission. 
2. A case study: neglect 
The method of studying the possible operation of the horror aequi principle is the same here as it was in Rudanko (2000) and Rudanko (2002). It consists in identifying a matrix verb that selects complements of two different syntactic types without there appearing to be a significant difference in the sense of the verb when it combines with either type of complement. A decision about an absence of a significant difference in meaning is a matter of judgment and subject to challenge in the light of further work. Each case study to be presented therefore has a provisional status, pending further research. 
When a potential matrix verb has been identified, the alternate types of complements are identified next. In this paper only verbs with two variant complements are considered and the alternation considered in each case is between to infitival and -ing complements. 
With the alternation being between to infinitival and -ing complements, it is easy to identify the environments where the horror aequi principle may be expected to be operative. There are two such environments, with the following predictions: 
4. a. In the enviroment of a to infinitive, an -ing complement is expected to be favored over a to infinitive complement. 
b. In the enviroment of an -ing form, a to infinitive is expected to be favored over an -ing complement. 
An important step in any discussion of the horror aequi principle is the identification of a tertium comparationis, that is, of a neutral environment, where the horror aequi principle can be expected not to operate. The past tense form of the matrix verb is used here as one such neutral environment. Another form to be used is the past perfect form of the matrix verb. These choices are not necessarily perfect and are subject to further investigation, but it does mean that the influence of modal verbs, which has been argued to be a potent source of context dependent impact on complement selection (see Rudanko (2000, 119 f.), can be neutralized, and there is thus some reason for it, beyond the obvious point that the environment is neutral from the point of view of the contexts identified in (4a) and (4b). 
Turning to the verb neglect, we may start by consulting the OED for an analysis of the sense or senses of the verb. Senses 4.a and 4.b in the OED are relevant: 
4. a. With inf. To omit through carelessness, to fail through negligence, to do something. 
b. To omit doing something. 
The most recent illustration of sense 4.a is the example reproduced in (5a), while that of sense 4.b is the example given in (5b): 
5. a. If they neglect To punish crime. (1819, Shelley, Cenci) 
b. It is not uncommon for persons .. entirely to neglect looking into the state of their affairs. (1729, Butler, Serm.) 
To consider the senses of the two types of complements with neglect in present-day English further, one or two additional sentences may be worth thinking about. Of the following, (6a) is an authentic 
663 
sentence from the Bank of English Corpus, and (6b) is a sentence modeled on it. Sentences (7a-b) and (a-b) are invented examples, devised by Ian Gurney (p.c.). 
6. a. Having taken the European Community for granted for so long, politicians had neglected to rally their publics to the European cause. 
b. Having taken the European Community for granted for so long, politicians had neglected rallying their publics to the European cause. 
7. a. It has been our policy to collect a fee from students for xeroxing every autumn, but this autumn we neglected to collect the money. 
b. It has been our policy to collect a fee from students for xeroxing every autumn, but this autumn we neglected collecting the money. 
8. a. I have neglected to water my flowers. 
b. I have neglected watering my flowers. 
As is predicted by Dwight Bolinger's (1968) principle, the senses of the two types of complements of the verb are not entirely equivalent in present-day English. There is a more nominal feel to the -ing complement. It brings to mind an interpretation with a suitable NP — of the type the practice or the habit — preceding the complement clause, and the construction may often imply a lapsed habit. For its part, a to infinitive complement may have more of a focus on the failure to do something and may be more apt to suggest a mistake. 
Even so, the difference in meaning between the two types of complements seems to be a matter of nuance today and should not be exaggerated. It is worth noting that the glosses of the two subsenses identified in the OED and linked to the two two types of complements both make use of the verb ‘omit'. The senses are so close to each other that it may be possible to consider the operation of the horror aequi principle in relation to this verb. 
Proceeding to do so, we may start by considering the past tense verb form neglected as a neutral environment. There are altogether 327 instances of the verb form neglected in the Chadwyck-Healey Corpus of Eighteenth Century Fiction. The verb form neglected is of course also a past participle, and when the form is examined in the material, it turns out that a large majority of the instances of the verb form are past participles, especially passives, as in (9): 
9. ... thanked the president for his wholesome counsel, which he assured him should not be neglected; ... (1751, Tobias Smollett, Peregrine Pickle) 
Among the remaining instances there are also numerous instances where the verb form neglected, while a past tense form, selects an NP complement, as in (10): 
10. ... his grief was so wild, and his passion so impetuous. He refused all sustenance, neglected his person, renounced his amusements, ... (1762, Tobias Smollett, Sir Launcelot Greaves) 
When irrelevant instances are excluded, there do still remain a fair number of relevant examples of the past tense verb form, 27 to be exact. Here is their breakdown, with three examples of each in (11): 
to infinitive complements: 21 
-ing complements: 5 
TABLE 1 
11. a. ... immediately concerns the Hero of it; whose Pardon I beg for having so long neglected to mention his Name. (1751, Francis Coventry, Pompey the Little) 
b. ... methinks I would now gladly return to the state of harmless infancy I then neglected to value. (177?, Henry Mackenzie, The Man of Feeling) 
c. ... they never went into the Town, but Don Sancho neglected not to go frequently to sell his Straw Baskets and Hats, ... (1739, Penelope Aubin, Charlotta Du Pont) 
d. I hastily asked for her; they told me Dinner waited: But I neglected eating, and entring the Chamber, found the charming Maria, ... (1739, Penelope Aubin, The Noble Slaves) 
664 
e. Once or twice I ask'd what was the matter, but the People neglected answering me, and I was not very importunate; ... (1722, Daniel Defoe, Moll Flanders) 
f. ... if she appeared slatternly in her dress, was remiss in her business, or neglected going to church, ... (1766, Sarah Scott, Sir George Ellison) 
Such authentic data are of interest from the point of view of the interpretation of the two complements in the eighteenth century. It would seem that a semantic explanation necessarily linking an -ing complement to habitual activity or practice is reductive and too simple. While the -ing form in (8f) would be compatible with this explanation, the -ing form in in (8d) seems linked to a single event, and the to infinitive in (8b), for instance, seems linked to a pattern of behavior. The lack of clear-cut semantic differentiation emphasizes the need to investigate the potential influence of the horror aequi principle. 
Proceeding now to an examination of the two marked environments identified in (4), we may start with the context where the matrix verb is in the form of to neglect. There are 52 examples of this combination in the material. The majority of them are irrelevant here because the complement of the verb is a nonsentential NP. In general these are easy enough to identify and to exclude. Perhaps the only example over which there may dangle question mark is the one given in (12): 
12. We discovered no land all that day, and Crampley was still so infatuated as to neglect sounding; But at three o'clock in the morning, the ship struck, ... (1748, Tobias Smollett, Roderick Random) 
It may be that if sounding were a noun, an indefinite article might be appropriate (Heidi Harley, p.c.), and the example is therefore included: 
to infinitive complements: 0 
-ing complements: 3 
TABLE 2 
Here are the other two examples encountered: 
13. a. ... she thought it would be no other than running wilfully into a Misfortune, to neglect making use of the smallest Means that should offer to take her from it. (1725, Eliza Haywood, Idalia) 
b. ... he had too much Complaisance to neglect endeavouring to overtake the Lady, and prevent the Pain he imagin'd she wou'd ... (1725, Eliza Haywood, Love in Excess) 
The second example is particularly interesting in that the sentential complement of neglect is introduced by endeavouring, which itself selects a to infinitive complement. This opens up the possibility of a further factor favoring an -ing complement for neglect in this environment. 
Setting aside this further refinement, the figures in table 2 speak for themselves. The low number of to infinitival complements is striking, and provides some confirmation for the relevance of the horror aequi principle. 
Turning to the other marked environment identified in (4), there are 47 instances of the verb form neglecting in the material. As in the case of the combination to neglect, their complements are again mostly nonsentential NPs, but there are altogether 13 sentential complements. Their breakdown is as follows: 
to infinitive complements: 13 
-ing complements: 0 
TABLE 3 
The lopsided preponderance of to infinitive complements here in relation to -ing complements provides further confirmation of the applicability of the horror aequi principle. 
The discussion so far has fleshed out and substantiated the suggestion made in Rudanko (2000,) that the horror aequi principle is potentially relevant to explaining the complement selection properties of 
665 
neglect in the eighteenth century. The discussion here makes it imperative that similar examination be conducted of nineteenth-century usage. 
Proceeding to conduct such an investigation and turning to the verb form neglected as the first step, it is observed that The Chadwyck-Healey Corpus of Nineteenth Century Fiction is so large, at almost 40 million words, that there are literally hundreds and thousands of this verb form in the corpus. Most of them are probably again irrelevant, being passive past participles. To keep the investigation manageable, the procedure here was to select three frames where such participles were excluded automatically. The frames in question are he neglected ——, she neglected —— and they neglected ——. These frames yield 46 instances in all, the first frame being the most frequent, with 26 instances. Most complements in the frames are nonsentential NPs, but there are also some sentential complements found. Here is their breakdown: 
no. of to infinitive complements: 10 
no. of -ing complements: 0 
TABLE 4 
Here are some examples: 
14. a. ... his will a very short time, comparatively speaking, before he died, and he neglected to make another. (1889, George Gissing, The Nether World) 
b. ... they attacked some travellers, and, as they thought, killed them. They neglected, however, to bury their victims, and one, who was not dead, revived: ... (1839, Meadows Taylor, Confessions of a Thug) 
The figures in table 4 substantiate the preponderance of to infinitive complements over -ing complements in an environment that is neutral from the point of view of the horror aequi principle. 
Another neutral environment may be worth considering, to get an additional perspective on the issue. The environment chosen is had neglected ——. There are 68 instances of this environment. NP complements are again frequent, but the breakdown of sentential complements is as follows: 
no. of to infinitive complements 27 
no. of -ing complements 0 
TABLE 5 
Here are one or two examples: 
15. a. The man having been at home ill some days had neglected to shave, and there was some mark upon his upper lip. (1885, Richard Jefferies, After London) 
b. The admiral was aware how often he had neglected to annoy or capture the enemy when he might have done it, and by such neglect, ... (1832-3, Frederick Marryat, Peter Simple) 
Turning now to the first of two non-neutral environments in nineteenth-century English, there are as many as 145 instances of the environment to neglect —— in the material. In the overwhelming majority of them the complement is a nonsentential NP, and the number of sentential complements is as low as 4. This low number may in itself be of significance. 
Here is the breakdown of the four instances: 
no. of to infinitive complements 3 
no. of -ing complements 2 
TABLE 6 
Here is one example of each: 
666 
15. a. ... he spent part of the day in a small boat on the water, having been mad enough to neglect to take his overcoat with him. (1892, George Grossmith, The Diary of Nobody) 
b. Mr. Lenville requests Mr. Johnson not to neglect making an appointment, as he has invited two or three professional friends to ... (1839, Charles Dickens, The Life and Adventures of ... Nicholas Nickleby) 
As regards the other non-neutral environment, neglecting ——, the number of instances is slightly higher, 175. Again, complements are generally nonsentential, but the proportion of sentential complements is noticeably higher than in the previous environment. Here is the breakdown: 
no. of to infinitive complements 25 
no. of -ing complements 0 
TABLE 7 
Here is one example: 
16. ... for the admonition or benefit of others, even while constantly neglecting to exert it for themselves. (1828, Edward Bulwer Lytton, Pelham) 
Turning to present-day English, it is not feasible to consider the simple frame neglected —— as a neutral environment, because of its high frequency and because of the high incidence of irrelevant examples. To exclude irrelevant examples and to keep the investigation manageable, the frame had neglected —— was used as a neutral frame instead. There are 135 instances of this frame in the Bank of English Corpus. In a large number of cases the complement is sentential. However, there is also a sizeable number of instances where the complement is sentential. Here is their breakdown: 
no. of to infinitive complements 62 
no. of -ing complements 0 
TABLE 8 
17. a. ... both former justice ministers had neglected to implement the law fully, ... 
b. ... what hurt him most was the prosecution's allegation that he had neglected to provide for his children. 
There is thus a huge preponderance of to infinitives in this environment, which is presumably neutral. 
Regarding the non-neutral environment of to neglect ——, there are plentiful examples of it in the Bank of English Corpus, 267 in all. One or two of them are irrelevant because the combination is of a preposition and NP, especially in the phrase due to neglect. Such examples must of course be set aside. 
In the huge majority of cases the word neglect is a verb with a complement. However, in the overwhelming majority of the instances the complement is a NP. Sentential complements are remarkably rare. Here is the breakdown of the cases that are found, with illustrations: 
no. of to infinitive complements: 1 
no. of -ing complements: 2 
TABLE 9 
Here are all the examples found: 
18. a. We might be tempted to neglect to tell people of the risks and let a few hundred die so we can develop a vaccine quickly, ... 
b. The pring empties into a well dug sometime around 1770. Around ten feet deep, it is lined with field stones and loosely covered with aging timbers. It is easy to neglect checking it because you would rather not know what has fallen or crawled into it, ... 
667 
c. Many employers have contributed sites that include self assessment material for potential applicants. To neglect using these is to put yourself at grave disadvantage, ... 
3. Conclusion 
This investigation suggests a number of conclusions about the complement selection properties of the matrix verb neglect. At the most basic level, this study shows that the verb has displayed variation between to infinitive and -ing complements in each of the last three centuries. At the same time, a comparison of the neutral environments suggests that the incidence of the variation has not remained constant. Instead, there appears to have been a clear rise in the proportion of to infinitive complements in relation to -ing complements during this period, or indeed even by the beginning of the nineteenth century. In the eighteenth century the ratio of the two types of complements was 21/5, thus heavily in favor of to infinitives, but in subsequent centuries it has become even more lopsided. 
Regarding the environments that are marked from the point of view of the horror aequi principle, there appears to be an qualitative difference between them with respect to the ease with which the verb combines with sentential complements. The verb form neglecting is seen to combine with sentential complements with ease, and these complements are to infinitives with great regularity. This is consistent with the operation of the horror aequi principle. However, because of the strong overall prevalence of to infinitive complements, this evidence is not very persuasive by itself. 
For its part, the combination to neglect is surprisingly rare with sentential complements in each of the three centuries. It may be worth bringing together the relevant figures here. In the eighteenth century material only in two sentences out of 52 potential examples is the complement sentential. In the nineteenth century material the corresponding figures are 5 out of 145 potential examples. In the Bank of English Corpus the figures are 3 out of 267 potential examples. 
The examples that are found in the environment of to neglect do suggest that the horror aequi principle is relevant. In the material for two of the centuries, it is actually the case that -ing complements are more frequent in this particular environment, while in the nineteenth century material the figures are three to two. In view of the overwhelming overall predominance of to infinitive complements with neglect, the high proportion of -ing complements in this environment does serve to motivate the postulation of a horror aequi principle in the analysis of the complement selection properties of the matrix verb neglect. 
References 
Bolinger, D 1968 Entailment and the Meaning of Structures. Glossa 2, 119-127. 
Rondenburg, G 2001 Mechanisms of Syntactic Change: the Rise and Fall of Prepositional Constructions in Late Modern English. Abstracts Booklet of the Late Modern English Conference, Department of English, University of Edinburgh. 
Rudanko, J 2000 Corpora and Complementation. Lanham, MD: University Press of America. 
Rudanko, J 2002 Complements and Constructions. Lanham, MD: University Press of America. 
668 

Constructing a database of French interlanguage oral corpora 
Sarah Rule, Emma Marsden, Florence Myles, Rosamond Mitchell. 
University of Southampton 
1 Introduction 
There is an increasingly recognised need for datasets of oral production in the second language acquisition research community. This requirement arises in response to two complementary agendas: to advance the second language acquisition research agenda and also for educational reasons. At the University of Southampton, a series of projects have concentrated on the collection of learner oral production data in French. In the mid 1990s the ‘Progression in Foreign Language Learning’ project collected data from Key Stage 3 classroom learners (Mitchell and Dickson 1997 R000234754). In 2001 –2, the Linguistic Development in Classroom Learners of French’ project collected similar data from Key Stage 4 learners (directed by Florence Myles 2001 R000223421). In 2002-3 further projects are running which are archiving these datasets in a standard format, and also collecting and archiving other existing datasets for more advanced learners 
This paper outlines the aims of these projects and the rationale behind the development of a database of French interlanguage oral corpora. The second part will report on the methods used for the collection, storage, analysis and sharing of such data, and decisions taken about methodology, considering both project specific needs and the requirements of the wider research community. There is a need for tools that can handle large data sets of oral production data, as working with such data presents several challenges: representing speech in written form, devising coding that is helpful both for specific hypothesis testing and carrying out efficient and reliable analyses. 
2 The arguments for developing French interlanguage corpora 
There are both linguistic and educational reasons for investing in the development of electronic interlanguage corpora. 
The field of second language acquisition research is still relatively young and the early work was predominantly exploratory and descriptive, using case studies and small-scale projects. As the field has developed the research questions are increasingly more focussed and theoretically informed. However the scale of empirical research has generally remained small. If larger, more easily accessible datasets were available, it should be possible to test a range of existing theoretical claims and hypotheses more fully. 
Complementing the theoretical research agenda are the needs of modern foreign language education. As the field of education renews its interest in specifying the linguistic content of foreign language programmes, for example in the UK Key Stage 3 MFLs Strategy being piloted in 2002-3, it is vital that the process be informed by detailed documentation of the course of interlanguage development. Solid empirically based accounts of learner development provide an extremely valuable perspective on curriculum proposals. The availability of large scale tagged interlanguage corpora will allow much more effective and systematic cross-checking of curriculum proposals against what is known about learner development. 
3 The research projects 
The Progression in Foreign Language research project took place in two English secondary schools over a period of three years (1993-6). It was a longitudinal study that tracked a cohort of 60 pupils from their first year of learning in year 7 to their third year of learning in year 9. The learners undertook a range of oral production tasks on six occasions spread over the period. Some tasks were carried out in learner pairs and some with a researcher. Some tasks were repeated between rounds in order to track progress more closely. The dataset from the project comprises some 200 hours of analogue audiorecordings and the 650 transcripts are in plain text files appropriate for use with concordancing tools such as Wordsmith. The resulting publications, however (see for example, Myles, 
669
Mitchell and Hooper 1999) have so far drawn on relatively small subsets of data from the corpus, partly due to the fact that the tools available did not facilitate rapid analyses of the complete dataset. 
The Linguistic Development Project (2001) was conceived to build on the knowledge base that was developed in the Progression Project. Its principle aim was to collect a new dataset of French oral learner language in order to be able to document and analyse progression in the years following on from those covered by the Progression Project, with a particular focus on linguistic development. Some of the same test instruments were used to compare performance on the same tasks at different educational stages. The data from the two projects combined will enable us to build a comprehensive picture of how children develop in French during their secondary education up to GCSE. 
The Linguistic Development in Learners of French Project had the following more specific aims 
• To analyse the development of a number of morphosyntactic structures in spoken learner French, including phrase structure, verb morphology, gender, interrogation, negation and pronominal reference. 
• To analyse the creative construction process, from Initial State and beyond, and its interaction with formulaic language among instructed learners. 
The sample consisted of three groups of 20 learners in each of Years 9, 10 and 11 in an English secondary school. Each learner undertook four oral tasks with an adult interlocutor (see Appendix one for description of tasks used in both projects). The project collected approximately 50 hours of spoken French and the data for the two projects combined constitutes a corpus of some 250 hours. 
The fact that in the Progression project only small subsets of learner data were used to produce theoretical research papers illustrates an issue that has been increasingly discussed in second language acquisition research: theoretical claims have proliferated while the scale of empirical research to test these have often remained quite small. Efficient means of carrying out detailed linguistic analyses on such data, given the nature of the research questions and size of the sample, are crucial. Another important issue was being able to share the data with the wider research community, given the expensive nature of corpus gathering. So, in the early stages of the Linguistic Development project decisions had to be made with regards to how the data was to be recorded, transcribed and stored in order to meet these objectives. 
Our investigations discovered that two attempts to develop software dedicated to the analysis of second language (L2) oral data are now inactive: COALA (Pienemann 1992) and COMOLA (Jagtman & Bonagerts 1994). The research team also investigated the possible use of Extensible Markup Language (XML) to tag and share our data. However, time considerations in developing the necessary skills and analysis software led us to use an ‘off-the-shelf’ language-specific package which we felt could meet our requirements: CHILDES (The Child Language Data Exchange System). In any case, CHILDES has now been converted to an XML compatible format. Moreover the recent addition of a French morphosyntactic parser to the CHILDES system meant that it was applicable to our research needs. 
4 A brief introduction to CHILDES 
The CHILDES set of tools was originally conceived for first language acquisition, but has also been used for research into language disorders and by some second language researchers (Malvern and Richards 2002, Housen 2003, Paradis, Le Corre and Genesee 1998). CHILDES tools have been used in well over 1300 published ranging studies ranging from L1 acquisition to computational linguistics, language disorders, narrative structures, literacy development, phonological analyses and sociolinguistics (Macwhinney 2002). 
Besides the features of specific interest to language researchers discussed in the following sections, CHILDES has several important advantages. First, the tools are constantly up-dated by a well-funded team of programmers, there is an active community of users, the system promotes data-sharing and all the tools can be downloaded free of charge from the internet. 
CHILDES consists of three integrated components: 
670
• The large and diversified database (Talkbank) consists primarily of child speech recordings and transcriptions, but also includes some language disorder data and bilingual data. It is a condition of using CHILDES tools that our data will become part of the Talkbank database. There are increasingly more Second Language Acquisition (SLA) datasets available in Talkbank. 
• CHAT (Codes for the Human Analysis of Transcripts) are the transcription procedures, which have been developed to be compatible with the analysis programmes. 
• CLAN (Computerized Language Analysis) consists of about 40 core computer commands for carrying out searches and counts, along with a range of 'switches' that can be used to customise each command. This is a powerful and flexible software package that can carry out rapid and detailed analyses and is designed to recognise the tagging conventions of CHAT. 
5 Recording the data 
Although digital soundfiles are not required in order to use the CHILDES tools, the TALKBANK database has now been entirely digitised1. This clearly facilitates complete data-sharing. The advantages of digital data also have important consequences for realising the potential of linguistic data. Digital recording machines themselves are quieter and less intrusive, the quality and durability of the sound is much better, negotiating through files is significantly more efficient than working with audiocassettes and the timing of pauses can be more easily done. Digital soundfiles can be ‘linked’ to the transcript, enabling simultaneous access to the written and spoken forms, compensating, to some extent, for the inevitable shortcomings of written representation of speech, and giving other researchers the opportunity to code the spoken data. Waveforms of the digital file can also be displayed and linked precisely with the transcription. 
6 CHAT transcribing and coding procedures 
6.1 Headers 
Every file has a set of ‘headers’ so that the software can process relevant details from each file. Anything that the researchers feel could potentially influence the findings (e.g. elicitation task, date, transcriber etc) can be recorded in these headings. A 'readme' file must accompany each dataset giving a brief description of the project and sample, and information to other researchers regarding the transcription and coding decisions taken (for example, in the current study, precise pauses and phonological codes were not documented). 
6.2 Main line 
The data is transcribed on to a main line as a set of standard language word forms. Each utterance is transcribed on to a separate line and starts with * followed by the speaker code; this line shows what was actually said, in contrast with lines starting with a % sign which contain linguistic tags or other relevant information. The CHAT manual (MacWhinney 2000a) contains codes that have been developed by various contributors addressing a wide variety of linguistic research agendas, including, for example, codes for Conversation Analysis, the analysis of written data, sign language, and for phonetic, prosodic, morphological and syntactic features of speech. CHAT also allows new codes to be used to address project-specific questions. 
1 The CHILDES research group offer free digitisation of data that will be offered to TALKBANK. For this project the digital recordings were stored as wav files. This is necessary in order to use Soundscriber software, described later, and it is also becoming the standard format adopted by those using CHILDES tools. 
671
A programme called CHECK can ensure your file meets minimum requirements to be recognised by the analysis software CLAN (for example by indicating where the human transcriber has not followed basic procedures, such as starting each main line with *)2. 
6.3 Tiers for coding 
In addition to the main line, there can be multiple ‘dependent tiers’ that provide ancillary information. These tiers are preceded by a % sign to indicate they are strings of tags and not the primary data. Researchers can decide how many dependent tiers are appropriate for their own purposes. Our data has a %err tier (error) and a %mor tier (morphosyntax), though researchers using our data in the future are free to add other coding tiers depending on their interests3, e.g. a % pho tier with phonological coding. 
6.3i %err tier 
The %err line (error tier) enables researchers to indicate errors in the interlanguage. It has a well-developed system of codes to be entered manually onto a tier beneath the main line. The % err tier is used by some SLA researchers but had several disadvantages for us e.g. it would have been a particularly arduous task given the interlanguage of beginners, resulting in cluttered coding. More importantly, project-specific error tiers can restrict the questions other researchers can ask of the data. In fact, most grammatical errors are retrievable systematically by searching in the morphosyntactic parsing produced by one of the CLAN programmes (MOR). For this to happen, the written forms on the main line have to be recognised by this programme as belonging to the French lexicon so that forms are correctly parsed. This meant that we did retain one feature of the error tier. Some words that were pronounced incorrectly but the context clearly indicated what was intended by the learner, were transcribed as target-like so that they were recognised by the automatic parser. For example, bouée (rubber ring) (given to the learners as a written prompt) was frequently mispronounced, but 'bouée' was written on the main line and what was actually said was recorded on the error tier. This illustrates that even with a highly systematized transcription procedure, project-specific aims still influence methodological decisions. 
6.3ii %mor tier 
The %mor tier encodes syntactic categories and morphological inflections, indicating tense, aspect, person, number and gender features. It is possible to generate a morphosyntactic description of the main line semi-automatically by using two CLAN tools, MOR and POST. Versions of MOR have been produced for a range of languages (ten at present4); the parser for French was developed by Christophe Parisse in 20015. For the programme to parse data from a particular corpus correctly, some time must be spent adding to the lexicon in the programme to ensure it recognises all the words in the corpus (though one of the CLAN programmes helpfully extracts all 'unrecognised' forms). 
Example of transcript with an added MOR tier from the Linguistic Development Corpus: A year 10 learner carrying out the negative elicitation task. 
*29N: mais il n' aime pas le musique . 
%mor: conj|mais pro:subj|il&MASC&_3S adv:neg|ne v|aimer-PRES&_3SV adv:neg|pas det|le&MASC&SING n|musique&_FEM . 
*29N: um il ne joue pas le basket # +/. 
%mor: co|um pro:subj|il&MASC&_3S adv:neg|ne v|jouer-PRES&_3SV 
adv:neg|pas det|le&MASC&SING n|basket&_MASC . 
*29N: +, et il aime le cola . 
2 Some CLAN commands can be used with transcriptions that are not in strict CHAT format by simply typing +y next to the normal command so the program would consider each line as one tier. +y1 would consider utterances as delimited by full stops, question marks, and exclamation marks. 
3 One CLAN command can take out all codes and tiers, leaving a ‘friendly’ transcript, useful for eyeballing and presentations. 
4 Cantonese, Danish, Dutch, English, French, German, Hungarian, Italian, Japanese and Spanish. 
5 Christophe Parisse's advice during the project is gratefully acknowledged, as is Brian MacWhinney's. 
672
%mor: conj|et pro:subj|il&MASC&_3S v|aimer-PRES&_3SV 
det|le&MASC&SING n|cola&_MASC . 
*ELD: mmm . 
*29N: il ne mange pas le glace . 
%mor: pro:subj|il&MASC&_3S adv:neg|ne v|manger-PRES&_3SV 
adv:neg|pas det|le&MASC&SING n|glace&_FEM . 
6.4 Other CLAN commands 
CLAN can carry out lexical, morphosyntactic, discourse and phonological analyses, amongst others, depending on how the data has been coded. CLAN programmes such as FREQ, KWAL and COMBO facilitate analyses of the frequency and linguistic context of interlanguage features. POSFREQ does a frequency analysis by sentence position and MLU calculates the mean length of utterance. In addition, the results of one analysis can be ‘piped’ through another analysis, allowing multiple analyses. 
Analyses can be carried out on specific tiers by customising the commands. For example, COMBO searches can search for specific words, word sequences or combinations of lexical items and morphosyntactical and/or ‘error’ codes. The COMBO command given here was one used in Rule & Marsden 2002 for the analysis of the expression of negation; ‘ne’ followed by a verb in the present then ‘pas'. 
COMBO combo +t%mor 10N 10N9ELD.mor.pst +s"*ne^v*pres*^*pas" 
A definition of what each part of the command means: 
combo = specifies the command 
+t = specifies the tier 
%mor = we want the command to be carried out on the %mor tier 
10N9ELD = name of file 
.mor.pst = extension given to file after it has undergone MOR and POST 
programmes 
+s = the switch to say look for the string 
" = a metacharacter to mark the start of the string you want to look for 
* = any character (on the MOR tier there can be some other symbols / characters between 'det' and the following word's code) 
ne = you are looking for the word 'ne' 
^ = followed by 
v*pres = you are looking for a verb with the code 'pres' which is given to any verbs inflected in the present 
^ = followed by 
pas 
" = marks the end of the string you are looking for 
All commands can be used to search individual files or whole sets of files stored in the same directory. 
7 Project-specific issues and flexibility of the tools 
Certain aspects of early emerging interlanguage grammars led us to make use of the flexibility of the CHAT and CLAN tools. Two such issues are illustrated below: 
673
7.1 Representing interlanguage forms 
English learners of French often use 'approximate' forms of definite and indefinite determiners, lying between ‘le’ and ‘la’ and between ‘un’ and ‘une'. Similarly for ‘a’ / ‘est’ and ‘je’ / ‘j'ai'. CHAT suggests @n can be used to code such morphological 'neologisms', which can be added to the lexicon and analysed by the parser as the researcher decides in order to track their use. For example we instructed MOR to code such items as follows (neologisms underlined): 
*21L: le@n famille 
%mor: neo:def:det|le@n n|famille&_FEM 
*21L: un@n grand+mere 
%mor: neo:indef:det|un@n n|grand+mere&_FEM 
Given our interest in morphosyntactic development this enabled us to track systematically the progression from a default, ‘genderless’ determiner towards the target system. It also avoids the problem of having to decide on a determiner when it cannot be heard clearly, potentially over-interpreting the data. 
7.2 Syntactic roles in code-switching 
Our data has a significant amount of both phrase-internal and phrase-external code-switching. CHILDES tools have been used for the study of bi-lingual data (e.g. Sebba http://talkbank.org/data/LIDES/) though we could not locate studies where the morphosyntactic role of both languages has been tracked. Tags for the main line were therefore devised to represent the different syntactic categories when English was used: @d, after first language (L1) nouns, @v after L1 verbs and @a after L1 adjectives and so on. These codes were then written into the MOR programme to produce the following coding: 
*21L: pour le skirt@d 
%mor: prep|pour det|le&MASC&SING n:eng|skirt 
As we were investigating phrase structure in the Linguistic Development project it was important to be able to track the architecture of the learners’ phrases regardless of the language used, for example whether verbs had the necessary number of arguments or whether determiners had the necessary noun complement. Additionally we could study the use of determiners with English nouns, e.g. whether predominantly masculine determiners are used with English nouns. 
8 Current Developments 
8.1 Digitising and converting the Progression Project data. 
A current project is now underway at the University of Southampton to digitise the analogue recordings from the Progression Project and also convert the transcripts from text formant into CHAT format (ESRC funded project R000220070). When completed, together with the Linguistic Development project data, we will be making available a substantial database of French oral interlanguage data produced by classroom learners in Key stages 3 and 4 (ages 11-16). The complete dataset will ultimately be freely available to other researchers on the web, as a set of digital soundfiles and linked transcripts, with various levels of coding. 
8.2 Integration of datasets from more advanced learners 
We have also recognised the potential advantages of extending the dataset beyond the material collected from early classroom learners in the two original Southampton projects. The research team at Southampton has now received funding from the AHRB for a further project to run from 2003-4 (Myles RE-AN9657/APN15456). The aim of this new project is to digitise and reformat existing French L2 oral corpora contributed by other researchers, as well as to create a web interface enabling researchers to interrogate these corpora according to a number of criteria. The corpora to be integrated into the overall database will complement the existing Southampton material, because it will largely be drawn from more advanced learners such as UK undergraduates. 
674
In this way, the considerable research effort involved in data gathering will be made more ‘profitable’ by making data more easily processed in large amounts, and also by making it more accessible to the wider research community. It is through these methods that we hope to meet the crucial requirement for second language acquisition research that there should be easy access to a wide range of corpora, transcribed and formatted to an internationally agreed standard (Ellis 1999, Rutherford and Thomas 2001). In addition we hope to provide a sound empirical basis for informing curriculum development and policy decisions. 
9 Conclusion 
The rationale behind using CHILDES, to transcribe, store, and analyse the Linguistic Development Project data, was to construct a dataset that was in a format that adhered to agreed procedures and could be analysed using powerful tools, and that it was readily available to the research community. It both meets the needs of our focused research questions and provides an extremely rich dataset that other researchers can interrogate in other ways. The investment into acquiring knowledge of CHILDES has proved extremely worthwhile. Already, analyses are ongoing using the Southampton material which relates to: 
• Use of rote-learned chunks by intermediate learners, and their role in ongoing development of the learner language system (Myles forthcoming, Mitchell and Myles 2003) 
• The development of the verb phrase in emerging grammars (Myles, in press a) 
• Development of negation by early learners (Rule and Marsden 2002) 
• Learner development of discourse management and narrative skill (Myles in press b) 
• Verb movement and the development of interrogatives 
• The role of NS-NNS and NNS-NNS scaffolding in the microgenesis of linguistic structure 
We look forward to sharing this data with the wider research community very soon and to seeing the potential of the CHILDES tools for SLA research being explored more fully in relation to further languages and learner levels. 
675
References 
Ellis, N. 1999, Cognitive approaches to SLA, Annual Review of Applied Linguistics 19, pp. 22-42. 
Housen, A. (2002) A corpus-based study of the L2-acquisition of the English verb system, in Computer Learner Corpora, Second Language Acquisition and Foreign Language Learning, S. Granger, J. Hung & S. Petch-Tyson (eds). John Benjamins. 
Jagtman, M. & Bongaerts, T. 1994, Report - COMOLA: a computer system for the anlysis of interlanguage data, Second Language Research 10 (1), pp. 49-83. 
MacWhinney, B. 1999, The CHILDES System, in Handbook of Child Language Acquisition, Academic Press, pp. 457-494. 
MacWhinney, B. 2000a, The CHILDES project: tools for analyzing talk. Volume 1: Transcription format and programs. Volume 2: The database. 3rd ed. Lawrence Erlbaum. 
MacWhinney, B. 2002, http://cnts.uia.ac.be/childes/ 
Malvern, D. & Richards, B. 2002, Investigating accommodation in language proficiency interviews using a new measure of lexical diversity. Language Testing, 19(1), pp. 85-104. 
Mitchell, R and Myles F, 2003, Rote learned chunks and interlanguage development, to be presented at AAAL annual conference 2003, Washington, USA. 
Mitchell, R. & Dickson, P. 1997, Progression in Foreign Language Learning, Centre for Language in Education Occasional Paper No.45. University of Southampton. 
Myles, F. (in press a) The emergence of morpho-syntactic structure in French L2. In J.-M Dewaele (ed), Focus on French as a foreign language: Multidisciplinary approaches. Clevedon: Multilingual Matters. 
Myles, F. (in press b), The early development of L2 narratives: a longitudinal study. Marges Linguistiques, v.5 Saint-Chamas: M.L.M.S. 
Myles, F. (forthcoming), From data to theory: the over-representation of linguistic knowledge in SLA. Transactions of the Philological Society 102. 
Myles, F., Mitchell, R., & Hooper, J. 1999, Interrogative chunks in French L2: A basis for creative construction? Studies in Second Language Acquisition 21, pp. 49-80. 
Paradis, J., Le Corre, M., & Genesee, F. 1998, The emergence of tense and agreement in child L2 French, Second Language Research 14(3), pp. 227-257. 
Pienemann, M. 1992, COALA -A Computational System for Interlanguage Analysis. Second Language Research 8 (1), pp. 59-92 
Rule, S. & Marsden, E. 2002, Expression of negation in French L2 classroom learners. Paper presented at BAAL / CUP seminar, July 2002, University of Southampton. 
676
Rule, S., Marsden, E. & Myles, F. 2002 "The acquisition of negatives in the French L2 classroom", Paper presented at Eurosla, Basel, 2002 
Rutherford, W. & Thomas, M. 2001, The Child Language Data Exchange System in research on second language acquisition. Second Language Research 17(2), pp. 195-212. 
677

"Are we nearly there yet, Mum?" 
Geoffrey Sampson 
University of Sussex 
School of Cognitive and Computing Sciences 
University of Sussex 
Falmer 
Brighton BN1 9QH 
geoffs@cogs.susx.ac.uk 
Tel. 01273 678525 
Fax 01273 671320 
ABSTRACT 
The reason why "corpus linguistics" is recognized as a distinct sub-discipline with a name of its own is that, until quite recently, much linguistic research was conducted without reference to empirical observation of 
real-life language use even when it concerned topics to which empirical observation seems relevant. The objections to that state of affairs in terms of scientific validity are well known; the success of the corpus "movement" over recent decades may suggest that the problem is now being dealt with satisfactorily. When Diana McCarthy and I used a historical basis for compiling our anthology of the field (Corpus 
Linguistics, now in press with Continuum of London and New York), the picture we found was one of slow growth from the pioneering work of Charles Fries in the 1950s onwards for several decades, giving way to almost explosive growth and diversification over about the last ten years. 
This paper seeks to discover how strong the impact of corpus work on the wider discipline has actually been to date, by counting the proportion of items in random samples from linguistic and computational- linguistic journals over a period of up to fifty years which appear to make empirical claims, and which base such claims on data that are in principle intersubjectively observable. Like the answer often received by the child piping up from the back seat of the car, the answer to our question may be that in reality we have much further left to go than we imagine and hope. 
678

Women's and men's style: fact or fiction? New grammatical evidence 
Hans-Jörg Schmid and Jens Fauth 
University of Bayreuth, Germany 
Hans-Joerg.Schmid@uni-bayreuth.de; jot.fauth@gmx.net 
The idea that there is a gender-specific style of writing has been around for a long time in literary studies. In linguistics there is also a fairly long-standing tradition of talking about women's language or women's style, but the focus of this line of research has traditionally been speech rather than writing. So far, empirical studies have focussed on phonological aspects (e.g. Trudgill's, Milroy's or Cheshire's well-known work), lexical aspects (“women's words”, cf. Lakoff 1975 or Schmid's contribution to Corpus Linguistics 2001 to be published in the Leech-Festschrift), and especially on styles of discourse and discourse behaviour (“women's cooperative style”, Tannen, Coates, Mulac and others). Investigations of grammatical aspects of gender-specific language have typically looked at questions, especially tag-questions, and a small number of other grammatical features that were envisaged to provide surface evidence for underlying gender-specific discourse styles. These handpicked areas aside, grammatical markers of gender-specific style have hardly been studied so far. 
This is the gap that the present paper wants to fill by investigating the gender-specific use of such basic gram-matical categories as the definite article, personal pronouns, nominal pre- and postmodifiers and various types of verbs in the spoken part of ICE-GB. It will be shown that statistically significant gender-specific differences in the frequency of such features can be found. The findings indicate that even under highly similar contextual circumstances women tend to use a more “involved” style of speaking and men a more “detached” one (cf. Chafe 1982, Chafe and Danielewicz 1987). 
Methodologically, this study is a substantial refinement as compared to Schmid's 2001 paper, which focussed almost exclusively on the parameter gender and thus neglected the influence of other user-related (age, educa-tion, social class etc.) and use-related factors (field of discourse, setting, topic, social roles). The range of inter-fering variables (most notably the important variables field of discourse and topic) will be narrowed down sys-tematically by focussing on mixed-sex conversations with balanced contributions by male and female speakers. (For a complementary study on same-sex conversations and a methodology for detecting gender-specific patterns of linguistic features see the abstract for this conference by Fauth and Schmid). 
679

Methods and tools for development of the Russian Reference Corpus 
Serge Sharoff, Russian Research Institute for Artificial Intelligence, P.O.Box 85, 125190, Moscow, Russia, sharoff@aha.ru 
From the viewpoint of corpus linguistics, Russian is one of few major world languages lacking a comprehensive corpus of modern language use, even though the need for constructing such a corpus is growing in the corpus linguistics community both in Russia and in the rest of the world. 
1. The history of development of Russian corpora 
The best-known attempt to develop a comprehensive Russian corpus has been made in Uppsala. The Uppsala Corpus consists of 1 million words of fiction and non-fiction texts, so it is too small and restricted in the genre coverage for modern standards. It also lacks morphosyntactic annotations. Another attempt has been made in the Soviet Union in the mid 1980s under the heading of the Machine Fund of Russian, though it did not produce the expected outcome. There are also multiple ad hoc collections of Russian texts, but they are not balanced and representative. 
2. The objective 
The objective of the project is to develop the Russian equivalent of the BNC, namely a corpus of 100 million words with proportional coverage of various functional registers, POS annotation and lemmatisation (the latter is required for Russian, which has dozens of word forms for a lemma). The annotation scheme (based on TEI) also allows to mark noun phrases and prepositional phrases, because they are important for the resolution of ambiguity. 
3. Problems and solutions 
First, there are problems in obtaining source texts. Some sources are readily available: fiction and news texts are widely accessible via the Internet and can be legally available for the corpus. Other types of the discourse, like business or private correspondence, are hard to obtain and make available in a corpus because of legal obstacles. Yet other types of sources, like samples of spontaneous speech, are rare for technical reasons. The decision is to increase the amount of ephemera whenever possible, because news and fiction texts will take the rest of the share. Personal and business letters are subjected to an anonymization procedure with respect to names of persons and companies. Another set of problems with sources concerns the choice of diachronic sampling, because the turbulent history of Russia in the 20th century radically affected the language. For instance, according to the frequency list (Zasorina, 1977) that was compiled on the basis of texts from 1930-1960, such words as sovetskij (Soviet) and tovarishch (comrade) belonged to the first hundred of Russian words on a par with function words, but this is no longer valid in modern texts. The decision on the chronological limits of the study is different for different functional registers, for instance, fiction texts are taken from 1970, scientific texts from 1980, while news texts from 1997. Second, there are problems with resolving the ambiguity of word forms. Many word forms correspond to several lemmas and POS classes, for instance, the pole is an instance of three different nouns pol (floor), pole (field) and pola (lap). Since they have distinct morphological properties (the case, number and gender), the ambiguity can be resolved on the basis of simple syntactic analysis, like the agreement in noun phrases or between the subject and the predicate in a sentence. Yet other frequent types of ambiguity can be resolved only on the basis of semantic and pragmatic constraints: Xranite svoi denjgi v banke (keep your money in a bank/in a jar). Such cases of genuine ambiguity are kept in the corpus using multiple <ana> tags. Third, there are problems with the query language for accessing the corpus. Typically corpus query languages (e.g. SARA or CQP) assume the fixed order of tokens, while in Russian the order of participants in a clause is not fixed, but depends on thematic development conditions. Special operators are introduced for expressing such conditions. 
4. The current state of the project 
Currently tools and techniques for working with the reference corpus are tested using a corpus of 40 million words. Its sub corpus of about 1 million words of fiction texts (The Russian Standard) has automatically assigned and manually inspected POS annotations (it is available from http://corpora.yandex.ru). It can be also used for correcting POS taggers used for processing the corpus. 
680

Using dialogue corpora to train a chatbot 
Bayan Abu Shawar (bshawar@comp.leeds.ac.uk) 
and Eric Atwell (eric@comp.leeds.ac.uk) 
School of Computing, University of Leeds, Leeds LS2 9JT England 
Abstract 
This paper presents two chatbot systems, ALICE and Elizabeth, illustrating the dialogue knowledge representation and pattern matching techniques of each. We discuss the problems which arise when using the Dialogue Diversity Corpus to retrain a chatbot system with human dialogue examples. A Java program to convert from dialog transcript to AIML format provides a basic implementation of corpus-based chatbot training.. We conclude that dialogue researchers should adopt clearer standards for transcription and markup format in dialogue corpora to be used in training a chatbot system more effectively. 
Keywords. Chatbot, matching algorithm, dialogue corpora. 
1. Introduction 
A chatbot is a conversational agent that interacts with users using natural language. Section two and three outline the linguistic knowledge representation and pattern matching algorithms of two chatbot systems: ALICE (ALICE 2002, Abu Shawar and Atwell 2002) and Elizabeth (Millican 2002, Abu Shawar and Atwell 2002). Both systems were adapted from the ELIZA program (Weizenbaum 1966) which emulated a psychotherapist. ALICE was found to be better suited for training using dialogue corpora because of its simple patterns templates and simple matching technique. The Dialogue Diversity Corpus (DDC) (Mann 2002) involves a collection of links to different dialogue corpuses in different domains. We used DDC samples to train ALICE, but we found several problems. Section three shows some example corpus transcripts and some problems these present. Section four presents the Java program that convert a dialogue from text to AIML format; this formalisation has helped us to see the main characteristics that must be found in dialogue corpora in order to use it for training a chatbot. 
2. ALICE 
ALICE (ALICE 2002, Abu Shawar and Atwell 2002): the Artificial Linguistic Internet Computer Entity, is a software robot or program that you can chat with using natural language. ALICE knowledge about English conversation patterns is stored in AIML files. AIML, or Artificial Intelligence Mark-up Language, is a derivative of Extensible Mark-up Language (XML). It was developed by the Alicebot free software community during 1995-2000 to enable people to input dialogue pattern knowledge into chatbots based on the A.L.I.C.E free software technology. 
AIML consists of data objects called AIML objects, which made up of units called topics and categories. The topic is an optional top-level element, has a name attribute and a set of categories related to that topic. Categories are the basic unit of knowledge in AIML. Each category is a rule for matching an input and converting to an output, and consists of a pattern which represents the user input and a template which implies the ALICE robot answer. The AIML pattern is simple, consisting only of words, spaces, and the wildcard symbols _ and *. The words may consist of letters and numerals, but no other characters. Words are separated by a single space, and the wildcard characters function like words. The pattern language is case invariant. 
3. Types of categories in ALICE 
There are three types of categories: atomic categories, default categories, and recursive categories. 
a. Atomic categories: are those with patterns that does not have wildcard symbols, _ and *, e.g.: 
<category> 
<pattern>10 DOLLARS</pattern> 
<template> WOW, what a cheap.</template> 
</category> 
In the above category, if the user inputs: 10 dollars, then ALICE answers: WOW, what a cheap. 
b. Default categories: are those with patterns having wildcard symbols * or _. The wildcard symbols match any input but they differ in their alphabetical order. Assuming the previous category, if the robot does not find the previous atomic pattern, then it will try to find the following default pattern: 
<category> 
<pattern>10 *</pattern> 
681
<template>It is ten.</template> 
</category> 
So ALICE answers: It is ten. 
c. Recursive categories: are those with templates having <srai> and <sr> tags which refers to simply recursive artificial intelligence and symbolic reduction. Recursive categories have many applications: symbolic reduction that reduces complex grammatical forms to simpler ones, divide and conquer that split an input into two or more subparts, and combine the responses to each, and dealing with synonyms by mapping different ways of saying the same thing to the same reply. 
c.1 Symbolic reduction 
<category> 
<pattern>DO YOU KNOW WHAT THE * IS</pattern> 
<template><srai>What is <star/></srai></template> 
</category> 
In this example <sari> is used to reduce the input to simpler form “what is”. 
c.2 Divide and conquer 
<category> 
<pattern>YES *</pattern> 
<template><srai>YES</srai><sr/><template> 
</category> 
The input is partitioned into two parts, “yes” and the second part; * is matched with the <sr/> tag. 
<sr/> = <srai><star/></srai> 
c.3 Synonyms 
<category> 
<pattern>HALO</pattern> 
<template><srai>Hello</srai></template> 
</category> 
The input is mapped to another form, which has the same meaning. 
4. Preparation for pattern matching in ALICE 
Before starting the pattern-matching algorithm, each input to the AIML interpreter must pass through two processes: normalization process, and producing input path from each sentence. 
1. Normalization process involves three phases. Firstly, substitution normalization is a heuristic applied to an input that attempts to retain information in the input that would otherwise be lost during the sentence splitting or pattern fitting normalization. Secondly, sentence splitting normalization splits input into two or more sentences using rules like “break sentence at periods”. Thirdly, pattern fitting normalization removes punctuation from the input, and converts input to uppercase. The normalization process is applied in this order and at least the pattern fitting normalization must be done. 
Example: If the input is: “I do not know. Do you, or will you, have a robots.txt file?” 
a. Substitution Normalized form is: 
“I do not know. Do you, or will you, have a robots dot txt file?” 
b. Sentence Splitting Normalized Form is: 
“I do not know.” 
“Do you, or will you, have a robots dot txt file” 
c. Pattern fitting Normalized Form is: 
“I DO NOT KNOW” 
“DO YOU OR WILL YOU HAVE A ROBOTS DOT TXT FILE” 
2. Producing Input Path, for each sentence an input path of the following form is produced. 
Normalized-Input<that>Tvalue<topic>Pvalue. Input path has three parts. Firstly, normalized input. Secondly, <that> tag which will be followed by a Tvalue that holds the previous robot answer if it exists or * otherwise. Thirdly, <topic> tag which will be followed by Pvalue that holds the topic name if it exists or * otherwise. Topic tag is an optional top level element having a name attribute and holding all categories related to that attribute name. 
Example: 
If the normalized input is: “YES”, previous robot output is: “DO YOU LIKE CHEESE”, and there is no topic name, then the input path will be: “YES <that> DO YOU LIKE CHEESE <topic> *”. 
682
5. ALICE Pattern matching algorithm 
The AIML interpreter tries to match word by word to obtain the longest pattern matching, which is the best one. This behaviour can be described in terms of the Graphmaster set of files and directories, which has a set of nodes called nodemappers and branches representing the first words of all patterns and wildcard symbols. 
Assume the user input starts with word X and the root of this tree structure is a folder of the file system that contains all patterns and templates, the pattern matching algorithm uses depth first search techniques: 
1. If the folder has a subfolder starts with underscore then turn to, “_/”, scan through it to match all words suffixed X, if no match then: 
2. Go back to folder, try to find a subfolder starts with word X, if so turn to “X/”, scan for matching the tail of X, if no match then: 
3. Go back to the folder, try to find a subfolder start with star notation, if so, turn to “*/”, try all remaining suffixes of input following “X” to see if one match. If no match was found, change directory back to the parent of this folder, and put “X” back on the head of the input. 
When a match is found, the process stops, and the template that belongs to that category is processed by the interpreter to construct the output. 
Example: assume the following categories: 
(1) <category> 
<pattern>_ WHAT IS 2 AND 2</pattern> 
<template> <sr/> <srai>WHAT IS 2 AND 2</srai> </template> 
</category> 
(2) <category> 
<pattern>WHAT IS 2 *</pattern> 
<template> <random> <li>Two.</li> <li>Four.</li> <li>Six.</li> <li>12.</li> </random> 
</template> 
</category> 
(3) <category> 
<pattern>HALO</pattern> 
<template> <srai>HELLO</srai> </template> 
</category> 
(4) <category> 
<pattern>HELLO</pattern> 
<template> <random> <li>Well hello there!</li> <li>Hi there!</li> 
<li>Hi there. I was just wanting to talk</li> <li>Hello there !</li> </random> 
</template> 
</category> 
If the user Input is: “halo what is 2 and 2 ?”, then the ALICE output will be: “Hi there! Four.” 
The process is as follows: 
1- This input will match category (1) that breaks it into two sentences: 
Sentence one: represented by <sr/> tag that matches (_ ) which is the word HALO. 
Sentence two: WHAT IS 2 AND 2. 
2- An atomic pattern is found for HALO and substituted by HELLO in category (3) and thenmatch it again with category (4); the answer will be selected randomly from the template list of category (4). Here it deals with synonyms since halo and hello has the same output. 
3- The interpreter tries to match WHAT IS 2 AND 2, no atomic pattern is found, so a backtracking technique is applied, scanning to find WHAT IS 2 AND *; again no match, another backtracking tries to find WHAT IS 2 *, a match found in category (2) and a response is selected randomly from the list. 
4- The AIML interpreter will combine these answers together and display them. 
683
6. Elizabeth 
Elizabeth is an adaptation of th Eliza program, in which the various selection, substitution, and phrase storage mechanisms have been enhanced and generalized to increase both flexibility and (potential) adaptability. Knowledge is stored as a script in a text file, where each line is started with a script command notation. These notations are single characters, one for each rule-type: 
W: Welcome message Q: quitting message N: No match 
V: Void input I: Input transformation / : Comment 
K: Key word pattern R: key word response M: Memorise phrase 
N: No match O: Output transformation &: Action to be perform 
Each script command has an index code that is generated automatically. It can also be indexed using a user special code. 
7. Elizabeth's Script file format 
A script file may contain at most four parts. Part one, script command lines holding welcome, void and no keyword messages. Part two, input transformation rules, which starts with “I” and maps input to another form to be compatible with the defined keywords. Part three, Output transformation rules, which starts with “O” and changes personal pronouns to be appropriate as a response. Part four, keyword patterns, which starts with “K” to denote keyword pattern to be matched, followed by robot response denoted by “R”. There are two type of keyword patterns: simple patterns: matching only single word, and composite patterns: match a sentence of phrase, words, string, letter, anything and nothing. For example: 
/ The script begins with Welcome, void, no keyword and quit messages 
W HELLO, I AM ELIZABETH. WHAT DO YOU LIKE TO TALK ABOUT? 
V CAN'T YOU THINK OF ANYTHING TO SAY? 
V ARE YOU ALWAYS CHIE? 
N TELL ME WHAT YOU LIKE DOING? 
Q GOODBYE! COME BACK SOON. 
/ Input transformation rules 
I MUM => mother 
I DAD => father 
/Output transformation rules 
O my => YOUR 
O YOU IS => YOU ARE 
/keyword transformations 
K MOTHER 
K FATHER 
R TELL ME MORE ABOUT YOUR FAMILY. 
R ARE YOU THE YOUNGEST IN YOUR FAMILY? 
K I LIKE [string]ING 
R HAVE YOU [string]ED AT ALL RECENTLY? 
If the user input is: “I like swimming”, robot answer will be: “HAVE YOU SWIMMED AT ALL RECENTLY?” 
8. Dynamic processing in Elizabeth 
Performing a set of actions modifies the script while the conversation is under progress. These actions have the following format: “& {script command}” and you can nest as many times as you want. These actions involve adding, memorization and deleting commands. These processes can be performed directly using “!” or after matching process. 
a. Adding new script commands, will add new script if it does not exist or replace it by new one if it is already found. Example: 
I my sister => my sister 
& {K MOTHER 
& {N DOES ANYTHING ELSE ABOUT YOUR MOTHER?} 
R HOW WELL YOUR MOTHER AND SISTER GET ON?} 
There are two actions to be performed. Action one, when matching occurs with the input rule, the system will search for the keyword “MOTHER”, if finds then the response denoted by “R” will be added to the response list of that keyword. If not found then the outer curly brackets will be added to the script. Action two, when matching with “MOTHER” occurs later on, then a no key word message will be added to the list or replace the old one if exists. 
684
b. Memorization commands, are used to hold the user or robot response for future matching. The command is denoted by M followed by memory name then the value. Example: 
K [ ] MY NAME [phrase] 
& {M name [phrase]} 
R YOUR NAME [phrase]? 
If the user input is: “my name is John” then the robot answer will be: YOUR NAME IS JOHN. And at the same time a memory name hold “John” is generated. 
c. Deleting commands delete any script command by using the back slash notation. Example: 
V\ : deletes all current void messages. 
M\ one : deletes the memorization command with index “one”. 
9. Elizabeth's Pattern matching algorithm 
Before starting the matching process, an active text is generated for each input by: converting user input to lower case, inserting spaces between words and punctuation, and removing some characters from the input, except: ! “ ‘ ( ) , - . 0..9 : ; ? a..z 
The matching process involves five stages. Stage one matches with input transformation rules. Stage two matches with keyword patterns. Stage three matches with output transformation rules. Stage four matches with void or no keyword messages. Stage five performs any dynamic process. 
Stage one: input transformation rules, all input rules are applied in turn to the active text, if match 
occurs with the left hand side of the rule then substitute it by the right hand side, and record 
or apply any dynamic process found. If the same rule is applicable to the active text more 
than 10 times in succession, then it is applied just once. 
Stage two: keyword pattern matching, all keywords are applied in turns until one match or no match at 
all. The response is giving according to the first match, so if there is a list of 
responses, one of them is selected randomly and the index code of this response will be 
recorded in order not to use it if the same match occurs in the next session. If there is a dynamic process, it will be applied or recorded to be performed in stage five. If no match at all found then go to phase four. 
Stage three: output transformation rules, includes two steps: apply the same algorithm used in stage 
one. Change the active text to upper case. 
Stage four: if no match occurs with the keyword patterns then either the active text is empty, and one 
of the void messages will be selected randomly, or no keyword found, and one of the no 
keyword messages will be selected. 
Stage five: perform any dynamic processes, performs the recorded actions if any exists. 
10. Implementing grammatical rules in Elizabeth 
Elizabeth has the ability to produce a grammar structure analysis of a sentence using a set of input transformation rules to represent grammar rules. This provides an introduction to some of the major concepts and techniques of natural language processing. 
Example: consider the following grammar: 
S => NP VP 
NP => D N 
VP => V NP 
This can be represented by the input rules to deal with certain nouns, verbs and determiners: 
I a => (A d) 
I the => (THE d) 
I cat => (CAT n) 
I dog => (DOG n) 
I likes => (LIKES v) 
I ([brak1] d) ([brak2] n) => (([brak1] D) ([brak2] N) np) 
I ([brak1] v) ([brak2] np) => (([brak1] V) ([brak2] NP) vp) 
I ([brak1] np) ([brak2] vp) => (([brak1] NP) ([brak2] VP) s) 
K [any?] 
R [any?] 
W TYPE A SENTENCE USING: A, THE, CAT, DOG, RABBIT, BITES, 
CHASES, LIKES 
This script starts from the Welcome message, so 
Robot: TYPE A SENTENCE USING: A, THE, CAT, DOG, RABBIT, BITES, 
685
CHASES, LIKES 
User input is: the cat likes the dog 
Response : (((THE D) (CAT N) NP) ((LIKES V) ((THE D) (DOG N) NP) VP) s) 
11. The Dialogue Diversity Corpus (DDC) 
From the our analysis of ALICE and Elizabeth chatbot systems, we conclude that ALICE is more suitable for corpus-based “retraining” because AIML is closer to the markup formats used in annotated corpora and because of AIML simplicity illustrated in its pattern templates and matching algorithm. We are investigating a corpus-based approach to generalizing AIML files to cover different dialogue domains in different languages. Since natural dialogue between a person and a computer should resemble a dialogue between humans as much as possible, we have learned from dialog corpora to generate AIML files, using a program to read dialog transcripts from the Dialog Diversity Corpus (DDC) (Mann 2002) and convert these to dialogue patterns and templates in AIML format. In this section we concentrate on the problems which arise when dealing with the DDC and illustrate these problems by examples and how the program reads dialogue and converts it to AIML format. 
12. Problems in the Dialog Diversity Corpus 
The DDC is not a single corpus, but a collection of links to different dialogue corpuses in different fields. For example: 
a. MICAS Corpus: Michigan Corpus of Academic Spoken English (MICASE 2002) , a collection of transcripts of academic speech events recorded at the university of Michigan. For example: 
Astronomy transcript: 
S1: circumpolar stars. So if I keep my pointer there, [S2: oh ] <ROTATES CEILING> 
everything else moves and we all get sick. <SS LAUGH> and we go backwards in time. And 
that's even more fun. 
S2: make it go really really fast. 
<SS LAUGH> 
S1: okay so that's how the sky is going to move, a couple of other things that we can do in here, 
um, this is a presentation of, the, grid, that we use to divide the sky, so these lines that run, 
north south what do we call those? 
S3: declination 
The Astronomy transcript was made using a digital audio tape recorder with two microphones. Problems illustrated are: long monologs, overlapping denoted by angle brackets, more than two speakers and extra annotations recorded actions such as <SS LAUGH>. 
b. CIRCLE corpus [6] (Center for interdisciplinary research on constructive learning environments), is a collection of transcripts holding different tutorial sessions, on topics such as physics, algebra and geometry. For example: 
b.1 Algebra transcript: 
2. TUTOR [Opening remarks and asks student to read out aloud and begin] 
3. STUD [Reads problem] Mike starts a job at McDonald's that will pay him 5 dollars and hour, 
Mike gets dropped off by his parents at the start of is shift. Mike works a “h” hour shift. Write 
an expression for how much he makes in one night? 
4. [Writes “h*5 = how much he makes”] 
5. TUTOR That's right number. 
b.2 Physics transcripts: 
T: [student name], I'd like you to read the problem carefully, and then tell me your strategy for 
solving this. 
S: ok 
[Pause 17 sec] 
hmm. 
[Pause 6 sec] 
T: thinking out loud as much as possible is good 
The Algebra transcript was made from a video tape of algebra session where the tutor and the student are sitting in the same room. Physics transcript was done from audio form tapes of the phone conversation, where the tutor was seated in a room and students were seated in a different one. The tutor and students communicated with a conference phone that let only one person talk at a time. The 
686
main problem here is that within the same corpora, different formats were used to distinguish speakers, and linguistic annotations such as pauses in the algebra corpus were denoted by colon whereas in the physics corpus it was directly written inside brackets. 
c. CSPA Corpus of Spoken Professional American-English (Athel 2002), includes transcripts conversations of various types occurring between 1994 and 1998. For example: 
LANGER: Hello, I'm delighted to be here. 
I have carefully read and heard about the University of Albany, the State University of New York. And I'm also the director of the National Research Center on English Learning and Achievement. 
STRICKLAND: Her mother wrote the stances. 
(Laughter) 
KAPINUS: Dorothy, I might add also that Judith probably has more history with NAEP than just about that I know of, you know, NAEP and reading. 
STRICKLAND: Yes, yes. We will really turn to you as a very important resource, Judith. 
And we have a new member, Gloria Lopez Gutierrez. 
And, Gloria, tell us a little bit about yourself. 
In this transcript, we noticed long turns/monologues, and the transcripts were not “anonymised”: speakers’ last names were used. 
d. The TRAINS Dialog Corpus (CISD 2002), a corpus of task-oriented spoken dialogue, that has been used in several studies of human-human dialogue. For example: 
utt9 : s: okay <sli> um 
utt10 : what you'll have to do is you'll have to uh pick out an <sli> 
uh an engine <sli> and schedule a train to do that 
utt11 : u: okay <sli> um <sli> engine <sli> two 
utt12 : s: + okay + 
utt13 : u: + from + Elmira 
utt14 : s: + mm-hm + 
utt15 : u: + to + Corning 
The main problem is deling with extra-linguistic annotation like + and <sli>. 
e. ICE-Singapore: International Corpus of English, Singapore English (Nelson 2002). For example: 
<$B> 
<ICE-SIN: S1A-099#33:1:B> 
How how are things otherwise 
<ICE-SIN:S1A-099#34:1:B> 
Are you okay 
<$A> 
<ICE-SIN:S1A-099#35:1:A> 
Uhm okay lah 
<ICE-SIN:S1A-099#36:1:A> 
Bearing up lah 
<$B> 
<ICE-SIN:S1A-099#37:1:B> 
Ah hah 
<$A> 
<ICE-SIN:S1A-099#38:1:A> 
Ya I mean I don't really feel comfortable talking about it over the phone so when I see you I'll tell you about it lah 
This is a spoken dialog recorded through the phone. Problems here are: unconstrained conversations, a lot of linguistic annotation, and great variation in turn length is noticed. 
<ICE-SIN:S1A-099#35:1:A> refers to text unit 35, in subtext 1, uttered by speaker A. ICE-SIN refers to corpus name ICE Singapore and S1A refers to academic spoken. 
e. Mishler Book Medical Interviews (Mishler 1985) is an example of a scanned text image, including a dialogue between a patient and his physician. The problem is that scanned image was not converted to 
687
text format and also uses extra-linguistic annotations. There are many more examples of printed books containing interview transcripts, but in general these are inaccessible if not in text-file format. 
13. Desired dialogue corpus characteristics for Machine Learning 
We have developed a Java program to read dialogue transcripts from the DDC and convert these to dialogue patterns and templates in AIML format in order to retrain ALICE. A lot of problems arise when dealing with the DDC, as illustrated in the previous section, which can be summarised as follows: 
a. No standard formats to distinguish between speakers. 
b. Extra-linguistic annotations were used. 
c. No standard format in using linguistic annotations. 
d. Long turns and monologues. 
e. Irregular turn taking (overlapping). 
f. More than one speaker. 
g. Scanned text-image not converted to text format. 
For example, if the program reads the dialog transcript from the astronomy transcript in MICAS corpus, every pair of speakers will generate a new AIML category where the first speaker represents the pattern, and the second speaker represents the template. But first the program calls a filtering procedure to remove the linguistic annotations such as: <SS LAUGH>, and the overlapping like: [S2: oh ]. The AIML category generated is: 
<category> 
<pattern> CIRCUMPOLAR STARS. SO IF I KEEP MY POINTER THERE, EVERYTHING ELSE MOVES AND WE ALL GET SICK. AND WE GO BACKWARDS IN TIME. AND THAT'S EVEN MORE FUN.</pattern> 
<template>make it go really really fast.</template> 
</category> 
It is clear that to solve these problems to obtain a good dialog model, a filtering process must be applied first to these transcripts to remove all unnecessary tags and other markup. Because of the variation in formats and annotations for theses transcripts even in the same corpus, each transcript has to be filtered and processes differently, which contradicts the generalization objective. 
When re-engineering ALICE to a new domain or conversation style, the patterns and templates learnt from a training corpus are only a raw prototype: the chatbot and AIML files must be tested and revised in user trials. One of the main design considerations is how to plan the dialogue. A good dialogue design would mean less time testing and re-implementing AIML files. 
In order to train a chatbot system with minimal need to post-edit the learnt AIML, dialogue corpuses should have the following characteristics: 
a. Two speakers. 
b. Structured format. 
c. Short, obvious turns without overlapping, and without any unnecessary notes, expressions or other symbols that are not used when writing a text. 
Even such “idealised” transcripts may still lead to a chatbot which does not seem entirely “natural”: although we aim to mimic the natural conversation between humans, the chatbot is constrained to chatting via typing, and the way we write is different from the way we speak. 
To date our machine-learnt models have not included linguistic analysis markup, such as grammatical, semantic or dialogue-act annotations (Atwell 1996, Atwell et al 2000), as ALICE/AIML makes no use of such linguistic knowledge in generating conversation responses. However, the Elizabeth chatbot does allow for more sophisticated conversation modelling including such linguistic features (see Abu Shawar and Atwell 2002). 
14. Using Wmatrix to compare human and chatbot dialogues 
Wmatrix (Rayson 2002) is a tool to provide a data driven method to compare between different sized corpora in three levels: word, Part-of-Speech (PoS), and semantic-tag analysis. The comparisons results are viewed as frequency lists ordered by log-likelihood ratio. LL values indicate the most important differences between corpora. We used Wmatrix to compare human-to-human dialogues extracted from the DDC corpora (Mann 2002) and human to computer dialogues extracted from chatting with ALICE (Spielberg 2000). Four different corpuses in different fields and sized were investigated against the same chatbot dialogue in semantic, PoS and lexical level, to illustrate the main 
688
common differences between human-to-human conversation and human to computer chatting. The DDC transcripts and ALICE transcript are not equal in size, but Wmatrix compensates by providing normalised difference scores. Screenshots below illustrate the use of Wmatrix, to compare ALICE transcripts with the Astronomy human dialogue transcript extracted from the MICASE Corpus (MICASE 2002), a conversation between four speakers in an introductory astronomy discussion. The screenshots show the most important differences in semantic, Part-of-Speech and lexical levels between ALICE chatbot dialogue (file 1) and the astronomy transcript (file 2). 
Sorted by log-likelihood value Item O1 %1 O2 %2 LL Z1 34 3.01 22 0.41 + 52.21 Personal names 
E2+ 16 1.42 6 0.11 + 32.44 Liking 
W1 2 0.18 102 1.91 - 26.27 The universe 
M6 7 0.62 151 2.82 - 24.95 Location and direction 
M1 0 0.00 59 1.10 - 22.59 Moving, coming and going 
H4 8 0.71 1 0.02 + 22.06 Residence 
F1 6 0.53 0 0.00 + 20.97 Food 
Q2.1 25 2.22 35 0.65 + 19.27 Speech etc:- ive 
T1.1.3 6 0.53 121 2.26 - 18.95 Time: General: Future 
Q2.2 37 3.28 69 1.29 + 18.60 Speech acts 
Screenshot 1; the semantic comparison 
Sorted by log-likelihood value Item O1 %1 O2 %2 LL PPIS1 55 4.88 0 0.00 + 192.23 
VD0 43 3.81 27 0.50 + 67.27 
PPIS2 1 0.09 129 2.41 - 41.15 
CC 10 0.89 230 4.30 - 39.86 
PPY 80 7.09 155 2.90 + 37.52 
CS 4 0.35 116 2.17 - 23.31 
ZZ1 0 0.00 56 1.05 - 21.44 
DD1 9 0.80 151 2.82 - 19.97 
MC 4 0.35 98 1.83 - 17.75 
DD2 0 0.00 34 0.64 - 13.02 
Screenshot 2; the POS comparison 
689
Sorted by log-likelihood value Item O1 %1 O2 %2 LL do 44 3.90 35 0.65 + 58.69 
i 54 4.79 67 1.25 + 48.04 
we 1 0.09 129 2.41 - 41.15 
so 1 0.09 117 2.19 - 36.75 
and 8 0.71 195 3.65 - 35.19 
Emily 9 0.80 0 0.00 + 31.46 
you 72 6.38 151 2.82 + 28.91 
this 0 0.00 70 1.31 - 26.80 
you_know 8 0.71 1 0.02 + 22.06 
here 0 0.00 55 1.03 - 21.06 
am 6 0.53 0 0.00 + 20.97 
'll 1 0.09 71 1.33 - 20.14 
Screenshot 3; the word comparison 
15. Conclusions 
Two chatbot systems ALICE and Elizabeth were reviewed in this paper. We decide to train ALICE rather than Elizabeth to learn from human dialogue corpora for two reasons. Firstly, the AIML format is closer to the markup format used in annotated corpora. Secondly, the simplicity in generating patterns/templates, and applying simple pattern matching technique. Our main conclusion relating to Corpus Linguistics is that the Dialogue Diversity Corpus (DDC) illustrates huge diversity in dialogues, not just in the subject area and speaker background/register but also in mark-up and annotation practices. We urge the dialogue corpus research community to agree standards for transcription and markup format: this would help us, and others too. 
Expanding AIML files using least frequent word and investigating how to incorporate corpus-derived linguistic annotation into an Elizabeth-style chatbot pattern file are the future direction of this research. 
16. References 
ALICE 2002 A.L.I.C.E AI Foundation , http://www.alicebot.org/ or http://alicebot.franz.com/ 
Abu Shawar B, Atwell E 2002 A comparison between ALICE and Elizabeth chatbot systems. School of Computing research report 2002.19, University of Leeds. 
Millican P 2002 Elizabeth's home page http://www.etext.leeds.ac.uk/elizabeth 
Mann W 2002 Dialog Diversity Corpus http://www-rcf.usc.edu/~billmann/diversity/DDivers-site.htm 
MICASE 2002 MICASE online homepage, http://www.hti.umich.edu/m/micase/ 
Rayson Paul, 2002 “Matrix: A statistical method and software tool for linguistic analysis through corpus comparison”, Unpublished PhD thesis, Department of Computing, Lancaster University 
Ringenberg M 2002 CIRCLE's tutorial archive http://www.pitt.edu/~circle/Archive.htm 
Athel 2002 Corpus of Spoken Professional American-English: description, http://www.athel.com/corpdes.html 
CISD 2002 TRAINS Dialogue Corpus, http://www.cs.rochester.edu/research/cisd/resources/trains.html 
Nelson G 2002 International Corpus of English: the Singapore Corpus user manual, http://www-rcf.usc.edu/~billmann/diversity/ICE-SIN_Manual.PDF 
Mishler E 1985 The discourse of medicine: dialectics of medical interviews, New Jersey, Ablex http://www-rcf.usc.edu/~billmann/diversity/Tr.5.1a.gif 
Atwell E 1996 Machine Learning from corpus resources for speech And handwriting recognition. In Thomas J, Short M (eds), Using corpora for language research: studies in the honour of Geoffrey Leech. Harlow, Longman, pp151-166. 
Atwell E, Demetriou G, Hughes J, Schiffrin A, Souter C, Wilcock S 2000 A comparative evaluation of modern English corpus grammatical annotation schemes. ICAME Journal 24: 7-23. 
690

Towards the Extraction of Conceptual Information from Corpora 
Gerardo Sierra, Alfonso Medina, Rodrigo Alarcón, César A. Aguilar 
Instituto de Ingeniería, UNAM. México 04510, D. F. 
Fax: (015255) 56 22 81 37 
{GSierraM, AMedinaU, RAlarconM, CAguilar}@iingen.unam.mx 
http://iling.torreingenieria.unam.mx 
1 Introduction 
To obtain the terminology of a specific domain, as well as the corresponding definitions of every term, terminologists could either consult specialists, or could consult texts in the area. In order to simplify this last option, computational terminology has developed some tools to facilitate textual analysis for terminographical purposes based on specialised corpora. 
From a computational linguistics point of view, specifically related to information extraction, terminology uses statistical and rule-based methods [Cabré et. al. 2001] to extract terms from specialised texts. Furthermore, terminology needs to identify the corresponding definitions of a specific term. Often, when an author introduces a new term, which is not well known to the readers, he/she provides the definition emphasising the new concept with a set of syntactic and typographical features. We thus call definitional context the structure consisting of the term, the definition and the emphatic features in a specialised text. 
In order to develop a tool capable of extracting definitional contexts from annotated corpora, an inventory of recurrent patterns used by authors to introduce concepts is necessary, as well as a computational linguistic technique capable of identifying concepts from specialised texts. 
One of the objectives of the Language Engineering Group, from the Instituto de Ingeniería, UNAM (National Autonomous University of Mexico) is to extract conceptual information from corpora. For this reason, the group is working on the study of recurrent patterns in definitional contexts, the elaboration of an annotated corpus on engineering, and the development of the tools required for conceptual information extraction. 
In this paper we present the process of developing a definitional context extraction tool. First, we present the minimal elements of a definitional context. Second, we define a typology of all the recurrent patterns found in definitional contexts of Spanish specialised texts. Third, we present the kinds of tagging necessary. And fourth, we briefly mention the characteristics that a search engine must have. 
2 Definitional contexts 
Our investigation is based on previous efforts such as the systematic identification of definitions based on lexical and metalinguistic patterns [Pearson, 1998], the analysis of Explicit Metalinguistic Operations [Rodríguez, 1999] and the analysis of Knowledge-Rich Contexts [Meyer, 2001]. 
As a result of these efforts, we have established that a definitional context is a structure in a specialised text consisting of two minimal elements, the term and the definition, and the emphatic features which accompany them. 
2.1 Term 
A term is a linguistic sign (i.e., one word or one set of words), that makes reference to a specialised concept (Cabré, 1999; Estopa, 2001). The most relevant features of a term are: 
  Syntactic structure. In Spanish a term may be constituted by a noun phrase, sometimes followed by one or more prepositional phrases. In some cases, a term can be a verbal phrase (infinitive verb functioning as a noun), and an optional set of prepositional phrases [Cardero, 2001]. 
  Highlighting elements. There are some cases where typographical marks (italic, bold, underline, capital letters) are employed to highlight the term within the definitional context. 
  Anaphoric relationship. Terms do not necessarily appear explicitly in a definitional context. That is, they do not appear next to their definition; instead, there is an anaphoric relation to the term within the definitional context. For example, a term can be presented as the theme of a section in 
691
a text, given by the title, and explained within the section without being mentioned. That is, the anaphoric reference between the term and the definition may be the only clue as to their association. 
2.2 Definition 
A definition is the description of a concept represented by a term. This description establishes relationships with other terms, in order to delimitate the meaning of the concept. The most relevant features to understand a definition are: 
  Typology. The starting point to identify the different types of definitions is the Aristotelian analytic definition [Sager/Ndi-Kimbi, 1995; Wilks/Slator/Guthrie, 1996], as represented in figure 1: 
 Analytic: genus + differentia. 
 Single genus: no further description of the differentia. 
 Synonymic: a strong semantic relationship with the genus. 
 Functional: differentia gives the function of the concept. 
 Extensional: differentia enumerates the parts composing the concept. 
Figure 1. Analytic definition 
  Syntactic structure. Most definitions start with a noun phrase, although verbal phrases are common in functional ones. 
 Noun phrases can start with a quantifier, determiner, or demonstratives. 
 The genus may consist of a set of prepositional phrases after the initial noun phrase. 
 The differentia may be introduced by subordinated sentences composed by noun, adjective and prepositional phrases. 
  Highlighting elements. Some highlighting features emphasise the presence of definitions. 
 Quotation marks are one of the most common typographical features. 
 Authoring references usually accompany definitions. 
3 Recurrent patterns in definitional contexts 
We have seen that a definitional context is composed of two minimal elements: the term and the definition. Also, we have observed that there are some explicit characteristics that function as visual effects or grammatical features to help readers to identify the presence of an important concept. All these elements will be called patterns. We use the following symbols to represent them: T (term), D (definition), tm (typographical mark), VP (verbal predication) and PP (pragmatic predication). 
Among possible sequences of these patterns, T could be connected with the definition by a VP or a tm. At the same time, a tm could characterise a T or a D. Here we will use the “+” sign to represent the sequences of those elements, while we concatenate the simultaneous combination of tm with T or D. 
To study these patterns systematically we propose grouping them in three different sets: typographical, syntactic and mixed patterns. 
3.1 Typographical patterns 
692
We will call typographical patterns those sequences of terms and definitions connected by punctuation marks and commonly highlighted with typographical features (see examples in table 1). 
  These patterns are the simplest forms found in specialised texts. They resemble the kind of definition commonly found in dictionaries. 
  There are not verbs connecting terms and definitions; instead, punctuation marks appear, i.e. period, colon, coma, end of paragraph, etc. 
  Typographical features are visual effects which emphasise the presence of either or both terms and definitions (bold, capital letters, quotation marks, etc.). 
Table 1. Examples of typographical patterns 
Pattern Definitional context T tm + tm + D tm Diseno: Desarrollo de configuraciones para la resolución de algún problema en base y sujetándose a sus restricciones. T tm + tm + D tm DESASTRE. Perturbación de la actividad normal que ocasiona pérdidas o danos extensos o graves. T tm + tm + D “Impactos agregados sociales” ¶ Los que impactan a la sociedad, produciendo, por ejemplo, la perturbación de las relaciones familiares 
3.2 Syntactic patterns 
Another kind of definitional contexts is that where syntactic forms are used either to connect the term and the definition, or to provide some additional information about the concept. These patterns are called syntactic patterns and use pragmatic or verbal predications. 
Table 2. Examples of syntactic patterns 
Pattern Definitional Context PV + T + D Se considera como protección civil a la actividad solidaria de los diversos sectores que integran a la sociedad… T + PP + PV + D Un soporte logístico de plataforma, de manera general, se define como un territorio equipado para el desarrollo de actividades logísticas… PP + T + PV + D De acuerdo con esta conceptualización, los danos probables se definen como el riesgo que corre el SA por ser expuesto al… 
  Pragmatic predications. These syntactic forms give us information about usage or treatment of the term. Also, they give us some clues to understand a concept in the context it appears. 
 Pragmatic predications include adverbial phrases, e.g. generalmente (generally), prepositional phrases, e.g. en términos generales (in general terms) and simple words, e.g. concepto (concept). 
 The structure of pragmatic predications relies on the different styles each author uses to write a specialised document. 
  Verbal predications. These forms use a verb to connect a term with a definition. The verbs used in definitional contexts are commonly called metalinguistic verbs. Generally, metalinguistic verbs are employed to refer to language itself. In Spanish, some of the most important metalinguistic verbs are definir, describir, denominar (to define, to describe, to denominate). Because of their structure, verbal predications could be classified in two groups: 
 Simple forms use a verb or a verbal periphrasis that could appear with a grammatical particle, e.g. afirma que (he/she/it asserts that). Normally, the name of the author who defines the term, or the theory or textual reference where it is defined, appears within the definitional context. 
 Complex forms use the pronoun se plus a verb or a verbal periphrasis; e.g. se define como (it is defined as). This pronoun is the only difference with respect to simple forms. There is a semantic distance between the author and the definition he or she provides. This semantic distance is often provided implicitly by the impersonal meaning of the pronoun se along with the name of the author. 
693
3.3 Mixed patterns 
There is a kind of pattern that combines both typographical and syntactic elements. These patterns will be called mixed patterns. 
  They represent a more robust structure than the non-mixed patterns, because they emphasize typographically (punctuation and typographical marks) and syntactically the presence of definitional contexts. 
  Mixed patterns are the most recurrent structure: they are present in 60% of the definitional contexts of our analysed corpus. 
  As described above, typographical marks emphasise both the term and/or definition, while verbal predications are used to connect the terms with the definitions. Some pragmatic predications may also appear. 
Table 3. Examples of mixed patterns 
Pattern Definitional Context T tm + VP + D a. Canal de comercialización es el conjunto de actores y actividades que interactúan para que un bien producido… T tm + VP + D - Las actividades se definen como los elementos principales de una acción… PP + T + VP + D tm Según G. Malagón (1996, p.18) un hospital se define como: “una parte integrante de la organización médica, cuya función es la de proporcionar a la población… 
4 Corpus tagging 
All of the above points to the idea of annotating corpora in order to extract the conceptual information contained in them. There are at least two levels at which the annotation schemes may be conceived; namely, that of visual effects or typographical tagging, and that of linguistic structure, consisting of both, part of speech tagging and syntactic or parsing structure tagging. 
The analysis of our corpus resulted in the following model, which represents the possible combinations of the patterns discussed above: 
(PP/VP) + T/D (tm) + (PP/VP/tm) + D/T (tm) + (PP) 
We can see from this model that a definitional context could be identified by determining the presence of certain combinations of these constitutive elements. The automatisation of this implies the elaboration of an annotation scheme, which would necessarily include typographical and syntactic tags 
4.1 Typographical tagging 
Even when typographical tagging is not very extended in corpus-based research, for conceptual information extraction it is very important to retain the marked typography from the original. As we have seen, terms and definitions are usually emphasised because of the intrinsic linguistic interest given by the author. Therefore, marking typographical features is relevant. 
Our preliminary analysis of definitional contexts let us identify the required tags to highlight the presence of concepts introduced in a printed text. The ways an author uses to emphasise the concepts differ from other authors and go beyond the most known tags in corpora. A deep study of the patterns shows this variety of ways: 
  Terms appearing as the title of a text section and only mentioned paragraphs below in an anaphoric relation with its definition. 
  Text superimposed outside the main text (pull quotes and footnotes) giving the definition for technical terms. 
  Bullets displaying the term followed by a comma and the definition. 
  Terms (and sometimes definitions) highlighted through word spacing, different font size, type or color, embedded within the text. 
694
  Quotation within the same paragraph distinguished only by quotation marks or reference to an author. 
Therefore, our corpus is rich in typographical encoding. Some of our tags include: different font, point type and color; word spacing; capital letters, small caps, subscripts and superscripts; division head elements such as titles, bylines, bullets, etc.; footnotes, endnotes, quotes and pull quotes. 
Tagging punctuation is unnecessary as it can be searched without any encoding and the cost to encoding it is too high. However, it is convenient that the search engine clusters similar punctuation elements for conceptual extraction, such as: parentheses (including square brackets and braces); points (either full stop, comma, period and dash, colon, etc.); quotation marks (single or double). 
4.2 POST 
From the discussion presented above, it can be seen that the application of traditional POS tags to corpora can help when subsequent conceptual information extraction procedures are applied. 
POS tagging is necessary to determine the internal structure of the prepositional, noun, verbal and adverbial phrases which build terms, definitions and, not less importantly, verbal and pragmatic predications. 
EAGLES tag set provides special codes to specify attributes associated to each POS tag, which may or may not be relevant to information extraction. For instance, Spanish adjectives exhibit several attributes such as gender (feminine/masculine) and number (singular/plural). 
However, for our purposes, according to the patterns presented above, most of the attributes typically associated to the Spanish part of speech categories are not needed for conceptual information extraction. First and second attributes of EAGLES encoding (category and type) will be enough for our purposes. 
  It does not matter whether the verb forms exhibit inflexion morphemes corresponding to the future, present or past tenses, or to the particular modes (indicative, subjunctive, infinitive, gerundive, etc.). 
  Whether or not the verb is an auxiliary verb seems to be very relevant. 
  Grammatical person can be expected to be important: in our data, conceptual information is predominantly introduced by means of third person verb forms. 
4.3 Parsing 
After considering the structure of terms, definitions, and syntactic and pragmatic predications, as well as the syntactic relations among all of these structures, it should not be surprising that parsing annotation may also be of help when it comes to automatic conceptual information extraction. 
  Noun phrases. They may function as terms and are important components of definitions. 
 As specified above, a term may consist of both a noun phrase and a prepositional phrase, or combinations of both [Cabré et al, 2001]. 
 Definitions may be composed of at least one well formed sentence, which is likely to contain any of the syntactic structures mentioned above. 
  Pragmatic predications. Their nature is more related to author's style or specialised jargon idiosyncrasies. Nevertheless, these are also constituted by syntactic phrases; .e.g. the prepositional phrase en términos generales (in general terms), the noun phrase la característica principal (the main characteristic) and the adverbial phrase tradicionalmente (traditionally). There might even be some overlapping, e.g. prepositional phrases may have an adverbial function and constitute pragmatic predications: de manera general (in a general manner), según G. Malagón (according to G. Malagón). 
  Verbal phrases. Metalinguistic verbs are important clues to detect definitional contexts, as specified above. However, we have observed that non-metalinguistic verbs must also be considered; namely, those constituting verbal predications such as se visualiza como (it is visualized as), se basa en (it is based on). 
  Other structures. Certain collocations have to be contemplated; for instance, those structures formed by a verb and a noun, where the verb has suffered some semantic erosion: tiene la finalidad de (it has the aim of), tiene la función de (it has the function of), etc. In a few words, we should consider the structure and variation of verbal predications in order to determine whether or not a verb functions as a connecting element within a definitional context. 
695
Tagging all of these structures will help a search engine to determine what sequences of syntactic patterns occur that could signal the presence of a definitional context and to discriminate between what belongs to the definition and what does not. 
5 Minimal requirements for a search engine 
A conceptual information extraction system must allow the user to retrieve the elements of a definitional context from a specialised corpus: terms, definitions, verbal and pragmatic predications, and the typographical features involved. This implies the search for concordances and collocations of these elements. 
  Term extraction. Need to search simple and lemmatised words, syntactic structures and typographical features. 
  Verbal and pragmatic predication extraction. Need to search simple and lemmatised words, and syntactic structures. 
  Definition extraction. Need to search syntactic structures and typographical features, as well as the retrieval of all of the above elements. 
To retrieve definitional contexts, the system will permit the search for combinations of the elements presented here. 
6 Conclusions and future work 
In this paper, we have examined some recurrent patterns that characterise definitional contexts, namely syntactic and typographical. The latter are the visual effects that authors may apply to texts in order to emphasise important information such as definitional contexts. The former refer to the linguistic structure typically found to codify conceptual information and can be embodied by syntactic and pragmatic predications. On the one hand, what we have called syntactic predications are connectors among terms and definitions and typically contain verbs called metalinguistic because they refer to language itself. On the other hand, pragmatic predications give us information about author's style and usage or treatment of the term. More importantly, they give us relevant clues to understand concepts in their contexts. All of this is a motivation to annotate corpora for automatic extraction of conceptual information. It is clear that our annotation scheme is to consider both typographical and linguistic information. In this sense, it will include at least typographical, part of speech, and parsing tagging. 
Our investigation can certainly be extended in different directions, some of which are in progress. 
  Definitions. Further studies of their structure should be permanently undertaken given the complexity of what a definition is. It will let us delimitate the boundaries of a definition in a definitional context. 
  Verbal predications. We need to identify more verbs that can be employed to define terms. Also, we need to evaluate their accuracy. 
  Pragmatic predications. It is important to study the structure of pragmatic patterns in order to expand our initial paradigm. 
  Verbs and definitions. We consider necessary to explore how a specific type of definition relies upon a particular verbal predication. L'Homme [2002] and Estopa, Lorente and Foguera [2002] have explored this topic. 
  Compound patterns. Although we have not described them here, there are other patterns of definitional contexts. These patterns are sequences where more than one term is defined. Pragmatic predications become essential in these sequences. 
  Anaphora. There are anaphoric relations between terms and definitions. They are mechanisms of textual cohesion (Botley & McEnery, 1999). Researching these may be of help for the localisation of definitional contexts and their boundaries. 
696
7 References 
Botley S & McEnery A M 1999. “Discourse Anaphora. The Need of Synthesis”. In Botley & McEnery (edits.), Corpus-based and Computational Approaches to Discourse Anaphora. Amsterdam/Philadelphia, John Benjamins Publish. 
Cabré, M T 1999. La terminología. Representación y comunicación. Barcelona, IULA-UPF. 
Cardero, A 2001. El procesamiento de una terminología. Referencia especial a la terminología de control de satélites en el área de las telecomunicaciones en México. PhD Thesis, Mexico, UNAM. 
Estopa, Rosa 2001. “Elementos lingüísticos de las unidades terminológicas para su extracción automática”. In Cabré T, Feliu J (eds), La terminología científico-técnica. Barcelona, IULA-UPF. 
L'Homme, M.C. 2002. "What can Verb and Adjectives tell us about Terms?" In Terminology and Knowledge Engineering, TKE 2002. Proceedings, August 28-30 2002, Nancy (France), pp. 65-70. 
Rodríguez, C. 1999. Operaciones Metalingüísticas Explícitas en Textos de Especialidad. Treball de Recerca. M.A. Thesis. Barcelona, IULA-UPF. 
Sager, J./Nidi-Kimbi, A. (1995); “The conceptual structure of terminological definition an their linguistic realisations: A report on research in progress”. Terminology, 2(1): 61-85. 
Wilks Y, Slator B, Guthrie L, 1996 Electric Words. Dictionaries, Computers and Meaning. Cambridge, Mass. MIT Press. 
697

Constraints for corpora development and validation1 
Kiril Simov, Alexander Simov, Milen Kouylekov 
BulTreeBank project 
http://www.BulTreeBank.org 
Linguistic Modelling Laboratory - CLPPI, Bulgarian Academy of Sciences 
Acad. G.Bonchev Str. 25A, 1113 Sofia, Bulgaria 
Tel: (+3592) 979 28 25, (+3592) 979 38 12, Fax: (+3592) 70 72 73 
kivs@bultreebank.org, adis_78@dir.bg, mkouylekov@dir.bg 
1 Introduction 
In this paper we consider corpora as a set of XML documents. The guidelines for the creation of the corpora determine the semantics of the data, stored in them. Usually the guidelines prescribe the actual structure of the corpora, the used symbols, their meaning and the relations among them. Ideally, the software supporting the creation of a corpus has to allow all the constraints that follow from the guidelines to be imposed over the XML representation of the corpus. To the best of our knowledge, such software does not exist yet. The main problems come from the complexity of the data in the corpus and the impossibility it to be to completely formalized. 
The XML technology provides two similar mechanisms for imposing constraints over XML documents (with respect to well-formedness): Document Type Definition (DTD) mechanism, and XML Schema. Both of them impose constraints over the content of the elements in the document. These constraints are local to the content of the given element and don't depend on the context of this element. In the paper we describe a tool for imposing additional constraints over XML documents, which allows representation of non-local dependencies among elements in one or several XML documents. The tool is a part of the CLaRK System (see (Simov et. al. 2001)) and it is based on XPath language. The CLaRK System is an XML-based system for corpora development. It incorporates several technologies: 
• XML technology (see (XML 2000), (XPath 1999), (XSLT 1999)); 
• Unicode (see (UNICODE 2003)); 
• Regular Cascade Grammars (see (Abney 1996), (Simov et al. 2002b)); 
• Constraints over XML Documents. 
On the basis of these technologies the following tools are implemented: XML Editor, Unicode Tokeniser, Sorting tool, Removing and Extracting tool, Concordancer, XSLT tool, Cascaded Regular Grammar tool, etc. The system is implemented in Java and it is freely available at the project site http://www.bultreebank.org/clark/index.html. 
The structure of the paper is as follows: in the next section we give a short description of the technologies behind the CLaRK System. The third section describes the constraints implemented in the system. In the next section we demonstrate the applicability of the constraints for semi-automatic disambiguation of Bulgarian morphologically annotated texts. The last section concludes the paper. 
2 Technologies behind CLaRK System 
As it was mentioned in the Introduction, the CLaRK System is based on the following technologies: XML technology, which supports document editing, storing, querying, etc; Unicode for internal representation of the documents; Regular cascade grammars, which facilitate the implementation of taggers, chunkers, searching; Constraints over XML documents for monitoring. Here we briefly consider each of these technologies. 
1 The work on the system is currently supported by BulTreeBank project funded by the Volkswagen-Stiftung, Federal Republic of Germany under the programme “Cooperation with Natural and Engineering Scientists in Central and Eastern Europe” contract I/76 887. 
698
2.1 eXtendable Markup Language (XML) 
XML stands for eXtendable Markup Language (see (XML 2000)) and it emerged as a new generation language for data description and exchange for Internet use. The language is more powerful than HTML and easier to implement than SGML. Starting as a markup language, XML evolved into a technology for structured data representation, exchange, manipulation, transformation, and querying. The popularity of XML and its ease for learning and use made it a natural basis of the CLaRK System. Additionally, the established standards for corpus descriptions like CES (Corpus Encoding Standard 2001) and TEI (Text Encoding Initiative 1997) are already adapted to meet the XML requirements. XML defines the notion of structured document in terms of sequences and inclusions of elements in the structure of the document. The whole document is considered as an element, which contains the rest of the elements. The elements in the structure of a document are marked-up by tags. Tags either surround the content of an element, or mark some points in the document. In the first case, the beginning of an element is marked-up by an opening tag, written as <tagname>, and the end is marked-up by a closing tag, written as </tagname>. The tags of the second kind, the so-called empty elements, are represented as <tagname/>. Each element can be connected with a set of attributes and their values. The currently assigned set of attributes of an element is recorded within the opening tag of the element or before the closing slash in an empty element. 
One important requirement for an XML document is that elements having common content must be strictly included one into another. It means that overlappings between elements is not allowed. Documents that meet this requirement (and some other - for details see (XML 2000)) are called well-formed. For example, the following document is not well-formed and thus it is not an acceptable XML document: 
<doc><el1> … <el2> … </el1> … </el2></doc> 
There are two mechanisms for imposing constraints over XML documents: DTD (Document Type Definition) and XML Schema. Both define the possible inclusions of elements and the possible sequences of elements within the content of an element. In addition, XML Schema defines the shape of textual elements, which in the DTD are treated as a whole string. An XML document containing elements, whose content obeys the restrictions, stated in a DTD or in an XML Schema, is valid with respect to the DTD or the XML Schema. 
2.2 XML Path Language (XPath) 
Another important language, defined within XML world and used within the CLaRK System, is the XPath language. XPath is a powerful language for selecting elements from an XML document. The XPath engine considers each XML document as a tree where the nodes of the tree represent the elements of the document. The document's most outer tag is the root of the tree and the children of a node represent the content of the corresponding element. Attributes and their values of each element are represented as an addition to the tree. This section is based on the following document (XML Path Language (XPath) version 1.0. W3C Recommendation. http://www.w3.org/TR/xpath). 
The XPath language plays a crucial role in the constraints of the CLaRK System, because it can be used for checking whether some configurations of data are presented in the document. In the implementation of the language in the system we have extended the above standard with variables and access to multiple XML documents. For instance, the following XML document: 
<S> 
<NP><N number="sing"> John </N></NP> 
<VP> 
<V tense="present"> loves </V> 
<NP><N number="sing"> Mary </N></NP> 
</VP> 
</S> 
is represented as the following tree: 
699
In this picture the rectangles correspond to elements in the XML document and they are called element nodes, the ovals represent the text elements in the document and they are called text nodes, and the rounded rectangles contain the attribute-value pairs and they are called attribute nodes. Each rectangle contains the tag of the corresponding element; each text node contains the text from the contents of the corresponding element in the document. Each attribute node is attached to the element that contains it. The root of the tree corresponds to the element containing the whole document. The immediate descendant nodes of a given node N represent the content of the node N. The attribute nodes are considered in a different way and they are connected with the corresponding nodes in a different dimension. Thus the attribute nodes are not descendant nodes of any node in the tree. 
The XPath language uses the tree-based terminology to point some direction within the tree. One of the basic notions of the XPath language is the so-called context node, i.e. a chosen node in the tree. Each expression in XPath language is evaluated with respect to some context node. The nodes in the tree are categorized with respect to the context node as follows: the nodes immediately under the context node are called children of the context node; all nodes under the context node are called descendant nodes of the context node; the node immediately above the context node is called parent of the context node; all nodes that are above the context node are called ancestor nodes of the context nodes; the nodes that have the same parent as the context node are called sibling nodes of the context node; siblings of the context node are divided into two types: preceding siblings and following siblings depending on their order with respect to the context node in the content of their parent - if the sibling is before the context node, then it is a preceding sibling, otherwise it is a following sibling. Attribute nodes are connected to the context node as attribute nodes and they are not children, descendant, parent or ancestor nodes of the context node. The context node with respect to itself is called self node. Some examples: let the "VP" node in the above tree is the context node, then its children are the "V" node and "NP" node that immediately dominates the node "Mary"; its descendant nodes (or simply descendants) are the nodes: "V", "loves", "NP", "N", "Mary"; its parent is the node "S"; at the same time this node is an ancestor of the node "VP"; its sibling is the left "NP" node and more precisely - this is its preceding sibling. If the context node is "V", then its ancestor nodes (or simply ancestors) are the nodes "VP" and "S"; its attribute node is the node 'tense="present"'; its following sibling is the right "NP" node. The result of the evaluation of one XPath expression could be a set of nodes in the tree (node-set), a string, a number, or a Boolean value. 
The most important concept of the XPath language is the location path. Informally, the location path is a description of how to reach some nodes in the tree from the context node. More formally, the location path is a kind of expression. The location path selects a set of nodes relative to the context node. The result of evaluation a location path expression is a node-set containing the nodes selected by the location path. Location paths can recursively contain expressions that are used to filter sets of nodes. 
700
Each location path expression consists of a sequence of location steps. Each location step has the following syntax: 
axis::node-test[predicate] 
where axis determines the part of the XML tree, in which the expression will select some nodes. The possible axes are self, child, parent, descendant, descendant-or-self, ancestor, ancestor-or-self, following-sibling, following, preceding-sibling, preceding, attribute. Each axis selects a set of nodes as it was informally described above. The node-test determines which of the nodes, selected by the axis, are of interest to us. Some of the node-tests are text() for selection of the text nodes only, tag-name for selection of nodes with this name etc. The predicate part [predicate] is optional. It can reduce the number of the selected nodes by rejecting those nodes for which the predicate is not satisfied. Such predicate could be equal to, not equal to, greater than, etc. A location path can start with the slash symbol (/) and, in this case, it is evaluated from the root of the tree and is called absolute location path. Otherwise the location path is relative. 
Additionally to location paths, an XPath expression can be formed by function call. There are several functions provided by the XPath language. They include position(_) - which determines the position of a node in its context, string(_) - which converts the argument to a string, substring(_,_) - which returns a substring of a given string, and many others. 
In the CLaRK System we have augmented XPath language with new predicates, a new axis and variables in order to facilitate the corpus construction and validation. The axis that we introduced is document(). It selects a different document for the evaluation of the next XPath expression. The new predicates include content(RE), which checks whether the content of the node satisfies the regular expression RE. Variables play a role of memory and can store any value, returned by some XPath expression. They can be used within other XPath expressions afterwards. 
2.3 Unicode tokenization 
XML DTD considers the content of each text element as a whole string; the XML Schema introduce a set of datatypes and regular expressions, which can impose constraints over the textual nodes in an XML document. From the point of view of corpora development the textual nodes usually are considered as a list of textual elements, such as wordforms, punctuation and other tokens. In order to provide possibility for imposing constraints over the textual node and to segment them in meaningful way, the CLaRK System supports a user-defined hierarchy of tokenisers. At the very basic level the user can define a tokeniser in terms of a set of token types. In this basic tokeniser each token type is defined by a set of UNICODE symbols. Above this basic level tokenisers, the user can define other tokenisers, for which the token types are defined as regular expressions over the tokens of some other tokeniser, the so called parent tokeniser. For each tokeniser an alphabetical order over the token types is defined. This order is used for operations like comparison between two tokens, sorting and similar. 
2.4 Regular Grammars 
The regular grammars are the basic mechanism for linguistic processing of the content of an XML document within the system. The regular grammar processor applies a set of rules over the content of some elements in the document and incorporates the categories of the rules back in the document as XML mark-up. The content is processed before the application of the grammar rules in the following way: textual nodes are tokenized with respect to some appropriate tokeniser, the element nodes are textualized on the basis of XPath expressions that determine the important information about the element. The recognized word is substituted by a new XML mark-up, which can or can not contain the word. 
3 Constraints 
The constraints that we implemented in the CLaRK System are generally based on the XPath language. We use XPath expressions to determine some data within one or several XML documents and thus we evaluate some predicates over the data. Generally, there are two modes of using a constraint. In the first mode the constraint is used for validity check, similar to the validity check, which is based on 
701
DTD or XML schema. In the second mode, the constraint is used to support the change of the document in order it to satisfy the constraint. In this section we present the constraints that are implemented in the system. 
The constraints in the CLaRK system are defined in the following way: 
(Selector, Condition, Event, Action) 
where the selector defines to which node(s) in the document the constraint is applicable; the condition defines the state of the document when the constraint is applied. The condition is stated as an XPath expression, which is evaluated with respect to each node, selected by the selector. If the result fom the evaluation is approving (a non-empty list of nodes, a non-empty string, the true Boolean value, or a positive number), then the constraint is applied; the event defines when this constraint is checked for application. Such events can be: selection of a menu item, pressing of key shortcut, some editing command as enter a child or a parent and similar; the action defines the way of the actual application of the constraint. There are three types of constraints, implemented in the system: regular expression constraints, number restriction constraints, value restriction constraints. 
3.1 Regular expression constraints 
In this kind of constraints the selection and condition are represented as one XPath expression. It determines the nodes, which the constraint will be applied to. The action, attached to each constraint of this kind, is defined as a regular expression, which is evaluated over the content of the selected elements. If the word, formed by the content of the element can be recognized as belonging to the language of the regular expression, then the constraint is evaluated as true. Otherwise it is evaluated as false and an appropriate message is given. The content of the element is treated as the content of the element when a grammar is applied to it (see above). The user can navigate over the nodes that do not satisfy the constraint or those that satisfy it. 
These constraints can be used for simulation of XML Schema constraints over textual nodes. In addition to checking the content, these constraints - via the XPath expression - can also determine the context of the elements, which they will be applied to. In this way they can be used for imposing regular constraints in addition to these in the DTD making them more specific on the basis of the surrounding context. This is very useful, for example, when someone is compiling a dictionary. Usually the DTD defines some very general content model for the elements, but in a concrete lexical entry a more specific model is realized. For example, let us assume that the <Paradigm> element in a lexical entry is defined as a disjunction of the following kind: 
<!ELEMENT Paradigm ((sg,pl)|(pres,past,particple))> 
The first part of the definition concerns the paradigm of nouns and the second - the paradigm of verbs. The information about the part of speech is presented in another element with tag POS and thus it is not available to the DTD validator. In order to support the consistency, we could write two constraints, which to express the dependency between the contents of the element POS and the element Paradigm: 
Selector: /descendant::Paradigm[preceding-sibling::POS/text()="N"] 
Action: (sg,pl) 
Selector: /descendant::Paradigm[preceding-sibling::POS/text()="V"] 
Action: (pres,past,particple) 
The first constraint is satisfied by each Paradigm element, whose neighbor - POS element preceding it, has textual value N and thus the lexical entry is of a noun and the content of the element Paradigm is a sequence of one sg element and one pl element. Similarly for the second constraint. 
3.2 Number restriction constraints 
Here again the selection is defined as an XPath expression. The action is evaluation of another XPath expression with respect to each node, selected by the first expression. Then the newly selected nodes are counted and checked whether their number is in a given range. The range is given by two numbers MIN and MAX. If the number is in this range, then the constraint is satisfied by the node, selected with 
702
the first expression. The MIN and MAX values can be dynamically determined by two XPath expressions, which return numbers as their values. 
These kinds of constraints can be useful for checking equal number of nodes of different type within a given context. For instance, let us suppose that for a finite set of words of the language anbn it is necessary to construct an XML representation. Then we can define an element <word> in the following way: 
<!ELEMENT word (a*,b*)> 
By regular expressions, however, it is impossible to specify that the as are always the same number as the bs. It becomes very simple with number restriction constraints: 
Selector: /descendant::word 
Min: count(child::a) 
Max: count(child::a) 
Action: Min <= count(child::b) <= Max and similar constraint for the number of as. 
The number constraints are very appropriate for stating general constraints over the whole document. Such constraints include arbitrary complex XPath expressions in their predicate part and otherwise they always select the root note if the predicate is satisfied in the document and require that there is exactly one such element. 3.3 Value restriction constraints 
There are four types of these constraints: parent, all children, some children, some attributes. In validation mode all of them check whether the closest surrounding of the node satisfies some conditions. In information entering support mode they offer to the user a possibility for entering information in order to satisfy the constraint. Let us take a look over them in turn. 
Parent constraints put additional restrictions over the possible parent of a node. In validation mode they are checked together with the constraints that are imposed in the DTD. Otherwise, parent constraints apply when the user inserts a new parent of a node. Then the system tries to calculate which are the possible parents for the node on the basis of the definitions in the DTD. The parent constraints reduce the number of the possible choices. All children constraints check whether the children of a node are among a list of possible elements. The list of possible elements can be relative to the context of the node and in this case they are selected by an XPath expression. If the content of the node is textual, then it is first tokenized and then checked whether each token is in a set of tokens, defined by the constraint. The all children constraints cannot be used in information entering support mode. Some children constraints also impose restrictions over the children of a node, but instead of all children; in this case some of them are necessary to be members of the possible children, determined by the constraint. In the mode of user entering information the constraints show to the user a list of the possible values for a new child of the node and he/she has to choose one. The new value can be incorporated in any position of the content of the node. Again, if the content is textual, then it is first tokenized. Similarly, some attribute constraints impose restrictions on the value of an attribute of the node. The difference is that in this case the values have to be textual. 
The three kinds of constraints, which can be used for changing the content of the document, are exploitable also as rules. They become rules when they determine exactly one value on the basis of the context. In this case the user is not consulted and the value is entered automatically into the document. 
4 Applications of constraints: semi-automatic disambiguation 
In this section we present several uses of the constraints for actual creation and validation of corpora. The examples are based on the uses within the BulTreeBank project (see (Simov, Popova and Osenova 2001), (Simov et al 2002a)). Here we demonstrate how constraints of the kind "some children" are used to support the manual disambiguation of the morpho-syntactic tags of wordforms in the text. For 
703
each wordform we encode the appropriate morpho-syntactic information from the dictionary as two elements: <aa> element, which contains a list of morpho-syntactic tags for the wordform, separated by a semicolon, and <ta> element, which contains the actual morpho-syntactic tag for this use of the wordform. Obviously the value of <ta> element has to be among the values in the list presented in the element <aa> for the same wordform. Here is one example: 
<w><ph>Chovek</ph> <aa>Ncmsi</aa> <ta></ta></w> 
<w><ph>s</ph> <aa>R</aa> <ta></ta></w> 
<w><ph>minalo</ph> <aa>Ansi;Ncnsi;Vpptcao-sni</aa> <ta></ta></w> 
<w><ph>i</ph> <aa>C</aa> <ta></ta></w> 
<w><ph>istinski</ph> <aa>Amsi;A-pi</aa> <ta></ta></w> 
<w><ph>char</ph> <aa>Ncmsi</aa> <ta></ta></w> 
where Ncmsi is a morphosyntactic tag for a common noun, masculine, singular, indefinite; Ncnsi is a morphosyntactic tag for a common noun, neuter, singular, indefinite; Amsi is a tag for an adjective, masculine, singular, indefinite; Ansi is a tag for an adjective, neuter, singular, indefinite; A-pi is a tag for an adjective, plural, indefinite; Vpptcao-sni is a tag for a participle, aorist, neuter, singular, indefinite; C is a tag for a conjunction; R is a tag for a preposition; 
The constraint supporting disambiguation is defined in the following way: 
Type: Some Children 
Selector: ta 
Condition: self::* 
Source: preceding-sibling::aa/text() 
The selector says that the constraint is about ta elements. There is no condition over the context of these elements. The source part determines that the appropriate values for the ta element are among the tokens (under one appropriate tokeniser) in the textual content of the preceding aa element. For the above example, the constraint will automatically fill the four unambiguous words and it will ask the user to choose among the following sets of values for the other two words: { Ansi Ncnsi Vpptcao-sni } and { Amsi A-pi }. The result after the correct choices is: 
<w><ph>Chovek</ph> <aa>Ncmsi</aa> <ta>Ncmsi</ta></w> 
<w><ph>s</ph> <aa>R</aa> <ta>R</ta></w> 
<w><ph>minalo</ph> <aa>Ansi;Ncnsi;Vpptcao-sni</aa> <ta>Ncnsi</ta></w> 
<w><ph>i</ph> <aa>C</aa> <ta>C</ta></w> 
<w><ph>istinski</ph> <aa>Amsi;A-pi</aa> <ta>Amsi</ta></w> 
<w><ph>char</ph> <aa>Ncmsi</aa> <ta>Ncmsi</ta></w> 
As it was already mentioned, one can use the constraints of this kind as rules. For instance, for the same data as above, we could encode a rule that considers the NP internal agreement in order to disambiguate for ambiguous adjectives: 
Type: Some Children 
Selector: ta 
Condition: preceding-sibling::aa[text()="Amsi;A-pi" 
and 
parent::w/following-sibling::w/aa[text()="Ncmsi"]] 
Source: Amsi 
The rule says: if the word in question can be an adjective, masculine, singular, indefinite, or plural, indefinite and the next word is a noun, masculine, singular, indefinite, then choose this word to be an adjective, masculine, singular, indefinite. The source in this case is treated as a string, not as an XPath expression. Using such kind of rules, we have written about 20 rules, which resolve more than 30% of all ambiguities in Bulgarian newspaper texts. All of the presented so far constraints can be used in validation mode. 
704
5 Conclusion 
In the paper we presented several ways of imposing constraints over XML documents which can be used during the implementation of a corpus and during validation of a corpus. They are based on XPath language as a means for checking the presence of some information in the XML files that encode the corpus. In CLaRK System three general kinds of constraints are implemented: regular expression constraints, number restriction constraints, value restriction constraints. They allow two modes of usage: validation mode and information entering mode. The first one just checks if the constraint is satisfied in the document(s) and reports errors otherwise. The second mode is available only for value constraint (except the all children constraints). In this mode the constraints assist the user to change the document in such a way that it to satisfies the constraints. When the context in which the constraints are applied allows only one choice for some value, then the constraints encode rules. 
References 
Abney St 1996 Partial Parsing via Finite-State Cascades. In: Proceedings of the ESSLLI'96 Robust Parsing Workshop. Prague, Czech Republic. 
Corpus Encoding Standard 2001 XCES: Corpus Encoding Standard for XML. Vassar College, New York, USA. http://www.cs.vassar.edu/XCES/ 
DOM 1998 Document Object Model (DOM) Level 1. Specification Version 1.0. W3C Recommendation. http://www.w3.org/TR/1998/REC-DOM-Level-1-19981001 
Simov K, Popova G, Osenova P 2001 HPSG-based syntactic treebank of Bulgarian (BulTreeBank). In: "A Rainbow of Corpora: Corpus Linguistics and the Languages of the World'', edited by Andrew Wilson, Paul Rayson, and Tony McEnery; Lincom-Europa, Munich, pp. 135-142. 
Simov K, Peev Z, Kouylekov M, Simov A, Dimitrov M, Kiryakov A. 2001. CLaRK - an XML-based System for Corpora Development. In: Proc. of the Corpus Linguistics 2001 Conference. Lancaster, England. pp: 558-560. 
Simov K, Osenova P, Slavcheva M, Kolkovska S, Balabanova E, Doikoff D, Ivanova K, Simov A, Kouylekov M. 2002. Building a Linguistically Interpreted Corpus of Bulgarian: the BulTreeBank. In: Proceedings from the LREC conference, Canary Islands, Spain. 
Simov K, Kouylekov M, Simov A 2002 Cascaded Regular Grammars over XML Documents. In: Proc. of the 2nd Workshop on NLP and XML (NLPXML-2002), Taipei, Taiwan. September 1, 2002. 
Text Encoding Initiative 1997 Guidelines for Electronic Text Encoding and Interchange. Sperberg-McQueen C.M., Burnard L (eds). 
UNICODE 2003 Unicode Home Page. http://www.unicode.org/ 
XML 2000 Extensible Markup Language (XML) 1.0 (Second Edition). W3C Recommendation. http://www.w3.org/TR/REC-xml 
XPath 1999 XML Path Lamguage (XPath) version 1.0. W3C Recommendation. http://www.w3.org/TR/xpath 
XSLT 1999 XSL Transformations (XSLT) version 1.0. W3C Recommendation. http://www.w3.org/TR/xslt 
705

Corpus shallow parsing: meeting point between paradigmatic knowledge encoding and syntagmatic pattern matching 
Milena Slavcheva 
Linguistic Modelling Department 
Central Laboratory for Parallel Processing 
Bulgarian Academy of Sciences 
25A, Acad. G. Bonchev St., Sofia, Bulgaria 
E-mail: milena@lml.bas.bg 
1. Introduction 
The production of language corpora augmented with linguistic information takes place in an environment where the researcher is constantly faced by the problem of the interaction between the paradigmatic knowledge representation and the syntagmatic properties of real-life language usage. This problem is especially apparent in the interaction between the first two steps in the development of a syntactic treebank, that is, the result of the morphosyntactic annotation of texts and the shallow parsing aiming at identifying reliable and meaningful segments in the text. Usually, when one comes across the problem of the interdependence between language as a system (i.e., langue) and language as real-life means of communication (i.e., parole), the attention is drawn to entities specific for the text like dates, titles, names, addresses (Karttunen et al. 1997), which are absent in a conventional lexicon used for morphosyntactic analysis. In general, more research has been dedicated to named entities and nominal objects than to the predicate. Similar observation is expressed in Nenadic et al. 2001. Certainly, named entities are those anchors in the text which, on one hand, have to be processed in order to resolve the problem of full text annotation, and, on the other hand, constitute, so to say, the substantive content of the text. The same holds for the noun groups in general. In this paper, however, I concentrate on the predicate of natural language sentences which is, so to say, the event information centre in texts. Since verbs are in the backbone of the lexicon and the grammar, it might seem at first glance, that they smoothly “shuttle” between the paradigmatic and syntagmatic dimensions of language formal description. But this is not the case. My target of exploration are Bulgarian verbs and the two aspects of difficulty that they represent: 1) complex forms expressing tense, mood and voice, which involve auxiliary verbs; 2) verbal units including clitic reflexive and personal pronominals, as well as different types of particles. The analysis and description of those entities are necessary for the appropriate assignment of the syntactic structure of sentences. They serve as the basis for the incremental augmentation of annotated texts with information about the temporal and discourse positioning of events, as well as of the argument structure of predicates. 
The work presented in this paper is a component of the integrated BulTreeBank framework (Simov et al. 2002b) where grammars are built for the segmentation, pattern recognition and category assignment of linguistic entities in XML documents. The software environment is provided by the CLARK system (Simov et al. 2001) which incorporates a number of tools for the creation and manipulation of XML documents, a cascaded regular grammar engine and a system of constraints over XML documents. 
The construction of a grammar for the segmentation, pattern recognition and category assignment of Bulgarian compound verb forms is carried out in an iterative, incremental mode, that is, the source and the target of grammar rule compilation and application are interchanged so that the grammar is refined and its discriminating power is increased. 
The advantages of the approach described in this paper lie in the exploitation of well-established and relatively simple techniques, that is, regular expressions and finite-state automata in a multi-task unified framework for the creation and manipulation of linguistically interpreted data sets. In this way, with reasonable efforts, satisfying results are obtained so that the results of given processes are considered relatively complete for use in certain applications, and, at the same time, there is the constant possibility for linguistically meaningful augmentation of the data sets within an underlying infrastructure. 
The paper is structured as follows. In Section 2 the relevant data categories are overviewed. Section 3 provides a survey of the sources of linguistic knowledge and the computational devices necessary in the initial phase of grammar writing. Section 4 presents the rule sets for automatic recognition and category 
706
assignment of compound verb forms. Section 5 provides some preliminary evaluation of the grammar application. In Section 6 some conclusions are drawn which serve as the basis for the further development of the grammar. Section 7 summarises some points in the further grammar development. 
2. An overview of the data categories 
The first problem that has to be solved in building a grammar for automatic recognition of compound verb forms is to determine the boundaries and the components of the linguistic entities that stand for the patterns to be recognized. The factors that determine the decision-making are the following: 
• language-specific idiosyncrasies; 
• shallow parsing strategies as a module in the text corpus processing; 
• interface between the segments identified as a result of shallow parsing and the deeper linguistic analysis performed at subsequent stages of the creation of the treebank. 
2.1. The tense, mood and voice paradigm of Bulgarian verbs 
Verbs in Bulgarian have a very rich tense, mood and voice paradigm consisting of simplex (synthetic) inflected forms and complex (analytic) forms. The latter are generally composed of a non-finite form of the full-content verb and one or more (in some cases omitted) auxiliaries. The reader can get an impression of the number and variety of forms if I mention that, traditionally, Bulgarian is considered to have nine tenses and four moods, including the “exotic” narrated (not-witness) mood. Passive voice is represented by the form consisting of auxiliary verbs and the passive participle of full-content verbs. (The forms with the reflexive pronominal se are not included in the passive voice paradigm in the current model). In fact, no matter how modern theoretical linguists might determine the different number and types of tenses, moods and voices in Bulgarian, the forms are there to challenge the computational linguist in grammar writing. 
Examples 1, 2 and 3 bellow illustrate some complex verb forms: 
1) Toj shte e napisal pismoto. 
He will-aux,pres,3p,sg be-aux,pres,3p,sg write-active_aorist_particip,m,sg letter-the 
‘He will have written the letter.’ 
2) Toj bil napisal pismoto. 
He be-past_participle,m,sg write-active_aorist_participle,m,sg letter-the 
‘They say, he has written the letter.’ 
3) Knigite sa bili namereni pod masata. 
Books-the be-aux,pres,3p,pl be-past_participle,pl find-passive_particip,pl under table-the 
‘The books have been found under the table.’ 
The difficulties in processing the complex tense, mood and voice forms stem from the fact that they incorporate features of both morphology and syntax. The morphological features are manifested in the grammatical meaning carried by the entire unit consisting of auxiliaries and a full-content verb. The syntactic features lie in the multi-word structure of the grammatical unit: the word order of the separate tokens in the construct can be permutated and different “external” syntactic elements can be inserted in-between the ingredients of the complex verb form. 
2.2. The relation between verbs and small words 
In Bulgarian, short pronominal elements and particles (which will be called small words for convenience) surrounding the verbs turn out to be a specific problem to encoding linguistic information in the lexicon, to sentence segmentation at the shallow parsing level, as well as to the phrase structure descriptions at the level of deeper linguistic analysis. In the segmentation model presented in this paper, the small words 
707
accompanying the verbs belong to the verb form patterns. The small words are the following: the negative particle ne, the interrogative particle li, the preverbal element da, the accusative reflexive pronominal element se, the dative reflexive pronominal element si, the short accusative and dative personal pronouns. 
The arguments for the inclusion of small words into compound verb forms can be outlined as follows. Both in the paradigmatic and syntagmatic dimension, the negative and interrogative particles, the preverbal element da, and the reflexive pronominals produce positional and meaningful clusters with the verb. The same holds for the combination between some impersonal verbs and accusative or dative pronominals expressing the number and person of the experiencer, thus producing a morphological entity with the verb. The arguments for the inclusion of short accusative and dative pronominals into compound forms with personal verbs are sought at the interface of shallow processing and deeper linguistic analysis. Object doubling is a phenomenon typical for Bulgarian. This is the case when the direct or indirect object is expressed twice: once by the short accusative or dative pronominal, and once by a full-fledged nominal phrase, which can be a full form of a pronoun, or a noun phrase. Oversimplifying the analysis, it is claimed in this paper that the short pronominals and the full-fledged complements are attached at different levels: the short pronominals at the lexical level, and the full-fledged complements at the syntactic level. In Avgustinova 1997, a similar approach to the segmentation of the Bulgarian verb complex is described. Similar analysis (although at a deeper linguistic level) of pronominal clitic attachment on a presyntactic level is observed for Italian (Monachesi 1998) and for French (Miller and Sag 1997). Personal pronominal clitics are part of the verb chunk in the shallow parsing grammar of French described in Clement and Kinyon 2000. 
3. Sources of linguistic knowledge and facilities for grammar writing 
Let us see what we have at the moment of being assigned the task of constructing a grammar for parsing compound verb forms. The team of the BulTreeBank project provides a corpus of one million word tokens which has the following special-purpose characteristics. The texts are from newspapers and are organised in a set of XML documents supplied with TEI conformant markup of the text structure at the paragraph level. The texts are processed by a morphological analyser and are manually disambiguated using the facilities of the system of constraints incorporated in ClaRK (Simov et al. 2002a). The electronic lexicon (Popov et al. 1998) used for the morphosyntactic analysis contains lexical entries for single words only. Consequently, the information about the verb tense, mood and voice is reduced to those portions, which are contained in single verb forms. In such a way three verb tenses (present, aorist and imperfect) are encoded in the tags attached to full-content verbs. Mood is represented by the imperative forms of full-content verbs and some special conditional forms of the auxiliary verb sam (‘to be'). Voice is present as a category in part of the paradigm of personal transitive verbs, that is, the passive participle ending in –n(a,o,i). There is no information about the attachment of short personal and reflexive pronominals, neither negative nor interrogative forms. 
The purpose of the grammar is the automatic identification of simplex or complex verb forms, which constitute the predicate of the sentence. Throughout this paper I will use the term compound verb form to refer to those verb clusters which consist of more than one orthographic word, that is, the compound verb forms are combinations of a full-content verb and a different number and type of auxiliary verbs, short pronominals or particles in varying word order. 
The grammar is determined by the ClaRK system environment (Simov et al. 2002a): it is of type regular, accepted by finite-state automata, and the rules are applied in a cascade where the output of a certain set of rules is the input to another set of rules (except for the initial rules). The rules of the grammar are of the type 
C -> R, 
where R is a regular expression and C is a category of the pattern matched by R. 
The process of grammar construction starts with abstracting over grammar books, which, as it is well-known, are written for humans and do not partition or combine the linguistic knowledge in a way that can directly serve practical applications. Grammar books and paper dictionaries do not contain the suitable exhaustive sets of data structures used in real-life software applications. Nevertheless, using grammar books and one's competence of language, a grammar is constructed in a deductive mode of inference. The result of the application of this grammar can be defined as a first approximation to resolving the problem of parsing. The grammar writer has to define principles of grammar construction to lead him in the “oceans of 
708
language” (Sampson 2003), and in this particular case in the ocean of Bulgarian verb forms. The principles of the initial phase of grammar construction can be defined as follows. 
1. Exhaustiveness. The grammar writer thinks of all constructs that can count as forms of tense, mood and voice: simplex forms, various occurrences of combined finite and non-finite auxiliary and main verb forms. Naturally, the positive forms are considered first, but then it is necessary to enumerate the forms of predicate negation and interrogation expressed by the addition of the negative particle ne and the interrogative particle li respectively. The variation of constructs grows rather big when the short pronominals are added into the set of ingredients of compound forms. 
2. Definition of key morphosyntactic features and combinations of features. The morphosyntactic tags attached to the verb tokens in the text are combinations of values of a whole bundle of morphosyntactic features differentiating the members of the verb paradigms. In the input words to the regular expressions, it is necessary to specify those values of features that are relevant and necessary for the program to perform the correct pattern matching. For instance, the lexical features Aspect and Transitivity are irrelevant, and they are denoted by wildcards in the input words to the regular expressions. At the same time, it is necessary to mention explicitly the indefinite forms of the past active and of the passive participles since they can be components of tense, mood or voice forms. Thus the definite forms which can be parts of noun or adjective phrases are discarded as possible ingredients of verb clusters. (The morphological expression of definiteness in Bulgarian is a definite article in the form of an ending to word forms.) The full description of the morphosyntactic specifications for verbs as a subset of the BulTreeBank tagset (Simov et al. 2002c) is given in Slavcheva 2003. 
3. Continuity. It is a linguistic fact that the continuity of compound verb forms can be interrupted at certain points and different elements, mainly adverbials and nominals can be inserted in them. But in the initial phase of grammar development, compound verb forms are thought of as prevailingly continuous entities, rather than discontinuous ones. This is due to the mainly paradigmatic perspective of thinking at the initial phase of modeling the verb complex. The alternative of foreseeing the eventual occurrence of “external” elements in-between the ingredients of the compound verb forms using regular expressions in an unsupervised manner does not help much (Nenadic 2001; Zackova and Pala 1999). It is necessary to work out carefully the set of rules that envisage the discontinuity of compound verb forms. That is possible after a closer and more extensive examination of syntagmatic patterns (Zackova and Pala 1999; Nenadic 2001). 
4. Longest versus shortest match principle. The cascaded regular grammar engine of the ClaRK system provides the possibility to identify segments of different length in a sequence suitable for defining the structure and supplying the annotation of compound verb forms. A strategy is defined for choosing the most suitable length and composition of the examined constructs at intermediate levels of parsing. 
4. Rules and category assignment at the initial phase of grammar development 
The methodology described in section 3 is illustrated by some rules for syntagmatic pattern matching. 
The first instance is a rule that recognises segments consisting of a full-content verb and a certain number and type of small words. 
<MV>\w</MV> --> 
<"da","#">?, <"ne","#">?, <"#","Pp@d@@@t">?, <"li","#">?, 
<"#","Pp@a@@@t">?, <"li","#">?, <"#",("Ppxa@@@t"|"Ppxd@@@t")>?, 
<"#",("Vp@@@f#"|"Vp@@@z#"| "Vn@@@f#"|"Vp@@@cao@@@i"| 
"Vp@@@cam@@@i"|"Vp@@@cv@@@@i"| "Vn@@@cao@@@i"| "Vn@@@cam@@@i")>, 
<"li","#">?,<"#","Pp@d@@@t">?, <"#","Pp@a@@@t">?, 
<"#",("Ppxa@@@t"|"Ppxd@@@t")>? 
709
The right hand side of the rule is a regular expression stating that a full-content verb cluster can consist of: 
1) a finite verb form inflected for the present, aorist or imperfect tense; 
2) a non-finite verb form which can be an indefinite form of the active aorist, active imperfect, or the passive participle; 
3) one of the verb forms enumerated in items 1 and 2 preceded or followed by a variety of sequences of small words. 
The left hand side of the rule contains the XML markup that is added to the output of the rule, that is, the tags that enclose the recognized pattern denoted by the variable \w. Since the input words to the regular expression of the rule are the contents of XML elements nested into XML local trees, an Element 
Value in the form of an XPath expression (Xpath 1999) is defined for the element serving as the context node of the tree structure that is computed by the grammar. In the case of the MV rule described here, the element (marked with <w>) is the context node for which the following XPath expressions are written: 
ph/text() ta/text() 
The XPath expressions state that the local tree structure should be searched for the PCDATA strings of the ph element (whose content is the running word string itself), and for the PCDATA strings of the ta element (whose content is the correct morphosyntactic tag) in order to match the input words to the regular expression. In the rule itself, the input words to the regular expressions are ordered pairs of ph content and ta content. The irrelevant content for a given input word is designated by the # symbol, for instance, <"da","#">, or <"#","Pp@d@@@t">. The grammar also indicates the elements in the XML document to which the grammar is applied. In the current example the MV rule is applied to the paragraph (p), head (head) and hilighted (hi) elements. This is indicated by the expression 
//p|//head|//hi 
At present the regular grammar cascade for recognition of compound verb forms consists of two levels. The set of rules that constitute Recognizer 1 and operate on the first level includes rules that produce: 
• verb forms consisting of a single main verb or a main verb accompanied by small words; 
• verb forms consisting of a single auxiliary verb or auxiliary verb accompanied by small words; 
• groups of small words, enclosed in a separate chunk, which remain unattached to a verb due to special cases of linear order and discontinuity. 
The second-level set of rules (Recognizer 2) outputs segments corresponding to full tense, mood and voice forms and including small words. For instance, the following rules belong to Recognizer 2: 
<VC>\w</VC> -> <CT>?, <XV>+, <MV> 
<VC>\w</VC> -> <MV>, <XV>+ 
The input words to the regular expressions on the right hand side of the rules are tags denoting XML elements that are added at Level 1 of pattern recognition. MV stands for a main verb chunk, XV for an auxiliary verb chunk, and CT for an independent chunk of small words. The regular expressions encode the variants of verb complex composition. It should be noted that the rules overgenerate verb complex constructs since they lack a detailed description of the possible verb complex constructs in relation with subordinate clauses. The left hand side of the rules state that the category of the patterns recognized by the rules are of type verb complex and the markup that delimits it in the XML document consists of a tag named VC. 
710
5. Some preliminary evaluation of the grammar application 
The application of Recognizer 2 provides the possibility to delimit correctly and to mark up the majority of the longest compound verb form patterns. The experiments with the newspaper corpus show that the chunks identified by Recognizer 1 tend to be adjacent within the longest compound verb form patterns in communicatively unmarked written prose. A paradigmatic representation of word order within the verb complex in view of the "communicative organization of Bulgarian sentences" provided in Avgustinova 1997 supports the conclusion expressed in this paper. 
Here are some numbers that illustrate the application of the regular grammars described in this paper. In a text of 4292 running words, there are: 
• 536 occurrences of main verbs with or without small words recognized by the set of rules of Recognizer 1; 
• 164 occurrences of auxiliary verbs with or without small words recognized by the set of rules of Recognizer 1; 
• 5 occurrences of small word groups recognized by Recognizer 1 as a separate chunk due to specific linear order in respect to the main verb. 
The application of Recognizer 2 resulted in the recognition of 77 compound verb forms per se which are longest matches. It should be noted that in the current grammar the combination between the copula and the primary predicatives is not identified as a pattern to be recognized by the grammar. They remain separate entities. The 77 occurrences of verb complexes were manually evaluated and the result was that in one case the verb complex was erroneously identified as such: the verbal part of a subordinate clause was incorrectly combined into a verb complex with a preceding copula. In the cases of discontinuous compound verb forms the grammar simply fails. 
The results of the grammar application to a number of text documents shows that the ratio of occurrences among the different types of segments is preserved. 
6. Some conclusions 
The first phase of the grammar development for parsing compound verb forms provides valuable input to the subsequent phases. Several conclusions are drawn which serve as the basis for the further development of the grammar. 
1. The complex tense forms consisting of auxiliaries and a full-content verb are not of such a great number and variety in real-life texts and that is why the grammar has good performance. At the same time, it is necessary to augment the paradigmatic knowledge by learning from the rare occurences of syntagmatic patterns, since this is the only way to build a wide-coverage grammar of compound verb forms. 
2. Discontinuous compound verb forms with adverbial and nominal inserts are not recognised by the grammar. They can be searched for due to the application of rules that identify shorter segments within the verb complex: auxiliary verb chunks and main verb chunks. In such a way the syntagmatic realisation compound verb forms serves as the source for paradigmatic knowledge necessary for the grammar rule writing. The direction from the syntagmatic to the paradigmatic dimension in grammar construction is also followed in the lemmatisation of compound verb forms in Serbo-Croatian in a method using local grammars described in Nenadic 2001. It is also obvious that discontinuity has to be explored in relation to the category of the main verb form. For instance, the passive participles are separated by adverbials from the accompanying auxiliary verbs in a greater number of cases compared to past active participles. 
3. In the process of building the treebank, a core set of sentences, excerpted from Bulgarian grammar books, are manually assigned syntactic structure. This is a resource for the account of possible compound verb forms which include or not small words, and whose components are adjacent or 
711
discontinuous. In such a way the possible inserts in compound verb forms can be classified and the grammar can be augmented with rules for the recognition of discontinuous compound verb forms as well (Zackova and Pala 1999). 
7. Further development 
The processes of grammar development within an integrated XML framework described in this paper provide a flexible method for augmenting data sets with linguistic information that is required as necessary for a certain application or research. The grammar developer can decide upon the degree of specification or underspecification of categories, the size and type of linguistic entities that should be analyzed. The decision making at the level of shallow parsing is performed in view of possible interfacing with the level of sophisticated linguistic analysis which is a subsequent step in building the treebank of Bulgarian. Having defined the underlying principles of grammar construction for the recognition of compound verb forms, it is necessary to perform more intensive testing of the application of grammar rules and to provide a detailed account of the frequency of occurrence of the different types of constructs. The exploration has to be performed in interrelation with the grammars developed by the team members for the other linguistic entities like noun groups, adjective and adverb phrases, named entities, etc. This is the way to extract the idiosyncratic cases where the grammar fails and to refine the grammar so that the deviating structures are possibly processed automatically as well. A significant further development is the interfacing between the shallow parsing level and the sophisticated linguistic analysis so that the process of treebank construction is facilitated. 
Acknowledgement 
The work presented in this paper is carried out within the BulTreebank project, funded by the Volkswagen Foundation, Federal Republic of Germany, under the Programme "Cooperation with Natural and Engineering Scientists in Central and Eastern Europe", contract I/76887. 
References 
Avgustinova T 1997 Word order and clitics in Bulgarian. Saarbruecken Dissertations, University of the Saarland. 
Clement L, Kinyon A 2000 Chunking, marking and searching a morphosyntactically annotated corpus for French." In Proceedings of ACIDCA'2000, Monastir. 
Karttunen L, Chanod J-P, Grefenstette G, Schiller A 1997 Regular expressions for language engineering. In Natural Language Engineering, Cambridge University Press, pp.1-24. 
Miller P, Sag I 1997 French clitic movement without clitics or movement." Natural Language and Linguistic Theory:15, pp.573-639. 
Monachesi P 1998 Decomposing Italian clitics. In Malari, S and Dini, L (eds) Romance in HPSG. CSLI Publications, Stanford, USA, pp.305-357. 
Nenadic G, Vitas D, Krstev C 2001 Local grammars and compound verb lemmatization in Serbo-Croatian. In Zybatow G, Junghanns U, Mehlhorn L, Szucsich L (eds) Current issues in formal Slavic linguistics. Peter Lang: Europaischer Verlag der Wissenschaften, pp 469-477. 
Popov D, Simov K, Vidinska S. 1998 Dictionary of writing, pronunciation and punctuation of Bulgarian, Atlantis LK, Sofia. 
Sampson G 2003 Thoughts on two decades of drawing trees. In Abeille A (ed), Treebanks. Building and using parsed corpora. Kluwer Academic Publishers, Boston/Dordrecht/London, pp 23-41. 
712
Simov K, Peev Z, Kouylekov M, Simov A, Dimitrov M, Kiryakov A 2001 CLaRK - an XML-based system for corpora development. In Proceedings of the Corpus Linguistics 2001 Conference, pp.558-560. 
Simov K, Kouylekov M, Simov A 2002a Cascaded regular grammars over XML Documents. In Proceedings of the Second Workshop on NLP and XML (NLPXML-2002), Taipei, Taiwan. 
Simov K, Osenova P, Slavcheva M, Kolhovska S, Balabanova E, Doikov D, Ivanova K. Simov, A. Kouylekov M 2002b Building a linguistically Interpreted Corpus of Bulgarian: the BulTreeBank. In Proceedings of LREC 2002, Canary Islands, Spain, pp.1729-1736. 
Simov K, Slavcheva M, Osenova P 2002c BulTreeBank morphosyntactic tagset. BulTreeBank Report, Sofia, Bulgaria. 
Slavcheva M 2003 Some aspects of the morphological processing of Bulgarian. In Proceedings of EACL Workshop on morphological processing of Slavic languages, Budapest (in press). 
XPath 1999. XML Path Language (XPath), Version 1.0. W3C Recommendation. http://www.w3.org/TR/xslt 
Zackova E, Pala K 1999 Corpus-based rules for Czech verb discontinuous constituents. In Proceedings of TSD'99 Workshop, LNAI 1692, Springer-Verlag, pp.325-328. 
713

A quirky progressive? A corpus-based exploration of the will + be + -ing construction in 
recent and present day British English. 1 
Nicholas Smith, Lancaster University 
1 Introduction 
1.1 Outline 
This paper investigates the properties and text distribution of English modal auxiliaries, especially will, in construction with be + -ing (as in I'll be leaving tomorrow, You should be going) in recent and present day English.2 Constructions of this kind are said to have some unusual aspectual and other semantic properties, quite against the grain of “regular” uses of the progressive. I consider the apparent eccentricity of the will+ be + -ing construction especially. I investigate its use in corpora of late twentieth British English in an attempt to discover more about the nature of the construction, including the respects in which recent, and possibly ongoing, changes in use have been occurring. 
1.2 A quirky progressive? 
The chief reason that the combination of a modal auxiliary with the be + -ing form of a verb might be deemed “quirky” is that two opposite aspectual values appear to be permissible within one and the same construction. In the case of will be + -ing, for example, many linguists identify a “regular” use, in the sense of an event or situation located in the future and viewed as unfolding – in progress – at the time of reference, cf. (1) and (2); and more strikingly, a use in which seemingly the event or situation is construed as an indivisible whole, with the notion of progressivity playing no part, cf. (3) and (4). A similar contrast seems to arise with other modal auxiliaries + be + -ing, cf. (5) and (6) against (7) and (8), although here the difference also involves temporal reference. 
(1) When you reach the end of the bridge, I'll be waiting there to show you the way. (Quirk et al. 1985 : 216) 
(2) Don't phone me at 7 o'clock – I'll be watching my favourite TV programme. (Leech 1987: 67) 
(3) The train will be arriving at 8 o'clock. (Leech 1987: 51) 
(4) Not in June! I'll be having my baby in June! (Adamczewski 1982 : 175) 
(5) He can't be working at this hour! (Leech 1987: 98) 
(6) She may be bluffing. (Leech 1987: 98) 
(7) She can't be coming on Monday. (Palmer 1979: 54) 
(8) Jill says she might be calling this afternoon. (Quirk et al. 1985 : 236) 
The nature of and speaker's motivation for the second use of will + be + -ing are far from clear-cut. For some commentators it provides compelling evidence that “progressive” is a misnomer for the be + -ing construction (cf. Adamczewksi 1982, Celle 2001, for example). Many see it as deriving from a need to refer to the future unclouded by attitudinal overtones such as volition, intention or promise - overtones that are liable to figure in the use of will + bare infinitive and be going to, for example. Indeed this factor has even been claimed to provide a continuing source of growth in the use of will + be + -ing in the present day (cf. Samuels 1972, Leech 1987, Close 1988). I would like to explore the viability of this claim with respect to registers of written English, and also assess whether the construction can truly be described as “progressive in form but non-progressive in meaning” (Wekker 1975: 116). 
1.3 Historical growth in frequency 
Historians of English tend to deal primarily with those modal + be + -ing combinations that have overtly future time reference, i.e. will + be + -ing, shall + be + -ing. According to Mossé (1938), these constructions are attested as early as Old English, but usage remains very rare right through Middle English, picking up again in the Modern era. It is claimed that before each construction became widespread, it was more frequent in texts from the north of England and Scotland (Mossé 1938, Mustanoja 1960, Strang 1970). Thus it is quite probable that the modal + be + -ing has its origins in Celtic varieties of English, or contact with Celtic languages (cf. Visser 1973: 2023 and below, 2.2.). 
1 I would like to thank Geoffrey Leech, Anna Siewierska, Amy Wang and Christopher Williams for their very helpful discussion and comments at various points in the preparation of this paper; and Agnes Celle for sending me a copy of her paper on will + be + -ing. 
2 I have tried to a avoid the term “modal progressive” as it has already been used by Wright (1994) to denote something else: affective or experiential (rather than aspectual) uses of be + -ing, rather than the combination of be + -ing with a modal auxiliary. 
714
In examining nineteenth century British English, Smitterberg (2002: 193-198) provides the first quantitative evidence that I can find. Across a range of genres sampled at three time intervals, he reports that will + be + -ing actually declines significantly in use, while all other modal + be + -ing constructions either decline or stagnate. Smitterberg's findings run counter to what we might expect, given that in the course of Modern English other parts of the progressive paradigm (permutations of tense, aspect and voice, such as the perfect progressive, passive progressive) have emerged or been realised with increasing frequency (cf., Rissanen 1998, Görlach 1999, Strang 1982). It seems necessary, therefore, to explain the gap between the depressed usage in the nineteenth century and the current healthy situation in the present day. My analysis attempts to shed some light on latter-day developments. First, I give an overview of recent changes in frequency distribution, then try to determine if stylistic or semantic factors, or both, are involved. 
2 Corpus data and frequency analysis 
2.1 The corpora 
The data used in this empirical investigation are the LOB corpus of British English written texts published in 1961, and its counterpart, FLOB, containing texts published in 1991–92. Each corpus contains one million words spread across fifteen text categories, nine of non–fiction, six of fiction. As they were originally intended for publication, the texts tend to be more formal in character and more carefully edited than private forms of writing, such as personal letters. (Further details of all these corpora are provided in Mair 1997). The categories and proportions of words within them are designed to match as closely as possible across the two corpora. The great benefit of this design is that it facilitates empirical investigation (both quantitative and qualitative) of very recent changes in the development of written English. 
I briefly look at four further corpora: two corpora of American English, Brown and Frown, that correspond to LOB and FLOB in terms of dates and other sampling criteria; and two supplementary corpora of British English: the ICE-GB and BNC (spoken part), in which the majority of texts are contemporaneous with FLOB (early 1990s). 
2.2 Frequencies of modal + be + -ing constructions 
Changes in frequency are expressed firstly in percentages, i.e. the difference between the two frequencies as a percentage of the first, and secondly by a probability value if this is calculated to be statistically significant.3 
Table 1: Frequencies of modal + be + -ing constructions in recent British and American English4 
British English American English LOB (1961) FLOB (1991) Change 
% Probab-ility Brown (1961) Frown (1991) Change 
% Probab-ility can 0 4 + % 4 1 -75.0% could 4 6 +49.6% 4 7 +75.0% may 10 15 +49.6% 9 8 -11.1% might 12 12 -0.3% 14 18 +28.6% must 10 7 -30.2% 16 1 -93.8% p < 0.05 shall 7 5 -28.8% 1 1 - should 9 18 +99.5% 11 9 -18.2% will 66 87 +33.0% 40 46 +15.0% would 32 34 +6.0% 36 41 +13.9% Total 150 190 +25.0% p < .05 135 132 -2.2% 
Collectively, modal + be + -ing constructions in British English have increased in use since the 1960s, with the most significant increase in will + be + -ing. This general buoyancy is in sharp contrast to a spectacular decline of almost all the modal auxiliaries in non-progressive environments (Table 2). Even though the ratio of progressives to non-progressives is rising, the former are still relatively few. 
3 The probability was obtained using the log likelihood test of significance. 
4 Need + be + -ing and ought (to) + be + -ing do not occur in these corpora. 
715
Table 2: Frequencies of modal auxiliaries + simple infinitive in British and American English 
British English American English LOB (1961) FLOB (1991) Change 
% Probab-ility Brown (1961) Frown (1991) Change 
% Probab-ility can 1997 2037 +1.7% 2189 2159 -1.3% could 1736 1776 +2.0% 1772 1648 -6.9% p < .05 may 1323 1086 -18.1% p < .01 1289 870 -32.4% p < .01 might 765 648 -15.5% p < .01 649 634 -2.2% must 1136 807 -29.2% p < .01 1004 650 -35.2% p < .01 need 78 44 -43.7% p < .01 40 35 -12.4% ought (to) 104 58 -44.4% p < .01 70 49 -29.9% shall 348 195 -44.1% p < .01 266 149 -43.9% p < .01 should 1292 1129 -12.9% p < .01 899 778 -13.4% p < .01 will 2756 2631 -4.8% 2662 2356 -11.4% p < .01 would 2996 2660 -11.5% p < .01 3017 2827 -6.2% p < .05 Total 14531 13071 -10.3% p < .01 13857 12155 -12.2% p < .01 
Meanwhile in American English there has been barely any change in the overall use of modal + be + -ing, while modal auxiliaries as a whole appear to be going out of fashion even faster than in British English. (For details, see Leech, forthcoming). The other main difference from British English is the significantly lower rate of use of will + be + -ing in American. While at first sight this might appear an unusual case of Britain leading the way in a grammatical change, evidence from other dialects of English suggests that this is not the case. Contrastive data provided by Filppula (2002), for example, point strongly to a much higher rate of usage of all variants of the progressive in the Celtic varieties of English (e.g. Irish English, Welsh English and Hebridean English), suggesting that these dialects are a more likely source of innovation, and standard British English may be influenced by contact with them. Since Filppula's data is mostly spoken and sampled in a very different way to the Brown family of corpora, his results are not directly comparable with my own. No doubt future construction of more closely matching dialectal corpora, ideally sampled at chronological intervals, could help to shed more light on underlying Celtic influence. 
Because the frequency of most individual modal auxiliaries combining with the progressive in these corpora is still too low to gain a more detailed picture, from this point on I will concentrate on will + be + -ing. 
2.3 Genre / medium breakdown of will + be + -ing 
Table 3: genre comparison of will + be + -ing in written British English 
LOB (1961) FLOB (1991) Change in frequency raw per 
frequency million raw 
frequency per 
million % probability Press 24 135 32 180 32.7% General prose 12 29 32 77 166.7% p < .01 Learned 4 25 5 31 24.3% Fiction 26 102 19 74 -27.2% Total 66 66 88 87 33.0% 
The increase in frequency occurs in non-fictional genres, most signficantly in the general prose category. Probably the lack of increase in fiction is because in fictional narrative it is unusual for the narrator to have call to refer to future plans, arrangements or expectations, while in fictional dialogue, in the 1960s a good approximation of everyday speech may already been achieved. 
An interesting question is whether the rise in press and general prose genres in FLOB is the result of conversational or informal influence on written language (a change “from below”). It is claimed by Close (1988: 53) and Leech (1987: 69), for example, that will + be + -ing, particularly in its so-called non-progressive use, is especially common in conversation. A direct sign of such influence would be a higher use of contracted forms ('ll/won't) in the FLOB examples. Yet most of the additional cases are not contracted but rather take the full form will. Moreover it is noteworthy that the frequency of will + be + -ing is not especially high in two conversational spoken corpora sampled around the same time as FLOB (early 1990s). In the BNC conversational part the frequency is 139 per million words, 
716
while in direct conversations in ICE-GB it is around 92 per million, and the entire spoken part of ICE-GB records less than 110 per million. These frequencies are all lower than the press reportage and editorials section of FLOB.5 
The most striking finding in the written section of ICE-GB is the vastly higher rate of use in letter-writing: 34 occurrences (more than 500 per million words), spread almost equally between social and business correspondence. One of the important functions of both types of correspondence is to describe or make arrangements for the future. In these examples from business letters, using the progressive seems to add tact or politeness, not informality. 
(9) I regret to now inform you that we will be terminating all our contracts with you as of Monday 22nd of July 1991. (ICE-GB W1B-028 #46:4) 
(10) Following our telephone conversation yesterday, I am writing to confirm that the theatre will be undergoing a reorganisation of staffing following the departure of the Deputy Manager William Brown. (ICE-GB W1B-018 #139:18) 
2.4 A brief comparison with other constructions referring to future time 
Table 4: Constructions referring to the future: comparative frequencies in British English (LOB and FLOB) 
LOB (1961) FLOB (1991) Change in frequency raw per raw 
frequency million frequency per 
million % probability will + infin. 
Press 
General Prose 
Learned/academic 
Fiction 
Total 617 
990 
293 
856 
2756 3481 
2395 
1845 
3344 
2741 680 
1013 
301 
637 
2631 3817 
2452 
1885 
2478 
2610 +9.7% 
+2.3% 
+2.2% 
-25.9% 
-4.8% p < .01 shall + infin. 
Press 
General prose 
Learned/academic 
Fiction 
Total 27 
169 
60 
99 
355 152 
409 
378 
387 
353 20 
85 
40 
55 
200 112 
206 
251 
214 
198 -26.3% 
-49.7% 
-33.7% 
-44.7% 
-43.8% p < .01 
p < .05 
p < .01 
p < .01 be going to (Present) 
Press 
General prose 
Learned/academic 
Fiction 
Total 24 
46 
5 
99 
174 135 
111 
31 
387 
173 42 
23 
5 
93 
163 236 
56 
31 
362 
162 +74.1% 
-50.0% 
-0.6% 
-6.5% 
-6.6% p < .05 
p < .05 be to: Present tense 
Press 
General Prose 
Learned/academic 
Fiction 
Total 85 
107 
42 
18 
252 480 
259 
265 
70 
251 45 
93 
35 
14 
187 253 
225 
219 
54 
186 -47.3% 
-13.1% 
-17.1% 
-22.6% 
-26.0% p < .01 
p < .01 
This very rudimentary survey shows some intriguing tendencies. Firstly, the rise of will + be + -ing reported earlier may not be so closely connected to a usurping of will + simple infinitive as one might assume. The decline of will + simple infinitive, rather than being part of a summary decline of the core modal auxiliaries, seems to be limited to the fiction section. In fact, if one breaks the corpora down into smaller subcategories of genre, one finds that two of the fiction sections (adventure fiction and romance fiction) account for most of the downturn, while other genre sub-categories fluctuate wildly (e.g. frequency in press reportage jumps up, but in press reviews it tumbles). 
On the other hand at least three other rival constructions do not appear to fare as well as will + be + -ing. There is a marked decline in shall and be to, while be going to rises only in the press sections. This last is puzzling since over the 
5 Clearly it will be necessary in the future to examine the frequency of a host of other indicators of conversational style across the two corpora, to get a better idea of the extent to which speech habits are increasingly mimicked in writing intended for publication. 
717
last several centuries be going to (including its variant gonna) has grown considerably (see e.g. Krug 2000), and in the matching American corpora (Brown and Frown) and corpora of spoken British English this trend has continued apace (see Leech forthcoming). 
Two important constructions not tabulated here are the present progressive and simple present with future time reference. In both cases it has proven very difficult to identify them positively as future-referring because of their strong basis in present time (plans or arrangements in existence at speech time). Regarding present progressive, if I limit examples to those where the culmination phase of the event does not appear to be “under way” at speech time, I retrieved around 45-55 occurrences in LOB and 30-40 in FLOB.6 Thus in the sample of written British English most of the major constructions that compete with will + be + -ing to refer to future time based on a present situation have either declined in use or shown growth in a few registers only. 
3 Semantic factors 
3.1 Semantic character and the rise of will + be + -ing 
Here I am especially interested in the motivation for the “non-progressive” use of will + be + -ing: What are its semantic characteristics? Should it really be considered eccentric? Why is it emerging in present day usage as an important means of marking future events? For reasons of space, my discussion is limited to the theories that I find the most persuasive. 
3.1.1 “Colourless future” 
By far the most popular explanation for its emergence is a communicative need for speakers to refer to the future “colourlessly”, i.e. without implying volition, intention or promise. Samuels (1972) offers a brief historical account of how this state of affairs might have arisen. He argues that absence of volition was already an implicit element of meaning in the regular use of will (and shall) + be + -ing, i.e. for future situations viewed in progress, as in (1) above and (11) below. This non-volitional element was later extended to situations not viewed as being in progress (cf. [12]), because the established way of referring to such events – will + the simple infinitive (cf. [12]) – was liable to be interpreted as marked by a personal attitude.7 
(11) I'll be leaving when you arrive. 
(12) I'll be leaving tomorrow. 
(13) I'll leave tomorrow. 
Samuels (1972: 57) adds: “It is therefore becoming more and more used as a colourless future without overtones of intention, wish, irritation and the like, and irrespective of whether the context demands a marked aspectual form”. It is difficult to confirm or reject this theory as no chronology or empirical data is provided,8 and no connection with other modal + be + -ing constructions is made. However, the transfer of colourlessness from one type of progressive to another is interesting in suggesting that the two types are not discrete. 
3.1.2 “Already decided future”; “future as matter-of-course” 
Several commentators on contemporary usage have pointed out that more than mere colourlessness is implied by 
will + be + -ing – and moreover by other modal + be + -ing constructions. A key additional ingredient according to Huddleston (2002: 172), for example, is the idea that the speaker feels that the future has already been settled in some way; he calls this “already decided future”. Thus in interrogative sentences like (14) below, a prior arrangement or plan is assumed, whereas will + infinitive (cf. [15]) is more likely to be taken as a direct request to the addressee, i.e. a decision has to be made on the spot whether to go to the shops. 
(14) Will you be going to the shops this afternoon? (Huddleston 2002: 171) 
(15) Will you go to the shops this afternoon? (Huddleston 2002: 171) 
6 I have not yet attempted to count cases of the simple present referring to the future. 
7 I have modified Samuels’ own example (I shall/will be leaving) by adding two kinds of adverbial, in order to clarify the aspectual differences. 
8 Nor can I find such a historical record in any other source. 
718
Most of the textbook examples chosen to illustrate the politeness and tact advantages of will + be + -ing are from conversational settings. But how far can this motivation explain the rising frequency of will + be + -ing in non-fictional written texts of LOB and FLOB, which are typically of a more impersonal, less interactive character? 
A similar notion to “already decided future” is what Leech (1971/1987: 69) calls “future-as-matter-of-course”. By this, will + be + -ing implies a certain kind of knowledge on the part of the speaker: the predicted event is viewed as happening in accordance with the course of nature, a routine, schedule or some other flow of expectation. This is most apparent in the examples like (16), which seems to treat as normal (in keeping with expectation) an event that most people would find bizarre or improbable:9 
(16) Margot will be poisoning her husband when he gets home. (Leech 1987: 69) 
Although Leech (1987: 69) himself argues that “‘matter-of-courseness’ … has little distinctive value” in situations lacking agentive subjects, such as (17) as opposed to (18), I feel (18) is more likely when the speaker wishes to convey a sense of judgement based on knowledge, as opposed to a bald prediction. Admittedly the difference is a subtle one, and it could also be treated as a case of stylistic preference. Quirk et al. (1985: 217) suggest that the latter type of sentence is more informal. 
(17) The sun will set in a minute. (Leech 1987: 69) 
(18) The sun will be setting in a minute. (Leech 1987: 69). 
Leech (1987) and Quirk et al. (1985) illustrate matter-of-courseness with other modal auxiliaries + be + -ing besides will. Their examples are discussed in terms of absence of colouring of personal intention or choice made by the speaker or the grammatical subject, rather than predicted events in line with the speaker's expectation. For instance, (21) is glossed by Quirk et al. as It is just possible that Jill will be calling and contrasted to Jill might call this afternoon, which “implies that the visit will depend on Jill's decision”. 
(19) I may not be working this afternoon. (Leech 1987: 98) 
(20) I must be going soon. (Leech 1987: 99) 
(21) Jill might be calling this afternoon. (Quirk et al. 1985: 236) 
As stated earlier, historical commentary on modals + be + -ing, particularly on the emergence of a special aspectual use, is rather thin. Thus I cannot say whether these modals developed later or earlier than will + be + -ing. 
It is not generally claimed that matter-of-courseness is implied in the following constructions: be going to, be to, futurate use of be + -ing, futurate use of simple present. However this is not to say that they do not overlap with will + be + -ing in several respects, the most important of which is that of a future prediction based on circumstances (typically arrangements or plans) existing in present time. 
3.1.3 Extended / virtual notion of “event in progress” 
It was stated earlier that situations like those in (3), (4), (7) and (8) do not appear to fit the description of “event in progress at reference time”. If we wish to account for them in terms of aspect, it seems that we have either to view them as complete wholes – exceptional cases of the progressive construction, or find a way to view them as progressive in some virtual or extended sense. A number of scholars10 have taken the second route for the so-called “futurate progressive” as in (22) 
(22) I'm leaving tomorrow. 
They consider that the speaker is saying it is as if the situation of leaving tomorrow has already begun and is ongoing at speech time. This is usually explained by the fact that some kind of preparation (mental or physical) for leaving has already begun; Hirtle (1967: 95) states: “The preparation stage may involve merely an engagement, yet the event may nevertheless be felt to be already started”. Williams (2002: 95-98) argues that examples like (22) as well as those with will + be + -ing, (23), convey a sense that the future actualisation of an event is part of a wider situation, consisting of implicit arrangements already begun, or states of affairs already in existence in the present. He adds that the difference 
9 Celle (2001) offers a number of insightful refinements to the “matter-of-course” theory. See further, 3.3 
10 For example, Hirtle (1967), Close (1988), Declerck (1991) and Williams (2002). 
719
with (22) above is that the focus of will + be + -ing is on the future actualisation rather than present circumstances, on account of will. 
(23) I'll be leaving tomorrow. 
Williams goes on to draw a systematic distinction between two kinds of be + -ing, both called “progressive”: (i) 
the “regular” or “basic” progressive, in which the situation described can accurately said to be ongoing at reference time, and (ii) an extended type in which a wider situation is envisaged, with some part of this deemed to be in progress.11 Will + be + -ing, he says, can be used both for regular progressive situations and for the extended type. 
I feel Williams makes a plausible case for connecting different future uses of be + -ing, with the caveat that the two types may not be as discrete as he suggests. How reliably can we separate literally-ongoing from virtually-ongoing situations? Moreover, an important omission in his description is the “interpretative” use of be + -ing, cf. (24) and (25).12 This too seems to require a “virtual” kind of construal if it is to qualify as an example of progressivity. It occurs across a range of tenses, not just with will. 
(24) If you pull that lever you will be killing me. (Ljung 1980: 77) 
(25) When I said the ‘boss’ I was referring to you. (Huddleston 2002: 165) 
Here the same world situation is captured by two predications, one with, one without be + -ing, the latter serving to explain or interpret the former. Huddleston adds: “the internal (imperfective) view is appropriate to the explanatory function of the clause – in emphasising duration, the progressive metaphorically slows down or extends the situation in order to be able to focus on clarifying its nature”. (2002: 165) 
3.2 Corpus analysis of semantic categories 
Following the description presented above, I have opted for two basic categories of will + be + -ing in the corpus analysis of LOB and FLOB. This primary division is based on aspectual interpretations, namely: 
Type I: future situation viewed as being in progress at reference time 
Type II: future situations that cannot be viewed “literally” as in progress at reference time; they are either viewed perfectively (as proposed in most theories), or viewed progressively by a special interpretation, as hypothesised by Williams (2002) above. 
Type I is clearest when a temporal frame can be assumed: 
(26) If the police don't keep me I'll be waiting for you when you finish tonight. (LOB L) 
Type II is clearest when it is difficult to impute durativity to the verb: 
(27) I'll be going out briefly at some stage, but Kay'll be here. (FLOB K) 
Within Type II, I try to distinguish cases where a volition-disclaiming effect is not a likely motivating factor for its use; and so-called “interpretative progressives” (see 4.1, and below, for discussion). However, epistemic uses of will + be + -ing with situations in present time (as in [28]) below) were given a category all on their own, even though their durative character puts them closer to Type I. 
(28) Castalia looked at her watch. The time was half-past nine. 
“Will she still be working now?” 
“Very probably. Why? Do you intend to call?” (FLOB N) 
In table 5, reporting the results of the semantic analysis, frequencies for each type are estimated. To give an idea of the 
extent of indeterminacy involved, I report both my own decisions (as “rater #1”) and those of another annotator (“rater #2”). It should be noted that rater #2 used “future-matter-of-course” as the basis for deciding on Type II, in other words the special aspectual value and the epistemic notion of matter-of-courseness were always felt to coincide. In my case 
11 Williams hypothesises an implicit wider situation in progress to explain other tense-aspect combinations with be + -ing, notably the indefinite use of the present perfect progressive, and the progressive imperative. 
12 See also (for example) Buyssens (1968) and Ljung (1980) for extensive further discussion of the interpretative progressive. 
720
the notions of “prior arrangement” or “matter-of-courseness expectation” were too subtle to be identified reliably, and my decisions were driven more by the aspectual interpretation. Moreover, rater #2 categorized interpretative examples as Type I, whereas I treated these as Type II. Both annotators made a distinction between clear and probable cases. 
Table 5: Semantic analysis of will + be + -ing: estimated frequencies in British English 
LOB (1961) frequency FLOB (1991) frequency rater #1 rater #2 rater #1 rater #2 Type I: clear 4 13 2 6 Type I: probable 15 9 21 10 Type II: clear 23 27 39 50 Type II : probable 21 12 30 21 Present epistemic: clear 4 3 4 3 Present epistemic: probable 4 3 3 0 
3.3 Discussion 
The aspectually marked (Type II) use of will + be + -ing appears to be more common in each corpus and to be rising in frequency. It is difficult to say whether this is a genuine trend or an accident of annotation, since the indeterminacy in aspectual properties and meanings such as absence of volition and matter-of-courseness is considerable. 
Type II cases where volition-disclaiming is likely include the following: 
(29) But if golfers are seeing trophies-before-the-eyes, I should point out that Joe Carr will be defending his title. (LOB A) 
(30) WEDNESDAY, apparently, is National Vegetarian Day. I will not be taking part. (FLOB B) 
Among the Type II examples there are a small number in LOB (3 clear cases), and a slightly higher number in FLOB (8 clear cases) where colourlessness seems an improbable motivation because because the verb phrase (even if will + be + -ing were replaced by will + infinitive) contains an inanimate subjects or a non-agentive verb. 
(31) The bursary made all the difference and, since last summer, Lindsay's been the proud owner of a wooden clarinet. It will be going with her at Easter when she tours Germany with the Rutland Concert Band (…) And it will be going with her to Austria this year and to America in 1992. (FLOB E) 
(32) In fact, it may well be that by the time these children have their children, the majority of girls will be maturing at ten. (LOB F) 
A slightly more convincing piece of evidence that volition is not (contrary to many theories) the only factor involved in the rise of will + be + -ing is the fact that the majority of grammatical subjects are third person, and in FLOB third singular grammatical subjects have increased more than other subjects (from 25 to 45 cases). It is more difficult – although of course not impossible – to impute volition to the subject in such contexts. 
This leads one to ask what other factors could be involved in the rise of will + be + -ing. One motivation could be “matter-of-courseness”, not so much in the sense of disclaiming volition, as “in the natural or expected order of things”. Celle (2001) has observed that this notion need not be confined to situations perceived as developing in a “natural” or ordinary way; it can also apply in a new or unexpected situation to which the speaker can adjust. Example (32) above occurs in a context where a “revolution” in the average age for onset of adolescence may be considered abnormal, but the consequence that “‘the girls will be maturing at 10’ is in keeping with the ‘revolution’ referred to and not with the natural course of events.” (Celle 2001: 33). The fuller context is supplied below: 
(32') For example, on an average, a girl of eight in 1959 was as tall and heavy as a girl of eight-and-a-half in 1949. And in ten years the average height of a ten-year-old has increased by half an inch, the average weight by three-and-a-half pounds. Nor does the advance show any signs of halting. In fact, it may well be that by the time these children have their children, the majority of girls will be maturing at ten. Doctors who are delving into the reasons why this revolution is taking place have come up with some intriguing theories. (LOB F) 
Celle goes further and treats matter-of-courseness as a property of all instances of will + be + -ing. In practice I found it difficult to detect the presence of this feature in the corpora because it is very subtle. The following is a rare case 
721
where the co-text provides support for matter-of-courseness (This year … And next [year] … if all goes according to plan): 
(33) This year, Denbies release their first commercial wine. And next, if all goes according to plan, we will be raising a glass of sparkling Denbies to a new phase in English viticulture. (FLOB E) 
Why is it so difficult to establish the aspectual interpretation or establish the presence of additional meanings like colourlessness, matter-of-courseness? I believe this is because the distinctions are rather subtle and in many cases cannot be distinguished with confidence. This is most likely because of overlaps between the different aspectual readings, and between the other semantic categories: 
(a) Difficulties in aspectual categorization 
As noticed by Celle (2001: 23), many authors give the erroneous impression that the future-in-progress meaning (Type I here) is perfectly “regular” and requires no further comment. Although a salient temporal framing effect is a helpful guide to classification, the absence of such a frame should not necessitate construal as a “special” aspectual use. 
Regarding the “special” aspectual use of will + be +ing, the kind of canonical examples found in many textboooks are in fact extremely rare in the corpora. Close (1988: 54), for example, speaks of “an event expected to take place at a definite hour rather than … a process continuing over a period of time” and provides a clear accomplishment situation, The plane will be leaving at 9.25. 
By generalizing into more than one aspectual value, it seems the construction will + be + -ing (like other modals + be + -ing) has lost specificity, and become “blurred”. I believe the fuzziness between the two types, as revealed in the corpus investigation, is better understood if Type II is not the opposite aspectual value from Type I, but instead a metaphorical or virtual extension of Type I. 
Different linguists (and different speakers) may view the same examples in different ways. Several textbooks offer “clear” cases of the special use of be + -ing (e.g. Williams 2002: 204 example [34] below), or clear cases of the regular use (e.g. Palmer 1979: 133, example [35]), which I find difficult to agree with. Thus the disagreements between the two annotators in the LOB-FLOB data appears not to be exceptional. 
(34) I'll be waiting for you outside the station tomorrow. 
(35) In a moment the queen will be alighting from her coach. 
An important sub-type of will + be + -ing that was consistently coded differently by both annotators of our data was the so-called interpretative use. I regarded these as Type II (metaphorical extension), the other annotator regarded these as regular use of Type I. 
(36) When the consultant surgeon puts his case, he will be speaking from the heart. (FLOB B) 
(b) Overlapping of other semantic categories 
As remarked by Samuels (1972: 57) in his theory of the origin of two uses of will + be + -ing, colourlessness (absence of volition) was probably originally an incidental property of Type I, before it was extended to Type II. Thus we can probably expect there to be traces of colourlessness in both types, and so it is not always a distinctive feature. 
(37) [from a parliamentary debate] Now that he has had a day to reflect, the whole House will be waiting for his answer to some vital questions. (FLOB H) 
Sometimes one annotator saw a sentence as “matter-of-course” while the other saw it as “interpretative”. This example could be viewed as one statement interpreting the deeper significance of another (interpretative), or as an event fully congruent with the speaker's expectation, with no personal choice involved (matter-of-course): 
(38) It is not practical to assume that just because you stop using tropical timber you will be automatically maintaining the rainforest. (FLOB E) 
722
4 Conclusions 
Evidence from two matching corpora of British English texts has pointed to a small but significant rise in recent times of modal auxiliary + be + -ing constructions, particularly as compared to a sharp fall in modal auxiliaries as a whole. I have explored the most frequent of these constructions, will + be + -ing, which has become more common outside of fictional texts. Its frequency is highest in correspondence registers, where there is common call for describing or making arrangements. (I cannot yet say whether there has been a diachronic shift in correspondence because this register is not included in the matching corpora surveyed.) In a brief comparison with other constructions it emerges that four other future-referring constructions have either declined or at least not risen in frequency (be to, shall, futurate present progressive, and – surprisingly – be going to), whereas will + infinitive presents a mixed picture. In terms of semantic developments, there are signs that disclaining of volition might not be the only motivation for the rise of will + be + -ing. The aspectual and other semantic categories investigated proved extremely difficult to identify with confidence in the corpora, seemingly because of considerable overlaps between the categories, and the subjective nature of construal. The paradox of two “opposite” aspectual classes may be explainable by a classification into “literally-ongoing” and “metaphorically-ongoing” types (along the lines of e.g. Williams 2002). This allows us (a) to retain the notion of progressive aspect for the construction as a whole and (b) perhaps explain better the fuzziness between the aspectual categories. For the distinction between literally and virtually progressive situations is not likely to be as sharply defined as that between progressive and non-progressive. There is plenty of scope for future research, in particular: earlier phases in the development of will and other modals + be + -ing; connections to other constructions referring to future time; and connection to the infinitive in be + -ing. 
References 
Adamczewski, H. & Delmas, C. 1982 Grammaire linguistique de l'anglais. Paris: Armand Colin. 
Celle, A. 2001 Men will be growing breasts - on will + be –ing in contemporary English In Anglophonia / Caliban n°9. Toulouse: Presses Universitaires du Mirail, 23-40. 
Close, R. 1988 The future in English.In Bald, W (ed.) Kernprobleme der englischen Grammatik: Sprachliche Fakten und Ihre Vermittlung. Munich: Langenscheidt-Longman 
Comrie, B. 1976 Aspect: An Introduction to the Study of Verbal Aspect and Related Problems. Cambridge: Cambridge University Press. 
Declerck, R. 1991 Tense in English: its Structure and Use in Discourse. London: Routledge. 
Görlach, M. 1999 English in Nineteenth-Century England: An Introduction. Cambridge: Cambridge University Press. 
Huddleston 2002 The Cambridge Encyclopedia of the English Language.??] Cambridge: Cambridge University Press. 
Krug, M. 2000 Emerging English Modals: A corpus-based study of grammaticalization. Berlin: Mouton de Gruyter. 
Leech, G. 1971/1986 Meaning and the English Verb. London : Longman. 
Leech forthcoming. Modality on the Move: The English Modal Auxiliaries 1961-1992. In Facchinetti, R., M. Krug and F. Palmer (eds.) Modality in Contemporary English. Berlin: Mouton de Gruyter. 
Ljung, M. 1980 Reflections on the English Progressive. Gothenburg Studies in English 46. Gothenburg: Acta Universitatis Gothoburgensis. 
Mair, C. 1997 Parallel Corpora: A Real–Time Approach to Language Change in Progress. In Corpus–based Studies in English: Papers from the 17th International Conference on English Language Research on Computerized Corpora (ICAME 17.), Magnus Ljung (ed.), 195–209. Amsterdam: Rodopi 
Mustanoja, T. 1960 A Middle English Syntax. Part I: Parts of Speech. (Mémoires de la Société Néophilologique de Helsinki 23.) Helsinki: Société Néophilologique. 
Palmer, F. 1974/1988 The English Verb. London and New York: Longman. 
Rissanen, M 1999 “Syntax” in Lass, R. (ed.) The Cambridge History of the English Language. Vol III: 1476-1776. Cambridge: Cambridge University Press 
Samuels, M. 1972 Linguistic evolution with special reference to English. Cambridge: Cambridge University Press 
Smitterberg, E. 2002 The Progressive in 19th-Century English: A Process of Integration. PhD thesis: Uppsala Universitet. 
Strang, B. 1970 A History of the English Language. London: Methuen. 
Strang. B. 1982 “Some Aspect of the History of the Be + -ing Construction”. In Anderson, J. (ed.), Language Form and Linguistic Variation: Papers dedicated to Angus McIntosh. (Current Issues in Linguistic Theory 15.) Amsterdam: John Benjamins, 427-474. 
Visser, F. Th. 1973 An Historical Syntax of English Language. Part III, 2nd Half: Syntactical Units with Two and with More Verbs. Leiden: E.J. Brill. 
Wekker, H. 1975 The expression of future time in contemporary British English. Amsterdam: North Holland Publishing Company. 
Williams 2002 Non-progressive and progressive aspect in English. Fasano di Puglia: Schena Editore. 
723

A profile-based calculation of region and register variation: the synchronic and diachronic status of the national variants of Dutch Dirk Speelman, Stefan Grondelaers & Dirk Geeraerts Department of Linguistics, Blijde-Inkomststraat 21, B-3000 Leuven, Belgium dirk.speelman@arts.kuleuven.ac.be 
Over the last decade, the Leuven Research Unit of Quantitative Lexicology and Variational Linguistics has developed a corpus-based method for the investigation of region and register variation in and between the national variants of Dutch, viz. Belgian Dutch and Netherlandic Dutch. The basic characteristics of the methodology are the following. Geeraerts, Grondelaers & Speelman (1999) introduces "linguistic uniformity" as an operational measure to determine the lexical distance between language varieties. Uniformity is calculated by quantifying the overlap between onomasiological profiles. (The onomasiological profile of a particular concept in a particular language variety is the set of synonymous alternative terms used to name that concept in that language variety, differentiated by the relative frequency of the alternative terms in a corpus.) These uniformity-based calculations have in the past been used in top-down analyses of convergence and stratification. More recently, they are incorporated in a broader framework of multivariate statistical analysis to achieve a bottom-up classification of language varieties. 
In this talk we want to present the technical basis of profile based calculations. After explaining the actual calculations, we illustrate the merits of the approach. By taking all alternative designations of a phenomenon (and their frequencies) into account, we can (i) put the statistical importance of isolated variables into perspective, (ii) avoid the blurring effect of polysemy and semantic ambiguity of isolated variables, and (iii) avoid confusion between the region & register differences that interest us, and thematic, situational or medium-specific biases in the corpus. 
From the list of practical issues that need to be tackled before profile-based uniformity calculations can safely be extrapolated to any kind of lexical and non-lexical indicators (an extension to function words can be found in Grondelaers e.a. 2001), we zoom in on the topic of variable selection. We show how this selection is facilitated by the automated generation of exhaustive lists of 'stable' markers of specific registers and regions (which are derived from all possible two-by-two comparisons of the word or n-gram frequency lists of the corpora that represent the registers and regions at issue - in this preparational step statistical significance is established with the method explained in Dunning 93. References Dunning T. 1993. ‘Accurate Methods for the Statistics of Surprise and Coincidence'. Computational Linguistics 19(1): 61-74. Geeraerts D., Grondelaers S., Speelman D. 1999. Convergentie en divergentie in de Nederlandse woordenschat. Een onderzoek naar kleding- en voetbaltermen. Amsterdam: Meerstensinstituut. Grondelaers S., van Aken H. Speelman D., Geeraerts D. 2001. ‘Inhoudswoorden en preposities als standaardiseringsindicatoren. De diachrone en synchrone status van het Belgische Nederlands'. Nederlandse Taalkunde 6: 179-202. 
733

Exploiting a parallel TEXT - DATA corpus 
Somayajulu G. Sripada, Ehud Reiter, Jim Hunter and Jin Yu 
Dept. of Computing Science 
University of Aberdeen 
{ssripada,ereiter,jhunter,jyu}@csd.abdn.ac.uk 
Fax: +44 (0)1224 273422 
Abstract 
In this paper, we describe SUMTIME-METEO, a parallel corpus of naturally occurring weather forecast texts and their corresponding forecast data; data that the human authors inspected while writing the forecast texts. We have analysed the corpus to acquire knowledge needed to build a text generator for automatically producing textual weather forecasts from numerical weather prediction data. Although parallel corpora are commonly used for the development and evaluation of machine translation technology, it is fairly novel in the text generation community. Our analyses of the corpus, in some cases, produced ambiguous results that are not useful and reflected inconsistencies in the underlying corpus. Despite the internal inconsistencies, the text-data parallel corpus was helpful in generating initial hypotheses, which were then tested with knowledge from other sources. We also describe how we have used the corpus for evaluating our prototype forecast text generator. 
1 Introduction 
SUMTIME-METEO is a parallel corpus of 1045 weather forecast texts and the numerical data (output of a numerical weather prediction (NWP) model) that human forecasters examined when writing the texts. While parallel corpora of texts and their translations into other languages are fairly common and are heavily used in machine translation (Brown et al 1990), parallel corpora of data and their “translations” into texts are more unusual. Barzilay and McKeown (2001) report use of parallel translations of texts (i.e. text to text but not data to text) for an NLG application. We are using SUMTIME-METEO to understand the structure, content selection rules and linguistic issues in weather forecast generation. More generally, we believe that parallel text-data corpora could be a valuable resource for researchers interested in semantics, as they provide empirical data on how non-linguistic content is expressed as words and sentences. SUMTIME-METEO is a naturally occurring corpus in the sense that the forecast texts were written for actual clients, and were not artificially written just for our project. 
The corpus is built as part of the SUMTIME project (http://www.csd.abdn.ac.in/research/sumtime). In the project our objective is to develop generic techniques for summarising time series data. Towards this end, we are working in three different domains – meteorology, gas turbines and neonatal intensive care units. In the domain of meteorology, we are trying to build a system, which automatically generates weather forecasts from numerical data, in collaboration with WNI Oceanroutes, Aberdeen, U.K. Our weather forecast generator, SUMTIME-MOUSAM is based on knowledge acquired from several sources, including analysis of the SUMTIME-METEO corpus. It is currently installed in WNI and is being run in parallel with human forecasters (with computer-generated forecasts compared against manually written ones), and may soon go “live” for some specialised types of forecasts. 
2 Corpus description 
2.1 Background 
Modern methods of weather forecasting are largely based on computer simulations of numerical weather prediction (NWP) models. These models generate predicted values of various weather parameters such as wind speed, wind direction and precipitation for various time points. In other words, the output of an NWP model is a multivariate time series. Human forecasters use the time series data sets generated by NWP models as the major source of information when writing forecast texts, although they also have access to other information such as satellite weather maps. See section 7 of (Sripada et al 2001a) for a more detailed description of the forecasting and writing process. 
Weather forecasts are produced for end-users with different information requirements. For instance, forecasts aimed at the general public often describe general outlook of the weather. On the other hand, forecasts aimed at more technically oriented audiences require comparatively more details of the weather. In SUMTIME, we focus on forecasts produced for oil company staff supporting offshore activities in the North Sea. Our collaborating organization works with three NWP models to generate these forecasts - Marine model, MaxMin model and Media Kernel model. Each NWP model is good at 
734
predicting a particular set of weather parameters; this is why several NWP models are used. For instance, Marine model is good at predicting wind and sea wave data where as the other two (which have many overlapping parameters) are good at predicting other parameters such as precipitation, temperature etc. 
2.2 Corpus contents 
We receive weather data corresponding to a specific offshore oil-drilling site in the form of three files corresponding to the three NWP models and one file containing the manually written forecast text: 
1. Files with extension .tab (or .etb) – these are the data files generated by the Marine model. This model generates the predicted values for wind and wave related parameters at three hourly intervals. 
2. Files with extension .mmo – these are the data files generated by the MaxMin model. This model generates the predicted values for weather parameters such as cloud, precipitation, at an hourly interval. 
3. Files with extension .csv – these files are generated by Media Kernel model and contain roughly similar information to the .mmo files. 
4. Files with extension .prn - the official forecast texts issued by our collaborating organization (written by human forecasters using the above data) 
A set of these four files forms one unit in the corpus because they all correspond to one official forecast issued by our collaborating organization. WNI has been sending us two such sets a day since the summer of 2000 (with occasional gaps and interruptions); most of these files are the 0300 and 1200 forecasts for the Magnus, Thistle, and NW Hutton oil fields. The current distribution version of our corpus contains files received by 10 May 2002. After this date our prototype system was installed at WNI and therefore we assumed that the human authors might write texts under its influence. This includes 1119 forecasts (.prn files) (ignoring files which are empty, duplicates, or updates) and similar numbers of data files. Due to gaps we do not have data files for all the forecasts; for example, we only have the corresponding .tab file for 1045 forecasts. 
2.3 Marine model file 
These files consist of a header and a body as shown in Figure 1. 
Header: Essentially the header information associates the data in the body part to a specific location and time. The first line of the tab (etb) file states information about the location (the name(s) of the location) to which the predicted data belongs. The sample data shown in Figure 1 belongs to three locations east of Shetland – MAGNUS, THISTLE and NW HUTTON. It should be noted that NWP models do not work with actual location names. Instead they compute data for the entire surface of the earth by dividing it into a two dimensional array called ‘grid’ and output data for a ‘grid point’ (node in the array). A grid point might map onto more than one location name. The second line of the header states the start date of the predicted data in GMT. The format of the date string is “dd-mm-yy”. In Figure 1, the date is 24 October 2000. ________________________________________________________________ 
MAGNUS / THISTLE / NW HUTTON FIELDS, EAST OF SHETLAND 
24-10-00 
25/00 SSW 12 15 18 2.0 3.2 WSW 1.7 8 
25/03 SSE 11 13 17 2.0 3.2 WSW 1.8 8 
25/06 ESE 18 22 28 2.4 3.8 SW 2.0 8 
25/09 ESE 16 20 24 2.7 4.3 SSW 2.4 8 
25/12 E 15 18 23 3.1 5.0 SSW 2.8 8 
25/15 ENE 15 18 23 3.2 5.1 SSW 3.0 9 
25/18 ENE 18 22 27 3.4 5.4 SSW 3.0 9 
25/21 NNE 20 25 31 3.4 5.4 SSW 2.9 9 
26/00 NNW 26 32 40 3.5 5.6 SSW 2.7 9 
_____________________________________________________________________ 
Figure 1. A portion of .tab (.etb) file for 24-10-2000. 

Header 

Date 

Location 

Body 

Time stamp 

Weather Data 
735
Body: This part of the file contains the weather data in rows and columns. Each row holds weather data predicted for a point of time (time stamp). The first column holds the time stamp and each of the other columns holds the value of a specific weather parameter. The format of the time stamp is “dd/HH”. In Figure 1, the time stamp in the first row is ‘25/00', which means that the data in that row belongs to 0000 hours on 25-10-00. The data columns are described below in Table 1 (the first column is already described as time stamp): 
Column Weather parameter Description 2 Wind Direction It is expressed as a string such as ‘E’ in Figure 1 representing the direction ‘East'. Numerically ‘E’ corresponds to 90 degrees. This model represents wind direction in quanta of 22.5 degrees. Thus wind direction can take any of the possible (360/22.5 =16) 16 values. Enumerating them we have N, NNE, NE, ENE, E, ESE, SE, SSE, S, SSW, SW, WSW, W, WNW, NW, NNW. 3 Wind Speed at 10m height It is measured in Knots at 10m height. These are also called surface winds. In Figure 1, for the time stamp 25/06 it is 18 Knots. 4 Gust at 10m height Gusts measured in Knots at 10m height. In Figure 1, for the time stamp 25/06 it is 28 Knots. 5 Gust at 50m height Gusts measured in Knots at 50m height. In Figure 1, for the time stamp 25/06 it is 28 Knots. 6 Significant Wave Height Expressed in metres. In Figure 1, for the time stamp 25/06 it is 2.4 7 Wave Period Expressed in seconds. In Figure 1, for the time stamp 25/06 it is 3.8 8 Swell Direction It is expressed in a similar way to wind direction. In Figure 1, for the time stamp 25/06 it is SW. 9 Swell Height Expressed in metres. In Figure 1, for the time stamp 25/06 it is 2.0 10 Swell Period Expressed in seconds. In Figure 1, for the time stamp 25/06 it is 8. 
Table 1. Parameters generated by marine model ________________________________________________________________________ 
OCEANROUTES SPECTRAL WAVE AND WEATHER FORECAST. 
DUTY FORECASTER AVAILABLE AT ALL TIMES.PHONE ABERDEEN (01224) 248080 
FORECAST FOR:- 
MAGNUS, THISTLE AND NW HUTTON FIELDS, EAST OF SHETLAND 
1.INFERENCE 0300 GMT, TUESDAY, 24-Oct 2000 
LOW PRESSURE WILL MOVE SLOWLY AWAY NE INTO THE NORWEGIAN SEA, WITH 
A RIDGE MOVING EAST ACROSS THE NORTH SEA THIS EVENING. A VIGOROUS 
FRONT WILL CROSS THE CENTRAL NORTH SEA TONIGHT, WITH THE PARENT 
LOW MOVING EAST ACROSS NORTHERN SCOTLAND TOMORROW MORNING THEN CONTINUING EAST ACROSS THE NORTHERN NORTH SEA BY TOMORROW 
NIGHT. 
AUTHOR NAME 
2.FORECAST 06-24 GMT, TUESDAY, 24-Oct 2000 
=====WARNINGS: NIL ======= 
WIND(KTS) CONFIDENCE HIGH 
10M: SW 10-14 VEERING W 18-22 GUSTS 30 BY LATE MORNING 
THEN BACKING AND EASING SSW 12-16 THIS EVENING 
50M: SW 12-18 VEERING W 23-28 GUSTS 38 BY LATE MORNING 
THEN BACKING AND EASING SSW 15-20 THIS EVENING 
736
WAVES(M) CONFIDENCE HIGH 
SIG HT: 2.5-3.0 FALLING 2.0-2.5 
MAX HT: 4.0-5.0 FALLING 3.0-4.0 
PER(SEC): SEA 3-4 RISING 5 FOR A TIME, SWELL 8-9 
WEATHER: SHOWERS DYING OUT THIS EVENING 
VIS(NM): OVER 10 FALLING 3-5 IN SHOWERS 
TEMP(C): 10-11 
CLOUD: 5-7 CU/SC 1500-2500 WITH 7-8 CU/CB 1000 IN SHOWERS 
(OKTAS/FT) 
LIGHTNING RISK:MODERATE (50-60 PER CENT) FALLING NIL LATER 
3.FORECAST 00-24 GMT, WEDNESDAY, 25-Oct 2000 
WIND(10M): SSW 12-16 BACKING ESE 16-20 IN THE MORNING, BACKING 
NE EARLY AFTERNOON THEN NNW 24-28 LATE EVENING 
(50M): SSW 15-20 BACKING ESE 20-25 IN THE MORNING, BACKING 
NE EARLY AFTERNOON THEN NNW 30-35 LATE EVENING 
SIG WAVE: 2.0-2.5 RISING 3.0-3.5 BY AFTERNOON 
MAX WAVE: 3.0-4.0 RISING 5.0-5.5 BY AFTERNOON 
WEATHER: RAIN SOON, CLEARING TO SHOWERS IN THE EVENING 
VIS: GOOD BECOMING MODERATE IN RAIN 
4.FORECAST 00-24 GMT, THURSDAY, 26-Oct 2000 
WIND(10M): NNW 24-28 EASING 14-18 IN THE MORNING THEN BACKING 
NW 10 OR LESS IN THE EVENING 
(50M): NNW 30-35 EASING 18-23 IN THE MORNING THEN BACKING 
NW 12 OR LESS IN THE EVENING 
SIG WAVE: 3.0-3.5 FALLING STEADILY TO 1.5-2.0 LATER 
MAX WAVE: 5.0-5.5 FALLING STEADILY TO 2.5-3.0 LATER 
WEATHER: SHOWERS DYING OUT 
5A.LONG RANGE OUTLOOK: FRI 27-Oct 2000, AND SAT 28-Oct 2000, 
WIND(10M): NW 10 BACKING S-SE 22-26 BY MID FRIDAY, EASING 18-22 
BY EARLY SATURDAY THEN RISING SSW 35-40 LATER 
SIG WAVE: 1.5-2.0 RISING 2.0-3.0 ON FRIDAY AND 5.0-5.5 
LATER ON SATURDAY 
5B.LONG RANGE OUTLOOK: SUN 29-Oct 2000, AND MON 30-Oct 2000, 
WIND(10M): S-SW 35-40 EASING 10-14 ON SUNDAY, RISING S 20-24 
ON MONDAY 
SIG WAVE: 5.0-5.5 FALLING 3.0-4.0 BY LATE SUNDAY 
N.B. THE HIGHEST INDIVIDUAL WAVE THAT MAY BE EXPERIENCED IS OF THE 
ORDER OF TWICE THE SIGNIFICANT WAVE HEIGHT. 
____________________________________________________________________ 
Figure 2. WNI Forecast for 24-10- 2000 
2.4 Forecast text file 
A sample forecast text is shown in Figure 2. 
Offshore weather forecasts organise the forecast in terms of forecast periods. Each forecast period describes the forecast for roughly a period of 24 hours. It is common to have three forecast periods or weather forecast information for 72 hours in an official forecast issued by WNI. Sometimes as in Figure 2, there is also a long-range forecast. Before predictions for any individual forecast period are presented, the official forecasts issued by WNI contain a description of the general outlook of the weather in that region under the heading ‘Inference'. Thus as shown in Figure 2, official forecasts are structured into five sections. The first section is the inference. The next three are the forecasts for the three forecast periods. The fifth one is what is called the long-range forecast. Forecast for each forecast period contains the following elements: 
737
Wind10M – this part of the forecast text summarises the behaviour of the wind at 10-meter height. This statement is based on parameters, wind speed, wind direction and gust at 10 meter from the marine model as shown in Figure 1. 
Wind50M – this part of the forecast text summarises the behaviour of the wind at 50-meter height. Wind direction and Gust50m are also used along with Wind50m for writing this element. 
Waves Sig. Ht (M) – Significant wave height; average height of the 1/3 highest waves in a record, defined as an approximation to the characteristic wave height (average height of the larger well-formed waves, observed visually). Swell height, swell direction and swell period are also used along with Sig. Ht. 
Waves Max. Ht (M) – Maximum wave height for the specified period of time. Swell height, swell direction and swell period are also used along with Max. Ht. 
Wave Period – summarises the wave period data 
Weather – summarises mainly the cloud cover and precipitation. 
Vis – summarises visibility 
Temp – temperature range 
Cloud – summarises amount of cloud 
Forecast texts in the first five fields (Wind10M, Wind50M, Sig. Wave, Max. Wave and Wave Period fields of Figure 2) are all produced primarily from the data shown in Figure 1. For example, the Wind10m field for the forecast period 06-24 GMT 25 Oct 2000 as shown in Figure 2 summarises data from the second (Wind direction), third (Wind speed at 10 meter altitude) and fourth (gust at 10 meter altitude) columns of the marine model data shown in Figure 1. The last four fields (Weather, Visibility, Temperature and Cloud) of the forecast shown in Figure 2 are produced primarily using the data from the other two models (MaxMin and Media Kernel). Details about the contents of each of these models have been presented in (Sripada et al 2002a). 
2.5 Distribution of the corpus 
SUMTIME-METEO can sometimes be distributed to other researchers. Interested researchers should contact Dr. Ehud Reiter (email: ereiter@csd.abdn.ac.uk). 
3 Knowledge acquisition studies using SUMTIME-METEO 
In this section, we describe how we are using the parallel text–data corpus in the SUMTIME project to acquire knowledge to build a weather forecast text generator, SUMTIME-MOUSAM. In our work, we follow the three-stage reference architecture for text generation as described in (Reiter and Dale 2000). Our main objective for corpus analysis has been to acquire knowledge needed for all the three stages of text generation (document planning, micro-planning and realization) from the parallel corpus. Corpus analysis happens to be one of the techniques in our knowledge acquisition studies. We have also tried other techniques familiar in the expert system community (Scott et al 1991). Our idea has been to initially collect knowledge from multiple sources and to finally consolidate the different findings into a consistent and usable model to be used for building SUMTIME-MOUSAM. 
For our discussion here we focus on texts from the Wind10M field (please refer to the Wind10M field in Figure 2) although our software generates all the fields. Forecasters write these texts primarily by inspecting data from the second (Wind direction) and the third (Wind speed at 10 meters altitude) columns of the marine model output (please refer to the Figure 1). Other data that is included in Wind10M texts such as gusts is secondary and therefore omitted in this discussion. 
3.1 Content determination 
The first task of a data summarization system is to select data items from the input data set to include in the summary text. Learning the content determination rules automatically, from the parallel corpus, was hard. Manual analysis of the parallel corpus revealed that human forecasters select data points that are representative of trends in the input data set. Accordingly we have used a well-known segmentation algorithm from the KDD community known as bottom-up segmentation algorithm (Keogh et al 2002) to segment the input data set (for example, Wind speed and direction at 10 meter altitude separately) in order to determine the trends in the input. However, our ‘think aloud’ sessions with an expert forecaster where he spoke out the process while writing forecast texts revealed an alternative procedure for data selection which is based on identifying ‘significant’ changes in the input data. 
738
Time Wind Speed in Knots Wind Direction 10-10-2003 00:00:00 5 S 10-10-2003 03:00:00 6 S 10-10-2003 06:00:00 7 S 10-10-2003 09:00:00 8 S 10-10-2003 12:00:00 9 S 10-10-2003 15:00:00 10 S 10-10-2003 18:00:00 11 S 10-10-2003 21:00:00 12 S 11-10-2003 00:00:00 13 S 
Table 2. Fictitious Wind Speed Data for 10-10-2003 
For example, consider the fictitious wind data shown in Table 2. The wind speed in this example case is monotonically increasing from 5 knots to 13 knots and the wind direction has always been S, southern. Segmentation approach fits the entire data set into one single rising line and accordingly selects the first (data corresponding to the time 00 hours on 10-10-2003) the last (data corresponding to the time 00 hours on 11-10-2003) data points. The wind statement generated in this case could be something like ‘S less than 10 rising steadily 10 –15 by midnight'. On the other hand, expert approach is based on an externally controlled threshold value for deciding ‘significant’ changes. If we assume a value of five (which is what the expert suggested) for the threshold, the expert approach selects data points as explained next. After picking up the first data point, this approach looks for the data point that is greater or equal (also smaller or equal) than the first data point by five (the threshold). In the example case, this happens to be the data corresponding to the time 15 hours on 10-10-2003. The wind statement generated in this case could be something like ‘S less than 10 rising steadily 8-13 by mid afternoon'. 
We have implemented both these procedures in our system and compared the two using an intrinsic evaluation (Sripada et al 2002b). The evaluation results indicated that segmentation procedure acquired from corpus studies is better than the procedure suggested by the expert. The expert meteorologist with whom we did the think aloud session also agreed with the result of evaluation eventually. This can be explained by observing that the meteorologist not being an expert in designing computer algorithms initially failed to develop a model reflecting his selection process accurately. 
3.2 Learning the meaning and usage of time phrases and change verbs 
The next task in a data summarization system is to make appropriate lexical choices to express the selected content. Here, we need good models of word meaning (or usage). Although there are existing procedures from the field of linguistics and lexicography for acquiring meanings of words, a parallel corpus with its text to data associations offers a novel resource to link words to numerical data and thereby to explore their meaning (or usage) in terms of the numerical data. Preliminary analysis of corpus showed that the majority of forecast texts (please refer to the forecast in Figure 2) are made up of time phrases (e.g. ‘by evening’ and ‘in the morning') and change verbs (e.g. ‘veering’ and ‘backing'). Thus we focused our attention on learning the meaning of these two types of phrases. 
First we describe our analysis of the parallel corpus for learning the interpretation of time phrases. Particularly, our objective is to determine what time the forecaster meant when he used a time phrase such as ‘by evening'. This knowledge will help our forecast generator decide the specific time phrase to mark a change in wind speed, say at 1800 hours. It should be mentioned in this context that expressions of time do not get standardised in the same way as expressions of cloud cover or precipitation. Therefore, forecasters don't have access to a list of acceptable time phases and their time mappings. Also it is very rare for forecasters to mention numerical time values in forecast texts. For this study, we aligned phrases from forecast texts with numerical data from the numerical weather simulation, and used this alignment to infer the meanings of time phrases. The precise alignment process is explained next. 
739
Sr. No. Phrases Information Speed Direction Time Phrase Change Verb 1 SSW 12-16 12-16 SSW None None 2 BACKING ESE 16-20 IN THE MORNING, 16-20 ESE IN THE MORNING BACKING 3 BACKING NE EARLY AFTERNOON 16-20 NE EARLY AFTERNOON BACKING 4 THEN NNW 24-28 LATE EVENING 24-28 NNW LATE EVENING BACKING 
Table 3. Parser Output for Wind10M text for 25-10-00. 
As stated earlier we performed our analysis on forecast texts from the Wind10m field. As shown in Figure 2 there are three Wind10m texts per forecast corresponding to the three days for which the forecast is issued. Initially we parsed these texts with a simple parser tuned to the linguistic structure of these texts. The parser first breaks each Wind10M statement up into phrases and then extracts information such as wind speed and direction from each phrase. This extracted information is recorded into a database. For example, for the Wind10m text from the 06-24 GMT, 25 Oct 2000 forecast in Figure 2, the parser extracts phrases and records information as shown in Table 3. 
In Table 3, phrase 3 has the speed information missing and phrase 4 has the verb missing. Our parser works in such cases by carrying forward the values from the previous phrase. Thus the wind speed for the third phrase is same as the second one and the change verb for the fourth phrase is same as the third. We have noticed that such cases of elision (omitted expressions) are very common in forecast texts, and carried out a separate study to acquire rules for eliding words or phrases. We describe this study in section 3.3. Our parser produced 8198 wind phrases and their related information from all the Wind10M statements in our corpus. For learning the meaning of time phrases not all of the 8198 phrases are useful for various reasons. For instance, the first phrase in Table 2 does not have a time phrase and therefore of no use in the corpus analysis (in fact, less than 1% of the initial phrases in wind statements included a time phrase). After applying all such restrictions, the useful number of time phrases reduced to 3654. 
Time Most common phrase in corpus Expert suggested phrases Phrases used in SUMTIME-MOUSAM 0 By late evening Around midnight By midnight 3 Tonight In early hours After midnight 6 Overnight In early morning By early morning 9 By midday During midday By morning 12 By midday Around midday By midday 15 By mid afternoon In mid afternoon By mid afternoon 18 By evening In early evening By early evening 21 By evening During night By evening 
Table 4. Time Phrase Mappings from Corpus studies, Experts and SUMTIME-MOUSAM 
Next we associated each wind phrase with an entry in the corresponding data file. This process, known as alignment, is very important for performing analysis on a parallel corpus (Och and Ney 2000). Good alignment techniques produce better results in the subsequent corpus analysis. Initially we have aligned all those phrases that matched with an entry in the data file without ambiguity. The proportion of such aligned pairs was 43%. Then we have used simple heuristics to resolve ambiguities in the remaining cases. After applying our heuristics we have been able to find 2539 aligned pairs, which is about 70% of the collected phrases. We are still working on improving the alignment process. More details about this study can be found in (Reiter and Sripada 2002). Using the aligned pairs (wind phrase and data entry) we have then mapped all the time phrases to their corresponding interpretations. For example, if the text contains the phrase “VEERING SW 10-14 BY EVENING” and the 1800 hours entry of the NWP model output is the only one with a direction of SW and a speed in the 10-14 range, then we assume that in this instance “BY EVENING” means 1800. This analysis showed that time phrases such as ‘by evening’ had much vaguer and more varied time mappings than we expected. This is bad for us because we cannot use these mappings in our forecast text generator. We have approached the experts 
740
to verify the results of our corpus analysis and to suggest time phrase interpretations. The first column in Table 4 shows the various times for which we need the time phrases. The second and third columns show the time phrases discovered in our corpus analysis and those suggested by the experts. The last column shows the consolidated time phrases that we finally used in SUMTIME-MOUSAM. 
We have also used the corpus for learning what verbs to use to describe changes in wind speed and direction. For example, the Wind10M text for 26-10-2000 in Figure 2 uses verbs ‘easing’ and ‘backing'. Our objective in this study is to learn which change in the underlying data prompted the forecaster for using these verbs. This was done by applying the machine learning algorithm, Ripper (Cohen 1995) to our corpus. This study highlighted that there were substantial individual differences in the use of near-synonyms such as EASING and DECREASING. Predictive rules could be created for individual forecasters, but not for the corpus as a whole (if we did not allow the rules to include author). We also ran C4.5 (Quinlan 1993) on our corpus; this gave similar results. 
3.3 Learning rules of expression 
The next important task in data summarization is to learn when to elide (omit) words and phrases. We have used only the forecast texts for this study without the parallel data component of the corpus. The results of this analysis were surprising because they disagreed with what expert forecasters had told us. For example, we noticed from our corpus studies that when the wind speed (or direction) varied steadily throughout a forecast period, (such as in fictitious data set shown in Table 2) forecasters often omitted a time phrase. For the example data in Table 2, the forecasters might something like ‘S less than 10 rising steadily 10 –15’ rather than ‘S less than 10 rising steadily 10-15 by midnight'. In contrast, the experts felt that end users find it better when a time phrase is included because in this case they are not required to remember when the forecast period ends. We suspect this may reflect forecasters trying to use latest data available to them; in other words, the corpus-derived rules in this case may not be appropriate ones to include in a computer system. 
We have carried out a number of other analyses on the corpus to understand how forecast content is expressed in the texts. A number of expression level variations have been observed in the textual forecasts. For example, single digit numerical values have been expressed with and without leading zero such as ‘06’ and ‘6'. 
3.4 Evaluation of prototype forecast text generator 
We have used the corpus for evaluating the output of our forecast generator by comparing computer-generated forecasts to manually written ones. This was done at a conceptual level that captures the content in a forecast at a higher level independent of the variations in expression. Comparison at the conceptual level was chosen because text level comparison would be difficult with all the expression level variations discovered in the above studies. For extracting the conceptual structures from the corpus texts we have used a parser, which is an improvement over the one described in section 3.2. It extracts additional information from the wind phrases. Instead of writing the extracted information into a database the parser here populates the conceptual structures. Conceptual representation of wind forecast consists of a tuple with the following elements 
- Time 
- Wind speed lower range 
- Wind speed upper range 
- Wind direction (in degrees) 
- Modifiers (gusts, shower, steady-change, gradual-change) 
For example, for the Wind10m text from the 06-24 GMT, 25 Oct 2000 forecast in Figure 2, we have four tuples as shown below: 
(00, 12, 16, 202.5, none) 
(06, 16, 20, 112.5, none) 
(15, 16, 20, 45, none) 
(00, 24, 28, 337.5, none) 
For this study, we have used one version of our prototype forecast text generator to produce the forecast in the conceptual form starting from the NWP model data in our parallel corpus. Using our 
741
parser we have generated the conceptual forecast from the human-written forecast text for the same data. We have compared both the forecasts in their conceptual forms and collected all those cases in the corpus where the machine-generated forecast is substantially different from the human written forecast. We have externally controlled the matching process by defining different matching criteria. In a majority of cases (about 85%) there was a good match between the two. Upon careful observation we have realised that these cases correspond to days when there was nothing complex happening with the weather, they involve routine forecasting. Though low in number, a majority of the other cases (15%) represent days when more complex weather conditions existed. We believe that the human forecasters used an overview of the weather while writing these (15%) texts as described in (Sripada et al 2001b). 
4 Conclusion 
We have described how we acquired knowledge from a parallel text-data corpus of naturally occurring weather forecasts for building a prototype forecast text generator, SUMTIME-MOUSAM. The knowledge acquired from the corpus was useful in a few cases but in many other cases required clarifications with expert meteorologists. Particularly, unlike what we expected, the corpus gave ambiguous results when we tried to find what words (or phrases) mean and what rules govern their expression in forecast texts. Among other things, this might partly be because the human forecasters who wrote these texts were writing many forecasts in a day under time pressure and therefore failed to maintain consistency. Despite these problems we found the parallel corpus useful and plan to explore it further. Particularly, we plan to try alignment techniques from the machine translation community, which might help us in obtaining better results. 
Acknowledgements 
Many thanks to our collaborators at WNI/Oceanroutes especially Ian Davy and Dave Selway; this work would not be possible without them! This project is supported by the UK Engineering and Physical Sciences Research Council (EPSRC), under grant GR/M76881. 
References 
Barzilay R, McKeown K 2001 Extracting paraphrases from a parallel corpus. In Proceedings of the 39th Meeting of the Association for Computational Linguistics (ACL-01), pages 50-57. 
Brown P, Cocke J, Della Pietra S, Della Pietra V, Jelinek F, Lafferty J, Merver R, and Roossin P 1990 A Statistical Approach to Machine Translation. Computational Linguistics 16:79-85. 
Cohen W 1995 Fast Effective Rule Induction. San Francisco, Morgan Kaufmann. 
Keogh E, Chu S, Hart D, Pazzani M 2001 An Online Algorithm for Segmenting Time Series. In Proceedings of IEEE International Conference on Data Mining. pp 289-296. 
Och F, Ney H 2000 A comparison of alignment models for statistical machine translation. In Proceedings of the 18th International Conference on Computational Linguistics (COLING-2000), pages 1086-1090. 
Quinlan R 1993 C4.5: Programs for Machine Learning. San Mateo, CA, Morgan Kaufmann. 
Reiter E, Dale R 2000 Building Natural Language Generation Systems. Cambridge, Cambridge University Press. 
Reiter E, Sripada S 2002 Should Corpora Texts be Gold Standards for NLG? In Proceedings of INLG-2002, pages 97-104. 
Scott A, Clayton J, Gibson E 1991 A Practical Guide to Knowledge Acquisition. Addison-Wesley. 
Sripada S, Reiter E, Hunter J, Yu J 2001a SUMTIME: Observations from KA for Weather Domain. Technical Report AUCS/TR0102, Department of Computing Science, University of Aberdeen. 
Sripada S, Reiter E, Hunter J, Yu J 2001b A Two-stage Model for Content Determination. In Proceedings of the 8th ACL-EWNLG'2001, pp3-10. 
742
Sripada S, Reiter E, Hunter J, Yu J 2002a SUMTIME-METEO: Parallel Corpus of Naturally Occurring Forecast Texts and Weather Data. Technical Report AUCS/TR0201, Department of Computing Science, University of Aberdeen. 
Sripada S, Reiter E, Hunter J, Yu J 2002b Segmenting Time Series for Weather Forecasting. In: Macintosh, A., Ellis, R. and Coenen, F. (ed) Proceedings of ES2002, pp. 193-206. 
743

A proposed mathematical theory explaining the sequence of grammatical categories in sentence structure. 
Asa M. Stepak, Independent Researcher 
(Cum Laude graduate of New York University) 
1. Introduction 
In this paper I propose a Mathematical Theory and Mathematical treatment of linguistic phenomenon. I refer to this theory as mathematical since it is derived from Information Theory which is a Mathematical Theory. But since the theory I propose concerns linguistic phenomenon and can only be validated or refuted based upon observable linguistic phenomenon, the theory might be better referred to as a Physical Linguistic Theory. Also, many of my suppositions and premises which I propose as a basis for the Theory are linguistic or cognitive in nature rather than mathematical. However, before I proceed to detail the actual Theory and how I have chosen to implement the Theory, I think it would be apropos if I first provide some of the broader theoretical background and differing theoretical perspectives. And, of course, since I will be proposing a new theoretical perspective, much of my effort will be directed to advancing my own theoretical perspective. 
In this section I will describe three perspectives of Natural Language Acquisition, (NLA), the first being the Chomskyan perspective based upon the notion of ‘Poverty of the Stimulus', the second a Learning Theory perspective, and a third perspective which is the one I propose based upon Information Theory principles. The Chomskyan and Learning Theory perspectives can be regarded as non-contradictory and consistent with one another. However, the Information Theory perspective entails an entirely different framework and does contradict the Chomskyan and Learning Theory perspectives. Based upon Information Theory principles it is my contention that the Chomskyan and Learning Theory proposals are inadequate in many respects. 
Chomsky's explanation of Natural Language Acquisition is based on the notion of ‘Poverty of the Stimulus'. From this perspective, since sentences can acquire infinite complex characteristics, the child needs to learn a set of rules governing the language rather than a set of patterns matching the language. The child's learning of these rules is guided internally since there are not sufficient data provided the child for language acquisition to occur through an inductive reasoning. Chomsky proposes we are born with an innate set of rules described as a ‘Language Acquisition Device',LAD, that limits the choices we have to make in determining the rules of a particular language. The set of internalized rules we are born with comprise the Universal Grammar, UG,--common limiting constraints found in all languages. The innate Universal Grammar is, also, referred to as the ‘Principles and Parameters Theory’ which states we are born with innate principles governing our use of language that for each language can be varied within finite parameter ranges. However, the shortcoming of the Chomsky perspective is that there is no attempt to provide an internal independent validity for the innate principles or universal grammar other than what is observed externally in all languages. Furthermore, Chomsky makes no effort to interconnect the many different features and aspects of the Universal Grammar by a common overriding principle. Thus, many of the principles comprising the Universal Grammar often appear disconnected and incidental, lacking a common cognitive or neuro-physiological justification. In fact, Chomsky points to many peculiarities and anomalies in language that seem to defy even a grammatical explanation and, therefore, presupposes language acquisition requires an innate LAD that cannot be fully described1. Chomsky, therefore, portrays the LAD as a ‘Black Box’ comprising of unknown underlying principles and serving as a common repository for all the known universal features of language. Therefore, if, in fact, there are universal features in language that have not yet been identified, these features would not be included in our description of the principle constraints of the LAD model. The LAD mirrors what is known of the universality of language through linguistic observation, what is not known or yet to be discovered is 
1 A classic example of idiosyncratic sentence structure suggested by James Garson would be found in the comparison of the following two sentences, “The dog that I saw has rabies.” compare with “The dog that bit me has rabies.” In the former, ‘that’ can be omitted without affecting the competency of the sentence whereas, in the latter, ‘that’ cannot be omitted. Chomsky claims these idiosyncratic phenomenon defy a linguistic explanation and can only be explained by a LAD serving as a ‘Black Box'. 
744
unaccounted for by the LAD model. Furthermore, the proposal of the LAD model excludes any factors that might be known about underlying cognitive neuro-physiological implementation mechanisms. The LAD's construction is purely based upon observations of language. The Chomsky LAD is intended to serve as a model for directing neuro-physiologists and cognitive scientists to make discoveries that would be consistent with an underlying implementation of the LAD but, the LAD, in its own right, is regarded inviolate of any counteropposing neuro-physiological findings. 
As stated above, Chomsky makes no effort to find an underlying principle that might interconnect and explain why the universal features of language are what they are. And, in fact, in so doing many of the hand picked Universal Grammar constraints that have been collected over the years based upon observation of all languages can be disconnected and unrelated to one another. In fact, Chomsky makes no requirement that they be related or connected in some way. Chomsky makes no effort to take a bottom-up approach that would draw upon principles that would provide independent justification and be able to independently predict the collection of grammar constraints attributed to a LAD and, in this regard, Chomsky treats the LAD as a ‘Black Box'. His justification for so doing is based upon his drawing upon a wide array of peculiarities in language that appear to defy any known linguistic explanation but, nevertheless, comprise of an integral part of language that is learned in the natural language acquisition phase. The Chomsky perspective, in this sense, is highly subjective and could be regarded as unproven or within the gamut of conceptual formulations that cannot be proven. 
A second approach to explaining Natural Language Acquisition is the Learning Theory or Computation Theory approach. Here, the basic argument is that since regular languages comprise of a subset of all possible languages to be learned and regular languages, unlike the complement of regular languages, can be either finite or infinite, there is no theoretical or practical way to decide through an inductive reasoning that is required for Natural Language Acquisition whether a language being learned was an infinite or finite language. Thus, theoretically, there would be a need for some internalized constraints represented as a Universal Grammar, UG, that would limit the search space. A review of this perspective is provided by Nowak2. However, it appears that Nowak's review may have oversimplified the required task in Natural Language Acquisition since learning theory makes no requirement that language be constrained in order to contain viable information that can be effectively communicated. From the learning theory perspective, a language comprising of as little as one or two words or even ten words would be amongst the multitude of possible languages a subject would have to choose from in NLA. Clearly, such abbreviated languages would not fulfill the information containing constraints required of a language. Similarly, languages comprising of one or two element characters would result in lengthy word strings that would not satisfy efficiency constraints of a language. Such languages also would be excluded from the effectual search space due to efficiency constraints. In NLA, the choice of choosing either a finite or infinite language is not at issue—the issue is choosing only a viable language that efficiently communicates information. Many languages, both finite and infinite, are automatically excluded because they do not fulfill this latter criteria. The fact that natural language can be described as belonging to the infinite class of languages based upon its infinite recursive properties is really irrelevant to the choices that would have to be made during the natural language learning phase. Thus, the search space would be constrained in its own right and the effectual search space would be substantially reduced and finite since many of the possible choices of a language simply do not satisfy the information theory criteria of serving as a viable platform for communicating information. 
Thus, the learning theory approach proves nothing and certainly does not prove the necessity for the existence of an internalized UG or LAD. The Learning and Computation Theory approach overlook an important feature of the search space—that the search space is finite in its own right due to information theory constraints that requires a language medium to convey viable information efficiently for a specific information source. 
The third approach to Natural Language Learning is the one I have proposed based upon Information Theory principles. In taking an information theory approach it becomes rather meaningless to think in terms of whether the search space is limited in its own right or by an external processing such as a LAD. Based upon Information Theory principles, the search space and the processing of the search space are both constrained and it becomes futile to attempt to make a distinction between the two. Information Theory principles require that a optimal homeostasis or equilibrium be achieved between the search space and the processing of the search space and places constraints on both simultaneously. From this vantage 
2 Nowak M 2002 Computational and evolutionary aspects of language. Nature Vol 417:611-617 
745
point, the mind as the embodiment of the brain takes on a whole new dimension since the information search space includes both and serves as a viable alternative to the more traditional notion of embodiment in a biological structure. 
From a Information Theory perspective there is no need to model a theory based upon an internal versus external dichotomy. Statistical Information Theory accounts for simultaneous events occurring within the organism and without and provides the schema that enables the information search space to be effectively used. The adjustments and modifications that are needed do not parallel the artificial boundary line established by the biological boundary of internal versus external. The objective is to use the information space as effectively as possible. Here the programmer is evolution as those organisms that do utilize the information search space most effectively are more certain to survive. But information theory is based purely on a statistical notion and is not impeded or detoured by the biological notion of embodiment. Thus, the contours for an effective information processing are not solely determined by an internalized processor. There is an also externalized refinement of the information to be processed and it becomes difficult to draw the line where externalization ends and internalization begins. They both overlap to large extents and may, in fact, be indistinguishable. In Information Theory terms, we don't think in terms of a LAD or an internalized Universal Grammar. We think in terms of encoders and decoders, channels, entropy of information source, information transmission and efficient coding. And the terminology that is used in Information Theory is ubiquitous, it can be applied to any communication of information or physical commodity and no less can it be applied to language. In language, the language is the code, the pharyngeal, oral cavity, tongue and lips produce the transmission signal, somewhere in the interface between neuronal stimulators and the pharyngeal and oral cavities lies the encoder, the air molecules are the channel, somewhere in the interface between the eardrum and neuronal receptors lies the decoder. The information source is our external and internal environments which embodies us. The nice thing about Information Theory, it is based upon mathematical statistical relationships that are linear. Thus, when applied to linguistics it can provide a format for obtaining simple, and quick solutions to otherwise thought to be complex issues in linguistics that, heretofore, one could only make an attempt to understand heuristically. And the concept of the LAD is precisely one such issue that might be better understood or put to rest based upon an Information Theory approach. 
Now, having embarked on an Information Theory approach and having described briefly language in Information Theory terms, perhaps, we can now do some of things that Chomsky has chosen not to do, that is, provide an independent, underlying justification for the observable phenomenon we refer to as Universal Grammar constraints that appear disconnected otherwise. And, in so doing, what has otherwise been referred to as a LAD as a ‘Black Box’ might otherwise be better referred to as a fundamental underlying principle we have evolved to implement language communication effectively. And we might suppose that such a fundamental principle based upon Information Theory, also, represents a common pathway to our other faculties such as walking, vision, etc. We can still think in terms of a highly specialized area of the brain devoted to language use, perhaps, based upon a Hopfield connectionist model, but, do not have to think of this specialized area relying on mystifying ‘Black Box’ principles separate and distinct from other cognitive functions. The LAD, in fact, is demystified and the ‘Black Box’ notion unraveled. 
2. An Information Theory Model for language 
In artificial coding there are two coding strategies to consider. 1. To minimize the entropy of the information source as much as possible. 2. To rely on coding that is the most efficient in transmitting a given information source. These two overriding principles can be seen as also operating on a natural language coding level. 
What is to be encoded in a natural language channel is our cognitive awareness that we choose for any given time duration to communicate rather than contemplate introspectively. Our cognitive awareness is the information source with a characteristic entropy. However, in achieving 1., above, the effective entropy can be reduced if portions of this information source do not have to be searched or their probability of occurrence are reduced to zero. Grammatical coding in language or what is more commonly referred to as syntax accomplishes this objective. The onset of one particular grammatical category in a sentence affects the probability of occurrence of the grammatical category that is to follow. For instance, in the case of the determiner, a noun has an extremely high probability of occurring immediately afterwards and, thus, other probabilities of occurrence of other grammatical categories are reduced to zero or a minimum. Grammatical coding in sentence structure, thus, serves to reduce the overall entropy of the given information source at any given point in time allowing for a more simplistic and efficient encoding. In achieving 2. ,above, an 
746
Information Theory Huffman algorithm is utilized. This is evident by the characteristic frequent word lists found in all languages. There are a predetermined small subset of words comprising of most common words in all languages that comprise of the majority of words utilized in the language. For English, roughly the 300 most frequently used words comprise of 65% of the words utilized in the language. Assuming there are roughly 300,000 available lexemes in a language, in effect, .1% of available words comprise of 65% of the words used. Furthermore, the most frequently used words tend to be the simplest encoded words of a language as is evident by their shorter lengths, fewer number of phonetic segments they comprise of and their shorter temporal duration. This observation is consistent with the implementation of a Huffman algorithm that determines the most efficient coding scheme for a specific n-ary information source in artificial coding. The Huffman algorithm assigns to information source items of high probability of occurrence the simplest codes resulting in a lowest overall average word length for a code. 
Zelig Harris was one of the first proponents of Information constraint implementation in natural language3. His view was that phonemic choices for words were constrained by the underlying requirement that words carry information. Harris illustrated this by the characteristic spikes in permissible phoneme choices evident at word boundaries, thus, verifying that the selection of a phoneme within a word or morpheme was constrained when compared to word boundaries. Similar constraints are revealed when examining a language's most frequently used words or MCW's. Here it is evident that the most frequently used words are also the simplest coded words consistent with Huffman but a substantial portion of the available phonetic combinations or permutations are never used for simplistic encoding. Thus, unlike artificial coding that relies on Huffman where all available combinations of simplistic codes are relied upon, there appears to be a constraint imposed on natural language that precludes the use of all phonetic combinations. As was independently suggested by Harris based upon spiking data, this additional constraint is an information carrying constraint or semantic constraint. Such constraints, also, would have to include an encoding for word boundaries so that in oral communication individual words could be distinguished without pauses that would otherwise markedly slow down oral communication and likely render it ineffectual. 
The two strategies described above in making language more code efficient and easier to utilize relative to an information source are, also, the strategies that determine the linear sequencing of grammatical categories in sentence structure. Sentences can be infinitely long and complex or extremely short or fragmental. But in terms of what grammatical categories occurs most frequently in a language, here too, the strategy would be to front end the most frequent categories and back end the less frequent categories in a phrasal structure. The same principle applies to constituents of categories, the most frequent being in the head or front and the less frequent or non-frequent being placed towards the tail or end of the phrase. This single unifying principle governs the sequencing of grammatical units in a particular language which, in turn, can be ascertained by the frequency of the grammatical units and constituents thereof appearing in large, representative corpuses of the language. To the extent the frequencies would be different in corpuses of different languages, the sequencing would be different to reflect the differences in frequencies. 
Now, what I just described for you above would seem to follow an Information Theory approach as applied to natural language but there is a subtle difference when compared to other artificial, man made forms of communication. In artificial communication, discrete events take place based upon a pre-coding of the information source. In others words, a telephone conversation that has to be conveyed over transmission lines is already pre-structured and coded at the source which is the telephone. Here the objective is to convey the message that has already been coded at the source. In natural language, however, there appears to be an ongoing processing or coding of the source message. In other words, the source is not completely coded or structured but exists in a partial gestation state. This results in codes not being transmitted merely on a predetermined discrete time event schedules, but allows for the most frequent coded words to take precedence in transmission over less frequent coded words to the extent other coding constraints are not violated. The advantage to this approach is obvious, it allows for the sequencing of grammatical units and their constituents based upon frequency precedence that would not be achievable otherwise, thus, promoting more efficient encoding. Thus, there is an added means by which efficiency in coding is achieved in natural language not available in artificial coding. Not only are information source data of high probability of occurrence attributed simple codes, but ,also, the high frequency simple codes are front ended in the coding. This exemplifies what I stated in section 2, that the natural language information source and the processing of the same almost become indistinguishable and much of the 
3 Harris Z, 1988 Language and information. New York, Columbia University Press 
747
refinement and processing of the information source occur simultaneously or in tandem. Thus it would be incorrect to describe the natural language phenomenon as determined by a static repository of rules and principles such as a LAD as Chomsky does. Language is a dynamic process that is governed by both internal and external processes that are in equilibrium with one another based upon the statistical properties of Information Theory applied to a representative corpus. 
3. A mathematical analysis of most common words (MCW) 
A simple probability analysis shows that the selection of codes for language is not random but subject to several constraints. The first constraint is that the most frequent codes be simplest in form so to minimize the average word length of the entire coding. A second constraint limiting the first is that the coding be such that it carries meaning and includes coding that distinguishes the boundaries between words so that oral communication can proceed at an effectual rate. The meaning constraint or semantic constraint comes in two types, iconic and non-iconic mental lexicon based. The retrieval of words belonging to the first type is autonomic, not requiring an underlying conceptual recognition of what is being communicated. Mental lexicon based words, on the other hand, do require some sort of on-line conceptual cognition corresponding to the words being used. It is words of the first type that serve as iconic markers in sentences that determine the grammatical competency of sentences since the iconic words are the MCW's of a language that, as described in the previous section, are coded to be frontloaded in phrases based on their frequency precedence to render the overall coding more efficient. The iconicism attributed to MCW's is verified by the substantial constraints placed on the coding that goes far beyond what would be needed for an arbitrary semantic coding of words. Mental lexicon words do not serve as iconic markers for grammatical purposes, (albeit, they may retain a remnant of semantic iconicity) and, therefore, are typically backloaded in phrases. 
Using Trager Smith's system of 32 phonetic segments that was used in the A. Hood Roberts study 4 and adding one null element so we have 33 possible choices, the number of possible combinations comprising of two phonetic segments or less is 33^2 or 1089. Based upon the Roberts study, only 50 words in Horn's 10,000 most frequent word list were found to have only two phonetic segments or less. These 50 words, however, accounted for approximately 33% of the words used in American English and, thus, are in the class of MCW's. Based upon a model that attempts to account for semantic constraints, an alternating pattern of vowel and consonant would be a better approximation of the available permutations for a word containing two phonetic segments. Such a model yields 8 * 25 * 2 = 400 possible choices. The factor of 2 is used since two-phonetic segment words exist in two canonical forms, VC and CV as was determined in the Roberts study. Thus, the conclusion one must reach is that the 350 additional available permutations that are not used are unsuitable since they do not satisfy semantic constraints or word boundary constraints. But if semantic coding is arbitrary and non-iconic then there should be no additional semantic constraint imposed in choosing additional available two or less phonetic segments permutations other than when it results in a one-phonetic segment word. In one –phonetic segment words, identical vowels appearing in the first or second position would represent the same code and one would have to be eliminated due to redundancy. The same would hold true for consonants. But, here, the most that would have to be eliminated would be eight potential choices for vowels and 24 for consonants. That still leaves 318 available unused choices. The limiting factor would, thus, have to be considered a word boundary constraint. But it is difficult to imagine that word boundary constraints alone would eliminate the use of the unused 318 available permutations. My conclusion, therefore, is that word boundary constraints serve only as a partial constraint and alone do not account for the 318 additional available permutations not being used. The additional constraint that accounts for additional available permutations not being used is an iconic semantic constraint that requires meaning be nestled in some aspect of articulation that metaphorically or otherwise relates to meaning in our internal or external environments5. The 318 permutations of two phonetic segments that are not used, other than those that are eliminated due to word boundary constraints, cannot satisfy the requirement of iconism and are not consistent with the Oral Metaphor Construct6 7, 
4 Roberts A. Hood 1965 A statistical linguistic analysis of American English. Netherlands, Mouton and Co. 
5 Stepak, A. 2002 Rationalism versus Empiricism: A new perspective. A work in progress. (filed in the copyright office, Washington D.C.) 
6 Stepak, A 2002 Oral Metaphor Construct. In Proceedings SSGRR Summer 2002, http://cogprints.ecs.soton.ac.uk/archive/00002294/ 
7 Stepak, A 2003 Oral Metaphor Construct: An information theory and cognitive perspective. A work in progress. (filed in the copyright office, Washington D.C.) 
748
OMC, of the language which serves as a characteristic iconic imprint for a language. Only having utilized 50 two-phonetic segment permutations even though more are available , language coding procedes to utilizes three phonetic segments. This fact alone would seem to contradict a Huffman algorithm hypothesis applied to natural language which utilizes the simplest available coding for the words of highest probability of occurrence. However, a Huffman application to natural language would not be refuted if we consider the parameter of simplicity not to be the character length of the code or the number of phonemes but rather the codes iconicity. In so doing, not choosing the additional 318 two-phonetic segment permutations due to iconic constraints would be consistent with a Huffman algorithm application to natural language. 
We can also do a statistical calculation of the available three-phonemic segment combinations and permutations in relation to the number that are actually used based upon the Roberts study and we will reach similar conclusions. Stated simply, there are a high number of three-phonetic segment permutations not utilized that cannot be merely explained by word boundary constraints or non-iconic semantic constraints. Eliminating the null character in this treatment since we want the exact number of three-phonetic segment combination or permutations and, not less than three, the number of three-phonetic combinations is 32^3 which equals 32,768 available combinations. Based upon the Roberts study, only 552 words out of a most frequent word list of 10,000 comprise of three phonemic segments. These three phonetic segment words, nevertheless, had an overall frequency of usage of approximately 26.5 %. They are, thus, included in the class of MCW's. Grouping semi-vowels with consonants, the canonical forms of these words based upon the Roberts study are CVC, or VCC. However, the frequency of the VCC canonical form is so low it can be ignored in this mathematical treatment. Therefore, based upon the predominating canonical form of CVC we can approximate the number of permutations to be 24 * 8 * 24 = 4608. Here again, the number of available permutations is exceedingly much higher than the 520 actual number of three-phonetic segment words that are used based upon the CVC canonical form. So great a disparity can only be explained by a substantial constraint that can only originate from a semantic iconic constraint where semantic coding is not arbitrary, but, intrinsically derived by some aspect of articulation that conveys semantic significance. 
Before ending this section and moving on to the next, I should state that in the above treatment I did not make reference to bound MCW's which the Robert's study did not consider. It is my view that common inflections and affixes of words, based upon there frequency of occurrence, should be considered with the class of MCW's. A basic tenet of my work on grammatical sequencing is that bound MCW's share the same features and serve the same functions as non-bound MCW's. Both serve as iconic markers in sentence structure. However, in American English, the bound MCW's comprise of a relatively small group, perhaps, no more than 30 such bound MCW's are present and the majority fall into the three-phonetic segment class. Thus, the mathematical conclusions reached in this section still would remain valid even having not considered the bound MCW's in the mathematical treatment. 
4. From a cognitive perspective 
A term often used in describing phenomenon from a cognitive perspective is the term ‘object'. However, in my review of the literature on this subject I have failed to find anyone attempting to provide an exact definition of the term ‘object'. To eliminate the plausibility of this term being misconstrued by those not having exposure to the cognitive sciences, I propose the following definition. ‘An object represents a foci of our attention that can be described as a unit distinguishable from its immediate environment'. In so far as discrete objects are the means by which we gain recognition of our environment and interact with the environment, our perception of objects in a non-language sense is also likely to be governed by information theory principles. In other words, we perceive and interact with objects in a discrete time event sense even though the true physical reality of our environments might be better portrayed as continuous, and non-discrete. Thus, herein lies the connection between what we perceive cognitively and that which we communicate linguisticly since language use, likewise, is best described as a discrete time event phenomenon. 
Objects can be in the form of words in which case we refer to them as word objects or they can be entities we interact with in a non-language sense in which case they would be referred to as cognitive objects. The number of objects we can possibly conceive of and routinely perceive is a very high number but it is not infinite. There are objects that we have not previously perceived that could be brought to our attention through instructive enlightenment. It could be said that enlightenment is a process by which the number of objects we routinely perceive is modified in some way. Objects can be reduced or subdivided and considered as new objects or objects of different classes might be grouped together as distinctly new 
749
objects. Objects can be tangible or conceptual, they can be in our sensory fields or beyond our sensory fields. 
The point to be made is that our conception of cognitive objects to a large extent dictates our information processing and representation of those objects in our language medium. Perception of cognitive objects categories determines the relative importance of corresponding word object categories and their frequencies in a language and their ultimate grammatical sequencing. To the extent that cognitive perceptions would tend to differ amongst different populations, so to would their corresponding languages. It is a basic tenet of this paper that a mathematical analysis of large representative corpuses of a language in terms of the relative sizes of word categories is a good measure of the relative size of classes of cognitive objects in the cognitive field of the users of the language. The point to be made is there are many perspectives and vantages points from which to view a cognitive field and these differences ultimately are reflected in different frequency precedent values for corresponding word categories in different languages that results in a unique sequencing of grammatical units in different languages. 
However, I do not advocate a pure cognitive perspective since it is my view that what I have described above is ultimately constrained by Information Theory considerations. In other words, it may be useful to attempt to construct cognitive schema to gain a semantic understanding of intra-sentence word dependencies in terms of subject, object etc. but once having been established grammatical sequencing becomes primarily an information theory phenomenon. The cognitive side of grammatical sequencing resides only in the cognitive discrimination of iconic markers and their proper precedence in a sentence structure based upon frequency values, which, in turn, is determined by the application of Information Theory principles to the cognitive field of the users of the language. 
5. The proposed theory 
In the previous sections I have attempted to provide the reader some of the broader theoretical background so I would not have to devote as much time and effort explaining the more detailed axioms and premises my proposed theory is based upon. Every theory requires axioms and premises. If the theory is able to predict phenomenon, then it can be assumed the axioms and premises upon which it is based are generally valid. However, that does not preclude the design of experimentation that could potentially test the validity of individual premises upon which a theory is constructed or, for that matter, that premises of a theory might be replaced with better premises. Based upon the limited data I have explored which involve only comparisons with English with some of the other Indo-European languages, my proposed theory can only be considered a working hypothesis at this stage subject to further refinement and supplementation. In this section, I will attempt to describe in a summary fashion all the tenets, premises, and perspectives which I have proposed as relevant to the formulation of my proposed theory and then state the Theory in its current form. I should note that the premises that I list may not represent a complete list of premises that should have been listed. Also, I wish to make the reader mindful of the fact that it is my belief that the actual Theory I propose may require further refinement, supplementation, and, perhaps, recapitulation to some extent as more data is explored from the diverse array of languages and dialects but, still, I firmly believe that my basic working hypothesis is valid. That working hypothesis is based upon a Information Theory perspective which claims that the different sequencing of grammatical units across different languages is determined by a fundamental unifying dynamic principle. In the next section, I will, then, apply my theory to both simple and complex grammatical constructions and will provide justification for different grammatical constructions prevailing in English and French. Some of the premises detailed below have already been discussed in previous sections of this paper. However, in a brief paper such as this, it is not possible to devote as much detailed discussion and background information as I would have liked to all the premises which follow. Some of this void can be filled by reference to my other works described in footnotes 5,6,7 on page 5. Other portions of this void will require my further indulgence in addressing the topics more specifically through additional writings. 
My Proposal: 
1. In natural language learning and usage, the subject relies on iconic markers comprising of the set of MCW's of a language that enable the user to distinguish a competent sentence. The iconic markers need to be sequenced in a sentence in a prescribed format based upon Information Theory principles for the sentence to be competent. 
2. In natural language acquisition, the subject not only learns the set of words comprising of the iconic markers in a language, but, also, learns the degree of iconism of the markers. Competence of sentence structure is achieved by the subject's intuitive ability to front end strong iconic markers and backend weak 
750
iconic markers in a sentence structure or phrase. In learning the degree of iconicism of iconic markers, the subject learns the relative precedence of iconic markers in a sentence structure or phrase. 
3. MCW's are iconic and serve as iconic markers. 
4. MCW's are cognated differently than uncommon words. Thus, the subject can distinguish between MCW's and uncommon words on a cognitive level. Iconic or MCW's are autonomic words, and their use does not require a concurrent semantic cognition. Uncommon words or mental lexicon based words, when used, due require to some extent a concurrent semantic cognition. 
5. All MCW's are iconic (with some few exceptions such as proper nouns and depending on the corpus). However, the iconicity of MCW's might be masked by a surface structure hardwiring that has evolved over long evolutionary epochs. A word will not become a MCW unless it is iconic. 
6. In written communication, MCW's can be represented as being bound to uncommon words. In oral communication, the distinction is less clear. Still, bound MCW's serve the same purpose and function as unbound MCW's in serving as iconic markers. Bound MCW's usually come in the form of common inflections and affixes. 
7. The information theory based strategy of language is to front end grammatical categories and constituents of high frequency and to back end grammatical categories and constituents of low frequency in a phrase or sentence. The subject accomplishes this objective by relying on iconic markers. Each phrase or grammatical unit has an associated iconic marker from which the frequency value of the constituents of the phrase are determined. 
8. Strong stresses on individual syllables and words as in the English language can attribute to a grammatical category iconicism that would not otherwise reside in MCW iconic markers alone. Examples are the adjective, adverb, and abstract noun. Corresponding strong stresses are not apparent in French or Spanish explaining the differing sequencing of grammatical units involving these categories. 
Theorem: 
Relying on large representative corpuses of a language, one can establish the relative size of word categories. Furthermore, by relying on the frequency of most common words and bound most common words, one can establish a frequency of occurrence of an iconic marker. Multiplying the frequency of occurrence times the size of the category of which the word is a constituent will yield positive frequency values for iconic markers and normalize non-iconic words to a value of 1. The size of the largest word category is arbitrarily assigned the value of K which is a high positive number but not infinite. The sizes of the other smaller categories are designated values proportionate to their relative size to the largest category in terms of K, i.e. 1/2K, 1/3K etc. To determine the overall frequency precedence of grammatical units within a sentence, rather, than merely within a phrase within the sentence, it is necessary to determine the function-argument word dependencies8 based upon a semantic analysis. Once established, in all instances where words are arguments of other words serving as functions, the upper limit of the class size of the argument is taken to be the class size of the word serving as the function. Determiners , prepositions, general determiners are not considered as having a fixed class size and are assigned the class size of the function to which they serve as argument, however, are considered as having a calculable frequency of occurrence based upon their frequency ranking among MCW's. Being that pronouns can serve as primary functions as do nouns, they are attributed as having the same class size of nouns. The above treatment does not exclude other factors such as phonetic stress that can attribute to a non-iconic category constituent a high frequency value. However, the frequency value attributed to stress cannot be independently approximated using a representative corpus and can only be fitted to the grammatical data. 
Based upon the methodology described above the following three principles govern the sequencing of grammatical units in a phrase or sentence. 
1. The frequency value of linguistic units relative to the frequency values of other linguistic unit determines the linguistic unit's relative precedence in a phrase or sentence. The higher the frequency value, the higher the precedence. 
8 Zelig Harris used word dependencies in his study of language except that he referred to word dependencies as operator-argument dependencies. Harris Z 1988 Language and information, New York, Columbia University Press. 
751
2. It is possible for frequency values to be too high immediately prior to a non-iconic constituent so that the non-iconic constituent is masked due to iconic persistence. This may occur when a constituent has two origins of iconism such as in the adverb, one being stress and the other being the iconic suffix ‘ly’ . 
3. Principle 2 above can serve to constrain principle 1 above. When such constraints occur, a linguistic unit of lower frequency value can proceed one of higher value. 
6. Implementation and Examples 
Using the above suggested methodology of the theorem, anyone familiar with corpuses could attempt to do their own mathematical computation of frequency values to justify grammatical sequence. There is still some guess work in approximation involved as to how one best goes about calculating frequency values. And I do not expect anyone's own calculation to match exactly my own. The important thing is that the calculations in some fundamental way explains the sequencing of grammatical units within language. And at this early stage there is still some fitting of the data that might have to be accomplished through refinement of the technique. For instance, there is no independent method for assigning frequency value attributed from phonetic stress. Here the assignment would solely depend on using a value that would explain the grammatical sequencing and hope that the same value works under other scenarios. The methodology still requires further testing across differing languages and dialects utilizing different corpuses. The results that follow were based on a mathematical analysis of the LOB corpuses as computed by Johansson and Hofland9. 
Based upon the Johansson and Hofland study, my approximation for word class size and frequency of occurrence are as follows: 
Class Size: Noun Objects = K; Verb Objects = K/10; Adj. Objects = K/3; Adv Objects = K/25 
Avg. Frequency of Occurrence: Nouns = 1/K; Verbs = 10/K; Adj = 1/10 from phonetic stress, fitting of the data, Adv.= 1 from ‘ly’ suffix and phonetic stress, fitting of the data; Prep = 1/40; Det(a,the) = 1/4; Gen. Det. = 1/6; Pronouns = 1/22 
I should reiterate, that the above frequency of occurrence values are approximations that serve to fit the data but are, also, reasonably deduced calculations based upon Johansson and Hofland's mathematical analysis of the LOB corpus. By reasonable, I mean that the values fit relative to one another but not necessarily reflect true absolute values. Also, I should indicate that in the following examples where I analyze French constructions, I am assuming the word class sizes determined in the LOB would also apply to the French language. This may not be an entirely correct assumption to make and any analysis of French based upon an English corpus would have to be considered rather limited in scope. 
Examples: 
1. Det noun 
arg1 f1 
Det arg1: freq. 1, size K(size of function) = fv K/4 
Noun f1: freq. 1/K, size K = fv 1 ( noun is non-iconic) 
Therefore Det precedes Noun. Principle 1 
In French, all nouns require to be fronted by a determiner, unlike English. This difference is due to the difference in amplication of stress in the two languages. In English, stresses carry sufficient amplitude for plural nouns and abstract nouns to stand alone without Determiners. In English, stress attributes to stand alone nouns iconicism. In French, stress is not of a sufficient amplitude to attribute to nouns iconicism and thus, all French nouns require the determiner. The same holds true for Spanish. 
2. Det noun verb adv adj noun. English, attributive adjective. 
arg1 f1 
arg2 f2 
Noun f1: freq 1/K, size K = fv 1 
Adj arg1, f2 : freq. 1/10, size K/3 = fv K/30 (fv is attributed from stress) 
Adv.arg2: freq. 1, size K/25 = fv K/25 ( fv is attributed from stress and suffix “ly”.) 
Principle 1 applies. 
3. Det noun verb noun adv. English 
f1 arg1 arg1 
Noun arg1: freq 1/K, size K = fv 1 (non-iconic) 
9 Johansson S, Hofland K 1989 Frequency analysis of English vocabulary and grammar based upon the LOB corpus, Vol.1&2, Oxford, Clarendon Press 
752
Adv.arg.1: freq 1, size K/25 =fv K/25 (doubly iconic due to stress, results in iconic persistence.) 
Principles 2 and 3 apply. 
4. Det noun verb adv noun. French 
f1 arg1 arg1 
Adv.arg1: freq 1/10, size K/25 = fv K/250(No stress, singly iconic in French, ment replaces ly) 
Noun arg1: freq 1/K, size K = fv 1 (non-iconic) 
Principle 1 applies. 
5. Det noun verb noun adj. French, attributive adjective. 
f1 arg1 
f2 arg2 
Noun: arg1, f2: freq 1/K, size K = fv 1 (non-iconic) 
Adj arg2: freq. K/3, size 3/K = fv 1 (non-iconic due to no stress) Explanation: Since there is minimal stress in French, noun and adj have virtually equal frequency values and adj can follow noun or vice of versa depending upon the relative frequency values of the two constituents based upon a representative corpus. 
Following are examples of more complex constructions. Here we do not have to rely on approximate values of frequency or class size but, rather, make an assessment based upon general iconic marker principles in explaining why certain constructions are forbidden or favored over others. The following examples, 1-4, are borrowed from Langacker10. Langacker claims that the forbidden constructions have a cognitive grammar explanation and that there is a continuum between the lexicon and grammar. However, Langacker's cognitive schema often appear to be self-justifying and lacking in formal constraints. It is my belief that the following interpretations I propose based upon information theory principles and iconic markers, alone, provide a more effective explanation to idiosyncratic sentence structure comparisons than do cognitive grammar schema. 
1. a. I baked her a cake. b.?I mowed her the lawn. 
b., above, is forbidden since lawn as a direct argument of mowed has a high frequency of occurrence. Mowed is almost idiomatic in that it takes only two arguments, lawn or grass. Thus, “I mowed the lawn for her” maximizes frequency values of the sentence and is required. In a. above, baked takes several potential arguments, any food item that can be baked. So cake does not have as a high a frequency of occurrence as a direct argument of bake as does lawn as a direct argument of mowed and the iconic marker her can be placed in between cake and baked. Now consider. 
2.a. I cleared the floor for Bill. b.? I cleared for Bill the floor. 
a. is an improvement over b. because in a. the definite article is front loaded. Remember that the definite article is a constitutent with a high frequency value. This serves to maximize the overall frequency value of the sentence. Now consider. 
3.a. I sent a walrus to Antarctica. b.?I sent Antarctica a walrus. 
Walrus as an argument of sent has a much higher frequency of occurrence then does Antarctica as a argument of sent. In fact, the latter probably represents a corpus frequency value of zero. Thus, a. is the favored construction. Now consider. 
4. a. I gave the fence a new coat of paint. b. ?I gave a new coat of paint to the fence. 
Here again we are dealing simply with an idiomatic phrase “a new coat of paint” that has very low frequency of occurrence as an argument of gave. Thus, it is backended in the sentence to allow “the fence” to take precedence that has a much higher frequency value due to the determiner “the”. 
5. Following is the idiosyncratic sentence structure described in footnote 1 on page 1. 
a. The dog that I saw has rabies. or The dog I saw has rabies. 
b. The dog that bit me has rabies. or ?The dog bit me has rabies. 
Explanation: “That” is an iconic marker that signals a consecutive action event in a sentence. When the consecutive action has embedded its own iconic marker,“that” can be dropped as in a. above where the following pronoun serves as the second iconic marker signaling the second action. In b. above, there is no second iconic marker to signal the second event and thus “that” cannot be dropped. Now consider this. 
“? The dog bit me that has rabies.” Here we have two iconic markers for two action events but the sentence is still forbidden. The reason is “that” has a high frequency value as a determiner and, thus, needs to be front loaded in the sentence to maximize the overall frequency value of the sentence. 
10 Langacker R, 2002 The cognitive basis of grammar, pp 14-15. Berlin, New York, Mouton de Gruyter 
753

The lexicographic use of corpora and computational tools for disambiguation 
Dr Petra Storjohann 
Institut für Deutsche Sprache Mannheim (IDS) 
1. Traditional vs. computerised lexicography 
Contemporary German lexicography is still mostly intuition-based and quite far from integrating corpus analysis into the process of dictionary making. Only a few projects are increasingly basing the compilation of their lexicographic structures on electronic corpora. The lexicographic practice remains a traditional “manual” art where compilers investigate numerous texts with the focus on spotting new words or detecting new senses of existing words.1 They gather material into card file systems which are still the most important instruments for German lexicographers (cf. Scholze-Stubenrecht 2001: 49, Hanks 1990: 31). Nowadays, the citation card files have been replaced by digitised searchable databases, but although such conventional systems have developed into elaborate and comprehensive information resources, they cannot compare to the volume and versatility of data in corpora. As Sinclair (1991: 4) comments: 
Especially in lexicography, there is a marked contrast between the data collected by computer and that collected by human readers exercising their judgment on what should or should not be selected for inclusion in a dictionary. 
The limits of such citation card file systems can be perceived when scrutinising any monolingual dictionary that works on such a basis. First, in some cases suitable examples cannot be found due to the paucity of evidence. Secondly, not every sense of a word can be encompassed in a lexicographer's brain. And finally, as corpus-oriented lexicographers would point out, deficits and inconsistencies arise from the lack of operational lexicographic algorithms. 
The revolutionary step to move away from intuition and employ a more objective approach through corpus-guided studies is a fairly recent development in German lexicography. The latest editions of monolingual dictionaries such as Duden (2000), however, have begun to combine traditional dictionary making with technological innovations, and it can be seen that corpora are being employed more effectively. Computer-based examinations of linguistic data are performed, particularly with regard to the selection of lexemes that are to be included in a new edition. On the one hand, it has been realised that the probability of finding suitable reference material is increased considerably through a large digital data collection and that specific data is accessible by defined search options. On the other hand, the chief asset of corpora is still seen in being a large pool of documentary evidence to support and verify the information gained by introspection (cf. Haß-Zumkehr 2002: 45). As far as the lexicographic process of describing lexemes semantically is concerned, corpora are not being used sufficiently. 
Whereas in other fields corpora have brought substantial change in working procedures and have led to the development of new approaches, German lexicography has not yet reached the stage where corpus-derived data delivers new approaches. Finding explanations that fit the evidence and adopting new approaches which enable us to extract new empirical linguistic knowledge still remain the 
754
principal objectives of empirical lexicography (cf. Sinclair 1991: 36). The potential of corpora and computer-based text analysing tools for lexicography needs to be fully recognised and to be more effectively employed in order to profitably supplement traditional lexicography. 
With the accessibility of comprehensive data and computational tools, lexical disambiguation2 should in particular be addressed from a new perspective. Currently, there is a demand for an disambiguation technique based on empirical and theoretical grounds. Although the issue of sense distinction has received much attention, most research is predominantly preoccupied in the field of machine translation, information retrieval, and hypertext navigation. Despite a long lexicographic tradition, a sound disambiguation practice for monolingual dictionaries has not been put forward. Today, such a procedure presupposes the existence of an elaborate linguistic disambiguation theory that is compatible with comprehensive empirical research. Its supporting instruments should be corpora, text analysing tools, and it also requires a good understanding of data interpreting. In the following part it is shown how such a need for an operational disambiguation algorithm with a rigorous empirical and theoretical basis can be met. 
2. Project Elexiko3 
At the Institut für Deutsche Sprache Mannheim (IDS) the project elexiko is developing a new corpus-driven lexicographic hypertext dictionary of the German lexicon and its present-day usage. Its aim is to construct a linguistic data warehouse where a diverse spectrum of readers can inexhaustibly explore the German language. By using a hypertext structure we are able to create a comprehensive dictionary with an extensive linking system for illustrating different types of language structures. As Hanks (1990: 35) argues: 
… there is a tendency for human lexicographers to focus on the way words are used to describe the world rather then on the way words interrelate with one another.” 
Its flexible search system will enable the reader to look for selective information according to individual needs. Elexiko shows a radical change from existing lexicographic conventions. The user will face different information with respect to quality, quantity and form of presentation. Our project follows a corpus-based approach, meaning we study linguistic instances on the basis of evidence found in corpus data. 
Elexiko has developed a disambiguation technique which is based on empirical and theoretical grounds. The lexicographic prerequisites of this disambiguation procedure are an elaborate theory, corpora, a data-processing software, and the linguistic competency of data interpreting. 
2.1 Elaborate disambiguation theory 
The principal criticism monolingual German dictionaries face today with respect to the issue of lexical disambiguation is the lack of a linguistic theory and formal grounding. Only a handful of dictionaries 
1 Also known as reading-and-marking method. 
2 In this paper disambiguation is not understood in terms of sense tagging/semantic tagging but refers to the lexicographic procedure of identifying and distinguishing the senses of a word for further semantic/syntactic description in a dictionary entry. 
3 Formerly named Wissen über Wörter, see http://www.ids-mannheim.de/wiw/ 
755
list meaning discriminating criteria; traditionally paradigmatic and syntagmatic patterns. However, a sound theoretical basis has not been developed so far. 
In order to avoid a one-sided theory, elexiko has striven to offer a multi-dimensional model which forms the basis for disambiguating content and function words. Crucially, we have constructed a systematic network of criteria consisting of semantic, syntactic, propositional, contextual, conceptual-referential and functional components combined in a complex cross-classification. The main components, however, are the semantic-syntactic and referential functions that correlate with the specific use of a search word within a collocational or sentential context. As Reichmann (1989) emphasises, the interaction of several complementary criteria illustrates differences in linguistic patterns of a word in different contexts and generates adequate results with regard to the correct identification of different functions. As pointed out by Sinclair (1987), the way word patterns relate to the use, function and meaning of individual words requires particular examination. Computer-processed data is hence classified according semantic, syntactic and referential functions with respect to the selected key word and its contexts. 
The advantage of describing a lexeme according to functional classes instead of traditional word classes is seen in the illustration of the connection between the semantic form and proposition or illocutionary potential of a communicative unit (cf. Strauß 1989). The classification is the linguistic method underlying the lexicographic exploration of sentential contexts and semantic relations to identify word senses, and it consists of the following levels: 
I Level of semantic classes of words 
This level is understood as the semantic classification of words into autosemantic or synsemantic classes. Words that contribute semantically to the proposition of a collocational or sentential unit are autosemantic/content words. Words that do not have a characteristic semantic contribution to a propositional and/or illocutionary meaning of a sentence, but are functional or syntactic constituents attached to a noun or verbal phrase are synsemantic/function words. 
II Level of sentential-semantic classes/syntactic classes and subclasses 
Content words in particular can be classified into propositional types according to their semantic characteristics in sentential contexts. Thus, autosemantic words are grouped into sentential-semantic classes which reflect propositional types and mainly consist of predicators4 (cf. von Polenz 1988). The class of predicators comprises for example event-denoting predicators, relation-denoting predicators, quality-denoting predicators, state-denoting predicators, classifying predicators etc. Furthermore, sentential-semantic classes also encompass the smaller groups of deictics, quantifiers, and partitives. Function words are disambiguated by their different syntactic functions in a sentence. This group 
4 Linguistically, the term predicators defines a verb in its functional relation to the clause, meaning an expression which takes a subject to form a sentence. We will, however, refer to the term as it is defined in terms of Propositional Logic and Predicate Logic. Here, a predicator designates a property or a relation and they can be ascribed to different objects. Grammatically, different word classes such as nouns, adjective and verbs can function as predicators (cf. Seiffert 1969: 23) 
756
includes conjunctions, prepositions, articles, particles etc. Like autosemantic words, they can be sorted into specific subgroups with corresponding functional properties. 
e.g.: This is a table (table = predicator), This is red (red = predicator), It moves (moves = predicator) 
The corresponding semantic or syntactic function for each word sense is mainly identified by the investigation of its paradigmatic and syntagmatic patterns, thematic roles, and modification patterns, as represented in contexts. The analysis of these patterns requires a corpus-driven investigation of surface relations, meaning the analysis of the co-occurrences of the word. Generally, it can be said that a polysemous word has several functions and can hence be grouped into different sentential-semantic classes, which function as lexical disambiguators. 
III Level of denotation 
The third level of sense differentiation is the level denotation. After each type of propositional function is classified, the reference or denotation of a word is determined. This level illustrates the senses of words by describing their different conceptual values. A system of conceptual classes helps to classify lexemes according to a taxonomic knowledge base which is being developed simultaneously.5 Words that belong to different entities tend to appear in recognisably different contexts. Therefore, in some cases a knowledge base can function as a context discriminator. The analysis of the denotational and conceptual content also follows a corpus-guided investigation of collocates where distinct word patterns which are associated with functions and the use of a word are examined. 
IV Level of specifications 
The fourth level comprises semantic specifications of words and provides further distinction of meaning. Specifications are understood as either inherent semantic properties, or semantic features which are identified by complements/adjuncts, e.g. aspectual features (aktionsarten) for process-denoting predicators. Others, like quality-denoting predicators, can be subcategorised according to their specifications into emphasising, classifying and modifying predicators. As far as function words are concerned, they often carry functional specifications. Conjunctions, for example, function as connectors of clauses. The type of connection between the clauses describes an individual specification of conjunctions such as conditional or concessive. 
V Level of relational properties 
In addition, some lexemes show relational characteristics which enable the lexicographer to further subdivide word senses. Relational properties refer to characteristics such as transitivity, symmetry, and reflexivity. Whereas the classification of sentential-semantic classes, denotation and specifications contain disjoint sets of features, a word can have more than one relational property. The classification of relational properties is a cross-classification which does not necessitate a preceding classification of specification. 
5 Disambiguation according to a knowledge base has become a widespread concept in Natural Language Processing (NLP) 
757
Altogether these criteria form a multi-faceted cross-classification functioning as a linguistic sense discriminator. Polysemous words and their different semantic or syntactic functions are classified according to their propositional features. If a word can be categorised into different classes we are able to distinguish word senses. Words that belong to different classes at one level occur in different contexts. Theoretical sense differentiation does not require a classification at every level. Tests have demonstrated that the senses of some words are sufficiently distinguished by classifying their propositional functions at the first or second level only. 
While the linguistic disambiguation model provides a theoretical basis of sense distinction, only the actual corpus data can provide an empirical validation of that model. In the next part attention will turn to the question of how theory meets corpus. It is also revealed how the necessary semantic analysis is conducted in a corpus-driven way and which tools are utilised in elexiko. 
2.2 COSMAS-Korpus-Recherchesystem and concordancing software 
The Institut für Deutsche Sprache Mannheim has compiled the largest German corpora. Currently, they are composed of about 1,900 million words from contemporary written and spoken texts. These corpora are accessible via a corpus query system called “COSMAS-Korpus-Recherchesystem“ (henceforth Cosmas6). The programme can be adjusted with respect to the settings of specific preference parameters.7 
Cosmas is an efficient statistical corpus query system with a concordancing software package Statistische Kollokationsanalyse und Clustering8 which has been utilised extensively in our project for lexical disambiguation. Its collocation analysis has been employed to detect statistically significant patterns of co-occurrences of word forms which are evaluated with regard to the use and semantic embedding of a word. Performing a collocation analysis results in the detection of linguistic regularities as well as irregularities within large text samples. Its main advantage is its ability to organise collocational structures by exploring semantic and syntactic neighbourhoods and calculating significance, thereby providing pre-structures which must be analysed systematically by the lexicographer. A further benefit can be seen in its ability to analyse language without introspective expectancy. It also offers empirical access to language in a comprehensive and systematic way, no lexicographer could perform. The lexicographer's task in disambiguating a polysemous word with Cosmas follows a procedure at three levels, the collocation-level, the KWIC-level, and the text-level. 
6 Cosmas is an abbreviation of Corpus Search, Management and Analysis System. It was developed at the IDS Mannheim and is publicly accessible via the internet (http://corpora.ids-mannheim.de/cosmas/). 
7 In the menu box there are several tabs which can be used to control the calculation of collocations, e.g. define the span of words around the hit or modify the performance of collocation calculation (by defining granularity of clustering, method for resolving cluster ambiguities, de/activating of function words, de/activating lemmatiser etc.). The setting of parameters impinge on the result of collocates and the hierarchy in the collocation list. 
8 The software Statistische Kollokationsanalyse und Clustering was developed by Cyril Belica (1995-2002) at the IDS Mannheim and can be used publicly via the internet since 1995. 
758
2.2.1. Collocation-level 
As indicated in the theoretical model, we believe that a sense distinction algorithm has to include the analysis of words by exploiting their contexts9. The disambiguation of polysemous words starts with the analysis of collocations10. The result is a retrieved list of co-occurrences (also called collocates) organised hierarchically and arranged according to the degree of lexical cohesion (lexical density).11 Although generally the result is only a statistical one, the advantage of a collocation list lies in its organisation and structuring of contexts, and the alignment of sense (cf. Sinclair 1991: 61). The collocation analysis identifies salient words which cluster together in a collective context. These contexts exemplify the semantic and syntactic dependency relationships that the key word participates in. This degree of clustering is expressed by frequency-based statistics where the most significant clustering have the highest score. 
Collocational structures are essential for lexicographic disambiguation, as they reveal linguistic patterns of the use and the propositional functions of a word by showing diverse paradigmatic, syntagmatic and syntactic structures, idioms, thematic domains, discourse analysis, and by uncovering co-referential co-occurrences. They reflect the complexity and the network of linguistic structures around the node by demonstrating, for example, the thematic roles of verbs and types of modifications of nouns, or simply any word that is closely associated with the key word. The itemised list of collocates offers the lexicographer different perspectives into the use of a word and its significant, as well as insignificant, semantic and syntactic neighbourhood. Restricted as well as non-restricted collocations give a picture of variant and invariant structures which can be classified into different linguistic categories. 
2.2.2 KWIC-level 
Although the organisation and structure of collocations offer great insights into the use of a word and its different semantic patterns, the co-occurrences and their significance must be evaluated as indications of senses only. Collocates cannot reveal the complex linguistic characteristics a key word exhibits in a larger empirical study of contexts. A quick insight into the actual contexts of the key word and its corresponding collocates can be gained at the KWIC-level (Key Word in Context12). KWICs are generated by concordancing software and are a way of displaying the search term and a selected collocate in a text so that the selected node word is listed in the middle column, with a certain amount of context on either side, usually a single line context. 
Lexicographers can draw five main benefits from the KWIC-level. First, the search term and all its corresponding collocates can be investigated individually and systematically by analysing their collective context. The advantage of this level is a selective and systematic analysis of co-occurrences 
9 We will exploit the term “context” when we refer to the notion of surrounding text of a search item, where strictly speaking “co-text” might be more appropriate. 
10 Collocations and the identification of salient words are not restricted to binominal structures. Also see Belica/Steyer (2002) (http://www.ids-mannheim.de/kt/kollok.html) 
11 Collocates are arranged by decreasing lexical density as determined by the corresponding log-likelihood ratio value. 
12 KWIC is a universal format for concordances. 
759
and their relationship to the key word. Secondly, this perspective on a small part of a common context reveals other words within the same semantic neighbourhood. Although they are usually statistically insignificant, they might contribute to the identification of the senses of the search item. Additional semantic partners such as paradigmatic co-occurrences, words functioning as modifiers, or words with semantically recurring patterns, and collocational or lexical restrictions are essential sense differentiating elements. Thirdly, the lexicographer is able to identify irrelevant collocates by establishing whether any collocate occurs, for example, due to faults of the lemmatising software.13 Fourthly, the KWIC is a small communicative unit which illustrates the proposition and its semantic components. Hence, it is vital to explore the predication in order to classify the involved predicators according to the aforementioned model. Finally, the systematics of this analysing procedure partly reveals unexpected results. As the concordancer works without any introspective expectancy and strict consistent statistical methods, the probability of capturing every sense of a word is higher than that of an intuitive search. Generally, it can be summarised that the KWIC-level is the first stage of a two-step verification process where the indications of the collocation-level are examined. 
2.2.3 Text-level 
The second step of the verification process is realised at the text-level. A KWIC is a one-line contextual display and in most cases does not display a full sentence. There are cases where the KWIC format is not adequate for the study of some words, as a detailed perspective into the meaning of the full proposition cannot be given. Therefore, the information provided cannot answer all questions in terms of the actual use of a search word. Here, the final level of text display must be consulted for closer study. The extent of the display can be selected according to the number of preceding and following sentences. The broad context must finally be selected to reveal the semantic behaviour of a word in its full environment. The larger the context is chosen, the larger semantic potential it can offer and, thus, the more differentiated an analysis can be performed.14 Only at this level is the lexicographer able to recognise linguistic patterns around the node word and its collocates as well as indications of referential components and the recurrence of semantic conglomerates which are important aspects of identifying semantic characteristics. 
Generally, we can conclude that the first level should be regarded as a means of pre-selection and pre-structuring for the disambiguation procedure. The actual systematic exploration of contexts – smaller and larger ones – is lexicographically simplified and systematised by the alignment of the collocational structures of the word. But only after the examination of the second and third level is the lexicographer able to linguistically categorise senses according to the aforementioned model and finally theoretically verify the disambiguation of the key word. 
13 As Sinclair (1997: 31) claims: “…we do not know which details are essential, which important, which optional, which indicative, which transitory, which random and which distracting.” 
14 The number of sentences to be selected is restricted due to copyright contracts. 
760
3 Conclusion 
Elexiko has put forward an elaborate disambiguation theory, a linguistic classification model which illustrates the propositional differences words can carry semantically, and hence their various senses can be categorised within a theoretical framework. In this way, lexicographic sense distinction has been elevated from a procedure conducted by introspection to a fully model-based task. Elexiko is also the first monolingual German dictionary which works in a corpus-based way. Linguistic information is derived from the complex empirical study of corpora and exemplified through citations. With respect to lexical disambiguation our project illustrates perspicuously the break with conventional German lexicography. Effectively, what has been achieved here is the development of a viable solution to the problems of establishing a disambiguation algorithm which links theoretical and empirical factors. 
Initial tests have clearly ascertained that the demands of modern lexicography can be met through a combination of the following: corpora as an empirical base, a corpus-processing software as a method to generate semantic information and as an instrument for structuring senses, and a robust theoretical model which linguistically justifies meaning discrimination. Although we can already derive valuable results from these tests, considerable testing of the proposed disambiguation technique is still required and is currently being conducted. So far, the results have demonstrated more detailed semantic descriptions and more objectively disambiguated senses than have ever been offered by other monolingual dictionaries. The hypertext structure of our dictionary allows a semantic complexity which can only be elicited from the exploration of corpora and the use of computational tools. A number of examples have shown different results in terms of the identified senses of a word. In most cases, compared to Duden (2000), the methodological basis, upon which the senses of a key word are identified in our project has led to the identification of additional senses. 
However advantageous any performance of computer analysis might be, Cosmas and its software components cannot replace the human element. Corpora and Cosmas are instruments for lexicographers. A corpus-processing tool can perform statistical analyses which are indicative, but they cannot interpret data linguistically. Although they give strong and necessary measurable evidence, so far we cannot envisage a world of dictionaries without lexicographers, a world where computers analyse a massive amount of linguistic data and where dictionaries would be generated automatically. The decision as to how to interpret corpus data and how to select irrelevant from relevant information remains the task of the lexicographer. At the same time modern lexicography must recognise that computer technology can supply us with the necessary volume of data (and the software for analysing it) and substantially enhance the compilation of a dictionary. It significantly improves, simplifies and systemises the lexicographer's work by performing large empirical explorations of data otherwise unmanageable for the unaided researcher. 
761
Bibliography 
Belica C, Steyer K 2002 Die COSMAS-Kollokationsanalyse – statistisches Modell, Funktionsweise und Interpretationsspielräume. (IDS-Homepage: http://www.ids-mannheim.de/kt/kollok.html). 
Belica C 1995-2002 Statistische Kollokationsanalyse und Clustering. Korpusanalysemodul. (IDS-Homepage: http://corpora.ids-mannheim.de/cosmas), Institut für Deutsche Sprache, Mannheim. 
Hanks P 1990 Evidence and Intuition in Lexicography. In Tomaszczyk J, Lewandowska-Tomaszczyk B (eds), Meaning and Lexicography. Amsterdam/Philadelphia, Benjamins. pp 31-41. 
Haß-Zumkehr U 2002 Das Wort in der Korpuslinguistik - Chancen und Probleme empirischer Lexikologie. In Ágel V (ed), Das Wort: seine strukturelle und kulturelle Dimension. Festschrift für Oskar Reichmann zum 65. Geburtstag. Tübingen, Niemeyer, pp 45-70. 
Polenz v. P 1985 Deutsche Satzsemantik. Grundbegriffe des Zwischen-den-Zeilen-Lesens, (=Sammlung Göschen 2226), Berlin/New York, de Gruyter. 
Reichmann O 1989 Einführung. In Anderson R R, Goebel U, Reichmann O (eds), Frühneuhochdeutsches Wörterbuch. Bd.1, a – äpfelkern. Berlin/ New York, de Gruyter, pp 1-164. 
Scholze-Stubenrecht W 2001 Das Internet und die korpusgestützte praktische Lexikographie. In Korhonen J (ed), Von der mono- zur bilingualen Lexikographie für das Deutsche. (Finnische Beiträge zur Germanistik Vol 6). Frankfurt, Peter Lang, pp 43-64. 
Seiffert H 1969 Einführung in die Wissenschaftstheorie. Vol 1: Sprachanalyse - Deduktion - Induktion in Natur und Sozialwissenschaften, München, C. H. Beck. 
Sinclair J 1987 Introduction. In Sinclair J et al (eds), Collins Cobuild English Language Dictionary. London/Glasgow, Collins, pp XV-XXIII. 
Sinclair J 1991 Corpus, Concordance, Collocation. Oxford, OUP. 
Sinclair J 1997 Corpus Evidence in Language Description. In Wichmann, A et al (eds), Teaching and Language Corpora. London/New York, Longman, pp 27-39. 
Strauß G 1989 Angabe traditioneller Wortarten oder Beschreibung nach funktionalen Wortklassen im allgemeinsprachigen Wörterbuch? In Hausmann F J, Reichmann O, Wiegand H E, Zgusta L (eds), Wörterbücher: ein internationales Handbuch zur Lexikographie. (Handbücher zur Sprach- und Kommunikationswissenschaft Vol 5.1). Berlin/New York, de Gruyter, pp 788-796. 
Dictionary 
Duden 2000 Das große Wörterbuch der deutschen Sprache. 10 Volumes on CD-Rom. Mannheim. 
762

Cultures and Corpora: Extracting Anthropological Information from Corpora of Formosan Endangered Languages 
Dr. Jozsef Szakos 
Department of English Language, Literature, and Linguistics 
Providence University, Taichung, Taiwan 
szakosdr@ms38.hinet.net 
Corpus research has concentrated on finding grammatically relevant information in collections of well-documented languages. While applying the methods of CL to minority languages, we have encountered another area worthy of discussion, namely the treatment of cultural and anthropological information in these corpora. 
For English, French or other major corpus research areas the solution has been either to ignore cultural issues influencing grammar, or to relocate such information into encyclopaedias, which in turn were not regarded as corpora. To illustrate this point, I just mention that Encyclopaedia Britannica is not BNC, but BNC cannot replace the knowledge contained in Britannica. At the same time, we are conscious that the relevant information in our culture also influences the grammar structures in BNC. 
While we are producing corpora for endangered languages, which I intend to exemplify on Austronesian languages of Taiwan (Formosan languages), we cannot escape two problem-fields. The first one is that we need to create bilingual corpora. The second necessity is the inclusion and treatment of anthropological and cultural information in these collections. We are going to deal with about a dozen languages, so it is desirable to work out the standards of bilingualor plurilingual corpora, and at this point we encounter the problem ofethnologically relevant information and its extraction. 
In my paper I intend to discuss the above questions while introducing the present stage of our research. I wish to speak about naming and culture (including conceptualisation of living and non living things), deixis and culture (geography, social life determining factors), language use and culture (constraints on the use of some constructions). This would also include analysing the existence or non-existence of ethnically conditioned concepts. I will also demonstrate some technical solutions for the bilingual corpora, using ParaConc. Besides, I want to show how this approach of taking into consideration the local cultural phenomena can contribute to the creation of authentic teaching materials for the preservation of these languages. 
763

Do we talk (or write?) differently over the Net? 
- A lexical enquiry into ‘a’ Net-EN - 
JUN ARATA TAKAHASHI 
UNIVERSITY OF ESSEX 
1. Introduction 
Emails, Internet discussion fora, chat rooms...a growing number of people use network connected computers for their personal, educational, and job-related correspondence. Some are very formal and some others are very casual. When people are in such computer mediated communication (CMC), do they talk (or write) differently from conventional written or oral communications? In lodging an on-line application form, for instance, it is easy to imagine that the form can be identical to a printed hard copy. The question is, however, what would it be that ones specific in computer mediated communications such as those emails, Internet fora, and chat rooms? Are they used in the same way as personal or professional letters (writing) or telephone conversations (speaking)? Herring (1996) defines the characteristics of the language in CMC as: 
...it is typed, and hence like writing, but exchanges are often rapid and informal, and hence more like spoken conversation...CMC is not homogeneous, but like any communicative modality, manifests itself in different styles and genres, some determined by the available technologies (e.g., real-time “chat” modes, as opposed to asynchronous e-mail), others by human factors such as communicative purpose and group membership. 
(pp. 3-4) 
She also identifies that users of CMC often use unique characters and acronyms conveying special meaning or shortening the time of composing. 
2. Previous research 
2.1 Linguistic enquiries and language in CMC 
Due to its anonymity and unique presentation in its communication, Herring (1996) identifies two major foci of research aspects in this field: socio-linguistic aspects (e.g., gender, identity, personality, and mode of their interactions) and linguistic aspects (e.g., lexis, syntax, phrase, mode, discoursal and genre functions). 
As is mentioned in the introduction, the issue of linguistic mode (speaking and writing) needs to be considered. Baron (2000) identifies some competing agenda and models in linguistic enquiry: ethnographic (function of form), technological (medium and message), and pedagogic (grammar) agenda and opposition, continuum, and crossover models between writing and speaking language. In terms of ethnographic agenda or descriptive language analysis, Tannen (1982), from her research on oral and literate narrative strategies, suggests that the features distinguishing one discourse from another are not only written or spoken mode but also genre and register. From another perspective on genre, Crystal (1991), inspired from a previous study with his colleagues on grammars of language impaired patients, shows an example of a stylistic language profiling analysis between newspaper articles and legal documents. The profiles include five categories: graphetics, graphology, phonology, grammar, and semantics. These categories are rated scales in three aspects, frequency, distinctiveness, and precision. He reports that legal documents’ distinctiveness is in nominal group and clause structure while newspaper articles’ is in sentence structure and connectivity. In his conclusion, Crystal (1991) foresees the invaluable outcomes of the research into stylistic aspect of language across different genres. 
764
Biber (1988 and 1995) has developed the notion of Multi-Dimensional analysis over the years, which categories the nature of genre into six dimensions. He argues for forms and styles are more genre specific in English language rather than medium oriented. In that sense, his research findings support the crossover model between writing and speaking language. However, does it apply to the emerging new mode, computer mediated communication (CMC) or it has something different from the cousins? 
As for such language models, Baron (2000) sees language in CMC through the continuum model with the idea of the crossover view of language: posting web version of printed articles such as online journal can be seen as the products of writing which are static while Internet discussion forum, chat rooms, and email correspondence can be called a process and thus dynamic communication. Even within this dynamic communication, however, there can be further two types of interactions, one is synchronous such as Internet relay chat (IRC) and chat rooms and the other, asynchronous (spontaneous responses though not synchronous) such as emails and BBS (Herring, 1996 and Simpson, 2002). Simpson (2002) investigates the former for discoursal features (turn taking and unique utterances) in IRC and instant messaging. He has found that like face-to-face conversation, interaction in IRC maintains strong coherence in turn taking despite the fact that multiple conversations may occur between more than two groups of participants in one message screen. For instance, interlocutors employ repetitions of style like the ones Tannen (1989) has found in face-to-face conversation. The present study will look into one of the latter form of interaction, which is dynamic and spontaneous but not synchronous communication, Internet forum. 
Not from the discoursal ones but from lexical perspectives, the present research will deal with the texts in Internet discussion forum. English language nowadays can be regarded as a multi-national language (EML), rather than inter-national language, which belongs to every user of English regardless weather it is mother tongue or not. Because of this, the present study will focus on the language change in the effect of the medium used among Japanese EML users. The concept of English as a multi-national language (EML), in Brutt-Griffler's (2002) term ‘World English', can be described as “World English is not simply made through speakers of other languages but by them” (p. ix). In terms of the impact of entire speech community on second language acquisition (SLA), she further introduces a new concept, “macroaqcuisition”; a new English is acquired through and nurtured by the speech community. Brutt-Griffler (2002) categorises “macroaqcuisition” into two types: Type A macroaqcuisition occurs in a multi-lingual setting that has adopted another unifying language and will develop an entirely new speech community while Type B occurs as transformation of a monolingual community into a bilingual one (p. iix). In the case of English as foreign language (EFL) setting, however, it does not fit into either of those but it can be categorised as another type, Type C, which introduces another (international) language as a tool of communication with other parts of the world rather than within the country. A publicly opened Internet discussion forum using English as its medium can be one of such macroaqcuisition settings falling in this Type C and it can be one of the gateways to English as multi-national language (EML), which belong to everyone who uses it. 
The present research will seek if texts in such a unique bleed of communication and acquisition setting reveal another side of English as multi-national language today. 
2.2 Lexical and functional research into Net-EN 
Lexical functionality in CMC English (hereafter Net-EN)1, general written English (LOB), and spoken English (London-Lund) are compared in Yates (1996). He has found that the Net-EN shows significantly different lexical choices from the other 
1 The Net-EN investigated in Yates (1996) is English language in Open University on-line discussion groups in U.K. See Yates (1996) for further details. 
765
two modes and Net-EN can be seen as a unique mode of medium. He adopts Halliday's (1978) concept of language function (textual, interpersonal, and ideational) in analysing Net-EN and comparing it with written and spoken English from lexical functional perspectives. In texts, the textual function deals with information and surrounding context, the interpersonal function realises the relationship of the participants in the texts, and the ideational function expresses the speakers'/writers’ external experience and/or internal reality (Halliday, 1978). Yates (1996) also adapts methods of other researchers’ previous studies to analyse these three functions of Net-EN. 
First, for the textual function, the three modes of communication (written, spoken, and Net-EN) are compared against type/token ratio and then lexical density. In simplest term, lexical density is “the number of lexical items [content words] as a proportion of the number of running words [total words]” (Halliday, 1985: 64). In short, the higher the lexical density, the more information is included in the texts. Referred as a previous investigation in Yates (1996), Ure (1971) has investigated the difference of the lexical density in both written and spoken texts with a focus on their register variances: 
The distinction between spoken and written texts is one of medium, and with it can be grouped all those other distinctions due to the physical conditions of the delivery of the text, in sound, in space and in time. The time available to prepare a text is also a factor that could be important on the dimension of physical circumstances. 
(p. 447) 
From the lexical aspect, Yates (1996) conducts a similar investigation with Ure's in written (LOB), spoken (London-Lund), and Net-EN. Yates reports that the level of lexical density is highest in written texts, then Net-EN and spoken with the lowest and the differences of the levels between them are statistically significant (p<0.05). He suggests that the way delivering information in Net-EN is more like written texts than spoken ones2. 
Yates (1996) then counts personal pronouns as a barometer of interpersonal function of texts3. Total number of first, second, and third pronoun use in written, spoken, and Net-EN are all statistically different from each other, however, he notices that the relative distributions of these three kinds of pronouns are distinctively different among the three modes of communication: Net-EN's proportion is very different from written one but somewhat similar to the one for spoken, but Net-EN uses more first and second pronouns (64%) than the other two (27% for written and 58% for spoken) (Yates, 1996: pp.42-3). 
Finally, Coates's (1983) model of analysing five categories of modal auxiliaries is adapted to Yates's (1996) investigation to the ideational function4: the five categories are obligation and necessity (e.g., must), ability and possibility (e.g., can), epistemic possibility (e.g., may), volition and prediction (e.g., will), and hypothetical modals (e.g., would). He has found that the use of all modal auxiliaries in Net-EN is greatest among the three modes and significantly different in every category from the other two modes of communication except in epistemic possibility against written mode. Again, the relative distribution of modal auxiliaries in those five categories in Net-EN is similar to spoken one (Yates 1996). 
From the findings of Yates (1996), although the actual values of lexical frequency in the three Hallidayan linguistic functions are significantly different from the other two modes of communication, one can interpret that the users of Net-EN deliver information (textual) as if they are writing texts but use personal 
2 See Yates (1996) for further discussions about the ease or difficulty in understanding the texts and about the repetition of the same or similar lexical items. 
3 See Chafe (1982) and Chafe and Danielewicz (1987) and Fowler and Kress (1979) for the original and further discussions on this issue. 
4 See Coates (1983) for detail. 
766
pronouns (interpersonal) and modal auxiliaries (ideational) as if they are speaking. 
As Ure (1971) notified, Collotte and Belmore (1996) on the other hand use categories of on-line and off-line language (a categorisation on the bases of the time available) rather than conventional written and spoken language. They apply Biber's (1995) Six Multi-Dimensional linguistic variation analysis model for informativity, narrativity, explicitness, persuasion, abstraction, and elaboration in analysing on-line and off-line international Internet bulletin board (BBS) English. Biber (1988) describes the notion of Multi-Dimensional linguistic variation analysis in three ways: 
1) Similar or the same type of texts shares particular types of linguistic features; 
2) Those features are measured in six bi-polar scales (dimensions) from positive to negative weights (except Dimension 4 and 5) rather than existence or non-existence of those features; and 
3) Conceptualisation of dimensions is not predetermined but they are the results of quantified empirical findings. 
A summary of textual features of the six dimensions (Biber, 1988 and 1995) and Collotte and Belmore's (1996) findings is as follows: 
Dimension 1: Involved versus informational production 
Nouns, word length, prepositions, high type-token ratio, and attributive adjectives are negative features in the scale of this dimension. Carefully elaborated long sentences with long words and frequent use of nouns are typical in texts in informational processing. In this Dimension 1 continuum, Collotte and Belmore's Net-EN falls between romantic fiction and personal letters. 
Dimension 2: Non-narrative versus narrative 
Past tense, third person personal pronouns, perfect aspect verbs, and public verbs are strong positive markers of narrative discourse while present tense verbs and attributive adjectives are negative features which make non-narrative discourse. In this continuum, their Net-EN falls between press reviews and interviews. 
Dimension 3: Situation-dependent versus elaborated reference 
Time and place adverbials and some adverbs are positive weights of situation-dependent texts while relative constructions, such as WH-relative clauses, are seen as features of elaborated reference. In the continuum, their Net-EN has scored very similar to humour and press reportage. 
Dimension 4: Overt expression of argumentation 
This dimension only has positive features. Infinitives, prediction modals, suasive verbs, conditional subordination, necessity modals, and split auxiliaries. These features are commonly found in professional letters and editorials but virtually non-existence in press reviews and broadcasts. In this dimension, Collotte and Belmore's Net-EN appears between personal letters and editorials which are close to the end of positive scale. However, both ‘off-line’ and others5in the Net-EN score closer to hobby and romantic fiction which fall in-between ‘off-line’ and others. 
Dimension 5: Abstract versus non-abstract styles 
This dimension only has negative features. Conjuncts, agentless passives, past participial adverbial, postnominal clauses, and by-passives are most important features of abstract styles, such as technical and engineering prose and 
5 For some ‘on-line’ texts, it is difficult to identify whether they are really on-line or pre-constructed off-line ones or not, therefore Collot and Belmore (1996) group their texts ‘off-line’ and ‘other'. 
767
official documents. In this continuum, their Net-EN appear between religion and official documents, however the Net-EN scores are not close to these genre and Collotte and Belmore further report that their Net-EN does not have the features which play a major role of this dimension. 
Dimension 6: On-line informational elaboration marking stance 
Demonstratives, some that-clauses (as verb complements, in object position, and as adjective complements), final prepositions, and demonstrative pronouns are typical markers of informational spoken (‘on-line') discourse, e.g., prepared speeches and public conversations, while edited or non-informational texts like most fictions have phrasal co-ordination as their feature. In this last dimension, Collotte and Belmore's Net-EN appear close to editorials and professional letters. 
In sammary, Collotte and Belmore (1996) suggest that the genre across Net-EN, such as BBS, Internet forum, and on-line news bulletin, will likely show different textual features to each other but the genres which most resemble their Net-EN are public interviews and personal and professional letters. 
All those researches support that Net-EN can be a third variety of language mode. In the present study, another kind of Net-EN is investigated, which is a Net-EN by Japanese EML (English as multi-national language) users. 
3. Data Gathering and Method of Analysis 
3.1 The source and the nature of the Net-EN corpora (NC) 
The texts are gathered from an internet BBS maintained by Keio University, Shonan Fujisawa Campus (SFC) in Japan, which is a part of their EFL curriculum. This aims to develop students’ interactive writing and communicative skills in computer mediated communication. This BBS, called IWC6 (Interactive Writing Community), has various sub-fora discussing from serious social and political issues (e.g., Euthanasia) to hobbies and sports. It also invites not only other university students but also participants from other countries without any status restrictions. 
The present research uses Net-EN corpora (NC) containing two sub-corpora, Base Text Corpus (BC) and Response Text Corpus (RC), of approximately 564,000 token from about 3050 base and response articles. The average length of an article in BC is about 370 words and for RC, 160 words. In order to achieve as general analysis as possible, selected 190 most common words are used. In the Net-EN corpora (NC), the 190 words occur more than the most commonly used proper noun that is specific in IWC, sfc, which is the abbreviated campus name of the university organising this forum. The 190 words cover 68.6%, 67.0%, and 70.2% of NC, BC, and RC respectively7. These words occur at least either 100 times in BC or 89 times in RC per 100,000 words. Any comparison between corpora and between given words are conducted within these 190 words. 
As is advocated in the introduction, the present research investigates English as multi-national language (EML) in the Internet discussion forum that is computer mediated communication. In this principle, any initial posting which has not attracted any response is regarded as neither EML in use nor a part of communication. Any articles, therefore, fall in the criteria below are excluded from the corpora: 
6 http://www.sfc.keio.ac.jp/iwc/IWC/index_2002f.html for the current fora. Previous IWC discussion fora can be accessed by replacing the part after the underscore (_) in the URL address to the year and semester looked for, e.g., for Spring semester in Year 2000 will be “.../index_2000s.html”. 
7 Only as a guide, the 190 words cover approximately 59.4% of BNC spoken corpus and 48.6% of BNC written corpus. 
768
1) Base articles that have not received any response; 
2) Any articles posted by other than Japanese participants; and 
3) Any articles posted by high school age Japanese. 
This maintains the NC, BC, and RC as corpora of EML in use by Japanese English users with certain linguistic level which enable the participants to communicate with others in English. Further figures of the NC, BC, and RC are in Table 1 below. 
Table 1: A profile of the Net-EN Corpora (NC, BC, and RC) 
Profile of the Net-EN Corpora (NC, BC, RC) Name of the BBS Forum No. of articles in BC Size of BC (token) No. of articles in RC Size of RC (token) Size of NC (token) IWC 409 151,000 2636 413,000 564,000 
3.2 Method of analysis 
As a preliminary comparison, the 50 most frequently used lexis are listed and compared between written and spoken English from British National Corpus (BNC) and the Net-EN corpus (NC). As its main analysis, it applies the same approach above in comparing the Base Text Corpus (BC) and Response Text Corpus (RC). A selected word with unique uses will then be investigated in the light of senses, collocates, and patterns8 of use. Collins COBUILD English Dictionary (CCED) (1995) and Francis, Hunston and Manning (1996) will be referred to for this purpose. 
4. Results 
4.1 A cross comparison of frequency information between written, spoken, and Net-EN. 
Like the findings in Yates (1996) reported in the earlier section, in the frequency list, a largest selection of prepositions and functional words (e.g., determiners and conjunctions) appears in written English, next Net-EN, and then the smallest in spoken English though some functional words specific to spoken language appear. Various types of content words are most used in Net-EN, next spoken English, and then least in written English. Like spoken English, Net-EN appears with high use of the first and second personal pronoun while written English the third person comes most frequently used pronoun. Further in the Net-EN, the word I is actually the most commonly used lexis while both written and spoken English have the in their first rank (See Table 2 below). 
Table 2: List of 50 most frequent word from in three corpora9 10 
Written (BNC) Spoken 
(BNC) Net-EN 
(NC) Written (BNC) Spoken 
(BNC) Net-EN 
(NC) 1 the the I 26 but well be 2 of I to* 27 from so very 
8 Hunston and Francis (2000) Pattern Grammar for theoretical detail of patterns. 
9 Frequency ranks of both written and spoken English in British National Corpus (BNC) are obtained and adapted from Leech, Rayson, and Wilson (2001) and thus it is a guide only. 
10 Word forms with ‘*’ include all the possible part of speech they are used in the corpora. 
769
3 and you the 28 which oh can 4 to* and and 29 she got as 5 a it is 30 they not people 6 in that* of 31 or are they 7 is to* you 32 an if like* 8 was a in 33 were with do 9 it of a 34 as no about 10 for in that* 35 we she what 11 that* we it 36 their at if 12 with is my 37 been there on 13 he do for 38 has* think essay 14 be they we 39 that yes when 15 on er think 40 will just school 16 I was have* 41 would all many 17 by what but 42 her can because 18 at he your 43 there then from 19 you but so 44 all get at 20 are for was 45 can did by 21 had* erm not 46 if or there 22 his be are 47 who like* also 23 not on this 48 said would life 24 this this me 49 do mm don't 25 have* know with 50 what them want 
A list of frequency rank of written, spoken, and Net-EN is provided in Table 2 above. The 50 most common word forms store up to 47.2% for Net-EN corpus (NC), 46.3% for Base Text Corpus (BC), and 48.1% for Response Text Corpus (RC). This shows a brief picture of the differences of the lexical use across the three different mediums. 
The frequencies of most common verbs are also noticeable. For instance in the first 50 most common words, had, have, has, and said are most common verb forms in written English, know, got, think, and get for spoken, and have, think, had, and thank for Net-EN. These lexis appear in the very different ranks in the frequency list. 
In addition, the frequencies of the top 190 words show statistically significant differences between the Net-EN corpora (NC, BC, NC) and BNC (spoken and written) ranging from p<0.001 to p<0.032 except between the Base Text Corpus (BC) and BNC spoken corpus (p<0.083) although it is a guide only. However, no significant differences have been found between the sub-corpora of Net-EN. 
5. Findings and Implications for Further Research 
5.1 A close investigation of a selected word in the use and patters 
One of most noticeable findings is that major verbs with high utility and in use are used in unique ways in the research corpora (Net-EN). The difference is not only the frequency of the word (including inflections) but also the ways (senses and patterns) they are used. The word make in the NC, for instance, is most commonly used as a causative use while the sense, carrying an action (e.g., make a suggestion), is the first in CCED which is based on the frequency information in Bank of English (See Table 3 below). 
Table 3: Frequencies of uses of make in Net-EN corpora (per 100,000 tokens). 
Order of uses in CCED Net-EN (NC) Base Text (BC) Res. Text (RC) 1 carrying an action 6964732 causative 140137143 
770
3 creating or producing 8983954 others 162210 TOTAL 314306321 
As it can be seen, about half the uses of make is causative in both BC and RC. Table 4 below further shows the frequency of each pattern of those causative uses. Again, the most common pattern of use is not the same as the one found in a corpus based on general English by native speakers. 
Table 4: Raw frequencies of patterns of causative uses of make in Net-EN corpora11. 
Order of appearances in CCED NC BC RC 1 V n inf. 30981 2282 be V to inf. 82 63 V n n 289 194 V n adj. 394100 2945 V it adj. (that)/to inf. 307 23* V n be adj. 122 10* V n to inf. 226 16* V n –ing 22 0 
This is possibly an example showing the unique use of English produced by Japanese users of English as multi-national language (EML) in today's context, computer mediated communication (CMC). Further research of this kind can reveal such unique use of English in CMC among Japanese EML users. This will contribute to capture a diverse use of English language today. Looking into the root of their unique use of lexical items can be a valuable and promising further research topic. 
6. References 
Baron N 2000 Alphabet to email. London, Routledge. 
Biber D 1988 Variation across speech and writing. Cambridge, Cambridge University Press. 
Biber D 1995 Dimensions of register variation. Cambridge, Cambridge University Press. 
Brutt-Griffler J 2002 World English: A study of its development. Clevedon, Multilingual Matters Ltd. 
Coates J 1983 The semantics of the modal auxiliaries. London, Croom Helm. 
Collins COBUILD English Dictionary 1995 London, HarperCollins. 
Collot M, Belmore O 1996 Electronic language: A new variety of English. In Herring S 1996 (ed) Computer-mediated communication: Linguistics, social and cross-cultural perspectives. Amsterdam, John Benjamins. pp 13-28. 
Crystal D 1991 Stylistic profiling. In Aijmer K, Altenberg B (eds) English Corpus Linguistics. London, Longman. 
Francis G, Hunston S, Manning E 1996 Collins COBUILD Grammar Patterns 1: Verbs. London, HarperCollins. 
11 Patterns with ‘*’ do not appear in CCEC as patterns for make. 
771
Halliday M A K 1978 Language as social semiotic: The social interpretation of language and meaning. London, Edward Arnold. 
Halliday M A K 1985 Spoken and written language. Oxford, Oxford University Press. 
Herring S 1996 Introduction. In Herring S (ed) Computer-mediated communication: Linguistics, social and cross-cultural perspectives. Amsterdam, John Benjamins. pp 1-10. 
Hunston S, Francis G 2000 Pattern Grammar. Amsterdam, John Benjamins. 
Leech G, Rayson P, Wilson A 2001 Word frequencies in written and spoken English: based on the British National Corpus. Harlow, Pearson Education Ltd. 
Simpson J 2002 Discourse and synchronous computer-mediated communication: Uniting speaking and writing? In Spelman Miller K, Thompson, P (eds.) Unity and diversity in language use: Selected papers from the Annual Meeting of the British Association for Applied Linguistics held at the University of Reading, September 2001. London, Continuum. pp. 57-71. 
Tannen D 1982 Oral and literate strategies in spoken and written narratives. Language, (58): pp 1-21. 
Tannen D 1989 Talking voices: Repetition, dialogue and imagery in conversational discourse. Cambridge, Cambridge University Press. 
Ure J 1971 Lexical density and register differentiation. In Perren, G E, Trim L M (eds.) Applications of Linguistics: selected papers of the second International Congress of Applied Linguistics, Cambridge, 1969. Cambridge, Cambridge University Press. 
Yates S J 1996 Oral and written linguistic aspects of computer conferencing: A corpus based study. In Herring S (ed.) Computer-mediated communication: Linguistics, social and cross-cultural perspectives. Amsterdam, John Benjamins. pp 29-46. 
772

A Study of Text Types and Register Variation in the British National Corpus 
TAKAHASHI, Kaoru 
Toyota National College of Technology 
takahasi@toyota-ct.ac.jp 
The paper is concerned with text typology. The British National Corpus (World Edition), which is a one hundred million-word collection of spoken and written British English, will be used to study the characterization of text types by identifying their linguistic characteristics. By means of multivariate analysis, the variation of the occurrence of selected linguistic features among genre categories will be classified. A multivariate analysis of this sort holds out the promise of being able to systematize the genre categories in the corpus while also revealing the characteristic linguistic features of the groups classified. The criteria for classification will be based exclusively on the dimensions revealed as significant by the multivariate analysis. The groupings will then be interpreted linguistically. 
In the research I also wish to focus on what kind of role language variation takes in the study of linguistics. Therefore, this paper deals with the study of sociolinguistic variables which are fairly systematically associated with ‘region', ‘class', ‘ethnic group', ‘age', and ‘gender’ and so on. The aim of an analysis of these variables is to yield a more sophisticated classification and systematization of textual sub-categorization in the British National Corpus. The groupings will provide sociolinguistic interpretations. 
773

The Structure of the Consonant Patterns in the Spanish Speech Sound Chain as a Clue of Typological Closeness 
Yuri Tambovtsev. Novosibirsk Ped. University, Novosibirsk 
yutamb@hotmail.com 
Spanish is said to belong to the group of Romanic languages of Indo-European family. Besides Spanish, Romanic group includes Portuguese, French, Italian, Rumanian, Moldavian and some other minor Romanic languages. 
Spanish, as any other human language, has a specific structure of the speech sound chain. It can be distinguished by its structure from any other language. Every language has a unique structure of distributions of speech sounds in its phonemic chain. The distribution of Spanish vowels will not be considered till the second stage of the investigation. Let's point out that consonants bear the semantic load in the word, not vowels. Therefore, it is more possible to understand the meaning of the message by consonants, rather by vowels. However, if we fail to recognise and distinguish two languages, then we resort to the structure of occurrence of vowels in the speech sound chain. While comparing languages, it is necessary to keep to the principle of commensurability. Having it in mind, it is not possible to compare languages on the basis of the frequency of occurrence of separate phonemes, because the sets of phonemes in languages are usually different. The articulatory features may serve as the basic features in phono-typological reasoning. First of all, it is the classification of consonants according to the work of the active organ of speech or place of articulation (4 features: labial, front, palatal, and velar). Secondly, it is the classification from the point of view of the manner of articulation or the type of the obstruction (3 features: sonorant, occlusive and fricative). Thirdly, it is the classification according to the work of the vocal cords (1 feature: voiced). In this way, 8 basic features are obtained: 1) labial; 2) front; 3) mediolingual or palatal; 4) back or velar; 5) sonorant; 6) occlusive; 7) fricative; and 8) voiced consonants. One should take the values of the frequency of occurrence of these 8 features in the speech chain of Spanish and compare them to those of the other Romanic languages. On the basis of the "chi-square" test and Euclidean distance, we have developed our own method of measuring the phono-typological distances between languages (Tambovtsev, 1994-a; 1994-b). It takes into account the frequency of occurrence of the 8 consonantal groups mentioned above to measure the overwhelming mosaic of the language sound picture. Having compared Spanish to some languages, we received the following phono-typological distances: 
Spanish – Esperanto 6.15 
Spanish – Portuguese 7.91 Spanish – Latin 7.98 Spanish – Italian 8.35 Spanish – Rumanian 9.26 Spanish – French 13.39 
As a conclusion, we can state that speech sound picture of Spanish is the most similar to Esperanto. One can explain it by the fact that Esperanto uses many Latin roots in the same way Spanish does. It is not a surprise that Spanish is close to the sound pattern of Portuguese or Latin. Actually, Latin is the parent language of all Romanic languages, thus Spanish uses many of its roots. Spanish is rather far away from the language which is geographically close, i.e. French, which undergone so many phonetic changes. It is logical that Spanish and French are distant from each other by the phonostatistical point of view. In fact, sound pattern of French is more different to that of Spanish, than Spanish to the rest of the Romanic languages. Our data state that the speech sound pattern of Spanish more or less resembles that of the other Romanic languages. Latin, the parent language for all the Romanic languages, gave all Romanic languages but French its sound system. We must point out, therefore, that it is not a coincidence that Portuguese, Italian and Romanian are more similar to Spanish than French. It was no surprise for us that Spanish consonants are distributed more similar to Portuguese, than Italian or Rumanian, because they were in the closest contact for many years. 
References: 
Tambovtsev, Yuri. 1994a Dinamika funktsionorovanija fonem v zvukovyh tsepochkah jazykov razlichnogo stroja. - Novosibirsk : NGU. 
Tambovtsev, Yuri. 1994b Tipologija uporjadochennosti zvukovyh tsepej v jazyke. - Novosibirsk: NGU. 
774

Phonological similarity between Basque and other world languages based on the frequency of occurrence of certain typological consonantal features. 
Yuri Tambovtsev. 
Novosibirsk Pedagogical University, Novosibirsk 
The aim of the present typological study of the Basque language based on the frequency of occurrence of consonants is to compare its sound chain to sound chains of various world languages. Then, one can construct typological distances between Basque and other world languages. 
The structure of the frequency of occurrence of consonants in the speech sound chain is a good clue of understanding the typological closeness of languages. Typological investigation of human languages involves the task of the analysis of their speech chains by a human being or computer. A human being can realise that this or that language sounds closer to his own native language without understanding the meaning. At the stage when it is hard to teach computer to understand a human language, it is quite possible to make it recognise the sound closeness of a language to this or that language on the basis of the analysis of its sound speech chain. We have computed the frequency of phonemic occurrence of 119 world languages as a teaching sample for the computer. Then we took Basque as a token language. Actually, Basque was chosen for only one reason. It was taken up mainly because Basque as well as Japanese, Korean, Ainu, Burushaski, Nivh (Gilyak), Yukaghir are considered to be isolated languages, i.e. languages that do not belong to the known language families. In fact, they may be relics of the former family of languages (Crystal, 1992: 425). Basque is a fair example of an isolate language. Efforts have been made to show a relationship with Caucasian languages, North African languages, and Iberian, but none has been convincing (Crystal, 1992: 40). Merritt Ruhlen considers John Bengtson to be correct to include Basque, Burushaski and Shumerian into the new Dene-Caucasian family (Ruhlen, 1994: 25). R. L. Trask believes Basque to be unquestionably the last surviving pre-Indo-European language in Europe. He links Basque with the dead Aquitanian language (Trask, 2001: 72). 
We tried to find out the distances between world languages with the help of computer in order to achieve the optimal and unbiased results. The computer had to measure the distances between a chosen language and the rest of the languages in the set. Then, the computer had to put it in a matrix: closer to some languages and far away from the others.We have chosen Basque and the other isolated languages (e.g. Basque, Yukaghir, Nivh, Korean and Japanese) because they are not categorically assigned to any language family. It is interesting to compare how the computer and how different linguists place them in different language families and super-families: Indo-European, Turkic, Finno-Ugric, Tungus-Manchurian, Uralic, Ural-Altaic, Paleo-Asiatic; Sino-Tibetan; Austro-Asiatic; etc. We measure the distances between Basque and the other world languages in the same way we did it with Japanese (Tambovtsev, 1988: 19). We believe that under the circumstances even some traits and hints from the typological point may help to find the languages genetically related to Basque (Tambovtsev, 2001: 83-85). 
In this study we have used the procedures that are usually used in pattern recognition First of all, one must choose the features which should be necessary and sufficient (Zagoruiko, 1972). We believe the chosen features to be the most informative from the phonetic point of view. 
Basque, as any other human language, has a specific structure of the speech sound chain. It can be distinguished by its structure from any other language. Every language has a unique structure of distributions of speech sounds in its phonemic chain. The distribution of Basque vowels will not be considered until the second stage of the investigation, that is, if the data on the frequency of occurrence of consonants will not allow us to distinguish between the languages under investigation. Let's point out that consonants bear the semantic load in the word, not vowels. Therefore, it is more possible to understand the meaning of the message by consonants, rather by vowels. Some writing systems (Hebrew, Arabic, etc.) are a fair example of that since they denote only consonants. However, if we fail to recognise and distinguish two languages by consonantal groups, then we resort to the second stage of investigation, in which the frequency structure of occurrence of vowels in the speech sound chain is taken into consideration. While comparing languages, it is necessary to keep to the principle of commensurability. Having it in mind, it is not possible to compare languages on the basis of the frequency of occurrence of separate phonemes, because the sets of phonemes in languages are usually different. The articulatory features may serve as the basic features in phono-typological reasoning. 
First of all, it is the classification of consonants according to the work of the active organ of speech or place of articulation (4 features: labial, front, palatal, velar). 
775
Secondly, it is the classification from the point of view of the manner of articulation or the type of the obstruction (3 features: sonorant, occlusive, fricative). 
Thirdly, it is the classification according to the work of the vocal cords (1 feature: voiced). 
In this way, 8 basic features are obtained: 1) labial; 2) front; 3) mediolingual or palatal; 4) back or velar; 5) sonorant; 6) occlusive; 7) fricative; and 8) voiced consonants. 
It is necessary to mention that the total of these 8 features embrace all the main space of features which serve as the fundamental for the characteristics of consonants from the point of view of its production in any human language. Every language is being characterised as a point in the 8 dimensional space. This is why, it is important to take the features, which don't cross, i.e. which can be derived from one another. Our 8 features comprise the complete articulation system. One can hardly find any other basic articulatory feature. This is why, one can be sure that our typological classification is not just one more arbitrary typological classification of languages since it involves all the basic articulatory features of consonants. 
One should take the values of the frequency of occurrence of these 8 features in the speech 
chain of Japanese and compare them to those of the other languages. On the basis of the "chi- 
square" test and Euclidean distance, we have developed our own method of measuring the phono-typological distances between languages (Tambovtsev, 2001). However, here our measurements use the well-known formula of Euclidean distance: 
D = U x1 -x2)? + (y1 - y2)? + (z1 - z2)? + etc. 
where x1 - is the frequency of occurrence of the first feature in the first language; in this case the frequency of labial consonants in Basque; 
x2 - is the frequency of occurrence of the first feature (labials) in the second language 
(any other world language); 
y1 - is the frequency of occurrence of the second feature 
(dentals and alveolars) in the first language; 
y2 - is the frequency of occurrence of the second feature 
(dentals and alveolars) in the second language; 
z1 - is the frequency of occurrence of the third feature 
(palatals) in the first language; 
z2 - is the frequency of occurrence of the third feature 
(palatals) in the second language; etc. 
In this way Basque is compared to all the languages of the world taken for this study. Therefore, one can see that this formula allows us to take any number of features and any number of languages. It is very important that the number of features and languages is not limited. However, now one should bear in mind that it takes into account the frequency of occurrence of the 8 basic consonantal groups mentioned above and builds up the overwhelming mosaic of the language sound picture. 
We could not find the data on the frequency of occurrence of Basque language, that is why, we think that it is calculated for the first time. 
We computed several Basque texts from two books. The first of them is "Emakume biboteduna" - The woman with a moustache (Xabier Montoia, 1992) with 3 tales in it and the second one is "Sei Ipuin Amodiozko"- Six stories of love (Xabier Mendiguren Elizegi, 1992) with 6 tales. We thank two native speakers of Basque Kepa Sarasola and Maite Oronoz for their help and useful advice. We defined Basque phonological segment inventory the same as Ian Maddieson (Maddieson, 1981: 168) and R. L. Trask (Trask, 2001: 73 - 74). The first text that we computed was "Amusa Margarita" by Xabier Mendiguren 
Elizegi. After computing the first Basque text, we received the following results: 
Frequency % to all phonemes % to consonants 
Labial: 1021 5.09 9.69 
Front: 7496 37.36 71.15 
Palatal: 100 0.50 0.95 
Velar: 1919 9.56 18.21 
Sonorant: 4458 22.22 42.32 
Occlusive: 4233 21.10 40.18 
Fricative: 1845 9.19 17.50 
Voiced: 2480 12.36 23.54 
776
Consonant total - 10536 phonemes or 52.51%. 
Vowel total - 9530 phonemes or 47.49%. 
Consonantal coefficient, i.e. ratio of consonants to vowels - 1.11 
Basque first linguistic sample - 20 066 phonemes. 
The frequency of occurrence of Basque phonemes in the greater sample (130 550 
phonemes) of 9 different stories showed similar results. 
After computing the greater Basque sample, we received the following results: 
Frequency % to all phonemes % to consonants 
Labial: 7470 5.72 11.13 
Front: 46857 35.89 69.87 
Palatal: 602 0.47 0.92 
Velar: 12130 9.29 18.08 
Sonorant: 26410 20.23 39.38 
Occlusive: 28613 21.92 42.67 
Fricative: 12036 9.22 17.95 
Voiced: 17046 13.06 25.42 
Consonant total - 67059 phonemes or 51.37%. 
Vowel total - 63491 phonemes or 48.63%. 
Consonantal coefficient, i.e. ratio of consonants to vowels - 1.06 
Basque linguistic sample - 130 550 phonemes. 
First of all, we measured the phonostatistical distances between Basque and 40 Indo-European languages. One can see the details in Tab. 1. The typologically closest language is Gypsy (7.89), the farthest - Vedic (21.09). The closest Finno-Ugric language is Ludikov dialect of Karel (6.90), the farthest - the Northern dialect of Mansi (23.30), see Tab. 2. Turkic languages seem to be the closest to Basque: Kazah (5.31), Tofalar (5.96), Tuvin (6.83), Altai- Chalkan (7.00), Shor (7.04), Kirgiz (7.09), Uzbek (7.42), Hakas (7.70), Tatar-Kazan (7.73), Turkish (7.84). Ten Turkic languages are closer to Basque, then the closest Indo-European language - Gypsy (Tab. 3). Mongolian languages are also close enough: Mongolian 
(7.28), Buriat (8.48) and Kalmyk (8.81). Tungus-Manchurian languages seem to be rather far away (Tab. 4): the nearest Nanai (12.51). Paleo-Asiatic languages are also far away (Tab. 5): the closest Koriak (18.07). The isolated languages have the following order: Korean (9.64), Ket (13.05), Yukaghir (20.43) and Nivh (24.29). In order to see if the distances given above are small enough, let us compare 
them to the distances between some other languages. 
It is interesting to find out that Korean value of the consonantal coefficient obtained here (1.16) is close to those of Japanese (1.08), Oroch (1.00), Nanaj (1.02) and Ulch (1.10), which all belong to the Tungus-Manchurian language family (Tambovtsev, 1985). 
Having compared Korean and Japanese to some languages by the formula of Euclidean distance, we received the following phono-typological distances: Korean - Baraba Tatar (5.19); Korean Turkish (5.63); Korean - Ujgur (5.69). It is interesting to find out that the phonostatistical distance to Japanese is greater than to the three Turkic languages mentioned above: Korean - Japanese (6.66). On the other hand, it is important to see that Japanes is also close to the Turkic languages: Japanese - Ujgur (6.77). One can see that Korean is closer to the Turkic languages than Japanese. Let's compare some distances: Korean - Jakut (7.58) and Japanese - Jakut (8.26); Korean - Kazah (7.52) and Japanese Kazah (9.02); Korean – Turkish (5.63) and Japanese - Turkish (9.05); Korean - Baraba Tatar (5.19) and Japanese – Baraba Tatar (9.76); Korean - Uzbek (8.67) and Japanese - Uzbek (10.63); Korean Kazan Tatar (7.52) and Japanese - Kazan Tatar (11.07) and so on. It is interesting to 
measure the pholological distances between Korean, Japanese, Ket, Nivh, Yukhagir and other isolated 
languages to American Indian languages since some scholars consider it possible that they are related. We obtained the following ordered series between Japanese and Korean on the one hand, and on the other hand some of the American Indian languages, the first number is for Japanese, the second for Korean: Siriano (11.71 - 12.20), Inga (11.72- 9.13), Nahuatl (12.73 - 10.94), Mam (13.47 - 14.66), Kadiweu (13.52 - 15.69), Cofan (13.57 - 18.48), Guambian (14.91 - 11.69), Kaiwa (16.87 - 19.54), Pokomchi (17.05 - 17.34), Guarani (18.18 - 19.42), Totonak (18.28 - 18.42), Apinaye (19.06 - 15.38), and Secoya (20.87 - 23.57). So, one can see, that Japanese and Korean have more or less the same values of distances, which show that they are not typologically close to the American Indian languages taken for this study. One can see that Ujgur, 
777
Jakut, Kazah, Turkish, Baraba Tatar, Uzbek and Kazan Tatar are Turkic languages. Nanaj is a Tungus-Manchurian language. Therefore, one can notice that Japanese is closer to the so-called Altaic languages which include Turkic, Mongolian and Tungus-Manchurian languages. All in all 120 world languages were compared to Japanese. We cannot show all the distances here for the lack of space. However, the maximum distances were found for Japanese - German (22,24); Japanese – English (19.83); Japanese - Rumanian (15,08) and Japanese - Swedish (17.03). 
As a conclusion, we can state that speech sound picture of Japanese is rather far away from the 
Languages that are geographically close: Korjak (15.35), Nivh (20.23), Itelmen (17.85) or Chookchee (19.53). It was a surprise to us. Our data state that the speech sound pattern of Japanese resembles that of Ujgur - one of the Turkic languages spoken in the Middle Asia. The Ujgur people are often linked to the Old Turkic tribes, who used to live in the stepps of Southern Russia before the Tatar-Mongols captured them in the IXth century A.D. We must point out that it is not a coincidence since the other native Altaic 
people have a very similar data of closeness to Japanese. Turkic and Tungus-Manchurian tribes may have 
had a sort of common origin with Japanese. Our typological results may verify the Altaic hypothesis of Japanese and Korean origin, since Korean showed a greater closeness to the Altaic languages than Japanese. It is especially vivid, when the Austro-Oceanic, Austronesian, Austro-Asiatic and other languages do not show such a closeness: Japanese- Chinese (17.52); Burmanese (20.24); Tagalog (14.36); Indonesian (10.33); Hawaiian (23.97); Samoan (21.56), but for Sea Dajak (8.86), which is a Polynisian language. The Dajaks live on the Kalimantan island. 
As a conclusion, we can say that the phonostatistical distances allow us to find out how similar Basque is to the other world languages under investigation. 
References: 
Crystal, David 1992 An Encyclopedic Dictionary of Language and Languages. Oxford: Blackwell 
Garry, Jane and Rubino, Carl (eds) 2001 Facts About the World's Languages: an Encyclopedia of the World's Major Languages, Past and Present. New York: The H. W. Wilson Company. 
Ramstedt G. J. 1951 A Korean Grammar. Moskva: IzdInostrLiter. 
Ruhlen, Merritt. 1994 On the Origin of Languages (Studies in Language Taxonomy). Stanford: Stanford university press. 
Tambovtsev, Yuri. 1985 The consonantal coefficient in selected languages. Canadian Journal of Linguistics, Vol.30, # 2: 179-188. 
Tambovtsev, Yuri.1988 The linguistics distances among some languages of Asia. - In: The Study of Sounds. Tokyo. Vol.22: 17 - 34. 
Tambovtsev, Yuri. 2001a Kompendium osnovnyh statisticheskih harakteristik funktsionirovanija soglasnyh fonem v zvukovoj tsepochke anglijskogo, nemetskogo, frentsuzskogo I drugih indoevropejskih jazykov. [Compendium of the basic statistical characteristics of functioning of consonants in the speech chain of English, German, French and other Indo-European languages] Novosibirsk: Novosibirskij klassicheskij institut. 
Tambovtsev, Yuri. 2001b Funktsionirovanie soglasnyh fonem v zvukovoj tsepochke uralo- altajskih jazykov [Functioning of consonants in the speech chain of Ural-Altaic languages]. - Novosibirsk: Novosibirskij klassicheskij institut. 
Tambovtsev, Yuri 2001c Nekotorye teoreticheskie polozhenija tipologii upor'adochennosti fonem v zvukovoj tsepochke jazyka i kompendium osnovnyh group soglasnyh [Some Theoretical foundations of the typology of orderliness of phonemes in the language sound chain and compendium of statistical characteristics of the basic groups of consonants]. Novosibirsk: Novosibirskij klassicheskij institut. 
Trask, R. L. 2001 Basque. - In: Garry, Jane and Rubino, Carl (eds). Facts About the World's Languages: an Encyclopedia of the World's Major Languages, Past and Present. New York: The H. W. Wilson Company. 
Zagoruiko Nikolay G. 1972 The methods of pattern recognition and their application. [in Russian]. - Moskva: Sovetskoe radio. 
778
Tab.1 
Distances between Basque and Indo-European languages Basque 
Basque 0 
Gyps 7.89 
Spanish 8.03 
Moldavian 8.63 
Bengal 8.96 
Norwegian 9.34 
Latvian 9.36 
Persian 9.36 
Osetian 9.36 
Italian 9.68 
Esperanto 10.04 
Marathi 10.14 
Latin 10.54 
Portuguese 10.70 
Dutch 10.71 
Russian 10.84 
Rumanian 11.21 
Old Greek 11.27 
SerboCro. 11.44 
Lithuanian 11.64 
Arminian 11.67 
Old English 11.80 
Hindi 12.00 
Tadjik 12.05 
Albanian 12.36 
Greek 12.75 
Sanscrit 12.97 
Czech 13.32 
Slovak 13.72 
English 13.78 
Bulgarian 14.06 
Belorussian 14.31 
German 14.45 
Danish 15.71 
French 16.10 
Irish 16.16 
Prakrit 17.55 
Ukrainian 17.68 
Gudjarati 17.73 
Polish 18.52 
Verdic 21.09 
779

Investigation on the uses of temporal subordinators by NS and NNS in academic spoken English 
Tess Yu-Shan, Ke; Liang-Feng, Chen; Chien-Chung, Chen 
National Chengchi University, R.O.C. 
tess@ms21.url.com.tw 
Abstract 
This study was prepared to investigate the semantic functions of four common temporal subordinators : “after”, “before”, “when” and “whenever” in the Michigan Corpus of Academic Spoken English (MICASE). The purpose of this paper was to examine: (1) the semantic functions of these four subordinators in academic spoken registers (2) the different semantic distributions by comparing the performance of American English native speakers (NS) and non-native speakers (NNS). In the first part, the frequency of each subordinator will be presented and subsequently the semantic functions of each will be investigated. In the second part, the emphasis will be put on the different semantic distribution of NS and NNS. 
In this study, we found out that the similar semantic functions of these four temporal subordinators displayed and NS and NNS showed different patterns on the semantic functions of “ after”, “before” and “whenever” clauses while they are the same on the semantic senses of “when” clauses. The result derived from MICASE are hoped to provide explicit pedagogical implications for second language teaching and learning. 
1. Introduction 
In 1987, Graeme conducted a corpus analysis on the linguistic devices used to express temporal frequency in academic written English. Among his findings, the most frequent type was subordinators which contain six items. Those words are: when, after, before, until, as and whenever. This study was prepared to investigate the semantic functions of four common temporal subordinators among these six items. They are “after”, “before”, “when” and “whenever”. 
By comparing how native and non-native speakers use “before” and “after” differently, we hope our study can provide insights into the acquisition of “before” and “after”. Quirk et al. (1985) proposed various functions of the two subordinators. We examined whether, when they are used in a temporal manner, the two subordinators display the distribution of semantic functions described by Quirk et al. The distinction between native and non-native usage was also one of our concern. 
The semantic usage of time sequence is not limited to the two words “before” and “after”. Literature records that the “when” subordinator not only displays temporality but in many cases show the semantic usage of “before” and “after”. Kennedy (1998) classified “when” into eight different senses. Two of them are synonymous with “before” and “after”. He has also investigated three corpora (LLC, LOB and Wellington) for the proportions of the eight senses of “when”. Our result of proportions would be compared with his. We further examined that under situations when “when” are used as “before” and “after”, whether they show different semantic functions as usual “before” and “after” do, and whether NS and NNS display different patterns. Finally, the difference between NS and NNS regarding the 
780
employment of semantic functions of ‘whenever’ was investigated. 
The organization of this paper is as follows: section 2 presents literature concerning our study. Section 3 discusses the methods we employ in conducting our study and the materials we use. Results and remarks are given in section 4. 
2. Literature review 
The traditional grammar defines the structure of a clause as a subject with its predicate. Clauses subsume many subtypes for different functions. One of those is the adverbial clause which serves to describe a cause, location or condition etc., and it can be omitted without affecting the host clause (Greenbaum et al., 1996). Therefore, the adverbial clauses frequently appear to be subordinate clauses which are dependent on the main clauses. A clause which fulfills a grammatical function inside another clause is called a subordinate clause. Subordinate clauses have specific structural features that distinguish them from main clauses and serve a range of grammatical functions. Their relationship with the main clause should be connected by the subordinators. According to Quirk et al. (1985), a subordinator is a conjunction connecting main clause and subordinate clause. Namely, the subordinating conjunctions relate the adverbial clause to other clauses. 
Among the adverbial clauses, the most common use is to denote the time sequences of some related events. Those are so-called “temporal adverbial clauses” defined by Quirk et al. (1985) as an adverbial clause of time which relates the time of the situation denoted in its clause to the time of the situation denoted in the matrix clause. Thompson and Longacre (1985) further classified time adverbials into time adverbs, time adverbial phrases and time adverbial clauses, but in this present paper, we focused mainly on adverbial clauses. 
Joo (2002) also give a simple interpretation that adverbial of time is a sentence combined with a temporal subordinator. Those temporal clauses indicating temporal sequence are introduced by different subordinators which denote different semantic meanings. Because our present study focuses mainly on “after”, “before”, ”when” and “whenever”, we will review some literature on the four subordinators in the following sections respectively. 
2.1 Semantic functions of “after” & “before” 
The uses of “after” and “before” are ubiquitous in English, but their sensitivities of semantic and syntactic functions to the contexts are less noticed. As a result, we intended to investigate their distinctive components. Several researchers have provided some useful classification of the semantic functions of “after” and “before”. Harper & Charniak tried to give the semantic functions of the two temporal connectives in terms of event sequences. They commented that ”before” means the anchor event of the main clause ends before the beginning of the main event of the subordinate clause. On the contrary, “after” implies the main event of the main clause begins after the end of the anchor event of the subordinate clause. In a similar point of view, Thompson et al (1985: 182) pointed out that “before-clauses are different from when and after clauses in that it is always the case that the event named in the before-clauses has not happened yet by the time of the event named in the main clause.” And “the event named in the before clauses is always incomplete with regard to the main clause event”. 
781
In addition, there are some peripheral meanings which may be performed by the two subordinators. Hudson (1998) identified that semantics of “after” may be combined with a causal meaning and sometimes “after “ can introduce a concessive meaning. However, besides the core meanings to indicate that the situation in the matrix clause occurred after or before to the situation in the subordinate clause, Quirk et al. (1985) gave a more comprehensive analysis on the semantic differences of ‘after’ and ‘before'. They singled out several categories as below: 
(1) the sequential meaning of “after” and “before” may induce an implication of cause and effect respectively. 
(2) Before-clause may imply purpose and result. 
(3) Nonfactual before-clauses may imply preference. 
(4) the situation in the matrix clause may prevent that in the before-clause from taking place. 
(5) when the matrix clause is imperative, the sentence with a before-clause may imply a condition relationship 
2.2 Semantic functions of “ when” 
nn Keedy (1998) noted that “when” marks sequence and is synonymous with temporal subordinator “after” and “before”, as in [1] and [2] respectively. 
] W [1hen you finished, you went away. 
[2] She had only been in her room a few minutes when there was a knock. 
According to Graeme (1971) and Renaat (1996), two point events referred to in a “when” clause and its main clause may be paraphrased as ‘just before, during or just after’ and two point events convey a sense of causality. Put it more precisely, in some cases, “when” clauses not only express sequentiality but also convey causal implication, as shown in [3]: the “when” clause leads to the causal interpretation that the riots stopped because of the police's intervence. 
[3] The riots stopped when the police intervened. 
In addition to temporal sequentiality, “when” is also synonymous with “whenever” if “when” marks indefinite or timeless frequency or iteration, as in [4]. (Kennedy and Quirk et. al) 
[4] He used to throw his children bananas to eat when they were hungry. 
2.3 Semantic functions of “whenever” 
irk Qu(1985) defined semantic uses of temporal subordinator “whenever”, as in the following: (1) “whenever” is primary used to introduce a frequency adverbial, denoting the situation is repeated. (2) “whenever” may induce an implication of cause. (3) “whenever” may combine time and condition. 
To sum up, we can conclude the semantic functions of “after”, “before”, “when” and “whenever” based on native speakers’ intuition, on which the traditional linguistic research heavily relies. However, the performance of the spontaneous speech, which was collected as a corpus, may not necessarily reflect what those classifications assumed completely. Furthermore, the deviation may appear to be even more marked for non-native speakers. Therefore, we will explore the aforementioned four subordinators by means of an empirical corpus analysis. 
782
3. Methods and materials 
3.1 Data resources 
The data for our linguistic analysis is drawn from MICASE, The Michigan Corpus of Academic Spoken English ( Simpson et al. 2000). Since MICASE consists of the database of native and nonnative English speakers, we take the liberty of using the database of MICASE to investigate the different uses of temporal subordinate clauses between American English native speakers (NS) and nonnative English speakers (NNS). For our data, we have used a subset of MICASE. 
3.2 Identification 
We used Wordsmith computational software to analyze the four temporal subordinators. Firstly, we searched the corpus for "after” “before” “when” and “whenever” in specified contexts (left and right 5 words of key words). The concordance results we obtained were 800 and 959 entries for “after” and “before” clauses in the database of NS respectively, 57 and 58 entries for “after” and “before” clauses in NNS’ database respectively, 3520 and 196 entries for “when” clauses in NS’ and NNS’ database respectively, and 76 and 5 entries for “whenever” clauses in NS’ and NNS’ database respectively. Since the entries for “when” clauses of NS were too many, we only extracted 1000 from 3520 entries for our analysis. After the concordance results in specific contexts were displayed, we classified their semantic meanings in a manual manner that we identified their whole transcriptions because the semantic meanings of “after” “before” “when” and “whenever”-clauses were not fully identifiable only by viewing their concordance results. The semantic meanings of these four kinds of temporal clauses were obtained more accurately by means of having access to their discourse contexts. 
3.2.1 Identification of “after” and “before” 
Our identification of the semantic meanings was based on Quirk's semantic functions of “after” and “before” clauses. However, on the basis of Quirk's definitions (1985) there were some gray areas between the different semantic functions of “after” and “before” clauses, especially in differentiating the cause-effect and purpose-result. Some entries regarded as difficult differentiating their semantic functions were put aside. For the sake of readers’ conveniences, Quirk's definitions of semantic meanings were illustrated again, and the examples were drawn from MICASE to characterize the semantic meanings of their own. 
(1 ) “after” and “before” indicate time sequence 
[1] S1: or for Jeff the program director, who who they're they_ again i mean they they they jumped to this concern before i did. [ADV285SG135] 
(2) the sequential meaning of “after” and “before” may induce an implication of cause and effect respectively. “Before” clause in [2] is the effect because the event referred to in the main clause provides the cause for the situation in the “before” clause. 
[2] S3: no it's not being we don't, yeah. we don't believe it's coming from the mitochondria because the calcium occurs before the gradient changes occur so we monitor the, gradient changes at the mitochondria with D-I-O-C-six or with J-C-one, [COL200MX133] 
Another “after” clause also describes the cause and effect implication. 
783
[3] S3: this could be an encounter between an m- a T-cell and an antigen, that could help fight off infection. after the infection is over and the antigen goes away [COL200MX133] 
“After”-clause indicates the cause of T-cell's disappearance or that of transforming to memory to help the immune system. 
(3) “After” and “Before”-clause may imply purpose and result 
Another semantic function of before clauses is to indicate purpose and result with the example in MICASE, as shown in [4]. 
[4] S2: well you might wanna meet her before you blow her off. [ADV700JU047] ” ] 
The speaker S2 suggested that S5 had better meet her advisor so that she does not annoy her advisor. Example [5] illustrates the “after”-clause which has a purpose-result implication: 
[5] S1: She is so dazed after she looks at the clock. [SGR385SU057] 
(4) Nonfactual before-clauses may imply preference, or implausibility. 
We did not find any entry denoting preference or implausibility on the basis of Quirk's definition. 
3.2.2 Identification of “when” 
The other focus of this paper was to analyze the semantic uses of “when” clauses functioning as temporal sequence “after” and “before”. Firstly, we identified “when” clauses functioning as “after” and “before”. Then, we subcategorized the semantic uses of “when” clauses functioning as “after” and “before” on the basis of Quirk's semantic functions of “after” and “before” clauses: cause-effect, purpose-result and temporality. Some examples were extracted from MICASE to make this subcategorization clear. 
(1) “When” clauses are synonymous with temporal subsequence, “after”. 
] [1] S1: when you're finished with your vocab quiz, would you, turn to the required readings… [LES215MU056 
(2) “When” clauses denote the combination of temporal subsequence and cause-effect, “after”. 
] [2] S3: the way that you usually do that... is you add an inducer. an inducer is a molecule that can bind the repressor, and when it binds the repressor it changes the conformation state… [LES175SU079 
(3) “When” clauses imply the combination of temporal subsequence and purpose-result, “after”. 
[3] S5: When I finally combined all three facets, I got a big fat zero..[LES335JG065] 
(4) “When” clauses are synonymous with temporal prior-ness, “before”. 
] [4] S6: cuz people didn't know about the the reversal until this morning when they opened up the New York Times and saw the poll in there and the, story about the reversal.[ LES220SU140 
3.2.3 Identification of “whenever 
As for the other temporal subordinator “ whenever”, the semantic uses of it are divided into three semantic functions: repetitiveness, condition and cause (Quirk, 1985), as shown in the following. 
(1) “whenever” clauses imply repetitiveness 
[1]S5: whenever you talk about those turtle populations i always think about the uh, study that...[LES425JG077 
784
(2) “whenever” clauses imply condition. 
[2]S2: no but we get Shannon we get to use his swimming pool whenever we want. we're gonna have a, chemistry reunion at his at his pool.[LAB200JU018] 
(3) “whenever” clauses denote cause. 
9] [3]S1: but instead, the law was design- to protect, m- minority voters whenever they confront more difficulties than whites, in trying to cast their ballots [STP095SU13 
4. Results and Discussion 
4.1 After vs. Before 
Table 4.1 The distribution of temporality in “after” and “before” clauses 
After Before NS 
(800) NNS 
(58) NS 
(959) NNS 
(57) 1. Temporal 27.75% 
(222) 31% 
(18) 35.25% 
(338) 40.35% 
(23) 2. Non-temporal 72.25% 
(578) 69% 
(40) 64.75% 
(621) 59.65% 
(34) 
Table 4.1 shows that for native speakers, among 959 “before” tokens, 338 were used as a temporal subordinator. Both finite and non-finite clauses were counted as temporal clauses. The remaining 621 tokens were used in a non-temporal manner. They were either used as adverbs or prepositions. The percentage of “after” used as a temporal subordinator was lower than that of “before”. This was so because that in our data more “after”s than “before”s were used as adverbs and prepositions. For non-native speaker data, they showed the same pattern as that of native speakers. The percentage of “before” used as a temporal subordinator was higher than that of “after”. 
Table 4.2 The semantic distributions of “after” and “before” 
SEMANTIC FUNCTIONS After Before NS 
(222) NNS 
(18) NS 
(338) NNS 
(23) 1. Cause & Effect 22% 
(49) 38.9% 
(7) 3.55% 
(12) 4.3% 
(1) 2. Purpose & Result 4.1% 
(9) 0% 
(0) 8.6% 
(29) 0% 
(0) 3. Preference & Implausibility 0% 
(0) 0% 
(0) 0% 
(0) 0% 
(0) 4. Conditional 1.8% 
(4) 0% 
(0) 0.29% 
(1) 0% 
(0) 
785
Table 4.2 indicates that, for NS, among 338 tokens when “before” was used as a temporal subordinator, there were 12 tokens when “before” clauses and their main clauses also bore a relation of cause and effect, 29 tokens a relation of purpose-result, and 1 token a conditional relation. For the relation of preference-implausibility, the number was zero. NS employed “before” clauses to represent different semantic meanings except “preference and implausibility”. When the tokens were converted into percentage, the rates were 3.55%, 8.6%, 0% and 0.29% respectively. For the subordinator “after”, the number of tokens were 49, 9, 0 and 4 in the four separate semantic functions. When converted into percentage, the rates were 22%, 4.1%, 0% and 1.8%. When the distributions of semantic functions of “after” and “before” were compared together, there did not show much a difference. Only that the distribution of “before” was more concentrated on the relation of purpose-result, and the distribution of “after” was more concentrated on the relation of cause-effect. Nevertheless, we noticed that in our data there were many cases when the “before” clauses and their main clauses showed a conditional relation which can not be counted under our categorizing framework. Our way of categorizing was adopted from Quirk et al.(1985). Under their framework, a “before” clause must be in an imperative mood to be counted under the conditional category. But there were many “before” clauses which showed a conditional relation with their main clauses but were not in an imperative mood. The “before” clause in a sentence like “We need to hang out here before the whole thing starts. [TOU999JU030] “ has a conditional relation with its main clause but is not in an imperative mood. In Quirk et al.(1985), they did not mention this kind of sentences. Their classification was not detailed enough. 
In the non-native data, there was cause-effect usage in both “before” and “after” clauses. There were two possible reasons. One was that those NNS had already learned to express cause-effect relation in temporal clauses. The other explanation was that besides showing time sequence, temporal relationship is strongly related to cause-effect. Thus it is not possible to separate the two. 
4.2. When 
Table 4.3 Comparison with other corpora ( LLC, LOB and Wellington corpora are quoted from Kennedy 1998) 
Sense of when MICASE 
(spoken academic American English) LLC 
(spoken British English) LOB 
(written British English) Wellington 
(written New Zealand English) Tokens % Tokens % Tokens % Tokens % After 140 14 257 20.3 763 30.1 113 29.4 Before 31 3.1 53 4.2 146 5.8 27 7.0 
Kennedy (1998) compared three corpora, one spoken and two written. The results showed that the proportions of tokens in which “when” is used synonymously as “before” and “after” were higher in the two written corpora than the spoken one. Our results were 14% and 3.1% for “after” and “before”, lower than 20.3% and 4.2% for “after” and “before” from LLC. Our results and Kennedy's results together seem to suggest that the proportions of cases of “when” used as “after” and “before” are higher in 
786
written corpora than oral ones. However, Kennedy (1998:151) cautions us that “The greater proportion of when as a synonym for after in the LOB and Wellington corpora may, of course, be a reflection of the reporting-narrative nature of particular genre in the corpora rather than necessarily reflecting the written medium as such.” Whether the discrepancy resulted from written-spoken difference or genre difference obliges further studies. 
4.4 The frequencies & percentages of “ when” for NS: 
Total Cause & Effect Purpose & ResultTemporality Fre. Per.% Fre. Per.% Fre. Per.% Fre. Per.% After 140 14 50 5 10 1 80 8 Before 31 3.1 0 0 0 0 31 3.1 whenever 74 7.4 * * * * 74 7.4 Subtotal 245 24.5 50 5 10 1 185 18.5 Total 1000 
4.5 The frequencies & percentages of “ when” for NNS: 
Total Cause & Effect Purpose & ResultTemporality Fre. Per.% Fre. Per.% Fre. Per.% Fre. Per.% After 22 11 6 3 1 0.5 15 8 Before 3 2 0 0 0 0 3 2 whenever 16 8 * * * * 16 8 Subtotal 41 21 6 3 1 0.5 34 17 Total 198 
Observing from Table 4.4 and 4.5, we found NS and NNS displayed a similar pattern in using “when” as “after” or “before”. The most pervasive use of “when” by both NS and NNS was the sense of “after” in the percentages of 14 and 11 respectively. To give a close scrutiny on the basis of the semantic categories provided by Quirk et al., we also examined the distribution in the uses of “after” and “before”. With respect to “after”, NS employed 5% denoting “cause and effect” and only 1% indicating “purpose and result”, likewise, NNS made use of 3% and 0.5% in the same semantic meanings. On the aspect of “before”, no case was found to substantiate the two semantic categories mentioned above, whatever the status of native-ness of the speakers. However, 8% of “after” and approximately 3% within “before” was found to express temporality. This may account for the fact that most English speakers, NS or NNS, do not get accustomed to using “when” to substitute “after” or “before” in their peripheral meanings. That is, they generally tended to produce time sequences through the primary meaning of “after” and “before”, i.e. temporality, in the form of “when”. This confirms what Kennedy (1998) suggested, “after” and “before” are used to mark “prior-ness’ and “subsequence”, while “when” is commonly used to mean 
787
“simultaneity”. 
Also, the proportion of occurrence of “whenever” was almost equivalent in NS (7.4%) and NNS (8%), which demonstrated another consistent dimension of “when” in this spoken corpus. Although there seemed to be no differential effect on the performance of “when” by NS and NNS, however, what of note was the relative proportion of “cause and effect” and “purpose and result”. The tokens of the NS ( 6%) were about two times than the NNS ( 3.5%), implying that NS may have more sensitivities to the subtle distinctions of the semantic subcategories thereof. 
4.3. Whenever 
Table 4.6 Distribution of the senses of whenever of NS &NNS 
Senses of whenever NS NNS Tokens % Tokens % Repetitiveness 47 61.8 5 100 Causality 2 2.6 0 0 Condition 27 35.6 0 0 Total 76 100 5 100 
“Repetitiveness” accounts for 61.8% of the tokens of “whenever”. This indicates that NS primarily use “whenever” to indicate repetitiveness. In contrast to the most prototypical meaning “repetitiveness”, “whenever” clauses are seldom used to convey causality by NS. Compared to that of NS, NNS's usage of “whenever” clauses seems limited to situations denoting repetitiveness. 
5. Conclusion 
This present paper has attempted to investigate the different uses of temporal subordinators by NS and NNS via the bulk of the data generated from an academic speech corpus. The consequences obtained unraveled some questions in relation to second language acquisition, and shed new light on language teaching and learning. To summarize, several crucial findings were reported as follows: 
First, with regard to “temporality”, there was a similar pattern of both NS and NNS. The “non-temporal” uses were sparser since all the subordinators under examination are inherent to express time sequence. Particularly, “after” was more preferred than “before” when “temporality” were involved. Moreover, speakers were found to be prone to use “after” and “before” instead of “when”, which has the same semantic functions. Secondly, when dealing with the minute distinctions of semantic functions, NS and NNS revealed somewhat differences. In general, our findings showed that NS and NNS displayed discrepancy in the use of semantic meanings. NNS had only learned the most prototypical semantic meaning. With respect to the several subordinators investigated, NNS displayed more limited language capacity compared to NS. They employed only the “cause & effect” among the various semantic functions for “before” and “after” and only the “repetitiveness” for “whenever”. However, a limitation should be noted. In comparison to the data of native speakers, that of non-native data is relatively small. Therefore, the statistic of non-native data we obtained should be treated with more caution. 
788
Quirk's semantic classifications of “after” and “before” clauses are too restricted. His classification of “before” clauses as conditional appears to be confined to imperative; however, there are some cases which are in the form of imperativeness and lack imperative voices, but the main clauses and the “before” clauses indeed bear conditional relations. Moreover, Quirk's definition of “preference and implausibility” is not clear enough, and it appears to only apply to “before” clauses which denote metaphorical relationship between main clauses and “before” clauses. 
In addition, some directions could be followed from our study. Firstly, future studies could compare different discourse modes in academic spoken English between English native speakers and non-native speakers. Furthermore, further studies could be done on comparison of written and spoken academic English of native speakers with that of non-native English speakers. 
Although limitations exist in our study, it is hoped that this study could make a contribution to language pedagogy and offer some insight for future researches. The disparity of competence between NS and NNS highlights the importance that language teachers should provide authentic materials to help language learners aware of other semantic functions of subordinators. 
6. References 
Greenbaum S, Nelson G 1996 Positions of adverbial clauses in British English. World Englishes 15(1): 69-81. 
Givon T 1993 English grammar. 
Grice P 1975 Logic and conversation. In Cole P & Morgan J (eds). Syntax and Semantics: Speech Acts (3): 41-58. 
Harper , Charniak. Time and tense in English. 
Hudson-Ettle D M 1998 Grammatical Subordination Strategies in Differing Text Types in the English Spoken and Written in Kenya. Dissertation, Technischen Universitat Chemnitz. 
Joo Y S 2002 Teaching grammar from discourse for Korean students in Korea. 
Leech G, Svartvik J 1994 A grammar of English. 
Quirk R, Greenbaum S, Leech G, Svartvik J 1985 A comprehensive grammar of the English language. London: Longman. 
Renaat D Tense and modality in English before clauses. English studies: a journal of English Language and Literature. Nijmegen Netherlands, 60:720-44. 
Renouf A, Sinclair J 1991 Collocational frameworks in English. In Aijmer K, Altenberg B (eds), English corpus linguistics, pp 128-144. 
Simpson C, Briggs S L, Ovens J, Swales J M. 2000 The Michigan Corpus of Academic Spoken English. Ann Arbor, MI: The Regents of the University of Michigan. 
Thompson L 1985 Adverbial clauses. In Timothy Shopen (eds), Language typology and syntactic description, 3, pp171-234. 
789

Ambiguity in Russian Morphology 
Carole Tiberius, Dunstan Brown, Greville Corbett 
Surrey Morphology Group, School of Arts 
University of Surrey 
Guildford GU2 7XH 
{c.tiberius,d.brown,g.corbett}@surrey.ac.uk 
The nature of the relationship between frequency of use and grammar in natural language is poorly understood. In order to understand this relationship better, we will look at textual frequency distributions in a language which encodes a reasonable number of grammatical distinctions in its word forms, namely Russian. In this paper, we will focus specifically on the relationship between ambiguity and frequency. 
To do this we shall undertake a corpus analysis of Russian texts. We will use two corpora, the Uppsala corpus (Lönngren 1993, Maier 1994) and a corpus of Russian newspapers from the late 1990's, to check whether we find similar distributions across both corpora. 
We will convert the corpora to unicode encoding to increase reusability and sharability. The corpora will then be segmented and lemmatised. The lemmatisation will be based on a predictive, formal model of Russian morphology, namely Network Morphology (Corbett and Fraser 1993, Brown 1998). This model has been implemented in the inheritance-based formalism DATR (Evans and Gazdar 1996). The lemmatiser will be based on theorem dumps derived from the DATR theory, augmented with a larger set of fully inflected forms derived from the electronic version of Zaliznjak's (1977) dictionary (Ilola and Mustajoki 1989). 
Rather than going for a fully tagged corpus, we will take the output of the lemmatiser and use this labelled unresolved data to determine the amount of ambiguity in Russian text. In the first place, we will focus on the analysis of nouns and adjectives, since these are the categories where we expect to find most ambiguity in Russian. We will be able to establish how many words get more than one morphological analysis, how many of those are two-way ambiguous, how many are three-way or more ambiguous. We can then determine whether there is a relationship between ambiguity and the frequency distributions of a given feature value (e.g. dative case) within a subset of nouns from the inheritance hierarchy of the Network Morphology model. 
Insight into these ambiguity distributions in the two Russian corpora will be of value for practical purposes and will also contribute to our understanding of the effects that frequency has on grammar. 
References 
Brown, Dunstan 1998. From the General to the Exceptional: A Network Morphology Account of Russian Nominal Inflection, PhD, University of Surrey. 
Corbett, Greville G. and Norman M. Fraser (1993) Network morphology: A DATR account of Russian nominal inflection. Journal of Linguistics 29. 113-42. 
Evans, Roger and Gerald Gazdar. 1996. DATR: A Language For Lexical Knowledge Representation. Computational Linguistics 22. 167-216. 
Lönngren, Lennart (ed.) 1993. Èastotnyj slovar´ sovremennogo russkogo jazyka. Uppsala: Uppsala University. (=Studia Slavica Upsaliensia 32) 
Ilola, Eeva & Mustajoki, Arto. 1989. Report on Russian Morphology as it appears in Zaliznyak's Grammatical Dictionary. Helsinki: Helsinki University Press. 
Maier, I. 1994. Review of Lönngren (ed.) Èastotnyj slovar´ sovremennogo russkogo jazyka. Rusistika Segodnja 1. 130-136 
Zaliznjak, A. A. 1977. Grammatièeskij slovar' russkogo jazyka. Moscow: Russkij jazyk. 
790

Creation and utilisation of the MediaTeam Emotional Speech Corpus 
Juhani Toivanen, Tapio Seppänen, Eero Väyrynen 
MediaTeam, University of Oulu, Finland 
Abstract 
The MediaTeam Emotional Speech Corpus is currently the largest database of emotional speech for colloquial modern Finnish, containing simulated emotional content. The specific aim of the research is to investigate in detail the phonetic and phonological/linguistic correlates of basic or primary emotions in spoken Finnish, to develop statistical classification methods of emotional speech signals and to develop methods to utilise this knowledge in speech corpus search engines. The vocal expression of emotion in speech is approached utilising both the abstract phonological information on the type of pitch contour and the distribution of focal sentence accents and the concrete phonetic information on the actual implementation of F0 curves and other global (continuously variable) prosodic parameters. The annotation procedure reflects these two levels, the aim being to investigate both the linguistic and paralinguistic ways of the vocal expression of emotion. The algorithms developed for prosodic parameter estimation, the statistical classification of emotional speech, and the estimation of the degree of emotions can open up new opportunities for future Internet database technologies and methods for content-based information retrieval from audio databases. 
1. Introduction 
Human social communication importantly rests on exchanges of non-verbal signals, including the (non-lexical) expression of emotion through speech. It is well known that emotions play a significant role in social interaction, both displaying and regulating patterns of behaviour and maintaining the homeostatic balance in the organism. In everyday communication, certain emotional states (e.g. boredom and fear/nervousness) are probably expressed mainly only non-verbally since social conventions demand that, as a rule, negative emotions be concealed. Today, the significance of emotions is acknowledged across scientific disciplines – “Descartes’ error” is thus being rectified, and emotions and feelings are no longer regarded as “intruders in the bastion of reason” (Damasio 1994). 
The vocal emotion literature is large but somewhat fragmented. There exists nothing like a standard model of the expression of emotion in speech although various aspects of affect-related vocal features have been described. However, the research has concentrated on the vocal correlates of emotions in spoken English and some other major languages (e.g. French and German): practically nothing is known about the specific acoustic correlates of emotions in (continuous) spoken Finnish. So far, the expression of emotion in spoken Finnish has been studied mainly in the context of very short nonce syllables. 
The investigation outlined in this paper represents basic research, and fills an obvious lacuna in the literature on the topic. On an applied side, to develop automatic methods for determining the emotional state of the (Finnish) speaker is an equally important goal, with immediate and immense application potential e.g. in situations where speech is the primary mode of interaction with the machine. Human-computer interfaces could be made to respond differently, depending on the emotional state of the user. Content-based information retrieval is a very important field of application: search engines that could locate happy or sad speech in an audio database would be very useful. One of the aims of the research described in this paper is to develop such methods. 
One cannot directly extrapolate from the vocal emotion literature on other languages. The intonational structure of Finnish has not been described in detail and very little is currently known about the correlation between e.g. pitch patterns (sentence melody) and emotions in spoken Finnish. It should be stressed that, from the viewpoint of intonation, Finnish is substantially different from English, for example; one important difference is that rising tones are uncommon in non-dialectal spoken Finnish. Thus, the findings concerning the correlation between different nuclear tones and perceived emotions/attitudes reported for English in the large literature on intonation cannot, as such, be applied to Finnish. 
2. Some IT background: information retrieval from digital databases 
The current stages of the digital revolution – in particular, the Internet and the new ways of storing and retrieving information – have produced very large collections of digital data: for example, text, audio and video libraries. Such digital collections are now an important part of a properly functioning society. 
791
Consequently, specific methods for effectively browsing and searching such digital libraries are badly needed. 
From the viewpoint of the end user, the problem is how to find the required information from a plethora of increasingly large digital databases. The problem concerns mainly the currently booming media types, such as digital speech, music and image, where the search criteria often include semantic concepts. The current demand for automatic interpretation of digital data has resulted in intensive research on content-based retrieval. Theoretically, the central problem is how to narrow down the semantic gap between the concept-based and the content-based approaches to data indexing. The aim is thus to close the gap between the natural semantic concepts used by a person seeking information and the computer-interpretable database descriptors automatically derived from the data contents. 
Information retrieval applications are based on an understanding of the content of the file(s) and there are, basically, two approaches. On the one hand, human interpretation can be utilised to generate semantic keywords but this tends to be inconsistent over time, and is very expensive. On the other hand, automatic interpretation of data can be developed: it is cheap, replicable and scalable. The automatic interpretation may be wrong but it is consistently so: the search results, whether they be good or bad, are at least systematic. 
As for the development of search engines, a critical fact is that the digital databases are so massive: the organising and indexing of the material is very demanding, and can never be done manually. Therefore, at least semiautomatic means for the interpretation of complex data must be developed. 
3. Utilisation of prosodic features in information retrieval 
Compared with information retrieval for text-media, semantically rich content-based information retrieval for audio/video is still an unattained goal (Bregman 1990, Smeaton 2000). With modern text mining methods, it is possible to automatically categorise and summarise very large documents, and locate and extract the relevant textual information. Image, video and audio, by contrast, are so rich in content that such generic information retrieval applications are still far away. However, some progress has already been made. 
The development of automatic speech recognition can offer a partial solution to the problem of information retrieval from large and semantically rich (audio) databases: it enables the computation of topic-related keywords, thus locating those specific parts of the file where the interesting and relevant theme-related words occur. The basis of all speech retrieval applications is some kind of recognition (where phone recognition precedes word recognition). Full speech recognition transcribes spoken utterances into a text which can be analysed to achieve a syntactic and semantic description of the utterances, whereas single word recognition only locates the relevant “spots” in the spoken messages. Full speech recognition is speaker-dependent and expensive, and requires training. Word spotting applications are often sufficient and they work because of the pre-defined vocabulary (the search is for these words only). 
Speech signal interpretation can be enhanced by means of prosody-based search features: general prosodic features of speech can be utilised to automatically chart the distribution of cohesive paragraphs, also known as paratones, in the speech data. Prosodic features such as F0 and intensity level can be used for topic and phrase boundary identification (Swerts and Ostendorf 1997). The speech can be segmented into audio paragraphs/paratones, using acoustic/prosodic information, after which automatic speech recognition can be applied to each such unit. By means of the more intelligent search features, the user of the information retrieval application could find all those lexically/syntactically relevant, and prosodically cohesive, parts of the audio which deal with a certain pre-defined topic. 
To improve the current content-based information retrieval methods, it may be necessary to add prosody-based elements to the list of search features. Prosody signals not only the textual structure of the discourse – for example, the beginnings and endings of topics, and turn-taking – but also the affective state of the speakers. Speech is, in actual fact, quite rich in different kinds of indexical markers relating, for example, to the affective/medical status or the socio-economic/professional background of the speaker (Laver 1994). These features are, to some extent at least, speaker- and language-independent (Toivanen 2001). 
Prosodic cues related to emotional content can be analysed at many different but intersecting levels by taking into account, in particular, the time-length of the speech segment. On the one hand, emotions can be detected with some accuracy in very brief speech segments/syllables of 200 ms in which the acoustic expression is, of course, entirely based on (paralinguistic) features of voice quality (Laukkanen et al. 1997). Such short units of emotional expression could be called affect bursts or semiautomatic biomechanical responses. On the other hand, phonological intonation contour types, as 
792
well as different types of accentuation, which signal affective states at the level of the utterance or speech event, may be more cognitively mediated, and more linguistic, components of the expression of emotion. 
At the signal level, a number of continuous variables convey emotions and affect. These variables are F0-related, intensity-related, temporal, and spectral features of the speech signal, including, for example, average F0 range, average RMS intensity, average speech/articulation rate, and the proportion of spectral energy below 1,000 Hz. These features, most of which are easily measurable from the signal, yield information about the paralinguistic aspects of speech. At the symbolic level, the distribution of tone types and focus structure in different syntactic structures can convey emotional content. Typically, these features convey linguistic information about how emotions are expressed in spoken language. In the research, the analysis of both of these two levels will eventually be necessary. 
The kind of prosodic information briefly described above could be valuable for database indexing and retrieval of audio material. Currently available search engines cannot make use of the acoustic parameters underlying “angry” voice, for example, but more intelligent robots could be designed to utilise acoustic information. As things stand now, prosodic features are an unexplored area in research on content-based information retrieval. 
4. Emotional speech databases 
The computer, or the search engine, can evaluate the emotional content of spoken messages on the basis of prosodic/acoustic information only if the computer is explicitly taught the way in which human listeners perceive emotions in speech. Thus, in order to find out the quantifiable relationship between prosodic features of speech and perceived emotions, listening experiments and subjective evaluations are necessary. 
Studies of emotional speech features are often based on recordings of professional actors simulating a number of (basic) emotions in a studio environment. The researchers then have a panel of listener- judges label the emotional content of each speech sample. Usually, the speech consists of a few standard phrases or sentences, the lexical content remaining the same for each simulated emotion. If the listener-judges can “hear” the intended emotions, it can be argued that the recordings are representative of the affective states under investigation (Iida et al. 1998). Because the speech material is identical, a detailed study of the acoustic/prosodic structure is justified: as the lexical content remains the same, the (spectral) differences between the speech samples, for example, should be principally caused by the intended emotional state, not by the varying sound structure. 
It is difficult to find authentic emotional speech data. On the one hand, if the recording is made in a (“genuine”) noisy and uncontrolled real-life situation, the stored speech signal is likely to be weak or distorted, which largely excludes a principled acoustic analysis of the data. On the other hand, ethical considerations are an important issue: is it morally acceptable, or even completely legal, to make the speaker, in a laboratory environment, genuinely angry, sad or scared for the purposes of collecting speech material? 
Certain previously existing speech data can be useful for research on the acoustic correlates of emotions. The radio news broadcast of the Hindenburg crash is perhaps the most famous publicly available emotional speech source. Similarly, recordings made in the context of interactive radio programs can contain interesting emotion-laden speech data. The problem is, however, that the researcher cannot verify which emotion the speaker intended to produce. Furthermore, copyright restrictions often apply to such data: one cannot take for granted that the data can be accessed even for strictly limited research purposes. Indeed, most of the emotional speech databases are not available for distribution because of copyright restrictions. 
5. Research utilising the MediaTeam Emotional Speech Corpus 
5.1. Speech data 
The MediaTeam Emotional Speech Corpus is currently the largest database of emotional speech for continuous spoken Finnish. Fourteen professional actors, eight men and six women, aged between 25 and 50, were recruited to simulate basic emotions. Each speaker was asked to read out a Finnish passage of 120 words in a “neutral” or “natural” tone of voice; the text was semantically as neutral as possible, dealing with the nutritional value of crowberry. Then each speaker was asked to read out the text simulating the following emotions: “happiness”, “sadness”, “fear”, “anger”, “boredom” and “disgust”. The speakers were allowed to retake the reading if they were not satisfied with the first rendition. In addition, the speakers acted out two pre-written dialogues containing emotional lines of 
793
varying length. Thus the corpus contains linguistic units with specific emotional content ranging from short exclamations to monologues of approximately 30 seconds. The speech material was digitally recorded with DAT in an unechoic studio to produce a 44.1-kHz, 16-bit CD-format recording. 
5.2. Acoustic analysis 
The acoustic analysis was carried out by means of analysis software, the F0 Tool, implemented in the MATLAB language. The software first distinguished between the voiced and voiceless parts of the speech signal (Ahmadi and Spanias 1999) and then determined the F0 contour for the voiced parts of the signal (Titze and Haixiang 1993). The F0 contours were determined by using cepstral analysis and interpolating waveform matching. 
Some 40 acoustic/prosodic parameters were automatically computed from the speech signal (see Appendix for a complete list of parameters). The parameters were F0-related, intensity-related, temporal and spectral features (below, the term “segment” refers to a part of the signal of varying duration, which may be realised as silence or as voiced or voiceless speech; thus the term does not describe any linguistic/phonological unit). The most significant parameters are listed below. 
The general F0-based parameters were the following: mean F0, median F0, maximum F0, minimum F0, F0 range, 95th/5th fractile of F0, the range between the fractiles, and F0 variance. The parameters concerning the “dynamics” of F0 were: average F0 rise/fall during a continuous voiced segment, average steepness of F0 rise/fall, maximum F0 rise/fall during a continuous voiced segment, maximum steepness of F0 rise/fall, and jitter. The intensity-related features were: mean RMS intensity, median RMS intensity, maximum RMS intensity, minimum RMS intensity, intensity range, 95th/5th fractile of intensity, the range between the fractiles, intensity variance, and shimmer. The temporal parameters were: average duration of voiced segments, average duration of voiceless segments shorter than 500 ms, average duration of silences/pauses shorter than 400 ms, average duration of voiceless segments longer than 500 ms, average duration of silences/pauses longer than 400 ms, maximum duration of voiced segments, maximum duration of voiceless segments, maximum duration of silences/pauses, silence-to-speech ratio and voicing-to-pauses ratio. Finally, the spectral features measured concerned the proportion of low-frequency energy (<500 Hz and <1000 Hz). 
5.3. Recognition and classification of emotional content of speech 
All the speech data was tested with listener-judges: 20 listeners, female university students aged between 19 and 28, listened to each speech sample and chose the emotional label which best described the affective content of the sample. In the test, the emotions to be differentiated were: neutral, anger, happiness and sadness, which can be considered to represent the basic emotions (Izard 1977). On an average, the listeners recognised the emotions with an accuracy of 84.7%. 
In the automatic recognition of emotions, k-NN (k-Nearest Neighbour) was used as a classifier. In the first experiment, the assumption was that the speaker had been authenticated earlier and the new emotional speech samples were compared with the corresponding speech models in the database. This experiment scenario was tested statistically with leave-one-out method. The automatic recognition of emotion was quite satisfactory, the average classification level being about 80 % (k-NN with k equal to 1). Further experiments are now being carried to investigate the accuracy of completely speaker-independent automatic classification of emotions (where the emotional speech samples of a person not included in the database are classified). 
The results of the listening tests (>84 % for the four basic emotions) are excellent in the light of the results reported in the literature. According to Scherer et al (2001), in the “Western cultural context” listeners can recognise basic emotions (neutral, anger, fear, happiness and sadness) on the basis of acoustic features with an average accuracy of 66 %. However, if more emotions are to be recognised, the task become more difficult: an accuracy of 60 % has been reported for anger, happiness, sadness, fear, disgust, love, pride and jealousy (Scherer 1995). Polzin (1999) reports a general rate of 70 % for recognition of emotion in English (sadness, anger and neutral). The conclusion is that the speech samples used in the present investigation contained clear acoustic cues related to emotional content as the recognition rate was so high. On the other hand, it should be pointed out that the speakers were professional actors: thus powerful vocal portrayals of emotions were to be expected. 
The preliminary results of the automatic recognition and classification of emotions seem very promising. Summarising research on the topic, Bosch (2000) argues that an accuracy of 60 per cent may be the best one can get in a (speaker-independent) limited happiness/joy, anger, sadness/grief discrimination test. McGilloway et al. (2000) suggest that 50 % correct classification can be achieved in automatic systems discriminating among five emotional states. Again, it must be emphasised that the 
794
nature of our data was probably conducive to good classification results: the speech samples contained acoustic cues which the speech analysis algorithm could extract and which were realised with clearly differentiated values for different emotional states. Furthermore, it should be pointed out that the successfulness of the speaker-independent classification of emotion is yet to be confirmed. 
5.4. Linguistic analysis 
Although it is true that the acoustic parameters described in section 5.2. reveal something about the general prosodic patterns in the speech data, a more linguistically oriented analysis is needed to investigate speech melody or intonation. The acoustic parameters can be measured completely independently of the linguistic/phonological structure of the speech data: for example, the parameters describing the rate of change (“steepness”) of the F0 movement do not indicate whether the most dynamic F0 contours are connected with sentence accent or not. Thus, automatic measures of mean F0-based values for the whole speech material do not provide information on the course of pitch at the utterance level, and they certainly do not provide any direct information on the linguistically relevant variation in pitch. To investigate speech melody, auditory analysis/annotation is necessary. The relation between syntax and intonation is the obvious place to start: it can be investigated whether statements, for example, contain a greater proportion of non-falling tones in certain emotional states. Our basic hypothesis is that emotional colouring might, in some cases, counteract the normal tendency for Finnish utterances to end with falling intonation. All in all, the speech material must be analysed in terms of linguistic/phonological features and units to find out how the functionally important parameters that are not directly measurable from the acoustic signal reflect different emotions. So far, the data has been analysed only in terms of automatically measurable acoustic parameters. 
The distribution of sentence accents and pauses is probably very important. It can be assumed that certain emotions – anger, for example – might be accompanied by a high percentage of strongly accented words. To chart the distribution of different types of accents in the speech data, the corpus must be transcribed manually. Similarly, frequent pausing is probably to some extent an emotion-specific feature of speech: a low speech-to-silence ratio can be an indication of “passive” emotions, e.g. sadness and boredom, but the type of pause may also be important. Unlike the speech-to-silence ratio, the percentages of different types of pauses are not automatically measurable. Therefore, auditory interpretation and manual measurements are needed to investigate the locations and types of pauses. In the analysis, it is thus important to come up with some kind of (syntactic/pragmatic) taxonomy of pauses, in addition to measuring e.g. the pause/speech ratio. 
To describe these features of the speech data, the corpus is currently being annotated utilising Transcriber, a tool for assisting the manual annotation of speech signals: the tool is especially handy as it has a graphical user interface for segmenting long recordings and transcribing them. The aim is to create interlinear transcription, where each word is annotated with prosodic and syntactic information (displayed under the word). If a piece of annotation is clicked on, the corresponding extent of audio signal is highlighted. As the recording is played back, the corresponding annotation sections are highlighted (waveform as well as text). The annotation records are stored in a spreadsheet. 
The data is being annotated in accordance with the following principles. For intonation, the ToBI model is used. ToBI (Tones and Break Indices) is a framework for developing generally agreed-upon conventions for transcribing the intonation and prosodic structure of spoken utterances in a language variety. Originally developed for English, the system has been used in the description of the prosody of a number of languages, including Dutch, Spanish, Italian, Japanese and French – to date, the ToBI framework has not been extensively applied to Finnish. In the ToBI system, the F0 contour is associated with textual information. The system includes four tiers: the ortographic tier (ortographic form of words), the break-index tier (a 5-step scale describing the nature of breaks in speech), the tone tier (including e.g. interphrasal accents and boundary tones), and the miscellaneous tier (for extra-linguistic information, for example). 
In the annotation used in this investigation, Välimaa-Blum's suggestion (1993) is followed: two complex accents, L+H* and L*+H, two boundary tones, H% and L%, and two initial boundary tones, %H and %L, are recognised. In spoken Finnish, according to Välimaa-Blum, the neutral prosodic contour is a gradually declining pattern of L+H* accents, with a low boundary tone (L%) in final position. The L*+H accent, involving late accentuation instead of the normal lexical stress on the first syllable in a word, is assumed to be typical of emotional speech acts: the entire utterance may consist of emphatic accents with delayed F0 peaks (Iivonen 1998). 
In addition to the ToBI system, prosody is annotated using the “traditional” contour analysis framework. At the clause/sentence level, the intonation is transcribed as representing one of the following prosodic options suggested for Finnish by Iivonen (1998): non-emphatic pattern, expressive 
795
pattern, and progredient pattern. The non-emphatic intonation is the basic pattern used in non-affective utterances, consisting of a descending F0 curve with rising-falling peaks on the accented syllables (i.e. the declining succession of L+H* accents in Välimaa-Blum's description). The expressive intonation involves a clear ascending F0 curve (at the end of a clause/sentence): in Finnish, this may indicate surprise, appeal or any clear emotional reaction. The progredient intonation is sometimes referred to as “comma intonation” (non-terminal intonation) in Finnish, involving a relatively high level of F0 before the boundary, in comparison with the low terminal intonation at the end of a declarative. Thus, while the non-emphatic pattern often involves laryngealisation (creak) with a very low F0 value in final position, the progredient intonation is typically completely periodic throughout. 
Each word is annotated with respect to accentuation/prominence applying the descriptive system used by Suomi et al. (in press). It has been assumed that spoken Finnish contains at least three degrees of accentuation: words are unaccented, moderately accented or strongly accented. In the investigation carried out by Suomi et al., the tripartite division turned out to be perceptually and phonetically valid: in listening experiments only three degrees of accentuation could be reliably distinguished. Phonetically, the differences were quite clear. Mere word stress (in unaccented words) was not signalled tonally (i.e. in terms of F0), whereas moderate accent and strong accent were signalled mainly tonally. Durationally only strong accent differed from the other degrees of prominence. It seems very likely that our speech data can be annotated in terms of accentuation by means of this descriptive framework. 
Finally, the breaks or pauses are annotated. The pauses are transcribed in terms of both phonetic and syntactic information: thus both manual phonetic measurement and syntactic analysis must be carried out. In the annotation, a silent pause is defined as a segment characterised by the absence of energy in the speech signal, with a minimum duration of 100 ms. Briefer pauses are not transcribed. Very brief pauses (50-100 ms in duration) typically occur between words, and may be caused by segmental features, such as plosives (Fant et al. 1990). A filled pause, with a minimum duration of 100 ms in our annotation, typically contains vocalisation of some kind. The pauses are classified along the following lines. At the first level of description, pauses can be sentence-internal (filled or unfilled) or sentence-final (filled or unfilled). At the second level of description, the (filled or unfilled) pauses may be syntactically motivated or unmotivated: syntactically motivated pauses occur at syntactic junctures (typically, between clauses and after long noun phrases). Finally, filled pauses may be “vocalisations” (e.g. uh, um) or “vocal noises” (inhalations, exhalations, laughter, sobbing, etc.). A basically linguistic approach is needed to annotate the speech corpus in terms of these features. The preliminary results indicate that the distribution of pauses signals some affective states more reliably than do most F0-related parameters: syntactically unmotivated long pauses (>500 ms) seem to be characteristic of speech reflecting sadness and, and syntactically unmotivated brief pauses (<200 ms) are connected with fearful speech. 
The aim of the linguistic annotation is to complement the acoustic/prosodic analysis. We aim to find perceptual contrasts in terms of tone and accent, and see how these “quantal” properties of speech convey emotional meaning (cf. Stevens 1972), over and above the continuously variable acoustic properties. In speech, cues related to emotional content interact in layers, none of which is, of course, entirely specific to emotional expression: for example, phonological prosodic contrasts signal syntactic and informational functions, and speech rate and average intensity may be connected with the stylistic features or genre of the speech situation. The listener will hear emotional cues smeared together at different levels, which also the analytic framework should reflect. 
A particularly interesting question is the time unit within which different emotional cues can be perceived in speech. It seems likely that emotional valence (i.e. whether the emotion is generally “positive” or “negative”) can be accurately perceived within seconds but a more nuanced analysis would seem to require a longer time. It can also be hypothesised that phonetic features of speech convey emotional content in very brief expressions while phonological prosodic contrasts can be perceived only in longer units of speech. These are all questions which the future research will address. 
6. Concluding remarks 
It is already clear that the speech corpus is useful in both basic and applied research on the vocal expression of emotions in Finnish. The results are readily applicable when developing content-based information retrieval methods for (Finnish) audio databases. Furthermore, especially the linguistic analysis of emotional speech outlined above significantly increases our understanding of an important (but underexplored) aspect of spoken Finnish. This knowledge is useful, for example, when creating a new Descriptive Grammar of Finnish – a topical issue in Finnish studies these days. 
796
The current speech material may represent unusually pure and intense emotional speech as the data was produced by professional actors. Thus, in the listening test, the listener-judges, though they, in the main, easily identified the basic emotions, were probably aware that the emotions represented simulated affect, differing (also prosodically) from real emotional speech. However, at the present stage of the research, this is not necessary a problem for information retrieval applications because much of the speech material stored in digital databases is, in fact, acted (radio plays, audio clips from movies, etc.). Yet it is clear that, eventually, more authentic data will be necessary. There are at least two ways to make the speech material more natural. Firstly, it is possible to use so-called emotive texts: the test subjects read out a text (a lengthy monologue, for example) which is emotionally biased. That is, the lexical content of the text is in tune with the intended emotion. In this way, instead of simulating the emotion to produce speech material, the emotion is stimulated or prompted, and the speech is then recorded. Secondly, genuinely authentic emotions can be induced – in the field of psychology of emotional behaviour, there are some widely used methods for evoking different emotions in the subjects as well as measuring different types of behavioural and physiological indices known to correlate with these emotions. In this type of research, the focus will eventually be geared towards the psychology of emotion, rather than the phonetic or linguistic basis of the outward manifestation of emotion. 
Appendix: Prosodic parameters measured from the speech signal 
MEAN Mean F0 
MEDIAN Median F0 
MAX Maximum F0 
MIN Minimum F0 
RANGE F0 range 
FRACMAX 95 % fractile of F0 
FRACMIN 5 % fractile of F0 
FRACRANGE 5 % - 95 % F0 range 
GDPOSAV Average F0 rise during voiced segment 
GDNEGAV Average F0 fall during voiced segment 
GDRISEAV Average steepness of F0 rise 
GDFALLAV Average steepness of F0 fall 
GDRISEMAX Maximum F0 rise during voiced segment 
GDFALLMIN Maximum F0 fall during voiced segment 
GDMAX Maximum steepness of F0 rise 
GDMIN Maximum steepness of F0 fall 
NORM FREQVAR Normalised segment frequency distribution width weighed sum 
F0VAR F0 variation 
JITTER Trend-corrected mean proportional random F0 
perturbation 
MEANINT Mean RMS intensity 
MEDIANINT Median RMS intensity 
MAXINT Maximum RMS intensity 
MININT Minimum RMS intensity 
INTRANGE Intensity range 
FRACMAXINT 95 % fractile of intensity 
FRACINTRANGE 5 % - 95 % intensity range 
NORM INTVAR Normalised segment intensity distribution width 
weighed sum 
INTVAR Intensity variation 
SHIMMER Trend-corrected mean proportional random 
intensity perturbation 
MAVLNGTH Average duration of voiced segments 
MANLNGTH Average duration of voiceless/silent segments 
shorter than 500 ms 
MASLNGTH Average duration of silences shorter than 400 ms 
MLNLNGTH Average duration of voiceless/silent segments 
longer than 500 ms 
797
MLSLNGTH Average duration of silences longer than 400 ms 
MAX VLNGTH Maximum duration of voiced segments 
MAX NLNGTH Maximum duration of voiceless/silent segments 
MAX SLNGTH Maximum duration of silent segments 
VRATIO Voicing-pauses ratio 
SRATIO Silence-speech ratio 
LFE500 Proportion of energy below 500 Hz 
LFE1000 Proportion of energy below 1000 Hz 
References 
Ahmadi S, Spanias A 1999 Cepstrum-based pitch detection using a new statistical V/UV classification algorithm. IEEE Transaction on Speech and Audio Processing 7: 333-338. 
Bosch L 2000 Emotions: what is possible in the ASR framework. In Proceedings of the ISCA Workshop on Speech and Emotion, Belfast, pp 183-188. 
Bregman A 1990 Auditory Scene Analysis. Cambridge, Massachusetts, MIT Press. 
Damasio A 1994 Descartes’ Error. New York, The Grosset Putnam. 
Fant G, Kruckenberg A, Nord L 1990 Acoustic correlates of rhythmical structures in text reading. In Wiik K, Raimo I (eds), Nordic Prosody V: Papers from a Symposium. Turku, University of Turku Press, pp 70-86. 
Iida A, Campbell N, Yasumura M 1998 Design and Evaluation of Synthesised Speech with Emotion. Journal of Information Processing Society of Japan 40: 479-486. 
Iivonen A 1998 Intonation in Finnish. In Hirst D, Di Cristo A (eds), Intonation Systems: A Survey of Twenty Languages. Cambridge, Cambridge University Press, pp 311-327. 
Izard C 1977 Human Emotions. New York, Plenum Press. 
Laukkanen A-M, Vilkman E, Alku P, Oksanen H 1997 On the perception of emotions in speech: the role of voice quality. Logopedics Phoniatrics Vocology 22: 157-168. 
Laver J 1994 Principles of Phonetics. Cambridge, Cambridge University Press. 
McGilloway S, Cowie R, Douglas-Cowie E, Gielen S, Westerdijk M, Stroeve S 2000. Approaching automatic recognition of emotion from voice: a rough benchmark. In Proceedings of the ISCA Workshop on Speech and Emotion, Belfast, pp 200-205. 
Polzin T S 1999 Detecting Verbal and Non-verbal Cues in the Communication of Emotions. Unpublished PhD thesis, Carnegie Mellon University. 
Scherer K R 1995 Expression of emotion in voice and music. Journal of Voice 9: 235-248. 
Scherer K R, Banse R, Wallbott H G 2001. Emotion inferences from vocal expression correlate across languages and cultures. Journal of Cross-Cultural Psychology 32: 76-92. 
Smeaton A 2000 TREC-6 Personal Highlights. Information Processing and Management 36: 87-94. 
Stevens K N (1972) The quantal nature of speech: evidence from articulatory-acoustic data. In David E, Denes P (eds), Human Communication: A Unified View. New York, McGraw Hill, pp 51-56. 
Suomi K, Toivanen J, Ylitalo R (in press) Durational and tonal correlates of accent in Finnish. Journal of Phonetics. 
Swerts M, Ostendorf M 1997 Prosodic and lexical indications of discourse structure in human-machine interactions. Speech Communication 22: 25-41. 
798
Titze I R, Haixiang L 1993 Comparison of F0 extraction methods for high-precision voice perturbation measurements. Journal of Speech and Hearing Research 36: 1120-1133. 
Toivanen J 2001 Perspectives on Intonation: English, Finnish and English Spoken by Finns. Frankfurt am Main, Peter Lang. 
Välimaa-Blum R 1993 A pitch accent analysis of intonation in Finnish. Ural-Altaische Jahrbucher 12: 82-89. 
799

Learner corpora: design, development and applications 
Yukio Tono 
Graduate School of Applied Linguistics 
Meikai University, JAPAN 
1. Introduction 
Following the 1996 AILA symposium, it was in 1999 when the first international symposium on learner corpora was held in Hong Kong, organized by Sylviane Granger and Joseph Hung. I remember that, with the exception of a few members, including myself, most of the participants were members of the ICLE project, which indicated that the majority of research activities were centred around the ICLE project and that not much else was going on except for some commercial projects such as Longman's. Now, 3 years after the Hong Kong conference, it is a pleasant surprise for the organizers of this pre-conference workshop to see that more than half of the speakers are working for projects other than ICLE, which shows that there has been a growing interest and diversity in this new field among SLA researchers and foreign language teachers. 
In this paper, I will give an overview of learner corpus research in terms of the theme of the workshop: its design, development and applications. I will clarify major theoretical and methodological issues involved in learner corpus research in order to facilitate discussions within this relatively new research community. I would also like to review the developments in this research area by introducing some of the major projects and their findings. 
2. Design issues of learner corpora 
Since people are interested in different aspects of learner language, it is quite natural that the design of learner corpora will vary from project to project, although the tremendous influence of the ICLE project cannot be underestimated here. Following the principles of Contrastive Interlanguage Analysis (CIA), we can use the ICLE corpora to compare native and non-native varieties of the same language, bringing out quantitative differences in frequency in the use of words, word categories, syntactic structures and discourse features (Granger 1998b: 47). Soon we will be able to enjoy the first harvest in the form of an ICLE CD-ROM, which will enable us to empirically verify how many of the ICLE research objectives have been achieved. To my knowledge, this is the first large collection of computerised learner data to be made available for research1 and we all welcome this valuable addition, considering the paucity of learner corpus data in the past. 
Table 1 shows some design considerations for building learner corpora. There are three major categories: (a) language-related criteria (e.g. mode, medium, genre, topic), (b) task-related criteria (e.g. longitudinal vs. cross-sectional; spontaneous vs. prepared), and (c) learner-related criteria (e.g. EFL or ESL, age, sex, mother tongue, overseas experience). 
Types of feature 
language-related task-related learner-related 
mode data collection internal-cognitive 
[written/spoken] [cross-sectional/longitudinal] [age/cognitive style] 
genre elicitation internal-affective 
[letter/diary/fiction/essay] [spontaneous/prepared] [motivation/attitude] 
style use of references L1 background 
[narration/argumentation] [dictionary/source text] L2 environment 
topic time limitation [ESL/EFL]/ [level of school] 
[general/leisure/ etc] [fixed/free/homework] L2 proficiency 
[standard test score] 
Table 1: Design considerations for building learner corpora 
Whilst the features in Table 1 can be seen in other types of learner data such as generic language 
1 One exception is the Longman Learner's Corpus, which has been commercially available for research. 
800
proficiency tests, learner corpora data will undergo further processing using the techniques of natural language processing. The types of processing available are illustrated in Table 2: 
Extra-textual information Header information (learner/ language/ task variables) 
Level of transcription Orthographic (+ phonemic/ phonetic for spoken corpora)2 
Level of annotation Sentence-boundary disambiguation 
Tokenisation 
POS tagging 
Lemmatisation 
Parsing (Treebanking) 
Semantic tagging (word senses/ semantic relationships and categories) 
Discourse tagging (apologies/greetings/politeness/?? moves/acts??/etc.) 
Error tagging 
Prosody annotation 
Anaphoric annotation 
Table 2: Processing of learner data 
One of the strengths of corpus-based research on learner language is that we can share corpora with other researchers so that findings can be subjected to the careful scrutiny of other researchers. It is in this very respect that most researchers who compile learner corpora for the first time have to be guided more carefully. There are quite a few projects in which not enough attention appears to have been paid to design considerations. If data is gathered in an opportunistic way without proper control and documentation of learner and task variables, the resulting corpus will be unlikely to be of much use. 
Granger proposed an approach called Contrastive Interlanguage Analysis (CIA), by which she means that a comparison can be made between native and non-native speakers as well as between learners with different L1 backgrounds in order to identify the features common to all learners and the ones unique to learners with a particular L1 background. This will enable us to distinguish “universal” errors from “L1-specific” errors. This is certainly a very interesting research avenue and one worth investigating, but it raises some methodological questions as well. Selection based upon external criteria such as school year or age does not necessarily ensure that the subjects selected are comparable in terms of language proficiency. This happens to be the case for the Japanese-speaking EFL learner group. Although their learner profile fulfilled all the criteria, their proficiency levels are so markedly lower than those from other European countries that the inclusion of the Japanese data seems to skew the overall results. This is inevitable, considering the learner-external criteria (i.e. second-year university English-majors) set for the project. It would be more appropriate to set an objective internal criterion such as a standard English proficiency test score. However, imposing such a strict criterion will reduce the number of subjects that can be included. This is a dilemma that we need to sort out in order to make our data truly comparable in the future. I would like to propose one solution to this problem. It would be good if we could administer a very simple, short language test for assessing the subjects’ proficiency levels. For instance, WHO?? are trying to develop a 5-minute vocabulary levels text, similar to Paul Nation's, but much shorter, in order to see whether such a short quiz can distinguish subjects’ proficiency levels in a valid and reliable way. If such a test were made available, it would greatly facilitate a better sampling of the subjects. 
Another methodological issue that I would like to mention here is concerned with the standardisation of corpus annotation. Within the corpus linguistics community generally, there is a growing awareness that corpus formatting and annotation should be standardised as much as possible (e.g. TEI, CES/XCES, EAGLES, ATLAS, TUSNELDA, MATE among others). It is of course useful to adopt generic annotation schemes such as above for learner data in general, but there is one area in which learner corpus researchers have yet to agree on a general scheme: “error annotation”. As shown in the history of error analysis, categorizing learner errors is a laborious and oftentimes fruitless job, for there are various ways of classifying errors, depending on research interest and theories involved and it is often the case that the classification is only as valid as the theory it is based on. Also, most people have different perspectives on error types, thus leading to very low inter-rater (or classifier) reliability. A generic error tagset, however, stills seems to be very useful goal to work towards, especially if an 
2 In this paper, phonetic transcription of the corpus is largely ignored because of its extra complexity, even though I am aware of such projects such as ISLE (http://nats-www.informatik.uni-hamburg.de/~isle/). 
801
international project such as ICLE can lead the way by producing a large set of learner corpora with standardised error annotations. It will give us an opportunity to survey the error patterns in the corpora in relation to the error analysis scheme and judge which areas we should focus future developments on. Even if it is not possible to use the generic or standard tagset wholesale for one's research purposes, one can still start with such a generic tagset and adapt it or add to it for more problem-oriented research purposes. 
3. Development of learner corpora 
Table 3 summarises the major learner corpus projects. As this is a rapidly growing field of research, it is increasingly difficult to be kept up-to-date and fully informed of all the various projects around the world, so if there is any project missing from this list, I would encourage you to contact me. 
Project Subjects/ Tasks Annotation 
Size Availability Comparison References Europe: International Corpus of 
Learner English (ICLE) - University EFL 
3/4 year students 
- 15 nationalities 
- Written essays 
- 3 million - Error tagged 
- POS tagged 
- Available in 2002 - IL – IL 
(different L1s) 
- TL – IL Granger (1993; 1994; 1996; 1998b; 2002) LINDSEI (Louvain International Database of Spoken English Interlanguage) - 50 interviews 
- University EFL 3/4 year students 
- 100,000 words 
- CH/IT/FR/JP - orthographic - IL – IL 
(different L1s) Granger (2001) Longman Learners' 
Corpus (LLC) - All-levels 
- Written essays 
- 10 million - POS tagged 
- Available for commercial purposes - IL – IL Gillard and Gadsby (1998) Polish-English Language Corpus Research and 
Applications 
(PELCRA) - All-levels 
- Written/spoken essays 
- Polish learners - POS tagged 
- Not available - IL – IL (developmental) 
- L1 – IL 
- TL – IL Uzar (1997) 
Mason & Uzar (2000) The UAM Corpus Corpus of teacher and students ’ production data. IL-IL Garcia 
longitudinal (in this workshop) The ISLE Corpus of non-native spoken English - 20 minute speech 
- German & Italian intermediate learners of English - Orthographic 
- Phone-stress 
- Available from ELRA - TL – IL http://nats-www.informatik.uni-hamburg.de/~isle/speech.html JPU (Janus Pannonius University) Corpus - University EFL 
- Written 
- c.400,000 - Plain text 
- Will be available - IL – IL (developmental) József (1998) Cambridge Learners 
Corpus (CLC) - All-levels 
- 10 million - POS tagged 
- Error-tagged (2.5 million) 
- In-house use only - IL – IL http://uk.cambridge.org/elt/reference/clc.htm Indianapolis Business 
Learner Corpus (IBLC) - US univ. business 
students - business writing 
- plain text - Plain text 
- Not available - IL – IL (different L1s) Connor & Precht (1998) 
Table 3: Learner corpus projects around the world Keys : IL = interlanguage ; TL = target language ; L1 = first language 
802
Table 3 (continued) 
Project Subjects/ Tasks Annotation 
Size Availability Comparison References ASIA: JEFLL Corpus (Japan) - All levels; EFL 
- Written & spoken 
- 1 million (expected in 2004) - POS-tagged 
- Error-tagged 
(partial) 
- Will be available - IL – IL 
cross-sectional 
- L1 – IL 
- TL – IL Tono (2000a, b) 
Tono and Aoki (1998) 
Tono (2002) Corpus of English 
by Japanese Learners - All levels; EFL 
- Written 
- 1 million - Plain text 
- Error tagged 
(partial) 
- Will be available - IL – IL 
cross-sectional Asao (1998) Japanese/ English 
Translation corpus - junior & senior high EFL students 
- L1/L2 translation - Plain text 
- Available via the web - TL – IL http://home.hiroshima-u.ac.jp/d052121/eigo1.html Standard Speaking Test (SST) Corpus (also called the TAO Corpus) - All levels; EFL 
- Spoken 
- 1,000,000 
- 15 min interview - Error tagged 
(partial) 
- Will be available - IL-IL 
(developmental) Tono et al. (2001) TELEC Student Corpus - Hong Kong 
learners 
- Univ. exam scripts 
- 3 million - Plain text 
- Restricted 
availability - TL – IL Allan (1998) Poly U Corpus - Postgraduates 
- thesis drafts, etc. 
- 282,000 - Plain text - TL – IL Farmer and Mead (1998) NTOU Corpus - EFL 
- 53,000 - Plain text - TL – IL 
- IL – IL Chen (1998) A parallel corpus of Japanese learners of English - Short English compositions 
- Paired with Japanese translations & NS's rewritings - Database format - TL – IL 
- IL – L1 Mark (1998a, b) MET Corpus - Chinese middle school students 
- Written 
- c. 150000 - Plain text - TL – IL He (1998) HKUST Corpus of 
Learner English (HKUST) - University EFL 
Chinese students 
- 10 million 
- Written essays & 
exam scripts -POS tagged (1M) 
- Error tagged (100,000 words) - IL – IL Flowerdew (1996) 
Flowerdew (1997) 
Milton (1998) 
Milton and Tsang (1993) 
The column “comparison” shows what types of comparison can be made by using the given corpus. Most corpora aim to be comparable with native corpora in order to reveal differences between NS and NNS performance. Some projects compare different stages of ILs (i.e. IL-IL) in order to identify the characteristics of different interlanguage stages. Very few learner corpora incorporate L1 data as an integral part of the design. This will become more important in future learner corpora projects as we are beginning to realise the need to identify specific features of L1-related errors or over/underuse patterns. The quality of TL and L1 corpora is also a critical issue, as over reliance on only one type of data will sometimes skew the picture. We also have to take into account what we should set as the target norm for the L2 learners that we are interested in. Should our students aim for the language proficiencies as represented in the British National Corpus, or should they aim towards something else? Since 
803
comparing frequencies and distributions is an essential part of the corpus-based study of learner language, we should have a clear understanding about the nature of the corpus data we use and how to make valid and meaningful comparisons. 
4. Learner corpus analysis 
In this section, I will briefly summarise the research areas in which modern-day learner corpora have been exploited. As shown in Table 3, most learner corpus projects were launched in the last decade and the research output based on them is consequently relatively limited. There are, however, a growing number of studies based on learner corpora, and these can show us how researchers are exploiting the new resources. I will summarise four main categories of study: studies related to error analysis (4.1), those investigating quantitative differences between native and non-native language (4.2), those describing the features of the interlanguage in its entirety (4.3), and those applying learner corpora-based research to language teaching methodology and materials design (4.4). 
4.1. Studies related to error analysis 
In this area, two further subcategories can be made: (1) studies on the development and evaluation of automatic error detection/tagging, and (2) computer learner-corpus-based error analysis. 
4.1.1. Studies on error analysis tools and error tagging 
Several studies (cf. Bolt 1992; Granger and Meunier 1994) tested the effectiveness of grammar and spelling checkers, demonstrating that while spelling checkers could be used for analysing interlanguage, these automatic tools correct only a very small number of learner errors. Whilst lists of the common errors of EFL learners are available (cf. Turton and Heaton 1997), we have no information on the frequency of these errors. Nor is there any information showing that certain error patterns occur more frequently in one particular learner group compared to others. We also do not know the contexts in which these errors are likely to be made. Without such information, it is impossible to develop either rule-based or probabilistic programs for identifying errors (Milton and Chowdhury 1994). 
Due to the lack of precision of currently available grammar checkers, some researchers attempt large-scale manual tagging of all lexical expressions in learner corpora. Since such error tagging is done manually, however, there is always the issue of validity and reliability. As regards validity, error taxonomy is a thorny issue. No matter how general the tagging scheme may be, it should at least include two aspects: (a) linguistic category classification (e.g. [grammar] - [verb] - [morpheme] - [tense]) and (b) target modification taxonomy (e.g. [omission/ addition/ misformation/etc.] (James 1998). As most error analysis studies in the 70s failed to provide a generic error taxonomy, we should learn a lesson from the past and make the tagging scheme purpose-oriented. Tono (2000b), for instance, replicated the morpheme studies of the 1970s and 80s. This involved manually error-tagging my learner data. I found that it was impossible not to have to develop my own tagging scheme for this particular study. Validity of error tagging should be assessed in the light of the research goals of any particular study. 
Reliability is a further issue. As Milton and Chowdhury (1994) commented, accounting for the uncertainty of error type is a serious problem. There are often cases where there is insufficient evidence to assign one unambiguous interpretation of an error. Thus we have to develop tagging schemes which allow for alternative possibilities in terms of target forms. However, while one may try to annotate reasonable alternative possibilities, it is doubtful that any analysis could guarantee total coverage of every possible option (ibid: 129). Granger et al. (1994) recommend that errors should not be normalised, as this involves a high degree of subjectivity, given that many errors can be corrected in many different ways (ibid: 105). This vague status of error correction makes the development of a tagging manual extremely important for the annotator. Granger and her team in ICLE have been developing a Windows-based error editor with an error-tagging manual (Dagneaux, Denness and Granger 1998; Dagneaux, Denness, Granger and Meunier 1996). Their attempt to make manual tagging work easy and consistent is worthwhile. Tono et al. (2001) and Izumi (in this workshop) have developed a generic error tagset and an associated editor. Sharing such tools will better facilitate the standardisation of corpus annotation in the future. 
There have been a few other attempts to automate parts of the error tagging process. Mason and Uzar (2000), for example, tested NLP (natural language processing) techniques for detecting zero articles in an interlanguage corpus and demonstrated the possibility of identifying missing articles in learner language. This could lead the way towards automatic error tagging based on POS information. In the same vein, Tono (2000a) also demonstrated the process of semi-automatically annotating learner 
804
data with morpheme tags, using automatically tagged POS information. 
4.1.2. Computer learner-corpus-based error analysis 
Another research area involving error analysis is the investigation of L2 learners' interlanguage errors using learner corpora. So far the results seem to be still fragmentary in nature, but there is a growing body of research into specific areas of interlanguage errors: for example, collocation (Chi et al. 1994; Lorenz 1997; Granger 1998c; Chen 1998), connectors (Milton and Tsang 1993; Granger and Tyson 1996; Satoh and Fang 1998), irregular past tense (Tono and Aoki 1998), and the English article system (Mason and Uzar 2000). 
4.2. Quantitative differences between native and non-native language 
There is genuine interest in quantitative differences in the use of certain syntactic, lexical and discoursal features between native and non-native speakers. This is especially true of learner corpora which consist of data from advanced learners, where there is general conformity to native speaker norms in terms of the basic rules of syntax and morphology. "Their deviations from the norm usually concern rather fine points of lexico-grammar and style" (Lorenz 1998:53). Thus the main interest is naturally shifted towards whether they use particular linguistic features more frequently or less frequently than native speakers. 
Comparisons are often made between NS and NNS as well as between different NNS groups. The ICLE project members have published articles extensively on this subject. The research topics include adverbial connectors (Altenberg and Tapper 1998; Lorenz 1998), multiword units (De Cock 1998, De Cock et al. 1998), direct questions (Virtanen 1998a), the progressive (Virtanen 1998b), tense morphology (Granger 1999) and phrasal verbs (Lam and Hung 1998). 
Biber and Reppen (1998) also conducted an analysis of complement clauses in a large native corpus (the Longman Grammar Corpus) and a small learner corpus (an early version of the Longman Learners' Corpus). While they found quite similar patterns of use for complement clauses, they are quick to caution that since there is also considerable variation among the different tasks required for student writing, it would be necessary to compile learner corpora designed to represent the full range of student writing (and speaking) tasks (ibid: 157) to draw firmer conclusions. 
4.3. Description of overall IL development 
Although the number is still small, some attempts have been made to exploit learner corpora to describe overall IL characteristics at a fixed stage or at different developmental stages. Granger and Rayson (1998), for example, demonstrated the potential of automatic profiling for revealing the stylistic characteristics of EFL texts vis-a-vis NS texts. They not only produced word frequency profiles for this purpose but also used various measures such as word category profiles (using POS information), which can reveal significant patterns of the over/underuse of major word categories. They concluded that their automatic profiling techniques highlighted the speech-like nature of learner writing (ibid: 129). Leñko-Szymañska (2000a, b) traced vocabulary growth in L2 learners' production by using learner corpora. 
Researchers involved in the ICLE project investigated the characteristics of learner language by examining sequences of POS tags (de Haan 1997, 1998; Aarts and Granger 1998). They investigated tag n-grams generated from POS-tagged corpora of three groups of learners (French, Dutch and Finnish). de Haan (1997) found that Finnish students tend to use the combinations involving articles least frequently, which is attributable to the fact that Finnish has no articles and that the use of articles in English is a notoriously difficult topic for Finnish learners (cf. Sajavaara 1981). This coincides with the observations made by Tono (2000a) and Mason and Uzar (2000) that the lack of articles in Japanese and Polish respectively is indeed shown to affect the use of the article system by Japanese- and Polish-speaking learners of English. Aarts and Granger (1998) found a marked similarity in tag sequence frequencies between the three categories of learners (Dutch, Finnish and French), and a striking difference in sentence beginnings: nouns are underused and pronouns overused in sentence-initial sequences in all three learner corpora (ibid: 134). Aarts and Granger also found consistent underuse of the four most common trigrams (sequences of three tags) in NS writing, all of which contain prepositions. Tono (2000b) investigated the tag sequences of developmental spoken interlanguage corpora and found that while L1 child data (taken from the CHILDES database) contain very high proportions of nouns in their top trigrams at the beginning stage, L2 learners start with verb-centred trigrams first instead of nouns. This shows that at least in the case of Japanese EFL learners, beginners produce utterances which contain the basic elements of syntax: subject, verb, and predicate, although the sentences are very short. What is striking in the subsequent developmental 
805
stages is the lack of tense/aspect morphology and the constant underuse of modal verbs as well as prepositional phrases. The Japanese EFL learner data also showed very different tag sequence patterns from the three groups examined in Aarts and Granger (1998), which indicates that the proficiency level of Japanese learners is much lower than the subjects represented in the ICLE Corpus. 
Finally, there are a few studies investigating the L2 acquisition of particular linguistic or lexical items. These studies are characterized by the use of learner corpora as a testbed for SLA theories. These studies cover such areas as the acquisition of tense/aspect morphology (Housen 1998; Tono and Aoki 1998), grammatical morphemes (Tono 1998, 2000a), and verb semantics (Oshita 1997). Focusing on L2 acquisition of verb Subcategorisation Frame (SF) patterns, Tono (2002) proposed a rigorous multiple-comparison between interlanguage, L1 and TL corpora in order to identify the relative effects of the four major variables: inherent verb semantics, similarities in SF patterns between L1 and TL, the influence of L2 input (as shown in the frequencies of SF patterns in the textbooks) and developmental effects. 
5. Pedagogical applications of learner corpora 
Those who are interested in pedagogical applications exploit the results of analyses of learner data to improve various aspects of foreign language teaching. At present, there are still very few studies which relate the findings from learner corpora to actual classroom practice. This is understandable if one takes into account the fact that investigation of interlanguage does not directly lead to better pedagogical practice. It may generate more research questions regarding the way formal instructionis given in a particular EFL context, but it is necessary to conduct follow-up studies to confirm the effect of such methodological changes as suggested by the corpus findings. I should say that we are still not at an advanced stage of development and further research will be needed first to accumulate solid findings from learner corpora. 
There are, however, some good examples of pedagogical innovations using learner corpora. For example, Milton (1998) investigated Chinese EFL learners' problematical areas in their writing and designed a CALL program (AutoWord) to assist students in essay writing. His research was based upon the 10-million-word HKUST Learner Corpus. The AutoWord program contains components such as error recognition (i.e. editing) exercises, a hypertext online grammar, and databases of the 'underused' lexical and grammatical phrases. Milton also developed the wordlist-driven concordancer, WordPilot , in order to integrate concordancing into the L2 essay writing system for novice learners (Milton 2001). Kevin Mark (1998a, b) has developed a very unique parallel corpus database based on his students’ writing. His data is firmly rooted in his classroom activities, thus his error classification is practical, which makes the results of his database extremely useful for improving the quality of the activities. 
Other areas of applications involve L2 lexicography (Tono 1996, 2001; Gillard and Gadsby 1998), ELT textbook design (Kaszubski 1998), teaching methodology (Granger and Tribble 1998), and developing a learning list of grammar items (Tono and Aoki 1998). 
6. Conclusion 
I hope that I have been able to show the dynamic nature of this new research area. I will close by making a few statements about desiderata for future research. Firstly, learner corpus researchers should exchange ideas with SLA researchers in a more structured and systematic way. Many corpus-based researchers do not know enough about the theoretical background of SLA research to communicate with them effectively, while SLA researchers typically know little about what corpora can do for them. By improving the communication lines, we will be able to learn from each other. Secondly, the compilation of a corpus takes time and effort. This means that it will take time to produce useful and useable results. Some people want quick solutions and tend to use cut corners when designing and building corpora, but this will confuse others by producing the results which are not always valid or reproducible. We should enrich the research community with the expertise we have gained from previous projects and should encourage one another not to jump on the bandwagon of corpus-based research without sufficient knowledge of corpus building. 
References 
Aart, J. and S. Granger, 1998 Tag sequences in learner corpora: a key to interlanguage grammar and discourse. In Granger (ed.) 1998a, pp. 132-141. 
Alderson, C. 1996 Do corpora have a role in language assessment? In Thomas, J. and M. Short (eds.) Using Corpora for Language Research. London: Longman, pp. 248-259. 
806
Allan, Q. G. 1998 The TELEC Student Corpus: a resource for teacher development. In S. Granger and J. Hung (eds) 1998, pp. 4-6. 
Altenberg, B. and M. Tapper 1998 The use of adverbial connectors in advanced Swedish learners' written English. In Granger (ed.) 1998a, pp. 80-93. 
Asao, K. 1998 Corpus of English by Japanese learners. In S. Granger and J. Hung (eds) 1998, pp. 10-13. 
Biber, D. and R. Reppen 1998 Comparing native and learner perspectives on English grammar: a study of complement clauses. In Granger (ed.) 1998a, pp. 145-158. 
Bolt, P. 1992 An evaluation of grammar-checking programs as self-helping learning aids for learners of English as a foreign language. CALL 5: 49-91. 
Chen, H-J.H. 1998 Underuse, overuse and misuse in Taiwanese EFL learner corpus. In S. Granger and J. Hung (eds) 1998, pp. 25-28. 
Chi, Amy, K. Wong Pui-yui and M.W. Chau-ping (1994) Collocational problems amongst ESL learners: a corpus-based study. In Flowerdew, L. and A. K.K. Tong (eds.) Entering Text. Language Centre. The Hong Kong University of Science and Technology. 
Dagneaux, E., S. Denness, S. Granger, and F. Meunier 1996 Error Tagging Manual Version 1.1. Centre for English Corpus Linguistics, Université catholique de Louvain, Louvain-la-Neuve. 
Dagneaux, E., S. Denness, and S. Granger, 1998 Computer-aided Error Analysis. System: An International Journal of Educational Technology and Applied Linguistics 26(2): 163-174. 
David, M. 1998 The pedagogical implication of a learner corpus. In S. Granger and J. Hung (eds) 1998, pp. 87-88. 
De Cock, S. 1998 A Recurrent Word Combination Approach to the Study of Formulae in the Speech of Native and Non-Native Speakers of English, International Journal of Corpus Linguistics 3(1): 59-80. 
De Cock, S., S. Granger, G. Leech and T. McEnery 1998 An automated approach to the phrasicon of EFL learners. In Granger (ed.) 1998a, pp. 67-79. 
Dodd, B. 1997 Exploiting a corpus of written German for advanced language learning. In Wichmann et al. 1997, pp. 131-145. 
Farmer, R. and K. Mead 1998 The language of citations: an analysis via computer learner corpus. In S. Granger and J. Hung (eds) 1998, pp. 34-37. 
Flowerdew, J. 1996 Concordancing in language learning. In M. Pennington (eds.) The Power of CALL, pp. 97-113. Houston, TX: Athelstan. 
Flowerdew, L. 1997 Interpersonal strategies: investigating interlanguage corpora. RELC Journal 28 (1): 72-88. 
Flowerdew, L. 1998 Concordancing on an expert and learner corpus in ESP. CALL Journal 8 (3): 3-7. 
Foster-Cohen, S.H. 1999 SLA and First Language Acquisition. Annual Review of Applied Linguistics 19: 3-21. 
French, F. 1949 Common Errors in English. London: Oxford University Press. 
Gillard, P. and A. Gadsby 1998 Using a learners' corpus in compiling ELT dictionaries. In Granger (ed.) 1998a, pp. 159-171. 
Granger, S. 1993 The International Corpus of Learner English. In Aarts, J., P. de Haan and N. Oostdijk (eds.) English Language Corpora: Design, Analysis and Exploitation. Amsterdam: Rodopi, pp. 57-69. 
Granger, S. 1994 The learner corpus: a revolution in applied linguistics. English Today 39 (10/3): 25-9. 
Granger, S. 1996 From CA to CIA and back: an integrated approach to computerized bilingual and learner corpora. In Aijmer, K., B. Altenberg and M. Johansson (eds.) Languages in Contrast: Papers from a Symposium on Text-based Cross-linguistic Studies, Lund 4-5 March 1994, Lund: Lund University Press, pp. 37-51.. 
Granger, S. (ed.) 1998a Learner English on Computer. London: Addison Wesley Longman. 
Granger, S. 1998b A bird's eye view of computer learner corpus research. In S. Granger and J. Hung (eds) 1998, pp. 45-48. 
Granger, S. 1998c Prefabricated patterns in advanced EFL writing: collocations and formulae. In A. Cowie (ed.) Phraseology: theory, analysis and applications. Oxford: Oxford University Press, pp. 145-160. 
Granger, S. 1999 Use of Tenses by Advanced EFL Learners: Evidence from an Error-tagged Computer Corpus. In Hasselgard, H. and S. Oksefjell (eds) Out of Corpora - Studies in Honour of Stig Johansson. Amsterdam: Rodopi, pp. 191-202. 
Granger, S. 2002 A Bird's-eye View of Computer Learner Corpus Research. In Granger, S., Hung, J. and Petch-Tyson, S. (eds) Computer Learner Corpora, Second Language Acquisition and Foreign Language Teaching. Amsterdam and Philadelphia: Benjamins. 
807
Granger, S. and F. Meunier 1994 Towards a grammar checker for learners of English. In Fries, U. and G. Tottie (eds.) Creating and Using English Language Corpora. Amsterdam and Atlanta: Rodopi, pp.79-89. 
Granger, S. and P. Rayson 1998 Automatic Lexical Profiling of Learner Texts. In Granger (ed.) 1998a, pp. 119-131. 
Granger, S. and C. Tribble 1998 Learner corpus data in the foreign language classroom: form-focused instruction and data-driven learning. In Granger (ed.) 1998a, pp. 199-209. 
Granger, S. and S. Tyson 1996 Connector usage in the English essay writing of native and non-native EFL speakers of English. World Englishes 15: 19-29. 
Granger, S. and J. Hung (eds.) 1998 First International Symposium on Computer Learner Corpora, Second Language Acquisition and Foreign Language Teaching, 14-16 December, 1998, The Chinese University of Hong Kong: Symposium Proceedings. 
de Haan, P. 1997 An experiment in English learner data analysis. In Aarts, J., de Mönnink, I. and Wekker, H. (eds) Studies in English Language and Teaching. Amsterdam: Rodopi, pp. 215-229. 
de Haan, P. 1998 How native-like are advanced learners of English? In Renouf, A. (ed.) Explorations in Corpus Linguistics, Amsterdam: Rodopi, pp. 55-65. 
de Haan, P. 1999 English writing by Dutch-speaking students. In Hasselgard, H. and Oksefjell, S. (eds) Out of Corpora, Amsterdam: Rodopi, pp. 203-212. 
de Haan, P. (forthcoming) Tagging non-native English with the TOSCA-ICLE tagger. In Mair, C. (ed.) Proceedings of the 20th ICAME Conference, Freiburg 1999. Amsterdam: Rodopi. 
He, A. 1998 A corpus-based analysis of middle school students’ English spelling errors. In Garnger and Hung (eds.), pp. 54-58. 
Housen, A. 1998 An analysis of grammatical form-function mapping in L2 data using the CHILDES system. In S. Granger and J. Hung (eds.) 1998, pp. 59-62. 
James, C. 1980 Contrastive Analysis. London: Longman. 
James, C. 1998 Errors in Language Learning and Use. London: Addison Wesley Longman. 
Johns, T. 1993 Data-driven learning: an update. TELL & CALL 3. 
Johns, T. 1997 Contexts: the background, development and trialling of a concordance-based CALL program. In Wichmann et al. 1997, pp. 100-115. 
József, H. 1998 Advanced Writing in English as a Foreign Language: A Corpus-based Study of Processes and Products. Unpublished PhD dissertation. Janus Pannonius University, Pécs, Hungary. 
Kaszubski, P. 1998 Enhancing a writing textbook: a national perspective. In Granger, S. (ed.) 1998a pp. 172-185. 
Lam, P and J.Hung 1998 The use of multi-word verbs in advanced Chinese ESL learners. In S. Granger and J. Hung (eds) 1998, pp. 80-82. 
Leñko-Szymañska, A. 2000a Passive and active vocabulary knowledge in advanced learners of English. In Lewandowska-Tomaszczyk, B. and J.P. Melia (eds), pp. 287-302. 
Leñko-Szymañska, A. 2000b How to trace the growth in learner's active vocabulary. A corpus-based study. Paper presented at the 4th International Conference on Teaching and Language Corpora. Graz, 19-23 July 2000. 
Lewandowska-Tomaszczyk, B., T. Leñko-Szymañska, and A. McEnery 2000 Lexical problem areas in the PELCRA learner corpus of English. In Lewandowska- Tomaszczyk, B. and J.P.Melia (eds.) 2000, pp. 303-312. 
Lewandowska-Tomaszczyk, B. and J.P. Melia (eds.) 2000 PALC’ 99: Practical Applications in Language Corpora. Frankfurt: Peter Lang. 
Lorenz, G. 1997 Introducing a Learner Corpus of English Writing: The Hidden Potentials of Adjective Intensification. Amsterdam: Rodopi. 
Mark, K. 1998a A parallel learner corpus approach to English curriculum development at a Japanese university. In S. Granger and J. Hung (eds) 1998, pp. 89-90. 
Mark, K. 1998b The Significance of Learner Corpus Data in Relation to the Problems of Language Teaching. Bulletin of General Education 312: 77-90. Meiji University. 
Mason, O. and R. Uzar 2000 NLP meets TEFL: Tracing the zero article. In Lewandowska-Tomaszczyk, B. and J.P. Melia (eds.) 2000, pp. 105-116. 
Meunier, F. 1995 Tagging and parsing interlanguage. In Beheydt, L. (ed.) La Linguistique Appliquée dans les années 90. ABLA Review 16, 21-29. 
Meunier, F. 1998 Computer tools for the analysis of learner corpora. In Granger (ed.) 1998a, pp. 19-37. 
Milton, J. 1998 WORDPILOT: enabling learners to navigate lexical universes. In S. Granger and J. Hung (eds) 1998: 97-98. 
Milton, J. 2001 Describing and overcoming environmental limitations on the interlanguage of Hong 
808
Kong Chinese learners of English: a computational and corpus-based methodology. Unpublished PhD thesis. Lancaster University. 
Milton, J. and E. Tsang 1993 A corpus-based study of logical connectors in EFL students’ writing. In R. Pemberton & E. Tsang (eds.) Studies in Lexis. Language Centre, The Hong Kong University of Science and Technology, pp. 215-246. 
Milton, J. and N. Chowdhury 1994 Tagging the interlanguage of Chinese learners of English. In Flowerdew, L. and A. K. K. Tong (eds.) Entering Text. Language Centre, The Hong Kong University of Science and Technology, pp. 127-143. 
Milton, J. and R. Freeman 1996 Lexical variation in the writing of Chinese learners of English. In C.E. Percy, C.F. Meyer and I. Lancashire (eds.) Synchronic Corpus Linguistics. Papers from the sixteenth International Conference on English Language Research on Computerized Corpora, Amsterdam: Rodopi, pp. 121-131. 
Milton, J. and K. Hyland 1999 Assertions in students’ academic essays: a comparison of L1 and L2 writers. In R. Berry, B. Asker, K. Hyland and M. Lam (eds.) Language Analysis, Description and Pedagogy. Hong Kong: HKUST, pp. 147-161. 
Oshita, H. 1997 “The unaccusative trap”: L2 acquisition of English intransitive verbs. Unpublished PhD thesis. University of Southern California. 
Sajavaara, K 1981 The nature of first language transfer: English as L2 in a foreign language setting. Paper presented at the first European-North American Workshop on Cross-Linguistics Second Language Acquisition Research, Lake Arrowhead, California. 
Satoh, K. and C.A. Fang 1998 A corpus-based study of the grammar and lexis of Japanese learners’ English. In S. Granger and J. Hung (eds) 1998, pp. 103-104. 
Tono, Y. 1996 Using learner corpora for L2 lexicography. LEXIKOS 6: 116-132. Stellenbosch: Universiteit van Stellenbosch. 
Tono, Y. 1998 A computer learner corpus-based analysis of the acquisition order of English grammatical morphemes. In TALC (Teaching and Language Corpora) 98 – Conference Proceedings, Keble College Oxford, 24-27 July 1998, pp. 183-187. 
Tono, Y. 2000a A computer learner corpus-based analysis of the acquisition order of English grammatical morphemes. In Burnard, L. and T. McEnery (eds.) Rethinking Language Pedagogy from a Corpus Perspective. Frankfurt: Peter Lang, pp. 123-132. 
Tono, Y. 2000b A corpus-based analysis of interlanguage development: Analysing POS tag sequences of EFL learner corpora. In Lewandowska-Tomaszczyk, B. and J.P.Melia 2000, pp. 323-343. 
Tono, Y. 2001 Research on Dictionary Use in the Context of Foreign Language Learning. Tübingen: Max Niemeyer Verlag. 
Tono, Y. 2002 The Role of Learner Corpora in SLA Research and Foreign Language Learning: The Multiple Comparison Approach. Unpublished PhD thesis. Lancaster University. 
Tono, Y. and K. Kanatani 1995 EFL learners’ proficiency and roles of feedback: towards the most appropriate feedback for EFL writing. Annual Review of English Language Education in Japan 6: 1-11. 
Tono, Y. and M. Aoki 1998 Developing the optimal learning list of irregular verbs based on the native and learner corpora. In S. Granger and J. Hung (eds) 1998, pp. 113-118. 
Tono, Y., Kaneko, T., Isahara, H., Saiga, T. and Izumi, E. 2001 The Standard Speaking Test (SST) Corpus: A 1 million-word spoken corpus of Japanese learners of English and its implications for L2 lexicography. Lee, S. (ed.) ASIALEX 2001 Proceedings: Asian Bilingualism and the Dictionary. The Second Asialex International Congress, August 8-10, 2001, Yonsei University, Korea, pp. 257-262. 
Tribble, C. 1997 Improvising corpora for ELT: quick-and-dirty ways of developing corpora for language teaching. In Lewandowska-Tomaszczyk, B. & P. J. Melia, (eds.) PALC '97 (Practical Applications in Language Corpora), pp. 106-117. Frankfurt: Peter Lang. 
Turton, N.D. and J.B. Heaton 1997 Longman Dictionary of Common Errors. Harlow: Longman. 
Uzar, R. 1997 Was PELE a linguist? In Lewandowska-Tomaszczyk, B. & P. J. Melia (eds.) PALC ’97 (Practical Applications in Language Corpora, Lódz, Poland 10-14 April 1997). 
Virtanen, T. 1998a Direct questions in argumentative student writing. In Granger (ed.) 1998a, pp.94-118. 
Virtanen, T. 1998b Argumentative uses of the progressive in NS and NNS student compositions: notes on clause status and grounding. In S. Granger and J. Hung (eds) 1998, pp. 119-120. 
809

Morphological knowledge and alignment of English-German parallel corpora
Patrick Tschorn,
Institute of Cognitive Science
University of Osnabrück
ptschorn@uos.de
Anke Lüdeling,
Institute of German Language and Linguistics
Humboldt-University, Berlin
anke.luedeling@rz-hu-berlin.de
1 Introduction
Alignment is an important step to linguistically exploit parallel corpora. In this paper we introduce a
morphological component that improves the alignment of German-English parallel texts and helps find
correspondences between morphological elements on the sub-word level. This paper deals with a small
aspect of an alignment system, namely the improvement of a dictionary-based distance measure
through a morphological analyser.
What is alignment? For the purposes of this paper we define a bilingual parallel text as a text (L1) and
its translation (L2). A sentence level alignment then maps groups of L1-sentences to corresponding
groups of L2-sentences. These groups are often called "beads". An alignment can be viewed as a
sequence of beads that covers the entire parallel text. While most beads usually express the
correspondence between a single L1-sentence and a single L2-sentence, other types of beads arise
when sentences are split, merged, deleted, added or changed in order by the translator. Each sentence
belongs to exactly one bead.
To illustrate some of the difficulties, consider the following excerpt from the very beginning of ‘The
War of the Worlds’ parallel text1:
0: The Eve of the War 0: 1. Am Vorabend des Krieges
1: But who shall dwell in these worlds if they be
inhabited ? ...
2: Are we or they Lords of the World ? ...
3: And how are all things made for man ?
4: - - Kepler ( quoted in The Anatomy of
Melancholy )
5: No one would have believed in the last years of
the nineteenth century that this world was being
watched keenly and closely by intelligences
greater than man 's and yet as mortal as his own ;
that as men busied themselves about their various
concerns they were scrutinised and studied ,
perhaps almost as narrowly as a man with a
microscope might scrutinise the transient creatures
that swarm and multiply in a drop of water .
1: Niemand hätte in den letzten Jahren des 19 .
Jahrhunderts daran gedacht , daß unsere Welt
beobachtet würde ; daß andere intelligente Wesen
, größer als die menschlichen und doch ebenso
sterblich , uns bei unserem täglichen Tun fast
ebenso intensiv belauschen und erforschen
könnten , wie jemand mit dem Mikroskop jene
kurzlebigen Lebewesen erforscht , die in einem
Wassertropfen ihr Wesen treiben und sich darin
vermehren .
6: With infinite complacency men went to and fro
over this globe about their little affairs , serene in
their assurance of their empire over matter .
2: Mit einem unendlichen Behagen schlenderte die
Menschheit mit ihren kleinen Sorgen kreuz und
quer auf dem Erdball umher , in gelassenem
Vertrauen auf ihre Herrschaft über die Materie .
7: It is possible that the infusoria under the
microscope do the same .
3: Es ist möglich , daß die mikroskopischen
Lebewesen unter dem Brennglas dasselbe tun .
8: No one gave a thought to the older worlds of
space as sources of human danger , or thought of
them only to dismiss the idea of life upon them as
impossible or improbable .
4: Niemand dachte daran , daß von anderen
Planeten Gefahren für die Menschheit herrühren
könnten .
5: Jede Vorstellung , daß sie bewohnt sein könnten
, wurde als unwahrscheinlich oder unmöglich
1 In this paper, we draw all our examples from ‘The War of the Worlds’ parallel text (Wells 1898). We also studied European
Union documents (RAPID 1995-2002) which contain many productively formed compounds and thus benefit from
morphological analysis.
818
abgetan . 9: It is curious to recall some of the mental habits of those departed days. 6: Es ist seltsam , sich heute der geistigen Verfassung jener vergangenen Tage zu entsinnen . 10: At most terrestrial men fancied there might be other men upon Mars , perhaps inferior to themselves and ready to welcome a missionary enterprise . 7: Es kam höchstens vor , daß Erdbewohner sich einbildeten , es könnten Wesen auf dem Mars leben , minderwertige allenfalls , zumindest aber solche , die eine irdische Forschungsreise freudig begrüßen würden . 11: Yet across the gulf of space , minds that are to our minds as ours are to those of the beasts that perish , intellects vast and cool and unsympathetic, regarded this earth with envious eyes , and slowly and surely drew their plans against us . 8: Aber jenseits des gähnend leeren Weltraums blickten Geister , uns so überlegen wie wir den Tieren , ungeheure , kalte und unheimliche Geister , mit neidischen Augen auf unsere Erde . 9: Bedächtig und sicherschmiedeten sie ihre Pläne gegen uns .
Table 1: Alignment of the first sentences of ‘The War-of-the-Worlds’ parallel text.
Computing the alignment is made difficult by the fact that the English version contains a quotation at the very beginning that is missing in the German version and by the two instances where more than one German sentence corresponds to one English sentence (so-called 1:2-beads; 8:4,5 and 11:8,9).
Different approaches to automatically compute sentence level alignments have been proposed since the mid nineteen eighties. A short history can be found in (Veronis, 2000).
Conceptually, an alignment program needs two main components:
• a distance function: a function to judge the degree of correspondence between two given segments of a parallel text
• an optimizer: a means of finding the sequence of beads that maximizes the overall degree of correspondence (composed of the assessments of the individual beads)
A popular choice is to solve the optimization task by dynamic programming, but a range of other possibilities exists. This aspect is of no importance to this paper.
Distance functions numerically express the relatedness of two segments of text. A variety of different distance functions have been proposed. The kind of knowledge employed to judge the relatedness of segments is of importance:
• Length-based distance functions are based on the observation that between many languages long sentences tend to be translated into long sentences, and short sentences into short sentences. The involved knowledge boils down to a probability distribution that predicts how likely it is that two segments, given their lengths, are translations of each other. Lengths can for example be measured in characters or words. This approach can easily be adapted to many language pairs. Despite its simplicity it works surprisingly well on clean parallel texts such as the Canadian Hansards. (Gale & Church 1991, Canadian Hansards) However, they do not work so well on texts with a slightly freer translation (e.g. literature). In many texts – as illustrated above - we find elisions or insertions. Because the length-based distance function does not assess the content of a sentence, missing sentences cannot reliably be detected. This causes problems in sequences of sentences with similar length.
• Lexical distance functions, on the other hand, try to find corresponding words in the two halves of the parallel text. The more correspondences between L1-words and L2-words found in a segment, the higher the degree of relatedness is deemed to be. The simplest and cheapest way of finding correspondences is by computing string similarity (cognates). Even though many sentence pairs contain cognates, they alone are not sufficient for a reliable distance function. Therefore it is desirable to employ a bilingual dictionary. There are in principle two possibilities to obtain a dictionary: either an ad hoc dictionary is automatically extracted from a parallel text or a high quality machine readable dictionary is taken.
o Cognates are strings that are orthographically (or otherwise) similar across languages, as for example observatory - Observatorium. Proper names, foreign words and numbers are often suitable cognates. String similarity can be judged by comparing the number of n-grams shared by two strings to the total number of n-grams. Other popular methods are related to the Levenshtein Edit Distance (see e.g. Steven 1994). Cognates are easy to detect and are often used to determine anchor-points for length-based alignment programs. Cognates can also be used to complement other lexical resources.
819
o Methods of automatically extracting an ad-hoc dictionary from aligned and unaligned parallel texts (see e.g. Fung 2000) or during the alignment (e.g. Kay & Röscheisen 1993) have been proposed. These approaches usually assume that translation relations hold between single words only and are based on some notion of frequency of co-occurence. It is, however, well-known that words in texts follow a Zipfian distribution, that is, texts consist of a small number of highly frequent words (such as articles and prepositions) and of a large number of very rare words (Baayen 2001). Highly frequent words occur in nearly every sentence and thus say virtually nothing about the relatedness of two segments of a parallel text. Rare words are a much better indicator for relatedness but are very hard to capture by co-occurence counting methods.
o The third possibility mentioned above are dictionary-based distance functions that rely on an existing machine-readable dictionary. Dictionary-based distance functions perform better than distance functions that employ less knowledge because one word can have many translations and multi-word units can be taken into account. There are, however, a number of disadvantages and problems: first, dictionary-based functions are language dependent and therefore have to be obtained for each new language pair. Dictionary resources can sometimes be difficult to obtain. Another fundamental problem is the necessary incompleteness of any dictionary resource (see the following section).
In this paper we take a dictionary-based distance function as a starting point and show how we can deal with the problem of incompleteness by using a morphological analyser. The distance function and the morphological analyser are part of an experimental alignment system (Tschorn 2002). The dictionary contains approx. 120,000 lemmatised words. The texts are also lemmatised (Schmid 1994). A list of stop words (frequent words that do not provide information on relatedness) is excluded.
The remainder of the paper is organized as follows: first we establish the need for a morphological analysis by showing how incomplete lexicons are and giving some examples which illustrate where morphological analysis would help find more correspondences (Section 2). Then we introduce the morphological analyser (Section 3) and show how it finds correspondences in the bitext (Section 5). In Section 6 we evaluate the morphological component.
2 Lexical knowledge and the incompleteness problem
Dictionary-based approaches are, of course, dependent on the quality and the scope of the employed dictionary. But no matter, how good and large the dictionary is, it will always be incomplete because of morphological productivity which can produce an “in principle unlimited number of new formations” (Schultink 1961, 113). This is especially obvious in languages like German which has excessive compounding in addition to derivation but it is just as true in languages like English or even Romance languages if the notion word does not refer to ‘graphemic word’ but rather to something like ‘semantic unit’ (as it should in translation). That means that there will often be unknown/unmatched words (or sequences) in the parallel text. Matching (some of) these words can significantly improve the distance function. The remainder of this paper shows how a morphological analysis of complex words can help in this task. The basic idea is the more matches the better – even if this results in partial matches and some morphological principles are violated along the way.
In this section we describe the phenomena that can be tackled by a morphological analyser – the analyser itself is described in Section 4.
For a first example, consider Table 2. Some unknown/unmatched words are marked in boldface. Consider first the case of the noun-noun compound Wassertropfen which consists of Wasser and Tropfen. Although this compound is not listed in the dictionary and can therefore not be matched, its parts can straightforwardly be matched to words (drop and water) in the English half of the parallel text. In this case it would be sufficient to analyse the German word. The case of the adjective menschlich 'human' which consist of the noun Mensch and the adjective suffix -lich is more complicated. Here we see that the corresponding word man belongs to a different part-of-speech – this happens very often in translation, and such cases can hardly ever be found in a dictionary. Here we see that complex derivations also have to be analysed. Only part of the derivation is used to improve the alignment (-lich remains unmatched). Sometimes complex English words have to be analysed as well, cf. Table 3.
820
5: No one would have believed in the last years of the nineteenth century that this world was being watched keenly and closely by intelligences greater than man 's and yet as mortal as his own ; that as men busied themselves about their various concerns they were scrutinised and studied , perhaps almost as narrowly as a man with a microscope might scrutinise the transient creatures that swarm and multiply in a drop of water . 1: Niemand hätte in den letzten Jahren des 19 . Jahrhunderts daran gedacht , daß unsere Welt beobachtet würde ; daß andere intelligente Wesen , größer als die menschlichen und doch ebenso sterblich , uns bei unserem täglichen Tun fast ebenso intensiv belauschen und erforschen könnten , wie jemand mit dem Mikroskop jene kurzlebigen Lebewesen erforscht , die in einem Wassertropfen ihr Wesen treiben und sich darin vermehren . 1. kurzlebigen :: transient
2. beobachtet :: watched
3. Welt :: world
4. größer :: greater
5. Mikroskop :: microscope
6. Jahren :: years
7. Jahrhunderts :: century
8. fast :: almost
9. Wesen :: being
10. letzten :: last
11. Lebewesen :: creatures
12. doch :: yet
13. sterblich :: mortal
14. Tun :: concerns
15. intelligente :: intelligences
16. 19 :: nineteenth
17. andere :: themselves
18. Wesen :: men
19. erforschen :: scrutinised
20. erforscht :: scrutinise
21. gedacht :: studied
22. vermehren :: multiply
23. intensiv :: keenly
24. wasser|tropfen:: drop of water
25. mensch|(lich) :: man
Table 2: A bead from ‘The War-of-the-Worlds’ parallel text (Wells, H G 1898). Dictionary correspondences are listed in the rightmost column under 1 – 23. 24 and 25 are correspondences found by the morphological analyser.
24: … at its nearest distance only 35,000,000 of miles sunward of them , a morning star … 24: ... in nächster Entfernung , nur 35.000.000 Meilen sonnenwärts , einen Morgenstern ... …
morgen|stern :: morning star
sonne|(n)|(wärts) :: sun|ward
…
Table 3: A bead from ‘The War-of-the-Worlds’ parallel text: Correspondences found by our tool are listed in the rightmost column; morphological elements are delimited by '|', bound morphological elements are bracketed
To summarize the desiderata. In order to find more matches – and thus to improve the distance function
o complex words should be analysed into their morphological elements in both languages
o as many elements as possible – but not necessarily all elements – should be matched with corresponding elements in the other language
In the following section we describe the morphological assumptions and the functioning of the word formation component.
3 The word-formation component
One basic assumption that lies behind our word formation component is that many of the unknown/unmatched complex words are productively formed words and as such are formed by regular morphological procedures. This means that chances are high that we will find translations for the morphological elements that are the result of our analysis. There are two general problems in the automatic analysis of complex words: (1) finding the morphological elements and (2) choosing the correct analysis from a number of possible analyses. Both problems are notoriously difficult (see ten Hacken & Lüdeling 2002 for an overview of automatic word formation systems).
Problem (1) – finding the morphological elements of a complex word – is difficult in concatenating languages like German (English or Romance languages, for example, have much more graphemic cues - spaces or hyphens that show morpheme boundaries). This is made even more difficult by stem changes such as linking elements (1a), elisions (1b) or umlauts (rounding of front vowels, 1c) in word
821
formation (morpheme boundaries are marked by '·'). The same element may have different forms in derivation and compounding (or even in different compounds). Stem changes are lexical and thus cannot be dealt with by rules (Fuhrhop 1998).
(1a) Glocke 'bell' – Glocken·läuten 'bell ringing';
Erlösung 'deliverance' – Erlösungs·botschaft 'message of deliverance'
(1b) Sprache 'language' – sprach·lich 'linguistic – Sprach·kurs 'language class'
(1c) Stadt 'city' - städt·isch 'urban' - Städte·bau 'urban development'
We deal with stem changes in a heuristic way. If a form is unknown we try to 'remove' all possible linkers and undo umlauting. This leads to many unwanted ambiguities, of course. This is a problem that could be dealt with by providing lexical word formation stems as suggested in (Lüdeling & Fitschen 2002).
Another factor that leads to ambiguities are bound elements. As described above, we need to analyse derivations as well as compounds.2 This can only be done if bound elements (affixes) are listed in the lexicon along with the free elements. Therefore we added 192 bound elements for German and 56 bound elements for English.3 These bound elements tend to be short and therefore also lead to many ambiguities. Many of these – such as (s)|(o)|(n)|(n)|(en)|(wärts) for sonnenwärts – 'sunwards' – are, of course, morphological nonsense but only a full-blown morphological system could resolve these, and such a system is currently not available for German (see Schmid et al. 2001).
This brings us to problem (2) : sometimes there are many possible analyses for one complex word. In our case – unlike in many other NLP applications – we do not have to choose a 'best' or 'correct' analysis between many possible analyses because we have as an added cue the corresponding words in the other language. As long as the correct analysis sonne|(n)|wärts is available the analysis (s)|(o)|(n)|(n)|(en)|(wärts), for example would not be chosen because in one case both elements can be matched while in the other case only wärts can be matched. We can therefore live with a crude and heuristic morphology system (although sometimes, of course, incorrect analyses are chosen, but in most cases our approach leads to more matches between two sentences and therefore to a better distance function).
Our algorithm first matches those word pairs that can be found in the dictionary and then processes unknown complex words: they are first analysed into their morphological parts – for all possible analyses, all the elements are then matched against the left over words in the other language. Disambiguation of multiple analyses is thus made possible by the other language and new correspondence pairs can be added to the dictionary. Note that not every morphological element needs to have a corresponding element in the other language – even partial matches improve the alignment quality. The next section describes how complex words are broken down into possible sequences of elements. This is followed by the presentation of two different strategies for matching the elements of a complex word with their correspondences in the other language.
A complex word is analysed by recognizing its elements from right to left.4 Usually, there are a number of strings (known free and bound morphemes) that are suitable as elements, which leads to a variety of possible analyses. All arising analysis alternatives are collected and returned.
An agenda-based search is conducted to construct alternative analyses. The agenda contains analysis states – i.e. snapshots of analyses in progress. A analysis state comprises of the string remaining to be decomposed and a sequence of previously recognized elements. An element can be a known free or bound morpheme.The agenda is initialized with an analysis state consisting of the complex word and an empty list of recognized elements.
While the agenda is non-empty
• the first analysis state is removed and examined.
o Should the analysis state be complete, it is collected in the result set. An analysis state is complete if its remaining string is empty, i.e. nothing is left to be analysed. The list of recognized elements then constitutes a legal analysis.
2 Conversion is not a problem in our approach We cannot deal with other types of word formation here. 3 Up to now the bound elements in our lexicon have no translations. It is possible to find regular translation correspondences for affixes that can then be added to the lexicon and used to establish more correspondences. There are obvious cases such as the German negative prefix un- that very often corresponds to in- or un- in English (unaufgeklärt – unexplained, unsichtbar – invisible). There are, however, interesting less obvious cases such as the German suffix -artig which often corresponds to -like (tischartig – tablelike, spinnenartig – spiderlike) but which is often -ular in neoclassical words (fühlerartig – tentacular) or -y in native English words (milchartig – milky, tintenartig – inky). We are still in the process of evaluating our data. 4 This is due to the fact that German and English are morphologically right-headed. However, since we take all possible analyses into account the analysis direction does not affect the result.
822
o Otherwise the analysis state is incomplete. In this case its successors are computed and appended to the agenda.
When the agenda has become empty, the result set is returned.
Successors can be computed for each incomplete analysis state as follows:
• Orthographic variations of the remaining string are obtained. For example umlauting in German is reversed.
• For each variation (including the original string)
o a set of free and bound morphological elements (stems, affixes, linkers) that match the right end of the remaining string is constructed using a dictionary.
o for each of the applicable morphological elements a new analysis state is created whose remaining string is shortened by the element which is added to the list of recognized elements. If no morpological element can be cut off the end of the remaining string, no successors are generated.
The computed analyses are ranked using a simple quality measure (quality := number of free morphemes / total number of morphological elements). The closer to 1 the better.
4 Finding correspondences of complex words in parallel text segments
As noted above, many complex words have compositional translations. The distance function component described in this paper is aimed at these. Non-compositional or non-literal translations cannot be found by the component. Each element of an analysis can have a variety of translations. Thus there is usually a large number of possible literal compositional translations for one complex word.
Two matching strategies are described in the following sections: the first strategy (the simple matching strategy) tries to match elements with unanalysed words while the second strategy (the complex matching strategy) tries to match elements with elements.
Two constraints apply: only 1:1 correspondences are possible and the involved elements must not cross sentence boundaries. Both strategies allow partial matches.
4.1 Simple matching strategy
In a given parallel text segment, all unmatched words of one language are analysed. Words that do not yield possible analyses are ignored. For each analysis an attempt is made to match its elements with unanalysed words of the other language half of the parallel text segment. A match between an L1-element and a L2-word is possible if the two are translations of each other according to the employed dictionary. All targeted words must be in the same sentence. Figure 1 illustrates this strategy.
Figure 1: Illustration of the simple matching strategy
If an element does not have a correspondence in the second language half of the segment, it is ignored, thus making partial matches possible, as illustrated above in Table 2 or the correspondence between neighbour and Nachbarplanet (literally 'neighbour planet') in Table 4. )
823
19: The secular cooling that must someday overtake our planet has already gone far indeed with our neighbour . 18: Die allmähliche Abkühlung , die auch unserem Planeten bevorsteht , ist bei unserem Nachbarplaneten schon weiter fortgeschritten . ...
nachbar|planet :: neighbour
...
Table 4: Partial correspondence between neighbour and Nachbarplanet. Example taken from ‘The War-of-the-Worlds'-parallel text.
The translation candidates are sorted according to their "length".
length := 1 + number of matched L2-words
Candidates with a length < 2 are rejected since they do not have any matching elements.
While the sorted list of translation candidates is not empty, the system
• commits itself to the longest candidate
• removes all further candidates that share words with the longest candidate to prevent the same words being part of more than one translation relation – an illegal situation
This strategy matches one complex L1-word with one or more unanalysed L2-words and prefers completely matchable analyses over partially matchable analyses. In this sense, the best decomposition of a complex word is picked from the set of available alternatives – disambiguation is achieved. The chosen analysis is not necessarily the correct one.
The procedure described above constitutes a simple but effective strategy for finding correspondences for complex words not included in the dictionary.
The elements of many English (and some German) compounds are joined together by hyphens. Separating tokens that contain hyphens in a preprocessing stage makes the simple matching strategy more successful.
67: That night another invisible missile started on its way to the earth from Mars , just a second or so under twenty - four hours after the first one . 73: In dieser Nacht nahm ein zweites unsichtbares Geschoß seinen Weg vom Mars hin zur Erde , bis auf ein oder zwei Sekunden genau vierundzwanzig Stunden nach dem ersten . …
vier|und|zwanzig :: twenty four
…
Table 5: A bead from ‘The War-of-the-Worlds’ parallel text. Separating hyphenated words during preprocessing improves the simple matching strategy. Twenty-four as one graphemic word can only be treated by the complex matching strategy.
4.2 Complex strategy: sub-word level matching
Whereas the simple strategy matches elements with words, the complex strategy matches elements with elements. In a given parallel text segment, all unmatched words of both language halves are analysed. All words are also made available as unanalysed matching targets. A great number of sub-word level matches are possible since there are several analysis alternatives whose elements possibly match a number of different elements of other analysis alternatives. Consider Figure 2.
Figure 2: Illustration of the complex matching strategy
824
The complex strategy aims towards selecting the most promising sub-word level match. All analyses of all thus processed words are sorted according to their "quality" as defined above and put into an agenda. As long as the agenda contains analyses, the best candidate is removed and an attempt to construct a sub-word level matching for the candidate is made. If the attempt is successful, all words involved in the matching are marked as unavailable and all remaining analysis candidates that depend on these words (or their elements) are removed from the agenda. The successful matching is stored in a result set which is returned when the agenda becomes empty.
A best-first search through the space of possible sub-word level matches is conducted. This space consists of states that represent such matchings in progress. Such a state consists of a list of elements belonging to an analysis alternative that still have to be matched (todo-list) and a list of those elements that have already been matched or deleted (done-list). A state with an empty todo-list is a final state that represents a complete matching. The search begins in an initial state that contains in its todo-list all the elements of the analysis chosen from the agenda. The successor states are explored until suitable complete states have been found.
To generate successor states
• the first element is removed from the todo-list
• for each element of the other language that is a correspondence according to the dictionary, a new state is created whith the following properties:
o the new done-list is the parent's done-list plus the first element and its corresponding element
o the new todo-list is the parent's todo-list without the first element and its correspondence plus all the unmatched elements that are also part of the analysis to whose element the first element corresponds.
• a further state with the first element moved to the done-list is created. This state represents a deletion of the first element which is useful if the element cannot be matched.
It is important that targeted elements of a word must belong to the same analysis alternative.
To guide the best-first search, the different sub-word level matching states are assessed using the following formula:
value := 1.0 - (completion / numwords)
completion := number of matched free elements / total number of elements
numwords := number of actual words involved
The closer to 0 the value is, the better the sub-word level matching state is deemed to be. The assessment aims to favour analyses with few elements (most of which are free) that involve a small number of words.
5 Evaluation
Two things have to be assessed. On the one hand we would like to know how well the morphological analyser performs its task and what kinds of mistakes we find. On the other hand it is important to consider the effect the analyser has on the alignment quality.
5.1 Performance of the morphological analyser
In order to judge the quality of the morphological analyser we qualitatively evaluated the longest chapter from ‘The War-of-the-Worlds’ bitext (Chapter 2-07). We looked at the complex words in the German half of the text and their processing by the simple and complex analysis strategy. Some numbers first: the simple strategy processed 118 words making 13 errors. The complex strategy processed 38 words making 7 errors – it can be seen that the simple strategy seems to perform better (see below). In many cases correspondences could be found for parts of the complex words only. We find two types of errors: (1) analysis errors and (2) matching errors. Analysis errors are most frequent. The are usually due to one or two letter elements. Examples for a German analysis error and an English analysis error are given in Table 6.
Some analysis errors can be excluded by simple heuristics (in the style of: a complex word in German can never begin with a one-letter element except a) but in the long run we would welcome a fully-fledged morphological analysis.
Matching errors are errors where the correct analysis is chosen but the elements are not matched to the corresponding words. Sometimes it happens that there are two (almost) identical possible matches, as shown by the example rötlich (which should have been matched with redly but instead was matched
825
with red) in Table 7 while sometimes a different reading is chosen, as illustrated by the example scheinen which in this case is matched with flash instead of glow (scheinen can mean both).
419: After an interminable string of games , we supped , and the artilleryman finished the champagne . 432: Nach einer endlosen Reihe von Spielen nahmen wir unser Abendessen ein , und der Artillerist trank den Champagner aus . …
ab|ende|(s)|(s)|(e):: finished
…
406: " We can dig better on this Thames - side burgundy , " said I. 419: " Es ist vielleicht besser , wenn wir bei unserem Burgunder weitergraben " , sagte ich . …
ich :: (th)|(a)|me|(s)
…
Table 6: Two beads from ‘The War-of-the-Worlds’ parallel text. Analysis errors. Abendessen 'supper' in the first row is incorrectly analysed to contain ende 'finish'. In the second row the German ich 'I' is matched to the me in Thames.
426: The northern hills were shrouded in darkness ; the fires near Kensington glowed redly , and now and then an orange - red tongue of flame flashed up and vanished in the deep blue night . 439: Die nördlichen Hügel waren in tiefes Dunkel gehüllt , die Feuer in der Nähe von Kensington schienen rötlich herüber , und hier und da zuckte eine orangefarbene Feuerzunge auf , um in der tiefblauen Nacht gleich wieder zu verschwinden . ...
schein|(en) :: flashed
rot|(lich) :: red
…
Table 7: A bead from ‘The War-of-the-Worlds’ parallel text: Matching errors
5.2 Improvement of the distance function
The War of the Worlds parallel text was aligned using four different distance functions: the base distance function consists of bilingual dictionary lookup plus cognate recognition. The other three functions are composed of the base function plus the simple and/or complex matching strategy. The thus obtained alignments were compared to a manually aligned reference. Table 8 shows the resulting recall (recall := number of correct beads / number of reference beads).
base base + simple base + simple + complex base + complex War of the Worlds 96.51 % 97.21 % 97.16 % 96.37 %
Table 8: recall on ‘The War of the Worlds’ parallel text using different distance functions
The simple matching strategy improves the overall alignment quality (recall) by 0.7%. While this might seem to be only a small improvement, the difference on individual chapters was as large as 7.6% which we consider to be very successful.
Contrary, it seems that the present implementation of the complex matching strategy leads to a decrease in performance. We assume that this can mainly be attributed to awkward analyses, some of which could be filtered out using simple heuristics such as introducing a minimum word length. The impact of incorrect matches seems to outweigh that of correct matches. Thus the complex matching strategy (in its present form) should not be used in the distance function. However, when applied to the beads of a previously computed alignment, the complex matching strategy can be used to extract interesting matches that can be reviewed by a human expert.
6 Discussion
We have shown that morphological analysis can improve a dictionary-based distance function. This is true for a literary text such as ‘The War of the Worlds’ as well as for the rather technical EU debates. There are still some problems, however: the rough analysis of complex words, sketched here, often fails – in the long run a linguistically more adequate morphological analyser is desirable. Our approach can also help to improve the dictionary since correspondences can be added to it.
826
Acknowledgements
We would like to thank Petra Prochazkova for her help in the evaluation and Kim Wallum for critical comments.
References
Canadian Hansards: aligned versions available from http://www.isi.edu/natural-language/download/hansard/
Fuhrhop, N 1998 Grenzfälle morphologischer Einheiten. Tübingen, Stauffenburg
Gale, W A & Church, K W 1991 A Program for Aligning Sentences in Bilingual Corpora. In: Proceedings of the 29th Annual Meeting of the Association for Computational Linguistics, Berkeley, CA, pp 177 –184
Fung, P 2000 A statistical view on bilingual lexicon extraction. In: Veronis, J (ed) 2000, pp. 219-243
ten Hacken, P & Lüdeling, A 2002 Word Formation in Computational Linguistics. Tutorial. In: Proceedings of TALN 2002, Nancy, pp. 61-87
Kay, M & Röscheisen, M 1993 Text-Translation Alignment. Computational Linguistics 19, pp. 121-142
Lüdeling, A & Fitschen, A 2002 An integrated lexicon für the automatic analysis of complex words. In: Proceedings of the Tenth International EURALEX Congress, Copenhagen, vol 1, pp. 145 – 152
RAPID, The Press and Communication Service of the European Commission 1995-2002, RAPID is a database giving a daily view of the activities of the European Union, http://europa.eu.int/rapid/start/welcome.htm
Schmid, T, Lüdeling, A, Säuberlich, B, Heid, U, Möbius, B 2001 DeKo: Ein System zur Analyse komplexer Wörter. In GLDV - Jahrestagung 2001 pp. 49-57.
Schmid, G 1994 TreeTagger - a language independent part-of-speech tagger. Manuscript. Available as http://www.ims.uni-stuttgart.de/Tools/DecisionTreeTagger.html.
Steven, G A 1994 String Searching Algorithms. Singapore, World Scientific Publishing
Tschorn, P 2002 Automatically aligning English-German parallel texts at sentence level using linguistic knowledge. Unpublished Master's thesis, University of Osnabrück
Veronis, J (ed) 2000 Parallel Text Processing – Alignment and use of Translation Corpora. Kluwer, Dordrecht
Wells, H G 1898 The War of the Worlds.
English version available from: www.fourmilab.ch/etexts/www/warworlds/
German version available from: www.geocities.com/Area51/Corridor/8282/
827

The Economist and The Financial Times. A study of movement metaphors 
Francesca Vaghi, Marco Venuti 
Dipartimento di Scienze Statistiche 
Universita degli Studi di Napoli Federico II 
1. Introduction 
This paper features the analysis of metaphors describing market movements in a corpus of web retrieved articles taken from the electronic archive of the periodical The Economist and the newspaper The Financial Times. The reason for choosing these two sources comes from the need to create a semi-technical corpus about the language of economics meant for a more specialised readership. All the articles have been published between July 2001 and June 2002, that is six months before and six months after the introduction of the single currency. The reasons for opting for this span of time are that it is the period of maximum interest in the historical event of the currency changeover, and that it could grant us the opportunity to examine market movements in texts focusing on an equivalent subject matter. 
As for the analytic framework chosen for the linguistic investigation, we have based it on the cognitive theories of metaphors (Lakoff and Johnson 1980) which see them as linguistic expressions and mental representations; in particular, we have based our analysis following the theories of previous researchers who applied cognitive theories to the study of the language of economics (Charteris-Black 2000, Charteris-Black and Ennis 2001, Henderson 1994). 
The aim of this study is to illustrate the use of equivalent categories of metaphors of movement in the two papers. Through the comparison of the hits it is also possible to reveal some significant differences in the frequency of particular instances even within the same domain. For example, a preliminary quantitative analysis of The Financial Times has shown twice as many hits as The Economist from the domain MARKET MOVEMENTS ARE NAUTICAL OR WAYS OF MOVING IN THE WATER. 
The examination of participants and circumstances involved in movement metaphors made it possible to enquire into the nature of the economy and the market. That is, we have differentiated between economy and market moving as autonomous entities and as entities whose movements depend on human intervention. 
The various electronic searches on the corpus will develop results which might give an insight into the way the newspaper and the magazine build their discourse strategies. 
2. British Press and the Euro Corpus 
This research is part of a work in progress involving the editing of a corpus of texts about the euro currency offering an exhaustive overview of the way the British press has portrayed the new-born European currency. The corpus is to be representative of a range of texts meant for a more specialised audience in economics, and texts taken from quality papers, therefore intended for a wider and more generalised readership. This is why it is a collection of articles retrieved on the one hand from the websites of The Economist and The Financial Times, and on the other hand from those of The Guardian and The Times. 
The articles have been selected according to two main criteria: the date of publication and the relevance to the topic. All selected articles have been published between July 1 2001 and June 30 2002, six months before and six months after the launch of the single currency, which gave us the opportunity to investigate the different attitudes of the papers also in terms of predictions and the immediate aftermath concerned with such an important event, a milestone in the history of European countries. 
As for the relevance to the topic, the selected articles contain either the phrase “the euro” or “single/European currency” in the headline and/or the lead. 
Of all the articles retrieved, a hundred were randomly sampled from each paper to create four sub-corpora of around 60.000 words each. The only exception to this rule being the sub-corpus of The Economist: the magazine featured an inferior number of articles which were much longer in length than those in the broadsheets, so we have decided to take only 50 articles scoring a total of around 50.000 words. In the sampling process of the four sub-corpora we have also taken into consideration the number of articles published each month, as we intended to assemble a corpus following precise criteria which would render it representative of the uneven editorial coverage throughout the twelve 
828
months. As it is, the articles published in December and January, the two months closer to the introduction of the new single currency, were three times as many as the monthly average. 
In a previous study based on our corpus (Vaghi and Venuti 2003) we chose a different approach for the analysis of the newspapers The Guardian and The Times. We investigated the way the comparison of the metaphors the EURO IS A CONTAINER and the EURO IS A MECHANICAL OBJECT could well represent the discourse strategies of two ideologically opposed broadsheets. 
The similar number of hits concerning the possible UK entry in the EMU was evidence to the inevitability of such an important topic in both newspapers, regardless of their political stance, but a more careful analysis also revealed some important differences between the two domains. 
For example, both broadsheets used the EURO IS A CONTAINER metaphor associating a constant discourse function to it. As container metaphors focus on the opposing ideas of being in/out, they were the perfect category representing this much discussed issue in both papers. The Times usually provided a negative connotation to the euro, emphasizing the existing contrast between Britain and the European Monetary Union. On the other hand, The Guardian had the tendency to highlight the positive aspects of being within the EMU, criticising the possible isolation resulting from staying outside of it. 
We think metaphors can help understanding discourse strategies, and this article is one more example of the role of these figures of speech in the comprehension of the underlying diverse ideologies in papers. 
Before analysing some examples from the two sub-corpora of The Economist and The Financial Times we will briefly introduce the theoretical background on metaphors we applied and the methodology we followed to identify the metaphors related to the topic of this research. 
3. Theoretical background 
The language of Economics is highly metaphorical (Mc Closkey 1993, Mason 1990) and uses images taken from other contexts because these figures of speech are useful in understanding and interpreting a text. This happens as 
Metaphors are supposed to facilitate and enhance understanding and to add an additional level of meaning by choice of a linguistic expression that may not be the conventional or ordinary way of expressing an idea (Charteris-Black and Ennis 2001, p 252). 
The first and foremost feature of metaphors can facilitate and enhance understanding because they describe reality exploiting the transference of meaning from one form to another (Beccaria 1989). Keeping this as the predictable starting point for an analysis of metaphors, we have decided to study the examples from our sub-corpora following the cognitive theory developed by Lakoff and Johnson 1980, which see them as both linguistic expressions and mental representations. According to this theory metaphors are conventionally represented via the relation A is B, where A is the ‘target’ domain and B is the ‘source’ domain. In this paper we will compare the metaphor MARKET MOVEMENTS ARE PHYSICAL MOVEMENTS in the two sub-corpora of The Economist and The Financial Times. After the above schema MARKET MOVEMENTS are the target domain A, whereas PHYSICAL MOVEMENTS are the source domain B. 
We think that the most significant reason why we have chosen this approach to the analysis of metaphors is that by subsuming the relation between target and source as constant, it always grants the comparability between entities with similar features, in fact we are comparing the same types of metaphors in two sets of texts. On the other hand, this theory gives space to inventiveness, as we can find virtually infinite variations to the types of combinations target/source involved in the creation of a metaphor. As the pioneers of the cognitive theory about metaphors put it: 
The most salient of the dimensions of poetic power … is the power to extend the basic metaphors … This is the power that metaphor has to reveal comprehensive hidden meaning to us, to allow us to find meaning beyond the surface, to interpret texts as wholes, and to make sense of patterns of events. (Lakoff and Turner 1989, p 159) 
As for the study of economy and market moving autonomously or depending on human intervention, we have based our analysis of functional theories, conceiving movement metaphors as processes involving participants and circumstances (Halliday 1994, pp 106-112). 
829
4. Methodology 

Table 1MARKET MOVEMENTS ARE WAYS OF MOVING IN THE AIR Economist Financial Times Lexis No. Lexis No. Bail out 1 Backdrop 2 Blow 2 Bail-out 1 Boost 8 Bloat 1 Breeze 1 Blow 1 Bubble n 1 Boost 16 Drop 8 Bubble 1 Expand 2 Drop 3 Fall 93 Expand 6 Lift 1 Fall 89 Rise 56 Land 3 Plummet 1 Lift 3 Pressure 12 Plummet 1 Soared 1 Pressure 17 Swing 1 Rise 56 Soar 1 Sway 1 Upset 1 Waver 1 Whiff 1 188 205 
We began our analysis having in mind the definition of the nature of metaphorical language applied to economic texts: “it is part of the generative aspect of metaphor to assist in the development of a routine vocabulary for handling economic ideas” (Henderson 1994, p. 358). 
Bearing Henderson's words in mind and considering that markets and economy are often seen as moving entities, we have decided to proceed to the examination of all the terms picturing these abstract 
entities as physical movements, and classify them within the metaphorical domain MARKET MOVEMENTS ARE PHYSICAL MOVEMENTS. 
Following the example of Charteris-Black and Ennis (2001, pp. 256-258) we have further 
categorised the metaphors into three sub-domains according to the medium where the movement takes place: MARKET MOVEMENTS ARE WAYS OF MOVING IN THE AIR, MARKET MOVEMENTS ARE NAUTICAL OR WAYS OF MOVING IN THE WATER and MARKET MOVEMENTS ARE WAYS OF MOVING ON THE GROUND. 
To carry out an extensive and systematic identification and inspection of the three sub-domains we have searched our two sub-corpora using the software package WordSmith Tools 3.01. 

Table 2MARKET MOVEMENTS ARE NAUTICAL OR WAYS OF MOVING IN THE WATER Economist Financial Times Lexis No. Lexis No. Dip 1 Boil 1 Drift 1 Dip 1 Flow 8 Ebb 1 Fluctuate 1 Float 1 Founder 1 Flow 21 Inflow 4 Fluctuation 4 Plunge 4 Freeze 1 Pump 1 Inflow 11 Squeeze 5 Outflow 12 Stream 1 Ripple 1 Surge 5 Squeeze 2 Undermine 5 Undermine 4 Watershed 1 Wade 1 Wave 1 Watershed 1 Wave 2 39 64 
Our first step was the study of the two wordlists obtained from the two sub-corpora of The Economist and The Financial Times: within the frequency lists we looked for all the verbs and nouns that could be related to the metaphorical domain MARKET MOVEMENTS ARE PHYSICAL MOVEMENTS. The extensive examination of WordList (A) and WordList (F) allowed us to identify all their inflected. 
The concordance lines of all the terms we had so far identified were printed out and manually checked in order to delete those cases where the search words expressed a literal rather than metaphorical meaning, and when they did not refer to market or economic movements. 
The results of this first analysis were included into Tables 1, 2 and 3 according to the classification of movements we have previously referred to. Each term within the lexis column includes the occurrences of both verbs and nouns, together with their inflected forms. 
1 For a detailed description of all the features of WordSmith Tools see Mike Scott, 1996. 
830
5. Analysis of data 

Table 3 MARKET MOVEMENTS ARE WAYS OF MOVING ON THE GROUND Economist Financial Times Lexis No. Lexis No. Lexis No. Lexis No. Backslide 1 Race 1 Bounce 1 Run 2 Bounce 1 Rebound 4 Bounceback 1 Run up 1 Climb 4 Run 1 Climb 5 Rush 1 Collapse 1 Run-up 2 Collapse 6 Slide 5 Creep 1 Rush 1 Crawl 1 Slide 3 Curve 2 Slide 5 Creep 9 Slip 2 Drag 2 Slip 3 Curve 1 Slope 1 Draw 1 Sneak 1 Drag 5 Step 7 Drive 4 Speed 2 Draw 2 Stuck 1 Edge 2 Spring 1 Drive 6 Stumble 1 Falter 2 Stall 2 Edge 4 Tumble 3 Hike 2 Step 8 Falter 1 Hobble 1 Stumble 2 Footing 1 Hurdle 3 Sweep 1 Hike 1 Jump 5 Trail 1 Jump 2 Leap 1 Tumble 7 Pace 6 Pace 3 Worm 1 Rebound 4 79 80 
A first comparison can be drawn by simply looking at the results of the total scores in the three tables. The figures show that, as far as the movement metaphors are concerned, The Financial Times employs a much richer figurative language than The Economist. In Table 1: MARKET MOVEMENTS ARE WAYS OF MOVING IN THE AIR, The Financial Times counts nearly 10% more metaphors than The Economist. The gap is even wider in Table 2: MARKET MOVEMENTS ARE NAUTICAL OR WAYS OF MOVING IN THE WATER, where the daily paper totals around 36% more hits than the magazine. This differentiation between the two papers is not repeated in Table 3: MARKET MOVEMENTS ARE WAYS OF MOVING ON THE GROUND. 
When editing the tables we have included all the terms referring to economy and market movements, but in studying the concordances we have decide to rule out words such as fall and rise whose frequency in the vocabulary of economic is so high we had doubts regarding their primary meaning being physical rather than not abstract. As Smith (1995) puts it 
A number of what were originally metaphors have become conventionalised in the language of economics, and can now better be considered as technical terms than “living metaphors”. Such terms are equilibrium, float, inflation, leakage, boom; liquidity and slump are now so familiar in the jargon of the subject that their metaphorical etymology is not immediately obvious. (Smith 1995, p 45) 
The terms: fall, rise, float, flow, inflow, outflow, drop and pressure were therefore excluded from our investigation. 
We then analysed the context of each term by looking at all concordances. In order to evaluate the role of the metaphors each concordance has been expanded to include, at times, the whole paragraph or even the whole article. 
During the examination we checked whether movement metaphors were referred to economy and market as independent, autonomous entities or as dependent as animate agents. Our approach derives from an established trend in the study of the language of economics relating animate agents with the possibility of controlling economic events, whereas inanimate ones express the impossibility of reliable forecast and lasting theories on market fluctuations (Charteris-Black 2000, pp 160-162). 
831
Our evaluation of movement metaphors is based on Halliday's notion of process2 “consist[ing], in principle, of three components: (i) the process itself; (ii) participants in the process; (iii) circumstances associated with the process” (Hallyday 1994, p 107). The participants can therefore be described as Actor, “the one that does the deed” (Hallyday 1994, p 109), and Goal “the one to which the process is extended” (Hallyday 1994, p 110). 
We present below a few expanded concordances featuring examples3 of animate and inanimate Actors of the processes of movement for both the Economist and the Financial Times. The following two examples are drawn from the domain MARKET MOVEMENTS ARE WAYS OF MOVING IN THE AIR. 
E1 The dollar will fall if they merely reduce the pace at which they add to their holdings. It will fall even more if American investors continue or expand this year's buying of foreign assets, notably European equities. Until recently, net capital inflows more than covered America's current-account deficit. 
E2 Hours worked also fell by 0.4%, after dropping by 0.3% in July. The downturn seems to be spreading to the services sector. The NAPM index of non-manufacturing activity plummeted to 45.5 in August from 48.9 in July. A number below 50 is a sign that activity is contracting. 
In E1 the Actor “American investors” of the Process “expand” is a human entity enacting on the Goal “buying foreign assets”. The whole process might cause an alteration of another process, that is a further decrease in the value of the dollar. On the other hand E2 is a process consisting of a non-human Actor “the NAPM index”, a Process “plummeted” lacking a Goal. The absence of a Goal together with the presence of a personified inanimate Actor exemplifies what we mean by an autonomous economic or market movement. The fall of the index is in fact presented as a self-governed movement that follows an independent trend. 
The same approach has been carried out for the concordances obtained from The Economist as exemplified by the analysis of the next two expanded concordances from the domain MARKET MOVEMENTS ARE NAUTICAL OR WAYS OF MOVING IN THE WATER. 
FT1 "You have to look back to 1995 for the last time the US intervened to push the dollar higher and, since then, it has twice waded into the market to bring the dollar lower," he added. In spite of the rhetorical support for a strong dollar it is thought unlikely that the US would stand in the way of a gradual depreciation of the dollar. 
FT2 Germany has no inflation problem. Annual inflation is estimated to have fallen to 1.2 per cent this month from 1.6 per cent in April. In June it may dip to 1 per cent or less, giving Germany the lowest inflation in the 12-nation eurozone. 
In FT1 “the US” is the animate Actor intervening into the market, wading into it to affect the exchange rate of the dollar. As in E1 we have two participants, an Actor extending the Process on a Goal, to express a direct intervention to change an economic tendency. The second example from The Financial Times shows once more the independent nature of market movements through the lack of a Goal. The Process “may dip” has an Actor “annual inflation” in Germany but no Goal. The impossibility to extend the process onto a second participant and the inanimate personified nature of the Actor have been used to convey the absence of a human involvement in what is portrayed as an autonomous progression. 
6. Concluding remarks 
The results of the analyses of animate and inanimate Actors are presented in Table 4. From the total scores it is possible to see how there is a tendency common to both publications to express economy and market movements through processes enacted by inanimate Actors rather than animate ones. This is true of all metaphorical domains but MARKET MOVEMENTS ARE NAUTICAL OR WAYS OF MOVING IN THE WATER. The explanation can be found in the very high frequency of the terms flow, inflow and outflow in this domain particularly in The Financial Times, as the newspaper makes an extensive use of semi-technical vocabulary, which has lost its original metaphorical nature, to express movement. 
2 In discussing the examples process refers to the hallidayan semantic category, while Process refers to ‘the process itsel'. 
3 Examples from The Economist will be introduced by a capital E followed by a progressive number, those from the Financial Times will be introduced by the capital letters FT followed by a progressive number. 
832

Table 4 Economist Financial Times AnimateInanimateAnimate Inanimate MARKET MOVEMENTS ARE WAYS OF MOVING IN THE AIR 11 22 21 46 MARKET MOVEMENTS ARE NAUTICAL OR WAYS OF MOVING IN THE WATER 14 13 11 9 MARKET MOVEMENTS ARE WAYS OF MOVING ON THE GROUND 29 50 20 60 54 85 52 114 
The predominance of inanimate Actors, generally in conjunction with the lack of an expressed Goal is a confirmation of what Charteris-Black (2000) had stated on the co-occurrence of animate subjects and transitive verbs, and that of inanimate subjects and intransitive verbs 
“We should also note that the verbs found with animate subjects are generally transitive while those found with inanimate subjects are generally intransitive. What we find here is further evidence of the concealment of human agency in the case of representations of the market.” (Charteris-Black 2000, p 162) 
If we compare the total scores of animate and inanimate Actors for The Economist and The Financial Times a further comment can be made. The latter tends to express economic movements as being autonomous and independent more often than the former. Together with the wider use of semi-technical terms expressing movement this observation could lead to the conclusion that The Financial Times tends to use a more technical vocabulary. This could be explained by the different nature of the two publications. 
Being a weekly paper The Economist tends to focus more on the commentary of market and economy movements, paying more attention to the causes that produced these changes, such as the direct intervention of governments and banking institutions. The Financial Times on the other hand has to follow and report the movements and fluctuations of the economy and the market day by day. This could explain both why it resorts more often to well established and consolidated terms, i.e. flow, and why it focuses its attention to the movements themselves, therefore portrayed as autonomous events, rather than investigating their causes. 
References 
Anderson P J, Weymouth A 1999 Insulting the Public? The British Press and the European Union. London, Longman. 
Beccaria G L (ed.) 1989 Dizionario di linguistica. Torino, Einaudi. 
Biber D 1993 Representativeness in Corpus Design. Literary and Linguistic Computing 8: 243-257. 
Charteris-Black J 2000 Metaphor and vocabulary teaching in ESP economics. English for Specific Purposes 19(2): 149-165. 
Charteris-Black J, Ennis T 2001 A Comparative Study of Metaphor in Spanish and English Financial Reporting. English for Specific Purposes 20(3): 249-266. 
Halliday M A K 1994 An Introduction to Functional Grammar. London, Arnold. 
Henderson W 1994 Metaphor and Economics. In Backhouse R E (ed), New Directions in Economic Methodology. London, Routledge, pp 343-367. 
Henderson W, Dudley-Evans T, Backhouse R (eds) 1993 Economics and Language. London, Routledge. 
Lakoff G, Johnson M 1980 Metaphors We Live By. Chicago, University of Chicago Press. 
Lakoff G, Turner M 1989 More than Cool Reason. A Field Guide to Poetic Metaphor. London, University of Chicago Press. 
Mason O 1990 Dancing on air: Analysis of a passages from an economics textbook. In Dudley-Evans T, Henderson W (eds) The Language of Economics: The Analysis of Economics Discourse. Hong Kong, ELT Document 134, Modern English Publications and the British Council, pp 16-28. 
McCloskey D 1983 The rhetoric of economics. Journal of Economic Literature 21: 481-517. 
Musolff A 1997 International metaphors: bridges or walls in international communication? In Debatin B, Jackson T R and Steuer D (eds) Metaphor and Rational Discourse. Tübingen, Niemeyer, pp 229-237. 
833
Musolff A 2000a Mirror Images of Europe. Metaphors in the Public Debate about Europe in Britain and Germany. München, Iudicium. 
Musolff A 2000b Maritime journey metaphors in British and German public discourse: transport vessels of international communication? German as a foreign language 3/2000 
Musolff A 2001 Cross-language metaphors: parents and children, love, marriage and divorce in the European family. In Cotterill J, Ife A (eds) Language across boundaries. Padstow, TJ International Ltd, pp 119-134. 
Partington A 1998 Patterns and Meaning. Amsterdam, Benjamins. 
Scott M 1996 Comparing corpora and identifying key words, collocations, and frequency distributions through the ‘WordSmith Tools’ suite of computer programs. In Ghadessy M et al (eds) Small Corpus Studies. Theory and Practice. Amsterdam, Benjamins, pp 47-69. 
Smith G 1995 How high can a dead cat bounce? Metaphor and the Hong Kong Stock Market. Hong Kong Papers in Linguistics and Language Teaching 18: 43-57. 
Vaghi F, Venuti M 2003 Metaphor and the euro, forthcoming article. 
834

An evaluation of three POS taggers for the tagging of the Tswana Learner English Corpus 
Bertus van Rooy & Lande Schäfer 
Potchefstroom University 
Private Bag X6001, Potchefstroom 
2520, South Africa 
Phone: +27-18-299 1487/1504 Fax: +27-18-299 1562 
E-mail: ntlajvr@puknet.puk.ac.za (Van Rooy) ntlyvs@puknet.puk.ac.za (Schäfer) 
1. Introduction 
Before starting with part of speech (POS) tagging on our corpus of learner English we decided to evaluate three POS taggers to see which one gives the best results when tagging written second language English. We evaluated the taggers' performance to determine which tagger would be most suitable for linguistic analyses on a POS-tagged corpus that had not been tag-edited. Once the accuracy of the taggers had been determined, we investigated the factors that contributed to inaccuracy with a view to establish time and cost effective ways of increasing tagger accuracy without necessarily tag-editing the corpus from beginning to end. The aim of this research was to explore the possibility of selective tag editing based upon specific tokens or tags frequently associated with tagging errors. 
2. The Tswana Learner English Corpus project and the POS taggers 
At the end of 2000 we started compiling the Tswana Learner English Corpus (TLEC). Setswana is spoken as a mother tongue in the North West and Northern Cape Provinces of South Africa and in Botswana. Most of the South African speakers of Setswana learn English as a second language in school. When it is complete, this 200 000-word corpus will form part of the International Corpus of Learner English (ICLE). 
Currently there is significant movement towards the fostering of a local human language technology industry in South Africa. Our aim is to collect and POS tag more corpora of South African varieties of English and of the other ten indigenous languages. As TLEC will be the first POS tagged corpus of a South African variety of English, we wanted to establish procedures for time and cost-effective POS tag-editing. 
Last year we started with POS tagging. As TLEC will form part of ICLE, it needs to be tagged by TOSCA-ICLE. TOSCA-ICLE has a very large tagset and undertakes complex subcategorisation of the main wordclasses. We were interested to see how TOSCA-ICLE would perform on our data and how its performance would compare to the performance of taggers with less complex subcategorisation withint its tagset. We decided to also evaluate the Brill-tagger (Brill 1999) and CLAWS7 (see Garside & Smith 1997 for discussion of an earlier version of CLAWS). 
3. Method of evaluation of taggers 
We based our evaluation of the taggers on the four considerations for the selection of a tagger that are identified by Van Halteren (1999b:95-98): 
 the tagset, 
 documentation, 
 the tagging process, and 
 performance. 
The Brill-tagger makes use of the Penn Treebank Tagset, which is quite a small tagset consisting of only 36 tags. The CLAWS7 tagset has 137 tags, excluding the punctuation tags. TOSCA-ICLE has a possible 220 tags that it can assign, due to the fact that it allows for subcategorisation on various levels. Thus, the tagset as criterion concerns the types of contrasts that the tagset identifies. We have taken the limitations of each tagset into account in our judgment of tag errors. Documentation refers to 
835
whether all the relevant documentation about the tagset, the tagging process and the types of assumptions underlying the various operations are available, to facilitate correct interpretation of results. We easily obtained documentation regarding all three taggers via the Internet, and they all contained the relevant information. Considerations regarding the tagging process includes the time it takes, the user interface it provides and the platform on which it operates. Performance refers to how accurate a tagger is at assigning POS-tags to corpus data. 
We focused our evaluation on measuring the taggers’ performance. Van Halteren (1999a) proposes a number of measures for this. The most basic measure is overall tagger accuracy, which is calculated by dividing the number of correct tags into the total number of tags. Precision measures how many tokens that received a tag Y received that tag correctly, thus precision EQUALS number of tokens tagged Y correctly DIVIDED BY total number of tokens tagged X. Recall measures how many tokens that should be tagged Y are indeed tagged Y. Thus, recall EQUALS number of tokens tagged Y correctly DIVIDED BY total number of tokens that must have been tagged Y. 
We randomly selected 13 essays (each approximately 400 words) from the corpus which provided us with a 5000 word sample (a total of 5618 tokens, including punctuation). The sample was tagged with all three taggers. The Brill and TOSCA-ICLE tagging were done in the language technology laboratory in Potchefstroom on Linux and DOS respectively. The CLAWS7 tagging was done by UCREL. 
We followed a comparative method in tag-editing. We entered the data into a Microsoft Excel file. Each tagger's data was represented in three columns. The first column contained the tokens, the second the tag assigned to that token, and the third the tag correction if there was a correction. We placed the data of the three taggers next to each other and aligned the data according to the tokens. The authors of this paper undertook the editing together in order to validate each other's interpretation of the learner's sentences as well as each other's judgement of the correctness of the tags. Being able to see the tags assigned by all three taggers sometimes helped us in deciding whether we would accept a certain POS-tag as correct or not. 
After editing the tags we calculated the accuracy of the taggers to determine whether it would be possible to undertake linguistic analyses and the accompanying POS-based searches on the unedited tagged corpus. We wanted to establish which tagger would give us the most peace of mind with searches prior to tag-editing, and whether some searches could take place without tag-editing having taken place. 
4. Results of evaluation of taggers 
The overall accuracy for all three taggers are presented in Table 1. In terms of performance, CLAWS7 fared the best, while the TOSCA-ICLE tagger fares substantially poorer than the 05% accuracy reported by De Haan (2000) for the same tagger on other varieties of learner English. It would be possible to undertake POS-based searches for specific grammatical constructions on a corpus that has been tagged with CLAWS7 but not tag-edited. However, the ideal is to work with a corpus with POS tags that are as accurate as possible, and the next step was to find ways to improve tagger accuracy by way of tag-editing, as explained in section 5. 
Table 1 Overall accuracy 
Accuracy Brill 86.34% CLAWS7 96.26% TOSCA-ICLE 88.04% 
We have made the decision to tag TLEC with both TOSCA-ICLE and CLAWS, and we therefore set out to examine ways of improving their accuracy. Because we don't rule out the possibility of using the Brill-tagger on data in future, we also looked at ways in which its accuracy could be improved. As we have already tagged most of our corpus with TOSCA-ICLE, we will begin by editing the TOSCA-ICLE tags first, and therefore tagging procedures for this tagger will be discussed in more detail in section 6. 
836
5. Method of improving tagger accuracy 
The time and effort that it took to tag-edit the three tagged versions of our 5000-word sample indicated to us that it would not be viable to tag-edit the whole corpus (and also other corpora) in this manner. We decided to carry out selective editing, by establishing which tags were frequently assigned incorrectly and which tokens were frequently tagged incorrectly. One would then be able to search for problematic tags or tokens to check whether they need correcting or not. 
After editing the tags, the precision and recall rates for each tag was calculated. From this it was possible to identify tags which had a particularly low precision or recall. Table 2 represents the three tags with the lowest precision for each tagger. For the sake of interest the recall rates for these tags are also given. The data in Table 2 indicates that it is not enough to simply consider the precision rates for the individual tags. The frequency with which the tag occurs in the corpus should also be taken into account. It would, for instance, be worthwhile to do a search for all CD tags in the Brill-tagged sample and to systematically correct all of them, because they constitute 5.59% of all tags, whereas it would not be worth the effort to search for all PRON(cleft) tags to ensure that they are correct, since they only constitute 0.02% of all the tags. 
Table 2: Tags with the lowest precision for each tagger 
Frequency Precision Recall Brill-tagger CD 5.59% 13.31% 95.35% RBS 0.16% 44.40% 100% NNPS 0.09% 60.00% 100% CLAWS7 RGR 0.08% 50.00% 100% RRR 0.24% 50.00% 100% DDQ 0.61% 58.06% 100% TOSCA-ICLE PRON(cleft) 0.02% 0.00% n/a PROFM(so,clause) 0.02% 0.00% n/a VB(aux,modal,infin) 0.04% 0.00% n/a 
The question then is not “What are the tags with the lowest accuracy?”, but “What are the tags that contribute most significantly to the overall error rate (the opposite of the overall accuracy)?”. We calculated each tag's contribution to the overall error rate of the tagger by dividing the number of times a specific tag was assigned incorrectly by the total number of tag errors. Tables 3, 4 and 5 shows which tags contributed most significantly to the tagging errors made by the various taggers. 
Table 3 Contribution to errors (Brill-tagger) 
Brill tag Percentage of total tag errors VB 19.55% NN 16.46% IN 14.81% VBP 11.11% 
Table 4 Contribution to errors (CLAWS7) 
CLAWS tag Percentage of total tag errors NN1 24.87% JJ 10.52% VV0 10.52% ND1 7.94% 
837
Table 5 Contribution to errors (TOSCA-ICLE) 
TOSCA-ICLE tag Percentage of total tag errors ADJ(ge,pos) 12.23% VB(lex,montr,infin) 8.10% ADV(ge,pos) 6.27% CONJUNC(subord) 5.50% N(sing) 5.50% PREP(ge) 5.20% N(plu) 3.82% VB(lex,montr,imp) 3.82% 
Editing only the tags contained in Table 3 would eliminate more than 60% of the tagging errors made by the Brill-tagger. Similarly, editing the tags listed in Table 4 would eliminate more than half of the tagging errors made by CLAWS7. Editing only the tags in Table 5 would also result in the elimination of more than half of the errors in the sample tagged by TOSCA-ICLE. 
Once we had established which tags should be submitted to selective manual editing by considering each tag's contribution to the overall error rate, we were faced with the question of developing procedures for selective editing based on the data contained in the 5000-word sample. One very important factor that we had to take into account was the availability of people who could do manual tag editing. Our team consists of two people who have a sound knowledge of parts of speech and who are familiar with the full tagsets of the taggers we used (henceforth referred to as full editors), and four people who have only a rudimentary knowledge of parts of speech and no knowledge of the tagsets (henceforth referred to as mini editors). Selective tag-editing meant that we would only need to provide selective training which is time-effective. In devising a work plan, we had to decide which tags could be successfully dealt with by mini editors, and which tags should be left to the full editors. This entailed a closer look at tagging results for each tagger. 
6. Selective tag editing: procedures for TOSCA-ICLE 
The aim was to provide each mini editor with tag-specific training. For instance a mini editor could be informed about the ADJ(ge,pos) class, which is the tag that made the greatest contribution to the tagging errors made by TOSCA-ICLE (see Table 5). The editor would also be informed that the ADJ(ge,pos) tag is often assigned to words that should have been tagged N(sing). Table 6 shows that in our data more than one fifth of the ADJ(ge,pos) tags, should have been N(sing) tags. The mini editor would be instructed about the identification of ADJ(ge,pos) and N(sing). This would also entail general instruction about the attributes of nouns and adjectives. 
Table 6 Assignment of the ADJ(ge,pos) tag by TOSCA-ICLE 
Correct Number of times ADJ(ge,pos) was assigned ADJ(ge,pos) 198 N(sing) 60 ADV(ge,pos) 7 PREP(ge) 4 NADJ(ge,pos) 1 PRON(quant) 1 PRON(pers,plu) 1 PRON(pers,sing) 1 VB(lex,*,*) 2 
The N(sing) tag, in turn, contributes to 5.5% of the errors (see Table 5). However, most of the N(sing) tag errors were made on lexical verbs (see Table 7). One could instruct the mini-editor working with N(sing) and ADJ(ge,pos) tags to check all the N(sing) tags and simply flag instances where the tag was not assigned correctly, except of course where the confusion is clearly with ADJ(ge,pos), since this mini editor already knows the identification criteria for ADJ(ge,pos) and can therefore correct the tag. 
838
Table 7 Assignment of the N(sing) tag by TOSCA-ICLE 
Correct Number of times N(sing) was assigned N(sing) 605 ADJ(ge,pos) 6 ADJ(ge,pos,ingp) 1 ADV(wh) 1 VB(aux,semip,pres) 1 VB(lex,ditr,pres) 2 VB(lex,intr,infin) 1 VB(lex,intr,ingp) 4 VB(lex,intr,pres) 3 VB(lex,montr,infin) 1 VB(lex,montr,ingp) 4 VB(lex,montr,pres) 7 
The same mini-editor could also edit the N(plu) tag, which contributed to 5.2% of the tagging errors as s/he would be familiar with the properties of nouns. Table 8 shows that N(plu) was mostly assigned incorrectly to N(sing). In adopting this procedure, one mini-editor can correct a substantial portion of the tag errors while only having to be trained on three tags. This should hopefully ensure fair accuracy in his/her work, since there is little room for mistakes to be made. At the same time, potentially difficult cases are flagged. The full editors can retrieve these cases much more easily, as they do not need to work through all instances of these three tags, and make the necessary corrections. 
Table 8 Assignment of the N(plu) tag by TOSCA-ICLE 
Correct Number of times N(plu) was assigned N(plu) 369 N(sing) 27 VB(lex,montr,pres) 6 
Errors pertaining to lexical verbs contributed significantly to the overall error rate (see Table 9). Therefore it would be viable to train one mini editor on the category of lexical verbs. The incorrect assignment of lexical verbs can sometimes be ascribed to confusion with another main word class, such as nouns, but most of the VB(lex,…) errors are due to confusion pertaining to transitivity (see table 10) and confusion pertaining to the form of the verb (see Table 11). 
Table 9 Contribution to error rate by VB(lex,…) tags 
TOSCA-ICLE lexical verb tag Contribution to error rate VB(lex,montr,infin) 8.10% VB(lex,montr,imp) 3.52% VB(lex,intr,infin) 3.37% VB(lex,montr,edp) 2.91% VB(lex,montr,pres) 7.99% VB(lex,cop,pres) 1.68% VB(lex,dirt,infin) 1.53% VB(lex,intr,imp) 1.53% VB(lex,intr,subjun) 1.53% VB(lex,(intr,pres) 1.38% VB(lex,montr,past) 1.07% 
Table 10 Confusion matrix for transitivity features of the lexical verb 
Correct « Number of times tag was assigned VB(lex, 
cxtr,…) VB(lex, 
dimontr,…) VB(lex, 
ditr,…) VB(lex, 
intr,…) VB(lex, 
montr,…) VB(lex,cxtr,…) 10 0 0 0 2 VB(lex,dimontr,…) 0 0 1 0 0 VB(lex,ditr,…) 0 0 24 1 11 VB(lex, intr, …) 1 0 0 166 14 VB(lex,montr,…) 3 0 8 7 370 
839
A closer look at the data in table 10 indicates that the two more frequent categories, monotransitive and intransitive verb, are the ones causing most of the problems although their precision is more than 90% in both cases, while the precision of the tag for ditransitives is less than 70%. The confusion caused by the ditransitive tag is mostly with the monotransitive tag. If one starts the mini editor on the tag for ditransitives, and also train him/her on monotransitives, then proceed to editing for monostransitives, which should include training for intransitives, before finally dealing with the intransitives themselves, a fair result should be obtained. 
Table 11 Confusion matrix for forms of the lexical verb 
Correct « Number of times tag was assigned edp imp infin ingp past pres subjun edp 84 0 0 0 0 0 0 imp 0 3 0 0 0 0 0 infin 0 12 263 0 0 3 0 ingp 0 0 0 79 0 0 0 past 3 0 0 0 31 0 1 pres 0 14 41 0 0 217 2 subjun 0 0 0 0 0 0 0 
We believe that the formal features of the verb are fairly easy to understand for our mini editors, and therefore propose to provide training on these when the first training on the transitivity features is given. No separate editing for the formal features will be done, rather, these should be inspected and corrected at the same time as the transitivity features. What should be clear from table 11 is that the tag for imperatives is assigned with a precision of about 10% and the tag for infinitives with a precision of about 87%. Very often, the tokens should rather have been tagged as present tense forms of the verb, although it also happens frequently that the tag imperative is assigned to infinitives. One should therefore concentrate on disambiguation criteria for these three tags in particular during training of the mini editor. 
As is clear from table 5, the tag responsible for the third most errors is the tag for general adverbs. What makes this tag more difficult to handle, is that there is no single category with which it is often confused. All the possible correct tags of tokens tagged as ADV(ge,pos) are presented in table 12. 
Table 12 Assignment of the ADV(ge,pos) tag in TOSCA-ICLE 
ADV(ge,pos) ADV(ge,pos) 161 ADJ(ge,pos) 2 ADV(connec) 4 ADV(phras) 3 ART(def) 1 ART(indef) 1 CONJUNC(subord) 3 EXTHERE 2 N(sing) 4 PREP(ge) 13 PRON(dem,sing) 2 PRON(quant) 3 PRON(univ) 1 VB(lex,montr,pres) 1 
The only tag that is confused with ADV(ge,pos) in a substantial number of cases is the tag PREP(ge), general prepositions. Since there is a limited number of lexical items, such as ‘out’ and ‘behind’ that are potentially ambiguous between PREP(ge) and ADV(ge,pos), it is feasible to train mini editors to distinguish between these two. We believe that it is similarly possible to train mini editors to resolve confusion between ADV(ge,pos) and EXTHERE, the tag for existential ‘there', since it is limited to a single lexical item. For the remainder, we propose to focus the training of the mini editor on good identification criteria for the tag ADV(ge,pos) itself, and instruct him/her to flag all cases that are not clearly correct or clear confusions with PREP(ge) or EXTHERE. If one happens to employ a mini editor who has already worked on another tag, and therefore knows how to identify that, he/she can be asked to also resolve confusions with that tag, e.g. the mini editor who has already handled N(sing). 
840
The fifth biggest contributor to tag errors is the tag for subordinating conjunctions CONJUNC(subord). As is clear from table 13, there is no single tag that is consistently confused with this tag, similar to the situation with ADV(ge,pos). 
Table 13 Assignment of the CONJUNC(subord) tag in TOSCA-ICLE 
CONJUNC(subord) CONJUNC(subord) 153 ADV(connec) 1 ADV(ge,pos) 2 ADV(wh) 5 PREP(ge) 5 PRON(dem,sing) 10 PRON(inter) 1 PRON(rel) 6 
However, there are a couple of lexical items that are subject to some confusion. It might therefore be worthwhile to rather concentrate on these lexical items. These are the tokens ‘so’ and ’that'. Let us look at confusion tables for individual tokens to come to an understanding of the type of confusion to see how we can use this information to improve the precision of the tag ADV(ge,pos). 
Table 14 Confusion table for the token ‘so’ 
Correct tag Proposed tag ADV(connec)CONJUNC (subord) CONJUNC (subord):1/2 ADV(ge,pos) PTCL(to):1/3ADV(connec) 3000 0CONJUNC(subord) 0503 0CONJUNC(subord):1/2 00110 0ADV(ge,pos) 0205 0PTCL(to):1/3 0000 0 
The only confusion that is evident from this confusion table is the confusion between the tags CONJUNC(subord) and ADV(ge,pos). If one reconsiders tables 12 and 13, it is clear that the confusion involving these two tags is related to the token ‘so’ in all cases. It would therefore be worthwhile to instruct a mini editor to deal with the tagging of the token ‘so’ before turning to either CONJUNC(subord) or ADV(ge,pos). 
Table 15 Confusion table for the token ‘that’ 
Correct tag Proposed tag CONJUNC (subord) CONJUNC (subord):2/2 PRON(dem,sing) PRON(rel) CONJUNC(subord) 3900 0CONJUNC(subord):2/2 0110 0PRON(dem,sing) 1007 2PRON(rel) 603 16 
From this table, it is clear that all instances involving confusion between the tags CONJUNC(subord) and PRON(dem,sing) in table 13 involve the token ‘that'. Additionally, this token is also responsible for all confusions with the tag PRON(rel) in table 13. Thus, instructing a mini editor to inspect and correct all instances of the token ‘that’ will contribute significantly to the overall precision of the tag CONJUNC(subord) and a number of other tags too. Correcting the tags for the tokens ‘so’ and ‘that’ will lead to the correction of 60% of tag errors involving the tag CONJUNC(subord), involving much less trouble than dealing with all tokens tagged as CONJUNC(subord). 
As a last example, we turn our attention to the seventh tag on the list in table 5, the tag for general prepositions, PREP(ge). The results in table 16 indicate that there is one tag responsible for almost half of the errors involving this tag, which is ADV(connec). 
841
Table 16 Assignment of the PREP(ge) tag in TOSCA-ICLE 
Correct tag Number of times tagged PREP(ge) PREP(ge) 104 ADV(connec) 14 ADV(ge,pos) 2 ADV(phras) 2 CONJUNC(coord) 1 CONJUNC(subord) 4 N(sing) 1 PREP(phras) 1 PRTCL(to) 2 VB(lex,montr,infin) 1 VB(lex,montr,pres) 3 
It would therefore be possible to instruct a mini editor to examine all occurrences of the tag PREP(ge) and inspect them for confusion with ADV(connec). However, to understand the meaning of the tag ADV(connec), it is also necessary to understand the meaning of the related tags for subordinating and coordinating conjunctions. However, close inspection of the confusion table for the token ‘like’ indicates that this token is responsible for 13 of the 14 errors involving the confusion between PREP(ge) and ADV(connec). It therefore seems far more efficient to instruct a mini editor to deal with the token ‘like'. It has the further gain that all instances of confusion between PREP(ge) and PREP(phras), as well as VB(lex,montr,pres) also involve ‘like'. Thus 17 of the 31 errors involving the tag PREP(ge) can be corrected by only inspecting and correcting the errors involving the token ‘like'. 
The preceding discussion in section 6 has hopefully demonstrated how significant improvement to overall tagger accuracy can be achieved by training mini editors on a selected number of tags and in some cases on the possible tags for a selected number of tokens. More than half of all tag errors (50,9%) can be removed following the procedure above, while examining far less than half of all the tokens in the corpus. 
Once this has been done, the full editors can work through the corpus on the basis of a different list, the one presented in table 2. Working through the tags in the order of least precision means that more substantial improvement in tag accuracy is effected per number of tags inspected. For instances, a first step could be to inspect and correct all tags with an accuracy of less than 80%. These tags are listed in table 17. 
842
Table 17 Tags not yet corrected with precision of less than 80% 
POS Tag Frequency Correct Precision PROFM(so,clause) 1 0 0.0000% PRON(cleft) 1 0 0.0000% VB(aux,modal,infin) 2 0 0.0000% VB(aux,modal,pres,encl) 3 0 0.0000% VB(aux,pass,past,neg) 1 0 0.0000% VB(aux,semi,pres) 5 0 0.0000% VB(lex,cxtr,past) 1 0 0.0000% ADJ(ge,pos,ingp) 10 1 10.0000% VB(aux,perf,pres) 21 4 19.0476% ADV(ge,sup) 4 1 25.0000% PRON(antit) 3 1 33.3333% VB(aux,perf,infin) 9 3 33.3333% VB(aux,semi,infin) 3 1 33.3333% VB(lex,cxtr,edp) 3 1 33.3333% VB(lex,cxtr,infin) 10 4 40.0000% VB(lex,montr,past) 14 7 50.0000% ADV(ge,comp) 11 6 54.5455% PRON(inter) 33 18 54.5455% VB(aux,pass,pres) 29 19 65.5172% NUM(ord,sing) 3 2 66.6667% ADJ(ge,pos,edp) 19 13 68.4211% VB(aux,do,pres) 28 21 75.0000% VB(lex,cop,ingp) 8 6 75.0000% VB(aux,pass,past) 9 7 77.7778% 
If a full editor were to inspect and correct the 231 tokens in table 17, he/she will encounter 115 errors. Thus a further 17,6% of the errors can be removed if a full editor is tasked with correcting the tags in table 17. Combining the two procedures leads to the removal of 68,5% of all tag errors, which is, we believe, a very good yield for relatively little input. If this remains valid for the entire corpus, we can achieve a tagging accuracy of 96,2%. Additioally, we will have an accurate record of which tags have been corrected and which haven't. For most of our purposes, this is sufficient. If a linguist or applied linguist were interested in a feature whose tags have not been corrected, assuming that that person will understand the grammar of that particular feature thoroughly, we can simply ask him/her to correct the relevant tags before commencing the study, as part of the licensing conditions for using our corpus of learner English. 
9. Conclusions 
This paper has presented results on the evaluation of three taggers, Brill, CLAWS7 and TOSCA-ICLE, using a non-native variety of English as testing data. Overall, CLAWS7 performs substantially better than the other two. 
Since our concern is with the accuracy of the TOSCA-ICLE tagger in particular, we have proposed a method for selective tag error editing to improve the overall accuracy of the tags in the tagged corpus. This procedure involves identifying the tags that contribute the biggest proportions to the total number of errors, together with a clear indication of the tags with which a particular tag is confused. On this basis, mini editors are trained to deal with one tag or a set of related tags at a time. Following this, full editors are employed to correct all remaining tags with a precision of less than 80%. In this way, which 
843
we believe is time efficient, we manage to remove more than two thirds of all tag errors, without having to manually edit two thirds of the corpus. 
10. References 
Brill E 1999 Corpus-based rules. In Van Halteren H (ed) Syntactic wordclass tagging. Dordrecht, Kluwer, pp 247-262. 
De Haan P 2000 Tagging non-native English with the TOSCA-ICLE tagger. In Mair C, Hundt M (eds) Corpus Linguistics and Linguistic Theory. Amsterdam, Rodopi, pp 69-79. 
Garside R, Smith N 1997 A hybrid- grammatical tagger: CLAWS4. In Garside R, Leech G, McEnery A (eds) Corpus Annotation: Linguistic Information from Computer Text Corpora. Essex, Addison Wesley Longman, pp 102-121. 
Van Halteren H 1999a Performance of taggers. In Van Halteren H (ed) Syntactic wordclass tagging. Dordrecht, Kluwer, pp 81-94. 
Van Halteren H 1999b Selection and operation of taggers. In Van Halteren H. (ed) Syntactic wordclass tagging. Dordrecht, Kluwer, pp 95-104. 
844

Shallow parsing of Hungarian business news 
Tamás Váradi 
Research Institute for Linguistics 
Hungarian Academy of Sciences 
varadi@nytud.hu 

Abstract 
The present paper reports on an attempt to annotate noun phrases in Hungarian using cascaded regular grammars. Hungarian presents several difficulties to shallow parsing such as discourse ori-ented constituent order as well as left-branching recursive possessive and participle structure inside noun phrases. The approach uses cascaded regular grammars and was developed with the CLaRK system. The NP grammar was tested on a morphologically tagged and disambiguated corpus of 928 sentences representing a sample of highly sophisticated written style of journalism. The results are encouraging even on this challenging text type. 
1 Introduction 
The present paper reports on ongoing work to develop a shallow parser for Hungarian. So far, there is no syntactic parser developed for Hungarian. There are two systems of morphosyntactic taggers developed at Szeged University (Gyimothy et al. 2000) and the Hungarian Academy of Sciences (Dienes Oravecz 2002) both using HUMOR, the morphological analyzer developed by Morphologic Kft. (Prószéky – Tihanyi 1996). These three centres are currently partners in two research projects aiming to develop an information extraction system of short business news and to build a treebank of Hungarian, which give the background to the work reported here. 
The paper is structured as follows. First, a brief overview of the peculiarities of Hungarian NPs is given, focusing on the practical problems they present for parsers. This will be followed by the description of the data (section 3) and the basic concept and the annotation used in the CLaRK system (section 4). Section 5 discusses the design principles of the NP grammar, which is presented in some detail in section 6. The pa-per ends with a discussion of the results and suggestions for future work. 
2 Problems of Hungarian Noun Phrase identification1 
Hungarian is customarily defined as a free word order language. Its basic sentence structure is designed on principles of topic-comment structure in the first place, which obey subtle rules of discourse perspective. Hungarian being a pro-drop language, the subject is often missing and prototypical sentences open with a topic instead. The topic position is optionally followed by the focus in the slot immediately preceding the verb. After the finite verb one finds an array of neutral elements in mostly free order (É. Kiss 1994). The automatic identification of the topic-comment structure is impossible to carry out on written language alone as the phenomenon is inherently intertwined with stress and other prosodic phenomena. 
Fortunately, word order within Noun Phrases follows strict rules. NPs are introduced by determiners, which may only be preceded by demonstratives or dative possessors. Determiners can be followed by nu- 
1 This section incorporates contributions to an earlier version of this article by Tibor Szécsényi, which is gratefully acknowledged. 
845
merals, which are followed by adjectives modifying the phrase-final noun head (Kenesei–Vago–Fenyvesi 1998) . 
Determiners 
Determiners cannot be relied on as anchor points to indicate the boundaries of noun phrases because they can be omitted, particularly when the NP is generic and functions as a kind of verb premodifier. Thus along with [1a] one also finds [1b]. 
[1a] [Péter] olvas [egy könyv-et]. 
Peter-NOM read a book-ACC 
‘Peter is reading a book.’ 
[1b] [Péter] [könyvet] olvas. 
Peter-NOM book-ACC read. 
‘Peter is reading a book.’ 
Participle structures 
In Hungarian there are two participles in active use, which are similar to the present and past participles in English. They are peculiar, however, in that they act as premodifiers and they can freely take any number of their own arguments in front of the NP head. In effect, this means that a participle premodifier can bring in the whole array of verb arguments and, even adjuncts, that would otherwise occur in a full-blown clause. To determine the leftmost boundary of such phrases is often a vexing problem especially because the de-terminer belonging to the head is often deleted, hence it cannot be used as a safe indicator of phrase bound-ary. The situation can be compounded by the fact that NPs containing participle premodifiers can be embedded recursively in each other. Such constructions are quite typical of careful, written Hungarian. As an example which is far from unusual consider: 
[2] [[[az ország 70 százalékát] [ellenõrzése] alatt tartó Unita] ellen harcoló [angolai kormány]] 
the country-NOM 70 percent-POSS-ACC control-POSS-NOM under holding Unita against fight-PrPART Angolan government 
‘The Angolan government fighting against Unita, which is holding 70 percent of the country under control’ 
Note how the corresponding English structure contains a mirror image, as it were, and involves a string of post-modifier structures. 
Missing noun heads 
It is not only the determiner that can be omitted from the NP, in certain cases the noun itself may be absent. If the noun head is identifiable from the context it can be elided. In cases like this the case marking suffix appears on the last constituent of the remaining NP: 
[3] [Péter] [a sárga könyv-et] olvas-sa, [Mari] pedig [a piros-at]. 
Peter-NOM the yellow(-NOM) book-ACC read-3SG Mary-NOM and the red-ACC 
Peter read the yellow book and Mary read the red one. 
This phenomenon means that practically everything that can serve as premodifier can also function as NP head. The only exception may be the articles. Unfortunately, the definite article is homophonous with the demonstrative pronoun, the indefinite article with the ordinal numeral meaning ‘one’ both of which can easily take the role of an inflected NP head. 
846
3 Description of the data 
The input to the grammar is text that is morphosyntactically annotated using a scaled-down version of the annotation scheme developed for the Hungarian National Corpus (Váradi 2002). For each token (tagged as <w>) its lemma, morphosyntactic description (msd) and a simplified corpus tag (ctag) is given as attributes of <w>. 
The quality of parser output depends to a great deal on the granularity of the morphosyntactic annotation and the precision of the disambiguation. As reported in Dienes and Oravecz 2002, the HNC tagging system basically uses the tagset of the HUMOR tool after applying some post-processing (such as stemming and some streamlining of the codes) on its output. The rate of precision of the disambiguation is reported to reach 98 percent. 
The texts were taken from a quality weekly economics journal, Heti Világgazdaság, which is the Hun-garian equivalent of the British magazine The Economist both as regards its prestige and its elaborate lan-guage style. The choice of this source was required by the information extraction project that focuses on short business news but at the same time it serves as the toughest possible test for the robustness of the NP grammar. This aspect of the data, though not easily amenable to quantitative measures, should be borne in mind when evaluating the precision/recall figures. 
At first, the regular grammar operates on a sequence of msd tags of words within the scope of a sentence. Figure 1 shows a sample of the annotated input data. The effect of a rule is typically to add an XML anno-tation over the chunk of data that it covered. The next set of rules operated on the output of the previous one, each adding further and further layers of XML annotation in the process. 
Figure 1 A sample of the annotated input data in XML </p> 
Figure 1 A sample of the annotated corpus 
4 The development environment 
The grammar was developed with the help of the CLaRK corpus development system, which is a graphical environment using a full blown XML processor and a finite state engine. The tool offers a whole number of useful facilities for grammar development, described in more detail in Simov 2001, 2002 and elsewhere. 
Here, we can only highlight some of the most important features of the system. Each rule of the gram-mar is defined in terms of a regular expression defining the target pattern together with left- and right-context and the XML markup that will be applied to the expression in case of a hit. For each rule, an Xpath expression is used to define the scope of the rule (using the apply to windows). and what part of an XML node will be fed to the regular expression processor (the so called element value). For example, most of the time the rules were applied to the msd attribute of the words (or, indeed, at a later stage the msd of con-stituents). The element value therefore was set to attribute::msd. As a unique extension, the system offers the possibility to assign a (user definable) tokenizer to each rule separately, making it possible, for example, to refer to capitalized words or hyphenated words, abbreviations etc. The functionality of XML processor is further enhanced by the use of a facility (called constraints) to manipulate the data in ways that are not easily done otherwise (e.g. automatically copying data, checking the XML tree in terms of the con-tent of their nodes, filling in attributes interactively from a set of dynamically defined alternatives etc. We used the constraints facility to percolate head features upwards the syntactic/XML tree. 

<?xml version="1.0" encoding="windows-1252"?> 
<!-- This Document is created with the Clark System! http://www.bultree-bank.org --> 
<text> 
<div> 
<div> 
<p> 
<s> 
<w lemma="új" msd="A.FOK.NOM" ctag="AS_Ac">Újabb</w> 
<w lemma="cég" msd="N.NOM" ctag="NS3NN">cég</w> 
<w lemma="kerül" msd="V.Me3" ctag="VS3PI">került</w> 
<w lemma="csõd" msd="N.ILL" ctag="NS3NX">csõdbe</w> 
</s> 
847
5 Design principles of the NP grammar 
The difficulties that Hungarian sentence structure presents to parsing are somewhat eased by the extremely rich morphological system, which encodes a lot of syntactic information that serves as useful clues for es-tablishing constituent structure at the sentence level. Many constituents serving as adjuncts to clauses are formed by (extensions of) inflected nouns, which explains the ubiquity of NPs in Hungarian sentences. 
As the internal structure of Hungarian NPs is left-branching, the NP head was used as a anchor point of the regular expression for the base NP. In defining the leftmost boundaries, the principle of longest match was used even in the compilation of the noun extensions. The base NP was allowed to contain a premodi-fier N provided it was in the nominative case. This constraint was imposed on other premodifiers capable of taking case (adjectives and pronouns). Such a rule is a slight oversimplification of the linguistic facts be-cause demonstrative pronouns can, in fact, occur inside NPs in non-nominative case provided they agree with the head. However, case agreement was not implemented in the current version of the grammar. 
The possibility for Noun heads to be omitted and their role to be filled by the premodifier standnig clos-est to them, calls for some heuristics in order to prevent overgeneralization. Accordingly, we adopted a 'depth first' strategy in the definition of NPs. First base NPs with noun heads were assembled and they were extended as far as it was possible. In the process, all premodifiers were associated with heads, thus making them unavailable as possible heads. Next, the premodifiers left out of NPs by earlier rules are identified as heads of noun-less NPs. Even here, it was necessary to introduce two stages in the assignment of heads, first allowing all premodifiers except participles to be heads (rule NP2) and participles are assigned head status only after testing if the new NPs formed with rule NP2 can form larger participle-NPs. 
6 The NP grammar 
The NP grammar is implemented as a series of cascaded regular grammars (Abney ??), twelve in its cur-rent version, applied cyclically fourteen times altogether. The various levels form two main stages: the level of base NP, which is built out of DetP and (simple and coordinated) AdJP chunks and includes a noun or a NamedEntity. 
AdJP(<DIG>|<NUM>|<"Num#">|<"Adv#">)*,<"A.NOM"|"FF.A.NOM">+NPbase<DP>?,(<"FF.Num.NOM">|<"Num.NOM">)?,<DIG>*,<NUM>*,(<DP>|<AdjP>)*,(<"N.#">|<NE>)AdjP_coord(<AdjP>,(<c>|<"Con#">)+)+,<AdjP>DetP(<"Num.NOM">|<"Pro.NOM">|<"Det">)+NamedEnt.(<$Word|$WORD>)+,(<"-">?,<"és">,<$Word|$WORD>+)*,<".">? 
Figure 2. Overview of the grammars producing basic NPs 
At present the NamedEntity rule only covers a very elementary approximations of proper names using only the token type $Word or $WORD standing for non-sentence-initial words that begin with an upper- 
848
case letter or words in uppercase. This is admittedly just a placeholder for an in-depth treatment of named entities, which we aim to carry out in the future. 
The process of producing basic NPs and the regular expressions used in the rules are illustrated in Figure 2. As an introduction to the notation, concatenation of tokens is indicated with commas, “<AdjP>” refers to a node of the label AdjP, <”Adv#”> is a regular expression evaluated on whatever the Element Value setting passes to the Regular Expression evaluator, typically the msd attribute of the node in our case. “#” is the wild card character in CLaRK standing for 0 or more accurrence of any character. Concatenation of the tokens in the patterns is signalled with a comma. 
The next stage consisted in building maximal extension NPs by combining possessive NPs, coordinated NPs and NPs linked with participles. Possessive and participle-NPs can be chained together, hence they are applied again whenever a larger NP is formed. For reasons detailed in 2) the structure of participle-NPs are obviously the most complex. 
After the application of each NP extension rule, the msd attribute of the head was copied onto the node. This was done with the help of the Constraints facility within CLaRK. Unfortunately, while grammars can be grouped together and run at once, constraints cannot be similarly automated, which means the cascade has to be run manually. 
NP_poss(<"N.#NOM#">|<"N.#DAT#">),<"N.PS#">NP_poss2<"N.PS#">,<"N.PS#">NP_coord(<NP>,<"és">|<”vagy&#161;>)+,<NP>NP_particple(<DP>|<"Det">)?,(<"Adv">|<"Con">|(<NP>+,<"NU">?)|(<AdjP>|<"Ad#">),((<c>|<"Con">)+,(<"Ad#">|<AdjP>))?|(<"Pre.V.INF">|<"V.INF">)|,((<c>|<"Con">)+,<"#V.INF">)?)+,(<"MIB.NOM">|<"MIF.NOM">),<NP>NP2<DP>?,(<"FF.Num.NOM">|<"Num.NOM">)?,<DIG>*,(<DP>|<AdjP>)*,<"N.NOM">*,(<AdjP>|<"A.#">|<"FF.A.#">|<"Num#">|<"Abb#">|<"Pro#">)NP_particple(<DP>|<"Det">)?,(<"Adv">|<"Con">|(<NP>+,<"NU">?)|(<AdjP>|<"Ad#">),((<c>|<"Con">)+,(<"Ad#">|<AdjP>))?|(<"Pre.V.INF">|<"V.INF">)|,((<c>|<"Con">)+,<"#V.INF">)?)+,(<"MIB.NOM">|<"MIF.NOM">),<NP>NP3<DP>?,(<"FF.Num.NOM">|<"Num.NOM">)?,<DIG>*,(<DP>|<AdjP>)*,<"N.NOM">*,(<AdjP>|<"A#">|<"FF.A#">|<"Num#">|<"Abb#">|<"Pro#">)*,<"MI#">NP_coord(<NP>,<"és">|<”vagy&#161;>)+,<NP> 
Figure 3. Overview of the grammars producing embedded NPs 
849
7 Results 
The grammar was developed on a file of 928 sentences containing 23991 tokens. The NP grammar recog-nized a total of 10500 NPs, 6807 of which were maximal extensions. At the current stage of our work, we are mostly concerned with the NPs at the sentence level. Therefore the parser will be evaluated in terms of its performance in handling NPs of maximal extension, ignoring their internal structure. 
In order to better evaluate the performance of the system, a dual measurement was applied. One reports precision and recall values for the standard structural bracketing, the other focusses on the number of to-kens that are analyzed as being under a top level NP chunk. The latter measure is motivated by the fact that the present system is designed as a prepocessor for manual annotation and even if a top level NP is not identified correctly, detected parts of it still serve as valuable information facilitating the annotators' work. 
A 100 sentence text chunk containing 2537 tokens served as the test set. A gold standard was produced from this by manual annotation and proof-reading, resulting in 488 top level NPs. Then the test set was processed by the system and evaluation measures were calculated in the standard ways: 
(i) structural measures: 
- precision: number of common NPs in gold standard and test set / number of NPs in the test set 
- recall: number of common NPs in gold standard and test set / number of NPs in gold standard 
(ii) per-token measures: 
- precision: number of common tokens under (common?) top level NPs in gold standard and test set / number of tokens under top level NPs in test set 
- recall: number of common tokens under (common?) top level NPs in gold standard and test set / number of tokens under top level NPs in gold standard 
FB1 scores were calculated as usual: 
FB1=2*prec * recall/(prec + recall) 
The results for the structural measures are displayed in Table 1, the per-token measures are shown in Table 2. 
Number of NPs in gold standard: 488 Number of NPs in test set: 611 Number of correct NPs 323 NP precision: 52.87% NP recall: 66.17% Per-word FB1 score: 58.78% 
Table 1 Structural measures 
Number of tokens in NPs in gold standard: 1660 Number of tokens in NPs in test set: 1577 Number of tokens in NPs in common: 1511 Per-word precision: 95.81% Per-word recall: 91.02% Per-word FB1 score: 93.36% 
Table 2 Per-token measures 
850
8 Conclusions 
Perusing the outcome of the grammar, we could not help finding the results far more encouraging than the numerical indices may otherwise warrant. In order to do justice to the results, the figures dislayed in the tables should be evaluated with the following points kept in mind. 1) The data the grammar is tested on is, without exaggeration, a sample of the most sophisticated written journalism in Hungarian. Even discount-ing flourishes of style so pervasive in this particular journal, one should note the high number of figures, dates, appellations, parenthetic and appositive materials etc. Thus, the grammar can be realistically ex-pected to stand the test of the most robust applications with at least these figures. To put it in another way, far better results can reasonably be expected on on average run-of the mill text type. 2) Hungarian sen-tences abound in NPs, owing to the fact that a large number of adjuncts are expressed with inflected nouns. To illustrate, 69% of the total of 2200 words were inside top level NPs. Therefore, once the NPs are tack-led, the major part of the parsing of sentences is already done. 
There are, obviously, shortcomings in the grammars as well, some were known, some were highlighted during a closer inspection of the results2. As mentioned above, the current implementation of the grammar cannot stipulate agreement, which would be important in a few cases even within NPs despite the fact that in general Hungarian does not usually display modifier-head agreement in NPs. In some instances, letting adjectives assume the role of NP heads led to false results. 
Inevitably, in a handful of cases, the wrong analysis could be tracked down to the wrong msd value. An example would be the occasional fault in disambiguating the past participle/past tense ambiguities. We also found examples where the NP grammar rules served to spot spurious part of speech analyses. The word szörnyen monster-adessive ‘terribly’ is again one of the ubiquitous inflected nouns that function as an ad-verb. Recall that the NP1 rules allowed no nouns next to the head unless they were in the nominative. However, as a booster expression it can clearly occur inside NPs as in Szörnyen drága kabát ‘terribly ex-pensive coat'. The fact that our grammar rejects this expression as a single NP indicates that the word szörnyen is incorrectly tagged as a noun instead of an adverb. Such cases, and also multi word expressions of a similar type (i.e. spurious nouns while actually adverbs) should be systematically addressed along other named entities. This point leads to the conclusion that a proper syntactic analysis of Hungarian NPs should be done with close integration of some degree of semantic markup. 
References 
Abney S 1996 Partial Parsing via Finite-State Cascades In Proceedings of the ESSLLI'96 Robust Parsing Workshop, pp 1 – 8 
Kenesei I, Vago R M, Fenyvesi A 1998 Hungarian. Descriptive Grammars. London, Routledge. 
É. Kiss K 1994 Sentence structure and word order. In: Kiefer–É. Kiss (eds): The Syntactic Structure of Hungarian. San Diego, Academic Press. 1–90. 
Oravecz Cs, Dienes P 2002: Efficient stochastic part of speech tagging for Hungarian. In Proceedings of the Second International Conference on Language Resources and Evaluation, Las Palmas, pp 710—717 
Prószéky G, Tihanyi L 1996 "Humor -- a Morphological System for Corpus Analysis." Proceedings of the first TELRI Seminar in Tihany. Budapest, pp 149-58. 
Simov K 2001 CLaRK – an XML-based System for Corpora Development in Proceedings of the Corpus Linguistics 2001 Conference, Lancaster, pp 553-560. 
Simov K et al. 2002 CLaRK System: Construction of Treebanks in The First Workshop on Treebanks and Linguistics Theories Sopozol: LML CLPP Bulgarian Academy of Sciences 183-199. 
Szabolcsi A 1994 The noun phrase. . In: Kiefer–É. Kiss (eds): The Syntactic Structure of Hungarian. San Diego, Academic Press. 179-274. 
Váradi T 2002 The Hungarian National Corpus. In Proceedings of the Second International Conference on Language Resources and Evaluation, Las Palmas, pp 385-389. 
2 We are indebted to Kata Gábor, Erzsébet Dancsecs and Andrea Littomericzky for their help in checking the results. 
851

Collocational and colligational patterns in lexical sets: A corpus-based study 
Isabel Verdaguer and Anna Poch 
University of Barcelona 
1. Introduction 
The need for a linguistic approach which integrates syntax and semantics in the description of language has now been known for a long time, since the interface between syntax and semantics allows us a systematic and coherent description of the word meaning. The lexicon, from which all types of linguistic information can be projected, thus acquires a relevance which it did not have in the past. A thorough understanding of all the relevant features of lexical items and of their interconnections allow the linguist to study their shared properties and behaviour, establish general patterns and draw generalisations. On the basis of the interdependence between their syntactic properties and their meaning, lexical items can then be classified and grouped into lexical sets. 
Following the assumption that the meaning of verbs is interrelated with their syntactic behaviour and that verbs can be classified into lexical sets or classes according to some shared syntactic and semantic properties (Levin 1993), the aim of this study is to see whether verbs forming a lexical set also share similar collocates. In the context of a project1 dealing with significant collocations in scientific research papers, aimed at helping Spanish scientists to use the right word combinations when writing their papers, we explore what collocations are recurrently used with a selection of verbs which Levin (1993) classifies as See verbs and Sight verbs. The starting point of this study is thus the question whether verbs which share a similar meaning and similar syntactic behaviour also share similar collocates. 
As corpus linguistics studies have confirmed the need to take into account the patterns of word co-occurrence in language description, the importance of collocations and other units of meaning larger than the word has been increasingly acknowledged (Sinclair 1991, 1996, McEnery and Wilson 1996, Gledhill 2000, Stubbs 2002, Oakey 2002). Corpus studies have shown that native speakers do not make up entirely new combinations of words when they speak or write, but very often make use of recurrent chains of words, semi-automatic chunks of language, or multi-word units. Language is therefore a mixture of creativity and repetition. This frequent co-occurrence of words has to be taken very much into account in the description of a language and in lexicography, especially that aimed at non-native speakers of a language, since, although they are usually not difficult to decode, they may cause difficulties in language production. The correct use of words in their context and of the right collocations is essential to show native-like competence and fluency, and it is especially necessary in the writing of scientific papers, which requires a precise expression of one's ideas and of research results. It is important, thus, to know what are the most frequent and expected word combinations used in a specific register and this evidence obviously derives from the observation and analysis of corpora of real text. 
Our analysis has started with the selection of a set of verbs which occur in scientific articles. Since observation is a necessary step, crucial in scientific research –most data are first perceived through the sense of sight before being processed by the mind– we have opted for a range of verbs of visual perception: see, examine, investigate, note, observe and view, which have been classified by Levin (1993) into two different subsets of verbs of perception: see is included in the See verbs, whereas the rest belong to Sight verbs. As we will see, most verbs of physical perception have extended their meanings to refer to mental perception.2 Some of them, such as observe, are verbs mainly of physical perception; in others, such as investigate, mental activity is predominant. 
1 This project, reference BFF2001-2988, is financed by the Spanish Ministry of Science and Technology and FEDER 
2 Most of these verbs have their origins as verbs of physical activity and through metaphoric transfer, mainly triggered by abstract direct objects, have become verbs of mental perception as well. Sweetser (1990) has provided ample evidence that change operates in from concrete to abstract domains. 
852
These verbs may be highly polysemous in general language. As Levin's classification is based on the meaning of the verbs, as well as on their syntactic properties,3 they are obviously found in other groups as well. For example, see is also grouped with Characterize verbs, along with view; observe is also classified as a Conjecture Verb; examine, investigate and observe as Investigate verbs; note and observe as Say verbs. It can be noticed then that, in addition to sharing the classification of verbs of perception, most of these verbs also share other similar senses and syntactic behaviour which makes it possible to classify them in the same group. This fact also indicates that changes and extensions of meaning follow a similar direction. It is to be expected then, that similar collocates will also be found. 
In specialized language, polysemous verbs generally appear in more restricted senses than in general language. Our corpus exploration and analysis aims to find out the patterns of these verbs and their recurrent collocates in scientific texts to see whether generalisations can be reached. We are interested in finding out what verbs can appear with a non-human subject, as well as a personal subject, what type of objects –from a semantic point of view– recur with these verbs, and what generalisations can be drawn from them, or what type of adverbs collocate with these verbs, in addition to their syntactic behaviour. And this can be very useful information in English for Specific Purposes, which will not be found in dictionaries only concerned with specialized vocabulary. 
2. Methodology 
The following study is based on the analysis of all the occurrences of the verbs see, observe, note, view, examine and investigate in a corpus of scientific research articles on biochemistry,4 which we have compared with a sample of the British National Corpus. The software5 used to process our data has provided the frequency of occurrence and the concordances that we have studied to look for patterns of word co-occurrence. 
We have started with the verb see, which is a highly frequent verb and may be taken as representative of the verbs of perception.6 Then we have explored the verbs observe, view and note, which are verbs more closely related to look at, which does not occur in our corpus. Next, we have dealt with examine, which could be considered a hyponym of observe (“observe attentively”) and finally with investigate. The verbs of greater frequency of occurrence, see, observe, examine and investigate, have been treated separately, whereas the rest have been dealt with in the general discussion. For each verb, the different structures and forms of occurrence were considered to try to set general patterns and at the same time establish differentiations among them. 
See verbs “describe the actual perception of some entity” (Levin 1993:186). They take the experiencer or perceiver as subject and what is perceived as object; they can also take that-clauses, -ing clauses or infinitive clauses as complements;7 and they do not take the middle alternation. Examples of all these structures can be easily found in a sample of a corpus of general English, such as the British National Corpus. However, the analysis of a corpus of English scientific texts provides us with information about the patterns and word co-selection which are specific to this register. 
3. Case Studies 
3.1. See 
In general English, see is a very frequent and highly polysemous verb. Some of the meanings found in the British National Corpus, however, such as for example “visit” (We were coming to see you last night), do not occur at all in our corpus. In scientific English, the meaning of see as a verb of visual perception generally implies mental understanding as well. It could be said that there is a continuum and in most occurrences both elements are involved, and the preponderance of physical or mental perception depends on the type of stimulus which co-occurs with it, either a physical entity or a more abstract process or phenomenon. Compare, for example, no significant staining was seen in the rest of the cell, 
3 Levin gives special prominence to the expression of arguments and diathesis alternations. 
4 The corpus used in this study consists of 104 articles published in Biochemical Journal in the years 1998-1999. 
5 WordSmith Tools 
6 The behaviour of see has received a lot of attention and has been dealt with in various studies. 
7 Levin does not attempt to deal thoroughly with all the possible complements of See verbs. 
853
where physical perception is predominant, and A similar induction pattern was seen with… where mental perception is more important. Except for the occurrences of see in the imperative form and a very small number of other cases, most occurrences of see in which the stimulus or perceived object is a Noun Phrase are passive sentences. And it should be mentioned in this regard that although it is well known that scientific language generally uses the passive voice, whereas with other verbs such as examine or investigate the Subject we is often explicit, with see, this is very rare. 
Although the perceived object can certainly be expressed by a Noun Phrase whose head is a noun referring to a physical entity such as < particles, protons or chromosomes >, the most recurrent collocates are of a more abstract nature, and many of them could be referred to as a state of affairs. Examples are < effects, trends, presence, changes, levels, pattern, interaction, induction, inhibition … >, generally either pre- or post-modified to specify their reference (e.g. A similar pattern of S activation in Z cells is seen / one distinct shift was seen for this fragment). Past participles with a passive meaning postmodifying similar collocates are also frequent < the level of induction seen in... / chromosomes frequently seen / the increase seen in cells…>. 
An even more frequent pattern than that of passive structures, however, is the imperative form see (60%), which is basically used for references. Its typical collocates are fairly simple Noun Phrases such as: < table X, section Y, figure Z, appendix A…>, or adverbs of place: < below, above…>. Another frequently recurring pattern is a comment clause introduced by as < As seen in figure, table …/ As can be seen > as a way of introducing a statement. This is much more frequent than the alternative that-clause, of which only two examples have been found, both with the modal can (It can be seen that…). No examples of see followed by an –ing clause occur, and only one where passive see is followed by a to-infinitive clause (X was seen to…). 
See can also occur in a to-infinitive clause, as a predicative complement, or complement to predicative adjectives (it is possible / difficult to see) or to nouns (it is of interest to see), which mainly convey the speaker's attitude. In 80% of occurrences of see in infinitive clauses, the verb is followed by whether /if and its meaning then is “find out.” 
3.2. Observe 
The rest of the verbs, which can be classified as Sight verbs (Levin 1993:186-187), in general English also take a perceiver as the subject and what is perceived as object, and cannot take an infinitive clause as complement. Some of them (except observe or note) cannot take a finite that-cl, either.8 And in the same way as See verbs, they do not allow the middle alternation. 
Observe is the most frequent verb of this group and the one that in principle is most closely connected to physical perception. In general English, observe is also highly polysemous. Levin classifies it in four different groups. However, in scientific English, observe is mainly used in the sense “look at, watch (attentively).” What can be observed is either something concrete, which is directly perceived, such as: < proteins, peaks or cells > or, more frequently, something more abstract or a state of affairs, which in many cases needs some sort of mental processing: < results (similar results…); activity (basal activity, low activity, PLC-bt activity, LPP activity…); processes (phosphorylation, catalysis, dimerization, degeneration…); phenomena (fluorescence, this phenomenon…); interaction (such an interaction…); attachment (some loss of attachment…); effects; trends; characteristics; changes (reduction-sensitive change…). The Noun Phrases which collocate with observe may be again highly complex, with premodification and /or postmodification, and in a a remarkable number of occurrences they indicate differences, similarities or some kind of variation or change: < A faster migrating X species / An increase in X levels / An accumulatin of PG or Cl mass / Any changes in / Any translocation of / A progressive age-dependent increase >. 
Some sort of comparison is sometimes established: < A smaller degree of induction was also observed / Similar results are also observed / A similar reduction in / A relatively broad signal is also observed / Increased Ab secretion is also observed / High similarity is also observed >. Those comparative structures are especially used with the pattern that / those observed, where some degree of 
8 Again, Levin does attempt to deal with all the possible complements. 
854
comparison is involved: < Expression of X was comparable to that observed in /…was greater than that observed in keratinocyte / …lower than that observed / process comparable to that observed with…>. 
Another interesting pattern is that observed with the past participle. Whereas with all the other verbs, the past participle follows the noun, in the case of observe, the past participle can also precede it. And there is a difference in use. The past participle precedes a noun (which can be < activity / change / curve / discrepancy / kinetics / phenotypes / values / radius / signals / stimulation…>) when an explanation is provided, and it gives support to what is explained (the data support the proposal that the observed activation…), whereas it follows the noun when the writer simply wants to express that something has been observed < low but significant activities observed / basal activity observed / phosphatase/ luciferase activity observed >. 
Sentences in the active do occur, although there are not many (9,7%). Again the direct objects which are used with we observed / we have observed are not physical entities or substances but < interaction, propensity, increase, difference…> Or, more frequently, in that case, observe is complemented by a finite that-clause conveying a statement (we observed that…). Interestingly enough, another pattern which occurs with this verb is one indicating negation, which is conveyed by verbs such as fail followed by the to-infinitive (we failed to observe), adjectives such as unable (were unable to observe) or negative determiners and particles: < no significant fluorescent staining / no such characteristic staining / no fragmentation / no inhibition was observed >. 
As see, observe is used in comment clauses, as observed / as we have observed, generally followed by an adjunct of place (in many other cell types), or time (after treatment / previously). Finite that-clauses can occur but are not frequent. 
3.3. Examine 
Its meaning in scientific texts could be glossed as “to observe attentively in order to discover or find out something” and here again physical perception and mental understanding are involved. Whereas the personal forms of see rarely occur in scientific English, examine occurs quite frequently with the personal pronoun we (26% of the occurrences) and it also allows a Subject which is not human < Further investigations / those studies / several papers >. Again, the collocates which appear as direct objects of the active structure do not refer to physical substances or entities, but are more abstract and general, referring to processes, effects, abilities, role, mechanisms, effects, influences, phenomena, role, possibility, interactions …, whereas in the passive structure more concrete entities and substances appear as well < plates, membranes, granules, cells, nuclei, filters, tissues, specimen, grids …>. 
Examine may take subordinate clauses introduced by whether, and a remarkable proportion of these (40%) follow examine in an infinitive clause of purpose (to examine whether this cell would show…), which represent 67% of the infinitive clauses occurring with examine, the rest occurring with catenatives or as complements to adjectives or nouns which mostly convey the writer's attitude (it is instructive to examine / it is of interest to examine). Non-finite –ing clauses may also appear in similar structures (we were interested in examining). On the other hand, the only modal auxiliary which has been found is will, to indicate future time reference. The past participle does not occur often at least in the occurrences found in our corpus, and it is always following the noun (tissues examined). 
As examine may be taken as a step in scientific research, it co-occurs with the adverbs also (we also examined), next (we next examined) then (we then examined) or further (to examine further), indicating the sequence of events, which can also be conveyed by constructions such as we began by examining. It is also for this reason that in the passive sentences, examined can appear coordinated with another verb < membranes were fixed and examined / cells were harvested and examined / starch granules were isolated and examined >, indicating the different parts of the procedure followed. It is of interest to remark in this respect the presence of the formation re-examine (re-examine the hypothesis / the nature of). 
3.4. Investigate 
Whereas in general English, most occurrences of investigate are used in the sense “to conduct an enquiry into,” which would correspond to Levin's Investigate verbs, needless to say, in scientific texts investigate is used in the sense “look into from a scientific point of view.” The predominant form used 
855
here is the infinitive of purpose, which is again used with the same type of abstract collocates as heads of the Noun Phrase that we have found with the other verbs: < To (in order to) investigate the effects / the role / the importance / the ability / the function / the mechanism / the distribution / the interaction >. However, the most significant characteristic which distinguishes it from the other verbs is the fact that it repeatedly co-occurs with expressions indicating possibility < this possibility / the possible dependence / the possible presence / a possible functional role / this hypothesis >. Although examine can also collocate with possibility, the number of such collocations is not as high as with investigate. Again, the quest involved in the meaning of this verb is also conveyed by the type of clausal complementation that it takes: interrogative wh-clauses < how / whether / what / which >. And the especially active implication of the Subject is shown again by the use of the pronoun we ( we investigated / we have investigated). Passive sentences are of course also used, and we find here the same pattern that has been previously mentioned in relation to the other verbs: concrete physical objects appear as passive subjects rather than as active direct objects, whereas more abstract or general objects can appear both in the active and passive sentences. 
As in the case of examine, the adjuncts which are mostly used here are < further / then / next > (we then investigated), indicating the sequence of events taking place in the investigation, or of manner < systematically / extensively / directly / thoroughly >. The need for further work which is involved in the meaning of investigate, which has been already indicated by its frequent co-occurrence with the adverb further is implied in the structures < remains to be investigated / will need to be investigated / can be investigated > or by the formation un-investigated. Non-finite clauses with a past participle following the noun can be used but, again like with examine, they are not especially frequent in our corpus (2.3%). 
4. Discussion and conclusion 
Collocations, which were once thought to be an idiosyncratic characteristic of individual words, have now been seen to form general patterns, since the collocates of a node may form sets which are semantically related. This corpus study has revealed the collocational and colligational preferences of verbs belonging to the same lexical set and has shown that they also share general patterns of collocates. However, this study has also revealed that, within these general patterns, each these verbs also have their own preferences, which are closely interrelated with the semantic elements that conform their meaning. We will first comment on the patterns that these verbs share and afterwards we will discuss the particular features that distinguish them. 
We will begin with the kinds of perceived objects which occur with the verbs studied. It will come as no surprise that the most frequent structure which occurs in scientific prose is a passive sentence and the perceived object will consequently be a passive subject. The passive is predominatly used with all the verbs in this register, except with investigate, whose to-infinitives (of purpose) are the most frequent forms, and see when it is used for references. Except with the verb view, which has a physical object < cells > when it is used as a Sight verb, the head of the complex Noun Phrases that conform the passive subject are not predominantly nouns referring to physical substances or objects, as could be expected when dealing with verbs of perception, but more abstract entities, such as effects, results, processes or phenomena, which require some mental processing. This indicates that all the verbs sharing this feature also share the extension of meaning from physical to mental perception. Unlike the active structure, however, which is typically used with abstract direct objects, except in the use of see mentioned above, in the passive structure we can also find physical entities, such as cells, granules or images which can be directly perceived. The fact that in the active voice we have only found abstract entities or state of affairs9 can be explained by the explicit involvement of the perceiver and the cognitive process which is implied here, which is not necessary in the direct perception of a physical entity, and thus the perceiver can be left implicit. Within this general pattern, the idiosyncratic characteristics of some verbs have to be mentioned: observe, and also note, frequently co-occur with Noun Phrases indicating similarities, differences, increase, decrease or some variation or change. Investigate, on the other hand repeatedly collocates with Noun Phrases indicating possibility. 
The perceiver is in most cases left implicit, as corresponds to passive structures. However, here again there are verbs which stand out, since observe, examine and investigate have a fairly high percentage of occurrences in active sentences with the explicit Subject we. In the case of observe, we 
9 Except with imperative see. 
856
observed is frequently followed by a that subordinate clause, introducing a statement or observation (we observed that). On the other hand, examine allows a metonymic extension of the Subject, which can be < studies / papers / investigation…>. 
The imperative form occurs with two verbs see and note, but with some differences in use and in collocates. In both cases, they are used to draw attention to something, but whereas see collocates with nouns such as tables or figures, for example, note collocates with lexical items which refer to properties or processes (note the co-precipitation of / the distribution / the co-enrichment of) or is followed by a statement introduced by a that-clause. 
Observe and see, as well as note, may take a subordinate content clause introduced by that, conveying a fact or an idea. The same verbs may also take a comment clause introduced by as (much more frequently in the case of see) typically co-occurring in this structure with an adjunct. They usually collocate with adjuncts of place (as is seen in Figure X / as observed in many other cell type ), of manner (as can be seen by comparing…), of time (as observed / noted previously). Note in this last example, that the meaning of both verbs in this context is “make a comment or remark”, which is also the case when note is followed by the agent (As noted by X). Examine, investigate and see (in the sense “find out”), on the other hand, can take a wh- interrogative clause. 
Past participles, with passive meaning, usually occur following the head of the Noun Phrase, (that is to say they occur postpositively). And in this structure, the meaning of observe occurring with previously (species previously observed) is of physical perception. Observed also stands out in the corpus because, differently from the other verbs, whose past participle follows the noun, observed can precede it to indicate not simply that something has been observed, but to provide an explanation for something. 
We have found examples of to-infinitives as predicative complements (the aim/ purpose of this study was to investigate) or as complements to adjectives (it was of interest to investigate), nouns (it was difficult to see / it is possible to see/ it is important to note), or verbs (we would expect to observe) which mostly express some kind of modality or the speaker's attitude. The modal auxiliary can, which typically occurs with see and other verbs of perception, is also recurrently found in the corpus. Other linguistic expressions of possibility, illustrated by Noun Phrases such as this possibility are used with examine and especially with investigate. These two verbs are also used with an infinitive of purpose, again indicating the search for new discoveries involved in the meaning of the verb. 
Apart from the adjuncts, such as those of place, time or manner, which are generally used, we also find collocational preferences which are related to the specific semantic elements of the different verbs. Along with the adjuncts indicating the sequence of events occurring with examine or investigate, we have found coordinated verbs (fixed and examined) or occurrences of constructions expressions such as remains to be investigated. 
In conclusion, verbs which have similar syntactic and semantic properties have been shown to share similar collocational patterns, too. Although specific differences within general patterns may arise, the more properties verbs have in common, the more similar their collocational preferences are. 
References 
Gledhill C 2000 The discourse function of collocation in research article introductions. English for Spacific Purposes. 19 (2): 115-135. 
Levin B 1993 English verb classes and alternations. Chicago and London, The University of Chicago Press. 
McEnery A, Wilson A 1996 Corpus linguistics. Edinburgh, Edinburgh University Press. 
Oakey D 2002 Formulaic language in English academic writing: A corpus-based study of the formal and functional variation of a lexical phrase in different academic disciplines. In Reppen R, Fitzmaurice S, Biber D (eds), Using corpora to explore linguistic variation. Amsterdam / Philadelphia, John Benjamins Publishing Company. 
857
Sinclair J 1991 Corpus, Concordance, Collocation. Oxford, Oxford University Press. 
Sinclair J 1996 The search for units of meaning. Textus 9: 75-106. 
Sweetser E 1990 From etymology to pragmatics. Metaphorical and cultural aspects of semantic structure. Cambridge, Cambridge University Press. 
Stubbs M 2002 Words and Phrases. Oxford, Blackwell. 
858

Shedding light on SHED, CAST and THROW as nodes of extended lexical units 
Maria Verde 
School of English, University of Birmingham 
1. Introduction 
1.1 The nature of the lexical item 
Considering linguistic form and meaning as two facets of the same phenomenon has been the fundamental assumption behind corpus work on phraseology. This systematic interconnection was originally proposed by Firth and then adopted by Sinclair to become the basis of the theoretical framework he developed. Such a view of language has important implications with regard to analyzing and describing meaning, in that first ‘given certain formal parameters in the context of a word, it is possible to arrive at a reliable meaning by formalizing the evidence of language usage’ and second ‘a variation in the formal profile of a word or an expression will always lead us to a change of meaning’ (Tognini-Bonelli 2002:76). 
In the light of corpus evidence, Sinclair challenged in the light of corpus evidence the traditional association of the unit of meaning with the word and postulated a redefinition of it (1991, 1996, 1998), based on collocation, colligation, semantic preference and semantic prosody. This new kind of unit is called ‘extended', because although it is described by means of using a word (or phrase) as a starting point, it is ultimately defined by the observed interrelation of the node and its context. Tognini-Bonelli (2001) defines meaning as ‘function in context’ and stresses the fact that it is essential for the extended unit of meaning to have reached its pragmatic function in order to be considered as ‘functionally complete’ (id.:19), which in turn draws attention to semantics and pragmatics as forming a continuum. 
1.2 Delexical verb and noun combinations 
The structural tendency of English towards analytic constructions is a current productive pattern constituting a vital part of its lexico-grammar and deeply rooted in the historical development of the language (Strang, 1970, Dixon, 1991). In this paper, we take an interest in periphrastic verbal expressions consisting of a verb and a noun complement, in which the overall meaning of the combination rests primarily on the noun (functioning as direct object in the logical structure), while the meaning of the verb appears to be ‘reduced’ or ‘weakened’ (when compared to its basic or prototypical meaning). Different names have been used by scholars with reference to these phraseological units, such as ‘verbo-nominal phrases’ (Renský, 1964), ‘complex verbal structures’ (Nickel, 1978), ‘composite predicates’ (Cattell, 1984), ‘V-NP’ constructions’ (Stein, 1991) or ‘expanded predicates’ (Algeo, 1995). The most frequent of the verbs employed in these expressions and the most commonly accepted as such in the literature are ‘have', take', ‘give', ‘do', ‘make', but there appear to be several others less frequently occurring in this kind of construction, such as ‘put', ‘get', ‘set', ‘provide’ etc. They too have been given various names, including ‘purely or mainly connective verbs', ‘verbs of vague or weakened meaning’ (Poutsma, 1926), ‘copulas’ (Curme, 1935), ‘light or insignificant verbs’ (Jespersen, 1933), ‘function verbs’ (Nickel, 1968), ‘lexically empty verbs’ (Live, 1973), ‘general or semantically fuzzy verbs’ (Leech & Svartvik, 1975) and ‘empty verbs’ (Allerton, 1982). We will be referring to the verbs of this type as ‘delexical’ and the structures they enter ‘delexical structures', which is the name commonly employed by corpus linguists. 
Sinclair (1991:113) introduced the term progressive delexicalization to refer to ‘a broad general tendency of frequent words or of frequent senses of words, to have less of a clear, and independent meaning than the less frequent words or senses'. While delexicalization is a growing area of interest, there has not been much corpus research on delexical verbs as yet and their treatment in the existing literature is far from comprehensive. The aim of this paper is the examination of the behaviour of SHED, CAST and THROW1 as verbs with a delexical tendency in naturally-occurring data taken from the ukbooks subcorpus of the Bank of English. These particular verbs are chosen as they are illustrative of how the phenomenon of delexicalization works, while at the same time they are not as frequent and/or productive as other more 
1 Capital letters represent lemmas, italics are used for forms of a lemma as well as for examples taken from the concordances, and quotes are used for semantic labelling. 
859
‘prototypical’ delexical verbs. In effect, in this paper we take the view that far more verbs than the few mentioned occasionally -as well as inconsistently- in the relevant literature could be claimed to have a delexical function. Nevertheless, they are more ‘peripheral’ or of ‘minor status’ with respect to either the number of discrete units of meaning they form and/or their frequency and distribution patterns. 
It was decided that 500 hundred concordances retrieved at random from the corpus would suffice for identifying typical lexicogrammatical patterning. Once yielded the concordance lines were examined manually in order to discount uses not relevant to our study. Our criteria of inclusion are much less restrictive than those postulated by the majority of linguists who have discussed this type of structures in the past; namely, we do not consider factors such as the form of the noun (i.e. morphology) or the existence of a semantically corresponding simple (i.e. non-periphrastic) verb expression as relevant; our focus is primarily on the existence or not of a unitary verbal meaning. The examples retained as instances of delexical uses were then subjected to a thorough analysis in terms of grammatical features of both the noun and verb group2 as well as lexical profiles. They provide evidence that various categories of nouns forming a number of lexical sets co-occur with the delexical verbs in question. Indeterminacy of membership is occasionally observed, which we see as relating to the question of the overall function of the predicate frame. Based on the analysed data, claims about typicality of usage are made. 
2 Lexicogrammatical patterning and meaning identification 
2.1 SHED as node 
Out of the 500 concordances that were retrieved at random from the corpus, the verb was observed to have a delexical function in 176 instances, which gives a frequency of 35%. The verb group3 is finite 53% of the time, a rather high 14% is marked as negative, with regard to voice 86% of the verb phrases are active and with regard to aspect, non-progressive and non-perfective are predominant (85% and 86% respectively). A relatively high 70% of the finite groups are in singular and all but one express the indicative mood and are in the past tense 60% of the time; the commonest person by far is the 3rd (84% of the time). Finally, some sort of modal meaning is expressed in 19% of the data. 
The three distinct lexical units which were identified on the basis of SHED's collocational relation to its noun complement as well as semantic preference are the following (in bold characters): 
SHED+TEAR SHED+‘light’ 
LIGHT 
GLOW 
RADIANCE 
BRIGHTNESS 
RAY 
REFLECTION 
TWILIGHT 
SMILE 
SHADOW 
DARK 
DARKNESS SHED+BLOOD 
SWEAT 
Figure 14 
Establishing semantic preferences among the patterns of co-selection of SHED, by virtue of identifying and grouping together its noun collocates on the grounds of shared semantic components, was rather straightforward. Fifteen different nouns co-occurred with SHED, of which 11 are one-offs. Most significant statistically is the noun object TEAR, which is the unique noun collocate of a distinct semantic 
2 The grammatical categories used in the analysis are taken from M.A.K. Halliday's Functional Grammar (1994). A summary of the findings for all 3 verbs is given in the form of tables at the end of the paper. 
3 ‘Verb group’ is used here in the traditional grammatical sense and does not take into account the semantic compositeness of the verb and its noun collocates. 
4 It has to be made clear that this is not meant to be a representation of the full meaning of the unit, but merely a schematic one consisting solely of its lexical core. All noun collocates are given here in the form of the lemma, however, it is acknowledged that those occurring only in the singular or plural form would be more accurately represented in italics. 
860
unit accounting for 59% of the data. The second most common unit, with a frequency of 45%, is the one involving nouns whose meaning would justify assigning the label ‘light’ to their set: light/-s, glow, brightness, radiance, ray, reflection, twilight, shadow, dark and darkness. It should be observed that LIGHT is the core member of the set, whereas the rest are one-off cases (apart from glow which occurs 3 times) and are in either a hyponymic or an antonymic relation with it. The last unit, with a frequency of 17%, is the one invariably composed by SHED+blood5. The remaining two nouns found in the data, smile and sweat, have quite distinct meanings from the rest. However it could be argued that they may still be viewed as reflecting semantic preference, though on the basis of a rather more remote semantic similarity to other paradigmatic realisations this time. Namely, sweat could be grouped together with ‘blood', since they both relate to human physiology, while smile, although appearing out of context to be semantically associated with ‘tear’ (carrying common reference to human feeling being facially expressed) should rather be grouped with ‘light’ on account of a simile being made contextually: ‘she doffed her mask with a swift gesture and shed upon me a smile like a sunbeam'. 
In terms of number, 68% of the noun heads are not marked for plural and in the majority (67%) are preceded by a modifier. Specific deictics and non-specific deictics are employed with a similar frequency. A characteristic feature of the delexical frame SHED+noun appears to be the use of partitive structures, mostly complex numeratives. It is not particularly high statistically (14%), but it does not seem negligible either, if compared to the frequency of use of adjectives (12%) or classifiers (2%). Quantification is often being done through nominal expressions which have a descriptive force (e.g. ‘its pool of light') and whose function in most cases is rather an ‘intensifying’ one: ‘rivers/ showers/ floods/ buckets of tears', ‘streams/ a drop of blood'. The important thing is not the quantity they refer to but the emphasis the add: ‘without shedding a drop of blood'. If we compare an expression like that -say ‘buckets of tears'- with the possible paraphrase using a single verb -‘cry a lot', the difference in expressivity and forcefulness is evident. The image evoked in the former constitutes a hyperbole and the tone of the writer could be described as humorous and/or self-sarcastic (the context supports this kind of interpretation: ‘I was sixteen and shed buckets of tears when he left'). Nevertheless, after examining individual delexical structures involving the same verb as well as structures with different delexical verbs, we have come to believe that generalisations about the internal structure (and consequently the overall semantic function) of multi-word semantic units bearing the same delexical verb can be misleading, and each unit has to be analysed individually in order to establish safely its typical realisations and overall function. 
To illustrate this point, we will now briefly comment on the colligational behaviour of the unit SHED+TEAR -which is the most typical realisation of the frame SHED+noun, with respect to its contribution in defining the overall meaning of the unit. TEAR is used in the plural 16% of the time, while all the other nouns listed above are singular (with the exception of one instance of lights). It is hardly surprising, as TEAR not only is a countable noun (while blood is not and light only when denoting ‘source of artificial light'), but most importantly refers to a concrete entity that is typically perceived as ‘physically produced in more than one unit at a time'6. A closer inspection of the relevant concordances showed that 9 out of 16 times where singular is used, it is combined with negative polarity of the verbal group. Negative polarity also appears to be of a relative importance in determining the structure of the unit, since it is present in about 20% of its instances (in fact, it accounts for 2/3 of all instances of the SHED+noun frame where non-positive meanings are expressed). Negation seems to be emphasized in cases where the deictic no is used or postdeictics like in the phrases not+SHED+ a single/ the smallest tear. These phrases could be viewed as carrying semantic prosody by being more overtly evaluative/ subjective on the part of the narrator or ‘commentator’ of the events, expressing disapproval, surprise etc. and are meant to influence the judgment of the reader (which clearly distinguishes them from possible paraphrases such as ‘did not cry at all') part of the narrator -or ‘commentator’ of the events, expressing disapproval, surprise etc. and be meant to influence the judgment of the reader. 
5 It should be noted that -as is often the case when one tries to categorize language in use with respect to meaning/function- there are some borderline cases where, although the noun object is one of those discussed so far, arguments in favour of a delexical interpretation of the expression appear rather weak: ‘it <=the Spanish soil> has been bought at the highest price and with the most precious coin: the Spanish blood shed here', ‘ I think that the tears that the doctors <…> had in mind were things like torn ligaments rather than the kind shed by Paul Gascoigne’ or ‘the scarlet colour of her lips reminded some Christians of the blood shed by Christ at his passion'. The nouns in the above examples are felt to be much more ‘concrete', an interpretation that is accentuated by (or is partly due to) the clause structure, namely non-finite v-ed in a restrictive qualifying relation with its understood subject. 
6 It not being the most typical realisation is, in fact, surprising, and the most likely reason for which SHED+tear feels intuitively more idiomatic. 
861
On the other hand, the use of singular in the 7 examples with positive polarity seems to add an extra nuance of meaning, that is aspectual: it minimises the duration of the activity, it presents it as lasting for a short time7 e.g. ‘shedding another tear of relief’ (through the use of another reference is made to individual instances of the activity), ‘The last clod of earth fell on the coffin. Michael finally shed a tear’ (here in line with the given aspectual meaning is the fact that the Agent restrains himself from indulging fully in it). This implied aspectual meaning can contribute contextually to a further meaning being carried by the expression, this time pragmatic, i.e. either the presentation of the activity as non-genuine, e.g. ‘Hoffenberg shed a very small, very pathetic tear', or the downplaying of the importance or the extent of the activity, e.g. ‘I've shed a tear or two myself'. Finally, we could mention the fact that pronominal substitution of this head is observed on 5 occasions, whereas pronominalisation does not occur with the other nouns entering the construction; the concrete reference of tear/-s to a physical object, might be used as a (partial) explanation. 
Turning to the unit SHED+‘light', it might be of interest to note that LIGHT appears to be typically expressing a figurative meaning -only in 7% of its instances is it used literally- and it accounts for all the SHED+noun structures with a metaphoric meaning in our data. Exceptionally, the noun dark is also used metaphorically in one example, where by being compared with LIGHT is readily perceived as expressing an opposite meaning: ‘The philosophy does have grey areas, there are not always clear differences between right and wrong, but there is a pattern to it, there is an on-going process which I think sheds light and dark over our lives'. Similarly, the unit SHED+blood typically refers to physical death, but can also have an alternative meaning as the following example illustrates: ‘<she> sustained a nasty bruising on the forehead and some blood was shed', ‘he undoubtedly shed blood in fighting other prisoners', where we find a more general reference to bodily harm (which also implies suffering as a consequence). In addition to this, on certain occasions it roughly corresponds to ‘die', the disambiguation between this meaning and ‘kill’ being achieved co-textually through a colligation, i.e. the use of a possessive deictic in the expressions intended to carry the former meaning8. 
Nevertheless, in order to be consistent with our treatment of these verb and noun combinations as lexical items, it would be more fitting to describe the whole unit as being used metaphorically and not only the noun collocates9. In relation to metaphoricity, Sinclair (1998:22) makes briefly a relevant point when discussing the word budge, that seems to be supported by our data, namely that the distinction between literal and non-literal uses ‘hardly seems necessary', since both the literal and figurative aspects of the meaning of other words in the wider context are highly relevant too for the overall interpretation. Effectively, a literal or metaphorical interpretation is ultimately determined by adding up (or balancing out) contextual clues, although literal as well as metaphorical elements might be simultaneously present, display a degree of subtlety, and interact in a way that the intended reading of a particular piece of language might remain inconclusive. For instance, SHED+TEAR appears to us as not necessarily implying ‘secreting drops of fluid’ but potentially designating merely ‘an emotional state of sadness', however the co-text does not seem to provide readily concrete -i.e. linguistically encoded- evidence for this. Similarly, the unit SHED+blood, it is noteworthy that although it typically refers to physical death, it can have an alternative meaning as the following example illustrates: ‘<she> sustained a nasty bruising on the forehead and some blood was shed', ‘he undoubtedly shed blood in fighting other prisoners', where we find a more general reference to bodily harm (which also implies suffering as a consequence). In addition to this, on certain occasions it roughly corresponds to ‘die', the disambiguation between this meaning and ‘kill’ being achieved co-textually through a colligation, i.e. the use of a possessive deictic in the expressions intended to carry the former meaning10. 
With SHED+‘light’ as well as with SHED+‘blood’ a second participant can be said to be implicated in the predicate's valency as an Affected entity. This is either suppressed linguistically or expressed by means of a prepositional phrase functioning on the surface level as a Spatial circumstantial element in the SHED+‘light’ unit, and as a noun qualifier (prepositional phrase introduced by of) in the SHED+‘blood’ 
7 This semantic trait is by no means restricted to this particular delexical structure. 
8 This by no means implies that we consider it synonymous with the verb DIE, the difference lying in the exact nature of the subject's agency among other things, but this is a complex issue that will not be dealt with here. 
9 As far as units that have both literal and (standard) metaphorical realisations are concerned, this would raise the issue of polysemy vs. synonymy. 
10 This by no means implies that we consider it synonymous with the verb DIE, the difference lying in the exact nature of the subject's agency among other things, but this is a complex issue that will not be dealt with here. 
862
unit (e.g. ‘He preferred to die, rather than shed the blood of his would-be-murderers'). A third participant occurs occasionally as part of the SHED+‘blood’ unit, actualised as a prepositional phrase introduced by for, which, although it functions as a Circumstance of Support, is semantically very close to the role typically expressed by a Beneficiary. Lastly, the structure SHED+TEAR is also accompanied quite often by a circumstantial element (for/over+noun) that either refers to a person or to an event or situation which at semantic level is the entity causing a particular instance of the activity. It is also important to assess the typicality as well as the centrality of colligational tendencies such as these for each unit separately11, and incorporate this kind of information in an eventual definition. 
We will now be turning to the other two delexical verbs; however space limitations oblige us to restrict ourselves to a much briefer commentary. 
2. 2 CAST as node 
214 concordances of CAST were considered instances of its delexical function and were kept for analysis, which makes a frequency of 43%, if we take into account the number of concordances initially retrieved. The analysis of the verb groups showed them to be 62% of the time finites, 99% of them have positive polarity, 88% are active and, in terms of aspect, they are typically non-progressive and non-perfective (79% and 91%respectively). The non-finites express the indicative mood 96% of the time and are marked for past 72% of the time; they are most frequently singular (74%) and as for person, the 3rd occurs 93% of the time. 
The following units were identified by examining the collational patterning of CAST with respect to its noun complements and the semantic preferences realised: 
CAST+ ‘glance’ 
GLANCE 
LOOK 
EYE CAST+‘shadow’ 
SHADOW 
SHADE 
LIGHT 
GLOW 
BEAM 
ILLUMINATION 
RADIANCE 
REFLECTION 
SHEEN 
COLOUR CAST+‘doubt’ 
DOUBT 
SUSPICION 
ASPERSIONS 
BLIGHT 
SLURS 
SCORN CAST+‘mind’ 
MIND 
THOUGHTS CAST+SPELL CAST+VOTE 
Figure 2 
Detailed analysis showed that the most common unit (accounting for 36% of the data) is CAST+‘shadow'. The words realising its semantic preference denote not only the actual form of energy but also the result of its succeeding to or not physical objects: shadow/-s, shade, light/-s, beam, illumination, glow, radiance, sheen, reflection and colour (beam is the only noun referring to a unit of light) 12. The next most frequent semantic unit is CAST+‘glance’ (accounting for 30% of the data), under which the nouns glance/-s, look/-s, eye/-s. We have also included here the noun smile, as it is semantically associated with them on the basis of its reference to bodily-facial activity, although the rest have to do with the human sensory system and it does not. In fact, at a closer inspection, all 4 nouns can be seen as carrying reference to human communication with either the environment or other people: the first two refer to the input and the third to the output of this kind of interaction. 15% of the data instantiate the unit CAST+‘doubt', whose collocate constituents are dount/-s, suspicion, scorn, slurs, aspersions and blight (with doubt/-s being clearly the core member of the set). The next two units CAST+SPELL and CAST+VOTE occur 6% and 7% of the time respectively. Last comes CAST+‘mind’ with a frequency of 4%. 
The units involving nouns from the ‘light’ and ‘glance’ sets are those assigned most frequently intrinsic qualities in the form of Epithets. An interesting trait of the adjectives modifying the ‘glance’ set is that they are most often descriptive of ‘a state of mind or emotion', e.g. furtive, anxious, wistful, curious 
11 For instance, out of the colligational features mentioned here the co-occurrence of an adverbial denoting Place with SHED+;LIGHT’ is of much greater significance than the Circumstance accompanying SHED+TEAR, as it occurs 91% of the time while the latter 16% of the time. 
12 The reason for which we gave ‘shadow’ as a label and not ‘light’ is that collocates expressing the first meaning are twice as frequent. In a more fine-grained description, these nouns should better be presented as constituents of two distinct units in antonymic relation with each other. 
863
etc. +glance/-s, fond, icy, sly, worried etc. +look/-s, hopeful, envious, critical etc. +eye/-s. A subset could be distinguished comprising adjectives with reference to speech acts: apologetic, questioning, speculative, doubtful. Other than that, they express ‘frequency or duration', e.g. frequent, quick, lingering etc., ‘direction', e.g. sidelong, backwards, retrospective etc. and in 2 instances they have some kind of evaluative meaning: sweet, wan +smile. The modifiers used are highly significant with respect to the construction of the meaning of the extended unit in question, as they specify it in most cases as an act of transfer of feelings, opinion etc. of the Agent with ‘eyes’ being an instrument of communication rather than perception. Modification does not seem particularly significant in the case of the other units, where epithets are far less common: the nouns of the ‘spell’ group are modified by an epithet 30% of the time, those of ‘doubt’ 10%, the ‘vote’ ones 7% and finally for the members of the ‘mind’ group Epithet is never realised. Once again this seems to support the claim made earlier that subsuming lexicogrammatical features of individual units under a single description by virtue of them sharing the same delexical verb can distort meaning identification and misrepresent it or under-specify it in a definition. 
Let us now turn to a few collocates, feeling (of desolation), ripples, eddies and spray (of rainwater), that we found problematic in terms of categorisation, in order to demonstrate a different point. The first one is a word with a rather general meaning, which despite being specified further in the co-text through the qualifier, cannot justify the noun's occurrence in terms of semantic preference (needless to say that a single occurrence would not justify positing a distinct unit of meaning). The same holds for the other three that arguably have a more specific lexical meaning. However, they neither cause a problem of interpretation in their co-occurrence with CAST, nor do they feel particularly unidiomatic. On the basis of having come across cases like these repeatedly in a current study we are carrying out of a great number of delexical structures, we maintain that lexical combinations of each particular delexical verb with a noun –which we call a ‘delexical frame', share certain aspects of meaning with each other, even when they do not constitute realisations of the same unit and this often seems to originate in the ‘prototypical’ meaning of the delexical verb. Moreover, some of the units display a closer ‘semantic affinity’ than others, which makes it very difficult at times to draw the line between classifying structures as realisations of the same unit or to identify them as distinct units. Consider for instance the obvious meaning relation of the structures CAST+SHADOW and CAST+LIGHT (that we grouped together in Figure 2), or a less straightforward one connecting CAST+‘shadow', CAST+SPELL with CAST+ripples CAST+eddies, i.e.designating some kind of energy transfer and the resulting effect of this transfer (the energy is supernatural in the case of spell and physical for the other two nouns. A clear component of the common semantic core of all the structures seems to be a general aspectual meaning of causation, which highlights the Agent's role and moving from top to bottom would reveal more specific meanings present in subgroupings of structures13. The nature of these semantic relations is a very interesting issue but too complex to be discussed in more detail here. 
We will close this section with a brief comment on the overlap that structures might display with respect to the nouns forming part of their lexical core. Our view is that their parallel existence can only be explained by them having a different function. Using the expressions CAST+LIGHT and SHED+LIGHT as an example, we can first mention that there is a clear difference of frequency –the latter being almost 4 times as frequent (and accounting for 39% of the delexical occurrences of SHED). Besides, it has 97% of the time a metaphorical meaning (i.e. ‘make clearer'), while CAST+LIGHT has a similar meaning in only 32% of its instances. In addition to this, half of the examples of SHED+LIGHT have modal meanings expressed, which could be attributed to the metaphorical meaning of the expression involving ‘an intellectual quest for knowledge or truth', which in turn is inherently associated with a degree of uncertainty and subjectivity. On the other hand, CAST+LIGHT is much more heavily premodified -and in that respect less ‘fixed'-, relying rather heavily on adjectives as well as predeictic elements for the construction of its overall meaning in context. It could be claimed that it is ultimately their discourse function -directly relating to textual distribution- that differentiates them: CAST+LIGHT is typically used in descriptive accounts of the natural environment (very common in fiction), whereas SHED+light typically refers to cognitive processes (characteristic of academic language). In effect, this point highlights the importance of examining the distribution and function of delexical structures across genres and registers in order to discover their intertextual, interdiscoursal functions, that are ultimately the purpose for which they are selected. 
13 Wierzbicka (1982), in her discussion of the construction ‘Have a V, argues for the existence of subtypes which must be defined in terms of semantic domains. 
864
2.3 THROW as node 
The verb THROW was found to be used delexically with a very low frequency: only 5% of its instances were delexical structures. Nevertheless, 18 different lexical items occurred as noun complements, 11 of which were one-offs. The majority of the verb groups are finite (88%) and active (92%) and primarily non-progressive and non-perfective (83% and 97% respectively). Finite clauses are marked for past reference 64% of the time. they are exclusively in the 3rd person and predominantly singular (74% of the time). Modality is encoded in 11% of the data. 
THROW+‘light’ 
LIGHT 
SHADOW 
GLOW THROW+‘look’ 
LOOK 
GLANCE 
SMILE THROW+‘utterance’ 
HINT 
EXPRESSION 
ACCUSATION 
QUESTION THROW+‘party’ 
BALL 
MATCH THROW+‘doubt’ 
DOUBT 
SUSPICION THROW+‘attack’ 
PUNCH 
BROADSIDE 
Figure 3 
THROW co-occurs with a wide range of noun collocates, some clearly forming sets on semantic grounds. The unit THROW+‘light’ was found to be realised in our data by the nouns light, shadow and glow, referring to brightness (or its absence); THROW+‘look’ by the nouns look, glance and smile, all three referring to a bodily function relating to human communication; THROW+‘utterance’ by question/-s, accusations, expressions and hints, where reference to interpersonal interaction is made as well, although in this case communication takes place through verbalised messages; THROW+‘party’ by party, ball and match, which refer to social events related to leisure and entertainment; THROW+‘doubt’ by doubt and suspicion; and lastly, THROW+‘attack’ by punches and broadside14. There are few more nouns co-occurring with throw in our data, which we were not able to assign to one of the previous groups, i.e. doubt, spell, salto-mortal, lock, ball. We could not stipulate them as distinct units either, as units have to be defined probabilistically, which necessitates sound evidence of routinization. 
An important point could be made here, namely that a particular (noun)lexeme not only can form part of different extended units in its co-occurrence with different delexical verbs, but can also belong to different units among those formed with the same verb by virtue of its polysemy. On the other hand, not all of its instances are necessarily realisations of extended units (this observation, however, concerns only non-deverbal nouns). Take ball as an example: different senses of the lexeme are observed in ‘pitchers who threw too many gopher balls’ and ‘he throws lavish balls (= ‘dances') at his clubs'; in ‘Pitchers like Jesus and Socrates were experts at throwing a curve ball’ it is understood as conveying the action performed, thus it is considered to enter a delexical structure and form part of the predicate, whereas in ‘I got up and went to throw the ball out to the wing’ it refers to the actual physical object and is discounted. It should also be stressed that co-text might dictate interpreting the meaning of a structure in a quite unexpected way, and could mark the structure as not being an instance of the unit that the noun in question typically realises (see for instance the comment made in section 2.1 regarding SHED+smile). 
THROW has a frequency of 12.7 per million words in the Bank of English, which makes it by far the most frequent out of the three verbs examined here -CAST occurs 3.5 times per million words and SHED 1.6. It could be speculated that the overall frequency of a verb (i.e. irrespective of its meanings/functions) is a correlate of its productivity with respect to the range of delexical structures it forms: the more common it is in language in general, the more likely it is to collocate with nouns and build the core of various units extending at the level of a clause. Moreover, we see the facility with which a delexical verb lends itself to the creation of phraseological verb-noun combinations, i.e. the number of discrete units it forms part of, as indicative of its ‘degree of delexicalisation'. We take delexicalisation to refer not only to the process of the semantic weakening of a lexical item but also to its resulting state, which can be presented as a cline along which different verbs occupy different positions. In this line of 
14 Broadside is used elliptically as the noun head: ‘with […] and her eight eighteen-pounders she <=ship> could throw a broadside of no less than 456 pounds, more than the gundeck of a line-of-battle ship', where an act of violence is referred to (a noun like shot or attack is substituted for). 
865
thought, THROW is the most delexical out of the three verbs, SHED is the least delexical, while CAST comes somewhere in the middle. Certainly, the description of the use of a verb characterised by a relatively high degree of delexicalisation, needs to be grounded in more extensive data, so that more patterns can surface and observations regarding strength of attraction be made possible, a prerequisite for any linguistic description in probabilistic terms. 
4. Delexical structures as instances of functionally complete units of meaning 
We believe that reconciling the paradigmatic and syntagmatic axes of patterning and allowing for descriptive flexibility of meaningful choices in language, makes the model of the lexical item as put forward by Sinclair a powerful descriptive framework. His proposed new kind of lexical item though, in order to be of real value, has to be defined in a comprehensive way and to be as independent as possible. Extended units involving a delexical verb co-occurring with a noun, due to the fact that they are phraseological units at the level of the clause, are to a certain extent self-contained and thus in principle would lend themselves more easily to such an analysis of meaning. We believe that descriptions of delexical uses of verbs restricted mainly to listing possible or probable noun collocates are of limited usefulness and are doomed to fail to capture their overall meaning potential. The meaning of these extended units is constructed through the interplay of their colligational and collocational patterning and is complete -and thus becomes their linguistic function- only when more abstract connotational meanings are specified. These prosodies15 are meanings acquired through habitual usage of the units in question in recurring contexts, and they can be located at different points on the semantics-pragmatics continuum. We see delexical verb+noun units as carrying primarily interpersonal meanings of various kinds, reflecting the way the speaker (or writer) position themselves with respect to the hearer (or reader) or other entities involved in a given situation. Taking into account our theoretical standpoint according to which meaning is function, their stylistic values, established intertextually and relating directly to their currency in particular genres and registers, are also integral part of their meaning. Moreover, our view is that these linguistic expressions are an inherently subjective way of conceptualising events, carrying conventionalised aspectual characteristics of various kinds. It could be maintained that all these semantic components are indispensable parts of a functional definition of their meaning, one that would merge ‘the two dimensions, the contextual one and the functional one’ (Tognini-Bonelli, 2002:79). Besides, only defining them as functional units would eventually allow, on one hand, differentiating structures that exhibit a partial lexical overlap, i.e. share items of their lexical core with certain objectivity, as well as specifying with precision the meaning components that are shared between units comprising the same delexical verb. 
We would like to close this section by commenting on an important characteristic of the extended units of meaning comprising delexical verbs, which can be readily observed with respect to the units we tentatively posited above. A unit of this type might be realised not only through one or more typical noun objects, but also by some other ‘peripheral’ nouns (consider for example the one-off collocates of SHED). Peripheral status is assigned to them on the grounds of low frequency and possible difficulty in allocating them to a lexical set (realising semantic preference), and the delexical structures they form can vary with respect to markedness, conventionality etc. It seems that the noun has to fulfil certain semantic criteria though, i.e. semantic closeness to a ‘core’ noun. In other words, the selection appears to be done paradigmatically not in an arbitrary way but in compliance with a meaning already expressed by an existing unit, hence reflecting semantic preference. It could be argued that the implication of this for the nature of these lexical units is that besides the greater or lesser degree of flexibility at the level of optional constituents (which by definition is part of the nature of an extended unit of meaning), they often display structural variability with respect to their internal constituents at the level of their core. 
5. Conclusion 
Our aim has been to demonstrate how the model proposed by Sinclair can be applied to the description of delexical structures, which lie in the largely unexplored grey area between idioms and collocations. Delexical verbs enter units, which can be variable in their lexical realisations and which might generally be 
15 Stubbs (2001:66) uses the term ‘discourse prosodies’ but suggests that ‘pragmatic prosodies’ might be preferable. 
866
considered conventional in their semantic patterning, but in fact carry meanings beyond a straightforward denotational core that need extensive, as well as detailed, corpus studies in order to be pinned down. This makes the relations used for defining the structure of the extended unit -collocation, colligation, semantic preference and semantic prosody-, a very fitting way for specifying their function. Given the limited space, we did not attempt any complete formal definition of the units discussed, but we hope to have given a glimpse on how all four categories of co-selection can be used for this purpose. In our opinion, verb delexicalisation is a far more pervasive phenomenon than is generally suggested, and delexical verbs, which we see as verbs with a ‘vague’ or ‘general’ meaning, rather than ‘semantically empty', can form part of a very wide variety of distinct semantic units, which also seem to carry certain common semantic components. 
References 
Algeo, J 1995 Having a look at the expanded predicate. In Aarts, B, Meyer, C (eds) The Verb in Contemporary English. Cambridge, Cambridge University Press, pp 203-217. 
Allerton, D 1982 Valency and the English verb. London, Academic Press. 
Cattell, R 1984 Composite Predicates in English. (Syntax and Semantics 17). Sydney, Academic Press. 
Curme, G 1931-1935 A grammar of the English language. Boston, D.C. Heath & Co. 
Dixon, R 1991 A New Approach to English Grammar, on Semantic Principles. Oxford, Clarendon Press. 
Jespersen, O 1933 Essentials of English Grammar .London, George Allen & Unwin. 
Leech, G, Svartvik, J (1994, 2nd ed) A communicative grammar of English. London, Longman. 
Live, A 1973 The take-have phrasal in English. Linguistics 95: 31-50. 
Nickel, G 1968 Complex verbal structures in English. IRAL VII: 1-21. 
Nickel, G 1978 Complex Verbal Structures in English. In Nehls, D (ed) Studies in Descriptive English Grammar. (Studies in Descriptive Linguistics 1). Heidelberg, Julius Groos, pp 63-83. 
Poutsma, H 1926 A Grammar of Late Modern English. Groningen, P. Noordhoff. 
Renský, M 1964 Nominal tendencies in English. Philologica Pragensia 7: 135-150. 
Sinclair, J 1991 Corpus, Concordance, Collocation. Oxford, Oxford University Press. 
Sinclair, J 1996 The Search for Units of Meaning. Textus IX: 75-106. 
Sinclair, J 1998 The lexical item. In Weigand, E (ed), Contrastive Lexical Semantics. Amsterdam and Philadelphia, Benjamins, pp 1-24. 
Stein, G 1991 The phrasal verb type ‘to have a look’ in modern English. IRAL XXIX: 1-29. 
Strang, B 1970 A History of English. London, Methuen & Co. 
Stubbs, M 2001 Words and Phrases: Corpus Studies of Lexical Semantics. Oxford, Blackwell. 
Tognini-Bonelli, E 2002 Functionally complete units of meaning across English and Italian: Towards a corpus-driven approach. In Altenberg, B, Granger, S (eds) Lexis in contrast: corpus-based approaches. Amsterdam and Philadelphia: Benjamins, pp 73-95. 
Tognini-Bonelli, E 2001 Corpus Linguistics at Work. Amsterdam and Philadelphia, Benjamins. 
867
Wierzbicka, A 1982 Why can you have a drink when you can't *have an eat?. Language 58: 753-799. 
List of tables 
Table 1 
FINITENESS POLARITY VOICE ASPECT MODALS Finite Non-f. Pos. Neg. Act. Pass. Non-Pr. Non-Pf. THROW 88% 12% 100% - 92% 8% 83% 97% 11% CAST 62% 38% 99% 1% 88% 12% 79% 91% 6% SHED 53% 47% 86% 14% 86% 14% 85% 86% 19% 
Table 2 
MOOD TENSE NUMBER PERSON Indicative Imperative Non-Past Past Singular Plural 1st 2nd 3rd THROW 100% - 36% 64% 74% 26% - - 100%CAST 96% 4% 28% 72% 74% 25% 3% 4% 93% SHED 99.5% 0.5% 40% 60% 70% 30% 13% 3% 84% 
Table 3 
THROW CAST SHED Number of nouns 18 > 11 one-offs 30 >16 one offs 14 >10 one-offs SINGULAR 75% 69% 70% PLURAL 25% 31% 30% PREMODIEFIED 75% 85% 68% Pre-deictics 7% Pre-deictics 5% Pre-deictics 0.8% Spec. deictics 14% Spec. deictics 31% Spec. deictics 34% Non-Spec. d. 42% Non-Spec. d. 43% Non-Spec. d. 37% Post-deictics 4% Post-deictics 3% Post-deictics 1% Numeratives 8% Numeratives 10% Numeratives 21% Epithets 31% Epithets 50% Epithets 17% Classifiers 11% Classifiers 0.5% Classifiers 2% POSTMODIFIED 8% 5% 2% Prepos. phr. as Qualifiers 33% Prepos. phr.as Qualifiers 80% Prepos. phr. as Qualifiers 100% Clause as Q. 33% Clause as Q. 20% Clause as Q. - Adjective as Q. 33% Adjective Q. - Adjective Q. - 
868

Mutual information and corpus-based approaches to English reduplication 
Shih-ping Wang 
Ming Chuan University and University of Nottingham 
spwang@mcu.edu.tw 
The aim of this project is to use corpus-based approaches to investigate reduplicative fixed expressions in English, e.g., sooner or later, first and foremost, part and parcel, etc. The probabilistic relations between two adjacent words were examined. A reduplication corpus has been constructed and the frequency of each token was calculated based on its occurrence in the British National Corpus (BNC). Then a word list with 116 items was proposed for related research in terms of SARA software, the built-in tool of BNC. Frequency is often considered as the main factor to decide the word order in a conjoined phrase (Fenk-Oczlon, 1989, 2001), but the frequency-based argument is shaky and not always reliable. Mutual information (MI) was therefore employed to calculate the probability of collocation and assess collocational significance. MI can be used to decide what to look for in a concordance (Church and Hanks, 1990). The higher the mutual information, the more genuine the association between two words. 
Keywords: BNC, corpus, reduplication, frequency, mutual information, and z-score 
1. INTRODUCTION 
The aim of this project, grant-funded by the National Science Council (NSC) is to use corpus-based approaches to investigate reduplicative fixed expressions in English, e.g., sooner or later, first and foremost, part and parcel, etc. Reduplication is important in language studies. Its word order at the phrasal level is explored in the present study. 
A reduplication corpus (1,700 items) had been accumulated. The frequency of each token was calculated based on its occurrence in the British National Corpus (BNC). A reduplicative wordlist with 232 items was established based on the self-constructed corpus. Then a questionnaire with 116 items, extracted from the wordlist, was proposed for related searches using the built-in SARA software. 
Frequency is important in creating a word list. It is also considered as the main factor to decide which item goes first in a conjoined phrase (Fenk-Oczlon, 1989, 2001). According to Fenk-Oczlon's rule, high frequency comes before low frequency in binomials. However, frequency is not the only criterion to decide the word order in fixed expressions. The probabilistic relations between two adjacent words should be examined as well. Mutual information (MI) and z-score are therefore employed to assess collocational significance (Church and Hanks, 1990). Both MI and z-score may provide useful insights into direction of collocability. Therefore the primary question of current researches is to explore fixed reduplications and collocational significance. Three aspects, frequency, MI and z-score, are discussed to evaluate which method is more appropriate to decide the word order in binomials and how they shed new light on word order and collocational analysis in terms of corpus-based approaches. 
869
2. LITERATURE REVIEW 
2.1 Multiword units and reduplicative fixed expressions 
In the research of word usage, linguists have recently turned their attention to multiword units (MWUs), which are strings of words acting as a unitary lexical item with a single meaning (Carter, 1998; Moon, 1998). MWU includes compound words, phrasal verbs, fixed phrases, idioms and proverbs (Schmitt, 2000, pp. 99-100). Freezes or fixed expressions consist of irreversible conjoined phrases and fixed reduplicatives (Pinker and Birdsong, 1979). Reduplicative MWUs can be compounds, fixed expressions and other types, e.g., first and foremost (248)1, deaf and dumb (276), bits and bobs (49) and the like. 
Reduplicated word-formation varies in English; examples consist of ablaut and rhyming terms: 'i-a' (riprap), 'i-o' (ping-pong), super-duper and hocus-pocus. Terminology to describe the phenomenon also shows a discrepancy and includes: fixed expressions, freezes, binomials and frozen locutions (Pinker and Birdsong, 1979; McCarthy, 1990; Landsberg, 1995; Moon, 1998). Binomials or trinomials are usually irreversible combinations with other conjunctions whose order may be different from language to language, e.g., sooner or later (503) 2 (McCarthy, 1998, 130-131). 
Basically fixed expressions can be divided into the concise formal types as shown in Table 1 (Carter, 1998; Moon, 1998): 
Table 1 Formal types of fixed expressions 
Types of freezes Examples irreversible conjoined phrases wear and tear 
hook, line and sinker 
first and foremost fixed reduplicatives 
•vowel alternations (ablaut) pitter-patter 
ping-pong •rhyming terms super-duper 
razzle-dazzle 
hocus-pocus 
2.2 Freezes, word order and frequency-based arguments 
Fenk-Oczlon (1989, 2001) argued that the frequency-based approaches can be simply used to solve some old questions. A new rule was proposed for the decision of word order in freezes: high frequency before low frequency, which implies ‘more frequent tokens come before less frequent ones.’ For example, the frequency of ‘plus’ (7,767) in the fixed expression, ‘plus or minus’ is higher than that of the second one, ‘minus’ (1,776); the frequency of occurrences for their collocation is 66. According to Fenk-Oczlon, the new rule achieves the highest accuracy with 84% correct predictions in his corpus. Frequency is considered 
1 The Arabic numerals mean the frequency of each token in the British National Corpus. 
2 In Mandarin Chinese the word order is opposite, i.e., chi-tsao (‘late-early')   ‘sooner or later'. 
870
as the main factor to decide which item goes first in a conjoined phrase (Landsberg, 1995). However, is frequency the only criterion to decide the word order in fixed expressions? It is often maintained that the probabilistic relations between two adjacent words should be considered as well when dealing with the fixed expressions (Moon, 1998; Schmitt, 2000). 
2.3 Mutual information, z-score and frequency in the British National Corpus 
There are three major kinds of scores frequently proposed to assess the collocational significance of each co-occurrence: i.e., mutual information (MI) score, t-score and z-score (McEnery and Wilson, 1996; Hunston, 2002). MI-score is probably the best known among them. T-score and z-score are most similar in terms of how they are calculated. MI and z-score are the two formulae mostly used to calculate the relationship of significant collocations. MI-score and z-score can be calculated using the SARA software, a built-in tool in the BNC, by which the z-score is generally recommended. Therefore, only both MI and Z-scores together with frequency will be disussed in the current research. 
MI is a measure of the strength of collocation, provides a summary of what company words keep and thus is used for assessing collocational significance (Aston and Burnard, 1998). The higher the MI score, the more genuine the association between two words (Oakes, 1998), which can be calculated in terms of the following formula (Church and Hanks, 1990; Stubbs, 1995): 
I = log2 ((f(n, c) x N) / (f(n) x f(c)) 
• I = MI = mutual information; n = node; c = collocate 
• f(n, c) is the collocation frequency 
• f(n) is the frequency of node word (the query focus) 
• f(c) is the frequency of the collocate 
• N is the number of words in the corpus (corpus size): 
- If I (n; c) > 3, then the pairs tend to be significant or ‘interesting'3. 
- If I (n; c) ~ 0, then the pairs are less interesting. 
- If I (n; c) < 0, then x and y are in complementary distribution. 
The z-score is the number of standard deviations from the mean frequency. It is used to measure how likely it is that the focus/node and collocate are related. The higher the z-score is for an item related to the node word, the greater is its degree of collocability with that word (McEnery and Wilson, 1996). 
3 An MI score greater than (or equal to) 3 may indicate a significant collocational link (Church and Hanks, 1990: 24; Hunston, 2002: p. 71). 
871
2.4 Problems and Research Questions 
Frequency is commonly assumed to influence the word order in fixed expressions. However, is frequency the main factor to decide which item goes first in a conjoined phrase? In addition, the current topic of reduplication and binomial freezes is often neglected because it presents problems for those theorists. Therefore the research questions of the present study will focus on the following topics: 
  Reduplication at the lexical level; 
  Frequency explored to see whether it is the major factor to influence word order in freezes; 
  The frequency and probability of collocation for reduplicative freezes, calculated in terms of MI (Church and Hanks, 1990; Moon, 1998); 
  MI and z-score employed to calculate the probability of collocation and assess collocational significance. 
Frequency, MI and z-score are explored to see how they shed new light on word order and collocational analysis in terms of an integrated methodology. 
3 METHOD & PROCEDURE 
3.1 Data and Analytical Tools 
The data for the present study include the author's own reduplication corpus and the BNC. The author's data have been gathered in the field, from research papers, dictionaries, websites, newspapers, advertisements, slogans, etc. since 1986. The ongoing collection (about 1,700 tokens) has undergone two stages for this research. The criteria for the establishment of the reduplication corpus are mainly based on the pattern (Wang, 2002a): 
• The form of each token should be reduplicated in various types (onset, rhyme, etc.): e.g., first and foremost (247), this and that (202), and town and gown (8). 
• The pattern may be reduplicative binomial or trinomial expressions: 
- Full copy, X1 {conj., prep} X1, e.g., so and so, and all in all; 
- Partial reduplication, X{conj., prep, art}Y, e.g., wine and dine (10), tit for tat (92), trick or treat (15), and bric-a-brac (52) ; 
- Triplet and others: X1 X2 X3, e.g., tic tac toe (3) and Milly Molly Mandy (2). 
Only the second pattern, partial reduplication, is explored for this current study. The following are the main foci to be investigated in order to demonstrate how reduplication is used pervasively in day-to-day discourse: 
• Corpus: using the BNC and Constructing the author's own corpus; 
• Calculating the frequency for each selected token in the BNC; 
• New power extracting data to be used in the questionnaire; 
• Using Google to surf general websites to download examples of reduplication; 
• Instruments: using SPSS and SARA for statistical analysis; 
• Using SARA searches for the frequency of each token, MI and z-score. 
872
3.2 Procedure and Method 
The initial step is to collect data, analyze it and then create the compiled reduplication corpus (1,700 items). The first 232 items were extracted as wordlist 1, and then 116 items were selected as a questionnaire (wordlist 2). All tokens are chosen according to the following principles: 
•All types are mainly based on the patterns, [X] + and/or + [Y], including MWU, fixed expressions and idioms. 
•All tokens identified in the corpus as reduplications undergo SARA searches for their frequency of occurrence, MI-score and z-score. 
•ANOVA and Sheffé tests were used to investigate which approach underwent significant difference in terms of frequency, MI-score and z-score. 
Figure 1 summarizes the basic procedure for the current studies: 
Figure 1 Flowchart for the data-processing 
4. RESULTS and SAMPLE ANALYSIS 
4.1 Frequency counts and frequency-based method 
4.1.1 Procedure 1: a word list with the frequency 
Fenk-Oczlon (1989) proposes frequency-based arguments to investigate freezes or fixed expressions. Therefore, the frequency for each token in the BNC was calculated first. The abridged ranking order in Table 2 is based on the results of frequency searches, i.e. the collocational F(x,y). For example, the following F(x) represents ‘either’ (27152), and F(y), ‘or’ (367981). The frequency of their collocation, F(x, y) ‘either…or…’ is 22111. The MI-score of 22.30 and the z-score of .1178 are calculated respectively. 

Create a questionnaire 
(wordlist 2) 

Extract 232 items for wordlist 1 

Use SARA to search the freq.ofeach item 

Construct a 
reduplication corpus (1,700) 

Present statistical analysis 

Do ANOVA and Scheffé Tests 

Calculate z-score in the BNC 

Calculate MI score in the BNC 
873
Table 2 Abridged results for frequency-based grouping with MI and Z score 
No. grouping F(x)>F(y) F(x) F(y) F(x, y)MI Z scoreTokens 1 High 27152 36798122111 22.30 .1178 either X or Y 2 High + 1931791 1972731152 30.50 .4021 in and out (X in and X out) 3 High 1710 38424 503 30.40 .6051 sooner or later 4 High + 36609 6086 458 32.70 1.4491 once or twice 5 High + 2629 724 276 34.00 3.2693 deaf and dumb 6 High + 120825 604 247 33.90 3.3861 first and foremost 7 High 454441 1119521 202 23.00 .0686 this and that 8 High 5328 12249 156 28.70 .5973 upper and lower 9 High + 197273 192000152 24.50 .1436 out and about 10 High + 51557 717 134 32.10 2.2891 part and parcel 11 High + 8335 4715 112 29.10 .8123 positive or negative 12 High + 24401 9171 83 27.30 .5036 mother and daughter 13 High 78 111 69 32.50 4.1748 comings and goings 14 High + 7767 1776 66 29.00 1.0207 plus or minus 15 High 1187 2048 67 28.80 .9577 fame and fortune 16 High + 74666 30303 53 24.30 .1978 last but not least 17 High 1490 4324 54 27.20 .5917 odds and ends 18 High 3424 9978 48 25.60 .3671 rough and ready 19 High + 3282 92 49 31.80 3.8643 bits and bobs 20 High + 23864 10059 46 25.50 .358 black and blue 21 High 5124 19891 44 24.40 .2489 birth and death 22 High 2663 9361 43 25.40 .3588 mix and match N=116 ; n=76 Correct rate for Fenk-rule (n/N) = 65.5% 
4.1.2 Procedure 2: “High frequency before low frequency” 
Fenk-Oczlon's rule (1989; hereby Fenk-rule), “high frequency before low frequency,” indicates that the frequency of F(x) is higher than that of F(y), i.e., F(x) > F (y). There are only 76 items in the wordlist (total = 116) whose F(x) is bigger than F (y). This means that the correct rate of Fenk-rule is 65.5% in our test, not so accurate as 84% correct predictions in Fenk-Oczlon's statement. For example, the frequency of the first part (1710) of ‘sooner or later’ is lower than that of the second one (38424). It is evident that Fenk rule fails to come up consistently with the correct prediction. 
4.1.3 Procedure 3: Frequency-based grouping 
Using the ANOVA test, there is no significant difference for frequency-based groups and Z-score groups. Only MI between groups shows a significant difference (*p<.01; see Table 3). Based on the ranking of collocational frequency of occurrences, all the items were divided into three groups, higher (1), middle (2) and lower (3) groups. 
Table 3 ANOVA for frequency-based grouping ANOVA4.60E+4.83E+1.18E+1.25E+Between GroupsWithin GroupsTotalBetween GroupsWithin GroupsTotalBetween GroupsWithin GroupsTotalFXY.0MI.0ZSCORE.0Sum ofMean
874
226627082113313542.069.133
08845475709.9
0886600.3422300.17116.437.0001533.9618418.2612134.303867251390.223625695.12.585.081
08841402822.6
0886
Squaresdf
SquareFSig. 
The Scheffé test further indicates that only MI between groups exhibits significant difference (Table 4). It is evident that something else is needed to make up the shortcomings of frequency-based argument. 
Table 4 Scheffé test for frequency-based grouping Multiple ComparisonsScheffe---2.8205*7.2955*-2.8205*2.902E-024.4749*-7.2955*4.0845-4.4749*1.6835---(J) GROUP2.003.001.003.001.002.002.003.001.003.001.002.002.003.001.003.001.00(I) GROUP1.002.003.001.002.003.001.002.003.00Dependent VariableFXY.0MI.0ZSCORE.0MeanDifference(I-J)Std. LowerBoundUpperBound95% Confidence Interval
he mean difference is significant at the .05 level. 4.2 MI-based and z-score approaches 2.00T*. 
4.2.1 Procedure 4: Mutual Information (MI-based grouping) 
The MI-based grouping is based on the results of calculating the association of mutual information between two items (e.g., binomials) in terms of using the Sara tool. It can be divided into three new groups, i.e., higher group (the first 23%), lower group (the last 23%), and middle group (the rest). Again, the ANOVA test shows there is no significant difference for frequency-based ranking between groups. On the other hand, both MI and Z- score groups indicate significant differences between groups. 
Table 5 ANOVA for MI-based grouping ANOVA1585.7892792.895121.425.0002134.303866132644323066322140.396.0006376204484759071.9581.25E+4805388.322402694.2.422.6574.78E+08845688297.04.83E+Between GroupsWithin GroupsTotalBetween GroupsWithin GroupsTotalBetween GroupsWithin GroupsTotalMIZSCOREMIFXY.MISum ofMean
The Scheffé test pinpoints that MI-based grouping illustrates the significant differences only for the relations between MI and Z-score groups except for one comparison. 
875
548.514846.530
1169.0814613.382.169-359.49832697.66111184.0000705.544.250-574.25002942.2500
1169.0814613.382.169-2697.6611359.498314.9186613.3821.000-1513.66111543.4983
0886
1184.0000705.544.250-2942.2500574.2500
14.9186613.3821.000-1543.49831513.6611
0886
1.120.0472.902E-025.6120
1.288.0004.084510.5064
1.120.047-5.6120-
Squaresdf
SquareFSig. 
1.120.0011.68357.2664
1.288.000-10.5064-
1.120.001-7.2664-
70.9095310.465.974-702.7845844.6035707.7545357.112.147-182.18761597.6967
70.9095310.465.974-844.6035702.7845636.8450310.465.128-136.84901410.5390
707.7545357.112.147-1597.6967182.1876
636.8450310.465.128-1410.5390136.8490
ErrorSig.
Table 6 Scheffé test for MI-based grouping Multiple ComparisonsScheffe-4.9176-10.0572-3.72111716.5628*2194.8227*-1716.5628*2285.6900-1147.4356-2194.8227*2849.4617-1540.1837----(J) GROUP2.003.001.003.001.002.002.003.001.003.001.002.002.003.001.003.001.002.00(I) GROUP1.002.003.001.002.003.001.002.003.00Dependent VariableMIZSCOREMIFXY.MIMeanDifference(I-J)Std. LowerBoundUpperBound95% Confidence IntervalThe mean difference is significant at the .05 level.*. 
4.2.2 Procedure 5: Z Score-based grouping 
Likewise, given the z-score-based grouping, the frequency-based ranking between groups displays insignificant difference. The Scheffé test verifies the observations that various groups based on MI grouping exhibit significant differences. 
4.3 Summary 
Given the ANOVA tests and multiple comparisons, the key grouping methods can be summarized in terms of Table 7: 
Table 7 Nine Different grouping types and the results of ANOVA tests 
Ranking 
Grouping Frequency 
F (x, y) MI Z Score Frequency-based    MI-based    Z Score-based    
**p<.01 
It is apparent that no matter what kind of grouping method is used, there is no significant difference for frequency ranking groups. However, MI ranking always shows significant difference between groups. If frequency-based grouping is adopted, there is no significant difference for z-score ranking. Multiple comparisons such as Scheffé confirm the above summary. Therefore frequency is not the main factor to decide the word order and the collocations in frozen expressions. It is evident that MI is more important to assess the probabilistic collocation between two adjacent words. 
5. CONCLUDING REMARKS 
The aim of this research, using corpus-based approaches, has been to explore fixed expressions in terms of 
876
6.5869*.670.0004.91768.256111.9773*.770.00010.057213.8973
6.5869*.670.000-8.2561-
5.3904*.670.0003.72117.0596
11.9773*.770.000-13.8973-
5.3904*.670.000-7.0596-
228.377.0001147.43562285.6900
262.691.0001540.18372849.4617
228.377.000-
478.2599228.377.118-90.86731047.3871
262.691.000-
478.2599228.377.118-1047.387190.8673
379.7368625.176.832-1937.70641178.2329154.5909719.109.977-1637.46501946.6468379.7368625.176.832-1178.23291937.7064534.3277625.176.695-1023.64202092.2974
154.5909719.109.977-1946.64681637.4650
534.3277625.176.695-2092.29741023.6420
ErrorSig.
frozen types, frequency, correct rate, collocation, and MI score. A corpus with a ranked wordlist was constructed. The frequency and probability of collocation for reduplication were calculated in terms of MI and z-score. Frequency was explored, but it is not the major factor to influence the word order in freezes. The MI-based method is instead proposed to confirm observations of word order and to revise Fenk-Oczlon's arguments (1989). In addition, ANOVA and post hoc tests, Scheffé, were employed to evaluate which grouping method is appropriate to come up with robust statistics for collocational significance. The key points for this research are summarized along the following lines: 
  Given corpus-based approaches, Fenk's rule, high frequency before low frequency, is not always reliable. Among the 116 items in the present corpus, only 76 items meet this rule. The correct rate is 65.5%. 
  MI-based approaches play a useful role in aligning, reinforcing or making up the shortcomings of frequency-based arguments. 
  MI provides a quick guide to decide what to look for in the collocation pairs. It is used to calculate the probabilistic collocation of two observed items, f(x) and f(y). If the first item, f(x), is larger than f(y), the MI score is higher. 
Statistics such as percentage coverage, and frequency of occurrences in a corpus are required to reinforce and constitute relevant arguments and research approaches. MI and z-scores are useful reference points while choosing fixed expressions to discuss. Further studies integrating probabilistic methods are definitely needed. 
ACKNOWLEDGEMENT 
The author is grateful to Michael McCarthy, John Ohala, Stuart Davis, Ming-wen Wu for their very useful comments. The research is partially supported by a grant, NSC 91-2411-H-130-008, from the National Science Council. 
REFERENCES 
Aston, G. & Burnard, L. 1998 The BNC handbook. Edinburgh: Edinburgh University Press. 
Biber, D., Conrad, S.&, Reppen, R. 1998 Corpus linguistics: investigating structure and use. Cambridge: Cambridge University Press. 
Birdsong, D. 1995 Iconicity, markedness, and processing constraints in frozen locutions. In M. Landsberg (Ed). Syntactic iconicity and linguistic freezes, pp. 31-45. 
Carter, R. 1998 Vocabulary: applied linguistic perspectives. London: Routledge. 
Church, K., &, Hanks, P. 1990 Word association norms, mutual information, and lexicography. Computational Linguistics, 16(1), 22-29. 
Cook, G. 2000 Language play, language learning. Oxford: Oxford University Press. 
Cooper, W. & J. Ross 1975 World order. Chicago Linguistic Society, 11, 63-111. 
Fenk-Oczlon, G. 1989 Word frequency and word order in freezes. Linguistics, 27, 517-556. 
Fenk-Oczlon, G. 2001 Familiarity, information flow, and linguistic form. In J. Bybee & P. Hopper (Eds) Frequency and the Emergence of Linguistic Structure, pp. 431-448. Amsterdam: John Benjamins. 
Hoey, M. 1991 Patterns of lexis in text. Oxford: Oxford University Press. 
Hunston, S. 2002 Corpora in applied linguistics. Cambridge: Cambridge University Press. 
Landsberg (Ed). 1995 Syntactic iconicity and linguistic freezes: the human dimension. New York: Mouton de Gruyter. 
Landsberg, M. 1995 Semantic constraints on phonologically independent freezes. In Landsberg (Ed). Syntactic iconicity and linguistic freezes: the human dimension, pp. 65-78. 
McCarthy, M. 1990 Vocabulary. Oxford: Oxford University Press. 
877
McCarthy, M. 2001 Good listenership made plain: British and American non-minimal 
response tokens in everyday conversation. MS, University of Nottingham. 
McEnery, T. & Wilson, A. 1996 Corpus linguistics. Edinburgh: Edinburgh University Press. 
Moon, R. 1998 Fixed Expressions and Idioms in English. Oxford: Oxford University Press. 
Nation, P. 1982 Beginning to learn foreign vocabulary: a review of the research. RELC Journal, 13 (1), 14-36. 
Oakes, M. 1998 Statistics for corpus linguistics. Edinburgh: Edinburgh University Press. 
Pinker S. & Birdsong, D. 1979 Speaker's sensitivity to rules of frozen word order. Journal of 
Verbal Learning and Verbal Behavior, 18, 497-508. 
Schmitt, N. 2000 Vocabulary in language teaching. Cambridge: Cambridge University Press. 
Schmitt, N. & McCarthy, M. 1997 Vocabulary. Cambridge: Cambridge University Press. 
Sinclair, J. 1991 Corpus, concordance, collocation. Oxford: Oxford University Press. 
Stubb, M. 1995 Collocations and semantic profiles: on the cause of the trouble with quantitative studies. Functions of Language, 2 (1), 23-55. 
Tao, H. & McCarthy, M. 2001 Understanding non-restrictive which-clauses in spoken English, which is not an easy thing. Language Sciences, 23, 651-677. 
Wang, S.P. 2001a Integrating corpus-based and vocabulary learning approaches into a linguistics project. Paper presented in 46th International Linguistics Association, New York: New York University. 
Wang, S.P. 2001b Reduplication and repetition in applied linguistics. The Proceedings of 2001 International Conference on the Application of English Teaching, 200~222. Taipei: Crane Publishing Co. 
Wang, S.P. 2002a Corpus-based approaches and discourse analysis to reduplication and repetition, paper submitted to journal. 
878

Finding and evaluating sets of nearest neighbours Julie Weeds and David Weir School of Cognitive and Computing Sciences University of Sussex, Brighton, BN1 9QH, UK 
Abstract 
In this paper, we consider two applications of distributional similarity measures, probability estimation and prediction of semantic similarity. We investigate whether high performance in one application area is correlated with high performance in the other. This work also provides an evaluation of two state-of-the-art distributional similarity measures and introduces a variant of one. Further, we overcome statistical biases in the standard pseudo-disambiguation task and look at the effect of word and co-occurrence frequency on the performance of the measures. 
1. Introduction 
Two words are said to be distributionally similar if they appear in similar contexts. However, such a loose definition leads to as many questions as it answers. For example, we can ask what constitutes context and whether all contexts should be treated equally. Accordingly, many techniques have been developed which use corpora to determine sets of distributionally similar words. These range from standard geometric measures, such as the L1 Norm, to information-theoretic measures, such as the one proposed by Lin (1998). 
Since the loose definition of distributional similarity also leads to a problem in evaluating the measures, evaluation is often based on the ability of sets of nearest neighbours, determined using the measures under consideration, to perform a given application-based task (Lin 1998; Lee 1999). However, there are many potential applications for sets of distributionally similar words and it is not clear whether the same properties which make a measure useful in one application will make it useful in another. One distinction that can be made between applications is whether they make use of the property of distributional similarity directly or as a predictor of semantic similarity. For example, in a probability estimation task (e.g. Dagan et al. 1999), where the aim is to predict the probabilities of unseen word co-occurrences, the property of distributional similarity can be used directly. Several researchers have proposed a semantic class based approach to the PP-attachment ambiguity resolution problem (e.g. Resnik 1993). However, this is in fact a syntactic task which might benefit from the words in classes being syntactically or distributionally related rather than semantically related. On the other hand, distributional similarity is also proposed as a predictor of semantic similarity and has therefore been used in tasks such as automatic thesaurus generation (e.g. Lin 1998) and word-for-word glossing (Pantel and Lin 2000). Further, as a predictor of semantic similarity, there may be potential for using sets of distributionally similar words in tasks which currently rely on a hand-crafted thesaurus such as WordNet (Fellbaum 1998) e.g. malapropism correction (Budanitsky and Hirst 2001), text simplification (Carroll et al. 1998), collocation extraction (Pearce 2002) and text summarization (Silber and McCoy 2002). 
We investigate whether high performance on a pseudo-disambiguation task, a purely distributional task often used as an evaluation task for probability estimation (Dagan et al. 1999), is correlated with high semantic similarity. In order to do this, we compare the performance of three distributional similarity measures and one semantic similarity measure on a pseudo-disambiguation task and also the ability of the distributional similarity measures to predict semantic similarity, as defined by the semantic similarity measure. 
Two measures which are widely used (e.g. Schulte Im Walde and Brew 2002; Wiebe 2000) and have been shown to outperform other similarity measures are Lin's mutual information (MI) based measure (Lin 1998) and Lee's á-skew divergence measure (Lee 1999). We also present a variation of Lin's MI-based measure which we will theoretically justify and will show performs slightly better than the standard MI-based measure on the pseudo-disambiguation task. However, not only do we consider which measure is best for the given application, we also seek to establish what properties of the measure leads to the better performance. In particular, we look at the interaction of a word's frequency and the quality of its nearest neighbours. 
The rest of this paper is organised as follows. First, the four similarity measures used throughout our experiments will be introduced. We will then discuss obtaining sets of neighbours from the data extracted from the BNC for two disjoint sets of nouns. This is followed by a discussion of the pseudo- 
879
disambiguation task and results on this task for all four measures. We will then compare the sets of neighbours derived using the distributional similarity measures with those derived using the semantic similarity measure. Finally, the effect of low frequency events on all three distributional similarity measures will be considered. 
2. Similarity measures 
In this section, each of the four similarity measures used throughout this work will be introduced in turn. However, it should be noted that when using any of the three distributional similarity measures, the description (or distribution) of a word is taken to be a set of dependency relation co-occurrences and frequencies. Further, we consider only similarity between nouns and describe nouns in terms of the verbs with which they co-occur in the direct-object relation. Accordingly, we use the notation P(v|n) to refer to the probability of the verb in a direct-object co-occurrence pair being v given that the noun is n. However, it would be possible, with all three measures, to consider other relations, other parts of speech and even other types of context. 
2.1. The á-skew divergence measure (Lee, 1999) 
Given two nouns, n1 and n2, and the notation q(v) = P(v|n1) and r(v) = P(v|n2) where v ranges over all verbs, the distance between the two nouns can be computed as the distance between the two probability distributions as defined by the á-skew divergence measure, simasd(r,q). 
)).1(.||(),(qrqDqrsimasdá.-+= 
where 
‡”×=vvrvqvqrqD)()(log)()||( 
The value of the alpha-skew divergence measure is zero if the two probability distributions are identical and increases positively as the distributions become less similar. The parameter á ranges between 0 and 1 and controls the extent to which the measure approximates D (the Kullback-Leibler divergence measure). When á is close to 1, the measure closely approximates the Kullback-Leibler divergence measure whilst avoiding the problem of zero probabilities associated with using the Kullback-Leibler divergence measure in NLP. Accordingly, we use á=0.99. Further, the measure is asymmetric and it defines how well the probability distribution associated with n2 approximates the probability distribution associated with n1. Consequently it can be used to determine how fit n2 is as a neighbour for n1. 
2.2. The MI-based measure (Lin, 1998) 
The MI-based measure is based on Lin's theorem that the similarity between two objects is defined to be the amount of information in the features they have in common divided by the sum of the information in the features of each object. If the only grammatical relation being considered is the direct-object relation dobj, n is a noun and v is a verb, v can be considered to be a feature of the noun n if I(n,v), the mutual information between n and v, is positive. The mutual information between two words occurring in the direct-object relation can be computed using the formula (Hindle 1990): 
)()|(log)()(),(log),(nPvnPnPvPnvPvnI== 
In other words, v is a feature of n if our expectation of seeing n increases on seeing v. Accordingly, if T(n) is the set of verbs v that are features of n, the similarity between the two nouns, n1 and n2 can be calculated as: 
‡”‡”¸¸‡”¿¸++=)()(21)()(21211221),(),()),(),((),(nTvnTvnTnTvmivnIvnIvnIvnInnsim 
The value of the MI-based measure lies in the range [0,1] where a value of 1 indicates that the two 
880
nouns are identical (Lin 1997). 
2.3. Variation on the MI-based Measure 
Lin's MI-based measure considers two nouns to be identical if they have exactly the same features. It is possible for two nouns to have different probability distributions yet be considered identical. This is because two nouns are considered to completely share a feature regardless of the extent to which each noun has that feature (i.e. the value of the mutual information between the noun and the feature verb). The difference in the extent to which each noun has a particular shared feature can be considered so that only nouns with identical probability distributions are considered to be identical. To do this, the similarity between two nouns, n1 and n2, can be defined as: 
‡”‡”‡”¸¸¿¸+--+=)()(21)()(2121211221),(),()),(),(),(),((),(nTvnTvnTnTvmivnIvnIvnIvnIvnIvnInnsimdiff 
The expression on the numerator of the equation can be rewritten in the following way: 
),min(2BABABA×=--+ 
and thus, this variation has parallels with the measure defined in Hindle (1990): 
‡”¿¸=)()(2121)),(),,(min(nTnTvHindlevnIvnIsim 
2.4. A WordNet Based Measure (Lin, 1997) 
There are many different measures of semantic similarity (see Budanitsky (1999) for a thorough review). Here, we use a measure which is based on the hyponymy relation (as defined by WordNet), proposed by Lin (1997), which has been shown to perform fairly well compared to other WordNet based measures on a spelling correction task (Budanitsky and Hirst 2001). 
If S(n) is the set of senses of the noun n in WordNet, sup(c) is the set of (possibly indirect) superclasses of concept c in WordNet then the similarity between n1 and n2 can be calculated as: 
ÿÿ.....+=¿¸¸È¸))()(log()(log2maxmax),(21)sup()sup()()(21212211cPcPcPnnsimcccnScnScwn 
In other words, the commonality (Lin 1997) of two concepts (or synsets) is defined as the maximally specific common superclass of those concepts. The similarity of two nouns is then defined as the similarity between their most similar senses. P(c) is the probability that a randomly selected noun refers to an instance of c. These probabilities P(c) are estimated by the frequency of concepts in SemCor (Miller et al. 1994), a sense-tagged subset of the Brown corpus, noting that the occurrence of a concept refers to instances of all the superclasses of that concept. 
3. Finding Sets of Neighbours 
Much previous evaluation of similarity measures has involved determining neighbours for the most frequently occurring nouns in a particular corpus (Lin 1998; Lee 1999). However, since we wish to investigate the effects of word frequency on similarity, we determine neighbours of 1000 high frequency nouns and also of 1000 low frequency nouns. 
2,852,300 lemmatised (noun,verb) direct-object pairs were extracted from the BNC using a parser developed by Briscoe and Carroll (1995, 1996). Having discarded all co-occurrence pairs where the noun does not occur in WordNet, we constructed two sets denoted: nounshighfreq made up of the pairs where the noun is one of the 1000 most frequently occurring nouns, and nounslowfreq made up of the pairs where the noun occurs between 70 and 120 times in the data set1. 
For each noun, we randomly select 80% of the available data as training data and set aside the other 20% as test data for the pseudo-disambiguation task. We compute the similarity between each pair of nouns using each of the four similarity measures under consideration. For each noun and similarity measure, a set of ranked neighbours is constructed. 
1 These frequencies correspond to ranks of 3001 to 4000. 
881
4. Pseudo-Disambiguation Experiment 
Pseudo-disambiguation experiments have become a standard method of evaluating a technique's ability to perform probability estimation (e.g. Pereira et al. 1993, Lee 1999 and Clark and Weir 2002). In the current context, we may use a word's neighbours to decide which of two co-occurrences (one of which occurred in the test data and the other of which did not) is more likely. However, various approaches can be taken to the way training/test sets are constructed and the way in which neighbours vote on their preferred co-occurrence and this will now be discussed. 
4.1. Test Set Construction 
In this section, we will first describe how the test set for our experiments was constructed and then explain the rationale behind it. Having set aside 20% of the data for each noun as test data, we converted each noun-verb (n,v1) pair in that data into a noun-verb-verb (n,v1,v2) triple where P(v1) is approximately equal to P(v2) over all of the training data and (n,v2) has not been seen in the test or training data. We selected ten test instances2 for each noun as follows. Whilst more than ten triples remained, duplicate triples were discarded. If there were still more than ten triples, after all duplicates had been discarded, ten triples were selected at random from the remaining triples. This process resulted in two sets of 10000 test triples. Each set of test triples was then split into five disjoint sets (containing 2 triples for each noun) so that five-fold cross validation could be performed. 
We now consider approaches taken by earlier researchers. In order to ensure that the test data is unseen, both Pereira et al. (1993) and Clark and Weir (2002) randomly select a portion of the co-occurrences as test data and then delete any instances of these co-occurrences from the training data. However, we feel that this approach is inappropriate for the large-scale evaluation of distributional similarity measures since completely removing co-occurrence types from the training data distorts the data used by the measures to determine similarity. A different approach, taken by Lee (1999), is to randomly select a portion of the co-occurrences as test data and then delete any instances of co-occurrences from this test data which also occur in the training data. The next step, in either approach, is to replace each remaining noun-verb (n,v1) pair in the test data with a noun-verb-verb triple (n,v1,v2) where v2 is selected such that it is of approximately equal probability to v1 in the training data. It is then standard to split test sets into five disjoint sets for cross-validation purposes. Neighbours of n are then required to decide which was the original co-occurrence which, by construction, is always (n,v1). 
Although discarding test pairs which appear in the training data ensures that the test data is unseen and overcomes the problem of removing items of training data, we argue that there are remaining problems to be considered. 
First, there is no guarantee that the inequality P(v1|n)>P(v2|n) holds for population probabilities (as opposed to sample probabilities) even if we introduce the restriction that (n,v2) is unseen in the training data. For (n,v1) not to have been seen in the training data (and thus remain in the test data), our estimate of the probability of this co-occurrence must be very small and is therefore not much greater than that of the completely unseen, in either training or test data, (n,v2). Accordingly, if we were to see a lot more data, (n,v2) may become more probable/frequent than (n,v1), and, even if it does not, we are still expecting neighbours to distinguish between events of very similar probabilities. 
Second, since the joint probability of n and v1 must be small (for the co-occurrence pair not to have appeared in the training data), the unigram probability of the verbs to be disambiguated tends to decrease as the unigram probability (or frequency) of the noun increases3. This in turn limits the number of neighbours the verbs may have occurred with and in turn may influence the performance results. 
Third, the test set, and therefore the average performance, is biased towards nouns which occur with many different verbs (typically high frequency nouns). In the extreme case, after discarding those pairs which also occur in the training data, there may be no pairs remaining for a very selective noun. Accordingly, we are not evaluating over the whole set of nouns. When 1000 random nouns were selected and a test set constructed in the way described by Lee (1999), we found that only 60% of the nouns appeared in the final test set. 
Our method of test set construction overcomes all three of these problems. We argue that there is no need to discard any test data on the basis of it having been seen in the training data (or vice versa). In the machine-learning paradigm, it is still unseen data which is not used in the training process. Further, by not discarding seen noun-verb pairs we can be more confident that the correct choice verb 
2 Ten being less than the minimum number (fourteen) of triples in the test data for any noun. 
3 In our experiments, we found significant negative correlation (Pearson's product-moment correlation coefficient between -0.12 and –0.18) between noun frequency and verb frequency. 
882
(v1) is more plausible than the incorrect choice verb (v2) and we do not introduce any statistical biases into the test set i.e. there is no significant negative correlation between noun and verb frequency. Further, we select a proportion of the data for each noun as test data which means that each noun is treated fairly as to how much data is lost for training purposes whilst allowing us to control the minimum number of test instances for each noun. Accordingly, we were able to remove the bias towards less selective/high frequency nouns. 
4.2. Nearest Neighbour Voting 
As mentioned in the previous section, the task is for the nearest neighbours to decide which of (n,v1) and (n,v2) was the original co-occurrence. In both Pereira et al. (1993) and Lee (1999), the k-nearest neighbours, where k is a parameter which is optimised, of the noun n vote as to which is the most likely co-occurrence (each neighbour m votes for the co-occurrence according to which verb it co-occurs with most in the training data i.e. by considering whether P(v1|m)>P(v2|m)). Performance is then measured as error rate. 
ÿ....+=2)(##1tiesofchoicesincorrectofTrateerror 
where T is the number of test instances and a tie results when the neighbours cannot decide between alternatives. 
However, in earlier work (Weeds 2002), we recognised that giving each neighbour a single vote does not allow us to distinguish between cases where a neighbour occurs with each verb approximately the same number of times and where a neighbour occurs with one verb significantly more times than the other. We also observed that a single vote per neighbour appears to introduce a bias towards v1. When all neighbours are considered, and therefore the two verbs have appeared approximately the same number of times, v1 tends to win more votes by virtue of the fact that it has occurred with more different nouns (whereas v2 wins its votes by a larger amount). Thus, we give each neighbour m a vote which is equal to the difference in frequencies of the co-occurrences (m,v1) and (m,v2). Performance is still measured as error rate. 
4.3. Results 
Table 1 shows the optimal error rates for each similarity measure when used to determine neighbours for high and low frequency nouns. This optimal error rate is given in terms of the mean error rate at the optimal value of k averaged over the five test sets used for the five-fold cross validation. The standard deviations of these means, not given, were of the order of 0.01. 
Measure Noun Frequency high low k opt. error rate k opt. error rate Asd 30 0.230 60 0.192 Mi 50 0.193 100 0.181 midiff 40 0.192 130 0.176 Wn 20 0.295 40 0.294 
Table 1 
We make the following observations based on Table 1. 
1. All of the distributional similarity measures significantly outperform the WordNet based measure on the pseudo-disambiguation task. This is regardless of the frequency of the original noun. 
2. Both MI-based measures significantly outperform the á-skew divergence measure on this task. 
3. The use of the midiff measure leads to a small improvement over the standard MI-based measure for low frequency nouns. 
4. All of the distributional similarity measures perform best for low frequency nouns whereas there is no significant difference when using the WordNet based measure. 
We will return to these observations later. 
5. Semantic Similarity 
The results in the previous section suggest that semantic similarity is not necessarily the best indicator of neighbours’ abilities to perform probability estimation, which could be considered a purely syntactic 
883
task. However, many potential applications of nearest neighbour sets in NLP are in the semantic domain and therefore would require words to be semantically similar. We evaluated how well each distributional similarity measure predicts semantic similarity by measuring the similarity between neighbour sets generated using the distributional similarity measures and neighbour sets generated using the WordNet based measure. 
In order to compare two neighbour sets, we convert each neighbour set so that each neighbour is given a score based on its rank order. We do not use the similarity directly in our comparison since 1) this would require us to convert the á-skew divergence measure distances into similarities and 2) many tasks, including the earlier pseudo-disambiguation task, make no use of the actual similarity scores. Neighbours are given a rank score of (k+1)-rank where k is the number of neighbours being considered and rank is the neighbour's rank in the neighbour set4. Having performed this transformation, we use the same calculation as Lin (1998) (except for simplifications due to the use of ranks) to compute the similarity between two neighbour sets. Supposing two neighbour sets for the same word, w, are represented by two ordered sets of words [wk….w1] and [wk'….w1'] , then their similarity is defined as: 
‡”‡”==×kiwwijiji12' 
This is similar to calculating a rank order correlation coefficient except for the fact that it does not require the same words to be in both sets. We could in fact calculate a rank order correlation coefficient based on the entire neighbour sets (i.e. where k=2000) 5 but we cannot do so directly for smaller subsets. Our method of scoring, however, means that we are approximating a product-moment coefficient between the ranks where all neighbours not within a given rank distance of the noun tie on the rank zero. 
In our experiments, the k nearest neighbours of each noun, computed using each of the distributional similarity measures, were compared with the k nearest neighbours of the noun according to the WordNet based measure (with no restrictions on the frequencies of the neighbours). The mean similarity between neighbour sets for k=200 for both the high frequency nouns and for the low frequency nouns are given in Table 2. The standard deviations of the means, not given, were all of the order of 0.1. k=200 was chosen in order to consider more than just closest neighbours whilst avoiding going into the tail of dissimilar words (which may be ranked in any order). However, similar results were obtained in experiments considering 50 and 100 nearest neighbours. 
Measure Noun Frequency high low asd 0.290 0.270 mi 0.307 0.210 midiff 0.303 0.172 
Table 2 
From these results we make the following observations: 
1. In terms of predicting semantic similarity, the MI-based measures outperform the á-skew divergence measure for high frequency nouns. 
2. However, the á-skew divergence measure performs significantly better than the MI-based measures for low frequency nouns. 
3. Similarity with the WordNet based measure is, in all cases, greater for high frequency nouns than low frequency nouns. 
Returning to the observations made in Section 4, it is interesting to note the differences with respect to high and low frequency nouns. In particular, semantic similarity is in all cases lower for low frequency nouns whilst performance on the pseudo-disambiguation task is higher for low frequency nouns. Further, there is a significant drop in performance of the MI-based measures for low frequency nouns in terms of semantic similarity. 
6. Dependence on Low Frequency Events 
We now explore why the MI-based measures perform poorly at the semantic evaluation for low 
4 Using a reverse-order rank score means that all words outside the neighbour set can be given the score 0 
5 we do not, however, wish to calculate the correlation over all neighbours since, generally, we are not concerned about the order in which a measure ranks dissimilar words. 
884
frequency nouns. We predict that it is because low frequency nouns are made too similar to each other. This would have less effect on the pseudo-disambiguation task since low frequency neighbours will have occurred with fewer verbs than their higher frequency counterparts. Further, we hypothesise that the unduly high similarity between low frequency nouns is because of the large influence of low frequency co-occurrences on the MI-based measures. We investigated the first of these predictions by comparing sets of neighbours where the neighbours are restricted in frequency and the second using a Bayesian Estimation technique. 
6.1. Neighbour Frequency 
For each noun and similarity measure, two new sets of ranked neighbours were constructed; one set where neighbour nouns are restricted to being in the nounshighfreq set and one set where neighbour nouns are restricted to being in the nounslowfreq set. We then used the neighbour set comparison technique described in Section 5 to compare the similarity between neighbour sets for a single measure. By comparing the complete neighbour set (where neighbours are of any frequency) to the restricted frequency neighbour sets, we get an estimate of how the complete neighbour set is made up of high and low frequency nouns. The results, in terms of mean similarity of the complete neighbour set to the high frequency neighbour set and to the low frequency set, for each of the four measures, are given in Table 3. Standard deviations of the means, not given, were all to the order of 0.1. 
Measure Noun Frequency high low sim high sim low sim high sim low Asd 0.992 0.021 0.956 0.102 Mi 0.931 0.161 0.459 0.748 midiff 0.950 0.122 0.209 0.915 Wn 0.845 0.330 0.798 0.392 
Table 3 
From the results in Table 3, we make the following observations: 
1. From the results for the WordNet based measure, it can be seen that both high and low frequency nouns tend to be more semantically similar to high frequency nouns than low frequency nouns. 
2. There is also a very strong tendency for all of the distributional similarity measures to select high frequency nouns as neighbours of high frequency nouns. 
3. The same tendency is seen for the alpha-skew divergence measure for low frequency nouns but it is reversed for the MI-based measures i.e. these measures have a greater tendency to select low frequency nouns as neighbours of low frequency nouns. 
This reversal in observation 3 explains the poor performance of the MI-based measures for low frequency nouns on the semantic evaluation task. We will now investigate why the MI-based measures tend to select low frequency neighbours for low frequency nouns. 
6.2. Bayesian Estimation 
Our second hypothesis was that the MI-based measures cause low frequency nouns to be overly similar to each other due to the large influence of low frequency co-occurrences of low frequency words. 
Let us consider the impact of a singular co-occurrence (i.e. a co-occurrence which is seen only once in the corpus) on mutual information and hence the similarity between nouns. From the definition of mutual information, a singular co-occurrence of a low frequency noun and a low frequency verb will lead to higher mutual information between that noun and that verb than if either noun or verb had been of a higher frequency. However, singular co-occurrences are a fairly unreliable source of information. It could be that that co-occurrence should not have occurred (it was due to an error in the corpus) or that it has a much lower probability than (1/size of corpus) but, because of discretization in the data, we either see such low probability events once or zero times in the corpus. As a direct consequence, events with the same population probabilities may be assigned either very high mutual information scores or negative mutual information scores. Further, if two low frequency nouns both have a singular co-occurrence with the same low frequency verb, this will lead to high similarity between those nouns. 
The dependence of such measures on low frequency events has been considered before. Lee (2001), based on a suggestion made in Katz (1987), discards all singular co-occurrences from the training data. 
885
Discarding all of these co-occurrences leads to a fairly small reduction in data set size in terms of indistinct co-occurrences (12% for our data) but a much larger reduction in terms of distinct co-occurrence types. 66% of co-occurrence types (i.e. distinct co-occurrences), which appear in our BNC co-occurrence data, occur only once. Over the 2000 nouns considered in our experiments, 57% of co-occurrence types occur only once. Further, the description of a low frequency noun may not contain many non-singular co-occurrences – so if we discard all singular co-occurrences, we will be left with the task of trying to determine similarities between low frequency nouns based on their co-occurrences with such frequently occurring verbs as have and make. It is fairly obvious, that these singular co-occurrences have to have some influence when we are considering low frequency nouns – however, it is our hypothesis that they have too much influence. 
The approach taken here is to tackle the root of the problem – that is that we do not have the true population probabilities of the co-occurrences or the words, only corpus-based maximum likelihood estimates. We use a Bayesian Estimation technique to smooth the sample probability distributions. 
6.3. A Uniform Prior 
Bayesian Estimation allows us to combine a prior distribution (what was expected before any data was observed) with an observed distribution to form a posterior distribution. In the simplest case, where the prior distribution is a uniform distribution, Bayesian Estimation reduces to add1 smoothing (see e.g. Friedman 1999). In our case, a uniform distribution has been assumed since, before any data is seen, we have no idea which noun-verb co-occurrences are more likely than others. However, rather than adding 1 to every possible noun-verb combination, we add a small fraction X so that we can control the total amount of probability mass which is assigned to unseen events. For our data, we use X=0.011 since adding this amount to every possible noun-verb combination is consistent with a total probability mass of 0.118 being assigned to previously unseen events. This value of 0.118 is Po as calculated using simple Good-Turing techniques (Gale and Sampson 1996) and is based on the number of hapax legomena in the data. 
Accordingly, having added 0.011 to every noun-verb combination, we recalculated the similarities between each pair of nouns using each distributional similarity measure. We then compared the neighbour sets generated for each measure (as in Section 6.1). 
6.4. Results 
Measure Noun Frequency high low sim high sim low sim high sim low Asd 0.992 0.022 0.981 0.050 Mi 0.953 0.081 0 1 midiff 0.9996 0.001 0 1 
Table 4 
Comparing the results in Table 4 with those in Table 3, we can observe that all of the tendencies seen in Table 3 have been exacerbated. However, the tendency of the MI-based measures to select low frequency neighbours for low frequency nouns has been increased by far the most significantly. In fact, the MI-based measures now select neighbours for low frequency nouns exclusively from the low frequency noun set. Accordingly, we can conclude that the Bayesian estimation has had most effect on the MI-based measures for low frequency nouns. 
The effects of the Bayesian smoothing on the similarity measures can also be seen by examining the results for the new neighbour sets on the pseudo-disambiguation task (Table 5) and the semantic evaluation against WordNet (Table 6). 
Measure Noun Frequency high lowMeasure Noun FrequencyK opt. error rate kopt. error ratehigh lowasd 30 0.228 400.199asd0.292 0.268mi 20 0.261 3000.313mi0.228 0.058midiff20 0.266 7000.389midiff0.226 0.047Table 5 Table 6 
886
By comparing the results in Tables 5 and 6 with their earlier counterparts (Tables 1 and 2), we can see again that the Bayesian Estimation has had a much greater (negative) effect on the performance of the MI-based measures than the á-skew divergence measure. 
6.5. Discussion 
In order to be able to understand the effects seen in Section 6.4, we need to consider carefully what is happening when we perform the Bayesian Estimation. Rather than reducing the impact of low frequency events, as might intuitively be expected, we have in fact created a large number of very low frequency events. The results for a measure which relies heavily on low frequency events will therefore be more affected than those for a measure which does not. Even though the frequency of co-occurrence added for unseen events (0.011) is very small, it is considered significant by the MI-based measures particularly when the nouns and verbs under consideration are low frequency ones. Let us consider an example. Suppose n and v have each occurred 60 times in the training data6 which contains 1596798 (indistinct) noun-verb co-occurrences but zero co-occurrences of n and v. There are a total of 2000 nouns in the data and 9587 verbs, so after adding 0.011 to every noun-verb combination, our frequency estimates are: 
f'(n1,v1) = 0.011 
f'(n1) = 60 + 0.011 × 9587 = 165.5 
f'(v1) = 60 + 0.011 × 2000 = 82 
1807712)95872000(011.01596798),(,=×~+=Œ‡”nvnvf 
The estimated mutual information between n1 and v1 (see Section 2.2) is 0.551. Thus, a low frequency verb which has not appeared with our low frequency noun has become a feature of that noun. Further, we can see that all low frequency verbs will become features of all low frequency nouns and hence low frequency nouns will tend to become more similar to each other. Further, in order for new features not to be created by the smoothing process, it was calculated that the parameter X would have to be very small (less than 6×107 if we allow for nouns and verbs only occurring once). In other words, it would have to be so small that it would have no effect on the mutual information scores. 
7. Conclusions and Further Work 
All of the distributional similarity measures perform the pseudo-disambiguation task better than the semantic similarity measure. Further, Lin's MI-based measure performs better than the á-skew divergence measure and the variant of Lin's MI-based measure presented here leads to further small improvements for low frequency nouns. 
One might have expected that the á-skew divergence measure would therefore perform the best at the semantic evaluation. However, this is not the case. The MI-based measures’ neighbour sets for high frequency words are closer to the WordNet derived neighbour sets than the á-skew divergence measure's neighbour sets are. 
The MI-based measures do, however, perform poorly at the semantic evaluation for low frequency nouns. We have explored this thoroughly and conclude that it is because low frequency nouns are made overly similar to each other due to the large influence of low frequency co-occurrences of low frequency events on the MI-based measures. This effect is a lot less for the á-skew divergence measure, which is why its neighbours of low frequency nouns are closer in terms of semantic relatedness. 
Our final observation is that the frequency of a neighbour influences how much weight it has in the pseudo-disambiguation task. Low frequency neighbours will have occurred with fewer verbs and will therefore contribute to fewer decisions. One consequence of this is that more neighbours will generally need to be considered in order to reach a decision. A second consequence is that, in order to obtain high performance on the pseudo-disambiguation task, we find we can sacrifice some semantic similarity in favour of finding lower frequency potential neighbours, safe in the knowledge that these neighbours, if wrong, will affect less test instance outcomes. 
In the future, we plan to look at ways of reducing the effects of word frequency on the distributional similarity measures. For example, Bayesian estimation with a non-uniform or hierarchical prior could be used and thus we could eliminate the problem of introducing extra features into the MI-based measures. We also intend to look at other applications of distributional similarity measures and how the frequency effects observed affect these applications. 
6 which corresponds to n occurring 75 times in the complete data set (training + test data) 
887
8. Acknowledgements 
We would like to thank John Carroll for the use of his parser and acknowledge that this work is supported by an Engineering and Physical Sciences Research Council studentship to the first author. 
References 
Briscoe E, Carroll J 1995 Developing and evaluating a probabilistic lr parser of part-of-speech and punctuation labels. In Proceedings of 4th ACL/SIGDAT International Workshop on Parsing Technologies, pp 48—58 
Budanitsky A, Hirst G 2001 Semantic distance in WordNet: an experimental, application-oriented evaluation of five measures. In Proceedings of NAACL-01 
Budanitsky A 1999 Lexical Semantic Relatedness and its Application in Natural Language Processing. Unpublished PhD thesis, University of Toronto 
Carroll J, Briscoe E 1996 Apportioning development effort in a probabilistic lr parsing system through evaluation. In Proceedings of ACL/SIGDAT Conference on Empirical Methods in Natural Language Processing, pp 92—100 
Carroll J, Minnen G, Canning Y, Devlin S, Tait J 1998 Practical simplification of English text to assist aphasic readers. In Proceedings of AAAI-98 Workshop on Integrating Artificial Intelligence and Assistive Technology 
Clark S, Weir D 2002 Class-based probability estimation using a semantic hierarchy. Computational Linguistics 28(2) 
Dagan I, Lee L, Pereira F 1999 Similarity-based models of word co-occurrence probabilities. Machine Learning 34(1-3) 
Fellbaum C (ed) 1998 WordNet: an electronic lexical database. Cambridge, Massachusetts, MIT Press 
Friedman N, Singer Y 1999 Efficient Bayesian parameter estimation in large discrete domains. Advances in Neural Information Processing Systems 11. 
Gale W, Sampson G 1996 Good-Turing estimation without tears. COGS Research Paper, Brighton 
Hindle D 1990 Noun classification from predicate-argument structures. In Proceedings of ACL-90, pp 268—275 
Katz S 1987 Estimation of probabilities from sparse data for the language model component of a speech recogniser. IEEE Transactions on Acoustics, Speech and Signal Processing, ASSP-35(3), pp 400—401 
Lee L 1999 Measures of distributional similarity. In Proceedings of ACL-99 
Lee L 2001 On the effectiveness of the skew divergence for statistical language analysis. Artificial Intelligence and Statistics:pp 65—72 
Lin D 1997 Using syntactic dependency as local context to resolve word sense ambiguity. In Proceedings of ACL/EACL-97. pp 64—71 
Lin D 1998 Automatic retrieval and clustering of similar words. In Proceedings of COLING-ACL'98 
Miller G, Chodorow M, Landes S, Leacock C, Thomas R 1994 Using a semantic concordance for sense identification. In Proceedings of ARPA Human Language Technology Workshop 
Pantel P, Lin D 2000 Word-for-word glossing of contextually similar words. In Proceedings of ANLP-NAACL'00. pp 78—85 
Pearce D 2002 A comparative evaluation of collocation extraction techniques. In Proceedings of LREC'02 
Pereira F, Tishby N, Lee L 1993 Distributional clustering of similar words. In Proceedings of ACL-93 
Resnik P 1993 Selection and Information: A Class-Based Approach to Lexical Relationships. Ph.D. thesis, University of Pennsylvania 
Schulte Im Walde S, Brew C 2002 Inducing German semantic verb classes from purely syntactic subcategorization information. In Proceedings of ACL-02. 
Silber H, McCoy, K 2002 Efficiently computed lexical chains as an intermediate representation for automatic text summarization. In Computational Linguistics 28(4) 
Weeds J 2002 The reliability of a similarity measure. In Proceedings of the 5th UK Special Interest Group for Computational Linguistics (CLUK5) 
Wiebe J 2000 Learning subjective adjectives from corpora. In Proceedings of AAAI'00. 
888

Using learner corpora to examine L2 acquisition of tense-aspect markings 
David Wible 
Ping-Yu Huang 
English Department: Graduate Programs, Tamkang University, 
Tamsui, Taipei Hsien, Taiwan 
1. Introduction 
This paper describes an application of learner corpus data to research on L2 learners’ acquisition of tense-aspect morphology. In the past twenty years, a series of studies in the fields of both first (L1) and second (L2) language acquisition have shown that, in beginning stages, language learners’ use of temporal markings are largely determined by lexical (or inherent) aspect (e.g. Antinucci and Miller 1976; Bardovi-Harlig and Reynolds 1995; Bloom, Lifter and Hafitz, 1980; Shirai and Anderson 1995). More specifically, these studies suggest that language learners tend to use past and perfective markings on verbs that denote events (e.g. die & jump) while adding progressive markings to semantically dynamic and durative verbs (e.g. play & swim). This sort of finding has been expressed in various forms under the name of The Aspect Hypothesis. Though most of the previous studies examining L1 and L2 learners’ use of tense-aspect markings generated results consistent with the claims of the Aspect Hypothesis, none of the previous researchers have incorporated L2 learner corpora of unrestricted written texts by L2 learners into their research. The present research serves as an initial study incorporating the data from an L2 learner corpus and aims to see if lexical aspect plays a role in L2 learners’ use of tense-aspect markings in these learners’ unrestricted written texts. More specifically, the data used here was extracted from a machine-readable corpus containing over one and a half million words freely written by English L2 learners in Taiwan, named English Taiwan Learner Corpus (English TLC). In addition to adding learner corpus data to the research literature on the Aspect Hypothesis, we present a novel use of corpus data to examine learners’ use of tense marking. Specifically, we extract tokens of obligatory contexts for non-past forms of verbs to identify cases of overuse of past tense forms. Our results show that, contrary to the predictions of the Aspect Hypothesis, the lexical aspect of the verbs wields virtually no influence on the overuse of past tense marking. The factor which does skew the overuse of past forms is whether the verb has a regular or irregular past form. Irregular past forms are far more likely to be overused than regular past forms. 
One point of interest that we address concerns the rather persistent finding in the Aspect Hypothesis literature that past tense forms used by learners appear on telic (or eventive) verbs at a much higher rate than on atelic verbs (states and activities). We suggest that this finding may be of limited interest if the same skewing of temporal markings is found not only in learner production of the target language but in standard written texts in the target language. In fact, in the present study, we have found the so-called standard or target texts show the same tendency of past tense markings to be used 
889
with eventive verbs rather than atelic verbs. This would seem to suggest that this facet of learner usage is a straightforward reflection of the target language rather than a distinctive property of learners’ interlanguage or the acquisition process. Hence, we propose investigating, instead, certain usages that are distinctive of L2 learners’ temporal markings. Thus, in addition to examining L2 learners’ use of temporal markings in a learner corpus of written texts, we also introduced the use of obligatory contexts in our research on our learner corpus (See Brown 1973). By looking for overuse of past tense forms in obligatory contexts for base verb forms (e.g. He should made it), we could see if the verbs learners are prone to overusing past markings on are eventive verbs, congruent with the claims of the Aspect Hypothesis. 
2. English TLC 
The research data used in the study was extracted from a L2 learner corpus named English TLC. What is novel about this corpus is its “data capture” (a term used by Granger in 1998) procedures. Unlike other learner corpora whose data capture methods are mainly scanning, keyboarding, or downloading from disks, the learner data inputted to English TLC can be seen as a byproduct of a teacher-student interaction from an online writing instruction environment. More specifically, we have an online EFL writing instruction platform, IWiLL, from which students’ essays submitted to teachers for corrections are, with these students’ permissions, automatically copied into English TLC. As students and teachers use the writing environment, the corpus grows by itself. Through this teacher-student interaction, nearly two million words of running texts have been archived into English TLC.1 Currently, English TLC can be searched online. A researcher can use key-word-in-context (KWIC) searches to extract a list of sentences that contain a keyword or a phrase from students’ essays. In this study, for instance, modal auxiliaries such as would and should were used as keywords to extract students’ overuse of past tense markings in obligatory contexts for base verb forms. 
3. Brief background on temporal morpheme acquisition research 
The study reported here is essentially intended to contribute to the literature on L2 acquisition of temporal morphology, specifically with respect to the predications of the Aspect Hypothesis. In this section, therefore, some basic relevant studies on temporal morpheme acquisition, particularly concerning corpus research, will be reviewed. 
The term “aspect” as it appears in the name of the Aspect Hypothesis refers basically to lexical rather than grammatical aspect.2 According to Comrie (1976), lexical aspect refers to “inherent aspectual (i.e. semantic aspectual) properties of various classes of lexical items” (p. 41). For example, the verb know is inherently stative while die, which denotes an event which takes place instantly, is semantically a punctual verb. Moreover, to clarify the differences in lexical aspectual meaning entailed 
1 See Wible et. al (2001) for a detailed description of the on-line writing environment and the corpus design. 
2 A basic distinction assumed in this literature concerns the semantic categories of lexical or verbal aspect. Generally, there are two types of aspect. The first one is grammatical aspect, which refers to “ways of viewing the internal temporal constituency of a situation” (Comrie 1976, p.3). The second one is lexical aspect. 
890
in verbs, Vendler (1967) proposed a classification of lexical aspect, dividing verbs into four semantic categories: states, activities, accomplishments, and achievements. According to this classification, stative verbs such as know, want, and love denote non-dynamic states of affairs which entail no inherent change or end-point in their meaning. Activity verbs, like states, also denote an eventuality that persists over time with no inherent endpoint entailed though they do entail change and are therefore semantically dynamic. In English, play, run, and swim are activity verbs. Both accomplishments and achievements entail inherent endpoints (i.e. telic, a term used by Verkuyl 1996, p. 323, inter alia) while activities and states do not (i.e. atelic). They differ in that accomplishments are durative while achievements are punctual or instantaneous. In English, accomplishment predicates includes run a mile and make a card while jump, win, and die belong to achievement verbs. 
To further describe the semantic aspects inherent in verbs, Dowty (1979) made clear the differences among the four categories of verbs. First, states differ from other types of verbs in that non-states can occur in progressive, be used as imperatives, and appear in pseudo-cleft constructions. Next, activities and accomplishments are distinguished by the adverbials they take. Specifically, accomplishment verbs can take in adverbials (e.g. He ran a mile in an hour) but not adverbials with for (e.g. *He ran a mile for an hour) while activity verbs can take for adverbials (e.g. He ran for an hour) but not in adverbials (e.g. *He ran in an hour). Finally, although both accomplishments and achievements entail inherent endpoints, achievements differ from accomplishments in that they do not take for adverbials (e.g. *John died for two days) since they occur punctually. 
From the early 1980s, a number of studies have examined patterns in language learners’ acquisition of temporal markings, and most of them supported the Aspect Hypothesis. Basically, according to Anderson and Shirai (1996), the Aspect Hypothesis consists of four central claims: 
1. Learners initially use past and perfect markers with telic verbs and then spread these markers to activity and state verbs. 
2. Imperfect markings (in languages such as Russian and Spanish) are first added to atelic verbs, and extended to accomplishments and achievements later. 
3. Progressive markers are first restricted to activity verbs and then spread to accomplishments and achievements. 
4. Progressive markers are not incorrectly used with stative verbs. 
We focus on the first claim, and specifically on past tense marking. 
From the early 1970s to 1980s, most studies concerning the correlation between learners’ use oftemporal markers and the inherent aspect of verbs were restricted to L1 acquisition research. For instance, Antinucci and Miller (1976) and Bronckart and Sinclair (1973) observing Italian L1learners’ and French L1 learners’ oral production, respectively, found children tended to use tense-aspect markers to mark lexical aspect rather than express temporal notions. Bloom, Lifter and Hafitz (1980) found that in English-speaking children's spontaneous speech, past tense inflections were mostly added to accomplishments and achievements while the progressive marker –ing was predominantly used with activities. More recently, since the 1990s, second language learners’ acquisition of temporal morphemes 
891
was investigated by several researchers. For example, Bardovi-Harlig and Reynolds (1995) used a cloze test showing ESL students were “sensitive to lexical aspectual class with respect to tense use” in these learners’ elicited written responses (p. 119). In addition, in Lee's (2001) research on 2 Korean children's acquisition of English as an L2, the subjects’ unrestricted oral productions also demonstrated a strong connection between lexical aspect and temporal inflections. 
Some researchers have used corpus data to test the Aspect Hypothesis. For instance, in studiesdone by Anderson and Shirai (1994) and Shirai and Anderson (1995), the data extracted from CHILDES showed that when children learn English as an L1 they first use past tense inflections with achievements and progressive markings with activity verbs. In addition, Robison (1995) working with a corpus with over 3,600 predicates found in 26 ESL Puerto Rican college students’ oral production, PAST markings (including past tense and perfect inflections) were used to describe punctual events and –ing was associated with activity verbs, as the Aspect Hypothesis predicts. More recently, Bardovi-Harlig (1998) constructed a specific and restricted sort of corpus with both spoken and written narratives gathered through a story re-telling procedure. She reports that the verbal morphology in L2 learners’ interlanguage was indeed initially determined by lexical aspect. 
Based on the review above, generally, most studies concerning L1 and L2 acquisition of temporalmorphemes confirmed the Aspect Hypothesis. However, most of the data examined in this and other previous research were oral (e.g. Antinucci and Miller 1976; Bloom, Lifter and Hafitz, 1980; Bronckart and Sinclair 1973; Lee 2001; Robison 1995; Shirai and Anderson 1995). In those studies which did use written data, the data examined were gathered from elicited or controlled tasks, but not freely produced by language learners (e.g. Bardovi-Harlig and Reynolds 1995; Bergstrom 1995). In addition, those studies which have incorporated corpus data of running text have used standard target language corpora rather than learner corpora (e.g. Anderson and Shirai 1994; Shirai and Anderson 1995). The purpose of this paper is to use data extracted from an L2 learner corpus, English TLC, to see if lexical aspect still plays a role in L2 learners’ unrestricted written production. Moreover, as mentioned in Section 1, in addition to examining the distribution of L2 learners’ temporal markings, which may be a mirroring of standard uses, in this study we also investigated non-target uses (i.e. overuse of past tense markings in contexts within which base forms are obligatory) in order to uncover certain distinctive L2 learners’ uses of temporal markings. The results of the two approaches to testing the Aspect Hypothesis are reported in the following section. 
4. Examining temporal markings in English TLC 
The preceding section offered a brief introduction of lexical aspect and its correlation with language learners’ use of tense-aspect morphology found in previous studies. In this section, results are reported from two approaches used to extract data relevant to the Aspect Hypothesis from English TLC. First, we consider the temporal markings used by Taiwan EFL students in 50 articles chosen from the learner corpus. 
4.1. Distribution of temporal markings in Taiwan EFL students’ writings 
To see if the distribution of verbal morphology used by Taiwan EFL students was influenced by lexical 
892
aspect, as the Aspect Hypothesis predicts, 50 articles whose contexts are mainly in the past tense were randomly chosen from English TLC and the verbs (or predicates) with their inflections used in these articles were analyzed. The reason why we chose articles whose contexts are in past tense was because most recent studies concerning the Aspect Hypothesis investigated temporal markings “in past-time contexts,” as Bardovi-Harlig indicated (1998, p. 475). More specifically, with the verbal morphology used in these 50 articles, we would like to see if past tense markings were predominantly added to telic verbs rather than atelic verbs, consistent with the claims of the Aspect Hypothesis. 
From the fifty articles, all finite verb forms with temporal markings were collected and grouped into two main categories: atelic and telic verbs. A further distinction was made among atelic verbs, specifically, states and activities. The method we used in the present study to code verbs with respect to lexical aspect was based on Shirai and Anderson's (1995) operational tests which were applied to 958 predicates in the 50 articles to determine their inherent aspect. These 958 partitioned as follows: 376 telic verbs, 137 activities, and 445 stative ones. Next, we focus on each type of verb separately and examine the tense-aspect markings attached to them. First, we consider stative verbs. Table 1 below displays the tokens and percentages of each temporal marker used with stative verbs. 
Tokens Percentage 
present 236 53% 
progressive 0 0% 
past 201 45% 
perfect 8 2% 
Table 1. Temporal markings used with STATIVE verbs 
As Table 1 shows, about half stative verbs were used in present tense (53%) and the other half were used with past tense markings (45%). In addition, we can find, as the Aspect Hypothesis predicts, none of the stative verbs in the 50 articles was incorrectly used with the English progressive marking –ing. One might want to claim that the data presented here are inconsistent with the claims of the Aspect Hypothesis since about half stative verbs were used in past tense. However, not until we consider the other two types of verbs and their inflections can we conclude temporal markings in English TLC did not correlate with lexical aspect. Next, we examined the other atelic category: activity verbs and their inflections, as displayed in Table 2. 
Tokens Percentage 
present 55 40% 
progressive 18 13% 
past 62 45% 
perfect 2 2% 
Table 2. Temporal markings used with ACTIVITY verbs 
Like the stative verbs, substantial proportions of the activity verbs were used in present (40%) as well as past tense (45%). In addition, the major difference between the inflections attached to states and activities was that students did use progressive markings with activity verbs, but not with stative 
893
ones. However, the proportion of activity verbs using progressive marking was relatively small at 13% of all activity verb tokens. 
Having examined the markings used with the atelic (state and activity) verbs, let us turn to the inflections attached to telic verbs. Table 3 displays the numbers of tokens and percentages of each temporal marking used with telic verbs. 
Tokens Percentage 
present 42 11% 
progressive 1 1% 
past 322 86% 
perfect 11 3% 
Table 3. Temporal markings used with TELIC verbs 
As Table 3 shows, obviously, the most frequent marking EFL students used with telic verbs was past tense marking (86%), congruent with the central claims of the Aspect Hypothesis. Moreover, the percentage was much higher than the proportion of past tense marking used with states (45%) and activities (45%). In addition, from Table 1-3, we can find, compared with stative and activity verbs, fewer telic verbs were used in present tense (10%). A detailed discussion of these findings will be provided in Section 5. 
In short, the results presented above were generally consistent with the Aspect Hypothesis since Taiwan EFL students were more likely to add past tense markings to eventive (that is, telic) verbs (86%) than to stative (45%) or activity verbs (45%). However, as mentioned in Section 1, the value of these findings may be limited if we can find a similar skewing of temporal markings in standard uses. In what follows, we will provide some evidence to indicate this skewing of distribution of verbal morphology in language learners’ production is simply a mirroring of target uses. 
4.2. Distribution of temporal markings in standard uses 
A point we would like to note concerning the prevalence of past tense marking used by learners on telic predicates, which is predicted by certain versions of the Aspect Hypothesis, is that in the literature on language learners’ acquisition of tense-aspect morphology, some researchers have indicated that a similar skewing of temporal markings is found in native speakers’ uses. For instance, Anderson (1993) reported the results of two experiments done by his students showing English native speakers, like English learners, tended to associate past tense markings with accomplishments and achievements and progressive markers with activity verbs. Based on the results, he suggested “proficient native speakers will exhibit in relative quantitative terms the same distributional bias found in more absolute terms in the acquisitional data.” (p. 320) In addition, Huang (1999) examined three English speakers and five English learners’ temporal markings and concluded that in their oral production, “both learners and native speakers demonstrate similar skewed distributions of verb morphology.” More specifically, her results showed both of the two groups of subjects were prone to using past tense markings more frequently with achievement verbs and were more likely to associate progressive morphology with activities. 
894
Based on these findings, some researchers have hypothesized that perhaps it is the native speakers’ uses of temporal markings serving as language acquirers’ input which causes learners to associate one temporal marking with a specific aspectual class of verbs. This view, named Distributional Bias Hypothesis by Anderson (1990, p. 58), refers to: 
Native speakers in normal interaction with other native speakers tend to use each verb morpheme with a specific class of verbs, also following the aspect hypothesis. When learners are then exposed to this language of native speakers, they initially interpret this skewed distribution of forms as an absolute characteristic of the forms themselves. 
Several studies had been conducted to test the hypothesis as well. For example, in Anderson and Shirai's (1994) and Shirai and Anderson's (1995) studies, the researchers examining the temporal markings in three mothers’ speech addressed to children found children's development of tense-aspect morphology could be attributed to their input. Turning to our learners in Taiwan, this suggested to us that we might find a similar skewing as well. We therefore examined the temporal markings used in six lessons in Taiwan high school students’ English textbooks and found past tense markings were predominantly associated with telic verbs (52%) rather than with states (38%) or activities (10%). (A more extensive analysis of these texts is given in Huang, in prep.) Consequently, it is not surprising students would tend to associate past tense inflections with verbs that denote events, and language learners’ use of temporal morphemes could simply be seen as a mirroring of the distribution found in so-called target or standard English texts that serve as their target language input. 
Since the considerations of the Distributional Bias Hypothesis suggest that simply tabulating statistics about past tense markings on telic and atelic verbs is insufficient for revealing distinctive features in learner English, in the present study, we have added a novel consideration by examining non-target overuse of past markings in obligatory contexts for base verb forms to further test the Aspect Hypothesis. The results are reported in Section 4.3 below. 
4.3. Checking obligatory contexts in learner corpora 
The concept of obligatory context (Brown, 1973) is used frequently by language acquisition researchers to determine whether a certain morpheme or structure has been acquired by a learner. For instance, in Kuczaj's (1981) research, 16 English L1 learners’ use of be in obligatory contexts was examined to see if these learners acquired the copula and auxiliary be verbs in English. None of the previous studies consider obligatory contexts in a corpus, either standard or learner. We do precisely this in order to overcome the limitations of studies in the literature reviewed above which tabulate statistical data on morpheme usage regardless of whether the morpheme is correctly used. We introduce a novel approach to check obligatory contexts for base forms of verbs in a learner corpus by KWIC concordancing. More specifically, keywords including (1) modal auxiliaries such as can, could, will, would, shall, should, may, might, do, does, and did and (2) infinitive to which take a base form verb as complement were used to search English TLC to extract Taiwan English learners’ overuse of past tense forms following these keywords (e.g. He should made it). That is, we hoped to see if the verbs learners were prone to overusing past markings on are eventive verbs, congruent with the claims of the Aspect Hypothesis. Totally, 82 
895
verbs with which students overused past markings were extracted from the learner corpus and were grouped into two main categories: atelic (states and activities) and telic verbs, based on the operational tests we used in Section 4.1. Table 4 shows the distribution of students’ overuse of past tense forms with respect to lexical aspect. 
Tokens Percentage 
atelic 40 49% 
telic 42 51% 
Table 4. Lexical aspect with overuse of past forms 
Based on the numbers shown in Table 4, it seemed our hypothesis was not confirmed since almost the same percentages of past tense markings were overused with telic (51%) and atelic verbs (49%). In addition, while we examined these past tensed verbs, we found an interesting fact. That is, we found among these verbs students incorrectly inflected for past tense, many had irregular past tense forms (e.g. send: sent & know: knew). Therefore, we further divided these verb tokens into two groups: regular and irregular ones, and the results are displayed in Table 5. 
Tokens Percentage 
regular 27 33% 
irregular 55 67% 
Table 5. Regular/irregular distinction with overuse of past forms 
Table 5 shows about 67% of the verbs students overused past tense markings with had irregular past tense forms, a percentage even higher than the proportion of telic verbs (51%) among the tokens of overused past forms. Therefore, based on the results here, it seems the main determinant of learners’ overuse of past forms was whether the verbs had regular or irregular past forms, but not their lexical aspectual class (stative, activity, or telic). 
In this section, we have presented data extracted from English TLC showing (1) Taiwan English L2 learners’ use of temporal markings in these learners’ unrestricted written texts conformed to the Aspect Hypothesis findings that past tense markings were significantly more prevalent on telic (event) predicates than on atelic (activity or state) predicates and (2), by checking obligatory contexts in a learner corpus, a much stronger determinant on the overuse of past forms than lexical aspect was found to be whether the past form was regular (-ed) or irregular. A more detailed discussion of these finding is provided in the following section. 
5. Discussion 
When investigating the temporal markings used in a learner corpus of English L2 learners’ unrestricted writings, we found, consistent with the claims of the Aspect Hypothesis, the use of temporal markings was indeed influenced by the semantic features of the verbs in English. Specifically, telic verbs were predominantly marked with past tense (86%), unlike states and activities, which attached past tense markings to a significantly smaller proportion of their total tokens (45% and 45%, respectively).3 
3 In addition, we also found most progressive markings were used with activity verbs (95%), though there were not many progressive markings used in the 50 articles for us to investigate since the contexts 
896
However, the finding, while confirming the Aspect Hypothesis, did not, we argue, reveal anything distinctive about learners’ English. In the present study, we provided evidence showing the target texts exhibited basically the same tendency of temporal markings reported for learners in the Aspect Hypothesis literature. Language learners’ use of tense-aspect markings, however, can be plausibly seen as simply a mirroring of the distribution that can be found in standard uses and in their target language input. Therefore, in this study, we introduced the use of obligatory contexts in our analysis of the learner corpus. By checking L2 learners’ overuse of past tense forms in contexts within which base verb forms are obligatory, we could examine whether these non-target uses were influenced by lexical aspect as well. However, in these obligatory contexts, we found the main determinant of learners’ overuse of past forms was whether the verbs had regular or irregular past forms.4 The regular/irregular distinction wielded much stronger influence on learners’ overuse of past forms than did the verbal aspect. In fact, the indication that regular/irregular distinction might determine language learners’ use of temporal markings is not new in the language acquisition research literature. After examining the verbal inflections used by English L2 learners, Salaberry (2000) has claimed what influenced L2 learner’ use of past tense morphology should be the frequency of the past tense morphology but not the lexical aspect of verbs (p. 135). More specifically, he found the verbs his subjects were prone to inflecting for past tense were mostly irregular, and the most frequent verbs in English are irregular. The finding obtained by examining L2 learners’ overuse of past forms in the present study, inconsistent with the Aspect Hypothesis, showed the regular/irregular distinction could well be stronger than lexical aspect in determining language learners’ use of past tense markings, at least in learners’ overuse of past forms. These findings suggest further research is needed to tease out the influence of frequency, regularity, and verbal semantics as factors in the acquisition of past tense markings in English by second language learners. 
of these articles were mainly in the past tense. 
In short, the results of the present study suggest that L2 learner corpora of unrestricted written texts can contribute to research on L2 learners’ acquisition of tense-aspect morphology. However, further CLC research is indeed needed to test these conflicting findings in the present research as well. 
Reference 
Anderson R W 1990 Models, processes, principles, and strategies: second language acquisition in and 
out of the classroom. In VanPatten B, Lee J F (eds), Second language acquisition-foreign language 
learning. Clevedon, Multilingual Matters, pp 45-78. 
Anderson R W 1993 Four operating principles and input distribution as explanations for underdeveloped and mature morphological systems. In Hyltenstam K, Viborg A (eds), Progression and regression in language. Cambridge, Cambridge University Press, pp 309-339. 
Anderson R W, Shirai 1994 Discourse motivations for some cognitive acquisition principles. Studies in 
4 This finding provides only part of the picture on overuse of past forms. A more complete picture will include the percentages of all uses of telic, atelic, regular, and irregular verbs which are overused in past tense. This additional data will be included in Huang (in prep). 
897
Second Language Acquisition 16: 133-156. 
Anderson R W, Shirai Y 1996 The primacy of aspect in first and second language acquisition: the pidgin-creole connection. In Ritchie W C, Bhatia T K (eds), Handbook of second language acquisition. San Diego, Academic, pp 527-570. 
Antinucci F, Miller R 1976 How children talk about what happened. Journal of Children Language 3: 167-189. 
Bardovi-Harlig K 1998 Narrative structure and lexical aspect. Studies in Second Language Acquisition 20: 471-541. 
Bardovi-Harlig K, Reynolds D 1995 The role of lexical aspect in the acquisition of tense and aspect. TESOL Quarterly 29: 107-131. 
Berstrom A 1995 The expression of past temporal reference by English-speaking learners of French. Unpublished PhD thesis, Pennsylvania State University, State College. 
Bloom L, Lifter K, Afitz J 1980 Semantics of verbs and the development of verbs inflection in child language. Language 56: 386-412. 
Bronckart J P, Sinclair H 1973 Time, tense, and aspect. Cognition 2: 107-130. 
Brown R 1973 A first language: the early stages. Cambridge, MA: Harvard University Press. 
Comrie B 1976 Aspect. Cambridge, Cambridge University Press. 
Dowty D 1979 Word meaning and montague grammar. Dordrecht, Reidel. 
Granger S 1998 The computer learner corpus: a versatile new source of data in the EFL classroom. In Granger S (ed), Learner English on computer. London, Longman, pp 3-17. 
Huang C C 1999 Tense-aspect marking by L2 learners of English and native English speakers: inherent lexical aspect and unitary vs. repeated situation types. Issues in Applied Linguistics 10: 113-129. 
Huang P Y in prep Lexical aspect in the interlanguage of Taiwan English learners. Unpublished Master thesis, Tamkang University. 
Kuczaj S A 1981 The acquisition of copula and auxiliary “be” forms. (ERIC Document Service No. 209932). 
Lee E J 2001 Interlanguage development by two Korean Speakers of English with a focus on temporality. Language Learning 51: 591-633. 
Robison R E 1995 The aspect hypothesis revisited: a cross-sectional study of tense and aspect marking in interlanguage. Applied Linguistics 16: 344-370. 
Salaberry M R 2000 The acquisition of English past tense in instructional setting. System 28: 135-152. 
Shirai Y, Anderson R W 1995 The acquisition of tense-aspect morphology: A prototype account. Language 71: 743-762. 
Vendler Z 1967 Verbs and times. Philosophical Review 66: 143-160. 
Verkuyl H J 1996 A theory of aspectuality. Cambridge, Cambridge University Press. 
Wible D, Kuo C-H, Chien F-Y, Liu A, Tsao N-L 2001 A web-based EFL writing environment: exploiting information for learners, teachers, and researchers. Computers and Education 37: 297-315. 
898

A corpus analysis of discourse relations for Natural Language Generation 
Sandra Williams and Ehud Reiter 
Department of Computing Science, University of Aberdeen, 
Aberdeen AB24 3UE. 
E-mail: swilliam/ereiter@csd.abdn.ac.uk 
Abstract 
We are developing a Natural Language Generation (NLG) system that generates texts tailored for the reading ability of individual readers. As part of building the system, GIRL (Generator for Individual Reading Levels), we carried out an analysis of the RST Discourse Treebank Corpus to find out how human writers linguistically realise discourse relations. The goal of the analysis was (a) to create a model of the choices that need to be made when realising discourse relations, and (b) to understand how these choices were typically made for “normal” readers, for a variety of discourse relations. We present our results for discourse relations: concession, condition, elaboration-additional, evaluation, example, reason and restatement. We discuss the results and how they were used in GIRL. 
1. Introduction 
The Generator for Individual Reading Levels (GIRL) project is developing a Natural Language Generation (NLG) system that generates feedback reports after a web-based literacy assessment (Williams 2002). The literacy assessment is for adults with poor basic literacy and it was designed by NFER-Nelson for the Target Skills application (Target Skills 2002). It produces a multi-level appraisal of a candidate's literacy skills based on the UK Adult Literacy Core Curriculum (Steeds 2001). 
GIRL is being developed with the goal of tailoring output texts to the individual reading skills of users (readers). In working towards this goal, we hope to find out more about generating documents for readers at different reading levels, how to test a system on real users (e.g. both good readers and learner readers), and how to implement reading level decision-making mechanisms as part of the generation process. Also, which decisions produce the most marked impact on the readability of the output texts. We particularly focus on discourse-level choices in GIRL, since earlier projects, e.g. PSET (Devlin et al., 2000), have looked at lexical and syntactic choices for poor readers. 
Discourse-level decisions are made in GIRL in a module called the microplanner. The microplanner's input is a tree of discourse relations joining pieces of information. The microplanner plans how the information from the tree will be ordered, whether it will be marked with discourse cue phrases (e.g. ‘but', ‘if’ and ‘for example'), and how it will be packed into sentences and punctuated. That is, it makes a number of decisions about how the output text will be realised linguistically. Below are listed a number of different ways in which an example discourse relation could be realised. 
a) Sometimes you did not pick the right letter. You did not click on the letter ‘d'. 
b) Sometimes you did not pick the right letter. For example, you did not click on the letter ‘d'. 
c) Sometimes you did not pick the right letter. You did not, for example, click on the letter ‘d'. 
d) Sometimes you did not pick the right letter – you did not click on the letter ‘d', for example. 
e) You did not click on the letter ‘d'. Sometimes you did not pick the right letter. 
f) Sometimes you did not pick the right letter. For instance, you did not click on the letter ‘d'. 
These are legal ways of realising an example relation that were encountered during this analysis. The discourse relation consists of two text spans, a statement ‘sometimes you did not pick the right letter’ and an example ‘you did not click on the letter ‘d''. In (a), the ordering of the two text spans is statement-example and each span is a separate sentence (there is a full stop between spans). The version in (b) is the same except that the discourse cue phrase ‘for example’ has been added before the second text span. In (c) it is positioned mid-second-span. In (d) there is an ‘em dash’ between spans and the cue phrase is at the end. In (e), the ordering is reversed and there is no cue phrase. (f) is the same as (b) but with ‘for instance’ rather than ‘for example'. 
Which version is the most readable? Readability calculators (e.g. Flesch 1949) tell us little about how to produce good readable texts. Flesch's calculator estimates that versions (a) and (e) both score equally as the most ‘readable', followed by (b) and (c) equal in second place, then (f) and lastly (d). The PSET project (Devlin et al., 2000) found that lexical selection has an impact on readability, and so cue phrase choice is important. Order of text spans, existence and position of cue phrases and between-span punctuation may also have an impact. Shorter sentences may help poor readers as Flesch's calculator predicts. 
It is very important to base the development of GIRL on solid empirical evidence rather than on our own intuitions. There has been very little empirical work to date on what kinds of texts are most appropriate for people with good reading skills and even less on what is appropriate for people with 
899
poor literacy. We were therefore motivated to do our own empirical studies such as the corpus analysis described in this paper. 
Aspects of the analysis are unique, since it was customised to the particular requirements of GIRL. Other aspects overlap with the much smaller analysis of Moser and Moore (1996). In this paper, we describe the corpus we used, the method we followed and our results. We go on to discuss how each of the features we analysed has an effect on all the others. We also compare our results with Moser and Moore's. Finally we describe how our corpus analysis results were used to build models for GIRL. 
2. Corpus 
We first collected a corpus of documents written by literacy tutors for readers at a variety of skill levels, and tried to analyse this to determine our model and the optimal choices for different readers. Unfortunately, this corpus was too small and inconsistent (different tutors wrote in very different ways) to allow us to extract useful information. We hence decided to focus on analysing existing corpora collected by others, even though these corpora do not in general contain texts written for poor readers. Later we adapted the model and choices to poor readers, based of psycholinguistics and experiments, but this is not the subject of this paper. 
Initially, we carried out an analysis of discourse cue phrases in the British National Corpus (http://www.hcu.ox.ac.uk/BNC/). That is, we searched the corpus for a number of cue phrases and parts-of-speech. For example, we searched for instances of the cue phrase “but” as a coordinating conjunction. We analysed the first 100 results and on this basis decided that our model would contain the following features: order of discourse text spans; existence and choice of cue phrase(s); position of cue phrase(s); existence and choice of between-text-span punctuation (including sentence-breaking punctuation) and length of first text span. The last two features obviously have an impact on sentence length. Choice of cue phrase has an impact on word length and sentence length. Both sentence length and word length are factors highlighted by readability calculators. 
We also attempted to determine choice rules from the BNC analysis, but this was difficult because the BNC only has part-of-speech annotation, not discourse relations annotation. Therefore we had no information about underlying discourse relations. For instance, to find out about the concession relation, a search in the BNC for a cue phrase such as ‘but’ may give a result containing a few concession relations, but it will not identify them nor, of course, give all the concession relations. We could not therefore answer some key questions about discourse relation realisation, such as how often authors omitted cue phrases completely, or which cue phrases were preferred for different relations. We obtained the RST Discourse Treebank corpus (RST-DTC) (Carlson, Marcu, and Okurowski, 2002) from the Linguistic Data Consortium (http://www.ldc.upenn.edu/). As far as we are aware, this is the largest publicly available and machine-readable corpus that has been annotated with discourse relations. So it was the best corpus for our purposes. An additional benefit of using this corpus is that the corpus annotators claim not to have been influenced by the presence of discourse cue phrases during their analysis (e-mail from Daniel Marcu). 
The RST Discourse Treebank Corpus (RST-DTC) (Carlson, Marcu, and Okurowski, 2002) is based on Rhetorical Structure Theory (RST) (Mann and Thompson 1987). The corpus contains 385 texts from the Wall Street Journal (WSJ) on a number of subjects (e.g. finance, world news and the arts). We analysed part of the TRAINING section. There are 342 files in this section. Each represents a single Wall Street Journal article. The files containing RST annotations are in the format shown in Figure 1. 
( Root (span 1 156) 
( Nucleus (span 1 129) (rel2par Topic-Drift) 
( Nucleus (span 1 40) (rel2par Problem-Solution) 
( 
Nucleus (span 1 21) (rel2par span) ( Nucleus (leaf 1) (rel2par span) (text _!Kidder, Peabody 
& Co. is trying to struggle back.<P>_!) ) 
( Satellite (span 2 21) (rel2par circumstance) 
( Nucleus (span 2 5) (rel2par Contrast) 
( Nucleus (span 2 4) (rel2par span) 
Figure 1. A fragment from the RST-DTC annotation file wsj_0604 
Segments of text are numbered and nodes are labelled Root, Nucleus or Satellite. The RST tree structure is denoted by embedding text segments and segment spans within parentheses. The figure displays the beginning of one of the WSJ files. It shows the root node spanning over the entire text (segments 1 to 156). Bold font highlights a circumstance relation with a single leaf node nucleus 
900
(segment 1), and text: ‘Kidder, Peabody...'. The satellite of this relation spans across segments 2 to 21 and contains relations at lower levels in the rhetorical structure tree. 
A software tool, RSTtool, is supplied with the corpus for viewing the RST trees. Figure 2 shows a screenshot of the RSTtool that displays part of an RST tree. The tree fragment shows a tri-nuclear Sequence relation between three text segments (segment 94:‘He bites it,'; segment 95:‘scowls’ and segment 96:‘and throws it down'). The other relation shown is evaluation-n with its satellite spanning across segments 94 to 96 and its nucleus is segment 97: ‘It's a real dog.'. 
Figure 2. Viewing part of an RST tree using the RSTTool 
There are a number of well-documented problems with RST analysis including segmentation of the text, selection of relations, and proliferation of relations (Hovy and Maier 1995). The RST-DTC analysts had to grapple with these and it is not surprising that the corpus suffers from poor inter-annotator agreement (Marcu et al. 1999), as does RST analysis in general. We are not experts in RST analysis, but we did notice some inconsistencies when we were carrying out our own investigations of the corpus, e.g. in segmentation, in choice of relations and in nucleus-satellite identification. 
In spite of these problems, we decided to use the RST-DTC in this research project. Investigating the annotated discourse relations within this corpus was the quickest and easiest way to discover how frequently discourse cue phrases are present. Furthermore, it allowed us to investigate how these relations are linguistically realised (text span ordering, cue phrase choice, cue position, etc). The corpus is a useful resource for computational discourse linguistics and it represents a great deal of hard work by its creators. 
3. METHOD 
The corpus analysis was carried out and checked entirely by the first author. Further checking by independent analysts, would, of course, be desirable, but this was not possible within the scope of the PhD project. 
3.1 Choosing discourse relations 
Our first task was to choose which of the 60+ discourse relations in the corpus to investigate (the exact number of types of relations present depends on how you count them, e.g. consequence-n, consequence-s and Consequence could be counted as one, two or three). We wanted to analyse the relations most commonly occurring in our small corpus of tutor-written reports, since this is the type of document our NLG system generates. We decided to analyse six relations: concession, condition, evaluation (evaluation-n and evaluation-s), example, reason and restatement. 
The semantics of some relations seem to overlap and we sometimes found it hard to distinguish between them, even though we referred to the definitions in Carlson and Marcu's Discourse Tagging Reference Manual (2002). For instance, in our tutor-written corpus, the tutors gave examples of things the students got right, or wrong. So we chose to investigate the example relation. However, another relation, called evidence, also seemed close in meaning. Furthermore, tutors often suggested reasons why they thought students had got questions wrong. We decided to investigate the reason relation, but we wavered between it and the cause, result and evidence relations. Similarly with the relations evaluation and interpretation. 
We discovered that the relations we chose to investigate were not very common in the TRAINING section of the RST-DTC: concession: 1.3%, condition: 1.0%, evaluation: 0.9%, example: 
901
1.6%, reason 0.9%, and restatement: 0.6%. We later investigated the most common relation of all, elaboration-additional (20% of all relations). Other very common relations are List: 15.8%, attribution: 14.4% and elaboration-object-attribute: 12.7%, but these were left for later work. 
3.2 Analysing each discourse relation 
The corpus analysis proceeded as follows for each discourse relation we investigated. First we searched the annotation files using a simple UNIX grep command for lines containing the relation. From the results of the search, we extracted the RST-DTC file names (e.g. wsj_0603), and the location of the relation (e.g. ‘leaf 4’ or ‘ span 102 114'). We produced a table of file names and locations. The table was transferred to an MS Excel worksheet where all analyses for the discourse relation under investigation were recorded. We analysed the first 100 occurrences of each relation. 
3.3 Determining the semantic roles of text spans and their ordering 
Figure 3. An ‘if’ text span embedded in a ‘then’ text span 
Next, we opened each file containing the relation under investigation with the RSTtool so that we could view the RST tree and read the text spans of the relation. When viewing the RST tree, we were able to judge (somewhat subjectively) which semantic role each text span of the relation played. For example, in a condition relation, one span plays the antecedent role (the one that could potentially have cue phrase ‘if') and the other plays the consequent role (the one that could have cue phrase 'then'). To avoid confusion, we simply call these text spans ‘if ’ and ‘then'. We recorded the order of the text spans, e.g. ‘if-then’ or ‘then-if'. Sometimes one span is embedded within another, as in Figure 3 where the ‘if’ text span, segment 34, (also the satellite) is shown embedded within the ‘then’ span. Segment 33 and segment 34 are shown linked as the nucleus of this relation by the bi-nuclear ‘Same-Unit’ keyword. We recorded this embedding as ‘th(if)en'. 
Note that we did not record nucleus-satellite orderings, but semantic orderings. Some RST relations are in fact defined in terms of semantic roles for nucleus and satellite (Carlson and Marcu 2001), e.g. the condition relation is defined as “the truth of the proposition associated with the nucleus is a consequence of the fulfilment of the condition in the satellite.” (p.51). However, we found examples in the RST-DTC where these semantic roles were not adhered to and we could not always rely on them. 
Nuclearity is also defined in terms of writers’ intentions. This is a similar concept to ‘intentional subordination’ described by Moser and Moore (1996). But RST-DTC annotators were not always able to determine writers’ intentions. For instance, in TRAINING there are 47 occurrences of the cause relation and 121 of result, but there are also 102 occurrences of the multi-nuclear relation Cause-Result. This relation is defined in an earlier tagging manual (Marcu 1999) as “When it is not clear what the intention of the writer is.” (p.24). The proportion of Cause-Result relations is relatively high and perhaps this is an indication of a real difficulty the annotators had with this aspect of nuclearity. They could only guess at a writer's intentions, at best. Since we required semantic roles to build our models of discourse realisation for GIRL, we recorded only the semantic role orderings. 
3.4 Identifying discourse cue phrases 
The next part of our analysis determined whether a discourse cue phrase was present, or not, in the relation being investigated. In most cases, this was fairly clearcut, for instance, we judge that ‘if’ is a cue phrase in the condition relation in Figure 3. But sometimes it was difficult to judge which relation a 
902
cue phrase ‘belonged’ with. See, for instance, Fig. 4 where we judge that the cue phrase ‘and’ in segment 35 belongs with the List relation, but the cue ‘however’ in segment 36 belongs with antithesis rather than attribution. This difficulty often occurred when text spans of the relation spanned across other relations, or vice versa. 
Figure 4. Identifying which relation a discourse cue phrase ‘belongs’ with. 
Another difficulty was judging whether a phrase was, in fact, a discourse cue phrase, or not. We applied the test described by Knott and Dale (1994), i.e. Is the text span containing the potential cue ‘incomplete’ when elided items have been included and references resolved? We were also helped greatly in identification of cue phrases by using lists that other researchers made available. We believe we have identified some new ones too: for example, ‘sure’ in an evaluation relation with first text span: ‘There are, of course, analysts who view the near-panic that briefly swept through investors on Oct. 13 and again on Oct. 24 as momentary lapses of good judgement that have only temporarily undermined a healthy stock market.'; and second text span: ‘Sure, price action is volatile and that's scary, but all-in-all stocks are still a good place to be...'. 
3.5 Recording discourse cue phrase positions and between-text span punctuation 
This part of our analysis was not subjective. We recorded the positions of cue phrases, if present, as before, mid- or after a text span. We also recorded between-text-span punctuation, if any. A simple coding scheme was devised to record these details. For instance, 0 . 0 represents two text spans with no cue phrases and a full stop between them. A text span is represented as 0 if there is no cue phrase, 1 if the cue phrase comes before the span, m if the cue is mid-span, and 2 if the cue comes after the span. Thus 0 , m denotes no cue on the first span, a comma between spans and a cue phrase occurring mid-second span. 2 n 1 denotes cue phrases after the first span and before the second span and no between-span punctuation (e.g. ‘hurry then if you are late'). We did not analyse in more detail exactly where mid-span cues occurred (e.g. after the first noun phrase), we leave this for future work. Also, we only recorded the final punctuation mark between spans (e.g. a quote mark followed by a full stop would be recorded simply as a full stop). We also ignored any mid-text-span punctuation. 
3.6 Finding the length of the first text span. 
We recorded the length of the first text span in words. There are a wide variety of techniques to count words. We tried to be consistent. We counted two hyphenated words (e.g. ‘near-panic') as two words. We counted people's initials as individual words (e.g. ‘A. B. Wilson’ was counted as three words). We counted as one single word: numbers, including monetary amounts (e.g. ‘$3,4200') and acronyms (e.g. ‘USA'). 
4. Results 
Here we present the results of our analyses of the RST-DTC. 
4.1 Presence of cue phrases 
The percentage of instances without discourse cue phrases for each relation is shown in Figure 5. Some relations, such as condition and concession, almost always have cue phrases, so the numbers of examples with no cues are very small. Other relations, such as restatement, almost never have cue phrases (99% of cases found). Other relations fall between these extremes. 
Our results for presence of cue phrases agree fairly well with Moser and Moore's (1996). They found 133 discourse relations with cue phrases out of a total of 286 relations in their corpus (47%). We found 310 relations with cue phrases out of a total of 700 relations (44%). Our result is not entirely representative of the WSJ texts since we isolated seven discourse relations for our study. However, we chose to include elaboration-additional because it is the most common relation and only 
903
24% of instances of this relation had cues. So 44% is an overestimate. The slight differences might be due to differences in genre of the two corpora. Moser and Moore's corpus consists of on-line tutor feedback and this is actually much closer to the kind of texts that GIRL generates, although, like RST-DTC it does not contain text written for poor readers. 
Figure 5. No cue phrases present 
4.2 Writers’ cue phrase preferences 
We compared the actual cue phrases chosen by authors to mark each relation. Table 1 shows all the cue phrases we found and in which of the seven relations investigated they occurred. 
N cue phrase r n cue phrase r n cue phrase r n cue phrase r 1 accordingly cc 1 even as cc 1 once cd 1 whatever el 1 after all re 1 even before ex 1 only if cd 1 when cc 1 also el 4 even if cd 1 or rs 2 when cd 11 although cc 3 even though cc 2 particularly because re 1 when ex 1 and cc 2 except el 1 particularly since re 3 when re 4 and el 14 for example ex 1 partly because re 1 whereas cc 8 and ev 3 for instance ex 1 rather cc 1 whether cc 4 and re 2 given cd 2 regardless cc 4 which el 1 and for that reason re 3 however cc 3 should cd 2 which ev 1 and therefore re 1 however el 1 so ev 8 while cc 1 as a result re 2 however ev 1 so re 4 who el 2 as long as cd 62 If cd 1 so far ev 1 with ev 2 as soon as cd 3 if ... then cd 5 still cc 2 without cd 40 because re 1 In fact el 2 such as ex 1 yet cc 35 but cc 1 indeed re 1 sure ev 1 yet ev 2 but el 1 instead re 3 that el 2 but ev 1 just as soon as cd 7 though cc 1 but re 1 mainly because re 5 unless cd 390 no cue phrase 8 despite cc 1 now el 5 until cd 700 TOTAL 
Table 1. Cue phrases used in 700 relations, where n=number, r=relation, cc=concession, cd=condition, el=elaboration-additional, ev=evaluation, ex=example, re=reason and rs=restatement. 
Table 1 shows marked differences in the range of cue phrases used. For instance, concession had 17, although many of these occurred only once. Restatement had only one occurrence of a cue phrase (‘or') in the 100 cases investigated. The cue phrase ‘if’ has a high frequency (62) in Table 1 because writers choose to mark over 90% of condition relations with cues, and the cue most commonly chosen is ‘if'. 
Of the relations with many different cue phrases, usually one or two cue phrases occur with much greater frequency than others. For instance ‘but’ most often marks a concession relation, ‘if’ a condition relation, and ‘because’ a reason relation. 
4.3 Order of text spans 
Figure 6 shows the semantic ordering of text spans. We found restatement always occurs in an order where a statement is followed by a restatement (statement-restatement), whilst example, and elaboration-additional almost always occur in orderings statement-example and statement-elaboration. Concession, evaluation and reason all have one preferred ordering, whereas condition occurs in both orders with almost equal frequencies. Embedding is relatively rare. In these seven relations, it is most 
904
common in condition. In TRAINING as a whole, the frequency of embedded relations is 13%, which is much higher. Embedding is likely to have an impact on readability and it should be investigated further. Note that we call the spans in concession ‘but’ and ‘though’ where ‘though’ denotes the span that could be marked with cue phrases ‘though', ‘although', etc. and ‘but’ denotes the other (see section 3.3). 
Figure 6. Results for text span ordering for seven discourse relations 
4.4 Positions of cue phrases and between-span punctuation 
Figure 7 shows the positions of cue phrases (before, mid-, or after the first or the second text span) for each relation and between-span punctuation. The coding scheme is described in section 3.5. We found one relation, restatement, was actually more commonly marked by an open parenthesis, than by a cue phrase. 
Figure 7. Positions of discourse cue phrase(s) and between-text-span punctuation 
Condition, has a distinct correlation between order of text spans, cue phrase position and punctuation. The code 0 n 1 (e.g. ‘hurry if you are late') accounts for 37% of all condition relations and 1 , 0 (e.g. ‘if you are late, hurry') accounts for a further 31%. So, if a cue phrase is present before the second 
905
span, a comma is not required and vice versa. This correlation appears to happen to a slightly lesser extent in reason where 0 n 1 occurs with frequency 36% and it occurs too in concession. 
4.5 Length of first text span 
Figure 8 shows the results for the length of first text spans for all seven relations. The curved lines drawn over the histograms indicate different trends. The relations fall into three categories showing similar trends. Concession and example have fairly flat profiles, indicating that these relations can have first text spans of any length. Condition, reason and restatement have quite short first text spans with the curves reaching a maximum at 6-10 words. Elaboration-additional and evaluation reach their maxima above 30 words. The reason why their first text spans are so long is that these two relations often have left-branching RST tree structures. That is, their first text spans tend to stretch across many relations at lower levels in the RST tree. 
Figure 8. Length in words of first text spans for seven discourse relations 
5. Cross-correlations 
We have presented our results for some individual features involved in the linguistic realisation of discourse relations, i.e. first text span length, span order, existence and choice of between-span punctuation, existence and choice of cue phrases and cue phrase position. But what we were really interested in is the impact of each feature on all the others, so that we could build models for GIRL. It is easiest to illustrate what we mean by Table 2. This table is a matrix of rules derived from results for just one of the relations: concession. The effect of one feature on any other can be seen by finding one feature on the left-hand side of one of the rows, then locating the other feature in one of the column headings (or vice versa) and reading the table cell located where the row and column intersect. The rules shown in this matrix are simplified with the less frequent results omitted. 
Some correlations are as we would intuitively expect. For instance, the impact of first span length on punctuation is that with a shorter span a comma is preferred (1-10words->comma in the table), whereas with a longer span a full stop is preferred. Other effects are less obvious, e.g. the cue choice rules. 
Moser and Moore (1996) use a theory of discourse relations with ‘core’ and ‘contributor'. These are somewhat similar to RST's nucleus and satellite (where core is roughly equivalent to nucleus and contributor to satellite). They discovered that a cue phrase never occurs on the core if it is ordered first. In our cross-correlation for concession in Table 2, the summary for order vs. cue position says that in ‘but-though’ order (see section 4.3 for an explanation), the cue cannot go on the first text span, only on the second. The concession relation is defined semantically in RST and what we call the ‘but’ 
906
text span is the nucleus. We did not find any counter examples in our semantic analysis and this result confirms that the RST-DTC annotators correctly assigned the nucleus and satellite for concession. Also our result agrees with Moser and Moore's result, since our but-though ordering is equivalent to nucleus-satellite ordering and hence in this order, a cue phrase can only go on the satellite. 
LENGTH ORDER PUNCTUATION CUE CHOICE CUE POSITION LENGTH No correlation 1-10words -> comma 
11-20words->comma, full stop 
>20words -> full stop 1—10->but, despite, while 
11-20->but, though, although, while 
>20->but, still, no cue 1-20words-> before either 
>20words->before 2nd ORDER though-but->comma, 
full stop 
but-though->comma, none though-but->no cue, but, although, despite, still, while, though 
but-though->although, even though, despite, though though-but->before either 
but-though->before 2nd PUNCTUATION comma->although, but, despite, though, while 
full stop->but, still, no cue comma->before either, 
full stop->before 1st, no cue 
none->before 2nd CUE CHOICE although->before either 
but->before 2nd 
despite->before either 
still->before 2nd 
though->before either 
while->before either 
Table 2. A simplified cross-correlation matrix showing the most common results for concession. 
6. Models for NLG 
Instead of a matrix of rules for each relation, such as the one for concession in Table 2, we constructed a decision tree and sets of rules using machine learning techniques similar to Di Eugenio, Moore and Paolucci (1997). We fed in a set of feature-values for each instance of a relation (e.g. for a condition relation this might be: first span length = 8, order = if-then, cue = if, position = before-first, punctuation = comma). Then the machine learning algorithm gave us a decision tree from which we derived a set of rules for the microplanner. It can then work out from the lengths of input text spans: their ordering, the existence, selection and position of cue phrase(s) and existence and selection of between-span punctuation. We built a model in this manner for each discourse relation. Our current microplanner uses these models, as described in the introduction. 
Although it is not the topic of this paper, we have subsequently adapted the choices specified by the models derived from this corpus analysis to be more appropriate for learner readers. We based these adaptions on psycholinguistic findings and on our own experiments (Williams and Reiter 2003). We are currently testing readability (reading speed and comprehension) for various readers on texts generated using different choice profiles. 
7. Conclusions 
This paper has described an analysis of the RST Discourse Treebank Corpus with the goal of acquiring rules for realising discourse relations. Our results show that discourse relations are commonly used in concession, condition and reason whilst they are rarely used in restatement, evaluation, example, and elaboration-additional. Human writers most commonly use the discourse cue phrases ‘but’ in concession, ‘if’ in condition, ‘because’ in reason and ‘for example’ in example. They tend to prefer one text span ordering over the other for all relations except condition. Evaluation and example are most commonly realised with no cue phrase and text spans in separate sentences. Elaboration-additional most commonly has no cue phrase and a comma between segments. Restatement is most commonly marked by punctuation,‘('. We found that correlations between order of spans, position of cue phrase and punctuation were indicated in condition, reason and concession. We discussed a correlation matrix for concession and showed examples of derived rules. 
This corpus analysis allowed us to achieve our goal of creating models in GIRL for the choices that need to be made when realising discourse relations and how these choices are typically made for “normal” readers. 
907
Acknowledgements 
An EPSRC studentship award supported this work. We would like to thank Alistair Knott, Daniel Marcu, Chrysanne Di Marco and Robert Mercer for lists of discourse cue phrases; everyone involved in the creation of the RST-DTC; Somayajulu Sripada and Judy Delin for helpful comments. 
References 
Carlson L, Marcu D, Okurowski M, 2002 Building a Discourse-Tagged Corpus in the Framework of Rhetorical Structure Theory. Kuppevelt and Smith (eds) Current Directions in Discourse and Dialogue, Kluwer. 
Carlson L and Marcu D 2001 Discourse Tagging Manual. Information Sciences Institute, University of Southern California. ISI Tech Report ISI-TR-545. 
Devlin S, Canning Y, Tait J, Carroll J, Minnen G and Pearce D 2000 An AAC aid for aphasic people with reading difficulties. Proceeding of the 9th Biennial Conference of the International Society for Augmentative and Alternative Communication. Washington D.C. 
Di Eugenio B, Moore J D and Paolucci M 1997 Learning Features that Predict Cue Usage , in Proceedings of the 35th Annual Meeting of the Association for Computational Linguistics, Madrid. 
Flesch R [1949] The Art of Readable Writing. Harper. USA. 
Hovy E H and Maier E 1995 Parsimonious or Profligate: How Many and Which Discourse Structure Relations? Information Sciences Institute, University of Southern California. Unpublished manuscript. 
Knott A and Dale R 1994 Using Linguistic Phenomena to Motivate a Set of Coherence Relations. Discourse Processes, Volume 18, Number 1, pp. 35-62. 
Mann W C and Thompson S A 1987 Rhetorical Structure Theory: A Theory of Text Organization. Information Sciences Institute (ISI) Reprint no. ISI/RS-87-190, University of Southern California. 
Marcu D 1999 Instructions for Manually Annotating the Discourse Structure of Texts. Information Sciences Institute, University of Southern California. 
Marcu D, Romera M, and Amorrortu E 1999 Experiments in Constructing a Corpus of Discourse Trees: Problems, Annotation Choices, Issues. Workshop on Levels of Representation in Discourse. 
Moser M and Moore J 1996 On the correlation of cues with discourse structure: results from a corpus study. Unpublished manuscript. 
Steeds A (ed) 2001 Adult Literacy core curriculum including Spoken Communication. Produced by Cambridge Training and Development Ltd. for The Basic Skills Agency. ISBN 1-85990-127-1 
Target Skills Literacy Assessment. 2002. Published by the Basic Skills Agency, in association with Cambridge Training and Development Ltd. and NFER-Nelson Ltd. 
Williams S H and Reiter E 2003 Experiments with discourse-level choices and readability. To appear in Proceedings of the 9th European Workshop on Natural Language Generation, Budapest. 
Williams S H 2002 Natural language generation of discourse connectives for different reading levels. In Proceedings of the 5th Annual CLUK Research Colloquium, University of Leeds. 
908

Building and annotating corpora of spoken Welsh and Gaelic1 
Andrew Wilson, Celia Worth 
Department of Linguistics and Modern English Language, Lancaster University 
1. Introduction 
The indigenous minority languages of the British Isles and Ireland (or "BIMLs") are becoming increasingly widely used in both public and private life, in some cases within a statutory framework. Speech and language technology applications for these languages are therefore now becoming an urgent need. These are needed not only for monolingual content management, but also to aid translators and interpreters since the BIMLs are nearly always used in bilingual contexts alongside English. To develop such applications, basic language resources (such as corpora of machine-readable texts, machine-readable dictionaries, speech databases and so on) are required. 
The LER-BIML project at Lancaster is helping to address this need. Two recent EPSRC-funded projects at Lancaster (MILLE and EMILLE2) have provided a great service to the non-indigenous minority language communities in the UK by locating existing resources, investigating end-user needs and wants, examining basic technical issues and beginning to generate appropriate resources. However, no such consolidated survey or examination of issues has yet been undertaken for the BIMLs. 
Within the project we have recently undertaken two surveys, one to locate existing machine-readable resources and tools for these languages, the other to investigate end-user needs and wants. In our paper, we will present the results of these surveys. Secondly, we are building two pilot corpora of spoken Welsh and Scottish Gaelic, focussing in particular on “task-oriented” spoken activity types. We will discuss in our paper the issues raised in sampling, building and annotating our corpora, including our work on developing a set of EAGLES-conformant part-of-speech tagsets for all the BIMLs with which we shall then tag the two corpora. 
2. The Languages 
The indigenous minority languages of the British Isles and Ireland (BIMLs) are those languages other than English that are autochthonous to the territory of the United Kingdom, the Republic of Ireland and the Isle of Man. They comprise Cornish, Scottish Gaelic, Irish, Manx, Scots, Ulster Scots (Ullans) and Welsh. 
Along with most of the other major languages of Europe, these languages all belong to the Indo-European language family. However, they belong to different sub-branches of Indo-European. Cornish, Scottish Gaelic, Irish, Manx and Welsh belong to the Celtic branch of Indo-European. Celtic, in turn, divides into two distinct subgroups: P-Celtic (or Brythonic) and Q-Celtic (or Goidelic). Cornish and Welsh are P-Celtic languages, whilst Scottish Gaelic, Irish and Manx are Q-Celtic languages. The names for the two subgroups arise from the reflexes of the Proto-Indo-European *qw. In P-Celtic this became a bilabial stop (/p/), whilst in Q-Celtic it became a velar stop (/k/): compare the word for ‘four’ in Welsh (pedwar) and Scottish Gaelic (ceithir), both descended from Proto-Indo-European *qwetwor. There also exists a further P-Celtic language, Breton, which is spoken in north-western France. As this is not indigenous to the British Isles or Ireland, it does not fall within the scope of our project. For the same reason the Channel Island languages Jeriaise and Guernsiaise are not included. 
In contrast, the two varieties of Scots are not Celtic languages, but belong instead to the Germanic branch of Indo-European. Although they are very closely related to some dialects of English, they are recognized as separate languages. Welsh is widely used in all sectors of life in Wales and has joint official status with English. Although only 18.7% of the total population consider themselves to be Welsh-speaking (1991 Census figures), this figure rises to 43.7% in Dyfed and 61% in Gwynedd. 
1 This research is funded by the Engineering and Physical Sciences Research Council, GR/R63301/01. 
2 http://www.emille.lancs.ac.uk 
909
Welsh is a compulsory subject in the national curriculum in Wales. Furthermore, as a consequence of the Welsh Language Act 1993, the public sector must offer its services bilingually in Wales. Irish and Ullans have both been given increased recognition in Northern Ireland under the Northern Ireland Agreement, and the Northern Ireland Executive has pledged to promote them. According to the 1991 Census, only 1.4% of the population are speakers of Scottish Gaelic, although this figure will be rather higher regionally since almost all Gaelic speakers live in the highlands and western islands. Following devolution the Scottish Executive has given considerable priority and funding to maintaining and encouraging the use of Gaelic. For instance, the report of the Executive's Gaelic Taskforce (Gaelic: Revitalising Gaelic a National Asset) states that “as a foundation stone in the building of the new Scotland, the Gaelic language will be an integral and dynamic component of a robust and self-assured community with economic and social stability and pride in its linguistic and cultural identity” (Education Department, Scottish Executive 2000). Both Manx and Cornish have undergone revival, and there are now once again some native speakers who are bilingual in these languages and English. In terms of direct transmission, the last native speaker of Cornish died in the 18th century and the last native speaker of Manx in 1974. However, the use of Manx is receiving substantial support from the Isle of Man government and it is being taught again in schools. Cornish, at present, has no such public-sector support, although there is a strong pressure group (Agan Tavas) seeking to acquire this. 
3. Surveying existing resources 
The BIMLs are presently in various states of health as regards extent of current use. This initial survey of electronic resources was concerned with the manner and extent of their distribution and the level of current activity surrounding them.3 
3.1 Methodology 
The BIMLs as noted fall into the two language families of Celtic and Scots. As it has not previously been general practice to regard and treat these two families as one individual cohesive group, there is no one central governing body responsible for coordinating and making their resources and facilities widely available. Consequently work on these languages is widespread and yet sparsely distributed. It was evident from the outset that the Internet was going to be the main starting point for locating material and suitable contacts. It was intended that our main focus should be on text corpora and machine-readable texts, but also concentrating on locating speech databases, term banks, lexicons and language analysis tools such as taggers and parsers. The Internet has the obvious benefit of supplying all texts in electronic form. Search engines proved a useful initial means for generating a general idea of the volume of material available for the BIMLs, and from here subsequent leads and contacts were then followed up. What was apparent was that most of this material would require sifting; whilst there is a wealth of information in English about the BIMLs, the interests of this project lie, however, with resources directly available in the languages themselves.4 
3.2 Areas of resource 
The Irish, Ulster Scots and Welsh languages are all well represented by having official Language Boards and Agencies whose websites occur entirely in bilingual format and offer useful links.5 Whilst no such official bodies exist for the remaining BIMLs, there are organisations whose work is invaluable to promoting their respective language such as Agan Tavas for Cornish, Cli for Scottish Gaelic and Mannin.Org.Im for Manx. These sites include histories of the language, manuscripts, reference materials, news items, and merchandise. Most of these pressure groups offer discussion fora and mailing lists in English and in the language in question where reports are archived for public reading. Webzines are particularly popular and are a very good source of BIML resources as they form the focus of special interest groups with a targeted loyal audience. Examples are An Gannas for Cornish speakers, Beo for Irish, Wir Leid for Scots and Ullans.com. They offer comment, stories, puzzles, quizzes, jokes, polls, and reviews amongst others. The Mercator project which serves as an 
3 See Working Paper 1 (Wilson and Worth 2002) for full details of all URLS cited. 
4 To avoid confusion the term “BIML data/material” will refer exclusively to material written in the languages themselves, rather than in English. 
5 http://www.bnag.ie/index.htm; http://www.ulsterscotsagency.com/; http://www.bwrdd-yr-iaith.org.uk/ 
910
information network for minority languages of the European Union profiles Cornish, Scottish Gaelic, Irish and Welsh amongst these languages and directs towards associated resources. 
3.2.1 Current Projects 
Aside from MILLE and EMILLE there are several other various projects in progress sharing themes with those of LER-BIML. These include CELT6, an online database of ancient and contemporary Irish literary and historical texts, the National Corpus of Irish incorporating 15 million words from a variety of contemporary books, newspapers, periodicals and discourse, marked up in accordance with the PAROLE encoding standards and MELIN which has produced dictionaries, grammars, spellcheckers and terminology lists for the initial four EU minority languages, Irish, Welsh, Catalan and Basque. Their sites offer good links to BIML data and other websites. The Oxford Text Archive7 has a catalogue of several thousand electronic texts and linguistic corpora in a range of languages including standard reference works and mono and bilingual dictionaries. The Universities of Edinburgh and Glasgow have recently begun collaborative work on the SCOTS8 project that aims to build a collection of electronic spoken and written texts for the languages of Scotland incorporating Scots, Scottish English and Gaelic but focussing primarily on Scots. Previous work in this field has included a one million word lexical database and frequency count for Welsh (CEG), developed at Bangor from a broad range of modern Welsh text types (Ellis et al 2001), and two speech databases for Welsh produced at Edinburgh and Swansea (Williams 1998, 1999; Jones et al 1998). 
3.2.2 Education 
In assessing the relative volume of BIML resources, it was clear that Welsh has the most widely available material. This is a direct result of the Welsh Language Act 1993 which states that the public sector must offer its services bilingually, something that much of the private sector has now also adopted. In locating BIML data we therefore took specific note of whether the material appeared solely in the original language, or in bilingual format. One particular good example of Welsh parallel text is to be found on university homepages. Sabhal Mór Ostaig, a Further Education College on Skye, offers a comprehensive index to key resources in Scottish Gaelic and is a primary gateway for BIML data. ACCAC has a parallel site of exhaustive Welsh language and bilingual educational sites for ages 4-18. Dublin City University has a centre, FIONTAR, which administers academic programs entirely through the medium of Irish, and has a bilingual website outlining this. The Centre for Manx Studies obliged in sending an extensive list of Manx resources, which included the main resources page developed by the Manx Language Officer at the Department of Education. This page offers links to short stories, the Manx Language Society's newsletter Dhooraght and the magazine CARN, dictionaries, grammars and glossaries. 
There are interactive and self-teaching language courses available for all the BIMLs, ranging in style and intensity from the colloquial to the more grammatically orientated instruction, but which cater for all levels of learner. Most sites also supply various vocabulary and phrase lists and glossaries. 
3.2.3 Government 
All Welsh council and parliamentary sites including the National Assembly are legally required to be presented in bilingual format, and the Scottish Parliament and Northern Ireland Executive are following their lead.9 Health Authorities in Wales that have developed their own websites also present them in this bilingual format. There is obviously much more “official” material available for Welsh than any of the other BIMLs, but Gaelic and Irish are certainly increasing their profile. 
6 http://www.ucc.ie.celt 
7 http://ota.ahds.ac.uk 
8 http://www.scottishcorpus.ac.uk/ 
9 http://www.wales.gov.uk/index.htm; http://www.scottish.parliament.uk/; http://www.nics.gov.uk/ 
911
3.2.4 Media 
Media resources are strong in BIML data with various online newspapers, radio broadcasts and television schedules. Newspapers vary from presenting their entire content in the original language, like the Welsh weeklies Y Cymro and Golwg, and the Irish weeklies Foinse and Lá, to producing special reports like An Phoblacht, Ireland's leading weekly Republican newspaper (archived).10 BBC Online is available in Welsh, BBC Scotland has pages in Gaelic, BBC Cornwall produces a weekly audio five minute news bulletin and the Welsh and Irish stations S4C and TG4 respectively have bilingual sites.11 The recently launched BBC4 channel broadcasts programmes in Gaelic. RTE, the national Irish radio service, provides bulletins online and Raidió na Gaeltachta has an extensive bilingual site which supplies audio downloads of its broadcasts 24 hours a day. There is a text and audio weekly review of Manx Radio. Search engines and web browsers can be used in Welsh, Gaelic and Irish: the Opera Web Browser is available in Irish, Scottish Gaelic and Welsh as well as Breton, and a detailed guide to Welsh software can be found at Meddal. 
3.2.5 Arts and Literature 
Much of the literary corpora is reproduced on various Internet sites. Poetry and song lyrics are favourites, some with audio recordings, and are often used alongside language lessons. Short stories have been written by contributors to the webzines. There are several online book inquiry services and bookshops including the Welsh Books Council, and the National Library of Wales has a bilingual website. Details and adverts for film and music festivals and local events are often posted in the respective BIML and linked through webzines and special interest groups. The National Museums and Galleries of Wales website is in parallel format. 
3.2.6 Religion 
As regards religious texts, the Cornish Language Board (in conjunction with the Bishop of Truro's Ecumenical Advisory Group for sevices in Cornish) has translated several books of the Bible into Cornish. Various excerpts have also been translated into Manx and there are Manx, Scottish Gaelic and Welsh versions of the Book of Common Prayer.12 
3.2.7 Tools 
Online dictionaries are widely available for all the BIMLs and spellcheckers have been developed for Cornish, Scottish Gaelic, Irish, Manx and Welsh. Canolfan Bedwyr at the University of Wales, Bangor has published extensively in the area of specialist terminology dictionaries, and include amongst others glossaries of finance and education terms produced for the National Assembly of Wales. 
3.3 Conclusion 
There is a reasonably healthy volume of BIML resources available on the Internet, but as predicted Welsh is the most prominent and prolific of these languages. This prominence will be a direct result of the Welsh Language Act 1993. As there is no such official legislation as regards the other BIMLs there is a strong bias amongst these languages towards popular entertainment, particularly webzines, and archaic literature as they still rely primarily on specialist interest. They are, however, increasing their profile within more official and political contexts. The paucity in particular of Scots and Ulster Scots resources is probably best accounted for by the difficulty in distinguishing the boundaries between what is a dialect of English and what is something entirely recognisable as “Scots”. This is a much disputed issue.13 Bilingual text tends to be the most favoured format of presenting BIML data, as it caters for the interests of the native BIML speaker, whilst also recognising the need to extend its 
10 http://www.ycymronow.co.uk/; http://www.foinse.ie/; http://www.irlnet.com/aprn/ 
11 http://www.s4c.co.uk/; http://www.tg4.ie/ 
12 www.justus.anglican.org/resources/bcp/bcp.htm 
13 In 2001 the UK Government signed and ratified the European Charter for Regional or Minority Languages. As a result in Scotland and in Northern Ireland, Scots is recognised as a regional language, except that in Northern Ireland it is referred to as Ulster Scots. 
912
resources to the non-BIML speaker. Much material exists on the subject of these languages, but in English. The positive results of this survey therefore support the project's claims of the increasing profile of the BIMLs in the UK and Ireland today. 
4. Surveying end user needs 
4.1 Methodology 
Previous experience has taught that the most effective way of ensuring as wide a scope as possible of potential end-users would be by means of a web questionnaire posted on the project website. Notice of this questionnaire was emailed to over fifteen Internet bulletin boards and mailing lists including HUMANIST, CORPORA, TERMCELT and CELTLING. This secured the questionnaire being disseminated to all the BIML linguistic regions, and also outside of the British Isles and Ireland to groups working with the BIMLs as non-indigenous minority languages. The questionnaire focuses on the response to language engineering resources and corpus construction for the BIMLs by varying groups of users and as such forms the basis of the project's Working Paper 2 (Wilson and Worth 2002). 
4.2 Results 
There were 127 responses, 57 of which were interested in receiving feedback from the survey. This would be done by emailing out a copy of the report. 
Scottish Gaelic had the highest demand for corpus resources; there was no demand at all for Ulster Scots.14 There was strong interest in seeing more availability of resources for Breton and Shetlandic, with individual requests for the Channel Island languages and Romany amongst others. 
BIML No. responses Scottish Gaelic 86 Irish 65 Welsh 63 Scots 47 Cornish 43 Manx 42 Ulster Scots 0 
Table 1 Responses showing the areas of demand for (more) BIML corpus resources. 
A bilingual corpus was the most favoured corpus type: 
• to contain English alongside the BIMLs in question, 
• to contain sentence-aligned translations of the same texts in each language. 
Most wanted to see an equal balance of written and spoken data built for the BIMLs, and for this to be done within general balanced corpora rather than in genre specific corpora. For genre specific corpora news, history and fiction proved the most popular areas of interest. Whilst people thought it important to envisage the ideal of all types of genre being made available for the individual languages, suggestions other than those proposed on the questionnaire included arts and music, youth culture, environment, travel, technology, oral literature, folklore, food and drink, media and advertising and religion and ethics. 
As regards linguistically annotating the data, most would prefer just plain text. However, of the methods of annotation on offer, part-of-speech was the next most popular. The respondents would be 
14 This zero return could be explained by the presumption that the respondents identified more with the category Scots than Ulster Scots, or from a more negative point of view, that there is simply a lack of interest in this area. 
913
happy with anything that could be made available, although there was special mention of IPA and metaphorical and dialect annotation. The question of textual mark-up returned the highest number of nil responses, but amongst those who were interested in seeing mark-up, html was the favourite. 
The Internet was the favourite medium for receiving corpus data, with the CD a close second. 
On the issue of the listed features and their perceived importance within a corpus, the general consensus was that there was ‘no opinion’ on their preferred status. The only features which received a majority rating of ‘essential’ were the header elements ‘author', ‘source of data’ and ‘extent of language usage'. The spoken data features attracted the only number of ‘not wanted’ responses. 
The majority of respondents were linguists rather than language engineers. Applications which the language engineers envisaged using the BIML data to build included frequency tables, speech synthesis and recognition, spelling, style, syntax and grammar checkers, bilingual dictionaries and lexica and pedagogical tools. Questions the linguists wanted to explore with the data included effects of linguistic shift and borrowings; frequencies and variations of syntactic structures, dialect, registers and discourse; patterns of code switching across genres; reported versus actual usage; patterns of growth and decline; reception by young people. Suggested support tools included concordances, checkers, search and recognition tools, glossaries, taggers, text aligners and audio and video files. 
The optimistic end result was that there was an overwhelming majority (63%) that people were very likely to be working with the BIMLs in the future. 
Future activity No. responses Very likely 81 Possibly 25 Unsure 13 Probably not 6 Very unlikely 2 
Table 2 Response to how many linguists plan to work with BIML corpus data in the future. 
4.3 Conclusion 
The encouraging number of responses to the survey indicated that work in progress and current activity regarding the BIMLs is healthy and positive. The higher demand for Scottish Gaelic, Irish and Welsh most likely correlates with the more specialised mailing lists that cater specifically for these languages and are in much wider use. An amendment for a future survey of this type would be to ask the respondents to indicate from which mailing list they received details of the questionnaire to secure a better overview of the distribution of the respondents. It would also be beneficial to determine the professional status of the respondents to indicate from what type of linguistic background they are approaching the data. These factors could help resolve why these results might appear to go somewhat against expectation in light of the results of the previous survey of existing resources. 
5. Creating the corpora 
The BIMLs, especially the spoken languages, have been relatively overlooked within a corpus-based framework and in order to investigate the practical and technical issues that they raise for language engineering development it is necessary to build corpora as a testbed for such research. Two corpora are currently under development, one for Welsh and one for Gaelic, each of 40,000 – 50,000 words. Size is clearly not the aim of this phase of work; rather the focus is on practical and technical feasibility. We are following the framework of the British National Corpus World Index for spoken data collection with a bias towards task-orientated data collection. 75% of the data for each corpus is being drawn from context governed “task-oriented” spoken activity types, and the other 25% from general everyday conversational speech. 
914
LER BIML Spoken Corpus 
40,000 – 50,000 words each for Welsh and Scottish Gaelic 
Context Governed speech Demographic speech 
« « 
40,000 words (75%) 10,000 words (25%) 
Business Education Leisure Public Services 
10,000 words 10,000 words 10,000 words 10,000 words 
It was the aim to include, as far as is possible for the language in question, at least one sample of each of the main activity types identified by Crowdy (1993) for the British National Corpus covering business, education, leisure and public services. A range of private and public companies and organisations were then contacted inviting them to participate in the project and supply appropriate speech data. Material has consequently been offered by medical and dental practices, cathedrals and churches, schools and universities, television networks and various individuals for the conversational speech. The speech data will be recorded onto mindiscs (although in some cases data has also been supplied on audio and video tape) and released as sound files. This will not only ease the transfer of the data onto computer, but minidiscs have been found to produce a high standard of audio recording. 
The ethical issues of confidentiality and consent are key priorities in assembling data of this type. We have followed Sampson's (2000) advice for anonymisation practices. Information sheets have been issued to all who have been approached to participate, explicitly stating that anonymity is assured and all confidential information will be bleeped out, and names and personal details duly altered to protect identity. Informed consent must be given by all who agree to be recorded, giving permission for the material to be used and published as the project demands. It is our intention to release the stored digitised sound waves alongside the final corpora. The additional option is given to all participants that they may ask that their recording not be released as a sound file should they prefer this. In accordance with copyright regulations (Ward 1995 and 2000), anyone who participates and supplies data has copyright of their own words and must also give signed consent for the recordings to be made publicly available. For the medical data, consent had to be granted by the dental defence union or society and the Local Research Ethics Committee for the relevant NHS Trust, and parental consent has to be granted by all those who participate under the age of 16. 
6. Transcription and annotation 
The data will be transcribed orthographically (though not phonetically) and will be encoded in a TEI-conformant format. The header and transcription guidelines for the LER-BIML corpora are based on the standards for spoken language encoding developed in the EMILLE project (see http://www.emille.lancs.ac.uk/spoken.htm), which, in turn, incorporate the recommendations of several user surveys and standards initiatives, including the EAGLES recommendations on dialogue (Leech, Weisser, Wilson & Grice 2000). 
The two corpora will then be annotated with parts of speech using tagsets which conform to the EAGLES recommendations for morphosyntactic annotation of corpora (Leech & Wilson 1999). Two tagsets exist for each language: a base tagset implementing the EAGLES obligatory features (i.e. major parts of speech only) and an extended tagset which also implements EAGLES recommended and language-specific features. (Since the EAGLES project did not address the Celtic languages, necessary language-specific features have been identified in consultation with experts on the various languages.) The Welsh extended tagset was devised within LER-BIML (Wilson 2002), whilst, for Gaelic, we will be using the PAROLE tagset developed for the National Corpus of Irish at the Linguistics Institute of Ireland.15 In view of the close relationship between Irish and Scottish Gaelic, it has been determined that this tagset is also appropriate for the annotation of the latter. Although we will not be collecting 915 15 http://www.ite.ie/pos.htm 

and annotating corpora of Cornish or Manx, extended tagsets have also been devised for these languages (Mills 2002, Phillips 2002). 
For the annotation of the Welsh corpus, an automatic tagger (Hepple 2000) is currently being re-trained using the annotated CEG corpus of written Welsh (Ellis et al 2001). It is envisaged that this tagger will apply only the base tagset for Welsh. This will allow more scope for the Gaelic corpus to be tagged by hand using the extended tagset. However, we also envisage re-training the tagger for Gaelic in the near future. 
7. Conclusion 
As has already happened to a great extent in Wales, the BIMLs look set to become increasingly important as the devolved governments, and consequent ethnic identities become more firmly established. Therefore although some BIMLs are presently spoken only by small numbers, it is nevertheless important strategically to begin to provide language engineering solutions for them. This fact has already been recognized at a policy level by EPSRC, who state that “the multilinguality of language engineering applications is of growing importance if these applications are to be equal use to all users of the many different languages and accents in the UK” and, furthermore, that “in EPSRC terms, the languages of particular importance are the indigenous minority languages of the UK (Welsh and Scots Gaelic), and the non-indigenous minority languages (such as Urdu, Hindi, Chinese, etc)...” (EPSRC 1999). Our project is the first step towards providing data for these applications in the Celtic languages. 
8. References 
Crowdy S 1993 Spoken Corpus Design. Literary and Linguistic Computing 8(4), 259-65. 
Education Department, Scottish Executive 2000 Gaelic: Revitalising Gaelic a National Asset. Report by the taskforce on public funding of Gaelic. Retrieved February 2003 from URL http://www.scotland.gov.uk/library3/heritage/gtfr-00.asp 
Ellis NC, O'Dochartaigh C, Hicks W, Morgan M, Laporte, N 2001 Cronfa Electroneg o Gymraeg (CEG): A 1 million word lexical database and frequency count for Welsh. Retrieved February 2003 from URL http://www.bangor.ac.uk/ar/cb/ceg/ceg_eng.html 
EPSRC 1999 People and Interactivity: Incorporating the Human Factors Review 1998/1999: The Results of the Consultation and Plans for the Future. 
Hepple M 2000 Independence and commitment: Assumptions for rapid training and execution of rule-based POS taggers. In Proceedings of the 38th Annual Meeting of the Association for Computational Linguistics (ACL-2000), Hong Kong, October 2000. 
Jones R, Mason J, Jones R, Helliker L, Pawlewski M 1998 SpeechDat Cymru: A large-scale Welsh telephony database, Proceedings of the LREC Workshop: Language Resources for European Minority Languages. 
Leech G, Weisser M, Wilson A, Grice M 2000 Representation and annotation of dialogue. In Gibbon D, Mertins I, Moore R (Ed.), Handbook of multimodal and spoken dialogue systems, pp.1-101. Kluwer, Dordrecht. 
Leech G, Wilson A 1999 Standards for tagsets. In van Halteren H (Ed.), Syntactic wordclass tagging, pp.55-80. Kluwer, Dordrecht. 
McEnery A, Sebba M, Burnard L 1999 Minority Language Engineering (MILLE) – Final report. Report to EPSRC, Lancaster University. 
Mills J 2002 Suggestions for a morphosyntactic tagset for Cornish, based on the EAGLES obligatory and recommended attributes. Retrieved February 2003 from URL http://www.ling.lancs.ac.uk/biml/cornish_tags.htm 
916
Oxford University Computing Services 2001 Second Edition British National Corpus World Edition. Retrieved February 2003 from URL http://www.hcu.ox.ac.uk/BNC/ 
Phillips JD 2002 A tagset for Manx. Retrieved February 2003 from URL http://www.ling.lancs.ac.uk/biml/manx.txt 
Sampson G 2000 CHRISTINE Corpus, Stage 1 Documentation Release 2. Retrieved February 2003 from URL http://www.cogs.susx.ac.uk/users/geoffs/ChrisDoc.html 
Ward A 2000 Copyright and Oral History, [Internet], Oral History Society. Retrieved February 2003 from URL http://www.nmgw.ac.uk/~ohs/ohs/copyright.html 
Ward A 1995 Copyright, Ethics and Oral History, Colchester: The Oral History Society. 
Williams B 1998 Levels of annotation for a Welsh speech database for phonetic research. 
Paper from conference proceedings: Workshop on Language Resources for European Minority Languages ( B. Williams, C. Nadeu, A. Monaghan, eds. ) [ 27th May 1998 ] Granada, Spain. 
Williams B 1999 A Welsh speech database: preliminary results. In Eurospeech 99 (European Conference on Speech Communication and Technology) [ 5-10 September 1999], Budapest, Hungary. 
Wilson A 2002a Suggestions for a morphosyntactic tagset for Welsh, based on the EAGLES obligatory and recommended attributes. Retrieved February 2003 from URL http://www.ling.lancs.ac.uk/biml/welsh_tags.html 
Wilson A, Worth C 2002b LER-BIML Working Paper 1: Surveying existing resources for the indigenous minority languages of the British Isles and Ireland, Lancaster University. Retrieved February 2003 from URL http://www.ling.lancs.ac.uk/biml/bimls3reports1.htm 
Wilson A, Worth C 2002 LER-BIML Working Paper2: Surveying end user needs for the indigenous minority languages of the British Isles and Ireland, Lancaster University. Retrieved February 2003 from URL http://www.ling.lancs.ac.uk/biml/bimls3reports2.htm 
917

Conceptual Glossaries of the Latin Vulgate Bible 
Andrew Wilson & Celia Worth 
Department of Linguistics and MEL 
Lancaster University 
Abstract 
A conceptual glossary is a textual reference work that combines the features of a thesaurus and an index verborum. In it, the word occurrences within a given text are classified, disambiguated, and indexed according to their membership of a set of conceptual (i.e. semantic) fields. Since 1994, we have been working towards building a set of conceptual glossaries for the Latin Vulgate Bible. So far, we have published a conceptual glossary to the Gospel according to John and are presently reaching the completion of the analysis of the Gospel according to Mark and the minor epistles (supported by a British Academy Small Research Grant). This poster will describe the background to our project and outlines the steps by which the glossaries are developed within a relational database framework. 
918

Quantitative or Qualitative Content Analysis? Experiences from a cross-cultural comparison of female students' attitudes to shoe fashions in Germany, Poland and Russia 
Andrew Wilson & Olga Moudraia 
Department of Linguistics and MEL 
Lancaster University 
The study of apparel and its meanings is of interest to linguists, in that it can help throw light on what different human communication codes have in common and what is unique to natural spoken/written language. However, it is also the case that we can only get at the meanings of apparel items by using natural language as a descriptive metalanguage. 
Although some early work on apparel meanings made use of free text response (e.g. Stone 1962), much of the recent research in the field has been carried out in an experimental Likert-scale-led paradigm (cf. Davis & Lennon 1988). The ethnomethodological free-text approach is arguably more valuable, however, because it can provide much richer data than the experimental approach: it taps into the actual attitudes and perceptions of the informants rather than having them agree or disagree with the researchers’ set of attitudes (cf. Golliher 1987). It was thus a variant of this data type that we selected for our study. 
Since much of the previous research on apparel meanings had been carried out in the USA, we wanted to examine the possibility of cross-cultural differences within Europe. In order to do this, we obtained a corpus of compositions from advanced students of English in three European locations: Greifswald (Germany), £ódŸ (Poland), and Volgograd (Russia). The students were asked to respond to three general stimulus questions, including one on shoes. It is this question that we consider here. 
Our initial aim was to analyse these sets of compositions quantitatively using Lancaster's USAS software for content analysis (Wilson & Rayson 1993; Rayson & Wilson 1996), in order determine which concepts predominated and how they differed between groups. However, our initial quantitative results revealed relatively little detailed information about the students’ classifications and schemata. To study these, we thus had to fall back on a form of “qualitative content analysis” (Kracauer 1952) based around communicative-functional language analysis (cf. Boeck 1981). 
In this paper, we will describe the results of our analysis, and will also explore some of the issues that are encountered in analysing free-text questionnaire responses such as these. We will address the relative merits of quantitative and qualitative content analysis and also look at some of the problems of operationalizing a qualitative content analysis. 
References 
Boeck, W. 1981. Kommunikativ-funktionale Sprachbetrachtung als theoretische Grundlage für den Fremdsprachenunterricht. Leipzig: Verlag Enzyklopädie. 
Davis, L.L. & Lennon, S.J. 1988. Social cognition and the study of clothing and human behavior. Social Behavior and Personality 16(2): 175-186. 
Golliher, J.M. 1987. The meaning of bodily artifacts: variation in domain structure, communicative functions, and social contexts. Semiotica 65(1/2): 107-127. 
Kracauer, S. 1952. The challenge of qualitative content analysis. Public Opinion Quarterly, 16, 631-642. 
Rayson, P. & Wilson, A. 1996. The ACAMRIT semantic tagging system: progress report. In L.J. Evett, and T.G. Rose (eds) Language Engineering for Document Analysis and Recognition, LEDAR, AISB96 Workshop proceedings, pp 13-20. Brighton, England. Faculty of Engineering and Computing, Nottingham Trent University, UK. 
Stone, G.P. 1962 "Appearance and the Self." Pp. 86-118 in Human Behavior and Social Processes, edited by Arnold Rose. Boston: Houghton Mifflin. 
Wilson, A. & Rayson, P. 1993. Automatic Content Analysis of Spoken Discourse. In: C. Souter and E. Atwell (eds), Corpus Based Computational Linguistics. Amsterdam: Rodopi. pp. 215-226 
919

  920 
Virtual Corpora at the Oxford Text Archive 
Martin Wynne, Rowan Wilson and Ylva Berglund 
Oxford Text Archive, Oxford University 
1  Deeper and wider access to electronic text holdings 
The Oxford Text Archive (OTA) has been collecting and distributing electronic texts since 1976. It 
now hosts the Arts and Humanities Data Service (AHDS) Centre for Literature, Languages and 
Linguistics, and provides a service as a data centre and a source of expertise and advice on the creation, 
storage, preservation, distribution and use of digital resources. 
The AHDS has a commitment to providing improved resource discovery mechanisms for its collections 
and to its collections. The OTA is committed to keeping up with current and future developments 
(Wynne 2002) and to offering deeper and wider access to the archive. 
In terms of resource discovery and wider access, various initiatives are underway, which are mentioned 
now but not covered in depth here, such as participation in the Open Archives Initiative, including the 
Open Language Archives Community (OLAC) and various portals projects. 
In terms of deeper access, we aim to make more texts more easily available and to improve online 
access to the electronic texts in our collection. This paper focuses on the keystone of the OTA's 
initiative to provide deeper access, the new online virtual corpus and online query system for the 
holdings of the Oxford Text Archive is described, demonstrated and explained.  
2  From magnetic tape to open archives 
The Oxford Text Archive is a large repository of electronic texts and corpora. In many ways, 
the archive has worked in much the same way since its inception. The user consults the catalogue, 
selects a text or a number of texts and then completes the relevant procedure in order to download the 
text or texts to their computer. The main development in terms of resource delivery in the past 27 years 
is that many of the resources can now be downloaded directly from the website, rather than being sent 
by post on magnetic media or downloaded by ftp. Users are then left to their own devices to find 
software to analyse the texts and try to extract information from them. 
In order to make the archive more useful and usable for linguistics researchers, learners and 
teachers, a system for the querying of texts online has now been developed at the Oxford Text Archive. 
It is possible for the user to construct a corpus of texts from the archive for downloading or querying 
online. Concordances and collocation information can be generated and displayed. Texts for this userdefined
 corpus can be selected on the basis of any of the metadata categories in the resource 
description, and the text selection can then be refined by simply picking and choosing from a list of 
texts generated by the initial query. 
3  Virtual corpora and online delivery 
Online concordancing is not new. Many sites and corpus projects offer this facility. Neither is 
the ability to select the texts on the fly and thus construct a virtual corpus a new innovation. The 
novelty of the new OTA service lies in the application of both a flexible text selection functionality and 
an online query system to a large online archive. 
A system of virtual corpus building based on fine-grained and detailed text-level metadata is 
implemented at the Max Planck Institut in Nijmegen (Wittenberg and Broeder 2002). This differs from 
the OTA system in that it is primarily for reassembling the contents of language corpora, rather than a 
means of using an archive of hitherto discrete and non-interoperable texts for virtual corpus building. 
There is also some use of the term virtual corpus referring to the use of the web as a corpus. 
Indeed, a logical extension of the concept of virtual corpus building as applied to the Oxford Text 
Archive could be to apply the concept to texts held in disparate locations on the web. There are 
certainly initiatives in this direction, one of the most significant of which is the Webcorp system 
(Renouf 2001). However, researchers currently find several important problems with the use of the web 
as corpus, particularly in the case of English language texts. One problem is the identification of the 
language of a page, or text which is part of a page. Another is the amount of non-native speaker 
writing, and the difficulty of identifying it. Similar and related problems are encountered with regional 
varieties. There are also strong biases in terms of the types of text which are present on the web, and in 
terms of the relative frequencies of different text types. For example, computing manuals and 
pornography are strongly represented, and formal correspondence is rare, in contrast to the relative 
importance of these text types in the language in general. There is also the problems of the so-called 
‘invisible web', whereby much textual content is served on the fly to browsers and is not available to 
the search engines and spider programs which would typically be expected to build corpora on the web. 
  921 
It is also often difficult to identify any information at all about texts, or at best it is difficult to 
have any confidence about the accuracy of information. To put it simply, textual metadata is sparse, 
unreliable and implemented in non-standard ways. 
The problem of metadata is addressed by proposals for the semantic web. It will be interesting 
to see what use linguists can make of this for corpus building and online linguistic analysis. However, 
even if satisfactory standards for encoding metadata for content on the web can be formulated, there 
will still be questions about the reliability of the implementation of these standards in any given case. 
The semantic web can be seen as one step towards using the web as a corpus, or of making 
corpora from texts on the web, by establishing interoperable textual metadata. Another step would be to 
distribute the text processing tasks. Interestingly there is a project to realise the distribution of data 
processing on the internet, in a initiative known as the Grid: 
The Grid is the name that describes the next significant development in Internet computing. A term first 
coined in the mid '90s to describe a vision for a distributed computing infrastructure for advanced science 
projects, the Grid was first properly explained by Ian Foster and Carl Kesselman in their book The Grid: 
Blueprint for a New Computing Infrastructure(Morgan Kaufmann, 1999; ISBN 1-55860-475-8). In this 
vision, the Grid will be to all computational resources what the World Wide Web presently is to 
documents containing information. Grid users will have at their disposal distributed high performance 
computers able to access and process terabytes of data stored in global databases, plus the appropriate 
tools to control these resources. The Grid is similar to Tim Berners-Lee's vision of theSemantic Web, an 
information space of meaning built upon the World Wide Web that he invented, but it extends that vision. 
At present, the Grid is in a similar state of infancy to that in which the World Wide Web was 10 years ago, 
when it was little more than "a very slow way to ftp files between distant computers". However, while the 
World Wide Web is simple to define in terms of a language for organizing information (HTML) and an 
information transport protocol for delivering it (HTTP), the form that the Grid will take is more complex, 
and the path of its development unclear. There are even debates about whether there will be different 
kinds of Grids for different purposes, one concerned primarily with number-crunching tasks on distributed 
supercomputers, another involved in integrating disparate information resources, private corporate Grids, 
etc. But that the Grid revolution is almost upon us is without doubt. 
(Oxford e.science centre website, http://e-science.ox.ac.uk/) 
At the OTA we have the advantage of being closely linked with the latest developments in the 
Grid and e-science (the term coined for research done using the Grid through a regional e-science 
centre based alongside the OTA in the Research Technologies Services of the Oxford University 
Computing Services. Interestingly, much of the early work on setting up the Grid and e-science 
projects is concerned with authentification, certification and the identification of trusted sites, users and 
data. It seems that both with e-science and with attempts to do linguistic analysis on the web, a crucial 
issue is the identification of reliable quality data sources, and this is constraining the more utopian 
visions of the possibilities for global distributed computing. 
At the current juncture then, our ambitions are restricted to working with mature and proven 
technologies, and therefore we aim to concentrate on providing virtual corpora at the OTA based on 
texts in our archive. The OTA is a trusted repository of quality texts created for academic research, and 
so there is some guarantee of text quality. It will however be interesting to watch and experiment 
ourselves with attempts to widen the concept to texts and tools in distributed locations in cyberspace. 
4  Making virtual corpora at the OTA 
A specific challenge of providing a virtual corpus service using the holdings of the Oxford 
Text Archive lies in the heterogenous nature of texts. There are more than two thousand texts in the 
archive and they have been collected and documented over a period of more than twenty-five years. 
They reflect a multitude of different practices in the encoding of the texts, in the construction of 
collections of texts, and in the documentation of the resources. The size and diversity of the archive 
makes it a potentially extremely rich linguistic resource. It is however necessary for the text selection 
and analysis software to be able to deal in a consistent fashion with the different texts and metadata. 
The first stage in the development of the virtual corpus system at the OTA was a selection of 
the texts which were to be used. Selection criteria were established with the aim of making it possible 
to construct a variety of corpora based on literary and non-literary prose in English from the 18th to the 
early 20th centuries. The texts selected were therefore all prose (excluding drama and poetry) in 
English, both fiction and non-fiction. A handful of texts which fall outside of these categories were 
included so that the text selection functionality could be tested, and so there are some earlier and some 
later texts, some poetry and drama, but not enough to make a representative corpus of these text types. 
Within the set of texts fitting the selection criteria, specific texts were chosen on the basis of textual 
  922 
integrity and markup. Less reliable texts were excluded. In terms of text encoding and markup, texts 
with no textual markup and with SGML (or XML) markup were selected. Linguistic and textual 
annotation encoded in non-standard ways was stripped out, in order to ensure that only the text was 
searched by the linguistic analysis software. SGML markup was left in because this can easily be 
identified automatically and differentiated from the text. 
As texts were selected, some additional information was added to the database of metadata. 
Categories identifying the sex and nationality of authors, original language (if the text is a translation), 
text type and the date of original publication of the printed work on which the electronic edition is 
based. These categories were chosen on the basis that they would be useful for linguistic researchers to 
attempting to control variation or identify correspondences. These categories were not used as selection 
criteria for the texts included in the system at this stage. 
The text selection and analysis software is implemented using CGI scripts written in Perl. 
These scripts access, manipulate and display metadata about the texts, and sections of the texts 
themselves. All back-end data is stored in a series of tables in a MySQL database. The Perl DBI 
module is used to pass queries from the script to the MySQL database, and to transmit the resulting 
tables back to the PERL script for display. The corpora generated by users' metadata queries are stored 
in temporary tables within the database. This allows comprehensive indexes to be generated and 
delivers increased performance. A house-keeping script is run periodically to handle the retiring of 
these temporary tables. 
In the following section, the functionality of the system is demonstrated by working through 
an example query. 
 
   
 
 
Session 'DEMO' 
 
Select your texts: 
   
PERIOD:  To   
AUTHOR:  Twain
 
NATIONALITY: 
British
US
Irish
German
Canadian
French
Russian
 
TEXT TYPE:  
Fiction
Non-Fiction
Poetry
Drama
 
SEX:  Male
Female  
Find Texts ^
 
 
Figure 1: the text selection form 
  923 
5  Extracting information from virtual corpora 
In this section, a small piece of lexical research is carried out. It has been observed that Mark 
Twain frequently uses the word ‘presently’ in his prose. A comparison of his writings with the Brown 
Corpus (a one million word representative corpus of written American English from the 1960s) using 
the keywords function of Wordsmith tools shows that the word ‘presently’ shows strongly as a keyword 
in Twain's writing. It occurs 438 times in 1.3 million words of Twain's writing and just 35 times in 1 
million words in the corpus. However, the mismatch between the type of language in the texts under 
investigation and the reference corpus in this case means that these results may merely reflect 
differences between fiction and non-fiction (since most of Brown is non-fiction), or change over time, 
since Twain's work is from the late 19th and early 20th centuries and the Brown corpus is from the 
1960s.  It would be more useful to compare Twain's writing with a corpus representative of his type of 
writing, from a similar time and place. 
What is therefore desirable is a corpus of Mark Twain's writing, plus a representative corpus 
of literary writing of his American contemporaries, and some tools for the simple lexical analysis of 
these corpora. The rest of this subsections details how this can be achieved with the OTA Virtual 
Corpus system. 
 
     
 
Matching texts: 
   
TEXTS:    Author  Title  Date  Sex  Nationality 
  Twain, Mark  Adventures of Huckleberry Finn   1885  M  US 
  Twain, Mark  The tragedy of Pudd'nhead Wilson   1894  M  US 
  Twain, Mark  Roughing it   1871  M  US 
  Twain, Mark  A ghost story   1870  M  US 
  Twain, Mark  Tom Sawyer abroad   1894  M  US 
  Twain, Mark  Extracts from Adam's diary   1893  M  US 
  Twain, Mark  Tom Sawyer, detective   1896  M  US 
  Twain, Mark  The great revolution in Pitcairn   1879  M  US 
  Twain, Mark  A Connecticut yankee in King 
Arthur's court  
1889  M  US 
  Twain, Mark 
The adventures of Huckleberry Finn 
: (Tom Sawyer's Comrade)  
1885  M  US 
  Twain, Mark  The adventures of Tom Sawyer   1876  M  US 
  Twain, Mark  A tramp abroad   1880  M  US 
 
Twain, Mark  Niagra   1869  M  US 
 
SELECT: 
Checks indicate inclusion 
Checks indicate exclusion 
Select These Texts
 
 
Figure 2: the text listing generated from the ‘Twain’ query 
  924 
5.1  Making a Mark Twain corpus 
Figure 1 shows the input form for selecting texts based on metadata categories. In this simple 
form, the user has a limited set of categories in which values may be selected or input. In a later stage 
of the development of the software we will also make available an ‘advanced’ form through which all 
metadata categories may be queried. 
In this example, a corpus of writings of Mark Twain is created by specifying the author's 
surname, which should be sufficient for an accurate search in this case. 
Clicking on ‘Find texts’ will load the page shown in Figure 2. This shows a listing of the texts 
which match the values entered in the text selection form. The user now has an opportunity to view and 
edit the list, or perhaps go back and change or refine the selection. At this stage the user may find that 
there are not sufficient texts matching the specified criteria to make a useful corpus. 
There are also multiple copies of some texts in the archive, so the user needs to be aware of 
this and to unselect repeated entries. 
Clicking on ‘Select these texts’ causes the user-defined corpus to be constructed from the 
selected texts. This involves building a database of all the words in the corpus, and so takes some time. 
The resulting corpus will however remain available to the user for the length of the current session, and 
so the user will not need to keep rebuilding. 
 
     
 
Selected texts: 
   
TEXTS:  >> Adventures of Huckleberry Finn - Twain, Mark - 1885  
>> Life on the Mississippi - Twain, Mark - 1883  
>> The tragedy of Pudd'nhead Wilson - Twain, Mark - 1894  
>> The Innocents abroad , or, The new Pilgrim's progress : being some account of the steamship Quaker City's 
pleasure excursion to Europe and the Holy Land : with descriptions of countries, nations, incidents and adventures, as 
they appeared to the author - Twain, Mark - 1869  
>> Roughing it - Twain, Mark - 1871  
>> A ghost story - Twain, Mark - 1870  
>> Tom Sawyer abroad - Twain, Mark - 1894  
>> Extracts from Adam's diary - Twain, Mark - 1893  
>> Tom Sawyer, detective - Twain, Mark - 1896  
>> The great revolution in Pitcairn - Twain, Mark - 1879  
>> A Connecticut yankee in King Arthur's court - Twain, Mark - 1889  
>> What is man? and other essays of Mark Twain - Twain, Mark - 1917  
>> The adventures of Huckleberry Finn : (Tom Sawyer's Comrade) - Twain, Mark - 1885  
>> The adventures of Tom Sawyer - Twain, Mark - 1876  
>> A tramp abroad - Twain, Mark - 1880  
>> Niagra - Twain, Mark - 1869  
WORD 
COUNT:  1311930  
Edit Selection ^
 
 
Enter Query: 
   
QUERY:  presently
 
QUERY 
TYPE:  Exact match
 
Submit Query >>>
 
 
Figure 3: entering the search query 
A word count of the new corpus is given, and the user has an option to go back and edit the 
text selection again. The user can choose whether to download the corpus of selected texts or to submit 
  925 
online queries to the corpus1. Downloading the corpus enables the user to carry out their analysis of the 
corpus unrestrained by the limited functionality of the OTA online service. This is considered an 
essential aspect of the functionality, as the authors do not wish to try to prescribe or predict all of the 
functions and options users might wish to have available for the analysis of the corpus. It is expected 
that some users will have their own tools which they want to run locally to process the data. They may 
also have their own text collections which they wish to integrate or compare with their virtual corpus. 
If the user does not opt to download the corpus and do the analysis locally, they can now enter 
the search query string in the form shown in Figure 3. The query type may also be selected, allowing 
the use of wildcards and substrings. In this case, we search for the word ‘presently'. Clicking on 
‘Submit query’ loads the concordance display shown in Figure 42. 
 
 
Figure 4: concordance display for ‘presently’ in the Twain corpus 
There are options for sorting the lines, finding the source text for a given line, and expanding 
the context. Clicking on the key word will give a list of collocates in the corpus. In this case the most 
frequent lexical collocates appear to be words involved in advancing the narrative (‘began'), as may be 
expected from the meaning of ‘presently', and in particular to do with movement (‘came', ‘away') and 
reported speech (‘says'). The way in which these words are used in co-occurrence with the keyword 
can easily be checked, as the entries in the collocation list are links to concordance lines where the two 
words co-occur. It is important to go back to the concordance lines in this way and look at the 
meanings of particular collocations if the interpretation of the collocations listings is not to be mere 
speculation. 
In this case, there are 438 occurrences of ‘presently, out of a total corpus of 1311930 words. 
This represents a relative frequency of 0.03%. 
The next section will describe how a reference corpus can be built and will compare the use of 
the word ‘presently’ in this corpus. 
 
                                                          
1 The option to download the corpus is not implemented in the version shown in Figure 3. 
2 A screen shot has been used for figure 4 to show the concordance output, because the HTML table produced is not the correct 
shape for this publication. The text is difficult to read the text in this picture, but it is intended only to demonstrate that a KWIC 
concordance is generated. 
  926 
5.2  Making a reference corpus 
It is first necessary to specify the selection criteria for the reference corpus. In this case a 
decision was made to select text from the time period 1850 to 1920 (roughly Twain's adult life), by US 
authors, prose and both fiction and non-fiction. Clicking on ‘Find texts’ therefore results in a list of the 
texts in the archive which are deemed to be of a similar type to Mark Twain's writings. The results are 
shown in Figure 6.  
 
     
 
Select your texts: 
   
PERIOD:  1850
To 
1920
 
AUTHOR: 
 
NATIONALITY: 
British
US
Irish
German
Canadian
French
Russian
 
TEXT TYPE:  
Fiction
Non-Fiction
Poetry
Drama
 
SEX:  Male
Female  
Find Texts ^
 
 
Figure 5: metadata selection for the reference corpus 
The listing in figure 6 only shows the first screenful of matching texts. (The full, final 
selection can be seen in Figure 7.). In this case the user needs to scroll down to see all of the texts. The 
first thing which it is necessary to do is to unselect the works of Mark Twain, which will also be in the 
list as they match the specified criteria. 
At this stage the user needs to critically analyse the corpus composition. The value of the 
results of any comparisons will depend on quality of the sample. Texts which exist in more than edition 
need to be identified if duplication is to be avoided. The user must also use this list of texts in order to 
identify the biases in the data. There may be too many works by a particular author or of a particular 
type which make the corpus unrepresentative. It may also be desirable to restrict the size of the corpus 
to a particular limit, in order to speed up the process, or produce a manageable number of hits, or to 
have a reference corpus of a similar size to the test corpus for particular statistical comparisons. 
Once the user is happy with the selection, clicking on ‘Select these texts’ will take the user to 
the online query input stage (Figure 7). The full list of texts selected from the archive can be seen in 
Figure 7, along with the total number of words in the reference corpus, a little over 2 million. 
 
 
 
 
  927 
 
     
 
Matching texts: 
   
TEXTS: 
  Author  Title  Date  Sex  Nationality 
 
Alcott, Louisa May  Little women   1869  F  US 
  Alger, Horatio  Ragged Dick   1868  M  US 
  Bierce, Ambrose  Can such things be   1893  M  US 
 
Burnett, Frances 
Hodgson 
The secret garden   1911  F  US 
 
Burnett, Frances 
Hodgson 
A little princess   1905  F  US 
  Haggard, H. Rider  King Solomon's mines   1885  M  US 
  Hawthorne, Nathaniel  The Blithedale romance   1852  M  US 
  Hawthorne, Nathaniel  The house of the seven gables   1883  M  US 
  Hawthorne, Nathaniel  The marble faun : or, The 
romance of Monte Beni  
1860  M  US 
  Hawthorne, Nathaniel  The scarlet letter   1850  M  US 
 
Henry, O  The gift of the magi   1906  M  US 
  Howells, William Dean  The rise of Silas Lapham   1885  M  US 
  James, Henry  Novels 1871-1880   1880  M  US 
  James, Henry  American writers : Henry James   1912  M  US 
 
SELECT: 
Checks indicate inclusion 
Checks indicate exclusion 
Select These Texts
 
 
Figure 6: list of texts matching the specified metadata (abbreviated) 
At this stage the search term can be entered, so the user now simply types ‘presently’ and 
clicks on ‘Submit query', as was done with the Twain corpus. The concordance display is now loaded, 
and shown in Figure 8. 
In this case, there are 201 hits for ‘presently', which represents a relative frequency of 0.01% 
(201/2003881). This can be compared with the relative frequency in the Twain corpus of 0.03%, which 
can be seen to be higher. This would appear to support the hypothesis that Twain uses the word more 
than the norm, and could be a useful starting point for further investigation. 
The usefulness of any reference or representative corpus that can be created using the virtual 
corpus system in this way will depend on what is available in the archive. The archive has necessarily 
been built up over the years depending on what texts were offered for deposit, and so no guarantees of 
coverage for any given text type can be offered. However, it may now be considered a priority for the 
  928 
OTA to adopt a collections development policy which will aim to fill gaps and expand the coverage so 
that representative corpora can more readily be constructed. 
     
 
Selected texts: 
   
TEXTS:  >> No treason : the constitution of no authority - Spooner, Lysander - 1870  
>> The Blithedale romance - Hawthorne, Nathaniel - 1852  
>> The house of the seven gables - Hawthorne, Nathaniel - 1883  
>> The marble faun : or, The romance of Monte Beni - Hawthorne, Nathaniel - 1860  
>> Essays on literature - James, Henry - 1915  
>> Cape Cod - Thoreau, Henry David - 1865  
>> Supplementary prose - Whitman, Walt - 1892  
>> The story of the other wise man - Van Dyke, Henry - 1896  
>> The Devil's dictionary - Bierce, Ambrose - 1906  
>> The gift of the magi - Henry, O - 1906  
>> Confidence - James, Henry - 1879  
>> The Europeans - James, Henry - 1878  
>> Roderick Hudson - James, Henry - 1876  
>> Watch and ward - James, Henry - 1871  
>> Selected Klondike Short Stories - London, Jack - 1918  
>> The sea wolf - London, Jack - 1918  
>> Selected Stories - London, Jack - 1918  
>> White Fang - London, Jack - 1918  
>> A descent into the Maelstrom ; The gold-bug ; Mellonta tauta ... - Poe, Edgar Allan - 1895  
>> The call of the wild - London, Jack - 1903  
>> To build a fire - London, Jack - 1902  
>> Thoreau's Walden , or, Life in the woods - Thoreau, Henry David - 1854  
>> The song of the cardinal - Stratton Porter, Gene - 1904  
>> Freckles - Stratton Porter, Gene - 1903  
>> At the foot of the rainbow - Stratton Porter, Gene - 1907  
>> Ragged Dick - Alger, Horatio - 1868  
>> Up from slavery : an autobiography - Washington, Booker T - 1901  
>> Just David - Porter, Eleanor H - 1916  
>> The uncrowned king - Wright, Harold Bell - 1910  
>> Our Mr. Wrenn : the romantic adventures of a gentle man - Lewis, Sinclair - 1914  
>> King Solomon's mines - Haggard, H. Rider - 1885  
WORD 
COUNT:   2 003 881 
Edit Selection ^
 
 
Enter Query: 
   
QUERY:  presently
 
QUERY 
TYPE:  Exact match
 
Submit Query >>>
 
Figure  7: Query entry for the reference corpus 
 
6  Future developments 
At the time of writing the user management and public access routines for the virtual corpus 
system have not yet been implemented, and so cannot be described here. Free public access will be 
granted to all holdings in the archive which do not have intellectual property rights (IPR) restrictions. 
However, the load imposed on the server by indexing of corpora will mean that the corpus construction 
process will not be instantaneous. 
 
  929 
There are clear limitations imposed limited number of texts in the corpus, the bias in favour of 
classic literary texts and the limited functionality of the software. The OTA Virtual Corpus system is 
not intended to be a replacement for existing or future corpora. Nor is it intended that the online query 
tools will be elaborated and expanded to a great degree, but they should only facilitate simple 
investigations for teachers and learners, or serve as the initial basis for hypothesis forming and testing 
in research. 
Online access to corpora through the same interface will be offered. There will also be the possibility to 
use text level metadata of the contents of corpora in order to pick and choose which texts to use and 
include in a virtual corpus, as described in (Wittenberg and Broeder 2002). 
 
 
Figure 8: concordance of ‘presently’ in the reference corpus 
 
7  Bibliography 
Renouf, A. (2001). The web as a source of linguistic information. Corpus Linguistics 2001, Lancaster, 
UK, UCREL. 
   
Wittenberg, P. and D. Broeder (2002). Management of Language Resources using Metadata. Third 
International Conference on Language Resources and Evaluation (LREC), Las Palmas, Gran Canaria, 
Spain, ELRA. 
   
Wynne, M. (2002). The Language Resources Archive of the 21st Century. Language Resources and 
Evaluation Conference (LREC), Las Palmas, Gran Canaria, Spain. 
   
 

Survey and Prospect of China's Corpus-Based Researches 
Yang Xiaojun 
Hunan University of Science & Technology 
yangxiaojun2008@yahoo.com.cn 
Abstract 
This paper makes a general survey of the progress in China's Corpus-Based researches. These researches fall into the following categories: 
(1) Corpora compilation---10 corpora have been established or being established such as Monolingual Corpora for Chinese and English respectively, English/Chinese & Chinese/English Parallel Corpus, Japanese/Chinese & Chinese/Japanese Parallel Corpus, etc. 
(2) Three stages of English corpus studies: the first--1980's; the second---1990's and the third---2000's. 
(3) Application of corpus-based researches in language researches, translation studies & teaching, and lexicography. 
The author collects nearly all the papers concerned in the major academic journals, which are classified into several categories. Some suggestions are put forward to better China's corpus-based researches. 
Key Words: corpus-based researches; survey; prospect; corpora, compilation; application 
930

Analysing a Corpus-based Semantic Investigation of English Dialects 
Dr Debra Ziegeler & Ms Sarah Lee 
School of English & Linguistics 
University of Manchester 
debra.ziegeler@man.ac.uk 
sarah.lee@man.ac.uk 
Fax: +44 (0)161-275 3256 
The “Causatives in Singapore English Project” investigates cross-dialectal preferences for encoding caused agency (agents with [+ volition], [+ intention]) (Talmy 1979, 2000) causative events. In more studied dialects such as British English, a common strategy used to encode caused agency causatives are the get and have possessive periphrastic constructions, i.e., [NP1-GET/HAVE-NP2-STEM/INF/-ING]. We conducted a contrastive corpus-based analysis of various English dialects (British English, Singapore English, New Zealand English, and Indian English) in the first instance, to obtain a general quantitative picture of the frequency distribution of possessive periphrastics and of other strategies enlisted by the respective dialects. It was found that the periphrastics were present in all dialects but there were significant frequency and in some cases, distribution differences across dialects. We also found competing strategies, again in varying frequencies across the dialects, such as the resultative [NP1-GET/HAVE-NP2-PP], ‘conventionalised scenario’ constructions (Goldberg 1995), and other verbs such as ask and tell used in an implicative sense. We concluded that a corpus-based analysis as an initial tool of investigation was indeed valuable as a springboard for further empirical investigation. 
Nevertheless, we also found some evidence to suggest that the design of a corpus itself can skew the statistics obtained. Our topic was concerned with semantic (and grammatical function) information of a particular type, and we found that caused agency, a salient property, was not equally distributed amongst the components found in each corpus. This was because the components were stratified across varied genres. Some contained written as well as spoken data, and others just the former. Some genres represented formal styles, others informal. Thus statistical testing strategies we used had to include deriving average comparative probabilities of particular components across dialects and determining the probability of occurrence of a particular causative strategy in a given semantic environment. 
931