Coling Grammar Engineering & Evaluation Workshop

[John Carroll]

Call for Papers

Grammar Engineering and Evaluation
Sunday 1 September 2002

Workshop to be held in conjunction with Coling 2002, Taipei
24 August - 1 September 2002

www.coling2002.sinica.edu.tw/
www.csis.ul.ie/gee02

Overview

Grammars are central components of many types of NLP system. The
workshop will be concerned with methods for the effective
engineering and evaluation of grammars with particular emphasis
on their use in real-world applications.

Background

Recent years have seen the development of techniques and
resources to support robust, deep grammatical analysis of
language in real-world domains, for instance in flexible
human-computer dialog systems (e.g.  the Dutch OVIS prototype
train information system) and speech-to-speech translation
(e.g. the Verbmobil system). The demands of these types of tasks
have driven significant advances in areas such as parser
efficiency, hybrid statistical / symbolic approaches to
disambiguation, and the acquisition of large-scale lexicons. In
response to these successes deep language processing is starting
to be deployed in commercial applications such as automated email
response.

The effective development, maintenance and enhancement of
grammars is a central issue in such efforts, and the size and
complexity of realistic grammars forces these processes to be
tackled in ways that have much in common with software
engineering. Thus, two common metrics defined over grammars are
coverage and degree of overgeneration; these can be evaluated by
applying the grammar to manually-constructed test suites of
grammatical and ungrammatical inputs, ideally supported by
automated profiling and visualisation tools. Examples of test
suites include those that have been produced on the TSNLP, DiET
and Verbmobil projects, while the Saarbruecken [incr tsdb()]
system is one of the established profiling tools. Since grammars
are expensive to develop, another important concern is the
effective re-use of existing grammatical resources: some grammar
formalisms facilitate this by for example allowing grammar
writers to structure the grammar hierarchically or in terms of
individual classes with modularised behaviour. A further issue is
how to support a team of grammarians working on the same or
related grammars; a notable effort in this area is the Xerox-led
collaborative ParGram project developing parallel grammars for
several different languages.


Objectives

The objectives of the workshop will be to summarise what has been
achieved in the areas of grammar engineering and evaluation, to
establish the common themes between different approaches and to
discuss future trends, with particular emphasis on real-world
applications. The focus will be on grammars rather than parsing
algorithms or the accuracy of parsing systems, on approaches
which enable re-use of resources, and on methods which are
suitable for multilingual systems.

In particular, contributions are solicited in the following areas:

  * Methods of grammar development and discussions of their strengths
  and weaknesses;

  * Standards for encoding grammatical information in a theory-neutral
  fashion;

  * Comparisons of manual techniques with those involving learning from
  treebanks;

  * Techniques for establishing the effectiveness, coverage or quality
  of a grammar;

  * The determination of time or effort required to achieve a level of
  performance or to adapt an existing grammar to a new application
  domain;

  * The application of a grammatical formalism to widely different
  languages; and

  * Issues in porting grammars between languages.


Submissions

Abstracts for workshop contributions should not exceed two A4 pages
(excluding references). An additional title page should state: the
title; author(s); affiliation(s); and contact author's e-mail address,
as well as postal address, telephone and fax numbers.

Submission is to be sent by email, preferably in Postscript or PDF
format, to Richard Sutcliffe by Friday 26 April 2002. Abstracts
will be reviewed by at least 3 members of the program committee.

Formatting instructions for the final full version of papers will be
sent to authors after notification of acceptance.

Accepted papers will appear in the printed proceedings which will be
available to all those who register for the workshop.

The proceedings of all workshops will also be included in the Coling
CD ROM along with the tutorials and the proceedings of the main
conference.

Important Dates

Deadline for Submissions: Fri 26 April 2002
Notification of Acceptance: Fri 24 May 2002
Final Versions of Papers Due: Fri 28 June 2002
Workshop: Sun 1 September 2002


Workshop Chairs

John A. Carroll
Cognitive and Computing Sciences
University of Sussex
Falmer, Brighton BN1 9QH
UK

johnca at cogs.susx.ac.uk
www.cogs.susx.ac.uk/lab/nlp/carroll/

Nelleke H. J. Oostdijk
Department of Language and Speech
University of Nijmegen
P.O. Box 9103
6500 HD Nijmegen
The Netherlands

n.oostdijk at let.kun.nl
lands.let.kun.nl/TSpublic/tosca/

Richard F. E. Sutcliffe (Contact Person)
Department of Computer Science
and Information Systems
University of Limerick
Limerick, Ireland

Richard.Sutcliffe at ul.ie
www.csis.ul.ie/staff/richard.sutcliffe


Programme Committee

Rens Bod, University of Amsterdam
Ted Briscoe, University of Cambridge
John Carroll, University of Sussex
Anette Frank, DFKI Saarbruecken
Gregory Grefenstette, Clairvoyance, Pittsburgh
Claire Grover, University of Edinburgh
Sadao Kurohashi, The University of Tokyo
Stephan Oepen, CSLI Stanford
Nelleke Oostdijk, University of Nijmegen
Richard Sutcliffe, University of Limerick
Atro Voutilainen, Conexor oy