[Corpora-List] Parser Evaluation Shared Task: Call For Participation

[Kenji Sagae]

Parser Evaluation SHARED TASK
(at the Coling 2008 Workshop on Cross-Framework and Cross-Domain
Parser Evaluation)

http://www-tsujii.is.s.u-tokyo.ac.jp/pe08-st/


CALL FOR PARTICIPATION

Registration deadline: April 25, 2008
Submission deadline: May 5, 2008

Shared Task Description

One of the goals of the Cross-Framework and Cross-Domain Parser
Evaluation Workshop at Coling 2008 is to establish an improved shared
knowledge among participants of the strengths and weaknesses of extant
annotation and evaluation schemes. In order to create a joint focus
and in-depth discussion, the workshop will feature a 'lightweight'
shared task.

For a selection of 34 sentences (of which ten are considered
obligatory, the rest optional) for which gold-standard annotations are
provided in several formats potentially useful for parser evaluation,
we invite contributors to scrutinize existing gold-standard
representations contrastively, identify perceived deficiencies, and
sketch what can be done to address these.

As an optional component, participants in the shared task are welcome
to include 'native', framework-specific output representations and
actual results for a parsing system of their choice (be it their own
or not) in the contrastive study. In either case, submissions to the
shared task should aim to reflect on the nature of different
representations, highlight which additional distinctions are made in
either scheme, and discuss why these are useful (for some task) or
unmotivated (in general).

The parser evaluation shared task is not a competition or a bake-off,
but a collaborative effort intended to provide a focus for discussion
on current issues on parser evaluation.

Shared task submissions will be in the form of papers following the
workshop format, but focusing on issues related to the shared-task
data sets.  Authors submitting to the workshop can decide whether they
want to participate in the shared task or submit a more general paper
on parser evaluation (or both). A call for papers for the workshop is
being sent separately. Depending on the volume and distribution of
accepted papers, we anticipate that the presentation and discussion of
shared task results may account for about half of the available time
at the workshop.


Data

There are three data sets. All shared task participants are required
to use the first data set, which will serve as a common basis for
discussion and comparison of different representations.

1. WSJ 10 sentences (required)

This set contains 10 sentences from the Wall Street Journal portion of
the Penn Treebank. The following representation formats are provided
(see the workshop web page for links to format descriptions):

- Penn Treebank (PTB): phrase structure trees.
- CoNLL-2008 shared task (CoNLL08): labeled syntactic dependencies
extracted from the PTB annotations, and predicate-argument
dependencies extracted from PropBank and NomBank.
- RASP Grammatical Relations (GR): the Grammatical Relation scheme
proposed by Briscoe, Carroll and colleagues for parser evaluation.
- PARC Dependency structures (PARC): dependency structures in the
scheme used by King et al. in the PARC 700 dependency bank.
- UTokyo HPSG Treebank Predicate-Argument structures (HPSG-PA):
predicate-argument dependencies extracted from the University of Tokyo
HPSG Treebank.
- CCGBank Predicate-Argument structures (CCG-PA): predicate-argument
dependencies extracted from the CCGBank.

2. WSJ 15 sentences (optional)

This set contains an additional 15 sentences from the Wall Street
Journal portion of the Penn Treebank.

Annotation is provided in the same formats as above, except for PARC.

3. GENIA 9 sentences (optional)

This set contains 9 sentences from the GENIA Treebank, which contains
syntactic structures for abstracts of biomedical papers. The sentences
in this data set correspond to one abstract from the GENIA Treebank.

Annotation is provided in the following formats: PTB, GR, PARC, and HPSG-PA.


Important Dates

- Release of data sets: March 22, 2008
- Deadline for shared task registration: April 25, 2008
- Shared task submission deadline: May 5, 2008


Registration

To register for the shared task, please send email to
pest AT delph-in DOT net
with the words "PEST registration" anywhere in your message.

If you have any questions or comments, please send email to the shared
task organizers at pest AT delph-in DOT net.


Shared task organizers

Yusuke Miyao (lead organizer), University of Tokyo (Japan)
Kenji Sagae (lead organizer), University of Tokyo (Japan)

Edward Briscoe, University of Cambridge (UK)
Aoife Cahill, University of Stuttgart (Germany)
Julia Hockenmaier, University of Illinois at Urbana-Champaign (USA)
Tracy Holloway King, PARC (USA)
Christopher Manning, Stanford University (USA)
Joakim Nivre, Växjö and Uppsala Universities (Sweden)
Stephan Oepen, University of Oslo (Norway) and CSLI Stanford (USA)


For more information

Contact: pest AT delph-in DOT net

Shared task website:
http://www-tsujii.is.s.u-tokyo.ac.jp/pe08-st/

Workshop website:
http://lingo.stanford.edu/events/08/pe/

_______________________________________________
Corpora mailing list
Corpora at uib.no
http://mailman.uib.no/listinfo/corpora