20.3641, FYI: Call for Participlation: GREC 2010 Shared Tasks

Wed Oct 28 16:40:34 UTC 2009

LINGUIST List: Vol-20-3641. Wed Oct 28 2009. ISSN: 1068 - 4875.

Subject: 20.3641, FYI: Call for Participlation: GREC 2010 Shared Tasks

Moderators: Anthony Aristar, Eastern Michigan U <aristar at linguistlist.org>
            Helen Aristar-Dry, Eastern Michigan U <hdry at linguistlist.org>

Reviews: Monica Macaulay, U of Wisconsin-Madison  
Eric Raimy, U of Wisconsin-Madison  
Joseph Salmons, U of Wisconsin-Madison  
Anja Wanner, U of Wisconsin-Madison  
       <reviews at linguistlist.org> 

Homepage: http://linguistlist.org/

The LINGUIST List is funded by Eastern Michigan University, 
and donations from subscribers and publishers.

Editor for this issue: Danielle St. Jean <danielle at linguistlist.org>
================================================================  

To post to LINGUIST, use our convenient web form at
http://linguistlist.org/LL/posttolinguist.html.

===========================Directory==============================  

1)
Date: 26-Oct-2009
From: Eric Kow < E.Y.Kow at brighton.ac.uk >
Subject: Call for Participlation: GREC 2010 Shared Tasks

-------------------------Message 1 ---------------------------------- 
Date: Wed, 28 Oct 2009 12:37:56
From: Eric Kow [E.Y.Kow at brighton.ac.uk]
Subject: Call for Participlation: GREC 2010 Shared Tasks

E-mail this message to a friend:
http://linguistlist.org/issues/emailmessage/verification.cfm?iss=20-3641.html&submissionid=235237&topicid=6&msgnumber=1

First Call for Participation

Generation of References in Context (GREC) Tasks 2010

Part of Generation Challenges 2010, in conjunction with INLG 2010.

Generation Challenges is a shared-task evaluation initiative for research
that involves the generation of language, in the widest sense. Generation
Challenges 2010 is the fourth event in this initiative, and aims to
continue to provide a common forum for a number of different Shared Tasks (see 
http://www.nltg.brighton.ac.uk/research/genchal10/).

As part of Generation Challenges 2010, we are organising three GREC Shared
Task Competitions. The first is the GREC-NEG (Named Entity Generation) task
which uses the GREC-People Corpus of 1,000 Wikipedia introduction sections
about people in which single and plural references to all people mentioned
in the text have been annotated; the task in GREC-NEG is to select
appropriate referential expressions for all mentions (singular and plural)
of people from a given list.

The second task, GREC-NER, is a straightforward Named Entity Recognition
task, using the new GREC-People-II extension to the GREC-People Corpus. The
third task is GREC-FULL which requires participating systems to combine NER
and NEG in order to improve references to people in texts, and this also
uses the GREC-People-II Corpus.

Submissions to all three tasks will be evaluated using a range of intrinsic
and extrinsic measures, some assessed automatically, some manually.
Submitted systems and evaluation results will be presented in a special
session at INLG'10 in Dublin in July 2010, and published in the INLG'10
proceedings.

1. Background
There has been increasing interest recently among text summarisation
researchers in postprocessing techniques to improve the referential clarity
and coherence of extractive summaries, and among language generation
researchers in generating referential expressions in context. The GREC
tasks are aimed at researchers in both of these groups, and the objective
is the development of methods for generating chains of referential
expressions for discourse entities in the context of a written discourse,
as is useful e.g. for postprocessing extractive summaries and repeatedly
edited texts (such as Wikipedia articles).

2. Data
The GREC data resources consist of introduction sections collected from
Wikipedia articles in which several broad syntactic categories of reference
to named entities have been annotated. The annotations include features
encoding syntactic and semantic information.

The GREC-People and GREC-People-II corpora consist of 1,000 texts each
in one domain, Wikipedia articles about people. Here, all references,
singular and plural, to all people mentioned in a text have been annotated.
Each text in these corpora therefore includes explicit coreference
annotation for at least one coreference chain.

For GREC-People and GREC-People-II we have test sets of 100 texts each,
where referential expressions have been selected by participants in an
elicitation experiment. In these test sets, there are three versions for
each corpus text, in each of which the referential expressions have been
manually selected by a single participant in the experiment.

Full details of the evaluation methods for GREC'10 will be given in the
Participants' Pack that will be distributed to registered participants.

3. Participation
Registration is now open at the GREC'10 homepage 
(http://www.nltg.brighton.ac.uk/research/genchal10/grec). Once registered,
participants in the GREC-NEG Task will receive the complete training and
development set, evaluation software and detailed documentation
(collectively known as the Participants' Pack) for this task. Participants
in the other two tasks will first receive a sample of the training and
development data, to enable them to start building systems; they will
receive the complete Participants' Pack for GREC-NER/FULL by the end of 2009.

4. Proceedings and Presentations
The Generation Challenges 2010 meeting will be held as a special session at
INLG 2010 in July 2010 in Dublin. The session will include overviews of all
the shared tasks, including the GREC'10 Tasks. The participating systems
will additionally be presented as papers in the INLG'10 proceedings, and as
posters during the INLG'10 poster session.

GREC'10 papers will not undergo a selection procedure with multiple
reviews, but the organisers reserve the right to reject material which is
not appropriate given the participation guidelines.

5. Important Dates
Oct 2009 - First Call for Participation in GREC'10 Tasks;
- GREC-NEG Participants' Pack available
- GREC-NER/FULL Data Sample available

Dec 31, 2009 - GREC-NER/FULL Participants Pack available

Apr 5-May 2, 2010 - Test data submission period; 4-step submission process:
1. Fill in submission form (available on website from Apr 5).
2. Upload 2-page paper describing approach and reporting development set
results.
3. Download test data (inputs only).
4. Submit test data outputs at the latest 48 hours after download, but in
any case no later than May 2.

May 2, 2010 - Final deadline for submission of test data outputs.

May 2-23, 2010 - GREC'10 Evaluation period

Jul 7-9, 2010 - Generation Challenges meeting at INLG'10

6. Organisation
Anja Belz, NLTG, University of Brighton, UK
Eric Kow, NLTG, University of Brighton, UK
Jette Viethen, Macquarie University, Australia

GREC'10 homepage: http://www.nltg.brighton.ac.uk/research/genchal10/grec

Generation Challenges homepage:
http://www.nltg.brighton.ac.uk/research/genchal10

Generation Challenges email: nlg-stec at itri.brighton.ac.uk 

Linguistic Field(s): Computational Linguistics

-----------------------------------------------------------
LINGUIST List: Vol-20-3641	