20.2336, FYI: Call: GIVE-2 Challenge Submissions

Tue Jun 30 20:09:00 UTC 2009

LINGUIST List: Vol-20-2336. Tue Jun 30 2009. ISSN: 1068 - 4875.

Subject: 20.2336, FYI: Call: GIVE-2 Challenge Submissions

Moderators: Anthony Aristar, Eastern Michigan U <aristar at linguistlist.org>
            Helen Aristar-Dry, Eastern Michigan U <hdry at linguistlist.org>

Reviews: Randall Eggert, U of Utah  
       <reviews at linguistlist.org> 

Homepage: http://linguistlist.org/

The LINGUIST List is funded by Eastern Michigan University, 
and donations from subscribers and publishers.

Editor for this issue: Danielle St. Jean <danielle at linguistlist.org>
================================================================  

To post to LINGUIST, use our convenient web form at
http://linguistlist.org/LL/posttolinguist.html.

===========================Directory==============================  

1)
Date: 29-Jun-2009
From: Alexander Koller < koller at mmci.uni-saarland.de >
Subject: Call: GIVE-2 Challenge Submissions

-------------------------Message 1 ---------------------------------- 
Date: Tue, 30 Jun 2009 16:07:16
From: Alexander Koller [koller at mmci.uni-saarland.de]
Subject: Call: GIVE-2 Challenge Submissions

E-mail this message to a friend:
http://linguistlist.org/issues/emailmessage/verification.cfm?iss=20-2336.html&submissionid=222111&topicid=6&msgnumber=1

Second Natural Language Generation (NLG) Challenge on Generating
Instructions in Virtual Environments (GIVE-2):  First Call for NLG System
Submissions. Part of Generation Challenges 2010 and endorsed by SIGGEN,
SIGDIAL, and SIGSEM.

http://www.give-challenge.org/research/

We invite contributions to the second round of the Challenge on Generating
Instructions in Virtual Environments (GIVE-2). See below for details of the
challenge and a timeline, or check our website.

Overview:
The Challenge on Generating Instructions in Virtual Environments (GIVE) is
a novel approach to the notoriously hard problem of evaluating NLG systems.
In this scenario, a human user performs a ''treasure hunt'' task in a
virtual 3D environment. The NLG system's job is to generate, in real time,
a sequence of natural-language instructions that will help the user perform
this task. The crucial thing is that users connect to the generation
systems over the Internet. By logging how well they were able to follow the
system's instructions, we can evaluate the quality of these instructions in
terms of task completion rates and times, subjective measures such as
helpfulness and friendliness, and runtime performance. Because the user and
the system don't need to be physically in the same place, access to
experimental subjects over the Internet becomes easy.

GIVE is a theory-neutral, end-to-end evaluation effort for NLG systems. It
involves research opportunities in text planning, sentence planning,
realization, and situated communication. One particularly interesting
aspect of situating the generation problem in a virtual environment is that
spatial and relational expressions play a bigger role than in other NLG
tasks. Beyond NLG, GIVE can be interesting as a testbed for improving the
NLG components of dialogue systems, and for computational semanticists
working on spatial language.

GIVE-2:
In the GIVE-1 Challenge, which we ran last year, five NLG systems were
evaluated using data from almost 1200 game runs. To our knowledge, this
made GIVE-1 the largest ever NLG evaluation effort in terms of the number
of experimental subjects. We presented the results of the evaluation at the
European Workshop on Natural Language Generation (ENLG Workshop), and have
verified that these results are consistent with (but more detailed than)
the results that could be obtained from a traditional lab-based evaluation.

Now we invite you to develop a GIVE NLG system and participate in GIVE-2.
The main novelty in GIVE-2 is that where GIVE-1 used discrete worlds (which
were based on square tiles, and the user could only jump from the center of
one tile to the center of the next, and turn in 90 degree steps), GIVE-2
will permit free, continuous movements in the worlds. This will make the
generation task more challenging because simple instructions of the form
''walk three steps forward'' are no longer possible.

Anyone is invited to submit an NLG system to participate in the GIVE-2
Challenge. We particularly invite contributions from students and student
teams. All participating systems will be evaluated, and the results will be
presented at International Natural Language Generation 2010 (INLG). To get
an idea of what this involves, you may want to go to the GIVE website
mentioned above and take a look at our European Chapter of the Association
for Computational Linguistics 2009 (EACL) demo paper describing the
software architecture, or download the GIVE-1 software and look at it in
more detail.

Timeline:
We plan to release a first version of the GIVE-2 software package in
August. You will then have time until February to develop your systems.
>From February to April, we will run the evaluation; at this time, users
will connect to the NLG systems over the Internet, through a central
matchmaking service which we will provide. You will then have the
opportunity to write a research report about your system, which will be
presented together with the evaluation results at the Generation Challenges
session of the INLG conference in July 2010.

Important dates:
August 2009- Materials distributed to participants
February-April 2010- Evaluation Period
July 2010- Presentation of results at INLG

If you are interested in participating, please let us know as soon as you
can at koller at mmci.uni-saarland.de. We look forward to hearing from you!

GIVE-2.5:
We plan to repeat GIVE-2 with minor modifications in 2010-11. If you happen
not to have time or available students this summer, this means you get a
second chance to participate. In addition, GIVE-2 participants will have an
opportunity to improve their systems based on their experiences in the
GIVE-2 evaluation.

Organizing committee:
Donna Byron, Northeastern University
Justine Cassell, Northwestern University
Robert Dale, Macquarie University
Alexander Koller, Saarland University
Johanna Moore, University of Edinburgh
Jon Oberlander, University of Edinburgh
Kristina Striegnitz, Union College 

Linguistic Field(s): Computational Linguistics

-----------------------------------------------------------
LINGUIST List: Vol-20-2336	