21.2188, FYI: Natural Language Generation Challenge (GIVE 2.5)

Tue May 11 18:45:01 UTC 2010

LINGUIST List: Vol-21-2188. Tue May 11 2010. ISSN: 1068 - 4875.

Subject: 21.2188, FYI: Natural Language Generation Challenge (GIVE 2.5)

Moderators: Anthony Aristar, Eastern Michigan U <aristar at linguistlist.org>
            Helen Aristar-Dry, Eastern Michigan U <hdry at linguistlist.org>

Reviews: Monica Macaulay, U of Wisconsin-Madison  
Eric Raimy, U of Wisconsin-Madison  
Joseph Salmons, U of Wisconsin-Madison  
Anja Wanner, U of Wisconsin-Madison  
       <reviews at linguistlist.org> 

Homepage: http://linguistlist.org/

The LINGUIST List is funded by Eastern Michigan University, 
and donations from subscribers and publishers.

Editor for this issue: Rachelle Felzien <rachelle at linguistlist.org>
================================================================  

To post to LINGUIST, use our convenient web form at
http://linguistlist.org/LL/posttolinguist.cfm.

===========================Directory==============================  

1)
Date: 07-May-2010
From: Alexander Koller < koller at mmci.uni-saarland.de >
Subject: Natural Language Generation Challenge (GIVE 2.5)

-------------------------Message 1 ---------------------------------- 
Date: Tue, 11 May 2010 14:43:58
From: Alexander Koller [koller at mmci.uni-saarland.de]
Subject: Natural Language Generation Challenge (GIVE 2.5)

E-mail this message to a friend:
http://linguistlist.org/issues/emailmessage/verification.cfm?iss=21-2188.html&submissionid=2632994&topicid=6&msgnumber=1

Second Natural Language Generation (NLG) Challenge on Generating
Instructions in Virtual Environments (GIVE-2.5).

Early Announcement:

http://www.give-challenge.org/research/

In the past two years, we have been organizing the Challenges on
Generating Instructions in Virtual Environments (GIVE).  In 2008-09,
we evaluated five natural language generation systems; the Second GIVE
Challenge (GIVE-2), in which we are evaluating seven systems, is
currently underway.

We are now announcing that we will organize the Second Second GIVE
Challenge (GIVE-2.5) in the winter of 2010-11.  The task in GIVE-2.5
will be basically identical to the one we now have in GIVE-2; this is
so GIVE-2 systems can be improved based on experiences from the
evaluation, and to allow more people to participate in the same task.

We invite you to consider participating in GIVE-2.5.  For more
information and to try out the GIVE-2 software, see
http://www.give-challenge.org/research.

If you are potentially interested in participating, please email us at
koller at mmci.uni-saarland.de so we know to keep you updated.

Overview
--------

The Challenge on Generating Instructions in Virtual Environments
(GIVE) is a novel approach to the notoriously hard problem of
evaluating NLG systems. In this scenario, a human user performs a
"treasure hunt" task in a virtual 3D environment. The NLG system's job
is to generate, in real time, a sequence of natural-language
instructions that will help the user perform this task. The crucial
thing is that users connect to the generation systems over the
Internet. By logging how well they were able to follow the system's
instructions, we can evaluate the quality of these instructions in
terms of task completion rates and times, subjective measures such as
helpfulness and friendliness, and runtime performance. Because the
user and the system don't need to be physically in the same place,
access to experimental subjects over the Internet becomes easy.

GIVE is a theory-neutral, end-to-end evaluation effort for NLG
systems. It involves research opportunities in text planning, sentence
planning, realization, and situated communication. One particularly
interesting aspect of situating the generation problem in a virtual
environment is that spatial and relational expressions play a bigger
role than in other NLG tasks. Beyond NLG, GIVE can be interesting as a
testbed for improving the NLG components of dialogue systems, and for
computational semanticists working on spatial language.

The GIVE-2 Task
---------------

In the GIVE-1 Challenge, which we ran last year, five NLG systems were
evaluated using data from almost 1200 game runs. To our knowledge,
this made GIVE-1 the largest ever NLG evaluation effort in terms of
the number of experimental subjects. We presented the results of the
evaluation at the ENLG Workshop, and have verified that these results
are consistent with (but more detailed than) the results that could be
obtained from a traditional lab-based evaluation.

In GIVE-2 we are evaluating seven systems; the public evaluation is
currently underway (see www.give-challenge.org).  The main novelty in
GIVE-2 is that where GIVE-1 used discrete worlds (which were based on
square tiles, and the user could only jump from the center of one tile
to the center of the next, and turn in 90 degree steps), GIVE-2
permits free, continuous movements in the worlds. This makes the
generation task more challenging because simple instructions of the
form "walk three steps forward" are no longer possible.  The results
of GIVE-2 will be presented at the INLG conference this year.

Anyone is invited to submit an NLG system to participate in the
GIVE-2.5 Challenge. We particularly invite contributions from students
and student teams. To get an idea of what this involves, you may want
to go to the GIVE website mentioned above and take a look at our EACL
2009 demo paper describing the software architecture, or download the
GIVE-2 software and look at it in more detail.

Provisional Timeline
--------------------

We plan to use essentially the same software for GIVE-2.5 that we used
in GIVE-2.  This means that GIVE-2 systems should be adaptable to
GIVE-2.5 with minimal effort; that is, you can start implementing your
GIVE-2.5 system based on the GIVE-2 software right now.   While we
don't have a precise schedule yet, we hope to present the results of
GIVE-2.5 at ENLG 2011.  This will probably entail that the public
evaluation phase will be at some point in the winter 2010-11.
We will distribute a call for participation with more details and
provide the GIVE-2.5 software in due time.

Organizing committee
--------------------

Donna Byron, Northeastern University
Justine Cassell, Northwestern University
Robert Dale, Macquarie University
Alexander Koller, Saarland University
Johanna Moore, University of Edinburgh
Jon Oberlander, University of Edinburgh
Kristina Striegnitz, Union College 

Linguistic Field(s): Computational Linguistics

-----------------------------------------------------------
LINGUIST List: Vol-21-2188	