19.3929, Qs: ResPubliQA2009: Call for Participation

[LINGUIST Network]

LINGUIST List: Vol-19-3929. Sun Dec 21 2008. ISSN: 1068 - 4875.

Subject: 19.3929, Qs: ResPubliQA2009: Call for Participation

Moderators: Anthony Aristar, Eastern Michigan U <aristar at linguistlist.org>
            Helen Aristar-Dry, Eastern Michigan U <hdry at linguistlist.org>
 
Reviews: Randall Eggert, U of Utah  
         <reviews at linguistlist.org> 

Homepage: http://linguistlist.org/

The LINGUIST List is funded by Eastern Michigan University, 
and donations from subscribers and publishers.

Editor for this issue: Dan Parker <dan at linguistlist.org>
================================================================  

We'd like to remind readers that the responses to queries are usually
best posted to the individual asking the question. That individual is
then strongly encouraged to post a summary to the list. This policy was
instituted to help control the huge volume of mail on LINGUIST; so we
would appreciate your cooperating with it whenever it seems appropriate.

In addition to posting a summary, we'd like to remind people that it
is usually a good idea to personally thank those individuals who have
taken the trouble to respond to the query.

To post to LINGUIST, use our convenient web form at
http://linguistlist.org/LL/posttolinguist.html.

===========================Directory==============================  

1)
Date: 19-Dec-2008
From: Pamela Forner < forner at celct.it >
Subject: ResPubliQA2009: Call for Participation

 

	
-------------------------Message 1 ---------------------------------- 
Date: Sun, 21 Dec 2008 10:08:10
From: Pamela Forner [forner at celct.it]
Subject: ResPubliQA2009: Call for Participation

E-mail this message to a friend:
http://linguistlist.org/issues/emailmessage/verification.cfm?iss=19-3929.html&submissionid=200189&topicid=8&msgnumber=1
  


ResPubliQA exercise at the 

Multilingual Question Answering Track at CLEF 2009


We are glad to announce that a new exercise will be proposed this year
within the Question Answering track at CLEF. For more information and
instructions visit the new ResPubliQA website at:

http://celct.isti.cnr.it/ResPubliQA/index.php

We invite participation, both from academic institutions and industrial
organizations, on this new task. 

Guidelines describing the task will be distributed among the participants
and will be downloadable from the ResPubliQA website. Participants will
also be provided with training data in order to have the opportunity to
test the systems with procedures to be used in the formal evaluation
campaign. The results of the evaluation will be disseminated at the final
workshop which will be organized in Corfu in conjunction with ECDL 2009.

ResPubliQA 2009: Task Overview

Task Description: Systems receive natural language questions as input, and
must return one paragraph containing the answer from the document
collection. No exact answer is required neither multiple responses.

Document Collection: The subset of JRC-Acquis documents that have parallel
aligned translations into all the languages involved will be used, namely
Bulgarian, Dutch, English, French, German, Italian, Portuguese, Romanian
and Spanish. The sub-collection is available at the ResPubliQA website:

http://celct.isti.cnr.it/ResPubliQA/index.php?page=Pages/downloads.php&option=newTrackSetup

Questions: a pool of 500 independent questions (factoid, definition,
reason, purpose and procedure) is provided
-	No list questions
-	No topic related questions (questions linked to the same topic)
-	No NIL questions

Answers: one of the following two responses must be returned
a) one single paragraph containing the candidate answer. Multi-paragraph
answers are not considered in this task
b) the string NOA to indicate that the system prefers not to answer the
question. 
Systems that give no answers (NOA) instead of wrong answers will be
rewarded by the evaluation measure. Answer Validation techniques (including
Machine Learning) are expected to be used for taking this final decision.

Languages Involved: Basque (EU), Bulgarian (BG), Dutch (NL), English (EN),
French (FR), German (DE), Italian (IT), Portuguese (PT), Romanian (RO) and
Spanish (ES).

A monolingual English (EN) task will also be activated this year, as both
the exercise and the collection are different from TREC. 
Basque has been included exclusively as a source language, as there is no
Basque collection available.

Schedule:
 
Corpora Release: December, 19 2008

Preliminary Track Guidelines: December, 19 2008

Training data: December 19, 2008

Registration Open: February 1, 2009

Final Track Guidelines: February 1, 2009

Test Sets Release: May 25, 2009

Submissions of Runs by Participants: June 5, 2009

Release of Individual Results: from July 15, 2009

Submission of Papers for Working Notes: August 14, 2009

CLEF Workshop (in Corfu, Greece): 30 September - 02 October 2009

The participants will have 5 DAYS to upload their submissions, starting
from the moment when the questions are downloaded, and not later than June
5, 2009.

Track Coordinators and Organizers:

-	UNED (coordinator)
Spanish Distance Learning University, Spain
Anselmo Peñas
	
-	CELCT (coordinator)
Center for the Evaluation of Language and Communication Technology, Italy
Pamela Forner and Danilo Giampiccolo 

-	ELDA/ELRA
Evaluations and Language Resources Distribution Agency, France
Nicolas Moreau

-	University of Limerick, Ireland
Richard Sutcliffe

-	BTB
Bulgarian Academy of Science, Bulgaria
Petya Osenova

-	UAIC and RACAI, Romania
Alexandru Ioan Cuza University and Romanian Academy Research Institute for
Artificial Intelligence, Romania
Corina Forascu

-	UBC 
University of Basque Country, Spain
Iñaki Alegria 

Linguistic Field(s): General Linguistics






-----------------------------------------------------------
LINGUIST List: Vol-19-3929