[Corpora-List] ResPubliQA 2009:Call for Participation and Final Track Guidelines

[Pamela Forner]

Apologies for multiple postings



~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
ResPubliQA exercise at the
MULTILINGUAL QUESTION ANSWERING TRACK AT CLEF 2009
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


NEW: The Final Guidelines of the ResPubliQA exercize are now available on the website!



------------------------------------------------------------------
Call for Participation
------------------------------------------------------------------



We are glad to announce that a NEW EXERCISE will be proposed this year within the Question Answering track at the Cross Language Evaluation Forum (CLEF).
For more information and updates visit the new ResPubliQA website at:

http://celct.isti.cnr.it/ResPubliQA/index.php

We invite participation, both from academic institutions and industrial organizations, on this new task.
Guidelines describing the task are distributed among the participants and are downloadable from the ResPubliQA website. http://celct.isti.cnr.it/ResPubliQA/Documents
Training data have also been made available to participants in order to test the systems with procedures to be used in the formal evaluation campaign. http://celct.isti.cnr.it/ResPubliQA/Downloads
The results of the evaluation will be disseminated at the final workshop which will be organized in Corfu in conjunction with ECDL 2009.




ResPubliQA 2009: TASK OVERVIEW



TASK DESCRIPTION: Systems receive natural language questions as input, and must return one paragraph containing the answer from the document collection. No exact answer is required neither multiple responses.



DOCUMENT COLLECTION: The subset of JRC-Acquis documents that have parallel aligned translations into all the languages involved will be used, namely Bulgarian, Dutch, English, French, German, Italian, Portuguese, Romanian and Spanish. The sub-collection is available at the ResPubliQA website http://celct.isti.cnr.it/ResPubliQA/Downloads



QUESTIONS: a pool of 500 independent questions (factoid, definition, reason, purpose and procedure) is provided

o   NO LIST questions
o   NO topic related questions (questions linked to the same topic)
o   NO NIL questions

ANSWERS: one of the following two responses must be returned

a) one single paragraph containing the candidate answer. Multi-paragraph answers are not considered in this task

b) the string NOA to indicate that the system prefers not to answer the question.

Systems that give no answers (NOA) instead of wrong answers will be rewarded by the evaluation measure. Answer Validation techniques (including Machine Learning) are expected to be used for taking this final decision.



LANGUAGES INVOLVED: Basque (EU), Bulgarian (BG), Dutch (NL), English (EN), French (FR), German (DE), Italian (IT), Portuguese (PT), Romanian (RO) and Spanish (ES).

A monolingual English (EN) task will also be activated this year, as both the exercize and the collection are different from TREC.
Basque has been included exclusively as a source language, as there is no Basque collection available.



SCHEDULE



Registration Open: February 4, 2009



Final Track Guidelines: February 2, 2009



Test Sets Release: May 25, 2009



Submissions of Runs by Participants: June 5, 2009



Release of Individual Results: from July 15, 2009



Submission of Papers for Working Notes: August 14, 2009



CLEF Workshop (in Corfu, Greece): 30 September - 02 October 2009



The participants will have 5 DAYS to upload their submissions, starting from the moment when the questions are downloaded, and not later than June 5, 2009.



TRACK COORDINATORS AND ORGANIZERS


-          UNED (coordinator)

Spanish Distance Learning University, Spain

Anselmo Peñas



-          CELCT (coordinator)

Center for the Evaluation of Language and Communication Technology, Italy

Pamela Forner and Danilo Giampiccolo



-          ELDA/ELRA

Evaluations and Language Resources Distribution Agency, France

Nicolas Moreau



-          University of Limerick, Ireland

Richard Sutcliffe



-          BTB

Bulgarian Academy of Science, Bulgaria

Petya Osenova


-          UAIC and RACAI, Romania

Alexandru Ioan Cuza University and Romanian Academy Research Institute for Artificial Intelligence, Romania

Corina Forascu


-          UBC

University of Basque Country, Spain

Iñaki Alegria



ADVISORY BOARD


-          Donna Harman (National Institute for Standards and Technology (NIST), USA)



-          Maarten de Rijke (University of Amsterdam, The Netherlands)



-          Dominique Laurent (Synapse Développement, France.)



================================
Pamela Forner
CELCT (web: www.celct.it<http://www.celct.it/>)
Center for the Evaluation of Language and Communication Technologies
Via alla Cascata 56/c
38100 Povo - TRENTO -Italy

email: forner at celct.it<mailto:forner at celct.it>
tel.:  +39 0461 314 804
fax:  +39 0461 314 846

Secretary Phone:  +39 0461 314 870

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listserv.linguistlist.org/pipermail/corpora/attachments/20090202/730972c6/attachment.htm>
-------------- next part --------------
_______________________________________________
Corpora mailing list
Corpora at uib.no
http://mailman.uib.no/listinfo/corpora