[Corpora-List] ResPubliQA 2009: Prelilminary Call for Participation

Fri Dec 19 14:47:22 UTC 2008

Apologies for cross-posting

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
ResPubliQA exercise at the
MULTILINGUAL QUESTION ANSWERING TRACK AT CLEF 2009
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

------------------------------------------------------------------
Preliminary Call for Participation
------------------------------------------------------------------

We are glad to announce that a NEW EXERCISE will be proposed this year within the Question Answering track at CLEF. For more information and instructions visit the new ResPubliQA website at:

http://celct.isti.cnr.it/ResPubliQA/index.php

We invite participation, both from academic institutions and industrial organizations, on this new task.
Guidelines describing the task will be distributed among the participants and will be downloadable from the ResPubliQA website. Participants will also be provided with training data in order to have the opportunity to test the systems with procedures to be used in the formal evaluation campaign. The results of the evaluation will be disseminated at the final workshop which will be organized in Corfu in conjunction with ECDL 2009.

ResPubliQA 2009: TASK OVERVIEW

TASK DESCRIPTION: Systems receive natural language questions as input, and must return one paragraph containing the answer from the document collection. No exact answer is required neither multiple responses.

DOCUMENT COLLECTION: The subset of JRC-Acquis documents that have parallel aligned translations into all the languages involved will be used, namely Bulgarian, Dutch, English, French, German, Italian, Portuguese, Romanian and Spanish. The sub-collection is available at the ResPubliQA website http://celct.isti.cnr.it/ResPubliQA/Downloads

QUESTIONS: a pool of 500 independent questions (factoid, definition, reason, purpose and procedure) is provided

o   NO LIST questions
o   NO topic related questions (questions linked to the same topic)
o   NO NIL questions

ANSWERS: one of the following two responses must be returned

a) one single paragraph containing the candidate answer. Multi-paragraph answers are not considered in this task

b) the string NOA to indicate that the system prefers not to answer the question.

Systems that give no answers (NOA) instead of wrong answers will be rewarded by the evaluation measure. Answer Validation techniques (including Machine Learning) are expected to be used for taking this final decision.

LANGUAGES INVOLVED: Basque (EU), Bulgarian (BG), Dutch (NL), English (EN), French (FR), German (DE), Italian (IT), Portuguese (PT), Romanian (RO) and Spanish (ES).

A monolingual English (EN) task will also be activated this year, as both the exercize and the collection are different from TREC.
Basque has been included exclusively as a source language, as there is no Basque collection available.

SCHEDULE

Corpora Release: December, 19 2008

Preliminary Track Guidelines: December, 19 2008

Training data: December 19, 2008

Registration Open: February 1, 2009

Final Track Guidelines: February 1, 2009

Test Sets Release: May 25, 2009

Submissions of Runs by Participants: June 5, 2009

Release of Individual Results: from July 15, 2009

Submission of Papers for Working Notes: August 14, 2009

CLEF Workshop (in Corfu, Greece): 30 September - 02 October 2009

The participants will have 5 DAYS to upload their submissions, starting from the moment when the questions are downloaded, and not later than June 5, 2009.

TRACK COORDINATORS AND ORGANIZERS

-          UNED (coordinator)

Spanish Distance Learning University, Spain

Anselmo Peñas

-          CELCT (coordinator)

Center for the Evaluation of Language and Communication Technology, Italy

Pamela Forner and Danilo Giampiccolo

-          ELDA/ELRA

Evaluations and Language Resources Distribution Agency, France

Nicolas Moreau

-          University of Limerick, Ireland

Richard Sutcliffe

-          BTB

Bulgarian Academy of Science, Bulgaria

Petya Osenova

-          UAIC and RACAI, Romania

Alexandru Ioan Cuza University and Romanian Academy Research Institute for Artificial Intelligence, Romania

Corina Forascu

-          UBC

University of Basque Country, Spain

Iñaki Alegria

================================
Pamela Forner
CELCT (web: www.celct.it<http://www.celct.it/>)
Center for the Evaluation of Language and Communication Technologies
Via alla Cascata 56/c
38100 Povo - TRENTO -Italy

email: forner at celct.it<mailto:forner at celct.it>
tel.:  +39 0461 314 804
fax:  +39 0461 314 846

Secretary Phone:  +39 0461 314 870

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listserv.linguistlist.org/pipermail/corpora/attachments/20081219/b880cb35/attachment.htm>
-------------- next part --------------
_______________________________________________
Corpora mailing list
Corpora at uib.no
http://mailman.uib.no/listinfo/corpora