[Corpora-List] QA4MRE 2011: Guidelines and Sample now available!

[Pamela Forner]

Apologies for multiple postings

================================================================================================

QA4MRE: Question Answering for Machine Reading Evaluation

a Lab at CLEF 2011, Amsterdam, Netherlands, September 19-22

http://celct.fbk.eu/QA4MRE/



UPDATE: We are pleased to announce that the Track Guidelines, a sample of tests with the respective reference collection is now available on the QA4MRE website http://celct.fbk.eu/QA4MRE/

================================================================================================

Call for Participation





Teams are solicited for QA4MRE at CLEF 2011[1], an evaluation campaign of Machine Reading systems through Question Answering and Reading Comprehension Tests. Systems should be able to extract knowledge from given texts and use this knowledge to answer questions. The task focuses on the reading of single documents, where correct answers require some inference and the consideration of previously acquired background knowledge. For more information visit http://celct.fbk.eu/QA4MRE/.
The results of the evaluation campaign will be disseminated at the final workshop which will be organized in conjunction with the CLEF 2011 conference, 19-22 September in Amsterdam, The Netherlands.



Test data

Participating systems will be given two or three topics as input.  Associated with each topic, the organization will provide a reference corpus consisting of about 30,000 un-annotated documents related to the topic. These documents should be used by the systems to acquire the reading capabilities and the background knowledge needed to answer a test on the topic.

The test for each topic will consist of one single document and a set of multiple choice questions having five options each. The participating systems will be required to answer the questions by choosing in each case one answer from the five alternatives.



Languages
Document collections and reading tests will be available in English, Spanish, German, Italian, and Romanian.

  *   The document collections created in the different languages will be available to all participants. Thus, the learning of knowledge could be in one language or several.
  *   The tests will be exactly the same in all languages, using parallel translations.



Evaluation

Evaluation will be performed automatically by comparing the answers given by systems to the ones given by humans. No manual assessment will be required.

Each test will receive an evaluation score between 0 and 1 using c at 1. This measure, already tried in previous CLEF QA Tracks, encourages systems to reduce the number of incorrect answers and maintain the number of correct ones by leaving some questions unanswered.



Tentative Schedule

-    Guidelines and Samples of tests: end of January;

-    Release of topics and reference corpus: end of March;

-    Test set release: beginning of June;

-    Run submissions: middle of June;

-    Results to the participants: 1st of July;

-    Submission of Notebook Papers: according to the CLEF 2011 schedule.



General Coordination

-   Anselmo Peñas, UNED NLP & IR Group, Spain

-   Eduard Hovy, Information Sciences Institute, USA



Technical Coordination

-   Pamela Forner, CELCT, Italy

-   Alvaro Rodrigo, UNED NLP & IR Group, Spain

-   Richard Sutcliffe, University of Limerick, Ireland



Technical Staff

-   Corina Forascu, Alexandru Ioan Cuza University, Romania

-   Caroline Sporleder, University of Saarland, Germany



Programme Committee

-   Ken Barker, University of Texas at Austin, USA

-   Johan Bos, Rijksuniversiteit Groningen, Netherlands

-   Peter Clark, Vulcan Inc., USA

-   Ido Dagan, Bar-Ilan University

-   Bernardo Magnini, Fondazione Bruno Kessler, Italy

-   Dan Moldovan, University of Texas at Dallas, USA

-   Emanuele Pianta, Fondazione Bruno Kessler, and CELCT, Italy

-   John Prager, IBM, USA

-   Hoa Trang Dang, NIST, USA
Dan Tufis, Research Institute for Artificial Intelligence, Romanian Academy


================================
Pamela Forner
CELCT (web: www.celct.it<http://www.celct.it/>)
Center for the Evaluation of Language and Communication Technologies
Via alla Cascata 56/c
38100 Povo - TRENTO -Italy

email: forner at celct.it<mailto:forner at celct.it>
tel.:  +39 0461 314 804
fax:  +39 0461 314 846

Secretary Phone:  +39 0461 314 870


________________________________

[1] http://clef2011.org/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listserv.linguistlist.org/pipermail/corpora/attachments/20110215/2bf8c174/attachment.htm>
-------------- next part --------------
_______________________________________________
Corpora mailing list
Corpora at uib.no
http://mailman.uib.no/listinfo/corpora