[Corpora-List] QA4MRE 2011: New Pilot Task

Tue Mar 29 14:26:36 UTC 2011

***Apologies for multiple postings. Please distribute to whom may be concerned.****

================================================

Call for Participation

================================================

QA4MRE

Question Answering for Machine Reading Evaluation

held in conjunction with CLEF 2011,

September 19-22, Amsterdam, The Netherlands

NEW!! One Pilot Task added

We are pleased to announce that beside the Main task, one pilot tasks is offered this year at QA4MRE 2011:

Processing Modality and Negation for Machine Reading
In this task we aim at evaluating whether systems understand extra-propositional aspects of meaning like modality and negation. Modality is a grammatical category that allows to express aspects related to the attitude of the speaker towards her statements. Modality understood in a broader sense is also related to the expression of certainty, factuality, and evidentiality. Negation is a grammatical category that allows to change the truth value of a proposition. Modality and negation interact to express extra-propositional aspects of meaning.
More info at: http://www.cnts.ua.ac.be/BiographTA/qa4mre.html

===============================================================================

Teams are solicited for QA4MRE at CLEF 2011[1], an evaluation campaign of Machine Reading systems through Question Answering and Reading Comprehension Tests. Systems should be able to extract knowledge from given texts and use this knowledge to answer questions. The task focuses on the reading of single documents, where correct answers require some inference and the consideration of previously acquired background knowledge. For more information visit http://celct.fbk.eu/QA4MRE/.
The results of the evaluation campaign will be disseminated at the final workshop which will be organized in conjunction with the CLEF 2011 conference, 19-22 September in Amsterdam, The Netherlands.

Test data

Participating systems will be given two or three topics as input.  Associated with each topic, the organization will provide a reference corpus consisting of about 30,000 un-annotated documents related to the topic. These documents should be used by the systems to acquire the reading capabilities and the background knowledge needed to answer a test on the topic.

The test for each topic will consist of one single document and a set of multiple choice questions having five options each. The participating systems will be required to answer the questions by choosing in each case one answer from the five alternatives.

Languages
Document collections and reading tests will be available in English, Spanish, German, Italian, and Romanian.

  *   The document collections created in the different languages will be available to all participants. Thus, the learning of knowledge could be in one language or several.
  *   The tests will be exactly the same in all languages, using parallel translations.

Evaluation

Evaluation will be performed automatically by comparing the answers given by systems to the ones given by humans. No manual assessment will be required.

Each test will receive an evaluation score between 0 and 1 using c at 1. This measure, already tried in previous CLEF QA Tracks, encourages systems to reduce the number of incorrect answers and maintain the number of correct ones by leaving some questions unanswered.

Tentative Schedule

-    Guidelines and Samples of tests: end of January;

-    Release of topics and reference corpus: end of March;

-    Test set release: beginning of June;

-    Run submissions: middle of June;

-    Results to the participants: 1st of July;

-    Submission of Notebook Papers: according to the CLEF 2011 schedule.

General Coordination

-   Anselmo Peñas, UNED NLP & IR Group, Spain

-   Eduard Hovy, Information Sciences Institute, USA

Technical Coordination

-   Pamela Forner, CELCT, Italy

-   Alvaro Rodrigo, UNED NLP & IR Group, Spain

-   Richard Sutcliffe, University of Limerick, Ireland

Technical Staff

-   Corina Forascu, Alexandru Ioan Cuza University, Romania

-   Caroline Sporleder, University of Saarland, Germany

Programme Committee

-   Ken Barker, University of Texas at Austin, USA

-   Johan Bos, Rijksuniversiteit Groningen, Netherlands

-   Peter Clark, Vulcan Inc., USA

-   Ido Dagan, Bar-Ilan University

-   Bernardo Magnini, Fondazione Bruno Kessler, Italy

-   Dan Moldovan, University of Texas at Dallas, USA

-   Emanuele Pianta, Fondazione Bruno Kessler, and CELCT, Italy

-   John Prager, IBM, USA

-   Hoa Trang Dang, NIST, USA

-    Dan Tufis, Research Institute for Artificial Intelligence, Romanian Academy

================================
Pamela Forner
CELCT (web: www.celct.it<http://www.celct.it/>)
Center for the Evaluation of Language and Communication Technologies
Via alla Cascata 56/c
38100 Povo - TRENTO -Italy

email: forner at celct.it<mailto:forner at celct.it>
tel.:  +39 0461 314 804
fax:  +39 0461 314 846

Secretary Phone:  +39 0461 314 870

________________________________

________________________________

________________________________

[1] http://clef2011.org/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listserv.linguistlist.org/pipermail/corpora/attachments/20110329/49f28e75/attachment.htm>
-------------- next part --------------
_______________________________________________
UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
Corpora mailing list
Corpora at uib.no
http://mailman.uib.no/listinfo/corpora