[Corpora-List] 1st CfP: Machine Reading of Biomedical Texts about Alzheimer's Disease

Fri Mar 9 15:35:10 UTC 2012

(Apologies for cross-postings)

First Call for Participation

==========================================================
MACHINE READING OF BIOMEDICAL TEXTS ABOUT ALZHEIMER'S DISEASE

A pilot task of the
Question Answering for Machine Reading Evaluation Lab at CLEF 2012

http://celct.fbk.eu/QA4MRE/index.php?page=Pages/biomedicalTask.html
==========================================================

------------
Motivation
------------

The task will focus on biomedical texts about Alzheimer's Disease (AD).  AD has been chosen as the focus of the task because there is a particular interest in more efficient processing of Alzheimer-related literature, as this condition constitutes a considerable health challenge for an aging population (Citron 2010).  The increasing importance of AD is reflected in the  recently approved US National Alzheimer's Project Act, which will result in considerable funding  being made available for research on this disease and  for  financing  better data infrastructure resources. The illness is being analyzed  from various perspectives in a growing number of scientific studies (Al-Mubaid and Singh 2005, Li et al. 2009, Barbosa-Silva et al. 2011).

-----------------
Task description
-----------------

'Machine reading of biomedical texts about Alzheimer's Disease' is a pilot task of the Question  Answering for Machine Reading Evaluation (QA4MRE) (http://celct.fbk.eu/QA4MRE) at CLEF 2012 (http://clef2012.org/).  The task follows the same set up and principles as the QA4MRE, with the difference that it focuses on the biomedical domain. More information about the QA4MRE can be found in the QA4MRE2012 Track Guidelines provided with the Pilot Task Sample at:  http://celct.fbk.eu/QA4MRE/scripts/downloadFile.php?file=/websites/ResPubliQA/resources/sample/pilot/Biomedical_about_Alzheimer_Sample.zip  This pilot task aims at exploring the ability of a machine reading system to answer questions about a scientific topic.

It is well known that scientific language poses additional challenges to natural language processing, since domain knowledge (from ontologies and databases) is essential to reach deep understanding. There exist NLP  tools to process scientific text at several levels of analysis, and several tasks have been organised to extract different types of information from scientific texts. With this task we aim at going a step further, by proposing a task where inference plays a main role.  As in the QA4MRE task, this task focuses on the reading of single documents and the identification of the answers to a set of questions about information that is stated or implied in the text. Questions are in the form of multiple choice, each having five options, and only one correct answer. The detection of correct answers is specifically designed to require various kinds of inference and the consideration of previously acquired background knowledge from reference document collections provided by the organization. Although the additional knowledge obtained through the background collection may be used to assist with answering the questions, the principal answer is to be found among the facts contained in the test documents given. Participants will be provided with a background collection and test documents about  AD. To solve the task, participants can make use of  existing resources, such as ontologies or databases, and tools, such as named entity taggers, event extractors, parsers, etc. In order to keep the task reasonably simple for systems, the organisation will provide the texts of the background collection and the test documents processed at several levels of linguistic analysis (lemmas, part-of-speech, named entities, chunking, dependency parsing). Publicly available state of the art tools will be used for this purpose.

The task design benefited from previous experiences gained in this domain such as the TREC Genomics track, BioCreative and the BioNLP shared task, as well as previous editions of QA4MRE.

------------------------
Background collection
------------------------

The background collection is a reference corpus consisting of un-annotated documents related to the topic. As in the main task, the background collection should be used by the systems to acquire the reading capabilities and the knowledge needed to answer questions about the test documents.   The collection will consist of full text articles and abstracts. Full articles from the PubMed Central Open Access subset related to Alzheimer will be provided (around 8.000) and abstracts from Medline (around 66.000). PMC is a free full-text archive of biomedical and life sciences journal literature at the U.S. National Institutes of Health's National Library of Medicine (NIH/NLM).  MEDLINE (Medical Literature Analysis and Retrieval System Online) is a bibliographic database of life sciences and biomedical information. It was compiled by the United States National Library of Medicine (NLM), and is freely available on the Internet.

This background collection, called the Alzheimer's Disease Literature Corpus (ADLC corpus) had been carefully selected to be as specific as possible for this topic and should constitute a comprehensive resource for this task in particular and for text mining efforts tailored to the Alzheimer's Diseases field in general.

------------------------
Test data
------------------------

The test set will be composed of 4 reading tests. Each reading test will consist of one single document, with 10 questions and a set of five choices per question. So, there will be in total 40 questions and 200 choices/options.

Participating systems will be required to answer these 40 questions by choosing in each case one answer from the five alternatives. There will always be one and only one correct option. Systems will also have the chance to leave some questions unanswered if they are not confident about the correctness of their response.

------------------------
Evaluation
------------------------

This pilot task will be evaluated following the same criteria as the QA4MRE task.  Evaluation will be performed automatically by comparing the answers given by systems to the ones given by humans. No manual assessment will be required. Each test will receive an evaluation score between 0 and 1 using c at 1 (Peñas and Rodrigo 2011). This measure, already tried in previous CLEF QA Tracks, encourages systems to reduce the amount of incorrect answers and maintain the number of correct ones by leaving some questions unanswered.

------------------------
Example test
------------------------

An example test has been provided so that participants can form an idea about the type of texts and questions that will be provided as evaluation set. A  preprocessed version of the test set is also provided, so that participants can get familiar with the format and annotations that will be provided. The example test is available from the download site of the main task (http://celct.fbk.eu/QA4MRE/index.php?page=Pages/downloads.php) .

------------------------
Registration
------------------------

Participants can register to this task via the registration website of CLEF 2012 (http://clef2012.org/index.php?page=Pages/registrationForm.php). Registration remains open until May 15.

------------------------
Important dates
------------------------

Registration closes: May, 15.
Release of the background collection: April, 6
Test set release: June, 5
Run submissions: June, 15
Individual Results to Participants: June, 25
Working Notes Papers: according to CLEF schedule
CLEF Workshop: September 17-20, Rome, Italy

------------------------
Organisers
------------------------

Roser Morante, Walter Daelemans - CLiPS, University of Antwerp, Belgium
Martin Krallinger and Alfonso Valencia - CNIO, Madrid, Spain

------------------------
Technical support
------------------------

Vincent Van Asch - CLiPS, University of Antwerp, Belgium
Florian Leitner - CNIO, Madrid, Spain
Cartic Ramakrishnan - Information Sciences Institute of the University of Southern California, USA
Gully A.P.C. Burns - Information Sciences Institute of the University of Southern California, USA

------------------------
Domain advisor
------------------------

Tim Clark, Massachusetts Alzheimer's Disease Research Center, USA

--------------------------------------
General coordinators of the QA4MRE Lab
--------------------------------------

Anselmo Peñas - IR&NLP Group, UNED, Madrid, Spain
Eduard Hovy - Information Sciences Institute of the University of Southern California, USA

---------------------------------------------
Technical Management and data collection infrastructure
----------------------------------------------

Pamela Forner - Giovanni Moretti, CELCT , Italy

---------
Contact
---------

Roser Morante -  roser.morante at ua.ac.be
Martin Krallinger - mkrallinger at cnio.es

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listserv.linguistlist.org/pipermail/corpora/attachments/20120309/faecdb6f/attachment.htm>
-------------- next part --------------
_______________________________________________
UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
Corpora mailing list
Corpora at uib.no
http://mailman.uib.no/listinfo/corpora