21.911, FYI: SemEval-2010 Task 12: PETE

Wed Feb 24 03:56:05 UTC 2010

LINGUIST List: Vol-21-911. Tue Feb 23 2010. ISSN: 1068 - 4875.

Subject: 21.911, FYI: SemEval-2010 Task 12: PETE

Moderators: Anthony Aristar, Eastern Michigan U <aristar at linguistlist.org>
            Helen Aristar-Dry, Eastern Michigan U <hdry at linguistlist.org>

Reviews: Monica Macaulay, U of Wisconsin-Madison  
Eric Raimy, U of Wisconsin-Madison  
Joseph Salmons, U of Wisconsin-Madison  
Anja Wanner, U of Wisconsin-Madison  
       <reviews at linguistlist.org> 

Homepage: http://linguistlist.org/

The LINGUIST List is funded by Eastern Michigan University, 
and donations from subscribers and publishers.

Editor for this issue: Elyssa Winzeler <elyssa at linguistlist.org>
================================================================  

To post to LINGUIST, use our convenient web form at
http://linguistlist.org/LL/posttolinguist.html.

===========================Directory==============================  

1)
Date: 22-Feb-2010
From: Deniz Yuret < dyuret at ku.edu.tr >
Subject: SemEval-2010 Task 12: PETE

-------------------------Message 1 ---------------------------------- 
Date: Tue, 23 Feb 2010 22:52:34
From: Deniz Yuret [dyuret at ku.edu.tr]
Subject: SemEval-2010 Task 12: PETE

E-mail this message to a friend:
http://linguistlist.org/issues/emailmessage/verification.cfm?iss=21-911.html&submissionid=2611954&topicid=6&msgnumber=1

SemEval-2010 Shared Task 12
Parser Evaluation using Textual Entailments (PETE)
(http://pete.yuret.com)

The purpose of this e-mail is to encourage participation in the task
'Parser Evaluation using Textual Entailments' in the 5th International
Workshop on Semantic Evaluations, SemEval-2010
(http://semeval2.fbk.eu/semeval2.php) collocated with ACL-2010, July 15-16,
Uppsala.

This shared task should be of interest to researchers working on
* parsing
* semantic relation extraction
* recognizing textual entailments

Description:

Parser Evaluation using Textual Entailments is a shared task in the
SemEval-2010 Evaluation Exercises on Semantic Evaluation.  The task
involves recognizing textual entailments (RTE) based solely on syntactic
information:

* The man with the hat was tired.
* The man was tired. (yes)
* The hat was tired. (no)

Our goals in introducing this task are:

* To focus parser evaluation on semantically relevant phenomena.
* To introduce a parser evaluation scheme that is formalism independent.
* To introduce a targeted textual entailment task focused on a single
linguistic competence.
* To be able to collect high quality evaluation data from untrained annotators.

The following criteria were used when constructing the entailments:

* They should be decidable using only syntactic inference.
* They should be easy to decide by untrained annotators.
* They should be challenging for state of the art parsers.

You can find more details about the entailment generation process at the
task website (http://pete.yuret.com).  The trial dataset can be downloaded
from the task website or the SemEval website (http://semeval2.fbk.eu). 
There will be no training data.  The evaluation will be similar to the past
RTE tasks.  There is a Google group
(http://groups.google.com/group/semeval-pete) for task related messages.

Instructions:

* Join the mailing list (http://groups.google.com/group/semeval-pete)
* Register in SemEval website (http://semeval2.fbk.eu)
* Download the trial data from task website (http://pete.yuret.com) or
SemEval website (http://semeval2.fbk.eu)
* Download task guide from task website (http://pete.yuret.com/guide)
* Download test data from SemEval website (http://semeval2.fbk.eu)
* Upload results to SemEval website (http://semeval2.fbk.eu)

Important Dates:

* February 19 - the development data available.
* March 26 - the test data available.
* April 2 - end of submission period for the task.
* April 17 - Submission of description papers.
* May 6 - Notification of acceptance.
* July 15-16 - Workshop at ACL 2010, Uppsala.

Contact:
* Deniz Yuret  dyuret at ku.edu.tr 

Linguistic Field(s): Computational Linguistics

-----------------------------------------------------------
LINGUIST List: Vol-21-911	