<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 12 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 2.0cm 2.0cm 2.0cm;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=IT link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span lang=EN-US>Apologies for multiple posting<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>*******************************************************<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>SEVENTH RECOGNIZING TEXTUAL ENTAILMENT CHALLENGE at TAC 2011<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US> (http://www.nist.gov/tac/2011/RTE/)<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>The Recognizing Textual Entailment (RTE) task consists of developing a system that, given two text fragments, can determine whether the meaning of one text is entailed, i.e. can be inferred, from the other text. The task has proved to work as a common framework in which to analyze, compare and evaluate different techniques used in NLP applications to deal with semantic inference.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>Since the introduction of the RTE task in 2005, RTE challenges have been organized annually. After three highly successful PASCAL RTE Challenges held in Europe, in 2008 RTE became a track at the Text Analysis Conference (TAC). During the last years RTE has constantly evolved, in the attempt to apply Textual Entailment to specific application settings and move it towards more realistic scenarios. After experimenting textual entailment recognition on a corpus in the RTE-5 Pilot Search task, in RTE-6 further innovations were introduced. First of all, the task of Recognizing Textual Entailment within a Corpus, a close variant of the RTE-5 Pilot task, replaced the traditional Main task. Furthermore, a Novelty Detection subtask aimed at specifically addressing the needs of the Summarization Update scenario was also proposed, where a system has to detect whether a statement is novel with respect to the content of a prior set of documents. Finally a new Knowledge Base Population (KBP) Validation Pilot, based on the TAC KBP Slot Filling task, was set up in order to investigate the potential utility of RTE systems for Knowledge Base Population.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>Encouraged by the positive response obtained so far, the RTE Organizing Committee is glad to launch the Seventh Recognizing Textual Entailment Challenge at TAC 2011.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>Organizations interested in participating in the RTE-7 Challenge are invited to submit a track registration form by June 3, 2011, at the TAC 2011 web site (http://www.nist.gov/tac/2011/).<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>THE RTE-7 TASKS<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>In order to ensure the continuity with the previous campaign and allow participants to get acquainted with the novelties introduced for the first time in RTE-6, the same tasks are proposed also in RTE-7 without significant changes, namely: <o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>MAIN TASK - RECOGNIZING TEXTUAL ENTAILMENT WITHIN A CORPUS <o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>In the RTE-7 Main task given a corpus, a hypothesis H, and a set of "candidate" entailing sentences for that H retrieved by Lucene from the corpus, RTE systems are required to identify all the sentences that entail H among the candidate sentences. <o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>The RTE-7 Main data set is based on the data created for the TAC 2008 and 2009 Update Summarization task, consisting of a number of topics, each containing two sets of documents, namely i) Cluster A, made up of the first 10 texts in chronological order of publication date, and ii) Cluster B, made up of the last 10 texts. H’s are standalone sentences taken from the TAC Update Summarization corpus, meanwhile candidate entailing sentences (T's) are the 100 top-ranked sentences retrieved for each H by Lucene from the Cluster A corpus, using H verbatim as the search query. While only the subset of the candidate entailing sentences must be judged for entailment, these sentences are not to be considered as isolated texts, but the entire Cluster A corpus, to which the candidate entailing sentences belong, is to be taken into consideration in order to resolve discourse references and appropriately judge the entailment relation.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>The example below presents a hypothesis referring to a given topic and some of the entailing sentences found in the subset of candidate sentences:<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US><H_sentence>Lance Armstrong is a Tour de France winner.</H_sentence> <o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US> <text doc_id="AFP_ENG_20050824.0557" s_id="1" evaluation="YES">Claims by a French newspaper that seven-time Tour de France winner Lance Armstrong had taken EPO were attacked as unsound and unethical by the director of the Canadian laboratory whose tests saw Olympic drug cheat Ben Johnson hit with a lifetime ban.</text> <o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US> <text doc_id="AFP_ENG_20050824.0557" s_id="2" evaluation="YES">L'Equipe on Tuesday carried a front page story headlined "Armstrong's Lie" suggesting the Texan had used the illegal blood booster EPO (erythropoeitin) during his first Tour win in 1999.</text> <o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>The second sentence in the example entails H because "the Texan" and "Tour" can be resolved as "Lance Armstrong" and "Tour the France" respectively, on the basis of the context in which they occur in the Cluster A document.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>NOVELTY DETECTION SUBTASK<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>The Novelty Detection subtask is based on the Main task and is aimed at specifically addressing the interests of the Summarization community, in particular with regard to the Update Summarization task, focusing on detection of novelty in Cluster B documents. <o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>The task consists of judging if the information contained in each H (drawn from the Cluster B documents) is novel with respect to the information contained in the set of Cluster A candidate entailing sentences. If for a given H one or more entailing sentences are found, it means that the content of the H is not new. On the contrary, if no entailing sentences are detected, it means that the information contained in the H is regarded as novel.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>The Novelty Detection task requires the same output format as the Main task - i.e. no additional type of decision is needed. Nevertheless, the Novelty Detection task differs from the Main task in the following ways: <o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>1) The set of H’s is not the same as that of the Main task;<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>2) The system outputs are scored differently, using specific scoring metrics designed for assessing novelty detection.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>The Main and Novelty Detection task guidelines and Development set are available at the RTE-7 Website (http://www.nist.gov/tac/2011/RTE/). <o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>KBP VALIDATION TASK<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>Based on the TAC Knowledge Base Population (KBP) Slot-Filling task, the KBP validation task is to determine whether a given relation (Hypothesis) is supported in an associated document (Text). Each slot fill that is proposed by a system for the KBP Slot-Filling task would create one evaluation item for the RTE-KBP Validation task: the Hypothesis would be a simple sentence created from the slot fill, while the Text would be the source document that was cited as supporting the slot fill.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>The KBP Validation task guidelines are available at the RTE-7 website (http://www.nist.gov/tac/2011/RTE/), together with the instructions to obtain the Development data (http://www.nist.gov/tac/2011/RTE/registration.html). <o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>RESOURCE AND TOOL EVALUATION THROUGH ABLATION TESTS <o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>The exploratory effort on resource evaluation, started in RTE-5 and extended to tools in RTE-6 , will be continued also in RTE-7. Ablation tests are required for systems participating in the RTE-7 Main task, in order to collect data to better understand the impact of both knowledge resources and tools used by RTE systems and evaluate their contribution to systems' performance. <o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>In an ablation test, a single resource or tool is removed from or added to a system, which is then rerun. Comparing the results to those obtained by the original system, it is possible to assess the practical contribution given by the individual resource or tool. <o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>More details on ablation tests are given in the Main and Novelty Detection Task guidelines (http://www.nist.gov/tac/2011/RTE/RTE7_Main_NoveltyDetection_Task_Guidelines.pdf). <o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>THE RTE RESOURCE POOL AND THE RTE KNOWLEDGE RESOURCES PAGE AT ACLwiki<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>(http://www.aclweb.org/aclwiki/index.php?title=Textual_Entailment_Resource_Pool) <o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US> <o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>The RTE Resource Pool, set up for the first time during RTE-3, serves as a portal and forum for publicizing and tracking resources, and reporting on their use. All the RTE participants and other members of the NLP community who develop or use relevant resources are encouraged to contribute to this important resource.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>The RTE Knowledge Resource page (http://www.aclweb.org/aclwiki/index.php?title=Textual_Entailment_Resource_Pool#Knowledge_Resources ) contains a list of the "standard" RTE resources, which have been selected and exploited majorly in the design of RTE systems during the RTE challenges held so far, together with the links to the locations where they are made available. Furthermore, the results of the ablation tests carried out in RTE-5 and in RTE-6, and their description, is also provided.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>PROPOSED RTE-7 SCHEDULE <o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>April 29 KBP Validation task: Release of Development Set<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>April 29 Main task: Release of Development Set<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>June 3 Deadline for TAC 2011 track registration<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>August 17 KBP Validation task: Release of Test Set<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>August 29 Main task: Release of Test Set<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>September 8 Main task: Deadline for task submissions<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>September 15 Main task: Release of individual evaluated results<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>September 16 KBP Validation task: Deadline for task submissions<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>September 23 KBP Validation task: Release of individual evaluated results<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>September 25 Deadline for TAC 2011 workshop presentation proposals<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>September 29 Main task: Deadline for ablation tests submissions<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>October 6 Main task: Release of individual ablation test results<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>October 25 Deadline for systems' reports<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>November 14-15: TAC 2011 workshop in Gaithersburg, Maryland, USA<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US> <o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>TRACK COORDINATORS AND ORGANIZERS:<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>Luisa Bentivogli, CELCT and FBK, Italy<o:p></o:p></span></p><p class=MsoNormal>Peter Clark, Vulcan Inc., USA<o:p></o:p></p><p class=MsoNormal>Ido Dagan, Bar Ilan University, Israel<o:p></o:p></p><p class=MsoNormal>Hoa Trang Dang, NIST, USA<o:p></o:p></p><p class=MsoNormal>Danilo Giampiccolo, CELCT, Italy<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><span style='font-size:10.5pt;font-family:Consolas'><o:p> </o:p></span></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p></div></body></html>