<div dir="ltr"><span id="docs-internal-guid-67c2fbdb-4bc5-6f7c-c877-f3f6bb049767"><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align:center"><span style="font-size:16px;font-family:Arial;font-weight:bold;vertical-align:baseline;white-space:pre-wrap">Call For Participation</span></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align:center"><span style="font-size:16px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap"><br class=""></span><span style="font-size:16px;font-family:Arial;font-weight:bold;vertical-align:baseline;white-space:pre-wrap">Cross-Level Semantic Similarity</span></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align:center"><span style="font-size:16px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap">SemEval 2014 - Task 3</span></p><br><span style="font-size:16px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap"></span><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align:center">
<a href="http://alt.qcri.org/semeval2014/task3/" style="text-decoration:none"><span style="font-size:16px;font-family:Arial;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap">http://alt.qcri.org/semeval2014/task3/</span></a><span style="font-size:16px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap"></span></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align:center"><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap"><font size="3"><br class=""></font><br class=""></span><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap"></span></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap">The aim of this task is to evaluate semantic similarity when comparing lexical items of different types, such as paragraphs, sentences, phrases, words, and senses.</span></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap"><br class=""></span><span style="font-family:Arial;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap">INTRODUCTION</span><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap"><br class="">
</span><span style="font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Semantic similarity is an essential component of many applications in Natural Language Processing (NLP). This task provides an evaluation for semantic similarity across different types of text, which we refer to as lexical levels. Unlike prior SemEval tasks on textual similarity that have focused on comparing similar-sized texts, this task evaluates the case where larger text must be compared to smaller text, or even to senses. Specifically, this task encompasses four types of semantic similarity comparisons:</span></p>
<ol style="margin-top:0pt;margin-bottom:0pt"><li dir="ltr" style="list-style-type:decimal;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline"><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt">
<span style="background-color:transparent;vertical-align:baseline;white-space:pre-wrap">paragraph to sentence,</span></p></li><li dir="ltr" style="list-style-type:decimal;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline">
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="background-color:transparent;vertical-align:baseline;white-space:pre-wrap">sentence to phrase,</span></p></li><li dir="ltr" style="list-style-type:decimal;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline">
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="background-color:transparent;vertical-align:baseline;white-space:pre-wrap">phrase to word, and</span></p></li><li dir="ltr" style="list-style-type:decimal;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline">
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="background-color:transparent;vertical-align:baseline;white-space:pre-wrap">word to sense.</span></p></li></ol><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt">
<span style="font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Task 3 unifies multiple objectives from different areas of NLP under a single task, e.g., Paraphrasing, Summarization, and Compositional Semantics. </span><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap">One of the major motivations of this task is to produce systems that handle all comparison types, thereby freeing downstream NLP applications from needing to consider the type of text being compared.</span><span style="font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"></span></p>
<br><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap"></span><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-family:Arial;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap">TASK DETAILS</span><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap">:</span></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Task participants will be provided with pairs of each comparison type and asked to rate how similar is the meaning of the smaller item to the overall meaning of the larger item. For example, given a sentence and a paragraph, a system would assess how similar is the meaning of the sentence to the meaning of the paragraph. Ideally, a high-similarity sentence would reflect overall meaning of the paragraph. </span></p>
<br><span style="font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"></span><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">For word-to-sense comparisons, two evaluation settings are used: (1) out-of-context and (2) in-context. In the out-of-context setting, a sense is paired with a word in isolation. In the in-context setting, a sense is compared with the meaning of a usage appearing in some context. Task 3 uses the WordNet 3.1 sense inventory.</span><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap"></span></p>
<br><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap"></span><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-family:Arial;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap">PARTICIPATING</span></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap">Teams are free to participate in one, some, or all comparison types. Given the unified setting of the task, we especially encourage systems that handle all comparison types. However, we also allow specialized systems that target only a single comparison type. </span></p>
<br><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap"></span><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap">Interested teams are encouraged to join the task’s </span><a href="https://groups.google.com/d/forum/semeval-2014-clss" style="text-decoration:none"><span style="font-family:Arial;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap">mailing list</span></a><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap"> for discussion and announcements.</span></p>
<br><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap"></span><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-family:Arial;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap">EVALUATION</span></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap">Systems will be evaluated against human similarity scores using both rank-based and score-based comparisons. See the task’s </span><a href="http://alt.qcri.org/semeval2014/task3/index.php?id=evaluation" style="text-decoration:none"><span style="font-family:Arial;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap">Evaluation</span></a><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap"> page for further details.</span></p>
<br><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap"></span><br><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap"></span><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt">
<span style="font-family:Arial;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap">DATASETS</span><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap">:</span></p><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt">
<span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap">The Task 3 trial data set is now available and contains tens of examples for each comparison type to use in building initial systems. The full training data will be released later in December. Please see the task’s </span><a href="http://alt.qcri.org/semeval2014/task3/index.php?id=data-and-tools" style="text-decoration:none"><span style="font-family:Arial;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap">Data</span></a><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap"> page for further details.</span></p>
<br><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap"></span><br><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap"></span><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt">
<span style="font-family:Arial;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap">IMPORTANT DATES</span></p><ul style="margin-top:0pt;margin-bottom:0pt"><li dir="ltr" style="list-style-type:none;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline">
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Trial data ready October 31, 2013</span></p></li><li dir="ltr" style="list-style-type:none;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline">
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Training data ready December 15, 2013</span></p></li><li dir="ltr" style="list-style-type:none;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline">
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Evaluation period March 15-30, 2014 </span></p></li><li dir="ltr" style="list-style-type:none;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline">
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Paper submission due April 30, 2014 [TBC]</span></p></li><li dir="ltr" style="list-style-type:none;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline">
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="background-color:transparent;vertical-align:baseline;white-space:pre-wrap">SemEval workshop August 23-24, 2014, co-located with COLING and *SEM in Dublin, Ireland.</span></p>
</li></ul><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap"><br class=""></span></p><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt">
<span style="font-family:Arial;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap">MORE INFORMATION</span></p><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap">The Semeval-2014 Task 3 website includes details on the training data, evaluation, and examples of the comparison types:</span></p>
<br><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap"></span><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap"> </span><a href="http://alt.qcri.org/semeval2014/task3/" style="text-decoration:none"><span style="font-family:Arial;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap">http://alt.qcri.org/semeval2014/task3/</span></a><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap"><br class="">
</span></p><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap">If interested in the task, please join our mailing list for updates:</span></p>
<br><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap"></span><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap"> </span><a href="https://groups.google.com/d/forum/semeval-2014-clss" style="text-decoration:none"><span style="font-family:Arial;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap">https://groups.google.com/d/forum/semeval-2014-clss</span></a><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap"></span></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap"><br class=""></span></p><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt">
<span style="font-family:Arial;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap">ORGANIZERS</span></p><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap"><br class="">
David Jurgens (</span><span style="font-family:Arial;color:rgb(17,85,204);vertical-align:baseline;white-space:pre-wrap"><a href="mailto:jurgens@di.uniroma1.it">jurgens@di.uniroma1.it</a></span><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap">), Sapienza University of Rome, Italy</span></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap">Mohammad Taher Pilehvar (</span><span style="font-family:Arial;color:rgb(17,85,204);vertical-align:baseline;white-space:pre-wrap"><a href="mailto:pilehvar@di.uniroma1.it">pilehvar@di.uniroma1.it</a></span><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap">), Sapienza University of Rome, Italy</span></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap">Roberto Navigli (</span><span style="font-family:Arial;color:rgb(17,85,204);vertical-align:baseline;white-space:pre-wrap"><a href="mailto:navigli@di.uniroma1.it">navigli@di.uniroma1.it</a></span><span style="font-family:Arial;vertical-align:baseline;white-space:pre-wrap">), Sapienza University of Rome, Italy</span></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-size:15px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap"><br class=""></span><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"></span></p>
</span></div>