<div dir="ltr"><div><b style="color:rgb(34,34,34);font-size:small;font-style:normal;font-variant:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);font-family:arial;font-weight:normal"><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align:center">
<span style="font-size:16px;font-family:Arial;font-weight:bold;vertical-align:baseline;white-space:pre-wrap">Call for Participation</span></p><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align:center">
<span style="font-size:16px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap"><br></span><span style="font-size:16px;font-family:Arial;font-weight:bold;vertical-align:baseline;white-space:pre-wrap">Cross-Level Semantic Similarity</span></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align:center"><span style="font-size:16px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap">SemEval 2014 - Task 3</span></p><br><span style="font-size:16px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap"></span><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align:center">
<a href="http://alt.qcri.org/semeval2014/task3/" target="_blank" style="color:rgb(17,85,204);text-decoration:none"><span style="font-size:16px;font-family:Arial;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap">http://alt.qcri.org/semeval2014/task3/</span></a><span style="font-size:16px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap"></span></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align:center"><span style="font-size:16px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap"><br></span></p><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt">
<span style="font-size:13px;font-family:Arial;color:rgb(0,0,0);vertical-align:baseline;white-space:pre-wrap">To what extent is the meaning of the sentence “do u know where i can watch free older movies online without download?” preserved in the phrase “streaming vintage movies for free”?  Or, how similar is “circumscribe” to the phrases “beating around the bush” and “avoiding the topic”? The aim of Task 3 is to evaluate semantic similarity when comparing items of different syntactic and semantic scales, such as paragraphs, sentences, phrases, words, and senses.</span></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><br><span style="font-size:13px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap"></span></p><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt">
<br><span style="font-size:13px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap"></span></p><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-size:13px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap"></span><b><span style="font-size:13px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap">INTRODUCTION</span></b><span style="font-size:13px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap"></span></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><br><span style="font-size:13px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap"></span></p><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt">
<span style="font-size:13px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap"></span><span style="font-size:13px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Semantic similarity is an essential component of many applications in Natural Language Processing (NLP). Task 3 provides an evaluation for semantic similarity across different types of text, which we refer to as lexical levels. Unlike prior SemEval tasks on textual similarity that have focused on comparing similar-sized texts, this task evaluates the case where a smaller text or sense is compared to the meaning of a larger text. Specifically, this task encompasses four types of semantic similarity comparisons:</span></p>
<br><span style="font-size:13px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"></span><ol style="margin-top:0pt;margin-bottom:0pt"><li dir="ltr" style="margin-left:15px;list-style-type:decimal;font-size:13px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline">
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="background-color:transparent;vertical-align:baseline;white-space:pre-wrap">paragraph to sentence,</span></p></li><li dir="ltr" style="margin-left:15px;list-style-type:decimal;font-size:13px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline">
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="background-color:transparent;vertical-align:baseline;white-space:pre-wrap">sentence to phrase,</span></p></li><li dir="ltr" style="margin-left:15px;list-style-type:decimal;font-size:13px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline">
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="background-color:transparent;vertical-align:baseline;white-space:pre-wrap">phrase to word, and</span></p></li><li dir="ltr" style="margin-left:15px;list-style-type:decimal;font-size:13px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline">
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="background-color:transparent;vertical-align:baseline;white-space:pre-wrap">word to sense.</span></p></li></ol><br><span style="font-size:13px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"></span><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt">
<span style="font-size:13px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Task 3 unifies multiple objectives from different areas of NLP under a single task, e.g., Paraphrasing, Summarization, and Compositional Semantics. </span><span style="font-size:13px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap">One of the major motivations of this task is to produce systems that handle all comparison types, thereby freeing downstream NLP applications from needing to consider the type of text being compared.</span><span style="font-size:13px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"></span></p>
<br><span style="font-size:13px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap"></span><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><b><span style="font-size:13px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap"><br>
</span></b></p><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><b><span style="font-size:13px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap">TASK DETAILS</span></b></p><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt">
<br><span style="font-size:13px;font-family:Arial;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap"></span><span style="font-size:13px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap"></span></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-size:13px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap"></span></p><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt">
<span style="font-size:13px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Task participants are provided with pairs of each comparison type and asked to rate to what extent does the smaller item preserve the meaning of the larger item.  For example, given a sentence and a paragraph, a system would assess how similar is the meaning of the sentence to the meaning of the paragraph. Ideally, a high-similarity sentence would summarize overall meaning of the paragraph.<b> </b>Similarly, in the case of word to sense, we assess how well the semantics of a word is captured by a WordNet 3.0 sense.</span><span style="font-size:13px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap"></span></p>
<b><br></b><b><span style="font-size:13px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap"></span></b><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><b><span style="font-size:13px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap"><br>
</span></b></p><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><b><span style="font-size:13px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap">PARTICIPATING</span></b></p><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt">
<b><span style="font-size:13px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap"><br></span></b></p><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-size:13px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap">Teams are free to participate in one, some, or all comparison types.  Given the unified setting of the task, we especially encourage systems that handle more than one comparison type.<b> </b>However, we also allow specialized systems that target only a single comparison type.  </span></p>
<br><span style="font-size:13px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap"></span><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-size:13px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap">Interested teams are encouraged to join the task’s </span><a href="https://groups.google.com/d/forum/semeval-2014-clss" target="_blank" style="color:rgb(17,85,204);text-decoration:none"><span style="font-size:13px;font-family:Arial;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap">mailing list</span></a><span style="font-size:13px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap"> for discussion and announcements.</span></p>
<br><span style="font-size:13px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap"></span><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><b><span style="font-size:13px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap">EVALUATION</span></b></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-size:13px;font-family:Arial;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap"><br></span></p><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt">
<span style="font-size:13px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap">Systems will be evaluated against human similarity scores using both rank-based and score-based comparisons.<b> </b>See the task’s </span><a href="http://alt.qcri.org/semeval2014/task3/index.php?id=evaluation" target="_blank" style="color:rgb(17,85,204);text-decoration:none"><span style="font-size:13px;font-family:Arial;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap">Evaluation</span></a><span style="font-size:13px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap"> page for further details.</span></p>
<br><br></b></div><div><b style="color:rgb(34,34,34);font-size:small;font-style:normal;font-variant:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);font-family:arial;font-weight:normal"><b>DATASETS</b></b></div>
<span style="color:rgb(34,34,34);font-size:small;font-style:normal;font-variant:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);font-family:arial;font-weight:normal"><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt">
<span style="font-size:13px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap"></span></p></span><b style="color:rgb(34,34,34);font-size:small;font-style:normal;font-variant:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);font-family:arial;font-weight:normal"><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt">
<br><span style="font-size:13px;font-family:Arial;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap"></span><span style="font-size:13px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap"></span></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-size:13px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap">The Task 3 training data set is now available and contains 500 scored pairs for each comparison type to use in building and training systems. Please see the task’s </span><a href="http://alt.qcri.org/semeval2014/task3/index.php?id=data-and-tools" target="_blank" style="color:rgb(17,85,204);text-decoration:none"><span style="font-size:13px;font-family:Arial;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap">Data</span></a><span style="font-size:13px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap"> page for further details.</span></p>
<br><br><span style="font-size:13px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap"></span><span style="font-size:13px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap"></span><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt">
<b><span style="font-size:13px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap">IMPORTANT DATES</span></b></p><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-size:13px;font-family:Arial;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap"><br>
</span></p><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-size:13px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap">• Test data ready: <span class="" tabindex="0" style="border-bottom:1px dashed rgb(204,204,204)"><span class="" style>March 10, 2014</span></span></span></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-size:13px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap">• Evaluation start: <span class="" tabindex="0" style="border-bottom:1px dashed rgb(204,204,204)"><span class="" style>March 15, 2014</span></span></span></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-size:13px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap">• Evaluation end: <span class="" tabindex="0" style="border-bottom:1px dashed rgb(204,204,204)"><span class="" style>March 30, 2014</span></span></span></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-size:13px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap">• Paper submission due: <span class="" tabindex="0" style="border-bottom:1px dashed rgb(204,204,204)"><span class="" style>April 30, 2014</span></span> [TBC]</span></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-size:13px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap">• Paper reviews due: <span class="" tabindex="0" style="border-bottom:1px dashed rgb(204,204,204)"><span class="" style>May 30, 2014</span></span></span></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-size:13px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap">• Camera ready due: <span class="" tabindex="0" style="border-bottom:1px dashed rgb(204,204,204)"><span class="" style>June 30, 2014</span></span></span></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-size:13px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap">• </span><span style="font-size:13px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">SemEval workshop <span class="" tabindex="0" style="border-bottom:1px dashed rgb(204,204,204)"><span class="" style>August 23-24, 2014</span></span>, co-located with COLING and *SEM in Dublin, Ireland.</span></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><br><span style="font-size:13px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap"></span></p><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt">
<span style="font-size:13px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap"><br></span></p><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><b><span style="font-size:13px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap">MORE INFORMATION</span></b></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-size:13px;font-family:Arial;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap"><br></span></p><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt">
<span style="font-size:13px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap">The Semeval-2014 Task 3 website includes details on the training data, evaluation, and examples of the comparison types:</span></p>
<span style="font-size:13px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap"></span><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-size:13px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap">   </span><a href="http://alt.qcri.org/semeval2014/task3/" target="_blank" style="color:rgb(17,85,204);text-decoration:none"><span style="font-size:13px;font-family:Arial;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap">http://alt.qcri.org/semeval2014/task3/</span></a><span style="font-size:13px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap"><br>
  <br></span></p><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-size:13px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap">If interested in the task, please join our mailing list for updates:</span></p>
<span style="font-size:13px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap"></span><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-size:13px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap">  </span><a href="https://groups.google.com/d/forum/semeval-2014-clss" target="_blank" style="color:rgb(17,85,204);text-decoration:none"><span style="font-size:13px;font-family:Arial;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap">https://groups.google.com/d/forum/semeval-2014-clss</span></a><span style="font-size:13px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap"></span></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><br><span style="font-size:13px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap"></span></p><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt">
<span style="font-size:13px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap"><br></span></p><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><b><span style="font-size:13px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap">ORGANIZERS</span></b></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-size:13px;font-family:Arial;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap"><br></span></p><p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt">
<span style="font-size:13px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap">David Jurgens (</span><span style="font-size:13px;font-family:Arial;color:rgb(17,85,204);vertical-align:baseline;white-space:pre-wrap"><a href="mailto:jurgens@di.uniroma1.it" target="_blank" style="color:rgb(17,85,204)">jurgens@di.uniroma1.it</a></span><span style="font-size:13px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap">), Sapienza University of Rome, Italy</span></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt"><span style="font-size:13px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap">Mohammad Taher Pilehvar (</span><span style="font-size:13px;font-family:Arial;color:rgb(17,85,204);vertical-align:baseline;white-space:pre-wrap"><a href="mailto:pilehvar@di.uniroma1.it" target="_blank" style="color:rgb(17,85,204)">pilehvar@di.uniroma1.it</a></span><span style="font-size:13px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap">), Sapienza University of Rome, Italy</span></p>
<span style="font-size:13px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap">Roberto Navigli (</span><span style="font-size:13px;font-family:Arial;color:rgb(17,85,204);vertical-align:baseline;white-space:pre-wrap"><a href="mailto:navigli@di.uniroma1.it" target="_blank" style="color:rgb(17,85,204)">navigli@di.uniroma1.it</a></span><span style="font-size:13px;font-family:Arial;vertical-align:baseline;white-space:pre-wrap">), Sapienza University of Rome, Italy</span></b></div>