[Corpora-List] [Call for Participation] SemEval-2014 Task 3 : Cross-Level Semantic Similarity

[Taher Pilehvar]

(sorry for cross-posting)


Call For Participation


Cross-Level Semantic Similarity

SemEval 2014 - Task 3

http://alt.qcri.org/semeval2014/task3/



To what extent is the meaning of the sentence "do u know where i can watch
free older movies online without download?" preserved in the phrase
"streaming vintage movies for free"?  Or, how similar is "circumscribe" to
the phrases "beating around the bush" and "avoiding the topic"? The aim of
Task 3 is to evaluate semantic similarity when comparing items of different
syntactic and semantic scales, such as paragraphs. sentences, phrases,
words, and senses.


INTRODUCTION


Semantic similarity is an essential component of many applications in
Natural Language Processing (NLP). Task 3 provides an evaluation for
semantic similarity across different types of text, which we refer to as
lexical levels. Unlike prior SemEval tasks on textual similarity that have
focused on comparing similar-sized texts, this task evaluates the case
where a smaller text or sense is compared to the meaning of a larger text.
Specifically, this task encompasses four types of semantic similarity
comparisons:


   1.

   paragraph to sentence,
   2.

   sentence to phrase,
   3.

   phrase to word, and
   4.

   word to sense.


Task 3 unifies multiple objectives from different areas of NLP under a
single task, e.g., Paraphrasing, Summarization, and Compositional
Semantics. One of the major motivations of this task is to produce systems
that handle all comparison types, thereby freeing downstream NLP
applications from needing to consider the type of text being compared.


TASK DETAILS

Task participants are provided with pairs of each comparison type and asked
to rate to what extent does the smaller item preserve the meaning of the
larger item.  For example, given a sentence and a paragraph, a system would
assess how similar is the meaning of the sentence to the meaning of the
paragraph. Ideally, a high-similarity sentence would summarize overall
meaning of the paragraph.  Similarly, in the case of word to sense, we
assess how well the semantics a word is captured by a WordNet 3.0 sense.


PARTICIPATING

Teams are free to participate in one, some, or all comparison types.  Given
the unified setting of the task, we especially encourage systems that
handle more than one comparison type.  However, we also allow specialized
systems that target only a single comparison type.

Interested teams are encouraged to join the task's mailing
list<https://groups.google.com/d/forum/semeval-2014-clss>for
discussion and announcements.


EVALUATION

Systems will be evaluated against human similarity scores using both
rank-based and score-based comparisons.  See the task's
Evaluation<http://alt.qcri.org/semeval2014/task3/index.php?id=evaluation>page
for further details.


DATASETS

The Task 3 training data set is now available and contains 500 scored pairs
for each comparison type to use in building and training systems. Please
see the task's Data<http://alt.qcri.org/semeval2014/task3/index.php?id=data-and-tools>page
for further details.


IMPORTANT DATES

* Test data ready: March 10, 2014

* Evaluation start: March 15, 2014

* Evaluation end: March 30, 2014

* Paper submission due: April 30, 2014 [TBC]

* Paper reviews due: May 30, 2014

* Camera ready due: June 30, 2014

* SemEval workshop August 23-24, 2014, co-located with COLING and *SEM in
Dublin, Ireland.



MORE INFORMATION


The Semeval-2014 Task 3 website includes details on the training data,
evaluation, and examples of the comparison types:

   http://alt.qcri.org/semeval2014/task3/
If interested in the task, please join our mailing list for updates:

  https://groups.google.com/d/forum/semeval-2014-clss



ORGANIZERS


David Jurgens (jurgens at uniroma), Sapienza University of Rome, Italy

Mohammad Taher Pilehvar (pilehvar at uniroma), Sapienza University of Rome,
Italy

Roberto Navigli (navigli at uniroma), Sapienza University of Rome, Italy

(uniroma=>di.uniroma1.it)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listserv.linguistlist.org/pipermail/corpora/attachments/20140310/33149d13/attachment.htm>
-------------- next part --------------
_______________________________________________
UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
Corpora mailing list
Corpora at uib.no
http://mailman.uib.no/listinfo/corpora