[Corpora-List] First CFP: SemEval-2014 Task 3 - Cross-Level Semantic Similarity

[David Jurgens]

Call For Participation


Cross-Level Semantic Similarity

SemEval 2014 - Task 3

http://alt.qcri.org/semeval2014/task3/



The aim of this task is to evaluate semantic similarity when comparing
lexical items of different types, such as paragraphs, sentences, phrases,
words, and senses.


INTRODUCTION
Semantic similarity is an essential component of many applications in
Natural Language Processing (NLP). This task provides an evaluation for
semantic similarity across different types of text, which we refer to as
lexical levels. Unlike prior SemEval tasks on textual similarity that have
focused on comparing similar-sized texts, this task evaluates the case
where larger text must be compared to smaller text, or even to senses.
Specifically, this task encompasses four types of semantic similarity
comparisons:

   1.

   paragraph to sentence,
   2.

   sentence to phrase,
   3.

   phrase to word, and
   4.

   word to sense.

Task 3 unifies multiple objectives from different areas of NLP under a
single task, e.g., Paraphrasing, Summarization, and Compositional
Semantics. One of the major motivations of this task is to produce systems
that handle all comparison types, thereby freeing downstream NLP
applications from needing to consider the type of text being compared.

TASK DETAILS:

Task participants will be provided with pairs of each comparison type and
asked to rate how similar is the meaning of the smaller item to the overall
meaning of the larger item.  For example, given a sentence and a paragraph,
a system would assess how similar is the meaning of the sentence to the
meaning of the paragraph. Ideally, a high-similarity sentence would reflect
overall meaning of the paragraph.

For word-to-sense comparisons, two evaluation settings are used: (1)
out-of-context and (2) in-context. In the out-of-context setting, a sense
is paired with a word in isolation. In the in-context setting, a sense is
compared with the meaning of a usage appearing in some context.  Task 3
uses the WordNet 3.1 sense inventory.

PARTICIPATING

Teams are free to participate in one, some, or all comparison types.  Given
the unified setting of the task, we especially encourage systems that
handle all comparison types.  However, we also allow specialized systems
that target only a single comparison type.

Interested teams are encouraged to join the task’s mailing
list<https://groups.google.com/d/forum/semeval-2014-clss>for
discussion and announcements.

EVALUATION

Systems will be evaluated against human similarity scores using both
rank-based and score-based comparisons.  See the task’s
Evaluation<http://alt.qcri.org/semeval2014/task3/index.php?id=evaluation>page
for further details.


DATASETS:

The Task 3 trial data set is now available and contains tens of examples
for each comparison type to use in building initial systems.  The full
training data will be released later in December. Please see the
task’s Data<http://alt.qcri.org/semeval2014/task3/index.php?id=data-and-tools>page
for further details.


IMPORTANT DATES

   -

   Trial data ready October 31, 2013
   -

   Training data ready December 15, 2013
   -

   Evaluation period March 15-30, 2014
   -

   Paper submission due April 30, 2014 [TBC]
   -

   SemEval workshop August 23-24, 2014, co-located with COLING and *SEM in
   Dublin, Ireland.


MORE INFORMATION

The Semeval-2014 Task 3 website includes details on the training data,
evaluation, and examples of the comparison types:

  http://alt.qcri.org/semeval2014/task3/


If interested in the task, please join our mailing list for updates:

 https://groups.google.com/d/forum/semeval-2014-clss


ORGANIZERS


David Jurgens (jurgens at di.uniroma1.it), Sapienza University of Rome, Italy

Mohammad Taher Pilehvar (pilehvar at di.uniroma1.it), Sapienza University of
Rome, Italy

Roberto Navigli (navigli at di.uniroma1.it), Sapienza University of Rome, Italy
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listserv.linguistlist.org/pipermail/corpora/attachments/20131112/a112a1ca/attachment.htm>
-------------- next part --------------
_______________________________________________
UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
Corpora mailing list
Corpora at uib.no
http://mailman.uib.no/listinfo/corpora