[Corpora] [Corpora-List] Call for participation: The First Shared Task on Russian Semantic Similarity Evaluation (RUSSE)

[Alexander Panchenko]

Dear all,

I hope some of you interested in semantic similarity can find this
interesting. I apologise for possible cross-postings.

The First Shared Task on Russian Semantic Similarity Evaluation (RUSSE)

We are happy to announce two shared tasks of semantic similarity
evaluation: the relatedness and the association tasks. These tasks are
quite accessible, so we encourage both beginners and experts to
participate. Short descriptions of the tasks are given below. Potential
participants should register in order to participate here as soon as
possible: http://goo.gl/xy8fWC.

Motivation, Contribution and Expected Results

A semantic similarity measure quantifies the semantic relatedness words or
multiword expressions. It yields high values for the pairs of words in a
semantic relation (synonyms, hyponyms, associations or co-hyponyms) and low
values for all other pairs. Such measures proved to be useful for text
processing applications, including text similarity, query expansion,
question answering and word sense disambiguation.

Many methods for semantic similarity were proposed and tested during the
last 20 years. Recently with the advent of neural network language models
yielding state-of-the-art results on the semantic similarity task the
interest in this field increased even more. Many authors tried to performed
exhaustive comparisons of semantic similarity measures and developed a
whole range of benchmarks and evaluations datasets.

Unfortunately, most of the approaches to semantic similarity were
implemented and evaluated only on a handful of European languages, mostly
in English. While some Russian researchers sporadically tried to adopt
several methods developed for English, these efforts were mostly done in a
context of some specific applications without any proper evaluation. To the
best of our knowledge, no systematic investigation of semantic similarity
measures of the Russian language was ever performed.

The goal of the RUSSE shared task is to fill this gap and to conduct an
evaluation campaign of key currently available methods. The RUSSE
competition will perform a systematic comparison of the currently available
methods in the context of Russian language. This will let us identify
specific features of the semantic similarity phenomena in Russian language.
The event will be organized in a form of a competition of systems that
calculate semantic similarity between word.

Tasks Descriptions

1. The Relatedness Task. In this task, two words are considered similar if
they are synonyms, hypernyms or co-hyponyms. Two benchmarks will be
provided for this task. The first dataset is based on human judgments about
semantic similarity (a translated version of the widely used MC, RG, and
WordSim353 datasets). The second dataset follows structure of the BLESS and
derived from the thesaurus RuThes Lite.

2. The Association Task. In this task, two words are considered similar if
the second is an association of the first one. We use results of the
large-scale Russian associative experiment in order to build training and
test collections.

Important Dates

- 22th October 2014 – announcement of the task
- 17th November 2014 – release of the training data
- 10th January 2015 – deadline for submissions
- 15th January 2015 – release of the test data
- 25th January 2015 – announcement of evaluation results
- 15th February 2015 – deadline for Dialogue paper submission

Organizing Committee

- Natalia Loukachevitch, Research Computing Center, Moscow State University
- Alexander Panchenko, Digital Society Laboratory LLC & Université
catholique de Louvain
- Dmitry Ustalov, NLPub & Krasovsky Institute of Mathematics and Mechanics
UB RAS
- Denis Paperno, Center for Brain/Mind Sciences, University of Trento
- Christian M. Meyer, UKP Lab, Technische Universität Darmstadt
- Ilia Chetviorkin, Opiner.ru & Moscow State University
- Andrey Philippovich, Bauman Moscow State Technical University

Further details, including task rationale, schedule and datasets can be
found on the RUSSE website: http://russe.nlpub.ru/. Participants will be
invited to submit a paper to the Dialogue 2015 conference describing their
system. Inquiries and suggestions are welcome at Alexander dot  Panchenko
AT uclouvain.be.

Best regards,
Organizing team of the RUSSE
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listserv.linguistlist.org/pipermail/corpora/attachments/20141028/99fecb10/attachment.htm>
-------------- next part --------------
_______________________________________________
UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
Corpora mailing list
Corpora at uib.no
http://mailman.uib.no/listinfo/corpora