[Corpora-List] Call for Participation: EVALUATING PHRASAL SEMANTICS (SemEval 2013, Task 5)

[Eugenie Giesbrecht]

				** CALL FOR PARTICIPATION **

			** EVALUATING PHRASAL SEMANTICS **

 						-- as part of --  

				** SemEval-2013, TASK 5 **

			http://www.cs.york.ac.uk/semeval-2013/task5/


Numerous tasks have focused on leveraging the meaning of words (noun categorization, TOEFL test), or of words in context (WSD, metonymy resolution, lexical substitution). These tasks have enjoyed a lot of success. A natural further step is to build up and evaluate models that can perform similar tasks in the face of multiword expressions and complex compositional structures.

We suggest two subtasks that are designed to evaluate phrasal models. Participating systems may attempt any or all of the subtasks, in any or all of the languages provided in the datasets. The sub-tasks are the following:

a) Semantic similarity of words and compositional phrases

b) Evaluating the compositionality of phrases in context


All subtasks are based on items drawn from the large-scale, freely available WaCky corpora (Baroni et al., 2009).  

** Training data as well as further details are available ** @ http://www.cs.york.ac.uk/semeval-2013/task5/

 
(a) Semantic Similarity of Words and Compositional Phrases

The aim of this subtask is to evaluate how well systems can judge the semantic similarity of a word and a short sequence of (two or more) words. For example, in the word-sequence pair: <contact, close interaction> the meaning of the sequence as a whole is semantically close to the meaning of the word. Contrarily, in the sequence-word pair: <megalomania, great madness> the meaning of the word is semantically different from the meaning of the sequence, although it is not entirely unrelated. This task is offered for English, German and Italian.

== Subtask Organizers == 

- Yannis Korkontzelos (National Centre for Text Mining, University of Manchester, UK)
- Fabio Massimo Zanzotto (University of Rome "Tor Vergata", Italy)


 (b) Semantic Compositionality in Context

An interesting sub-problem of semantic compositionality is to decide whether a phrase is used in its literal or figurative meaning in a **given context**. For example, “big picture” may be used literally as in “Click here for a bigger picture”, or figuratively as in “To solve this problem, you have to look at the bigger picture”.

Another example is “old school” which can also be used literally (“During the 1970's the hall of the old school was converted into the library”) or figuratively (“He will go down in history as one of the old school, a true gentlemen”).

Participants of this subtask are provided with a list of target phrases together with real usage examples sampled from WaCky (Baroni et al., 2009) corpora. For each usage example, the task is to make a binary decision whether the target phrase is used literally or figuratively in this context. There may be cases where a binary decision is not possible, then the classification as “both” is allowed.

For English, we define two different subsets that will be scored separately and in combination. One subset of target phrases is accompanied by a large number of contexts, which allows learning single classifiers for each phrase. The validation/test sets will only contain phrases that occur in the training set. This is comparable to the lexical sample task, which encourages the participation of supervised systems.

The other subset contains target phrases together with a smaller number of usage contexts, probably favoring unsupervised approaches. The validation/test sets will contain new target phrases. This is comparable to the all-word task. Here, systems must grasp a notion of literal vs. idiomatic use in general, without training classifiers for each phrase.

For German, only the second setup is offered.

 
== Subtask Organizers ==

- Chris Biemann, UKP Lab, Technische Universität Darmstadt, Germany
- Eugenie Giesbrecht, Karlsruhe Institute of Technology, Germany
- Torsten Zesch, UKP Lab, Technische Universität Darmstadt, Germany

 
SemEval-2013 schedule

	• February 15, 2013 - Registration Deadline [for Task Participants]
	• March 1, 2013 onwards - Start of evaluation period [Task Dependent]
	• March 15, 2013 - End of evaluation period
	• April 9, 2013 - Paper submission deadline [TBC]
	• April 23, 2013 - Reviews Due [TBC]
	• May 4, 2013 - Camera ready Due [TBC]
_______________________________________________
UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
Corpora mailing list
Corpora at uib.no
http://mailman.uib.no/listinfo/corpora