33.142, Calls: Computational Linguistics/USA

Tue Jan 18 05:41:51 UTC 2022

LINGUIST List: Vol-33-142. Tue Jan 18 2022. ISSN: 1069 - 4875.

Subject: 33.142, Calls: Computational Linguistics/USA

Moderator: Malgorzata E. Cavar (linguist at linguistlist.org)
Student Moderator: Billy Dickson
Managing Editor: Lauren Perkins
Team: Helen Aristar-Dry, Everett Green, Sarah Goldfinch, Nils Hjortnaes,
      Joshua Sims, Billy Dickson, Amalia Robinson, Matthew Fort
Jobs: jobs at linguistlist.org | Conferences: callconf at linguistlist.org | Pubs: pubs at linguistlist.org

Homepage: http://linguistlist.org

Please support the LL editors and operation with a donation at:
           https://funddrive.linguistlist.org/donate/

Editor for this issue: Everett Green <everett at linguistlist.org>
================================================================

Date: Tue, 18 Jan 2022 00:40:49
From: Roberto Zamparelli [roberto.zamparelli at unitn.it]
Subject: SEMEVAL 2022 Task 3

Full Title: SEMEVAL 2022 Task 3 
Short Title: SEMEVAL PreTENS 

Date: 10-Jul-2022 - 15-Jul-2022
Location: Online / Seattle WA, USA 
Contact Person: Roberto Zamparelli
Meeting Email: roberto.zamparelli at unitn.it
Web Site: https://sites.google.com/view/semeval2022-pretens 

Linguistic Field(s): Computational Linguistics 

Subject Language(s): Ehueun (ehu)

Call Deadline: 20-Jan-2022 

Meeting Description:

SemEval 2022 Task 3 Presupposed Taxonomies: Evaluating Neural Network
Semantics (PreTENS)

Task Page: https://sites.google.com/view/semeval2022-pretens/

This SemEval 2022 Task is aimed to encourage the development of general
methods to detect semantically infelicitous sentences. 
All participating teams will be invited to submit a task description paper in
the proceedings published by ACL.

Call for Papers:

This SemEval 2022 Task is aimed to encourage the development of general
methods to detect semantically infelicitous sentences. All participating teams
will be invited to submit a task description paper in the proceedings
published by ACL.

Motivation

A growing body of literature on the computational linguistics has tried to
probe the metalinguistic abilities of modern
language models, including the ability to recognize linguistic structures that
are deviant at the syntactic and/or
semantic level (e.g. ??''Who does speaking to bothers Anna'', ??''I like cats,
and in particular hamsters''). This can be used
to probe the cognitive plausibility of modern NLP models, but can also find
application in the detection of
writing/reasoning errors more subtle than what current grammar checkers can
detect. We focus on a case of purely
semantic deviance that requires the ability to recognize lexical relationships
between words and a capacity for
generalization.

Task Overview

SEMEVAL 2022 Task 3 will comprise datasets in 3 languages: English, Italian,
French. The French and Italian are slightly
adapted, randomly ordered translations of the English dataset. Each dataset
will contain about 20,394 artificially
generated sentences that exemplify constructions which enforce presuppositions
on a certain taxonomic status of their
arguments A and B (i.e. whether A denotes a subset of B or vice-versa). Some
constructions require their arguments not to
be in a taxonomic relation (e.g. comparatives ''I like A more than B'', see
??''I like trees more than oaks''), others
require a taxonomic relation in a specific order, e.g. exemplifications (I
like A, and in particular B) or
generalizations (I like A, and B in general), yet others may be ambiguous with
respect to their taxonomic requirements.

The argument nouns A and B are taken from 30 semantic categories (e.g. dogs,
birds, mammals, cars, motorcycles, cutlery,
clothes, trees, plastics...).

Participants have the freedom to choose a subset of sub-tasks or settings that
they'd like to participate in (see
sections detailing each of the subtasks). The evaluation will be carried out
as the average of the three languages (with
0 given to languages not submitted).

This task consists of two subtasks:
Subtask A

A binary classification task aimed at determining whether a sentence contains
the correct taxonomic configuration.  All
sentences for this sub-task will be provided with an acceptability label such
as in the following examples:

    I like trees, and in particular birches    1
    I like oaks, and in particular trees    0

Where the labels (1 = acceptable, 0 = unacceptable) are derived from the
theoretical semantic analysis of the various constructions.  
For this binary classification sub-task, the evaluation metric will be based
on Precision, Recall and F-score; the final
ranking will be based on F-score.

Subtask B

A set of 1,533 sentences (mostly a subset of the whole dataset), corresponding
to about 5% of the total and
representative of the patterns considered, was judged by human annotators via
a crowdsourcing campaign on a seven point
Likert-scale, ranging from 1 (not at all acceptable) to 7 (completely
acceptable). In this case, the sentences will be
provided with the average judgment they received, which could be affected by
plausibility considerations, argument order
and other factors. Examples of data for this conditions are:

    I like politicians, an interesting type of farmer 1.42
    I like governors, an interesting type of politician 6.16

For this sub-task, the evaluation metric will be based on Spearman’s rank
correlation coefficient between the task
participants’ scores and the test set scores

The two sub-tasks are independent. Participants can decide to participate in
just one of the  Participants can decide to participate in just one of them,
though we encourage participation in multiple subtasks.

Important Dates

Evaluation starts: January 15, 2022
Evaluation ends: January 20, 2022
Paper submissions due: (TBC) February 23, 2022
Notification to authors: March 31, 2022

Organization:
Dominique Brunato - Institute for Computational Linguistics ''A. Zampolli''
(CNR), Pisa, Italy
Cristiano Chesi - University School for Advanced Studies (IUSS), Pavia, Italy
Shammur Absar Chowdhury - Qatar Computing Research Institute, HBKU, Qatar
Felice Dell'Orletta - Institute for Computational Linguistics ''A. Zampolli''
(CNR), Pisa, Italy
Simonetta Montemagni - Institute for Computational Linguistics ''A. Zampolli''
(CNR), Pisa, Italy
Giulia Venturi -  Institute for Computational Linguistics ''A. Zampolli''
(CNR), Pisa, Italy
Roberto Zamparelli - CIMEC - Mind/Brain Center - University of Trento

------------------------------------------------------------------------------

***************************    LINGUIST List Support    ***************************
 The 2020 Fund Drive is under way! Please visit https://funddrive.linguistlist.org
  to find out how to donate and check how your university, country or discipline
     ranks in the fund drive challenges. Or go directly to the donation site:
                   https://crowdfunding.iu.edu/the-linguist-list

                        Let's make this a short fund drive!
                Please feel free to share the link to our campaign:
                    https://funddrive.linguistlist.org/donate/

----------------------------------------------------------
LINGUIST List: Vol-33-142	
----------------------------------------------------------