33.577, Calls: Computational Linguistics, Text/Corpus Linguistics/USA

[The LINGUIST List]

LINGUIST List: Vol-33-577. Tue Feb 15 2022. ISSN: 1069 - 4875.

Subject: 33.577, Calls: Computational Linguistics, Text/Corpus Linguistics/USA

Moderator: Malgorzata E. Cavar (linguist at linguistlist.org)
Student Moderator: Billy Dickson
Managing Editor: Lauren Perkins
Team: Helen Aristar-Dry, Everett Green, Sarah Goldfinch, Nils Hjortnaes,
      Joshua Sims, Billy Dickson, Amalia Robinson, Matthew Fort
Jobs: jobs at linguistlist.org | Conferences: callconf at linguistlist.org | Pubs: pubs at linguistlist.org

Homepage: http://linguistlist.org

Please support the LL editors and operation with a donation at:
           https://funddrive.linguistlist.org/donate/

Editor for this issue: Everett Green <everett at linguistlist.org>
================================================================


Date: Tue, 15 Feb 2022 01:50:26
From: Björn Ross [b.ross at ed.ac.uk]
Subject: 1st Workshop on Novel Evaluation Approaches for Text Classification Systems on Social Media

 
Full Title: 1st Workshop on Novel Evaluation Approaches for Text Classification Systems on Social Media 
Short Title: NEATCLasS 

Date: 06-Jun-2022 - 06-Jun-2022
Location: Atlanta, Georgia, USA 
Contact Person: Björn Ross
Meeting Email: b.ross at ed.ac.uk
Web Site: https://neatclass-workshop.github.io/ 

Linguistic Field(s): Computational Linguistics; Text/Corpus Linguistics 

Call Deadline: 27-Mar-2022 

Meeting Description:

The automatic or semiautomatic analysis of textual data is a key approach to
analyse the massive amounts of user-generated content online, from the
identification of sentiment in text and topic classification to the detection
of abusive language, misinformation or propaganda. However, the development of
such systems faces a crucial challenge. Static benchmarking datasets and
performance metrics are the primary method for measuring progress in the
field, and the publication of research on new systems typically requires
demonstrating an improvement over state-of-the-art approaches in this way.
Yet, these performance metrics can obscure critical failings in current
models. Improvements in metrics often do not reflect improvements in the
real-world performance of models. There is clearly a need to rethink
performance evaluation for text classification and analysis systems to be
usable and trustable.

If unreliable systems achieve astonishing scores with traditional metrics, how
do we recognise progress when we see it? The goal of the workshop on Novel
Evaluation Approaches for Text Classification Systems on Social Media
(NEATCLasS) is to promote the development and use of novel metrics for abuse
detection, sentiment analysis and similar tasks within the community, to
better be able to measure whether models really improve upon the state of the
art, and to encourage a wide range of models to be tested on these new
metrics.


Call for Papers:

Recently there have been attempts to address the problem of benchmarks and
metrics that do not represent performance well. For example, in abusive
language detection, there are both static datasets of hard-to-detect examples
(Röttger et al. 2021) and dynamic approaches for generating such examples
(Calabrese et al. 2021). On the platform DynaBench (Kiela et al. 2021),
benchmarks are dynamic and constantly updated with hard-to-classify examples,
avoiding overfitting a predetermined dataset. However, these approaches only
capture a tiny fraction of issues with benchmarking. There is still much work
to do. 

For the first edition of the workshop on Novel Evaluation Approaches for Text
Classification Systems (NEATCLasS) we welcome submissions discussing such new
evaluation approaches, introducing new or refining existing ones, promoting
the use of novel metrics for abuse detection, sentiment analysis and similar
tasks within the community. Furthermore, the workshop will promote discussion
on the importance, potential and danger of disagreement in tasks that require
subjective judgements. This discussion will also focus on how to evaluate
human annotations, and how to find the most suitable set of annotators (if
any) for a given instance and task. The workshop will solicit, among others,
research papers about

* Issues with current evaluation metrics and benchmarking datasets
* New evaluation metrics
* User-centred (qualitative or quantitative) evaluation of social media text
analysis tools
* Adaptations and translations of novel evaluation metrics for other languages
 
* New datasets for benchmarking  
* Increasing data quality in benchmarking datasets, e.g., avoidance of
selection bias, identification of suitable expert human annotators for tasks
involving subjective judgements  
* Systems that facilitate dynamic evaluation and benchmarking  
* Models that perform better at hard-to-classify instances and novel
evaluation metrics such as AAA, DynaBench and HateCheck  
* Bias, error analysis and model diagnostics  
* Phenomena not captured by existing evaluation metrics (such as models making
the right predictions for the wrong reason)  
* Approaches to mitigating bias and common errors  
* Alternative designs for NLP competitions that evaluate a wide range of model
characteristics (such as bias, error analysis, cross-domain performance)  
* Challenges of downstream applications (in industry, computational social
science, computational communication science, and others) and reflections on
how these challenges can be captured in evaluation metrics

The full call for papers can be found here:
https://neatclass-workshop.github.io/call-for-papers/




------------------------------------------------------------------------------

***************************    LINGUIST List Support    ***************************
 The 2020 Fund Drive is under way! Please visit https://funddrive.linguistlist.org
  to find out how to donate and check how your university, country or discipline
     ranks in the fund drive challenges. Or go directly to the donation site:
                   https://crowdfunding.iu.edu/the-linguist-list

                        Let's make this a short fund drive!
                Please feel free to share the link to our campaign:
                    https://funddrive.linguistlist.org/donate/
 


----------------------------------------------------------
LINGUIST List: Vol-33-577	
----------------------------------------------------------