36.1746, Confs: LeWiDi: Shared task on Learning with Disagreement (China)

[The LINGUIST List]

LINGUIST List: Vol-36-1746. Wed Jun 04 2025. ISSN: 1069 - 4875.

Subject: 36.1746, Confs: LeWiDi: Shared task on Learning with Disagreement (China)

Moderator: Steven Moran (linguist at linguistlist.org)
Managing Editor: Justin Fuller
Team: Helen Aristar-Dry, Steven Franks, Joel Jenkins, Daniel Swanson, Erin Steitz
Jobs: jobs at linguistlist.org | Conferences: callconf at linguistlist.org | Pubs: pubs at linguistlist.org

Homepage: http://linguistlist.org

Editor for this issue: Valeriia Vyshnevetska <valeriia at linguistlist.org>

================================================================


Date: 02-Jun-2025
From: Silvia Casola [s.casola at lmu.de]
Subject: LeWiDi: Shared task on Learning with Disagreement


LeWiDi: Shared task on Learning with Disagreement

Location: Suzhou, China

Linguistic Field(s): Computational Linguistics

Submission Deadline: 01-Aug-2025

LeWiDi: Shared task on Learning With Disagreement - Call for
participation
We'd like to invite researchers in disagreement and variation  to
participate in the third edition of the LeWidi shared tasks held in
conjunction with the NLPerspectives workshop at the EMNLP conference
in Suzhou, China.
The LeWiDi series is positioned within the growing body of research
that questions the practice of label harmonization and the reliance on
a single ground truth in AI and NLP. This year's shared task
challenges participants to leverage both instance-level disagreement
and annotator-level information in classification. The proposed tasks
include ones that address disagreement in both generation and
labeling—with a dataset for Natural Language Inference (NLI) and
another for paraphrase detection—as well as subjective tasks,
including irony and sarcasm detection.
Competition webpage:  https://www.codabench.org/competitions/7192/
Subtasks and Datasets:
Participants will be able to submit to subtasks exploring different
types of disagreement through dedicated datasets:
1. The Conversational Sarcasm corpus (CSC) – a dataset of
context+response pairs rated for sarcasm, with ratings from 1 to 6.
2. The MultiPico dataset (MP) – a crowdsourced multilingual irony
detection dataset. Annotators were tasked to detect whether a reply
was ironic in the context of a brief post-reply exchange on social
media. Annotators ids and metadata (gender, age, nationality, etc) are
available. Languages include Arabic, German, English, Spanish, French,
Hindi, Italian, Dutch, and Portuguese.
3. The Paraphrase dataset (Par) – a dataset of question pairs for
which the annotators had to tell whether the two questions are
paraphrases of each other, using values on a Likert scale.
4. TheVariErrNLI dataset (VariErrNLI) – a dataset originally designed
for automatic error detection, distinguishing between annotation
errors and legitimate human label variations in Natural Language
Inference.
Participants will be able to submit to one or multiple datasets.
Tasks and Evaluation:
In this edition, only soft evaluation metrics will be used. We will
however experiment with two forms of tasks and evaluation:
TASK A (SOFT LABEL PREDICTION): Systems will be asked to output a
probability distribution of the values. EVALUATION: the distance
between this predicted soft label and that resulting from human
annotations will be computed.
TASK B (PERSPECTIVIST PREDICTION): Systems will be asked to predict
each annotator's label on items. EVALUATION: a measure of correctness
of the predictions
Participants will be able to submit to one or both tasks.
Important Dates:
Training data ready: May 15th 2025
Evaluation starts: June 15th 2025
Evaluation ends: July 10th 2025
Paper submission due: August 1st, 2025
Notification to authors: August 25th, 2025
Camera-ready deadline: September 12, 2025
NLPerspectives workshop: November 8, 2025
We are looking forward to your submission!



------------------------------------------------------------------------------

********************** LINGUIST List Support ***********************
Please consider donating to the Linguist List to support the student editors:

https://www.paypal.com/donate/?hosted_button_id=87C2AXTVC4PP8

LINGUIST List is supported by the following publishers:

Bloomsbury Publishing http://www.bloomsbury.com/uk/

Cambridge University Press http://www.cambridge.org/linguistics

Cascadilla Press http://www.cascadilla.com/

De Gruyter Mouton https://cloud.newsletter.degruyter.com/mouton

Edinburgh University Press http://www.edinburghuniversitypress.com

Elsevier Ltd http://www.elsevier.com/linguistics

John Benjamins http://www.benjamins.com/

Language Science Press http://langsci-press.org

Lincom GmbH https://lincom-shop.eu/

Multilingual Matters http://www.multilingual-matters.com/

Netherlands Graduate School of Linguistics / Landelijke (LOT) http://www.lotpublications.nl/

Oxford University Press http://www.oup.com/us

Wiley http://www.wiley.com


----------------------------------------------------------
LINGUIST List: Vol-36-1746
----------------------------------------------------------