34.1451, Calls: GenBench: The First Workshop on (benchmarking) Generalisation in NLP

Wed May 10 02:05:02 UTC 2023

LINGUIST List: Vol-34-1451. Wed May 10 2023. ISSN: 1069 - 4875.

Subject: 34.1451, Calls: GenBench: The First Workshop on (benchmarking) Generalisation in NLP

Moderator: Malgorzata E. Cavar, Francis Tyers (linguist at linguistlist.org)
Managing Editor: Lauren Perkins
Team: Helen Aristar-Dry, Steven Franks, Everett Green, Joshua Sims, Daniel Swanson, Matthew Fort, Maria Lucero Guillen Puon, Zackary Leech, Lynzie Coburn
Jobs: jobs at linguistlist.org | Conferences: callconf at linguistlist.org | Pubs: pubs at linguistlist.org

Homepage: http://linguistlist.org

Please support the LL editors and operation with a donation at:
           https://funddrive.linguistlist.org/donate/

Editor for this issue: Everett Green <everett at linguistlist.org>
================================================================

Date: 09-May-2023
From: Dieuwke Hupkes [dieuwkehupkes at gmail.com]
Subject: GenBench: The First Workshop on (benchmarking) Generalisation in NLP

Full Title: GenBench: The First Workshop on (benchmarking)
Generalisation in NLP
Short Title: GenBench

Date: 06-Dec-2023 - 07-Dec-2023
Location: Singapore, Singapore
Contact Person: Dieuwke Hupkes
Meeting Email: genbench at googlegroups.com
Web Site: https://genbench.org/workshop/

Linguistic Field(s): Computational Linguistics

Call Deadline: 01-Sep-2023

Meeting Description:

The ability to generalise well is often mentioned as one of the
primary desiderata for computational models of natural language
processing (NLP). Yet, what good generalisation entails and how it
should be evaluated is not well understood, nor are there any common
standards to evaluate it in the field of NLP (Hupkes et al., 2022). As
a result, it is difficult to understand what the current state of the
field is when it comes to generalisation. It is difficult to
understand how results in this area relate to each other, what sorts
of generalisation are being addressed and which are neglected, which
forms of generalisation testing we should prioritise in which types of
scenarios, and how we can adequately assess generalisation in the
first place. Missing answers to all of those questions are standing in
the way of better model development: what we cannot measure, we cannot
improve. The GenBench workshop on (benchmarking) generalisation in NLP
aims to serve as a cornerstone to catalyse research on generalisation
in the NLP community. The workshop has two concrete goals:

- Bring together different expert communities to discuss challenging
questions relating to generalisation in NLP.
- Establish a shared platform for state-of-the-art generalisation
testing in NLP, with a leaderboard for a selection of tests that are
created and selected not by one group, but by a larger community.

Call for Papers:

GenBench: The First Workshop on (benchmarking) Generalisation in NLP

The ability to generalise well is often mentioned as one of the
primary desiderata for models of natural language processing. It is
crucial to ensure that models behave robustly, reliably and fairly
when making predictions about data that is different from the data
that they were trained on. Generalisation is also important when NLP
models are considered from a cognitive perspective, as models of human
language. Yet, there are still many open questions related to what it
means for an NLP model to generalise well, and how generalisation
should be evaluated. The first GenBench workshop aims to serve as a
cornerstone to catalyse research on generalisation in the NLP
community. In particular the workshop aims to:
- Bring together different expert communities to discuss challenging
questions relating to generalisation in NLP;
- Crowd-source a collaborative generalisation benchmark, hosted on a
platform for democratic state-of-the-art (SOTA) generalisation testing
in NLP

For both goals, we believe it is of utmost important to get
information from domain experts such as linguists, which is why we
reach out specifically to your community.

The first GenBench workshop on generalisation (benchmarking) in NLP
will be co-located with EMNLP 2023.

*Submission types*
We call for two types of submissions: regular workshop submissions and
collaborative benchmarking task submissions. The latter will consist
of a data/task artefact and a companion paper motivating and
evaluating the submission. In both cases, we accept archival papers
and extended abstracts.

1. Regular workshop submissions
Regular workshop submissions present papers on the topic of
generalisation (see examples listed below), but are not intended to be
included on the GenBench evaluation platform. Regular workshop papers
may be submitted as an archival paper, when they report on completed,
original and unpublished research; or as a shorter extended abstract.
More details on this category can be found on our website.

If you are unsure whether a specific topic is well-suited for
submission, feel free to reach out to the organisers of the workshop
at genbench at googlegroups.com.

2. Collaborative Benchmarking Task submissions
Collaborative benchmarking task submissions consist of a data/task
artefact and a paper describing and motivating the submission and
showcasing it on a select number of models.

We accept submissions that introduce new datasets, resplits of
existing datasets along particular dimensions, or in-context learning
tasks, with the goal of measuring generalisation of NLP models. We
especially encourage submissions that focus on:
- Generalisation in the context of fairness and inclusivity
- Multilingual generalisation
- Generalisation in LLMs, where we have no control over the training
data

More details about the collaborative benchmark submissions and example
submissions can be found on our website: genbench.org/cbt.

*Important dates*
August 1, 2023 – Sample data submission deadline
September 1, 2023 – Paper submission deadline
September 15, 2023 – ARR submission deadline
October 6, 2023 – Notification deadline
October 18, 2023 – Camera ready deadline
December 6/7, 2023 – Workshop

*Contact*
Email address: genbench at googlegroups.com
Website: http://genbench.org/workshop

------------------------------------------------------------------------------

LINGUIST List is supported by the following publishers:

American Dialect Society/Duke University Press http://dukeupress.edu

Bloomsbury Publishing (formerly The Continuum International Publishing Group) http://www.bloomsbury.com/uk/

Brill http://www.brill.com

Cambridge Scholars Publishing http://www.cambridgescholars.com/

Cambridge University Press http://www.cambridge.org/linguistics

Cascadilla Press http://www.cascadilla.com/

De Gruyter Mouton https://cloud.newsletter.degruyter.com/mouton

Dictionary Society of North America http://dictionarysociety.com/

Edinburgh University Press www.edinburghuniversitypress.com

Equinox Publishing Ltd http://www.equinoxpub.com/

European Language Resources Association (ELRA) http://www.elra.info

Georgetown University Press http://www.press.georgetown.edu

John Benjamins http://www.benjamins.com/

Lincom GmbH https://lincom-shop.eu/

Linguistic Association of Finland http://www.ling.helsinki.fi/sky/

MIT Press http://mitpress.mit.edu/

Multilingual Matters http://www.multilingual-matters.com/

Narr Francke Attempto Verlag GmbH + Co. KG http://www.narr.de/

Netherlands Graduate School of Linguistics / Landelijke (LOT) http://www.lotpublications.nl/

Oxford University Press http://www.oup.com/us

Wiley http://www.wiley.com

----------------------------------------------------------
LINGUIST List: Vol-34-1451
----------------------------------------------------------