34.2192, Calls: GenBench: The First Workshop on (benchmarking) Generalisation in NLP

[The LINGUIST List]

LINGUIST List: Vol-34-2192. Wed Jul 12 2023. ISSN: 1069 - 4875.

Subject: 34.2192, Calls: GenBench: The First Workshop on (benchmarking) Generalisation in NLP

Moderators: Malgorzata E. Cavar, Francis Tyers (linguist at linguistlist.org)
Managing Editor: Justin Fuller
Team: Helen Aristar-Dry, Steven Franks, Everett Green, Daniel Swanson, Maria Lucero Guillen Puon, Zackary Leech, Lynzie Coburn, Natasha Singh, Erin Steitz
Jobs: jobs at linguistlist.org | Conferences: callconf at linguistlist.org | Pubs: pubs at linguistlist.org

Homepage: http://linguistlist.org

Please support the LL editors and operation with a donation at:
           https://funddrive.linguistlist.org/donate/

Editor for this issue: Everett Green <everett at linguistlist.org>
================================================================


Date: 11-Jul-2023
From: Dieuwke Hupkes [dieuwkehupkes at gmail.com]
Subject: GenBench: The First Workshop on (benchmarking) Generalisation in NLP


Full Title: GenBench: The First Workshop on (benchmarking)
Generalisation in NLP
Short Title: GenBench

Date: 06-Dec-2023 - 07-Dec-2023
Location: Singapore, Singapore
Contact Person: Dieuwke Hupkes
Meeting Email: genbench at googlegroups.com
Web Site: https://genbench.org/workshop/

Linguistic Field(s): Computational Linguistics

Call Deadline: 01-Sep-2023

Meeting Description:

The ability to generalise well is often mentioned as one of the
primary desiderata for computational models of natural language
processing (NLP). Yet, what good generalisation entails and how it
should be evaluated is not well understood, nor are there any common
standards to evaluate it in the field of NLP (Hupkes et al., 2022). As
a result, it is difficult to understand what the current state of the
field is when it comes to generalisation. It is difficult to
understand how results in this area relate to each other, what sorts
of generalisation are being addressed and which are neglected, which
forms of generalisation testing we should prioritise in which types of
scenarios, and how we can adequately assess generalisation in the
first place. Missing answers to all of those questions are standing in
the way of better model development: what we cannot measure, we cannot
improve. The GenBench workshop on (benchmarking) generalisation in NLP
aims to serve as a cornerstone to catalyse research on generalisation
in the NLP community. The workshop has two concrete goals:

- Bring together different expert communities to discuss challenging
questions relating to generalisation in NLP.
- Establish a shared platform for state-of-the-art generalisation
testing in NLP, with a leaderboard for a selection of tests that are
created and selected not by one group, but by a larger community.

2nd Call for Papers:

***New: the Collaborative Benchmarking Task submissions are now open;
visit https://github.com/GenBench/genbench_cbt.
With the support of our workshop sponsor Amazon, we are now offering
scholarships for travel expenses.***

*Workshop description
The ability to generalise well is often mentioned as one of the
primary desiderata for models of natural language processing.
It is crucial to ensure that models behave robustly, reliably and
fairly when making predictions about data that is different from the
data that they were trained on. Generalisation is also important when
NLP models are considered from a cognitive perspective, as models of
human language. Yet, there are still many open questions related to
what it means for an NLP model to generalise well and how
generalisation should be evaluated.

The first GenBench workshop aims to serve as a cornerstone to catalyse
research on generalisation in the NLP community.
In particular, the workshop aims to
- bring together different expert communities to discuss challenging
questions relating to generalisation in NLP;
- crowd-source a collaborative generalisation benchmark hosted on a
platform for democratic state-of-the-art (SOTA) generalisation testing
in NLP.

*Submission types*
We call for two types of submissions: regular workshop submissions and
collaborative benchmarking task (CBT) submissions.
The latter will consist of a data/task artefact and a companion paper
motivating and evaluating the submission. In both cases, we accept
archival papers and extended abstracts.

1. Regular workshop submissions
Regular workshop submissions present papers on the topic of
generalisation.  Topics of interest include, but are not limited to:
- Opinion or position papers about generalisation and how it should be
evaluated;
- Analyses of how existing or new models generalise;
- Empirical studies that propose new paradigms to evaluate
generalisation;
- Meta-analyses that investigate how results from different
generalisation studies compare to one another;
- Meta-analyses that study how different types of generalisation are
related;
- Papers that discuss how generalisation of LLMs can be evaluated
without access to training data;
- Papers that discuss why generalisation is (not) important in the era
of LLMs;
- Studies on the relationship between generalisation and fairness or
robustness.

2. Collaborative Benchmarking Task submissions
CBT submissions consist of a data/task artefact and a paper describing
and motivating the submission and showcasing it on a select number of
models. We accept submissions that introduce new datasets, resplits of
existing datasets along particular dimensions, or in-context learning
tasks, with the goal of measuring generalisation of NLP models.

We especially encourage submissions that focus on:
- Generalisation in the context of fairness and inclusivity;
- Multilingual generalisation;
- Generalisation in LLMs, where we have no control over the training
data.

More details about the CBT submissions and example submissions can be
found on our website: visit genbench.org/cbt or or
github.com/GenBench/genbench_cbt

*Important Dates*
August 1, 2023 – Sample data submission deadline
September 1, 2023 – Paper submission deadline
September 15, 2023 – ARR submission deadline
October 6, 2023 – Notification deadline
October 18, 2023 – Camera-ready deadline
December 6, 2023 – Workshop @EMNLP

All papers can be submitted through OpenReview:
https://openreview.net/group?id=GenBench.org/2023/Workshop

*Preprints*
We do not have an anonymity deadline. Preprints are allowed, both
before the submission deadline as well as after.

*Scholarships*
With the support of our workshop sponsor Amazon, we are offering 6
scholarships each covering up to to $500 of travel expenses and/or
(virtual) registration fees. Please check out our website for more
information about the application process.

*Contact*
Email address: genbench at googlegroups.com
Website: genbench.org/workshop



------------------------------------------------------------------------------

Please consider donating to the Linguist List https://give.myiu.org/iu-bloomington/I320011968.html


LINGUIST List is supported by the following publishers:

American Dialect Society/Duke University Press http://dukeupress.edu

Bloomsbury Publishing (formerly The Continuum International Publishing Group) http://www.bloomsbury.com/uk/

Brill http://www.brill.com

Cambridge Scholars Publishing http://www.cambridgescholars.com/

Cambridge University Press http://www.cambridge.org/linguistics

Cascadilla Press http://www.cascadilla.com/

De Gruyter Mouton https://cloud.newsletter.degruyter.com/mouton

Dictionary Society of North America http://dictionarysociety.com/

Edinburgh University Press www.edinburghuniversitypress.com

Equinox Publishing Ltd http://www.equinoxpub.com/

European Language Resources Association (ELRA) http://www.elra.info

Georgetown University Press http://www.press.georgetown.edu

John Benjamins http://www.benjamins.com/

Lincom GmbH https://lincom-shop.eu/

Linguistic Association of Finland http://www.ling.helsinki.fi/sky/

MIT Press http://mitpress.mit.edu/

Multilingual Matters http://www.multilingual-matters.com/

Narr Francke Attempto Verlag GmbH + Co. KG http://www.narr.de/

Netherlands Graduate School of Linguistics / Landelijke (LOT) http://www.lotpublications.nl/

Oxford University Press http://www.oup.com/us

SIL International Publications http://www.sil.org/resources/publications

Springer Nature http://www.springer.com

Wiley http://www.wiley.com


----------------------------------------------------------
LINGUIST List: Vol-34-2192
----------------------------------------------------------