28.727, FYI: Call for Task Proposals - SemEval-2018

Tue Feb 7 18:51:35 UTC 2017

LINGUIST List: Vol-28-727. Tue Feb 07 2017. ISSN: 1069 - 4875.

Subject: 28.727, FYI: Call for Task Proposals - SemEval-2018

Moderators: linguist at linguistlist.org (Damir Cavar, Malgorzata E. Cavar)
Reviews: reviews at linguistlist.org (Helen Aristar-Dry, Robert Coté,
                                   Michael Czerniakowski)
Homepage: http://linguistlist.org

Editor for this issue: Yue Chen <yue at linguistlist.org>
================================================================

Date: Tue, 07 Feb 2017 13:51:19
From: Saif Mohammad [yue at linguistlist.org]
Subject: Call for Task Proposals - SemEval-2018

SemEval-2018: International Workshop on Semantic Evaluations

Call for Task Proposals:

We invite proposals for tasks to be run as part of SemEval-2018. SemEval
(Semantic Evaluation) is an ongoing series of evaluations of computational
semantic analysis systems, organized under the umbrella of SIGLEX, the Special
Interest Group on the Lexicon of the Association for Computational
Linguistics.

The SemEval evaluations explore the nature of meaning in natural languages in
practical terms, by providing an emergent mechanism to identify the problems
(e.g., how to characterize meaning and what is necessary to compute it) and to
explore the strengths of possible solutions by means of standardized
evaluation on shared datasets. SemEval evaluations initially focused on
identifying word senses computationally, but have later grown to investigate
the interrelationships among the elements in a sentence (e.g., semantic
relations, semantic parsing, semantic role labeling), relations between
sentences (e.g., coreference), and author attitudes (e.g., sentiment
analysis), among other research directions.

For SemEval-2018, we welcome any task that can test an automatic system for
semantic analysis of text, be it application-dependent or
application-independent. We especially welcome tasks for different languages,
cross-lingual tasks, tasks requiring semantic interpretation, and tasks with
both intrinsic and application-based evaluation.  See the websites of previous
editions of SemEval to get an idea about the range of tasks explored, e.g.,
for SemEval-2017: http://alt.qcri.org/semeval2017/

We strongly encourage proposals based on pilot studies that have already
generated initial data which can provide concrete examples and discuss the
challenges of preparing the full task. In the event of receiving many
proposals, preference will be given to tasks that have already run a pilot
study for the proposed task.

We encourage the following aspects in task design:

- Application-oriented tasks:
We welcome tasks that are devoted to developing novel applications of
computational semantics. We will encourage tasks that have a clearly defined
end-user application showcasing and enhancing our understanding of
computational semantics, as well as extending the current state-of-the-art.

- Umbrella tasks
In order to reduce fragmentation of similar tasks and increase community
effort towards solving the underlying research problems, we encourage task
organizers to propose larger tasks that include several related subtasks. For
example, a Semantic Similarity umbrella task might include subtasks for
different kinds of similarity and different languages. Similarly, a Sentiment
Analysis umbrella task might include subtasks for Twitter, Product Reviews,
and Service Reviews. We also welcome task proposals for umbrella tasks
focusing on different aspects of the same phenomena. For example, an Attitude
Inference task might have subtasks for detecting an author’s emotional state,
the sentiment of their writing, and the writing’s objectivity. In addition,
the program committee will actively encourage task organizers proposing
similar tasks to combine their efforts into larger umbrella tasks.

Task Selection

Task proposals will be reviewed by experts, and the reviews will serve as the
basis for acceptance decisions. In case of conflict, more innovative new tasks
will be given preference over task re-runs. Task proposals will be evaluated
on:
- Interest: Is the proposed task likely to attract a sufficient number of
participants?
- Data: Are the plans for collecting data convincing? Will the resulting data
be high quality? Will the data annotation be ready on time?
- Evaluation: Is the methodology for evaluation sound? Is the necessary
infrastructure available or can it be built in time for the shared task?
- Impact: What is the expected impact of the data in this task on future
research beyond the SemEval Workshop?

Task Organization 

Task organizers are expected to provide to task participants format checkers
and standard scorers. Moreover, in order to lower the obstacles to
participation, we encourage task organizers to provide baseline systems that
participants can use as a starting point. A baseline system typically contains
code that reads the data, creates a baseline response (e.g., random guessing,
majority class prediction, etc.), and outputs the evaluation results. Whenever
possible, baseline systems should be written in widely used programming
languages and/or should be implemented as a component for standard NLP
pipelines such as UIMA or GATE.

New Tasks vs. Task Reruns

We welcome both new tasks and task reruns. For a new task, a major concern to
be addressed in the proposal is whether it would be able to attract
participants. For task reruns, the organizers should in their proposal defend
the need for another iteration of their task, explain, for example, why there
is need for a new form of evaluation (e.g., a new metric to test new
phenomena, a new application-oriented scenario, etc.) or need to test on new
types of data (e.g., social media, domain-specific corpora), whether there is
significant expansion in scale over a previous trial run of the task, etc.

In the case of a rerun, we further discourage carrying over the same tasks
year after year and just adding new subtasks as this can lead to the
accumulation of too many subtasks. Evaluating on a different dataset with the
same task formulation typically should not be considered a separate subtask.

Tasks that have already run for three years will not be accepted for
SemEval-2018. If however the organizers estimate there is need for another
iteration of their task, they are welcome to submit a task rerun proposal for
SemEval-2019 (the calendar for submissions will be announced in Feb, 2018).
Solid justification for the re-run will be needed highlighting its novel
aspects compared to previous editions, in respect to the criteria discussed
above.

Important Dates

SemEval-2018

1st Call for task proposals February 10, 2017
Task proposals due     April 3, 2017
Task selection notification May 22, 2017
Tasks merged      June 12, 2017
Trial data ready      July 31, 2017
Training data ready     September 4, 2017
Test data ready      December 1, 2017
Evaluation start     January 10, 2018
Evaluation end      January 31, 2018
Paper submission due February 28, 2018 
Paper reviews due      March 31, 2018
Camera ready due      April 30, 2018 
SemEval workshop     Summer 2018

The SemEval-2018 Workshop will be co-located with a major NLP conference in
2018.

Submission Details

The task proposals should be a self-contained document of roughly 4-8 pages. 
Each proposal should contain the following:

- Overview
-- A summary of the task in general
-- Motivation, why this task is needed and which communities would be
interested in participating
--What the expected impact of the task will be

- Data & Resources
-- How the training/testing data will be built and/or procured
-- What source texts/corpora are going to be used? Please discuss whether
existing corpora have been re-used or not.
-- How much data is going to be produced
-- How will quality of the data be ensured and evaluated
-- An example of how a data instance would look like
-- The anticipated availability of the necessary resources to the participants
(copyright, etc.)
-- The resources required to prepare the task (computation and annotation
time, costs of annotations, etc.) and their availability

- Pilot Task
-- Details of the pilot task, if any
-- What lessons were learned and how these will impact the future task design

- Evaluation
-- The evaluation methodology to be used, including clear evaluation criteria

- For Task Reruns
-- Justification for why a new iteration of the task is needed, using the
criteria discussed above
-- What will differ from the previous instance
-- The expected impact of the re-run compared with the previous instance

- Task organizers
-- Names, affiliations, brief description of research interests and relevant
experience, contact information (email).

Proposals will be reviewed by an independent group of area experts who may not
have familiarity with recent SemEval tasks and therefore, all proposals should
be written in a self-explanatory manner and contain sufficient examples.

Please submit proposals by mail in PDF format to the SemEval email address:
semeval-organizers at googlegroups.com

In case you are not sure whether a task is suitable for SemEval, please feel
free to get in touch to discuss your idea.

Chairs:

Marianna Apidianaki, LIMSI-CNRS & University of Pennsylvania
Saif M. Mohammad, National Research Council Canada
Steven Bethard, University of Alabama at Birmingham
Marine Carpuat, University of Maryland

The SemEval discussion group:

Please join our discussion group at semeval3 at googlegroups.com in order to
receive announcements and participate in discussions.

The SemEval-2018 Website:

http://alt.qcri.org/semeval2018/

Contact: semeval-organizers at googlegroups.com

Linguistic Field(s): Computational Linguistics

----------------------------------------------------------
LINGUIST List: Vol-28-727	
----------------------------------------------------------