Appel: First Call for Participation, SemEval - Task 4, TimeLine: Cross-Document Event Ordering (pilot task)

Tue Jul 29 20:39:13 UTC 2014

Date: Mon, 28 Jul 2014 13:32:44 +0000
From: "Erp, M.G.J. van" <marieke.van.erp at vu.nl>
Message-ID: <A521AB7F-BDA0-4396-91B9-671057B54291 at vu.nl>
X-url: http://alt.qcri.org/semeval2015/task4
X-url: https://groups.google.com/forum/#!forum/semeval-task4-timeline

SemEval-2015 Task 4: TimeLine: Cross-Document Event Ordering (pilot
task)
First Call for Participation

Website: http://alt.qcri.org/semeval2015/task4
Google Group: https://groups.google.com/forum/#!forum/semeval-task4-timeline
Evaluation period: November 15 - 30, 2014
Paper submission: January 2015

*Introduction*
In any domain, professionals need to have access to knowledge in order
to take well-informed decisions. An insightful way of presenting
information in an easily updatable and complete manner is to present it
on a timeline that is continuously updated with new information. The aim
of the task is to build timelines from written news in English. More
specifically, the goal is to order on a timeline all the events in which
a target entity is involved. We focus mainly on cross-document event
coreference resolution and cross-document temporal relation extraction.

Temporal relation extraction has been the topic of the three past
TempEval tasks as part of SemEval:
- TempEval-1 (2007): Temporal Relation Identification
- TempEval-2 (2010): Evaluating Events, Time Expressions, and Temporal
  Relations
- TempEval-3 (2013): Temporal Annotation

In addition, temporal relation extraction has been the focus of the 6th
i2b2 NLP Challenge for clinical records but the cross-document aspect,
has not been often explored. At RANLP 2009 there was a cross-document
temporal relation extraction task, in which the goal was to link
pre-defined events involving the same centroid entities (i.e. entities
frequently participating in events) on a timeline. Nominal coreference
resolution has been the topic of SemEval 2010 Task on Coreference
Resolution in Multiple Languages.

Partially motivated by the work in the NewsReader project
(http://www.newsreader-project.eu), TimeLine goes beyond the these tasks
by addressing coreference resolution for events and temporal relation
identification across documents.

*Task Description*
Given a set of documents and a target entity, the task is to build an
event TimeLine related to that entity, i.e. to detect, anchor in time
and order the events involving the target entity.

As input data, we provide a set of documents and a set of target
entities (people, organization, product or financial entity); only
entities of interest will be selected as target entities, i.e. entities
involved in many events across different documents and for which it is
relevant to build a timeline.

There are two tracks in this task based on the data used as input. For
Track A only raw text is provided to the participants, while for Track B
gold-standard event mentions are also given.

For both tracks the expected output is one TimeLine for each target
entity. Each TimeLine consists of an ordered list of events in which
each event is associated to a time anchor.

For both tracks a sub-track in which the events are not associated to a
time anchor is proposed.

Participants can choose to participate in any track and subtrack.
Participants can submit up to two runs for each track/subtrack.

*Data*
The trial data consists of a set of 30 documents collected from Wikinews
(http://en.wikinews.org) about Apple Inc. A set of target entities
(input) and the corresponding ordered list of events (the output
timeline) is provided with the set of documents.

The trial data have been annotated with the extents of event mentions
and are available
fromhttp://alt.qcri.org/semeval2015/task4/index.php?id=data-and-tools. The
evaluation tool can also be found there.

The evaluation data will consist of 3 sets of documents annotated with
event mentions and a set of target entities. Each set will contain
around 30 documents from Wikinews, totalling around 30,000 tokens. For
each set of documents, one file is provided containing the list of
target entities.

No training corpus will be provided for this task.

*Evaluation Methodology*
Participants will submit the TimeLines produced by their system for all
target entities. Systems will be ranked based on their temporal
awareness.

*More information*
See http://alt.qcri.org/semeval2015/task4/ for further information about
the task, data formats, examples, data downloads, tools and registration
information.

*Important dates*
- Evaluation period starts: November 15, 2014
- Evaluation period ends: November 30, 2014
- Paper submission due: January 30, 2015
- Paper notification: Early March, 2015
- Camera-ready due: March 30, 2015
- SemEval workshop: Summer 2015

*Organisers*
- Anne-Lyse Minard, Fondazione Bruno Kessler, Italy
- Eneko Agirre, The University of the Basque Country, Spain
- Itziar Aldabe, The University of the Basque Country, Spain
- Marieke van Erp, VU University Amsterdam, Netherlands
- Bernardo Magnini, Fondazione Bruno Kessler, Italy
- German Rigau, The University of the Basque Country, Spain
- Manuela Speranza, Fondazione Bruno Kessler, Italy
- Rubén Urizar, The University of the Basque Country, Spain

-------------------------------------------------------------------------
Message diffuse par la liste Langage Naturel <LN at cines.fr>
Informations, abonnement : http://www.atala.org/article.php3?id_article=48
English version       : 
Archives                 : http://listserv.linguistlist.org/archives/ln.html
                                http://liste.cines.fr/info/ln

La liste LN est parrainee par l'ATALA (Association pour le Traitement
Automatique des Langues)
Information et adhesion  : http://www.atala.org/

ATALA décline toute responsabilité concernant le contenu des
messages diffusés sur la liste LN
-------------------------------------------------------------------------