[Corpora-List] Call for Participation: Temporal Information Extraction (TempEval-3) - SemEval task 1

Tue Dec 11 09:04:46 UTC 2012

(apologies for cross-posting)

SECOND CALL FOR PARTICIPATION

                               TempEval-2013
                                 as part of

                               SemEval-2013
                 International Workshop on Semantic Evaluations
                            an ACL-SIGLEX event
                         Second Call for Participation

                  http://www.cs.york.ac.uk/semeval-2013

The aim of TempEval is to advance research on temporal information
processing, which could eventually help NLP applications like question
answering, textual entailment, summarization, etc. TempEval-3 follows
on from previous TempEval events, incorporating: a three-part task
structure covering event, temporal expression and temporal relation
extraction; the use of the complete set of TimeML temporal relations,
that was simplified in previous editions; a 10-times larger dataset;
and single overall performance scores, which allow the ranking of the
participating systems in each task and also in general.

Introduction:
-------------

Temporal annotation is a time-consuming task for humans, which has
limited the size of annotated data in previous TempEvals. Current
systems, however, are performing close to the inter-annotator
reliability, which suggests that larger corpora could be built
starting with automatically annotated data. One of the main goals of
this TempEval edition is to explore whether there is value in adding a
large automatically created silver standard to a hand-crafted gold
standard. It might be that for some tasks an auto-annotated larger
corpus might be more useful than a hand annotated small corpus.

TempEval-3, a temporal evaluation task, is a follow-up to TempEval-1
and 2. TempEval-3 diﬀers from its ancestors in the following respects:
   (i) size of the corpus: the dataset used comprises about 500K
tokens of silver standard data and about 100K tokens of gold standard
data for training, compared to the corpus of roughly 50K tokens corpus
used in TempEval 1 and 2;
   (ii) temporal relation task: the temporal relation classiﬁcation
tasks are to be performed from raw text, i.e. participants need to
extract events and temporal expressions ﬁrst, determine which ones to
link and then obtain the relation types;
   (iii) tasks not independent: participants must annotate temporal
expressions and events in order to do the relation task;
   (iv) temporal relation types: the full set of temporal interval
relations in TimeML is used, rather than the reduced set used in
earlier TempEvals;
   (v) annotation: most of the corpus was automatically annotated by
the stateof-the-art systems from TempEval-2, a portion of the corpus,
including the test dataset, that is human reviewed;
   (vi) evaluation: we will report a temporal awareness score for
evaluating temporal relations, to help to rank systems with a single
score.

TempEval 3 Tasks:
----------------
The tasks proposed for TempEval-3 are related to each one of the main
TimeML tags. These are:

* Task A: Temporal expression extraction and normalization
Determine the extent of the time expressions in a text as deﬁned by
the TimeML TIMEX3 tag. In addition, determine the value of the
features TYPE and VAL. The possible values of TYPE are time, date,
duration, and set; the value of VAL is a normalized value as deﬁned by
the TIMEX3 standard. The main attribute to annotate is VAL.

* Task B: Event extraction
As in TempEval-2, participants will determine the extent of the events
in a text as deﬁned by the TimeML EVENT tag. In addition, systems may
determine the value of the features CLASS, TENSE, ASPECT, POLARITY,
MODALITY and also identify if the event is a main event or not. The
main attribute to annotate is CLASS.

* Task C: Annotating temporal relations
Identify the pairs of temporal entities (events or temporal
expressions) that have a temporal link and classify the temporal
relation between them as a TLINK. Possible pairs of entities that can
have a temporal link are: (i) event and temporal expressions in the
same sentence, (ii) event and document creation time, (iii) main
events of consecutive sentences and (iv) pairs of events in the same
sentence. For this task, we now require that the participating systems
determine which entities need to be linked.
The relation labels will be same as in TimeML, i.e.: before, after,
includes, is-included, during, simultaneous, immediately after,
immediately before, identity, begins, ends, begun-by and ended-by.

Task selection
Participants may choose to do task A, B, or C. Choosing task C
(relation annotation) entails doing tasks A and B (interval
annotation). However, a participant may perform only task C by
applying existing tools to carry out tasks A and B.

Dataset Creation
----------------
In TempEval-3, we release new data, as well as signiﬁcantly reviewing
and modifying existing corpora.

A large portion of the TempEval-3 data is automatically generated,
using a temporal merging system. We include over half a million
temporally-annotated tokens from English Gigaword, as well as 40,000
tokens of new gold-standard data.

Task Organizers:
----------------

James Allen, University of Rochester
Leon Derczynski, University of Sheffield
Hector Llorens, University of Alicante
James Pustejovsky, Brandeis University
Naushad UzZaman, University of Rochester [Primary Contact]
Marc Verhagen, Brandeis University

Important Dates:
----------------

September 12, 2012 First Call for participation
November 1, 2012 onwards Full Training Data available for participants
February 15, 2013 Test set ready
February 15, 2013 Registration Deadline [for Task Participants]
March 1, 2013 onwards Start of evaluation period [Task Dependent]
March 15, 2013 End of evaluation period
April 9, 2013 Paper submission deadline [TBC]
April 23, 2013 Reviews Due [TBC]
May 4, 2013 Camera ready Due [TBC]

Summer 2013 Workshop co-located with NAACL, as part of SemEval-2013

More infomation:
----------------

The TempEval-3 website, for signup and details, is:

                http://www.cs.york.ac.uk/semeval-2013

For details, check the task description paper here:
http://arxiv.org/pdf/1206.5333v1.pdf
Naushad UzZaman, Hector Llorens, James F. Allen, Leon Derczynski, Marc
Verhagen, James Pustejovsky. 2012. TempEval-3: Evaluating Events, Time
Expressions, and Temporal Relations. arXiv:1206.5333v1.

--
Leon R A Derczynski
NLP Research Group

Department of Computer Science
University of Sheffield
Regent Court, 211 Portobello
Sheffield S1 4DP, UK

+45 5157 4948
http://www.dcs.shef.ac.uk/~leon/

_______________________________________________
UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
Corpora mailing list
Corpora at uib.no
http://mailman.uib.no/listinfo/corpora