[Corpora-List] Deadline Extension and Final Call: TempEval-3 / Temporal Information Extraction (SemEval 2012, Task 1)

Wed Feb 20 13:50:02 UTC 2013

(apologies for cross-posting)

Deadline extended: Register by 22 February

New subtask: linking-only

FINAL CALL FOR PARTICIPATION

                               TempEval-2013
                                 as part of

                               SemEval-2013
                 International Workshop on Semantic Evaluations
                            an ACL-SIGLEX event
                         Second Call for Participation

                  http://www.cs.york.ac.uk/semeval-2013

The aim of TempEval is to advance research on temporal information
processing, which could eventually help NLP applications like question
answering, textual entailment, summarization, etc. TempEval-3 follows
on from previous TempEval events, incorporating: a three-part task
structure covering event, temporal expression and temporal relation
extraction; the use of the complete set of TimeML temporal relations,
that was simplified in previous editions; a 10-times larger dataset;
and single overall performance scores, which allow the ranking of the
participating systems in each task and also in general.

Introduction:
-------------

Temporal annotation is a time-consuming task for humans, which has
limited the size of annotated data in previous TempEvals. Current
systems, however, are performing close to the inter-annotator
reliability, which suggests that larger corpora could be built
starting with automatically annotated data. One of the main goals of
this TempEval edition is to explore whether there is value in adding a
large automatically created silver standard to a hand-crafted gold
standard. It might be that for some tasks an auto-annotated larger
corpus might be more useful than a hand annotated small corpus.

TempEval-3, a temporal evaluation task, is a follow-up to TempEval-1
and 2. TempEval-3 diﬀers from its ancestors in the following respects:
   (i) size of the corpus: the dataset used comprises about 500K
tokens of silver standard data and about 100K tokens of gold standard
data for training, compared to the corpus of roughly 50K tokens corpus
used in TempEval 1 and 2;
   (ii) temporal relation task: the temporal relation classiﬁcation
tasks are to be performed from raw text, i.e. participants need to
extract events and temporal expressions ﬁrst, determine which ones to
link and then obtain the relation types;
   (iii) tasks not independent: participants must annotate temporal
expressions and events in order to do the relation task;
   (iv) temporal relation types: the full set of temporal interval
relations in TimeML is used, rather than the reduced set used in
earlier TempEvals;
   (v) annotation: most of the corpus was automatically annotated by
the stateof-the-art systems from TempEval-2, a portion of the corpus,
including the test dataset, that is human reviewed;
   (vi) evaluation: we will report a temporal awareness score for
evaluating temporal relations, to help to rank systems with a single
score.

TempEval 3 Tasks:
----------------
The tasks proposed for TempEval-3 are related to each one of the main
TimeML tags. These are:

* Task A: Temporal expression extraction and normalization
Determine the extent of the time expressions in a text as deﬁned by
the TimeML TIMEX3 tag. In addition, determine the value of the
features TYPE and VAL. The possible values of TYPE are time, date,
duration, and set; the value of VAL is a normalized value as deﬁned by
the TIMEX3 standard. The main attribute to annotate is VAL.

* Task B: Event extraction
As in TempEval-2, participants will determine the extent of the events
in a text as deﬁned by the TimeML EVENT tag. In addition, systems may
determine the value of the features CLASS, TENSE, ASPECT, POLARITY,
MODALITY and also identify if the event is a main event or not. The
main attribute to annotate is CLASS.

* Task C: Annotating temporal relations
Identify the pairs of temporal entities (events or temporal
expressions) that have a temporal link and classify the temporal
relation between them as a TLINK. For this task, we now require
that the participating systems determine which entities need to be linked.
The relation labels will be same as in TimeML, i.e.: before, after,
includes, is-included, during, simultaneous, immediately after,
immediately before, identity, begins, ends, begun-by and ended-by.

Task selection
Participants may choose to do task A, B, or C. Choosing task C
(relation annotation) entails doing tasks A and B (interval
annotation). However, a participant may perform only task C by
applying existing tools to carry out tasks A and B.

Further, the linking-only task is now available; gold-standard timex
and event annotations will be release and a separate one-week
evaluation window is available for teams only wishing to do the
temporal relation labelling task.

Dataset Creation
----------------
In TempEval-3, we release new data, as well as signiﬁcantly reviewing
and modifying existing corpora.

A large portion of the TempEval-3 data is automatically generated,
using a temporal merging system. We include over half a million
temporally-annotated tokens from English Gigaword, as well as 40,000
tokens of new gold-standard data.

Task Organizers:
----------------

James Allen, University of Rochester
Leon Derczynski, University of Sheffield
Hector Llorens, University of Alicante
James Pustejovsky, Brandeis University
Naushad UzZaman, University of Rochester [Primary Contact]
Marc Verhagen, Brandeis University

Important Dates:
----------------

*February 22, 2013: Team registration deadline*

March 1, 2013 (mid-day EST): Start of evaluation period for Task A; for
Task B; for Combined Task ABC, from raw text
March 15, 2013 (23.59 UTC-10 Hawaii): End of evaluation period for Task A;
for Task B; and for Combined Task ABC, from raw text
March 16, 2013 (mid-day EST): Start of evaluation period for Linking-only
Task C using reference timex and event annotations
March 23, 2013 (23.59 UTC-10 Hawaii): End of evaluation period for
Linking-only Task C
April 14, 2013 (23.59 UTC-10 Hawaii): System description & analysis paper
submission deadline
April 24, 2013: Description paper notifications / reviews
May 4, 2013: Camera-ready deadline
June 13-14: *SEM conference co-hosted with NAACL
June 14-15: SemEval workshop (tentative)

More infomation:
----------------

The TempEval-3 website, for signup and details, is:

                http://www.cs.york.ac.uk/semeval-2013

For details, check the task description paper here:
http://arxiv.org/pdf/1206.5333v1.pdf
Naushad UzZaman, Hector Llorens, James F. Allen, Leon Derczynski, Marc
Verhagen, James Pustejovsky. 2012. TempEval-3: Evaluating Events, Time
Expressions, and Temporal Relations. arXiv:1206.5333v1.

--
Leon R A Derczynski
NLP Research Group

Department of Computer Science
University of Sheffield
Regent Court, 211 Portobello
Sheffield S1 4DP, UK

+45 5157 4948
http://www.dcs.shef.ac.uk/~leon/

-- 
Leon R A Derczynski
Research Associate, NLP Group

Department of Computer Science
University of Sheffield
Regent Court, 211 Portobello
Sheffield S1 4DP, UK

+45 5157 4948
http://www.dcs.shef.ac.uk/~leon/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listserv.linguistlist.org/pipermail/corpora/attachments/20130220/75a2f977/attachment.htm>
-------------- next part --------------
************************************************ 
This SIGIR-IRList message and the SIG-IRList Digest (a moderated IR newsletter), are brought to you by SIGIR, distributed from the University of Sheffield and edited by Claudia Hauff (irlist-editor at acm.org). 
o	To submit an article, e-mail IRList at lists.shef.ac.uk 
o	To subscribe, send mail to sympa at lists.shef.ac.uk , with the subject: SUBSCRIBE irlist firstname lastname
o	To unsubscribe, send mail to sympa at lists.shef.ac.uk, with the subject: UNSUBSCRIBE irlist YourEmailAddressHere
[The email address is required only if you want to unsubscribe with an address other than the address with which you send the message]

o	For more info, visit: http://www.sigir.org/sigirlist/
These files are not to be sold or used for commercial purposes. 
THE OPINIONS EXPRESSED WITHIN THIS DOCUMENT DO NOT REPRESENT THOSE OF THE EDITOR, THE DELFT UNIVERSITY OF TECHNOLOGY OR THE UNIVERSITY OF SHEFFIELD. 
AUTHORS ASSUME FULL RESPONSIBILITY FOR THEIR MATERIAL.

-------------- next part --------------
_______________________________________________
UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
Corpora mailing list
Corpora at uib.no
http://mailman.uib.no/listinfo/corpora