13.2941, Confs: Reuse of Evaluation Resources

Wed Nov 13 17:51:49 UTC 2002

LINGUIST List:  Vol-13-2941. Wed Nov 13 2002. ISSN: 1068-4875.

Subject: 13.2941, Confs: Reuse of Evaluation Resources

Moderators: Anthony Aristar, Wayne State U.<aristar at linguistlist.org>
            Helen Dry, Eastern Michigan U. <hdry at linguistlist.org>

Reviews (reviews at linguistlist.org):
	Simin Karimi, U. of Arizona
	Terence Langendoen, U. of Arizona

Consulting Editor:
        Andrew Carnie, U. of Arizona <carnie at linguistlist.org>

Editors (linguist at linguistlist.org):
	Karen Milligan, WSU 		Naomi Ogasawara, Arizona U.
	James Yuells, EMU		Marie Klopfenstein, WSU
	Michael Appleby, EMU		Heather Taylor, EMU
	Ljuba Veselinova, Stockholm U.	Richard John Harvey, EMU
	Dina Kapetangianni, EMU		Renee Galvis, WSU
	Karolina Owczarzak, EMU		Anita Huang, EMU
	Tomoko Okuno, EMU		Steve Moran, EMU
	Lakshmi Narayanan, EMU		Sarah Murray, WSU
	Marisa Ferrara, EMU

Software: Gayathri Sriram, E. Michigan U. <gayatri at linguistlist.org>
          Zhenwei Chen, E. Michigan U. <chen at linguistlist.org>
	  Prashant Nagaraja, E. Michigan U. <prashant at linguistlist.org>

Home Page:  http://linguistlist.org/

The LINGUIST List is funded by Eastern Michigan University, Wayne
State University, and donations from subscribers and publishers.

Editor for this issue: Karolina Owczarzak <karolina at linguistlist.org>
 ==========================================================================
Please keep conferences announcement as short as you can; LINGUIST
will not post conference announcements which in our opinion are
excessively long.

=================================Directory=================================

1)
Date:  Wed, 13 Nov 2002 05:05:51 -0800 (PST)
From:  ekaterini pastra <pastrak at yahoo.com>
Subject:  Reuse of Evaluation Resources, Budapest Hungary

-------------------------------- Message 1 -------------------------------

Date:  Wed, 13 Nov 2002 05:05:51 -0800 (PST)
From:  ekaterini pastra <pastrak at yahoo.com>
Subject:  Reuse of Evaluation Resources, Budapest Hungary

EACL 2003 Workshop on:

Evaluation Initiatives in Natural Language Processing:
are evaluation methods, metrics and resources reusable?

13 April 2003, Budapest, Hungary
11th conference of the European Chapter of the
Association for Computational Linguistics (April 12-17, 2003)
http://www.dcs.shef.ac.uk/~katerina/EACL03-eval

Introduction:

Systems that accomplish different Natural Language Processing (NLP)
tasks have different characteristics and therefore, it would seem,
different requirements for evaluation. However, are there common
features in evaluation methods used in various language technologies?
Could the evaluation methods established for one type of systems be
ported/adapted to another NLP research area? Could automatic
evaluation metrics be ported? For instance, could Papineni's MT
evaluation metric be used for the evaluation of generated summaries?
Could the extrinsic evaluation method used within SUMMAC be applied to
the evaluation of Natural Language Generation systems? What are the
reusability obstacles encountered and how could they be overcome? What
are the evaluation needs of system types such as dialogue systems,
which have been less strenuously evaluated till now, and how could
they benefit from current practices in evaluating Language Engineering
technologies? What are the evaluation challenges that emerge from
systems that integrate a number of different language processing
functions (e.g. multimodal dialogue systems such as Smartkom)? Could
resources (e.g. corpora) used for a specific NLP task, be reused for
the evaluation of the output of an NLP system and if so, what
adaptations would this require? John White suggested some years ago a
hierarchy of difficulty, or compositionality, of NLP tasks; if
correct, does this have implications for evaluation?

End-to-end evaluation of systems in a specific NLP area of research
has been attempted within both European initiatives (e.g. EAGLES/ISLE,
ELSE, TEMAA etc.) and U.S. evaluation regimes with international
participation (e.g. MUC, TREC, SUMMAC). It has been reported that
evaluation techniques in the different Language Engineering areas grow
more similar (Hovy et al. 1999), a fact that emphasizes the need for
co-ordinated and reusable evaluation techniques and measures. The time
has come to bring together all the above attempts to address the
evaluation of NLP systems as a whole and explore ways for reusing
established evaluation methods, metrics and resources, thus,
contributing to a more co-ordinated approach to the evaluation of
language technology.

Target audience:

The aim of this workshop is to bring together leading researchers from
various NLP areas (such as Machine Translation, Information
Extraction, Information Retrieval, Automatic Summarization,
Question-Answering, Dialogue Systems and Natural Language Generation)
in order to explore ways in making the most of currently available
evaluation methods, metrics and resources.

Workshop format:

The workshop will be opened with an invited speaker who will introduce
the topic and present the research questions and challenges that need
to be addressed. Oral presentations divided into thematic sessions
will follow; at the end of each session a panel discussion will take
place. The panels will consist of members of the program
committee. The workshop will close with an overview talk.

Topics of interest:

We welcome submissions of both discussion-papers and papers presenting
applied experiments relevant with -but not limited to- the following
topics:

- cross-fertilization of evaluation methods and metrics
- reuse of resources for evaluation (corpora, evaluation tools etc.)
- feasibility experiments for the reuse of established evaluation
methods/metrics/resources in different NLP system types
- reusability obstacles and the notion of compositionality of NLP tasks
- evaluation needs and challenges for less strenuously evaluated
system types (e.g. multimodal dialogue systems ), possible benefits
from established evaluation practices
- evaluation standards and reusability
- reuse within big evaluation initiatives
- application of e.g. Machine Translation methods to Information
Retrieval: implications for evaluation

Submission format:

Submissions must be electronic only, and should consist of full papers
of max. 8 pages (inclusive of references, tables,figures and
equations). Authors are strongly encouraged to use the style-files
suggested for the EACL main conference submissions at:
http://ufal.ms.mff.cuni.cz/~hajic/eacl03/submission.html Please, mail
your submissions to Katerina Pastra: e.pastra at dcs.shef.ac.uk

Important dates:
* Deadline for workshop paper submissions:
TUESDAY, 7 January 2003 (NOTE: strict deadline)
* Notification of workshop paper acceptance:
TUESDAY, 28 January 2003
* Deadline for camera-ready workshop papers:
THURSDAY, 13 February 2003
* Workshop Date:
SUNDAY, 13 April 2003

Program Committee:
Rob Gaizauskas (University of Sheffield, UK)
Donna Harman (NIST, US)
Lynette Hirschman (MITRE, US)
Maghi King (ISSCO, Switzerland)
Steven Krauwer (Utrecht University, Netherlands)
Inderjeet Mani (MITRE, US)
Joseph Mariani (LIMSI, France)
Patrick Paroubek (LIMSI, France)
Katerina Pastra (University of Sheffield, UK)
Martin Rajman (EPFL - Switzerland)
Karen Sparck-Jones (University of Cambridge, UK)
Horacio Saggion (University of Sheffield, UK)
Simone Teufel (University of Cambridge, UK)
Yorick Wilks (University of Sheffield, UK)

Registration details:
Information on registration fees and procedures will be published at
the main EACL 2003 conference pages at:
http://www.conferences.hu/EACL03/

For detailed and up-to-date information on the workshop please visit
the workshop's website:
http://www.dcs.shef.ac.uk/~katerina/EACL03-eval

For any queries, please don't hesitate to contact me
e.pastra at dcs.shef.ac.uk

---------------------------------------------------------------------------
LINGUIST List: Vol-13-2941