Corpora: call for papers

[Reeder,Florence M.]

Please post the following call for papers for a workshop in
Machine Translation Evaluation to be held this fall:

Machine Translation Evaluation Workshop in conjunction with AMTA-2000
October 10, 2000
Mision del Sol, Cuernavaca, Mexico

MT EVALUATION WORKSHOP:  Hands-On Evaluation

Motivation:  Arguments about the evaluation of machine translation (MT)
are even more plentiful than systems to accomplish MT.  In an effort to
drive the evaluation process to the next level and to unify past work,
this workshop is going to focus on a challenge and a framework in which
the challenge could fit. The challenge is Hands-On Evaluation.  The
framework is being developed by the ISLE MT Evaluation effort.  Both
will be discussed throughout the course of the workshop.

Focus of the first part of the workshop will be on real-world
evaluation, encouraging both developers and users.   In an effort to
facilitate a common ground for discussions, if they desire, participants
may be given a) sample online data to be translated; b) a minimal task
to accomplish with this data; c) currently existing tools for processing
this data.  With these three items, participants are expected to address
issues in the evaluation of machine translation.  A  domain of
particular interest is evaluation for Arabic data; Arabic tools, and
filtering and text mining applications although participants are
required only to evaluate using real-world data and actual systems.
This common framework will give insights into the evaluation process and
useful metrics for driving the development process.   Additionally, we
hope that the common challenge will motivate interesting discussion.  As
part of this, we are expecting to set up a web page to host previous
work in evaluation.  The URL will be released when the page is
prepared.
The second part of the workshop will concentrate on the ISLE MT
Evaluation effort, funded by NSF and the EU, to create a general
framework of characteristics in terms of which MT evaluations, past and
future, can be described and classified.  The framework, whose
antecedents are the JEIDA and EAGLES reports, consists of two taxonomies
of increasingly specific features, with associated measures and pointers
to systems.  The discussion will review the current state of the
classification effort, critique the framework in light of the hands-on
evaluation performed earlier in the workshop, and suggest additional
systems and measures to be considered.

Questions and Issues:  The questions and issues to be answered are
diverse.  The constants are the situation:  the available systems; the
available data; the sample tasks.  This will, hopefully, eliminate some
of the variables of evaluation.  All papers on evaluation and evaluation
issues will be considered, but preference will be given to papers
following the framework.  The following questions suggest possible
evaluation threads within the framework.
        - What kind of metrics are useful for users versus system
        developers?
        - What kinds of tools automate the evaluation process?
        - What kinds of tasks are suited to which evaluation schemes?
        - How can we use the evaluation process to speed or improve the
        MT development process?
        - What kind of impacts does real-world data imply?
        - How can we evaluate MT when MT is a small part of the data
        flow?
        - How independent is MT of the subsequent processing?-that is,
        cleaning up the data improves performance, but does it improve
        it enough?  How do we quantify that?

While we encourage papers on Arabic MT evaluation, we will consider
papers on related issues such as real-world data evaluation or that
which is related to the ISLE work.

Submission Guidelines:
        - Intention to participate:  For those participants wishing to
        receive either data or systems, it may be necessary to generate
        a non-disclosure agreement.  Alternatively, participants are
        encouraged to bring their own tools / data in light of the
        constraints listed above.
        - Submission for review:  papers of no more than 6000 words are
        expected.
        - Submission for publication:   A template will be provided for
        accepted papers.

Important Dates:
        Intention to participate: 28th July 2000
        Release of data / software: 9th August 2000
        Draft submission: 1st September 2000
        Notification of acceptance: 15th September 2000
        Final Papers Due: 29th September 2000
        Workshop: 10th October 2000

Contact Points & Organizers:
Florence Reeder (MITRE Corporation)  freeder at mitre.org
Eduard Hovy (ISI/USC) hovy at isi.edu

Main conference site:
http://www.isi.edu/natural-language/conferences/amta2000

Venue site:
http://www.misiondelsol.com.mx

Reviewing Committee:
Michelle Vanni (Georgetown University / Department of Defense)
Keith Miller (MITRE Corporation)
Jack Benoit (MITRE Corporation)
Maghi King (ISSCO, University of Geneva)
Carol Van Ess-Dykema (Department of Defense)
John White (Litton/PRC)