Conf: LREC 2002 workshop : Machine Translation Evaluation

Fri Mar 22 10:03:57 UTC 2002

Dear members of the LN list,

Please find below the call for participation for the
MT Evaluation workshop organized on May 27th, 2002 at
the LREC 2002 Conference, Canary Islands.

Please accept our apologies if you receive multiple
copies of this announcement. Thank you,
Andrei Popescu-Belis

---------------------------------------------------------------

              Machine Translation Evaluation: Human
                Evaluators Meet Automated Metrics

                          27 May 2002

                 A hands-on evaluation workshop at
                 LREC 2002 (27 May - 2 June 2002)
                    Las Palmas, Canary Islands

               Second call for interest and participation

---------------------------------------------------------------

                        Important dates

LREC 2002 advance registration deadline: March 29th, 2002

Please check the Conference's webpage at:
   http://www.lrec-conf.org/lrec2002/

Distribution of pre-workshop material: April 2002

Workshop: May 27th, 2002
          09:00 to 13:00 morning session
          14:30 to 18:30 afternoon session
---------------------------------------------------------------

                           Preliminary Schedule

Morning
   introduction and welcome
   background on workshop theme
   integration of evaluation exercises (start)
Afternoon
   integration of evaluation exercise (continue)
   reports
   cross-evaluation analysis
   final wrap-up

-------------------------------------------------------------

Background

The Evaluation Working Group of the ISLE project has organised a
series of workshops on MT evaluation. Each of these workshops has
contained a practical component, where participants have been asked to
carry out exercises involving MT evaluation. These workshops
proved to be very illuminating, and have stimulated ongoing work
in the area, much of it was reported in the latest workshop in the
series, held at the MT Summit meeting in September 2001.

Results from previous workshops can be consulted at
   http://www.issco.unige.ch/projects/isle/ewg.html,
and the proceedings from the MT Summit in Santiago de Compostela can
be requested from the organisers.

The workshop at LREC 2002 will continue the series, and will consist
primarily of hands-on exercises defined to investigate empirically a
small number of metrics proposed for evaluation of MT systems and the
potential relationships between them.

In an effort to develop a more systematic MT evaluation
methodology, recent work in the EAGLES and ISLE projects, funded by
the EU and NSF, has created a framework of characteristics in terms
of which MT evaluations and systems, past and future, can be described
and classified.  The resulting taxonomy can be consulted at:
   http://issco-www.unige.ch/projects/isle/taxonomy2/.

Previous workshops have led to critical analysis of measures drawn
from the literature, and to the creation of new measures.  Of the
latter, several are aimed at eventual automation of the evaluation
task and/or at finding relatively simple and inexpensive measures
which correlate well with more complex measures that are hard
to automate or expensive to implement.

Given this background, the time has come to concentrate on
systematizing the actual evaluation measures themselves.  For any
particular measure, one would like to know how accurate it is, how
expensive and/or difficult to apply, how independent of other measures,
etc.  Very little of this type of information is available to date.

This workshop will focus on these issues.  The organizers will provide
the participants in advance with the materials required to:
  - perform a small evaluation, using one or two measures
  - perform a cross-measure analysis of the resulting scores
  - create a general characterization of the measure's performance.

The participants will then apply these measures to the data made
available, and bring their results to the workshop in order to
integrate them with other participants' results.

The overall intention of the workshop is to discover, empirically,
what kinds of characteristics are easily determinable, and how
accurate they actually are.  Only through a process of assessing the
evaluations can we eventually arrive at a small but accurate set of
measures that adequately cover the set of phenomena MT system
evaluators, system developers, and potential MT users care about.

It is our hope that participants will feel inspired to continue
this process, so that the combined results can be assembled later,
integrated into the framework, and become a valuable resource to
anyone interested in MT evaluation.

-------------------------------------------------------------
Organizing Committee

Marianne Dabbadie
  EVALING, Paris, France
Tony Hartley
  Centre for Translation Studies, University of Leeds, UK
Eduard Hovy
  USC Information Sciences Institute, Marina del Rey, USA
Margaret King
  ISSCO/TIM/ETI, University of Geneva, Switzerland
Bente Maegaard
  Center for Sprogteknologi, Copenhagen, Denmark
Sandra Manzi
  ISSCO/TIM/ETI, University of Geneva, Switzerland
Keith J. Miller
  The MITRE Corporation, USA
Widad Mustafa El Hadi
  Université Lille III - Charles de Gaulle, France
Andrei Popescu-Belis
  ISSCO/TIM/ETI, University of Geneva, Switzerland
Florence Reeder
  The MITRE Corporation, USA
Michelle Vanni
  U.S. Department of Defense, USA

-------------------------------------------------------------

Intention to participate:

Participants wishing to receive preparatory data should send the
the following information to contact person below:
- name, address, email contact;
- experience in MT evaluation;
- languages known and level of comprehension (elementary, fair,
  good, near-native, native);

Contact:
   Andrei Popescu-Belis
   Email: andrei.popescu-belis at issco.unige.ch
   Fax:   (41 22) 705 86 89
   Regular mail:
   ISSCO/TIM/ETI, University of Geneva
   40, bd du Pont d'Arve
   CH-1211 Geneva 4 - SWITZERLAND

Cost of the Workshop:
   LREC 2002 participants: 90 EURO
   Other participants: 140 EURO

Registration forms are available on the LREC 2002 conference site:
   http://www.lrec-conf.org/lrec2002/

Main conference and workshop site:
   Palacio de Congresos, Las Palmas, Canary Islands

---------------------------------------------------------------
-------------------------------------------------------------------------
Message diffusé par la liste Langage Naturel <LN at cines.fr>
Informations, abonnement : http://www.biomath.jussieu.fr/LN/LN-F/
English version          : http://www.biomath.jussieu.fr/LN/LN/
Archives                 : http://listserv.linguistlist.org/archives/ln.html

La liste LN est parrainée par l'ATALA (Association pour le Traitement
Automatique des Langues)
Information et adhésion  : http://www.atala.org/
-------------------------------------------------------------------------