Appel: MT Summit VIII: workshop

[alexis nasr]

  ( Apologies if you have already seen an announcement for this  workshop )

                               MT Evaluation:
                            Who Did What To Whom

     A workshop held in conjunction with Machine Translation Summit VIII
        Saturday, September 22nd, 2001, Santiago de Compostela, Spain


                              CALL FOR PAPERS


        Introduction

     Many measures and methods of evaluating MT systems have been
     developed over the years. The ISLE project, funded jointly by the
     European Union and the National Science Foundation of the USA, is
     continuing with work started in the EU's EAGLES project on
     systematizing these measures and procedures. This workshop will
     be the fourth in a series that report on and flesh out portions
     of the systematization.

     The main thrust of the work is to build up schemes that classify
     various aspects of import for MT, including user needs, the
     suggested system characteristics and associated metrics for
     measuring each of these. The classification schemes relate to ISO
     work on software evaluation. The work is intended to be useful to
     those who are considering using machine translation, those
     interested in comparative evaluation of several MT systems and to
     MT system developers.  A fuller account of the ISLE evaluation
     work and an overview of the current classification schemes can be
     found at http://www.isi.edu/natural-language/mteval/.

     By the time of the MT Summit, the ISLE project will have
     organized or been involved in a number of workshops on MT
     evaluation, namely at LREC in Athens, AMTA in Mexico, Geneva
     Workshop, and NAACL in Pittsburgh. Participants are
     encouraged, but not required, to get involved with the Geneva
     Workshop or NAACL Workshop as these are hands-on exercises in
     MT. The results of the previous workshops, as possible, will
     be presented here and feedback based on them serves directly
     the goal of developing proposals of concrete usefulness to the
     whole community. This workshop will be seen as a preliminary
     attempt at reaching conclusions which synthesize results.
     Information about the previous workshops can be found at:

      LREC: http://www.icp.grenet.fr/ELRA/lrec2000.html
      AMTA: http://www.isi.edu/natural-language/conferences/AMTA2000.html
      NAACL 2001: http://www.cs.cmu.edu/~ref/naacl2001.html
      MT Evaluation workshop: An invitation to get your hands dirty!
        http://www.issco.unige.ch/projects/isle/mt-eval-workshop.html,
      and finally at the MT Summit VIII:  http://www.eamt.org/summitVIII.

        Papers

     We invite submission of papers along the themes discussed
     above. The questions and issues to be answered are diverse.
     All papers on evaluation and evaluation issues will be
     considered, but preference will be given to papers relating to
     the ISLE framework. The following questions suggest possible
     evaluation threads within the framework.

     * What kind of metrics are useful for what system
       characteristics?
     * What system characteristics reflect what user needs?
     * Is there a radical difference between evaluation focusing
       on research or development needs and evaluation focusing
       on end-user needs?
     * When should real-world data be used, and what is the
       impact of using it?
     * What constitutes a valid metric? How can you demonstrate
       that a metric is valid?
     * What are the advantages/disadvantages of specific
       metrics?
     * What kinds of tools automate the evaluation process? Can
       the process (or any part of it) be automated? What are
       the difficulties inherent in choosing particular metrics
       for automation?
     * What kinds of tasks are suited to which evaluation
       schemes?
     * How can we use the evaluation process to speed or improve
       the MT development process?
     * How can we evaluate MT when MT is a small part of the
       data flow? How independent is MT of the subsequent
       processing? Cleaning up the data improves performance,
       but does it improve it enough? How do we quantify that?


        Submission Guidelines

     Papers may be submitted electronically to Sandra Manzi,
     (Sandra.Manzi at issco.unige.ch) by the submission date. The
     format may be RTF, Microsoft Word, or PostScript. Any
     specialized fonts should be included in the submission
     package. Users are advised to follow the main conference
     formatting as described at:
http://www.eamt.org/summitVIII/guidelines.html.

     Submission for review:
                      papers of no more than 6000 words are expected.
     Submission for publication:
                      A template will be provided for accepted papers.


        Important Dates

     Paper submission deadline
                      April 15, 2001
     Notification of acceptance
                      May 31, 2001
     Final camera-ready due
                      July 1, 2001


        Schedule and Venue

    This one-day workshop will take place on September 22nd, 2001,
    after the regular conference.

    The main conference site contains information about
    registration and accommodation for both the conference and any
    satellite events, including this workshop.


        About Santiago

    Santiago de Compostela in North West Spain (Galicia) was the
    most visited pilgrimage shrine in the Middle Ages, and to this
    day the Road to Santiago is a popular route for walkers
    through France and northern Spain. The goal is the imposing
    11th century cathedral containing the relics of St. James the
    Great. But the well-preserved historical center of the city
    offers much more: medieval streets, excellent restaurants,
    ancient churches, monasteries, palaces, etc.

    The venue for the conference is the Hostal de los Reyes
    Católicos, founded in 1499 as a Royal hospital for pilgrims; a
    magnificent building in the Spanish Renaissance style, and now
    a hotel (the oldest and one of the most beautiful in the
    world). Those staying at the Hostal will have a
    once-in-a-lifetime experience, but there is also accommodation
    nearby in many other good hotels, all within easy walking
    distance of the center and the Hostal. Santiago de Compostela
    has its own airport with daily direct flights from Barcelona,
    Madrid, Brussels and London.


        Contact Points and Organizing Committee:

    For any further information, please contact Sandra Manzi,
    (sandra.manzi at issco.unige.ch)
    TIM/ISSCO --  University of Geneva
    40 blvd du Pont d'Arve
    CH-1211 GENEVA 4 (Switzerland)
    Tel: +41-22-705.8680
    Fax: +41-22-705.8689


        Organizing Committee

    E. Hovy, ISI, hovy at isi.edu
    B. Maegaard, CST, bente at cst.ku.dk
    S. Manzi, ISSCO, sandra.manzi at issco.unige.ch
    M. King, ISSCO, margaret.king at issco.unige.ch
    F. Reeder, MITRE, freeder at mitre.org

___________________________________________________________________
Message diffusé par la liste Langage Naturel <LN at cines.fr>
Informations, abonnement : http://www.biomath.jussieu.fr/LN/LN-F/
English version          : http://www.biomath.jussieu.fr/LN/LN/
Archives                 : http://web-lli.univ-paris13.fr/ln/