8.778, Sum: Natural lg dialogue systems evaluation

[linguist at linguistlist.org]

LINGUIST List:  Vol-8-778. Sat May 24 1997. ISSN: 1068-4875.

Subject: 8.778, Sum: Natural lg dialogue systems evaluation

Moderators: Anthony Rodrigues Aristar: Texas A&M U. <aristar at linguistlist.org>
            Helen Dry: Eastern Michigan U. <hdry at linguistlist.org>
            T. Daniel Seely: Eastern Michigan U. <seely at linguistlist.org>

Review Editor:     Andrew Carnie <carnie at linguistlist.org>

Associate Editors: Ljuba Veselinova <ljuba at linguistlist.org>
                   Ann Dizdar <ann at linguistlist.org>
Assistant Editor:  Sue Robinson <sue at linguistlist.org>

Software development: John H. Remmers <remmers at emunix.emich.edu>
                      Zhiping Zheng <zzheng at online.emich.edu>

Home Page:  http://linguistlist.org/


Editor for this issue: Ann Dizdar <ann at linguistlist.org>

=================================Directory=================================

1)
Date:           Wed, 21 May 1997 15:06:20 GMT+1
From:  "gas0" <GAS0 at elvira.ugr.es>
Subject:  Natural language dialogue systems evaluation

-------------------------------- Message 1 -------------------------------

Date:           Wed, 21 May 1997 15:06:20 GMT+1
From:  "gas0" <GAS0 at elvira.ugr.es>
Subject:  Natural language dialogue systems evaluation

Ramon Lopez-Cozar Delgado
Dpto. Electronica y Tecnologia de Computadores
Universidad de Granada
18071 Granada, Espana (Spain)
e-mail: gas0 at elvira.ugr.es
Fax: +34-58-243230

Dear LINGUIST Subscribers,

A few weeks ago I sent out an E-mail inquiry on
natural language dialogue systems evaluation.
I would like to thank the following people for
their kind replies. I enclose a summary of their
answers.

Nicholas Ostler        - nostler at chibcha.demon.co.uk
Jean Caelen            - Jean.Caelen at imag.fr
Frederik Fouvry        - fouvry at llsun10.essex.ac.uk
Hugo Zaragoza          - Hugo.Zaragoza at laforia.ibp.fr
Gerhard Hanrieder      - hanriede at vs.dasa.de
Gavin E Churcher       - gavin at scs.leeds.ac.uk
Philips Bouwman        - bouwman at pfa.research.philips.com
Eric Atwell            - eric at scs.leeds.ac.uk
Matthias Denecke       - denecke at cs.cmu.edu
Robert John Gaizauskas - R.Gaizauskas at dcs.shef.ac.uk
Boda Peter             - peter.boda at research.nokia.com
Doug Arnold            - doug at essex.ac.uk
Bill Manaris           - manaris at ucs.usl.edu
Maria Teresa Taboada   - taboada+ at andrew.cmu.edu
Alan Biermann          - awb at cs.duke.edu
Mai Kuha               - mkuha at indiana.edu
Hans Dybkjaer          - dybkjaer at cog.ruc.dk

Special thanks to Nicholas Ostler for sending "Lying in
Wait at the Heart of the Web", a very interesting
report on the development of linguistic technology in
Japan.

The summary follows:

******* Interesting www sites ***********************

- The DIET project:
http://www2.echo.lu/langeng/en/le3/diet/diet.html

- Dialogue systems evaluation
http://xxx.lanl.gov/cmp-lg

- A Generic Template to evaluate integrated components
in spoken dialogue
http://www.scs.leed.ac.uk/gavin/papers/acl.htm

- PARADISE: A framework for Evaluating Spoken Dialogue
Agents
http://xxx.lanl.gov/abs/cmp-lg/?9704004

- Metrics for evaluating Dialogue Strategies in a Spoken
Language System
http://xxx.lanl.gov/abs/cmp-lg/?9612003

- EAGLES
http://www.ilc.pi.cnr.it/EAGLES/home.html

- Tests suites for Natural Language Processing
http://clwww.essex.ac.uk/group/projects/tsnlp/

- "End-to-end evaluation in Janus: a speech-to-speech
translation system"
http://www.is.cs.cmu.edu/ISL.speech.publications.html

********** Interesting ftp addresses *****************

ftp://ftp.ilc.pi.cnr.it/pub/eagles/evaluation/ewg95.ps
ftp://agora.leeds.ac.uk/scs/doc/reports/1997/97_06.ps.Z

************ Interesting reports **********************

- Neal J. Feit E. et al (1992): "An Evaluation Methodology
for Natural Processing Systems", Final Technical Report,
Rome Laboratory.

*************** Interesting books *********************

- "Evaluating Natural Language Processing Systems"
Karen Sparck-Jones and Julia Galliers, Springer, 1995

- "An Architecture for Voice-Dialog"
Smith, Hipp and Biermann,
Computational Linguistics, vol. 21, no. 3

Best regards,
Ramon.

---------------------------------------------------------------------------
LINGUIST List: Vol-8-778