Appel: LREC-2004 workshop : LESSONS LEARNED FROM EVALUATION ...
alexis.nasr at LINGUIST.JUSSIEU.FR
alexis.nasr at LINGUIST.JUSSIEU.FR
Tue Feb 3 08:47:00 UTC 2004
Call for papers
Workshop on
LESSONS LEARNED FROM EVALUATION:
TOWARDS TRANSPARENCY AND INTEGRATION
IN CROSS-LINGUAL INFORMATION RETRIEVAL
(LECLIQ)
to be held in
Lisbon, Portugal, on May 29, 2004
in conjunction with the
4th Conference on Language Resources and Evaluation (LREC-2004)
---------------------------------------------------------------------------
A half-day workshop will be held in conjunction with the LREC
Conference in Lisbon on the morning of May 29, 2004. The topic of the
workshop is "Lessons Learned from Evaluation: Towards Transparency and
Integration in Cross-Lingual Information Retrieval (LECLIQ)" with
special focus on the use and benefits of quality gates.
TECHNICAL DESCRIPTION
Recent work in Cross-Lingual Information Retrieval (CLIR) has shown
that systems perform differently with respect to queries, topics, data
sets and the corresponding query or result language. Since the
state-of-the-art in CLIR is far from finding an ideal method for this
variety of parameters, a better performance might be possible through
continuous analysis and integration of evaluation results achieved
from several retrieval devices. The process of integrating these
different methodologies might be controlled by quality gates. Quality
gates - having their origin in car manufacturing and being used in
IT-project management as well - are characterized as preventive and
process accompanying control mechanisms to check critical parameters
and ensure quality standards during the design, development, and
deployment of software. They generally consist of checklists combined
with appropriate rules to guarantee that work procedure fail is
recognized in time to prevent repetition.
The objective of this workshop is to bring together researchers from
the field of Information Retrieval and Language Engineering with three
primary goals:
(1) to discuss, evaluate and judge the different methodologies used in
CLIR and to relate them to the languages being retrieved,
(2) to design the requirements for a dynamic network of quality gates
for the tuning of retrieval results,
(3) to define desired learn effects caused by the interaction of
quality gates.
Topics of interest include, but are not restricted to:
- Lessons learned in evaluation initiatives such as CLEF, TREC, INEX,
NTCIR, etc.
- Criteria, checklists, parameters and metrics for the evaluation of
search and retrieval methods
- Quality criteria and evaluation for resources and tools in CLIR
(thesauri stemmer, translation tools and services, etc.)
- Explanation and verification of success stories and failure analysis
in CLIR
- Data and Text Mining on CLIR evaluation results
- Current evaluation issues: shortcomings, gaps, reliability of
results, etc.
- Transfer of evaluation results to system design and product
development
- Design for (coupled or networked) quality gates in CLIR (for systems
and building blocks)
- Semaphore logic for the integration of distributed retrieval systems
- Crossing-over techniques for quality assurance
- Hybrid systems for CLIR
The results of this workshop will be a first step toward outlining a
general framework for quality gates in (Cross-Language) Information
Retrieval with further elaboration in future workshops.
We are inviting contributions discussing the themes above from
participants in evaluation initiatives as well as from experienced
researches in the areas of (Cross Language) Information Retrieval,
quality assurance, Human Language Technology (HLT) or related fields.
PROGRAMME COMMITTEE
Norbert Fuhr, University of Duisburg, Germany
Marcello Federico, I.R.S.T, Italy
Gregory Grefenstette, Clairvoyance Corp., Pittsburgh, USA
Donna Harman, NIST, USA
Paul Heisterkamp, DaimlerChrysler AG, Germany
Thomas Mandl, University of Hildesheim, Germany
Dagobert Soergel, University of Maryland, USA
Christa Womser-Hacker, University of Hildesheim, Germany
[Final list forthcoming]
SUBMISSION OF CONTRIBUTIONS
Presentations and active participation in the workshop will be based
on a 2 - 6 page position statement or research paper. Submissions must
be in English and should consist of an abstract, plus a title page
providing the following information: official title of the paper;
names and affiliations of the authors; full address of the first
author including phone, fax, e-mail, URL; presentation facilities
required. Only electronic submissions in Word, PDF or RTF will be
accepted. Demonstrations of cross-lingual retrieval tools will be
considered as well if the demonstration outlines the rationale of the
system and sheds light on the success or failure of the methods
applied. Contributions should be submitted to:
lecliq at uni-hildesheim.de Reception will be acknowledged within three
days. If you do not get an acknowledgement, please check the e-mail
address of your submission.
PROCEEDINGS
The workshop organizers will publish proceedings of the
workshop. Ready-to-print versions of the papers have to be submitted
according to LREC Guidelines for Authors, which can be found at the
conference homepage (to be announced in February 2004). They should
not exceed six pages. These final versions should be submitted
electronically to the following e-mail address:
lecliq at uni-hildesheim.de
IMPORTANT DATES
Deadline for submitting papers: February 15
Notification of acceptance: March 15
Final versions of papers for proceedings: April 4
Workshop: May 29
FURTHER INFORMATION
Information about the workshop such as call, outline, and program can
be found on the LREC web-page: http://www.lrec-conf.org/lrec2004/
where you can also find information about the LREC conference itself.
Information about the general purpose of quality gates in
manufacturing and software-engineering and their transduction to the
field of (Cross Lingual) Information Retrieval will be available under
http://www.uni-hildesheim.de/~lecliq
REGISTRATION
The registration fee for the workshop is:
- 50 EURO for conference participants
- 85 EURO for those not attending LREC-2004
Registration and payment procedures are explained on the LREC-2004
web-page.
The registration fee includes one copy of the proceedings and a coffee
break.
QUESTIONS
For all questions on the focus of the workshop, please contact the
workshop co-ordinator under lecliq at uni-hildesheim.de.
ORGANIZING COMMITTEE
Christa Womser-Hacker, University of Hildesheim, Center for
Information Science (Chair)
Norbert Fuhr, University of Duisburg
Paul Heisterkamp, DaimlerChrysler AG
Thomas Mandl, University of Hildesheim
René Schneider, University of Hildesheim (Co-ordinator)
------------------------
René Schneider
University of Hildesheim, Marienburger Platz 22, D-31141 Hildesheim,
Germany
Phone: +49(0)5121/883-834 Fax: +49(0)5121/883-802
e-mail: rschneid @ uni-hildesheim.de
-------------------------------------------------------------------------
Message diffusé par la liste Langage Naturel <LN at cines.fr>
Informations, abonnement : http://www.biomath.jussieu.fr/LN/LN-F/
English version : http://www.biomath.jussieu.fr/LN/LN/
Archives : http://listserv.linguistlist.org/archives/ln.html
La liste LN est parrainée par l'ATALA (Association pour le Traitement
Automatique des Langues)
Information et adhésion : http://www.atala.org/
-------------------------------------------------------------------------
More information about the Ln
mailing list