Call for participation - INFILE

[Helene Mazo]

Apologies for cross-postings


        Call for participation

             INFILE at CLEF2008

     Information, Filtering, Evaluation
     
          http://www.infile.org


INFILE welcomes participation of any institution to its first evaluation 
campaign. This participation is free of charge and
participants can keep and use the development and evaluation data for 
free after the evaluations for research and development purposes.

INFILE (INformation, Filtrage, Evaluation) is a cross-language adaptive 
filtering evaluation campaign jointly organized by CEA,
Université de Lille 3 and ELDA. It is organized as a pilot track in CLEF 
2008 and is supported by NIST TREC.

INFILE extends the last filtering track of TREC 2002 in the following ways:
-INFILE is crosslingual (English, French and Arabic); a corpus of  
100,000 comparable news-wire stories from Agence France Presse (AFP)
 for each language is used for the evaluations.

-Evaluation will be performed using an automatic interrogation of test  
systems with a simulated user feedback. Each system will be able to
 use the feedback at any time to increase performance.

The participant systems will have to provide a Boolean decision for each 
document according to each filtering  profile. A curve of the evolution 
of efficiency will be computed.
Although cross-lingual systems are encouraged, the campaign is also open 
to monolingual systems.


Tasks and languages
-------------------
Two tasks and three languages are considered. The first task is 
Information Filtering on general news and events. For this task, 
participants will have to classify each transmitted news-wire into
zero, one or more different profiles. 30 general profiles will be made 
available in 3 languages (Arabic, English and French).

The second task is Information Filtering on science and technology 
domain. Participants will have to associate to each news-wire zero, one 
or more science and technology profiles.
A total of 20 profiles will be available in 3 languages (Arabic, English 
and French).

For each task, participants are free to register to monolingual 
filtering (e.g. information filtering using profiles and news-wires in 
the same language) or to crosslingual filtering (e.g. information
filtering according to profiles in one language and news-wires in 
another language).

Corpus
------
The corpus consists of 300,000 news-wires in Arabic, English and French 
from the news agency Agence France Presse covering the 2004-2006 period.
The news-wires are related to general news and events information and 
are comparable between Arabic, English and French.


Protocol description
---------------------
General information about the domain of profiles is given to each 
participant. 15 days afterwards, 50 profiles are given to participants 
(30 general
profiles and 20 profiles related to science and technology). Profiles 
are composed of a list of keywords (simple and complex noun phrases) and 
up to 3
documents illustrating each profile.
 
Then, news-wires are transmitted by the organizer to an automated  
interface of each participating system. The interface returns a
 Boolean response for each profile. After reception of this response, 
and if requested by the participant, the organizer sends a feedback
 consisting of expected profile assignments for each document  
submitted. Participants may adapt their system at any time using this 
feedback.


Important Dates
---------------
    Registration Opens - Feb 11th, 2008
    Dry Run - June 2nd to June 14th, 2008
    Evaluation Run June 30th - July 19th
    Release of Human Assessments and Individual Results - August 4th, 2008
    Submission of Paper for Working Notes - 15 August 2008
    Workshop - 17-19 September 2008 CLEF Workshop

Contact
-------
    info at infile.org
    http://www.infile.org