Appel: CLEF 2015, Call for Labs proposals, 10 September 2014

Sun Aug 17 10:53:36 UTC 2014

Date: Wed, 13 Aug 2014 11:26:12 +0200
From: Eric SanJuan <eric.sanjuan at univ-avignon.fr>
Message-ID: <53EB2F34.5080908 at univ-avignon.fr>
X-url: http://clef2015.clef-initiative.eu/

*********************************************
CLEF 2015 - Call for Labs proposals
-----------------------------------------

http://clef2015.clef-initiative.eu/

Toulouse, France, September 8 - 11, 2015

Important Dates
---------------
* Final lab proposals: 10 September 2014
* Notification of lab acceptance: 19 September 2014
* 3 November 2014: Labs registration opens

CLEF is a leading annual international conference exploring topics in
information access technologies.
The CLEF Conference uniquely combines evaluation laboratories and
workshops with research presentations, panels, poster and demo sessions
interleaved during three and a half days of intense and stimulating
research activities. CLEF 2015 will be the sixth edition of the current
CLEF conference launched as a continuation of the annual CLEF evaluation
campaign series which ran with great success from 2000 to 2009
contributing to the systematic evaluation of information access systems,
primarily through experimentation on shared tasks.

CLEF 2015 will be held in September at the University of Toulouse,
France.  Researchers and practitioners from all areas of information
access and related communities are invited to submit proposals for
evaluation labs for consideration for inclusion in CLEF 2015. Lab
proposals will be reviewed by the lab selection committee. Authors of
selected proposals will be invited to include their lab in the CLEF 2015
lab programme, possibly subject to small suggested modifications to
their proposal to better suit the CLEF lab workflow or timeline.

About CLEF
----------
The CLEF Initiative (http://clef2015.clef-initiative.eu/) is a 
self-organized body whose main mission is to promote research, 
innovation, and development of information access systems with an 
emphasis on multilingual information in different modalities - including 
text and multimedia - with various levels of structure. CLEF promotes 
research and development by providing an infrastructure for:

* independent evaluation of information access systems;
* investigation of the use of unstructured, semi-structured, 
  highly-structured, and semantically enriched data in information
  access;
* creation of reusable test collections for benchmarking;
* exploration of new evaluation methodologies and innovative ways of 
  using experimental data;
* discussion of results, comparison of approaches, exchange of ideas, 
  and transfer of knowledge.

Scope of CLEF Labs
------------------
Proposals are accepted for two different types of “labs”:

* Evaluation Labs that are a “campaign-style” evaluation for specific
  information access problems (during the twelve month period preceding
  the conference), similar in nature to the traditional CLEF campaign
  “tracks” (see 2014 Labs at
  http://clef2014.clef-initiative.eu/). Topics covered by campaign-style
  labs can be inspired by any information access-related domain or task.

* Labs that follow a more classical “workshop” pattern, exploring issues
  of evaluation methodology, metrics, processes etc. in information
  access and closely related fields, such as natural language
  processing, machine translation, and human-computer interaction.

If the organisers of the proposal are new to CLEF or other shared task
evaluation campaigns, we highly recommend that a lab workshop first be
organised to discuss the format, the problem space, and the
practicalities of the shared task. The CLEF 2015 programme will reserve
about half of the conference schedule for lab sessions which will take
place within the conference in Toulouse. The labs will present their
overall results “overview presentations” during the plenary scientific
paper sessions to allow non-participants to get a sense of where the
research frontiers are moving. Organisers of each lab are expected to
organise the separate sessions for their lab at the conference so that
they contain ample time for general discussion and engagement by all
participants - not just those presenting campaign results and papers.
Organisers should plan time in their sessions for activities such as
panels, demos, poster sessions etc. as appropriate.

CLEF is always interested in receiving and facilitating innovate and
creative lab proposals. Potential task proposers who are unsure of the
suitability of their task proposal or its format for inclusion at CLEF
are encouraged to contact the CLEF 2015 Lab Organizing Committee Chairs
to discuss its suitability or design at an early stage.

Proposal Submission
-------------------
Lab proposals must provide sufficient information to enable to be able
to judge the relevance, timeliness, scientific quality, benefits for the
research community, and the competence of the proposers to co-ordinate
the lab. Each lab proposal should identify one or more organizers as
responsible for ensuring the timely execution of the lab.

Proposals should be 2-4 pages long and should provide the following 
information:

1. Title of the proposed lab.
2. A brief description of the lab topic and goals, its relevance to CLEF
   and the significance for the field.
3. A brief but clear statement of usage scenarios or domain to which the
   activity is intended to contribute, including the evaluation setup
   and metrics.
4. Details on the lab organizer(s), including identifying the task
   chair(s) responsible for the ensuring the running of the task. This
   should include details of any previous of involvement in organising
   or participating in evaluation tasks at CLEF or similar campaigns.
5. The planned format of the lab, i.e. campaign-style (“track”) or
   workshop.
7. A statement on the intended development/growth path if the proposal
   is for a continuation of activities previously undertaken at CLEF
   workshops.
8. Details of the expected target audience, i.e. who do you expect to
   participate in the task, and how do you propose to reach them.
9. The number of tasks to be carried out in a lab. The proposal should
   clearly motivate the need for each of the proposed tasks and provide
   evidence of its capability of attracting enough participation. It is
   fine for a lab to have a single task, but labs often contain multiple
   closely related tasks.
10. Expected length of the lab session at the conference: half-day, one
    day, two days. This should include very high level details of
    planned structure of the session, e.g. participant presentations,
    invited speaker(s), panels, etc, to justify the requested session
    length.
11. Arrangements for the organization of the lab campaign: who will be
    responsible for activities within the task; how will data to
    acquired or created, including highlighting any IP issues that need
    to be addressed to enable data to be distributed to lab
    participants; tools or methods which will be used, e.g. how will
    necessary queries be created or relevance assessment carried out;
    any other information which is relevant to the conduct of your lab.
12. If the lab proposes to set up a steering committee to oversee and
    advise its activities, include names, addresses, and homepage links
    of people your propose to be involved.

Reviewing Process
-----------------
Each submitted proposal will be reviewed by the CLEF 2015 lab organizing
committee.
The decision will be sent by email to the responsible organizer by 
September 19, 2014.
The final length of the lab session will be determined based on the
overall organization of the conference and the number of participant
submissions received by a lab. Reviewing criteria can be found on the
website of the conference - http://clef2015.clef-initiative.eu/ .

Important Dates
---------------
* Final lab proposals: 10 September 2014
* Notification of lab acceptance: 19 September 2014
* 3 November 2014: Labs registration opens

Submission Details
------------------
Lab proposals (or questions) should be submitted via e-mail (either 
plain text or PDF format, using “CLEF labs proposal” in the subject
line) to both Lab Organizing Committee Chairs:

Eric San-Juan , eric.sanjuan at univ-avignon.fr
Gareth Jones, gareth.jones at computing.dcu.ie

Further information is available at: http://clef2015.clef-initiativeeu/

-------------------------------------------------------------------------
Message diffuse par la liste Langage Naturel <LN at cines.fr>
Informations, abonnement : http://www.atala.org/article.php3?id_article=48
English version       : 
Archives                 : http://listserv.linguistlist.org/archives/ln.html
                                http://liste.cines.fr/info/ln

La liste LN est parrainee par l'ATALA (Association pour le Traitement
Automatique des Langues)
Information et adhesion  : http://www.atala.org/

ATALA décline toute responsabilité concernant le contenu des
messages diffusés sur la liste LN
-------------------------------------------------------------------------