[Corpora-List] CLEF 2011 Call for Lab Proposals - Deadline: Monday November 8

[Pamela Forner]

*******************************************************************

CLEF 2011 - Call for Lab proposals - Deadline: Monday November 8

*****************************************************************

The CLEF 2011 conference is the continuation of the popular CLEF 
campaigns and workshops that ran 2000-2009 
(http://www.clef-campaign.org/). In 2010, CLEF was organized in a 
different way: the CLEF 2010 Conference (http://clef2010.org 
<http://clef2010.org/>) presenting related research papers followed by a 
series of labs covering a broad range of issues from fields of 
multilingual and multimodal information access evaluation. CLEF 2011 
will be organized in a similar way. Researchers and practitioners from 
all segments of the information access and related communities are 
invited to submit proposals for review.


SCOPE

Proposals are accepted for two different types of labs:

    * Labs that are a campaign-style evaluation for specific
      information access problems (during the twelve month period
      preceding the conference), similar in nature to the traditional
      CLEF campaign tracks. In 2010 there were 5 labs
      (http://clef2010.org/index.php?page=pages/labs.html): CLEF-IP,
      ImageCLEF, PAN, RespublicQA and WePS. Topics covered by
      campaign-style labs can be any information access-related task.

    * Labs that follow a more classical workshop pattern, exploring
      issues of evaluation methodology, metrics, processes etc. in
      information access and closely related fields, such as natural
      language processing, machine translation, and human-computer
      interaction. In 2010 there were 2 workshops: CriES and LogCLEF.

In both cases, it is expected that lab sessions at the conference will 
contain ample time for general discussion and engagement by all 
participants - not just those presenting campaign results and papers. 
Organizers should plan time for panels, demos, etc. where applicable.
Location

Two days of the four-day CLEF 2011 conference have been reserved 
exclusively for lab sessions. The lab sessions will take place at the 
site of the conference at the University of Amsterdam, Netherlands. The 
potential for labs to present their overall results overview 
presentations during the scientific paper session of the conference is 
presently being explored.


PROPOSAL SUBMISSION

Lab proposals should provide sufficient information for the lab 
organizing committee to be able to judge the importance, quality, and 
benefits for the research community.

Each lab should have one or more organizers responsible for the 
execution of the lab.
Proposals should be 2-4 pages long and should provide the following 
information:

    * Title of the proposed lab.
    * The planned format of the lab, i.e. campaign-style (track) or
      workshop.
    * Planned length of the lab session at the conference: half-day, one
      day, two days.
    * Names and full addresses, including contact details, of the lab
      organizer(s), a brief description of the organizers' experience
      and background in the topic, and links to web pages of the lab
      organizers.
    * A brief description of the lab topic and goals, its relevance to
      CLEF 2011 and significance for the research field together with
      indications as to how the proposed tasks/actions contribute to
      their implementation
    * A statement on the intended development/growth path if the
      proposal is for a continuation of activities previously undertaken
      at CLEF workshops.
    * A description of the target audience, areas from which the
      participants are expected to come, an analysis of the potential
      for participants (number, statements of intent to participate
      where applicable), potential industry stakeholders, strategy for
      publicizing the lab
    * Arrangements for the organization of the lab campaign, if
      applicable, including a brief outline of the campaign milestones,
      test data to be used, indications of the size of the data
      collections, issues of scalability, tasks to be proposed to
      participants, and format of presentation at the conference.
    * If the lab proposes to set up a steering committee to oversee its
      activities, include names, addresses, and home page links of
      people who have agreed to be part of the steering committee if the
      lab proposal is accepted. This list should ideally include people
      from at least three different countries.



REVIEWING PROCESS

Each submitted proposal will be reviewed by the CLEF2011 lab organizing 
committee. The decision will be sent by email to the responsible 
organizer by November 15, 2010.
The final length of the lab session will be determined based on the 
overall organization of the conference and the number of submissions 
received by a lab. Due to space restrictions, only a limited number of 
lab sessions can be conducted in parallel at the conference. The 
reviewers may suggest modifications to the proposed lab programme in 
order to fit it with the above reviewing criteria and the overall 
organization of CLEF 2011.

Reviewing criteria for the "campaign-style" labs include:

    * Soundness of methodology, feasibility of task.
    * Use case (description of the underlying problems), business
      case/industrial stakeholders (potential for market).
    * Number of potential participants, critical mass.
    * Clear movement along a growth path, development of field.
    * Others factors such as: innovation, minimize overlap with other
      evaluation initiatives or events, focus (appropriate number of
      subtasks, avoid inflation of labs), interdisciplinary character.

Reviewing criteria for the workshop-style labs include:

    * The appropriateness of the workshop-style lab to the overall
      information access agenda pursued by CLEF.
    * Number of potential participants, critical mass.
    * Likelihood that the outcomes of the workshop will constitute a
      significant contribution to the field. Is the workshop focused
      enough that useful conclusions are likely?
    * Other factors such as: innovation, minimize overlap with other
      evaluation initiatives and events, vision for a potential
      continuation, interdisciplinary character.

LAB ORGANIZER'S TASKS:

    * Producing a Call for Participation for the campaign-style lab or
      a Call for Papers for a workshop-style lab and disseminating it
      through all appropriate means. Please provide a web page URL which
      can be linked into the CLEF 2011 home page (asap).
    * Providing a brief description of the lab for the conference program.
    * Signing up campaign participants, and executing the campaign in
      the case of campaign-style labs.
    * Reviewing/accepting submitted papers and position papers in case
      of workshop-style labs.
    * Scheduling lab session activities in collaboration with the local
      organizers and the lab chairs.
    * Sending the lab schedule and other lab material, all in PDF
      format, to the lab chairs (deadline to be defined).
    * The lab material (papers, presentations etc.) will be distributed
      by the CLEF 2011 organization to the conference participants in
      electronic format (copyright will not be asked for from the
      authors, but only permission to publish and disseminate).
    * Organization of post-conference publication of lab results in
      appropriate form (special issue, lab proceedings, etc.).


PUBLICATION

The working notes of the labs will be published online in time for the 
conference. It is foreseen that this online publication will have an 
ISBN number and be indexed in relevant services. It is the 
responsibility of lab organizers to arrange for appropriate 
post-conference publication of the lab results.
Important Dates

    * Final lab proposals: 8 November 2010
    * Notification of lab acceptance: 15 November 2010
    * CLEF 2011 Conference: September 2011


SUBMISSION DETAILS

Lab proposals (or questions) should be submitted via e-mail (either 
plain text or PDF format) to both Lab Organizing Committee Chairs:

Vivien Petras, Berlin School of Library and Information Science,
Humboldt-Universitaet zu Berlin
vivien.petras[at]ibi.hu-berlin.de

Paul Clough, The Information School, 
University of Sheffield, 
p.d.clough[at]sheffield.ac.uk





_______________________________________________
Corpora mailing list
Corpora at uib.no
http://mailman.uib.no/listinfo/corpora