[Corpora-List] CLEF 2010 Call for Lab Proposals

[Edgar Meij]

CLEF 2010
Conference on Multilingual and Multimodal Information Access Evaluation
http://clef2010.org

Call for Lab Proposals

The CLEF 2010 conference is the continuation of the popular CLEF
campaigns and workshops that have run for the past ten years
(2000-2009).  It will cover a broad range of issues from fields
of multilingual and multimodal information access evaluation.
One of the two main parts of CLEF 2010 will be a series of
"labs".  Two different forms of labs will be offered: labs can
either be run "campaign-style" during the twelve month period
preceding the conference, or adopt a more "workshop"-style format
that can explore issues of information access evaluation and
related fields.  The labs will culminate in sessions of a
half-day, one full day or two days at the CLEF 2010 conference.

Researchers and practitioners from all segments of the
information access and related communities are invited to submit
proposals for review.

SCOPE

Proposals are accepted for two different types of "labs":

· Labs that are a "campaign-style" evaluation for (a) specific
  information access problem(s).  These labs will be very similar
  to the current CLEF campaign "tracks" in their nature.  Topics
  covered by campaign-style labs can be any information
  access-related task(s).
· Labs that follow a more classical "workshop" pattern, exploring
  issues of evaluation methodology, metrics, processes etc.  in
  information access and closely related fields, such as natural
  language processing, machine translation, and human-computer
  interaction.

In both cases, it is expected that lab sessions at the conference
will contain ample time for general discussion and engagement by
all participants - not just those presenting campaign results and
papers.  Organizers should plan time for panels, demos, etc.
where applicable.

LOCATION

Two days of the four-day CLEF 2010 conference have been reserved
exclusively for lab sessions.  The lab sessions will take place
at the site of the conference at the University of Padua, Padua,
Italy.  The potential for labs to present their overall results
("overview presentations") during the scientific paper session of
the conference is presently being explored.

PROPOSAL SUBMISSION

Lab proposals should provide sufficient information for the lab
organizing committee to be able to judge the importance, quality,
and benefits for the research community.

Each lab should have one or more organizers responsible for the
execution of the lab.  Proposals should be 2-4 pages long and
should provide the following information:

· Title of the proposed lab.
· The planned format of the lab, i.e. campaign-style ("track") or
  workshop.
· Planned length of the lab session at the conference: half-day,
  one day, two days.
· Names and full addresses, including contact details, of the lab
  organizer(s).
· A brief description of the lab topic and goals, its relevance
  to CLEF 2010 and significance for the research field.
· A statement on the intended development/growth path if the
  proposal is for a continuation of activities previously
  undertaken at CLEF workshops.
· A list of the main goals of the proposed labs, together with
  indications as to how the proposed tasks/actions contribute to
  their implementation
· A description of the target audience, areas from which the
  participants are expected to come, an analysis of the potential
  for participants (number, statements of intent to participate
  where applicable), potential industry stakeholders, strategy for
  publicizing the lab
· Arrangements for the organization of the lab campaign, if
  applicable, including a brief outline of the campaign milestones,
  test data to be used, indications of the size of the data
  collections, issues of scalability, tasks to be proposed to
  participants, and format of presentation at the conference.
· If the lab proposes to set up a steering committee to oversee
  its activities, include names, addresses, and home page links of
  people who have agreed to be part of the steering committee if
  the lab proposal is accepted.  This list should ideally include
  people from at least three different countries.
· A brief description of the organizers' experience and
  background in the topic, and links to web pages of the lab
  organizers.

REVIEWING PROCESS

Each submitted proposal will be reviewed by the CLEF2010 lab
organizing committee.  The decision will be sent by email to the
responsible organizer by November 10, 2009.  The final length of
the lab session will be determined based on the overall
organization of the conference and the number of submissions
received by a lab.  Due to space restrictions, only a limited
number of lab sessions can be conducted in parallel at the
conference.  The reviewers may suggest modifications to the
proposed lab programme in order to fit it with the above
reviewing criteria and the overall organization of CLEF 2010.

Reviewing criteria for the "campaign-style" labs include:

· Soundness of methodology, feasibility of task
· Use case (description of the underlying problems), business
  case/industrial stakeholders (potential for market)
· Number of potential participants, critical mass
· Clear movement along a growth path, development of field
· Others factors such as: innovation, minimize overlap with other
  evaluation initiatives or events, focus (appropriate number of
  subtasks, avoid "inflation" of labs), interdisciplinary character

Reviewing criteria for the "workshop-style" labs include:

· The appropriateness of the workshop-style lab to the overall
  information access agenda pursued by CLEF
· Number of potential participants, critical mass
· Likelihood that the outcomes of the workshop will constitute a
  significant contribution to the field.  Is the workshop focused
  enough that useful conclusions are likely?
· Other factores such as: innovation, minimize overalp with other
  evaluation initiatives and events, vision for a potential
  continuation, interdisciplinary character

LAB ORGANIZER'S TASKS:
· Producing a "Call for Participation" for the camapaign-style
  lab or a "Call for Papers" for a workshop-style lab and
  disseminating it through all appropriate means.  Please provide a
  web page URL which can be linked into the CLEF 2010 home page
  asap.
· Providing a brief description of the lab for the conference
  program.
· Signing up campaign participants, and executing the campaign in
  the case of campaign-style labs
· Reviewing/accepting submitted papers and position papers in
  case of workshop-style labs
· Scheduling lab session activities in collaboration with the
  local organizers and the lab chairs.
· Sending the lab schedule and other lab material, all in PDF
  format, to the lab chairs (deadline to be defined).
· The lab material (papers, presentations etc.)  will be
  distributed by the CLEF 2010 organization to the conference
  participants in electronic format (copyright will not be asked
  for from the authors, but only permission to publish and
  disseminate).
· Organization of post-conference publication of lab results in
  appropriate form (special issue, lab proceedings, etc.)

PUBLICATION

The working notes of the labs will be published online in time
for the conference.  It is foreseen that this online publication
will have an ISBN number and be indexed in relevant services.  It
is the responsibility of lab organizers to arrange for
appropriate post-conference publication of the lab results.

IMPORTANT DATES

· Final lab proposals: 31 October 2009
· Notification of lab acceptance: 10 November 2009
· Lab web page on the conference site: tbd

SUBMISSION DETAILS

Lab proposals should be submitted via e-mail (either plain text
or PDF format) to both Lab Organizing Committee Chairs:

Martin Braschler, Zurich University of Applied Sciences, bram at zhaw.ch
Donna Harman, National Institute of Standards and Technology, donna.harman at nist.gov

Please direct all questions to the lab chairs.

Preliminary information about CLEF 2010 can be found at
http://clef2010.org
_______________________________________________
Corpora mailing list
Corpora at uib.no
http://mailman.uib.no/listinfo/corpora