[Corpora-List] why LREC2012 NOT blind-reviewed?

Mon Oct 3 15:16:41 UTC 2011

Dear colleagues,

Note sure if this was mentioned before in the discussion:
Concerning LREC, as many papers deal with language resources
which are often made available on the web, it makes sense that 
reviewers can also look at these sites. This is not compatible 
with double blind reviewing.

Concerning bad reviews, I think reviewers usually have the
best intentions (otherwise they would not volunteer), but life 
can be very harsh with us poor scientists,  so completely
unforeseen problems such as the following may arise:

1) The reviewer was too optimistic timewise and finally has to 
   act in a last minute fashion (90% probability).

2) The assigned paper simply does not match the reviewer's
    competences or interests (50% probability).

3) The paper is so bad that it does not deserve a detailed review
    (10% probability; some help for newcomers might nevertheless 
    be nice).

In my experience, bidding is very helpful to obtain better matches
between papers and reviewers' interests and thus helps to solve
problem 2). But it is usually based on abstracts which can be 
misleading.

So let me suggest to take this one step further:

Allow reviewers to return papers which they don't like to review
after looking at them.

This could be realized as follows:

>>From the pool of all papers reviewers can pick freely the ones that
look interesting to them (of course based on some pre-classification 
and up to a maximum number). These papers are then assigned 
to the reviewer for a certain time slot (a fraction of the total reviewing 
period). If a paper has enough assignments, it is locked, i.e. invisible 
for new reviewers. If a reviewer does not provide a review within the 
given time slot (alternatively: makes a binding commitment), the 
paper is unlocked, i.e. returned to the pool.

This method would hopefully solve problem 2) above. But it would also
reduce problem 1): As early reviewers have a better choice, fast action
is encouraged. Papers which are returned several times would finally
(towards the end of the reviewing period) have to be dealt with in a 
last minute fashion. But the hope is that many of these papers would 
fall into category 3) anyway (rather than papers on less popular
topics finding no reviewer).

This procedure would also remove from the programme committee
(some of) the burden of making appropriate paper assignments, and 
would encourage or even force reviewers to provide their reviews at 
an early stage.

But needless to say that I am not sure whether this would work in 
practice, and whether the benefits would outweigh the extra effort. 

Kind regards,

Reinhard

  ----- Original Message ----- 
  From: Ted Pedersen 
  To: corpora at uib.no 
  Sent: Monday, October 03, 2011 1:39 PM
  Subject: Re: [Corpora-List] why LREC2012 NOT blind-reviewed?

  I think one thing we've learned here is that there are some fairly significant concerns and frustrations with reviewing. 

  For myself, I review too much, and am probably a little lazy in some cases as a result. I started to notice that a bit more recently, and so have started saying "No" with much more frequency, so hopefully I am doing my small part to correct something in the process. Reviewers who feel they don't have time to review really need to say "No" more often - the field will survive without us, and we aren't doing anyone any favors by submitting reviews that we don't really spend enough time on. LIkewise, I don't think saying "Yes" and then farming out reviews to graduate students is all that helpful, unless the senior person is willing to spend some time with the student on reviewing (until they are sufficiently experienced). I realize more folks doing this will make it harder to get reviews, but I think a smaller number of better reviews is in the end more helpful and healthy.

  I would like to suggest that maybe we ought to ask people who submit papers to provide the answers to the following two questions (separate from their papers) in an effort to streamline the process.

  1) What is the most important idea presented in this submission (in 50 words or less)

  2) What other paper is most similar to this submission, and how does this paper improve upon or extend that? (in 100 words or less)

  If I'm not sufficiently excited by the answers to both 1 and 2, then the paper can be rejected without further review. A good paper will of course make 1 and 2 fairly clear, but sometimes you have to dig a little, so I'd like to dispense with the kabuki dance and simply ask authors to answer these questions at the start, and then we decide as reviewers if we should read further. 

  The other part of the equation is that most published papers don't end up having much impact beyond advancing the author's career (Zipf's Law for Papers? A few papers cited a lot, most not cited much at all). This doesn't mean they shouldn't be published,  and career advancement is a good thing generally both for the authors and our field, but it can also make for lots and lots of  incremental papers that just aren't all that interesting and so they aren't cited much, and they tend to have a mind numbing effect on reviewers and is part of what I think makes reviewing such a chore sometimes.  So, maybe if we make authors self-identify the incremental work versus the big new ideas then reviewers can have a better idea of what to expect. Incremental in the area you care most about can be fascinating stuff, so I don't think incremental is always a bad thing, but there also needs to be a balance between the incremental and the more novel. As a reviewer I feel like I spend huge amounts of time on incremental work, and it just gets a little dull to be honest...

  Cordially,
  Ted

  On Fri, Sep 30, 2011 at 9:45 AM, Isabella Chiari <isabella.chiari at uniroma1.it> wrote:

    Dear Corpora members,
    I just noticed that the LREC2012 call specifies that submissions are NOT anonymous and there will not be blind-reviewing.

    Does anyone know why? Which is the policy under this decision?
    Best regards,
    Isabella Chiari

    Dipartimento di Scienze documentarie, linguistico-filologiche e geografiche

    Università di Roma “La Sapienza”

    pl.le Aldo Moro, 5, III Piano, Edificio ex Facoltà di Lettere e Filosofia, 00185 Roma, tel. +30 06 4991 3575 

    E.mail: isabella.chiari at uniroma1.it

    Website: www.alphabit.net

    _______________________________________________
    UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
    Corpora mailing list
    Corpora at uib.no
    http://mailman.uib.no/listinfo/corpora

  -- 
  Ted Pedersen
  http://www.d.umn.edu/~tpederse

------------------------------------------------------------------------------

  _______________________________________________
  UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
  Corpora mailing list
  Corpora at uib.no
  http://mailman.uib.no/listinfo/corpora
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listserv.linguistlist.org/pipermail/corpora/attachments/20111003/72341a51/attachment.htm>
-------------- next part --------------
_______________________________________________
UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
Corpora mailing list
Corpora at uib.no
http://mailman.uib.no/listinfo/corpora