[Corpora-List] why LREC2012 NOT blind-reviewed?

Sun Oct 2 16:54:27 UTC 2011

Well, yes, crappy papers do get sent (and resent or multiple-sent), but I
don't see why non-blind reviewing cannot deal with that. In fact, the truly
crappy papers are the easiest to deal with and the most easy to have
consensus on.

And if the reviewers are not hidden behind the curtain and can be
identified, then why not just have them sign their reviews anyway?

The most important thing, in my opinion is what you see as the primary goal
of the peer review process. I see it as ensuring that good research is not
blacked out, rather than ensuring that an occassional bad (or even
unreliable or unethically written) paper does not get into the proceedings
or the journal. I would say let's take the risk of an occassional
crappy/unreliable/unethical/fraudulent paper being published in the most
prestigious conference proceedings or journals, but don't take the risk of
'killing' research that might turn out to be good. Because, to restate the
point I made earlier, the peer review process for publication is just the
first step in a longer process of peer review. That requires that any
research that could turn out to be good should get disseminated to the
community: should be put on the records. And that's why conferences like
LREC are precious, even if they sometimes allow papers of lower quality.
There is, in any case, no foolproof way to ensure the 'reliability' or
quality of a paper, with or without double blind reviewing.

Basically what I am saying is just an adaptation of an established legal
maxim in all societies where the rule of law is supposed to be present and
justice is taken seriously: "Let a few guilty get away, but never let an
innocent man be punished." (May be a paraphrase).

In the case of scientific (and by extension, engineering) research, we need
more than that because there can't be just one final trial. The case always
remains open. Or it should.

Once again I would say that if double blind reviewing had been working (that
is, had it been effective in ensuring what it is supposed to ensure), I
would not oppose it. But it's not working. And it's being misused.

On Sun, Oct 2, 2011 at 7:52 PM, Michal Ptaszynski <
ptaszynski at media.eng.hokudai.ac.jp> wrote:

> Just two humble cents from me.
> I'm sure most of us, even those strongly against double blind reviewing,
> remember or at least realize why this process was introduced.
> The fact is many people DO send many crappy papers, often re-sending
> exactly the same crap simultaneously to many conferences. If blind reviewing
> can flush at least some of this, and improve the world of science even in
> 1/100 of a percent, I wouldn't vote against it.
>
> Although what I would really like to have (and I think it is feasible) is a
> system telling me more-less if the paper is a candidate for a crap or a
> reliable piece of work.
>
> Also a word about the reviewers being hidden behind the curtain. This is
> not exactly true, since the list of all reviewers is always available on the
> conference homepage (and the list of sub-reviewers is sometimes also added
> in the proceedings). If you spent some time in the field you can more less
> guess who does things similar to you and can narrow down the list of your
> potential reviewers (so its very much like with guessing the paper authors).
>
> Best,
>
> Michal
>
> ------------------------------**-----
> Od: Anil Singh <anil.phdcl at gmail.com>
> Kopia dla: corpora at uib.no, Yorick Wilks <Y.Wilks at dcs.shef.ac.uk>
> Do: Yassine Benajiba <benajibayassine at gmail.com>
> Data: Sat, 1 Oct 2011 16:02:56 +0530
> Temat: Re: [Corpora-List] why LREC2012 NOT blind-reviewed?
>
> As some academicians/researchers I have respect for have expressed opinions
> which I believe to be right and which I have been writing about on my blog
> and elsewhere (apart from arguing for them in conversations), I dare to chip
> in and say my bit.
>
> When I had entered this area and was thinking of submitting my first paper
> (around 2003-2004) and I found out that reviewing will be blind, I was
> delighted. I was a nobody (a graduate student) from a developing country
> (India, but not even from one of the IITs and not with a very good
> pedigree), I thought double blind reviewing will be definitely more fair for
> people like me.
>
> My experience since then has completely disabused me of that naive idea.
> While it may not be possible to exactly identify the author(s) of the paper,
> one does get enough information (and meta-information) that is more than
> enough to trigger all the prejudices, biases etc. that blind reviewing is
> supposed to be an antidote against. This happens in almost all the cases.
> Needless to add that there can be exceptions.
>
> You can, of course, give numerous counter-examples from cases where no bias
> or prejudice is likely anyway or is very unlikely. But those examples are
> not the ones that matter here.
>
> As far as I am concerned, if you can just identify the fact that the author
> is from India, that alone removes at least half of the supposed
> effectiveness of the idea of double blind reviewing. And if you work on
> Indian languages and do certain kind of work, it's a no-brainer.
>
> Then there can be things like whether the author is just a student or an
> established researcher, whether the project is funded or non-funded, whether
> the language is that of a native speaker or not etc. These are the very
> things that double blind reviewing is supposed to guard against, but it
> simply can't. It just can't and I am sorry that it can't. Theoretically the
> idea still appeals to me, but may be like many other theoretically good
> things, it is not practically implementable.
>
> I especially like Yorick's comment about undignified gymnastics that one is
> required to perform to hide one's identity. It even lowers the academic
> quality of the paper quite often because you can't add information that is
> very relevant. And I am totally in favour of the reviewer taking
> responsibility for his comments. I have a corpus of reviews and some of the
> comments simply make one embarrassed that academicians (which one is too)
> can behave like that -- and that too in writing.
>
> One of the things that has always left me wondering (to put it lightly) is
> the fact that the conduct of academicians during the actual meetings, i.e.,
> paper presentations, panel discussions etc. is so exceedingly civilized (for
> want of a better word) that I sometimes feel out of place there (coming from
> a chaotic third world country and being disordered personally). But a lot of
> the same academicians, when they blind-review a paper, behave like bullies,
> vigilantes or just plain hooligans. Fortunately, their number is still a
> minority.
>
> Of course, like everyone else, I have received wonderful (even if very
> critical) reviews. But that can happen even with non-blind reviewing. Just
> read literary supplements of papers that take literature seriously.
>
> To conclude, I would just say that if for nothing else, at least to
> maintain the basic dignity of the academic community and of individual
> academicians, it would be best if we switch to a reviewing process that does
> not pretend to be blind and where reviewers take responsibility for their
> comments.
>
> I am agnostic about whether extended abstracts should be reviewed or full
> papers. Both seem to have their merits. For a conference like LREC, extended
> abstracts do seem better to me, though I won't fight for that (borrowing a
> phrase from review forms).
>
> I hope am not doing anything wrong by adding this link here:
>
> http://reviewscontd.org/
>
>
>
>  On Sat, Oct 1, 2011 at 6:28 AM, Yassine Benajiba <
> benajibayassine at gmail.com> wrote:
>  Hi everyone,
>
> I say let's judge the conference by the results. LREC is an awesome
> conference constantly improving year after year. Even though it would be
> great if somebody from the organizing committee could join this conversation
> and tell us a bit more about the reasons.
>
> Best,
>
> --Yassine.
>
> On Fri, Sep 30, 2011 at 11:31 PM, Eric Ringger <ringger at cs.byu.edu> wrote:
> Thanks to all for the open discussion.
>
> Graeme’s reason (1)(a) – the impact on merit review – is for me the
> strongest reason to encourage LREC to move away from reviewing extended
> abstracts and toward reviewing full papers.
>
> Best,
> --Eric
>
> From: Graeme Hirst [mailto:gh at cs.toronto.edu]
> Sent: Friday, September 30, 2011 11:02 AM
> To: Yorick Wilks
> Cc: Eric Ringger; corpora at uib.no
> Subject: Re: [Corpora-List] why LREC2012 NOT blind-reviewed?
>
> Yorick,
>
> (1)  Whether a conference is reviewed by abstract or by full paper makes an
> enormous difference:
>
>   (a) to merit, as perceived by tenure committees, granting agencies, and
> others, who count only fully peer-reviewed papers.
>   (b) to funding for travel.  Right now, one of my colleagues has the
> problem that he cannot be funded to travel to give a paper at LREC because
> it isn't a fully-reviewed conference, so he doesn't even bother submitting.
>
> You might say that these situations aren't desirable, but they are
> nonetheless reality right now.
>
> (2)  I wonder how you are so sure that you almost invariably identify the
> author of an anonymous paper correctly.  If the paper is not ultimately
> accepted at the conference, which is 60 to 80% of them at ACL and COLING
> conferences, you will never find out who the authors actually are.  I've
> certainly guessed wrongly in the past.  And in my own papers, I often throw
> in "hidden signals" to deceive the reviewers.
>
> (3)  I think Eric Ringger is 100% right about LREC.  As you say, LREC's
> reputation and quality have grown, and for that reason it has to start
> acting like a grown-up conference.
>
> Regards,
> Graeme
>
> --
> ::::  Graeme Hirst
> ::::  University of Toronto * Department of Computer Science
>
>  On 2011-09-30, at 11:27, Yorick Wilks wrote:
>
>
> I disagree strongly. I dont see why all conferences should be exactly like
> all others. Extended abstracts are less of a burden on busy academics --both
> as writers and reviewers----and there is no evidence they lower the final
> quality; COLING used to do this and I am sorry it changed. The whole
> blind-review business is a huge nonsense: I rarely meet a paper to review
> where i cannot identify the authors from a careful trawl of hidden signals
> and the references. Trying to make a paper genuinely anonymous is almost
> impossible if one has a body of past work and publication to link it
> to---the mental gymnastics required are undignified and best avoided. LRECs
> reputation has grown steadily and it will be the quality of its papers that
> sustain it--there is no evidence at all anonymity would improve matters in
> the least. if it ain't broke........
> Yorick Wilks
>
>
> On 30 Sep 2011, at 16:02, Eric Ringger wrote:
>
>
> Greetings.
>
> LREC has been operated in this manner since its inception.  Personally and
> for the sake of LREC’s reputation, I would like to see the reviewing process
> for LREC upgraded to double-blind review.
>
> I believe that LREC fills a couple of important niches: its focus on
> language resources and evaluation/validation is important and not well
> served elsewhere, and it does a good job of bringing a large, diverse group
> together.  (I should add that it does a good job of selecting attractive
> venues as well!)  If implemented well, I believe that double-blind review
> would not detract from the primary objectives of the conference but would
> refine the quality of the program and improve the reputation of the venue.
>  I have said as much in private feedback after past LRECs.
>
> I also think it is time for LREC to move up from reviewing extended
> abstracts to reviewing full papers.
>
> Regards,
> --Eric
>
>
> From: corpora-bounces at uib.no [mailto:corpora-bounces at uib.no**] On Behalf
> Of Isabella Chiari
> Sent: Friday, September 30, 2011 8:45 AM
> To: corpora at uib.no
> Subject: [Corpora-List] why LREC2012 NOT blind-reviewed?
>
> Dear Corpora members,
> I just noticed that the LREC2012 call specifies that submissions are NOT
> anonymous and there will not be blind-reviewing.
>
> Does anyone know why? Which is the policy under this decision?
> Best regards,
> Isabella Chiari
>
>
>
>  ______________________________**_________________
>  UNSUBSCRIBE from this page: http://mailman.uib.no/options/**corpora<http://mailman.uib.no/options/corpora>
>  Corpora mailing list
> Corpora at uib.no
> http://mailman.uib.no/**listinfo/corpora<http://mailman.uib.no/listinfo/corpora>
>
>
>
> ______________________________**_________________
>  UNSUBSCRIBE from this page: http://mailman.uib.no/options/**corpora<http://mailman.uib.no/options/corpora>
>  Corpora mailing list
> Corpora at uib.no
> http://mailman.uib.no/**listinfo/corpora<http://mailman.uib.no/listinfo/corpora>
>
> ______________________________**_________________
> UNSUBSCRIBE from this page: http://mailman.uib.no/options/**corpora<http://mailman.uib.no/options/corpora>
> Corpora mailing list
> Corpora at uib.no
> http://mailman.uib.no/**listinfo/corpora<http://mailman.uib.no/listinfo/corpora>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listserv.linguistlist.org/pipermail/corpora/attachments/20111002/6e54ff3b/attachment.htm>
-------------- next part --------------
_______________________________________________
UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
Corpora mailing list
Corpora at uib.no
http://mailman.uib.no/listinfo/corpora