[Corpora-List] why LREC2012 NOT blind-reviewed?

Sun Oct 2 20:12:11 UTC 2011

I've been avoiding this thread for some time, but couln't resist seconding
Laurence's (brilliant) summary and comment.  Just my 2 cents.

C:
--
Dr Christopher Tribble
TEL     || +44 (0)207 833 4271
EMAIL   || ctribble at clara.co.uk
WEB     || www.ctribble.co.uk

> -----Original Message-----
> From: corpora-bounces at uib.no [mailto:corpora-bounces at uib.no] On Behalf Of
> Laurence Anthony
> Sent: 02 October 2011 16:30
> To: corpora at uib.no
> Subject: Re: [Corpora-List] why LREC2012 NOT blind-reviewed?
> 
> Just to follow up on Michal's comments,
> 
> I'm quite surprised how much animosity there seems to be against
> reviewers. In this thread people have said; "the reviewers make
> linguistics not brilliant", "when they blind-review a paper, behave like
> bullies, vigilantes or just plain hooligans."
> "it would be best if we switch to a reviewing process ... where reviewers
> take responsibility for their comments."
> 
> It seems to me that reviewers are just peers who have been asked to
> volunteer their service to a conference/journal by the editor/chair.
> They accept the request because they feel it helps the conference/journal
> in some way, and they attempt to the best of their abilities to provide a
> balanced judgement on the merits of the papers they are reviewing. For a
> good conference/journal, usually two reviewers are asked to submit their
> opinions, and almost always, the editor/chair will have the final say on
> whether or not to accept the reviews or seek advice of a third or fourth
> reviewer. Obviously, there will be times when the reviews will be of
> lower quality than desired.
> But the entire process is designed to avoid these making an impact on the
> final decision to accept/reject a paper.
> 
> Of course, not all reviewers will be "brilliant". And, few would ever
> claim to be so. They are *peers*, doing peer reviews. If the author does
> not accept the review they almost always have the ability to respond to
> the review, explaining where they see its faults. And, if reviewers acted
> like "bullies, vigilantes or just plain hooligans", editors/chairs would
> very quickly stop asking them to review work.
> Finally, I'm sure that most reviewers consider very carefully the
> responsibility they have, and they certainly take responsibility for
> their comments because they submit them openly to the journal
> editor/conference chair (even if their names are ultimately hidden to the
> author). They do not hide behind user names, IDs, or avatars. As Michal
> writes, they allow their names to appear on the conference/journal
> website or in the proceedings.
> 
> Saying all this, if there is such animosity to reviewers and most peers
> feel they are unintelligent, irresponsible bullies, vigilantes, and
> hooligans, I'm sure reviewers would be quite happy to stop providing
> their *voluntary service*. But, I would imagine that the loss to the
> conferences/journals would be much greater than the gains they would
> make.
> 
> On the point of the double blind review system, it is certainly not
> perfect. But, arguing for a completely open system that invites all kinds
> of biases, simply because the current system does not work perfectly,
> seems rather odd. Are people arguing that allowing both authors and
> reviewers to know the names of each other will lead to a fairer system?
> If so, is there any evidence for this? I haven't checked the history but
> would expect that the double blind review system was introduced
> *precisely because* of the problems of an open system.
> 
> I am not sure why LREC2012 has made the decision it has. I would be very
> interested to hear how their revised system will improve the conference.
> 
> Laurence Anthony
> (taking full responsibility for my comments even if they are not the most
> brilliant in the world!)
> 
> 
> 
> 
> 
> 
> 
> 
> 
> On Sun, Oct 2, 2011 at 11:23 PM, Michal Ptaszynski
> <michal.ptaszynski at gmail.com> wrote:
> > Just two humble cents from me.
> > I'm sure most of us, even those strongly against double blind
> > reviewing, remember or at least realize why this process was
> introduced.
> > The fact is many people DO send many crappy papers, often re-sending
> > exactly the same crap simultaneously to many conferences. If blind
> > reviewing can flush at least some of this, and improve the world of
> > science even in 1/100 of a percent, I wouldn't vote against it.
> >
> > Although what I would really like to have (and I think it is feasible)
> > is a system telling me more-less if the paper is a candidate for a
> > crap or a reliable piece of work.
> >
> > Also a word about the reviewers being hidden behind the curtain. This
> > is not exactly true, since the list of all reviewers is always
> > available on the conference homepage (and the list of sub-reviewers is
> > sometimes also added in the proceedings). If you spent some time in
> > the field you can more less guess who does things similar to you and
> > can narrow down the list of your potential reviewers (so its very much
> > like with guessing the paper authors).
> >
> > Best,
> >
> > Michal
> >
> > -----------------------------------
> > Od: Anil Singh <anil.phdcl at gmail.com>
> > Kopia dla: corpora at uib.no, Yorick Wilks <Y.Wilks at dcs.shef.ac.uk>
> > Do: Yassine Benajiba <benajibayassine at gmail.com>
> > Data: Sat, 1 Oct 2011 16:02:56 +0530
> > Temat: Re: [Corpora-List] why LREC2012 NOT blind-reviewed?
> >
> > As some academicians/researchers I have respect for have expressed
> > opinions which I believe to be right and which I have been writing
> > about on my blog and elsewhere (apart from arguing for them in
> > conversations), I dare to chip in and say my bit.
> >
> > When I had entered this area and was thinking of submitting my first
> > paper (around 2003-2004) and I found out that reviewing will be blind,
> > I was delighted. I was a nobody (a graduate student) from a developing
> > country (India, but not even from one of the IITs and not with a very
> > good pedigree), I thought double blind reviewing will be definitely
> > more fair for people like me.
> >
> > My experience since then has completely disabused me of that naive
> idea.
> > While it may not be possible to exactly identify the author(s) of the
> > paper, one does get enough information (and meta-information) that is
> > more than enough to trigger all the prejudices, biases etc. that blind
> > reviewing is supposed to be an antidote against. This happens in
> > almost all the cases. Needless to add that there can be exceptions.
> >
> > You can, of course, give numerous counter-examples from cases where no
> > bias or prejudice is likely anyway or is very unlikely. But those
> > examples are not the ones that matter here.
> >
> > As far as I am concerned, if you can just identify the fact that the
> > author is from India, that alone removes at least half of the supposed
> > effectiveness of the idea of double blind reviewing. And if you work
> > on Indian languages and do certain kind of work, it's a no-brainer.
> >
> > Then there can be things like whether the author is just a student or
> > an established researcher, whether the project is funded or
> > non-funded, whether the language is that of a native speaker or not
> > etc. These are the very things that double blind reviewing is supposed
> > to guard against, but it simply can't. It just can't and I am sorry
> > that it can't. Theoretically the idea still appeals to me, but may be
> > like many other theoretically good things, it is not practically
> implementable.
> >
> > I especially like Yorick's comment about undignified gymnastics that
> > one is required to perform to hide one's identity. It even lowers the
> > academic quality of the paper quite often because you can't add
> > information that is very relevant. And I am totally in favour of the
> > reviewer taking responsibility for his comments. I have a corpus of
> > reviews and some of the comments simply make one embarrassed that
> > academicians (which one is
> > too) can behave like that -- and that too in writing.
> >
> > One of the things that has always left me wondering (to put it
> > lightly) is the fact that the conduct of academicians during the
> > actual meetings, i.e., paper presentations, panel discussions etc. is
> > so exceedingly civilized (for want of a better word) that I sometimes
> > feel out of place there (coming from a chaotic third world country and
> > being disordered personally). But a lot of the same academicians, when
> > they blind-review a paper, behave like bullies, vigilantes or just
> plain hooligans.
> > Fortunately, their number is still a minority.
> >
> > Of course, like everyone else, I have received wonderful (even if very
> > critical) reviews. But that can happen even with non-blind reviewing.
> > Just read literary supplements of papers that take literature
> seriously.
> >
> > To conclude, I would just say that if for nothing else, at least to
> > maintain the basic dignity of the academic community and of individual
> > academicians, it would be best if we switch to a reviewing process
> > that does not pretend to be blind and where reviewers take
> > responsibility for their comments.
> >
> > I am agnostic about whether extended abstracts should be reviewed or
> > full papers. Both seem to have their merits. For a conference like
> > LREC, extended abstracts do seem better to me, though I won't fight
> > for that (borrowing a phrase from review forms).
> >
> > I hope am not doing anything wrong by adding this link here:
> >
> > http://reviewscontd.org/
> >
> >
> >
> >   On Sat, Oct 1, 2011 at 6:28 AM, Yassine Benajiba
> > <benajibayassine at gmail.com> wrote:
> >   Hi everyone,
> >
> > I say let's judge the conference by the results. LREC is an awesome
> > conference constantly improving year after year. Even though it would
> > be great if somebody from the organizing committee could join this
> > conversation and tell us a bit more about the reasons.
> >
> > Best,
> >
> > --Yassine.
> >
> > On Fri, Sep 30, 2011 at 11:31 PM, Eric Ringger <ringger at cs.byu.edu>
> wrote:
> > Thanks to all for the open discussion.
> >
> > Graeme’s reason (1)(a) – the impact on merit review – is for me the
> > strongest reason to encourage LREC to move away from reviewing
> > extended abstracts and toward reviewing full papers.
> >
> > Best,
> > --Eric
> >
> > From: Graeme Hirst [mailto:gh at cs.toronto.edu]
> > Sent: Friday, September 30, 2011 11:02 AM
> > To: Yorick Wilks
> > Cc: Eric Ringger; corpora at uib.no
> > Subject: Re: [Corpora-List] why LREC2012 NOT blind-reviewed?
> >
> > Yorick,
> >
> > (1)  Whether a conference is reviewed by abstract or by full paper
> > makes an enormous difference:
> >
> >     (a) to merit, as perceived by tenure committees, granting
> > agencies, and others, who count only fully peer-reviewed papers.
> >     (b) to funding for travel.  Right now, one of my colleagues has
> > the problem that he cannot be funded to travel to give a paper at LREC
> > because it isn't a fully-reviewed conference, so he doesn't even bother
> submitting.
> >
> > You might say that these situations aren't desirable, but they are
> > nonetheless reality right now.
> >
> > (2)  I wonder how you are so sure that you almost invariably identify
> > the author of an anonymous paper correctly.  If the paper is not
> > ultimately accepted at the conference, which is 60 to 80% of them at
> > ACL and COLING conferences, you will never find out who the authors
> > actually are.  I've certainly guessed wrongly in the past.  And in my
> > own papers, I often throw in "hidden signals" to deceive the reviewers.
> >
> > (3)  I think Eric Ringger is 100% right about LREC.  As you say,
> > LREC's reputation and quality have grown, and for that reason it has
> > to start acting like a grown-up conference.
> >
> > Regards,
> > Graeme
> >
> > --
> > ::::  Graeme Hirst
> > ::::  University of Toronto * Department of Computer Science
> >
> >   On 2011-09-30, at 11:27, Yorick Wilks wrote:
> >
> >
> > I disagree strongly. I dont see why all conferences should be exactly
> > like all others. Extended abstracts are less of a burden on busy
> > academics --both as writers and reviewers----and there is no evidence
> > they lower the final quality; COLING used to do this and I am sorry it
> > changed. The whole blind-review business is a huge nonsense: I rarely
> > meet a paper to review where i cannot identify the authors from a
> > careful trawl of hidden signals and the references. Trying to make a
> > paper genuinely anonymous is almost impossible if one has a body of
> > past work and publication to link it to---the mental gymnastics
> required are undignified and best avoided.
> > LRECs reputation has grown steadily and it will be the quality of its
> > papers that sustain it--there is no evidence at all anonymity would
> > improve matters in the least. if it ain't broke........
> > Yorick Wilks
> >
> >
> > On 30 Sep 2011, at 16:02, Eric Ringger wrote:
> >
> >
> > Greetings.
> >
> > LREC has been operated in this manner since its inception.  Personally
> > and for the sake of LREC’s reputation, I would like to see the
> > reviewing process for LREC upgraded to double-blind review.
> >
> > I believe that LREC fills a couple of important niches: its focus on
> > language resources and evaluation/validation is important and not well
> > served elsewhere, and it does a good job of bringing a large, diverse
> > group together.  (I should add that it does a good job of selecting
> > attractive venues as well!)  If implemented well, I believe that
> > double-blind review would not detract from the primary objectives of
> > the conference but would refine the quality of the program and improve
> > the reputation of the venue.  I have said as much in private feedback
> > after past LRECs.
> >
> > I also think it is time for LREC to move up from reviewing extended
> > abstracts to reviewing full papers.
> >
> > Regards,
> > --Eric
> >
> >
> > From: corpora-bounces at uib.no [mailto:corpora-bounces at uib.no] On Behalf
> > Of Isabella Chiari
> > Sent: Friday, September 30, 2011 8:45 AM
> > To: corpora at uib.no
> > Subject: [Corpora-List] why LREC2012 NOT blind-reviewed?
> >
> > Dear Corpora members,
> > I just noticed that the LREC2012 call specifies that submissions are
> > NOT anonymous and there will not be blind-reviewing.
> >
> > Does anyone know why? Which is the policy under this decision?
> > Best regards,
> > Isabella Chiari
> >
> >
> >
> >   _______________________________________________
> >   UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
> >   Corpora mailing list
> > Corpora at uib.no
> > http://mailman.uib.no/listinfo/corpora
> >
> >
> >
> > _______________________________________________
> >   UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
> >   Corpora mailing list
> > Corpora at uib.no
> > http://mailman.uib.no/listinfo/corpora
> >
> > _______________________________________________
> > UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
> > Corpora mailing list
> > Corpora at uib.no
> > http://mailman.uib.no/listinfo/corpora
> >
> 
> _______________________________________________
> UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
> Corpora mailing list
> Corpora at uib.no
> http://mailman.uib.no/listinfo/corpora

_______________________________________________
UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
Corpora mailing list
Corpora at uib.no
http://mailman.uib.no/listinfo/corpora