[Corpora-List] why LREC2012 NOT blind-reviewed?
Geoffrey Williams
geoffrey.williams at wanadoo.fr
Sat Oct 1 14:21:57 UTC 2011
Now I am pushed to reply.
I am a frequent blind reviewer. I see an amazing amount of rubbish which
ought never to have been submitted to a journal. I follow my doctoral
students closely and help them with their papers, which is why I get
angry when inundated by bad papers which a supervisor should have
stopped. If the Humanities is in any danger, it is because of a
caring-sharing tendency that refuses to demand clearly structured and
thought out work. A bad paper takes much more time to review than a good
one.
Blind review may not be the answer to all, but it does keep journals at
a decent standard. As a reviewer, I always send detailed comments, and
have seen many bad papers become good ones. Rather than complaining
about reviewers, it might be worth looking at appauling supervisors who
do not fulful their duty in guiding students to a good publication.
It is not for me to name and shame, but I can guess the universities
that are the source of some papers, and thus the supervisors who are
letting rubblish through.
I once got caught out when the writer took an educated guess at who I
was. I never admitted to being a reviewer, but was clearly expected to
rewrite his thesis. Doing a review job for free is one thing, doing the
job of bad supervisors is another.
I essentially review for two journals and several major conferences, I
am proud of the quality standard of all, and it comes from blind review
and the excellent review editors with whom I work.
We have all come up against pig ignorant (an insult as pigs are very
pleasant and intelligent animals, and I would rather share a pint with a
pig than many colleagues), but the quality standard is maintained by the
great majority who give their time freely to help guide others.
Oversized conferences encourage bad reviewing as time is of the essence,
and the quaity variable in the extreme. Do not tarnish all events with
the same briush.
Do not let the blind lead the blind reveiwers astray, and two is always
better than one.
Geoffrey
Having eyes, see ye not? and having ears, hear ye not? and do ye not
remember? Mark 8. 18. King James, of course.
Le 01/10/2011 12:32, Anil Singh a écrit :
> As some academicians/researchers I have respect for have expressed
> opinions which I believe to be right and which I have been writing
> about on my blog and elsewhere (apart from arguing for them in
> conversations), I dare to chip in and say my bit.
>
> When I had entered this area and was thinking of submitting my first
> paper (around 2003-2004) and I found out that reviewing will be blind,
> I was delighted. I was a nobody (a graduate student) from a developing
> country (India, but not even from one of the IITs and not with a very
> good pedigree), I thought double blind reviewing will be definitely
> more fair for people like me.
>
> My experience since then has completely disabused me of that naive
> idea. While it may not be possible to exactly identify the author(s)
> of the paper, one does get enough information (and meta-information)
> that is more than enough to trigger all the prejudices, biases etc.
> that blind reviewing is supposed to be an antidote against. This
> happens in almost all the cases. Needless to add that there can be
> exceptions.
>
> You can, of course, give numerous counter-examples from cases where no
> bias or prejudice is likely anyway or is very unlikely. But those
> examples are not the ones that matter here.
>
> As far as I am concerned, if you can just identify the fact that the
> author is from India, that alone removes at least half of the supposed
> effectiveness of the idea of double blind reviewing. And if you work
> on Indian languages and do certain kind of work, it's a no-brainer.
>
> Then there can be things like whether the author is just a student or
> an established researcher, whether the project is funded or
> non-funded, whether the language is that of a native speaker or not
> etc. These are the very things that double blind reviewing is supposed
> to guard against, but it simply can't. It just can't and I am sorry
> that it can't. Theoretically the idea still appeals to me, but may be
> like many other theoretically good things, it is not practically
> implementable.
>
> I especially like Yorick's comment about undignified gymnastics that
> one is required to perform to hide one's identity. It even lowers the
> academic quality of the paper quite often because you can't add
> information that is very relevant. And I am totally in favour of the
> reviewer taking responsibility for his comments. I have a corpus of
> reviews and some of the comments simply make one embarrassed that
> academicians (which one is too) can behave like that -- and that too
> in writing.
>
> One of the things that has always left me wondering (to put it
> lightly) is the fact that the conduct of academicians during the
> actual meetings, i.e., paper presentations, panel discussions etc. is
> so exceedingly civilized (for want of a better word) that I sometimes
> feel out of place there (coming from a chaotic third world country and
> being disordered personally). But a lot of the same academicians, when
> they blind-review a paper, behave like bullies, vigilantes or just
> plain hooligans. Fortunately, their number is still a minority.
>
> Of course, like everyone else, I have received wonderful (even if very
> critical) reviews. But that can happen even with non-blind reviewing.
> Just read literary supplements of papers that take literature seriously.
>
> To conclude, I would just say that if for nothing else, at least to
> maintain the basic dignity of the academic community and of individual
> academicians, it would be best if we switch to a reviewing process
> that does not pretend to be blind and where reviewers take
> responsibility for their comments.
>
> I am agnostic about whether extended abstracts should be reviewed or
> full papers. Both seem to have their merits. For a conference like
> LREC, extended abstracts do seem better to me, though I won't fight
> for that (borrowing a phrase from review forms).
>
> I hope am not doing anything wrong by adding this link here:
>
> http://reviewscontd.org/
>
>
>
> On Sat, Oct 1, 2011 at 6:28 AM, Yassine Benajiba
> <benajibayassine at gmail.com <mailto:benajibayassine at gmail.com>> wrote:
>
> Hi everyone,
>
> I say let's judge the conference by the results. LREC is an
> awesome conference constantly improving year after year. Even
> though it would be great if somebody from the organizing committee
> could join this conversation and tell us a bit more about the reasons.
>
> Best,
>
> --Yassine.
>
> On Fri, Sep 30, 2011 at 11:31 PM, Eric Ringger <ringger at cs.byu.edu
> <mailto:ringger at cs.byu.edu>> wrote:
>
> Thanks to all for the open discussion.
>
> Graeme’s reason (1)(a) – the impact on merit review – is for
> me the strongest reason to encourage LREC to move away from
> reviewing extended abstracts and toward reviewing full papers.
>
> Best,
>
> --Eric
>
> *From:*Graeme Hirst [mailto:gh at cs.toronto.edu
> <mailto:gh at cs.toronto.edu>]
> *Sent:* Friday, September 30, 2011 11:02 AM
> *To:* Yorick Wilks
> *Cc:* Eric Ringger; corpora at uib.no <mailto:corpora at uib.no>
> *Subject:* Re: [Corpora-List] why LREC2012 NOT blind-reviewed?
>
> Yorick,
>
> (1) Whether a conference is reviewed by abstract or by full
> paper makes an enormous difference:
>
> (a) to merit, as perceived by tenure committees, granting
> agencies, and others, who count only fully peer-reviewed papers.
>
> (b) to funding for travel. Right now, one of my colleagues
> has the problem that he cannot be funded to travel to give a
> paper at LREC because it isn't a fully-reviewed conference, so
> he doesn't even bother submitting.
>
> You might say that these situations aren't desirable, but they
> are nonetheless reality right now.
>
> (2) I wonder how you are so sure that you almost invariably
> identify the author of an anonymous paper correctly. If the
> paper is not ultimately accepted at the conference, which is
> 60 to 80% of them at ACL and COLING conferences, you will
> never find out who the authors actually are. I've certainly
> guessed wrongly in the past. And in my own papers, I often
> throw in "hidden signals" to deceive the reviewers.
>
> (3) I think Eric Ringger is 100% right about LREC. As you
> say, LREC's reputation and quality have grown, and for that
> reason it has to start acting like a grown-up conference.
>
> Regards,
>
> Graeme
>
> --
> :::: Graeme Hirst
> :::: University of Toronto * Department of Computer Science
>
> On 2011-09-30, at 11:27, Yorick Wilks wrote:
>
>
>
> I disagree strongly. I dont see why all conferences should be
> exactly like all others. Extended abstracts are less of a
> burden on busy academics --both as writers and
> reviewers----and there is no evidence they lower the final
> quality; COLING used to do this and I am sorry it changed. The
> whole blind-review business is a huge nonsense: I rarely meet
> a paper to review where i cannot identify the authors from a
> careful trawl of hidden signals and the references. Trying to
> make a paper genuinely anonymous is almost impossible if one
> has a body of past work and publication to link it to---the
> mental gymnastics required are undignified and best avoided.
> LRECs reputation has grown steadily and it will be the quality
> of its papers that sustain it--there is no evidence at all
> anonymity would improve matters in the least. if it ain't
> broke........
>
> Yorick Wilks
>
> On 30 Sep 2011, at 16:02, Eric Ringger wrote:
>
>
>
> Greetings.
>
> LREC has been operated in this manner since its inception.
> Personally and for the sake of LREC’s reputation, I would like
> to see the reviewing process for LREC upgraded to double-blind
> review.
>
> I believe that LREC fills a couple of important niches: its
> focus on language resources and evaluation/validation is
> important and not well served elsewhere, and it does a good
> job of bringing a large, diverse group together. (I should
> add that it does a good job of selecting attractive venues as
> well!) If implemented well, I believe that double-blind
> review would not detract from the primary objectives of the
> conference but would refine the quality of the program and
> improve the reputation of the venue. I have said as much in
> private feedback after past LRECs.
>
> I also think it is time for LREC to move up from reviewing
> extended abstracts to reviewing full papers.
>
> Regards,
>
> --Eric
>
> *From:*corpora-bounces at uib.no
> <mailto:corpora-bounces at uib.no>[mailto:corpora-bounces at uib.no]
> <mailto:[mailto:corpora-bounces at uib.no]>*On Behalf Of*Isabella
> Chiari
> *Sent:*Friday, September 30, 2011 8:45 AM
> *To:*corpora at uib.no <mailto:corpora at uib.no>
> *Subject:*[Corpora-List] why LREC2012 NOT blind-reviewed?
>
> Dear Corpora members,
>
> I just noticed that the LREC2012 call specifies that
> submissions are NOT anonymous and there will not be
> blind-reviewing.
>
> Does anyone know why? Which is the policy under this decision?
>
> Best regards,
>
> Isabella Chiari
>
>
> _______________________________________________
> UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
> Corpora mailing list
> Corpora at uib.no <mailto:Corpora at uib.no>
> http://mailman.uib.no/listinfo/corpora
>
>
>
> _______________________________________________
> UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
> Corpora mailing list
> Corpora at uib.no <mailto:Corpora at uib.no>
> http://mailman.uib.no/listinfo/corpora
>
>
>
> _______________________________________________
> UNSUBSCRIBE from this page:http://mailman.uib.no/options/corpora
> Corpora mailing list
> Corpora at uib.no
> http://mailman.uib.no/listinfo/corpora
--
**
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listserv.linguistlist.org/pipermail/corpora/attachments/20111001/1d307b93/attachment.htm>
-------------- next part --------------
_______________________________________________
UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
Corpora mailing list
Corpora at uib.no
http://mailman.uib.no/listinfo/corpora
More information about the Corpora
mailing list