[Corpora-List] why LREC2012 NOT blind-reviewed?
Mitkov, Ruslan
R.Mitkov at wlv.ac.uk
Sun Oct 2 17:35:55 UTC 2011
Dear all,
Let me make a related point. There are many reviewers who instead of providing constructive and encouraging comments,
seem to enjoy criticising a specific work, somehow trying to show their superiority. Sometimes this is done in an arrogant way.
I somehow find such reviews not very helpful. I think such comments could dishearten an author and push him/her away from further work.
Personally whether in a role of a conference programme chair or journal editor,
I remove comments which are too negative and even on occasions bullying, from the feedback sent to the authors
I hope most of you share my views.
Regards, R.
-----Original Message-----
From: corpora-bounces at uib.no [mailto:corpora-bounces at uib.no] On Behalf Of Michael Zock
Sent: 02 October 2011 18:11
To: Laurence Anthony
Cc: corpora at uib.no
Subject: Re: [Corpora-List] why LREC2012 NOT blind-reviewed?
Hello Laurence,
having been (like most of you) on both sides, I'm afraid to say that my own experience does not quite correspond to Laurence's. Hence, I cannot fully share his comments and optimisms, eventhough I'd love to.
True, there are many people who make a real effort to be useful, but there is also amazingly large proportion of reviewers who do not spend the time it takes to read carefully a paper and to provide the feedback that authors deserve, feedback on which decisions are made. You'd be surprised about the kind of reviews I've gotten as organizer of conferences and workshops, and the same holds for other conferences, including major ones where we can see the comments made by our colleages. Sometimes it's really a shame.
Also, how many authors will write to the organizers of a conference challenging the conclusions of a reviewer? I think most authors (especially younger ones) will refrain from it, for various reasons.
While I don't know what the solution to all this is, I do believe that it is good that peers can see the reviews written by a colleague. This may prevent some reviewers to come up a with a 's...y' (sorry) 2 lines comment, saying :
- poor English
- lack of references
and, that was it. I have received quite a few of such reviews, sometimes from very well known 'peers', and this after having them reminded several times that the reviews were overdue.
Hoping that your experience is closer to Laurence's
best
Michael Zock (at least, the sun keeps shining over Marseille)
> Just to follow up on Michal's comments,
>
> I'm quite surprised how much animosity there seems to be against
> reviewers. In this thread people have said; "the reviewers make
> linguistics not brilliant", "when they blind-review a paper, behave
> like bullies, vigilantes or just plain hooligans."
> "it would be best if we switch to a reviewing process ... where
> reviewers take responsibility for their comments."
>
> It seems to me that reviewers are just peers who have been asked to
> volunteer their service to a conference/journal by the editor/chair.
> They accept the request because they feel it helps the
> conference/journal in some way, and they attempt to the best of their
> abilities to provide a balanced judgement on the merits of the papers
> they are reviewing. For a good conference/journal, usually two
> reviewers are asked to submit their opinions, and almost always, the
> editor/chair will have the final say on whether or not to accept the
> reviews or seek advice of a third or fourth reviewer. Obviously, there
> will be times when the reviews will be of lower quality than desired.
> But the entire process is designed to avoid these making an impact on
> the final decision to accept/reject a paper.
>
> Of course, not all reviewers will be "brilliant". And, few would ever
> claim to be so. They are *peers*, doing peer reviews. If the author
> does not accept the review they almost always have the ability to
> respond to the review, explaining where they see its faults. And, if
> reviewers acted like "bullies, vigilantes or just plain hooligans",
> editors/chairs would very quickly stop asking them to review work.
> Finally, I'm sure that most reviewers consider very carefully the
> responsibility they have, and they certainly take responsibility for
> their comments because they submit them openly to the journal
> editor/conference chair (even if their names are ultimately hidden to
> the author). They do not hide behind user names, IDs, or avatars. As
> Michal writes, they allow their names to appear on the
> conference/journal website or in the proceedings.
>
> Saying all this, if there is such animosity to reviewers and most
> peers feel they are unintelligent, irresponsible bullies, vigilantes,
> and hooligans, I'm sure reviewers would be quite happy to stop
> providing their *voluntary service*. But, I would imagine that the
> loss to the conferences/journals would be much greater than the gains
> they would make.
>
> On the point of the double blind review system, it is certainly not
> perfect. But, arguing for a completely open system that invites all
> kinds of biases, simply because the current system does not work
> perfectly, seems rather odd. Are people arguing that allowing both
> authors and reviewers to know the names of each other will lead to a
> fairer system? If so, is there any evidence for this? I haven't
> checked the history but would expect that the double blind review
> system was introduced *precisely because* of the problems of an open
> system.
>
> I am not sure why LREC2012 has made the decision it has. I would be
> very interested to hear how their revised system will improve the
> conference.
>
> Laurence Anthony
> (taking full responsibility for my comments even if they are not the
> most brilliant in the world!)
>
>
>
>
>
>
>
>
>
> On Sun, Oct 2, 2011 at 11:23 PM, Michal Ptaszynski
> <michal.ptaszynski at gmail.com> wrote:
>> Just two humble cents from me.
>> I'm sure most of us, even those strongly against double blind
>> reviewing, remember or at least realize why this process was introduced.
>> The fact is many people DO send many crappy papers, often re-sending
>> exactly the same crap simultaneously to many conferences. If blind
>> reviewing can flush at least some of this, and improve the world of
>> science even in 1/100 of a percent, I wouldn't vote against it.
>>
>> Although what I would really like to have (and I think it is
>> feasible) is a system telling me more-less if the paper is a
>> candidate for a crap or a reliable piece of work.
>>
>> Also a word about the reviewers being hidden behind the curtain. This
>> is not exactly true, since the list of all reviewers is always
>> available on the conference homepage (and the list of sub-reviewers
>> is sometimes also added in the proceedings). If you spent some time
>> in the field you can more less guess who does things similar to you
>> and can narrow down the list of your potential reviewers (so its very
>> much like with guessing the paper authors).
>>
>> Best,
>>
>> Michal
>>
>> -----------------------------------
>> Od: Anil Singh<anil.phdcl at gmail.com>
>> Kopia dla: corpora at uib.no, Yorick Wilks<Y.Wilks at dcs.shef.ac.uk>
>> Do: Yassine Benajiba<benajibayassine at gmail.com>
>> Data: Sat, 1 Oct 2011 16:02:56 +0530
>> Temat: Re: [Corpora-List] why LREC2012 NOT blind-reviewed?
>>
>> As some academicians/researchers I have respect for have expressed
>> opinions which I believe to be right and which I have been writing
>> about on my blog and elsewhere (apart from arguing for them in
>> conversations), I dare to chip in and say my bit.
>>
>> When I had entered this area and was thinking of submitting my first
>> paper (around 2003-2004) and I found out that reviewing will be
>> blind, I was delighted. I was a nobody (a graduate student) from a
>> developing country (India, but not even from one of the IITs and not
>> with a very good pedigree), I thought double blind reviewing will be
>> definitely more fair for people like me.
>>
>> My experience since then has completely disabused me of that naive idea.
>> While it may not be possible to exactly identify the author(s) of the
>> paper, one does get enough information (and meta-information) that is
>> more than enough to trigger all the prejudices, biases etc. that
>> blind reviewing is supposed to be an antidote against. This happens
>> in almost all the cases. Needless to add that there can be exceptions.
>>
>> You can, of course, give numerous counter-examples from cases where
>> no bias or prejudice is likely anyway or is very unlikely. But those
>> examples are not the ones that matter here.
>>
>> As far as I am concerned, if you can just identify the fact that the
>> author is from India, that alone removes at least half of the
>> supposed effectiveness of the idea of double blind reviewing. And if
>> you work on Indian languages and do certain kind of work, it's a no-brainer.
>>
>> Then there can be things like whether the author is just a student or
>> an established researcher, whether the project is funded or
>> non-funded, whether the language is that of a native speaker or not
>> etc. These are the very things that double blind reviewing is
>> supposed to guard against, but it simply can't. It just can't and I
>> am sorry that it can't. Theoretically the idea still appeals to me,
>> but may be like many other theoretically good things, it is not practically implementable.
>>
>> I especially like Yorick's comment about undignified gymnastics that
>> one is required to perform to hide one's identity. It even lowers the
>> academic quality of the paper quite often because you can't add
>> information that is very relevant. And I am totally in favour of the
>> reviewer taking responsibility for his comments. I have a corpus of
>> reviews and some of the comments simply make one embarrassed that
>> academicians (which one is
>> too) can behave like that -- and that too in writing.
>>
>> One of the things that has always left me wondering (to put it
>> lightly) is the fact that the conduct of academicians during the
>> actual meetings, i.e., paper presentations, panel discussions etc. is
>> so exceedingly civilized (for want of a better word) that I sometimes
>> feel out of place there (coming from a chaotic third world country
>> and being disordered personally). But a lot of the same academicians,
>> when they blind-review a paper, behave like bullies, vigilantes or just plain hooligans.
>> Fortunately, their number is still a minority.
>>
>> Of course, like everyone else, I have received wonderful (even if
>> very
>> critical) reviews. But that can happen even with non-blind reviewing.
>> Just read literary supplements of papers that take literature seriously.
>>
>> To conclude, I would just say that if for nothing else, at least to
>> maintain the basic dignity of the academic community and of
>> individual academicians, it would be best if we switch to a reviewing
>> process that does not pretend to be blind and where reviewers take
>> responsibility for their comments.
>>
>> I am agnostic about whether extended abstracts should be reviewed or
>> full papers. Both seem to have their merits. For a conference like
>> LREC, extended abstracts do seem better to me, though I won't fight
>> for that (borrowing a phrase from review forms).
>>
>> I hope am not doing anything wrong by adding this link here:
>>
>> http://reviewscontd.org/
>>
>>
>>
>> On Sat, Oct 1, 2011 at 6:28 AM, Yassine Benajiba
>> <benajibayassine at gmail.com> wrote:
>> Hi everyone,
>>
>> I say let's judge the conference by the results. LREC is an awesome
>> conference constantly improving year after year. Even though it would
>> be great if somebody from the organizing committee could join this
>> conversation and tell us a bit more about the reasons.
>>
>> Best,
>>
>> --Yassine.
>>
>> On Fri, Sep 30, 2011 at 11:31 PM, Eric Ringger<ringger at cs.byu.edu> wrote:
>> Thanks to all for the open discussion.
>>
>> Graeme's reason (1)(a) - the impact on merit review - is for me the
>> strongest reason to encourage LREC to move away from reviewing
>> extended abstracts and toward reviewing full papers.
>>
>> Best,
>> --Eric
>>
>> From: Graeme Hirst [mailto:gh at cs.toronto.edu]
>> Sent: Friday, September 30, 2011 11:02 AM
>> To: Yorick Wilks
>> Cc: Eric Ringger; corpora at uib.no
>> Subject: Re: [Corpora-List] why LREC2012 NOT blind-reviewed?
>>
>> Yorick,
>>
>> (1) Whether a conference is reviewed by abstract or by full paper
>> makes an enormous difference:
>>
>> (a) to merit, as perceived by tenure committees, granting
>> agencies, and others, who count only fully peer-reviewed papers.
>> (b) to funding for travel. Right now, one of my colleagues has
>> the problem that he cannot be funded to travel to give a paper at
>> LREC because it isn't a fully-reviewed conference, so he doesn't even bother submitting.
>>
>> You might say that these situations aren't desirable, but they are
>> nonetheless reality right now.
>>
>> (2) I wonder how you are so sure that you almost invariably identify
>> the author of an anonymous paper correctly. If the paper is not
>> ultimately accepted at the conference, which is 60 to 80% of them at
>> ACL and COLING conferences, you will never find out who the authors
>> actually are. I've certainly guessed wrongly in the past. And in my
>> own papers, I often throw in "hidden signals" to deceive the reviewers.
>>
>> (3) I think Eric Ringger is 100% right about LREC. As you say,
>> LREC's reputation and quality have grown, and for that reason it has
>> to start acting like a grown-up conference.
>>
>> Regards,
>> Graeme
>>
>> --
>> :::: Graeme Hirst
>> :::: University of Toronto * Department of Computer Science
>>
>> On 2011-09-30, at 11:27, Yorick Wilks wrote:
>>
>>
>> I disagree strongly. I dont see why all conferences should be exactly
>> like all others. Extended abstracts are less of a burden on busy
>> academics --both as writers and reviewers----and there is no evidence
>> they lower the final quality; COLING used to do this and I am sorry
>> it changed. The whole blind-review business is a huge nonsense: I
>> rarely meet a paper to review where i cannot identify the authors
>> from a careful trawl of hidden signals and the references. Trying to
>> make a paper genuinely anonymous is almost impossible if one has a
>> body of past work and publication to link it to---the mental gymnastics required are undignified and best avoided.
>> LRECs reputation has grown steadily and it will be the quality of its
>> papers that sustain it--there is no evidence at all anonymity would
>> improve matters in the least. if it ain't broke........
>> Yorick Wilks
>>
>>
>> On 30 Sep 2011, at 16:02, Eric Ringger wrote:
>>
>>
>> Greetings.
>>
>> LREC has been operated in this manner since its inception.
>> Personally and for the sake of LREC's reputation, I would like to see
>> the reviewing process for LREC upgraded to double-blind review.
>>
>> I believe that LREC fills a couple of important niches: its focus on
>> language resources and evaluation/validation is important and not
>> well served elsewhere, and it does a good job of bringing a large,
>> diverse group together. (I should add that it does a good job of
>> selecting attractive venues as well!) If implemented well, I believe
>> that double-blind review would not detract from the primary
>> objectives of the conference but would refine the quality of the
>> program and improve the reputation of the venue. I have said as much
>> in private feedback after past LRECs.
>>
>> I also think it is time for LREC to move up from reviewing extended
>> abstracts to reviewing full papers.
>>
>> Regards,
>> --Eric
>>
>>
>> From: corpora-bounces at uib.no [mailto:corpora-bounces at uib.no] On
>> Behalf Of Isabella Chiari
>> Sent: Friday, September 30, 2011 8:45 AM
>> To: corpora at uib.no
>> Subject: [Corpora-List] why LREC2012 NOT blind-reviewed?
>>
>> Dear Corpora members,
>> I just noticed that the LREC2012 call specifies that submissions are
>> NOT anonymous and there will not be blind-reviewing.
>>
>> Does anyone know why? Which is the policy under this decision?
>> Best regards,
>> Isabella Chiari
>>
>>
>>
>> _______________________________________________
>> UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
>> Corpora mailing list
>> Corpora at uib.no
>> http://mailman.uib.no/listinfo/corpora
>>
>>
>>
>> _______________________________________________
>> UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
>> Corpora mailing list
>> Corpora at uib.no
>> http://mailman.uib.no/listinfo/corpora
>>
>> _______________________________________________
>> UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
>> Corpora mailing list
>> Corpora at uib.no
>> http://mailman.uib.no/listinfo/corpora
>>
> _______________________________________________
> UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
> Corpora mailing list
> Corpora at uib.no
> http://mailman.uib.no/listinfo/corpora
>
--
------------------------------------
Michael ZOCK& Lih-Juang FANG ZOCK
83, bd du Redon, Bât. E - 8
F-13009 Marseille
Mail: zock at free.fr
Tel. +33 (0) 9.51.89.97.07
portable: +33 (0) 6.01.34.81.22
------------------------------------
_______________________________________________
UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
Corpora mailing list
Corpora at uib.no
http://mailman.uib.no/listinfo/corpora
--
Scanned by iCritical.
_______________________________________________
UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
Corpora mailing list
Corpora at uib.no
http://mailman.uib.no/listinfo/corpora
More information about the Corpora
mailing list