[Corpora-List] Blind reviewing

Wed Oct 12 17:55:03 UTC 2011

Dear all,

I've been reading this thread with interest, and I think there is one
advantage of author-blind reviewing which has not come up: I think
part of the reason that author-blind reviewing is more associated
with scientific venues (as opposed to Anil's example of book reviews
in the press) is the issue of *positive* prejudice.  The peer-review
system is supposed to be part of scientific process; author-blind reviewing
means (if it's truly author-blind) that ideas are more likely to be evaluated
on their own merits, rather than on the basis of who said them.

Of course, we can (and should) consider how all of these considerations
balance against each other, and which carry more weight in which
contexts (types of conferences, journals), while also recognizing
that it is not always possible to be truly author-blind (I certainly have
that problem when publishing on long-standing research projects!).

And the issue of author-blind reviewing is independent (as others
have rightly noted) of the issue of reviewer-blind reviewing and
accountability for reviewers.  I believe we need more accountability
for reviewers, though I'm not yet convinced that removing anonymity
of the reviewers is the best way to go about that.

Emily

-- 
Emily M. Bender
Associate Professor
Department of Linguistics
Check out CLMA on facebook! http://www.facebook.com/uwclma

On Wed, Oct 12, 2011 at 10:47 AM, Anil Singh <anil.phdcl at gmail.com> wrote:
> As I have made many comments on this thread, I will clearly state my
> position. In most contexts, I would prefer the three kinds of reviewing
> systems as below (in decreasing order of preference, i.e., the first one
> being the most preferable):
> 1. Open review system: The reviewer and the author both know each other's
> identity
> 2. Single blind system: Reviewer knows who the author is, but the author
> doesn't know who the reviewer is
> 3. Double blind system: Neither the author nor the reviewer knows the
> other's identity
> I have tried to give arguments in support of my position. I will skip
> repeating them (though I like repetition as a stylistic device).
> I think it's a mistake to think that system-1 is not practical at all. The
> fact is, most of the peer reviews in the world out there work on this very
> basis. Consider book reviews in the press, in journals, wherever. Even a few
> scientific journals/conferences are using open peer review. The evidence, so
> far as it goes, scientifically speaking, is not conclusive either way. It is
> not even very heavily in favour of the one (blind) or the other (open).
> System-2 is better than system-3, primarily because of the problems in and
> impracticality of hiding one's identity (often needs undigified gymnastic,
> as Yorick expressed it: And it's not simply a matter of making it "as Smith
> (2008) showed").
> The only strong argument in favour of system-3 (based on my knowledge and
> experienced) is the possibility of retaliation. I would agree that the
> chances of that happening would be higher in system-1 than in system-2 or
> system-3, but I am pretty sure that it can happen even in these two systems.
> The reason is that often the author can guess who the reviewer was. But with
> system-1 the process is transparent, so that if some such thing happens, the
> person at the receiving end can at least say that this has happened and
> possibly seek some sort of remedy. If those who retaliate for such reasons
> know that their actions (which are very seriously unethical and immoral, and
> I don't see why they shouldn't be given as much consideration as issues like
> plagiarism) can be brought to public attention, they might think twice
> before indulging in them. As it happens, with blind reviewing (systems 2 and
> 3), the person (the reviewer) at the receiving end can't even say that any
> such thing has happened, because the immediate dismissive answer would be
> that "our system ensures that the identities are not revealed" and you have
> run into a wall. A similar thing applies when a reviewer unfairly rejects a
> paper (the standard method now seems to be to ensure the paper is killed, to
> borrow Church's memorable words, by giving it a score of 1, irrespective of
> anything in the paper and sometimes even irrespective of the same reviewer's
> comments for the same paper) because he has guessed the author's identity or
> has narrowed it down sufficiently. Here I will have to repeat that this can
> happen because of various kinds of prejudices (e.g. the
> formalism/paradigm/framework that enrages him, to give an example of a
> purely technical kind of prejudice, even though the issue of the validity of
> that formalism/paradigm/framework is still very much open).
>
> On Wed, Oct 12, 2011 at 10:18 PM, Laurence Anthony <anthony0122 at gmail.com>
> wrote:
>>
>> On Thu, Oct 13, 2011 at 1:23 AM, Yorick Wilks <Y.Wilks at dcs.shef.ac.uk>
>> wrote:
>>>
>>> Thanks, I remember the details. The discussion has gone many ways, some
>>> of them arguing the (de)merits of  author-blind --as well as reviewer-blind
>>> ---systems. The starting point was LREC and the author-blind system.  Much
>>> later, you wrote, after I used the phrase "both systems":
>>> ".....what exactly is the alternative system to blind reviewing that is
>>> being referred to in the phrase "both systems". Obviously, "against blind
>>> reviewing" is not a system in itself. Am I correct in assuming that the
>>> 'alternative system' being proposed on this list is simply an open one where
>>> both reviewers and authors know each others' names? "
>>> My "both systems" referred, as I thought was clear in the context I wrote
>>> it, to author-blind and non-blind systems---ACL being like the former and
>>> LREC the latter (COLING has oscillated, if memory serves). So no, the
>>> opposites are those just listed. Does that clear it up?
>>> YW
>>
>> Sorry, I'm still confused. I think ACL uses a double-blind system (authors
>> and reviewers don't know who the other is). See here:
>>
>> http://www.aclweb.org/archive/policies/current/program-committee-guide.html
>> LREC uses an single-blind system (the reviewer knows the author but the
>> author doesn't know the reviewer). See here:
>> http://www.lrec-conf.org/lrec2012/?Abstract-for-Oral-or-Poster
>> In view of earlier comments about reviewers needing to reveal their
>> identity, neither ACL nor LREC adopt such a policy. In fact, the LREC policy
>> in effect gives even more power to the reviewer than a double-blind policy.
>> Is this what you were supporting when you wrote,  "The whole blind-review
>> business is a huge nonsense...LRECs reputation has grown steadily and it
>> will be the quality of its papers that sustain it--there is no evidence at
>> all anonymity would improve matters in the least. if it ain't broke........"
>> Laurence.
>> (p.s. If it's just me that's confused, feel free to ignore me!)
>>
>>
>>
>>
>> _______________________________________________
>> UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
>> Corpora mailing list
>> Corpora at uib.no
>> http://mailman.uib.no/listinfo/corpora
>>
>
>
> _______________________________________________
> UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
> Corpora mailing list
> Corpora at uib.no
> http://mailman.uib.no/listinfo/corpora
>
>

_______________________________________________
UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
Corpora mailing list
Corpora at uib.no
http://mailman.uib.no/listinfo/corpora