[Corpora-List] Corpora Digest, Vol 52, Issue 3 [was blind reviewing]

[Alon Lischinsky]

On 2011/10/11 Laurence Anthony <anthony0122 at gmail.com> wrote:

> Really? Why would you say that the reviewers' 'clients' are not usually in a
> position to retaliate? I review many papers and I'm sure many of my
> 'clients' are more well-known and influential than I am.

This has not been my experience. I review quite often as well,
although probably for a different set of journals, and it's quite
seldom that I find myself reviewing the work of a more senior scholar
(at least, judging from the articles I have positively reviewed, the
identity of whose authors I can identify by looking at the published
version; although possible in principle, it is unlikely that senior
scholars should be underrepresented in this set). Perhaps this is
something that differs across subfields.

> (This is under the assumption that neither side is blind).

My arguments have been mostly directed to the problems arising from
blinding the reviewer's identity. I have my doubts about the
effectiveness of author blinding, but in any case it does not raise
the questions I'm concerned with here.

> Many juries in many countries can give a verdict without the decision
> being unanimous.

"Many countries" is a somewhat specious claim, given that peer juries
are mostly restricted to common law jurisdictions, a clear minority
among legal systems. But my main point is that peer juries hardly
resemble peer reviews.

>> The reviewing process is, if anything, more similar to a
>> bench trial, where a single person is in charge of all decisions, and
>> no checks exist to limit their discretionary judgement.
>
> Why? In the reviewing process, multiple people give reviews on a paper.  If
> you are trying to argue that a single person makes all the decisions, that
> would be the Editor, who the author will know directly.

Some papers use multiple reviewers. Others don't. And even those that
require multiple reviews do not (a) allow the different reviewers to
discuss the case and seek a common opinion, as happens in jury trials;
(b) require enough reviews to ensure that personal biases regress to
the mean.

In my experience (and this is, of course, anecdotal evidence, but I
don't know of any systematic studies), the editor rarely has anything
to do with the review process (beyond, perhaps, an initial screening
of obviously rejectable papers). The decision of the reviewer(s) is
what determines acceptance. And I haven't found a single journal that
encourages authors to challenge the reviewers' decisions, or even
two-way communication during the review process.

In sum: unlike voters, or even jurors, who have only limited power
over the final result because of the (relatively) large number of
decision-makers, reviewers exert significant individual influence. I
believe that public scrutiny is beneficial in those cases. If a
reviewer is required to publicly stand by their words, the potential
for retaliation also becomes limited: an obviously biased or malicious
review would reflect poorly on the reviewer.

> As I wrote before, I just think it is strange that people are proposing
> dropping the blind-review process altogether just because of its
> limitations.

I don't think anyone is proposing to dispense with it outright.
Rather, it's a matter of examining our experiences about when it works
and when it doesn't.

A.

_______________________________________________
UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
Corpora mailing list
Corpora at uib.no
http://mailman.uib.no/listinfo/corpora