[Corpora-List] Corpora Digest, Vol 52, Issue 3 [was blind reviewing]

[Adam Lopez]

> Of course, I have no statistics on the quality of reviews so I cannot
> comment on the accuracy of the word "often" above.

Indeed, it is surprising that in a discussion prompted by conference
about data, no one has mentioned conclusions derived from statistical
analysis of data, however tentative. Below is just a sample of what
pops up in a Google Scholar search for "blind review", and while it is
neither complete nor representative, the conclusions in each example
are at least drawn from data rather than anecdote.
Cheers
Adam

-Double-blind review favours increased representation of female authors
Double-blind peer review, in which neither author nor reviewer
identity are revealed, is rarely practised in ecology or evolution
journals. However, in 2001, double-blind review was introduced by the
journal Behavioral Ecology. Following this policy change, there was a
significant increase in female first-authored papers, a pattern not
observed in a very similar journal that provides reviewers with author
information. No negative effects could be identified, suggesting that
double-blind review should be considered by other journals.
http://www.sciencedirect.com/science/article/pii/S0169534707002704

-How blind is blind review?
Surveyed the beliefs of 132 academic psychologists to assess their
beliefs about the blindness of blind review as exhibited by journals
in their own field of interest. The Ss estimated that reviewers were
able to deduce the author's identities 72% of the time. Responses of
reviewers to a questionnaire sent by each of 6 editors with 20–30
manuscripts indicate that, excluding mechanical detection, only 25.7%
of reviewers are able to detect authors' identities. It is concluded
that blind reviewing is fairly blind. (16 ref) (PsycINFO Database
Record (c) 2010 APA, all rights reserved)
http://psycnet.apa.org/psycinfo/1985-29014-001

-How blind is blind review?
BACKGROUND: No representative surveys of scientific opinion about
blind review have been published, and there is very little information
on the success of the blinding process. The American Journal of Public
Health has practiced blind review since 1977. METHODS: In 1989 to 1990
312 of its reviewers were asked to identify author and institution in
the manuscript they reviewed, to provide clues to such identification,
to express their opinion concerning blind review, and to offer reasons
for their opinion. RESULTS: Reviewers claimed to be able to identify
author and/or institution in 47% of the 614 chances offered;
identification was incorrect 16% of the time, overall identification
correct 39% of the time. Self-referencing was the clue to
identification in 62%, personal knowledge in 38% of the cases. If only
personal knowledge cases are considered, blinding was successful 83%
of the time. Blinding was favored by 75% of the reviewers with most
asserting it eliminated bias. Reasons given for opposing blind review
included the following: blinding not possible, identification will not
influence judgment, and its obverse, identification assists judgment.
CONCLUSIONS: For the American Journal of Public Health blinding is
usually, but not always, successful; and the majority of its reviewers
favor current policy. Until more definitive data are in, reviewer
preference, which differs from journal to journal, seems the most
legitimate guide to journal policy on blind review.
http://ajph.aphapublications.org/cgi/content/abstract/81/7/843

The myth of the double-blind review?: author identification using only citations
Prior studies have questioned the degree of anonymity of the
double-blind review process for scholarly research articles. For
example, one study based on a survey of reviewers concluded that
authors often could be identified by reviewers using a combination of
the author's reference list and the referee's personal background
knowledge. For the KDD Cup 2003 competition's "Open Task," we examined
how well various automatic matching techniques could identify authors
within the competition's very large archive of research papers. This
paper describes the issues surrounding author identification, how
these issues motivated our study, and the results we obtained. The
best method, based on discriminative self-citations, identified
authors correctly 40--45% of the time. One main motivation for
double-blind review is to eliminate bias in favor of well-known
authors. However, identification accuracy for authors with substantial
publication history is even better (60% accuracy for the top-10% most
prolific authors, 85% for authors with 100 or more prior papers).
http://dl.acm.org/citation.cfm?id=981001

A Citation Analysis of the Impact of Blinded Peer Review
Objective. —To determine whether articles published in journals using
blinded peer review receive significantly more or fewer citations than
those published in journals using nonblinded peer review. Design.
—Drawing from a sample of 1051 full articles published in 28 economics
journals during 1984, we used nonlinear regression and ordered probit
techniques to estimate the impact of blinded peer review on citations
of these articles in 1985 through 1989. Outcomes. —Citations of
articles. Results. —Articles published in journals using blinded peer
review were cited significantly more than articles published in
journals using nonblinded peer review, controlling for a variety of
author, article, and journal attributes. Conclusions. —Nonblinded peer
review apparently suffers from type I error to a greater extent than
blinded peer review. That is, journals using nonblinded peer review
publish a larger fraction of papers that should not have been
published than do journals using blinded peer review. When reviewers
know the identity of the author(s) of an article, they are able to
(and evidently do) substitute particularistic criteria for
universalistic criteria in their evaluative process.
http://jama.ama-assn.org/content/272/2/147.short

_______________________________________________
UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
Corpora mailing list
Corpora at uib.no
http://mailman.uib.no/listinfo/corpora