15.355, Disc: Penultimate Posting: Blind Peer Review

[LINGUIST List]

LINGUIST List:  Vol-15-355. Thu Jan 29 2004. ISSN: 1068-4875.

Subject: 15.355, Disc: Penultimate Posting: Blind Peer Review

Moderators: Anthony Aristar, Wayne State U.<aristar at linguistlist.org>
            Helen Dry, Eastern Michigan U. <hdry at linguistlist.org>

Reviews (reviews at linguistlist.org):
	Sheila Collberg, U. of Arizona
	Terence Langendoen, U. of Arizona

Home Page:  http://linguistlist.org/

The LINGUIST List is funded by Eastern Michigan University, Wayne
State University, and donations from subscribers and publishers.

Editor for this issue: Sarah Murray <sarah at linguistlist.org>
 ==========================================================================
To post to LINGUIST, use our convenient web form at
http://linguistlist.org/LL/posttolinguist.html.
=================================Directory=================================

1)
Date:  Thu, 29 Jan 2004 01:57:11 -0500
From:  Franca Ferrari <ff244 at MAIL.nyu.edu>
Subject:  Re: 15.324, Disc: Re: Blind Peer Review

2)
Date:  Thu, 29 Jan 2004 10:04:49 -0500
From:  Sally Thomason <thomason at umich.edu>
Subject:  Re: 15.324, Disc: Re: Blind Peer Review

-------------------------------- Message 1 -------------------------------

Date:  Thu, 29 Jan 2004 01:57:11 -0500
From:  Franca Ferrari <ff244 at MAIL.nyu.edu>
Subject:  Re: 15.324, Disc: Re: Blind Peer Review


I just want to add a comment to this discussion. I am not in favor of
blind peer review, but I can understand some of its validities in
terms of freedom of expression for the reviewers. But what I find out
of place is the freedom of the reviewers to insult openly the author
of the article under review.

It happened to me and to many other linguists I know. Our work was
trashed, we were insulted and almost called stupid, just because what
we wrote was not conform with the reviewers' way to conceive a
particular linguistic topic.

I do not find proper of a so called 'scientific review' to insult
someone just because he might disagree with the reviewers' opinions.

 As Ron Sheed did, I am going to complain with the editorial board. I
hope to have more luck! The editorial board must be made responsible
for the tones and the language used by the reviewers. Along this long
review process, someone must be responsible to mediate between the
authors and the reviewers. After all, reviewers and authors are human
beings and it is human to make mistakes.

Maybe for some controversial and difficult reviews, the editorial board
should review the reviews and act as a mediator. A third reviewer should be
hired to re-examine paper and reviews, his primary task should be to
understand why reviewers and authors are so far apart in their way of
thinking about a specific topic.

-franca ferrari



-------------------------------- Message 2 -------------------------------

Date:  Thu, 29 Jan 2004 10:04:49 -0500
From:  Sally Thomason <thomason at umich.edu>
Subject:  Re: 15.324, Disc: Re: Blind Peer Review


I've read with interest the postings of Ron Sheen and others on the
blind review process and related issues of journal submission.  I'm an
ex-editor (or rather a recovering editor, in various senses of the
phrase) -- I edited the LSA's journal Language from 1988 to 1994 --
and it's disturbing, though not surprising, to see how widespread
misunderstanding of the refereeing process is. Before I comment on
some of the criticisms and suggestions that have been made on LINGUIST
LIST lately, I should say that I completely agree with one criticism:
the long and apparently lengthening time gap between date of
submission and date of decision on a manuscript is a serious problem
for our field. By working 30-40 hours a week on my journal duties, I
managed to keep the average turn-around time for submissions down
below 4 months; but it wasn't easy, and I doubt if I could've done it
if I hadn't had generous support from my university (staff, course
releases) as well as the LSA's support. Editors don't get paid for
editing (though the LSA did, in the middle of my editorship, institute
a system of honoraria for the editor) any more than referees do. My
office handled all book reviews as well as ca. 135-150 manuscript
submissions each year. It's a lot of work. About the length of the
refereeing process specifically: editors are in a bind over this,
because if you nag a referee too frequently for a long-overdue report,
it might annoy the referee sufficiently that s/he will take it out on
the poor author.

   Now, about the refereeing process: like David Odden (not Ogden, as
Ron Sheen's most recent message has it), I support the blind
refereeing system. It's basically the same principle as with writing
letters of recommendation for students: if the student doesn't waive
his/her right to see the letter, many writers will pull their punches,
because they don't want the aggravation of complaints from the
student, or threatened or actual lawsuits. Ditto for tenure letters,
though in that case the letter-writers' identities will (usually) be
kept confidential automatically, unless the university is ordered by a
court of law to turn them over to an unsuccessful candidate for
tenure. If you think lawsuits are not something that would ever occur
in the world of academic journals, think again: I was threatened with
three lawsuits by disgruntled authors of books that had been
unfavorably reviewed in the journal, and there's no reason to think
that a very few disgruntled authors of rejected papers wouldn't try
this route too. (A very few because, in my experience, the vast
majority of authors were gracious, prepared to acknowledge that editor
& referees were doing their best to be fair even if they made
mistakes.)

   The point is that you get more objective letters if the referees
know that their names will not be revealed to authors of mss. Unlike
Ron Sheen and some others, I don't see anything sinister in this: it's
a lot easier to be frank about shortcomings (if any) in a paper if you
don't have to duck every time you encounter the author at a conference
afterward. I have written a sizable number of critical referee reports
on mss.  submitted to various journals by authors I like and respect;
I wouldn't want my collegial relations to be strained by possible hurt
feelings. And in addition, it would take a lot more time to write a
suitably critical report, where criticism seems necessary to me, if I
knew the author would know who wrote it.  Time, for busy people, is
never plentiful.

   In any case, and whatever other referees' motives are for remaining
anonymous, an editor has a powerful motive for insisting on blind
refereeing. It is very, very hard to get people to write referee
reports, and especially hard to get them to write reports in a timely
fashion.  Many referees would not write reports at all if they weren't
going to be anonymous; others would write less critical reports that
didn't reflect their actual opinions of the mss., and that would
directly affect the quality of the journal. Reducing the number of
good available referees would of course also have serious consequences
for efforts to keep the time lag between submission & decision to a
minimum.

   That said, I sympathize with the suspicion, which is no doubt
justified in some cases, that some referees and some editors are not
objective, but instead are concerned to ensure that papers outside
their preferred frameworks don't get published. All I can say, after
seven years of experience with hundreds of referees, is that cases of
detectable inappropriate referee bias are much rarer than some people
tend to believe. I detected a few, but not many; of course I probably
missed some cases because the bias was too subtle for my nonspecialist
eye. (The journal Language is a general ling. journal, so many
submissions are going to be far from any editor's area(s) of
specialization.)

   Which brings me to my next point: I suspect that I wasn't alone in
having a long blacklist of referees never to be consulted a second
time. The people on my blacklist fell into several categories: they
wrote incompetent, rude, and/or perfunctory reports; they displayed
inappropriate bias against certain approaches; they took more than two
months to get a report to me. And then there were a few very senior
and prominent linguists who, after benefitting in their salad days
from reports written by hard-working senior linguists, decided that
they had no professional obligation to do the same service for junior
linguists in their turn -- those types were at the top of my
blacklist. It's true that referees don't get paid, they don't get
thanked (except by editors), and they don't gain professional
prominence by performing this service: they merely support the
profession.  People who don't feel that they owe the profession
anything once they've achieved the heights are (to my mind) just as
unappealing as people who write hatchet-job referee reports on papers
whose approaches they dislike. Fortunately, they're rareish.

   I don't have much to say on the topic of editor bias: there are
certainly inappropriately biased editors, and none of us is totally
free of bias in any case. But most of us do try not to let our own
biases dictate, or even influence, our decisions; no doubt some of us
succeed more than others. Editing a journal with an 85-90% rejection
rate (like Language during my stint as editor) means that rejection is
the norm, for papers in all frameworks. One week, actually in the same
week, I got two letters that accused me of opposite sins: one
complained that I was publishing only generative papers, the other
accused me of never publishing generative papers.  Like my
predecessor, Bill Bright, who also got such complaints of bias from
both generativists and nongenerativists, I took this as at least a
small sign that my efforts to be even-handed were not totally in vain.

   And finally (sorry for all the verbosity), nobody, but nobody,
would claim that the system always works perfectly, so that all & only
the best papers get accepted by a journal. I am sure I accepted weak
papers that I shouldn't have accepted, and that I rejected strong
papers that I shouldn't have rejected. I just don't know which was
which.

   Wait, sorry, that wasn't the last thing: my final comment is that
there are good and not so good ways to deal with rejection and with a
suspicion that the referees for your paper were biased or missed your
point and/or made serious mistakes in characterizing your arguments.
Writing an outraged letter to the editor is not so good. But editors
don't like making mistakes, and any reasonable editor will listen to a
reasonable complaint -- expressed non-shrilly, and without denouncing
the referee(s). It's always wisest (in my opinion) to assume, even if
deep down you don't believe it, that misunderstandings were innocent.
Revise the paper to make your points less easy to misunderstand, and
resubmit the paper with a cover letter that says where & how & why you
have made revisions in response to the referees' criticisms, and where
& how & why you have *not* made revisions in response to the
criticisms. On occasion I accepted a paper, without a second round of
refereeing, because I was so impressed with the cogency (and calmness,
non-defensiveness, etc.) of an author's cover letter submitted with a
revision. [Note: EVERY paper can use some revision!]  True, you can't
assume that an editor will be willing to consider a revised ms. at
all; some editors consider rejection to be absolute, unless they
explicitly encouraged revision & resubmission. (I don't like that
stance myself, but reasonable people could disagree on the policy,
certainly.)  But even then, revision before submitting to another
journal is a *really* good idea, especially if you think the referees
misunderstood you.

  -- Sally Thomason

---------------------------------------------------------------------------
LINGUIST List: Vol-15-355