[Corpora-List] Against the reviewer mediation stage

[Diana Santos]

Sorry Adam, not only I do not share it, as I am an enthusiast about this.

This is the only effective way to prevent people doing dishonest, careless or uninformed reviews and getting away with it.

Except if one accepts the principle of Signed Reviews (http://www.linguateca.pt/Diana/SignedReviews.html), but this may have other consequences.

I suggest you read
Chubin, D. R. & E. J. Hackett. Peerless Science, Peer Review and U.S. Science Policy. New York, State University of New York Press. 1990.
for a debate and some suggestions.
For those of you who read Portuguese, I have a page on these issues as well, with some further references: http://www.linguateca.pt/Diana/avalpubl.html
Best,
Diana



________________________________
From: corpora-bounces at uib.no [mailto:corpora-bounces at uib.no] On Behalf Of Adam Kilgarriff
Sent: 29. mai 2009 10:47
To: corpora at uib.no
Subject: [Corpora-List] Against the reviewer mediation stage

Corpora readers,

Do any of you share my feeling about the 'review mediation phase'?

I do reviewing partly out of duty and partly because it's a way of making sure I read closely at least one arbitrary subset of new work in my area  - and sometimes I find out about really interesting work in this way.  I do like the innovation of being able to bid for the papers you actively want to review.

But an innovation I don't like is the 'review mediation process', as now widely used by ACL and EMNLP where, if two reviewers disagree, they are expected to contribute to a discussion where they see if they can reconcile their differences.  The image is very nice - academics sitting down to sort out their differences etc., but the reality is (for me) quite different.  I reviewed the paper maybe three weeks ago and (at this frenetic time of year) have probably reviewed half a dozen other papers between times.  To make a considered comment, I need to take my time to re-acquaint myself with the paper, remind myself of what I said in my review, give careful thought to the other reviewers' comments, and work out how to respond, which involves delicate processes (with both interpersonal and intellectual components) of standing  up for my considered opinion while giving due heed to what others have said (and being polite even if I think the other person's opinion is rubbish - no anonymity here).  One good thing about initial reviewing is that you can do it in your own time.  But that's not true for review mediation, because there are only two or three days allocated to that phase.  And here I am expected to devote as much time again to it as I did to the original version, and there's nothing in it for me, as I've already read it so I won't find any new ideas.

I think the reviewer mediation phase should be scrapped.  Either use maths to merge reviewers' scores, or if the chair thinks that would not get a good result in a particular case, let him/her read and decide.  That's his/her job.

Adam

--
================================================
Adam Kilgarriff                                      http://www.kilgarriff.co.uk
Lexical Computing Ltd                   http://www.sketchengine.co.uk
Lexicography MasterClass Ltd      http://www.lexmasterclass.com
Universities of Leeds and Sussex       adam at lexmasterclass.com<mailto:adam at lexmasterclass.com>
================================================
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listserv.linguistlist.org/pipermail/corpora/attachments/20090529/61927165/attachment.htm>
-------------- next part --------------
_______________________________________________
Corpora mailing list
Corpora at uib.no
http://mailman.uib.no/listinfo/corpora