[Corpora-List] Against the reviewer mediation stage

[German Rigau]

Hi Adam,

Thanks for opening this new conversation threat ... ;-)

I also believe that the "review mediation phase" is improving the 
overall quality of the reviewing process. Obviously, this process can 
always be improved (more time, need to reach a consensus, different 
scoring schemas among area-chairs, etc.)

However, it would be nice to see the real effect of this phase with 
respect the original scoring. Only a few changes? Many changes? Changes 
not in the scoring but on the reviews? ... Where is the effect of this 
phase reported?

Best,

German


Diana Santos wrote:
> Sorry Adam, not only I do not share it, as I am an enthusiast about this.
>  
> This is the only effective way to prevent people doing dishonest, 
> careless or uninformed reviews and getting away with it.
>  
> Except if one accepts the principle of Signed Reviews 
> (http://www.linguateca.pt/Diana/SignedReviews.html), but this may have 
> other consequences.
>  
> I suggest you read
> Chubin, D. R. & E. J. Hackett. /Peerless Science, Peer Review and U.S. 
> Science Policy/. New York, State University of New York Press. 1990.
> for a debate and some suggestions.
> For those of you who read Portuguese, I have a page on these issues as 
> well, with some further references: 
> http://www.linguateca.pt/Diana/avalpubl.html
> Best,
> Diana
>  
>  
>
>     ------------------------------------------------------------------------
>     *From:* corpora-bounces at uib.no [mailto:corpora-bounces at uib.no] *On
>     Behalf Of *Adam Kilgarriff
>     *Sent:* 29. mai 2009 10:47
>     *To:* corpora at uib.no
>     *Subject:* [Corpora-List] Against the reviewer mediation stage
>
>     Corpora readers,
>      
>     Do any of you share my feeling about the 'review mediation phase'?
>      
>     I do reviewing partly out of duty and partly because it's a way of
>     making sure I read closely at least one arbitrary subset of new
>     work in my area  - and sometimes I find out about really
>     interesting work in this way.  I do like the innovation of being
>     able to bid for the papers you actively want to review.
>      
>     But an innovation I don't like is the 'review mediation process',
>     as now widely used by ACL and EMNLP where, if two reviewers
>     disagree, they are expected to contribute to a discussion where
>     they see if they can reconcile their differences.  The image is
>     very nice - academics sitting down to sort out their differences
>     etc., but the reality is (for me) quite different.  I reviewed the
>     paper maybe three weeks ago and (at this frenetic time of year)
>     have probably reviewed half a dozen other papers between times. 
>     To make a considered comment, I need to take my time to
>     re-acquaint myself with the paper, remind myself of what I said in
>     my review, give careful thought to the other reviewers' comments,
>     and work out how to respond, which involves delicate processes
>     (with both interpersonal and intellectual components) of standing 
>     up for my considered opinion while giving due heed to what others
>     have said (and being polite even if I think the other person's
>     opinion is rubbish - no anonymity here).  One good thing about
>     initial reviewing is that you can do it in your own time.  But
>     that's not true for review mediation, because there are only two
>     or three days allocated to that phase.  And here I am expected to
>     devote as much time again to it as I did to the original version,
>     and there's nothing in it for me, as I've already read it so I
>     won't find any new ideas.
>      
>     I think the reviewer mediation phase should be scrapped.  Either
>     use maths to merge reviewers' scores, or if the chair thinks that
>     would not get a good result in a particular case, let him/her read
>     and decide.  That's his/her job.
>      
>     Adam
>
>     -- 
>     ================================================
>     Adam Kilgarriff                                    
>      http://www.kilgarriff.co.uk              
>     Lexical Computing Ltd                   http://www.sketchengine.co.uk
>     Lexicography MasterClass Ltd      http://www.lexmasterclass.com
>     Universities of Leeds and Sussex       adam at lexmasterclass.com
>     <mailto:adam at lexmasterclass.com>
>     ================================================
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Corpora mailing list
> Corpora at uib.no
> http://mailman.uib.no/listinfo/corpora
>   


_______________________________________________
Corpora mailing list
Corpora at uib.no
http://mailman.uib.no/listinfo/corpora