[Corpora-List] Against the reviewer mediation stage (German Rigau)

[Michal Ptaszynski]

How about using opinion mining methods to automatically extract the  
reviewer's opinions, present it in scores and co-reference them with the  
numerical scores provided by the reviewer? :)

Best,
--
Michal PTASZYNSKI
Language Media Laboratory
Graduate School of Information Science and Technology
Hokkaido University
Address: Kita-ku, Kita 14 Nishi 9, 060-0814 Sapporo, Japan
E-mail (work): ptaszynski at media.eng.hokudai.ac.jp
E-mail (home): michal.ptaszynski at gmail.com
http://sig.media.eng.hokudai.ac.jp/~ptaszynski/

--------------
Od: German Rigau <german.rigau at ehu.es>
Do: corpora at uib.no
Data: Fri, 29 May 2009 11:45:00 +0200
Temat: Re: [Corpora-List] Against the reviewer mediation stage

Hi Adam,

Thanks for opening this new conversation threat ...

I also believe that the "review mediation phase" is improving the overall  
quality of the reviewing process. Obviously, this process can always be  
improved (more time, need to reach a consensus, different scoring schemas  
among area-chairs, etc.)

However, it would be nice to see the real effect of this phase with  
respect the original scoring. Only a few changes? Many changes? Changes  
not in the scoring but on the reviews? ... Where is the effect of this  
phase reported?

Best,

German


Diana Santos wrote:
Sorry Adam, not only I do not share it, as I am an enthusiast about this.
  This is the only effective way to prevent people doing dishonest,  
careless or uninformed reviews and getting away with it.
  Except if one accepts the principle of Signed Reviews  
(http://www.linguateca.pt/Diana/SignedReviews.html), but this may have  
other consequences.
  I suggest you read
Chubin, D. R. & E. J. Hackett. /Peerless Science, Peer Review and U.S.  
Science Policy/. New York, State University of New York Press. 1990.
for a debate and some suggestions.
For those of you who read Portuguese, I have a page on these issues as  
well, with some further references:  
http://www.linguateca.pt/Diana/avalpubl.html
Best,
Diana

     ------------------------------------------------------------------------
     *From:* corpora-bounces at uib.no [mailto:corpora-bounces at uib.no] *On
     Behalf Of *Adam Kilgarriff
     *Sent:* 29. mai 2009 10:47
     *To:* corpora at uib.no
     *Subject:* [Corpora-List] Against the reviewer mediation stage

     Corpora readers,
      Do any of you share my feeling about the 'review mediation phase'?
      I do reviewing partly out of duty and partly because it's a way of
     making sure I read closely at least one arbitrary subset of new
     work in my area  - and sometimes I find out about really
     interesting work in this way.  I do like the innovation of being
     able to bid for the papers you actively want to review.
      But an innovation I don't like is the 'review mediation process',
     as now widely used by ACL and EMNLP where, if two reviewers
     disagree, they are expected to contribute to a discussion where
     they see if they can reconcile their differences.  The image is
     very nice - academics sitting down to sort out their differences
     etc., but the reality is (for me) quite different.  I reviewed the
     paper maybe three weeks ago and (at this frenetic time of year)
     have probably reviewed half a dozen other papers between times.     To  
make a considered comment, I need to take my time to
     re-acquaint myself with the paper, remind myself of what I said in
     my review, give careful thought to the other reviewers' comments,
     and work out how to respond, which involves delicate processes
     (with both interpersonal and intellectual components) of standing      
up for my considered opinion while giving due heed to what others
     have said (and being polite even if I think the other person's
     opinion is rubbish - no anonymity here).  One good thing about
     initial reviewing is that you can do it in your own time.  But
     that's not true for review mediation, because there are only two
     or three days allocated to that phase.  And here I am expected to
     devote as much time again to it as I did to the original version,
     and there's nothing in it for me, as I've already read it so I
     won't find any new ideas.
      I think the reviewer mediation phase should be scrapped.  Either
     use maths to merge reviewers' scores, or if the chair thinks that
     would not get a good result in a particular case, let him/her read
     and decide.  That's his/her job.
      Adam

     --     ================================================
     Adam Kilgarriff                                          
http://www.kilgarriff.co.uk                  Lexical Computing  
Ltd                   http://www.sketchengine.co.uk
     Lexicography MasterClass Ltd      http://www.lexmasterclass.com
     Universities of Leeds and Sussex       adam at lexmasterclass.com
     <mailto:adam at lexmasterclass.com>
     ================================================

------------------------------------------------------------------------

_______________________________________________
Corpora mailing list
Corpora at uib.no
http://mailman.uib.no/listinfo/corpora

 

_______________________________________________
Corpora mailing list
Corpora at uib.no
http://mailman.uib.no/listinfo/corpora