[Corpora-List] Against the reviewer mediation stage

[Ted Pedersen]

Interesting thoughts Adam, and I think it's an important issue to discuss.

I do think there can be a problem in how the mediation stage is
presented to the reviewers - if there is a specific request to reach a
consensus from a chair, then that sometimes seems to be steering
things towards a sort of bland middle ground (where everyone ends up
saying "borderline"). On the other hand, if the instructions are
"please read the other reviews, and if you feel like changing your
scores / comments as a result you have until Thursday to do that",
then there's not quite that same subtle pressure.

Ken Church makes some related points in his "Reviewing the Reviewers"
Last Words from a couple of years ago.
http://aclweb.org/anthology-new/J/J05/J05-4006.pdf I think one of the
more interesting points was that we tend to shoot down papers with one
bad review, and favor those papers that are a bit safer (those with no
bad scores). I think he was suggesting that we ought to look for
papers that can attract at least one strong "champion", and that might
make things a bit more interesting (if we accepted or rejected papers
based on whether or not it inspired particularly powerful reviews (pro
or con)).

Perhaps we just need to try and be a bit more bold in our reviewing -
do away with the "3" score altogether and if a paper doesn't make you
want to fight for it, reject it. :)

Cordially,
Ted

On Fri, May 29, 2009 at 3:46 AM, Adam Kilgarriff
<adam at lexmasterclass.com> wrote:
> Corpora readers,
>
> Do any of you share my feeling about the 'review mediation phase'?
>
> I do reviewing partly out of duty and partly because it's a way of making
> sure I read closely at least one arbitrary subset of new work in my area  -
> and sometimes I find out about really interesting work in this way.  I do
> like the innovation of being able to bid for the papers you actively want to
> review.
>
> But an innovation I don't like is the 'review mediation process', as now
> widely used by ACL and EMNLP where, if two reviewers disagree, they are
> expected to contribute to a discussion where they see if they can reconcile
> their differences.  The image is very nice - academics sitting down to sort
> out their differences etc., but the reality is (for me) quite different.  I
> reviewed the paper maybe three weeks ago and (at this frenetic time of year)
> have probably reviewed half a dozen other papers between times.  To make a
> considered comment, I need to take my time to re-acquaint myself with the
> paper, remind myself of what I said in my review, give careful thought to
> the other reviewers' comments, and work out how to respond, which involves
> delicate processes (with both interpersonal and intellectual components) of
> standing  up for my considered opinion while giving due heed to what others
> have said (and being polite even if I think the other person's opinion is
> rubbish - no anonymity here).  One good thing about initial reviewing is
> that you can do it in your own time.  But that's not true for review
> mediation, because there are only two or three days allocated to that
> phase.  And here I am expected to devote as much time again to it as I did
> to the original version, and there's nothing in it for me, as I've already
> read it so I won't find any new ideas.
>
> I think the reviewer mediation phase should be scrapped.  Either use maths
> to merge reviewers' scores, or if the chair thinks that would not get a good
> result in a particular case, let him/her read and decide.  That's his/her
> job.
>
> Adam
>
> --
> ================================================
> Adam Kilgarriff
>  http://www.kilgarriff.co.uk
> Lexical Computing Ltd                   http://www.sketchengine.co.uk
> Lexicography MasterClass Ltd      http://www.lexmasterclass.com
> Universities of Leeds and Sussex       adam at lexmasterclass.com
> ================================================
>
> _______________________________________________
> Corpora mailing list
> Corpora at uib.no
> http://mailman.uib.no/listinfo/corpora
>
>



-- 
Ted Pedersen
http://www.d.umn.edu/~tpederse

_______________________________________________
Corpora mailing list
Corpora at uib.no
http://mailman.uib.no/listinfo/corpora