[Corpora-List] Against the reviewer mediation stage

[Jason Eisner]

On Fri, May 29, 2009 at 4:46 AM, Adam Kilgarriff
<adam at lexmasterclass.com> wrote:
> I reviewed the paper maybe three weeks ago and (at this frenetic time of year)
> have probably reviewed half a dozen other papers between times.

THERE's your problem.  Why don't you just review the paper at the last
minute, like everyone else?  Then it will still be fresh in your mind
when discussion period starts a day or two later. ;-)

The discussion period does add a bit of time for some papers, but
you've got to admit that at least it's still easier than journal
reviewing, which can drag on for many months with revisions and
resubmissions and author correspondence.

Conferences are designed to get faster turnaround than journals, with
a quick up-or-down decision and quick publication.  But since
conferences play the same central role in our field that journals play
in other fields, they are only willing to cede so much when it comes
to reviewing quality or the quality of feedback to authors.

> if two reviewers disagree, they are expected to contribute to a discussion
> where they see if they can reconcile their differences.

That's not quite it.  If the area chair can't figure out how to rank a
paper, then he or she will go back to the reviewers with questions.
Not all discussion is about "reconciling differences," nor do all
differences have to be reconciled.

> The image is very nice - academics sitting down to sort out their differences etc.,
> but the reality is (for me) quite different.

Hmm.  In my own experience as a reviewer (and as a chair), sometimes
there is real technical back-and-forth, which can result in a deeper
reading of the paper, better suggestions to the authors, and a better
understanding of the area (including related literature).  I've been
in discussions where we have proved or disproved theorems, come up
with new models or experiments to suggest, or made interesting
connections to other work.  I have learned a lot myself from some of
these discussions, as well as getting to know my co-reviewers better.

> I think the reviewer mediation phase should be scrapped.  Either use maths
> to merge reviewers' scores,

Anything but that!  From "How to chair a conference":

   "Please, please, please don't just sort the papers by the 3 reviewers'
   average overall recommendation! There is too much variance in these
   scores for n=3 to be a large enough sample. Maybe reviewer #1 tends to
   give high scores to everyone, reviewer #2 has warped priorities, and
   reviewer #3 barely read the paper or barely knows the area. Whereas
   another paper drew a different set of 3 reviewers."

The review is not a single score.  It is 8 or 9 scores, a bunch of
nuanced comments, and perhaps a discussion.  ALL of which provide
useful guidance to whoever is making the decisions.

> or if the chair thinks that would not get a good result in a particular case,
> let him/her read and decide.  That's his/her job.

Not practical because the chair can't read all the papers carefully,
or be expert in all the topics, or think of all sides.  That's why he
or she recruits and assigns reviewers who are well-equipped to help.

Of course, it's best when the chair plays an active role by reading
the paper to some degree and asking good questions of the reviewers.
But the chair is just one person.

-cheers, jason

p.s. I've often heard that NLP/CL has a lot to be proud of, including
our reviewing practices.  That's not to say that everything is perfect
as it is, but I would hate to be in one of the related subfields where
conference reviewing loads are unreasonable and superficial anonymous
one-line reviews are common, with no attribution or discussion to hold
reviewers to account.

_______________________________________________
Corpora mailing list
Corpora at uib.no
http://mailman.uib.no/listinfo/corpora