[Corpora-List] Against the reviewer mediation stage

[John F. Sowa]

I sympathize with both sides of this argument, but the problems of
"reviewer roulette" are very real.  In the olden days, many conferences
had meetings with the program committee members in the same room
where they could resolve disputes.

At one such meeting, I rescued a paper that had two bad reviews
and one good review.  I could see that the paper had merit, but the
main point wasn't stated as clearly as it should have been.  The
committee agreed, and we gave a conditional acceptance provided
that the authors would make certain revisions.  The revised paper
turned out to be one of the best in the proceedings.

One another occasion, two of my colleagues at IBM submitted the
following paper to IJCAI:

    Binot, Jean-Louis and Karen Jensen. 1987. "A semantic expert using
    an online standard dictionary." Proceedings of IJCAI-87, Milan,
    Italy, August 1987.

They presented a method of resolving ambiguities in parsing by using
the sample sentences from a dictionary (the MW 7th) instead of the
definitions.  They printed five copies of the paper and sent them
by first-class mail.

However, they were worried that their paper might not be delivered
before the deadline.  So they printed another five copies, which
they sent by express mail with a cover letter saying that it should
be discarded if the other set had been received.

What happened was that both sets were received, and they were sent
to different reviewers.  One set got three negative reviews, including
one that just said "I never saw anyone do anything like this before."
(Apparently, research should be new, but not too new.)  The other set
got three positive reviews and was accepted.  So the authors presented
the paper, and it became an early example of how a corpus (admittedly
small) could be used to resolve ambiguities.

Another colleague at IBM, Eva Mueckstein, worked in Fred Jelinek's
group.  She implemented a parser that used a finite-state machine
to select which grammar rules to try at each step.  It also collected
statistics about how frequently each option was taken, and it would
use them to change the order in which the FSM would select rules
and to choose between alternative parses.  She submitted a paper
about the parser to IJCAI '81, and it was rejected with the comment
"Statistics is not AI!"

Today, many reviewers have a more favorable view of statistics,
and they might bias their reviews somewhat differently.

John Sowa


_______________________________________________
Corpora mailing list
Corpora at uib.no
http://mailman.uib.no/listinfo/corpora