[Corpora-List] why LREC2012 NOT blind-reviewed?

Thu Oct 6 18:03:13 UTC 2011

Hi everyone,

I'm not exactly an established researcher in this field, but I sensed interest in _everyone_'s view, so I will contribute a few points. First, my reviewing activity has been exclusively limited to computer vision so far. That implies that I don't know anything about the system in NLP, so please don't rate me for making wrong assumptions.

Now, here are a few points:

- comparing journals and conferences seems inappropriate: many journals primarily get extended versions of conference publications. Here, hiding the authors is not sensible. Also, the information of where the paper was accepted can help greatly to overcome any prejudices the reviewer might have about the authors.
- I very much liked the idea of having a two-round system with feedback from both the authors and the reviewers.
- I wouldn't worry too much about things like searching for similar passages (I support fighting plagiarisms, but in case the paper is original all knowledge accidentally gained in the process should be ignored for the review). As we all agreed, reviewers have little time to spend anyway. But on the other hand, I know that some reviewers are intentionally abusing their position: someone once told me he was doing that (I won't tell who, but I no longer talk to him). And I think we should at least try to improve the system against such persons.
- If we want to go for less anonymity, I think we should start with revealing the area chair of each paper. After all, these established names are least likely to get a serious disadvantage.
- Maybe it would be good if one could complain about a reviewer, with the effect that (maybe after several complaints) this pair of authors and reviewers will not be matched for some time. This could still be double-blind, only some roof organisation (ACL?) would have the list. Of course, for big names it might be rather easy to circumvent the system. One would have to point out serious errors in the reviews to complain.

- The proposal that reviewers are identified upon complaints is something that should at least be considered - in my opinion.
- I believe that the area chairs are in a very good position to improve the reviews: by assigning reviewers that are familiar with the problem.

- Like someone else said already, I frequently guess wrongly about who has written the papers I review. 
- Teaching students to review seems an essential part of a supervisor's work. But at least in the beginning, the students should not work alone, and be warned against typical errors (narrow scope, etc.).

- I really don't like this "I'm an Indian, so I will be rejected"-attitude. I have lived abroad several times (and am living abroad now), and I have always tried to avoid the assumption that something happened because I was a foreigner. Now, here's my proposal: I will attend IJCNLP this year (which is not that far from India), and since I don't have many contacts yet, I explicitly welcome all Indians to come and chat with me! Of course the others are welcome, too.

In any case, change always comes slowly, so we should take our time to discuss this - preferably in a larger scope than here. There are quite a few things that NLP is doing better than computer vision. E.g. that references don't count for the page limitations.

Best regards,
  Thomas (currently University of Pisa, Italy)

________________________________
Von: Michal Ptaszynski <ptaszynski at media.eng.hokudai.ac.jp>
An: "corpora at uib.no" <corpora at uib.no>; "corpora-request at uib.no" <corpora-request at uib.no>
Cc: ptaszynski at hgu.jp
Gesendet: 17:30 Donnerstag, 6.Oktober 2011 
Betreff: [Corpora-List]  why LREC2012 NOT blind-reviewed?

Ladies and Gents,

This fantastic reviewing-ralated topic has been probably the most active
one for some time. It looks like most of Corpora members put their 2 cents
into the bag. The bag seems now quite heavy and shows somewhat big
potential to turn it into something creative.

My question/proposition is,

How about working out a better reviewing system than the present double
blind reviewing?

It could actually change something for the better, and since most of us
took an active part in the discussion we probably have some ideas.

To give a good start let me say my bunch of ideas.

- Field assessment before reviewing. Something that has been implemented
in easychair and several other systems to some extent. A reviewer selects
her fields of expertise to give the conference organizers a hint on what
papers he/she could review with the highest confidence.
- Make blinding a choice. Some researchers want to be anonymous, and some
want to use their previous research, but don't want to describe their
previous works too extensively (saving paper space).
- Discussion between authors and reviewers. For conferences there could be
too short time span for double rebuttal, but for journals there could be
allowed a longer open discussion. Its sometimes difficult to convince a
reviewer only in one rebuttal session.
- Using NLP technology to help reviewers and authors. Automatic
summarization, credibility verification of infromation contained in a
paper - for the reviewers, and for the other side (or both), opinion
mining/sentiment analysis of all other reviews of the paper (comparing
reviews) and of all reviews of one reviewer (is he/she an understanding,
tolerant reviewer or an innocent bunny killer?)

Let me know about your ideas.

Michal

--------------------------
Od: Anil Singh <anil.phdcl at gmail.com>
Kopia dla: corpora at uib.no
Do: Gemma Boleda <gemma.boleda at upf.edu>
Data: Thu, 6 Oct 2011 02:39:41 +0530
Temat: Re: [Corpora-List] why LREC2012 NOT blind-reviewed?

Not to forget the fact that getting a visa from countries like India to
the US or to Europe is a project in itself. And it costs quite something.
And if the visa is rejected after all that time, effort and expenditure...
Rejection of visa is not rare at all: for the US (where a substantial
proportion of the highest profile conferences are held) the rate is very
high, especially for gradutate students.

On Thu, Oct 6, 2011 at 2:09 AM, Anil Singh <anil.phdcl at gmail.com> wrote:
   I mostly agree on this point, except that this is quite a radical
proposal and can be implemented only in the long term. We will need a lot
of good quality journal to replace conference proceedings. Will be these
be printed or purely online? Will those purely online considered to have
the same credibility as those printed (why not?)?

The cost of attending a conference/workshop is, of course, a major hurdle
for all researchers in developing countries. Nowadays even registration
fee is so high that it is hard to afford it (say, in India) just for one
author, even if no one travels abroad to attend the event and present the
paper. And no solution for this is in sight (never mind the Emerging
Economy, New Economic Powerhouses  etc.). We are clearly being told
explicity to not try to publish in conferences, but to try for journals
(students by supervisors and administration, teachers/researchers by
administrations and funding agencies). Easy to say, but how many journals
are out there for the whole of CL/NLP (as compared to the number
conferences and workshops)?

Still, shifting to journals from conferences (for publication) is
something that has to happen in CL/NLP.

The academic evaluation forms (in India at least) give a much higher
weightage to journal publications than to conference publications, which
is a big disadvantage for those working in CL/NLP under the current
situation.

Of course, as even Church (2005) had hinted, there is practical problem
involved. Conference publications mean registrations and registrations
mean $$. Where will the money for the journals come from? Who will sponsor
them? If commercial publishers publish them, won't other factors come in
which might affect their cridibility?

Just some doubts. But generally I support the idea.

On Wed, Oct 5, 2011 at 9:45 PM, Gemma Boleda <gemma.boleda at upf.edu> wrote:
   Dear list members,

   some of the concerns that have been raised in this discussion, such as
reviewer load and "incrementality" in papers, could be addressed if the
field moved to journal, rather than conference, publishing, and used
conferences for dissemination of ideas (where only abstracts would be
reviewed) and journals for actual publication. This would have the
positive side-effect of making the citation indices of computational
linguistics as a subfield go up, thus making it more visible in the
"scientific market". For a 1.5-page long elaboration of these ideas, see
http://www.lsi.upc.edu/~gboleda/pubs/gboleda_publishingCL.pdf

   Best,
   Gemma Boleda

   _______________________________________________
   UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
   Corpora mailing list
Corpora at uib.no
http://mailman.uib.no/listinfo/corpora

_______________________________________________
UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
Corpora mailing list
Corpora at uib.no
http://mailman.uib.no/listinfo/corpora
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listserv.linguistlist.org/pipermail/corpora/attachments/20111006/780b6a53/attachment.htm>
-------------- next part --------------
_______________________________________________
UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
Corpora mailing list
Corpora at uib.no
http://mailman.uib.no/listinfo/corpora