[Corpora-List] why LREC2012 NOT blind-reviewed?

[Ted Pedersen]

Greetings all,

One related thought has to do with how blind blind reviewing turns out
to be. There is of course the parlor trick of reviewers guessing the
authors of anonymous papers, but then there is also the practice that
some of us have of running papers we are going to review through
commercial (or otherwise) plagiarism detection services. I started
doing this a few years ago, and while I haven't found too many cases
of flat out plagiarism, I often have a pretty good idea of who the
authors of "blinded" papers turn out to be. Services like
https://turnitin.com/static/index.php and http://safeassign.com/ (the
one I use, which the UMN has a license for) are reasonably effective
at using information found on the web to identify overly similar
passages from other material found online. Usually this is earlier
material by the same author (earlier version of paper with similar
background, perhaps a tech report they have posted on their web site,
similar reference lists, etc).

This raises a troublesome issue I think - if we really want blind
reviewing, should we consider submissions whose authors can be
identified via the use of this kind of service as not-anonymous, and
reject them immediately? That seems a bit harsh, but the fact is some
percentage of submissions just don't pass this test (I'd say 25%-50%).
If we don't want to follow this course of action, do we then
discourage the use of automatic services by reviewers? That doesn't
seem realistic, particularly when actual plagiarism does occur and it
seems a bit cruel and self-defeating to require reviewers to spot that
"manually". Or, do we go ahead and identify authors? Or do we just
(continue to) pretend to be blind? ;)

Cordially,
Ted

On Mon, Oct 3, 2011 at 7:39 AM, Ted Pedersen <tpederse at d.umn.edu> wrote:
>
> I think one thing we've learned here is that there are some fairly
> significant concerns and frustrations with reviewing.
> For myself, I review too much, and am probably a little lazy in some cases
> as a result. I started to notice that a bit more recently, and so have
> started saying "No" with much more frequency, so hopefully I am doing my
> small part to correct something in the process. Reviewers who feel they
> don't have time to review really need to say "No" more often - the field
> will survive without us, and we aren't doing anyone any favors by submitting
> reviews that we don't really spend enough time on. LIkewise, I don't think
> saying "Yes" and then farming out reviews to graduate students is all that
> helpful, unless the senior person is willing to spend some time with the
> student on reviewing (until they are sufficiently experienced). I realize
> more folks doing this will make it harder to get reviews, but I think a
> smaller number of better reviews is in the end more helpful and healthy.
> I would like to suggest that maybe we ought to ask people who submit papers
> to provide the answers to the following two questions (separate from their
> papers) in an effort to streamline the process.
> 1) What is the most important idea presented in this submission (in 50 words
> or less)
> 2) What other paper is most similar to this submission, and how does this
> paper improve upon or extend that? (in 100 words or less)
> If I'm not sufficiently excited by the answers to both 1 and 2, then the
> paper can be rejected without further review. A good paper will of course
> make 1 and 2 fairly clear, but sometimes you have to dig a little, so I'd
> like to dispense with the kabuki dance and simply ask authors to answer
> these questions at the start, and then we decide as reviewers if we should
> read further.
> The other part of the equation is that most published papers don't end up
> having much impact beyond advancing the author's career (Zipf's Law for
> Papers? A few papers cited a lot, most not cited much at all). This doesn't
> mean they shouldn't be published,  and career advancement is a good thing
> generally both for the authors and our field, but it can also make for lots
> and lots of  incremental papers that just aren't all that interesting and so
> they aren't cited much, and they tend to have a mind numbing effect on
> reviewers and is part of what I think makes reviewing such a chore
> sometimes.  So, maybe if we make authors self-identify the incremental work
> versus the big new ideas then reviewers can have a better idea of what to
> expect. Incremental in the area you care most about can be fascinating
> stuff, so I don't think incremental is always a bad thing, but there also
> needs to be a balance between the incremental and the more novel. As a
> reviewer I feel like I spend huge amounts of time on incremental work, and
> it just gets a little dull to be honest...
> Cordially,
> Ted
> On Fri, Sep 30, 2011 at 9:45 AM, Isabella Chiari
> <isabella.chiari at uniroma1.it> wrote:
>>
>> Dear Corpora members,
>> I just noticed that the LREC2012 call specifies that submissions are NOT
>> anonymous and there will not be blind-reviewing.
>> Does anyone know why? Which is the policy under this decision?
>> Best regards,
>> Isabella Chiari
>>
>> Dipartimento di Scienze documentarie, linguistico-filologiche e
>> geografiche
>>
>> Università di Roma “La Sapienza”
>>
>> pl.le Aldo Moro, 5, III Piano, Edificio ex Facoltà di Lettere e Filosofia,
>> 00185 Roma, tel. +30 06 4991 3575
>>
>> E.mail: isabella.chiari at uniroma1.it
>>
>> Website: www.alphabit.net
>>
>> _______________________________________________
>> UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
>> Corpora mailing list
>> Corpora at uib.no
>> http://mailman.uib.no/listinfo/corpora
>>
>
>
>
> --
> Ted Pedersen
> http://www.d.umn.edu/~tpederse
>



-- 
Ted Pedersen
http://www.d.umn.edu/~tpederse

_______________________________________________
UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
Corpora mailing list
Corpora at uib.no
http://mailman.uib.no/listinfo/corpora