[Corpora-List] why LREC2012 NOT blind-reviewed?

[Ted Pedersen]

Greetings all,

Another alternative that might not have been mentioned in this far
ranging discussion is the idea of crowd sourcing peer review. As a
community we are now coming to rely (somewhat) on crowd sourced data,
so why not crowd sourced reviewing? ;) I say that somewhat tongue in
cheek, although if it's good enough for our data...

In any case, here's a NYT article discussing this a bit with respect
to the humanities...

http://www.nytimes.com/2010/08/24/arts/24peer.html?_r=1

A related model, I think, is simply to have a pre-print archive where
everyone uploads their latest and greatest work, and then let world
decide impact and merit via downloads and subsequent citations. That
may not work well for folks who need tenure committees to evaluate
them, but it does cut out the very expensive and time consuming review
process that often results in quite a few papers in a distinguished
proceedings (like ACL, etc.) that are relatively uncited and have
fairly low impact despite the rigorous review process they survived.
My understanding is that the physics community operates a bit like
this with http://arxiv.org/ In fact there was a time when I submitted
papers to arxiv but that was usually after acceptance at a conference.

In some sense you could think of our current publication model as a
sort of controlled upload to a pre-print archive (the ACL Anthology).
This represents a kind of safety net for authors, where a paper that
is accepted at ACL etc. will be impressive to a tenure committee
whether or not it is cited much and has much impact. If the only
papers that "counted" for tenure or promotion were those that had a
lot of downloads or citations...well, that would probably change
things quite a bit. I suspect we'd write fewer papers and maybe be a
bit less incremental. That sounds good, although I'm sure there are
many other side effects I'm not considering.

Cordially,
Ted

On Mon, Oct 3, 2011 at 7:39 AM, Ted Pedersen <tpederse at d.umn.edu> wrote:
>
> I think one thing we've learned here is that there are some fairly
> significant concerns and frustrations with reviewing.
> For myself, I review too much, and am probably a little lazy in some cases
> as a result. I started to notice that a bit more recently, and so have
> started saying "No" with much more frequency, so hopefully I am doing my
> small part to correct something in the process. Reviewers who feel they
> don't have time to review really need to say "No" more often - the field
> will survive without us, and we aren't doing anyone any favors by submitting
> reviews that we don't really spend enough time on. LIkewise, I don't think
> saying "Yes" and then farming out reviews to graduate students is all that
> helpful, unless the senior person is willing to spend some time with the
> student on reviewing (until they are sufficiently experienced). I realize
> more folks doing this will make it harder to get reviews, but I think a
> smaller number of better reviews is in the end more helpful and healthy.
> I would like to suggest that maybe we ought to ask people who submit papers
> to provide the answers to the following two questions (separate from their
> papers) in an effort to streamline the process.
> 1) What is the most important idea presented in this submission (in 50 words
> or less)
> 2) What other paper is most similar to this submission, and how does this
> paper improve upon or extend that? (in 100 words or less)
> If I'm not sufficiently excited by the answers to both 1 and 2, then the
> paper can be rejected without further review. A good paper will of course
> make 1 and 2 fairly clear, but sometimes you have to dig a little, so I'd
> like to dispense with the kabuki dance and simply ask authors to answer
> these questions at the start, and then we decide as reviewers if we should
> read further.
> The other part of the equation is that most published papers don't end up
> having much impact beyond advancing the author's career (Zipf's Law for
> Papers? A few papers cited a lot, most not cited much at all). This doesn't
> mean they shouldn't be published,  and career advancement is a good thing
> generally both for the authors and our field, but it can also make for lots
> and lots of  incremental papers that just aren't all that interesting and so
> they aren't cited much, and they tend to have a mind numbing effect on
> reviewers and is part of what I think makes reviewing such a chore
> sometimes.  So, maybe if we make authors self-identify the incremental work
> versus the big new ideas then reviewers can have a better idea of what to
> expect. Incremental in the area you care most about can be fascinating
> stuff, so I don't think incremental is always a bad thing, but there also
> needs to be a balance between the incremental and the more novel. As a
> reviewer I feel like I spend huge amounts of time on incremental work, and
> it just gets a little dull to be honest...
> Cordially,
> Ted
> On Fri, Sep 30, 2011 at 9:45 AM, Isabella Chiari
> <isabella.chiari at uniroma1.it> wrote:
>>
>> Dear Corpora members,
>> I just noticed that the LREC2012 call specifies that submissions are NOT
>> anonymous and there will not be blind-reviewing.
>> Does anyone know why? Which is the policy under this decision?
>> Best regards,
>> Isabella Chiari
>>
>> Dipartimento di Scienze documentarie, linguistico-filologiche e
>> geografiche
>>
>> Università di Roma “La Sapienza”
>>
>> pl.le Aldo Moro, 5, III Piano, Edificio ex Facoltà di Lettere e Filosofia,
>> 00185 Roma, tel. +30 06 4991 3575
>>
>> E.mail: isabella.chiari at uniroma1.it
>>
>> Website: www.alphabit.net
>>
>> _______________________________________________
>> UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
>> Corpora mailing list
>> Corpora at uib.no
>> http://mailman.uib.no/listinfo/corpora
>>
>
>
>
> --
> Ted Pedersen
> http://www.d.umn.edu/~tpederse
>



-- 
Ted Pedersen
http://www.d.umn.edu/~tpederse

_______________________________________________
UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
Corpora mailing list
Corpora at uib.no
http://mailman.uib.no/listinfo/corpora