[Corpora-List] discussion on reproducibility at ACL 2011 business meeting

[Ruvan Weerasinghe]

Thanks for the clarification Mike. What I meant was that we may be able to learn the pros and cons they experienced first hand - without going through what could be a painful experience ourselves. 

I know that some of their referees actually run the code of the author against their own data to verify the results claimed! I guess that's rather extreme. Being from the two-thirds world, what I would really benefit from is free availability of data more than code (though as has been suggested, the availability of the core of the algorithm would greatly reduce the time taken to reproduce its results on our data for instance). This would also allow us to check our algorithms on other people's data and make sure that we haven't for instance 'overtrained' our models to some specific situation. 


Ruvan Weerasinghe 
University of Colombo School of Computing 
Colombo 00700, 
Sri Lanka. 

Web: http://www.ucsc.lk 
Phone: +94112158953; Fax: +94112587239 




From: "Mike Maxwell" <maxwell at umiacs.umd.edu> 
To: "John F. Sowa" <sowa at bestweb.net> 
Cc: corpora at uib.no 
Sent: Tuesday, July 5, 2011 3:19:56 AM 
Subject: Re: [Corpora-List] discussion on reproducibility at ACL 2011 business meeting 

On 7/4/2011 4:58 PM, John F. Sowa wrote: 
> On 7/4/2011 4:25 PM, Mark Sammons wrote: 
>> Might it not be sufficient to simply encourage this practice without 
>> formalizing the requirements, and rely on positive feedback to do the 
>> job? 
> 
> Yes. That's what I was suggesting. 
> 
> There are many different research practices that have led to important 
> new discoveries and breakthroughs. Imposing too many conditions and 
> restrictions on what is considered publishable is more likely to 
> inhibit rather than promote progress. 

Which of course is what this thread was about in the first place. 
Quoting Ted (but adding emphasis): 
> ...in favor of the innovation this year on *allowing* 
> submissions of data and code along with paper submissions. 

I don't think Ruvan was suggesting making code/data submission a 
requirement in our field, he was just saying we could look at how the 
biology/ bio-informatics people (who reportedly do make it a 
requirement) addressed the issues which had been raised. 

And I didn't say anything about requiring it, either. 

So can we stop beating the dead horse of requiring code/data submission, 
and get back to the point Ted raised: the concerns over how and whether 
to encourage reproducible research by means of code and data publication. 
-- 
Mike Maxwell 
maxwell at umiacs.umd.edu 
"My definition of an interesting universe is 
one that has the capacity to study itself." 
--Stephen Eastmond 

_______________________________________________ 
UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora 
Corpora mailing list 
Corpora at uib.no 
http://mailman.uib.no/listinfo/corpora 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listserv.linguistlist.org/pipermail/corpora/attachments/20110705/0f772f31/attachment.htm>
-------------- next part --------------
_______________________________________________
UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
Corpora mailing list
Corpora at uib.no
http://mailman.uib.no/listinfo/corpora