<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:st1="urn:schemas-microsoft-com:office:smarttags" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 11 (filtered medium)">
<!--[if !mso]>
<style>
v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style>
<![endif]--><o:SmartTagType
namespaceuri="urn:schemas-microsoft-com:office:smarttags" name="country-region"/>
<o:SmartTagType namespaceuri="urn:schemas-microsoft-com:office:smarttags"
name="City"/>
<o:SmartTagType namespaceuri="urn:schemas-microsoft-com:office:smarttags"
name="PlaceType"/>
<o:SmartTagType namespaceuri="urn:schemas-microsoft-com:office:smarttags"
name="PlaceName"/>
<o:SmartTagType namespaceuri="urn:schemas-microsoft-com:office:smarttags"
name="place"/>
<!--[if !mso]>
<style>
st1\:*{behavior:url(#default#ieooui) }
</style>
<![endif]-->
<style>
<!--
/* Font Definitions */
@font-face
{font-family:"MS Mincho";
panose-1:2 2 6 9 4 2 5 8 3 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:"\@MS Mincho";
panose-1:2 2 6 9 4 2 5 8 3 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman";}
a:link, span.MsoHyperlink
{color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{color:blue;
text-decoration:underline;}
p
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman";}
p.Computer1, li.Computer1, div.Computer1
{margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:Arial;
color:blue;
font-weight:normal;
font-style:normal;
text-decoration:none none;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 1.25in 1.0in 1.25in;}
div.Section1
{page:Section1;}
-->
</style>
</head>
<body lang=EN-US link=blue vlink=blue>
<div class=Section1>
<p class=MsoNormal><font size=3 color=blue face=Arial><span style='font-size:
12.0pt;font-family:Arial;color:blue'>Hi Ruvan,<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=3 color=blue face=Arial><span style='font-size:
12.0pt;font-family:Arial;color:blue'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=3 color=blue face=Arial><span style='font-size:
12.0pt;font-family:Arial;color:blue'>Generally, publishing academic code is a
good idea, but publishing real code isn’t feasible. If the code is small
enough to publish, it is merely an aspect of the full requirements for
linguistic interaction. But even programs of less than a hundred thousand
lines simply don’t have that much functionality. <o:p></o:p></span></font></p>
<p class=MsoNormal><font size=3 color=blue face=Arial><span style='font-size:
12.0pt;font-family:Arial;color:blue'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=3 color=blue face=Arial><span style='font-size:
12.0pt;font-family:Arial;color:blue'>However, publishing code snippets, like
the Link Grammar developers did, would be very useful. In the LG case, the
publications included a very clear exposition of how constraint propagation can
be applied to simple context free grammars to cover, at most, ten percent of
the kind of linguistic conversations that are required. But the abstraction is
limited to parsing, not to linguistic interaction. That made the complexity of
the ideas match the originality of the published procedures. <o:p></o:p></span></font></p>
<p class=MsoNormal><font size=3 color=blue face=Arial><span style='font-size:
12.0pt;font-family:Arial;color:blue'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=3 color=blue face=Arial><span style='font-size:
12.0pt;font-family:Arial;color:blue'>I agree that publishing such abstraction
snippets is a good idea, but only for appropriate levels of detail. Beyond
that, it gets unreasonably complicated for others to learn from effectively. <o:p></o:p></span></font></p>
<p class=MsoNormal><font size=3 color=blue face=Arial><span style='font-size:
12.0pt;font-family:Arial;color:blue'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=3 color=blue face=Arial><span style='font-size:
12.0pt;font-family:Arial;color:blue'>Abstractions teach. But full code
publication merely confuses. So it should be a question of which papers should
contain published algorithms to demonstrate simple slices of a full system. <o:p></o:p></span></font></p>
<p class=MsoNormal><font size=3 color=blue face=Arial><span style='font-size:
12.0pt;font-family:Arial;color:blue'><o:p> </o:p></span></font></p>
<div>
<p class=MsoNormal><font size=3 color=blue face=Arial><span style='font-size:
12.0pt;font-family:Arial;color:blue'>-Rich</span></font><font color=black><span
style='color:black'><o:p></o:p></span></font></p>
<p class=MsoNormal><font size=3 color=black face="Times New Roman"><span
style='font-size:12.0pt;color:black'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=3 color=black face="Times New Roman"><span
style='font-size:12.0pt;color:black'>Sincerely,<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=3 color=black face="Times New Roman"><span
style='font-size:12.0pt;color:black'>Rich Cooper<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=3 color=black face="Times New Roman"><span
style='font-size:12.0pt;color:black'>EnglishLogicKernel.com</span></font><font
color=blue><span style='color:blue'><o:p></o:p></span></font></p>
<p class=MsoNormal><font size=3 color=black face="Times New Roman"><span
style='font-size:12.0pt;color:black'>Rich AT EnglishLogicKernel DOT com</span></font><font
color=blue><span style='color:blue'><o:p></o:p></span></font></p>
<p class=MsoNormal><font size=3 color=black face="Times New Roman"><span
style='font-size:12.0pt;color:black'>9 4 9 \ 5 2 5 - 5 7 1 2</span></font><o:p></o:p></p>
</div>
<div>
<div class=MsoNormal align=center style='text-align:center'><font size=3
face="Times New Roman"><span style='font-size:12.0pt'>
<hr size=2 width="100%" align=center tabindex=-1>
</span></font></div>
<p class=MsoNormal><b><font size=2 face=Tahoma><span style='font-size:10.0pt;
font-family:Tahoma;font-weight:bold'>From:</span></font></b><font size=2
face=Tahoma><span style='font-size:10.0pt;font-family:Tahoma'> corpora-bounces@uib.no
[mailto:corpora-bounces@uib.no] <b><span style='font-weight:bold'>On Behalf Of </span></b>Ruvan
Weerasinghe<br>
<b><span style='font-weight:bold'>Sent:</span></b> Sunday, July 03, 2011 6:47
PM<br>
<b><span style='font-weight:bold'>To:</span></b> tpederse@d.umn.edu<br>
<b><span style='font-weight:bold'>Cc:</span></b> nlpatumd@yahoogroups.com;
corpora@uib.no<br>
<b><span style='font-weight:bold'>Subject:</span></b> Re: [Corpora-List]
discussion on reproducibility at ACL 2011 business meeting</span></font><o:p></o:p></p>
</div>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'><o:p> </o:p></span></font></p>
<div>
<p class=MsoNormal style='margin-bottom:12.0pt'><font size=3 color=black
face=Arial><span style='font-size:12.0pt;font-family:Arial;color:black'>May be
we can address some of the issues raised by talking to the Biology
(Bioinformatics) people who seem to make publishing data and code a
precondition for publication?<br>
<br>
Regards.<br>
<br>
<br>
Ruvan Weerasinghe<br>
University of <st1:place w:st="on"><st1:PlaceName w:st="on">Colombo</st1:PlaceName>
<st1:PlaceType w:st="on">School</st1:PlaceType></st1:place> of Computing<br>
<st1:City w:st="on"><st1:place w:st="on">Colombo</st1:place></st1:City> 00700,<br>
<st1:country-region w:st="on"><st1:place w:st="on">Sri Lanka</st1:place></st1:country-region>.<br>
<br>
Web: http://www.ucsc.lk<br>
Phone: +94112158953; Fax: +94112587239<span name=x><o:p></o:p></span></font></p>
</span>
<div class=MsoNormal align=center style='text-align:center'><font size=3
color=black face=Arial><span style='font-size:12.0pt;font-family:Arial;
color:black'>
<hr size=2 width="100%" align=center id=zwchr>
</span></font></div>
<blockquote style='border:none;border-left:solid #1010FF 1.5pt;padding:0in 0in 0in 4.0pt;
margin-left:3.75pt;margin-top:5.0pt;margin-bottom:5.0pt'
mce_style="border-left:2px solid #1010ff;margin-left:5px;padding-left:5px;">
<p class=MsoNormal><b><font size=3 color=black face=Arial><span
style='font-size:12.0pt;font-family:Arial;color:black;font-weight:bold'>From: </span></font></b><font
color=black face=Arial><span style='font-family:Arial;color:black'>"Ted
Pedersen" <tpederse@d.umn.edu><br>
<b><span style='font-weight:bold'>To: </span></b>corpora@uib.no<br>
<b><span style='font-weight:bold'>Cc: </span></b>nlpatumd@yahoogroups.com<br>
<b><span style='font-weight:bold'>Sent: </span></b>Sunday, July 3, 2011
11:40:05 PM<br>
<b><span style='font-weight:bold'>Subject: </span></b>[Corpora-List] discussion
on reproducibility at ACL 2011
business meeting<br>
<br>
Greetings all,<o:p></o:p></span></font></p>
<div>
<p class=MsoNormal><font size=3 color=black face=Arial><span style='font-size:
12.0pt;font-family:Arial;color:black'><o:p> </o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=3 color=black face=Arial><span style='font-size:
12.0pt;font-family:Arial;color:black'>I made a few remarks during the ACL 2011
business meeting in favor of the innovation this year on allowing submissions
of data and code along with paper submissions. I suggested this is something we
want to continue and encourage, particularly for papers submitted to the
empirical track at ACL (which is the majority of papers these days) so that we
might be able to reproduce results more easily. I had some slides prepared that
I didn't use, but I've put those here that summarize part of what I said at
least (I forgot a few points, but the gist is fairly consistent I guess...):<o:p></o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=3 color=black face=Arial><span style='font-size:
12.0pt;font-family:Arial;color:black'><o:p> </o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=3 color=black face=Arial><span style='font-size:
12.0pt;font-family:Arial;color:black'><a
href="http://www.slideshare.net/duluthted/pedersen-acl2011businessmeeting"
target="_blank"
mce_href="http://www.slideshare.net/duluthted/pedersen-acl2011businessmeeting">http://www.slideshare.net/duluthted/pedersen-acl2011businessmeeting</a><o:p></o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=3 color=black face=Arial><span style='font-size:
12.0pt;font-family:Arial;color:black'><o:p> </o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=3 color=black face=Arial><span style='font-size:
12.0pt;font-family:Arial;color:black'>There were quite a few comments
thereafter and I took a few notes, and I guess I thought it would be possibly
useful to preserve these "for the record" at least, since I think
that discussion raised many of the common concerns about this issue. It might
also be an opportunity for folks to follow up or at least continue
thinking. <o:p></o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=3 color=black face=Arial><span style='font-size:
12.0pt;font-family:Arial;color:black'><o:p> </o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=3 color=black face=Arial><span style='font-size:
12.0pt;font-family:Arial;color:black'>Below are the comments, approximately in
the order made....note that I'm trying here to simply reproduce the gist of
comments, and not offer any opinion on them. I think it was great there was
such an extensive discussion, and I guess I just wanted to note that and
preserve it as best I could. If anyone feels like they have been misquoted,
forgotten, or misunderstood, please feel free to jump in and elaborate. <o:p></o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=3 color=black face=Arial><span style='font-size:
12.0pt;font-family:Arial;color:black'><o:p> </o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=3 color=black face=Arial><span style='font-size:
12.0pt;font-family:Arial;color:black'>0) Speaker was in support of the
encouraging more submissions of code and data, and noted that he was happy to
see quite a few presentations at ACL where code and data were being made available. <o:p></o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=3 color=black face=Arial><span style='font-size:
12.0pt;font-family:Arial;color:black'><o:p> </o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=3 color=black face=Arial><span style='font-size:
12.0pt;font-family:Arial;color:black'>1) Data is sometimes expensive to
create (especially speech data) and releasing it after one publication may not
be in the best interests of the creators.<o:p></o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=3 color=black face=Arial><span style='font-size:
12.0pt;font-family:Arial;color:black'><o:p> </o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=3 color=black face=Arial><span style='font-size:
12.0pt;font-family:Arial;color:black'>2) Reviewing code is time consuming (and
another concern raised during the business meeting was reviewer overload, so
this certainly fit into that theme).<o:p></o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=3 color=black face=Arial><span style='font-size:
12.0pt;font-family:Arial;color:black'><o:p> </o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=3 color=black face=Arial><span style='font-size:
12.0pt;font-family:Arial;color:black'>3) It is often hard or impossible for
people in industrial settings to release code - the licensing issues are
sometimes very complex and would need to be resolved before any code was submitted.<o:p></o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=3 color=black face=Arial><span style='font-size:
12.0pt;font-family:Arial;color:black'><o:p> </o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=3 color=black face=Arial><span style='font-size:
12.0pt;font-family:Arial;color:black'>4) There could be a prize offered for the
best code / best data submitted . <o:p></o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=3 color=black face=Arial><span style='font-size:
12.0pt;font-family:Arial;color:black'><o:p> </o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=3 color=black face=Arial><span style='font-size:
12.0pt;font-family:Arial;color:black'>5) It is hard to know how to review
software.<o:p></o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=3 color=black face=Arial><span style='font-size:
12.0pt;font-family:Arial;color:black'><o:p> </o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=3 color=black face=Arial><span style='font-size:
12.0pt;font-family:Arial;color:black'>6) Maybe software could be made available
on an ACL cloud, in order to solve some licensing concerns (especially of industry)<o:p></o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=3 color=black face=Arial><span style='font-size:
12.0pt;font-family:Arial;color:black'><o:p> </o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=3 color=black face=Arial><span style='font-size:
12.0pt;font-family:Arial;color:black'>7) Code at submission time is very hard
to anonymize - maybe we need separate reviewers for code and data
(from paper).<o:p></o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=3 color=black face=Arial><span style='font-size:
12.0pt;font-family:Arial;color:black'><o:p> </o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=3 color=black face=Arial><span style='font-size:
12.0pt;font-family:Arial;color:black'>8) Simply releasing or submitting code
isn't necessarily useful (if it is bad code). How do we make sure the code is
of high quality and/or useful?<o:p></o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=3 color=black face=Arial><span style='font-size:
12.0pt;font-family:Arial;color:black'><o:p> </o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=3 color=black face=Arial><span style='font-size:
12.0pt;font-family:Arial;color:black'>9) There is a tension between having new
and exciting ideas and producing well engineered code. Put another way, there's
a tension between pushing the envelope and playing it safe. The speaker was
concerned we might be moving too far away from encouraging new ideas. <o:p></o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=3 color=black face=Arial><span style='font-size:
12.0pt;font-family:Arial;color:black'><o:p> </o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=3 color=black face=Arial><span style='font-size:
12.0pt;font-family:Arial;color:black'>10) Releasing code will in the end help
the impact of work. If you look at high impact work in our field, it often
centers around a resource (eg Penn Treebank). Releasing code can also help
people in industry, because sometimes publishing code is the only way that it
will ever get out (eg sentence alignment code from CL in 1993 by Gale and
Church)<o:p></o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=3 color=black face=Arial><span style='font-size:
12.0pt;font-family:Arial;color:black'><o:p> </o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=3 color=black face=Arial><span style='font-size:
12.0pt;font-family:Arial;color:black'>11) Have a retroactive prize after a few
years for software systems that are released and are proven to have some
impact.<o:p></o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=3 color=black face=Arial><span style='font-size:
12.0pt;font-family:Arial;color:black'><o:p> </o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=3 color=black face=Arial><span style='font-size:
12.0pt;font-family:Arial;color:black'>12) During the discussion of the new
journal, it was mentioned that maybe that could be a vehicle for releasing code
and data. <o:p></o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=3 color=black face=Arial><span style='font-size:
12.0pt;font-family:Arial;color:black'><o:p> </o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=3 color=black face=Arial><span style='font-size:
12.0pt;font-family:Arial;color:black'>I'm grateful that the ACL opened up the
business meeting to these kinds of remarks, and really appreciate both the
opportunity to say a few words, and also hear all these different views. It's
given me a lot to think about, and I just wanted to pass along my notes in the
hopes of encouraging others to do the same. Keep talking. :) <o:p></o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=3 color=black face=Arial><span style='font-size:
12.0pt;font-family:Arial;color:black'><o:p> </o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=3 color=black face=Arial><span style='font-size:
12.0pt;font-family:Arial;color:black'>Enjoy,<o:p></o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=3 color=black face=Arial><span style='font-size:
12.0pt;font-family:Arial;color:black'>Ted<br clear=all>
<br>
-- <br>
Ted Pedersen<br>
<a href="http://www.d.umn.edu/%7Etpederse" target="_blank"
mce_href="http://www.d.umn.edu/~tpederse">http://www.d.umn.edu/~tpederse</a><o:p></o:p></span></font></p>
</div>
<p class=MsoNormal><font size=3 color=black face=Arial><span style='font-size:
12.0pt;font-family:Arial;color:black'><br>
_______________________________________________<br>
UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora<br>
Corpora mailing list<br>
Corpora@uib.no<br>
http://mailman.uib.no/listinfo/corpora<o:p></o:p></span></font></p>
</blockquote>
<p class=MsoNormal><font size=3 color=black face=Arial><span style='font-size:
12.0pt;font-family:Arial;color:black'><o:p> </o:p></span></font></p>
</div>
</div>
</body>
</html>