<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:st1="urn:schemas-microsoft-com:office:smarttags" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=Content-Type content="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 11 (filtered medium)">
<!--[if !mso]>
<style>
v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style>
<![endif]--><o:SmartTagType
namespaceuri="urn:schemas-microsoft-com:office:smarttags" name="country-region"/>
<o:SmartTagType namespaceuri="urn:schemas-microsoft-com:office:smarttags"
name="PostalCode"/>
<o:SmartTagType namespaceuri="urn:schemas-microsoft-com:office:smarttags"
name="place"/>
<o:SmartTagType namespaceuri="urn:schemas-microsoft-com:office:smarttags"
name="Street"/>
<o:SmartTagType namespaceuri="urn:schemas-microsoft-com:office:smarttags"
name="City"/>
<o:SmartTagType namespaceuri="urn:schemas-microsoft-com:office:smarttags"
name="address"/>
<!--[if !mso]>
<style>
st1\:*{behavior:url(#default#ieooui) }
</style>
<![endif]-->
<style>
<!--
/* Font Definitions */
@font-face
{font-family:"MS Mincho";
panose-1:2 2 6 9 4 2 5 8 3 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:"\@MS Mincho";
panose-1:0 0 0 0 0 0 0 0 0 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman";}
a:link, span.MsoHyperlink
{color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{color:blue;
text-decoration:underline;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:Arial;
color:navy;}
@page Section1
{size:595.3pt 841.9pt;
margin:72.0pt 90.0pt 72.0pt 90.0pt;}
div.Section1
{page:Section1;}
-->
</style>
</head>
<body lang=EN-GB link=blue vlink=blue>
<div class=Section1>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>Hi Sean,<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>You could start with something
straightforward such as evaluating coverage against a lexicon across the
various corpora. In LREC2004, we looked at this when evaluating a semantic
tagger over written, spoken, domain-specific and historical corpora:<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>Piao, Scott S. L., Paul Rayson, Dawn
Archer, Tony McEnery (2004). Evaluating Lexical Resources for A Semantic
Tagger. In proceedings of 4th International Conference on Language Resources
and Evaluation (LREC 2004), May 2004, Lisbon, Portugal, Volume II, pp. 499-502.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><a
href="http://www.comp.lancs.ac.uk/computing/users/paul/publications/pram_lrec04.pdf">http://www.comp.lancs.ac.uk/computing/users/paul/publications/pram_lrec04.pdf</a><o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>Related to misspellings (actually
historical variants) and grammatical variation across corpora (over time), we’ve
recently compared the accuracy of CLAWS on modern and historical corpora:<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>Rayson, P., Archer, D., Baron, A.,
Culpeper, J. and Smith, N. (2007). Tagging the Bard: Evaluating the accuracy of
a modern POS tagger on Early Modern English corpora. In proceedings of Corpus
Linguistics 2007, July 27-30, <st1:place w:st="on"><st1:City w:st="on">University
of Birmingham</st1:City>, <st1:country-region w:st="on">UK</st1:country-region></st1:place>.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><a
href="http://ucrel.lancs.ac.uk/publications/CL2007/paper/192_Paper.pdf">http://ucrel.lancs.ac.uk/publications/CL2007/paper/192_Paper.pdf</a><o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>Regards,<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>Paul.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<div>
<div>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>Dr. Paul Rayson<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>Director of UCREL<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>Computing Department, Infolab21, <st1:address
w:st="on"><st1:Street w:st="on">South Drive</st1:Street>, <st1:City w:st="on">Lancaster</st1:City></st1:address>
University, <st1:place w:st="on"><st1:City w:st="on">Lancaster</st1:City>, <st1:PostalCode
w:st="on">LA1 4WA</st1:PostalCode>, <st1:country-region w:st="on">UK</st1:country-region></st1:place>.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>Web: <a
href="http://www.comp.lancs.ac.uk/computing/users/paul/"><font color=purple><span
style='color:purple'>http://www.comp.lancs.ac.uk/computing/users/paul/</span></font></a><o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>Tel: +44 1524 510357 Fax: +44 1524 510492<o:p></o:p></span></font></p>
</div>
</div>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<div>
<div class=MsoNormal align=center style='text-align:center'><font size=3
face="Times New Roman"><span lang=EN-US style='font-size:12.0pt'>
<hr size=2 width="100%" align=center tabindex=-1>
</span></font></div>
<p class=MsoNormal><b><font size=2 face=Tahoma><span lang=EN-US
style='font-size:10.0pt;font-family:Tahoma;font-weight:bold'>From:</span></font></b><font
size=2 face=Tahoma><span lang=EN-US style='font-size:10.0pt;font-family:Tahoma'>
corpora-bounces@uib.no [mailto:corpora-bounces@uib.no] <b><span
style='font-weight:bold'>On Behalf Of </span></b>Adam Kilgarriff<br>
<b><span style='font-weight:bold'>Sent:</span></b> 03 February 2008 06:56<br>
<b><span style='font-weight:bold'>To:</span></b> Sean Igo<br>
<b><span style='font-weight:bold'>Cc:</span></b> CORPORA@uib.no<br>
<b><span style='font-weight:bold'>Subject:</span></b> Re: [Corpora-List]
Metrics for corpus "parseability"</span></font><span lang=EN-US><o:p></o:p></span></p>
</div>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'><o:p> </o:p></span></font></p>
<div>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'>Sean,<o:p></o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'> <o:p></o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'>very interesting question. We are approaching a similar task
with similar methods: using RASP, if we set the timeout threshold to 1s, how
many sentences time out? We're also planning something similar with
Clark&Curran parser.<o:p></o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'> <o:p></o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'>> Are there standard or widely accepted metrics for describing the<br>
> well-behavedness of corpora?<br>
<o:p></o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'>The answer is, I think, a resounding 'no'. There is
disappointingly little work on systematically comparing corpora, or making
objective general observations of one corpus in comparison to others.
(Citations proving me wrong are most welcome. I'm aware of Sekine,
Roland and Jurafsky, Cavaglia, also work on genre by eg Karlgren, Santini,
Sharoff, which touches on the topic)<o:p></o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'><br>
Adam<br>
<br>
================================================<br>
Adam
Kilgarriff <a
href="http://www.kilgarriff.co.uk">http://www.kilgarriff.co.uk</a> <br>
Lexical Computing
Ltd
<a href="http://www.sketchengine.co.uk">http://www.sketchengine.co.uk</a><br>
Lexicography MasterClass Ltd <a
href="http://www.lexmasterclass.com">http://www.lexmasterclass.com</a><br>
Universities of Leeds and <st1:country-region w:st="on"><st1:place w:st="on">Sussex</st1:place></st1:country-region>
<a href="mailto:adam@lexmasterclass.com">adam@lexmasterclass.com</a><br>
================================================ <o:p></o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><span class=gmailquote><font size=3 face="Times New Roman"><span
style='font-size:12.0pt'>On 01/02/2008, <b><span style='font-weight:bold'>Sean
Igo</span></b> <<a href="mailto:sgigo@xmission.com">sgigo@xmission.com</a>>
wrote:</span></font></span> <o:p></o:p></p>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'>Good day,<br>
<br>
I'm working on a project in which we are attempting to characterize a<br>
few different corpora according to how "well-behaved" they are. That
is,<br>
we want to show that some are more amenable in particular to parsing and<br>
part-of-speech tagging than others. Some of the corpora consist of<br>
complete, grammatical sentences and others are telegraphic, fragmentary<br>
text including a large number of abbreviations and misspellings.<br>
<br>
One approach I've tried is to tag and parse each of the corpora with the<br>
Stanford tagger and parser, generating ranked lists of the unique tokens<br>
and tags and looking for certain errors / warnings / phrase structures<br>
in the parser output. For instance, I'm counting how many sentences the<br>
parser had to retry, how many it failed to find any parse for, how many<br>
it ran out of memory while processing, and how many FRAG (sentence<br>
fragment) phrases are found in the parser output.<br>
<br>
Are there standard or widely accepted metrics for describing the<br>
well-behavedness of corpora?<br>
<br>
Many thanks,<br>
Sean Igo<br>
<br>
_______________________________________________<br>
Corpora mailing list<br>
<a href="mailto:Corpora@uib.no">Corpora@uib.no</a><br>
<a href="http://mailman.uib.no/listinfo/corpora">http://mailman.uib.no/listinfo/corpora</a><o:p></o:p></span></font></p>
</div>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'><br>
<br clear=all>
<br>
-- <o:p></o:p></span></font></p>
</div>
</body>
</html>