<div>Hi, Adam,</div>
<div> </div>
<div>Well, I think and believe, so to speak, that that may depend on just how large the numbers are: if we're talking about *the* most frequent words (pun intended) (eg, so-called 'stop words'), we can be fairly sure of finding at least similar frequencies. I'll grant you some exceptions for, say, that book written so as not to have any e's. If that's your corpus (and other similar texts), then 'the' won't appear, and so won't be the commonest word. For any other non-artificial corpus, it would be surprising indeed if 'the' weren't the most, or at worst second-most, common word (of course, talking about English corpora). On the other hand, we would *expect* a decent amount of variation in place of, say, the twentieth word, and not be surprised if in another biggish corpus it came in 15th or 25th. If it came in, say, 1000th in a second corpus, we could be damned sure that it's a very 'bursty' sort of word, and we just got unlucky (?or lucky?) in our first corpus. Likewise, the further down the list we go, the bigger we would expect the variation to be. </div>
<div> </div>
<div>I'm not disputing, really, what you say. It's partly a question of focus, perhaps, and the question of 'burstiness' is obviously very important in, say, dividing up subcorpora by the field they belong to. I'm just saying, I guess, that linguists may very well study, in depth, 'stop words', while corpus linguists are most unlikely to study them, since they usually look for other things in their corpora.</div>
<div> </div>
<div>Jim<br><br> </div>
<div><span class="gmail_quote">On 5/15/09, <b class="gmail_sendername">Adam Kilgarriff</b> <<a href="mailto:adam@lexmasterclass.com">adam@lexmasterclass.com</a>> wrote:</span>
<blockquote class="gmail_quote" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid">
<div>Sorry, Jim, law of large numbers is not relevant as it assumes independent effects. In language, effects are never independent (for details see <a onclick="return top.js.OpenExtLink(window,event,this)" href="http://kilgarriff.co.uk/Publications/2005-K-lineer.pdf" target="_blank">Language is never ever ever random.</a> )</div>
<div> </div>
<div>So the short answer to Tina's question - </div><span class="q">
<blockquote dir="ltr" style="MARGIN-RIGHT: 0px">
<div><font face="Arial" size="2">"Could you tell me what the frequency would be in a corpus of 1 million if I extrapolated </font><font face="Arial" size="2">from the frequency of 20 in a corpus of 300K?"</font></div>
</blockquote></span>
<div dir="ltr"> is "no". It all depends on the structure and composition of the 300,000 corpus, the structure and composition of the (probably hypothetical) 1m corpus, how 'bursty' the word is, and how the two corpora relate to each other. (For burstiness, see Ken Church's "<strong>Empirical estimates of adaptation: the chance of two noriegas is closer to p/2 than p<sup>2"</sup></strong> ) If the word in question is term-like and all 20 occurrences come from one doc, then it is likely that the frequency in the 1m corpus will be 20 (if we include the doc in the first corpus) or 0 (if we don't). </div>
<div dir="ltr"> </div>
<div dir="ltr">Extrapolation of frequencies from corpora is a risky business, highly dependent on the sampling procedure for the corpus and the nature of the term for which the frequency is being extrapolated. It's generally safer to extrapolate on the basis of document frequencies (eg, how many docs does the word/term appear in) than word/term frequencies, though still, think hard about the nature of the corpus and its claims to representativeness.</div>
<blockquote dir="ltr" style="MARGIN-RIGHT: 0px">
<div> Adam</div></blockquote>
<div class="gmail_quote">2009/5/15 James L. Fidelholtz <span dir="ltr"><<a onclick="return top.js.OpenExtLink(window,event,this)" href="mailto:fidelholtz@gmail.com" target="_blank">fidelholtz@gmail.com</a>></span></div>
<div><span class="e" id="q_12146379c3318d2c_3">
<div class="gmail_quote"> </div>
<blockquote class="gmail_quote" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid">
<div>Hi, Lluis, Tina, & Al.,</div>
<div> </div>
<div>Firstly, the math is a little kinky (though Lluis is right--it's roughly OK): it should be 20 * 1M/300K, or 63.3.... </div>
<div> </div>
<div>The point Lluis makes about the corpus containing more rarer words as we augment the size of the corpus is, of course, correct. Nevertheless (here I haven't done much work, but I just appeal to common sense and the 'law of large numbers' (not sure this is relevant, but 300K is a *pretty* large number)), we should expect, even with more obscure words to muddy up the picture, that the percentage of *common* words in the 300K corpus should be roughly the same in a corpus of 1M words, especially (but not quite only, for the more common words) if the corpora are selected from similar universes. Naturally, different selection criteria might affect even very common words, and it has been shown many times that the 'rarer' the words are, the more variable the exact percentage can be, but I wouldn't expect a priori that ever bigger corpora should lower the percentages of common (or even necessarily of rare) words. Indeed, for the hapax legomena, say, that enter in the new 'complement' to the corpus, their percentage even *increases* from 0 to 0.0001, correspondingly more for the other new words.</div>
<div> </div>
<div>Of course there can always be variations in the percentages. But, equally always, we *expect* that our sampling of the universe will give us for a word W something reasonably close to its real percentage frequency. And that when we repeat the process (or augment it), we will again get reasonably close to its 'real' frequency, so that we expect both frequencies to be close to each other. The real world often lets us down (and don't bet the family farm on any of this), but I guess statisticians tend to be optimists in this regard. And mathematicians even more (after all, we have an edge, and so tend to gain 5 family farms for each one we lose). In this sense, think: Bell curve, which, with the appropriate tweaks, is the exact representation of what our expectations should be in a particular case.</div>
<div> </div>
<div>Jim
<div> </div>
<div> </div></div>
<div>
<div>
<div></div>
<div><span class="gmail_quote">On 5/15/09, <b class="gmail_sendername">Lluís Padró</b> <<a onclick="return top.js.OpenExtLink(window,event,this)" href="mailto:padro@lsi.upc.edu" target="_blank">padro@lsi.upc.edu</a>> wrote:</span> </div>
</div>
<blockquote class="gmail_quote" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid">
<div>
<div></div>
<div>
<div bgcolor="#ffffff" text="#000000">En/na Tina Waldman ha escrit: <span>
<blockquote type="cite">
<div><font face="Arial" size="2">Dear members</font></div>
<div><font face="Arial" size="2">Could you tell me what the frequency would be in a corpus of 1 million if I extrapolated from the frequency of 20 in a corpus of 300K?</font></div>
<div> </div>
<div><font face="Arial" size="2">Would it be 60 - 20 x 3 ?</font></div>
<div> </div></blockquote></span> As a rough estimate, that may work.
<div> </div>
<div> </div>
<div> Nevertheless, due to Zipf's laws, when you go from 300K to 1M, you're getting lots of previously unseen words with very low frequencies, but they modify the proability distribution</div>
<div> </div>
<div> For this and other reasons, relative frequencies seem to be less stable than that when you use larger corpora.</div>
<div> </div>
<div> You can find out more about it in:</div>
<div>Baroni M., Evert S., "Words and echoes: assessing and mitigating the non-randomness problem in word frequency distribution modeling". In:Proceedings of ACL 2007, East Stroudsburg PA: ACL, 2007. p. 904-911, Atti del convegno: "Association for Computational Linguistics (ACL)", Prague, 23rd-30th June 2007.</div>
<div> </div>
<div> best,</div>
<div> </div>
<div> </div>
<div>--
<div>
<table>
<tbody>
<tr>
<td align="middle" colspan="2">
<hr width="100%">
</td></tr>
<tr>
<td valign="top"><font color="#0000aa"><b>Lluís Padró</b></font>
<div><font color="#2f2f66">Despatx Ω-S112
<div>Campus Nord UPC
<div>C/ Jordi Girona 1-3
<div>08034 Barcelona, Spain</div></div></div></font></div></td></tr></tbody></table></div></div></div>
<table>
<tbody>
<tr>
<td valign="top"><font color="#0000aa">Tel: <tt><font size="+1">+34 934 134 015</font></tt>
<div>Fax: <tt><font size="+1">+34 934 137 833</font></tt></div></font>
<div><tt><font size="+1"><a onclick="return top.js.OpenExtLink(window,event,this)" href="mailto:padro@lsi.upc.es" target="_blank">padro@lsi.upc.edu</a>
<div><a onclick="return top.js.OpenExtLink(window,event,this)" href="http://www.lsi.upc.es/~padro" target="_blank">www.lsi.upc.edu/~padro</a></div></font></tt></div></td></tr></tbody></table></div></div>
<table>
<tbody>
<tr>
<td align="middle" colspan="2">
<hr width="100%">
<font color="#2f2f66">UNIVERSITAT POLITÈCNICA DE CATALUNYA
<div>Dept. <a onclick="return top.js.OpenExtLink(window,event,this)" href="http://www.lsi.upc.es/" target="_blank">Llenguatges i Sistemes Informàtics</a>
<div><a onclick="return top.js.OpenExtLink(window,event,this)" href="http://www.talp.upc.es/" target="_blank">TALP</a> Research Center</div></div></font>
<hr width="100%">
</td></tr></tbody></table></blockquote></div>
<div> </div>_______________________________________________
<div>Corpora mailing list</div>
<div><a onclick="return top.js.OpenExtLink(window,event,this)" href="mailto:Corpora@uib.no" target="_blank">Corpora@uib.no</a></div>
<div><a onclick="return top.js.OpenExtLink(window,event,this)" href="http://mailman.uib.no/listinfo/corpora" target="_blank">http://mailman.uib.no/listinfo/corpora</a></div>
<div> </div>
<div> </div></blockquote><font color="#888888">
<div> </div>
<div clear="all"> </div>
<div>-- </div>
<div>James L. Fidelholtz</div>
<div>Posgrado en Ciencias del Lenguaje</div>
<div>Instituto de Ciencias Sociales y Humanidades</div>
<div>Benemérita Universidad Autónoma de Puebla, MÉXICO </div></font>
<div>_______________________________________________</div>
<div>Corpora mailing list</div>
<div><a onclick="return top.js.OpenExtLink(window,event,this)" href="mailto:Corpora@uib.no" target="_blank">Corpora@uib.no</a></div>
<div><a onclick="return top.js.OpenExtLink(window,event,this)" href="http://mailman.uib.no/listinfo/corpora" target="_blank">http://mailman.uib.no/listinfo/corpora</a></div>
<div> </div>
<div> </div>
<div> </div>
<div clear="all"> </div>
<div></div>
<div>-- </div></span></div>
<div>================================================</div>
<div>Adam Kilgarriff <a onclick="return top.js.OpenExtLink(window,event,this)" href="http://www.kilgarriff.co.uk/" target="_blank">http://www.kilgarriff.co.uk</a> </div><span class="sg">
<div>Lexical Computing Ltd <a onclick="return top.js.OpenExtLink(window,event,this)" href="http://www.sketchengine.co.uk/" target="_blank">http://www.sketchengine.co.uk</a></div>
<div>Lexicography MasterClass Ltd <a onclick="return top.js.OpenExtLink(window,event,this)" href="http://www.lexmasterclass.com/" target="_blank">http://www.lexmasterclass.com</a></div>
<div>Universities of Leeds and Sussex <a onclick="return top.js.OpenExtLink(window,event,this)" href="mailto:adam@lexmasterclass.com" target="_blank">adam@lexmasterclass.com</a></div>
<div>================================================</div>
<div> </div></span></blockquote></div><br><br clear="all"><br>-- <br>James L. Fidelholtz<br>Posgrado en Ciencias del Lenguaje<br>Instituto de Ciencias Sociales y Humanidades<br>Benemérita Universidad Autónoma de Puebla, MÉXICO