<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>Dear Mark,</div><div><br></div><div>As has already been suggested in this thread -- and as Alberto illustrated below -- authorship analysis has been researched in some depth and successfully by many forensic linguists worldwide. </div><div>I thought that, besides other references already suggested, your colleague might be interested in some of the work with done, such as:</div><div><br></div><div>[1] Sousa-Silva, R., Sarmento, L., Grant, T., Oliveira, E.C. & Maia, B. (2011) 'Comparing Sentence-Level Features for Authorship Analysis in Portuguese'. IN Proceedings of the Computational Processing of the Portuguese Language. </div><div>(This paper presents results on authorship analysis of newspaper editorials, and might be of particular interest your colleague - <a href="http://paginas.fe.up.pt/~niadr/PUBLICATIONS/2010/60010051.pdf">http://paginas.fe.up.pt/~niadr/PUBLICATIONS/2010/60010051.pdf</a>)</div><div><br></div><div><div>[2] Sousa-Silva, R., Laboreiro, G., Sarmento, L., Grant, T., Oliveira, E.C. & Maia, B. (2011) ''twazn me!!! ;(' Automatic Authorship Analysis of Micro-Blogging Messages'. IN R. Muñoz, A. Montoyo and E. Métais (Eds.). Lecture Notes in Computer Science 6716 Springer 2011</div><div>(Paper on authorship of micro-blogging messages - <a href="http://paginas.fe.up.pt/~niadr/PUBLICATIONS/2011/Twitter-NLDB2011.pdf">http://paginas.fe.up.pt/~niadr/PUBLICATIONS/2011/Twitter-NLDB2011.pdf</a>)</div></div><div><br></div><div>[3] Grant, T. (2010) 'Txt 4n6: Idiolect free authorship analysis'. IN M. Coulthard and A. Johnson (Eds.) The Routledge Handbook of Forensic Linguistics. London: Routledge.</div><div><br></div><div>I hope these help!</div><div><br></div><div>Regards,</div><div>Rui</div><div><br></div><div><br></div><div><br></div><br><div><div>On 18/04/2012, at 10:02, Alberto Barron Cedeño wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div>Dear Mark,<br><br><blockquote type="cite">From the numbers you mention ({6,7,8,9}-grams in common), it is very<br></blockquote>likely that the book chapters have a co-derivation relationship (either<br>one of them was considered when producing the other or both considered a<br>common source).<br><br>You both can first look at the point of view of forensic linguistics.<br>[1] considers that "the longer a phrase, the less likely you are going<br>to find anybody use it". Experts estimate that (assuming circa 40% of<br>the words in a text are lexical) documents on the same topic could share<br>around 25% of lexical words. But if two documents contain circa 60% of<br>lexical words in common, they can be considered related [2]. Obviously<br>in this case we are talking about 1-grams. For higher level n-grams the<br>expected amount of shared terms is much lower. <br><br>This fact takes us to the concept of "uniqueness": every person is<br>linguistically unique; no two people exist that express their ideas in<br>the exact same way [3]. Inspired in some slides presented by M.<br>Coulthard and M.T. Turell at PAN 2011 (see below), I tried a simple<br>"uniqueness" experiment. I took a set of phrases and split them in<br>n-grams of increasing order (0<n<14). The resulting chunk was quoted and<br>queried to a commercial search engine. I attach the results (don't worry<br>about the different colours, consider all of them as randomly selected<br>phrases): it is extremely unlikely that two sequences of text (already<br>from n=6) will occur in two presumably independent documents. You could<br>try the same exercise with the fragments you mention.<br><br>Now, what about two documents written by one single author? Table 1 in<br>[4] shows a toy experiment we carried out considering four documents<br>written by the same authors: On average only 3% of the 4-grams in two<br>documents occurred in common (versus 16% of 1-grams and 11% of 2-grams).<br>Note we are talking about documents on the same topic, by the same<br>authors.<br><br>You or your colleague might be interested in the PAN Initiative<br>(<a href="http://pan.webis.de">http://pan.webis.de</a>), where automatic plagiarism detection and<br>authorship identification tasks are included, among others. You can get<br>an overview of the different models applied to these tasks from the<br>previous editions of the lab (everything is available online). The<br>Coulthard and Turell slides I mentioned before are available from the<br>2011 edition site (PAN @ CLEF'11), accesible from the same PAN website.<br><br>[1] Coulthard, Malcolm. ‘Author Identification, Idiolect, and Linguistic<br>Uniqueness’. Applied Linguistics 25 (December 1, 2004): 431–447.<br>[2] Coulthard, M. (2010). The Linguist as Detective: Forensic<br>Applications of Language Description.<br>[<a href="http://bit.ly/madrid_lingforense">http://bit.ly/madrid_lingforense</a>], Madrid, Spain. Talk at: Jornadas<br>(In)formativas de Lingüística Forense ((In)formative Conference on<br>Forensic Linguistics).<br>[3] Coulthard, M. and Alison, J. (2007). An Introduction to Forensic<br>Linguistics: Language in Evidence. Routledge, Oxon, UK.<br>[4] Barrón-Cedeño, A., Rosso, P. On Automatic Plagiarism Detection based<br>on n-grams Comparison. In: Boughanem et al. (Eds.) ECIR 2009, LNCS 5478,<br>pp. 696-700, Springer-Verlag Berlin Heidelberg (2009) <br><br>Kind regards,<br>Alberto<br><br>-- <br>Alberto Barrón-Cedeño <br>Department of Information Systems and Computation (Ph.D. student)<br>Universidad Politécnica de Valencia<br><a href="http://www.dsic.upv.es/~lbarron">http://www.dsic.upv.es/~lbarron</a><br><br><br>On Tue, 2012-04-17 at 19:47 +0000, Mark Davies wrote:<br><blockquote type="cite">I am sending the following question on behalf of a colleague at BYU. Thanks in advance for any suggestions you have; I'll forward them to the researcher who is working on this problem.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Mark Davies, BYU<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">-------------------------------------------<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">I am working with a 250,000 word text. Within this text there are two chapters, A and B (1,200 and 2,400 words respectively). The authorship of these two chapters is unknown, but we have reason to believe to that the author(s) of A and B have a relationship that is different from the majority of the rest of the book. There are two 4-grams, three 6-grams, one 7-gram, one 8-gram, and one 9-gram shared in common in chapters A and B that appear nowhere else in the book. Intuitively it seems like there is a unique relationship between chapters A and B. <br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">The question is:<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Is there a statistical method of measuring whether the types of n-grams above establish a reasonable probability that the two texts are linked.<br></blockquote><blockquote type="cite">_______________________________________________<br></blockquote><blockquote type="cite">UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora<br></blockquote><blockquote type="cite">Corpora mailing list<br></blockquote><blockquote type="cite">Corpora@uib.no<br></blockquote><blockquote type="cite">http://mailman.uib.no/listinfo/corpora<br></blockquote><span><uniqueness_example.png></span>_______________________________________________<br>UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora<br>Corpora mailing list<br>Corpora@uib.no<br>http://mailman.uib.no/listinfo/corpora<br></div></blockquote></div><br></body></html>