<html><body><div style="color:#000; background-color:#fff; font-family:HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif;font-size:10pt"><div><span>As I recall there are so many papers used Wikipedia articles as comparable corpora in CLIR. Because in CLIR there is no need to have documents that are exact translations of each other. For our task, a collection of topically related aligned documents is enough. However, I couldn't find any free comparable corpus which is extracted from Wikipedia. Is there any free corpus extracted from Wikipedia? I know there are some comparable corpora in CLEF datasets but they aren't free. <br></span></div><div> </div><div>Best Regards,<br>Hosein Azarbonyad</div> <div class="qtdSeparateBR"><br><br></div><div style="display: block;" class="yahoo_quoted"> <div style="font-family: HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif; font-size: 10pt;"> <div
style="font-family: HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif; font-size: 12pt;"> <div dir="ltr"> <font face="Arial" size="2"> On Sunday, June 15, 2014 8:42 AM, Joel Nothman <joel@it.usyd.edu.au> wrote:<br> </font> </div> <br><br> <div class="y_msg_container"><div id="yiv2777319177"><div><div dir="ltr">Perhaps the most characteristic feature of Wikipedia is its long tail, and the apparently different features (and editorial behaviour?) of the tail and head. What is true of the most important/popular articles may rarely be true of the majority (it's unclear which we care about in this case). For example, <a rel="nofollow" shape="rect" target="_blank" href="http://downloads.schwa.org/pubs/pdf/aij10wikiner.pdf">our work in entity type classification</a> has compared training and testing on a random or a "popular" sample, each of about 2000 articles altogether. A random model achieves 92% F1 over popular
articles, but the reverse only yields 75%, although random can learn random to 90% F1. This is mostly indicative of type distributions, but no doubt editing patterns face similar discrepancies.<div>
<br clear="none"></div><div>Therefore I might guess that the more universally popular articles like [[Tennis]] are going to appear different, while the plethora of more minor entries (e.g. bands, corporations) are likely to have clearer parallels.<div>
<br clear="none"></div><div>Additionally, there will be divergence after translation (notably restructuring in the most popular articles of actively edited Wikipedias) which makes cognates (may I?) hard to identify from the current pages. Thus "clicking a random sample of languages for the page on tennis" may be made more precise if one compares a foundational edit, or perhaps the historical edit that introduced the largest portion of text to a page, to the state of the English Wikipedia equivalent <i>at that time</i>. However, the example of <a rel="nofollow" shape="rect" target="_blank" href="http://ja.wikipedia.org/w/index.php?title=%E3%83%86%E3%83%8B%E3%82%B9&oldid=356563">Japanese tennis</a> in 2004 compared to <a rel="nofollow" shape="rect" target="_blank" href="http://en.wikipedia.org/w/index.php?title=Tennis&oldid=2702021">English</a> is not very suggestive.<div>
<br clear="none"></div><div>And I recall <a rel="nofollow" shape="rect" target="_blank" href="http://storm.cis.fordham.edu/~filatova/publications.html">Elena Filatova</a> did some pioneering work in computationally exploiting parallels and differences in multilingual Wikipedia.<br clear="none">
</div></div><div><br clear="none"></div><div>Cheers,</div><div><br clear="none"></div><div class="yiv2777319177gmail_extra">Joel Nothman</div><div class="yiv2777319177gmail_extra">School of IT</div><div class="yiv2777319177gmail_extra">University of Sydney<br clear="none"><br clear="none"><div class="yiv2777319177yqt5046587770" id="yiv2777319177yqt34271"><div class="yiv2777319177gmail_quote">
On 15 June 2014 11:58, Francis Bond <span dir="ltr"><<a rel="nofollow" shape="rect" ymailto="mailto:bond@ieee.org" target="_blank" href="mailto:bond@ieee.org">bond@ieee.org</a>></span> wrote:<br clear="none"><blockquote class="yiv2777319177gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
G'day.<br clear="none">
<div class="yiv2777319177"><br clear="none">
> No, articles from Wikipedia in different languages are NOT a comparable<br clear="none">
> corpus, for many reasons<br clear="none">
><br clear="none">
</div><div class="yiv2777319177">> First, most of the time they are a (more or less free) translation of a<br clear="none">
> master/initial one.<br clear="none">
<br clear="none">
</div>Do you have a citation for this? As far as I know it is not<br clear="none">
generally true, pages are written pretty much entirely independently<br clear="none">
(at least for the English and Japanese Wikipedias which I am<br clear="none">
experienced with). I also clicked a random sample of languages for<br clear="none">
the page on tennis, and they are all very differently structured.<br clear="none">
<br clear="none">
I seem to recall a shared task on aligning sentences in wikipedia<br clear="none">
articles that found them not at all similar, but I am afraid I can't<br clear="none">
find the paper: does anyone else recall it?<br clear="none">
<span class="yiv2777319177HOEnZb"><font color="#888888"><br clear="none">
--<br clear="none">
Francis Bond <<a rel="nofollow" shape="rect" target="_blank" href="http://www3.ntu.edu.sg/home/fcbond/">http://www3.ntu.edu.sg/home/fcbond/</a>><br clear="none">
Division of Linguistics and Multilingual Studies<br clear="none">
Nanyang Technological University<br clear="none">
</font></span><div class="yiv2777319177HOEnZb"><div class="yiv2777319177h5"><br clear="none">
_______________________________________________<br clear="none">
UNSUBSCRIBE from this page: <a rel="nofollow" shape="rect" target="_blank" href="http://mailman.uib.no/options/corpora">http://mailman.uib.no/options/corpora</a><br clear="none">
Corpora mailing list<br clear="none">
<a rel="nofollow" shape="rect" ymailto="mailto:Corpora@uib.no" target="_blank" href="mailto:Corpora@uib.no">Corpora@uib.no</a><br clear="none">
<a rel="nofollow" shape="rect" target="_blank" href="http://mailman.uib.no/listinfo/corpora">http://mailman.uib.no/listinfo/corpora</a><br clear="none">
</div></div></blockquote></div></div><br clear="none"></div></div></div></div></div><br><div class="yqt5046587770" id="yqt82405">_______________________________________________<br clear="none">UNSUBSCRIBE from this page: <a shape="rect" href="http://mailman.uib.no/options/corpora" target="_blank">http://mailman.uib.no/options/corpora</a><br clear="none">Corpora mailing list<br clear="none"><a shape="rect" ymailto="mailto:Corpora@uib.no" href="mailto:Corpora@uib.no">Corpora@uib.no</a><br clear="none"><a shape="rect" href="http://mailman.uib.no/listinfo/corpora" target="_blank">http://mailman.uib.no/listinfo/corpora</a></div><br><br></div> </div> </div> </div> </div></body></html>