<HTML><HEAD>
<META content=text/html;charset=ISO-8859-1 http-equiv=Content-Type></HEAD>
<BODY dir=ltr bgColor=#ffffff text=#000000>
<DIV dir=ltr>
<DIV style="FONT-SIZE: 12pt; FONT-FAMILY: 'Calibri'; COLOR: #000000">
<DIV>Hi Darren, Dekai, and all,</DIV>
<DIV> </DIV>
<DIV>Let me only add that there are a number of definitions around of what
constitutes a comparable corpus, and I think none of them is really
authorative.</DIV>
<DIV> </DIV>
<DIV>E.g. another one can be found in section 2.1 of the introductory chapter of
the book “Building and Using Comparable Corpora”. This chapter can be freely
downloaded from <A
href="http://www.springer.com/computer/ai/book/978-3-642-20127-1">http://www.springer.com/computer/ai/book/978-3-642-20127-1</A>
by clicking on “Download sample pages”.</DIV>
<DIV> </DIV>
<DIV>Kind regards,</DIV>
<DIV> </DIV>
<DIV>Reinhard</DIV>
<DIV> </DIV>
<DIV
style='FONT-SIZE: small; TEXT-DECORATION: none; FONT-FAMILY: "Calibri"; FONT-WEIGHT: normal; COLOR: #000000; FONT-STYLE: normal; DISPLAY: inline'>
<DIV style="FONT: 10pt tahoma">
<DIV> </DIV>
<DIV style="BACKGROUND: #f5f5f5">
<DIV style="font-color: black"><B>From:</B> <A title=dekai@cs.ust.hk
href="mailto:dekai@cs.ust.hk">Dekai Wu</A> </DIV>
<DIV><B>Sent:</B> Monday, June 16, 2014 8:01 PM</DIV>
<DIV><B>To:</B> <A title=Corpora@uib.no href="mailto:Corpora@uib.no">corpora</A>
</DIV>
<DIV><B>Subject:</B> Re: [Corpora-List] Seeking for a free comparable
corpus</DIV></DIV></DIV>
<DIV> </DIV></DIV>
<DIV
style='FONT-SIZE: small; TEXT-DECORATION: none; FONT-FAMILY: "Calibri"; FONT-WEIGHT: normal; COLOR: #000000; FONT-STYLE: normal; DISPLAY: inline'>Hi
Darren,<BR><BR>In a note at <A
href="http://www.cs.ust.hk/%7Edekai/library/WU_Dekai/nonparallel.html">http://www.cs.ust.hk/~dekai/library/WU_Dekai/nonparallel.html</A>
you can find a systematized overview of relevant discussions on these
terminology issues, synthesized from some surveys within some papers by Pascale
Fung (who is of course one of the folks who pioneered research in all typies of
non-parallel corpora). At least four different levels of (non)parallelism are
identified:<BR><BR>parallel corpus<BR>noisy parallel corpus<BR>comparable
corpus<BR>quasi-comparable (very-non-parallel) corpus<BR><BR>The note includes
an HTML table summarizing the differences (probably can't safely insert the HTML
table here, as corpora-list tends to turn HTML formatted stuff into plain
text).<BR><BR>References<BR><BR>Pascale Fung & Percy Cheung (2004). Mining
very-non-parallel corpora: Parallel sentence and lexicon extraction via
bootstrapping and EM. In Dekang Lin and Dekai Wu (editors), Proceedings of the
2004 Conference on Empirical Methods in Natural Language Processing (EMNLP
2004). Barcelona, Spain: July 2004.<BR><BR>Pascale Fung & Percy Cheung
(2004). Multi-level bootstrapping for extracting parallel sentences from a
quasi-comparable Corpus. In Proceedings of the 20th International Conference on
Computational Linguistics (COLING 2004). Geneva, Switzerland: August
2004.<BR><BR>Dekai Wu & Pascale Fung (2005). Inversion Transduction Grammar
constraints for mining parallel sentences from quasi-comparable corpora. In
Proceedings of the Second International Joint Conference on Natural Language
Processing (IJCNLP 2005), Lecture Notes in Computer Science 3651:
257-268.<BR><BR><BR>Hope this helps!<BR>-Dekai<BR>-- <BR>Dekai Wu<BR>Hong Kong
University of Science & Technology (HKUST)<BR>Human Language Technology
Center<BR>Department of Computer Science and Engineering<BR><BR><BR>John D
Burger wrote:
<BLOCKQUOTE cite=mid:E61010FA-5247-4C28-A8D1-F9FB29FFA1C5@mitre.org
type="cite"><BLOCKQUOTE type="cite"><PRE wrap="">No, articles from Wikipedia in different languages are NOT a comparable corpus, for many reasons.
First, most of the time they are a (more or less free) translation of a master/initial one.
</PRE></BLOCKQUOTE><PRE wrap=""><!---->
Even if this is true (and for most articles it's not, in my experience), the articles quickly deviate as the two language communities begin to make changes. There is of course no particular effort made to keep two articles in sync.
</PRE>
<BLOCKQUOTE type="cite"><PRE wrap="">Second, they are about the same (narrow) subject, while a comparable corpus would be about the same theme but different many subjects. Examples of comparable corpora would be: original articles in two languages about violations of human rights; or about fashion, or about complaints about health system facilities.
</PRE></BLOCKQUOTE><PRE wrap=""><!---->
In my opinion, this is a rather narrow definition of comparable corpora, and leaves out many resources that are clearly not parallel, but are nonetheless extremely useful (as evidenced by the substantial amount of research based on such corpora).
As an aside, using the category structure in two Wikipedias, it would be very easy to construct a comparable corpora in almost any domain one wished, if in fact the notion of domain-centeredness is important.
- John Burger
MITRE
On 14Jun2014, at 10:31, Diana Santos <A class=moz-txt-link-rfc2396E href="mailto:dianamsmpsantos@gmail.com">mailto:dianamsmpsantos@gmail.com</A> wrote:
</PRE>
<BLOCKQUOTE type="cite"><PRE wrap="">Hi Darren
No, articles from Wikipedia in different languages are NOT a comparable corpus, for many reasons.
First, most of the time they are a (more or less free) translation of a master/initial one.
Second, they are about the same (narrow) subject, while a comparable corpus would be about the same theme but different many subjects. Examples of comparable corpora would be: original articles in two languages about violations of human rights; or about fashion, or about complaints about health system facilities.
If you are interested in CLIR you could try the CLEF collections which were precisely created for this.
Second, a parallel corpus is not defined in terms of SENTENCE alignment, unit is a parameter for parallel. So a Wikipedia collection as the one you suggest is a parallel corpus where the unit is the wikipedia article, not the sentence.
Paralell means in a nutshell that you can put the units in direct corespondence (most of them), while comparable means that the selection criteria are the same, but you cannot pair the elements of the two coprora.
I hope to have helped.
Best
Diana
2014-06-14 16:15 GMT+02:00 Darren Cook <A class=moz-txt-link-rfc2396E href="mailto:darren@dcook.org">mailto:darren@dcook.org</A>:
</PRE>
<BLOCKQUOTE type="cite"><PRE wrap="">I'm working on Cross Language Information Retrieval based on
comparable corpora. In order to test my approach, I need a free
comparable corpus between English language and an European language.
</PRE></BLOCKQUOTE><PRE wrap="">I was just trying to understand the difference between "parallel corpus"
and "comparable corpus". Am I correct in thinking that if an article is
translated (by a professional human translator, or a machine) from one
language to another, such that there is a sentence-level correspondence,
then it is a parallel corpus. Whereas a comparable corpus is one where
the two articles were written about the same subject, but neither is a
translation of the other, and mostly the same knowledge is covered, but
a sentence-level mapping would not exist?
If so, Wikipedia sounds like an ideal source.
E.g.
<A class=moz-txt-link-freetext href="http://en.wikipedia.org/wiki/Paris">http://en.wikipedia.org/wiki/Paris</A>
<A class=moz-txt-link-freetext href="http://fr.wikipedia.org/wiki/Paris">http://fr.wikipedia.org/wiki/Paris</A>
<A class=moz-txt-link-freetext href="http://en.wikipedia.org/wiki/Association_football">http://en.wikipedia.org/wiki/Association_football</A>
<A class=moz-txt-link-freetext href="http://fr.wikipedia.org/wiki/Football">http://fr.wikipedia.org/wiki/Football</A>
etc.
Darren
--
Darren Cook, Software Researcher/Developer
My new book: Data Push Apps with HTML5 SSE
Published by O'Reilly: (ask me for a discount code!)
<A class=moz-txt-link-freetext href="http://shop.oreilly.com/product/0636920030928.do">http://shop.oreilly.com/product/0636920030928.do</A>
Also on Amazon and at all good booksellers!
_______________________________________________
UNSUBSCRIBE from this page: <A class=moz-txt-link-freetext href="http://mailman.uib.no/options/corpora">http://mailman.uib.no/options/corpora</A>
Corpora mailing list
<A class=moz-txt-link-abbreviated href="mailto:Corpora@uib.no">Corpora@uib.no</A>
<A class=moz-txt-link-freetext href="http://mailman.uib.no/listinfo/corpora">http://mailman.uib.no/listinfo/corpora</A>
_______________________________________________
UNSUBSCRIBE from this page: <A class=moz-txt-link-freetext href="http://mailman.uib.no/options/corpora">http://mailman.uib.no/options/corpora</A>
Corpora mailing list
<A class=moz-txt-link-abbreviated href="mailto:Corpora@uib.no">Corpora@uib.no</A>
<A class=moz-txt-link-freetext href="http://mailman.uib.no/listinfo/corpora">http://mailman.uib.no/listinfo/corpora</A>
</PRE></BLOCKQUOTE><PRE wrap=""><!---->
_______________________________________________
UNSUBSCRIBE from this page: <A class=moz-txt-link-freetext href="http://mailman.uib.no/options/corpora">http://mailman.uib.no/options/corpora</A>
Corpora mailing list
<A class=moz-txt-link-abbreviated href="mailto:Corpora@uib.no">Corpora@uib.no</A>
<A class=moz-txt-link-freetext href="http://mailman.uib.no/listinfo/corpora">http://mailman.uib.no/listinfo/corpora</A>
</PRE></BLOCKQUOTE><BR>
<P>
<HR>
_______________________________________________<BR>UNSUBSCRIBE from this page:
http://mailman.uib.no/options/corpora<BR>Corpora mailing
list<BR>Corpora@uib.no<BR>http://mailman.uib.no/listinfo/corpora<BR></DIV></DIV></DIV></BODY></HTML>