[Corpora-List] Seeking for a free comparable corpus

Mon Jun 16 18:01:25 UTC 2014

Hi Darren,

In a note at 
http://www.cs.ust.hk/~dekai/library/WU_Dekai/nonparallel.html 
<http://www.cs.ust.hk/%7Edekai/library/WU_Dekai/nonparallel.html> you 
can find a systematized overview of relevant discussions on these 
terminology issues, synthesized from some surveys within some papers by 
Pascale Fung (who is of course one of the folks who pioneered research 
in all typies of non-parallel corpora). At least four different levels 
of (non)parallelism are identified:

parallel corpus
noisy parallel corpus
comparable corpus
quasi-comparable (very-non-parallel) corpus

The note includes an HTML table summarizing the differences (probably 
can't safely insert the HTML table here, as corpora-list tends to turn 
HTML formatted stuff into plain text).

References

Pascale Fung & Percy Cheung (2004). Mining very-non-parallel corpora: 
Parallel sentence and lexicon extraction via bootstrapping and EM. In 
Dekang Lin and Dekai Wu (editors), Proceedings of the 2004 Conference on 
Empirical Methods in Natural Language Processing (EMNLP 2004). 
Barcelona, Spain: July 2004.

Pascale Fung & Percy Cheung (2004). Multi-level bootstrapping for 
extracting parallel sentences from a quasi-comparable Corpus. In 
Proceedings of the 20th International Conference on Computational 
Linguistics (COLING 2004). Geneva, Switzerland: August 2004.

Dekai Wu & Pascale Fung (2005). Inversion Transduction Grammar 
constraints for mining parallel sentences from quasi-comparable corpora. 
In Proceedings of the Second International Joint Conference on Natural 
Language Processing (IJCNLP 2005), Lecture Notes in Computer Science 
3651: 257-268.

Hope this helps!
-Dekai
-- 
Dekai Wu
Hong Kong University of Science & Technology (HKUST)
Human Language Technology Center
Department of Computer Science and Engineering

John D Burger wrote:
>> No, articles from Wikipedia in different languages are NOT a comparable corpus, for many reasons.
>>
>> First, most of the time they are a (more or less free) translation of a master/initial one.
>>     
>
> Even if this is true (and for most articles it's not, in my experience), the articles quickly deviate as the two language communities begin to make changes. There is of course no particular effort made to keep two articles in sync.
>
>   
>> Second, they are about the same (narrow) subject, while a comparable corpus would be about the same theme but different many  subjects. Examples of comparable corpora would be: original articles in two languages about violations of human rights; or about fashion, or about complaints about health system facilities.
>>     
>
> In my opinion, this is a rather narrow definition of comparable corpora, and leaves out many resources that are clearly not parallel, but are nonetheless extremely useful (as evidenced by the substantial amount of research based on such corpora).
>
> As an aside, using the category structure in two Wikipedias, it would be very easy to construct a comparable corpora in almost any domain one wished, if in fact the notion of domain-centeredness is important.
>
> - John Burger
>   MITRE
>
> On 14Jun2014, at 10:31, Diana Santos <dianamsmpsantos at gmail.com> wrote:
>
>   
>> Hi Darren
>> No, articles from Wikipedia in different languages are NOT a comparable corpus, for many reasons.
>>
>> First, most of the time they are a (more or less free) translation of a master/initial one.
>> Second, they are about the same (narrow) subject, while a comparable corpus would be about the same theme but different many  subjects. Examples of comparable corpora would be: original articles in two languages about violations of human rights; or about fashion, or about complaints about health system facilities.
>>
>> If you are interested in CLIR you could try the CLEF collections which were precisely created for this.
>>
>> Second, a parallel corpus is not defined in terms of SENTENCE alignment, unit is a parameter for parallel. So a Wikipedia collection as the one you suggest is a parallel corpus where the unit is the wikipedia article, not the sentence.
>>
>> Paralell means in a nutshell that you can put the units in direct corespondence (most of them), while comparable means that the selection criteria are the same, but you cannot pair the elements of the two coprora.
>>
>> I hope to have helped.
>> Best
>> Diana
>>
>>
>> 2014-06-14 16:15 GMT+02:00 Darren Cook <darren at dcook.org>:
>>     
>>> I'm working on Cross Language Information Retrieval based on
>>> comparable corpora. In order to test my approach, I need a free
>>> comparable corpus between English language and an European language.
>>>       
>> I was just trying to understand the difference between "parallel corpus"
>> and "comparable corpus". Am I correct in thinking that if an article is
>> translated (by a professional human translator, or a machine) from one
>> language to another, such that there is a sentence-level correspondence,
>> then it is a parallel corpus. Whereas a comparable corpus is one where
>> the two articles were written about the same subject, but neither is a
>> translation of the other, and mostly the same knowledge is covered, but
>> a sentence-level mapping would not exist?
>>
>> If so, Wikipedia sounds like an ideal source.
>> E.g.
>>   http://en.wikipedia.org/wiki/Paris
>>   http://fr.wikipedia.org/wiki/Paris
>>
>>   http://en.wikipedia.org/wiki/Association_football
>>   http://fr.wikipedia.org/wiki/Football
>>
>> etc.
>>
>> Darren
>>
>>
>> --
>> Darren Cook, Software Researcher/Developer
>> My new book: Data Push Apps with HTML5 SSE
>> Published by O'Reilly: (ask me for a discount code!)
>>   http://shop.oreilly.com/product/0636920030928.do
>> Also on Amazon and at all good booksellers!
>>
>> _______________________________________________
>> UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
>> Corpora mailing list
>> Corpora at uib.no
>> http://mailman.uib.no/listinfo/corpora
>>
>> _______________________________________________
>> UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
>> Corpora mailing list
>> Corpora at uib.no
>> http://mailman.uib.no/listinfo/corpora
>>     
>
>
> _______________________________________________
> UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
> Corpora mailing list
> Corpora at uib.no
> http://mailman.uib.no/listinfo/corpora
>
>
>   

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listserv.linguistlist.org/pipermail/corpora/attachments/20140617/630536a9/attachment.htm>
-------------- next part --------------
_______________________________________________
UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
Corpora mailing list
Corpora at uib.no
http://mailman.uib.no/listinfo/corpora