<div class="gmail_quote">Hi all,<div><br></div><div>As a disclaimer, I have not worked with any of the tokenizers. For the type of results originally reported, however, I do have a suggestion for a possible partial explanation, based on some experience with Spanish. There is a real stylistic rule in Spanish which makes speakers and especially writers avoid repeating the same 'content word' within the same or contiguous sentences or clauses, using instead a synonym or paraphrase (incidentally, this latter may be a partial explanation for the well-known fact that Spanish sentences are longer than English ones on average). We might expect, then, to find a larger number of words in smaller comparable corpora in Spanish than in English. *If* Japanese had some similar stylistic tendencies, then that is likely to be a part of such results. <br>
</div><div><br></div><div>It would be instructive to examine Spanish vs. English in this regard, which would show that pragmatic considerations (word choice restrictions) may influence distributions as much as 'grammatical' ones (like the German 'monster nouns' vs. the English in compound nouns that John mentioned).</div>
<div><br></div><div>Jim </div><div><div><div></div><div class="h5"><br><div class="gmail_quote">On Thu, Aug 12, 2010 at 8:17 AM, Jim Breen <span dir="ltr"><<a href="mailto:jimbreen@gmail.com" target="_blank">jimbreen@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>On 12 August 2010 18:45, Emmanuel Prochasson wrote:<br>
> I am working on a trilingual comparable corpus of French/English and<br>
> Japanese. I am running a simple word count on each part of the corpus<br>
> but found surprising results for Japanese.<br>
><br>
> For each part, I count the total number of words and the number of<br>
> /unique words/, that is I count every words only once, even if they<br>
> appear 1, 5 or 100 times. I POS-tagged each part of the corpus and<br>
> only keep the lemmatized version of every words (to group different<br>
> flexion of one words). Furthermore, I only focus on nouns, keeping the<br>
</div>> "??:??" tag for Japanese (noun:general) and all noun (including proper<br>
<div>> nouns) in French/English. I use MeCab for Japanese and TreeTagger for<br>
> French/English.<br>
><br>
> Here are the results (Unique words/Total words).<br>
> Japanese : 189,798 / 5,174,800<br>
> English : 66,821 / 4,589,465<br>
> French : 23,970 / 1,796,183<br>
><br>
> What surprises me is that the number of unique nouns in Japanese is<br>
> three times the number of unique nouns in English, even though the<br>
> difference of total number of words in both language is not that large<br>
</div>> (the ratio for French/English is more consistent for example).<br>
<div>><br>
> As far as I can tell, the tokenization/POS-tagging looks /ok/ (ie : I<br>
> checked using google translate, it seems to make sense, but my lack of<br>
> skill in Japanese prevents me from investigating deeper).<br>
<br>
</div>Japanese morphological analysers such as MeCab, Chasen, etc. tend to<br>
over-split so that what might be considered a single word in English or<br>
French may end up as two or three elements in MeCab's output. For<br>
example, "industrialization" is "jigyouka" in Japanese. MeCab (depending<br>
on which lexicon you are using) will typically break it into "jigyou" and "ka",<br>
i.e. "industry" and "ization". Both are tagged as nouns; noun-general<br>
and noun-suffix.<br>
<br>
That said, I would not expect a factor of three difference.<br>
<br>
As a test, I put the Japanese and English components of the Tanaka<br>
Corpus (approx. 150,000 sentence pairs) through MeCab and TreeTagger.<br>
The unique noun counts (all meishi in Japanese and NN in English) were<br>
13,725 and 12,106 respectively. That is more what I would expect.<br>
<br>
Your number of unique words in Japanese seems extraordinarily large.<br>
As a comparison, the MeCab output from the Tanaka sentences is only<br>
about 19,000 unique tokens.<br>
<br>
If you contact me offline, I may be able to help with analysing the output from<br>
MeCab.<br>
<br>
Jim<br>
<font color="#888888"><br>
--<br>
Jim Breen<br>
Adjunct Snr Research Fellow, Clayton School of IT, Monash University<br>
Treasurer: Hawthorn Rowing Club, Japanese Studies Centre<br>
Graduate student: Language Technology Group, University of Melbourne<br>
</font><div><div></div><div><br>
_______________________________________________<br>
Corpora mailing list<br>
<a href="mailto:Corpora@uib.no" target="_blank">Corpora@uib.no</a><br>
<a href="http://mailman.uib.no/listinfo/corpora" target="_blank">http://mailman.uib.no/listinfo/corpora</a><br>
</div></div></blockquote></div><br><br clear="all"><br></div></div>-- <br>James L. Fidelholtz<br>Posgrado en Ciencias del Lenguaje<br>Instituto de Ciencias Sociales y Humanidades<br>Benemérita Universidad Autónoma de Puebla, MÉXICO<br>
</div>
</div><br><br clear="all"><br>-- <br>James L. Fidelholtz<br>Posgrado en Ciencias del Lenguaje<br>Instituto de Ciencias Sociales y Humanidades<br>Benemérita Universidad Autónoma de Puebla, MÉXICO<br>