Dear Ralf, <div><br></div><div>are the translated terms ("descriptors") used in the classification process? If they are not, then I'm way off target. But it looks like they are, and if they are then the accuracy and precision of those translations are very relevant to the system performance</div>
<div><br></div><div>Related diatribe:
<a href="http://kilgarriff.co.uk/Publications/1998-K-ELSNET-lex_quality.txt">http://kilgarriff.co.uk/Publications/1998-K-ELSNET-lex_quality.txt</a> </div><div><br></div><div>Adam<br><br><div class="gmail_quote">On 2 June 2012 13:29, Ralf Steinberger <span dir="ltr"><<a href="mailto:ralf.steinberger@jrc.ec.europa.eu" target="_blank">ralf.steinberger@jrc.ec.europa.eu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div lang="EN-GB" link="blue" vlink="purple"><div><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Dear Adam,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Thanks for your proposal and for allowing me to clarify: EuroVoc is a <i>classification scheme</i> with exactly the same 6700 subject domain classes in all languages, i.e. each class has a numerical identifier and exactly <i>one class</i> <i>label</i> that has been translated into all 27 or so languages. Example EuroVoc categories are ‘nuclear materials’, ‘Austria’, ‘fishery management’, ‘xenophobia’, ‘budget’, ‘population statistics’, ...<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">I cannot see how such a classification scheme would favour one language over another, especially as the documents are parallel translations, as well: they have the same contents in all languages. EuroVoc is in no way comparable to a resource such as WordNet, which rather lists and organises existing words of a language, with varying coverage. <u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Greetings from Italy to the UK.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Ralf<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> <a href="mailto:adam.kilgarriff@gmail.com" target="_blank">adam.kilgarriff@gmail.com</a> [mailto:<a href="mailto:adam.kilgarriff@gmail.com" target="_blank">adam.kilgarriff@gmail.com</a>] <b>On Behalf Of </b>Adam Kilgarriff<br>
<b>Sent:</b> 02 June 2012 14:13<br><b>To:</b> Ralf Steinberger<br><b>Cc:</b> <a href="mailto:corpora@uib.no" target="_blank">corpora@uib.no</a>; <a href="mailto:clef@dei.unipd.it" target="_blank">clef@dei.unipd.it</a>; <a href="mailto:ln@cines.fr" target="_blank">ln@cines.fr</a><br>
<b>Subject:</b> Re: [Corpora-List] Q: Classification performance across languages and language families<u></u><u></u></span></p><div><div class="h5"><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal">Ralf,<u></u><u></u></p>
<div><p class="MsoNormal"><u></u> <u></u></p></div><div><p class="MsoNormal">Please excuse scepticism, but what about the simple hypothesis that it all depends on thesaurus-quality. My hunch would be that it started from a Germanic language, hence good performance there, and that Slavic lgs have been added more recently, so there have been less years for debugging/improving, and that there was a particularly inspired Hungarian translator!<u></u><u></u></p>
<div><p class="MsoNormal"><u></u> <u></u></p></div><div><p class="MsoNormal">Maltese has a special problem - Maltese hasn't ever had a technical vocabulary so there was nothing the Maltese thesaurus-translators could do except make things up.<u></u><u></u></p>
</div><div><p class="MsoNormal"><u></u> <u></u></p></div><div><p class="MsoNormal">(Of course I'll be happy to have my hypothesis quashed by someone who knows the history of Eurovoc)<u></u><u></u></p></div><div><p class="MsoNormal">
<u></u> <u></u></p></div><div><p class="MsoNormal">Adam<u></u><u></u></p></div><div><div><p class="MsoNormal" style="margin-bottom:12.0pt"><u></u> <u></u></p><div><p class="MsoNormal">On 2 June 2012 12:40, Ralf Steinberger <<a href="mailto:ralf.steinberger@jrc.ec.europa.eu" target="_blank">ralf.steinberger@jrc.ec.europa.eu</a>> wrote:<u></u><u></u></p>
<div><div><p class="MsoNormal">A question and an invitation to discussion.<u></u><u></u></p><p class="MsoNormal"> <u></u><u></u></p><p class="MsoNormal" style="text-autospace:none">We recently carried out <a href="http://langtech.jrc.ec.europa.eu/Documents/2012_LREC-JEX-final.pdf" target="_blank">multi-label categorisation experiments</a> on a mostly parallel set of documents in 22 languages, covering the language families Germanic, Romance, Slavic, Hellenic, Finno-Ugric, Baltic and Semitic. The document set is reasonably large (22K to 42K documents per language), using the thousands of subject domain categories from the <a href="http://eurovoc.europa.eu/" target="_blank">EuroVoc thesaurus</a>. The performance across languages was rather uniform, with the exception of the outlier Maltese, which performed considerably less well. The languages covered are Bulgarian, Czech, Danish, Dutch, English, Estonian, Finnish, French, German, Greek, Hungarian, Italian, Latvian, Lithuanian, Maltese, Polish, Portuguese, Romanian, Slovak, Slovenian, Spanish and Swedish. <u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p><p class="MsoNormal">To my great surprise, the highly inflected agglutinative language <b>Hungarian</b> produced the best results of all. The five Germanic languages ended up in the top ten positions, the five Slavic languages in the bottom half. The results for the other language families were less consistent. <u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p><p class="MsoNormal"><b>Q1:</b> Does anyone have an intuition how these results could be explained?<u></u><u></u></p><p class="MsoNormal"> <u></u><u></u></p><p class="MsoNormal"><b>Q2:</b> Has anyone ran similar experiments with other types of classifiers or data? Are the results similar?<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p><p class="MsoNormal">My initial expectation had been that highly inflected languages would perform less well and that feature space reduction using lemmatisation would improve the results. However, our experiments for Czech, English, Estonian and French (described in Ebrahim et al., forthcoming) showed the contrary, rather consistently for all four languages and language families: (1) lemmatisation reduces the performance and (2) adding part-of-speech (POS) information to the word form and/or to the lemma improves the performance. <u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p><p class="MsoNormal"><b>Q3:</b> Can we conclude that: the scarcer the feature space, the better the classification performance? <u></u><u></u></p><p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal"><b>Q4:</b> If that were the case, why did Slavic languages (and Maltese) perform less well in our experiments? <u></u><u></u></p><p class="MsoNormal"> <u></u><u></u></p><p class="MsoNormal" style="text-autospace:none">
I would be pleased if you could share your own experience and/or your opinions.<u></u><u></u></p><p class="MsoNormal" style="text-autospace:none"> <u></u><u></u></p><p class="MsoNormal" style="text-autospace:none">The classification tool (<a href="http://langtech.jrc.ec.europa.eu/Eurovoc.html" target="_blank">JRC EuroVoc Indexer JEX</a>) and the multilingual document set can be downloaded from <a href="http://langtech.jrc.ec.europa.eu/Eurovoc.html" target="_blank">http://langtech.jrc.ec.europa.eu/Eurovoc.html</a> . Details of our experiments are given in the two papers below.<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p><p class="MsoNormal" style="margin-left:36.0pt">Steinberger Ralf, Mohamed Ebrahim & Marco Turchi (2012). <strong><span style="font-family:"Calibri","sans-serif"">JRC EuroVoc Indexer JEX - A freely available multi-label categorisation tool</span></strong>. Proceedings of the 8<sup>th</sup> international conference on Language Resources and Evaluation (LREC'2012), Istanbul, 21-27 May 2012. (<a href="http://langtech.jrc.ec.europa.eu/Documents/2012_LREC-JEX-final.pdf" title="Reference publication for the JRC Eurovoc Indexer JEX" target="_blank">PDF</a>)<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:36.0pt"> <u></u><u></u></p><p class="MsoNormal" style="margin-left:36.0pt">Ebrahim Mohamed, Maud Ehrmann, Marco Turchi & Ralf Steinberger (forthcoming). <strong><span style="font-family:"Calibri","sans-serif"">Multi-label EuroVoc classification for Eastern and Southern EU Languages</span></strong>. In: Cristina Vertan & Walther v. Hahn: Multilingual processing in Eastern and Southern EU languages - Low-resourced technologies and translation. Cambridge Scholars Publishing, Cambridge, UK.<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p><p class="MsoNormal">Greetings,<u></u><u></u></p><p class="MsoNormal"> <u></u><u></u></p><p class="MsoNormal">Ralf<u></u><u></u></p><p class="MsoNormal"> <u></u><u></u></p><p class="MsoNormal">
<u></u><u></u></p><p class="MsoNormal"> <u></u><u></u></p><p class="MsoNormal"><b><span style="font-size:9.0pt;color:#4a442a">Ralf Steinberger</span></b><span style="font-size:9.0pt;color:#4a442a"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:9.0pt;color:#4a442a">European Commission – Joint Research Centre (JRC)</span><u></u><u></u></p><p class="MsoNormal"><span lang="EN-US" style="font-size:9.0pt;color:#4a442a">URL: <a href="http://langtech.jrc.ec.europa.eu/RS.html" target="_blank">http://langtech.jrc.ec.europa.eu/RS.html</a></span><span lang="EN-US" style="font-size:9.0pt"> <span style="color:#4a442a"> </span></span><u></u><u></u></p>
</div></div><p class="MsoNormal" style="margin-bottom:12.0pt"><br>_______________________________________________<br>UNSUBSCRIBE from this page: <a href="http://mailman.uib.no/options/corpora" target="_blank">http://mailman.uib.no/options/corpora</a><br>
Corpora mailing list<br><a href="mailto:Corpora@uib.no" target="_blank">Corpora@uib.no</a><br><a href="http://mailman.uib.no/listinfo/corpora" target="_blank">http://mailman.uib.no/listinfo/corpora</a><u></u><u></u></p></div>
<p class="MsoNormal"><br><br clear="all"><u></u><u></u></p><div><p class="MsoNormal"><u></u> <u></u></p></div><p class="MsoNormal">-- <br>========================================<br><a href="http://www.kilgarriff.co.uk/" target="_blank">Adam Kilgarriff</a> <a href="mailto:adam@lexmasterclass.com" target="_blank">adam@lexmasterclass.com</a> <br>
Director <a href="http://www.sketchengine.co.uk/" target="_blank">Lexical Computing Ltd</a> <br>Visiting Research Fellow <a href="http://leeds.ac.uk" target="_blank">University of Leeds</a> <u></u><u></u></p>
<div><p class="MsoNormal"><i><span style="color:#006600">Corpora for all</span></i> with <a href="http://www.sketchengine.co.uk" target="_blank">the Sketch Engine</a> <u></u><u></u></p></div><div><p class="MsoNormal">
<i><a href="http://www.webdante.com" target="_blank">DANTE: <span style="color:#009900">a lexical database for English</span></a><span style="color:#009900"> </span> </i><u></u><u></u></p>
<div><p class="MsoNormal">========================================<u></u><u></u></p></div></div><p class="MsoNormal"><u></u> <u></u></p></div></div></div></div></div></div></div></blockquote></div><br><br clear="all"><div>
<br></div>-- <br>========================================<br><a href="http://www.kilgarriff.co.uk/" target="_blank">Adam Kilgarriff</a> <a href="mailto:adam@lexmasterclass.com" target="_blank">adam@lexmasterclass.com</a> <br>
Director <a href="http://www.sketchengine.co.uk/" target="_blank">Lexical Computing Ltd</a> <br>Visiting Research Fellow <a href="http://leeds.ac.uk" target="_blank">University of Leeds</a> <div>
<i><font color="#006600">Corpora for all</font></i> with <a href="http://www.sketchengine.co.uk" target="_blank">the Sketch Engine</a> </div><div> <i><a href="http://www.webdante.com" target="_blank">DANTE: <font color="#009900">a lexical database for English</font></a><font color="#009900"> </font> </i><div>
========================================</div></div><br>
</div>