My point was not that MULTEXT-EAST already meets Mike's needs exactly, but<div>that it is a good basis for an extensible, cross-linguistically useful tagset that does</div><div>do this. </div><div><br class="webkit-block-placeholder">
</div><div>The PDT 2.0 documentation, chapters 3 and 4, has a detailed discussion</div><div>of names and abbreviations,including the names of horses, DJs and Julie Sedivy (whose</div><div>name is, they say, of Czech origin, but adjusted/smoothed over at some point to fit in with </div>
<div>non-Czech expectations).</div><div><br class="webkit-block-placeholder"></div><div>See <a href="http://ufal.mff.cuni.cz/pdt2.0/doc/manuals/en/m-layer/html/">http://ufal.mff.cuni.cz/pdt2.0/doc/manuals/en/m-layer/html/</a></div>
<div><br class="webkit-block-placeholder"></div><div>There is much more.</div><div>Chapter 13.7 documents how to tag chess moves, should such occur in the data.</div><div>Much of this is specific to Czech and Czech newspaper norms. It is the general</div>
<div>approach (non-atomic tags, excellent documentation, care for the realities of the</div><div>data) that I advocate. It is not a ready made solution, but a good basis for one.</div><div><br><div><span class="gmail_quote">On 06/02/2008, <b class="gmail_sendername">Eric Atwell</b> <<a href="mailto:eric@comp.leeds.ac.uk">eric@comp.leeds.ac.uk</a>> wrote:</span><blockquote class="gmail_quote" style="margin:0;margin-left:0.8ex;border-left:1px #ccc solid;padding-left:1ex">
Correct me if I'm wrong, but I thought EAGLES, MULTEXT-EAST etc tagsets<br>dont make the kind of distinctions Mike alluded to - eg distinguisihng<br>3 from three, foreign-names from local-names, and categorising non-dictionary<br>
words with something more than "unknown". This isnt PoS0tagging in the<br>traditional sense, whcih EAGLES etc extended from English to other<br>languages<br><br>Eric Atwell Leeds University<br><br><br><br>On Wed, 6 Feb 2008, Serge Sharoff wrote:<br>
<br>> My vote goes to MULTEXT-EAST (MTE). For its next version it has been adapted to include Persian, FInnish and Hungarian in addition to Slavonic languages in Version 3, so it's quite flexible. However, MTE might be an overkill for your purposes, as the tagset for Russian has more than 600 tags (in you take into account all combinations of cases, numbers, genders, tenses, etc), but the English set is much smaller.<br>
> S<br>><br>> -----Original Message-----<br>> From: <a href="mailto:corpora-bounces@uib.no">corpora-bounces@uib.no</a> on behalf of chris brew<br>> Sent: Tue 05/02/2008 22:50<br>> To: <a href="mailto:maxwell@umiacs.umd.edu">maxwell@umiacs.umd.edu</a><br>
> Cc: corpora<br>> Subject: Re: [Corpora-List] Tag sets<br>><br>> You could start from something that has already been applied multilingually,<br>> such as<br>> the MULTEXT-EAST materials at<br>> <a href="http://nl.ijs.si/ME/V3/msd/html/">http://nl.ijs.si/ME/V3/msd/html/</a><br>
><br>> or<br>><br>> <a href="http://ufal.mff.cuni.cz/pdt2.0/doc/manuals/en/m-layer/html/index.html">http://ufal.mff.cuni.cz/pdt2.0/doc/manuals/en/m-layer/html/index.html</a><br>><br>> which is a similar thing developed by the Prague group for Czech.<br>
> Anna Feldman, Jiri Hana (who co-wrote the pdt manual above)<br>> and I have some experience in using<br>> adapted versions for Russian, Polish, Spanish and Catalan. It<br>> would be fun to find out if the same ideas work for Bengali etc.<br>
><br>> Chris<br>><br>><br>> On 05/02/2008, <a href="mailto:maxwell@umiacs.umd.edu">maxwell@umiacs.umd.edu</a> <<a href="mailto:maxwell@umiacs.umd.edu">maxwell@umiacs.umd.edu</a>> wrote:<br>>><br>
>> We're looking at annotating a small sample (~5k words) of Bengali text,<br>>> and later maybe Urdu and Punjabi. The annotation will be the dictionary<br>>> citation form of each word. The texts are mostly news articles, so there<br>
>> are a fair number of words for which there won't be any dictionary<br>>> citation form. These include many proper names, numerals, acronyms, and<br>>> who knows what else. I'll refer to these as "non-dictionary words",<br>
>> whereas "dictionary words" will include words whose citation form is in<br>>> the dictionary we're using, even if the inflected wordform itself is not.<br>>> (We're doing this to test a morphological parser.)<br>
>><br>>> This is not quite the same as the inverse of named entity tagging, since<br>>> some parts of names may have citation forms. For example, in English one<br>>> would tag "Mississippi River" as a name. But "River" would be found in<br>
>> the dictionary, so for our purposes we would only want to tag<br>>> "Mississippi" as a non-dictionary word.<br>>><br>>> The simplest thing for us to do would be to just tag all such<br>
>> non-dictionary words the same way, e.g. with a tag "NOT". However, in the<br>>> interest of future uses to which we might put such a tagged text, it might<br>>> be good to differentiate among the various kinds of non-dictionary words.<br>
>><br>>> We could easily make up our own tagset for non-dictionary words, but it<br>>> strikes me that better would be to use some standard tagset for such<br>>> words, if such a tagset exists. There is a table of tagsets in Manning<br>
>> and Schutze pg. 141-2, including the Penn Treebank, Brown, and CLAWS.<br>>> However, the tagsets are English-specific. This is especially noticeable<br>>> in the punctuation tags for the PTB and Brown sets, but also e.g. in the<br>
>> decision to tag singular and plural proper nouns differently. (Some<br>>> languages attach case markers to proper nouns.) Also, it appears that<br>>> none of the tagsets distinguishes between numerals ('3', '4.5') and<br>
>> numbers written out ('three', 'four point five'), which we need to do, nor<br>>> are acronyms distinguished from "symbols".<br>>><br>>> Another distinction I thought about making is between "ordinary" Bengali<br>
>> names, and foreign names, since one might later want to develop a<br>>> transducer to convert the latter into their more common Latin forms.<br>>> However, I suspect that might be too difficult a distinction for<br>
>> annotators to make, and in any case some well-known Bengali names are<br>>> likely to have "standard" transliterations.<br>>><br>>> Does anyone know of a semi-standard tagset that would be less<br>
>> English-specific, and would make the kinds of distinctions among<br>>> non-dictionary words that we want to (or should) make? Or should we just<br>>> make up our own set?<br>>><br>>> Mike Maxwell<br>
>> CASL/ U MD<br>>><br>>><br>>> _______________________________________________<br>>> Corpora mailing list<br>>> <a href="mailto:Corpora@uib.no">Corpora@uib.no</a><br>>> <a href="http://mailman.uib.no/listinfo/corpora">http://mailman.uib.no/listinfo/corpora</a><br>
>><br>><br>> _______________________________________________<br>> Corpora mailing list<br>> <a href="mailto:Corpora@uib.no">Corpora@uib.no</a><br>> <a href="http://mailman.uib.no/listinfo/corpora">http://mailman.uib.no/listinfo/corpora</a><br>
><br><br>--<br>Eric Atwell,<br> Senior Lecturer, Language research group leader, School of Computing,<br> Faculty of Engineering, UNIVERSITY OF LEEDS, Leeds LS2 9JT, England<br> TEL: 0113-3435430 FAX: 0113-3435468 WWW/email: google Eric Atwell<br>
</blockquote></div><br> </div>