<div dir="ltr">Hi, Mike,<div><br></div><div style>Your second situation:</div><div style><br></div><div style><span style="font-family:arial,sans-serif;font-size:12.800000190734863px"><<2) You're trying to detect code switching in text, e.g. the occasional use of English or French loanwords or other terms inside Arabic text, where the loanwords are written in the same script as the Arabic.>></span><br style="font-family:arial,sans-serif;font-size:12.800000190734863px">
</div><div style><span style="font-family:arial,sans-serif;font-size:12.800000190734863px"> </span></div><div style><span style="font-family:arial,sans-serif;font-size:12.800000190734863px">is NOT code-switching (almost never, anyway). My standard proof is the name 'Bach' as pronounced by minimally-educated speakers of English: [bax]. There is absolutely *no* requisite for such speakers to be bilingual in any degree in German, and, if they are Americans, they are not, mostly. Nevertheless, they use the non-English segment [x] if they are minimally musically literate. It is important to recognize that the segment [x] used by English speakers is *not* the same as that segment in the German pronunciation of the name, which is more postvelar. That is, they are *not* speaking German, even most of those who are bilingual in German. The word is an *English* word (borrowed from German, but not, in most cases, pronounced exactly as in German as it would be if the speaker were engaged in code-switching). See below my signature for a pertinent limerick on this point.</span></div>
<div style><span style="font-family:arial,sans-serif;font-size:12.800000190734863px"><br></span></div><div style><span style="font-family:arial,sans-serif;font-size:12.800000190734863px">I am *not* claiming that there can be no one-word code-switching. Indeed, I have witnessed in southern Texas a conversation by locals in a store where there was code-switching between Spanish and English after every single word! But there is a difference between borrowings and the use of code-switching.</span></div>
<div style><span style="font-family:arial,sans-serif;font-size:12.800000190734863px"><br></span></div><div style><span style="font-family:arial,sans-serif;font-size:12.800000190734863px">Jim</span></div><div style><span style="font-family:arial,sans-serif;font-size:12.800000190734863px"><br>
</span></div><div style><span style="font-family:arial,sans-serif;font-size:12.800000190734863px">There once was a fellow named Hatch</span></div><div style><span style="font-family:arial,sans-serif;font-size:12.800000190734863px">Who was fond of the music of Bach.</span></div>
<div style><span style="font-family:arial,sans-serif;font-size:12.800000190734863px">He said, "It's not fussy,</span></div><div style><span style="font-family:arial,sans-serif;font-size:12.800000190734863px">Like Brahms or Debussy;</span></div>
<div style><span style="font-family:arial,sans-serif;font-size:12.800000190734863px">Sit down and I'll play you a snatch."</span></div><div style><span style="font-family:arial,sans-serif;font-size:12.800000190734863px"><br>
</span></div><div style><span style="font-family:arial,sans-serif;font-size:12.800000190734863px">Now *that's* a borrowing! (But so is [bax].)</span></div><div style><span style="font-family:arial,sans-serif;font-size:12.800000190734863px"><br>
</span></div><div style><span style="font-family:arial,sans-serif;font-size:12.800000190734863px"><br></span></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Apr 25, 2013 at 8:58 AM, maxwell <span dir="ltr"><<a href="mailto:maxwell@umiacs.umd.edu" target="_blank">maxwell@umiacs.umd.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im">saadane houda <<a href="mailto:saadane_houda@yahoo.fr" target="_blank">saadane_houda@yahoo.fr</a>> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I want to ask you if you know of software or programs<br>
(open source) for the detection of the origin of<br>
language(Arabic, French or English)<br>
</blockquote>
<br></div>
The usual sorts of language ID programs work quite well if there's enough text, and the system has been trained on that kind of text. Mostly they use character n-grams. There are at least two situations where they can go wrong:<br>
<br>
1) They haven't been trained on the particular type of text, e.g. they've been trained on Unicode Arabic text, but not on Arabizi.<br>
<br>
2) You're trying to detect code switching in text, e.g. the occasional use of English or French loanwords or other terms inside Arabic text, where the loanwords are written in the same script as the Arabic.<br>
<br>
Problem (2) is made worse by the fact that many such English-in-Arabic words won't be in dictionaries (even assuming you try to map the Arabic script in a fuzzy way to roman script), because they're place names or person names.<br>
<br>
There is also a problem of deciding whether a word *is* Arabic, English or French; I've heard the Arabic word 'mufti' used in English in ways that I'm guessing it wouldn't be used in Arabic. Does it count as an Arabic word, or as English? And if you think that one is clear (perhaps because its meaning in English is so divergent from its meaning in Arabic), then there are other, more borderline, examples. (The same problem arises with place names; is 'Cairo' an Arabic word in English text, just because it refers to a place in the Arabic-speaking world?) Of course, whether sort of thing this is a problem depends on what use you want to put the results to. Maybe place names don't matter for your purposes.<br>
<br>
There are doubtless papers, even books, written on these issues. (And there was a discussion on this list awhile back about it.) Of course, if your task is to decide whether paragraphs-sized stretches of text are (mostly) English, French or Arabic, then the usual language ID programs will work just fine.<br>
<br>
Mike Maxwell<br>
University of Maryland<div class="HOEnZb"><div class="h5"><br>
<br>
______________________________<u></u>_________________<br>
UNSUBSCRIBE from this page: <a href="http://mailman.uib.no/options/corpora" target="_blank">http://mailman.uib.no/options/<u></u>corpora</a><br>
Corpora mailing list<br>
<a href="mailto:Corpora@uib.no" target="_blank">Corpora@uib.no</a><br>
<a href="http://mailman.uib.no/listinfo/corpora" target="_blank">http://mailman.uib.no/<u></u>listinfo/corpora</a><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br>James L. Fidelholtz<br>Posgrado en Ciencias del Lenguaje<br>Instituto de Ciencias Sociales y Humanidades<br>Benemérita Universidad Autónoma de Puebla, MÉXICO
</div>