If you don't need smoothing, we have a program for generating n-grams as part of a system called Sanchay. It can work on UTF-8 text.<br><br>- Anil Kumar Singh<br><br><div class="gmail_quote">On Tue, Mar 11, 2008 at 4:34 PM, Paul Johnston <<a href="mailto:paul.a.johnston@manchester.ac.uk">paul.a.johnston@manchester.ac.uk</a>> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div link="blue" vlink="purple" lang="EN-US">
<div>
<p><font face="Arial" size="2"><span style="font-size: 10pt; font-family: Arial;">Can anyone recommend a wordgram generator similar to
text2wngram in the CMU-Toolkit which can handle Unicode encoded texts,
preferably utf-8 or UCS-2.</span></font></p>
<p><font face="Arial" size="2"><span style="font-size: 10pt; font-family: Arial;">I've been using the CMU-Toolkit successfully on English
text files especially from the BNC but seem to have problems when using a UTF-8
file.</span></font></p>
<p><font face="Arial" size="2"><span style="font-size: 10pt; font-family: Arial;"> </span></font></p>
<p><font face="Arial" size="2"><span style="font-size: 10pt; font-family: Arial;">Error reading temp file count
/usr/tmp/text2wngram.tmp.hb-0021205.4217.1</span></font></p>
<p><font face="Arial" size="2"><span style="font-size: 10pt; font-family: Arial;"> </span></font></p>
<p><font face="Arial" size="2"><span style="font-size: 10pt; font-family: Arial;">It seems to have problems reading the tmp files (see above)
permissions are fine and it works with ascii texts.</span></font></p>
<p><font face="Arial" size="2"><span style="font-size: 10pt; font-family: Arial;"> </span></font></p>
<p><font face="Arial" size="2"><span style="font-size: 10pt; font-family: Arial;">I've tried this on a couple of Linux systems (Fedora
and SUSE) with clean builds and in both cases text2wfreq works fine but
text2wngram does not.</span></font></p>
<p><font face="Arial" size="2"><span style="font-size: 10pt; font-family: Arial;">Any suggestions?</span></font></p>
<p><font face="Arial" size="2"><span style="font-size: 10pt; font-family: Arial;"> </span></font></p>
<p><font face="Arial" size="2"><span style="font-size: 10pt; font-family: Arial;">Cheers Paul</span></font></p>
<p><font face="Arial" size="2"><span style="font-size: 10pt; font-family: Arial;"> </span></font></p>
<p><font face="Arial" size="2"><span style="font-size: 10pt; font-family: Arial;"> </span></font></p>
<p><font face="Times New Roman" size="2"><span style="font-size: 10pt;">Paul Johnston</span></font></p>
<p><font face="Times New Roman" size="2"><span style="font-size: 10pt;">Humanities Development</span></font></p>
<p><font face="Times New Roman" size="2"><span style="font-size: 10pt;">Room 2.12</span></font></p>
<p><font face="Times New Roman" size="2"><span style="font-size: 10pt;">Bridgeford</span></font><font size="2"><span style="font-size: 10pt;"> Building</span></font></p>
<p><font face="Times New Roman" size="2"><span style="font-size: 10pt;">Manchester</span></font><font size="2"><span style="font-size: 10pt;"> University</span></font></p>
<p><font face="Times New Roman" size="2"><span style="font-size: 10pt;">0161 275 1396</span></font></p>
<p><font face="Times New Roman" size="3"><span style="font-size: 12pt;"> </span></font></p>
<p><font face="Times New Roman" size="2"><span style="font-size: 10pt;">Programmers are in a race with the Universe to create bigger and better
idiot-proof programs, </span></font></p>
<p><font face="Times New Roman" size="2"><span style="font-size: 10pt;">while the Universe is trying to create bigger and better idiots. </span></font></p>
<p><font face="Times New Roman" size="2"><span style="font-size: 10pt;">So far the Universe is winning. </span></font></p>
<p><font face="Times New Roman" size="2"><span style="font-size: 10pt;">Rich Cook</span></font></p>
<p><font face="Times New Roman" size="3"><span style="font-size: 12pt;"> </span></font></p>
</div>
</div>
<br>_______________________________________________<br>
Corpora mailing list<br>
<a href="mailto:Corpora@uib.no">Corpora@uib.no</a><br>
<a href="http://mailman.uib.no/listinfo/corpora" target="_blank">http://mailman.uib.no/listinfo/corpora</a><br>
<br></blockquote></div><br>