[Corpora-List] Grep for Unicode (was: Grep for Windows)

[Mike Maxwell]

Rob Malouf wrote:
> On Dec 15, 2006, at 8:36 AM, maxwell at ldc.upenn.edu wrote:
>> Besides, none of the standard grep implementations that I know of 
>> handle Unicode (at least not in any useful way).
> 
> Gnu grep 2.5.1 supports Unicode, though I guess it's debatable just how 
> useful it is.  The next version is supposed to be much better on that 
> front.

I suspect this has been hashed over somewhere, and if so just point me 
in the right direction.  But I don't see the string 'unicode' (upper or 
lower case) anywhere in the Gnu grep 2.5.1 that I just downloaded, save 
in the .po files (which are messages, and haven't been updated in a long 
time anyway).  I did google some Red Hat info on updates to grep, which 
do speak about a Unicode issue (apparently an earlier version had an 
extreme inefficiency in the way it searched UTF-8 streams).  Since I 
thought Linux distros usually came with the GNU tools, I'm a little puzzled.

Stepping back a bit: I can think of two ways one might want to use grep 
with Unicode files.

One is to search for a particular byte sequence, and I presume grep can 
do that.

The other is to search for a particular character sequence.  For that, 
two things seem to be necessary: it needs to know the encoding of the 
incoming stream (UTF-8, UTF-16 big-end/little-end,...), and it needs to 
handle normalization.  (And it needs to know what to do with these in 
the output.)  I think the normalization issue is doable, provided the 
encoding issue is correctly handled.  But there are numerous issues with 
determining the encoding of an input stream, and I'm not knowledgeable 
enough to know whether it is always possible to reliably tell from 
looking at a stream of bytes which one knows to be Unicode which 
encoding it is.

At any rate, I don't see anything that tells me how Gnu grep deals with 
Unicode encodings and normalization.  Am I missing something?
-- 
	Mike Maxwell
	maxwell at ldc.upenn.edu