Corpora: (Robust and fast) PC concordance programs

[Mark Davies]

I'm writing a paper in which I need to briefly comment on concordance
programs for the PC platform, and how they handle large corpora ( >
100,000,000 words).  I'm trying to address the question of whether or not
there is a program out there that can search a corpus of this size in a
matter of a few seconds (perhaps 5-10 seconds or less).

My own experience is with the following programs:

-- WordCruncher (for DOS)
-- TACT
-- WordSmith
-- MonoConc

My experience is with WordCruncher is that it is extremely fast, because it
creates and searches an every-word index that can be used in successive
sessions.  Search time is usually one or two seconds, even for the most
complex searches.  The problem with this program is that it cannot handle
corpora larger than about 20,000,000 words because of a limit on the number
of "unique forms".  TACT, if I recall correctly, reachers a limit at a much
smaller size -- about 10,000,000 words or less.

Both WordSmith and MonoConc can handle large files --I've used both
successfully on corpora over 100,000,000 words.  The limitation of both of
these programs is that they do not create an every-word index that can be
searched in later sessions.  This means that it has to traverse the entire
corpus for each separate search, which takes 3-4 minutes for a 100,000,000+
word corpus.

My question then: are there in fact any **PC-based**, **commercially
available** programs that can search  100,000,000+ word corpora in a couple
of seconds (or less)?  I realize that there may be some custom solutions
that researchers have created, but I need to focus here on
commercially-available software (or shareware/freeware, if in fact such a
program existed that meets these specs).

Thanks in advance for your help.

Mark Davies

=======================================
Mark Davies, Associate Professor, Spanish Linguistics
Dept. of Foreign Languages, Illinois State University
Normal, IL 61790-4300

Voice:309/438-7975      email:mdavies at ilstu.edu
Fax:309/438-8038          http://mdavies.for.ilstu.edu/
=======================================