[Corpora-List] Query on the use of Google for corpus research

[Philip Resnik]

> Your tools sound really interesing, and in part similar to what we are
> developing/adapting. Is anything (besides GATES, of course) publicly
> available?

Marco and Nancy, we are soon (within a month or two) going to be doing
an open source release of the codebase for the Linguist's Search
Engine (LSE, http://lse.umiacs.umd.edu).  Although the LSE does not
currently do some of the Web page processing you're describing, other
aspects of its architecture might be useful to you or others.

The LSE currently piggybacks on Altavista, rather than doing its own
crawling.  Its facility for building custom collections currently
includes the retrieval of pages, extraction of text from HTML,
sentence breaking, tokenization, POS tagging, parsing, and indexing
sentences by their syntactic structure.  The architecture is highly
modular and it's easy to add new annotation modules and to configure
dependencies between modules (e.g adding a parser that requires
POS-tagged input).  The LSE is designed so that the processing of
collected pages takes place in parallel on as many machines as you'd
like.  Annotation processes run concurrently as pages are added to the
collection -- i.e. you are processing the pages, including indexing
and making material searchable, while crawling is still taking place,
and you can distribute multiple copies of the annotation processes on
a computing cluster.

It is very simple to modify the LSE code to draw from other Web
sources (certainly anything available via a CPAN WWW::Search module),
and although we do not identify tables, headers and footers, etc., I'm
sure the architecture would be flexible enough to add that sort of
functionality as part of the document-level processing before sentence
identification and sentence-level processing take place.

The resulting database and index are, of course, well suited to the
kinds of lexically and syntactically driven searching the LSE was
designed to support, including a very linguist-friendly user
interface.  But I expect the software could easily be adapted for
other purposes, and we're hoping the open source release will make it
easy for people to develop their own variations of linguistic search.

Best,

  Philip

P.S.  The LSE will appear in the demo session at the upcoming ACL
conference.  Perhaps we'll get a chance to talk in person there!

  ----------------------------------------------------------------
  Philip Resnik, Associate Professor
  Department of Linguistics and Institute for Advanced Computer Studies

  1401 Marie Mount Hall            UMIACS phone: (301) 405-6760
  University of Maryland           Linguistics phone: (301) 405-8903
  College Park, MD 20742 USA	   Fax: (301) 314-2644 / (301) 405-7104
  http://umiacs.umd.edu/~resnik	   E-mail: resnik at umiacs.umd.edu