[Ads-l] Improving text databases

[ADSGarson O'Toole]

Research in linguistics, history, machine learning, and more relies on
digital databases of books, magazines, and newspapers. Yet the quality
of these databases is often poor.

Optical Character Recognition (OCR): Too many words are mangled or
omitted from OCR results. Often multicolumn formats are not segmented
correctly. Words from different columns are interleaved. When OCR is
applied to low-quality scans the result is complete gibberish. When
this occurs, the item that is supposed to be in the database is
effectively absent.

Dates: Too many items are misdated. Often the correct dates can be
carefully extracted from scans of newspapers and magazines. This
should be done. In addition, today's AI technology should allow an
estimate of the correct date to be generated based on extended text
excerpts. The vocabulary, grammatical structures, and events help to
identify the period. These estimates can provide sanity checks. When
the estimate differs significantly from the date listed in the
metadata, a flag should be set for manual review of the date.

Do other list members have ideas for improving the text databases?
Garson

------------------------------------------------------------
The American Dialect Society - http://www.americandialect.org