[Ads-l] Improving text databases

[ADSGarson O'Toole]

Estimating the publication date from text passages is complicated for
reprints. If an item is reprinted, the estimate yields the original
publication date, not the reprint date.

Also, some documents yield a patchwork of estimated dates. For
example, a collection of short stories from different eras might yield
a different date for each story. A document with extensive quotations
from different time periods would also yield multiple dates.

Garson

On Fri, Oct 30, 2026 at 6:13 PM ADSGarson O'Toole
<adsgarsonotoole at gmail.com> wrote:
>
> Research has been conducted on estimating publication dates from text
> passages. Here is a pertinent article based on a corpus of New York
> Times articles:
>
> Date: April 24, 2023
> Article Title: Text2Time: Transformer-based Article Time Period Prediction
> Authors: Karthick Prasad Gunasekaran, B Chase Babrich, Saurabh
> Shirodkar, Hee Hwang
> Location: University of Massachusetts, Amherst, USA
> https://arxiv.org/abs/2304.10859v2
>
> [Begin excerpt]
> The task of predicting the publication period of text documents, such
> as news articles, is an important but less studied problem in the
> field of natural language processing. Predicting the year of a news
> article can be useful in various contexts, such as historical
> research, sentiment analysis, and media monitoring. In this work, we
> investigate the problem of predicting the publication period of a text
> document, specifically a news article, based on its textual content.
> In order to do so, we created our own extensive labeled dataset of
> over 350,000 news articles published by The New York Times over six
> decades. In our approach, we use a pretrained BERT model fine-tuned
> for the task of text classification, specifically for time period
> prediction. This model exceeds our expectations and provides some very
> impressive results in terms of accurately classifying news articles
> into their respective publication decades. The results beat the
> performance of the baseline model for this relatively unexplored task
> of time prediction from text.
> [End excerpt]
>
> A more comprehensive and improved algorithm of this type would be
> helpful. Note, the goal is not to recklessly replace existing dates
> with estimated dates. Instead, approximate dates would be used to flag
> books and newspaper pages that have previously been assigned dates
> that are dramatically inconsistent with the text, e.g., a forty-year
> difference. Humans would examine the flagged items manually. For this
> to be practical the rate of false positives must be low. The goal is
> to help clean up the metadata.
>
> Garson
>
> On Thu, Oct 29, 2026 at 12:50 PM ADSGarson O'Toole
> <adsgarsonotoole at gmail.com> wrote:
> >
> > Research in linguistics, history, machine learning, and more relies on
> > digital databases of books, magazines, and newspapers. Yet the quality
> > of these databases is often poor.
> >
> > Optical Character Recognition (OCR): Too many words are mangled or
> > omitted from OCR results. Often multicolumn formats are not segmented
> > correctly. Words from different columns are interleaved. When OCR is
> > applied to low-quality scans the result is complete gibberish. When
> > this occurs, the item that is supposed to be in the database is
> > effectively absent.
> >
> > Dates: Too many items are misdated. Often the correct dates can be
> > carefully extracted from scans of newspapers and magazines. This
> > should be done. In addition, today's AI technology should allow an
> > estimate of the correct date to be generated based on extended text
> > excerpts. The vocabulary, grammatical structures, and events help to
> > identify the period. These estimates can provide sanity checks. When
> > the estimate differs significantly from the date listed in the
> > metadata, a flag should be set for manual review of the date.
> >
> > Do other list members have ideas for improving the text databases?
> > Garson

------------------------------------------------------------
The American Dialect Society - http://www.americandialect.org