[Ads-l] Improving text databases

[dave@wilton.net]

That's good to know. If the goal is to flag titles with suspect dates for human review, that's a reasonable use. 
 
But I note the study was done using only NY Times articles. That's one publication with a distinct style that primarily publishes in one genre, news. I wonder how it would fare with publications from a variety of publishes in a variety of genres: news, fiction, poetry, academic papers, long-form non-fiction, etc. I suspect the results would not be as good.
 
 
-----Original Message-----
From: "ADSGarson O'Toole" <00001aa1be50b751-dmarc-request at LISTSERV.UGA.EDU>
Sent: Friday, October 30, 2026 6:13pm
To: ADS-L at LISTSERV.UGA.EDU
Subject: Re: [ADS-L] Improving text databases



Research has been conducted on estimating publication dates from text
passages. Here is a pertinent article based on a corpus of New York
Times articles:

Date: April 24, 2023
Article Title: Text2Time: Transformer-based Article Time Period Prediction
Authors: Karthick Prasad Gunasekaran, B Chase Babrich, Saurabh
Shirodkar, Hee Hwang
Location: University of Massachusetts, Amherst, USA
https://arxiv.org/abs/2304.10859v2

[Begin excerpt]
The task of predicting the publication period of text documents, such
as news articles, is an important but less studied problem in the
field of natural language processing. Predicting the year of a news
article can be useful in various contexts, such as historical
research, sentiment analysis, and media monitoring. In this work, we
investigate the problem of predicting the publication period of a text
document, specifically a news article, based on its textual content.
In order to do so, we created our own extensive labeled dataset of
over 350,000 news articles published by The New York Times over six
decades. In our approach, we use a pretrained BERT model fine-tuned
for the task of text classification, specifically for time period
prediction. This model exceeds our expectations and provides some very
impressive results in terms of accurately classifying news articles
into their respective publication decades. The results beat the
performance of the baseline model for this relatively unexplored task
of time prediction from text.
[End excerpt]

A more comprehensive and improved algorithm of this type would be
helpful. Note, the goal is not to recklessly replace existing dates
with estimated dates. Instead, approximate dates would be used to flag
books and newspaper pages that have previously been assigned dates
that are dramatically inconsistent with the text, e.g., a forty-year
difference. Humans would examine the flagged items manually. For this
to be practical the rate of false positives must be low. The goal is
to help clean up the metadata.

Garson

On Thu, Oct 29, 2026 at 12:50 PM ADSGarson O'Toole
<adsgarsonotoole at gmail.com> wrote:
>
> Research in linguistics, history, machine learning, and more relies on
> digital databases of books, magazines, and newspapers. Yet the quality
> of these databases is often poor.
>
> Optical Character Recognition (OCR): Too many words are mangled or
> omitted from OCR results. Often multicolumn formats are not segmented
> correctly. Words from different columns are interleaved. When OCR is
> applied to low-quality scans the result is complete gibberish. When
> this occurs, the item that is supposed to be in the database is
> effectively absent.
>
> Dates: Too many items are misdated. Often the correct dates can be
> carefully extracted from scans of newspapers and magazines. This
> should be done. In addition, today's AI technology should allow an
> estimate of the correct date to be generated based on extended text
> excerpts. The vocabulary, grammatical structures, and events help to
> identify the period. These estimates can provide sanity checks. When
> the estimate differs significantly from the date listed in the
> metadata, a flag should be set for manual review of the date.
>
> Do other list members have ideas for improving the text databases?
> Garson

------------------------------------------------------------
The American Dialect Society - http://www.americandialect.org

------------------------------------------------------------
The American Dialect Society - http://www.americandialect.org