34.1319, Books: Image Captioning with External Knowledge: Nikiforova

[The LINGUIST List]

LINGUIST List: Vol-34-1319. Wed Apr 26 2023. ISSN: 1069 - 4875.

Subject: 34.1319, Books: Image Captioning with External Knowledge: Nikiforova

Moderator: Malgorzata E. Cavar, Francis Tyers (linguist at linguistlist.org)
Managing Editor: Lauren Perkins
Team: Helen Aristar-Dry, Steven Franks, Everett Green, Joshua Sims, Daniel Swanson, Matthew Fort, Maria Lucero Guillen Puon, Zackary Leech, Lynzie Coburn
Jobs: jobs at linguistlist.org | Conferences: callconf at linguistlist.org | Pubs: pubs at linguistlist.org

Homepage: http://linguistlist.org

Please support the LL editors and operation with a donation at:
           https://funddrive.linguistlist.org/donate/

Editor for this issue: Maria Lucero Guillen Puon <luceroguillen at linguistlist.org>
================================================================


Date: 19-Apr-2023
From: Tessa Arneri [lotdissertations-fgw at uva.nl]
Subject: Image Captioning with External Knowledge: Nikiforova


Title: Image Captioning with External Knowledge
Series Title: LOT Dissertation Series
Publication Year: 2023
Publisher: Netherlands Graduate School of Linguistics / Landelijke
(LOT)
                http://www.lotpublications.nl/
Book URL: https://www.lotpublications.nl/image-captioning-with-externa
l-knowledge

Author: Sofia Nikiforova
Paperback: ISBN: 9789460934261 Pages: 147 Price: Europe EURO 31
Abstract:

In modern automatic image captioning, generating straightforward
visual descriptions of images is largely a solved problem. One of the
biggest challenges that still remains is incorporating information
that cannot be inferred from the image alone: its context and related
real world knowledge. In this dissertation, we tackle this challenge
by developing a new method of enriching an otherwise standard
captioning pipeline with contextually relevant image-external
knowledge.

Our method starts with identifying the subset of data from external
sources that is relevant to a given image. The retrieved data is
integrated into the caption generation process, aiming to influence
the resulting caption and extend it beyond a purely visual
description. Based on this general method, we develop three neural
image captioning models. The first model addresses a specific problem
of generating references to the geographic context of the image. The
second model expands to broad encyclopedic knowledge about the
depicted geographic entities. Finally, the third model generalizes
beyond the geographic domain and applies our method to diverse images
from newspaper articles. The evaluation of the models shows that our
method is indeed effective for producing contextualized and
informative captions with factually accurate references to relevant
external knowledge.

Linguistic Field(s): Computational Linguistics

Written In: English (eng)

See this book announcement on our website:
http://linguistlist.org/pubs/books/get-book.cfm?BookID=170273



------------------------------------------------------------------------------


LINGUIST List is supported by the following publishers:

American Dialect Society/Duke University Press http://dukeupress.edu

Bloomsbury Publishing (formerly The Continuum International Publishing Group) http://www.bloomsbury.com/uk/

Brill http://www.brill.com

Cambridge Scholars Publishing http://www.cambridgescholars.com/

Cambridge University Press http://www.cambridge.org/linguistics

Cascadilla Press http://www.cascadilla.com/

De Gruyter Mouton https://cloud.newsletter.degruyter.com/mouton

Dictionary Society of North America http://dictionarysociety.com/

Edinburgh University Press www.edinburghuniversitypress.com

Equinox Publishing Ltd http://www.equinoxpub.com/

European Language Resources Association (ELRA) http://www.elra.info

Georgetown University Press http://www.press.georgetown.edu

John Benjamins http://www.benjamins.com/

Lincom GmbH https://lincom-shop.eu/

Linguistic Association of Finland http://www.ling.helsinki.fi/sky/

Multilingual Matters http://www.multilingual-matters.com/

Narr Francke Attempto Verlag GmbH + Co. KG http://www.narr.de/

Netherlands Graduate School of Linguistics / Landelijke (LOT) http://www.lotpublications.nl/

Oxford University Press http://www.oup.com/us

Wiley http://www.wiley.com


----------------------------------------------------------
LINGUIST List: Vol-34-1319
----------------------------------------------------------