[Ads-l] An initial assessment of ChatGPT's skills in historical linguistics and etymology

[Daphne Preston-Kendal]

The Internet has been aflutter lately with people trying out ChatGPT, a supposedly artificially intelligent chatbot which can solve various tasks posed to it in natural language, answering likewise in fluent English text. To do this, it has ‘read’ a vast amount of literature on all subjects from many sources (OpenAI, the company who made it, don’t say which ones) and combines its knowledge of these sources to produce new texts. Its actual knowledge in its does not go beyond the mere linguistic level: as some have described it, all it is doing is ‘predicting the next word’ in its answer using corpus statistics.

It is an interesting experiment, but general consensus seems to be that it cannot tell what’s true and what isn’t very well, sometimes just making things up. It is quite good at some creative tasks, including even writing computer programs, but its answers are generally very mixed in quality. (Its poetry is about at the level of William McGonagall.)

I decided to experiment to see how well it can answer questions about historical linguistics and English etymology.

I started with what I would consider a moderate question for an introductory historical linguistics exam, with potential for it to really shine and give a detailed, advanced-level answer:

> Explain the interactions of Grimm's and Verner's laws in Indo-European linguistics, giving examples

ChatGPT’s response:

> Grimm's Law and Verner's Law are two sound laws that describe the changes in the consonants of the proto-Indo-European language as it evolved into various daughter languages. Grimm's Law explains the systematic shift of the proto-Indo-European voiceless stops to voiceless fricatives in certain environments, while Verner's Law explains the alternation between voiced and voiceless fricatives in certain other environments.
> 
> Grimm's Law describes how the proto-Indo-European voiceless stops *p, *t, and *k shifted to voiceless fricatives *f, *θ, and *x (or *h) in the Germanic languages. For example, the proto-Indo-European word *pénkʷe (meaning "five") became *fimf in Old High German, *fimf in Old English, and *fimm in Old Norse.
> 
> Verner's Law explains why some voiceless fricatives became voiced in certain environments. Specifically, it describes how an originally voiced sound in the preceding syllable, specifically a voiced consonant or a vowel, prevented the devoicing of a voiceless fricative in the following syllable. For example, the proto-Indo-European word *bʰréh₂tēr (meaning "brother") became *brōþēr in Old English due to the voiced vowel in the preceding syllable. Similarly, the proto-Indo-European word *déḱm̥t (meaning "ten") became *tehun in Old High German due to the voiced consonant in the preceding syllable.

An inauspicious start, since, while it explained the relevant laws, it did so badly (missing all but the (logically) final stage of Grimm's law), provided a bad example of Grimm's law (I would grudgingly admit that *pénkʷe >>> five is an example of Grimm's law, but there are better ones, given the irregular *kʷ > f change), two IE to Germanic examples which are not actually examples of Verner’s law, and most crucially, failed to explain what I had asked it for: the *interaction* between the two laws. So much for my hope that it might give a truly excellent answer: it fails to give an answer I would hope from a first semester student.

While disappointing, its example at least hints at where it went wrong: I think the difference between the developments of *bʰréh₂tēr and *ph₂tḗr is fairly standard as an example of Verner's law. If I were to speculate, I’d say its ‘language model’ saw *bʰréh₂tēr mentioned in a discussion of Verner’s law but missed the crucial point that it’s actually a *contrast* to a semantically and phonologically similar word which underwent a different development, and that this is the example which *doesn’t* show Verner’s law.

I decided to give it a little help along:

> In what environments does Verner's law take place?


Its answer:

> Verner's Law is a sound law that describes a specific pattern of sound change in the Germanic languages. It explains why certain voiceless fricatives in Germanic languages became voiced in certain environments. Specifically, Verner's Law applies when a voiceless fricative (such as *f, *þ, *s, or *h) appears in the onset of a stressed syllable in a word, and the preceding syllable in the word was originally unstressed and had an unstressed vowel, a laryngeal consonant (*h1, *h2, or *h3), or a voiced obstruent (a stop or a fricative).
> 
> In these specific environments, Verner's Law explains that the voiceless fricative would become voiced. For example, the proto-Germanic word *frijaz (meaning "free") had a voiceless fricative in the onset of the stressed syllable, but because the preceding syllable contained an unstressed vowel, it underwent Verner's Law and the fricative became voiced, resulting in the Old English word frēo (meaning "free").


This time, it gave a correct explanation of the law but added an additional, but (as far as I can tell) meaningless extra condition. Its example, however, once again fails to illustrate Verner's law. It doesn’t even give the PIE root. Indeed, it’s only by interpreting the answer in the context of its previous answer that we can glean that its talk of stressed and unstressed syllables refers to the original situation in PIE and not in Proto-Germanic.

I tried something slightly more obscure:

> Explain Pinault's law in Proto-Indo-European

This time, it came up with:

> "Pinault's Law" is a proposed sound change in Proto-Indo-European (PIE) linguistics that suggests a regular correspondence between PIE *s and Proto-Indo-Iranian *h. Specifically, the law suggests that in certain phonological environments, PIE *s regularly shifted to Proto-Indo-Iranian *h.
> 
> According to Pinault's Law, PIE *s would have shifted to *h in Proto-Indo-Iranian in the following environments:
>     • Before voiced stops and voiced aspirates
>     • Before a syllabic resonant (i.e. *r̥, *l̥, *m̥, *n̥)
>     • Before a laryngeal consonant (*h1, *h2, *h3)
> 
> For example, the PIE word *k̑sépṓs ("vine") would be expected to become *k̑hépṓs in Proto-Indo-Iranian under Pinault's Law, due to the voicing of the following stop *p.
> 
> While Pinault's Law has not yet been widely accepted as a sound law in PIE linguistics, it remains a topic of ongoing research and discussion.

As far as I can tell, this is completely made up. Its example (again) doesn’t even illustrate the law it made up/attempted to describe, and I think is an entirely invented PIE word. But the nature of the thing is that it might have picked up some obscure paper from goodness-knows-when and found this PIE reconstruction there. You can certainly see that it’s probably synthesized some information about Indo-Iranian sound laws to produce this, possibly indeed from papers by or citing Pinault.

Satisfied that the jobs of Indo-European scholars are safe from AI takeover, at least for the immediately foreseeable future, I turned my attention to questions of English etymology. ChatGPT is known to be at least somewhat vulnerable to repeating urban legends and myths. I asked it:

> Explain the origin of the English word Easter

wondering if it would give me the popular story about the Babylonian goddess Ishtar. It came up with:

> The origin of the English word "Easter" is somewhat uncertain, but it is generally believed to come from the Old English word "Ēastre" or "Ēostre," which was the name of a pagan festival celebrating the spring equinox. This festival was dedicated to the goddess Ēostre, who was associated with fertility, new beginnings, and the dawn.
> 
> When Christianity began to spread throughout England, missionaries sought to incorporate Christian beliefs and practices into the existing culture. One strategy was to appropriate pagan festivals and rebrand them with Christian meaning. As a result, the festival of Ēostre was transformed into a Christian holiday celebrating the resurrection of Jesus Christ.
> 
> The Christian holiday became known as "Easter" in English, possibly borrowing the name from the earlier pagan festival. Another theory suggests that the English word "Easter" may have been influenced by the Latin word "alba," which means "white," since newly baptized Christians would wear white robes during the Easter season.
> 
> Overall, while the exact origins of the English word "Easter" are not entirely clear, it is clear that the holiday has evolved over time to become an important Christian celebration of resurrection and renewal.

Its level of hedging could be considered a parody of an English etymologist (it claims the association between the pagan Easter festival and the Christianized Easter festival is certain, but then hedges and says the name itself is only ‘possibly’ borrowed from it). I’ve never heard the theory about alba before — sound-wise it’s obvious nonsense, but perhaps it means some semantic connection? I’m not sure. Connections between white and Christian Easter are linguistically fairly common.

I challenged it again along similar lines:

> What's the origin of the English word posh?

Its answer:

> The origin of the English word "posh" is uncertain, but there are a few theories about its origins. One theory suggests that "posh" may be an acronym for "Port Out, Starboard Home," which refers to the more desirable cabins on ships traveling between England and India during the 19th century. Passengers who could afford to travel in these cabins would avoid the sun's heat by having their cabins on the shaded side of the ship during both the outward and return voyages.
> 
> However, this theory is not widely accepted by linguists and etymologists. Another theory is that "posh" is derived from the Romani word "posh-ken," which means "half" or "a portion," and was used by Romani people to describe their living conditions. From this, the word "posh" may have come to be associated with luxury and wealth.
> 
> However, there is little concrete evidence to support this theory, either. It is also possible that "posh" simply evolved from a slang word meaning "dandy" or "foppish," which was used in England in the early 19th century to describe a fashionable or stylish person. Over time, the meaning of "posh" may have shifted to refer more broadly to any person or thing that is considered luxurious or high-class.

My response to this would be along much more nit-picking lines than any previous answer it gave. It fails to say up-front and explicitly why the ‘Port Out, Starboard Home’ theory is not widely accepted, although it later implies it with its ‘little concrete evidence … either’. I think a responsible etymologist would also have guarded against lazy readers by saying the theory is wrong *before* actually describing it.

As one final challenge, I gave it a problem combining Indo-European and English etymology, again hoping to catch it out:

> What is the Indo-European origin of the English word bone?

Its answer:

> The English word "bone" is believed to have originated from the Proto-Indo-European root *os-/*ost- (meaning "bone"), which is also the source of many other words in various Indo-European languages.
> 
> In Sanskrit, the word for "bone" is "asthi," which can be traced back to the same Proto-Indo-European root. In Greek, the word for "bone" is "osteon," which is derived from the same root as well. Similarly, in Latin, the word for "bone" is "os," also derived from the same Proto-Indo-European root.
> 
> The Proto-Indo-European root *os-/*ost- likely referred specifically to "bone" as opposed to "flesh" or other parts of the body. This is reflected in the various daughter languages, where the same root is used to refer to bone, but different roots are used to refer to flesh or muscle.
> 
> Overall, the English word "bone" can be traced back to a common Proto-Indo-European root that has cognates in many other Indo-European languages.

Indeed, I did catch it out, but not in the way I expected. It found a PIE root for bone, but one that has no connection to the English word. It also gave a pre-laryngeal reconstruction this time.

In summary, ChatGPT seems from this informal and limited test to be quite good at avoiding uncritically repeating etymological myths, but its actual skills at resolving etymologies and even explaining basic facts of historical linguistics are limited. While other fields are rightly concerned about students using GPT models to write essays for them,* I can say with moderate confidence that historical linguists can probably thank the obscurity of their subject for the fact that ChatGPT is not even yet at the point of being able to write a satisfactory homework answer on this subject. Perhaps the model will evolve in future, of course – this has also been the source of much public speculation.

Apart from the limited number of questions I asked, one additional point which limits the validity of this investigation is that I’m not (yet) very experienced in using ChatGPT. I know that regular users claim to have learned to formulate their questions such that they get higher-quality answers.

* I’m not sure how valid this fear is as applied to other subjects either, at least at the moment. When I first tried it out, a month or so ago, I asked it to explain some basic facts about the Crimean War to me, a subject considerably less obscure in the scheme of general knowledge than Indo-European sound laws. Its answer (which I unfortunately didn’t save) was full of basic factual errors.

-- 
dpk (Daphne Preston-Kendal) ·· 12103 Berlin, Germany ·· http://dpk.io/
 ‘On two occasions I have been asked [by Members of Parliament] “Pray,
    Mr. Babbage, if you put into the machine wrong figures, will the
    right answers come out?” I am not able rightly to apprehend the
    kind of confusion of ideas that could provoke such a question.’
                                                  — Charles Babbage

------------------------------------------------------------
The American Dialect Society - http://www.americandialect.org