[Ads-l] An initial assessment of ChatGPT's skills in historical linguistics and etymology
Laurence Horn
laurence.horn at YALE.EDU
Tue Feb 14 17:23:01 UTC 2023
Back to etymology: I decided to have the grudge match feature ChatGPT vs. OED and focus on terms for creative etymologies. Both contestants handled “mondegreen” successfully, noting the connection to the Sylvia Wright piece in Harper’s in 1984. “Eggcorn” was more of a challenge, especially for the bot. The OED entry is accurate enough, although somewhat compressed:
An alteration of a word or phrase through the mishearing or reinterpretation of one or more of its elements as a similar-sounding word.
2003 M. Liberman Egg Corns: Folk Etymol., Malapropism, Mondegreen, ???: update in languagelog.ldc.upenn.edu 30 Sept. (Internet Archive Wayback Machine 8 Oct. 2003) Geoff Pullum suggests that if no suitable term already exists for cases like this, we should call them ‘egg corns’, in the metonymic tradition of ‘mondegreen’.
2004 Boston Globe <> (Nexis) 12 Dec. k5 Shakespeare's Hamlet said he was ‘to the manner born’, but the eggcorn ‘to the manor born’ has wide currency.
Here’s the more voluble ChatGPT:
The term "eggcorn" was coined by linguist Mark Liberman in 2003, based on a mishearing of the word "acorn" as "eggcorn" by a woman named Sylvia, which she used to describe a type of seed. The term "eggcorn" was then used by Liberman to refer to a specific type of linguistic phenomenon known as a "folk etymology," where a word or phrase is misheard, misspelled, or misinterpreted in a way that creates a new word or phrase that is similar in sound or meaning to the original. Eggcorns are different from other types of language errors, like a mondegreen, because they often involve the creation of a new word or phrase that has a different meaning from the original, rather than simply misinterpreting the original.
In fact, as per the OED entry, the 9.30.2003 LL post, http://itre.cis.upenn.edu/~myl/languagelog/archives/2003_09.html <http://itre.cis.upenn.edu/~myl/languagelog/archives/2003_09.html>, confirms that the suggested coinage is largely Pullum's, although introduced to the wider world by Liberman, but more crucially no "woman named Sylvia" is featured in the story. I suspect ChatGPT is free associating from Sylvia Wright of “mondegreen” fame.
The next challenge, selfishly, was “etymythology”. The OED doesn’t mention it (suggesting I try “etymology” or “mythology” instead), and ChatGPT enters its familiar alternate history mode:
The term "etymythology" was first introduced by the linguist Laurence Horn in his 1984 book "A Natural History of Negation." Horn combined the words "etymology" and "mythology" to describe false or fanciful etymologies, which are popular but incorrect explanations of the origins of words. The term has since been used in linguistics and other fields to describe similar misconceptions about word origins or histories.
As is often the case, the bot should have quit while it was ahead. The book in question was published in 1989 (perhaps the 1984 date comes from that same Sylvia Wright article introducing “mondegreen”) and in any case the book contained no mention of “etymythology", which was first proposed on this list.
My last try was the etymology of “spitting image”, which the OED used to claim “Repr. corrupted pronunc. spit and (image, picture)”, but now derives as “an alteration of spitten image”, as persuasively argued in a certain 2004 paper in American Speech. ChatGPT is evidently unfamiliar with that work and still nods to “spit and image”.
So the OED wins the derive-off on points, but as noted by others the chatbot can be expected to improve, and certainly wins on creativity.
LH
> On Feb 14, 2023, at 11:35 AM, Geoffrey Nathan <geoffnathan at WAYNE.EDU> wrote:
>
> Just for fun, I reran Daphne's test of ChatGPT with
> the prompt:
>
> 'Explain the relationship between Grimm's law
> and Verner's law.'
>
> It got the relationship between the laws correct (i.e. it
> mentioned accent as a crucial variable, and mentioned
> exceptions to Grimm's law), but
> it defined Grimm's Law as consisting only of the changes
> to voiceless stops, omitting voiced and voiced aspirates.
> I would have given the answer a B- for that reason.
>
> Geoff
>
> Geoffrey S. Nathan
> WSU Information Privacy Officer (Retired)
> Emeritus Professor, Linguistics Program
> https://clasprofiles.wayne.edu/profile/an6993
> geoffnathan at wayne.edu
>
> From: Daphne Preston-Kendal<mailto:dpk at NONCEWORD.ORG>
> Sent: Tuesday, February 14, 2023 5:58 AM
> To: ADS-L at LISTSERV.UGA.EDU<mailto:ADS-L at LISTSERV.UGA.EDU>
> Subject: An initial assessment of ChatGPT's skills in historical linguistics and etymology
>
> [EXTERNAL]
>
> ---------------------- Information from the mail header -----------------------
> Sender: American Dialect Society <ADS-L at LISTSERV.UGA.EDU>
> Poster: Daphne Preston-Kendal <dpk at NONCEWORD.ORG>
> Subject: An initial assessment of ChatGPT's skills in historical
> linguistics and etymology
> -------------------------------------------------------------------------------
>
> The Internet has been aflutter lately with people trying out ChatGPT, a =
> supposedly artificially intelligent chatbot which can solve various =
> tasks posed to it in natural language, answering likewise in fluent =
> English text. To do this, it has =E2=80=98read=E2=80=99 a vast amount of =
> literature on all subjects from many sources (OpenAI, the company who =
> made it, don=E2=80=99t say which ones) and combines its knowledge of =
> these sources to produce new texts. Its actual knowledge in its does not =
> go beyond the mere linguistic level: as some have described it, all it =
> is doing is =E2=80=98predicting the next word=E2=80=99 in its answer =
> using corpus statistics.
>
> It is an interesting experiment, but general consensus seems to be that =
> it cannot tell what=E2=80=99s true and what isn=E2=80=99t very well, =
> sometimes just making things up. It is quite good at some creative =
> tasks, including even writing computer programs, but its answers are =
> generally very mixed in quality. (Its poetry is about at the level of =
> William McGonagall.)
>
> I decided to experiment to see how well it can answer questions about =
> historical linguistics and English etymology.
>
> I started with what I would consider a moderate question for an =
> introductory historical linguistics exam, with potential for it to =
> really shine and give a detailed, advanced-level answer:
>
>> Explain the interactions of Grimm's and Verner's laws in Indo-European =
> linguistics, giving examples
>
> ChatGPT=E2=80=99s response:
>
>> Grimm's Law and Verner's Law are two sound laws that describe the =
> changes in the consonants of the proto-Indo-European language as it =
> evolved into various daughter languages. Grimm's Law explains the =
> systematic shift of the proto-Indo-European voiceless stops to voiceless =
> fricatives in certain environments, while Verner's Law explains the =
> alternation between voiced and voiceless fricatives in certain other =
> environments.
>> =20
>> Grimm's Law describes how the proto-Indo-European voiceless stops *p, =
> *t, and *k shifted to voiceless fricatives *f, *=CE=B8, and *x (or *h) =
> in the Germanic languages. For example, the proto-Indo-European word =
> *p=C3=A9nk=CA=B7e (meaning "five") became *fimf in Old High German, =
> *fimf in Old English, and *fimm in Old Norse.
>> =20
>> Verner's Law explains why some voiceless fricatives became voiced in =
> certain environments. Specifically, it describes how an originally =
> voiced sound in the preceding syllable, specifically a voiced consonant =
> or a vowel, prevented the devoicing of a voiceless fricative in the =
> following syllable. For example, the proto-Indo-European word =
> *b=CA=B0r=C3=A9h=E2=82=82t=C4=93r (meaning "brother") became *br=C5=8D=C3=BE=
> =C4=93r in Old English due to the voiced vowel in the preceding =
> syllable. Similarly, the proto-Indo-European word *d=C3=A9=E1=B8=B1m=CC=A5=
> t (meaning "ten") became *tehun in Old High German due to the voiced =
> consonant in the preceding syllable.
>
> An inauspicious start, since, while it explained the relevant laws, it =
> did so badly (missing all but the (logically) final stage of Grimm's =
> law), provided a bad example of Grimm's law (I would grudgingly admit =
> that *p=C3=A9nk=CA=B7e >>> five is an example of Grimm's law, but there =
> are better ones, given the irregular *k=CA=B7 > f change), two IE to =
> Germanic examples which are not actually examples of Verner=E2=80=99s =
> law, and most crucially, failed to explain what I had asked it for: the =
> *interaction* between the two laws. So much for my hope that it might =
> give a truly excellent answer: it fails to give an answer I would hope =
> from a first semester student.
>
> While disappointing, its example at least hints at where it went wrong: =
> I think the difference between the developments of *b=CA=B0r=C3=A9h=E2=82=82=
> t=C4=93r and *ph=E2=82=82t=E1=B8=97r is fairly standard as an example of =
> Verner's law. If I were to speculate, I=E2=80=99d say its =E2=80=98languag=
> e model=E2=80=99 saw *b=CA=B0r=C3=A9h=E2=82=82t=C4=93r mentioned in a =
> discussion of Verner=E2=80=99s law but missed the crucial point that =
> it=E2=80=99s actually a *contrast* to a semantically and phonologically =
> similar word which underwent a different development, and that this is =
> the example which *doesn=E2=80=99t* show Verner=E2=80=99s law.
>
> I decided to give it a little help along:
>
>> In what environments does Verner's law take place?
>
>
> Its answer:
>
>> Verner's Law is a sound law that describes a specific pattern of sound =
> change in the Germanic languages. It explains why certain voiceless =
> fricatives in Germanic languages became voiced in certain environments. =
> Specifically, Verner's Law applies when a voiceless fricative (such as =
> *f, *=C3=BE, *s, or *h) appears in the onset of a stressed syllable in a =
> word, and the preceding syllable in the word was originally unstressed =
> and had an unstressed vowel, a laryngeal consonant (*h1, *h2, or *h3), =
> or a voiced obstruent (a stop or a fricative).
>> =20
>> In these specific environments, Verner's Law explains that the =
> voiceless fricative would become voiced. For example, the proto-Germanic =
> word *frijaz (meaning "free") had a voiceless fricative in the onset of =
> the stressed syllable, but because the preceding syllable contained an =
> unstressed vowel, it underwent Verner's Law and the fricative became =
> voiced, resulting in the Old English word fr=C4=93o (meaning "free").
>
>
> This time, it gave a correct explanation of the law but added an =
> additional, but (as far as I can tell) meaningless extra condition. Its =
> example, however, once again fails to illustrate Verner's law. It =
> doesn=E2=80=99t even give the PIE root. Indeed, it=E2=80=99s only by =
> interpreting the answer in the context of its previous answer that we =
> can glean that its talk of stressed and unstressed syllables refers to =
> the original situation in PIE and not in Proto-Germanic.
>
> I tried something slightly more obscure:
>
>> Explain Pinault's law in Proto-Indo-European
>
> This time, it came up with:
>
>> "Pinault's Law" is a proposed sound change in Proto-Indo-European =
> (PIE) linguistics that suggests a regular correspondence between PIE *s =
> and Proto-Indo-Iranian *h. Specifically, the law suggests that in =
> certain phonological environments, PIE *s regularly shifted to =
> Proto-Indo-Iranian *h.
>> =20
>> According to Pinault's Law, PIE *s would have shifted to *h in =
> Proto-Indo-Iranian in the following environments:
>> =E2=80=A2 Before voiced stops and voiced aspirates
>> =E2=80=A2 Before a syllabic resonant (i.e. *r=CC=A5, *l=CC=A5, =
> *m=CC=A5, *n=CC=A5)
>> =E2=80=A2 Before a laryngeal consonant (*h1, *h2, *h3)
>> =20
>> For example, the PIE word *k=CC=91s=C3=A9p=E1=B9=93s ("vine") would be =
> expected to become *k=CC=91h=C3=A9p=E1=B9=93s in Proto-Indo-Iranian =
> under Pinault's Law, due to the voicing of the following stop *p.
>> =20
>> While Pinault's Law has not yet been widely accepted as a sound law in =
> PIE linguistics, it remains a topic of ongoing research and discussion.
>
> As far as I can tell, this is completely made up. Its example (again) =
> doesn=E2=80=99t even illustrate the law it made up/attempted to =
> describe, and I think is an entirely invented PIE word. But the nature =
> of the thing is that it might have picked up some obscure paper from =
> goodness-knows-when and found this PIE reconstruction there. You can =
> certainly see that it=E2=80=99s probably synthesized some information =
> about Indo-Iranian sound laws to produce this, possibly indeed from =
> papers by or citing Pinault.
>
> Satisfied that the jobs of Indo-European scholars are safe from AI =
> takeover, at least for the immediately foreseeable future, I turned my =
> attention to questions of English etymology. ChatGPT is known to be at =
> least somewhat vulnerable to repeating urban legends and myths. I asked =
> it:
>
>> Explain the origin of the English word Easter
>
> wondering if it would give me the popular story about the Babylonian =
> goddess Ishtar. It came up with:
>
>> The origin of the English word "Easter" is somewhat uncertain, but it =
> is generally believed to come from the Old English word "=C4=92astre" or =
> "=C4=92ostre," which was the name of a pagan festival celebrating the =
> spring equinox. This festival was dedicated to the goddess =C4=92ostre, =
> who was associated with fertility, new beginnings, and the dawn.
>> =20
>> When Christianity began to spread throughout England, missionaries =
> sought to incorporate Christian beliefs and practices into the existing =
> culture. One strategy was to appropriate pagan festivals and rebrand =
> them with Christian meaning. As a result, the festival of =C4=92ostre =
> was transformed into a Christian holiday celebrating the resurrection of =
> Jesus Christ.
>> =20
>> The Christian holiday became known as "Easter" in English, possibly =
> borrowing the name from the earlier pagan festival. Another theory =
> suggests that the English word "Easter" may have been influenced by the =
> Latin word "alba," which means "white," since newly baptized Christians =
> would wear white robes during the Easter season.
>> =20
>> Overall, while the exact origins of the English word "Easter" are not =
> entirely clear, it is clear that the holiday has evolved over time to =
> become an important Christian celebration of resurrection and renewal.
>
> Its level of hedging could be considered a parody of an English =
> etymologist (it claims the association between the pagan Easter festival =
> and the Christianized Easter festival is certain, but then hedges and =
> says the name itself is only =E2=80=98possibly=E2=80=99 borrowed from =
> it). I=E2=80=99ve never heard the theory about alba before =E2=80=94 =
> sound-wise it=E2=80=99s obvious nonsense, but perhaps it means some =
> semantic connection? I=E2=80=99m not sure. Connections between white and =
> Christian Easter are linguistically fairly common.
>
> I challenged it again along similar lines:
>
>> What's the origin of the English word posh?
>
> Its answer:
>
>> The origin of the English word "posh" is uncertain, but there are a =
> few theories about its origins. One theory suggests that "posh" may be =
> an acronym for "Port Out, Starboard Home," which refers to the more =
> desirable cabins on ships traveling between England and India during the =
> 19th century. Passengers who could afford to travel in these cabins =
> would avoid the sun's heat by having their cabins on the shaded side of =
> the ship during both the outward and return voyages.
>> =20
>> However, this theory is not widely accepted by linguists and =
> etymologists. Another theory is that "posh" is derived from the Romani =
> word "posh-ken," which means "half" or "a portion," and was used by =
> Romani people to describe their living conditions. =46rom this, the word =
> "posh" may have come to be associated with luxury and wealth.
>> =20
>> However, there is little concrete evidence to support this theory, =
> either. It is also possible that "posh" simply evolved from a slang word =
> meaning "dandy" or "foppish," which was used in England in the early =
> 19th century to describe a fashionable or stylish person. Over time, the =
> meaning of "posh" may have shifted to refer more broadly to any person =
> or thing that is considered luxurious or high-class.
>
> My response to this would be along much more nit-picking lines than any =
> previous answer it gave. It fails to say up-front and explicitly why the =
> =E2=80=98Port Out, Starboard Home=E2=80=99 theory is not widely =
> accepted, although it later implies it with its =E2=80=98little concrete =
> evidence =E2=80=A6 either=E2=80=99. I think a responsible etymologist =
> would also have guarded against lazy readers by saying the theory is =
> wrong *before* actually describing it.
>
> As one final challenge, I gave it a problem combining Indo-European and =
> English etymology, again hoping to catch it out:
>
>> What is the Indo-European origin of the English word bone?
>
> Its answer:
>
>> The English word "bone" is believed to have originated from the =
> Proto-Indo-European root *os-/*ost- (meaning "bone"), which is also the =
> source of many other words in various Indo-European languages.
>> =20
>> In Sanskrit, the word for "bone" is "asthi," which can be traced back =
> to the same Proto-Indo-European root. In Greek, the word for "bone" is =
> "osteon," which is derived from the same root as well. Similarly, in =
> Latin, the word for "bone" is "os," also derived from the same =
> Proto-Indo-European root.
>> =20
>> The Proto-Indo-European root *os-/*ost- likely referred specifically =
> to "bone" as opposed to "flesh" or other parts of the body. This is =
> reflected in the various daughter languages, where the same root is used =
> to refer to bone, but different roots are used to refer to flesh or =
> muscle.
>> =20
>> Overall, the English word "bone" can be traced back to a common =
> Proto-Indo-European root that has cognates in many other Indo-European =
> languages.
>
> Indeed, I did catch it out, but not in the way I expected. It found a =
> PIE root for bone, but one that has no connection to the English word. =
> It also gave a pre-laryngeal reconstruction this time.
>
> In summary, ChatGPT seems from this informal and limited test to be =
> quite good at avoiding uncritically repeating etymological myths, but =
> its actual skills at resolving etymologies and even explaining basic =
> facts of historical linguistics are limited. While other fields are =
> rightly concerned about students using GPT models to write essays for =
> them,* I can say with moderate confidence that historical linguists can =
> probably thank the obscurity of their subject for the fact that ChatGPT =
> is not even yet at the point of being able to write a satisfactory =
> homework answer on this subject. Perhaps the model will evolve in =
> future, of course =E2=80=93 this has also been the source of much public =
> speculation.
>
> Apart from the limited number of questions I asked, one additional point =
> which limits the validity of this investigation is that I=E2=80=99m not =
> (yet) very experienced in using ChatGPT. I know that regular users claim =
> to have learned to formulate their questions such that they get =
> higher-quality answers.
>
> * I=E2=80=99m not sure how valid this fear is as applied to other =
> subjects either, at least at the moment. When I first tried it out, a =
> month or so ago, I asked it to explain some basic facts about the =
> Crimean War to me, a subject considerably less obscure in the scheme of =
> general knowledge than Indo-European sound laws. Its answer (which I =
> unfortunately didn=E2=80=99t save) was full of basic factual errors.
>
> --=20
> dpk (Daphne Preston-Kendal) =C2=B7=C2=B7 12103 Berlin, Germany =C2=B7=C2=B7=
> http://dpk.io/
> =E2=80=98On two occasions I have been asked [by Members of Parliament] =
> =E2=80=9CPray,
> Mr. Babbage, if you put into the machine wrong figures, will the
> right answers come out?=E2=80=9D I am not able rightly to apprehend =
> the
> kind of confusion of ideas that could provoke such a question.=E2=80=99=
>
> =E2=80=94 Charles =
> Babbage
>
> ------------------------------------------------------------
> The American Dialect Society - http://www.americandialect.org
>
>
> ------------------------------------------------------------
> The American Dialect Society - http://www.americandialect.org
------------------------------------------------------------
The American Dialect Society - http://www.americandialect.org
More information about the Ads-l
mailing list