<html><head></head><body><div class="ydpf059903byahoo-style-wrap" style="font-family:Helvetica Neue, Helvetica, Arial, sans-serif;font-size:13px;"><div></div>
<div dir="ltr" data-setdir="false">Hi all,</div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false">Thank you for this very helpful discussion and it is heartening to see the hard work that different scholars are putting into improving pedagogical methods given the wrench that LLMs have put into things. A few small thoughts: although Nepali certainly is a less-commonly-taught language, many of the large tech companies treat Indics by bootstrapping from Hindi, for which there is much, much more data, to other members of the family, including varieties like Nepali and Assamese, for which there are much less data. The same is to some degree true for other language families, but a bit dependent upon the perceived size and socioeconomic value of the speaker populations and the complexity of adding the language varieties to the models for the given company. Another thing to consider is that companies developing LLMs have essentially scraped the entire internet of scrapable data (incredible as this may seem), so if there are books, dissertations, or journal articles on a language that are available on the open internet, they have almost certainly been scraped to train the LLMs. Linguists have worked really hard in the last almost two decades to make publications open access or otherwise freely available so that members of native language communities and other scholars can have access to them, so there is a huge amount of language data out there.</div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false">Best,</div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false">Liberty</div><div><br></div>
</div><div id="ydp8bbc286fyahoo_quoted_3161638533" class="ydp8bbc286fyahoo_quoted">
<div style="font-family:'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:13px;color:#26282a;">
<div>
On Thursday, November 6, 2025 at 09:12:01 PM PST, Spike Gildea via Lingtyp <lingtyp@listserv.linguistlist.org> wrote:
</div>
<div><br></div>
<div><br></div>
<div><div id="ydp8bbc286fyiv8933599793"><div>
<div dir="ltr" style="font-family:Aptos, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0, 0, 0);">
Hi all,</div>
<div dir="ltr" style="font-family:Aptos, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0, 0, 0);">
<br clear="none">
</div>
<div dir="ltr" style="font-family:Aptos, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0, 0, 0);">
This last summer, a team here at the University of Oregon tested a number of assignments from across the Humanities for susceptibility to AI. I offered them the take-home midterm from my advanced syntax class, a complex problem set using examples I had created
from my personal knowledge of the Nepali language. The data featured SOV order, postpositions, case-marking suffixes, optionality of core arguments, tense-based split ergativity, dative subjects, and differential object marking. I was confident that AI would
have little chance of finding and describing all these patterns. </div>
<div dir="ltr" style="font-family:Aptos, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0, 0, 0);">
<br clear="none">
</div>
<div dir="ltr" style="font-size:12pt;"><span style="font-family:Aptos, Arial, Helvetica, sans-serif;color:rgb(0, 0, 0);">On 5/28/2025, they gave the assignment to
</span><span style="font-family:Arial, sans-serif;color:rgb(102, 102, 102);background-color:rgb(255, 238, 128);line-height:20.7px;"><i>Gemini</i></span><span style="font-family:Arial, sans-serif;color:rgb(102, 102, 102);line-height:20.7px;"><i> 2.5
Pro Preview </i>and it </span><span style="font-family:Aptos, Arial, Helvetica, sans-serif;color:rgb(0, 0, 0);">not only identified most of the relevant patterns and successfully answered my descriptive questions, it also generated a strong essay about
the relevance of morphological vs. syntactic subject properties in the data. This essay correctly synthesized the relevant patterns in the assigned data, but it would have been suspicious to me because it also drew on theoretical perspectives (presumably from
scraping the internet) that I deliberately did not include in the class. So even though it was quite high-level work, I would certainly have called in the student to ask where they had picked up these out-of-class ideas, after which I suspect the truth would
have come out, since the student would have been unlikely to have the capacity to discuss the theoretical literature and why they had chosen to use these concepts instead of the ones I taught in class.</span></div>
<div dir="ltr" style="font-family:Aptos, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0, 0, 0);">
<br clear="none">
</div>
<div dir="ltr"><span style="font-family:Aptos, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0, 0, 0);">When I expressed my surprise at the success of AI at solving this problem, the testers told me that over the last two years, AI has taken massive
leaps forward in sophistication. They added: "</span><span style="font-family: Arial, sans-serif; font-size: 14.6667px;">These models are typically good at following instructions, and
they are trained in a large variety of languages and linguistics-related texts. As these models have considerable data in their training sets (or access to it on the internet) and entirely language-centric, they’re likely to do a reasonable if not very competent job."
and "Language is the specialty of LLMs. There is a lot of text out there likely scraped online and fed to these models. They can speak tens of languages, and they know a lot about linguistics. As non-experts, we cannot be certain that these answers are correct,
although they seem so at first glance. We don’t doubt the model’s capabilities in this field."</span></div>
<div dir="ltr" style="font-family:Aptos, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0, 0, 0);">
<br clear="none">
</div>
<div dir="ltr" style="font-family:Aptos, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0, 0, 0);">
Cheers!</div>
<div dir="ltr" style="font-family:Aptos, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0, 0, 0);">
Spike</div>
<div dir="ltr" style="font-family:Aptos, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0, 0, 0);">
<br clear="none">
</div>
<div id="ydp8bbc286fyiv8933599793mail-editor-reference-message-container">
<div dir="ltr" class="ydp8bbc286fyiv8933599793ms-outlook-mobile-reference-message ydp8bbc286fyiv8933599793skipProofing"></div>
<div id="ydp8bbc286fyiv8933599793yqt92167" class="ydp8bbc286fyiv8933599793yqt9607512131"><div style="text-align:left;padding:3pt 0in 0in;border-width:1pt medium medium;border-style:solid none none;border-color:rgb(181, 196, 223) currentcolor currentcolor;font-family:Aptos;font-size:12pt;color:black;" class="ydp8bbc286fyiv8933599793ms-outlook-mobile-reference-message ydp8bbc286fyiv8933599793skipProofing">
<b>From: </b>Lingtyp <lingtyp-bounces@listserv.linguistlist.org> on behalf of Alexander Coupe via Lingtyp <lingtyp@listserv.linguistlist.org><br clear="none">
<b>Date: </b>Thursday, November 6, 2025 at 7:27 PM<br clear="none">
<b>To: </b>Juergen Bohnemeyer <jb77@buffalo.edu>, Mark Post <mark.post@sydney.edu.au>, typology list <lingtyp@listserv.linguistlist.org><br clear="none">
<b>Subject: </b>Re: [Lingtyp] "AI" and linguistics problem sets<br clear="none">
<br clear="none">
</div>
<div dir="ltr" id="ydp8bbc286fyiv8933599793pfptBanner1mfwmpo" lang="en" style="background-color: rgb(208, 216, 220); visibility: visible !important; max-width: none !important; display: block !important; text-align: left !important; margin: 0px 0px 10px !important; padding: 7px 16px 8px !important; border-radius: 4px !important; min-width: 200px !important; border-top: 4px solid rgb(144, 164, 174) !important;">
<div id="ydp8bbc286fyiv8933599793pfptBanner1mfwmpo" style="background-color: rgb(208, 216, 220); visibility: visible !important; float: left !important; display: block !important; margin: 1px 0px !important; max-width: 600px !important;">
<div id="ydp8bbc286fyiv8933599793pfptBanner1mfwmpo" style="background-color: rgb(208, 216, 220); color: rgb(0, 0, 0); max-width: none !important; display: block !important; visibility: visible !important; font-family: Arial, sans-serif !important; font-weight: bold !important; font-size: 14px !important; line-height: 1.29 !important;">
This message originated outside the UO email ecosystem.</div>
<div id="ydp8bbc286fyiv8933599793pfptBanner1mfwmpo" style="font-weight: normal; background-color: rgb(208, 216, 220); color: rgb(0, 0, 0); max-width: none !important; display: block !important; visibility: visible !important; font-family: Arial, sans-serif !important; font-size: 12px !important; line-height: 1.5 !important; margin-top: 2px !important;">
Use caution with links and attachments. Learn more about this <span style="color:blue;">
<u><a shape="rect" href="https://service.uoregon.edu/TDClient/2030/Portal/KB/ArticleDet?ID=141098" style="color:blue;" rel="nofollow" target="_blank">email warning tag</a></u></span>.</div>
</div>
<div id="ydp8bbc286fyiv8933599793pfptBanner1mfwmpo" style="background-color: rgb(208, 216, 220); visibility: visible !important; max-width: none !important; float: right !important; display: block !important; margin-left: 16px !important; margin-top: 1px !important; text-align: right !important; font-size: 12px !important;">
<div style="margin-top: 3px; margin-bottom: 3px; margin-left: 16px; padding: 7.5px 16px; border-width: 1px; border-style: solid; border-color: rgb(102, 102, 102); border-radius: 2px; display: inline-block; font-family: Arial, sans-serif; font-size: 14px; color: blue;" class="ydp8bbc286fyiv8933599793pfptPrimaryButton1mfwmpo">
<span style="background-color:rgb(208, 216, 220);"><a shape="rect" id="ydp8bbc286fyiv8933599793pfptBanner1mfwmpo" href="https://us-phishalarm-ewt.proofpoint.com/EWT/v1/C5qS4YX3!OymHZdAFSneBnUakM_O6Cg_4Bzz61Ui1jyPxXoMRTCoE94kD55qqwRvBZQ2oQ9Q1oRPooC5CgOy7mct_yAnqYjOz0q-jzpRmUlXZhWj01MIxdbgKHA$" style="color: blue; text-decoration: none; display: inline-block; background-color: rgb(208, 216, 220); max-width: none;" rel="nofollow" target="_blank">Report Suspicious</a></span></div>
</div>
<div style="line-height:0;min-height:0px;display:block;font-size:0.01px;"> </div>
</div></div>
</div></div><div id="ydp8bbc286fyiv8933599793yqt34624" class="ydp8bbc286fyiv8933599793yqt9607512131"><div><p style="margin:0cm;font-family:Aptos, sans-serif;font-size:12pt;" class="ydp8bbc286fyiv8933599793MsoNormal">
<span style="font-size:11pt;">Dear Mark and Juergen,</span></p>
<p style="margin:0cm;font-family:Aptos, sans-serif;font-size:12pt;" class="ydp8bbc286fyiv8933599793MsoNormal">
<span style="font-size:11pt;"> </span></p>
<p style="margin:0cm;font-family:Aptos, sans-serif;font-size:12pt;" class="ydp8bbc286fyiv8933599793MsoNormal">
<span style="font-size:11pt;">A while ago when I was teaching an undergraduate morphology & syntax course I had the same concerns about students relying on AI to solve problem sets, so I tested ChatGPT (probably v. 3.5) on some fairly obscure data prior to
setting assignments. The first task was a grammatical sketch based on ~two dozen sentences in Nagamese with English translations. While it did quite well with identifying word classes, tense marking, and other details of morphology, it struggled to make sense
of the postpositional case markers (I had included example sentences of differential marking of P arguments in the data set). Nevertheless, it would have gotten through with a pass. I then tested it on some Dyirbal data with sentences demonstrating the split
alignment system in the case marking/pronominals. This time it did extremely poorly and would have earned an F for its attempt. Naturally I shared the findings with my students
</span><span style="font-family:Color;font-size:11pt;">😊</span></p>
<p style="margin:0cm;font-family:Aptos, sans-serif;font-size:12pt;" class="ydp8bbc286fyiv8933599793MsoNormal">
<span style="font-size:11pt;"> </span></p>
<p style="margin:0cm;font-family:Aptos, sans-serif;font-size:12pt;" class="ydp8bbc286fyiv8933599793MsoNormal">
<span style="font-size:11pt;">This suggests that if there is language data available that a LLM can access to learn, then it is risky to use a data set of that or a typologically similar language for assessment. At the stage of ChatGPT 3.5 it seemed that it
hadn’t had much exposure to head-final languages, and that may explain its inability to identify postpositional case markers. But this may change in the future, and its performance might have already improved vastly.</span></p>
<p style="margin:0cm;font-family:Aptos, sans-serif;font-size:12pt;" class="ydp8bbc286fyiv8933599793MsoNormal">
<span style="font-size:11pt;"> </span></p>
<p style="margin:0cm;font-family:Aptos, sans-serif;font-size:12pt;" class="ydp8bbc286fyiv8933599793MsoNormal">
<span style="font-size:11pt;">Alec</span></p>
<p style="margin:0cm;font-family:Aptos, sans-serif;font-size:12pt;" class="ydp8bbc286fyiv8933599793MsoNormal">
<span style="font-family:Calibri, sans-serif;font-size:11pt;">-- </span></p>
<p style="line-height:9pt;margin:0cm;font-family:Aptos, sans-serif;font-size:12pt;" class="ydp8bbc286fyiv8933599793MsoNormal">
<span style="font-family:Arial, sans-serif;font-size:8pt;color:black;">Assoc. Prof. Alexander R. Coupe, Ph.D.
</span><span style="font-family:Arial, sans-serif;font-size:8pt;color:rgb(251, 0, 7);">|
</span><span style="font-family:Arial, sans-serif;font-size:8pt;color:black;">Associate Chair (Research)
</span><span style="font-family:Arial, sans-serif;font-size:8pt;color:rgb(251, 0, 7);">|</span><span style="font-family:Arial, sans-serif;font-size:8pt;color:black;"> School of Humanities
</span><span style="font-family:Arial, sans-serif;font-size:8pt;color:rgb(251, 0, 7);">|</span><span style="font-family:Arial, sans-serif;font-size:8pt;color:black;"> Nanyang Technological University</span></p>
<p style="margin:0cm;font-family:Aptos, sans-serif;font-size:12pt;" class="ydp8bbc286fyiv8933599793MsoNormal">
<span style="font-family:Arial, sans-serif;font-size:8pt;color:rgb(33, 33, 33);">48 Nanyang Avenue, SHHK-03-84D, Singapore 639818</span></p>
<p style="line-height:9pt;margin:0cm;font-family:Aptos, sans-serif;font-size:12pt;" class="ydp8bbc286fyiv8933599793MsoNormal">
<span style="font-family:Arial, sans-serif;font-size:8pt;color:black;">Tel: +65
</span><span style="font-family:Arial, sans-serif;font-size:8pt;color:rgb(33, 33, 33);">6904 2072</span><span style="font-family:Arial, sans-serif;font-size:8pt;color:black;"> </span><span style="font-family:Arial, sans-serif;font-size:8pt;color:rgb(109, 109, 109);">GMT+8h</span><span style="font-family:Arial, sans-serif;font-size:8pt;color:black;"> </span><span style="font-family:Arial, sans-serif;font-size:8pt;color:rgb(251, 0, 7);">|</span><span style="font-family:Arial, sans-serif;font-size:8pt;color:black;"> Email:
</span><span style="font-family:Arial, sans-serif;font-size:8pt;color:rgb(4, 50, 255);"><u><a shape="rect" href="mailto:arcoupe@ntu.edu.sg" title="mailto:arcoupe@ntu.edu.sg" style="color:rgb(4, 50, 255);margin-top:0px;margin-bottom:0px;" rel="nofollow" target="_blank">arcoupe@ntu.edu.sg</a></u></span></p>
<p style="line-height:9pt;margin:0cm;font-family:Aptos, sans-serif;font-size:12pt;" class="ydp8bbc286fyiv8933599793MsoNormal">
<span style="font-family:Arial, sans-serif;font-size:8pt;color:black;">Academia.edu:
</span><span style="font-family:Arial, sans-serif;font-size:8pt;color:rgb(4, 50, 255);"><u><a shape="rect" href="https://urldefense.com/v3/__https://nanyang.academia.edu/AlexanderCoupe__;!!C5qS4YX3!AYlp9c-DTaVHsnTMRlYSmuMzxsqa-fSpOLYlPhat7VpZOgroj3MWgyzR-PyRCdxbnCdyYVW0qixHRA80tDIIYZckE-6L$" title="https://nanyang.academia.edu/AlexanderCoupe" style="color:rgb(4, 50, 255);margin-top:0px;margin-bottom:0px;" rel="nofollow" target="_blank">https://nanyang.academia.edu/AlexanderCoupe</a></u> </span><span style="font-family:Calibri, sans-serif;font-size:11pt;color:rgb(4, 50, 255);"> </span></p>
<p style="margin:0cm;font-family:Aptos, sans-serif;font-size:12pt;" class="ydp8bbc286fyiv8933599793MsoNormal">
<span style="font-family:Arial, sans-serif;font-size:8pt;color:black;">ORCID ID:
</span><span style="font-family:Arial, sans-serif;font-size:8pt;color:rgb(4, 50, 255);"><u><a shape="rect" href="https://urldefense.com/v3/__https://orcid.org/0000-0003-1979-2370__;!!C5qS4YX3!AYlp9c-DTaVHsnTMRlYSmuMzxsqa-fSpOLYlPhat7VpZOgroj3MWgyzR-PyRCdxbnCdyYVW0qixHRA80tDIIYf61tY-S$" title="https://orcid.org/0000-0003-1979-2370" style="color:rgb(4, 50, 255);margin-top:0px;margin-bottom:0px;" rel="nofollow" target="_blank">https://orcid.org/0000-0003-1979-2370</a></u></span></p>
<p style="line-height:9pt;margin:0cm;font-family:Aptos, sans-serif;font-size:12pt;" class="ydp8bbc286fyiv8933599793MsoNormal">
<span style="font-family:Arial, sans-serif;font-size:8pt;color:black;">Webpage:
</span><span style="font-family:Arial, sans-serif;font-size:8pt;color:rgb(4, 50, 255);"><u><a shape="rect" href="https://urldefense.com/v3/__https://blogs.ntu.edu.sg/arcoupe/__;!!C5qS4YX3!AYlp9c-DTaVHsnTMRlYSmuMzxsqa-fSpOLYlPhat7VpZOgroj3MWgyzR-PyRCdxbnCdyYVW0qixHRA80tDIIYdHSOv5c$" title="https://blogs.ntu.edu.sg/arcoupe/" style="color:rgb(4, 50, 255);margin-top:0px;margin-bottom:0px;" rel="nofollow" target="_blank">https://blogs.ntu.edu.sg/arcoupe/</a></u></span></p>
<p style="margin:0cm;font-family:Aptos, sans-serif;font-size:12pt;" class="ydp8bbc286fyiv8933599793MsoNormal">
<span style="font-size:11pt;"> </span></p>
<p style="margin:0cm;font-family:Aptos, sans-serif;font-size:12pt;" class="ydp8bbc286fyiv8933599793MsoNormal">
<span style="font-size:11pt;"> </span></p>
<p style="margin:0cm;font-family:Aptos, sans-serif;font-size:12pt;" class="ydp8bbc286fyiv8933599793MsoNormal">
<span style="font-size:11pt;"> </span></p>
<div style="padding:3pt 0cm 0cm;border-width:1pt medium medium;border-style:solid none none;border-color:rgb(181, 196, 223) currentcolor currentcolor;">
<p style="margin:0cm;font-family:Aptos, sans-serif;font-size:12pt;" class="ydp8bbc286fyiv8933599793MsoNormal">
<span style="font-family:Calibri, sans-serif;color:black;"><b>From: </b>Lingtyp <lingtyp-bounces@listserv.linguistlist.org> on behalf of "lingtyp@listserv.linguistlist.org" <lingtyp@listserv.linguistlist.org><br clear="none">
<b>Reply to: </b>Juergen Bohnemeyer <jb77@buffalo.edu><br clear="none">
<b>Date: </b>Friday, 7 November 2025 at 1:09 AM<br clear="none">
<b>To: </b>Mark Post <mark.post@sydney.edu.au>, "lingtyp@listserv.linguistlist.org" <lingtyp@listserv.linguistlist.org><br clear="none">
<b>Subject: </b>Re: [Lingtyp] "AI" and linguistics problem sets</span></p>
</div>
<p style="margin:0cm;font-family:Aptos, sans-serif;font-size:12pt;" class="ydp8bbc286fyiv8933599793MsoNormal">
</p>
<table cellpadding="0" style="width: 100%; border-collapse: collapse;" class="ydp8bbc286fyiv8933599793MsoNormalTable"><tbody><tr><td colspan="1" rowspan="1" style="background-color:white;padding:0.75pt;" class="ydp8bbc286fyiv8933599793MsoNormalTable">
<table align="left" cellspacing="0" cellpadding="0" style="width: 100%; border-collapse: collapse;" class="ydp8bbc286fyiv8933599793MsoNormalTable"><tbody><tr><td colspan="1" rowspan="1" style="background-color:rgb(0, 70, 255);padding:5.25pt 1.5pt;width:1%;" class="ydp8bbc286fyiv8933599793MsoNormalTable">
</td><td colspan="1" rowspan="1" style="background-color:rgb(238, 247, 251);padding:5.25pt 3.75pt 5pt 11.25pt;width:100%;" class="ydp8bbc286fyiv8933599793MsoNormalTable">
<p class="ydp8bbc286fyiv8933599793MsoNormalTable"><span style="font-family:Aptos, sans-serif;color:black;"><b>[Alert: Non-NTU Email] Be cautious before clicking any link or attachment.</b></span></p>
</td></tr></tbody></table>
</td></tr></tbody></table>
<p style="margin:0cm;font-family:Aptos, sans-serif;font-size:12pt;" class="ydp8bbc286fyiv8933599793MsoNormal">
<span style="color:black;">Dear Mark — I’m actually surprised to hear that an AI bot is able to adequately solve your problem sets. My assumption, based on my own very limited experience with ChatGPT, has been that LMMs would perform so poorly at linguistic
analysis that the results would dissuade students from trying again in the future. Would it be possible at all to share more details with us?</span></p>
<p style="margin:0cm;font-family:Aptos, sans-serif;font-size:12pt;" class="ydp8bbc286fyiv8933599793MsoNormal">
<span style="color:black;"> </span></p>
<p style="margin:0cm;font-family:Aptos, sans-serif;font-size:12pt;" class="ydp8bbc286fyiv8933599793MsoNormal">
<span style="color:black;">(One recommendation I have, which I however haven’t actually tried out, is to put a watermark of sorts in your assignments, in the form of a factual detail about some lesser-studied language. Even though such engines are of course
quite capable of information retrieval, their very nature seems to predispose them toward predicting the answer rather than to looking it up. With the results being likely straightforwardly false.)</span></p>
<p style="margin:0cm;font-family:Aptos, sans-serif;font-size:12pt;" class="ydp8bbc286fyiv8933599793MsoNormal">
<span style="color:black;"> </span></p>
<p style="margin:0cm;font-family:Aptos, sans-serif;font-size:12pt;" class="ydp8bbc286fyiv8933599793MsoNormal">
<span style="color:black;">Best — Juergen</span></p>
<p style="margin:0cm;font-family:Aptos, sans-serif;font-size:12pt;" class="ydp8bbc286fyiv8933599793MsoNormal">
<span style="color:black;"> </span></p>
<p style="margin:0cm;font-family:Aptos, sans-serif;font-size:12pt;" class="ydp8bbc286fyiv8933599793MsoNormal">
<span style="color:black;"> </span></p>
<div id="ydp8bbc286fyiv8933599793ms-outlook-mobile-signature">
<p style="margin:0cm;"><span style="font-family:Helvetica;font-size:9pt;color:black;">Juergen Bohnemeyer (He/Him)<br clear="none">
Professor, Department of Linguistics<br clear="none">
University at Buffalo <br clear="none">
<br clear="none">
Office: 642 Baldy Hall, UB North Campus<br clear="none">
Mailing address: 609 Baldy Hall, Buffalo, NY 14260 <br clear="none">
Phone: (716) 645 0127 <br clear="none">
Fax: (716) 645 3825<br clear="none">
Email: </span><span style="font-family:Helvetica;font-size:9pt;color:rgb(0, 120, 212);"><u><a shape="rect" href="mailto:jb77@buffalo.edu" title="mailto:jb77@buffalo.edu" style="color:rgb(0, 120, 212);margin-top:0px;margin-bottom:0px;" rel="nofollow" target="_blank">jb77@buffalo.edu</a></u></span><span style="font-family:Helvetica;font-size:9pt;color:black;"><br clear="none">
Web: </span><span style="font-family:Helvetica;font-size:9pt;color:rgb(5, 99, 193);"><u><a shape="rect" href="https://urldefense.com/v3/__http://www.acsu.buffalo.edu/*jb77/__;fg!!C5qS4YX3!AYlp9c-DTaVHsnTMRlYSmuMzxsqa-fSpOLYlPhat7VpZOgroj3MWgyzR-PyRCdxbnCdyYVW0qixHRA80tDIIYQ1qRBWH$" title="http://www.acsu.buffalo.edu/~jb77/" style="color:rgb(5, 99, 193);margin-top:0px;margin-bottom:0px;" rel="nofollow" target="_blank">http://www.acsu.buffalo.edu/~jb77/</a></u></span><span style="font-family:Helvetica;font-size:9pt;color:black;"> <br clear="none">
<br clear="none">
</span><span style="font-family:Calibri, sans-serif;font-size:11pt;color:black;">Office hours Tu/Th 3:30-4:30pm in 642 Baldy or via Zoom (Meeting ID 585 520 2411; Passcode Hoorheh) </span><span style="font-family:Helvetica;font-size:9pt;color:black;"><br clear="none">
<br clear="none">
There’s A Crack In Everything - That’s How The Light Gets In <br clear="none">
(Leonard Cohen) </span></p>
<p style="margin:0cm;"><span style="font-family:Calibri, sans-serif;font-size:11pt;">-- </span></p>
<p style="margin:0cm;"><span style="font-family:Calibri, sans-serif;font-size:11pt;"> </span></p>
</div>
<div id="ydp8bbc286fyiv8933599793mail-editor-reference-message-container">
<div style="padding:3pt 0cm 0cm;border-width:1pt medium medium;border-style:solid none none;border-color:currentcolor;">
<p style="margin:0cm 0cm 12pt;font-family:Aptos, sans-serif;font-size:12pt;" class="ydp8bbc286fyiv8933599793MsoNormal">
<span style="color:black;"><b>From: </b>Lingtyp <lingtyp-bounces@listserv.linguistlist.org> on behalf of Mark Post via Lingtyp <lingtyp@listserv.linguistlist.org><br clear="none">
<b>Date: </b>Tuesday, November 4, 2025 at 18:27<br clear="none">
<b>To: </b>typology list <lingtyp@listserv.linguistlist.org><br clear="none">
<b>Subject: </b>[Lingtyp] "AI" and linguistics problem sets</span></p>
</div>
<p style="margin:0cm;font-family:Aptos, sans-serif;font-size:12pt;" class="ydp8bbc286fyiv8933599793MsoNormal">
<span style="color:black;">Dear Listmembers,</span></p>
<p style="margin:0cm;font-family:Aptos, sans-serif;font-size:12pt;" class="ydp8bbc286fyiv8933599793MsoNormal">
<span style="color:black;"> </span></p>
<p style="margin:0cm;font-family:Aptos, sans-serif;font-size:12pt;" class="ydp8bbc286fyiv8933599793MsoNormal">
<span style="color:black;">I trust that most lingtyp subscribers will have engaged with “problem sets” of the type found in Language Files, Describing Morphosyntax, and my personal favourite oldie-but-goodie the Source Book for Linguistics. Since the advent
of ChatGPT, I’ve been migrating away from these (and even edited/obscured versions of them) for assessments, and relying more and more on private/unpublished data sets, mostly from languages with lots of complex morphology and less familiar category types,
that LLMs seemed to have a much harder time with. This was not an ideal situation for many reasons, not least of which being that these were not the only types of languages students should get practice working with. But the problem really came to a head this
year, when I found that perhaps most off-the-shelf LLMs were now able to solve almost all of my go-to problem sets to an at least reasonable degree, even after I obscured much of the data. </span></p>
<p style="margin:0cm;font-family:Aptos, sans-serif;font-size:12pt;" class="ydp8bbc286fyiv8933599793MsoNormal">
<span style="color:black;"> </span></p>
<p style="margin:0cm;font-family:Aptos, sans-serif;font-size:12pt;" class="ydp8bbc286fyiv8933599793MsoNormal">
<span style="color:black;">Leaving aside issues around how LLMs work, what role(s) they can or should (not) play in linguistic research, etc., I’d like to ask if any listmembers would be willing to share their experiences, advice, etc., specifically in the
area of student assessment in the teaching of linguistic data analysis, and in particular morphosyntax, in the unfolding AI-saturated environment. Is the “problem set” method of teaching distributional analysis irretrievably lost? Can it still be employed,
and if so how? Are there different/better ways of teaching more or less the same skills? </span></p>
<p style="margin:0cm;font-family:Aptos, sans-serif;font-size:12pt;" class="ydp8bbc286fyiv8933599793MsoNormal">
<span style="color:black;"> </span></p>
<p style="margin:0cm;font-family:Aptos, sans-serif;font-size:12pt;" class="ydp8bbc286fyiv8933599793MsoNormal">
<span style="color:black;">Note that I would really like to avoid doomsdayisms if possible here (“the skills traditionally taught to linguists have already been made obsolete by AIs, such that there’s no point in teaching them anymore” - an argument with which
I am all-too-familiar), and focus, if possible, on <i>how</i> it is possible to assess/evaluate students’ performance
<i>under the assumption</i> that there is at least some value in teaching at least some human beings how to do a distributional analysis “by hand” - such that they are actually able, for example, to evaluate a machine’s performance in analysing a new/unfamiliar
data set, and under the further assumption that assessment/evaluation of student performance in at least many institutions will continue to follow existing models.</span></p>
<p style="margin:0cm;font-family:Aptos, sans-serif;font-size:12pt;" class="ydp8bbc286fyiv8933599793MsoNormal">
<span style="color:black;"> </span></p>
<p style="margin:0cm;font-family:Aptos, sans-serif;font-size:12pt;" class="ydp8bbc286fyiv8933599793MsoNormal">
<span style="color:black;">Many thanks in advance!</span></p>
<p style="margin:0cm;font-family:Aptos, sans-serif;font-size:12pt;" class="ydp8bbc286fyiv8933599793MsoNormal">
<span style="color:black;">Mark</span></p>
<p style="margin:0cm;font-family:Aptos, sans-serif;font-size:12pt;" class="ydp8bbc286fyiv8933599793MsoNormal">
<span style="color:black;"> </span></p>
</div>
<hr>
<p style="font-family:Arial;font-size:10pt;color:rgb(128, 128, 128);" class="ydp8bbc286fyiv8933599793ms-outlook-mobile-reference-message ydp8bbc286fyiv8933599793skipProofing">
CONFIDENTIALITY: This email is intended solely for the person(s) named and may be confidential and/or privileged. If you are not the intended recipient, please delete it, notify us and do not copy, use, or disclose its contents.<br clear="none">
Towards a sustainable earth: Print only when necessary. Thank you.</p>
</div></div></div><div class="ydp8bbc286fyqt9607512131" id="ydp8bbc286fyqt57020">_______________________________________________<br clear="none">Lingtyp mailing list<br clear="none"><a shape="rect" href="mailto:Lingtyp@listserv.linguistlist.org" rel="nofollow" target="_blank">Lingtyp@listserv.linguistlist.org</a><br clear="none"><a shape="rect" href="https://listserv.linguistlist.org/cgi-bin/mailman/listinfo/lingtyp" rel="nofollow" target="_blank">https://listserv.linguistlist.org/cgi-bin/mailman/listinfo/lingtyp</a><br clear="none"></div></div>
</div>
</div><div id="DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2"><br><table style="border-top: 1px solid #D3D4DE;"><tr><td style="width: 55px; padding-top: 13px;"><a href="https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail" target="_blank"><img src="https://s-install.avcdn.net/ipm/preview/icons/icon-envelope-tick-round-orange-animated-no-repeat-v1.gif" alt="" width="46" height="29" style="width: 46px; height: 29px;"></a></td><td style="width: 470px; padding-top: 12px; color: #41424e; font-size: 13px; font-family: Arial, Helvetica, sans-serif; line-height: 18px;">Virus-free.<a href="https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail" target="_blank" style="color: #4453ea;">www.avast.com</a></td></tr></table><a href="#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2" width="1" height="1"></a></div></body></html>