<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body>
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Dear Mark — I’m actually surprised to hear that an AI bot is able to adequately solve your problem sets. My assumption, based on my own very limited experience with ChatGPT, has been that LMMs would perform so poorly at linguistic analysis that the results
would dissuade students from trying again in the future. Would it be possible at all to share more details with us?</div>
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
(One recommendation I have, which I however haven’t actually tried out, is to put a watermark of sorts in your assignments, in the form of a factual detail about some lesser-studied language. Even though such engines are of course quite capable of information
retrieval, their very nature seems to predispose them toward predicting the answer rather than to looking it up. With the results being likely straightforwardly false.)</div>
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Best — Juergen</div>
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="ms-outlook-mobile-signature">
<p style="margin: 0in; font-family: Calibri, sans-serif; font-size: 11pt;"><span style="font-family: Helvetica; font-size: 9pt; color: black;">Juergen Bohnemeyer (He/Him)<br>
Professor, Department of Linguistics<br>
University at Buffalo <br>
<br>
Office: 642 Baldy Hall, UB North Campus<br>
Mailing address: 609 Baldy Hall, Buffalo, NY 14260 <br>
Phone: (716) 645 0127 <br>
Fax: (716) 645 3825<br>
Email: </span><span style="font-family: Helvetica; font-size: 9pt; color: rgb(0, 120, 212);"><u><a href="mailto:jb77@buffalo.edu" title="mailto:jb77@buffalo.edu" data-outlook-id="2e48ce68-9d23-43a8-bec9-d1c8469319f9" style="color: rgb(0, 120, 212); margin-top: 0px; margin-bottom: 0px;">jb77@buffalo.edu</a></u></span><span style="font-family: Helvetica; font-size: 9pt; color: black;"><br>
Web: </span><span style="font-family: Helvetica; font-size: 9pt; color: rgb(5, 99, 193);"><u><a href="http://www.acsu.buffalo.edu/~jb77/" title="http://www.acsu.buffalo.edu/~jb77/" data-outlook-id="50123fc3-4659-4423-a91a-3c2c39a4b944" style="color: rgb(5, 99, 193); margin-top: 0px; margin-bottom: 0px;">http://www.acsu.buffalo.edu/~jb77/</a></u></span><span style="font-family: Helvetica; font-size: 9pt; color: black;"> <br>
<br>
</span><span style="color: black;">Office hours Tu/Th 3:30-4:30pm in 642 Baldy or via Zoom (Meeting ID 585 520 2411; Passcode Hoorheh) </span><span style="font-family: Helvetica; font-size: 9pt; color: black;"><br>
<br>
There’s A Crack In Everything - That’s How The Light Gets In <br>
(Leonard Cohen) </span></p>
<p style="margin: 0in; font-family: Calibri, sans-serif; font-size: 11pt;">-- </p>
<p style="margin: 0in; font-family: Calibri, sans-serif; font-size: 11pt;"> </p>
</div>
<div id="mail-editor-reference-message-container">
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing"></div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="text-align: left; padding: 3pt 0in 0in; border-width: 1pt medium medium; border-style: solid none none; border-color: rgb(181, 196, 223) currentcolor currentcolor; font-family: Aptos; font-size: 12pt; color: black;">
<b>From: </b>Lingtyp <lingtyp-bounces@listserv.linguistlist.org> on behalf of Mark Post via Lingtyp <lingtyp@listserv.linguistlist.org><br>
<b>Date: </b>Tuesday, November 4, 2025 at 18:27<br>
<b>To: </b>typology list <lingtyp@listserv.linguistlist.org><br>
<b>Subject: </b>[Lingtyp] "AI" and linguistics problem sets<br>
<br>
</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Dear Listmembers,</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I trust that most lingtyp subscribers will have engaged with “problem sets” of the type found in Language Files, Describing Morphosyntax, and my personal favourite oldie-but-goodie the Source Book for Linguistics. Since the advent of ChatGPT, I’ve been migrating
away from these (and even edited/obscured versions of them) for assessments, and relying more and more on private/unpublished data sets, mostly from languages with lots of complex morphology and less familiar category types, that LLMs seemed to have a much
harder time with. This was not an ideal situation for many reasons, not least of which being that these were not the only types of languages students should get practice working with. But the problem really came to a head this year, when I found that perhaps
most off-the-shelf LLMs were now able to solve almost all of my go-to problem sets to an at least reasonable degree, even after I obscured much of the data. </div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Leaving aside issues around how LLMs work, what role(s) they can or should (not) play in linguistic research, etc., I’d like to ask if any listmembers would be willing to share their experiences, advice, etc., specifically in the area of student assessment
in the teaching of linguistic data analysis, and in particular morphosyntax, in the unfolding AI-saturated environment. Is the “problem set” method of teaching distributional analysis irretrievably lost? Can it still be employed, and if so how? Are there different/better
ways of teaching more or less the same skills? </div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Note that I would really like to avoid doomsdayisms if possible here (“the skills traditionally taught to linguists have already been made obsolete by AIs, such that there’s no point in teaching them anymore” - an argument with which I am all-too-familiar),
and focus, if possible, on <i>how</i> it is possible to assess/evaluate students’ performance
<i>under the assumption</i> that there is at least some value in teaching at least some human beings how to do a distributional analysis “by hand” - such that they are actually able, for example, to evaluate a machine’s performance in analysing a new/unfamiliar
data set, and under the further assumption that assessment/evaluation of student performance in at least many institutions will continue to follow existing models.</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Many thanks in advance!</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Mark</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
</div>
</body>
</html>