<html><head><meta http-equiv="content-type" content="text/html; charset=UTF-8"/></head><body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">Hi Mark,<div>LLMs work by induction (statistical probability), the same as us in terms of identifying patterns that repeat in the data. But the problem we have seen with technology generally is that if you rely on the technology, you lose the ability to do it yourself. For example, in the past, each time I moved to a different city I would develop a map of the city in my head from studying maps and driving by reference to them. But after using GPS, I find I never learn the cities, and just have to continually rely on GPS. In Hong Kong (and myself as well), many people who use computers for writing Chinese characters all the time lose the ability to remember how to write the characters by hand, and if they can write it, they can no longer write it in a nice style (important in Chinese culture). The same problem came up in the teaching of arithmetic once electronic calculators were small enough for everyone to use. It might be useful to see how Math teachers dealt with that.</div><div>But where we differ from machines is that in doing linguistics we can also use abductive inference to try to explain why the forms are the way they are (something you have argued we should be doing). The LLMs are not reliable for that. So in terms of teaching, assessment should be done in the classroom with no open phones or computers (these should actually never be on in the classroom), and there should be not only an inductive part of the assessment, but also an abductive part of the assessment. Harder to write, but certainly more interesting and challenging for the students.</div><div><br/></div><div>Hope this helps.</div><div><br/></div><div>Best always,</div><div>Randy<br id="lineBreakAtBeginningOfMessage"/><div><br/><blockquote type="cite"><div>On 5 Nov 2025, at 7:23 AM, Mark Post via Lingtyp <lingtyp@listserv.linguistlist.org> wrote:</div><br class="Apple-interchange-newline"/><div>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
<div>
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt;">
Dear Listmembers,</div>
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt;">
<br/>
</div>
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt;">
I trust that most lingtyp subscribers will have engaged with “problem sets” of the type found in Language Files, Describing Morphosyntax, and my personal favourite oldie-but-goodie the Source Book for Linguistics. Since the advent of ChatGPT, I’ve been migrating
away from these (and even edited/obscured versions of them) for assessments, and relying more and more on private/unpublished data sets, mostly from languages with lots of complex morphology and less familiar category types, that LLMs seemed to have a much
harder time with. This was not an ideal situation for many reasons, not least of which being that these were not the only types of languages students should get practice working with. But the problem really came to a head this year, when I found that perhaps
most off-the-shelf LLMs were now able to solve almost all of my go-to problem sets to an at least reasonable degree, even after I obscured much of the data. </div>
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt;">
<br/>
</div>
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt;">
Leaving aside issues around how LLMs work, what role(s) they can or should (not) play in linguistic research, etc., I’d like to ask if any listmembers would be willing to share their experiences, advice, etc., specifically in the area of student assessment
in the teaching of linguistic data analysis, and in particular morphosyntax, in the unfolding AI-saturated environment. Is the “problem set” method of teaching distributional analysis irretrievably lost? Can it still be employed, and if so how? Are there different/better
ways of teaching more or less the same skills? </div>
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt;">
<br/>
</div>
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt;">
Note that I would really like to avoid doomsdayisms if possible here (“the skills traditionally taught to linguists have already been made obsolete by AIs, such that there’s no point in teaching them anymore” - an argument with which I am all-too-familiar),
and focus, if possible, on <i>how</i> it is possible to assess/evaluate students’ performance
<i>under the assumption</i> that there is at least some value in teaching at least some human beings how to do a distributional analysis “by hand” - such that they are actually able, for example, to evaluate a machine’s performance in analysing a new/unfamiliar
data set, and under the further assumption that assessment/evaluation of student performance in at least many institutions will continue to follow existing models.</div>
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt;">
<br/>
</div>
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt;">
Many thanks in advance!</div>
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt;">
Mark</div>
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt;">
<br/>
</div>
</div>
</div></blockquote></div><br/></div></body></html>