<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"MS Gothic";
panose-1:2 11 6 9 7 2 5 8 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Aptos;
panose-1:2 11 0 4 2 2 2 2 2 4;}
@font-face
{font-family:"맑은 고딕";
panose-1:2 11 5 3 2 0 0 2 0 4;}
@font-face
{font-family:"Apple Color Emoji";
panose-1:0 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"\@MS Gothic";
panose-1:2 11 6 9 7 2 5 8 2 4;}
@font-face
{font-family:"\@맑은 고딕";}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:12.0pt;
font-family:"Aptos",sans-serif;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Arial",sans-serif;
color:windowtext;
font-weight:normal;
font-style:normal;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
mso-ligatures:none;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="ko-JP" link="#467886" vlink="#96607D" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal" style="word-break:break-hangul"><span lang="EN-US" style="font-family:"Arial",sans-serif;mso-fareast-language:#0C00">Dear Mark,<o:p></o:p></span></p>
<p class="MsoNormal" style="word-break:break-hangul"><span lang="EN-US" style="font-family:"Arial",sans-serif;mso-fareast-language:#0C00"><o:p> </o:p></span></p>
<p class="MsoNormal" style="word-break:break-hangul"><span lang="EN-US" style="font-family:"Arial",sans-serif;mso-fareast-language:#0C00">I simply don’t give any take-home assignments (in fact I don’t think I ever did). All evaluations in my courses are either
closed-book exams or chalk-talks (presentations without slides). Not only is this an easy solution to the LLM issue, but also it lessens the burdens of the university students, who usually already have a lot going in their lives.
<o:p></o:p></span></p>
<p class="MsoNormal" style="word-break:break-hangul"><span lang="EN-US" style="font-family:"Arial",sans-serif;mso-fareast-language:#0C00"><o:p> </o:p></span></p>
<p class="MsoNormal" style="word-break:break-hangul"><span lang="EN-US" style="font-family:"Arial",sans-serif;mso-fareast-language:#0C00">My rule of thumb is that using LLM for whatever purpose is okay as long as you know what you’re doing and you can be fully
responsible for the end result. So in general I don’t see the point of employing LLM for courses whose goal is to precisely help the students understand what they’re doing.<o:p></o:p></span></p>
<p class="MsoNormal" style="word-break:break-hangul"><span lang="EN-US" style="font-family:"Arial",sans-serif;mso-fareast-language:#0C00"><o:p> </o:p></span></p>
<p class="MsoNormal" style="word-break:break-hangul"><span lang="EN-US" style="font-family:"Arial",sans-serif;mso-fareast-language:#0C00">From Otaru, Japan,<o:p></o:p></span></p>
<p class="MsoNormal" style="word-break:break-hangul"><span lang="EN-US" style="font-family:"Arial",sans-serif;mso-fareast-language:#0C00">Ian<o:p></o:p></span></p>
<p class="MsoNormal" style="word-break:break-hangul"><span lang="EN-US" style="font-family:"Arial",sans-serif;mso-fareast-language:#0C00"><o:p> </o:p></span></p>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:black">- - - - - - - - - - - - - - - - - - - - - -</span><span style="font-family:"Arial",sans-serif;color:black">
<span lang="EN-US">-</span></span><span style="font-family:"Arial",sans-serif;color:black"><br>
</span><span lang="KO" style="font-family:"MS Gothic";color:black">朱 易安</span><span style="font-family:"Arial",sans-serif;color:black"> <br>
JOO, IAN <br>
</span><span lang="KO" style="font-family:"MS Gothic";color:black">准教授</span><span style="font-family:"Arial",sans-serif;color:black"> <br>
Associate Professor <br>
</span><span lang="KO" style="font-family:"MS Gothic";color:black">小樽商科大学</span><span style="font-family:"Arial",sans-serif;color:black"> <br>
Otaru University of Commerce<o:p></o:p></span></p>
<p class="MsoNormal" style="word-break:break-hangul"><span lang="EN-US" style="font-family:"Arial",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
</div>
<p class="MsoNormal" style="word-break:break-hangul"><span lang="EN-US" style="font-family:"Apple Color Emoji";color:black;mso-fareast-language:#0C00">🌐</span><span lang="EN-US" style="font-family:"Arial",sans-serif;color:black;mso-fareast-language:#0C00"> <a href="http://ianjoo.github.io/"><span style="color:#714FBC">ianjoo.github.io</span></a><br>
- - - - - - - - - - - - - - - - - - - - - - -</span><span lang="EN-US" style="font-family:"Arial",sans-serif;mso-fareast-language:#0C00"><o:p></o:p></span></p>
<p class="MsoNormal" style="word-break:break-hangul"><span lang="EN-US" style="font-family:"Arial",sans-serif;mso-fareast-language:#0C00"><o:p> </o:p></span></p>
<div id="mail-editor-reference-message-container">
<div>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" style="mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:12.0pt;margin-left:39.95pt;mso-margin-top-alt:0cm;mso-para-margin-right:0cm;mso-para-margin-bottom:12.0pt;mso-para-margin-left:3.33gd">
<b><span lang="KO" style="font-family:"맑은 고딕",sans-serif;color:black">보낸</span></b><b><span lang="KO" style="color:black">
</span></b><b><span lang="KO" style="font-family:"맑은 고딕",sans-serif;color:black">사람</span></b><b><span style="color:black">:
</span></b><span style="color:black">Lingtyp <lingtyp-bounces@listserv.linguistlist.org></span><span lang="KO" style="font-family:"맑은 고딕",sans-serif;color:black">이</span><span style="color:black">(</span><span lang="KO" style="font-family:"맑은 고딕",sans-serif;color:black">가</span><span style="color:black">)
</span><span lang="KO" style="font-family:"맑은 고딕",sans-serif;color:black">다음</span><span lang="KO" style="color:black">
</span><span lang="KO" style="font-family:"맑은 고딕",sans-serif;color:black">사람</span><span lang="KO" style="color:black">
</span><span lang="KO" style="font-family:"맑은 고딕",sans-serif;color:black">대신</span><span lang="KO" style="color:black">
</span><span lang="KO" style="font-family:"맑은 고딕",sans-serif;color:black">보냄</span><span style="color:black">: Mark Post via Lingtyp <lingtyp@listserv.linguistlist.org><br>
</span><b><span lang="KO" style="font-family:"맑은 고딕",sans-serif;color:black">날짜</span></b><b><span style="color:black">:
</span></b><span lang="KO" style="font-family:"맑은 고딕",sans-serif;color:black">수요일</span><span style="color:black">, 2025</span><span lang="KO" style="font-family:"맑은 고딕",sans-serif;color:black">년</span><span style="color:black"> 11</span><span lang="KO" style="font-family:"맑은 고딕",sans-serif;color:black">월</span><span style="color:black">
5</span><span lang="KO" style="font-family:"맑은 고딕",sans-serif;color:black">일</span><span style="color:black"> 08:29<br>
</span><b><span lang="KO" style="font-family:"맑은 고딕",sans-serif;color:black">받는</span></b><b><span lang="KO" style="color:black">
</span></b><b><span lang="KO" style="font-family:"맑은 고딕",sans-serif;color:black">사람</span></b><b><span style="color:black">:
</span></b><span style="color:black">typology list <lingtyp@listserv.linguistlist.org><br>
</span><b><span lang="KO" style="font-family:"맑은 고딕",sans-serif;color:black">주제</span></b><b><span style="color:black">:
</span></b><span style="color:black">[Lingtyp] "AI" and linguistics problem sets<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:0cm;margin-right:12.0pt;margin-bottom:0cm;margin-left:12.0pt;margin-bottom:.0001pt;mso-margin-top-alt:0cm;mso-para-margin-right:1.0gd;mso-para-margin-bottom:0cm;mso-para-margin-left:1.0gd;mso-para-margin-bottom:.0001pt">
<span style="color:black">Dear Listmembers,<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:0cm;margin-right:12.0pt;margin-bottom:0cm;margin-left:12.0pt;margin-bottom:.0001pt;mso-margin-top-alt:0cm;mso-para-margin-right:1.0gd;mso-para-margin-bottom:0cm;mso-para-margin-left:1.0gd;mso-para-margin-bottom:.0001pt">
<span style="color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:0cm;margin-right:12.0pt;margin-bottom:0cm;margin-left:12.0pt;margin-bottom:.0001pt;mso-margin-top-alt:0cm;mso-para-margin-right:1.0gd;mso-para-margin-bottom:0cm;mso-para-margin-left:1.0gd;mso-para-margin-bottom:.0001pt">
<span style="color:black">I trust that most lingtyp subscribers will have engaged with “problem sets” of the type found in Language Files, Describing Morphosyntax, and my personal favourite oldie-but-goodie the Source Book for Linguistics. Since the advent
of ChatGPT, I’ve been migrating away from these (and even edited/obscured versions of them) for assessments, and relying more and more on private/unpublished data sets, mostly from languages with lots of complex morphology and less familiar category types,
that LLMs seemed to have a much harder time with. This was not an ideal situation for many reasons, not least of which being that these were not the only types of languages students should get practice working with. But the problem really came to a head this
year, when I found that perhaps most off-the-shelf LLMs were now able to solve almost all of my go-to problem sets to an at least reasonable degree, even after I obscured much of the data. <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:0cm;margin-right:12.0pt;margin-bottom:0cm;margin-left:12.0pt;margin-bottom:.0001pt;mso-margin-top-alt:0cm;mso-para-margin-right:1.0gd;mso-para-margin-bottom:0cm;mso-para-margin-left:1.0gd;mso-para-margin-bottom:.0001pt">
<span style="color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:0cm;margin-right:12.0pt;margin-bottom:0cm;margin-left:12.0pt;margin-bottom:.0001pt;mso-margin-top-alt:0cm;mso-para-margin-right:1.0gd;mso-para-margin-bottom:0cm;mso-para-margin-left:1.0gd;mso-para-margin-bottom:.0001pt">
<span style="color:black">Leaving aside issues around how LLMs work, what role(s) they can or should (not) play in linguistic research, etc., I’d like to ask if any listmembers would be willing to share their experiences, advice, etc., specifically in the area
of student assessment in the teaching of linguistic data analysis, and in particular morphosyntax, in the unfolding AI-saturated environment. Is the “problem set” method of teaching distributional analysis irretrievably lost? Can it still be employed, and
if so how? Are there different/better ways of teaching more or less the same skills? <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:0cm;margin-right:12.0pt;margin-bottom:0cm;margin-left:12.0pt;margin-bottom:.0001pt;mso-margin-top-alt:0cm;mso-para-margin-right:1.0gd;mso-para-margin-bottom:0cm;mso-para-margin-left:1.0gd;mso-para-margin-bottom:.0001pt">
<span style="color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:0cm;margin-right:12.0pt;margin-bottom:0cm;margin-left:12.0pt;margin-bottom:.0001pt;mso-margin-top-alt:0cm;mso-para-margin-right:1.0gd;mso-para-margin-bottom:0cm;mso-para-margin-left:1.0gd;mso-para-margin-bottom:.0001pt">
<span style="color:black">Note that I would really like to avoid doomsdayisms if possible here (“the skills traditionally taught to linguists have already been made obsolete by AIs, such that there’s no point in teaching them anymore” - an argument with which
I am all-too-familiar), and focus, if possible, on <i>how</i> it is possible to assess/evaluate students’ performance
<i>under the assumption</i> that there is at least some value in teaching at least some human beings how to do a distributional analysis “by hand” - such that they are actually able, for example, to evaluate a machine’s performance in analysing a new/unfamiliar
data set, and under the further assumption that assessment/evaluation of student performance in at least many institutions will continue to follow existing models.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:0cm;margin-right:12.0pt;margin-bottom:0cm;margin-left:12.0pt;margin-bottom:.0001pt;mso-margin-top-alt:0cm;mso-para-margin-right:1.0gd;mso-para-margin-bottom:0cm;mso-para-margin-left:1.0gd;mso-para-margin-bottom:.0001pt">
<span style="color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:0cm;margin-right:12.0pt;margin-bottom:0cm;margin-left:12.0pt;margin-bottom:.0001pt;mso-margin-top-alt:0cm;mso-para-margin-right:1.0gd;mso-para-margin-bottom:0cm;mso-para-margin-left:1.0gd;mso-para-margin-bottom:.0001pt">
<span style="color:black">Many thanks in advance!<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:39.95pt;mso-para-margin-left:3.33gd"><span style="color:black">Mark<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:0cm;margin-right:12.0pt;margin-bottom:0cm;margin-left:12.0pt;margin-bottom:.0001pt;mso-margin-top-alt:0cm;mso-para-margin-right:1.0gd;mso-para-margin-bottom:0cm;mso-para-margin-left:1.0gd;mso-para-margin-bottom:.0001pt">
<span style="color:black"><o:p> </o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</body>
</html>