<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Ari,<div class=""><br class=""></div><div class="">Given all your subcodes, it looks like you have 1122 ((4x5x14x4) +2) possible codes. So that would be the N for Cohen's kappa.<div class=""><br class=""></div><div class="">In order to calculate the reliability of subcodes such as those for #3 by themselves, you might need to edit the coding tier in your files to remove everything but those codes. RELY wasn't designed to do this, but if you edit the data to just focus on those choices, it should be able to do it okay. You would make a copy of your files and them use a GREP search and replace string to get rid of the first three and last elements in your string. Having them separated by colons makes this not too hard. Of course, then your N for Cohen's kappa would only be 14.<br class=""><div class=""><br class=""></div><div class="">-- Brian MacWhinney</div><div class=""><div class=""><br class=""><div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">Teresa Heinz Professor of </div><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">Cognitive Psychology,</div><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">Computational Linguistics, </div><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">and Modern Languages, CMU</div></div></div></div></div></div></div></div></div></div>
</div>
<div><br class=""><blockquote type="cite" class=""><div class="">On Apr 3, 2020, at 4:34 PM, Ariadne Nelson <<a href="mailto:nelsonat@bc.edu" class="">nelsonat@bc.edu</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class="">Good evening,<br class=""><br class="">I have a question about calculating the inter-rater reliability (Cohen's kappa coefficient) of codes in CLAN using the RELY command. <br class=""><br class="">I have been using the command: rely +dN +c @ +t%num -t*, with N equaling the number of categories. Does this look correct?<br class=""><br class="">I am unsure how to calculate the number of categories for my coding scheme. This table breaks down my coding scheme. There are three main codes ($OT, $RE, $NT). If $NT is selected, there are four subcodes, with differing quantities of options to choose from. A full $NT might look like: $NT:PAR:ALL:EME:P.<table cellpadding="0" class="gmail-ajC" style="border-spacing:0px;line-height:20px;font-family:Roboto,RobotoDraft,Helvetica,Arial,sans-serif;font-size:14px"><tbody class=""><tr class="gmail-ajv"><td colspan="2" tabindex="0" class="gmail-gL" style="vertical-align:top;width:auto;padding:2px 0px"><span class="gmail-gI" style="vertical-align:top"><br class=""><table class="gmail-MsoTableGrid" border="1" cellspacing="0" cellpadding="0" style="border-collapse:collapse;border:none">
<tbody class=""><tr class="">
<td width="120" style="width:89.75pt;border:1pt solid windowtext;padding:0in 5.4pt" class=""><div style="margin: 0in 0in 0.0001pt; text-align: center; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><b class="">Main
code</b></div>
</td>
<td width="130" style="width:97.25pt;border-top:1pt solid windowtext;border-right:1pt solid windowtext;border-bottom:1pt solid windowtext;border-left:none;padding:0in 5.4pt" class=""><div style="margin: 0in 0in 0.0001pt; text-align: center; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><b class="">Sub-Code
#1</b></div>
</td>
<td width="125" style="width:93.5pt;border-top:1pt solid windowtext;border-right:1pt solid windowtext;border-bottom:1pt solid windowtext;border-left:none;padding:0in 5.4pt" class=""><div style="margin: 0in 0in 0.0001pt; text-align: center; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><b class="">Sub-Code
#2</b></div>
</td>
<td width="125" style="width:93.5pt;border-top:1pt solid windowtext;border-right:1pt solid windowtext;border-bottom:1pt solid windowtext;border-left:none;padding:0in 5.4pt" class=""><div style="margin: 0in 0in 0.0001pt; text-align: center; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><b class="">Sub-Code
#3</b></div>
</td>
<td width="125" style="width:93.5pt;border-top:1pt solid windowtext;border-right:1pt solid windowtext;border-bottom:1pt solid windowtext;border-left:none;padding:0in 5.4pt" class=""><div style="margin: 0in 0in 0.0001pt; text-align: center; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><b class="">Sub-Code
#4</b></div>
</td>
</tr>
<tr class="">
<td width="120" valign="top" style="width:89.75pt;border-right:1pt solid windowtext;border-bottom:1pt solid windowtext;border-left:1pt solid windowtext;border-top:none;padding:0in 5.4pt" class=""><div style="margin: 0in 0in 0.0001pt; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;" class="">$OT</div>
</td>
<td width="130" valign="top" style="width:97.25pt;border-top:none;border-left:none;border-bottom:1pt solid windowtext;border-right:1pt solid windowtext;padding:0in 5.4pt" class=""><p class="MsoNormal" style="margin:0in 0in 0.0001pt;line-height:normal;font-size:11pt;font-family:Calibri,sans-serif"> </p>
</td>
<td width="125" valign="top" style="width:93.5pt;border-top:none;border-left:none;border-bottom:1pt solid windowtext;border-right:1pt solid windowtext;padding:0in 5.4pt" class=""><p class="MsoNormal" style="margin:0in 0in 0.0001pt;line-height:normal;font-size:11pt;font-family:Calibri,sans-serif"> </p>
</td>
<td width="125" valign="top" style="width:93.5pt;border-top:none;border-left:none;border-bottom:1pt solid windowtext;border-right:1pt solid windowtext;padding:0in 5.4pt" class=""><p class="MsoNormal" style="margin:0in 0in 0.0001pt;line-height:normal;font-size:11pt;font-family:Calibri,sans-serif"> </p>
</td>
<td width="125" valign="top" style="width:93.5pt;border-top:none;border-left:none;border-bottom:1pt solid windowtext;border-right:1pt solid windowtext;padding:0in 5.4pt" class=""><p class="MsoNormal" style="margin:0in 0in 0.0001pt;line-height:normal;font-size:11pt;font-family:Calibri,sans-serif"> </p>
</td>
</tr>
<tr class="">
<td width="120" valign="top" style="width:89.75pt;border-right:1pt solid windowtext;border-bottom:1pt solid windowtext;border-left:1pt solid windowtext;border-top:none;padding:0in 5.4pt" class=""><div style="margin: 0in 0in 0.0001pt; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;" class="">$RE</div>
</td>
<td width="130" valign="top" style="width:97.25pt;border-top:none;border-left:none;border-bottom:1pt solid windowtext;border-right:1pt solid windowtext;padding:0in 5.4pt" class=""><p class="MsoNormal" style="margin:0in 0in 0.0001pt;line-height:normal;font-size:11pt;font-family:Calibri,sans-serif"> </p>
</td>
<td width="125" valign="top" style="width:93.5pt;border-top:none;border-left:none;border-bottom:1pt solid windowtext;border-right:1pt solid windowtext;padding:0in 5.4pt" class=""><p class="MsoNormal" style="margin:0in 0in 0.0001pt;line-height:normal;font-size:11pt;font-family:Calibri,sans-serif"> </p>
</td>
<td width="125" valign="top" style="width:93.5pt;border-top:none;border-left:none;border-bottom:1pt solid windowtext;border-right:1pt solid windowtext;padding:0in 5.4pt" class=""><p class="MsoNormal" style="margin:0in 0in 0.0001pt;line-height:normal;font-size:11pt;font-family:Calibri,sans-serif"> </p>
</td>
<td width="125" valign="top" style="width:93.5pt;border-top:none;border-left:none;border-bottom:1pt solid windowtext;border-right:1pt solid windowtext;padding:0in 5.4pt" class=""><p class="MsoNormal" style="margin:0in 0in 0.0001pt;line-height:normal;font-size:11pt;font-family:Calibri,sans-serif"> </p>
</td>
</tr>
<tr class="">
<td width="120" valign="top" style="width:89.75pt;border-right:1pt solid windowtext;border-bottom:1pt solid windowtext;border-left:1pt solid windowtext;border-top:none;padding:0in 5.4pt" class=""><div style="margin: 0in 0in 0.0001pt; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;" class="">$NT</div>
</td>
<td width="130" valign="top" style="width:97.25pt;border-top:none;border-left:none;border-bottom:1pt solid windowtext;border-right:1pt solid windowtext;padding:0in 5.4pt" class=""><div style="margin: 0in 0in 0.0001pt; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;" class="">4 options: </div><div style="margin: 0in 0in 0.0001pt; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Target child (TCH), Parent (PAR), </div><div style="margin: 0in 0in 0.0001pt; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Other Child (OCH), </div><div style="margin: 0in 0in 0.0001pt; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Other Adult (OAD)</div>
</td>
<td width="125" valign="top" style="width:93.5pt;border-top:none;border-left:none;border-bottom:1pt solid windowtext;border-right:1pt solid windowtext;padding:0in 5.4pt" class=""><div style="margin: 0in 0in 0.0001pt; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;" class="">5 options </div>
</td>
<td width="125" valign="top" style="width:93.5pt;border-top:none;border-left:none;border-bottom:1pt solid windowtext;border-right:1pt solid windowtext;padding:0in 5.4pt" class=""><div style="margin: 0in 0in 0.0001pt; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;" class="">14 options</div>
</td>
<td width="125" valign="top" style="width:93.5pt;border-top:none;border-left:none;border-bottom:1pt solid windowtext;border-right:1pt solid windowtext;padding:0in 5.4pt" class=""><div style="margin: 0in 0in 0.0001pt; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;" class="">If adult selected for sub-code #1 (PAR or OAD), 4 options for
sub-code #4. </div><p class="MsoNormal" style="margin:0in 0in 0.0001pt;line-height:normal;font-size:11pt;font-family:Calibri,sans-serif"> </p><div style="margin: 0in 0in 0.0001pt; line-height: normal; font-size: 11pt; font-family: Calibri, sans-serif;" class="">If child selected for sub-code #1 (TCH or OCH), 3 options for
sub-code #4. </div>
</td>
</tr>
</tbody></table><br class="gmail-Apple-interchange-newline"></span></td></tr></tbody></table>Should N = 5? Or the total number of possibilities? Or...? <div class=""><br class=""></div><div class="">Additionally, if coders tend to be very reliable on the main code, sub-code #1, and sub-code #2, which are more straightforward and leave less room for interpretation, how can I go about calculating the reliability of sub-code #3 and sub-code #4 separately? </div><div class=""><br class=""></div><div class="">Thank you so much for your consideration! </div><div class=""><br class=""></div><div class="">Best, </div><div class="">Ari Nelson<br class=""><div class=""><br class=""></div><div class="">-- <br class=""><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr" class=""><div class=""><div dir="ltr" class="">Ariadne Nelson, M.A.</div><div dir="ltr" class="">Pronouns: she/her/hers<br class="">Doctoral Candidate<br class="">Applied Developmental and Educational Psychology<br class="">Lynch School of Education<br class="">Boston College</div></div></div></div></div></div></div><div class=""><br class="webkit-block-placeholder"></div>
-- <br class="">
You received this message because you are subscribed to the Google Groups "chibolts" group.<br class="">
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="mailto:chibolts+unsubscribe@googlegroups.com" class="">chibolts+unsubscribe@googlegroups.com</a>.<br class="">
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/chibolts/CAF%3DFrmyauN49KG00E5deDN1dayBErMs0_CXWPAVjnGwxUi2Myw%40mail.gmail.com?utm_medium=email&utm_source=footer" class="">https://groups.google.com/d/msgid/chibolts/CAF%3DFrmyauN49KG00E5deDN1dayBErMs0_CXWPAVjnGwxUi2Myw%40mail.gmail.com</a>.<br class="">
</div></blockquote></div><br class=""></div></div></div></div></body></html>
<p></p>
-- <br />
You received this message because you are subscribed to the Google Groups "chibolts" group.<br />
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="mailto:chibolts+unsubscribe@googlegroups.com">chibolts+unsubscribe@googlegroups.com</a>.<br />
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/chibolts/7B11DE90-21AD-4A33-896B-637B52174788%40andrew.cmu.edu?utm_medium=email&utm_source=footer">https://groups.google.com/d/msgid/chibolts/7B11DE90-21AD-4A33-896B-637B52174788%40andrew.cmu.edu</a>.<br />