Number of categories for computing kappa with RELY

[Brian MacWhinney]

Ari,

Given all your subcodes, it looks like you have 1122  ((4x5x14x4) +2) possible codes. So that would be the N for Cohen's kappa.

In order to calculate the reliability of subcodes such as those for #3 by themselves, you might need to edit the coding tier in your files to remove everything but those codes.  RELY wasn't designed to do this, but if you edit the data to just focus on those choices, it should be able to do it okay.   You would make a copy of your files and them use a GREP search and replace string to get rid of the first three and last elements in your string.  Having them separated by colons makes this not too hard.  Of course, then your N for Cohen's kappa would only be 14.

-- Brian MacWhinney

Teresa Heinz Professor of 
Cognitive Psychology,
Computational Linguistics, 
and Modern Languages, CMU

> On Apr 3, 2020, at 4:34 PM, Ariadne Nelson <nelsonat at bc.edu> wrote:
> 
> Good evening,
> 
> I have a question about calculating the inter-rater reliability (Cohen's kappa coefficient) of codes in CLAN using the RELY command. 
> 
> I have been using the command: rely +dN +c @ +t%num -t*, with N equaling the number of categories. Does this look correct?
> 
> I am unsure how to calculate the number of categories for my coding scheme. This table breaks down my coding scheme. There are three main codes ($OT, $RE, $NT). If $NT is selected, there are four subcodes, with differing quantities of options to choose from. A full $NT might look like: $NT:PAR:ALL:EME:P.
> 
> Main code
> Sub-Code #1
> Sub-Code #2
> Sub-Code #3
> Sub-Code #4
> $OT
>  
>  
>  
>  
> $RE
>  
>  
>  
>  
> $NT
> 4 options:
> Target child (TCH), Parent (PAR),
> Other Child (OCH),
> Other Adult (OAD)
> 5 options
> 14 options
> If adult selected for sub-code #1 (PAR or OAD), 4 options for sub-code #4.
>  
> If child selected for sub-code #1 (TCH or OCH), 3 options for sub-code #4.
> 
> Should N = 5? Or the total number of possibilities? Or...? 
> 
> Additionally, if coders tend to be very reliable on the main code, sub-code #1, and sub-code #2, which are more straightforward and leave less room for interpretation, how can I go about calculating the reliability of sub-code #3 and sub-code #4 separately? 
> 
> Thank you so much for your consideration! 
> 
> Best, 
> Ari Nelson
> 
> -- 
> Ariadne Nelson, M.A.
> Pronouns: she/her/hers
> Doctoral Candidate
> Applied Developmental and Educational Psychology
> Lynch School of Education
> Boston College
> 
> -- 
> You received this message because you are subscribed to the Google Groups "chibolts" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to chibolts+unsubscribe at googlegroups.com <mailto:chibolts+unsubscribe at googlegroups.com>.
> To view this discussion on the web visit https://groups.google.com/d/msgid/chibolts/CAF%3DFrmyauN49KG00E5deDN1dayBErMs0_CXWPAVjnGwxUi2Myw%40mail.gmail.com <https://groups.google.com/d/msgid/chibolts/CAF%3DFrmyauN49KG00E5deDN1dayBErMs0_CXWPAVjnGwxUi2Myw%40mail.gmail.com?utm_medium=email&utm_source=footer>.

-- 
You received this message because you are subscribed to the Google Groups "chibolts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chibolts+unsubscribe at googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/chibolts/7B11DE90-21AD-4A33-896B-637B52174788%40andrew.cmu.edu.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listserv.linguistlist.org/pipermail/chibolts/attachments/20200403/e2da8b70/attachment.htm>