Number of categories for computing kappa with RELY

[Ariadne Nelson]

Good evening,

I have a question about calculating the inter-rater reliability (Cohen's
kappa coefficient) of codes in CLAN using the RELY command.

I have been using the command: rely +dN +c @ +t%num -t*, with N equaling
the number of categories. Does this look correct?

I am unsure how to calculate the number of categories for my coding scheme.
This table breaks down my coding scheme. There are three main codes ($OT,
$RE, $NT). If $NT is selected, there are four subcodes, with differing
quantities of options to choose from. A full $NT might look like:
$NT:PAR:ALL:EME:P.

*Main code*

*Sub-Code #1*

*Sub-Code #2*

*Sub-Code #3*

*Sub-Code #4*

$OT









$RE









$NT

4 options:

Target child (TCH), Parent (PAR),

Other Child (OCH),

Other Adult (OAD)

5 options

14 options

If adult selected for sub-code #1 (PAR or OAD), 4 options for sub-code #4.



If child selected for sub-code #1 (TCH or OCH), 3 options for sub-code #4.

Should N = 5? Or the total number of possibilities? Or...?

Additionally, if coders tend to be very reliable on the main code, sub-code
#1, and sub-code #2, which are more straightforward and leave less room for
interpretation, how can I go about calculating the reliability of sub-code
#3 and sub-code #4 separately?

Thank you so much for your consideration!

Best,
Ari Nelson

-- 
Ariadne Nelson, M.A.
Pronouns: she/her/hers
Doctoral Candidate
Applied Developmental and Educational Psychology
Lynch School of Education
Boston College

-- 
You received this message because you are subscribed to the Google Groups "chibolts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chibolts+unsubscribe at googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/chibolts/CAF%3DFrmyauN49KG00E5deDN1dayBErMs0_CXWPAVjnGwxUi2Myw%40mail.gmail.com.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listserv.linguistlist.org/pipermail/chibolts/attachments/20200403/1191b34a/attachment.htm>