Rely command - assessing reliability for each code separately

[Leonid Spektor]

Hi Marissa,

I am not sure if I understand correctly what you want. I have made two sample files. Here is the first one:

*MOT:	child's word.
%spa:	lowq-s
*MOT:	something other .
%spa:	lowq-s
*MOT:	child's word.
%spa:	refq-s
*MOT:	something other .
%spa:	refq-e

and here is the second one:

*MOT:	child's word.
%spa:	lowq-s
*MOT:	something other .
%spa:	refq-e
*MOT:	child's word.
%spa:	lowq-s
*MOT:	something other .
%spa:	refq-m

-----------------------------------
if I ran the following command, then I get result:
> rely +c -t* +t%spa +t*MOT +d Marissa1.cha Marissa2.cha

Marissa1.cha: # utterances examined: 4
Marissa2.cha: # utterances examined: 4
% of all utterances examined with matching codes %: 100.0000

Marissa1.cha: # words examined: 4
Marissa2.cha: # words examined: 4
# words (tokens, codes) matched: 3
% of all words examined with matching codes %: 75.0000

I believe it is a correct result, because all codes except for "refq-s" and "refq-m" match. Please note that the match is for the whole file and not based on what speaker utterances those code are associated with.

-----------------------------------
if I ran the following command, then I get result:
> rely +c -t* +t%spa +t*MOT +d Marissa1.cha Marissa2.cha +slowq-s

Marissa1.cha: total lowq-s: 2
Marissa2.cha: total lowq-s: 2
total lowq-s matched: 2
total lowq-s %: 100.0000

In this case RELY only looks at "lowq-s" codes and those two codes per file match exactly between two files.

-----------------------------------
if I ran the following command, then I get result:
> rely +c -t* +t%spa +t*MOT +d Marissa1.cha Marissa2.cha +s*-s

Marissa1.cha: total *-s: 3
Marissa2.cha: total *-s: 2
total *-s matched: 2
total *-s %: 66.6667

 Again there are three codes with -s in the first file  and there are only two codes in the second file.


Leonid.
-----------------------------------
I am attaching my two sample files here. If this is not what you wanted, then please explain in more details and maybe give me an example of what you are trying to achieve. Please email to me directly at spektor at andrew.cmu.edu.


Leonid.


> On Aug 2, 2024, at 18:34, Marissa Castellana <macaste8 at asu.edu> wrote:
> 
> Hello,
> 
> I am trying to assess inter-rater reliability using the RELY command for each of the four main codes (lowq, refq, recq, and infq) on the dependent tier (%spa) in our transcripts. Each code was also denoted with a "-s", "-m", or "-e" depending on the language used. For example, an utterance may be tagged as "lowq-s", "lowq-m", or "lowq-e".
> 
> However, my code (rely +c  -t* +t%spa +t*MOT +d) seems to capture overall reliability instead of reliability per code. I have tried adding +s"lowq" and also attempted adding the more specific sub-code +s"lowq-e", but these are not producing the intended results. Does anyone have any advice or potential solutions so that I can gather reliability per code rather than overall?
> 
> Thank you!
> Best,
> Marissa
> 
> -- 
> You received this message because you are subscribed to the Google Groups "chibolts" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to chibolts+unsubscribe at googlegroups.com <mailto:chibolts+unsubscribe at googlegroups.com>.
> To view this discussion on the web visit https://groups.google.com/d/msgid/chibolts/db4a7ef9-4541-4d8d-900a-d4c49d973ba7n%40googlegroups.com <https://groups.google.com/d/msgid/chibolts/db4a7ef9-4541-4d8d-900a-d4c49d973ba7n%40googlegroups.com?utm_medium=email&utm_source=footer>.

-- 
You received this message because you are subscribed to the Google Groups "chibolts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chibolts+unsubscribe at googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/chibolts/EACA6EF9-AE62-48CE-9FB4-7CB28F239511%40andrew.cmu.edu.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listserv.linguistlist.org/pipermail/chibolts/attachments/20240802/c84dd917/attachment-0003.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Marissa1.cha
Type: text/chat
Size: 353 bytes
Desc: not available
URL: <http://listserv.linguistlist.org/pipermail/chibolts/attachments/20240802/c84dd917/attachment-0002.bin>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listserv.linguistlist.org/pipermail/chibolts/attachments/20240802/c84dd917/attachment-0004.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Marissa2.cha
Type: text/chat
Size: 354 bytes
Desc: not available
URL: <http://listserv.linguistlist.org/pipermail/chibolts/attachments/20240802/c84dd917/attachment-0003.bin>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listserv.linguistlist.org/pipermail/chibolts/attachments/20240802/c84dd917/attachment-0005.htm>