RELY

[Brian MacWhinney]

Dear Karin,
    Looking now at that section of the CLAN manual, I see that it had not been properly updated to include both the +dm1 and +dN switches.  Here is the revised section:

The second function of the RELY command is to evaluate the correctness of a student transcript against a master transcript. The student is given a version of the complete master transcript without the coding tier and the goal is to add codes. In this case, it is assumed that the coding in the master transcript is fully correct. The question is whether the student's codes are correct. To specify that the first file given is the master, you use the +dm1 switch.  In addition, to compute Cohen's kappa you need to add the +dN switch. In this example, we can say that there are only two levels to the category, making the +dN switch to be +d2.  However, the number used would depend on the actual levels of the coding system.
rely +d2 +dm1 sample.cha samplea.cha +t%spa

The output from this command provides two counts: precision and accuracy.  These are the two dimensions used in fields such as computational linguistics to evaluate correctness.  Precision is the percentage of fields in the master file that are matched by the student. Accuracy is the percentage of fields in the student file that are matched to the master. A failure in precision occurs when the student misses a code in the master file.  A failure in accuracy occurs when the student inserts a code that is not found in the master. If a researcher is interested in going beyond these two scores, they can also compute the F score which is the harmonic mean of precision and accuracy.  By adding the +d2 switch, the command also produces Cohen's kappa.


--Brian MacWhinney

> On Jun 11, 2020, at 5:17 AM, Karin Edlund <karin.eva.edlund at gmail.com> wrote:
> 
> Hi,
> What is the accurate command for RELY and computing percentage agreement on a dependant tier (i.e. %cod)?
> I have tried 
> rely +d sample.cha samplea.cha +t%cod -t*
> 
> Also, what is the command for calculating Cohen's kappa on %cod tier?
> I have tried with +dN, but I can't get it to work unfortunately.
> 
> Happy to receive advice on how to solve this.
> 
> Best Regards,
> Karin Edlund
> 
> -- 
> You received this message because you are subscribed to the Google Groups "chibolts" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to chibolts+unsubscribe at googlegroups.com <mailto:chibolts+unsubscribe at googlegroups.com>.
> To view this discussion on the web visit https://groups.google.com/d/msgid/chibolts/CANkUE2xW3-jbmfqNXyCVLAwZOh8dpCccdTeEWqujDsJQqTgH%3DQ%40mail.gmail.com <https://groups.google.com/d/msgid/chibolts/CANkUE2xW3-jbmfqNXyCVLAwZOh8dpCccdTeEWqujDsJQqTgH%3DQ%40mail.gmail.com?utm_medium=email&utm_source=footer>.

-- 
You received this message because you are subscribed to the Google Groups "chibolts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chibolts+unsubscribe at googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/chibolts/FF171C4D-A67B-4752-B054-98CD31394E58%40andrew.cmu.edu.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listserv.linguistlist.org/pipermail/chibolts/attachments/20200611/a1279614/attachment.htm>