reliability test

[Brian MacWhinney]

Dear Karin,
    RELY was originally designed to check reliability of coding tiers, rather than the main line. For that, it does a great job, because the shape of coding tiers is so well defined.  If that is what you are doing, then I would say that the decision about what percentage to check is really a function of your overall corpus.  For example, if you only have two hours of recording and you check 10% of that, then that would seem incomplete.  However, if you have 200 hours and you check a random 10% of that and show good reliability, then I would be pretty convinced that you are okay.  So, percentage is a function of corpus size. 
    I suppose a similar argument could be made regarding corpus or data type, but that argument would be a bit less convincing.
       Later on, Leonid expanded the function of RELY to include comparisons of the main line.  This method just looks for overall match of words.  Please just read the manual about these differences.

Best,

-- Brian MacWhinney

> On Jun 1, 2020, at 4:09 AM, Karin Edlund <karin.eva.edlund at gmail.com> wrote:
> 
> Hi,
> 
> I have a question about reliability test and deciding on a proper amount of my material to check reliability for. 
> 
> I am planning to use the RELY function in CLAN to check for reliability and to do it on 10% of my data. 
> 
> I know in some studies as much as 25% is checked for agreement. 
> 
> What are your guidelines? is it depending on the collected data, how much it encompasses or the data's content? 
> If you have suggestions on references, I am grateful to take part of them.
> 
> Best Regards,
> Karin Edlund,
> PhD student in Special Education
> Stockholm University
> 
> -- 
> You received this message because you are subscribed to the Google Groups "chibolts" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to chibolts+unsubscribe at googlegroups.com <mailto:chibolts+unsubscribe at googlegroups.com>.
> To view this discussion on the web visit https://groups.google.com/d/msgid/chibolts/CANkUE2ze32zpkYg%2Bkw9TtH7YUfLe2-J8gzws1n%2Bq06He9-rmRw%40mail.gmail.com <https://groups.google.com/d/msgid/chibolts/CANkUE2ze32zpkYg%2Bkw9TtH7YUfLe2-J8gzws1n%2Bq06He9-rmRw%40mail.gmail.com?utm_medium=email&utm_source=footer>.

-- 
You received this message because you are subscribed to the Google Groups "chibolts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chibolts+unsubscribe at googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/chibolts/BDCEAD24-8DC5-456C-8C73-5D6CADAE86C7%40andrew.cmu.edu.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listserv.linguistlist.org/pipermail/chibolts/attachments/20200601/e508ec9e/attachment.htm>