Inter-rater reliability of utterance segmentation

Sat Apr 15 14:07:50 UTC 2023

Correct; it is one quantitative way of evaluating reliability, not the only
way or the most comprehensive picture.

Brielle C. Stark, PhD
Assistant Professor
Department of Speech, Language and Hearing Sciences
Program in Neuroscience, Cognitive Science Program
Indiana University Bloomington

sent from mobile, please excuse errors

On Sat, Apr 15, 2023, 09:22 Brian Macwhinney <macw at andrew.cmu.edu> wrote:

> Oh, yes, that ICC.  That could work.  But wouldn’t that be limited to
> comparing two or more raters in terms of their scores for the total number
> of utterances in a sample.  It would not look at how they got to that
> number.  So, one rater could break up utterance 52 into two pieces, but
> then join utterances 83 and 84 into one and end up with a score that is the
> same as the gold standard segmentation.
>
> More generally, the standard measures of reliability, including those
> computed by RELY are meant to apply to coding systems, such as one would
> find on the %spa or %cod lines and not the main line of a transcript.
>
> To compute word-level agreement, one could turn to something like a BLUE
> score that is used in judging machine translations. There is a good
> discussion of this in the Wikipedia page.  However, BLEU is working on
> pairs of utterances and we already have a problem if the two coders have
> segmented things differently.
>
> Grant and journal eviewers often ask whether transcription reliability has
> been computed. This seems like a reasonable request, but the fact is that
> there is no straightforward way to compute this.  Perhaps the “bag of
> words” method available through the fourth function for RELY in section
> 7.21 of the CLAN manual makes the most sense.
>
> —Brian MacWhinney
>
> > On Apr 14, 2023, at 9:22 PM, Brielle Stark <brielle.stark at gmail.com>
> wrote:
> >
> > Intraclass Correlation Coefficient, as is usually used for reliability
> estimates :)
> >
> > Brielle C. Stark, PhD
> > Assistant Professor
> > Department of Speech, Language and Hearing Sciences
> > Program in Neuroscience, Cognitive Science Program
> > Indiana University Bloomington
> >
> > sent from mobile, please excuse errors
> >
> > On Fri, Apr 14, 2023, 20:19 Brian Macwhinney <macw at andrew.cmu.edu>
> wrote:
> > Brie,
> >     Sounds like a good idea, but does ICC refer to:  International
> Criminal Court?  Illinois Commerce Commission? International Cricket
> Council? Illinois Central College? International Color Consortium?
> International Christian Concern? or maybe the Industrial Chimney Company?
> So far, googling this hasn’t helped.
> >
> > — Brian MacWhinney
> >
> > > On Apr 14, 2023, at 4:28 PM, Brielle Stark <brielle.stark at gmail.com>
> wrote:
> > >
> > > One way we've done it is to look at number of utterances using outside
> tools like ICC. That's also a more robust metric for that then percent
> agreement, I'd argue.
> > >
> > > Brielle C. Stark, PhD
> > > Assistant Professor
> > > Department of Speech, Language and Hearing Sciences
> > > Program in Neuroscience, Cognitive Science Program
> > > Indiana University Bloomington
> > >
> > > sent from mobile, please excuse errors
> > >
> > > On Fri, Apr 14, 2023, 16:13 Leonid Spektor <spektor at andrew.cmu.edu>
> wrote:
> > > Cynthia,
> > >
> > > For RELY to work the speaker utterance in the first file must have
> corresponding utterance of the same speaker in the second file. Your
> example will produce an error that utterance do not match.
> > >
> > > Perhaps someone else can have a better suggestion.
> > >
> > >
> > > Leonid.
> > >
> > >> On Apr 14, 2023, at 15:54, Cynthia Audisio <cpaudisio at gmail.com>
> wrote:
> > >>
> > >> Dear chibolts,
> > >>
> > >> With some colleagues, we are trying to analyze inter-rater
> reliability of utterance segmentation in CHAT transcripts of spontaneous
> situations. We were wondering whether we can use RELY for this purpose.
> > >>
> > >> Here is an example of how transcripts by two different transcribers
> might look:
> > >>
> > >> Transcriber 1:
> > >> CHI: Mami, tenés < el vasito > [/] el vasito de Juan ?
> > >>
> > >> Transcriber 2:
> > >> CHI: Mami, tenés el vasito ?
> > >> CHI: el vasito de Juan ?
> > >>
> > >> Best,
> > >> Cynthia
> > >>
> > >> --
> > >> You received this message because you are subscribed to the Google
> Groups "chibolts" group.
> > >> To unsubscribe from this group and stop receiving emails from it,
> send an email to chibolts+unsubscribe at googlegroups.com.
> > >> To view this discussion on the web visit
> https://groups.google.com/d/msgid/chibolts/3902be92-562e-4e3f-b4cd-35328e60f465n%40googlegroups.com
> .
> > >
> > >
> > > --
> > > You received this message because you are subscribed to the Google
> Groups "chibolts" group.
> > > To unsubscribe from this group and stop receiving emails from it, send
> an email to chibolts+unsubscribe at googlegroups.com.
> > > To view this discussion on the web visit
> https://groups.google.com/d/msgid/chibolts/FC7F6924-A5ED-4586-89F6-2130D5EE1442%40andrew.cmu.edu
> .
> > >
> > > --
> > > You received this message because you are subscribed to the Google
> Groups "chibolts" group.
> > > To unsubscribe from this group and stop receiving emails from it, send
> an email to chibolts+unsubscribe at googlegroups.com.
> > > To view this discussion on the web visit
> https://groups.google.com/d/msgid/chibolts/CAEs2yTqZYz-O7U9Kw4mQ1uDxkggmRSyuJqs1Kp5wiJCAKbpqJQ%40mail.gmail.com
> .
> >
>
>

-- 
You received this message because you are subscribed to the Google Groups "chibolts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chibolts+unsubscribe at googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/chibolts/CAEs2yTpAKQ%3DKWjQJ%2BrN151rRdJa-RuSEiVNdiZvDvv9GY2vR%3Dg%40mail.gmail.com.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listserv.linguistlist.org/pipermail/chibolts/attachments/20230415/e2be0690/attachment-0001.htm>