[Lingtyp] typology projects that use inter-rater reliability?

[Johann-Mattis List]

In 2010, after a conference in 2009 of the Indo-European circle in 
Würzburg, we submitted a study comparing two independently compiled 
lexicostatistical datasets. We did not really know about 
inter-annotator-agreement by then, because I had just started my PhD and 
barely knew how to even program. But we provided a first account on the 
problems of even just having two people comparing a Swadesh list for the 
same language.

This study has never been officially published, but I submitted it to a 
preprint server, where it has been accessible since then.

https://hal.science/hal-01298493v1/file/beautiful__trees.pdf

Even if nobody followed up on this and I did never find the motivation 
to re-do the study, I would consider this as an early example for a 
study that actively asks for inter-annotator-agreement.

Best,

Mattis

Am 15.10.23 um 23:29 schrieb Juergen Bohnemeyer:
> Hello Adam – In this 
> <https://www.dropbox.com/scl/fi/mqupkt7weval8g4fhaa89/01_07_2023_8_00am_Bohnemeyer.pdf?rlkey=1gsg2yfa2r7bigvchktk7ww3l&dl=0> as yet unpublished study, we didn’t do an interrater reliability study, but we used MDS to compare interrater variation as one measure of prototypicality. See Slides 37-41. Here’s <https://www.dropbox.com/scl/fi/cp2ir7vyu9btaua7cn5q8/Using-statistical-classification-to-discover-cross-linguistic-semantic-prototypes-copy.mp4?rlkey=njd01v8duqoiabu0lcklz8o36&dl=0> a video of me presenting this paper at the LSA. – Best – Juergen
> 
> Juergen Bohnemeyer (He/Him)
> Professor, Department of Linguistics
> University at Buffalo
> 
> Office: 642 Baldy Hall, UB North Campus
> Mailing address: 609 Baldy Hall, Buffalo, NY 14260
> Phone: (716) 645 0127
> Fax: (716) 645 3825
> Email: jb77 at buffalo.edu <mailto:jb77 at buffalo.edu>
> Web: http://www.acsu.buffalo.edu/~jb77/ <http://www.acsu.buffalo.edu/~jb77/>
> 
> Office hours Tu/Th 3:30-4:30pm in 642 Baldy or via Zoom (Meeting ID 585 
> 520 2411; Passcode Hoorheh)
> 
> There’s A Crack In Everything - That’s How The Light Gets In
> (Leonard Cohen)
> 
> -- 
> 
> *From: *Lingtyp <lingtyp-bounces at listserv.linguistlist.org> on behalf of 
> Adam James Ross Tallman <ajrtallman at utexas.edu>
> *Date: *Saturday, October 14, 2023 at 18:16
> *To: *LINGTYP at LISTSERV.LINGUISTLIST.ORG <LINGTYP at listserv.linguistlist.org>
> *Subject: *[Lingtyp] typology projects that use inter-rater reliability?
> 
> Hello all,
> 
> I am gathering a list of projects / citations / papers that use or refer 
> to inter-rater reliability. So far I have.
> 
> Himmelmann et al. On the universality of intonational phrases: a 
> cross-linguistic interrater study. /Phonology /35.
> 
> Gast & Koptjevskaja-Tamm. 2022. Patterns of persistence and 
> diffusibility in the European lexicon. /Linguistic Typology /(not 
> explicitly the topic of the paper, but interrater reliability metrics 
> are used)
> 
> I understand people working with Grambank have used it, but I don't know 
> if there is a publication on that.
> 
> best,
> 
> Adam
> 
> 
> -- 
> 
> Adam J.R. Tallman
> 
> Post-doctoral Researcher
> 
> Friedrich Schiller Universität
> 
> Department of English Studies
> 
> 
> _______________________________________________
> Lingtyp mailing list
> Lingtyp at listserv.linguistlist.org
> https://listserv.linguistlist.org/cgi-bin/mailman/listinfo/lingtyp