28.1047, Qs: Establishing Credibility in Annotations

The LINGUIST List linguist at listserv.linguistlist.org
Tue Feb 28 19:22:33 UTC 2017


LINGUIST List: Vol-28-1047. Tue Feb 28 2017. ISSN: 1069 - 4875.

Subject: 28.1047, Qs: Establishing Credibility in Annotations

Moderators: linguist at linguistlist.org (Damir Cavar, Malgorzata E. Cavar)
Reviews: reviews at linguistlist.org (Helen Aristar-Dry, Robert Coté,
                                   Michael Czerniakowski)
Homepage: http://linguistlist.org

Editor for this issue: Kenneth Steimel <ken at linguistlist.org>
================================================================


Date: Tue, 28 Feb 2017 14:22:26
From: Jonathan Brown [yohane711 at gmail.com]
Subject: Establishing Credibility in Annotations

 
I am conducting a contrastive study between Japanese and English expository
prose within the framework of Rhetorical Structure Theory (RST) for my PhD. I
have analyzed 22 texts in each corpus using RST and want to show that my
analyses are credible. It simply is not feasible to find another annotator to
analyze all 44 texts, so I was wondering if there is a way to show that my
analyses are in fact credible. I underwent about a four-hour training session
with my PhD advisor until we came to an agreement on six of the 44 texts. He
seems to think that this should suffice, but I feel like I can make it more
rigorous by taking a few texts from each corpus and having a second rater
analyze these few texts within the RST framework and then calculate Cohen's
kappa. If the reliability is good, would this be sufficient to suggest that
the remaining texts (that were not annotated by a second rater) are credible
as well? If so, is there any literature/studies on this to which I could
refer?

Another, more simplistic, idea I had was just to have a second rater agree or
disagree with my annotations. If he disagrees, it really doesn't matter the
relation he chooses. I thought that perhaps this would be okay since I had
already done a training session with my PhD advisor. This would also be the
easiest and fastest approach (I cannot ask my second rater to spend too much
time on it since he is not involved in my PhD at all), but, again, I'm not
sure if it will be criticized. 

Would either of these approaches work? If not, is there a more appropriate
approach that you could suggest that would not require too much work on a
second rater?
 

Linguistic Field(s): Discourse Analysis
                     Text/Corpus Linguistics

Subject Language(s): English (eng)
                     Japanese (jpn)



----------------------------------------------------------
LINGUIST List: Vol-28-1047	
----------------------------------------------------------






More information about the LINGUIST mailing list