[Corpora-List] Kappa coefficient for more than two annotators

[Ron Artstein]

On Tue, 18 Jul 2006, Narjes Boufaden wrote:

> I'm looking for a perl script that estimates the Kappa 
> coefficient for data coming from more than two annotators and 
> with multiple classes.

I have such a script; I'll send it to you privately, as it's 
probably not interesting to the entire list.

But I want to include a cautionary note, which should be of 
interest to at least some members of the list: the term "kappa" is 
ambiguous, as it refers to more than one coefficient. For two 
coders, there is kappa (Cohen 1960) which calculates chance 
agreement using individual coder marginals, and pi (Scott 1955) 
which averages over these marginals; unfortunately pi is also often 
called "kappa". For multiple coders, there is kappa (Fleiss 1971, 
Siegel and Castellan 1988, Carletta 1996) which is actually a 
generalization of Scott's pi; there's also another kappa (Davies 
and Fleiss 1982) which is a generalization of Cohen's kappa...

These issues are explained in some detail DiEugenio and Glass 2004 
(Computational Linguistics 30(1): 95-101), and in more detail in 
Artstein and Poesio 2005 (currently under revision):

http://cswww.essex.ac.uk/technical-reports/2005/csm-437.pdf

Fortunately, you have more than two annotators, and indeed the 
difference between the various coefficients called kappa decreases 
as the number of coders grows (another Artstein and Poesio 2005):

http://privatewww.essex.ac.uk/~artstein/publications/bias-a5.pdf

The numerical differences between the variants of kappa are likely 
to be not very large (especially if you have many annotators or 
reach high agreement), but I feel it's important to report exactly 
which coefficient is used.

-Ron.