[Corpora-List] Kappa coefficient for more than two annotators

[Adam Kilgarriff]

Narjes,

Don't assume it will get you an informative result.  Kappa depends on what a
random guesser does, but the random guesser's results depend on the
probability function you assume.  It is easy to use kappa to spread more
confusion that light.  See section 5 of
http://www.kilgarriff.co.uk/Publications/2003-K-EACLBuda-noburo.pdf 

Adam


-----Original Message-----
From: owner-corpora at lists.uib.no [mailto:owner-corpora at lists.uib.no] On
Behalf Of Narjes Boufaden
Sent: 18 July 2006 17:35
To: corpora
Subject: [Corpora-List] Kappa coefficient for more than two annotators

Hi everybody,

I'm looking for a perl script that estimates the Kappa coefficient for
data coming from more than two annotators and  with multiple classes.

Thanks a lot for any help,

Narjès.