[Corpora-List] Results of the query: Kappa coefficient for more than two annotators

[Narjes Boufaden]

Hi everybody,

Thank you for all of you who answered my question. Here are the list of
your suggestions:

+ Adam Kilgarriff pointed out some of the problems related to the
assumptions made when computing kappa.

+ Norton Roman suggested the use of Krippendorf's alpha to evaluate
annotator agreement.

+ Ron Artstein kindly sent a script for the kappa (multiple juges). You
ca ask him directly to have the script.

+ Andrei Popescu-Belis proposed a website where is available an
implementation  (MS-DOS) of the Fleiss approcah (an extention of the
Cohen's kappa coefficient for multiple juges) : http://kappa.chez-alice.fr


Thanks again for all your suggestions.

Best regards, Narjès.