<html dir="ltr"><head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style title="owaParaStyle">P {
MARGIN-TOP: 0px; MARGIN-BOTTOM: 0px
}
</style>
</head>
<body ocsi="x">
<div dir="ltr"><font face="Tahoma" color="#000000" size="2">Dear all,</font></div>
<div dir="ltr"><font face="tahoma" size="2"></font> </div>
<div dir="ltr"><font face="tahoma" size="2">one way to measure inter-coder reliability is Cohen's kappa. But this can only be applied to pairs of raters, at least in the standard use.</font></div>
<div dir="ltr"><font face="tahoma" size="2"></font> </div>
<div dir="ltr"><font face="tahoma" size="2">One solution to the problem of having more than two coders is to average Cohen's kappas across all possible pairs of raters, but I am not sure how this is looked upon in the testing community.</font></div>
<div dir="ltr"><font face="tahoma" size="2"></font> </div>
<div dir="ltr"><font face="tahoma" size="2">Another solution to this problem appears to be Fleiss' kappa, which can accommodate more raters in one reliability analysis. What sort of experience do you have with this statistic? And are there any software packages
that include it (since SPSS does not seem to have it)?</font></div>
<div dir="ltr"><font face="tahoma" size="2"></font> </div>
<div dir="ltr"><font face="tahoma" size="2">Any advice will be greatly appreciated.</font></div>
<div dir="ltr"><font face="tahoma" size="2"></font> </div>
<div dir="ltr"><font face="tahoma" size="2">Many thanks,</font></div>
<div dir="ltr"><font face="tahoma" size="2"></font> </div>
<div dir="ltr"><font face="tahoma" size="2">Gerard Steen</font></div>
<div dir="ltr"><font face="tahoma" size="2"></font> </div>
<div dir="ltr"><font face="tahoma" size="2"></font> </div>
<div dir="ltr"><font face="tahoma" size="2"></font> </div>
<div class="BodyFragment"><font size="2">
<div class="PlainText">Professor of Language Use and Cognition<br>
Director, Language, Cognition, Communication program<br>
Faculty of Arts, 11A-35<br>
Department of Language and Communication<br>
VU University Amsterdam<br>
De Boelelaan 1105<br>
1081 HV Amsterdam<br>
<br>
T: ++31-20-5986433<br>
F: ++31-20-5986500<br>
<br>
http://www.let.vu.nl/staf/gj.steen/</div>
</font></div>
</body>
</html>