inter-rater agreement

[Michele Koven]

I was wondering what experience people on this list have had with different
ways of calculating inter-rater agreement for languagey data. With another
coder,  I'm trying to get decent reliability for coding lexical register in
French and have four different register categories.  Given that my values
aren't well distributed across my four categories (as one might expect),
this is giving me poor success with Cohen's kappa.  Unfortunately,
percentage agreement doesn't cut it for many.

All ideas welcome!

Thanks,

Michele Koven



Michèle Koven
Assistant Professor
Department of Speech Communication
University of Illinois at Urbana-Champaign
244 Lincoln Hall
702 S. Wright St.
Urbana, Illinois 61801

ph:(217) 333-8969
fax: (217) 244-1598