agreement in hierarchical structures

[Anna Kazantseva]

Hi everybody,

It seems that I need to ask for help again. Does anyone know of any 
agreement metrics that would be suitable for hierarchical structures 
(trees, in my case)? I am particularly interested in metrics that would 
compare agreement at different levels.

I am working with a corpus of discourse trees annotated by 2 people and 
I want to check the hypothesis that they agree quite well at the leaves 
but the agreement worsens as the spans become larger. I consider both 
segmentation and labeling of the arcs. I do not seem to find a metric 
that would not over-penalize disagreements that are due to disagreement 
at lower levels.

I would appreciate any pointers or thoughts.

Thank you very much in advance


Anna

-- 
Anna Kazantseva, Ph.D. Candidate
School of Information Technology and Engineering
University of Ottawa
800 King Edward Ave. Ottawa, Ontario, Canada, K1N 6N5