agreement in hierarchical structures

[Eduard Hovy]

Hi Anna,

At 4:41 PM -0500 12/7/09, Anna Kazantseva wrote:
>I am actually analyzing the corpus of discourse trees created by 
>Carlson, Marcu and Okurowski (2003). ...What I am trying to capture 
>is the fact that the annotations are similar, if not identical, when 
>one considers stand-alone sentences and paragraphs. The annotations 
>differ when people connect larger segments. I did not want to choose 
>a long text for the example, but the phenomenon is more evident in 
>large texts.

The problem of parser output evaluation (comparing to gold standard 
parse trees) involves the same question.

So you can read
http://www.google.com/url?sa=t&source=web&ct=res&cd=2&ved=0CAoQFjAB&url=http%3A%2F%2Fwww.cogs.susx.ac.uk%2Flab%2Fnlp%2Fcarroll%2Fpapers%2Flre98.pdf&rct=j&q=parser+evaluation&ei=pqYdS6q5H4euswOZs7yHCg&usg=AFQjCNEATUkAFlmLTra-G4r-bywKg87OTw
for a list of tree structure comparison methods and metrics.  (Second 
entry on google for "parser evaluation".)

Regards,
E


-- 
Eduard Hovy
email: hovy at isi.edu            USC Information Sciences Institute
tel: 310-448-8731            4676 Admiralty Way
fax: 310-823-6714            Marina del Rey, CA 90292-6695
http://www.isi.edu/natural-language/