agreement in hierarchical structures

[Anna Kazantseva]

It seems that I have not provided enough information in my question.

I am actually analyzing the corpus of discourse trees created by 
Carlson, Marcu and Okurowski (2003). It consists mostly of Wall Street 
Journal articles annotated with discourse structure. I have posted an 
example of a small text and the two discourse trees available for it at:

http://site.uottawa.ca/~ankazant/hierarchical_annotations.html

What I am trying to capture is the fact that the annotations are 
similar, if not identical, when one considers stand-alone sentences and 
paragraphs. The annotations differ when people connect larger segments. 
I did not want to choose a long text for the example, but the phenomenon 
is more evident in large texts.

Thanks again!

Regards,

Anna

Rich Cooper wrote:
> Hi Anna,
>
> Could you post a sample with discourse tree?  That might help stimulate
> agreement metrics.  
>
> -Rich
>  
> Sincerely,
> Rich Cooper
> EnglishLogicKernel.com
> Rich AT EnglishLogicKernel DOT com
>
> -----Original Message-----
> From: RST Discussion List [mailto:rstlist at listserv.linguistlist.org] On
> Behalf Of Anna Kazantseva
> Sent: Monday, December 07, 2009 11:27 AM
> To: RSTLIST at LISTSERV.LINGUISTLIST.ORG
> Subject: [RST-LIST] agreement in hierarchical structures
>
> Hi everybody,
>
> It seems that I need to ask for help again. Does anyone know of any 
> agreement metrics that would be suitable for hierarchical structures 
> (trees, in my case)? I am particularly interested in metrics that would 
> compare agreement at different levels.
>
> I am working with a corpus of discourse trees annotated by 2 people and 
> I want to check the hypothesis that they agree quite well at the leaves 
> but the agreement worsens as the spans become larger. I consider both 
> segmentation and labeling of the arcs. I do not seem to find a metric 
> that would not over-penalize disagreements that are due to disagreement 
> at lower levels.
>
> I would appreciate any pointers or thoughts.
>
> Thank you very much in advance
>
>
> Anna
>
>   


-- 
Anna Kazantseva, Ph.D. Candidate
School of Information Technology and Engineering
University of Ottawa
800 King Edward Ave. Ottawa, Ontario, Canada, K1N 6N5