[Corpora-List] semi-automatic annotation tool evaluation

[Eric Atwell]

Samira,

To evaluate a system for annotating dialogue acts, you could take a
Discourse Corpus where the text has been manually annotated with
discourse connectives and realtions, apply your system to the same text,
and compare your system's analysis with the manual analysis. For example, 
if you work with Arabic, you could try the Arabic Disoourse Corpus 
http://www.arabicdiscourse.net/ built by Leeds PhD student Amal Alsaif.

One problem you may find is that analyses are only directly comparable
if you use the same tag-set of discourse connectives and relations
  - eg for the Arabic Discourse Corpus, see
  http://www.arabicdiscourse.net/connectives/ and
  http://www.arabicdiscourse.net/annotation-scheme/

I assume you want to evaluate accuracy; you might also want to
evaluate ease-of-use, portability, speed etc of your annotation tool,
and for this you could compare it to other annotation tools eg
Amal Alsaif's READ tool: http://www.arabicdiscourse.net/annotation-tool/


I hope this is useful

Eric Atwell, Leeds University 
http://www.comp.leeds.ac.uk/nlp/



On Tue, 1 May 2012, samira ben dbabis wrote:

> Hi,
> I am developing a semi-automatic annotation tool specific for Dialogue Acts
> and I want to evaluate my system.
> So, does anyone knows how to evaluate the performance of semi-automatic
> annotation tools (like GATE, MATE, XDML and DAT tools).
> 
> Best ,
> Samira
>  
> **************************************************************************
> Samira Ben Dbabis
> PhD Computer Science Student
> Faculty of Economic Sciences and management of Sfax
> 
> MIRACL Laboratory
> www.miracl.rnu.tn
> 
> 
> ANLP Research Group
> http://sites.google.com/site/anlprg
> 
> 
> ***************************************************************************
> 
>

-- 
Eric Atwell, Associate Professor, Language research group,
  I-AIBS Institute for Artificial Intelligence and Biological Systems
  School of Computing, Faculty of Engineering, UNIVERSITY OF LEEDS
  Leeds LS2 9JT, England.        TEL: 0113-3435430  FAX: 0113-3435468
  WWW: http://www.comp.leeds.ac.uk/eric
       http://www.comp.leeds.ac.uk/nlp
       http://www.comp.leeds.ac.uk/arabic
-------------- next part --------------
_______________________________________________
UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
Corpora mailing list
Corpora at uib.no
http://mailman.uib.no/listinfo/corpora