[Corpora-List] dependency-based semantic role labeling annotation tool

[John F Sowa]

On 4/24/2012 10:55 AM, anders bjorkelund wrote:
> Just for clarification, are you looking for a (statistical) system to
> parse semantic roles, or a tool (GUI) to create a gold standard
> annotation?

I just wanted to make a comment about the term "gold standard".

Inter-annotator agreement is so low that it is misleading to call
human annotation the "gold standard".

Anything that is called a gold standard for some purpose should be
thoroughly vetted by multiple annotators -- both human and computerized.

The goal for computer annotation should be to *exceed* the quality
of average human annotators.  That does not imply that the computers
surpass human ability.  It just means that they can be more thorough
and consistent in annotating documents.

And by the way, I wouldn't equate computer annotation with statistical
annotation.  It is true that most of them are statistical, but the
best quality is achieved by using multiple *independent* methods --
for cross checking, for better coverage of rare but significant cases,
and for avoiding systemic errors caused by using a single paradigm.

Ideally, you should only need to call a human expert for the unusual
cases where the multiple computerized tests disagree.

John

_______________________________________________
UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
Corpora mailing list
Corpora at uib.no
http://mailman.uib.no/listinfo/corpora