[Corpora-List] WSD / # WordNet senses / Mechanical Turk

[Kilian Evang]

Off the top of my head, here's two relevant studies on inter-rater
reliability for WSD, one for the case of expert annotators and one for
the case of non-experts:

http://link.springer.com/article/10.1023/A:1002693207386#page-1

http://dl.acm.org/citation.cfm?id=1613751

Cheers,
Kilian


On 15/07/13 23:59, Mark Davies wrote:
> Sorry if this is a basic question for computational linguists; I'm a
> corpus linguist.
> 
> I'm wondering if there has been much research on inter-rater
> reliability of word sense disambiguation by raters on something like
> Mechanical Turk. For example:
> 
> -- Given some verbs that have 5 word senses each in WordNet (e.g. the
> words tag, tame, taste, temper), how well do native speakers agree on
> the word sense for these verbs in context -- How does this
> inter-rater reliability change for words that might have just two
> senses (e.g. the verbs taint, tamper, tan, tank) or maybe 10 senses
> (e.g. the verbs shift, spread, stop, trim). (In other words,
> intuition suggests that for words with two WordNet senses, there
> might be higher inter-rater reliability than those words with five
> senses, and that for words with 10 WN senses, inter-rate reliability
> would be pretty bad.) -- Semantically, which kinds of 2 / 5 / 10 WN
> entry words have the best inter-rater reliability, and which have the
> worst?
> 
> Thanks in advance.
> 
> Mark Davies
> 
> ============================================ Mark Davies Professor of
> Linguistics / Brigham Young University 
> http://davies-linguistics.byu.edu/
> 
> ** Corpus design and use // Linguistic databases ** ** Historical
> linguistics // Language variation ** ** English, Spanish, and
> Portuguese ** ============================================
> 
> _______________________________________________ UNSUBSCRIBE from this
> page: http://mailman.uib.no/options/corpora Corpora mailing list 
> Corpora at uib.no http://mailman.uib.no/listinfo/corpora
> 


_______________________________________________
UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
Corpora mailing list
Corpora at uib.no
http://mailman.uib.no/listinfo/corpora