Re: the 0.994 accuracy result reported by Snow et al: there was precisely one word used for this task, 'president',<div>with the 3-way ambiguity between</div><blockquote style="margin:0 0 0 40px;border:none;padding:0px">
<div><div>1) executive officer of a firm, corporation, or university</div></div><div><div>2) head of a country (other than the U.S.)</div></div><div><div>3) head of the U.S., President of the United States</div></div></blockquote>
<div><div>Open a dictionary at random and you'll see that most polysemy isn't like that. The result, based on one word, provides no insight into the difficulty of the WSD task</div><div><br></div><div>Adam</div>
<br>
<div class="gmail_quote">On 16 July 2013 13:32, Benjamin Van Durme <span dir="ltr"><<a href="mailto:vandurme@cs.jhu.edu" target="_blank">vandurme@cs.jhu.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Rion Snow, Brendan O'Connor, Daniel Jurafsky and Andrew Y. Ng. Cheap<br>
and Fast - But is it Good? Evaluating Non-Expert Annotations for<br>
Natural Language Tasks. EMNLP 2008.<br>
<a href="http://ai.stanford.edu/~rion/papers/amt_emnlp08.pdf" target="_blank">http://ai.stanford.edu/~rion/papers/amt_emnlp08.pdf</a><br>
<br>
"We collect 10 annotations for each of 177 examples of the noun<br>
“president” for the three senses given in SemEval. [...]<br>
performing simple majority voting (with random tie-breaking) over<br>
annotators results in a rapid accuracy plateau at a very high rate of<br>
0.994 accuracy. In fact, further analysis reveals that there was only<br>
a single disagreement between the averaged non-expert vote and the<br>
gold standard; on inspection it was observed that the annotators voted<br>
strongly against the original gold la-bel (9-to-1 against), and that<br>
it was in fact found to be an error in the original gold standard<br>
annotation.6 After correcting this error, the non-expert accuracy rate<br>
is 100% on the 177 examples in this task. This is a specific example<br>
where non-expert annotations can be used to correct expert<br>
annotations. "<br>
<br>
<br>
<br>
<br>
<br>
Xuchen Yao, Benjamin Van Durme and Chris Callison-Burch. Expectations<br>
of Word Sense in Parallel Corpora. NAACL Short. 2012.<br>
<a href="http://cs.jhu.edu/~vandurme/papers/YaoVanDurmeCallison-BurchNAACL12.pdf" target="_blank">http://cs.jhu.edu/~vandurme/papers/YaoVanDurmeCallison-BurchNAACL12.pdf</a><br>
<br>
<br>
"2 Turker Reliability<br>
<br>
While Amazon’s Mechanical Turk (MTurk) has been been considered in the<br>
past for constructing lexical semantic resources (e.g., (Snow et al.,<br>
2008; Akkaya et al., 2010; Parent and Eskenazi, 2010; Rumshisky,<br>
2011)), word sense annotation is sensi- tive to subjectivity and<br>
usually achieves low agree- ment rate even among experts. Thus we<br>
first asked Turkers to re-annotate a sample of existing gold- standard<br>
data. With an eye towards costs saving, we also considered how many<br>
Turkers would be needed per item to produce results of sufficient<br>
quality.<br>
<br>
Turkers were presented sentences from the test portion of the word<br>
sense induction task of SemEval-2007 (Agirre and Soroa, 2007),<br>
covering 2,559 instances of 35 nouns, expert-annotated with OntoNotes<br>
(Hovy et al., 2006) senses. [...]<br>
<br>
We measure inter-coder agreement using Krip- pendorff’s Alpha<br>
(Krippendorff, 2004; Artstein and Poesio, 2008), [...]"<br>
<div class="HOEnZb"><div class="h5"><br>
_______________________________________________<br>
UNSUBSCRIBE from this page: <a href="http://mailman.uib.no/options/corpora" target="_blank">http://mailman.uib.no/options/corpora</a><br>
Corpora mailing list<br>
<a href="mailto:Corpora@uib.no">Corpora@uib.no</a><br>
<a href="http://mailman.uib.no/listinfo/corpora" target="_blank">http://mailman.uib.no/listinfo/corpora</a><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br>========================================<br><a href="http://www.kilgarriff.co.uk/" target="_blank">Adam Kilgarriff</a> <a href="mailto:adam@lexmasterclass.com" target="_blank">adam@lexmasterclass.com</a> <br>
Director <a href="http://www.sketchengine.co.uk/" target="_blank">Lexical Computing Ltd</a> <br>Visiting Research Fellow <a href="http://leeds.ac.uk" target="_blank">University of Leeds</a> <div>
<i><font color="#006600">Corpora for all</font></i> with <a href="http://www.sketchengine.co.uk" target="_blank">the Sketch Engine</a> </div><div> <i><a href="http://www.webdante.com" target="_blank">DANTE: <font color="#009900">a lexical database for English</font></a><font color="#009900"> </font> </i><div>
========================================</div></div>
</div>