<div class="gmail_quote">
<p>I am working on a proposal to the Obama administration to sponsor a large-scale annotation effort, as one way to ameliorate the high unemployment rate in the United States. Thinking large, it would be billed as the "The Million American Annotation Effort". A plaint text version is shown at the end of this message, and the formatted version of the proposal can be downloaded via <a href="http://cs.txstate.edu/~to17/temp/million-american-annotation-effort-draft.docx">http://cs.txstate.edu/~to17/<span>temp</span>/million-american-annotation-effort-draft.docx</a>.</p>
<p>I would like to get feedback on the proposal as well as input on some aspects (e.g., target areas for annotations). Below is a brief survey to be filled out, along with space for general comments. To minimize traffic to the Corpora list, please just reply to me (<a href="mailto:thomas.paul.ohara@gmail.com" target="_blank">thomas.paul.ohara@gmail.com</a>). I will summarize the responses in a few weeks, maintaining anonymity.</p>
<p>The survey is also intended to gauge the extent to which the research community feels such an annotation effort would be worthwhile as well as to see if others might be interested in collaborating on it. Naturally, getting full approval for such a proposal is a long shot to say the least. However, it might be possible to get a pilot study funded, especially in the context of exploring novel approaches to the addressing the unemployment problem (e.g., part of campaign promise).</p>
<p>Best,<br>Tom</p>
<p>----------</p>
<p><font face="courier new,monospace">Survey:<br>[x] Keep anonymous (e.g., with respect to 'Other' or 'General Feedback')</font></p>
<p><font face="courier new,monospace">1. Overall assessment of proposal's merit<br>Choose one:<br>a. ( ) Basically meritless<br>b. ( ) Not at all practical<br>c. ( ) Feasible if sufficiently constrained<br>d. ( ) Entirely worthwhile<br>
e. (*) No response</font></p>
<p><font face="courier new,monospace">2. Level of expected participation by responder<br>Choose one:<br>a. ( ) Interested in organizational work<br>b. ( ) Interested in consulting role (e.g., resource development)<br>c. ( ) Can't participate directly but can provide moral support<br>
d. ( ) No interest whatsoever (e.g., see 1a above)<br>e. (*) No response</font></p>
<p><font face="courier new,monospace">3. Target inventory for word-sense annotations<br>Choose one or more:<br>a. [ ] WordNet (</font><a href="http://wordnet.princeton.edu/" target="_blank"><font face="courier new,monospace">http://wordnet.princeton.edu</font></a><font face="courier new,monospace">)<br>
b. [ ] FrameNet (</font><a href="https://framenet.icsi.berkeley.edu/" target="_blank"><font face="courier new,monospace">https://framenet.icsi.berkeley.edu</font></a><font face="courier new,monospace">)<br>c. [ ] Dante (</font><a href="http://www.webdante.com/" target="_blank"><font face="courier new,monospace">http://www.webdante.com</font></a><font face="courier new,monospace">)<br>
d. [ ] Other: ________________________________________<br>e. [x] No response</font></p>
<p><font face="courier new,monospace">4. Additional natural language annotations target areas<br>Choose one or more:<br>a. [ ] Machine translation<br>b. [ ] Semantic roles (e.g., FrameNet)<br>c. [ ] Information retrieval<br>
d. [ ] Other: ________________________________________<br>e. [x] No response</font></p>
<p><font face="courier new,monospace">5. Other target areas<br>Choose one or more:<br>a. [ ] Image analysis (e.g., object recognition)<br>b. [ ] Music information retrieval (e.g., verse-level annotations)<br>c. [ ] Web page annotation (e.g., for Semantic Web)<br>
d. [ ] Other: ________________________________________<br>e. [x] No response</font></p>
<div><br><font face="courier new,monospace">General Feedback:<br>________________________________________________________________________________<br></font><font face="courier new,monospace">________________________________________________________________________________<br>
________________________________________________________________________________<br>________________________________________________________________________________<br>________________________________________________________________________________<br>
[Use as much space as desired]</font></div>
<div> </div>
<div>================================================================================</div>
<div> </div>
<div><strong>The Million American Annotation Effort: DRAFT</strong></div>
<div>note: See attachment for formatted version (e.g., hyperlinks with pointers to further information)</div>
<div> </div>
<div>Tom O'Hara<br>Adjunct Professor<br>Computer Science Department<br>Texas State University<br>2 October 2012</div>
<div> </div>
<div>Idea</div>
<div> </div>
<div>The fallout from the 2008 economic crisis is still being felt, particularly with respect to unemployment. A novel way to reduce unemployment would be to hire up to one million Americans to perform annotations of data in support of intelligent computer applications. For example, the field of natural language processing (NLP) endeavors to get computers to understand English to facilitate intelligent search and other applications. NLP commonly exploits learning by example, a popular technique for artificial intelligence (AI), which is trying to make computers achieve human-level intelligence. A large-scale effort on annotating data to provide training examples for NLP and other areas of AI can be used to spur high-tech innovations (e.g., in support of semantic web). This can be viewed as a stimulus package that promotes both employment and high-tech investment.</div>
<div> </div>
<div>Motivation</div>
<div> </div>
<div>Exploiting human annotations via example-based learning has led to significant advancements in NLP. Sample application areas include the following: 1) grammatical parsing, which derives syntactic parse trees, a generalization of the sentence diagrams used in grade school; 2) word-sense disambiguation, such as in choosing which dictionary sense definition best fits a word in context; and 3) semantic role tagging, which indicates how phrases contribute to sentence meaning (e.g., the who, what, where). Previously, development of such systems would require significant programming and knowledge engineering efforts, often yielding systems tailored to particular domains to achieve better precision. With example-based machine learning, the engineering efforts are focused on extracting features from tagged data. An advantage is that such system can be readily adapted to different domains by using different training data.</div>
<div> </div>
<div>Producing some of the NLP annotation types require technical backgrounds in linguistics as with the parse trees in the Penn Treebank, but word-sense disambiguation (WSD) can readily be done by native English speakers without specialized training. Currently, a small subset of the common English vocabulary has been sufficiently annotated to allow for accurate WSD (roughly 1,000 distinct words[1]), so a large-scale effort is critical to ensure much broader coverage (> 50,000 words). Assuming 10,000 instances for each word with five different annotators per example (for quality assurance purposes), there would need to be over a billion total word-sense annotations for basic coverage of English. Thus, a large-scale annotation effort would be instrumental in achieving this goal.</div>
<div> </div>
<div>Having such a large annotated corpus of word senses would be indispensable for the Semantic Web, a long-term effort now under way to make web pages explicit regarding the entities they discuss. For example, a web page discussing dogs as pets would use internal labels indicating that the web page pertains to Canis familiaris (rather than say hotdogs). These technical usage labels would normally be hidden from end users, who would still be querying using the more natural keyword approach (e.g., simple English phrases like "kid-friendly dog"). Therefore, making full use of the semantic web would require the ability to disambiguate word senses by search engines. For instance, if a web page is explicitly tagged with category labels unrelated to canines, searches for "smart guide dog" should generally omit that page, even if all the keywords matched. The latter could happen in web pages containing slang usages (e.g., "lucky dog").</div>
<div> </div>
<div>Other areas exploiting example-based learning can benefit from having annotations by large numbers of non-technical users. For example, this would make it possible to have detailed topic and mood annotations covering the entire Million Song Dataset, which is popular music information retrieval (MIR) research. Currently, detailed annotations have only been done for datasets involving around 10,000 songs. Having complete annotations for the Million Song Dataset would help maintain the American competitive edge in the burgeoning MIR market. In addition, image annotations can help with computer recognition of objects; and, document annotations can help with text categorization. Two or three such annotation target areas will be selected, based on input from the research community.</div>
<div> </div>
<div>Result</div>
<div> </div>
<div>The end result will be a variety of human tagged datasets that U.S. companies can license for use in R&D. The data can be made available for academic research for modest fees. Higher license fees can be assessed for foreign corporations. (Data protection might be an issue, so that special safeguards might be needed as in reserving certain portions for commercial use by companies meeting strict security safeguarding criteria.)</div>
<div> </div>
<div>Benefits</div>
<div> </div>
<div>- A large number of unskilled workers can be hired to perform annotation tasks just requiring basic reading and analysis skills (e.g., only high school education required).<br>- Welfare recipients can be required to perform annotations in proportion to amount of benefits received, such as full-time work for those receiving equivalent of full-time pay. For example, this would allow a single mother to work at home while still caring for children. (Depending on circumstances, reduced workloads might be allowed.)<br>
- Minimal infrastructure will be required, as many people will already have home computers and internet service, the main requirements for annotation work from home. (The computers do not need to be state of the art, as the annotation software would be run on a server.)</div>
<div> </div>
<div>Proposal chances</div>
<div> </div>
<div>- With unemployment still at high levels (> 8%), this can be attractive to the White House as a novel way to get large numbers of jobs with minimal infrastructure costs (unlike public works).<br>- Partisan politics might preclude the proposal getting immediately approved, but a pilot study might be feasible. For example, $1 billion dollars would allow for about 40,000 annotator jobs.</div>
<div> </div>
<div>Related work</div>
<div> </div>
<div>Several different approaches have been applied to produce word-sense disambiguation annotations. The most common has been the use of professional annotators, notably in support of the Senseval word-sense disambiguation competitions (now Semeval). The approach used is typically to have trained linguists concentrate on specific words, rather than trying to annotate all words in a sentence at the same time as done in earlier annotations (e.g., SemCor for WordNet, the lexicon commonly used in NLP). Other approaches have relied upon on online users to provide ad hoc annotations in the context of interaction with a web service (e.g., language games). Word-sense annotations are also done with respect to FrameNet, which concentrates on semantic role tagging. In addition, there has been work on annotations for the American National Corpus.</div>
<div> </div>
<div>More recently, crowd sourcing has developed into a cost-effective alternative to traditional annotations, such as via Amazon's Mechanical Turk. This involves soliciting humans to perform tasks that computers have difficulty in return for payment upon successful completion. There has been success in certain application areas (e.g., document relevancy judgments for information retrieval). However, large-scale annotations have not been produced for word-sense disambiguation. A drawback of crowd sourcing is that there is no guarantee that the same pool of annotators will be used throughout. In addition, as the payment model typically is based on individual tasks, it is less suitable than traditional annotation in the context of jobs creation.</div>
<div> </div>
<div>Cost estimate</div>
<div> </div>
<div>1. Annotators<br>- rate: $7.50-10/hour (minimum wage is $7.25/hour[2])<br>- wages (52 weeks; 40 hours/week): $15,600-20,800/person<br>- benefits, employer taxes, etc.: $7,000-10,000/person (???)<br>subtotal: Roughly $24 billion for 1M annotators full-time. See below for cost sharing ideas.<br>
note: Some annotators might only be working part-time, in which case a higher wage might be offered to offset lack of benefits.</div>
<div> </div>
<div>2. Low-level Management<br>note: It is unclear how much management would be needed. The following assumes one manager (supervisor) per 100 annotators.<br>todo: rework as range (e.g., $250M - $1B)<br>- rate: $15-20/hour<br>
- wages (52 weeks; 40 hours/week): $31,200-41,600/person<br>- benefits, employer taxes, etc.: $15,000/person (???)<br>- tasks: coordinating annotators; performing quality assurance (QA)<br>subtotal: roughly $500M (1O,000 low-level managers)</div>
<div> </div>
<div>3. Upper Management<br>note: Likewise unsure amount of upper management needed: assuming 1 per 100 low-level ones. Managers should have a strong quantitative background to help in the overall data analysis.<br>- rate: $25-30/hour<br>
- wages (52 weeks; 40 hours/week): $52,000-$62,000/person<br>- benefits, employer taxes, etc.: $20,000/person (???)<br>- tasks: Workload distribution for annotation task; data analysis<br>subtotal: roughly $500K (100 2nd-level)</div>
<div> </div>
<div>4. Research Staff<br>note: Assuming 3 fulltime primary investigators (co-PI's) and 4 halftime graduate assistants (GA) each. todo: add in technical support staff as well; rework along the lines of a large NSF grant<br>
- PI: $100K/per<br>- GA: $40K/per (1/2 mgt [$27.5K + 10K tuition + $2.5K other???])<br>- university costs listed below<br>subtotal: $780K</div>
<div> </div>
<div>5. Infrastructure<br>note: annotators will provide own computer and internet access (there can be some funds set aside to support the indigent)<br>servers (storage and analysis): 10 @ $2K each => $20K<br>admin: $50K (or done by GA's)<br>
university costs (estimated via research staff costs [a la infrastructure as half research grant]): $780K<br>subtotal: $850K</div>
<div> </div>
<div>6. Resource Preparation<br>Certain annotation tasks would require preparation of external resources. For example, with word-sense disambiguation, it is critical that the target sense inventory matches actual word usage in practice. For Senseval, this was achieved by modifying the WordNet lexicon to make the distinctions clearer. In addition, in cases where alternative lexicons were used, mappings into WordNet were developed, as the latter is the most common lexicon used in computational linguistics.<br>
subtotal: $500K ???<br>Total: ~$25B</div>
<div> </div>
<div>Cost sharing ideas</div>
<div> </div>
<div>- Can be required for welfare recipients so existing Health and Human Services budget can be used to cover part of the costs (e.g., 25-50% of total).<br>- Likewise, extended unemployment benefits can be conditional upon annotation work (e.g., 10-25% of total).<br>
- Individual states can be required to fund a percentage of the cost in proportion to number of employees hired there (e.g., 5-10% of total).</div>
<div> </div>
<div>Potential criticisms</div>
<div> </div>
<div>- Naive annotations unreliable.<br> counter: Require multiple annotations per item (e.g., 5 or more).<br>- Niche market so can't recoup costs by licensing.<br> counter: view as infrastructure support (i.e., intangible)<br>
- Fiscal hawks will make it hard to fund entire project ($25B), especially in short term.<br> counter: just getting trial study might be worthwhile in itself<br>- Other researchers might feel cost inordinate. For example, entire NSF budget is $6.9 billion per year[3], and that for the NIH is $26.4 billion per year[4].<br>
counter: small portion actually goes into research</div>
<div>Complications</div>
<div>- Converting welfare recipients to annotators might incur significant cost overhead for other federal departments.<br>- Might be viewed as creating yet another federal bureaucracy (e.g., management structure required).<br>
</div>
<div>Miscellaneous<br></div>
<div>- For the trial, services like eLance or oDesk can be used to supply time-tracking infrastructure. Both have a high surcharge (roughly 10%), so perhaps negotiation can be done to make it more cost effective (less than 5%).<br>
- Another advantage of targeting natural language annotations is that it can help the annotators improve their language skills. Furthermore, it might even interest some of them into pursuing a career in linguistics.</div>
<div> </div>
<div>Footnotes<br></div>
<div>1. See <a href="http://www.senseval.org/data.html" target="_blank">http://www.senseval.org/data.html</a> for a representative sample of word-sense annotation datasets.<br>2. <a href="http://www.dol.gov/dol/topic/wages/minimumwage.htm" target="_blank">www.dol.gov/dol/topic/wages/minimumwage.htm</a><br>
3. <a href="http://en.wikipedia.org/wiki/National_Science_Foundation" target="_blank">http://en.wikipedia.org/wiki/National_Science_Foundation</a><br>4. <a href="http://en.wikipedia.org/wiki/National_Institutes_of_Health" target="_blank">http://en.wikipedia.org/wiki/National_Institutes_of_Health</a><br>
</div></div><br>