<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
<!--StartFragment--> <div class="MsoNormal">Dear Corpora members,</div> <div class="MsoNormal"> <o:p></o:p></div> <div class="MsoNormal">Here’s the summary of responses I received regarding a question about using the Amazon Mechanical Turk. Thanks to everyone who has helped me.</div> <div class="MsoNormal"> <o:p></o:p></div> <div class="MsoNormal">Regards,</div> <div class="MsoNormal"> <o:p></o:p></div> <div class="MsoNormal">Anna Kazantseva</div> <div class="MsoNormal"> <o:p></o:p></div><p class="MsoNormal" style="mso-pagination:none;mso-layout-grid-align:none; text-autospace:none"><span style="font-family:Helvetica">1.</span><span style=""><font class="Apple-style-span" face="Arial"> </font></span><span style="color: black; "><font class="Apple-style-span" face="Arial">Fadi Biadsy, Julia Hirschberg, Elena Filatova. "An Unsupervised Approach to Biography Production using Wikipedia", ACL-08: HLT, Columbus, Ohio, June 2008</font></span><span style="font-size:13.0pt;font-family:"Lucida Grande";color:black">.</span><span style="font-family:Helvetica"><o:p></o:p></span></p> <div class="MsoNormal"><span style="font-family:Helvetica"> <o:p></o:p></span></div> <div class="MsoNormal"><span style="font-family:Helvetica">2. Fast. Cheap and Creative: Evaluating Translation Quality Using Amazon’s Mechanical Turk. Chris Callison-Burch. To be published.<o:p></o:p></span></div><div class="MsoNormal"><br class="webkit-block-placeholder"></div><div class="MsoNormal">Regards,</div><div class="MsoNormal"><br class="webkit-block-placeholder"></div><div class="MsoNormal">Anna Kazantseva</div><div class="MsoNormal"><br class="webkit-block-placeholder"></div> <!--EndFragment--> <div><div>On 11-May-09, at 2:18 PM, Anna Kazantseva wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Dear Corpora members,</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Have any of you had any experience using the Amazon Mechanical Turk as<span class="Apple-converted-space"> </span></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">an evaluation tool for relatively complex NLP tasks?</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Myself and the research group at the Univeristy of Ottawa are looking at<span class="Apple-converted-space"> </span></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">using it to evaluate summaries in the context of the Text Analysis<span class="Apple-converted-space"> </span></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Conference. I know that several researchers used the Mechanical Turk<span class="Apple-converted-space"> </span></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">successfully for cognitively simple tasks such as choosing<span class="Apple-converted-space"> </span>a correct<span class="Apple-converted-space"> </span></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">word sense or ordering events.<span class="Apple-converted-space"> </span>But I am not aware of any studies where<span class="Apple-converted-space"> </span></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">subjects would be required to spend significant amount of time to<span class="Apple-converted-space"> </span></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">complete a single task (15 minutes minutes or more,<span class="Apple-converted-space"> </span>although this is<span class="Apple-converted-space"> </span></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">very tentative). I would appreciate any pointers.</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">I will post a summary of responses I receive.</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Many thanks in advance.</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Regards,</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Anna</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">--<span class="Apple-converted-space"> </span></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Anna Kazantseva, Ph.D. Candidate</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">School of Information Technology and Engineering</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">University of Ottawa</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">800 King Edward Ave. Ottawa, Ontario, Canada, K1N 6N5</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">_______________________________________________</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Corpora mailing list</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><a href="mailto:Corpora@uib.no">Corpora@uib.no</a></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><a href="http://mailman.uib.no/listinfo/corpora">http://mailman.uib.no/listinfo/corpora</a></div> </blockquote></div><br></body></html>