[Corpora-List] Amazon Mechanical Turk as an evaluation tool

[Kevin B. Cohen]

I believe that Fadi Biadsy et al. used Amazon's Mechanical Turk for part of
their evaluation in Biadsy et al. 2008:

An unsupervised approach to biography production using Wikipedia.

Kevin

On Mon, May 11, 2009 at 12:18 PM, Anna Kazantseva
<ankazant at site.uottawa.ca>wrote:

> Dear Corpora members,
>
>
>
> Have any of you had any experience using the Amazon Mechanical Turk as
> an evaluation tool for relatively complex NLP tasks?
>
> Myself and the research group at the Univeristy of Ottawa are looking at
> using it to evaluate summaries in the context of the Text Analysis
> Conference. I know that several researchers used the Mechanical Turk
> successfully for cognitively simple tasks such as choosing  a correct
> word sense or ordering events.  But I am not aware of any studies where
> subjects would be required to spend significant amount of time to
> complete a single task (15 minutes minutes or more,  although this is
> very tentative). I would appreciate any pointers.
>
> I will post a summary of responses I receive.
>
> Many thanks in advance.
>
> Regards,
>
> Anna
>
> --
> Anna Kazantseva, Ph.D. Candidate
> School of Information Technology and Engineering
> University of Ottawa
> 800 King Edward Ave. Ottawa, Ontario, Canada, K1N 6N5
>
>
> _______________________________________________
> Corpora mailing list
> Corpora at uib.no
> http://mailman.uib.no/listinfo/corpora
>



-- 
K. B. Cohen
Biomedical Text Mining Group Lead, Center for Computational Pharmacology
and
Lead Artificial Intelligence Engineer, The MITRE Corporation, Human Language
Technology Division
303-916-2417 (cell) 303-377-9194 (home)
http://compbio.uchsc.edu/Hunter_lab/Cohen
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listserv.linguistlist.org/pipermail/corpora/attachments/20090511/34306e1b/attachment.htm>
-------------- next part --------------
_______________________________________________
Corpora mailing list
Corpora at uib.no
http://mailman.uib.no/listinfo/corpora