[Corpora-List] Amazon Mechanical Turk as an evaluation tool

[Anna Kazantseva]

Dear Corpora members,

Here’s the summary of responses I received regarding a question about  
using the Amazon Mechanical Turk. Thanks to everyone who has helped me.

Regards,

Anna Kazantseva

1. Fadi Biadsy, Julia Hirschberg, Elena Filatova. "An Unsupervised  
Approach to Biography Production using Wikipedia", ACL-08: HLT,  
Columbus, Ohio, June 2008.


2. Fast. Cheap and Creative: Evaluating Translation Quality Using  
Amazon’s Mechanical Turk. Chris Callison-Burch. To be published.

Regards,

Anna Kazantseva

On 11-May-09, at 2:18 PM, Anna Kazantseva wrote:

> Dear Corpora members,
>
>
>
> Have any of you had any experience using the Amazon Mechanical Turk as
> an evaluation tool for relatively complex NLP tasks?
>
> Myself and the research group at the Univeristy of Ottawa are  
> looking at
> using it to evaluate summaries in the context of the Text Analysis
> Conference. I know that several researchers used the Mechanical Turk
> successfully for cognitively simple tasks such as choosing  a correct
> word sense or ordering events.  But I am not aware of any studies  
> where
> subjects would be required to spend significant amount of time to
> complete a single task (15 minutes minutes or more,  although this is
> very tentative). I would appreciate any pointers.
>
> I will post a summary of responses I receive.
>
> Many thanks in advance.
>
> Regards,
>
> Anna
>
> -- 
> Anna Kazantseva, Ph.D. Candidate
> School of Information Technology and Engineering
> University of Ottawa
> 800 King Edward Ave. Ottawa, Ontario, Canada, K1N 6N5
>
>
> _______________________________________________
> Corpora mailing list
> Corpora at uib.no
> http://mailman.uib.no/listinfo/corpora

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listserv.linguistlist.org/pipermail/corpora/attachments/20090518/923ed4a3/attachment.htm>
-------------- next part --------------
_______________________________________________
Corpora mailing list
Corpora at uib.no
http://mailman.uib.no/listinfo/corpora