[Corpora-List] Amazon Mechanical Turk as an evaluation tool

[Anna Kazantseva]

Dear Corpora members,

 

Have any of you had any experience using the Amazon Mechanical Turk as 
an evaluation tool for relatively complex NLP tasks?

Myself and the research group at the Univeristy of Ottawa are looking at 
using it to evaluate summaries in the context of the Text Analysis 
Conference. I know that several researchers used the Mechanical Turk 
successfully for cognitively simple tasks such as choosing  a correct 
word sense or ordering events.  But I am not aware of any studies where 
subjects would be required to spend significant amount of time to 
complete a single task (15 minutes minutes or more,  although this is 
very tentative). I would appreciate any pointers.

I will post a summary of responses I receive.

Many thanks in advance.

Regards,

Anna

-- 
Anna Kazantseva, Ph.D. Candidate
School of Information Technology and Engineering
University of Ottawa
800 King Edward Ave. Ottawa, Ontario, Canada, K1N 6N5


_______________________________________________
Corpora mailing list
Corpora at uib.no
http://mailman.uib.no/listinfo/corpora