[Corpora-List] Amazon Mechanical Turk as an evaluation tool

[Kevin B. Cohen]

Sorry, I should have noted that this was a paper on summarization.

Kevin

On Mon, May 11, 2009 at 12:58 PM, Kevin B. Cohen <kevin.cohen at gmail.com>wrote:

> I believe that Fadi Biadsy et al. used Amazon's Mechanical Turk for part of
> their evaluation in Biadsy et al. 2008:
>
> An unsupervised approach to biography production using Wikipedia.
>
> Kevin
>
>
> On Mon, May 11, 2009 at 12:18 PM, Anna Kazantseva <
> ankazant at site.uottawa.ca> wrote:
>
>> Dear Corpora members,
>>
>>
>>
>> Have any of you had any experience using the Amazon Mechanical Turk as
>> an evaluation tool for relatively complex NLP tasks?
>>
>> Myself and the research group at the Univeristy of Ottawa are looking at
>> using it to evaluate summaries in the context of the Text Analysis
>> Conference. I know that several researchers used the Mechanical Turk
>> successfully for cognitively simple tasks such as choosing  a correct
>> word sense or ordering events.  But I am not aware of any studies where
>> subjects would be required to spend significant amount of time to
>> complete a single task (15 minutes minutes or more,  although this is
>> very tentative). I would appreciate any pointers.
>>
>> I will post a summary of responses I receive.
>>
>> Many thanks in advance.
>>
>> Regards,
>>
>> Anna
>>
>> --
>> Anna Kazantseva, Ph.D. Candidate
>> School of Information Technology and Engineering
>> University of Ottawa
>> 800 King Edward Ave. Ottawa, Ontario, Canada, K1N 6N5
>>
>>
>> _______________________________________________
>> Corpora mailing list
>> Corpora at uib.no
>> http://mailman.uib.no/listinfo/corpora
>>
>
>
>
> --
> K. B. Cohen
> Biomedical Text Mining Group Lead, Center for Computational Pharmacology
> and
> Lead Artificial Intelligence Engineer, The MITRE Corporation, Human
> Language Technology Division
> 303-916-2417 (cell) 303-377-9194 (home)
> http://compbio.uchsc.edu/Hunter_lab/Cohen
>
>


-- 
K. B. Cohen
Biomedical Text Mining Group Lead, Center for Computational Pharmacology
and
Lead Artificial Intelligence Engineer, The MITRE Corporation, Human Language
Technology Division
303-916-2417 (cell) 303-377-9194 (home)
http://compbio.uchsc.edu/Hunter_lab/Cohen
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listserv.linguistlist.org/pipermail/corpora/attachments/20090511/3199bc7b/attachment.htm>
-------------- next part --------------
_______________________________________________
Corpora mailing list
Corpora at uib.no
http://mailman.uib.no/listinfo/corpora