[Corpora-List] Corpora Digest, Vol 23, Issue 12

[Steve DeRose]

>Date: Mon, 11 May 2009 14:18:00 -0400
>From: Anna Kazantseva <ankazant at site.uottawa.ca>
>Subject: [Corpora-List] Amazon Mechanical Turk as an evaluation tool
>To: corpora at uib.no

Several colleagues and I have used Mechanical Turk and found it useful,
though haven't tried it for large-scale tasks yet. I have seen large-scale
tasks advertised on mturk, though. One thing I would be especially careful
of, is tracking how much time volunteers spend on your task. We found a few
volunteers who obviously didn't do the task at all; but more interesting was
that on average people spent less than half the time we expected them to
need just to read the text to be evaluated. So count on people skimming
quickly and responding with only their first impression, unless you can
design some controls to somehow prevent that. The problem of overly-hurried
responses may well grow with task size.

Steve DeRose
Director of Linguistics
Hapax LLC





_______________________________________________
Corpora mailing list
Corpora at uib.no
http://mailman.uib.no/listinfo/corpora