We need your help to make sure Turker Nation stays online. Please click the thermometer to learn more.
Note: clicking on the above banners and making ANY purchase returns a commission to Turker Nation.
If you can't see the ad, please click on Shop on Amazon instead. | Want to advertise here? PM Spamgirl to learn more!




Results 1 to 10 of 12

Thread: The Myth of Low Cost, High Quality on Amazon's Mechanical Turk

Threaded View

  1. #1
    Community Manager spamgirl's Avatar
    Join Date
    Aug 2011
    Gender
    Female
    Location
    Canada
    Posts
    12,089
    Thanks
    2,502
    Thanked 6,550 Times in 2,118 Posts

    Exclamation The Myth of Low Cost, High Quality on Amazon's Mechanical Turk

    "So far, I've earned $8.06 for almost four hours of Turking."
    http://www.boston.com/business/techn...borer_exp.html

    "... but all of it is characterized by rates of pay that are usually well below the U.S. minimum wage of $7.25 an hour."
    http://www.huffingtonpost.com/julian...b_2687431.html

    "Estimates of what workers can earn on these crowdsourced tasks range from about $1.20 to $5 an hour without any benefits."
    http://economix.blogs.nytimes.com/20...ype=blogs&_r=0

    "Some requesters aim to pay close to the equivalent of minimum wage (e.g., $1 for a 10-minute assignment). Other requesters opt to pay significantly less than this, as low as 50 cents per hour. We have found that approximately 75 cents is a reasonable rate for a 30-minute survey, though if you need to collect data very quickly, or have a complex task or study, then consider paying more per HIT."
    http://www.siop.org/tip/oct11/03barger.aspx

    "The first wave of participants included 530 participants, who were paid $0.75 for approximately 20 minutes. This rate of pay ($2.25 per hour) is above average for MTurk HITs; the median hourly wage for tasks performed on MTurk is $1.38 (Horton & Chilton, 2010)"
    http://cpx.sagepub.com/content/early...612469015.full

    The refrain is the same - Turkers are being paid slave wages! Requesters, come join in this free-for-all and post your work for $1 per hour! Over and over again, both in academic papers and newspapers, we're told that mTurk is a cheap platform for getting work done. Sounds great, but have you considered that maybe this isn't actually the case if you want high quality work done?

    If you dig down into these studies and articles, you'll find the secret: sure, if you pay $1 the work will get done, but it will be completed by scammers using "bots" (automated answering systems), people who don't necessarily understand the instructions (as English is not their strongest language), or those who don't care about the quality of the work they complete. That's fine for simple work that doesn't require mastery of the English language, but every HIT is posted with the requirement that the answers provided actually offer what is being asked.

    An excellent article (http://turkrequesters.blogspot.ca/20...ical-turk.html) touches on this issue in order to provide requesters with the truth about mTurk. A "constant stream of complaints to Amazon about the poor quality of work from international IP addresses" has caused Amazon to block anyone from outside the US to register for mTurk in the last few years. That means that those who can afford to work for so little are blocked from the platform, therefore when you pay slave wages you are now doing so to Americans - your neighbors, your friends, your family.

    There is no point in spending money on using mTurk if all you're going to get back is crummy work. In fact, some requesters actually build in a "buffer" to compensate for running their HITs twice, three times or even more just to make up for the low quality answers they receive at a low price point. If they instead doubled their HIT pay and used a qualification test, they'd end up SAVING money.

    In fact, studies which focus on getting quality out of your workers are emphatic that pay = quality:

    "Payments on mTurk are suggested to follow a reasonable hourly rate, with an example of $8 per hour or about 13c per minute [2]. In practice, many mTurk tasks pay much less overall, with the median study paying just 5-10c for a task taking "a few minutes," like watching and providing feedback on 3 short (15-second) videos, summarizing a website, and evaluating hypothetical and real market products. Indeed, "wages" this low have been shown to result in lower quality output than could be had for no payment at all, by pure volunteers [3]. "
    http://lorrie.cranor.org/pubs/note1552-downs.pdf

    And that fair pay=getting the work done quickly:

    "Services such as Qualtrics or Survey Monkey commonly charge a researcher $5-$10 for the completion of a ten-minute survey. In contrast, a higher-end worker on MTurk can expect to make $6-$12 per hour, which means the requester should only have to pay $1-$2 for the completion of a ten-minute survey. Despite the lower pay rate, finding willing survey takers is not a major issue. A requester that is offering $1.00 for the completion of 50 ten-minute surveys should expect to have that request filled within a few days."
    http://library.constantcontact.com/d...ather+Data.pdf

    And that there is a difference between high quality and low quality workers:

    "New Turkers and Turkers who don't meet the strict filters can be paid less, but most of my high-quality workers expect to make about $8-14 an hour. "
    http://blog.echen.me/2012/04/25/maki...est-practices/

    You have to pay for quality. Whether you have people writing articles, categorizing photographs or answering a survey, paying less than minimum wage will garner you less than high quality results. This means you will have to take the time to double check every HIT submitted to ensure it is up to par for your project. Time is money and you can easily save both by paying fairly and using a qualification test. If you want to learn more about some best practices for Requesters, which both encourage Turkers to do your HITS and ensure your results will be top notch, check out this post: http://turkrequesters.blogspot.ca/20...-on-mturk.html and this post: http://blog.echen.me/2012/04/25/maki...est-practices/ and the Guidelines for Academic Requesters.
    It's all good.

  2. The Following 23 Users Say Thank You to spamgirl For This Useful Post:

    + Show/Hide list of Thanks


Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •