Note: clicking on the above banners and making ANY purchase returns a commission to Turker Nation.
If you can't see the ad, please click on Shop on Amazon instead. | Want to advertise here? PM Spamgirl to learn more!




Page 1 of 2 12 LastLast
Results 1 to 10 of 12

Thread: The Myth of Low Cost, High Quality on Amazon's Mechanical Turk

  1. #1
    Community Manager spamgirl's Avatar
    Join Date
    Aug 2011
    Gender
    Female
    Location
    Canada
    Posts
    12,493
    Thanks
    2,621
    Thanked 6,852 Times in 2,213 Posts

    Exclamation The Myth of Low Cost, High Quality on Amazon's Mechanical Turk

    "So far, I've earned $8.06 for almost four hours of Turking."
    http://www.boston.com/business/techn...borer_exp.html

    "... but all of it is characterized by rates of pay that are usually well below the U.S. minimum wage of $7.25 an hour."
    http://www.huffingtonpost.com/julian...b_2687431.html

    "Estimates of what workers can earn on these crowdsourced tasks range from about $1.20 to $5 an hour without any benefits."
    http://economix.blogs.nytimes.com/20...ype=blogs&_r=0

    "Some requesters aim to pay close to the equivalent of minimum wage (e.g., $1 for a 10-minute assignment). Other requesters opt to pay significantly less than this, as low as 50 cents per hour. We have found that approximately 75 cents is a reasonable rate for a 30-minute survey, though if you need to collect data very quickly, or have a complex task or study, then consider paying more per HIT."
    http://www.siop.org/tip/oct11/03barger.aspx

    "The first wave of participants included 530 participants, who were paid $0.75 for approximately 20 minutes. This rate of pay ($2.25 per hour) is above average for MTurk HITs; the median hourly wage for tasks performed on MTurk is $1.38 (Horton & Chilton, 2010)"
    http://cpx.sagepub.com/content/early...612469015.full

    The refrain is the same - Turkers are being paid slave wages! Requesters, come join in this free-for-all and post your work for $1 per hour! Over and over again, both in academic papers and newspapers, we're told that mTurk is a cheap platform for getting work done. Sounds great, but have you considered that maybe this isn't actually the case if you want high quality work done?

    If you dig down into these studies and articles, you'll find the secret: sure, if you pay $1 the work will get done, but it will be completed by scammers using "bots" (automated answering systems), people who don't necessarily understand the instructions (as English is not their strongest language), or those who don't care about the quality of the work they complete. That's fine for simple work that doesn't require mastery of the English language, but every HIT is posted with the requirement that the answers provided actually offer what is being asked.

    An excellent article (http://turkrequesters.blogspot.ca/20...ical-turk.html) touches on this issue in order to provide requesters with the truth about mTurk. A "constant stream of complaints to Amazon about the poor quality of work from international IP addresses" has caused Amazon to block anyone from outside the US to register for mTurk in the last few years. That means that those who can afford to work for so little are blocked from the platform, therefore when you pay slave wages you are now doing so to Americans - your neighbors, your friends, your family.

    There is no point in spending money on using mTurk if all you're going to get back is crummy work. In fact, some requesters actually build in a "buffer" to compensate for running their HITs twice, three times or even more just to make up for the low quality answers they receive at a low price point. If they instead doubled their HIT pay and used a qualification test, they'd end up SAVING money.

    In fact, studies which focus on getting quality out of your workers are emphatic that pay = quality:

    "Payments on mTurk are suggested to follow a reasonable hourly rate, with an example of $8 per hour or about 13c per minute [2]. In practice, many mTurk tasks pay much less overall, with the median study paying just 5-10c for a task taking "a few minutes," like watching and providing feedback on 3 short (15-second) videos, summarizing a website, and evaluating hypothetical and real market products. Indeed, "wages" this low have been shown to result in lower quality output than could be had for no payment at all, by pure volunteers [3]. "
    http://lorrie.cranor.org/pubs/note1552-downs.pdf

    And that fair pay=getting the work done quickly:

    "Services such as Qualtrics or Survey Monkey commonly charge a researcher $5-$10 for the completion of a ten-minute survey. In contrast, a higher-end worker on MTurk can expect to make $6-$12 per hour, which means the requester should only have to pay $1-$2 for the completion of a ten-minute survey. Despite the lower pay rate, finding willing survey takers is not a major issue. A requester that is offering $1.00 for the completion of 50 ten-minute surveys should expect to have that request filled within a few days."
    http://library.constantcontact.com/d...ather+Data.pdf

    And that there is a difference between high quality and low quality workers:

    "New Turkers and Turkers who don't meet the strict filters can be paid less, but most of my high-quality workers expect to make about $8-14 an hour. "
    http://blog.echen.me/2012/04/25/maki...est-practices/

    You have to pay for quality. Whether you have people writing articles, categorizing photographs or answering a survey, paying less than minimum wage will garner you less than high quality results. This means you will have to take the time to double check every HIT submitted to ensure it is up to par for your project. Time is money and you can easily save both by paying fairly and using a qualification test. If you want to learn more about some best practices for Requesters, which both encourage Turkers to do your HITS and ensure your results will be top notch, check out this post: http://turkrequesters.blogspot.ca/20...-on-mturk.html and this post: http://blog.echen.me/2012/04/25/maki...est-practices/ and the Guidelines for Academic Requesters.
    It's all good.

  2. The Following 23 Users Say Thank You to spamgirl For This Useful Post:

    + Show/Hide list of Thanks


  3. #2
    Super Sized Member
    Join Date
    Aug 2011
    Gender
    Other
    Posts
    3,379
    Thanks
    2,102
    Thanked 4,396 Times in 1,503 Posts

    Spamgirl, can you make this public plox ?

  4. The Following User Says Thank You to taintturk For This Useful Post:


  5. #3
    Member djjustice24's Avatar
    Join Date
    Dec 2013
    Gender
    Male
    Location
    Los Angeles CA
    Posts
    228
    Thanks
    776
    Thanked 195 Times in 102 Posts

    Awesome writing Spamgirl this is a great defense of all of us turkers out here just trying to make a decent wage on Mturk. Thank you SO MUCH for this!

  6. #4
    Member Dreadnought's Avatar
    Join Date
    Dec 2013
    Gender
    Male
    Location
    The International Space Station
    Posts
    11
    Thanks
    5
    Thanked 8 Times in 7 Posts

    What will take for requesters to take notice? As long as there is low quality workers and bots that suck up the low paying HITs, there will always be requesters that create low paying HITs. The only way that I think it will change is if Amazon steps in and mandates some sort of minimum pay. For example

    $ 0.01 HITs cannot take longer than 5 seconds to complete
    $ 0.02 HITs cannot take longer than 10 seconds
    $ 0.03 HITs cannot take longer than 15 seconds

    And so forth. At least that will average 12 cents a minute or $7.20 an hour but then how many requesters will that drive away.


    You may be a doctor. But I'm the Doctor. The definite article, you might say.


  7. The Following 2 Users Say Thank You to Dreadnought For This Useful Post:


  8. #5
    Member thegrinder's Avatar
    Join Date
    Oct 2013
    Gender
    Female
    Location
    North Carolina
    Posts
    592
    Thanks
    2,737
    Thanked 1,665 Times in 513 Posts

    Quote Originally Posted by Dreadnought View Post
    What will take for requesters to take notice? As long as there is low quality workers and bots that suck up the low paying HITs, there will always be requesters that create low paying HITs. The only way that I think it will change is if Amazon steps in and mandates some sort of minimum pay. For example

    $ 0.01 HITs cannot take longer than 5 seconds to complete
    $ 0.02 HITs cannot take longer than 10 seconds
    $ 0.03 HITs cannot take longer than 15 seconds

    And so forth. At least that will average 12 cents a minute or $7.20 an hour but then how many requesters will that drive away.
    Nice idea, but some turkers are slower than others, e.g. I'm a fast typist & you may not be (not saying that's the case, just hypothesizing), so do you get paid more than me because I get the task done more quickly?

    Ideally, educating requesters that if they want top notch work they have to pay more and have some sort of qualification for their HIT, is the way to go but sadly Amazon aren't about to tell new (or old) requesters that there are a bunch of shite workers out there and they need to be weeded out!!

  9. #6
    Community Manager spamgirl's Avatar
    Join Date
    Aug 2011
    Gender
    Female
    Location
    Canada
    Posts
    12,493
    Thanks
    2,621
    Thanked 6,852 Times in 2,213 Posts

    The issue at hand is educating requesters - they come to mTurk thinking they can pay $1 per hour, since the academics and articles told them so, and then are shocked - *shocked* - when the quality of the work they get back is ridiculously low. If they went into mTurk knowing that you get what you pay for, they'd be less likely to leave when they get those subpar results. If they were directed to the community to run a poll on what they could ask for their task in order to get good workers (those willing to join the community and participate therein) then they would a) get good work and end up more likely to stick around to post more work, and b) make Turkers happy people. The problem is that Amazon doesn't facilitate an easy way for us to talk to requesters, so we can't get in touch with new people since we can't tell who they are.
    It's all good.

  10. The Following User Says Thank You to spamgirl For This Useful Post:


  11. #7
    I have noticed surveys putting in control questions. I can imagine they get a lot of bogus stuff in there. However, I agree decent work takes a decent wage.

  12. #8
    Member
    Join Date
    Aug 2011
    Gender
    Female
    Location
    San Diego
    Posts
    22
    Thanks
    37
    Thanked 37 Times in 15 Posts

    This is a great compilation. If it would help, I could post a link to this from the Turkopticon and my twitter. I don't know if you want people directed here... It would be good for academics who are interested in these issues to read this thread about the perils of well-intentioned dumb surveys. (Btw, we're taking down the Ross Silberman Irani Zaldivar survey paper that has one of these $1-$2 stats from all the co-author personal websites where it appeared and adding a note that the results don't reflect the realities of Turking. Spamgirl explained the problems that such stats cause recently and I'm sorry I was so slow to get it.)

    Or maybe that turkrequesters blog taint runs could x-post it. In any case, let me know if I can be any help in disseminating it a bit further out...

  13. The Following User Says Thank You to differenceengines For This Useful Post:


  14. #9
    Researcher
    Join Date
    Dec 2011
    Gender
    Male
    Posts
    21
    Thanks
    19
    Thanked 48 Times in 17 Posts

    I have added a link to this page to the front page of the Turkopticon site, under a new category called "Notes for requesters". If anyone has better ideas (or potential problems with doing that that I didn't think of), please let me know.

  15. #10
    *** Forum Supporter *** moffett's Avatar
    Join Date
    Mar 2012
    Gender
    Female
    Location
    Sherman Oaks, CA
    Posts
    655
    Thanks
    355
    Thanked 623 Times in 252 Posts

    It would be nice if the requestors knew that so many of us, particularly those on this forum, have college degrees and full time jobs. We are doing this low paying work as a way to earn some supplemental income. I also have great admiration for the workers who Turk full time and produce extremely good work.

  16. The Following 2 Users Say Thank You to moffett For This Useful Post:


Page 1 of 2 12 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •