We need your help to make sure Turker Nation stays online. Please click the thermometer to learn more.
Note: clicking on the above banners and making ANY purchase returns a commission to Turker Nation.
If you can't see the ad, please click on Shop on Amazon instead. | Want to advertise here? PM Spamgirl to learn more!




Results 1 to 4 of 4

Thread: MTurk Character Misrepresentation: Assessment and Solutions

  1. #1
    Community Manager spamgirl's Avatar
    Join Date
    Aug 2011
    Gender
    Female
    Location
    Canada
    Posts
    12,127
    Thanks
    2,503
    Thanked 6,554 Times in 2,122 Posts

    MTurk Character Misrepresentation: Assessment and Solutions

    This tutorial provides evidence that character misrepresentation in survey screeners by Amazon Mechanical Turk Workers (“Turkers”) can substantially and significantly distort research findings. Using five studies, we demonstrate that a large proportion of respondents in paid MTurk studies claim a false identity, ownership, or activity in order to qualify for a study. The extent of misrepresentation can be unacceptably high, and the responses to subsequent questions can have little correspondence to responses from appropriately identified participants. We recommend a number of remedies to deal with the problem, largely involving strategies to take away the economic motive to misrepresent and to make it difficult for Turkers to recognize that a particular response will gain them access to a study. The major short-run solution involves a two-survey process that first asks respondents to identify their characteristics when there is no motive to deceive, and then limits the second survey to those who have passed this screen. The long-run recommendation involves building an ongoing MTurk participant pool (“panel”) that (1) continuously collects information that could be used to classify respondents, and (2) eliminates from the panel those who misrepresent themselves.

    https://academic.oup.com/jcr/article...dFrom=fulltext
    Last edited by spamgirl; 07-25-2017 at 09:02 PM.
    It's all good.

  2. The Following 2 Users Say Thank You to spamgirl For This Useful Post:


  3. #2
    Moderator bromeliad's Avatar
    Join Date
    Aug 2011
    Gender
    Male
    Location
    Fritz Lang's Metropolis
    Posts
    1,002
    Thanks
    156
    Thanked 3,084 Times in 749 Posts

    Quote Originally Posted by spamgirl View Post
    This tutorial provides evidence that character misrepresentation in survey screeners by Amazon Mechanical Turk Workers (“Turkers”) can substantially and significantly distort research findings. Using five studies, we demonstrate that a large proportion of respondents in paid MTurk studies claim a false identity, ownership, or activity in order to qualify for a study. The extent of misrepresentation can be unacceptably high, and the responses to subsequent questions can have little correspondence to responses from appropriately identified participants. We recommend a number of remedies to deal with the problem, largely involving strategies to take away the economic motive to misrepresent and to make it difficult for Turkers to recognize that a particular response will gain them access to a study. The major short-run solution involves a two-survey process that first asks respondents to identify their characteristics when there is no motive to deceive, and then limits the second survey to those who have passed this screen. The long-run recommendation involves building an ongoing MTurk participant pool (“panel”) that (1) continuously collects information that could be used to classify respondents, and (2) eliminates from the panel those who misrepresent themselves.

    https://oup.silverchair-cdn.com/oup/...CZBIA4LVPAVW3Q
    In case the above link doesn't work https://academic.oup.com/jcr/article...Assessment-and

  4. The Following 2 Users Say Thank You to bromeliad For This Useful Post:


  5. #3
    I wonder how many people are giving them incorrect data on purpose because the pay is too low overall. It infuriates most turkers when academic research papers come out that tout our advanced educations and fantastic work ability all for slave wages. Do you think it could be retaliation? You can't unionize, but you can mess up their data sort of deal? I also wonder if terribly written surveys could be part of the problem.

  6. The Following User Says Thank You to chatter11 For This Useful Post:


  7. #4
    Community Manager spamgirl's Avatar
    Join Date
    Aug 2011
    Gender
    Female
    Location
    Canada
    Posts
    12,127
    Thanks
    2,503
    Thanked 6,554 Times in 2,122 Posts

    Quote Originally Posted by chatter11 View Post
    I wonder how many people are giving them incorrect data on purpose because the pay is too low overall. It infuriates most turkers when academic research papers come out that tout our advanced educations and fantastic work ability all for slave wages. Do you think it could be retaliation? You can't unionize, but you can mess up their data sort of deal? I also wonder if terribly written surveys could be part of the problem.
    I can't say that I haven't done it myself, so it wouldn't surprise me if many others do it. Or just hurry in order to make a not horrible wage. Or lie to get the money.
    It's all good.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •