We need your help to make sure Turker Nation stays online. Please click the thermometer to learn more.
Note: clicking on the above banners and making ANY purchase returns a commission to Turker Nation.
If you can't see the ad, please click on Shop on Amazon instead. | Want to advertise here? PM Spamgirl to learn more!




Results 1 to 2 of 2

Thread: MTurk Character Misrepresentation: Assessment and Solutions

  1. #1
    Community Manager spamgirl's Avatar
    Join Date
    Aug 2011
    Gender
    Female
    Location
    Canada
    Posts
    11,830
    Thanks
    2,472
    Thanked 6,455 Times in 2,088 Posts

    MTurk Character Misrepresentation: Assessment and Solutions

    This tutorial provides evidence that character misrepresentation in survey screeners by Amazon Mechanical Turk Workers (“Turkers”) can substantially and significantly distort research findings. Using five studies, we demonstrate that a large proportion of respondents in paid MTurk studies claim a false identity, ownership, or activity in order to qualify for a study. The extent of misrepresentation can be unacceptably high, and the responses to subsequent questions can have little correspondence to responses from appropriately identified participants. We recommend a number of remedies to deal with the problem, largely involving strategies to take away the economic motive to misrepresent and to make it difficult for Turkers to recognize that a particular response will gain them access to a study. The major short-run solution involves a two-survey process that first asks respondents to identify their characteristics when there is no motive to deceive, and then limits the second survey to those who have passed this screen. The long-run recommendation involves building an ongoing MTurk participant pool (“panel”) that (1) continuously collects information that could be used to classify respondents, and (2) eliminates from the panel those who misrepresent themselves.

    https://oup.silverchair-cdn.com/oup/...CZBIA4LVPAVW3Q
    It's all good.

  2. The Following User Says Thank You to spamgirl For This Useful Post:


  3. #2
    Moderator bromeliad's Avatar
    Join Date
    Aug 2011
    Gender
    Male
    Location
    Fritz Lang's Metropolis
    Posts
    979
    Thanks
    155
    Thanked 2,982 Times in 725 Posts

    Quote Originally Posted by spamgirl View Post
    This tutorial provides evidence that character misrepresentation in survey screeners by Amazon Mechanical Turk Workers (“Turkers”) can substantially and significantly distort research findings. Using five studies, we demonstrate that a large proportion of respondents in paid MTurk studies claim a false identity, ownership, or activity in order to qualify for a study. The extent of misrepresentation can be unacceptably high, and the responses to subsequent questions can have little correspondence to responses from appropriately identified participants. We recommend a number of remedies to deal with the problem, largely involving strategies to take away the economic motive to misrepresent and to make it difficult for Turkers to recognize that a particular response will gain them access to a study. The major short-run solution involves a two-survey process that first asks respondents to identify their characteristics when there is no motive to deceive, and then limits the second survey to those who have passed this screen. The long-run recommendation involves building an ongoing MTurk participant pool (“panel”) that (1) continuously collects information that could be used to classify respondents, and (2) eliminates from the panel those who misrepresent themselves.

    https://oup.silverchair-cdn.com/oup/...CZBIA4LVPAVW3Q
    In case the above link doesn't work https://academic.oup.com/jcr/article...Assessment-and

  4. The Following User Says Thank You to bromeliad For This Useful Post:


Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •