We need help getting enough donations to pay the monthly server bill. Click the thermometer to learn more.
Note: clicking on the above banners and making ANY purchase returns a commission to Turker Nation.
If you can't see the ad, please click on Shop on Amazon instead. | Want to advertise here? PM Spamgirl to learn more!




Results 1 to 3 of 3

Thread: Setting a HIT to get feedback for a HIT?

  1. #1

    Setting a HIT to get feedback for a HIT?

    Hope you guys are doing well!

    I am conducting a survey on Life Satisfaction for my research, which takes about 10 minutes to complete ($1.2 pay). I set the HIT Approval Rate (%) for all Requesters' HITs greater than 98, and the Number of HITs Approved greater than 10000.

    With the above setup, I think I was able to attract great workers who are willing to give quality responses for the survey. The problem I am having is that perhaps the survey design was too complicated (or I am providing too much information for the workers to analyze). For the first few questions, I am getting good responses, but for the later parts of the survey, I can see that the workers are speeding through just to complete the survey.

    Questions:

    1. Do you think it is worthwhile to set up a HIT and ask workers directly to evaluate the survey (what needs to be change? Is it too long? Anything not clear?) I will not require them to complete the entire survey, but I will need their feedback.

    2. For those of you who have experience working on online surveys conducted on Mturk, are they usually a short questionnaire that are relatively fast and simple to complete? What are some of the things that would make you "speed through and just get it done".

    Thanks!

  2. #2
    Community Manager spamgirl's Avatar
    Join Date
    Aug 2011
    Gender
    Female
    Location
    Canada
    Posts
    13,122
    Thanks
    2,842
    Thanked 7,337 Times in 2,375 Posts

    Quote Originally Posted by Henri_F View Post
    1. Do you think it is worthwhile to set up a HIT and ask workers directly to evaluate the survey (what needs to be change? Is it too long? Anything not clear?) I will not require them to complete the entire survey, but I will need their feedback.
    Why not just do that here? Have some of the workers here take a look, then pay them through a HIT for their time. Then you're guaranteed to have people doing it who will put in the effort to provide you with good insight.

    2. For those of you who have experience working on online surveys conducted on Mturk, are they usually a short questionnaire that are relatively fast and simple to complete? What are some of the things that would make you "speed through and just get it done".
    The longer it is, the more you'll have to pay. What you could try is running two surveys: the first is just the first few questions they're actually trying on, and it pays as much as the time taken to complete those questions alone. Then post a second survey for which those who complete the first would receive a qualification for later, so by the time they come back, they're rested. You may not have as many people completing both necessarily, but the break would likely improve the quality of the second set of questions.

    With all of that said, 10 minutes is NOT a long survey. We do hour-long surveys (if the pay is right). My guess is at $7.20/hour you're just not attracting the sorts of workers who are willing to put in their all for 10 minutes. The best workers are either slow and conscientious, therefore they think that the survey might take them more than 10 minutes because of the effort they put in and therefore they won't do it, or are quick and confident, but expect a higher rate of pay. Also, many top Turkers don't do surveys because the pay is so low, so they may never have even looked at it in the first place.

    My recommendation based on all of the above is to offer a bonus to workers who clearly put in effort throughout the survey. State that clearly in both the HIT and the survey, both at the beginning of the survey (on it's own page, none of us read IC pages), and at the point of the survey where people are losing their steam. Something like, "You're halfway done! Remember, if you continue putting in the concerted effort you have so far, I'll be sending you a $1 bonus, which will almost double your hourly wage. I appreciate how hard you're working on this survey, and thank you so much for your time." Coming from a psych background (and I'm not sure what your field is, so this might just be old hat to you), this idea that the Requester is going to double your pay creates reciprocation guilt which will lead many (not all) participants to feel they have to keep working hard in return for your generosity. I don't consider this trickery or duplicity as the expectation of all of us is that if we choose to do a HIT, we do it well. It's just reminding workers of this expectation.
    It's all good.

  3. #3
    Thanks for the great feedback I think it is a great idea, I will post the link of the HIT in the "Survey Area" later today.

    Yes, I think 10 minutes is not that long, but the effort that is required is more than some of the surveys I have tried myself on Mturk. I will also try your suggestions by giving a bonus to the workers midway through the survey.

  4. The Following User Says Thank You to Henri_F For This Useful Post:


Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •