We need help getting enough donations to pay the monthly server bill. Click the thermometer to learn more.
Note: clicking on the above banners and making ANY purchase returns a commission to Turker Nation.
If you can't see the ad, please click on Shop on Amazon instead. | Want to advertise here? PM Spamgirl to learn more!




Results 1 to 7 of 7

Thread: Multi-session experiment as a single HIT? (Researcher new to mechanical turk)

  1. #1

    Multi-session experiment as a single HIT? (Researcher new to mechanical turk)

    Hello,
    I'm a PhD student thinking about conducting a memory experiment employing mTurk workers and have not used mTurk before. The experiment would have three sessions and the second session needs to be approximately 24 hours after the first session and the third session needs to be a week after the second session (e.g., a worker could do 1st session: Monday 4pm, 2nd session: Tuesday 4pm and then the 3rd session: Tuesday the following week). So my question is would there be any issues for making the whole experiment as a single HIT with workers getting paid after they complete the third session?
    (This seems a little different from the past multi-stage studies I've read about on mechanical turk which seem to use multiple HIT's, but paying after the third session is what I've done for offline experiments and seems to make the most sense for online ones as well).

  2. #2
    Quote Originally Posted by Smith View Post
    Hello,
    I'm a PhD student thinking about conducting a memory experiment employing mTurk workers and have not used mTurk before. The experiment would have three sessions and the second session needs to be approximately 24 hours after the first session and the third session needs to be a week after the second session (e.g., a worker could do 1st session: Monday 4pm, 2nd session: Tuesday 4pm and then the 3rd session: Tuesday the following week). So my question is would there be any issues for making the whole experiment as a single HIT with workers getting paid after they complete the third session?
    (This seems a little different from the past multi-stage studies I've read about on mechanical turk which seem to use multiple HIT's, but paying after the third session is what I've done for offline experiments and seems to make the most sense for online ones as well).
    The problem with this is that you would then be expecting a worker to hold a hit in their queue for over a week. There is almost nothing that a turker hates more. You would be better off if you offered a bonus for completing all 3 parts, or increased the pay as an incentive as they went along.

  3. The Following User Says Thank You to tscreener For This Useful Post:


  4. #3
    Moderator Rosie The Robot's Avatar
    Join Date
    May 2015
    Gender
    Female
    Location
    Serving God
    Posts
    3,830
    Thanks
    9,009
    Thanked 7,567 Times in 2,844 Posts

    Quote Originally Posted by tscreener View Post
    The problem with this is that you would then be expecting a worker to hold a hit in their queue for over a week. There is almost nothing that a turker hates more. You would be better off if you offered a bonus for completing all 3 parts, or increased the pay as an incentive as they went along.
    I totally agree. Not only is holding a HIT for a week very inconvenient, it is likely to be accidentally returned. There is no way for us to put it aside to come back to later, it's always there at the front of the queue, getting in the way and slowing us down.

    Since it is important to you that we do all of them, you could make the greater payment after the last one. You'd want to send reminders for that last HIT, which would not be hard to do.

    Good workers will make every effort to do all 3 HITs. In order to make sure you get good workers, you'll probably want to use quals. I recommend you avoid the Masters qual, it costs you more and does nothing special to guarantee good workers. You can be assured of seasoned workers by setting limits such as 5000 approved HITs and >98% approval. (I don't recommend you do >99 because you'd be limited to only workers who never got even one rejection, of which there are very few.)

    Sent by means of Tapatalk
    If Turker Nation has helped you make more money than you would have without it, please consider chipping in to help pay the monthly server bill. When you see the big thermometer above the board, please click on it for more info! Thank you!

  5. #4
    Moderator RippedWarrior's Avatar
    Join Date
    Sep 2011
    Gender
    Male
    Location
    Canada
    Posts
    2,032
    Thanks
    1,539
    Thanked 2,439 Times in 1,018 Posts

    I agree with what the others said regarding a single HIT. It's not reasonable to ask a worker to hold on to a HIT for a week, this would be best published as 3 separate HITs.

    I have published several longitudinal studies myself on MTurk. Worker retention is always an issue. One thing I do is automate the process of sending out reminder messages prior to the release of each HIT.

    You can use your payment process to help with retention as well. As in, pay for the majority of the work after the 3rd HIT. (but don't under-pay on the first 2 HITs!).

    For example, suppose your survey takes 15 minutes to complete. Using federal minimum wage as your guideline, the reward should be approx. $2. So you could set up first HIT for $1, the second for 1$ and the final one for $4 (providing the worker has completed the first 2 HITs). This will incentivise the worker to complete all 3 HITs.

    Good luck!

    Also, check this out for some guidelines and best practices

    http://wiki.wearedynamo.org/index.ph...mic_Requesters
    Eschew obfuscation, espouse elucidation. Batch HITs for newbs. 1000 req'd. 5000 req'd.

  6. The Following User Says Thank You to RippedWarrior For This Useful Post:


  7. #5
    Thank you to everybody for the advice. I'll apply it when I make the study.

  8. The Following User Says Thank You to Smith For This Useful Post:


  9. #6
    Did you ever make this study? I am in the exact same boat and really struggling to understand all of the ins and outs involved in conducting longitudinal research on mTurk. I have been advised to use bonus payments, instead of separate HITs. I am wondering if you got any good advise/guideance. I would certainly pay someone fluent in MTurk to set my study up for me!

  10. #7
    Quote Originally Posted by anchor5898 View Post
    Did you ever make this study? I am in the exact same boat and really struggling to understand all of the ins and outs involved in conducting longitudinal research on mTurk. I have been advised to use bonus payments, instead of separate HITs. I am wondering if you got any good advise/guideance. I would certainly pay someone fluent in MTurk to set my study up for me!
    Hello:
    Can you contact @spamgirl and get set up on the board as a requester? She might be a good resource to help you out with some of these questions also.

  11. The Following 2 Users Say Thank You to tscreener For This Useful Post:


Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •