Note: clicking on the above banners and making ANY purchase returns a commission to Turker Nation.
If you can't see the ad, please click on Shop on Amazon instead. | Want to advertise here? PM Spamgirl to learn more!




Page 2 of 2 FirstFirst 12
Results 11 to 20 of 20

Thread: Excluding workers (from retaking surveys, doing too many HITs, etc.)

  1. #11
    Community Manager spamgirl's Avatar
    Join Date
    Aug 2011
    Gender
    Female
    Location
    Canada
    Posts
    12,554
    Thanks
    2,636
    Thanked 6,885 Times in 2,221 Posts

    It isn't correct, but Amazon CS perpetuates the myth when Turkers ask. Amazon is manually checking people with multiple blocks before they're suspended now, so it's not such a big deal to get a few blocks.

    That said, blocking isn't the best technique for stopping people from doing your work as it COULD potentially lead to someone being suspended.
    It's all good.

  2. #12
    Community Manager spamgirl's Avatar
    Join Date
    Aug 2011
    Gender
    Female
    Location
    Canada
    Posts
    12,554
    Thanks
    2,636
    Thanked 6,885 Times in 2,221 Posts

    Researchers: How to prevent retakes on your surveys without risking worker accounts

    Preventing retakes on your survey is the most common issue that requesters have with mTurk. This guide explains an easy way to prevent retakes just by using the Amazon mTurk Command Line Tools (CLT) that work on Windows, OS X and GNU/Linux.

    http://turkrequesters.blogspot.ca/20...o-prevent.html
    It's all good.

  3. #13
    Community Manager spamgirl's Avatar
    Join Date
    Aug 2011
    Gender
    Female
    Location
    Canada
    Posts
    12,554
    Thanks
    2,636
    Thanked 6,885 Times in 2,221 Posts

    How to block past workers from doing your new HITs (API)

    Through the API:
    Generic API instructions
    Python instructions

    -*-

    Through the RUI/GUI:

    1. Create a Qualification

    Go here.

    Click on .

    Friendly Name: I'd recommend using your requester name, date and HIT type. For example, Spamgirl 12.8.13 Block Past Workers
    Description: Describe the work this qualification entitles the Worker to do. For example, Workers with this qualification have already done my HITs before, so I'm using this qualification instead of blocking.

    2. Assign Qualifications

    Go here.

    Click on the first Worker ID you'd like to give the qual to. Click .

    Check the box next to your new Qualification, then click on .

    Repeat for every Worker on your list.

    3. Create your HITs

    Go here.

    Choose a template of your choice. Note that you cannot use your qualification with the Categorization or Sentiment templates as they don't allow you to change your qualifications, they force you to use the expensive Masters qualification. For categorization or sentiment HITs, use the Other custom template instead. Click Create Project.

    The only settings which matter towards using your qualification are found when you click the Advanced >> link at the bottom right corner of the big gray area. Next, click the Worker Requirements >> link.

    Worker Requirements: Choose Customize worker requirements.

    Click Remove twice. On the last row, click Clear.

    In the first drop down box, choose your qualification under Qualification Types you have created.
    In the second drop down box, choose has not been granted.

    Require qualification for preview: Yes.

    Everything else is up to you.

    Repeat step 2 every time your HITs are completed as you will need to give the qualification to all new people who do your HITs. Be sure to do this before reposting your HITs to ensure they can't retake it.

    -*-

    Why this is your best bet:

    1) If a worker has to accept a HIT and then submit their ID, or try to remember if they've done it before (trust me, they won't), they'll get upset and your Turk Opticon rating will suffer.

    2) Asking a worker to email you to find out if they've done it before is time consuming and more than any Turker will want to bother with for one HIT.

    3) Putting a PUBLIC list of worker IDs online and asking Turkers to see if their ID is there IS A BIG NO-NO! This exposes their ID and all associated data, such as their wish list and any demographic information listed there, to the world. If you take this route, reputation damage will be swift and severe. On top of that, websites aren't 100% reliable, so they may not even be able to access the list.

    4) Blocks put workers' accounts in jeopardy of suspension, so using them will totally ruin your reputation. Here is the hard block email text they receive when you block them:

    Greetings from Mechanical Turk. We regret to inform you that you were blocked from working on HITs by the following requester(s): (That's YOU)

    Requesters typically block Workers who submit poor quality work.

    Requesters rely on Mechanical Turk for high quality work results. In order to maintain quality, we continuously monitor service activity. Should additional Requesters block you in the future, we may suspend your account. Please ensure your work quality is at a high standard. We encourage you to read the HIT instructions and follow them carefully.

    We realize that this block may be a onetime occurrence for you. Should you maintain high work quality with no further complaints for the next few months we will dismiss this event.

    Regards, The Mechanical Turk Team

    -*-

    If your survey will only be posted once, the best way to ensure Turkers only complete it once is to post a single HIT and specify how many workers you want to do it. For example, 1 HIT, 100 Turkers. This can be done through the GUI, Command Line Tools and API.

    You can also use Qualtrics and other third-party services to do the work for you, but that will cost you money.

    The best, easiest and most effective way to ensure no retakes is a qualification, though, so check out the link at the beginning of this post!h
    It's all good.

  4. The Following User Says Thank You to spamgirl For This Useful Post:


  5. #14
    Btw, one of the comments from the link above describes a different method - http://www.tylerjohnburleigh.com/?p=321 (and you also need this - http://www.tylerjohnburleigh.com/mtu...urker-database - if you have workers from previous studies that you want to exclude).

    I wonder whether someone here has tried it?

  6. #15
    Requester
    Join Date
    Jan 2012
    Gender
    Male
    Location
    Based out of Georgia
    Posts
    18
    Thanks
    5
    Thanked 3 Times in 1 Post
    We set up ~100 hits per month and we find that we get the same Turkers accepting the HIT's. We need more of a sampling than that so would you still recommend a qualification? I would think specifying hit limits would be a better way to go.

  7. #16
    Community Manager spamgirl's Avatar
    Join Date
    Aug 2011
    Gender
    Female
    Location
    Canada
    Posts
    12,554
    Thanks
    2,636
    Thanked 6,885 Times in 2,221 Posts

    The qualification is your way of blocking workers who have done your HITs before. Anyone who has a qualification score of 1 can't do your HITs, for example, yet those with 0 (which can be autogranted through the API) can do it. This ensures no one who's done your HITs do them again, ever.
    It's all good.

  8. #17
    Requester
    Join Date
    Jan 2012
    Gender
    Male
    Location
    Based out of Georgia
    Posts
    18
    Thanks
    5
    Thanked 3 Times in 1 Post
    Thanks for the quick information!!

  9. #18
    Community Manager spamgirl's Avatar
    Join Date
    Aug 2011
    Gender
    Female
    Location
    Canada
    Posts
    12,554
    Thanks
    2,636
    Thanked 6,885 Times in 2,221 Posts

    They've added the new comparators to the GUI, so I've updated my post above to tell you how to block workers using DoesNotExist! Yay!
    It's all good.

  10. The Following User Says Thank You to spamgirl For This Useful Post:


  11. #19
    Community Manager spamgirl's Avatar
    Join Date
    Aug 2011
    Gender
    Female
    Location
    Canada
    Posts
    12,554
    Thanks
    2,636
    Thanked 6,885 Times in 2,221 Posts

    Excluding workers (from retaking surveys, doing too many HITs, etc.)

    TurkCheck: Ensuring that workers cannot participate in multiple studies from the same research project.

    Imagine that you have a research project in mind, with two Qualtrics studies that you want to run on mTurk: A and B. You want to prevent participant A, who has taken Study A to later participate in Study B. Should be simple, right?


    There are a few ways out there on how you can modify your Qualtrics surveys to check on completed turkIds, but it requires lots of manual labor. TurkCheck allows you to restrict your participants, by project name, using my database system, with minimal work on your part.


    Namely, it works through my database, and requires you almost no work on Qualtrics other than: 1) create two (blank) embedded data fields (no actual turkId input required) and, 2) change the end of survey url to one I'll generate for you automatically by clicking the button below. No need to add a bunch of turkIds to your survey flow. The total process is about 3 minutes.


    Here's how TurkCheck does it:

    http://faculty.georgetown.edu/sjb247...als/turkcheck/
    It's all good.

  12. #20

Page 2 of 2 FirstFirst 12

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •