We need help getting enough donations to pay the monthly server bill. Click the thermometer to learn more.
Note: clicking on the above banners and making ANY purchase returns a commission to Turker Nation.
If you can't see the ad, please click on Shop on Amazon instead. | Want to advertise here? PM Spamgirl to learn more!




Page 1 of 2 12 LastLast
Results 1 to 10 of 20

Thread: Excluding workers (from retaking surveys, doing too many HITs, etc.)

  1. #1
    Requester
    Join Date
    Sep 2011
    Gender
    Male
    Posts
    4
    Thanks
    0
    Thanked 0 Times in 0 Posts

    How to block past workers from doing your new HITs (API & GUI!)

    For detailed instructions on our recommended process of blocking past workers, click here to get to the post below.

    Hi all,
    I'm using mechanical turk to run surveys, and I don't want turkers who have done previous versions of a survey to be able to do the survey again. The easiest way for us to implement this is by blocking workers who have done the previous versions of the survey from taking newer versions.

    I'm concerned that blocking a worker hurts their reputation, so might piss off workers with whom we have no quality issues. Would I hurt a worker's reputation on mTurk by blocking them from a HIT?

    The other option is to ask workers for their mTurk worker ids when they start the survey so workers can see if they've done the survey before. Is it easy for turkers to look up their worker id?

    Thanks for your help!

  2. #2
    Yes, we know our ID. If you can set it up this way, it would:

    1. Be very smart
    2. Be good for your reputation!

    Thanks so much for taking the time to check this out, blocking is very bad for us.

  3. #3
    Requester
    Join Date
    Sep 2011
    Gender
    Male
    Posts
    4
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Thanks librarian50!

  4. #4
    Three better options:

    1) Use ad-hoc qualifications.
    - Grant "did survey" to those who completed that version.
    - When posting the second version apply the qualification with the comparitor "does not exist"
    This will always work. The downside is lots of manual management of quals.

    2) Host the HITs yourself.
    - workerId is sent to your server up front, so you can check against your own db.
    - You can run one HIT for all survey versions and route workers as needed.
    - You have complete control over your data, HIT format, etc.
    This will also always work, as long as you have basic web and database coding ability.

    3) Outsource.
    - Let CrowdSource.com deal with all this. Just give them your survey and get back your results.
    - Additional advantage is a heavily pre-screened worker pool.
    - Cost is pretty modest if you put any value on your own time.


    n.b., DoesNotExist is currently not functional as a comparator, so for strategy #1 you'd have to do something like assign "my survey takers" or whatnot to anyone approved to work for you, and assign/update the scores for that qualification to indicate what surveys a worker has done and/or is eligible for.
    Last edited by techlist; 09-18-2011 at 10:22 PM. Reason: correction

  5. #5
    Community Manager spamgirl's Avatar
    Join Date
    Aug 2011
    Gender
    Female
    Location
    Canada
    Posts
    12,679
    Thanks
    2,683
    Thanked 6,971 Times in 2,258 Posts

    You should definitely block the user from ACCEPTING instead of having them accept the HIT, look up their ID, find out they can't do the HIT, then have to return it. Return rates are in our stats, too, so who knows if someday they could hurt us.

    I vote for a qual - they're very easy to do, even in the GUI. If you're using the GUI, you can assign a qual to everyone who has worked for you which is super easy. Just keep assigning that qual as people do your surveys and make all your surveys only open to people who DON'T have the qual.

    Oh, and a qualification is free Don't have to pay anyone anything.
    It's all good.

  6. #6
    What we use is a Google Site (free) to list the WorkerIDs for people who aren't qualified to do the current HIT (usually due to having done a previous version).

    We use an external data collection site (usually Survey Monkey or Inquisit), so just above the link to that, we have a link to the Google Site page. Potential workers can search for their ID and only accept if their ID does NOT appear on that list (example here: https://sites.google.com/site/mturkinelligible/). Workers appreciate this, as it assures them that they won't be rejected if they don't appear on the list.

    To achieve this, we download the results of every HIT which has a column of WorkerIDs and then simply copy and paste those into the text of the main page of the Google Site. You can create as many sites as you need to cover unique combinations of ineligible workers

    Note, this is a workaround, since you can achieve a similar thing using qualifications, but for us, it works!

  7. #7
    Requester
    Join Date
    Sep 2011
    Gender
    Male
    Posts
    4
    Thanks
    0
    Thanked 0 Times in 0 Posts

    thanks to everyone for the helpful suggestions!

  8. #8
    Requester
    Join Date
    Sep 2011
    Gender
    Male
    Posts
    4
    Thanks
    0
    Thanked 0 Times in 0 Posts

    There doesn't seem to be a 'does not exist' comparator for the sake of qualifying workers for HITs. We could use a "not equal to" comparator, but it seems to disqualify workers who we haven't assigned the qualification to. So we haven't been able to get this to work (yet).

    Quote Originally Posted by techlist View Post
    Three better options:

    1) Use ad-hoc qualifications.
    - Grant "did survey version 1" to those who completed that version.
    - When posting the second version apply the qualification with the comparitor "does not exist"
    This will always work. The downside is lots of manual management of quals.

    2) Host the HITs yourself.
    - workerId is sent to your server up front, so you can check against your own db.
    - You can run one HIT for all survey versions and route workers as needed.
    - You have complete control over your data, HIT format, etc.
    This will also always work, as long as you have basic web and database coding ability.

    3) Outsource.
    - Let CrowdSource.com deal with all this. Just give them your survey and get back your results.
    - Additional advantage is a heavily pre-screened worker pool.
    - Cost is pretty modest if you put any value on your own time.

  9. #9
    UBC is correct, and I've updated my suggestion above with the appropriate workaround for the qualification approach.

    In regard to publishing workerIds on a public site, that seems both clumsy and time consuming for workers, not to mention a privacy breach for both the workers and the requester.

  10. #10
    Requester
    Join Date
    Dec 2011
    Gender
    Male
    Location
    Provo, Utah
    Posts
    32
    Thanks
    8
    Thanked 5 Times in 5 Posts

    In the workers area of this forum, I've read people posting about "soft blocks" and "hard blocks." It appears their notion is that some blocks can be done by the requester without affecting the worker's reputation. However, I don't believe that's correct. As I read Amazon's documentation, a block is a block is a block.

    It's really too bad, for two reasons:
    (1) A "soft block" (i.e., nothing personal, it's just that for my reasons I don't want you working on my HITs right now), as compared to a "hard block" (i.e., this person contributed shady, unethical work), would allow requesters to manage their workforce better. The qualification thing seems like it would be useful, but I've worked with qualifications extensively and the hobbles that Amazon applies to qualifications truly limit their effectiveness for the use you're describing.
    (2) Requesters are not told their blocks hurt workers. Once I caught wind of it from a worker forum, I spent a lot of time going through Amazon's documentation, and I finally found confirmation that a number of blocks (three? I forget) will indeed lead Amazon to suspend the worker's account. Amazon should do a better job of informing requesters of the implications of the block.

Page 1 of 2 12 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •