Note: clicking on the above banners and making ANY purchase returns a commission to Turker Nation.
If you can't see the ad, please click on Shop on Amazon instead. | Want to advertise here? PM Spamgirl to learn more!




Results 1 to 6 of 6

Thread: Blocking workers who are scammers

  1. #1
    Requester BitLit's Avatar
    Join Date
    Jul 2014
    Gender
    Male
    Location
    Vancouver BC
    Posts
    38
    Thanks
    20
    Thanked 136 Times in 30 Posts

    Blocking workers who are scammers

    I'm a requester and I post a reasonably large number of HITs. I think I spend about $2000/month on mTurk. My HITs are the "label books on a shelf" ones. The HITs are generated when users take pictures of their bookshelves ("shelfies") in my app. The software identifies most of the books, but some need to be manually identified -- these are the ones that show up in HITs.

    In each HIT there are 12 book spines of which there are 3 which are "tests" where the right answer is known and I'm checking to see if the worker is making a good effort to identify the book. I know that the tests aren't perfect, so I allow workers to get a percentage of them wrong without revoking the Qual.

    However, recently I discovered that I had a bug in my code and was allowing workers to mark all books as "blurry/unreadable" and still pass the tests. Some workers did 10000+ of my HITs using this loophole before I caught it. I've fixed it now... and will be re-posting all the HITs that were done by the workers who used this exploit (it's going to cost me several thousand dollars since I can't reject the scammers' work). I went through my results database and found all workers who used this scamming loophole and blocked them.

    But I've read that blocking a worker can mean that Amazon will then ban them from mTurk. I'm torn... I don't want to destroy somebody's only source of income. But at the same time they were cheating on my HITs. But the cheat was only possible because I had a bug in my code. What's the right thing to do? I could have just removed the Qual form those workers, but then I'd be letting them scam some other requester... and because these workers were scamming on my HITs, it means that honest workers weren't getting access to my HITs since I'm sure the scammers developed scripts to automatically accept and complete my HITs. Thoughts?

  2. The Following User Says Thank You to BitLit For This Useful Post:


  3. #2
    Moderator
    Join Date
    Aug 2011
    Gender
    Male
    Posts
    1,075
    Thanks
    251
    Thanked 2,565 Times in 593 Posts

    I would point out that you are not responsible for the actions of these workers or any possible action AMT may take as a result of those actions. I think you answered your own question with this:

    I'd be letting them scam some other requester... and because these workers were scamming on my HITs, it means that honest workers weren't getting access to my HITs since I'm sure the scammers developed scripts to automatically accept and complete my HITs.


    If your evidence is solid, and they were scamming on the HITs as well as the test, I think you did the right thing.

  4. The Following 3 Users Say Thank You to bid For This Useful Post:


  5. #3
    Member thegrinder's Avatar
    Join Date
    Oct 2013
    Gender
    Female
    Location
    North Carolina
    Posts
    592
    Thanks
    2,737
    Thanked 1,665 Times in 513 Posts

    I do as many of your HITs as possible. I find them to be interesting and reasonably priced - except I just bought a book I saw on one of your HITs! Great way to save Bought it on Amazon, folks, not his site.

    Scammers have no place on MTurk. Go ahead and block the execrable pieces of rubbish - the more blocks these asses get the sooner we get rid of them (this batch, anyway!). I've emailed with you in the past (nothing bad folks) and you've always been helpful and quick to respond. Thank you for being a great requester.

  6. The Following User Says Thank You to thegrinder For This Useful Post:


  7. #4
    Member Zeph85's Avatar
    Join Date
    Aug 2016
    Gender
    Male
    Location
    Michigan
    Posts
    8
    Thanks
    0
    Thanked 6 Times in 4 Posts

    I want you to consider this alternative. If a person cheats on a test and they are caught the penalties are severe. For example, if a student is caught cheating on a test they could be expelled from their university which would most likely destroy their academic career. So yes, if you have spent thousands of dollars and have been scammed then you need to take action against those workers.

  8. The Following User Says Thank You to Zeph85 For This Useful Post:


  9. #5
    Requester BitLit's Avatar
    Join Date
    Jul 2014
    Gender
    Male
    Location
    Vancouver BC
    Posts
    38
    Thanks
    20
    Thanked 136 Times in 30 Posts

    Thanks so much for the feedback! And that's awesome that you found a book in one of the HITs and then went out and bought it!!!!

    This might be TL;DR, but I figured it was worth sharing... all the work that everybody on mTurk does in labeling the book spines is critical to us boot strapping our computer vision. We've seen over 2 million book spines come through our app. In the early days 100% of the book spines needed to be labeled by mTurk, but as more and more books were labeled we've been able to develop automated recognition software that's able to identify easy to read book (e.g. Catcher in the Rye) or very common (e.g. Harry Potter) books. In December 2014 our app was featured on LifeHacker and 10,000 new users signed up and submitted pics of their bookshelves. The load on mTurk was HUGE... our servers that power the auto-complete drop down list were over powered and it took us days to get enough machines running so that workers could do HITs quickly again. But as the automated recognition has been able to take on more and more of the recongition, we've been able to handle bigger and bigger user loads. For example, a few weeks ago Apple featured Shelfie in the #1 spot in the App Store... we had 30,000 users submit pictures of their bookshelves over the course of a few days... but this time instead of our mTurk servers melting, the load was pretty reasonable. We'll always need to have mTurkers labeling book spines since new books are always been published and there are always tattered covers that the computers won't be able to recognize. But that's why it's so important that we have good workers. If things get labeled wrong, it messes up all the hard work for all the workers who've labelled the millions of book spines so far.

  10. The Following 2 Users Say Thank You to BitLit For This Useful Post:


  11. #6
    Member thegrinder's Avatar
    Join Date
    Oct 2013
    Gender
    Female
    Location
    North Carolina
    Posts
    592
    Thanks
    2,737
    Thanked 1,665 Times in 513 Posts

    This is actually the second book I've bought, having seen it on one of your HITs! Thank you for the above info. It really helps to know a little background to what we're doing.

  12. The Following User Says Thank You to thegrinder For This Useful Post:


Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •