Note: clicking on the above banners and making ANY purchase returns a commission to Turker Nation.
If you can't see the ad, please click on Shop on Amazon instead. | Want to advertise here? PM Spamgirl to learn more!




Page 1 of 4 123 ... LastLast
Results 1 to 10 of 32

Thread: Aaron Sojourner, University of Minnesota, manipulates Turk Opticon for a study

  1. #1
    *** Forum Supporter *** Tribune's Avatar
    Join Date
    Oct 2013
    Gender
    Male
    Location
    Bay Area
    Posts
    241
    Thanks
    578
    Thanked 817 Times in 140 Posts

    Aaron Sojourner, University of Minnesota, manipulates Turk Opticon for a study

    Don't do this HIT!

    Title: Survey about your MTurk work history - less than 1 minute
    Requester: Brian Robinson [AYSQZUL9GWE6P] (TO)
    TO Ratings:

    5.00 Communicativity
    4.71 Generosity
    4.86 Fairness
    4.29 Promptness
    Number of Reviews: 7
    (Submit a new TO rating for this requester)

    Description: Survey to better understand Turkers' earning opportunities
    Time: 30 minutes
    HITs available:1
    Reward: $0.12
    Qualifications: Location is US

    Do not do this HIT.
    It has been published several times by "different" requesters over the last 2-3 days.


    The one I did was ostensibly from a Brian Adams A37PEFVPTL6UKA.
    The reddit thread (below) was about a Brian Clark AUP8ZZ1Z21KHE .


    Looks like at least twenty-one new requesters, some "good" and some "bad," have fake TO from the same handful of sockpuppets.
    All of the sames sound WASP-y.


    ref: Rand4m on reddit HWTF
    w/ a tip of the community hat to clickhappier

  2. The Following 9 Users Say Thank You to Tribune For This Useful Post:

    + Show/Hide list of Thanks


  3. #2
    Community Manager spamgirl's Avatar
    Join Date
    Aug 2011
    Gender
    Female
    Location
    Canada
    Posts
    12,494
    Thanks
    2,625
    Thanked 6,855 Times in 2,213 Posts

    So for anyone following the Aaron Sojourner drama, here's how it goes:

    There is discussion about figuring out that this person is manipulating TO reviews to do a study here: https://groups.google.com/d/msg/turk...w/KRAQQOvWe-4J

    The bottom line is that it turns out Aaron Sojourner ran two studies:

    1) He had someone sign up for mTurk as a worker and do HITs. They checked whether the HITs done for requesters with good, neutral and bad reputations would match up with the number of rejections they got. It did. This is not mindblowing, we already know Turk Opticon works. This study was of no benefit to Turkers.

    2) He created 45 fake requester accounts on mTurk and at least 27 fake worker accounts on Turk Opticon. The fake worker accounts posted either good, neutral or bad reviews for each of the requesters on Turk Opticon. He then posted the same HIT on each requester account for 12 cents asking demographic information, including what your worker statistics were for the last 45 days, and intended to see if the rating affected how many people worked for each requester account. This totally skewed his data as only new people who would work for little pay and offer such personal information up would do the HIT, so he wasn't going to get a comprehensive Turker subject pool. The data was further compromised when he was found out by Tribune, Taintturk and Redditors on July 10th.

    Today, Tribune is attempting to get in touch with the IRB at the University of Minnesota. You can read more about this here: https://groups.google.com/d/msg/turk...w/mhGOnLeBtHwJ
    It's all good.

  4. The Following 8 Users Say Thank You to spamgirl For This Useful Post:

    + Show/Hide list of Thanks


  5. #3
    Interesting. I remember looking at that Brian Adams hit, and thought it seemed suspicious, so I didn't do it.

  6. The Following 2 Users Say Thank You to Rosemary For This Useful Post:


  7. #4
    Member tdalton43's Avatar
    Join Date
    Jan 2014
    Gender
    Female
    Location
    Ohio
    Posts
    185
    Thanks
    153
    Thanked 138 Times in 90 Posts

    Thanks for letting us know!

  8. The Following 2 Users Say Thank You to tdalton43 For This Useful Post:


  9. #5
    Member
    Join Date
    Oct 2011
    Gender
    Male
    Location
    Houston, Texas
    Posts
    379
    Thanks
    298
    Thanked 141 Times in 87 Posts

    Wow. Didn't know there was so much drama percolating concerning TO. I don't refer to TO that much any more since I now have my few trusted requestors that keep me busy. I would think that Aaron Sojourner may as well stay away from AMT for conducting his research after being so publicly exposed. Kudos to the many folk who tracked down all the information about him. Many of the researchers that are on AMT seem to be sincerely trying to hone their craft to become reliable researchers but there seem to be some who think that hypothetical (nice word for 'fake') questions can produce anything better than hypothetical answers that can only be used to reach hypothetical conclusions. How can research like this be approved to carry the university's name. Aren't classes in ethics being taught any more?

    ttfn
    ----------------------------
    -Robert

    Ever wonder what the speed of lightning would be if it didn't zigzag?

  10. The Following 2 Users Say Thank You to genoa For This Useful Post:


  11. #6
    Thanks for working on this Spamgirl.

  12. The Following 3 Users Say Thank You to chrisfuccione For This Useful Post:


  13. #7
    Member
    Join Date
    Oct 2011
    Gender
    Male
    Location
    Houston, Texas
    Posts
    379
    Thanks
    298
    Thanked 141 Times in 87 Posts

    Now that I've taken the time to follow the links provided by spamgirl I would like make a few more comments about Sojourner and his research methods. It appears to me that the University of Minnesota would ban him from ever associating their name with him. It looks like he may have broken some laws associated with using multiple fake identities or at least acted very unethically in collecting his data. It looks like, too, that Amazon should ban the University of Minnesota from ever being a requester due to TOS violations.

    Anyway, just my 2 cents.
    ----------------------------
    -Robert

    Ever wonder what the speed of lightning would be if it didn't zigzag?

  14. The Following 3 Users Say Thank You to genoa For This Useful Post:


  15. #8
    Super Sized Member
    Join Date
    Aug 2011
    Gender
    Other
    Posts
    3,379
    Thanks
    2,102
    Thanked 4,396 Times in 1,503 Posts

    It seems that ethics and research do not go together genoa. He had approval from the IRB, but they are conducting an investigation into his possible breech of the university code of conduct and misuse of the computer systems.

    He just posted a list of over 20 requester names he made on amazon and a list of dozens of email addresses for fake TO accounts.
    Just ridiculous.

  16. The Following 4 Users Say Thank You to taintturk For This Useful Post:


  17. #9
    Super Sized Member
    Join Date
    Aug 2011
    Gender
    Other
    Posts
    3,379
    Thanks
    2,102
    Thanked 4,396 Times in 1,503 Posts

    Here is the list he posted of the Amazon accounts he created and all the TO fake profiles he created.
    He did pay everyone who completed his hits.

    I submitted a formal complaint to UM. I think a few other people did as well. I am not going to take it any further, but IMO he should have asked for permission to do this from the TO administrators, but he knew they would tell him no.



    Here are all the (same) requester names with associated mTurk id#s
    Thomas Adams A1JLHAQ01QNJEM
    Thomas Robinson A3J35OP0A1YX0L
    Thomas Warren A1TXD65RUWMS0V
    Thomas Miller A26BM3BHSSZMD6
    Thomas Owens A35G4HTYCIS8JQ
    Thomas Kelly A3FIH03UPIYIH
    Thomas Martin APJQZOOO4A4MP
    Thomas Lewis AKKOAMBS3CHZB
    Thomas Clark A106FPPVLENXEU
    Thomas Roberts A2E6IEO8EC86RJ
    Thomas Jordan A1C1AUAULVM7NS
    Thomas Johnson A1Q5RFV41O11PT
    Mark Adams A8U54PRN9NDFN
    Mark Robinson A2PIAO9AIXAQ2X
    Mark Warren A3MTL6R2AE48ER
    Mark Miller A2VXVY94GSXR5B
    Mark Owens A19NC7RPBMGJ4A
    Mark Kelly A2RNLYGJ4EVT38
    Mark Martin A3BO8OQU32LQJJ
    Mark Lewis A35DF4IJ5P4RI5
    Mark Clark A2LIGJACYQVFMV
    Mark Roberts A13EZQBVS463S0
    Mark Jordan A302JS6L16QUV5
    Mark Johnson AL7ZOT9503LJH
    Joseph Adams A2YW4V3QAX5CF6
    Joseph Robinson A35AGHTJKAAYYR
    Joseph Warren A1U5MA5S785GJE
    Joseph Miller A3DRJPPIXVCIGL
    Joseph Owens A1PGBFMJ0LF8HC
    Joseph Kelly A32SN4PM56QF3G
    Joseph Martin A2YOXT5QB84ZM8
    Joseph Lewis A1IU130P64DPHE
    Joseph Clark A17N3WTWD025ED
    Joseph Roberts A2BSL3XCJH3P9Y
    Joseph Jordan A2II2B2A50UOIA
    Joseph Johnson A2WBP3GNGQ7Z01
    Brian Adams A37PEFVPTL6UKA
    Brian Clark AUP8ZZ1Z21KHE
    Brian Robinson AYSQZUL9GWE6P

    Here are the TO reviewers' ids:
    34545
    34546
    34547
    34548
    34549
    34551
    34552
    34553
    34554
    34555
    34557
    34558
    34560
    34561
    34563
    34564
    34565
    34566
    34567
    34568
    34570
    34571
    34573
    34574
    34575
    34576
    34578
    34579
    34580
    34581
    34585
    34586
    34587
    34588
    34589
    34590
    34591
    34592
    34593
    34594
    34596
    34597
    34598
    34599
    34600
    34601
    34602
    34604
    34605
    34606
    34607
    34608
    34609
    34611
    34612
    34613
    34614
    34615
    34616
    34617
    34618
    34619
    34620
    34621

  18. The Following 7 Users Say Thank You to taintturk For This Useful Post:

    + Show/Hide list of Thanks


  19. #10
    Super Sized Member
    Join Date
    Aug 2011
    Gender
    Other
    Posts
    3,379
    Thanks
    2,102
    Thanked 4,396 Times in 1,503 Posts

    I guess what it comes down to is that it is better to trust the people here on TN who you have worked with for years as opposed to using TO as a definitive review site. If you want to know about a requester, pop into chat and talk about them.

  20. The Following 3 Users Say Thank You to taintturk For This Useful Post:


Page 1 of 4 123 ... LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •