We did not have enough package purchases to pay our July or August server bills. Please help us get enough for September! Click the thermometer to learn more.
Note: clicking on the above banners and making ANY purchase returns a commission to Turker Nation.
If you can't see the ad, please click on Shop on Amazon instead. | Want to advertise here? PM Spamgirl to learn more!




Results 1 to 10 of 10

Thread: New requester seeking feedback on a series of HITs

  1. #1

    New requester seeking feedback on a series of HITs

    Hi everyone,

    I'm a newbie requester looking to get some feedback on a series of HITs for identifying and labeling forum/blog/Q&A sites. Early prototypes of the HITs (on the worker sandbox) are up at:

    HIT #1: Identify forum sites:
    https://workersandbox.mturk.com/proj...UORCVUY6/tasks
    This is a simple single question with instructions. I've got keyboard shortcuts for the options for easy completion. Would it be helpful to have the site itself in an iframe so it displays inline? Or is it better as a link that opens in a new tab/window?

    HIT #2: Find forum root:
    https://workersandbox.mturk.com/proj...NFP58UX5/tasks
    This is a text box entry, again after looking at and manipulating a website. Same question about iframe vs. link as before. Also, is it clear enough what I mean by forum "root"? Would some examples help?

    HIT #3-5: Label index/thread/post pages:
    https://workersandbox.mturk.com/proj...2NT5YV02/tasks
    https://workersandbox.mturk.com/proj...3QX6ZUZX/tasks
    https://workersandbox.mturk.com/proj...QU9VUVUZ/tasks
    These involve a custom Javascript overlay that walks you through labeling several (~20) fields on a webpage. I'm most concerned about the granularity of these HITs - I know that it's best to break HITs up into individual tasks that can be completed quickly, but in these cases, the later stages are dependent upon what's selected in early stages, and I can use software to reduce the amount of work the Turker has to do or improve the error-checking on their input. (For example, if no categories are selected, it should hide the request to label category fields, and the set of legal click targets for the field is modified by the items selected in earlier stages.) Are these too daunting & long? Should I break them up so that each "stage" is a separate HIT, moving some of the software logic that's currently in the Javascript to the server? So that each individual labeling task is a separate HIT? Or is it more convenient to be able to click through a number of fields at once all on the same document?

    I'm also interested in feedback on wording, instruction clarity, UI, and pricing, if you have any opinions on those. I'll likely change the color scheme to match the default templates and add some example screenshots in the future.

    Thanks for any input!

  2. #2
    Moderator and Forum Suppo Rosie The Robot's Avatar
    Join Date
    May 2015
    Gender
    Female
    Location
    Serving God
    Posts
    5,616
    Thanks
    11,720
    Thanked 11,452 Times in 4,168 Posts

    Welcome to Turker Nation!

    Since you're a Requester, you should respond to the private message Spamgirl has sent you titled "URGENT! Please read!" so that you can get full access to the forum. Thanks!

    Sent from my Moto E (4) using Tapatalk
    If Turker Nation has helped you make more money than you would have without it, please consider chipping in to help pay the monthly server bill. When you see the big thermometer above the board, please click on it for more info! Thank you!

  3. #3
    Member
    Join Date
    Aug 2011
    Gender
    Female
    Location
    Canada
    Posts
    251
    Thanks
    213
    Thanked 656 Times in 227 Posts

    my preference is for links to open in a new tab or window so I can move it to my second monitor - that makes it easier for me to reference as I work

    and if these could be open to Canada, that'd be awesome!

  4. #4
    *** Forum Supporter ***
    Join Date
    Oct 2017
    Gender
    Female
    Posts
    4,408
    Thanks
    13,994
    Thanked 10,827 Times in 3,867 Posts

    #1 - I agree that it's easier to have it open in a new window
    #2 - the language is pretty clear, but I would err on the side of always giving examples
    #3 - 5 - I can't see these, but speaking for myself I don't mind more intricate hits, particularly if you're walking workers through it.

  5. #5
    Thanks for the responses!

    Quote Originally Posted by thehoncho View Post
    and if these could be open to Canada, that'd be awesome!
    I'd only submitted a limited number of HITs (2) to the worker sandbox with this design, and it looks like someone had accepted them and so it was showing up as "no hits available". I've added a bunch more - it's all dummy data now, to test out the labeling interface - so hopefully that'll last a while. Also had to fix a bug on my end, which'll likely not be the last - this is just a rough cut to get feedback before polishing it up.

    Quote Originally Posted by Rosie The Robot View Post
    Welcome to Turker Nation!

    Since you're a Requester, you should respond to the private message Spamgirl has sent you titled "URGENT! Please read!" so that you can get full access to the forum. Thanks!
    That's the one with the form linking my Amazon ID to the forum identity, right? I submitted that concurrently, and just read Spamgirl's reply. I'll head over to the Introductions thread shortly...is this forum okay for this discussion, or should it be moved to "Everything Else"?

  6. #6
    Moderator and Forum Suppo Rosie The Robot's Avatar
    Join Date
    May 2015
    Gender
    Female
    Location
    Serving God
    Posts
    5,616
    Thanks
    11,720
    Thanked 11,452 Times in 4,168 Posts

    Quote Originally Posted by pilazza View Post
    Thanks for the responses!



    I'd only submitted a limited number of HITs (2) to the worker sandbox with this design, and it looks like someone had accepted them and so it was showing up as "no hits available". I've added a bunch more - it's all dummy data now, to test out the labeling interface - so hopefully that'll last a while. Also had to fix a bug on my end, which'll likely not be the last - this is just a rough cut to get feedback before polishing it up.



    That's the one with the form linking my Amazon ID to the forum identity, right? I submitted that concurrently, and just read Spamgirl's reply. I'll head over to the Introductions thread shortly...is this forum okay for this discussion, or should it be moved to "Everything Else"?
    Thank you!

    I'm not actually sure where the best place is. @spamgirl would know.

    Sent from my Moto E (4) using Tapatalk
    If Turker Nation has helped you make more money than you would have without it, please consider chipping in to help pay the monthly server bill. When you see the big thermometer above the board, please click on it for more info! Thank you!

  7. #7
    Hi everyone, it's me again,

    I did a bunch of changes to this MTurk sequence based on feedback and also some requirements on our end:

    • I combined the first two HITs into one that asks all the questions at once.
    • I broke apart the three later multi-stage HITs into their individual tasks, so that Turkers don't feel like they have to do a huge monolithic HIT all at once or end up getting stuck halfway through a large task with a lot invested in it.
    • I then recombined the resulting HITs based on the type of work the Turker has to do, rather than how it's used within our system. So instead of splitting it up by page type, I've got all tasks where you have to select a bunch of repeated elements in one HIT type, and all the ones where you need to label a bunch of fields in another HIT type. That makes it easy to do a bunch of similar HITs in a row.
    • I added the ability to unselect & reselect elements, so that if you make a mistake you don't have to abandon the whole HIT.
    • I added a bunch of examples, with screenshots


    A trial batch of a dozen or so HITs is up on the real (non-sandbox) site - search for "pilazza" for it, though it seems like they disappear fast so there often aren't results. The first HIT in the sequence has gone well, but the next HITs seem like they're getting accepted by workers but then not finished and just allowed to time out after the 3 hours.

    Any idea why that might be? Software bug? Unclear instructions? Price too low to make it worthwhile? Since they disappear from the queue so fast, it's hard to view how they actually appear to Turkers live, but here are some examples of the raw HTML+JS files for the HIT:



    Any feedback is appreciated, of course, particularly on where instructions might not be clear or why Turkers may be abandoning this sequence.

  8. #8
    *** Forum Supporter ***
    Join Date
    Oct 2017
    Gender
    Female
    Posts
    4,408
    Thanks
    13,994
    Thanked 10,827 Times in 3,867 Posts

    So I just found: Select repeated elements on webpage - is this the one that is being abandoned? Here's why I'd be worried about doing this hit:

    You've got a popup on the right side of the window that says click to see example - but the different images send me all over - it's not clear what I'm supposed to be looking at. I clicked expecting to see a blow up of the examples or something.

    Below that is your submit button, and the popup is blocking the page. I figured out that I could click on the page and the popup would shrink, but generally that's not expected behavior.

    Finally, the page I'm looking at is the front page to a site. I can't really see any threads as such. Do you have an expectation explained if someone cannot find threads?

    I hope that helps

  9. #9
    Thanks for the feedback! I'm hearing a couple major takeaways:

    • I spent so much trying to explain what a "forum", "thread", "category" is that I forgot to clearly explain exactly what the task is. The examples are supposed to be just guidelines to illuminate what to click - the focus should be on the page behind, where you actually click on things but that's obscured by the popup.
    • Mistakes in the first HIT of the series, which determines which page the popup displays on, can make it impossible to complete subsequent HITs, so I need to be really clear in the instructions there and ideally have some way to compensate for mislabeled forum roots or types.



    I'll go try and re-do the instructions based on that and see how it performs...may also need to rework the backend workflows to increase the accuracy of that first HIT.

  10. The Following User Says Thank You to pilazza For This Useful Post:


  11. #10
    *** Forum Supporter ***
    Join Date
    Oct 2017
    Gender
    Female
    Posts
    4,408
    Thanks
    13,994
    Thanked 10,827 Times in 3,867 Posts

    Quote Originally Posted by pilazza View Post
    Thanks for the feedback! I'm hearing a couple major takeaways:

    • I spent so much trying to explain what a "forum", "thread", "category" is that I forgot to clearly explain exactly what the task is. The examples are supposed to be just guidelines to illuminate what to click - the focus should be on the page behind, where you actually click on things but that's obscured by the popup.
    • Mistakes in the first HIT of the series, which determines which page the popup displays on, can make it impossible to complete subsequent HITs, so I need to be really clear in the instructions there and ideally have some way to compensate for mislabeled forum roots or types.



    I'll go try and re-do the instructions based on that and see how it performs...may also need to rework the backend workflows to increase the accuracy of that first HIT.
    I think you've got it - the first thing I do when I go to a hit is check the instructions - if I can understand what is going on quickly, there's a much better chance that I'll do it. I think the explanations are almost a 2nd level of information - if someone needs it, they're there, but they could just look at the information at a glance to see examples.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •