We need help getting enough donations to pay the monthly server bill. Click the thermometer to learn more.
Note: clicking on the above banners and making ANY purchase returns a commission to Turker Nation.
If you can't see the ad, please click on Shop on Amazon instead. | Want to advertise here? PM Spamgirl to learn more!




Results 1 to 3 of 3

Thread: How to start? Advise on designing good HITs / Setting Price

  1. #1

    How to start? Advise on designing good HITs / Setting Price

    Hi, I am new to mTurk. I have a website data annotation task. Basically, workers are expected to tag/annotate product attributes like Price, Color, Product Title in a given webpage. Each HIT task will be a question to the workers to highlight one product attribute, (e-g Can you identify and highlight product price in this page). Workers are expected to find the product attribute (e-g price) in that page and double click that element. I have few questions

    1. What will be the right price for this type task?
    2. How to collect feedback on this task? How Do I start before I posting thousands of hit task?
    3. How I can validate whether the workers selected right answer in this case.


    Note: My Task uses an external web application rendering the product pages (External Question Type).

  2. #2
    Moderator RippedWarrior's Avatar
    Join Date
    Sep 2011
    Gender
    Male
    Location
    Canada
    Posts
    2,032
    Thanks
    1,539
    Thanked 2,439 Times in 1,018 Posts

    HI sdscience1, thanks for posting.

    Quote Originally Posted by sdscience1 View Post
    1. What will be the right price for this type task?
    This is a difficult one to determine without trying the task first. In general, you want to aim for $0.20/minute of work. Of course with micro-task, this can be tough to determine. It sounds like this should pay around 4 cents.

    Quote Originally Posted by sdscience1 View Post
    2. How to collect feedback on this task? How Do I start before I posting thousands of hit task?
    Before you post thousands of HITs, definitely work out any kinks in the HIT before going live. There is a sandbox environment you can use to test your actual HIT interface. Next, post a small batch and ask for feedback, either in our IRC Chat (for real-time feedback) or in the Daily HIT thread. .

    Quote Originally Posted by sdscience1 View Post
    3. How I can validate whether the workers selected right answer in this case.
    Quality assurance is always tough as well. For starters, you can use qualifications to filter your pool of workers. For example:

    "HIT Approval Rating is not less than 98%"
    "Total Approved HITs is not less than 1000"
    "Location is one of: AU, CA, GB, US"

    You could also try "plurality" (aka majority-rules). This is where you post multiple instances of the same question (say to 3 different workers) and accept the majority answer as the correct one. Note, 'majority-rules' is NOT a good approval/rejection policy, but can help with quality assurance.

    IN my experience, the keys to running a successful project on MTurk is:

    1) Fair pay
    2) Proper use of qualifications
    3) Fair and transparent approval/rejection policy
    4) Efficient HIT design
    5) Clear and concise instructions
    6) Communication with the workforce.

    Sounds like you're well on your way and good luck with your project!
    Eschew obfuscation, espouse elucidation. Batch HITs for newbs. 1000 req'd. 5000 req'd.

  3. The Following User Says Thank You to RippedWarrior For This Useful Post:


  4. #3
    Awesome. Thanks for the reply, really helps.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •