A Plea to Amazon: Fix Mechanical Turk

Aw dammit, Turk is great.<p>I spent $300 in one day on Turk and Amazon gave me a personal account manager... And that's the problem, if you spend $300 on Turk in a day you're a high roller, Turk just isn't that big of a market. I've had days when I've been responsible for 20% of the HITs in the system and I'm just a poor computer programmer who writes code for a penny a line. I just got an email the other day from a Turk who was begging for more HITs.<p>Personally I've got no problem with the API and for certain kinds of tasks I've got good answers for the quality problems. For other ones, I haven't figured it out yet.<p>If you think Amazon's offering a service that's too bare bones, you ought to start a start-up that layers services on top of it. It may be a tough row to hoe though... My layer that interfaces to turk is specialized to my purposes and isn't spiffy, commercial and easy to use. It would probably cost you 20-50 times as much to develop something "usable" than it cost me to develop something that fits my workflow. Unless you can amortize this cost over a LARGE number of customers, your spiffy new system is going to cost more than it costs to pay the Turks. And good luck getting Angel or VC money to develop something that might never get all that big... And that will probably get regulated with the government when they find out that you're getting people to work for you for $2.50 an hour!<p>For now I'm tickled pink because I can get labor cheaper than my competitors can -- I can escape the economic problems that keep silicon valley companies going in circles chasing each other's tails.<p>What I really wish is that I could pay people Facebook credits to do work for me: my guess is that I could be paying people $1.25 an hour that way... However, Facebook knows the first thing people would do if it was easy to pay credits out to people is create gambling apps, so they don't make it easy.

My main issue with Mturk is not being able to sign up as a non-US company.<p>It's not that I'm angry at Amazon or anything, I just wish that they would come to the conclusion that our money is just as green as everyone elses.

I have used Mechanical Turk quite extensively the last few months. But I have always used it via Crowdflower. They offer a more sensible API and interface. And they keep scores on all workers, and calculate how well they perform. This influcences the weight of their job submissions. It takes a bit getting used to their terms, but I'm quite happy with the end results.

I have a product that I hope to launch before Jan 1 that will address these issues.

Completely agree with his sentiment regarding TurkIt and the prospect of making iterative tasks easily accessible to the masses. Read some of the papers that Greg Little has put out from MIT to open up your eyes to some advanced uses of Mechanical Turk.<p>Using the RTurk gem and my recently released Turkee gem ( <a href="http://github.com/aantix/turkee" rel="nofollow">http://github.com/aantix/turkee</a> ), I've been able to quickly throw together a demo Rails application that allows Turkers to vote on previous Turker submissions or use other turker's submissions as the basis of their own own submission.<p>With my experience under this type of interface most Turkers will vote for one of the previous submissions but if you offer a small bonus for the highest voted submission, more turkers will take a chance at submitting something original (even if it's a trivial 50 cents bonus).<p>The demo's code is available under my github account (aantix/Turkee-Iterator). Here's some of screenshots of my demo; the top image shows my initial parameters for setting up my Turkee task. The second screen shows the responses I received from the Turkers. The third screen shows the form that was presented to the Turkers (either vote or submit a joke).<p><a href="http://imgur.com/13rCM.png" rel="nofollow">http://imgur.com/13rCM.png</a><p><a href="http://imgur.com/ghveH.png" rel="nofollow">http://imgur.com/ghveH.png</a><p><a href="http://imgur.com/229uc.png" rel="nofollow">http://imgur.com/229uc.png</a><p>Right now I am writing a game that generates content using a couple iterative approaches to content generation with Mechanical Turk (hence my interest in this area).<p>Don't hesitate to drop me a line if you have any problems with getting my demo running.

Chris Conley, a member of this community and a well known Philly developer, has been building a powerful API for developers to use to interact with Mechanical Turk (it started with his own frustrations using theirs). You can check it out here: <a href="http://houdinihq.com/" rel="nofollow">http://houdinihq.com/</a>.<p>If you're looking for a solution to some of the problems addressed in this article, his API (and the corresponding Rails gem) are a good place to start.

I think MT is failing because they only allow busiless in the US, which is a tiny market for jobs that pay pennies/hour. But if Amazon managed to open up say, India or China, there would be a much richer ecosystem in place.<p>An interesting finding is that it costs about $5K-10K to open a US bank account from overseas, post 9/11. Pre 9/11 it was probably around $100.<p>Although you can use MT credit to buy stuff out of the Amazon store.

These change requests are (mostly) from one particular perspective, so it ignores some other problems from the worker's perspective; namely, that the market hasn't quite settled on reasonable rates for work, and that a worker has insufficient protection from exploitive work providers. These are also serious problems that could hinder it as a platform.

Great community and service, terrible and confusing interface. I wonder maybe their UI is so bad as they only want to attract a certain demographic, though it seems their demographic(turkers) runs beyond just computer nerds.

Fix these problems and you may have yourself a viable startup.

Has anyone tried CloudCrowd? They have a similar system to MTurk but it's built on facebook and it's a managed process similar to crowdflower... Interested to hear any experiences.