Archive for the ‘Amazon Mechanical Turk’ Tag

Work3.0 Featured on Crowdsourcing.org!

Tuesday, December 18th, 2012 by Seth Weinstein

crowdsoucing.orgThree cheers for recognition! The article I posted last week about Automan and Amazon Mechanical Turk was picked up as an editorial feature on Crowdsourcing.org, the leading site for news and discussion regarding crowdsourcing, crowd funding, cloud labor, and distributed knowledge.

Professionals in the field would be wise to check out the content on Crowdsourcing.org with some degree of regularity. Their articles are well-sorted, submitted from a variety of sources, and will be seen by bright minds from a multitude of distinct industries. When I reported for Tiny Work, many of my news leads could be traced back to Crowdsourcing.org, and the attention my articles received on the site was instrumental in getting me where I am today.

Thank you, Crowdsourcing.org, and I hope we continue to have a mutually beneficial relationship with each other.

Advertisements

CrowdFlower Under Fire: Lawsuit Threatens Employment Status

Friday, December 14th, 2012 by Seth Weinstein

crowdflowerIt is not a good day to be CrowdFlower. The website, based on crowdsourcing, offers a service akin to Amazon Mechanical Turk: huge amounts of data handled cheaply and quickly by large crowds that are paid per task. But a new lawsuit leveled by an Oregon man who works for the company claims that its employees are drastically underpaid. According to the suit, CrowdFlower pays its workers much less than the federally-mandated minimum wage for employees, with some fees going as low as $2 to 3 an hour.

The sticking point in the suit is the classification of CrowdFlower’s work force. The suit claims that they are employees, like at any other company, and must therefore not only be paid the minimum wage, but also must have income and other taxes withheld. CrowdFlower, however, claims that their employees are independent contractors, and subject to less strict guidelines. The results of the suit will most likely hinge on the classification of their employees into one category or the other.

It’s tricky, though. According to the official IRS website, there is no one deciding factor that determines if an employee is or is not a contractor:

Businesses must weigh [behavioral, financial, and relationship-related] factors when determining whether a worker is an employee or independent contractor. Some factors may indicate that the worker is an employee, while other factors indicate that the worker is an independent contractor. There is no “magic” or set number of factors that “makes” the worker an employee or an independent contractor, and no one factor stands alone in making this determination. Also, factors which are relevant in one situation may not be relevant in another.

The keys are to look at the entire relationship, consider the degree or extent of the right to direct and control, and finally, to document each of the factors used in coming up with the determination.

In a nutshell, this means that the plaintiff’s side of this case will likely launch a large-scale investigation into CrowdFlower’s employment methods and worker duties, as well as peer into how the workers are paid, and by whom. If I can be honest, it does not look great for CrowdFlower. The lawsuit claims that some of their employees aren’t even compensated monetarily, and are instead given online gaming credits or points towards award programs. If it turns out that CrowdFlower’s workers cannot be classified as contractors, that is going to be all but a fatal blow. Further complicating the matter is the fact that CrowdFlower does not directly employ its workers, but instead hires them through channels like MTurk.

Further reading on the IRS site details what will happen if the employees are found to not be contractors. In the most likely scenario, CrowdFlower will have to pay both back taxes to the IRS and full retroactive compensation to the plaintiff, plus damages. Since the plaintiff is also attempting to get his suit into class action status, he may not be the only one who gets paid. However, CrowdFlower can claim that it had a “reasonable basis” for treating its workers as contractors, in which case they may not have to pay the aforementioned compensation.

There is also the possibility that CrowdFlower will be forced to reclassify its employees and give them all the tax- and compensation-related baggage that goes with it. Again, this could be a huge blow to the company since they’ve built their model on the idea that their workers are contractors. Without drastically increasing prices or dramatically cutting costs, they are simply not going to have the money to keep their namesake Crowd around.

It’s times like these that I am very thankful for how Ziptask runs things. We hire contractors as well, but with two major differences when compared to the CrowdFlower model: we are unequivocal about labeling our workers as contractors, and we pay them a fair wage. Crazy, right? But it turns out people like feeling like their time is worth something. There’s a reason that there isn’t a huge line at McDonald’s to be the next fry cook. Some work is menial and less-than-desirable, but it still has to get done. And if CrowdFlower wants that to happen, it’s in their best interest to not get sued for treating their employees poorly.

Simple stuff, really.

Cloud Labor Scuffle: Ziptask, AutoMan, and MTurk’s Flaws

Tuesday, December 11th, 2012 by Seth Weinstein

AutoMan

Researchers at the University of Massachusetts have recently created AutoMan, a new cloud labor algorithm that intends to outsource not the worker, but the boss. New Scientist’s Douglas Haven reports that AutoMan is a fully automatic system that analyses and delegates tasks to human workers on Amazon Mechanical Turk. Where Ziptask simplifies task outsourcing via our task management team and a “set it and forget it” setup, AutoMan seeks to tackle the process completely automatically. If AutoMan is successful, it could end up wildly improving on the original Turk by automating oversight, the one remaining untouched process.

In a report published by the UMass researchers, the grievances against MTurk are laid out quite succinctly on the very first page. Turk doesn’t scale well to complicated tasks, it’s often difficult to determine the appropriate payment or time scale for a job, and there’s no guarantee that the finished work will be of acceptable quality. Being so similar, both Ziptask and AutoMan have their own unique ways of addressing these flaws.

Scale and Complexity

MTurk is great for simple tasks like identifying the subjects of photos, but when it comes to complicated, iterative, or interrelated tasks, its power often falls short. The problem lies in the fact that clients need to separate complex tasks into bite-sized chunks of work, which are better suited to the platform. Ziptask solves this problem with its team of project managers, who can break down and assign tricky tasks to multiple workers, or pore through their database for a worker who is qualified for all aspects of the task. Unfortunately, it does not appear as though AutoMan will have any innate capability to split up or delegate a task in such a way; perhaps this functionality will be addressed in a later update. We’ve discussed the strength of Ziptask’s scalability before, so I hope the UMass researchers have something good up their sleeves.

Payment and Time

Those who wish to assign work via MTurk not only have to format and post their task, but must also determine how long it should take and how much money they think it’s worth. Since task posters are already short on time by definition, this step becomes an unnecessary speed bump. Ziptask, again with its human team of supervisors, assigns prices to jobs automatically based on the difficulty and type of work. Since the labor is compensated per-minute, they’ll also determine a cutoff price to help you avoid going over budget. By contrast, AutoMan turns the process into trial-and-error based on a series of formulas. Price is calculated based on the duration of the work and federal minimum wage, and task time limits are set to 30 seconds by default. AutoMan will automatically adjust both the task price and time limit (upwards) if it’s not getting the results it requires. Clients can set these parameters to other defaults if the task requires, but the process is otherwise very standardized.

Quality Assurance

Any cloud labor platform, regardless of its makeup or the details of its process, will live and die by work quality. Who wants to pay for substandard results? Quality assurance is an absolute necessity, and MTurk has next to none built in. Ziptask once again turns to its supervision team, who personally make sure that every document is up to standards before presenting it to the client. The client provides the final pass/fail check, and no money changes hands until everyone agrees that the work makes the cut. AutoMan, by comparison, automates the process in the simplest possible way; it has multiple workers complete the task, and waits to see which results are the most common. The workers are paid once the majority has reached a statistically viable agreement, with no payment going to workers who provided incorrect answers.

Will My New Boss Be A Robot?

Rest assured, it’s probably not gonna happen anytime soon. The relative inflexibility of both the AutoMan algorithm and the MTurk interface mean that this combination is going to be very effective, but only for certain kinds of tasks. In a nutshell, this isn’t going to add any muscle to MTurk; it will continue to be bad at intricate or skill-based work, but good at work that’s just above “a monkey could do it”-level. The only difference is that the AutoMan algorithm could highly increase Turk’s effectiveness at completing these types of tasks. For all other office work, especially things that you can’t wait around for five or six workers to agree on, Ziptask is going to get you better results, faster, and most likely for a better price.

Ziptask: The Killer App of Cloud Labor

Friday, November 2nd, 2012 by Seth Weinstein

Quite an assertion, right? You’ve gotta be pretty bold to make the claim that your service is the be-all-end-all, better than the rest, the only one worth using. But in the case of Ziptask, it’s pretty close to the truth. We pride ourselves on a “whole package”-type deal, and that’s the key factor that sets us above all the other outlets. They may provide aspects of cloud labor or outsourcing, and some of them are very good at what they do, but if you want the full package, Ziptask is the only one that’s gonna do it for you.

Of course, I couldn’t make a claim like this without some evidence to back it up. So allow me to contrast Ziptask against some of our top competitors, so you can see exactly where Ziptask fills in the gaps.

Ziptask

Just so we have a baseline, allow me to describe the Total Ziptask Package. When you send your work to Ziptask, the only other thing you are required to do is provide instructions on how to complete said work and approve a price. Ziptask’s in-house team and pool of freelancers then work together, completely autonomously, to find the best possible worker for the job, assign them the work, establish a per-minute price estimate, check progress, and assure the quality of the completed work. For the person using Ziptask, the process looks something like this: submit the assignment and instructions, receive and approve a price estimate, go do something else for a while, and pay a fee when the work is completed to your satisfaction. Completely independent and hands-off.

oDesk/Elance

Let’s get this straight; here at Ziptask, we friggin’ love oDesk. We get a ton of our freelancers directly from their pool, and they and Elance are both very good at what they do. But what they do is simply provide a space for freelancers to gather. They don’t do any hiring, work assignments, cost estimates, or quality assurance in-house. These platforms do not handle any of the interaction between the freelancer and the hiring entity. They basically boil down to an online stack of resumés, albeit an easily-searchable one.

Amazon Mechanical Turk

With Turk, we see a shift towards the automation that Ziptask users enjoy so much. Once a user creates a job on Turk and submits it, the work is done automatically by whatever users are sufficiently qualified and motivated. The actual process of the work getting done can be unmonitored and hands-off, which is nice for simple work that doesn’t need a lot of skill. The downside is the enormous amount of effort and brainpower that it takes to set up a project on the platform. Even though great leaps have been made in usability, the mTurk interface is still very clunky and confusing for a newcomer. Add that to the standard headaches of finding a way to format your assignment on their system, deciding on a “sweet-spot” price point that will attract workers without breaking your wallet, and the fact that quality assurance is not guaranteed, and many will find that Turk is too much of a hassle for anything but the most rudimentary tasks.

A Real-Life, Flesh-And-Blood In-House Worker

Just for fun. I probably don’t have to describe what an in-house employee does; you most likely are one, or have several working for you. An in-house employees can offer a lot of advantages, but Ziptask still has them beat. A worker can only take so much work in a day, of course, and scalability is rough since it requires you to actually go through the process of locating, interviewing, hiring, negotiating with, and providing office space for a new employee. Additionally, many workers are skilled in one area and not so hot in others, meaning that if it’s versatility you’re after, you once again have to go to the ol’ Resumé Well. And lastly, most workers are either salaried or paid hourly, meaning that unless you’re monitoring them for the entire time they’re at work, you’re most likely paying for them to browse the Internet at some point or another.

I may be slightly biased, but from where I’m sitting, Ziptask looks like a pretty sweet deal. I like to describe it as a “black box”, where the only things you have to worry about are the input and the output. The rest is completely automated by Ziptask’s team.

And honestly, we could all use one less thing we have to worry about in our daily lives.

Oh, and if you know of a platform that compares favorably to Ziptask, be sure to let us know in the comments!

%d bloggers like this: