Collaboratively Crowdsourcing Workflows with Turkomatic
To be published in the proceedings of CSCW2012:
Completing work on crowdsourcing marketplaces like Amazon's Mechanical Turk requires careful attention to workflow design, the problem of dividing problems among several workers to ensure accurate and effective completion of work. We present Turkomatic, a new tool for creating workflows that recruits workers from the crowd to aid requesters in planning and solving complex tasks. Requesters can view the status of workflows in real time, intervene to change workflow tasks and solutions, and request new solutions to subtasks from the crowd. Unlike existing approaches to workflow design, Turkomatic does not require requesters to program workflows, giving nontechnical users the ability to create workflows rapidly. Our evaluation examines how members of the crowd interacted while planning tasks. We also examined online post-hoc correction by requesters as an alternative to advance planning. We argue that this collaborative approach can be more efficient than the conventional workflow design process and discuss several implications for the design of collaborative crowd planning systems.