Abstract
Consistency is a practical metric that evaluates an instrument's reliability based on its ability to yield the same output when repeatedly given a particular input. Despite its broad usage, little is understood about the feasibility of using consistency as a measure of worker reliability in crowdwork. In this paper, we explore the viability of measuring a worker's reliability by their ability to conform to themselves. We introduce and describe Deja Vu, a mechanism for dynamically generating task queues with consistency probes to measure the consistency of workers who repeat the same task twice. We present a study that utilizes Deja Vu to examine how generic characteristics of the duplicate task - such as placement, difficulty, and transformation - affect a workers task consistency in the context of two unique object detection tasks. Our findings provide insight into the design and use of consistency-based reliability metrics.
| Original language | English (US) |
|---|---|
| Title of host publication | Proceedings of the 5th AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2017 |
| Editors | Steven Dow, Adam Tauman |
| Publisher | AAAI press |
| Pages | 197-205 |
| Number of pages | 9 |
| ISBN (Electronic) | 9781577357933 |
| State | Published - Oct 27 2017 |
| Externally published | Yes |
| Event | 5th AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2017 - Quebec City, Canada Duration: Oct 24 2017 → Oct 26 2017 |
Publication series
| Name | Proceedings of the 5th AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2017 |
|---|
Conference
| Conference | 5th AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2017 |
|---|---|
| Country/Territory | Canada |
| City | Quebec City |
| Period | 10/24/17 → 10/26/17 |
Bibliographical note
Publisher Copyright:Copyright © 2017, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved.