Skip to main navigation Skip to search Skip to main content

Understanding Confusion: A Case Study of Training a Machine Model to Predict and Interpret Consensus From Volunteer Labels

  • Ramanakumar Sankar
  • , Kameswara Mantha
  • , Cooper Nesmith
  • , Lucy Fortson
  • , Shawn Brueshaber
  • , Candice Hansen-Koharcheck
  • , Glenn Orton

Research output: Contribution to journalArticlepeer-review

Abstract

Citizen science has become a valuable and reliable method for interpreting and processing big datasets, and is vital in the era of ever-growing data volumes. However, there are inherent difficulties in the generating labels from citizen scientists, due to the inherent variability between the members of the crowd, leading to variability in the results. Sometimes, this is useful — such as with serendipitous discoveries, which corresponds to rare/unknown classes in the data — but it might also be due to ambiguity between classes. The primary issue is then to distinguish between the intrinsic variability in the dataset and the uncertainty in the citizen scientists’ responses, and leveraging that to extract scientifically useful relationships. In this paper, we explore using a neural network to interpret volunteer confusion across the dataset, to increase the purity of the downstream analysis. We focus on the use of learned features from the network to disentangle feature similarity across the classes, and the ability of the machines’ “attention” in identifying features that lead to confusion. We use data from Jovian Vortex Hunter, a citizen science project to study vortices in Jupiter’s atmosphere, and find that the latent space from the model helps effectively identify different sources of image-level features that lead to low volunteer consensus. Furthermore, the machine’s attention highlights features corresponding to specific classes. This provides meaningful image-level feature-class relationships, which is useful in our analysis for identifying vortex-specific features to better understand vortex evolution mechanisms. Finally, we discuss the applicability of this method to other citizen science projects.

Original languageEnglish (US)
Article number41
JournalCitizen Science: Theory and Practice
Volume9
Issue number1
DOIs
StatePublished - 2024

Bibliographical note

Publisher Copyright:
© 2024 The Author(s).

Keywords

  • auto-encoder
  • citizen science
  • crowdsourcing
  • machine attention
  • planetary atmospheres
  • semi-supervised network

Fingerprint

Dive into the research topics of 'Understanding Confusion: A Case Study of Training a Machine Model to Predict and Interpret Consensus From Volunteer Labels'. Together they form a unique fingerprint.

Cite this