Effective Strategies for Crowd-Powered Cognitive Reappraisal Systems: A Field Deployment of the Flip∗Doubt Web Application for Mental Health

Estelle Smith, William Lane, Hannah Miller Hillberg, Daniel Kluver, Loren Terveen, Svetlana Yarosh

Research output: Contribution to journalArticlepeer-review

3 Scopus citations


Online technologies offer great promise to expand models of delivery for therapeutic interventions to help users cope with increasingly common mental illnesses like anxiety and depression. For example, "cognitive reappraisal"is a skill that involves changing one's perspective on negative thoughts in order to improve one's emotional state. In this work, we present Flip∗Doubt, a novel crowd-powered web application that provides users with cognitive reappraisals ("reframes") of negative thoughts. A one-month field deployment of Flip∗Doubt with 13 graduate students yielded a data set of negative thoughts paired with positive reframes, as well as rich interview data about how participants interacted with the system. Through this deployment, our work contributes: (1) an in-depth qualitative understanding of how participants used a crowd-powered cognitive reappraisal system in the wild; and (2) detailed codebooks that capture informative context about negative input thoughts and reframes. Our results surface data-derived hypotheses that may help to explain what types of reframes are helpful for users, while also providing guidance to future researchers and developers interested in building collaborative systems for mental health. In our discussion, we outline implications for systems research to leverage peer training and support, as well as opportunities to integrate AI/ML-based algorithms to support the cognitive reappraisal task. (Note: This paper includes potentially triggering mentions of mental health issues and suicide.)

Original languageEnglish (US)
Article number417
JournalProceedings of the ACM on Human-Computer Interaction
Issue numberCSCW2
StatePublished - Oct 18 2021

Bibliographical note

Funding Information:
We thank our participants for sharing their thoughts and our reviewers for their supportive feedback. We are grateful to Xinyi Wang and Karan Jaswani who made excellent coding contributions to a preliminary Flip*Doubt prototype built for a web development course taught by our wonderful instructor, F. Maxwell Harper. We also thank Stephen Schueller for his preliminary comments that helped shape our research direction, and our undergraduate research assistants, Benjamin Wiley, Eric Sortland, Shubhavi Arya, and Ishan Joshi. This work was partially funded by the University of Minnesota Social Media Business Analytics Collaborative (SOBACO) faculty award.

Publisher Copyright:
© 2021 ACM.


  • amazon mechanical turk
  • cognitive reappraisal
  • crowdsourcing
  • human-centered machine learning
  • mental health
  • online health communities
  • peer support
  • social support


Dive into the research topics of 'Effective Strategies for Crowd-Powered Cognitive Reappraisal Systems: A Field Deployment of the Flip∗Doubt Web Application for Mental Health'. Together they form a unique fingerprint.

Cite this