Suicide risk and protective factors in online support forum posts: annotation scheme development and validation study

Stevie Chancellor, Steven A. Sumner, Corinne David-Ferdon, Tahirah Ahmad, Munmun de Choudhury

Research output: Contribution to journalArticlepeer-review


Background: Online communities provide support for individuals looking for help with suicidal ideation and crisis. As community data are increasingly used to devise machine learning models to infer who might be at risk, there have been limited efforts to identify both risk and protective factors in web-based posts. These annotations can enrich and augment computational assessment approaches to identify appropriate intervention points, which are useful to public health professionals and suicide prevention researchers. Objective: This qualitative study aims to develop a valid and reliable annotation scheme for evaluating risk and protective factors for suicidal ideation in posts in suicide crisis forums. Methods: We designed a valid, reliable, and clinically grounded process for identifying risk and protective markers in social media data. This scheme draws on prior work on construct validity and the social sciences of measurement. We then applied the scheme to annotate 200 posts from r/SuicideWatch—a Reddit community focused on suicide crisis. Results: We documented our results on producing an annotation scheme that is consistent with leading public health information coding schemes for suicide and advances attention to protective factors. Our study showed high internal validity, and we have presented results that indicate that our approach is consistent with findings from prior work. Conclusions: Our work formalizes a framework that incorporates construct validity into the development of annotation schemes for suicide risk on social media. This study furthers the understanding of risk and protective factors expressed in social media data. This may help public health programming to prevent suicide and computational social science research and investigations that rely on the quality of labels for downstream machine learning tasks.

Original languageEnglish (US)
Article numbere24471
JournalJMIR Mental Health
Issue number11
StatePublished - Nov 2021

Bibliographical note

Funding Information:
SC and MDC were supported in part by National Institutes of Health grant #R01GM112697-01. SC completed much of this work while she was at the Georgia Institute of Technology and Northwestern University. The authors would like to thank Michael Xiao for the early piloting of the project, Harman Kaur and Jacob Thebault-Spieker for feedback on early drafts, and the University of Minnesota Institute for Research in Statistics and its Applications for assistance with factor analysis and correlations.

Publisher Copyright:
© Stevie Chancellor, Steven A Sumner, Corinne David-Ferdon, Tahirah Ahmad, Munmun De Choudhury. Originally published in JMIR Mental Health (, 08.11.2021. This is an open-access article distributed under the terms of the Creative Commons Attribution License.


  • Annotation
  • Annotation scheme
  • Construct validity
  • Online communities
  • Reddit
  • Suicide crisis


Dive into the research topics of 'Suicide risk and protective factors in online support forum posts: annotation scheme development and validation study'. Together they form a unique fingerprint.

Cite this