Validity Concerns in Research Using Organic Data

Heng Xu, Nan Zhang, Le Zhou

Research output: Contribution to journalArticlepeer-review

27 Scopus citations


With the advent of computing technologies, researchers across social science fields are using increasingly complex methods to collect, process, and analyze data in pursuit of scientific evidence. Given the complexity of research methods used, it is important to ensure that the research findings produced by a study are robust instead of being affected significantly by uncertainties associated with the design or implementation of the study. The field of metascience—the use of scientific methodology to study science itself—has examined various aspects of this robustness requirement for research that uses conventional designed studies (e.g., surveys, laboratory experiments) to collect data. Largely missing, however, are efforts to examine the robustness of empirical research using “organic data,” namely, data that are generated without any explicit research design elements and are continuously documented by digital devices (e.g., video captured by ubiquitous sensing devices; content and social interactions extracted from social networking sites, Twitter feeds, and click streams). Given the growing popularity of using organic data in management research, it is essential to understand issues concerning the usage and processing of organic data that may affect the robustness of research findings. This commentary first provides an overview of commonly present issues that threaten the validity of inferences drawn from empirical studies using organic data. This is followed by a discussion on some key considerations and suggestions for making organic data a robust and integral part of future research endeavors in management.

Original languageEnglish (US)
Pages (from-to)1257-1274
Number of pages18
JournalJournal of Management
Issue number7
StatePublished - Sep 1 2020

Bibliographical note

Funding Information:
The authors are deeply grateful to Mo Wang and Gwendolyn K. Lee for their encouragement and guidance in the process of developing this manuscript. They are also grateful for helpful feedback and comments from the two anonymous reviewers as well as the participants of NSF Workshop on Promoting Robust and Reliable Research Practice. The authors would also like to thank Elizabeth M. Campbell and John D. Kammeyer-Mueller for their valuable comments and suggestions. Heng Xu’s and Nan Zhang’s work on this manuscript was supported in part by the National Science Foundation under Grant 1760059. Le Zhou’s work on this manuscript was supported in part by the National Science Foundation under Grant 1734134. Any opinions, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.

Publisher Copyright:
© The Author(s) 2019.


  • open science (e.g., transparency in research practices)
  • replication studies
  • research design
  • research methods


Dive into the research topics of 'Validity Concerns in Research Using Organic Data'. Together they form a unique fingerprint.

Cite this