One data set, many analysts: Implications for practicing scientists

Research output: Contribution to journalArticlepeer-review


Researchers routinely face choices throughout the data analysis process. It is often opaque to readers how these choices are made, how they affect the findings, and whether or not data analysis results are unduly influenced by subjective decisions. This concern is spurring numerous investigations into the variability of data analysis results. The findings demonstrate that different teams analyzing the same data may reach different conclusions. This is the “many-analysts” problem. Previous research on the many-analysts problem focused on demonstrating its existence, without identifying specific practices for solving it. We address this gap by identifying three pitfalls that have contributed to the variability observed in many-analysts publications and providing suggestions on how to avoid them.

Original languageEnglish (US)
Article number1094150
JournalFrontiers in Psychology
StatePublished - 2023

Bibliographical note

Funding Information:
EK was supported by funding from Grant No. NCRR 1UL1TR002494-01 and GJ was partially supported by NSF grant DMS-2152746. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Publisher Copyright:
Copyright © 2023 Kummerfeld and Jones.


  • data analysis
  • metascience
  • multilab analysis
  • reproducibility
  • statistical problem-solving process

PubMed: MeSH publication types

  • Journal Article


Dive into the research topics of 'One data set, many analysts: Implications for practicing scientists'. Together they form a unique fingerprint.

Cite this