Applying Machine Learning Across Sites: External Validation of a Surgical Site Infection Detection Algorithm

Ying Zhu, Gyorgy J. Simon, Elizabeth C. Wick, Yumiko Abe-Jones, Nader Najafi, Adam Sheka, Roshan Tourani, Steven J. Skube, Zhen Hu, Genevieve B. Melton

Research output: Contribution to journalArticlepeer-review

11 Scopus citations


Background: Surgical complications have tremendous consequences and costs. Complication detection is important for quality improvement, but traditional manual chart review is burdensome. Automated mechanisms are needed to make this more efficient. To understand the generalizability of a machine learning algorithm between sites, automated surgical site infection (SSI) detection algorithms developed at one center were tested at another distinct center. Study design: NSQIP patients had electronic health record (EHR) data extracted at one center (University of Minnesota Medical Center, Site A) over a 4-year period for model development and internal validation, and at a second center (University of California San Francisco, Site B) over a subsequent 2-year period for external validation. Models for automated NSQIP SSI detection of superficial, organ space, and total SSI within 30 days postoperatively were validated using area under the curve (AUC) scores and corresponding 95% confidence intervals. Results: For the 8,883 patients (Site A) and 1,473 patients (Site B), AUC scores were not statistically different for any outcome including superficial (external 0.804, internal [0.784, 0.874] AUC); organ/space (external 0.905, internal [0.867, 0.941] AUC); and total (external 0.855, internal [0.854, 0.908] AUC) SSI. False negative rates decreased with increasing case review volume and would be amenable to a strategy in which cases with low predicted probabilities of SSI could be excluded from chart review. Conclusions: Our findings demonstrated that SSI detection machine learning algorithms developed at 1 site were generalizable to another institution. SSI detection models are practically applicable to accelerate and focus chart review.

Original languageEnglish (US)
Pages (from-to)963-971.e1
JournalJournal of the American College of Surgeons
Issue number6
Early online dateApr 5 2021
StatePublished - Jun 1 2021

Bibliographical note

Funding Information:
Support: This research was supported by the University of Minnesota Academic Health Center Faculty Development Award, Agency for Healthcare Research and Quality (R01HS24532), NIH Clinical and Translational Science Award program (UL1TR000114), NIH National Institute of General Medical Sciences (R01GM120079), Fairview Health Services, University of Minnesota Physicians, and University of California, San Francisco Medical Center.

Publisher Copyright:
© 2021 American College of Surgeons


Dive into the research topics of 'Applying Machine Learning Across Sites: External Validation of a Surgical Site Infection Detection Algorithm'. Together they form a unique fingerprint.

Cite this