Towards more Accessible Precision Medicine

Building a more Transferable Machine Learning Model to Support Prognostic Decisions for Micro- and Macrovascular Complications of Type 2 Diabetes Mellitus

Era Kim, Pedro J. Caraballo, M. Regina Castro, David S Pieczkiewicz, Gyorgy J Simon

Research output: Contribution to journalArticle

1 Citation (Scopus)

Abstract

Although machine learning models are increasingly being developed for clinical decision support for patients with type 2 diabetes, the adoption of these models into clinical practice remains limited. Currently, machine learning (ML) models are being constructed on local healthcare systems and are validated internally with no expectation that they would validate externally and thus, are rarely transferrable to a different healthcare system. In this work, we aim to demonstrate that (1) even a complex ML model built on a national cohort can be transferred to two local healthcare systems, (2) while a model constructed on a local healthcare system’s cohort is difficult to transfer; (3) we examine the impact of training cohort size on the transferability; and (4) we discuss criteria for external validity. We built a model using our previously published Multi-Task Learning-based methodology on a national cohort extracted from OptumLabs® Data Warehouse and transferred the model to two local healthcare systems (i.e., University of Minnesota Medical Center and Mayo Clinic) for external evaluation. The model remained valid when applied to the local patient populations and performed as well as locally constructed models (concordance:.73–.92), demonstrating transferability. The performance of the locally constructed models reduced substantially when applied to each other’s healthcare system (concordance:.62–.90). We believe that our modeling approach, in which a model is learned from a national cohort and is externally validated, produces a transferable model, allowing patients at smaller healthcare systems to benefit from precision medicine.

Original languageEnglish (US)
Article number185
JournalJournal of Medical Systems
Volume43
Issue number7
DOIs
StatePublished - Jul 1 2019

Fingerprint

Precision Medicine
Medical problems
Type 2 Diabetes Mellitus
Medicine
Learning systems
Delivery of Health Care
Clinical Decision Support Systems
Machine Learning
Learning
Data warehouses
Population

Keywords

  • Complications of type 2 diabetes
  • External validation
  • Large national data
  • Machine learning
  • Precision medicine
  • Transferable model

PubMed: MeSH publication types

  • Journal Article

Cite this

@article{81e9fe7d0836401eb02156b8fed44a35,
title = "Towards more Accessible Precision Medicine: Building a more Transferable Machine Learning Model to Support Prognostic Decisions for Micro- and Macrovascular Complications of Type 2 Diabetes Mellitus",
abstract = "Although machine learning models are increasingly being developed for clinical decision support for patients with type 2 diabetes, the adoption of these models into clinical practice remains limited. Currently, machine learning (ML) models are being constructed on local healthcare systems and are validated internally with no expectation that they would validate externally and thus, are rarely transferrable to a different healthcare system. In this work, we aim to demonstrate that (1) even a complex ML model built on a national cohort can be transferred to two local healthcare systems, (2) while a model constructed on a local healthcare system’s cohort is difficult to transfer; (3) we examine the impact of training cohort size on the transferability; and (4) we discuss criteria for external validity. We built a model using our previously published Multi-Task Learning-based methodology on a national cohort extracted from OptumLabs{\circledR} Data Warehouse and transferred the model to two local healthcare systems (i.e., University of Minnesota Medical Center and Mayo Clinic) for external evaluation. The model remained valid when applied to the local patient populations and performed as well as locally constructed models (concordance:.73–.92), demonstrating transferability. The performance of the locally constructed models reduced substantially when applied to each other’s healthcare system (concordance:.62–.90). We believe that our modeling approach, in which a model is learned from a national cohort and is externally validated, produces a transferable model, allowing patients at smaller healthcare systems to benefit from precision medicine.",
keywords = "Complications of type 2 diabetes, External validation, Large national data, Machine learning, Precision medicine, Transferable model",
author = "Era Kim and Caraballo, {Pedro J.} and Castro, {M. Regina} and Pieczkiewicz, {David S} and Simon, {Gyorgy J}",
year = "2019",
month = "7",
day = "1",
doi = "10.1007/s10916-019-1321-6",
language = "English (US)",
volume = "43",
journal = "Journal of Medical Systems",
issn = "0148-5598",
publisher = "Springer New York",
number = "7",

}

TY - JOUR

T1 - Towards more Accessible Precision Medicine

T2 - Building a more Transferable Machine Learning Model to Support Prognostic Decisions for Micro- and Macrovascular Complications of Type 2 Diabetes Mellitus

AU - Kim, Era

AU - Caraballo, Pedro J.

AU - Castro, M. Regina

AU - Pieczkiewicz, David S

AU - Simon, Gyorgy J

PY - 2019/7/1

Y1 - 2019/7/1

N2 - Although machine learning models are increasingly being developed for clinical decision support for patients with type 2 diabetes, the adoption of these models into clinical practice remains limited. Currently, machine learning (ML) models are being constructed on local healthcare systems and are validated internally with no expectation that they would validate externally and thus, are rarely transferrable to a different healthcare system. In this work, we aim to demonstrate that (1) even a complex ML model built on a national cohort can be transferred to two local healthcare systems, (2) while a model constructed on a local healthcare system’s cohort is difficult to transfer; (3) we examine the impact of training cohort size on the transferability; and (4) we discuss criteria for external validity. We built a model using our previously published Multi-Task Learning-based methodology on a national cohort extracted from OptumLabs® Data Warehouse and transferred the model to two local healthcare systems (i.e., University of Minnesota Medical Center and Mayo Clinic) for external evaluation. The model remained valid when applied to the local patient populations and performed as well as locally constructed models (concordance:.73–.92), demonstrating transferability. The performance of the locally constructed models reduced substantially when applied to each other’s healthcare system (concordance:.62–.90). We believe that our modeling approach, in which a model is learned from a national cohort and is externally validated, produces a transferable model, allowing patients at smaller healthcare systems to benefit from precision medicine.

AB - Although machine learning models are increasingly being developed for clinical decision support for patients with type 2 diabetes, the adoption of these models into clinical practice remains limited. Currently, machine learning (ML) models are being constructed on local healthcare systems and are validated internally with no expectation that they would validate externally and thus, are rarely transferrable to a different healthcare system. In this work, we aim to demonstrate that (1) even a complex ML model built on a national cohort can be transferred to two local healthcare systems, (2) while a model constructed on a local healthcare system’s cohort is difficult to transfer; (3) we examine the impact of training cohort size on the transferability; and (4) we discuss criteria for external validity. We built a model using our previously published Multi-Task Learning-based methodology on a national cohort extracted from OptumLabs® Data Warehouse and transferred the model to two local healthcare systems (i.e., University of Minnesota Medical Center and Mayo Clinic) for external evaluation. The model remained valid when applied to the local patient populations and performed as well as locally constructed models (concordance:.73–.92), demonstrating transferability. The performance of the locally constructed models reduced substantially when applied to each other’s healthcare system (concordance:.62–.90). We believe that our modeling approach, in which a model is learned from a national cohort and is externally validated, produces a transferable model, allowing patients at smaller healthcare systems to benefit from precision medicine.

KW - Complications of type 2 diabetes

KW - External validation

KW - Large national data

KW - Machine learning

KW - Precision medicine

KW - Transferable model

UR - http://www.scopus.com/inward/record.url?scp=85065914810&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85065914810&partnerID=8YFLogxK

U2 - 10.1007/s10916-019-1321-6

DO - 10.1007/s10916-019-1321-6

M3 - Article

VL - 43

JO - Journal of Medical Systems

JF - Journal of Medical Systems

SN - 0148-5598

IS - 7

M1 - 185

ER -