Use and Customization of Risk Scores for Predicting Cardiovascular Events Using Electronic Health Record Data

Julian Wolfson, David M Vock, Sunayan Bandyopadhyay, Thomas Kottke, Gabriela Vazquez-Benitez, Paul Johnson, Gediminas Adomavicius, Patrick J. O'Connor

Research output: Contribution to journalArticle

16 Citations (Scopus)

Abstract

BACKGROUND: Clinicians who are using the Framingham Risk Score (FRS) or the American College of Cardiology/American Heart Association Pooled Cohort Equations (PCE) to estimate risk for their patients based on electronic health data (EHD) face 4 questions. (1) Do published risk scores applied to EHD yield accurate estimates of cardiovascular risk? (2) Are FRS risk estimates, which are based on data that are up to 45 years old, valid for a contemporary patient population seeking routine care? (3) Do the PCE make the FRS obsolete? (4) Does refitting the risk score using EHD improve the accuracy of risk estimates?

METHODS AND RESULTS: Data were extracted from the EHD of 84 116 adults aged 40 to 79 years who received care at a large healthcare delivery and insurance organization between 2001 and 2011. We assessed calibration and discrimination for 4 risk scores: published versions of FRS and PCE and versions obtained by refitting models using a subset of the available EHD. The published FRS was well calibrated (calibration statistic K=9.1, miscalibration ranging from 0% to 17% across risk groups), but the PCE displayed modest evidence of miscalibration (calibration statistic K=43.7, miscalibration from 9% to 31%). Discrimination was similar in both models (C-index=0.740 for FRS, 0.747 for PCE). Refitting the published models using EHD did not substantially improve calibration or discrimination.

CONCLUSIONS: We conclude that published cardiovascular risk models can be successfully applied to EHD to estimate cardiovascular risk; the FRS remains valid and is not obsolete; and model refitting does not meaningfully improve the accuracy of risk estimates.

Original languageEnglish (US)
JournalJournal of the American Heart Association
Volume6
Issue number4
DOIs
StatePublished - Apr 24 2017

Fingerprint

Electronic Health Records
Health
Calibration
Cardiovascular Models

Keywords

  • electronic health record data
  • risk prediction
  • risk score

Cite this

Use and Customization of Risk Scores for Predicting Cardiovascular Events Using Electronic Health Record Data. / Wolfson, Julian; Vock, David M; Bandyopadhyay, Sunayan; Kottke, Thomas; Vazquez-Benitez, Gabriela; Johnson, Paul; Adomavicius, Gediminas; O'Connor, Patrick J.

In: Journal of the American Heart Association, Vol. 6, No. 4, 24.04.2017.

Research output: Contribution to journalArticle

Wolfson, Julian ; Vock, David M ; Bandyopadhyay, Sunayan ; Kottke, Thomas ; Vazquez-Benitez, Gabriela ; Johnson, Paul ; Adomavicius, Gediminas ; O'Connor, Patrick J. / Use and Customization of Risk Scores for Predicting Cardiovascular Events Using Electronic Health Record Data. In: Journal of the American Heart Association. 2017 ; Vol. 6, No. 4.
@article{a586d43d04424a6e84681017d58a2da6,
title = "Use and Customization of Risk Scores for Predicting Cardiovascular Events Using Electronic Health Record Data",
abstract = "BACKGROUND: Clinicians who are using the Framingham Risk Score (FRS) or the American College of Cardiology/American Heart Association Pooled Cohort Equations (PCE) to estimate risk for their patients based on electronic health data (EHD) face 4 questions. (1) Do published risk scores applied to EHD yield accurate estimates of cardiovascular risk? (2) Are FRS risk estimates, which are based on data that are up to 45 years old, valid for a contemporary patient population seeking routine care? (3) Do the PCE make the FRS obsolete? (4) Does refitting the risk score using EHD improve the accuracy of risk estimates?METHODS AND RESULTS: Data were extracted from the EHD of 84 116 adults aged 40 to 79 years who received care at a large healthcare delivery and insurance organization between 2001 and 2011. We assessed calibration and discrimination for 4 risk scores: published versions of FRS and PCE and versions obtained by refitting models using a subset of the available EHD. The published FRS was well calibrated (calibration statistic K=9.1, miscalibration ranging from 0{\%} to 17{\%} across risk groups), but the PCE displayed modest evidence of miscalibration (calibration statistic K=43.7, miscalibration from 9{\%} to 31{\%}). Discrimination was similar in both models (C-index=0.740 for FRS, 0.747 for PCE). Refitting the published models using EHD did not substantially improve calibration or discrimination.CONCLUSIONS: We conclude that published cardiovascular risk models can be successfully applied to EHD to estimate cardiovascular risk; the FRS remains valid and is not obsolete; and model refitting does not meaningfully improve the accuracy of risk estimates.",
keywords = "electronic health record data, risk prediction, risk score",
author = "Julian Wolfson and Vock, {David M} and Sunayan Bandyopadhyay and Thomas Kottke and Gabriela Vazquez-Benitez and Paul Johnson and Gediminas Adomavicius and O'Connor, {Patrick J.}",
year = "2017",
month = "4",
day = "24",
doi = "10.1161/JAHA.116.003670",
language = "English (US)",
volume = "6",
journal = "Journal of the American Heart Association",
issn = "2047-9980",
publisher = "Wiley-Blackwell",
number = "4",

}

TY - JOUR

T1 - Use and Customization of Risk Scores for Predicting Cardiovascular Events Using Electronic Health Record Data

AU - Wolfson, Julian

AU - Vock, David M

AU - Bandyopadhyay, Sunayan

AU - Kottke, Thomas

AU - Vazquez-Benitez, Gabriela

AU - Johnson, Paul

AU - Adomavicius, Gediminas

AU - O'Connor, Patrick J.

PY - 2017/4/24

Y1 - 2017/4/24

N2 - BACKGROUND: Clinicians who are using the Framingham Risk Score (FRS) or the American College of Cardiology/American Heart Association Pooled Cohort Equations (PCE) to estimate risk for their patients based on electronic health data (EHD) face 4 questions. (1) Do published risk scores applied to EHD yield accurate estimates of cardiovascular risk? (2) Are FRS risk estimates, which are based on data that are up to 45 years old, valid for a contemporary patient population seeking routine care? (3) Do the PCE make the FRS obsolete? (4) Does refitting the risk score using EHD improve the accuracy of risk estimates?METHODS AND RESULTS: Data were extracted from the EHD of 84 116 adults aged 40 to 79 years who received care at a large healthcare delivery and insurance organization between 2001 and 2011. We assessed calibration and discrimination for 4 risk scores: published versions of FRS and PCE and versions obtained by refitting models using a subset of the available EHD. The published FRS was well calibrated (calibration statistic K=9.1, miscalibration ranging from 0% to 17% across risk groups), but the PCE displayed modest evidence of miscalibration (calibration statistic K=43.7, miscalibration from 9% to 31%). Discrimination was similar in both models (C-index=0.740 for FRS, 0.747 for PCE). Refitting the published models using EHD did not substantially improve calibration or discrimination.CONCLUSIONS: We conclude that published cardiovascular risk models can be successfully applied to EHD to estimate cardiovascular risk; the FRS remains valid and is not obsolete; and model refitting does not meaningfully improve the accuracy of risk estimates.

AB - BACKGROUND: Clinicians who are using the Framingham Risk Score (FRS) or the American College of Cardiology/American Heart Association Pooled Cohort Equations (PCE) to estimate risk for their patients based on electronic health data (EHD) face 4 questions. (1) Do published risk scores applied to EHD yield accurate estimates of cardiovascular risk? (2) Are FRS risk estimates, which are based on data that are up to 45 years old, valid for a contemporary patient population seeking routine care? (3) Do the PCE make the FRS obsolete? (4) Does refitting the risk score using EHD improve the accuracy of risk estimates?METHODS AND RESULTS: Data were extracted from the EHD of 84 116 adults aged 40 to 79 years who received care at a large healthcare delivery and insurance organization between 2001 and 2011. We assessed calibration and discrimination for 4 risk scores: published versions of FRS and PCE and versions obtained by refitting models using a subset of the available EHD. The published FRS was well calibrated (calibration statistic K=9.1, miscalibration ranging from 0% to 17% across risk groups), but the PCE displayed modest evidence of miscalibration (calibration statistic K=43.7, miscalibration from 9% to 31%). Discrimination was similar in both models (C-index=0.740 for FRS, 0.747 for PCE). Refitting the published models using EHD did not substantially improve calibration or discrimination.CONCLUSIONS: We conclude that published cardiovascular risk models can be successfully applied to EHD to estimate cardiovascular risk; the FRS remains valid and is not obsolete; and model refitting does not meaningfully improve the accuracy of risk estimates.

KW - electronic health record data

KW - risk prediction

KW - risk score

UR - http://www.scopus.com/inward/record.url?scp=85042659208&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85042659208&partnerID=8YFLogxK

U2 - 10.1161/JAHA.116.003670

DO - 10.1161/JAHA.116.003670

M3 - Article

C2 - 28438733

AN - SCOPUS:85042659208

VL - 6

JO - Journal of the American Heart Association

JF - Journal of the American Heart Association

SN - 2047-9980

IS - 4

ER -