Performance of the Net Reclassification Improvement for Nonnested Models and a Novel Percentile-Based Alternative

Shannon B. McKearnan, Julian Wolfson, David M Vock, Gabriela Vazquez-Benitez, Patrick J. O'Connor

Research output: Contribution to journalArticle

1 Citation (Scopus)

Abstract

The net reclassification improvement (NRI) is a widely used metric used to assess the relative ability of 2 risk models to distinguish between low- and high-risk individuals. However, the validity and usefulness of the NRI have been questioned. Criticism of the NRI focuses on its use comparing nested risk models, whereas in practice it is often used to compare nonnested risk models derived from distinct data sources. In this study, we evaluated the performance of the NRI in a nonnested context by using it to compare competing cardiovascular risk-prediction models. We explored the NRI's sensitivity to variations in risk categories and to the calibration of the compared models. We found that the NRI was very sensitive to changes in the definition of risk categories, especially when at least 1 model was miscalibrated. To address these shortcomings, we describe a novel alternative to the usual NRI that uses percentiles of risk instead of cutoffs based on absolute risk. This percentile-based NRI demonstrates the relative ability of 2 models to rank patient risk. It displays more stable behavior, and we recommend its use when there are no established risk categories or when models are miscalibrated.

Original languageEnglish (US)
Pages (from-to)1327-1335
Number of pages9
JournalAmerican journal of epidemiology
Volume187
Issue number6
DOIs
StatePublished - Jun 1 2018

Fingerprint

Information Storage and Retrieval
Calibration

Keywords

  • discrimination
  • electronic health data
  • model comparison
  • net reclassification improvement
  • reclassification
  • risk assessment

Cite this

Performance of the Net Reclassification Improvement for Nonnested Models and a Novel Percentile-Based Alternative. / McKearnan, Shannon B.; Wolfson, Julian; Vock, David M; Vazquez-Benitez, Gabriela; O'Connor, Patrick J.

In: American journal of epidemiology, Vol. 187, No. 6, 01.06.2018, p. 1327-1335.

Research output: Contribution to journalArticle

McKearnan, Shannon B. ; Wolfson, Julian ; Vock, David M ; Vazquez-Benitez, Gabriela ; O'Connor, Patrick J. / Performance of the Net Reclassification Improvement for Nonnested Models and a Novel Percentile-Based Alternative. In: American journal of epidemiology. 2018 ; Vol. 187, No. 6. pp. 1327-1335.
@article{189a983cf6dc4ff2883775aecaa70db7,
title = "Performance of the Net Reclassification Improvement for Nonnested Models and a Novel Percentile-Based Alternative",
abstract = "The net reclassification improvement (NRI) is a widely used metric used to assess the relative ability of 2 risk models to distinguish between low- and high-risk individuals. However, the validity and usefulness of the NRI have been questioned. Criticism of the NRI focuses on its use comparing nested risk models, whereas in practice it is often used to compare nonnested risk models derived from distinct data sources. In this study, we evaluated the performance of the NRI in a nonnested context by using it to compare competing cardiovascular risk-prediction models. We explored the NRI's sensitivity to variations in risk categories and to the calibration of the compared models. We found that the NRI was very sensitive to changes in the definition of risk categories, especially when at least 1 model was miscalibrated. To address these shortcomings, we describe a novel alternative to the usual NRI that uses percentiles of risk instead of cutoffs based on absolute risk. This percentile-based NRI demonstrates the relative ability of 2 models to rank patient risk. It displays more stable behavior, and we recommend its use when there are no established risk categories or when models are miscalibrated.",
keywords = "discrimination, electronic health data, model comparison, net reclassification improvement, reclassification, risk assessment",
author = "McKearnan, {Shannon B.} and Julian Wolfson and Vock, {David M} and Gabriela Vazquez-Benitez and O'Connor, {Patrick J.}",
year = "2018",
month = "6",
day = "1",
doi = "10.1093/aje/kwx374",
language = "English (US)",
volume = "187",
pages = "1327--1335",
journal = "American Journal of Epidemiology",
issn = "0002-9262",
publisher = "Oxford University Press",
number = "6",

}

TY - JOUR

T1 - Performance of the Net Reclassification Improvement for Nonnested Models and a Novel Percentile-Based Alternative

AU - McKearnan, Shannon B.

AU - Wolfson, Julian

AU - Vock, David M

AU - Vazquez-Benitez, Gabriela

AU - O'Connor, Patrick J.

PY - 2018/6/1

Y1 - 2018/6/1

N2 - The net reclassification improvement (NRI) is a widely used metric used to assess the relative ability of 2 risk models to distinguish between low- and high-risk individuals. However, the validity and usefulness of the NRI have been questioned. Criticism of the NRI focuses on its use comparing nested risk models, whereas in practice it is often used to compare nonnested risk models derived from distinct data sources. In this study, we evaluated the performance of the NRI in a nonnested context by using it to compare competing cardiovascular risk-prediction models. We explored the NRI's sensitivity to variations in risk categories and to the calibration of the compared models. We found that the NRI was very sensitive to changes in the definition of risk categories, especially when at least 1 model was miscalibrated. To address these shortcomings, we describe a novel alternative to the usual NRI that uses percentiles of risk instead of cutoffs based on absolute risk. This percentile-based NRI demonstrates the relative ability of 2 models to rank patient risk. It displays more stable behavior, and we recommend its use when there are no established risk categories or when models are miscalibrated.

AB - The net reclassification improvement (NRI) is a widely used metric used to assess the relative ability of 2 risk models to distinguish between low- and high-risk individuals. However, the validity and usefulness of the NRI have been questioned. Criticism of the NRI focuses on its use comparing nested risk models, whereas in practice it is often used to compare nonnested risk models derived from distinct data sources. In this study, we evaluated the performance of the NRI in a nonnested context by using it to compare competing cardiovascular risk-prediction models. We explored the NRI's sensitivity to variations in risk categories and to the calibration of the compared models. We found that the NRI was very sensitive to changes in the definition of risk categories, especially when at least 1 model was miscalibrated. To address these shortcomings, we describe a novel alternative to the usual NRI that uses percentiles of risk instead of cutoffs based on absolute risk. This percentile-based NRI demonstrates the relative ability of 2 models to rank patient risk. It displays more stable behavior, and we recommend its use when there are no established risk categories or when models are miscalibrated.

KW - discrimination

KW - electronic health data

KW - model comparison

KW - net reclassification improvement

KW - reclassification

KW - risk assessment

UR - http://www.scopus.com/inward/record.url?scp=85049010801&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85049010801&partnerID=8YFLogxK

U2 - 10.1093/aje/kwx374

DO - 10.1093/aje/kwx374

M3 - Article

C2 - 29304237

AN - SCOPUS:85049010801

VL - 187

SP - 1327

EP - 1335

JO - American Journal of Epidemiology

JF - American Journal of Epidemiology

SN - 0002-9262

IS - 6

ER -