Predicting Performance of First-Year Residents: Correlations between Structured Interview, Licensure Exam, and Competency Scores in a Multi-Institutional Study

Brittany Marcus-Blank, Jeffrey A. Dahlke, Jonathan P Braman, Emily C Borman-Shoap, Ezgi Tiryaki, Jeffrey G Chipman, John S. Andrews, Paul R Sackett, Michael J. Cullen

Research output: Contribution to journalArticle

2 Citations (Scopus)

Abstract

Purpose To determine whether scores on structured interview (SI) questions designed to measure noncognitive competencies in physicians (1) predict subsequent first-year resident performance on Accreditation Council for Graduate Medical Education (ACGME) milestones and (2) add incremental validity over United States Medical Licensing Examination (USMLE) Step 1 and Step 2 Clinical Knowledge scores in predicting performance. Method The authors developed 18 behavioral description questions to measure key noncognitive competencies (e.g., teamwork). In 2013-2015, 14 programs (13 residency, 1 fellowship) from 6 institutions used subsets of these questions in their selection processes. The authors conducted analyses to determine the validity of SI and USMLE scores in predicting first-year resident milestone performance in the ACGME's core competency domains and overall. Results SI scores predicted midyear and year-end overall performance (r = 0.18 and 0.19, respectively, P <.05) and year-end performance on patient care, interpersonal and communication skills, and professionalism competencies (r = 0.23, r = 0.22, and r = 0.20, respectively, P <.05). SI scores contributed incremental validity over USMLE scores in predicting year-end performance on patient care (ΔR = 0.05), interpersonal and communication skills (ΔR = 0.09), and professionalism (ΔR = 0.09; all P <.05). USMLE scores contributed incremental validity over SI scores in predicting year-end performance overall and on patient care and medical knowledge. Conclusions SI scores predict first-year resident year-end performance in the interpersonal and communication skills, patient care, and professionalism competency domains. Future research should investigate whether SIs predict a range of clinically relevant outcomes.

Original languageEnglish (US)
Pages (from-to)378-387
Number of pages10
JournalAcademic Medicine
Volume94
Issue number3
DOIs
StatePublished - Mar 1 2019

Fingerprint

resident
interview
performance
patient care
communication skills
examination
teamwork
accreditation
physician
graduate
knowledge
professionalism
education

Cite this

Predicting Performance of First-Year Residents : Correlations between Structured Interview, Licensure Exam, and Competency Scores in a Multi-Institutional Study. / Marcus-Blank, Brittany; Dahlke, Jeffrey A.; Braman, Jonathan P; Borman-Shoap, Emily C; Tiryaki, Ezgi; Chipman, Jeffrey G; Andrews, John S.; Sackett, Paul R; Cullen, Michael J.

In: Academic Medicine, Vol. 94, No. 3, 01.03.2019, p. 378-387.

Research output: Contribution to journalArticle

@article{25a78f2f41504c87b868ef7f762015b2,
title = "Predicting Performance of First-Year Residents: Correlations between Structured Interview, Licensure Exam, and Competency Scores in a Multi-Institutional Study",
abstract = "Purpose To determine whether scores on structured interview (SI) questions designed to measure noncognitive competencies in physicians (1) predict subsequent first-year resident performance on Accreditation Council for Graduate Medical Education (ACGME) milestones and (2) add incremental validity over United States Medical Licensing Examination (USMLE) Step 1 and Step 2 Clinical Knowledge scores in predicting performance. Method The authors developed 18 behavioral description questions to measure key noncognitive competencies (e.g., teamwork). In 2013-2015, 14 programs (13 residency, 1 fellowship) from 6 institutions used subsets of these questions in their selection processes. The authors conducted analyses to determine the validity of SI and USMLE scores in predicting first-year resident milestone performance in the ACGME's core competency domains and overall. Results SI scores predicted midyear and year-end overall performance (r = 0.18 and 0.19, respectively, P <.05) and year-end performance on patient care, interpersonal and communication skills, and professionalism competencies (r = 0.23, r = 0.22, and r = 0.20, respectively, P <.05). SI scores contributed incremental validity over USMLE scores in predicting year-end performance on patient care (ΔR = 0.05), interpersonal and communication skills (ΔR = 0.09), and professionalism (ΔR = 0.09; all P <.05). USMLE scores contributed incremental validity over SI scores in predicting year-end performance overall and on patient care and medical knowledge. Conclusions SI scores predict first-year resident year-end performance in the interpersonal and communication skills, patient care, and professionalism competency domains. Future research should investigate whether SIs predict a range of clinically relevant outcomes.",
author = "Brittany Marcus-Blank and Dahlke, {Jeffrey A.} and Braman, {Jonathan P} and Borman-Shoap, {Emily C} and Ezgi Tiryaki and Chipman, {Jeffrey G} and Andrews, {John S.} and Sackett, {Paul R} and Cullen, {Michael J.}",
year = "2019",
month = "3",
day = "1",
doi = "10.1097/ACM.0000000000002429",
language = "English (US)",
volume = "94",
pages = "378--387",
journal = "Academic Medicine",
issn = "1040-2446",
publisher = "Lippincott Williams and Wilkins",
number = "3",

}

TY - JOUR

T1 - Predicting Performance of First-Year Residents

T2 - Correlations between Structured Interview, Licensure Exam, and Competency Scores in a Multi-Institutional Study

AU - Marcus-Blank, Brittany

AU - Dahlke, Jeffrey A.

AU - Braman, Jonathan P

AU - Borman-Shoap, Emily C

AU - Tiryaki, Ezgi

AU - Chipman, Jeffrey G

AU - Andrews, John S.

AU - Sackett, Paul R

AU - Cullen, Michael J.

PY - 2019/3/1

Y1 - 2019/3/1

N2 - Purpose To determine whether scores on structured interview (SI) questions designed to measure noncognitive competencies in physicians (1) predict subsequent first-year resident performance on Accreditation Council for Graduate Medical Education (ACGME) milestones and (2) add incremental validity over United States Medical Licensing Examination (USMLE) Step 1 and Step 2 Clinical Knowledge scores in predicting performance. Method The authors developed 18 behavioral description questions to measure key noncognitive competencies (e.g., teamwork). In 2013-2015, 14 programs (13 residency, 1 fellowship) from 6 institutions used subsets of these questions in their selection processes. The authors conducted analyses to determine the validity of SI and USMLE scores in predicting first-year resident milestone performance in the ACGME's core competency domains and overall. Results SI scores predicted midyear and year-end overall performance (r = 0.18 and 0.19, respectively, P <.05) and year-end performance on patient care, interpersonal and communication skills, and professionalism competencies (r = 0.23, r = 0.22, and r = 0.20, respectively, P <.05). SI scores contributed incremental validity over USMLE scores in predicting year-end performance on patient care (ΔR = 0.05), interpersonal and communication skills (ΔR = 0.09), and professionalism (ΔR = 0.09; all P <.05). USMLE scores contributed incremental validity over SI scores in predicting year-end performance overall and on patient care and medical knowledge. Conclusions SI scores predict first-year resident year-end performance in the interpersonal and communication skills, patient care, and professionalism competency domains. Future research should investigate whether SIs predict a range of clinically relevant outcomes.

AB - Purpose To determine whether scores on structured interview (SI) questions designed to measure noncognitive competencies in physicians (1) predict subsequent first-year resident performance on Accreditation Council for Graduate Medical Education (ACGME) milestones and (2) add incremental validity over United States Medical Licensing Examination (USMLE) Step 1 and Step 2 Clinical Knowledge scores in predicting performance. Method The authors developed 18 behavioral description questions to measure key noncognitive competencies (e.g., teamwork). In 2013-2015, 14 programs (13 residency, 1 fellowship) from 6 institutions used subsets of these questions in their selection processes. The authors conducted analyses to determine the validity of SI and USMLE scores in predicting first-year resident milestone performance in the ACGME's core competency domains and overall. Results SI scores predicted midyear and year-end overall performance (r = 0.18 and 0.19, respectively, P <.05) and year-end performance on patient care, interpersonal and communication skills, and professionalism competencies (r = 0.23, r = 0.22, and r = 0.20, respectively, P <.05). SI scores contributed incremental validity over USMLE scores in predicting year-end performance on patient care (ΔR = 0.05), interpersonal and communication skills (ΔR = 0.09), and professionalism (ΔR = 0.09; all P <.05). USMLE scores contributed incremental validity over SI scores in predicting year-end performance overall and on patient care and medical knowledge. Conclusions SI scores predict first-year resident year-end performance in the interpersonal and communication skills, patient care, and professionalism competency domains. Future research should investigate whether SIs predict a range of clinically relevant outcomes.

UR - http://www.scopus.com/inward/record.url?scp=85064818170&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85064818170&partnerID=8YFLogxK

U2 - 10.1097/ACM.0000000000002429

DO - 10.1097/ACM.0000000000002429

M3 - Article

C2 - 30157088

AN - SCOPUS:85064818170

VL - 94

SP - 378

EP - 387

JO - Academic Medicine

JF - Academic Medicine

SN - 1040-2446

IS - 3

ER -