Does Step 3 of the United States Medical Licensing Exam Measure Clinical Competence? A Predictive Validity Study

Purpose: The main purpose of the present study was to evaluate the predictive validity of medical school admissions data, preclinical and clinical achievement data, postgraduate evaluations, and the United States Medical Licensing Exam Step 1 and 2 data for performance on Step 3. Methods: A total of 321 physicians (178 men, 55.4 %; 143 women, 44.6 %) participated. Admissions data, medical school preclinical achievement data, medical school clerkship clinical performance data, postgraduate clinical performance data, and USMLE Step 1 and 2 data were independent variables. Step 3 was the dependent variable in a backwards multiple regression. An analysis of covariance with Step 3 scores as the dependent variable, Step 2 scores as the covariate, and residency program (narrowly focused vs. broadly focused two levels) as the independent variable was conducted. Results: The regression analyses resulted in an optimal model fit with the most parsimonious combination of independent variables (multiple R = 0.686, R ²  = 0.470, p < 0.001): Step 2 (β = 0.606) + verbal reasoning (β = 0.125) + PGY1 (β = 0.161) + Year 3 (β = 0.157). Candidates in broadly focused residencies scored higher on Step 3 compared to those in narrowly focused residencies (F = 5.17, p < 0.05). Conclusions: The predictive validity of Step 2, verbal reasoning, PGY1 assessment, Year 3 average for Step 3 performance is supported by the optimal regression model. Candidates in broadly focused residencies scored higher on Step 3 compared to those in narrowly focused residencies. The results are evidence for the validity of Step 3 as a measure of clinical competence.