Effects of predictor weighting methods on incremental validity

Paul R Sackett, Jeffrey A. Dahlke, Oren R. Shewach, Nathan R Kuncel

Research output: Contribution to journalArticle

2 Citations (Scopus)

Abstract

It is common to add an additional predictor to a selection system with the goal of increasing criterionrelated validity. Research on the incremental validity of a second predictor is generally based on forming a regression-weighted composite of the predictors. However, in practice predictors are commonly used in ways other than regression-weighted composites, and we examine the robustness of incremental validity findings to other ways of using predictors, namely, unit weighting and multiple hurdles. We show that there are settings in which the incremental value of a second predictor disappears, and can even produce lower validity than the first predictor alone, when these alternatives to regression weighting are used. First, we examine conditions under which unit weighting will negate gain in predictive power attainable via regression weights. Second, we revisit Schmidt and Hunter's (1998) summary of incremental validity of predictors over cognitive ability, evaluating whether the reported incremental value of a second predictor is different when predictors are unit weighted rather than regression weighted. Third, we analyze data reported in the published literature to discern the frequency with which unit weighting might affect conclusions about whether there is value in adding a second predictor to a first. Finally, we shift from unit weighting to multiple hurdle selection, examining conditions under which conclusions about incremental validity differ when regression weighting is replaced by multiple-hurdle selection.

Original languageEnglish (US)
Pages (from-to)1421-1434
Number of pages14
JournalJournal of Applied Psychology
Volume102
Issue number10
DOIs
StatePublished - Oct 1 2017

Fingerprint

Weights and Measures
Research

Keywords

  • Incremental validity
  • Personnel selection
  • Validity

Cite this

Effects of predictor weighting methods on incremental validity. / Sackett, Paul R; Dahlke, Jeffrey A.; Shewach, Oren R.; Kuncel, Nathan R.

In: Journal of Applied Psychology, Vol. 102, No. 10, 01.10.2017, p. 1421-1434.

Research output: Contribution to journalArticle

Sackett, Paul R ; Dahlke, Jeffrey A. ; Shewach, Oren R. ; Kuncel, Nathan R. / Effects of predictor weighting methods on incremental validity. In: Journal of Applied Psychology. 2017 ; Vol. 102, No. 10. pp. 1421-1434.
@article{a20df28d89b4440c93a7286b0eacb61c,
title = "Effects of predictor weighting methods on incremental validity",
abstract = "It is common to add an additional predictor to a selection system with the goal of increasing criterionrelated validity. Research on the incremental validity of a second predictor is generally based on forming a regression-weighted composite of the predictors. However, in practice predictors are commonly used in ways other than regression-weighted composites, and we examine the robustness of incremental validity findings to other ways of using predictors, namely, unit weighting and multiple hurdles. We show that there are settings in which the incremental value of a second predictor disappears, and can even produce lower validity than the first predictor alone, when these alternatives to regression weighting are used. First, we examine conditions under which unit weighting will negate gain in predictive power attainable via regression weights. Second, we revisit Schmidt and Hunter's (1998) summary of incremental validity of predictors over cognitive ability, evaluating whether the reported incremental value of a second predictor is different when predictors are unit weighted rather than regression weighted. Third, we analyze data reported in the published literature to discern the frequency with which unit weighting might affect conclusions about whether there is value in adding a second predictor to a first. Finally, we shift from unit weighting to multiple hurdle selection, examining conditions under which conclusions about incremental validity differ when regression weighting is replaced by multiple-hurdle selection.",
keywords = "Incremental validity, Personnel selection, Validity",
author = "Sackett, {Paul R} and Dahlke, {Jeffrey A.} and Shewach, {Oren R.} and Kuncel, {Nathan R}",
year = "2017",
month = "10",
day = "1",
doi = "10.1037/apl0000235",
language = "English (US)",
volume = "102",
pages = "1421--1434",
journal = "Journal of Applied Psychology",
issn = "0021-9010",
publisher = "American Psychological Association",
number = "10",

}

TY - JOUR

T1 - Effects of predictor weighting methods on incremental validity

AU - Sackett, Paul R

AU - Dahlke, Jeffrey A.

AU - Shewach, Oren R.

AU - Kuncel, Nathan R

PY - 2017/10/1

Y1 - 2017/10/1

N2 - It is common to add an additional predictor to a selection system with the goal of increasing criterionrelated validity. Research on the incremental validity of a second predictor is generally based on forming a regression-weighted composite of the predictors. However, in practice predictors are commonly used in ways other than regression-weighted composites, and we examine the robustness of incremental validity findings to other ways of using predictors, namely, unit weighting and multiple hurdles. We show that there are settings in which the incremental value of a second predictor disappears, and can even produce lower validity than the first predictor alone, when these alternatives to regression weighting are used. First, we examine conditions under which unit weighting will negate gain in predictive power attainable via regression weights. Second, we revisit Schmidt and Hunter's (1998) summary of incremental validity of predictors over cognitive ability, evaluating whether the reported incremental value of a second predictor is different when predictors are unit weighted rather than regression weighted. Third, we analyze data reported in the published literature to discern the frequency with which unit weighting might affect conclusions about whether there is value in adding a second predictor to a first. Finally, we shift from unit weighting to multiple hurdle selection, examining conditions under which conclusions about incremental validity differ when regression weighting is replaced by multiple-hurdle selection.

AB - It is common to add an additional predictor to a selection system with the goal of increasing criterionrelated validity. Research on the incremental validity of a second predictor is generally based on forming a regression-weighted composite of the predictors. However, in practice predictors are commonly used in ways other than regression-weighted composites, and we examine the robustness of incremental validity findings to other ways of using predictors, namely, unit weighting and multiple hurdles. We show that there are settings in which the incremental value of a second predictor disappears, and can even produce lower validity than the first predictor alone, when these alternatives to regression weighting are used. First, we examine conditions under which unit weighting will negate gain in predictive power attainable via regression weights. Second, we revisit Schmidt and Hunter's (1998) summary of incremental validity of predictors over cognitive ability, evaluating whether the reported incremental value of a second predictor is different when predictors are unit weighted rather than regression weighted. Third, we analyze data reported in the published literature to discern the frequency with which unit weighting might affect conclusions about whether there is value in adding a second predictor to a first. Finally, we shift from unit weighting to multiple hurdle selection, examining conditions under which conclusions about incremental validity differ when regression weighting is replaced by multiple-hurdle selection.

KW - Incremental validity

KW - Personnel selection

KW - Validity

UR - http://www.scopus.com/inward/record.url?scp=85021742151&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85021742151&partnerID=8YFLogxK

U2 - 10.1037/apl0000235

DO - 10.1037/apl0000235

M3 - Article

C2 - 28530417

AN - SCOPUS:85021742151

VL - 102

SP - 1421

EP - 1434

JO - Journal of Applied Psychology

JF - Journal of Applied Psychology

SN - 0021-9010

IS - 10

ER -