Exploratory Study of Individual Assessment Practices: Interrater Reliability and Judgments of Assessor Effectiveness

Ann Marie Ryan, Paul R. Sackett

Research output: Contribution to journalArticle

21 Scopus citations

Abstract

Three psychologists who regularly conduct individual assessments were asked to assess 3 individuals posing as job candidates for the same position. The materials from these 9 assessments-test scores, biographical information, and audiotapes of interviews-served as protocols for 50 industrial/organizational psychologists who rated the candidates and assessors. Comparisons of the approaches and conclusions of the assessors indicated variability in job/organizational information obtained, test instruments used, personal history information gathered, interview and the report generated, and conclusions regarding the candidates. On average, only one third of the raters agreed with the conclusions of the assessor whose protocol they were reviewing. Significant differences were found in the raters' evaluations of 2 of the assessors, depending on which candidate the assessor had evaluated. The study's design limits generalizability; however, the low interrater agreement is disturbing.

Original languageEnglish (US)
Pages (from-to)568-579
Number of pages12
JournalJournal of Applied Psychology
Volume74
Issue number4
DOIs
StatePublished - Aug 1989

Fingerprint Dive into the research topics of 'Exploratory Study of Individual Assessment Practices: Interrater Reliability and Judgments of Assessor Effectiveness'. Together they form a unique fingerprint.

Cite this