Interrater reliability and agreement of subjective judgments

Howard E. Tinsley, David J. Weiss

Research output: Contribution to journalArticlepeer-review

551 Scopus citations

Abstract

Indexes of interrater reliability and agreement are reviewed, and suggestions are made regarding their use in counseling psychology research. The distinction between agreement and reliability is clarified, and the relationships between these indexes and the level of measurement and type of replication are discussed. Indexes of interrater reliability appropriate for use with ordinal and interval scales are considered. The intraclass correlation as a measure of interrater reliability is discussed in terms of the treatment of between-raters variance and the appropriateness of reliability estimates based on composite or individual ratings. The advisability of optimal weighting schemes for calculating composite ratings is also considered. Measures of interrater agreement for ordinal and interval scales are described, as are measures of interrater agreement for data at the nominal level of measurement. (54 ref) (PsycINFO Database Record (c) 2006 APA, all rights reserved).

Original languageEnglish (US)
Pages (from-to)358-376
Number of pages19
JournalJournal of counseling psychology
Volume22
Issue number4
DOIs
StatePublished - Jul 1975

Keywords

  • review of indexes & interrelations with measurement level & type of replication, counseling psychology research

Fingerprint

Dive into the research topics of 'Interrater reliability and agreement of subjective judgments'. Together they form a unique fingerprint.

Cite this