TY - JOUR
T1 - Evaluating the validity of educational rating data
AU - Harwell, Michael
PY - 1999/2
Y1 - 1999/2
N2 - The use of trained raters has a long tradition in educational research. A standard feature of studies employing raters is the use of indica ors of agreement among raters such as interrater reliability coefficients. Surprisingly, the validity of rating data has received relatively little attention, raising the undesirable prospect of ratings with satisfactory reliability but little validity. This article suggests two complementary frameworks for providing validity evidence for rating data. One conceptualizes raters as data collection instruments that should be subject to traditional procedures for establishing validity evidence; another evaluates studies employing raters from an experimental design perspective, permitting the internal validity of the study to be assessed and used as an indicator of the extent to which ratings are attributable to the training of the raters. Two studies employing raters are used to illustrate these ideas.
AB - The use of trained raters has a long tradition in educational research. A standard feature of studies employing raters is the use of indica ors of agreement among raters such as interrater reliability coefficients. Surprisingly, the validity of rating data has received relatively little attention, raising the undesirable prospect of ratings with satisfactory reliability but little validity. This article suggests two complementary frameworks for providing validity evidence for rating data. One conceptualizes raters as data collection instruments that should be subject to traditional procedures for establishing validity evidence; another evaluates studies employing raters from an experimental design perspective, permitting the internal validity of the study to be assessed and used as an indicator of the extent to which ratings are attributable to the training of the raters. Two studies employing raters are used to illustrate these ideas.
UR - http://www.scopus.com/inward/record.url?scp=0033249395&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=0033249395&partnerID=8YFLogxK
U2 - 10.1177/0013164499591002
DO - 10.1177/0013164499591002
M3 - Article
AN - SCOPUS:0033249395
SN - 0013-1644
VL - 59
SP - 25
EP - 37
JO - Educational and Psychological Measurement
JF - Educational and Psychological Measurement
IS - 1
ER -