What makes experts reliable? Expert reliability and the estimation of latent traits

Kyle L. Marquardt, Daniel Pemstein, Brigitte Seim, Yi Ting Wang

Research output: Contribution to journalArticlepeer-review

10 Scopus citations

Abstract

Experts code latent quantities for many influential political science datasets. Although scholars are aware of the importance of accounting for variation in expert reliability when aggregating such data, they have not systematically explored either the factors affecting expert reliability or the degree to which these factors influence estimates of latent concepts. Here we provide a template for examining potential correlates of expert reliability, using coder-level data for six randomly selected variables from a cross-national panel dataset. We aggregate these data with an ordinal item response theory model that parameterizes expert reliability, and regress the resulting reliability estimates on both expert demographic characteristics and measures of their coding behavior. We find little evidence of a consistent substantial relationship between most expert characteristics and reliability, and these null results extend to potentially problematic sources of bias in estimates, such as gender. The exceptions to these results are intuitive, and provide baseline guidance for expert recruitment and retention in future expert coding projects: attentive and confident experts who have contextual knowledge tend to be more reliable. Taken as a whole, these findings reinforce arguments that item response theory models are a relatively safe method for aggregating expert-coded data.

Original languageEnglish (US)
JournalResearch and Politics
Volume6
Issue number4
DOIs
StatePublished - Oct 2019
Externally publishedYes

Bibliographical note

Publisher Copyright:
© The Author(s) 2019.

Keywords

  • Bayesian methods
  • Cross-national panel data
  • expert surveys
  • IRT models
  • measurement

Fingerprint

Dive into the research topics of 'What makes experts reliable? Expert reliability and the estimation of latent traits'. Together they form a unique fingerprint.

Cite this