Evaluating a Natural Language Processing Approach to Estimating KSA and Interest Job Analysis Ratings

Collecting job analysis ratings for a large number of jobs via surveys, interviews, or focus groups can put a very large burden on organizations. In this study, we describe and evaluate a streamlined, natural language processing-based approach to estimating (a) the importance of various knowledges, skills, abilities, and other characteristics (KSAOs) to jobs, and (b) how descriptive various interests are of work on a job. Specifically, we evaluate whether we can train a machine to accurately estimate KSAO ratings for jobs using job description and task statement text as the sole input. Data for 963 occupations from the U.S. Department of Labor’s Occupational Information Network (O*NET) system and an independent set of 229 occupations from a large organization provided the basis for the evaluation. Our approach produced KSAO predictions that had cross-validated correlations with subject matter expert (SME) ratings of knowledges, skills, abilities, and interests of.74,.80,.75, and.84, respectively (on average, across the 126 KSAOs examined). We found clear evidence for the validity of machine-based predictions based on (a) convergence among machine-based and SME-furnished ratings, (b) conceptually meaningful patterns of prediction model regression coefficients among the KSAOs examined, and (c) conceptual relevance of top predictor models underlying related clusters of KSAOs. We also found that prediction models developed on O*NET data produced meaningful results when applied to an independent set of job descriptions and tasks. Implications of this work, as well as suggested directions for future job analysis research and practice, are discussed.