Pushing the Limits for Judgmental Consistency: Comparing Random Weighting Schemes with Expert Judgments

Authors

Martin C. Yu, HumRROFollow
Nathan R. Kuncel, University of Minnesota - Twin CitiesFollow

DOI

https://doi.org/10.25035/pad.2020.02.002

Abstract

Consistent use of information has been identified as a critical issue that can undermine expert predictions. Using three personnel assessment datasets, we conduct Monte Carlo simulations to compare the accuracy of expert judgements for predicting the job performance of managers against four different weighting schemes: consistent random weights, completely random weights, unit weights, and optimal weights. Expert accuracy fell within the completely random weight distribution in two samples and at the low end of the consistent random weight distribution in one sample. In other words, consistent random weights reliably outperformed expert judgment for hiring decisions across three datasets with a total sample size of 847. We see this as a call to develop decision making systems that help control consistency or to manage consistency by aggregating multiple expert judgments.

Recommended Citation

Yu, Martin C. and Kuncel, Nathan R. (2020) "Pushing the Limits for Judgmental Consistency: Comparing Random Weighting Schemes with Expert Judgments," Personnel Assessment and Decisions: Number 6 : Iss. 2 , Article 2.
DOI: https://doi.org/10.25035/pad.2020.02.002
Available at: https://scholarworks.bgsu.edu/pad/vol6/iss2/2

Corresponding Author Information

Martin C. Yu

myu@humrro.org

66 Canal Center Plaza Suite 700 Alexandria, VA 22314