Examining Factors Influencing Use of a Decision Aid in Personnel Selection

In this research, two studies were conducted to examine factors influencing reliance on a decision aid in personnel selection. Specifically, this study examined the effect of feedback, validity of selection predictors, and presence of a decision aid on the use of the aid in personnel selection. The results demonstrate that when people are provided with the decision aid, their predictions were significantly more similar to the predictions made by the aid than people who were not provided with the aid. This suggests that when people are provided with an aid, they will use it to some degree. This research also shows that when provided with a decision aid with high cue validity, people will increase their reliance on the decision aid over multiple decisions.

Assessing job candidates and selecting those with the highest qualifications is of utmost importance as organizations attempt to win the war for talent. Personnel selection systems aim to assess applicants on physical and psychological attributes required to perform the job; ideally, these attributes help identify individuals who will demonstrate better performance and improve organizational effectiveness and efficiency (Farr & Tippins, 2010). A personnel selection system, however, is only as good as the measures used to assess the specified attributes, as well as the evaluators assessing applicants.
Researchers have spent decades investigating the validity of various constructs and assessment methods. From meta-analytic studies, several conclusions can be made regarding the overall effectiveness of job performance predictors in selection. Specifically, general cognitive ability is one of the best predictors of performance (Schmidt & Hunter, 2004), whereas conscientiousness is the most valid of the five-factor model of personality dimensions (Barrick, Mount, & Judge, 2001). Further, structured interviews are superior to unstructured interviews (Huffcutt & Arthur Jr., 1994;Huffcutt, Culbertson, & Weyhrauch, 2014), and using multiple valid predictors can improve predictions 1998). Moreover, research has demonstrated that practitioners should use decision aids (e.g., scores on cognitive ability tests) when making hiring decisions (Highhouse, 2008;Schmidt & Hunter, 1998). Nevertheless, decision makers tend to disregard statistically validated predictors and over-rely on their intuition, usually to the detriment of the selection decision (Highhouse, 2008;Slaughter & Kausel, 2014).

Decision Aid Use
Researchers have shown that people are hesitant to rely on decision aids when making predictions or decisions (Arkes, Dawes & Christensen, 1986;Ashton, 1990;Diab, Pui, Yankelevich, & Highhouse, 2011). The reasons include an assumption that perfect prediction is possible and people can consider more information than an aid. People believe they themselves are capable of perfect prediction (Highhouse, 2008), and any evidence to the contrary is downplayed or discounted. However, people cannot in fact perfectly predict behavior and the "variance in [employee] success is simply not predictable prior to employment" (Highhouse, 2008, pp. 335-336). Therefore, when predicting human behavior, there is a guarantee of error. Furthermore, Dietvorst, Simmons, and Masey (2015) demonstrated that when people see a decision aid err, they distrust the aid more than they distrust themselves after making the

Impact of Validity on Decision Aid Use
Uncertainty is a key factor influencing managerial reliance on intuition. In a sample of 200 executives, almost all reported using intuition to guide decision making and noted reliance on intuition most heavily when a high level of uncertainty existed (Agor, 1986). Managers also reported relying on intuition when outcomes were less scientifically predictable, when information was limited, when the information available did not provide clear direction on how to proceed, when statistical data had limited utility, and when time pressures were greatest.
Additionally, researchers have demonstrated that the accuracy, or validity, of a decision aid influences its use. Gomaa et al. (2011) directly manipulated the validity of a decision aid, such that the decision aid participants were presented with had an accuracy of 50%, 60%, 70%, 80%, or 90%. Specifically, they informed participants that the decision aid gave correct estimates "in every X out of 10 cases" (p. 211). They found that more valid decision aids were used to a significantly greater extent. Similarly, across sev-eral studies Dietvorst et al. (2015) manipulated participants' experience with a decision aid by providing the decision aid's previous forecasting performance, their own previous forecasting performance, previous forecasting performance for both the decision aid and their self, or no previous performance information. Their results showed that after viewing the forecasting performance of the decision aid, people were less likely to use it because they were less tolerant of the decision aid's smaller errors than their own larger errors. Further, Gomaa et al. demonstrated that people utilize a decision aid more when it is more valid. All of this information suggests that managers are most likely to rely on a decision aid when it has a higher level of validity.

Feedback
Slaughter and Kausel (2014) noted that providing decision makers with feedback regarding their personnel selection decisions can improve those decisions. Feedback may be a vital source of information in calibrating one's decision strategies when it assesses the accuracy of one's own decisions (e.g., Louie, 1999). Such feedback has had meaningful influence on individuals' decision-making processes (Brown, 2006;Louie, 1999) and may influence one's future decision-making strategies. Louie (1999) demonstrated that individuals who receive positive feedback regarding a decision exhibit a strong hindsight bias or believe the outcome was predictable after learning the outcome (Roese & Vohs, 2012). Additionally, Brown (2006) demonstrated that when decision outcomes are less uncertain, decision feedback actually leads to decreases in the effectiveness of decision-making strategies; however, when decision outcomes are more uncertain, decision feedback leads to more effective decision making. Wofford and Goodwin (1990) found that repeated negative feedback changed the decision-making strategies individuals used. In essence, the feedback was a form of operant conditioning whereby positive feedback reinforced a person's decision strategy and negative feedback punished a decision strategy. Because the negative feedback led to a change in the decision-making strategies individuals used, it would be expected that providing negative feedback in the form of information about the magnitude of one's errors would lead them to utilize different decision-making strategies.
Additionally, it is likely that feedback will interact with the cue validity. When cues have lower validity, people who receive feedback may be more likely to rely on their own pre-existing beliefs. Arkes et al. (1986) examined the effect of different types of feedback on decision aid reliance when the decision aid was 70% accurate, a high level of validity. They found that feedback type had a significant effect on decision aid reliance. However, the validity of the decision aid was not manipulated. Further, Gomaa et al. (2011) demonstrated that when a decision aid is more valid, people utilize the decision aid to a greater extent. Conversely, after observing a model make mistakes, participants instead relied on their own decision-making processes. Furthermore, researchers have directly examined the interactive effects of future uncertainty and feedback on optimal decision making strategies. When provided with feedback regarding uncertain future outcomes, people made less prudent decisions than when provided with feedback regarding certain outcomes (Brown, 2006).

Cue Learning
Within the field of judgment and decision making, researchers have focused on understanding how people make inferences and judgments about some unknown criterion based on probabilistic cues (Brunswik, 1943). For example, every year faculty members utilize cues (undergraduate GPA, GRE scores, letters of recommendation) to make inferences about graduate school applicants' likelihood of success (graduate school GPA). Researchers have also examined whether and how people can accurately learn the appropriate weighting of various cues for making judgments. For instance, Santarcangelo, Cribbie, and Ebesu Hubbard (2004) demonstrated that training participants on the appropriate use of visual, vocal, and verbal content cues leads to more accurate judgments of the truthfulness of messages. Similarly, in their test of whether the modality of cue-based training impacts appropriate use of cues, Henriksson and Enkvist (2018) found that feedback-based training, observational learning, and training focusing on cue profile matching all significantly increased accuracy of judgments. Trippas and Pachur (2019) found that feedback and continuous criterion information lead to cue learning. Further, when cues are experienced as being predictive of important outcomes (compared to not being predictive), people are better able to discriminated between cues when the cue predictiveness is established during cue training (Le Pelley, Turnbull, Reimers, & Knipe, 2010). However, as Dawes (1979) suggests, even improper cue weighting can be more accurate than normal human judgments, in part because people tend to change the relative weighting of the cues between judgments.
The present study can be construed as a training design in which decision makers are taught the relative importance of various selection cues. Specifically, in the current study, when the decision aid is present, participants are given information about the proper model and relative importance of the predictors as well as scores on the predictors (or cues). When the decision aid is not present, participants are not given information about how good the different predictors are yet still see the applicant scores on the various predictors. Thus, we contribute not only to the judgment and decision making literature but also to research involving cue training effects.

The Current Study
Hypothesis 1: Participants' hiring choices and performance predictions will more closely match those made by the decision aid when cues are more valid than when they are less valid. Hypothesis 2: Participants' hiring choice and performance predictions will more closely match the choice and performance predictions made by the decision aid when it is provided. Hypothesis 3: The presence of the decision aid will interact with the validity of the cues, such that when the decision aid is present and the cues are more valid, participants' hiring choices and performance predictions will more closely match those made by the decision aid than in all other conditions. Hypothesis 4: Participants' hiring choice and performance predictions will more closely match those made by the decision aid when negatively framed feedback is provided regarding participants' predictions than when no feedback is provided. Hypothesis 5: The effect of feedback on decision aid reliance will depend on the validity of the cues, such that when the cues are more valid and feedback is provided, participants' hiring choices and performance predictions will more closely match those made by the decision aid than all other conditions. The hypotheses we are testing in this study build upon the existing literature in several ways. First, we directly evaluate recommendations made by Slaughter and Kausel (2014), who argued that in order to improve personnel selection decisions, decision makers should be asked to make precise estimates of performance and be provided with feedback regarding those estimates. In both of the studies we discuss below, we presented participants with feedback regarding the performance predictions they made. Further, in Study 2, we directly manipulated the presence of feedback to examine its effects on decision aid use. We also extend the literature on feedback by examining the role of feedback over multiple occasions to determine whether people will learn from previous decisions and predictions (e.g., Louie, 1999;Wofford & Goodwin, 1990). Slaughter and Kausel (2014) also argued that instead of instructing decision makers to make a decision based solely on a statistical prediction, decision makers should be provided with decision support on how to select among applicants (e.g., using a decision aid). In both studies, we directly tested this assertion. We sought to replicate and extend the findings of previous studies examining the effects of cue validity (e.g., Gomaa et al., 2011). Last, we extend each of these assertions by examining the interactive effects they have on Personnel Assessment And decisions Use of A decision Aid in Personnel selection decision aid use.

Method
Participants. Participants were recruited from Amazon's Mechanical Turk program. Attention check and screening items were used to identify and exclude participants who were not paying attention and were simply clicking through the survey. Usable data were obtained from 154 participants. Participants were paid one US dollar for their participation. Approximately 57% of participants were male with an average age of 37.7 (SD = 11.8), 73% were Caucasian, and 89% were employed. For employed individuals, the mean number of hours worked per week was 40.3 (SD = 9.5). Participant hiring experience was measured using a 6 point Likert scale (1 = no experience to 6 = extremely experienced). The average hiring experience level of participants was 3.19 (SD = 1.59).
Decision task. The decision task was adapted from Kausel, Culbertson, and Madrid (2016). Participants completed 10 trials which they compared two applicants for a sales agent job. Applicant data came from an actual organization that was validating their selection procedures. Over 200 applicants were assessed with a variety of selection tools and three months later their performance was assessed by their supervisors. We randomly selected 10 pairs of applicants for study participants to evaluate. For each trial, participants were presented with the two applicants' percentile scores on tests of cognitive ability, conscientiousness, and an unstructured interview. Participants were asked to predict each candidate's performance percentile rank from 0 (will perform worse than all other employees) to 99 (will perform better than all other employees). Participants then selected the candidate that the company should hire.
Feedback information. Participants received feedback after each decision. Participants were shown their original predictions (i.e., their estimated performance percentile rank), job performance of both candidates once hired (i.e., their actual performance percentile rank), and the prediction error for each candidate's performance (e.g., "Your prediction for Candidate A was off by X% points"). As such, participants were informed about the extent to which their predictions differed from the candidates' actual performance.
Cue validity manipulation. Participants were randomly assigned to a high validity condition or a moderate validity condition. Participants were unaware of which condition they were in. In the high validity condition, the job candidates' eventual performance was highly predictable (R2 = .962) from an appropriate weighting of the three predictors. In the moderate validity condition, the job candidates' eventual performance was less predictable (R2 = .504) from an appropriate weighting of the three predictors. The weighting of the predictors in both conditions was .50 for cognitive ability, .40 for conscientiousness, and .10 for the unstructured interview based on the results of meta-analyses (e.g., Huffcutt & Arthur, 1994, Schmidt & Hunter, 1998.
The model used to create the high validity condition was:

Equation 2
y lp = round((logistic(logistic percent(.50 * x 1 + .40 * x 2 + .10 * x 3 ) + x r~N (0,1)) * 100) Where y lp represents the candidate's eventual performance in the moderate validity condition. In both equations, x 1 represents the candidate's cognitive ability score, x 2 represents the candidate's conscientiousness score, and x 3 represents the candidate's interview score. Additionally, x r~N (0,1) represents the value randomly sampled from a standard normal distribution with a mean of 0 and a standard deviation of 1.
In order to determine the actual validity of the cues once the random error has been introduced in the eventual performance of the candidates, the candidates' test scores were used to predict their eventual performance. The model used to predict the candidates' eventual performance used the same weighting used in Equations 1 and 2. Therefore, the formula used to predict the candidates' eventual performance was:

Equation 3
ŷ = .50 * x 1 + .40 * x 2 + .10 * x 3 Where ŷ = the predicted eventual performance for the candidate. In the high cue validity condition, Equation 3 resulted in an R 2 = .962. In the moderate cue validity condition, Equation 3 resulted in an R 2 = .504. This confirms that the conditions represent situations in which the selection predictors are highly valid and moderately valid, respectively.

Decision Aid Manipulation
Two operationalizations of decision aid reliance were utilized: the degree of match between the participant's and model's predicted performance as assessed by the percentile rank, and the degree of match between the participant's and model's hire choice. Participants were randomly assigned to one of two conditions in which a decision aid was either present or absent. In the decision aid present condition, participants were provided information about the validity of the three predictors and information regarding a statistical model that should be used to predict candidate performance. In the decision aid absent condition, participants did not receive any information regarding the validity of the three selection predictors or the model.
Participants were asked to utilize the candidates' scores to estimate the candidates' performance as well as select one of the candidates to hire. For participants in the decision aid present condition, participants were presented with Equation 3, but they were not provided with the results of the calculations for each candidate. Instead, participants were only presented with the result of the validity weights multiplied by the predictor scores. Thus, participants would still be required to add the three weighted predictor scores. The rationale for this was that participants who engaged in more systematic information processing (i.e., relied more on the statistical model's prediction) would actually add these scores. Thus, their predictions should match the predictions made by the model. In contrast, individuals who engaged in more automatic information processing would not rely on the information provided by the model. Instead, they would rely on their own decision-making processes to make their predictions, which would likely result in predictions that do not match the predictions made by the model. In summary, when the decision aid is provided, participants are provided with information about the proper statistical model, the relative importance of each of the predictors (cues), and the scores on the predictors (cues) for both candidates. When the decision aid is not provided, participants do not receive any information about the relative importance of the predictors but are provided with the candidates' scores on the predictors. Thus, this study design can be thought of as a training design with the attempt of training the decision makers to use the decision aid and about the relative importance of the predictors. The instructions provided and the example decision stimuli are presented in Appendix C.

Results
Match in hire choice. To examine reliance on the decision aid based on the match between the participant's and model's hire choice, a repeated measures logistic regression was conducted using the generalized linear mixed-effects modeling package in R (Bates, Maechler, Bolker, & Walker, 2014). The cue validity, decision aid presence, trial, and their interactions were entered as fixed effects. The match in hire choice was entered as the dependent variable. To reduce the effects of multicollinearity, the predictors were centered before being entered into the model by using effect coding of cue validity and decision aid presence and mean centering of trial.
The results of Model 1 are displayed in Figure 1 (all figures are displayed in the Appendix A), which showed a significant main effect of model presence, B = 0.453, z = 5.126, p < .001. When model information was provided, participants' hire choices were significantly more likely to match the model's hire choices than when model information was not provided. As can be seen in Figure 1, participants who were provided with the decision aid on average made hiring decisions that were approximately 11% more likely to match the decision aid's choices. There was not a significant main effect of cue validity or trial. Further, no interactions were significant (see Table 1 -all tables are displayed in Appendix B).
Match in predicted performance. To examine reliance on the decision aid based on the match between the participant's and model's predictions about the candidates' performance, a repeated measures linear regression was conducted using the linear mixed-effects modeling package in R (Bates et al., 2014). Cue validity, decision aid presence, trial, and their interactions were entered as fixed effects. The absolute value of the difference between the participants' and model's performance predictions for each candidate was used as the dependent variable. To reduce effects of multicollinearity, predictors were centered before being entered into the model.
The results revealed a significant main effect of decision aid presence (B = -0.750, t(3071) = -5.969, p < .05), cue validity (B = -0.368, t(3071) = 2.932, p < .05), and trial (B = -0.036, t(3071) = -3.514, p < .05). These main effects were qualified by significant interactions. Specifically, there was a significant interaction between cue validity and decision aid presence (B = -0.307, t(3071) = 2.443, p < .05), such that the effect of the cue validity was stronger when the decision aid was provided (B = -0.675) than when it was not provided (B = -0.061). In other words, when the decision aid was provided, participants' performance predictions were on average 5.78% closer to the decision aid's performance predictions (see Figure 2). Additionally, there was a significant interaction between trial and cue validity (B = -0.038, t(3071) = -3.754, p < .05), such that the effect of trial was stronger when the cue validity was high (B = -.074) than when the cue validity was moderate (B = 0.002). For those in the high validity condition, performance predictions improved from 2.42% to 1.24% difference with the decision aid's predictions between Trial 1 and Trial 10. In contrast, the difference in performance predictions between participants in the moderate validity condition and the decision aid's predictions did not significantly change from Trial 1 to Trial 10 (Trial 1: 5.45%, Trial 10: 5.57%). This suggests that learning occurred over the 10 trials in the high validity condition but not the moderate validity condition Use of A decision Aid in Personnel selection (see Figure 3). Table 2 summarizes these results. Figures 4 through 7 show how participants' weighting of the different predictors (cognitive ability, conscientiousness, and unstructured interview ratings) changed over the course of the 10 trials for each of the study conditions. Exploratory analyses. As a result of a query made during the review process, we conducted exploratory analyses to examine whether our findings would be applicable to real work scenarios in which decision makers are often experienced. Specifically, we explored the role of hiring experience as a moderating variable in our analyses. First, we repeated the analyses predicting match in hiring choice, but we added hiring experience and all subsequent interactions as fixed effects. To reduce the effects of multicollinearity, hiring experience was mean centered. As can be seen in Table 3, none of the interactions including hiring experience were significant. We then repeated the analyses predicting match in performance predictions with mean-centered hiring experience and the subsequent interactions entered as fixed effects. As can be seen in Table 4, there is a significant four-way interaction among cue validity, decision aid presence, trial, and hiring experience. Figure 8 displays the four-way interaction. As can be seen in the figure, previous hiring experience does impact use of a decision aid. Specifically, when the decision aid is provided, the cue validity is moderate, and experience is low, decision makers only perform slightly worse than the decision aid itself. However, when the decision aid is provided, the cue validity is moderate, and experience is high, decision makers perform much worse than the decision aid. This suggests that more experience may lead people to be less willing to use the decision aid. However, when the decision aid is provided, cue validity is moderate, and experience is high, we do see an increase in the match in performance predictions between the decision aid and the participants over time. This suggests that those with higher experience increased their used of the decision aid across the 10 trials.

Discussion
The first study sought to examine the interactive effects of decision aid presence and cue validity on reliance on a decision aid over a series of hiring decisions. Cue validity was not a significant predictor when examining the degree of match in hiring choices. However, cue validity was a significant predictor when examining the degree of match in performance predictions, such that when cues had higher validity, there was a greater degree of match between participants' performance predictions and the model's performance predictions. Therefore, Hypothesis 1 was partially supported. When examining the degree of match in hiring choices and in performance predictions, the presence of the decision aid was a significant predictor, thus supporting Hypothesis 2. Only when examining the degree of match in performance predictions was the interaction significant, such that the greatest degree of match in performance predictions occurred when the decision aid was provided and cues were highly valid. Therefore, Hypothesis 3 was partially supported. Results also revealed a significant effect of decision trial, suggesting learning effect over time. Indeed, the exploratory analyses showed that individuals with higher experience tended to increase their use of the decision aid over time.

Study 2
Study 2 extended Study 1 in four ways. First, Study 2 utilized 20 decision trials instead of 10 (to better examine learning). Second, a third cue-validity condition was introduced to represent realistic hiring situations (R 2 = .204). Third, feedback was manipulated, such that half of the participants received feedback while the other half did not. Finally, handwriting analysis was added as a fourth cue and distractor to determine whether participants' cue weighting strategies could accommodate a cue with a near-zero relationship with job performance (Reilly & Chao, 1982;Schmidt & Hunter, 1998).

Method
Participants. The same attention check items from Study 1 were used. Usable data were obtained from 519 hiring professionals recruited using Qualtrics participant panels. Participants had approximately 7.7 (SD = 6.7) years of hiring experience. Most (93%) were currently employed, and those employed worked an average of 43.0 (SD = 10.0) hours per week. Approximately 52% of participants were female with an average age of 39.0 (SD = 11.3), and 80% were Caucasian.
Materials and procedure. This study used the same decision task used in Study 1 except with 20 instead of 10 selection decisions. The ordering of the 20 decisions was randomized to account for order effects. As in Study 1, participants were randomly assigned to the cue validity conditions. However, a third condition was added. Participants were randomly assigned to the high (R 2 = .962), moderate (R 2 = .504), or realistic (R 2 = .204) cue validity condition. The same procedures used Study 1 were used to create the realistic validity condition, except with a greater degree of random error introduced. The formula used to create the realistic cue validity was:

Equation 4
y r = round(logistic(logistic percent(.50 * x 1 + .40 * x 2 + .10 * x 3 + .0 * x 4 ) + 1.5 * (x r~N (0,1))) * 100) Where yr represents the candidate's eventual performance in the realistic condition, x 1 represents the candidate's cognitive ability score, x 2 represents the candidate's conscientiousness score, x 3 represents the candidate's interview score, and x4 represents the candidate's handwriting analysis score. Additionally, x r~N (0,1) represents the value randomly sampled from a standard normal distribution. In order to determine the actual validity of the cues once the random error has been introduced in the eventual performance of the candidates, the candidates' test scores were used to predict their eventual performance. The formula used to predict the candidates' eventual performance was:
Like Study 1, participants were randomly assigned to receive or not receive the decision aid. Participants were also randomly assigned to receive or not receive feedback regarding their performance predictions and hiring choices after each decision. Those assigned to the feedback condition were shown what their original performance predictions were, the actual job performance of both candidates once they were hired, and their prediction error for each candidate's performance. Participants assigned to not receive feedback did not receive any feedback regarding what their original performance predictions were, the actual job performance of both candidates once they were hired, or their prediction error for each candidate's performance.

Results
Match in hire choice. The analytic procedures used in Study 1 were also used in Study 2. Cue validity, model presence, the presence of feedback, trial, and their interactions were entered as fixed effects. The match in hire choice was entered as the dependent variable. Categorical predictors were centered using effects coding, and trial was mean centered.
No significant main effect of cue validity on match between the participants' and model's hiring choices emerged. However, there was a significant main effect of decision aid presence, B = 0.153, z = 3.98, p < .001. When the decision aid was provided, participants' hire choices were significantly more likely to match the model's hire choices than when model information was not provided. Additionally, there was a significant main effect of feedback on whether participants' hiring choices matched the model's choices, B = -0.088, z = -2.29, p = .022. When feedback was provided, participants' hiring choices were significantly less likely to match the model's choices. Further, there was a significant three-way interaction among cue validity, feedback, and trial, B = 0.017, z = 2.47, p = .013. Table 5 summarizes these model effects. Figure 8 displays the significant three-way interaction. As can be seen in the figure, when feedback is provided and cue validity is high, people are more likely to make choices that match the decision aid's over time. However, when the cue validity is moderate or realistic, there is essentially no change in the likelihood that participants' hiring choices match the decision aid's over time. This suggests that when feedback is provided and the cue validity is high, people are more likely to use the decision aid over time than when no feedback is provided or when the cues have realistic to moderate validity.
Match in performance predictions. Cue validity, decision aid presence, feedback, trial, and their interactions were entered as fixed effects. The absolute value of the difference between the participants' and model's performance predictions for each candidate was used as the dependent variable. The predictors were centered before being entered into the model. Table 6 displays the model effects. Results showed no significant effect of cue validity on the degree of similarity in the participants' and model's performance predictions. There was also no significant main effect of feedback. This likely suggests that our feedback manipulation did not significantly impact participant's reliance on the decision aid, and participants were unable to actually learn from the feedback in the way it was presented. However, a significant main effect of decision aid presence emerged (B = -0.534, t(20719) = -10.13, p < .05), such that when provided with the decision aid, participants' performance predictions were significantly more similar to the model's performance predictions than participants who were not provided with the decision aid. There was also a significant main effect of trial (B = -0.015, t(20719) = -4.78, p < .05), such that participants' predictions regarding the candidates' performance became more similar to the model's predictions over time. However, these main effects were qualified by significant interactions.
There was a significant interaction between cue validity and decision aid presence, F(2, 507) = 3.566, p = .029. This interaction was further qualified by a significant three-way interaction among cue validity, decision aid presence, and trial (F(2, 507) = 7.211, p < .001, see Figure 5). Therefore, post hoc comparisons of the simple slopes in the interaction using Bonferroni corrected p-values were conducted. Post hoc analyses revealed that for the high validity condition, the slope for trial when the decision aid was provided (B = -0.030) was significantly different than when the decision aid was not provided B < 0.001, z = -2.931, p = .027. Additionally, when the model was provided, the slope for trial in the high validity condition (B = -.030) was significantly different from the moderate validity condition (B = 0.006, z = -2.888, p = .030) and from the realistic validity condition, B = .022, z = -4.547, p < .001). There was also a significant three-way interaction among cue validity, feedback, and trial. Figures 9 and 10 display these interactions. Figures  11 through 22 show how participants' weighting of the different predictors (cognitive ability, conscientiousness, and unstructured interview ratings) changed over the course of the 20 trials for each of the study conditions. Exploratory analyses. As in Study 1, we explored the role of hiring experience as a moderator in our analyses. First, we repeated the analyses predicting match in hiring choice, but we added hiring experience and all subsequent interactions as fixed effects. To reduce the effects of multicollinearity, hiring experience was mean centered. As shown in Table 7, there was a significant interaction between experience and cue validity, B = .023, z = 2.28, p = .022. There were no other significant interactions with experience. For the purposes of illustration, Figure 11 displays the five-way interaction among decision aid presence, cue validity, feedback, trial, and experience. We then repeated the analyses predicting match in performance predictions with mean-centered hiring experience and the subsequent interactions entered as fixed effects. When predicting match in performance ratings, there were several significant interactions including experience. Specifically, there were significant four-way interactions among cue validity, decision aid presence, trial, and hiring experience, Bcue validity 1 = -.001, t = -2.205, p = .027, Bcue validity 2 =.002, t = 3.695, p < .001. There was also a significant four-way interaction among cue validity, feedback, trial, and hiring experience, Bcue validity 1 = -.001, t = -2.233, p = .026. Last, there was a significant four-way interaction among decision aid presence, feedback, trial, and hiring experience, B = .001, t = -2.478, p = .013. For brevity and ease of interpretation of all of these interactions, the five-way interaction is displayed in Figure 24. As can be seen in the figure, when the cue validity is high, the only difference observed was when the decision aid was provided. When people were provided with the decision aid they were more likely to make performance predictions that matched those of the decision aid, suggesting that they were using the decision aid. The figure also shows that in the moderate validity condition, we see that not providing feedback had a more pronounced effect on individuals with higher experience when they were provided with the decision aid. Specifically, they were less likely to make performance predictions that matched the decision aid over time. A similar pattern of decreased match in performance predictions over time occurred in the realistic validity condition when people were not provided with feedback.

Discussion
Study 2 was conducted to replicate the findings of Study 1 as well as test Hypotheses 4 and 5. All three analyses showed no significant main effect of cue validity on the degree to which participants' hiring choices and performance predictions match those made by the model. Therefore, Hypothesis 1 was not supported. In contrast, analyses did show a significant main effect of the presence of the decision aid on the degree to which participants' hiring choices and performance predictions matched those made by the model, supporting Hypothesis 2. Further, there was not a significant interaction when predicting the match in hiring choice.
However, when predicting similarity in performance predictions, there was a significant interaction between the presence of the decision aid and the validity of the cues. Specifically, when the decision aid was provided and the cues had high validity, participants' relied on the decision aid more than when the validity of the cues was realistic, but not when they had a moderate level of validity. Therefore, Hypothesis 3 was partially supported. Unfortunately, the observed validity of selection predictors more closely resembles the realistic validity condition (Schmidt & Hunter, 1998). Therefore, a practical reason people are hesitant to rely on decision aids is that decision aids err, which led to a slight (nonsignificant) decrease in the reliance over time in the realistic validity condition.
The presence of feedback was only a significant predictor when examining the match between participants' hiring choices and the model's hiring choices and in the opposite direction than predicted. Therefore, Hypothesis 4 was not supported. Across both analyses, there was not a significant interaction between the presence of feedback and cue validity. Therefore, Hypothesis 5 was also not supported. This is surprising, especially given the three-way interaction among trial, decision aid presence, and cue validity. The significant interaction would suggest that for the high validity condition, people are able to learn to use the decision aid when it is provided. However, people cannot learn about the validity of the decision aid without feedback, and this may depend on the form and content of feedback.
A secondary purpose of Study 2 was to increase the number of decisions participants made to better examine learning effects. In contrast to Study 1, there was a significant three-way interaction among the validity of the cues, the presence of the decision aid, and trial (see Figure 5). Participants experienced the greatest degree of learning when the decision aid was provided and the cues were highly valid. As the validity of the cues decreased, learning decreased. When the validity of the cues was weakest and thus more realistically mirrored the validity of current hiring cues, learning was not observed. There are two possible conclusions from this finding. First, there may have been too few decisions for participants to learn the predictive relationships in the presence of such high degrees of uncertainty. Alternatively, there may be so much uncertainty that the relationships are unlearnable.
Last, the exploratory analyses revealed several significant interactions with hiring experience. Together these findings suggest that providing feedback regarding the accuracy of one's decisions compared to that of a decision aid may be essential to getting people, especially those with greater hiring experience, to rely on decision aids.

GENERAL DISCUSSION
The purpose of these two studies was to examine the conditions under which people will utilize decision aids in a personnel selection context. Specifically, this study sought to examine whether (a) the mere presence of a decision aid will lead people to rely on the decision aid, (b) the validity of the predictors used in the selection context influence reliance on a decision aid, (c) the presence of feedback regarding one's predictions of a candidate's performance, and (d) the interactions among these factors influence reliance on a decision aid. In this study, the decision aid took the form of a statistical model that should be used to select the candidate to be hired. In both Study 1 and Study 2, the evidence clearly demonstrated that the mere presence of a decision aid leads people to rely on the decision aid. Although this is not an overly profound finding, it does have its own merit. By having a comparison group (those who did not receive the decision aid), we were able to examine whether participants were actually relying on the decision aid.
The finding that participants rely, to some extent, on a decision aid when it is provided, also has practical importance. Both studies demonstrated that when a decision aid is present, people do indeed rely on it, albeit not entirely. Therefore, organizations should provide individuals with a decision aid. This should ultimately make their performance predictions and hiring choices more accurate.
A second major finding in the present research is that the validity of the cues interacts with the presence of a decision aid to influence reliance on the decision aid when making performance predictions. In both Study 1 and Study 2, the validity of the cues interacted with the presence of the decision aid, such that there was the greatest degree of match between participants' predictions of candidates' performance and the model's predictions of the candidates' performance when the decision aid was provided and the validity of the cues was high. The importance of this finding is inherent in nearly all personnel selection research. Specifically, personnel selection research aims to identify and develop methods of assessment that maximize the relationship between selection tests and future job performance. This research demonstrated that reliance on the decision aid was greatest when the validity of the predictors was greatest. Unfortunately, the observed validity of selection predictors more closely resembles the realistic validity condition (Schmidt & Hunter, 1998). Therefore, a practical reason why people are hesitant to rely on decision aids is that decision aids do err. This leads people to distrust decision aids (e.g., Dietvorst et al., 2015). This is especially apparent in Figure 5. In the high validity condition, people saw the ac-curacy of the decision aid, which lead to an increase in the reliance over time. However, in the realistic validity condition, people saw the decision aid err, which led to a slight (nonsignificant) decrease in the reliance over time.
This research also sought to answer the call by researchers to examine the effect of immediate feedback on reliance on a decision aid in a personnel selection context (Slaughter & Kausel, 2014). The results of Study 2 showed that feedback did not have a significant effect on reliance on the decision aid. Nor did feedback interact with trial, decision aid presence, or the validity of the cues to influence reliance on the decision aid. This is surprising, especially given the three-way interaction among trial, decision aid presence, and cue validity. The significant interaction would suggest that for the high validity condition, people are able to learn to use the decision aid when it is provided. However, people cannot learn about the validity of the decision aid without feedback. It may be the case that the form and content of feedback may influence the reliance on a decision aid.

Limitations and Future Directions
One limitation of the present research is that participants simply saw candidates' scores, which may not resemble real hiring decisions where managers likely have more information about the candidates (e.g., résumés, references, etc.). In the context of the present research, participants' information was limited, which may have lowered the psychological fidelity of the hiring situation. Thus, the current studies may represent a best-case scenario in which fewer invalid cues are present that could draw a hiring manager's attention.
In Study 2, participants were randomly assigned to receive feedback or not receive feedback. Thus, one limitation of this research is that participants in the no feedback condition were not able to learn the validity of the cues. As such, there should be further investigation regarding whether providing feedback interacts with cue validity to influence reliance on the decision aid. Previous researchers have argued that resistance to using decision aids stems from a lack of trust in the aid (e.g., Dietvorst et al., 2015). Therefore, future research should assess participants' trust in a decision aid and how it changes over a series of decisions.

General Conclusions
This research sought to examine the effects of cue validity, presence of a decision aid, and feedback on reliance on a decision aid in a personnel selection context. Providing a decision aid led to reliance on that aid, at least to some degree. Finally, when the cues had high validity and the decision aid was provided, people learned to increase their reliance on the aid. Use of A decision Aid in Personnel selection Two-way interaction between decision aid presence and decision aid validity predicting match in performance predictions in Study 1. Note that the y-axis has been inverted to ease comparison across operationalizations of decision aid reliance. Error bars represent +/-1 standard error. Two-way interaction between trial and cue validity predicting match in performance predictions in Study 1. Note that the y-axis has been inverted to ease comparison across operationalizations of decision aid reliance. Error bars represent +/-1 standard error. Use of A decision Aid in Personnel selection Change in participants' weighting of the predictors over time when the decision aid is provided and the cue validity is high. Error bars represent +/-1 standard error.

Figure 5.
Change in participants' weighting of the predictors over time when the decision aid is provided and the cue validity is moderate. Error bars represent +/-1 standard error.
ReseaRch aRticles Figure 6.
Change in participants' weighting of the predictors over time when the decision aid is not provided and the cue validity is high. Error bars represent +/-1 standard error.

Figure 7.
Change in participants' weighting of the predictors over time when the decision aid is not provided and the cue validity is moderate. Error bars represent +/-1 standard error. Use of A decision Aid in Personnel selection Four-way interaction among decision aid presence, decision aid validity, trial, and hiring experience predicting match in performance predictions in Study 1. Note that the y-axis has been inverted to ease comparison across operationalizations of decision aid reliance. Error bars represent +/-1 standard error. Three-way interaction among cue validity, decision aid presence, and trial predicting match in performance predictions in Study 2. Note that the y-axis has been inverted for ease of comparison across operationalizations of decision aid reliance. Error bars represent +/-1 standard error. Figure 11.
Change in participants' weighting of the predictors over time when the decision aid is provided, cue validity is high, and feedback is provided. Error bars represent +/-1 standard error. Use of A decision Aid in Personnel selection Change in participants' weighting of the predictors over time when the decision aid is provided, cue validity is moderate, and feedback is provided. Error bars represent +/-1 standard error. Figure 13.
Change in participants' weighting of the predictors over time when the decision aid is provided, cue validity is realistic, and feedback is provided. Error bars represent +/-1 standard error.
ReseaRch aRticles Figure 14.
Change in participants' weighting of the predictors over time when the decision aid is provided, cue validity is realistic, and feedback is provided. Error bars represent +/-1 standard error. Figure 15.
Change in participants' weighting of the predictors over time when the decision aid is provided, cue validity is moderate, and no feedback is provided. Error bars represent +/-1 standard error. Use of A decision Aid in Personnel selection Figure 16.
Change in participants' weighting of the predictors over time when the decision aid is provided, cue validity is realistic, and no feedback is provided. Error bars represent +/-1 standard error. Figure 17.
Change in participants' weighting of the predictors over time when the decision aid is provided, cue validity is high, and feedback is provided. Error bars represent +/-1 standard error.
ReseaRch aRticles Figure 18.
Change in participants' weighting of the predictors over time when the decision aid is provided, cue validity is moderate, and feedback is provided. Error bars represent +/-1 standard error. Figure 19.
Change in participants' weighting of the predictors over time when the decision aid is provided, cue validity is realistic, and feedback is provided. Error bars represent +/-1 standard error. Use of A decision Aid in Personnel selection Change in participants' weighting of the predictors over time when the decision aid is provided, cue validity is high, and no feedback is provided. Error bars represent +/-1 standard error.

Figure 21.
Change in participants' weighting of the predictors over time when the decision aid is provided, cue validity is moderate, and no feedback is provided. Error bars represent +/-1 standard error.

Figure 23.
Five-way interaction among decision aid presence, cue validity, feedback, trial, and hiring experience predicting match in hiring choices in Study 2. Error bars represent +/-1 standard error. Use of A decision Aid in Personnel selection Five-way interaction among decision aid presence, cue validity, feedback, trial, and hiring experience predicting match in performance predictions in Study 2. Note that the y-axis has been inverted to ease comparison across operationalizations of decision aid reliance. Error bars represent +/-1 standard error.     Note. Bolded values are significant at p < .05. Variables were coded using effects coding. Cue validity 1 was coded as 1 = highly valid cues, 0 = moderately valid cues, -1 = low validity cues. Cue validity 2 was coded as 0 = highly valid cues, 1 = moderately valid cues, -1 = low validity cues. Decision aid presence was coded as 1 = decision aid present, -1 = decision aid not present. Feedback was coded as 1 = feedback provided, -1 = feedback not provided.      (Note: Percentile is the percentage of individuals who score less than the candidate. For example, a percentile score of 50 on the cognitive ability test means that the candidate performed better than 50% of the other individuals).

Participants assigned to decision aid condition
Recall that the prediction formula was: 0.50 x (cognitive ability score) + 0.40 x (conscientiousness score) + 0.10 x (unstructured interview score) = Predicted Job Performance Based on the scores for each candidate, the formula for each candidate is: Candidate A: 42.5 + 38 + 5 = Predicted Job Performance Candidate B: 41 + 3.6 + 7 = Predicted Job Performance

Instructions (all conditions)
Thanks for participating in this study. One of the major objectives of personnel selection is to predict candidates' performance based on available information.
In this study, we are interested in how people make hiring decisions using limited information. As such, your opinions are very important to us.
The following is from a large airline company. The firm was validating their selection procedures for the ticket agent job. As such, more than 200 applicants took a standardized personality test (conscientiousness factor), standardized cognitive ability test, and completed an unstructured interview before being hired. Three months after being hired, these same individuals were assessed by their supervisors in terms of their general performance.
On the following pages, you'll be presented with prehiring information of 20 pairs of applicants. Based on this information, for each pair, we ask you to • Make a prediction of each candidate's potential job performance as rated by his or her supervisor, and • Choose which candidate should be hired.

Information about the decision aid (decision aid present condition)
According to research examining various selection procedures, scores on standardized cognitive ability tests are good predictors of future job performance. Scores on the conscientiousness factor of standardized personality tests are moderate predictors of future job performance. Last, scores on unstructured interviews are weak predictors of future job performance, and scores on the handwriting analysis do not predict future job performance. Based on this information, one can use the following equation to estimate a candidate's job performance 0.50 x (cognitive ability score) + 0.40 x (conscientiousness score) + 0.10 x (unstructured interview score) + 0.00 x (handwriting analysis) = Predicted Job Performance