“Where’s the I-O?” Artificial Intelligence and Machine Learning in Talent Management Systems

Artificial intelligence (AI) and machine learning (ML) have seen widespread adoption by organizations seeking to identify and hire high-quality job applicants. Yet the volume, variety, and velocity of professional involvement among I-O psychologists remains relatively limited when it comes to developing and evaluating AI/ML applications for talent assessment and selection. Furthermore, there is a paucity of empirical research that investigates the reliability, validity, and fairness of AI/ML tools in organizational contexts. To stimulate future involvement and research, we share our review and perspective on the current state of AI/ML in talent assessment as well as its benefits and potential pitfalls; and in addressing the issue of fairness, we present experimental evidence regarding the potential for AI/ML to evoke adverse reactions from job applicants during selection procedures. We close by emphasizing increased collaboration among I-O psychologists, computer scientists, legal scholars, and members of other professional disciplines in developing, implementing, and evaluating AI/ML applications in organizational contexts.

the extent such training is neglecting technical AI/ML expertise and applications, I-O psychologists may miss out on powerful predictive modeling tools that uniquely or more powerfully address complex organizational problems (Putka et al., 2018). In short, I-O psychologists can inform other disciplines-and themselves-in the AI/ML talent assessment arena.
Thus, our goals in this paper are threefold: (a) review some key potential benefits and limitations of AI/ML talent assessment applications, (b) highlight areas where I-O psychologists can help improve the development and use of AI/ ML, and (c) offer some current empirical evidence regarding applicant reactions to AI/ML technologies.

AI/ML Applications in Talent Assessment: The State of the Market
AI/ML tools have gained great traction within the talent acquisition arena. An estimated 33% of organizations have adopted such tools to identify, recruit, and select job applicants with greater speed and efficiency (Stephan, Brown, & Erickson, 2017). Numerous existing and forthcoming technology companies offer various AI/ML-based applications, such as those claiming to remove bias from job descriptions to identify more diverse candidates (e.g., TalentSonar), score asynchronous video interviews that aid with hiring (e.g., HireVue, Montage), leverage neuroscience and the engaging nature of games (e.g., Pymetrics, Knack), and evaluate person-job fit by scraping social media profiles (e.g., Entelo). Of course, these tools demonstrate their value when accompanied by professionally developed and standards-based research evidence that supports their reliability, validity, and fairness. Unfortunately, from our viewpoint, such evidence generally seems to be lacking.
To this latter point, I-O psychologists will continue to learn more as a community, as AI/ML tools are more frequently evaluated for their reliability, validity, and fairness across various contexts (i.e., organizational, consulting, legal, and policy making). However, the development and implementation rates of AI/ML applications are already outpacing relevant scientific research and legal guidelines. Companies' proprietary concerns restrict sharing; limited access by researchers to AI/ML tools prevents publications from emerging (Dastin, 2018); and relevant case law that would circumscribe the nature and application of AI/ML tools is scant. Therefore, at least currently, organizations seem to be placing great trust and acceptance in AI/ML talent acquisition tools, despite potential lingering limitations and ethical questions.

Organizational Benefits of AI/ML
Most appealingly, AI/ML applications offer efficiency by helping organizations sift through massive volumes of applicant data and consider larger applicant pools in a shorter time frame (e.g., Das, Pandey, & Rautaray, 2018). Even if AI/ML algorithms did not produce better employees, they lend potentially greater decision-making speed and efficiency than traditional assessments, saving organizations time and money. These savings may be the primary bottom-line improvement, such that AI/ML tools are actually changing or replacing the work of HR as much or more than their intended effect of changing the quality of hired applicants (Scholz, 2017). Regarding this latter effect, as we noted earlier, any improvements in selection accuracy from AI/ML over traditional methods remain a largely open question, and more evidence seems necessary in this space.
AI/ML methods could yield higher validity for organizations by offering access to new sources of data and applying sophisticated algorithms to them (e.g., Sajjadani, Sojourner, Kammeyer-Mueller, & Mykerezi, 2019). AI/ML tools have made various types of data more accessible for organizations to analyze (Bâra, Șimonca, Belciu, & Nedelcu, 2015), such as resumé content (e.g., Indira & Kumar, 2016, online social media activity (e.g., Park et al., 2015;Zang & Ye, 2015), and open-ended applicant responses (e.g., Campion, Campion, Campion, & Reider, 2016). Techniques such as Natural Language Processing (NLP) can collect and analyze these new data sources in a rapid, automated manner. One might argue that NLP, then, might improve reliability because, even if the text itself is unstandardized, the NLP analysis is standardized and faster to train than humans (given that the accuracy of human judgments fluctuates unreliably within and across people). Keeping the context and ethics of AI/ML applications in mind, organizations might use such text data to assign scores on psychological variables (e.g., verbal fluency, honesty, emotionality, aggression; Tausczik & Pennebaker, 2010) to predict important employee and organizational outcomes.
Within limited construct domains and settings, researchers have found comparable quality measurement between human raters and algorithmic techniques such as NLP (Campion et al., 2016;Park et al., 2015). For example, social media data have been shown to produce reliable personality measurement, at a level comparable to self-report measures and informant ratings (Park et al., 2015;Youyou, Kosinski, & Stillwell, 2015), although additional convergent and discriminant validity evidence is needed. Therefore, if used judiciously -and with the critical guidance of job analysis information -AI/ML tools can potentially yield more effective recruitment and hiring decisions over traditional approaches by measuring job-relevant attributes (e.g., using natural language that describes and predicts leadership potential; Campion et al., 2016).

Potential Limitations of AI/ML
As I-O psychologists collaborate on AI/ML tools with HR specialists, data scientists, organizational decision makers, lawyers, and policy makers, they should (strategically, over the course of their collaboration) soberly balance the actual and potential strengths and implications of AI/ML with its potential limitations (see Table 1).
Potential limitation #1: Data quality and decision making. To be clear, many of the primary concerns that arise with AI/ML tools and techniques pertain equally to traditional approaches: for example, (a) assessing the appropriateness of the nature and quality of the data captured, and the inferences and decisions derived from them (e.g., Sessions & Valtorta, 2006); and (b) assessing and addressing the potential problems (selection irrelevancies, biases) imbued within AI/ML datasets that impact reliability, validity, and fairness (see Dastin, 2018, for an example from Amazon). AI/ML applications must satisfy the same core psychometric, professional, and legal standards as traditional selection systems, the latter being outlined in Title VII as supported by the Uniform Guidelines (EEOC, 1978) and as addressed in the recently updated SIOP Principles for the Validation and Use of Personnel Selection Procedures (5th ed., 2018). In short, companies must provide evidence of minimal risk of adverse impact against members of protected groups (EEOC, 1978) and protect such findings through thorough job analytic and validity-based evidence that supports their employment systems. Ideally, AI/ML tools and algorithms will someday help to fulfill these obligationsnot obscure or threaten them.
Data quality and the appropriateness of the algorithm can jointly facilitate (or hinder) the effectiveness of personnel selection decisions. In general, algorithms seek out relationships or regularities in datasets for the purposes of clustering/classification or prediction. AI/ML algorithms can more flexibly adjust to the data to seek robust prediction than traditional modeling approaches, especially with larger quantities of data (e.g., via k-fold cross-validation or bootstrapping, or by averaging predictions across trees in a random forest or other model ensembles; see Putka et al., 2018). Specifically, algorithms seek out regularities in the data that are unconstrained by traditional model specifications found in t-tests, ANOVA, or linear regression models. To improve prediction (i.e., reduce mean squared error), predictive models are developed within a subset of the data (training sample) and subsequently tested on the remainder of the data that were held out (test sample). When based on sophisticated AI/ML algorithms, cross-validated predictions are potentially based on more complex relationships than can possibly be modeled in traditional approaches. AI/ML algorithms can be applied to any data set, yet the aforementioned complexity will generally not be detected when the data set is not sufficiently large enough and/or there is no complex relationship to detect in the first place.
Like more traditional modeling approaches (e.g., regression, ANOVA), AI/ML algorithms risk "garbage in, garbage out" when provided with low-quality data (Redman, 2018). For example, despite their sophistication, AI/ML algorithms often face the same impediment as traditional selection tools/methods: the criterion problem (e.g., Austin & Villanova, 1992;Campbell, McCloy, Oppler, & Sager, 1993). Limited appreciation for measuring the multidimensionality of criteria (e.g., performance, satisfaction) can conspire with having low-quality measures and a limited window of measurement to capture the performance processes of interest. A measure might be conceptually multidimensional, but empirically unidimensional, for instance, when supervisors hurriedly rate employees or have insufficient opportunity to observe certain behaviors, among other reasons (e.g., Murphy, 2008), or because the instrument does not measure the intended constructs reliably or representatively. AI/ML algorithms cannot correct for such real-world problems, just like psychometric corrections to Potential Benefits of AI/ML Potential Limitations of AI/ML

Reduced costs in terms of time, effort, money, and human resources
Quality of prediction is bound by the quality of the data obtained and the appropriateness of the algorithm used Increased power to handle large quantities of data Algorithms are to some extent a "black box," such that it is often unclear how or why AI/ML arrived at a given prediction Greater accessibility to previously burdensome forms of data (e.g., large volumes of resumes, social media content, interview responses and other forms of open-ended assessment data) Potential for future legal issues around data privacy and automated handling of personal data Potential for increased predictive accuracy for important individual and organizational outcomes of interest Unfavorable reactions toward AI/ML from job applicants, media, and the general public Note. AI/ML refers to "artificial intelligence/machine learning".  (Oswald, Ercan, McAbee, & Shaw, 2015). Perhaps future AI/ML-enhanced technologies will aid in gathering better criterion data that, in turn, improve the performance of AI/ML-based tests and algorithms. But history could disappointingly repeat itself here should AI/ML technologies and techniques remain more focused on the predictor than on the criterion. Potential limitation #2: "Black box" predictions. As a second and related limitation, the application of AI/ML algorithms to big data (e.g., random forests, deep learning) can yield "black box" predictive results. Here, "black box" means that even when the algorithm itself is well understood, and even when prediction is impressive, the predictive relationships are too complex to interpret (Adadi & Berrada, 2018). For example, random forests average across hundreds or thousands of different predictive trees that each reflect interactions (Breiman, 2001). In this case, end-users of AI/ML technologies may face challenges in justifying how specific predictors and outcomes in big data led to a set of hiring decisions (Tonidandel, King, & Cortina, 2018).
Even when professionals obtain personnel selection data using sound practices-and even when those data, in tandem with AI/ML algorithms, demonstrate robust incremental validity over traditional statistical approaches-it is still not guaranteed that the detected patterns of prediction or clustering predominantly reflect important and interpretable constructs (for a similar argument regarding big data and adverse impact, see Jacobs, Murphy, & Silva, 2013). For example, research on educational testing has revealed that automated and human scores can sometimes diverge for certain linguistic subgroups (Bridgeman, Trapani, & Attali, 2012), and both sets of scores can share common sources of contamination, such as essay length (Attali, 2007). Research of this nature is critical for exploring the benefits and limitations of AI/ML applications, and how job applicants can potentially "game" AI/ML talent assessment tools (e.g., receiving higher scores simply by typing longer responses).
Considering this formidable "black box" issue, recent interest in the topic of "explainable AI" has surged. Explainable AI refers to a set of techniques applied to AI/ ML results to enhance interpretation (Adadi & Berrada, 2018). For example, regarding our earlier example of random forests, researchers can use techniques such as local interpretable model-agnostic explanations (LIME) to understand which features most strongly drive predictions within specific predictive regions (Ribeiro, Singh, & Guestrin, 2016). However, this research area is relatively young, and challenges remain. Several explainable AI techniques involve decrements to flexibility and accuracy, such that more easily understood models tend to be less flexible and less predictive (Adadi & Berrada, 2018), which can diminish the original promise of incremental validity of AI/ ML tools over traditional statistical methods. Furthermore, explainable AI methods may indicate which features drive prediction but not why they drive prediction. Traditional job analysis should therefore drive the selection of measures as well as the understanding and interpretation of data.
Importantly, we note how AI/ML algorithms are often applied to very large datasets with many possible predictors (features) that may yield predictive benefit. In these cases, drilling large datasets down to a small handful of predictors for the sake of interpretability may be difficult or even counterproductive. Yet ultimately, the set of predictors used should remain defensible as job relevant. Here, the expertise of I-O psychologists seems critical, then, in partnering with relevant technical disciplines (e.g., computer science, applied statistics) to inform relevant data and the AI/ML tools applied to them. In this context, the traditional emphasis and practice of job analysis by I-O psychologists also remain highly relevant.
Potential limitation #3: Ethical and legal issues. Third, organizations should identify and appropriately address relevant ethical, professional, and legal issues when accessing and analyzing applicant data, such as by obtaining applicants' and employees' consent to collect and use their data and by using such data appropriately. To provide some ethical context: The media has celebrated AI/ML algorithms' successes at solving problems with clear goals and criteria that are almost uniformly valued in their own right: for example, winning at games of chess, Go, or Jeopardy!; or successfully identifying zip codes on envelopes (Markoff, 2011;Schaeffer & van den Herik, 2002;Thórisson, Bieger, Thorarensen, Sigurðardóttir, & Steunebrink, 2016). By contrast, employee selection involves multiple and complex goals and criteria that, as such, are informed by ethical, professional, and legal contexts (e.g., capturing selection-relevant characteristics of jobs and employees, avoiding adverse impact, accommodating disability). Thus, the regularities found in historical employment data by algorithms (AI/ML, regression, or otherwise) should not necessarily guide tomorrow's decisions. Sometimes organizations should pursue something very different from or even opposite of their empirical findings, such as when an algorithm captures and relies upon irrelevant or legally inappropriate data (e.g., race/ethnicity, gender, age; Tonidandel et al., 2018).
Vendors who develop AI/ML talent assessment products and the companies that use them should strive to maximize benefits while mitigating risks to both job applicants (e.g., data privacy) and the organization (e.g., legal repercussions). Regarding data privacy, the Global Data Protection Regulations (GDPR, 2016) in the European Union (EU) and European Economic Area (EEA) present challenges by requiring companies to disclose the use of AI/ML to applicants and remain transparent about the data used to make selection decisions (Liem et al., 2018). Similar legislation is emerging in the United States, such as in Illinois, where employers must obtain applicants' consent to use AI/ML in their hiring processes (Bologna, 2019). Furthermore, HIPAA requires AI/ML users to specify which types of data are needed to fulfill a specific objective, given the risk that AI/ML techniques could potentially identify sensitive health information from nonclinical sources such as social media (Weintraub, 2017). Research regarding the implications of such regulations on employers and applicants is needed, as AI/ML and the interpretation of these regulations co-evolve.
Potential limitation #4: Applicant reactions. A final factor that could impede the progress of AI/ML-based talent assessment products is negative reactions, both from job applicants and from the broader population. Recent events like the Cambridge Analytica scandal, in which big data were used to influence national elections (Granville, 2018), and the Chinese government's use of AI/ML to establish a "social credit score system" for its citizens (Cao, 2019), have generated data privacy concerns and high levels of distrust of AI/ML systems (e.g., Meos & Roosipuu, 2018;Rao & Cameron, 2018). It would be unfortunate if litigation and negative perceptions of AI/ML forestalled significant developments that would provide high benefit to both employees and organizations.
For several reasons, applicants may perceive selection procedures and decisions as unfair when such processes are left to AI/ML alone. First, many people are currently unfamiliar with how AI/ML works (Johnson & Verdicchio, 2017), and therefore they may perceive AI/ML-based decision processes as too narrow, impersonal, inaccurate, or opaque. Second, AI/ML can operate more or less automatically on a massive number of applicants using a uniform process, which might deprive applicants of their ability (or perceived ability) to appeal decisions, provide voice, exercise other forms of process control (Leventhal, 1980) or otherwise erode the reality and perceptions of procedural fairness (Thibaut & Walker, 1975). Conversely, with human decision makers such as recruiters or hiring managers, applicants might feel that they have more control in an interview, where they can provide unique information, ask questions, and engage in interpersonal conversation (or impression management; Gilmore & Ferris, 1989). However, human decision makers clearly have their own idiosyncratic and fluctuating biases that may be no better or more manageable by applicants than those found in AI/ ML applications (Judge, Cable, & Higgins, 2000). Third, organizational practices signal the organization's values to applicants (e.g., Bangerter, Roulin, & König, 2012), and applicants could perceive the organization, rightly or wrongly, as not caring about applicants (or human capital in general) by essentially outsourcing the selection process to AI/ML technologies.
Empirical research: Reactions to AI/ML. A subset of the authors (Gonzalez, Capman, Martin, Justenhoven, & Preuss, 2019) conducted research on reactions toward AI/ ML in employee selection. We hypothesized that participants in an applicant simulation would generally react negatively during selection procedures automated by current AI/ML tools rather than managed by a human (e.g., recruiter, hiring manager) and that these negative reactions would spill over toward the organizations that adopt these procedures. Furthermore, these differences in applicant reactions would likely be more apparent in hired applicants, given how past applicant reactions research shows that applicants who are not hired display uniformly negative reactions (Bauer, Maertz, Dolen, & Campion, 1998).
The study followed a 2 (decision-maker: human, AI) x 2 (outcome favorability: hired, not hired) between-subjects experimental design. Data were collected from 192 Amazon Mechanical Turk participants (59.4% male; 79.2% Caucasian; age M = 35.49 years, SD = 10.90; work experience M = 13.68 years, SD = 10.70). Participants imagined they were applying for a job in which either a hiring manager or an AI/ML tool would make the hiring decision. For the sake of brevity, we note they completed several reaction measures, mostly adapted from existing organizational justice measures (Bauer et al., 2001;Colquitt, 2001). Participants were next told to imagine being either hired or not hired for the job, and reported additional reactions toward the organization. Last, participants reported their knowledge, experience, and general attitudes involving AI/ML. All multi-item measures demonstrated acceptable reliability (αs ≥ .80). 1 First, we found that participants generally reacted less favorably to AI/ML decision makers, compared to human decision makers ( Table 2)-particularly expressing stronger interpersonal concerns (e.g., dignified treatment, communication) than procedural concerns (e.g., consistency, accuracy). Furthermore, participants trusted the organization less and they were less likely to promote the organization if they either had an AI/ML decision maker or were not hired (Table  3). Participants' open-ended comments revealed concerns regarding the impersonal nature and inaccuracy of AI/ML. The latter concern regarding inaccuracy is striking because meta-analytic evidence supports the validity of selection decisions that use standardized decision rules, versus subjective or holistic judgments that tend to base themselves on more inconsistent and unreliable information (Kuncel, Klieger, Connelly, & Ones, 2013; see also, Highhouse, 1997). Perhaps inaccuracy concerns arose because appli-  Note. N = 192. Scores on all dependent variables ranged from 1 to 5. For all F-tests, df 1 = 1 and df 2 = 188. a To aid with interpreting outcome favorability main effects, we present the M (SD) when collapsing across the two outcome favorability conditions. b To aid with interpreting decision-maker main effects, we present the M (SD) when collapsing across the two decision-maker conditions. c We present the grand mean and SD, collapsing across all four experimental conditions.

Analysis of Variance Main Effects and Interactions of Decision Maker and Outcome Favorability on Reactions to Selection Decisions
cants do not consider AI/ML to be standardized in the same way as job knowledge tests or structured application forms. Second, hired participants were significantly more distrustful and less likely to promote the organization if the decision maker was an AI/ML rather than a human, whereas nonhired participants exhibited uniformly less positive reactions, regardless of the type of decision maker (Figures 1  and 2). Thus, perhaps the use of AI/ML in employee selec-tion sets a negative tone even for hired applicants entering the organization. Last, some of these less positive reactions toward AI/ML (i.e., lower trust in the decision criteria, lower likelihood of promoting the organization) were reduced when participants had higher familiarity toward AI/ML (Table 4; Figures 3 and 4). Thus, less trust toward AI/ML may partly stem from unfamiliarity with how AI/ML functions (Johnson & Verdicchio, 2017). Certainly these findings can FIGURE 1.
Significant two-way interaction between decision-maker and outcome favorability on organizational distrust, F(1,188) = 7.14, p < .01, η p 2 = .04, using two-way analysis of variance (ANOVA). With the exception of the human/ rejected versus AI/rejected conditions, all conditions differed significantly from one another, ps < .001.

FIGURE 2.
Significant two-way interaction between decision-maker and outcome favorability on organizational promotion, F(1,188) = 4.86, p < .05, η p 2 = .03, using two-way analysis of variance (ANOVA). With the exception of the human/ rejected versus AI/rejected conditions, all conditions differed significantly from one another, ps < .01.

Personnel Assessment And decisions
Ai/ml And i-o Psychology   Statistically two-way significant interaction between AI/ML familiarity and decision maker on organizational distrust, B = .38, SE = .18, p < .05, using hierarchical linear regression. Based on follow-up simple slopes analyses, slopes for lower and higher AI/ML familiarity are modeled at -1SD below the mean and +1SD above the mean, respectively. Participants with lower AI/ML familiarity reported less trust in the factors used for decision making when there was an AI/ML decision maker, relative to a human decision maker, B = -.79, SE = .23, p <.001. Participants with higher AI/ ML familiarity reported similar, higher levels of trust on average, regardless of the decision maker, B = -.10, SE = .27, p = .67. Interactions were plotted and probed using procedures from Aiken and West (1991) and Dawson (2014).
be usefully replicated in actual hiring situations versus this simulation study; yet the findings may still have important implications for organizations that use AI/ML-based selection tools, as we discuss below.

Where Will AI/ML Go Next? A Call for Collaboration
Given the many organizational issues surrounding data privacy, job relevance, legal challenges, and negative applicant reactions, we believe that well-trained I-O psychology researchers and practitioners should play a much stronger role in the development and assessment of AI/ML-based organizational tools. I-O psychologists obviously do not control the speed of the development and implementation of AI/ML; but we strongly suggest that AI/ML companies will obtain a competitive advantage by involving those I-O psychologists who are trained experts in developing and implementing their employment-based technologies in a practical, legal, valid, and fair manner.
Together, I-O psychologists and data scientists can contribute a diverse and relevant range of knowledge, skills, and abilities to improve the quality and viability of AI/ML talent assessment technologies (Handler & Landers, 2018). For example, data scientists and AI/ML experts possess unique training and expertise in programming, data mining/ wrangling, linear algebra, statistical theory, and advanced analytical modeling, whereas I-O psychologists offer proficiency in personnel selection and recruiting practices, measure development, and psychometrics (i.e., ensuring stronger measurement "signals" in the data for AI/ML algorithms to use), and Equal Employment Opportunity (EEO) law.
Regarding our findings on applicant reactions, organi- Significant three-way interaction between AI/ML familiarity, outcome favorability, and decision maker on organizational promotion, B = .70, se = .32, p < .05, using hierarchical linear regression analysis. Based on follow-up simple slopes analyses, slopes for lower and higher AI/ML familiarity are modeled at -1SD below the mean and +1SD above the mean, respectively. Among participants with lower AI/ML familiarity, participants with a favorable outcome (i.e., accepted) reported were generally less likely to promote the organization when there was an AI/ML decision maker, relative to a human decision maker, whereas the decision maker did not affect organizational promotion among participants with an unfavorable outcome (i.e., rejected), B = -1.21, se = .40, p < .01. This interaction was not found among participants with higher AI/ML familiarity, B = .05, se = .40, p = .91. Interactions were plotted and probed using procedures from Aiken and West (1991), Dawson (2014), and Dawson and Richter (2006).
Personnel Assessment And decisions Ai/ml And i-o Psychology zations should consider how to frame AI/ML-based selection procedures when communicating with applicants and the context surrounding the assessment process. Organizations may also consider educating applicants up front on how and why they use AI/ML. Even if the algorithms are not always as transparent as would be ideal, the process by which the algorithms are applied can be made clear to applicants. This approach could alleviate applicants' interpersonal concerns because the organization is investing time to ensure that applicants feel respected and treated fairly (Bies & Moag, 1986). Our participants in the hypothetical hiring scenario were not told why AI/ML technology was used or how it would make selection decisions. It would therefore be theoretically and practically meaningful to investigate whether organizations can tailor their messaging, and the AI/ML tools themselves, to proactively influence applicant reactions to AI/ML. However, findings from Langer, König, and Fitili (2018) suggest that such transparency poses a double-edged sword, in that it can positively affect organizational attractiveness indirectly, via perceived open treatment, but transparency can also have a direct negative effect on organizational attractiveness. Communication strategies and the assessment context are thus potentially important factors to examine, representing fruitful areas for partnership between researchers from I-O psychology and computer science. The current state of affairs in the development of AI/ ML talent management tools reflects a yawning gap between I-O psychologists-who study the science and practice of personnel selection and have learned from decades of organizational, legal, and ethical lessons in the selection and workplace contexts-and computer scientists and applied statisticians-who implement AI/ML talent assessment technologies. Although data scientists and AI/ML experts are inventing and advancing talent assessment technologies, the practical experience and scientific knowledge base of I-O psychologists can critically aid in developing and implementing psychometrically reliable and valid measures. Having psychometrically solid measures ultimately provides high-quality data (not just "big data"), thereby enabling AI/ML algorithms to yield more substantively interpretable results that are then more defensible at the legal, organizational, individual, and ethical levels. Thus, we argue that by forming cross-functional partnerships with data scientists and AI/ML experts, I-O psychologists can contribute invaluably to today's exponential progress, substantially raising the value of AI/ML technologies and ensuring that organizations implement fair, practical, and valid AI/ ML solutions.

Conclusion
More broadly, we encourage researchers and practitioners across all relevant scientific disciplines (e.g., computer sciences, applied statistics, human factors, I-O psychology) and organizational roles (e.g., human resources, line and staff managers, employees) to strengthen their collaborative communities, working together to continue investigating the vast, interrelated, and interesting applications and implications of AI/ML in talent assessment (e.g., recruiting, promotion, retention, diversity and inclusion). The most urgent task, we believe, requires sharing AI/ML development, implementation, and evaluation experiences and developing practical community-wide recommendations for researchers to implement reliable, valid, and fair AI/ML-based solutions that conform to ethical, professional, and legal guidelines. This will be no small feat, but meaningful progress can be made. Such progress for AI/ML tools will require teamwork, expertise, and difficult discussions of the sort that cannot be conducted or solved by AI/ ML tools themselves-not just yet.