Higher Education Ph.D. Dissertations

A Study of the Inter-rater Reliability of University Application Readers in a Holistic Admissions Review Process

Date of Award

2017

Document Type

Dissertation

Degree Name

Doctor of Philosophy (Ph.D.)

Department

Higher Education Administration

First Advisor

Dafina-Lazarus Stewart (Advisor)

Second Advisor

Steven Cady (Other)

Third Advisor

Hyun Kyoung Ro (Committee Member)

Fourth Advisor

Patrick Pauken (Committee Member)

Abstract

In 2015, the American Council on Education surveyed undergraduate admission and enrollment management leaders at 338 four-year institutions to understand holistic admissions review (Espinosa, Gaertner, and Orfield, 2015). In the report titled, Race, Class and College Access: Achieving Diversity in a Shifting Legal Landscape, 92% of selective institutions reported utilizing a holistic admissions review process and 72% cited the Gratz v. Bollinger (2003) and Grutter v. Bollinger (2003) court cases as the impetus for increasing their use (Espinosa et al., 2015). Holistic admissions review has led to admission directors relying heavily on applicant qualities beyond GPA and test scores, such as leadership and extracurricular experience (Hoover, 2013; Sandlin, 2008; Spencer, 2008). Yet, little research has been conducted on the reliability of holistic admissions as an assessment or review tool, which is inherently subjective.

This study sought to examine a holistic admissions review process at one institution to simply show if variance occurred and possible explanations for it. The study analyzed reader reviews for approximately 15,000 individual undergraduate admission reviews over three years. The primary method was focused on the inter-rater reliability of the application reviews. If there was significant variance for particular reviews, there was an analysis of the content behind such reviews. Findings revealed that subjective measures of performance review, and in this case holistic admissions review, may not be susceptible to high levels of variability with 97% of reader reviews showing agreement for this study. Overall, just 3% of the subjective application reviews showed variability. The content analysis of the 208 application reviews with variance revealed 20 categories within six themes of Concern, Diversity, Extracurricular, Lacking Substance, Missing Information, and Writing Style. Additionally, female and international applicants were more represented in the variability group when comparing to applicants overall. Implications for future research and professional practice should focus on reader characteristics and experiences, quantity of readers utilized, reader review systematization, and reader hiring, training, and supervision.

Link to Full Text

Share

COinS