A Randomized Study to Assess the Effect of Including the Graduate Record Examinations (GRE) Results on Reviewers Scores for Underrepresented Minorities

Whether requiring Graduate Record Examinations (GRE) results for PhD applicants affects the diversity of admitted cohorts remains uncertain. This study randomized applications to two population health University of California San Francisco PhD programs to assess whether masking reviewers to applicant GRE results differentially affects reviewers’ scores for underrepresented minorities (URM) applicants from 2018-2020. Applications with GRE results and those without were randomly assigned to reviewers to designate scores for each copy (1-10, 1 being best). URM was defined as self-identification as African American/Black, Filipino, Hmong, Vietnamese, Hispanic/Latinx, Native American/Alaska Native, or Native Hawaiian/Other Pacific Islander. We used linear mixed models with random effects for applicant and fixed effects for each reviewer to evaluate the effect of masking the GRE results on the overall application score and whether this effect differed by URM status. Reviewer scores did not significantly differ for unmasked versus masked applications among non-URM applicants (b=0.15; 95% CI: [-0.03, 0.33]) or URM applicants (b=0.02, 95% CI: [-0.36, 0.40]). We did not find evidence that removing GREs differentially affected URM compared to non-URM students (b for interaction= -0.13, 95% CI: [-0.55, 0.29]). Within these doctoral programs, results indicate that GRE scores do not harm nor help URM applicants.

Do GRE scores help predict getting a physics Ph.D.? A comment on a paper by Miller et al.

A recent paper in Science Advances by Miller et al. concludes that Graduate Record Examinations (GREs) do not help predict whether physics graduate students will get Ph.D.’s. Here, I argue that the presented analyses reflect collider-like stratification bias, variance inflation by collinearity and range restriction, omission of parts of a needed correlation matrix, a peculiar choice of null hypothesis on subsamples, blurring the distinction between failure to reject a null and accepting a null, and an unusual procedure that inflates the confidence intervals in a figure. Release of results of a model that leaves out stratification by the rank of the graduate program would fix many of the problems.