Single- or double-blind review? A field study of system preference, reliability, bias, and validity

Many scientists experience a practice-preference gap about peer review. Single-blind review---where authors' identities are revealed to reviewers---is often used for evaluation. Yet, double-blind review---where authors' identities are concealed---is seen as more fair. To understand this gap, we compared both systems in a high-stakes field study: submissions to the Society for Judgment and Decision Making’s annual conference, the leading international conference on this topic. Each submission received both review types. Reviewers were randomly assigned to the review system and submissions. Selected conference talks were evaluated for quality, popularity, and subsequent publication status. We assessed the two systems on reliability, bias, and validity. On reliability, while both systems had moderate reliability, agreement was higher on what constituted a poor submission than a strong one (Anna Karenina Principle). On bias, double-blind reviews showed a slight bias against submission by women (Matilda Effect), while single-blind reviews showed a preference for submissions with senior co-authors (Matthew Effect). On validity, neither system predicted talk quality or popularity, but both predicted publication status. Author characteristics did not consistently predict outcomes. Thus, we suggest the costs of single-blind review do not outweigh its benefits. Yet, double-blind review is also not a perfect solution. We propose an equitable approach for selecting scientific work may be an informed lottery: use double-blind review to identify submissions of merit, then randomly choose from this set.

The cases for and against double-blind reviews

To date, the majority of authors on scientific publications have been men. While much of this gender bias can be explained by historic sexism and discrimination, there is concern that women may still be disadvantaged by the peer review process if reviewers’ biases lead them to reject publications with female authors more often. One potential solution to this perceived gender bias in the reviewing process is for journals to adopt double-blind reviews whereby neither the authors nor the reviewers are aware of each other’s identity and gender. To test the efficacy of double-blind reviews in one behavioral ecology journal (Behavioral Ecology, BE), we assigned gender to every authorship of every paper published for 2010–2018 in that journal compared to four other journals with single-blind reviews but similar subject matter and impact factors. While female authorships comprised only 35% of the total in all journals, the double-blind journal (BE) did not have more female authorships than its single-blind counterparts. Interestingly, the incidence of female authorship is higher at behavioral ecology journals (BE and Behavioral Ecology and Sociobiology) than in the ornithology journals (Auk, Condor, Ibis) for papers on all topics as well as those on birds. These analyses suggest that double-blind review does not currently increase the incidence of female authorship in the journals studied here. We conclude, at least for these journals, that double-blind review no longer benefits female authors and we discuss the pros and cons of the double-blind reviewing process based on our findings.

The Case For and Against Double-blind Reviews

To date, the majority of authors on scientific publications have been men. While much of this gender bias can be explained by historic sexism and discrimination, there is concern that women may still be disadvantaged by the peer review process if reviewers' unconscious biases lead them to reject publications with female authors more often. One potential solution to this perceived gender bias in the reviewing process is for journals to adopt double-blind reviews whereby neither the authors nor the reviewers are aware of each other's identities and genders. To test the efficacy of double-blind reviews, we assigned gender to every authorship of every paper published in 5 different journals with different peer review processes (double-blind vs. single blind) and subject matter (birds vs. behavioral ecology) from 2010-2018 (n = 4865 papers). While female authorships comprised only 35% of the total, the double-blind journal Behavioral Ecology did not have more female authorships than its single-blind counterparts. Interestingly, the incidence of female authorship is higher at behavioral ecology journals (Behavioral Ecology and Behavioral Ecology and Sociobiology) than in the ornithology journals (Auk, Condor, Ibis), for papers on all topics as well as those on birds. These analyses suggest that double-blind review does not currently increase the incidence of female authorship in the journals studied here. We conclude, at least for these journals, that double-blind review does not benefit female authors and may, in the long run, be detrimental.

Diversity in Social Work Doctoral Programs

Students of color (SOC) face multiple barriers to being admitted and retained in social work doctoral programs. We describe these hurdles and propose approaches to circumvent them. Admission challenges include a lack of guidance, financial hurdles, underpreparedness, and racism directed at applicants. Retention challenges include a dearth of mentorship, financial hardship, lack of academic support, and discrimination experienced by students. We propose institutional and structural responses to these challenges. Programs should recruit from milieus marked by diversity, de-emphasize standardized scores and blind reviews in admissions, increase faculty of color numbers, establish crosscutting mentorship networks, and expand academic and financial support. At the structural level, teaching credentials for core classes should be reconsidered, research and publication reimagined, and community engagement emphasized. Finally, moves like the replacement of tenured with adjunct faculty and the initiation of doctorate in social work programs should be examined for their implications to the career prospects of SOC.

Double blind reviews in software engineering venues: Practicability, promises and perils

The peer review process is central to the scientific method, the advancement and spread of research as well as crucial for individual careers. However, the single blind review process currently used in most Software Engineering (SE) venues is susceptible towards apparent and hidden biases, since reviewers know the identity of authors. In this paper, we perform a study on the benefits and costs that are associated with introducing double-blind reviews in SE venues. We surveyed the SE community’s opinion, interviewed experts on double-blind reviewing, and estimated the likelihood of reviewers being able to guess the authors. Our results indicate that double-blind reviewing could be introduced in large SE conferences at lower-than-generally believed costs and that the majority of the SE community is in favor of introducing it.