scholarly journals The Participation and Motivations of Grant Peer Reviewers: A Comprehensive Survey

2018 ◽  
Author(s):  
Stephen A Gallo ◽  
Lisa A Thompson ◽  
Karen B Schmaling ◽  
Scott R Glisson
Keyword(s):  
Peer Review ◽  
Selection Process ◽  
Funding Agency ◽  
Grant Applications ◽  
Time Period ◽  
Funding Agencies ◽  
Integral Role ◽  
Comprehensive Survey ◽  
Peer Reviewers ◽  
Grant Proposals

AbstractScientific peer reviewers play an integral role in the grant selection process, yet very little has been reported on the levels of participation or the motivations of scientists to take part in peer review. AIBS developed a comprehensive peer review survey that examined the motivations and levels of participation of grant reviewers. The survey was disseminated to 13,091 scientists in AIBS’s proprietary database. Of the 874 respondents, 76% indicated they had reviewed grant applications in the last 3 years; however, the number of reviews was unevenly distributed across this sample. Higher review loads were associated with respondents who had submitted more grant proposals over this time period, some of whom were likely to be study section members for large funding agencies. The most prevalent reason to participate in a review was to give back to the scientific community (especially among frequent grant submitters) and the most common reason to decline an invitation to review was lack of time. Interestingly, few suggested that expectation from the funding agency was a motivation to review. Most felt that review participation positively influenced their careers through improving grantsmanship and exposure to new scientific ideas. Of those who reviewed, respondents reported dedicating 2-5% of their total annual work time to grant review and, based on their self-reported maximum review loads, it is estimated they are participating at 56%-89% of their capacity, which may have important implications regarding the sustainability of the system. Overall, it is clear that participation in peer review is uneven and in some cases near capacity, and more needs to be done to create new motivations and incentives to increase the future pool of reviewers.

scholarly journals Gender and other potential biases in peer review: cross-sectional analysis of 38 250 external peer review reports

BMJ Open ◽  
2020 ◽  
Vol 10 (8) ◽  
pp. e035058 ◽  
Author(s):  
Anna Severin ◽  
Joao Martins ◽  
Rachel Heyard ◽  
François Delavy ◽  
Anne Jorstad ◽  
...  
Keyword(s):  
Peer Review ◽  
Multivariable Analysis ◽  
Evidence Base ◽  
Funding Agency ◽  
Cross Sectional ◽  
Cross Sectional Analysis ◽  
Grant Applications ◽  
Peer Reviewers ◽  
Review Reports ◽  
Sectional Analysis

ObjectivesTo examine whether the gender of applicants and peer reviewers and other factors influence peer review of grant proposals submitted to a national funding agency.SettingSwiss National Science Foundation (SNSF).DesignCross-sectional analysis of peer review reports submitted from 2009 to 2016 using linear mixed effects regression models adjusted for research topic, applicant’s age, nationality, affiliation and calendar period.ParticipantsExternal peer reviewers.Primary outcome measureOverall score on a scale from 1 (worst) to 6 (best).ResultsAnalyses included 38 250 reports on 12 294 grant applications from medicine, architecture, biology, chemistry, economics, engineering, geology, history, linguistics, mathematics, physics, psychology and sociology submitted by 26 829 unique peer reviewers. In univariable analysis, male applicants received more favourable evaluation scores than female applicants (+0.18 points; 95% CI 0.14 to 0.23), and male reviewers awarded higher scores than female reviewers (+0.11; 95% CI 0.08 to 0.15). Applicant-nominated reviewers awarded higher scores than reviewers nominated by the SNSF (+0.53; 95% CI 0.50 to 0.56), and reviewers from outside of Switzerland more favourable scores than reviewers affiliated with Swiss institutions (+0.53; 95% CI 0.49 to 0.56). In multivariable analysis, differences between male and female applicants were attenuated (+0.08; 95% CI 0.04 to 0.13) whereas results changed little for source of nomination and affiliation of reviewers. The gender difference increased after September 2011, when new evaluation forms were introduced (p=0.033 from test of interaction).ConclusionsPeer review of grant applications at SNSF might be prone to biases stemming from different applicant and reviewer characteristics. The SNSF abandoned the nomination of peer reviewers by applicants. The new form introduced in 2011 may inadvertently have given more emphasis to the applicant’s track record. We encourage other funders to conduct similar studies, in order to improve the evidence base for rational and fair research funding.

Peer Review or Lottery? A Critical Analysis of Two Different Forms of Decision-making Mechanisms for Allocation of Research Grants

2019 ◽  
Vol 44 (6) ◽  
pp. 994-1019 ◽  
Author(s):  
Lambros Roumbanis
Keyword(s):  
Decision Making ◽  
Peer Review ◽  
Review Process ◽  
Selection Process ◽  
Peer Review Process ◽  
Dynamic Selection ◽  
Funding Agencies ◽  
Epistemic Diversity ◽  
Alternative Decision ◽  
Made In

At present, peer review is the most common method used by funding agencies to make decisions about resource allocation. But how reliable, efficient, and fair is it in practice? The ex ante evaluation of scientific novelty is a fundamentally uncertain endeavor; bias and chance are embedded in the final outcome. In the current study, I will examine some of the most central problems of peer review and highlight the possible benefits of using a lottery as an alternative decision-making mechanism. Lotteries are driven by chance, not reason. The argument made in the study is that the epistemic landscape could benefit in several respects by using a lottery, thus avoiding all types of bias, disagreement, and other limitations associated with the peer review process. Funding agencies could form a pool of funding applicants who have minimal qualification levels and then select randomly within that pool. The benefits of a lottery would not only be that it saves time and resources, but also that it contributes to a more dynamic selection process and increases the epistemic diversity, fairness, and impartiality within academia.

scholarly journals Boom & Bust: Bird Stories for a Dry Country

2010 ◽  
Vol 16 (1) ◽  
pp. 70
Author(s):  
Stephen J. J. F. Davies
Keyword(s):  
Population Dynamics ◽  
Peer Review ◽  
Extended Time ◽  
Journal Editors ◽  
Time Period ◽  
Peer Reviewers ◽  
The Subject ◽  
Extended Time Period

As the subtitle says, this is a book of bird stories. The overall theme examines the population dynamics and evolution of some Australian inland birds in 12 separate essays, each by a different author or group of authors. It is apparent that the dates of preparation of these chapters cover an extended time period, one prepared by an author who died in 2001, and there is little attempt to integrate the stories. Each of the essays is well referenced and written by a respected authority on the bird, birds or topic that are the subject of the essay. In this way the book is authoritative, but it is also innovative, because it is clear that peer review has not been applied, or if applied then applied gently, because the authors have been allowed to speculate on the interpretations of the data they present. Modern journal editors and peer reviewers do their best to eliminate such speculation and yet it is from the presentation of such untested ideas that inquiry begins and science advances. The editors are to be applauded for taking this approach.

scholarly journals Measuring bias, burden and conservatism in research funding processes

F1000Research ◽  
2019 ◽  
Vol 8 ◽  
pp. 851 ◽  
Author(s):  
Susan Guthrie ◽  
Daniela Rodriguez Rincon ◽  
Gordon McInroy ◽  
Becky Ioppolo ◽  
Salil Gunashekar
Keyword(s):  
Peer Review ◽  
Research Funding ◽  
Assessment Process ◽  
Grant Funding ◽  
Funding Allocation ◽  
Complex Process ◽  
Funding Agencies ◽  
Grant Proposals ◽  
Funding Scheme ◽  
Measuring Bias

Background: Grant funding allocation is a complex process that in most cases relies on peer review. A recent study identified a number of challenges associated with the use of peer review in the evaluation of grant proposals. Three important issues identified were bias, burden, and conservatism, and the work concluded that further experimentation and measurement is needed to assess the performance of funding processes. Methods: We have conducted a review of international practice in the evaluation and improvement of grant funding processes in relation to bias, burden and conservatism, based on a rapid evidence assessment and interviews with research funding agencies. Results: The evidence gathered suggests that efforts so far to measure these characteristics systematically by funders have been limited. However, there are some examples of measures and approaches which could be developed and more widely applied. Conclusions: The majority of the literature focuses primarily on the application and assessment process, whereas burden, bias and conservatism can emerge as challenges at many wider stages in the development and implementation of a grant funding scheme. In response to this we set out a wider conceptualisation of the ways in which this could emerge across the funding process.

scholarly journals Low agreement among reviewers evaluating the same NIH grant applications

2018 ◽  
Vol 115 (12) ◽  
pp. 2952-2957 ◽  
Author(s):  
Elizabeth L. Pier ◽  
Markus Brauer ◽  
Amarette Filut ◽  
Anna Kaatz ◽  
Joshua Raclaw ◽  
...  
Keyword(s):  
Peer Review ◽  
Review Process ◽  
Peer Review Process ◽  
National Institutes Of Health ◽  
Success Rates ◽  
Grant Applications ◽  
Peer Reviewers ◽  
Funding Success ◽  
Numeric Rating

Obtaining grant funding from the National Institutes of Health (NIH) is increasingly competitive, as funding success rates have declined over the past decade. To allocate relatively scarce funds, scientific peer reviewers must differentiate the very best applications from comparatively weaker ones. Despite the importance of this determination, little research has explored how reviewers assign ratings to the applications they review and whether there is consistency in the reviewers’ evaluation of the same application. Replicating all aspects of the NIH peer-review process, we examined 43 individual reviewers’ ratings and written critiques of the same group of 25 NIH grant applications. Results showed no agreement among reviewers regarding the quality of the applications in either their qualitative or quantitative evaluations. Although all reviewers received the same instructions on how to rate applications and format their written critiques, we also found no agreement in how reviewers “translated” a given number of strengths and weaknesses into a numeric rating. It appeared that the outcome of the grant review depended more on the reviewer to whom the grant was assigned than the research proposed in the grant. This research replicates the NIH peer-review process to examine in detail the qualitative and quantitative judgments of different reviewers examining the same application, and our results have broad relevance for scientific grant peer review.

scholarly journals Navigating the Road to Success: Guidelines for Preparing Competitive Grant Proposals

2007 ◽  
Vol 2 (1) ◽  
pp. 23 ◽  
Author(s):  
Lynn L. Langille ◽  
Theresa Mackenzie
Keyword(s):  
Best Practices ◽  
Research Funding ◽  
Research Process ◽  
Grant Writing ◽  
Original Research ◽  
The Road ◽  
Grant Applications ◽  
Funding Agencies ◽  
Grant Proposals ◽  
Research Grant

Purpose - Difficulty in securing research funding has been cited as one barrier to the involvement of more librarians and information professionals in conducting original research. This article seeks to support the work of librarians who wish to secure research funding by describing some key approaches to the creation of successful grant applications. Approach - The authors draw on more than 15 years experience in supporting the development of successful research grant proposals. Twelve grant-writing best practices or ‘key approaches’ are described, and a planning timeline is suggested. Conclusions - Use of these best practices can assist researchers in creating successful research grant proposals that will also help streamline the research process once it is underway. It is important to recognize the competitive nature of research grant competitions, to obtain feedback from an internal review panel, and to use feedback from funding agencies to strengthen future grant applications.

scholarly journals Grant Review Feedback: Appropriateness and Usefulness

2021 ◽  
Vol 27 (2) ◽  
Author(s):  
Stephen A. Gallo ◽  
Karen B. Schmaling ◽  
Lisa A. Thompson ◽  
Scott R. Glisson
Keyword(s):  
Peer Review ◽  
Funding Agency ◽  
Constructive Feedback ◽  
Competitive Funding ◽  
Grant Proposals ◽  
Funding Success ◽  
Negative Perceptions ◽  
Grant Review ◽  
Research Grant

AbstractThe primary goal of the peer review of research grant proposals is to evaluate their quality for the funding agency. An important secondary goal is to provide constructive feedback to applicants for their resubmissions. However, little is known about whether review feedback achieves this goal. In this paper, we present a multi-methods analysis of responses from grant applicants regarding their perceptions of the effectiveness and appropriateness of peer review feedback they received from grant submissions. Overall, 56–60% of applicants determined the feedback to be appropriate (fair, well-written, and well-informed), although their judgments were more favorable if their recent application was funded. Importantly, independent of funding success, women found the feedback better written than men, and more white applicants found the feedback to be fair than non-white applicants. Also, perceptions of a variety of biases were specifically reported in respondents’ feedback. Less than 40% of applicants found the feedback to be very useful in informing their research and improving grantsmanship and future submissions. Further, negative perceptions of the appropriateness of review feedback were positively correlated with more negative perceptions of feedback usefulness. Importantly, respondents suggested that highly competitive funding pay-lines and poor inter-panel reliability limited the usefulness of review feedback. Overall, these results suggest that more effort is needed to ensure that appropriate and useful feedback is provided to all applicants, bolstering the equity of the review process and likely improving the quality of resubmitted proposals.

scholarly journals Gender and other potential biases in peer review: Analysis of 38,250 external peer review reports

2019 ◽  
Author(s):  
Anna Severin ◽  
Joao Martins ◽  
François Delavy ◽  
Anne Jorstad ◽  
Matthias Egger ◽  
...  
Keyword(s):  
Peer Review ◽  
Multivariable Analysis ◽  
Evidence Base ◽  
Calendar Period ◽  
Swiss National Science Foundation ◽  
Track Record ◽  
Grant Applications ◽  
Evaluation Scores ◽  
Peer Reviewers ◽  
Review Reports

Background The Swiss National Science Foundation (SNSF) supports fundamental and use-inspired research in all disciplines. Peer reviewers assess the proposals submitted to the SNSF. We examined whether the gender of applicants and reviewers and other factors influenced the summary scores awarded. Methods We analysed 38,250 reports on 12,294 grant applications across all disciplines 2006 to 2016. Proposals were rated on a scale from 1 (=worst) to 6 (=best) by 26,836 reviewers. We used linear mixed effects regression models adjusted for research topic, applicant's age, nationality, affiliation and calendar period to examine associations, and interactions between gender of the applicant and other variables. Results In univariable analysis, male applicants received more favourable evaluation scores than female applicants (+0.19 points; 95% CI 0.14-0.23), and male reviewers awarded higher scores than female reviewers (+0.12; 95% CI 0.08-0.15). Applicant-nominated reviewers awarded higher scores than reviewers nominated by the SNSF (+0.53; 95% CI 0.50-0.56), and reviewers affiliated with research institutions outside of Switzerland more favourable scores than reviewers affiliated with Swiss institutions (+0.53; 95% CI 0.49-0.56). In multivariable analysis, differences between male and female applicants were attenuated (to +0.08; 95% CI 0.04-0.13) whereas results changed little for source of nomination and affiliation of reviewers. There was an interaction between gender of applicant and reviewer, and between gender of applicant and calendar period. Male reviewers gave higher scores than female reviewers, with a greater difference for male than for female applicants (P=0.037 from test of interaction). The gender difference increased after September 2011, when new evaluation forms were introduced (P=0.033 from test of interaction). Conclusions Our study showed that peer review of grant applications at SNSF might be prone to biases stemming from different applicant and reviewer characteristics. The SNSF abandoned the nomination of peer reviewers by applicants, and made members of panels aware of the other systematic differences in scores. The new form introduced in 2011 may inadvertently have given more emphasis to the applicant’s track record, and a revision is now under discussion. We encourage other funders to conduct similar studies, in order to improve the evidence base for rational and fair research funding.

scholarly journals Quality Control through Peer Review Process in Scholarly Communication: Review of Related Literature

2017 ◽  
Vol 8 (3) ◽  
pp. 248
Author(s):  
Vishnu Kumar Gupta
Keyword(s):  
Quality Control ◽  
Peer Review ◽  
Review Process ◽  
Peer Review Process ◽  
Scholarly Communication ◽  
Related Literature ◽  
Grant Applications ◽  
The Status ◽  
Peer Reviewers

<p>This review of related literature on the theme of peer review process in scholarly communication explains the status of research on periodicals, grant peer review and fellowships. The paper highlights the quality related issues of the scholarly communication and peer review process. Peer reviewers are invited to grant applications or assess fellowship or review manuscript in a peer review process undertake the responsibility for confirming top-level quality and standards in their concerned subject fields. <em></em></p>

scholarly journals Gender and other potential biases in peer review: Analysis of 38,250 external peer review reports

2019 ◽  
Author(s):  
Anna Severin ◽  
Joao Martins ◽  
François Delavy ◽  
Anne Jorstad ◽  
Matthias Egger ◽  
...  
Keyword(s):  
Peer Review ◽  
Multivariable Analysis ◽  
Evidence Base ◽  
Calendar Period ◽  
Swiss National Science Foundation ◽  
Track Record ◽  
Grant Applications ◽  
Evaluation Scores ◽  
Peer Reviewers ◽  
Review Reports

Background The Swiss National Science Foundation (SNSF) supports fundamental and use-inspired research in all disciplines. Peer reviewers assess the proposals submitted to the SNSF. We examined whether the gender of applicants and reviewers and other factors influenced the summary scores awarded. Methods We analysed 38,250 reports on 12,294 grant applications across all disciplines 2006 to 2016. Proposals were rated on a scale from 1 (=worst) to 6 (=best) by 26,836 reviewers. We used linear mixed effects regression models adjusted for research topic, applicant's age, nationality, affiliation and calendar period to examine associations, and interactions between gender of the applicant and other variables. Results In univariable analysis, male applicants received more favourable evaluation scores than female applicants (+0.19 points; 95% CI 0.14-0.23), and male reviewers awarded higher scores than female reviewers (+0.12; 95% CI 0.08-0.15). Applicant-nominated reviewers awarded higher scores than reviewers nominated by the SNSF (+0.53; 95% CI 0.50-0.56), and reviewers affiliated with research institutions outside of Switzerland more favourable scores than reviewers affiliated with Swiss institutions (+0.53; 95% CI 0.49-0.56). In multivariable analysis, differences between male and female applicants were attenuated (to +0.08; 95% CI 0.04-0.13) whereas results changed little for source of nomination and affiliation of reviewers. There was an interaction between gender of applicant and reviewer, and between gender of applicant and calendar period. Male reviewers gave higher scores than female reviewers, with a greater difference for male than for female applicants (P=0.037 from test of interaction). The gender difference increased after September 2011, when new evaluation forms were introduced (P=0.033 from test of interaction). Conclusions Our study showed that peer review of grant applications at SNSF might be prone to biases stemming from different applicant and reviewer characteristics. The SNSF abandoned the nomination of peer reviewers by applicants, and made members of panels aware of the other systematic differences in scores. The new form introduced in 2011 may inadvertently have given more emphasis to the applicant’s track record, and a revision is now under discussion. We encourage other funders to conduct similar studies, in order to improve the evidence base for rational and fair research funding.

Sign in / Sign up

Export Citation Format

Share Document