scholarly journals Expertise versus Bias in Evaluation: Evidence from the NIH

2017 ◽  
Vol 9 (2) ◽  
pp. 60-92 ◽  
Author(s):  
Danielle Li
Keyword(s):  
Peer Review ◽  
National Institutes Of Health ◽  
Versus Bias ◽  
The Us ◽  
Informational Advantage

Evaluators with expertise in a particular field may have an informational advantage in separating good projects from bad. At the same time, they may also have personal preferences that impact their objectivity. This paper examines these issues in the context of peer review at the US National Institutes of Health. I show that evaluators are both better informed and more biased about the quality of projects in their own area. On net, the benefits of expertise weakly dominate the costs of bias. As such, policies designed to limit bias by seeking impartial evaluators may reduce the quality of funding decisions. (JEL D82, H51, I10, I23, O38)

scholarly journals Low agreement among reviewers evaluating the same NIH grant applications

2018 ◽  
Vol 115 (12) ◽  
pp. 2952-2957 ◽  
Author(s):  
Elizabeth L. Pier ◽  
Markus Brauer ◽  
Amarette Filut ◽  
Anna Kaatz ◽  
Joshua Raclaw ◽  
...  
Keyword(s):  
Peer Review ◽  
Review Process ◽  
Peer Review Process ◽  
National Institutes Of Health ◽  
Success Rates ◽  
Grant Applications ◽  
Peer Reviewers ◽  
Funding Success ◽  
Numeric Rating

Obtaining grant funding from the National Institutes of Health (NIH) is increasingly competitive, as funding success rates have declined over the past decade. To allocate relatively scarce funds, scientific peer reviewers must differentiate the very best applications from comparatively weaker ones. Despite the importance of this determination, little research has explored how reviewers assign ratings to the applications they review and whether there is consistency in the reviewers’ evaluation of the same application. Replicating all aspects of the NIH peer-review process, we examined 43 individual reviewers’ ratings and written critiques of the same group of 25 NIH grant applications. Results showed no agreement among reviewers regarding the quality of the applications in either their qualitative or quantitative evaluations. Although all reviewers received the same instructions on how to rate applications and format their written critiques, we also found no agreement in how reviewers “translated” a given number of strengths and weaknesses into a numeric rating. It appeared that the outcome of the grant review depended more on the reviewer to whom the grant was assigned than the research proposed in the grant. This research replicates the NIH peer-review process to examine in detail the qualitative and quantitative judgments of different reviewers examining the same application, and our results have broad relevance for scientific grant peer review.

scholarly journals The data and analysis underlying NIH’s decision to cap research support lacked rigor and transparency: a commentary

2017 ◽  
Author(s):  
A Cecile JW Janssens ◽  
Gary W Miller ◽  
K Venkat Narayan
Keyword(s):  
Scientific Output ◽  
National Institutes Of Health ◽  
Quality Data ◽  
Rigorous Analysis ◽  
Research Support ◽  
High Quality Data ◽  
Grant Support ◽  
Threshold Data ◽  
The Us

The US National Institutes of Health (NIH) recently announced that they would limit the number of grants per scientist and redistribute their funds across a larger group of researchers. The policy was withdrawn a month later after criticism from the scientific community. Even so, the basis of this defunct policy was flawed and it merits further examination. The amount of grant support would have been quantified using a new metric, the Grant Support Index (GSI), and limited to a maximum of 21 points, the equivalent of three R01 grants. This threshold was decided based upon analysis of a new metric of scientific output, the annual weighted Relative Citation Ratio, which showed a pattern of diminishing returns at higher values of the GSI. In this commentary, we discuss several concerns about the validity of the two metrics and the quality of the data that the NIH had used to set the grant threshold. These concerns would have warranted a re-analysis of new data to confirm the legitimacy of the GSI threshold. Data-driven policies that affect the careers of scientists should be justified by nothing less than a rigorous analysis of high-quality data.

scholarly journals NIH peer review percentile scores are poorly predictive of grant productivity

eLife ◽  
2016 ◽  
Vol 5 ◽  
Author(s):  
Ferric C Fang ◽  
Anthony Bowen ◽  
Arturo Casadevall
Keyword(s):  
Peer Review ◽  
National Institutes Of Health ◽  
Recent Analysis ◽  
Success Rates ◽  
Grant Applications ◽  
The Us ◽  
Percentile Scores

Peer review is widely used to assess grant applications so that the highest ranked applications can be funded. A number of studies have questioned the ability of peer review panels to predict the productivity of applications, but a recent analysis of grants funded by the National Institutes of Health (NIH) in the US found that the percentile scores awarded by peer review panels correlated with productivity as measured by citations of grant-supported publications. Here, based on a re-analysis of these data for the 102,740 funded grants with percentile scores of 20 or better, we report that these percentile scores are a poor discriminator of productivity. This underscores the limitations of peer review as a means of assessing grant applications in an era when typical success rates are often as low as about 10%.

scholarly journals The data and analysis underlying NIH’s decision to cap research support lacked rigor and transparency: a commentary

2017 ◽  
Author(s):  
A Cecile JW Janssens ◽  
Gary W Miller ◽  
K Venkat Narayan
Keyword(s):  
Scientific Output ◽  
National Institutes Of Health ◽  
Quality Data ◽  
Rigorous Analysis ◽  
Research Support ◽  
High Quality Data ◽  
Grant Support ◽  
Threshold Data ◽  
The Us

The US National Institutes of Health (NIH) recently announced that they would limit the number of grants per scientist and redistribute their funds across a larger group of researchers. The policy was withdrawn a month later after criticism from the scientific community. Even so, the basis of this defunct policy was flawed and it merits further examination. The amount of grant support would have been quantified using a new metric, the Grant Support Index (GSI), and limited to a maximum of 21 points, the equivalent of three R01 grants. This threshold was decided based upon analysis of a new metric of scientific output, the annual weighted Relative Citation Ratio, which showed a pattern of diminishing returns at higher values of the GSI. In this commentary, we discuss several concerns about the validity of the two metrics and the quality of the data that the NIH had used to set the grant threshold. These concerns would have warranted a re-analysis of new data to confirm the legitimacy of the GSI threshold. Data-driven policies that affect the careers of scientists should be justified by nothing less than a rigorous analysis of high-quality data.

Feasibility and Clinical Implications of Remotely Guided Ultrasound Examinations

2020 ◽  
Vol 91 (7) ◽  
pp. 592-596
Author(s):  
Quinn Dufurrena ◽  
Kazi Imran Ullah ◽  
Erin Taub ◽  
Connor Leszczuk ◽  
Sahar Ahmad
Keyword(s):  
Clinical Utility ◽  
Regional Wall Motion ◽  
Subjective Assessment ◽  
Clinical Implications ◽  
Pilot Studies ◽  
Objective Quality ◽  
The Us ◽  
Regional Wall Motion Abnormalities ◽  
Wall Motion Abnormalities

BACKGROUND: Remotely guided ultrasound (US) examinations carried out by nonmedical personnel (novices) have been shown to produce clinically useful examinations, at least in small pilot studies. Comparison of the quality of such exams to those carried out by trained medical professionals is lacking in the literature. This study compared the objective quality and clinical utility of cardiac and pulmonary US examinations carried out by novices and trained physicians.METHODS: Cardiac and pulmonary US examinations were carried out by novices under remote guidance by an US expert and independently by US trained physicians. Exams were blindly evaluated by US experts for both a task-based objective score as well as a subjective assessment of clinical utility.RESULTS: Participating in the study were 16 novices and 9 physicians. Novices took longer to complete the US exams (median 641.5 s vs. 256 s). For the objective component, novices scored higher in exams evaluating for pneumothorax (100% vs. 87.5%). For the subjective component, novices more often obtained clinically useful exams in the assessment of cardiac regional wall motion abnormalities (56.3% vs. 11.1%). No other comparisons yielded statistically significant differences between the two groups. Both groups had generally higher scores for pulmonary examinations compared to cardiac. There was variability in the quality of exams carried out by novices depending on their expert guide.CONCLUSION: Remotely guided novices are able to carry out cardiac and pulmonary US examinations with similar, if not better, technical proficiency and clinical utility as US trained physicians, though they take longer to do so.Dufurrena Q, Ullah KI, Taub E, Leszczuk C, Ahmad S. Feasibility and clinical implications of remotely guided ultrasound examinations. Aerosp Med Hum Perform. 2020; 91(7):592–596.

A Study of Medical Board Peer Reviews in Nevada

2010 ◽  
Vol 96 (1) ◽  
pp. 20-29
Author(s):  
Jerry C. Calvanese
Keyword(s):  
Peer Review ◽  
Board Members ◽  
Study Objective ◽  
Clark County ◽  
Peer Reviews ◽  
Medical Examiners ◽  
Washoe County ◽  
Total Average Cost ◽  
Total Average

ABSTRACT Study Objective: The purpose of this study was to obtain data on various characteristics of peer reviews. These reviews were performed for the Nevada State Board of Medical Examiners (NSBME) to assess physician licensees' negligence and/or incompetence. It was hoped that this data could help identify and define certain characteristics of peer reviews. Methods: This study examined two years of data collected on peer reviews. The complaints were initially screened by a medical reviewer and/or a committee composed of Board members to assess the need for a peer review. Data was then collected from the peer reviews performed. The data included costs, specialty of the peer reviewer, location of the peer reviewer, and timeliness of the peer reviews. Results: During the two-year study, 102 peer reviews were evaluated. Sixty-nine percent of the peer-reviewed complaints originated from civil malpractice cases and 15% originated from complaints made by patients. Eighty percent of the complaint physicians were located in Clark County and 12% were located in Washoe County. Sixty-one percent of the physicians who performed the peer reviews were located in Washoe County and 24% were located in Clark County. Twelve percent of the complaint physicians were in practice in the state for 5 years or less, 40% from 6 to 10 years, 20% from 11 to 15 years, 16% from 16 to 20 years, and 13% were in practice 21 years or more. Forty-seven percent of the complaint physicians had three or less total complaints filed with the Board, 10% had four to six complaints, 17% had 7 to 10 complaints, and 26% had 11 or more complaints. The overall quality of peer reviews was judged to be good or excellent in 96% of the reviews. A finding of malpractice was found in 42% of the reviews ordered by the medical reviewer and in 15% ordered by the Investigative Committees. There was a finding of malpractice in 38% of the overall total of peer reviews. The total average cost of a peer review was $791. In 47% of the peer reviews requested, materials were sent from the Board to the peer reviewer within 60 days of the original request and 33% took more than 120 days for the request to be sent. In 48% of the reviews, the total time for the peer review to be performed by the peer reviewer was less than 60 days. Twenty seven percent of the peer reviews took more than 120 days to be returned. Conclusion: Further data is needed to draw meaningful conclusions from certain peer review characteristics reported in this study. However, useful data was obtained regarding timeliness in sending out peer review materials, total times for the peer reviews, and costs.

Neuro-Optometric Treatment of Complications from a Cerebellar Astrocytoma

2020 ◽  
pp. 304-312
Keyword(s):  
Visual System ◽  
Common Site ◽  
Us Army ◽  
Cranial Nerve Palsies ◽  
Pilocytic Astrocytomas ◽  
The Us ◽  
Extent Of Damage ◽  
Functional Components ◽  
The Brain

Background: Insult to the brain, whether from trauma or other etiologies, can have a devastating effect on an individual. Symptoms can be many and varied, depending on the location and extent of damage. This presentation can be a challenge to the optometrist charged with treating the sequelae of this event as multiple functional components of the visual system can be affected. Case Report: This paper describes the diagnosis and subsequent ophthalmic management of an acquired brain injury in a 22 year old male on active duty in the US Army. After developing acute neurological symptoms, the patient was diagnosed with a pilocytic astrocytoma of the cerebellum. Emergent neurosurgery to treat the neoplasm resulted in iatrogenic cranial nerve palsies and a hemispheric syndrome. Over the next 18 months, he was managed by a series of providers, including a strabismus surgeon, until presenting to our clinic. Lenses, prism, and in-office and out-of-office neurooptometric rehabilitation therapy were utilized to improve his functioning and make progress towards his goals. Conclusions: Pilocytic astrocytomas are the most common primary brain tumors, and the vast majority are benign with excellent surgical prognosis. Although the most common site is the cerebellum, the visual pathway is also frequently affected. If the eye or visual system is affected, optometrists have the ability to drastically improve quality of life with neuro-optometric rehabilitation.

La evaluación del interprofesionalismo en la educación basada en simulación

2019 ◽  
Vol 62 (0) ◽  
pp. 56-63
Author(s):  
Laura Silvia Hernández Gutiérrez ◽  
Angélica García-Gómez ◽  
Argimira Vianey Barona Nuñez ◽  
Erick López Léon
Keyword(s):  
Health Professionals ◽  
Interprofessional Education ◽  
Collaborative Work ◽  
Educational Process ◽  
Educational Institutions ◽  
Controlled Environment ◽  
Study Group ◽  
The Us ◽  
Improvement Strategies

The education based on simulation is an educationalstrategy where students learn from their errors, developing skills, knowledge, competences,etc. in a controlled environment. During the process of teaching by simulation, it is necessaryto execute various types of assessments (diagnostic, summative, formative) in order tomake adjustments or changes in the educational process of the students, therefore identifying areas of opportunity for improvement. With the simulation, different processes can be taught, like interprofessionalism and collaborative work. Nowadays, there is a major concern for added safety and the quality of care for the patients and their families. Therefore, a WHO study group determined the basic interprofessional competences, and has been given the task of disseminating and promoting interprofessional education. Some educational institutions in the US, Canada and Europe have integrated interprofessional and collaborative work in simulation practices. All the activity by simulation must be evaluated in order to provide feedback to the participants and establish improvement strategies. The assessment of the interprofessional work focuses on the evaluation of common skills and competencies among various health professionals.

Editorial peer-review for improving the quality of reports of biomedical studies

2001 ◽  
Author(s):  
TO Jefferson ◽  
P Alderson ◽  
F Davidoff ◽  
E Wager
Keyword(s):  
Peer Review ◽  
Quality Of Reports
Sign in / Sign up

Export Citation Format

Share Document