scholarly journals Analysis of >30,000 abstracts suggests higher false discovery rates for oncology journals, especially those with low impact factors

2018 ◽  
Author(s):  
LM Hall ◽  
AE Hendricks
Keyword(s):  
Open Access ◽  
Impact Factor ◽  
False Discovery Rate ◽  
Journal Impact Factor ◽  
General Medicine ◽  
False Discovery Rates ◽  
False Discovery ◽  
Published Research ◽  
Journal Impact ◽  
Discovery Rates

AbstractBackgroundRecently, there has been increasing concern about the replicability, or lack thereof, of published research. An especially high rate of false discoveries has been reported in some areas motivating the creation of resource-intensive collaborations to estimate the replication rate of published research by repeating a large number of studies. The substantial amount of resources required by these replication projects limits the number of studies that can be repeated and consequently the generalizability of the findings.Methods and findingsIn 2013, Jager and Leek developed a method to estimate the empirical false discovery rate from journal abstracts and applied their method to five high profile journals. Here, we use the relative efficiency of Jager and Leek’s method to gather p-values from over 30,000 abstracts and to subsequently estimate the false discovery rate for 94 journals over a five-year time span. We model the empirical false discovery rate by journal subject area (cancer or general medicine), impact factor, and Open Access status. We find that the empirical false discovery rate is higher for cancer vs. general medicine journals (p = 5.14E-6). Within cancer journals, we find that this relationship is further modified by journal impact factor where a lower journal impact factor is associated with a higher empirical false discovery rates (p = 0.012, 95% CI: -0.010, -0.001). We find no significant differences, on average, in the false discovery rate for Open Access vs closed access journals (p = 0.256, 95% CI: -0.014, 0.051).ConclusionsWe find evidence of a higher false discovery rate in cancer journals compared to general medicine journals, especially those with a lower journal impact factor. For cancer journals, a lower journal impact factor of one point is associated with a 0.006 increase in the empirical false discovery rate, on average. For a false discovery rate of 0.05, this would result in over a 10% increase to 0.056. Conversely, we find no significant evidence of a higher false discovery rate, on average, for Open Access vs. closed access journals from InCites. Our results provide identify areas of research that may need of additional scrutiny and support to facilitate replicable science. Given our publicly available R code and data, others can complete a broad assessment of the empirical false discovery rate across other subject areas and characteristics of published research.

scholarly journals High-throughput analysis suggests differences in journal false discovery rate by subject area and impact factor but not open access status

2020 ◽  
Vol 21 (1) ◽  
Author(s):  
L. M. Hall ◽  
A. E. Hendricks
Keyword(s):  
Open Access ◽  
Impact Factor ◽  
False Discovery Rate ◽  
Journal Impact Factor ◽  
General Medicine ◽  
Subject Area ◽  
Replication Rate ◽  
High Throughput Analysis ◽  
False Discovery ◽  
Journal Impact

Abstract Background A low replication rate has been reported in some scientific areas motivating the creation of resource intensive collaborations to estimate the replication rate by repeating individual studies. The substantial resources required by these projects limits the number of studies that can be repeated and consequently the generalizability of the findings. We extend the use of a method from Jager and Leek to estimate the false discovery rate for 94 journals over a 5-year period using p values from over 30,000 abstracts enabling the study of how the false discovery rate varies by journal characteristics. Results We find that the empirical false discovery rate is higher for cancer versus general medicine journals (p = 9.801E−07, 95% CI: 0.045, 0.097; adjusted mean false discovery rate cancer = 0.264 vs. general medicine = 0.194). We also find that false discovery rate is negatively associated with log journal impact factor. A two-fold decrease in journal impact factor is associated with an average increase of 0.020 in FDR (p = 2.545E−04). Conversely, we find no statistically significant evidence of a higher false discovery rate, on average, for Open Access versus closed access journals (p = 0.320, 95% CI − 0.015, 0.046, adjusted mean false discovery rate Open Access = 0.241 vs. closed access = 0.225). Conclusions Our results identify areas of research that may need additional scrutiny and support to facilitate replicable science. Given our publicly available R code and data, others can complete a broad assessment of the empirical false discovery rate across other subject areas and characteristics of published research.

scholarly journals High-throughput analysis suggests differences in journal false discovery rate by subject area and impact factor but not open access status

2020 ◽  
Author(s):  
Lauren Hall ◽  
Audrey E Hendricks
Keyword(s):  
Open Access ◽  
False Discovery Rate ◽  
General Medicine ◽  
Subject Area ◽  
Impact Factors ◽  
Replication Rate ◽  
High Throughput Analysis ◽  
False Discovery ◽  
Subject Areas ◽  
Published Research

Abstract Background: A low replication rate has been reported in some scientific areas motivating the creation of resource-intensive collaborations to estimate the replication rate by repeating individual studies. The substantial resources required by these projects limits the number of studies that can be repeated and consequently the generalizability of the findings. We extend the use of a method from Jager and Leek to estimate the false discovery rate (FDR) for 94 journals over a five-year period using p-values from 30,000 abstracts enabling the study of how the FDR varies by journal characteristics. Results: We find that the empirical FDR is higher for cancer vs. general medicine journals (p = 5.14E-6), especially for those with lower journal impact factors. Conversely, we find no significant evidence of a higher FDR, on average, for Open Access vs. closed access journals (p = 0.256, 95% CI: -0.014, 0.051). Conclusions: Our results identify areas of research that may need additional scrutiny and support to facilitate replicable science. Given our publicly available R code and data, others can complete a broad assessment of the empirical FDR across other subject areas and characteristics of published research.

Replication of Candidate Gene Single Nucleotide Polymorphisms (SNPs) Previously Reported As Associated with Venous Thromboembolism (VTE)

Blood ◽  
2011 ◽  
Vol 118 (21) ◽  
pp. 1238-1238
Author(s):  
Anita D'Souza ◽  
Sebastian M. Armasu ◽  
Mariza de Andrade ◽  
John A. Heit
Keyword(s):  
Candidate Gene ◽  
False Discovery Rate ◽  
Population Stratification ◽  
Multiple Comparisons ◽  
Blood Type ◽  
European Ancestry ◽  
Q Value ◽  
False Discovery Rates ◽  
False Discovery ◽  
Discovery Rates

Abstract Abstract 1238 Background: SNPs within genes encoding factor XI (F11), fibrinogen genes (FGA, FGG) and other candidate genes within the procoagulant, anticoagulant, fibrinolytic, innate immunity and endocrine pathways have been reported as associated with VTE. However, the independent risk of VTE associated with many of these SNPs after controlling for factor V Leiden, Prothrombin G20210A and ABO blood group non-O carrier status is uncertain. Objective: To replicate candidate gene SNPs previously reported as associated with VTE. Methods: As part of a large replication study, we included 17 SNPs previously reported as associated with VTE in a custom Illumina Golden gate (total n=1093 SNPs) genotyping array. We genotyped 1270 non-Hispanic adults of European ancestry with objectively-diagnosed VTE (cases; no cancer, venous catheter or antiphospholipid antibodies) and 1302 controls (frequency-matched on case age, gender, race, MI/stroke status). Genotyping results from high-quality control DNA (SNP call rate ≥ 95%) was used to generate a cluster algorithm. The primary outcome was VTE status, a binary measure. The covariates were age at interview or blood sample collection, sex, stroke and/or MI status, and state of residence. To adjust for population stratification, we performed the multidimensional scaling (MDS) analysis option in PLINK v 1.07 to identify outliers in our population using the ancestry informative markers. We tested for an association between each SNP and VTE using unconditional logistic regression, adjusting for age, sex, stroke/MI status, state of residence and ABO rs514659 (in high linkage disequilibrium with non-O blood type). The analyses were corrected for multiple comparisons using an extension of false discovery rates. The false discovery rate (reported as a Q-value) is an analogue measure of the p-value that takes into account the number of statistical tests and estimates the expected proportion of false positive tests incurred when a particular SNP is significant. All analyses were performed using PLINK v 1.07. Results: MDS gave no evidence of population stratification. Genotyping was unsuccessful for two of the 17 SNPs. We found significant associations between VTE and SNPs in F11, FGG, TC2D and FGA (Table). However, the false discovery rates for all significant SNPs except F11 rs3756008 were >0.05, suggesting that the observed associations were likely falsely positive due to multiple comparisons. Even at a false discovery rate of Q-value=0.0099, one would expect ∼13 SNPs (0.0099 × 1302 SNPs) to be falsely associated with VTE due to multiple comparisons. Consequently, even our observed association between F11 rs3756008 and VTE remains tentative. Conclusions: We were unable to replicate reported associations between 15 SNPs and VTE. Our results emphasize the necessity of replication studies in different populations to confirm reported associations of SNPs with VTE. Disclosures: Heit: Daiichi Sankyo: Consultancy, Honoraria.

scholarly journals “Worth(less) papers” – are journal impact factor and number of citations suitable indicators to evaluate quality of scientists?

2018 ◽  
Vol XVI (2) ◽  
pp. 369-388 ◽  
Author(s):  
Aleksandar Racz ◽  
Suzana Marković
Keyword(s):  
Open Access ◽  
Impact Factor ◽  
Journal Impact Factor ◽  
Academic Publishing ◽  
Digital Repository ◽  
Open Access Journals ◽  
On Line ◽  
Number Of Citations ◽  
Journal Impact

Technology driven changings with consecutive increase in the on-line availability and accessibility of journals and papers rapidly changes patterns of academic communication and publishing. The dissemination of important research findings through the academic and scientific community begins with publication in peer-reviewed journals. Aim of this article is to identify, critically evaluate and integrate the findings of relevant, high-quality individual studies addressing the trends of enhancement of visibility and accessibility of academic publishing in digital era. The number of citations a paper receives is often used as a measure of its impact and by extension, of its quality. Many aberrations of the citation practices have been reported in the attempt to increase impact of someone’s paper through manipulation with self-citation, inter-citation and citation cartels. Authors revenues to legally extend visibility, awareness and accessibility of their research outputs with uprising in citation and amplifying measurable personal scientist impact has strongly been enhanced by on line communication tools like networking (LinkedIn, Research Gate, Academia.edu, Google Scholar), sharing (Facebook, Blogs, Twitter, Google Plus) media sharing (Slide Share), data sharing (Dryad Digital Repository, Mendeley database, PubMed, PubChem), code sharing, impact tracking. Publishing in Open Access journals. Many studies and review articles in last decade have examined whether open access articles receive more citations than equivalent subscription toll access) articles and most of them lead to conclusion that there might be high probability that open access articles have the open access citation advantage over generally equivalent payfor-access articles in many, if not most disciplines. But it is still questionable are those never cited papers indeed “Worth(less) papers” and should journal impact factor and number of citations be considered as only suitable indicators to evaluate quality of scientists? “Publish or perish” phrase usually used to describe the pressure in academia to rapidly and continually publish academic work to sustain or further one’s career can now in 21. Century be reformulate into “Publish, be cited and maybe will not Perish”.

scholarly journals Modeling and Prediction of the Impact Factor of Journals Using Open-Access Databases

2020 ◽  
Vol 49 (5) ◽  
pp. 35-58
Author(s):  
Matthias Templ
Keyword(s):  
Open Access ◽  
Impact Factor ◽  
Journal Impact Factor ◽  
Web Of Science ◽  
Functional Relationships ◽  
Modeling And Prediction ◽  
Novel Approach ◽  
The Impact ◽  
Journal Impact ◽  
The Web

This article is motivated by the work as editor-in-chief of the Austrian Journal of Statistics and contains detailed analyses about the impact of the Austrian Journal of Statistics. The impact of a journal is typically expressed by journal metrics indicators. One of the important ones, the journal impact factor is calculated from the Web of Science (WoS) database by Clarivate Analytics. It is known that newly established journals or journals without membership in big publishers often face difficulties to be included, e.g., in the Science Citation Index (SCI) and thus they do not receive a WoS journal impact factor, as it is the case for example, for the Austrian Journal of Statistics. In this study, a novel approach is pursued modeling and predicting the WoS impact factor of journals using open access or partly open-access databases, like Google Scholar, ResearchGate, and Scopus. I hypothesize a functional linear dependency between citation counts in these databases and the journal impact factor. These functional relationships enable the development of a model that may allow estimating the impact factor for new, small, and independent journals not listed in SCI. However, only good results could be achieved with robust linear regression and well-chosen models. In addition, this study demonstrates that the WoS impact factor of SCI listed journals can be successfully estimated without using the Web of Science database and therefore the dependency of researchers and institutions to this popular database can be minimized. These results suggest that the statistical model developed here can be well applied to predict the WoS impact factor using alternative open-access databases. 

scholarly journals Metrics: Measuring importance

2012 ◽  
Vol 34 (1) ◽  
pp. 38-41
Author(s):  
Caroline Black
Keyword(s):  
Impact Factor ◽  
Journal Impact Factor ◽  
Academic Literature ◽  
Research Article ◽  
The Impact ◽  
Published Research ◽  
Journal Impact ◽  
Quality Of Research

Bibliometrics is the term used to describe various approaches to analysing measures of the use of academic literature, in particular articles in peer-reviewed journals. More broadly, the topic addresses the validity or otherwise of these measures as indicators of the impact, influence or value of the research being reported. These measures, and in particular the journal Impact Factor, are used as evidence for the quality of research, to make decisions about appointments, to judge a journal editor's success, and (it is assumed) to make funding decisions. Until recently, bibliometrics was mainly about citations, but now it is increasingly common to measure online usage, and even tweets, blogging and user star-ratings when assessing the contribution of a published research article.

scholarly journals Lag times in the publication of network meta-analyses: a survey

BMJ Open ◽  
2021 ◽  
Vol 11 (9) ◽  
pp. e048581
Author(s):  
Fernanda S Tonin ◽  
Ariane G Araujo ◽  
Mariana M Fachi ◽  
Vinicius L Ferreira ◽  
Roberto Pontarolo ◽  
...  
Keyword(s):  
Open Access ◽  
Impact Factor ◽  
Online Publication ◽  
Journal Impact Factor ◽  
Secondary Outcome ◽  
Allocation Process ◽  
Lag Times ◽  
Meta Analyses ◽  
Drug Interventions ◽  
Journal Impact

ObjectiveWe assessed the extent of lag times in the publication and indexing of network meta-analyses (NMAs).Study designThis was a survey of published NMAs on drug interventions.SettingNMAs indexed in PubMed (searches updated in May 2020).Primary and secondary outcome measuresLag times were measured as the time between the last systematic search and the article submission, acceptance, online publication, indexing and Medical Subject Headings (MeSH) allocation dates. Time-to-event analyses were performed considering independent variables (geographical origin, Journal Impact Factor, Scopus CiteScore, open access status) (SPSS V.24, R/RStudio).ResultsWe included 1245 NMAs. The median time from last search to article submission was 6.8 months (204 days (IQR 95–381)), and to publication was 11.6 months. Only 5% of authors updated their search after first submission. There is a very slightly decreasing historical trend of acceptance (rho=−0.087; p=0.010), online publication (rho=−0.080; p=0.008) and indexing (rho=−0.080; p=0.007) lag times. Journal Impact Factor influenced the MeSH allocation process, but not the other lag times. The comparison between open access versus subscription journals confirmed meaningless differences in acceptance, online publication and indexing lag times.ConclusionEfforts by authors to update their search before submission are needed to reduce evidence production time. Peer reviewers and editors should ensure authors’ compliance with NMA standards. The accuracy of these findings depends on the accuracy of the metadata used; as we evaluated only NMA on drug interventions, results may not be generalisable to all types of studies.

scholarly journals ANNALS OF GEOPHYSICS: AD MAJORA

2014 ◽  
Vol 57 (1) ◽  
Author(s):  
Fabio Florindo ◽  
Francesca Bianco ◽  
Paola De Michelis ◽  
Simona Masina ◽  
Giovanni Muscari ◽  
...  
Keyword(s):  
Open Access ◽  
Impact Factor ◽  
Solid Earth ◽  
Journal Impact Factor ◽  
Volume Number ◽  
Scientific Papers ◽  
Science Community ◽  
Journal Impact

<p>Annals of Geophysics is a bimonthly international journal, which publishes scientific papers in the field of geophysics sensu lato. It derives from Annali di Geofisica, which commenced publication in January 1948 as a quarterly periodical devoted to general geophysics, seismology, earth magnetism, and atmospheric studies. The journal was published regularly for a quarter of a century until 1982 when it merged with the French journal Annales de Géophysique to become Annales Geophysicae under the aegis of the European Geophysical Society. In 1981, this journal ceased publication of the section on solid earth geophysics, ending the legacy of Annali di Geofisica. In 1993, the Istituto Nazionale di Geofisica (ING), founder of the journal, decided to resume publication of its own journal under the same name, Annali di Geofisica. To ensure continuity, the first volume of the new series was assigned the volume number XXXVI (following the last issue published in 1982). In 2002, with volume XLV, the name of the journal was translated into English to become Annals of Geophysics and in consequence the journal impact factor counter was restarted. Starting in 2010, in order to improve its status and better serve the science community, Annals of Geophysics has instituted a number of editorial changes including full electronic open access, freely accessible online, the possibility to comment on and discuss papers online, and a board of editors representing Asia and the Americas as well as Europe. [...]</p>

scholarly journals FDRestimation: Flexible False Discovery Rate Computation in R

F1000Research ◽  
2021 ◽  
Vol 10 ◽  
pp. 441
Author(s):  
Megan H. Murray ◽  
Jeffrey D. Blume
Keyword(s):  
False Discovery Rate ◽  
Estimation Procedure ◽  
False Discovery Rates ◽  
P Values ◽  
False Discovery ◽  
Software Packages ◽  
Broad Array ◽  
Potential Impact ◽  
User Friendly ◽  
Discovery Rates

False discovery rates (FDR) are an essential component of statistical inference, representing the propensity for an observed result to be mistaken. FDR estimates should accompany observed results to help the user contextualize the relevance and potential impact of findings. This paper introduces a new user-friendly R pack-age for estimating FDRs and computing adjusted p-values for FDR control. The roles of these two quantities are often confused in practice and some software packages even report the adjusted p-values as the estimated FDRs. A key contribution of this package is that it distinguishes between these two quantities while also offering a broad array of refined algorithms for estimating them. For example, included are newly augmented methods for estimating the null proportion of findings - an important part of the FDR estimation procedure. The package is broad, encompassing a variety of adjustment methods for FDR estimation and FDR control, and includes plotting functions for easy display of results. Through extensive illustrations, we strongly encourage wider reporting of false discovery rates for observed findings.

Sign in / Sign up

Export Citation Format

Share Document