scholarly journals The Methodological Quality Score of COVID-19 Systematic Reviews is Low, Except for Cochrane Reviews: A Meta-epidemiological Study

2021 ◽  
Vol 3 (2) ◽  
pp. 46-55
Author(s):  
Yuki Kataoka ◽  
Shiho Oide ◽  
Takashi Ariie ◽  
Yasushi Tsujimoto ◽  
Toshi A. Furukawa
Keyword(s):  
Epidemiological Study ◽  
Systematic Reviews ◽  
Methodological Quality ◽  
Quality Score ◽  
Cochrane Reviews

scholarly journals The methodological quality of COVID-19 systematic reviews is low, except for Cochrane reviews: a meta-epidemiological study

2020 ◽  
Author(s):  
Yuki KATAOKA ◽  
Shiho Oide ◽  
Takashi Ariie ◽  
Yasushi Tsujimoto ◽  
Toshi A. Furukawa
Keyword(s):  
Epidemiological Study ◽  
Systematic Reviews ◽  
Methodological Quality ◽  
Mean Difference ◽  
Measurement Tool ◽  
Cross Sectional ◽  
Cochrane Reviews ◽  
Cochrane Database ◽  
The Mean

Objectives: The objective of this study was to investigate the methodological quality of COVID-19 systematic reviews (SRs) indexed in medRxiv and PubMed, compared with Cochrane COVID Reviews. Study Design and Setting: This is a cross-sectional meta-epidemiological study. We searched medRxiv, PubMed, and Cochrane Database of Systematic Reviews for SRs of COVID-19. We evaluated the methodological quality using A MeaSurement Tool to Assess systematic Reviews (AMSTAR) checklists. The maximum AMSTAR score is 11, and minimum is 0. Higher score means better quality. Results: We included 9 Cochrane reviews as well as randomly selected 100 non-Cochrane reviews in medRxiv and PubMed. Compared with Cochrane reviews (mean 9.33, standard deviation 1.32), the mean AMSTAR scores of the articles in medRxiv were lower (mean difference -2.85, 95%confidence intervals (CI): -0.96 to -4.74) and those in PubMed was also lower (mean difference -3.28, 95% CI: -1.40 to -5.15), with no difference between the latter two. Conclusions: It should be noted that AMSTAR is not a perfect tool of assessing quality SRs other than intervention. Readers should pay attention to the potentially low methodological quality of COVID-19 SRs in both PubMed and medRxiv but less so in Cochrane COVID reviews.

scholarly journals Methodological quality of meta-analyses indexed in PsycINFO: leads for enhancements: a meta-epidemiological study

BMJ Open ◽  
2020 ◽  
Vol 10 (8) ◽  
pp. e036349
Author(s):  
Victoria Leclercq ◽  
Charlotte Beaudart ◽  
Sara Ajamieh ◽  
Ezio Tirelli ◽  
Olivier Bruyère
Keyword(s):  
Epidemiological Study ◽  
Systematic Reviews ◽  
Methodological Quality ◽  
Positive Influence ◽  
Measurement Tool ◽  
Data Collection And Analysis ◽  
Literature Search Strategy ◽  
The Individual ◽  
Meta Analyses

ObjectivesMeta-analyses (MAs) are often used because they are lauded to provide robust evidence that synthesises information from multiple studies. However, the validity of MA conclusions relies on the procedural rigour applied by the authors. Therefore, this meta-research study aims to characterise the methodological quality and meta-analytic practices of MAs indexed in PsycINFO.DesignA meta-epidemiological study.ParticipantsWe evaluated a random sample of 206 MAs indexed in the PsycINFO database in 2016.Primary and secondary outcomesTwo authors independently extracted the methodological characteristics of all MAs and checked their quality according to the 16 items of the A MeaSurement Tool to Assess systematic Reviews (AMSTAR2) tool for MA critical appraisal. Moreover, we investigated the effect of mentioning Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) on the methodological quality of MAs.ResultsAccording to AMSTAR2 criteria, 95% of the 206 MAs were rated as critically low quality. Statistical methods were appropriate and publication bias was well evaluated in 87% and 70% of the MAs, respectively. However, much improvement is needed in data collection and analysis: only 11% of MAs published a research protocol, 44% had a comprehensive literature search strategy, 37% assessed and 29% interpreted the risk of bias in the individual included studies, and 11% presented a list of excluded studies. Interestingly, the explicit mentioning of PRISMA suggested a positive influence on the methodological quality of MAs.ConclusionThe methodological quality of MAs in our sample was critically low according to the AMSTAR2 criteria. Some efforts to tremendously improve the methodological quality of MAs could increase their robustness and reliability.

scholarly journals Low Methodological Quality of Systematic Reviews on Acupuncture: A Cross- Sectional Study

2021 ◽  
Author(s):  
Leonard TF Ho ◽  
Fiona YT Ke ◽  
Charlene HL Wong ◽  
Irene XY Wu ◽  
Andy KL Cheung ◽  
...  
Keyword(s):  
Systematic Reviews ◽  
Methodological Quality ◽  
Meta Analysis ◽  
A Priori ◽  
Cross Sectional Study ◽  
Sectional Study ◽  
Cross Sectional ◽  
Quality Ratings ◽  
Cochrane Reviews

Abstract Background: While well-conducted systematic reviews (SRs) can provide best evidence on the potential effectiveness of acupuncture, limitations on methodological rigour of SRs may impact trustworthiness of their conclusions. This cross-sectional study aimed to evaluate the methodological quality of a representative sample of SRs on acupuncture effectiveness.Methods: CDSR, MEDLINE, and EMBASE were searched for SRs on acupuncture. AMSTAR2 was applied for assessing methodological quality. Associations between bibliographical characteristics and methodological quality ratings were examined. Results: A total of 106 SRs were appraised. Only one (0.9%) SR was of high overall methodological quality, zero (0%) was of moderate-quality, six (5.7%) and 99 (93.4%) were of low-quality and critically low-quality respectively. Among appraised SRs, only ten (9.4%) provided an a priori protocol, four (3.8%) conducted a comprehensive literature search, five (4.7%) provided a list of excluded study, and six (5.7%) performed meta-analysis appropriately. Cochrane reviews, update reviews, reviews with corresponding authors from the America, and reviews that searched non-English databases had relatively higher overall quality. Conclusions: Methodological quality of SRs on acupuncture is unsatisfactory. Future reviewers should improve critical areas of publishing protocols, performing comprehensive search, providing a list of excluded studies with justifications for exclusion, and conducting meta-analysis appropriately.

scholarly journals A systematic assessment of Cochrane reviews and systematic reviews published in high-impact medical journals related to cancer

BMJ Open ◽  
2018 ◽  
Vol 8 (3) ◽  
pp. e020869 ◽  
Author(s):  
Marius Goldkuhle ◽  
Vikram M Narayan ◽  
Aaron Weigl ◽  
Philipp Dahm ◽  
Nicole Skoetz
Keyword(s):  
Systematic Reviews ◽  
Methodological Quality ◽  
Grey Literature ◽  
High Impact ◽  
Impact Factors ◽  
High Impact Journal ◽  
Eligibility Criteria ◽  
Medical Journals ◽  
Cochrane Reviews ◽  
Meta Analyses

ObjectiveTo compare cancer-related systematic reviews (SRs) published in the Cochrane Database of SRs (CDSR) and high-impact journals, with respect to type, content, quality and citation rates.DesignMethodological SR with assessment and comparison of SRs and meta-analyses. Two authors independently assessed methodological quality using an Assessment of Multiple Systematic Reviews (AMSTAR)-based extraction form. Both authors independently screened search results, extracted content-relevant characteristics and retrieved citation numbers of the included reviews using the Clarivate Analytics Web of Science database.Data sourcesCancer-related SRs were retrieved from the CDSR, as well as from the 10 journals which publish oncological SRs and had the highest impact factors, using a comprehensive search in both the CDSR and MEDLINE.Eligibility criteria for selecting studiesWe included all cancer-related SRs and meta-analyses published from January 2011 to May 2016. Methodological SRs were excluded.ResultsWe included 346 applicable Cochrane reviews and 215 SRs from high-impact journals. Cochrane reviews consistently met more individual AMSTAR criteria, notably with regard to an a priori design (risk ratio (RR) 3.89; 95% CI 3.10 to 4.88), inclusion of the grey literature and trial registries (RR 3.52; 95% CI 2.84 to 4.37) in their searches, and the reporting of excluded studies (RR 8.80; 95% CI 6.06 to 12.78). Cochrane reviews were less likely to address questions of prognosis (RR 0.04; 95% CI 0.02 to 0.09), use individual patient data (RR 0.03; 95% CI 0.01 to 0.09) or be based on non-randomised controlled trials (RR 0.04; 95% CI 0.02 to 0.09). Citation rates of Cochrane reviews were notably lower than those for high-impact journals (Cochrane reviews: mean number of citations 6.52 (range 0–143); high-impact journal SRs: 74.45 (0–652)).ConclusionsWhen comparing cancer-related SRs published in the CDSR versus those published in high-impact medical journals, Cochrane reviews were consistently of higher methodological quality, but cited less frequently.

scholarly journals Assessing bias in osteoarthritis trials included in Cochrane reviews: protocol for a meta-epidemiological study

BMJ Open ◽  
2014 ◽  
Vol 4 (10) ◽  
pp. e005491 ◽  
Author(s):  
Julie B Hansen ◽  
Carsten B Juhl ◽  
Isabelle Boutron ◽  
Peter Tugwell ◽  
Elizabeth A T Ghogomu ◽  
...  
Keyword(s):  
Systematic Reviews ◽  
Treatment Effect ◽  
Methodological Quality ◽  
Cochrane Review ◽  
Risk Of Bias ◽  
Primary Data ◽  
Forest Plot ◽  
Mean Values ◽  
Cochrane Reviews ◽  
Meta Analyses

IntroductionThe validity of systematic reviews and meta-analysis depends on methodological quality and unbiased dissemination of trials. Our objective is to evaluate the association of estimates of treatment effects with different bias-related study characteristics in meta-analyses of interventions used for treating pain in osteoarthritis (OA). From the findings, we hope to consolidate guidance on interpreting OA trials in systematic reviews based on empirical evidence from Cochrane reviews.Methods and analysisOnly systematic reviews that compare experimental interventions with sham, placebo or no intervention control will be considered eligible. Bias will be assessed with the risk of bias tool, used according to the Cochrane Collaboration’s recommendations. Furthermore, center status, trial size and funding will be assessed. The primary outcome (pain) will be abstracted from the first appearing forest plot for overall pain in the Cochrane review. Treatment effect sizes will be expressed as standardised mean differences (SMDs), where the difference in mean values available from the forest plots is divided by the pooled SD. To empirically assess the risk of bias in treatment benefits, we will perform stratified analyses of the trials from the included meta-analyses and assess the interaction between trial characteristics and treatment effect. A relevant study-level covariate is defined as one that decreases the between-study variance (τ2, estimated as Tau-squared) as a consequence of inclusion in the mixed effects statistical model.Ethics and disseminationMeta-analyses and randomised controlled trials provide the most reliable basis for treatment of patients with OA, but the actual impact of bias is unclear. This study will systematically examine the methodological quality in OA Cochrane reviews and explore the effect estimates behind possible bias. Since our study does not collect primary data, no formal ethical assessment and informed consent are required.Trial registration numberPROSPERO (CRD42013006924).

scholarly journals Methodological Quality Assessment of Systematic Review or Meta-Analysis Using AMSTAR-2: the Long-Term Effectiveness or Efficacy of Opioids for Chronic Non-Cancer Pain

2019 ◽  
Author(s):  
Qian Li ◽  
Ke Deng ◽  
Xiaoyuan Jiang ◽  
Huan Tao ◽  
Hui Liu ◽  
...  
Keyword(s):  
Cancer Pain ◽  
Systematic Reviews ◽  
Methodological Quality ◽  
Meta Analysis ◽  
Risk Of Bias ◽  
Cochrane Central Register ◽  
Cochrane Reviews ◽  
Meta Analyses

Abstract Background：Systematic review or meta-analysis, the strong study design of high quality evidence, give inconsistent conclusion of long-term effectiveness or efficacy of opioids for chronic non-cancer pain. We appraised the methodological quality of systematic reviews or meta-analyses. Methods: We found the relevant systematic reviews or meta-analyses by searching Medline, EMBASE, the Cochrane Database of Systematic Reviews, the Cochrane Central Register of Controlled Trials, the International prospective register of systematic reviews, Psyc ARTICLES/OVID, the Chinese Bio-Medical Literature Database, the China National Knowledge Infrastructure, and the Wan Fang Data and VIP Database on March 1st, 2019. The methodological quality was assessed by A Measurement Tool to Assess Systematic Reviews-2(AMSTAR-2). Spearman correlation analysis and non-parametric tests were used to assess the association between quality and factors. Results: Twenty-one systematic reviews or meta-analyses were included in our study. One has no individual study. In terms of methodological quality, twelve reviews were critically low in overall confidence, four reviews were low, two reviews were moderate, two reviews were high. When referring to the systematic reviews or meta-analyses of relatively better methodological quality with more credible results and conclusions, the effectiveness or efficacy of opioids was small to questionable. Cochrane reviews performed better than non-Cochrane reviews in establishing prior protocol (100% vs 17%, P<0.05), providing an excluded studies list (100% vs 50%, P<0.05) and taking risk of bias into account when interpreting the results of the review (100% vs 75%, P<0.05). There was a strong correlation (ρ=0.526, P<0.05) between the impact factor of systematic reviews or meta-analyses in published journals and methodological quality. Conclusion The methodological quality of the included systematic reviews or meta-analyses is far from satisfactory and needs improvement, especially in establishing prior protocol and justifying significant deviations from the protocol, providing an excluded primary studies list, reporting the funding information of primary studies, and assessing the potential impact of risk of bias on individual studies.

PP015 Methodological Quality Of Health Technology Assessment Reports

2017 ◽  
Vol 33 (S1) ◽  
pp. 74-75
Author(s):  
Elke Hausner ◽  
Marco Knelangen ◽  
Laura Sanders ◽  
Siw Waffenschmidt
Keyword(s):  
Health Care ◽  
Health Technology Assessment ◽  
Systematic Reviews ◽  
Technology Assessment ◽  
Methodological Quality ◽  
Health Technology ◽  
Cochrane Reviews ◽  
Literature Source ◽  
The One

INTRODUCTION:Health Technology Assessment (HTA) reports may have a major impact on the health care provided in a country. Hence, one would assume that these reports have a high methodological quality and thus represent a potentially important source of information, for instance, for identifying primary studies for inclusion in the evidence syntheses (for example, systematic reviews, Cochrane reviews, HTA reports). The aim of the present analysis is to evaluate the methodological quality of HTA reports used as a literature source for HTA reports produced by the German Institute for Quality and Efficiency in Health Care (IQWiG).METHODS:Eligible IQWiG reports were assessments of drug or non-drug interventions considering HTA reports as the literature source for primary studies and published up to October 2016. An HTA report included in the IQWIG report was considered in the analysis if it was a complete report published in English or German and indexed in the Health Technology Assessment Database (Wiley) or MEDLINE. Only the most current HTA report in an IQWiG report was considered; if more than one current HTA report was available, the one for inclusion in the analysis was randomly selected. The methodological quality of the HTA reports identified was evaluated with the AMSTAR (“Assessment of Multiple Systematic Reviews”) tool (1), which comprises 11 items on methodological quality (meaning a maximum achievable score of 11).RESULTS:A total of fifty eligible IQWiG reports using fourty-one eligible HTA reports as literature sources were identified. The mean AMSTAR score of these HTA reports was 5.3 (95 percent Confidence Interval, CI: 4.3, 6.2). None of the HTA reports achieved a score of 11, nineteen (46 percent) had a score between 6 and 10, and twenty-two had a score below 6.CONCLUSIONS:HTA reports included in IQWiG reports only have an average methodological quality.

scholarly journals Methodological quality of systematic reviews on interventions for osteoarthritis: a cross-sectional study

2020 ◽  
Vol 12 ◽  
pp. 1759720X2095996
Author(s):  
Irene XY Wu ◽  
Huan Wang ◽  
Lin Zhu ◽  
Yancong Chen ◽  
Charlene HL Wong ◽  
...  
Keyword(s):  
Systematic Reviews ◽  
Methodological Quality ◽  
Random Sequence ◽  
Meta Analysis ◽  
Healthcare Providers ◽  
Cross Sectional Study ◽  
Risk Of Bias ◽  
Sectional Study ◽  
Cross Sectional ◽  
Cochrane Reviews

Background: Healthcare providers need reliable evidence for supporting the adoption of new interventions, of which the source of evidence often originates from systematic reviews (SRs). However, little assessment on the rigor of SRs related to osteoarthritis interventions has been conducted. This cross-sectional study aimed to evaluate the methodological quality and predictors among SRs on osteoarthritis interventions. Methods: Four electronic databases (Cochrane Database of Systematic Reviews, MEDLINE, Embase, and PsycINFO) were searched, from 1 January 2008 to 10 October 2019. An SR was eligible if it focused on osteoarthritis interventions, and we performed at least one meta-analysis. Methodological quality was assessed using the validated AMSTAR 2 instrument. Multivariate regression analyses were conducted to assess predictors of methodological quality. Results: In total, 167 SRs were included. The most SRs were non-Cochrane reviews (88.6%), and 54.5% investigated non-pharmacological interventions. Only seven (4.2%) had high methodological quality. Respectively, eight (4.8%), 25 (15.0%), and 127 (76.0%) SRs had moderate, low, and critically low quality. Main methodological weaknesses were as follows: only 16.8% registered protocol a priori, 4.2% searched literature comprehensively, 25.7% included lists of excluded studies with justifications, and 30.5% assessed risk of bias appropriately by considering allocation concealment, blinding of patients and assessors, random sequence generation and selective reported outcomes. Cochrane reviews [adjusted odds ratio (AOR) 251.5, 95% confidence interval (CI) 35.5–1782.6], being updates of previous SRs (AOR 3.9, 95% CI 1.1–13.7), and SRs published after 2017 (AOR 7.7, 95% CI 2.8–21.5) were positively related to higher methodological quality. Conclusion: Despite signs of improvement in recent years, most of the SRs on osteoarthritis interventions have critically low methodological quality, especially among non-Cochrane reviews. Future SRs should be improved by conducting comprehensive literature search, justifying excluded studies, publishing a protocol, and assessing the risk of bias of included studies appropriately.

Reporting of the methodological quality of search strategies in orthodontic quantitative systematic reviews

2020 ◽  
Author(s):  
Danah AlMubarak ◽  
Nikolaos Pandis ◽  
Martyn T Cobourne ◽  
Jadbinder Seehra
Keyword(s):  
Systematic Reviews ◽  
Methodological Quality ◽  
Search Strategy ◽  
Grey Literature ◽  
Reporting Quality ◽  
Search Strategies ◽  
Cochrane Library ◽  
Eligibility Criteria ◽  
Cochrane Reviews

Summary Background This study aimed to assess the reporting of the methodological quality of search strategies undertaken in orthodontic quantitative systematic reviews (SRs) and hence their reproducibility. Materials and methods A search of a single electronic database (Medline via PubMed) was undertaken to identify interventional orthodontic SRs with meta-analysis published within a 10-year period. The Cochrane Library of Systematic Reviews was also sourced. Full articles were reviewed by two assessors against the eligibility criteria. The reporting quality of each search strategy was assessed using a previously validated checklist with a score of 1 or 2 given for each of the eight items. Cumulative totals were calculated. Guided by previous research, the authors agreed the following cut-offs to categorize the overall level of quality: 8–10 (poor), 10–12 (fair), and greater than 13 (good). Results A total of 127 SRs were analysed. The overall median quality score for the reporting of the search strategy was 14 [interquartile range (IQR): 13–15]. Cochrane SRs and those originating in Europe received higher aggregate scores, whereas no difference was evident based on Prospero registration. The continent of the corresponding author predicated the overall score. Non-Cochrane reviews achieved lower overall scores compared to Cochrane reviews (−1.0, 95% confidence interval: −1.65, −0.34, P = 0.003). The most frequently searched database was EMBASE (N = 93) and the median number of authors was 5 (IQR 4–6). Authors of 26.8% of SRs searched the grey literature. Language restrictions were applied to the search strategies of 88 (69.3%) SRs. Conclusions The reporting quality of search strategies undertaken in orthodontic SRs is at a good level but differences between Cochrane and non-Cochrane reviews currently exist. The reporting of searching of the grey literature and application of no language restrictions can be improved.

Sign in / Sign up

Export Citation Format

Share Document