scholarly journals Large-scale evidence generation and evaluation across a network of databases (LEGEND): assessing validity using hypertension as a case study

2020 ◽  
Vol 27 (8) ◽  
pp. 1268-1277 ◽  
Author(s):  
Martijn J Schuemie ◽  
Patrick B Ryan ◽  
Nicole Pratt ◽  
RuiJun Chen ◽  
Seng Chan You ◽  
...  
Keyword(s):  
Propensity Score ◽  
Confidence Intervals ◽  
External Validity ◽  
Large Scale ◽  
Internal Validity ◽  
Medical Decision ◽  
Observational Research ◽  
High Quality ◽  
Meta Analyses ◽  
Evidence Generation

Abstract Objectives To demonstrate the application of the Large-scale Evidence Generation and Evaluation across a Network of Databases (LEGEND) principles described in our companion article to hypertension treatments and assess internal and external validity of the generated evidence. Materials and Methods LEGEND defines a process for high-quality observational research based on 10 guiding principles. We demonstrate how this process, here implemented through large-scale propensity score modeling, negative and positive control questions, empirical calibration, and full transparency, can be applied to compare antihypertensive drug therapies. We assess internal validity through covariate balance, confidence-interval coverage, between-database heterogeneity, and transitivity of results. We assess external validity through comparison to direct meta-analyses of randomized controlled trials (RCTs). Results From 21.6 million unique antihypertensive new users, we generate 6 076 775 effect size estimates for 699 872 research questions on 12 946 treatment comparisons. Through propensity score matching, we achieve balance on all baseline patient characteristics for 75% of estimates, observe 95.7% coverage in our effect-estimate 95% confidence intervals, find high between-database consistency, and achieve transitivity in 84.8% of triplet hypotheses. Compared with meta-analyses of RCTs, our results are consistent with 28 of 30 comparisons while providing narrower confidence intervals. Conclusion We find that these LEGEND results show high internal validity and are congruent with meta-analyses of RCTs. For these reasons we believe that evidence generated by LEGEND is of high quality and can inform medical decision-making where evidence is currently lacking. Subsequent publications will explore the clinical interpretations of this evidence.

External Validity Reconsidered

2011 ◽  
Author(s):  
Diana C. Mutz
Keyword(s):  
Political Science ◽  
Real World ◽  
External Validity ◽  
Observational Studies ◽  
Internal Validity ◽  
Conventional Wisdom ◽  
Observational Research ◽  
Convincing Case

This chapter talks about the significance of generalizability. Experimentalists often go to great lengths to argue that student or other convenience samples are not problematic in terms of external validity. Likewise, a convincing case for causality is often elusive with observational research, no matter how stridently one might argue to the contrary. The conventional wisdom is that experiments are widely valued for their internal validity, and experiments lack external validity. These assumptions are so widespread as to go without question in most disciplines, particularly those emphasizing external validity, such as political science and sociology. But observational studies, such as surveys, are still supposed to be better for purposes of maximizing external validity because this method allows studying people in real world settings.

What conclusions should we draw from the data?

1995 ◽  
Vol 84 (02) ◽  
pp. 95-101 ◽  
Author(s):  
Andrew Vickers
Keyword(s):  
Cohort Studies ◽  
External Validity ◽  
Large Scale ◽  
Critical Appraisal ◽  
Internal Validity ◽  
Research Process ◽  
Clinical Value ◽  
Blind Trials

AbstractCritical appraisal of a scientific trial involves deciding on its internal validity— whether the hypothesis has been correctly accepted or rejected—and its external validity—the extent to which the trial's findings can be generalized. Discourse on homoeopathic research has focused on the former at the expense of the latter and an analysis of homoeopathic research demonstrates that it has low external validity. One solution would be to split the research process in two. Large scale, triple-blind trials could be used to determine the extent to which the action of homoeopathy may be explained by placebo. Importantly, no assessment of external validity would be made. Audit and cohort studies could then be used to examine questions usually associated with external validity, such as the conditions most suitable for treatment and the long-term clinical value of homoeopathy.

scholarly journals Recommendations for Designing, Conducting and Reporting Observational Studies in Homeopathy

Homeopathy ◽  
2020 ◽  
Vol 109 (03) ◽  
pp. 114-125
Author(s):  
Michael Teut ◽  
Harald Walach ◽  
Roja Varanasi ◽  
Raj K. Manchanda ◽  
Praveen Oberai ◽  
...  
Keyword(s):  
External Validity ◽  
Observational Studies ◽  
Internal Validity ◽  
Cross Sectional Study ◽  
Research Field ◽  
Observational Research ◽  
Special Focus ◽  
Cross Sectional ◽  
Randomised Controlled ◽  
Strobe Statement

Abstract Background Randomized placebo-controlled trials are considered to be the gold standard in clinical research and have the highest importance in the hierarchical system of evidence-based medicine. However, from the viewpoint of decision makers, due to lower external validity, practical results of efficacy research are often not in line with the huge investments made over decades. Method We conducted a narrative review. With a special focus on homeopathy, we give an overview on cohort, comparative cohort, case-control and cross-sectional study designs and explain guidelines and tools that help to improve the quality of observational studies, such as the STROBE Statement, RECORD, GRACE and ENCePP Guide. Results Within the conventional medical research field, two types of arguments have been employed in favor of observational studies. First, observational studies allow for a more generalizable and robust estimation of effects in clinical practice, and if cohorts are large enough, there is no over-estimation of effect sizes, as is often feared. We argue that observational research is needed to balance the current over-emphasis on internal validity at the expense of external validity. Thus, observational research can be considered an important research tool to describe “real-world” care settings and can assist with the design and inform the results of randomised controlled trails. Conclusions We present recommendations for designing, conducting and reporting observational studies in homeopathy and provide recommendations to complement the STROBE Statement for homeopathic observational studies.

scholarly journals Partially randomised patient preference trials as an alternative design to randomised controlled trials: systematic review and meta-analyses

BMJ Open ◽  
2019 ◽  
Vol 9 (10) ◽  
pp. e031151 ◽  
Author(s):  
Karin A Wasmann ◽  
Pieta Wijsman ◽  
Susan van Dieren ◽  
Willem Bemelman ◽  
Christianne Buskens
Keyword(s):  
Systematic Review ◽  
External Validity ◽  
Randomised Controlled Trials ◽  
Primary Outcome ◽  
Internal Validity ◽  
Controlled Trials ◽  
Randomised Controlled ◽  
Meta Analyses ◽  
Lost To Follow Up

ObjectiveRandomised controlled trials (RCT) are the gold standard to provide unbiased data. However, when patients have a treatment preference, randomisation may influence participation and outcomes (eg, external and internal validity). The aim of this study was to assess the influence of patients’ preference in RCTs by analysing partially randomised patient preference trials (RPPT); an RCT and preference cohort combined.DesignSystematic review and meta-analyses.Data sourcesMEDLINE, Embase, PsycINFO and the Cochrane Library.Eligibility criteria for selecting studiesRPPTs published between January 2005 and October 2018 reporting on allocation of patients to randomised and preference cohorts were included.Data extraction and synthesisTwo independent reviewers extracted data. The main outcomes were the difference in external validity (participation and baseline characteristics) and internal validity (lost to follow-up, crossover and the primary outcome) between the randomised and the preference cohort within each RPPT, compared in a meta-regression using a Wald test. Risk of bias was not assessed, as no quality assessment for RPPTs has yet been developed.ResultsIn total, 117 of 3734 identified articles met screening criteria and 44 were eligible (24 873 patients). The participation rate in RPPTs was >95% in 14 trials (range: 48%–100%) and the randomisation refusal rate was >50% in 26 trials (range: 19%–99%). Higher education, female, older age, race and prior experience with one treatment arm were characteristics of patients declining randomisation. The lost to follow-up and cross-over rate were significantly higher in the randomised cohort compared with the preference cohort. Following the meta-analysis, the reported primary outcomes were comparable between both cohorts of the RPPTs, mean difference 0.093 (95% CI −0.178 to 0.364, p=0.502).ConclusionsPatients’ preference led to a substantial proportion of a specific patient group refusing randomisation, while it did not influence the primary outcome within an RPPT. Therefore, RPPTs could increase external validity without compromising the internal validity compared with RCTs.PROSPERO registration numberCRD42019094438.

Development of Unit Commitment Model Considering Confidence Intervals of Photovoltaics Forecast and Analysis of a Large Scale Power System

2016 ◽  
Vol 136 (5) ◽  
pp. 484-496 ◽  
Author(s):  
Yusuke Udagawa ◽  
Kazuhiko Ogimoto ◽  
Takashi Oozeki ◽  
Hideaki Ohtake ◽  
Takashi Ikegami ◽  
...  
Keyword(s):  
Power System ◽  
Confidence Intervals ◽  
Large Scale ◽  
Unit Commitment

Identification of and Correction for Publication Bias: Comment

2019 ◽  
Author(s):  
Amanda Kvarven ◽  
Eirik Strømland ◽  
Magnus Johannesson
Keyword(s):  
Publication Bias ◽  
False Positive ◽  
Large Scale ◽  
Meta Analysis ◽  
False Positive Rate ◽  
Effect Sizes ◽  
Replication Studies ◽  
Moderate Reduction ◽  
Positive Rate ◽  
Meta Analyses

Andrews & Kasy (2019) propose an approach for adjusting effect sizes in meta-analysis for publication bias. We use the Andrews-Kasy estimator to adjust the result of 15 meta-analyses and compare the adjusted results to 15 large-scale multiple labs replication studies estimating the same effects. The pre-registered replications provide precisely estimated effect sizes, which do not suffer from publication bias. The Andrews-Kasy approach leads to a moderate reduction of the inflated effect sizes in the meta-analyses. However, the approach still overestimates effect sizes by a factor of about two or more and has an estimated false positive rate of between 57% and 100%.

scholarly journals Virtual Reality in Neurorehabilitation: An Umbrella Review of Meta-Analyses

2021 ◽  
Vol 10 (7) ◽  
pp. 1478
Author(s):  
Alexandra Voinescu ◽  
Jie Sui ◽  
Danaë Stanton Fraser
Keyword(s):  
Virtual Reality ◽  
Cerebral Palsy ◽  
Brain Injury ◽  
Upper Limb ◽  
Umbrella Review ◽  
High Quality ◽  
Quality Of Evidence ◽  
Upper Limb Function ◽  
Meta Analyses

Neurological disorders are a leading cause of death and disability worldwide. Can virtual reality (VR) based intervention, a novel technology-driven change of paradigm in rehabilitation, reduce impairments, activity limitations, and participation restrictions? This question is directly addressed here for the first time using an umbrella review that assessed the effectiveness and quality of evidence of VR interventions in the physical and cognitive rehabilitation of patients with stroke, traumatic brain injury and cerebral palsy, identified factors that can enhance rehabilitation outcomes and addressed safety concerns. Forty-one meta-analyses were included. The data synthesis found mostly low- or very low-quality evidence that supports the effectiveness of VR interventions. Only a limited number of comparisons were rated as having moderate and high quality of evidence, but overall, results highlight potential benefits of VR for improving the ambulation function of children with cerebral palsy, mobility, balance, upper limb function, and body structure/function and activity of people with stroke, and upper limb function of people with acquired brain injury. Customization of VR systems is one important factor linked with improved outcomes. Most studies do not address safety concerns, as only nine reviews reported adverse effects. The results provide critical recommendations for the design and implementation of future VR programs, trials and systematic reviews, including the need for high quality randomized controlled trials to test principles and mechanisms, in primary studies and in meta-analyses, in order to formulate evidence-based guidelines for designing VR-based rehabilitation interventions.

Tiered Sampling

2021 ◽  
Vol 15 (5) ◽  
pp. 1-52
Author(s):  
Lorenzo De Stefani ◽  
Erisa Terolli ◽  
Eli Upfal
Keyword(s):  
Large Scale ◽  
Analysis Of Algorithms ◽  
Base Layer ◽  
Single Edge ◽  
Real World Data ◽  
High Quality ◽  
Large Graphs ◽  
Massive Graphs ◽  
Variance Estimate ◽  
Low Probability

We introduce Tiered Sampling , a novel technique for estimating the count of sparse motifs in massive graphs whose edges are observed in a stream. Our technique requires only a single pass on the data and uses a memory of fixed size M , which can be magnitudes smaller than the number of edges. Our methods address the challenging task of counting sparse motifs—sub-graph patterns—that have a low probability of appearing in a sample of M edges in the graph, which is the maximum amount of data available to the algorithms in each step. To obtain an unbiased and low variance estimate of the count, we partition the available memory into tiers (layers) of reservoir samples. While the base layer is a standard reservoir sample of edges, other layers are reservoir samples of sub-structures of the desired motif. By storing more frequent sub-structures of the motif, we increase the probability of detecting an occurrence of the sparse motif we are counting, thus decreasing the variance and error of the estimate. While we focus on the designing and analysis of algorithms for counting 4-cliques, we present a method which allows generalizing Tiered Sampling to obtain high-quality estimates for the number of occurrence of any sub-graph of interest, while reducing the analysis effort due to specific properties of the pattern of interest. We present a complete analytical analysis and extensive experimental evaluation of our proposed method using both synthetic and real-world data. Our results demonstrate the advantage of our method in obtaining high-quality approximations for the number of 4 and 5-cliques for large graphs using a very limited amount of memory, significantly outperforming the single edge sample approach for counting sparse motifs in large scale graphs.

Sign in / Sign up

Export Citation Format

Share Document