scholarly journals Machine learning for identifying Randomized Controlled Trials: An evaluation and practitioner's guide

2018 ◽  
Vol 9 (4) ◽  
pp. 602-614 ◽  
Author(s):  
Iain J. Marshall ◽  
Anna Noel-Storr ◽  
Joël Kuiper ◽  
James Thomas ◽  
Byron C. Wallace
Keyword(s):  
Machine Learning ◽  
Randomized Controlled Trials ◽  
Controlled Trials ◽  
Randomized Controlled

scholarly journals Deep Neural Network for Reducing the Screening Workload in Systematic Reviews for Clinical Guidelines: Algorithm Validation Study

10.2196/22422 ◽  
2020 ◽  
Vol 22 (12) ◽  
pp. e22422
Author(s):  
Tomohide Yamada ◽  
Daisuke Yoneoka ◽  
Yuta Hiraike ◽  
Kimihiro Hino ◽  
Hiroyoshi Toyoshiba ◽  
...  
Keyword(s):  
Machine Learning ◽  
Randomized Controlled Trials ◽  
Systematic Reviews ◽  
Clinical Guidelines ◽  
Learning System ◽  
Controlled Trials ◽  
Data Set ◽  
Randomized Controlled ◽  
Fold Reduction ◽  
Meta Analyses

Background Performing systematic reviews is a time-consuming and resource-intensive process. Objective We investigated whether a machine learning system could perform systematic reviews more efficiently. Methods All systematic reviews and meta-analyses of interventional randomized controlled trials cited in recent clinical guidelines from the American Diabetes Association, American College of Cardiology, American Heart Association (2 guidelines), and American Stroke Association were assessed. After reproducing the primary screening data set according to the published search strategy of each, we extracted correct articles (those actually reviewed) and incorrect articles (those not reviewed) from the data set. These 2 sets of articles were used to train a neural network–based artificial intelligence engine (Concept Encoder, Fronteo Inc). The primary endpoint was work saved over sampling at 95% recall (WSS@95%). Results Among 145 candidate reviews of randomized controlled trials, 8 reviews fulfilled the inclusion criteria. For these 8 reviews, the machine learning system significantly reduced the literature screening workload by at least 6-fold versus that of manual screening based on WSS@95%. When machine learning was initiated using 2 correct articles that were randomly selected by a researcher, a 10-fold reduction in workload was achieved versus that of manual screening based on the WSS@95% value, with high sensitivity for eligible studies. The area under the receiver operating characteristic curve increased dramatically every time the algorithm learned a correct article. Conclusions Concept Encoder achieved a 10-fold reduction of the screening workload for systematic review after learning from 2 randomly selected studies on the target topic. However, few meta-analyses of randomized controlled trials were included. Concept Encoder could facilitate the acquisition of evidence for clinical guidelines.

scholarly journals Identifying reports of randomized controlled trials (RCTs) via a hybrid machine learning and crowdsourcing approach

2017 ◽  
Vol 24 (6) ◽  
pp. 1165-1168 ◽  
Author(s):  
Byron C Wallace ◽  
Anna Noel-Storr ◽  
Iain J Marshall ◽  
Aaron M Cohen ◽  
Neil R Smalheiser ◽  
...  
Keyword(s):  
Machine Learning ◽  
Randomized Controlled Trials ◽  
Hybrid Approach ◽  
Controlled Trials ◽  
Randomized Controlled ◽  
Simple Strategy ◽  
Highly Sensitive ◽  
Hybrid Machine ◽  
Identification Strategy

Abstract Objectives Identifying all published reports of randomized controlled trials (RCTs) is an important aim, but it requires extensive manual effort to separate RCTs from non-RCTs, even using current machine learning (ML) approaches. We aimed to make this process more efficient via a hybrid approach using both crowdsourcing and ML. Methods We trained a classifier to discriminate between citations that describe RCTs and those that do not. We then adopted a simple strategy of automatically excluding citations deemed very unlikely to be RCTs by the classifier and deferring to crowdworkers otherwise. Results Combining ML and crowdsourcing provides a highly sensitive RCT identification strategy (our estimates suggest 95%–99% recall) with substantially less effort (we observed a reduction of around 60%–80%) than relying on manual screening alone. Conclusions Hybrid crowd-ML strategies warrant further exploration for biomedical curation/annotation tasks.

scholarly journals Deep Neural Network for Reducing the Screening Workload in Systematic Reviews for Clinical Guidelines: Algorithm Validation Study (Preprint)

2020 ◽  
Author(s):  
Tomohide Yamada ◽  
Daisuke Yoneoka ◽  
Yuta Hiraike ◽  
Kimihiro Hino ◽  
Hiroyoshi Toyoshiba ◽  
...  
Keyword(s):  
Machine Learning ◽  
Randomized Controlled Trials ◽  
Systematic Reviews ◽  
Clinical Guidelines ◽  
Learning System ◽  
Controlled Trials ◽  
Data Set ◽  
Randomized Controlled ◽  
Fold Reduction ◽  
Meta Analyses

BACKGROUND Performing systematic reviews is a time-consuming and resource-intensive process. OBJECTIVE We investigated whether a machine learning system could perform systematic reviews more efficiently. METHODS All systematic reviews and meta-analyses of interventional randomized controlled trials cited in recent clinical guidelines from the American Diabetes Association, American College of Cardiology, American Heart Association (2 guidelines), and American Stroke Association were assessed. After reproducing the primary screening data set according to the published search strategy of each, we extracted correct articles (those actually reviewed) and incorrect articles (those not reviewed) from the data set. These 2 sets of articles were used to train a neural network–based artificial intelligence engine (Concept Encoder, Fronteo Inc). The primary endpoint was work saved over sampling at 95% recall (WSS@95%). RESULTS Among 145 candidate reviews of randomized controlled trials, 8 reviews fulfilled the inclusion criteria. For these 8 reviews, the machine learning system significantly reduced the literature screening workload by at least 6-fold versus that of manual screening based on WSS@95%. When machine learning was initiated using 2 correct articles that were randomly selected by a researcher, a 10-fold reduction in workload was achieved versus that of manual screening based on the WSS@95% value, with high sensitivity for eligible studies. The area under the receiver operating characteristic curve increased dramatically every time the algorithm learned a correct article. CONCLUSIONS Concept Encoder achieved a 10-fold reduction of the screening workload for systematic review after learning from 2 randomly selected studies on the target topic. However, few meta-analyses of randomized controlled trials were included. Concept Encoder could facilitate the acquisition of evidence for clinical guidelines.

Rethinking the Gold Standard With Multi-armed Bandits: Machine Learning Allocation Algorithms for Experiments

2019 ◽  
Vol 24 (1) ◽  
pp. 78-103
Author(s):  
Chris Kaibel ◽  
Torsten Biemann
Keyword(s):  
Machine Learning ◽  
Randomized Controlled Trials ◽  
Machine Learning Algorithms ◽  
Controlled Trials ◽  
Randomized Controlled ◽  
Treatment Conditions ◽  
Scientific Disciplines ◽  
New Information ◽  
Methodological Approaches ◽  
Allocation Algorithms

In experiments, researchers commonly allocate subjects randomly and equally to the different treatment conditions before the experiment starts. While this approach is intuitive, it means that new information gathered during the experiment is not utilized until after the experiment has ended. Based on methodological approaches from other scientific disciplines such as computer science and medicine, we suggest machine learning algorithms for subject allocation in experiments. Specifically, we discuss a Bayesian multi-armed bandit algorithm for randomized controlled trials and use Monte Carlo simulations to compare its efficiency with randomized controlled trials that have a fixed and balanced subject allocation. Our findings indicate that a randomized allocation based on Bayesian multi-armed bandits is more efficient and ethical in most settings. We develop recommendations for researchers and discuss the limitations of our approach.

scholarly journals A machine learning method for subgroup analysis of randomized controlled trials

2018 ◽  
Author(s):  
Ljubomir Buturović
Keyword(s):  
Machine Learning ◽  
Randomized Controlled Trials ◽  
Controlled Trials ◽  
Machine Learning Method ◽  
Learning Method ◽  
Randomized Controlled ◽  
Subgroup Analyses ◽  
Treatment Of Hypertension ◽  
Multiple Predictor ◽  
General Method

AbstractWe developed a machine learning method for subgroup analyses of randomized controlled trials (RCT), and applied it to the results of the SPRINT RCT for treatment of hypertension. To date, the subgroup analyses mostly focused on detecting associations between certain factors and outcome, in the hope that the results will point out biologically (for example, carriers of a certain mutation) or clinically (for example, smokers) distinct subgroups with different outcomes. This seldom worked in the sense of re-launching the intervention for the detected subgroup only and successfully treating it. In contrast, we propose an empirical and general method to develop a predictive multivariate classifier using the RCT outcomes and baseline data. The classifier identifies patients likely to benefit from the intervention, is not limited to a single factor of interest, and is ready for validation in a subsequent pivotal trial. We believe this approach has a better chance of succeeding in identifying the relevant subgroups because of increased accuracy made possible by the use of multiple predictor variables, and opportunity to use advanced machine learning. The method effectiveness is demonstrated by the analysis of the SPRINT trial.

Sign in / Sign up

Export Citation Format

Share Document