Designing and Evaluating a Real-Time Automated Patient Screening System in an Emergency Department (Preprint)

2019 ◽  
Author(s):  
Yizhao Ni ◽  
Monica Bermudez ◽  
Stephanie Kennebeck ◽  
Stacey Liddy-Hicks ◽  
Judith Dexheimer
Keyword(s):  
Machine Learning ◽  
Clinical Trial ◽  
Emergency Department ◽  
Clinical Trials ◽  
Real Time ◽  
Patient Identification ◽  
Post Evaluation ◽  
Patient Screening ◽  
Automated Screening ◽  
Quantitative Assessments

BACKGROUND One critical hurdle for clinical trial recruitment is the lack of an efficient method for identifying subjects who meet eligibility criteria. Given the large volume of data documented in electronic health records (EHRs), it is labor-intensive for the staff to screen relevant information, particularly within the time frame needed. To facilitate subject identification, we developed a natural language processing (NLP) and machine learning-based system, Automated Clinical Trial Eligibility Screener© (ACTES), which analyzed structured data and unstructured narratives automatically to determine patients' suitability for clinical trial enrollment. In this study, we integrated the ACTES into clinical practice to support real-time patient screening. OBJECTIVE Our objective was to evaluate the ACTES's impact on the institutional workflow prospectively and comprehensively. We hypothesized that compared with the manual screening process, using EHR-based automated screening would improve efficiency of patient identification, streamline patient recruitment workflow, and increase enrollment in clinical trials. METHODS The ACTES was fully integrated into the clinical research coordinator (CRC) workflow in the pediatric emergency department (ED) at Cincinnati Children's Hospital Medical Center. The system continuously analyzed EHR information for current ED patients and recommended potential candidates for clinical trials. Relevant patient eligibility information was presented in real-time on a dashboard available to CRCs to facilitate their recruitment. To assess the system's effectiveness, we performed a multidimensional, prospective evaluation for a 12-month period, including a time-and-motion study, quantitative assessments of enrollment, and post-evaluation usability surveys collected from the CRCs. RESULTS Compared to manual screening, use of ACTES reduced the patient screening time by 34% (P<0.0001). The saved time was redirected to other work-related activities that streamlined teamwork among the CRCs (P <0.05). The quantitative assessments showed that automated screening improved the numbers of subjects screened, approached and enrolled by more than 10%, suggesting the potential of ACTES in streamlining recruitment workflow. The post-evaluation surveys indicated that the system was a good computerized solution with satisfactory usability. CONCLUSIONS By leveraging NLP and machine learning technologies, the ACTES demonstrated good capacity for improving efficiency of patient identification. The quantitative assessments demonstrated the potential of ACTES in streamlining recruitment workflow and improving patient approach and enrollment. The post-evaluation surveys suggested that the system was a good computerized solution with satisfactory usability.

scholarly journals A Real-Time Automated Patient Screening System for Clinical Trials Eligibility in an Emergency Department: Design and Evaluation

10.2196/14185 ◽  
2019 ◽  
Vol 7 (3) ◽  
pp. e14185 ◽  
Author(s):  
Yizhao Ni ◽  
Monica Bermudez ◽  
Stephanie Kennebeck ◽  
Stacey Liddy-Hicks ◽  
Judith Dexheimer
Keyword(s):  
Emergency Department ◽  
Clinical Trials ◽  
Real Time ◽  
Screening System ◽  
Patient Screening

scholarly journals Will they participate? Predicting patients’ response to clinical trial invitations in a pediatric emergency department

2016 ◽  
Vol 23 (4) ◽  
pp. 671-680 ◽  
Author(s):  
Yizhao Ni ◽  
Andrew F Beck ◽  
Regina Taylor ◽  
Jenna Dyas ◽  
Imre Solti ◽  
...  
Keyword(s):  
Machine Learning ◽  
Clinical Trial ◽  
Emergency Department ◽  
Clinical Trials ◽  
Roc Curve ◽  
Pediatric Emergency ◽  
Machine Learning Algorithms ◽  
Pediatric Emergency Department ◽  
Multiple Sources ◽  
F Measure

Abstract Objective (1) To develop an automated algorithm to predict a patient’s response (ie, if the patient agrees or declines) before he/she is approached for a clinical trial invitation; (2) to assess the algorithm performance and the predictors on real-world patient recruitment data for a diverse set of clinical trials in a pediatric emergency department; and (3) to identify directions for future studies in predicting patients’ participation response. Materials and Methods We collected 3345 patients’ response to trial invitations on 18 clinical trials at one center that were actively enrolling patients between January 1, 2010 and December 31, 2012. In parallel, we retrospectively extracted demographic, socioeconomic, and clinical predictors from multiple sources to represent the patients’ profiles. Leveraging machine learning methodology, the automated algorithms predicted participation response for individual patients and identified influential features associated with their decision-making. The performance was validated on the collection of actual patient response, where precision, recall, F-measure, and area under the ROC curve were assessed. Results Compared to the random response predictor that simulated the current practice, the machine learning algorithms achieved significantly better performance (Precision/Recall/F-measure/area under the ROC curve: 70.82%/92.02%/80.04%/72.78% on 10-fold cross validation and 71.52%/92.68%/80.74%/75.74% on the test set). By analyzing the significant features output by the algorithms, the study confirmed several literature findings and identified challenges that could be mitigated to optimize recruitment. Conclusion By exploiting predictive variables from multiple sources, we demonstrated that machine learning algorithms have great potential in improving the effectiveness of the recruitment process by automatically predicting patients’ participation response to trial invitations.

Shared-Task Worklists Improve Clinical Trial Recruitment Workflow in an Academic Emergency Department

2021 ◽  
Vol 12 (02) ◽  
pp. 293-300
Author(s):  
Kevin S. Naceanceno ◽  
Stacey L. House ◽  
Phillip V. Asaro
Keyword(s):  
Clinical Trial ◽  
Emergency Department ◽  
Clinical Trials ◽  
Clinical Research ◽  
Substantial Reduction ◽  
Text Message ◽  
High Volume ◽  
Text Messages ◽  
Shared Task ◽  
Clinical Research Coordinators

Abstract Background Clinical trials performed in our emergency department at Barnes-Jewish Hospital utilize a centralized infrastructure for alerting, screening, and enrollment with rule-based alerts sent to clinical research coordinators. Previously, all alerts were delivered as text messages via dedicated cellular phones. As the number of ongoing clinical trials increased, the volume of alerts grew to an unmanageable level. Therefore, we have changed our primary notification delivery method to study-specific, shared-task worklists integrated with our pre-existing web-based screening documentation system. Objective To evaluate the effects on screening and recruitment workflow of replacing text-message delivery of clinical trial alerts with study-specific shared-task worklists in a high-volume academic emergency department supporting multiple concurrent clinical trials. Methods We analyzed retrospective data on alerting, screening, and enrollment for 10 active clinical trials pre- and postimplementation of shared-task worklists. Results Notifications signaling the presence of potentially eligible subjects for clinical trials were more likely to result in a screen (p < 0.001) with the implementation of shared-task worklists compared with notifications delivered as text messages for 8/10 clinical trials. The change in workflow did not alter the likelihood of a notification resulting in an enrollment (p = 0.473). The Director of Research reported a substantial reduction in the amount of time spent redirecting clinical research coordinator screening activities. Conclusion Shared-task worklists, with the functionalities we have described, offer a viable alternative to delivery of clinical trial alerts via text message directly to clinical research coordinators recruiting for multiple concurrent clinical trials in a high-volume academic emergency department.

scholarly journals Understanding and predicting COVID-19 clinical trial completion vs. cessation

PLoS ONE ◽  
2021 ◽  
Vol 16 (7) ◽  
pp. e0253789
Author(s):  
Magdalyn E. Elkin ◽  
Xingquan Zhu
Keyword(s):  
Machine Learning ◽  
Clinical Trial ◽  
Clinical Trials ◽  
Computational Methods ◽  
Predictive Models ◽  
State Of The Art ◽  
Area Under The Curve ◽  
Information Criteria ◽  
The Other ◽  
Satisfactory Accuracy

As of March 30 2021, over 5,193 COVID-19 clinical trials have been registered through Clinicaltrial.gov. Among them, 191 trials were terminated, suspended, or withdrawn (indicating the cessation of the study). On the other hand, 909 trials have been completed (indicating the completion of the study). In this study, we propose to study underlying factors of COVID-19 trial completion vs. cessation, and design predictive models to accurately predict whether a COVID-19 trial may complete or cease in the future. We collect 4,441 COVID-19 trials from ClinicalTrial.gov to build a testbed, and design four types of features to characterize clinical trial administration, eligibility, study information, criteria, drug types, study keywords, as well as embedding features commonly used in the state-of-the-art machine learning. Our study shows that drug features and study keywords are most informative features, but all four types of features are essential for accurate trial prediction. By using predictive models, our approach achieves more than 0.87 AUC (Area Under the Curve) score and 0.81 balanced accuracy to correctly predict COVID-19 clinical trial completion vs. cessation. Our research shows that computational methods can deliver effective features to understand difference between completed vs. ceased COVID-19 trials. In addition, such models can also predict COVID-19 trial status with satisfactory accuracy, and help stakeholders better plan trials and minimize costs.

scholarly journals Novel Clinical Trial Designs to Improve the Efficiency of Research

2020 ◽  
Vol 132 (1) ◽  
pp. 69-81 ◽  
Author(s):  
Daniel I. Sessler ◽  
Paul S. Myles
Keyword(s):  
Clinical Trial ◽  
Clinical Trials ◽  
Real Time ◽  
Randomized Trials ◽  
Clinical Evidence ◽  
Adaptive Designs ◽  
Large Numbers ◽  
Clinical Trial Designs ◽  
Study Sites ◽  
Trial Efficiency

Abstract SUMMARY Large randomized trials provide the highest level of clinical evidence. However, enrolling large numbers of randomized patients across numerous study sites is expensive and often takes years. There will never be enough conventional clinical trials to address the important questions in medicine. Efficient alternatives to conventional randomized trials that preserve protections against bias and confounding are thus of considerable interest. A common feature of novel trial designs is that they are pragmatic and facilitate enrollment of large numbers of patients at modest cost. This article presents trial designs including cluster designs, real-time automated enrollment, and practitioner-preference approaches. Then various adaptive designs that improve trial efficiency are presented. And finally, the article discusses the advantages of embedding randomized trials within registries.

Accelerating clinical trial enrollment with comprehensive genomic profiling (CGP) and just-in-time clinical trial sites: An index case of a paradigm shift.

2017 ◽  
Vol 35 (15_suppl) ◽  
pp. 6539-6539
Author(s):  
Gaurav Singal ◽  
Deepu Madduri ◽  
Lara Yuan ◽  
David Luo ◽  
Aparna Upadhyay ◽  
...  
Keyword(s):  
Clinical Trial ◽  
Clinical Trials ◽  
Drug Development ◽  
Therapeutic Option ◽  
Just In Time ◽  
Trial Participation ◽  
Community Settings ◽  
Patient Identification ◽  
Access Methods ◽  
Comprehensive Genomic Profiling

6539 Background: In many instances, trials may offer the best or only therapeutic option for patients with rare findings. However, conducting clinical trials of novel therapeutics targeting rare molecular variants is challenging. Patient populations are small, distributed, and predominantly in community settings where trial access remains limited by awareness and site availability. These challenges increase costs of drug development and approval, delaying widespread patient access. Methods: Foundation Medicine deployed a trial education and access program, “Precision Enrollment,” with Ignyta (a trial sponsor) and Pharmatech (a site management organization, or SMO, enabling “Just-In-Time” clinical trials) (Wiener, JCO 2007). Infrastructure and algorithms developed at Foundation Medicine (“SmartTrials Engine”) matched sequenced patients (avg n = 800/wk) with activating NTRK, ROS1, or ALK fusions to the phase II study of Entrectinib (NCT02568267). Oncologists at Foundation Medicine, through peer-to-peer outreach, facilitated trial access by providing trial and nearest site information to treating providers of matched patients. Results: 107 treatment-eligible patients with NTRK, ROS1, or ALK fusions were matched by the SmartTrials Engine; 36 (33%) expressed interest in trial participation. One such patient with NSCLC and a CD74-ROS1 fusion was unable to participate at an open trial site due to inability to travel. The patient’s site was part of the “Just-In-Time” network, with IRB and contract pre-approval, and was activated in only 3 days. Total time from patient identification to initiation of therapy was 7 days. Conclusions: We demonstrate a novel methodology for patient matching to trials targeting rare genomic findings, including in community settings. If extended, such innovative partnerships combined with computational matching infrastructure, could improve drug development and therapeutic access.

An automated EHR-based tool to facilitate patient identification for biomarker-driven trials.

2021 ◽  
Vol 39 (15_suppl) ◽  
pp. 1539-1539
Author(s):  
Shailendra Lakhanpal ◽  
Kailee Hawkins ◽  
Steven G. Dunder ◽  
Karri Donahue ◽  
Madeline Richey ◽  
...  
Keyword(s):  
Machine Learning ◽  
Clinical Trial ◽  
Sensitivity And Specificity ◽  
High Sensitivity ◽  
Test Sample ◽  
Patient Identification ◽  
Community Oncology ◽  
Online Setting ◽  
N Gram ◽  
Sensitivity Specificity

1539 Background: Clinical trial eligibility increasingly requires information found in NGS tests; lack of structured NGS results hinders the automation of trial matching for this criterion, which may be a deterrent to open biomarker-driven trials in certain sites. We developed a machine learning tool that infers the presence of NGS results in the EHR, facilitating clinical trial matching. Methods: The Flatiron Health EHR-derived database contains patient-level pathology and genetic counseling reports from community oncology practices. An internal team of clinical experts reviewed a random sample of patients across this network to generate labels of whether each patient had been NGS tested. A supervised ML model was trained by scanning documents in the EHR and extracting n-gram features from text snippets surrounding relevant keywords (i.e. 'Lung biomarker', 'Biomarker negative'). Through k-fold cross-validation and l2-regularization, we found that a logistic regression was able to classify patients' NGS testing status. The model's offline performance on a 20% hold-out test set was measured with standard classification metrics: sensitivity, specificity, positive predictive value (PPV) and NPV. In an online setting, we integrated the tool into Flatiron's clinical trial matching software OncoTrials by including in each patient's profile an indicator of "likely NGS tested" or "unlikely NGS tested" based on the classifier's prediction. For patients inferred as tested, the model linked users to a test report view in the EHR. In this online setting, we measured sensitivity and specificity of the model after user review in two community oncology practices. Results: This NGS testing status inference model was characterized using a test sample of 15,175 patients. The model sensitivity and specificity (95%CI) were 91.3% (90.2, 92.3) and 96.2% (95.8, 96.5), respectively; PPV was 84.5% (83.2, 85.8) and NPV was 98.0% (97.7, 98.2). In the validation sample (N = 200 originated from 2 distinct care sites), users identified NGS testing status with a sensitivity of 95.2% (88.3%, 98.7%). Conclusions: This machine learning model facilitates the screening for potential patient enrollment in biomarker-driven trials by automatically surfacing patients with NGS test results at high sensitivity and specificity into a trial matching application to identify candidates. This tool could mitigate a key barrier for participation in biomarker-driven trials for community clinics.

scholarly journals Predictive modeling of clinical trial terminations using feature engineering and embedding learning

2021 ◽  
Vol 11 (1) ◽  
Author(s):  
Magdalyn E. Elkin ◽  
Xingquan Zhu
Keyword(s):  
Machine Learning ◽  
Clinical Trial ◽  
Clinical Trials ◽  
Area Under The Curve ◽  
Common Factors ◽  
Information Criteria ◽  
Feature Engineering ◽  
Exclusion Criteria ◽  
Direct Estimate ◽  
Satisfactory Prediction

AbstractIn this study, we propose to use machine learning to understand terminated clinical trials. Our goal is to answer two fundamental questions: (1) what are common factors/markers associated to terminated clinical trials? and (2) how to accurately predict whether a clinical trial may be terminated or not? The answer to the first question provides effective ways to understand characteristics of terminated trials for stakeholders to better plan their trials; and the answer to the second question can direct estimate the chance of success of a clinical trial in order to minimize costs. By using 311,260 trials to build a testbed with 68,999 samples, we use feature engineering to create 640 features, reflecting clinical trial administration, eligibility, study information, criteria etc. Using feature ranking, a handful of features, such as trial eligibility, trial inclusion/exclusion criteria, sponsor types etc., are found to be related to the clinical trial termination. By using sampling and ensemble learning, we achieve over 67% Balanced Accuracy and over 0.73 AUC (Area Under the Curve) scores to correctly predict clinical trial termination, indicating that machine learning can help achieve satisfactory prediction results for clinical trial study.

scholarly journals Machine learning enabled subgroup analysis with real-world data to inform better clinical trial design

2021 ◽  
Author(s):  
Jie Xu ◽  
Hao Zhang ◽  
Hansi Zhang ◽  
Jiang Bian ◽  
Fei Wang
Keyword(s):  
Machine Learning ◽  
Clinical Trial ◽  
Clinical Trials ◽  
Real World ◽  
Clinical Manifestations ◽  
Research Network ◽  
Real World Data ◽  
Eligibility Criteria ◽  
World Data ◽  
Patient Subgroups

Restrictive eligibility criteria for clinical trials may limit the generalizability of treatment effectiveness and safety to real-world patients. In this paper, we propose a machine learning approach to derive patient subgroups from real-world data (RWD), such that the patients within the same subgroup share similar clinical characteristics and safety outcomes. The effectiveness of our approach was validated on two existing clinical trials with the electronic health records (EHRs) from a large clinical research network. One is the donepezil trial for Alzheimer's disease (AD), and the other is the Bevacizumab trial on colon cancer (CRC). The results show that our proposed algorithm can identify patient subgroups with coherent clinical manifestations and similar risk levels of encountering severe adverse events (SAEs). We further exemplify that potential rules for describing the patient subgroups with less SAEs can be derived to inform the design of clinical trial eligibility criteria.

Sign in / Sign up

Export Citation Format

Share Document