scholarly journals A/B testing using rapid randomized controlled trials to optimize clinical decision support (Preprint)

2019 ◽  
Author(s):  
Devin Mann ◽  
Adam Szerencsy ◽  
Leora Horwitz ◽  
Simon Jones ◽  
Masha Kuznetsova ◽  
...  
Keyword(s):  
Decision Support ◽  
Randomized Controlled Trials ◽  
Clinical Decision Support ◽  
Positive Impact ◽  
Clinical Decision ◽  
Controlled Trials ◽  
Alert Fatigue ◽  
Randomized Controlled ◽  
Registration Number ◽  
The Impact

BACKGROUND Clinical decision support (CDS) is a valuable feature of electronic health records (EHRs) designed to improve quality and safety. However, due to the complexities of system design and inconsistent results, CDS tools may inadvertently increase alert fatigue and contribute to physician burnout. A/B testing, or rapid-cycle randomized tests, is a useful method that can be applied to the EHR in order to understand and iteratively improve design choices embedded within CDS tools. OBJECTIVE This paper describes how rapid randomized controlled trials (RCTs) embedded within EHRs can be used to quickly ascertain the superiority of potential CDS tools to improve their usability, reduce alert fatigue and promote quality of care. METHODS A multi-step process combining tools from user-centered design, A/B testing and implementation science is used to understand, ideate, prototype, test, analyze and improve each candidate CDS. CDS engagement metrics (alert views, ignores, orders) are used to evaluate which CDS version is superior. RESULTS Two experiments are highlighted to demonstrate the impact of the process. First, after multiple rounds of usability testing, a revised CDS influenza alert was tested against usual care in a rapid RCT. The new alert text resulted in minimal impact but the failure triggered another round of testing that identified key issues and led to a 70% reduction in alert volume in the next round. In the second experiment, the process was used to test three versions (financial, quality, regulatory) of text supporting tobacco cessation alerts as well as three supporting images. Three rounds of RCTs showed that the financial framing was 5-10% more effective than the other two but that adding images did not have a positive impact. CONCLUSIONS These data support the potential for this new process to rapidly develop, deploy and improve CDS within an EHR. This approach may be an important tool for improving the impact and experience of CDS. CLINICALTRIAL Our flu alert trial was registered in January 2018 with ClinicalTrials.gov, registration number NCT03415425. Our tobacco alert trial was registered in October 2018 with ClinicalTrials.gov, registration number NCT03714191.

scholarly journals Applying A/B Testing to Clinical Decision Support: Rapid Randomized Controlled Trials

10.2196/16651 ◽  
2021 ◽  
Vol 23 (4) ◽  
pp. e16651
Author(s):  
Jonathan Austrian ◽  
Felicia Mendoza ◽  
Adam Szerencsy ◽  
Lucille Fenelon ◽  
Leora I Horwitz ◽  
...  
Keyword(s):  
Decision Support ◽  
Randomized Controlled Trials ◽  
Clinical Decision Support ◽  
Clinical Decision ◽  
Controlled Trials ◽  
Alert Fatigue ◽  
Randomized Controlled ◽  
Significant Difference ◽  
Acceptance Rates ◽  
The Impact

Background Clinical decision support (CDS) is a valuable feature of electronic health records (EHRs) designed to improve quality and safety. However, due to the complexities of system design and inconsistent results, CDS tools may inadvertently increase alert fatigue and contribute to physician burnout. A/B testing, or rapid-cycle randomized tests, is a useful method that can be applied to the EHR in order to rapidly understand and iteratively improve design choices embedded within CDS tools. Objective This paper describes how rapid randomized controlled trials (RCTs) embedded within EHRs can be used to quickly ascertain the superiority of potential CDS design changes to improve their usability, reduce alert fatigue, and promote quality of care. Methods A multistep process combining tools from user-centered design, A/B testing, and implementation science was used to understand, ideate, prototype, test, analyze, and improve each candidate CDS. CDS engagement metrics (alert views, acceptance rates) were used to evaluate which CDS version is superior. Results To demonstrate the impact of the process, 2 experiments are highlighted. First, after multiple rounds of usability testing, a revised CDS influenza alert was tested against usual care CDS in a rapid (~6 weeks) RCT. The new alert text resulted in minimal impact on reducing firings per patients per day, but this failure triggered another round of review that identified key technical improvements (ie, removal of dismissal button and firings in procedural areas) that led to a dramatic decrease in firings per patient per day (23.1 to 7.3). In the second experiment, the process was used to test 3 versions (financial, quality, regulatory) of text supporting tobacco cessation alerts as well as 3 supporting images. Based on 3 rounds of RCTs, there was no significant difference in acceptance rates based on the framing of the messages or addition of images. Conclusions These experiments support the potential for this new process to rapidly develop, deploy, and rigorously evaluate CDS within an EHR. We also identified important considerations in applying these methods. This approach may be an important tool for improving the impact of and experience with CDS. Trial Registration Flu alert trial: ClinicalTrials.gov NCT03415425; https://clinicaltrials.gov/ct2/show/NCT03415425. Tobacco alert trial: ClinicalTrials.gov NCT03714191; https://clinicaltrials.gov/ct2/show/NCT03714191

scholarly journals A Narrative Review of Clinical Decision Support for Inpatient Clinical Pharmacists

2021 ◽  
Vol 12 (02) ◽  
pp. 199-207
Author(s):  
Liang Yan ◽  
Thomas Reese ◽  
Scott D. Nelson
Keyword(s):  
Patient Safety ◽  
Quality Of Care ◽  
Decision Support ◽  
Clinical Decision Support ◽  
Clinical Decision ◽  
Controlled Trials ◽  
Clinical Pharmacists ◽  
Randomized Controlled ◽  
The Impact

Abstract Objective Increasingly, pharmacists provide team-based care that impacts patient care; however, the extent of recent clinical decision support (CDS), targeted to support the evolving roles of pharmacists, is unknown. Our objective was to evaluate the literature to understand the impact of clinical pharmacists using CDS. Methods We searched MEDLINE, EMBASE, and Cochrane Central for randomized controlled trials, nonrandomized trials, and quasi-experimental studies which evaluated CDS tools that were developed for inpatient pharmacists as a target user. The primary outcome of our analysis was the impact of CDS on patient safety, quality use of medication, and quality of care. Outcomes were scored as positive, negative, or neutral. The secondary outcome was the proportion of CDS developed for tasks other than medication order verification. Study quality was assessed using the Newcastle–Ottawa Scale. Results Of 4,365 potentially relevant articles, 15 were included. Five studies were randomized controlled trials. All included studies were rated as good quality. Of the studies evaluating inpatient pharmacists using a CDS tool, four showed significantly improved quality use of medications, four showed significantly improved patient safety, and three showed significantly improved quality of care. Six studies (40%) supported expanded roles of clinical pharmacists. Conclusion These results suggest that CDS can support clinical inpatient pharmacists in preventing medication errors and optimizing pharmacotherapy. Moreover, an increasing number of CDS tools have been developed for pharmacists' roles outside of order verification, whereby further supporting and establishing pharmacists as leaders in safe and effective pharmacotherapy.

scholarly journals A Closer Look at the “Right” Format for Clinical Decision Support: Methods for Evaluating a Storyboard BestPractice Advisory

2020 ◽  
Vol 10 (4) ◽  
pp. 142
Author(s):  
Brian J. Douthit ◽  
R. Clayton Musser ◽  
Kay S. Lytle ◽  
Rachel L. Richesson
Keyword(s):  
Decision Support ◽  
Clinical Decision Support ◽  
Local Level ◽  
Clinical Decision ◽  
System Level ◽  
Evaluation Methodology ◽  
Task Completion ◽  
Free Text ◽  
Alert Fatigue ◽  
The Impact

(1) Background: The five rights of clinical decision support (CDS) are a well-known framework for planning the nuances of CDS, but recent advancements have given us more options to modify the format of the alert. One-size-fits-all assessments fail to capture the nuance of different BestPractice Advisory (BPA) formats. To demonstrate a tailored evaluation methodology, we assessed a BPA after implementation of Storyboard for changes in alert fatigue, behavior influence, and task completion; (2) Methods: Data from 19 weeks before and after implementation were used to evaluate differences in each domain. Individual clinics were evaluated for task completion and compared for changes pre- and post-redesign; (3) Results: The change in format was correlated with an increase in alert fatigue, a decrease in erroneous free text answers, and worsened task completion at a system level. At a local level, however, 14% of clinics had improved task completion; (4) Conclusions: While the change in BPA format was correlated with decreased performance, the changes may have been driven primarily by the COVID-19 pandemic. The framework and metrics proposed can be used in future studies to assess the impact of new CDS formats. Although the changes in this study seemed undesirable in aggregate, some positive changes were observed at the level of individual clinics. Personalized implementations of CDS tools based on local need should be considered.

scholarly journals Improving Provider Adoption With Adaptive Clinical Decision Support Surveillance: An Observational Study (Preprint)

2018 ◽  
Author(s):  
Sundas Khan ◽  
Safiya Richardson ◽  
Andrew Liu ◽  
Vinodh Mechery ◽  
Lauren McCullagh ◽  
...  
Keyword(s):  
Decision Support ◽  
Clinical Decision Support ◽  
Clinical Decision ◽  
Intermediate Risk ◽  
Intravenous Contrast ◽  
Alert Fatigue ◽  
Trigger Rate ◽  
Tertiary Center ◽  
The Impact ◽  
Academic Year

BACKGROUND Successful clinical decision support (CDS) tools can help use evidence-based medicine to effectively improve patient outcomes. However, the impact of these tools has been limited by low provider adoption due to overtriggering, leading to alert fatigue. We developed a tracking mechanism for monitoring trigger (percent of total visits for which the tool triggers) and adoption (percent of completed tools) rates of a complex CDS tool based on the Wells criteria for pulmonary embolism (PE). OBJECTIVE We aimed to monitor and evaluate the adoption and trigger rates of the tool and assess whether ongoing tool modifications would improve adoption rates. METHODS As part of a larger clinical trial, a CDS tool was developed using the Wells criteria to calculate pretest probability for PE at 2 tertiary centers’ emergency departments (EDs). The tool had multiple triggers: any order for D-dimer, computed tomography (CT) of the chest with intravenous contrast, CT pulmonary angiography (CTPA), ventilation-perfusion scan, or lower extremity Doppler ultrasound. A tracking dashboard was developed using Tableau to monitor real-time trigger and adoption rates. Based on initial low provider adoption rates of the tool, we conducted small focus groups with key ED providers to elicit barriers to tool use. We identified overtriggering of the tool for non-PE-related evaluations and inability to order CT testing for intermediate-risk patients. Thus, the tool was modified to allow CT testing for the intermediate-risk group and not to trigger for CT chest with intravenous contrast orders. A dialogue box, “Are you considering PE for this patient?” was added before the tool triggered to account for CTPAs ordered for aortic dissection evaluation. RESULTS In the ED of tertiary center 1, 95,295 patients visited during the academic year. The tool triggered for an average of 509 patients per month (average trigger rate 2036/30,234, 6.73%) before the modifications, reducing to 423 patients per month (average trigger rate 1629/31,361, 5.22%). In the ED of tertiary center 2, 88,956 patients visited during the academic year, with the tool triggering for about 473 patients per month (average trigger rate 1892/29,706, 6.37%) before the modifications and for about 400 per month (average trigger rate 1534/30,006, 5.12%) afterward. The modifications resulted in a significant 4.5- and 3-fold increase in provider adoption rates in tertiary centers 1 and 2, respectively. The modifications increased the average monthly adoption rate from 23.20/360 (6.5%) tools to 81.60/280.20 (29.3%) tools and 46.60/318.80 (14.7%) tools to 111.20/263.40 (42.6%) tools in centers 1 and 2, respectively. CONCLUSIONS Close postimplementation monitoring of CDS tools may help improve provider adoption. Adaptive modifications based on user feedback may increase targeted CDS with lower trigger rates, reducing alert fatigue and increasing provider adoption. Iterative improvements and a postimplementation monitoring dashboard can significantly improve adoption rates.

scholarly journals The impact of the Nutri-Score front-of-pack nutrition label on purchasing intentions of unprocessed and processed foods: post-hoc analyses from three randomized controlled trials

2021 ◽  
Vol 18 (1) ◽  
Author(s):  
Manon Egnell ◽  
Pilar Galan ◽  
Morgane Fialon ◽  
Mathilde Touvier ◽  
Sandrine Péneau ◽  
...  
Keyword(s):  
Randomized Controlled Trials ◽  
Low Income ◽  
P Value ◽  
Controlled Trials ◽  
Nutrition Label ◽  
Randomized Controlled ◽  
Food Purchases ◽  
Relative Contribution ◽  
Processed Products ◽  
The Impact

Abstract Background The Nutri-Score summary graded front-of-pack nutrition label has been identified as an efficient tool to increase the nutritional quality of pre-packed food purchases. However, no study has been conducted to investigate the effect of the Nutri-Score on the shopping cart composition, considering the type of foods. The present paper aims to investigate the effect of the Nutri-Score on the type of food purchases, in terms of the relative contribution of unpacked and pre-packed foods, or the processing degree of foods. Methods Between September 2016 and April 2017, three consecutive randomized controlled trials were conducted in three specific populations – students (N = 1866), low-income individuals (N = 336) and subjects suffering from cardiometabolic diseases (N = 1180) – to investigate the effect of the Nutri-Score on purchasing intentions compared to the Reference Intakes and no label. Using these combined data, the proportion of unpacked products in the shopping carts, as well as the distribution of products across food categories taking into account the degree of processing (NOVA classification) were assessed by trials arm. Results The shopping carts of participants simulating purchases with the Nutri-Score affixed on pre-packed foods contained higher proportion of unpacked products – especially raw fruits and meats, i.e. with no FoPL –, compared to participants purchasing with no label (difference of 5.93 percentage points [3.88–7.99], p-value< 0.0001) or with the Reference Intakes (difference of 5.27[3.25–7.29], p-value< 0.0001). This higher proportion was partly explained by fewer purchases of pre-packed processed and ultra-processed products overall in the Nutri-Score group. Conclusions These findings provide new insights on the positive effect of the Nutri-Score, which appears to decrease purchases in processed products resulting in higher proportions of unprocessed and unpacked foods, in line with public health recommendations.

Sign in / Sign up

Export Citation Format

Share Document