External Validation and Comparison of Clostridioides difficile Severity Scoring Systems

2021 ◽  
Author(s):  
D Alexander Perry ◽  
Daniel Shirley ◽  
Dejan Micic ◽  
C Pratish Patel ◽  
Rosemary Putler ◽  
...  
Keyword(s):  
Characteristic Curve ◽  
External Validation ◽  
Predictive Ability ◽  
Scoring Systems ◽  
Receiver Operator Characteristic Curve ◽  
Adverse Outcomes ◽  
Risk Scores ◽  
Clostridioides Difficile ◽  
Multicenter Cohort ◽  
Severity Scores

Abstract Background Many models have been developed to predict severe outcomes from Clostridioides difficile infection. These models are usually developed at a single institution and largely are not externally validated. This aim of this study was to validate previously published risk scores in a multicenter cohort of patients with CDI. Methods Retrospective study on four separate inpatient cohorts with CDI from three distinct sites: The Universities of Michigan (2010-2012 and 2016), Chicago (2012), and Wisconsin (2012). The primary composite outcome was admission to an intensive care unit, colectomy, and/or death attributed to CDI within 30 days of positive testing. Both within each cohort and combined across all cohorts, published CDI severity scores were assessed and compared to each other and the IDSA guideline definitions of severe and fulminant CDI. Results A total of 3,646 patients were included for analysis. Including the two IDSA guideline definitions, fourteen scores were assessed. Performance of scores varied within each cohort and in the combined set (mean area under the receiver operator characteristic curve(AUC 0.61, range 0.53-0.66). Only half of the scores had performance at or better than IDSA severe and fulminant definitions (AUCs 0.64 and 0.63, respectively). Most of the scoring systems had more false than true positives in the combined set (mean: 81.5%, range:0-91.5%). Conclusions No published CDI severity score showed stable, good predictive ability for adverse outcomes across multiple cohorts/institutions or in a combined multicenter cohort.

scholarly journals 2409. External Validation and Comparison of Clostridioides difficile Severity Scoring Systems

2019 ◽  
Vol 6 (Supplement_2) ◽  
pp. S831-S832
Author(s):  
Donald A Perry ◽  
Daniel Shirley ◽  
Dejan Micic ◽  
Rosemary K B Putler ◽  
Pratish Patel ◽  
...  
Keyword(s):  
Binary Classification ◽  
Characteristic Curve ◽  
External Validation ◽  
Receiver Operator Characteristic Curve ◽  
Adverse Outcomes ◽  
Risk Scores ◽  
Composite Outcome ◽  
Clostridioides Difficile ◽  
Severity Scores ◽  
Primary Composite Outcome

Abstract Background Annually in the US alone, Clostridioides difficile infection (CDI) afflicts nearly 500,000 patients causing 29,000 deaths. Since early and aggressive interventions could save lives but are not optimally deployed in all patients, numerous studies have published predictive models for adverse outcomes. These models are usually developed at a single institution, and largely are not externally validated. This aim of this study was to validate the predictability for severe CDI with previously published risk scores in a multicenter cohort of patients with CDI. Methods We conducted a retrospective study on four separate inpatient cohorts with CDI from three distinct sites: the Universities of Michigan (2010–2012 and 2016), Chicago (2012), and Wisconsin (2012). The primary composite outcome was admission to an intensive care unit, colectomy, and/or death attributed to CDI within 30 days of positive test. Structured query and manual chart review abstracted data from the medical record at each site. Published CDI severity scores were assessed and compared with each other and the IDSA guideline definition of severe CDI. Sensitivity, specificity, area under the receiver operator characteristic curve (AuROC), precision-recall curves, and net reclassification index (NRI) were calculated to compare models. Results We included 3,775 patients from the four cohorts (Table 1) and evaluated eight severity scores (Table 2). The IDSA (baseline comparator) model showed poor performance across cohorts(Table 3). Of the binary classification models, including those that were most predictive of the primary composite outcome, Jardin, performed poorly with minimal to no NRI improvement compared with IDSA. The continuous score models, Toro and ATLAS, performed better, but the AuROC varied by site by up to 17% (Table 3). The Gujja model varied the most: from most predictive in the University of Michigan 2010–2012 cohort to having no predictive value in the 2016 cohort (Table 3). Conclusion No published CDI severity score showed stable, acceptable predictive ability across multiple cohorts/institutions. To maximize performance and clinical utility, future efforts should focus on a multicenter-derived and validated scoring system, and/or incorporate novel biomarkers. Disclosures All authors: No reported disclosures.

External Validation for Acute Kidney Injury Severity Scores: A Multicenter Retrospective Study in 14 Japanese ICUs

2015 ◽  
Vol 42 (1) ◽  
pp. 57-64 ◽  
Author(s):  
Tetsu Ohnuma ◽  
Shigehiko Uchino ◽  
Noriyoshi Toki ◽  
Kenta Takeda ◽  
Yoshitomo Namba ◽  
...  
Keyword(s):  
Acute Kidney Injury ◽  
Intensive Care ◽  
Injury Severity ◽  
Characteristic Curve ◽  
External Validation ◽  
Kidney Injury ◽  
Receiver Operator Characteristic Curve ◽  
Saps Ii ◽  
Severity Scores ◽  
Saps Ii Score

Background/Aims: Acute kidney injury (AKI) is associated with high mortality. Multiple AKI severity scores have been derived to predict patient outcome. We externally validated new AKI severity scores using the Japanese Society for Physicians and Trainees in Intensive Care (JSEPTIC) database. Methods: New AKI severity scores published in the 21st century (Mehta, Stuivenberg Hospital Acute Renal Failure (SHARF) II, Program to Improve Care in Acute Renal Disease (PICARD), Vellore and Demirjian), Liano, Simplified Acute Physiology Score (SAPS) II and lactate were compared using the JSEPTIC database that collected retrospectively 343 patients with AKI who required continuous renal replacement therapy (CRRT) in 14 intensive care units. Accuracy of the severity scores was assessed by the area under the receiver-operator characteristic curve (AUROC, discrimination) and Hosmer-Lemeshow test (H-L test, calibration). Results: The median age was 69 years and 65.8% were male. The median SAPS II score was 53 and the hospital mortality was 58.6%. The AUROC curves revealed low discrimination ability of the new AKI severity scores (Mehta 0.65, SHARF II 0.64, PICARD 0.64, Vellore 0.64, Demirjian 0.69), similar to Liano 0.67, SAPS II 0.67 and lactate 0.64. The H-L test also demonstrated that all assessed scores except for Liano had significantly low calibration ability. Conclusions: Using a multicenter database of AKI patients requiring CRRT, this study externally validated new AKI severity scores. While the Demirjian's score and Liano's score showed a better performance, further research will be required to confirm these findings.

scholarly journals External validation of two prediction tools for patients at risk for recurrent Clostridioides difficile infection

2021 ◽  
Vol 14 ◽  
pp. 175628482097738
Author(s):  
Tessel M. van Rossen ◽  
Laura J. van Dijk ◽  
Martijn W. Heymans ◽  
Olaf M. Dekkers ◽  
Christina M. J. E. Vandenbroucke-Grauls ◽  
...  
Keyword(s):  
At Risk ◽  
Validation Cohort ◽  
Characteristic Curve ◽  
External Validation ◽  
Risk Scores ◽  
Prediction Tools ◽  
Clostridioides Difficile ◽  
Clostridioides Difficile Infection ◽  
Patients At Risk ◽  
Definition Of

Background: One in four patients with primary Clostridioides difficile infection (CDI) develops recurrent CDI (rCDI). With every recurrence, the chance of a subsequent CDI episode increases. Early identification of patients at risk for rCDI might help doctors to guide treatment. The aim of this study was to externally validate published clinical prediction tools for rCDI. Methods: The validation cohort consisted of 129 patients, diagnosed with CDI between 2018 and 2020. rCDI risk scores were calculated for each individual patient in the validation cohort using the scoring tools described in the derivation studies. Per score value, we compared the average predicted risk of rCDI with the observed number of rCDI cases. Discrimination was assessed by calculating the area under the receiver operating characteristic curve (AUC). Results: Two prediction tools were selected for validation (Cobo 2018 and Larrainzar-Coghen 2016). The two derivation studies used different definitions for rCDI. Using Cobo’s definition, rCDI occurred in 34 patients (26%) of the validation cohort: using the definition of Larrainzar-Coghen, we observed 19 recurrences (15%). The performance of both prediction tools was poor when applied to our validation cohort. The estimated AUC was 0.43 [95% confidence interval (CI); 0.32–0.54] for Cobo’s tool and 0.42 (95% CI; 0.28–0.56) for Larrainzar-Coghen’s tool. Conclusion: Performance of both prediction tools was disappointing in the external validation cohort. Currently identified clinical risk factors may not be sufficient for accurate prediction of rCDI.

Abstract TP13: External Validation of Stroke Prognostication Scales - A Retrospective Cohort Study

Stroke ◽  
2020 ◽  
Vol 51 (Suppl_1) ◽  
Author(s):  
Nauman Jahangir ◽  
Nicholas Lanzotti ◽  
Kyle Gollon ◽  
Mehwish Farooqi ◽  
Michael Buhnerkempe ◽  
...  
Keyword(s):  
Ischemic Stroke ◽  
Acute Ischemic Stroke ◽  
Characteristic Curve ◽  
External Validation ◽  
Area Under The Curve ◽  
Predictive Ability ◽  
Scoring Systems ◽  
Predictive Performance ◽  
Discriminatory Ability ◽  
Single Center Study

Introduction: In recent years, many scoring models have been proposed to predict clinical outcomes after acute ischemic stroke. Aim of our study was to perform a comparative analysis of these scoring systems to assess predictive reliability. Method: This retrospective single center study included 166 community-based patients presenting with an acute ischemic stroke between 2015 and 2018 who had undergone mechanical thrombectomy with or without IV r-tPA administration prior to the procedure. Patients with unknown 90 day Modified Ranking Scale (mRS) were excluded from the study. We included SPAN-100, THRIVE, HIAT2, iScore , TPI, DRAGON, ASTRAL and HAT predictive models to our study. To predict MRS at 90 days, we first dichotomize mRS into two groups: scores of 0 and 1 and scores 2 and above. We then used logistic regression to find the association between a stroke score and the probability of having a 90-day mRS of 2 or above. Separate univariate logistic regressions were fit for each stroke score. We assessed the ability of each stroke score to predict 90-day mRS using the area-under-the-curve (AUC) of the receiver operating characteristic curve (ROC - plot of sensitivity against 1-specificity). AUC values range from 0.5 to 1 with values above 0.7 showing good discriminatory ability. Results: SPAN-100, HIAT2, iScore, and ASTRAL scores have similar predictive ability with AUC values over 0.7 (Table 1). The ASTRAL score had the highest predictive ability with a score above 31.5 indicating a high likelihood of a 90-day MRS ≥ 2 (Table 1). The TPI, DRAGON, and HAT scores all had AUCs below 0.65 indicating poor predictive performance in our data. Conclusion: The SPAN-100, HIAT2, iScore, and ASTRAL scores reliably predicts 90-day mRS of 2 or greater in patients with acute ischemic stroke.

scholarly journals Haemoconcentration as a predictor of severity in acute pancreatitis

2016 ◽  
Vol 4 (1) ◽  
pp. 3-7
Author(s):  
Tanka Prasad Bohara ◽  
Dimindra Karki ◽  
Anuj Parajuli ◽  
Shail Rupakheti ◽  
Mukund Raj Joshi
Keyword(s):  
Acute Pancreatitis ◽  
Pearson Correlation ◽  
Characteristic Curve ◽  
Scoring Systems ◽  
Cost Effective ◽  
Receiver Operator Characteristic Curve ◽  
First Episode ◽  
P Value ◽  
Receiver Operator Characteristic ◽  
Chi Square

Background: Acute pancreatitis is usually a mild and self-limiting disease. About 25 % of patients have severe episode with mortality up to 30%. Early identification of these patients has potential advantages of aggressive treatment at intensive care unit or transfer to higher centre. Several scoring systems are available to predict severity of acute pancreatitis but are cumbersome, take 24 to 48 hours and are dependent on tests that are not universally available. Haematocrit has been used as a predictor of severity of acute pancreatitis but some have doubted its role.Objectives: To study the significance of haematocrit in prediction of severity of acute pancreatitis.Methods: Patients admitted with first episode of acute pancreatitis from February 2014 to July 2014 were included. Haematocrit at admission and 24 hours of admission were compared with severity of acute pancreatitis. Mean, analysis of variance, chi square, pearson correlation and receiver operator characteristic curve were used for statistical analysis.Results: Thirty one patients were included in the study with 16 (51.61%) male and 15 (48.4%) female. Haematocrit at 24 hours of admission was higher in severe acute pancreatitis (P value 0.003). Both haematocrit at admission and at 24 hours had positive correlation with severity of acute pancreatitis (r: 0.387; P value 0.031 and r: 0.584; P value 0.001) respectively.Area under receiver operator characteristic curve for haematocrit at admission and 24 hours were 0.713 (P value 0.175, 95% CI 0.536 - 0.889) and 0.917 (P value 0.008, 95% CI 0.813 – 1.00) respectively.Conclusion: Haematocrit is a simple, cost effective and widely available test and can predict severity of acute pancreatitis.Journal of Kathmandu Medical College, Vol. 4(1) 2015, 3-7

scholarly journals Development and External Validation of Deep-Learning-Based Tumor Grading Models in Soft-Tissue Sarcoma Patients Using MR Imaging

Cancers ◽  
2021 ◽  
Vol 13 (12) ◽  
pp. 2866
Author(s):  
Fernando Navarro ◽  
Hendrik Dapper ◽  
Rebecca Asadpour ◽  
Carolin Knebel ◽  
Matthew B. Spraker ◽  
...  
Keyword(s):  
Deep Learning ◽  
Soft Tissue ◽  
Mr Imaging ◽  
Characteristic Curve ◽  
External Validation ◽  
Soft Tissue Sarcomas ◽  
Treatment Decision ◽  
Receiver Operator Characteristic Curve ◽  
Low Grade ◽  
Tumor Grading

Background: In patients with soft-tissue sarcomas, tumor grading constitutes a decisive factor to determine the best treatment decision. Tumor grading is obtained by pathological work-up after focal biopsies. Deep learning (DL)-based imaging analysis may pose an alternative way to characterize STS tissue. In this work, we sought to non-invasively differentiate tumor grading into low-grade (G1) and high-grade (G2/G3) STS using DL techniques based on MR-imaging. Methods: Contrast-enhanced T1-weighted fat-saturated (T1FSGd) MRI sequences and fat-saturated T2-weighted (T2FS) sequences were collected from two independent retrospective cohorts (training: 148 patients, testing: 158 patients). Tumor grading was determined following the French Federation of Cancer Centers Sarcoma Group in pre-therapeutic biopsies. DL models were developed using transfer learning based on the DenseNet 161 architecture. Results: The T1FSGd and T2FS-based DL models achieved area under the receiver operator characteristic curve (AUC) values of 0.75 and 0.76 on the test cohort, respectively. T1FSGd achieved the best F1-score of all models (0.90). The T2FS-based DL model was able to significantly risk-stratify for overall survival. Attention maps revealed relevant features within the tumor volume and in border regions. Conclusions: MRI-based DL models are capable of predicting tumor grading with good reproducibility in external validation.

scholarly journals Recurrent Clostridioides difficile infection can be predicted using inflammatory mediator and toxin activity levels

2020 ◽  
Vol 41 (S1) ◽  
pp. s77-s78
Author(s):  
Jonathan Motyka ◽  
Aline Penkevich ◽  
Vincent Young ◽  
Krishna Rao
Keyword(s):  
Machine Learning ◽  
Inflammatory Mediators ◽  
Inflammatory Mediator ◽  
Characteristic Curve ◽  
Receiver Operator Characteristic Curve ◽  
Sensitivity Analyses ◽  
Activity Levels ◽  
Clinical Criteria ◽  
Clostridioides Difficile ◽  
Potential Biomarker

Background:Clostridioides difficile infection (CDI) frequently recurs after initial treatment. Predicting recurrent CDI (rCDI) early in the disease course can assist clinicians in their decision making and improve outcomes. However, predictions based on clinical criteria alone are not accurate and/or do not validate other results. Here, we tested the hypothesis that circulating and stool-derived inflammatory mediators predict rCDI. Methods: Consecutive subjects with available specimens at diagnosis were included if they tested positive for toxigenic C. difficile (+enzyme immunoassay [EIA] for glutamate dehydrogenase and toxins A/B, with reflex to PCR for the tcdB gene for discordants). Stool was thawed on ice, diluted 1:1 in PBS with protease inhibitor, centrifuged, and used immediately. A 17-plex panel of inflammatory mediators was run on a Luminex 200 machine using a custom antibody-linked bead array. Prior to analysis, all measurements were normalized and log-transformed. Stool toxin activity levels were quantified using a custom cell-culture assay. Recurrence was defined as a second episode of CDI within 100 days. Ordination characterized variation in the panel between outcomes, tested with a permutational, multivariate ANOVA. Machine learning via elastic net regression with 100 iterations of 5-fold cross validation selected the optimal model and the area under the receiver operator characteristic curve (AuROC) was computed. Sensitivity analyses excluding those that died and/or lived >100 km away were performed. Results: We included 186 subjects, with 95 women (51.1%) and average age of 55.9 years (±20). More patients were diagnosed by PCR than toxin EIA (170 vs 55, respectively). Death, rCDI, and no rCDI occurred in 32 (17.2%), 36 (19.4%), and 118 (63.4%) subjects, respectively. Ordination revealed that the serum panel was associated with rCDI (P = .007) but the stool panel was not. Serum procalcitonin, IL-8, IL-6, CCL5, and EGF were associated with recurrence. The machine-learning models using the serum panel predicted rCDI with AuROCs between 0.74 and 0.8 (Fig. 1). No stool inflammatory mediators independently predicted rCDI. However, stool IL-8 interacted with toxin activity to predict rCDI (Fig. 2). These results did not change significantly upon sensitivity analysis. Conclusions: A panel of serum inflammatory mediators predicted rCDI with up to 80% accuracy, but the stool panel alone was less successful. Incorporating toxin activity levels alongside inflammatory mediator measurements is a novel, promising approach to studying stool-derived biomarkers of rCDI. This approach revealed that stool IL-8 is a potential biomarker for rCDI. These results need to be confirmed both with a larger dataset and after adjustment for clinical covariates.Funding: NoneDisclosure: Vincent Young is a consultant for Bio-K+ International, Pantheryx, and Vedanta Biosciences.

Vaginal examination vs. cervical length – which is superior in predicting preterm birth?

2017 ◽  
Vol 45 (8) ◽  
Author(s):  
Merav Sharvit ◽  
Reut Weiss ◽  
Yael Ganor Paz ◽  
Keren Tzadikevitch Geffen ◽  
Netanella Danielli Miller ◽  
...  
Keyword(s):  
Preterm Birth ◽  
Medical Center ◽  
Characteristic Curve ◽  
Multiple Pregnancy ◽  
Predictive Ability ◽  
Cervical Length ◽  
Receiver Operator Characteristic Curve ◽  
Retrospective Case ◽  
Nulliparous Women ◽  
Significant Difference

AbstractObjective:To compare the predictive value of preterm birth (PTB) by transvaginal sonographic cervical length (CL) measurement to digital examination of the cervix (Bishop score – BS), in patients with premature contractions (PC) and intact membranes.Design:A retrospective case-control study.Setting:Meir Medical Center, Kfar Saba, Israel.Population:Women at 24–34 weeks of gestation who were hospitalized with PC and intact membranes.Methods:All patients underwent CL and BS measurements upon admission. Power analysis revealed that 375 patients were needed to show a significant difference between the two methods for predicting PTB. Each one served as her own control.Main outcome measures:PTB<37 and<34 weeks.Results:Receiver-operator characteristic curve (ROC) and logistic regression analyses indicated a correlation between both shortened CL and increased BS to PTB (P<0.001). Neither test offered an advantage in predicting PTB. Areas under the curve for BS and CL ROC were similar for PTB before 37 weeks gestation (0.611 vs. 0.640, P=0.28). For nulliparous women, CL predicted PTB better that BS (0.642 vs. 0.724, P=0.03). For singleton and multiple pregnancy pregnancies, BS and CL did not differ significantly in predicting PTB (P=0.9, P=0.2, respectively). For nulliparous with multiple pregnancy, the BS and CL ROC curves differ nearly significantly (0.554 vs. 0.709, P=0.07), with better predictive ability for CL.Conclusions:CL and BS have similar value in predicting PTB in patients with PC. For nulliparous women, CL is superior over the BS.

scholarly journals A new approach to scoring systems to improve identification of acute medical admissions that will require critical care

2011 ◽  
Vol 56 (4) ◽  
pp. 195-202 ◽  
Author(s):  
H A Carmichael ◽  
E Robertson ◽  
J Austin ◽  
D Mccruden ◽  
C M Messow ◽  
...  
Keyword(s):  
Early Warning ◽  
Characteristic Curve ◽  
Medical Patient ◽  
Scoring Systems ◽  
Receiver Operator Characteristic Curve ◽  
Early Warning Score ◽  
Peripheral Oxygen Saturation ◽  
Logistic Regression Models ◽  
Prospective Application ◽  
Gas Measurements

Removal of the intensive care unit (ICU) at the Vale of Leven Hospital mandated the identification and transfer out of those acute medical admissions with a high risk of requiring ICU. The aim of the study was to develop triaging tools that identified such patients and compare them with other scoring systems. The methodology included a retrospective analysis of physiological and arterial gas measurements from 1976 acute medical admissions produced PREEMPT-1 (PRE-critical Emergency Medical Patient Triage). A simpler one for ambulance use (PREAMBLE-1 [PRE-Admission Medical Blue-Light Emergency]) was produced by the addition of peripheral oxygen saturation to a modification of MEWS (Modified Early Warning Score). Prospective application of these tools produced a larger database of 4447 acute admissions from which logistic regression models produced PREEMPT-2 and PREAMBLE-2, which were then compared with the original systems and seven other early warning scoring systems. Results showed that in patients with arterial gases, the area under the receiver operator characteristic curve was significantly higher in PREEMPT-2 (89·1%) and PREAMBLE-2 (84.4%) than all other scoring systems. Similarly, in all patients, it was higher in PREAMBLE-2 (92.4%) than PREAMBLE-1 (88.1%) and the other scoring systems. In conclusion, risk of requiring ICU can be more accurately predicted using PREEMPT-2 and PREAMBLE-2, as described here, than by other early warning scoring systems developed over recent years.

Scoring Systems for Estimating the Risk of Anticoagulant-Associated Bleeding

2017 ◽  
Vol 43 (05) ◽  
pp. 514-524 ◽  
Author(s):  
Anna Parks ◽  
Margaret Fang
Keyword(s):  
Clinical Practice ◽  
Risk Index ◽  
Predictive Ability ◽  
Scoring Systems ◽  
Bleeding Risk ◽  
Assessment Tools ◽  
Risk Scores ◽  
Bleeding Complications ◽  
Individual Risk ◽  
Thrombosis Risk

AbstractAnticoagulant medications are frequently used to prevent and treat thromboembolic disease. However, the benefits of anticoagulants must be balanced with a careful assessment of the risk of bleeding complications that can ensue from their use. Several bleeding risk scores are available, including the Outpatient Bleeding Risk Index, HAS-BLED, ATRIA, and HEMORR2HAGES risk assessment tools, and can be used to help estimate patients' risk for bleeding on anticoagulants. These tools vary by their individual risk components and in how they define and weigh clinical factors. However, it is not yet clear how best to integrate bleeding risk tools into clinical practice. Current bleeding risk scores generally have modest predictive ability and limited ability to predict the most devastating complication of anticoagulation, intracranial hemorrhage. In clinical practice, bleeding risk tools should be paired with a formal determination of thrombosis risk, as their results may be most influential for patients at the lower end of thrombosis risk, as well as for highlighting potentially modifiable risk factors for bleeding. Use of bleeding risk scores may assist clinicians and patients in making informed and individualized anticoagulation decisions.

Sign in / Sign up

Export Citation Format

Share Document