Cohen's kappa coefficient as a performance measure for feature selection

2010 ◽  
Author(s):  
Susana M. Vieira ◽  
Uzay Kaymak ◽  
Joao M. C. Sousa
Keyword(s):  
Feature Selection ◽  
Performance Measure ◽  
Kappa Coefficient ◽  
Cohen’S Kappa ◽  
Cohen's Kappa ◽  
Cohen’S Kappa Coefficient ◽  
A Performance

scholarly journals Bayesian network-based methodology for selecting a cost-effective sewer asset management model

2020 ◽  
Author(s):  
Julián Guzmán-Fierro ◽  
Sharel Charry ◽  
Ivan González ◽  
Felipe Peña-Heredia ◽  
Nathalie Hernández ◽  
...  
Keyword(s):  
Asset Management ◽  
Kappa Coefficient ◽  
Wilcoxon Test ◽  
Cohen’S Kappa ◽  
Validation Data ◽  
Statistical Measures ◽  
Proactive Management ◽  
Cohen's Kappa ◽  
Cohen’S Kappa Coefficient

Abstract This paper presents a methodology based on Bayesian Networks (BN) to prioritise and select the minimal number of variables that allows predicting the structural condition of sewer assets to support the strategies in proactive management. The integration of BN models, statistical measures of agreement (Cohen's Kappa coefficient) and a statistical test (Wilcoxon test) were useful for a robust and straightforward selection of a minimum number of variables (qualitative and quantitative) that ensure a suitable prediction level of the structural conditions of sewer pipes. According to the application of the methodology to a specific case study (Bogotás sewer network, Colombia), it found that with only two variables (age and diameter) the model could achieve the same capacity of prediction (Cohen's Kappa coefficient = 0.43) as a model considering several variables. Furthermore, the methodology allows finding the calibration and validation percentage subsets that best fit (80% for calibration and 20% for validation data in the case study) in the model to increase the capacity of prediction with low variations. Furthermore, it found that a model, considering only pipes in critical and excellent conditions, increases the capacity of successful predictions (Cohen's Kappa coefficient from 0.2 to 0.43) for the proposed case study.

scholarly journals Automatic Sleep Stage Classification Based on Two-channel EOG and One-channel EMG

2021 ◽  
Author(s):  
Yanjun LI ◽  
Xianglin Yang ◽  
Zhi Xu ◽  
Yu Zhang ◽  
Zhongping Cao
Keyword(s):  
Sleep Stage ◽  
Kappa Coefficient ◽  
Sleep Monitoring ◽  
Cohen’S Kappa ◽  
Cohen's Kappa ◽  
Stage Classification ◽  
Sleep Stage Classification ◽  
Feature Normalization ◽  
Leave One Out ◽  
Cohen’S Kappa Coefficient

Abstract The sleep monitoring with PSG severely degrades the sleep quality. In order to simplify the hygienic processing and reduce the load of sleep monitoring, an approach to automatic sleep stage classification without electroencephalogram (EEG) was explored. Totally 108 features from two-channel electrooculogram (EOG) and 6 features from one-channel electromyogram (EMG) were extracted. After feature normalization, the random forest (RF) was used to classify five stages, including wakefulness, REM sleep, N1 sleep, N2 sleep and N3 sleep. Using 114 normalized features from the combination of EOG (108 features) and EMG (6 features), the Cohen’s kappa coefficient was 0.749 and the accuracy was 80.8% by leave-one -out cross-validation (LOOCV) for 124 records from ISRUC-Sleep. As a reference for AASM standard, the Cohen’s kappa coefficient was 0.801 and the accuracy was 84.7% for the same dataset based on 438 normalized features from the combination of EEG (324 features), EOG (108 features) and EMG (6 features). In conclusion, the approach by EOG+EMG with the normalization can reduce the load of sleep monitoring, and achieves comparable performances with the "gold standard" EEG+EOG+EMG on sleep classification.

scholarly journals Patient-Specific Explanations for Predictions of Clinical Outcomes

ACI Open ◽  
2019 ◽  
Vol 03 (02) ◽  
pp. e88-e97
Author(s):  
Mohammadamin Tajgardoon ◽  
Malarkodi J. Samayamuthu ◽  
Luca Calzoni ◽  
Shyam Visweswaran
Keyword(s):  
Machine Learning ◽  
Clinical Outcomes ◽  
Characteristic Curve ◽  
Community Acquired Pneumonia ◽  
Kappa Coefficient ◽  
Patient Specific ◽  
P Value ◽  
Cohen’S Kappa ◽  
Cohen's Kappa ◽  
Cohen’S Kappa Coefficient

Abstract Background Machine learning models that are used for predicting clinical outcomes can be made more useful by augmenting predictions with simple and reliable patient-specific explanations for each prediction. Objectives This article evaluates the quality of explanations of predictions using physician reviewers. The predictions are obtained from a machine learning model that is developed to predict dire outcomes (severe complications including death) in patients with community acquired pneumonia (CAP). Methods Using a dataset of patients diagnosed with CAP, we developed a predictive model to predict dire outcomes. On a set of 40 patients, who were predicted to be either at very high risk or at very low risk of developing a dire outcome, we applied an explanation method to generate patient-specific explanations. Three physician reviewers independently evaluated each explanatory feature in the context of the patient's data and were instructed to disagree with a feature if they did not agree with the magnitude of support, the direction of support (supportive versus contradictory), or both. Results The model used for generating predictions achieved a F1 score of 0.43 and area under the receiver operating characteristic curve (AUROC) of 0.84 (95% confidence interval [CI]: 0.81–0.87). Interreviewer agreement between two reviewers was strong (Cohen's kappa coefficient = 0.87) and fair to moderate between the third reviewer and others (Cohen's kappa coefficient = 0.49 and 0.33). Agreement rates between reviewers and generated explanations—defined as the proportion of explanatory features with which majority of reviewers agreed—were 0.78 for actual explanations and 0.52 for fabricated explanations, and the difference between the two agreement rates was statistically significant (Chi-square = 19.76, p-value < 0.01). Conclusion There was good agreement among physician reviewers on patient-specific explanations that were generated to augment predictions of clinical outcomes. Such explanations can be useful in interpreting predictions of clinical outcomes.

scholarly journals Seroprevalence of SARS-CoV-2 Assessed by Four Chemiluminescence Immunoassays and One Immunocromatography Test for SARS-Cov-2

2021 ◽  
Vol 9 ◽  
Author(s):  
Pellegrino Cerino ◽  
Alfonso Gallo ◽  
Biancamaria Pierri ◽  
Carlo Buonerba ◽  
Denise Di Concilio ◽  
...  
Keyword(s):  
Screening Program ◽  
Kappa Coefficient ◽  
Cohen’S Kappa ◽  
Campania Region ◽  
Entire Cohort ◽  
Cohen's Kappa ◽  
Rapid Tests ◽  
Second Wave ◽  
Cohen’S Kappa Coefficient ◽  
Good Agreement

The onset of the new SARS-CoV-2 coronavirus encouraged the development of new serologic tests that could be additional and complementary to real-time RT-PCR-based assays. In such a context, the study of performances of available tests is urgently needed, as their use has just been initiated for seroprevalence assessment. The aim of this study was to compare four chemiluminescence immunoassays and one immunochromatography test for SARS-Cov-2 antibodies for the evaluation of the degree of diffusion of SARS-CoV-2 infection in Salerno Province (Campania Region, Italy). A total of 3,185 specimens from citizens were tested for anti-SARS-CoV-2 antibodies as part of a screening program. Four automated immunoassays (Abbott and Liaison SARS-CoV-2 CLIA IgG and Roche and Siemens SARS-CoV-2 CLIA IgM/IgG/IgA assays) and one lateral flow immunoassay (LFIA Technogenetics IgG–IgM COVID-19) were used. Seroprevalence in the entire cohort was 2.41, 2.10, 1.82, and 1.85% according to the Liaison IgG, Abbott IgG, Siemens, and Roche total Ig tests, respectively. When we explored the agreement among the rapid tests and the serologic assays, we reported good agreement for Abbott, Siemens, and Roche (Cohen's Kappa coefficient 0.69, 0.67, and 0.67, respectively), whereas we found moderate agreement for Liaison (Cohen's kappa coefficient 0.58). Our study showed that Abbott and Liaison SARS-CoV-2 CLIA IgG, Roche and Siemens SARS-CoV-2 CLIA IgM/IgG/IgA assays, and LFIA Technogenetics IgG-IgM COVID-19 have good agreement in seroprevalence assessment. In addition, our findings indicate that the prevalence of IgG and total Ig antibodies against SARS-CoV-2 at the time of the study was as low as around 3%, likely explaining the amplitude of the current second wave.

scholarly journals Using Video Recording in Evaluating Skills of Medical Students in the Performance of the Orthopedic Examination

2021 ◽  
Vol 55 (3) ◽  
Author(s):  
Jose Ma D. Bautista ◽  
Peter B. Bernardo ◽  
Mark Anthony R. Ruanto
Keyword(s):  
Clinical Skills ◽  
Video Recording ◽  
Kappa Coefficient ◽  
Cohen’S Kappa ◽  
Perfect Agreement ◽  
Rater Reliability ◽  
Video Recordings ◽  
Cohen's Kappa ◽  
Improve Reliability ◽  
Cohen’S Kappa Coefficient

Objective. The study aims to assess the similarity between the results of the evaluation of students during an Objective Structured Clinical Examination (OSCE) and a video recording of the same OSCE (VOSCE). Methods. All Orthopedic surgeon preceptors in the actual OSCE were recruited to the study. Video recordings of the students taking the OSCE were collected and later reviewed and re-evaluated by the same preceptor after at least four weeks. The grades of actual OSCE and VOSCE were collected and analyzed using Cohen’s kappa coefficient. Results. High variability of intra-rater reliability was observed in different preceptors and station (slight agreement to perfect agreement). Overall intra-rater reliability between actual and video OSCE showed moderate agreement with Cohen’s kappa coefficient equal to 0.43 (n-219). Conclusion. Video OSCE is a reliable tool in assessing student clinical skills and knowledge in the musculoskeletal examination. Some factors have been suggested to further improve reliability.

Prevalence and Risk Factors for Bilateral Meniscal Tears Identified during Treatment for Cranial Cruciate Ligament Disease Via Tibial Plateau Levelling Osteotomy in Dogs

2020 ◽  
Author(s):  
Rebecca L. Laube ◽  
Kyle K. Kerstetter
Keyword(s):  
Risk Factors ◽  
Tibial Plateau ◽  
Cruciate Ligament ◽  
Significant Risk ◽  
Kappa Coefficient ◽  
Meniscal Tears ◽  
Cohen’S Kappa ◽  
Cranial Cruciate Ligament ◽  
Cohen's Kappa ◽  
Cohen’S Kappa Coefficient

Abstract Objective The aim of this study was to report the prevalence and risk factors of bilateral meniscal tears during a tibial plateau levelling osteotomy (TPLO). Methods Data from 362 dogs that underwent staged or simultaneous TPLO between January 2006 and April 2019 were retrospectively collected. Variables such as breed, sex, weight change and intervals between surgeries were analysed with logistic regression. Preoperative tibial plateau angle, age, cranial cruciate ligament status and body weight were analysed with a generalized linear mixed model. All analyses were performed to assess the likelihood of bilateral meniscal tears versus unilateral tears and no tears. Correlation of meniscal tears between stifles was assessed with Cohen's kappa coefficient. Results Prevalence of bilateral meniscal tears was 48.0% (95% confidence interval [CI]: 43.0–53.0%). There was moderate agreement of the presence of meniscal tears between stifles (Cohen's kappa coefficient = 0.41, 95% CI: 0.31–0.51).The odds for bilateral meniscal tears were higher for Rottweilers (odds ratio [OR:] 4.5 [95% CI 1.1–30.3], p = 0.033), older dogs (OR: 1.2 [95% CI: 1.1–1.4 per year], p < 0.0001), smaller dogs (OR: 0.98 [95% CI: 0.97–0.99 per 0.45-kg], p = 0.001), stifles with complete cranial cruciate ligament tears (OR: 21.1 [95% CI: 7.1–62.4], p < 0.0001). Conclusion Contralateral meniscal tears, breed, older age, lower patient weight and complete cranial cruciate ligament tear were significant risk factors for bilateral meniscal tears. Surgeons can use these results to determine prognoses and propensities for meniscal tears in at-risk dogs.

scholarly journals POS1436 EPIDEMIOLOGY OF LATENT TUBERCULOSIS INFECTION IN PATIENTS WITH RHEUMATIC IMMUNE-MEDIATED DISEASES. SINGLE UNIVERSITY STUDY OF 1117 PATIENTS

2021 ◽  
Vol 80 (Suppl 1) ◽  
pp. 1002.2-1003
Author(s):  
D. Martínez-López ◽  
J. Osorio-Chavez ◽  
C. Álvarez-Reguera ◽  
V. Portilla ◽  
M. A. González-Gay ◽  
...  
Keyword(s):  
Latent Tuberculosis Infection ◽  
Latent Tuberculosis ◽  
Kappa Coefficient ◽  
Tuberculosis Infection ◽  
Cohen’S Kappa ◽  
Cohen's Kappa ◽  
Immune Mediated ◽  
Igra Test ◽  
Ltbi Prevalence ◽  
Cohen’S Kappa Coefficient

Background:Patients with rheumatologic immune-mediated diseases (R-IMID) with Latent tuberculosis infection (LTBI) requiring biologic therapy (BT) are at an increased risk of active tuberculosis (TB). Screening of LTBI with tuberculin skin test (TST) and/or Interferon (IFN)-γ release assays (IGRA) is recommended before starting of BT.Objectives:In patients with R-IMID previously to BT our aim was to assess a) prevalence of LTBI, b) importance of using a booster test in negative TST and c) to compare TST with the IGRA test.Methods:Cross-sectional single University Hospital study including all patients diagnosed with R-IMID who underwent a TST and/or IGRA in the last five years (2016-2020).TST was performed by a subcutaneous injection of 0.1 ml of purified protein derivative (PPD) with a reading after 72 hours. TST was considered positive with an induration of more than 5 mm of diameter. If the first TST was negative, a new TST (Booster) was performed between 1 and 2 weeks after the first TST.LTBI was diagnosed by a positive IGRA and/or TST and absence of active TB (Chest radiograph). Diagnosis with IGRA vs TST was compared (Cohen’s kappa coefficient).Results:We included 1117 patients (741 women/376 men), mean age 53±15 years with LTBI. Chest radiograph was normal in most of the patients, only 39 patients (3.5%) presented signs of previous TB infection, mostly granuloma. Total LTBI prevalence was 31.7% (354/1117). LTBI prevalence in different underlying R-IMID ranges from 35% in vasculitis up to 26.5% in conectivopathies (Figure 1).Booster was positive in 66 patients (7.7%) out of 859 patients with a negative simple TST. Results of TST (+booster) and IGRA tests are shown in Table 1. TST (+booster) was positive in 187 patients (22.9%) out of 817 with a negative or indeterminate IGRA test. IGRA test was positive in 30 (3.8%) out of 793 patients with a negative TST (+booster). Cohen’s Kappa coefficient between TST (+booster) and IGRA (QFT-plus), was 0.381.Conclusion:LTBI is frequent between patients with R-IMID. Booster after negative simple TST may be useful, since it can detect false negatives for LTBI. IGRA and TST(+booster) show a low grade of agreement. Therefore, performing both tests before BT may be recommendable.Table 1.Results of TST (+booster) and IGRA testIGRA (QFT-Plus)PositiveNegativeIndeterminateUnavailableTotalTST(+Booster)Positive891424548324Negative30500130133793Total1196421751811117* Cohen’s kappa coefficient: 0.381Figure 1.Prevalence of LTBI in different underlying R-IMIDLTBI: Latent tuberculosis infection, PsA: Psoriatic arthritis, RA: Rheumatoid arthritis, SpA: Axial spondyloarthritis.Diagnosis of LTBI: Positive TST(+booster) and/or IGRA test.Disclosure of Interests:David Martínez-López: None declared, Joy Osorio-Chavez: None declared, Carmen Álvarez-Reguera: None declared, Virginia Portilla: None declared, Miguel A González-Gay Speakers bureau: Abbvie, Pfizer, Roche, Sanofi and MSD, Consultant of: Abbvie, Pfizer, Roche, Sanofi and MSD, Grant/research support from: Abbvie, MSD, Jansen and Roche, Ricardo Blanco Speakers bureau: Abbvie, Pfizer, Roche, Bristol-Myers, Janssen, Lilly and MSD, Consultant of: Abbvie, Pfizer, Roche, Bristol-Myers, Janssen, Lilly and MSD, Grant/research support from: Abbvie, MSD, and Roche

scholarly journals Lung MRI to predict lack of response to treatment in interstitial lung disease: initial observations on SSFSE/PROPELLER T2 mismatch

2020 ◽  
Author(s):  
Wagner Diniz de Paula ◽  
Marcelo Palmeira Rodrigues ◽  
Nathali Mireise Costa Ferreira ◽  
Viviane Vieira Passini ◽  
César Augusto Melo e Silva
Keyword(s):  
Interstitial Lung Disease ◽  
Lung Disease ◽  
Kappa Coefficient ◽  
Response To Treatment ◽  
Rater Agreement ◽  
Cohen’S Kappa ◽  
Relative Signal Intensity ◽  
Cohen's Kappa ◽  
Lung Mri ◽  
Cohen’S Kappa Coefficient

Abstract BackgroundHigh-resolution chest computed tomography (HRCT) signs of interstitial lung disease (ILD) are varied, some corresponding to irreparable parenchymal destruction and fibrosis, others representing potentially reversible changes, such as fine reticulation and ground-glass opacities (GGO). GGO frequently correspond to sites of active inflammation that may be responsive to steroids or immunosuppressive agents, but they might also represent intralobular interstitial fibrosis not resolved by current HRCT technique. Our aim was to investigate the ability of lung MRI to predict treatment response in individuals with ILD presenting with predominant GGO.MethodsIn this prospective cohort, 15 participants (4 male and 11 female) aged 38–84 years, presenting with ILD manifested as predominant GGO and referred for a new treatment regimen with a systemic glucocorticoid and/or an immunosuppressive agent, underwent 1.5 T lung MRI with breath-hold (SSFSE) and respiratory-gated (PROPELLER) T2-weighted pulse sequences, and with dynamic contrast-enhanced fat-suppressed T1-weighted pulse sequence (LAVA). Relative signal intensity on T2-weighted images and relative enhancement of lung lesions were compared to functional response in a dichotomous fashion (response versus non-response) with t test for independent samples. SSFSE/PROPELLER T2 mismatch was compared to response with Fisher’s exact test. Inter-rater agreement was evaluated with Cohen’s kappa coefficient. The primary endpoint for response was a greater than 10% increase in forced vital capacity in 10 weeks.ResultsResponders (4/15, 27%) and non-responders (11/15, 73%) showed similar relative signal intensity on T2-weighted images and relative enhancement measurements. SSFSE/PROPELLER T2 mismatch was able to discriminate responders from non-responders in 12 of 15 participants (80% accuracy, p = 0.026) for readers 1 and 2, and in 13 of 15 participants (87% accuracy, p = 0.011) for reader 3, with inter-rater agreement of 87% between readers 1 and 2 (Cohen’s kappa coefficient of 0.732) and 93% between readers 1/2 and 3 (Cohen’s kappa coefficient of 0.865).ConclusionsSSFSE-PROPELLER T2 mismatch was predictive of lack of response to treatment in this small group of ILD patients presenting with predominant GGO at HRCT.Key PointSSFSE/PROPELLER T2 mismatch may help predict lack of response to anti-inflammatory/immunosuppressive treatment in interstitial lung disease, with high accuracy and high inter-rater agreement.

scholarly journals A Comparative Clinical Diagnostic Examination of Deceased Poisoned Patients in Baharloo Hospital Through Autopsy and Toxicology at Tehran’s MedicoLegal Organization During 2014-2015

2019 ◽  
Author(s):  
Behnam Behnoush ◽  
Elham Bazmi ◽  
Zahra Aghili Tekye
Keyword(s):  
Clinical Diagnosis ◽  
Causes Of Death ◽  
Acute Poisoning ◽  
Kappa Coefficient ◽  
High Dose ◽  
Cohen’S Kappa ◽  
Hospital Records ◽  
Cohen's Kappa ◽  
Common Causes ◽  
Cohen’S Kappa Coefficient

Abstract- Acute poisoning is nowadays a common medical emergency. The important point about admitting patients with acute poisoning is its timely and early diagnosis. However, the rate of consistency between clinical diagnosis and autopsy of patients with acute lethal poisoning has not been investigated broadly. The goal of this study was to compare the clinical symptoms of poisoned deceased patients through autopsy and toxicology at Tehran’s Medico-Legal Organization. A cross-sectional study was conducted on all the admitted poisoning cases at Baharloo Hospital in 2014 to 2015 who passed away and were referred to Kahrizak Autopsy Hall. Through investigating the medical records of the Hospital and Autopsy Hall, the researcher examined and compared the existent files. The collected data were entered into SPSS 16. Data were analyzed using Cohen’s kappa coefficient. In all, 98 people who had died from acute poisoning were investigated. The most common causes of poisoning were drugs (49%), followed by pesticides (39%). The least common cause of poisoning was alcohol (1%). The most common poisonous drugs were opioids. The most common causes of death based on hospital records and autopsy results were rice tablets and high-dose opioid poisoning. Using Cohen’s kappa coefficient, the rate of agreement between the causes of death in hospital records and autopsy results was estimated at 0.744. Bearing in mind the consistency of clinical diagnosis with postmortem autopsy results, we may begin life-saving management of patients of acute poisoning in the shortest possible time

Sign in / Sign up

Export Citation Format

Share Document