scholarly journals Episodix: a serious game to detect cognitive impairment in senior adults. A psychometric study

PeerJ ◽  
2018 ◽  
Vol 6 ◽  
pp. e5478 ◽  
Author(s):  
Sonia Valladares-Rodriguez ◽  
Manuel J. Fernández-Iglesias ◽  
Luis Anido-Rifón ◽  
David Facal ◽  
Roberto Pérez-Rodríguez
Keyword(s):  
Machine Learning ◽  
Cognitive Impairment ◽  
Performance Metrics ◽  
Cognitive Impairments ◽  
Senior Adults ◽  
Cohen’S Kappa ◽  
Machine Learning Classification ◽  
Cohen's Kappa ◽  
Preliminary Study ◽  
Psychometric Validity

Introduction Assessment of episodic memory is traditionally used to evaluate potential cognitive impairments in senior adults. The present article discusses the capabilities of Episodix, a game to assess the aforementioned cognitive area, as a valid tool to discriminate among mild cognitive impairment (MCI), Alzheimer’s disease (AD) and healthy individuals (HC); that is, it studies the game’s psychometric validity study to assess cognitive impairment. Materials and Methods After a preliminary study, a new pilot study, statistically significant for the Galician population, was carried out from a cross-sectional sample of senior adults as target users. A total of 64 individuals (28 HC, 16 MCI, 20 AD) completed the experiment from an initial sample of 74. Participants were administered a collection of classical pen-and-paper tests and interacted with the games developed. A total of six machine learning classification techniques were applied and four relevant performance metrics were computed to assess the classification power of the tool according to participants’ cognitive status. Results According to the classification performance metrics computed, the best classification result is obtained using the Extra Trees Classifier (F1 = 0.97 and Cohen’s kappa coefficient = 0.97). Precision and recall values are also high, above 0.9 for all cognitive groups. Moreover, according to the standard interpretation of Cohen’s kappa index, classification is almost perfect (i.e., 0.81–1.00) for the complete dataset for all algorithms. Limitations Weaknesses (e.g., accessibility, sample size or speed of stimuli) detected during the preliminary study were addressed and solved. Nevertheless, additional research is needed to improve the resolution of the game for the identification of specific cognitive impairments, as well as to achieve a complete validation of the psychometric properties of the digital game. Conclusion Promising results obtained about psychometric validity of Episodix, represent a relevant step ahead towards the introduction of serious games and machine learning in regular clinical practice for detecting MCI or AD. However, more research is needed to explore the introduction of item response theory in this game and to obtain the required normative data for clinical validity.

Neuropsychiatric symptoms as predictors of conversion from MCI to dementia: a machine learning approach

2019 ◽  
Vol 32 (3) ◽  
pp. 381-392 ◽  
Author(s):  
Sabela C. Mallo ◽  
Sonia Valladares-Rodriguez ◽  
David Facal ◽  
Cristina Lojo-Seoane ◽  
Manuel J. Fernández-Iglesias ◽  
...  
Keyword(s):  
Machine Learning ◽  
Severity Score ◽  
Performance Metrics ◽  
Neuropsychiatric Symptoms ◽  
Depression Scale ◽  
Forest Plot ◽  
Total Stress ◽  
Cohen’S Kappa ◽  
Cohen's Kappa ◽  
Stress Score

ABSTRACTObjectives:To use a Machine Learning (ML) approach to compare Neuropsychiatric Symptoms (NPS) in participants of a longitudinal study who developed dementia and those who did not.Design:Mann-Whitney U and ML analysis. Nine ML algorithms were evaluated using a 10-fold stratified validation procedure. Performance metrics (accuracy, recall, F-1 score, and Cohen’s kappa) were computed for each algorithm, and graphic metrics (ROC and precision-recall curves) and features analysis were computed for the best-performing algorithm.Setting:Primary care health centers.Participants:128 participants: 78 cognitively unimpaired and 50 with MCI.Measurements:Diagnosis at baseline, months from the baseline assessment until the 3rd follow-up or development of dementia, gender, age, Charlson Comorbidity Index, Neuropsychiatric Inventory-Questionnaire (NPI-Q) individual items, NPI-Q total severity, and total stress score and Geriatric Depression Scale-15 items (GDS-15) total score.Results:30 participants developed dementia, while 98 did not. Most of the participants who developed dementia were diagnosed at baseline with amnestic multidomain MCI. The Random Forest Plot model provided the metrics that best predicted conversion to dementia (e.g. accuracy=.88, F1=.67, and Cohen’s kappa=.63). The algorithm indicated the importance of the metrics, in the following (decreasing) order: months from first assessment, age, the diagnostic group at baseline, total NPI-Q severity score, total NPI-Q stress score, and GDS-15 total score.Conclusions:ML is a valuable technique for detecting the risk of conversion to dementia in MCI patients. Some NPS proxies, including NPI-Q total severity score, NPI-Q total stress score, and GDS-15 total score, were deemed as the most important variables for predicting conversion, adding further support to the hypothesis that some NPS are associated with a higher risk of dementia in MCI.

scholarly journals Utility of MemTrax and Machine Learning Modeling in Classification of Mild Cognitive Impairment

2020 ◽  
Vol 77 (4) ◽  
pp. 1545-1558
Author(s):  
Michael F. Bergeron ◽  
Sara Landset ◽  
Xianbo Zhou ◽  
Tao Ding ◽  
Taghi M. Khoshgoftaar ◽  
...  
Keyword(s):  
Machine Learning ◽  
Cognitive Impairment ◽  
Mild Cognitive Impairment ◽  
Performance Metrics ◽  
Early Stage ◽  
Characteristic Curve ◽  
Classification Performance ◽  
Cognitive Screening ◽  
Cross Sectional ◽  
Machine Learning Classification

Background: The widespread incidence and prevalence of Alzheimer’s disease and mild cognitive impairment (MCI) has prompted an urgent call for research to validate early detection cognitive screening and assessment. Objective: Our primary research aim was to determine if selected MemTrax performance metrics and relevant demographics and health profile characteristics can be effectively utilized in predictive models developed with machine learning to classify cognitive health (normal versus MCI), as would be indicated by the Montreal Cognitive Assessment (MoCA). Methods: We conducted a cross-sectional study on 259 neurology, memory clinic, and internal medicine adult patients recruited from two hospitals in China. Each patient was given the Chinese-language MoCA and self-administered the continuous recognition MemTrax online episodic memory test on the same day. Predictive classification models were built using machine learning with 10-fold cross validation, and model performance was measured using Area Under the Receiver Operating Characteristic Curve (AUC). Models were built using two MemTrax performance metrics (percent correct, response time), along with the eight common demographic and personal history features. Results: Comparing the learners across selected combinations of MoCA scores and thresholds, Naïve Bayes was generally the top-performing learner with an overall classification performance of 0.9093. Further, among the top three learners, MemTrax-based classification performance overall was superior using just the top-ranked four features (0.9119) compared to using all 10 common features (0.8999). Conclusion: MemTrax performance can be effectively utilized in a machine learning classification predictive model screening application for detecting early stage cognitive impairment.

scholarly journals Prediction of Streptococcus uberis clinical mastitis treatment success in dairy herds by means of mass spectrometry and machine-learning

2021 ◽  
Vol 11 (1) ◽  
Author(s):  
Alexandre Maciel-Guerra ◽  
Necati Esener ◽  
Katharina Giebel ◽  
Daniel Lea ◽  
Martin J. Green ◽  
...  
Keyword(s):  
Machine Learning ◽  
Mass Spectrometry ◽  
Treatment Success ◽  
Clinical Mastitis ◽  
Supervised Machine Learning ◽  
Structural Protein ◽  
Cohen’S Kappa ◽  
Maldi Tof ◽  
Streptococcus Uberis ◽  
Cohen's Kappa

AbstractStreptococcus uberis is one of the leading pathogens causing mastitis worldwide. Identification of S. uberis strains that fail to respond to treatment with antibiotics is essential for better decision making and treatment selection. We demonstrate that the combination of supervised machine learning and matrix-assisted laser desorption ionization/time of flight (MALDI-TOF) mass spectrometry can discriminate strains of S. uberis causing clinical mastitis that are likely to be responsive or unresponsive to treatment. Diagnostics prediction systems trained on 90 individuals from 26 different farms achieved up to 86.2% and 71.5% in terms of accuracy and Cohen’s kappa. The performance was further increased by adding metadata (parity, somatic cell count of previous lactation and count of positive mastitis cases) to encoded MALDI-TOF spectra, which increased accuracy and Cohen’s kappa to 92.2% and 84.1% respectively. A computational framework integrating protein–protein networks and structural protein information to the machine learning results unveiled the molecular determinants underlying the responsive and unresponsive phenotypes.

scholarly journals Patient-Specific Explanations for Predictions of Clinical Outcomes

ACI Open ◽  
2019 ◽  
Vol 03 (02) ◽  
pp. e88-e97
Author(s):  
Mohammadamin Tajgardoon ◽  
Malarkodi J. Samayamuthu ◽  
Luca Calzoni ◽  
Shyam Visweswaran
Keyword(s):  
Machine Learning ◽  
Clinical Outcomes ◽  
Characteristic Curve ◽  
Community Acquired Pneumonia ◽  
Kappa Coefficient ◽  
Patient Specific ◽  
P Value ◽  
Cohen’S Kappa ◽  
Cohen's Kappa ◽  
Cohen’S Kappa Coefficient

Abstract Background Machine learning models that are used for predicting clinical outcomes can be made more useful by augmenting predictions with simple and reliable patient-specific explanations for each prediction. Objectives This article evaluates the quality of explanations of predictions using physician reviewers. The predictions are obtained from a machine learning model that is developed to predict dire outcomes (severe complications including death) in patients with community acquired pneumonia (CAP). Methods Using a dataset of patients diagnosed with CAP, we developed a predictive model to predict dire outcomes. On a set of 40 patients, who were predicted to be either at very high risk or at very low risk of developing a dire outcome, we applied an explanation method to generate patient-specific explanations. Three physician reviewers independently evaluated each explanatory feature in the context of the patient's data and were instructed to disagree with a feature if they did not agree with the magnitude of support, the direction of support (supportive versus contradictory), or both. Results The model used for generating predictions achieved a F1 score of 0.43 and area under the receiver operating characteristic curve (AUROC) of 0.84 (95% confidence interval [CI]: 0.81–0.87). Interreviewer agreement between two reviewers was strong (Cohen's kappa coefficient = 0.87) and fair to moderate between the third reviewer and others (Cohen's kappa coefficient = 0.49 and 0.33). Agreement rates between reviewers and generated explanations—defined as the proportion of explanatory features with which majority of reviewers agreed—were 0.78 for actual explanations and 0.52 for fabricated explanations, and the difference between the two agreement rates was statistically significant (Chi-square = 19.76, p-value < 0.01). Conclusion There was good agreement among physician reviewers on patient-specific explanations that were generated to augment predictions of clinical outcomes. Such explanations can be useful in interpreting predictions of clinical outcomes.

scholarly journals Performance Evaluation of Different Machine Learning Classification Algorithms for Disease Diagnosis

2021 ◽  
Vol 12 (6) ◽  
pp. 1-28
Author(s):  
Munder Abdulatef Al-Hashem ◽  
Ali Mohammad Alqudah ◽  
Qasem Qananwah
Keyword(s):  
Machine Learning ◽  
Nearest Neighbor ◽  
Performance Metrics ◽  
Confusion Matrix ◽  
Learning Algorithms ◽  
Disease Diagnosis ◽  
Machine Learning Algorithms ◽  
Classification Algorithms ◽  
K Nearest Neighbor ◽  
Machine Learning Classification

Knowledge extraction within a healthcare field is a very challenging task since we are having many problems such as noise and imbalanced datasets. They are obtained from clinical studies where uncertainty and variability are popular. Lately, a wide number of machine learning algorithms are considered and evaluated to check their validity of being used in the medical field. Usually, the classification algorithms are compared against medical experts who are specialized in certain disease diagnoses and provide an effective methodological evaluation of classifiers by applying performance metrics. The performance metrics contain four criteria: accuracy, sensitivity, and specificity forming the confusion matrix of each used algorithm. We have utilized eight different well-known machine learning algorithms to evaluate their performances in six different medical datasets. Based on the experimental results we conclude that the XGBoost and K-Nearest Neighbor classifiers were the best overall among the used datasets and signs can be used for diagnosing various diseases.

scholarly journals Mass spectrometry and machine learning for the accurate diagnosis of benzylpenicillin and multidrug resistance of Staphylococcus aureus in bovine mastitis

2021 ◽  
Vol 17 (6) ◽  
pp. e1009108
Author(s):  
Necati Esener ◽  
Alexandre Maciel Guerra ◽  
Katharina Giebel ◽  
Daniel Lea ◽  
Martin J. Green ◽  
...  
Keyword(s):  
Neural Network ◽  
Machine Learning ◽  
Mass Spectrometry ◽  
Staphylococcus Aureus ◽  
Logistic Regression ◽  
Bovine Mastitis ◽  
Cohen’S Kappa ◽  
Mlp Neural Network ◽  
Cohen's Kappa ◽  
Linear Svm

Staphylococcus aureus is a serious human and animal pathogen threat exhibiting extraordinary capacity for acquiring new antibiotic resistance traits in the pathogen population worldwide. The development of fast, affordable and effective diagnostic solutions capable of discriminating between antibiotic-resistant and susceptible S. aureus strains would be of huge benefit for effective disease detection and treatment. Here we develop a diagnostics solution that uses Matrix-Assisted Laser Desorption/Ionisation–Time of Flight Mass Spectrometry (MALDI-TOF) and machine learning, to identify signature profiles of antibiotic resistance to either multidrug or benzylpenicillin in S. aureus isolates. Using ten different supervised learning techniques, we have analysed a set of 82 S. aureus isolates collected from 67 cows diagnosed with bovine mastitis across 24 farms. For the multidrug phenotyping analysis, LDA, linear SVM, RBF SVM, logistic regression, naïve Bayes, MLP neural network and QDA had Cohen’s kappa values over 85.00%. For the benzylpenicillin phenotyping analysis, RBF SVM, MLP neural network, naïve Bayes, logistic regression, linear SVM, QDA, LDA, and random forests had Cohen’s kappa values over 85.00%. For the benzylpenicillin the diagnostic systems achieved up to (mean result ± standard deviation over 30 runs on the test set): accuracy = 97.54% ± 1.91%, sensitivity = 99.93% ± 0.25%, specificity = 95.04% ± 3.83%, and Cohen’s kappa = 95.04% ± 3.83%. Moreover, the diagnostic platform complemented by a protein-protein network and 3D structural protein information framework allowed the identification of five molecular determinants underlying the susceptible and resistant profiles. Four proteins were able to classify multidrug-resistant and susceptible strains with 96.81% ± 0.43% accuracy. Five proteins, including the previous four, were able to classify benzylpenicillin resistant and susceptible strains with 97.54% ± 1.91% accuracy. Our approach may open up new avenues for the development of a fast, affordable and effective day-to-day diagnostic solution, which would offer new opportunities for targeting resistant bacteria.

scholarly journals Automatic Detection of Cognitive Impairments through Acoustic Analysis of Speech

2020 ◽  
Vol 17 (1) ◽  
pp. 60-68 ◽  
Author(s):  
Ryosuke Nagumo ◽  
Yaming Zhang ◽  
Yuki Ogawa ◽  
Mitsuharu Hosokawa ◽  
Kengo Abe ◽  
...  
Keyword(s):  
Machine Learning ◽  
Cognitive Impairment ◽  
Mild Cognitive Impairment ◽  
Acoustic Analysis ◽  
Cognitive Impairments ◽  
Learning Model ◽  
Community Dwelling ◽  
Healthy Controls ◽  
Acoustic Features ◽  
Machine Learning Model

Background: Early detection of mild cognitive impairment is crucial in the prevention of Alzheimer’s disease. The aim of the present study was to identify whether acoustic features can help differentiate older, independent community-dwelling individuals with cognitive impairment from healthy controls. Methods: A total of 8779 participants (mean age 74.2 ± 5.7 in the range of 65-96, 3907 males and 4872 females) with different cognitive profiles, namely healthy controls, mild cognitive impairment, global cognitive impairment (defined as a Mini Mental State Examination score of 20-23), and mild cognitive impairment with global cognitive impairment (a combined status of mild cognitive impairment and global cognitive impairment), were evaluated in short-sentence reading tasks, and their acoustic features, including temporal features (such as duration of utterance, number and length of pauses) and spectral features (F0, F1, and F2), were used to build a machine learning model to predict their cognitive impairments. Results: The classification metrics from the healthy controls were evaluated through the area under the receiver operating characteristic curve and were found to be 0.61, 0.67, and 0.77 for mild cognitive impairment, global cognitive impairment, and mild cognitive impairment with global cognitive impairment, respectively. Conclusion: Our machine learning model revealed that individuals’ acoustic features can be employed to discriminate between healthy controls and those with mild cognitive impairment with global cognitive impairment, which is a more severe form of cognitive impairment compared with mild cognitive impairment or global cognitive impairment alone. It is suggested that language impairment increases in severity with cognitive impairment.

scholarly journals Multi-Level Comparison of Machine Learning Classifiers and Their Performance Metrics

Molecules ◽  
2019 ◽  
Vol 24 (15) ◽  
pp. 2811 ◽  
Author(s):  
Rácz ◽  
Bajusz ◽  
Héberger
Keyword(s):  
Machine Learning ◽  
Performance Metrics ◽  
Learning Algorithm ◽  
Classification Problems ◽  
Machine Learning Classification ◽  
Learning Tasks ◽  
Sum Of Ranking Differences ◽  
Multi Level

Machine learning classification algorithms are widely used for the prediction and classification of the different properties of molecules such as toxicity or biological activity. the prediction of toxic vs. non-toxic molecules is important due to testing on living animals, which has ethical and cost drawbacks as well. The quality of classification models can be determined with several performance parameters. which often give conflicting results. In this study, we performed a multi-level comparison with the use of different performance metrics and machine learning classification methods. Well-established and standardized protocols for the machine learning tasks were used in each case. The comparison was applied to three datasets (acute and aquatic toxicities) and the robust, yet sensitive, sum of ranking differences (SRD) and analysis of variance (ANOVA) were applied for evaluation. The effect of dataset composition (balanced vs. imbalanced) and 2-class vs. multiclass classification scenarios was also studied. Most of the performance metrics are sensitive to dataset composition, especially in 2-class classification problems. The optimal machine learning algorithm also depends significantly on the composition of the dataset.

Sign in / Sign up

Export Citation Format

Share Document