scholarly journals Exploring Symmetry of Binary Classification Performance Metrics

Symmetry ◽  
2019 ◽  
Vol 11 (1) ◽  
pp. 47 ◽  
Author(s):  
Amalia Luque ◽  
Alejandro Carrasco ◽  
Alejandro Martín ◽  
Juan Ramón Lama
Keyword(s):  
Performance Metrics ◽  
Binary Classification ◽  
Confusion Matrix ◽  
Full Range ◽  
Classification Performance ◽  
Classification Problems ◽  
Performance Metric ◽  
Selection For ◽  
Proper Performance ◽  
Insight Into

Selecting the proper performance metric constitutes a key issue for most classification problems in the field of machine learning. Although the specialized literature has addressed several topics regarding these metrics, their symmetries have yet to be systematically studied. This research focuses on ten metrics based on a binary confusion matrix and their symmetric behaviour is formally defined under all types of transformations. Through simulated experiments, which cover the full range of datasets and classification results, the symmetric behaviour of these metrics is explored by exposing them to hundreds of simple or combined symmetric transformations. Cross-symmetries among the metrics and statistical symmetries are also explored. The results obtained show that, in all cases, three and only three types of symmetries arise: labelling inversion (between positive and negative classes); scoring inversion (concerning good and bad classifiers); and the combination of these two inversions. Additionally, certain metrics have been shown to be independent of the imbalance in the dataset and two cross-symmetries have been identified. The results regarding their symmetries reveal a deeper insight into the behaviour of various performance metrics and offer an indicator to properly interpret their values and a guide for their selection for certain specific applications.

scholarly journals Classifier uncertainty: evidence, potential impact, and probabilistic treatment

2021 ◽  
Vol 7 ◽  
pp. e398
Author(s):  
Niklas Tötsch ◽  
Daniel Hoffmann
Keyword(s):  
Performance Metrics ◽  
Scientific Literature ◽  
Probability Model ◽  
Confusion Matrix ◽  
Classification Performance ◽  
Small Data ◽  
Data Sets ◽  
Performance Metric ◽  
Small Data Sets ◽  
Potential Impact

Classifiers are often tested on relatively small data sets, which should lead to uncertain performance metrics. Nevertheless, these metrics are usually taken at face value. We present an approach to quantify the uncertainty of classification performance metrics, based on a probability model of the confusion matrix. Application of our approach to classifiers from the scientific literature and a classification competition shows that uncertainties can be surprisingly large and limit performance evaluation. In fact, some published classifiers may be misleading. The application of our approach is simple and requires only the confusion matrix. It is agnostic of the underlying classifier. Our method can also be used for the estimation of sample sizes that achieve a desired precision of a performance metric.

scholarly journals The impact of class imbalance in classification performance metrics based on the binary confusion matrix

2019 ◽  
Vol 91 ◽  
pp. 216-231 ◽  
Author(s):  
Amalia Luque ◽  
Alejandro Carrasco ◽  
Alejandro Martín ◽  
Ana de las Heras
Keyword(s):  
Performance Metrics ◽  
Confusion Matrix ◽  
Class Imbalance ◽  
Classification Performance ◽  
The Impact

scholarly journals A new learning algorithm based on strengthening boundary samples for convolutional neural networks

2020 ◽  
Vol 327 ◽  
pp. 02004
Author(s):  
Dongning Zhou ◽  
Lu Lu ◽  
Junhong Zhao ◽  
Dali Wang ◽  
Wenlian Lu ◽  
...  
Keyword(s):  
Learning Algorithm ◽  
Binary Classification ◽  
Classification Performance ◽  
Data Sets ◽  
Classification Problems ◽  
Data Set ◽  
Penalty Term ◽  
Classification Boundary ◽  
Multi Classification ◽  
Hard Samples

CNN is an artificial neural network that can automatically extract features with relatively few parameters, which is the advantage of CNN in image classification tasks. The purpose of this paper is to propose a new algorithm to improve the classification performance of CNN by strengthening boundary samples. The samples with predicted values near the classification boundary are recorded as hard samples. In this algorithm, the errors of hard samples are added as a penalty term of the original loss function. Multi-classification and binary classification experiments were performed using the MNIST data set and three sub-data sets of CIFAR-10, respectively. The experimental results prove that the accuracy of the new algorithm is improved in both binary classification and multi-classification problems.

Genetic Algorithm Based Feature Subset Selection for Fetal State Classification

2015 ◽  
Vol 2 ◽  
pp. 13 ◽  
Author(s):  
Subha Velappan ◽  
Murugan D ◽  
Prabha S ◽  
Manivanna Boopathi A
Keyword(s):  
Genetic Algorithm ◽  
Performance Metrics ◽  
Classification Performance ◽  
Feature Subset Selection ◽  
Support Vector ◽  
Feature Subset ◽  
Huge Amount ◽  
State Classification ◽  
Multiclass Support Vector Machine ◽  
Selection For

Huge amount of data are available in the field of medicine which are used for diagnosing the diseases by analyzing them. Presently, prediction of diseases are made easier and accurate by employing various data mining techniques to extract information from these medical data. This paper presents an improved method of classifying the cardiotocogram (CTG) data using Multiclass Support Vector Machine (MSVM) through an optimized feature subset produced by Genetic Algorithm (GA). Various performance metrics have been evaluated and the experimental results exhibit improved classification performance when using optimized feature set comparing to the full feature set.

Dynamic Fusion and Parameter Optimization of Multiple Classifier Systems

2008 ◽  
Author(s):  
Anuradha Kodali ◽  
William Donat ◽  
Satnam Singh ◽  
Kihoon Choi ◽  
Krishna Pattipati
Keyword(s):  
Performance Metrics ◽  
Time Window ◽  
Binary Classification ◽  
Diagnostic Errors ◽  
Probability Of Detection ◽  
Multiple Classifier Systems ◽  
Classification Problems ◽  
Classifier Systems ◽  
Primal Dual ◽  
Over Time

We propose a fusion architecture that combines a set of classifier decisions over a time window to isolate dynamically evolving faults in gas turbine engines. The dynamic fusion problem is formulated as a maximum a posteriori decision problem of inferring the fault sequence based on uncertain outcomes of multiple classifiers over time. The resulting problem is solved via a primal-dual optimization framework. The fusion process involves three steps: the first step transforms the multi-class problem into dichotomies using error correcting output codes (ECOC) and thus solving the concomitant binary classification problems; the second step fuses the outcomes of multiple binary classifiers over time using a sliding window dynamic fusion method that exploits temporal data correlations over time. The window size provides a trade-off between diagnostic errors and decision delays. The third step optimizes the fusion parameters using a genetic algorithm. The probability of detection and false alarm probability of each classifier are the fusion parameters; these probabilities are jointly optimized as part of the fusion architecture instead of optimizing the parameters of each classifier separately. The proposed algorithm is demonstrated by computing the diagnostic performance metrics on a twin-spool commercial jet engine data.

scholarly journals Analysis of multi-class classification performance metrics for remote sensing imagery imbalanced datasets

2021 ◽  
pp. 11-17
Author(s):  
Andrea González-Ramírez ◽  
Josué Lopez ◽  
Deni Torres ◽  
Israel Yañez-Vargas
Keyword(s):  
Remote Sensing ◽  
Performance Evaluation ◽  
Performance Metrics ◽  
Matthews Correlation Coefficient ◽  
Confusion Matrix ◽  
Classification Performance ◽  
Support Vector ◽  
Multi Class Classification

Remote sensing imaging datasets for classification generally present high levels of imbalance between classes of interest. This work presented a study of a set of performance evaluation metrics for an imbalance dataset. In this work, a support vector machine (SVM) was used to perform the classification of seven classes of interest in a popular dataset called Salinas-A. The performance evaluation of the classifier was performed using two types of metrics: 1) Metrics for multi-class classification, and 2) Metrics based on the binary confusion matrix. In the results, a comparison of the scores of each metric is developed, some being more optimistic than others due to the bias that they present given the imbalance. In addition, our case study helps to conclude that the Matthews correlation coefficient (MCC) presents the lowest bias in imbalanced cases and is regarded to be robust metric. These results can be extended to any imbalanced dataset taking into account the equations developed by Luque.

scholarly journals Emphasis Learning (EL), Feature Repetition in Width Instead of Length, to Obtain Better Classification Performance; Case Study: Alzheimer’s Disease Diagnosis

2019 ◽  
Author(s):  
Hamid Akramifard ◽  
MohammadAli Balafar ◽  
SeyedNaser Razavi ◽  
Abd Rahman Ramli
Keyword(s):  
Cognitive Impairment ◽  
Mild Cognitive Impairment ◽  
Binary Classification ◽  
Disease Diagnosis ◽  
Classification Performance ◽  
Classification Problems ◽  
Computer Aided ◽  
Binary Problems ◽  
Do So

A method for classification is introduced in this article, and it is tested on ADNI database to diagnose alzheimer’s disease (AD). It is obvious that tunning the performance of a classification to get better results is a complicated problem, and when we want model’s accuracy or other peformance measurments higher than 90%, the problem will be more complicated. In this study, we tried and succeeded to discover a method to solve this problem. The final feature set can be used clustering too, because outgrowth feature set of the proposed method is invigorated. In the recent years, a lot of activities is done to develop computer aided systems (CAD) for alzheimer’s disease diagnosis. Most of these recently developed systems concenterated on extracting and combining features from MRI, PET, CSF, and …; in this article, we made attempt to do so and utilized one more technique to increase classification performance. Finding and producing the best features to solve three binary classification problems of AD vs. Normal Control (NC), Mild Cognitive Impairment (MCI) vs. NC, and MCI vs. AD are the purposes of this article. Experiments indicate performance and effectiveness rates of the proposed method, which are accuracies of 98.81%, 81.61%, and 81.40% for AD vs. NC, MCI vs. NC, and AD vs. MCI classification problems, respectively. As can be seen, using this method increased the performance of the three binary problems incredibly.

scholarly journals Multiclass Confusion Matrix Reduction Method and Its Application on Net Promoter Score Classification Problem

Technologies ◽  
2021 ◽  
Vol 9 (4) ◽  
pp. 81
Author(s):  
Ioannis Markoulidakis ◽  
Ioannis Rallis ◽  
Ioannis Georgoulas ◽  
George Kopsiaftis ◽  
Anastasios Doulamis ◽  
...  
Keyword(s):  
Performance Metrics ◽  
Reduction Method ◽  
Confusion Matrix ◽  
Area Under The Curve ◽  
Classification Problem ◽  
Machine Learning Algorithms ◽  
Classification Problems ◽  
Net Promoter Score ◽  
Novel Method ◽  
Net Promoter

The current paper presents a novel method for reducing a multiclass confusion matrix into a 2×2 version enabling the exploitation of the relevant performance metrics and methods such as the receiver operating characteristic and area under the curve for the assessment of different classification algorithms. The reduction method is based on class grouping and leads to a special type of matrix called the reduced confusion matrix. The developed method is then exploited for the assessment of state of the art machine learning algorithms applied on the net promoter score classification problem in the field of customer experience analytics indicating the value of the proposed method in real world classification problems.

Sign in / Sign up

Export Citation Format

Share Document