The Complexity of a Probabilistic Approach to Deal with Missing Values in a Decision Tree

2006 ◽  
Author(s):  
Lamis Hawarah ◽  
Ana Simonet ◽  
Michel Simonet
Keyword(s):  
Decision Tree ◽  
Missing Values ◽  
Probabilistic Approach

scholarly journals Accuracy of probabilistic and deterministic record linkage: the case of tuberculosis

2016 ◽  
Vol 50 (0) ◽  
Author(s):  
Gisele Pinto de Oliveira ◽  
Ana Luiza de Souza Bierrenbach ◽  
Kenneth Rochel de Camargo Júnior ◽  
Cláudia Medina Coeli ◽  
Rejane Sobrino Pinheiro
Keyword(s):  
Record Linkage ◽  
Missing Values ◽  
Probabilistic Approach ◽  
Roc Curves ◽  
Cutoff Point ◽  
Accurate Analysis ◽  
Linkage Algorithm ◽  
Probabilistic Linkage ◽  
Key Variables ◽  
High Level

ABSTRACT OBJECTIVE To analyze the accuracy of deterministic and probabilistic record linkage to identify TB duplicate records, as well as the characteristics of discordant pairs. METHODS The study analyzed all TB records from 2009 to 2011 in the state of Rio de Janeiro. A deterministic record linkage algorithm was developed using a set of 70 rules, based on the combination of fragments of the key variables with or without modification (Soundex or substring). Each rule was formed by three or more fragments. The probabilistic approach required a cutoff point for the score, above which the links would be automatically classified as belonging to the same individual. The cutoff point was obtained by linkage of the Notifiable Diseases Information System – Tuberculosis database with itself, subsequent manual review and ROC curves and precision-recall. Sensitivity and specificity for accurate analysis were calculated. RESULTS Accuracy ranged from 87.2% to 95.2% for sensitivity and 99.8% to 99.9% for specificity for probabilistic and deterministic record linkage, respectively. The occurrence of missing values for the key variables and the low percentage of similarity measure for name and date of birth were mainly responsible for the failure to identify records of the same individual with the techniques used. CONCLUSIONS The two techniques showed a high level of correlation for pair classification. Although deterministic linkage identified more duplicate records than probabilistic linkage, the latter retrieved records not identified by the former. User need and experience should be considered when choosing the best technique to be used.

scholarly journals A Contemporary Machine Learning Method for Accurate Prediction of Cervical Cancer

2021 ◽  
Vol 102 ◽  
pp. 04004
Author(s):  
Jesse Jeremiah Tanimu ◽  
Mohamed Hamada ◽  
Mohammed Hassan ◽  
Saratu Yusuf Ilu
Keyword(s):  
Machine Learning ◽  
Cervical Cancer ◽  
Feature Selection ◽  
Decision Tree ◽  
Sensitivity And Specificity ◽  
Missing Values ◽  
New Technologies ◽  
Machine Learning Techniques ◽  
Screening Tests ◽  
Tree Classifier

With the advent of new technologies in the medical field, huge amounts of cancerous data have been collected and are readily accessible to the medical research community. Over the years, researchers have employed advanced data mining and machine learning techniques to develop better models that can analyze datasets to extract the conceived patterns, ideas, and hidden knowledge. The mined information can be used as a support in decision making for diagnostic processes. These techniques, while being able to predict future outcomes of certain diseases effectively, can discover and identify patterns and relationships between them from complex datasets. In this research, a predictive model for predicting the outcome of patients’ cervical cancer results has been developed, given risk patterns from individual medical records and preliminary screening tests. This work presents a Decision tree (DT) classification algorithm and shows the advantage of feature selection approaches in the prediction of cervical cancer using recursive feature elimination technique for dimensionality reduction for improving the accuracy, sensitivity, and specificity of the model. The dataset employed here suffers from missing values and is highly imbalanced. Therefore, a combination of under and oversampling techniques called SMOTETomek was employed. A comparative analysis of the proposed model has been performed to show the effectiveness of feature selection and class imbalance based on the classifier’s accuracy, sensitivity, and specificity. The DT with the selected features and SMOTETomek has better results with an accuracy of 98%, sensitivity of 100%, and specificity of 97%. Decision Tree classifier is shown to have excellent performance in handling classification assignment when the features are reduced, and the problem of imbalance class is addressed.

An Integrated CRO and FLANN Based Classifier for a Non-Imputed and Inconsistent Dataset

2019 ◽  
Vol 28 (03) ◽  
pp. 1950013 ◽  
Author(s):  
Ch. Sanjeev Kumar Dash ◽  
Ajit Kumar Behera ◽  
Sarat Chandra Nayak ◽  
Satchidananda Dehuri ◽  
Sung-Bae Cho
Keyword(s):  
Missing Values ◽  
Probabilistic Approach ◽  
Integrated Approach ◽  
Bench Mark ◽  
Chemical Reaction Optimization ◽  
Functional Link ◽  
Missing Value ◽  
Promising Alternative ◽  
Reaction Optimization ◽  
Selection Of

This paper presents an integrated approach by considering chemical reaction optimization (CRO) and functional link artificial neural networks (FLANNs) for building a classifier from the dataset with missing value, inconsistent records, and noisy instances. Here, imputation is carried out based on the known value of two nearest neighbors to address dataset plagued with missing values. The probabilistic approach is used to remove the inconsistency from either of the datasets like original or imputed. The resulting dataset is then given as an input to boosted instance selection approach for selection of relevant instances to reduce the size of the dataset without loss of generality and compromising classification accuracy. Finally, the transformed dataset (i.e., from non-imputed and inconsistent dataset to imputed and consistent dataset) is used for developing a classifier based on CRO trained FLANN. The method is evaluated extensively through a few bench-mark datasets obtained from University of California, Irvine (UCI) repository. The experimental results confirm that our preprocessing tasks along with integrated approach can be a promising alternative tool for mitigating missing value, inconsistent records, and noisy instances.

scholarly journals FINANCIAL FORECASTING USING DECISION TREE (REPTree & C4.5) AND NEURAL NETWORKS (K*) FOR HANDLING THE MISSING VALUES

2017 ◽  
Vol 7 (3) ◽  
pp. 1473-1477
Author(s):  
J Jayanthi ◽  
◽  
Gurpreet Kaur ◽  
K Suresh Joseph ◽  
◽  
...  
Keyword(s):  
Neural Networks ◽  
Decision Tree ◽  
Missing Values ◽  
Financial Forecasting

Development of Web Tools to Predict Axillary lymph Node Metastasis and Pathological Response to Neoadjuvant Chemotherapy in Breast Cancer Patients

2014 ◽  
Vol 29 (4) ◽  
pp. 372-379 ◽  
Author(s):  
Masahiro Sugimoto ◽  
Masahiro Takada ◽  
Masakazu Toi
Keyword(s):  
Breast Cancer ◽  
Lymph Node ◽  
Neoadjuvant Chemotherapy ◽  
Decision Tree ◽  
Cancer Patients ◽  
Axillary Lymph Node ◽  
Missing Values ◽  
Axillary Lymph ◽  
Breast Cancer Patients ◽  
Response To Neoadjuvant Chemotherapy

Nomograms are a standard computational tool to predict the likelihood of an outcome using multiple available patient features. We have developed a more powerful data mining methodology, to predict axillary lymph node (AxLN) metastasis and response to neoadjuvant chemotherapy (NAC) in primary breast cancer patients. We developed websites to use these tools. The tools calculate the probability of AxLN metastasis (AxLN model) and pathological complete response to NAC (NAC model). As a calculation algorithm, we employed a decision tree–based prediction model known as the alternative decision tree (ADTree), which is an analog development of if-then type decision trees. An ensemble technique was used to combine multiple ADTree predictions, resulting in higher generalization abilities and robustness against missing values. The AxLN model was developed with training datasets (n=148) and test datasets (n=143), and validated using an independent cohort (n=174), yielding an area under the receiver operating characteristic curve (AUC) of 0.768. The NAC model was developed and validated with n=150 and n=173 datasets from a randomized controlled trial, yielding an AUC of 0.787. AxLN and NAC models require users to input up to 17 and 16 variables, respectively. These include pathological features, including human epidermal growth factor receptor 2 (HER2) status and imaging findings. Each input variable has an option of “unknown,” to facilitate prediction for cases with missing values. The websites developed facilitate the use of these tools, and serve as a database for accumulating new datasets.

Sign in / Sign up

Export Citation Format

Share Document