scholarly journals Leveraging Expert Knowledge for Label Noise Mitigation in Machine Learning

2021 ◽  
Vol 11 (22) ◽  
pp. 11040
Author(s):  
Quoc Nguyen ◽  
Tomoaki Shikina ◽  
Daichi Teruya ◽  
Seiji Hotta ◽  
Huy-Dung Han ◽  
...  
Keyword(s):  
Machine Learning ◽  
Expert Knowledge ◽  
Classification Problem ◽  
Training Data ◽  
Noise Mitigation ◽  
Label Noise ◽  
Training Models ◽  
Gradient Descent Algorithm ◽  
Machine Learning Applications ◽  
Model Training

In training-based Machine Learning applications, the training data are frequently labeled by non-experts and expose substantial label noise which greatly alters the training models. In this work, a novel method for reducing the effect of label noise is introduced. The rules are created from expert knowledge to identify the incorrect non-expert training data. Using the gradient descent algorithm, the violating data samples are weighted less to mitigate their effects during model training. The proposed method is applied to the image classification problem using Manga109 and CIFAR-10 dataset. The experiments show that when the noise level is up to 50% our proposed method significantly increases the accuracy of the model compared to conventional learning methods.

scholarly journals Brain Activity-Based Metrics for Assessing Learning States in VR under Stress among Firefighters: An Explorative Machine Learning Approach in Neuroergonomics

2021 ◽  
Vol 11 (7) ◽  
pp. 885
Author(s):  
Maher Abujelala ◽  
Rohith Karthikeyan ◽  
Oshin Tyagi ◽  
Jing Du ◽  
Ranjana K. Mehta
Keyword(s):  
Machine Learning ◽  
Environmental Conditions ◽  
Brain Activity ◽  
Memory Task ◽  
Classification Problem ◽  
Brain Regions ◽  
Training Data ◽  
Information Encoding ◽  
Machine Learning Approach ◽  
Encoding And Retrieval

The nature of firefighters` duties requires them to work for long periods under unfavorable conditions. To perform their jobs effectively, they are required to endure long hours of extensive, stressful training. Creating such training environments is very expensive and it is difficult to guarantee trainees’ safety. In this study, firefighters are trained in a virtual environment that includes virtual perturbations such as fires, alarms, and smoke. The objective of this paper is to use machine learning methods to discern encoding and retrieval states in firefighters during a visuospatial episodic memory task and explore which regions of the brain provide suitable signals to solve this classification problem. Our results show that the Random Forest algorithm could be used to distinguish between information encoding and retrieval using features extracted from fNIRS data. Our algorithm achieved an F-1 score of 0.844 and an accuracy of 79.10% if the training and testing data are obtained at similar environmental conditions. However, the algorithm’s performance dropped to an F-1 score of 0.723 and accuracy of 60.61% when evaluated on data collected under different environmental conditions than the training data. We also found that if the training and evaluation data were recorded under the same environmental conditions, the RPM, LDLPFC, RDLPFC were the most relevant brain regions under non-stressful, stressful, and a mix of stressful and non-stressful conditions, respectively.

scholarly journals Using Machine Learning Methods Incorporating Individual Reader Annotations to Classify Paediatric Chest Radiographs in Epidemiological Studies

2021 ◽  
Vol 6 ◽  
pp. 309
Author(s):  
Paul Mwaniki ◽  
Timothy Kamanu ◽  
Samuel Akech ◽  
M. J. C Eijkemans
Keyword(s):  
Machine Learning ◽  
Epidemiological Studies ◽  
Chest Radiographs ◽  
World Health ◽  
Data Sets ◽  
Learning Models ◽  
Middle Income ◽  
Training Models ◽  
Model Training ◽  
Machine Learning Models

Introduction: Epidemiological studies that involve interpretation of chest radiographs (CXRs) suffer from inter-reader and intra-reader variability. Inter-reader and intra-reader variability hinder comparison of results from different studies or centres, which negatively affects efforts to track the burden of chest diseases or evaluate the efficacy of interventions such as vaccines. This study explores machine learning models that could standardize interpretation of CXR across studies and the utility of incorporating individual reader annotations when training models using CXR data sets annotated by multiple readers. Methods: Convolutional neural networks were used to classify CXRs from seven low to middle-income countries into five categories according to the World Health Organization's standardized methodology for interpreting paediatric CXRs. We compared models trained to predict the final/aggregate classification with models trained to predict how each reader would classify an image and then aggregate predictions for all readers using unweighted mean. Results: Incorporating individual reader's annotations during model training improved classification accuracy by 3.4% (multi-class accuracy 61% vs 59%). Model accuracy was higher for children above 12 months of age (68% vs 58%). The accuracy of the models in different countries ranged between 45% and 71%. Conclusions: Machine learning models can annotate CXRs in epidemiological studies reducing inter-reader and intra-reader variability. In addition, incorporating individual reader annotations can improve the performance of machine learning models trained using CXRs annotated by multiple readers.

Broadening volcanic eruption forecasting using transfer machine learning

2021 ◽  
Author(s):  
David Dempsey ◽  
Shane Cronin ◽  
Andreas Kempa-Liehr ◽  
Martin Letourneur
Keyword(s):  
Machine Learning ◽  
Seismic Station ◽  
Feature Space ◽  
Forecast Model ◽  
Linear Interpolation ◽  
Lessons Learned ◽  
Training Data ◽  
Single Station ◽  
Data Driven Approach ◽  
Model Training

<p>Sudden steam-driven eruptions at tourist volcanoes were the cause of 63 deaths at Mt Ontake (Japan) in 2014, and 22 deaths at Whakaari (New Zealand) in 2019. Warning systems that can anticipate these eruptions could provide crucial hours for evacuation or sheltering but these require reliable forecasting. Recently, machine learning has been used to extract eruption precursors from observational data and train forecasting models. However, a weakness of this data-driven approach is its reliance on long observational records that span multiple eruptions. As many volcano datasets may only record one or no eruptions, there is a need to extend these techniques to data-poor locales.</p><p>Transfer machine learning is one approach for generalising lessons learned at data-rich volcanoes and applying them to data-poor ones. Here, we tackle two problems: (1) generalising time series features between seismic stations at Whakaari to address recording gaps, and (2) training a forecasting model for Mt Ruapehu augmented using data from Whakaari. This required that we standardise data records at different stations for direct comparisons, devise an interpolation scheme to fill in missing eruption data, and combine volcano-specific feature matrices prior to model training.</p><p>We trained a forecast model for Whakaari using tremor data from three eruptions recorded at one seismic station (WSRZ) and augmented by data from two other eruptions recorded at a second station (WIZ). First, the training data from both stations were standardised to a unit normal distribution in log space. Then, linear interpolation in feature space was used to infer missing eruption features at WSRZ. Under pseudo-prospective testing, the augmented model had similar forecasting skill to one trained using all five eruptions recorded at a single station (WIZ). However, extending this approach to Ruapehu, we saw reduced performance indicating that more work is needed in standardisation and feature selection.</p>

Machine Learning and Value-Based Software Engineering

2012 ◽  
pp. 287-301
Author(s):  
Du Zhang
Keyword(s):  
Machine Learning ◽  
Software Engineering ◽  
Full Range ◽  
Training Data ◽  
Learning Methods ◽  
Engineering Research ◽  
Machine Learning Methods ◽  
Machine Learning Applications ◽  
Value Propositions ◽  
Software Engineering Research

Software engineering research and practice thus far are primarily conducted in a value-neutral setting where each artifact in software development such as requirement, use case, test case, and defect, is treated as equally important during a software system development process. There are a number of shortcomings of such value-neutral software engineering. Value-based software engineering is to integrate value considerations into the full range of existing and emerging software engineering principles and practices. Machine learning has been playing an increasingly important role in helping develop and maintain large and complex software systems. However, machine learning applications to software engineering have been largely confined to the value-neutral software engineering setting. In this paper, the general message to be conveyed is to apply machine learning methods and algorithms to value-based software engineering. The training data or the background knowledge or domain theory or heuristics or bias used by machine learning methods in generating target models or functions should be aligned with stakeholders’ value propositions. An initial research agenda is proposed for machine learning in value-based software engineering.

Objective evaluation of fabric smoothness appearance with an ordinal classification framework based on label noise estimation

2020 ◽  
pp. 004051752093957
Author(s):  
Jingan Wang ◽  
Meng Shuo ◽  
Lei Wang ◽  
Fengxin Sun ◽  
Ruru Pan ◽  
...  
Keyword(s):  
Measurement Errors ◽  
General Pattern ◽  
Image Features ◽  
Noise Estimation ◽  
Training Data ◽  
Label Noise ◽  
Ordinal Classification ◽  
Ordinal Information ◽  
Classification Framework ◽  
Model Training

Objective fabric smoothness appearance evaluation plays an important role in the textile and apparel industry. In most previous studies, objective fabric smoothness appearance evaluation is defined as a general pattern classification problem. However, the labels in this problem exhibit a natural ordering. Nominal classification ignores the ordinal information, which may cause overfitting in model training. In addition, for the existence of subjective errors, measurement errors, manual errors, etc., the labels in the data might be noisy, which has been rarely discussed previously. This paper proposes an ordinal classification framework based on label noise estimation (OCF-LNE) to objectively evaluate the fabric smoothness appearance degree, which takes the ordinal information and noise of the label in the training data into consideration. The OCF-LNE uses the basic classifier in pre-training as a label noise estimator, and uses the estimated label noise to adjust the labels in further training. The adjusted labels can introduce the ordinal constrain implicitly and reduce the negative impact of label noise in model training. Within a 10 × 10 nested cross-validation, the proposed OCF-LNE achieves 82.86%, 94.29%, and 100% average accuracies under errors of 0, 0.5, and 1 degree, respectively. Experiments on different fabric image features and basic classification models verify the effectiveness of the OCF-LNE. In addition, the proposed method outperforms the state-of-the-art methods for fabric smoothness evaluation and ordinal classification. Promisingly, the OCF-LNE can provide novel ideas for image-based fabric smoothness evaluation.

Entropy Ensemble Filter: Does information content assessment of bootstrapped training datasets before model training lead to better trade-off between ensemble size and predictive performance?

2020 ◽  
Author(s):  
Hossein Foroozand ◽  
Steven V. Weijs
Keyword(s):  
Machine Learning ◽  
Computational Cost ◽  
Predictive Performance ◽  
Original Data ◽  
Training Data ◽  
Computational Time ◽  
Limiting Factor ◽  
Ensemble Size ◽  
Content Assessment ◽  
Model Training

<p>Machine learning is the fast-growing branch of data-driven models, and its main objective is to use computational methods to become more accurate in predicting outcomes without being explicitly programmed. In this field, a way to improve model predictions is to use a large collection of models (called ensemble) instead of a single one. Each model is then trained on slightly different samples of the original data, and their predictions are averaged. This is called bootstrap aggregating, or Bagging, and is widely applied. A recurring question in previous works was: how to choose the ensemble size of training data sets for tuning the weights in machine learning? The computational cost of ensemble-based methods scales with the size of the ensemble, but excessively reducing the ensemble size comes at the cost of reduced predictive performance. The choice of ensemble size was often determined based on the size of input data and available computational power, which can become a limiting factor for larger datasets and complex models’ training. In this research, it is our hypothesis that if an ensemble of artificial neural networks (ANN) models or any other machine learning technique uses the most informative ensemble members for training purpose rather than all bootstrapped ensemble members, it could reduce the computational time substantially without negatively affecting the performance of simulation.</p>

scholarly journals Literature Mining for Incorporating Inductive Bias in Biomedical Prediction Tasks (Student Abstract)

2020 ◽  
Vol 34 (10) ◽  
pp. 13983-13984
Author(s):  
Qizhen Zhang ◽  
Audrey Durand ◽  
Joelle Pineau
Keyword(s):  
Machine Learning ◽  
Prior Knowledge ◽  
Expert Knowledge ◽  
Training Data ◽  
Literature Mining ◽  
Prediction Task ◽  
Inductive Bias ◽  
Prediction Outcome ◽  
Applications Of Machine Learning

Applications of machine learning in biomedical prediction tasks are often limited by datasets that are unrepresentative of the sampling population. In these situations, we can no longer rely only on the the training data to learn the relations between features and the prediction outcome. Our method proposes to learn an inductive bias that indicates the relevance of each feature to outcomes through literature mining in PubMed, a centralized source of biomedical documents. The inductive bias acts as a source of prior knowledge from experts, which we leverage by imposing an extra penalty for model weights that differ from this inductive bias. We empirically evaluate our method on a medical prediction task and highlight the importance of incorporating expert knowledge that can capture relations not present in the training data.

scholarly journals Methodology for Collecting a Training Dataset for an Intrusion Detection Model

2021 ◽  
Vol 33 (5) ◽  
pp. 83-104
Author(s):  
Aleksandr Igorevich Getman ◽  
Maxim Nikolaevich Goryunov ◽  
Andrey Georgievich Matskevich ◽  
Dmitry Aleksandrovich Rybolovlev
Keyword(s):  
Attack Detection ◽  
Training Data ◽  
Training Dataset ◽  
Training Models ◽  
The Public ◽  
Detection Model ◽  
Computer Attacks ◽  
Model Training ◽  
Public Datasets

The paper discusses the issues of training models for detecting computer attacks based on the use of machine learning methods. The results of the analysis of publicly available training datasets and tools for analyzing network traffic and identifying features of network sessions are presented sequentially. The drawbacks of existing tools and possible errors in the datasets formed with their help are noted. It is concluded that it is necessary to collect own training data in the absence of guarantees of the public datasets reliability and the limited use of pre-trained models in networks with characteristics that differ from the characteristics of the network in which the training traffic was collected. A practical approach to generating training data for computer attack detection models is proposed. The proposed solutions have been tested to evaluate the quality of model training on the collected data and the quality of attack detection in conditions of real network infrastructure.

scholarly journals Noise Removal Process from Label Classification using Machine Learning

2019 ◽  
Vol 8 (3) ◽  
pp. 172-175
Keyword(s):  
Machine Learning ◽  
Big Data ◽  
Supervised Learning ◽  
Noise Removal ◽  
Error Rates ◽  
Training Data ◽  
Learning Performance ◽  
Training Dataset ◽  
Noise Filtering ◽  
Label Noise

Text classification and clustering approach is essential for big data environments. In supervised learning applications many classification algorithms have been proposed. In the era of big data, a large volume of training data is available in many machine learning works. However, there is a possibility of mislabeled or unlabeled data that are not labeled properly. Some labels may be incorrect resulted in label noise which in turn regress learning performance of a classifier. A general approach to address label noise is to apply noise filtering techniques to identify and remove noise before learning. A range of noise filtering approaches have been developed to improve the classifiers performance. This paper proposes noise filtering approach in text data during the training phase. Many supervised learning algorithms generates high error rates due to noise in training dataset, our work eliminates such noise and provides accurate classification system.

Use of machine learning to identify relevant research publications in clinical oncology.

2019 ◽  
Vol 37 (15_suppl) ◽  
pp. 6558-6558
Author(s):  
Fernando Jose Suarez Saiz ◽  
Corey Sanders ◽  
Rick J Stevens ◽  
Robert Nielsen ◽  
Michael W Britt ◽  
...  
Keyword(s):  
Machine Learning ◽  
Expert Knowledge ◽  
Grey Literature ◽  
Clinical Decision ◽  
Training Data ◽  
Surgical Approaches ◽  
High Quality ◽  
Test Set ◽  
N Gram ◽  
Abstract Content

6558 Background: Finding high-quality science to support decisions for individual patients is challenging. Common approaches to assess clinical literature quality and relevance rely on bibliometrics or expert knowledge. We describe a method to automatically identify clinically relevant, high-quality scientific citations using abstract content. Methods: We used machine learning trained on text from PubMed papers cited in 3 expert resources: NCCN, NCI-PDQ, and Hemonc.org. Balanced training data included text cited in at least two sources to form an “on topic” set (i.e., relevant and high quality), and an “off-topic” set, not cited in any of the above 3 sources. The off-topic set was published in lower ranked journals, using a citation-based score. Articles were part of an Oncology Clinical Trial corpus generated using a standard PubMed query. We used a gradient boosted-tree approach with a binary logistic supervised learning classification. Briefly, 988 texts were processed to produce a term frequency-inverse document frequency (tf-idf) n-gram representation of both the training and the test set (70/30 split). Ideal parameters were determined using 1000-fold cross validation. Results: Our model classified papers in the test set with 0.93 accuracy (95% CI (0.09:0.96) p ≤ 0.0001), with sensitivity 0.95 and specificity 0.91. Some false positives contained language considered clinically relevant that may have been missed or not yet included in expert resources. False negatives revealed a potential bias towards chemotherapy-focused research over radiation therapy or surgical approaches. Conclusions: Machine learning can be used to automatically identify relevant clinical publications from biographic databases, without relying on expert curation or bibliometric methods. The use of machine learning to identify relevant publications may reduce the time clinicians spend finding pertinent evidence for a patient. This approach is generalizable to cases where a corpus of high-quality publications that can serve as a training set exists or cases where document metadata is unreliable, as is the case of “grey” literature within oncology and beyond to other diseases. Future work will extend this approach and may integrate it into oncology clinical decision-support tools.

Sign in / Sign up

Export Citation Format

Share Document