scholarly journals An Incremental Local Outlier Detection Method in the Data Stream

2018 ◽  
Vol 8 (8) ◽  
pp. 1248 ◽  
Author(s):  
Haiqing Yao ◽  
Xiuwen Fu ◽  
Yongsheng Yang ◽  
Octavian Postolache
Keyword(s):  
Outlier Detection ◽  
Data Streams ◽  
Data Stream ◽  
Nearest Neighbor ◽  
Nearest Neighbors ◽  
Detection Accuracy ◽  
K Nearest Neighbor ◽  
Major Work ◽  
Wide Range ◽  
Local Outlier

Outlier detection has attracted a wide range of attention for its broad applications, such as fault diagnosis and intrusion detection, among which the outlier analysis in data streams with high uncertainty and infinity is more challenging. Recent major work of outlier detection has focused on principle research of the local outlier factor, and there are few studies on incremental updating strategies, which are vital to outlier detection in data streams. In this paper, a novel incremental local outlier detection approach is introduced to dynamically evaluate the local outlier in the data stream. An extended local neighborhood consisting of k nearest neighbors, reverse nearest neighbors and shared nearest neighbors is estimated for each data. The theoretical evidence of algorithm complexity for the insertion of new data and deletion of old data in the composite neighborhood shows that the amount of affected data in the incremental calculation is finite. Finally, experiments performed on both synthetic and real datasets verify its scalability and outlier detection accuracy. All results show that the proposed approach has comparable performance with state-of-the-art k nearest neighbor-based methods.

scholarly journals Efficient Shared Execution Processing of k-Nearest Neighbor Joins in Road Networks

2018 ◽  
Vol 2018 ◽  
pp. 1-17 ◽  
Author(s):  
Hyung-Ju Cho
Keyword(s):  
Euclidean Distance ◽  
Nearest Neighbor ◽  
Real Life ◽  
Road Networks ◽  
Nearest Neighbors ◽  
Superior Performance ◽  
K Nearest Neighbor ◽  
Wide Range ◽  
Primitive Operation ◽  
Nested Loop

We investigate the k-nearest neighbor (kNN) join in road networks to determine the k-nearest neighbors (NNs) from a dataset S to every object in another dataset R. The kNN join is a primitive operation and is widely used in many data mining applications. However, it is an expensive operation because it combines the kNN query and the join operation, whereas most existing methods assume the use of the Euclidean distance metric. We alternatively consider the problem of processing kNN joins in road networks where the distance between two points is the length of the shortest path connecting them. We propose a shared execution-based approach called the group-nested loop (GNL) method that can efficiently evaluate kNN joins in road networks by exploiting grouping and shared execution. The GNL method can be easily implemented using existing kNN query algorithms. Extensive experiments using several real-life roadmaps confirm the superior performance and effectiveness of the proposed method in a wide range of problem settings.

scholarly journals TADILOF: Time Aware Density-Based Incremental Local Outlier Detection in Data Streams

Sensors ◽  
2020 ◽  
Vol 20 (20) ◽  
pp. 5829 ◽  
Author(s):  
Jen-Wei Huang ◽  
Meng-Xun Zhong ◽  
Bijay Prasad Jaysawal
Keyword(s):  
Outlier Detection ◽  
Data Streams ◽  
Data Stream ◽  
State Of The Art ◽  
Streaming Data ◽  
Current State ◽  
Data Points ◽  
Local Outlier ◽  
Time Aware ◽  
Over Time

Outlier detection in data streams is crucial to successful data mining. However, this task is made increasingly difficult by the enormous growth in the quantity of data generated by the expansion of Internet of Things (IoT). Recent advances in outlier detection based on the density-based local outlier factor (LOF) algorithms do not consider variations in data that change over time. For example, there may appear a new cluster of data points over time in the data stream. Therefore, we present a novel algorithm for streaming data, referred to as time-aware density-based incremental local outlier detection (TADILOF) to overcome this issue. In addition, we have developed a means for estimating the LOF score, termed "approximate LOF," based on historical information following the removal of outdated data. The results of experiments demonstrate that TADILOF outperforms current state-of-the-art methods in terms of AUC while achieving similar performance in terms of execution time. Moreover, we present an application of the proposed scheme to the development of an air-quality monitoring system.

K-NN Based Outlier Detection Technique on Intrusion Dataset

2017 ◽  
Vol 7 (1) ◽  
pp. 58-70 ◽  
Author(s):  
Santosh Kumar Sahu ◽  
Sanjay Kumar Jena ◽  
Manish Verma
Keyword(s):  
Outlier Detection ◽  
Time Complexity ◽  
Nearest Neighbor ◽  
Nearest Neighbors ◽  
The Other ◽  
Detection Technique ◽  
K Nearest Neighbors ◽  
Time Required ◽  
Local Outlier ◽  
Quick Sort

Outliers in the database are the objects that deviate from the rest of the dataset by some measure. The Nearest Neighbor Outlier Factor is considering to measure the degree of outlier-ness of the object in the dataset. Unlike the other methods like Local Outlier Factor, this approach shows the interest of a point from both neighbors and reverse neighbors, and after that, an object comes into consideration. We have observed that in GBBK algorithm that based on K-NN, used quick sort to find k nearest neighbors that take O (N log N) time. However, in proposed method, the time required for searching on K times which complete in O (KN) time to find k nearest neighbors (k < < log N). As a result, the proposed method improves the time complexity. The NSL-KDD and Fisher iris dataset is used, and experimental results compared with the GBBK method. The result is same in both the methods, but the proposed method takes less time for computation.

scholarly journals Tropical Balls and Its Applications to K Nearest Neighbor over the Space of Phylogenetic Trees

Mathematics ◽  
2021 ◽  
Vol 9 (7) ◽  
pp. 779
Author(s):  
Ruriko Yoshida
Keyword(s):  
Supervised Learning ◽  
Phylogenetic Trees ◽  
Nearest Neighbor ◽  
Nearest Neighbors ◽  
High Dimensional ◽  
Learning Method ◽  
Dimensional Vector ◽  
K Nearest Neighbor ◽  
K Nearest Neighbors

A tropical ball is a ball defined by the tropical metric over the tropical projective torus. In this paper we show several properties of tropical balls over the tropical projective torus and also over the space of phylogenetic trees with a given set of leaf labels. Then we discuss its application to the K nearest neighbors (KNN) algorithm, a supervised learning method used to classify a high-dimensional vector into given categories by looking at a ball centered at the vector, which contains K vectors in the space.

scholarly journals Assessing the Relation between Mud Components and Rheology for Loss Circulation Prevention Using Polymeric Gels: A Machine Learning Approach

Energies ◽  
2021 ◽  
Vol 14 (5) ◽  
pp. 1377
Author(s):  
Musaab I. Magzoub ◽  
Raj Kiran ◽  
Saeed Salehi ◽  
Ibnelwaleed A. Hussein ◽  
Mustafa S. Nasser
Keyword(s):  
Machine Learning ◽  
Rheological Properties ◽  
Nearest Neighbor ◽  
Drilling Fluid ◽  
Gradient Boosting ◽  
K Nearest Neighbor ◽  
Wide Range ◽  
Machine Learning Approach ◽  
Drilling Operations

The traditional way to mitigate loss circulation in drilling operations is to use preventative and curative materials. However, it is difficult to quantify the amount of materials from every possible combination to produce customized rheological properties. In this study, machine learning (ML) is used to develop a framework to identify material composition for loss circulation applications based on the desired rheological characteristics. The relation between the rheological properties and the mud components for polyacrylamide/polyethyleneimine (PAM/PEI)-based mud is assessed experimentally. Four different ML algorithms were implemented to model the rheological data for various mud components at different concentrations and testing conditions. These four algorithms include (a) k-Nearest Neighbor, (b) Random Forest, (c) Gradient Boosting, and (d) AdaBoosting. The Gradient Boosting model showed the highest accuracy (91 and 74% for plastic and apparent viscosity, respectively), which can be further used for hydraulic calculations. Overall, the experimental study presented in this paper, together with the proposed ML-based framework, adds valuable information to the design of PAM/PEI-based mud. The ML models allowed a wide range of rheology assessments for various drilling fluid formulations with a mean accuracy of up to 91%. The case study has shown that with the appropriate combination of materials, reasonable rheological properties could be achieved to prevent loss circulation by managing the equivalent circulating density (ECD).

Scalable KDE-based top-n local outlier detection over large-scale data streams

2020 ◽  
Vol 204 ◽  
pp. 106186 ◽  
Author(s):  
Fang Liu ◽  
Yanwei Yu ◽  
Peng Song ◽  
Yangyang Fan ◽  
Xiangrong Tong
Keyword(s):  
Outlier Detection ◽  
Data Streams ◽  
Large Scale ◽  
Large Scale Data ◽  
Scale Data ◽  
Local Outlier

scholarly journals The K Nearest Neighbor Algorithm for Imputation of Missing Longitudinal Prenatal Alcohol Data

2021 ◽  
Author(s):  
Ayesha Sania ◽  
Nicolo Pini ◽  
Morgan Nelson ◽  
Michael Myers ◽  
Lauren Shuffrey ◽  
...  
Keyword(s):  
Missing Data ◽  
Missing Values ◽  
Nearest Neighbor ◽  
Learning Algorithm ◽  
Drinking Behavior ◽  
Nearest Neighbors ◽  
First Trimester ◽  
Epidemiologic Studies ◽  
K Nearest Neighbor ◽  
Timeline Followback

Abstract Background — Missing data are a source of bias in epidemiologic studies. This is problematic in alcohol research where data missingness is linked to drinking behavior. Methods — The Safe Passage study was a prospective investigation of prenatal drinking and fetal/infant outcomes (n=11,083). Daily alcohol consumption for last reported drinking day and 30 days prior was recorded using Timeline Followback method. Of 3.2 million person-days, data were missing for 0.36 million. We imputed missing data using a machine learning algorithm; “K Nearest Neighbor” (K-NN). K-NN imputes missing values for a participant using data of participants closest to it. Imputed values were weighted for the distances from nearest neighbors and matched for day of week. Validation was done on randomly deleted data for 5-15 consecutive days. Results — Data from 5 nearest neighbors and segments of 55 days provided imputed values with least imputation error. After deleting data segments from with no missing days first trimester, there was no difference between actual and predicted values for 64% of deleted segments. For 31% of the segments, imputed data were within +/-1 drink/day of the actual. Conclusions — K-NN can be used to impute missing data in longitudinal studies of alcohol use during pregnancy with high accuracy.

Outlier Detection for Soft-Sensor Modeling Data Based on k-Nearest Neighbor

2012 ◽  
Vol 468-471 ◽  
pp. 2504-2509
Author(s):  
Qiang Da Yang ◽  
Zhen Quan Liu
Keyword(s):  
Outlier Detection ◽  
Nearest Neighbor ◽  
Detection Method ◽  
Soft Sensor ◽  
K Nearest Neighbor ◽  
Sensor Model ◽  
Sensor Technique ◽  
On Line ◽  
Modeling Data ◽  
Sensor Modeling

The on-line estimation of some key hard-to-measure process variables by using soft-sensor technique has received extensive concern in industrial production process. The precision of on-line estimation is closely related to the accuracy of soft-sensor model, while the accuracy of soft-sensor model depends strongly on the accuracy of modeling data. Aiming at the special character of the definition for outliers in soft-sensor modeling process, an outlier detection method based on k-nearest neighbor (k-NN) is proposed in this paper. The proposed method can be realized conveniently from data without priori knowledge and assumption of the process. The simulation result and practical application show that the proposed outlier detection method based on k-NN has good detection effect and high application value.

A Comparison of the Analysis of Methods for Feature Extraction and Classification by Wavelet Transform in SSVEP BCIs

2020 ◽  
Author(s):  
Hoda Heidari ◽  
Zahra Einalou ◽  
Mehrdad Dadgostar ◽  
Hamidreza Hosseinzadeh
Keyword(s):  
Feature Extraction ◽  
Feature Selection ◽  
Wavelet Transform ◽  
Decision Tree ◽  
Nearest Neighbor ◽  
Support Vector ◽  
K Nearest Neighbor ◽  
Iir Filters ◽  
Wide Range ◽  
New Feature

Abstract Most of the studies in the field of Brain-Computer Interface (BCI) based on electroencephalography have a wide range of applications. Extracting Steady State Visual Evoked Potential (SSVEP) is regarded as one of the most useful tools in BCI systems. In this study, different methods such as feature extraction with different spectral methods (Shannon entropy, skewness, kurtosis, mean, variance) (bank of filters, narrow-bank IIR filters, and wavelet transform magnitude), feature selection performed by various methods (decision tree, principle component analysis (PCA), t-test, Wilcoxon, Receiver operating characteristic (ROC)), and classification step applying k nearest neighbor (k-NN), perceptron, support vector machines (SVM), Bayesian, multiple layer perceptron (MLP) were compared from the whole stream of signal processing. Through combining such methods, the effective overview of the study indicated the accuracy of classical methods. In addition, the present study relied on a rather new feature selection described by decision tree and PCA, which is used for the BCI-SSVEP systems. Finally, the obtained accuracies were calculated based on the four recorded frequencies representing four directions including right, left, up, and down.

Sign in / Sign up

Export Citation Format

Share Document