Something Is Better Than Everything: A Distributed Approach to Audit Log Anomaly Detection

2017 ◽  
Author(s):  
Isis Rose ◽  
Nicholas Felts ◽  
Alexander George ◽  
Emily Miller ◽  
Max Planck
Keyword(s):  
Anomaly Detection ◽  
Distributed Approach ◽  
Better Than

A Novel Network Traffic Anomaly Detection Based on Multi-Scale Fusion

2011 ◽  
Vol 48-49 ◽  
pp. 102-105
Author(s):  
Guo Zhen Cheng ◽  
Dong Nian Cheng ◽  
He Lei
Keyword(s):  
Anomaly Detection ◽  
False Alarm ◽  
False Alarm Rate ◽  
Network Traffic ◽  
Self Similarity ◽  
Data Set ◽  
Multi Scale ◽  
Traffic Anomaly ◽  
Detection Evaluation ◽  
Better Than

Detecting network traffic anomaly is very important for network security. But it has high false alarm rate, low detect rate and that can’t perform real-time detection in the backbone very well due to its nonlinearity, nonstationarity and self-similarity. Therefore we propose a novel detection method—EMD-DS, and prove that it can reduce mean error rate of anomaly detection efficiently after EMD. On the KDD CUP 1999 intrusion detection evaluation data set, this detector detects 85.1% attacks at low false alarm rate which is better than some other systems.

scholarly journals Valid Probabilistic Anomaly Detection Models for System Logs

2020 ◽  
Vol 2020 ◽  
pp. 1-12
Author(s):  
Chunbo Liu ◽  
Lanlan Pan ◽  
Zhaojun Gu ◽  
Jialiang Wang ◽  
Yitong Ren ◽  
...  
Keyword(s):  
Anomaly Detection ◽  
Large Scale ◽  
Learning Algorithm ◽  
Recall Rate ◽  
Support Vector ◽  
Fusion Algorithm ◽  
Flexible Tool ◽  
System Logs ◽  
Output Only ◽  
Better Than

System logs can record the system status and important events during system operation in detail. Detecting anomalies in the system logs is a common method for modern large-scale distributed systems. Yet threshold-based classification models used for anomaly detection output only two values: normal or abnormal, which lacks probability of estimating whether the prediction results are correct. In this paper, a statistical learning algorithm Venn-Abers predictor is adopted to evaluate the confidence of prediction results in the field of system log anomaly detection. It is able to calculate the probability distribution of labels for a set of samples and provide a quality assessment of predictive labels to some extent. Two Venn-Abers predictors LR-VA and SVM-VA have been implemented based on Logistic Regression and Support Vector Machine, respectively. Then, the differences among different algorithms are considered so as to build a multimodel fusion algorithm by Stacking. And then a Venn-Abers predictor based on the Stacking algorithm called Stacking-VA is implemented. The performances of four types of algorithms (unimodel, Venn-Abers predictor based on unimodel, multimodel, and Venn-Abers predictor based on multimodel) are compared in terms of validity and accuracy. Experiments are carried out on a log dataset of the Hadoop Distributed File System (HDFS). For the comparative experiments on unimodels, the results show that the validities of LR-VA and SVM-VA are better than those of the two corresponding underlying models. Compared with the underlying model, the accuracy of the SVM-VA predictor is better than that of LR-VA predictor, and more significantly, the recall rate increases from 81% to 94%. In the case of experiments on multiple models, the algorithm based on Stacking multimodel fusion is significantly superior to the underlying classifier. The average accuracy of Stacking-VA is larger than 0.95, which is more stable than the prediction results of LR-VA and SVM-VA. Experimental results show that the Venn-Abers predictor is a flexible tool that can make accurate and valid probability predictions in the field of system log anomaly detection.

scholarly journals Semi-supervised generative adversarial networks for anomaly detection

2022 ◽  
Vol 132 ◽  
pp. 01016
Author(s):  
Juan Montenegro ◽  
Yeojin Chung
Keyword(s):  
Anomaly Detection ◽  
Video Surveillance ◽  
Daily Life ◽  
Generative Adversarial Networks ◽  
Gamma Correction ◽  
Generative Adversarial Network ◽  
Adversarial Network ◽  
Adversarial Networks ◽  
Different Types ◽  
Better Than

Advancements in security have provided ways of recording anomalies of daily life through video surveillance. For the present investigation, a semi-supervised generative adversarial network model to detect and classify different types of crimes on videos. Additionally, we intend to tackle one of the most recurring difficulties of anomaly detection: illumination. For this, we propose a light augmentation algorithm based on gamma correction to help the semi-supervised generative adversarial networks on its classification task. The proposed process performs slightly better than other proposed models.

A New Prior for Bayesian Anomaly Detection

2010 ◽  
Vol 49 (01) ◽  
pp. 44-53 ◽  
Author(s):  
G. F. Cooper ◽  
Y. Shen
Keyword(s):  
Anomaly Detection ◽  
Control Chart ◽  
Bayesian Method ◽  
Prior Probability ◽  
Disease Outbreak ◽  
Detection Performance ◽  
Outbreak Detection ◽  
Informative Prior ◽  
Disease Outbreak Detection ◽  
Better Than

Summary Objectives: Bayesian anomaly detection computes posterior probabilities of anomalous events by combining prior beliefs and evidence from data. However, the specification of prior probabilities can be challenging. This paper describes a Bayesian prior in the context of disease outbreak detection. The goal is to provide a meaningful, easy-to-use prior that yields a posterior probability of an outbreak that performs at least as well as a standard frequentist approach. If this goal is achieved, the resulting posterior could be usefully incorporated into a decision analysis about how to act in light of a possible disease outbreak. Methods: This paper describes a Bayesian method for anomaly detection that combines learning from data with a semi-informative prior probability over patterns of anomalous events. A univariate version of the algorithm is presented here for ease of illustration of the essential ideas. The paper describes the algorithm in the context of disease-outbreak detection, but it is general and can be used in other anomaly detection applications. For this application, the semi-informative prior specifies that an increased count over baseline is expected for the variable being monitored, such as the number of respiratory chief complaints per day at a given emergency department. The semi-informative prior is derived based on the baseline prior, which is estimated from using historical data. Results: The evaluation reported here used semi-synthetic data to evaluate the detection performance of the proposed Bayesian method and a control chart method, which is a standard frequentist algorithm that is closest to the Bayesian method in terms of the type of data it uses. The disease-outbreak detection performance of the Bayesian method was statistically significantly better than that of the control chart method when proper baseline periods were used to estimate the baseline behavior to avoid seasonal effects. When using longer baseline periods, the Bayesian method performed as well as the control chart method. The time complexity of the Bayesian algorithm is linear in the number of the observed events being monitored, due to a novel, closed-form derivation that is introduced in the paper. Conclusions: This paper introduces a novel prior probability for Bayesian outbreak detection that is expressive, easy-to-apply, computationally efficient, and performs as well or better than a standard frequentist method.

scholarly journals Vision Transformer-Based Tailing Detection in Videos

2021 ◽  
Vol 11 (24) ◽  
pp. 11591
Author(s):  
Jaewoo Lee ◽  
Sungjun Lee ◽  
Wonki Cho ◽  
Zahid Ali Siddiqui ◽  
Unsang Park
Keyword(s):  
Time Series ◽  
Anomaly Detection ◽  
Spatial Information ◽  
Detection Problem ◽  
Walking Pattern ◽  
Series Of Experiments ◽  
Time Series Images ◽  
Small Dataset ◽  
Better Than ◽  
Do So

Tailing is defined as an event where a suspicious person follows someone closely. We define the problem of tailing detection from videos as an anomaly detection problem, where the goal is to find abnormalities in the walking pattern of the pedestrians (victim and follower). We, therefore, propose a modified Time-Series Vision Transformer (TSViT), a method for anomaly detection in video, specifically for tailing detection with a small dataset. We introduce an effective way to train TSViT with a small dataset by regularizing the prediction model. To do so, we first encode the spatial information of the pedestrians into 2D patterns and then pass them as tokens to the TSViT. Through a series of experiments, we show that the tailing detection on a small dataset using TSViT outperforms popular CNN-based architectures, as the CNN architectures tend to overfit with a small dataset of time-series images. We also show that when using time-series images, the performance of CNN-based architecture gradually drops, as the network depth is increased, to increase its capacity. On the other hand, a decreasing number of heads in Vision Transformer architecture shows good performance on time-series images, and the performance is further increased as the input resolution of the images is increased. Experimental results demonstrate that the TSViT performs better than the handcrafted rule-based method and CNN-based method for tailing detection. TSViT can be used in many applications for video anomaly detection, even with a small dataset.

A Neural Network for Image Anomaly Detection with Deep Pyramidal Representations and Dynamic Routing

2020 ◽  
Vol 30 (10) ◽  
pp. 2050060
Author(s):  
Pankaj Mishra ◽  
Claudio Piciarelli ◽  
Gian Luca Foresti
Keyword(s):  
Neural Network ◽  
Anomaly Detection ◽  
State Of The Art ◽  
Structural Similarity ◽  
Input Image ◽  
Image Features ◽  
Anomalous Data ◽  
Scale Levels ◽  
Anomaly Detector ◽  
Better Than

Image anomaly detection is an application-driven problem where the aim is to identify novel samples, which differ significantly from the normal ones. We here propose Pyramidal Image Anomaly DEtector (PIADE), a deep reconstruction-based pyramidal approach, in which image features are extracted at different scale levels to better catch the peculiarities that could help to discriminate between normal and anomalous data. The features are dynamically routed to a reconstruction layer and anomalies can be identified by comparing the input image with its reconstruction. Unlike similar approaches, the comparison is done by using structural similarity and perceptual loss rather than trivial pixel-by-pixel comparison. The proposed method performed at par or better than the state-of-the-art methods when tested on publicly available datasets such as CIFAR10, COIL-100 and MVTec.

scholarly journals Medical Fraud and Abuse Detection System Based on Machine Learning

2020 ◽  
Vol 17 (19) ◽  
pp. 7265 ◽  
Author(s):  
Conghai Zhang ◽  
Xinyao Xiao ◽  
Chao Wu
Keyword(s):  
Neural Network ◽  
Machine Learning ◽  
Health Care ◽  
Anomaly Detection ◽  
Healthcare System ◽  
Loss Function ◽  
Detection System ◽  
Relative Probability ◽  
Fully Connected ◽  
Better Than

It is estimated that approximately 10% of healthcare system expenditures are wasted due to medical fraud and abuse. In the medical area, the combination of thousands of drugs and diseases make the supervision of health care more difficult. To quantify the disease–drug relationship into relationship score and do anomaly detection based on this relationship score and other features, we proposed a neural network with fully connected layers and sparse convolution. We introduced a focal-loss function to adapt to the data imbalance and a relative probability score to measure the model’s performance. As our model performs much better than previous ones, it can well alleviate analysts’ work.

Sign in / Sign up

Export Citation Format

Share Document