Online Fault and Anomaly Detection for Large-Scale Scientific Workflows

2011 ◽  
Author(s):  
Taghrid Samak ◽  
Dan Gunter ◽  
Monte Goode ◽  
Ewa Deelman ◽  
Gideon Juve ◽  
...  
Keyword(s):  
Anomaly Detection ◽  
Large Scale ◽  
Scientific Workflows

scholarly journals Multi-task learning based Encoder-Decoder: A comprehensive detection and diagnosis system for multi-sensor data

2021 ◽  
Vol 13 (5) ◽  
pp. 168781402110131
Author(s):  
Junfeng Wu ◽  
Li Yao ◽  
Bin Liu ◽  
Zheyuan Ding ◽  
Lei Zhang
Keyword(s):  
Anomaly Detection ◽  
Event Detection ◽  
Large Scale ◽  
Multivariate Time Series ◽  
Sensor Data ◽  
Unified Framework ◽  
Diagnosis System ◽  
Learning Framework ◽  
Task Learning ◽  
Detection And Diagnosis

As more and more sensor data have been collected, automated detection, and diagnosis systems are urgently needed to lessen the increasing monitoring burden and reduce the risk of system faults. A plethora of researches have been done on anomaly detection, event detection, anomaly diagnosis respectively. However, none of current approaches can explore all these respects in one unified framework. In this work, a Multi-Task Learning based Encoder-Decoder (MTLED) which can simultaneously detect anomalies, diagnose anomalies, and detect events is proposed. In MTLED, feature matrix is introduced so that features are extracted for each time point and point-wise anomaly detection can be realized in an end-to-end way. Anomaly diagnosis and event detection share the same feature matrix with anomaly detection in the multi-task learning framework and also provide important information for system monitoring. To train such a comprehensive detection and diagnosis system, a large-scale multivariate time series dataset which contains anomalies of multiple types is generated with simulation tools. Extensive experiments on the synthetic dataset verify the effectiveness of MTLED and its multi-task learning framework, and the evaluation on a real-world dataset demonstrates that MTLED can be used in other application scenarios through transfer learning.

Hierarchical Anomaly Detection and Multimodal Classification in Large-Scale Photovoltaic Systems

2019 ◽  
Vol 10 (3) ◽  
pp. 1351-1361 ◽  
Author(s):  
Yingying Zhao ◽  
Qi Liu ◽  
Dongsheng Li ◽  
Dahai Kang ◽  
Qin Lv ◽  
...  
Keyword(s):  
Anomaly Detection ◽  
Large Scale ◽  
Photovoltaic Systems ◽  
Multimodal Classification

Data Management in Scientific Workflows

2012 ◽  
pp. 177-187
Author(s):  
Ewa Deelman ◽  
Ann Chervenak
Keyword(s):  
Data Management ◽  
Gravitational Wave ◽  
Large Scale ◽  
Real Life ◽  
Scientific Workflows ◽  
Workflow Systems ◽  
Automated Generation ◽  
Workflow Execution ◽  
Data Products ◽  
Workflow Planning

Scientific applications such as those in astronomy, earthquake science, gravitational-wave physics, and others have embraced workflow technologies to do large-scale science. Workflows enable researchers to collaboratively design, manage, and obtain results that involve hundreds of thousands of steps, access terabytes of data, and generate similar amounts of intermediate and final data products. Although workflow systems are able to facilitate the automated generation of data products, many issues still remain to be addressed. These issues exist in different forms in the workflow lifecycle. This chapter describes a workflow lifecycle as consisting of a workflow generation phase where the analysis is defined, the workflow planning phase where resources needed for execution are selected, the workflow execution part, where the actual computations take place, and the result, metadata, and provenance storing phase. The authors discuss the issues related to data management at each step of the workflow cycle. They describe challenge problems and illustrate them in the context of real-life applications. They discuss the challenges, possible solutions, and open issues faced when mapping and executing large-scale workflows on current cyberinfrastructure. They particularly emphasize the issues related to the management of data throughout the workflow lifecycle.

Generative and Autoencoder Models for Large-Scale Mutivariate Unsupervised Anomaly Detection

2021 ◽  
pp. 45-58
Author(s):  
Nabila Ounasser ◽  
Maryem Rhanoui ◽  
Mounia Mikram ◽  
Bouchra El Asri
Keyword(s):  
Anomaly Detection ◽  
Large Scale ◽  
Unsupervised Anomaly Detection

Logging Analysis and Prediction in Open Source Java Project

2021 ◽  
pp. 733-761
Author(s):  
Sangeeta Lal ◽  
Neetu Sardana ◽  
Ashish Sureka
Keyword(s):  
Machine Learning ◽  
Content Analysis ◽  
Software Development ◽  
Anomaly Detection ◽  
Open Source ◽  
Large Scale ◽  
Source Code ◽  
Scale Analysis ◽  
Large Scale Analysis ◽  
Research Questions

Log statements present in source code provide important information to the software developers because they are useful in various software development activities such as debugging, anomaly detection, and remote issue resolution. Most of the previous studies on logging analysis and prediction provide insights and results after analyzing only a few code constructs. In this chapter, the authors perform an in-depth, focused, and large-scale analysis of logging code constructs at two levels: the file level and catch-blocks level. They answer several research questions related to statistical and content analysis. Statistical and content analysis reveals the presence of differentiating properties among logged and nonlogged code constructs. Based on these findings, the authors propose a machine-learning-based model for catch-blocks logging prediction. The machine-learning-based model is found to be effective in catch-blocks logging prediction.

scholarly journals Valid Probabilistic Anomaly Detection Models for System Logs

2020 ◽  
Vol 2020 ◽  
pp. 1-12
Author(s):  
Chunbo Liu ◽  
Lanlan Pan ◽  
Zhaojun Gu ◽  
Jialiang Wang ◽  
Yitong Ren ◽  
...  
Keyword(s):  
Anomaly Detection ◽  
Large Scale ◽  
Learning Algorithm ◽  
Recall Rate ◽  
Support Vector ◽  
Fusion Algorithm ◽  
Flexible Tool ◽  
System Logs ◽  
Output Only ◽  
Better Than

System logs can record the system status and important events during system operation in detail. Detecting anomalies in the system logs is a common method for modern large-scale distributed systems. Yet threshold-based classification models used for anomaly detection output only two values: normal or abnormal, which lacks probability of estimating whether the prediction results are correct. In this paper, a statistical learning algorithm Venn-Abers predictor is adopted to evaluate the confidence of prediction results in the field of system log anomaly detection. It is able to calculate the probability distribution of labels for a set of samples and provide a quality assessment of predictive labels to some extent. Two Venn-Abers predictors LR-VA and SVM-VA have been implemented based on Logistic Regression and Support Vector Machine, respectively. Then, the differences among different algorithms are considered so as to build a multimodel fusion algorithm by Stacking. And then a Venn-Abers predictor based on the Stacking algorithm called Stacking-VA is implemented. The performances of four types of algorithms (unimodel, Venn-Abers predictor based on unimodel, multimodel, and Venn-Abers predictor based on multimodel) are compared in terms of validity and accuracy. Experiments are carried out on a log dataset of the Hadoop Distributed File System (HDFS). For the comparative experiments on unimodels, the results show that the validities of LR-VA and SVM-VA are better than those of the two corresponding underlying models. Compared with the underlying model, the accuracy of the SVM-VA predictor is better than that of LR-VA predictor, and more significantly, the recall rate increases from 81% to 94%. In the case of experiments on multiple models, the algorithm based on Stacking multimodel fusion is significantly superior to the underlying classifier. The average accuracy of Stacking-VA is larger than 0.95, which is more stable than the prediction results of LR-VA and SVM-VA. Experimental results show that the Venn-Abers predictor is a flexible tool that can make accurate and valid probability predictions in the field of system log anomaly detection.

scholarly journals Anomaly Detection in the Open Supernova Catalog

2019 ◽  
Author(s):  
M V Pruzhinskaya ◽  
K L Malanchev ◽  
M V Kornilov ◽  
E E O Ishida ◽  
F Mondon ◽  
...  
Keyword(s):  
Anomaly Detection ◽  
Large Scale ◽  
Physical Nature ◽  
Large Data ◽  
Machine Learning Algorithms ◽  
Classical Type ◽  
Detection Analysis ◽  
Astronomical Surveys ◽  
Type Ia ◽  
Ia Supernovae

Abstract In the upcoming decade large astronomical surveys will discover millions of transients raising unprecedented data challenges in the process. Only the use of the machine learning algorithms can process such large data volumes. Most of the discovered transients will belong to the known classes of astronomical objects. However, it is expected that some transients will be rare or completely new events of unknown physical nature. The task of finding them can be framed as an anomaly detection problem. In this work, we perform for the first time an automated anomaly detection analysis in the photometric data of the Open Supernova Catalog (OSC), which serves as a proof of concept for the applicability of these methods to future large scale surveys. The analysis consists of the following steps: 1) data selection from the OSC and approximation of the pre-processed data with Gaussian processes, 2) dimensionality reduction, 3) searching for outliers with the use of the isolation forest algorithm, 4) expert analysis of the identified outliers. The pipeline returned 81 candidate anomalies, 27 (33%) of which were confirmed to be from astrophysically peculiar objects. Found anomalies correspond to a selected sample of 1.4% of the initial automatically identified data sample of ∼2000 objects. Among the identified outliers we recognised superluminous supernovae, non-classical Type Ia supernovae, unusual Type II supernovae, one active galactic nucleus and one binary microlensing event. We also found that 16 anomalies classified as supernovae in the literature are likely to be quasars or stars. Our proposed pipeline represents an effective strategy to guarantee we shall not overlook exciting new science hidden in the data we fought so hard to acquire. All code and products of this investigation are made publicly available‡.

Sign in / Sign up

Export Citation Format

Share Document