Parallel outlier detection in real time data streams

In pursuit of real-time data analysis, approximate summarization structures, i.e., synopses, have gained importance over the years. However, existing stream processing systems, such as Flink, Spark, and Storm, do not support synopses as first class citizens, i.e., as pipeline operators. Synopses' implementation is upon users. This is mainly because of the diversity of synopses, which makes a unified implementation difficult. We present Condor, a framework that supports synopses as first class citizens. Condor facilitates the specification and processing of synopsis-based streaming jobs while hiding all internal processing details. Condor's key component is its model that represents synopses as a particular case of windowed aggregate functions. An inherent divide and conquer strategy allows Condor to efficiently distribute the computation, allowing for high-performance and linear scalability. Our evaluation shows that Condor outperforms existing approaches by up to a factor of 75x and that it scales linearly with the number of cores.

Download Full-text

Visual Analysis of Dynamic Data Streams

Information Visualization ◽

10.1057/ivs.2009.18 ◽

2009 ◽

Vol 8 (3) ◽

pp. 212-229 ◽

Cited By ~ 8

Author(s):

George Chin ◽

Mudita Singhal ◽

Grant Nakamura ◽

Vidhya Gurumoorthi ◽

Natalie Freeman-Cadoret

Keyword(s):

Real Time ◽

Data Streams ◽

High Volume ◽

Scientific Data ◽

General Strategy ◽

Dynamic Visualization ◽

Time Data ◽

Dynamic Data ◽

Intensive Training ◽

Real Time Data

For scientific data visualizations, real-time data streams present many interesting challenges when compared to static data. Real-time data are dynamic, transient, high-volume and temporal. Effective visualizations need to be able to accommodate dynamic data behavior as well as Abstract and present the data in ways that make sense to and are usable by humans. The Visual Content Analysis of Real-Time Data Streams project at the Pacific Northwest National Laboratory is researching and prototyping dynamic visualization techniques and tools to help facilitate human understanding and comprehension of high-volume, real-time data. The general strategy of the project is to develop and evolve visual contexts that will organize and orient high-volume dynamic data in conceptual and perceptive views. The goal is to allow users to quickly grasp dynamic data in forms that are intuitive and natural without requiring intensive training in the use of specific visualization or analysis tools and methods. Thus far, the project has prototyped five different visualization prototypes that represent and convey dynamic data through human-recognizable contexts and paradigms such as hierarchies, relationships, time and geography. We describe the design considerations and unique features of these dynamic visualization prototypes as well as our findings in the exploration and evaluation of their use.

Download Full-text

Assessing the transmission performance of iGMAS real-time data streams

MATEC Web of Conferences ◽

10.1051/matecconf/201711901031 ◽

2017 ◽

Vol 119 ◽

pp. 01031

Author(s):

Zhanke He ◽

Wenda Tang ◽

Xuhai Yang ◽

Liming Wang

Keyword(s):

Real Time ◽

Data Streams ◽

Transmission Performance ◽

Time Data ◽

Real Time Data

Download Full-text

MCDAStream: a real-time data stream clustering based on micro-cluster density and attraction

International Journal of Engineering & Technology ◽

10.14419/ijet.v7i2.9051 ◽

2018 ◽

Vol 7 (2) ◽

pp. 270 ◽

Cited By ~ 1

Author(s):

Shyam Sunder Reddy K ◽

Shoba Bindu C

Keyword(s):

Real Time ◽

Data Streams ◽

Data Stream ◽

Clustering Algorithm ◽

Positional Information ◽

Time Data ◽

Stream Clustering ◽

Real Time Data ◽

Cluster Density ◽

Data Stream Clustering

Real-time data stream clustering has been widely used in many fields, and it can extract useful information from massive sets of data. Most of the existing density-based algorithms cluster the data streams based on the density within the micro-clusters. These algorithms completely omit the data density in the area between the micro-clusters and recluster the micro-clusters based on erroneous assumptions about the distribution of the data within and between the micro-clusters that lead to poor clustering results. This paper describes a novel density-based clustering algorithm for evolving data streams called MCDAStream, which clusters the data stream based on micro-cluster density and attraction between the micro-clusters. The attraction of micro-clusters characterizes the positional information of the data points in each micro-cluster. We generate better clustering results by considering both micro-cluster density and attraction of micro-clusters. The quality of the proposed algorithm is evaluated on various synthetic and real-time datasets with distinct characteristics and quality metrics.

Download Full-text

Dynamic redirection of real-time data streams for elastic stream computing

Future Generation Computer Systems ◽

10.1016/j.future.2020.05.021 ◽

2020 ◽

Vol 112 ◽

pp. 193-208

Author(s):

Dawei Sun ◽

Shang Gao ◽

Xunyun Liu ◽

Xindong You ◽

Rajkumar Buyya

Keyword(s):

Real Time ◽

Data Streams ◽

Stream Computing ◽

Time Data ◽

Real Time Data

Download Full-text