scholarly journals A Scalable Architecture for Real-Time Stream Processing of Spatiotemporal IoT Stream Data—Performance Analysis on the Example of Map Matching

2018 ◽  
Vol 7 (7) ◽  
pp. 238 ◽  
Author(s):  
Marius Laska ◽  
Stefan Herle ◽  
Ralf Klamma ◽  
Jörg Blankenbach
Keyword(s):  
Performance Analysis ◽  
Real Time ◽  
Stream Processing ◽  
Map Matching ◽  
Stream Data ◽  
Scalable Architecture

Algorithm on real-time map-matching of multi-weight probability

2013 ◽  
Vol 26 (2) ◽  
pp. 166-170
Author(s):  
Yang Li ◽  
Xiaodong Zhang ◽  
Yuanlv Bao
Keyword(s):  
Real Time ◽  
Map Matching ◽  
Time Map

scholarly journals s2p: Provenance Research for Stream Processing System

2021 ◽  
Vol 11 (12) ◽  
pp. 5523
Author(s):  
Qian Ye ◽  
Minyan Lu
Keyword(s):  
Data Storage ◽  
Stream Processing ◽  
Processing System ◽  
Coarse Grained ◽  
Stream Data ◽  
Related Data ◽  
Dsp System ◽  
Provenance Research ◽  
Dsp Systems ◽  
Abnormal Results

The main purpose of our provenance research for DSP (distributed stream processing) systems is to analyze abnormal results. Provenance for these systems is not nontrivial because of the ephemerality of stream data and instant data processing mode in modern DSP systems. Challenges include but are not limited to an optimization solution for avoiding excessive runtime overhead, reducing provenance-related data storage, and providing it in an easy-to-use fashion. Without any prior knowledge about which kinds of data may finally lead to the abnormal, we have to track all transformations in detail, which potentially causes hard system burden. This paper proposes s2p (Stream Process Provenance), which mainly consists of online provenance and offline provenance, to provide fine- and coarse-grained provenance in different precision. We base our design of s2p on the fact that, for a mature online DSP system, the abnormal results are rare, and the results that require a detailed analysis are even rarer. We also consider state transition in our provenance explanation. We implement s2p on Apache Flink named as s2p-flink and conduct three experiments to evaluate its scalability, efficiency, and overhead from end-to-end cost, throughput, and space overhead. Our evaluation shows that s2p-flink incurs a 13% to 32% cost overhead, 11% to 24% decline in throughput, and few additional space costs in the online provenance phase. Experiments also demonstrates the s2p-flink can scale well. A case study is presented to demonstrate the feasibility of the whole s2p solution.

scholarly journals Stream Data Load Prediction for Resource Scaling Using Online Support Vector Regression

Algorithms ◽  
2019 ◽  
Vol 12 (2) ◽  
pp. 37 ◽  
Author(s):  
Zhigang Hu ◽  
Hui Kang ◽  
Meiguang Zheng
Keyword(s):  
Support Vector Regression ◽  
Real Time ◽  
Virtual Machines ◽  
Time Window ◽  
Performance Model ◽  
Streaming Data ◽  
Support Vector ◽  
Load Prediction ◽  
Stream Data ◽  
Online Support Vector Regression

A distributed data stream processing system handles real-time, changeable and sudden streaming data load. Its elastic resource allocation has become a fundamental and challenging problem with a fixed strategy that will result in waste of resources or a reduction in QoS (quality of service). Spark Streaming as an emerging system has been developed to process real time stream data analytics by using micro-batch approach. In this paper, first, we propose an improved SVR (support vector regression) based stream data load prediction scheme. Then, we design a spark-based maximum sustainable throughput of time window (MSTW) performance model to find the optimized number of virtual machines. Finally, we present a resource scaling algorithm TWRES (time window resource elasticity scaling algorithm) with MSTW constraint and streaming data load prediction. The evaluation results show that TWRES could improve resource utilization and mitigate SLA (service level agreement) violation.

Sign in / Sign up

Export Citation Format

Share Document