Resolve Hotspots and Load Imbalance Problem in Event Stream Processing System

2013 ◽  
Author(s):  
Baojian Zhou ◽  
Zhongzhi Luan ◽  
JieQian Wu ◽  
Ming Xie
Keyword(s):  
Stream Processing ◽  
Processing System ◽  
Event Stream ◽  
Imbalance Problem ◽  
Load Imbalance

scholarly journals An Architectural Guide and Design of StreamEPS

2021 ◽  
Author(s):  
Frank Appiah
Keyword(s):  
Open Source ◽  
Stream Processing ◽  
Processing System ◽  
Event Stream ◽  
Source Event

<i>This is about the overall functionality and complexity (size) of the open source event stream processing system or StreamEPS for short. The elements of the platform will be functional if the design follows application interfaces as described in this work. The engine architecture details the overall functionality in terms of engine core, engine context, engine processing and of itself.</i>

scholarly journals Development of an Event Stream Processing System for the Vehicle Telematics Environment

ETRI Journal ◽  
2009 ◽  
Vol 31 (4) ◽  
pp. 463-465 ◽  
Author(s):  
Jongik Kim ◽  
Oh-Cheon Kwon ◽  
Hyunsuk Kim
Keyword(s):  
Stream Processing ◽  
Processing System ◽  
Event Stream

scholarly journals An Architectural Guide and Design of StreamEPS

2021 ◽  
Author(s):  
Frank Appiah
Keyword(s):  
Open Source ◽  
Stream Processing ◽  
Processing System ◽  
Event Stream ◽  
Source Event

<i>This is about the overall functionality and complexity (size) of the open source event stream processing system or StreamEPS for short. The elements of the platform will be functional if the design follows application interfaces as described in this work. The engine architecture details the overall functionality in terms of engine core, engine context, engine processing and of itself.</i>

scholarly journals A QoS-Latency Aware Event Stream Processing with Elastic-FaaS

2019 ◽  
Vol 8 (10) ◽  
pp. 3756-3762
Keyword(s):  
Real Time ◽  
Virtual Machines ◽  
Stream Processing ◽  
Processing System ◽  
Low Latency ◽  
Event Stream ◽  
Huge Impact ◽  
Time Required ◽  
The Right ◽  
Cloud Technologies

Stream processing systems need to be elastically scalable to process and respond the unpredictable massive load spike in real-time with high throughput and low latency. Though the modern cloud technologies can help in elastically provisioning the required computing resources on-the-fly, finding out the right point-in-time varies among systems based on their expected QoS characteristics. The latency sensitivity of the stream processing applications varies based on their nature and pre-set requirements. For few applications, even a little latency in the response will have huge impact, whereas for others the little latency will not have that much impact. For the former ones, the processing systems are expected to be highly available, elastically scalable, and fast enough to perform, whenever there is a spike. The time required to elasticity provision the systems under FaaS is very high, comparing to provisioning the Virtual Machines and Containers. However, the current FaaS systems have some limitations that need to be overcome to handle the unexpected spike in real-time. This paper proposes a new algorithm called Elastic-FaaS on top of the existing FaaS to overcome this QoS latency issue. Our proposed algorithm will provision required number of FaaS container instances than any typical FaaS can provision normally, whenever there is a demand to avoid the latency issue. We have experimented our algorithm with an event stream processing system and the result shows that our proposed Elastic-FaaS algorithm performs better than typical FaaS by improving the throughput that meets the high accuracy and low latency requirements.

scholarly journals s2p: Provenance Research for Stream Processing System

2021 ◽  
Vol 11 (12) ◽  
pp. 5523
Author(s):  
Qian Ye ◽  
Minyan Lu
Keyword(s):  
Data Storage ◽  
Stream Processing ◽  
Processing System ◽  
Coarse Grained ◽  
Stream Data ◽  
Related Data ◽  
Dsp System ◽  
Provenance Research ◽  
Dsp Systems ◽  
Abnormal Results

The main purpose of our provenance research for DSP (distributed stream processing) systems is to analyze abnormal results. Provenance for these systems is not nontrivial because of the ephemerality of stream data and instant data processing mode in modern DSP systems. Challenges include but are not limited to an optimization solution for avoiding excessive runtime overhead, reducing provenance-related data storage, and providing it in an easy-to-use fashion. Without any prior knowledge about which kinds of data may finally lead to the abnormal, we have to track all transformations in detail, which potentially causes hard system burden. This paper proposes s2p (Stream Process Provenance), which mainly consists of online provenance and offline provenance, to provide fine- and coarse-grained provenance in different precision. We base our design of s2p on the fact that, for a mature online DSP system, the abnormal results are rare, and the results that require a detailed analysis are even rarer. We also consider state transition in our provenance explanation. We implement s2p on Apache Flink named as s2p-flink and conduct three experiments to evaluate its scalability, efficiency, and overhead from end-to-end cost, throughput, and space overhead. Our evaluation shows that s2p-flink incurs a 13% to 32% cost overhead, 11% to 24% decline in throughput, and few additional space costs in the online provenance phase. Experiments also demonstrates the s2p-flink can scale well. A case study is presented to demonstrate the feasibility of the whole s2p solution.

Architecture of a stream processing system

2014 ◽  
pp. 203-217
Author(s):  
Henrique Andrade ◽  
Bugra Gedik ◽  
Deepak Turaga
Keyword(s):  
Stream Processing ◽  
Processing System

scholarly journals Prefix Imputation of Orphan Events in Event Stream Processing

2021 ◽  
Vol 4 ◽  
Author(s):  
Rashid Zaman ◽  
Marwan Hassani ◽  
Boudewijn F. Van Dongen
Keyword(s):  
Process Model ◽  
Process Mining ◽  
State Of The Art ◽  
Processing System ◽  
Conformance Checking ◽  
Event Stream ◽  
Window Width ◽  
Event Logs ◽  
Relevant Case ◽  
Log File

In the context of process mining, event logs consist of process instances called cases. Conformance checking is a process mining task that inspects whether a log file is conformant with an existing process model. This inspection is additionally quantifying the conformance in an explainable manner. Online conformance checking processes streaming event logs by having precise insights into the running cases and timely mitigating non-conformance, if any. State-of-the-art online conformance checking approaches bound the memory by either delimiting storage of the events per case or limiting the number of cases to a specific window width. The former technique still requires unbounded memory as the number of cases to store is unlimited, while the latter technique forgets running, not yet concluded, cases to conform to the limited window width. Consequently, the processing system may later encounter events that represent some intermediate activity as per the process model and for which the relevant case has been forgotten, to be referred to as orphan events. The naïve approach to cope with an orphan event is to either neglect its relevant case for conformance checking or treat it as an altogether new case. However, this might result in misleading process insights, for instance, overestimated non-conformance. In order to bound memory yet effectively incorporate the orphan events into processing, we propose an imputation of missing-prefix approach for such orphan events. Our approach utilizes the existing process model for imputing the missing prefix. Furthermore, we leverage the case storage management to increase the accuracy of the prefix prediction. We propose a systematic forgetting mechanism that distinguishes and forgets the cases that can be reliably regenerated as prefix upon receipt of their future orphan event. We evaluate the efficacy of our proposed approach through multiple experiments with synthetic and three real event logs while simulating a streaming setting. Our approach achieves considerably higher realistic conformance statistics than the state of the art while requiring the same storage.

Sign in / Sign up

Export Citation Format

Share Document