Meteor Shower: A Reliable Stream Processing System for Commodity Data Centers

2012 ◽  
Author(s):  
Huayong Wang ◽  
Li-Shiuan Peh ◽  
Emmanouil Koukoumidis ◽  
Shao Tao ◽  
Mun Choon Chan
Keyword(s):  
Data Centers ◽  
Stream Processing ◽  
Processing System ◽  
Meteor Shower

scholarly journals s2p: Provenance Research for Stream Processing System

2021 ◽  
Vol 11 (12) ◽  
pp. 5523
Author(s):  
Qian Ye ◽  
Minyan Lu
Keyword(s):  
Data Storage ◽  
Stream Processing ◽  
Processing System ◽  
Coarse Grained ◽  
Stream Data ◽  
Related Data ◽  
Dsp System ◽  
Provenance Research ◽  
Dsp Systems ◽  
Abnormal Results

The main purpose of our provenance research for DSP (distributed stream processing) systems is to analyze abnormal results. Provenance for these systems is not nontrivial because of the ephemerality of stream data and instant data processing mode in modern DSP systems. Challenges include but are not limited to an optimization solution for avoiding excessive runtime overhead, reducing provenance-related data storage, and providing it in an easy-to-use fashion. Without any prior knowledge about which kinds of data may finally lead to the abnormal, we have to track all transformations in detail, which potentially causes hard system burden. This paper proposes s2p (Stream Process Provenance), which mainly consists of online provenance and offline provenance, to provide fine- and coarse-grained provenance in different precision. We base our design of s2p on the fact that, for a mature online DSP system, the abnormal results are rare, and the results that require a detailed analysis are even rarer. We also consider state transition in our provenance explanation. We implement s2p on Apache Flink named as s2p-flink and conduct three experiments to evaluate its scalability, efficiency, and overhead from end-to-end cost, throughput, and space overhead. Our evaluation shows that s2p-flink incurs a 13% to 32% cost overhead, 11% to 24% decline in throughput, and few additional space costs in the online provenance phase. Experiments also demonstrates the s2p-flink can scale well. A case study is presented to demonstrate the feasibility of the whole s2p solution.

Architecture of a stream processing system

2014 ◽  
pp. 203-217
Author(s):  
Henrique Andrade ◽  
Bugra Gedik ◽  
Deepak Turaga
Keyword(s):  
Stream Processing ◽  
Processing System

A demonstration of the MaxStream federated stream processing system

2010 ◽  
Author(s):  
Irina Botan ◽  
Younggoo Cho ◽  
Roozbeh Derakhshan ◽  
Nihal Dindar ◽  
Ankush Gupta ◽  
...  
Keyword(s):  
Stream Processing ◽  
Processing System

Alovera: A Fast Stream Processing System for Large-Scale Data

2013 ◽  
Author(s):  
Zhen'an Zhang ◽  
Dongjie Zhang ◽  
Xiaopeng Yu ◽  
Jing Wang ◽  
Chunjiang He ◽  
...  
Keyword(s):  
Large Scale ◽  
Stream Processing ◽  
Processing System ◽  
Large Scale Data ◽  
Scale Data

scholarly journals Fault-tolerance in the borealis distributed stream processing system

2008 ◽  
Vol 33 (1) ◽  
pp. 1-44 ◽  
Author(s):  
Magdalena Balazinska ◽  
Hari Balakrishnan ◽  
Samuel R. Madden ◽  
Michael Stonebraker
Keyword(s):  
Fault Tolerance ◽  
Stream Processing ◽  
Processing System ◽  
Distributed Stream Processing
Sign in / Sign up

Export Citation Format

Share Document