scholarly journals Probabilistic Hesitant Fuzzy Methods for Prioritizing Distributed Stream Processing Frameworks for IoT Applications

2021 ◽  
Vol 2021 ◽  
pp. 1-12
Author(s):  
Zhimin Lin ◽  
Chao Huang ◽  
Mingwei Lin
Keyword(s):  
Large Scale ◽  
Evaluation Criteria ◽  
Stream Processing ◽  
Multicriteria Decision Making ◽  
Practical Case ◽  
Time Data ◽  
Iot Applications ◽  
Definition Of ◽  
Real Time Data Processing ◽  
Distributed Stream Processing

Distributed stream processing frameworks (DSPFs) are the vital engine, which can handle real-time data processing and analytics for IoT applications. How to prioritize DSPFs and select the most suitable one for special IoT applications is an open issue. To help developers of IoT applications to solve this complex issue, a novel probabilistic hesitant fuzzy multicriteria decision making (MCDM) model is put forward in this paper. To characterize the requirements for large-scale IoT data stream processing, a novel evaluation criteria system including qualitative and quantitative criteria is established. To accurately model the collective opinions from skilled developers and consider their psychological distance, the definition of probabilistic hesitant fuzzy sets (PHFSs) is used. To derive the importance degrees of criteria, a novel probabilistic hesitant fuzzy best-worst (PHFBW) method is proposed based on the score value. To prioritize the DSPFs and choose the most suitable one, a novel probabilistic hesitant fuzzy MULTIMOORA method is put forward. Finally, a practical case composed of four Apache stream processing frameworks, namely, Storm, Flink, Spark, and Samza, is studied. The obtained results indicate that throughput, latency, and reliability are considered to be the three most important criteria, and Flink is the most suitable stream framework.

scholarly journals A Metadata and Z Score-based Load-Shedding Technique in IoT-based Data Collection Systems

2020 ◽  
Vol 6 (1) ◽  
pp. 363-382
Author(s):  
Mario José Diván ◽  
María Laura Sánchez-Reynoso
Keyword(s):  
Data Collection ◽  
Stream Processing ◽  
Original Data ◽  
Load Shedding ◽  
Distributed Data ◽  
Z Score ◽  
Time Data ◽  
Measurement Framework ◽  
Real Time Data Processing ◽  
Future Work

The Internet-of-Things (IoT) has emerged as an alternative to communicate different pieces of technology to foster the distributed data collection. The measurement projects and the Real-time data processing are articulated to take advantage of this environment, fostering a sustainable data-driven decision making. The Data Stream Processing Strategy (DSPS) is a Stream Processing Engine focused on measurement projects, where each concept is previously agreed through a measurement framework. The Measurement Adapter (MA) is a component whose responsibility is to pair each metric’s definition from the measurement project with data sensors to transmit data (i.e., measures) with metadata (i.e., tags indicating the data meaning) together. The Gathering Function (GF) receives and derivates data for its processing from each MA, while it implements load-shedding (LS) techniques based on Metadata to avoid a processing collapse when all MAs informs jointly and frequently. Here, a Metadata and Z-score based load-shedding technique implemented locally in the MA is proposed. Thus, the load-shedding is located at the same data source to avoid data transmission and saving resources. Also, an incremental estimation of average, deviations, covariance, and correlations are implemented and employed to calculate the Z-scores and to data retain/discard selectively. Four simulations discrete were designed and performed to analyze the proposal. Results indicate that the local LS required only 24% of the original data transmissions, a minimum of 18.61 ms as the data lifespan, while it consumes 890.26 KB. As future work, other kinds of dependencies analysis will be analyzed to provide local alternatives to LS.

INS real-time data processing composite system using large scale general purpose computers

1983 ◽  
Vol 213 (2-3) ◽  
pp. 317-327 ◽  
Author(s):  
Jun Kokame ◽  
Motonobu Takano ◽  
Tomoko Oshikubo ◽  
Kurazo Chiba ◽  
Kumataro Ukai ◽  
...  
Keyword(s):  
Data Processing ◽  
Real Time ◽  
Large Scale ◽  
Composite System ◽  
General Purpose ◽  
Time Data ◽  
Real Time Data ◽  
Real Time Data Processing

scholarly journals Distributed stream processing for genomics pipelines

2017 ◽  
Author(s):  
Francesco Versaci ◽  
Luca Pireddu ◽  
Gianluigi Zanetti
Keyword(s):  
Personalized Medicine ◽  
Large Scale ◽  
Fault Tolerant ◽  
Great Part ◽  
Stream Processing ◽  
Sequencing Data ◽  
Modern Biology ◽  
Scale Population ◽  
Conventional Solution ◽  
Distributed Stream Processing

Personalized medicine is in great part enabled by the progress in data acquisition technologies for modern biology, such as next-generation sequencing (NGS). Conventional NGS processing workflows are composed by independent tools implementing shared-memory parallelism which communicate by means of intermediate files. With increasing data sizes this approach is showing its limited scalability and robustness characteristics – problems that make it unsuitable for large-scale, population-wide personalized medicine applications. In this work we propose the adoption of the stream computing architecture to make the genomics pipeline more scalable, and fault-tolerant. We implemented the first processing phases for Illumina sequencing data – from raw data to alignment – using the Apache Flink distributed stream processing framework and Apache Kafka. The new pipeline has been tested processing the raw output of an Illumina HiSeq3000 sequencer and producing aligned reads in CRAM format. The results show near optimal scalability characteristics on experiments from 1 to 12 computing nodes, with a speed-up of 9.5x over the conventional solution (which cannot automatically run on multiple nodes). This result is particularly positive considering that the very short runtime of the experiment – less than 15 minutes – makes significant the constant time costs imposed by the overheads of the frameworks.

Sign in / Sign up

Export Citation Format

Share Document