Kafka interfaces for composable streaming genomics pipelines

Mapping Intimacies ◽

10.1101/182030 ◽

2017 ◽

Author(s):

Francesco Versaci ◽

Luca Pireddu ◽

Gianluigi Zanetti

Keyword(s):

Fault Tolerance ◽

Shared Memory ◽

Real Data ◽

Stream Computing ◽

Constant Increase ◽

Processing Pipeline ◽

Memory Tools

AbstractModern sequencing machines produce order of a terabyte of data per day, which need subsequently to go through a complex processing pipeline. The standard workflow begins with a few independent, shared-memory tools, which communicate by means of intermediate files. Given the constant increase of the amount of data produced, this approach is proving more and more unmanageable, due to its lack of robustness and scalability.In this work we propose the adoption of stream computing to simplify the genomic pipeline, boost its performance and improve its fault-tolerance. We decompose the first steps of the genomic processing in two distinct and specialized modules (preprocessing and alignment) and we loosely compose them via communication through Kafka streams, in order to allow for easy composability and integration in the already existing Hadoop-based pipelines. The proposed solution is then experimentally validated on real data and shown to scale almost linearly.

Download Full-text

Survey of fault tolerance techniques for shared memory multicore/multiprocessor systems

2011 IEEE 6th International Design and Test Workshop (IDT) ◽

10.1109/idt.2011.6123094 ◽

2011 ◽

Cited By ~ 14

Author(s):

Hamid Mushtaq ◽

Zaid Al-Ars ◽

Koen Bertels

Keyword(s):

Fault Tolerance ◽

Shared Memory ◽

Multiprocessor Systems

Download Full-text

Performance Improvement IoT Applications Through Multimedia Analytics Using Big Data Stream Computing Platforms

Exploring the Convergence of Big Data and the Internet of Things - Advances in Data Mining and Database Management ◽

10.4018/978-1-5225-2947-7.ch015 ◽

2018 ◽

pp. 200-221

Author(s):

Rizwan Patan ◽

Rajasekhara Babu M ◽

Suresh Kallam

Keyword(s):

Big Data ◽

Real Time ◽

Performance Improvement ◽

Data Stream ◽

Real Data ◽

Stream Computing ◽

Time Data ◽

Real Time Data ◽

Computing Platforms ◽

Time And Energy

A Big Data Stream Computing (BDSC) Platform handles real-time data from various applications such as risk management, marketing management and business intelligence. Now a days Internet of Things (IoT) deployment is increasing massively in all the areas. These IoTs engender real-time data for analysis. Existing BDSC is inefficient to handle Real-data stream from IoTs because the data stream from IoTs is unstructured and has inconstant velocity. So, it is challenging to handle such real-time data stream. This work proposes a framework that handles real-time data stream through device control techniques to improve the performance. The frame work includes three layers. First layer deals with Big Data platforms that handles real data streams based on area of importance. Second layer is performance layer which deals with performance issues such as low response time, and energy efficiency. The third layer is meant for Applying developed method on existing BDSC platform. The experimental results have been shown a performance improvement 20%-30% for real time data stream from IoT application.

Download Full-text

A Comparative Analysis of Performance of Shared Memory Cluster Computing Interconnection Systems

Journal of Computer Networks and Communications ◽

10.1155/2014/128438 ◽

2014 ◽

Vol 2014 ◽

pp. 1-9

Author(s):

Minakshi Tripathy ◽

C. R. Tripathy

Keyword(s):

Comparative Analysis ◽

Fault Tolerance ◽

Load Balancing ◽

Shared Memory ◽

Cluster Computing ◽

Distributed Shared Memory ◽

System Size ◽

Cluster Architecture ◽

Analysis Of Performance ◽

Made In

In recent past, many types of shared memory cluster computing interconnection systems have been proposed. Each of these systems has its own advantages and limitations. With the increase in system size of the cluster interconnection systems, the comparative analysis of their various performance measures becomes quite inevitable. The cluster architecture, load balancing, and fault tolerance are some of the important aspects, which need to be addressed. The comparison needs to be made in order to choose the best one for a particular application. In this paper, a detailed comparative study on four important and different classes of shared memory cluster architectures has been made. The systems taken up for the purpose of the study are shared memory clusters, hierarchical shared memory clusters, distributed shared memory clusters, and the virtual distributed shared memory clusters. These clusters are analyzed and compared on the basis of the architecture, load balancing, and fault tolerance aspects. The results of comparison are reported.

Download Full-text