Proceedings of the 2nd Joint International Workshop on Parallel Data Storage & Data Intensive Scalable Computing Systems - PDSW-DISCS '17

2017 ◽  
Keyword(s):  
Data Storage ◽  
International Workshop ◽  
Computing Systems ◽  
Scalable Computing ◽  
Data Intensive ◽  
Parallel Data

Overview of Big Data-Intensive Storage and its Technologies for Cloud and Fog Computing

2021 ◽  
pp. 112-153
Author(s):  
Richard S. Segall ◽  
Jeffrey S Cook ◽  
Gao Niu
Keyword(s):  
Big Data ◽  
Data Storage ◽  
High Performance ◽  
Storage Systems ◽  
Fog Computing ◽  
Storage Management ◽  
Data Intensive Computing ◽  
Computing Systems ◽  
Application Performance ◽  
Data Intensive

Computing systems are becoming increasingly data-intensive because of the explosion of data and the needs for processing the data, and subsequently storage management is critical to application performance in such data-intensive computing systems. However, if existing resource management frameworks in these systems lack the support for storage management, this would cause unpredictable performance degradation when applications are under input/output (I/O) contention. Storage management of data-intensive systems is a challenge. Big Data plays a most major role in storage systems for data-intensive computing. This article deals with these difficulties along with discussion of High Performance Computing (HPC) systems, background for storage systems for data-intensive applications, storage patterns and storage mechanisms for Big Data, the Top 10 Cloud Storage Systems for data-intensive computing in today's world, and the interface between Big Data Intensive Storage and Cloud/Fog Computing. Big Data storage and its server statistics and usage distributions for the Top 500 Supercomputers in the world are also presented graphically and discussed as data-intensive storage components that can be interfaced with Fog-to-cloud interactions and enabling protocols.

scholarly journals Performance-efficient Recommendation and Prediction Service for Big Data frameworks focusing on Data Compression and In-memory Data Storage Indicators

2021 ◽  
Vol 22 (4) ◽  
pp. 401-412
Author(s):  
Hrachya Astsatryan ◽  
Arthur Lalayan ◽  
Aram Kocharyan ◽  
Daniel Hagimont
Keyword(s):  
Big Data ◽  
Data Compression ◽  
Data Storage ◽  
File Systems ◽  
Large Datasets ◽  
Data Sets ◽  
Mapreduce Framework ◽  
Data Intensive ◽  
Parallel Data ◽  
Data Intensive Applications

The MapReduce framework manages Big Data sets by splitting the large datasets into a set of distributed blocks and processes them in parallel. Data compression and in-memory file systems are widely used methods in Big Data processing to reduce resource-intensive I/O operations and improve I/O rate correspondingly. The article presents a performance-efficient modular and configurable decision-making robust service relying on data compression and in-memory data storage indicators. The service consists of Recommendation and Prediction modules, predicts the execution time of a given job based on metrics, and recommends the best configuration parameters to improve Hadoop and Spark frameworks' performance. Several CPU and data-intensive applications and micro-benchmarks have been evaluated to improve the performance, including Log Analyzer, WordCount, and K-Means.

Overview of Big Data-Intensive Storage and its Technologies for Cloud and Fog Computing

2019 ◽  
Vol 2 (1) ◽  
pp. 74-113 ◽  
Author(s):  
Richard S. Segall ◽  
Jeffrey S Cook ◽  
Gao Niu
Keyword(s):  
Big Data ◽  
Data Storage ◽  
High Performance ◽  
Storage Systems ◽  
Fog Computing ◽  
Storage Management ◽  
Data Intensive Computing ◽  
Computing Systems ◽  
Application Performance ◽  
Data Intensive

Computing systems are becoming increasingly data-intensive because of the explosion of data and the needs for processing the data, and subsequently storage management is critical to application performance in such data-intensive computing systems. However, if existing resource management frameworks in these systems lack the support for storage management, this would cause unpredictable performance degradation when applications are under input/output (I/O) contention. Storage management of data-intensive systems is a challenge. Big Data plays a most major role in storage systems for data-intensive computing. This article deals with these difficulties along with discussion of High Performance Computing (HPC) systems, background for storage systems for data-intensive applications, storage patterns and storage mechanisms for Big Data, the Top 10 Cloud Storage Systems for data-intensive computing in today's world, and the interface between Big Data Intensive Storage and Cloud/Fog Computing. Big Data storage and its server statistics and usage distributions for the Top 500 Supercomputers in the world are also presented graphically and discussed as data-intensive storage components that can be interfaced with Fog-to-cloud interactions and enabling protocols.

Sign in / Sign up

Export Citation Format

Share Document