SciDFS: An In-Situ Processing System for Scientific Array Data Based on Distributed File System

2018 ◽  
Author(s):  
Donghyoung Han ◽  
Yoon-Min Nam ◽  
Min-Soo Kim ◽  
Kyongseok Park ◽  
Sunggeun Han
Keyword(s):  
File System ◽  
Processing System ◽  
Distributed File System ◽  
Array Data ◽  
In Situ Processing

scholarly journals A Seismic Data Processing System based on Fast Distributed File System

2015 ◽  
Vol 14 (5) ◽  
pp. 5779-5788
Author(s):  
Jun Li ◽  
Changsen Pan ◽  
Menghan Lu
Keyword(s):  
Data Storage ◽  
Seismic Data ◽  
File System ◽  
Processing System ◽  
Social Needs ◽  
Distributed File System ◽  
Seismic Exploration ◽  
Seismic Data Processing ◽  
The Social ◽  
Effective System

Big data has attracted an increasingly number of attentions with the advent of the cloud era, and in the field of seismic exploration, the amount of data created by seismic exploration has also experienced an incredible growth in order to satisfy the social needs. In this case, it is necessary to build a highly-effective system of data storage and process. In our paper, we aim at the properties of the seismic data and the requirement to the performance of IO, and establish a distributed file system with the goal of processing seismic data based on the Fast Distributed File System (Fast DFS), then test our system through a series of operations such as file write and read, and the results show that our file system is very proper and effective when processing seismic data.

scholarly journals A Simple Measuring Model for Evaluating the Performance of Small Block Size Accesses in Lustre File System

2017 ◽  
Vol 7 (6) ◽  
pp. 2313-2318
Author(s):  
N. Jayakumar ◽  
A. M. Kulkarni
Keyword(s):  
Big Data ◽  
File System ◽  
File Systems ◽  
Block Size ◽  
Storage Virtualization ◽  
Hadoop Mapreduce ◽  
Parallel Data ◽  
Hadoop Distributed File System ◽  
In Situ Processing

Storage performance is one of the vital characteristics of a big data environment. Data throughput can be increased to some extent using storage virtualization and parallel data paths. Technology has enhanced the various SANs and storage topologies to be adaptable for diverse applications that improve end to end performance. In big data environments the mostly used file systems are HDFS (Hadoop Distributed File System) and Lustre. There are environments in which both HDFS and Lustre are connected, and the applications directly work on Lustre. In Lustre architecture with out-of-band storage virtualization system, the separation of data path from metadata path is acceptable (and even desirable) for large files since one MDT (Metadata Target) open RPC is typically a small fraction of the total number of read or write RPCs. This hurts small file performance significantly when there is only a single read or write RPC for the file data. Since applications require data for processing and considering in-situ architecture which brings data or metadata close to applications for processing, how the in-situ processing can be exploited in Lustre is the domain of this dissertation work. The earlier research exploited Lustre supporting in-situ processing when Hadoop/MapReduce is integrated with Lustre, but still, the scope of performance improvement existed in Lustre. The aim of the research is to check whether it is feasible and beneficial to move the small files to the MDT so that additional RPCs and I/O overhead can be eliminated, and read/write performance of Lustre file system can be improved.

Energy-Efficient Algorithms for Distributed File System HDFS

2014 ◽  
Vol 36 (5) ◽  
pp. 1047-1064 ◽  
Author(s):  
Bin LIAO ◽  
Jiong YU ◽  
Tao ZHANG ◽  
Xing-Yao YANG
Keyword(s):  
Energy Efficient ◽  
File System ◽  
Efficient Algorithms ◽  
Distributed File System

Performance Testing and Analysis to Storage Medium of Distributed File System

2010 ◽  
Vol 33 (10) ◽  
pp. 1873-1880 ◽  
Author(s):  
Chun-Cong XU ◽  
Xiao-Meng HUANG ◽  
Nuo WU ◽  
Ning-Wei SUN ◽  
Guang-Wen YANG
Keyword(s):  
File System ◽  
Performance Testing ◽  
Distributed File System ◽  
Storage Medium

Improving downloading performance in hadoop distributed file system

2010 ◽  
Vol 30 (8) ◽  
pp. 2060-2065 ◽  
Author(s):  
Ning CAO ◽  
Zhong-hai WU ◽  
Hong-zhi LIU ◽  
Qi-xun ZHANG
Keyword(s):  
File System ◽  
Distributed File System ◽  
Hadoop Distributed File System
Sign in / Sign up

Export Citation Format

Share Document