scholarly journals Erasure-Coding-Based Storage and Recovery for Distributed Exascale Storage Systems

2021 ◽  
Vol 11 (8) ◽  
pp. 3298
Author(s):  
Jeong-Joon Kim
Keyword(s):  
File System ◽  
File Systems ◽  
Data Availability ◽  
Distributed File System ◽  
Distributed File Systems ◽  
Erasure Coding ◽  
Input Output ◽  
Space Efficiency ◽  
Random Block ◽  
Replication Technique

Various techniques have been used in distributed file systems for data availability and stability. Typically, a method for storing data in a replication technique-based distributed file system is used, but due to the problem of space efficiency, an erasure-coding (EC) technique has been utilized more recently. The EC technique improves the space efficiency problem more than the replication technique does. However, the EC technique has various performance degradation factors, such as encoding and decoding and input and output (I/O) degradation. Thus, this study proposes a buffering and combining technique in which various I/O requests that occurred during encoding in an EC-based distributed file system are combined into one and processed. In addition, it proposes four recovery measures (disk input/output load distribution, random block layout, multi-thread-based parallel recovery, and matrix recycle technique) to distribute the disk input/output loads generated during decoding.

scholarly journals Modeling of distributed file System in big data storage by event- B

2018 ◽  
Vol 210 ◽  
pp. 04042
Author(s):  
Ammar Alhaj Ali ◽  
Pavel Varacha ◽  
Said Krayem ◽  
Roman Jasek ◽  
Petr Zacek ◽  
...  
Keyword(s):  
Big Data ◽  
Data Storage ◽  
High Performance ◽  
File System ◽  
Formal Method ◽  
File Systems ◽  
Distributed File System ◽  
Distributed File Systems ◽  
Data Systems ◽  
Big Data Systems

Nowadays, a wide set of systems and application, especially in high performance computing, depends on distributed environments to process and analyses huge amounts of data. As we know, the amount of data increases enormously, and the goal to provide and develop efficient, scalable and reliable storage solutions has become one of the major issue for scientific computing. The storage solution used by big data systems is Distributed File Systems (DFSs), where DFS is used to build a hierarchical and unified view of multiple file servers and shares on the network. In this paper we will offer Hadoop Distributed File System (HDFS) as DFS in big data systems and we will present an Event-B as formal method that can be used in modeling, where Event-B is a mature formal method which has been widely used in a number of industry projects in a number of domains, such as automotive, transportation, space, business information, medical device and so on, And will propose using the Rodin as modeling tool for Event-B, which integrates modeling and proving as well as the Rodin platform is open source, so it supports a large number of plug-in tools.

Research and Design of a Trusted Distributed File System Based on HDFS

2014 ◽  
Vol 602-605 ◽  
pp. 3282-3284
Author(s):  
Fa Gui Liu ◽  
Xiao Jie Zhang
Keyword(s):  
File System ◽  
File Systems ◽  
Distributed File System ◽  
Distributed File Systems ◽  
Security Mechanism ◽  
Advanced Persistent Threat ◽  
Research And Design

Distributed file systems such as HDFS are facing the threat of Advanced Persistent Threat, APT. Although security mechanisms such as Kerberos and ACL are implemented in distributed file systems, most of them are not sufficient to solve the threats caused by APT. With the observation into traits of APT, we propose a trusted distributed file system based on HDFS, which guarantees another further security facing APT compared to the current security mechanism.

An Adaptive Performance Prediction Method of Distributed File System Based on Performance Correlation

2014 ◽  
Vol 998-999 ◽  
pp. 1362-1365
Author(s):  
Wei Feng Gao ◽  
Tie Zhu Zhao ◽  
Ming Bin Lin
Keyword(s):  
Prediction Model ◽  
Performance Prediction ◽  
Large Scale ◽  
File System ◽  
File Systems ◽  
Prediction Method ◽  
Distributed File System ◽  
Distributed File Systems ◽  
Continuous Growth ◽  
Modeling And Analysis

Distributed file systems are emerging as a key component of large scale cloud storage platform due to the continuous growth of the amount of application data. Performance modeling and analysis is an important concern in the distributed file system area. This paper focuses on the performance prediction and modeling issues. An adaptive prediction model (APModel) is proposed to predict the performance of distributed file systems by capturing the performance correlation of different performance factors. We perform a series of experiments to validate the proposed prediction model. The experiment results indicate our proposed approach can get better prediction accuracy. It is practical and can achieve sufficient performance analysis for distributed file systems.

Using Hadoop Distributed and Deduplicated File System (HD2FS) in Astronomy

2019 ◽  
Vol 15 (S367) ◽  
pp. 464-466
Author(s):  
Paul Bartus
Keyword(s):  
Data Storage ◽  
Storage Capacity ◽  
File System ◽  
Storage Systems ◽  
Storage System ◽  
File Systems ◽  
Distributed File System ◽  
Distributed File Systems ◽  
Output Performance ◽  
Hadoop Distributed File System

AbstractDuring the last years, the amount of data has skyrocketed. As a consequence, the data has become more expensive to store than to generate. The storage needs for astronomical data are also following this trend. Storage systems in Astronomy contain redundant copies of data such as identical files or within sub-file regions. We propose the use of the Hadoop Distributed and Deduplicated File System (HD2FS) in Astronomy. HD2FS is a deduplication storage system that was created to improve data storage capacity and efficiency in distributed file systems without compromising Input/Output performance. HD2FS can be developed by modifying existing storage system environments such as the Hadoop Distributed File System. By taking advantage of deduplication technology, we can better manage the underlying redundancy of data in astronomy and reduce the space needed to store these files in the file systems, thus allowing for more capacity per volume.

Requirements for a Forensically Ready Cloud Storage Service

2011 ◽  
Vol 3 (3) ◽  
pp. 19-36 ◽  
Author(s):  
Theodoros Spyridopoulos ◽  
Vasilios Katos
Keyword(s):  
Cloud Storage ◽  
File System ◽  
Storage Systems ◽  
File Systems ◽  
Distributed File System ◽  
Distributed File Systems ◽  
Cloud Storage Service ◽  
Storage Service ◽  
Acquisition Processes

This paper examines the feasibility of developing a forensic acquisition tool in a distributed file system. Using GFS and KFS distributed file systems as vehicles and through representative scenarios and examples, the authors develop forensic acquisition processes and examine both the requirements of the tool and the distributed file system must meet in order to facilitate the acquisition. The authors conclude that cloud storage has features that can be leveraged to perform acquisition (such as redundancy and replication triggers) but also maintains a complexity, which is higher than traditional storage systems leading to a need for forensic-readiness-by-design.

Octopus + : An RDMA-Enabled Distributed Persistent Memory File System

2021 ◽  
Vol 17 (3) ◽  
pp. 1-25
Author(s):  
Bohong Zhu ◽  
Youmin Chen ◽  
Qing Wang ◽  
Youyou Lu ◽  
Jiwu Shu
Keyword(s):  
High Speed ◽  
High Performance ◽  
File System ◽  
Direct Memory Access ◽  
File Systems ◽  
Distributed File Systems ◽  
Persistent Memory ◽  
Memory Modules ◽  
Non Volatile Memory ◽  
Volatile Memory

Non-volatile memory and remote direct memory access (RDMA) provide extremely high performance in storage and network hardware. However, existing distributed file systems strictly isolate file system and network layers, and the heavy layered software designs leave high-speed hardware under-exploited. In this article, we propose an RDMA-enabled distributed persistent memory file system, Octopus + , to redesign file system internal mechanisms by closely coupling non-volatile memory and RDMA features. For data operations, Octopus + directly accesses a shared persistent memory pool to reduce memory copying overhead, and actively fetches and pushes data all in clients to rebalance the load between the server and network. For metadata operations, Octopus + introduces self-identified remote procedure calls for immediate notification between file systems and networking, and an efficient distributed transaction mechanism for consistency. Octopus + is enabled with replication feature to provide better availability. Evaluations on Intel Optane DC Persistent Memory Modules show that Octopus + achieves nearly the raw bandwidth for large I/Os and orders of magnitude better performance than existing distributed file systems.

Dynamic Load Rebalancing Algorithm for Private Cloud

2014 ◽  
Vol 573 ◽  
pp. 556-559
Author(s):  
A. Shenbaga Bharatha Priya ◽  
J. Ganesh ◽  
Mareeswari M. Devi
Keyword(s):  
Dynamic Load ◽  
Large Scale ◽  
File System ◽  
Single Point ◽  
File Systems ◽  
Distributed File System ◽  
Private Cloud ◽  
Global Knowledge ◽  
Load Imbalance ◽  
And Storage

Infrastructure-As-A-Service (IAAS) provides an environmental setup under any type of cloud. In Distributed file system (DFS), nodes are simultaneously serve computing and storage functions; that is parallel Data Processing and storage in cloud. Here, file is considered as a data or load. That file is partitioned into a number of File chunks (FC) allocated in distinct nodes so that Map Reduce tasks can be performed in parallel over the nodes. Files and Nodes can be dynamically created, deleted, and added. This results in load imbalance in a distributed file system; that is, the file chunks are not distributed as uniformly as possible among the Chunk Servers (CS). Emerging distributed file systems in production systems strongly depend on a central node for chunk reallocation or Distributed node to maintain global knowledge of all chunks. This dependence is clearly inadequate in a large-scale, failure-prone environment because the central load balancer is put under considerable workload that is linearly scaled with the system size, it may thus become the performance bottleneck and the single point of failure and memory wastage in distributed nodes. So, we have to enhance the Client side module with server side module to create, delete and update the file chunks in Client Module. And manage the overall private cloud and apply dynamic load balancing algorithm to perform auto scaling options in private cloud. In this project, a fully distributed load rebalancing algorithm is presented to cope with the load imbalance problem.

Comparative Analysis of Distributed File Systems

2021 ◽  
Vol 9 (VIII) ◽  
pp. 20-26
Author(s):  
Basireddy Ithihas Reddy
Keyword(s):  
Comparative Analysis ◽  
File Systems ◽  
Distributed File Systems ◽  
Input Output ◽  
Primary Input ◽  
Multiple Systems ◽  
Execution Engine ◽  
Distributed Execution ◽  
Testing Algorithms ◽  
Distributed Files

It has been observed that there has been a great interest in computing experiments which has been useful on shared nothing computers and commodity machines. We need multiple systems running in parallel working closely together towards the same goal. Frequently it has been experienced and observed that the distributed execution engine named MapReduce handles the primary input-output workload for such clusters. There are numerous distributed file systems around viz. NTFS,ReFS,FAT,FAT32 in windows and linux, we studied them and implemented a few distributed file systems. It has been studied that distributed file systems (DFS) work very well on many small files but some do not generate expected output on large files. We implemented benchmark testing algorithms in each distributed files systems for small and large files, and the analysis is been put forward in this paper. Even we came across the various implementation issues of various DFS, they have also been mentioned in this paper.

scholarly journals Log-Less Metadata Management on Metadata Server for Parallel File Systems

2014 ◽  
Vol 2014 ◽  
pp. 1-8
Author(s):  
Jianwei Liao ◽  
Guoqiang Xiao ◽  
Xiaoning Peng
Keyword(s):  
Performance Improvement ◽  
File System ◽  
File Systems ◽  
Metadata Management ◽  
Distributed File Systems ◽  
Data Throughput ◽  
Management Schemes ◽  
Metadata Processing ◽  
Metadata Server ◽  
Metadata Service

This paper presents a novel metadata management mechanism on the metadata server (MDS) for parallel and distributed file systems. In this technique, the client file system backs up the sent metadata requests, which have been handled by the metadata server, so that the MDS does not need to log metadata changes to nonvolatile storage for achieving highly available metadata service, as well as better performance improvement in metadata processing. As the client file system backs up certain sent metadata requests in its memory, the overhead for handling these backup requests is much smaller than that brought by the metadata server, while it adopts logging or journaling to yield highly available metadata service. The experimental results show that this newly proposed mechanism can significantly improve the speed of metadata processing and render a better I/O data throughput, in contrast to conventional metadata management schemes, that is, logging or journaling on MDS. Besides, a complete metadata recovery can be achieved by replaying the backup logs cached by all involved clients, when the metadata server has crashed or gone into nonoperational state exceptionally.

Sign in / Sign up

Export Citation Format

Share Document