Metadata Distribution and Consistency Techniques for Large-Scale Cluster File Systems

2011 ◽  
Vol 22 (5) ◽  
pp. 803-816 ◽  
Author(s):  
Jin Xiong ◽  
Yiming Hu ◽  
Guojie Li ◽  
Rongfeng Tang ◽  
Zhihua Fan
Keyword(s):  
Large Scale ◽  
File Systems ◽  
Cluster File

scholarly journals G-SD: Achieving Fast Reverse Lookup using Scalable Declustering Layout in Large-Scale File Systems

2018 ◽  
Vol 6 (4) ◽  
pp. 1017-1030
Author(s):  
Jun Wang ◽  
Dezhi Han ◽  
Junyao Zhang ◽  
Jiangling Yin
Keyword(s):  
Large Scale ◽  
File Systems

Optimizing of metadata management in large-scale file systems

2018 ◽  
Vol 21 (4) ◽  
pp. 1865-1879 ◽  
Author(s):  
Nae Young Song ◽  
Hwajung Kim ◽  
Hyuck Han ◽  
Heon Young Yeom
Keyword(s):  
Large Scale ◽  
File Systems ◽  
Metadata Management

Grid Data Handling

2013 ◽  
pp. 294-321
Author(s):  
Alexandru Costan
Keyword(s):  
Fault Tolerance ◽  
Data Storage ◽  
Large Scale ◽  
File Systems ◽  
Future Research ◽  
Distributed Data ◽  
Data Handling ◽  
Grid Data ◽  
Distributed Data Storage ◽  
Grid Environments

To accommodate the needs of large-scale distributed systems, scalable data storage and management strategies are required, allowing applications to efficiently cope with continuously growing, highly distributed data. This chapter addresses the key issues of data handling in grid environments focusing on storing, accessing, managing and processing data. We start by providing the background for the data storage issue in grid environments. We outline the main challenges addressed by distributed storage systems: high availability which translates into high resilience and consistency, corruption handling regarding arbitrary faults, fault tolerance, asynchrony, fairness, access control and transparency. The core part of the chapter presents how existing solutions cope with these high requirements. The most important research results are organized along several themes: grid data storage, distributed file systems, data transfer and retrieval and data management. Important characteristics such as performance, efficient use of resources, fault tolerance, security, and others are strongly determined by the adopted system architectures and the technologies behind them. For each topic, we shortly present previous work, describe the most recent achievements, highlight their advantages and limitations, and indicate future research trends in distributed data storage and management.

Cluster File Systems

2011 ◽  
pp. 289-289
Author(s):  
Guy L. Steele ◽  
Xiaowei Shen ◽  
Josep Torrellas ◽  
Mark Tuckerman ◽  
Eric J. Bohm ◽  
...  
Keyword(s):  
File Systems ◽  
Cluster File

An Efficient Metadata Distribution Policy for Cluster File Systems

2005 ◽  
Author(s):  
Jin Xiong ◽  
Rongfeng Tang ◽  
Sining Wu ◽  
Dan Meng ◽  
Ninghui Sun
Keyword(s):  
File Systems ◽  
Cluster File

Energy Efficient Prefetching with Buffer Disks for Cluster File Systems

2010 ◽  
Author(s):  
Adam Manzanares ◽  
Xiaojun Ruan ◽  
Shu Yin ◽  
Jiong Xie ◽  
Zhiyang Ding ◽  
...  
Keyword(s):  
Energy Efficient ◽  
File Systems ◽  
Cluster File

Dynamic Load Rebalancing Algorithm for Private Cloud

2014 ◽  
Vol 573 ◽  
pp. 556-559
Author(s):  
A. Shenbaga Bharatha Priya ◽  
J. Ganesh ◽  
Mareeswari M. Devi
Keyword(s):  
Dynamic Load ◽  
Large Scale ◽  
File System ◽  
Single Point ◽  
File Systems ◽  
Distributed File System ◽  
Private Cloud ◽  
Global Knowledge ◽  
Load Imbalance ◽  
And Storage

Infrastructure-As-A-Service (IAAS) provides an environmental setup under any type of cloud. In Distributed file system (DFS), nodes are simultaneously serve computing and storage functions; that is parallel Data Processing and storage in cloud. Here, file is considered as a data or load. That file is partitioned into a number of File chunks (FC) allocated in distinct nodes so that Map Reduce tasks can be performed in parallel over the nodes. Files and Nodes can be dynamically created, deleted, and added. This results in load imbalance in a distributed file system; that is, the file chunks are not distributed as uniformly as possible among the Chunk Servers (CS). Emerging distributed file systems in production systems strongly depend on a central node for chunk reallocation or Distributed node to maintain global knowledge of all chunks. This dependence is clearly inadequate in a large-scale, failure-prone environment because the central load balancer is put under considerable workload that is linearly scaled with the system size, it may thus become the performance bottleneck and the single point of failure and memory wastage in distributed nodes. So, we have to enhance the Client side module with server side module to create, delete and update the file chunks in Client Module. And manage the overall private cloud and apply dynamic load balancing algorithm to perform auto scaling options in private cloud. In this project, a fully distributed load rebalancing algorithm is presented to cope with the load imbalance problem.

Sign in / Sign up

Export Citation Format

Share Document