scholarly journals iez: Resource Contention Aware Load Balancing for Large-Scale Parallel File Systems

2019 ◽  
Author(s):  
Bharti Wadhwa ◽  
Arnab K. Paul ◽  
Sarah Neuwirth ◽  
Feiyi Wang ◽  
Sarp Oral ◽  
...  
Keyword(s):  
Load Balancing ◽  
Large Scale ◽  
File Systems ◽  
Parallel File Systems ◽  
Resource Contention ◽  
Parallel File

DMFSsim: A Distributed Metadata File System Simulator

2012 ◽  
Vol 241-244 ◽  
pp. 1556-1561
Author(s):  
Qi Meng Wu ◽  
Ke Xie ◽  
Ming Fa Zhu ◽  
Li Min Xiao ◽  
Li Ruan
Keyword(s):  
Large Scale ◽  
File System ◽  
File Systems ◽  
Performance Gain ◽  
Parallel File Systems ◽  
Management Mechanism ◽  
Metadata File ◽  
Parallel File ◽  
Distribution Algorithms

Parallel file systems deploy multiple metadata servers to distribute heavy metadata workload from clients. With the increasing number of metadata servers, metadata-intensive operations are facing some problems related with collaboration among them, compromising the performance gain. Consequently, a file system simulator is very helpful to try out some optimization ideas to solve these problems. In this paper, we propose DMFSsim to simulate the metadata-intensive operations on large-scale distributed metadata file systems. DMFSsim can flexibly replay traces of multiple metadata operations, support several commonly used metadata distribution algorithms, simulate file system tree hierarchy and underlying disk blocks management mechanism in real systems. Extensive simulations show that DMFSsim is capable of demonstrating the performance of metadata-intensive operations in distributed metadata file system.

Rethinking key–value store for parallel I/O optimization

2016 ◽  
Vol 31 (4) ◽  
pp. 335-356 ◽  
Author(s):  
Anthony Kougkas ◽  
Hassan Eslami ◽  
Xian-He Sun ◽  
Rajeev Thakur ◽  
William Gropp
Keyword(s):  
Cloud Storage ◽  
Large Scale ◽  
Storage Systems ◽  
Storage System ◽  
File Systems ◽  
Data Synchronization ◽  
Input Output ◽  
Parallel File Systems ◽  
Parallel File ◽  
And Performance

Key–value stores are being widely used as the storage system for large-scale internet services and cloud storage systems. However, they are rarely used in HPC systems, where parallel file systems are the dominant storage solution. In this study, we examine the architecture differences and performance characteristics of parallel file systems and key–value stores. We propose using key–value stores to optimize overall Input/Output (I/O) performance, especially for workloads that parallel file systems cannot handle well, such as the cases with intense data synchronization or heavy metadata operations. We conducted experiments with several synthetic benchmarks, an I/O benchmark, and a real application. We modeled the performance of these two systems using collected data from our experiments, and we provide a predictive method to identify which system offers better I/O performance given a specific workload. The results show that we can optimize the I/O performance in HPC systems by utilizing key–value stores.

Alleviating I/O Interference Through Workload-Aware Striping and Load-Balancing on Parallel File Systems

2017 ◽  
pp. 315-333 ◽  
Author(s):  
Yuichi Tsujita ◽  
Tatsuhiko Yoshizaki ◽  
Keiji Yamamoto ◽  
Fumichika Sueyasu ◽  
Ryoji Miyazaki ◽  
...  
Keyword(s):  
Load Balancing ◽  
File Systems ◽  
Parallel File Systems ◽  
Parallel File

I/O Interference Alleviation on Parallel File Systems Using Server-Side QoS-Based Load-Balancing

2018 ◽  
pp. 36-48
Author(s):  
Yuichi Tsujita ◽  
Yoshitaka Furutani ◽  
Hajime Hida ◽  
Keiji Yamamoto ◽  
Atsuya Uno ◽  
...  
Keyword(s):  
Load Balancing ◽  
File Systems ◽  
Parallel File Systems ◽  
Server Side ◽  
Parallel File
Sign in / Sign up

Export Citation Format

Share Document