Improving Efficiency and Performance of Distributed File-Systems

2008 ◽  
Author(s):  
Micah Galizia ◽  
Hanan Lutfiyya
Keyword(s):  
File Systems ◽  
Distributed File Systems ◽  
And Performance

scholarly journals Performance Evaluations of Distributed File Systems for Scientific Big Data in FUSE Environment

Electronics ◽  
2021 ◽  
Vol 10 (12) ◽  
pp. 1471
Author(s):  
Jun-Yeong Lee ◽  
Moon-Hyun Kim ◽  
Syed Asif Raza Raza Shah ◽  
Sang-Un Ahn ◽  
Heejun Yoon ◽  
...  
Keyword(s):  
Data Storage ◽  
Scale Up ◽  
File Systems ◽  
Performance Evaluations ◽  
Distributed File Systems ◽  
Data Intensive Computing ◽  
Data Intensive ◽  
Tremendous Amount ◽  
Computing Environments ◽  
And Performance

Data are important and ever growing in data-intensive scientific environments. Such research data growth requires data storage systems that play pivotal roles in data management and analysis for scientific discoveries. Redundant Array of Independent Disks (RAID), a well-known storage technology combining multiple disks into a single large logical volume, has been widely used for the purpose of data redundancy and performance improvement. However, this requires RAID-capable hardware or software to build up a RAID-enabled disk array. In addition, it is difficult to scale up the RAID-based storage. In order to mitigate such a problem, many distributed file systems have been developed and are being actively used in various environments, especially in data-intensive computing facilities, where a tremendous amount of data have to be handled. In this study, we investigated and benchmarked various distributed file systems, such as Ceph, GlusterFS, Lustre and EOS for data-intensive environments. In our experiment, we configured the distributed file systems under a Reliable Array of Independent Nodes (RAIN) structure and a Filesystem in Userspace (FUSE) environment. Our results identify the characteristics of each file system that affect the read and write performance depending on the features of data, which have to be considered in data-intensive computing environments.

scholarly journals Machine Learning-based Anomaly Detection of Ganglia Monitoring Data in HEP Data Center

2020 ◽  
Vol 245 ◽  
pp. 07061
Author(s):  
Juan Chen ◽  
Lu Wang ◽  
Qingbao Hu
Keyword(s):  
Machine Learning ◽  
Anomaly Detection ◽  
Data Center ◽  
File Systems ◽  
Monitoring Data ◽  
Detection Methods ◽  
Distributed File Systems ◽  
Operation And Maintenance ◽  
Deviation Measurement ◽  
And Performance

This paper introduces a generic and scalable anomaly detection framework. Anomaly detection can improve operation and maintenance efficiency and assure experiments can be carried out effectively. The framework facilitates common tasks such as data sample building, retagging and visualization, deviation measurement and performance measurement for machine learning-based anomaly detection methods. The samples we used are sourced from Ganglia monitoring data. There are several anomaly detection methods to handle spatial and temporal anomalies within the framework. Finally, we show the rudimental application of the framework on Lustre distributed file systems in daily operation and maintenance.

PABIRS: A data access middleware for distributed file systems

2015 ◽  
Author(s):  
Sai Wu ◽  
Gang Chen ◽  
Xianke Zhou ◽  
Zhenjie Zhang ◽  
Anthony K. H. Tung ◽  
...  
Keyword(s):  
File Systems ◽  
Data Access ◽  
Distributed File Systems

ALDM: Adaptive Loading Data Migration in Distributed File Systems

2013 ◽  
Vol 49 (6) ◽  
pp. 2645-2652 ◽  
Author(s):  
Zhipeng Tan ◽  
Wei Zhou ◽  
Dan Feng ◽  
Wenhua Zhang
Keyword(s):  
File Systems ◽  
Data Migration ◽  
Distributed File Systems

Octopus + : An RDMA-Enabled Distributed Persistent Memory File System

2021 ◽  
Vol 17 (3) ◽  
pp. 1-25
Author(s):  
Bohong Zhu ◽  
Youmin Chen ◽  
Qing Wang ◽  
Youyou Lu ◽  
Jiwu Shu
Keyword(s):  
High Speed ◽  
High Performance ◽  
File System ◽  
Direct Memory Access ◽  
File Systems ◽  
Distributed File Systems ◽  
Persistent Memory ◽  
Memory Modules ◽  
Non Volatile Memory ◽  
Volatile Memory

Non-volatile memory and remote direct memory access (RDMA) provide extremely high performance in storage and network hardware. However, existing distributed file systems strictly isolate file system and network layers, and the heavy layered software designs leave high-speed hardware under-exploited. In this article, we propose an RDMA-enabled distributed persistent memory file system, Octopus + , to redesign file system internal mechanisms by closely coupling non-volatile memory and RDMA features. For data operations, Octopus + directly accesses a shared persistent memory pool to reduce memory copying overhead, and actively fetches and pushes data all in clients to rebalance the load between the server and network. For metadata operations, Octopus + introduces self-identified remote procedure calls for immediate notification between file systems and networking, and an efficient distributed transaction mechanism for consistency. Octopus + is enabled with replication feature to provide better availability. Evaluations on Intel Optane DC Persistent Memory Modules show that Octopus + achieves nearly the raw bandwidth for large I/Os and orders of magnitude better performance than existing distributed file systems.

Sign in / Sign up

Export Citation Format

Share Document