The Use of Locality Information on Data Intensive Parallel File Systems

2013 ◽  
Author(s):  
Ricardo Ryoiti Sugawara Junior ◽  
Liria Matsumoto Sato
Keyword(s):  
File Systems ◽  
Parallel File Systems ◽  
Data Intensive ◽  
Parallel File ◽  
Locality Information

Towards Data Intensive Many-Task Computing

2012 ◽  
pp. 28-73 ◽  
Author(s):  
Ioan Raicu ◽  
Ian Foster ◽  
Yong Zhao ◽  
Alex Szalay ◽  
Philip Little ◽  
...  
Keyword(s):  
High Performance ◽  
File Systems ◽  
Data Locality ◽  
Resource Provisioning ◽  
Parallel File Systems ◽  
Data Intensive ◽  
Dynamic Resource Provisioning ◽  
Rate Of Increase ◽  
Parallel File ◽  
Data Intensive Applications

Many-task computing aims to bridge the gap between two computing paradigms, high throughput computing and high performance computing. Traditional techniques to support many-task computing commonly found in scientific computing (i.e. the reliance on parallel file systems with static configurations) do not scale to today’s largest systems for data intensive application, as the rate of increase in the number of processors per system is outgrowing the rate of performance increase of parallel file systems. In this chapter, the authors argue that in such circumstances, data locality is critical to the successful and efficient use of large distributed systems for data-intensive applications. They propose a “data diffusion” approach to enable data-intensive many-task computing. They define an abstract model for data diffusion, define and implement scheduling policies with heuristics that optimize real world performance, and develop a competitive online caching eviction policy. They also offer many empirical experiments to explore the benefits of data diffusion, both under static and dynamic resource provisioning, demonstrating approaches that improve both performance and scalability.

UniIndex: An index and query middleware for parallel file systems

2020 ◽  
Vol 32 (9) ◽  
Author(s):  
Peng Cheng ◽  
Yong Wang ◽  
Yutong Lu ◽  
Yunfei Du ◽  
Zhiguang Chen
Keyword(s):  
File Systems ◽  
Parallel File Systems ◽  
Parallel File
Sign in / Sign up

Export Citation Format

Share Document