A Distributed Shared Parallel IO System for HPC

2008 ◽  
Author(s):  
Yu-Feng Guo ◽  
Qiong Li ◽  
Guang-Ming Liu ◽  
Yue-Sheng Cao ◽  
Lei Zhang
Keyword(s):  
Parallel Io

Parallel IO Prefetching and Caching

2002 ◽  
pp. 96-151
Author(s):  
MAHESH KALLAHALLA ◽  
PETER J. VARMAN
Keyword(s):  
Parallel Io

scholarly journals Improving Parallel IO Performance of Cell-based AMR Cosmology Applications

2012 ◽  
Author(s):  
Yongen Yu ◽  
Douglas H. Rudd ◽  
Zhiling Lan ◽  
Nickolay Y. Gnedin ◽  
Andrey Kravtsov ◽  
...  
Keyword(s):  
Parallel Io

scholarly journals An Offline Framework for High-dimensional Ensemble Kalman Filters to Reduce the Time-to-solution

2019 ◽  
Author(s):  
Yongjun Zheng ◽  
Clément Albergel ◽  
Simon Munier ◽  
Bertrand Bonan ◽  
Jean-Christophe Calvet
Keyword(s):  
Data Assimilation ◽  
Kalman Filters ◽  
High Performance ◽  
Large Scale ◽  
Message Passing Interface ◽  
High Dimensional ◽  
Ensemble Kalman Filters ◽  
Running Time ◽  
Online Mode ◽  
Parallel Io

Abstract. The high computational resources and the time-consuming IO (Input/Output) are major issues in offline ensemble- based high-dimentional data assimilation systems. Bearing these in mind, this study proposes a sophisticated dynamically running job scheme as well as an innovative parallel IO algorithm to reduce the time-to-solution of an offline framework for high-dimensional ensemble Kalman filters. The dynamically running job scheme runs as many tasks as possible within a single job to reduce the queuing time and minimize the overhead of starting/ending a job. The parallel IO algorithm reads or writes non-overlapping segments of multiple files with an identical structure to reduce the IO times by minimizing the IO competitions and maximizing the overlapping of the MPI (Message Passing Interface) communications with the IO operations. Results based on sensitive experiments shown that the proposed parallel IO algorithm can significantly reduce the IO times and has a very good scalability, too. Based on these two advanced techniques, the offline and online modes of ensemble Kalman filters are built based on PDAF (Parallel Data Assimilation Framework) to comprehensively assess their efficiencies. It can be seen from the comparisons between the offline and online modes that the IO time only accounts for a small fraction of the total time with the proposed parallel IO algorithm. The queuing time might be less than the running time in a low-loaded supercomputer such as in an operational context but the offline mode can be nearly as fast as, if not faster than, the online mode in terms of time-to-solution. However, the queuing time is dominant and several times larger than the running time in a high-loaded supercomputer. Thus, the offline mode is substantially faster than the online mode in terms of time-to-solution, especially for large-scale assimilation problems. From this point of view, it suggests that an offline ensemble Kalman filter with an efficient implementation and a high performance parallel file system should be preferred over its online counterpart for the intermittent data assimilation in many situations.

scholarly journals An offline framework for high-dimensional ensemble Kalman filters to reduce the time to solution

2020 ◽  
Vol 13 (8) ◽  
pp. 3607-3625
Author(s):  
Yongjun Zheng ◽  
Clément Albergel ◽  
Simon Munier ◽  
Bertrand Bonan ◽  
Jean-Christophe Calvet
Keyword(s):  
Data Assimilation ◽  
Kalman Filters ◽  
High Performance ◽  
Large Scale ◽  
Message Passing Interface ◽  
High Dimensional ◽  
Ensemble Kalman Filters ◽  
Running Time ◽  
Online Mode ◽  
Parallel Io

Abstract. The high computational resources and the time-consuming IO (input/output) are major issues in offline ensemble-based high-dimensional data assimilation systems. Bearing these in mind, this study proposes a sophisticated dynamically running job scheme as well as an innovative parallel IO algorithm to reduce the time to solution of an offline framework for high-dimensional ensemble Kalman filters. The dynamically running job scheme runs as many tasks as possible within a single job to reduce the queuing time and minimize the overhead of starting and/or ending a job. The parallel IO algorithm reads or writes non-overlapping segments of multiple files with an identical structure to reduce the IO times by minimizing the IO competitions and maximizing the overlapping of the MPI (Message Passing Interface) communications with the IO operations. Results based on sensitive experiments show that the proposed parallel IO algorithm can significantly reduce the IO times and have a very good scalability, too. Based on these two advanced techniques, the offline and online modes of ensemble Kalman filters are built based on PDAF (Parallel Data Assimilation Framework) to comprehensively assess their efficiencies. It can be seen from the comparisons between the offline and online modes that the IO time only accounts for a small fraction of the total time with the proposed parallel IO algorithm. The queuing time might be less than the running time in a low-loaded supercomputer such as in an operational context, but the offline mode can be nearly as fast as, if not faster than, the online mode in terms of time to solution. However, the queuing time is dominant and several times larger than the running time in a high-loaded supercomputer. Thus, the offline mode is substantially faster than the online mode in terms of time to solution, especially for large-scale assimilation problems. From this point of view, results suggest that an offline ensemble Kalman filter with an efficient implementation and a high-performance parallel file system should be preferred over its online counterpart for intermittent data assimilation in many situations.

Abstract: Auto-Tuning of Parallel IO Parameters for HDF5 Applications

2012 ◽  
Author(s):  
Babak Behzad ◽  
Joey Huchette ◽  
Huong Luu ◽  
Ruth Aydt ◽  
Quincey Koziol ◽  
...  
Keyword(s):  
Auto Tuning ◽  
Parallel Io
Sign in / Sign up

Export Citation Format

Share Document