scholarly journals Log Analysis-Based Resource and Execution Time Improvement in HPC: A Case Study

2020 ◽  
Vol 10 (7) ◽  
pp. 2634
Author(s):  
JunWeon Yoon ◽  
TaeYoung Hong ◽  
ChanYeol Park ◽  
Seo-Young Noh ◽  
HeonChang Yu
Keyword(s):  
Execution Time ◽  
High Performance ◽  
Large Scale ◽  
Experimental Result ◽  
Optimization Approach ◽  
Root Cause ◽  
Large Systems ◽  
Job Scheduler ◽  
Performance Computing

High-performance computing (HPC) uses many distributed computing resources to solve large computational science problems through parallel computation. Such an approach can reduce overall job execution time and increase the capacity of solving large-scale and complex problems. In the supercomputer, the job scheduler, the HPC’s flagship tool, is responsible for distributing and managing the resources of large systems. In this paper, we analyze the execution log of the job scheduler for a certain period of time and propose an optimization approach to reduce the idle time of jobs. In our experiment, it has been found that the main root cause of delayed job is highly related to resource waiting. The execution time of the entire job is affected and significantly delayed due to the increase in idle resources that must be ready when submitting the large-scale job. The backfilling algorithm can optimize the inefficiency of these idle resources and help to reduce the execution time of the job. Therefore, we propose the backfilling algorithm, which can be applied to the supercomputer. This experimental result shows that the overall execution time is reduced.

scholarly journals Scalability of DL_POLY on High Performance Computing Platform

2017 ◽  
Vol 29 (3) ◽  
Author(s):  
Mabule Samuel Mabakane ◽  
Daniel Mojalefa Moeketsi ◽  
Anton Lopis
Keyword(s):  
Molecular Dynamics ◽  
High Performance Computing ◽  
High Performance ◽  
Computing Platform ◽  
Speed Up ◽  
Linux Cluster ◽  
Large Systems ◽  
Performance Computing ◽  
Better Than

This paper presents a case study on the scalability of several versions of the molecular dynamics code (DL_POLY) performed on South Africa‘s Centre for High Performance Computing e1350 IBM Linux cluster, Sun system and Lengau supercomputers. Within this study different problem sizes were designed and the same chosen systems were employed in order to test the performance of DL_POLY using weak and strong scalability. It was found that the speed-up results for the small systems were better than large systems on both Ethernet and Infiniband network. However, simulations of large systems in DL_POLY performed well using Infiniband network on Lengau cluster as compared to e1350 and Sun supercomputer.

Parallel PPI Prediction Performance Study on HPC Platforms

2015 ◽  
Vol 24 (05) ◽  
pp. 1550074 ◽  
Author(s):  
Ali A. El-Moursy ◽  
Wael S. Afifi ◽  
Fadi N. Sibai ◽  
Salwa M. Nassar
Keyword(s):  
Protein Interactions ◽  
Execution Time ◽  
High Performance ◽  
Large Scale ◽  
Parallel Implementation ◽  
Prediction Method ◽  
Protein Protein Interactions ◽  
Performance Study ◽  
Ppi Prediction ◽  
Performance Computing

STRIKE is an algorithm which predicts protein–protein interactions (PPIs) and determines that proteins interact if they contain similar substrings of amino acids. Unlike other methods for PPI prediction, STRIKE is able to achieve reasonable improvement over the existing PPI prediction methods. Although its high accuracy as a PPI prediction method, STRIKE consumes a large execution time and hence it is considered to be a compute-intensive application. In this paper, we develop and implement a parallel STRIKE algorithm for high-performance computing (HPC) systems. Using a large-scale cluster, the execution time of the parallel implementation of this bioinformatics algorithm was reduced from about a week on a serial uniprocessor machine to about 16.5 h on 16 computing nodes, down to about 2 h on 128 parallel nodes. Communication overheads between nodes are thoroughly studied.

scholarly journals Deriving Large-Scale Coastal Bathymetry from Sentinel-2 Images Using an HIGH-Performance Cluster: A Case Study Covering North Africa’s Coastal Zone

Sensors ◽  
2021 ◽  
Vol 21 (21) ◽  
pp. 7006
Author(s):  
Mohamed Wassim Baba ◽  
Gregoire Thoumyre ◽  
Erwin W. J. Bergsma ◽  
Christopher J. Daly ◽  
Rafael Almar
Keyword(s):  
High Performance ◽  
Large Scale ◽  
Computational Cost ◽  
Remote Sensing Data ◽  
Coastal Areas ◽  
New Perspective ◽  
Sentinel 2A ◽  
Performance Computing ◽  
High Computational Cost

Coasts are areas of vitality because they host numerous activities worldwide. Despite their major importance, the knowledge of the main characteristics of the majority of coastal areas (e.g., coastal bathymetry) is still very limited. This is mainly due to the scarcity and lack of accurate measurements or observations, and the sparsity of coastal waters. Moreover, the high cost of performing observations with conventional methods does not allow expansion of the monitoring chain in different coastal areas. In this study, we suggest that the advent of remote sensing data (e.g., Sentinel 2A/B) and high performance computing could open a new perspective to overcome the lack of coastal observations. Indeed, previous research has shown that it is possible to derive large-scale coastal bathymetry from S-2 images. The large S-2 coverage, however, leads to a high computational cost when post-processing the images. Thus, we develop a methodology implemented on a High-Performance cluster (HPC) to derive the bathymetry from S-2 over the globe. In this paper, we describe the conceptualization and implementation of this methodology. Moreover, we will give a general overview of the generated bathymetry map for NA compared with the reference GEBCO global bathymetric product. Finally, we will highlight some hotspots by looking closely to their outputs.

Hardware Accelerator Integration Tradeoffs for High-Performance Computing: A Case Study of GEMM Acceleration in N-Body Methods

2021 ◽  
Vol 32 (8) ◽  
pp. 2035-2048
Author(s):  
Mochamad Asri ◽  
Dhairya Malhotra ◽  
Jiajun Wang ◽  
George Biros ◽  
Lizy K. John ◽  
...  
Keyword(s):  
High Performance Computing ◽  
High Performance ◽  
Hardware Accelerator ◽  
Performance Computing

scholarly journals Fast Playback Framework for Analysis of Ground-Based Doppler Radar Observations Using MapReduce Technology

2016 ◽  
Vol 33 (4) ◽  
pp. 621-634 ◽  
Author(s):  
Jingyin Tang ◽  
Corene J. Matyas
Keyword(s):  
Spatial Analysis ◽  
High Performance ◽  
Large Scale ◽  
Doppler Radar ◽  
Weather Events ◽  
Data Architecture ◽  
Research Grade ◽  
High Performance Computing Cluster ◽  
Time Systems ◽  
Performance Computing

AbstractThe creation of a 3D mosaic is often the first step when using the high-spatial- and temporal-resolution data produced by ground-based radars. Efficient yet accurate methods are needed to mosaic data from dozens of radar to better understand the precipitation processes in synoptic-scale systems such as tropical cyclones. Research-grade radar mosaic methods of analyzing historical weather events should utilize data from both sides of a moving temporal window and process them in a flexible data architecture that is not available in most stand-alone software tools or real-time systems. Thus, these historical analyses require a different strategy for optimizing flexibility and scalability by removing time constraints from the design. This paper presents a MapReduce-based playback framework using Apache Spark’s computational engine to interpolate large volumes of radar reflectivity and velocity data onto 3D grids. Designed as being friendly to use on a high-performance computing cluster, these methods may also be executed on a low-end configured machine. A protocol is designed to enable interoperability with GIS and spatial analysis functions in this framework. Open-source software is utilized to enhance radar usability in the nonspecialist community. Case studies during a tropical cyclone landfall shows this framework’s capability of efficiently creating a large-scale high-resolution 3D radar mosaic with the integration of GIS functions for spatial analysis.

scholarly journals High-Performance Computing Framework Based on Distributed Systems for Large-Scale Neurophysiological Data

2021 ◽  
Author(s):  
Mohsen Hadianpour ◽  
Ehsan Rezayat ◽  
Mohammad-Reza Dehaqani
Keyword(s):  
High Performance Computing ◽  
High Performance ◽  
Large Scale ◽  
Electrophysiological Recording ◽  
Neural Data ◽  
Data Framework ◽  
Neurophysiological Data ◽  
Computing Framework ◽  
Performance Computing ◽  
Neuroscience Community

Abstract Due to the significantly drastic progress and improvement in neurophysiological recording technologies, neuroscientists have faced various complexities dealing with unstructured large-scale neural data. In the neuroscience community, these complexities could create serious bottlenecks in storing, sharing, and processing neural datasets. In this article, we developed a distributed high-performance computing (HPC) framework called `Big neuronal data framework' (BNDF), to overcome these complexities. BNDF is based on open-source big data frameworks, Hadoop and Spark providing a flexible and scalable structure. We examined BNDF on three different large-scale electrophysiological recording datasets from nonhuman primate’s brains. Our results exhibited faster runtimes with scalability due to the distributed nature of BNDF. We compared BNDF results to a widely used platform like MATLAB in an equitable computational resource. Compared with other similar methods, using BNDF provides more than five times faster performance in spike sorting as a usual neuroscience application.

scholarly journals Taxonomic assignment for large-scale metagenomic data on high-perfomance systems

2017 ◽  
Vol 33 (2) ◽  
pp. 119-130
Author(s):  
Vinh Van Le ◽  
Hoai Van Tran ◽  
Hieu Ngoc Duong ◽  
Giang Xuan Bui ◽  
Lang Van Tran
Keyword(s):  
High Performance Computing ◽  
Assignment Problem ◽  
High Performance ◽  
Large Scale ◽  
Computing System ◽  
Metagenomic Data ◽  
Taxonomic Assignment ◽  
High Performance Computing System ◽  
Powerful Approach ◽  
Performance Computing

Metagenomics is a powerful approach to study environment samples which do not require the isolation and cultivation of individual organisms. One of the essential tasks in a metagenomic project is to identify the origin of reads, referred to as taxonomic assignment. Due to the fact that each metagenomic project has to analyze large-scale datasets, the metatenomic assignment is very much computation intensive. This study proposes a parallel algorithm for the taxonomic assignment problem, called SeMetaPL, which aims to deal with the computational challenge. The proposed algorithm is evaluated with both simulated and real datasets on a high performance computing system. Experimental results demonstrate that the algorithm is able to achieve good performance and utilize resources of the system efficiently. The software implementing the algorithm and all test datasets can be downloaded at http://it.hcmute.edu.vn/bioinfo/metapro/SeMetaPL.html.

Sign in / Sign up

Export Citation Format

Share Document