Bridging VisTrails Scientific Workflow Management System to High Performance Computing

2013 ◽  
Author(s):  
Jia Zhang ◽  
Petr Votava ◽  
Tsengdar J. Lee ◽  
Owen Chu ◽  
Clyde Li ◽  
...  
Keyword(s):  
High Performance Computing ◽  
Management System ◽  
High Performance ◽  
Workflow Management ◽  
Scientific Workflow ◽  
Workflow Management System ◽  
Performance Computing

scholarly journals Scientific workflows applied to the coupling of a continuum (Elmer v8.3) and a discrete element (HiDEM v1.0) ice dynamic model

2019 ◽  
Vol 12 (7) ◽  
pp. 3001-3015 ◽  
Author(s):  
Shahbaz Memon ◽  
Dorothée Vallot ◽  
Thomas Zwinger ◽  
Jan Åström ◽  
Helmut Neukirchen ◽  
...  
Keyword(s):  
Management System ◽  
High Performance ◽  
Heterogeneous Computing ◽  
Workflow Management ◽  
Scientific Workflow ◽  
Workflow Management System ◽  
Data Intensive ◽  
Cpu Utilization ◽  
Computing Environments ◽  
High Level

Abstract. Scientific computing applications involving complex simulations and data-intensive processing are often composed of multiple tasks forming a workflow of computing jobs. Scientific communities running such applications on computing resources often find it cumbersome to manage and monitor the execution of these tasks and their associated data. These workflow implementations usually add overhead by introducing unnecessary input/output (I/O) for coupling the models and can lead to sub-optimal CPU utilization. Furthermore, running these workflow implementations in different environments requires significant adaptation efforts, which can hinder the reproducibility of the underlying science. High-level scientific workflow management systems (WMS) can be used to automate and simplify complex task structures by providing tooling for the composition and execution of workflows – even across distributed and heterogeneous computing environments. The WMS approach allows users to focus on the underlying high-level workflow and avoid low-level pitfalls that would lead to non-optimal resource usage while still allowing the workflow to remain portable between different computing environments. As a case study, we apply the UNICORE workflow management system to enable the coupling of a glacier flow model and calving model which contain many tasks and dependencies, ranging from pre-processing and data management to repetitive executions in heterogeneous high-performance computing (HPC) resource environments. Using the UNICORE workflow management system, the composition, management, and execution of the glacier modelling workflow becomes easier with respect to usage, monitoring, maintenance, reusability, portability, and reproducibility in different environments and by different user groups. Last but not least, the workflow helps to speed the runs up by reducing model coupling I/O overhead and it optimizes CPU utilization by avoiding idle CPU cores and running the models in a distributed way on the HPC cluster that best fits the characteristics of each model.

scholarly journals Integrating ScientificWorkflows with Scientific Gateways: A Bioinformatics Experiment in the Brazilian National High-Performance Computing Network

2018 ◽  
Author(s):  
Maria Luiza Mondelli ◽  
Marcelo Monteiro Galheigo ◽  
V´ıvian Medeiros ◽  
Bruno F. Bastos ◽  
Antônio Tadeu Azevedo Gomes ◽  
...  
Keyword(s):  
High Performance Computing ◽  
High Performance ◽  
Workflow Management ◽  
Scientific Workflow ◽  
Management Systems ◽  
Workflow Management Systems ◽  
Sequencing Technologies ◽  
Geographically Distributed ◽  
Performance Computing ◽  
High Performance Computation

Bioinformatics experiments are rapidly and constantly evolving due improvements in sequencing technologies. These experiments usually demand high performance computation and produce huge quantities of data. They also require different programs to be executed in a certain order, allowing the experiments to be modeled as workflows. However, users do not always have the infrastructure needed to perform these experiments. Our contribution is the integration of scientific workflow management systems and grid-enabled scientific gateways, providing the user with a transparent way to run these workflows in geographically distributed computing resources. The availability of the workflow through the gateway allows for a better usability of these experiments.  

scholarly journals ScientificWorkflows Applied to the Coupling of a Continuum (Elmer v8.3) and a Discrete Element (HiDEM v1.0) Ice Dynamic Model

2018 ◽  
Author(s):  
Shahbaz Memon ◽  
Dorothée Vallot ◽  
Thomas Zwinger ◽  
Jan Åström ◽  
Helmut Neukirchen ◽  
...  
Keyword(s):  
Management System ◽  
High Performance ◽  
Heterogeneous Computing ◽  
Workflow Management ◽  
Scientific Workflow ◽  
Workflow Management System ◽  
Data Intensive ◽  
User Groups ◽  
Task Structures

Abstract. Scientific computing applications involving complex simulations and data-intensive processing are often composed of multiple tasks forming a workflow of computing jobs. Scientific communities running such applications on distributed and heterogeneous computing resources find it cumbersome to manage and monitor the execution of these tasks. Scientific workflow management systems (WMS) can be used to automate and simplify complex task structures by providing tooling for the composition and execution of workflows across distributed and heterogeneous computing environments. As a case study, we apply the UNICORE workflow management system to a formerly hard-coded coupling of a glacier sliding and calving simulation that contains many tasks and dependencies, ranging from pre-processing and data management to repetitive executions in heterogeneous high-performance computing (HPC) resource environments. Using the UNICORE workflow management system, the composition, management, and execution of the glacier modelling workflow becomes easier with respect to usage, monitoring, maintenance, re-usability, portability, and reproducibility in different environments and by different user groups.

Integrating Scientific Workflows with Scientific Gateways: A Bioinformatics Experiment in the Brazilian National High-Performance Computing Network

2020 ◽  
Author(s):  
Maria Luiza Mondelli ◽  
Marcelo Monteiro Galheigo ◽  
Vivivan Medeiros ◽  
Bruno F. Bastos ◽  
Antônio Tadeu Azevedo Gomes ◽  
...  
Keyword(s):  
High Performance ◽  
Workflow Management ◽  
Scientific Workflow ◽  
Scientific Workflows ◽  
Management Systems ◽  
Workflow Management Systems ◽  
Sequencing Technologies ◽  
Geographically Distributed ◽  
Performance Computing ◽  
High Performance Computation

Bioinformatics experiments are rapidly and constantly evolving due improvements in sequencing technologies. These experiments usually demand high performance computation and produce huge quantities of data. They also require different programs to be executed in a certain order, allowing the experiments to be modeled as workflows. However, users do not always have the infrastructure needed to perform these experiments. Our contribution is the integration of scientific workflow management systems and grid-enabled scientific gateways, providing the user with a transparent way to run these workflows in geographically distributed computing resources. The availability of the workflow through the gateway allows for a better usability of these experiments.

HPC cloud services based on the Proxmox VE platform

2019 ◽  
Author(s):  
А.В. Баранов ◽  
Е.А. Киселёв
Keyword(s):  
High Performance Computing ◽  
Management System ◽  
High Performance ◽  
Virtual Machines ◽  
Cloud Services ◽  
Job Management ◽  
Job Management System ◽  
High Level ◽  
Performance Computing ◽  
Hpc Cloud

Организация облачных сервисов для высокопроизводительных вычислений затруднена, во-первых, по причине высоких накладных расходов на виртуализацию, во-вторых, из-за специфики систем управления заданиями и ресурсами в научных суперкомпьютерных центрах. В настоящей работе рассмотрен подход к построению облачных сервисов видов PaaS и SaaS, основанных на совместном функционировании облачной платформы Proxmox VE и системы управления прохождением параллельных заданий, применяемой в качестве менеджера ресурсов в Межведомственном суперкомпьютерном центре РАН. Purpose. The purpose of this paper is to develop methods and technologies for building high-performance computing cloud services in scientific supercomputer centers. Methodology.To build a cloud environment for high-performance scientific calculations (HPC), the corresponding three-level model and the method of combining flows of supercomputer tasks of various types were applied. Results.A high-level HPC cloud services technology based on the free Proxmox VE software platform has been developed. The Proxmox VE platform has been integrated with the domestic supercomputer job management system called SUPPZ. Experimental estimates of the overheads introduced in the high-performance computing process by the Proxmox components are obtained. Findings.An approach to the integration a supercomputer job management system and a virtualization platform is proposed. The presented approach is based on the representation of the supercomputer jobs as virtual machines or containers. Using the Proxmox VE platform as an example, the influence of a virtual environment on the execution time of parallel programs is investigated experimentally. The possibility of applying the proposed approach to building cloud services of the PaaS and SaaS type in scientific supercomputing centers of collective use is substantiated for a class of applications for which the overhead costs introduced by the Proxmox components are acceptable.

Sign in / Sign up

Export Citation Format

Share Document