Resource Provisioning Options for Large-Scale Scientific Workflows

2008 ◽  
Author(s):  
Gideon Juve ◽  
Ewa Deelman
Keyword(s):  
Large Scale ◽  
Resource Provisioning ◽  
Scientific Workflows

scholarly journals Beyond HEP: Photon and accelerator science computing infrastructure at DESY

2020 ◽  
Vol 245 ◽  
pp. 07036
Author(s):  
Christoph Beyer ◽  
Stefan Bujack ◽  
Stefan Dietrich ◽  
Thomas Finnern ◽  
Martin Flemming ◽  
...  
Keyword(s):  
High Performance ◽  
Large Scale ◽  
High Energy Physics ◽  
High Energy ◽  
Resource Provisioning ◽  
Small Scale ◽  
Online Processing ◽  
Offline Processing ◽  
National Analysis ◽  
Energy Physics

DESY is one of the largest accelerator laboratories in Europe. It develops and operates state of the art accelerators for fundamental science in the areas of high energy physics, photon science and accelerator development. While for decades high energy physics (HEP) has been the most prominent user of the DESY compute, storage and network infrastructure, various scientific areas as science with photons and accelerator development have caught up and are now dominating the demands on the DESY infrastructure resources, with significant consequences for the IT resource provisioning. In this contribution, we will present an overview of the computational, storage and network resources covering the various physics communities on site. Ranging from high-throughput computing (HTC) batch-like offline processing in the Grid and the interactive user analyses resources in the National Analysis Factory (NAF) for the HEP community, to the computing needs of accelerator development or of photon sciences such as PETRA III or the European XFEL. Since DESY is involved in these experiments and their data taking, their requirements include fast low-latency online processing for data taking and calibration as well as offline processing, thus high-performance computing (HPC) workloads, that are run on the dedicated Maxwell HPC cluster. As all communities face significant challenges due to changing environments and increasing data rates in the following years, we will discuss how this will reflect in necessary changes to the computing and storage infrastructures. We will present DESY compute cloud and container orchestration plans as a basis for infrastructure and platform services. We will show examples of Jupyter notebooks for small scale interactive analysis, as well as its integration into large scale resources such as batch systems or Spark clusters. To overcome the fragmentation of the various resources for all scientific communities at DESY, we explore how to integrate them into a seamless user experience in an Interdisciplinary Data Analysis Facility.

Data Management in Scientific Workflows

2012 ◽  
pp. 177-187
Author(s):  
Ewa Deelman ◽  
Ann Chervenak
Keyword(s):  
Data Management ◽  
Gravitational Wave ◽  
Large Scale ◽  
Real Life ◽  
Scientific Workflows ◽  
Workflow Systems ◽  
Automated Generation ◽  
Workflow Execution ◽  
Data Products ◽  
Workflow Planning

Scientific applications such as those in astronomy, earthquake science, gravitational-wave physics, and others have embraced workflow technologies to do large-scale science. Workflows enable researchers to collaboratively design, manage, and obtain results that involve hundreds of thousands of steps, access terabytes of data, and generate similar amounts of intermediate and final data products. Although workflow systems are able to facilitate the automated generation of data products, many issues still remain to be addressed. These issues exist in different forms in the workflow lifecycle. This chapter describes a workflow lifecycle as consisting of a workflow generation phase where the analysis is defined, the workflow planning phase where resources needed for execution are selected, the workflow execution part, where the actual computations take place, and the result, metadata, and provenance storing phase. The authors discuss the issues related to data management at each step of the workflow cycle. They describe challenge problems and illustrate them in the context of real-life applications. They discuss the challenges, possible solutions, and open issues faced when mapping and executing large-scale workflows on current cyberinfrastructure. They particularly emphasize the issues related to the management of data throughout the workflow lifecycle.

Resource Provisioning Analysis in Virtual Machines Using Statistical Performance of Workloads for Business Applications

2020 ◽  
Vol 17 (9) ◽  
pp. 4156-4161
Author(s):  
Jeny Varghese ◽  
S. Jagannatha
Keyword(s):  
Cloud Computing ◽  
Large Scale ◽  
Virtual Machines ◽  
Resource Provisioning ◽  
Business Applications ◽  
Cloud Federation ◽  
Cloud Datacenter

Cloud Federation is the interconnection of two or more cloud computing settings in order to share configurable processing components such as networks, servers, apps that can be dynamically delivered to customers. Virtualization has been an integral part of cloud computing which provides manageability and utilization of resources. This paper analyses on how jobs of business applications demand and efficiently use the capacity of the resources that are provisioned by the VMs, thereby managing the performance of the applications. The in-depth assessment is based on two large-scale and constant performance traces gathered in a cloud datacenter that host company tools for running distinct apps with regard to requested and used resources.

Online Task Resource Consumption Prediction for Scientific Workflows

2015 ◽  
Vol 25 (03) ◽  
pp. 1541003 ◽  
Author(s):  
Rafael Ferreira da Silva ◽  
Gideon Juve ◽  
Mats Rynge ◽  
Ewa Deelman ◽  
Miron Livny
Keyword(s):  
Input Data ◽  
Resource Provisioning ◽  
Scientific Workflows ◽  
Memory Consumption ◽  
Online Estimation ◽  
Workflow Execution ◽  
Disk Space ◽  
Estimation Process ◽  
Fine Grained ◽  
Task Requirements

Estimates of task runtime, disk space usage, and memory consumption, are commonly used by scheduling and resource provisioning algorithms to support efficient and reliable workflow executions. Such algorithms often assume that accurate estimates are available, but such estimates are difficult to generate in practice. In this work, we first profile five real scientific workflows, collecting fine-grained information such as process I/O, runtime, memory usage, and CPU utilization. We then propose a method to automatically characterize workflow task requirements based on these profiles. Our method estimates task runtime, disk space, and peak memory consumption based on the size of the tasks’ input data. It looks for correlations between the parameters of a dataset, and if no correlation is found, the dataset is divided into smaller subsets using a clustering technique. Task estimates are generated based on the ratio parameter/input data size if they are correlated, or based on the probability distribution function of the parameter. We then propose an online estimation process based on the MAPE-K loop, where task executions are monitored and estimates are updated as more information becomes available. Experimental results show that our online estimation process results in much more accurate predictions than an offline approach, where all task requirements are estimated prior to workflow execution.

scholarly journals Scheduling and Resource Provisioning Algorithms for ScientificWorkflows on Commercial Clouds

2021 ◽  
Author(s):  
◽  
Vahid Arabnejad
Keyword(s):  
High Performance ◽  
Large Scale ◽  
Cost Model ◽  
Universal Access ◽  
Scientific Workflow ◽  
Resource Provisioning ◽  
Scheduling Problem ◽  
Computing Paradigm ◽  
Computationally Intensive ◽  
Workflow Tasks

<p>Basic science is becoming ever more computationally intensive, increasing the need for large-scale compute and storage resources, be they within a High-Performance Computer cluster, or more recently, within the cloud. Commercial clouds have increasingly become a viable platform for hosting scientific analyses and computation due to their elasticity, recent introduction of specialist hardware, and pay-as-you-go cost model. This computing paradigm therefore presents a low capital and low barrier alternative to operating dedicated eScience infrastructure. Indeed, commercial clouds now enable universal access to capabilities previously available to only large well funded research groups. While the potential benefits of cloud computing are clear, there are still significant technical hurdles associated with obtaining the best execution efficiency whilst trading off cost. In most cases, large scale scientific computation is represented as a workflow for scheduling and runtime provisioning. Such scheduling becomes an even more challenging problem on cloud systems due to the dynamic nature of the cloud, in particular, the elasticity, the pricing models (both static and dynamic), the non-homogeneous resource types and the vast array of services. This mapping of workflow tasks onto a set of provisioned instances is an example of the general scheduling problem and is NP-complete. In addition, certain runtime constraints, the most typical being the cost of the computation and the time which that computation requires to complete, must be met. This thesis addresses 'the scientific workflow scheduling problem in cloud', which is to schedule workflow tasks on cloud resources in a way that users meet their defined constraints such as budget and deadline, and providers maximize profits and resource utilization. Moreover, it explores different mechanisms and strategies for distributing defined constraints over a workflow and investigate its impact on the overall cost of the resulting schedule.</p>

scholarly journals An Exotic IWD - SVR Based Approach for Failure Prognostication in Cloud-Based Scientific Workflows

2021 ◽  
Author(s):  
Sridevi S ◽  
Jeevaa Katiravan Jeevaa Katiravan
Keyword(s):  
Large Scale ◽  
Performance Metrics ◽  
Prediction Models ◽  
Fault Tolerant ◽  
Scientific Workflow ◽  
Scientific Workflows ◽  
Support Vector ◽  
Learning Approaches ◽  
Task Failure ◽  
Proactive Measures

Abstract Scientific workflows deserve the emerging attention in sophisticated large-scale scientific problem-solving environments. Though a single task failure occurs in workflow based applications, due to its task dependency nature the reliability of the overall system will be affected drastically. Hence rather than reactive fault tolerant approaches, proactive measures are vital in scientific workflows. This work puts forth an attempt to concentrate on the exploration issue of structuring an Exotic Intelligent Water Drops - Support Vector Regression-based approach for task failure prognostication which facilitates proactive fault tolerance in scientific workflow applications. The failure prediction models in this study have been implemented through SVR-based machine learning approaches and its precision accuracy is optimized by IWDA and various performance metrics were evaluated. The experimental results prove that the proposed approach performs better compared with the other existing techniques.

Sign in / Sign up

Export Citation Format

Share Document