Workflow management — An exercise in distributed computing

1997 ◽  
pp. 15-19 ◽  
Author(s):  
Andreas Reuter
Keyword(s):  
Distributed Computing ◽  
Workflow Management

Data-Aware Distributed Computing

2012 ◽  
pp. 1-27 ◽  
Author(s):  
Esma Yildirim ◽  
Mehmet Balman ◽  
Tevfik Kosar
Keyword(s):  
Distributed Computing ◽  
Workflow Management ◽  
Data Access ◽  
New Paradigm ◽  
Distributed Computing Systems ◽  
Computing Systems ◽  
Data Movement ◽  
End To End ◽  
Commercial Applications ◽  
Main Components

With the continuous increase in the data requirements of scientific and commercial applications, access to remote and distributed data has become a major bottleneck for end-to-end application performance. Traditional distributed computing systems closely couple data access and computation, and generally, data access is considered a side effect of computation. The limitations of traditional distributed computing systems and CPU-oriented scheduling and workflow management tools in managing complex data handling have motivated a newly emerging era: data-aware distributed computing. In this chapter, the authors elaborate on how the most crucial distributed computing components, such as scheduling, workflow management, and end-to-end throughput optimization, can become “data-aware.” In this new computing paradigm, called data-aware distributed computing, data placement activities are represented as full-featured jobs in the end-to-end workflow, and they are queued, managed, scheduled, and optimized via a specialized data-aware scheduler. As part of this new paradigm, the authors present a set of tools for mitigating the data bottleneck in distributed computing systems, which consists of three main components: a data-aware scheduler, which provides capabilities such as planning, scheduling, resource reservation, job execution, and error recovery for data movement tasks; integration of these capabilities to the other layers in distributed computing, such as workflow planning; and further optimization of data movement tasks via dynamic tuning of underlying protocol transfer parameters.

scholarly journals Implementation of ATLAS Distributed Computing monitoring dashboards using InfluxDB and Grafana

2020 ◽  
Vol 245 ◽  
pp. 03031 ◽  
Author(s):  
Thomas Beermann ◽  
Aleksandr Alekseev ◽  
Dario Baberis ◽  
Sabine Crépé-Renaudin ◽  
Johannes Elmsheuser ◽  
...  
Keyword(s):  
Distributed Computing ◽  
Workflow Management ◽  
Monitoring Tools ◽  
Data Store ◽  
Standard Code ◽  
Data Flows ◽  
Code Base ◽  
Monitoring Applications ◽  
New Infrastructure

For the last 10 years, the ATLAS Distributed Computing project has based its monitoring infrastructure on a set of custom designed dashboards provided by CERN. This system functioned very well for LHC Runs 1 and 2, but its maintenance has progressively become more difficult and the conditions for Run 3, starting in 2021, will be even more demanding; hence a more standard code base and more automatic operations are needed. A new infrastructure has been provided by CERN, based on InfluxDB as the data store and Grafana as the display environment. ATLAS has adapted and further developed its monitoring tools to use this infrastructure for data and workflow management monitoring and accounting dashboards, expanding the range of previous possibilities with the aim to achieve a single, simpler, environment for all monitoring applications. This document describes these tools and the data flows for monitoring and accounting.

scholarly journals The Future of Distributed Computing Systems in ATLAS: Boldly Venturing Beyond Grids

2019 ◽  
Vol 214 ◽  
pp. 03047 ◽  
Author(s):  
Fernando Barreiro ◽  
Doug Benjamin ◽  
Taylor Childers ◽  
Kaushik De ◽  
Johannes Elmsheuser ◽  
...  
Keyword(s):  
Distributed Computing ◽  
Workflow Management ◽  
Distributed Computing Systems ◽  
Computing Systems ◽  
External Resources ◽  
Network Provisioning ◽  
Software Distribution ◽  
Computing Grid ◽  
Analysis System

Since 2010 the Production and Distributed Analysis system (PanDA) for the ATLAS experiment at the Large Hadron Colliderhas seen big changes to accommodate new types of distributed computing resources: clouds, HPCs, volunteer computers and other external resources. While PanDA was originally designed for fairly homogeneous resources available through the Worldwide LHC Computing Grid, the new resources are heterogeneous, at diverse scales and with diverse interfaces. Up to a fifth of the resources available to ATLAS are of such new types and require special techniques for integration into PanDA. In this talk, we present the nature and scale of these resources. We provide an overview of the various challenges faced, spanning infrastructure, software distribution, workload requirements, scaling requirements, workflow management, data management, network provisioning, and associated software and computing facilities. We describe the strategies for integrating these heterogeneous resources into ATLAS, and the new software components being developed in PanDA to efficiently use them. Plans for software and computing evolution to meet the needs of LHC operations and upgrade in the long term future will be discussed.

Power Consumption Evaluation of Distributed Computing Network Considering Traffic Locality

2012 ◽  
Vol E95.B (8) ◽  
pp. 2538-2548
Author(s):  
Yukio OGAWA ◽  
Go HASEGAWA ◽  
Masayuki MURATA
Keyword(s):  
Distributed Computing ◽  
Power Consumption

Matlab and Parallel Computing

2012 ◽  
Vol 17 (4) ◽  
pp. 207-216 ◽  
Author(s):  
Magdalena Szymczyk ◽  
Piotr Szymczyk
Keyword(s):  
Image Processing ◽  
Signal Processing ◽  
Parallel Computing ◽  
Distributed Computing ◽  
Control Systems ◽  
High Performance ◽  
Parallel Applications ◽  
Process Simulations ◽  
Key Features ◽  
Financial Process

Abstract The MATLAB is a technical computing language used in a variety of fields, such as control systems, image and signal processing, visualization, financial process simulations in an easy-to-use environment. MATLAB offers "toolboxes" which are specialized libraries for variety scientific domains, and a simplified interface to high-performance libraries (LAPACK, BLAS, FFTW too). Now MATLAB is enriched by the possibility of parallel computing with the Parallel Computing ToolboxTM and MATLAB Distributed Computing ServerTM. In this article we present some of the key features of MATLAB parallel applications focused on using GPU processors for image processing.

Activity of public control entities and development of distributed computing and distributed data storage systems

2018 ◽  
pp. 14-22
Author(s):  
D. V. Gribanov
Keyword(s):  
Distributed Computing ◽  
Data Storage ◽  
Storage Systems ◽  
Legal Regulation ◽  
Distributed Data ◽  
Distributed Data Storage ◽  
Public Control ◽  
Blockchain Technology ◽  
Legal Method ◽  
Digital Assets

Introduction. This article is devoted to legal regulation of digital assets turnover, utilization possibilities of distributed computing and distributed data storage systems in activities of public authorities and entities of public control. The author notes that some national and foreign scientists who study a “blockchain” technology (distributed computing and distributed data storage systems) emphasize its usefulness in different activities. Data validation procedure of digital transactions, legal regulation of creation, issuance and turnover of digital assets need further attention.Materials and methods. The research is based on common scientific (analysis, analogy, comparing) and particular methods of cognition of legal phenomena and processes (a method of interpretation of legal rules, a technical legal method, a formal legal method and a formal logical one).Results of the study. The author conducted an analysis which resulted in finding some advantages of the use of the “blockchain” technology in the sphere of public control which are as follows: a particular validation system; data that once were entered in the system of distributed data storage cannot be erased or forged; absolute transparency of succession of actions while exercising governing powers; automatic repeat of recurring actions. The need of fivefold validation of exercising governing powers is substantiated. The author stresses that the fivefold validation shall ensure complex control over exercising of powers by the civil society, the entities of public control and the Russian Federation as a federal state holding sovereignty over its territory. The author has also conducted a brief analysis of judicial decisions concerning digital transactions.Discussion and conclusion. The use of the distributed data storage system makes it easier to exercise control due to the decrease of risks of forge, replacement or termination of data. The author suggests defining digital transaction not only as some actions with digital assets, but also as actions toward modification and addition of information about legal facts with a purpose of its establishment in the systems of distributed data storage. The author suggests using the systems of distributed data storage for independent validation of information about activities of the bodies of state authority. In the author’s opinion, application of the “blockchain” technology may result not only in the increase of efficiency of public control, but also in the creation of a new form of public control – automatic control. It is concluded there is no legislation basis for regulation of legal relations concerning distributed data storage today.

Sign in / Sign up

Export Citation Format

Share Document