A comparative evaluation of high-performance file transfer systems for data-intensive grid applications

2005 ◽  
Author(s):  
C. Anglano ◽  
M. Canonico
Keyword(s):  
Comparative Evaluation ◽  
High Performance ◽  
File Transfer ◽  
Data Intensive ◽  
Grid Applications

Beyond MPI

2021 ◽  
Vol 49 (4) ◽  
pp. 12-17
Author(s):  
Feilong Liu ◽  
Claude Barthels ◽  
Spyros Blanas ◽  
Hideaki Kimura ◽  
Garret Swart
Keyword(s):  
High Performance ◽  
Processing System ◽  
Complex Interaction ◽  
Remote Memory ◽  
Interaction Patterns ◽  
Round Trip ◽  
Data Processing System ◽  
Data Intensive ◽  
Multiple Round ◽  
Programming Interface

Networkswith Remote DirectMemoryAccess (RDMA) support are becoming increasingly common. RDMA, however, offers a limited programming interface to remote memory that consists of read, write and atomic operations. With RDMA alone, completing the most basic operations on remote data structures often requires multiple round-trips over the network. Data-intensive systems strongly desire higher-level communication abstractions that supportmore complex interaction patterns. A natural candidate to consider is MPI, the de facto standard for developing high-performance applications in the HPC community. This paper critically evaluates the communication primitives of MPI and shows that using MPI in the context of a data processing system comes with its own set of insurmountable challenges. Based on this analysis, we propose a new communication abstraction named RDMO, or Remote DirectMemory Operation, that dispatches a short sequence of reads, writes and atomic operations to remote memory and executes them in a single round-trip.

Custom templates based heterogeneous resource allocation for data-intensive applications

2020 ◽  
Author(s):  
◽  
Ronny Bazan Antequera
Keyword(s):  
High Performance ◽  
Real Data ◽  
University Of Missouri ◽  
Application Performance ◽  
Data Intensive ◽  
Edge Based ◽  
The Right ◽  
Heterogeneous Cloud ◽  
Data Intensive Applications ◽  
Cloud Resources

[ACCESS RESTRICTED TO THE UNIVERSITY OF MISSOURI-COLUMBIA AT REQUEST OF AUTHOR.] The increase of data-intensive applications in science and engineering fields (i.e., bioinformatics, cybermanufacturing) demand the use of high-performance computing resources. However, data-intensive applications' local resources usually present limited capacity and availability due to sizable upfront costs. Moreover, using remote public resources presents constraints at the private edge network domain. Specifically, mis-configured network policies cause bottlenecks due to the other application cross-traffic attempting to use shared networking resources. Additionally, selecting the right remote resources can be cumbersome especially for those users who are interested in the application execution considering nonfunctional requirements such as performance, security and cost. The data-intensive applications have recurrent deployments and similar infrastructure requirements that can be addressed by creating templates. In this thesis, we handle applications requirements through intelligent resource 'abstractions' coupled with 'reusable' approaches that save time and effort in deploying new cloud architectures. Specifically, we design a novel custom template middleware that can retrieve blue prints of resource configuration, technical/policy information, and benchmarks of workflow performance to facilitate repeatable/reusable resource composition. The middleware considers hybrid-recommendation methodology (Online and offline recommendation) to leverage a catalog to rapidly check custom template solution correctness before/during resource consumption. Further, it prescribes application adaptations by fostering effective social interactions during the application's scaling stages. Based on the above approach, we organize the thesis contributions under two main thrusts: (i) Custom Templates for Cloud Networking for Data-intensive Applications: This involves scheduling transit selection, engineering at the campus-edge based upon real-time policy control. Our solution ensures prioritized application performance delivery for multi-tenant traffic profiles from a diverse set of actual data intensive applications in bioinformatics. (ii) Custom Templates for Cloud Computing for Data-intensive Applications: This involves recommending cloud resources for data-intensive applications based on a custom template catalog. We develop a novel expert system approach that is implemented as a middleware to abstracts data-intensive application requirements for custom templates composition. We uniquely consider heterogeneous cloud resources selection for the deployment of cloud architectures for real data-intensive applications in cybermanufacturing.

scholarly journals Infrastructure and Energy Conservation in Big Data Computing: A Survey

2019 ◽  
Vol 2 ◽  
pp. 73-82
Author(s):  
Ewa Niewiadomska-Szynkiewicz ◽  
Michał P. Karpowicz
Keyword(s):  
Big Data ◽  
High Performance ◽  
Physical Sciences ◽  
Communication Management ◽  
Data Intensive ◽  
Efficient Data ◽  
Software Platforms ◽  
Allocation Algorithms ◽  
And Storage ◽  
Big Data Computing

Progress in life, physical sciences and technology depends on efficient data-mining and modern computing technologies. The rapid growth of data-intensive domains requires a continuous development of new solutions for network infrastructure, servers and storage in order to address Big Datarelated problems. Development of software frameworks, include smart calculation, communication management, data decomposition and allocation algorithms is clearly one of the major technological challenges we are faced with. Reduction in energy consumption is another challenge arising in connection with the development of efficient HPC infrastructures. This paper addresses the vital problem of energy-efficient high performance distributed and parallel computing. An overview of recent technologies for Big Data processing is presented. The attention is focused on the most popular middleware and software platforms. Various energy-saving approaches are presented and discussed as well.

Empirical Performance Analysis of HPC Benchmarks Across Variations in Cloud Computing

2013 ◽  
Vol 3 (1) ◽  
pp. 13-26 ◽  
Author(s):  
Sanjay P. Ahuja ◽  
Sindhu Mani
Keyword(s):  
Data Storage ◽  
High Performance ◽  
Large Data ◽  
Extensive Study ◽  
Memory Bandwidth ◽  
Platform As A Service ◽  
Data Intensive ◽  
Computational Performance ◽  
Empirical Performance ◽  
Data Intensive Applications

High Performance Computing (HPC) applications are scientific applications that require significant CPU capabilities. They are also data-intensive applications requiring large data storage. While many researchers have examined the performance of Amazon’s EC2 platform across some HPC benchmarks, an extensive study and their comparison between Amazon’s EC2 and Microsoft’s Windows Azure is largely missing with metrics such as memory bandwidth, I/O performance, and communication and computational performance. The purpose of this paper is to implement existing benchmarks to evaluate and analyze these metrics for EC2 and Windows Azure that span both Infrastructure-as-a-Service and Platform-as-a-Service types. This was accomplished by running MPI versions of STREAM, Interleaved or Random (IOR) and NAS Parallel (NPB) benchmarks on small and medium instance types. In addition a new EC2 medium instance type (m1.medium) was also included in the analysis. These benchmarks measure the memory bandwidth, I/O performance, communication and computational performance.

Sign in / Sign up

Export Citation Format

Share Document