scholarly journals NUMA-Aware Thread Scheduling for Big Data Transfers over Terabits Network Infrastructure

2018 ◽  
Vol 2018 ◽  
pp. 1-8
Author(s):  
Taeuk Kim ◽  
Awais Khan ◽  
Youngjae Kim ◽  
Preethika Kasu ◽  
Scott Atchley
Keyword(s):  
Big Data ◽  
Data Transfer ◽  
Analytical Data ◽  
File Systems ◽  
Memory Access ◽  
Thread Scheduling ◽  
Data Scheduling ◽  
Data Transfers ◽  
With Memory ◽  
Existing Data

The evergrowing trend of big data has led scientists to share and transfer the simulation and analytical data across the geodistributed research and computing facilities. However, the existing data transfer frameworks used for data sharing lack the capability to adopt the attributes of the underlying parallel file systems (PFS). LADS (Layout-Aware Data Scheduling) is an end-to-end data transfer tool optimized for terabit network using a layout-aware data scheduling via PFS. However, it does not consider the NUMA (Nonuniform Memory Access) architecture. In this paper, we propose a NUMA-aware thread and resource scheduling for optimized data transfer in terabit network. First, we propose distributed RMA buffers to reduce memory controller contention in CPU sockets and then schedule the threads based on CPU socket and NUMA nodes inside CPU socket to reduce memory access latency. We design and implement the proposed resource and thread scheduling in the existing LADS framework. Experimental results showed from 21.7% to 44% improvement with memory-level optimizations in the LADS framework as compared to the baseline without any optimization.

scholarly journals Optimized Data Transfers Based on the OpenCL Event Management Mechanism

2015 ◽  
Vol 2015 ◽  
pp. 1-16 ◽  
Author(s):  
Hiroyuki Takizawa ◽  
Shoichi Hirasawa ◽  
Makoto Sugawara ◽  
Isaac Gelado ◽  
Hiroaki Kobayashi ◽  
...  
Keyword(s):  
Big Data ◽  
Data Transfer ◽  
Data File ◽  
Transfer Performance ◽  
Event Management ◽  
Management Mechanism ◽  
File Access ◽  
Programming Techniques ◽  
Data Transfers ◽  
Main Thread

In standard OpenCL programming, hosts are supposed to control their compute devices. Since compute devices are dedicated to kernel computation, only hosts can execute several kinds of data transfers such as internode communication and file access. These data transfers require one host to simultaneously play two or more roles due to the need for collaboration between the host and devices. The codes for such data transfers are likely to be system-specific, resulting in low portability. This paper proposes an OpenCL extension that incorporates such data transfers into the OpenCL event management mechanism. Unlike the current OpenCL standard, the main thread running on the host is not blocked to serialize dependent operations. Hence, an application can easily use the opportunities to overlap parallel activities of hosts and compute devices. In addition, the implementation details of data transfers are hidden behind the extension, and application programmers can use the optimized data transfers without any tricky programming techniques. The evaluation results show that the proposed extension can use the optimized data transfer implementation and thereby increase the sustained data transfer performance by about 18% for a real application accessing a big data file.

scholarly journals BIG DATA TRANSFER FOR TABLET-CLASS MACHINES

2014 ◽  
pp. 316-323
Author(s):  
Tevaganthan Veluppillai ◽  
Brandon Ortiz ◽  
Robert E. Hiromoto
Keyword(s):  
Big Data ◽  
Comparative Study ◽  
Data Transfer ◽  
Data File ◽  
Transport Protocol ◽  
Transport Protocols ◽  
Client Server ◽  
Remote Method Invocation ◽  
Block Data ◽  
Data Transfers

Several well-known data transfer protocols are presented in a comparative study to address the issue of big data transfer for tablet-class machines. The data transfer protocols include standard Java and C++, and block-data transfers protocols that use both the Java New IO (NIO) and the Zerocopy libraries, and a block-data C++ transfer protocol. Several experiments are described and results compared against the standard Java IO and C++ (stream-based file transport protocols). The motivation for this study is the development of a client/server big data file transport protocol for tablet-class client machines that rely on the Java Remote Method Invocation (RMI) package for distributed computing.

scholarly journals FTLADS: Object-Logging Based Fault-Tolerant Big Data Transfer System Using Layout Aware Data Scheduling

IEEE Access ◽  
2019 ◽  
Vol 7 ◽  
pp. 37448-37462
Author(s):  
Preethika Kasu ◽  
Taeuk Kim ◽  
Jung-Ho Um ◽  
Kyongseok Park ◽  
Scott Atchley ◽  
...  
Keyword(s):  
Big Data ◽  
Data Transfer ◽  
Fault Tolerant ◽  
Transfer System ◽  
Data Scheduling

scholarly journals Bandwidth scheduling for big data transfer with two variable node-disjoint paths

2020 ◽  
Vol 22 (2) ◽  
pp. 130-144
Author(s):  
Aiqin Hou ◽  
Chase Qishi Wu ◽  
Liudong Zuo ◽  
Xiaoyang Zhang ◽  
Tao Wang ◽  
...  
Keyword(s):  
Big Data ◽  
Data Transfer ◽  
Disjoint Paths ◽  
Bandwidth Scheduling ◽  
Variable Node

scholarly journals DynDL: Scheduling Data-Locality-Aware Tasks with Dynamic Data Transfer Cost for Multicore-Server-Based Big Data Clusters

2018 ◽  
Vol 8 (11) ◽  
pp. 2216
Author(s):  
Jiahui Jin ◽  
Qi An ◽  
Wei Zhou ◽  
Jiakai Tang ◽  
Runqun Xiong
Keyword(s):  
Big Data ◽  
Data Processing ◽  
Processing Time ◽  
Data Transfer ◽  
Data Locality ◽  
Free Time ◽  
Time Data ◽  
Dynamic Data ◽  
Network Bandwidth ◽  
Transfer Cost

Network bandwidth is a scarce resource in big data environments, so data locality is a fundamental problem for data-parallel frameworks such as Hadoop and Spark. This problem is exacerbated in multicore server-based clusters, where multiple tasks running on the same server compete for the server’s network bandwidth. Existing approaches solve this problem by scheduling computational tasks near the input data and considering the server’s free time, data placements, and data transfer costs. However, such approaches usually set identical values for data transfer costs, even though a multicore server’s data transfer cost increases with the number of data-remote tasks. Eventually, this hampers data-processing time, by minimizing it ineffectively. As a solution, we propose DynDL (Dynamic Data Locality), a novel data-locality-aware task-scheduling model that handles dynamic data transfer costs for multicore servers. DynDL offers greater flexibility than existing approaches by using a set of non-decreasing functions to evaluate dynamic data transfer costs. We also propose online and offline algorithms (based on DynDL) that minimize data-processing time and adaptively adjust data locality. Although DynDL is NP-complete (nondeterministic polynomial-complete), we prove that the offline algorithm runs in quadratic time and generates optimal results for DynDL’s specific uses. Using a series of simulations and real-world executions, we show that our algorithms are 30% better than algorithms that do not consider dynamic data transfer costs in terms of data-processing time. Moreover, they can adaptively adjust data localities based on the server’s free time, data placement, and network bandwidth, and schedule tens of thousands of tasks within subseconds or seconds.

Performance Evaluation of Protocols for Big Data Transfers

Big Data ◽  
2016 ◽  
pp. 43-95
Author(s):  
Se-young Yu ◽  
Nevil Brownlee ◽  
Aniket Mahanti
Keyword(s):  
Big Data ◽  
Performance Evaluation ◽  
Data Transfers

High Performance Storage for Big Data Analytics and Visualization

2018 ◽  
pp. 254-275
Author(s):  
Armando Fandango ◽  
William Rivera
Keyword(s):  
Big Data ◽  
High Speed ◽  
High Performance ◽  
File System ◽  
Predictive Analytics ◽  
Big Data Analytics ◽  
File Systems ◽  
Distributed Applications ◽  
System Level ◽  
File Formats

Scientific Big Data being gathered at exascale needs to be stored, retrieved and manipulated. The storage stack for scientific Big Data includes a file system at the system level for physical organization of the data, and a file format and input/output (I/O) system at the application level for logical organization of the data; both of them of high-performance variety for exascale. The high-performance file system is designed with concurrent access, high-speed transmission and fault tolerance characteristics. High-performance file formats and I/O are designed to allow parallel and distributed applications with easy and fast access to Big Data. These specialized file formats make it easier to store and access Big Data for scientific visualization and predictive analytics. This chapter provides a brief review of the characteristics of high-performance file systems such as Lustre and GPFS, and high-performance file formats such as HDF5, NetCDF, MPI-IO, and HDFS.

Sign in / Sign up

Export Citation Format

Share Document