Prophet: Speeding up Distributed DNN Training with Predictable Communication Scheduling

2021 ◽  
Author(s):  
Zhenwei Zhang ◽  
Qiang Qi ◽  
Ruitao Shang ◽  
Li Chen ◽  
Fei Xu
Keyword(s):  
Communication Scheduling

DERIVING AND SCHEDULING COMMUNICATION OPERATIONS FOR GENERIC SKELETON IMPLEMENTATIONS

2005 ◽  
Vol 15 (03) ◽  
pp. 337-352 ◽  
Author(s):  
THOMAS NITSCHE
Keyword(s):  
Iterative Solution ◽  
Synchronous Communication ◽  
Data Types ◽  
Communication Scheduling ◽  
Overlapping Data ◽  
Abstract Notion ◽  
Algebraic Data Types ◽  
Data Elements ◽  
Communication Operations

Data distributions are an abstract notion for describing parallel programs by means of overlapping data structures. A generic data distribution layer serves as a basis for implementing specific data distributions over arbitrary algebraic data types and arrays as well as generic skeletons. The necessary communication operations for exchanging overlapping data elements are derived automatically from the specification of the overlappings. This paper describes how the communication operations used internally by the generic skeletons are derived, especially for the asynchronous and synchronous communication scheduling. As a case study, we discuss the iterative solution of PDEs and compare a hand-coded MPI version with a skeletal one based on overlapping data distributions.

Sign in / Sign up

Export Citation Format

Share Document