A Distributed Computing Service for Neural Networks and Its Application to Flood Peak Forecasting

2006 ◽  
pp. 890-896 ◽  
Author(s):  
Jun Zhu ◽  
Chunbo Liu ◽  
Jianhua Gong ◽  
Daojun Wang ◽  
Tao Song
Keyword(s):  
Neural Networks ◽  
Distributed Computing ◽  
Flood Peak

scholarly journals DeepWeave: Accelerating Job Completion Time with Deep Reinforcement Learning-based Coflow Scheduling

2020 ◽  
Author(s):  
Penghao Sun ◽  
Zehua Guo ◽  
Junchao Wang ◽  
Junfei Li ◽  
Julong Lan ◽  
...  
Keyword(s):  
Neural Networks ◽  
Reinforcement Learning ◽  
Distributed Computing ◽  
Completion Time ◽  
Heuristic Algorithms ◽  
Expert Knowledge ◽  
Real Life ◽  
Processing Efficiency ◽  
Multi Stage ◽  
The Neural Networks

To improve the processing efficiency of jobs in distributed computing, the concept of coflow is proposed. A coflow is a collection of flows that are semantically correlated in a multi-stage computation task. A job consists of multiple coflows and can be usually formulated as a Directed-Acyclic Graph (DAG). A proper scheduling of coflows can significantly reduce the completion time of jobs in distributed computing. However, this scheduling problem is proved to be NP-hard. Different from existing schemes that use hand-crafted heuristic algorithms to solve this problem, in this paper, we propose a Deep Reinforcement Learning (DRL) framework named DeepWeave to generate coflow scheduling policies. To improve the inter-coflow scheduling ability in the job DAG, DeepWeave employs a Graph Neural Network (GNN) to process the DAG information. DeepWeave learns from the history workload trace to train the neural networks of the DRL agent and encodes the scheduling policy in the neural networks, which make coflow scheduling decisions without expert knowledge or a pre-assumed model. The proposed scheme is evaluated with a simulator using real-life traces. Simulation results show that DeepWeave completes jobs at least 1.7X faster than the state-of-the-art solutions.

Sign in / Sign up

Export Citation Format

Share Document