scholarly journals The Control Method of Twin Delayed Deep Deterministic Policy Gradient with Rebirth Mechanism to Multi-DOF Manipulator

Electronics ◽  
2021 ◽  
Vol 10 (7) ◽  
pp. 870
Author(s):  
Yangyang Hou ◽  
Huajie Hong ◽  
Zhaomei Sun ◽  
Dasheng Xu ◽  
Zhe Zeng
Keyword(s):  
Reinforcement Learning ◽  
Control Method ◽  
Kinematic Model ◽  
Degree Of Freedom ◽  
Learning Ability ◽  
Reward Function ◽  
Policy Gradient ◽  
Evaluation Algorithm ◽  
Improved Learning ◽  
Point To Point

As a research hotspot in the field of artificial intelligence, the application of deep reinforcement learning to the learning of the motion ability of a manipulator can help to improve the learning of the motion ability of a manipulator without a kinematic model. To suppress the overestimation bias of values in Deep Deterministic Policy Gradient (DDPG) networks, the Twin Delayed Deep Deterministic Policy Gradient (TD3) was proposed. This paper further suppresses the overestimation bias of values for multi-degree of freedom (DOF) manipulator learning based on deep reinforcement learning. Twin Delayed Deep Deterministic Policy Gradient with Rebirth Mechanism (RTD3) was proposed. The experimental results show that RTD3 applied to multi degree freedom manipulators is in place, with an improved learning ability by 29.15% on the basis of TD3. In this paper, a step-by-step reward function is proposed specifically for the learning and innovation of the multi degree of freedom manipulator’s motion ability. The view of continuous decision-making and process problem is used to guide the learning of the manipulator, and the learning efficiency is improved by optimizing the playback of experience. In order to measure the point-to-point position motion ability of a manipulator, a new evaluation index based on the characteristics of the continuous decision process problem, energy efficiency distance, is presented in this paper, which can evaluate the learning quality of the manipulator motion ability by a more comprehensive and fair evaluation algorithm.

scholarly journals Building a Connected Communication Network for UAV Clusters Using DE-MADDPG

Symmetry ◽  
2021 ◽  
Vol 13 (8) ◽  
pp. 1537
Author(s):  
Zixiong Zhu ◽  
Nianhao Xie ◽  
Kang Zong ◽  
Lei Chen
Keyword(s):  
Communication Network ◽  
Reinforcement Learning ◽  
Control Method ◽  
Control Mechanisms ◽  
Motion Information ◽  
Reward Function ◽  
Learning Framework ◽  
Policy Gradient ◽  
Multi Agent ◽  
Virtual Leader

Clusters of unmanned aerial vehicles (UAVs) are often used to perform complex tasks. In such clusters, the reliability of the communication network connecting the UAVs is an essential factor in their collective efficiency. Due to the complex wireless environment, however, communication malfunctions within the cluster are likely during the flight of UAVs. In such cases, it is important to control the cluster and rebuild the connected network. The asymmetry of the cluster topology also increases the complexity of the control mechanisms. The traditional control methods based on cluster consistency often rely on the motion information of the neighboring UAVs. The motion information, however, may become unavailable because of the interrupted communications. UAV control algorithms based on deep reinforcement learning have achieved outstanding results in many fields. Here, we propose a cluster control method based on the Decomposed Multi-Agent Deep Deterministic Policy Gradient (DE-MADDPG) to rebuild a communication network for UAV clusters. The DE-MADDPG improves the framework of the traditional multi-agent deep deterministic policy gradient (MADDPG) algorithm by decomposing the reward function. We further introduce the reward reshaping function to facilitate the convergence of the algorithm in sparse reward environments. To address the instability of the state-space in the reinforcement learning framework, we also propose the notion of the virtual leader–follower model. Extensive simulations show that the success rate of the DE-MADDPG is higher than that of the MADDPG algorithm, confirming the effectiveness of the proposed method.

Preceding vehicle following algorithm with human driving characteristics

2021 ◽  
pp. 095440702098154
Author(s):  
Feng Pan ◽  
Hong Bao
Keyword(s):  
Reinforcement Learning ◽  
Weight Vector ◽  
Gradient Algorithm ◽  
Inner Product ◽  
Inverse Reinforcement Learning ◽  
Reward Function ◽  
Human Driver ◽  
Policy Gradient ◽  
Preceding Vehicle ◽  
Action Spaces

This paper proposes a new approach of using reinforcement learning (RL) to train an agent to perform the task of vehicle following with human driving characteristics. We refer to the ideal of inverse reinforcement learning to design the reward function of the RL model. The factors that need to be weighed in vehicle following were vectorized into reward vectors, and the reward function was defined as the inner product of the reward vector and weights. Driving data of human drivers was collected and analyzed to obtain the true reward function. The RL model was trained with the deterministic policy gradient algorithm because the state and action spaces are continuous. We adjusted the weight vector of the reward function so that the value vector of the RL model could continuously approach that of a human driver. After dozens of rounds of training, we selected the policy with the nearest value vector to that of a human driver and tested it in the PanoSim simulation environment. The results showed the desired performance for the task of an agent following the preceding vehicle safely and smoothly.

Continuous reinforcement learning based ramp jump control for single-track two-wheeled robots

2021 ◽  
pp. 014233122110378
Author(s):  
Qingyuan Zheng ◽  
Duo Wang ◽  
Zhang Chen ◽  
Yiyong Sun ◽  
Bin Liang
Keyword(s):  
Reinforcement Learning ◽  
Energy Savings ◽  
Control Method ◽  
Learning Control ◽  
Action Space ◽  
Gradient Algorithm ◽  
Single Track ◽  
Wheeled Robots ◽  
Reward Function ◽  
Wheeled Robot

Single-track two-wheeled robots have become an important research topic in recent years, owing to their simple structure, energy savings and ability to run on narrow roads. However, the ramp jump remains a challenging task. In this study, we propose to realize a single-track two-wheeled robot ramp jump. We present a control method that employs continuous action reinforcement learning techniques for single-track two-wheeled robot control. We design a novel reward function for reinforcement learning, optimize the dimensions of the action space, and enable training under the deep deterministic policy gradient algorithm. Finally, we validate the control method through simulation experiments and successfully realize the single-track two-wheeled robot ramp jump task. Simulation results validate that the control method is effective and has several advantages over high-dimension action space control, reinforcement learning control of sparse reward function and discrete action reinforcement learning control.

scholarly journals Deep Reinforcement Learning Based Left-Turn Connected and Automated Vehicle Control at Signalized Intersection in Vehicle-to-Infrastructure Environment

Information ◽  
2020 ◽  
Vol 11 (2) ◽  
pp. 77 ◽  
Author(s):  
Juan Chen ◽  
Zhengxuan Xue ◽  
Daiqian Fan
Keyword(s):  
Reinforcement Learning ◽  
Control Method ◽  
Signalized Intersection ◽  
Signal Control ◽  
Left Turn ◽  
Automated Vehicle ◽  
Whole Process ◽  
Policy Gradient ◽  
Experience Replay ◽  
Automated Vehicle Control

In order to solve the problem of vehicle delay caused by stops at signalized intersections, a micro-control method of a left-turning connected and automated vehicle (CAV) based on an improved deep deterministic policy gradient (DDPG) is designed in this paper. In this paper, the micro-control of the whole process of a left-turn vehicle approaching, entering, and leaving a signalized intersection is considered. In addition, in order to solve the problems of low sampling efficiency and overestimation of the critic network of the DDPG algorithm, a positive and negative reward experience replay buffer sampling mechanism and multi-critic network structure are adopted in the DDPG algorithm in this paper. Finally, the effectiveness of the signal control method, six DDPG-based methods (DDPG, PNRERB-1C-DDPG, PNRERB-3C-DDPG, PNRERB-5C-DDPG, PNRERB-5CNG-DDPG, and PNRERB-7C-DDPG), and four DQN-based methods (DQN, Dueling DQN, Double DQN, and Prioritized Replay DQN) are verified under 0.2, 0.5, and 0.7 saturation degrees of left-turning vehicles at a signalized intersection within a VISSIM simulation environment. The results show that the proposed deep reinforcement learning method can get a number of stops benefits ranging from 5% to 94%, stop time benefits ranging from 1% to 99%, and delay benefits ranging from −17% to 93%, respectively compared with the traditional signal control method.

Stiffness-based trajectory planning of a 6-DOF cable-driven parallel manipulator

2016 ◽  
Vol 231 (21) ◽  
pp. 3999-4011 ◽  
Author(s):  
Wenjia Zhang ◽  
Weiwei Shang ◽  
Bin Zhang ◽  
Fei Zhang ◽  
Shuang Cong
Keyword(s):  
Velocity Profile ◽  
Trajectory Planning ◽  
Parallel Manipulator ◽  
Kinematic Model ◽  
Degree Of Freedom ◽  
Performance Indices ◽  
Curved Trajectories ◽  
Quintic Polynomial ◽  
Point To Point ◽  
Six Degree Of Freedom

The stiffness of the cable-driven parallel manipulator is usually poor because of the cable flexibility, and the existing methods on trajectory planning mainly take the minimum time and the optimal energy into account, not the stiffness. To solve it, the effects of different trajectories on stiffness are studied for a six degree-of-freedom cable-driven parallel manipulator, according to the kinematic model and the dynamic model. The condition number and the minimum eigenvalue of the dimensionally homogeneous stiffness matrix are selected as performance indices to analyze the stiffness changes during the motion. The simulation experiments are implemented on a six degree-of-freedom cable-driven parallel manipulator, to study the stiffness of three different trajectory planning approaches such as S-type velocity profile, quintic polynomial, and trigonometric function. The accelerations of different methods are analyzed, and the stiffness performances for the methods are compared after planning the point-to-point straight and the curved trajectories. The simulation results indicate that the quintic polynomial and S-type velocity profile have the optimal performance to keep the stiffness stable during the motion control and the travel time of the quintic polynomial can be optimized sufficiently while keeping stable.

scholarly journals Toward Diverse Text Generation with Inverse Reinforcement Learning

2018 ◽  
Author(s):  
Zhan Shi ◽  
Xinchi Chen ◽  
Xipeng Qiu ◽  
Xuanjing Huang
Keyword(s):  
Reinforcement Learning ◽  
Generative Models ◽  
Training Data ◽  
Great Success ◽  
Text Generation ◽  
Inverse Reinforcement Learning ◽  
Reward Function ◽  
Total Reward ◽  
Policy Gradient ◽  
Adversarial Models

Text generation is a crucial task in NLP. Recently, several adversarial generative models have been proposed to improve the exposure bias problem in text generation. Though these models gain great success, they still suffer from the problems of reward sparsity and mode collapse. In order to address these two problems, in this paper, we employ inverse reinforcement learning (IRL) for text generation. Specifically, the IRL framework learns a reward function on training data, and then an optimal policy to maximum the expected total reward. Similar to the adversarial models, the reward and policy function in IRL are optimized alternately. Our method has two advantages: (1) the reward function can produce more dense reward signals. (2) the generation policy, trained by ``entropy regularized'' policy gradient, encourages to generate more diversified texts. Experiment results demonstrate that our proposed method can generate higher quality texts than the previous methods.

scholarly journals A Decision-Making Model for Self-Driving Vehicles Based on Overtaking Frequency

2021 ◽  
Vol 2021 ◽  
pp. 1-13
Author(s):  
Mengyuan Huang ◽  
Shiwu Li ◽  
Mengzhu Guo ◽  
Lihong Han
Keyword(s):  
Decision Model ◽  
Assessment Method ◽  
State Assessment ◽  
The Self ◽  
Decision System ◽  
Reward Function ◽  
State Evaluation ◽  
Policy Gradient ◽  
Evaluation Algorithm ◽  
Decision Making Model

The driving state of a self-driving vehicle represents an important component in the self-driving decision system. To ensure the safe and efficient driving state of a self-driving vehicle, the driving state of the self-driving vehicle needs to be evaluated quantitatively. In this paper, a driving state assessment method for the decision system of self-driving vehicles is proposed. First, a self-driving vehicle and surrounding vehicles are compared in terms of the overtaking frequency (OTF), and an OTF-based driving state evaluation algorithm is proposed considering the future driving efficiency. Next, a decision model based on the deep deterministic policy gradient (DDPG) algorithm and the proposed method is designed, and the driving state assessment method is integrated with the existing time-to-collision (TTC) and minimum safe distance. In addition, the reward function and multiple driving scenarios are designed so that the most efficient driving strategy at the current moment can be determined by optimal search under the condition of ensuring safety. Finally, the proposed decision model is verified by simulations in four three-lane highway scenarios. The simulation results show that the proposed decision model that integrates the self-driving vehicle driving state assessment method can help self-driving vehicles to drive safely and to maintain good maneuverability.

On the use of the policy gradient and Hessian in inverse reinforcement learning

2020 ◽  
Vol 14 (1) ◽  
pp. 117-150
Author(s):  
Alberto Maria Metelli ◽  
Matteo Pirotta ◽  
Marcello Restelli
Keyword(s):  
Reinforcement Learning ◽  
Sequential Decision ◽  
Inverse Reinforcement Learning ◽  
Reward Function ◽  
Model Free ◽  
Learning Speed ◽  
Policy Gradient ◽  
Continuous Domains ◽  
Learning Policies ◽  
Finite Domains

Reinforcement Learning (RL) is an effective approach to solve sequential decision making problems when the environment is equipped with a reward function to evaluate the agent’s actions. However, there are several domains in which a reward function is not available and difficult to estimate. When samples of expert agents are available, Inverse Reinforcement Learning (IRL) allows recovering a reward function that explains the demonstrated behavior. Most of the classic IRL methods, in addition to expert’s demonstrations, require sampling the environment to evaluate each reward function, that, in turn, is built starting from a set of engineered features. This paper is about a novel model-free IRL approach that does not require to specify a function space where to search for the expert’s reward function. Leveraging on the fact that the policy gradient needs to be zero for an optimal policy, the algorithm generates an approximation space for the reward function, in which a reward is singled out employing a second-order criterion. After introducing our approach for finite domains, we extend it to continuous ones. The empirical results, on both finite and continuous domains, show that the reward function recovered by our algorithm allows learning policies that outperform those obtained with the true reward function, in terms of learning speed.

Quadrotor Motion Control Using Deep Reinforcement Learning

2021 ◽  
Author(s):  
Zifei Jiang ◽  
Alan F. Lynch
Keyword(s):  
Reinforcement Learning ◽  
Neural Nets ◽  
Neural Net ◽  
Reward Function ◽  
Model Free ◽  
Policy Gradient ◽  
Aerial Vehicle ◽  
Stochastic Controller ◽  
Policy Optimization ◽  
Gradient Approach

We present a deep neural net-based controller trained by a model-free reinforcement learning (RL) algorithm to achieve hover stabilization for a quadrotor unmanned aerial vehicle (UAV). With RL, two neural nets are trained. One neural net is used as a stochastic controller which gives the distribution of control inputs. The other maps the UAV state to a scalar which estimates the reward of the controller. A proximal policy optimization (PPO) method, which is an actor-critic policy gradient approach, is used to train the neural nets. Simulation results show that the trained controller achieves a comparable level of performance to a manually-tuned PID controller, despite not depending on any model information. The paper considers different choices of reward function and their influence on controller performance.

scholarly journals Reward-Free Reinforcement Learning Algorithm Using Prediction Network

2020 ◽  
Author(s):  
Zhen Yu ◽  
Yimin Feng ◽  
Lijun Liu
Keyword(s):  
Reinforcement Learning ◽  
Learning Algorithm ◽  
Value Functions ◽  
Learning Method ◽  
Reward Function ◽  
Network Training ◽  
Learning Tasks ◽  
Reward Value ◽  
Policy Gradient ◽  
Reward Functions

In general reinforcement learning tasks, the formulation of reward functions is a very important step in reinforcement learning. The reward function is not easy to formulate in a large number of systems. The network training effect is sensitive to the reward function, and different reward value functions will get different results. For a class of systems that meet specific conditions, the traditional reinforcement learning method is improved. A state quantity function is designed to replace the reward function, which is more efficient than the traditional reward function. At the same time, the predictive network link is designed so that the network can learn the value of the general state by using the special state. The overall structure of the network will be improved based on the Deep Deterministic Policy Gradient (DDPG) algorithm. Finally, the algorithm was successfully applied in the environment of FrozenLake, and achieved good performance. The experiment proves the effectiveness of the algorithm and realizes rewardless reinforcement learning in a class of systems.

Sign in / Sign up

Export Citation Format

Share Document