scholarly journals A Reward Optimization Method Based on Action Subrewards in Hierarchical Reinforcement Learning

2014 ◽  
Vol 2014 ◽  
pp. 1-6
Author(s):  
Yuchen Fu ◽  
Quan Liu ◽  
Xionghong Ling ◽  
Zhiming Cui
Keyword(s):  
Reinforcement Learning ◽  
Learning Algorithm ◽  
Optimization Method ◽  
Curse Of Dimensionality ◽  
Convergence Speed ◽  
Learning Method ◽  
Trial And Error ◽  
State Spaces ◽  
Reward Function ◽  
Hierarchical Reinforcement Learning

Reinforcement learning (RL) is one kind of interactive learning methods. Its main characteristics are “trial and error” and “related reward.” A hierarchical reinforcement learning method based on action subrewards is proposed to solve the problem of “curse of dimensionality,” which means that the states space will grow exponentially in the number of features and low convergence speed. The method can reduce state spaces greatly and choose actions with favorable purpose and efficiency so as to optimize reward function and enhance convergence speed. Apply it to the online learning in Tetris game, and the experiment result shows that the convergence speed of this algorithm can be enhanced evidently based on the new method which combines hierarchical reinforcement learning algorithm and action subrewards. The “curse of dimensionality” problem is also solved to a certain extent with hierarchical method. All the performance with different parameters is compared and analyzed as well.

scholarly journals Reward-Free Reinforcement Learning Algorithm Using Prediction Network

2020 ◽  
Author(s):  
Zhen Yu ◽  
Yimin Feng ◽  
Lijun Liu
Keyword(s):  
Reinforcement Learning ◽  
Learning Algorithm ◽  
Value Functions ◽  
Learning Method ◽  
Reward Function ◽  
Network Training ◽  
Learning Tasks ◽  
Reward Value ◽  
Policy Gradient ◽  
Reward Functions

In general reinforcement learning tasks, the formulation of reward functions is a very important step in reinforcement learning. The reward function is not easy to formulate in a large number of systems. The network training effect is sensitive to the reward function, and different reward value functions will get different results. For a class of systems that meet specific conditions, the traditional reinforcement learning method is improved. A state quantity function is designed to replace the reward function, which is more efficient than the traditional reward function. At the same time, the predictive network link is designed so that the network can learn the value of the general state by using the special state. The overall structure of the network will be improved based on the Deep Deterministic Policy Gradient (DDPG) algorithm. Finally, the algorithm was successfully applied in the environment of FrozenLake, and achieved good performance. The experiment proves the effectiveness of the algorithm and realizes rewardless reinforcement learning in a class of systems.

scholarly journals Hierarchical Reinforcement Learning Based Self-balancing Algorithm for Two-wheeled Robots

2016 ◽  
Vol 10 (1) ◽  
pp. 69-79 ◽  
Author(s):  
Juan Yan ◽  
Huibin Yang
Keyword(s):  
Reinforcement Learning ◽  
Learning Algorithm ◽  
Weight Vector ◽  
The Self ◽  
Simulation Experiments ◽  
Wheeled Robots ◽  
Reward Function ◽  
Hierarchical Reinforcement Learning ◽  
Balancing Control ◽  
Reinforcement Learning Algorithm

Self-balancing control is the basis for applications of two-wheeled robots. In order to improve the self-balancing of two-wheeled robots, we propose a hierarchical reinforcement learning algorithm for controlling the balance of two-wheeled robots. After describing the subgoals of hierarchical reinforcement learning, we extract features for subgoals, define a feature value vector and its corresponding weight vector, and propose a reward function with additional subgoal reward function. Finally, we give a hierarchical reinforcement learning algorithm for finding the optimal strategy. Simulation experiments show that, the proposed algorithm is more effectiveness than traditional reinforcement learning algorithm in convergent speed. So in our system, the robots can get self-balanced very quickly.

Safety-aware Adversarial Inverse Reinforcement Learning (S-AIRL) for Highway Autonomous Driving

2022 ◽  
pp. 1-14
Author(s):  
Fangjian Li ◽  
John R Wagner ◽  
Yue Wang
Keyword(s):  
Reinforcement Learning ◽  
Learning Algorithm ◽  
Risky Behaviors ◽  
Autonomous Driving ◽  
Inverse Reinforcement Learning ◽  
Safety Issues ◽  
Reward Function ◽  
Sampling Process ◽  
Safety Awareness ◽  
Driving Scenario

Abstract Inverse reinforcement learning (IRL) has been successfully applied in many robotics and autonomous driving studies without the need for hand-tuning a reward function. However, it suffers from safety issues. Compared to the reinforcement learning (RL) algorithms, IRL is even more vulnerable to unsafe situations as it can only infer the importance of safety based on expert demonstrations. In this paper, we propose a safety-aware adversarial inverse reinforcement learning algorithm (S-AIRL). First, the control barrier function (CBF) is used to guide the training of a safety critic, which leverages the knowledge of system dynamics in the sampling process without training an additional guiding policy. The trained safety critic is then integrated into the discriminator to help discern the generated data and expert demonstrations from the standpoint of safety. Finally, to further improve the safety awareness, a regulator is introduced in the loss function of the discriminator training to prevent the recovered reward function from assigning high rewards to the risky behaviors. We tested our S-AIRL in the highway autonomous driving scenario. Comparing to the original AIRL algorithm, with the same level of imitation learning (IL) performance, the proposed S-AIRL can reduce the collision rate by 32.6%.

Hierarchical Reinforcement Learning

2011 ◽  
pp. 825-830
Author(s):  
Carlos Diuk ◽  
Michael Littman
Keyword(s):  
Reinforcement Learning ◽  
Learning Problems ◽  
Underlying Structure ◽  
Sequential Decision ◽  
State Spaces ◽  
Hierarchical Reinforcement Learning ◽  
Markov Decision ◽  
Finite Set ◽  
State Abstraction ◽  
Main Ideas

Reinforcement learning (RL) deals with the problem of an agent that has to learn how to behave to maximize its utility by its interactions with an environment (Sutton & Barto, 1998; Kaelbling, Littman & Moore, 1996). Reinforcement learning problems are usually formalized as Markov Decision Processes (MDP), which consist of a finite set of states and a finite number of possible actions that the agent can perform. At any given point in time, the agent is in a certain state and picks an action. It can then observe the new state this action leads to, and receives a reward signal. The goal of the agent is to maximize its long-term reward. In this standard formalization, no particular structure or relationship between states is assumed. However, learning in environments with extremely large state spaces is infeasible without some form of generalization. Exploiting the underlying structure of a problem can effect generalization and has long been recognized as an important aspect in representing sequential decision tasks (Boutilier et al., 1999). Hierarchical Reinforcement Learning is the subfield of RL that deals with the discovery and/or exploitation of this underlying structure. Two main ideas come into play in hierarchical RL. The first one is to break a task into a hierarchy of smaller subtasks, each of which can be learned faster and easier than the whole problem. Subtasks can also be performed multiple times in the course of achieving the larger task, reusing accumulated knowledge and skills. The second idea is to use state abstraction within subtasks: not every task needs to be concerned with every aspect of the state space, so some states can actually be abstracted away and treated as the same for the purpose of the given subtask.

scholarly journals An Application of Reinforced Learning-Based Dynamic Pricing for Improvement of Ridesharing Platform Service in Seoul

Electronics ◽  
2020 ◽  
Vol 9 (11) ◽  
pp. 1818
Author(s):  
Jaein Song ◽  
Yun Ji Cho ◽  
Min Hee Kang ◽  
Kee Yeon Hwang
Keyword(s):  
Reinforcement Learning ◽  
Waiting Time ◽  
Dynamic Pricing ◽  
Learning Algorithm ◽  
Residential Areas ◽  
Private Companies ◽  
Reward Function ◽  
Time Period ◽  
Reinforced Learning ◽  
Centrality Analysis

As ridesharing services (including taxi) are often run by private companies, profitability is the top priority in operation. This leads to an increase in the driver’s refusal to take passengers to areas with low demand where they will have difficulties finding subsequent passengers, causing problems such as an extended waiting time when hailing a vehicle for passengers bound for these regions. The study used Seoul’s taxi data to find appropriate surge rates of ridesharing services between 10:00 p.m. and 4:00 a.m. by region using a reinforcement learning algorithm to resolve this problem during the worst time period. In reinforcement learning, the outcome of centrality analysis was applied as a weight affecting drivers’ destination choice probability. Furthermore, the reward function used in the learning was adjusted according to whether the passenger waiting time value was applied or not. The profit was used for reward value. By using a negative reward for the passenger waiting time, the study was able to identify a more appropriate surge level. Across the region, the surge averaged a value of 1.6. To be more specific, those located on the outskirts of the city and in residential areas showed a higher surge, while central areas had a lower surge. Due to this different surge, a driver’s refusal to take passengers can be lessened and the passenger waiting time can be shortened. The supply of ridesharing services in low-demand regions can be increased by as much as 7.5%, allowing regional equity problems related to ridesharing services in Seoul to be reduced to a greater extent.

A Multi-Step Reinforcement Learning Algorithm

2010 ◽  
Vol 44-47 ◽  
pp. 3611-3615 ◽  
Author(s):  
Zhi Cong Zhang ◽  
Kai Shun Hu ◽  
Hui Yu Huang ◽  
Shuai Li ◽  
Shao Yong Zhao
Keyword(s):  
Reinforcement Learning ◽  
Markov Decision Process ◽  
Decision Process ◽  
Large Scale ◽  
Learning Algorithm ◽  
Machine Learning Method ◽  
Learning Method ◽  
K Value ◽  
Markov Decision ◽  
Action Value

Reinforcement learning (RL) is a state or action value based machine learning method which approximately solves large-scale Markov Decision Process (MDP) or Semi-Markov Decision Process (SMDP). A multi-step RL algorithm called Sarsa(,k) is proposed, which is a compromised variation of Sarsa and Sarsa(). It is equivalent to Sarsa if k is 1 and is equivalent to Sarsa() if k is infinite. Sarsa(,k) adjust its performance by setting k value. Two forms of Sarsa(,k), forward view Sarsa(,k) and backward view Sarsa(,k), are constructed and proved equivalent in off-line updating.

scholarly journals Vision-Based Robot Navigation through Combining Unsupervised Learning and Hierarchical Reinforcement Learning

Sensors ◽  
2019 ◽  
Vol 19 (7) ◽  
pp. 1576 ◽  
Author(s):  
Xiaomao Zhou ◽  
Tao Bai ◽  
Yanbin Gao ◽  
Yuntao Han
Keyword(s):  
Reinforcement Learning ◽  
Unsupervised Learning ◽  
Learning Algorithm ◽  
Spatial Scales ◽  
Learning Performance ◽  
Head Direction ◽  
Topological Map ◽  
Topological Maps ◽  
Hierarchical Reinforcement Learning ◽  
Continuous State

Extensive studies have shown that many animals’ capability of forming spatial representations for self-localization, path planning, and navigation relies on the functionalities of place and head-direction (HD) cells in the hippocampus. Although there are numerous hippocampal modeling approaches, only a few span the wide functionalities ranging from processing raw sensory signals to planning and action generation. This paper presents a vision-based navigation system that involves generating place and HD cells through learning from visual images, building topological maps based on learned cell representations and performing navigation using hierarchical reinforcement learning. First, place and HD cells are trained from sequences of visual stimuli in an unsupervised learning fashion. A modified Slow Feature Analysis (SFA) algorithm is proposed to learn different cell types in an intentional way by restricting their learning to separate phases of the spatial exploration. Then, to extract the encoded metric information from these unsupervised learning representations, a self-organized learning algorithm is adopted to learn over the emerged cell activities and to generate topological maps that reveal the topology of the environment and information about a robot’s head direction, respectively. This enables the robot to perform self-localization and orientation detection based on the generated maps. Finally, goal-directed navigation is performed using reinforcement learning in continuous state spaces which are represented by the population activities of place cells. In particular, considering that the topological map provides a natural hierarchical representation of the environment, hierarchical reinforcement learning (HRL) is used to exploit this hierarchy to accelerate learning. The HRL works on different spatial scales, where a high-level policy learns to select subgoals and a low-level policy learns over primitive actions to specialize on the selected subgoals. Experimental results demonstrate that our system is able to navigate a robot to the desired position effectively, and the HRL shows a much better learning performance than the standard RL in solving our navigation tasks.

Sign in / Sign up

Export Citation Format

Share Document