scholarly journals Quantifying the impact of non-stationarity in reinforcement learning-based traffic signal control

2021 ◽  
Vol 7 ◽  
pp. e575
Author(s):  
Lucas N. Alegre ◽  
Ana L.C. Bazzan ◽  
Bruno C. da Silva
Keyword(s):  
Reinforcement Learning ◽  
Traffic Control ◽  
Traffic Signals ◽  
Traffic Signal ◽  
Traffic Signal Control ◽  
Traffic Patterns ◽  
Learning Agent ◽  
Traffic Optimization ◽  
The Impact ◽  
Different Sources

In reinforcement learning (RL), dealing with non-stationarity is a challenging issue. However, some domains such as traffic optimization are inherently non-stationary. Causes for and effects of this are manifold. In particular, when dealing with traffic signal controls, addressing non-stationarity is key since traffic conditions change over time and as a function of traffic control decisions taken in other parts of a network. In this paper we analyze the effects that different sources of non-stationarity have in a network of traffic signals, in which each signal is modeled as a learning agent. More precisely, we study both the effects of changing the context in which an agent learns (e.g., a change in flow rates experienced by it), as well as the effects of reducing agent observability of the true environment state. Partial observability may cause distinct states (in which distinct actions are optimal) to be seen as the same by the traffic signal agents. This, in turn, may lead to sub-optimal performance. We show that the lack of suitable sensors to provide a representative observation of the real state seems to affect the performance more drastically than the changes to the underlying traffic patterns.

scholarly journals Adaptive Traffic Signal Control Model on Intersections Based on Deep Reinforcement Learning

2020 ◽  
Vol 2020 ◽  
pp. 1-14
Author(s):  
Duowei Li ◽  
Jianping Wu ◽  
Ming Xu ◽  
Ziheng Wang ◽  
Kezhen Hu
Keyword(s):  
Reinforcement Learning ◽  
Waiting Time ◽  
Traffic Signals ◽  
Control Model ◽  
Traffic Signal ◽  
Signal Control ◽  
Traffic Signal Control ◽  
Average Waiting Time ◽  
Adaptive Traffic Signal Control ◽  
Proposed Model

Controlling traffic signals to alleviate increasing traffic pressure is a concept that has received public attention for a long time. However, existing systems and methodologies for controlling traffic signals are insufficient for addressing the problem. To this end, we build a truly adaptive traffic signal control model in a traffic microsimulator, i.e., “Simulation of Urban Mobility” (SUMO), using the technology of modern deep reinforcement learning. The model is proposed based on a deep Q-network algorithm that precisely represents the elements associated with the problem: agents, environments, and actions. The real-time state of traffic, including the number of vehicles and the average speed, at one or more intersections is used as an input to the model. To reduce the average waiting time, the agents provide an optimal traffic signal phase and duration that should be implemented in both single-intersection cases and multi-intersection cases. The co-operation between agents enables the model to achieve an improvement in overall performance in a large road network. By testing with data sets pertaining to three different traffic conditions, we prove that the proposed model is better than other methods (e.g., Q-learning method, longest queue first method, and Webster fixed timing control method) for all cases. The proposed model reduces both the average waiting time and travel time, and it becomes more advantageous as the traffic environment becomes more complex.

Developing Signal Warrants for Restricted Crossing U-Turn Intersections

2021 ◽  
pp. 036119812110469
Author(s):  
Justice Appiah
Keyword(s):  
Traffic Control ◽  
Simulation Analysis ◽  
Traffic Signals ◽  
Traffic Volume ◽  
Traffic Signal ◽  
Signal Control ◽  
Traffic Signal Control ◽  
Left Turn ◽  
Minor Road ◽  
Intersection Configuration

The restricted crossing U-turn (RCUT) intersection is a form of innovative intersection design that reroutes left-turn and through traffic from the minor road to U-turn crossovers on the major road. When implemented correctly, an RCUT intersection can provide significant safety and operational benefits over the conventional intersection configuration. The RCUT may be controlled by traffic signals, STOP control, merges and diverges, or a combination of these. There is currently no concrete guidance in relation to when the use of traffic signal control is warranted at an RCUT intersection. This study investigated traffic volume conditions that may warrant consideration of traffic signal control at an RCUT intersection. Simulation experiments including two geometric configurations and three traffic control schemes were designed and run in VISSIM to evaluate the effects of traffic conditions on intersection delay and queue lengths. Traffic was varied by changing the composition, approach volumes, and origin–destination flow patterns to reflect different conditions that may occur at the intersection on any given day. For the range of conditions studied, the results of the simulation analysis suggested that the RCUT intersection may operate better with traffic signals (at all junctions) when the minor roadway traffic volume is more than 450 vehicles per hour (vph) and the major roadway has two through lanes. The corresponding minor roadway volume threshold increases to 575 vph when the major roadway has four through lanes.

Integrated Traffic Signal Control Along Corridors With Multiple Intersections Near Highway-Rail Grade Crossings

2013 ◽  
Author(s):  
Yifeng Chen ◽  
Laurence R. Rilett
Keyword(s):  
Traffic Control ◽  
Traffic Signals ◽  
Traffic Signal ◽  
Railway Track ◽  
Test Bed ◽  
Control Logic ◽  
Grade Crossings ◽  
The Impact ◽  
Multiple Intersections ◽  
Near Highway

Traffic signal optimization for traffic signals located near highway-rail grade crossings (HRGC) can be difficult because of the complex nature of the interactions between the two systems and the necessity of considering multiple objectives, such as safety and operational efficiency. The problems are magnified when considering traffic control for corridors that have multiple intersections located near HRGCs. This paper develops a methodology for optimizing traffic signals along a highway-railway corridor while considering the dual objectives of maximizing safety and efficiency. The Highway 6 (Cornhusker Hwy) corridor in Lincoln, Nebraska was used as a test bed. The corridor was modeled in VISSIM, and was used to emulate the traffic control along Highway 6, including the preemption logic. The traffic control logic was modeled using the Vehicle Actuated Programming (VAP) in the VISSIM simulation model. In addition, the logic allows multiple train events on the railway track that runs parallel to Highway 6 to be modeled. The model was calibrated to local traffic conditions using empirical field data. The impact of train frequency, length, direction, speed, etc., on the performance of the network and pedestrian safety will be evaluated.

scholarly journals DQN-BASED TRAFFIC SIGNAL CONTROL SYSTEMS

2021 ◽  
Vol 6 (7(57)) ◽  
pp. 16-18
Author(s):  
Ivan Vladimirovich Kondratov
Keyword(s):  
Reinforcement Learning ◽  
Control Systems ◽  
Real Time ◽  
Traffic Control ◽  
Optimization Methods ◽  
Traffic Signal ◽  
Modern World ◽  
Signal Control ◽  
Traffic Signal Control ◽  
Traffic Signal Control Systems

Real-time adaptive traffic control is an important problem in modern world. Historically, various optimization methods have been used to build adaptive traffic signal control systems. Recently, reinforcement learning has been advanced, and various papers showed efficiency of Deep-Q-Learning (DQN) in solving traffic control problems and providing real-time adaptive control for traffic, decreasing traffic pressure and lowering average travel time for drivers. In this paper we consider the problem of traffic signal control, present the basics of reinforcement learning and review the latest results in this area.

Control and Coordination of Self-Adaptive Traffic Signal Using Deep Reinforcement Learning

2021 ◽  
Vol 9 (1) ◽  
pp. 373-379
Author(s):  
Pallavi Mandhare, Dr. Jyoti Yadav, Prof. Vilas Kharat, Prof. C.Y. Patil
Keyword(s):  
Reinforcement Learning ◽  
Traffic Flow ◽  
Traffic Signals ◽  
Traffic Signal ◽  
Traffic Signal Control ◽  
Sustainable Mobility ◽  
Traffic Demand ◽  
Traffic Systems ◽  
Traffic Signal Timing ◽  
Traffic Signal Control Systems

The most observable obstacle to sustainable mobility is traffic congestions. These congestions cannot effectively be fixed by traditional control of traffic signals. Safe and smooth movement of traffic is ensured by a self-controlled traffic signal. As such, to coordinate the traffic flow it is necessary to implement dynamic traffic signal subsequences. Primarily, Traffic Signal Controllers (TSC) provides sophisticated control and coordination of vehicles. The control and coordination of traffic signal control systems can be effectively achieved by implementing the Deep Reinforcement Learning (DRL) approaches. The decision-making capabilities at intersections are improved by having variations of traffic signal timing using an adaptive TSC. Alternatively, the actual traffic demand is nothing but managing the traffic systems. It analyses the incoming number and type of vehicles and gives a real-time response at intersection geometrics and controls the traffic signals accordingly. The proposed DRL algorithm observes traffic data and operates optimum management plans for the regulation of the traffic flow. Furthermore, an existing traffic simulator is used to help provide a realistic environment to support the proposed algorithm.  

Recent Advances in Reinforcement Learning for Traffic Signal Control

2021 ◽  
Vol 22 (2) ◽  
pp. 12-18 ◽  
Author(s):  
Hua Wei ◽  
Guanjie Zheng ◽  
Vikash Gayah ◽  
Zhenhui Li
Keyword(s):  
Reinforcement Learning ◽  
Real World ◽  
Intelligent Transportation Systems ◽  
Transportation Systems ◽  
Traffic Signal ◽  
Signal Control ◽  
Traffic Signal Control ◽  
Control Methods ◽  
Advantages And Disadvantages ◽  
Recent Advances

Traffic signal control is an important and challenging real-world problem that has recently received a large amount of interest from both transportation and computer science communities. In this survey, we focus on investigating the recent advances in using reinforcement learning (RL) techniques to solve the traffic signal control problem. We classify the known approaches based on the RL techniques they use and provide a review of existing models with analysis on their advantages and disadvantages. Moreover, we give an overview of the simulation environments and experimental settings that have been developed to evaluate the traffic signal control methods. Finally, we explore future directions in the area of RLbased traffic signal control methods. We hope this survey could provide insights to researchers dealing with real-world applications in intelligent transportation systems

scholarly journals An Evaluation Methodology for Interactive Reinforcement Learning with Simulated Users

Biomimetics ◽  
2021 ◽  
Vol 6 (1) ◽  
pp. 13
Author(s):  
Adam Bignold ◽  
Francisco Cruz ◽  
Richard Dazeley ◽  
Peter Vamplew ◽  
Cameron Foale
Keyword(s):  
Reinforcement Learning ◽  
Information Source ◽  
Human Interaction ◽  
Evaluation Methodology ◽  
External Information ◽  
Preliminary Evaluation ◽  
Learning Agents ◽  
Learning Agent ◽  
Knowledge Bias ◽  
The Impact

Interactive reinforcement learning methods utilise an external information source to evaluate decisions and accelerate learning. Previous work has shown that human advice could significantly improve learning agents’ performance. When evaluating reinforcement learning algorithms, it is common to repeat experiments as parameters are altered or to gain a sufficient sample size. In this regard, to require human interaction every time an experiment is restarted is undesirable, particularly when the expense in doing so can be considerable. Additionally, reusing the same people for the experiment introduces bias, as they will learn the behaviour of the agent and the dynamics of the environment. This paper presents a methodology for evaluating interactive reinforcement learning agents by employing simulated users. Simulated users allow human knowledge, bias, and interaction to be simulated. The use of simulated users allows the development and testing of reinforcement learning agents, and can provide indicative results of agent performance under defined human constraints. While simulated users are no replacement for actual humans, they do offer an affordable and fast alternative for evaluative assisted agents. We introduce a method for performing a preliminary evaluation utilising simulated users to show how performance changes depending on the type of user assisting the agent. Moreover, we describe how human interaction may be simulated, and present an experiment illustrating the applicability of simulating users in evaluating agent performance when assisted by different types of trainers. Experimental results show that the use of this methodology allows for greater insight into the performance of interactive reinforcement learning agents when advised by different users. The use of simulated users with varying characteristics allows for evaluation of the impact of those characteristics on the behaviour of the learning agent.

Sign in / Sign up

Export Citation Format

Share Document