scholarly journals Deep Reinforcement Learning with Uncertain Data for Real-Time Stormwater System Control and Flood Mitigation

Water ◽  
2020 ◽  
Vol 12 (11) ◽  
pp. 3222
Author(s):  
Sami M. Saliba ◽  
Benjamin D. Bowes ◽  
Stephen Adams ◽  
Peter A. Beling ◽  
Jonathan L. Goodall
Keyword(s):  
Reinforcement Learning ◽  
Real Time ◽  
Real World ◽  
Input Data ◽  
Flood Control ◽  
Sequential Optimization ◽  
System Control ◽  
Passive System ◽  
General Technique ◽  
Noisy Input

Flooding in many areas is becoming more prevalent due to factors such as urbanization and climate change, requiring modernization of stormwater infrastructure. Retrofitting standard passive systems with controllable valves/pumps is promising, but requires real-time control (RTC). One method of automating RTC is reinforcement learning (RL), a general technique for sequential optimization and control in uncertain environments. The notion is that an RL algorithm can use inputs of real-time flood data and rainfall forecasts to learn a policy for controlling the stormwater infrastructure to minimize measures of flooding. In real-world conditions, rainfall forecasts and other state information are subject to noise and uncertainty. To account for these characteristics of the problem data, we implemented Deep Deterministic Policy Gradient (DDPG), an RL algorithm that is distinguished by its capability to handle noise in the input data. DDPG implementations were trained and tested against a passive flood control policy. Three primary cases were studied: (i) perfect data, (ii) imperfect rainfall forecasts, and (iii) imperfect water level and forecast data. Rainfall episodes (100) that caused flooding in the passive system were selected from 10 years of observations in Norfolk, Virginia, USA; 85 randomly selected episodes were used for training and the remaining 15 unseen episodes served as test cases. Compared to the passive system, all RL implementations reduced flooding volume by 70.5% on average, and performed within a range of 5%. This suggests that DDPG is robust to noisy input data, which is essential knowledge to advance the real-world applicability of RL for stormwater RTC.

scholarly journals Mitigation of Flooding in Stormwater Systems Utilizing Imperfect Forecasting and Sensor Data with Deep Deterministic Policy Gradient Reinforcement Learning

2020 ◽  
Author(s):  
Sami Saliba ◽  
Benjamin Bowes ◽  
Stephen Adams ◽  
Peter Beling ◽  
Jonathan Goodall
Keyword(s):  
Reinforcement Learning ◽  
Real Time ◽  
Real World ◽  
Input Data ◽  
Flood Control ◽  
Sensor Data ◽  
Sequential Optimization ◽  
Passive System ◽  
Noisy Input ◽  
Policy Gradient

Climate change and development have increased urban flooding, requiring modernization of stormwater infrastructure. Retrofitting standard passive systems with controllable valves/pumps is promising, but requires real-time control (RTC). One method of automating RTC is reinforcement learning (RL), a general technique for sequential optimization and control in uncertain environments. The notion is that an RL algorithm can use inputs of real-time flood data and rainfall forecasts to learn a policy for controlling the stormwater infrastructure to minimize measures of flooding. In real-world conditions, rainfall forecasts and other state information, are subject to noise and uncertainty. To account for these characteristics of the problem data, we implemented Deep Deterministic Policy Gradient (DDPG), an RL algorithm that is distinguished by its capability to handle noise in the input data. DDPG implementations were trained and tested against a passive flood control policy. Three primary cases were studied: (i) perfect data, (ii) imperfect rainfall forecasts, and (iii) imperfect water level and forecast data. Rainfall episodes (100) that caused flooding in the passive system were selected from 10 years of observations in Norfolk, Virginia, USA; 85 randomly selected episodes were used for training and the remaining 15 unseen episodes served as test cases. Compared to the passive system, all RL implementations reduced flooding volume by 70.5% on average, and performed within a range of 5%. This suggests that DDPG is robust to noisy input data, which is essential knowledge to advance the real-world applicability of RL for stormwater RTC.

scholarly journals iADA*-RL: Anytime Graph-Based Path Planning with Deep Reinforcement Learning for an Autonomous UAV

2021 ◽  
Vol 11 (9) ◽  
pp. 3948
Author(s):  
Aye Aye Maw ◽  
Maxim Tyan ◽  
Tuan Anh Nguyen ◽  
Jae-Woo Lee
Keyword(s):  
Reinforcement Learning ◽  
Path Planning ◽  
Real Time ◽  
Collision Avoidance ◽  
Real World ◽  
Mission Planning ◽  
Planning System ◽  
Local Planning ◽  
Planning Algorithm ◽  
Path Planning Algorithm

Path planning algorithms are of paramount importance in guidance and collision systems to provide trustworthiness and safety for operations of autonomous unmanned aerial vehicles (UAV). Previous works showed different approaches mostly focusing on shortest path discovery without a sufficient consideration on local planning and collision avoidance. In this paper, we propose a hybrid path planning algorithm that uses an anytime graph-based path planning algorithm for global planning and deep reinforcement learning for local planning which applied for a real-time mission planning system of an autonomous UAV. In particular, we aim to achieve a highly autonomous UAV mission planning system that is adaptive to real-world environments consisting of both static and moving obstacles for collision avoidance capabilities. To achieve adaptive behavior for real-world problems, a simulator is required that can imitate real environments for learning. For this reason, the simulator must be sufficiently flexible to allow the UAV to learn about the environment and to adapt to real-world conditions. In our scheme, the UAV first learns about the environment via a simulator, and only then is it applied to the real-world. The proposed system is divided into two main parts: optimal flight path generation and collision avoidance. A hybrid path planning approach is developed by combining a graph-based path planning algorithm with a learning-based algorithm for local planning to allow the UAV to avoid a collision in real time. The global path planning problem is solved in the first stage using a novel anytime incremental search algorithm called improved Anytime Dynamic A* (iADA*). A reinforcement learning method is used to carry out local planning between waypoints, to avoid any obstacles within the environment. The developed hybrid path planning system was investigated and validated in an AirSim environment. A number of different simulations and experiments were performed using AirSim platform in order to demonstrate the effectiveness of the proposed system for an autonomous UAV. This study helps expand the existing research area in designing efficient and safe path planning algorithms for UAVs.

scholarly journals Real-time reservoir flood control operation enhanced by data assimilation

2020 ◽  
Author(s):  
Jingwen Zhang ◽  
Ximing Cai ◽  
Xiaohui Lei ◽  
Pan Liu ◽  
Hao Wang
Keyword(s):  
Data Assimilation ◽  
Real Time ◽  
Real World ◽  
Flood Risk ◽  
Optimization Model ◽  
Reservoir Operation ◽  
Flood Control ◽  
Interactive Method ◽  
Operation Conditions ◽  
Time Period

Abstract. Real world reservoir operations are usually not fully automatic based on computer models; instead, reservoir operators conduct the operations based on their experiences, professional justification, as well as modeling support for some cases due to unavoidable gap between computer modeling and real world reservoir operation conditions. In this paper, we propose a human-machine interactive method, namely Real-time Optimization Model Enhanced by Data Assimilation (ROMEDA) for reservoirs which have complex storage and stage relations (e.g. long and narrow reservoirs). The system is composed of 1) an optimization model to search for optimal releases, 2) reservoir operators’ choices based on their experiences, knowledge, and behaviors, and 3) a reservoir storage-stage simulation and data assimilation schedule to update the storage based on real-time reservoir stage observations. For every time period and based on the updated storage, ROMEDA provides optimal releases as recommendations, actual releases made by operators, as well as a warning of flood risk when the storage exceeds a threshold level. ROMEDA does not assume that operators strictly accept the recommendations, and storage will be updated based on actual release at each time period. Via a case study on-channel reservoir, it is found that for both small and large flood events, ROMEDA, which integrates the advantages of both machine and human, shows better performance on flood risk mitigation and water use (hydropower) benefit than the case with historical operation records (HOR) or optimization with single/multi-objective. ROMEDA is one of the first attempts of a human-machine interactive method for online use of an optimization model for real-time reservoir operation based on integrated modeling, observation, and operators’ choice.

scholarly journals Flood mitigation in coastal urban catchments using real-time stormwater infrastructure control and reinforcement learning

2020 ◽  
Author(s):  
Benjamin D. Bowes ◽  
Arash Tavakoli ◽  
Cheng Wang ◽  
Arsalan Heydarian ◽  
Madhur Behl ◽  
...  
Keyword(s):  
Reinforcement Learning ◽  
Real Time ◽  
Control Strategies ◽  
Computationally Efficient ◽  
Passive System ◽  
Real Time Control ◽  
Control Policies ◽  
Time Control ◽  
System A ◽  
Urban Catchments

Abstract Flooding in coastal cities is increasing due to climate change and sea-level rise, stressing the traditional stormwater systems these communities rely on. Automated real-time control (RTC) of these systems can improve performance, and creating control policies for smart stormwater systems is an active area of study. This research explores reinforcement learning (RL) to create control policies to mitigate flood risk. RL is trained using a model of hypothetical urban catchments with a tidal boundary and two retention ponds with controllable valves. RL's performance is compared to the passive system, a model predictive control (MPC) strategy, and a rule-based control strategy (RBC). RL learns to proactively manage pond levels using current and forecast conditions and reduced flooding by 32% over the passive system. Compared to the MPC approach using a physics-based model and genetic algorithm, RL achieved nearly the same flood reduction, just 3% less than MPC, with a significant 88× speedup in runtime. Compared to RBC, RL was able to quickly learn similar control strategies and reduced flooding by an additional 19%. This research demonstrates that RL can effectively control a simple system and offers a computationally efficient method that could scale to RTC of more complex stormwater systems.

A Simple and Practical Method for Incorporating Augmented Reality into the Classroom and Laboratory

2018 ◽  
Author(s):  
Kyle Plunkett
Keyword(s):  
Organic Chemistry ◽  
Augmented Reality ◽  
Real Time ◽  
Real World ◽  
Smart Phone ◽  
Practical Method ◽  
Virtual Expert ◽  
Video Projection

This manuscript provides two demonstrations of how Augmented Reality (AR), which is the projection of virtual information onto a real-world object, can be applied in the classroom and in the laboratory. Using only a smart phone and the free HP Reveal app, content rich AR notecards were prepared. The physical notecards are based on Organic Chemistry I reactions and show only a reagent and substrate. Upon interacting with the HP Reveal app, an AR video projection shows the product of the reaction as well as a real-time, hand-drawn curved-arrow mechanism of how the product is formed. Thirty AR notecards based on common Organic Chemistry I reactions and mechanisms are provided in the Supporting Information and are available for widespread use. In addition, the HP Reveal app was used to create AR video projections onto laboratory instrumentation so that a virtual expert can guide the user during the equipment setup and operation.

scholarly journals A Frequency Pattern Mining Model Based on Deep Neural Network for Real-Time Classification of Heart Conditions

Healthcare ◽  
2020 ◽  
Vol 8 (3) ◽  
pp. 234 ◽  
Author(s):  
Hyun Yoo ◽  
Soyoung Han ◽  
Kyungyong Chung
Keyword(s):  
Neural Network ◽  
Big Data ◽  
Fourier Transform ◽  
Fast Fourier Transform ◽  
Real Time ◽  
Normal Control ◽  
Input Data ◽  
Deep Neural Network ◽  
Pattern Mining ◽  
F Measure

Recently, a massive amount of big data of bioinformation is collected by sensor-based IoT devices. The collected data are also classified into different types of health big data in various techniques. A personalized analysis technique is a basis for judging the risk factors of personal cardiovascular disorders in real-time. The objective of this paper is to provide the model for the personalized heart condition classification in combination with the fast and effective preprocessing technique and deep neural network in order to process the real-time accumulated biosensor input data. The model can be useful to learn input data and develop an approximation function, and it can help users recognize risk situations. For the analysis of the pulse frequency, a fast Fourier transform is applied in preprocessing work. With the use of the frequency-by-frequency ratio data of the extracted power spectrum, data reduction is performed. To analyze the meanings of preprocessed data, a neural network algorithm is applied. In particular, a deep neural network is used to analyze and evaluate linear data. A deep neural network can make multiple layers and can establish an operation model of nodes with the use of gradient descent. The completed model was trained by classifying the ECG signals collected in advance into normal, control, and noise groups. Thereafter, the ECG signal input in real time through the trained deep neural network system was classified into normal, control, and noise. To evaluate the performance of the proposed model, this study utilized a ratio of data operation cost reduction and F-measure. As a result, with the use of fast Fourier transform and cumulative frequency percentage, the size of ECG reduced to 1:32. According to the analysis on the F-measure of the deep neural network, the model had 83.83% accuracy. Given the results, the modified deep neural network technique can reduce the size of big data in terms of computing work, and it is an effective system to reduce operation time.

scholarly journals A Novel Rail-Network Hardware Simulator for Embedded System Programming

Electronics ◽  
2020 ◽  
Vol 10 (1) ◽  
pp. 13
Author(s):  
Balaji M ◽  
Chandrasekaran M ◽  
Vaithiyanathan Dhandapani
Keyword(s):  
Embedded Systems ◽  
Embedded System ◽  
Real Time ◽  
Learning Outcomes ◽  
Real World ◽  
Time Constraints ◽  
Network Simulator ◽  
Practical Applications ◽  
Rail Network ◽  
Knowledge Enhancement

A Novel Rail-Network Hardware with simulation facilities is presented in this paper. The hardware is designed to facilitate the learning of application-oriented, logical, real-time programming in an embedded system environment. The platform enables the creation of multiple unique programming scenarios with variability in complexity without any hardware changes. Prior experimental hardware comes with static programming facilities that focus the students’ learning on hardware features and programming basics, leaving them ill-equipped to take up practical applications with more real-time constraints. This hardware complements and completes their learning to help them program real-world embedded systems. The hardware uses LEDs to simulate the movement of trains in a network. The network has train stations, intersections and parking slots where the train movements can be controlled by using a 16-bit Renesas RL78/G13 microcontroller. Additionally, simulating facilities are provided to enable the students to navigate the trains by manual controls using switches and indicators. This helps them get an easy understanding of train navigation functions before taking up programming. The students start with simple tasks and gradually progress to more complicated ones with real-time constraints, on their own. During training, students’ learning outcomes are evaluated by obtaining their feedback and conducting a test at the end to measure their knowledge acquisition during the training. Students’ Knowledge Enhancement Index is originated to measure the knowledge acquired by the students. It is observed that 87% of students have successfully enhanced their knowledge undergoing training with this rail-network simulator.

Recent Advances in Reinforcement Learning for Traffic Signal Control

2021 ◽  
Vol 22 (2) ◽  
pp. 12-18 ◽  
Author(s):  
Hua Wei ◽  
Guanjie Zheng ◽  
Vikash Gayah ◽  
Zhenhui Li
Keyword(s):  
Reinforcement Learning ◽  
Real World ◽  
Intelligent Transportation Systems ◽  
Transportation Systems ◽  
Traffic Signal ◽  
Signal Control ◽  
Traffic Signal Control ◽  
Control Methods ◽  
Advantages And Disadvantages ◽  
Recent Advances

Traffic signal control is an important and challenging real-world problem that has recently received a large amount of interest from both transportation and computer science communities. In this survey, we focus on investigating the recent advances in using reinforcement learning (RL) techniques to solve the traffic signal control problem. We classify the known approaches based on the RL techniques they use and provide a review of existing models with analysis on their advantages and disadvantages. Moreover, we give an overview of the simulation environments and experimental settings that have been developed to evaluate the traffic signal control methods. Finally, we explore future directions in the area of RLbased traffic signal control methods. We hope this survey could provide insights to researchers dealing with real-world applications in intelligent transportation systems

Sign in / Sign up

Export Citation Format

Share Document