scholarly journals Analyzing Real Options and Flexibility in Engineering Systems Design using Decision Rules and Deep Reinforcement Learning

2021 ◽  
pp. 1-31
Author(s):  
Cesare Caputo ◽  
Michel-Alexandre Cardin
Keyword(s):  
Reinforcement Learning ◽  
Real Options ◽  
Decision Rules ◽  
Systems Design ◽  
Computational Design ◽  
Risk Tolerance ◽  
Action Space ◽  
Waste To Energy ◽  
Engineering Systems ◽  
Standard Design

Abstract Engineering systems provide essential services to society e.g., power generation, transportation. Their performance, however, is directly affected by their ability to cope with uncertainty, especially given the realities of climate change and pandemics. Standard design methods often fail to recognize uncertainty in early conceptual activities, leading to rigid systems that are vulnerable to change. Real Options and Flexibility in Design are important paradigms to improve a system's ability to adapt and respond to unforeseen conditions. Existing approaches to analyze flexibility, however, do not leverage sufficiently recent developments in machine learning enabling deeper exploration of the computational design space. There is untapped potential for new solutions that are not readily accessible using existing methods. Here, a novel approach to analyze flexibility is proposed based on Deep Reinforcement Learning (DRL). It explores available datasets systematically and considers a wider range of adaptability strategies. The methodology is evaluated on an example waste-to-energy system. Low and high flexibility DRL models are compared against stochastically optimal inflexible and flexible solutions using decision rules. The results show highly dynamic solutions, with action space parametrized via artificial neural network. They show improved expected economic value up to 69% compared to previous solutions. Combining information from action space probability distributions along expert insights and risk tolerance helps make better decisions in real-world design and system operations. Out of sample testing shows that the policies are generalizable, but subject to tradeoffs between flexibility and inherent limitations of the learning process.

A Framework for Designing and Managing Flexibility and Real Options in Engineering Systems Based on Decision Rules

2017 ◽  
Author(s):  
Qihui Xie ◽  
Michel-Alexandre Cardin
Keyword(s):  
Real Options ◽  
Performance Improvement ◽  
Decision Rules ◽  
Decision Makers ◽  
Waste To Energy ◽  
Engineering Systems ◽  
Triggering Mechanism ◽  
Implementation Phase ◽  
Uncertainty Sources ◽  
Standard Design

This paper introduces a framework to design and manage flexibility in engineering systems based on the concept of decision rules. A decision rule can be described as a heuristic triggering mechanism that is used to determine when it is appropriate to exercise flexibility in systems operations. The proposed framework differs from existing real options analysis (ROA) approaches used in a design and management setting by focusing on the practicability in the implementation phase of engineering systems. By incorporating decision rules in the design process, this framework not only helps generate better performing designs, it also provides intuitive guidance for decision makers (DMs) to manage the system in operations. The proposed framework is applied as demonstration to the design and management of an anaerobic digestion (AD) waste-to-energy (WTE) plant. It demonstrates significant lifecycle performance improvement as compared to a standard design analysis. A comparison with existing ROA approaches shows that another advantage of the proposed framework is the ability to analyze systems facing multiple uncertainty sources and relying on multiple flexibility strategies as a way to improve expected lifecycle performance.

scholarly journals Spatiotemporally Constrained Action Space Attacks on Deep Reinforcement Learning Agents

2020 ◽  
Vol 34 (04) ◽  
pp. 4577-4584
Author(s):  
Xian Yeow Lee ◽  
Sambit Ghadai ◽  
Kai Liang Tan ◽  
Chinmay Hegde ◽  
Soumik Sarkar
Keyword(s):  
Reinforcement Learning ◽  
Optimization Problem ◽  
Limited Resource ◽  
Action Space ◽  
Engineering Systems ◽  
Physical Systems ◽  
Learning Agents ◽  
Look Ahead ◽  
Real World Applications ◽  
Temporal Dimensions

Robustness of Deep Reinforcement Learning (DRL) algorithms towards adversarial attacks in real world applications such as those deployed in cyber-physical systems (CPS) are of increasing concern. Numerous studies have investigated the mechanisms of attacks on the RL agent's state space. Nonetheless, attacks on the RL agent's action space (corresponding to actuators in engineering systems) are equally perverse, but such attacks are relatively less studied in the ML literature. In this work, we first frame the problem as an optimization problem of minimizing the cumulative reward of an RL agent with decoupled constraints as the budget of attack. We propose the white-box Myopic Action Space (MAS) attack algorithm that distributes the attacks across the action space dimensions. Next, we reformulate the optimization problem above with the same objective function, but with a temporally coupled constraint on the attack budget to take into account the approximated dynamics of the agent. This leads to the white-box Look-ahead Action Space (LAS) attack algorithm that distributes the attacks across the action and temporal dimensions. Our results showed that using the same amount of resources, the LAS attack deteriorates the agent's performance significantly more than the MAS attack. This reveals the possibility that with limited resource, an adversary can utilize the agent's dynamics to malevolently craft attacks that causes the agent to fail. Additionally, we leverage these attack strategies as a possible tool to gain insights on the potential vulnerabilities of DRL agents.

scholarly journals THE ROLE OF MACHINE LEARNING FOR FLEXIBILITY AND REAL OPTIONS ANALYSIS IN ENGINEERING SYSTEMS DESIGN

2021 ◽  
Vol 1 ◽  
pp. 3121-3130
Author(s):  
Cesare Caputo ◽  
Michel-Alexandre Cardin
Keyword(s):  
Machine Learning ◽  
Real Options ◽  
Systems Design ◽  
Engineering Systems ◽  
Real Options Analysis ◽  
Domain Experts ◽  
Design Variables ◽  
Parameters Uncertainty ◽  
Infrastructure System ◽  
Computational Intractability

AbstractFlexibility analysis helps improve the expected value of engineering systems under uncertainty (economic and/or social). Designing for flexibility, however, can be challenging as a large number of design variables, parameters, uncertainty drivers, decision making possibilities and metrics must be considered. Many available techniques either rely on assumptions that are not suitable for an engineering setting, or may be limited due to computational intractability. This paper makes the case for an increased integration of Machine Learning into flexibility and real options analysis in engineering systems design to complement existing design methods. Several synergies are found and discussed critically between the fields in order to explore better solutions that may exist by analyzing the data, which may not be intuitive to domain experts. Reinforcement Learning is particularly promising as a result of the theoretical common grounds with latest methodological developments e.g. decision-rule based real options analysis. Relevance to the field of computational creativity is examined, and potential avenues for further research are identified. The proposed concepts are illustrated through the design of an example infrastructure system.

scholarly journals Computational Design of Modular Robots Based on Genetic Algorithm and Reinforcement Learning

Symmetry ◽  
2021 ◽  
Vol 13 (3) ◽  
pp. 471
Author(s):  
Jai Hoon Park ◽  
Kang Hoon Lee
Keyword(s):  
Genetic Algorithm ◽  
Reinforcement Learning ◽  
Design Space ◽  
Learning Algorithm ◽  
Computational Design ◽  
Computational Method ◽  
Learning Ability ◽  
Modular Robots ◽  
Control Mechanisms ◽  
Candidate Structure

Designing novel robots that can cope with a specific task is a challenging problem because of the enormous design space that involves both morphological structures and control mechanisms. To this end, we present a computational method for automating the design of modular robots. Our method employs a genetic algorithm to evolve robotic structures as an outer optimization, and it applies a reinforcement learning algorithm to each candidate structure to train its behavior and evaluate its potential learning ability as an inner optimization. The size of the design space is reduced significantly by evolving only the robotic structure and by performing behavioral optimization using a separate training algorithm compared to that when both the structure and behavior are evolved simultaneously. Mutual dependence between evolution and learning is achieved by regarding the mean cumulative rewards of a candidate structure in the reinforcement learning as its fitness in the genetic algorithm. Therefore, our method searches for prospective robotic structures that can potentially lead to near-optimal behaviors if trained sufficiently. We demonstrate the usefulness of our method through several effective design results that were automatically generated in the process of experimenting with actual modular robotics kit.

scholarly journals A game strategy model in the digital curling system based on NFSP

2021 ◽  
Author(s):  
Yuntao Han ◽  
Qibin Zhou ◽  
Fuqing Duan
Keyword(s):  
Reinforcement Learning ◽  
Nash Equilibrium ◽  
Action Space ◽  
Learning Networks ◽  
Game Tree ◽  
Continuous Action ◽  
Extensive Game ◽  
Strategy Model ◽  
Zero Sum ◽  
Tree Searching

AbstractThe digital curling game is a two-player zero-sum extensive game in a continuous action space. There are some challenging problems that are still not solved well, such as the uncertainty of strategy, the large game tree searching, and the use of large amounts of supervised data, etc. In this work, we combine NFSP and KR-UCT for digital curling games, where NFSP uses two adversary learning networks and can automatically produce supervised data, and KR-UCT can be used for large game tree searching in continuous action space. We propose two reward mechanisms to make reinforcement learning converge quickly. Experimental results validate the proposed method, and show the strategy model can reach the Nash equilibrium.

scholarly journals On-Demand Channel Bonding in Heterogeneous WLANs: A Multi-Agent Deep Reinforcement Learning Approach

Sensors ◽  
2020 ◽  
Vol 20 (10) ◽  
pp. 2789 ◽  
Author(s):  
Hang Qi ◽  
Hao Huang ◽  
Zhiqun Hu ◽  
Xiangming Wen ◽  
Zhaoming Lu
Keyword(s):  
Reinforcement Learning ◽  
Transmission Rate ◽  
Single Agent ◽  
Time Of Day ◽  
Action Space ◽  
Traffic Load ◽  
Traffic Demand ◽  
Channel Bonding ◽  
On Demand ◽  
Multi Agent

In order to meet the ever-increasing traffic demand of Wireless Local Area Networks (WLANs), channel bonding is introduced in IEEE 802.11 standards. Although channel bonding effectively increases the transmission rate, the wider channel reduces the number of non-overlapping channels and is more susceptible to interference. Meanwhile, the traffic load differs from one access point (AP) to another and changes significantly depending on the time of day. Therefore, the primary channel and channel bonding bandwidth should be carefully selected to meet traffic demand and guarantee the performance gain. In this paper, we proposed an On-Demand Channel Bonding (O-DCB) algorithm based on Deep Reinforcement Learning (DRL) for heterogeneous WLANs to reduce transmission delay, where the APs have different channel bonding capabilities. In this problem, the state space is continuous and the action space is discrete. However, the size of action space increases exponentially with the number of APs by using single-agent DRL, which severely affects the learning rate. To accelerate learning, Multi-Agent Deep Deterministic Policy Gradient (MADDPG) is used to train O-DCB. Real traffic traces collected from a campus WLAN are used to train and test O-DCB. Simulation results reveal that the proposed algorithm has good convergence and lower delay than other algorithms.

Sign in / Sign up

Export Citation Format

Share Document