A mathematical programming approach to a problem in variance penalised Markov decision processes

OR Spectrum ◽  
1994 ◽  
Vol 15 (4) ◽  
pp. 225-230 ◽  
Author(s):  
D. J. White
Keyword(s):  
Mathematical Programming ◽  
Markov Decision Processes ◽  
Decision Processes ◽  
Programming Approach ◽  
Markov Decision ◽  
Mathematical Programming Approach

scholarly journals The Expected Total Cost Criterion for Markov Decision Processes under Constraints

2013 ◽  
Vol 45 (3) ◽  
pp. 837-859 ◽  
Author(s):  
François Dufour ◽  
A. B. Piunovskiy
Keyword(s):  
Markov Decision Processes ◽  
Optimal Solution ◽  
Decision Processes ◽  
Linear Program ◽  
Programming Approach ◽  
Stationary Policy ◽  
Total Cost ◽  
Optimal Value ◽  
Markov Decision ◽  
Expected Total Cost

In this work, we study discrete-time Markov decision processes (MDPs) with constraints when all the objectives have the same form of expected total cost over the infinite time horizon. Our objective is to analyze this problem by using the linear programming approach. Under some technical hypotheses, it is shown that if there exists an optimal solution for the associated linear program then there exists a randomized stationary policy which is optimal for the MDP, and that the optimal value of the linear program coincides with the optimal value of the constrained control problem. A second important result states that the set of randomized stationary policies provides a sufficient set for solving this MDP. It is important to note that, in contrast with the classical results of the literature, we do not assume the MDP to be transient or absorbing. More importantly, we do not impose the cost functions to be nonnegative or to be bounded below. Several examples are presented to illustrate our results.

scholarly journals Using mathematical programming to solve Factored Markov Decision Processes with Imprecise Probabilities

2011 ◽  
Vol 52 (7) ◽  
pp. 1000-1017 ◽  
Author(s):  
Karina Valdivia Delgado ◽  
Leliane Nunes de Barros ◽  
Fabio Gagliardi Cozman ◽  
Scott Sanner
Keyword(s):  
Mathematical Programming ◽  
Markov Decision Processes ◽  
Decision Processes ◽  
Imprecise Probabilities ◽  
Markov Decision

scholarly journals On Linear Programming for Constrained and Unconstrained Average-Cost Markov Decision Processes with Countable Action Spaces and Strictly Unbounded Costs

2021 ◽  
Author(s):  
Huizhen Yu
Keyword(s):  
Linear Programming ◽  
Markov Decision Processes ◽  
Average Cost ◽  
Decision Processes ◽  
The State ◽  
Programming Approach ◽  
One Stage ◽  
Markov Decision ◽  
Action Spaces ◽  
Borel Measurable

We consider the linear programming approach for constrained and unconstrained Markov decision processes (MDPs) under the long-run average-cost criterion, where the class of MDPs in our study have Borel state spaces and discrete countable action spaces. Under a strict unboundedness condition on the one-stage costs and a recently introduced majorization condition on the state transition stochastic kernel, we study infinite-dimensional linear programs for the average-cost MDPs and prove the absence of a duality gap and other optimality results. Our results do not require a lower-semicontinuous MDP model. Thus, they can be applied to countable action space MDPs where the dynamics and one-stage costs are discontinuous in the state variable. Our proofs make use of the continuity property of Borel measurable functions asserted by Lusin’s theorem.

scholarly journals The Linear Programming Approach to Reach-Avoid Problems for Markov Decision Processes

2017 ◽  
Vol 60 ◽  
pp. 263-285 ◽  
Author(s):  
Nikolaos Kariotoglou ◽  
Maryam Kamgarpour ◽  
Tyler H. Summers ◽  
John Lygeros
Keyword(s):  
Markov Decision Processes ◽  
Dimensional Space ◽  
Decision Processes ◽  
Linear Program ◽  
Control Synthesis ◽  
Programming Approach ◽  
Finite Dimensional Space ◽  
Infinite Dimensional ◽  
Finite Dimensional ◽  
Markov Decision

One of the most fundamental problems in Markov decision processes is analysis and control synthesis for safety and reachability specifications. We consider the stochastic reach-avoid problem, in which the objective is to synthesize a control policy to maximize the probability of reaching a target set at a given time, while staying in a safe set at all prior times. We characterize the solution to this problem through an infinite dimensional linear program. We then develop a tractable approximation to the infinite dimensional linear program through finite dimensional approximations of the decision space and constraints. For a large class of Markov decision processes modeled by Gaussian mixtures kernels we show that through a proper selection of the finite dimensional space, one can further reduce the computational complexity of the resulting linear program. We validate the proposed method and analyze its potential with numerical case studies.

scholarly journals The Expected Total Cost Criterion for Markov Decision Processes under Constraints

2013 ◽  
Vol 45 (03) ◽  
pp. 837-859 ◽  
Author(s):  
François Dufour ◽  
A. B. Piunovskiy
Keyword(s):  
Markov Decision Processes ◽  
Optimal Solution ◽  
Decision Processes ◽  
Linear Program ◽  
Programming Approach ◽  
Stationary Policy ◽  
Total Cost ◽  
Optimal Value ◽  
Markov Decision ◽  
Expected Total Cost

In this work, we study discrete-time Markov decision processes (MDPs) with constraints when all the objectives have the same form of expected total cost over the infinite time horizon. Our objective is to analyze this problem by using the linear programming approach. Under some technical hypotheses, it is shown that if there exists an optimal solution for the associated linear program then there exists a randomized stationary policy which is optimal for the MDP, and that the optimal value of the linear program coincides with the optimal value of the constrained control problem. A second important result states that the set of randomized stationary policies provides a sufficient set for solving this MDP. It is important to note that, in contrast with the classical results of the literature, we do not assume the MDP to be transient or absorbing. More importantly, we do not impose the cost functions to be nonnegative or to be bounded below. Several examples are presented to illustrate our results.

Sign in / Sign up

Export Citation Format

Share Document