Optimal control in light traffic Markov decision processes

1997 ◽  
Vol 45 (1) ◽  
pp. 63-79 ◽  
Author(s):  
Ger Koole ◽  
Olaf Passchier
Keyword(s):  
Optimal Control ◽  
Markov Decision Processes ◽  
Decision Processes ◽  
Light Traffic ◽  
Markov Decision

scholarly journals Impulsive Control for Continuous-Time Markov Decision Processes

2015 ◽  
Vol 47 (1) ◽  
pp. 106-127 ◽  
Author(s):  
François Dufour ◽  
Alexei B. Piunovskiy
Keyword(s):  
Optimal Control ◽  
Control Problem ◽  
Markov Decision Processes ◽  
Control Strategy ◽  
Continuous Time ◽  
Sufficient Conditions ◽  
Decision Processes ◽  
Optimal Control Strategy ◽  
Optimality Equation ◽  
Markov Decision

In this paper our objective is to study continuous-time Markov decision processes on a general Borel state space with both impulsive and continuous controls for the infinite time horizon discounted cost. The continuous-time controlled process is shown to be nonexplosive under appropriate hypotheses. The so-called Bellman equation associated to this control problem is studied. Sufficient conditions ensuring the existence and the uniqueness of a bounded measurable solution to this optimality equation are provided. Moreover, it is shown that the value function of the optimization problem under consideration satisfies this optimality equation. Sufficient conditions are also presented to ensure on the one hand the existence of an optimal control strategy, and on the other hand the existence of a ε-optimal control strategy. The decomposition of the state space into two disjoint subsets is exhibited where, roughly speaking, one should apply a gradual action or an impulsive action correspondingly to obtain an optimal or ε-optimal strategy. An interesting consequence of our previous results is as follows: the set of strategies that allow interventions at time t = 0 and only immediately after natural jumps is a sufficient set for the control problem under consideration.

scholarly journals Impulsive Control for Continuous-Time Markov Decision Processes

2015 ◽  
Vol 47 (01) ◽  
pp. 106-127 ◽  
Author(s):  
François Dufour ◽  
Alexei B. Piunovskiy
Keyword(s):  
Optimal Control ◽  
Control Problem ◽  
Markov Decision Processes ◽  
Control Strategy ◽  
Continuous Time ◽  
Sufficient Conditions ◽  
Decision Processes ◽  
Optimal Control Strategy ◽  
Optimality Equation ◽  
Markov Decision

In this paper our objective is to study continuous-time Markov decision processes on a general Borel state space with both impulsive and continuous controls for the infinite time horizon discounted cost. The continuous-time controlled process is shown to be nonexplosive under appropriate hypotheses. The so-called Bellman equation associated to this control problem is studied. Sufficient conditions ensuring the existence and the uniqueness of a bounded measurable solution to this optimality equation are provided. Moreover, it is shown that the value function of the optimization problem under consideration satisfies this optimality equation. Sufficient conditions are also presented to ensure on the one hand the existence of an optimal control strategy, and on the other hand the existence of a ε-optimal control strategy. The decomposition of the state space into two disjoint subsets is exhibited where, roughly speaking, one should apply a gradual action or an impulsive action correspondingly to obtain an optimal or ε-optimal strategy. An interesting consequence of our previous results is as follows: the set of strategies that allow interventions at time t = 0 and only immediately after natural jumps is a sufficient set for the control problem under consideration.

scholarly journals The Expected Total Cost Criterion for Markov Decision Processes under Constraints: A Convex Analytic Approach

2012 ◽  
Vol 44 (3) ◽  
pp. 774-793 ◽  
Author(s):  
François Dufour ◽  
M. Horiguchi ◽  
A. B. Piunovskiy
Keyword(s):  
Optimal Control ◽  
Markov Decision Processes ◽  
Optimal Solution ◽  
Decision Processes ◽  
Linear Program ◽  
Occupation Measure ◽  
Stationary Policy ◽  
Total Cost ◽  
Markov Decision ◽  
Expected Total Cost

This paper deals with discrete-time Markov decision processes (MDPs) under constraints where all the objectives have the same form of expected total cost over the infinite time horizon. The existence of an optimal control policy is discussed by using the convex analytic approach. We work under the assumptions that the state and action spaces are general Borel spaces, and that the model is nonnegative, semicontinuous, and there exists an admissible solution with finite cost for the associated linear program. It is worth noting that, in contrast to the classical results in the literature, our hypotheses do not require the MDP to be transient or absorbing. Our first result ensures the existence of an optimal solution to the linear program given by an occupation measure of the process generated by a randomized stationary policy. Moreover, it is shown that this randomized stationary policy provides an optimal solution to this Markov control problem. As a consequence, these results imply that the set of randomized stationary policies is a sufficient set for this optimal control problem. Finally, our last main result states that all optimal solutions of the linear program coincide on a special set with an optimal occupation measure generated by a randomized stationary policy. Several examples are presented to illustrate some theoretical issues and the possible applications of the results developed in the paper.

scholarly journals A particle-based policy for the optimal control of Markov decision processes

2014 ◽  
Vol 47 (3) ◽  
pp. 10518-10523 ◽  
Author(s):  
M. Pirotta ◽  
G. Manganini ◽  
L. Piroddi ◽  
M. Prandini ◽  
M. Restelli
Keyword(s):  
Optimal Control ◽  
Markov Decision Processes ◽  
Decision Processes ◽  
Markov Decision
Sign in / Sign up

Export Citation Format

Share Document