scholarly journals Regret Bounds for Reinforcement Learning via Markov Chain Concentration

2020 ◽  
Vol 67 ◽  
pp. 115-128
Author(s):  
Ronald Ortner
Keyword(s):  
Markov Chain ◽  
Reinforcement Learning ◽  
Markov Decision Processes ◽  
Mixing Time ◽  
Decision Processes ◽  
Time Parameter ◽  
Markov Decision ◽  
Regret Bounds ◽  
Optimistic Algorithm ◽  
S States

We give a simple optimistic algorithm for which it is easy to derive regret bounds of O(sqrt{t-mix SAT}) steps in uniformly ergodic Markov decision processes with S states, A actions, and mixing time parameter t-mix. These bounds are the first regret bounds in the general, non-episodic setting with an optimal dependence on all given parameters. They could only be improved by using an alternative mixing time parameter.

Average optimal policies in Markov decision drift processes with applications to a queueing and a replacement model

1983 ◽  
Vol 15 (2) ◽  
pp. 274-303 ◽  
Author(s):  
Arie Hordijk ◽  
Frank A. Van Der Duyn Schouten
Keyword(s):  
Markov Decision Processes ◽  
Optimal Policy ◽  
Continuous Time ◽  
Sufficient Conditions ◽  
Decision Processes ◽  
Time Parameter ◽  
Queueing Model ◽  
Replacement Model ◽  
Optimal Policies ◽  
Markov Decision

Recently the authors introduced the concept of Markov decision drift processes. A Markov decision drift process can be seen as a straightforward generalization of a Markov decision process with continuous time parameter. In this paper we investigate the existence of stationary average optimal policies for Markov decision drift processes. Using a well-known Abelian theorem we derive sufficient conditions, which guarantee that a ‘limit point' of a sequence of discounted optimal policies with the discounting factor approaching 1 is an average optimal policy. An alternative set of sufficient conditions is obtained for the case in which the discounted optimal policies generate regenerative stochastic processes. The latter set of conditions is easier to verify in several applications. The results of this paper are also applicable to Markov decision processes with discrete or continuous time parameter and to semi-Markov decision processes. In this sense they generalize some well-known results for Markov decision processes with finite or compact action space. Applications to an M/M/1 queueing model and a maintenance replacement model are given. It is shown that under certain conditions on the model parameters the average optimal policy for the M/M/1 queueing model is monotone non-decreasing (as a function of the number of waiting customers) with respect to the service intensity and monotone non-increasing with respect to the arrival intensity. For the maintenance replacement model we prove the average optimality of a bang-bang type policy. Special attention is paid to the computation of the optimal control parameters.

Sign in / Sign up

Export Citation Format

Share Document