Learning control of finite Markov chains with periodically varying transition probabilities

1990 ◽  
Author(s):  
M. Sato ◽  
H. Takeda
Keyword(s):  
Markov Chains ◽  
Transition Probabilities ◽  
Learning Control ◽  
Finite Markov Chains

Perturbation theory and finite Markov chains

1968 ◽  
Vol 5 (2) ◽  
pp. 401-413 ◽  
Author(s):  
Paul J. Schweitzer
Keyword(s):  
Perturbation Theory ◽  
Markov Chain ◽  
Markov Chains ◽  
Stationary Distribution ◽  
Fundamental Matrix ◽  
Transition Probabilities ◽  
Semi Group ◽  
Partial Derivatives ◽  
Finite Markov Chains ◽  
Derivatives Of

A perturbation formalism is presented which shows how the stationary distribution and fundamental matrix of a Markov chain containing a single irreducible set of states change as the transition probabilities vary. Expressions are given for the partial derivatives of the stationary distribution and fundamental matrix with respect to the transition probabilities. Semi-group properties of the generators of transformations from one Markov chain to another are investigated. It is shown that a perturbation formalism exists in the multiple subchain case if and only if the change in the transition probabilities does not alter the number of, or intermix the various subchains. The formalism is presented when this condition is satisfied.

Self-Learning Control of Finite Markov Chains

2018 ◽  
Author(s):  
A.S. Poznyak ◽  
Kaddour Najim ◽  
E. Gomez-Ramirez
Keyword(s):  
Markov Chains ◽  
Learning Control ◽  
Finite Markov Chains ◽  
Self Learning

A Note on External Uniformization for Finite Markov Chains in Continuous Time

1992 ◽  
Vol 6 (1) ◽  
pp. 127-131 ◽  
Author(s):  
Masaaki Kijima
Keyword(s):  
Markov Chains ◽  
Continuous Time ◽  
Numerical Experiments ◽  
Law Of Large Numbers ◽  
Transition Probabilities ◽  
Strong Law ◽  
Large Numbers ◽  
Finite Markov Chains

An external uniformization technique was developed by Ross [4] to obtain approximations of transition probabilities of finite Markov chains in continuous time. Yoon and Shanthikumar [7] then reported through extensive numerical experiments that this technique performs quite well compared to other existing methods. In this paper, we show that external uniformization results from the strong law of large numbers whose underlying distributions are exponential. Based on this observation, some remarks regarding properties of the approximation are given.

General transition probabilities for finite Markov chains

1964 ◽  
Vol 60 (1) ◽  
pp. 83-91 ◽  
Author(s):  
Marcel F. Neuts
Keyword(s):  
Markov Chain ◽  
Markov Chains ◽  
Finite Number ◽  
Discrete Time ◽  
Transition Probabilities ◽  
Initial State ◽  
Discrete Time Markov Chain ◽  
Finite Markov Chains

We consider a stationary discrete-time Markov chain with a finite number m of possible states which we designate by 1,…,m. We assume that at time t = 0 the process is in an initial state i with probability (i = 1,…, m) and such that and .

Perturbation theory and finite Markov chains

1968 ◽  
Vol 5 (02) ◽  
pp. 401-413 ◽  
Author(s):  
Paul J. Schweitzer
Keyword(s):  
Perturbation Theory ◽  
Markov Chain ◽  
Markov Chains ◽  
Stationary Distribution ◽  
Fundamental Matrix ◽  
Transition Probabilities ◽  
Semi Group ◽  
Partial Derivatives ◽  
Finite Markov Chains ◽  
Derivatives Of

A perturbation formalism is presented which shows how the stationary distribution and fundamental matrix of a Markov chain containing a single irreducible set of states change as the transition probabilities vary. Expressions are given for the partial derivatives of the stationary distribution and fundamental matrix with respect to the transition probabilities. Semi-group properties of the generators of transformations from one Markov chain to another are investigated. It is shown that a perturbation formalism exists in the multiple subchain case if and only if the change in the transition probabilities does not alter the number of, or intermix the various subchains. The formalism is presented when this condition is satisfied.

XXIV.—Conditions for the Ergodicity of Non-homogeneous Finite Markov Chains

1957 ◽  
Vol 64 (4) ◽  
pp. 369-380 ◽  
Author(s):  
J. L. Mott
Keyword(s):  
Markov Chain ◽  
Markov Chains ◽  
Asymptotic Behaviour ◽  
Transition Probabilities ◽  
Finite Markov Chain ◽  
Finite Markov Chains ◽  
Product Of Matrices

SynopsisIn this note we study the asymptotic behaviour of a product of matrices where Pj is a matrix of transition probabilities in a non-homogeneous finite Markov chain. We give conditions that (i) the rows of P(n) tend to identity and that (ii) P(n) tends to a limit matrix with identical rows.

Sign in / Sign up

Export Citation Format

Share Document