Nondiscounted Continuous Time Markovian Decision Process with Countable State Space

1972 ◽  
Vol 10 (1) ◽  
pp. 210-220 ◽  
Author(s):  
Prasadarao Kakumanu
Keyword(s):  
State Space ◽  
Continuous Time ◽  
Decision Process ◽  
Countable State Space ◽  
Markovian Decision Process ◽  
Countable State

scholarly journals Two Approaches to the Construction of Perturbation Bounds for Continuous-Time Markov Chains

Mathematics ◽  
2020 ◽  
Vol 8 (2) ◽  
pp. 253 ◽  
Author(s):  
Alexander Zeifman ◽  
Victor Korolev ◽  
Yacov Satin
Keyword(s):  
Markov Chains ◽  
State Space ◽  
Continuous Time ◽  
Countable State Space ◽  
Perturbation Bounds ◽  
Queuing Models ◽  
Continuous Time Markov Chains ◽  
Quantitative Estimates ◽  
Countable State

This paper is largely a review. It considers two main methods used to study stability and to obtain appropriate quantitative estimates of perturbations of (inhomogeneous) Markov chains with continuous time and a finite or countable state space. An approach is described to the construction of perturbation estimates for the main five classes of such chains associated with queuing models. Several specific models are considered for which the limit characteristics and perturbation bounds for admissible “perturbed” processes are calculated.

Quasi-ergodicity for non-homogeneous continuous-time Markov chains

1989 ◽  
Vol 26 (3) ◽  
pp. 643-648 ◽  
Author(s):  
A. I. Zeifman
Keyword(s):  
Differential Equation ◽  
Markov Chain ◽  
Markov Chains ◽  
State Space ◽  
Continuous Time ◽  
Sufficient Conditions ◽  
Continuous Time Markov Chain ◽  
Countable State Space ◽  
Continuous Time Markov Chains ◽  
Countable State

We consider a non-homogeneous continuous-time Markov chain X(t) with countable state space. Definitions of uniform and strong quasi-ergodicity are introduced. The forward Kolmogorov system for X(t) is considered as a differential equation in the space of sequences l1. Sufficient conditions for uniform quasi-ergodicity are deduced from this equation. We consider conditions of uniform and strong ergodicity in the case of proportional intensities.

scholarly journals Path integrals for continuous-time Markov chains

2002 ◽  
Vol 39 (4) ◽  
pp. 901-904 ◽  
Author(s):  
P. K. Pollett ◽  
V. T. Stefanov
Keyword(s):  
Markov Chains ◽  
State Space ◽  
Path Integral ◽  
Continuous Time ◽  
Path Integrals ◽  
Countable State Space ◽  
Continuous Time Markov Chains ◽  
Countable State

This note presents a method of evaluating the distribution of a path integral for Markov chains on a countable state space.

Quasi-ergodicity for non-homogeneous continuous-time Markov chains

1989 ◽  
Vol 26 (03) ◽  
pp. 643-648 ◽  
Author(s):  
A. I. Zeifman
Keyword(s):  
Differential Equation ◽  
Markov Chain ◽  
Markov Chains ◽  
State Space ◽  
Continuous Time ◽  
Sufficient Conditions ◽  
Continuous Time Markov Chain ◽  
Countable State Space ◽  
Continuous Time Markov Chains ◽  
Countable State

We consider a non-homogeneous continuous-time Markov chain X(t) with countable state space. Definitions of uniform and strong quasi-ergodicity are introduced. The forward Kolmogorov system for X(t) is considered as a differential equation in the space of sequences l 1 . Sufficient conditions for uniform quasi-ergodicity are deduced from this equation. We consider conditions of uniform and strong ergodicity in the case of proportional intensities.

scholarly journals Path integrals for continuous-time Markov chains

2002 ◽  
Vol 39 (04) ◽  
pp. 901-904 ◽  
Author(s):  
P. K. Pollett ◽  
V. T. Stefanov
Keyword(s):  
Markov Chains ◽  
State Space ◽  
Path Integral ◽  
Continuous Time ◽  
Path Integrals ◽  
Countable State Space ◽  
Continuous Time Markov Chains ◽  
Countable State

This note presents a method of evaluating the distribution of a path integral for Markov chains on a countable state space.

THE SHANNON–MCMILLAN THEOREM FOR MARKOV CHAINS INDEXED BY A CAYLEY TREE IN RANDOM ENVIRONMENT

2017 ◽  
Vol 32 (4) ◽  
pp. 626-639 ◽  
Author(s):  
Zhiyan Shi ◽  
Pingping Zhong ◽  
Yan Fan
Keyword(s):  
Markov Chains ◽  
State Space ◽  
Random Environment ◽  
Cayley Tree ◽  
Law Of Large Numbers ◽  
Strong Law ◽  
Countable State Space ◽  
Large Numbers ◽  
Countable State ◽  
Definition Of

In this paper, we give the definition of tree-indexed Markov chains in random environment with countable state space, and then study the realization of Markov chain indexed by a tree in random environment. Finally, we prove the strong law of large numbers and Shannon–McMillan theorem for Markov chains indexed by a Cayley tree in a Markovian environment with countable state space.

Sign in / Sign up

Export Citation Format

Share Document