Value iteration in a class of average controlled Markov chains with unbounded costs: necessary and sufficient conditions for pointwise convergence

1996 ◽  
Vol 33 (4) ◽  
pp. 986-1002 ◽  
Author(s):  
Rolando Cavazos-Cadena ◽  
Emmanuel Fernández-Gaucherand
Keyword(s):  
Markov Chains ◽  
Cost Function ◽  
Sufficient Conditions ◽  
Value Functions ◽  
Value Iteration ◽  
Controlled Markov Chains ◽  
Cost Criterion ◽  
Convergence Results ◽  
Differential Costs ◽  
Necessary And Sufficient

This work concerns controlled Markov chains with denumerable state space, (possibly) unbounded cost function, and an expected average cost criterion. Under a Lyapunov function condition, together with mild continuity-compactness assumptions, a simple necessary and sufficient criterion is given so that the relative value functions and differential costs produced by the value iteration scheme converge pointwise to the solution of the optimality equation; this criterion is applied to obtain convergence results when the cost function is bounded below or bounded above.

Value iteration in a class of average controlled Markov chains with unbounded costs: necessary and sufficient conditions for pointwise convergence

1996 ◽  
Vol 33 (04) ◽  
pp. 986-1002 ◽  
Author(s):  
Rolando Cavazos-Cadena ◽  
Emmanuel Fernández-Gaucherand
Keyword(s):  
Markov Chains ◽  
Cost Function ◽  
Sufficient Conditions ◽  
Value Functions ◽  
Value Iteration ◽  
Controlled Markov Chains ◽  
Cost Criterion ◽  
Convergence Results ◽  
Differential Costs ◽  
Necessary And Sufficient

This work concerns controlled Markov chains with denumerable state space, (possibly)unboundedcost function, and an expectedaveragecost criterion. Under aLyapunov function condition, together with mild continuity-compactness assumptions, a simplenecessary and sufficientcriterion is given so that the relative value functions and differential costs produced by thevalue iterationscheme converge pointwise to the solution of the optimality equation; this criterion is applied to obtain convergence results when the cost function is bounded below or bounded above.

Reversibility of Markov chains with applications to storage models

1985 ◽  
Vol 22 (01) ◽  
pp. 123-137 ◽  
Author(s):  
Hideo Ōsawa
Keyword(s):  
Markov Chains ◽  
State Space ◽  
Discrete Time ◽  
Risk Model ◽  
Sufficient Conditions ◽  
Single Server ◽  
General State ◽  
Insurance Risk ◽  
General State Space ◽  
Necessary And Sufficient

This paper studies the reversibility conditions of stationary Markov chains (discrete-time Markov processes) with general state space. In particular, we investigate the Markov chains having atomic points in the state space. Such processes are often seen in storage models, for example waiting time in a queue, insurance risk reserve, dam content and so on. The necessary and sufficient conditions for reversibility of these processes are obtained. Further, we apply these conditions to some storage models and present some interesting results for single-server queues and a finite insurance risk model.

Lumpability and marginalisability for continuous-time Markov chains

1993 ◽  
Vol 30 (3) ◽  
pp. 518-528 ◽  
Author(s):  
Frank Ball ◽  
Geoffrey F. Yeo
Keyword(s):  
Markov Chain ◽  
Markov Chains ◽  
Ion Channel ◽  
Continuous Time ◽  
Sufficient Conditions ◽  
Continuous Time Markov Chain ◽  
Continuous Time Markov Chains ◽  
Probabilistic Proof ◽  
Necessary And Sufficient ◽  
Mutually Independent

We consider lumpability for continuous-time Markov chains and provide a simple probabilistic proof of necessary and sufficient conditions for strong lumpability, valid in circumstances not covered by known theory. We also consider the following marginalisability problem. Let {X{t)} = {(X1(t), X2(t), · ··, Xm(t))} be a continuous-time Markov chain. Under what conditions are the marginal processes {X1(t)}, {X2(t)}, · ··, {Xm(t)} also continuous-time Markov chains? We show that this is related to lumpability and, if no two of the marginal processes can jump simultaneously, then they are continuous-time Markov chains if and only if they are mutually independent. Applications to ion channel modelling and birth–death processes are discussed briefly.

Poisson theorem for non-homogeneous Markov chains

1989 ◽  
Vol 26 (03) ◽  
pp. 637-642 ◽  
Author(s):  
Janusz Pawłowski
Keyword(s):  
Markov Chains ◽  
Poisson Distribution ◽  
Sufficient Conditions ◽  
Necessary And Sufficient Conditions ◽  
Compound Poisson Distribution ◽  
Convergence In Distribution ◽  
Compound Poisson ◽  
Necessary And Sufficient

This paper gives necessary and sufficient conditions for the convergence in distribution of sums of the 0–1 Markov chains to a compound Poisson distribution.

Reversibility of Markov chains with applications to storage models

1985 ◽  
Vol 22 (1) ◽  
pp. 123-137 ◽  
Author(s):  
Hideo Ōsawa
Keyword(s):  
Markov Chains ◽  
State Space ◽  
Discrete Time ◽  
Risk Model ◽  
Sufficient Conditions ◽  
Single Server ◽  
General State ◽  
Insurance Risk ◽  
General State Space ◽  
Necessary And Sufficient

This paper studies the reversibility conditions of stationary Markov chains (discrete-time Markov processes) with general state space. In particular, we investigate the Markov chains having atomic points in the state space. Such processes are often seen in storage models, for example waiting time in a queue, insurance risk reserve, dam content and so on. The necessary and sufficient conditions for reversibility of these processes are obtained. Further, we apply these conditions to some storage models and present some interesting results for single-server queues and a finite insurance risk model.

scholarly journals Necessary and sufficient conditions for recurrence and transience of Markov chains, in terms of inequalities

1978 ◽  
Vol 15 (4) ◽  
pp. 848-851 ◽  
Author(s):  
Jean-François Mertens ◽  
Ester Samuel-Cahn ◽  
Shmuel Zamir
Keyword(s):  
Markov Chain ◽  
Markov Chains ◽  
State Space ◽  
Bounded Solution ◽  
Sufficient Conditions ◽  
Necessary And Sufficient Conditions ◽  
Existence Of A Solution ◽  
Necessary And Sufficient ◽  
Irreducible Markov Chain

For an aperiodic, irreducible Markov chain with the non-negative integers as state space it is shown that the existence of a solution to in which yi → ∞is necessary and sufficient for recurrence, and the existence of a bounded solution to the same inequalities, with yk < yo, · · ·, yN–1 for some k ≧ N, is necessary and sufficient for transience.

scholarly journals Convergence on Gauss-Seidel iterative methods for linear systems with general H-matrices

2015 ◽  
Vol 30 ◽  
pp. 843-870 ◽  
Author(s):  
Cheng-yi Zhang ◽  
Dan Ye ◽  
Cong-Lei Zhong ◽  
SHUANGHUA SHUANGHUA
Keyword(s):  
Linear Systems ◽  
Iterative Methods ◽  
Sufficient Conditions ◽  
Numerical Linear Algebra ◽  
Convergence Results ◽  
Diagonally Dominant Matrices ◽  
Sufficient Conditions For Convergence ◽  
Diagonally Dominant ◽  
Necessary And Sufficient ◽  
Theoretical Results

It is well known that as a famous type of iterative methods in numerical linear algebra, Gauss-Seidel iterative methods are convergent for linear systems with strictly or irreducibly diagonally dominant matrices, invertible H−matrices (generalized strictly diagonally dominant matrices) and Hermitian positive definite matrices. But, the same is not necessarily true for linear systems with non-strictly diagonally dominant matrices and general H−matrices. This paper firstly proposes some necessary and sufficient conditions for convergence on Gauss-Seidel iterative methods to establish several new theoretical results on linear systems with nonstrictly diagonally dominant matrices and general H−matrices. Then, the convergence results on preconditioned Gauss-Seidel (PGS) iterative methods for general H−matrices are presented. Finally, some numerical examples are given to demonstrate the results obtained in this paper.

On the existence of submultiplicative moments for the stationary distributions of some Markovian random walks

1999 ◽  
Vol 36 (1) ◽  
pp. 78-85 ◽  
Author(s):  
M. S. Sgibnev
Keyword(s):  
Markov Chains ◽  
Random Walks ◽  
Sufficient Conditions ◽  
Random Variables ◽  
Necessary And Sufficient Conditions ◽  
Stationary Distributions ◽  
Necessary And Sufficient ◽  
Submultiplicative Function ◽  
Independent Identically Distributed

This paper is concerned with submultiplicative moments for the stationary distributions π of some Markov chains taking values in ℝ+ or ℝ which are closely related to the random walks generated by sequences of independent identically distributed random variables. Necessary and sufficient conditions are given for ∫φ(x)π(dx) < ∞, where φ(x) is a submultiplicative function, i.e. φ(0) = 1 and φ(x+y) ≤ φ(x)φ(y) for all x, y.

Sign in / Sign up

Export Citation Format

Share Document