Reversibility, invariance and μ-invariance

1988 ◽  
Vol 20 (3) ◽  
pp. 600-621 ◽  
Author(s):  
P. K. Pollett
Keyword(s):  
Stationary Processes ◽  
Transition Rates ◽  
Dual Processes ◽  
Transition Matrices ◽  
Stationary Distributions ◽  
Birth And Death Processes ◽  
The Family ◽  
Transient Case ◽  
Detail Balance ◽  
Quasi Stationary Distribution

In this paper we consider a number of questions relating to the problem of determining quasi-stationary distributions for transient Markov processes. First we find conditions under which a measure or vector that is µ-invariant for a matrix of transition rates is also μ-invariant for the family of transition matrices of the minimal process it generates. These provide a means for determining whether or not the so-called stationary conditional quasi-stationary distribution exists in the λ-transient case. The process is not assumed to be regular, nor is it assumed to be uniform or irreducible. In deriving the invariance conditions we reveal a relationship between μ-invariance and the invariance of measures for related processes called the μ-reverse and the μ-dual processes. They play a role analogous to the time-reverse process which arises in the discussion of stationary distributions. Secondly we bring the related notions of detail-balance and reversibility into the realm of quasi-stationary processes. For example, if a process can be identified as being μ-reversible, the problem of determining quasi-stationary distributions is made much simpler. Finally, we consider some practical problems that emerge when calculating quasi-stationary distributions directly from the transition rates of the process. Our results are illustrated with reference to a variety of processes including examples of birth and death processes and the birth, death and catastrophe process.

Reversibility, invariance and μ-invariance

1988 ◽  
Vol 20 (03) ◽  
pp. 600-621
Author(s):  
P. K. Pollett
Keyword(s):  
Stationary Processes ◽  
Transition Rates ◽  
Dual Processes ◽  
Transition Matrices ◽  
Stationary Distributions ◽  
Birth And Death Processes ◽  
The Family ◽  
Transient Case ◽  
Detail Balance ◽  
Quasi Stationary Distribution

In this paper we consider a number of questions relating to the problem of determining quasi-stationary distributions for transient Markov processes. First we find conditions under which a measure or vector that is µ-invariant for a matrix of transition rates is also μ-invariant for the family of transition matrices of the minimal process it generates. These provide a means for determining whether or not the so-called stationary conditional quasi-stationary distribution exists in the λ-transient case. The process is not assumed to be regular, nor is it assumed to be uniform or irreducible. In deriving the invariance conditions we reveal a relationship between μ-invariance and the invariance of measures for related processes called the μ-reverse and the μ-dual processes. They play a role analogous to the time-reverse process which arises in the discussion of stationary distributions. Secondly we bring the related notions of detail-balance and reversibility into the realm of quasi-stationary processes. For example, if a process can be identified as being μ-reversible, the problem of determining quasi-stationary distributions is made much simpler. Finally, we consider some practical problems that emerge when calculating quasi-stationary distributions directly from the transition rates of the process. Our results are illustrated with reference to a variety of processes including examples of birth and death processes and the birth, death and catastrophe process.

An Efficient Procedure for Computing Quasi-Stationary Distributions of Markov Chains by Sparse Transition Structure

1994 ◽  
Vol 26 (01) ◽  
pp. 68-79
Author(s):  
P. K. Pollett ◽  
D. E. Stewart
Keyword(s):  
Markov Chain ◽  
Continuous Time ◽  
Computational Procedure ◽  
Transition Rates ◽  
Continuous Time Markov Chain ◽  
Transition Structure ◽  
Stationary Distributions ◽  
Efficient Procedure ◽  
The Matrix ◽  
Quasi Stationary Distribution

We describe a computational procedure for evaluating the quasi-stationary distributions of a continuous-time Markov chain. Our method, which is an ‘iterative version' of Arnoldi's algorithm, is appropriate for dealing with cases where the matrix of transition rates is large and sparse, but does not exhibit a banded structure which might otherwise be usefully exploited. We illustrate the method with reference to an epidemic model and we compare the computed quasi-stationary distribution with an appropriate diffusion approximation.

Quasi-stationary distributions of birth-and-death processes

1978 ◽  
Vol 10 (03) ◽  
pp. 570-586 ◽  
Author(s):  
James A. Cavender
Keyword(s):  
State Space ◽  
Stationary Distribution ◽  
Parameter Family ◽  
Death Process ◽  
Birth And Death Process ◽  
Stationary Distributions ◽  
Birth And Death Processes ◽  
Finite State ◽  
Finite State Space ◽  
Quasi Stationary Distribution

Letqn(t) be the conditioned probability of finding a birth-and-death process in statenat timet,given that absorption into state 0 has not occurred by then. A family {q1(t),q2(t), · · ·} that is constant in time is a quasi-stationary distribution. If any exist, the quasi-stationary distributions comprise a one-parameter family related to quasi-stationary distributions of finite state-space approximations to the process.

An Efficient Procedure for Computing Quasi-Stationary Distributions of Markov Chains by Sparse Transition Structure

1994 ◽  
Vol 26 (1) ◽  
pp. 68-79 ◽  
Author(s):  
P. K. Pollett ◽  
D. E. Stewart
Keyword(s):  
Markov Chain ◽  
Continuous Time ◽  
Computational Procedure ◽  
Transition Rates ◽  
Continuous Time Markov Chain ◽  
Transition Structure ◽  
Stationary Distributions ◽  
Efficient Procedure ◽  
The Matrix ◽  
Quasi Stationary Distribution

We describe a computational procedure for evaluating the quasi-stationary distributions of a continuous-time Markov chain. Our method, which is an ‘iterative version' of Arnoldi's algorithm, is appropriate for dealing with cases where the matrix of transition rates is large and sparse, but does not exhibit a banded structure which might otherwise be usefully exploited. We illustrate the method with reference to an epidemic model and we compare the computed quasi-stationary distribution with an appropriate diffusion approximation.

Quasi-stationary distributions of birth-and-death processes

1978 ◽  
Vol 10 (3) ◽  
pp. 570-586 ◽  
Author(s):  
James A. Cavender
Keyword(s):  
State Space ◽  
Stationary Distribution ◽  
Parameter Family ◽  
Death Process ◽  
Birth And Death Process ◽  
Stationary Distributions ◽  
Birth And Death Processes ◽  
Finite State ◽  
Finite State Space ◽  
Quasi Stationary Distribution

Let qn(t) be the conditioned probability of finding a birth-and-death process in state n at time t, given that absorption into state 0 has not occurred by then. A family {q1(t), q2(t), · · ·} that is constant in time is a quasi-stationary distribution. If any exist, the quasi-stationary distributions comprise a one-parameter family related to quasi-stationary distributions of finite state-space approximations to the process.

scholarly journals Denumerable state continuous time Markov decision processes with unbounded cost and transition rates under average criterion

2002 ◽  
Vol 43 (4) ◽  
pp. 541-557 ◽  
Author(s):  
Xianping Guo ◽  
Weiping Zhu
Keyword(s):  
Markov Decision Processes ◽  
Continuous Time ◽  
Decision Processes ◽  
Transition Rates ◽  
Birth And Death Processes ◽  
Optimality Equation ◽  
Average Criterion ◽  
Markov Decision ◽  
Unbounded Cost ◽  
Queue Model

AbstractIn this paper, we consider denumerable state continuous time Markov decision processes with (possibly unbounded) transition and cost rates under average criterion. We present a set of conditions and prove the existence of both average cost optimal stationary policies and a solution of the average optimality equation under the conditions. The results in this paper are applied to an admission control queue model and controlled birth and death processes.

On the relationship between µ-invariant measures and quasi-stationary distributions for continuous-time Markov chains

1993 ◽  
Vol 25 (01) ◽  
pp. 82-102
Author(s):  
M. G. Nair ◽  
P. K. Pollett
Keyword(s):  
Invariant Measure ◽  
Stationary Distribution ◽  
Continuous Time ◽  
Sufficient Condition ◽  
Transition Rates ◽  
Necessary And Sufficient Condition ◽  
Absorbing State ◽  
Necessary And Sufficient ◽  
Forward Equations ◽  
Quasi Stationary Distribution

In a recent paper, van Doorn (1991) explained how quasi-stationary distributions for an absorbing birth-death process could be determined from the transition rates of the process, thus generalizing earlier work of Cavender (1978). In this paper we shall show that many of van Doorn's results can be extended to deal with an arbitrary continuous-time Markov chain over a countable state space, consisting of an irreducible class, C, and an absorbing state, 0, which is accessible from C. Some of our results are extensions of theorems proved for honest chains in Pollett and Vere-Jones (1992). In Section 3 we prove that a probability distribution on C is a quasi-stationary distribution if and only if it is a µ-invariant measure for the transition function, P. We shall also show that if m is a quasi-stationary distribution for P, then a necessary and sufficient condition for m to be µ-invariant for Q is that P satisfies the Kolmogorov forward equations over C. When the remaining forward equations hold, the quasi-stationary distribution must satisfy a set of ‘residual equations' involving the transition rates into the absorbing state. The residual equations allow us to determine the value of µ for which the quasi-stationary distribution is µ-invariant for P. We also prove some more general results giving bounds on the values of µ for which a convergent measure can be a µ-subinvariant and then µ-invariant measure for P. The remainder of the paper is devoted to the question of when a convergent µ-subinvariant measure, m, for Q is a quasi-stationary distribution. Section 4 establishes a necessary and sufficient condition for m to be a quasi-stationary distribution for the minimal chain. In Section 5 we consider ‘single-exit' chains. We derive a necessary and sufficient condition for there to exist a process for which m is a quasi-stationary distribution. Under this condition all such processes can be specified explicitly through their resolvents. The results proved here allow us to conclude that the bounds for µ obtained in Section 3 are, in fact, tight. Finally, in Section 6, we illustrate our results by way of two examples: regular birth-death processes and a pure-birth process with absorption.

Optimal control of random walks, birth and death processes, and queues

1981 ◽  
Vol 13 (01) ◽  
pp. 61-83 ◽  
Author(s):  
Richard Serfozo
Keyword(s):  
Optimal Control ◽  
Random Walks ◽  
Queue Length ◽  
Transition Probabilities ◽  
Transition Rates ◽  
Average Reward ◽  
Birth And Death Processes ◽  
Optimal Policies ◽  
One Step ◽  
Increasing Functions

This is a study of simple random walks, birth and death processes, and M/M/s queues that have transition probabilities and rates that are sequentially controlled at jump times of the processes. Each control action yields a one-step reward depending on the chosen probabilities or transition rates and the state of the process. The aim is to find control policies that maximize the total discounted or average reward. Conditions are given for these processes to have certain natural monotone optimal policies. Under such a policy for the M/M/s queue, for example, the service and arrival rates are non-decreasing and non-increasing functions, respectively, of the queue length. Properties of these policies and a linear program for computing them are also discussed.

Generalized two-dimensional bounded birth and death processes and some applications

1981 ◽  
Vol 18 (02) ◽  
pp. 335-347 ◽  
Author(s):  
L. Billard
Keyword(s):  
Species Interaction ◽  
Transition Rates ◽  
Two Dimensional ◽  
Birth And Death Processes ◽  
Arbitrary Time ◽  
Interaction Processes ◽  
Epidemic Theory ◽  
Independent Functions ◽  
Population Sizes

Two-dimensional bounded birth and death processes are considered. State probabilities are given for the generalized case in which the transition rates are arbitrary time-independent functions of the population sizes. Some applications in epidemic theory and species interaction processes are briefly described.

A note on quasi-stationary distributions of birth–death processes and the SIS logistic epidemic

2003 ◽  
Vol 40 (3) ◽  
pp. 821-825 ◽  
Author(s):  
Damian Clancy ◽  
Philip K. Pollett
Keyword(s):  
Probability Distributions ◽  
Explicit Representation ◽  
Death Process ◽  
Conditional Distributions ◽  
Stationary Distributions ◽  
Absorbing State ◽  
Likelihood Ratio Ordering ◽  
Positive Integers ◽  
Quasi Stationary Distribution ◽  
Birth Death

For Markov processes on the positive integers with the origin as an absorbing state, Ferrari, Kesten, Martínez and Picco studied the existence of quasi-stationary and limiting conditional distributions by characterizing quasi-stationary distributions as fixed points of a transformation Φ on the space of probability distributions on {1, 2, …}. In the case of a birth–death process, the components of Φ(ν) can be written down explicitly for any given distribution ν. Using this explicit representation, we will show that Φ preserves likelihood ratio ordering between distributions. A conjecture of Kryscio and Lefèvre concerning the quasi-stationary distribution of the SIS logistic epidemic follows as a corollary.

Sign in / Sign up

Export Citation Format

Share Document