Optimal decision procedures for finite markov chains. Part I: Examples

A Markov process in discrete time with a finite state space is controlled by choosing the transition probabilities from a prescribed set depending on the state occupied at any time. Given the immediate cost for each choice, it is required to minimise the expected cost over an infinite future, without discounting. Various techniques are reviewed for the case when there is a finite set of possible transition matrices and an example is given to illustrate the unpredictable behaviour of policy sequences derived by backward induction. Further examples show that the existing methods may break down when there is an infinite family of transition matrices. A new approach is suggested, based on the idea of classifying the states according to their accessibility from one another.

Download Full-text

Optimal decision procedures for finite Markov chains. Part II: Communicating systems

Advances in Applied Probability ◽

10.2307/1425832 ◽

1973 ◽

Vol 5 (3) ◽

pp. 521-540 ◽

Cited By ~ 54

Author(s):

John Bather

Keyword(s):

Markov Process ◽

General Solution ◽

Transition Probabilities ◽

Optimal Decision ◽

Positive Probability ◽

Finite State ◽

Finite Markov Chains ◽

Special Case ◽

Family Of Distributions ◽

Finite State Space

A Markov process in discrete time with a finite state space is controlled by choosing the transition probabilities from a given convex family of distributions depending on the present state. The immediate cost is prescribed for each choice and it is required to minimise the average expected cost over an infinite future. The paper considers a special case of this general problem and provides the foundation for a general solution. The main result is that an optimal policy exists if each state of the system can be reached with positive probability from any other state by choosing a suitable policy.

Download Full-text

Optimal decision procedures for finite Markov chains. Part II: Communicating systems

Advances in Applied Probability ◽

10.1017/s0001867800039409 ◽

1973 ◽

Vol 5 (03) ◽

pp. 521-540 ◽

Cited By ~ 5

Author(s):

John Bather

Keyword(s):

Markov Process ◽

General Solution ◽

Transition Probabilities ◽

Optimal Decision ◽

Positive Probability ◽

Finite State ◽

Finite Markov Chains ◽

Special Case ◽

Family Of Distributions ◽

Finite State Space

A Markov process in discrete time with a finite state space is controlled by choosing the transition probabilities from a given convex family of distributions depending on the present state. The immediate cost is prescribed for each choice and it is required to minimise the average expected cost over an infinite future. The paper considers a special case of this general problem and provides the foundation for a general solution. The main result is that an optimal policy exists if each state of the system can be reached with positive probability from any other state by choosing a suitable policy.

Download Full-text

Strong consistency of a modified maximum likelihood estimator for controlled Markov chains

Journal of Applied Probability ◽

10.1017/s0021900200033830 ◽

1980 ◽

Vol 17 (03) ◽

pp. 726-734 ◽

Cited By ~ 4

Author(s):

Bharat Doshi ◽

Steven E. Shreve

Keyword(s):

Likelihood Function ◽

Transition Probabilities ◽

Control Law ◽

Likelihood Estimator ◽

Modified Maximum Likelihood ◽

Control Scheme ◽

Finite State ◽

Finite Set ◽

Stationary Control ◽

Finite State Space

A controlled Markov chain with finite state space has transition probabilities which depend on an unknown parameter α lying in a known finite set A. For each α, a stationary control law ϕ α is given. This paper develops a control scheme whereby at each stage t a parameter α t is chosen at random from among those parameters which nearly maximize the log likelihood function, and the control ut is chosen according to the control law ϕ αt. It is proved that this algorithm leads to identification of the true α under conditions weaker than any previously considered.

Download Full-text

Strong consistency of a modified maximum likelihood estimator for controlled Markov chains

Journal of Applied Probability ◽

10.2307/3212966 ◽

1980 ◽

Vol 17 (3) ◽

pp. 726-734 ◽

Cited By ~ 20

Author(s):

Bharat Doshi ◽

Steven E. Shreve

Keyword(s):

Likelihood Function ◽

Transition Probabilities ◽

Control Law ◽

Likelihood Estimator ◽

Modified Maximum Likelihood ◽

Control Scheme ◽

Finite State ◽

Finite Set ◽

Stationary Control ◽

Finite State Space

A controlled Markov chain with finite state space has transition probabilities which depend on an unknown parameter α lying in a known finite set A. For each α, a stationary control law ϕ α is given. This paper develops a control scheme whereby at each stage t a parameter α t is chosen at random from among those parameters which nearly maximize the log likelihood function, and the control ut is chosen according to the control law ϕ αt. It is proved that this algorithm leads to identification of the true α under conditions weaker than any previously considered.

Download Full-text

On quasi-stationary distributions in absorbing continuous-time finite Markov chains

Journal of Applied Probability ◽

10.2307/3212311 ◽

1967 ◽

Vol 4 (1) ◽

pp. 192-196 ◽

Cited By ~ 126

Author(s):

J. N. Darroch ◽

E. Seneta

Keyword(s):

Markov Chains ◽

Discrete Time ◽

Continuous Time ◽

Present Note ◽

Time Parameter ◽

Stationary Distributions ◽

Finite State ◽

Absorbing Markov Chains ◽

Finite Markov Chains ◽

Finite State Space

In a recent paper, the authors have discussed the concept of quasi-stationary distributions for absorbing Markov chains having a finite state space, with the further restriction of discrete time. The purpose of the present note is to summarize the analogous results when the time parameter is continuous.

Download Full-text

Quasi-Statically Cooled Markov Chains

Probability in the Engineering and Informational Sciences ◽

10.1017/s0269964800003168 ◽

1994 ◽

Vol 8 (1) ◽

pp. 1-19 ◽

Cited By ~ 2

Author(s):

Madhav Desai ◽

Sunil Kumar ◽

P. R. Kumar

Keyword(s):

Markov Chains ◽

Transition Probabilities ◽

Graph Algorithm ◽

Steady State Distribution ◽

State Distribution ◽

Critical Rate ◽

Occupation Measures ◽

Finite State ◽

Precise Relationship ◽

Finite State Space

We consider time-inhomogeneous Markov chains on a finite state-space, whose transition probabilitiespij(t) = cijε(t)Vij are proportional to powers of a vanishing small parameter ε(t). We determine the precise relationship between this chain and the corresponding time-homogeneous chains pij= cijε(t)vij, as ε ↘ 0. Let {} be the steady-state distribution of this time-homogeneous chain. We characterize the orders {ηι} in = θ(εηι). We show that if ε(t) ↘ 0 slowly enough, then the timewise occupation measures βι := sup { q > 0 | Prob(x(t) = i) = + ∞}, called the recurrence orders, satisfy βi — βj = ηj — ηi. Moreover, : = { ηι|ηι = minj} is the set of ground states of the time-homogeneous chain, then x(t) → . in an appropriate sense, whenever η(t) is “cooled” slowly. We also show that there exists a critical ρ* such that x(t) → if and only if = + ∞. We characterize this critical rate as ρ* = max.min min max. Finally, we provide a graph algorithm for determining the orders [ηi] [βi] and the critical rate ρ*.

Download Full-text

ON THE EARTHQUAKE OCCURRENCES IN JAPAN AND THE SURROUNDING AREA VIA SEMI MARKOV MODELING

Bulletin of the Geological Society of Greece ◽

10.12681/bgsg.11866 ◽

2017 ◽

Vol 50 (3) ◽

pp. 1535

Author(s):

C. Panorias ◽

A. Papadopoulou ◽

T. Tsapanos

Keyword(s):

Markov Model ◽

Transition Probabilities ◽

Seismic Zone ◽

Markov Modeling ◽

Surrounding Area ◽

Large Magnitude ◽

Limiting State ◽

Finite State ◽

Limiting Behaviour ◽

Finite State Space

In the present paper, the earthquake occurrences in the area of Japan, are studied by a semi Markov model which is considered homogeneous in time. The data applied refer to earthquakes of large magnitude (Mw>6.0) during the period 1900-2012. We consider 9 seismic zones derived from the typical 11 zones for the area of Japan, due to the lack of data for 3 zones (9-th,10-th and 11-th). Also, we define 3 groups for the magnitudes, corresponding to 6-7,7.1-8 and M> 8.0. Thus, we consider for our semi Markov model a finite state space, S={ ( ,)j i ZR | i=1,...9, j=1,2,3}, where i Z defines the i-th seismic zone and j R states the j-th magnitude scale. We applied the data to describe the interval transition probabilities for the states and the model's limiting behaviour for which is sufficient an interval of time of seven years. The time unit of the model is considered to be one day. Some interesting results, concerning the interval transition probabilities and the limiting state vector, are derived.

Download Full-text

Why is Kemeny’s constant a constant?

Journal of Applied Probability ◽

10.1017/jpr.2018.68 ◽

2018 ◽

Vol 55 (4) ◽

pp. 1025-1036 ◽

Cited By ~ 7

Author(s):

Dario Bini ◽

Jeffrey J. Hunter ◽

Guy Latouche ◽

Beatrice Meini ◽

Peter Taylor

Keyword(s):

Markov Chains ◽

State Space ◽

Continuous Time ◽

Death Process ◽

Birth And Death Process ◽

Finite State ◽

Finite Markov Chains ◽

Infinite State ◽

Special Case ◽

Finite State Space

Abstract In their 1960 book on finite Markov chains, Kemeny and Snell established that a certain sum is invariant. The value of this sum has become known as Kemeny’s constant. Various proofs have been given over time, some more technical than others. We give here a very simple physical justification, which extends without a hitch to continuous-time Markov chains on a finite state space. For Markov chains with denumerably infinite state space, the constant may be infinite and even if it is finite, there is no guarantee that the physical argument will hold. We show that the physical interpretation does go through for the special case of a birth-and-death process with a finite value of Kemeny’s constant.

Download Full-text

On a characterization property of finite irreducible Markov chains

Journal of Applied Probability ◽

10.2307/3211955 ◽

1970 ◽

Vol 7 (3) ◽

pp. 771-775

Author(s):

I. V. Basawa

Keyword(s):

Transition Matrix ◽

Transition Probabilities ◽

Homogeneous Markov Chain ◽

Finite State ◽

One Step ◽

Characterization Property ◽

Step Transition ◽

The One ◽

Image Position ◽

Finite State Space

Let {Xk}, k = 1, 2, ··· be a sequence of random variables forming a homogeneous Markov chain on a finite state-space, S = {1, 2, ···, s}. Xk could be thought of as the state at time k of some physical system for which are the (one-step) transition probabilities. It is assumed that all the states are inter-communicating, so that the transition matrix P = ((pij)) is irreducible.

Download Full-text