Optimal decision procedures for finite Markov chains. Part III: General convex systems

This paper is concerned with the general problem of finding an optimal transition matrix for a finite Markov chain, where the probabilities for each transition must be chosen from a given convex family of distributions. The immediate cost is determined by this choice, but it is required to minimise the average expected cost in the long run. The problem is investigated by classifying the states according to the accessibility relations between them. If an optimal policy exists, it can be found by considering the convex subsystems associated with the states at different levels in the classification scheme.

Download Full-text

Addenda to processes defined on a finite Markov chain

Mathematical Proceedings of the Cambridge Philosophical Society ◽

10.1017/s0305004100041062 ◽

1967 ◽

Vol 63 (1) ◽

pp. 187-193 ◽

Cited By ~ 15

Author(s):

Julian Keilson ◽

David M. G. Wishart

Keyword(s):

Distribution Function ◽

Markov Chain ◽

Transition Matrix ◽

Finite Markov Chain ◽

Marginal Process ◽

Stochastic Transition ◽

Matrix Distribution ◽

Finite Markov Chains ◽

Image Position ◽

Irreducible Markov Chain

Introduction. Two previous papers (1,2) have dealt with additive processes defined on finite Markov chains. Such a process in discrete time may be treated as a bivariate Markov process {R(k), X(k)}. The process R(k) is an irreducible Markov chain on states r = 1, 2, …, R governed by a stochastic transition matrix B0 with components brs. The marginal process X(k) ‘defined’ on the chain R(k) is a sum of random increments ξ(i) dependent on the chain, i.e. if the ith transition takes the chain from state r to state s, ξ,(i) is chosen from a distribution function Drs(x) indexed by r and s. The distribution of the bivariate process may be represented by a vector F(x, k) with componentsThese are generated recursively by the relationwhere the increment matrix distribution B(x) has components brsDrs(x). We denote the nth moment of B(x) by Bn = ∫xndB(x), so that B0 = B(∞).

Download Full-text

Estimating the Entropy Rate of Finite Markov Chains With Application to Behavior Studies

Journal of Educational and Behavioral Statistics ◽

10.3102/1076998618822540 ◽

2019 ◽

Vol 44 (3) ◽

pp. 282-308 ◽

Cited By ~ 8

Author(s):

Brian G. Vegetabile ◽

Stephanie A. Stout-Oswald ◽

Elysia Poggi Davis ◽

Tallie Z. Baram ◽

Hal S. Stern

Keyword(s):

Markov Chain ◽

Markov Chains ◽

Markov Processes ◽

Simulation Study ◽

Transition Matrix ◽

Sliding Window ◽

Entropy Rate ◽

Homogeneous Markov Chain ◽

The Third ◽

Finite Markov Chains

Predictability of behavior is an important characteristic in many fields including biology, medicine, marketing, and education. When a sequence of actions performed by an individual can be modeled as a stationary time-homogeneous Markov chain the predictability of the individual’s behavior can be quantified by the entropy rate of the process. This article compares three estimators of the entropy rate of finite Markov processes. The first two methods directly estimate the entropy rate through estimates of the transition matrix and stationary distribution of the process. The third method is related to the sliding-window Lempel–Ziv compression algorithm. The methods are compared via a simulation study and in the context of a study of interactions between mothers and their children.

Download Full-text

Normal distributions of finite Markov chains

International Journal of Algebra and Computation ◽

10.1142/s0218196719500577 ◽

2019 ◽

Vol 29 (08) ◽

pp. 1431-1449

Author(s):

John Rhodes ◽

Anne Schilling

Keyword(s):

Markov Chain ◽

Markov Chains ◽

Stationary Distribution ◽

Planar Graphs ◽

Finite Semigroup ◽

Finite Markov Chain ◽

Normal Distributions ◽

Straight Line ◽

Finite Markov Chains ◽

The Right

We show that the stationary distribution of a finite Markov chain can be expressed as the sum of certain normal distributions. These normal distributions are associated to planar graphs consisting of a straight line with attached loops. The loops touch only at one vertex either of the straight line or of another attached loop. Our analysis is based on our previous work, which derives the stationary distribution of a finite Markov chain using semaphore codes on the Karnofsky–Rhodes and McCammond expansion of the right Cayley graph of the finite semigroup underlying the Markov chain.

Download Full-text

Optimal decision procedures for finite Markov chains. Part II: Communicating systems

Advances in Applied Probability ◽

10.2307/1425832 ◽

1973 ◽

Vol 5 (3) ◽

pp. 521-540 ◽

Cited By ~ 54

Author(s):

John Bather

Keyword(s):

Markov Process ◽

General Solution ◽

Transition Probabilities ◽

Optimal Decision ◽

Positive Probability ◽

Finite State ◽

Finite Markov Chains ◽

Special Case ◽

Family Of Distributions ◽

Finite State Space

A Markov process in discrete time with a finite state space is controlled by choosing the transition probabilities from a given convex family of distributions depending on the present state. The immediate cost is prescribed for each choice and it is required to minimise the average expected cost over an infinite future. The paper considers a special case of this general problem and provides the foundation for a general solution. The main result is that an optimal policy exists if each state of the system can be reached with positive probability from any other state by choosing a suitable policy.

Download Full-text

Convergence Analysis of Surrogate Assisted (1+1) Evolutionary Algorithm

Applied Mechanics and Materials ◽

10.4028/www.scientific.net/amm.263-266.2339 ◽

2012 ◽

Vol 263-266 ◽

pp. 2339-2343

Author(s):

Ying Ming Jin

Keyword(s):

Markov Chain ◽

Evolutionary Algorithm ◽

Convergence Analysis ◽

Perturbation Analysis ◽

Surrogate Model ◽

Transition Matrix ◽

Matrix Perturbation ◽

Finite Markov Chain ◽

Matrix Perturbation Analysis

This paper analyzes the convergence deviation of surrogate assisted (1+1)EA. A model of surrogate assisted (1+1)EA can be built by the finite markov chain, then we got the transition matrix of this algorithm. The deviation of surrogate model can be expressed by the perturbation of transition matrix. So we can estimate the convergence deviation with the method of matrix perturbation analysis. Analyzing of the convergence changes brought by surrogate model’s deviations can help us to have a better select of the surrogate model.

Download Full-text

XXIV.—Conditions for the Ergodicity of Non-homogeneous Finite Markov Chains

Proceedings of the Royal Society of Edinburgh Section A Mathematical and Physical Sciences ◽

10.1017/s0080454100007585 ◽

1957 ◽

Vol 64 (4) ◽

pp. 369-380 ◽

Cited By ~ 3

Author(s):

J. L. Mott

Keyword(s):

Markov Chain ◽

Markov Chains ◽

Asymptotic Behaviour ◽

Transition Probabilities ◽

Finite Markov Chain ◽

Finite Markov Chains ◽

Product Of Matrices

SynopsisIn this note we study the asymptotic behaviour of a product of matrices where Pj is a matrix of transition probabilities in a non-homogeneous finite Markov chain. We give conditions that (i) the rows of P(n) tend to identity and that (ii) P(n) tends to a limit matrix with identical rows.

Download Full-text

A Matrix Factorization Problem in the Theory of Random Variables Defined on a Finite Markov Chain

Mathematical Proceedings of the Cambridge Philosophical Society ◽

10.1017/s0305004100036495 ◽

1962 ◽

Vol 58 (2) ◽

pp. 268-285 ◽

Cited By ~ 39

Author(s):

H. D. Miller

Keyword(s):

Markov Chain ◽

Matrix Factorization ◽

Transition Matrix ◽

Random Variables ◽

Finite Markov Chain ◽

Stieltjes Transform ◽

Factorization Problem ◽

Regular Elements ◽

The Matrix ◽

Image Position

SummaryLet {kr} (r = 0, 1, 2, …; 1 ≤ kr ≤ h) be a positively regular, finite Markov chain with transition matrix P = (pjk). For each possible transition j → k let gjk(x)(− ∞ ≤ x ≤ ∞) be a given distribution function. The sequence of random variables {ξr} is defined where ξr has the distribution gjk(x) if the rth transition takes the chain from state j to state k. It is supposed that each distribution gjk(x) admits a two-sided Laplace-Stieltjes transform in a real t-interval surrounding t = 0. Let P(t) denote the matrix {Pjkmjk(t)}. It is shown, using probability arguments, that I − sP(t) admits a Wiener-Hopf type of factorization in two ways for suitable values of s where the plus-factors are non-singular, bounded and have regular elements in a right half of the complex t-plane and the minus-factors have similar properties in an overlapping left half-plane (Theorem 1).

Download Full-text

Some passage-time generating functions for discrete-time and continuous-time finite Markov chains

Journal of Applied Probability ◽

10.1017/s002190020002550x ◽

1967 ◽

Vol 4 (03) ◽

pp. 496-507 ◽

Cited By ~ 5

Author(s):

J. N. Darroch ◽

K. W. Morris

Keyword(s):

Markov Chain ◽

Markov Chains ◽

Discrete Time ◽

Generating Functions ◽

Continuous Time ◽

Passage Time ◽

Finite Markov Chain ◽

Finite Markov Chains

Let T denote a subset of the possible transitions between the states of a finite Markov chain and let Yk denote the time of the kth occurrence of a T-transition. Formulae are derived for the generating functions of Yk , of Yj + k — Yj and of Yj + k — Yj in the limit as j → ∞, for both discrete-time and continuoustime chains. Several particular cases are briefly discussed.

Download Full-text

Weak lumpability in finite Markov chains

Journal of Applied Probability ◽

10.1017/s0021900200037190 ◽

1982 ◽

Vol 19 (03) ◽

pp. 685-691 ◽

Cited By ~ 4

Author(s):

Atef M. Abdel-moneim ◽

Frederick W. Leysieffer

Keyword(s):

Markov Chain ◽

Markov Chains ◽

State Space ◽

Transition Probability ◽

Linear Equations ◽

Transition Probability Matrix ◽

Finite Markov Chain ◽

Systems Of Linear Equations ◽

Finite Markov Chains ◽

The Given

Criteria are given to determine whether a given finite Markov chain can be lumped weakly with respect to a given partition of its state space. These conditions are given in terms of solution classes of systems of linear equations associated with the transition probability matrix of the Markov chain and the given partition.

Download Full-text