Estimation of the derivative of a stationary measure with respect to a control parameter

The paper deals with a problem which arises in the Monte Carlo optimization of steady state or ergodic systems which can be modelled by Markov chains. The transition probability depends on a parameter, and one wishes to find the parameter value at which some performance function is minimum. The only available data are obtained from either simulation or actual operating information. For such a problem one needs good statistical estimates of the derivatives. Conditions are given for the existence of the derivative of the stationary measure with respect to the parameter, in the sense that the derivative is a signed measure, and is the limit of the natural approximating sequence. Some properties and a useful characterization of the derivative are obtained. It is also shown that, under appropriate conditions, the derivative of the n-step transition function converges to the derivative of the stationary measure as n tends to ∞. This latter result is of particular importance whether one is simply estimating or is actually optimizing via some sequential Monte Carlo procedure, since the basic observations are always taken over a finite time interval.