On the curved exponential family in the Stochatic Approximation Expectation Maximization Algorithm

The Expectation-Maximization Algorithm (EM) is a widely used method allowing to estimate the maximum likelihood of models involving latent variables. When the Expectation step cannot be computed easily, one can use stochastic versions of the EM such as the Stochastic Approximation EM. This algorithm, however, has the drawback to require the joint likelihood to belong to the curved exponential family. To overcome this problem, \cite{kuhn2005maximum} introduced a rewriting of the model which ``exponentializes'' it by considering the parameter as an additional latent variable following a Normal distribution centered on the newly defined parameters and with fixed variance. The likelihood of this new exponentialized model now belongs to the curved exponential family. Although often used, there is no guarantee that the estimated mean is close to the maximum likelihood estimate of the initial model. In this paper, we quantify the error done in this estimation while considering the exponentialized model instead of the initial one. By verifying those results on an example, we see that a trade-off must be made between the speed of convergence and the tolerated error. Finally, we propose a new algorithm allowing a better estimation of the parameter in a reasonable computation time to reduce the bias.

Forward-reverse expectation-maximization algorithm for Markov chains: convergence and numerical analysis

We develop a forward-reverse expectation-maximization (FREM) algorithm for estimating parameters of a discrete-time Markov chain evolving through a certain measurable state-space. For the construction of the FREM method, we develop forward-reverse representations for Markov chains conditioned on a certain terminal state. We prove almost sure convergence of our algorithm for a Markov chain model with curved exponential family structure. On the numerical side, we carry out a complexity analysis of the forward-reverse algorithm by deriving its expected cost. Two application examples are discussed.

Ancillarity and the Limited Information Maximum-Likelihood Estimation of a Structural Equation in a Simultaneous Equation System

The concepts of the curved exponential family of distributions and ancillarity are applied to estimation problems of a single structural equation in a simultaneous equation model, and the effect of conditioning on ancillary statistics on the limited information maximum-likelihood (LIML) estimator is investigated. The asymptotic conditional covariance matrix of the LIML estimator conditioned on the second-order asymptotic maximal ancillary statistic is shown to be efficiently estimated by Liu and Breen's formula. The effect of conditioning on a second-order asymptotic ancillary statistic, i.e., the smallest characteristic root associated with the LIML estimation, is analyzed by means of an asymptotic expansion of the distribution as well as the exact distribution. The smallest root helps to give an intuitively appealing measure of precision of the LIML estimator.