Variational Bayes Estimation of Mixing Coefficients

2005 ◽  
pp. 281-295 ◽  
Author(s):  
Bo Wang ◽  
D. M. Titterington
Keyword(s):  
Variational Bayes ◽  
Bayes Estimation ◽  
Mixing Coefficients

Variational Bayes Solution of Linear Neural Networks and Its Generalization Performance

2007 ◽  
Vol 19 (4) ◽  
pp. 1112-1153 ◽  
Author(s):  
Shinichi Nakajima ◽  
Sumio Watanabe
Keyword(s):  
Neural Networks ◽  
Free Energy ◽  
Variational Bayes ◽  
Bayes Estimation ◽  
Generalization Error ◽  
Positive Part ◽  
Ml Estimation ◽  
Generalization Performance ◽  
Training Error ◽  
And Training

It is well known that in unidentifiable models, the Bayes estimation provides much better generalization performance than the maximum likelihood (ML) estimation. However, its accurate approximation by Markov chain Monte Carlo methods requires huge computational costs. As an alternative, a tractable approximation method, called the variational Bayes (VB) approach, has recently been proposed and has been attracting attention. Its advantage over the expectation maximization (EM) algorithm, often used for realizing the ML estimation, has been experimentally shown in many applications; nevertheless, it has not yet been theoretically shown. In this letter, through analysis of the simplest unidentifiable models, we theoretically show some properties of the VB approach. We first prove that in three-layer linear neural networks, the VB approach is asymptotically equivalent to a positive-part James-Stein type shrinkage estimation. Then we theoretically clarify its free energy, generalization error, and training error. Comparing them with those of the ML estimation and the Bayes estimation, we discuss the advantage of the VB approach. We also show that unlike in the Bayes estimation, the free energy and the generalization error are less simply related with each other and that in typical cases, the VB free energy well approximates the Bayes one, while the VB generalization error significantly differs from the Bayes one.

Pattern Theory

2006 ◽  
Author(s):  
Ulf Grenander ◽  
Michael I. Miller
Keyword(s):  
Computational Linguistics ◽  
Parametric Representation ◽  
Bayes Estimation ◽  
Geometric Transformation ◽  
Central Component ◽  
Pattern Theory ◽  
Infinite Dimensional ◽  
Engineering Mathematics ◽  
Dimensional Shape ◽  
The Continuum

Pattern Theory provides a comprehensive and accessible overview of the modern challenges in signal, data, and pattern analysis in speech recognition, computational linguistics, image analysis and computer vision. Aimed at graduate students in biomedical engineering, mathematics, computer science, and electrical engineering with a good background in mathematics and probability, the text includes numerous exercises and an extensive bibliography. Additional resources including extended proofs, selected solutions and examples are available on a companion website. The book commences with a short overview of pattern theory and the basics of statistics and estimation theory. Chapters 3-6 discuss the role of representation of patterns via condition structure. Chapters 7 and 8 examine the second central component of pattern theory: groups of geometric transformation applied to the representation of geometric objects. Chapter 9 moves into probabilistic structures in the continuum, studying random processes and random fields indexed over subsets of Rn. Chapters 10 and 11 continue with transformations and patterns indexed over the continuum. Chapters 12-14 extend from the pure representations of shapes to the Bayes estimation of shapes and their parametric representation. Chapters 15 and 16 study the estimation of infinite dimensional shape in the newly emergent field of Computational Anatomy. Finally, Chapters 17 and 18 look at inference, exploring random sampling approaches for estimation of model order and parametric representing of shapes.

Sign in / Sign up

Export Citation Format

Share Document