Efficient computation of normalized maximum likelihood coding for Gaussian mixtures with its applications to optimal clustering

2011 ◽  
Author(s):  
So Hirai ◽  
Kenji Yamanishi
Keyword(s):  
Maximum Likelihood ◽  
Gaussian Mixtures ◽  
Efficient Computation ◽  
Normalized Maximum Likelihood

scholarly journals Word2vec Skip-Gram Dimensionality Selection via Sequential Normalized Maximum Likelihood

Entropy ◽  
2021 ◽  
Vol 23 (8) ◽  
pp. 997
Author(s):  
Pham Thuc Hung ◽  
Kenji Yamanishi
Keyword(s):  
Probability Theory ◽  
Maximum Likelihood ◽  
Bayesian Information Criterion ◽  
Minimum Description Length ◽  
Information Criterion ◽  
Information Criteria ◽  
Efficient Computation ◽  
Data Sequence ◽  
Word Similarity ◽  
Normalized Maximum Likelihood

In this paper, we propose a novel information criteria-based approach to select the dimensionality of the word2vec Skip-gram (SG). From the perspective of the probability theory, SG is considered as an implicit probability distribution estimation under the assumption that there exists a true contextual distribution among words. Therefore, we apply information criteria with the aim of selecting the best dimensionality so that the corresponding model can be as close as possible to the true distribution. We examine the following information criteria for the dimensionality selection problem: the Akaike’s Information Criterion (AIC), Bayesian Information Criterion (BIC), and Sequential Normalized Maximum Likelihood (SNML) criterion. SNML is the total codelength required for the sequential encoding of a data sequence on the basis of the minimum description length. The proposed approach is applied to both the original SG model and the SG Negative Sampling model to clarify the idea of using information criteria. Additionally, as the original SNML suffers from computational disadvantages, we introduce novel heuristics for its efficient computation. Moreover, we empirically demonstrate that SNML outperforms both BIC and AIC. In comparison with other evaluation methods for word embedding, the dimensionality selected by SNML is significantly closer to the optimal dimensionality obtained by word analogy or word similarity tasks.

scholarly journals A Note on the Applied Use of MDL Approximations

2004 ◽  
Vol 16 (9) ◽  
pp. 1763-1768 ◽  
Author(s):  
Daniel J. Navarro
Keyword(s):  
Maximum Likelihood ◽  
Fisher Information ◽  
Minimum Description Length ◽  
Geometric Interpretation ◽  
Applied Problem ◽  
Full Model ◽  
Large Samples ◽  
Psychological Models ◽  
Normalized Maximum Likelihood ◽  
Practical Implications

An applied problem is discussed in which two nested psychological models of retention are compared using minimum description length (MDL). The standard Fisher information approximation to the normalized maximum likelihood is calculated for these two models, with the result that the full model is assigned a smaller complexity, even for moderately large samples. A geometric interpretation for this behavior is considered, along with its practical implications.

Sign in / Sign up

Export Citation Format

Share Document