scholarly journals Topologically Ordered Feature Extraction Based on Sparse Group Restricted Boltzmann Machines

2015 ◽  
Vol 2015 ◽  
pp. 1-12
Author(s):  
Zhong Chen ◽  
Shengwu Xiong ◽  
Zhixiang Fang ◽  
Ruiling Zhang ◽  
Xiangzhen Kong ◽  
...  
Keyword(s):  
Likelihood Function ◽  
Feature Learning ◽  
Restricted Boltzmann Machines ◽  
Weight Decay ◽  
Boltzmann Machines ◽  
Learning Domains ◽  
Log Likelihood ◽  
New Type ◽  
Historical Periods ◽  
Group Weights

How to extract topologically ordered features efficiently from high-dimensional data is an important problem of unsupervised feature learning domains for deep learning. To address this problem, we propose a new type of regularization for Restricted Boltzmann Machines (RBMs). Adding two extra terms in the log-likelihood function to penalize the group weights and topologically ordered factors, this type of regularization extracts topologically ordered features based on sparse group Restricted Boltzmann Machines (SGRBMs). Therefore, it encourages an RBM to learn a much smoother probability distribution because its formulations turn out to be a combination of the group weight-decay and topologically ordered factor regularizations. We apply this proposed regularization scheme to image datasets of natural images and Flying Apsara images in the Dunhuang Grotto Murals at four different historical periods. The experimental results demonstrate that the combination of these two extra terms in the log-likelihood function helps to extract more discriminative features with much sparser and more aggregative hidden activation probabilities.

scholarly journals On the Training Algorithms for Restricted Boltzmann Machines

2019 ◽  
Author(s):  
Leandro Aparecido Passos ◽  
João Paulo Papa
Keyword(s):  
Feature Learning ◽  
Parameters Optimization ◽  
Fine Tuning ◽  
Stochastic Neural Networks ◽  
Heuristic Methods ◽  
Training Algorithms ◽  
Restricted Boltzmann Machines ◽  
Boltzmann Machines ◽  
Heuristic Techniques ◽  
Temperature Parameter

Deep learning techniques have been studied extensively in the last years due to their good results related to essential tasks on a large range of applications, such as speech and face recognition, as well as object classification. Restrict Boltzmann Machines (RBMs) are among the most employed techniques, which are energy-based stochastic neural networks composed of two layers of neurons whose objective is to estimate the connection weights between them. Recently, the scientific community spent much effort on sampling methods since the effectiveness of RBMs is directly related to the success of such a process. Thereby, this work contributes to studies concerning different training algorithms for RBMs, as well as its variants Deep Belief Networks and Deep Boltzmann Machines. Further, the work covers the application of meta-heuristic methods concerning a proper fine-tune of these techniques. Moreover, the validation of the model is presented in the context of image reconstruction and unsupervised feature learning. In general, we present different approaches to training these techniques, as well as the evaluation of meta-heuristic methods for fine-tuning parameters, and its main contributions are: (i) temperature parameter introduction in DBM formulation, (ii) DBM using adaptive temperature, (iii) DBM meta-parameter optimization through meta-heuristic techniques, and (iv) infinity Restricted Boltzmann Machine (iRBM) meta-parameters optimization through meta-heuristic techniques.

Bounding the Bias of Contrastive Divergence Learning

2011 ◽  
Vol 23 (3) ◽  
pp. 664-673 ◽  
Author(s):  
Asja Fischer ◽  
Christian Igel
Keyword(s):  
Gibbs Sampling ◽  
Upper Bound ◽  
Single Variable ◽  
Restricted Boltzmann Machines ◽  
Boltzmann Machines ◽  
Maximum Change ◽  
Log Likelihood ◽  
Contrastive Divergence ◽  
The Absolute

Optimization based on k-step contrastive divergence (CD) has become a common way to train restricted Boltzmann machines (RBMs). The k-step CD is a biased estimator of the log-likelihood gradient relying on Gibbs sampling. We derive a new upper bound for this bias. Its magnitude depends on k, the number of variables in the RBM, and the maximum change in energy that can be produced by changing a single variable. The last reflects the dependence on the absolute values of the RBM parameters. The magnitude of the bias is also affected by the distance in variation between the modeled distribution and the starting distribution of the Gibbs chain.

scholarly journals Adaptive hyperparameter updating for training restricted Boltzmann machines on quantum annealers

2021 ◽  
Vol 11 (1) ◽  
Author(s):  
Guanglei Xu ◽  
William S. Oates
Keyword(s):  
Neural Network ◽  
Maximum Likelihood ◽  
Image Reconstruction ◽  
Image Recognition ◽  
Shannon Entropy ◽  
Reconstruction Error ◽  
Likelihood Method ◽  
Restricted Boltzmann Machines ◽  
Boltzmann Machines ◽  
D Wave

AbstractRestricted Boltzmann Machines (RBMs) have been proposed for developing neural networks for a variety of unsupervised machine learning applications such as image recognition, drug discovery, and materials design. The Boltzmann probability distribution is used as a model to identify network parameters by optimizing the likelihood of predicting an output given hidden states trained on available data. Training such networks often requires sampling over a large probability space that must be approximated during gradient based optimization. Quantum annealing has been proposed as a means to search this space more efficiently which has been experimentally investigated on D-Wave hardware. D-Wave implementation requires selection of an effective inverse temperature or hyperparameter ($$\beta $$ β ) within the Boltzmann distribution which can strongly influence optimization. Here, we show how this parameter can be estimated as a hyperparameter applied to D-Wave hardware during neural network training by maximizing the likelihood or minimizing the Shannon entropy. We find both methods improve training RBMs based upon D-Wave hardware experimental validation on an image recognition problem. Neural network image reconstruction errors are evaluated using Bayesian uncertainty analysis which illustrate more than an order magnitude lower image reconstruction error using the maximum likelihood over manually optimizing the hyperparameter. The maximum likelihood method is also shown to out-perform minimizing the Shannon entropy for image reconstruction.

scholarly journals Evaluating the Observed Log-Likelihood Function in Two-Level Structural Equation Modeling with Missing Data: From Formulas to R Code

Psych ◽  
2021 ◽  
Vol 3 (2) ◽  
pp. 197-232
Author(s):  
Yves Rosseel
Keyword(s):  
Structural Equation ◽  
Structural Equation Models ◽  
Likelihood Function ◽  
Likelihood Estimation ◽  
Missing At Random ◽  
Equation Modeling ◽  
Computationally Efficient ◽  
Log Likelihood ◽  
Efficient Expression ◽  
Special Case

This paper discusses maximum likelihood estimation for two-level structural equation models when data are missing at random at both levels. Building on existing literature, a computationally efficient expression is derived to evaluate the observed log-likelihood. Unlike previous work, the expression is valid for the special case where the model implied variance–covariance matrix at the between level is singular. Next, the log-likelihood function is translated to R code. A sequence of R scripts is presented, starting from a naive implementation and ending at the final implementation as found in the lavaan package. Along the way, various computational tips and tricks are given.

Sign in / Sign up

Export Citation Format

Share Document