Learning Algorithms for Neural Networks Based on Quasi-Newton Methods With Self-Scaling

1993 ◽  
Vol 115 (1) ◽  
pp. 38-43 ◽  
Author(s):  
H. S. M. Beigi ◽  
C. J. Li
Keyword(s):  
Neural Networks ◽  
State Of The Art ◽  
Learning Algorithms ◽  
Feedforward Neural Networks ◽  
Optimization Techniques ◽  
Newton Methods ◽  
Convergence Properties ◽  
Variable Metric ◽  
Quasi Newton ◽  
Number Of Iterations

Previous studies have suggested that, for moderate sized neural networks, the use of classical Quasi-Newton methods yields the best convergence properties among all the state-of-the-art [1]. This paper describes a set of even better learning algorithms based on a class of Quasi-Newton optimization techniques called Self-Scaling Variable Metric (SSVM) methods. One of the characteristics of SSVM methods is that they provide a set of search directions which are invariant under the scaling of the objective function. With an XOR benchmark and an encoder benchmark, simulations using the SSVM algorithms for the learning of general feedforward neural networks were carried out to study their performance. Compared to classical Quasi-Newton methods, it is shown that the SSVM method reduces the number of iterations required for convergence by 40 percent to 60 percent that of the classical Quasi-Newton methods which, in general, converge two to three orders of magnitude faster than the steepest descent techniques.

scholarly journals Active Learning for Node Classification: An Evaluation

Entropy ◽  
2020 ◽  
Vol 22 (10) ◽  
pp. 1164
Author(s):  
Kaushalya Madhawa ◽  
Tsuyoshi Murata
Keyword(s):  
Neural Networks ◽  
Active Learning ◽  
State Of The Art ◽  
Learning Algorithms ◽  
Classification Performance ◽  
Data Types ◽  
Neural Network Models ◽  
Attributed Graph ◽  
Attributed Graphs ◽  
Graph Neural Networks

Current breakthroughs in the field of machine learning are fueled by the deployment of deep neural network models. Deep neural networks models are notorious for their dependence on large amounts of labeled data for training them. Active learning is being used as a solution to train classification models with less labeled instances by selecting only the most informative instances for labeling. This is especially important when the labeled data are scarce or the labeling process is expensive. In this paper, we study the application of active learning on attributed graphs. In this setting, the data instances are represented as nodes of an attributed graph. Graph neural networks achieve the current state-of-the-art classification performance on attributed graphs. The performance of graph neural networks relies on the careful tuning of their hyperparameters, usually performed using a validation set, an additional set of labeled instances. In label scarce problems, it is realistic to use all labeled instances for training the model. In this setting, we perform a fair comparison of the existing active learning algorithms proposed for graph neural networks as well as other data types such as images and text. With empirical results, we demonstrate that state-of-the-art active learning algorithms designed for other data types do not perform well on graph-structured data. We study the problem within the framework of the exploration-vs.-exploitation trade-off and propose a new count-based exploration term. With empirical evidence on multiple benchmark graphs, we highlight the importance of complementing uncertainty-based active learning models with an exploration term.

scholarly journals Detecting Emotions in English and Arabic Tweets

Information ◽  
2019 ◽  
Vol 10 (3) ◽  
pp. 98 ◽  
Author(s):  
Tariq Ahmad ◽  
Allan Ramsay ◽  
Hanady Ahmed
Keyword(s):  
Machine Learning ◽  
Neural Networks ◽  
Deep Neural Networks ◽  
State Of The Art ◽  
Learning Algorithms ◽  
General Purpose ◽  
Machine Learning Algorithms ◽  
Current State ◽  
Optimal Thresholds ◽  
Alternative Approach

Assigning sentiment labels to documents is, at first sight, a standard multi-label classification task. Many approaches have been used for this task, but the current state-of-the-art solutions use deep neural networks (DNNs). As such, it seems likely that standard machine learning algorithms, such as these, will provide an effective approach. We describe an alternative approach, involving the use of probabilities to construct a weighted lexicon of sentiment terms, then modifying the lexicon and calculating optimal thresholds for each class. We show that this approach outperforms the use of DNNs and other standard algorithms. We believe that DNNs are not a universal panacea and that paying attention to the nature of the data that you are trying to learn from can be more important than trying out ever more powerful general purpose machine learning algorithms.

scholarly journals Empirical Investigation of Optimization Algorithms in Neural Machine Translation

2017 ◽  
Vol 108 (1) ◽  
pp. 13-25 ◽  
Author(s):  
Parnia Bahar ◽  
Tamer Alkhouli ◽  
Jan-Thorsten Peter ◽  
Christopher Jan-Steffen Brix ◽  
Hermann Ney
Keyword(s):  
Neural Networks ◽  
Machine Translation ◽  
Optimization Problem ◽  
Empirical Investigation ◽  
State Of The Art ◽  
Optimization Techniques ◽  
Neural Machine Translation ◽  
Translation Quality ◽  
And Training ◽  
Dimensional Optimization

AbstractTraining neural networks is a non-convex and a high-dimensional optimization problem. In this paper, we provide a comparative study of the most popular stochastic optimization techniques used to train neural networks. We evaluate the methods in terms of convergence speed, translation quality, and training stability. In addition, we investigate combinations that seek to improve optimization in terms of these aspects. We train state-of-the-art attention-based models and apply them to perform neural machine translation. We demonstrate our results on two tasks: WMT 2016 En→Ro and WMT 2015 De→En.

scholarly journals Investigating Deep Feedforward Neural Networks for Classification of Transposon-Derived piRNAs

2020 ◽  
Author(s):  
Alisson Hayasi da Costa ◽  
Renato Augusto C. dos Santos ◽  
Ricardo Cerri
Keyword(s):  
Neural Networks ◽  
Deep Learning ◽  
State Of The Art ◽  
Feedforward Neural Networks ◽  
The State ◽  
Machine Learning Algorithms ◽  
Support Vector ◽  
Advantages And Disadvantages ◽  
Large Application

AbstractPIWI-Interacting RNAs (piRNAs) form an important class of non-coding RNAs that play a key role in the genome integrity through the silencing of transposable elements. However, despite their importance and the large application of deep learning in computational biology for classification tasks, there are few studies of deep learning and neural networks for piRNAs prediction. Therefore, this paper presents an investigation on deep feedforward networks models for classification of transposon-derived piRNAs. We analyze and compare the results of the neural networks in different hyperparameters choices, such as number of layers, activation functions and optimizers, clarifying the advantages and disadvantages of each configuration. From this analysis, we propose a model for human piRNAs classification and compare our method with the state-of-the-art deep neural network for piRNA prediction in the literature and also traditional machine learning algorithms, such as Support Vector Machines and Random Forests, showing that our model has achieved a great performance with an F-measure value of 0.872, outperforming the state-of-the-art method in the literature.

Recent Advances in Gradient Based Unconstrained Optimization Techniques for Large Problems

1983 ◽  
Vol 105 (2) ◽  
pp. 155-159 ◽  
Author(s):  
D. F. Shanno
Keyword(s):  
Gradient Methods ◽  
Optimization Techniques ◽  
Newton Methods ◽  
Variable Metric Methods ◽  
Distributed Software ◽  
Variable Metric ◽  
Gradient Based ◽  
Truncated Newton Methods ◽  
Truncated Newton ◽  
Theoretical Results

The paper surveys recent results in conjugate gradient methods, variable storage variable metric methods, sparse variable metric and finite difference Newton methods, and truncated Newton methods. Both computational and theoretical results will be discussed, as well as currently distributed software.

scholarly journals Functional connectome fingerprinting using shallow feedforward neural networks

2021 ◽  
Vol 118 (15) ◽  
pp. e2021852118
Author(s):  
Gokce Sarar ◽  
Bhaskar Rao ◽  
Thomas Liu
Keyword(s):  
Neural Networks ◽  
Recurrent Neural Networks ◽  
Resting State ◽  
State Of The Art ◽  
Feedforward Neural Networks ◽  
High Accuracy ◽  
Correlation Matrices ◽  
Functional Connectome ◽  
Temporal Features ◽  
Data Points

Although individual subjects can be identified with high accuracy using correlation matrices computed from resting-state functional MRI (rsfMRI) data, the performance significantly degrades as the scan duration is decreased. Recurrent neural networks can achieve high accuracy with short-duration (72 s) data segments but are designed to use temporal features not present in the correlation matrices. Here we show that shallow feedforward neural networks that rely solely on the information in rsfMRI correlation matrices can achieve state-of-the-art identification accuracies (≥99.5%) with data segments as short as 20 s and across a range of input data size combinations when the total number of data points (number of regions × number of time points) is on the order of 10,000.

Sign in / Sign up

Export Citation Format

Share Document