scholarly journals Non‐linear activation function approximation using a REMEZ algorithm

2021 ◽  
Author(s):  
Samba Raju Chiluveru ◽  
Manoj Tripathy ◽  
Bibhudutta
Keyword(s):  
Function Approximation ◽  
Activation Function ◽  
Remez Algorithm ◽  
Non Linear

scholarly journals A Non-Polynomial, Non-Sigmoidal, Bounded and Symmetric Activation Function for Feed – Forward Artificial Neural Networks

2019 ◽  
Vol 8 (12) ◽  
pp. 405-410
Keyword(s):  
Neural Networks ◽  
Artificial Neural Networks ◽  
Function Approximation ◽  
Activation Function ◽  
Continuous Functions ◽  
Feed Forward ◽  
Learning Tasks ◽  
Non Linear ◽  
Artificial Neural ◽  
Hidden Layer

Feed-forward artificial neural networks are universal approximators of continuous functions. This property enables the use of these networks to solve learning tasks. Learning tasks in this paradigm are cast as function approximation problems. The universal approximation results for these networks require at least one hidden layer with non-linear nodes, and also require that the non-linearities be non-polynomial in nature. In this paper a non-polynomial and non-sigmoidal non-linear function is proposed as a suitable activation function for these networks. The usefulness of the proposed activation function is shown on 12 function approximation task. The obtained results demonstrate that the proposed activation function outperforms the logistic / log-sigmoid and the hyperbolic tangent activation functions.

Analysis of Non-Linear Activation Functions for Classification Tasks Using Convolutional Neural Networks

2019 ◽  
Vol 12 (3) ◽  
pp. 156-161 ◽  
Author(s):  
Aman Dureja ◽  
Payal Pahwa
Keyword(s):  
Neural Networks ◽  
Deep Neural Networks ◽  
Activation Function ◽  
Primary Objective ◽  
Experimental Comparison ◽  
Activation Functions ◽  
Practical Applications ◽  
Network Activation ◽  
Non Linear ◽  
Hidden Layer

Background: In making the deep neural network, activation functions play an important role. But the choice of activation functions also affects the network in term of optimization and to retrieve the better results. Several activation functions have been introduced in machine learning for many practical applications. But which activation function should use at hidden layer of deep neural networks was not identified. Objective: The primary objective of this analysis was to describe which activation function must be used at hidden layers for deep neural networks to solve complex non-linear problems. Methods: The configuration for this comparative model was used by using the datasets of 2 classes (Cat/Dog). The number of Convolutional layer used in this network was 3 and the pooling layer was also introduced after each layer of CNN layer. The total of the dataset was divided into the two parts. The first 8000 images were mainly used for training the network and the next 2000 images were used for testing the network. Results: The experimental comparison was done by analyzing the network by taking different activation functions on each layer of CNN network. The validation error and accuracy on Cat/Dog dataset were analyzed using activation functions (ReLU, Tanh, Selu, PRelu, Elu) at number of hidden layers. Overall the Relu gave best performance with the validation loss at 25th Epoch 0.3912 and validation accuracy at 25th Epoch 0.8320. Conclusion: It is found that a CNN model with ReLU hidden layers (3 hidden layers here) gives best results and improve overall performance better in term of accuracy and speed. These advantages of ReLU in CNN at number of hidden layers are helpful to effectively and fast retrieval of images from the databases.

scholarly journals On sharpness of error bounds for multivariate neural network approximation

2020 ◽  
Author(s):  
Steffen Goebbels
Keyword(s):  
Error Bounds ◽  
Uniform Boundedness ◽  
Feedforward Neural Networks ◽  
Activation Function ◽  
Multivariate Functions ◽  
Non Linear ◽  
Uniform Boundedness Principle ◽  
Ridge Functions ◽  
Hidden Layer ◽  
Polynomial Activation Functions

AbstractSingle hidden layer feedforward neural networks can represent multivariate functions that are sums of ridge functions. These ridge functions are defined via an activation function and customizable weights. The paper deals with best non-linear approximation by such sums of ridge functions. Error bounds are presented in terms of moduli of smoothness. The main focus, however, is to prove that the bounds are best possible. To this end, counterexamples are constructed with a non-linear, quantitative extension of the uniform boundedness principle. They show sharpness with respect to Lipschitz classes for the logistic activation function and for certain piecewise polynomial activation functions. The paper is based on univariate results in Goebbels (Res Math 75(3):1–35, 2020. https://rdcu.be/b5mKH)

On the Remez algorithm for non-linear families

1970 ◽  
Vol 15 (5) ◽  
pp. 382-391 ◽  
Author(s):  
Richard B. Barrar ◽  
Henry L. Loeb
Keyword(s):  
Remez Algorithm ◽  
Non Linear
Sign in / Sign up

Export Citation Format

Share Document