scholarly journals Lp and almost sure rates of convergence of averaged stochastic gradient algorithms: locally strongly convex objective

2019 ◽  
Vol 23 ◽  
pp. 841-873 ◽  
Author(s):  
Antoine Godichon-Baggioni
Keyword(s):  
Big Data ◽  
Rates Of Convergence ◽  
Stochastic Gradient ◽  
Gradient Estimates ◽  
High Dimensional ◽  
Strongly Convex ◽  
Large Samples ◽  
Gradient Algorithms ◽  
Locally Strongly Convex ◽  
Usual Problem

An usual problem in statistics consists in estimating the minimizer of a convex function. When we have to deal with large samples taking values in high dimensional spaces, stochastic gradient algorithms and their averaged versions are efficient candidates. Indeed, (1) they do not need too much computational efforts, (2) they do not need to store all the data, which is crucial when we deal with big data, (3) they allow to simply update the estimates, which is important when data arrive sequentially. The aim of this work is to give asymptotic and non asymptotic rates of convergence of stochastic gradient estimates as well as of their averaged versions when the function we would like to minimize is only locally strongly convex.

scholarly journals A Continuous-Time Analysis of Distributed Stochastic Gradient

2020 ◽  
Vol 32 (1) ◽  
pp. 36-96 ◽  
Author(s):  
Nicholas M. Boffi ◽  
Jean-Jacques E. Slotine
Keyword(s):  
Quorum Sensing ◽  
Stochastic Gradient ◽  
Global Minimizer ◽  
Data Set ◽  
Distributed Agents ◽  
Strongly Convex ◽  
Gradient Algorithms ◽  
Second Order In Time ◽  
The Individual ◽  
Common Signal

We analyze the effect of synchronization on distributed stochastic gradient algorithms. By exploiting an analogy with dynamical models of biological quorum sensing, where synchronization between agents is induced through communication with a common signal, we quantify how synchronization can significantly reduce the magnitude of the noise felt by the individual distributed agents and their spatial mean. This noise reduction is in turn associated with a reduction in the smoothing of the loss function imposed by the stochastic gradient approximation. Through simulations on model nonconvex objectives, we demonstrate that coupling can stabilize higher noise levels and improve convergence. We provide a convergence analysis for strongly convex functions by deriving a bound on the expected deviation of the spatial mean of the agents from the global minimizer for an algorithm based on quorum sensing, the same algorithm with momentum, and the elastic averaging SGD (EASGD) algorithm. We discuss extensions to new algorithms that allow each agent to broadcast its current measure of success and shape the collective computation accordingly. We supplement our theoretical analysis with numerical experiments on convolutional neural networks trained on the CIFAR-10 data set, where we note a surprising regularizing property of EASGD even when applied to the non-distributed case. This observation suggests alternative second-order in time algorithms for nondistributed optimization that are competitive with momentum methods.

scholarly journals On the rates of convergence of parallelized averaged stochastic gradient algorithms

Statistics ◽  
2020 ◽  
Vol 54 (3) ◽  
pp. 618-635
Author(s):  
Antoine Godichon-Baggioni ◽  
Sofiane Saadane
Keyword(s):  
Rates Of Convergence ◽  
Stochastic Gradient ◽  
Gradient Algorithms

scholarly journals Opportunities and Challenges of Feature Selection Methods for High Dimensional Data: A Review

2021 ◽  
Vol 26 (1) ◽  
pp. 67-77
Author(s):  
Siva Sankari Subbiah ◽  
Jayakumar Chinnappan
Keyword(s):  
Feature Selection ◽  
Big Data ◽  
Large Scale ◽  
High Dimensional Data ◽  
Research Work ◽  
Basic Feature ◽  
High Dimensional ◽  
Selection Methods ◽  
Fast Development ◽  
Improved Accuracy

Now a day, all the organizations collecting huge volume of data without knowing its usefulness. The fast development of Internet helps the organizations to capture data in many different formats through Internet of Things (IoT), social media and from other disparate sources. The dimension of the dataset increases day by day at an extraordinary rate resulting in large scale dataset with high dimensionality. The present paper reviews the opportunities and challenges of feature selection for processing the high dimensional data with reduced complexity and improved accuracy. In the modern big data world the feature selection has a significance in reducing the dimensionality and overfitting of the learning process. Many feature selection methods have been proposed by researchers for obtaining more relevant features especially from the big datasets that helps to provide accurate learning results without degradation in performance. This paper discusses the importance of feature selection, basic feature selection approaches, centralized and distributed big data processing using Hadoop and Spark, challenges of feature selection and provides the summary of the related research work done by various researchers. As a result, the big data analysis with the feature selection improves the accuracy of the learning.

Sign in / Sign up

Export Citation Format

Share Document