Statistical Learning: Stability is Sufficient for Generalization and Necessary and Sufficient for Consistency of Empirical Risk Minimization

2004 ◽  
Author(s):  
Sayan Mukherjee ◽  
Partha Niyogi ◽  
Tomaso Poggio ◽  
Ryan Rifkin
Keyword(s):  
Statistical Learning ◽  
Empirical Risk Minimization ◽  
Risk Minimization ◽  
Empirical Risk ◽  
Necessary And Sufficient

STABILITY RESULTS IN LEARNING THEORY

2005 ◽  
Vol 03 (04) ◽  
pp. 397-417 ◽  
Author(s):  
ALEXANDER RAKHLIN ◽  
SAYAN MUKHERJEE ◽  
TOMASO POGGIO
Keyword(s):  
Sufficient Conditions ◽  
Empirical Risk Minimization ◽  
Risk Minimization ◽  
Generalization Bounds ◽  
Empirical Risk ◽  
Sufficient Stability Condition ◽  
Sufficient Stability ◽  
Exponential Bounds ◽  
Necessary And Sufficient ◽  
Stability Results

The problem of proving generalization bounds for the performance of learning algorithms can be formulated as a problem of bounding the bias and variance of estimators of the expected error. We show how various stability assumptions can be employed for this purpose. We provide a necessary and sufficient stability condition for bounding the bias and variance for the Empirical Risk Minimization algorithm, and various sufficient conditions for bounding bias and variance of estimators for general algorithms. We discuss settings in which it is possible to obtain exponential bounds, and we prove an extension of the bounded-difference inequality for "almost always" stable algorithms.

scholarly journals Learning Bounds of ERM Principle for Sequences of Time-Dependent Samples

2015 ◽  
Vol 2015 ◽  
pp. 1-8
Author(s):  
Mingchen Yao ◽  
Chao Zhang ◽  
Wei Wu
Keyword(s):  
Time Dependent ◽  
Dependent Data ◽  
Empirical Risk Minimization ◽  
Risk Minimization ◽  
Practical Applications ◽  
Empirical Risk ◽  
Learning Tasks ◽  
Generalization Bound ◽  
Erm Principle ◽  
Dependent Samples

Many generalization results in learning theory are established under the assumption that samples are independent and identically distributed (i.i.d.). However, numerous learning tasks in practical applications involve the time-dependent data. In this paper, we propose a theoretical framework to analyze the generalization performance of the empirical risk minimization (ERM) principle for sequences of time-dependent samples (TDS). In particular, we first present the generalization bound of ERM principle for TDS. By introducing some auxiliary quantities, we also give a further analysis of the generalization properties and the asymptotical behaviors of ERM principle for TDS.

Differentially Private Empirical Risk Minimization for AUC Maximization

2021 ◽  
Author(s):  
Puyu Wang ◽  
Zhenhuan Yang ◽  
Yunwen Lei ◽  
Yiming Ying ◽  
Hai Zhang
Keyword(s):  
Empirical Risk Minimization ◽  
Risk Minimization ◽  
Empirical Risk ◽  
Auc Maximization

Asymptotic Properties of Stationary Solutions of Coupled Nonconvex Nonsmooth Empirical Risk Minimization

2021 ◽  
Author(s):  
Zhengling Qi ◽  
Ying Cui ◽  
Yufeng Liu ◽  
Jong-Shi Pang
Keyword(s):  
Phase Retrieval ◽  
Convergence Rates ◽  
Stationary Solutions ◽  
Statistical Properties ◽  
Global Minimizer ◽  
General Distribution ◽  
Empirical Risk Minimization ◽  
Risk Minimization ◽  
Minimization Problems ◽  
Empirical Risk

This paper has two main goals: (a) establish several statistical properties—consistency, asymptotic distributions, and convergence rates—of stationary solutions and values of a class of coupled nonconvex and nonsmooth empirical risk-minimization problems and (b) validate these properties by a noisy amplitude-based phase-retrieval problem, the latter being of much topical interest. Derived from available data via sampling, these empirical risk-minimization problems are the computational workhorse of a population risk model that involves the minimization of an expected value of a random functional. When these minimization problems are nonconvex, the computation of their globally optimal solutions is elusive. Together with the fact that the expectation operator cannot be evaluated for general probability distributions, it becomes necessary to justify whether the stationary solutions of the empirical problems are practical approximations of the stationary solution of the population problem. When these two features, general distribution and nonconvexity, are coupled with nondifferentiability that often renders the problems “non-Clarke regular,” the task of the justification becomes challenging. Our work aims to address such a challenge within an algorithm-free setting. The resulting analysis is, therefore, different from much of the analysis in the recent literature that is based on local search algorithms. Furthermore, supplementing the classical global minimizer-centric analysis, our results offer a promising step to close the gap between computational optimization and asymptotic analysis of coupled, nonconvex, nonsmooth statistical estimation problems, expanding the former with statistical properties of the practically obtained solution and providing the latter with a more practical focus pertaining to computational tractability.

scholarly journals Learning Theory Estimates with Observations from General Stationary Stochastic Processes

2016 ◽  
Vol 28 (12) ◽  
pp. 2853-2889 ◽  
Author(s):  
Hanyuan Hang ◽  
Yunlong Feng ◽  
Ingo Steinwart ◽  
Johan A. K. Suykens
Keyword(s):  
Stochastic Processes ◽  
Type Inequality ◽  
Empirical Risk Minimization ◽  
Risk Minimization ◽  
Mixing Processes ◽  
Gaussian Kernels ◽  
Empirical Risk ◽  
Learning Rates ◽  
Stationary Stochastic Processes ◽  
Learning Schemes

This letter investigates the supervised learning problem with observations drawn from certain general stationary stochastic processes. Here by general, we mean that many stationary stochastic processes can be included. We show that when the stochastic processes satisfy a generalized Bernstein-type inequality, a unified treatment on analyzing the learning schemes with various mixing processes can be conducted and a sharp oracle inequality for generic regularized empirical risk minimization schemes can be established. The obtained oracle inequality is then applied to derive convergence rates for several learning schemes such as empirical risk minimization (ERM), least squares support vector machines (LS-SVMs) using given generic kernels, and SVMs using gaussian kernels for both least squares and quantile regression. It turns out that for independent and identically distributed (i.i.d.) processes, our learning rates for ERM recover the optimal rates. For non-i.i.d. processes, including geometrically [Formula: see text]-mixing Markov processes, geometrically [Formula: see text]-mixing processes with restricted decay, [Formula: see text]-mixing processes, and (time-reversed) geometrically [Formula: see text]-mixing processes, our learning rates for SVMs with gaussian kernels match, up to some arbitrarily small extra term in the exponent, the optimal rates. For the remaining cases, our rates are at least close to the optimal rates. As a by-product, the assumed generalized Bernstein-type inequality also provides an interpretation of the so-called effective number of observations for various mixing processes.

Sign in / Sign up

Export Citation Format

Share Document