scholarly journals Learning Rates for -Regularized Kernel Classifiers

2013 ◽  
Vol 2013 ◽  
pp. 1-11 ◽  
Author(s):  
Hongzhi Tong ◽  
Di-Rong Chen ◽  
Fenghong Yang
Keyword(s):  
Marginal Distribution ◽  
Approximation Error ◽  
Misclassification Error ◽  
Classification Algorithms ◽  
Learning Rates ◽  
Convex Loss Function ◽  
Sample Error ◽  
Convex Loss ◽  
Error Learning ◽  
Error Decomposition

We consider a family of classification algorithms generated from a regularization kernel scheme associated with -regularizer and convex loss function. Our main purpose is to provide an explicit convergence rate for the excess misclassification error of the produced classifiers. The error decomposition includes approximation error, hypothesis error, and sample error. We apply some novel techniques to estimate the hypothesis error and sample error. Learning rates are eventually derived under some assumptions on the kernel, the input space, the marginal distribution, and the approximation error.

Learning with Convex Loss and Indefinite Kernels

2014 ◽  
Vol 26 (1) ◽  
pp. 158-184 ◽  
Author(s):  
Hongzhi Tong ◽  
Di-Rong Chen ◽  
Fenghong Yang
Keyword(s):  
Approximation Error ◽  
Positive Semidefinite ◽  
Learning Rate ◽  
Support Vector ◽  
Empirical Process Theory ◽  
Sample Error ◽  
General Convex ◽  
Convex Loss ◽  
Detailed Mathematical Analysis ◽  
Error Decomposition

We consider a kind of kernel-based regression with general convex loss functions in a regularization scheme. The kernels used in the scheme are not necessarily symmetric and thus are not positive semidefinite; l1−norm of the coefficients in the kernel ensembles is taken as the regularizer. Our setting in this letter is quite different from the classical regularized regression algorithms such as regularized networks and support vector machines regression. Under an established error decomposition that consists of approximation error, hypothesis error, and sample error, we present a detailed mathematical analysis for this scheme and, in particular, its learning rate. A reweighted empirical process theory is applied to the analysis of produced learning algorithms, which plays a key role in deriving the explicit learning rate under some assumptions.

A Note on Support Vector Machines with Polynomial Kernels

2016 ◽  
Vol 28 (1) ◽  
pp. 71-88 ◽  
Author(s):  
Hongzhi Tong
Keyword(s):  
Support Vector Machines ◽  
Marginal Distribution ◽  
Theoretical Foundation ◽  
Approximation Error ◽  
Support Vector ◽  
Gaussian Kernels ◽  
Learning Rates ◽  
Vector Machines ◽  
Sample Error ◽  
Polynomial Kernels

We present a better theoretical foundation of support vector machines with polynomial kernels. The sample error is estimated under Tsybakov’s noise assumption. In bounding the approximation error, we take advantage of a geometric noise assumption that was introduced to analyze gaussian kernels. Compared with the previous literature, the error analysis in this note does not require any regularity of the marginal distribution or smoothness of Bayes’ rule. We thus establish the learning rates for polynomial kernels for a wide class of distributions.

ONLINE REGRESSION WITH VARYING GAUSSIANS AND NON-IDENTICAL DISTRIBUTIONS

2011 ◽  
Vol 09 (04) ◽  
pp. 395-408 ◽  
Author(s):  
TING HU
Keyword(s):  
Error Analysis ◽  
Loss Function ◽  
Learning Ability ◽  
Probability Measures ◽  
Gaussian Kernels ◽  
Learning Rates ◽  
Convex Loss Function ◽  
General Convex ◽  
Convex Loss

We consider a fully online regression algorithm associated with a general convex loss function and Gaussian kernels with changing variances. Error analysis is conducted in a setting with samples drawn from a non-identical sequence of probability measures. When a fixed Gaussian is used, it was known that the learning ability of induced algorithms is weak. By allowing varying Gaussians, we show that the achieved learning rates can be of polynomial decays.

Sparse additive machine with ramp loss

2020 ◽  
pp. 1-20
Author(s):  
Hong Chen ◽  
Changying Guo ◽  
Huijuan Xiong ◽  
Yingjie Wang
Keyword(s):  
Descent Method ◽  
Misclassification Error ◽  
High Dimensional ◽  
Block Coordinate Descent ◽  
Regularization Scheme ◽  
Hinge Loss ◽  
Dimensional Classification ◽  
Benchmark Datasets ◽  
Ramp Loss ◽  
Error Decomposition

Sparse additive machines (SAMs) have attracted increasing attention in high dimensional classification due to their representation flexibility and interpretability. However, most of existing methods are formulated under Tikhonov regularization scheme with the hinge loss, which are susceptible to outliers. To circumvent this problem, we propose a sparse additive machine with ramp loss (called ramp-SAM) to tackle classification and variable selection simultaneously. Misclassification error bound is established for ramp-SAM with the help of detailed error decomposition and constructive hypothesis error analysis. To solve the nonsmooth and nonconvex ramp-SAM, a proximal block coordinate descent method is presented with convergence guarantees. The empirical effectiveness of our model is confirmed on simulated and benchmark datasets.

Learning rates for regularized least squares ranking algorithm

2017 ◽  
Vol 15 (06) ◽  
pp. 815-836 ◽  
Author(s):  
Yulong Zhao ◽  
Jun Fan ◽  
Lei Shi
Keyword(s):  
Least Squares ◽  
Reproducing Kernel ◽  
Reproducing Kernel Hilbert Space ◽  
Ranking Algorithm ◽  
Generalization Error ◽  
Ranking Problem ◽  
Regularized Least Squares ◽  
U Statistics ◽  
Learning Rates ◽  
Error Decomposition

The ranking problem aims at learning real-valued functions to order instances, which has attracted great interest in statistical learning theory. In this paper, we consider the regularized least squares ranking algorithm within the framework of reproducing kernel Hilbert space. In particular, we focus on analysis of the generalization error for this ranking algorithm, and improve the existing learning rates by virtue of an error decomposition technique from regression and Hoeffding’s decomposition for U-statistics.

scholarly journals Constructive Analysis for Least Squares Regression with GeneralizedK-Norm Regularization

2014 ◽  
Vol 2014 ◽  
pp. 1-7 ◽  
Author(s):  
Cheng Wang ◽  
Weilin Nie
Keyword(s):  
Integral Operator ◽  
Least Squares ◽  
Learning Rate ◽  
Spectral Theorem ◽  
Least Squares Regression ◽  
Constructive Approach ◽  
Projection Technique ◽  
Sample Error ◽  
Regularization Error ◽  
Error Decomposition

We introduce a constructive approach for the least squares algorithms with generalizedK-norm regularization. Different from the previous studies, a stepping-stone function is constructed with some adjustable parameters in error decomposition. It makes the analysis flexible and may be extended to other algorithms. Based on projection technique for sample error and spectral theorem for integral operator in regularization error, we finally derive a learning rate.

Sign in / Sign up

Export Citation Format

Share Document