This paper aims at refined error analysis for binary classification using support vector machine (SVM) with Gaussian kernel and convex loss. Our first result shows that for some loss functions such as the truncated quadratic loss and quadratic loss, SVM with Gaussian kernel can reach the almost optimal learning rate, p…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “Kernelized Loss Functions”
Regularized empirical risk minimization including support vector machines plays an important role in machine learning theory. In this paper regularized pairwise learning (RPL) methods based on kernels will be investigated. One example is regularized minimization of the error entropy loss which has recently attracted qu…
A new algorithm tackles adversarial linear contextual bandits using kernelized loss functions.
problem Online learning in adversarial linear contextual bandits with flexible loss functions.
method Proposes a computationally efficient algorithm using an optimistically biased estimator for reproducing kernel Hilbert space loss functions.
result Achieves near-optimal regret guarantees under polynomial and exponential eigendecay assumptions.
RKUM is an R package for robust kernel-based unsupervised methods.
problem Robust analysis under contaminated or noisy data conditions.
method Robust kernel covariance and cross-covariance operators using generalized loss functions.
result RKUM reduces sensitivity to contamination and effectively identifies outliers.
A novel dictionary-based approach for predicting functions.
problem Functional-output regression with non-orthogonal dictionaries.
method Projection learning (PL) with reproducing kernel Hilbert spaces (KPL).
result KPL offers a flexible and computationally efficient solution.
Least squares kernel based methods have been widely used in regression problems due to the simple implementation and good generalization performance. Among them, least squares support vector regression (LS-SVR) and extreme learning machine (ELM) are popular techniques. However, the noise sensitivity is a major bottlene…
Operator-Valued Kernels (OVKs) and associated vector-valued Reproducing Kernel Hilbert Spaces provide an elegant way to extend scalar kernel methods when the output space is a Hilbert space. Although primarily used in finite dimension for problems like multi-task regression, the ability of this framework to deal with i…
We present a generalization of the adversarial linear bandits framework, where the underlying losses are kernel functions (with an associated reproducing kernel Hilbert space) rather than linear functions. We study a version of the exponential weights algorithm and bound its regret in this setting. Under conditions on …
To the best of our knowledge, there are no general well-founded robust methods for statistical unsupervised learning. Most of the unsupervised methods explicitly or implicitly depend on the kernel covariance operator (kernel CO) or kernel cross-covariance operator (kernel CCO). They are sensitive to contaminated data, …
On the Connection between and Risk Consistency and its Implications on Regularized Kernel Methodsstat.ML
Paper connects risk consistency to L_p consistency for broader loss functions.
problem Establishing risk consistency for a wider class of loss functions.
method Analyzes the connection between risk consistency and L_p-consistency for various loss functions.
result Shifted loss functions do not reduce assumptions as much as other results.
We discover scaling laws for kernel regression loss under various learning rate schedules.
problem Understanding loss dynamics and learning rate schedules in kernel regression.
method Theoretical analysis of stochastic gradient descent on a power-law kernel regression model.
result Established a Functional Scaling Law (FSL) capturing the full loss trajectory under arbitrary learning rate schedules.
Novel method learns memory kernels in Langevin equations.
problem Estimating memory kernels in Langevin equations.
method Regularized Prony method for correlation functions, followed by regression over Sobolev norm-based loss function with RKHS regularization.
result Method outperforms other regression estimators in exponentially weighted L^2 space.
Many unsupervised kernel methods rely on the estimation of the kernel covariance operator (kernel CO) or kernel cross-covariance operator (kernel CCO). Both kernel CO and kernel CCO are sensitive to contaminated data, even when bounded positive definite kernels are used. To the best of our knowledge, there are few well…
Support vector machines (SVMs) are special kernel based methods and belong to the most successful learning methods since more than a decade. SVMs can informally be described as a kind of regularized M-estimators for functions and have demonstrated their usefulness in many complicated real-life problems. During the last…
Deep regularization and direct training of the inner layers of Neural Networks with Kernel Flowsstat.ML
New method uses Kernel Flows to improve neural network training without changing structure.
problem Improving neural network training without altering structure or output classifier.
method Combines KFs with a classical output loss to aggregate a subset of KF losses.
result Reduced test errors, decreased generalization gaps, increased robustness to distribution shift.
Paper tackles tensor decomposition for unaligned observations using RKHS and novel loss functions.
problem Tackles tensor decomposition for unaligned observations.
method Uses functions in RKHS to represent mode with unaligned observations, introduces versatile loss function, proposes optimization algorithm and stochastic gradient method.
result Demonstrates improved tensor decomposition efficiency and effectiveness with synthetic and real data.
Paper introduces robust distribution regression using kernel methods.
problem Distribution regression from probability measures to real-valued responses.
method Introduces a robust loss function and a windowing function for two-stage sampling problems.
result Shows improved learning rates and robustness with the robust distribution regression (RDR) scheme.
Unified analysis of kernel-based methods under covariate shift.
problem Covariate shift in learning problems.
method Unified analysis of nonparametric methods in RKHS.
result Sharp convergence rates for general loss functions.
DGKIP extends KIP for dataset distillation without bi-level optimization.
problem Efficiently distill datasets for various loss functions.
method Leverages duality theory to avoid bi-level optimization.
result DGKIP supports a wider range of loss functions.
Develops a new framework for robust regression with EGM.
problem Addressing robust regression with heavy-tailed noise or outliers.
method Empirical gain maximization (EGM) to approximate noise density.
result Unified analysis of robust regression approaches.
Kernel DRO uses RKHS to optimize under distributional uncertainty.
problem Optimizing under distributional uncertainty with limited knowledge.
method Kernel DRO using RKHS ambiguity sets and duality theory.
result Unified approach to robust and stochastic optimization.
Pairwise learning usually refers to a learning task which involves a loss function depending on pairs of examples, among which most notable ones include ranking, metric learning and AUC maximization. In this paper, we study an online algorithm for pairwise learning with a least-square loss function in an unconstrained …
Gaussian process regression loses locality in high dimensions, affecting molecular energy surface fitting.
problem Loss of locality in high-dimensional Gaussian process regression.
method Analysis of Matern family kernels and multi-zeta basis functions.
result The property of locality disappears in high dimensions, impacting regression quality.
We apply a fast kernel method for mask-based single-channel speech enhancement. Specifically, our method solves a kernel regression problem associated to a non-smooth kernel function (exponential power kernel) with a highly efficient iterative method (EigenPro). Due to the simplicity of this method, its hyper-parameter…
Proposes a method for inference in high-dimensional classification with non-differentiable surrogate losses.
problem Lack of inference procedures for identifying driving factors in high-dimensional classification with non-differentiable surrogate losses.
method Kernel-smoothed decorrelated score and cross-fitted version for hypothesis tests and interval estimators.
result Valid and superior inference methods for high-dimensional classification with non-differentiable surrogate losses.
Develop a comprehensive theory for regularized M-estimation in reproducing kernel Hilbert spaces.
problem Regularized M-estimation in reproducing kernel Hilbert spaces
method Existence and measurability of the estimator, sharp rates of convergence
result New rates for tensor product Sobolev spaces
We analyze double descent in finite-width neural networks using influence functions.
problem Understanding double descent in finite-width neural networks.
method Using influence functions to derive population loss bounds and investigate loss function effects.
result Derived bounds exhibit double descent behavior at the interpolation threshold.
Neural networks and linear systems linked, revealing training loss and kernel limitations.
problem Exploring the training loss and limitations of neural networks and their kernels.
method Drawing connections between neural networks and under-determined linear systems, providing lower bounds, and analyzing gradient descent.
result Zero training loss achievable for neural networks under certain conditions, but not for ReLU kernels.
The study analyzes spectral algorithms for kernel methods and derives generalization error.
problem Estimating generalization error of spectral algorithms for kernel methods.
method Considered spectral algorithms including KRR and GD, derived generalization error as a functional of learning profile.
result Showed the loss localizes on certain spectral scales and conjectured universality of the loss for noisy observations.
Paper introduces AIF to analyze robust optimization effects.
problem Quantifying robust optimization's impact on model optimizers and losses.
method Inspired by robust statistics, AIF is introduced to measure model sensitivity.
result AIF reveals how model complexity and randomized smoothing affect model sensitivity.
Kernel online convex optimization (KOCO) is a framework combining the expressiveness of non-parametric kernel models with the regret guarantees of online learning. First-order KOCO methods such as functional gradient descent require only time and space per iteration, and, when the only information on t…
Paper studies a robust online learning algorithm for regression.
problem Develops a robust online learning algorithm for regression problems.
method Introduces an online learning algorithm with a robust loss function over RKHS.
result The algorithm achieves optimal convergence rates in mean square and RKHS.
Functional Output Regression with Infimal Convolution: Exploring the Huber and -insensitive Lossesstat.ML
The paper introduces a new FOR framework using Huber and ε-insensitive losses.
problem Handling outliers and sparsity in functional output regression.
method Proposes a flexible FOR framework with infimal convolution losses and computable algorithms.
result Demonstrates efficiency and effectiveness on synthetic and real-world data.
A new framework learns differentiable structured losses from data.
problem Learning effective losses for complex structured prediction tasks.
method Contrastive learning to learn differentiable structured losses from output data.
result Achieves similar or better performance than kernel-based methods.
Paper introduces robust kernel ridge regression using Cauchy loss for handling various noise types.
problem Developing robust regression methods for noisy data.
method Introduces kernel Cauchy ridge regressor (KCRR) using Cauchy loss function.
result Establishes almost minimax-optimal convergence rate for KCRR in terms of -risk.
In this paper, we study two general classes of optimization algorithms for kernel methods with convex loss function and quadratic norm regularization, and analyze their convergence. The first approach, based on fixed-point iterations, is simple to implement and analyze, and can be easily parallelized. The second, based…
Reduces dynamic regret to static problem in RKHS.
problem Minimizing cumulative loss in online convex optimization.
method Reduces dynamic regret to static regret problem in RKHS.
result Optimal dynamic regret guarantees for linear losses and new bounds for exp-concave and improper linear regression.
It is shown that bootstrap approximations of support vector machines (SVMs) based on a general convex and smooth loss function and on a general kernel are consistent. This result is useful to approximate the unknown finite sample distribution of SVMs by the bootstrap approach.
Study risk bounds for distributed ERM with general loss functions and hypothesis spaces.
problem Limited theoretical analysis for distributed ERM with general loss functions and hypothesis spaces.
method Derive tight risk bounds under assumptions on hypothesis space and loss function.
result Developed more general risk bound for distributed ERM without strong convexity restriction.
Enhanced Hopfield model boosts memory retrieval capacity.
problem Memory retrieval in modern Hopfield models with limited capacity.
method Introduces a learnable feature map transforming energy function into kernel space, minimizing separation loss for uniform memory distribution.
result Significant reduction in metastable states, enhancing memory capacity and retrieval accuracy.
MKL--SVMstat.ML
Paper introduces MKL--SVM for SVM with loss.
problem Optimization of SVM with loss function.
method MKL framework combined with ADMM algorithm for solving the optimization problem.
result Performance of MKL--SVM comparable to SimpleMKL.
New solver for MKL-SVM with 0/1 loss function.
problem Optimizing MKL-SVM with -loss function.
method Developed an ADMM solver for nonconvex, nonsmooth optimization.
result Shows promise in simple numerical experiment.
Tilting loss functions improves machine learning performance.
problem Improving machine learning models, especially in under- and over-parameterized networks.
method Using evolving loss functions that emphasize different classes cyclically.
result Dynamical loss functions lead to better generalization and stability in training.
Generative adversarial nets (GANs) are widely used to learn the data sampling process and their performance may heavily depend on the loss functions, given a limited computational budget. This study revisits MMD-GAN that uses the maximum mean discrepancy (MMD) as the loss function for GAN and makes two contributions. F…
We study strictly proper scoring rules in the Reproducing Kernel Hilbert Space. We propose a general Kernel Scoring rule and associated Kernel Divergence. We consider conditions under which the Kernel Score is strictly proper. We then demonstrate that the Kernel Score includes the Maximum Mean Discrepancy as a special …
A well-recognized limitation of kernel learning is the requirement to handle a kernel matrix, whose size is quadratic in the number of training examples. Many methods have been proposed to reduce this computational cost, mostly by using a subset of the kernel matrix entries, or some form of low-rank matrix approximatio…
Learning "best" kernels from data in Gaussian process regression. With application to aerodynamicsstat.ML
Paper learns optimal kernels for Gaussian process regression in aerodynamics.
problem Approximating complex functions from limited data in aerodynamics.
method Two algorithms: Kernel Flow and Spectral Kernel Ridge Regression.
result Explicit construction of optimal kernels based on target function features.
In nonparametric classification and regression problems, regularized kernel methods, in particular support vector machines, attract much attention in theoretical and in applied statistics. In an abstract sense, regularized kernel methods (simply called SVMs here) can be seen as regularized M-estimators for a parameter …