Unified derivation of PAC-Bayes and MI bounds for general VC classes with fast rates.
problem Generalization bounds for machine learning models with VC classes.
method Unified derivation of conditional PAC-Bayesian and mutual information bounds, including MAC-Bayesian bounds.
result Nontrivial bounds for general VC classes and faster rates for specific conditions.
The developments of Rademacher complexity and PAC-Bayesian theory have been largely independent. One exception is the PAC-Bayes theorem of Kakade, Sridharan, and Tewari (2008), which is established via Rademacher complexity theory by viewing Gibbs classifiers as linear operators. The goal of this paper is to extend thi…
Improved fast rates for decision making with forward-KL regularization in contextual bandits.
problem Improving fast rates for decision making with forward-KL regularization in contextual bandits.
method Streamlined analysis of forward-KL-regularized offline CBs, exploiting the pessimism principle and convex-analytical pipeline.
result First ildeO(ε−1) upper bounds in tabular and general function approximation settings. New algorithm reduces best-in-class regret in contextual bandits.
problem Compete with the best policy in a class without model restrictions.
method Proposes an algorithm that updates policies by minimizing a pessimistic objective, including a clipped inverse-propensity estimate and variance penalty.
result Achieves fast best-in-class regret rates, including polylogarithmic rates in the parametric case.
New bounds for multi-task learning with varying task sizes.
problem Generalization in multi-task learning with tasks of different sizes.
method PAC-Bayesian bounds for unbalanced settings.
result Stronger generalization bounds for multi-task learning with varying task sizes.
The paper tackles fast rates in structured prediction problems.
problem Structured prediction problems with discrete outputs.
method Introducing continuous surrogate problems and leveraging their convergence rates for discrete problems.
result Super fast rates, including exponential rates, for excess risk in structured prediction problems.
Error bound conditions (EBC) are properties that characterize the growth of an objective function when a point is moved away from the optimal set. They have recently received increasing attention in the field of optimization for developing optimization algorithms with fast convergence. However, the studies of EBC in st…
The paper improves PAC-Bayes bounds for losses with finite moments.
problem Bounding generalization for losses with heavy tails and finite moments.
method Truncation method and PAC-Bayes bounds for unbounded losses with heavy tails and bounded variance.
result Bounds interpolate between slow and fast rates depending on the moment.
Deep neural nets estimate operators between infinite-dimensional spaces with fast rates.
problem Estimating operators between infinite-dimensional spaces.
method Deep neural networks for nonparametric estimation of Lipschitz operators.
result Error bounds decay with fast rates depending on intrinsic dimension.
Study shows convergence rate for empirical minimizer of unbounded functions with fast growth.
problem Convergence rate of empirical minimizer for unbounded functions with fast growth.
method Analyzes L1-distance convergence rate of the empiric minimizer for coercive functions sampled with noise. result Convergence rate is bounded above by ann−1/q, where q is the dimension and an=o(nε) for every ε>0. TD(0) with Polyak-Ruppert averaging achieves robust and fast convergence rates
problem TD(0) learning under Markovian sampling
method Polyak-Ruppert averaging with a single stepsize
result Simultaneous high-probability convergence guarantees for TD(0) iterates and PR average
Investigates fast prediction rates with limited expert advice.
problem Minimizing excess generalization error with limited expert access.
method Assumes Lipschitz and strongly convex loss, designs novel algorithms.
result Achieves fast rates of O(1/T) with optimal number of expert advices.
Paper improves PAC-Bayes bounds for various loss types.
problem Improving PAC-Bayes bounds for different types of losses.
method Introducing new high-probability PAC-Bayes bounds for bounded and general tail behaviors losses, and extending to anytime-valid bounds.
result New fast-rate and mixed-rate bounds for losses with bounded ranges, and parameter-free bounds for losses with general tail behaviors.
Paper improves a method for fast global and local convergence in optimization.
problem Slow global convergence in optimization methods with noisy Hessian estimates.
method Stochastic Newton Proximal Extragradient method using HPE framework.
result Faster global linear rate and superlinear convergence in fewer iterations.
Paper optimizes multi-fidelity function with fast learning rates.
problem Optimizing a locally smooth function with limited budget and varying fidelity approximations.
method Kometo algorithm that achieves simple regret rates without knowing function smoothness or fidelity assumptions.
result Kometo algorithm outperforms previous methods empirically.
New learning dynamics achieve fast convergence in games without needing to know utility scales.
problem Fast convergence guarantees in learning games require prior knowledge of utility scales.
method Developed scale-free and scale-invariant learning dynamics using optimistic follow-the-regularized-leader with adaptive learning rates and clipping techniques.
result Achieved fast convergence rates to Nash and correlated equilibria without prior utility scale knowledge.
New bounds for balanced classification improve understanding of imbalanced datasets.
problem Negligible size of the minority class in imbalanced datasets.
method Developed non-asymptotic and consistent bounds for balanced empirical risk minimization and balanced nearest neighbors estimates.
result Improved understanding of class-weighting benefits in real-world imbalanced classification settings.
We study fast learning rates when the losses are not necessarily bounded and may have a distribution with heavy tails. To enable such analyses, we introduce two new conditions: (i) the envelope function supf∈F∣ℓ∘f∣, where ℓ is the loss function and F is the hypothesis class…
In the setting of sequential prediction of individual {0,1}-sequences with expert advice, we show that by allowing the learner to abstain from the prediction by paying a cost marginally smaller than 21 (say, 0.49), it is possible to achieve expected regret bounds that are independent of the time horizon …
The speed with which a learning algorithm converges as it is presented with more data is a central problem in machine learning --- a fast rate of convergence means less data is needed for the same level of performance. The pursuit of fast rates in online and statistical learning has led to the discovery of many conditi…
New bounds show polyhedral surrogates are optimal for generalization.
problem Proving generalization rates for polyhedral loss functions.
method Developed two general results for polyhedral surrogates.
result Polyhedral surrogates provide linear surrogate regret bounds, translating directly to target rates.
In this paper, we give a new sharp generalization bound of lp-MKL which is a generalized framework of multiple kernel learning (MKL) and imposes lp-mixed-norm regularization instead of l1-mixed-norm regularization. We utilize localization techniques to obtain the sharp learning rate. The bound is characterized by the d…
For binary classification we establish learning rates up to the order of n−1 for support vector machines (SVMs) with hinge loss and Gaussian RBF kernels. These rates are in terms of two assumptions on the considered distributions: Tsybakov's noise assumption to establish a small estimation error, and a new geometr…
New bounds on KPCA efficiency reveal conditions for fast convergence.
problem Lack of theoretical understanding of KPCA efficiency.
method Lower and upper bounds on KPCA efficiency involving empirical eigenvalues and new variance quantities.
result Fast convergence rates achievable for certain kernels, highlighting dataset properties.
Linear-Core Surrogates combine fast optimization and statistical efficiency in classification and structured prediction.
problem The trade-off between smoothness and margin-based losses in classification and structured prediction.
method Linear-Core (LC) Surrogates, a family of convex loss functions that stitch a linear core to a smooth tail.
result LC Surrogates achieve fast linear consistency rates while maintaining differentiability and strict H-consistency bounds. An infinitely wide model is a weighted integration ∫φ(x,v)dμ(v) of feature maps. This model excels at handling an infinite number of features, and thus it has been adopted to the theoretical study of deep learning. Kernel quadrature is a kernel-based numerical integration scheme developed for fast approxi…
In theoretical analysis of deep learning, discovering which features of deep learning lead to good performance is an important task. In this paper, using the framework for analyzing the generalization error developed in Suzuki (2018), we derive a fast learning rate for deep neural networks with more general activation …
Improved bounds for non-linear SA with fast convergence.
problem Stochastic approximation with non-linear mappings and multiple time scales.
method Mean squared error bounds with O(1/k) rate for contractive mappings. result First O(1/k) rate for non-linear two-time-scale SA without additional smoothness assumptions. The paper offers error bounds for quantized dynamical models.
problem Accuracy of dynamical models from dependent data sequences.
method Developed uniform error bounds for quantized models and imperfect optimization algorithms.
result Unified bounds for slow and fast rates, scaling with model encoding bits.
Study fast learning rates for square loss in dependent data with hypercontractivity condition.
problem Learning from dependent data with fast rates matching independent data.
method Martingale difference noise, trajectory hypercontractivity condition, least-squares estimator.
result Excess risk bound matches iid rate after burn-in time, independent of mixing-time.
Study shows how fast a specific matrix completion method works.
problem Completing a rank-one matrix from a subset of revealed entries.
method Alternating minimization approach for matrix completion.
result Polynomial upper bound on convergence rate.
The stochastic gradient descent (SGD) optimization algorithm plays a central role in a series of machine learning applications. The scientific literature provides a vast amount of upper error bounds for the SGD method. Much less attention as been paid to proving lower error bounds for the SGD method. It is the key cont…
In this paper, we present a simple analysis of {\bf fast rates} with {\it high probability} of {\bf empirical minimization} for {\it stochastic composite optimization} over a finite-dimensional bounded convex set with exponential concave loss functions and an arbitrary convex regularization. To the best of our knowledg…
Neural operators achieve fast convergence rates for solving PDEs.
problem Solving partial differential equations (PDEs) efficiently.
method Two-layer neural operators with gradient descent analysis in RKHS.
result Fast convergence rates are minimax optimal for early-stopped GD.
Study shows exponential error reduction in multiclass classification without bias-variance trade-off.
problem Multiclass classification with margin conditions.
method Analysis of classification error under hard-margin conditions.
result Exponential decrease in classification error without bias-variance trade-off.
We derive an upper bound on the local Rademacher complexity of ℓp-norm multiple kernel learning, which yields a tighter excess risk bound than global approaches. Previous local approaches aimed at analyzed the case p=1 only while our analysis covers all cases 1≤p≤∞, assuming the different feature …
Paper studies deep learning for solving elliptic PDEs, proving optimal bounds and neural scaling laws.
problem Solving elliptic PDEs from random samples using machine learning.
method Deep Ritz Method and Physics-Informed Neural Networks (PINNs) for the Schrödinger equation.
result Proves minimax optimal bounds and neural scaling laws for deep PDE solvers.
This paper tightens information-theoretic bounds on generalization errors.
problem Understanding the discrepancy between training and testing data losses.
method Investigates the tightness of information-theoretic bounds on generalization error.
result The individual sample mutual information bound can be asymptotically tight under specific assumptions.
In this paper, we give a new generalization error bound of Multiple Kernel Learning (MKL) for a general class of regularizations, and discuss what kind of regularization gives a favorable predictive accuracy. Our main target in this paper is dense type regularizations including \ellp-MKL. According to the recent numeri…
Paper offers a fast convergence theory for offline decision making.
problem Offline decision making problems, including reinforcement learning and off-policy evaluation.
method Introduces a framework (DMOF) and algorithm (EDD) with a fast convergence guarantee.
result Demonstrates a fast convergence guarantee with a lower bound complement.
The paper explains how data augmentation improves semi-supervised learning efficiency.
problem Improving accuracy from a small fraction of labeled data.
method Data augmentation induces a similarity graph, which is graph-Laplacian-regularized for downstream learning.
result A fast transductive rate of O(1/nL) is achieved, reducing the number of labels needed. Paper analyzes mistake and generalization of MNIC classifiers.
problem Understanding the performance of interpolating classifiers.
method Elementary analyses of MNIC's regret and generalization.
result MNIC generalizes with a rate proportional to the norm of the interpolating solution and inversely proportional to the number of data points.
We investigate the learning rate of multiple kernel leaning (MKL) with elastic-net regularization, which consists of an ℓ1-regularizer for inducing the sparsity and an ℓ2-regularizer for controlling the smoothness. We focus on a sparse setting where the total number of kernels is large but the number of non…
Empirical risk minimization (ERM) is a fundamental learning rule for statistical learning problems where the data is generated according to some unknown distribution P and returns a hypothesis f chosen from a fixed class F with small loss ℓ. In the parametric setting, depending upon $(\ell…
The paper bounds the expectation of empirical processes indexed by Hölder classes.
problem Estimating the expectation of the supremum of empirical processes for distributions on bounded sets.
method Providing upper bounds on the expectation of the supremum of empirical processes indexed by Hölder classes.
result Deriving non-asymptotic risk bounds for estimating distributions using empirical processes and IPM.
We study the problem of empirical minimization for variance-type functionals over functional classes. Sharp non-asymptotic bounds for the excess variance are derived under mild conditions. In particular, it is shown that under some restrictions imposed on the functional class fast convergence rates can be achieved incl…
This paper deals with two related problems, namely distance-preserving binary embeddings and quantization for compressed sensing . First, we propose fast methods to replace points from a subset X⊂Rn, associated with the Euclidean metric, with points in the cube {±1}m and we associa…
Paper analyzes normal approximation for two-timescale stochastic algorithms, revealing interaction between fast and slow timescales.
problem Non-asymptotic bounds for accuracy of normal approximation in linear two-timescale stochastic approximation algorithms.
method Established bounds for normal approximation in terms of convex distance, focusing on last iterate and Polyak-Ruppert averaging.
result Normal approximation rate for the last iterate improves with increased timescale separation, while it decreases in the averaged setting.