The paper tackles fast rates in structured prediction problems.
problem Structured prediction problems with discrete outputs.
method Introducing continuous surrogate problems and leveraging their convergence rates for discrete problems.
result Super fast rates, including exponential rates, for excess risk in structured prediction problems.
Empirical risk minimization (ERM) is a fundamental learning rule for statistical learning problems where the data is generated according to some unknown distribution P and returns a hypothesis f chosen from a fixed class F with small loss ℓ. In the parametric setting, depending upon $(\ell…
Paper shows SVM can achieve super fast convergence rates.
problem Understanding fast convergence rates for SVM.
method Presented a simple mechanism to obtain fast convergence rates for SVM.
result SVM can exhibit exponential convergence rates without hard Tsybakov margin condition.
The paper analyzes reinforcement learning methods for estimating weights and quality functions with fast convergence rates.
problem Estimating weights and quality functions in reinforcement learning with function approximation.
method The paper uses minimax methods for estimating marginal importance weights and q-functions.
result The minimax approach enables fast rates of convergence for weights and quality functions, achieving first-order efficiency.
Study shows convergence rate for empirical minimizer of unbounded functions with fast growth.
problem Convergence rate of empirical minimizer for unbounded functions with fast growth.
method Analyzes L1-distance convergence rate of the empiric minimizer for coercive functions sampled with noise. result Convergence rate is bounded above by ann−1/q, where q is the dimension and an=o(nε) for every ε>0. The developments of Rademacher complexity and PAC-Bayesian theory have been largely independent. One exception is the PAC-Bayes theorem of Kakade, Sridharan, and Tewari (2008), which is established via Rademacher complexity theory by viewing Gibbs classifiers as linear operators. The goal of this paper is to extend thi…
This paper shows how to learn variational inequalities fast with strong monotonicity.
problem Learning variational inequalities efficiently.
method Extending convex optimization techniques to variational inequalities with strong monotonicity.
result Fast generalization rates of Θ(1/ε) for learning variational inequalities. The speed with which a learning algorithm converges as it is presented with more data is a central problem in machine learning --- a fast rate of convergence means less data is needed for the same level of performance. The pursuit of fast rates in online and statistical learning has led to the discovery of many conditi…
Error bound conditions (EBC) are properties that characterize the growth of an objective function when a point is moved away from the optimal set. They have recently received increasing attention in the field of optimization for developing optimization algorithms with fast convergence. However, the studies of EBC in st…
Paper improves a method for fast global and local convergence in optimization.
problem Slow global convergence in optimization methods with noisy Hessian estimates.
method Stochastic Newton Proximal Extragradient method using HPE framework.
result Faster global linear rate and superlinear convergence in fewer iterations.
Unified derivation of PAC-Bayes and MI bounds for general VC classes with fast rates.
problem Generalization bounds for machine learning models with VC classes.
method Unified derivation of conditional PAC-Bayesian and mutual information bounds, including MAC-Bayesian bounds.
result Nontrivial bounds for general VC classes and faster rates for specific conditions.
Improved fast rates for decision making with forward-KL regularization in contextual bandits.
problem Improving fast rates for decision making with forward-KL regularization in contextual bandits.
method Streamlined analysis of forward-KL-regularized offline CBs, exploiting the pessimism principle and convex-analytical pipeline.
result First ildeO(ε−1) upper bounds in tabular and general function approximation settings. Study shows fast rates for inverse reinforcement learning with linear rewards.
problem Entropy-regularized min-max inverse reinforcement learning in finite-horizon MDPs.
method Structural and statistical analysis of Min-Max-IRL with pseudo-self-concordance.
result Both trajectory-level KL divergence and parameter error decay at O(n−1). New algorithm reduces best-in-class regret in contextual bandits.
problem Compete with the best policy in a class without model restrictions.
method Proposes an algorithm that updates policies by minimizing a pessimistic objective, including a clipped inverse-propensity estimate and variance penalty.
result Achieves fast best-in-class regret rates, including polylogarithmic rates in the parametric case.
We study fast learning rates when the losses are not necessarily bounded and may have a distribution with heavy tails. To enable such analyses, we introduce two new conditions: (i) the envelope function supf∈F∣ℓ∘f∣, where ℓ is the loss function and F is the hypothesis class…
Paper optimizes multi-fidelity function with fast learning rates.
problem Optimizing a locally smooth function with limited budget and varying fidelity approximations.
method Kometo algorithm that achieves simple regret rates without knowing function smoothness or fidelity assumptions.
result Kometo algorithm outperforms previous methods empirically.
We consider the classical problem of learning rates for classes with finite VC dimension. It is well known that fast learning rates up to O(nd) are achievable by the empirical risk minimization algorithm (ERM) if low noise or margin assumptions are satisfied. These usually require the optimal Baye…
Deep neural nets estimate operators between infinite-dimensional spaces with fast rates.
problem Estimating operators between infinite-dimensional spaces.
method Deep neural networks for nonparametric estimation of Lipschitz operators.
result Error bounds decay with fast rates depending on intrinsic dimension.
Investigates fast prediction rates with limited expert advice.
problem Minimizing excess generalization error with limited expert access.
method Assumes Lipschitz and strongly convex loss, designs novel algorithms.
result Achieves fast rates of O(1/T) with optimal number of expert advices.
We consider the stochastic contextual bandit problem with additional regularization. The motivation comes from problems where the policy of the agent must be close to some baseline policy which is known to perform well on the task. To tackle this problem we use a nonparametric model and propose an algorithm splitting t…
We derive the fast convergence rates of a deep neural network (DNN) classifier with the rectified linear unit (ReLU) activation function learned using the hinge loss. We consider three cases for a true model: (1) a smooth decision boundary, (2) smooth conditional class probability, and (3) the margin condition (i.e., t…
An infinitely wide model is a weighted integration ∫φ(x,v)dμ(v) of feature maps. This model excels at handling an infinite number of features, and thus it has been adopted to the theoretical study of deep learning. Kernel quadrature is a kernel-based numerical integration scheme developed for fast approxi…
When applied to training deep neural networks, stochastic gradient descent (SGD) often incurs steady progression phases, interrupted by catastrophic episodes in which loss and gradient norm explode. A possible mitigation of such events is to slow down the learning process. This paper presents a novel approach to contro…
Paper analyzes faster convergence rates for reinforcement learning from offline data.
problem Analyzing faster convergence rates for reinforcement learning from offline data.
method Fine analysis of reinforcement learning from offline data, providing fast rates for regret convergence.
result The paper provides fast rates for the regret convergence, showing that the level of exponentiation depends on the noise in the decision-making problem.
Research on nearest-neighbor methods tends to focus somewhat dichotomously either on the statistical or the computational aspects -- either on, say, Bayes consistency and rates of convergence or on techniques for speeding up the proximity search. This paper aims at bridging these realms: to reap the advantages of fast …
Noise affects the effectiveness of interpolating models, especially those with strong inductive biases.
problem The impact of noise on interpolating models with strong inductive biases.
method Analyzing linear and classification models with sparse ground truths, proving fast rates for interpolators.
result Strong inductive biases can lead to faster but noisier interpolators, contrary to intuition.
The gauge theory of arbitrage was introduced by Ilinski in [arXiv:hep-th/9710148] and applied to fast money flows in [arXiv:cond-mat/9902044]. The theory of fast money flow dynamics attempts to model the evolution of currency exchange rates and stock prices on short, e.g.\ intra-day, time scales. It has been used to ex…
We prove new fast learning rates for the one-vs-all multiclass plug-in classifiers trained either from exponentially strongly mixing data or from data generated by a converging drifting distribution. These are two typical scenarios where training data are not iid. The learning rates are obtained under a multiclass vers…
Weak labels can significantly speed up learning for strong tasks.
problem Learning with limited strong labels.
method Using weak labels to accelerate learning of strong tasks.
result Weak labels can accelerate learning to O(icefrac1n) rate. This paper proposes a new method for estimating sparse precision matrices in the high dimensional setting. It has been popular to study fast computation and adaptive procedures for this problem. We propose a novel approach, called Sparse Column-wise Inverse Operator, to address these two issues. We analyze an adaptive …
TD(0) with Polyak-Ruppert averaging achieves robust and fast convergence rates
problem TD(0) learning under Markovian sampling
method Polyak-Ruppert averaging with a single stepsize
result Simultaneous high-probability convergence guarantees for TD(0) iterates and PR average
A new topology improves decentralized learning efficiency and accuracy.
problem Finding efficient decentralized learning topologies with fast consensus and low maximum degree.
method Proposed the Base-(k+1) Graph topology for decentralized learning. result The Base-(k+1) Graph enables faster convergence and better communication efficiency than the exponential graph. Paper proposes a boosting method with fast learning rates and early stopping.
problem Missing theoretical guarantees for boosting methods in binary classification.
method Fully-corrective gradient boosting with squared hinge loss and ADMM algorithm.
result Derives fast learning rates of O((m/logm)−1/4) and O((m/logm)−1/2). Algorithm minimizes risk for multiclass classification of stochastic diffusion paths.
problem Multiclass classification of stochastic diffusion paths with distinct drift functions.
method Empirical risk minimization using L2 risk.
result Achieves fast rates of convergence under margin assumption.
The effect of errors in variables in quantization is investigated. We prove general exact and non-exact oracle inequalities with fast rates for an empirical minimization based on a noisy sample Zi=Xi+εi,i=1,…,n, where Xi are i.i.d. with density f and εi are i.i.d. with density η. These rates depend …
This research examines how the error rate of nearest neighbor classifiers varies with dataset size.
problem The scaling of classification error rates with dataset size is not uniform.
method Theoretical analysis of nearest neighbor classifiers, focusing on early and late phases of dataset size.
result The error rate of nearest neighbor classifiers can have fine-grained rates depending on the dataset size and data distribution.
In this paper we propose a new method to assist in labeling data arriving from fast running processes using anomaly detection. A result is the possibility to manually classify data arriving at a high rates to train machine learning models. To circumvent the problem of not having a real ground truth we propose specific …
Paper shows how meta-learning can reduce prior learning cost.
problem Learning the prior in meta-learning with fast rates.
method Examined Gibbs algorithm in meta-learning context.
result Bernstein's condition holds at meta level, reducing prior learning cost.
New bounds for multi-task learning with varying task sizes.
problem Generalization in multi-task learning with tasks of different sizes.
method PAC-Bayesian bounds for unbalanced settings.
result Stronger generalization bounds for multi-task learning with varying task sizes.
In this work we introduce a new optimisation method called SAGA in the spirit of SAG, SDCA, MISO and SVRG, a set of recently proposed incremental gradient algorithms with fast linear convergence rates. SAGA improves on the theory behind SAG and SVRG, with better theoretical convergence rates, and has support for compos…
Neural operators achieve fast convergence rates for solving PDEs.
problem Solving partial differential equations (PDEs) efficiently.
method Two-layer neural operators with gradient descent analysis in RKHS.
result Fast convergence rates are minimax optimal for early-stopped GD.
Linear-Core Surrogates combine fast optimization and statistical efficiency in classification and structured prediction.
problem The trade-off between smoothness and margin-based losses in classification and structured prediction.
method Linear-Core (LC) Surrogates, a family of convex loss functions that stitch a linear core to a smooth tail.
result LC Surrogates achieve fast linear consistency rates while maintaining differentiability and strict H-consistency bounds. Develops a fast variational approximation for high-dimensional empirical Bayes posteriors.
problem Optimal posterior computation in high-dimensional settings with prior tails effect.
method Variational approximation of empirical Bayes posterior with data-driven centers and thin-tailed conjugate priors.
result Retains optimal concentration rate properties and superior performance compared to existing methods.
New learning dynamics achieve fast convergence in games without needing to know utility scales.
problem Fast convergence guarantees in learning games require prior knowledge of utility scales.
method Developed scale-free and scale-invariant learning dynamics using optimistic follow-the-regularized-leader with adaptive learning rates and clipping techniques.
result Achieved fast convergence rates to Nash and correlated equilibria without prior utility scale knowledge.
PPGD solves nonconvex nonsmooth optimization problems without KL property.
problem Nonconvex and nonsmooth optimization problems in statistics and machine learning.
method Projective Proximal Gradient Descent (PPGD) for solving a class of nonconvex and nonsmooth problems.
result PPGD achieves a fast convergence rate of O(1/k^2) for k ≥ k_0.
The alternating direction method of multipliers (ADMM) is a powerful optimization solver in machine learning. Recently, stochastic ADMM has been integrated with variance reduction methods for stochastic gradient, leading to SAG-ADMM and SDCA-ADMM that have fast convergence rates and low iteration complexities. However,…
New findings on flatness of certain metrics with fast decay.
problem Rigidity of positive mass theorem under fast metric decay.
method Considered metrics with nonnegative scalar curvature and rapid decay at infinity.
result Any such metric is necessarily flat in dimensions 4 and higher if decay rate exceeds Schwarzschild metric.
FAST improves fast and stable task adaptation in DNNs.
problem Catastrophic forgetting in fine-tuned pretrained models.
method Introducing FAST, an easy-to-implement fine-tuning algorithm.
result FAST learns target tasks faster and retains source knowledge longer.