arXiv research

The paper tackles fast rates in structured prediction problems.

Empirical risk minimization (ERM) is a fundamental learning rule for statistical learning problems where the data is generated according to some unknown distribution $\mathsf{P}$ and returns a hypothesis $f$ chosen from a fixed class $\mathcal{F}$ with small loss $\ell$. In the parametric setting, depending upon $(\ell…

Paper shows SVM can achieve super fast convergence rates.

The paper analyzes reinforcement learning methods for estimating weights and quality functions with fast convergence rates.

Study shows convergence rate for empirical minimizer of unbounded functions with fast growth.

The developments of Rademacher complexity and PAC-Bayesian theory have been largely independent. One exception is the PAC-Bayes theorem of Kakade, Sridharan, and Tewari (2008), which is established via Rademacher complexity theory by viewing Gibbs classifiers as linear operators. The goal of this paper is to extend thi…

This paper shows how to learn variational inequalities fast with strong monotonicity.

The speed with which a learning algorithm converges as it is presented with more data is a central problem in machine learning --- a fast rate of convergence means less data is needed for the same level of performance. The pursuit of fast rates in online and statistical learning has led to the discovery of many conditi…

Error bound conditions (EBC) are properties that characterize the growth of an objective function when a point is moved away from the optimal set. They have recently received increasing attention in the field of optimization for developing optimization algorithms with fast convergence. However, the studies of EBC in st…

Paper improves a method for fast global and local convergence in optimization.

Unified derivation of PAC-Bayes and MI bounds for general VC classes with fast rates.

Improved fast rates for decision making with forward-KL regularization in contextual bandits.

Study shows fast rates for inverse reinforcement learning with linear rewards.

New algorithm reduces best-in-class regret in contextual bandits.

We study fast learning rates when the losses are not necessarily bounded and may have a distribution with heavy tails. To enable such analyses, we introduce two new conditions: (i) the envelope function $\sup_{f \in \mathcal{F}}|\ell \circ f|$, where $\ell$ is the loss function and $\mathcal{F}$ is the hypothesis class…

Paper optimizes multi-fidelity function with fast learning rates.

We consider the classical problem of learning rates for classes with finite VC dimension. It is well known that fast learning rates up to $O\left(\frac{d}{n}\right)$ are achievable by the empirical risk minimization algorithm (ERM) if low noise or margin assumptions are satisfied. These usually require the optimal Baye…

Deep neural nets estimate operators between infinite-dimensional spaces with fast rates.

Investigates fast prediction rates with limited expert advice.

We consider the stochastic contextual bandit problem with additional regularization. The motivation comes from problems where the policy of the agent must be close to some baseline policy which is known to perform well on the task. To tackle this problem we use a nonparametric model and propose an algorithm splitting t…

We derive the fast convergence rates of a deep neural network (DNN) classifier with the rectified linear unit (ReLU) activation function learned using the hinge loss. We consider three cases for a true model: (1) a smooth decision boundary, (2) smooth conditional class probability, and (3) the margin condition (i.e., t…

An infinitely wide model is a weighted integration $\int \varphi(x,v) d μ(v)$ of feature maps. This model excels at handling an infinite number of features, and thus it has been adopted to the theoretical study of deep learning. Kernel quadrature is a kernel-based numerical integration scheme developed for fast approxi…

When applied to training deep neural networks, stochastic gradient descent (SGD) often incurs steady progression phases, interrupted by catastrophic episodes in which loss and gradient norm explode. A possible mitigation of such events is to slow down the learning process. This paper presents a novel approach to contro…

Paper analyzes faster convergence rates for reinforcement learning from offline data.

Research on nearest-neighbor methods tends to focus somewhat dichotomously either on the statistical or the computational aspects -- either on, say, Bayes consistency and rates of convergence or on techniques for speeding up the proximity search. This paper aims at bridging these realms: to reap the advantages of fast …

Noise affects the effectiveness of interpolating models, especially those with strong inductive biases.

The gauge theory of arbitrage was introduced by Ilinski in [arXiv:hep-th/9710148] and applied to fast money flows in [arXiv:cond-mat/9902044]. The theory of fast money flow dynamics attempts to model the evolution of currency exchange rates and stock prices on short, e.g.\ intra-day, time scales. It has been used to ex…

We prove new fast learning rates for the one-vs-all multiclass plug-in classifiers trained either from exponentially strongly mixing data or from data generated by a converging drifting distribution. These are two typical scenarios where training data are not iid. The learning rates are obtained under a multiclass vers…

This paper proposes a new method for estimating sparse precision matrices in the high dimensional setting. It has been popular to study fast computation and adaptive procedures for this problem. We propose a novel approach, called Sparse Column-wise Inverse Operator, to address these two issues. We analyze an adaptive …

TD(0) with Polyak-Ruppert averaging achieves robust and fast convergence rates

A new topology improves decentralized learning efficiency and accuracy.

Paper proposes a boosting method with fast learning rates and early stopping.

Algorithm minimizes risk for multiclass classification of stochastic diffusion paths.

The effect of errors in variables in quantization is investigated. We prove general exact and non-exact oracle inequalities with fast rates for an empirical minimization based on a noisy sample $Z_i=X_i+ε_i,i=1,\ldots,n$, where $X_i$ are i.i.d. with density $f$ and $ε_i$ are i.i.d. with density $η$. These rates depend …

This research examines how the error rate of nearest neighbor classifiers varies with dataset size.

In this paper we propose a new method to assist in labeling data arriving from fast running processes using anomaly detection. A result is the possibility to manually classify data arriving at a high rates to train machine learning models. To circumvent the problem of not having a real ground truth we propose specific …

Paper shows how meta-learning can reduce prior learning cost.

New bounds for multi-task learning with varying task sizes.

In this work we introduce a new optimisation method called SAGA in the spirit of SAG, SDCA, MISO and SVRG, a set of recently proposed incremental gradient algorithms with fast linear convergence rates. SAGA improves on the theory behind SAG and SVRG, with better theoretical convergence rates, and has support for compos…

Neural operators achieve fast convergence rates for solving PDEs.

Linear-Core Surrogates combine fast optimization and statistical efficiency in classification and structured prediction.

Develops a fast variational approximation for high-dimensional empirical Bayes posteriors.

New learning dynamics achieve fast convergence in games without needing to know utility scales.

PPGD solves nonconvex nonsmooth optimization problems without KL property.

We study generalization properties of weakly supervised learning. That is, learning where only a few "strong" labels (the actual target of our prediction) are present but many more "weak" labels are available. In particular, we show that having access to weak labels can significantly accelerate the learning rate for th…

The alternating direction method of multipliers (ADMM) is a powerful optimization solver in machine learning. Recently, stochastic ADMM has been integrated with variance reduction methods for stochastic gradient, leading to SAG-ADMM and SDCA-ADMM that have fast convergence rates and low iteration complexities. However,…

New findings on flatness of certain metrics with fast decay.

FAST improves fast and stable task adaptation in DNNs.