arXiv research

The paper analyzes reinforcement learning methods for estimating weights and quality functions with fast convergence rates.

This paper shows how to learn variational inequalities fast with strong monotonicity.

Empirical risk minimization (ERM) is a fundamental learning rule for statistical learning problems where the data is generated according to some unknown distribution $\mathsf{P}$ and returns a hypothesis $f$ chosen from a fixed class $\mathcal{F}$ with small loss $\ell$. In the parametric setting, depending upon $(\ell…

The speed with which a learning algorithm converges as it is presented with more data is a central problem in machine learning --- a fast rate of convergence means less data is needed for the same level of performance. The pursuit of fast rates in online and statistical learning has led to the discovery of many conditi…

The paper tackles fast rates in structured prediction problems.

Study shows convergence rate for empirical minimizer of unbounded functions with fast growth.

We study fast learning rates when the losses are not necessarily bounded and may have a distribution with heavy tails. To enable such analyses, we introduce two new conditions: (i) the envelope function $\sup_{f \in \mathcal{F}}|\ell \circ f|$, where $\ell$ is the loss function and $\mathcal{F}$ is the hypothesis class…

Study shows fast rates for inverse reinforcement learning with linear rewards.

Error bound conditions (EBC) are properties that characterize the growth of an objective function when a point is moved away from the optimal set. They have recently received increasing attention in the field of optimization for developing optimization algorithms with fast convergence. However, the studies of EBC in st…

We consider the classical problem of learning rates for classes with finite VC dimension. It is well known that fast learning rates up to $O\left(\frac{d}{n}\right)$ are achievable by the empirical risk minimization algorithm (ERM) if low noise or margin assumptions are satisfied. These usually require the optimal Baye…

Deep neural nets estimate operators between infinite-dimensional spaces with fast rates.

When applied to training deep neural networks, stochastic gradient descent (SGD) often incurs steady progression phases, interrupted by catastrophic episodes in which loss and gradient norm explode. A possible mitigation of such events is to slow down the learning process. This paper presents a novel approach to contro…

Paper optimizes multi-fidelity function with fast learning rates.

Paper shows SVM can achieve super fast convergence rates.

Weak labels can significantly speed up learning for strong tasks.

We derive the fast convergence rates of a deep neural network (DNN) classifier with the rectified linear unit (ReLU) activation function learned using the hinge loss. We consider three cases for a true model: (1) a smooth decision boundary, (2) smooth conditional class probability, and (3) the margin condition (i.e., t…

We prove new fast learning rates for the one-vs-all multiclass plug-in classifiers trained either from exponentially strongly mixing data or from data generated by a converging drifting distribution. These are two typical scenarios where training data are not iid. The learning rates are obtained under a multiclass vers…

Paper analyzes faster convergence rates for reinforcement learning from offline data.

A new topology improves decentralized learning efficiency and accuracy.

Paper proposes a boosting method with fast learning rates and early stopping.

Paper shows how meta-learning can reduce prior learning cost.

An infinitely wide model is a weighted integration $\int \varphi(x,v) d μ(v)$ of feature maps. This model excels at handling an infinite number of features, and thus it has been adopted to the theoretical study of deep learning. Kernel quadrature is a kernel-based numerical integration scheme developed for fast approxi…

Improved fast rates for decision making with forward-KL regularization in contextual bandits.

New bounds for multi-task learning with varying task sizes.

In this paper we propose a new method to assist in labeling data arriving from fast running processes using anomaly detection. A result is the possibility to manually classify data arriving at a high rates to train machine learning models. To circumvent the problem of not having a real ground truth we propose specific …

New learning dynamics achieve fast convergence in games without needing to know utility scales.

In this paper, we give a new sharp generalization bound of lp-MKL which is a generalized framework of multiple kernel learning (MKL) and imposes lp-mixed-norm regularization instead of l1-mixed-norm regularization. We utilize localization techniques to obtain the sharp learning rate. The bound is characterized by the d…

FAST improves fast and stable task adaptation in DNNs.

New guarantees for ERM with adaptively collected data.

The developments of Rademacher complexity and PAC-Bayesian theory have been largely independent. One exception is the PAC-Bayes theorem of Kakade, Sridharan, and Tewari (2008), which is established via Rademacher complexity theory by viewing Gibbs classifiers as linear operators. The goal of this paper is to extend thi…

Paper develops an online learning algorithm for functional data models.

Random Fourier features classification achieves fast learning rates with fewer features.

New method resolves density ratio estimation saturation issues.

In theoretical analysis of deep learning, discovering which features of deep learning lead to good performance is an important task. In this paper, using the framework for analyzing the generalization error developed in Suzuki (2018), we derive a fast learning rate for deep neural networks with more general activation …

Proposes a new adaptive learning rate for SGD.

Paper improves a method for fast global and local convergence in optimization.

Unified derivation of PAC-Bayes and MI bounds for general VC classes with fast rates.

Study fast learning rates for square loss in dependent data with hypercontractivity condition.

In this paper, we describe a phenomenon, which we named "super-convergence", where neural networks can be trained an order of magnitude faster than with standard training methods. The existence of super-convergence is relevant to understanding why deep networks generalize well. One of the key elements of super-converge…

We investigate the learning rate of multiple kernel leaning (MKL) with elastic-net regularization, which consists of an $\ell_1$-regularizer for inducing the sparsity and an $\ell_2$-regularizer for controlling the smoothness. We focus on a sparse setting where the total number of kernels is large but the number of non…

Neural operators achieve fast convergence rates for solving PDEs.

The paper explains how data augmentation improves semi-supervised learning efficiency.

In this paper, we give a new generalization error bound of Multiple Kernel Learning (MKL) for a general class of regularizations, and discuss what kind of regularization gives a favorable predictive accuracy. Our main target in this paper is dense type regularizations including \ellp-MKL. According to the recent numeri…

New algorithm reduces best-in-class regret in contextual bandits.

New approach uses 'growth' and 'harvesting' concepts to improve deep learning models.

Optimizing deep neural networks is largely thought to be an empirical process, requiring manual tuning of several hyper-parameters, such as learning rate, weight decay, and dropout rate. Arguably, the learning rate is the most important of these to tune, and this has gained more attention in recent works. In this paper…

The alternating direction method of multipliers (ADMM) is a powerful optimization solver in machine learning. Recently, stochastic ADMM has been integrated with variance reduction methods for stochastic gradient, leading to SAG-ADMM and SDCA-ADMM that have fast convergence rates and low iteration complexities. However,…

Cyclical learning rate improves neural machine translation performance.