arXiv research

FIRE PBT improves neural network training by focusing on long-term performance.

We propose the stochastic average gradient (SAG) method for optimizing the sum of a finite number of smooth convex functions. Like stochastic gradient (SG) methods, the SAG method's iteration cost is independent of the number of terms in the sum. However, by incorporating a memory of previous gradient values the SAG me…

SALR improves deep learning generalization by dynamically adjusting learning rates.

This is our second paper in a series to study gravitational instantons, i.e. complete hyperkäler 4-manifolds with faster than quadratic curvature decay. We prove two main theorems: 1.The asymptotic rate of gravitational instantons to the standard models can be improved automatically. 2.Any ALF-D_k gravitational instant…

We provide improved convergence rates for various \emph{non-smooth} optimization problems via higher-order accelerated methods. In the case of $\ell_\infty$ regression, we achieves an $O(ε^{-4/5})$ iteration complexity, breaking the $O(ε^{-1})$ barrier so far present for previous methods. We arrive at a similar rate fo…

New algorithms optimize private convex optimization with faster rates for functions with κ-growth.

Private optimization faster on interpolation problems with quadratic growth.

Faster algorithms for solving multichain MDPs under average-reward criterion.

There has been significant recent work on the theory and application of randomized coordinate descent algorithms, beginning with the work of Nesterov [SIAM J. Optim., 22(2), 2012], who showed that a random-coordinate selection rule achieves the same convergence rate as the Gauss-Southwell selection rule. This result su…

Riemannian stochastic gradient descent converges faster with increasing batch size.

Nesterov's accelerated gradient descent (AGD), an instance of the general family of "momentum methods", provably achieves faster convergence rate than gradient descent (GD) in the convex setting. However, whether these methods are superior to GD in the nonconvex setting remains open. This paper studies a simple variant…

Paper improves learning rates for GSC loss functions using iterated Tikhonov regularization.

Improved robust regression algorithms with faster runtime and better estimation rates.

Improved SGD methods converge faster for nonconvex optimization.

Faster algorithms solve convex function learning problems.

We study Frank-Wolfe methods for nonconvex stochastic and finite-sum optimization problems. Frank-Wolfe methods (in the convex case) have gained tremendous recent interest in machine learning and optimization communities due to their projection-free property and their ability to exploit structured constraints. However,…

learn2mix trains neural nets faster by adjusting class proportions dynamically.

Improved first-order algorithm for entropy regularized OT with faster convergence.

A new algorithm improves convergence rates for convex optimization problems.

A faster ADMM method for nonconvex optimization with improved complexity.

LALR adapts learning rate for faster convergence in regression and neural nets.

Warm-up improves training by adapting learning rate based on curvature.

FedLion improves Federated Learning by speeding up convergence and reducing communication costs.

New method tracks significant arm switches to improve bandit algorithms.

Armijo line-search speeds up gradient descent for various functions.

Improved private AdaGrad achieves faster convergence rates for convex functions.

In this paper, we present the Bennett-type generalization bounds of the learning process for i.i.d. samples, and then show that the generalization bounds have a faster rate of convergence than the traditional results. In particular, we first develop two types of Bennett-type deviation inequality for the i.i.d. learning…

Improved LV model for interest rate swaptions and caplets.

We present a unified framework for low-rank matrix estimation with nonconvex penalties. We first prove that the proposed estimator attains a faster statistical rate than the traditional low-rank matrix estimator with nuclear norm penalty. Moreover, we rigorously show that under a certain condition on the magnitude of t…

A new topology improves decentralized learning efficiency and accuracy.

Batch normalization (BN) is a technique to normalize activations in intermediate layers of deep neural networks. Its tendency to improve accuracy and speed up training have established BN as a favorite technique in deep learning. Yet, despite its enormous success, there remains little consensus on the exact reason and …

New algorithm finds approximate stationary points faster under differential privacy constraints.

We study in this paper the rate of convergence for learning densities under the Generative Adversarial Networks (GAN) framework, borrowing insights from nonparametric statistics. We introduce an improved GAN estimator that achieves a faster rate, through simultaneously leveraging the level of smoothness in the target d…

Paper proposes MS-k-NN for improved convergence rate in k-NN classification.

This paper investigates the nonparametric regression problem using SVMs with anisotropic Gaussian RBF kernels. Under the assumption that the target functions are resided in certain anisotropic Besov spaces, we establish the almost optimal learning rates, more precisely, optimal up to some logarithmic factor, presented …

We prove that, under low noise assumptions, the support vector machine with $N\ll m$ random features (RFSVM) can achieve the learning rate faster than $O(1/\sqrt{m})$ on a training set with $m$ samples when an optimized feature map is used. Our work extends the previous fast rate analysis of random features method from…

Stochastic methods with coordinate-wise adaptive stepsize (such as RMSprop and Adam) have been widely used in training deep neural networks. Despite their fast convergence, they can generalize worse than stochastic gradient descent. In this paper, by revisiting the design of Adagrad, we propose to split the network par…

Stochastic gradient descent (SGD) gives an optimal convergence rate when minimizing convex stochastic objectives $f(x)$. However, in terms of making the gradients small, the original SGD does not give an optimal rate, even when $f(x)$ is convex. If $f(x)$ is convex, to find a point with gradient norm $\varepsilon$, we …

The paper improves the probability flow ODE sampler for faster sampling of natural images.

RR algorithm improves convergence rate without strong convexity assumptions.

Bayesian rating system for large competitions improves prediction and efficiency.

Gradient boosting improved with lassoed trees achieves faster convergence.

Adaptive gradient methods converge faster with over-parameterization and line-search.

New methods boost first-order optimization with faster rates.

Different technological domains have significantly different rates of performance improvement. Prior theory indicates that such differing rates should influence the relative speed of diffusion of the products embodying the different technologies since improvement in performance during the diffusion process increases th…

New algorithms solve DR-submodular maximization with faster convergence.

Paper improves a method for fast global and local convergence in optimization.

Paper speeds up IoT device detection and data decoding.