arXiv research

Unified analysis for decentralized SGD across various topologies and updates.

Paper proves multiplicative weight updates can train neural networks without learning rate tuning.

In several recently proposed stochastic optimization methods (e.g. RMSProp, Adam, Adadelta), parameter updates are scaled by the inverse square roots of exponential moving averages of squared past gradients. Maintaining these per-parameter second-moment estimators requires memory equal to the number of parameters. For …

MetaGrad adapts multiple learning rates for faster online optimization.

Study separates learning rate effects from adaptive gradient methods.

DEAM optimizes momentum weights dynamically to improve deep learning model training.

We present a confidence-based single-layer feed-forward learning algorithm SPIRAL (Spike Regularized Adaptive Learning) relying on an encoding of activation spikes. We adaptively update a weight vector relying on confidence estimates and activation offsets relative to previous activity. We regularize updates proportion…

We present a novel method for convex unconstrained optimization that, without any modifications, ensures: (i) accelerated convergence rate for smooth objectives, (ii) standard convergence rate in the general (non-smooth) setting, and (iii) standard convergence rate in the stochastic optimization setting. To the best of…

Adaptive momentum method solves non-convex min-max problems.

Deep neural networks are traditionally trained using human-designed stochastic optimization algorithms, such as SGD and Adam. Recently, the approach of learning to optimize network parameters has emerged as a promising research topic. However, these learned black-box optimizers sometimes do not fully utilize the experi…

We introduce a general method for improving the convergence rate of gradient-based optimizers that is easy to implement and works well in practice. We demonstrate the effectiveness of the method in a range of optimization problems by applying it to stochastic gradient descent, stochastic gradient descent with Nesterov …

Improved LoRA+ adapts large models more efficiently.

Spectral Adaptive Conformal Prediction for Structured Non-Exchangeable Data

Recent work has established an empirically successful framework for adapting learning rates for stochastic gradient descent (SGD). This effectively removes all needs for tuning, while automatically reducing learning rates over time on stationary problems, and permitting learning rates to grow appropriately in non-stati…

Adaptive OMD reduces variance in learning optimal strategies for imperfect information games.

New algorithm reduces adaptation lag in online model selection.

Adaptive clipping improves privacy in federated learning.

CAdam optimizes online learning by adapting to distribution shifts and noise.

We consider a generic convex-concave saddle point problem with separable structure, a form that covers a wide-ranged machine learning applications. Under this problem structure, we follow the framework of primal-dual updates for saddle point problems, and incorporate stochastic block coordinate descent with adaptive st…

FedDuA adapts global learning rate for federated learning.

This work analyzes $Q$-learning with adaptive stepsizes for finite-time convergence.

Adaptive learning rate improves FTRL's performance in online learning.

A Bayesian estimation of a GARCH model is performed for US Dollar/Japanese Yen exchange rate by the Metropolis-Hastings algorithm with a proposal density given by the adaptive construction scheme. In the adaptive construction scheme the proposal density is assumed to take a form of a multivariate Student's t-distributi…

New adaptive methods solve weakly convex stochastic optimization problems.

One of the main obstacles to broad application of reinforcement learning methods is the parameter sensitivity of our core learning algorithms. In many large-scale applications, online computation and function approximation represent key strategies in scaling up reinforcement learning algorithms. In this setting, we hav…

Paper introduces a simpler gated RNN structure to better capture long-term dependencies.

Optimistic method adapted for faster convex-concave min-max problems.

Adambs adapts Adam to prioritize important training examples.

TDprop uses Jacobi preconditioning to improve adaptive optimizers in Deep RL.

Untuned warmup of Adam performs similarly to RAdam in practical settings.

Adaptive online learning algorithms without manual tuning of a Lipschitz hyperparameter.

AdaS adapts SGD learning rate based on knowledge gain metrics.

Alt-GDA outperforms Sim-GDA in minimax games with near-optimal local convergence.

DynBRO learns robustly from dynamic Byzantine workers.

PCGS-TF uses a Transformer to adaptively control expert switching in non-stationary environments.

Epidemiological model updates infection rates in China, US, Italy.

A new method for efficient neural network fine-tuning using queryable low-rank update atoms.

New adaptive scheduler improves SAM for better model training.

OnlineSCI extends ACI for adaptive selective inference with improved coverage and IER control.

This paper studies a class of adaptive gradient based momentum algorithms that update the search directions and learning rates simultaneously using past gradients. This class, which we refer to as the "Adam-type", includes the popular algorithms such as the Adam, AMSGrad and AdaGrad. Despite their popularity in trainin…

Stochastic gradient methods are dominant in nonconvex optimization especially for deep models but have low asymptotical convergence due to the fixed smoothness. To address this problem, we propose a simple yet effective method for improving stochastic gradient methods named predictive local smoothness (PLS). First, we …

Local adaptive methods in FL can accelerate convergence but introduce bias, which is corrected.

Local update methods' performance depends on learning rates, affecting convergence rates and alignment with true loss.

The paper provides convergence guarantees for multicalibration gradient boosting.

Paper unifies off-policy learning algorithms and introduces C-trace for better trade-offs.

New algorithm reduces best-in-class regret in contextual bandits.

This paper proves AdaGrad and Adam converge linearly under PL inequality.

A new method improves few-shot image classification by updating top layers.