Improved analysis for fair federated learning reduces dependence on noise floor.
problem Asymptotic stationarity in group fair federated learning with reduced noise floor dependence.
method DS FedProxGrad framework with inexact local proximal solutions and fairness regularization.
result Algorithm converges asymptotically to stationarity without dependence on a noise floor.
Paper proves SHB convergence with biased gradients and approximate step sizes.
problem Establishing convergence of SHB with biased gradients and approximate step sizes.
method Generalizes SHB convergence conditions for biased gradients, approximate step sizes, and block updating.
result Proves convergence of SHB with new conditions for biased gradients and approximate step sizes.
New method improves Robbins-Monro algorithm convergence with prior information.
problem Improving convergence speed of Robbins-Monro algorithm.
method Integrates prior information into Robbins-Monro iteration without regression model.
result Prior-information Robbins-Monro sequence converges faster than standard.
The need for parameter estimation with massive datasets has reinvigorated interest in stochastic optimization and iterative estimation procedures. Stochastic approximations are at the forefront of this recent development as they yield procedures that are simple, general, and fast. However, standard stochastic approxima…
We formulate simple assumptions, implying the Robbins-Monro conditions for the Q-learning algorithm with the local learning rate, depending on the number of visits of a particular state-action pair (local clock) and the number of iteration (global clock). It is assumed that the Markov decision process is communicatin…
The paper analyzes two ISGD modes for statistical inference, deriving error bounds and confidence intervals.
problem Statistical inference with implicit SGD for smooth convex functions.
method Proximal Robbins-Monro (proxRM) and proximal Polyak-Ruppert (proxPR) procedures for ISGD.
result Derives non-asymptotic error bounds and confidence interval estimators for model parameters.
Paper develops efficient methods for estimating Hessian inverses in stochastic optimization.
problem Estimating the inverse Hessian for convex function minimization.
method Robbins-Monro procedure for recursive estimation of the inverse Hessian.
result Develops universal stochastic Newton methods with improved efficiency.
TCP provides well-calibrated prediction intervals for nonstationary time series.
problem Nonstationary time series forecasting with well-calibrated prediction intervals.
method Temporal Conformal Prediction (TCP) couples a modern quantile forecaster with a rolling split-conformal calibration layer.
result TCP achieves near-nominal coverage, providing slightly wider intervals than Historical Simulation.
Implicit Q-learning and SARSA adjust step-sizes automatically, improving stability and performance.
problem Numerical instability and slow progress in Q-learning and SARSA due to step-size calibration.
method Reformulate iterative updates as fixed-point equations, scaling step-sizes inversely with feature norms.
result Implicit methods maintain stability over broader step-size ranges and achieve comparable convergence rates.
The paper analyzes fixed step-size SA schemes on Riemannian manifolds.
problem Developing efficient algorithms for optimization on curved spaces.
method Fixed step-size stochastic approximation schemes in a Riemannian framework.
result The schemes converge to the solution as the step-size approaches zero.
The paper analyzes and validates two step size schedules for SGD: exponential and cosine, proving their adaptivity and performance.
problem The variability of SGD performance due to step size choice.
method Analysis and empirical evaluation of exponential and cosine step sizes.
result Exponential and cosine step sizes are adaptive to noise and achieve optimal performance without tuning hyperparameters.
In this paper, we introduce a method for adapting the step-sizes of temporal difference (TD) learning. The performance of TD methods often depends on well chosen step-sizes, yet few algorithms have been developed for setting the step-size automatically for TD learning. An important limitation of current methods is that…
New algorithm improves stability of optimization algorithms by adapting step-size.
problem Optimization algorithms' effectiveness is sensitive to step-size hyperparameters.
method Adapts NGN step-size method with momentum to enhance stability.
result Achieves convergence rate of O(1/√K) without restrictive assumptions.
AutoStep MCMC adapts step size locally for better sampling efficiency.
problem Challenging step size selection for complex, multiscale targets.
method AutoStep MCMC uses a locally adaptive step size for involutive proposals.
result AutoStep MCMC is π-invariant, irreducible, and aperiodic.
The CSA-ES is an Evolution Strategy with Cumulative Step size Adaptation, where the step size is adapted measuring the length of a so-called cumulative path. The cumulative path is a combination of the previous steps realized by the algorithm, where the importance of each step decreases with time. This article studies …
Adaptive step-size improves optimization in complex geometries.
problem Optimizing functions with non-Euclidean geometries.
method Adaptive step-size strategy for optimization algorithms.
result Guaranteed convergence for Adaptive Conditional Gradient Descent.
Negative step sizes improve second-order methods for neural networks.
problem Second-order methods discard negative curvature, limiting their effectiveness.
method Introduce negative step sizes in second-order methods combined with Wolfe line search.
result Negative step sizes lead to global convergence and improved performance.
The practical performance of online stochastic gradient descent algorithms is highly dependent on the chosen step size, which must be tediously hand-tuned in many applications. The same is true for more advanced variants of stochastic gradients, such as SAGA, SVRG, or AdaGrad. Here we propose to adapt the step size by …
The main goal of this work is equipping convex and nonconvex problems with Barzilai-Borwein (BB) step size. With the adaptivity of BB step sizes granted, they can fail when the objective function is not strongly convex. To overcome this challenge, the key idea here is to bridge (non)convex problems and strongly convex …
New insights into SGD and SGD-M in high dimensions.
problem Understanding and comparing SGD and SGD-M in high-dimensional settings.
method Developed high-dimensional scaling limits for SGD-M and online SGD, examining their dynamics and performance.
result SGD-M amplifies high-dimensional effects, potentially degrading performance compared to online SGD.
Improved variational inequality algorithms using adaptive step sizes.
problem Solving monotone variational inequalities and convex-concave min-max problems efficiently.
method Adaptive step sizes that eliminate hyperparameters and global Lipschitz continuity requirements.
result Eliminated the need for the golden ratio in the algorithm and improved complexity bounds.
We consider d-dimensional linear stochastic approximation algorithms (LSAs) with a constant step-size and the so called Polyak-Ruppert (PR) averaging of iterates. LSAs are widely applied in machine learning and reinforcement learning (RL), where the aim is to compute an appropriate θ∗∈Rd (that is a…
Sparse coding is typically solved by iterative optimization techniques, such as the Iterative Shrinkage-Thresholding Algorithm (ISTA). Unfolding and learning weights of ISTA using neural networks is a practical way to accelerate estimation. In this paper, we study the selection of adapted step sizes for ISTA. We show t…
New step-size methods improve SHB convergence for stochastic optimization.
problem Tuning step-size and momentum parameters in SHB is challenging.
method Proposed MomSPSmax, MomDecSPS, and MomAdaSPS for SHB. result Convergence guarantees for SHB to solution neighborhoods and exact minimizers.
Adaptive step-size method improves compressed SGD performance in machine learning.
problem Communication bottleneck in distributed and decentralized optimization.
method Developed an adaptive step-size method for compressed SGD.
result Order-optimal convergence rates for various objective functions.
New adaptive step-size method for convex optimization without tuning.
problem Optimizing convex functions efficiently with stochastic gradients.
method Adapted Adaptive Gradient Descent Without Descent to stochastic setting.
result Stochastic gradient descent converges under various assumptions.
One of the major issues in stochastic gradient descent (SGD) methods is how to choose an appropriate step size while running the algorithm. Since the traditional line search technique does not apply for stochastic optimization algorithms, the common practice in SGD is either to use a diminishing step size, or to tune a…
Large SGD step sizes lead to sparse feature learning in neural networks.
problem Sparse feature learning in neural networks with large step sizes.
method Empirical observations and theoretical analysis of SGD dynamics.
result Large step sizes induce implicit regularization leading to sparse predictors.
A recent algorithmic family for distributed optimization, DIGing's, have been shown to have geometric convergence over time-varying undirected/directed graphs. Nevertheless, an identical step-size for all agents is needed. In this paper, we study the convergence rates of the Adapt-Then-Combine (ATC) variation of the DI…
Polyak step size GD reaches final radius of convergence after log iterations.
problem Statistical and computational complexities of Polyak step size GD.
method Generalized smoothness and Lojasiewicz conditions, stability of gradients.
result Polyak step size GD reaches final statistical radius of convergence after logarithmic number of iterations.
WITCHcraft improves PGD attacks with random step size, enhancing efficiency.
problem Efficiently crafting adversarial attacks on neural networks.
method Variant of PGD using random step size.
result Superior performance to classical PGD without additional computational cost.
Develops a generalized version of Chung's Lemma for stochastic optimization methods.
problem Establishing asymptotic convergence rates for stochastic optimization methods under various step size rules.
method Generalized version of Chung's Lemma for a broader family of step size rules.
result Demonstrates tight non-asymptotic convergence rates for various stochastic methods.
The variance reduction class of algorithms including the representative ones, SVRG and SARAH, have well documented merits for empirical risk minimization problems. However, they require grid search to tune parameters (step size and the number of iterations per inner loop) for optimal performance. This work introduces `…
New convergence analysis for ADAM algorithm in non-convex optimization with adaptive step size.
problem Convergence issues in ADAM algorithm for non-convex optimization.
method Study of ADAM algorithm under bounded adaptive step size assumption, providing safe step sizes.
result Novel first order convergence rate result in deterministic and stochastic contexts.
We consider the least-squares regression problem and provide a detailed asymptotic analysis of the performance of averaged constant-step-size stochastic gradient descent (a.k.a. least-mean-squares). In the strongly-convex case, we provide an asymptotic expansion up to explicit exponentially decaying terms. Our analysis…
Proposes a new adaptive learning rate for SGD.
problem Finding an efficient learning rate for SGD.
method Stochastic Polyak step-size (SPS).
result SGD with SPS converges faster for over-parameterized models.
Study optimizes step size for Metropolis algorithm in non-identifiable cases.
problem Optimizing step size for Metropolis algorithm in non-identifiable models.
method Analytical derivation of average acceptance rate for non-identifiable cases.
result Developed optimization principle for step size based on average acceptance rate.
The paper interprets learned step sizes in deep-unfolded gradient descent.
problem Intuitive interpretation of learned non-constant step sizes in deep-unfolded gradient descent.
method Theoretical analysis and optimization of spectral radius.
result Chebyshev steps achieve the lower bound of convergence rate for first-order methods.
New convergence results for NGVI with various step sizes and sample sizes.
problem Understanding convergence of stochastic NGVI for various schedules.
method Projected stochastic NGVI for exponential family variational distributions.
result Geometric convergence and $\mathcal{O}\left(\frac{1}{T^ρ}
ight)$ rates for different schedules.
Proposes a neural network for learning step-size policies for L-BFGS optimization.
problem Optimizing step sizes for L-BFGS in large-scale problems.
method Neural network architecture using local iterate information, trained via stochastic optimization.
result Outperforms existing step size selection methods in training classifiers.
Sparse Polyak improves high-dimensional statistical estimation.
problem High-dimensional statistical estimation problems with growing problem dimension.
method Sparse Polyak modifies Polyak's adaptive step size to estimate restricted Lipschitz smoothness.
result Sparse Polyak achieves optimal statistical precision with fewer iterations.
SGD converges almost surely in non-convex problems, avoiding saddle points and accelerating convergence.
problem Understanding convergence of SGD in non-convex optimization problems.
method Analysis of SGD trajectories, focusing on boundedness, convergence to strict saddle points, and rate of convergence.
result SGD converges almost surely to a minimizer in non-convex problems, avoiding strict saddle points.
Paper develops an online learning algorithm for functional data models.
problem Recovering slope functions or predictors in functional data models.
method Online regularized learning algorithm in reproducing kernel Hilbert spaces with polynomially decaying step-size.
result Established fast convergence rates for estimation error without capacity assumption.
Applying standard Markov chain Monte Carlo (MCMC) algorithms to large data sets is computationally infeasible. The recently proposed stochastic gradient Langevin dynamics (SGLD) method circumvents this problem in three ways: it generates proposed moves using only a subset of the data, it skips the Metropolis-Hastings a…
New TD algorithms stabilize RL tasks by reformulating updates into fixed point equations.
problem TD learning's sensitivity to step size specification.
method Implicit TD algorithms reformulate TD updates into fixed point equations.
result Implicit TD algorithms are more stable and less sensitive to step size.
This manuscript shows that AdaBoost and its immediate variants can produce approximate maximum margin classifiers simply by scaling step size choices with a fixed small constant. In this way, when the unscaled step size is an optimal choice, these results provide guarantees for Friedman's empirically successful "shrink…
Gradient descent with large steps leads to chaotic parameter space and unpredictable outcomes.
problem Understanding the behavior of gradient descent with large step sizes in matrix factorization.
method Analyzing the fractal structure of the parameter space and deriving critical step sizes for convergence.
result Gradient descent with large steps exhibits chaotic behavior and sensitivity to initialization, creating a fractal boundary between converging and diverging minimizers.
Proposes an exponentially increasing step-size for faster parameter estimation in statistical models.
problem Slow convergence of gradient descent in locally convex loss functions.
method Exponentially increasing step-size in gradient descent algorithm.
result Converges linearly to optimal solution under homogeneous assumptions.