Efficient RNN algorithm guarantees convergence in online learning.
problem Online nonlinear regression with RNNs.
method First-order training algorithm with convergence guarantee.
result The algorithm converges to optimum network parameters.
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “first-order convergence”
OSGM uses online learning to adapt stepsize for faster convergence.
problem Improving convergence rates of first-order methods.
method OSGM combines online learning and feedback functions to adjust stepsize.
result OSGM achieves convergence rates asymptotically no worse than optimal.
This paper analyzes convergence of RMSProp and Adam in non-convex optimization with tight complexity bounds.
problem Analyzing convergence of RMSProp and Adam in non-convex optimization with relaxed assumptions.
method Developed new convergence analyses for RMSProp and Adam, considering adaptive learning rates and affine noise variance.
result RMSProp and Adam converge to ε-stationary points with iteration complexities of O(ε^(-4)) under proper hyperparameters.
First-order method solves stochastic bilevel optimization with linear constraints.
problem Stochastic bilevel optimization with linear constraints and noise.
method Developed a novel framework using gradient-based techniques and smoothed penalty functions.
result Achieved finite-time convergence guarantees for -Goldstein stationary points.
We simplify deep learning convergence analysis using basic math.
problem Ensuring reliable convergence of optimization algorithms in deep networks.
method Elementary arguments and computations to estimate smoothness constants.
result Systematic computation of smoothness constants for first-order algorithms.
The purpose of this paper is to provide a sharp analysis on the asymptotic behavior of the Durbin-Watson statistic. We focus our attention on the first-order autoregressive process where the driven noise is also given by a first-order autoregressive process. We establish the almost sure convergence and the asymptotic n…
First Order Methods take Exponential Time to Converge to Global Minimizers of Non-Convex Functionscs.LG
First order methods can take extremely long to find global minima of non-convex functions.
problem Finding global minimizers of non-convex functions.
method Designing a family of non-convex functions and using statistical lower bounds for parameter estimation.
result First order methods can take exponential time to converge to a global minimizer.
Many problems in machine learning and game theory can be formulated as saddle-point problems, for which various first-order methods have been developed and proven efficient in practice. Under the general convex-concave assumption, most first-order methods only guarantee an ergodic convergence rate, that is, the uniform…
A Unified Analysis of First-Order Methods for Smooth Games via Integral Quadratic Constraintsmath.OC
Unified analysis of first-order methods for smooth games using IQCs.
problem Certify convergence rates of first-order methods for smooth and strongly-monotone games.
method Adapted integral quadratic constraints (IQCs) to study first-order methods and derive tight upper bounds of convergence rates.
result First global convergence rate for the negative momentum method with iteration complexity.
In this paper, we consider first-order convergence theory and algorithms for solving a class of non-convex non-concave min-max saddle-point problems, whose objective function is weakly convex in the variables of minimization and weakly concave in the variables of maximization. It has many important applications in mach…
New method accelerates convergence for entropy-regularized reinforcement learning problems.
problem Slow convergence of standard first-order methods for entropy-regularized Markov decision processes.
method Introduce a quadratically convexified primal-dual formulation and a new interpolating metric to accelerate convergence.
result Global convergence and exponential convergence rate for the new method.
A Generic First-Order Algorithmic Framework for Bi-Level Programming Beyond Lower-Level Singletoncs.LG
New framework tackles bi-level optimization without LLS condition.
problem Bi-level optimization problems without LLS condition.
method Bi-level Descent Aggregation (BDA) framework, proving convergence without LLS condition.
result Proves convergence of BDA without LLS condition.
Computing Nash equilibrium (NE) of multi-player games has witnessed renewed interest due to recent advances in generative adversarial networks. However, computing equilibrium efficiently is challenging. To this end, we introduce the Gradient-based Nikaido-Isoda (GNI) function which serves: (i) as a merit function, vani…
The paper analyzes reinforcement learning methods for estimating weights and quality functions with fast convergence rates.
problem Estimating weights and quality functions in reinforcement learning with function approximation.
method The paper uses minimax methods for estimating marginal importance weights and q-functions.
result The minimax approach enables fast rates of convergence for weights and quality functions, achieving first-order efficiency.
Logistic regression is one of the most popular methods in binary classification, wherein estimation of model parameters is carried out by solving the maximum likelihood (ML) optimization problem, and the ML estimator is defined to be the optimal solution of this problem. It is well known that the ML estimator exists wh…
New method improves FO-BLO convergence without increasing memory or time complexity.
problem Lack of theoretical understanding of FO-BLO convergence.
method Unbiased first-order bilevel optimization (UFO-BLO) to improve convergence.
result Theoretical guarantee of convergence for FO-BLO-based stochastic optimization.
Unified framework for analyzing batch updating methods with noisy gradients.
problem Analyzing convergence of batch updating methods with noisy gradients and approximations.
method Unified framework using convergence of stochastic processes.
result Establishes a general theorem for most known convergence results.
In this paper, we study the Kurdyka-Łojasiewicz (KL) exponent, an important quantity for analyzing the convergence rate of first-order methods. Specifically, we develop various calculus rules to deduce the KL exponent of new (possibly nonconvex and nonsmooth) functions formed from functions with known KL exponents. In …
Hamiltonian Descent Methodsmath.OC
We propose a family of optimization methods that achieve linear convergence using first-order gradient information and constant step sizes on a class of convex functions much larger than the smooth and strongly convex ones. This larger class includes functions whose second derivatives may be singular or unbounded at th…
New algorithm guarantees optimal convergence rate for stochastic optimization.
problem Optimal convergence rate for stochastic optimization algorithms.
method Regularized versions of Minimization by Incremental Surrogate Optimization (MISO) with arbitrary recurrent data sampling.
result Expected optimality gap converges at under general recurrent sampling schemes.
Develops first-order methods for average-reward MDPs with strong guarantees.
problem Lack of strong theoretical guarantees for first-order methods in AMDPs.
method Average-reward stochastic policy mirror descent (SPMD) and variance-reduced temporal difference (VRTD) methods.
result Establishes sample complexity results for solving AMDPs.
New method solves saddle-point problems faster than existing methods.
problem Large-scale saddle-point problems in optimization.
method Sequential subspace optimization with proximal regularization.
result Significantly better convergence compared to first-order methods.
The filtering-clustering models, including trend filtering and convex clustering, have become an important source of ideas and modeling tools in machine learning and related fields. The statistical guarantee of optimal solutions in these models has been extensively studied yet the investigations on the computational as…
Develops a first-order interior-point method for solving constrained variational inequalities.
problem Solving constrained variational inequalities with nontrivial constraints.
method ADMM-based interior-point method for constrained VIs (ACVI).
result First-order interior-point method with global convergence guarantees for general cVI problems.
New insights into convergence and accuracy trade-offs in federated and meta-learning.
problem Understanding the trade-offs between convergence and accuracy in federated and meta-learning.
method Generalized local update methods, proving equivalence to first-order optimization on a surrogate loss.
result Novel convergence rates and insights into the importance of algorithmic choices in communication-limited settings.
Derives FACT, an alternative to NFA for neural networks, explaining feature learning.
problem Understanding how neural networks learn representations.
method First-principles approach using first-order optimality conditions.
result FACT explains why NFA holds and provides a principled alternative.
BMM algorithm improves convergence for nonconvex optimization problems.
problem Constrained nonsmooth nonconvex optimization problems.
method Block majorization-minimization with diminishing radius.
result Improved convergence rate for nonconvex optimization problems.
This paper studies quasar-convex functions to improve optimization methods.
problem Improving optimization methods for non-convex functions.
method Study of first order methods for quasar-convex functions.
result Proves complexity upper bounds similar to convex functions.
New methods boost first-order optimization with faster rates.
problem Designing efficient first-order methods for convex problems.
method Shifted objective function with interpolation condition.
result New schemes achieve faster convergence rates.
Gradient descent finds global optima in ResNets with sufficient parameters.
problem Finding optimal parameters in ResNet models.
method Mean-field analysis and gradient-flow PDE to study convergence of first-order optimization methods.
result First-order methods can find global minimizers in overparameterized ResNets.
Last-Iterate Convergence of Saddle-Point Optimizers via High-Resolution Differential Equationsmath.OC
New ODE models show saddle-point optimization methods converge differently, with last-iterate convergence for OGDA.
problem Analyzing convergence properties of saddle-point optimization methods.
method High-Resolution Differential Equations (HRDEs) to design differential equation models for saddle-point optimization methods.
result HRDEs reveal last-iterate convergence for Optimistic Gradient Descent Ascent (OGDA) in bilinear games.
In this paper we study general Schatten- quasi-norm (SPQN) regularized matrix minimization problems. In particular, we first introduce a class of first-order stationary points for them, and show that the first-order stationary points introduced in [11] for an SPQN regularized minimization problem are equiva…
In this paper, we study optimization methods consisting of iteratively minimizing surrogates of an objective function. By proposing several algorithmic variants and simple convergence analyses, we make two main contributions. First, we provide a unified viewpoint for several first-order optimization techniques such as …
SP-NGD improves deep learning models' generalization with large mini-batch sizes.
problem Worse generalization performance with large mini-batch sizes in deep learning.
method SP-NGD, a natural gradient descent approach for large-scale deep learning.
result SP-NGD achieves similar generalization performance to first-order methods with accelerated convergence and negligible overhead.
Geodesic convexity generalizes the notion of (vector space) convexity to nonlinear metric spaces. But unlike convex optimization, geodesically convex (g-convex) optimization is much less developed. In this paper we contribute to the understanding of g-convex optimization by developing iteration complexity analysis for …
Two classes of methods have been proposed for escaping from saddle points with one using the second-order information carried by the Hessian and the other adding the noise into the first-order information. The existing analysis for algorithms using noise in the first-order information is quite involved and hides the es…
We consider empirical risk minimization of linear predictors with convex loss functions. Such problems can be reformulated as convex-concave saddle point problems, and thus are well suitable for primal-dual first-order algorithms. However, primal-dual algorithms often require explicit strongly convex regularization in …
A new algorithm for solving constrained convex optimization problems efficiently.
problem Constrained convex optimization problems requiring high accuracy solutions.
method Second-Order Conditional Gradient Sliding (SOCGS) algorithm, using projection-free methods to solve quadratic subproblems inexactly.
result Converges quadratically in primal gap after a finite number of linearly convergent iterations.
A new algorithm for decentralized optimization over directed graphs.
problem Decentralized stochastic optimization over directed networks.
method Gradient tracking and S-ADDOPT algorithm with constant and decaying step-sizes.
result S-ADDOPT converges linearly with constant step-size and sublinearly with decaying step-size.
Motivated by applications in Optimization, Game Theory, and the training of Generative Adversarial Networks, the convergence properties of first order methods in min-max problems have received extensive study. It has been recognized that they may cycle, and there is no good understanding of their limit points when they…
Study revisits AdaGrad convergence with relaxed noise assumptions.
problem Non-convex smooth optimization problems with general noise.
method General noise model with function value gap and gradient magnitude control.
result Probabilistic convergence rate of ( ilde{\mathcal{O}}(1/\sqrt{T})) under general noise.
Efficient First-Order Contextual Bandits: Prediction, Allocation, and Triangular Discriminationcs.LG
Efficient algorithm for contextual bandits with first-order guarantees.
problem Adapting to low noise in contextual bandits.
method Reduction to online regression with logarithmic loss.
result Optimal and efficient first-order guarantees for contextual bandits.
First-order methods such as stochastic gradient descent (SGD) are currently the standard algorithm for training deep neural networks. Second-order methods, despite their better convergence rate, are rarely used in practice due to the prohibitive computational cost in calculating the second-order information. In this pa…
In this work we introduce a conditional accelerated lazy stochastic gradient descent algorithm with optimal number of calls to a stochastic first-order oracle and convergence rate improving over the projection-free, Online Frank-Wolfe based stochastic gradient descent of Hazan an…
We provide two fundamental results on the population (infinite-sample) likelihood function of Gaussian mixture models with components. Our first main result shows that the population likelihood function has bad local maxima even in the special case of equally-weighted mixtures of well-separated and spherical…
New RL algorithms learn policies competitive with best in class without assuming optimal policy.
problem Agnostic RL where best policy not necessarily optimal.
method First-order optimization in non-Euclidean space, reducing to policy learning.
result Sample complexity upper bounds for three algorithms under VGD condition.
A new first-order sampler improves diffusion probabilistic model sampling quality.
problem The belief that first-order methods are inherently slower for diffusion probabilistic model sampling.
method A novel training-free, first-order sampler that approximates the forward-value evaluation via a one-step lookahead predictor.
result The proposed sampler provably approximates the ideal forward-value trajectory while retaining first-order convergence and can improve sample quality under the same NFE budget.
LMC algorithm converges to target in Chi-squared and Renyi divergence.
problem Sampling from target distribution using LMC with strong dissipativity and smoothness conditions.
method LMC algorithm with strong dissipativity and first-order smoothness, initialized with Gaussian.
result LMC reaches ε-neighborhood of target in Chi-squared and Renyi divergence in O(λ²dε⁻¹) steps.