Coordinate descent algorithms solve optimization problems by minimizing along coordinates.
problem Solving large-scale optimization problems in data science and engineering.
method Solve optimization problems by successively minimizing along each coordinate or coordinate hyperplane.
result Coordinate descent algorithms are effective for parallelized and distributed computing.
Accelerates coordinate descent methods for machine learning problems.
problem Slowness of coordinate descent methods in machine learning.
method Extrapolation-based accelerated coordinate descent.
result Significant speed-up in practice compared to existing methods.
Meta-descent improves online prediction in non-stationary settings.
problem Non-stationary online prediction problems.
method Meta-gradient descent to adapt step-size parameters.
result Meta-descent methods outperform other approaches in non-stationary prediction.
Stochastic Gradient Descent prefers minimizers with flat basins in nonconvex problems.
problem Understanding why SGD prefers minimizers with flat basins in nonconvex problems.
method Detailed analysis of a generic stochastic quadratic problem, deriving a deterministic mechanism.
result Derives a deterministic mechanism explaining why SGD prefers flat minimizers.
Paper proposes a pre-conditioning technique to speed up gradient-descent convergence in distributed linear least-squares problems.
problem Expediting convergence of gradient-descent method for ill-conditioned distributed linear least-squares problems.
method Iterative pre-conditioning technique to improve convergence rate of gradient-descent method.
result Pre-conditioned gradient-descent achieves superlinear convergence for unique solutions and improved linear convergence otherwise.
Blind Descent avoids gradient issues, using a different learning approach.
problem Gradient issues like exploding and vanishing gradients.
method Does not use gradients to guide learning; instead, it is a more fundamental learning process.
result Gradient descent is a specific case of Blind Descent.
New algorithm reduces optimization complexity in adaptive mirror descent.
problem Optimizing complex, non-smooth, non-convex functions efficiently.
method SVRAMD: Variance Reduced Adaptive Mirror Descent.
result Variance reduction accelerates convergence in adaptive mirror descent.
Paper proves linear convergence of R-FDM and RC-FDM under weak strong convexity.
problem Optimizing SVM dual problem and LASSO problem.
method Randomized feasible descent method (R-FDM) and coordinate-wise random feasible descent method (RC-FDM).
result Both R-FDM and RC-FDM converge linearly under weak strong convexity assumption.
Gradient descent and SGD solve nonlinear inverse problems efficiently.
problem Solving nonlinear inverse problems with random design.
method Gradient descent and SGD with mini-batching, under classical assumptions.
result Achieves optimal convergence rates in RKHS framework.
Gradient descent optimizes deep ReLU networks with proper initialization.
problem Training deep neural networks with ReLU activation.
method Gradient descent and stochastic gradient descent with proper random weight initialization.
result Gradient descent finds global minima for over-parameterized deep ReLU networks.
Langevin algorithms enhance training of deep neural networks for stochastic control problems.
problem Training acceleration for deep neural networks in stochastic control problems.
method Application of Langevin algorithms to minimize the loss of deep neural networks in stochastic control problems.
result Langevin algorithms improve training on various stochastic control problems.
Continuous-time mirror descent solves sparse phase retrieval efficiently.
problem Recovering sparse signals from magnitude-only measurements.
method Continuous-time mirror descent applied to unconstrained empirical risk minimization problem.
result Mirror descent recovers k k k -sparse vectors with minimum non-zero entry order of ∥ x ⋆ ∥ 2 / k \| \mathbf{x}^\star \|_2/\sqrt{k} ∥ x ⋆ ∥ 2 / k from k 2 k^2 k 2 Gaussian measurements. Extends geometric descent method for convex composite problems.
problem Nonsmooth and strongly convex composite problems.
method Geometric Proximal Gradient Method (GeoPG)
result Achieves optimal linear convergence rate of (1-1/\sqrtκ).
Reparameterizes mirror descent as gradient descent for efficient sparse learning.
problem Efficiently training small sparse networks with mirror descent.
method Develops a framework to convert mirror descent updates into gradient descent updates on different parameters.
result Mirror descent can be reparameterized as gradient descent on modified parameters, facilitating standard backpropagation.
Gradient descent variants improve phase retrieval accuracy.
problem Phase retrieval problem in high-dimensional spaces.
method Gradient descent, stochastic gradient descent, Langevin algorithm, dynamical mean-field theory.
result Stochastic variants of gradient descent achieve better generalization in phase retrieval.
Proposes Hebbian-descent for neural network learning, addressing Hebbian and gradient descent issues.
problem Learning issues with correlated data and vanishing error term in gradient descent.
method Integrates Hebbian and gradient descent principles without activation function derivatives, centering neural activities.
result Biologically plausible, convergent, and effective in online learning with correlated data.
Distributed block coordinate descent solves large kernel problems.
problem Large scale kernel learning problems.
method Distributed block coordinate descent for kernel regression and classification.
result Nyström method generally achieves better statistical accuracy than random features.
Gradient descent with random initialization solves phase retrieval problems efficiently.
problem Solving systems of quadratic equations for phase retrieval.
method Gradient descent with random initialization for nonconvex least squares problem.
result Gradient descent achieves near-optimal computational and sample complexities for phase retrieval.
Stochastic gradient descent on manifolds improves low-rank approximation.
problem Efficiently approximate large matrices with lower rank.
method Stochastic gradient descent on a manifold.
result Algorithm outperforms Euclidean space methods on Netflix Prize data.
Gradient descent learns two-layer networks well for classification problems.
problem Understanding the performance of gradient descent on classification problems.
method Refined convergence analysis of gradient descent for two-layer networks with smooth activations.
result Gradient descent can learn less over-parameterized networks for classification problems.
Optimistic mirror descent improves convergence in saddle-point problems.
problem Training generative adversarial networks (GANs) with saddle-point problems.
method Analyzed mirror descent (MD) and optimistic mirror descent (OMD) in coherent non-monotone problems.
result Optimistic mirror descent converges in all coherent problems, improving upon vanilla MD.
Efficient method solves large-scale saddle point problems with parallel updates.
problem Large-scale convex-concave saddle point problems with separable structure.
method Stochastic parallel block coordinate descent with adaptive primal-dual updates.
result Significantly better performance than state-of-the-art methods in various applications.
New method for faster optimization of convex functions.
problem Optimizing finite sums of smooth convex functions efficiently.
method Accelerated gradient descent combined with stochastic variance reduction in mini-batches.
result Achieves lower overall complexity and faster convergence rates.
Paper studies early-stopped mirror descent for noisy sparse phase retrieval.
problem Recovering a sparse signal from noisy quadratic measurements.
method Early-stopped mirror descent with hyperbolic entropy mirror map.
result Achieves nearly minimax-optimal rate of convergence for k k k -sparse signals. New method improves convergence of gradient descent for non-convex, non-reversible Markov chains.
problem Improving convergence of gradient descent for non-convex, non-reversible Markov chains.
method Introducing a new technique that varies the mixing levels of the Markov chains to establish non-ergodic convergence under wider step sizes.
result Established non-ergodic convergence for non-convex problems and non-reversible finite-state Markov chains.
Paper proposes a pre-conditioning method to speed up gradient descent in multi-agent optimization.
problem Speed up convergence of gradient descent in multi-agent optimization problems.
method Iterative pre-conditioning approach to mitigate the effect of problem conditioning.
result Significant improvement in convergence speed of gradient descent method.
Gradient descent works for low-rank matrix problems under certain conditions.
problem Optimization problems with rank constraints in matrix regression, PCA, completion, and decomposition.
method Projected gradient descent on nonconvex factorized optimization problems.
result Projected gradient descent converges geometrically to a statistically useful solution under certain conditions.
Stochastic mirror descent improves multi-class classification accuracy.
problem Efficient multi-class classification using stochastic optimization.
method Derive risk bounds for stochastic mirror descent algorithm.
result Algorithm provides efficient error bounds for classification.
Simple rules ensure gradient descent adapts to local geometry, converging for convex and nonconvex problems.
problem Minimizing convex and nonconvex functions efficiently.
method Two rules: don't increase stepsize too fast and don't overstep local curvature.
result Method converges for convex and nonconvex problems, even with infinite global smoothness.
Paper analyzes fast SAGA method for nonconvex optimization problems.
problem Optimizing nonconvex problems of the form min x ∑ i f i ( x ) \min_x \sum_i f_i(x) min x ∑ i f i ( x ) method Incremental aggregated gradient method (SAGA) within an Incremental First-order Oracle framework
result SAGA converges to a stationary point faster than gradient descent and stochastic gradient descent, and at a linear rate to the global optimum for a specific class of nonconvex problems.
Grassmannian gradient descent solves subspace estimation problems.
problem Estimating a d d d -dimensional subspace from streaming data. method Incremental gradient descent constrained to the Grassmannian manifold with adaptive step sizes.
result Converges from any initialization to the global minimum for noise-free data.
Study shows double descent curve in high-dimensional linear regression with random projections.
problem Understanding the generalization performance in high-dimensional settings with random projections.
method Fixed prediction problem, ridge regression estimator, minimum norm least-squares fit, random matrix theory, asymptotic equivalents.
result Exhibit a double descent curve for high-dimensional linear regression with random projections.
New method learns good initialization for gradient descent from past solutions.
problem Challenges in solving non-convex optimization problems.
method Learning good initialization rules from previous solutions.
result Our approach performs better than random initialization in various non-convex problems.
Information geometry applies concepts in differential geometry to probability and statistics and is especially useful for parameter estimation in exponential families where parameters are known to lie on a Riemannian manifold. Connections between the geometric properties of the induced manifold and statistical properti…
This work accelerates gradient descent with anytime convergence guarantees.
problem Improving the convergence rate of gradient descent methods.
method Proposes a stepsize schedule for gradient descent that achieves anytime convergence rates.
result Gradient descent can achieve convergence rates of O ( T − 1.119 ) O(T^{-1.119}) O ( T − 1.119 ) for any stopping time T T T . Gradient descent on measure positions and weights solves sparse optimization problems.
problem Sparse optimization of measures with sparsity-inducing penalties.
method Discretize measure, run non-convex gradient descent on positions and weights.
result Global optimization with complexity scaling as log(1/ε), improving over convex methods.
Paper analyzes ADMM convergence for nonconvex Gaussian phase retrieval.
problem Nonconvex optimization in Gaussian phase retrieval.
method Block coordinate descent as ADMM with dual variable fixed.
result Block coordinate descent converges linearly to global minimizer.
Paper improves accelerated coordinate descent by sqrt(n) with non-uniform sampling.
problem Optimization and machine learning problems, especially empirical risk minimization and solving linear systems.
method Non-uniform sampling of coordinates with probability proportional to the square root of their smoothness parameter.
result Improves the best known running time of accelerated coordinate descent by a factor up to sqrt(n).
Gradient descent implicitly regularizes nonconvex problems, achieving near-optimal results.
problem Statistical estimation problems like phase retrieval, matrix completion, and blind deconvolution.
method Gradient descent without explicit regularization.
result Gradient descent achieves near-optimal statistical and computational guarantees.
Disputes the empirical Fisher approximation for natural gradient descent.
problem The empirical Fisher approximation fails to capture second-order information in general.
method Comparison of empirical Fisher and Fisher information matrices.
result The empirical Fisher does not generally approximate the Fisher or Hessian.
Paper proposes an algorithm to solve complex minimax problems efficiently.
problem Stochastic nonconvex-concave minimax problems in various fields.
method Accelerated first-order regularized momentum descent ascent algorithm (FORMDA).
result Achieves best-known complexity bound of i l d e O ( ε − 6.5 ) ilde{\mathcal{O}}(\varepsilon ^{-6.5}) i l d e O ( ε − 6.5 ) for single-loop algorithms. A new algorithm solves sparse optimization problems on measures efficiently.
problem Sparse optimization problems on measures.
method Over-parameterized Stochastic Gradient Descent with Random Features.
result Global convergence with rate O ( log ( K ) / K ) O(\log(K)/\sqrt{K}) O ( log ( K ) / K ) and bounded total variation norms. MT-SGD samples from multiple target distributions using gradient descent.
problem Sampling from multiple unnormalized target distributions.
method Proposes MT-SGD, a flow of intermediate distributions to sample from multiple target distributions.
result Asymptotic analysis shows MT-SGD reduces to multiple-gradient descent for multi-objective optimization.
This work proposes a new method for variational inference using Wasserstein gradient descent.
problem Optimizing variational parameters to match a true posterior distribution.
method Reinterpreting VI as an optimization problem over a variational parameter space, using Wasserstein gradient descent.
result The proposed Wasserstein gradient descent can be seen as a generalization of existing optimization techniques in VI.
The paper analyzes reg-SGD for convex problems, proving convergence and quantifying the rate of convergence.
problem Minimizing convex, L-smooth functions in a Hilbert space.
method Regularized stochastic gradient descent with decaying regularization.
result Strong convergence to the minimum-norm solution without boundedness assumptions.
This paper explains why double descent sometimes occurs weakly or not at all from an optimization perspective.
problem Understanding the role of optimization in the phenomenon of double descent.
method Investigates model-wise double descent from an optimization perspective, proposing a unified explanation for its occurrence.
result Model-wise double descent is observed if and only if the optimizer can find a sufficiently low-loss minimum.
Characterizes problems solvable via linear convergence algorithms.
problem Optimization problems solvable with linear convergence.
method Riemannian gradient descent.
result Characterized problems solvable via linear convergence.
Gradient descent with noise converges to a unique optimum in nonconvex matrix factorization.
problem Gradient descent with noise converges to a unique optimum in nonconvex matrix factorization.
method A perturbed form of gradient descent with arbitrary initialization.
result Gradient descent with noise converges to a unique optimum.