Characterizes problems solvable via linear convergence algorithms.
problem Optimization problems solvable with linear convergence.
method Riemannian gradient descent.
result Characterized problems solvable via linear convergence.
The Sinkhorn-Knopp derivatives converge with linear rate.
problem Optimal transport problem with entropic regularization.
method Iterative proportional fitting procedure.
result Derivatives converge with linear rate.
Policy gradient converges linearly with Hadamard parameterization in tabular settings.
problem Convergence of policy gradient methods under Hadamard parameterization.
method Studied convergence rate and established linear convergence after k0 iterations. result Algorithm converges linearly with rate $O( rac{1}{k})$ and faster locally after k0. EM algorithm converges globally for two-component mixed linear regression.
problem Global convergence of EM algorithm for mixed linear regression.
method Developed new theoretical analysis for EM algorithm convergence in mixed linear regression.
result EM algorithm converges globally for two-component mixed linear regression.
We investigate finite-time decoupled convergence in nonlinear two-time-scale stochastic approximation.
problem Achieving decoupled convergence in nonlinear two-time-scale stochastic approximation.
method Nested local linearity assumption, suitable step size selection, convergence analysis of matrix cross term, fourth-order moment convergence rates.
result Finite-time decoupled convergence rates can be achieved in nonlinear two-time-scale stochastic approximation with proper step size selection.
Entropy-regularized NPG converges linearly with linear function approximation.
problem Analyzing convergence of entropy-regularized NPG with function approximation.
method Established finite-time convergence analyses with entropy regularization and linear function approximation.
result Entropy-regularized NPG achieves linear convergence up to a function approximation error.
Policy gradient methods achieve linear convergence in simple MDPs.
problem Analyzing convergence rates of policy gradient methods in finite MDPs.
method Connections with policy iteration to show linear convergence with large step-sizes.
result Policy gradient methods succeed with large step-sizes and achieve linear rate of convergence.
Paper proves linear convergence of SCMS algorithm for directional data.
problem Identifying density ridges in directional data.
method Generalized SCMS algorithm to directional data, derived from SCGA with adaptive step size.
result Linear convergence of the proposed directional SCMS algorithm.
The paper improves convergence for linear systems using entropic mirror descent with Polyak stepsizes.
problem Convergence analysis for linear systems with unbounded domain.
method Entropic mirror descent with Polyak stepsizes, sublinear and linear convergence results.
result Generalized convergence result for arbitrary convex functions.
We consider linear slices of the space of Kleinian once-punctured torus groups; a linear slice is obtained by fixing the value of the trace of one of the generators. The linear slice for trace 2 is called the Maskit slice. We will show that if traces converge `horocyclically' to 2 then associated linear slices converge…
New analysis shows GMD can converge linearly under PL-like conditions.
problem Establishing linear convergence for generalized mirror descent.
method PL-based analysis for time-dependent mirrors, Taylor-series approach for stochastic GMD.
result Linear convergence of stochastic GMD under PL-like conditions.
Deep linear ResNets converge globally with certain transformations.
problem Global convergence of training deep linear ResNets.
method Gradient descent and stochastic gradient descent for training L-hidden-layer linear ResNets. result GD and SGD can converge to global minimum for deep linear ResNets with specific transformations.
Gradient descent achieves exact linear convergence rate for symmetric matrix completion.
problem Low-rank symmetric matrix completion using gradient descent.
method Local analysis of gradient descent for symmetric matrices without additional assumptions.
result Closed-form expression of exact linear convergence rate matches practice.
Study shows a linear quadratic regulator's imitation learning converges globally.
problem Global convergence of imitation learning for linear quadratic regulators.
method Analyzed alternating gradient algorithm and established Q-linear rate of convergence.
result Established a unique saddle point for globally optimal policy and reward function.
Study shows long-term solutions for complex equations on curved spaces.
problem Long-term behavior of solutions to fully non-linear parabolic equations on Hermitian manifolds.
method Used general assumptions and derived a Harnack inequality for the linearized equation.
result Proved the long-time existence and convergence of solutions.
Gradient descent converges globally in deep linear residual networks with ZAS initialization.
problem Optimizing deep linear residual networks for convergence.
method Zero-asymmetric (ZAS) initialization for gradient descent.
result Gradient descent converges to an ε-optimal point in O(L^3 log(1/ε)) iterations.
Gradient descent converges linearly for deep linear networks under specific conditions.
problem Speed of convergence in gradient descent for deep linear neural networks.
method Analysis of gradient descent training for deep linear neural networks minimizing ℓ2 loss. result Gradient descent converges linearly under specific conditions on layer dimensions, initialization, and initial loss.
New insights into continual learning for deep models, showing convergence issues but local linear solutions.
problem Challenges in continual learning for homogeneous deep models.
method Sequential projections onto task margin sets, leveraging nonconvex projection theory.
result Local linear convergence under certain conditions for homogeneous deep networks.
New method for natural policy gradients converges linearly.
problem Improving natural policy gradient methods for better convergence.
method Fisher-Rao gradient flow applied to state-action distributions.
result Linear convergence rate with geometry-dependent factor.
This paper proves AdaGrad and Adam converge linearly under PL inequality.
problem Understanding the convergence of adaptive gradient methods.
method Unified approach proving AdaGrad and Adam converge linearly under PL inequality.
result AdaGrad and Adam converge linearly when the cost function is smooth and satisfies PL inequality.
Study shows directional convergence for neural networks under spherical symmetry.
problem Learning linear predictors with neural networks under spherically symmetric data.
method Analysis of gradient flow and gradient descent for two-layer and deep linear networks.
result Directional convergence guarantees with exact convergence rate for specific network architectures.
Stochastic gradient descent achieves polynomial convergence rates for noiseless linear models.
problem Convergence analysis of stochastic gradient descent in noiseless linear models.
method Fixed step-size stochastic gradient descent on least-square risk.
result Polynomial convergence rates depend on the regularities of the optimum and feature vectors.
New algorithm improves convergence rates for convex optimization problems.
problem Convex optimization problems with noisy stochastic data.
method Stochastic proximal point algorithm with weak linear regularity condition.
result Achieves $\mathcal{O}\left(\frac{1}{k}
ight)$ convergence rate for SPP.
Gradient descent converges linearly for overparameterized linear networks.
problem Convergence of gradient descent for overparameterized neural networks.
method Local Polyak-Lojasiewicz and Descent Lemma for overparameterized linear models.
result Gradient descent achieves linear convergence for two-layer linear networks under relaxed assumptions.
GD converges faster to flatter minima than gradient flow in shallow networks.
problem Understanding the dynamics of gradient descent in shallow linear networks.
method Analyzing the convergence rate and solution of gradient descent in depth-2 linear neural networks.
result GD converges linearly to flatter minima than gradient flow, even with large step sizes.
AdaLoss optimizes adaptive learning rates for efficient convergence in various models.
problem Efficiently optimizing adaptive learning rates for gradient descent methods.
method AdaLoss uses loss function information to dynamically adjust step sizes.
result AdaLoss achieves linear convergence in linear regression and robust global convergence in neural networks.
Online learning of linear operators between infinite-dimensional spaces is possible but with limitations.
problem Learning linear operators between infinite-dimensional Hilbert spaces in an online setting.
method Online learning approach for linear operators with bounded p-Schatten norm, proving impossibility for operator norm. result Separation between online learnability and uniform convergence for bounded linear operators.
EM converges for mixtures of many linear regressions with SNR > Ω(k).
problem Convergence of EM algorithm for mixtures of linear regressions.
method Analysis of EM algorithm convergence for mixtures of linear regressions with arbitrary number of components.
result EM converges to true parameters with SNR > Ω(k), independent of parameter norms.
FA algorithm provides convergence guarantees for deep linear networks.
problem Training efficiency and convergence of deep neural networks.
method Theoretical analysis of Feedback Alignment (FA) algorithm for deep linear networks.
result Certain initializations lead to implicit anti-regularization, affecting learning effectiveness.
Improved SGD for robust linear and ReLU regression with adversarial corruptions.
problem Robust regression with adversarial corruptions in streaming data.
method Stochastic gradient descent (SGD-exp) with exponentially decaying step size.
result Nearly linear convergence to true parameter with up to 50% Massart corruption rate.
AM converges super-linearly for solving mixed linear regression problems.
problem Learning linear regressors from unlabeled observations in multiple linear regression models.
method Alternating Minimization (AM) algorithm, which alternates between label estimation and regression solving.
result AM converges super-linearly in certain parameter regimes, requiring only O(log log(1/ε)) iterations to achieve an error of ε.
Paper revisits set membership estimation for linear systems with relaxed disturbance bounds.
problem Set membership estimation for linear systems with disturbances bounded by convex sets.
method Adopted block-martingale small-ball condition and random perturbed control policies to establish convergence rates.
result Established convergence rates for disturbances bounded by general convex sets.
AdaGrad-Norm achieves linear convergence for certain functions.
problem Proving linear convergence for specific types of functions.
method Introducing RUIG, a measure of gradient balance; developing a two-stage framework.
result AdaGrad-Norm achieves linear convergence for certain functions.
We show that gradient descent on full-width linear convolutional networks of depth L converges to a linear predictor related to the ℓ2/L bridge penalty in the frequency domain. This is in contrast to linearly fully connected networks, where gradient descent converges to the hard margin linear support vector m…
Geometric analysis shows gradient descent in linear neural nets converges to global minima.
problem Analyzing convergence of gradient descent in linear neural networks.
method Geometric framework and invariance property of network structure.
result Gradient descent trajectories converge to global minima for linear neural nets.
Proposes a convergent TD algorithm for off-policy RL.
problem Learning value function from different policies in RL.
method Convergent on-policy TD algorithm with linear function approximation.
result Proposes a convergent TD algorithm for off-policy RL.
Paper proposes a pre-conditioning technique to speed up gradient-descent convergence in distributed linear least-squares problems.
problem Expediting convergence of gradient-descent method for ill-conditioned distributed linear least-squares problems.
method Iterative pre-conditioning technique to improve convergence rate of gradient-descent method.
result Pre-conditioned gradient-descent achieves superlinear convergence for unique solutions and improved linear convergence otherwise.
FedSARSA converges with heterogeneous agents, achieving linear speed-up.
problem Convergence analysis of Federated SARSA with heterogeneous agents.
method Linear function approximation, local training, multi-step error expansion.
result FedSARSA achieves linear speed-up with respect to the number of agents.
Unified framework for solving linear systems with improved convergence rates.
problem Efficiently solving linear systems with randomized batch-sampling methods.
method Developed a unified randomized batch-sampling Kaczmarz framework with concentration inequalities for analysis.
result Derived new expected linear convergence rate bounds that are tighter and more reflective of empirical behavior.
Linear Q-learning converges to a bounded set without divergence.
problem Proving linear Q-learning does not diverge and converges to a bounded set.
method No modifications to the original linear Q-learning algorithm, no Bellman completeness or near-optimality assumptions, only an ε-softmax behavior policy with adaptive temperature.
result First L2 convergence rate of linear Q-learning iterates to a bounded set. A fully decentralized multi-agent algorithm converges linearly with minimal memory.
problem Efficiently evaluating policies in multi-agent settings with limited exploration.
method Fully decentralized, combining off-policy learning, eligibility traces, and linear function approximation.
result Achieves linear convergence with minimal memory requirements.
Actor-critic converges globally in LQR with ergodic cost.
problem Theoretical understanding of actor-critic algorithm's global convergence.
method Nonasymptotic convergence analysis of actor-critic in linear quadratic regulator (LQR) setting.
result Actor-critic finds globally optimal policy and value function at a linear rate.
Orthogonal initialization speeds up convergence in deep linear networks.
problem The impact of initialization on convergence speed and model performance in deep neural networks.
method Analysis of orthogonal initialization in deep linear networks, proving its superiority over Gaussian initialization.
result Orthogonal initialization speeds up convergence relative to Gaussian initialization in deep networks.
Paper analyzes EM algorithm's trajectory in 2MLR, revealing cycloid behavior.
problem Understanding the convergence and trajectory of EM algorithm in 2MLR.
method Explicit closed-form expressions for EM updates, recurrence relation derivation at population level.
result EM iterations lie on a cycloid trajectory, leading to theoretical estimate of convergence exponent.
Study derivative-free methods for linear policies in linear-quadratic systems.
problem Optimizing policies in linear-quadratic systems with limited derivative information.
method Derivative-free methods applied to linear policies over various noise and reward feedback settings.
result These methods converge to near-optimal policies with a polynomial number of zero-order evaluations.
Guaranteed convergence for tensor factorization using Riemannian gradient descent.
problem Recovering tensor train format from linear measurements.
method Optimization over left-orthogonal TT format using Riemannian gradient descent on Stiefel manifold.
result RGD converges linearly to the ground-truth tensor with polynomial error growth in tensor order.
Policy gradient methods find Nash equilibrium in noisy games.
problem Finding Nash equilibrium in noisy games.
method Policy gradient methods with noise added.
result Policy gradient methods converge to Nash equilibrium in noisy games.
Paper shows linear convergence of ISTA and FISTA for ill-conditioned images.
problem Solving linear inverse problems with sparse representation in signal and image processing.
method Revisits iterative shrinkage-thresholding algorithms (ISTA) and improves their convergence properties.
result Linear convergence of ISTA and FISTA for strongly convex smooth parts, even in ill-conditioned cases.