TOLD++ improves convergence of diffusion models by critically damping the forward transition matrix.
problem Improving the convergence of Denoising Diffusion Probabilistic Models.
method Critically damping the Third-Order Langevin Dynamics (TOLD) forward transition matrix using eigen-analysis.
result TOLD++ converges faster than TOLD, verified on toy and real datasets.
Random permutations can offer faster convergence than with-replacement sampling for some functions.
problem Understanding when and how random permutations outperform with-replacement sampling in SGD convergence.
method Analyzing convergence rates for different function classes (1D strongly convex, general strongly convex, quadratic strongly convex).
result The optimal convergence gap between random and permutation-based SGD varies from exponential to nonexistent, depending on the function class.
In this paper, we present the Bennett-type generalization bounds of the learning process for i.i.d. samples, and then show that the generalization bounds have a faster rate of convergence than the traditional results. In particular, we first develop two types of Bennett-type deviation inequality for the i.i.d. learning…
This paper proves AdaGrad and Adam converge linearly under PL inequality.
problem Understanding the convergence of adaptive gradient methods.
method Unified approach proving AdaGrad and Adam converge linearly under PL inequality.
result AdaGrad and Adam converge linearly when the cost function is smooth and satisfies PL inequality.
learn2mix trains neural nets faster by adjusting class proportions dynamically.
problem Training neural nets efficiently with limited resources and imbalanced classes.
method Adaptive class proportion adjustment during training.
result Neural nets trained with learn2mix converge faster than static methods.
Riemannian stochastic gradient descent converges faster with increasing batch size.
problem Improving convergence rate of Riemannian stochastic gradient descent.
method Theoretical analysis and numerical investigation of increasing batch size effects.
result Riemannian stochastic gradient descent converges faster with increasing batch size.
Faster convergence in federated learning for non-convex problems.
problem Accelerating convergence in federated learning for non-convex models.
method Reformulated federated learning as gradient-based method with biased gradients, proving convergence for non-convex problems and proposing an accelerated algorithm.
result Proved federated averaging algorithm converges for non-convex problems and proposed an accelerated federated learning algorithm with convergence guarantee.
A faster algorithm for ranking from pairwise comparisons.
problem Efficiently ranking individuals or objects from pairwise comparisons.
method An alternative and simpler iterative algorithm for ranking that converges faster.
result The new algorithm is over 100 times faster in some cases.
Faster algorithms for solving multichain MDPs under average-reward criterion.
problem Navigating towards the best connected component in multichain MDPs.
method Developed algorithms to better solve the navigational subproblem, achieving faster convergence rates.
result Improved rates of convergence and sharper complexity measures for multichain MDPs.
Adaptive gradient methods converge faster with over-parameterization and line-search.
problem Training over-parameterized models using adaptive gradient methods.
method Simplified setting of smooth, convex losses with over-parameterized models, proving convergence rates and demonstrating improvements with line-search techniques.
result Adaptive gradient methods, particularly AMSGrad, converge faster with line-search techniques.
FedCluster accelerates federated learning convergence by cycling device groups.
problem Federated learning convergence issues with device-level data heterogeneity.
method FedCluster groups devices into clusters that cycle through learning rounds, boosting convergence with meta-updates.
result FedCluster achieves faster convergence in nonconvex optimization compared to FedAvg.
Framework improves gradient estimation for faster training convergence.
problem Efficiently estimating noisy gradients in stochastic optimization.
method Dynamic adaptive importance sampling combining multiple distributions.
result Adaptively weighted multiple importance sampling yields superior gradient estimates.
New algorithm speeds up Lasso computation by proving faster convergence.
problem Lasso estimator's slow convergence rate due to ℓ1 penalty. method Homotopic approach using surrogate functions.
result Proves O([log(1/ε)]2) convergence rate for Lasso computation. Stochastic methods with coordinate-wise adaptive stepsize (such as RMSprop and Adam) have been widely used in training deep neural networks. Despite their fast convergence, they can generalize worse than stochastic gradient descent. In this paper, by revisiting the design of Adagrad, we propose to split the network par…
Gradient boosting improved with lassoed trees achieves faster convergence.
problem Improving gradient boosting convergence in large nonparametric spaces.
method Lassoed gradient boosted trees with early stopping.
result Achieves faster than n−1/4 L2 convergence rate. Improved neural-ODE for faster convergence and stability.
problem Stability, consistency, and convergence issues in neural-ODE solvers.
method Proposed a first-order Nesterov's accelerated gradient (NAG) based ODE-solver.
result Efficacy demonstrated in three tasks: supervised classification, density estimation, and time-series modelling.
There has been significant recent work on the theory and application of randomized coordinate descent algorithms, beginning with the work of Nesterov [SIAM J. Optim., 22(2), 2012], who showed that a random-coordinate selection rule achieves the same convergence rate as the Gauss-Southwell selection rule. This result su…
Previous studies on stochastic primal-dual algorithms for solving min-max problems with faster convergence heavily rely on the bilinear structure of the problem, which restricts their applicability to a narrowed range of problems. The main contribution of this paper is the design and analysis of new stochastic primal-d…
MSGD outperforms SGD in overparametrized settings with faster convergence rates.
problem Optimization of non-convex functions with momentum.
method Momentum Stochastic Gradient Descent (MSGD) with rigorous analysis.
result MSGD converges exponentially faster than SGD in overparametrized settings.
New convergence bounds for shuffling-based SGD methods in distributed learning.
problem Analyzing the performance of shuffling-based variants of SGD in distributed learning.
method Study of minibatch and local Random Reshuffling methods, proving convergence bounds and lower bounds.
result Shuffling-based variants converge faster than with-replacement sampling methods, and the bounds are tight.
Sampling without replacement speeds up optimization in minimax problems.
problem Optimizing minimax problems with faster convergence rates.
method Analysis of gradient descent ascent and proximal point method with two sampling strategies.
result Sampling without replacement leads to faster convergence rates in minimax optimization.
Armijo line-search speeds up gradient descent for various functions.
problem Improving convergence rate of gradient descent.
method Applying Armijo line-search to adjust step-size in gradient descent.
result GD with Armijo line-search converges faster than GD with a fixed step-size.
We study Frank-Wolfe methods for nonconvex stochastic and finite-sum optimization problems. Frank-Wolfe methods (in the convex case) have gained tremendous recent interest in machine learning and optimization communities due to their projection-free property and their ability to exploit structured constraints. However,…
Paper proposes faster method to find local minima in nonconvex optimization.
problem Escaping saddle points and finding local minima in nonconvex optimization.
method LENA (Last stEp shriNkAge) framework for faster perturbed stochastic gradient methods.
result LENA finds (ε,εH)-approximate local minima within ildeO(ε−3+εH−6) evaluations. We extend the Fourier cosine method to discrete probability distributions, achieving faster convergence rates.
problem Extending Fourier cosine method to discrete probability distributions.
method Spectral filters and convergence rates analysis.
result Spectral filters achieve one order faster convergence rates than previously recognized.
New research shows sparse topologies can lead to faster convergence in distributed optimization.
problem The impact of worker communication topology on convergence speed in distributed optimization.
method Consensus-based distributed optimization methods with local averaging and correction based on local data.
result Sparse topologies can lead to faster convergence in distributed optimization without communication delays.
Drago optimizes DRO problems with faster convergence.
problem Distributionally robust optimization with closed, convex uncertainty sets.
method Primal-dual coupled variance reduction algorithm with cyclic and randomized updates.
result Achieves state-of-the-art linear convergence rate on strongly convex-strongly concave problems.
We estimate from above the rate at which a solution to the normalized Ricci flow on a closed manifold may converge to a limit soliton. Our main result implies that any solution which converges modulo diffeomorphisms to a soliton faster than any fixed exponential rate must itself be self-similar.
Paper proposes a new estimator for nested expectations with faster convergence.
problem Estimating nested expectations is computationally challenging.
method Nested kernel quadrature estimators with proof of faster convergence rate.
result The proposed method requires fewer samples for accurate estimation.
Recent advances in optimization theory have shown that smooth strongly convex finite sums can be minimized faster than by treating them as a black box "batch" problem. In this work we introduce a new method in this class with a theoretical convergence rate four times faster than existing methods, for sums with sufficie…
Deep neural networks have gained tremendous popularity in last few years. They have been applied for the task of classification in almost every domain. Despite the success, deep networks can be incredibly slow to train for even moderate sized models on sufficiently large datasets. Additionally, these networks require l…
This paper analyzes and improves convergence in federated learning with biased client selection.
problem Analyzing convergence in federated learning with biased client selection.
method First convergence analysis of federated optimization for biased client selection strategies, proposing Power-of-Choice framework.
result Power-of-Choice strategies converge up to 3 times faster and give 10% higher test accuracy than random selection.
We find faster-converging sub-networks that significantly reduce adversarial training time.
problem Finding optimal sub-networks for adversarial training is costly and time-consuming.
method We identify a subset of sub-networks that converge faster during training.
result Sub-networks can reduce adversarial training time by up to 49%.
A new optimizer, Grad-Avg, converges faster than SGD and improves classification task performance.
problem Optimizing error functions in regression and classification tasks.
method Grad-Avg optimizer based on gradient averaging, with mathematical convergence proof and parameter scaling for classification.
result Grad-Avg converges faster than other optimizers for classification tasks on benchmark datasets.
SAPPHIRE tackles ill-conditioned rERM problems with faster convergence.
problem Ill-conditioned objectives and non-smooth regularizers in rERM.
method Sketch-based preconditioning and scaled proximal mapping.
result Achieves condition-number-free linear convergence.
Proposes a neural density estimator that adapts to low-dimensional structures and integrates into generative models.
problem Challenges in implementing neural density estimators and lack of theoretical understanding.
method Structure-agnostic neural density estimator that is easy to implement and provably adaptive.
result Adapts to low-dimensional structures and achieves faster convergence rates.
Accelerates optimal transport computation by 10x with spectral insights.
problem Exponential slow-down of convergence in Entropic Optimal Transport as regularization weakens.
method Spectral insights and spectral warm-start strategy to mitigate convergence issues.
result Faster convergence compared to the reference method Sinkhorn algorithm.
Paper adapts diffusion sampler training for faster convergence and better sampling.
problem Training limitations in diffusion samplers.
method Decouples generation and destruction variances, learns both as unconstrained Gaussians.
result Training both processes leads to faster convergence and improved sampling quality.
Estimates the rate of convergence of mean curvature flow solutions.
problem Understanding the convergence rate of mean curvature flow solutions.
method Estimates the upper bound of convergence rate to a limit self-similar solution.
result Solutions converging faster than any fixed exponential rate must be shrinkers themselves.
Stagewise training strategy is widely used for learning neural networks, which runs a stochastic algorithm (e.g., SGD) starting with a relatively large step size (aka learning rate) and geometrically decreasing the step size after a number of iterations. It has been observed that the stagewise SGD has much faster conve…
Unified framework for training SNNs using EP, faster convergence.
problem Training spiking neural networks with Expectation-Propagation.
method Message-passing framework for learning marginal distributions of SNN parameters.
result Faster convergence compared to gradient-based methods.
Screening rules allow to early discard irrelevant variables from the optimization in Lasso problems, or its derivatives, making solvers faster. In this paper, we propose new versions of the so-called safe rules for the Lasso. Based on duality gap considerations, our new rules create safe test regions whose d…
LALR adapts learning rate for faster convergence in regression and neural nets.
problem Finding optimal learning rates for faster convergence in regression and neural networks.
method Lipschitz continuity theory applied to Mean Absolute Error and Quantile loss functions.
result Adaptive learning rate policy enables up to 20x faster convergence.
We propose the stochastic average gradient (SAG) method for optimizing the sum of a finite number of smooth convex functions. Like stochastic gradient (SG) methods, the SAG method's iteration cost is independent of the number of terms in the sum. However, by incorporating a memory of previous gradient values the SAG me…
Stochastic proximal point algorithm with momentum converges faster and is more stable than standard methods.
problem Improving convergence and stability of stochastic optimization methods.
method Developed and analyzed the convergence and stability of the stochastic proximal point algorithm with momentum (SPPAM).
result SPPAM converges faster and is more stable than standard stochastic proximal point algorithm (SPPA) and stochastic gradient descent with momentum (SGDM).
Sparse perturbations improve convergence in SZO methods for faster training.
problem Dependency of SZO methods on function dimensionality limits their convergence speed.
method Sparse perturbations reduce the effective dimensionality of the optimization problem.
result Sparse SZO optimization leads to faster convergence in training loss and test accuracy.
New proof shows faster convergence rate for robust estimation with Lasso in adversarially contaminated outputs.
problem Robust estimation of parameters in the presence of adversarial output contamination.
method Extended Lasso with Huber loss function and L1 penalty, focusing on specific properties of the Huber function. result Same convergence rate as Dalalyan and Thompson (2019), but with a different proof.
New method improves Robbins-Monro algorithm convergence with prior information.
problem Improving convergence speed of Robbins-Monro algorithm.
method Integrates prior information into Robbins-Monro iteration without regression model.
result Prior-information Robbins-Monro sequence converges faster than standard.