This paper improves convergence guarantees for gradient clipping in deep learning.
problem Improving convergence guarantees for gradient clipping in deep learning models.
method Analyzes and provides precise convergence guarantees for arbitrary clipping thresholds.
result Shows tight convergence guarantees for clipped stochastic gradient descent.
Efficient RNN algorithm guarantees convergence in online learning.
problem Online nonlinear regression with RNNs.
method First-order training algorithm with convergence guarantee.
result The algorithm converges to optimum network parameters.
Two new algorithms recover ridge lines from point clouds with convergence guarantees.
problem Extracting filamentary structure from point clouds.
method Proposes two novel algorithms with convergence guarantees.
result The algorithms can asymptotically recover the full ridge set.
The paper provides convergence guarantees for VAEs using SGD and Adam.
problem Understanding theoretical convergence guarantees for VAEs.
method Derives non-asymptotic convergence rates for VAEs trained with SGD and Adam.
result Convergence rate of \(\mathcal{O}(\log n / \sqrt{n})\) with explicit hyperparameter dependencies.
This paper improves convergence guarantees for SGD algorithms in non-convex smooth functions.
problem Theoretical convergence properties of SGD algorithms for non-convex smooth functions.
method Analysis of SGD algorithms with arbitrary data ordering for non-convex smooth functions.
result Enhanced convergence guarantees for incremental gradient and single shuffle SGD, improving the optimization term of convergence guarantee.
Paper provides exponential convergence guarantees for Iterative Markovian Fitting.
problem Addressing the Schrödinger Bridge problem in computational optimal transport and generative modeling.
method Develops non-asymptotic exponential convergence guarantees for Iterative Markovian Fitting.
result First non-asymptotic exponential convergence guarantees for IMF under mild structural assumptions.
New guarantees for black-box variational inference methods.
problem Insufficient theoretical guarantees for black-box variational inference.
method Novel convergence guarantees for stochastic optimization of variational inference.
result Provable convergence of proximal and projected stochastic gradient descent for variational inference.
New convergence guarantees for SGDA and SCO under expected co-coercivity.
problem Solving smooth games with stochastic gradient descent-ascent and consensus optimization.
method Introducing expected co-coercivity and proving convergence guarantees for SGDA and SCO.
result Linear convergence of SGDA and SCO to a neighborhood of the solution with constant step-size, and convergence to the exact solution with stepsize-switching rules.
This work accelerates gradient descent with anytime convergence guarantees.
problem Improving the convergence rate of gradient descent methods.
method Proposes a stepsize schedule for gradient descent that achieves anytime convergence rates.
result Gradient descent can achieve convergence rates of O ( T − 1.119 ) O(T^{-1.119}) O ( T − 1.119 ) for any stopping time T T T . Square-root natural-gradient improves variational inference convergence.
problem Challenges in establishing theoretical convergence guarantees for natural-gradient descent.
method Square-root parameterization for Gaussian covariance.
result Establishes novel convergence guarantees for natural-gradient Gaussian inference.
Paper proves smoothness for variational inference, giving convergence guarantees.
problem Proving convergence guarantees for black-box variational inference.
method Describes gradients in an inner-product space, using Bessel's inequality.
result Objective is M-Lipschitz smooth if target is, excluding entropy.
Study on Adam-family methods for nonsmooth optimization with convergence guarantees.
problem Training nonsmooth neural networks with convergence guarantees.
method Two-timescale updating scheme and stochastic subgradient methods with gradient clipping.
result Convergence guarantees for various Adam-family methods in training nonsmooth neural networks.
Guarantees uniform convergence for square-root Lipschitz losses.
problem Uniform convergence guarantees for square-root Lipschitz losses.
method Using Rademacher complexity and square root of scalar loss function Lipschitz constant.
result Generalizes previous results and handles non-smooth loss functions.
In this paper, we provide local and global convergence guarantees for recovering CP (Candecomp/Parafac) tensor decomposition. The main step of the proposed algorithm is a simple alternating rank- 1 1 1 update which is the alternating version of the tensor power iteration adapted for asymmetric tensors. Local convergence g…
We extend contrastive learning theory for multiway classification and prove convergence guarantees.
problem Efficient self-supervised training for multiway classification tasks.
method Contrastive representation learning with multiple negative samples and convergence guarantees for gradient descent.
result Convergence guarantees for contrastive learning with gradient descent of an overparametrized encoder.
Guarantees convergence for black-box variational inference without modifications.
problem Convergence guarantees for black-box variational inference.
method Analysis of log-smooth posterior densities, location-scale variational family, and convergence rates of algorithm design choices.
result Proximal stochastic gradient descent fixes suboptimal convergence rates and achieves strongest known guarantees.
Improved convergence for Polyak steps with momentum in smooth convex optimization.
problem Optimizing smooth strongly convex functions with limited information.
method Polyak steps with momentum, derived for accelerated gradient method.
result Convergence guarantees for accelerated gradient method with Polyak steps and momentum.
Develops uniform convergence guarantees for a broad class of risk functionals in supervised learning.
problem Bounding generalization gaps for various risk functionals beyond the expectation.
method Establishes uniform convergence for Hölder risk functionals, providing guarantees for empirical risk minimization.
result First uniform convergence results for estimating the CDF of loss distributions, applicable to various risk functionals.
Improved KL bounds and Wasserstein guarantees for diffusion flow matching under minimal conditions.
problem Theoretical convergence properties of Brownian motion based diffusion flow matching.
method Refined analysis under Kullback-Leibler and 2-Wasserstein distances.
result State-of-the-art scaling in KL convergence bounds under minimal conditions.
Simpler, parameter-free AdaGrad and Adam variants with convergence guarantees.
problem Inefficiencies in ad-hoc learning rate tuning for optimization algorithms.
method Developed AdaGrad++ and Adam++ without predefined learning rates and proved their convergence.
result AdaGrad++ and Adam++ achieve comparable convergence rates to AdaGrad and Adam respectively.
New polynomial convergence guarantees for SGM on general data distributions.
problem Efficient guarantees for multimodal and non-smooth distributions in SGM.
method Polynomial convergence guarantees for denoising diffusion models on general data distributions, with no assumptions on functional inequalities or smoothness.
result Wasserstein distance guarantees for distributions of bounded support or decaying tails, and TV guarantees for further smoothness assumptions.
Mutation improves FTRL convergence in zero-sum games.
problem Lack of last-iterate convergence in FTRL variants.
method Introduced mutation to perturb action probabilities in FTRL.
result M-FTRL converges to Nash equilibria under full-information feedback.
LMC algorithm receives first convergence guarantees under weak smoothness conditions.
problem Convergence guarantees for LMC under weak smoothness conditions.
method Using Latała--Oleszkiewicz or modified log-Sobolev inequalities.
result First convergence guarantees for LMC under weak smoothness conditions.
New methods solve non-Lipschitz smooth problems with guaranteed convergence.
problem Non-Lipschitz smooth problems in machine learning and signal processing.
method Bregman-divergence based algorithms for relatively smooth problems.
result Guaranteed convergence to second-order stationary points for any relatively smooth problem.
Proposes a new backpropagation algorithm for deep learning with guaranteed convergence.
problem Backward locking in backpropagation limits parallel updates in deep neural networks.
method Decouples gradients and splits the network into modules for parallel updates, proving convergence for non-convex problems.
result The proposed algorithm achieves significant speedup without accuracy loss in training deep convolutional neural networks.
Nonlinear MCMC improves Bayesian machine learning sampling.
problem Sampling problems in Bayesian machine learning.
method Nonlinear MCMC technique with convergence guarantees.
result Improves sampling in Bayesian neural networks.
New algorithm for computing Wasserstein barycenters with guarantees.
problem Computing Wasserstein barycenters with varying regularization strengths.
method Damped Sinkhorn iterations followed by exact maximization/minimization steps.
result First non-asymptotic convergence guarantees for approximating Wasserstein barycenters.
New method improves convergence for smooth games.
problem Improving convergence for smooth games.
method Stochastic Hamiltonian Gradient Methods (SHGD).
result SHGD converges linearly to the neighbourhood of a stationary point.
New convergence guarantees for learning with unknown nuisance parameters.
problem Learning problems with unknown nuisance parameters.
method Stochastic gradient optimization with Neyman orthogonality and approximately orthogonalized updates.
result Stochastic gradient algorithms can converge under conditions of nuisance parameters.
New quasi-Newton method guarantees global superlinear convergence.
problem Global convergence and superlinear convergence of quasi-Newton methods.
method Hybrid proximal extragradient method with online learning for Hessian approximation.
result First globally convergent quasi-Newton method with explicit superlinear convergence rate.
New step-size methods improve SHB convergence for stochastic optimization.
problem Tuning step-size and momentum parameters in SHB is challenging.
method Proposed MomSPS max _{\max} m a x , MomDecSPS, and MomAdaSPS for SHB. result Convergence guarantees for SHB to solution neighborhoods and exact minimizers.
New analysis improves convergence guarantees for diffusion-based samplers in Wasserstein distance.
problem Improving convergence guarantees for diffusion-based generative models.
method Simple framework to analyze discretization, initialization, and score estimation errors.
result First Wasserstein convergence bound for the Heun sampler and improved results for Euler sampler.
BCD algorithm finds global minima in neural networks.
problem Training deep neural networks to find global minima.
method Block coordinate descent with skip connections and non-negative projection.
result Proves convergence to global minima for strictly monotonic and ReLU activations.
FedSARSA converges with heterogeneous agents, achieving linear speed-up.
problem Convergence analysis of Federated SARSA with heterogeneous agents.
method Linear function approximation, local training, multi-step error expansion.
result FedSARSA achieves linear speed-up with respect to the number of agents.
KSS method converges and recovers correct clustering under certain conditions.
problem Subspace clustering for semi-randomly sampled data.
method Local convergence analysis and recovery guarantee for KSS method.
result KSS method converges superlinearly and finds correct clustering within loglog N iterations.
Paper shows SVM can achieve super fast convergence rates.
problem Understanding fast convergence rates for SVM.
method Presented a simple mechanism to obtain fast convergence rates for SVM.
result SVM can exhibit exponential convergence rates without hard Tsybakov margin condition.
Inference problems in graphical models are often approximated by casting them as constrained optimization problems. Message passing algorithms, such as belief propagation, have previously been suggested as methods for solving these optimization problems. However, there are few convergence guarantees for such algorithms…
The paper analyzes PPM for nonconvex-nonconcave problems, identifying three regions with varying convergence guarantees.
problem Challenges in nonconvex-nonconcave minimax optimization.
method Classic proximal point method with insights from the Moreau envelope.
result Identification of three regions with varying convergence guarantees for PPM.
We simplify deep learning convergence analysis using basic math.
problem Ensuring reliable convergence of optimization algorithms in deep networks.
method Elementary arguments and computations to estimate smoothness constants.
result Systematic computation of smoothness constants for first-order algorithms.
Global convergence of multilayer neural networks proven for any depth.
problem Global convergence of multilayer neural networks in the mean field regime.
method Mean field limit framework, neuronal embedding, bidirectional diversity condition.
result Global convergence for multilayer networks of any depths, including correlated initializations.
The study provides statistical guarantees for Bayesian variational boosting.
problem Statistical and convergence issues in variational boosting.
method Proposed a novel variational family and a functional Frank-Wolfe optimization algorithm.
result Demonstrated stochastic boundedness and provided convergence rate for boosting iterates.
We propose a distributed approach to train deep neural networks (DNNs), which has guaranteed convergence theoretically and great scalability empirically: close to 6 times faster on instance of ImageNet data set when run with 6 machines. The proposed scheme is close to optimally scalable in terms of number of machines, …
Optimizes audio codec selection with statistical guarantees.
problem Selecting the best audio encoding scheme for various data types.
method Supervised learning with uniform convergence theory.
result Rigorous statistical guarantees for codec selection.
The paper provides convergence guarantees for multicalibration gradient boosting.
problem Understanding the convergence properties of multicalibration gradient boosting.
method Computational guarantees for multicalibration gradient boosting algorithms, including adaptive variants.
result The magnitude of successive prediction updates decays at O ( 1 / T ) O(1/\sqrt{T}) O ( 1/ T ) , leading to convergence in empirical multicalibration error. BCD methods provide provable convergence guarantees for deep learning models.
problem Theoretical convergence guarantees for BCD methods in deep learning.
method Established global convergence rate of O(1/k) for DNN training models.
result Global convergence to a critical point at a rate of O(1/k) for most DNN training models.
Guaranteed convergence for tensor factorization using Riemannian gradient descent.
problem Recovering tensor train format from linear measurements.
method Optimization over left-orthogonal TT format using Riemannian gradient descent on Stiefel manifold.
result RGD converges linearly to the ground-truth tensor with polynomial error growth in tensor order.
Neural network learns fast PDE solvers with proven guarantees.
problem Designing fast iterative solvers for specific PDE problems.
method Learn to modify an existing solver using a deep neural network.
result Achieves 2-3 times speedup compared to state-of-the-art solvers.
The paper analyzes a geometrical algorithm for statistical inference with convergence guarantees.
problem Statistical inference on nonparametric cases.
method Derives a bound for learning rate to ensure local convergence of a geometrical projection algorithm.
result Specific forms of the bound are calculated for m-mixture and e-mixture estimation problems.