Gradient EM converges globally for over-parameterized Gaussian mixtures.
problem Global convergence of gradient EM for Gaussian mixtures with more than 2 components.
method Likelihood-based convergence analysis framework.
result Gradient EM converges globally with a sublinear rate of O(1/√t).
Gradient descent proves global convergence for 4-layer matrix factorization.
problem Global convergence of gradient descent on four-layer matrix factorization under random initialization.
method New techniques to show saddle-avoidance properties and extend eigenvalue theories.
result Polynomial-time global convergence guarantee for randomly initialized gradient descent on four-layer matrix factorization.
EM algorithm converges to global max in latent Gaussian tree models.
problem Optimizing log-likelihood in latent Gaussian tree models.
method Analyzed the optimization landscape and convergence of EM algorithm.
result EM algorithm converges to global maximum in latent Gaussian tree models.
Smooth convergence shown for curve diffusion flows.
problem Embeddedness and global existence of curves.
method Exponentially fast convergence established.
result Smooth convergence for curve diffusion flows.
This work studies the location estimation problem for a mixture of two rotation invariant log-concave densities. We demonstrate that Least Squares EM, a variant of the EM algorithm, converges to the true location parameter from a randomly initialized point. We establish the explicit convergence rates and sample complex…
Gradient EM converges globally for over-parameterized Gaussian mixtures.
problem Recovering ground truth Gaussian mixtures with over-parameterized models.
method Gradient EM with over-parameterization, using Hermite polynomials and tensor decomposition.
result Gradient EM globally converges to ground truth with n=Ω(mlogm) over-parameterization. Paper proves global convergence of NCELM model.
problem Ensuring global convergence of NCELM model.
method Two-stage process: random base learners and NCL penalty term updates.
result Global convergence of NCELM proved using Banach theorem.
Variational inference methods for latent variable statistical models have gained popularity because they are relatively fast, can handle large data sets, and have deterministic convergence guarantees. However, in practice it is unclear whether the fixed point identified by the variational inference algorithm is a local…
GAIL with neural networks converges to global optima and has a known rate.
problem Uncertainty about GAIL with neural networks achieving global optimality.
method Gradient-based alternating updates algorithm.
result Established sublinear convergence to globally optimal solution.
New quasi-Newton method guarantees global superlinear convergence.
problem Global convergence and superlinear convergence of quasi-Newton methods.
method Hybrid proximal extragradient method with online learning for Hessian approximation.
result First globally convergent quasi-Newton method with explicit superlinear convergence rate.
Paper proves global optimality of a simple optimization scheme for learning DAG models.
problem Learning acyclic directed graphical models from data.
method Path-following optimization scheme for bivariate setting.
result Simple optimization scheme globally converges to global minimum.
This work analyzes how overparameterization aids GANs in reaching global saddle points.
problem Understanding the role of overparameterization in GANs for convergence to global saddle points.
method Theoretical and empirical analysis of overparameterized GANs with various architectures and datasets.
result GDA converges to a global saddle point in overparameterized GANs with certain assumptions.
New framework for DNN training guarantees convergence to global minimum.
problem Training deep neural networks to converge to global minimum.
method Reformulated minimization problem with recursive algorithmic framework, using bounded style assumptions.
result Convergence to an ε-(global) minimum with O(1/ε^3) gradient computations.
Policy gradient methods with actor-critic schemes demonstrate tremendous empirical successes, especially when the actors and critics are parameterized by neural networks. However, it remains less clear whether such "neural" policy gradient methods converge to globally optimal policies and whether they even converge at …
AGGLIO optimizes non-convex functions with local convexity guarantees.
problem Optimizing non-convex functions with local convexity.
method Stage-wise, graduated optimization technique for locally convex functions.
result Global convergence to the global optimum for non-convex and locally convex objectives.
Global convergence of multilayer neural networks proven for any depth.
problem Global convergence of multilayer neural networks in the mean field regime.
method Mean field limit framework, neuronal embedding, bidirectional diversity condition.
result Global convergence for multilayer networks of any depths, including correlated initializations.
Global convergence for robust regression problems via IRLS with enhancements.
problem Global convergence for robust regression problems.
method Augmentations to IRLS to ensure global recovery and improved robustness.
result Global recovery guarantees for robust regression problems, outperforming state-of-the-art algorithms.
Deep linear ResNets converge globally with certain transformations.
problem Global convergence of training deep linear ResNets.
method Gradient descent and stochastic gradient descent for training L-hidden-layer linear ResNets. result GD and SGD can converge to global minimum for deep linear ResNets with specific transformations.
The study examines when MAML's objective has a benign landscape.
problem Understanding when MAML's objective landscape is benign.
method Analyzing the landscape of MAML objective on LQR tasks.
result The benign landscape of the MAML objective depends on task similarities.
Accelerates convergence in global non-convex optimization with reversible diffusion.
problem Global non-convex optimization challenges.
method Utilizes reversible diffusion processes with adaptive diffusion coefficients.
result Accelerated convergence with reduced discretization error.
Temporal-difference learning (TD), coupled with neural networks, is among the most fundamental building blocks of deep reinforcement learning. However, due to the nonlinearity in value function approximation, such a coupling leads to nonconvexity and even divergence in optimization. As a result, the global convergence …
SGD converges to global minimum for certain non-convex functions.
problem Theoretical challenges in optimizing non-convex functions in machine learning.
method Perturbed SGD on a broad class of non-convex functions.
result SGD converges to global minimum for certain non-convex functions.
Wide neural networks converge linearly to zero loss with feature learning.
problem Optimizing wide neural networks with feature learning guarantees.
method Gradient flow analysis for wide shallow and multi-layer NNs.
result Training loss converges linearly to zero for wide NNs under GF, demonstrating feature learning and better generalization.
Paper refutes EM convergence theory and introduces a new EM algorithm.
problem The convergence theory of the EM algorithm is incorrect and affects its performance.
method Proposes a new EM algorithm called the Channel Matching (CM) EM algorithm and provides an initialization map.
result The locally maximal Q can affect the convergent speed but not the global convergence.
This paper studies GAIL's global convergence for general MDP and nonlinear rewards.
problem Understanding when GAIL algorithms achieve global convergence for general MDP and nonlinear rewards.
method Characterization of global convergence for various policy gradient algorithms applied to GAIL.
result First systematic theoretical study of GAIL for global convergence.
Paper proposes a quasi-Newton method for nonlinear equations with global convergence guarantees.
problem Solving smooth and monotone nonlinear equations efficiently and globally.
method Hybrid proximal extragradient framework combined with online learning for Jacobian approximation.
result First global convergence results showing quasi-Newton method's advantage over extragradient method.
Continuous-time distributed mirror descent with integral feedback converges to global optimum.
problem Distributed optimization of a global strongly convex function with local convex components.
method Continuous-time distributed mirror descent with integral feedback.
result Asymptotic convergence to global optimum with constant step-size.
The Expectation-Maximization algorithm is perhaps the most broadly used algorithm for inference of latent variable problems. A theoretical understanding of its performance, however, largely remains lacking. Recent results established that EM enjoys global convergence for Gaussian Mixture Models. For Mixed Linear Regres…
Paper improves a method for fast global and local convergence in optimization.
problem Slow global convergence in optimization methods with noisy Hessian estimates.
method Stochastic Newton Proximal Extragradient method using HPE framework.
result Faster global linear rate and superlinear convergence in fewer iterations.
Study describes global existence and convergence of flows on surfaces and fibrations.
problem Global existence and convergence of flows on surfaces and fibrations.
method Complete description of Ricci-Yang-Mills flow and pluriclosed flow on Tk bundles over Riemann surfaces. result Equivalence of solutions to generalized Ricci flow and pluriclosed flow with symmetry.
Local adaptive methods in FL can accelerate convergence but introduce bias, which is corrected.
problem The effect of using adaptive optimization methods for local updates in federated learning.
method Proposed correction techniques to overcome the bias introduced by local adaptive methods.
result Correction techniques can achieve faster convergence and higher test accuracy than baseline methods.
SGD converges globally to logistic loss minima for two-layer nets.
problem Global convergence of SGD for logistic loss on two-layer neural nets.
method Demonstrates existence of Frobenius norm regularized logistic loss functions as Villani functions, proving convergence and exponential rate.
result SGD converges globally to the global minima of appropriately regularized logistic empirical risk of depth 2 nets.
In this paper, the convergence time of federated learning (FL), when deployed over a realistic wireless network, is studied. In particular, a wireless network is considered in which wireless users transmit their local FL models (trained using their locally collected data) to a base station (BS). The BS, acting as a cen…
The DANE algorithm is an approximate Newton method popularly used for communication-efficient distributed machine learning. Reasons for the interest in DANE include scalability and versatility. Convergence of DANE, however, can be tricky; its appealing convergence rate is only rigorous for quadratic objective, and for …
Deep equilibrium models converge globally without explicit computation.
problem Global convergence of deep learning models with implicit layers.
method Analysis of gradient dynamics and proof of convergence rate.
result Deep equilibrium models converge to global optimum at a linear rate.
This paper extends the convergence rate of DEQs with ReLU to any general activation.
problem Proving global convergence rate for DEQs with general activations.
method Developed a novel population Gram matrix and new form of dual activation with Hermite polynomial expansion.
result Gradient descent converges to a globally optimal solution at a linear rate for DEQs with general activations.
We study the implicit bias of gradient descent methods in solving a binary classification problem over a linearly separable dataset. The classifier is described by a nonlinear ReLU model and the objective function adopts the exponential loss function. We first characterize the landscape of the loss function and show th…
RWR converges to global optimum in certain settings.
problem Proving convergence of RWR to optimal policy.
method Iterative learning with return-weighted log-likelihood.
result RWR converges to global optimum under certain conditions.
Deep learning has aroused extensive attention due to its great empirical success. The efficiency of the block coordinate descent (BCD) methods has been recently demonstrated in deep neural network (DNN) training. However, theoretical studies on their convergence properties are limited due to the highly nonconvex nature…
Wide neural networks with asymmetrical node scaling converge globally and learn features.
problem Global convergence and feature learning in over-parameterised shallow networks.
method Gradient-based optimisation of wide, shallow neural networks with asymmetrical node scaling.
result Gradient flow and gradient descent converge to a global minimum and learn features, unlike in the NTK parameterisation.
SGD methods fail to converge to global minimizers in deep neural networks with ReLU activation.
problem Failure of SGD methods to converge to global minimizers in deep neural networks.
method Stochastic Gradient Descent (SGD) and its variants like Adam, RMSProp, etc.
result SGD methods fail to converge to global minimizers with high probability in deep neural networks with ReLU activation.
AdaLoss optimizes adaptive learning rates for efficient convergence in various models.
problem Efficiently optimizing adaptive learning rates for gradient descent methods.
method AdaLoss uses loss function information to dynamically adjust step sizes.
result AdaLoss achieves linear convergence in linear regression and robust global convergence in neural networks.
Coded Federated Learning speeds up model convergence by preemptively computing on parity data.
problem Federated learning's convergence is slow on heterogeneous platforms due to stragglers.
method Develops CFL scheme where clients generate parity data and share it once, allowing the server to compute redundantly.
result CFL allows global model to converge nearly four times faster than uncoded federated learning.
Paper develops NPG for risk-averse RL with ECRMs, proving global convergence.
problem Ensuring reliable performance in stochastic RL problems with risk-averse policies.
method Developed natural policy gradient updates for ECRMs-based RL problems, proving global optimality and iteration complexity.
result Global convergence of risk-averse NPG algorithm with ECRMs.
BCD algorithm finds global minima in neural networks.
problem Training deep neural networks to find global minima.
method Block coordinate descent with skip connections and non-negative projection.
result Proves convergence to global minima for strictly monotonic and ReLU activations.
Global convergence proved for three-layer neural networks in mean field regime.
problem Optimization efficiency of multilayer neural networks in the mean field regime.
method Developed a rigorous framework for mean field limit of three-layer networks using stochastic gradient descent and neuronal embedding.
result Global convergence guarantee for unregularized feedforward three-layer networks in the mean field regime.
Federated learning (FL) provides a communication-efficient approach to solve machine learning problems concerning distributed data, without sending raw data to a central server. However, existing works on FL only utilize first-order gradient descent (GD) and do not consider the preceding iterations to gradient update w…
GBML with deep nets converges globally and generalizes well.
problem Theoretical guarantees for few-shot learning with deep nets.
method Proving global convergence and generalization bounds for GBML with over-parameterized DNNs.
result GBML with over-parameterized DNNs converges globally to the optimum at a linear rate and achieves good generalization.