Natural gradient descent speeds up convergence in neural networks, especially with overparameterization.
problem Mitigating the effects of curvature in neural network optimization.
method Analysis of natural gradient descent on nonlinear neural networks with stability conditions.
result Natural gradient descent converges efficiently under specific conditions for overparameterized networks.
Gradient descent converges globally in deep linear residual networks with ZAS initialization.
problem Optimizing deep linear residual networks for convergence.
method Zero-asymmetric (ZAS) initialization for gradient descent.
result Gradient descent converges to an ε-optimal point in O(L^3 log(1/ε)) iterations.
Neural networks trained with actor-critic algorithms converge to ODEs under weak convergence analysis.
problem Challenges in convergence analysis due to changing data distributions in online learning.
method Geometric ergodicity of data samples, Poisson equation, weak convergence techniques.
result Actor and critic networks converge to solutions of ODEs with random initial conditions.
We simplify deep learning convergence analysis using basic math.
problem Ensuring reliable convergence of optimization algorithms in deep networks.
method Elementary arguments and computations to estimate smoothness constants.
result Systematic computation of smoothness constants for first-order algorithms.
Study shows neural networks trained with GD converge to Gaussian processes with polynomial decay.
problem Understanding convergence of neural networks to Gaussian processes during training.
method Explicit upper bounds on quadratic Wasserstein distance between trained networks and Gaussian approximations.
result Polynomial decay of approximation error with network width and training time.
Study shows directional convergence for neural networks under spherical symmetry.
problem Learning linear predictors with neural networks under spherically symmetric data.
method Analysis of gradient flow and gradient descent for two-layer and deep linear networks.
result Directional convergence guarantees with exact convergence rate for specific network architectures.
The paper improves theoretical bounds on deep neural networks' convergence.
problem Understanding convergence of over-parameterized deep neural networks.
method Surrogate network construction with fixed activation patterns.
result Convergence to a global minimum guaranteed for networks with quadratic width and linear depth.
New method shows AdaGrad converges globally to neural network minima.
problem Convergence of adaptive gradient methods for neural networks.
method Proposed adaptive gradient method for over-parameterized neural networks.
result Converges to global minimum in polynomial time for two-layer networks.
Study on SGD for overparameterized neural networks, focusing on convergence rates.
problem Understanding convergence rates of SGD in overparameterized two-layer neural networks.
method Combines NTK approximation with RKHS analysis to explore SGD dynamics.
result Established sharp convergence rates for SGD in overparameterized two-layer neural networks.
GAIL with neural networks converges to global optima and has a known rate.
problem Uncertainty about GAIL with neural networks achieving global optimality.
method Gradient-based alternating updates algorithm.
result Established sublinear convergence to globally optimal solution.
Deep neural networks converge to Gaussian mixtures as layer width increases.
problem Understanding the distribution of outputs from deep neural networks.
method Proof and experiments with a simple model showing the convergence of neural network outputs to Gaussian mixtures.
result Neural networks converge to Gaussian mixtures as the width of the last hidden layer increases.
We consider the problem of accelerating distributed optimization in multi-agent networks by sequentially adding edges. Specifically, we extend the distributed dual averaging (DDA) subgradient algorithm to evolving networks of growing connectivity and analyze the corresponding improvement in convergence rate. It is know…
Neural networks with bounded weights converge to unique fixed points.
problem Convergence and fixed points of neural networks with bounded weights.
method Analysis of iterative neural networks in Hilbert spaces with a single mild condition on weights.
result Guaranteed convergence to a unique fixed point with a bound on its norm.
Neural networks with Xavier initialization converge to global minimum in the scaling limit.
problem Optimizing neural networks with Xavier initialization in the large network limit.
method Stochastic analysis and convergence to a random ODE with a Gaussian distribution.
result The neural network converges to a global minimum in the limit, with zero loss.
This paper proves SGD converges to global minimum for over-parameterized ReLU networks.
problem Theoretical understanding of implicit neural networks is limited.
method Gradient flow analysis of ReLU activated implicit neural networks.
result Randomly initialized gradient descent converges to global minimum at a linear rate for square loss function in over-parameterized ReLU networks.
Gradient flow of ReLU networks converges in low-correlation high-dimensional data.
problem Convergence of shallow ReLU networks trained on weakly interacting data.
method Gradient flow analysis with Polyak-Łojasiewicz viewpoint.
result Network width of order log(n) neurons suffices for global convergence with high probability.
Averaged SGD achieves optimal convergence rate for neural networks in the NTK regime.
problem Convergence analysis of averaged stochastic gradient descent for neural networks.
method Analyzed convergence of averaged stochastic gradient descent for overparameterized two-layer neural networks.
result Achieved minimax optimal convergence rate with global convergence guarantee.
Gradient descent converges to global minima for ResNets with linearly scaled width.
problem Understanding the convergence of deep residual networks with varying network width and dataset size.
method Analyzing the Jacobian of ResNets and applying gradient descent for quadratic loss.
result Gradient descent converges to global minima for ResNets with linearly scaled width and independent of depth.
Gradient descent converges to minimum Bayes risk for two-layer ReLU networks in mean field regime.
problem Training two-layer ReLU networks using gradient descent in the mean field regime.
method Describes a condition for convergence to minimum Bayes risk, extending previous results to ReLU-activated networks.
result The condition for convergence does not depend on initialization and concerns weak convergence of network realization.
Global convergence of multilayer neural networks proven for any depth.
problem Global convergence of multilayer neural networks in the mean field regime.
method Mean field limit framework, neuronal embedding, bidirectional diversity condition.
result Global convergence for multilayer networks of any depths, including correlated initializations.
Deep neural networks with adversarial training achieve sup-norm convergence for nonparametric regression.
problem Achieving sup-norm convergence for deep neural network estimators in nonparametric regression.
method Developed an adversarial training scheme to address the sup-norm convergence issue.
result Deep neural network estimators achieve optimal sup-norm convergence with the proposed adversarial training.
Global convergence proved for three-layer neural networks in mean field regime.
problem Optimization efficiency of multilayer neural networks in the mean field regime.
method Developed a rigorous framework for mean field limit of three-layer networks using stochastic gradient descent and neuronal embedding.
result Global convergence guarantee for unregularized feedforward three-layer networks in the mean field regime.
The paper proves stability and convergence of minimal networks under curvature motion.
problem Stability and convergence of minimal networks under curvature motion.
method Proved Lojasiewicz-Simon gradient inequalities for minimal networks.
result Motion by curvature starting from networks close to minimal ones exists for all times and smoothly converges.
Convolutional neural networks improve image classification accuracy.
problem Improving accuracy in image classification.
method Analyzing the convergence rate of misclassification risk for image classifiers.
result A rate of convergence independent of image dimension proves the effectiveness of CNNs.
SGD fails to converge for deep ReLU networks with limited random initializations.
problem SGD convergence in deep neural networks with limited random initializations.
method Analysis of four discretization parameters: network architecture, training data, gradient steps, and random initializations.
result SGD fails to converge for ReLU networks with depth much larger than width.
Deep networks converge in direction, with implications for predictions and margins.
problem Understanding convergence and alignment in deep learning networks.
method Developed a theory of unbounded nonsmooth Kurdyka-Łojasiewicz inequalities for functions definable in an o-minimal structure.
result Network weights, predictions, training errors, and margin distribution converge in direction and align with gradient flow.
Paper analyzes convergence of SGD in RNNs, proving linear convergence rate.
problem Analyzing convergence of SGD in multi-layer recurrent neural networks.
method Developed tools to analyze multi-layer networks with ReLU activations.
result SGD achieves linear convergence rate in training RNNs with sufficient neurons.
Paper finds conditions for benign overfitting in neural networks.
problem Benign overfitting in leaky ReLU two-layer neural networks.
method Established directional convergence and classification error bounds.
result Benign overfitting occurs with high probability on mixture data.
DCDC calculates convergence rates for Markov chains using neural networks.
problem Computing precise convergence rates for Markov chains is hard.
method Developed a neural network-based algorithm (DCDC) to bound convergence rates in Wasserstein distance.
result Demonstrated effective convergence bounds for real-world Markov chains.
Study on Adam-family methods for nonsmooth optimization with convergence guarantees.
problem Training nonsmooth neural networks with convergence guarantees.
method Two-timescale updating scheme and stochastic subgradient methods with gradient clipping.
result Convergence guarantees for various Adam-family methods in training nonsmooth neural networks.
Deep linear ResNets converge globally with certain transformations.
problem Global convergence of training deep linear ResNets.
method Gradient descent and stochastic gradient descent for training L-hidden-layer linear ResNets. result GD and SGD can converge to global minimum for deep linear ResNets with specific transformations.
SGD converges with positive probability for non-convex deep neural networks under specific conditions.
problem Convergence of SGD for non-convex deep neural networks.
method Established local convergence with positive probability under local Łojasiewicz condition and additional structural assumption.
result SGD converges with positive probability for non-convex deep neural networks under specific conditions.
Improved neural network convergence with causal Bayesian modeling in retail performance.
problem Improving neural network convergence in retail performance models.
method Causal Bayesian neural network implementation, removal of weakest SEM path, Flipout layers, Vadam optimizer.
result Neural network convergence improved with removal of the weakest SEM path.
New birth-death dynamics accelerates convergence in neural networks.
problem Accelerating convergence in neural networks with large parameters.
method Proposed a non-local mass transport dynamics as a stochastic neuronal birth-death process.
result Proved that the birth-death dynamics accelerates the rate of convergence in the mean-field limit.
Gaussian BP algorithm converges exponentially under walk summability for cyclic graphs.
problem Convergence rate of Gaussian BP for cyclic graphs.
method Extending known results on walk summability, proving exponential convergence rate.
result Gaussian BP converges exponentially under walk summability for cyclic graphs.
Frank-Wolfe optimization applied to a small deep network shows slower convergence compared to gradient descent.
problem Training deep neural networks is challenging due to many parameters and optimization difficulties.
method Frank-Wolfe optimization method applied to a deep network.
result Frank-Wolfe optimization converges slowly and is unstable in a stochastic setting.
MFCNs use sparse graphs to approximate manifold convergence.
problem Understanding manifold neural networks (MNNs).
method Sparse graph approximation for manifold convergence.
result Method converges to continuum limit as data points increase.
Convolutional neural networks converge quickly with gradient descent.
problem Learning efficient image classifiers with over-parameterized networks.
method Gradient descent for training over-parametrized CNNs with global average-pooling.
result Gradient descent quickly reduces the misclassification risk of CNNs.
Wide residual networks generalize well with uniform convergence to RNTK as width increases.
problem Understanding the generalization ability of wide residual networks.
method Uniform convergence of residual network kernel to residual neural tangent kernel (RNTK).
result Generalization error converges to kernel regression error with respect to RNTK.
Study on convergence of graph neural networks on random graphs.
problem Convergence of message passing graph neural networks on large random graphs.
method Extended convergence results to a broad class of aggregation functions using McDiarmid inequality.
result Non-asymptotic bounds for convergence quantified with high probability.
Polynomial networks converge to Gaussian processes at a rate of O(n^(-1/2)).
problem Understanding the convergence rate of polynomial networks to Gaussian processes.
method Examined one-hidden-layer neural networks with random weights, focusing on polynomial activations and their convergence rate in the 2-Wasserstein metric.
result The rate of convergence for polynomial networks to Gaussian processes is $O(n^{- rac{1}{2}})$.
Orthogonal initialization speeds up convergence in deep linear networks.
problem The impact of initialization on convergence speed and model performance in deep neural networks.
method Analysis of orthogonal initialization in deep linear networks, proving its superiority over Gaussian initialization.
result Orthogonal initialization speeds up convergence relative to Gaussian initialization in deep networks.
GD converges faster to flatter minima than gradient flow in shallow networks.
problem Understanding the dynamics of gradient descent in shallow linear networks.
method Analyzing the convergence rate and solution of gradient descent in depth-2 linear neural networks.
result GD converges linearly to flatter minima than gradient flow, even with large step sizes.
The paper studies neural networks' convergence near origin and saddle points.
problem Directional convergence of neural networks near small initializations and saddle points.
method Gradient flow dynamics analysis of two-homogeneous neural networks.
result Neural networks' weights approximately converge in direction to KKT points for small initializations.
Researchers develop neural networks for manifold data with a convergence rate.
problem Analyzing high-dimensional data on non-Euclidean domains.
method Constructing manifold neural networks using spectral decomposition of the Laplace Beltrami operator.
result Established a rate of convergence for the neural network scheme that depends on intrinsic manifold dimension.
Gradient methods converge exponentially in concave network games.
problem Finding Nash equilibria in concave network zero-sum games.
method Gradient Ascent and Optimistic Gradient Ascent analyses.
result Exponential convergence rates in various game settings.
Kolmogorov-Arnold Networks achieve optimal convergence rates in nonparametric regression.
problem Nonparametric function approximation in multivariate settings.
method Structured additive and multiplicative KANs using B-splines.
result Achieve minimax-optimal convergence rate O(n−2r/(2r+1)) for Sobolev space functions. FA algorithm provides convergence guarantees for deep linear networks.
problem Training efficiency and convergence of deep neural networks.
method Theoretical analysis of Feedback Alignment (FA) algorithm for deep linear networks.
result Certain initializations lead to implicit anti-regularization, affecting learning effectiveness.