Analyzes SGD dynamics in two-layer networks, bridging different regimes.
problem Understanding SGD dynamics in high-dimensional and mean-field settings.
method Rigorous analysis via deterministic low-dimensional description of sufficient statistics.
result Infinite-width dynamics remains close to a low-dimensional subspace.
Study shows how large neural networks avoid overfitting through decoupling of feature learning and complexity growth.
problem Understanding inductive bias and generalization in large neural networks.
method Dynamical mean field theory applied to large two-layer networks.
result Training dynamics of large networks exhibit a separation of timescales, decoupling feature learning and overfitting.
The paper studies learning dynamics in two-layer neural networks.
problem Learning dynamics and time scales in two-layer neural networks.
method Gradient flow dynamics of a wide two-layer neural network in high-dimension, with data following a single-index model.
result The learning dynamics exhibit separation of timescales and intermittency.
Two-layer neural networks learn efficiently using kernel methods in mean-field analysis.
problem Feature learning ability of two-layer neural networks in the mean-field regime.
method Mean-field analysis through kernel methods, focusing on dynamics of the first layer's kernel.
result Two-layer neural networks can learn a union of multiple reproducing kernel Hilbert spaces more efficiently than kernel methods.
Study on symmetries in wide neural networks' dynamics without bias.
problem Understanding symmetries in the dynamics of wide two-layer neural networks.
method Analyzing symmetries in gradient flow on population risk for infinitely wide networks.
result Symmetries can simplify the dynamics of predictors and reduce the dimensionality of the problem.
This work shows linear convergence for two-layer neural networks in mean-field regime.
problem Optimizing two-layer neural networks in the mean-field regime.
method Mean-field analysis and continuous-time noisy gradient descent.
result Establishes linear convergence rate for two-layer neural networks.
This paper presents a phase diagram for two-layer neural networks under different initialization scales.
problem Understanding the behavior of neural networks under varying scales of initialization.
method Analysis of a phase diagram for two-layer neural networks.
result Condensation of weight vectors on isolated orientations during training.
Two-layer networks learn faster with batch reuse, overcoming information and leap exponents.
problem Limitations of gradient flow and single-pass GD in learning multi-index target functions.
method Multi-pass gradient descent that reuses batches, analyzed using Dynamical Mean-Field Theory.
result Two-time-step overlap with target subspace for non-staircase functions, overcoming information and leap exponents.
Study of two-layer ReLU neural network phase diagram at infinite-width limit.
problem Characterize the dynamical regimes of two-layer ReLU neural networks.
method Combining experimental and theoretical approaches, including phase diagram analogy.
result Identification of three regimes: linear, critical, and condensed.
Gradient descent dynamics in neural networks show quenching and activation phases.
problem Understanding training dynamics in neural networks.
method Numerical and phenomenological study of gradient descent algorithm for two-layer neural networks.
result Gradient descent dynamics exhibit quenching and activation phases in under-parametrized networks.
It remains a puzzle that why deep neural networks (DNNs), with more parameters than samples, often generalize well. An attempt of understanding this puzzle is to discover implicit biases underlying the training process of DNNs, such as the Frequency Principle (F-Principle), i.e., DNNs often fit target functions from lo…
Auto-regressive models learn latent states from partially observed linear dynamical systems.
problem Understanding how auto-regressive models learn latent representations from partially observed linear dynamical systems.
method Empirical risk minimization on partially observed linear dynamical systems.
result Two-layer linear auto-regressive models learn to approximate Kalman filtering, coinciding with optimal state estimates.
SDA method reduces memory and time for assimilating noisy geophysical data.
problem Challenges in identifying state trajectories of high-dimensional geophysical systems.
method Score-based data assimilation with modified score network architecture.
result Promising results for a two-layer quasi-geostrophic model.
Study on SignGD optimization of two-layer transformer on noisy data.
problem Understanding how SignGD optimizes transformers and its generalization.
method Analysis of a two-layer transformer with SignGD on a linearly separable noisy dataset.
result SignGD converges fast but has poor generalization on noisy data.
Stochastic gradient methods converge for training wide PINNs.
problem Convergence of stochastic gradient descent in training over-parameterized PINNs.
method Established linear convergence of stochastic gradient descent/flow in training over-parameterized two-layer PINNs.
result Linear convergence with high probability for general activation functions.
Study on SGD for overparameterized neural networks, focusing on convergence rates.
problem Understanding convergence rates of SGD in overparameterized two-layer neural networks.
method Combines NTK approximation with RKHS analysis to explore SGD dynamics.
result Established sharp convergence rates for SGD in overparameterized two-layer neural networks.
A fairly comprehensive analysis is presented for the gradient descent dynamics for training two-layer neural network models in the situation when the parameters in both layers are updated. General initialization schemes as well as general regimes for the network width and training data size are considered. In the over-…
Two-layer networks learn hard GLMs with SGD in high dimensions.
problem Learning hard generalized linear models with SGD in high-dimensional settings.
method Reduction of SGD dynamics to a stochastic process in lower dimensions, focusing on the role of stochasticity.
result Overparameterization enhances convergence by a constant factor, suggesting minimal role of stochasticity.
Anisotropic data structure affects learning dynamics and generalization error in linear networks.
problem Understanding the impact of data anisotropy on learning dynamics and generalization error in linear networks.
method Examined a spiked covariance structure as a model of anisotropy in a two-layer linear network in a linear regression setting.
result Learning dynamics proceed in two phases: initially driven by input-output correlation, then by other principal directions of the data structure. Derived an analytical expression for the generalization error.
Two-layer CNNs can overfit well if initialized correctly.
problem Understanding the conditions for benign overfitting in over-parameterized CNNs.
method Extending analysis to fully trainable two-layer CNNs, examining initialization scaling effects.
result Initialization scaling of the output layer is crucial; large scales lead to fixed output behavior, small scales to complex interactions.
In this paper, we made an extension to the convergence analysis of the dynamics of two-layered bias-free networks with one ReLU output. We took into consideration two popular regularization terms: the ℓ1 and ℓ2 norm of the parameter vector w, and added it to the square loss function with coefficient $λ/…
Study reveals sharp characterisation of local minima in neural network loss landscapes.
problem Characterizing local minima in high-dimensional two-layer ReLU neural networks.
method Exact low-dimensional representation of local minima using summary statistics and link with one-pass SGD dynamics.
result Local minima in overparameterized neural networks form discrete families with varying stability and reachability.
A new perspective on self-attention models using MLPs.
problem Improving sequence modeling with self-attention mechanisms.
method Introducing HyperMLP and HyperGLU, which use dynamic two-layer MLPs with reverse-offset layout.
result HyperMLP/HyperGLU consistently outperform softmax-attention baselines.
This paper explains double descent in linear neural networks, identifying new factors.
problem Understanding double descent in linear neural networks.
method Gradient flow derivation and necessary conditions for double descent.
result Singular values of input-output covariance matrix are important for double descent in two-layer models.
Gradient descent learns over-param neural nets better than NTK.
problem Learning over-parametrized neural networks with ReLU activations.
method Gradient descent from random initialization on a Gaussian input distribution.
result Gradient descent achieves population loss o(1/d), while NTK achieves Ω(1/d). The study analyzes a three-layer neural network's training dynamics using a functional-space mean-field theory.
problem Understanding the training dynamics of partially-trained three-layer neural networks.
method Generalized mean-field theory to functional spaces, proving convergence and feature learning.
result The training loss of the model decays to zero at a linear rate in the L2 regression setting. This work improves sample efficiency in neural function approximation for reinforcement learning.
problem Improving sample efficiency in reinforcement learning with neural function approximation.
method Study of function approximation with two-layer neural networks (ReLU and polynomial activations) under generative and realizability models.
result Significant improvement in sample complexity compared to linear methods.
Study how neural networks learn from non-Gaussian data models.
problem Understanding neural network learning dynamics with non-Gaussian data.
method Developed a two-layer neural network with Hermite polynomial activations to control high-order cumulants.
result Neural networks progressively learn high-order cumulants after capturing low-order statistics.
AGF explains feature learning in neural networks through alternating steps.
problem Understanding what features neural networks learn and how they learn them.
method AGF is an algorithmic framework that approximates the dynamics of feature learning in two-layer networks.
result AGF provides a unified framework to understand feature learning in neural networks, matching experimental results across various architectures.
Study shows directional convergence for neural networks under spherical symmetry.
problem Learning linear predictors with neural networks under spherically symmetric data.
method Analysis of gradient flow and gradient descent for two-layer and deep linear networks.
result Directional convergence guarantees with exact convergence rate for specific network architectures.
New GM layers improve neural network performance.
problem Improving neural network performance.
method Employing Gaussian mixture models and Wasserstein gradient flows.
result GM layers achieve comparable performance to two-layer networks.
Improved sampling from mean-field stationary distributions.
problem Sampling from the stationary distribution of mean-field SDEs.
method Decoupling the problem into two aspects: approximation of mean-field SDE and sampling from finite-particle distribution.
result Improved guarantees in various settings, including optimizing neural networks.
Convolutional neural networks (CNNs) have achieved remarkable performance in various fields, particularly in the domain of computer vision. However, why this architecture works well remains to be a mystery. In this work we move a small step toward understanding the success of CNNs by investigating the learning dynamics…
Deep neural networks achieve stellar generalisation even when they have enough parameters to easily fit all their training data. We study this phenomenon by analysing the dynamics and the performance of over-parameterised two-layer neural networks in the teacher-student setup, where one network, the student, is trained…
We introduce exact macroscopic on-line learning dynamics of two-layer neural networks with ReLU units in the form of a system of differential equations, using techniques borrowed from statistical physics. For the first experiments, numerical solutions reveal similar behavior compared to sigmoidal activation researched …
Two-layer networks struggle with high frequencies due to numerical and computational limitations.
problem High frequency approximation and learning in shallow networks.
method Mathematical and computational analysis focusing on numerical error, computational cost, and stability.
result Explicit answers to fundamental computational issues in shallow networks' high frequency handling.
Global convergence proved for three-layer neural networks in mean field regime.
problem Optimization efficiency of multilayer neural networks in the mean field regime.
method Developed a rigorous framework for mean field limit of three-layer networks using stochastic gradient descent and neuronal embedding.
result Global convergence guarantee for unregularized feedforward three-layer networks in the mean field regime.
Multi-layer neural networks are among the most powerful models in machine learning, yet the fundamental reasons for this success defy mathematical understanding. Learning a neural network requires to optimize a non-convex high-dimensional objective (risk function), a problem which is usually attacked using stochastic g…
Efficient methods reduce projections in non-stationary online learning.
problem Optimizing dynamic and adaptive regret in non-stationary online learning environments.
method Presented efficient methods reducing the number of projections per round from O(logT) to 1. result Reduced number of projections per round from O(logT) to 1 for optimizing dynamic and adaptive regret. We consider networks, trained via stochastic gradient descent to minimize ℓ2 loss, with the training labels perturbed by independent noise at each iteration. We characterize the behavior of the training dynamics near any parameter vector that achieves zero training error, in terms of an implicit regularization te…
Gradient descent with logistic loss can make two-layer networks interpolate binary classification data.
problem Training two-layer networks for binary classification.
method Gradient descent with logistic loss applied to two-layer networks.
result Gradient descent can drive training loss to zero under certain conditions.
Two-layer neural networks can approximate functions with fractal singularities.
problem Characterizing functions that can be represented by infinitely wide two-layer neural networks.
method Representation formulas and pointwise properties analysis.
result Functions with fractal or curved singularities cannot be represented by two-layer networks with finite path-norm.
SGD learns two-layer neural networks efficiently, revealing scaling laws.
problem Learning efficiency and scaling laws in SGD for two-layer networks.
method Precise analysis of SGD dynamics for MSE minimization.
result Smooth scaling law in cumulative objective despite abrupt transitions for individual neurons.
Extends Langevin dynamics for constrained domains.
problem Optimization of constrained probability measures.
method Mirror mean-field Langevin dynamics (MMFLD).
result Linear convergence guarantees and propagation of chaos results.
Gradient descent amplifies random features in neural networks to useful ones.
problem Generalization in neural networks trained on corrupted data.
method Characterization of feature-learning process in two-layer ReLU networks trained by gradient descent.
result Gradient descent amplifies random features to useful ones, achieving near optimal generalization error.
When optimizing over-parameterized models, such as deep neural networks, a large set of parameters can achieve zero training error. In such cases, the choice of the optimization algorithm and its respective hyper-parameters introduces biases that will lead to convergence to specific minimizers of the objective. Consequ…
Analytical method finds deeper optima in two-layer ReLU networks.
problem Training two-layer ReLU networks with analytical methods.
method Analytically finding critical points of the loss function for one layer while keeping the other fixed.
result Significantly smaller training loss values on real datasets compared to gradient descent methods.
Studied how SGD's stability regularization affects generalization in neural networks.
problem Understanding why SGD often generalizes better than GD in neural networks.
method Analyzed stability of SGD and GD through Frobenius norm and trace of Hessian, and compared their generalization properties.
result Stable minima of SGD generalize well, while GD's stability-induced regularization is too weak.