Overview of high-dimensional dynamical systems and their applications to machine learning.
problem Characterizing behavior of high-dimensional dynamical systems driven by random matrices.
method Cavity method arguments, path integrals, dynamical mean field theory (DMFT), and random matrix resolvents.
result Connections between random matrix resolvents and DMFT response, and non-monotonic loss curves in training.
Analyzes high-dimensional SGD dynamics using DMFT.
problem Understanding the high-dimensional behavior of multi-pass SGD with small batch sizes.
method Derives DMFT equations for high-dimensional SGD dynamics.
result Proves DMFT equations characterize the asymptotic distribution of SGF parameters.
Unified analysis of DLNs using DMFT reveals dynamics of loss convergence and generalization trade-offs.
problem Understanding the overall dynamics of diagonal linear networks (DLNs) in neural network training.
method Dynamical Mean-Field Theory (DMFT) applied to DLNs.
result Derives low-dimensional effective process capturing high-dimensional gradient flow dynamics.
Adaptive kernels from neural networks improve model performance.
problem Improving neural network performance through adaptive kernels.
method Deriving adaptive kernels from infinite-width neural networks using feature learning and gradient flow training.
result Adaptive kernels achieve lower test loss compared to traditional kernels.
New scaling framework for MoE architectures ensures stability and optimal performance at scale.
problem Lack of principled understanding of how hyperparameters should scale in MoE architectures.
method Developed a novel Dynamical Mean Field Theory (DMFT) for three scaling regimes of MoE architectures.
result Derived Maximally Scale-Stable Parameterization (MSSP) for SGD and Adam, providing robust learning rate transfer and monotonic improvement with scale.
Analysis of deep neural networks under various learning rules reveals dynamics of feature and prediction learning.
problem Understanding how different learning rules affect feature and prediction dynamics in deep neural networks.
method Analysis of infinite-width deep networks trained with gradient descent and various learning rules.
result The evolution of the output function is governed by an effective neural tangent kernel (eNTK), which varies depending on the learning rule and training regime.
Study examines infinite limits of transformer dynamics, identifying key parameterizations.
problem Understanding the training dynamics of transformer models in the feature learning regime.
method Analysis of infinite scaling limits using dynamical mean field theory.
result Identified parameterizations that admit well-defined infinite width and depth limits.
Proves equations for high-dimensional gradient-based methods from Gaussian data.
problem High-dimensional asymptotics of gradient-based learning algorithms.
method Closed-form equations derived from dynamical mean-field theory.
result Equations match those from discretized DMFT for gradient flow.
Study on neural network dynamics in high dimensions with quadratic activation.
problem Understanding training dynamics in overparameterized neural networks.
method Derivation of gradient flow equations and analysis under l2-regularization.
result Characterization of estimator performance and spectral properties in the high-dimensional limit.
Residual networks with depthwise hyperparameter scaling transfer optimal hyperparameters across width and depth.
problem The challenge of hyperparameter tuning in deep learning, especially for large models.
method Combining μ μ μ P parameterization with residual networks having a residual branch scale of 1 / e x t d e p t h 1/\sqrt{ ext{depth}} 1/ e x t d e pt h . result Optimal hyperparameters transfer across width and depth in residual networks trained with this parameterization.
Two-layer networks learn faster with batch reuse, overcoming information and leap exponents.
problem Limitations of gradient flow and single-pass GD in learning multi-index target functions.
method Multi-pass gradient descent that reuses batches, analyzed using Dynamical Mean-Field Theory.
result Two-time-step overlap with target subspace for non-staircase functions, overcoming information and leap exponents.
The study examines spectral dynamics in deep neural networks, predicting how outliers evolve during training.
problem Understanding spectral evolution in deep neural networks during training.
method Developed a two-level dynamical mean-field theory (DMFT) to track spectral dynamics.
result The theory predicts how outliers evolve with training time, width, output scale, and initialization variance.
Study on fluctuations in neural network kernels and predictions, focusing on finite width effects.
problem Characterizing fluctuations in finite width neural networks.
method Dynamical mean field theory analysis of wide but finite feature learning neural networks.
result Fluctuations in kernels and predictions are dynamically coupled, leading to reduced variance in feature learning regimes.
ResNets converge to a limit model with improved error rates.
problem Understanding convergence of ResNets in the large-scale limit.
method Combining cavity method and propagation of chaos arguments on skeleton maps.
result Convergence rate of O(1/sqrt(D)) for ResNets in the large-scale limit.