Study shows critical initialisation not crucial for ReLU networks under dropout limits.
problem Effect of initialisation on training speed and generalisation in ReLU networks.
method Large-scale statistical analysis of over 12,000 trained networks.
result Non-critical initialisations perform similarly to critical initialisations in terms of performance.
Study on functions computed by deep-layered machines finds same distribution in neural networks and Boolean circuits.
problem Understanding the space of functions computed by deep-layered machines.
method Investigation of Boolean functions on random-layered machines, including neural networks and Boolean circuits.
result The space of functions computed at large depth limit is characterized and the macroscopic entropy of Boolean functions is either monotonically increasing or decreasing with depth.
Study of deep linear neural networks with proportional width and depth.
problem Lack of descriptive power in Gaussian limit of deep linear neural networks.
method Proportional infinite-width infinite-depth limit for deep linear neural networks.
result Characterization of limiting distribution as a nontrivial mixture of Gaussians.
Study on feature learning dynamics in infinite-depth neural networks, focusing on ResNets.
problem Understanding how features evolve during training in deep neural networks, especially in the large-depth limit.
method Conditional Gaussian representations and SDE system with decoupled backward weights.
result Depth-induced suppression of forward-backward coupling in infinite-depth networks, leading to a decoupled forward-backward SDE system.
This paper analyzes convergence rates of neural networks in the deep learning regime.
problem Understanding convergence rates of neural networks in the deep learning regime.
method Analyzing the Neural Tangent Kernel (NTK) convergence rates in the large depth limit.
result Quantifies the impact of initialization and activation function on NTK convergence rates.
Scene recognition with RGB images has been extensively studied and has reached very remarkable recognition levels, thanks to convolutional neural networks (CNN) and large scene datasets. In contrast, current RGB-D scene data is much more limited, so often leverages RGB large datasets, by transferring pretrained RGB CNN…
Dropout and RaM become equivalent in large ResNets as depth and width increase.
problem Improving performance in deep learning models.
method Comparing Dropout and Random Gradient Masking in ResNets.
result Dropout and RaM converge to the same large-scale limiting dynamics in ResNets.
Study on neural network initialization with shaped infinite depth-and-width networks.
problem Understanding the distribution of random covariance matrices in shaped infinite-depth-and-width networks.
method Introduced the Neural Covariance SDE to model the distribution of the random covariance matrix.
result Identified the precise scaling of the activation function necessary for a non-trivial limit.
In order-driven markets, limit-order book (LOB) resiliency is an important microscopic indicator of market quality when the order book is hit by a liquidity shock and plays an essential role in the design of optimal submission strategies of large orders. However, the evolutionary behavior of LOB resilience around liqui…
New algorithm proves deep networks can learn better than shallow ones.
problem Understanding the power difference between shallow and deep neural networks.
method Identifying a class of Boolean functions and proving that logarithmic-depth networks can learn them efficiently using hierarchical reconstruction.
result First algorithmic separation between constant-depth and logarithmic-depth neural networks.
Introduces Polar Depth for analyzing multivariate heavy-tailed data extremes.
problem Analyzing the behavior of extremes from multivariate heavy-tailed distributions.
method Introduces Polar Depth, a novel statistical depth function expressed in polar coordinates.
result The polar depth of the largest observations converges to the polar depth of the limiting distribution as the threshold increases.
Study infinite-depth limits of neural networks with fixed width.
problem Understanding the behavior of neural networks as depth increases with fixed width.
method Analyzing finite-width residual networks with random Gaussian weights, focusing on the infinite-depth limit.
result The pre-activations converge to a zero-drift diffusion process, differing from the infinite-width limit.
Residual networks with depthwise hyperparameter scaling transfer optimal hyperparameters across width and depth.
problem The challenge of hyperparameter tuning in deep learning, especially for large models.
method Combining μ μ μ P parameterization with residual networks having a residual branch scale of 1 / e x t d e p t h 1/\sqrt{ ext{depth}} 1/ e x t d e pt h . result Optimal hyperparameters transfer across width and depth in residual networks trained with this parameterization.
New Transformer architecture prevents rank degeneracy in deep attention models.
problem Rank degeneracy in deep attention models.
method Modified Softmax-based attention model with skip connections, centered at identity, and scaled logits.
result Existence of a stable SDE implies well-behaved covariance structure, preventing rank degeneracy.
Uniform scaling limits in AdamW-trained transformers converge to ODEs.
problem Understanding the dynamics of large-depth transformers trained with AdamW.
method Modeling transformer dynamics as an interacting particle system coupled through attention, proving convergence to ODEs.
result The joint dynamics of hidden states and backpropagated variables converge uniformly to an ODE system.
Stable ResNet stabilizes gradients in deep networks.
problem Gradient vanishing and exploding in deep ResNet architectures.
method Introducing Stable ResNet architectures with gradient stabilization and infinite depth expressivity.
result Stable ResNet maintains gradient stability and expressivity in deep networks.
The paper proves barriers to approximating functions with small weights and depth in neural networks.
problem Proving barriers to approximating functions with constant depth neural networks.
method Reduction to open problems and natural-proof barriers in circuit complexity, and a new approach to polynomially-bounded functions.
result There are fundamental barriers to proving results beyond depth 4 for constant-depth neural networks.
This work challenges the Neural Tangent Kernel's role in overparameterized neural networks, especially with large width and depth.
problem The Neural Tangent Kernel's behavior in overparameterized neural networks with large width and depth is unclear.
method Experimental and theoretical analysis of ReLU networks with large width and depth.
result The aggregate norm of hidden neuron deviations does not vanish in infinitely-wide ReLU networks, indicating non-trivial behavior.
Deep and wide ReLU networks learn data-dependent features even in the lazy training regime.
problem Understanding the behavior of neural networks with finite depth and width.
method Analyzing the mean and variance of the neural tangent kernel (NTK) in a randomly initialized ReLU network.
result The NTK has a non-trivial evolution during training, with the mean of its first SGD update being exponential in the ratio of depth to width.
Theoretical limits of deep residual networks show consistent covariance structures.
problem Understanding the limits of deep residual networks.
method Analyzing the behavior of deep residual networks with skip connections as width and depth approach infinity.
result Theoretical analysis confirms that the covariance structure remains consistent regardless of the order of width and depth.
A new depth measure and median defined on Hadamard manifolds.
problem Statistical depth and median on Hadamard manifolds.
method Horospherical depth and Busemann median defined using renormalized distance functions.
result The Busemann median exists for every Borel probability measure on Hadamard manifolds.
ResNets approximate log-Gaussian at initialization, improving network performance.
problem Understanding the initialization behavior of deep neural networks like ResNets.
method Analyzing ReLU ResNets in the infinite-depth-and-width limit, showing log-Gaussian behavior.
result ResNets at initialization exhibit hypoactivation and interlayer correlations, which are not captured by Gaussian limits.
Structured dropout reduces transformer depth for efficient training and inference.
problem Overparameterized transformer networks overfit and require large computation.
method LayerDrop structured dropout for efficient pruning and regularization.
result Sub-networks of any depth can be selected from a large network without performance loss.
The study analyzes deep linear networks from random initialization, capturing dynamics and hyperparameter effects.
problem Understanding training dynamics in deep linear networks from random initialization.
method Theoretical analysis of gradient descent dynamics in deep linear networks with random initialization and large data.
result Captures the 'wider is better' effect and hyperparameter transfer effects, contrasting with neural-tangent parameterization.
One-shot neural architecture search limits depth search space and prunes networks for better performance and uncertainty.
problem Finding optimal depth in residual networks for efficient training and inference.
method Formulated a variational objective to approximate the depth distribution and pruned networks based on this distribution.
result Pruned networks achieve competitive accuracy with unpruned networks and better uncertainty calibration.
This paper explores how neural network width and depth behave as they approach infinity.
problem Understanding the behavior of neural functions as width and depth go to infinity.
method Formal definition of commutativity framework, study of neural covariance kernel, novel proof techniques.
result Taking width and depth to infinity in a deep neural network with skip connections results in the same covariance structure, regardless of the order of taking limits.
Bayesian linear networks reveal optimal depth and width trade-offs.
problem Understanding how depth, width, and dataset size affect model quality in linear networks.
method Zero noise Bayesian inference with Gaussian weight priors and mean squared error.
result Optimal predictions at infinite depth and maximized Bayesian model evidence at infinite depth.
ResNets converge to a limit model with improved error rates.
problem Understanding convergence of ResNets in the large-scale limit.
method Combining cavity method and propagation of chaos arguments on skeleton maps.
result Convergence rate of O(1/sqrt(D)) for ResNets in the large-scale limit.
Deep imagination optimizes decision-making in large trees with limited resources.
problem Optimal planning in large decision trees with limited resources and time.
method Analytical solutions and numerical analysis of sampling capacity allocation.
result Optimal policy is to allocate few samples per level for deep exploration, favoring depth over breadth.
Study how depth affects inference in deep Bayesian neural networks.
problem Understanding how depth impacts inference in overparameterized linear Bayesian neural networks.
method Interpreting finite deep linear Bayesian neural networks as scale mixtures of Gaussian process predictors.
result Advances analytical understanding of how depth affects inference in a simple class of Bayesian neural networks.
Deep ResNets can achieve zero loss with enough layers and weights.
problem Finding parameters in deep ResNets that fit training data perfectly.
method Mean-field analysis and gradient descent flow to a PDE.
result Gradient descent for ResNet training converges to a zero-loss solution in the large-NN limit.
Inverse depth scaling found in LLMs due to similar layers averaging error.
problem Understanding how depth affects loss in large language models.
method Analysis of LLMs and toy residual networks.
result Loss scales inversely proportional to depth in LLMs.
Investigates the relationship between ResNets and Neural ODEs, quantifying their closeness and providing training methods.
problem Quantifying the distance between ResNet dynamics and Neural ODE solutions.
method Bounding the distance between hidden state trajectories and Neural ODE solutions, using gradient descent and Heun's method.
result Gradient descent and Heun's method can implicitly regularize ResNets towards Neural ODEs, especially for smooth residual functions.
This work studies fluctuation in multilayer neural networks using mean field theory.
problem Understanding fluctuation in multilayer neural networks with mean field training.
method Developed a second-order mean field limit to capture fluctuation, demonstrating stability of gradient descent training.
result Gradient descent training in multilayer networks biases towards minimal fluctuation, even after convergence.
Study reveals how depth of reasoning affects generalization in models.
problem Understanding scaling behavior of generalization with CoT depth.
method Theoretical model of CoT in linear regression using random matrix theory.
result Sharp phase transition between exponential and polynomial improvement, saturation, and overthinking.
New insights into how depth and width affect in-context learning in deep models.
problem Understanding how various resources impact in-context learning in deep models.
method Analyzed linear regression in a deep linear self-attention model, varying resources like depth, width, context length, and training steps.
result Increasing depth improves in-context learning even at infinite context length, contrary to previous findings.
Deep convolutional neural networks (CNNs) used in practice employ potentially hundreds of layers and 10 10 10 , 000 000 000 s of nodes. Such network sizes entail significant computational complexity due to the large number of convolutions that need to be carried out; in addition, a large number of parameters needs to be learned and…
We study the behavior of untrained neural networks whose weights and biases are randomly distributed using mean field theory. We show the existence of depth scales that naturally limit the maximum depth of signal propagation through these random networks. Our main practical result is to show that random networks may be…
Graphs avoid oversmoothing with properly initialized weights.
problem Oversmoothing in graph convolutional networks (GCNs) limits their depth.
method Used Gaussian process equivalence to analyze GCNs, identifying a non-oversmoothing phase.
result Properly initialized GCNs can be deep and expressive.
Study examines infinite limits of transformer dynamics, identifying key parameterizations.
problem Understanding the training dynamics of transformer models in the feature learning regime.
method Analysis of infinite scaling limits using dynamical mean field theory.
result Identified parameterizations that admit well-defined infinite width and depth limits.
Wide neural networks can degrade performance, contrary to conventional wisdom.
problem Understanding the limitations of increasing network width in neural networks.
method Using Deep Gaussian Processes to decouple capacity and width, analyzing their effects on representational power and non-Gaussianity.
result Wide neural networks can become less adaptable and more Gaussian, leading to performance degradation.
Framework captures neural network learning in large-width limit.
problem Understanding learning dynamics in large neural networks.
method Developed a rigorous framework for multilayer neural networks in mean field limit.
result Global convergence guarantees for various network architectures and initializations.
New method improves reliability of depth estimation models.
problem Uncertainty quantification in large-scale vision models.
method Parameter-efficient Bayesian neural networks with PEFT methods.
result Combining PEFT methods with Bayesian inference enhances predictive performance.
We propose an analytically tractable class of models for the dynamics of a limit order book, described through a stochastic partial differential equation (SPDE) with multiplicative noise for the order book centered at the mid-price, along with stochastic dynamics for the mid-price which is consistent with the order flo…
New method for multi-modal depth prediction challenges.
problem Challenges in applying unimodal coreset selection to multi-modal data.
method Adapted state-of-the-art coreset selection technique for multimodal data.
result Challenges in extending unimodal algorithms to multi-modal scenarios.
This paper studies activation sparsity in large language models, finding key trends and implications.
problem Activation sparsity in large language models (LLMs) can be improved for efficiency and interpretability.
method Proposes PPL- p % p\% p % sparsity, analyzes trends with training data, width-depth ratio, and parameter scale. result ReLU is more efficient for sparsity than SiLU, and deeper architectures can improve sparsity.
Our research proves neural collapse in deep ResNets and transformers is globally optimal.
problem Understanding neural collapse in deep learning models.
method Analysis of deep regularized transformers and ResNets trained with cross entropy or mean squared error loss.
result Global optima of deep regularized transformers and ResNets are approximately collapsed, becoming more prominent as depth increases.
The paper explores how the depth of neural networks affects their ability to represent data accurately.
problem Understanding the implicit bias and rank of neural networks with large depth.
method Analyzing the convergence of representation cost to a notion of rank as network depth increases, and investigating conditions for recovering the true rank of data.
result There is a range of network depths where the true rank of data is recovered, and this affects the topology of class boundaries.