Wide neural networks can benefit from multi-task learning in their infinite-width limit.
problem The generalization behavior of wide neural networks in multi-task learning settings.
method Optimizing wide ReLU neural networks with L2-regularization promotes multi-task learning in the infinite-width limit.
result An exact quantitative characterization of multi-task learning in the infinite-width limit of wide ReLU neural networks.
Stable processes emerge as limits of deep neural networks with symmetric stable distributions.
problem Understanding the behavior of deep neural networks as they become infinitely wide.
method Analyzing fully connected feed-forward deep neural networks with symmetric stable distributions and showing the limit as a stable process.
result The infinite wide limit of the network is a stable process with multivariate stable distributions.
There has recently been much work on the "wide limit" of neural networks, where Bayesian neural networks (BNNs) are shown to converge to a Gaussian process (GP) as all hidden layers are sent to infinite width. However, these results do not apply to architectures that require one or more of the hidden layers to remain n…
Wide stochastic networks show Gaussian behavior and improve training with PAC-Bayesian methods.
problem Analyzing and training over-parameterised neural networks with large width.
method Establishing Gaussian behavior for a stochastic architecture, applying PAC-Bayesian training.
result PAC-Bayesian training on large but finite-width networks outperforms standard methods.
Wide deep neural networks with Gaussian weights approximate Gaussian processes closely.
problem Understanding the approximation of deep neural networks with Gaussian weights to Gaussian processes.
method Established novel rates for the Gaussian approximation of random deep neural networks with Gaussian parameters and Lipschitz activation functions in the wide limit.
result The distance between the network output and the Gaussian approximation scales inversely with the width of the network.
The paper examines how deep linear neural networks behave as they become infinitely wide.
problem Understanding the behavior of deep linear neural networks as they approach infinite width.
method Analyzes the infinite-width limit of deep linear neural networks, proving convergence to deterministic models and providing precise laws for random weights.
result The training dynamics of deep linear neural networks converge to those of a deterministic model, and the weights' behavior is precisely described.
Infinite CNNs lose spatial correlations, but can be restored by correlated weights.
problem Infinite CNNs lose spatial correlations, which are crucial for their performance.
method Introduced correlated weights to restore spatial correlations in infinite CNNs.
result Optimal performance is achieved with a moderate level of weight correlation.
Embedded ensembles improve neural network performance efficiently.
problem Improving neural network performance with fewer resources.
method Analyzing the wide network limit of gradient descent dynamics using Neural-Tangent-Kernel.
result Embedded ensembles exhibit two regimes: independent and collective, affecting performance.
A longstanding goal in deep learning research has been to precisely characterize training and generalization. However, the often complex loss landscapes of neural networks have made a theory of learning dynamics elusive. In this work, we show that for wide neural networks the learning dynamics simplify considerably and…
Understanding the asymptotic behavior of wide networks is of considerable interest. In this work, we present a general method for analyzing this large width behavior. The method is an adaptation of Feynman diagrams, a standard tool for computing multivariate Gaussian integrals. We apply our method to study training dyn…
Study of deep Stable neural networks with various activation functions.
problem Characterizing the infinitely wide limits of deep Stable neural networks.
method Investigation of large-width properties of deep Stable NNs with a generalized central limit theorem for heavy tails.
result Extension of characterization to a broader class of activation functions, including sub-linear, asymptotically linear, and super-linear functions.
Tensor programs prove neural network limits for any architecture.
problem Understanding the limits of neural networks of any architecture.
method Prove convergence of neural network's Tangent Kernel (NTK) to a deterministic limit as network widths increase.
result Identify conditions for correct NTK limit calculation based on gradient independence assumption.
Wide networks with polynomial activations have proven asymptotic behavior.
problem Understanding the behavior of neural networks in the large width limit.
method Proving a conjecture for deep networks with polynomial activation functions.
result Tight bounds on the behavior of wide networks during stochastic gradient descent and derivation of their finite-width dynamics.
The paper studies neural networks with wide layers and finds a deformed semicircle law.
problem Investigating spectral distributions of neural networks in the ultra-wide regime.
method Analyzes empirical kernel matrices, proves deformed semicircle law, provides nonlinear Hanson-Wright inequality.
result Emergence of a deformed semicircle law in the ultra-wide neural network regime.
This paper shows how infinitely wide Tensor Networks converge to Gaussian Processes.
problem Understanding the relationship between Tensor Networks and Gaussian Processes.
method Analyzing the infinite-width limit of Tensor Networks and comparing them to Gaussian Processes.
result Infinitely wide Tensor Networks converge to Gaussian Processes, proving their equivalence.
Elman-type RNNs converge to globally optimal solutions in the mean-field regime.
problem Optimizing feature learning in wide RNNs.
method Analysis of gradient descent dynamics and mean-field limits.
result Fixed points of infinite-width dynamics are globally optimal.
Gradient descent dynamics in wide neural networks are analyzed using a dynamical CLT.
problem Understanding the fluctuations in wide shallow neural networks trained via gradient descent.
method Dynamical Central Limit Theorem (CLT) applied to neural network dynamics.
result Asymptotic fluctuations remain bounded in mean square throughout training.
Study on MC dropout in wide neural networks and its convergence to Gaussian processes.
problem Understanding the behavior of Monte Carlo dropout in wide neural networks.
method Rigorously studied the limiting distribution of wide untrained NNs under dropout, proving convergence to Gaussian processes. Investigated correlations and non-Gaussian behavior in finite width NNs.
result Wide untrained neural networks under dropout converge to Gaussian processes for fixed sets of weights and biases.
This paper provides a mathematical foundation for deep neural networks solving PDEs.
problem Mathematical foundation for deep neural networks solving high-dimensional PDEs.
method Decomposed generalization error into approximation and training errors; derived gradient flow in the wide network limit.
result Generalization error tends to zero as the number of neurons and training time tend to infinity.
Randomly initialized wide neural networks with zero-mean activations are nearly independent, potentially solving AI interpretability limits.
problem Measuring the limits of AI interpretability.
method Randomly initialized neural networks with large width and zero-mean activation functions.
result Neural networks with zero-mean activations are nearly independent, solving the computational no-coincidence conjecture.
Wide neural networks learn features under μP, identifying weights and decomposing support.
problem Feature learning in wide neural networks under μP. method Proving mean-field limit, characterizing identifiability, sparse-dictionary decomposition, and feature-learning-error decomposition.
result The triple (w∗,Dorb∗,S∗) identifies the natural learning cell of the architecture-data pair (σ,ρ). This paper explores loss landscapes of sparse neural networks, finding unique characteristics compared to dense networks.
problem Understanding the loss landscape of sparse neural networks, especially one-hidden-layer networks.
method Analyzes sparse networks with dense and sparse final layers, focusing on linear and non-linear models.
result Sparse networks can have no spurious valleys under certain conditions, but spurious valleys and minima can exist for wide sparse networks.
GNNs learn graph representations, with new theory on their power and limitations.
problem Understanding the capabilities and limitations of GNNs.
method Theoretical analysis of GNNs, focusing on approximation and learning properties.
result New insights into the representation, generalization, and extrapolation of GNNs.
We prove the precise scaling, at finite depth and width, for the mean and variance of the neural tangent kernel (NTK) in a randomly initialized ReLU network. The standard deviation is exponential in the ratio of network depth to width. Thus, even in the limit of infinite overparameterization, the NTK is not determinist…
Study reveals three limiting regimes for neural network functionals.
problem Understanding the behavior of functionals of random neural networks.
method Central and non-central limit theorems, Hermite expansions, Diagram Formula, Stein-Malliavin techniques.
result Three distinct limiting regimes based on fixed points of covariance function.
Stabilizes GAN training with limited data.
problem Overfitting in GANs with scarce data.
method Adaptive discriminator augmentation.
result Good results possible with few thousand images.
Deep Gaussian Processes are reinterpreted as deep trigonometric networks for tractable inference.
problem Challenging inference in DGPs due to intractable marginalization in latent function space.
method Viewing DGPs as deep trigonometric networks with Bochner's theorem, and using the wide limit with a bottleneck to translate DGPs into deep trigonometric networks.
result The weight space view yields the same effective covariance functions as obtained in function space, and varying prior distributions over network parameters is equivalent to employing different kernels.
We prove that a single-layer neural network trained with the Q-learning algorithm converges in distribution to a random ordinary differential equation as the size of the model and the number of training steps become large. Analysis of the limit differential equation shows that it has a unique stationary solution which …
Training of large-scale deep neural networks is often constrained by the available computational resources. We study the effect of limited precision data representation and computation on neural network training. Within the context of low-precision fixed-point computations, we observe the rounding scheme to play a cruc…
Deep neural networks with heavy-tailed weights converge to stable distributions.
problem Understanding the convergence of heavy-tailed weights in infinitely-wide neural networks.
method Analyzing infinitely-wide multi-layer perceptrons with i.i.d. symmetric α-stable weight distributions. result The vector of pre-activation values converges to i.i.d. symmetric α-stable distributions. Wide neural networks converge to Gaussian processes, improving generalization.
problem Understanding the generalization of wide neural networks, especially deep equilibrium models.
method Investigation of deep equilibrium models (DEQs) with infinite-depth layers, focusing on their convergence to Gaussian processes as width and depth approach infinity.
result Wide DEQs converge to Gaussian processes, maintaining generalization performance.
New method for Bayesian neural networks with unbounded weights.
problem Posterior inference for Bayesian neural networks with unbounded weights.
method Conditionally Gaussian representation for efficient posterior inference.
result Interpretable and computationally efficient procedure for posterior inference.
Analyzes dynamics of quantum neural networks, predicting exponential decay of training error.
problem Understanding convergence rate of quantum neural networks training.
method Analytic theory for gradient descent dynamics of wide quantum neural networks.
result Simple analytic formula predicts exponential decay of training error.
Pruning filters in neural networks improves performance and reduces costs.
problem Large neural networks have high inference costs and limited resource usage.
method Batch Bridgeout for efficient pruning of convolutional filters.
result Batch Bridgeout trained networks achieve higher accuracy across various pruning intensities.
Derives a family of hyperparameter scaling strategies for neural networks.
problem Optimizing hyperparameters for wide and deep neural networks.
method Introduces a one-parameter family of hyperparameter scaling strategies.
result Reveals proper scaling of depth with width for large-scale models.
Bayesian neural networks ignore data in infinite units limit.
problem Pathological behavior of posterior in over-parameterized networks.
method Mean-field variational inference in infinite hidden units limit.
result Posterior mean converges to zero, ignoring data.
Several recent trends in machine learning theory and practice, from the design of state-of-the-art Gaussian Process to the convergence analysis of deep neural nets (DNNs) under stochastic gradient descent (SGD), have found it fruitful to study wide random neural networks. Central to these approaches are certain scaling…
Wide neural networks can be closely approximated by Gaussian processes, with rates depending on the activation function's properties.
problem Approximating the behavior of wide neural networks using Gaussian processes.
method Established convergence rates for the central limit theorem in an infinite-dimensional functional space, using a transportation distance metric.
result Explicit convergence rates for neural networks approximated by Gaussian processes, varying based on the activation function's properties.
Motivation: Cell-biological processes are regulated through a complex network of interactions between genes and their products. The processes, their activating conditions, and the associated transcriptional responses are often unknown. Organism-wide modeling of network activation can reveal unique and shared mechanisms…
A well-conditioned Jacobian spectrum has a vital role in preventing exploding or vanishing gradients and speeding up learning of deep neural networks. Free probability theory helps us to understand and handle the Jacobian spectrum. We rigorously show almost sure asymptotic freeness of layer-wise Jacobians of deep neura…
The paper examines when NTK theory applies to real finite-width neural networks.
problem Understanding when NTK theory accurately predicts the behavior of finite-width neural networks.
method Empirical study of fully-connected ReLU and sigmoid DNNs with various hyperparameters and depths.
result NTK theory does not always apply to sufficiently deep networks with exploding gradients, and the kernel changes significantly during training.
Bayesian neural networks with dependent weights converge to Gaussian mixtures.
problem Limitations of standard Gaussian priors in neural networks.
method Posterior analysis with Gaussian likelihood for networks with dependent weights.
result Posterior distribution identified in the wide-width limit, ensuring invertibility of random covariance matrix.
The study investigates how data variability impacts the generalization of neural networks.
problem Understanding the impact of data variability on neural network generalization.
method Developed a field-theoretic formalism to compute generalization properties of neural networks, focusing on data variability.
result Data variability leads to non-Gaussian action, affecting the learning curve and generalization properties of neural networks.
Wide networks learn from adversarial perturbations effectively.
problem Understanding why adversarial examples deceive classifiers and transfer between models.
method Assumed wide two-layer networks, proved with theoretical analysis.
result Adversarial perturbations contain class-specific features for networks to generalize.
Bounds neural network output distribution to Gaussian for random initialization.
problem Quantifying the distribution of randomly initialized deep neural networks.
method Quantitative Gaussian approximation using quadratic Wasserstein distance.
result Explicit inequalities show how network sizes affect Gaussian behavior.
The paper studies deep neural networks with Gaussian weights and finds their asymptotic behavior.
problem Understanding the behavior of deep neural networks with large width.
method Function-space perspective, Gaussian process analysis, weak convergence in large-width limit.
result Deep neural networks with large width converge to a continuous Gaussian process.
A number of recent papers have provided evidence that practical design questions about neural networks may be tackled theoretically by studying the behavior of random networks. However, until now the tools available for analyzing random neural networks have been relatively ad-hoc. In this work, we show that the distrib…
Paper introduces a new edge exchangeable block model for complex networks.
problem Limitations of the stochastic block model in analyzing complex networks.
method Develops a Bayesian nonparametric edge exchangeable block model.
result The new model outperforms state-of-the-art SBMs for link prediction.