This note explains when neural networks can be seen as Gaussian processes.
problem Understanding the relationship between neural networks and Gaussian processes.
method Formulating a Gaussian process regression based on neural network outputs and analyzing the resulting posterior mean functions.
result The posterior mean functions of neural networks follow a Gaussian process in certain cases, providing an interpretation of reproducing kernel Hilbert spaces.
Bounds neural network output distribution to Gaussian for random initialization.
problem Quantifying the distribution of randomly initialized deep neural networks.
method Quantitative Gaussian approximation using quadratic Wasserstein distance.
result Explicit inequalities show how network sizes affect Gaussian behavior.
Study challenges the Gaussian pre-activations assumption in neural networks.
problem Challenges the assumption that pre-activations are Gaussian in neural networks.
method Constructs pairs of activation functions and initialization distributions to ensure Gaussian pre-activations.
result Discovered constraints for ensuring Gaussian pre-activations in neural networks.
Bayesian inference for wide neural networks using Edgeworth expansion.
problem Analyzing the non-Gaussian behavior of wide neural networks in Bayesian inference.
method Proposed a non-Gaussian distribution using multivariate Edgeworth expansion for finite-width neural networks.
result Derived non-Gaussian posterior distribution in Bayesian regression tasks.
The paper studies deep neural networks with Gaussian weights and finds their asymptotic behavior.
problem Understanding the behavior of deep neural networks with large width.
method Function-space perspective, Gaussian process analysis, weak convergence in large-width limit.
result Deep neural networks with large width converge to a continuous Gaussian process.
Deep neural networks converge to Gaussian mixtures as layer width increases.
problem Understanding the distribution of outputs from deep neural networks.
method Proof and experiments with a simple model showing the convergence of neural network outputs to Gaussian mixtures.
result Neural networks converge to Gaussian mixtures as the width of the last hidden layer increases.
We propose a simple method that combines neural networks and Gaussian processes. The proposed method can estimate the uncertainty of outputs and flexibly adjust target functions where training data exist, which are advantages of Gaussian processes. The proposed method can also achieve high generalization performance fo…
Deep neural networks' infinite-width behavior approximated by Gaussian models.
problem Understanding the behavior of deep neural networks in the limit of infinite width.
method Using the Lindeberg exchange principle to approximate weights by Gaussian random variables.
result Quantitative bounds on the 2-Wasserstein distance between deep neural networks and Gaussian limits.
Wide deep neural networks with Gaussian weights approximate Gaussian processes closely.
problem Understanding the approximation of deep neural networks with Gaussian weights to Gaussian processes.
method Established novel rates for the Gaussian approximation of random deep neural networks with Gaussian parameters and Lipschitz activation functions in the wide limit.
result The distance between the network output and the Gaussian approximation scales inversely with the width of the network.
Deep neural networks and Gaussian processes are shown to be equivalent through activation functions.
problem Understanding the relationship between neural networks and Gaussian processes.
method Developing an equivalence theory based on activation functions and kernels.
result Models can be seen as neural networks with improved uncertainty prediction or deep Gaussian processes with increased accuracy.
This work approximates finite neural networks with Gaussian processes, providing error bounds and applications in prior selection.
problem Approximating finite neural networks with Gaussian processes for error bounds and uncertainty quantification.
method Iterative approximation of neural network layers as mixtures of Gaussian processes, using optimal transport and Gaussian processes.
result The ability to return a mixture of Gaussian processes that is ε-close to the neural network at a finite set of input points.
Proposes TAGI for efficient Gaussian inference in Bayesian neural networks.
problem Efficient inference in Bayesian neural networks with complex architectures.
method Analytical method for tractable approximate Gaussian inference (TAGI).
result Matches performance of gradient-based methods with O ( n ) \mathcal{O}(n) O ( n ) computational complexity. Study of deep linear neural networks with proportional width and depth.
problem Lack of descriptive power in Gaussian limit of deep linear neural networks.
method Proportional infinite-width infinite-depth limit for deep linear neural networks.
result Characterization of limiting distribution as a nontrivial mixture of Gaussians.
New analysis shows a gap between Gaussian RKHS and neural networks on unbounded domains.
problem Understanding the function space bias of neural networks compared to Gaussian RKHS.
method Infinite-center asymptotic analysis of neural network Banach space and Gaussian RKHS on unbounded domains.
result Certain functions in Gaussian RKHS have infinite norm in neural network Banach space on unbounded domains.
Wide neural networks converge to Gaussian processes, improving generalization.
problem Understanding the generalization of wide neural networks, especially deep equilibrium models.
method Investigation of deep equilibrium models (DEQs) with infinite-depth layers, focusing on their convergence to Gaussian processes as width and depth approach infinity.
result Wide DEQs converge to Gaussian processes, maintaining generalization performance.
Study deep maxout networks and their equivalence to Gaussian processes.
problem Understanding neural networks with infinite width.
method Derive equivalence between deep maxout networks and Gaussian processes, characterize maxout kernel, and provide efficient numerical implementation.
result Bayesian inference based on deep maxout network kernel leads to competitive results compared to finite-width counterparts and deep neural network kernels.
Study on MC dropout in wide neural networks and its convergence to Gaussian processes.
problem Understanding the behavior of Monte Carlo dropout in wide neural networks.
method Rigorously studied the limiting distribution of wide untrained NNs under dropout, proving convergence to Gaussian processes. Investigated correlations and non-Gaussian behavior in finite width NNs.
result Wide untrained neural networks under dropout converge to Gaussian processes for fixed sets of weights and biases.
New method for Bayesian neural networks with unbounded weights.
problem Posterior inference for Bayesian neural networks with unbounded weights.
method Conditionally Gaussian representation for efficient posterior inference.
result Interpretable and computationally efficient procedure for posterior inference.
Bayesian neural networks use ridgelet prior for uncertainty quantification.
problem Combining strong predictive performance with uncertainty quantification in Bayesian neural networks.
method Proposes a ridgelet prior that approximates a Gaussian process covariance function in the output space of the network.
result Establishes universality property allowing Bayesian neural networks to approximate any Gaussian process.
The paper provides non-asymptotic Edgeworth expansions for neural network outputs.
problem Approximating deviations of finite-width neural networks from their Gaussian limit.
method Multidimensional Edgeworth expansions of arbitrary order for neural network outputs.
result Established a bound on the total variation distance between neural network output and its Edgeworth approximation.
Study shows neural networks trained with GD converge to Gaussian processes with polynomial decay.
problem Understanding convergence of neural networks to Gaussian processes during training.
method Explicit upper bounds on quadratic Wasserstein distance between trained networks and Gaussian approximations.
result Polynomial decay of approximation error with network width and training time.
New neural network approach for optimizing latent variable models.
problem Stability issues in marginalizing Gaussian Bayesian networks.
method Developed a new graphical structure and a neural network algorithm.
result Established a duality between parameter optimization and neural network training.
Study Gaussian-process limits of neural networks using tensor programs.
problem Understanding the behavior of neural networks as they approach infinite width.
method Quantitative analysis through tensor programs and Wasserstein distance.
result Explicit finite-width error bounds, showing convergence to Gaussian-process limits.
Gaussian process models simplify neural network behavior for easier understanding.
problem Understanding and predicting the behavior of deep learning systems.
method Constructing surrogate models using Gaussian processes from finite neural networks.
result Surrogate models capture phenomena like spectral bias and predict generalization well.
Deterministic method for certifying neural network robustness.
problem Certifying neural network robustness against adversarial attacks.
method Equivalence between training and Gaussian averaging for robustness certification.
result Comparable certified accuracy and robustness to stochastic methods but with single model evaluation.
SFSVI uses Gaussian mixtures to approximate neural network outputs for continual learning.
problem Learning new tasks without forgetting old ones in neural networks.
method Sequential function-space variational inference with Gaussian mixture approximation.
result Gaussian mixture SFSVI outperforms other methods in continual learning.
We establish large deviation principles for convolutional neural networks.
problem Understanding the behavior of convolutional neural networks in the infinite-channel limit.
method We establish large deviation principles for convolutional neural networks under Gaussian prior and posterior distributions.
result We provide a large deviation principle for the sequence of conditional covariance matrices and the posterior distribution.
The abstract proposes a neural network theory using quantum field theory.
problem Understanding the behavior of neural networks in the asymptotic and non-asymptotic limits.
method Mapping neural networks to Wilsonian effective field theory, using Gaussian processes and Feynman diagrams.
result Established a direct connection between overparameterization and simplicity of neural network likelihoods.
Gaussian processes are ubiquitous in nature and engineering. A case in point is a class of neural networks in the infinite-width limit, whose priors correspond to Gaussian processes. Here we perturbatively extend this correspondence to finite-width neural networks, yielding non-Gaussian processes as priors. The methodo…
Wide neural networks can degrade performance, contrary to conventional wisdom.
problem Understanding the limitations of increasing network width in neural networks.
method Using Deep Gaussian Processes to decouple capacity and width, analyzing their effects on representational power and non-Gaussianity.
result Wide neural networks can become less adaptable and more Gaussian, leading to performance degradation.
This paper shows how infinitely wide Tensor Networks converge to Gaussian Processes.
problem Understanding the relationship between Tensor Networks and Gaussian Processes.
method Analyzing the infinite-width limit of Tensor Networks and comparing them to Gaussian Processes.
result Infinitely wide Tensor Networks converge to Gaussian Processes, proving their equivalence.
Study Gaussian approximation for deep neural networks with random weights.
problem Understanding the distribution of deep neural networks with random weights.
method Established Gaussian approximation bounds in Wasserstein-1 norm.
result Convergence rates of order n − ( 1 / 6 ) L − 1 + ε n^{-({1}/{6})^{L-1} + ε} n − ( 1 / 6 ) L − 1 + ε for deep networks with proportional layer widths. uGMM-NN integrates probabilistic reasoning into neural networks.
problem Capturing multimodality and uncertainty in neural network activations.
method Parameterizes activations as univariate Gaussian mixtures with learnable parameters.
result Competitive discriminative performance with probabilistic activations.
Researchers derive exact priors for finite Bayesian neural networks.
problem Understanding non-Gaussian priors in finite Bayesian neural networks.
method Analytical derivation of function space priors for finite fully-connected feedforward networks.
result Exact solutions for priors of finite networks, including Meijer G-function for linear networks and mixtures for ReLU networks.
Wide neural networks can be closely approximated by Gaussian processes, with rates depending on the activation function's properties.
problem Approximating the behavior of wide neural networks using Gaussian processes.
method Established convergence rates for the central limit theorem in an infinite-dimensional functional space, using a transportation distance metric.
result Explicit convergence rates for neural networks approximated by Gaussian processes, varying based on the activation function's properties.
Study how depth affects inference in deep Bayesian neural networks.
problem Understanding how depth impacts inference in overparameterized linear Bayesian neural networks.
method Interpreting finite deep linear Bayesian neural networks as scale mixtures of Gaussian process predictors.
result Advances analytical understanding of how depth affects inference in a simple class of Bayesian neural networks.
Large deviation principle for deep neural networks with ReLU activation.
problem Understanding the behavior of deep neural networks with ReLU activation.
method Proving a large deviation principle for networks with Gaussian weights and ReLU activation functions.
result Simplified expressions and power-series expansions for the ReLU case.
Proposes scale mixture of NNGPs for more flexible stochastic processes.
problem Limited focus on broadening the class of stochastic processes from NNGPs.
method Scale mixture of NNGPs with scale priors on last-layer parameters.
result Turns neural networks into a richer class of stochastic processes.
TDistNNs improve prediction intervals for neural networks by using t-distributions.
problem Traditional neural networks provide only point estimates, lacking predictive uncertainty.
method TDistNNs generate t-distributed outputs with adjustable degrees of freedom, enhancing robustness to non-Gaussian data.
result TDistNNs produce narrower prediction intervals with proper coverage compared to Gaussian-based PNNs.
The paper develops generalization bounds for deep compound Gaussian neural networks.
problem Developing theoretical guarantees for the performance of deep neural networks.
method Novel generalization error bounds using a compound Gaussian prior and Dudley's integral.
result Theoretical bounds show generalization error scales O ( n ln ( n ) ) \mathcal{O}(n\sqrt{\ln(n)}) O ( n ln ( n ) ) in signal dimension and O ( ( N e t w o r k S i z e ) 3 / 2 ) \mathcal{O}((Network Size)^{3/2}) O (( N e tw or k S i z e ) 3/2 ) in network size. Quantitative CLTs show neural network distributions converge to Gaussian as width increases.
problem Understanding the distribution of fully connected neural networks with random weights and biases.
method Analyzing the distribution of a fully connected neural network with random Gaussian weights and biases, proving quantitative bounds on normal approximations.
result The distance between a random fully connected network and the corresponding infinite width Gaussian process scales like n − γ n^{-γ} n − γ for γ > 0 γ>0 γ > 0 . Random neural networks with ReLU activations are non-Gaussian processes.
problem Understanding the behavior of neural networks with random initialization and rectified linear units.
method Proving these networks are non-Gaussian processes and deriving their properties.
result These networks can converge to non-Gaussian processes under certain conditions.
Hybrid Bayesian neural networks use function uncertainty for probabilistic inference.
problem Uncertainty in neural network weights is hard to specify and interpret.
method Integrates probabilistic layers with standard deterministic layers for function uncertainty.
result Improves probabilistic inference by encoding function uncertainty.
Gradient descent memorizes many Gaussians efficiently.
problem Memorizing many Gaussians with minimal parameters.
method Gradient descent on a depth-two neural network.
result One step of gradient descent memorizes $Ω\left(\frac{dq}{\log^4(d)}
ight)$ Gaussians.
Deep learning models converge to Gaussian dynamics with mixed structured inputs.
problem Understanding neural network dynamics with complex input distributions.
method Extended hidden manifold model to Gaussian mixtures, analyzed via SGD.
result Learning dynamics with mixed inputs converge to Gaussian behavior.
Neural networks outperform kernel methods in classifying high-dimensional Gaussian mixtures.
problem Classifying high-dimensional Gaussian mixtures using kernel methods and neural networks.
method Theoretical analysis and derivation of learning dynamics for 2LNN and comparison with kernel methods.
result 2LNN can achieve near-optimal performance on high-dimensional Gaussian mixture classification tasks, surpassing kernel methods.
Paper connects neural networks to Gaussian processes for understanding double-descent.
problem Understanding the double-descent phenomenon in neural networks.
method Uses techniques from random matrix theory and Gaussian processes.
result Establishes a connection between NNGP and random matrix theory for neural networks.
Bounds on Gaussian approximation for neural networks with novel smoothing techniques.
problem Approximating the distribution of wide random neural networks.
method Stein's method, Gaussian smoothing, Laplacian operators, Cameron-Martin space.
result First bounds on Gaussian approximation of wide random neural networks.