We study how finite Bayesian neural networks adapt their hidden representations.
problem Understanding how finite Bayesian neural networks differ from infinite ones.
method We analyze the asymptotics of learned feature kernels for various network architectures.
result The leading finite-width corrections to feature kernels have a universal form.
Researchers derive exact priors for finite Bayesian neural networks.
problem Understanding non-Gaussian priors in finite Bayesian neural networks.
method Analytical derivation of function space priors for finite fully-connected feedforward networks.
result Exact solutions for priors of finite networks, including Meijer G-function for linear networks and mixtures for ReLU networks.
Empirical study compares wide neural networks to kernel methods, resolving open questions.
problem Understanding the relationship between wide neural networks and kernel methods.
method Large-scale empirical study using various neural network architectures and kernel methods.
result Wide neural networks outperform fully-connected finite-width networks in some cases, but underperform convolutional finite-width networks.
Finite-precision learning of a n h anh anh networks is limited by the Monte Carlo rate.
problem Learning a n h anh anh neural networks under finite precision method Using iterated a n h anh anh activations to construct localized bump functions result No adaptive randomized algorithm can achieve higher convergence rate than Monte Carlo rate in finite precision
Study on hidden units in finite Bayesian neural networks and their tail properties.
problem Understanding the behavior of hidden units in finite Bayesian neural networks.
method Introduced a generalized Weibull-tail property to describe hidden units tails.
result Unit priors become heavier-tailed going deeper, providing insights into finite Bayesian neural networks.
Study of deep neural networks using finite-time Lyapunov exponents.
problem Understanding the geometric structures in input space formed by deep neural networks.
method Analogy with dynamical systems, computing finite-time Lyapunov exponents.
result Ridges of large positive exponents divide input space into regions associated with different classes.
Finite singular times for symmetric network curvature flow.
problem Formation of singularities in network curvature flow.
method Curvature flow of networks with symmetric initial data and two triple junctions.
result The set of singular times is finite.
This work uses sampling theory to analyze smoothness and error bounds of finite neural networks.
problem Analyzing the function space of finite neural networks and providing error bounds.
method Applying sampling theory to finite neural networks with non-expansive activation functions, considering both deterministic and random sampling.
result Novel error bounds for univariate neural networks under band-limited input assumption, highlighting the advantage of deterministic uniform sampling.
Study on fluctuations in neural network kernels and predictions, focusing on finite width effects.
problem Characterizing fluctuations in finite width neural networks.
method Dynamical mean field theory analysis of wide but finite feature learning neural networks.
result Fluctuations in kernels and predictions are dynamically coupled, leading to reduced variance in feature learning regimes.
New method identifies parameters of wider shallow neural networks with biases.
problem Identifying parameters of wide shallow neural networks with biases from finite samples.
method Two-step pipeline: direction of weights via second order information, signs via algebraic evaluations, biases via gradient descent.
result Constructive methods and theoretical guarantees of finite sample identification for wider shallow networks with biases.
New method reduces over-parametrization in neural networks, ensuring sparsity and finite network size.
problem Over-parametrization leads to too many active neurons in neural networks, especially with large data.
method Investigates a nonconvex regularization method for shallow ReLU networks.
result Locally optimal networks are finite even with infinite data, maintaining approximation guarantees and network size bounds.
Analyzes DNNs trained with noisy gradients, finding FWCs negligible for large n.
problem Analyzing DNNs trained with noisy gradients.
method Introduced analytical framework to analyze non-Gaussian stochastic process.
result FWCs negligible for large n, improving CNN performance.
We show that finite-width deep ReLU neural networks yield rate-distortion optimal approximation (Bölcskei et al., 2018) of polynomials, windowed sinusoidal functions, one-dimensional oscillatory textures, and the Weierstrass function, a fractal function which is continuous but nowhere differentiable. Together with thei…
The paper analyzes deep neural networks using control theory to set a time limit for their convergence.
problem Understanding the finite-time convergence of deep neural networks.
method Lyapunov based analysis of the loss function, control theory framework, finite-time control of non-linear systems.
result A priori guarantees of finite-time convergence for deep neural networks are provided.
Empirical study compares finite- and infinite-width BNNs, revealing performance differences under model mismatch.
problem Comparing BNNs with different widths due to conflicting model properties and inference intractability.
method Empirical comparison of finite- and infinite-width BNNs, analyzing performance under model mismatch.
result Increasing width can hurt BNN performance when the model is mis-specified, and finite-width BNNs generalize better under model mismatch.
Taylorized training improves neural network training at finite width.
problem Understanding and improving neural network training at finite width.
method Training the k-th order Taylor expansion of the neural network at initialization.
result Taylorized training agrees with full neural network training better as k increases and can significantly close the performance gap.
Generalizes neural networks for infinite-dimensional mappings, including PDE solutions.
problem Learning mappings between infinite-dimensional spaces and finite-dimensional approximations.
method Graph kernel network architecture with message passing for kernel integration.
result Competitive performance compared to state-of-the-art solvers for PDEs.
New findings show modern neural networks have finite sample complexity in o-minimal structures.
problem Understanding the learnability of modern neural networks in a broad context.
method Analyzing feedforward neural networks definable in o-minimal structures.
result Modern neural networks, including MLPs, CNNs, GNNs, and transformers, have finite sample complexity in the agnostic PAC setting.
Anisotropic curvature flow of networks shows unique solutions and behavior under finite time.
problem Existence and behavior of networks under anisotropic curvature flow.
method Existence, uniqueness, and regularity of maximal geometric solutions proven.
result Existence of maximal geometric solutions and behavior under finite time.
Study examines dependence properties of Bayesian neural network units in finite-width networks.
problem Understanding dependence properties of hidden units in practical finite-width Bayesian neural networks.
method Theoretical analysis and empirical evaluation of depth and width impacts.
result Hidden units in finite-width Bayesian neural networks are dependent, contrary to the infinite-width limit assumption.
Theory captures feature learning effects in finite CNNs.
problem Feature learning in finite deep neural networks.
method Derive a self-consistent Gaussian Process theory.
result Good agreement with experiments and sharp transition between regimes.
Study Gaussian-process limits of neural networks using tensor programs.
problem Understanding the behavior of neural networks as they approach infinite width.
method Quantitative analysis through tensor programs and Wasserstein distance.
result Explicit finite-width error bounds, showing convergence to Gaussian-process limits.
The paper examines when NTK theory applies to real finite-width neural networks.
problem Understanding when NTK theory accurately predicts the behavior of finite-width neural networks.
method Empirical study of fully-connected ReLU and sigmoid DNNs with various hyperparameters and depths.
result NTK theory does not always apply to sufficiently deep networks with exploding gradients, and the kernel changes significantly during training.
Improved bounds for neural network approximations of functions.
problem Bounding the width of neural networks for function approximation.
method Extending Radon-based norms to bounded open sets and deriving new approximation bounds.
result Improved sparse approximation bounds for neural networks.
Paper analyzes Q-learning with neural networks, proving a fast convergence rate.
problem Analyzing the convergence rate of neural Q-learning.
method Finite-time analysis of neural Q-learning with a deep ReLU network.
result Neural Q-learning converges to optimal policy with O ( 1 / T ) O(1/\sqrt{T}) O ( 1/ T ) rate. This work approximates finite neural networks with Gaussian processes, providing error bounds and applications in prior selection.
problem Approximating finite neural networks with Gaussian processes for error bounds and uncertainty quantification.
method Iterative approximation of neural network layers as mixtures of Gaussian processes, using optimal transport and Gaussian processes.
result The ability to return a mixture of Gaussian processes that is ε-close to the neural network at a finite set of input points.
FDNet learns PDEs from data with fast predictions.
problem Discovering complex systems behavior from data.
method Finite difference neural networks (FDNet) to learn PDEs from trajectory data.
result FDNet predicts future behavior with few trainable parameters.
Novel algorithms improve efficiency of finite width NTK computation.
problem Efficiency of computing finite width Neural Tangent Kernel (NTK).
method Leveraging neural network structure, propose two novel algorithms.
result Significantly improved efficiency in compute and memory requirements.
Investigates the impact of finite VC dimension on neural network approximation and learning.
problem The influence of VC dimension on neural network approximation and learning from samples.
method Analysis of high-dimensional geometry and statistical learning theory, focusing on VC dimension.
result Finite VC dimension is beneficial for uniform convergence of empirical errors but not for approximation of functions from a probability distribution.
Study minimal networks on spheres and balls near standard metrics.
problem Existence of minimal networks in spheres and balls with metrics close to standard.
method Finite-dimensional reduction method, inspired by configuration of networks and triods.
result Existence of minimal networks in spheres and balls for metrics close to standard.
Wide CNNs outperform infinite width networks, revealing scaling laws.
problem Understanding the performance difference between finite and infinite width convolutional networks.
method Diagrammatic approach to derive asymptotic width dependence for various quantities.
result The difference in performance between finite and infinite width models vanishes at a definite rate with respect to model width.
Study Gaussian approximation for deep neural networks with random weights.
problem Understanding the distribution of deep neural networks with random weights.
method Established Gaussian approximation bounds in Wasserstein-1 norm.
result Convergence rates of order n − ( 1 / 6 ) L − 1 + ε n^{-({1}/{6})^{L-1} + ε} n − ( 1 / 6 ) L − 1 + ε for deep networks with proportional layer widths. Field theory explains optimal scaling in ResNets for signal propagation.
problem Understanding optimal scaling parameter for ResNet performance.
method Finite-size field theory for ResNets to study signal propagation and scaling.
result Analytical expressions for optimal scaling parameter, independent of other hyperparameters.
Empirical study shows standard CNNs deviate from NTK predictions.
problem Understanding how standard finite-width CNNs behave compared to their infinite-width NTK counterparts.
method Empirical analysis of AlexNet and LeNet architectures.
result Standard CNNs deviate significantly from their NTK counterparts, but deviation decreases with wider networks.
Study on how noise and variation-norm regularisation help shallow ReLU networks use fewer neurons.
problem Understanding how shallow ReLU networks use a finite number of neurons in the infinitely wide limit.
method Analysis of two regularisation strategies: noise injection and variation-norm.
result Both regularisation methods minimize functions with a finite number of neurons, regardless of overparametrisation.
We analyze double descent in finite-width neural networks using influence functions.
problem Understanding double descent in finite-width neural networks.
method Using influence functions to derive population loss bounds and investigate loss function effects.
result Derived bounds exhibit double descent behavior at the interpolation threshold.
Two neural network methods solve the master equation for MFGs.
problem Approximating Nash equilibria in stochastic, finite-agent games.
method Backward induction and direct PDE tackling neural networks.
result Neural networks can approximate the master equation's solution.
Bayesian inference for wide neural networks using Edgeworth expansion.
problem Analyzing the non-Gaussian behavior of wide neural networks in Bayesian inference.
method Proposed a non-Gaussian distribution using multivariate Edgeworth expansion for finite-width neural networks.
result Derived non-Gaussian posterior distribution in Bayesian regression tasks.
The paper provides non-asymptotic Edgeworth expansions for neural network outputs.
problem Approximating deviations of finite-width neural networks from their Gaussian limit.
method Multidimensional Edgeworth expansions of arbitrary order for neural network outputs.
result Established a bound on the total variation distance between neural network output and its Edgeworth approximation.
Quantitative CLTs show neural network distributions converge to Gaussian as width increases.
problem Understanding the distribution of fully connected neural networks with random weights and biases.
method Analyzing the distribution of a fully connected neural network with random Gaussian weights and biases, proving quantitative bounds on normal approximations.
result The distance between a random fully connected network and the corresponding infinite width Gaussian process scales like n − γ n^{-γ} n − γ for γ > 0 γ>0 γ > 0 . Novel analysis of EFP for finite-sum problems in neural networks.
problem Optimization of two-layer neural networks in the mean-field regime.
method Primal-dual analysis of entropic fictitious play (EFP) for finite-sum problems.
result Established global convergence guarantees for EFP dynamics.
Analyzes symmetries in neural networks to predict learning dynamics.
problem Understanding the dynamics of neural network parameters during training.
method Unified theoretical framework based on symmetries and conservation laws.
result Symmetries impose geometric constraints on gradients and Hessians, leading to conservation laws.
Wide networks with polynomial activations have proven asymptotic behavior.
problem Understanding the behavior of neural networks in the large width limit.
method Proving a conjecture for deep networks with polynomial activation functions.
result Tight bounds on the behavior of wide networks during stochastic gradient descent and derivation of their finite-width dynamics.
Recurrent neural networks trained on regular languages exhibit stable states that can recover from noise.
problem Stability of internal states in recurrent neural networks trained on regular languages.
method Empirical study with analysis of network activation and transitions between states.
result Recurrent neural networks trained on regular languages can recover from random perturbations and maintain stable states.
Develops a fast, accurate method for comparing networks.
problem Comparing networks with repeated observations and varying sizes/sparsity.
method A novel two-sample hypothesis testing method with theoretical guarantees.
result Outperforms existing tools in speed and accuracy, power-optimal.
Renormalization in neural networks linked to quantum field theory.
problem Implementing renormalization in neural networks.
method Mapping neural networks to quantum field theory, applying renormalization techniques.
result Changing weight standard deviation corresponds to a renormalization flow.
Our work investigates how to identify privacy violations in models using finite adversaries.
problem Identifying privacy violations in models with limited adversary capabilities.
method Investigates requirements for finite adversaries to identify privacy violations.
result Parameters quantify the capabilities of finite adversaries.
Finite-time queue peaks in stochastic networks have logarithmic scaling after geometric thresholds.
problem Queue peak laws in stochastic networks with geometric thresholds.
method Self-normalization mechanism
result Logarithmic scaling of queue peaks after geometric thresholds.