Improved sample complexity bounds for neural networks with depth independence.
problem Understanding the sample complexity of neural networks with depth and size independence.
method New bounds on Rademacher complexity with norm constraints on parameter matrices.
result Improved sample complexity bounds that are fully independent of network size under certain assumptions.
Proposes a new method to estimate Bayesian neural network depth.
problem Estimating the depth of Bayesian neural networks.
method Uses a discrete truncated normal distribution to learn depth mean and variance, inferring posterior distributions by minimizing variational free energy.
result Improves test accuracy and reduces posterior depth variance on the spiral dataset.
A new data-level recombination strategy improves RGB-D salient object detection.
problem RGB-D salient object detection struggles with depth information.
method Proposes a novel data-level recombination strategy to fuse RGB and depth data before feature extraction.
result Achieves a new state-of-the-art performance in RGB-D salient object detection.
The paper proves barriers to approximating functions with small weights and depth in neural networks.
problem Proving barriers to approximating functions with constant depth neural networks.
method Reduction to open problems and natural-proof barriers in circuit complexity, and a new approach to polynomially-bounded functions.
result There are fundamental barriers to proving results beyond depth 4 for constant-depth neural networks.
Improved sample complexity for ReLU networks with norm constraints.
problem Estimating sample complexity for ReLU networks under norm constraints.
method Refined Rademacher complexity analysis for function class.
result Often no explicit depth-dependence in sample complexity bound.
Proves depth 2 neural networks can't approximate certain functions as well as depth 3 networks.
problem Approximating functions with depth 2 networks in high dimensions.
method Lower bound proof using worst-to-average-case random self-reducibility.
result Proves depth 2 networks can't approximate certain functions as well as depth 3 networks, resolving an open problem.
ResNets and DenseNets converge to NTK with depth and width, offering advantages for kernel regression.
problem Understanding convergence of ResNets and DenseNets to Neural Tangent Kernel (NTK).
method Analysis of finite width and depth corrections for NTK of ResNets and DenseNets.
result ResNets and DenseNets can converge to NTK with depth and width, unlike vanilla networks.
Deep networks with orthogonal weights show stable fluctuations, improving generalization and training speed.
problem Fluctuations in deep networks with Gaussian weights can impair training, especially in networks with depth comparable to width.
method Analytical and numerical studies of fully-connected networks with orthogonal weight initialization and tanh activations.
result Rectangular networks with orthogonal weights have stable fluctuations independent of network depth, leading to better generalization and training speed.
Study on learning rates in neural networks of varying depth.
problem Dependence of maximal update learning rate on network depth.
method Analysis of random fully connected ReLU networks with mean-field weight initialization.
result Maximal update learning rate scales like L−3/2 with network depth. New findings show depth is more important than width in neural networks.
problem Understanding the role of width and depth in neural networks.
method Constructed networks with bounded weights and width at most d+2, showing depth plays a more significant role.
result Depth is more important than width in the expressive power of neural networks.
Normalizing flows are shown to be equivalent to Bayesian networks, revealing new insights.
problem Understanding the limitations and capabilities of normalizing flows.
method Revisiting normalizing flows as probabilistic graphical models and analyzing their structure.
result Normalizing flows can be reduced to Bayesian networks, revealing new insights into their structure and capabilities.
A new depth measure captures anomalies in time series data.
problem Anomaly detection in time series data.
method Convex hull area of sampled curves for functional depth.
result Captures gradual departures from centrality.
Adversarial examples in deep ReLU networks with constant depth.
problem Adversarial perturbations lead to large output changes in deep ReLU networks.
method Analyzes the phenomenon in networks with independent Gaussian parameters and constant depth.
result Adversarial examples arise due to functions being close to linear.
Study on size and depth of neural networks for approximating benign functions, showing barriers and explicit results.
problem Understanding how size and depth of neural networks affect their ability to approximate benign functions.
method Analyzing ReLU networks for benign functions, proving barriers and explicit results.
result Explicit benign functions that cannot be approximated by networks of certain sizes or depths, showing barriers to size and depth separation.
Uniform scaling limits in AdamW-trained transformers converge to ODEs.
problem Understanding the dynamics of large-depth transformers trained with AdamW.
method Modeling transformer dynamics as an interacting particle system coupled through attention, proving convergence to ODEs.
result The joint dynamics of hidden states and backpropagated variables converge uniformly to an ODE system.
Study reveals how Fisher information changes with network depth, finding it grows linearly.
problem Understanding the trainability of deep neural networks (DNNs).
method Investigates the spectral distribution of the conditional Fisher information matrix (FIM) for fully-connected networks achieving dynamical isometry.
result The conditional FIM's spectrum concentrates around the maximum and grows linearly with depth.
This paper improves neural network approximation for analytic functions with adjustable depth and width.
problem Approximating analytic functions using neural networks with depth and width parameters.
method Characterizes approximation rates as a joint function of width (N) and depth (L) for ReLU networks.
result Establishes upper bounds for analytic function approximation rates of O(N^(-CL^τ)) with τ influenced by N and L.
We consider rotationally symmetric spaces with low regularity, which we regard as integral currents spaces or manifolds with Sobolev regularity and are assumed to have nonnegative scalar curvature. Relying on the flat distance and on Sobolev norms, we establish several nonlinear stability estimates about the ``distance…
This paper analyzes convergence rates of neural networks in the deep learning regime.
problem Understanding convergence rates of neural networks in the deep learning regime.
method Analyzing the Neural Tangent Kernel (NTK) convergence rates in the large depth limit.
result Quantifies the impact of initialization and activation function on NTK convergence rates.
Independent component analysis (ICA) has been widely used for blind source separation in many fields such as brain imaging analysis, signal processing and telecommunication. Many statistical techniques based on M-estimates have been proposed for estimating the mixing matrix. Recently, several nonparametric methods have…
Wide CNNs with shared weights and max pooling have linearly independent features and can achieve zero training error.
problem Understanding the optimization landscape and expressiveness of deep CNNs.
method Analysis of loss landscape and expressiveness of practical deep CNNs with shared weights and max pooling layers.
result Wide CNNs can achieve zero training error and have a well-behaved loss surface with almost no bad local minima.
Mean field theory explains gradient backpropagation in deep dropout networks.
problem Understanding gradient backpropagation in deep dropout networks.
method Applied mean field theory to dropout networks, considering realistic training conditions.
result Gradient backpropagation length is limited by depth scales, not just independence assumption.
Algorithm efficiently learns deep ReLU networks with polynomial runtime in depth and parameters.
problem Learning deep ReLU networks with polynomial runtime.
method Algorithm using filtered PCA and lattice polynomial analysis.
result First nontrivial results for networks of depth more than two with polynomial runtime.
This work challenges the Neural Tangent Kernel's role in overparameterized neural networks, especially with large width and depth.
problem The Neural Tangent Kernel's behavior in overparameterized neural networks with large width and depth is unclear.
method Experimental and theoretical analysis of ReLU networks with large width and depth.
result The aggregate norm of hidden neuron deviations does not vanish in infinitely-wide ReLU networks, indicating non-trivial behavior.
Path-independent equilibrium models improve network performance on harder problems.
problem Improving network performance on harder problem instances.
method Investigated path-independent equilibrium models and their impact on network performance.
result Path independence correlates with better performance on harder problem instances.
Feedback alignment methods need to be evaluated for accuracy and gradient cosine similarity.
problem Evaluating feedback alignment methods
method Proposed diagnostic evaluation protocol
result Identified silent failures in standard reporting pair
Despite the widespread practical success of deep learning methods, our theoretical understanding of the dynamics of learning in deep neural networks remains quite sparse. We attempt to bridge the gap between the theory and practice of deep learning by systematically analyzing learning dynamics for the restricted case o…
DiffEnc improves diffusion models by adding flexibility and achieving better likelihood on CIFAR-10.
problem Improving the likelihood of diffusion models on image datasets.
method Introducing a data- and depth-dependent mean function and a free weight parameter for noise variance.
result Achieved statistically significant improvement in likelihood on CIFAR-10.
Gradient descent memorizes many Gaussians efficiently.
problem Memorizing many Gaussians with minimal parameters.
method Gradient descent on a depth-two neural network.
result One step of gradient descent memorizes $Ω\left(\frac{dq}{\log^4(d)}
ight)$ Gaussians.
Optimizes synthetic image augmentation for sim2real policy transfer in robotics.
problem Difficulty in transferring learned policies from simulated to real environments.
method Optimizes random transformations to augment synthetic images, enabling policy learning without real data.
result Significant improvement in policy accuracy on real robots for three manipulation tasks.
This paper controls the capacity of weight-normalized deep neural networks using rectified linear units.
problem Capacity control of weight-normalized deep neural networks.
method Establishes upper bounds on Rademacher complexities and analyzes approximation properties of Lp,q weight normalized networks. result For L1,∞ weight normalized networks, the approximation error is controlled by the L1 norm of the output layer, and generalization error depends on the square root of depth. New algorithm speeds up sampling from logconcave densities.
problem Sampling from logconcave functions in statistics and ML.
method Solves ODEs to improve HMC and other sampling methods.
result Nearly linear runtime for polylogarithmic depth.
This paper simplifies deep learning networks by mapping them to a linear function of a feature map.
problem Understanding how weights in deep networks coordinate across layers and generalize.
method Reparameterizes DNNs as a linear function of a feature map, transforming depth-dependencies into tensor products.
result Develops sample compression representation of neural networks in terms of support vectors, showing sample complexity of O(ns/epsilon).
Quantum circuits are hard to learn on average.
problem Learning the output distributions of quantum circuits is hard.
method Statistical query model analysis.
result Learning quantum circuits requires exponentially many queries.
A new depth measure for non-convex data supports, faster than halfspace depth.
problem Non-convex data supports in multivariate statistics.
method Extending halfspace depth to Reproducing Kernel Hilbert Space (RKHS).
result The new depth measure is consistent and can be computed faster.
The paper studies randomized approximations of Tukey's depth for log-concave isotropic data.
problem The challenge of approximating Tukey's depth in high dimensions.
method The study examines randomized algorithms for approximating Tukey's depth for log-concave isotropic data.
result Randomized algorithms correctly approximate maximal depth and close to zero depths but not intermediate depths.
This paper connects functional data analysis with machine learning techniques.
problem Lack of theoretical analysis for functional depths.
method Viewing functional depths as kernel mean embeddings in machine learning.
result Facilitates answers to open questions about functional depths.
Self-attention models benefit equally from width and depth, but beyond a certain point, depth becomes less efficient.
problem Understanding the optimal balance between depth and width in self-attention models.
method Theoretical predictions and empirical ablations on networks of varying depths and widths.
result An optimal width of 30K is recommended for a 1-Trillion parameter network, marking a significant width for self-attention models.
Study on deep multi-head self-attention dynamics, proving homogenized limits under specific scalings.
problem Understanding the behavior of deep multi-head self-attention models as depth increases.
method Random model of deep multi-head self-attention, viewing depth as time, and analyzing the residual stream as a particle system.
result Homogenized limit of the dynamics, leading to deterministic or stochastic behavior depending on scaling, with implications for representation collapse.
We employ Hirzebruch-type invariants obtained from iterated p-covers to investigate concordance of links and string links. We show that the invariants naturally give various group homomorphisms of the string link concordance group into L-groups over number fields. We also obtain homomorphisms of successive quotients of…
A new depth measure based on optimal control theory captures multi-modal data.
problem Statistical depths for high-dimensional data.
method Eikonal equations and optimal control theory.
result The new depth measure is robust under adversarial models.
A new depth measure and median defined on Hadamard manifolds.
problem Statistical depth and median on Hadamard manifolds.
method Horospherical depth and Busemann median defined using renormalized distance functions.
result The Busemann median exists for every Borel probability measure on Hadamard manifolds.
New approach uses loss functions to extend data depth for anomaly detection.
problem Anomaly detection in high-dimensional data.
method Introducing loss depths to generalize halfspace depth.
result New loss depths improve anomaly detection efficiency and interpretability.
New findings on depth vs. width in neural networks, showing depth can improve learnability.
problem Understanding the role of depth in neural networks, especially when width is unbounded.
method Analyzing sample complexity for learnability in norm-controlled depth-2 and depth-3 ReLU networks.
result Depth can improve learnability of functions that are otherwise unlearnable with depth-2 networks.
Learning based methods have shown very promising results for the task of depth estimation in single images. However, most existing approaches treat depth prediction as a supervised regression problem and as a result, require vast quantities of corresponding ground truth depth data for training. Just recording quality d…
Introduces Polar Depth for analyzing multivariate heavy-tailed data extremes.
problem Analyzing the behavior of extremes from multivariate heavy-tailed distributions.
method Introduces Polar Depth, a novel statistical depth function expressed in polar coordinates.
result The polar depth of the largest observations converges to the polar depth of the limiting distribution as the threshold increases.
Study on feature learning dynamics in infinite-depth neural networks, focusing on ResNets.
problem Understanding how features evolve during training in deep neural networks, especially in the large-depth limit.
method Conditional Gaussian representations and SDE system with decoupled backward weights.
result Depth-induced suppression of forward-backward coupling in infinite-depth networks, leading to a decoupled forward-backward SDE system.
New algorithm proves deep networks can learn better than shallow ones.
problem Understanding the power difference between shallow and deep neural networks.
method Identifying a class of Boolean functions and proving that logarithmic-depth networks can learn them efficiently using hierarchical reconstruction.
result First algorithmic separation between constant-depth and logarithmic-depth neural networks.