The paper proves deep ReLU networks can die and proposes a new initialization method to prevent it.
problem Dying ReLU neurons in deep neural networks.
method The paper rigorously proves the dying ReLU problem and proposes a new randomized asymmetric initialization method.
result The new initialization method effectively prevents the dying ReLU problem.
The paper establishes principles for initializing and designing GNNs with ReLU activations to avoid oversmoothing and correlation collapse.
problem Oversmoothing and correlation collapse in deep ReLU GNNs.
method The paper derives and validates three principles for initialization and architecture selection in finite width graph neural networks with ReLU activations.
result Correct initialization, residual aggregation operators, and residual connections significantly improve early training dynamics in deep ReLU GNNs.
New properties of deep ReLU networks at initialization improve performance.
problem Improving performance of deep ReLU networks.
method PAC analysis framework to prove novel properties of He initialization.
result Hidden activation norms and weight gradient norms are preserved under He initialization.
Randomly initialized ReLU networks of depth two can approximate smooth functions well.
problem Approximation power of two-layer networks of random ReLUs.
method Harmonic analysis and ridgelet representation theory for upper bounds, dimensionality arguments for lower bounds.
result Near-matching upper and lower bounds for L2-approximation and Sobolev norms. Gradient descent optimizes deep ReLU networks with proper initialization.
problem Training deep neural networks with ReLU activation.
method Gradient descent and stochastic gradient descent with proper random weight initialization.
result Gradient descent finds global minima for over-parameterized deep ReLU networks.
The paper explores neural network initialization strategies and proposes a new method.
problem The initialization phase of neural networks is underexplored compared to optimization.
method The authors discuss and develop an alternative initialization strategy for DNNs with ReLU activations.
result The new initialization strategy improves the quality of neural network training.
Deep ReLU networks have surprisingly few activation patterns at initialization.
problem Limited expressivity of deep ReLU networks despite theoretical potential.
method Analyzed the number of activation patterns in ReLU networks at initialization and during training.
result The average number of activation patterns is bounded by the total number of neurons raised to the input dimension.
A new scaling calculus helps design and initialize ReLU networks more effectively.
problem Optimizing the design and initialization of ReLU neural networks.
method Proposes a scaling constant for neural network layers and weights, relating it to optimizability.
result A network with a uniform scaling constant is easier to train, and the geometric mean of fan-in and fan-out is a better initialization variance.
Study on trainability of ReLU networks and proposes data-dependent initialization method.
problem Understanding and quantifying the trainability of ReLU networks.
method Introduced death states of neurons, studied probability distribution of active neurons at initialization, proposed data-dependent initialization method.
result Trainability is a necessary condition for successful training and over-parameterization is both necessary and sufficient for minimizing training loss.
New ReLU initialization improves network performance and dynamical isometry.
problem Improving the initialization of ReLU units for better network performance.
method Derive exact joint signal output distribution for fully-connected networks with Gaussian weights and biases, and propose a new initialization scheme for ReLU units.
result Proposed initialization scheme achieves dynamical isometry, improving network performance.
Maximal initial learning rate for deep ReLU networks identified.
problem Finding the optimal initial learning rate for deep neural networks.
method Simple approach to estimate maximal initial learning rate η∗, analyzing its behavior in constant-width fully-connected ReLU networks. result Maximal initial learning rate η∗ is well predicted as a power of depth × width, with specific conditions for network width and input layer training. Study on gradient dynamics of shallow ReLU networks for least-squares interpolation.
problem Understanding the gradient dynamics of shallow ReLU networks for interpolation.
method Theoretical and empirical analysis of gradient flow in non-redundant parameterization.
result Identification of two learning regimes: kernel and adaptive, with distinct interpolant shapes.
New algorithm trains ReLU networks via alternating minimization.
problem Training deep neural networks with ReLU activations.
method Alternating minimization of activation patterns and weight updates.
result Proves linear convergence for recovering true parameters.
New method for symmetric matrix completion using ReLU sampling.
problem Symmetric positive semi-definite low-rank matrix completion with deterministic entry-dependent sampling.
method ReLU sampling, gradient descent with tailored initialization.
result Gradient descent with tailored initialization achieves global minima.
Gradient descent fails to train two-layer ReLU networks, leading to poor performance.
problem Gradient descent training of two-layer ReLU networks initialized by He et al. (2015) fails to find optimal solutions.
method Gradient descent on a least-squares loss for training two-layer (Leaky)ReLU networks.
result Gradient descent only finds bad local minima, leading to linear regression for non-linear target functions.
SGD fails to converge for deep ReLU networks with limited random initializations.
problem SGD convergence in deep neural networks with limited random initializations.
method Analysis of four discretization parameters: network architecture, training data, gradient steps, and random initializations.
result SGD fails to converge for ReLU networks with depth much larger than width.
We reparametrize ReLU NNs as splines to understand their learning dynamics.
problem Understanding the learning dynamics and inductive bias of neural networks.
method Reparametrize ReLU NNs as continuous piecewise linear splines to study learning dynamics.
result Standard weight initializations yield very flat functions, leading to strength and type of implicit regularization.
The paper proves deep ReLU networks avoid spurious local minima in NTK regime.
problem The existence of spurious local minima in deep ReLU neural networks.
method Theoretical proof under Neural Tangent Kernel regime.
result Deep ReLU networks do not lie in spurious local minima in NTK regime.
A new method to rescale ReLU neural networks based on path-lifting.
problem Lack of principled ways to leverage rescaling symmetries in ReLU neural networks.
method Introduces a geometrically motivated criterion to rescale neural network parameters, aligning a kernel in the path-lifting space with a chosen reference.
result Proposed method can speed up training and aligns a kernel in the path-lifting space with a chosen reference.
New algorithms optimize neural networks with ReLU activations using sampling.
problem Optimizing trained neural networks with ReLU activations.
method Iterative algorithm and enhanced algorithm using sampling and neighborhood search.
result The methods reduce the initial MIP problem into smaller LP or MIP problems.
ReLU networks don't exponentially distort curve lengths as previously thought.
problem Understanding how neural networks distort curve lengths with depth.
method Analyzing expected length distortion of ReLU networks with random initialization.
result Expected length distortion does not grow with depth, and shrinks slightly.
AutoInit automatically finds good neural network initialization.
problem Finding optimal neural network initialization is crucial but time-consuming.
method Uses Jacobian tuning to automatically adjust network hyperparameters.
result The method finds good initialization for various network architectures.
Gradient descent learns ReLU functions with non-zero bias efficiently.
problem Learning ReLU functions with non-zero bias under Gaussian distributions.
method Gradient descent starting from random initialization.
result Gradient descent achieves near-optimal error with high probability.
This paper studies large-width asymptotics for ReLU neural networks with α-Stable initializations.
problem Characterizing the large-width behavior of ReLU neural networks with α-Stable initializations.
method Analysis of the large-width distributions and training dynamics of ReLU neural networks initialized with α-Stable distributions.
result For ReLU neural networks with α-Stable initializations, the large-width training dynamics achieve zero training error at a linear rate, characterized by a random kernel.
Gradient descent with logistic loss can interpolate deep networks with smoothed ReLU activations under certain conditions.
problem Conditions for gradient descent to drive logistic loss to zero in deep networks with smoothed ReLU activations.
method Gradient descent applied to fixed-width deep networks with smoothed ReLU approximations (e.g., Swish, Huberized ReLU).
result Gradient descent can drive logistic loss to zero under specific conditions, providing bounds on convergence rate.
Sample variance decay is shown in deep ReLU networks, impacting training dynamics.
problem Sample variance decay in deep ReLU networks during training.
method Decomposed total variance into sample variance and network-averaged sum of sample mean and variance.
result Sample variance decays in later layers of deep ReLU networks, impacting training dynamics.
Gradient descent achieves good generalization for over-parameterized deep ReLU networks.
problem Understanding good generalization in over-parameterized deep neural networks.
method Algorithm-dependent generalization error bound for deep ReLU networks using gradient descent.
result Gradient descent with proper initialization can achieve arbitrarily small generalization error for over-parameterized DNNs.
New method for initializing low-rank neural networks improves performance.
problem Training low-rank neural networks efficiently and accurately.
method Inspired by function approximation, proposes a novel low-rank initialization framework.
result Demonstrates significant gap between spectral and low-rank initialization approaches.
Gradient descent converges to minimum Bayes risk for two-layer ReLU networks in mean field regime.
problem Training two-layer ReLU networks using gradient descent in the mean field regime.
method Describes a condition for convergence to minimum Bayes risk, extending previous results to ReLU-activated networks.
result The condition for convergence does not depend on initialization and concerns weak convergence of network realization.
Single ReLU neuron's gradient dynamics reveal support vectors as key to generalization.
problem Understanding the generalization capability of ReLU networks.
method Examined gradient flow dynamics and support vectors in single ReLU neuron training.
result Support vectors play a crucial role in the generalization of ReLU networks.
Study reveals how initial weights influence convergence in deep ReLU networks.
problem Understanding the dynamics and generalization of deep ReLU networks.
method Teacher-student setting, gradient analysis, and activation assumptions.
result Initial weights close to teacher nodes lead to faster convergence, and fan-out weights of other nodes converge to zero in over-parameterized cases.
Improved bounds on neural network regions using activation histograms.
problem Bounding the number of affine regions in ReLU networks.
method Analysis of algebraic topology problem, extension of framework to subnetwork composition.
result Slightly tighter bounds and insights into parameter initialization.
Two-layer CNNs can overfit well if initialized correctly.
problem Understanding the conditions for benign overfitting in over-parameterized CNNs.
method Extending analysis to fully trainable two-layer CNNs, examining initialization scaling effects.
result Initialization scaling of the output layer is crucial; large scales lead to fixed output behavior, small scales to complex interactions.
Study on learning rates in neural networks of varying depth.
problem Dependence of maximal update learning rate on network depth.
method Analysis of random fully connected ReLU networks with mean-field weight initialization.
result Maximal update learning rate scales like L−3/2 with network depth. New method stabilizes deep neural networks by setting Lyapunov exponent to zero.
problem Stability issues in deep neural networks with low width.
method Lyapunov initialization method to set Lyapunov exponent to zero.
result Lyapunov exponent governs stability of deep networks; standard methods fail for low width.
Study on the complexity of 1D ReLU neural networks, proving growth in linear regions.
problem Understanding the complexity and expressivity of 1D ReLU neural networks.
method Analyzing the number of linear regions in randomly initialized, fully connected 1D ReLU networks in the infinite-width limit.
result The expected number of linear regions grows as a function of the number of neurons in each layer.
Develops ADMM for deep neural networks with sigmoid activations to avoid saturation and improve approximation.
problem Gradient saturation in deep neural networks with sigmoid activations.
method Introduces sigmoid-ADMM pair for training deep sigmoid nets and proves its convergence.
result ADMM avoids saturation and improves approximation of deep sigmoid nets compared to ReLU nets.
ReLU units can 'die' in neural networks, causing slower convergence.
problem ReLU units sometimes produce near-zero outputs during training.
method Simulation and statistical analysis of a simplified ReLU unit model.
result Activation probability decreases as training progresses, leading to slower convergence.
The paper examines how the angle between inputs in ReLU networks decreases with depth, impacting training.
problem Depth degeneracy in neural networks, leading to constant function behavior on initialization.
method Combinatorial expansions and Monte Carlo experiments to analyze the angle between inputs in ReLU networks of increasing depth.
result The angle between inputs in ReLU networks decreases exponentially with depth, leading to constant function behavior on initialization.
Stable unactivated neurons reduce expressiveness in ReLU networks.
problem Reducing expressiveness in ReLU neural networks due to stably unactivated neurons.
method Investigated the probability of neurons being stably unactivated in ReLU networks with symmetric weight and bias distributions.
result Proved the probability of a neuron being stably unactivated in the second hidden layer of a ReLU network.
Early training of deep neural networks leads to small, directionally converging weights.
problem Training dynamics of deep homogeneous neural networks with small initializations.
method Gradient flow analysis and study of KKT points for neural correlation function.
result Weights converge in direction to KKT points during early training stages.
Gradient descent learns ReLU networks with Gaussian inputs and noisy outputs.
problem Learning one-hidden-layer ReLU networks with Gaussian inputs and noisy outputs.
method Gradient descent with tensor initialization for empirical risk minimization.
result Gradient descent converges to ground-truth parameters at a linear rate up to statistical error.
New initialization schemes preserve fractional moments of weights in deep networks, improving training and test performance.
problem Heavy-tailed distribution of stochastic gradients in DNNs during training.
method Developed initialization schemes that preserve any given fractional moment of order s < 2 over layers for various activations.
result The network output admits a heavy-tailed distribution with finite moments, improving training and test performance.
Gradient descent solves non-convex neural networks with random initialization.
problem Gradient descent can solve non-convex neural networks with random initialization.
method Gradient descent, over-parameterized neural networks, random initialization, strong convexity-like property.
result Gradient descent converges to a globally optimal solution at a linear rate.
Gradient descent amplifies random features in neural networks to useful ones.
problem Generalization in neural networks trained on corrupted data.
method Characterization of feature-learning process in two-layer ReLU networks trained by gradient descent.
result Gradient descent amplifies random features to useful ones, achieving near optimal generalization error.
Functional dimension varies in ReLU networks, with implications for symmetry and connectivity.
problem Understanding the functional dimension of ReLU neural networks.
method Careful definition and analysis of functional dimension, study of quotient space and fibers.
result Functional dimension is inhomogeneous and can be non-constant, with implications for symmetry and connectivity.
In this paper, we consider parameter recovery for non-overlapping convolutional neural networks (CNNs) with multiple kernels. We show that when the inputs follow Gaussian distribution and the sample size is sufficiently large, the squared loss of such CNNs is locally strongly convex in a basin of attraction…
Deep ReLU networks escape from the origin via saddle points with a low-rank bias.
problem Understanding the dynamics of gradient descent in deep ReLU networks.
method Analysis of escape directions and singular values of weight matrices.
result The first singular value of the ℓ-th layer weight matrix is at least ℓ41 larger than any other singular value.