We improve neural network robustness verification by training for faster stability.
problem Efficient verification of adversarial robustness in deep networks.
method Co-design of weight sparsity and ReLU stability to simplify verification.
result Improving ReLU stability leads to a 4-13x speedup in verification times.
New method reveals properties of ReLU-networks via invertibility.
problem Understanding the behavior of deep neural networks.
method Deriving a theory on invertibility of ReLU-layers and exploring mechanisms affecting stability.
result Numerical results show characteristic properties of ReLU-networks.
The paper examines stability of ReLU networks in tangent space and activation regions.
problem Stability and sensitivity of ReLU networks to small changes.
method Tangent sensitivity measure for ReLU networks, focusing on stability induced by individual examples.
result Tangent sensitivity correlates with the distribution of activation regions and generalization gap.
Novel approach analyzes ReLU networks' training dynamics and proposes GmP for improved optimization.
problem Stochastic optimization instability in ReLU networks impedes convergence and generalization.
method Characteristic activation boundaries analysis and Geometric Parameterization (GmP) technique.
result GmP resolves instability, leading to better optimization, convergence, and generalization.
A new Shapley value approach for neural networks interpretable and stable.
problem Neural networks' interpretability and training stability issues.
method Shapley value approximation for ReLU activation, globally continuous Shapley gradient, Shapley Activation function.
result SA consistently outperforms ReLU in training convergence, accuracy, and stability.
This paper examines how noise affects deep neural networks and improves their performance.
problem The impact of noise on the stability of deep ReLU neural networks for nonparametric regression.
method Investigates the optimal rate of convergence for deep ReLU neural networks under Huber loss, considering the p-th moment of noise and the smoothness of the function.
result The optimal rate of convergence cannot be achieved by ordinary least squares but can be by Huber loss with a properly chosen parameter.
New activation function BrownianReLU improves LSTM network performance on financial time series.
problem Gradient instability in noisy financial time series data.
method Introduces BrownianReLU, a stochastic activation function based on Brownian motion.
result Significantly improved predictive accuracy and generalization on financial datasets.
New results show flat minima in neural networks suffer from high dimensionality.
problem Flat minima in neural networks generalize poorly in high dimensions.
method Theoretical analysis of two-layer ReLU networks with multivariate inputs.
result Flat minima lead to exponentially slower convergence in high dimensions.
The paper explores how data geometry influences generalization in neural networks.
problem Understanding generalization in overparameterized neural networks.
method Theoretical exploration of overparametrized two-layer ReLU networks trained below the edge of stability.
result Generalization bounds adapt to the intrinsic dimension of data distributions and deteriorate as data concentrates towards the unit sphere.
Study on products of large random matrices and neural network gradients stability.
problem Understanding exploding and vanishing gradient problem in deep neural networks.
method Analyzes products of many large random matrices and applies to neural network gradients stability.
result Precise information about the stability of gradients in randomly initialized deep neural networks.
New method stabilizes deep neural networks by setting Lyapunov exponent to zero.
problem Stability issues in deep neural networks with low width.
method Lyapunov initialization method to set Lyapunov exponent to zero.
result Lyapunov exponent governs stability of deep networks; standard methods fail for low width.
Studied how SGD's stability regularization affects generalization in neural networks.
problem Understanding why SGD often generalizes better than GD in neural networks.
method Analyzed stability of SGD and GD through Frobenius norm and trace of Hessian, and compared their generalization properties.
result Stable minima of SGD generalize well, while GD's stability-induced regularization is too weak.
Deep neural networks struggle with numerical instability during training.
problem Numerical instability in gradient descent training of deep neural networks.
method Analysis of floating-point arithmetic and gradient descent in ReLU neural networks.
result It is highly unlikely for ReLU networks to maintain a superlinear number of affine pieces during training.
Study on the optimization of neural networks with ReLU activation and the degeneracy of their parametrizations.
problem Understanding the optimization landscape of neural networks with ReLU activation.
method Analyzing the optimization problem over the space of neural network realizations and establishing inverse stability of the realization map.
result Inverse stability of the realization map is not guaranteed in general but can be established for shallow networks, allowing optimization over restricted sets.
MGDL refines deep neural networks by training grades sequentially, improving stability.
problem Training deep neural networks is challenging due to nonconvex optimization landscapes.
method MGDL trains deep networks grade by grade, freezing previously learned grades and training new ones to fit residuals.
result MGDL guarantees vanishing error in a fixed-width multigrade ReLU architecture.
Injectivity of ReLU networks is characterized for generative models and inverse problems.
problem Injectivity in ReLU networks for generative models and inverse problems.
method Layerwise analysis, worst-case Lipschitz constants, differential topology, random projections.
result Global injectivity of ReLU networks requires expansivity between 3.4 and 10.5 for Gaussian matrices.
Batch normalization prevents rank collapse in deep networks, improving training stability.
problem Rank collapse in randomly initialized deep networks with increasing depth.
method Investigates spectral instabilities in random matrices and uses batch normalization to avoid rank collapse.
result Batch normalization prevents rank collapse in both linear and ReLU networks, improving training stability.
New theory shows large learning rates prevent overfitting in neural networks.
problem Generalization of two-layer ReLU neural networks in noisy regression problems.
method Gradient descent with constant learning rate converges to stable minima.
result Gradient descent with large learning rates finds smooth, sparse fits.
We analyze why some models resist unlearning using linear stability theory.
problem Understanding and predicting when machine learning models resist unlearning.
method Linear stability theory applied to machine learning models, focusing on data coherence and optimization dynamics.
result Data coherence and signal-to-noise ratio (SNR) influence unlearning resistance; lower SNR makes unlearning easier.
Study reveals sharp characterisation of local minima in neural network loss landscapes.
problem Characterizing local minima in high-dimensional two-layer ReLU neural networks.
method Exact low-dimensional representation of local minima using summary statistics and link with one-pass SGD dynamics.
result Local minima in overparameterized neural networks form discrete families with varying stability and reachability.
Sparse connectivity improves generalization in neural networks below the Edge of Stability.
problem Generalization guarantees for fully-connected networks fail at the Edge of Stability.
method Analyzed sparse connectivity's impact on generalization in two-layer ReLU networks.
result Sparse connectivity changes the effective constraint, leading to non-vacuous generalization bounds.
Our paper characterizes how ReLU affects GD's implicit bias in high-dimensional neural networks.
problem Understanding the implicit bias of gradient descent on neural networks.
method Novel primal-dual analysis tracking predictions and coefficients.
result The implicit bias approximates the minimum- ℓ 2 \ell_2 ℓ 2 -norm solution with high probability. This work analyzes Batch Normalization through convex optimization, providing insights and improved training methods.
problem Understanding and improving the effectiveness of Batch Normalization in deep neural networks.
method Introducing convex duality to model weight-decay regularized ReLU networks with BN, and designing an explicit regularization approach.
result Gradient Descent provides an algorithmic bias effect on BN networks, which can be explicitly encoded into the convex objective.
Algorithm learns weight matrix from single trajectory of nonlinear dynamical system.
problem Learning weight matrix from a single trajectory of nonlinear dynamical system.
method Algorithm uses global stability and well-conditioned covariance to recover weight matrix.
result Algorithm recovers weight matrix with optimal sample complexity and linear running time.
Noise stability improves understanding of Transformer models.
problem Lack of robustness metrics for real-valued domains and junta-like input dependence in modern LLMs.
method Proposed noise stability as a new metric and developed a practical regularization method.
result Noise stability regularization method accelerates training by 35-75%.
The paper analyzes deep neural networks using rectified linear units.
problem Understanding the individual affine linear representations of deep neural networks.
method Signal processing perspective, atomic decompositions, Lipschitz regularity estimation.
result Conditions for stabilizing learning in deep neural networks without network depth constraints.
Flat solutions don't guarantee generalization for logistic loss in neural networks.
problem Proving flat solutions imply generalization for logistic loss in neural networks.
method Analyzing overparameterized two-layer ReLU networks with univariate input under logistic loss.
result Flat solutions enjoy near-optimal generalization bounds within uncertain sets but can still overfit at infinity.
This study connects ReLU neural networks to toric geometry to analyze function realization.
problem Determining which continuous piecewise linear functions can be realized by ReLU neural networks.
method Established a connection between toric geometry and ReLU neural networks, defining key structures like the ReLU fan, toric variety, and Cartier divisor.
result Proved a criterion for functions realizable by unbiased shallow ReLU networks using intersection numbers.
ResNets promote smoother interpolations than MLPs, enhancing generalization.
problem Understanding the difference in smoothness between ResNets and MLPs.
method Neural Tangent Kernel (NTK) analysis during gradient descent training.
result ResNet's NTK results in smoother interpolations than MLPs.
The paper proves deep ReLU networks can die and proposes a new initialization method to prevent it.
problem Dying ReLU neurons in deep neural networks.
method The paper rigorously proves the dying ReLU problem and proposes a new randomized asymmetric initialization method.
result The new initialization method effectively prevents the dying ReLU problem.
The paper investigates how target normalization and momentum affect dying ReLUs in neural networks.
problem Understanding and mitigating the dying ReLU problem in neural networks.
method Empirical analysis and theoretical modeling of a discrete-time linear autonomous system.
result Target variance plays a crucial role in the dying ReLU phenomenon, and momentum exacerbates this issue.
Single ReLU neuron's gradient dynamics reveal support vectors as key to generalization.
problem Understanding the generalization capability of ReLU networks.
method Examined gradient flow dynamics and support vectors in single ReLU neuron training.
result Support vectors play a crucial role in the generalization of ReLU networks.
Large deviation principle for deep neural networks with ReLU activation.
problem Understanding the behavior of deep neural networks with ReLU activation.
method Proving a large deviation principle for networks with Gaussian weights and ReLU activation functions.
result Simplified expressions and power-series expansions for the ReLU case.
Study shows efficient neural network approach for stochastic bandits.
problem Optimizing decisions in uncertain environments with neural network models.
method OFU-ReLU algorithm that balances exploration and exploitation, using a transformed feature space.
result Achieves i l d e O ( T ) ilde{O}(\sqrt{T}) i l d e O ( T ) regret guarantee for stochastic bandits with ReLU neural networks. Hard to learn ReLU with Gaussian data, but can approximate efficiently.
problem Learning a ReLU with Gaussian marginals under arbitrary labels.
method Proved hardness and developed an efficient approximation algorithm.
result Efficient approximation algorithm for best-fitting ReLU with error O ( o p t 2 / 3 ) O(\mathsf{opt}^{2/3}) O ( opt 2/3 ) . The paper analyzes deep ReLU CNNs' approximation properties in 2D space.
problem Establishing L 2 L^2 L 2 approximation properties for deep ReLU CNNs. method Analysis based on decomposition theorem for convolutional kernels, properties of ReLU activation, and connections with one-hidden-layer ReLU NNs.
result Universal approximation theorem for deep ReLU CNNs with classic structure.
Proves existence of optimal shallow neural networks with ReLU activation.
problem Proving the existence of optimal shallow feedforward networks with ReLU activation.
method Proves existence of global minima in the loss landscape for continuous target functions using shallow feedforward neural networks with ReLU activation.
result Existence of global minima in the loss landscape for shallow feedforward networks with ReLU activation.
Gradient descent learns ReLU functions with non-zero bias efficiently.
problem Learning ReLU functions with non-zero bias under Gaussian distributions.
method Gradient descent starting from random initialization.
result Gradient descent achieves near-optimal error with high probability.
The paper studies ReLU layers, introducing new tools to understand their singular values and Gaussian mean width.
problem Understanding the role of ReLU layers in neural networks and their impact on network performance.
method Introducing ReLU singular values and Gaussian mean width of operators to study ReLU layers.
result ReLU singular values and Gaussian mean width provide metrics for distinguishing correctly and incorrectly classified data.
Random ReLU features are shown to be a universally consistent learning algorithm but struggle with complex functions.
problem Approximating complex functions with random ReLU features.
method Study of random ReLU features through their RKHS and composition of functions.
result Random ReLU features can efficiently approximate complex functions but not as well as multi-layer ReLU networks.
Deep ReLU networks can be simplified to a three-layer model.
problem Understanding the behavior of deep neural networks.
method Constructive proof and algorithm to transform deep networks into shallow ones.
result Deep ReLU networks can be represented by a simpler three-layer structure.
This article concerns the expressive power of depth in neural nets with ReLU activations and bounded width. We are particularly interested in the following questions: what is the minimal width w min ( d ) w_{\text{min}}(d) w min ( d ) so that ReLU nets of width w min ( d ) w_{\text{min}}(d) w min ( d ) (and arbitrary depth) can approximate any continuous functio…
Bayesian ReLU nets fix asymptotic overconfidence with infinite features.
problem Bayesian ReLU nets can be asymptotically overconfident far from training data.
method Extend finite ReLU BNNs with infinite ReLU features via a Gaussian process.
result The resulting model is asymptotically maximally uncertain far from the data.
Modified ReLU networks improve regression estimation rates.
problem Regression estimation with smooth functions.
method Using modified ReLU neural networks with specific weight modifications.
result Empirical risk minimizers achieve minimax rate of prediction.
Improved bounds on neural network expressivity.
problem Understanding neural network expressivity and approximation capabilities.
method Improved bounds on the maximal number of linear regions of ReLU-networks.
result New insights into the expressivity of neural networks.
Restricted Boltzmann Machine (RBM) is a bipartite graphical model that is used as the building block in energy-based deep generative models. Due to numerical stability and quantifiability of the likelihood, RBM is commonly used with Bernoulli units. Here, we consider an alternative member of exponential family RBM with…
Paper tackles learning ReLU networks for binary classification with linearly separable data.
problem Learning two-layer ReLU networks for binary classification with linearly separable data.
method Stochastic gradient descent (SGD) algorithm with random noise perturbation.
result Proves global optimality of SGD for training any single-hidden-layer ReLU network.
The paper proves deep ReLU networks avoid spurious local minima in NTK regime.
problem The existence of spurious local minima in deep ReLU neural networks.
method Theoretical proof under Neural Tangent Kernel regime.
result Deep ReLU networks do not lie in spurious local minima in NTK regime.