Single ReLU neuron's gradient dynamics reveal support vectors as key to generalization.
problem Understanding the generalization capability of ReLU networks.
method Examined gradient flow dynamics and support vectors in single ReLU neuron training.
result Support vectors play a crucial role in the generalization of ReLU networks.
Researchers can reverse-engineer deep ReLU networks from outputs.
problem Recovering a deep ReLU network from its outputs.
method Dissecting region boundaries to identify neuron states and weights.
result Weights and neuron arrangement of a deep ReLU network can be recovered.
Algorithm learns arbitrary ReLU neurons under Gaussian inputs.
problem Learn an arbitrary ReLU activation over Gaussian marginals.
method Statistical Query (SQ) algorithm that outputs a ReLU activation achieving O(OPT)+ε loss. result First constant factor approximation for arbitrary bias in polynomial time.
The paper proves deep ReLU networks can die and proposes a new initialization method to prevent it.
problem Dying ReLU neurons in deep neural networks.
method The paper rigorously proves the dying ReLU problem and proposes a new randomized asymmetric initialization method.
result The new initialization method effectively prevents the dying ReLU problem.
Stable unactivated neurons reduce expressiveness in ReLU networks.
problem Reducing expressiveness in ReLU neural networks due to stably unactivated neurons.
method Investigated the probability of neurons being stably unactivated in ReLU networks with symmetric weight and bias distributions.
result Proved the probability of a neuron being stably unactivated in the second hidden layer of a ReLU network.
New training method for ReLU networks achieves optimal weight size for memorization.
problem Approximate memorization of arbitrary real labels with neural networks.
method Complex recombination training procedure for ReLU networks.
result Approximate memorization with nearly optimal weight size and neuron count.
Study on how noise and variation-norm regularisation help shallow ReLU networks use fewer neurons.
problem Understanding how shallow ReLU networks use a finite number of neurons in the infinitely wide limit.
method Analysis of two regularisation strategies: noise injection and variation-norm.
result Both regularisation methods minimize functions with a finite number of neurons, regardless of overparametrisation.
New method improves neural network verification by considering multivariate input space of ReLU neurons.
problem Improving the effectiveness of neural network verification algorithms.
method A new tightened convex relaxation for ReLU neurons considering multivariate input space.
result Our convex relaxation is significantly stronger than the commonly used univariate-input relaxation.
Study on trainability of ReLU networks and proposes data-dependent initialization method.
problem Understanding and quantifying the trainability of ReLU networks.
method Introduced death states of neurons, studied probability distribution of active neurons at initialization, proposed data-dependent initialization method.
result Trainability is a necessary condition for successful training and over-parameterization is both necessary and sufficient for minimizing training loss.
Gradient descent struggles with learning a single neuron with bias.
problem Learning a single neuron with a bias term in the realizable setting with ReLU activation.
method Theoretical study using gradient descent, characterizing critical points, and providing convergence guarantees.
result Gradient descent faces significant challenges in learning a single neuron with bias, unlike the bias-less case.
Paper finds how many neurons are needed to approximate histogram distributions.
problem How many neurons are needed to approximate a target probability distribution?
method Examined for uniform input distribution and histogram target distributions, using efficient neural net construction.
result Obtained a new upper bound on the number of required neurons, strictly better than previous bounds.
Polynomial-time algorithm learns ReLU networks without assumptions.
problem Learning linear combinations of ReLU activations with Gaussian inputs.
method Random contractions of moment tensors and multi-scale analysis.
result First polynomial-time algorithm without additional assumptions.
New method learns neural network activation functions from data.
problem Learning activation functions for neural networks.
method Model each neuron's activation function as a small neural network.
result Learned activation functions improve network performance.
New measure FTC quantifies how much a ReLU network can fine-tune.
problem Analyzing memorization capacity in fine-tuned neural networks.
method Defined Fine-Tuning Capacity (FTC) for additive fine-tuning of ReLU networks.
result Upper and lower bounds on FTC for 2 and 3-layer ReLU networks.
Single neuron with ADA learns XOR and outperforms other functions.
problem Classifying linearly non-separable data.
method Proposed a new artificial neuron with apical dendrite activation.
result ADA function achieves 100% accuracy on XOR and superior performance on benchmark datasets.
Paper analyzes GLM-tron for high-dimensional ReLU regression, providing upper and lower bounds.
problem Learning a single ReLU neuron in high-dimensional settings with overparameterization.
method Perceptron-type algorithm GLM-tron, with finite-sample analysis.
result Sharp characterization of high-dimensional ReLU regression problems via GLM-tron, contrasting with SGD.
Training neural networks is hard in fixed dimensions.
problem Training two-layer neural networks is computationally hard in fixed dimensions.
method Parameterized complexity analysis considering dimension and number of neurons.
result Training two-layer neural networks is NP-hard for two dimensions.
Study shows SQ lower bounds for learning ReLUs with Massart noise.
problem Learning a single neuron in the presence of Massart noise.
method Statistical Query (SQ) lower bounds for efficient learning algorithms.
result No efficient SQ algorithm can approximate the optimal error within any constant factor.
Paper studies shallow ReLU networks' approximation rates for Hölder functions.
problem Understanding shallow ReLU networks' efficiency in approximating Hölder functions.
method Analyzes rates of uniform approximation by ReLU shallow neural networks with m hidden neurons. result Shows ReLU shallow neural networks can uniformly approximate Hölder functions with rates close to optimal.
New Lipschitz bound for ReLU networks resists weight rescaling.
problem Lack of robustness guarantees for ReLU networks under weight perturbations.
method Rescaling-invariant Lipschitz bound based on path-metrics.
result The new bound applies to various ReLU-DAG architectures and resists neuron-wise rescalings.
New findings on hidden symmetries in ReLU networks.
problem Understanding the redundancy and symmetries in ReLU network parameter space.
method Analyzing parameter settings and function classes for various network architectures.
result For certain network architectures, there are no hidden symmetries.
The study examines if ReLU activation function is optimal for modularity in neural networks.
problem Finding the best activation function for modularity in neural networks.
method Comparing ReLU with other activation functions for modularity and performance.
result ReLU may not be the best choice for modularity, suggesting other functions could be more suitable.
Study reveals properties of local minima in ReLU networks.
problem Understanding the loss landscape of neural networks.
method Theoretical analysis of one-hidden-layer ReLU networks.
result All differentiable local minima are global within certain regions.
Gradient descent slows significantly in over-parameterized single neuron learning.
problem Learning a single neuron with over-parameterization and square loss.
method Analysis of gradient descent dynamics, proving convergence rates and lower bounds.
result Over-parameterization can exponentially slow down the convergence rate of gradient descent.
New Gated Linear Unit (GaLU) improves neural network performance.
problem Theoretical gap between ReLU networks and empirical observations in shallow networks.
method Introduced Gated Linear Unit (GaLU) by decoupling linearity from gating.
result Improved memorization and generalization results for GaLU networks.
Paper compresses deep neural networks by eliminating redundant neurons.
problem Challenges in deploying deep learning models due to high parameter count.
method Exploits non-linear redundancy to compress neural networks without loss.
result Reduces network size by up to 99% with minimal performance loss.
Paper proves hardness of learning various complex models under local pseudorandom generators.
problem Hardness of learning various complex models.
method Existence of local pseudorandom generators.
result Proves hardness of learning shallow ReLU neural networks and other models.
`Biologically inspired' activation functions, such as the logistic sigmoid, have been instrumental in the historical advancement of machine learning. However in the field of deep learning, they have been largely displaced by rectified linear units (ReLU) or similar functions, such as its exponential linear unit (ELU) v…
Estimates nonlinear Hawkes processes using RKHSs with ReLU rectification.
problem Nonlinear multivariate Hawkes processes with complex interaction functions.
method Nonparametric estimation using RKHSs with approximations for ReLU and integral operators.
result Proposes an estimation method with bounds on approximation errors.
Study on the complexity of 1D ReLU neural networks, proving growth in linear regions.
problem Understanding the complexity and expressivity of 1D ReLU neural networks.
method Analyzing the number of linear regions in randomly initialized, fully connected 1D ReLU networks in the infinite-width limit.
result The expected number of linear regions grows as a function of the number of neurons in each layer.
Novel approach analyzes ReLU networks' training dynamics and proposes GmP for improved optimization.
problem Stochastic optimization instability in ReLU networks impedes convergence and generalization.
method Characteristic activation boundaries analysis and Geometric Parameterization (GmP) technique.
result GmP resolves instability, leading to better optimization, convergence, and generalization.
A method to interpret neural networks by isolating active subnetworks.
problem Interpreting the complex computations of deep neural networks.
method Switched linear projections to decompose network activity.
result Patterns in deactivated neurons are meaningful for network understanding.
Efficiently learns a single neuron with adversarial noise, improving on prior work.
problem Learning a single neuron with adversarial label noise.
method Efficient algorithm using local error bounds from optimization theory.
result Approximates optimal L22-error within a constant factor. Study shows shallow ReLU networks struggle with high-dimensional Lipschitz functions.
problem Expressing high-dimensional Lipschitz functions with shallow ReLU networks.
method Established lower bounds on shallow network complexity for polynomial approximation.
result Shallow ReLU networks suffer from the curse of dimensionality for Lipschitz functions.
Neural networks can represent complex piecewise functions efficiently.
problem Representing continuous piecewise affine functions with neural networks.
method Two hidden layers with ReLU activation, O(p) neurons for p pieces. result CPA functions can be represented by a neural network with linear size.
Gradient-trained shallow networks can generalize well but are vulnerable to small-radius adversarial attacks.
problem Adversarial robustness of gradient-trained shallow networks.
method Analysis of neuron alignment and polynomial ReLU activation.
result Gradient-trained shallow networks with polynomial ReLU activation are robust to small-radius adversarial attacks.
Neural networks learn to mimic brain neurons with two-input activation functions, improving performance and robustness.
problem Training neural networks to mimic the complex interactions of brain neurons.
method Developed a network-in-network architecture with two-input activation functions, optimized hyperparameters, and compared to conventional ReLU networks.
result Two-input activation functions can learn soft XOR functions, improving network performance and robustness.
Deep gated networks help understand training and generalization in deep learning.
problem Understanding the role of SGD in training and generalization of deep neural networks with ReLU activation.
method Developed deep gated networks (DGNs) as a framework to analyze training and generalization in DNNs with ReLU activation.
result Gate adaptation is key for generalization in deep neural networks.
Gradient descent learns ReLU networks with Gaussian inputs and noisy outputs.
problem Learning one-hidden-layer ReLU networks with Gaussian inputs and noisy outputs.
method Gradient descent with tensor initialization for empirical risk minimization.
result Gradient descent converges to ground-truth parameters at a linear rate up to statistical error.
Probabilistic bounds on neuron death in deep networks, showing depth can be increased indefinitely.
problem Understanding neuron death in deep rectifier networks and its impact on model trainability.
method Deriving upper and lower bounds on neuron death probability as a function of model hyperparameters.
result The probability of neuron death decreases as network depth increases, provided width increases proportionally.
New method converts ANN gates to SNNs with AMOS neurons for improved image classification.
problem Efficiently converting ANN gates to SNNs for neuromorphic hardware.
method Introducing AMOS conversion for gates in ANNs, improving accuracy and throughput.
result Improved accuracy of SNNs for ImageNet from 74.60% to 80.97%.
Over-parameterization makes optimization easier for simple neural networks, even with minor extra neurons.
problem Understanding the impact of over-parameterization on optimization landscapes of shallow neural networks.
method Analyzing a simple ReLU neural network with Gaussian inputs, focusing on optimization properties and landscape changes.
result Over-parameterization makes the objective function one-point strongly convex in most directions, aiding optimization.
A corrective neural network approach improves memorization and learning efficiency.
problem Improving neural network memorization and learning efficiency.
method Divide neural network into groups to sequentially approximate and correct errors.
result Two-layer neural networks can memorize arbitrary labels with optimal number of ReLUs.
A new Shapley value approach for neural networks interpretable and stable.
problem Neural networks' interpretability and training stability issues.
method Shapley value approximation for ReLU activation, globally continuous Shapley gradient, Shapley Activation function.
result SA consistently outperforms ReLU in training convergence, accuracy, and stability.
Gradient descent learns a single neuron without knowing the relationship between inputs and labels.
problem Learning a single neuron without knowing the relationship between inputs and labels.
method Using gradient descent to minimize empirical risk over i.i.d. samples, with a nonconvex and nonsmooth optimization problem.
result Gradient descent achieves near-optimal population risk in polynomial time and sample complexity.
New algorithm trains ReLU networks via alternating minimization.
problem Training deep neural networks with ReLU activations.
method Alternating minimization of activation patterns and weight updates.
result Proves linear convergence for recovering true parameters.
Improved bounds on neural network expressivity.
problem Understanding neural network expressivity and approximation capabilities.
method Improved bounds on the maximal number of linear regions of ReLU-networks.
result New insights into the expressivity of neural networks.
Deep ReLU networks can approximate various signal types with exponential error decay.
problem Approximating different signal structures with deep neural networks.
method Demonstrated approximation of polynomials, sinusoidal functions, oscillatory textures, and fractals.
result Finite-width deep ReLU networks require fewer connections than wide finite-depth networks for smooth function approximation.