The paper improves lower bounds on deep neural network trajectories.
problem Understanding the expressivity of deep neural networks.
method Generalized method for lower bounding trajectory growth in random sparse deep ReLU networks.
result Trajectory growth can remain exponential in depth with sparse variants of random nets.
Model-based neural networks generalize better than ReLU networks for sparse recovery.
problem Understanding and quantifying the superior generalization of model-based neural networks.
method Using complexity measures like global and local Rademacher complexities, the paper provides theoretical bounds on generalization and estimation errors.
result Model-based neural networks exhibit higher generalization capabilities for sparse recovery problems compared to ReLU networks.
ReLU networks learn simple models even with many parameters, overcoming traditional wisdom.
problem Generalization of overparameterized neural networks.
method Convex optimization and sparse recovery perspective applied to two-layer ReLU networks with standard weight decay.
result ReLU networks learn simple models that explain the data, analogous to sparse recovery in compressed sensing.
Guarantees sparse recovery for neural networks with iterative hard thresholding.
problem Recovering sparse network weights in neural networks.
method Structural properties of sparse network weights and iterative hard thresholding algorithm.
result Simple iterative hard thresholding algorithm recovers sparse network weights exactly using linear memory.
Paper reveals hidden convexities in deep learning models using sparse signal processing.
problem Non-convex loss functions in deep learning models complicate optimization and theoretical understanding.
method Developed convex equivalences of ReLU NNs and their connections to sparse signal processing models.
result Recent research has uncovered hidden convexities in certain NN architectures, notably two-layer ReLU networks and other architectures.
New proof shows neural networks can memorize training data with high accuracy.
problem Proving neural networks can memorize training data with high accuracy.
method Probabilistic construction exploiting sparsity.
result Proves neural networks with threshold or mixed threshold-ReLU activations can memorize training data.
This paper finds sparsest ReLU networks for interpolating data.
problem Finding the sparsest neural network that fits a dataset.
method Proposes a continuous, differentiable objective function based on ℓp quasinorms. result Global minimizers of the proposed objective correspond to sparsest ReLU networks.
Improved bounds for neural network approximations of functions.
problem Bounding the width of neural networks for function approximation.
method Extending Radon-based norms to bounded open sets and deriving new approximation bounds.
result Improved sparse approximation bounds for neural networks.
New capacity measure for deep ReLU networks derived from weight norms.
problem Identifying a suitable capacity measure for deep ReLU networks.
method Generalization of a recently proposed sampling argument to demonstrate the existence of sparse approximants of positive homogeneous networks.
result Bounding generalization error in multi-class classification using covering number bounds.
A new method uses Coordinate Descent to optimize ResNet networks for private inference.
problem Reducing ReLU count in ResNet networks for private inference.
method Directly optimizing in the discrete domain using Coordinate Descent.
result Our method yields a sparse solution and is state-of-the-art.
Activation functions play a key role in providing remarkable performance in deep neural networks, and the rectified linear unit (ReLU) is one of the most widely used activation functions. Various new activation functions and improvements on ReLU have been proposed, but each carry performance drawbacks. In this paper, w…
This work shows how penalising bias terms in norm regularisation leads to sparse solutions.
problem Understanding the relation between parameter norm regularization and the sparsity of neural network solutions.
method Analyzes one hidden ReLU layer networks with unidimensional data, showing the norm required for function representation and the importance of the bias term's norm.
result Penalising the bias terms in regularisation leads to sparse solutions, enforcing the uniqueness and sparsity of the minimal norm interpolator.
Sparse connectivity improves generalization in neural networks below the Edge of Stability.
problem Generalization guarantees for fully-connected networks fail at the Edge of Stability.
method Analyzed sparse connectivity's impact on generalization in two-layer ReLU networks.
result Sparse connectivity changes the effective constraint, leading to non-vacuous generalization bounds.
Deep ReLU networks can efficiently approximate Sobolev and Besov functions.
problem Approximating functions in Sobolev and Besov spaces using deep neural networks.
method Used deep ReLU neural networks with varied width and depth to approximate functions in Sobolev and Besov spaces.
result Generalized the approximation rate to hold under the Sobolev embedding condition.
Neural networks can approximate positive homogeneous functions, especially with multiple hidden layers.
problem Approximating positive homogeneous functions with neural networks.
method Using scale-invariant ReLU networks with multiple hidden layers.
result Approximation of positive homogeneous functions is possible with neural networks, especially with two hidden layers.
Consider the multivariate nonparametric regression model. It is shown that estimators based on sparsely connected deep neural networks with ReLU activation function and properly chosen network architecture achieve the minimax rates of convergence (up to logn-factors) under a general composition assumption on the re…
Paper analyzes how neural networks learn from a teacher in a specific setting.
problem Understanding how two-layer ReLU neural networks learn from a teacher in a regression model.
method Used gradient descent with specific regularization and over-parameterization, combined with measure representation and sparse estimation.
result Student network can identify teacher network parameters with high probability via gradient descent.
Whereas recovery of the manifold from data is a well-studied topic, approximation rates for functions defined on manifolds are less known. In this work, we study a regression problem with inputs on a d∗-dimensional manifold that is embedded into a space with potentially much larger ambient dimension. It is shown tha…
Curriculum learning helps neural networks learn parities more efficiently.
problem Improving learning efficiency for neural networks on parity targets.
method Using a curriculum learning approach with a mixture of sparse and dense inputs.
result A 2-layer ReLU neural network can learn parities more efficiently than a fully connected network.
Deep ReLU networks approximate functions in Sobolev and Besov spaces efficiently.
problem Efficiently approximating functions in Sobolev and Besov spaces using deep ReLU networks.
method Novel bit-extraction technique and VC-dimension method for deriving approximation bounds.
result Sharp upper and lower bounds for Lp-approximation of functions in Sobolev and Besov spaces. The paper tightens bounds on covering numbers for deep ReLU networks.
problem Characterizing the capacity and performance of deep ReLU networks.
method Derives tight lower and upper bounds on metric entropy of ReLU networks.
result Establishes optimality in nonparametric regression via deep networks.
Deep neural networks with specific parameter sets can approximate smooth functions efficiently.
problem Approximating smooth functions with deep neural networks.
method Deep neural networks with ReLU activation and specific parameter sets {0,±21,±1,2} are used to approximate Cβ-smooth functions. result The constructed networks can approximate Cβ-smooth functions with parameters {0,±21,±1,2} efficiently, achieving the same convergence rate as sparse networks with parameters in [−1,1]. Deep neural networks improve mean function estimation for functional data.
problem Estimating mean functions of functional data.
method Deep neural networks with ReLU activation, sparsely connected.
result Achieves optimal nonparametric convergence rate in empirical norm.
This work studies how contrastive learning extracts features from unlabeled data.
problem How neural networks trained by contrastive learning can extract features from unlabeled data.
method Formal analysis of contrastive learning's feature learning process, considering two types of features: sparse and dense.
result Contrastive learning using ReLU networks can learn sparse features if proper augmentations are adopted.
The angular power spectrum characterizes neural network complexity.
problem Characterizing the complexity of deep neural networks.
method Using the angular power spectrum of the limiting field to characterize network complexity.
result Classified neural networks as low-disorder, sparse, or high-disorder.
There has been a large amount of interest, both in the past and particularly recently, into the power of different families of universal approximators, e.g. ReLU networks, polynomials, rational functions. However, current research has focused almost exclusively on understanding this problem in a worst-case setting, e.g…
Rectified linear units, or ReLUs, have become the preferred activation function for artificial neural networks. In this paper we consider two basic learning problems assuming that the underlying data follow a generative model based on a ReLU-network -- a neural network with ReLU activations. As a primarily theoretical …
New method reduces over-parametrization in neural networks, ensuring sparsity and finite network size.
problem Over-parametrization leads to too many active neurons in neural networks, especially with large data.
method Investigates a nonconvex regularization method for shallow ReLU networks.
result Locally optimal networks are finite even with infinite data, maintaining approximation guarantees and network size bounds.
This study connects ReLU neural networks to toric geometry to analyze function realization.
problem Determining which continuous piecewise linear functions can be realized by ReLU neural networks.
method Established a connection between toric geometry and ReLU neural networks, defining key structures like the ReLU fan, toric variety, and Cartier divisor.
result Proved a criterion for functions realizable by unbiased shallow ReLU networks using intersection numbers.
We discuss approximation of functions using deep neural nets. Given a function f on a d-dimensional manifold Γ⊂Rm, we construct a sparsely-connected depth-4 neural network and bound its error in approximating f. The size of the network depends on dimension and curvature of the manifold Γ, the…
QSD enhances deep network performance through biologically plausible dropout.
problem Overfitting in deep networks.
method Quantal Synaptic Dilution (QSD) model based on neuronal synapses.
result QSD outperforms standard dropout in various deep network architectures.
Bayesian approach improves uncertainty in deep learning models.
problem Uncertainty quantification in deep learning models.
method Bayesian point of view, Gaussian approximability, semi-parametric Bernstein-von Mises theorems.
result Bayesian credible regions have valid frequentist coverage, providing theoretical justification for deep learning.
Spike-and-Slab Deep Learning (SS-DL) is a fully Bayesian alternative to Dropout for improving generalizability of deep ReLU networks. This new type of regularization enables provable recovery of smooth input-output maps with unknown levels of smoothness. Indeed, we show that the posterior distribution concentrates at t…
Optimal sketching bounds for sparse linear regression under various loss functions are established.
problem Sparse linear regression under different loss functions.
method Distribution over oblivious sketches for sparse ℓ2 norm regression and hinge-like loss functions. result Optimal sketching bounds with O(klog(d/k)/ε2) rows for sparse ℓ2 norm regression and O(μ2klog(μnd/ε)/ε2) rows for hinge-like loss functions. Deep ReLU networks can be simplified to a three-layer model.
problem Understanding the behavior of deep neural networks.
method Constructive proof and algorithm to transform deep networks into shallow ones.
result Deep ReLU networks can be represented by a simpler three-layer structure.
Theoretical analysis of deep neural networks for time series data.
problem Theoretical development for deep neural networks on temporally dependent observations is lacking.
method Established non-asymptotic bounds for prediction error of deep neural networks under mixing-type assumptions.
result Deep neural networks can model non-linear time series data with additional logarithmic factors due to dependence.
New theory shows large learning rates prevent overfitting in neural networks.
problem Generalization of two-layer ReLU neural networks in noisy regression problems.
method Gradient descent with constant learning rate converges to stable minima.
result Gradient descent with large learning rates finds smooth, sparse fits.
Single ReLU neuron's gradient dynamics reveal support vectors as key to generalization.
problem Understanding the generalization capability of ReLU networks.
method Examined gradient flow dynamics and support vectors in single ReLU neuron training.
result Support vectors play a crucial role in the generalization of ReLU networks.
Large deviation principle for deep neural networks with ReLU activation.
problem Understanding the behavior of deep neural networks with ReLU activation.
method Proving a large deviation principle for networks with Gaussian weights and ReLU activation functions.
result Simplified expressions and power-series expansions for the ReLU case.
Proves existence of optimal shallow neural networks with ReLU activation.
problem Proving the existence of optimal shallow feedforward networks with ReLU activation.
method Proves existence of global minima in the loss landscape for continuous target functions using shallow feedforward neural networks with ReLU activation.
result Existence of global minima in the loss landscape for shallow feedforward networks with ReLU activation.
Modified ReLU networks improve regression estimation rates.
problem Regression estimation with smooth functions.
method Using modified ReLU neural networks with specific weight modifications.
result Empirical risk minimizers achieve minimax rate of prediction.
Hard to learn ReLU with Gaussian data, but can approximate efficiently.
problem Learning a ReLU with Gaussian marginals under arbitrary labels.
method Proved hardness and developed an efficient approximation algorithm.
result Efficient approximation algorithm for best-fitting ReLU with error O(opt2/3). Deep neural networks (DNNs) have emerged as key enablers of machine learning. Applying larger DNNs to more diverse applications is an important challenge. The computations performed during DNN training and inference are dominated by operations on the weight matrices describing the DNN. As DNNs incorporate more layers a…
Early alignment in neural networks leads to sparse representations but hinders convergence.
problem The implicit bias of gradient descent during early training phases.
method Quantitative description of early alignment phase in small initialisation, one hidden layer networks.
result Early alignment induces a sparse representation but also hinders convergence to global minima.
Paper analyzes approximation and learning of MoEs with (P)ReLU activation.
problem Scaling up deep learning models with MoEs and (P)ReLU activation.
method Approximation and learning-theoretic analysis of MoMLPs with (P)ReLU.
result MoMLPs can uniformly approximate Lipschitz functions with ε accuracy using O(ε−1) parameters. The paper analyzes deep ReLU CNNs' approximation properties in 2D space.
problem Establishing L2 approximation properties for deep ReLU CNNs. method Analysis based on decomposition theorem for convolutional kernels, properties of ReLU activation, and connections with one-hidden-layer ReLU NNs.
result Universal approximation theorem for deep ReLU CNNs with classic structure.
Improved bounds on neural network expressivity.
problem Understanding neural network expressivity and approximation capabilities.
method Improved bounds on the maximal number of linear regions of ReLU-networks.
result New insights into the expressivity of neural networks.
TILT improves target domain performance by penalizing an auxiliary component on unlabeled target inputs.
problem Improving performance on target domain under covariate shift.
method TILT uses a novel objective function to decompose the source predictor and penalize an auxiliary component on unlabeled target inputs.
result TILT improves target domain performance over source-only training and other baselines.