Shallow neural networks can represent polynomials efficiently.
problem Representing polynomials using shallow neural networks.
method Using shallow neural networks of width 2(R+d)d to represent d-variate polynomials of degree R. result Derives minimax optimal convergence rate for shallow networks to unknown univariate regression functions.
Wide and shallow networks approximate convex functions well.
problem Understanding why wide and shallow neural networks perform well.
method Analyzing the epigraph of the input-output map of shallow and wide neural networks.
result The epigraph of the input-output map approximates a convex function.
Optimal rates for shallow ReLU networks in nonparametric regression.
problem Approximating smooth and non-smooth functions with shallow ReLU networks.
method Analysis of shallow ReLUk neural networks, using variation norms and deep learning theory. result Optimal approximation rates for shallow ReLU networks in nonparametric regression.
The study compares shallow and deep learning methods for text classification.
problem High-dimensional, sparse data from customer calls.
method Comparative evaluation of shallow, deep, and ensemble methods.
result Ensemble methods improve classification accuracy.
This paper reports the performances of shallow word-level convolutional neural networks (CNN), our earlier work (2015), on the eight datasets with relatively large training data that were used for testing the very deep character-level CNN in Conneau et al. (2016). Our findings are as follows. The shallow word-level CNN…
Deep ReLU networks can be simplified to a three-layer model.
problem Understanding the behavior of deep neural networks.
method Constructive proof and algorithm to transform deep networks into shallow ones.
result Deep ReLU networks can be represented by a simpler three-layer structure.
Deep neural networks can learn smooth functions without parameters.
problem Learning smooth functions from shallow ReLU neural networks.
method Using over-parameterized shallow ReLU neural networks with norm constraints.
result Least squares estimators based on shallow neural networks are minimax optimal.
Deep nets outperform shallow nets in complex feature realization.
problem Realizing complex data features with deep nets.
method Refined covering number estimates and analysis of approximation rates.
result Deep nets can improve performance without additional capacity costs for complex features.
Study compares shallow and deep learning for MS lesion segmentation.
problem Automated segmentation of white matter lesions in early-stage MS patients.
method Training and testing shallow and deep learning architectures on 32 patients.
result Combining shallow and deep architectures improves lesion-wise metrics.
CHOOSE enhances shallow Transformers for wireless symbol detection.
problem Improving wireless symbol detection with shallow Transformers.
method Introducing autoregressive latent reasoning steps within hidden space.
result Lightweight Transformers achieve comparable performance to deep models.
This paper shows linear over-parametrization suffices for shallow neural networks to fit training data.
problem Training shallow neural networks with optimal over-parametrization.
method Used a simple variant of stochastic gradient descent.
result Linear over-parametrization is sufficient for shallow neural networks to fit training data.
New bounds for shallow neural networks with deterministic parameters.
problem Developing generalisation bounds for shallow neural networks.
method PAC-Bayesian theory applied to shallow neural networks with deterministic parameters.
result Empirical non-vacuous bounds for shallow neural networks trained with vanilla SGD.
Gradient descent can solve shallow neural network training problems globally.
problem Training shallow linear neural networks.
method Analyzed the geometric properties of the loss landscape.
result Gradient descent can converge globally due to the landscape's properties.
Sharp lower bounds on shallow neural networks' approximation rates are derived.
problem The efficiency of shallow neural networks in approximating functions.
method Lower bounding the L2-metric entropy and Kolmogorov n-widths of the convex hull of neural network basis functions. result Sharp lower bounds on the approximation rates for shallow neural networks are provided.
Paper proves bounds for shallow networks without dimensionality issues.
problem Avoiding curse of dimensionality and estimating approximation degree.
method Abstract theorem for G-networks on compact metric measure spaces. result Dimension independent bounds for approximation by shallow networks.
Study shows shallow ReLU networks struggle with high-dimensional Lipschitz functions.
problem Expressing high-dimensional Lipschitz functions with shallow ReLU networks.
method Established lower bounds on shallow network complexity for polynomial approximation.
result Shallow ReLU networks suffer from the curse of dimensionality for Lipschitz functions.
Paper tackles invariance of demodulation in shallow water acoustic communications.
problem Frequency-selective signal distortion (Doppler effect) in shallow water environments.
method Developed ML-based demodulation methods using DBN-NN and DBN-CNN.
result Demonstrated invariance of the proposed method to Doppler effect with 2dB error margin.
Proves existence of optimal shallow neural networks with ReLU activation.
problem Proving the existence of optimal shallow feedforward networks with ReLU activation.
method Proves existence of global minima in the loss landscape for continuous target functions using shallow feedforward neural networks with ReLU activation.
result Existence of global minima in the loss landscape for shallow feedforward networks with ReLU activation.
Gradient-trained shallow networks can generalize well but are vulnerable to small-radius adversarial attacks.
problem Adversarial robustness of gradient-trained shallow networks.
method Analysis of neuron alignment and polynomial ReLU activation.
result Gradient-trained shallow networks with polynomial ReLU activation are robust to small-radius adversarial attacks.
Unified model for shallow and deep learning models.
problem Unified representation of shallow and deep learning models.
method Reconciled polynomial machine predicting output via feature kernel and variable reconciling function.
result Existing models can be reduced to reconciled polynomial machine representations.
In this paper, we introduce transformations of deep rectifier networks, enabling the conversion of deep rectifier networks into shallow rectifier networks. We subsequently prove that any rectifier net of any depth can be represented by a maximum of a number of functions that can be realized by a shallow network with a …
A new algorithm for learning shallow neural networks with infinite width.
problem Learning shallow over-parameterized neural networks.
method Sinkhorn proximal algorithm approximating mean field learning dynamics.
result The algorithm performs gradient descent of the free energy associated with the risk functional.
Study shows limits on deep and shallow neural networks for approximating compact sets.
problem Understanding the limitations of deep and shallow neural networks in approximating compact sets.
method Proved Carl's type inequalities for approximation error, using Lipschitz widths.
result Lower bounds on approximation error for neural network outputs.
Paper studies shallow ReLU networks' approximation rates for Hölder functions.
problem Understanding shallow ReLU networks' efficiency in approximating Hölder functions.
method Analyzes rates of uniform approximation by ReLU shallow neural networks with m hidden neurons. result Shows ReLU shallow neural networks can uniformly approximate Hölder functions with rates close to optimal.
Deep ReLU networks approximate as well as shallow ones in kernel regimes.
problem Understanding the limitations of kernel methods for deep ReLU networks.
method Characterizing eigenvalue decays of kernels derived from deep ReLU networks.
result Deep ReLU networks and shallow two-layer networks have equivalent approximation properties in kernel regimes.
Deep networks don't improve on shallow ones for finding minima.
problem Improving representation of multidimensional mappings with deep neural networks.
method Numerical training methods to find minima in deep and shallow networks.
result Minima found with deep networks are worse than those found with shallow networks.
Deep networks without non-linearities are equivalent to shallow ones.
problem Training deep orthogonal linear networks with no non-linearity.
method Riemannian gradient descent and gradient descent on factorization.
result Training deep overparametrized networks is equivalent to shallow ones.
It is well established that neural networks with deep architectures perform better than shallow networks for many tasks in machine learning. In statistical physics, while there has been recent interest in representing physical data with generative modelling, the focus has been on shallow neural networks. A natural ques…
Unified theorem for deep and shallow joint-equivariant machines.
problem Universal approximation of joint-equivariant machines.
method Constructive universal approximation theorem based on ridgelet transform.
result Unified approximation of deep and shallow networks.
New method for efficient proximal mapping of 1-path-norm in shallow networks.
problem Efficiently handling the 1-path-norm of shallow neural networks.
method Closed-form proximal operator for efficient computation and upper bound on Lipschitz constant.
result Proximal mapping allows robust training against adversarial perturbations.
A new method approximates tangent spaces to simplify neural networks.
problem Efficiency of hierarchical neural networks is hindered by their complexity and training requirements.
method Approximates tangent subspace to enable sparse representation and switch to shallow networks.
result The method improves and sometimes surpasses the performance of original networks after a few epochs.
Study shows overparameterization helps shallow neural networks recover signals in high dimensions.
problem Signal recovery in shallow neural networks with overparameterization.
method Gradient flow on population risk, Gaussian distribution assumption, high-dimensional limit analysis.
result Minimal overparameterization is sufficient for strong recovery of signals.
Deep networks learn hierarchical functions more efficiently than shallow ones.
problem Understanding the advantage of deep neural networks over shallow models.
method Analytical study of learning dynamics and generalization performance of deep networks compared to shallow ones.
result Deep networks reduce effective dimensionality, enabling learning with fewer samples.
Combines machine learning and convex limiting for accurate subgrid flux modeling in shallow-water equations.
problem Accurate subgrid flux modeling in shallow-water equations.
method Machine learning and flux limiting for property-preserving subgrid scale modeling.
result The proposed method produces meaningful closures even in untrained scenarios.
This paper analyzes shallow ReLU networks in L^p and Sobolev spaces, focusing on approximation and generalization.
problem Approximation and generalization of shallow ReLU networks in L^p and Sobolev spaces.
method Spherical harmonic analysis and embeddings into spectral Barron spaces for L^p spaces, path-norm control for Sobolev spaces.
result Minimax-optimal rates for nonparametric regression with shallow ReLU networks under path-norm control.
Deep sum-product networks learn faster than shallow models.
problem The speed of parameter optimization in sum-product networks.
method Theoretical analysis and empirical experiments on overparameterized sum-product networks.
result Gradient-based optimization in deep sum-product networks is equivalent to gradient ascent with adaptive and time-varying learning rates and additional momentum terms.
Gradient descent trains shallow neural networks to approximate functions in 1D.
problem Approximating functions in 1D with shallow neural networks trained by gradient descent.
method Gradient descent optimization of non-convex weight space for finite width networks in 1D.
result Gradient descent can approximate functions in 1D with a minimal number of weights, balancing practical performance and theoretical capabilities.
Bonsai learns fast, deep trees for XMC with fast training and high accuracy.
problem Efficiently learning multi-label classification models with millions of labels.
method Develops Bonsai suite of algorithms that generalize label representation and learn shallow trees.
result Bonsai achieves best of fast training and high accuracy on XMC tasks.
Generative model improves noise estimation in stochastic rotating shallow water models.
problem Improving noise estimation in stochastic partial differential equations for fluid dynamics.
method Replaced PCA with a generative model to avoid constraints on stochastic increments.
result Generative model produces better RMSE, CRPS score, and forecast rank histograms.
Dropout speeds up convergence in shallow linear NNs, with a rate bound.
problem Analyzing convergence rate of Dropout in shallow linear NNs.
method Gradient flow analysis and Hessian examination.
result Bound on convergence rate depends on data, dropout probability, and NN width.
New method trains shallow neural networks with subquadratic width scaling.
problem Training shallow neural networks with optimal width scaling.
method Polyak-Lojasiewicz condition, smoothness, standard data assumptions, random matrix theory.
result Subquadratic scaling on network width with standard initialization strategies.
Simpler algorithm learns shallow networks faster.
problem Learning a linear combination of ReLU activations.
method A simpler one-stage algorithm with improved runtime.
result Runs in (d/ε)O(k2) time. Revisits shallow neural networks using Lipschitz norms and measures.
problem Existence and compactness of minimizers in neural network formulations.
method Mean field parametrization, signed measures, duality pairings, Kantorovich-Rubinstein norms.
result Compactness results and uniform large data limits for empirical risk minimization.
Study shows consistency of shallow GCNNs on sampled point clouds under manifold assumption.
problem Consistency of shallow GCNNs on sampled point clouds under manifold assumption.
method Functional analysis perspective, weakly compact product of unit balls, Sobolev regularity, frequency cutoff.
result Proves Γ-convergence of regularized empirical risk minimization functionals and convergence of their global minimizers. Shallow neural nets classify objects perfectly if their distribution is linearly separable.
problem Designing efficient neural networks for classification.
method Constructed shallow sigmoid-type neural networks.
result Achieves 100% accuracy for datasets following a linear separability condition.
Proposes a learning rate method for shallow nets based on gradient Lipschitz constant.
problem Finding optimal learning rates for shallow neural networks.
method Associates learning rate with gradient Lipschitz constant and proposes a search algorithm.
result The proposed method significantly outperforms existing tuning methods.
In this paper, we address the issue of how to enhance the generalization performance of convolutional neural networks (CNN) in the early learning stage for image classification. This is motivated by real-time applications that require the generalization performance of CNN to be satisfactory within limited training time…
SRFRN accelerates image super-resolution using shallow residual units.
problem High computational complexity and time in deep learning image super-resolution.
method SRFRN uses a bicubic interpolated low-resolution image and residual representative units (RFR) for faster and more efficient high-resolution image reconstruction.
result SRFRN achieves superior performance and faster execution time compared to existing methods.