New proof shows neural networks can represent all multivariate functions.
problem Representing all multivariate functions with neural networks.
method Proved that three-layer neural networks can represent both continuous and discontinuous functions.
result Three-layer neural networks can represent all multivariate functions, including discontinuous ones.
Rational neural networks approximate functions more efficiently with less depth.
problem Choosing optimal nonlinear activation functions in neural networks.
method Rational activation functions with optimal bounds and efficiency proofs.
result Rational neural networks approximate smooth functions more efficiently than ReLU networks with exponentially smaller depth.
Improved neural network approximates analytic and L^p functions efficiently.
problem Efficiently approximating analytic and L^p functions using neural networks.
method Three-dimensional ReLU network architecture for sawtooth functions, improving approximation rates.
result Substantially improved exponential approximation rates for analytic functions and general L^p functions.
Kolmogorov neural networks can represent various types of functions.
problem Representing different types of functions with neural networks.
method Continuous, discontinuous bounded or unbounded activation functions in a two hidden layer model.
result Kolmogorov neural networks can represent continuous, discontinuous bounded and all unbounded multivariate functions.
Optimizes function networks by selectively evaluating parts of the network, reducing costs.
problem Optimizing expensive function networks where full evaluations are costly.
method Knowledge gradient acquisition function for cost-aware partial evaluations.
result Outperforms existing BOFN methods and benchmarks across various problems.
Complex-valued neural networks can approximate any continuous function.
problem Generalizing the universal approximation theorem to complex-valued networks.
method Characterizing activation functions for complex networks to approximate any continuous function.
result Different activation functions are required for deep vs shallow complex networks to achieve universal approximation.
New neural networks learn distribution functions using quantiles and moments.
problem Approximating functions of distributions in probability spaces.
method Quantile and moment neural networks, mixing quantile and moment features.
result Moment neural network outperforms others for bivariate distributions.
We describe convolutional networks using harmonic functions.
problem Understanding the function space and smoothness of convolutional networks.
method Using reproducing kernel Hilbert spaces and functional ANOVA decomposition.
result Convolutional networks can be decomposed into a sum of elementary functions.
New neural network models for functional data.
problem Handling non-linear functional data.
method Functional Direct Neural Network (FDNN) and Functional Basis Neural Network (FBNN) with gradient-based optimization.
result Demonstrated effectiveness in complex functional models.
PDEs constrain smooth functions in neural networks.
problem Understanding functions computable by neural networks.
method Analyzing smooth hierarchical functions via PDEs.
result Established algebraic PDEs for smooth functions.
Wide neural networks can learn complex functions like gravitational force law.
problem Learning complex functions like gravitational force law with neural networks.
method Extending theoretical bounds to analytic functions on the sphere using SGD and ReLU networks.
result Wide ReLU networks can learn analytic functions efficiently with proportional number of samples.
Neural networks can approximate functionals on RKHS with error bounds.
problem Approximating functionals on RKHS using neural networks.
method Interpolating orthogonal projections in RKHS using point evaluations.
result Explicit error bounds for various kernels (inverse multiquadric, Gaussian, Sobolev).
Deep networks with path norm regularization can approximate analytic functions.
problem Approximating analytic functions with neural networks.
method Path norm regularized deep networks with activation function.
result Deep networks can approximate analytic functions with logarithmic dependence on approximation error.
Sparse neural networks can match dense models on Lipschitz functions.
problem Sparse networks are more efficient but lack theoretical guarantees.
method Formal model of sparse networks, LSH-based routing function, Lipschitz function approximation.
result Sparse networks can approximate dense networks on Lipschitz functions.
New bounds on ReLU networks for low-regular functions.
problem Bounding approximation error for ReLU networks on low-regular functions.
method Complexity analysis of Fourier features residual networks to ReLU networks.
result Approximation error bound proportional to target function norm and inversely proportional to network width and depth.
New RBF networks can approximate any continuous function.
problem Approximating any continuous function on a compact subset.
method Replacing smoothing factors with shifts in RBF networks and proving approximation under certain conditions.
result RBF networks can approximate any continuous function on any compact subset.
A neural network with a single hidden layer can't represent certain multivariable functions.
problem Representing certain multivariable functions with a neural network having only one hidden layer.
method Developed a continuum version of a one-hidden-layer neural network with ReLU activation, and proved constraints on its parameters and second derivative.
result Existence of a smooth binary function that cannot be precisely represented by any such neural network.
Study shows how activation functions impact the storage capacity of treelike neural networks.
problem Understanding the role of activation functions in neural network expressive power.
method Analysis of treelike two-layer networks with various activation functions in the infinite-width limit.
result Activation functions affect storage capacity and robustness, with nonlinearity increasing capacity and decreasing robustness.
A new Kolmogorov-Arnold network improves function approximation and optimization.
problem Approximating potentially irregular functions in high dimensions.
method Proposes a new Kolmogorov-Arnold network (KAN) and provides error bounds and universal approximation theorems.
result Outperforms multilayer perceptrons in accuracy and convergence speed for irregular functions.
Study on size and depth of neural networks for approximating benign functions, showing barriers and explicit results.
problem Understanding how size and depth of neural networks affect their ability to approximate benign functions.
method Analyzing ReLU networks for benign functions, proving barriers and explicit results.
result Explicit benign functions that cannot be approximated by networks of certain sizes or depths, showing barriers to size and depth separation.
New neural network models for complex functional data analysis.
problem Complex relations between functional predictors and responses.
method Function-on-Function regression models using neural networks with continuous hidden layers.
result Demonstrated power and flexibility in handling complex functional models.
Deep networks can learn functions approximated by shallow networks, but not all functions.
problem The learnability of functions by deep neural networks and the approximation capacity of simpler classes.
method Study the connection between learnability and approximation capacity of functions by deep neural networks and simpler classes.
result A necessary condition for a function to be learnable by deep neural networks is to be approximable by shallow networks.
Neural networks can approximate functions uniformly across various measures.
problem Universal approximation of functions across different probability measures.
method Proving neural networks are dense in Orlicz spaces, extending classical theorems.
result Neural networks uniformly approximate functions for weakly compact families of measures.
Study approximates nonlinear functionals using deep ReLU networks.
problem Approximating nonlinear continuous functionals with neural networks.
method Constructs continuous piecewise linear interpolation under simple triangulation, analyzes rates of approximation.
result Established rates of approximation for functional deep ReLU networks.
Wide and shallow networks approximate convex functions well.
problem Understanding why wide and shallow neural networks perform well.
method Analyzing the epigraph of the input-output map of shallow and wide neural networks.
result The epigraph of the input-output map approximates a convex function.
A new deep neural network tackles nonlinear functional regression with improved dimensionality reduction.
problem Nonlinear functional regression in infinite-dimensional functional data analysis.
method Functional deep neural network with adaptive kernel embedding and projection steps.
result Explicit rates of approximating nonlinear smooth functionals are derived, and the network is shown to be effective in both simulated and real datasets.
Deep, wide ConvResNets can approximate functions and their smoothness.
problem Function approximation and smoothness in deep networks.
method Analyzing ConvResNets, proving their ability to approximate functions and their smoothness.
result Large ConvResNets can approximate functions and exhibit sufficient first-order smoothness.
Most deep neural networks use simple, fixed activation functions, such as sigmoids or rectified linear units, regardless of domain or network structure. We introduce differential equation units (DEUs), an improvement to modern neural networks, which enables each neuron to learn a particular nonlinear activation functio…
New method for fair resource allocation in AI-aware networks with unknown utility functions.
problem Fair resource allocation in AI-aware communication networks with unknown utility functions.
method Distributed, data-driven bilevel optimization approach to learn surrogate utility functions.
result The proposed algorithm learns from data to autotune surrogate utility functions for unknown utility functions.
Neural networks cannot approximate certain functions in Sobolev spaces, leading to unbounded parameter growth.
problem Non-closedness of sets of neural networks in Sobolev spaces.
method Construction of sequences of neural networks whose realizations converge to functions not realizable by neural networks.
result Sets of realized neural networks are not closed in order-(m−1) Sobolev spaces Wm−1,p for p∈[1,∞]. GroupSort neural networks can approximate Lipschitz continuous functions.
problem Understanding and improving the expressive power of neural networks with Lipschitz constraints.
method Introduced and studied GroupSort neural networks with constraints on weights, proving their ability to approximate Lipschitz continuous functions.
result GroupSort networks can represent any Lipschitz continuous piecewise linear functions and are well-suited for approximating general Lipschitz continuous functions.
Most deep neural networks use simple, fixed activation functions, such as sigmoids or rectified linear units, regardless of domain or network structure. We introduce differential equation units (DEUs), an improvement to modern neural networks, which enables each neuron to learn a particular nonlinear activation functio…
Deep networks adapt to function regularity and data distribution.
problem Understanding deep learning's adaptability to function regularity and data distribution.
method Developed nonparametric approximation and estimation theories for a broad class of functions using deep ReLU networks.
result Deep neural networks are adaptive to different regularity of functions and nonuniform data distributions.
Hybrid Bayesian neural networks use function uncertainty for probabilistic inference.
problem Uncertainty in neural network weights is hard to specify and interpret.
method Integrates probabilistic layers with standard deterministic layers for function uncertainty.
result Improves probabilistic inference by encoding function uncertainty.
Regularizers change the geometric properties of loss functions in neural networks.
problem Understanding how different regularizers affect the geometric properties of loss functions in neural networks.
method Examined several regularizers, including weight decay, to determine if the regularized loss function becomes Morse.
result For certain regularizers, the regularized loss function becomes Morse, indicating a change in geometric properties.
Complexity measures for neural nets with general activations using path-based norms.
problem Control complexity of neural networks with arbitrary activation functions.
method Approximate general activations with ReLU networks and derive path-based norms for complexity control.
result Preliminary analyses of function spaces and regularized estimators.
We show that deep networks are better than shallow networks at approximating functions that can be expressed as a composition of functions described by a directed acyclic graph, because the deep networks can be designed to have the same compositional structure, while a shallow network cannot exploit this knowledge. Thu…
Researchers derive exact priors for finite Bayesian neural networks.
problem Understanding non-Gaussian priors in finite Bayesian neural networks.
method Analytical derivation of function space priors for finite fully-connected feedforward networks.
result Exact solutions for priors of finite networks, including Meijer G-function for linear networks and mixtures for ReLU networks.
Quantized neural networks can represent all fixed-point functions under certain conditions.
problem Expressive power of quantized neural networks under fixed-point arithmetic.
method Analyzing necessary and sufficient conditions for quantized networks to represent all fixed-point functions.
result Various popular activation functions satisfy the sufficient condition for representing all fixed-point functions.
There has been a growing interest in expressivity of deep neural networks. However, most of the existing work about this topic focuses only on the specific activation function such as ReLU or sigmoid. In this paper, we investigate the approximation ability of deep neural networks with a broad class of activation functi…
Neural networks with integer weights approximate continuous functions efficiently.
problem Approximating continuous functions using neural networks with integer weights.
method Integrates superexpressive activation functions and integer weights.
result Convergence rate of order n2β+d−2βlog2n for neural network regression. This paper provides an overview of activation functions in neural networks.
problem Confusion in activation function selection and properties in deep learning.
method Analytic review of popular activation functions.
result Clarification of activation function properties and selection.
Revisits the connection between neural networks and the Kolmogorov-Arnold theorem.
problem Explains the limitations of using the Kolmogorov-Arnold theorem to explain neural networks with multiple hidden layers.
method Derives modifications of the Kolmogorov-Arnold representation that transfer smoothness properties to the outer function and can be well approximated by ReLU networks.
result Shows that a deep neural network with most layers approximating the interior function is a more natural interpretation of the Kolmogorov-Arnold representation.
Proposes ANOVA-TPNN for stable interpretation of complex functions.
problem Stability issues in estimating components of functional ANOVA models.
method Introduces ANOVA-TPNN based on tensor product basis expansion.
result ANOVA-TPNN provides stable estimation of components.
Modified ReLU networks improve regression estimation rates.
problem Regression estimation with smooth functions.
method Using modified ReLU neural networks with specific weight modifications.
result Empirical risk minimizers achieve minimax rate of prediction.
Neural networks minimize error with shallow ReLU models for function estimation.
problem Estimating unknown functions from noisy data.
method Minimizing squared errors plus weight decay regularization.
result Neural network estimators are minimax optimal up to logarithmic factors.
Functional dimension varies in ReLU networks, with implications for symmetry and connectivity.
problem Understanding the functional dimension of ReLU neural networks.
method Careful definition and analysis of functional dimension, study of quotient space and fibers.
result Functional dimension is inhomogeneous and can be non-constant, with implications for symmetry and connectivity.
Neural networks are dense among Lipschitz functions with fixed Lipschitz constant.
problem Characterizing neural network approximations to Lipschitz functions.
method Analyzing L-Lipschitz neural networks and their density in L-Lipschitz functions. result One layer neural networks are dense in the set of all L-Lipschitz functions.