Softmax attention approximates complex functions and subsumes many known universal approximators.
problem Universal approximation of continuous sequence-to-sequence functions.
method Interpolation-based analysis of attention's internal mechanism, showing its ability to approximate ReLU functions.
result Softmax attention is a universal approximator for continuous sequence-to-sequence functions.
Universal approximation for stochastic processes using Brownian motion.
problem Approximating stochastic processes with linear functionals.
method Establishing Lp-type universal approximation theorems for rough path spaces. result Linear functionals on the signature of time-extended Brownian motion can approximate any p-integrable stochastic process. Minimum width for ReLU networks to approximate L^p functions is max(d_x+1, d_y).
problem Characterizing the minimum width for ReLU networks to approximate L^p functions.
method Analyzing networks with ReLU activation functions and proving the minimum width required.
result The minimum width required for the universal approximation of L^p functions is exactly max(d_x+1, d_y).
CF-INNs can approximate any invertible function, resolving a long-standing problem.
problem Whether CF-INNs can approximate any invertible function.
method Demonstrated CF-INNs are universal approximators for invertible functions by showing a convenient criterion.
result CF-INNs are universal approximators for invertible functions.
Universal approximation theorem for differentiable maps on infinite-dimensional manifolds
problem Approximation of differentiable maps on infinite-dimensional manifolds
method Weighted universal approximation theorem
result Universal approximation theorem for differentiable maps
Novel approach to financial derivatives pricing using rough path theory.
problem No-arbitrage conditions in financial markets necessitating precise integration methods.
method Developed a polynomial-based approximation class for rough path functionals, extending to non-geometric rough paths.
result Motivated a hypothesis for payoff functionals in financial markets, facilitating analysis.
Transformers enable in-context learning with guarantees for a wide range of tasks.
problem How to enable in-context learning with transformers for various tasks.
method Developed a universal approximation theory integrating Barron's function approximation with transformer capabilities.
result Transformers can approximate any target function with vanishingly small risk using a few in-context examples.
Framework for universal graph function approximators outperforms existing methods.
problem Graph classification and separation of graph classes.
method Inspired by persistent homology, dependency parsing, and multivalued functions, the framework constructs universal approximators on graph isomorphism classes.
result Achieves state-of-the-art performance on four graph datasets.
Quantum neural networks can approximate noisy functions accurately.
problem Approximating noisy functions with quantum neural networks.
method Universal approximation theorem with error bounds for noisy quantum neural networks.
result Quantum neural networks can approximate noisy functions with precise error bounds.
NODEs can approximate a wide range of diffeomorphisms with strong guarantees.
problem The approximation power of NODEs under certain conditions.
method Leveraging a structure theorem of the diffeomorphism group.
result NODEs can approximate a large class of diffeomorphisms with a stronger guarantee.
Complex-valued neural networks can approximate any continuous function.
problem Generalizing the universal approximation theorem to complex-valued networks.
method Characterizing activation functions for complex networks to approximate any continuous function.
result Different activation functions are required for deep vs shallow complex networks to achieve universal approximation.
MLPs can approximate any function in context, challenging the importance of in-context universality.
problem Understanding why transformers are more effective than classical models.
method Proved MLPs with trainable activation functions are universal in context.
result Transformer success is likely due to factors other than in-context universality.
Unified theorem for deep and shallow joint-equivariant machines.
problem Universal approximation of joint-equivariant machines.
method Constructive universal approximation theorem based on ridgelet transform.
result Unified approximation of deep and shallow networks.
The paper shows neural networks can approximate functions over non-compact domains with non-polynomial activation.
problem Approximating functions over non-compact domains using neural networks.
method Using single-hidden-layer feedforward neural networks with non-polynomial activation functions over non-compact subsets of Euclidean spaces.
result Neural networks can approximate functions in weighted Ck-spaces and weighted Sobolev spaces over unbounded domains. HDNNs can approximate any continuous function, proving their expressivity.
problem Lack of a comprehensive study on the expressivity of HDNNs.
method Discretization of Hamiltonian Neural Ordinary Differential Equations (HNN-ODEs).
result HDNNs can approximate any continuous function over a compact domain.
TVS-FNNs can approximate any continuous function on expanded input spaces.
problem Processing a broader range of inputs like sequences and matrices.
method Proving a universal approximation theorem for TVS-FNNs.
result TVS-FNNs can approximate any continuous function on expanded input spaces.
Sumformer simplifies Transformers to handle long sequences efficiently.
problem Quadratic complexity of Transformers limits their use with long sequences.
method Introducing Sumformer, a simple architecture that universally approximates equivariant sequence-to-sequence functions.
result Sumformer achieves the first universal approximation results for Linformer and Performer.
Universal approximation for rough paths and Lévy processes.
problem Approximating continuous functionals of càdlàg paths.
method Linear functionals of time-extended signatures.
result Universal approximation theorem for continuous functionals of càdlàg paths.
Unified method for CNNs to approximate equivariant maps across various groups.
problem Limited universal approximation theorems for CNNs with specific groups and settings.
method Unified approach to derive universal approximation theorems for equivariant maps by CNNs in diverse settings.
result Ability to handle non-linear equivariant maps between infinite-dimensional spaces for non-compact groups.
Generalizes neural network approximation to infinite-dimensional manifolds and derivatives.
problem Approximating differentiable maps on infinite-dimensional manifolds.
method Proves a weighted Nachbin theorem to establish universal approximation for differentiable maps, including derivatives.
result Linear functions of the signature can approximate path space functionals including their derivatives.
PLN-Nets with two linear layers and parallel LN achieve universal approximation.
problem Limitations of standard neural network architectures in universal approximation.
method Introduced PLN-Nets combining two linear layers with parallel LN.
result PLN-Nets achieve universal approximation, while standard LN has limited power.
We show that deep narrow Boltzmann machines are universal approximators of probability distributions on the activities of their visible units, provided they have sufficiently many hidden layers, each containing the same number of units as the visible layer. We show that, within certain parameter domains, deep Boltzmann…
Neural networks can approximate functions uniformly across various measures.
problem Universal approximation of functions across different probability measures.
method Proving neural networks are dense in Orlicz spaces, extending classical theorems.
result Neural networks uniformly approximate functions for weakly compact families of measures.
MPE framework proves universal approximation for quantum data distribution.
problem Challenges in generating quantum data from underlying distributions.
method Many-body Projected Ensemble (MPE) framework for quantum state design.
result MPE can approximate any quantum distribution within 1-Wasserstein distance error.
Universal approximation for ODENet and ResNet with a single activation function.
problem Approximating complex dynamical systems with limited vector fields.
method Examined ODENet and ResNet with vector fields composed of a single activation function and affine mapping.
result ODENet and ResNet with restricted vector fields can uniformly approximate those with general vector fields.
Develops hierarchical reinforcement learning value function approximators.
problem Estimating long-term returns in reinforcement learning with multiple goals.
method Introduces hierarchical universal value function approximators (H-UVFAs) using the options framework.
result Demonstrates generalization and improved performance of H-UVFAs over UVFAs.
Single-head attention approximates any function under various norms.
problem Universal approximation of functions using attention mechanisms.
method Interpreting attention as partitioning and summing linear transformations.
result Single-head attention can approximate any continuous function under L∞-norm and Lebesgue integrable functions under Lp-norm. Dense neural networks can't approximate all functions.
problem Approximation capabilities of dense neural networks.
method Model compression approach combining weak regularity lemma and graph neural networks.
result Existence of Lipschitz continuous functions not approximable by dense neural networks.
Paper proves GDL models can approximate any continuous function on non-Euclidean data.
problem Processing non-Euclidean data with universal feedforward models.
method Introduces geometric deep learning framework for differentiable manifold geometries.
result GDL models can uniformly approximate any continuous function on compact sets.
Deep residual networks can approximate any continuous function using control theory.
problem Universal approximation capabilities of deep residual neural networks.
method Relating residual networks to control systems and using Lie algebraic techniques.
result Deep residual networks with adequately deep layers can approximate any continuous function on a compact set.
Deep Sets approximates functions on sets with high-dimensional latent space.
problem Modeling functions of sets (permutation-invariant functions).
method Deep Sets, a method known to be a universal approximator for continuous set functions.
result Deep Sets' universal approximation property is only guaranteed with a sufficiently high-dimensional latent space.
Echo state networks with random weights can approximate any continuous system.
problem Approximating continuous dynamical systems using echo state networks.
method Randomly generated internal weights and a sampling procedure for activation functions.
result Echo state networks with random weights can approximate any continuous casual time-invariant operators with high probability.
The paper proves signatures of non-geometric rough paths can approximate functionals uniformly.
problem Approximating functionals of non-geometric rough paths.
method Extending rough paths with time and quadratic variation terms, proving uniform approximation.
result Linear functionals of extended signatures uniformly approximate continuous functionals.
New conditions ensure deep neural networks can approximate any function on non-Euclidean spaces.
problem Understanding how to modify neural network architectures to approximate functions on non-Euclidean spaces.
method Developed conditions for feature and readout maps that preserve universal approximation capabilities.
result Modified architectures can deterministically approximate any classifier on non-Euclidean spaces.
Paper proves neural networks can be approximated using interval bounds.
problem Verifying safety and robustness of neural networks.
method Introduces interval universal approximation (IUA) theorem for neural networks.
result Neural networks can be approximated using interval bounds for any continuous function and squashable activation functions.
The universal approximation property of various machine learning models is currently only understood on a case-by-case basis, limiting the rapid development of new theoretically justified neural network architectures and blurring our understanding of our current models' potential. This paper works towards overcoming th…
Kernel methods have been widely applied to machine learning and other questions of approximating an unknown function from its finite sample data. To ensure arbitrary accuracy of such approximation, various denseness conditions are imposed on the selected kernel. This note contributes to the study of universal, characte…
A new machine learning model uses matrix exponentials for universal approximation.
problem Developing a robust and efficient machine learning model.
method Introduces a novel architecture using matrix exponentials as the only nonlinearity.
result The model achieves universal approximation properties and outperforms other models on benchmark tasks.
Study proves deep narrow RNNs can approximate any function, with minimum width independent of data length.
problem Proving universality of deep narrow RNNs with bounded widths.
method Analyzing RNNs as dynamical systems, proving universality for deep narrow structures with specific widths.
result Minimum width for universality of deep narrow RNNs is independent of data length.
Despite the widespread adoption of Transformer models for NLP tasks, the expressive power of these models is not well-understood. In this paper, we establish that Transformer models are universal approximators of continuous permutation equivariant sequence-to-sequence functions with compact support, which is quite surp…
ELF simplifies normalizing flows, making them more efficient and universal.
problem Computational inefficiency of normalizing flows.
method ELF introduces a simple, one-layer network with closed-form Lipschitz constants, combining the ease of residual flows with the performance of autoregressive flows.
result ELF is a provably universal density approximator, more efficient computationally and parameter-wise.
New proof shows incremental flow models are essential for universal generation.
problem Understanding the universality of flow-based models in generating natural maps.
method Topological-dynamical argument and algebraic properties of flows.
result Incremental generation is necessary and sufficient for universal flow-based generation.
Functional input neural networks approximate continuous functions on weighted spaces.
problem Approximating continuous functions on infinite-dimensional weighted spaces.
method Additive family mapping, non-linear activation, linear readouts, Stone-Weierstrass theorem.
result Global universal approximation of continuous functions on weighted spaces.
Minimum width for ReLU networks on compact domain is exactly max{d_x, d_y, 2}
problem Characterizing the minimum width for ReLU networks to approximate functions on compact domains
method Analyzing the minimum width for Lp approximation of Lp functions from [0,1]d to Rdy using ReLU-like activation functions result The minimum width for Lp approximation on a compact domain is exactly max{d_x, d_y, 2} for ReLU-like activation functions The study of universal approximation of arbitrary functions f:X→Y by neural networks has a rich and thorough history dating back to Kolmogorov (1957). In the case of learning finite dimensional maps, many authors have shown various forms of the universality of both fixed depth and fixed width…
Paper proves CFlows can approximate any diffeomorphism and applies it in Bayesian optimization.
problem Proving the universality of CFlows in approximating diffeomorphisms.
method Deriving the universality of Para-CFlows through affine coupling layers and invertible linear transforms.
result Para-CFlows can approximate any diffeomorphism in C^k-norm.
New RBF networks can approximate any continuous function.
problem Approximating any continuous function on a compact subset.
method Replacing smoothing factors with shifts in RBF networks and proving approximation under certain conditions.
result RBF networks can approximate any continuous function on any compact subset.
o1Neuro neural network approximates complex functions and converges quickly.
problem Approximating complex functions and ensuring convergence in neural networks.
method Sparse indicator activation neurons, population and sample level convergence properties.
result o1Neuro achieves optimal model approximation and convergence with high probability.