Transformers use ReLUs to approximate softmax efficiently.
problem Analyzing resource usage in softmax transformer models.
method Translating ReLU approximation results to softmax attention mechanisms.
result Economic resource bounds for softmax attention mechanisms.
It is a survey on recent results in constructive sparse approximation. Three directions are discussed here: (1) Lebesgue-type inequalities for greedy algorithms with respect to a special class of dictionaries, (2) constructive sparse approximation with respect to the trigonometric system, (3) sparse approximation with …
New findings on how convolutional architectures approximate time series data.
problem Understanding the approximation properties of convolutional architectures in time series modeling.
method Mathematical analysis of convolutional architectures applied to time series modeling.
result A new definition of spectrum-based regularity for measuring temporal relationships under convolutional approximation.
Stochastic approximation algorithms show exponential progress bounds.
problem Analyzing the convergence of stochastic approximation algorithms.
method Developed geometric ergodicity proofs to establish exponential concentration bounds.
result Proved faster convergence rates for specific algorithms.
Paper shows deep neural networks can approximate Korobov functions nearly optimally.
problem Approximating Korobov functions with deep neural networks.
method Used deep neural networks and measured approximation rates with Lp and H1 norms. result Achieved a super-convergence rate, outperforming traditional methods.
Neural approximate computing gains enormous energy-efficiency at the cost of tolerable quality-loss. A neural approximator can map the input data to output while a classifier determines whether the input data are safe to approximate with quality guarantee. However, existing works cannot maximize the invocation of the a…
We are interested in approximation of a multivariate function f(x1,…,xd) by linear combinations of products u1(x1)⋯ud(xd) of univariate functions ui(xi), i=1,…,d. In the case d=2 it is a classical problem of bilinear approximation. In the case of approximation in the L2 space the bili…
Study approximates operator learning for PDEs using Fourier multipliers.
problem Approximating operator behavior for PDE simulations.
method Approximation of operator symbols in Fourier domain using semi-norms.
result Identifies conditions for achieving predefined approximation error.
Neural network based approximate computing is a universal architecture promising to gain tremendous energy-efficiency for many error resilient applications. To guarantee the approximation quality, existing works deploy two neural networks (NNs), e.g., an approximator and a predictor. The approximator provides the appro…
The paper shows neural networks can approximate functions over non-compact domains with non-polynomial activation.
problem Approximating functions over non-compact domains using neural networks.
method Using single-hidden-layer feedforward neural networks with non-polynomial activation functions over non-compact subsets of Euclidean spaces.
result Neural networks can approximate functions in weighted Ck-spaces and weighted Sobolev spaces over unbounded domains. Review of neural network expressivity and architectures.
problem Understanding neural network expressivity across different architectures.
method Comprehensive overview of approximation results for various neural network types.
result Deep neural networks offer advantages over shallow ones for specific function classes.
Length metrics can be closely approximated by conformally flat metrics.
problem Approximating length metrics with conformally flat metrics.
method Uniform approximation of length metrics by conformally flat Riemannian metrics.
result Any length metric on \(\mathbb{R}^d\) can be uniformly approximated by conformally flat Riemannian metrics.
Gradient descent trains shallow neural networks to approximate functions in 1D.
problem Approximating functions in 1D with shallow neural networks trained by gradient descent.
method Gradient descent optimization of non-convex weight space for finite width networks in 1D.
result Gradient descent can approximate functions in 1D with a minimal number of weights, balancing practical performance and theoretical capabilities.
We consider in this paper the optimal approximations of convex univariate functions with feed-forward Relu neural networks. We are interested in the following question: what is the minimal approximation error given the number of approximating linear pieces? We establish the necessary and sufficient conditions and uniqu…
Paper develops Gaussian approximations and bootstrap for federated LSA with trade-off bounds.
problem Analyzing convergence rates and trade-offs in federated linear stochastic approximation.
method Established Berry-Esseen-type bounds for federated LSA, developed multiplier bootstrap for inference.
result First federated Gaussian approximations with explicit trade-off terms and non-asymptotic validity guarantees.
Paper proves neural networks can be approximated using interval bounds.
problem Verifying safety and robustness of neural networks.
method Introduces interval universal approximation (IUA) theorem for neural networks.
result Neural networks can be approximated using interval bounds for any continuous function and squashable activation functions.
Flexible approach for normal approximations in geometric and topological statistics.
problem Normal approximation for complex statistics not expressible as sums of score functions.
method Flexible add-one cost operator combined with strong stabilization theory.
result Established normal approximation results for geometric and topological statistics.
The study assesses low-rank approximations in Gaussian Process regression.
problem Improving Gaussian Process regression efficiency with low-rank approximations.
method Analyzes two low-rank approximations: random Fourier features and Mercer expansion truncation.
result Bounds on the divergence and error between exact and approximate GP models.
High-probability bound for distributed stochastic approximation tracking error.
problem Analyzing the convergence of distributed stochastic approximation schemes.
method Analysis using ODE approach to stochastic approximation.
result High probability bound for tracking error between iterates and limiting differential equation.
The study assesses low-rank approximations in Gaussian Process regression.
problem Improving the efficiency of Gaussian Process regression while maintaining accuracy.
method Analyzes two low-rank approximations: random Fourier features and Mercer expansion truncation, and bounds the divergence and error between exact and approximate models.
result Theoretical bounds on the divergence and error between exact and approximate Gaussian Process models are provided.
Minimum width for ReLU networks to approximate L^p functions is max(d_x+1, d_y).
problem Characterizing the minimum width for ReLU networks to approximate L^p functions.
method Analyzing networks with ReLU activation functions and proving the minimum width required.
result The minimum width required for the universal approximation of L^p functions is exactly max(d_x+1, d_y).
Dropout neural networks can approximate any function with high probability.
problem Approximating functions with dropout neural networks.
method Two universal approximation theorems for dropout neural networks in random and deterministic modes.
result Dropout neural networks can approximate any function in probability and in Lq. Paper develops a new kernel approximation framework.
problem High time and space complexity of kernel methods for large datasets.
method Perturbation-based kernel approximation framework using classical perturbation theory.
result Framework generalizes and improves upon existing methods.
Extends Tian theorem to Vaisman manifolds for approximations.
problem Approximating Vaisman metrics by immersions/embeddings.
method Study Vaisman metrics on compact manifolds.
result Extend Tian's theorem to Vaisman manifolds.
This work improves sample efficiency in neural function approximation for reinforcement learning.
problem Improving sample efficiency in reinforcement learning with neural function approximation.
method Study of function approximation with two-layer neural networks (ReLU and polynomial activations) under generative and realizability models.
result Significant improvement in sample complexity compared to linear methods.
Let U⊆Rn be open and convex. We show that every (not necessarily Lipschitz or strongly) convex function f:U→R can be approximated by real analytic convex functions, uniformly on all of U. In doing so we provide a technique which transfers results on uniform approximation on bounded …
Paper approximates fractional harmonic maps with numerical methods.
problem Approximating fractional harmonic maps with constraints and nonlocality.
method Weak compactness results and numerical methods for various PDEs.
result Convergence of numerical approximations for fractional harmonic maps.
Sharp bounds for approximating Sobolev functions by ridge functions and networks.
problem Approximating Sobolev functions with multivariate ridge functions and networks.
method Proving sharp upper and lower bounds for approximation order.
result Order of approximation asymptotically behaves as n−r/(d−ℓ). Transformers can learn noisy linear systems with depth and IID data.
problem Learning noisy linear dynamical systems with transformers.
method Theoretical analysis of multi-layer and single-layer transformers with respect to L2-testing loss. result Single-layer transformers have a non-diminishing lower bound on approximation error, suggesting depth separation.
In this article, we prove energy quantization for approximate (intrinsic and extrinsic) biharmonic maps into spheres where the approximate map is in LlogL. Moreover, we demonstrate that if the LlogL norm of the approximate maps does not concentrate, the image of the bubbles are connected without necks.
Adaptive optimal control using value iteration (VI) initiated from a stabilizing policy is theoretically analyzed in various aspects including the continuity of the result, the stability of the system operated using any single/constant resulting control policy, the stability of the system operated using the evolving/ti…
Recently, artificial neural networks (ANNs) in conjunction with stochastic gradient descent optimization methods have been employed to approximately compute solutions of possibly rather high-dimensional partial differential equations (PDEs). Very recently, there have also been a number of rigorous mathematical results …
The paper provides approximation guarantees for neural networks trained with gradient flow.
problem Approximating neural networks trained with gradient flow in continuous L2(Sd−1)-norm. method NTK argument for non-convex second but last layer, under-parametrized regime.
result Gradient flow convergence guarantees for neural networks under Sobolev smoothness assumptions.
POUnets combine partitions of unity and monomials for efficient deep learning.
problem Efficiently approximating functions with deep neural networks in high dimensions.
method Integrates partitions of unity and monomials into neural network architecture.
result POUnets achieve hp-convergence for smooth functions and outperform MLPs for discontinuous functions.
Paper introduces new approximations for lognormal sums, matching comonotonicity and moments.
problem Approximating sums of lognormal random variables accurately.
method Introduces new approximations based on weighted distribution theory, emphasizing comonotonicity and moment matching.
result Approximations perform better than classical methods, especially in the right tail of the distribution.
For optimization on large-scale data, exactly calculating its solution may be computationally difficulty because of the large size of the data. In this paper we consider subsampled optimization for fast approximating the exact solution. In this approach, one gets a surrogate dataset by sampling from the full data, and …
In this paper, we propose a generic framework for devising an adaptive approximation scheme for value function approximation in reinforcement learning, which introduces multiscale approximation. The two basic ingredients are multiresolution analysis as well as tree approximation. Starting from simple refinable function…
We build on the dynamical systems approach to deep learning, where deep residual networks are idealized as continuous-time dynamical systems, from the approximation perspective. In particular, we establish general sufficient conditions for universal approximation using continuous-time deep residual networks, which can …
Proposes approximating computationally expensive explainability techniques using conformal regression.
problem Computational expense of score-based explainability techniques limits their applicability in time-critical contexts.
method Uses conformal prediction framework to approximate SHAP and TreeSHAP explanations.
result Significantly improves execution time and produces tight validity guarantees.
Study shows limits on deep and shallow neural networks for approximating compact sets.
problem Understanding the limitations of deep and shallow neural networks in approximating compact sets.
method Proved Carl's type inequalities for approximation error, using Lipschitz widths.
result Lower bounds on approximation error for neural network outputs.
Study on RNNs' ability to approximate past-dependent Hölder functions and their application to regression.
problem Understanding and optimizing the approximation capacity of RNNs for regression tasks.
method Derivation of upper bounds on RNN approximation error for Hölder smooth functions and application to regression.
result Achievement of minimax optimal prediction error bounds for RNNs under various data assumptions.
This study is aimed at answering the famous question of how the approximation errors at each iteration of Approximate Dynamic Programming (ADP) affect the quality of the final results considering the fact that errors at each iteration affect the next iteration. To this goal, convergence of Value Iteration scheme of ADP…
The future predictive performance of a Bayesian model can be estimated using Bayesian cross-validation. In this article, we consider Gaussian latent variable models where the integration over the latent values is approximated using the Laplace method or expectation propagation (EP). We study the properties of several B…
Minimum width for ReLU networks on compact domain is exactly max{d_x, d_y, 2}
problem Characterizing the minimum width for ReLU networks to approximate functions on compact domains
method Analyzing the minimum width for Lp approximation of Lp functions from [0,1]d to Rdy using ReLU-like activation functions result The minimum width for Lp approximation on a compact domain is exactly max{d_x, d_y, 2} for ReLU-like activation functions This paper extends depth separation results to piece-wise oscillatory functions.
problem Approximating functions with piece-wise oscillatory structure using neural networks.
method Extends existing results to piece-wise oscillatory functions using proof strategy from (Eldan and Shamir, 2016).
result Approximation by one-hidden-layer networks holds at a poly(d) rate for functions with constant domain radius and oscillation rate.
Asymptotic results for weighted floating bodies are established and used to obtain new proofs for the existence of floating areas on the sphere and in hyperbolic space and to establish the existence of floating areas in Hilbert geometries. Results on weighted best and random approximation and the new approach to floati…
The paper provides Gaussian approximations for decentralized Federated Learning.
problem Lack of asymptotic statistical guarantees for local SGD in Federated Learning.
method Two generalized Gaussian approximation results for local SGD trajectories.
result Valid multiplier bootstrap procedures and Gaussian bootstrap-based tests for detecting adversarial attacks.
Improved semialgebraic choices with linear complexity.
problem Finding semialgebraic choices in projections with exponential complexity.
method Allowing approximate selections in Hausdorff sense.
result Constructed an approximate selection with linear degree in complexity.