Smooth approximations for continuous functions on orbit spaces.
problem Approximating continuous functions on orbit spaces.
method Study of subcartesian spaces and proper Lie group actions.
result Continuous functions can be approximated by smooth functions.
Neural networks with integer weights approximate continuous functions efficiently.
problem Approximating continuous functions using neural networks with integer weights.
method Integrates superexpressive activation functions and integer weights.
result Convergence rate of order n2β+d−2βlog2n for neural network regression. Study analyzes a new algorithm for complex optimization problems.
problem Stochastic bilevel optimisation problems in continuous-time models.
method Continuous-time, two-timescale stochastic approximation algorithm.
result Obtained weak convergence rate using central limit theorem.
iMOCA optimizes multiple objectives with continuous approximations for resource efficiency.
problem Optimizing multiple objectives with continuous function approximations that balance accuracy and evaluation cost.
method Information-Theoretic Multi-Objective Bayesian Optimization with Continuous Approximations (iMOCA) selects input and function approximations to maximize information gain per unit cost.
result iMOCA significantly improves over existing single-fidelity methods in approximating the optimal Pareto set.
Fluid approximations have seen great success in approximating the macro-scale behaviour of Markov systems with a large number of discrete states. However, these methods rely on the continuous-time Markov chain (CTMC) having a particular population structure which suggests a natural continuous state-space endowed with a…
Three-hidden-layer neural networks can approximate Hölder continuous functions uniformly with exponential rate.
problem Approximating Hölder continuous functions with neural networks.
method Introduced Floor-Exponential-Step (FLES) networks with three hidden layers.
result Uniform approximation of Hölder continuous functions with an exponential rate.
New Transformers maintain Lipschitz continuity for robustness.
problem Ensuring robustness in Transformers for safety-sensitive applications.
method Introducing gradient-descent-type in-context Transformers with explicit Euler steps of negative gradient flows.
result Universal approximation theorem for Lipschitz continuous Transformers.
TVS-FNNs can approximate any continuous function on expanded input spaces.
problem Processing a broader range of inputs like sequences and matrices.
method Proving a universal approximation theorem for TVS-FNNs.
result TVS-FNNs can approximate any continuous function on expanded input spaces.
New RBF networks can approximate any continuous function.
problem Approximating any continuous function on a compact subset.
method Replacing smoothing factors with shifts in RBF networks and proving approximation under certain conditions.
result RBF networks can approximate any continuous function on any compact subset.
This paper optimizes ReLU networks for approximating Hölder continuous functions.
problem Optimizing the approximation rate of ReLU networks in terms of width and depth.
method Constructive proof of ReLU networks' approximation power with specific width and depth constraints.
result Optimal approximation rate of ReLU networks with width and depth constraints.
This paper tackles discontinuous neural networks for better approximation of piecewise continuous functions.
problem Limitation of neural networks in approximating piecewise continuous functions due to discontinuities.
method Proposes a decoupled two-step procedure to train a discontinuous deep neural network model.
result Provides approximation guarantees for the proposed model in piecewise continuous function spaces.
Some machine learning applications require continual learning - where data comes in a sequence of datasets, each is used for training and then permanently discarded. From a Bayesian perspective, continual learning seems straightforward: Given the model posterior one would simply use this as the prior for the next task.…
Extends neural network approximations to guarantee continuity of real-world learning tasks.
problem Guaranteeing continuity of real-world learning tasks given by conditional expectations.
method Establishing conditions on learning tasks that guarantee their continuity under a factorization of the data-generating process.
result Conditions guaranteeing the continuity of practically any derived learning task.
DQNs can approximate optimal Q-functions with high accuracy on compact sets.
problem Approximating optimal Q-functions in continuous-time Markov Decision Processes.
method Stochastic control, FBSDEs, residual network approximation theorems, large deviation bounds, viscosity solutions.
result DQNs can approximate optimal Q-functions on compact sets with arbitrary accuracy and high probability.
GroupSort neural networks can approximate Lipschitz continuous functions.
problem Understanding and improving the expressive power of neural networks with Lipschitz constraints.
method Introduced and studied GroupSort neural networks with constraints on weights, proving their ability to approximate Lipschitz continuous functions.
result GroupSort networks can represent any Lipschitz continuous piecewise linear functions and are well-suited for approximating general Lipschitz continuous functions.
This paper tackles catastrophic forgetting in neural networks by providing a unified framework for regularization-based continual learning.
problem Catastrophic forgetting in neural networks trained sequentially on multiple tasks.
method Formulates regularization-based continual learning as a second-order Taylor approximation of the loss function, leading to a unified framework.
result Theoretical results indicate the importance of accurate approximation of the Hessian matrix for optimization and generalization.
Simple neural networks approximate any continuous function with fixed neurons.
problem Approximating arbitrary continuous functions with limited neurons.
method Developed simple feed-forward neural networks with a specific activation function.
result Proven that networks with 36d(2d+1) neurons and depth 11 can approximate any continuous function.
Researchers develop neural networks for approximating functions in Banach spaces.
problem Approximating Banach space valued continuous functions.
method Quasi-interpolation Banach space valued neural network operators using algebraic sigmoid functions.
result Jackson type inequalities for function approximation.
This paper uses normalizing flows to approximate transport maps between densities.
problem Approximating transport maps between given densities.
method Construct time-dependent controls using normalizing flows.
result Provides bounds on the number of switches for piecewise constant approximations.
We build on the dynamical systems approach to deep learning, where deep residual networks are idealized as continuous-time dynamical systems, from the approximation perspective. In particular, we establish general sufficient conditions for universal approximation using continuous-time deep residual networks, which can …
Dense neural networks can't approximate all functions.
problem Approximation capabilities of dense neural networks.
method Model compression approach combining weak regularity lemma and graph neural networks.
result Existence of Lipschitz continuous functions not approximable by dense neural networks.
Analyzes non-Markovian environments in stochastic approximation.
problem Understanding learning mechanisms in non-ergodic, non-Markovian settings.
method Analytic framework for transformer learning and continual learning.
result Proposes a new approach to transformer and continual learning.
This article concerns the expressive power of depth in deep feed-forward neural nets with ReLU activations. Specifically, we answer the following question: for a fixed din≥1, what is the minimal width w so that neural nets with ReLU activations, input dimension din, hidden layer widths at most w, and …
RSAC improves lightweight continuous learning efficiency.
problem Excessive training time and memory usage in continuous learning.
method Regularized subspace approximation classifier with feature reduction and regularization.
result RSAC achieves more efficient continuous learning than prior methods.
This work develops efficient methods for continuous-time distributional reinforcement learning.
problem Continuous-time reinforcement learning with return distributions.
method Parameterizing return distributions using quantile representation and showing topological properties.
result Efficient approximation algorithm for continuous-time distributional reinforcement learning.
New algorithm approximates continuous Wasserstein barycenters efficiently.
problem Computing Wasserstein barycenters for continuous distributions.
method Stochastic algorithm using dual potentials and stochastic gradient descent.
result Efficient online approximation of continuous Wasserstein barycenters.
Continuous-time SGD converges under certain conditions, useful for deep learning.
problem Minimizing population expected loss in learning problems.
method Continuous-time approximation of stochastic gradient descent.
result Establishes sufficient conditions for convergence, applicable to overparametrized neural networks.
Proves FR-NGD optimally approximates evolutionary dynamics and continuous Bayesian inference.
problem Optimizing continuous time replicator equations and continuous Bayesian inference.
method Fisher-Rao natural gradient descent (FR-NGD) and its correspondence with evolutionary dynamics.
result FR-NGD optimally approximates continuous time replicator equations and continuous Bayesian inference.
We show that the shortfall risk of binomial approximations of game (Israeli) options converges to the shortfall risk in the corresponding Black--Scholes market considering Lipschitz continuous path-dependent payoffs for both discrete- and continuous-time cases. These results are new also for usual American style option…
New network approximates functions with error decreasing with network width and depth.
problem Approximating functions with high-dimensional data.
method Floor-ReLU networks with specific width and depth.
result Approximation error decreases as network width and depth increase.
We develop methods to efficiently approximate data in metric spaces without additional assumptions.
problem Efficiently approximating data in metric spaces without imposing structural assumptions.
method Identify discrete modulus of continuity, investigate consistency, propose algorithm, and develop approximation theory.
result Consistent approximation of data in metric spaces without structural assumptions.
SFSVI uses Gaussian mixtures to approximate neural network outputs for continual learning.
problem Learning new tasks without forgetting old ones in neural networks.
method Sequential function-space variational inference with Gaussian mixture approximation.
result Gaussian mixture SFSVI outperforms other methods in continual learning.
UDENet and ResNet can approximate any function, with ODENet showing UAP for continuous functions.
problem Approximating any function using ODENet and ResNet.
method Proved UAP for ODENet and ResNet, derived gradient, and applied to various problems.
result UDENet and ResNet can approximate any function, with ODENet showing UAP for continuous functions.
Improved continual learning for neural networks with BN layers using K-FAC extension.
problem Continual learning challenges in neural networks with BN layers.
method Extended K-FAC method to account for inter-example relations, weight merging, and reparameterization for BN layers; proposed weight merging and reparameterization for BN layers; proposed method to select hyperparameters without source task data.
result Better performance in continual learning tasks with BN layers compared to baselines.
New algorithm learns value and advantage functions for continuous-time Markov processes without structural assumptions.
problem Learning value and advantage functions for continuous-time Markov processes without structural assumptions.
method Proposes Sobolev-prox fitted q-learning algorithm based on Hilbert-space positive definiteness and boundedness properties of Bellman operators. result Identifies ellipticity as a key structural property enabling reinforcement learning for Markov diffusions.
George Cybenko's landmark 1989 paper showed that there exists a feedforward neural network, with exactly one hidden layer (and a finite number of neurons), that can arbitrarily approximate a given continuous function f on the unit hypercube. The paper did not address how to find the weight/parameters of such a networ…
Neural networks approximate high-dimensional functions better than theory predicts.
problem Current theory struggles to explain why small neural networks work well in high-dimensional inverse problems.
method Bounding complexity required for neural networks to approximate Hölder or uniformly continuous functions on high-dimensional sets.
result A general theoretical framework explaining empirical successes of smaller networks in inverse problems.
Bandit methods for black-box optimisation, such as Bayesian optimisation, are used in a variety of applications including hyper-parameter tuning and experiment design. Recently, \emph{multi-fidelity} methods have garnered considerable attention since function evaluations have become increasingly expensive in such appli…
Echo state networks with random weights can approximate any continuous system.
problem Approximating continuous dynamical systems using echo state networks.
method Randomly generated internal weights and a sampling procedure for activation functions.
result Echo state networks with random weights can approximate any continuous casual time-invariant operators with high probability.
Continuous-time Bayesian networks (CTBNs) constitute a general and powerful framework for modeling continuous-time stochastic processes on networks. This makes them particularly attractive for learning the directed structures among interacting entities. However, if the available data is incomplete, one needs to simulat…
Paper proves GDL models can approximate any continuous function on non-Euclidean data.
problem Processing non-Euclidean data with universal feedforward models.
method Introduces geometric deep learning framework for differentiable manifold geometries.
result GDL models can uniformly approximate any continuous function on compact sets.
New Banach spaces for ReLU networks enable better function approximation and gradient dynamics analysis.
problem Function approximation and gradient dynamics in multi-layer ReLU networks.
method Developed Banach spaces for ReLU networks, defined new function representations, and analyzed gradient flow dynamics.
result Gradient flow dynamics of the new representation is the continuous analog of gradient descent for ReLU networks.
We show how to use a variational approximation to the logistic function to perform approximate inference in Bayesian networks containing discrete nodes with continuous parents. Essentially, we convert the logistic function to a Gaussian, which facilitates exact inference, and then iteratively adjust the variational par…
In this paper, we introduce the notion of liquid time-constant (LTC) recurrent neural networks (RNN)s, a subclass of continuous-time RNNs, with varying neuronal time-constant realized by their nonlinear synaptic transmission model. This feature is inspired by the communication principles in the nervous system of small …
DR-submodular continuous functions are important objectives with wide real-world applications spanning MAP inference in determinantal point processes (DPPs), and mean-field inference for probabilistic submodular models, amongst others. DR-submodularity captures a subclass of non-convex functions that enables both exact…
We consider the inverse problem of reconstructing the posterior measure over the trajec- tories of a diffusion process from discrete time observations and continuous time constraints. We cast the problem in a Bayesian framework and derive approximations to the posterior distributions of single time marginals using vari…
New bound on neural nets complexity for approximating functions.
problem Approximating continuous functions with shallow neural networks.
method Inspired by Stone-Weierstrass theorem, constructive proof.
result General upper bound on neuron count for accuracy.
Optimized neural network approximates high-dimensional functions with minimal parameters.
problem Achieving optimal approximation of high-dimensional continuous functions with minimal parameters.
method Developed a neural network with a specific activation function and architecture to achieve super approximation property.
result A composed network with at most 10889d + 10887 nonzero parameters achieves super approximation property, suggesting optimality in parameter growth.