Study shows infoGAN's generalization error bound for two-layer networks.
problem Understanding generalization error in infoGAN for two-layer neural networks.
method Analyzes the difference between empirical and population objective functions, derives Rademacher complexity bounds.
result Derives error bound for infoGAN's generalization error in a two-layer network.
Modified training direction reduces generalization error in neural networks.
problem Reducing generalization error in neural networks.
method Theoretical analysis of modified natural gradient descent in function space.
result Modifying training direction in function space reduces total generalization error.
Langevin dynamics fails to produce accurate samples even with small score function errors.
problem Robustness of Langevin dynamics to score function errors.
method Analysis of Langevin dynamics and score function errors.
result Langevin dynamics produces a distribution far from the target distribution in TV distance even with small L2 errors in the score function. The success of deep learning has led to a rising interest in the generalization property of the stochastic gradient descent (SGD) method, and stability is one popular approach to study it. Existing works based on stability have studied nonconvex loss functions, but only considered the generalization error of the SGD in…
Proposes a new sparse recovery method using generalized error function.
problem Sparse recovery in signal processing and imaging.
method Introduces a penalty function with shape and scale parameters for sparse recovery.
result The method improves MRI reconstruction and is theoretically sound.
This research analyzes the error convergence rate of GAN models.
problem Understanding the error convergence rate of GAN models.
method Applying Talagrand inequality and Borel-Cantelli lemma to establish a tight convergence rate.
result Established a tight convergence rate for the error of GAN models.
New dual formulation reduces generalization error for ERM-fDR.
problem Generalization error in constrained optimization problems.
method Introduces a dual formulation of ERM-fDR using Legendre-Fenchel transform and implicit function theorem.
result Explicit characterizations of generalization error for algorithms under mild conditions.
We address the problem of automatic generation of features for value function approximation. Bellman Error Basis Functions (BEBFs) have been shown to improve the error of policy evaluation with function approximation, with a convergence rate similar to that of value iteration. We propose a simple, fast and robust algor…
Study finds simple model-agreement scores perform well in various error estimation scenarios.
problem Evaluating model performance on unseen distributions using disparate scoring functions.
method Rigorously studied popular scoring functions (confidence, local manifold smoothness, model agreement) independently of mechanism choice.
result Simple model-agreement scores outperform confidence- and smoothness-based scores in realistic settings with compromised training data.
The paper explores how neural networks learn logical functions and their generalization error.
problem Learning logical functions with neural networks and understanding generalization error.
method Gradient descent on neural networks, analyzing noise-stability and Boolean influence.
result Gradient descent on certain neural architectures tends to favor low-degree representations, impacting generalization error.
Next generation deep neural networks for classification hosted on embedded platforms will rely on fast, efficient, and accurate learning algorithms. Initialization of weights in learning networks has a great impact on the classification accuracy. In this paper we focus on deriving good initial weights by modeling the e…
Paper introduces a new cost function to improve deep learning model generalization.
problem Overfitting and poor extrapolation of deep learning models.
method Introduces a 'whitening' cost function based on the Ljung-Box statistic.
result Significant improvement in generalization for RNNs and image autoencoders.
OTS error shows small training error doesn't guarantee small test error.
problem The relationship between training and test errors is unclear.
method An analysis of the conditions under which small training set error guarantees small OTS error.
result The theorem is limited to models with distinct training and test distributions.
In theoretical analysis of deep learning, discovering which features of deep learning lead to good performance is an important task. In this paper, using the framework for analyzing the generalization error developed in Suzuki (2018), we derive a fast learning rate for deep neural networks with more general activation …
We consider least squares estimation in a general nonparametric regression model. The rate of convergence of the least squares estimator (LSE) for the unknown regression function is well studied when the errors are sub-Gaussian. We find upper bounds on the rates of convergence of the LSE when the errors have uniformly …
New bounds on machine learning model generalization error moments.
problem Understanding the performance of machine learning models.
method Information-theoretic bounds on the moments of the generalization error of learning algorithms.
result Proposed bounds on generalization error moments and their high-probability bounds.
We consider active, semi-supervised learning in an offline transductive setting. We show that a previously proposed error bound for active learning on undirected weighted graphs can be generalized by replacing graph cut with an arbitrary symmetric submodular function. Arbitrary non-symmetric submodular functions can be…
The paper improves error bounds for Bayesian quadrature in noisy settings.
problem Improving error bounds for Bayesian quadrature in noisy settings.
method Develops a two-step meta-algorithm to relate average-case quadrature error to L2-function approximation error. result Provides new average-case results for various kernels and noise settings.
Group-invariant neural networks improve approximation accuracy for symmetric functions.
problem Improving approximation accuracy for symmetric functions using neural networks.
method Investigates the generalization error of group-invariant neural networks within the Barron framework.
result Group invariance introduces a factor δ that can significantly improve approximation accuracy when it is small.
Analysis of non-asymptotic estimation error and structured statistical recovery based on norm regularized regression, such as Lasso, needs to consider four aspects: the norm, the loss function, the design matrix, and the noise model. This paper presents generalizations of such estimation error analysis on all four aspe…
The paper explores MAE as a loss function for DNN vector-to-vector regression, proving its advantages over MSE.
problem Improving loss function for deep neural network based vector-to-vector regression.
method Presenting performance bounds and new properties of MAE, deriving generalized upper bounds, and interpreting MAE as a Laplacian distribution.
result MAE is a more suitable loss function than MSE for DNN based vector-to-vector regression, especially when errors follow a Laplacian distribution.
New bound matches exact generalization error for quadratic Gaussian problem.
problem Understanding generalization error in quadratic Gaussian problems.
method Information-theoretic approach with new ingredients.
result Exact tight bound for generalization error.
New method uses extreme value theory to estimate neural network errors.
problem Quantifying the error of neural networks, especially for large values.
method Applying extreme value theory to approximate the distribution of error.
result Developed a new estimator for the shape parameter of the Pareto distribution.
In many signal detection and classification problems, we have knowledge of the distribution under each hypothesis, but not the prior probabilities. This paper is aimed at providing theory to quantify the performance of detection via estimating prior probabilities from either labeled or unlabeled training data. The erro…
Error bounds, which refer to inequalities that bound the distance of vectors in a test set to a given set by a residual function, have proven to be extremely useful in analyzing the convergence rates of a host of iterative methods for solving optimization problems. In this paper, we present a new framework for establis…
New error bounds for GANs with nonlinear objective functions derived.
problem Statistical consistency of GANs with nonlinear objective functions.
method Derivation of statistical error bounds for (f,Γ)-GANs using Rademacher complexity. result Proves the statistical consistency of (f,Γ)-GANs. Estimates generalization error for two-layer ReLU NNs through minimum norm solutions.
problem Estimating generalization error for two-layer ReLU NNs trained by mean squared error.
method Uses minimum norm solutions and Neural Tangent Kernel (NTK) regime to derive generalization error bounds.
result Derives an a priori generalization error bound for two-layer ReLU NNs without requiring exponentially large number of neurons.
Study loop corrections in random feature models affecting training and test errors.
problem Analyzing loop corrections in random feature models to understand training and test errors.
method Statistical physics and effective field theory approach to study loop corrections.
result Derived loop corrections to training error, test error, and generalization gap.
In statistical learning theory, generalization error is used to quantify the degree to which a supervised machine learning algorithm may overfit to training data. Recent work [Xu and Raginsky (2017)] has established a bound on the generalization error of empirical risk minimization based on the mutual information $I(S;…
New error bound for diffusion models without dimensionality constraints.
problem Error estimation in diffusion generative models without dimensionality constraints.
method Derive dimension-free error bound using a smooth test functional.
result Explicit, dimension-free bound on generated vs true data distributions.
The paper explores neural scaling laws for deep operator networks, offering a theoretical foundation.
problem Understanding neural scaling laws in deep operator networks.
method Theoretical analysis of approximation and generalization errors.
result Established a theoretical framework to quantify neural scaling laws for deep operator networks.
This paper analyzes SGD with increasingly weighted averaging for optimization and generalization.
problem Improving optimization and generalization for non-strongly convex objectives.
method Comprehensive analysis of increasingly weighted averaging schemes for convex, strongly convex, and non-convex objectives.
result The weight α affects both optimization and generalization errors, revealing a trade-off. This article analyzes the weak error of SGD optimization schemes.
problem Analyzing the error in SGD optimization schemes with respect to a test function.
method Weak error analysis for SGD type optimization schemes.
result The weak error decays at the same speed as in the strong sense.
This work analyzes how different layers in deep neural networks contribute to generalization error.
problem Understanding the role of each layer in deep neural networks for generalization.
method Spectral analysis, Neural Tangent Kernel, Hermite polynomials, Spherical Harmonics.
result Initial layers in deep neural networks have a larger bias towards high-frequency functions.
DeepONet learns nonlinear operators from data to identify differential equations.
problem Learning nonlinear operators from data to identify differential equations.
method DeepONet architecture with branch and trunk nets.
result DeepONet significantly reduces generalization error compared to fully-connected networks.
New bounds for neural networks without loss boundedness assumption.
problem Generalization error bounds for two-layer neural networks.
method Wasserstein distance estimates and moment bounds for stochastic gradient method.
result Dimension-free rate of order O(n−1/2) for independent test data. Paper develops an online learning algorithm for functional data models.
problem Recovering slope functions or predictors in functional data models.
method Online regularized learning algorithm in reproducing kernel Hilbert spaces with polynomially decaying step-size.
result Established fast convergence rates for estimation error without capacity assumption.
There has been a growing interest in expressivity of deep neural networks. However, most of the existing work about this topic focuses only on the specific activation function such as ReLU or sigmoid. In this paper, we investigate the approximation ability of deep neural networks with a broad class of activation functi…
Develops high-dimensional measurement error models for non-linear loss functions.
problem Measurement errors in ultrahigh-dimensional biomedical data.
method Lipschitz loss functions, L1 norm minimization, Lasso analog.
result Improved accuracy in classification and quantile regression problems.
Consider the problem of minimizing functions that are Lipschitz and strongly convex, but not necessarily differentiable. We prove that after T steps of stochastic gradient descent, the error of the final iterate is O(log(T)/T) with high probability. We also construct a function from this class for which the error …
First order discretizations of Langevin diffusion can achieve better generalization error with additional smoothness assumptions.
problem Analyzing generalization error for first order discretizations of Langevin diffusion.
method Providing a sufficient smoothness condition to show that first order methods can achieve arbitrarily runtime complexity for a given expected generalization error.
result First order methods can achieve arbitrarily runtime complexity with additional smoothness assumptions.
Improved generalization bounds for SGD in non-convex learning.
problem Understanding generalization properties of SGD in non-convex settings.
method Introducing Type II perturbed SGD (T2pm-SGD) to analyze generalization error bounds.
result Tighter generalization error bounds for SGD in non-convex learning, especially for sub-Gaussian and bounded loss functions.
Neural networks can approximate functionals on RKHS with error bounds.
problem Approximating functionals on RKHS using neural networks.
method Interpolating orthogonal projections in RKHS using point evaluations.
result Explicit error bounds for various kernels (inverse multiquadric, Gaussian, Sobolev).
We propose a data aggregation-based algorithm with monotonic convergence to a global optimum for a generalized version of the L1-norm error fitting model with an assumption of the fitting function. The proposed algorithm generalizes the recent algorithm in the literature, aggregate and iterative disaggregate (AID), whi…
Data-aware activation function customization reduces neural network error.
problem Current neural networks lack consideration for specific activation functions.
method Linear algebraic explanation and Diaconis-Shahshahani Approximation Theorem criteria for activation functions.
result Using an even activation function like seagull can reduce neural network error by orders of magnitude.
Develops wavelet-based neural network approximation theory.
problem Analyzing neural network approximation capabilities over various activation functions.
method Wavelet frame theory on spaces of homogeneous type, sufficient conditions for approximation, error estimates.
result Derives sufficient conditions for neural networks to approximate any functions in a given space, including non-smooth activations.
This work uses sampling theory to analyze smoothness and error bounds of finite neural networks.
problem Analyzing the function space of finite neural networks and providing error bounds.
method Applying sampling theory to finite neural networks with non-expansive activation functions, considering both deterministic and random sampling.
result Novel error bounds for univariate neural networks under band-limited input assumption, highlighting the advantage of deterministic uniform sampling.
New algorithm reduces neural net error in contextual bandits.
problem Neural contextual bandits with general activation functions.
method Proposed an efficient algorithm with sublinear regret bound.
result Demonstrated provably sublinear regret bound in finite regime.