This article analyzes the weak error of SGD optimization schemes.
problem Analyzing the error in SGD optimization schemes with respect to a test function.
method Weak error analysis for SGD type optimization schemes.
result The weak error decays at the same speed as in the strong sense.
Paper develops an online learning algorithm for functional data models.
problem Recovering slope functions or predictors in functional data models.
method Online regularized learning algorithm in reproducing kernel Hilbert spaces with polynomially decaying step-size.
result Established fast convergence rates for estimation error without capacity assumption.
Langevin dynamics fails to produce accurate samples even with small score function errors.
problem Robustness of Langevin dynamics to score function errors.
method Analysis of Langevin dynamics and score function errors.
result Langevin dynamics produces a distribution far from the target distribution in TV distance even with small L2 errors in the score function. Study shows infoGAN's generalization error bound for two-layer networks.
problem Understanding generalization error in infoGAN for two-layer neural networks.
method Analyzes the difference between empirical and population objective functions, derives Rademacher complexity bounds.
result Derives error bound for infoGAN's generalization error in a two-layer network.
Modified training direction reduces generalization error in neural networks.
problem Reducing generalization error in neural networks.
method Theoretical analysis of modified natural gradient descent in function space.
result Modifying training direction in function space reduces total generalization error.
The Bellman error is a poor proxy for value function accuracy, even with all state-action pairs.
problem The Bellman error is a poor proxy for the accuracy of the value function.
method Study of the Bellman equation as a surrogate objective for value prediction accuracy.
result The magnitude of the Bellman error is only weakly related to the distance to the true value function, even with all state-action pairs.
This research analyzes the error convergence rate of GAN models.
problem Understanding the error convergence rate of GAN models.
method Applying Talagrand inequality and Borel-Cantelli lemma to establish a tight convergence rate.
result Established a tight convergence rate for the error of GAN models.
Study finds simple model-agreement scores perform well in various error estimation scenarios.
problem Evaluating model performance on unseen distributions using disparate scoring functions.
method Rigorously studied popular scoring functions (confidence, local manifold smoothness, model agreement) independently of mechanism choice.
result Simple model-agreement scores outperform confidence- and smoothness-based scores in realistic settings with compromised training data.
We consider least squares estimation in a general nonparametric regression model. The rate of convergence of the least squares estimator (LSE) for the unknown regression function is well studied when the errors are sub-Gaussian. We find upper bounds on the rates of convergence of the LSE when the errors have uniformly …
Proposes a new sparse recovery method using generalized error function.
problem Sparse recovery in signal processing and imaging.
method Introduces a penalty function with shape and scale parameters for sparse recovery.
result The method improves MRI reconstruction and is theoretically sound.
We consider active, semi-supervised learning in an offline transductive setting. We show that a previously proposed error bound for active learning on undirected weighted graphs can be generalized by replacing graph cut with an arbitrary symmetric submodular function. Arbitrary non-symmetric submodular functions can be…
Optimal AFs minimize RFR test error and sensitivity.
problem Finding optimal AFs for RFR to minimize test error and sensitivity.
method Closed-form solution for AFs minimizing test error and sensitivity under different functional parsimony.
result Optimal AFs can be linear, saturated linear, or Hermite polynomial expressions.
In many signal detection and classification problems, we have knowledge of the distribution under each hypothesis, but not the prior probabilities. This paper is aimed at providing theory to quantify the performance of detection via estimating prior probabilities from either labeled or unlabeled training data. The erro…
New bounds on ReLU networks for low-regular functions.
problem Bounding approximation error for ReLU networks on low-regular functions.
method Complexity analysis of Fourier features residual networks to ReLU networks.
result Approximation error bound proportional to target function norm and inversely proportional to network width and depth.
Learning reward functions can lead to poor policy performance despite low error.
problem Low error in learned reward functions does not guarantee low regret in policy performance.
method Mathematical analysis of reward learning and policy optimization.
result A low expected test error of the reward model guarantees low worst-case regret, but error-regret mismatch can occur with certain data distributions.
The success of deep learning has led to a rising interest in the generalization property of the stochastic gradient descent (SGD) method, and stability is one popular approach to study it. Existing works based on stability have studied nonconvex loss functions, but only considered the generalization error of the SGD in…
We address the problem of automatic generation of features for value function approximation. Bellman Error Basis Functions (BEBFs) have been shown to improve the error of policy evaluation with function approximation, with a convergence rate similar to that of value iteration. We propose a simple, fast and robust algor…
Inexact subgradient methods work well for semialgebraic functions with additive errors.
problem Approximate gradients in machine learning and optimization.
method Inexact subgradient methods with persistent additive errors in semialgebraic functions.
result Iterates eventually fluctuate near the critical set with a proximity of O(ερ), where ε is the magnitude of subgradient evaluation errors. K-DAREK improves KKANs for efficient function approximation with robust error bounds.
problem Efficient function approximation with uncertainty quantification for large-scale problems.
method Developed a novel learning algorithm, K-DAREK, for KKANs.
result Established robust error bounds that are distance-aware, improving efficiency and scalability.
Robust variable selection for high-dimensional data with missing and measurement errors.
problem Missing data and measurement errors confound data distribution.
method Exponential loss function with inverse probability weighting and additive error models.
result The Atan punishment method improves robust variable selection.
We approximate derivatives of functions on manifolds by embedding them and applying vector-valued operators.
problem Derivatives of manifold-valued functions are harder to approximate than vector-valued functions.
method Embed the manifold into a higher space, approximate the derivative of the vector-valued function, and project back.
result We provide error bounds for the approximation of manifold-valued function derivatives.
Unified framework for estimating random forest prediction errors.
problem Estimating prediction errors for random forests.
method Novel estimator of conditional prediction error distribution function.
result Proposed estimators enable competitive prediction intervals.
Data-aware activation function customization reduces neural network error.
problem Current neural networks lack consideration for specific activation functions.
method Linear algebraic explanation and Diaconis-Shahshahani Approximation Theorem criteria for activation functions.
result Using an even activation function like seagull can reduce neural network error by orders of magnitude.
New dual formulation reduces generalization error for ERM-fDR.
problem Generalization error in constrained optimization problems.
method Introduces a dual formulation of ERM-fDR using Legendre-Fenchel transform and implicit function theorem.
result Explicit characterizations of generalization error for algorithms under mild conditions.
New method uses extreme value theory to estimate neural network errors.
problem Quantifying the error of neural networks, especially for large values.
method Applying extreme value theory to approximate the distribution of error.
result Developed a new estimator for the shape parameter of the Pareto distribution.
New method calibrates asynchronous, error-prone covariates for longitudinal data.
problem Estimation biases and slow convergence in analyzing time-varying covariates with measurement error.
method Functional calibration approach based on functional principal component analysis.
result Asymptotically unbiased and consistent estimators for time-invariant coefficients; optimal convergence rate for time-varying coefficients.
Paper studies offline RL with linear approx, focusing on inherent Bellman error.
problem Offline RL with linear approx, focusing on inherent Bellman error.
method Algorithm that succeeds under single-policy coverage condition, leveraging inherent Bellman error.
result Algorithm yields first known guarantee under single-policy coverage, even for linear Bellman completeness.
Next generation deep neural networks for classification hosted on embedded platforms will rely on fast, efficient, and accurate learning algorithms. Initialization of weights in learning networks has a great impact on the classification accuracy. In this paper we focus on deriving good initial weights by modeling the e…
The paper explores how neural networks learn logical functions and their generalization error.
problem Learning logical functions with neural networks and understanding generalization error.
method Gradient descent on neural networks, analyzing noise-stability and Boolean influence.
result Gradient descent on certain neural architectures tends to favor low-degree representations, impacting generalization error.
Novel Hilbert space Gaussian process improves sequential design accuracy and efficiency.
problem Efficiently implementing Gaussian process acquisition functions for expensive simulations.
method Proposed a truncated eigenbasis representation for closed-form evaluation of IMSE acquisition function.
result Significantly lower prediction error and reduced computation time compared to benchmarks.
New algorithm learns value and advantage functions for continuous-time Markov processes without structural assumptions.
problem Learning value and advantage functions for continuous-time Markov processes without structural assumptions.
method Proposes Sobolev-prox fitted q-learning algorithm based on Hilbert-space positive definiteness and boundedness properties of Bellman operators. result Identifies ellipticity as a key structural property enabling reinforcement learning for Markov diffusions.
Consider the problem of minimizing functions that are Lipschitz and strongly convex, but not necessarily differentiable. We prove that after T steps of stochastic gradient descent, the error of the final iterate is O(log(T)/T) with high probability. We also construct a function from this class for which the error …
In this paper, we obtain asymptotic formulas with error estimates for the implied volatility associated with a European call pricing function. We show that these formulas imply Lee's moment formulas for the implied volatility and the tail-wing formulas due to Benaim and Friz. In addition, we analyze Pareto-type tails o…
The paper improves error bounds for Bayesian quadrature in noisy settings.
problem Improving error bounds for Bayesian quadrature in noisy settings.
method Develops a two-step meta-algorithm to relate average-case quadrature error to L2-function approximation error. result Provides new average-case results for various kernels and noise settings.
Improved numerical solution for BSDEs with reduced boundary errors.
problem Boundary errors in numerical solution of BSDEs.
method Modified damping and shifting schemes to transform target function into a bounded periodic function, applying Fourier transforms.
result Significant reduction in boundary errors with improved accuracy and convergence.
We carefully study how well minimizing convex surrogate loss functions, corresponds to minimizing the misclassification error rate for the problem of binary classification with linear predictors. In particular, we show that amongst all convex surrogate losses, the hinge loss gives essentially the best possible bound, o…
Paper introduces a new cost function to improve deep learning model generalization.
problem Overfitting and poor extrapolation of deep learning models.
method Introduces a 'whitening' cost function based on the Ljung-Box statistic.
result Significant improvement in generalization for RNNs and image autoencoders.
The paper explores MAE as a loss function for DNN vector-to-vector regression, proving its advantages over MSE.
problem Improving loss function for deep neural network based vector-to-vector regression.
method Presenting performance bounds and new properties of MAE, deriving generalized upper bounds, and interpreting MAE as a Laplacian distribution.
result MAE is a more suitable loss function than MSE for DNN based vector-to-vector regression, especially when errors follow a Laplacian distribution.
The paper analyzes the probabilistic structure of DDPMs and bounds their sampling error.
problem Understanding and controlling errors in discrete-time DDPMs.
method Structural analysis of score functions, Schrödinger's problem, and FBSDEs.
result Explicit upper bound for total variation distance between sampling and target distributions.
Error bounds, which refer to inequalities that bound the distance of vectors in a test set to a given set by a residual function, have proven to be extremely useful in analyzing the convergence rates of a host of iterative methods for solving optimization problems. In this paper, we present a new framework for establis…
A new algorithm for K-means clustering in evolving data streams.
problem Clustering of continuously arriving data in streaming scenarios with concept drift.
method Formal definition of Streaming K-means, surrogate error function, algorithm for minimizing surrogate error. result The surrogate error function effectively approximates the Streaming K-means error. New Bellman error estimator improves offline model selection performance.
problem Selecting the best policy from logged data using mean squared Bellman error.
method Developed a more accurate estimator of MSBE and analyzed conditions for successful OMS.
result New estimator achieves impressive offline model selection performance on diverse tasks.
Neural networks can approximate functionals on RKHS with error bounds.
problem Approximating functionals on RKHS using neural networks.
method Interpolating orthogonal projections in RKHS using point evaluations.
result Explicit error bounds for various kernels (inverse multiquadric, Gaussian, Sobolev).
In this paper we study the consistency of an empirical minimum error entropy (MEE) algorithm in a regression setting. We introduce two types of consistency. The error entropy consistency, which requires the error entropy of the learned function to approximate the minimum error entropy, is shown to be always true if the…
Paper analyzes error in stochastic approximation for discontinuous functions.
problem Estimating expected error in discontinuous stochastic approximation.
method Uses finite differences and O(n−1/5) error estimate for discontinuous functions. result Achieves error estimate of O(n−1/5) for discontinuous stochastic representation. OTS error shows small training error doesn't guarantee small test error.
problem The relationship between training and test errors is unclear.
method An analysis of the conditions under which small training set error guarantees small OTS error.
result The theorem is limited to models with distinct training and test distributions.
We predict generalization error across model and dataset sizes.
problem Understanding the dependency of neural network generalization error on model and dataset size.
method Model scaling concept applied to construct a functional form of generalization error.
result The constructed functional form accurately predicts generalization error across scales.
Polynomial error equidistribution for SL2 groups.
problem Equidistribution of orbits in arithmetic quotients.
method Margulis function, incidence geometry, spectral gap.
result Polynomial error rate for equidistribution.