Sharp bounds on uniform generalization errors in binary linear classification.
problem Understanding the uniform generalization errors in binary linear classification.
method Isoperimetric arguments, Poincaré and log-Sobolev inequalities for joint distributions.
result Sharp concentration bounds on uniform generalization errors, almost sure convergence in broad settings.
Study shows gap between uniform convergence and test error in random feature models.
problem Understanding the gap between uniform convergence and test error in random feature models.
method Analytical expressions for uniform convergence over norm balls, interpolators, and minimum norm interpolator risk derived and proved.
result Uniform convergence over interpolators still gives a non-trivial bound of test error even when classical uniform convergence is vacuous.
Berestovskii and Plaut introduced the concept of a coverable uniform space when developing their theory of generalized universal covering maps for uniform spaces. Brodskiy, Dydak, LaBuz, and Mitra introduced the concept of a locally uniformly joinable uniform space when developing their theory of generalized uniform co…
The paper explores why a specific type of predictor works well in noisy data.
problem Understanding why a specific type of predictor (minimum-norm interpolator) works well in noisy data.
method The paper uses uniform convergence and zero-error predictors in a norm ball to explain the success of the minimum-norm interpolator.
result The minimum-norm interpolator is consistent, and this can be explained by uniform convergence of zero-error predictors in a norm ball.
New bounds show multicalibration error is close to prediction error.
problem Addressing fairness in machine learning systems.
method Sample complexity bounds for uniform convergence of multicalibration error.
result Uniform convergence guarantees for multicalibration error, independent of prediction error.
Study problem-dependent rates in statistical learning theory, achieving optimal generalization error bounds.
problem Generalization error in statistical learning theory.
method Uniform localized convergence framework.
result Optimal generalization error bounds for various learning problems.
Study uniform consistency in nonparametric mixture models and mixed regression.
problem Uniform consistency in nonparametric mixture models and mixed regression models.
method Construct uniformly consistent estimators under general conditions, develop novel technical tools.
result Prove uniform consistency results for nonparametric mixtures and mixed regression models.
Data-driven models are subject to model errors due to limited and noisy training data. Key to the application of such models in safety-critical domains is the quantification of their model error. Gaussian processes provide such a measure and uniform error bounds have been derived, which allow safe control based on thes…
Improved error estimate for SGLD sampling algorithm.
problem Establishing a precise error bound for SGLD.
method Sharp uniform-in-time error estimate for SGLD under mild assumptions.
result Uniform-in-time O(η2) bound for KL-divergence between SGLD and Langevin diffusion. Uniform bounds for neural networks' generalization error in overparameterized settings.
problem Generalization error in overparameterized neural networks.
method Neural Tangent kernel theory and Mercer decomposition of the NT kernel in spherical harmonics.
result Uniform generalization bounds for overparameterized neural networks in RKHS.
The paper tackles learning from non-irreducible Markov chains, proving learnability and generalization bounds.
problem Learning from temporal dependent data with non-irreducible Markov chains.
method Uniform convergence and generalization bounds for sample error under uniform ergodicity.
result Learnability and generalization bounds for approximate sample error minimization algorithm.
The paper provides a uniform convergence bound for smooth calibration error and its relationship with functional gradient.
problem Limited theoretical understanding of learning algorithms achieving high accuracy and good calibration.
method Focuses on smooth calibration error, providing a uniform convergence bound and proving the relationship with functional gradient.
result Derives conditions for simultaneous classification and calibration guarantees in gradient boosting trees, kernel boosting, and neural networks.
New bounds for SGLD show error decreases with more data.
problem Establishing generalization error bounds for SGLD in non-convex settings.
method Using dissipativity, smoothness, and uniform stability, time-independent bounds are derived.
result Error bounds decay to zero as sample size increases.
Unified error analysis for discrete flow models.
problem Error analysis of discrete flow models.
method Stochastic calculus theory, Girsanov theorem, generator matching, uniformization.
result First error analysis for discrete flow models.
Deep ReLU networks generalize well with few parameters.
problem Generalization of overparametrized deep neural networks.
method Explicit bounds on test error independent of overparametrization and VC dimension.
result Generalization error is independent of network architecture and overparametrization.
New algorithms achieve uniform stability for empirical risk minimization.
problem Designing uniformly stable optimization algorithms for empirical risk minimization.
method Black-box conversion of smooth optimization algorithms and development of Mirror Descent for smooth optimization.
result Optimal algorithms with uniform stability and convergence rates for smooth optimization.
We show that the sets in a family with finite VC dimension can be uniformly approximated within a given error by a finite partition. Immediate corollaries include the fact that VC classes have finite bracketing numbers, satisfy uniform laws of averages under strong dependence, and exhibit uniform mixing. Our results ar…
The paper provides guarantees for feedback control with sensor errors.
problem Certifying performance and safety in feedback control systems with sensor errors.
method Solving a supervised learning problem to characterize sensor errors and providing uniform error bounds.
result Finite-time convergence rate on sub-optimality of using a regressor in closed-loop for waypoint tracking.
This paper addresses error bounds and posterior variance for Gaussian process regression.
problem Deriving performance guarantees for Gaussian process regression without prior knowledge.
method Lipschitz continuity and analysis of posterior variance function.
result Uniform error bounds for Gaussian process regression are derived.
This study uses neural networks to approximate Bayesian filtering problems.
problem Estimating latent time-series signal statistics from observation sequences.
method Formulated a generic recurrent neural network framework to learn recursive mappings directly.
result Approximation error bounds for filtering in non-compact domains and strong time-uniform bounds.
No algorithm outperforms uniform sampling in A/B testing.
problem Identifying the best arm in A/B testing with fixed budget.
method Introducing consistent and stable algorithms, deriving lower bounds, and proving optimality of uniform sampling.
result No algorithm performs better than uniform sampling in A/B testing.
Paper analyzes ECE bias and provides bounds for its estimation.
problem Understanding the estimation bias in ECE for machine learning models.
method Information-theoretic approach to analyze bias in uniform mass and uniform width binning strategies.
result Established upper bounds on ECE estimation bias and optimal number of bins.
The paper offers error bounds for quantized dynamical models.
problem Accuracy of dynamical models from dependent data sequences.
method Developed uniform error bounds for quantized models and imperfect optimization algorithms.
result Unified bounds for slow and fast rates, scaling with model encoding bits.
We propose to study the generalization error of a learned predictor h^ in terms of that of a surrogate (potentially randomized) predictor that is coupled to h^ and designed to trade empirical risk for control of generalization error. In the case where h^ interpolates the data, it is interesting to con…
This letter presents an improved version of diffusion least mean ppower (LMP) algorithm for distributed estimation. Instead of sum of mean square errors, a weighted sum of mean square error is defined as the cost function for global and local cost functions of a network of sensors. The weight coefficients are updated b…
The method to derive uniform bounds with Gaussian and Rademacher complexities is extended to the case where the sample average is replaced by a nonlinear statistic. Tight bounds are obtained for U-statistics, smoothened L-statistics and error functionals of l2-regularized algorithms.
New loss function equivalence reveals PER's uniform sampling can be improved.
problem Improving Prioritized Experience Replay (PER) for better learning efficiency.
method Transforming non-uniformly sampled data loss functions into uniformly sampled ones.
result Some environments can replace PER with a new loss function without performance loss.
Continuous-time PCD for MLE with explicit error bounds.
problem Maximum likelihood estimation of unnormalised densities.
method Continuous-time formulation as coupled SDEs, deriving UiT bounds.
result Explicit error bounds between PCD iterates and MLE solution.
Aimed at explaining the surprisingly good generalization behavior of overparameterized deep networks, recent works have developed a variety of generalization bounds for deep learning, all based on the fundamental learning-theoretic technique of uniform convergence. While it is well-known that many of these existing bou…
A new loss function HUG decouples and generalizes neural collapse.
problem Neural collapse limits in deep learning models.
method Hyperspherical uniformity gap (HUG) as a unified framework.
result HUG decouples and generalizes neural collapse, improving model flexibility and robustness.
Develops UKP for comparing feature representations in multitask learning.
problem Comparing feature representations learned by different models without access to test data.
method Uniform Kernel Prober (UKP) for comparing representations in kernel ridge regression tasks.
result UKP provides a uniform measure of prediction error on test data without access to test data.
We study discrete curvatures computed from nets of curvature lines on a given smooth surface, and prove their uniform convergence to smooth principal curvatures. We provide explicit error bounds, with constants depending only on properties of the smooth limit surface and the shape regularity of the discrete net.
Recently, artificial neural networks (ANNs) in conjunction with stochastic gradient descent optimization methods have been employed to approximately compute solutions of possibly rather high-dimensional partial differential equations (PDEs). Very recently, there have also been a number of rigorous mathematical results …
Algorithm-dependent generalization error bounds are central to statistical learning theory. A learning algorithm may use a large hypothesis space, but the limited number of iterations controls its model capacity and generalization error. The impacts of stochastic gradient methods on generalization error for non-convex …
Deep learning generalizes well despite being overparameterized.
problem Why deep networks generalize well despite fitting training data perfectly.
method Empirical study of training methods and derivation of data-dependent generalization bounds.
result Uniform convergence alone is insufficient for explaining generalization in overparameterized settings.
The paper develops bounds for predictive values in binary classification.
problem Lack of confidence intervals for positive and negative predictive values.
method Bi-criterion framework and distribution-free large deviation and uniform convergence bounds.
result New bounds for predictive values without relying on concentration inequalities.
We explore in some detail the notion of algorithmic stability as a viable framework for analyzing the generalization error of learning algorithms. We introduce the new notion of training stability of a learning algorithm and show that, in a general setting, it is sufficient for good bounds on generalization error. In t…
Uniform convergence of interpolators proven for Gaussian data.
problem Interpolation learning in high-dimensional linear regression with Gaussian data.
method Generic uniform convergence guarantee in terms of Gaussian width.
result Consistency of interpolators for minimum-norm and near-minimal-norm cases.
Learning ReLU networks to high uniform accuracy requires exponentially many samples.
problem Achieving high uniform accuracy on ReLU networks for security-critical applications.
method Quantified the number of training samples needed for any algorithm to guarantee uniform accuracy.
result The minimal number of training samples scales exponentially with network depth and input dimension.
This paper has been withdrawn by the author due to an error in an inequality in the proof of Theorem 1.1.
Let Γ<SL2(Z) be a non-elementary finitely generated subgroup and let Γ(q) be its congruence subgroup of level q for each q∈N. We obtain an asymptotic formula for the matrix coefficients of L2(Γ(q)\SL2(R)) with a {\it uniform} exponential error term…
Higher-order motif structures and multi-vertex interactions are becoming increasingly important in studies that aim to improve our understanding of functionalities and evolution patterns of networks. To elucidate the role of higher-order structures in community detection problems over complex networks, we introduce the…
Unified framework for uniform signal recovery in nonlinear GCS with 1-bit/quantized measurements.
problem Uniform recovery guarantees for nonlinear generative compressed sensing.
method Unified framework using generalized Lasso and Lipschitz approximation.
result Uniform recovery of all signals in the ball up to an error of ε using approximately O(k/ε^2) samples.
New bounds for learning polynomial surrogates with L∞ guarantees.
problem Learning polynomial surrogates for bounded binary functions with L∞ error guarantees. method Characterized minimax sample complexity for two classes of polynomials under subgaussian noise.
result Sample complexity rates differ from noiseless case, scaling as nd+1 for degree d polynomials and ns2 for sparse polynomials. SS-GEN simulates rare events in heavy and light-tailed data.
problem Estimating probabilities of extreme events in multivariate data.
method Self-Similar Generative Estimation (SS-GEN) decomposes tail distribution into radial and angular components.
result SS-GEN generates representative extreme scenarios and estimates rare-event probabilities beyond observed data.
SGD-trained deep nets have bounds on their generalization error.
problem Bounding generalization error for deep neural networks trained by SGD.
method Combining dynamical control of parameter norms and Rademacher complexity estimates.
result Explicit bounds depend on loss trajectory, work for various architectures.
Wide residual networks generalize well with uniform convergence to RNTK as width increases.
problem Understanding the generalization ability of wide residual networks.
method Uniform convergence of residual network kernel to residual neural tangent kernel (RNTK).
result Generalization error converges to kernel regression error with respect to RNTK.
Analytic networks with bounded coefficients can't outperform polynomial approximations.
problem Approximation limits of neural networks with analytic activation functions under coefficient constraints.
method Deterministic analysis using comparison argument and Bernstein-type estimates.
result Networks with analytic activation functions and controlled coefficients cannot outperform classical polynomial approximation rates on non-analytic targets.