New bounds on machine learning model generalization error moments.
problem Understanding the performance of machine learning models.
method Information-theoretic bounds on the moments of the generalization error of learning algorithms.
result Proposed bounds on generalization error moments and their high-probability bounds.
GANs learn distributions well from samples, with rates depending on intrinsic dimension.
problem Learning distributions from samples using GANs.
method Oracle inequality, Hölder functions approximation, neural network approximation, integral probability metrics.
result Convergence rates of GANs depend on intrinsic dimension, not ambient dimension.
New bounds tighten the generalization error of Gibbs algorithm.
problem Bounding the generalization error of Gibbs algorithm.
method Characterization of generalization error in terms of symmetrized KL information.
result Exact characterization of Gibbs algorithm's expected generalization error.
Study loop corrections in random feature models affecting training and test errors.
problem Analyzing loop corrections in random feature models to understand training and test errors.
method Statistical physics and effective field theory approach to study loop corrections.
result Derived loop corrections to training error, test error, and generalization gap.
Efficient classifier error estimation without re-training.
problem Estimating classifier error without re-training.
method Generalized resubstitution based on empirical measures.
result Consistent and asymptotically unbiased error estimation.
Study shows how classifiers can approach Bayes error in high-dimensional settings.
problem Generalization error in high-dimensional perceptrons.
method Proved a formula for generalization error using convex optimization and observed that logistic and hinge regression can approach Bayes error closely.
result Logistic and hinge regression can approach Bayes-optimal generalization error closely in high-dimensional settings.
Confidence measures for the generalization error are crucial when small training samples are used to construct classifiers. A common approach is to estimate the generalization error by resampling and then assume the resampled estimator follows a known distribution to form a confidence set [Kohavi 1995, Martin 1996,Yang…
Full-batch GD achieves generalization close to any stationary point with fewer assumptions.
problem Generalization and excess risk bounds for smooth losses, including non-Lipschitz and nonconvex cases.
method Path-dependent analysis of GD's generalization error, focusing on optimization error and stability.
result Generalization error is tightly bound in terms of optimization error and iteration count, bypassing common assumptions.
The paper analyzes CycleGAN's error components for unpaired data generation.
problem Analyzing approximation and estimation errors in CycleGAN for unpaired data.
method Decomposes risk into approximation and estimation errors, analyzing each separately and considering their trade-offs.
result Theoretical insights into CycleGAN's performance through error analysis.
This research analyzes the error convergence rate of GAN models.
problem Understanding the error convergence rate of GAN models.
method Applying Talagrand inequality and Borel-Cantelli lemma to establish a tight convergence rate.
result Established a tight convergence rate for the error of GAN models.
Study shows infoGAN's generalization error bound for two-layer networks.
problem Understanding generalization error in infoGAN for two-layer neural networks.
method Analyzes the difference between empirical and population objective functions, derives Rademacher complexity bounds.
result Derives error bound for infoGAN's generalization error in a two-layer network.
Paper analyzes Gibbs and Langevin Monte Carlo for interpolation regime, showing generalization from low errors.
problem Analyzing Gibbs and Langevin Monte Carlo in overparameterized interpolation regime.
method Data-dependent bounds and stability under approximation with Langevin Monte Carlo.
result Generalization is signaled by small training errors in noisy regime, with bounds stable under approximation.
SGD reduces test error by decorrelating updates.
problem Improving generalization error in machine learning models.
method Derive a formula for generalization gap change due to SGD updates, compare to GD, and show decorrelation effect.
result SGD implicitly regularizes generalization error by decorrelating updates.
Random forest training yields confidence intervals for generalization error.
problem Computing accurate confidence intervals for random forest generalization error.
method Directly computes confidence intervals from training data without data splitting.
result Confidence intervals have good coverage and appropriate width.
First order discretizations of Langevin diffusion can achieve better generalization error with additional smoothness assumptions.
problem Analyzing generalization error for first order discretizations of Langevin diffusion.
method Providing a sufficient smoothness condition to show that first order methods can achieve arbitrarily runtime complexity for a given expected generalization error.
result First order methods can achieve arbitrarily runtime complexity with additional smoothness assumptions.
The success of deep learning has led to a rising interest in the generalization property of the stochastic gradient descent (SGD) method, and stability is one popular approach to study it. Existing works based on stability have studied nonconvex loss functions, but only considered the generalization error of the SGD in…
Randomly sampled interpolators achieve zero generalization error with enough data.
problem Understanding the high generalization ability of machine learning models.
method Algebraic geometry tools to prove zero generalization error for random interpolators.
result Generalization error of randomly sampled interpolators becomes zero once the number of training samples exceeds a geometric threshold.
New bounds study class-specific generalization error in machine learning.
problem Existing generalization theories assume uniform class performance, but in practice, classes vary significantly.
method Developed novel information-theoretic bounds using KL divergence and CMI.
result Theoretical bounds accurately capture complex class-generalization error behavior.
Paper calculates the exact error of LDA models.
problem Bayesian generalization error in Latent Dirichlet Allocation (LDA).
method Theoretical analysis of learning coefficient using algebraic geometry.
result Exact asymptotic form of LDA's generalization error.
Method estimates LLM error rates using Pareto optimization.
problem Quantifying error rates in text-generating models.
method Pareto optimization for generating risk scores.
result Risk scores correlate well with true error rates.
Modified training direction reduces generalization error in neural networks.
problem Reducing generalization error in neural networks.
method Theoretical analysis of modified natural gradient descent in function space.
result Modifying training direction in function space reduces total generalization error.
Unified error analysis for discrete flow models.
problem Error analysis of discrete flow models.
method Stochastic calculus theory, Girsanov theorem, generator matching, uniformization.
result First error analysis for discrete flow models.
A form of generalisation error known as Off Training Set (OTS) error was recently introduced in [Wolpert, 1996b], along with a theorem showing that small training set error does not guarantee small OTS error, unless assumptions are made about the target function. Here it is shown that the applicability of this theorem …
The paper examines prediction and estimation risks of ridgeless least squares under general error assumptions.
problem Prediction and estimation risks of ridgeless least squares under realistic error structures.
method Analysis of prediction and estimation risks under general regression error assumptions, including clustered or serial dependence.
result The benefits of overparameterization extend to time series, panel, and grouped data.
Analyzes generalization error in distributed linear regression.
problem Understanding generalization performance in distributed learning.
method Analytical characterization of generalization error in linear regression with distributed learning.
result Generalization error increases dramatically when nodes estimate close to the number of observations.
Study shows overparameterization helps in generalizing from smooth interpolants.
problem Understanding generalization in overparameterized linear models.
method Analysis of random Fourier series model with weighted trigonometric interpolation.
result Weighted trigonometric interpolation leads to lower generalization error in overparameterized scenarios.
Study on distributed linear regression performance, focusing on generalization error.
problem Performance of distributed learning in large-scale linear regression.
method Statistical learning approach, focusing on generalization error.
result Generalization error of distributed solution can be higher than centralized solution.
Analyzes the generalization and training errors of the random feature model over time.
problem Understanding the temporal behavior of generalization and training errors in deep learning.
method Uses Cauchy complex integral representations and random matrix methods based on linear pencils.
result Analytical solution of the full time-evolution path of generalization and training errors.
The study examines how equivariance in networks affects generalization error using PAC-Bayesian bounds.
problem Understanding how equivariance in networks impacts generalization error.
method Utilized PAC-Bayesian analysis for equivariant networks, deriving norm-based bounds for generalization error.
result The bound indicates that using larger group size in the model improves generalization error.
Grammatical error correction, like other machine learning tasks, greatly benefits from large quantities of high quality training data, which is typically expensive to produce. While writing a program to automatically generate realistic grammatical errors would be difficult, one could learn the distribution of naturally…
The Gibbs algorithm's generalization error is bounded, improving with prior volume in low temperatures.
problem Bounding the generalization error of the Gibbs algorithm in low temperature regimes.
method Analyzes the Gibbs algorithm's performance, extending known high-temperature bounds to low-temperature scenarios.
result With high probability, the generalization error decreases with the total prior volume of similar hypotheses.
This paper presents the hierarchical generalized linear model (HGLM) for loss reserving in a non-life insurance company. Because in this case the error of prediction is expressed by a complex analytical formula, the error bootstrap estimator is proposed instead. Moreover, the bootstrap procedure is used to obtain full …
Paper discusses prediction errors for penalized regressions using GAMP and LOOCV.
problem Prediction accuracy of penalized regression models.
method Derives prediction error estimators using GAMP and LOOCV.
result Information criteria and LOOCV error estimators differ in large parameter regions.
Paper presents a method to reduce prediction variance of DNNs for unknown systems.
problem Uncertainty in DNN predictions due to high variance.
method Ensemble averaging of multiple DNN models trained independently.
result Reduction in variance of DNN predictions, improving reliability.
We explore in some detail the notion of algorithmic stability as a viable framework for analyzing the generalization error of learning algorithms. We introduce the new notion of training stability of a learning algorithm and show that, in a general setting, it is sufficient for good bounds on generalization error. In t…
Study on ridge regression in convolutional models shows double descent error behavior.
problem Understanding generalization and estimation error in over-parameterized convolutional models.
method Analysis of ridge estimators for convolutional linear models, derivation of exact error formulae.
result Ridge estimators exhibit double descent error behavior in high-dimensional convolutional models.
The paper connects three machine learning methods to reduce generalization errors.
problem Reducing generalization errors in machine learning models.
method Distributionally robust optimization, Bayesian methods, and regularization.
result Machine learning models can be characterized using distributional uncertainty and robustness measures.
New generalization concept considers distribution of errors, not just average error.
problem Classical generalization fails to capture distributional differences in classifier outputs.
method Formal conjectures about distributional generalization based on model architecture, training procedure, and data distribution.
result Distributional generalization can be expected in specific conditions, as evidenced by empirical results.
Sharp bounds on uniform generalization errors in binary linear classification.
problem Understanding the uniform generalization errors in binary linear classification.
method Isoperimetric arguments, Poincaré and log-Sobolev inequalities for joint distributions.
result Sharp concentration bounds on uniform generalization errors, almost sure convergence in broad settings.
This article studies the achievable guarantees on the error rates of certain learning algorithms, with particular focus on refining logarithmic factors. Many of the results are based on a general technique for obtaining bounds on the error rates of sample-consistent classifiers with monotonic error regions, in the real…
We focus on estimating \emph{a priori} generalization error of two-layer ReLU neural networks (NNs) trained by mean squared error, which only depends on initial parameters and the target function, through the following research line. We first estimate \emph{a priori} generalization error of finite-width two-layer ReLU …
This paper analyzes SGD with increasingly weighted averaging for optimization and generalization.
problem Improving optimization and generalization for non-strongly convex objectives.
method Comprehensive analysis of increasingly weighted averaging schemes for convex, strongly convex, and non-convex objectives.
result The weight α affects both optimization and generalization errors, revealing a trade-off. Enhances reinforcement learning uncertainty estimation with a generalized Gaussian error model.
problem Inaccurate error representations and compromised uncertainty estimation in conventional uncertainty-aware TD learning.
method Introduces a novel framework for generalized Gaussian error modeling in deep reinforcement learning, incorporating higher-order moments, particularly kurtosis, to improve uncertainty estimation and mitigation.
result Significant performance gains in policy gradient algorithms with the proposed framework.
Analysis of non-asymptotic estimation error and structured statistical recovery based on norm regularized regression, such as Lasso, needs to consider four aspects: the norm, the loss function, the design matrix, and the noise model. This paper presents generalizations of such estimation error analysis on all four aspe…
This paper introduces a new method for model selection and more generally hyperparameter selection in machine learning. Minimum description length (MDL) is an established method for model selection, which is however not directly aimed at minimizing generalization error, which is often the primary goal in machine learni…
Study bounds graph neural networks' over-parameterized error.
problem Understanding graph neural networks' performance in over-parameterized regimes.
method Developed mean-field regime bounds for graph convolutional and message passing neural networks.
result Established upper bounds with a convergence rate of O(1/n) for generalization error. It is generally difficult to make any statements about the expected prediction error in an univariate setting without further knowledge about how the data were generated. Recent work showed that knowledge about the real underlying causal structure of a data generation process has implications for various machine learni…
In this paper, we improve the PAC-Bayesian error bound for linear regression derived in Germain et al. [10]. The improvements are twofold. First, the proposed error bound is tighter, and converges to the generalization loss with a well-chosen temperature parameter. Second, the error bound also holds for training data t…