New bounds on machine learning model generalization error moments.
problem Understanding the performance of machine learning models.
method Information-theoretic bounds on the moments of the generalization error of learning algorithms.
result Proposed bounds on generalization error moments and their high-probability bounds.
New bound on machine learning model performance using Jensen-Shannon information.
problem Understanding the performance of machine learning models.
method Proposes a new information-theoretic bound on generalization error.
result Shows that the new bound can be tighter than mutual information-based bounds under certain conditions.
PAC-Bayesian bounds for stochastic LTI systems derived.
problem Error bounds for stochastic LTI systems.
method PAC-Bayesian theory applied to autonomous stochastic LTI models.
result Error bounds for stochastic LTI systems derived.
In this paper, we improve the PAC-Bayesian error bound for linear regression derived in Germain et al. [10]. The improvements are twofold. First, the proposed error bound is tighter, and converges to the generalization loss with a well-chosen temperature parameter. Second, the error bound also holds for training data t…
New bounds tighten the generalization error of Gibbs algorithm.
problem Bounding the generalization error of Gibbs algorithm.
method Characterization of generalization error in terms of symmetrized KL information.
result Exact characterization of Gibbs algorithm's expected generalization error.
New bound matches exact generalization error for quadratic Gaussian problem.
problem Understanding generalization error in quadratic Gaussian problems.
method Information-theoretic approach with new ingredients.
result Exact tight bound for generalization error.
The study examines how equivariance in networks affects generalization error using PAC-Bayesian bounds.
problem Understanding how equivariance in networks impacts generalization error.
method Utilized PAC-Bayesian analysis for equivariant networks, deriving norm-based bounds for generalization error.
result The bound indicates that using larger group size in the model improves generalization error.
New bounds study class-specific generalization error in machine learning.
problem Existing generalization theories assume uniform class performance, but in practice, classes vary significantly.
method Developed novel information-theoretic bounds using KL divergence and CMI.
result Theoretical bounds accurately capture complex class-generalization error behavior.
Proposes a new bound on generalization error using conditional mutual information.
problem Improving the generalization error bound in machine learning.
method Combines error decomposition and conditional mutual information techniques.
result New bound is order-wise better than previous ones in a simple Gaussian setting.
Paper improves risk bounds for nonconvex-strongly-concave minimax problems.
problem Achieving sharper risk bounds for nonconvex-strongly-concave minimax problems.
method Using uniform localized convergence to derive high probability generalization error bounds.
result Derives n times faster excess primal risk bounds for popular algorithms.
New bounds on generalization error using information density moments.
problem Bounding the generalization error of randomized learning algorithms.
method Derives bounds on average and tail probabilities of generalization error using mth central moments of the information density.
result Explicit bounds on generalization error are derived, showing better dependence on confidence level with higher-order information density moments.
This article studies the achievable guarantees on the error rates of certain learning algorithms, with particular focus on refining logarithmic factors. Many of the results are based on a general technique for obtaining bounds on the error rates of sample-consistent classifiers with monotonic error regions, in the real…
The paper bounds generalization error for iterative learning with bounded updates.
problem Generalization error of iterative learning algorithms with bounded updates for non-convex loss functions.
method Information-theoretic techniques, reformulating mutual information as update uncertainty, variance decomposition.
result Improved generalization error bounds for iterative learning algorithms with bounded updates.
The success of deep learning has led to a rising interest in the generalization property of the stochastic gradient descent (SGD) method, and stability is one popular approach to study it. Existing works based on stability have studied nonconvex loss functions, but only considered the generalization error of the SGD in…
Paper addresses generalization error bounds for learning with censored feedback.
problem Impact of censored feedback on generalization error bounds.
method Derives an extension of DKW inequality for non-IID data due to censored feedback and uses it to bound generalization error.
result Existing generalization error bounds fail to account for censored feedback, necessitating new bounds.
New error bounds for flow matching methods using deterministic sampling.
problem Improving the accuracy of flow matching methods for generating probability distributions.
method Derived error bounds for flow matching methods under deterministic sampling conditions.
result Presented error bounds for flow matching methods using L 2 L^2 L 2 loss and regularity conditions. Paper analyzes Gibbs and Langevin Monte Carlo for interpolation regime, showing generalization from low errors.
problem Analyzing Gibbs and Langevin Monte Carlo in overparameterized interpolation regime.
method Data-dependent bounds and stability under approximation with Langevin Monte Carlo.
result Generalization is signaled by small training errors in noisy regime, with bounds stable under approximation.
An information-theoretic upper bound on the generalization error of supervised learning algorithms is derived. The bound is constructed in terms of the mutual information between each individual training sample and the output of the learning algorithm. The bound is derived under more general conditions on the loss func…
Paper introduces new bounds linking data compressibility to generalization error.
problem Establishing data-dependent generalization bounds.
method Variable-size compressibility framework linking generalization error to compression rate of input data.
result New bounds depend on empirical data measure, subsuming existing PAC-Bayes and intrinsic dimension bounds.
The paper bounds generalization errors for deep neural networks with Markov datasets.
problem Bounding generalization errors for deep learning with Markov datasets.
method Developed new symmetrization inequalities for Markov chains, using spectral gap of the infinitesimal generator.
result Derived upper bounds on generalization errors for deep neural networks with Markov datasets.
Establishes upper bounds on generalization error in active learning.
problem Improving query algorithms in active learning.
method Derives upper bounds on generalization error using informativeness and representativeness query strategies.
result Validates the use of regularization techniques to ensure bounds' validity.
SGD-trained deep nets have bounds on their generalization error.
problem Bounding generalization error for deep neural networks trained by SGD.
method Combining dynamical control of parameter norms and Rademacher complexity estimates.
result Explicit bounds depend on loss trajectory, work for various architectures.
This paper examines error bounds for deep learning classifiers with noisy labels.
problem Understanding the performance of classifiers trained on noisy data.
method Derives error bounds for excess risk, decomposing it into statistical and approximation errors. Uses independent block construction for statistical dependencies and vector-valued setting for approximation error.
result Established theoretical results for error bounds in deep learning with noisy labels, mitigating the impact of high-dimensional input spaces.
GANs learn distributions well from samples, with rates depending on intrinsic dimension.
problem Learning distributions from samples using GANs.
method Oracle inequality, Hölder functions approximation, neural network approximation, integral probability metrics.
result Convergence rates of GANs depend on intrinsic dimension, not ambient dimension.
The Gibbs algorithm's generalization error is bounded, improving with prior volume in low temperatures.
problem Bounding the generalization error of the Gibbs algorithm in low temperature regimes.
method Analyzes the Gibbs algorithm's performance, extending known high-temperature bounds to low-temperature scenarios.
result With high probability, the generalization error decreases with the total prior volume of similar hypotheses.
Study bounds graph neural networks' over-parameterized error.
problem Understanding graph neural networks' performance in over-parameterized regimes.
method Developed mean-field regime bounds for graph convolutional and message passing neural networks.
result Established upper bounds with a convergence rate of O ( 1 / n ) O(1/n) O ( 1/ n ) for generalization error. Upper bounds for CV errors apply to lasso and other models.
problem Bounding CV errors for lasso and similar models.
method Rademacher complexity and Orlicz- Ψ ν Ψ_ν Ψ ν norm. result Upper bounds are tight and stable for lasso.
This paper tightens information-theoretic bounds on generalization errors.
problem Understanding the discrepancy between training and testing data losses.
method Investigates the tightness of information-theoretic bounds on generalization error.
result The individual sample mutual information bound can be asymptotically tight under specific assumptions.
In statistical learning theory, generalization error is used to quantify the degree to which a supervised machine learning algorithm may overfit to training data. Recent work [Xu and Raginsky (2017)] has established a bound on the generalization error of empirical risk minimization based on the mutual information $I(S;…
In this work, we present a novel upper bound of target error to address the problem for unsupervised domain adaptation. Recent studies reveal that a deep neural network can learn transferable features which generalize well to novel tasks. Furthermore, a theory proposed by Ben-David et al. (2010) provides a upper bound …
Sharp bounds on uniform generalization errors in binary linear classification.
problem Understanding the uniform generalization errors in binary linear classification.
method Isoperimetric arguments, Poincaré and log-Sobolev inequalities for joint distributions.
result Sharp concentration bounds on uniform generalization errors, almost sure convergence in broad settings.
In this paper, we obtain generic bounds on the variances of estimation and prediction errors in time series analysis via an information-theoretic approach. It is seen in general that the error bounds are determined by the conditional entropy of the data point to be estimated or predicted given the side information or p…
The paper bounds the mean absolute error in DNN vector-to-vector regression.
problem Bounding the mean absolute error in deep neural network based vector-to-vector regression.
method Error decomposition techniques in statistical learning theory and non-convex optimization theory were used to derive upper bounds for approximation, estimation, and optimization errors.
result Theoretical upper bounds for mean absolute error in DNN vector-to-vector regression were derived and validated experimentally.
Unified bounds for random subset generalization error and improved SGD Langevin dynamics.
problem Generalization error bounds for random subsets and stochastic gradient Langevin dynamics.
method Unified framework based on Hellström and Durisi's work, extending bounds for Langevin dynamics.
result Unified and refined bounds for generalization error in stochastic gradient Langevin dynamics.
This work bounds the generalization error of private algorithms for discrete data.
problem Bounding the generalization error of private algorithms for discrete data.
method Information-theoretic approach using relative entropy and the method of types.
result Explicit upper bounds on the generalization error of stable private algorithms for discrete data.
New method improves understanding of machine learning model performance.
problem Understanding how well machine learning models generalize from training data to unseen data.
method Auxiliary Distribution Method to derive new generalization error bounds.
result Upper bounds on generalization errors are tighter and more applicable.
New bounds for neural networks without loss boundedness assumption.
problem Generalization error bounds for two-layer neural networks.
method Wasserstein distance estimates and moment bounds for stochastic gradient method.
result Dimension-free rate of order O ( n − 1 / 2 ) O(n^{-1/2}) O ( n − 1/2 ) for independent test data. Generalization error (also known as the out-of-sample error) measures how well the hypothesis learned from training data generalizes to previously unseen data. Proving tight generalization error bounds is a central question in statistical learning theory. In this paper, we obtain generalization error bounds for learnin…
This research evaluates generalization measures in deep learning.
problem Understanding why deep learning models generalize well despite small training error.
method Empirical evaluation of generalization bounds and measures.
result Generalization measures should be evaluated using distributional robustness.
New method preserves unitarity for Schrödinger equation learning, reducing errors and improving time generalization.
problem Learning the evolution operator for time-dependent Schrödinger equation with varying Hamiltonians.
method Linear estimator preserving weak unitarity, with theoretical error bounds and time generalization.
result Achieves up to two orders of magnitude smaller relative errors than existing methods.
This work tightens generalization error bounds using Wasserstein distance.
problem Improving expected generalization error bounds in machine learning.
method Introduces bounds based on Wasserstein distance for various settings.
result New, tighter bounds based on relative entropy and other information measures.
Study shows infoGAN's generalization error bound for two-layer networks.
problem Understanding generalization error in infoGAN for two-layer neural networks.
method Analyzes the difference between empirical and population objective functions, derives Rademacher complexity bounds.
result Derives error bound for infoGAN's generalization error in a two-layer network.
Study error bounds in evaluating distributional computational graphs.
problem Error analysis in evaluating graphs with inputs as probability distributions.
method Establish non-asymptotic error bounds using Wasserstein-1 distance.
result Non-asymptotic error bounds for discretization errors in distributional computational graphs.
Improved generalization bounds for SGD in non-convex learning.
problem Understanding generalization properties of SGD in non-convex settings.
method Introducing Type II perturbed SGD (T2pm-SGD) to analyze generalization error bounds.
result Tighter generalization error bounds for SGD in non-convex learning, especially for sub-Gaussian and bounded loss functions.
Study shows data heterogeneity affects distributed learning's generalization error.
problem Effect of data heterogeneity on distributed learning performance.
method Established bounds on generalization error using information-theoretic rate-distortion theory.
result Data heterogeneity improves generalization error for distributed learning.
The paper develops generalization bounds for deep compound Gaussian neural networks.
problem Developing theoretical guarantees for the performance of deep neural networks.
method Novel generalization error bounds using a compound Gaussian prior and Dudley's integral.
result Theoretical bounds show generalization error scales O ( n ln ( n ) ) \mathcal{O}(n\sqrt{\ln(n)}) O ( n ln ( n ) ) in signal dimension and O ( ( N e t w o r k S i z e ) 3 / 2 ) \mathcal{O}((Network Size)^{3/2}) O (( N e tw or k S i z e ) 3/2 ) in network size. This paper tackles worst-class error rate in classification tasks.
problem Minimizing worst-class error rate in classification tasks, especially in medical image classification.
method Designing a boosting approach to bound the worst-class error rate using Deep Neural Networks (DNNs).
result The proposed boosting approach lowers worst-class test error rates while avoiding overfitting.
Full-batch GD achieves generalization close to any stationary point with fewer assumptions.
problem Generalization and excess risk bounds for smooth losses, including non-Lipschitz and nonconvex cases.
method Path-dependent analysis of GD's generalization error, focusing on optimization error and stability.
result Generalization error is tightly bound in terms of optimization error and iteration count, bypassing common assumptions.