New bounds show limitations of sample-wise information-theoretic generalization.
problem Limitations of sample-wise information-theoretic generalization bounds.
method Analysis of existing bounds and derivation of new bounds.
result No sample-wise information-theoretic bounds exist for expected squared generalization gap.
New bounds improve generalization in learning scenarios.
problem Limitations of existing information-theoretic bounds in SCO problems.
method Sample-conditioned hypothesis stability and neighboring-hypothesis matrix.
result Sharper generalization guarantees in various learning scenarios.
New tighter bounds for learning algorithms from Steinke & Zakynthinou's supersample setting.
problem Improving generalization bounds for machine learning algorithms.
method Information-theoretic approach using projected loss and Rademacher sequence.
result The new bounds are tighter than previous information-theoretic bounds.
In three-dimensional computational topology, the theory of normal surfaces is a tool of great theoretical and practical significance. Although this theory typically leads to exponential time algorithms, very little is known about how these algorithms perform in "typical" scenarios, or how far the best known theoretical…
This work improves generalisation bounds using chaining and information theory.
problem Improving generalisation bounds for supervised learning algorithms.
method Developed a theoretical framework linking generalisation bounds to their chained counterparts, derived new bounds using Wasserstein distance.
result Chained generalisation bounds can be tighter than standard bounds, especially for concentrated hypothesis distributions.
Information-theoretic Bayesian regret bounds of Russo and Van Roy capture the dependence of regret on prior uncertainty. However, this dependence is through entropy, which can become arbitrarily large as the number of actions increases. We establish new bounds that depend instead on a notion of rate-distortion. Among o…
New bounds estimate learning algorithm performance using prediction information.
problem Estimating the performance of black-box learning algorithms.
method Information-theoretic bounds based on prediction information.
result Improved bounds applicable to deterministic algorithms and easier to estimate.
New research shows existing information-theoretic methods can't establish minimax rates for gradient descent in stochastic convex optimization.
problem Establishing minimax rates for gradient descent in stochastic convex optimization using information-theoretic methods.
method Examined several information-theoretic frameworks including input-output mutual information bounds, conditional mutual information bounds, PAC-Bayes bounds, and their variants.
result Proved that none of the examined information-theoretic frameworks can establish minimax rates for gradient descent in stochastic convex optimization.
Paper formalizes and analyzes a new bound for variational inference.
problem Lack of theoretical guarantees in variational algorithms.
method Introduces VR-IWAE bound, a generalization of IWAE.
result VR-IWAE bound leads to unbiased gradient estimators.
Study reveals mutual information is crucial for understanding algorithm performance in stochastic convex optimization.
problem Uncertainty in capturing the exceptional performance of learning algorithms using existing information-theoretic generalization bounds.
method Examined the relationship between mutual information and generalization in stochastic convex optimization.
result Mutual information is necessary for true risk minimization in stochastic convex optimization, indicating existing bounds fall short.
Paper analyzes ECE bias and provides bounds for its estimation.
problem Understanding the estimation bias in ECE for machine learning models.
method Information-theoretic approach to analyze bias in uniform mass and uniform width binning strategies.
result Established upper bounds on ECE estimation bias and optimal number of bins.
Novel bounds for SGLD show generalization error decreases with more samples.
problem Understanding the generalization error of SGLD in non-convex optimization.
method Information-theoretic approach focusing on Kullback-Leibler divergence and sub-exponential loss function.
result Time-independent generalization bounds for SGLD, independent of step size and number of iterations.
Unified framework for information-theoretic bounds on learning algorithms.
problem Deriving generalization bounds for learning algorithms.
method Probabilistic decorrelation lemma, symmetrization, couplings, chaining, Young's inequality.
result New upper bounds on generalization error in expectation and high probability.
New bounds on IDS for RL show how to balance computation and learning efficiency.
problem Understanding and optimizing information-directed sampling (IDS) for reinforcement learning.
method Developed novel information-theoretic tools to bound information ratio and cumulative information gain.
result Derived prior-free Bayesian regret bounds for IDS in tabular finite-horizon MDPs and improved computational efficiency.
The paper analyzes the generalizability of linear autoencoders and multivariate linear regression.
problem Limited theoretical understanding of linear autoencoders' performance.
method Proposes a PAC-Bayes bound for multivariate linear regression and shows LAEs as constrained models.
result The proposed PAC-Bayes bound is tight and correlates with practical metrics.
The paper sets information-theoretic lower bounds for neural networks' parameter recovery and excess risk.
problem Establishing sample complexity lower bounds for neural network parameters and excess risk.
method Using information-theoretic tools, the paper proves lower bounds by constructing a generative network.
result Proves information-theoretic lower bounds for exact parameter recovery and positive excess risk.
Unified framework improves meta-learning generalization bounds.
problem Limited sharpness of existing meta-generalization bounds.
method Unified information-theoretic derivation for single-step bounds.
result Unified bounds exhibit tighter scaling and computational advantages.
Optimizes SGLD noise structure for better generalization bounds.
problem Improving generalization bounds for large models trained with SGLD.
method Manipulates the noise structure in SGLD to optimize information-theoretical bounds.
result Optimal noise covariance is the square root of the expected gradient covariance under certain constraints.
New analysis improves generalization bounds for meta-learning.
problem Improving generalization in meta-learning algorithms.
method Information-theoretic analysis of MAML and its stochastic variant.
result Data-dependent generalization bound is tighter and non-vacuous.
We integrate information-theoretic concepts into the design and analysis of optimistic algorithms and Thompson sampling. By making a connection between information-theoretic quantities and confidence bounds, we obtain results that relate the per-period performance of the agent with its information gain about the enviro…
In this paper, we derive generic bounds on the maximum deviations in prediction errors for sequential prediction via an information-theoretic approach. The fundamental bounds are shown to depend only on the conditional entropy of the data point to be predicted given the previous data points. In the asymptotic case, the…
New bounds study class-specific generalization error in machine learning.
problem Existing generalization theories assume uniform class performance, but in practice, classes vary significantly.
method Developed novel information-theoretic bounds using KL divergence and CMI.
result Theoretical bounds accurately capture complex class-generalization error behavior.
This paper tightens information-theoretic bounds on generalization errors.
problem Understanding the discrepancy between training and testing data losses.
method Investigates the tightness of information-theoretic bounds on generalization error.
result The individual sample mutual information bound can be asymptotically tight under specific assumptions.
Transfer learning has been proven effective when within-target labeled data is scarce. A lot of works have developed successful algorithms and empirically observed positive transfer effect that improves target generalization error using source knowledge. However, theoretical analysis of transfer learning is more challe…
This work sets theoretical limits on meta-learning performance.
problem Understanding the difficulty of adapting machine learning models to real-world data distributions.
method Information-theoretic lower bounds on convergence rates for meta-learning algorithms.
result Theoretical bounds on parameter estimation error for hierarchical Bayesian models of meta-learning.
The paper analyzes SMOTE for imbalanced classification, providing theoretical bounds and guidelines.
problem The challenge of imbalanced classification problems, especially with minority classes.
method Theoretical analysis of SMOTE and related oversampling techniques for minority classes.
result Derives concentration and excess risk bounds for SMOTE and kernel-based classifiers.
EBUCB framework achieves optimal regret with bounded approximate inference error.
problem Theoretical gap between practical performance and theoretical justification of Bayesian bandit algorithms with approximate inference.
method Enhanced Bayesian Upper Confidence Bound (EBUCB) framework that accommodates bandit problems with approximate inference.
result EBUCB achieves optimal regret order O(logT) under certain conditions on inference error. New bound matches exact generalization error for quadratic Gaussian problem.
problem Understanding generalization error in quadratic Gaussian problems.
method Information-theoretic approach with new ingredients.
result Exact tight bound for generalization error.
The paper establishes bounds for transductive learning using information theory.
problem Transductive learning generalization gap control.
method Information theory, PAC-Bayes, mutual information, conditional mutual information, different information measures.
result Established transductive information-theoretic and PAC-Bayesian bounds.
Study on maximizing submodular functions with limited updates, achieving tight bounds and poly-time algorithms.
problem Online submodular maximization with constant recourse.
method Information-theoretic bounds and poly-time randomized algorithms.
result Achieved tight bounds of 2/3 and 3/4 for general and coverage functions, respectively, with a 0.51 approximation.
The study establishes risk bounds for distributional regression estimators.
problem Estimating distributional regression models with nonparametric methods.
method Theoretical bounds for CRPS and MSE are derived for convex and non-convex constraints.
result Theoretical risk bounds are validated through experiments on simulated and real data.
New information-theoretic bounds improve machine learning generalization.
problem Improving machine learning generalization beyond traditional complexity-based methods.
method Introducing bounds using Wasserstein distance and structured methods to incorporate geometry and individual data dependence.
result Established connections between different bounds and introduced new tighter bounds for various loss functions.
This paper introduces a new bound to explain generalization in over-parameterized models.
problem Understanding why some over-parameterized models generalize well while others do not.
method PAC-Chernoff bounds and smoothness measures based on large deviation theory.
result Interpolators with smoother structures generalize better, according to the new theoretical framework.
Determines surgeries on chain links bounding rational homology balls using lattice-theoretic methods.
problem Integral surgeries on chain links bounding rational homology balls.
method Lattice-theoretic cubiquity obstruction and practical computation methods.
result Proves slice-ribbon conjecture for quasi-alternating 3-braid links, extending previous results.
Framework sharpens causal effect estimates without external assumptions.
problem Estimating causal effects under unmeasured confounding.
method Information-theoretic divergence bounds, Neyman orthogonality, machine learning.
result Sharp partial identification of conditional causal effects from observational data.
Novel MOBO method for risk measures under input uncertainty.
problem Efficiently identifying Pareto front for black-box functions with input uncertainty.
method Assumes Gaussian process model and constructs bounding boxes for risk measures.
result The method can return an arbitrary-accurate solution with high probability.
Develops hypothesis tests for conditional distributions using learning-theoretic bounds.
problem Testing differences in conditional distributions and functionals.
method Transforming learning-theoretic bounds into hypothesis tests for conditional expectations.
result Establishes comprehensive foundation for conditional testing, including theoretical guarantees and practical implementations.
New framework controls generalization for heavy-tailed data in RLHF and SGLD.
problem Heavy-tailed data in modern learning pipelines.
method Tail-dependent information-theoretic framework for sub-Weibull data.
result Sharp generalization bounds for heavy-tailed data.
The paper improves risk certificate tightness for neural networks using PAC-Bayes bounds.
problem Improving the usability of risk certificates for neural networks based on PAC-Bayes bounds.
method Theoretical contributions including KL divergence bounds, efficient methodology for optimization, and methods for optimizing non-differentiable objectives.
result First non-vacuous generalization bounds on CIFAR-10 for neural networks.
Meta-learning bound uses conditional mutual information.
problem Bounding generalization performance in meta-learning.
method Extends CMI framework to meta-learning with a meta-supersample.
result Explicit bound involving two CMI terms.
New bound for neural networks with full-rank weights, independent of network width.
problem Understanding generalization of neural networks with full-rank weight matrices.
method Using Koopman operators to derive a tighter generalization bound for full-rank weight matrices.
result The bound is tighter than existing norm-based bounds when condition numbers are small.
Proposes a new bound on generalization error using conditional mutual information.
problem Improving the generalization error bound in machine learning.
method Combines error decomposition and conditional mutual information techniques.
result New bound is order-wise better than previous ones in a simple Gaussian setting.
This paper analyzes multi-view learning using information theory to improve generalization.
problem Lack of theoretical understanding of multi-view learning's generalization behavior.
method Developed information-theoretic generalization bounds for multi-view learning.
result Capturing both consensus and complementary information maximizes representation disentanglement.
Lower bounds show many sampling algorithms need many gradient queries.
problem Sampling from strongly log-concave densities in high dimensions.
method Information theory and stochastic gradient methods.
result Lower bound on number of gradient queries needed.
Novel kernelized Renyi's entropy improves deep learning generalization bounds.
problem Improving generalization bounds for deep learning algorithms.
method Kernelized Renyi's entropy, a new information theoretical measure.
result Theoretical bounds are tighter than current SOTA results.
Paper analyzes how contrastive learning structures learned representations.
problem Understanding the structure of learned representations in contrastive learning.
method Kernel-based contrastive learning framework (KCL) and statistical dependency viewpoint.
result Theoretical upper bound and generalization error bound for KCL.
We derive upper bounds on the generalization error of a learning algorithm in terms of the mutual information between its input and output. The bounds provide an information-theoretic understanding of generalization in learning problems, and give theoretical guidelines for striking the right balance between data fit an…
New method for tensor completion using nonconvex dual total variation.
problem Tensor completion from partial measurements with exponential-family noise.
method Proposed dual-TV (DTV) regularizers for tensor completion under exponential-family noise.
result Theoretical upper bounds on recovery error for tensor completion.