This work uses PAC-Bayes for structured prediction with ILE, yielding insights and algorithms.
problem Structured prediction with interdependent outputs and implicit loss embeddings.
method PAC-Bayes perspective applied to ILE framework, deriving generalization bounds and learning algorithms.
result Two learning algorithms derived from PAC-Bayes bounds, analyzed and implemented.
New algorithm constructs PAC prediction sets for meta-learning.
problem Uncertainty quantification in safety-critical systems.
method Proposes a novel algorithm to construct PAC prediction sets.
result Prediction sets satisfy a PAC guarantee with high probability over future tasks.
PAC-Bayesian framework for fairness in stochastic and deterministic classifiers.
problem Theoretical guarantees on fairness for balancing predictive risk and fairness constraints.
method PAC-Bayesian framework for both stochastic and deterministic classifiers, covering a broad class of fairness measures.
result Derives generalization bounds for fairness, demonstrating tightness with empirical evaluation.
Algorithm constructs prediction sets with PAC guarantees in label shift settings.
problem Reliable uncertainty quantification in the face of distribution shift.
method Estimates predicted probabilities and confusion matrix, then propagates uncertainty through Gaussian elimination to compute confidence intervals and construct prediction sets.
result Satisfies PAC guarantees and produces smaller, more informative prediction sets.
The paper improves off-policy evaluation in contextual bandits using conformal prediction.
problem Quantifying the performance of a target policy using data from a different behavior policy.
method Proposes a novel algorithm based on a PAC-valid conformal prediction framework to construct probably approximately correct prediction intervals.
result Establishes PAC-type bounds on coverage, improving theoretical guarantees.
Improves Bayesian predictive performance in misspecified models.
problem Misspecification gap between inferential and predictive risks.
method Develops a multi-sample loss (PAC m ^m m ) to bridge the gap. result Empirical study shows improved predictive distribution.
Paper derives PAC-Bayesian bounds for LTI systems learning from empirical data.
problem Characterizing predictive power of LTI systems learned from data.
method PAC-Bayesian bounds for LTI stochastic dynamical systems with inputs.
result Finite-sample error bounds for learning algorithms of LTI systems.
Develops anytime-valid conformal and PAC prediction for streaming data.
problem Lack of guarantee in traditional conformal methods for sequential settings.
method Extends conformal and PAC prediction frameworks to handle streaming data.
result Provides anytime-valid prediction sets for sequential settings.
Proposes a method to quantify uncertainty in predictions under covariate shift.
problem Uncertainty quantification challenges in machine learning with covariate shifts.
method Constructs PAC prediction sets with given importance weights and confidence intervals for weights.
result Algorithm gives prediction sets with the smallest average normalized size.
We propose an algorithm combining calibrated prediction and generalization bounds from learning theory to construct confidence sets for deep neural networks with PAC guarantees---i.e., the confidence set for a given input contains the true label with high probability. We demonstrate how our approach can be used to cons…
New PAC-Bayesian bounds for online learning with data streams.
problem Challenges of traditional PAC-Bayesian bounds in dynamic data collection.
method Developed new PAC-Bayesian bounds in online learning framework, using updated regret definition and batch-to-online conversion.
result PAC-Bayesian bounds hold for online learning with dependent data and non-convex losses.
The paper improves PAC-Bayes bounds for data-dependent predictors.
problem Guaranteeing the quality of predictions on unseen examples.
method Basic PAC-Bayes inequality for stochastic kernels, leading to various bounds.
result Validates PAC-Bayes bounds without fixed 'data-free' priors and bounded losses.
Learn conditional averages in PAC framework for better predictions.
problem Learning average labels over neighborhoods in unknown concept class.
method Characterization of learnability using combinatorial parameters.
result Complete characterization and sample complexity bounds.
Enhances predictive models against misspecification and outliers.
problem Suboptimal generalization under misspecification and outliers.
method Combines PAC m ^m m ensemble bounds with a generalized logarithm score function. result Produces predictive distributions resistant to both misspecification and outliers.
New PAC-Bayesian framework for flexible meta-learning.
problem Limitations in transferring knowledge between tasks in meta-learning.
method PAC-Bayesian theory applied to learning the learning algorithm.
result Flexibility in meta-learning mechanisms and improved prediction quality.
PAC-Bayes theory improves ICP efficiency and coverage.
problem Inefficient and unreliable uncertainty estimates in deep learning models.
method PAC-Bayes theory to optimize model and score function parameters for efficient and reliable prediction sets.
result Generalization bounds on coverage and efficiency of optimized prediction sets.
We consider using an ensemble of binary classifiers for transductive prediction, when unlabeled test data are known in advance. We derive minimax optimal rules for confidence-rated prediction in this setting. By using PAC-Bayes analysis on these rules, we obtain data-dependent performance guarantees without distributio…
The paper explores learning from label proportions, showing differences in efficiency between LLP and PAC learning.
problem Learning from label proportions (LLP) in unlabeled data with given label proportions.
method Formal definition and computational complexity analysis of LLP learning.
result LLP learning is more restrictive than PAC learning for finite VC classes, and some classes are uncharacterizable.
This research improves binary classification by balancing overfitting and generalization with a novel Bayesian approach.
problem Improving binary classification models to avoid overfitting and generalize well.
method Introduces a PAC-Bayes type learning rule with a balancing parameter λ to balance training error and KL divergence to a prior.
result A choice of λ ensures uniformly vanishing excess loss, even in the agnostic case, by under-regularizing or over-regularizing appropriately.
Paper presents a new probabilistic approach for high-dimensional quantile prediction.
problem High-dimensional quantile prediction challenges in robust statistical methods.
method Pseudo-Bayesian framework with scaled Student-t prior and Langevin Monte Carlo.
result Demonstrates strong theoretical guarantees and competitive performance in simulations and real-world data.
Meta-learning can successfully acquire useful inductive biases from data. Yet, its generalization properties to unseen learning tasks are poorly understood. Particularly if the number of meta-training tasks is small, this raises concerns about overfitting. We provide a theoretical analysis using the PAC-Bayesian framew…
Characterizes the sample complexity of list regression tasks.
problem Understanding the sample complexity of list learning tasks in regression.
method Introducing two combinatorial dimensions: k-OIG dimension and k-fat-shattering dimension.
result These dimensions characterize realizable and agnostic k-list regression.
New bounds predict deep learning generalization better than existing methods.
problem Predicting generalization errors for deep learning models.
method Function-based PAC-Bayesian bounds that meet multiple desiderata.
result The new bound performs significantly better than existing parameter-based PAC-Bayes bounds.
Paper develops tighter risk certificates for contrastive learning models.
problem Statistical theory for contrastive learning is lacking, especially for practical models like SimCLR.
method Develops non-vacuous PAC-Bayesian risk certificates considering practical SimCLR factors.
result Risk certificates for contrastive loss and downstream prediction are much tighter than previous results.
Sharp sample complexity for multiclass PAC learning with bandit feedback.
problem Multiclass PAC learning with bandit feedback.
method Characterization of optimal sample complexity using a new combinatorial dimension (bandit DS dimension) and a learning algorithm (ListCascade).
result Sharp sample complexity characterization for every concept class up to logarithmic factors.
The paper addresses the reliability of conformal prediction under covariate shift.
problem Ensuring reliable prediction sets under covariate shift.
method Derives upper bounds on training-conditional coverage.
result Offers PAC guarantees for conformal prediction methods.
We formulate weighted graph clustering as a prediction problem: given a subset of edge weights we analyze the ability of graph clustering to predict the remaining edge weights. This formulation enables practical and theoretical comparison of different approaches to graph clustering as well as comparison of graph cluste…
We informally call a stochastic process learnable if it admits a generalization error approaching zero in probability for any concept class with finite VC-dimension (IID processes are the simplest example). A mixture of learnable processes need not be learnable itself, and certainly its generalization error need not de…
Private classification and online prediction are shown to be equivalent.
problem Learning with differential privacy and online prediction equivalence.
method Introducing global stability and proving equivalence between online learnability and private PAC learnability.
result Every concept class with finite Littlestone dimension can be learned by a differentially-private algorithm.
Study improves theoretical understanding of Bayesian deep learning for classification tasks.
problem Theoretical gap in understanding Bayesian approaches in deep learning for classification.
method PAC-Bayes bounds techniques and Spike-and-Slab priors for sparse deep learning.
result Established non-asymptotic results for prediction error, achieving minimax optimal rates.
New method uses quotient predictor space for better PAC-Bayes bounds, reducing KL divergence and improving model performance.
problem Overparameterized models with continuous symmetries can lead to biased predictions.
method Perform PAC-Bayesian analysis on quotient predictor space, constructing a canonical prior that reflects model's implicit bias.
result The new prior reduces KL divergence and improves model performance in experiments.
Bayesian neural networks use temperature adjustments to improve predictive performance.
problem Lack of theoretical generalization guarantees for Bayesian neural networks.
method Temperature adjustments to balance likelihood and prior regularization.
result Improved predictive performance through temperature adjustments.
Novel risk bound for structured prediction tackles non-i.i.d. data.
problem Structured prediction challenges due to non-factorizable target objects.
method PAC-Bayesian risk bound with explicit structure distillation.
result Generalization rate scales with data size and structure.
We propose a novel method for closed-form predictive distribution modeling with neural nets. In quantifying prediction uncertainty, we build on Evidential Deep Learning, which has been impactful as being both simple to implement and giving closed-form access to predictive uncertainty. We employ it to model aleatoric un…
New findings on PAC learning and marginal distribution estimation.
problem Understanding how PAC learning relates to marginal distribution estimation under distributional constraints.
method Revisited the connection between PAC learning, uniform convergence, and density estimation, considering a known family of marginal distributions.
result PAC learning is sandwiched between two refined models of density estimation, differing only in whether the learner knows the set of well-estimated events in H.
New framework improves model reliability under distribution shifts.
problem Lack of formal guarantees connecting shift magnitude to prediction reliability in TTA methods.
method Develops a PAC-Bayesian framework interpreting MMD-balls as credal sets.
result Establishes generalization bounds and provides epistemic uncertainty quantification.
Improved Bayesian inference via variational approximations of generalized rho-posteriors.
problem Robust Bayesian inference under model misspecification and data contamination.
method Introducing a modified ρ ρ ρ -posterior and using PAC-Bayesian analysis with variational approximations. result Theoretical guarantees for tractable inference with competitive robustness and computational efficiency.
Improved PAC learning algorithm for multiclass classification with bandit feedback.
problem Efficiently learning multiclass classification with limited feedback.
method Novel algorithm using stochastic optimization and Frank-Wolfe updates.
result Improved sample complexity bounds for multiclass PAC learning.
Improves model accuracy for neural nets in stochastic dynamics with partial prior knowledge.
problem Stability and accuracy in neural nets modeling stochastic dynamics with many parameters.
method Three steps: probabilistic weights, partial knowledge incorporation, and PAC-Bayesian training.
result Improved model fit with partial and noisy prior knowledge.
Novel Bayesian method for high-dimensional count data prediction.
problem Count data in high-dimensional settings requires feature selection.
method Pseudo-Bayesian framework with scaled Student prior and exponential weights.
result Strong performance compared to Lasso in various settings.
Bayesian framework reduces online optimization regret.
problem Sequential optimization in dynamic environments with bounded losses.
method PAC-Bayes theory and Bayesian updating principles.
result Achieves O ( T ) \mathcal{O}(\sqrt{T}) O ( T ) regret for bounded losses. DAMVI algorithm improves imbalanced binary classification by adjusting weights of examples and classifiers.
problem Imbalanced binary classification tasks where minority class is underrepresented.
method DAMVI algorithm increases positive example weights and optimizes classifier weights using PAC-Bayesian C-Bound.
result DAMVI outperforms state-of-the-art models on various imbalanced datasets.
Study on sequential prediction with log-loss, focusing on well-specified and misspecified cases.
problem Sequential prediction with log-loss under different specification conditions.
method Analysis of cumulative regret in well-specified and misspecified cases for a Gaussian location hypothesis class.
result Cumulative regrets in well-specified and misspecified cases asymptotically coincide for the d d d -dimensional Gaussian location hypothesis class. Study analyzes EU in variational inference for Bayesian deep learning.
problem Analyzing epistemic uncertainty in variational inference for Bayesian deep learning.
method Theoretical analysis and derivation of relations between generalization error and EU measurements. Proposing a new objective function for VI.
result Proposed new objective function significantly improves EU evaluation over existing VI methods.
PAC-Bayesian bounds for MLPs with cross entropy loss validated.
problem Generalization bounds for MLPs with cross entropy loss.
method Introduced probabilistic explanations and proved PAC-Bayesian bounds using ELBO.
result MLPs with cross entropy loss inherently guarantee PAC-Bayesian generalization bounds.
PAC-Bayesian bounds show fully connected DNNs with Gaussian priors match minimax rates.
problem Theoretical limits of fully connected deep neural networks with Gaussian priors.
method PAC-Bayesian bounds for fully connected Bayesian DNNs with Gaussian priors.
result PAC-Bayesian bounds match minimax-optimal rates in Besov space for nonparametric regression and binary classification.
Unified derivation of PAC-Bayes and MI bounds for general VC classes with fast rates.
problem Generalization bounds for machine learning models with VC classes.
method Unified derivation of conditional PAC-Bayesian and mutual information bounds, including MAC-Bayesian bounds.
result Nontrivial bounds for general VC classes and faster rates for specific conditions.
The developments of Rademacher complexity and PAC-Bayesian theory have been largely independent. One exception is the PAC-Bayes theorem of Kakade, Sridharan, and Tewari (2008), which is established via Rademacher complexity theory by viewing Gibbs classifiers as linear operators. The goal of this paper is to extend thi…