The study compares clustering risk in Hidden Markov and i.i.d. models, showing the Bayes classifier is nearly optimal.
problem Comparing clustering risk in Hidden Markov and i.i.d. models.
method Analysis of Bayes risk, theoretical bounds, and simulations.
result The Bayes classifier is nearly optimal for clustering in both Hidden Markov and i.i.d. models.
Study calculates Bayes risk for semi-supervised learning with uncertain labels.
problem Uncertain labeling in semi-supervised classification.
method Gaussian mixture model, Bayes risk computation, comparison with algorithm performance.
result New insights into semi-supervised learning algorithm performance.
New method reduces computational cost for estimating PAC-Bayes bounds.
problem High computational cost in estimating PAC-Bayes bounds.
method General alternative method that makes computational savings.
result Reduces computational cost on the order of the dataset size.
We study the rates of convergence from empirical surrogate risk minimizers to the Bayes optimal classifier. Specifically, we introduce the notion of \emph{consistency intensity} to characterize a surrogate loss function and exploit this notion to obtain the rate of convergence from an empirical surrogate risk minimizer…
The study calculates the risk of semi-supervised multitask learning on Gaussian mixtures.
problem Understanding the risk in semi-supervised multitask learning on Gaussian mixtures.
method Statistical physics methods applied to Gaussian mixture models.
result The study evaluates the performance gain of learning tasks together versus separately.
Lower bounds on Bayes risk for realizable models derived using information theory.
problem Deriving lower bounds on Bayes risk for realizable machine learning models.
method Information-theoretic analysis using rate-distortion theory and mutual information.
result Lower bounds on Bayes risk for realizable models, matching known bounds up to logarithmic factors.
Adversarial consistency depends on the uniqueness of adversarial Bayes classifiers.
problem Consistency of adversarial surrogate losses is not guaranteed.
method Connected consistency of adversarial surrogate losses to the uniqueness of adversarial Bayes classifiers.
result A convex surrogate loss is statistically consistent for adversarial learning if and only if the adversarial Bayes classifier is unique.
PAC-Bayes bound requires prior to place mass on high-performing predictors.
problem Explaining generalization in machine learning.
method Analyzing necessary conditions for PAC-Bayes bounds to provide meaningful generalization guarantees.
result Achieving a target generalisation level requires the prior to place sufficient mass on high-performing predictors.
Meta-learning bounds derived using PAC-Bayes theory for improved generalization.
problem Uncertainty in generalization performance for meta-learning with new tasks.
method PAC-Bayes relative entropy bounds and empirical risk minimization (ERM) method.
result Competitive generalization performance and rapid convergence with data-dependent prior.
New method certifies deep graph classifiers with tighter risk bounds.
problem Certifying the reliability of deep graph classifiers.
method Linearized deep assignment flows with random initial conditions, using PAC-Bayes risk certification.
result Computes tighter out-of-sample risk certificates efficiently.
The paper improves risk certificate tightness for neural networks using PAC-Bayes bounds.
problem Improving the usability of risk certificates for neural networks based on PAC-Bayes bounds.
method Theoretical contributions including KL divergence bounds, efficient methodology for optimization, and methods for optimizing non-differentiable objectives.
result First non-vacuous generalization bounds on CIFAR-10 for neural networks.
This work uses PAC-Bayes for structured prediction with ILE, yielding insights and algorithms.
problem Structured prediction with interdependent outputs and implicit loss embeddings.
method PAC-Bayes perspective applied to ILE framework, deriving generalization bounds and learning algorithms.
result Two learning algorithms derived from PAC-Bayes bounds, analyzed and implemented.
New estimator reduces risk in slate bandits by leveraging Bayes risk criterion.
problem Evaluating slate policies using logged data when policies factorize over slots.
method Developed a new estimator using a control variate approach, showing risk improvement over existing methods.
result The new estimator has lower risk than the pseudoinverse estimator in slate bandit problems.
New PAC-Bayes bounds derived using Legendre transform and f-divergences.
problem Deriving PAC-Bayes bounds under various assumptions.
method Combining Legendre transform and Fenchel--Young inequality to derive change-of-measure inequalities.
result Extended PAC-Bayesian guarantees under tailored assumptions.
PAC-Bayesian framework for fairness in stochastic and deterministic classifiers.
problem Theoretical guarantees on fairness for balancing predictive risk and fairness constraints.
method PAC-Bayesian framework for both stochastic and deterministic classifiers, covering a broad class of fairness measures.
result Derives generalization bounds for fairness, demonstrating tightness with empirical evaluation.
New method optimises learning via surrogate PAC-Bayes bounds.
problem Computational intractability of optimising generalisation bounds.
method Iteratively optimising surrogate training objectives derived from PAC-Bayes bounds.
result Iteratively optimising surrogates implies optimising original generalisation bounds.
Paper improves risk estimation for extreme events.
problem Estimating extreme risks accurately.
method Modified Bayes risk for expectiles, asymptotic expansions, efficient estimators.
result Asymptotic normality of estimators proved.
PAC-Bayes bounds have been proposed to get risk estimates based on a training sample. In this paper the PAC-Bayes approach is combined with stability of the hypothesis learned by a Hilbert space valued algorithm. The PAC-Bayes setting is used with a Gaussian prior centered at the expected output. Thus a novelty of our …
New PAC-Bayes training method improves model generalization for unbounded loss.
problem Improving generalization of complex models under unbounded loss.
method Established new PAC-Bayes bound for unbounded loss, jointly training prior and posterior.
result Outperforms existing PAC-Bayes training algorithms and matches ERM accuracy.
New data-dependent priors improve PAC-Bayes bounds.
problem Improving PAC-Bayes bounds for nonconvex learning.
method Using data to learn a conditional expectation of the posterior, given a subset of training data.
result Data-dependent oracle priors lead to stronger PAC-Bayes bounds.
PAC-Bayes bounds for Gibbs posteriors derived via singular learning theory.
problem Generalization bounds for overparameterized models with data-dependent priors.
method Explicit non-asymptotic PAC-Bayes bounds using singular learning theory.
result Explicit posterior-averaged risk bounds for overparameterized models.
New bound improves on weighted majority vote risk estimation.
problem Improving risk estimation for weighted majority vote.
method Novel Chebyshev-Cantelli inequality and PAC-Bayes-Bennett inequality.
result New bounds improve on existing methods.
A statistical decision problem is hidden in the core of option pricing. A simple form for the price C of a European call option is obtained via the minimum Bayes risk, R_B, of a 2-parameter estimation problem, thus justifying calling C Bayes (B-)price. The result provides new insight in option pricing, among others obt…
Gradient descent converges to minimum Bayes risk for two-layer ReLU networks in mean field regime.
problem Training two-layer ReLU networks using gradient descent in the mean field regime.
method Describes a condition for convergence to minimum Bayes risk, extending previous results to ReLU-activated networks.
result The condition for convergence does not depend on initialization and concerns weak convergence of network realization.
Paper presents new training methods for neural networks with tighter risk certificates.
problem Training probabilistic neural networks with tighter risk certificates.
method Derived from PAC-Bayes bounds, two training objectives implemented for the first time in neural networks.
result Competitive test set errors and non-vacuous risk bounds with tighter values than previous results.
Paper extends transfer learning for decision rules, improving treatment rule estimation.
problem Estimating optimal individualized treatment rules under changing conditions.
method Bayes decision rules and low-dimensional empirical risk minimization.
result Consistent estimators and risk bounds established under mild conditions.
New method certifies risks of LLM outputs, improving accuracy and reliability.
problem Uncertain and incorrect outputs from large language models.
method Information-lift certificates using PAC-Bayes bounds and skeleton design.
result Achieves 77.0% coverage at 2% risk, outperforming baselines.
We show that Entropy-SGD (Chaudhari et al., 2017), when viewed as a learning algorithm, optimizes a PAC-Bayes bound on the risk of a Gibbs (posterior) classifier, i.e., a randomized classifier obtained by a risk-sensitive perturbation of the weights of a learned classifier. Entropy-SGD works by optimizing the bound's p…
Enhances normal mean estimation with side info using NIT approach.
problem Compound estimation of normal means with side information.
method Empirical Bayes, nonparametric integrative Tweedie (NIT) approach.
result NIT approach improves estimation risk and convergence rate with increasing auxiliary data.
Study generalizes property elicitation to imprecise probabilities.
problem Minimizing risk over imprecise probability distributions.
method Maximin risk minimization over a set of imprecise probabilities.
result Conditions for elicitability of IP-properties.
The paper proves how Transformers learn from context and generalize well.
problem Understanding how Transformers generalize from diverse tasks.
method Developed a statistical theory for in-context learning, separating risk into Bayes Gap and Posterior Variance.
result The Posterior Variance is task-independent, and the Bayes Gap decreases with more in-context examples.
VIB balances empirical and Bayesian approaches in predictive models.
problem Balancing empirical and Bayesian methods in predictive models.
method VIB as a compromise between empirical and Bayesian objectives.
result VIB minimizes risks due to finite sampling of targets.
The study analyzes multi-class teacher-student perceptron performance and generalization errors.
problem Analyzing multi-class classification with the teacher-student perceptron.
method Deriving asymptotic expressions for Bayes-optimal and empirical risk minimization (ERM) generalization errors.
result Regularised cross-entropy minimization yields close-to-optimal accuracy for multi-class classification.
We presented Bayesian portfolio selection strategy, via the k factor asset pricing model. If the market is information efficient, the proposed strategy will mimic the market; otherwise, the strategy will outperform the market. The strategy depends on the selection of a portfolio via Bayesian multiple testing methodol…
The problem of estimating a high-dimensional sparse vector θ∈Rn from an observation in i.i.d. Gaussian noise is considered. The performance is measured using squared-error loss. An empirical Bayes shrinkage estimator, derived using a Bernoulli-Gaussian prior, is analyzed and compared with the…
We explore the family of methods "PAC-Bayes with Backprop" (PBB) to train probabilistic neural networks by minimizing PAC-Bayes bounds. We present two training objectives, one derived from a previously known PAC-Bayes bound, and a second one derived from a novel PAC-Bayes bound. Both training objectives are evaluated o…
A new procedure for learning cost-sensitive SVM(CS-SVM) classifiers is proposed. The SVM hinge loss is extended to the cost sensitive setting, and the CS-SVM is derived as the minimizer of the associated risk. The extension of the hinge loss draws on recent connections between risk minimization and probability elicitat…
A new sequential method estimates Poisson means in streaming data, achieving optimality and efficiency.
problem Estimating Poisson means in a streaming, or online, framework.
method A quasi-Bayesian approach based on Newton's algorithm for a sequential estimate.
result Established frequentist guarantees including consistency and asymptotic optimality.
The developments of Rademacher complexity and PAC-Bayesian theory have been largely independent. One exception is the PAC-Bayes theorem of Kakade, Sridharan, and Tewari (2008), which is established via Rademacher complexity theory by viewing Gibbs classifiers as linear operators. The goal of this paper is to extend thi…
Research on nearest-neighbor methods tends to focus somewhat dichotomously either on the statistical or the computational aspects -- either on, say, Bayes consistency and rates of convergence or on techniques for speeding up the proximity search. This paper aims at bridging these realms: to reap the advantages of fast …
Distance weighted discrimination (DWD) is a margin-based classifier with an interesting geometric motivation. DWD was originally proposed as a superior alternative to the support vector machine (SVM), however DWD is yet to be popular compared with the SVM. The main reasons are twofold. First, the state-of-the-art algor…
In this document we are going to derive the equations needed to implement a Variational Bayes i-vector extractor. This can be used to extract longer i-vectors reducing the risk of overfittig or to adapt an i-vector extractor from a database to another with scarce development data. This work is based on Patrick Kenny's …
In this work, we propose a PAC-Bayes bound for the generalization risk of the Gibbs classifier in the multi-class classification framework. The novelty of our work is the critical use of the confusion matrix of a classifier as an error measure; this puts our contribution in the line of work aiming at dealing with perfo…
In many signal detection and classification problems, we have knowledge of the distribution under each hypothesis, but not the prior probabilities. This paper is aimed at providing theory to quantify the performance of detection via estimating prior probabilities from either labeled or unlabeled training data. The erro…
New method uses SURE to denoise signals, outperforming NPMLE.
problem Learning to optimally denoise signals corrupted by Gaussian noise.
method Hyvärinen's score matching (SM) is shown equivalent to SURE minimization.
result SURE achieves nearly parametric rates of convergence in empirical Bayes settings.
Predicting an individual's risk of experiencing a future clinical outcome is a statistical task with important consequences for both practicing clinicians and public health experts. Modern observational databases such as electronic health records (EHRs) provide an alternative to the longitudinal cohort studies traditio…
This paper examines various definitions of adversarial risk and their implications.
problem Quantifying the performance of classifiers under adversarial perturbations.
method Optimal transport, robust statistics, functional analysis, and game theory.
result Generalization of Strassen's theorem and new connections to Choquet capacities and game theory.
MARLA uses deep reinforcement learning for multi-agent AHT, reducing Bayes risk.
problem Decentralized active hypothesis testing among multiple agents.
method MARLA algorithm using deep multi-agent reinforcement learning.
result MARLA outperforms single-agent learning in AHT problems.