ContraBAR uses contrastive learning to learn Bayes-optimal policies in RL.
problem Learning optimal policies for unknown tasks sampled from a known distribution.
method Proposes ContraBAR, a meta RL algorithm using contrastive predictive coding (CPC) for belief inference.
result ContraBAR achieves comparable performance to state-of-the-art methods and is computationally efficient.
Unified framework for Bayes-optimal classifiers under group fairness.
problem Mitigating disparate impacts from algorithmic predictions in high-stakes decision-making.
method Unified framework based on Neyman-Pearson argument for deriving Bayes-optimal classifiers under group fairness constraints.
result Proposes FairBayes method that directly controls disparity and achieves optimal fairness-accuracy tradeoff.
Bayes-optimal learning of deep random networks with Gaussian weights is studied.
problem Learning a target function corresponding to a deep, extensive-width, non-linear neural network with random Gaussian weights.
method Closed-form expressions for Bayes-optimal test error, ridge regression, kernel and random features regression are computed.
result Optimally regularized ridge regression and kernel regression achieve Bayes-optimal performances, while logistic loss yields a near-optimal test error for classification.
We address the problem of learning to benchmark the best achievable classifier performance. In this problem the objective is to establish statistically consistent estimates of the Bayes misclassification error rate without having to learn a Bayes-optimal classifier. Our learning to benchmark framework improves on previ…
Bayes-optimal learning of a neural network with quadratic activations is achieved with GAMP-RIE.
problem Learning a neural network with quadratic activations from quadratic samples.
method Combining approximate message passing with rotationally invariant matrix denoising.
result Derives a closed-form expression for Bayes-optimal test error.
Paper proves fair classification can be done via simple thresholding.
problem Achieving fair binary classification subject to group fairness constraints.
method Proves Bayes optimal fair learning rule is a group-wise thresholding rule over the Bayes regressor with randomization.
result Proposes an efficient unconstrained optimization algorithm for post-processing fair classification.
Adaptive variational Bayes framework improves inference adaptively.
problem Lack of general and computationally tractable variational Bayes method for adaptive inference.
method Proposes a novel adaptive variational Bayes framework combining variational posteriors over individual models.
result Adaptive variational Bayes achieves optimal contraction rates adaptively under general conditions.
SAM improves deep learning by relaxing Bayes objective.
problem Improving generalization in deep learning models.
method SAM as a relaxation of Bayes objective using Fenchel biconjugate.
result SAM connects adversarial and Bayesian methods for robustness.
Bayes-optimal classifiers are robust to adversarial attacks, unlike CNNs trained on the same data.
problem The vulnerability of modern CNN classifiers to adversarial examples.
method Constructing realistic image datasets and deriving analytic conditions for Bayes-optimal classifiers.
result Bayes-optimal classifiers are robust to adversarial attacks, unlike CNNs trained on the same data.
Study of Bayes optimal learning in high-dimensional linear regression with network side information.
problem Bayes optimal learning in high-dimensional linear regression with network side information.
method Introduce a Reg-Graph model and an iterative AMP algorithm for Bayes optimality under general conditions.
result Characterization of the limiting mutual information between latent signal and data observed.
Researchers compute Bayes error for classification models using normalizing flows.
problem Evaluating the inherent difficulty of classification problems.
method Invertible transformations and Gaussian base distributions to compute Bayes error.
result State-of-the-art models can achieve near-optimal accuracy but not always.
The study analyzes multi-class teacher-student perceptron performance and generalization errors.
problem Analyzing multi-class classification with the teacher-student perceptron.
method Deriving asymptotic expressions for Bayes-optimal and empirical risk minimization (ERM) generalization errors.
result Regularised cross-entropy minimization yields close-to-optimal accuracy for multi-class classification.
PAC-Bayes bound requires prior to place mass on high-performing predictors.
problem Explaining generalization in machine learning.
method Analyzing necessary conditions for PAC-Bayes bounds to provide meaningful generalization guarantees.
result Achieving a target generalisation level requires the prior to place sufficient mass on high-performing predictors.
Bayesian model-based reinforcement learning is a formally elegant approach to learning optimal behaviour under model uncertainty, trading off exploration and exploitation in an ideal way. Unfortunately, finding the resulting Bayes-optimal policies is notoriously taxing, since the search space becomes enormous. In this …
Trading off exploration and exploitation in an unknown environment is key to maximising expected return during learning. A Bayes-optimal policy, which does so optimally, conditions its actions not only on the environment state but on the agent's uncertainty about the environment. Computing a Bayes-optimal policy is how…
New PAC-Bayes bounds use Wasserstein distances to improve generalization.
problem Lack of geometric properties in existing PAC-Bayes bounds.
method Developed new PAC-Bayes bounds with Wasserstein distances.
result Optimization guarantees translate to good generalization abilities.
Paper refines PAC-Bayes bounds for bandit problems.
problem Improving probabilistic bounds for off-policy learning.
method Optimizes PAC-Bayesian bounds using a new parameter optimization approach.
result Provides two parameter-free PAC-Bayes bounds that nearly match optimal rates.
The study calculates the risk of semi-supervised multitask learning on Gaussian mixtures.
problem Understanding the risk in semi-supervised multitask learning on Gaussian mixtures.
method Statistical physics methods applied to Gaussian mixture models.
result The study evaluates the performance gain of learning tasks together versus separately.
PAC-Bayesian theory applied to learning optimization algorithms with generalization guarantees.
problem Learning optimization algorithms with provable generalization guarantees and explicit trade-offs.
method PAC-Bayes theory applied to learning-to-optimize, reformulating the learning procedure into a one-dimensional minimization problem.
result Learned optimization algorithms outperform deterministic worst-case analysis algorithms, even in the limit case of guaranteed convergence.
A new hyperparameter optimization method reduces overfitting.
problem Overfitting in hyperparameter optimization.
method PAC-Bayes bound minimization using gradient-based algorithm.
result Significant reduction in out-of-sample error.
New framework shows cross-attention improves multi-modal in-context learning.
problem Understanding multi-modal in-context learning in neural networks.
method Mathematical framework and linearized cross-attention mechanism.
result Cross-attention mechanism is provably optimal for multi-modal in-context learning.
AIM models explain deep learning in attention layers, offering solvable insights.
problem Understanding how deep learning models learn in attention layers.
method Statistical mechanics and random matrix theory.
result Closed-form predictions for Bayes-optimal generalization error and gradient descent performance.
Bayesian framework reduces online optimization regret.
problem Sequential optimization in dynamic environments with bounded losses.
method PAC-Bayes theory and Bayesian updating principles.
result Achieves O ( T ) \mathcal{O}(\sqrt{T}) O ( T ) regret for bounded losses. Study on conditions for Bayes optimal classifier in adversarial robustness.
problem Existence of Bayes optimal classifier for adversarial robustness.
method General sufficient conditions for existence of Bayes optimal classifier.
result Guaranteed existence of Bayes optimal classifier under certain conditions.
Researchers estimate optimal PAC-Bayes bounds using Hamiltonian Monte Carlo.
problem Estimating tight PAC-Bayes bounds with restricted posterior families.
method Sampling from optimal Gibbs posterior using Hamiltonian Monte Carlo, estimating KL divergence, and proposing high-probability bounds.
result Significant tightness gaps in PAC-Bayes bounds, up to 5-6% in some cases.
New PAC-Bayes training method improves model generalization for unbounded loss.
problem Improving generalization of complex models under unbounded loss.
method Established new PAC-Bayes bound for unbounded loss, jointly training prior and posterior.
result Outperforms existing PAC-Bayes training algorithms and matches ERM accuracy.
The study compares clustering risk in Hidden Markov and i.i.d. models, showing the Bayes classifier is nearly optimal.
problem Comparing clustering risk in Hidden Markov and i.i.d. models.
method Analysis of Bayes risk, theoretical bounds, and simulations.
result The Bayes classifier is nearly optimal for clustering in both Hidden Markov and i.i.d. models.
New method handles unknown task boundaries in continual learning.
problem Catastrophic forgetting in neural networks.
method Fixed-point equations for online variational Bayes optimization.
result Approximates online Bayes update for non-stationary data.
Paper develops PAC-Bayes bounds for unknown linear systems.
problem Learning controllers for unknown stochastic linear discrete-time systems.
method PAC-Bayes framework for data-dependent high probability bounds.
result Proposes efficient learning algorithms with theoretical guarantees.
RQMC improves optimization in variational Bayes problems.
problem Optimizing variational Bayes problems with noisy objective functions.
method Use of randomized quasi-Monte Carlo (RQMC) sampling with stochastic L-BFGS.
result RQMC can significantly speed up optimization and find better parameter values.
Paper extends transfer learning for decision rules, improving treatment rule estimation.
problem Estimating optimal individualized treatment rules under changing conditions.
method Bayes decision rules and low-dimensional empirical risk minimization.
result Consistent estimators and risk bounds established under mild conditions.
The paper proves how Transformers learn from context and generalize well.
problem Understanding how Transformers generalize from diverse tasks.
method Developed a statistical theory for in-context learning, separating risk into Bayes Gap and Posterior Variance.
result The Posterior Variance is task-independent, and the Bayes Gap decreases with more in-context examples.
Memory-based models can learn to approximate Bayes-optimal predictors for non-stationary data.
problem Learning from non-stationary data with unobserved switching points.
method Memory-based neural models, including Transformers, LSTMs, and RNNs, trained to minimize log loss.
result Memory-based models can accurately approximate known Bayes-optimal algorithms and perform Bayesian inference over latent switching points.
Develops methods for fair classification under linear disparity constraints.
problem Disparate impacts of machine learning algorithms on protected groups.
method Bayes-optimal fair classification methods via pre-, in-, and post-processing.
result Explicit forms of Bayes-optimal fair classifiers under linear disparity measures.
We study the rates of convergence from empirical surrogate risk minimizers to the Bayes optimal classifier. Specifically, we introduce the notion of \emph{consistency intensity} to characterize a surrogate loss function and exploit this notion to obtain the rate of convergence from an empirical surrogate risk minimizer…
Deep neural networks are optimal for dependent data using PAC-Bayes bounds.
problem Optimizing deep neural networks for dependent data.
method PAC-Bayes oracle inequalities and Bernstein inequality.
result Upper and lower bounds match, proving minimax optimality.
The paper proves statistical consistency and fairness guarantees for a plug-in algorithm.
problem Establishing statistical guarantees for fairness-aware binary classification.
method Proves statistical consistency and derives finite sample guarantees for the plug-in algorithm.
result The plug-in algorithm is statistically consistent and guarantees fairness and differential privacy.
Paper studies how few pretraining tasks are needed for a linear model to solve new tasks.
problem How many pretraining tasks are needed for a linear model to solve new tasks?
method Pretrained a linear attention model for linear regression with a Gaussian prior.
result Effective pretraining requires a small number of independent tasks, and the model closely matches Bayes optimal.
Proposes a non-convex optimization method for a parsimonious weighted naive Bayes classifier.
problem Improving naïve Bayes classifier performance with a large number of input variables.
method Sparse regularization of model log-likelihood for direct estimation of variable weights.
result Optimization-based weighted naïve Bayes classifiers achieve equivalent performance to averaging-based classifiers.
Optimizes pruning masks for neural networks using probabilistic fine-tuning and PAC-Bayes bounds.
problem Improving neural network performance through adaptive pruning of weights.
method Optimizes stochastic pruning masks by minimizing expected loss, considering data-adaptive regularization and feature alignment.
result Probabilistic fine-tuning leads to improved test error over baseline methods in neural networks.
New method quantifies uncertainty for near-optimal ML algorithms.
problem Uncertainty quantification for near-Bayes optimal ML algorithms.
method Developed a martingale posterior to recover Bayesian posterior from ML algorithms.
result Proved practical uncertainty quantification method applicable to general ML algorithms.
We present the first PAC optimal algorithm for Bayes-Adaptive Markov Decision Processes (BAMDPs) in continuous state and action spaces, to the best of our knowledge. The BAMDP framework elegantly addresses model uncertainty by incorporating Bayesian belief updates into long-term expected return. However, computing an e…
This paper tackles fair Bayes-optimal classifiers under predictive parity, proving their limitations and proposing a new algorithm.
problem Ensuring fair Bayes-optimal classifiers under predictive parity, especially when group performance levels vary widely.
method Proving the limitations of fair Bayes-optimal classifiers under predictive parity and proposing a new adaptive thresholding algorithm, FairBayes-DPP.
result Fair Bayes-optimal classifiers under predictive parity may not hold if group performance levels vary widely, leading to within-group unfairness.
The paper tackles fair classification with multiple sensitive features.
problem Existing fair classification methods often consider a single sensitive feature, but in practice, individuals are defined by multiple sensitive features.
method Characterizes Bayes-optimal fair classifiers for multiple sensitive features under various fairness measures, proposing in-processing and post-processing algorithms.
result Bayes-optimal fair classifiers for multiple sensitive features are instance-dependent thresholding rules that rely on a weighted sum of group membership probabilities.
Study shows how classifiers can approach Bayes error in high-dimensional settings.
problem Generalization error in high-dimensional perceptrons.
method Proved a formula for generalization error using convex optimization and observed that logistic and hinge regression can approach Bayes error closely.
result Logistic and hinge regression can approach Bayes-optimal generalization error closely in high-dimensional settings.
Proposes OBS, a method to adaptively combine Bayesian models online.
problem Learning optimal combinations of Bayesian models in online learning.
method Empirical Bayes lens, Online Bayesian Stacking (OBS).
result Establishes a novel connection between OBS and portfolio selection.
Study exact limits of matrix reconstruction from noisy projections.
problem Reconstructing matrices from linear projections with high-dimensional data.
method Asymptotic analysis, universality properties, and generalized linear models.
result Exact asymptotic equations for optimal learning performance.
Neural Bayes simplifies computing complex stats for unsupervised learning.
problem Computing mutual information and optimal labeling of disjoint manifolds in unsupervised learning.
method Parameterization using neural networks to express statistical quantities in closed form.
result Neural Bayes enables efficient computation of mutual information and optimal labeling of disjoint manifolds.