Unified framework for Bayes-optimal classifiers under group fairness.
problem Mitigating disparate impacts from algorithmic predictions in high-stakes decision-making.
method Unified framework based on Neyman-Pearson argument for deriving Bayes-optimal classifiers under group fairness constraints.
result Proposes FairBayes method that directly controls disparity and achieves optimal fairness-accuracy tradeoff.
ContraBAR uses contrastive learning to learn Bayes-optimal policies in RL.
problem Learning optimal policies for unknown tasks sampled from a known distribution.
method Proposes ContraBAR, a meta RL algorithm using contrastive predictive coding (CPC) for belief inference.
result ContraBAR achieves comparable performance to state-of-the-art methods and is computationally efficient.
Bayes-optimal learning of deep random networks with Gaussian weights is studied.
problem Learning a target function corresponding to a deep, extensive-width, non-linear neural network with random Gaussian weights.
method Closed-form expressions for Bayes-optimal test error, ridge regression, kernel and random features regression are computed.
result Optimally regularized ridge regression and kernel regression achieve Bayes-optimal performances, while logistic loss yields a near-optimal test error for classification.
Study of Bayes optimal learning in high-dimensional linear regression with network side information.
problem Bayes optimal learning in high-dimensional linear regression with network side information.
method Introduce a Reg-Graph model and an iterative AMP algorithm for Bayes optimality under general conditions.
result Characterization of the limiting mutual information between latent signal and data observed.
Bayes-optimal classifiers are robust to adversarial attacks, unlike CNNs trained on the same data.
problem The vulnerability of modern CNN classifiers to adversarial examples.
method Constructing realistic image datasets and deriving analytic conditions for Bayes-optimal classifiers.
result Bayes-optimal classifiers are robust to adversarial attacks, unlike CNNs trained on the same data.
Bayes-optimal learning of a neural network with quadratic activations is achieved with GAMP-RIE.
problem Learning a neural network with quadratic activations from quadratic samples.
method Combining approximate message passing with rotationally invariant matrix denoising.
result Derives a closed-form expression for Bayes-optimal test error.
The study analyzes multi-class teacher-student perceptron performance and generalization errors.
problem Analyzing multi-class classification with the teacher-student perceptron.
method Deriving asymptotic expressions for Bayes-optimal and empirical risk minimization (ERM) generalization errors.
result Regularised cross-entropy minimization yields close-to-optimal accuracy for multi-class classification.
New framework shows cross-attention improves multi-modal in-context learning.
problem Understanding multi-modal in-context learning in neural networks.
method Mathematical framework and linearized cross-attention mechanism.
result Cross-attention mechanism is provably optimal for multi-modal in-context learning.
We study the rates of convergence from empirical surrogate risk minimizers to the Bayes optimal classifier. Specifically, we introduce the notion of \emph{consistency intensity} to characterize a surrogate loss function and exploit this notion to obtain the rate of convergence from an empirical surrogate risk minimizer…
Memory-based models can learn to approximate Bayes-optimal predictors for non-stationary data.
problem Learning from non-stationary data with unobserved switching points.
method Memory-based neural models, including Transformers, LSTMs, and RNNs, trained to minimize log loss.
result Memory-based models can accurately approximate known Bayes-optimal algorithms and perform Bayesian inference over latent switching points.
AIM models explain deep learning in attention layers, offering solvable insights.
problem Understanding how deep learning models learn in attention layers.
method Statistical mechanics and random matrix theory.
result Closed-form predictions for Bayes-optimal generalization error and gradient descent performance.
The paper proves statistical consistency and fairness guarantees for a plug-in algorithm.
problem Establishing statistical guarantees for fairness-aware binary classification.
method Proves statistical consistency and derives finite sample guarantees for the plug-in algorithm.
result The plug-in algorithm is statistically consistent and guarantees fairness and differential privacy.
Develops methods for fair classification under linear disparity constraints.
problem Disparate impacts of machine learning algorithms on protected groups.
method Bayes-optimal fair classification methods via pre-, in-, and post-processing.
result Explicit forms of Bayes-optimal fair classifiers under linear disparity measures.
The paper tackles fair classification with multiple sensitive features.
problem Existing fair classification methods often consider a single sensitive feature, but in practice, individuals are defined by multiple sensitive features.
method Characterizes Bayes-optimal fair classifiers for multiple sensitive features under various fairness measures, proposing in-processing and post-processing algorithms.
result Bayes-optimal fair classifiers for multiple sensitive features are instance-dependent thresholding rules that rely on a weighted sum of group membership probabilities.
This paper tackles fair Bayes-optimal classifiers under predictive parity, proving their limitations and proposing a new algorithm.
problem Ensuring fair Bayes-optimal classifiers under predictive parity, especially when group performance levels vary widely.
method Proving the limitations of fair Bayes-optimal classifiers under predictive parity and proposing a new adaptive thresholding algorithm, FairBayes-DPP.
result Fair Bayes-optimal classifiers under predictive parity may not hold if group performance levels vary widely, leading to within-group unfairness.
Paper studies how few pretraining tasks are needed for a linear model to solve new tasks.
problem How many pretraining tasks are needed for a linear model to solve new tasks?
method Pretrained a linear attention model for linear regression with a Gaussian prior.
result Effective pretraining requires a small number of independent tasks, and the model closely matches Bayes optimal.
Study exact limits of matrix reconstruction from noisy projections.
problem Reconstructing matrices from linear projections with high-dimensional data.
method Asymptotic analysis, universality properties, and generalized linear models.
result Exact asymptotic equations for optimal learning performance.
New method quantifies uncertainty for near-optimal ML algorithms.
problem Uncertainty quantification for near-Bayes optimal ML algorithms.
method Developed a martingale posterior to recover Bayesian posterior from ML algorithms.
result Proved practical uncertainty quantification method applicable to general ML algorithms.
Study on conditions for Bayes optimal classifier in adversarial robustness.
problem Existence of Bayes optimal classifier for adversarial robustness.
method General sufficient conditions for existence of Bayes optimal classifier.
result Guaranteed existence of Bayes optimal classifier under certain conditions.
Bayes optimal algorithm under certain conditions doesn't achieve exponential simple regret.
problem Best arm identification with normal rewards over time.
method Fixed-budget best arm identification problem with rewards from normal distributions. Evaluates performance via simple regret.
result Bayes optimal algorithm does not yield exponential decrease in simple regret.
Meta-learning method improves PU classification performance.
problem Improving binary classifiers from PU data in unseen tasks.
method Adapts model to PU data using related tasks and neural networks.
result Proposed method outperforms existing methods on synthetic and real-world datasets.
Sharp feature transitions revealed in extensive-width networks.
problem Learning hierarchical features from noisy queries in large networks.
method Information-theoretic analysis and heuristic decoupling argument.
result Sequential phase transitions in feature learnability and effective width.
Unified approach for fair classification with overlapping groups.
problem Ensuring fairness across multiple overlapping groups in prediction problems.
method Probabilistic population analysis leading to Bayes-optimal classifier, unifying existing methods.
result Outperforms baselines in fairness-performance tradeoff on real datasets.
Paper proves fair classification can be done via simple thresholding.
problem Achieving fair binary classification subject to group fairness constraints.
method Proves Bayes optimal fair learning rule is a group-wise thresholding rule over the Bayes regressor with randomization.
result Proposes an efficient unconstrained optimization algorithm for post-processing fair classification.
The paper explores robust classifiers for imbalanced Gaussian data.
problem Adversarial robustness in machine learning with imbalanced data.
method Developed exact and approximate Bayes-optimal robust classifiers for Gaussian classification problems.
result Revealed fundamental tradeoffs between standard and robust accuracy.
Bayesian model-based reinforcement learning is a formally elegant approach to learning optimal behaviour under model uncertainty, trading off exploration and exploitation in an ideal way. Unfortunately, finding the resulting Bayes-optimal policies is notoriously taxing, since the search space becomes enormous. In this …
Trading off exploration and exploitation in an unknown environment is key to maximising expected return during learning. A Bayes-optimal policy, which does so optimally, conditions its actions not only on the environment state but on the agent's uncertainty about the environment. Computing a Bayes-optimal policy is how…
LDA-GO improves LDA for high-dimensional data via gradient optimization.
problem LDA struggles in high-dimensional settings due to unreliable covariance matrix estimation.
method LDA-GO learns a low-rank precision matrix via gradient optimization, automatically selecting between Gaussian likelihood and cross-entropy loss.
result LDA-GO outperforms other LDA variants in sparse-signal high-dimensional regimes.
Study optimizes best-arm identification with minimax and Bayes strategies.
problem Efficiently identifying the best arm in fixed-budget scenarios.
method Adaptive procedure with two stages: pilot phase and minimax game.
result Single strategy is asymptotically minimax and Bayes optimal.
In many applications of black-box optimization, one can evaluate multiple points simultaneously, e.g. when evaluating the performances of several different neural network architectures in a parallel computing environment. In this paper, we develop a novel batch Bayesian optimization algorithm --- the parallel knowledge…
We present an alternative to the pseudo-inverse method for determining the hidden to output weight values for Extreme Learning Machines performing classification tasks. The method is based on linear discriminant analysis and provides Bayes optimal single point estimates for the weight values.
Instance- and Label-dependent label Noise (ILN) widely exists in real-world datasets but has been rarely studied. In this paper, we focus on Bounded Instance- and Label-dependent label Noise (BILN), a particular case of ILN where the label noise rates -- the probabilities that the true labels of examples flip into the …
Paper tackles regression with cost-based rejection, balancing prediction and rejection costs.
problem Regression with cost-based rejection, balancing prediction and rejection costs in a continuous target space.
method Formulated expected risk, derived Bayes optimal solution, proposed surrogate loss function.
result Bayes optimal solution can be recovered by the proposed surrogate loss function.
Optimal classifiers derived from GMMs are approximated by deep neural networks.
problem Binary classification of high-dimensional overlapping Gaussian mixtures.
method Closed-form expressions for Bayes optimal decision boundaries derived from GMMs' eigenstructure. Empirical validation through synthetic and real-world data.
result Deep neural networks approximate optimal classifiers for GMMs, with decision thresholds related to covariance eigenvectors.
We consider effort allocation in crowdsourcing, where we wish to assign labeling tasks to imperfect homogeneous crowd workers to maximize overall accuracy in a continuous-time Bayesian setting, subject to budget and time constraints. The Bayes-optimal policy for this problem is the solution to a partially observable Ma…
Optimal graph classification uses message-passing neural networks.
problem Node classification on sparse graphs with fixed feature dimensions.
method Asymptotic local Bayes optimality, message-passing graph neural networks.
result Optimal message-passing architecture interpolates between MLP and convolution.
This work analyzes two methods for combining multiple binary labels in bipartite ranking.
problem Combining multiple binary labels for optimal bipartite ranking.
method Loss aggregation vs. label aggregation approaches.
result Label aggregation is preferable to loss aggregation due to label dictatorship issues.
Recent advances in Bayesian reinforcement learning (BRL) have shown that Bayes-optimality is theoretically achievable by modeling the environment's latent dynamics using Flat-Dirichlet-Multinomial (FDM) prior. In self-interested multi-agent environments, the transition dynamics are mainly controlled by the other agent'…
This study optimizes multi-modal learning thresholds and algorithms in high dimensions.
problem Optimizing multi-modal learning performance in high-dimensional data.
method Analytical quantification and derivation of AMP algorithm with state evolution analysis.
result Bayes-optimal performance and recovery thresholds derived for multi-modal data.
Bayes-optimal limits in PCA with structured noise are determined.
problem Analyzing statistical dependencies in measurement noise for high-dimensional inference.
method Study of spiked matrix model with low-order polynomial orthogonal noise, providing Bayes-optimal limits and proposing a novel AMP.
result A novel AMP algorithm reaches the information-theoretic limits for more general priors.
Bad models can teach well by replicating noise.
problem Overparameterized models can replicate noise in training data.
method Knowledge distillation from noisy samplers.
result Distillation from samplers approximates Bayes optimal classifier.
Paper establishes limits for accurately estimating low-rank matrices from noisy, non-linear data.
problem Estimating low-rank matrices from noisy, non-linear observations.
method Proves strong universality result with equivalent Gaussian model and effective prior parameters.
result Signal-to-noise ratio requirement grows as $N^{ rac 12 (1-1/k_F)}$ for accurate reconstruction.
Improved neural active learning algorithms reduce regret and improve performance.
problem Improving performance and reducing regret in neural active learning for non-parametric streaming data.
method Introducing two new regret metrics and leveraging NNs for both exploitation and exploration. The algorithm uses tailored query decision-makers and full feedback.
result Achieved an instance-dependent regret upper bound improving by a multiplicative factor of O(logT) and removing the curse of dimensionality. LGD algorithm learns optimal hyperparameters for regression tasks.
problem Learning to learn hyperparameters for regression problems.
method Langevin Gradient Descent (LGD) approximates posterior distribution.
result Meta-learning optimal hyperparameters achieves Bayes' optimal solution.
Transformer attention layers solve single-location regression tasks.
problem Understanding token-wise sparsity and internal linear representations in attention-based models.
method Introduce single-location regression task and a simplified predictor based on self-attention layers.
result Transformer attention layers are asymptotically Bayes optimal and can learn underlying structures effectively.
Bayesian approach optimizes in-context learning for state space models.
problem Optimizing in-context learning for state space models.
method Bayesian optimal sequential prediction over latent sequence tasks.
result Bayesian optimal predictor converges to posterior predictive mean.
Bayesian Pseudo Label Selection reduces overfitting in semi-supervised learning.
problem Overfitting in pseudo-label selection for semi-supervised learning.
method BPLS, a Bayesian framework that approximates the posterior predictive of pseudo-samples.
result BPLS outperforms traditional PLS methods, especially in high-dimensional data.
We analyze the problem of learning a single user's preferences in an active learning setting, sequentially and adaptively querying the user over a finite time horizon. Learning is conducted via choice-based queries, where the user selects her preferred option among a small subset of offered alternatives. These queries …