Neural Empirical Bayes estimates source distributions from noisy simulations.
problem Estimating source distributions from noisy, simulated data.
method Uses neural density estimators to estimate a prior or source distribution over uncorrupted samples, then performs posterior inference.
result Recovering ground truth source distributions up to symmetries.
Bayesian Empirical Bayes extends EB to complex structures using probabilistic symmetry.
problem Improving simultaneous inference in complex settings like arrays and graphs.
method Generalized empirical Bayes approach based on probabilistic symmetry.
result BEB outperforms existing methods in denoising arrays and spatial data.
Improved Naive Bayes classifier with neural network models.
problem Limited complexity handling and independence assumption in Naive Bayes.
method Introducing Neural Naive Bayes and Neural Pooled Markov Chain models.
result Error rate reduced by 4.5 on IMDB dataset.
Stochastic variational inference for Bayesian deep neural network (DNN) requires specifying priors and approximate posterior distributions over neural network weights. Specifying meaningful weight priors is a challenging problem, particularly for scaling variational inference to deeper architectures involving high dime…
Unified empirical and variational Bayes for unnormalized densities.
problem Approximating unnormalized densities using latent variable models.
method Formulate a latent variable model for Y=X+N(0,σ2Id), use ELBO as parametrization of Y's energy function, and estimate X with empirical Bayes least-squares. result UVB has higher capacity to approximate energy functions than MLPs in DEEN.
New algorithm optimizes PAC-Bayes bound without surrogate loss.
problem Mismatch between optimisation objective and generalisation bound in stochastic neural networks.
method Proposes a novel training algorithm that optimizes the PAC-Bayesian bound directly.
result Empirical results show improved performance over existing PAC-Bayesian training methods.
New PAC-Bayes training method improves model generalization for unbounded loss.
problem Improving generalization of complex models under unbounded loss.
method Established new PAC-Bayes bound for unbounded loss, jointly training prior and posterior.
result Outperforms existing PAC-Bayes training algorithms and matches ERM accuracy.
The paper improves risk certificate tightness for neural networks using PAC-Bayes bounds.
problem Improving the usability of risk certificates for neural networks based on PAC-Bayes bounds.
method Theoretical contributions including KL divergence bounds, efficient methodology for optimization, and methods for optimizing non-differentiable objectives.
result First non-vacuous generalization bounds on CIFAR-10 for neural networks.
LLEB uses a learned prior to quantify neural network uncertainty.
problem Quantifying uncertainty in neural network predictions.
method LLEB uses a learnable prior as a normalizing flow to maximize the evidence lower bound.
result LLEB performs on par with existing approaches in uncertainty quantification.
Empirical Bayes rates via variational approximations and prior decomposition.
problem Nonparametric and high-dimensional inference convergence rates.
method Variational perspective and prior decomposition.
result Empirical Bayes posterior rates derived from variational Bayes.
We unify kernel density estimation and empirical Bayes and address a set of problems in unsupervised learning with a geometric interpretation of those methods, rooted in the concentration of measure phenomenon. Kernel density is viewed symbolically as X⇀Y where the rand…
Improves model accuracy for neural nets in stochastic dynamics with partial prior knowledge.
problem Stability and accuracy in neural nets modeling stochastic dynamics with many parameters.
method Three steps: probabilistic weights, partial knowledge incorporation, and PAC-Bayesian training.
result Improved model fit with partial and noisy prior knowledge.
Deep ensembles mimic Bayesian averaging with learned priors.
problem Quantifying uncertainty in neural networks.
method Showed deep ensembles perform exact Bayesian averaging with an implicitly learned data-dependent prior.
result Deep ensembles are Bayesian and provide an explanation for their strong empirical performance.
New estimators outperform maximum likelihood without hyper-parameter estimation.
problem Improving system identification performance without hyper-parameter estimation.
method Developed generalized Bayes and closed-form biased estimators using excess MSE.
result New estimators have comparable performance to empirical-Bayes-based regularized estimator.
Accurate statistical models of neural spike responses can characterize the information carried by neural populations. But the limited samples of spike counts during recording usually result in model overfitting. Besides, current models assume spike counts to be Poisson-distributed, which ignores the fact that many neur…
New work shows limits of certifying neural network robustness.
problem Certified training improves robustness but decreases accuracy.
method Bayes error analysis to investigate robustness limits.
result Upper bound for certified robust accuracy established.
New empirical PAC-Bayes bound for Markov chains with finite state space.
problem Lack of empirical bounds for Markov chains with temporal dependence.
method Proved a new PAC-Bayes bound for Markov chains, providing an empirical pseudo-spectral gap.
result First fully empirical PAC-Bayes bound for Markov chains with finite state space.
Bayesian neural networks (BNNs) hold great promise as a flexible and principled solution to deal with uncertainty when learning from finite data. Among approaches to realize probabilistic inference in deep neural networks, variational Bayes (VB) is theoretically grounded, generally applicable, and computationally effic…
Neural Bayes simplifies computing complex stats for unsupervised learning.
problem Computing mutual information and optimal labeling of disjoint manifolds in unsupervised learning.
method Parameterization using neural networks to express statistical quantities in closed form.
result Neural Bayes enables efficient computation of mutual information and optimal labeling of disjoint manifolds.
Bayes-optimal learning of a neural network with quadratic activations is achieved with GAMP-RIE.
problem Learning a neural network with quadratic activations from quadratic samples.
method Combining approximate message passing with rotationally invariant matrix denoising.
result Derives a closed-form expression for Bayes-optimal test error.
Nash integrates covariate-specific side info into sparse regression via neural networks.
problem Sparse linear regression struggles with covariates exhibiting structure or coming from heterogeneous sources.
method Neural Adaptive Shrinkage (Nash) framework that integrates side information into sparse regression via neural networks. Uses split variational empirical Bayes algorithm.
result Nash improves accuracy and adaptability over existing methods in real data experiments.
New method for high-dimensional linear regression using empirical Bayes.
problem Estimating prior in high-dimensional linear regression.
method Variational empirical Bayes approach with NPMLE and mean field approximation.
result Established asymptotic consistency and computational efficiency of the method.
In this study, we consider an empirical Bayes method for Boltzmann machines and propose an algorithm for it. The empirical Bayes method allows estimation of the values of the hyperparameters of the Boltzmann machine by maximizing a specific likelihood function referred to as the empirical Bayes likelihood function in t…
Unified smoothing for robust classification improves accuracy.
problem Improving robustness of classifiers against adversarial attacks.
method Learned smoothed densities and randomized smoothing.
result Provable robust accuracies higher than state-of-the-art defenses.
Optimal classifiers derived from GMMs are approximated by deep neural networks.
problem Binary classification of high-dimensional overlapping Gaussian mixtures.
method Closed-form expressions for Bayes optimal decision boundaries derived from GMMs' eigenstructure. Empirical validation through synthetic and real-world data.
result Deep neural networks approximate optimal classifiers for GMMs, with decision thresholds related to covariance eigenvectors.
New method improves training stochastic neural networks with tighter guarantees.
problem Training stochastic neural networks with provable guarantees.
method Developed partially-aggregated estimators and reformulated PAC-Bayesian bounds.
result Derives a differentiable objective leading to tighter generalisation guarantees.
Develops a fast variational approximation for high-dimensional empirical Bayes posteriors.
problem Optimal posterior computation in high-dimensional settings with prior tails effect.
method Variational approximation of empirical Bayes posterior with data-driven centers and thin-tailed conjugate priors.
result Retains optimal concentration rate properties and superior performance compared to existing methods.
The paper analyzes high-dimensional linear regression using parametric empirical Bayes methods.
problem Estimation of i.i.d. priors in high-dimensional Bayesian linear regression with random design.
method Parametric empirical Bayes estimation, variational lower bound maximization, phase transition analysis.
result The vEB estimator is information theoretically optimal up to p=o(n2/3) but sub-optimal in higher dimensions. EB-PCA reduces noise in high-dimensional PCA by estimating a joint prior distribution.
problem High-dimensional PCA noise in samples comparable to or larger than data.
method Empirical Bayes PCA using Kiefer-Wolfowitz MLE, random matrix theory, and AMP algorithm.
result EB-PCA achieves Bayes-optimal accuracy in spiked models and significantly improves over PCA in simulations and real data.
A new sequential method estimates Poisson means in streaming data, achieving optimality and efficiency.
problem Estimating Poisson means in a streaming, or online, framework.
method A quasi-Bayesian approach based on Newton's algorithm for a sequential estimate.
result Established frequentist guarantees including consistency and asymptotic optimality.
Neural-g models mixtures of densities with flexibility and accuracy.
problem Accurately estimating prior densities in g-models. method Neural network with softmax output for valid probability densities.
result Neural-g captures various prior shapes including flat, heavy-tailed, and discontinuous.
We study the rates of convergence from empirical surrogate risk minimizers to the Bayes optimal classifier. Specifically, we introduce the notion of \emph{consistency intensity} to characterize a surrogate loss function and exploit this notion to obtain the rate of convergence from an empirical surrogate risk minimizer…
New algorithm reduces bias in trained models, near-optimal performance proven.
problem Reduction of bias in trained machine learning models.
method Scalable post-processing algorithm for debiasing trained models, including deep neural networks (DNNs).
result Proven to be near-optimal by bounding its excess Bayes risk.
EB-VAE combines tumor growth and dropout data for personalized treatment response modeling.
problem Challenges in integrating longitudinal tumor measurements, dropout information, and genetic covariates.
method Extended EB-VAE framework to jointly model longitudinal and time-to-event data, incorporating dropout hazard and genetic covariates.
result Hybrid decoder formulation yields consistent treatment-effect parameters and prior predictive performance comparable to neural decoder.
Paper presents new training methods for neural networks with tighter risk certificates.
problem Training probabilistic neural networks with tighter risk certificates.
method Derived from PAC-Bayes bounds, two training objectives implemented for the first time in neural networks.
result Competitive test set errors and non-vacuous risk bounds with tighter values than previous results.
Solution to sparse PCA tuning problem using Empirical Bayes.
problem Sparse PCA multiple tuning problem (MTP).
method Empirical Bayes covariance decomposition for penalized PCA.
result Empirical Bayes approach efficiently solves MTP in sparse PCA.
Meta-learning bounds derived using PAC-Bayes theory for improved generalization.
problem Uncertainty in generalization performance for meta-learning with new tasks.
method PAC-Bayes relative entropy bounds and empirical risk minimization (ERM) method.
result Competitive generalization performance and rapid convergence with data-dependent prior.
Bayesian framework for policy learning in decision problems.
problem Maximizing expected welfare in decision-making problems.
method Loss-based Bayesian updating and squared-loss surrogate for welfare maximization.
result General Bayes posterior over decision rules with Gaussian pseudo-likelihood interpretation.
Improved neural active learning algorithms reduce regret and improve performance.
problem Improving performance and reducing regret in neural active learning for non-parametric streaming data.
method Introducing two new regret metrics and leveraging NNs for both exploitation and exploration. The algorithm uses tailored query decision-makers and full feedback.
result Achieved an instance-dependent regret upper bound improving by a multiplicative factor of O(logT) and removing the curse of dimensionality. New method predicts binary matrix entries using empirical Bayes and low-rank structure.
problem Predicting unobserved entries in binary matrices.
method Empirical Bayes method motivated by Efron--Morris estimator, exploiting low-rank structure.
result Superior performance in predictive accuracy, calibration, and efficiency compared to existing methods.
Graph neural networks extend neural Bayes estimators to irregular spatial data.
problem Estimating parameters from irregular spatial data with computational efficiency.
method Employing graph neural networks to approximate Bayes estimators for irregular spatial data.
result Extending neural Bayes estimation to irregular spatial data with computational benefits.
VL finds flatter solutions at edge of stability, matching theory with practice.
problem Understanding implicit regularization in deep learning.
method Edge of Stability framework, controlling variational posterior shape and sample number.
result VL finds even flatter solutions than gradient descent.
PAC-Bayes bounds for Gibbs posteriors derived via singular learning theory.
problem Generalization bounds for overparameterized models with data-dependent priors.
method Explicit non-asymptotic PAC-Bayes bounds using singular learning theory.
result Explicit posterior-averaged risk bounds for overparameterized models.
Flexible empirical Bayes for large-scale multiple linear regression.
problem Large-scale multiple linear regression with flexible priors and efficient computation.
method Adaptive shrinkage priors combined with variational approximations for hyperparameter estimation.
result The posterior mean from the empirical Bayes method solves a penalized regression problem.
Neural Bayes methods simplify fitting complex bivariate extremal models.
problem Inference on complex multivariate extremal dependence models with computationally expensive likelihood functions.
method Use neural networks to approximate Bayes estimators and classifiers for model selection.
result Proposed neural Bayes methods enable routine implementation of complex extreme-value dependence models.
New inequality for ternary variables improves on existing measures.
problem Analyzing excess losses and weighted majority votes with ternary random variables.
method Developed a split-kl inequality and its PAC-Bayes extension.
result Outperforms existing inequalities in certain regimes.
We study the Nonparametric Maximum Likelihood Estimator (NPMLE) for estimating Gaussian location mixture densities in d-dimensions from independent observations. Unlike usual likelihood-based methods for fitting mixtures, NPMLEs are based on convex optimization. We prove finite sample results on the Hellinger accurac…
IIC provides a PAC-Bayes bound for interpolating models, revealing factors affecting generalization.
problem Theoretical challenges in understanding overparameterized models and their performance.
method PAC-Bayesian perspective applied to the Interpolating Information Criterion (IIC).
result Test error for overparameterized models achieving zero training error depends on various factors.