We explore the family of methods "PAC-Bayes with Backprop" (PBB) to train probabilistic neural networks by minimizing PAC-Bayes bounds. We present two training objectives, one derived from a previously known PAC-Bayes bound, and a second one derived from a novel PAC-Bayes bound. Both training objectives are evaluated o…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,742 papers · 148 categories
Trend · papers per month
48 results for “Bayes by Backprop”
Hidden Markov Neural Networks balance adaptation and forgetting in time-series data.
problem Balancing adaptation to new data and forgetting outdated information in time-series forecasting.
method Modeling weights as hidden states of a Hidden Markov model, using a filtering algorithm for learning a variational approximation of the posterior distribution over weights, and employing sequential Bayes by Backprop with variational DropConnect for regularization.
result Achieves strong predictive performance and effective uncertainty quantification on various tasks.
Paper derives CLT for Bayesian neural networks trained with variational inference.
problem Analyzing the fluctuation behavior of Bayesian neural networks trained with different variational inference schemes.
method Rigorous derivation of CLT for three variational inference schemes: idealized, Bayes-by-Backprop, and Minimal VI.
result Minimal VI scheme has larger variances but is more computationally efficient.
Modern neural networks tend to be overconfident on unseen, noisy or incorrectly labelled data and do not produce meaningful uncertainty measures. Bayesian deep learning aims to address this shortcoming with variational approximations (such as Bayes by Backprop or Multiplicative Normalising Flows). However, current appr…
We present a new algorithm that significantly improves the efficiency of exploration for deep Q-learning agents in dialogue systems. Our agents explore via Thompson sampling, drawing Monte Carlo samples from a Bayes-by-Backprop neural network. Our algorithm learns much faster than common exploration strategies such as …
We introduce a new, efficient, principled and backpropagation-compatible algorithm for learning a probability distribution on the weights of a neural network, called Bayes by Backprop. It regularises the weights by minimising a compression cost, known as the variational free energy or the expected lower bound on the ma…
Bayesian neural networks predict stress fields and uncertainty in materials.
problem Uncertainty in stress field predictions for complex materials.
method Modified Bayesian U-net architecture with three inference algorithms.
result High accuracy predictions and interpretable uncertainty estimates.
This paper introduces Selective-Backprop, a technique that accelerates the training of deep neural networks (DNNs) by prioritizing examples with high loss at each iteration. Selective-Backprop uses the output of a training example's forward pass to decide whether to use that example to compute gradients and update para…
BYOV combines SSL and Bayesian methods for uncertainty estimation.
problem Model uncertainty in applications.
method Combines Bootstrap Your Own Latent (BYOL) and Bayes by Backprop (BBB).
result BYOV improves model calibration and reliability with various augmentations.
A new method reduces deep learning training costs by 92%.
problem High computational cost of backpropagation in deep neural networks.
method Dithered backprop with non-subtractive dither quantization.
result 92% sparsity with minimal accuracy loss.
We introduce a novel uncertainty estimation for classification tasks for Bayesian convolutional neural networks with variational inference. By normalizing the output of a Softplus function in the final layer, we estimate aleatoric and epistemic uncertainty in a coherent manner. The intractable posterior probability dis…
In statistical dialogue management, the dialogue manager learns a policy that maps a belief state to an action for the system to perform. Efficient exploration is key to successful policy optimisation. Current deep reinforcement learning methods are very promising but rely on epsilon-greedy exploration, thus subjecting…
Backpropagation is explained as a diffusion process in neural networks.
problem The biological plausibility of Backpropagation is questioned.
method Demonstrated that time-delayed neurons and forward-backward waves approximate the gradient in deep networks.
result Backpropagation can be interpreted as a diffusion process, approximating the gradient for non-fast inputs.
Several recent papers investigate Active Learning (AL) for mitigating the data dependence of deep learning for natural language processing. However, the applicability of AL to real-world problems remains an open question. While in supervised learning, practitioners can try many different methods, evaluating each agains…
Representations are fundamental to artificial intelligence. The performance of a learning system depends on the type of representation used for representing the data. Typically, these representations are hand-engineered using domain knowledge. More recently, the trend is to learn these representations through stochasti…
Gradient-free deep learning for large datasets.
problem Training deep neural networks on large-scale datasets is resource-intensive and requires specialized techniques.
method Recursive Local Representation Alignment (RLRA) for gradient-free training.
result RLRA achieves comparable performance to backprop while converging faster and being parallelizable.
Training deep neural networks (DNNs) efficiently is a challenge due to the associated highly nonconvex optimization. The backpropagation (backprop) algorithm has long been the most widely used algorithm for gradient computation of parameters of DNNs and is used along with gradient descent-type algorithms for this optim…
In real-world scenarios, it is appealing to learn a model carrying out stochastic operations internally, known as stochastic computation graphs (SCGs), rather than learning a deterministic mapping. However, standard backpropagation is not applicable to SCGs. We attempt to address this issue from the angle of cost propa…
Law of Large Numbers for Bayesian two-layer Neural Network trained with Variational Inferencestat.ML
The paper analyzes Bayesian neural networks trained with VI, proving a law of large numbers for different schemes.
problem Training Bayesian neural networks with variational inference.
method Analyzes three training schemes: exact estimation, Bayes by Backprop, and Minimal VI.
result All training schemes converge to the same mean-field limit.
The question why deep learning algorithms generalize so well has attracted increasing research interest. However, most of the well-established approaches, such as hypothesis capacity, stability or sparseness, have not provided complete explanations (Zhang et al., 2016; Kawaguchi et al., 2017). In this work, we focus on…
A new method improves meta-learning for few-shot learning.
problem Catastrophic forgetting and robustness issues in meta-learning.
method CML-BGNN: Bayesian Graph Neural Networks for continual meta-learning.
result Improves performance by 42.8% on miniImageNet 5-way 1-shot classification.
This paper defines a new learning architecture, Layered Self-Organizing Maps (LSOMs), that uses the SOM and supervised-SOM learning algorithms. The architecture is validated with the MNIST database of hand-written digit images. LSOMs are similar to convolutional neural nets (covnets) in the way they sample data, but di…
This work tackles exploding inverses in INNs, revealing and mitigating their numerical non-invertibility.
problem Exploding inverses in INNs cause numerical non-invertibility, leading to failures in various tasks.
method Derived bi-Lipschitz properties of INN building blocks, proposed regularizers for local invertibility, and stable INN designs for global invertibility.
result Bi-Lipschitz properties and stable INN designs are crucial for addressing numerical non-invertibility.
Exponential-family harmoniums (EFHs), which extend restricted Boltzmann machines (RBMs) from Bernoulli random variables to other exponential families (Welling et al., 2005), are generative models that can be trained with unsupervised-learning techniques, like contrastive divergence (Hinton et al. 2006; Hinton, 2002), a…
Artificial Neural Networks are connectionist systems that perform a given task by learning on examples without having prior knowledge about the task. This is done by finding an optimal point estimate for the weights in every node. Generally, the network using point estimates as weights perform well with large datasets,…
Backprop Evolutioncs.NE
The back-propagation algorithm is the cornerstone of deep learning. Despite its importance, few variations of the algorithm have been attempted. This work presents an approach to discover new variations of the back-propagation equation. We use a domain specific lan- guage to describe update equations as a list of primi…
DeConvNet, Guided BackProp, LRP, were invented to better understand deep neural networks. We show that these methods do not produce the theoretically correct explanation for a linear model. Yet they are used on multi-layer networks with millions of parameters. This is a cause for concern since linear models are simple …
New uniqueness concept for adversarial Bayes classifier.
problem Understanding adversarial Bayes classifiers in binary classification.
method Developed a new notion of uniqueness and analyzed it for a family of one-dimensional data distributions.
result Improved regularity of adversarial Bayes classifiers as perturbation radius increases.
Empirical Bayes rates via variational approximations and prior decomposition.
problem Nonparametric and high-dimensional inference convergence rates.
method Variational perspective and prior decomposition.
result Empirical Bayes posterior rates derived from variational Bayes.
The study compares clustering risk in Hidden Markov and i.i.d. models, showing the Bayes classifier is nearly optimal.
problem Comparing clustering risk in Hidden Markov and i.i.d. models.
method Analysis of Bayes risk, theoretical bounds, and simulations.
result The Bayes classifier is nearly optimal for clustering in both Hidden Markov and i.i.d. models.
Despite its simplicity, the naive Bayes classifier has surprised machine learning researchers by exhibiting good performance on a variety of learning problems. Encouraged by these results, researchers have looked to overcome naive Bayes primary weakness - attribute independence - and improve the performance of the algo…
Researchers prove NP-hardness of learning parameter-bounded Bayes nets.
problem Learning parameter-bounded Bayes nets is computationally hard.
method Proved NP-hardness of learning parameter-bounded Bayes nets and a promise search variant.
result Proved NP-hardness of a promise search variant of LEARN.
Deep Bayes Factorsstat.ME
Deep learning improves Bayes factor computation for likelihood-free models.
problem Computing Bayes factors for likelihood-free models is challenging.
method Proposes a deep learning estimator of Bayes factors using simulated data.
result Establishes consistency of the Deep Bayes Factor estimator.
Meta-learning bounds derived using PAC-Bayes theory for improved generalization.
problem Uncertainty in generalization performance for meta-learning with new tasks.
method PAC-Bayes relative entropy bounds and empirical risk minimization (ERM) method.
result Competitive generalization performance and rapid convergence with data-dependent prior.
PAC-Bayes framework fails on simple 1D linear classification task.
problem Proving the learnability of simple 1D linear classification tasks using PAC-Bayes bounds.
method Demonstrated a specific 1D linear classification task that PAC-Bayes cannot analyze.
result PAC-Bayes framework cannot prove learnability of simple 1D linear classification tasks.
New method adapts without backprop, faster and better.
problem Efficient domain adaptation without source data.
method Computing class prototypes from pre-trained model.
result Significant accuracy improvements over pre-trained model.
PAC-Bayes bound requires prior to place mass on high-performing predictors.
problem Explaining generalization in machine learning.
method Analyzing necessary conditions for PAC-Bayes bounds to provide meaningful generalization guarantees.
result Achieving a target generalisation level requires the prior to place sufficient mass on high-performing predictors.
New estimators outperform maximum likelihood without hyper-parameter estimation.
problem Improving system identification performance without hyper-parameter estimation.
method Developed generalized Bayes and closed-form biased estimators using excess MSE.
result New estimators have comparable performance to empirical-Bayes-based regularized estimator.
Generalized Naive Bayesstat.ML
Paper introduces Generalized Naive Bayes for better data fitting.
problem Improving Naive Bayes for better data fitting.
method Developed new greedy and optimal algorithms for GNB.
result Proves GNB fits data at least as well as classical NB.
Discriminative classification with generative features: bridging Naive Bayes and logistic regressionstat.ML
Smart Bayes integrates generative and discriminative features for improved classification.
problem Improving classification performance by combining generative and discriminative modeling.
method Integrates generative likelihood-ratio features into a logistic-regression-style classifier.
result Often outperforms logistic regression and Naive Bayes in simulations and real data.
New method reduces computational cost for estimating PAC-Bayes bounds.
problem High computational cost in estimating PAC-Bayes bounds.
method General alternative method that makes computational savings.
result Reduces computational cost on the order of the dataset size.
This work bounds classification error in machine learning for low Bayes error conditions.
problem Understanding the error mismatch between Bayes error and model-based classification error.
method Applying classification error bounds to study the relationship with Kullback-Leibler divergence and proposing a linear approximation for low Bayes error conditions.
result A linear approximation of the classification error bound for low Bayes error conditions is proposed.
Improved Naive Bayes classifier with neural network models.
problem Limited complexity handling and independence assumption in Naive Bayes.
method Introducing Neural Naive Bayes and Neural Pooled Markov Chain models.
result Error rate reduced by 4.5 on IMDB dataset.
Study calculates Bayes risk for semi-supervised learning with uncertain labels.
problem Uncertain labeling in semi-supervised classification.
method Gaussian mixture model, Bayes risk computation, comparison with algorithm performance.
result New insights into semi-supervised learning algorithm performance.
We extend a recently proposed 1-nearest-neighbor based multiclass learning algorithm and prove that our modification is universally strongly Bayes-consistent in all metric spaces admitting any such learner, making it an "optimistically universal" Bayes-consistent learner. This is the first learning algorithm known to e…
Unified derivation of PAC-Bayes and MI bounds for general VC classes with fast rates.
problem Generalization bounds for machine learning models with VC classes.
method Unified derivation of conditional PAC-Bayesian and mutual information bounds, including MAC-Bayesian bounds.
result Nontrivial bounds for general VC classes and faster rates for specific conditions.
Bayes classifier cannot be learned from noisy labels without knowing noise distribution.
problem Learning a Bayes classifier from noisy labels when the noise distribution is unknown.
method Demonstrates the identifiability issues and proposes a simple algorithm for learning the Bayes decision rule.
result The Bayes decision rule is generally unidentified and cannot be learned without knowing the noise distribution.
This paper improves meta-learning by developing new PAC-Bayes bounds.
problem Meta-learning generalization gap across multiple tasks.
method Upper bounding convex functions linking environment and task-level losses.
result New PAC-Bayes bounds for meta-learning with improved algorithms.