Dropout is one of the key techniques to prevent the learning from overfitting. It is explained that dropout works as a kind of modified L2 regularization. Here, we shed light on the dropout from Bayesian standpoint. Bayesian interpretation enables us to optimize the dropout rate, which is beneficial for learning of wei…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
169,291 papers · 148 categories
Trend · papers per month
48 results for “Bayesian Dropout”
Bayesian Dropoutstat.ML
Dropout has recently emerged as a powerful and simple method for training neural networks preventing co-adaptation by stochastically omitting neurons. Dropout is currently not grounded in explicit modelling assumptions which so far has precluded its adoption in Bayesian modelling. Using Bayesian entropic reasoning we s…
Bayesian inference using stochastic neural networks ensembles.
problem Approximating Bayesian posterior distributions.
method Formulate stochastic ensembles of neural networks, train with variational inference, and evaluate using Monte Carlo dropout.
result Stochastic ensembles provide more accurate posterior estimates than other methods.
Qualitative analysis of MC dropout for NN model uncertainty.
problem Measuring uncertainty in neural network models.
method Mathematical formulation of Monte Carlo dropout and its benefits/costs in NN models.
result Potential benefits and associated costs of using MC dropout in NN models.
Improves Bayesian neural networks inference efficiency and accuracy.
problem Inflexibility of factorized structure in Dropout posterior.
method Introduces Variational Structured Dropout (VSD) with orthogonal transformation.
result VSD induces adaptive regularization and better generalization.
Variational dropout and ARD are shown to be equivalent, improving sparsity in neural networks.
problem Improper priors in variational dropout models and theoretical issues.
method Applying ARD to Bayesian DNNs with Gaussian approximations and exploring hierarchical priors.
result Variational dropout and ARD objectives are equivalent, with improved sparsity.
Structured dropout improves confidence calibration in convolutional neural networks.
problem Poor confidence calibration in convolutional neural networks using dropout.
method Structured dropout to promote model diversity and improve calibration.
result Structured dropout reduces calibration error compared to naive dropout.
We show that a neural network with arbitrary depth and non-linearities, with dropout applied before every weight layer, is mathematically equivalent to an approximation to a well known Bayesian model. This interpretation might offer an explanation to some of dropout's key properties, such as its robustness to over-fitt…
New method uses dropout to optimize high-dimensional Bayesian functions.
problem Challenges in scaling Bayesian optimization to high dimensions.
method Proposes a new method using dropout to optimize a subset of variables.
result Derives theoretical bounds for regret and demonstrates algorithm efficacy.
An adaptive dropout approach improves high-dimensional Bayesian optimization.
problem High-dimensional black-box optimization problems.
method Adaptive dropout of variables in the acquisition function.
result AdaDropout effectively tackles high-dimensional challenges and improves solution quality.
MC Dropout is re-evaluated as not Bayesian, affecting predictive posterior and multimodality.
problem MC Dropout's Bayesian properties and predictive posterior are questioned.
method Re-evaluation of MC Dropout's properties, including a new VI engine in pytorch.
result MC Dropout does not produce a faithful Bayesian predictive posterior.
The paper improves predictive uncertainty estimation using Dropout and Hamiltonian Monte Carlo.
problem Improving predictive uncertainty in classification tasks.
method Combining Dropout regularization and Hamiltonian Monte Carlo for Bayesian inference.
result The method effectively estimates predictive accuracy and generalizes better for difficult test examples.
Recurrent neural networks (RNNs) stand at the forefront of many recent developments in deep learning. Yet a major difficulty with these models is their tendency to overfit, with dropout shown to fail when applied to recurrent layers. Recent results at the intersection of Bayesian modelling and deep learning offer a Bay…
Improved dropout inference for Bayesian neural networks using alpha-divergences.
problem Uncertainty underestimation in dropout variational inference.
method Proposed a re-parametrisation of alpha-divergence objectives for dropout networks.
result Improved uncertainty estimates and accuracy compared to VI in dropout networks.
Concrete Dropoutstat.ML
New dropout method improves uncertainty estimates in deep models and RL.
problem Problems with current dropout methods in obtaining well-calibrated uncertainty estimates.
method Continuous relaxation of dropout masks, principled optimisation objective.
result Improved performance and better calibrated uncertainties in large models and RL.
Advanced Dropout improves DNN performance without requiring model-specific dropout techniques.
problem Overfitting in deep neural networks due to lack of data.
method Model-free, end-to-end training with adaptive dropout rate optimization.
result Advanced Dropout outperforms other techniques on various datasets.
New method reduces parameter overhead for Bayesian neural networks.
problem High parameter overhead and difficulty of implementation in variational Bayesian neural networks.
method Constructs a general variational family for ensemble-based Bayesian neural networks that works well with batch normalization layers.
result Improves predictive accuracy and achieves almost perfect calibration on a ResNet-18 trained with ImageNet.
Bayesian sparsification reduces memory usage in RNNs without significant loss of accuracy.
problem High memory usage in RNNs for text analysis tasks.
method Apply Sparse Variational Dropout and Binary Variational Dropout to sparsify RNNs.
result Achieve 99.5% sparsity on sentiment analysis with no quality drop and up to 87% on language modeling with slight loss of accuracy.
Proposes LBD for more robust deep learning models.
problem Improving robustness and uncertainty in deep learning models.
method Model-agnostic learnable Bernoulli dropout with joint optimization of dropout rates.
result Superior performance compared to other dropout schemes.
Researchers fix issues with variational Bayesian dropout.
problem Variational Bayesian dropout's theoretical framework has issues.
method Proposed new approximate inference objective called Quasi-KL (QKL).
result QKL addresses singularity issue and leads to Principal Component Analysis solution.
Enhances uncertainty estimation in neural networks using Dirichlet-based MC Dropout.
problem Deterministic predictions without uncertainty estimates in neural networks.
method Integrates Dirichlet-based framework within Monte Carlo Dropout.
result Improves quality of uncertainty estimates in deep learning models.
VBD improves variational dropout by using a hierarchical prior, enabling better regularization.
problem Improper log-uniform prior in VD causes ill-posed posterior inference.
method Introduces a hierarchical prior with a zero-mean Gaussian distribution and a uniform hyper-prior.
result VBD enables well-posed posterior inference and superior regularization performance.
Evaluates uncertainty quality in neural networks using anomaly detection.
problem Evaluating the quality of uncertainty in neural networks.
method Extract uncertainty measures, use them as features for an anomaly detector, and compare different neural network models.
result Bayesian Dropout and OSBA provide better uncertainty information than Maximum Likelihood, and are faster.
CODA uses a new dropout technique inspired by constructivism learning to improve deep learning performance.
problem Existing dropout methods fail to differentiate among instances, leading to overfitting.
method CODA incorporates structural information and uses a Bayesian nonparametric method to create a better dropout technique.
result CODA outperformed other state-of-the-art dropout techniques on 5 real-world datasets.
A fast single-shot MC dropout method for neural networks.
problem Inability of DNNs to provide uncertainty measures for new situations.
method Analytically approximates MC dropout for fully connected networks.
result Approach preserves BDNN advantages while being faster.
Dropout increases the generalization of neural networks by expanding the weight space.
problem Understanding and improving the generalization of neural networks.
method Introducing weight expansion and showing that dropout leads to it.
result Dropout increases the generalization of neural networks by expanding the weight space.
Deep learning tools have gained tremendous attention in applied machine learning. However such tools for regression and classification do not capture model uncertainty. In comparison, Bayesian models offer a mathematically grounded framework to reason about model uncertainty, but usually come with a prohibitive computa…
Variational Dropout sparsifies deep neural networks by reducing parameter count.
problem Overfitting and high parameter count in deep neural networks.
method Extending Variational Dropout to unbounded dropout rates, reducing gradient variance, and applying individual dropout rates per weight.
result Significant reduction in parameter count (up to 280 times) with minimal accuracy loss.
New research shows Gaussian dropout in neural networks is not a proper Bayesian method.
problem Gaussian dropout's Bayesian interpretation is flawed and leads to improper posteriors.
method Analyzed the log-uniform prior and correlated weight noise approximation in Gaussian dropout.
result Gaussian dropout does not provide proper Bayesian inference and can lead to overfitting.
Bayesian approach improves activity recognition accuracy and uncertainty quantification.
problem Lack of predictive uncertainty in multimodal audiovisual activity recognition.
method Uncertainty aware multimodal Bayesian fusion framework combining deterministic and variational layers.
result Improved precision-recall AUC by 10.2% on MiT dataset.
A fast method for neural networks that provides uncertainty measures.
problem Inability of neural networks to provide reliable uncertainty measures for new situations.
method Moment propagation (MP) for analytically approximating MC dropout.
result Single-shot MC dropout approximation as fast as NNs with similar uncertainty measures.
Improved neural network predicts tropical storm trajectories and Bayesian intervals.
problem Accurately predicting the trajectories of tropical storms to prevent damage.
method Developed an improved RNN model with dropout to predict Bayesian intervals.
result Neural network dropout values significantly affect prediction accuracy and intervals.
New Bayesian model injects noise to improve neural network sparsity and acceleration.
problem Improving neural network sparsity and acceleration.
method Proposes a new Bayesian model that injects noise to neurons outputs while keeping weights unregularized, using log-normal multiplicative noise.
result Provides significant acceleration on deep neural architectures.
Spiking neural networks enable efficient approximate Bayesian inference via permanent dropout.
problem Efficient uncertainty quantification in neural network predictions for critical tasks.
method Conversion of classical neural networks to spiking neural networks, applying permanent dropout for inference.
result Predictive distributions from spiking neural networks using permanent dropout are nearly identical to those from classical networks.
Study on MC dropout in wide neural networks and its convergence to Gaussian processes.
problem Understanding the behavior of Monte Carlo dropout in wide neural networks.
method Rigorously studied the limiting distribution of wide untrained NNs under dropout, proving convergence to Gaussian processes. Investigated correlations and non-Gaussian behavior in finite width NNs.
result Wide untrained neural networks under dropout converge to Gaussian processes for fixed sets of weights and biases.
The paper calibrates uncertainty in dropout variational inference models.
problem Miscalibration of model uncertainty in dropout variational inference.
method Logit scaling methods are extended to recalibrate model uncertainty.
result Logit scaling reduces miscalibration, improving reliability of predictions.
We investigate a local reparameterizaton technique for greatly reducing the variance of stochastic gradients for variational Bayesian inference (SGVB) of a posterior over model parameters, while retaining parallelizability. This local reparameterization translates uncertainty about global parameters into local noise th…
New method calibrates Bayesian neural network approximations for better task-specific predictions.
problem Inaccurate approximations of Bayesian neural networks without task-specific knowledge.
method Introduces a loss-calibrated evidence lower bound informed by Bayesian decision theory.
result Achieves higher utility for applications with asymmetric utility functions.
Radial Bayesian Neural Networks: Beyond Discrete Support In Large-Scale Bayesian Deep Learningstat.ML
Radial BNNs offer a scalable, continuous weight distribution for Bayesian deep learning.
problem Discrete support in Bayesian deep learning methods like MC dropout.
method Radial BNNs with full support over weight-space.
result Radial BNNs outperform discrete-support methods in real-world applications.
Privacy-preserving dropout technique for neural networks.
problem Overfitting in deep neural networks and privacy of training data.
method Modified variational dropout with concentrated differential privacy.
result The technique provides differential privacy for neural networks.
Bayesian active learning improves natural language processing models.
problem Lack of model comparison in AL for NLP tasks.
method Large-scale empirical study of Bayesian active learning with Dropout and Bayes-by-Backprop uncertainty estimates.
result Bayesian active learning by disagreement significantly improves NLP model performance.
This paper calibrates uncertainty in dropout variational inference models.
problem Uncertainty in variational inference with dropout is poorly calibrated.
method Temperature scaling is extended to dropout variational inference.
result Temperature scaling reduces miscalibration of uncertainty.
Differentially private dropout technique preserves privacy in neural network training.
problem Preserving privacy in large datasets used for neural network training.
method Introduces a Bayesian dropout technique that adds intrinsic noise for regularization and differential privacy.
result Demonstrates that the iterative nature of neural network training can be handled with a relaxed differential privacy concept.
Batch normalisation doesn't affect variational inference but fails for larger batch sizes.
problem Failure of Monte Carlo Batch Normalisation (MCBN) for capturing epistemic uncertainty in larger batch sizes.
method Investigated MCBN as an approximate inference technique for Bayesian neural networks, showing its limitations and providing insights for improvement.
result For larger batch sizes, MCBN fails to capture epistemic uncertainty, requiring the batch size to be a variational parameter.
Bayesian Layers adds uncertainty to neural networks, enabling faster experimentation and scalability.
problem Enabling neural networks to quantify uncertainty in predictions.
method Drop-in replacements for common layers, capturing uncertainty over weights, activations, etc.
result Bayesian Layers can fit large models like 5-billion parameter Bayesian Transformers.
Bayesian deep learning predicts price movements from LOBs, improving trading profits.
problem Predicting price movements from limit order books for better trading decisions.
method Applies dropout variational inference to deep neural networks, using uncertainty information for position sizing.
result Bayesian techniques improve predictive performance and deliver useful uncertainty information for trading.
Locally learned synaptic failure enables complete Bayesian inference.
problem Bayesian inference in neural networks.
method Biologically constrained neural network with synaptic failure and lateral inhibition.
result Synaptic failure allows sampling from both epistemic and aleatoric distributions.
Proposes a new method for uncertainty estimation in neural networks.
problem Estimating uncertainty in neural networks.
method Samples outputs from Gaussian distributions parametrized by mean and variance sub-layers.
result Achieves better uncertainty quality than other methods.