Bayesian sparsification reduces memory usage in RNNs without significant loss of accuracy.
problem High memory usage in RNNs for text analysis tasks.
method Apply Sparse Variational Dropout and Binary Variational Dropout to sparsify RNNs.
result Achieve 99.5% sparsity on sentiment analysis with no quality drop and up to 87% on language modeling with slight loss of accuracy.
Continuous dropout mimics brain neuron firing rates to prevent feature detector co-adaptation.
problem Preventing overfitting in deep neural networks.
method Extending binary dropout to continuous dropout, inspired by brain neuron firing rates.
result Continuous dropout improves test performance by preventing feature detector co-adaptation.
A new loss function using noisy activations improves deep learning performance.
problem Improving deep learning performance through optimal representation learning.
method Regularization with multiplicative noise in activations, Information Dropout.
result Information Dropout achieves comparable or better generalization performance than binary dropout.
This research explores using Alpha-Divergences in variational dropout for better inference.
problem Improving variational inference methods using alternative divergences.
method Extending the Stochastic Gradient Variational Bayes (SGVB) framework with Alpha-Divergences.
result The α-divergence with αightarrow1 yields the lowest training error and optimizes the ELBO. Variational dropout and ARD are shown to be equivalent, improving sparsity in neural networks.
problem Improper priors in variational dropout models and theoretical issues.
method Applying ARD to Bayesian DNNs with Gaussian approximations and exploring hierarchical priors.
result Variational dropout and ARD objectives are equivalent, with improved sparsity.
Variational Dropout sparsifies deep neural networks by reducing parameter count.
problem Overfitting and high parameter count in deep neural networks.
method Extending Variational Dropout to unbounded dropout rates, reducing gradient variance, and applying individual dropout rates per weight.
result Significant reduction in parameter count (up to 280 times) with minimal accuracy loss.
Privacy-preserving dropout technique for neural networks.
problem Overfitting in deep neural networks and privacy of training data.
method Modified variational dropout with concentrated differential privacy.
result The technique provides differential privacy for neural networks.
Improves Bayesian neural networks inference efficiency and accuracy.
problem Inflexibility of factorized structure in Dropout posterior.
method Introduces Variational Structured Dropout (VSD) with orthogonal transformation.
result VSD induces adaptive regularization and better generalization.
The paper calibrates uncertainty in dropout variational inference models.
problem Miscalibration of model uncertainty in dropout variational inference.
method Logit scaling methods are extended to recalibrate model uncertainty.
result Logit scaling reduces miscalibration, improving reliability of predictions.
Adaptive network sparsification improves model compactness and accuracy.
problem Suboptimal network sparsification due to input-independent dropout.
method Dependent variational beta-Bernoulli dropout.
result Significantly more compact networks with consistent accuracy improvements.
This paper calibrates uncertainty in dropout variational inference models.
problem Uncertainty in variational inference with dropout is poorly calibrated.
method Temperature scaling is extended to dropout variational inference.
result Temperature scaling reduces miscalibration of uncertainty.
We investigate a local reparameterizaton technique for greatly reducing the variance of stochastic gradients for variational Bayesian inference (SGVB) of a posterior over model parameters, while retaining parallelizability. This local reparameterization translates uncertainty about global parameters into local noise th…
VBD improves variational dropout by using a hierarchical prior, enabling better regularization.
problem Improper log-uniform prior in VD causes ill-posed posterior inference.
method Introduces a hierarchical prior with a zero-mean Gaussian distribution and a uniform hyper-prior.
result VBD enables well-posed posterior inference and superior regularization performance.
Researchers fix issues with variational Bayesian dropout.
problem Variational Bayesian dropout's theoretical framework has issues.
method Proposed new approximate inference objective called Quasi-KL (QKL).
result QKL addresses singularity issue and leads to Principal Component Analysis solution.
Advanced Dropout improves DNN performance without requiring model-specific dropout techniques.
problem Overfitting in deep neural networks due to lack of data.
method Model-free, end-to-end training with adaptive dropout rate optimization.
result Advanced Dropout outperforms other techniques on various datasets.
Deep learning improves GW signal detection efficiency and robustness.
problem Traditional matched-filtering techniques are limited in detecting new GW signals.
method Optimized CNN models with techniques like batch normalization and dropout.
result CNN models are robust to the variation of GW waveform parameters.
Proposes LBD for more robust deep learning models.
problem Improving robustness and uncertainty in deep learning models.
method Model-agnostic learnable Bernoulli dropout with joint optimization of dropout rates.
result Superior performance compared to other dropout schemes.
Bayesian inference using stochastic neural networks ensembles.
problem Approximating Bayesian posterior distributions.
method Formulate stochastic ensembles of neural networks, train with variational inference, and evaluate using Monte Carlo dropout.
result Stochastic ensembles provide more accurate posterior estimates than other methods.
MC Dropout is re-evaluated as not Bayesian, affecting predictive posterior and multimodality.
problem MC Dropout's Bayesian properties and predictive posterior are questioned.
method Re-evaluation of MC Dropout's properties, including a new VI engine in pytorch.
result MC Dropout does not produce a faithful Bayesian predictive posterior.
Recurrent neural networks (RNNs) stand at the forefront of many recent developments in deep learning. Yet a major difficulty with these models is their tendency to overfit, with dropout shown to fail when applied to recurrent layers. Recent results at the intersection of Bayesian modelling and deep learning offer a Bay…
New method reduces parameter overhead for Bayesian neural networks.
problem High parameter overhead and difficulty of implementation in variational Bayesian neural networks.
method Constructs a general variational family for ensemble-based Bayesian neural networks that works well with batch normalization layers.
result Improves predictive accuracy and achieves almost perfect calibration on a ResNet-18 trained with ImageNet.
Improved dropout inference for Bayesian neural networks using alpha-divergences.
problem Uncertainty underestimation in dropout variational inference.
method Proposed a re-parametrisation of alpha-divergence objectives for dropout networks.
result Improved uncertainty estimates and accuracy compared to VI in dropout networks.
QSD enhances deep network performance through biologically plausible dropout.
problem Overfitting in deep networks.
method Quantal Synaptic Dilution (QSD) model based on neuronal synapses.
result QSD outperforms standard dropout in various deep network architectures.
Batch normalisation doesn't affect variational inference but fails for larger batch sizes.
problem Failure of Monte Carlo Batch Normalisation (MCBN) for capturing epistemic uncertainty in larger batch sizes.
method Investigated MCBN as an approximate inference technique for Bayesian neural networks, showing its limitations and providing insights for improvement.
result For larger batch sizes, MCBN fails to capture epistemic uncertainty, requiring the batch size to be a variational parameter.
Dropout has recently emerged as a powerful and simple method for training neural networks preventing co-adaptation by stochastically omitting neurons. Dropout is currently not grounded in explicit modelling assumptions which so far has precluded its adoption in Bayesian modelling. Using Bayesian entropic reasoning we s…
Adaptive dropout and regularization are shown to be dual in linear networks.
problem Sparsifying deep neural networks.
method Examining dropout in the linear case, revealing a duality with regularization.
result Adaptive dropout methods lead to sparse solutions with effective penalties similar to classical sparse optimization penalties.
Enhances uncertainty estimation in medical image segmentation.
problem Frequency-related noise in medical imaging leads to biased uncertainty estimates.
method Extends MC-Dropout to the frequency domain for better uncertainty estimation.
result MC-Frequency Dropout improves calibration and uncertainty in semantic segmentation.
Evaluates uncertainty quality in neural networks using anomaly detection.
problem Evaluating the quality of uncertainty in neural networks.
method Extract uncertainty measures, use them as features for an anomaly detector, and compare different neural network models.
result Bayesian Dropout and OSBA provide better uncertainty information than Maximum Likelihood, and are faster.
Proposes using Monte Carlo Dropout in Autoencoder and VAE for synthetic data generation.
problem Handling large amounts of data in costly or difficult-to-collect scenarios.
method Incorporates Monte Carlo Dropout within Autoencoder and Variational Autoencoder.
result Generated data sets are statistically and predictively similar to actual data.
Prb-GAN uses dropout and variational inference to improve GAN performance.
problem GANs struggle with mode loss and training instability.
method Introduces Prb-GANs with dropout and variational inference for parameter distribution.
result Improves GAN performance through dropout and variational inference.
Dropout improves matrix factorization by controlling factor size.
problem Understanding regularization properties of dropout for matrix factorization.
method Theoretical analysis of dropout's equivalence to a deterministic model with adaptive dropout rates.
result Dropout's regularization effect is limited by the fixed dropout rate, suggesting adaptive rates.
Novel framework improves deep learning uncertainty estimates.
problem Improving uncertainty estimates in deep learning models.
method Statistical mechanics based framework for dropout, focusing on variance of loss.
result Generates don't-know answers in classification tasks, improving classifier reliability.
New method improves neural network compression.
problem Efficiently compressing neural networks for mobile devices and inference.
method Combining Soft-Weight Sharing and Variational Dropout.
result New approach achieves state-of-the-art results in model compression.
EB-VAE combines tumor growth and dropout data for personalized treatment response modeling.
problem Challenges in integrating longitudinal tumor measurements, dropout information, and genetic covariates.
method Extended EB-VAE framework to jointly model longitudinal and time-to-event data, incorporating dropout hazard and genetic covariates.
result Hybrid decoder formulation yields consistent treatment-effect parameters and prior predictive performance comparable to neural decoder.
Proposes a new method to prevent overfitting in deep neural networks.
problem Overfitting in deep neural networks with many trainable parameters.
method Randomly replaces elements in feature maps with specific values during training.
result Improves the testing performance of deep neural networks on benchmark datasets.
Bayesian approach improves activity recognition accuracy and uncertainty quantification.
problem Lack of predictive uncertainty in multimodal audiovisual activity recognition.
method Uncertainty aware multimodal Bayesian fusion framework combining deterministic and variational layers.
result Improved precision-recall AUC by 10.2% on MiT dataset.
Study evaluates uncertainty estimation methods in binary classification models.
problem Difficulty in quantifying uncertainty in complex models like deep learning.
method Approximate Bayesian inference with synthetic datasets and empirical tests.
result Deep learning-based algorithms do not consistently reflect lack of evidence for out-of-distribution data.
Paper proposes a new method for imputing missing biomedical data.
problem Missing values in biomedical data.
method Monte Carlo Dropout within Autoencoders.
result The method improves imputation error and predictive similarity.
Detects out-of-distribution sentences in Neural Machine Translation.
problem Identifying sentences from a different language than the training data.
method Developed a new uncertainty measure for long sequences of words in Transformers.
result Shows ability to identify Dutch sentences as German input.
Novel variational sampling improves generative model optimization.
problem Optimizing binary latent variable generative models efficiently.
method Truncated variational EM with efficient sampling.
result Efficiently increases variational free energy objective.
New method simplifies Bayesian analysis for categorical data.
problem Difficulties in scaling GLMs for categorical data due to non-conjugacy or posterior dependencies.
method Defining CB models with binary approximations for tractable inference.
result Fast and scalable inference for thousands of categories, outperforming competitors.
Bayesian inference for DNN synaptic connectivity improves regularization without manual tuning.
problem Improving deep neural network performance through effective regularization.
method Bayesian inference with Beta priors over binary synaptic variables, using Black-Box Variational inference.
result Effective, heuristics-free network regularization scheme.
Lower bound for VAE training objective for binary data.
problem Finding a lower bound for the ELBO of Bernoulli VAE.
method Interpretable lower bound, modified initialization, faster training architecture, PCA for latent space dimension.
result Theoretical result and improved performance of new architecture.
Radial BNNs offer a scalable, continuous weight distribution for Bayesian deep learning.
problem Discrete support in Bayesian deep learning methods like MC dropout.
method Radial BNNs with full support over weight-space.
result Radial BNNs outperform discrete-support methods in real-world applications.
Efficiently identifies important variables in binary outcomes using variational Bayes.
problem Bayesian variable selection for binary outcomes with computational challenges.
method Mean-field variational Bayes approximation with closed-form updates and efficient inference algorithm.
result Successfully identifies important variables and is orders of magnitude faster than MCMC.
Detects out-of-distribution samples in deep reinforcement learning using uncertainty estimation.
problem Detecting out-of-distribution samples in deep reinforcement learning.
method Use uncertainty estimation techniques on the agent's value estimating neural network.
result Bootstrap-based approaches tend to produce more reliable epistemic uncertainty estimates.
New Bayesian model injects noise to improve neural network sparsity and acceleration.
problem Improving neural network sparsity and acceleration.
method Proposes a new Bayesian model that injects noise to neurons outputs while keeping weights unregularized, using log-normal multiplicative noise.
result Provides significant acceleration on deep neural architectures.
Proposes an alternative method to train RBMs with binary synapses using Bayesian learning rule.
problem Training RBMs with binary synapses is challenging due to discrete nature of synapses.
method Proposes an alternative optimization method using the Bayesian learning rule, updating natural parameters instead of expectation parameters.
result No additional clipping is needed as natural parameters take values in the entire real domain.