Advanced Dropout improves DNN performance without requiring model-specific dropout techniques.
problem Overfitting in deep neural networks due to lack of data.
method Model-free, end-to-end training with adaptive dropout rate optimization.
result Advanced Dropout outperforms other techniques on various datasets.
Wasserstein Dropout improves uncertainty estimation in neural networks.
problem Estimating neural uncertainties for safe machine learning.
method A purely non-parametric approach using dropout-based sub-network distributions and Wasserstein distance.
result Wasserstein Dropout outperforms state-of-the-art methods in uncertainty estimation.
Proposes multinomial dropout for improved convergence and performance in shallow and deep learning.
problem Suboptimal convergence of independent dropout in deep learning.
method Uses multinomial sampling for dropout probabilities, and adaptive dropout for evolving distributions.
result Achieves faster convergence and smaller testing error compared to standard dropout.
Dropout training improves neural networks' performance.
problem Improving neural network convergence and generalization.
method Two-layer neural networks with ReLU activations, overparametrization, and positive margin assumption.
result Dropout training achieves ε-suboptimality in test error in O(1/ε) iterations.
Enhances uncertainty estimation in medical image segmentation.
problem Frequency-related noise in medical imaging leads to biased uncertainty estimates.
method Extends MC-Dropout to the frequency domain for better uncertainty estimation.
result MC-Frequency Dropout improves calibration and uncertainty in semantic segmentation.
Proposes a curriculum-based dropout discriminator for domain adaptation.
problem Improving domain adaptation using deep learning networks trained on large labeled datasets.
method Introduces a curriculum-based dropout discriminator that gradually increases sample variance and uses reverse gradients to align source and target feature representations.
result The proposed model outperforms state-of-the-art results in domain adaptation tasks.
Dropout training is shown to be optimal for adversarial covariate corruption.
problem Adversarial corruption of covariates in machine learning models.
method Minimax game analysis with dropout noise as the adversarial strategy.
result Dropout training minimizes expected loss for multiplicative perturbations of covariates.
QSD enhances deep network performance through biologically plausible dropout.
problem Overfitting in deep networks.
method Quantal Synaptic Dilution (QSD) model based on neuronal synapses.
result QSD outperforms standard dropout in various deep network architectures.
Variational dropout and ARD are shown to be equivalent, improving sparsity in neural networks.
problem Improper priors in variational dropout models and theoretical issues.
method Applying ARD to Bayesian DNNs with Gaussian approximations and exploring hierarchical priors.
result Variational dropout and ARD objectives are equivalent, with improved sparsity.
Dropout is a simple but effective technique for learning in neural networks and other settings. A sound theoretical understanding of dropout is needed to determine when dropout should be applied and how to use it most effectively. In this paper we continue the exploration of dropout as a regularizer pioneered by Wager,…
The paper analyzes SGD with dropout regularization in linear models, proving asymptotic properties and providing inference tools.
problem Analyzing the behavior of SGD with dropout regularization in linear models.
method Establishing geometric-moment contraction (GMC) and proving quenched central limit theorems (CLT).
result The existence of a unique stationary distribution and asymptotic normality results for SGD with dropout.
New dropout technique reduces training time by 20-77%.
problem Efficient training of deep neural networks consumes excessive time and energy.
method Approximate Random Dropout replaces random neuron/synapse dropout with regular patterns to reduce computation and data access.
result Reduces training time by 20-77% with minimal accuracy loss.
Enhances uncertainty estimation in neural networks using Dirichlet-based MC Dropout.
problem Deterministic predictions without uncertainty estimates in neural networks.
method Integrates Dirichlet-based framework within Monte Carlo Dropout.
result Improves quality of uncertainty estimates in deep learning models.
Rate-In dynamically adjusts dropout rates during inference to improve uncertainty estimation in neural networks.
problem Static dropout rates lead to suboptimal uncertainty estimates in neural networks.
method Rate-In dynamically adjusts dropout rates using information-theoretic principles.
result Rate-In improves calibration and sharpens uncertainty estimates compared to fixed or heuristic dropout rates.
The paper calibrates uncertainty in dropout variational inference models.
problem Miscalibration of model uncertainty in dropout variational inference.
method Logit scaling methods are extended to recalibrate model uncertainty.
result Logit scaling reduces miscalibration, improving reliability of predictions.
New method reduces the gap between dropout training and inference phases.
problem The gap between dropout training and inference phases in neural networks.
method Formulate dropout as a tractable approximation of a latent variable model, introduce expectation-linear dropout neural networks, and regularize the training objective.
result Reduction in the inference gap can improve performance consistently.
Bayesian inference using stochastic neural networks ensembles.
problem Approximating Bayesian posterior distributions.
method Formulate stochastic ensembles of neural networks, train with variational inference, and evaluate using Monte Carlo dropout.
result Stochastic ensembles provide more accurate posterior estimates than other methods.
A fast single-shot MC dropout method for neural networks.
problem Inability of DNNs to provide uncertainty measures for new situations.
method Analytically approximates MC dropout for fully connected networks.
result Approach preserves BDNN advantages while being faster.
Researchers fix issues with variational Bayesian dropout.
problem Variational Bayesian dropout's theoretical framework has issues.
method Proposed new approximate inference objective called Quasi-KL (QKL).
result QKL addresses singularity issue and leads to Principal Component Analysis solution.
Improves Bayesian neural networks inference efficiency and accuracy.
problem Inflexibility of factorized structure in Dropout posterior.
method Introduces Variational Structured Dropout (VSD) with orthogonal transformation.
result VSD induces adaptive regularization and better generalization.
New method improves fault detection by adding unsupervised learning to Monte Carlo dropout models.
problem Detecting and diagnosing incipient and unknown faults in deep neural networks.
method Augmenting Monte Carlo dropout models with unsupervised learning tasks.
result Improved fault detection and diagnosis performance, especially on out-of-distribution examples.
Detects out-of-distribution sentences in Neural Machine Translation.
problem Identifying sentences from a different language than the training data.
method Developed a new uncertainty measure for long sequences of words in Transformers.
result Shows ability to identify Dutch sentences as German input.
VBD improves variational dropout by using a hierarchical prior, enabling better regularization.
problem Improper log-uniform prior in VD causes ill-posed posterior inference.
method Introduces a hierarchical prior with a zero-mean Gaussian distribution and a uniform hyper-prior.
result VBD enables well-posed posterior inference and superior regularization performance.
Dropout is explained as a structured shrinkage prior in neural networks.
problem Understanding the effectiveness of dropout in preventing overfitting.
method Proposes a novel framework to explain dropout as a structured shrinkage prior, considering continuous distributions and Bernoulli noise.
result Dropout's Monte Carlo training objective approximates marginal MAP estimation.
Improved uncertainty estimation through diverse sampling in neural networks.
problem Enhancing uncertainty estimation for machine learning models.
method Data-driven correlations and determinantal point processes-based sampling for dropout layers.
result State-of-the-art results in uncertainty estimation for regression and classification tasks.
Detects out-of-distribution samples in deep reinforcement learning using uncertainty estimation.
problem Detecting out-of-distribution samples in deep reinforcement learning.
method Use uncertainty estimation techniques on the agent's value estimating neural network.
result Bootstrap-based approaches tend to produce more reliable epistemic uncertainty estimates.
Improved dropout inference for Bayesian neural networks using alpha-divergences.
problem Uncertainty underestimation in dropout variational inference.
method Proposed a re-parametrisation of alpha-divergence objectives for dropout networks.
result Improved uncertainty estimates and accuracy compared to VI in dropout networks.
Paper proposes a new method for imputing missing biomedical data.
problem Missing values in biomedical data.
method Monte Carlo Dropout within Autoencoders.
result The method improves imputation error and predictive similarity.
Real-time uncertainty estimation for computer vision tasks.
problem Real-time inference of uncertainty in deep learning models.
method Uncertainty-Aware Distribution Distillation method for fast inference.
result Significantly reduced inference time with improved uncertainty and predictive performance.
Study on MC dropout in wide neural networks and its convergence to Gaussian processes.
problem Understanding the behavior of Monte Carlo dropout in wide neural networks.
method Rigorously studied the limiting distribution of wide untrained NNs under dropout, proving convergence to Gaussian processes. Investigated correlations and non-Gaussian behavior in finite width NNs.
result Wide untrained neural networks under dropout converge to Gaussian processes for fixed sets of weights and biases.
Meta-dropout learns to perturb training examples for better generalization.
problem Achieving good generalization in unseen test examples.
method Meta-learning a noise generator to perturb latent features of training examples.
result Meta-dropout significantly improves generalization performance on few-shot classification datasets.
Guided dropout improves neural network performance by selectively dropping nodes.
problem Overfitting in deep neural networks.
method Guided selection of nodes for dropout based on their strength.
result Guided dropout leads to better generalization compared to conventional dropout.
Enhanced dropout technique improves training speed and generalization.
problem Improving generalization and training speed of deep neural networks.
method Multi-sample dropout technique, creating multiple dropout samples and averaging their losses.
result Multi-sample dropout accelerates training and achieves lower error rates.
A fast method for neural networks that provides uncertainty measures.
problem Inability of neural networks to provide reliable uncertainty measures for new situations.
method Moment propagation (MP) for analytically approximating MC dropout.
result Single-shot MC dropout approximation as fast as NNs with similar uncertainty measures.
Proposes a new noise injection method for neural networks that improves accuracy and representation clarity.
problem Improving neural network performance and representation clarity.
method Adaptive Structured Noise Injection (ASNI) for shallow and deep neural networks.
result Boosts the accuracy of neural networks and disentangles hidden layer representations.
Locally learned synaptic failure enables complete Bayesian inference.
problem Bayesian inference in neural networks.
method Biologically constrained neural network with synaptic failure and lateral inhibition.
result Synaptic failure allows sampling from both epistemic and aleatoric distributions.
Dropout in deep networks produces unexpected behavior and counteracts weight decay.
problem Understanding the behavior of dropout in deep networks.
method Analysis of dropout in deep networks with rectified linear units and quadratic loss.
result Dropout training can produce negative weights, contrary to the expectation of discouraging co-adaptation of weights.
Continuous dropout mimics brain neuron firing rates to prevent feature detector co-adaptation.
problem Preventing overfitting in deep neural networks.
method Extending binary dropout to continuous dropout, inspired by brain neuron firing rates.
result Continuous dropout improves test performance by preventing feature detector co-adaptation.
Dropout regularizes against high-order interactions by canceling interaction rates.
problem Overfitting to high-order interactions in neural networks.
method Analyzes Dropout through the lens of interaction effects, showing how it effectively cancels out the probability of surviving interactions of different orders.
result Dropout regularizes against high-order interactions by effectively canceling out the probability of surviving interactions of different orders.
Integrates global information into dropout for better text classification.
problem Improving neural networks for text classification.
method GI-Dropout, a novel dropout method integrating global information.
result Demonstrates the effectiveness of GI-Dropout on seven text classification tasks.
The paper introduces a method to decompose variance in twin networks for better treatment effect estimation.
problem Accurate treatment effect estimation requires reliable uncertainty measures to locate model failures.
method Layer-wise variance decomposition using Monte Carlo Dropout in twin networks.
result The encoder component dominates under distributional shift, providing a practical diagnostic for data collection.
Proposes a new method for uncertainty estimation in neural networks.
problem Estimating uncertainty in neural networks.
method Samples outputs from Gaussian distributions parametrized by mean and variance sub-layers.
result Achieves better uncertainty quality than other methods.
Tabu Dropout improves performance of standard Dropout by generating more diverse neural network architectures.
problem Preventing co-adaptation of neurons in deep neural networks.
method Integrates a diversification strategy into dropout, marking units from the last forward propagation for re-selection in the current forward propagation.
result Improves performance of standard Dropout on MNIST and Fashion-MNIST datasets.
Dropout is one of the key techniques to prevent the learning from overfitting. It is explained that dropout works as a kind of modified L2 regularization. Here, we shed light on the dropout from Bayesian standpoint. Bayesian interpretation enables us to optimize the dropout rate, which is beneficial for learning of wei…
Dropout improves online learning by avoiding convergence issues.
problem The effect of dropout in online learning is not well understood.
method Analyzed the effect of dropout on convergence speed near the singular point.
result Dropout is effective in online learning, avoiding convergence issues near the singular point.
Adversarial dropout improves RNNs' performance on sequential data tasks.
problem Improving generalization performance of RNNs for sequential data.
method Adversarial dropout technique for RNNs using intentionally generated dropout masks.
result Adversarial dropout improves RNNs' effectiveness on sequential tasks.
Spiking neural networks enable efficient approximate Bayesian inference via permanent dropout.
problem Efficient uncertainty quantification in neural network predictions for critical tasks.
method Conversion of classical neural networks to spiking neural networks, applying permanent dropout for inference.
result Predictive distributions from spiking neural networks using permanent dropout are nearly identical to those from classical networks.
EB-VAE combines tumor growth and dropout data for personalized treatment response modeling.
problem Challenges in integrating longitudinal tumor measurements, dropout information, and genetic covariates.
method Extended EB-VAE framework to jointly model longitudinal and time-to-event data, incorporating dropout hazard and genetic covariates.
result Hybrid decoder formulation yields consistent treatment-effect parameters and prior predictive performance comparable to neural decoder.