Dropout technique improves deep learning from noisy labels.
problem Training deep networks on datasets with unreliable labels.
method Augment deep network with a noise model and apply dropout regularization.
result The technique outperforms state-of-the-art methods on noisy datasets.
Paper finds dropout noise approximation invalid for logistic regression and neural networks.
problem Invalidity of dropout noise approximation for logistic regression and neural networks.
method Derived equivalence between dropout noise injection and L2 regularisation using divergent Taylor expansion. result Approximation approach is invalid for robust constraints and general neural network topologies.
Dropout is typically interpreted as bagging a large number of models sharing parameters. We show that using dropout in a network can also be interpreted as a kind of data augmentation in the input space without domain knowledge. We present an approach to projecting the dropout noise within a network back into the input…
Analyzes dropout in neural networks using differential equations.
problem Lack of theoretical understanding of dropout's effectiveness.
method Analytical study of dropout in two-layer neural networks with online stochastic gradient descent.
result Dropout reduces detrimental correlations and optimal probability increases with data noise.
Proposes a new noise injection method for neural networks that improves accuracy and representation clarity.
problem Improving neural network performance and representation clarity.
method Adaptive Structured Noise Injection (ASNI) for shallow and deep neural networks.
result Boosts the accuracy of neural networks and disentangles hidden layer representations.
Dropout is explained as a structured shrinkage prior in neural networks.
problem Understanding the effectiveness of dropout in preventing overfitting.
method Proposes a novel framework to explain dropout as a structured shrinkage prior, considering continuous distributions and Bernoulli noise.
result Dropout's Monte Carlo training objective approximates marginal MAP estimation.
Privacy-preserving dropout technique for neural networks.
problem Overfitting in deep neural networks and privacy of training data.
method Modified variational dropout with concentrated differential privacy.
result The technique provides differential privacy for neural networks.
Differentially private dropout technique preserves privacy in neural network training.
problem Preserving privacy in large datasets used for neural network training.
method Introduces a Bayesian dropout technique that adds intrinsic noise for regularization and differential privacy.
result Demonstrates that the iterative nature of neural network training can be handled with a relaxed differential privacy concept.
DropLasso improves scRNA-seq data analysis by handling dropout noise.
problem Noisy scRNA-seq data with many dropout events.
method Extends dropout regularization to sparse linear models.
result DropLasso provides better molecular signature inference than standard regularizations.
Dropout improves GAN outputs by preserving input constraints.
problem Generating multiple outputs from one input.
method Using Dropout in GANs to maintain input constraints.
result Enhanced generation of diverse outputs from a single input.
Proposes NCMN to remove feature correlation in multiplicative noise.
problem High feature correlation in multiplicative noise reduces network performance.
method Exploits batch normalization to remove correlation effect.
result Significantly improves performance on image classification tasks.
New Bayesian model injects noise to improve neural network sparsity and acceleration.
problem Improving neural network sparsity and acceleration.
method Proposes a new Bayesian model that injects noise to neurons outputs while keeping weights unregularized, using log-normal multiplicative noise.
result Provides significant acceleration on deep neural architectures.
Proposes Neural SDE for better model robustness and generalization.
problem Missing regularization mechanisms in Neural ODE networks.
method Integrates various regularization mechanisms via stochastic noise injection.
result Improves robustness and generalization compared to Neural ODE.
Meta-dropout learns to perturb training examples for better generalization.
problem Achieving good generalization in unseen test examples.
method Meta-learning a noise generator to perturb latent features of training examples.
result Meta-dropout significantly improves generalization performance on few-shot classification datasets.
A new loss function using noisy activations improves deep learning performance.
problem Improving deep learning performance through optimal representation learning.
method Regularization with multiplicative noise in activations, Information Dropout.
result Information Dropout achieves comparable or better generalization performance than binary dropout.
Dropout and other feature noising schemes control overfitting by artificially corrupting the training data. For generalized linear models, dropout performs a form of adaptive regularization. Using this viewpoint, we show that the dropout regularizer is first-order equivalent to an L2 regularizer applied after scaling t…
We investigate a local reparameterizaton technique for greatly reducing the variance of stochastic gradients for variational Bayesian inference (SGVB) of a posterior over model parameters, while retaining parallelizability. This local reparameterization translates uncertainty about global parameters into local noise th…
Critical initialisation strategies are identified for noisy ReLU networks.
problem Understanding signal propagation in noisy rectifier neural networks.
method Developed a new framework for signal propagation in stochastic regularized neural networks, incorporating various noise distributions.
result Critical initialisation strategies for multiplicative noise (e.g. dropout) are identified, but not for additive noise.
Continuum Dropout improves neural differential equations by preventing overfitting.
problem Overfitting in Neural Differential Equations (NDEs).
method Introduces Continuum Dropout, a regularization technique based on alternating renewal processes.
result Continuum Dropout outperforms existing methods in various tasks, improving generalization and uncertainty quantification.
Enhances uncertainty estimation in medical image segmentation.
problem Frequency-related noise in medical imaging leads to biased uncertainty estimates.
method Extends MC-Dropout to the frequency domain for better uncertainty estimation.
result MC-Frequency Dropout improves calibration and uncertainty in semantic segmentation.
This research explores using Alpha-Divergences in variational dropout for better inference.
problem Improving variational inference methods using alternative divergences.
method Extending the Stochastic Gradient Variational Bayes (SGVB) framework with Alpha-Divergences.
result The α-divergence with αightarrow1 yields the lowest training error and optimizes the ELBO. Two methods use DNN-HMM for global SNR estimation of speech signals.
problem Estimating global SNR of speech signals in various noise conditions.
method Dropout approximation for uncertainty estimation and noise-specific regressors.
result Improved SNR estimation accuracy compared to existing methods.
Rate-In dynamically adjusts dropout rates during inference to improve uncertainty estimation in neural networks.
problem Static dropout rates lead to suboptimal uncertainty estimates in neural networks.
method Rate-In dynamically adjusts dropout rates using information-theoretic principles.
result Rate-In improves calibration and sharpens uncertainty estimates compared to fixed or heuristic dropout rates.
We formalize the notion of a pseudo-ensemble, a (possibly infinite) collection of child models spawned from a parent model by perturbing it according to some noise process. E.g., dropout (Hinton et. al, 2012) in a deep neural network trains a pseudo-ensemble of child subnetworks generated by randomly masking nodes in t…
Advancements in parallel processing have lead to a surge in multilayer perceptrons' (MLP) applications and deep learning in the past decades. Recurrent Neural Networks (RNNs) give additional representational power to feedforward MLPs by providing a way to treat sequential data. However, RNNs are hard to train using con…
Dropout improves matrix factorization by controlling factor size.
problem Understanding regularization properties of dropout for matrix factorization.
method Theoretical analysis of dropout's equivalence to a deterministic model with adaptive dropout rates.
result Dropout's regularization effect is limited by the fixed dropout rate, suggesting adaptive rates.
Improves Bayesian neural networks inference efficiency and accuracy.
problem Inflexibility of factorized structure in Dropout posterior.
method Introduces Variational Structured Dropout (VSD) with orthogonal transformation.
result VSD induces adaptive regularization and better generalization.
Dropout and RaM become equivalent in large ResNets as depth and width increase.
problem Improving performance in deep learning models.
method Comparing Dropout and Random Gradient Masking in ResNets.
result Dropout and RaM converge to the same large-scale limiting dynamics in ResNets.
Dropout training is shown to be optimal for adversarial covariate corruption.
problem Adversarial corruption of covariates in machine learning models.
method Minimax game analysis with dropout noise as the adversarial strategy.
result Dropout training minimizes expected loss for multiplicative perturbations of covariates.
In this paper, a simple, general method of adding auxiliary stochastic neurons to a multi-layer perceptron is proposed. It is shown that the proposed method is a generalization of recently successful methods of dropout (Hinton et al., 2012), explicit noise injection (Vincent et al., 2010; Bishop, 1995) and semantic has…
Defense against ASR attacks using dropout uncertainty.
problem Adversarial attacks on ASR systems.
method Dropout uncertainty in neural networks.
result High detection accuracy across various ASR systems and datasets.
Curriculum Dropout improves neural network training by gradually increasing difficulty.
problem Overfitting and suboptimal performance during training.
method Adaptive dropout probability scheduling.
result Curriculum Dropout leads to better generalization and performance.
Deep networks can memorize noise data but prioritize simple patterns.
problem Understanding the role of memorization in deep learning.
method Examination of deep learning capacity, generalization, and adversarial robustness; experiments with gradient-based optimization and dropout.
result Deep networks prioritize simple patterns over complex ones when memorizing noise data.
RotationOut rotates input vectors to regularize neural networks.
problem Reduction of co-adaptation in neural networks.
method Randomly rotates input vectors of the input layer.
result RotationOut reduces co-adaptation better than Dropout.
Paper tackles noisy labels by compressing feature representations.
problem Learning with noisy labels leads to overfitting and poor generalization.
method Introduces compression inductive bias using Dropout and Nested Dropout.
result Compression helps in combating label noise and improving performance.
Improved speech recognition models with data augmentation and dropout.
problem Overfitting in end-to-end speech recognition models.
method Data augmentation and dropout applied to all layers of the network.
result Combination of data augmentation and dropout gives over 20% performance improvement.
Dropout improves MIL performance on noisy WSI classification.
problem Noisy feature embeddings and weak supervision in MIL for WSI classification.
method Empirical exploration of dropout in MIL, proposing MIL-Dropout.
result MIL-Dropout boosts MIL performance with minimal computational cost.
Neuron-specific dropout reduces overfitting and data needs for neural networks.
problem Overfitting and insufficient training data for deep neural networks.
method Compares training and validation passes of a layer, drops targeted neurons based on feature analysis.
result Achieves similar or better testing accuracy with less data, reducing overfitting.
New research shows Gaussian dropout in neural networks is not a proper Bayesian method.
problem Gaussian dropout's Bayesian interpretation is flawed and leads to improper posteriors.
method Analyzed the log-uniform prior and correlated weight noise approximation in Gaussian dropout.
result Gaussian dropout does not provide proper Bayesian inference and can lead to overfitting.
Researchers fix issues with variational Bayesian dropout.
problem Variational Bayesian dropout's theoretical framework has issues.
method Proposed new approximate inference objective called Quasi-KL (QKL).
result QKL addresses singularity issue and leads to Principal Component Analysis solution.
Regularization is essential when training large neural networks. As deep neural networks can be mathematically interpreted as universal function approximators, they are effective at memorizing sampling noise in the training data. This results in poor generalization to unseen data. Therefore, it is no surprise that a ne…
Derives VC bounds for deep neural networks using radius-margin approach.
problem Understanding the capacity of deep neural networks.
method Applies radius-margin bounds from SVM to deep feed-forward architectures.
result Derives new VC bounds different from earlier ones.
Tensor dropout improves CNN robustness and efficiency.
problem Limited generalization and robustness of deep CNNs.
method Tensor layers with low-rank structures and tensor dropout.
result Tensor dropout outperforms other methods in robustness and efficiency.
Dropout is shown to be a simplified version of SDR, which improves deep learning performance.
problem Overfitting and misspecification in deep learning models.
method SDR redefines weights as random variables, updating them based on prediction error and local history.
result SDR outperforms Dropout on standard benchmarks, achieving similar accuracy in fewer epochs.
NADPEx uses dropout to enable temporally consistent exploration in reinforcement learning.
problem Achieving temporally consistent exploration in reinforcement learning agents.
method Integrates dropout into reinforcement learning policies to ensure temporal consistency.
result NADPEx outperforms naive exploration and parameter noise in tasks with sparse rewards.
Proposes a feed-forward method for uncertainty propagation in neural networks.
problem Uncertainty in neural network inputs and activations.
method Feed-forward inference method for belief and neural networks, considering both mean and variance.
result Propagates uncertainty through neural network layers and parameters.
Whiteout injects adaptive Gaussian noise to prevent overfitting in deep neural networks.
problem Preventing overfitting in deep neural networks.
method Introducing a family of noise injection regularization techniques (NIRT) through Gaussian noise.
result Whiteout stabilizes neural network training and converges to ideal loss function.
Tikhonov regularization improves LSTM network performance without dropout issues.
problem Improving LSTM network performance without dropout-induced memory loss.
method Derives a Tikhonov regularizer for LSTM networks, considering interactions between weights.
result Proposes a regularizer with three parameters for LSTM networks, maintaining stability during training.