Dropout improves online learning by avoiding convergence issues.
problem The effect of dropout in online learning is not well understood.
method Analyzed the effect of dropout on convergence speed near the singular point.
result Dropout is effective in online learning, avoiding convergence issues near the singular point.
Dropout is one of the key techniques to prevent the learning from overfitting. It is explained that dropout works as a kind of modified L2 regularization. Here, we shed light on the dropout from Bayesian standpoint. Bayesian interpretation enables us to optimize the dropout rate, which is beneficial for learning of wei…
Guided dropout improves neural network performance by selectively dropping nodes.
problem Overfitting in deep neural networks.
method Guided selection of nodes for dropout based on their strength.
result Guided dropout leads to better generalization compared to conventional dropout.
Dropout has been witnessed with great success in training deep neural networks by independently zeroing out the outputs of neurons at random. It has also received a surge of interest for shallow learning, e.g., logistic regression. However, the independent sampling for dropout could be suboptimal for the sake of conver…
Structured dropout improves confidence calibration in convolutional neural networks.
problem Poor confidence calibration in convolutional neural networks using dropout.
method Structured dropout to promote model diversity and improve calibration.
result Structured dropout reduces calibration error compared to naive dropout.
Advanced Dropout improves DNN performance without requiring model-specific dropout techniques.
problem Overfitting in deep neural networks due to lack of data.
method Model-free, end-to-end training with adaptive dropout rate optimization.
result Advanced Dropout outperforms other techniques on various datasets.
Dropout controls model capacity in deep learning and matrix completion.
problem Controlling model capacity in deep learning and matrix completion problems.
method Investigates dropout's effect on model capacity and Rademacher complexity.
result Dropout induces a regularizer that controls model capacity in expectation.
New dropout method improves uncertainty estimates in deep models and RL.
problem Problems with current dropout methods in obtaining well-calibrated uncertainty estimates.
method Continuous relaxation of dropout masks, principled optimisation objective.
result Improved performance and better calibrated uncertainties in large models and RL.
Dropout learning is analyzed as ensemble learning to prevent overfitting.
problem Overfitting in deep learning models.
method Dropout learning ignores some inputs and hidden units with a probability, p, and combines them with the learned network.
result Combining neglected hidden units with the learned network can be seen as ensemble learning.
Dropout biases neural networks by equalizing hidden node weights.
problem Understanding implicit bias in dropout for neural networks.
method Characterization of optimization landscape for linear neural networks with dropout.
result Dropout equalizes the norm of hidden node weight vectors.
The paper improves dropout's utility by reducing interactions in deep neural networks.
problem Over-fitting problem in deep learning.
method Game-theoretic interactions analysis and interaction loss.
result Interaction loss improves dropout's utility and boosts DNN performance.
CODA uses a new dropout technique inspired by constructivism learning to improve deep learning performance.
problem Existing dropout methods fail to differentiate among instances, leading to overfitting.
method CODA incorporates structural information and uses a Bayesian nonparametric method to create a better dropout technique.
result CODA outperformed other state-of-the-art dropout techniques on 5 real-world datasets.
Enhanced dropout technique improves training speed and generalization.
problem Improving generalization and training speed of deep neural networks.
method Multi-sample dropout technique, creating multiple dropout samples and averaging their losses.
result Multi-sample dropout accelerates training and achieves lower error rates.
Proposes LBD for more robust deep learning models.
problem Improving robustness and uncertainty in deep learning models.
method Model-agnostic learnable Bernoulli dropout with joint optimization of dropout rates.
result Superior performance compared to other dropout schemes.
Wasserstein Dropout improves uncertainty estimation in neural networks.
problem Estimating neural uncertainties for safe machine learning.
method A purely non-parametric approach using dropout-based sub-network distributions and Wasserstein distance.
result Wasserstein Dropout outperforms state-of-the-art methods in uncertainty estimation.
QSD enhances deep network performance through biologically plausible dropout.
problem Overfitting in deep networks.
method Quantal Synaptic Dilution (QSD) model based on neuronal synapses.
result QSD outperforms standard dropout in various deep network architectures.
Recurrent neural networks (RNNs) stand at the forefront of many recent developments in deep learning. Yet a major difficulty with these models is their tendency to overfit, with dropout shown to fail when applied to recurrent layers. Recent results at the intersection of Bayesian modelling and deep learning offer a Bay…
Dropout is a simple but effective technique for learning in neural networks and other settings. A sound theoretical understanding of dropout is needed to determine when dropout should be applied and how to use it most effectively. In this paper we continue the exploration of dropout as a regularizer pioneered by Wager,…
This paper interprets Dropout as creating an equiangular tight frame in autoencoders.
problem Understanding the success of Dropout in deep learning.
method Connecting Dropout to analog channel coding and equiangular tight frames (ETF).
result Optimizing autoencoders with dropout leads to an equiangular tight frame structure.
Dropout and RaM become equivalent in large ResNets as depth and width increase.
problem Improving performance in deep learning models.
method Comparing Dropout and Random Gradient Masking in ResNets.
result Dropout and RaM converge to the same large-scale limiting dynamics in ResNets.
Characterizes dropout's regularizer in deep linear networks.
problem Understanding dropout's regularization effect in deep learning.
method Formal characterization of dropout's regularizer, showing it is composed of an ℓ2-path regularizer and the squared nuclear norm. result For large dropout rates, the global optima of the dropout objective can be characterized.
Dropout improves MIL performance on noisy WSI classification.
problem Noisy feature embeddings and weak supervision in MIL for WSI classification.
method Empirical exploration of dropout in MIL, proposing MIL-Dropout.
result MIL-Dropout boosts MIL performance with minimal computational cost.
Curriculum Dropout improves neural network training by gradually increasing difficulty.
problem Overfitting and suboptimal performance during training.
method Adaptive dropout probability scheduling.
result Curriculum Dropout leads to better generalization and performance.
Dropout training is shown to be optimal for adversarial covariate corruption.
problem Adversarial corruption of covariates in machine learning models.
method Minimax game analysis with dropout noise as the adversarial strategy.
result Dropout training minimizes expected loss for multiplicative perturbations of covariates.
Dropout and other feature noising schemes control overfitting by artificially corrupting the training data. For generalized linear models, dropout performs a form of adaptive regularization. Using this viewpoint, we show that the dropout regularizer is first-order equivalent to an L2 regularizer applied after scaling t…
Dropout training, originally designed for deep neural networks, has been successful on high-dimensional single-layer natural language tasks. This paper proposes a theoretical explanation for this phenomenon: we show that, under a generative Poisson topic model with long documents, dropout training improves the exponent…
Dropout improves matrix factorization by acting as a low-rank regularizer.
problem Improving matrix factorization performance through regularization.
method Using Bernoulli random variables to drop columns of factors, demonstrating equivalence to a deterministic model with sum of squared Euclidean norms.
result Dropout achieves the global minimum of a convex approximation problem with squared nuclear norm regularization.
New dropout technique reduces training time by 20-77%.
problem Efficient training of deep neural networks consumes excessive time and energy.
method Approximate Random Dropout replaces random neuron/synapse dropout with regular patterns to reduce computation and data access.
result Reduces training time by 20-77% with minimal accuracy loss.
Continuum Dropout improves neural differential equations by preventing overfitting.
problem Overfitting in Neural Differential Equations (NDEs).
method Introduces Continuum Dropout, a regularization technique based on alternating renewal processes.
result Continuum Dropout outperforms existing methods in various tasks, improving generalization and uncertainty quantification.
We show that a neural network with arbitrary depth and non-linearities, with dropout applied before every weight layer, is mathematically equivalent to an approximation to a well known Bayesian model. This interpretation might offer an explanation to some of dropout's key properties, such as its robustness to over-fitt…
Privacy-preserving dropout technique for neural networks.
problem Overfitting in deep neural networks and privacy of training data.
method Modified variational dropout with concentrated differential privacy.
result The technique provides differential privacy for neural networks.
Training deep belief networks (DBNs) requires optimizing a non-convex function with an extremely large number of parameters. Naturally, existing gradient descent (GD) based methods are prone to arbitrarily poor local minima. In this paper, we rigorously show that such local minima can be avoided (upto an approximation …
Privileged Information Dropout improves RL performance without distillation.
problem Improving sample efficiency and performance in reinforcement learning.
method Introducing Privileged Information Dropout to directly incorporate privileged information into RL agent inputs.
result Privileged Information Dropout outperforms distillation and auxiliary tasks in a partially-observed environment.
Dropout technique improves deep learning from noisy labels.
problem Training deep networks on datasets with unreliable labels.
method Augment deep network with a noise model and apply dropout regularization.
result The technique outperforms state-of-the-art methods on noisy datasets.
We investigate a local reparameterizaton technique for greatly reducing the variance of stochastic gradients for variational Bayesian inference (SGVB) of a posterior over model parameters, while retaining parallelizability. This local reparameterization translates uncertainty about global parameters into local noise th…
Enhances uncertainty estimation in neural networks using Dirichlet-based MC Dropout.
problem Deterministic predictions without uncertainty estimates in neural networks.
method Integrates Dirichlet-based framework within Monte Carlo Dropout.
result Improves quality of uncertainty estimates in deep learning models.
A new technique called fraternal dropout improves RNN performance.
problem Optimizing recurrent neural networks (RNNs) is harder than feed-forward networks.
method Train two identical RNNs with different dropout masks to encourage robust representations.
result Achieves state-of-the-art results on sequence modeling tasks and improves image captioning and semi-supervised learning.
New method uses dropout with varying variance to improve model learning efficiency.
problem Improving model learning efficiency with limited training data.
method Proposes heteroscedastic dropout for CNNs and RNNs.
result Significantly increases sample efficiency and accuracy with limited data.
Neuron-specific dropout reduces overfitting and data needs for neural networks.
problem Overfitting and insufficient training data for deep neural networks.
method Compares training and validation passes of a layer, drops targeted neurons based on feature analysis.
result Achieves similar or better testing accuracy with less data, reducing overfitting.
Internal node bagging uses dropout-like training to improve model fitting with fewer parameters.
problem Improving model fitting with fewer parameters.
method Explicitly forces a group of nodes to learn a certain feature, combining them in inference time.
result Internal node bagging performs significantly better than dropout on small models.
A fast active learning algorithm for regression using dropout uncertainty.
problem Challenges in high-dimensional regression models with expensive annotation.
method Uncertainty estimation from stochastic dropout output of neural networks.
result Comparable or better performance compared to baselines on synthetic and real-world datasets.
Proposes a curriculum-based dropout discriminator for domain adaptation.
problem Improving domain adaptation using deep learning networks trained on large labeled datasets.
method Introduces a curriculum-based dropout discriminator that gradually increases sample variance and uses reverse gradients to align source and target feature representations.
result The proposed model outperforms state-of-the-art results in domain adaptation tasks.
Deep learning tools have gained tremendous attention in applied machine learning. However such tools for regression and classification do not capture model uncertainty. In comparison, Bayesian models offer a mathematically grounded framework to reason about model uncertainty, but usually come with a prohibitive computa…
Paper proposes using MC-dropout to detect and diagnose incipient faults in buildings.
problem Lack of labeled incipient fault data in buildings.
method Proposes using Monte Carlo dropout (MC-dropout) to enhance deep neural networks for fault detection.
result Demonstrates effectiveness of MC-dropout in indicating likely incipient fault types.
Study predicts high school dropout risk in Louisiana using imbalanced learning techniques.
problem Predicting high school dropout risk in Louisiana.
method Applied imbalanced learning techniques including resampling, case weighting, and cost-sensitive learning.
result Imbalanced learning techniques improve recall but decrease precision.
Predicts student dropout using transcript data.
problem Student attrition in higher education.
method Machine learning model using transcript data.
result Dropout can be accurately predicted from a single term of transcript data.
A fast single-shot MC dropout method for neural networks.
problem Inability of DNNs to provide uncertainty measures for new situations.
method Analytically approximates MC dropout for fully connected networks.
result Approach preserves BDNN advantages while being faster.
Paper explains why Dropout and BN lead to worse performance when combined and proposes solutions.
problem Worse performance when Dropout and BN are combined.
method Theoretical analysis and experiments on various networks to identify variance shift and propose solutions.
result Dropout shifts variance of a specific neural unit, while BN maintains accumulated variance, leading to unstable predictions.