While variational dropout approaches have been shown to be effective for network sparsification, they are still suboptimal in the sense that they set the dropout rate for each neuron without consideration of the input data. With such input-independent dropout, each neuron is evolved to be generic across inputs, which m…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
169,051 papers · 148 categories
Trend · papers per month
48 results for “Bernoulli dropout”
Proposes LBD for more robust deep learning models.
problem Improving robustness and uncertainty in deep learning models.
method Model-agnostic learnable Bernoulli dropout with joint optimization of dropout rates.
result Superior performance compared to other dropout schemes.
Dropout is a simple yet effective algorithm for regularizing neural networks by randomly dropping out units through Bernoulli multiplicative noise, and for some restricted problem classes, such as linear or logistic regression, several theoretical studies have demonstrated the equivalence between dropout and a fully de…
Regularization for matrix factorization (MF) and approximation problems has been carried out in many different ways. Due to its popularity in deep learning, dropout has been applied also for this class of problems. Despite its solid empirical performance, the theoretical properties of dropout as a regularizer remain qu…
Dropout is explained as a structured shrinkage prior in neural networks.
problem Understanding the effectiveness of dropout in preventing overfitting.
method Proposes a novel framework to explain dropout as a structured shrinkage prior, considering continuous distributions and Bernoulli noise.
result Dropout's Monte Carlo training objective approximates marginal MAP estimation.
New method views residual networks as stochastic differential equations.
problem Improving generalization of neural networks.
method Applying modified equations to show residual networks as weak approximations of stochastic differential equations.
result Stochastic training of residual networks can be understood through the lens of optimal control of backward Kolmogorov's equations.
Researchers fix issues with variational Bayesian dropout.
problem Variational Bayesian dropout's theoretical framework has issues.
method Proposed new approximate inference objective called Quasi-KL (QKL).
result QKL addresses singularity issue and leads to Principal Component Analysis solution.
Dropout is shown to be a simplified version of SDR, which improves deep learning performance.
problem Overfitting and misspecification in deep learning models.
method SDR redefines weights as random variables, updating them based on prediction error and local history.
result SDR outperforms Dropout on standard benchmarks, achieving similar accuracy in fewer epochs.
Convolutional neural networks (CNNs) work well on large datasets. But labelled data is hard to collect, and in some applications larger amounts of data are not available. The problem then is how to use CNNs with small data -- as CNNs overfit quickly. We present an efficient Bayesian CNN, offering better robustness to o…
Noise injection (NI) is an efficient technique to mitigate over-fitting in neural networks (NNs). The Bernoulli NI procedure as implemented in dropout and shakeout has connections with and regularization for the NN model parameters. We propose whiteout, a family NI regularization techniques (NIRT) through i…
Characterizes symmetric Bernoulli distributions with minimal convex sums.
problem Understanding minimal dependence among Bernoulli random vectors.
method Geometric and algebraic representations of multivariate symmetric Bernoulli distributions.
result Characterizes extremal negative dependence and builds minimal dependence copulas.
Guided Dropoutcs.LG
Guided dropout improves neural network performance by selectively dropping nodes.
problem Overfitting in deep neural networks.
method Guided selection of nodes for dropout based on their strength.
result Guided dropout leads to better generalization compared to conventional dropout.
Advanced Dropout improves DNN performance without requiring model-specific dropout techniques.
problem Overfitting in deep neural networks due to lack of data.
method Model-free, end-to-end training with adaptive dropout rate optimization.
result Advanced Dropout outperforms other techniques on various datasets.
Enhanced dropout technique improves training speed and generalization.
problem Improving generalization and training speed of deep neural networks.
method Multi-sample dropout technique, creating multiple dropout samples and averaging their losses.
result Multi-sample dropout accelerates training and achieves lower error rates.
Upper bound on expected supremum of Bernoulli process.
problem Bounding the supremum of Bernoulli processes.
method Using properties of the index set and function class, extending earlier results on Gaussian processes.
result An upper bound on the expected supremum of a Bernoulli process.
Continuous Dropoutcs.CV
Continuous dropout mimics brain neuron firing rates to prevent feature detector co-adaptation.
problem Preventing overfitting in deep neural networks.
method Extending binary dropout to continuous dropout, inspired by brain neuron firing rates.
result Continuous dropout improves test performance by preventing feature detector co-adaptation.
Dropout regularizes against high-order interactions by canceling interaction rates.
problem Overfitting to high-order interactions in neural networks.
method Analyzes Dropout through the lens of interaction effects, showing how it effectively cancels out the probability of surviving interactions of different orders.
result Dropout regularizes against high-order interactions by effectively canceling out the probability of surviving interactions of different orders.
We propose a feed-forward inference method applicable to belief and neural networks. In a belief network, the method estimates an approximate factorized posterior of all hidden units given the input. In neural networks the method propagates uncertainty of the input through all the layers. In neural networks with inject…
Integrates global information into dropout for better text classification.
problem Improving neural networks for text classification.
method GI-Dropout, a novel dropout method integrating global information.
result Demonstrates the effectiveness of GI-Dropout on seven text classification tasks.
We analyze dropout in deep networks with rectified linear units and the quadratic loss. Our results expose surprising differences between the behavior of dropout and more traditional regularizers like weight decay. For example, on some simple data sets dropout training produces negative weights even though the output i…
Tabu Dropout improves performance of standard Dropout by generating more diverse neural network architectures.
problem Preventing co-adaptation of neurons in deep neural networks.
method Integrates a diversification strategy into dropout, marking units from the last forward propagation for re-selection in the current forward propagation.
result Improves performance of standard Dropout on MNIST and Fashion-MNIST datasets.
Dropout is one of the key techniques to prevent the learning from overfitting. It is explained that dropout works as a kind of modified L2 regularization. Here, we shed light on the dropout from Bayesian standpoint. Bayesian interpretation enables us to optimize the dropout rate, which is beneficial for learning of wei…
Adversarial dropout improves RNNs' performance on sequential data tasks.
problem Improving generalization performance of RNNs for sequential data.
method Adversarial dropout technique for RNNs using intentionally generated dropout masks.
result Adversarial dropout improves RNNs' effectiveness on sequential tasks.
Dropout is a simple but effective technique for learning in neural networks and other settings. A sound theoretical understanding of dropout is needed to determine when dropout should be applied and how to use it most effectively. In this paper we continue the exploration of dropout as a regularizer pioneered by Wager,…
Dropout has been witnessed with great success in training deep neural networks by independently zeroing out the outputs of neurons at random. It has also received a surge of interest for shallow learning, e.g., logistic regression. However, the independent sampling for dropout could be suboptimal for the sake of conver…
Rate-In dynamically adjusts dropout rates during inference to improve uncertainty estimation in neural networks.
problem Static dropout rates lead to suboptimal uncertainty estimates in neural networks.
method Rate-In dynamically adjusts dropout rates using information-theoretic principles.
result Rate-In improves calibration and sharpens uncertainty estimates compared to fixed or heuristic dropout rates.
Simple algorithm approximates rare event frequencies.
problem Approximating the frequency of rare events.
method Iterative update of categorical click-distribution, resulting in a random walk on an n-dimensional simplex.
result The random walk corresponds to a biased Bernoulli convolution under certain conditions.
Proposes a framework to handle missing data in traffic forecasting with sensor blackouts.
problem Missing data in traffic forecasting due to sensor blackouts, especially when correlated with traffic conditions.
method Latent state-space framework that models traffic dynamics and sensor dropout probabilities.
result Improves traffic forecasting by reducing blackout imputation RMSE from 7.02 to 4.23, with MNAR modeling providing additional gains.
QSD enhances deep network performance through biologically plausible dropout.
problem Overfitting in deep networks.
method Quantal Synaptic Dilution (QSD) model based on neuronal synapses.
result QSD outperforms standard dropout in various deep network architectures.
In this paper, we consider the multivariate Bernoulli distribution as a model to estimate the structure of graphs with binary nodes. This distribution is discussed in the framework of the exponential family, and its statistical properties regarding independence of the nodes are demonstrated. Importantly the model can e…
Analyzes dropout in neural networks using differential equations.
problem Lack of theoretical understanding of dropout's effectiveness.
method Analytical study of dropout in two-layer neural networks with online stochastic gradient descent.
result Dropout reduces detrimental correlations and optimal probability increases with data noise.
Develops a multilevel Monte Carlo framework with dropout for efficient uncertainty quantification.
problem Efficiently quantify uncertainty in complex models using dropout.
method Integrates multilevel Monte Carlo with Monte Carlo dropout, creating coupled estimators to reduce variance.
result Demonstrates significant variance reduction and efficiency gains over single-level Monte Carlo dropout.
Finite index solutions to Bernoulli problem are always axially symmetric.
problem Entire solutions to the Bernoulli free boundary problem with finite Morse index in 3D.
method Proof of axial symmetry for finite index solutions.
result Finite index solutions to the Bernoulli problem in 3D are axially symmetric.
Dropout introduces both explicit and implicit regularization effects.
problem Understanding the full impact of dropout regularization.
method Disentangled explicit and implicit regularization effects through experiments and analytic simplifications.
result Explicit and implicit regularization effects of dropout are distinct and can be characterized analytically.
Structured dropout improves confidence calibration in convolutional neural networks.
problem Poor confidence calibration in convolutional neural networks using dropout.
method Structured dropout to promote model diversity and improve calibration.
result Structured dropout reduces calibration error compared to naive dropout.
Dropout technique is analyzed using information geometry.
problem Understanding the regularization performance of dropout in neural networks.
method Unified analysis from information geometry viewpoint.
result Dropout flattens the model manifold and its performance depends on curvature.
The paper analyzes SGD with dropout regularization in linear models, proving asymptotic properties and providing inference tools.
problem Analyzing the behavior of SGD with dropout regularization in linear models.
method Establishing geometric-moment contraction (GMC) and proving quenched central limit theorems (CLT).
result The existence of a unique stationary distribution and asymptotic normality results for SGD with dropout.
Deep learning is the state-of-the-art in fields such as visual object recognition and speech recognition. This learning uses a large number of layers and a huge number of units and connections. Therefore, overfitting is a serious problem with it, and the dropout which is a kind of regularization tool is used. However, …
Variational dropout and ARD are shown to be equivalent, improving sparsity in neural networks.
problem Improper priors in variational dropout models and theoretical issues.
method Applying ARD to Bayesian DNNs with Gaussian approximations and exploring hierarchical priors.
result Variational dropout and ARD objectives are equivalent, with improved sparsity.
Algorithm learns smooth probability functions from Bernoulli tests with guarantees.
problem Learning smooth probability functions from Bernoulli tests with contextual features.
method Scalable algorithm with rigorous L2-norm convergence guarantees for posterior update rule.
result Empirical convergence rates match theoretical guarantees, superior to state-of-the-art.
Dropout training improves neural networks' performance.
problem Improving neural network convergence and generalization.
method Two-layer neural networks with ReLU activations, overparametrization, and positive margin assumption.
result Dropout training achieves ε-suboptimality in test error in O(1/ε) iterations.
The paper improves dropout's utility by reducing interactions in deep neural networks.
problem Over-fitting problem in deep learning.
method Game-theoretic interactions analysis and interaction loss.
result Interaction loss improves dropout's utility and boosts DNN performance.
Study compares dropout and l2 regularization in linear models.
problem Understanding the statistical behavior of dropout and l2 regularization in linear models.
method Derives non-asymptotic bounds for gradient descent iterates with dropout and compares them to l2 regularization.
result Indicates a more subtle relationship between dropout and l2 regularization, highlighting interactions between dynamics and randomness.
Qualitative analysis of MC dropout for NN model uncertainty.
problem Measuring uncertainty in neural network models.
method Mathematical formulation of Monte Carlo dropout and its benefits/costs in NN models.
result Potential benefits and associated costs of using MC dropout in NN models.
Dropout as data augmentationstat.ML
Dropout is typically interpreted as bagging a large number of models sharing parameters. We show that using dropout in a network can also be interpreted as a kind of data augmentation in the input space without domain knowledge. We present an approach to projecting the dropout noise within a network back into the input…
Proves a principle for one-phase Bernoulli problem minimizers.
problem One-phase Bernoulli problem minimizers.
method Strong maximum principle, Alt-Caffarelli functional, Hardt-Simon-type foliation.
result Constructs a foliation for global minimizers.
Dropout controls model capacity in deep learning and matrix completion.
problem Controlling model capacity in deep learning and matrix completion problems.
method Investigates dropout's effect on model capacity and Rademacher complexity.
result Dropout induces a regularizer that controls model capacity in expectation.
Characterizes dropout's regularizer in deep linear networks.
problem Understanding dropout's regularization effect in deep learning.
method Formal characterization of dropout's regularizer, showing it is composed of an -path regularizer and the squared nuclear norm.
result For large dropout rates, the global optima of the dropout objective can be characterized.