A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
QSD enhances deep network performance through biologically plausible dropout.
Dropout is a simple but efficient regularization technique for achieving better generalization of deep neural networks (DNNs); hence it is widely used in tasks based on DNNs. During training, dropout randomly discards a portion of the neurons to avoid overfitting. This paper presents an enhanced dropout technique, whic…
Dropout is often used in deep neural networks to prevent over-fitting. Conventionally, dropout training invokes \textit{random drop} of nodes from the hidden layers of a Neural Network. It is our hypothesis that a guided selection of nodes for intelligent dropout can lead to better generalization as compared to the tra…
Analyzes dropout in neural networks using differential equations.
We analyze dropout in deep networks with rectified linear units and the quadratic loss. Our results expose surprising differences between the behavior of dropout and more traditional regularizers like weight decay. For example, on some simple data sets dropout training produces negative weights even though the output i…
Dropout training improves neural networks' performance.
Qualitative analysis of MC dropout for NN model uncertainty.
Dropout has proven to be an effective technique for regularization and preventing the co-adaptation of neurons in deep neural networks (DNN). It randomly drops units with a probability during the training stage of DNN. Dropout also provides a way of approximately combining exponentially many different neural networ…
Study on dropout in neural networks using percolation theory.
Dropout is typically interpreted as bagging a large number of models sharing parameters. We show that using dropout in a network can also be interpreted as a kind of data augmentation in the input space without domain knowledge. We present an approach to projecting the dropout noise within a network back into the input…
Dropout improves neural network performance by promoting low-rank solutions.
Continuous dropout mimics brain neuron firing rates to prevent feature detector co-adaptation.
While variational dropout approaches have been shown to be effective for network sparsification, they are still suboptimal in the sense that they set the dropout rate for each neuron without consideration of the input data. With such input-independent dropout, each neuron is evolved to be generic across inputs, which m…
Dropout technique is analyzed using information geometry.
We introduce dropout compaction, a novel method for training feed-forward neural networks which realizes the performance gains of training a large model with dropout regularization, yet extracts a compact neural network for run-time efficiency. In the proposed method, we introduce a sparsity-inducing prior on the per u…
Wasserstein Dropout improves uncertainty estimation in neural networks.
Dropout is used to avoid overfitting by randomly dropping units from the neural networks during training. Inspired by dropout, this paper presents GI-Dropout, a novel dropout method integrating with global information to improve neural networks for text classification. Unlike the traditional dropout method in which the…
Adaptive dropout and regularization are shown to be dual in linear networks.
Dropout regularizes against high-order interactions by canceling interaction rates.
In classification applications, we often want probabilistic predictions to reflect confidence or uncertainty. Dropout, a commonly used training technique, has recently been linked to Bayesian inference, yielding an efficient way to quantify uncertainty in neural network models. However, as previously demonstrated, conf…
Advanced Dropout improves DNN performance without requiring model-specific dropout techniques.
Rate-In dynamically adjusts dropout rates during inference to improve uncertainty estimation in neural networks.
Neuron-specific dropout reduces overfitting and data needs for neural networks.
Successful application processing sequential data, such as text and speech, requires an improved generalization performance of recurrent neural networks (RNNs). Dropout techniques for RNNs were introduced to respond to these demands, but we conjecture that the dropout on RNNs could have been improved by adopting the ad…
Dropout introduces both explicit and implicit regularization effects.
Variational dropout (VD) is a generalization of Gaussian dropout, which aims at inferring the posterior of network weights based on a log-uniform prior on them to learn these weights as well as dropout rate simultaneously. The log-uniform prior not only interprets the regularization capacity of Gaussian dropout in netw…
Study on Monte-Carlo dropout for uncertainty estimation in neural networks.
The paper improves dropout's utility by reducing interactions in deep neural networks.
Dropout controls model capacity in deep learning and matrix completion.
We explore a recently proposed Variational Dropout technique that provided an elegant Bayesian interpretation to Gaussian Dropout. We extend Variational Dropout to the case when dropout rates are unbounded, propose a way to reduce the variance of the gradient estimator and report first experimental results with individ…
A fast single-shot MC dropout method for neural networks.
An important problem in training deep networks with high capacity is to ensure that the trained network works well when presented with new inputs outside the training dataset. Dropout is an effective regularization technique to boost the network generalization in which a random subset of the elements of the given data …
Improves Bayesian neural networks inference efficiency and accuracy.
Bayesian inference using stochastic neural networks ensembles.
Dropout neural networks can approximate any function with high probability.
Great successes of deep neural networks have been witnessed in various real applications. Many algorithmic and implementation techniques have been developed, however, theoretical understanding of many aspects of deep neural networks is far from clear. A particular interesting issue is the usefulness of dropout, which w…
As one of standard approaches to train deep neural networks, dropout has been applied to regularize large models to avoid overfitting, and the improvement in performance by dropout has been explained as avoiding co-adaptation between nodes. However, when correlations between nodes are compared after training the networ…
Dropout is a very effective way of regularizing neural networks. Stochastically "dropping out" units with a certain probability discourages over-specific co-adaptations of feature detectors, preventing overfitting and improving network generalization. Besides, Dropout can be interpreted as an approximate model aggregat…
Efficient FPGA dropout algorithm reduces memory usage.
Enhances uncertainty estimation in neural networks using Dirichlet-based MC Dropout.
Mean field theory explains gradient backpropagation in deep dropout networks.
Deep neural networks (DNNs) are known vulnerable to adversarial attacks. That is, adversarial examples, obtained by adding delicately crafted distortions onto original legal inputs, can mislead a DNN to classify them as any target labels. This work provides a solution to hardening DNNs under adversarial attacks through…
Deep neural networks with their large number of parameters are highly flexible learning systems. The high flexibility in such networks brings with some serious problems such as overfitting, and regularization is used to address this problem. A currently popular and effective regularization technique for controlling the…
We give a formal and complete characterization of the explicit regularizer induced by dropout in deep linear networks with squared loss. We show that (a) the explicit regularizer is composed of an -path regularizer and other terms that are also re-scaling invariant, (b) the convex envelope of the induced regula…
Study on MC dropout in wide neural networks and its convergence to Gaussian processes.
Recurrent neural networks (RNNs) are important class of architectures among neural networks useful for language modeling and sequential prediction. However, optimizing RNNs is known to be harder compared to feed-forward neural networks. A number of techniques have been proposed in literature to address this problem. In…
Study shows critical initialisation not crucial for ReLU networks under dropout limits.
Dropout increases the generalization of neural networks by expanding the weight space.