Select-DC reduces GFLOPS for uncertainty estimation in neural networks.
problem Computational inefficiency in estimating model uncertainty for low-latency applications.
method Select-DC uses a subset of layers to model epistemic uncertainty with MCDC, reducing GFLOPS.
result Significant reduction in GFLOPS required for uncertainty estimation with marginal performance loss.
DropConnect improves uncertainty estimation in Bayesian deep networks.
problem Modeling uncertainty in Bayesian deep networks for safety-critical applications.
method Developed a theoretical framework to approximate Bayesian inference for DNNs using MC-DropConnect.
result Significant improvement in both prediction accuracy and uncertainty estimation quality.
Dropout improves neural network performance by promoting low-rank solutions.
problem Improving neural network generalization through regularization.
method Analyzing Dropout, DropBlock, and DropConnect as regularizers for linear networks and extending to deep networks.
result Dropout, DropBlock, and DropConnect induce low-rank solutions and can be computed in closed form.
A new neural network architecture reduces parameters by 94% while maintaining performance.
problem Reduction of trainable parameters in neural networks.
method Spatially-coupled sparse construction to allocate trainable parameters efficiently.
result Performance comparable to traditional neural networks with 94% fewer parameters.
Derives VC bounds for deep neural networks using radius-margin approach.
problem Understanding the capacity of deep neural networks.
method Applies radius-margin bounds from SVM to deep feed-forward architectures.
result Derives new VC bounds different from earlier ones.
Bayesian Neural Networks improve uncertainty modeling in facial emotion recognition.
problem High aleatoric uncertainty and visual ambiguity in facial emotion recognition.
method Bayesian Neural Networks approximated using MC-Dropout, MC-DropConnect, or Ensemble methods.
result Bayesian Neural Networks produce more human-like output probabilities.
Hidden Markov Neural Networks balance adaptation and forgetting in time-series data.
problem Balancing adaptation to new data and forgetting outdated information in time-series forecasting.
method Modeling weights as hidden states of a Hidden Markov model, using a filtering algorithm for learning a variational approximation of the posterior distribution over weights, and employing sequential Bayes by Backprop with variational DropConnect for regularization.
result Achieves strong predictive performance and effective uncertainty quantification on various tasks.
We investigate the use of Deep Neural Networks for the classification of image datasets where texture features are important for generating class-conditional discriminative representations. To this end, we first derive the size of the feature space for some standard textural features extracted from the input dataset an…
We introduce Delay Pruning, a simple yet powerful technique to regularize dynamic Boltzmann machines (DyBM). The recently introduced DyBM provides a particularly structured Boltzmann machine, as a generative model of a multi-dimensional time-series. This Boltzmann machine can have infinitely many layers of units but al…
Bayesian inference using stochastic neural networks ensembles.
problem Approximating Bayesian posterior distributions.
method Formulate stochastic ensembles of neural networks, train with variational inference, and evaluate using Monte Carlo dropout.
result Stochastic ensembles provide more accurate posterior estimates than other methods.
Regularization is essential when training large neural networks. As deep neural networks can be mathematically interpreted as universal function approximators, they are effective at memorizing sampling noise in the training data. This results in poor generalization to unseen data. Therefore, it is no surprise that a ne…
Paper introduces DropFilter and DropFilter-PLUS for CNN regularization.
problem Overfitting in Convolutional Neural Networks (CNNs).
method Randomly modifies convolution filters in CNNs.
result Improves performance on image classification tasks.
Neural models can realize decision trees with parameter sharing and improved performance.
problem Training and optimizing oblique decision trees.
method Locally constant networks based on ReLU gradients, parameter sharing, and neural tools.
result Locally constant networks can implicitly model oblique decision trees with fewer neurons.
Dropout speeds up convergence in shallow linear NNs, with a rate bound.
problem Analyzing convergence rate of Dropout in shallow linear NNs.
method Gradient flow analysis and Hessian examination.
result Bound on convergence rate depends on data, dropout probability, and NN width.
Deep neural networks (DNNs) often require good regularizers to generalize well. Currently, state-of-the-art DNN regularization techniques consist in randomly dropping units and/or connections on each iteration of the training algorithm. Dropout and DropConnect are characteristic examples of such regularizers, that are …
Continuous dropout mimics brain neuron firing rates to prevent feature detector co-adaptation.
problem Preventing overfitting in deep neural networks.
method Extending binary dropout to continuous dropout, inspired by brain neuron firing rates.
result Continuous dropout improves test performance by preventing feature detector co-adaptation.
RIFLE improves deep transfer learning by reinitializing fully-connected layers.
problem Limited improvement in transfer learning accuracy with pre-trained models on small datasets.
method Re-Initializing fully-connected layers with random scratch during fine-tuning.
result Significant improvement in deep transfer learning accuracy across various datasets.