Bayesian Layers adds uncertainty to neural networks, enabling faster experimentation and scalability.
problem Enabling neural networks to quantify uncertainty in predictions.
method Drop-in replacements for common layers, capturing uncertainty over weights, activations, etc.
result Bayesian Layers can fit large models like 5-billion parameter Bayesian Transformers.
Study on deep neural networks using concentration inequalities and optimal stopping.
problem Understanding the performance and structure of stochastic deep neural networks.
method Introduced concentration inequalities for SDNN outputs and an EC classifier. Determined the optimal number of layers via an optimal stopping procedure.
result Optimal number of layers for SDNNs determined via an optimal stopping procedure.
Stochastic neural networks can approximate any function, even with correlated outputs.
problem Approximating functions with stochastic outputs and correlations.
method Investigating deep sigmoid belief networks to approximate any stochastic mapping.
result Minimal number of layers and units needed for approximation.
Proposes BATer for improved adversarial example detection.
problem Detecting adversarial examples in neural networks.
method Introduces a Bayesian adversarial example detector (BATer) using random components in a Bayesian neural network.
result BATer outperforms state-of-the-art detectors in adversarial example detection.
New bounds enable training of probabilistic models for deep networks.
problem Training scalable latent variable models for deep networks.
method Introducing new variational bounds for specific output layers of neural networks.
result Analytical bounds for certain output layers allow training without re-parameterization or Monte Carlo approximations.
Enhances neural network regularization with no extra cost.
problem Improving neural network robustness and generalization.
method Train an ensemble of weight matrices with stochastic regularization and explicitly average outputs.
result Consistent improvement on various image classification tasks.
Deep Gaussian processes reduce uncertainty in porous media flow modeling.
problem Uncertainty quantification in flow through heterogeneous porous media.
method Multi-layer hierarchical Gaussian process with variational approximation.
result Automatic selection of hidden layer dimensions and uncertainty propagation.
BSF algorithm reduces neural network size and selects features efficiently.
problem Neural network size and feature selection optimization.
method Binary Stochastic Filtering (BSF) layer that penalizes information, stochastically passes or drops features.
result Multifold decrease in neural network size and optimal feature selection.
Stochastic LWTA networks resist adversarial attacks while maintaining accuracy.
problem Adversarial robustness of neural networks.
method Replaced ReLU with stochastic LWTA activations, trained with Variational Bayesian and PGD.
result Stochastic LWTA networks achieve state-of-the-art robustness against adversarial attacks.
Recently, we proposed to transform the outputs of each hidden neuron in a multi-layer perceptron network to have zero output and zero slope on average, and use separate shortcut connections to model the linear dependencies instead. We continue the work by firstly introducing a third transformation to normalize the scal…
Deep linear ResNets converge globally with certain transformations.
problem Global convergence of training deep linear ResNets.
method Gradient descent and stochastic gradient descent for training L-hidden-layer linear ResNets. result GD and SGD can converge to global minimum for deep linear ResNets with specific transformations.
A new method for learning Bayesian neural networks using layerwise inference.
problem Learning Bayesian neural networks efficiently and accurately.
method Bayesian layerwise inference, treating neural networks as stacked Bayesian linear models, with pseudo-targets defined by backpropagated gradients.
result The method converges quickly and performs well on various benchmarks.
We analyze deep neural networks in the large size and iteration limit, revealing a deterministic system of equations.
problem Understanding the behavior of deep neural networks in the asymptotic regime of large network sizes and iterations.
method Sequential limit of each hidden layer and characterization of parameter evolution, using weak convergence and stochastic analysis.
result The limit neural network recovers a global minimum with zero loss for the objective function.
The study shows removing fully connected output layers improves efficiency without sacrificing performance.
problem Large number of parameters in fully connected layers for high-category datasets.
method Examined architectures replacing fully connected output layers with fixed layers and compared performance.
result Fixed classifiers offer no additional benefit over removing the output layer and its parameters.
ReLU networks trained with MILPs match deep learning accuracy.
problem Training deep neural networks efficiently.
method Iterative training with Mixed Integer Linear Programs (MILPs).
result ReLU networks can be trained with MILPs achieving similar accuracy to deep learning methods.
We establish upper bounds for the minimal number of hidden units for which a binary stochastic feedforward network with sigmoid activation probabilities and a single hidden layer is a universal approximator of Markov kernels. We show that each possible probabilistic assignment of the states of n output units, given t…
Conditional restricted Boltzmann machines are undirected stochastic neural networks with a layer of input and output units connected bipartitely to a layer of hidden units. These networks define models of conditional probability distributions on the states of the output units given the states of the input units, parame…
Deep generative networks provide a powerful tool for modeling complex data in a wide range of applications. In inverse problems that use these networks as generative priors on data, one must often perform inference of the inputs of the networks from the outputs. Inference is also required for sampling during stochastic…
Combines deep and statistical learning for structured data.
problem Structured high-dimensional data challenges.
method Generates nonlinear features via sparse regularization and stochastic optimisation, uses probabilistic output layer for uncertainty.
result Achieves best of scalability and uncertainty quantification.
Algorithm predicts performance of learning in multi-layer networks with matrix-valued hidden variables.
problem Signal recovery and learning in multi-layer neural networks with matrix-valued hidden variables.
method Unified approximation algorithm for MAP and MMSE inference, extending ML-VAMP to handle matrix-valued unknowns.
result Performance of ML-Mat-VAMP algorithm can be predicted in a random large-system limit.
Two-layer CNNs can overfit well if initialized correctly.
problem Understanding the conditions for benign overfitting in over-parameterized CNNs.
method Extending analysis to fully trainable two-layer CNNs, examining initialization scaling effects.
result Initialization scaling of the output layer is crucial; large scales lead to fixed output behavior, small scales to complex interactions.
New bounds for neural networks without loss boundedness assumption.
problem Generalization error bounds for two-layer neural networks.
method Wasserstein distance estimates and moment bounds for stochastic gradient method.
result Dimension-free rate of order O(n−1/2) for independent test data. DeepRSCN models nonlinear systems using stochastic configurations.
problem Modeling nonlinear dynamic systems efficiently.
method Incrementally constructed deep reservoir computing framework with random parameters and online weight updates.
result DeepRSCN outperforms single-layer networks in efficiency, learning, and generalization.
New deep learning model robust to adversarial attacks using stochastic LWTA units.
problem Adversarial robustness in deep learning networks.
method Introduces deep networks with stochastic LWTA activations, combining them with Bayesian non-parametric tools.
result Achieves high robustness to adversarial perturbations, outperforming state-of-the-art methods.
In this article we review computational aspects of Deep Learning (DL). Deep learning uses network architectures consisting of hierarchical layers of latent variables to construct predictors for high-dimensional input-output models. Training a deep learning architecture is computationally intensive, and efficient linear…
Our proposed deeply-supervised nets (DSN) method simultaneously minimizes classification error while making the learning process of hidden layers direct and transparent. We make an attempt to boost the classification performance by studying a new formulation in deep networks. Three aspects in convolutional neural netwo…
Stochastic gradient descent (SGD) has achieved great success in training deep neural network, where the gradient is computed through back-propagation. However, the back-propagated values of different layers vary dramatically. This inconsistence of gradient magnitude across different layers renders optimization of deep …
Paper shows SGD can learn RNNs efficiently for certain functions.
problem Understanding what concept class RNNs can learn and how efficiently.
method Vanilla stochastic gradient descent (SGD) approach.
result RNNs can learn some functions efficiently with polynomial complexity in input length.
This paper proposes a fast method for estimating input-dependent prediction intervals in Extreme Learning Machines.
problem Estimating reliable prediction intervals for Extreme Learning Machines with heteroscedastic outputs.
method A separate Extreme Learning Machine model estimates input-dependent prediction intervals using a weighted Jackknife method to correct for model uncertainty.
result The proposed method is fast, robust to heteroscedastic outputs, and handles large datasets and insufficient training data.
Study on Bayesian deep linear networks with multiple outputs and convolutional layers.
problem Characterize feature learning in finite-width Bayesian deep linear networks.
method Exact and analytical formulas for joint and posterior distributions, using large deviation theory.
result Quantitative description of feature learning in infinite-width regime.
A residual network (or ResNet) is a standard deep neural net architecture, with state-of-the-art performance across numerous applications. The main premise of ResNets is that they allow the training of each layer to focus on fitting just the residual of the previous layer's output and the target output. Thus, we should…
Study clarifies Bayesian generalization error in CBM for 3-layered linear neural networks.
problem Understanding the generalization error in concept bottleneck models.
method Mathematical analysis of Bayesian generalization error and free energy in CBM for 3-layered linear neural networks.
result CBM significantly alters the parameter region and Bayesian generalization error compared to standard models.
NOVAS uses adaptive stochastic search for non-convex optimization in deep networks.
problem Non-convex optimization challenges in deep neural networks.
method Adaptive stochastic search for non-convex optimization.
result NOVAS outperforms existing alternatives in a structured prediction task.
Neural networks solve copositive programs, revealing insights into training problems.
problem Training two-layer vector-output ReLU neural networks.
method Convex analysis and copositive programming.
result Neural networks solve copositive programs, providing insights into training problems.
In this paper we first present a class of algorithms for training multi-level neural networks with a quadratic cost function one layer at a time starting from the input layer. The algorithm is based on the fact that for any layer to be trained, the effect of a direct connection to an optimized linear output layer can b…
In this paper we introduce deep Gaussian process (GP) models. Deep GPs are a deep belief network based on Gaussian process mappings. The data is modeled as the output of a multivariate GP. The inputs to that Gaussian process are then governed by another GP. A single layer model is equivalent to a standard GP or the GP …
RaNNDy uses randomized neural networks to learn transfer operators efficiently.
problem Efficiently learning transfer operators from data.
method Randomized neural network approach with randomly initialized hidden layers and trained output layer.
result Significant reduction in training time and resources with improved stability.
Stochastic binary hidden units in a multi-layer perceptron (MLP) network give at least three potential benefits when compared to deterministic MLP networks. (1) They allow to learn one-to-many type of mappings. (2) They can be used in structured prediction problems, where modeling the internal structure of the output i…
The paper proves neural networks with ReLU and softmax can approximate any function.
problem Approximating functions and class labels in neural networks.
method Extended universal approximator theory to neural networks with ReLU and softmax.
result Neural networks with ReLU and softmax can approximate any function and class labels.
Stochastic gradient method converges as fast as deterministic for overparametrized models.
problem Convergence rate of stochastic gradient methods in overparametrized models.
method Proposes a regularity condition enabling fast convergence of SGD.
result Stochastic gradient method achieves the same convergence rate as deterministic gradient method.
This research analyzes how input and output layers affect deep neural networks' resistance to adversarial attacks.
problem The vulnerability of deep neural networks to adversarial inputs, especially non-gradient based attacks.
method Analysis of three different fully connected dense network classes with manipulated input and output layers.
result Manipulating input and output layers can significantly enhance a deep neural network's robustness against adversarial attacks.
Study on neural network initialization with shaped infinite depth-and-width networks.
problem Understanding the distribution of random covariance matrices in shaped infinite-depth-and-width networks.
method Introduced the Neural Covariance SDE to model the distribution of the random covariance matrix.
result Identified the precise scaling of the activation function necessary for a non-trivial limit.
Paper presents a defense framework against adversarial examples.
problem Vulnerability of deep neural networks to adversarial examples.
method Cross-layer strategic ensemble defense with input and output transformations.
result Strategic ensemble defense achieves high defense success rates and robustness.
Formalizes explanations as blending input and model output.
problem Creating clear and consistent explanations for model predictions.
method Defines properties of explanation functions and links them to model layers.
result Consistency of activations across layers implies consistency of explanations.
Bayesian approach improves deep image prior for image reconstruction.
problem Improving performance of deep image prior for image reconstruction tasks.
method Derive Bayesian approach using stochastic gradient Langevin, showing asymptotic equivalence to Gaussian process prior.
result Improves denoising and impainting results for image reconstruction tasks.
New normalization technique balances positive and negative weights for faster convergence.
problem Balancing positive and negative weights for faster convergence.
method Transformation of layer weights instead of outputs, balancing positive and negative contributions.
result Balanced normalization leads to faster convergence on standard benchmarks.
Study defends shallow neural networks from data-poisoning attacks.
problem Protecting shallow neural networks from adversarial attacks during training.
method Developed a non-gradient stochastic algorithm for depth-2 neural networks, proving near-optimal trade-offs.
result Demonstrated improved performance over stochastic gradient descent under various data distributions.
We use smoothed analysis techniques to provide guarantees on the training loss of Multilayer Neural Networks (MNNs) at differentiable local minima. Specifically, we examine MNNs with piecewise linear activation functions, quadratic loss and a single output, under mild over-parametrization. We prove that for a MNN with …