Formula found for neural network error with fixed weights.
problem Understanding error in neural networks with fixed weights.
method Provided an explicit formula for approximation error.
result Explicit formula for neural network error with fixed weights.
Universal MLPs with a single hidden layer can learn any function.
problem Learning on various data structures like sequences, images, sets, and graphs.
method Using group theory, the paper proves the universality of a broad class of equivariant MLPs with a single hidden layer.
result Having a hidden layer on which the group acts regularly is sufficient for universal equivariance (invariance).
In this paper, we introduce transformations of deep rectifier networks, enabling the conversion of deep rectifier networks into shallow rectifier networks. We subsequently prove that any rectifier net of any depth can be represented by a maximum of a number of functions that can be realized by a shallow network with a …
LIFE framework improves model accuracy and interpretability.
problem Achieving high prediction accuracy and interpretability in neural networks.
method Three-step process: subset definition, feature creation, and linear model combination.
result LIFE consistently outperforms other models in prediction accuracy and interpretability.
A neural network with a single hidden layer can't represent certain multivariable functions.
problem Representing certain multivariable functions with a neural network having only one hidden layer.
method Developed a continuum version of a one-hidden-layer neural network with ReLU activation, and proved constraints on its parameters and second derivative.
result Existence of a smooth binary function that cannot be precisely represented by any such neural network.
To infer a multilayer representation of high-dimensional count vectors, we propose the Poisson gamma belief network (PGBN) that factorizes each of its layers into the product of a connection weight matrix and the nonnegative real hidden units of the next layer. The PGBN's hidden layers are jointly trained with an upwar…
The muti-layer information bottleneck (IB) problem, where information is propagated (or successively refined) from layer to layer, is considered. Based on information forwarded by the preceding layer, each stage of the network is required to preserve a certain level of relevance with regards to a specific hidden variab…
Traditionally, when generative models of data are developed via deep architectures, greedy layer-wise pre-training is employed. In a well-trained model, the lower layer of the architecture models the data distribution conditional upon the hidden variables, while the higher layers model the hidden distribution prior. Bu…
Gradient descent proves global convergence for deep networks with a single wide layer.
problem Proving global convergence of gradient descent for deep ReLU networks.
method Simplified proof using a single wide layer, leveraging ReLU's Lipschitz property.
result Gradient descent converges globally for networks with a single wide layer.
Lean 2-layer RBMs achieve similar representational power as single-layer RBMs with fewer parameters.
problem Understanding and quantifying the representational power of multi-layer RBMs.
method Inherent Structure Capacity (ISC) and Lean RBMs.
result 2-layer RBMs can achieve the same representational power as single-layer RBMs with fewer parameters.
Deep ResNets with single-neuron hidden layers can approximate any function.
problem The challenge of universal approximation by deep neural networks.
method A ResNet architecture with one neuron per hidden layer in each module.
result ResNet with one-neuron hidden layers is a universal approximator.
Paper discusses the pseudoinverse learning algorithm and its variants.
problem Improving learning algorithms for neural networks.
method Review and discussion of the pseudoinverse learning algorithm and its variants.
result Extreme Learning Machine (ELM) is a variant of the pseudoinverse learning algorithm.
Single wide layer followed by a pyramidal structure ensures global convergence in deep networks.
problem Ensuring global convergence in deep neural networks with limited width constraints.
method Proves that a single wide layer followed by a pyramidal structure guarantees global convergence for over-parameterized networks.
result Single wide layer of width N suffices for global convergence in deep networks with constant-width remaining layers. Neural networks can approximate any continuous function with a single hidden layer.
problem Approximating any continuous function using neural networks.
method Direct algebraic proof and explicit quantification of hidden units required.
result Explicit bounds on the number of hidden units needed for approximation.
Enhanced ELM reduces randomness in neural network training.
problem Challenges in ELM architecture design and sensitivity to random weight initialization.
method Introduces Effective Non-Random ELM (ENR-ELM) incorporating signal processing concepts.
result ENR-ELM simplifies architecture design and eliminates random weight selection.
Single neuron with ADA learns XOR and outperforms other functions.
problem Classifying linearly non-separable data.
method Proposed a new artificial neuron with apical dendrite activation.
result ADA function achieves 100% accuracy on XOR and superior performance on benchmark datasets.
We establish upper bounds for the minimal number of hidden units for which a binary stochastic feedforward network with sigmoid activation probabilities and a single hidden layer is a universal approximator of Markov kernels. We show that each possible probabilistic assignment of the states of n output units, given t…
Softmax policy gradient achieves global optimality in wide neural networks with entropy regularization.
problem Optimizing softmax policies with neural networks in the mean-field regime.
method Modeling neural networks as Wasserstein gradient flows and proving global optimality of fixed points.
result Global optimality of softmax policy gradient in wide single hidden layer neural networks with entropy regularization.
New method trains neural networks with local error signals, outperforming global methods.
problem Training neural networks with global error signals.
method Layer-wise training with local error signals.
result Layer-wise training with local error signals can approach state-of-the-art performance.
We study how finite Bayesian neural networks adapt their hidden representations.
problem Understanding how finite Bayesian neural networks differ from infinite ones.
method We analyze the asymptotics of learned feature kernels for various network architectures.
result The leading finite-width corrections to feature kernels have a universal form.
It is well-known that neural networks are universal approximators, but that deeper networks tend in practice to be more powerful than shallower ones. We shed light on this by proving that the total number of neurons m required to approximate natural classes of multivariate polynomials of n variables grows only line…
Recurrent Neural Networks (RNNs), which are a powerful scheme for modeling temporal and sequential data need to capture long-term dependencies on datasets and represent them in hidden layers with a powerful model to capture more information from inputs. For modeling long-term dependencies in a dataset, the gating mecha…
Proves a central limit theorem for neural networks with hidden layers.
problem Understanding the statistical behavior of neural networks with large numbers of hidden units and training iterations.
method Rigorous mathematical proof using weak convergence methods and stochastic analysis.
result Neural network fluctuations around mean-field limit follow a Gaussian distribution and satisfy a stochastic partial differential equation.
Negative results for neural network approximations on multi-dimensional spaces.
problem Approximating functions on compact subsets of R^d (d≥2) using neural networks.
method Proof of negative results for single and multi-layer feedforward neural networks with various activation functions.
result Existence of target functions difficult to approximate by these neural networks.
We apply the spike-and-slab Restricted Boltzmann Machine (ssRBM) to texture modeling. The ssRBM with tiled-convolution weight sharing (TssRBM) achieves or surpasses the state-of-the-art on texture synthesis and inpainting by parametric models. We also develop a novel RBM model with a spike-and-slab visible layer and bi…
Parsimonious Bayesian deep networks infer optimal architectures from data.
problem Optimizing deep network architectures for efficiency and accuracy.
method Combining Bayesian nonparametrics with a greedy layer-wise learning algorithm.
result Achieves state-of-the-art classification accuracy with low computational complexity.
Paper introduces a new method to compute pseudoinverse for ELM with large datasets.
problem Efficient computation of pseudoinverse for ELM with large datasets.
method Rank-based matrix decomposition of the hidden layer matrix.
result Optimal training time and reduced computational complexity for large hidden nodes.
This paper provides a theoretical justification of the superior classification performance of deep rectifier networks over shallow rectifier networks from the geometrical perspective of piecewise linear (PWL) classifier boundaries. We show that, for a given threshold on the approximation error, the required number of b…
SNEPPPs use squared neural networks to efficiently model Poisson point processes.
problem Efficiently modeling Poisson point processes with flexibility.
method Parameterizing intensity function with squared norm of a two-layer neural network.
result Closed-form integration of intensity function for quadratic time computation.
Shallow networks with local learning rules can match deep learning performance.
problem Training deep neural networks is biologically implausible; the goal is to achieve similar performance with shallow networks.
method Investigated shallow networks with one hidden layer and a single readout layer, using various local learning rules for the hidden layer and supervised learning for the readout layer.
result Shallow networks can achieve test accuracy comparable to deep learning models, suggesting the use of different datasets for testing.
Deep neural networks can efficiently classify Gaussian mixture models with fewer neurons.
problem Classifying Gaussian mixture models efficiently.
method Used deep neural networks with two hidden layers to classify GMMs optimally.
result Deep neural networks require significantly fewer neurons than shallow ones for optimal classification of GMMs.
We study model recovery for data classification, where the training labels are generated from a one-hidden-layer neural network with sigmoid activations, also known as a single-layer feedforward network, and the goal is to recover the weights of the neural network. We consider two network models, the fully-connected ne…
Establishes connection between MTDNN and multitask GP, revealing weight correlation as key to task sharing.
problem Limited theoretical understanding of information sharing in MTDNN.
method Derives multitask GP kernels for MTDNN and MTBNN, showing shared hyper-parameters and last layer weights.
result Information sharing in MTDNN is due to weight correlation, not intermediate layer weights.
Learning and inferring features that generate sensory input is a task continuously performed by cortex. In recent years, novel algorithms and learning rules have been proposed that allow neural network models to learn such features from natural images, written text, audio signals, etc. These networks usually involve de…
Dropout biases neural networks by equalizing hidden node weights.
problem Understanding implicit bias in dropout for neural networks.
method Characterization of optimization landscape for linear neural networks with dropout.
result Dropout equalizes the norm of hidden node weight vectors.
Extreme learning machine (ELM) is a new single hidden layer feedback neural network. The weights of the input layer and the biases of neurons in hidden layer are randomly generated, the weights of the output layer can be analytically determined. ELM has been achieved good results for a large number of classification ta…
Dropout improves neural network performance by promoting low-rank solutions.
problem Improving neural network generalization through regularization.
method Analyzing Dropout, DropBlock, and DropConnect as regularizers for linear networks and extending to deep networks.
result Dropout, DropBlock, and DropConnect induce low-rank solutions and can be computed in closed form.
3-layer NTK models generalize better than 2-layer models, especially with large input dimensions.
problem Understanding the generalization of overparameterized neural networks.
method Analyzing the 3-layer NTK model's test error and comparing it to 2-layer NTK models.
result 3-layer NTK models have a faster descent in test error with respect to the number of neurons in the second hidden layer.
To infer multilayer deep representations of high-dimensional discrete and nonnegative real vectors, we propose an augmentable gamma belief network (GBN) that factorizes each of its hidden layers into the product of a sparse connection weight matrix and the nonnegative real hidden units of the next layer. The GBN's hidd…
Wide neural networks with narrow bottlenecks behave like deep Gaussian processes.
problem Understanding the behavior of neural networks with narrow layers in the wide limit.
method Analyzing the wide limit of BNNs with narrow bottlenecks, showing they behave like a composition of GPs.
result Wide neural networks with narrow bottlenecks form a composition of GPs, termed a bottleneck NNGP.
We use smoothed analysis techniques to provide guarantees on the training loss of Multilayer Neural Networks (MNNs) at differentiable local minima. Specifically, we examine MNNs with piecewise linear activation functions, quadratic loss and a single output, under mild over-parametrization. We prove that for a MNN with …
Deep Gaussian processes reduce uncertainty in porous media flow modeling.
problem Uncertainty quantification in flow through heterogeneous porous media.
method Multi-layer hierarchical Gaussian process with variational approximation.
result Automatic selection of hidden layer dimensions and uncertainty propagation.
Background: Statistical mechanics results (Dauphin et al. (2014); Choromanska et al. (2015)) suggest that local minima with high error are exponentially rare in high dimensions. However, to prove low error guarantees for Multilayer Neural Networks (MNNs), previous works so far required either a heavily modified MNN mod…
A method to interpret neural networks by isolating active subnetworks.
problem Interpreting the complex computations of deep neural networks.
method Switched linear projections to decompose network activity.
result Patterns in deactivated neurons are meaningful for network understanding.
Study shows training input-to-hidden weights reduces generalization error in wide neural networks.
problem Understanding the mathematical properties of hierarchical neural networks in the infinite-width limit.
method Examined a three-layer neural network with a large number of hidden units, comparing training input-to-hidden weights to keeping them fixed.
result Training input-to-hidden weights yields a smaller generalization error and avoids singularities.
Randomly chosen primary hidden units and derived secondary units reduce neural network complexity.
problem Large number of hidden units in neural networks.
method Introducing primary and secondary hidden units with random weights for primary units and derived weights for secondary units.
result Significant reduction in the number of hidden units without compromising accuracy.
Algorithm approximates continuous functions using a neural network with one hidden layer.
problem Finding the weights/parameters for a neural network to approximate continuous functions.
method Algorithm outlines an approach to reconstruct any continuous function using a neural network with one hidden layer.
result Algorithm successfully reconstructs any continuous function using a neural network with one hidden layer.
Closed-form variational objectives for Bayesian neural networks with ReLU layers.
problem Efficient computation of Bayesian neural networks with closed-form variational objectives.
method Single-layer networks with piecewise polynomial activations (ReLU). Structured Normal variational distributions for Normal likelihoods. Approximate lower bounds for other likelihoods.
result Closed-form computation of variational lower bounds, predictive mean, and variance for Bayesian neural networks.