Randomly chosen primary hidden units and derived secondary units reduce neural network complexity.
problem Large number of hidden units in neural networks.
method Introducing primary and secondary hidden units with random weights for primary units and derived weights for secondary units.
result Significant reduction in the number of hidden units without compromising accuracy.
Study examines dependence properties of Bayesian neural network units in finite-width networks.
problem Understanding dependence properties of hidden units in practical finite-width Bayesian neural networks.
method Theoretical analysis and empirical evaluation of depth and width impacts.
result Hidden units in finite-width Bayesian neural networks are dependent, contrary to the infinite-width limit assumption.
Study on hidden units in finite Bayesian neural networks and their tail properties.
problem Understanding the behavior of hidden units in finite Bayesian neural networks.
method Introduced a generalized Weibull-tail property to describe hidden units tails.
result Unit priors become heavier-tailed going deeper, providing insights into finite Bayesian neural networks.
Extends neuron importance to hidden units using conductance.
problem Understanding the importance of hidden units in deep networks.
method Introducing conductance to measure hidden unit importance.
result Effective in identifying important hidden units for predictions.
Reduced hidden units in RBM without performance loss.
problem Maintaining performance with fewer hidden units in RBM.
method Proposed algorithm to decrease hidden units while keeping performance constant.
result Demonstrated through numerical simulations that fewer hidden units can be used without performance degradation.
Deep belief networks can approximate any multivariate density with binary hidden units.
problem Approximating multivariate probability densities with binary hidden units.
method Sharp quantitative bounds on approximation error in terms of hidden units.
result Deep belief networks can approximate any multivariate density with binary hidden units under mild integrability requirements.
Since learning is typically very slow in Boltzmann machines, there is a need to restrict connections within hidden layers. However, the resulting states of hidden units exhibit statistical dependencies. Based on this observation, we propose using l1/l2 regularization upon the activation possibilities of hidden unit…
Self-regularizing RBMs learn optimal hidden units efficiently.
problem Learning optimal number of hidden units in RBMs.
method Grand-canonical extension of RBMs with varying hidden units, using chemical potential to control size.
result Efficiently deduces optimal number of hidden units with small generalization error.
RBM models reveal how hidden unit tail behavior affects pattern reconstruction.
problem Understanding how the tail behavior of hidden units in RBMs influences pattern reconstruction.
method Identified an effective energy function for RBMs and studied its local minima.
result The ability to reconstruct patterns depends on the tail behavior of the hidden unit prior distribution.
Minimal model reveals how permutation symmetry affects unsupervised learning.
problem Understanding permutation symmetry in unsupervised learning.
method Proposed a minimal model with two hidden units in a restricted Boltzmann machine.
result Proved that critical data size is independent of the number of hidden units once finite.
Model shows neural nets can learn categorical perception.
problem Understanding how categorical perception arises in neural networks.
method Developed a neural network model to simulate learning-induced categorical perception.
result Neural nets can learn to perceive categories in a way similar to human perception.
Deep learning is the state-of-the-art in fields such as visual object recognition and speech recognition. This learning uses a large number of layers, huge number of units, and connections. Therefore, overfitting is a serious problem. To avoid this problem, dropout learning is proposed. Dropout learning neglects some i…
A novel training strategy speeds up learning of infinite RBM models.
problem Slow convergence of infinite RBM models due to dependency between hidden units.
method Randomly regrouping hidden units before each gradient descent step.
result Significant acceleration in learning and enhancement of generalization ability.
ReLU activations lead to smoother learning curves compared to sigmoidal activations in neural networks.
problem Comparing the performance of ReLU and sigmoidal activations in neural networks.
method Analytical computation of learning curves in shallow networks with different activation functions.
result ReLU networks exhibit continuous transitions in performance, while sigmoidal networks show discontinuous transitions.
New method reduces overfitting in deep neural networks by measuring and regulating hidden unit diversity.
problem Overfitting in deep neural networks.
method Introduces a new redundancy measure based on mutual information to improve generalization.
result Reduction of redundancy improves generalization capacity, reducing overfitting.
New method estimates sigmoids' parameters using gradient estimation.
problem Learning parameters of models with hidden variables.
method Estimate gradients at random points, cluster, use cluster centers as parameter estimates.
result Proven that estimated gradients concentrate around true parameter vectors.
Infinite RBMs use Frank-Wolfe for efficient training and initialization.
problem Training infinite RBMs with sparse solutions.
method Frank-Wolfe algorithm for constrained convex optimization.
result Infinite RBMs can be trained efficiently and initialized effectively.
We establish upper bounds for the minimal number of hidden units for which a binary stochastic feedforward network with sigmoid activation probabilities and a single hidden layer is a universal approximator of Markov kernels. We show that each possible probabilistic assignment of the states of n output units, given t…
New MBL hidden Born machine learns various tasks.
problem Learning from quantum many-body systems.
method MBL dynamics and hidden units for training.
result Enhanced trainability and stability in learning.
New approach to deep learning for domain adaptation.
problem Learning a model on a target domain using a similar source domain.
method Introducing a search framework for correct alignment of high-level representations.
result Conceptual domain adaptation improves deep learning performance.
Combining interpretability and stability methods improves DNN robustness.
problem Improving interpretability and robustness of deep neural networks.
method Combining interpretability (conductance) and stability (binary classifier) methods to detect and discard wrong predictions.
result Combining interpretability and stability methods increases model robustness.
Unsupervised deep learning is one of the most powerful representation learning techniques. Restricted Boltzman machine, sparse coding, regularized auto-encoders, and convolutional neural networks are pioneering building blocks of deep learning. In this paper, we propose a new building block -- distributed random models…
A novel quantum model improves RBM performance and is efficiently trainable.
problem Improving the performance of RBM models.
method Quantum model with parametrically coupled fermions to classical signals.
result The model outperforms classical RBM with the same number of hidden units.
Unsupervised learning by hidden units with biological plausibility.
problem Training neural networks without labeled data.
method Global inhibition in hidden layer to learn feature detectors.
result Learned feature detectors enable supervised training of higher layers.
To infer a multilayer representation of high-dimensional count vectors, we propose the Poisson gamma belief network (PGBN) that factorizes each of its layers into the product of a connection weight matrix and the nonnegative real hidden units of the next layer. The PGBN's hidden layers are jointly trained with an upwar…
Automatically updates both network weights and architecture.
problem Manual selection of network architecture limits flexibility and efficiency.
method Continuous parameterization of network depth and automatic adjustment of architecture and weights.
result Correctly adjusts network complexity to task complexity.
We investigate the problem of factorizing a matrix into several sparse matrices and propose an algorithm for this under randomness and sparsity assumptions. This problem can be viewed as a simplification of the deep learning problem where finding a factorization corresponds to finding edges in different layers and valu…
Improved mean-field theory for two-layer neural networks with stronger bounds and generalizations.
problem Learning dynamics of two-layer neural networks using stochastic gradient descent.
method Mean-field approximation and gradient flow in Wasserstein space.
result Stronger approximation guarantees for learning two-layer neural networks, independent of dimensionality.
We present explicit classes of probability distributions that can be learned by Restricted Boltzmann Machines (RBMs) depending on the number of units that they contain, and which are representative for the expressive power of the model. We use this to show that the maximal Kullback-Leibler divergence to the RBM model w…
We introduce dropout compaction, a novel method for training feed-forward neural networks which realizes the performance gains of training a large model with dropout regularization, yet extracts a compact neural network for run-time efficiency. In the proposed method, we introduce a sparsity-inducing prior on the per u…
M-PHATE visualizes neural network learning dynamics.
problem Understanding neural network performance and learning dynamics.
method Multislice PHATE (M-PHATE) for visualizing neural network hidden representations.
result M-PHATE provides detailed summaries of learning dynamics without needing validation data.
Study on neural networks' performance under different normalizations as N grows.
problem Characterizing neural networks' performance under various normalizations.
method Developed an asymptotic expansion to analyze statistical output of shallow neural networks.
result No bias-variance trade-off exists to leading order in N, and variance decreases as normalization approaches mean field.
New framework analyzes deep neural networks using feature probabilities.
problem Degenerate situation in over-parameterized DNNs.
method Mean-field framework representing DNNs by feature probabilities and functions.
result Global convergence proof for over-parameterized Res-Net training.
We present a novel neural network algorithm, the Tensor Switching (TS) network, which generalizes the Rectified Linear Unit (ReLU) nonlinearity to tensor-valued hidden units. The TS network copies its entire input vector to different locations in an expanded representation, with the location determined by its hidden un…
Mathematical methods characterize RNNs' asymptotics as hidden units and data grow.
problem Characterize recurrent neural networks' behavior as hidden units and data grow.
method Developed mathematical methods to analyze RNNs' convergence to an infinite-dimensional ODE coupled with a fixed point of a random algebraic equation.
result RNNs converge to an infinite-dimensional ODE coupled with a fixed point of a random algebraic equation.
Structural RBM reduces parameters for image denoising and classification.
problem High parameter count in RBMs limits their applicability to large datasets.
method Introduces SRBM with constrained connections to reduce parameters.
result SRBM achieves better performance and faster training than vanilla RBM.
We describe a model for capturing the statistical structure of local amplitude and local spatial phase in natural images. The model is based on a recently developed, factorized third-order Boltzmann machine that was shown to be effective at capturing higher-order structure in images by modeling dependencies among squar…
DEVDAN adapts to changing data streams by dynamically adding and removing hidden units.
problem Fixed DAE network capacity limits adaptability to rapidly changing environments.
method DEVDAN features an open structure with dynamically adjustable hidden units.
result DEVDAN outperforms state-of-the-art methods on ten datasets.
Pruning neural networks reduces parameters without sacrificing interpretability.
problem Reducing unnecessary structure in neural networks to improve efficiency.
method Examined the effect of pruning on the number of hidden units learning disentangled representations.
result Pruning does not harm interpretability until a significant portion of parameters are removed.
Bayesian inference for deep neural networks using trace-class priors and MLMC.
problem Efficient Bayesian inference for deep neural networks.
method Trace-class neural network priors and Multilevel Monte Carlo method.
result Optimal computational complexity for Bayesian inference of TNN models.
DSSCN improves lifelong learning of non-stationary data streams through adaptive network construction.
problem Lifelong learning of non-stationary data streams with efficient and adaptive models.
method Deep stacked stochastic configuration network (DSSCN) with self-constructing deep stacked network structure and adaptive hidden unit parameters.
result DSSCN outperforms existing data stream algorithms in continual learning of non-stationary data streams.
Neural networks can approximate any continuous function with a single hidden layer.
problem Approximating any continuous function using neural networks.
method Direct algebraic proof and explicit quantification of hidden units required.
result Explicit bounds on the number of hidden units needed for approximation.
Proposes a method to estimate neural network statistics analytically for normalization.
problem Estimating statistics of hidden units in neural networks for better initialization and normalization.
method Analytic moment propagation of mean and variance through the network structure.
result Analytic estimates of statistics are useful for initialization and normalization, independent of batch input.
Newton's method converges faster than gradient descent in overparameterized neural networks.
problem Training neural networks efficiently in the overparameterized limit.
method Developed a convergence analysis for the regularized Newton method in this context.
result The NN training dynamics converge to the solution of a deterministic limit equation involving a Newton neural tangent kernel (NNTK).
Bayesian neural networks ignore data in infinite units limit.
problem Pathological behavior of posterior in over-parameterized networks.
method Mean-field variational inference in infinite hidden units limit.
result Posterior mean converges to zero, ignoring data.
Proves a central limit theorem for neural networks with hidden layers.
problem Understanding the statistical behavior of neural networks with large numbers of hidden units and training iterations.
method Rigorous mathematical proof using weak convergence methods and stochastic analysis.
result Neural network fluctuations around mean-field limit follow a Gaussian distribution and satisfy a stochastic partial differential equation.
LSTM neural networks improve fiber nonlinearities in coherent systems.
problem Compensating fiber nonlinearities in digital coherent systems.
method Utilization of Long short-term memory (LSTM) neural networks.
result LSTM neural networks provide superior performance compared to digital back propagation, especially in multi-channel scenarios.
This work concerns testing the number of parameters in one hidden layer multilayer perceptron (MLP). For this purpose we assume that we have identifiable models, up to a finite group of transformations on the weights, this is for example the case when the number of hidden units is know. In this framework, we show that …