Study of infinitely deep but narrow neural networks using NTK theory.
problem Analyzing the role of depth in deep learning with overparameterized networks.
method Infinite-depth limit analysis of MLP and CNN using Neural Tangent Kernel (NTK) theory.
result Established trainability guarantee for infinitely deep but narrow neural networks.
We show that for neural network functions that have width less or equal to the input dimension all connected components of decision regions are unbounded. The result holds for continuous and strictly monotonic activation functions as well as for the ReLU activation function. This complements recent results on approxima…
Recent theoretical work has demonstrated that deep neural networks have superior performance over shallow networks, but their training is more difficult, e.g., they suffer from the vanishing gradient problem. This problem can be typically resolved by the rectified linear unit (ReLU) activation. However, here we show th…
Improved bounds on neural network expressivity.
problem Understanding neural network expressivity and approximation capabilities.
method Improved bounds on the maximal number of linear regions of ReLU-networks.
result New insights into the expressivity of neural networks.
Wide networks are often believed to have a nice optimization landscape, but what rigorous results can we prove? To understand the benefit of width, it is important to identify the difference between wide and narrow networks. In this work, we prove that from narrow to wide networks, there is a phase transition from havi…
Embedding principle explains loss landscape of deep neural networks.
problem Understanding the structure of loss landscapes in deep neural networks.
method Proposed an embedding principle that critical points of narrower DNNs can be embedded to critical points of wider DNNs.
result Wide DNNs are often attracted by highly-degenerate critical points embedded from narrower DNNs.
Large learning rates improve generalization, but optimal ranges are narrower than previously thought.
problem Optimizing learning rates for neural network training.
method Detailed exploration of learning rate ranges in a simplified setup, validating findings in a practical setting.
result Optimal learning rate ranges are significantly narrower than previously assumed.
Neural networks can approximate functions uniformly across various measures.
problem Universal approximation of functions across different probability measures.
method Proving neural networks are dense in Orlicz spaces, extending classical theorems.
result Neural networks uniformly approximate functions for weakly compact families of measures.
Bayesian Quadrature improves ensembling for neural networks with dispersed likelihood peaks.
problem Ensembling neural networks struggles with dispersed, narrow peaks in likelihood surfaces.
method Uses Bayesian Quadrature to construct weighted ensembles of architectures.
result Empirically outperforms state-of-the-art baselines in test likelihood, accuracy, and expected calibration error.
We review recent results about the maximal values of the Kullback-Leibler information divergence from statistical models defined by neural networks, including naive Bayes models, restricted Boltzmann machines, deep belief networks, and various classes of exponential families. We illustrate approaches to compute the max…
The classical Universal Approximation Theorem holds for neural networks of arbitrary width and bounded depth. Here we consider the natural `dual' scenario for networks of bounded width and arbitrary depth. Precisely, let n be the number of inputs neurons, m be the number of output neurons, and let ρ be any nonaff…
We show that deep narrow Boltzmann machines are universal approximators of probability distributions on the activities of their visible units, provided they have sufficiently many hidden layers, each containing the same number of units as the visible layer. We show that, within certain parameter domains, deep Boltzmann…
Complex-valued neural networks can approximate any continuous function with bounded widths and depths.
problem Approximating continuous functions with complex-valued neural networks of bounded widths and depths.
method Analyzing activation functions and proving universality for complex-valued networks.
result Deep narrow complex-valued networks are universal if and only if their activation function is neither holomorphic, nor antiholomorphic, nor R-affine. We theoretically study the landscape of the training error for neural networks in overparameterized cases. We consider three basic methods for embedding a network into a wider one with more hidden units, and discuss whether a minimum point of the narrower network gives a minimum or saddle point of the wider one. Our re…
Randomly trained neural networks can generalize well if there's a simpler underlying teacher model.
problem Why randomly trained neural networks generalize well despite interpolating training data.
method Examined a random neural network that interpolates training data and showed it generalizes well if there's a simpler underlying teacher model.
result Randomly trained neural networks can generalize well if there's a simpler underlying teacher model.
Tilting loss functions improves machine learning performance.
problem Improving machine learning models, especially in under- and over-parameterized networks.
method Using evolving loss functions that emphasize different classes cyclically.
result Dynamical loss functions lead to better generalization and stability in training.
This paper presents a method to automatically generate high-quality prediction intervals for neural networks.
problem Accurate uncertainty quantification for deep learning models in real-world applications.
method Dual neural network approach with a novel loss function to balance prediction interval width and coverage.
result Our method produces significantly narrower prediction intervals with higher probability coverage compared to state-of-the-art methods.
This paper improves deep neural network approximation for fully connected networks, achieving optimal convergence rates.
problem Improving approximation of fully connected deep neural networks for optimal convergence rates.
method Deriving approximation bounds specifically for a narrower fully connected deep neural network.
result Achieves an optimal rate (up to a logarithmic factor) for fully connected deep neural networks.
Study proves deep narrow RNNs can approximate any function, with minimum width independent of data length.
problem Proving universality of deep narrow RNNs with bounded widths.
method Analyzing RNNs as dynamical systems, proving universality for deep narrow structures with specific widths.
result Minimum width for universality of deep narrow RNNs is independent of data length.
Study compares random and learned features in deep Bayesian linear models.
problem Understanding how feature learning affects generalization in deep learning.
method Comparing deep random feature models to deep networks with trained layers.
result Random feature models can display double-descent behavior, while deep networks do not.
A new method reduces memory requirements for Graph Transformers by sparsely training a network.
problem Quadratic memory complexity in Graph Transformers limits their scalability to large graphs.
method Spexphormer: trains a narrow network on augmented graph, then uses only active connections in a wider network.
result Spexphormer achieves good performance with drastically reduced memory requirements.
MPNN improves on UniFL approximation with provable guarantees.
problem Uniform Facility Location (UniFL) optimization problem.
method Graph Neural Network (MPNN) incorporating approximation-algorithmic principles.
result Empirically outperforms standard approximation algorithms.
Statistical physics explains deep learning's feature learning capacity.
problem Understanding neural networks' ability to learn complex features.
method Study of a multi-layer perceptron in the interpolation regime.
result Optimal learning requires specialization across layers and neurons.
Deep learning uncovers patterns between knot types.
problem Discovering connections between combinatorial and hyperbolic knot invariants.
method Statistical approach using linear regression and deep learning.
result Revealed empirical connections between knot types.
We prove that for an L-layer fully-connected linear neural network, if the width of every hidden layer is Ω~(L⋅r⋅dout⋅κ3), where r and κ are the rank and the condition number of the input data, and dout is the output dimension, then gradient descent with Gaussi…
Proposes a method to create prediction intervals for neural networks using cross-validation.
problem Lack of prediction intervals for neural networks.
method k-fold cross-validation to construct conformal prediction intervals.
result Proposed method produces narrower intervals with similar coverage compared to SC method.
There has recently been much work on the "wide limit" of neural networks, where Bayesian neural networks (BNNs) are shown to converge to a Gaussian process (GP) as all hidden layers are sent to infinite width. However, these results do not apply to architectures that require one or more of the hidden layers to remain n…
A neural framework corrects bias in estimating individual treatment effects.
problem Estimating individual treatment effects from observational data.
method An anchored neural architecture and precision-corrected intersection-bound inference.
result Corrected bias and maintained nominal coverage in high-dimensional settings.
As deep neural networks continue to revolutionize various application domains, there is increasing interest in making these powerful models more understandable and interpretable, and narrowing down the causes of good and bad predictions. We focus on recurrent neural networks (RNNs), state of the art models in speech re…
Enhances FDR control in variable selection using neural networks.
problem Balancing rigorous error control with statistical power in high-dimensional variable selection.
method Learning-augmented T-Rex Selector framework with a neural network trained on synthetic datasets.
result Achieves superior detection of true variables compared to existing approaches.
Improves matrix multiplication throughput for asymmetric bit-width operands.
problem Matrix multiplications between asymmetric bit-width operands, especially 8- and 4-bit, are not efficiently handled by existing SIMD instructions.
method Proposes a new SIMD matrix multiplication instruction that uses mixed precision on inputs (8- and 4-bit) and accumulates into 16-bit output, improving throughput.
result Offers 2x improvement in throughput compared to existing symmetric-operand-size instructions, with negligible overflow.
New findings show depth is more important than width in neural networks.
problem Understanding the role of width and depth in neural networks.
method Constructed networks with bounded weights and width at most d+2, showing depth plays a more significant role.
result Depth is more important than width in the expressive power of neural networks.
Unified neural network for linear and nonlinear dimension reduction.
problem Efficiently perform linear and nonlinear sufficient dimension reduction.
method Belted and Ensembled Neural Network (BENN) framework.
result Unified framework for both linear and nonlinear dimension reduction.
Gradient descent in neural networks analyzed using RKBS for broader applicability.
problem Analyzing neural network training in the over-parametrized limit.
method Constructing an exact power-series representation of neural networks in RKBS, proving replicability of gradient descent sequences.
result Gradient descent sequences can be exactly replicated by regularized sequential learning in RKBS, providing new theoretical insights.
TDistNNs improve prediction intervals for neural networks by using t-distributions.
problem Traditional neural networks provide only point estimates, lacking predictive uncertainty.
method TDistNNs generate t-distributed outputs with adjustable degrees of freedom, enhancing robustness to non-Gaussian data.
result TDistNNs produce narrower prediction intervals with proper coverage compared to Gaussian-based PNNs.
Investigates how SGD behaves in high-dimensional neural networks, distinguishing between global convergence and local minima.
problem Understanding the behavior of SGD in high-dimensional shallow neural networks.
method Extends statistical physics analysis to study SGD dynamics, focusing on mean-field/hydrodynamic regime and learning rate.
result Identifies the critical number of hidden units and learning rate for SGD to avoid local minima.
Study challenges the Gaussian pre-activations assumption in neural networks.
problem Challenges the assumption that pre-activations are Gaussian in neural networks.
method Constructs pairs of activation functions and initialization distributions to ensure Gaussian pre-activations.
result Discovered constraints for ensuring Gaussian pre-activations in neural networks.
New framework analyzes deep learning optimization with finite width networks, revealing generalization gaps and excess risks.
problem Analyzing generalization error of deep learning with finite width networks.
method Formulating neural network training as transportation map estimation and analyzing via infinite dimensional Langevin dynamics.
result Achieves fast learning rate and minimax optimal rates for classification and regression problems.
Study shows DNNs can recover functions with fewer samples than model parameters at overparameterization.
problem Determining reliable function recovery in overparameterized deep neural networks.
method Introducing 'local linear recovery' (LLR) and proving upper bounds on sample sizes for recovery.
result Upper bounds on optimistic sample sizes for function recovery in overparameterized DNNs are achieved.
Piecewise linear activations create many spurious local minima in neural networks.
problem Understanding the loss surface of neural networks with piecewise linear activations.
method Proved the existence of infinite spurious local minima and partitioned the loss surface into smooth cells.
result Piecewise linear activations create many spurious local minima that are invariant under a continuous path.
Transferability of learned features between tasks can massively reduce the cost of training a neural network on a novel task. We investigate the effect of network width on learned features using activation atlases --- a visualization technique that captures features the entire hidden state responds to, as opposed to in…
Comparing different neural network representations and determining how representations evolve over time remain challenging open questions in our understanding of the function of neural networks. Comparing representations in neural networks is fundamentally difficult as the structure of representations varies greatly, e…
New findings on hidden symmetries in ReLU networks.
problem Understanding the redundancy and symmetries in ReLU network parameter space.
method Analyzing parameter settings and function classes for various network architectures.
result For certain network architectures, there are no hidden symmetries.
Empirical study compares wide neural networks to kernel methods, resolving open questions.
problem Understanding the relationship between wide neural networks and kernel methods.
method Large-scale empirical study using various neural network architectures and kernel methods.
result Wide neural networks outperform fully-connected finite-width networks in some cases, but underperform convolutional finite-width networks.
We present a novel tractable generative model that extends Sum-Product Networks (SPNs) and significantly boosts their power. We call it Sum-Product-Quotient Networks (SPQNs), whose core concept is to incorporate conditional distributions into the model by direct computation using quotient nodes, e.g. $P(A|B) = \frac{P(…
In this paper, we propose a simple but effective method for training neural networks with a limited amount of training data. Our approach inherits the idea of knowledge distillation that transfers knowledge from a deep or wide reference model to a shallow or narrow target model. The proposed method employs this idea to…
Deep neural network pruning and quantization techniques have demonstrated it is possible to achieve high levels of compression with surprisingly little degradation to test set accuracy. However, this measure of performance conceals significant differences in how different classes and images are impacted by model compre…
New method for PINNs uncertainty quantification without prior distribution.
problem Lack of reliable uncertainty quantification for PINNs.
method Extended fiducial inference with narrow-neck hyper-network.
result Construction of honest confidence sets based on observed data.