Study on size and depth of neural networks for approximating benign functions, showing barriers and explicit results.
problem Understanding how size and depth of neural networks affect their ability to approximate benign functions.
method Analyzing ReLU networks for benign functions, proving barriers and explicit results.
result Explicit benign functions that cannot be approximated by networks of certain sizes or depths, showing barriers to size and depth separation.
SSFN self-estimates network size with low complexity and consistent performance.
problem Designing a self-estimating feed-forward network with low complexity and consistent performance.
method Joint optimization for layer and node estimation, low computational complexity, and use of lossless flow property and convex optimization.
result Consistent performance across Monte-Carlo trials and monotonically non-increasing cost with network growth.
Neural network learns its size and structure during training.
problem Adapting neural network architecture to specific datasets.
method Flexible setup allowing neural network to learn size and topology during training.
result Trained networks achieve virtually identical performance and have learned optimal structure.
Decomposable-Net compresses neural networks without retraining for various sizes.
problem Performance degradation when changing model size after training.
method Decomposes weight matrices via SVD and adjusts ranks for different sizes.
result Maintains and improves performance across multiple model sizes.
How big is the risk that a few initial failures of nodes in a network amplify to large cascades that span a substantial share of all nodes? Predicting the final cascade size is critical to ensure the functioning of a system as a whole. Yet, this task is hampered by uncertain or changing parameters and missing informati…
Training deep neural networks with Stochastic Gradient Descent, or its variants, requires careful choice of both learning rate and batch size. While smaller batch sizes generally converge in fewer training epochs, larger batch sizes offer more parallelism and hence better computational efficiency. We have developed a n…
New neural network models learn symmetric functions of varying input sizes.
problem Learning symmetric functions with varying input sizes.
method Functional perspective on neural networks, treating symmetric functions as functions over probability measures.
result Established approximation and generalization bounds for shallow architectures that extend across input sizes.
Study shows how algorithmic choices affect optimal batch sizes in neural networks.
problem Understanding how batch size impacts neural network training efficiency.
method Experiments and analysis of a simple quadratic model to study algorithmic choices.
result Preconditioned optimizers like Adam and K-FAC allow larger batch sizes before diminishing returns.
Deep neural networks can approximate complex functions through repeated compositions of a fixed-size ReLU network.
problem Understanding the expressive power of deep neural networks through function compositions.
method Demonstrated the surprising expressive power of repeated compositions of a single fixed-size ReLU network.
result Repeated compositions of a single fixed-size ReLU network can approximate 1-Lipschitz continuous functions on [0,1]d with an error O(r−1/d). The study reveals a transition in neural network performance from infinite-width to variance-limited behavior as dataset size increases.
problem Understanding the transition from infinite-width to variance-limited behavior in neural networks.
method Empirical study of the transition from infinite-width to variance-limited behavior as a function of sample size and network width.
result The critical sample size \( P^* \) is approximately \( \sqrt{N} \) for polynomial regression with ReLU networks.
Gradient descent finds global minima in deep neural networks of practical sizes.
problem Finding global minima in large, non-convex neural network optimization problems.
method Gradient descent applied to non-convex optimization of deep neural networks with practical degrees of over-parameterization.
result Gradient descent can find global minima in deep neural networks of sizes commonly encountered in practice.
This paper optimizes step sizes for faster convergence in sparse coding.
problem Improving convergence rate of ISTA for sparse coding.
method Learning step sizes for ISTA using neural networks.
result A simple step size strategy can improve convergence rate of ISTA.
Study shows no degradation in neural network performance with larger batch sizes.
problem Characterizing the effects of increasing batch size on neural network training time.
method Experimentally measured training time for various neural network models and datasets.
result No evidence of degradation in out-of-sample performance with larger batch sizes.
As neural networks become widely deployed in different applications and on different hardware, it has become increasingly important to optimize inference time and model size along with model accuracy. Most current techniques optimize model size, model accuracy and inference time in different stages, resulting in subopt…
Two methods reduce BN and DNN complexity, balancing size and accuracy.
problem Balancing model size and prediction accuracy in Bayesian networks and deep neural networks.
method Quantization-aware training and tree-augmented naive Bayes structure learning extension.
result Pareto optimal models found for small-scale scenarios.
Monotone neural networks can approximate and interpolate functions efficiently.
problem Understanding the efficiency and expressiveness of monotone neural networks.
method Solving the monotone interpolation problem using depth-4 networks and comparing size bounds with arbitrary networks.
result Monotone neural networks can approximate and interpolate functions efficiently, but may require exponential size in high dimensions.
Gradient descent converges to global minima for ResNets with linearly scaled width.
problem Understanding the convergence of deep residual networks with varying network width and dataset size.
method Analyzing the Jacobian of ResNets and applying gradient descent for quadratic loss.
result Gradient descent converges to global minima for ResNets with linearly scaled width and independent of depth.
Unified scaling laws reveal how model size and training time impact neural network performance.
problem Understanding how much performance improvement can be expected from scaling model size or data volume.
method Established scale-time equivalence and combined it with a linear model analysis of double descent.
result Unified theoretical scaling laws explain previously unexplained phenomena and offer a more accessible path to training large models.
Auto-sizing improves machine translation efficiency by 3.9 BLEU points with fewer parameters.
problem Optimizing neural network architecture and hyperparameters for low-resource machine translation is computationally expensive.
method Auto-sizing uses regularization to dynamically adjust network size during training.
result Auto-sizing improves BLEU scores by up to 3.9 points with one-third fewer parameters.
This paper analyzes the generalization risk of unrolled neural networks using Stein's Unbiased Risk Estimator.
problem Analyzing the generalization risk of unrolled neural networks and its relationship to network design and train sample size.
method Using Stein's Unbiased Risk Estimator (SURE), the paper analyzes the generalization risk with bias and variance components for recurrent unrolled networks, focusing on the degrees-of-freedom (DOF) component and the trace of the end-to-end network Jacobian.
result DOF is well-approximated by the weighted path sparsity of the network under incoherence conditions on the trained weights, and DOF increases with train sample size and converges to the generalization risk for both recurrent and non-recurrent schemes.
New training method for ReLU networks achieves optimal weight size for memorization.
problem Approximate memorization of arbitrary real labels with neural networks.
method Complex recombination training procedure for ReLU networks.
result Approximate memorization with nearly optimal weight size and neuron count.
New activation functions achieve arbitrary-accuracy Sobolev approximation by fixed-size neural networks.
problem Approximation of Sobolev functions by neural networks
method Elementary Universal Activation Function and Differentiable Universal Activation Functions
result Arbitrary-accuracy Sobolev approximation by fixed-size neural networks
Large SGD step sizes lead to sparse feature learning in neural networks.
problem Sparse feature learning in neural networks with large step sizes.
method Empirical observations and theoretical analysis of SGD dynamics.
result Large step sizes induce implicit regularization leading to sparse predictors.
Paper improves graph convolutional networks by adjusting filter size.
problem Improving predictive performance of graph convolutional networks.
method Introducing a hyper-parameter to influence filter size in graph convolutions.
result Improves predictive performance of Deep Graph Convolutional Networks.
NodeDrop reduces network size by eliminating unnecessary features.
problem Reducing network size for systems with limited memory or processing power.
method Identifies and eliminates nodes carrying no information using regularization.
result Reduces the number of features by a factor of 114x for a VGG-like network on CIFAR10 without a drop in accuracy.
Alignment of neural network representations is influenced by SNR and sample size.
problem Understanding how neural network representations align across different conditions.
method Controlled training of neural networks on perturbed datasets, analyzing alignment and generalization.
result Alignment varies monotonically with SNR but non-monotonically with sample size, with minimal alignment near the interpolation threshold.
This work investigates power laws in deep neural network ensembles and predicts their performance.
problem Understanding the performance of deep neural network ensembles and their optimal structure.
method Investigated the behavior of negative log-likelihood (CNLL) of a deep ensemble as a function of ensemble size and member network size, identifying power law dependencies.
result One large network may perform worse than an ensemble of several medium-size networks, known as a memory split.
The paper investigates why GNNs struggle to generalize from small to large graphs.
problem Challenges in graph neural networks' ability to generalize across different graph sizes.
method Identified and studied the effect of local structure on size generalization; proposed a novel SSL task.
result GNNs can converge to non-generalizing solutions when there is a discrepancy in local structure.
New study reveals how heavy-tailed SGD dynamics lead to compressible neural networks.
problem Understanding why large neural networks can be compressed effectively.
method Linking SGD dynamics to compressibility properties of neural networks.
result Large step-size/batch-size ratios and overparametrization lead to heavy-tailed SGD dynamics, making networks compressible.
A new pruning method reduces DNN size and interconnectivity using brain network principles.
problem Over-parameterization in deep neural networks causes memory and hardware cost issues.
method Structural pruning scheme based on Small-World model, trimming network before training.
result Reduced model size by 2.3% on LeNet-5 for MNIST and 9.02% on VGG-16 for CIFAR-10.
This paper shows neural networks can solve complex graph problems efficiently.
problem Solving exact maximum flow computation and minimum spanning tree problems.
method Introduces Max-Affine Arithmetic Programs and shows equivalence to neural networks.
result Two combinatorial optimization problems can be solved with polynomial-size neural networks.
Study shows DNNs often extract redundant features, influenced by network size and activation function.
problem Redundancy in deep neural network features.
method Hierarchical clustering of features based on cosine distances, varying network sizes and activation functions.
result Network size and activation function are key factors in DNN redundancy.
Negative step sizes improve second-order methods for neural networks.
problem Second-order methods discard negative curvature, limiting their effectiveness.
method Introduce negative step sizes in second-order methods combined with Wolfe line search.
result Negative step sizes lead to global convergence and improved performance.
Standard optimizers perform as well as LARS and LAMB at large batch sizes.
problem Comparing optimizers for neural network training at large batch sizes.
method Used standard optimizers like Nesterov momentum and Adam to match or exceed LARS and LAMB results.
result Standard optimizers can match or exceed LARS and LAMB at large batch sizes.
New method reduces training time for deep hedging networks.
problem Challenges in training deep hedging networks with large batch sizes.
method Integrates topological features to reduce batch sizes.
result Practical training of deep hedging models without sacrificing performance.
New neural network reduces CT radiation, works for any ROI size.
problem CT ROI reconstruction suffers from cupping artifacts and high computation.
method Two neural networks: one learns ROI-specific artifacts, the other learns DBP inversion.
result New network outperforms existing methods for any ROI size.
Study reveals optimal scaling conditions for photonic neural networks.
problem Impact of reservoir size and learning routines on convergence-speed during learning.
method Used a greedy algorithm to train a photonic neural network for chaotic signals prediction.
result Determined convergence speed of learning as a function of reservoir size and found close to linear scaling.
This paper examines the convergence of adaptive sampling methods for Bayesian neural networks.
problem Uncertainty quantification in deep neural networks, especially for medical applications.
method Locally adaptive and scalable diffusion-based sampling methods.
result These methods can have a substantial bias in the distribution they sample, even in the limit of vanishing step sizes.
GD networks improve lung nodule classification accuracy.
problem Difficult classification of lung nodules of varying sizes.
method Proposes Gated-Dilated (GD) networks with Context-Aware sub-network.
result GD network outperforms state-of-the-art models with AUC > 0.95.
We give a covering number bound for deep learning networks that is independent of the size of the network. The key for the simple analysis is that for linear classifiers, rotating the data doesn't affect the covering number. Thus, we can ignore the rotation part of each layer's linear transformation, and get the coveri…
Faster convergence and handling larger mini-batches for deep neural networks.
problem Generalization gap in large-scale distributed training of deep neural networks.
method Second-order optimization using Kronecker-factored approximate curvature.
result Achieved 75% Top-1 validation accuracy with mini-batch size of 131,072 in 978 iterations.
DAL improves active learning for neural networks with large batch sizes.
problem Efficiently choosing examples to label for neural networks with large batch sizes.
method DAL treats active learning as a binary classification task to make labeled and unlabeled sets indistinguishable.
result DAL performs on par with state-of-the-art methods in medium and large query batch sizes.
Proposes FCNs for text classification with size-invariant inputs.
problem Classifying text of varying sizes.
method Uses fully convolutional networks with modifications to attention mechanisms.
result Suboptimal results on ITAmoji task, proposes fixes.
Study compares RL and SL for TSP, finds RL better for variable graph sizes.
problem Training deep neural networks for the Travelling Salesman Problem.
method Controlled experiments with supervised and reinforcement learning models on fixed and variable sized graphs.
result Reinforcement learning leads to better generalization to variable graph sizes.
Paper reduces recommender system model size by 90%.
problem Large model size in recommender systems.
method Hybrid frequency and double hashing for model size reduction.
result 90% reduction in model size with no performance loss.
A modified VDCNN model reduces size and latency for mobile platforms.
problem Memory and processing constraints on mobile platforms.
method Temporal Depthwise Separable Convolutions and Global Average Pooling.
result The squeezed model (SVDCNN) is 10x-20x smaller with minimal accuracy loss.
New method finds compact strong lottery tickets in partially frozen networks.
problem Finding compact strong lottery tickets in randomly pruned networks.
method Freezing a random subset of initial weights in a partially frozen network.
result Better accuracy and compression achieved with frozen networks.
RBMs learn archetypes when trained on blurred copies of them, revealing a critical sample size.
problem Determining the critical sample size for RBMs to learn archetypes.
method Formal equivalence between RBMs and Hopfield networks, statistical-mechanics of disordered systems, Monte Carlo simulations.
result A phase diagram highlights regions where learning can be accomplished.