One-layer transformers can't solve induction heads task efficiently.
problem Solving the induction heads task efficiently with one-layer transformers.
method Communication complexity argument showing exponential size requirement.
result No one-layer transformer can solve the induction heads task efficiently.
Using detailed statistical analyses of the size distribution of a universe of equity exchange-traded funds (ETFs), we discover a discrete hierarchy of sizes, which imprints a log-periodic structure on the probability distribution of ETF sizes that dominates the details of the asymptotic tail. This allows us to propose …
Proposes continuous convolution layers for flexible feature map resizing.
problem Fixed stride limitations in discrete convolution layers.
method Introduces Continuous Convolution (CC) layers that use learned continuous functions.
result Dynamic and consistent resizing of feature maps at any scale, non-integer and axis-dependent.
This paper compares Transformers and RNNs in various tasks, showing size differences.
problem Comparing representational capabilities of Transformers and RNNs across tasks.
method Analysis of differences in tasks like index lookup, nearest neighbor, and string equality.
result Size differences in Transformers and RNNs for various tasks.
Improved recommendation systems using multi-layer embeddings reduce model size while maintaining accuracy.
problem Improving model accuracy in recommendation systems while minimizing model size.
method Introducing a multi-layer embedding training (MLET) architecture that trains embeddings via a sequence of linear layers.
result Substantial advantages in model accuracy and memory footprint are achieved with reduced embedding dimensions.
This work proposes a collaborative multi-head attention layer to reduce model size without sacrificing accuracy.
problem Over-parameterization in transformer models trained with large datasets.
method Proposes a collaborative multi-head attention layer that shares key/query projections.
result Reduction in model size by 4 for same accuracy and speed.
BSF algorithm reduces neural network size and selects features efficiently.
problem Neural network size and feature selection optimization.
method Binary Stochastic Filtering (BSF) layer that penalizes information, stochastically passes or drops features.
result Multifold decrease in neural network size and optimal feature selection.
FRN layer eliminates batch dependence in deep learning, improving performance across various tasks.
problem Batch Normalization's dependency on mini-batch elements can degrade performance for small batches.
method Filter Response Normalization (FRN) operates independently on each activation channel of each batch element.
result FRN layer outperforms BN and other alternatives in various settings for all batch sizes.
A method predicts GNS of transformer layers using normalization layer norms.
problem Estimating gradient noise scale with minimal variance.
method Simultaneously compute per-example gradient norms and parameter gradients.
result Total GNS is predicted well by normalization layer GNS.
Deep neural networks can handle high dimensions without losing accuracy.
problem The curse of dimensionality in nonparametric regression.
method Least squares estimates on fully connected neural networks with ReLU activation functions.
result Similar convergence results to deep learning models with sparsity constraints can be achieved without such constraints.
We study the implicit regularization imposed by gradient descent for learning multi-layer homogeneous functions including feed-forward fully connected and convolutional deep neural networks with linear, ReLU or Leaky ReLU activation. We rigorously prove that gradient flow (i.e. gradient descent with infinitesimal step …
Transformers show strengths and weaknesses in complexity analysis.
problem Understanding the strengths and limitations of attention layers in transformers.
method Analysis of representation power through complexity parameters and task-specific constructions.
result Transformers can solve sparse averaging tasks with logarithmic complexity, but triple detection tasks require linear complexity.
SSFN self-estimates network size with low complexity and consistent performance.
problem Designing a self-estimating feed-forward network with low complexity and consistent performance.
method Joint optimization for layer and node estimation, low computational complexity, and use of lossless flow property and convex optimization.
result Consistent performance across Monte-Carlo trials and monotonically non-increasing cost with network growth.
New approach replaces BN layers with shifted-ReLU for embedded systems.
problem Complexity and overheads of BN layers in low-power embedded systems.
method Used shifted-ReLU layers instead of BN layers in wide residual networks.
result Shifted-ReLU layers offer advantages in speed, memory, and complexity without significant accuracy loss.
Optimal scaling found to depend on operator norm across large models and datasets.
problem Lack of unifying principle for optimal hyperparameter scaling across models and datasets.
method Discovered that optimal scaling is conditioned on the operator norm of the output layer.
result The optimal learning rate/batch size pair (η∗,B∗) consistently has the same operator norm value. This paper analyzes training and generalization of simple 2-layer neural nets.
problem Why overparameterized neural nets generalize well despite fitting any data.
method Tighter characterization of training speed, data-dependent complexity measure, and tracking training and generalization dynamics via a related kernel.
result Generalization bound independent of network size, with completely independent sample complexity.
Bounds neural network output distribution to Gaussian for random initialization.
problem Quantifying the distribution of randomly initialized deep neural networks.
method Quantitative Gaussian approximation using quadratic Wasserstein distance.
result Explicit inequalities show how network sizes affect Gaussian behavior.
WavPool improves deep neural networks with wavelet-based pooling.
problem Improving efficiency and performance of deep neural networks.
method Introducing WavPool, a wavelet-transform-based pooling layer.
result WavPool outperforms existing network architectures by 10% on CIFAR-10.
Study shows infoGAN's generalization error bound for two-layer networks.
problem Understanding generalization error in infoGAN for two-layer neural networks.
method Analyzes the difference between empirical and population objective functions, derives Rademacher complexity bounds.
result Derives error bound for infoGAN's generalization error in a two-layer network.
The study explores the compressive power of Boolean threshold autoencoders, finding that seven layers are necessary but three are not.
problem Understanding the compressive limits of Boolean threshold autoencoders.
method Investigation into the minimum number of layers and nodes required for autoencoders to accurately transform binary vectors.
result There exists a seven-layer autoencoder with a logarithmic middle layer size for any set of distinct vectors, but not a three-layer one.
New estimates for the population risk are established for two-layer neural networks. These estimates are nearly optimal in the sense that the error rates scale in the same way as the Monte Carlo error rates. They are equally effective in the over-parametrized regime when the network size is much larger than the size of…
Global pooling, such as max- or sum-pooling, is one of the key ingredients in deep neural networks used for processing images, texts, graphs and other types of structured data. Based on the recent DeepSets architecture proposed by Zaheer et al. (NIPS 2017), we introduce a Set Aggregation Network (SAN) as an alternative…
Novel neural network layer improves long-range interactions in point clouds.
problem Efficiently treating long-range interactions in point clouds.
method Long-range convolutional (LRC)-layer that incorporates Fourier transforms.
result Global all-to-all convolution operation performed in nearly-linear time.
This work explores representation complexity in RL paradigms, revealing model-based RL as the easiest task.
problem Investigating the representation complexity gap among model-based, policy-based, and value-based RL.
method Demonstrated through analysis of Markov decision processes (MDPs) and introduced new classes of MDPs.
result Representation complexity hierarchy: model-based RL > policy-based RL > value-based RL.
It has been experimentally observed in recent years that multi-layer artificial neural networks have a surprising ability to generalize, even when trained with far more parameters than observations. Is there a theoretical basis for this? The best available bounds on their metric entropy and associated complexity measur…
DNArch learns CNN architectures by backpropagation.
problem Discovering optimal CNN architectures.
method Differentiable Neural Architectures (DNArch) learns CNN architectures by backpropagation, controlling kernel sizes, channels, downsampling positions, and depth.
result DNArch finds performant CNN architectures across various tasks.
A method to analyze neural network performance by measuring layer saturation.
problem Understanding which layers contribute to network performance.
method Layer saturation method: restricts layer output to eigenspace of variance matrix.
result Layer saturation indicates which layers contribute to network performance.
Gradient descent optimizes neural networks and random features similarly, achieving zero loss fast.
problem Optimizing two-layer neural networks and random feature models under gradient descent.
method Comprehensive analysis of gradient descent dynamics, considering various network widths and data sizes.
result Gradient descent achieves zero training loss exponentially fast in the over-parametrized regime.
Bayesian method learns neural network architecture parameters.
problem Estimating optimal neural network architecture parameters.
method Bayesian learning of concrete distributions over layer size and network depth.
result Regular networks with learnt structure generalize better on small datasets, while stochastic networks are more robust to initialisation.
We analyze deep neural networks in the large size and iteration limit, revealing a deterministic system of equations.
problem Understanding the behavior of deep neural networks in the asymptotic regime of large network sizes and iterations.
method Sequential limit of each hidden layer and characterization of parameter evolution, using weak convergence and stochastic analysis.
result The limit neural network recovers a global minimum with zero loss for the objective function.
Study shows DNNs often extract redundant features, influenced by network size and activation function.
problem Redundancy in deep neural network features.
method Hierarchical clustering of features based on cosine distances, varying network sizes and activation functions.
result Network size and activation function are key factors in DNN redundancy.
A new layer, funnel, reduces dimensionality in flows for better performance.
problem Training high-dimensional models efficiently and accurately.
method Constructing dimension-reducing surjective flows using the funnel layer.
result The funnel layer improves model performance with a smaller latent space.
New training method for ReLU networks achieves optimal weight size for memorization.
problem Approximate memorization of arbitrary real labels with neural networks.
method Complex recombination training procedure for ReLU networks.
result Approximate memorization with nearly optimal weight size and neuron count.
This paper proposes a new Quantum Spatial Graph Convolutional Neural Network (QSGCNN) model that can directly learn a classification function for graphs of arbitrary sizes. Unlike state-of-the-art Graph Convolutional Neural Network (GCNN) models, the proposed QSGCNN model incorporates the process of identifying transit…
GP-DRF model handles variable-sized input and learns deep features.
problem Scaling deep Gaussian processes for variable-sized data.
method GP-DRF model combining GPs and DRF layers for efficient inference.
result GP-DRF outperforms standard GP and DRF models across various datasets.
Deep learning methods have predominantly been applied to large artificial neural networks. Despite their state-of-the-art performance, these large networks typically do not generalize well to datasets with limited sample sizes. In this paper, we take a different approach by learning multiple layers of kernels. We combi…
HaarPooling compresses graphs by Haar transforms, improving graph classification and regression.
problem Handling graphs of varying size and structure in GNNs.
method HaarPooling, a cascade of clusterings and compressive Haar transforms.
result HaarPooling synthesizes graph features into uniform size, achieving state-of-the-art performance.
Study on double descent behavior in two-layer neural networks for binary classification.
problem Understanding the double descent phenomenon in model test error.
method Two-layer neural network with ReLU activation for binary classification. Quantified model size by sample-to-dimension ratio. Empirical risk minimization using Convex Gaussian Min Max Theorem.
result Observed and investigated the double descent behavior of model test error.
Estimates and compares the capabilities of neural network architectures.
problem Developing quantitative methods to estimate and compare neural network architectures.
method Defining capacity as the logarithm of the number of functions an architecture can compute, and proving a cubic polynomial formula for layered architectures.
result The capacity of a neural network is given by a cubic polynomial in the layer sizes, with bottlenecks affecting the capacity.
A Bloom filter approach combined with Transformer models improves accuracy for machine learning tasks on opaque IDs.
problem Improving accuracy for machine learning tasks on opaque IDs with large vocabulary sizes.
method Applying hash functions to map opaque IDs to multiple hash tokens, similar to a Bloom filter, and using a multi-layer Transformer to process these digests.
result Models outperform those without hashing and sampled softmax, achieving high accuracy with a smaller computational budget.
We study cascades on a two-layer multiplex network, with asymmetric feedback that depends on the coupling strength between the layers. Based on an analytical branching process approximation, we calculate the systemic risk measured by the final fraction of failed nodes on a reference layer. The results are compared with…
Global convergence of SGD proven for two-layer neural nets with regularization.
problem Proving global convergence of SGD for two-layer neural nets.
method Regularized empirical risk, SGD iterates, Villani functions.
result Global convergence of SGD for a special class of initializations.
We give a covering number bound for deep learning networks that is independent of the size of the network. The key for the simple analysis is that for linear classifiers, rotating the data doesn't affect the covering number. Thus, we can ignore the rotation part of each layer's linear transformation, and get the coveri…
Extends capacity analysis to neural networks, showing how capacity is distributed across layers.
problem How capacity is distributed in neural networks with non-linear layers.
method Introduces layer decoupling to quantify non-linear activation's impact, and uses a markovian rule for capacity propagation in deep networks.
result Shows that under certain conditions, capacity allocation in neural networks is equivalent to linear capacity allocation in an extended input space.
In this paper we investigate the family of functions representable by deep neural networks (DNN) with rectified linear units (ReLU). We give an algorithm to train a ReLU DNN with one hidden layer to *global optimality* with runtime polynomial in the data size albeit exponential in the input dimension. Further, we impro…
We introduce a simple permutation equivariant layer for deep learning with set structure.This type of layer, obtained by parameter-sharing, has a simple implementation and linear-time complexity in the size of each set. We use deep permutation-invariant networks to perform point-could classification and MNIST-digit sum…
Hardness proven for learning neural networks with polynomial size and Gaussian inputs.
problem Learning one hidden layer ReLU neural networks with polynomial size and Gaussian inputs.
method Based on the hardness of the Continuous Learning with Errors (CLWE) problem.
result Hardness of learning neural networks is proven under standard cryptographic assumptions.
Whereas CNNs have demonstrated immense progress in many vision problems, they suffer from a dependence on monumental amounts of labeled training data. On the other hand, dictionary learning does not scale to the size of problems that CNNs can handle, despite being very effective at low-level vision tasks such as denois…