arXiv research

Efficient algorithm for CMDPs reduces to offline density estimation.

Use simplified layerwise linear models to understand neural dynamics.

A new method for learning Bayesian neural networks using layerwise inference.

This work proves the asymptotic freeness of layerwise Jacobians in MLPs with Haar orthogonal matrices.

Proves deep networks can learn hierarchical structures efficiently.

New method $μP^2$ improves neural network training by scaling perturbations layerwise.

SAEs struggle with curved activation manifolds, revealing layer-dependent scaling laws.

AlphaPruning optimizes LLM pruning using HT-SR theory for better performance.

The lack of transparency of neural networks stays a major break for their use. The Layerwise Relevance Propagation technique builds heat-maps representing the relevance of each input in the model s decision. The relevance spreads backward from the last to the first layer of the Deep Neural Network. Layer-wise Relevance…

New optimizers control network width scaling, improving stability and transfer across different model sizes.

Training large deep neural networks on massive datasets is computationally very challenging. There has been recent surge in interest in using large batch stochastic optimization methods to tackle this issue. The most prominent algorithm in this line of research is LARS, which by employing layerwise adaptive learning ra…

Proposes efficient training method for deep thin networks.

Deep Gaussian Processes (DGP) are hierarchical generalizations of Gaussian Processes (GP) that have proven to work effectively on a multiple supervised regression tasks. They combine the well calibrated uncertainty estimates of GPs with the great flexibility of multilayer models. In DGPs, given the inputs, the outputs …

New bounds adaptively control spectral complexity of trained Transformers.

Estimating machine learning performance 'in the wild' is both an important and unsolved problem. In this paper, we seek to examine, understand, and predict the pointwise competence of classification models. Our contributions are twofold: First, we establish a statistically rigorous definition of competence that general…

We give algorithms with provable guarantees that learn a class of deep nets in the generative model view popularized by Hinton and others. Our generative model is an $n$ node multilayer neural net that has degree at most $n^γ$ for some $γ<1$ and each edge has a random edge weight in $[-1,1]$. Our algorithm learns {\em …

In this note we present a generative model of natural images consisting of a deep hierarchy of layers of latent random variables, each of which follows a new type of distribution that we call rectified Gaussian. These rectified Gaussian units allow spike-and-slab type sparsity, while retaining the differentiability nec…

Deep learning is also known as hierarchical learning, where the learner _learns_ to represent a complicated target function by decomposing it into a sequence of simpler functions to reduce sample and time complexity. This paper formally analyzes how multi-layer neural networks can perform such hierarchical learning _ef…

Training avoids edge of stability by aligning Jacobian matrices.

Injectivity of ReLU networks is characterized for generative models and inverse problems.

Develops methods to measure and set function-space learning rates in neural networks.

A new robust prefix-tuning framework improves model robustness against adversarial attacks.

Continuous-depth Evoformer reduces protein folding prediction time and resource usage.

The staircase property aids deep learning by guiding hierarchical feature learning.

With the development of cloud computing and big data, the reliability of data storage systems becomes increasingly important. Previous researchers have shown that machine learning algorithms based on SMART attributes are effective methods to predict hard drive failures. In this paper, we use SMART attributes to predict…

MTL-NAS combines NAS with GP-MTL for task-agnostic multi-task learning.

Scalable method bounds Lipschitz constant of generative models.

In this paper, we study two aspects of the variational autoencoder (VAE): the prior distribution over the latent variables and its corresponding posterior. First, we decompose the learning of VAEs into layerwise density estimation, and argue that having a flexible prior is beneficial to both sample generation and infer…

Deep Optimisation (DO) combines evolutionary search with Deep Neural Networks (DNNs) in a novel way - not for optimising a learning algorithm, but for finding a solution to an optimisation problem. Deep learning has been successfully applied to classification, regression, decision and generative tasks and in this paper…

A new measure predicts deep learning model performance.

Deep neural networks (DNNs) depend on the storage of a large number of parameters, which consumes an important portion of the energy used during inference. This paper considers the case where the energy usage of memory elements can be reduced at the cost of reduced reliability. A training algorithm is proposed to optim…

We propose a distributed approach to train deep neural networks (DNNs), which has guaranteed convergence theoretically and great scalability empirically: close to 6 times faster on instance of ImageNet data set when run with 6 machines. The proposed scheme is close to optimally scalable in terms of number of machines, …

Large learning rates work surprisingly well in standard parameterization, contrary to theory.

New view: Deep GCNs learn to anti-oversmooth during training.

The paper studies how noise synchronizes tokens in deep transformer models.

Adversarial training has been shown to regularize deep neural networks in addition to increasing their robustness to adversarial examples. However, its impact on very deep state of the art networks has not been fully investigated. In this paper, we present an efficient approach to perform adversarial training by pertur…

Multitask learning has shown promising performance in many applications and many multitask models have been proposed. In order to identify an effective multitask model for a given multitask problem, we propose a learning framework called learning to multitask (L2MT). To achieve the goal, L2MT exploits historical multit…

Low complexity decentralized neural net with centralized performance.

Gradient-based meta-learning methods leverage gradient descent to learn the commonalities among various tasks. While previous such methods have been successful in meta-learning tasks, they resort to simple gradient descent during meta-testing. Our primary contribution is the {\em MT-net}, which enables the meta-learner…

While the backpropagation of error algorithm enables deep neural network training, it implies (i) bidirectional synaptic weight transport and (ii) update locking until the forward and backward passes are completed. Not only do these constraints preclude biological plausibility, but they also hinder the development of l…

Bayesian KANs achieve near-minimax posterior contraction rates in anisotropic Besov spaces.

TSSM splits neural networks for parallel training with minimal accuracy loss.

Three-layer neural networks learn hierarchical polynomial functions efficiently.

New learning rules for wide neural networks without backpropagation.

SmartExchange trades memory for computation in neural networks.

While the authors of Batch Normalization (BN) identify and address an important problem involved in training deep networks-- Internal Covariate Shift-- the current solution has certain drawbacks. Specifically, BN depends on batch statistics for layerwise input normalization during training which makes the estimates of …

While the authors of Batch Normalization (BN) identify and address an important problem involved in training deep networks-- \textit{Internal Covariate Shift}-- the current solution has certain drawbacks. For instance, BN depends on batch statistics for layerwise input normalization during training which makes the esti…

New theory maps neural network weights to optimize faster and scale.