arXiv research

A new method for learning Bayesian neural networks using layerwise inference.

Use simplified layerwise linear models to understand neural dynamics.

This work proves the asymptotic freeness of layerwise Jacobians in MLPs with Haar orthogonal matrices.

Injectivity of ReLU networks is characterized for generative models and inverse problems.

In this paper, we study two aspects of the variational autoencoder (VAE): the prior distribution over the latent variables and its corresponding posterior. First, we decompose the learning of VAEs into layerwise density estimation, and argue that having a flexible prior is beneficial to both sample generation and infer…

Efficient algorithm for CMDPs reduces to offline density estimation.

New method $μP^2$ improves neural network training by scaling perturbations layerwise.

We improve neural network explainability by bypassing batch normalization.

Deep neural networks (DNNs) depend on the storage of a large number of parameters, which consumes an important portion of the energy used during inference. This paper considers the case where the energy usage of memory elements can be reduced at the cost of reduced reliability. A training algorithm is proposed to optim…

AlphaPruning optimizes LLM pruning using HT-SR theory for better performance.

Training large deep neural networks on massive datasets is computationally very challenging. There has been recent surge in interest in using large batch stochastic optimization methods to tackle this issue. The most prominent algorithm in this line of research is LARS, which by employing layerwise adaptive learning ra…

Proposes efficient training method for deep thin networks.

We give algorithms with provable guarantees that learn a class of deep nets in the generative model view popularized by Hinton and others. Our generative model is an $n$ node multilayer neural net that has degree at most $n^γ$ for some $γ<1$ and each edge has a random edge weight in $[-1,1]$. Our algorithm learns {\em …

Deep Gaussian Processes (DGP) are hierarchical generalizations of Gaussian Processes (GP) that have proven to work effectively on a multiple supervised regression tasks. They combine the well calibrated uncertainty estimates of GPs with the great flexibility of multilayer models. In DGPs, given the inputs, the outputs …

Estimating machine learning performance 'in the wild' is both an important and unsolved problem. In this paper, we seek to examine, understand, and predict the pointwise competence of classification models. Our contributions are twofold: First, we establish a statistically rigorous definition of competence that general…

In this note we present a generative model of natural images consisting of a deep hierarchy of layers of latent random variables, each of which follows a new type of distribution that we call rectified Gaussian. These rectified Gaussian units allow spike-and-slab type sparsity, while retaining the differentiability nec…

Deep learning is also known as hierarchical learning, where the learner _learns_ to represent a complicated target function by decomposing it into a sequence of simpler functions to reduce sample and time complexity. This paper formally analyzes how multi-layer neural networks can perform such hierarchical learning _ef…

Training avoids edge of stability by aligning Jacobian matrices.

SAEs struggle with curved activation manifolds, revealing layer-dependent scaling laws.

New optimizers control network width scaling, improving stability and transfer across different model sizes.

Proves deep networks can learn hierarchical structures efficiently.

Develops methods to measure and set function-space learning rates in neural networks.

New bounds adaptively control spectral complexity of trained Transformers.

A new robust prefix-tuning framework improves model robustness against adversarial attacks.

With the development of cloud computing and big data, the reliability of data storage systems becomes increasingly important. Previous researchers have shown that machine learning algorithms based on SMART attributes are effective methods to predict hard drive failures. In this paper, we use SMART attributes to predict…

MTL-NAS combines NAS with GP-MTL for task-agnostic multi-task learning.

SmartExchange trades memory for computation in neural networks.

Scalable method bounds Lipschitz constant of generative models.

A method to automatically choose feature dimensions in linear attention for better approximation quality.

A new algorithm for efficiently removing specific classes from a model without retraining.

A new measure predicts deep learning model performance.

We propose a distributed approach to train deep neural networks (DNNs), which has guaranteed convergence theoretically and great scalability empirically: close to 6 times faster on instance of ImageNet data set when run with 6 machines. The proposed scheme is close to optimally scalable in terms of number of machines, …

Large learning rates work surprisingly well in standard parameterization, contrary to theory.

Neural networks have become increasingly prevalent within the geosciences, although a common limitation of their usage has been a lack of methods to interpret what the networks learn and how they make decisions. As such, neural networks have often been used within the geosciences to most accurately identify a desired o…

New view: Deep GCNs learn to anti-oversmooth during training.

LASE learns graph embeddings by unrolling GD iterations into a neural network.

AlgebraNets use alternative algebras for neural networks, improving performance on image and language tasks.

The paper studies how noise synchronizes tokens in deep transformer models.

Adversarial training has been shown to regularize deep neural networks in addition to increasing their robustness to adversarial examples. However, its impact on very deep state of the art networks has not been fully investigated. In this paper, we present an efficient approach to perform adversarial training by pertur…

Multitask learning has shown promising performance in many applications and many multitask models have been proposed. In order to identify an effective multitask model for a given multitask problem, we propose a learning framework called learning to multitask (L2MT). To achieve the goal, L2MT exploits historical multit…

Low complexity decentralized neural net with centralized performance.

Gradient-based meta-learning methods leverage gradient descent to learn the commonalities among various tasks. While previous such methods have been successful in meta-learning tasks, they resort to simple gradient descent during meta-testing. Our primary contribution is the {\em MT-net}, which enables the meta-learner…

While the backpropagation of error algorithm enables deep neural network training, it implies (i) bidirectional synaptic weight transport and (ii) update locking until the forward and backward passes are completed. Not only do these constraints preclude biological plausibility, but they also hinder the development of l…

TSSM splits neural networks for parallel training with minimal accuracy loss.

New learning rules for wide neural networks without backpropagation.

While the authors of Batch Normalization (BN) identify and address an important problem involved in training deep networks-- Internal Covariate Shift-- the current solution has certain drawbacks. Specifically, BN depends on batch statistics for layerwise input normalization during training which makes the estimates of …

While the authors of Batch Normalization (BN) identify and address an important problem involved in training deep networks-- \textit{Internal Covariate Shift}-- the current solution has certain drawbacks. For instance, BN depends on batch statistics for layerwise input normalization during training which makes the esti…

New theory maps neural network weights to optimize faster and scale.