A new method for learning Bayesian neural networks using layerwise inference.
problem Learning Bayesian neural networks efficiently and accurately.
method Bayesian layerwise inference, treating neural networks as stacked Bayesian linear models, with pseudo-targets defined by backpropagated gradients.
result The method converges quickly and performs well on various benchmarks.
Use simplified layerwise linear models to understand neural dynamics.
problem Complex neural network dynamics are hard to grasp.
method Apply simplified layerwise linear models to explain neural phenomena.
result Simplified models explain neural collapse, emergence, etc.
This work proves the asymptotic freeness of layerwise Jacobians in MLPs with Haar orthogonal matrices.
problem Proving the asymptotic freeness of layerwise Jacobians in multilayer perceptrons (MLPs).
method Replacing each layer's parameter matrix with itself multiplied by a Haar orthogonal matrix, and using the invariance of the MLP.
result Proves the asymptotic freeness of layerwise Jacobians in MLPs with Haar orthogonal matrices.
Injectivity of ReLU networks is characterized for generative models and inverse problems.
problem Injectivity in ReLU networks for generative models and inverse problems.
method Layerwise analysis, worst-case Lipschitz constants, differential topology, random projections.
result Global injectivity of ReLU networks requires expansivity between 3.4 and 10.5 for Gaussian matrices.
In this paper, we study two aspects of the variational autoencoder (VAE): the prior distribution over the latent variables and its corresponding posterior. First, we decompose the learning of VAEs into layerwise density estimation, and argue that having a flexible prior is beneficial to both sample generation and infer…
Efficient algorithm for CMDPs reduces to offline density estimation.
problem Offline learning for CMDPs with horizon H.
method Reduction to offline density estimation, layerwise exploration-exploitation tradeoff.
result First efficient and near-optimal reduction from CMDPs to offline density estimation.
New method μP2 improves neural network training by scaling perturbations layerwise.
problem Improving neural network performance as models scale up.
method Layerwise perturbation scaling in the infinite-width limit of neural networks.
result Layerwise perturbation scaling ensures all layers are effectively perturbed in the limit.
We improve neural network explainability by bypassing batch normalization.
problem Lack of transparency in neural networks.
method Layer-wise Relevance Propagation with a method to include normalization layers.
result Heatmaps are more accurate for convolutional layers with our method.
Deep neural networks (DNNs) depend on the storage of a large number of parameters, which consumes an important portion of the energy used during inference. This paper considers the case where the energy usage of memory elements can be reduced at the cost of reduced reliability. A training algorithm is proposed to optim…
AlphaPruning optimizes LLM pruning using HT-SR theory for better performance.
problem Improving pruning of large language models to reduce size without sacrificing performance.
method AlphaPruning uses HT-SR theory to allocate layerwise sparsity ratios more theoretically.
result AlphaPruning prunes LLaMA-7B to 80% sparsity with reasonable perplexity.
Training large deep neural networks on massive datasets is computationally very challenging. There has been recent surge in interest in using large batch stochastic optimization methods to tackle this issue. The most prominent algorithm in this line of research is LARS, which by employing layerwise adaptive learning ra…
Proposes efficient training method for deep thin networks.
problem Deploying deep learning models with accuracy and compactness.
method Three-stage method: widen, warm up, fine tune.
result Deep thin networks trained with method outperform standard deep networks.
We give algorithms with provable guarantees that learn a class of deep nets in the generative model view popularized by Hinton and others. Our generative model is an n node multilayer neural net that has degree at most nγ for some γ<1 and each edge has a random edge weight in [−1,1]. Our algorithm learns {\em …
Deep Gaussian Processes (DGP) are hierarchical generalizations of Gaussian Processes (GP) that have proven to work effectively on a multiple supervised regression tasks. They combine the well calibrated uncertainty estimates of GPs with the great flexibility of multilayer models. In DGPs, given the inputs, the outputs …
Estimating machine learning performance 'in the wild' is both an important and unsolved problem. In this paper, we seek to examine, understand, and predict the pointwise competence of classification models. Our contributions are twofold: First, we establish a statistically rigorous definition of competence that general…
In this note we present a generative model of natural images consisting of a deep hierarchy of layers of latent random variables, each of which follows a new type of distribution that we call rectified Gaussian. These rectified Gaussian units allow spike-and-slab type sparsity, while retaining the differentiability nec…
Training avoids edge of stability by aligning Jacobian matrices.
problem Training neural networks on the edge of stability causes inaccuracies.
method Used an exponential Euler solver to prevent entering the edge of stability.
result Alignment of Jacobian matrices causes sharpness increase in Hessian.
SAEs struggle with curved activation manifolds, revealing layer-dependent scaling laws.
problem Sparse autoencoders' reconstruction error varies across layers, not fitting existing scaling laws.
method Cross-layer study of 844 SAE checkpoints, fitting and regressing on manifold geometry.
result Manifold geometry predicts layer-dependent width exponents in SAEs, with transferable coefficients.
New optimizers control network width scaling, improving stability and transfer across different model sizes.
problem Designing stable optimizers for networks of varying widths.
method Interpreting optimizers as steepest descent under mean-normalized operator norms, enabling layerwise composability and width-independent bounds.
result New optimizers like row normalization and column normalization provide stable learning-rate transfer across different model widths.
Proves deep networks can learn hierarchical structures efficiently.
problem Understanding how deep networks learn hierarchical structures in data.
method Random Hierarchy Models, gradient-based methods, layerwise training.
result Proves deep networks can efficiently learn hierarchical structures.
This paper explains how deep learning performs hierarchical learning efficiently.
problem How deep learning can perform hierarchical learning efficiently.
method Backward feature correction principle and SGD training.
result Deep learning can efficiently train complex hierarchical tasks using SGD.
Develops methods to measure and set function-space learning rates in neural networks.
problem Measuring and optimizing changes in neural network output functions.
method Efficient methods to measure and set function-space learning rates, requiring minimal computational overhead.
result Demonstrates FLeRM (Function-space Learning Rate Matching) for hyperparameter transfer across model scales.
New bounds adaptively control spectral complexity of trained Transformers.
problem Understanding why Transformers generalize well in machine learning.
method Spectrum-adaptive post hoc generalization bounds for multi-layer Transformers.
result Bounds adaptively trade off spectral complexity against dimension and depth factors.
A new robust prefix-tuning framework improves model robustness against adversarial attacks.
problem Lack of robustness in prefix-tuning for adversarial attacks.
method Leveraging layerwise activations of pretrained models for additional prefix finetuning during the test phase.
result Framework substantially improves robustness over strong baselines while maintaining comparable accuracy on clean texts.
With the development of cloud computing and big data, the reliability of data storage systems becomes increasingly important. Previous researchers have shown that machine learning algorithms based on SMART attributes are effective methods to predict hard drive failures. In this paper, we use SMART attributes to predict…
MTL-NAS combines NAS with GP-MTL for task-agnostic multi-task learning.
problem Designing architectures for diverse tasks with varying priors.
method Disentangled GP-MTL networks, hierarchical feature sharing, and gradient-based search.
result General-purpose model trained once can adapt to multiple tasks.
SmartExchange trades memory for computation in neural networks.
problem Energy-efficient inference of deep neural networks.
method Algorithm-hardware co-design framework integrating sparsification, decomposition, and quantization.
result SmartExchange reduces energy consumption and improves performance.
Scalable method bounds Lipschitz constant of generative models.
problem Bounding the Lipschitz constant of generative models.
method Layerwise convex approximations using zonotopes.
result Efficient and tight bounds on generative models.
A method to automatically choose feature dimensions in linear attention for better approximation quality.
problem Choosing the feature dimension in linear attention to balance quality and efficiency.
method Statistical degrees of freedom for determining feature dimension, layer-wise training strategy.
result Our method achieves smaller approximation error compared to fixed dimensions and improves model performance.
A new algorithm for efficiently removing specific classes from a model without retraining.
problem Class forgetting in machine learning models.
method Estimating retain and forget spaces using SVD, removing shared information, and updating weights.
result Achieved up to 1.38% accuracy improvement on ImageNet dataset with minimal samples.
A new measure predicts deep learning model performance.
problem Predicting the generalization error of deep learning models.
method 2sED measure based on effective dimension, layerwise iterative approximation.
result 2sED correlates well with training error and generalization error.
We propose a distributed approach to train deep neural networks (DNNs), which has guaranteed convergence theoretically and great scalability empirically: close to 6 times faster on instance of ImageNet data set when run with 6 machines. The proposed scheme is close to optimally scalable in terms of number of machines, …
Large learning rates work surprisingly well in standard parameterization, contrary to theory.
problem Theoretical limits of large learning rates do not match practical network behavior.
method Fine-grained analysis of learning rates and network behavior under cross-entropy loss.
result There are two distinct sub-regimes of unstable learning rates, with a controlled divergence regime where features continue to evolve.
Neural networks have become increasingly prevalent within the geosciences, although a common limitation of their usage has been a lack of methods to interpret what the networks learn and how they make decisions. As such, neural networks have often been used within the geosciences to most accurately identify a desired o…
New view: Deep GCNs learn to anti-oversmooth during training.
problem Performance drop in deep GCNs due to oversmoothing.
method Interpreted GCN as MLP + graph regularization, analyzed training process.
result Deep GCNs learn to anti-oversmooth during training, not over-smooth.
LASE learns graph embeddings by unrolling GD iterations into a neural network.
problem Learning graph embeddings from graph inputs.
method Truncating and interpreting GD iterations as layers in a GNN.
result LASE outperforms eigendecomposition routines and GNNs in link prediction and node classification.
AlgebraNets use alternative algebras for neural networks, improving performance on image and language tasks.
problem Improving neural network performance on large-scale image and language tasks.
method Considered alternative algebras (C, H, M2(R), M2(C), M3(R), M4(R)) for activations and weights, and studied their performance on ImageNet and enwiki8 datasets.
result Alternative algebras deliver better parameter and computational efficiency compared with real numbers, especially in sparse and auto-regressive inference scenarios.
The paper studies how noise synchronizes tokens in deep transformer models.
problem Understanding synchronization in deep learning models with noise.
method Proves convergence to a stochastic particle system and identifies the limiting SDE.
result The limiting model displays synchronization by noise and exponential dissipation of interaction energy.
Adversarial training has been shown to regularize deep neural networks in addition to increasing their robustness to adversarial examples. However, its impact on very deep state of the art networks has not been fully investigated. In this paper, we present an efficient approach to perform adversarial training by pertur…
Multitask learning has shown promising performance in many applications and many multitask models have been proposed. In order to identify an effective multitask model for a given multitask problem, we propose a learning framework called learning to multitask (L2MT). To achieve the goal, L2MT exploits historical multit…
Low complexity decentralized neural net with centralized performance.
problem Training large neural networks in distributed nodes without data sharing.
method Layer-wise learning using ADMM for low complexity and centralized performance.
result Equivalent learning performance to centralized training in distributed nodes.
Gradient-based meta-learning methods leverage gradient descent to learn the commonalities among various tasks. While previous such methods have been successful in meta-learning tasks, they resort to simple gradient descent during meta-testing. Our primary contribution is the {\em MT-net}, which enables the meta-learner…
While the backpropagation of error algorithm enables deep neural network training, it implies (i) bidirectional synaptic weight transport and (ii) update locking until the forward and backward passes are completed. Not only do these constraints preclude biological plausibility, but they also hinder the development of l…
TSSM splits neural networks for parallel training with minimal accuracy loss.
problem Accuracy degradation in parallel training of deep neural networks.
method TSSM reformulates alternating minimization to achieve parallelism with minimal accuracy loss.
result TSSM achieves significant speedup without accuracy loss on multiple datasets.
New learning rules for wide neural networks without backpropagation.
problem Training wide neural networks efficiently and without backpropagation.
method Input-weight alignment driven by gradient descent in the NTK regime.
result Biologically-motivated learning rules equivalent to backpropagation in wide networks.
While the authors of Batch Normalization (BN) identify and address an important problem involved in training deep networks-- Internal Covariate Shift-- the current solution has certain drawbacks. Specifically, BN depends on batch statistics for layerwise input normalization during training which makes the estimates of …
While the authors of Batch Normalization (BN) identify and address an important problem involved in training deep networks-- \textit{Internal Covariate Shift}-- the current solution has certain drawbacks. For instance, BN depends on batch statistics for layerwise input normalization during training which makes the esti…
New theory maps neural network weights to optimize faster and scale.
problem Optimizing neural networks for speed and scalability.
method Constructing a duality map using layer-wise operator norms.
result Derived GPU-friendly algorithms for various layers.