A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
This work proves the asymptotic freeness of layerwise Jacobians in MLPs with Haar orthogonal matrices.
New method improves neural network training by scaling perturbations layerwise.
Proposes efficient training method for deep thin networks.
Training large deep neural networks on massive datasets is computationally very challenging. There has been recent surge in interest in using large batch stochastic optimization methods to tackle this issue. The most prominent algorithm in this line of research is LARS, which by employing layerwise adaptive learning ra…
Use simplified layerwise linear models to understand neural dynamics.
A new method for learning Bayesian neural networks using layerwise inference.
Training avoids edge of stability by aligning Jacobian matrices.
Deep Gaussian Processes (DGP) are hierarchical generalizations of Gaussian Processes (GP) that have proven to work effectively on a multiple supervised regression tasks. They combine the well calibrated uncertainty estimates of GPs with the great flexibility of multilayer models. In DGPs, given the inputs, the outputs …
Deep learning is also known as hierarchical learning, where the learner _learns_ to represent a complicated target function by decomposing it into a sequence of simpler functions to reduce sample and time complexity. This paper formally analyzes how multi-layer neural networks can perform such hierarchical learning _ef…
Proves deep networks can learn hierarchical structures efficiently.
AlphaPruning optimizes LLM pruning using HT-SR theory for better performance.
New bounds adaptively control spectral complexity of trained Transformers.
Estimating machine learning performance 'in the wild' is both an important and unsolved problem. In this paper, we seek to examine, understand, and predict the pointwise competence of classification models. Our contributions are twofold: First, we establish a statistically rigorous definition of competence that general…
Efficient algorithm for CMDPs reduces to offline density estimation.
Develops methods to measure and set function-space learning rates in neural networks.
With the development of cloud computing and big data, the reliability of data storage systems becomes increasingly important. Previous researchers have shown that machine learning algorithms based on SMART attributes are effective methods to predict hard drive failures. In this paper, we use SMART attributes to predict…
Deep neural networks (DNNs) depend on the storage of a large number of parameters, which consumes an important portion of the energy used during inference. This paper considers the case where the energy usage of memory elements can be reduced at the cost of reduced reliability. A training algorithm is proposed to optim…
New optimizers control network width scaling, improving stability and transfer across different model sizes.
New view: Deep GCNs learn to anti-oversmooth during training.
We propose a distributed approach to train deep neural networks (DNNs), which has guaranteed convergence theoretically and great scalability empirically: close to 6 times faster on instance of ImageNet data set when run with 6 machines. The proposed scheme is close to optimally scalable in terms of number of machines, …
The lack of transparency of neural networks stays a major break for their use. The Layerwise Relevance Propagation technique builds heat-maps representing the relevance of each input in the model s decision. The relevance spreads backward from the last to the first layer of the Deep Neural Network. Layer-wise Relevance…
A new robust prefix-tuning framework improves model robustness against adversarial attacks.
Adversarial training has been shown to regularize deep neural networks in addition to increasing their robustness to adversarial examples. However, its impact on very deep state of the art networks has not been fully investigated. In this paper, we present an efficient approach to perform adversarial training by pertur…
A new measure predicts deep learning model performance.
In this note we present a generative model of natural images consisting of a deep hierarchy of layers of latent random variables, each of which follows a new type of distribution that we call rectified Gaussian. These rectified Gaussian units allow spike-and-slab type sparsity, while retaining the differentiability nec…
MTL-NAS combines NAS with GP-MTL for task-agnostic multi-task learning.
Low complexity decentralized neural net with centralized performance.
While the backpropagation of error algorithm enables deep neural network training, it implies (i) bidirectional synaptic weight transport and (ii) update locking until the forward and backward passes are completed. Not only do these constraints preclude biological plausibility, but they also hinder the development of l…
Large learning rates work surprisingly well in standard parameterization, contrary to theory.
TSSM splits neural networks for parallel training with minimal accuracy loss.
Multitask learning has shown promising performance in many applications and many multitask models have been proposed. In order to identify an effective multitask model for a given multitask problem, we propose a learning framework called learning to multitask (L2MT). To achieve the goal, L2MT exploits historical multit…
SAEs struggle with curved activation manifolds, revealing layer-dependent scaling laws.
While the authors of Batch Normalization (BN) identify and address an important problem involved in training deep networks-- Internal Covariate Shift-- the current solution has certain drawbacks. Specifically, BN depends on batch statistics for layerwise input normalization during training which makes the estimates of …
While the authors of Batch Normalization (BN) identify and address an important problem involved in training deep networks-- \textit{Internal Covariate Shift}-- the current solution has certain drawbacks. For instance, BN depends on batch statistics for layerwise input normalization during training which makes the esti…
New analysis shows how to balance privacy and accuracy in deep learning.
Injectivity of ReLU networks is characterized for generative models and inverse problems.
Traditionally, when generative models of data are developed via deep architectures, greedy layer-wise pre-training is employed. In a well-trained model, the lower layer of the architecture models the data distribution conditional upon the hidden variables, while the higher layers model the hidden distribution prior. Bu…
Shallow supervised 1-hidden layer neural networks have a number of favorable properties that make them easier to interpret, analyze, and optimize than their deep counterparts, but lack their representational power. Here we use 1-hidden layer learning problems to sequentially build deep networks layer by layer, which ca…
New learning rules for wide neural networks without backpropagation.
A method to automatically choose feature dimensions in linear attention for better approximation quality.
New theory maps neural network weights to optimize faster and scale.
Dropout and RaM become equivalent in large ResNets as depth and width increase.
Spectral gradient methods outperform Euclidean in certain deep learning scenarios.
Scalable method bounds Lipschitz constant of generative models.
In this paper, we study two aspects of the variational autoencoder (VAE): the prior distribution over the latent variables and its corresponding posterior. First, we decompose the learning of VAEs into layerwise density estimation, and argue that having a flexible prior is beneficial to both sample generation and infer…
Looped transformers with LN converge to power method for principal component prediction.
The paper explores how invertibility affects the complexity of encoder models in VAEs.
DNNs with regularization reveal feature learning dynamics and sparsity.