A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
Layer-wise preconditioning methods improve neural network optimization and feature learning.
Unified framework LPCD optimizes quantization of complex submodules.
A fundamental question in deep learning concerns the role played by individual layers in a deep neural network (DNN) and the transferable properties of the data representations which they learn. To the extent that layers have clear roles, one should be able to optimize them separately using layer-wise loss functions. S…
We propose a new optimization method for training feed-forward neural networks. By rewriting the activation function as an equivalent proximal operator, we approximate a feed-forward neural network by adding the proximal operators to the objective function as penalties, hence we call the lifted proximal operator machin…
Proposes QEP to mitigate quantization error propagation in layer-wise post-training quantization.
Gluon optimizes LMO-based methods for large-scale tasks, improving performance and theory-practice gap.
Physics-inspired methods optimize SVD compression of LLMs.
Layer-wise networks have a closed-form solution and a stopping criterion.
Layer-wise networks have a closed-form solution and a stopping criterion.
Drop-Muon updates only some layers, speeding up training.
Deep neural networks have been used in various machine learning applications and achieved tremendous empirical successes. However, training deep neural networks is a challenging task. Many alternatives have been proposed in place of end-to-end back-propagation. Layer-wise training is one of them, which trains a single …
Training deep neural networks using a large batch size has shown promising results and benefits many real-world applications. However, the optimizer converges slowly at early epochs and there is a gap between large-batch deep learning optimization heuristics and theoretical underpinnings. In this paper, we propose a no…
In this paper, we tackle the problem of explanations in a deep-learning based model for recommendations by leveraging the technique of layer-wise relevance propagation. We use a Deep Convolutional Neural Network to extract relevant features from the input images before identifying similarity between the images in featu…
LEWIS merges LLMs without training, improving performance on specific tasks.
Random layer-wise pruning profiles are as effective as metric-based ones for various datasets.
Low complexity decentralized neural net with centralized performance.
Analyzes layer-wise quantization effects in neural networks.
Automatically learns flexible symmetry constraints in neural networks using gradients.
Improves synthetic-to-real generalization without real data.
This article proposes a communication-efficient decentralized deep learning algorithm, coined layer-wise federated group ADMM (L-FGADMM). To minimize an empirical risk, every worker in L-FGADMM periodically communicates with two neighbors, in which the periods are separately adjusted for different layers of its deep ne…
Pre-training is crucial for learning deep neural networks. Most of existing pre-training methods train simple models (e.g., restricted Boltzmann machines) and then stack them layer by layer to form the deep structure. This layer-wise pre-training has found strong theoretical foundation and broad empirical support. Howe…
A new method improves few-shot image classification by updating top layers.
DSA efficiently allocates sparsity across layers for budgeted pruning.
To reduce the long training time of large deep neural network (DNN) models, distributed synchronous stochastic gradient descent (S-SGD) is commonly used on a cluster of workers. However, the speedup brought by multiple workers is limited by the communication overhead. Two approaches, namely pipelining and gradient spar…
Joslim optimizes both width and weight configurations for slimmable neural networks, improving model efficiency.
Speeds up deep neural networks training by 10x using GPU concurrency.
Study on shortcuts in deep networks, revealing their layer-wise distribution and impact.
EF21-Muon optimizes deep learning with error feedback, improving efficiency and accuracy.
Stochastic gradient descent (SGD) has achieved great success in training deep neural network, where the gradient is computed through back-propagation. However, the back-propagated values of different layers vary dramatically. This inconsistence of gradient magnitude across different layers renders optimization of deep …
Despite their great success in practical applications, there is still a lack of theoretical and systematic methods to analyze deep neural networks. In this paper, we illustrate an advanced information theoretic methodology to understand the dynamics of learning and the design of autoencoders, a special type of deep lea…
We propose NovoGrad, an adaptive stochastic gradient descent method with layer-wise gradient normalization and decoupled weight decay. In our experiments on neural networks for image classification, speech recognition, machine translation, and language modeling, it performs on par or better than well tuned SGD with mom…
This work analyzes how different layers in deep neural networks contribute to generalization error.
We establish the first benchmark for federated learning with differential privacy in ASR, achieving competitive performance.
Supervised training of neural networks for classification is typically performed with a global loss function. The loss function provides a gradient for the output layer, and this gradient is back-propagated to hidden layers to dictate an update direction for the weights. An alternative approach is to train the network …
We propose a novel family of connectionist models based on kernel machines and consider the problem of learning layer-by-layer a compositional hypothesis class, i.e., a feedforward, multilayer architecture, in a supervised setting. In terms of the models, we present a principled method to "kernelize" (partly or complet…
We propose two approaches of locally adaptive activation functions namely, layer-wise and neuron-wise locally adaptive activation functions, which improve the performance of deep and physics-informed neural networks. The local adaptation of activation function is achieved by introducing a scalable parameter in each lay…
Model compression has been widely adopted to obtain light-weighted deep neural networks. Most prevalent methods, however, require fine-tuning with sufficient training data to ensure accuracy, which could be challenged by privacy and security issues. As a compromise between privacy and performance, in this paper we inve…
QuantEase optimizes LLMs with CD-based quantization, achieving state-of-the-art performance.
RedEx improves neural network optimization with convex optimization guarantees.
This work improves OOD detection using deep generative models by approximating Fisher information metrics.
Improved neural quantization reduces accuracy loss to less than 1% with 4-bit weights.
InteractionNet models noncovalent protein-ligand interactions with GNNs and explains predictions.
Recurrent Neural Networks (RNNs) achieve state-of-the-art results in many sequence-to-sequence modeling tasks. However, RNNs are difficult to train and tend to suffer from overfitting. Motivated by the Data Processing Inequality (DPI), we formulate the multi-layered network as a Markov chain, introducing a training met…
TempBalance improves neural network training by balancing learning rates across layers.
When using deep, multi-layered architectures to build generative models of data, it is difficult to train all layers at once. We propose a layer-wise training procedure admitting a performance guarantee compared to the global optimum. It is based on an optimistic proxy of future performance, the best latent marginal. W…
Standard optimizers perform as well as LARS and LAMB at large batch sizes.
The lack of transparency of neural networks stays a major break for their use. The Layerwise Relevance Propagation technique builds heat-maps representing the relevance of each input in the model s decision. The relevance spreads backward from the last to the first layer of the Deep Neural Network. Layer-wise Relevance…
A new method prevents forgetting during knowledge transfer.