A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
Deep learning reduces training overhead in massive MIMO systems.
When scaling distributed training, the communication overhead is often the bottleneck. In this paper, we propose a novel SGD variant with reduced communication and adaptive learning rates. We prove the convergence of the proposed algorithm for smooth but non-convex problems. Empirical results show that the proposed alg…
TorchGAN is a PyTorch based framework for writing succinct and comprehensible code for training and evaluation of Generative Adversarial Networks. The framework's modular design allows effortless customization of the model architecture, loss functions, training paradigms, and evaluation metrics. The key features of Tor…
Paper introduces a new adaptive gradient method with gradient compression for distributed training.
ISAAC Newton uses input-based curvature for efficient training.
Two main obstacles preventing the widespread adoption of variational Bayesian neural networks are the high parameter overhead that makes them infeasible on large networks, and the difficulty of implementation, which can be thought of as "programming overhead." MC dropout [Gal and Ghahramani, 2016] is popular because it…
Distributed implementations of mini-batch stochastic gradient descent (SGD) suffer from communication overheads, attributed to the high frequency of gradient updates inherent in small-batch training. Training with large batches can reduce these overheads; however, large batches can affect the convergence properties and…
Paper proposes ADC framework to reduce ViT SL training communication overhead.
SOTERIA optimizes neural networks for secure inference with minimal overhead.
Low-precision training reduces computational cost and produces efficient models. Recent research in developing new low-precision training algorithms often relies on simulation to empirically evaluate the statistical effects of quantization while avoiding the substantial overhead of building specific hardware. To suppor…
New method reduces communication in deep learning training.
We use GG distributions to optimize LLMs, reducing size and training time.
Federated LIDAR aided beam selection reduces mmWave beam search overhead.
Distributed model training suffers from communication overheads due to frequent gradient updates transmitted between compute nodes. To mitigate these overheads, several studies propose the use of sparsified stochastic gradients. We argue that these are facets of a general sparsification method that can operate on any p…
Paper proposes a method to estimate variance reduction in DNN training using importance sampling.
On-device machine learning (ML) enables the training process to exploit a massive amount of user-generated private data samples. To enjoy this benefit, inter-device communication overhead should be minimized. With this end, we propose federated distillation (FD), a distributed model training algorithm whose communicati…
Global-QSGD accelerates distributed training by up to 3.51%.
Training modern deep learning models requires large amounts of computation, often provided by GPUs. Scaling computation from one GPU to many can enable much faster training and research progress but entails two complications. First, the training library must support inter-GPU communication. Depending on the particular …
A new FL algorithm reduces communication overhead by selectively updating model parameters.
BayesAdapter turns pre-trained NNs into reliable BNNs with minimal overhead.
One of the most significant bottleneck in training large scale machine learning models on parameter server (PS) is the communication overhead, because it needs to frequently exchange the model gradients between the workers and servers during the training iterations. Gradient quantization has been proposed as an effecti…
Learning to learn has emerged as an important direction for achieving artificial intelligence. Two of the primary barriers to its adoption are an inability to scale to larger problems and a limited ability to generalize to new tasks. We introduce a learned gradient descent optimizer that generalizes well to new tasks, …
SIFT reduces training time by selecting samples with approximate losses.
Lapse improves parameter servers by dynamically allocating parameters, achieving near-linear scaling.
Generative Adapter adapts LMs with a single forward pass, reducing inference overhead.
Turbo-Aggregate reduces secure aggregation time from quadratic to nearly linear.
Adaptive gradient-based optimizers such as Adagrad and Adam are crucial for achieving state-of-the-art performance in machine translation and language modeling. However, these methods maintain second-order statistics for each parameter, thus introducing significant memory overheads that restrict the size of the model b…
TCR improves DNN robustness to noisy labels with minimal overhead.
A new method combines generative models and regularization to prevent forgetting in continual learning.
New method SF-AdamW trains large models without decay phases or memory overhead.
Automatically detects and down-weights noisy samples in machine learning training.
FedLog reduces communication in federated learning by sharing data summaries.
A new method combines regularization and generative rehearsal for continual learning.
Deep neural networks have achieved impressive performance in many applications but their large number of parameters lead to significant computational and storage overheads. Several recent works attempt to mitigate these overheads by designing compact networks using pruning of connections. However, we observe that most …
A basic question in the theory of fault-tolerant quantum computation is to understand the fundamental resource costs for performing a universal logical set of gates on encoded qubits to arbitrary accuracy. Here we consider qubits encoded with constant space overhead (i.e. finite encoding rate) in the limit of arbitrari…
The most straightforward method to accelerate Stochastic Gradient Descent (SGD) computation is to distribute the randomly selected batch of inputs over multiple processors. To keep the distributed processors fully utilized requires commensurately growing the batch size. However, large batch training often leads to poor…
Nystrom approximation speeds up kernel model training.
The impact of the maximally possible batch size (for the better runtime) on performance of graphic processing units (GPU) and tensor processing units (TPU) during training and inference phases is investigated. The numerous runs of the selected deep neural network (DNN) were performed on the standard MNIST and Fashion-M…
New approach detects adversarial samples with certifiable guarantees.
GPA improves LLM training speed by 8.71% for Llama-160M models.
Drone optimizes collaborative learning for neural networks.
We consider the problem of learning a binary classifier from different data sources, among which at most an fraction are adversarial. The overhead is defined as the ratio between the sample complexity of learning in this setting and that of learning the same hypothesis class on a single data distribution. We pr…
Ensuring differential privacy of models learned from sensitive user data is an important goal that has been studied extensively in recent years. It is now known that for some basic learning problems, especially those involving high-dimensional data, producing an accurate private model requires much more data than learn…
Clapping reduces memory usage in distributed optimization by reusing data samples.
The performance and efficiency of distributed machine learning (ML) depends significantly on how long it takes for nodes to exchange state changes. Overly-aggressive attempts to reduce communication often sacrifice final model accuracy and necessitate additional ML techniques to compensate for this loss, limiting their…
We propose a novel data-dependent structured gradient regularizer to increase the robustness of neural networks vis-a-vis adversarial perturbations. Our regularizer can be derived as a controlled approximation from first principles, leveraging the fundamental link between training with noise and regularization. It adds…
Distributed stochastic gradient descent (SGD) algorithms are widely deployed in training large-scale deep learning models, while the communication overhead among workers becomes the new system bottleneck. Recently proposed gradient sparsification techniques, especially Top- sparsification with error compensation (To…
EControl improves fast distributed optimization with compression and error control.