arXiv research

Improved NiNo networks accelerate Adam training by up to 50%.

Unified framework for understanding and optimizing training acceleration.

Stochastic gradient decent~(SGD) and its variants, including some accelerated variants, have become popular for training in machine learning. However, in all existing SGD and its variants, the sample size in each iteration~(epoch) of training is the same as the size of the full training set. In this paper, we propose a…

Improved training of large-scale neural networks with reduced variance noise.

Selective-Backprop accelerates deep learning by focusing on high-loss examples.

Stacking improves deep neural network training efficiency.

A new method accelerates deep neural network training using minimal margin score.

Loss-guided training accelerates node embedding methods on graphs.

PipeDream-2BW accelerates large model training by 20x with minimal memory usage.

Surrogate models speed up RL training in dynamic systems.

A common problem in training neural networks is the vanishing and/or exploding gradient problem which is more prominently seen in training of Recurrent Neural Networks (RNNs). Thus several algorithms have been proposed for training RNNs. This paper proposes a novel adaptive stochastic Nesterov accelerated quasiNewton (…

Hardware-accelerated RBM solves large combinatorial problems and integer factorization.

ES improves training efficiency by dynamically selecting data samples.

Training generative adversarial networks (GANs) often suffers from cyclic behaviors of iterates. Based on a simple intuition that the direction of centripetal acceleration of an object moving in uniform circular motion is toward the center of the circle, we present the Simultaneous Centripetal Acceleration (SCA) method…

SEED RL accelerates deep RL training on modern accelerators.

SuNCEt accelerates contrastive learning with minimal labeled data.

In recent years, deep neural networks (DNN) have demonstrated significant business impact in large scale analysis and classification tasks such as speech recognition, visual object detection, pattern extraction, etc. Training of large DNNs, however, is universally considered as time consuming and computationally intens…

New insights into GNN optimization reveal skip connections and depth accelerate training.

Polyak's momentum accelerates training of neural networks.

A significant advance in accelerating neural network training has been the development of normalization methods, permitting the training of deep models both faster and with better accuracy. These advances come with practical challenges: for instance, batch normalization ties the prediction of individual examples with o…

Superposition accelerates training to a universal power-law exponent.

GPU acceleration speeds up financial machine learning training time.

Paper finds sharpness differences in transformer blocks accelerating LLM training.

Paper improves DNN accelerator robustness against bit errors with energy savings.

Generative model synthesizes earthquake acceleration data.

AB dynamically scales gradients to mitigate asynchronous training delays.

Sparsification is an efficient approach to accelerate CNN inference, but it is challenging to take advantage of sparsity in training procedure because the involved gradients are dynamically changed. Actually, an important observation shows that most of the activation gradients in back-propagation are very close to zero…

Accelerates data loading in deep neural network training by 30x.

We propose a generic algorithmic building block to accelerate training of machine learning models on heterogeneous compute systems. Our scheme allows to efficiently employ compute accelerators such as GPUs and FPGAs for the training of large-scale machine learning models, when the training data exceeds their memory cap…

State-of-the-art convolutional neural networks (CNNs) used in vision applications have large models with numerous weights. Training these models is very compute- and memory-resource intensive. Much research has been done on pruning or compressing these models to reduce the cost of inference, but little work has address…

FedNAG improves federated learning accuracy and reduces training time.

Seesaw optimizes training by balancing learning rate and batch size, accelerating model pretraining.

Recent technological advances have proliferated the available computing power, memory, and speed of modern Central Processing Units (CPUs), Graphics Processing Units (GPUs), and Field Programmable Gate Arrays (FPGAs). Consequently, the performance and complexity of Artificial Neural Networks (ANNs) is burgeoning. While…

Un-trained neural networks outperform trained methods in MRI reconstruction.

The paper accelerates LLM inference by adding early exit heads trained in a self-supervised manner.

GradSkip reduces local training steps for better communication efficiency.

Deep neural networks have yielded superior performance in many applications; however, the gradient computation in a deep model with millions of instances lead to a lengthy training process even with modern GPU/TPU hardware acceleration. In this paper, we propose AutoAssist, a simple framework to accelerate training of …

Poor (even random) starting points for learning/training/optimization are common in machine learning. In many settings, the method of Robbins and Monro (online stochastic gradient descent) is known to be optimal for good starting points, but may not be optimal for poor starting points -- indeed, for poor starting point…

HAL accelerates the generation of training sets for accurate interatomic potentials.

The Regularized Nonlinear Acceleration (RNA) algorithm is an acceleration method capable of improving the rate of convergence of many optimization schemes such as gradient descend, SAGA or SVRG. Until now, its analysis is limited to convex problems, but empirical observations shows that RNA may be extended to wider set…

Deep unfolding accelerates MCMC-based COP solvers.

Dropout is a simple but efficient regularization technique for achieving better generalization of deep neural networks (DNNs); hence it is widely used in tasks based on DNNs. During training, dropout randomly discards a portion of the neurons to avoid overfitting. This paper presents an enhanced dropout technique, whic…

New algorithm PRACTISE accelerates networks with tiny sets, reducing latency by 22%.

This paper improves NAQ method for faster convergence on Tensorflow.

Proposes EDM algorithm to accelerate model training in distributed networks.

Fast emulators built with neural search accelerate expensive scientific simulations.

Accelerates BERT pretraining from 3 days to 54 minutes.

SUOD accelerates OD for large, diverse models.