arXiv research

DANCE optimizes neural network and accelerator design for faster, more efficient DNN execution.

Improved NiNo networks accelerate Adam training by up to 50%.

Recent technological advances have proliferated the available computing power, memory, and speed of modern Central Processing Units (CPUs), Graphics Processing Units (GPUs), and Field Programmable Gate Arrays (FPGAs). Consequently, the performance and complexity of Artificial Neural Networks (ANNs) is burgeoning. While…

Improved training of large-scale neural networks with reduced variance noise.

Stacking improves deep neural network training efficiency.

Improved SDE-BNN model reduces NFEs and accelerates convergence.

Optimized neural networks for Edge TPU achieve high accuracy in real-time image classification.

New algorithm PRACTISE accelerates networks with tiny sets, reducing latency by 22%.

Smartfluidnet accelerates Eulerian fluid simulation with neural networks.

Graph neural networks speed up nonnegative matrix factorization.

Polyak's momentum accelerates training of neural networks.

DeepLight accelerates CTR predictions in ad serving by 46X.

A new method accelerates deep neural network training using minimal margin score.

Memory-augmented neural networks (MANNs) are designed for question-answering tasks. It is difficult to run a MANN effectively on accelerators designed for other neural networks (NNs), in particular on mobile devices, because MANNs require recurrent data paths and various types of operations related to external memory a…

In this paper we explore acceleration techniques for large scale nonconvex optimization problems with special focuses on deep neural networks. The extrapolation scheme is a classical approach for accelerating stochastic gradient descent for convex optimization, but it does not work well for nonconvex optimization typic…

New methods accelerate distributed optimization in noisy networks.

ALONE uses neural networks to accelerate cine MRI without needing ground truth.

Selective-Backprop accelerates deep learning by focusing on high-loss examples.

Survey of Graph Neural Networks for efficient computation.

We consider the problem of accelerating distributed optimization in multi-agent networks by sequentially adding edges. Specifically, we extend the distributed dual averaging (DDA) subgradient algorithm to evolving networks of growing connectivity and analyze the corresponding improvement in convergence rate. It is know…

Nesterov SGD is widely used for training modern neural networks and other machine learning models. Yet, its advantages over SGD have not been theoretically clarified. Indeed, as we show in our paper, both theoretically and empirically, Nesterov SGD with any parameter selection does not in general provide acceleration o…

Super-acceleration of gradient descent with momentum improves loss function minimization.

In this paper, we propose a novel progressive parameter pruning method for Convolutional Neural Network acceleration, named Structured Probabilistic Pruning (SPP), which effectively prunes weights of convolutional layers in a probabilistic manner. Unlike existing deterministic pruning approaches, where unimportant weig…

Accelerates sampling from Gibbs distributions using ARWP method.

GD and NAG accelerate matrix factorization and neural networks.

Paper proposes a new binary quantization method for faster DNN inference.

GRU models with Adam optimizer outperform other combinations in stock market forecasting.

Survey on pruning CNN models to reduce size for edge devices.

New algorithms accelerate SVGD convergence using deep unfolding.

Differentiable voxelization for 3D meshes with GPU acceleration.

Training generative adversarial networks (GANs) often suffers from cyclic behaviors of iterates. Based on a simple intuition that the direction of centripetal acceleration of an object moving in uniform circular motion is toward the center of the circle, we present the Simultaneous Centripetal Acceleration (SCA) method…

Generative model synthesizes earthquake acceleration data.

We consider efficiency in the implementation of deep neural networks. Hardware accelerators are gaining interest as machine learning becomes one of the drivers of high-performance computing. In these accelerators, the directed graph describing a neural network can be implemented as a directed graph describing a Boolean…

Proposes EDM algorithm to accelerate model training in distributed networks.

A common problem in training neural networks is the vanishing and/or exploding gradient problem which is more prominently seen in training of Recurrent Neural Networks (RNNs). Thus several algorithms have been proposed for training RNNs. This paper proposes a novel adaptive stochastic Nesterov accelerated quasiNewton (…

In recent years, convolutional neural networks (CNNs) have shown great performance in various fields such as image classification, pattern recognition, and multi-media compression. Two of the feature properties, local connectivity and weight sharing, can reduce the number of parameters and increase processing speed dur…

New insights into GNN optimization reveal skip connections and depth accelerate training.

The paper models and analyzes faults in TPU-based neural networks.

A significant advance in accelerating neural network training has been the development of normalization methods, permitting the training of deep models both faster and with better accuracy. These advances come with practical challenges: for instance, batch normalization ties the prediction of individual examples with o…

Recently, deep neural networks (DNNs) have shown advantages in accelerating optimization algorithms. One approach is to unfold finite number of iterations of conventional optimization algorithms and to learn parameters in the algorithms. However, these are forward methods and are indeed neither iterative nor convergent…

Accelerated magnetic resonance (MR) scan acquisition with compressed sensing (CS) and parallel imaging is a powerful method to reduce MR imaging scan time. However, many reconstruction algorithms have high computational costs. To address this, we investigate deep residual learning networks to remove aliasing artifacts …

Interneurons improve learning in neural networks by accelerating convergence.

Accelerates Bayesian optimization of function networks with partial evaluations.

The past several years have seen both an explosion in the use of Convolutional Neural Networks (CNNs) and the design of accelerators to make CNN inference practical. In the architecture community, the lion share of effort has targeted CNN inference for image recognition. The closely related problem of video recognition…

The Regularized Nonlinear Acceleration (RNA) algorithm is an acceleration method capable of improving the rate of convergence of many optimization schemes such as gradient descend, SAGA or SVRG. Until now, its analysis is limited to convex problems, but empirical observations shows that RNA may be extended to wider set…

Unpaired deep learning reconstructs MRI images from accelerated data.

Hardware-accelerated RBM solves large combinatorial problems and integer factorization.

Sparsity helps reduce the computational complexity of deep neural networks by skipping zeros. Taking advantage of sparsity is listed as a high priority in next generation DNN accelerators such as TPU. The structure of sparsity, i.e., the granularity of pruning, affects the efficiency of hardware accelerator design as w…