arXiv research

Speeds up deep neural networks training by 10x using GPU concurrency.

Study the effects of data parallelism and sparsity on neural network training.

PALMS reconstructs large-scale networks efficiently with parallel computing.

Parallel neural network training yields better long-term prediction accuracy.

Improves parallel deep model performance by restructuring and pruning.

TSSM splits neural networks for parallel training with minimal accuracy loss.

We propose a new integrated method of exploiting model, batch and domain parallelism for the training of deep neural networks (DNNs) on large distributed-memory computers using minibatch stochastic gradient descent (SGD). Our goal is to find an efficient parallelization strategy for a fixed batch size using $P$ process…

Ensembled neural networks improve MRI image quality.

This paper describes neural-fortran, a parallel Fortran framework for neural networks and deep learning. It features a simple interface to construct feed-forward neural networks of arbitrary structure and size, several activation functions, and stochastic gradient descent as the default optimization algorithm. Neural-f…

Paper presents efficient algorithms for convolutional neural networks using Winograd minimal filtering.

Optimized parallel RNN training reaches up to 845x speedup.

Linear algebra approach for parallel deep learning models.

A new framework for efficient Bayesian network inference.

RandNet-Parareal uses neural networks to speed up time-parallel PDE solving.

The paper introduces a multilevel initialization method for deep neural networks.

Parallel sentence extraction is a task addressing the data sparsity problem found in multilingual natural language processing applications. We propose a bidirectional recurrent neural network based approach to extract parallel sentences from collections of multilingual texts. Our experiments with noisy parallel corpora…

PLN-Nets with two linear layers and parallel LN achieve universal approximation.

Artificial Neural Networks (ANNs) have received increasing attention in recent years with applications that span a wide range of disciplines including vital domains such as medicine, network security and autonomous transportation. However, neural network architectures are becoming increasingly complex and with an incre…

Deep neural networks, and in particular recurrent networks, are promising candidates to control autonomous agents that interact in real-time with the physical world. However, this requires a seamless integration of temporal features into the network's architecture. For the training of and inference with recurrent neura…

NEST optimizes deep learning training by placing devices efficiently across networks and memory.

OptEx accelerates first-order optimization with parallelized iterations.

Parallel unlearning framework for inherited models reduces computational overhead.

New neural network can learn multiple patterns at once.

Enhances parallelism in decentralized learning for larger networks.

In this research, Artificial Neural Networks (ANNs) have been used as a powerful tool to solve the inverse kinematic equations of a parallel robot. For this purpose, we have developed the kinematic equations of a Tricept parallel kinematic mechanism with two rotational and one translational degrees of freedom (DoF). Us…

Benefitting from large-scale training datasets and the complex training network, Convolutional Neural Networks (CNNs) are widely applied in various fields with high accuracy. However, the training process of CNNs is very time-consuming, where large amounts of training samples and iterative operations are required to ob…

PipeMare enables efficient DNN training with minimal memory and pipeline sacrifices.

Paper proposes DCT for efficient hybrid parallel training of large recommendation models.

Parallel score matching accelerates DPM training and improves density estimation.

Backpropagation algorithm is indispensable for the training of feedforward neural networks. It requires propagating error gradients sequentially from the output layer all the way back to the input layer. The backward locking in backpropagation algorithm constrains us from updating network layers in parallel and fully l…

Training a neural network using backpropagation algorithm requires passing error gradients sequentially through the network. The backward locking prevents us from updating network layers in parallel and fully leveraging the computing resources. Recently, there are several works trying to decouple and parallelize the ba…

A new trading model combines GARCH and PPO for better stock trading profits.

This paper proposes a method that allows non-parallel many-to-many voice conversion (VC) by using a variant of a generative adversarial network (GAN) called StarGAN. Our method, which we call StarGAN-VC, is noteworthy in that it (1) requires no parallel utterances, transcriptions, or time alignment procedures for speec…

Efficiently optimizes orthogonal and Stiefel matrices on parallel units.

Neural networks compress uninformative input directions, improving test error.

Although voice conversion (VC) algorithms have achieved remarkable success along with the development of machine learning, superior performance is still difficult to achieve when using nonparallel data. In this paper, we propose using a cycle-consistent adversarial network (CycleGAN) for nonparallel data-based VC train…

We propose a parallel-data-free voice-conversion (VC) method that can learn a mapping from source to target speech without relying on parallel data. The proposed method is general purpose, high quality, and parallel-data free and works without any extra data, modules, or alignment procedure. It also avoids over-smoothi…

Training deep networks is expensive and time-consuming with the training period increasing with data size and growth in model parameters. In this paper, we provide a framework for distributed training of deep networks over a cluster of CPUs in Apache Spark. The framework implements both Data Parallelism and Model Paral…

Asynchronous parallel implementations of stochastic gradient (SG) have been broadly used in solving deep neural network and received many successes in practice recently. However, existing theories cannot explain their convergence and speedup properties, mainly due to the nonconvexity of most deep learning formulations …

New algorithms reduce communication in GNN training.

Due to the need for robust uncertainty quantification, Bayesian neural learning has gained attention in the era of deep learning and big data. Markov Chain Monte-Carlo (MCMC) methods typically implement Bayesian inference which faces several challenges given a large number of parameters, complex and multimodal posterio…

We propose a new algorithm called Parle for parallel training of deep networks that converges 2-4x faster than a data-parallel implementation of SGD, while achieving significantly improved error rates that are nearly state-of-the-art on several benchmarks including CIFAR-10 and CIFAR-100, without introducing any additi…

New networks improve MRI image reconstruction without calibration.

HPSGD speeds up DNN training by paralleling data sync with local training.

Cyclic Data Parallelism reduces memory usage and balances gradient communications.

We develop a convex relaxation method for analyzing neural network generalization.

Path regularization reveals convex optimization in deep ReLU networks.

AgEBO-Tabular combines NAS and hyperparameter tuning for fast, high-performing tabular models.