This work optimizes deep learning training by combining data and model parallelism.
problem Training large models with multiple GPUs suffers from high communication overhead and statistical efficiency loss.
method Hybrid parallelization combining data and model parallelism.
result Hybrid training provides significant speedup compared to data parallelism alone.
Proposes a new architecture to speed up large-scale CNN training.
problem Time-consuming training process of large-scale CNNs.
method Bi-layered Parallel Training (BPT-CNN) architecture with outer-layer and inner-layer parallelism.
result Improves training performance of CNNs while maintaining accuracy.
Parallel neural network training yields better long-term prediction accuracy.
problem Choosing the right training strategy for neural networks in dynamical systems.
method Comparison of parallel and series-parallel training strategies on five neural network architectures and two examples.
result Parallel training consistently outperforms series-parallel training in long-term prediction accuracy.
Study the effects of data parallelism and sparsity on neural network training.
problem Understanding the effects of data parallelism and sparsity on neural network training.
method Conducted extensive experiments and developed a theoretical analysis.
result Found a general scaling trend between batch size and number of training steps to convergence for the effect of data parallelism, and difficulty of training under sparsity.
Improved voice conversion with semi-supervised learning.
problem Voice conversion with limited parallel data.
method Amortized variational inference with parallel and non-parallel utterances.
result Semi-supervised training improves voice conversion performance.
Adaptive batch size schedules improve language model training efficiency and generalization.
problem Dilemma of choosing batch sizes in large-scale model training.
method General-purpose adaptive batch size schedules compatible with data and model parallelism.
result Adaptive batch size schedules outperform constant batch sizes and heuristic warmup schedules.
Linear algebra approach for parallel deep learning models.
problem Training large DNNs in distributed environments.
method Linear algebraic approach to model parallelism.
result Manual development of backward operators for gradient-based training.
Optimized parallel RNN training reaches up to 845x speedup.
problem Expensive RNN training through back-propagation through time (BPTT).
method Optimized parallel algorithm \opt based on ELM, leveraging GPU shared memory and QR factorization.
result Up to 845x speedup over sequential training and 20x less time to train.
TensorOpt finds optimal parallelization strategies for DNN training.
problem Finding efficient parallelization strategies for DNN training.
method TensorOpt uses an algorithm (FT) to search for an optimal set of parallelization strategies considering multiple objectives.
result TensorOpt provides accurate runtime cost estimation and adapts to resource availability.
Speeds up deep neural networks training by 10x using GPU concurrency.
problem Training deep residual neural networks efficiently.
method Layer-wise parallel training with GPU concurrency and Nonlinear Multigrid.
result 10.2x speedup over traditional techniques.
Optimizes parallel training of linear models, improving convergence.
problem Improving convergence of parallel training of linear models.
method Data partitioning scheme across threads to improve convergence.
result Achieved up to 42x speedup in convergence compared to state of the art implementations.
SWAP uses large mini-batches to train DNNs faster with good generalization.
problem Training deep neural networks with small mini-batches is time-consuming.
method SWAP computes an approximate solution with large mini-batches and refines it by averaging weights of multiple parallel models.
result SWAP trains models as well as small-batch training but in significantly less time.
Paper proposes DCT for efficient hybrid parallel training of large recommendation models.
problem Training large recommendation models at scale with efficient communication.
method Dynamic Communication Thresholding (DCT) for both Data Parallelism and Model Parallelism.
result Reduces communication by 100x and 20x during DP and MP, respectively, improving training time by 37%.
PETRA enables parallel training of deep models with reversible architectures.
problem Challenges in parallelizing deep model training.
method Introduces PETRA, a novel approach for parallelizing gradient computations in reversible architectures.
result Achieves competitive accuracies on CIFAR-10, ImageNet32, and ImageNet using ResNet models.
HPSGD speeds up DNN training by paralleling data sync with local training.
problem Low cluster utilization in distributed deep neural network training.
method Hierarchical Parallel SGD (HPSGD) with improved model updating for stale gradients.
result Significantly boosts distributed DNN training and reduces stale gradients.
Efficiently trains HDP topic models on large datasets using a sparse data-parallel sampler.
problem Scaling non-parametric topic models to large datasets.
method Data-parallel training with a doubly sparse sampler for HDP topic models.
result Trains HDP topic models on a 8m document, 768m token PubMed corpus in under 4 days.
Parallel score matching accelerates DPM training and improves density estimation.
problem Extended training periods and limited modeling flexibility in DPMs.
method Partitioning the learning task into independent time sub-intervals and modeling the score at each time point separately.
result Significant acceleration of training process and improved density estimation performance.
We propose a new integrated method of exploiting model, batch and domain parallelism for the training of deep neural networks (DNNs) on large distributed-memory computers using minibatch stochastic gradient descent (SGD). Our goal is to find an efficient parallelization strategy for a fixed batch size using P process…
TSSM splits neural networks for parallel training with minimal accuracy loss.
problem Accuracy degradation in parallel training of deep neural networks.
method TSSM reformulates alternating minimization to achieve parallelism with minimal accuracy loss.
result TSSM achieves significant speedup without accuracy loss on multiple datasets.
Improved RBM training speed using parallel computing.
problem High computational cost of training RBM models.
method Parallelized training using Horovod framework.
result Training time reduced to less than 12 minutes on 64 CPU nodes.
PipeMare enables efficient DNN training with minimal memory and pipeline sacrifices.
problem Sacrificing hardware efficiency to maintain statistical efficiency in pipeline parallel DNN training.
method PipeMare is a simple yet robust training method that tolerates asynchronous updates during pipeline parallelism without sacrificing pipeline utilization or memory.
result PipeMare achieves up to 2.7x less memory usage or 4.3x higher pipeline utilization compared to state-of-the-art synchronous PP training techniques.
A new training method speeds up ResNet training by 3x with minimal accuracy loss.
problem Training ResNets is slow due to dependencies between modules.
method Serial-parallel hybrid training strategy with data augmentation and downsampling.
result Significant speedup over traditional methods with comparable accuracy.
ADMMiRNN solves RNN training issues with stable convergence.
problem Training RNN with stable convergence and avoiding gradient issues.
method Built ADMMiRNN framework on unfolded RNN, providing novel update rules and theoretical analysis.
result ADMMiRNN achieves convergent results and outperforms baselines.
In AI research and industry, machine learning is the most widely used tool. One of the most important machine learning algorithms is Gradient Boosting Decision Tree, i.e. GBDT whose training process needs considerable computational resources and time. To shorten GBDT training time, many works tried to apply GBDT on Par…
Cyclic Data Parallelism reduces memory usage and balances gradient communications.
problem Training large deep learning models requires efficient parallelism to scale.
method Cyclic Data Parallelism shifts micro-batches from simultaneous to sequential execution, balancing memory and gradient communications.
result Cyclic Data Parallelism reduces total memory usage and balances gradient communications.
SySCD improves SCD scalability and speeds up training.
problem Scalability issues in parallel SCD algorithms.
method Developed a system-aware parallel SCD algorithm (SySCD) to avoid bottlenecks.
result Offers up to x42 speedup compared to state-of-the-art GLM solvers.
Parallelizes autoregressive generation using VSSM.
problem Autoregressive models' inability to parallelize generation.
method Variational SSM (VSSM) with parallelizable sampling and decoding.
result Parallel generation possible with VSSM.
Ouroboros accelerates training of large Transformer models.
problem Training large Transformer-based language models is slow and resource-intensive.
method Proposes a novel model-parallel algorithm to speed up training.
result Demonstrates significantly faster training speed compared to data parallelism.
AgEBO-Tabular combines NAS and hyperparameter tuning for fast, high-performing tabular models.
problem Developing high-performing predictive models for large tabular data sets is challenging.
method Combines aging evolution NAS and asynchronous Bayesian optimization for hyperparameter tuning in data-parallel training.
result Automatically discovered neural network models outperform state-of-the-art AutoML ensembles in inference speed by two orders of magnitude.
Proposes NUQSGD for efficient parallel training of large models.
problem Efficiently compressing gradients for parallel SGD training.
method Nonuniform quantization scheme for improved theoretical and empirical performance.
result NUQSGD outperforms QSGDinf and other compression methods.
We parallelize backpropagation for deep learning models, achieving significant speedups.
problem Sequential dependency in backpropagation limits scalability on parallel systems.
method Reformulated backpropagation as a scan operation, using Blelloch scan algorithm.
result Up to 2.75x speedup on overall training time and 108x on backward pass.
Mesh-TensorFlow enables efficient deep learning on large clusters.
problem Memory constraints and inefficiency in batch-splitting for large models.
method Introduces Mesh-TensorFlow for specifying general tensor computations across a multi-dimensional mesh of processors.
result Trains Transformer models with up to 5 billion parameters on TPU meshes of up to 512 cores.
We consider learning problems over training sets in which both, the number of training examples and the dimension of the feature vectors, are large. To solve these problems we propose the random parallel stochastic algorithm (RAPSA). We call the algorithm random parallel because it utilizes multiple parallel processors…
This work improves ASR noise robustness using parallel data and T/S learning.
problem Noise robustness in automatic speech recognition.
method Teacher-student learning with parallel clean and noisy data, logits selection.
result Best student model yields significant WER reductions in noisy conditions.
A new gradient quantization scheme improves communication efficiency in distributed training.
problem Efficiently compressing gradients for parallel training of large models.
method Proposes a new gradient quantization scheme with theoretical guarantees and empirical performance.
result The new scheme matches and exceeds the performance of existing methods.
The paper introduces a multilevel initialization method for deep neural networks.
problem Training very deep neural networks with layer-parallel methods.
method Continuous interpretation of training as optimal control, using time-dependent ODEs for neural network discretization, and a refinement strategy across the time domain.
result The method creates deep networks with good initializations from coarser networks, reducing training time and providing regularization.
EP-GFlowNets parallelize GFlowNet training for large-scale Bayesian inference.
problem Prohibitive repeated evaluations of unnormalized distributions for large-scale posterior sampling.
method Divide-and-conquer approach with server learning from local models.
result EP-GFlowNets enable efficient parallel and federated Bayesian inference.
Neural networks compress uninformative input directions, improving test error.
problem Data lie in a high-dimensional space but labels vary along a lower-dimensional manifold.
method One-hidden layer network trained with gradient descent, analyzing weight evolution and compression.
result Compression factor λ ∼ √p improves test error, with β Feature > β Lazy.
PipeDream-2BW accelerates large model training by 20x with minimal memory usage.
problem Training large models requires memory beyond single accelerator capacity.
method Pipeline parallelism, weight gradient coalescing, double buffering.
result Accelerates large model training by up to 20x.
Training examples are not all equally informative. Active learning strategies leverage this observation in order to massively reduce the number of examples that need to be labeled. We leverage the same observation to build a generic strategy for parallelizing learning algorithms. This strategy is effective because the …
ZeRO optimizes memory for training large models, scaling to trillions of parameters.
problem Training models with billions to trillions of parameters is challenging due to limited device memory.
method ZeRO eliminates memory redundancies in data- and model-parallel training, scaling model size proportional to the number of devices.
result ZeRO trains models of up to 13B parameters without model parallelism, achieving super-linear speedup and throughput of 15 Petaflops.
Improves parallel deep model performance by restructuring and pruning.
problem Latency in parallel deep model execution due to interdependency among sub-models.
method Layer-wise model restructuring and pruning, using ℓ0 optimization and Munkres assignment algorithm. result Significantly improves efficiency of distributed inference in terms of communication and computational complexity.
New algorithms reduce communication in GNN training.
problem Higher communication costs in GNNs due to sparse connectivity.
method Parallel algorithms for sparse-dense matrix multiplication.
result Asymptotic reduction in communication compared to previous methods.
We propose a flexible framework for spectral conversion (SC) that facilitates training with unaligned corpora. Many SC frameworks require parallel corpora, phonetic alignments, or explicit frame-wise correspondence for learning conversion functions or for synthesizing a target spectrum with the aid of alignments. Howev…
Artificial Neural Networks (ANNs) have received increasing attention in recent years with applications that span a wide range of disciplines including vital domains such as medicine, network security and autonomous transportation. However, neural network architectures are becoming increasingly complex and with an incre…
Adaptive quantization improves SGD accuracy in data-parallel settings.
problem Fixed gradient quantization schemes lead to suboptimal performance in deep learning.
method Developed adaptive quantization schemes ALQ and AMQ that update compression schemes based on gradient statistics.
result Improved validation accuracy on CIFAR-10 and ImageNet datasets by 2% and 1% respectively.
Efficiently optimizes orthogonal and Stiefel matrices on parallel units.
problem Optimization over orthogonal groups on parallel units.
method CWY and T-CWY transforms for parametrization and optimization.
result CWY and T-CWY methods lead to convergence on parallel units.
DSSP improves deep learning training speed by dynamically adjusting staleness thresholds.
problem Time-consuming deep learning training on large datasets.
method Dynamic Stale Synchronous Parallel (DSSP) framework that adapts staleness threshold at runtime.
result DSSP converges faster and achieves higher accuracy than other paradigms.