Poseidon optimizes deep learning training on GPU clusters by reducing network communication.
problem Substantial parameter synchronization over the network in distributed DL implementations.
method Overlap communication and computation, use a hybrid communication scheme.
result Achieves significant speed-ups in DL training on GPU clusters.
cuSLINK clusters data faster on GPUs, saving space and time.
problem Efficiently clustering large datasets on GPUs.
method Reformulated SLINK algorithm on GPU with optimized building blocks.
result cuSLINK uses O(Nk) space and trades off space and time with parameter k. New framework quantifies uncertainty in flexible density-based clustering.
problem Uncertainty quantification in clustering with non-parametric density estimation.
method Martingale posterior distributions and density-based clustering.
result Efficient GPU-compatible inference on clustering structures with uncertainty.
We implement a master-slave parallel genetic algorithm (PGA) with a bespoke log-likelihood fitness function to identify emergent clusters within price evolutions. We use graphics processing units (GPUs) to implement a PGA and visualise the results using disjoint minimal spanning trees (MSTs). We demonstrate that our GP…
Paper proposes a GPU-based system for training massive deep learning models in ads systems.
problem Training massive deep learning models with terabyte-scale parameters in ads systems.
method Hierarchical GPU parameter server with 3-layer storage (GPU High-Bandwidth Memory, CPU main memory, SSD).
result 4-node hierarchical GPU parameter server trains a model 2X faster than a 150-node in-memory system.
CuLE accelerates reinforcement learning on GPUs.
problem Limited CPU-GPU communication bandwidth in Atari Learning Environment.
method CUDA port of ALE, leveraging GPU parallelization and direct frame rendering.
result Up to 155M frames per hour on a single GPU, accelerating reinforcement learning.
aweSOM accelerates SOM clustering for large datasets.
problem Scalability issues in existing SOM implementations for large, multidimensional data.
method CPU/GPU-accelerated Self-organizing Maps (SOM) with ensemble stacking.
result 10-100x speed up and improved memory efficiency for large datasets.
torchsom simplifies SOMs in PyTorch with GPU acceleration and scikit-learn API.
problem Efficient implementation and usability of SOMs in PyTorch.
method PyTorch backend, GPU acceleration, scikit-learn API, 90% test coverage.
result Ease of use and scalability for SOMs in PyTorch.
Our system trains ImageNet in 6.6 minutes using 2048 Tesla P40 GPUs.
problem Training large-scale deep neural networks efficiently and accurately.
method Mixed-precision training, extremely large mini-batch size optimization, and optimized all-reduce algorithms.
result Trains ResNet-50 to 75.8% top-1 test accuracy in 6.6 minutes.
Synaptic cluster-driven evolution improves deep neural networks by reducing synapses and clusters.
problem Efficiently synthesizing deep neural networks with fewer synapses and clusters.
method Synaptic cluster-driven genetic encoding scheme.
result Significantly smaller number of synapses and clusters in offspring networks.
New algorithms reduce communication in GNN training.
problem Higher communication costs in GNNs due to sparse connectivity.
method Parallel algorithms for sparse-dense matrix multiplication.
result Asymptotic reduction in communication compared to previous methods.
SaberLDA learns a large number of topics from big text data on GPUs.
problem Handling large number of topics with existing GPU-based LDA systems.
method Sparsity-aware algorithm, novel data layout, warp-based sampling kernel, sparse count matrix updating.
result SaberLDA learns up to 10,000 topics from billions of tokens in a few hours.
NetFuse merges different DNN models with varying weights for faster inference.
problem Inference speed of DNN models with different weights cannot be improved using existing techniques.
method NetFuse merges models with the same architecture but different weights and inputs, replacing operations with more general ones.
result NetFuse can speed up DNN inference time up to 3.6x on a NVIDIA V100 GPU.
End-to-end speech recognition system trained on GPUs and CPUs.
problem Building state-of-the-art speech recognition systems.
method Utilizes CPUs and GPUs for training, data augmentation, and neural network updates. Uses vocal tract length perturbation and acoustic simulator for data augmentation. Employed Horovod allreduce for training.
result Achieved 7.92% WER on proprietary English Bixby open domain test set using a Bidirectional Full Attention (BFA) model.
74.7 seconds to train ResNet-50 on ImageNet with 2,048 GPUs.
problem Training deep learning models on large mini-batches efficiently.
method Optimization methods applied to distributed deep learning.
result Top-1 validation accuracy of 75.08% with training time of 74.7 seconds.
New software package for scalable DPMM inference on large datasets.
problem Scalability and practical adoption of Dirichlet Process Mixture Models.
method Efficient distributed sampling-based inference on CPUs and GPUs.
result Significant speedups and fitting of larger datasets.
DSSP improves deep learning training speed by dynamically adjusting staleness thresholds.
problem Time-consuming deep learning training on large datasets.
method Dynamic Stale Synchronous Parallel (DSSP) framework that adapts staleness threshold at runtime.
result DSSP converges faster and achieves higher accuracy than other paradigms.
This paper surveys distributed training techniques for deep learning models.
problem Substantial compute needed for training deep learning models.
method Exploration of various algorithms and techniques for distributed training.
result Recent advancements have reduced training time from weeks to minutes.
Efficiently trains BERT on academic GPUs in 12 days.
problem Training large-scale BERT models is expensive and time-consuming.
method Optimizes training on multiple GPUs and nodes, reducing costs.
result Trains BERT on academic GPUs in 12 days, not requiring expensive hardware.
Stage-based hyper-parameter optimization reduces GPU-hours and training time.
problem Efficiently executing hyper-parameter optimization for deep learning models.
method Stage-based execution strategy to remove redundant computations.
result Stage-based execution outperforms trial-based method by up to 6.60 times in GPU-hours and 4.13 times in training time.
A novel method clusters protein conformations from MD simulations.
problem Clustering long MD protein dynamics for identifying states and behavior.
method Adversarial Autoencoder (AAE) for conformation clustering.
result Identifies many salient features of the folding process.
SpinSVAR estimates SVAR models with sparse input, improving accuracy and scalability.
problem Estimating SVAR models with sparse input assumptions.
method SpinSVAR models input as independent Laplacian variables, enforcing sparsity and using least absolute error regression.
result SpinSVAR outperforms state-of-the-art methods in accuracy and runtime, identifying significant structural shocks.
There has been significant recent interest towards achieving highly efficient deep neural network architectures. A promising paradigm for achieving this is the concept of evolutionary deep intelligence, which attempts to mimic biological evolution processes to synthesize highly-efficient deep neural networks over succe…
New GPU algorithm boosts machine learning with larger datasets.
problem Limited GPU memory restricts training data size.
method Out-of-core GPU gradient boosting algorithm.
result Training larger datasets on GPUs without accuracy loss.
Massively parallel architectures such as the GPU are becoming increasingly important due to the recent proliferation of data. In this paper, we propose a key class of hybrid parallel graphlet algorithms that leverages multiple CPUs and GPUs simultaneously for computing k-vertex induced subgraph statistics (called graph…
New framework for efficient PD averaging and clustering.
problem Challenges in averaging and clustering persistence diagrams.
method Reformulate PD metrics as optimal transport problems, leveraging recent computational advances.
result Scalable computations of PD barycenters and clustering on thousands of diagrams.
Given a multivariate data set, sparse principal component analysis (SPCA) aims to extract several linear combinations of the variables that together explain the variance in the data as much as possible, while controlling the number of nonzero loadings in these combinations. In this paper we consider 8 different optimiz…
EBIC is a biclustering tool for big genomic data, achieving significant speedup.
problem Mining genetic data for high-dimensional and big data challenges.
method EBIC is a biclustering algorithm enhanced for big data, including support for missing values and integration with R.
result EBIC achieves over 6.6 fold speedup on large datasets, demonstrating high scalability.
XGBoost accelerates machine learning on GPUs.
problem Training large datasets efficiently on GPUs.
method Multi-GPU gradient boosting with data compression and end-to-end GPU parallelism.
result Processed 115 million instances in 3 minutes.
A new method for efficient kernel k-means clustering.
problem Inefficiency of kernel k-means due to memory scaling issues.
method Approximate strategy and parallelization for large-scale clustering.
result Improved efficiency and scalability of kernel k-means.
Extends GENO framework for GPU optimization of constrained ML problems.
problem Constrained optimization in classical machine learning.
method Extends GENO framework to GPU optimization, specifying problems in a modeling language.
result Solvers on GPU outperform state-of-the-art approaches by several orders of magnitude.
GPU computing has become popular in computational finance and many financial institutions are moving their CPU based applications to the GPU platform. Since most Monte Carlo algorithms are embarrassingly parallel, they benefit greatly from parallel implementations, and consequently Monte Carlo has become a focal point …
GPU acceleration for tree boosting improves speed and scalability.
problem Scalability and performance issues in GPU-based tree building algorithms.
method Histogram-based algorithm for approximate split finding on GPUs.
result 7-8 times faster training on CPU and 25 times faster on Xeon server.
GPU speeds up derivatives sensitivity computation for Heston model options.
problem Efficient computation of option Greeks under the Heston model.
method Implemented exact simulation and novel Milstein discretisation methods on GPU.
result GPU speeds up Greeks computation up to 200x compared to CPU methods.
New approach splits deep models for parallel training.
problem Training deep models requires significant computing resources.
method Split the model into parts and train them separately.
result High-performance model-parallel training achieved.
JAXFit speeds up curve fitting on GPUs.
problem Nonlinear least squares curve fitting problems.
method Trust region method on GPU with automatic differentiation.
result Significantly faster than CPU and other GPU libraries.
Horovod simplifies multi-GPU training in TensorFlow.
problem Efficient multi-GPU training in TensorFlow with minimal code changes.
method Efficient inter-GPU communication via ring reduction and minimal code modifications.
result Faster, easier distributed training in TensorFlow.
GPU optimization speeds up large-scale classification tasks.
problem Efficiently training large-scale classification models on GPUs.
method Judecious GPU-optimization principles applied to TRON algorithm.
result Significant speedups for logistic regression and SVM classification.
A GPU framework speeds up BnB for discrete optimization problems.
problem Optimizing large-scale discrete problems with GPU limitations.
method Parallel BnB nodes in GPU batches, using padding and custom kernels.
result One to two orders of magnitude speedup and zero optimality gap.
Echo optimizes LSTM RNN training on GPUs by reducing memory footprint.
problem Memory bottleneck in LSTM RNN training on GPUs.
method Compiler-based feature map recomputation to estimate and balance memory and execution time.
result Average 1.89X reduction in GPU memory footprint.
Speeds up deep neural networks training by 10x using GPU concurrency.
problem Training deep residual neural networks efficiently.
method Layer-wise parallel training with GPU concurrency and Nonlinear Multigrid.
result 10.2x speedup over traditional techniques.
RFX accelerates and compresses Random Forests for large datasets.
problem Memory bottleneck in proximity matrices limits Random Forest analysis.
method QLORA compression, CPU TriBlock storage, GPU batch sizing, 3D MDS visualization.
result Proximity-based Random Forest analysis on larger datasets is feasible.
UMAP speeds up significantly with GPU acceleration.
problem Efficient GPU implementation of UMAP.
method Various techniques to optimize UMAP for GPU, including data partitioning and parallelization.
result Up to 100x speedup in practice.
The realized stochastic volatility (RSV) model that utilizes the realized volatility as additional information has been proposed to infer volatility of financial time series. We consider the Bayesian inference of the RSV model by the Hybrid Monte Carlo (HMC) algorithm. The HMC algorithm can be parallelized and thus per…
MoGA optimizes mobile networks for GPU efficiency.
problem Lack of GPU-aware optimization in mobile network design.
method Proposes MoGA, a GPU-aware neural architecture search.
result Achieves higher accuracy with similar latency constraints.
TorchKM: A GPU-Oriented Library for Kernel Learning and Model Selection
problem Kernel learning and model selection
method GPU acceleration
result Competitive predictive performance with speedups
GPU-optimized ES-RNN boosts time series forecasting speed by 322x.
problem Efficiently forecasting time series data.
method Vectorized GPU implementation of ES-RNN.
result Up to 322x speedup in training time.
New GPU kernels boost deep learning speed and memory efficiency.
problem Sparse deep learning matrices are not well-suited for existing sparse kernels.
method Identified favorable properties of sparse matrices from deep learning, developed high-performance GPU kernels for sparse matrix operations.
result 27% of single-precision peak performance on Nvidia V100 GPUs achieved with new kernels.