TensorOpt finds optimal parallelization strategies for DNN training.
problem Finding efficient parallelization strategies for DNN training.
method TensorOpt uses an algorithm (FT) to search for an optimal set of parallelization strategies considering multiple objectives.
result TensorOpt provides accurate runtime cost estimation and adapts to resource availability.
Parallelizes feedforward computation using nonlinear equation solving.
problem Sequential nature of feedforward computation limits parallelization.
method Frame feedforward computation as solving nonlinear equations; use Jacobi or Gauss-Seidel methods for parallel updates.
result Accelerates feedforward computation with reduced parallelizable iterations.
Paper speeds up DGD by allowing multiple transmissions from non-straggling servers.
problem Limited parallel execution due to straggling servers in DGD.
method Allowing multiple transmissions from each CS at each iteration.
result Significant reduction in average completion time per iteration.
Speeds up deep neural networks training by 10x using GPU concurrency.
problem Training deep residual neural networks efficiently.
method Layer-wise parallel training with GPU concurrency and Nonlinear Multigrid.
result 10.2x speedup over traditional techniques.
Improves parallel deep model performance by restructuring and pruning.
problem Latency in parallel deep model execution due to interdependency among sub-models.
method Layer-wise model restructuring and pruning, using ℓ0 optimization and Munkres assignment algorithm. result Significantly improves efficiency of distributed inference in terms of communication and computational complexity.
New approach predicts deep learning training times more accurately.
problem Inaccurate prediction of deep learning training times.
method Trains a deep learning network to predict execution time for parts of a network.
result Can predict execution times for unseen scenarios and hardware choices.
This paper optimizes deep neural networks for resource-constrained devices.
problem Efficient deployment of deep neural networks on resource-constrained devices.
method Across-stack optimization of CNNs using weight pruning, channel pruning, quantization, and parallel execution.
result Comprehensive Pareto curves for trade-offs between accuracy, execution time, and memory space.
Deep neural networks, and in particular recurrent networks, are promising candidates to control autonomous agents that interact in real-time with the physical world. However, this requires a seamless integration of temporal features into the network's architecture. For the training of and inference with recurrent neura…
The construction of Mapper has emerged in the last decade as a powerful and effective topological data analysis tool that approximates and generalizes other topological summaries, such as the Reeb graph, the contour tree, split, and joint trees. In this paper, we study the parallel analysis of the construction of Mappe…
Parallel optimization limits are tight for non-smooth convex functions.
problem Limiting parallel acceleration in convex optimization.
method Information-theoretic measure of adaptivity, lower bounds for parallel runtime.
result No randomized algorithm can achieve better convergence rates than a one-query-per-round algorithm with adaptivity better than o(n1/3). Adds recursion to deep learning frameworks for better handling of recursive data structures.
problem Lack of support for recursion in existing deep learning frameworks.
method Complements existing frameworks with recursive execution of dataflow graphs and APIs for recursive definitions.
result Recursive implementation reduces training and inference time by more effectively using resources.
Accelerator synthesizes deep learning inference from C code using FPGA hardware.
problem Efficiently implementing deep learning models on FPGAs.
method Parallelized C code using Pthreads, high-level synthesis, reduced precision, zero-weight-skipping.
result Peak performance of 138 effective GOPS on VGG-16 on Intel Arria 10 SoC FPGA.
We present a general framework for accelerating a large class of widely used Markov chain Monte Carlo (MCMC) algorithms. Our approach exploits fast, iterative approximations to the target density to speculatively evaluate many potential future steps of the chain in parallel. The approach can accelerate computation of t…
Cyclic Data Parallelism reduces memory usage and balances gradient communications.
problem Training large deep learning models requires efficient parallelism to scale.
method Cyclic Data Parallelism shifts micro-batches from simultaneous to sequential execution, balancing memory and gradient communications.
result Cyclic Data Parallelism reduces total memory usage and balances gradient communications.
This research optimizes energy consumption forecasting in Puno using parallel computing and ARIMA models.
problem Improving energy consumption forecasting accuracy and efficiency in Puno.
method Parallel computing and ARIMA models for forecasting energy consumption.
result Notable improvements in computational efficiency and data processing capabilities.
This paper optimizes how deep learning models are distributed across different devices.
problem Optimizing how large, complex neural networks are split across multiple devices.
method Identified and solved an optimization problem for device placement of DNN operators.
result Automated algorithms that solve the device placement problem for modern pipelined settings.
DistShap parallelizes GNN explanation for large graphs.
problem Computational expense in attributing GNN predictions to specific edges or features.
method Distributed Shapley values across multiple GPUs for scalable GNN explanations.
result DistShap outperforms existing methods and scales to models with millions of features.
PipeMare enables efficient DNN training with minimal memory and pipeline sacrifices.
problem Sacrificing hardware efficiency to maintain statistical efficiency in pipeline parallel DNN training.
method PipeMare is a simple yet robust training method that tolerates asynchronous updates during pipeline parallelism without sacrificing pipeline utilization or memory.
result PipeMare achieves up to 2.7x less memory usage or 4.3x higher pipeline utilization compared to state-of-the-art synchronous PP training techniques.
We present CYCLADES, a general framework for parallelizing stochastic optimization algorithms in a shared memory setting. CYCLADES is asynchronous during shared model updates, and requires no memory locking mechanisms, similar to HOGWILD!-type algorithms. Unlike HOGWILD!, CYCLADES introduces no conflicts during the par…
The pricing of American style and multiple exercise options is a very challenging problem in mathematical finance. One usually employs a Least-Square Monte Carlo approach (Longstaff-Schwartz method) for the evaluation of conditional expectations which arise in the Backward Dynamic Programming principle for such optimal…
Neural networks learn to execute graph algorithms.
problem Training GNNs to mimic graph algorithm steps.
method Training GNN architectures to imitate graph algorithm steps, including breadth-first search, Bellman-Ford, Prim's algorithm, and reachability.
result Maximization-based message passing neural networks are best-suited for graph algorithm decision-making.
SwitchML accelerates machine learning training by aggregating model updates in the network.
problem Training machine learning models in parallel consumes a lot of data.
method Designing a communication primitive that uses a programmable switch to aggregate model updates.
result SwitchML reduces data exchange by up to 5.5 times, speeding up training.
POLO simplifies parallel optimization research with a flexible library.
problem Ease of implementing and evaluating parallel optimization algorithms.
method Uses policies and template programming for algorithm decomposition and code reuse.
result Allows implementation of state-of-the-art algorithms in minimal code.
KineticSim: A lightweight, high-performance execution engine for real-time market simulators
problem Simulating financial markets at scale with multi-agent models
method Reusable parallel design pattern: persistent, state-carrying clearing for iterative multi-agent reductions
result Reduces per-step critical-path depth from Theta(L+A) to Theta(log L + ceil(A/L))
Sideways trains video models by overwriting activations as new frames arrive, potentially improving generalization.
problem Training deep video models synchronously slows down and requires storing activations, limiting parallelism.
method Sideways trains video models by overwriting activations as new frames arrive, breaking the precise correspondence between gradients and activations.
result Sideways training can converge and potentially generalize better than standard synchronized backpropagation.
KineticSim accelerates financial market simulations 3406x over CPU.
problem Simulating financial markets at scale with multi-agent models is bottlenecked by sequential processing and GPU kernel overhead.
method Formalized and implemented a reusable parallel design pattern for iterative multi-agent reductions in thread-block shared memory.
result Achieved a peak throughput of over 54.7 billion agent-events per second, delivering 3406x speedup over CPU.
Training large machine learning (ML) models with many variables or parameters can take a long time if one employs sequential procedures even with stochastic updates. A natural solution is to turn to distributed computing on a cluster; however, naive, unstructured parallelization of ML algorithms does not usually lead t…
New technique reduces memory usage and boosts neural network training speed.
problem Training large neural networks requires significant memory and computational resources.
method L2L (layer-to-layer) execution technique with eager param-server (EPS) and micro-batching.
result 45% reduction in memory usage and 40% increase in throughput for BERT-Large.
This paper uses supervised learning to predict optimal chunk-size for parallel linear algebra operations.
problem Finding the optimal chunk-size for parallel linear algebra operations.
method The paper uses supervised learning models (logistic regression, neural networks, decision trees) to predict the optimal chunk-size for multiple linear algebra operations.
result The custom decision tree model outperforms classical decision trees and other models in predicting optimal chunk-size for linear algebra operations.
Benanza speeds up DL model optimization by automatically generating micro-benchmarks and identifying inefficiencies.
problem Slow characterization/optimization cycles for DL models on GPUs.
method Benanza includes a model processor, benchmark generator, database, and analyzer.
result Benanza identifies optimizations in parallel layer execution, cuDNN, framework inefficiency, layer fusion, and Tensor Cores.
The emergence and development of cancer is a consequence of the accumulation over time of genomic mutations involving a specific set of genes, which provides the cancer clones with a functional selective advantage. In this work, we model the order of accumulation of such mutations during the progression, which eventual…
Tasks like code generation and semantic parsing require mapping unstructured (or partially structured) inputs to well-formed, executable outputs. We introduce abstract syntax networks, a modeling framework for these problems. The outputs are represented as abstract syntax trees (ASTs) and constructed by a decoder with …
Analog arrays speed up ConvNets by parallelizing kernel matrix training.
problem Early ConvNets struggle with analog arrays due to small kernel matrices.
method Replicate kernel matrix on multiple analog arrays, training in parallel.
result Analog arrays achieve high acceleration factors (16-128) for ConvNets.
New method improves speech synthesis quality.
problem Efficiently train parallel speech synthesis models.
method Spectral energy distance for implicit generative models.
result State-of-the-art generation quality achieved.
LLMs improve parent-order execution in trading.
problem Improving execution costs in algorithmic trading.
method PACE (Plan-Ahead Controlled Execution) framework.
result LLMs outperform existing methods by 0.65 bps.
LEMs extend transformer-based architectures for complex execution problems.
problem Handling flexible time boundaries and multiple execution constraints in deep learning.
method Decouples market information processing from execution allocation decisions using TKANs, VSNs, and multi-head attention mechanisms.
result LEMs achieve superior execution performance compared to traditional benchmarks.
The next generation of AI applications will continuously interact with the environment and learn from these interactions. These applications impose new and demanding systems requirements, both in terms of performance and flexibility. In this paper, we consider these requirements and present Ray---a distributed system t…
Chicle tackles elastic machine learning training by avoiding micro-tasks.
problem Elasticity and load balancing in distributed machine learning training.
method Chicle is a new elastic distributed training framework that exploits machine learning algorithms to implement elasticity and load balancing without micro-tasks.
result Chicle achieves performance competitive with state-of-the-art rigid frameworks while enabling elastic execution and dynamic load balancing.
Proposes a method to allocate time budgets in mixed criticality systems.
problem Managing execution time variability in mixed criticality systems.
method Quantifies execution time variability using statistical dispersion parameters and proposes a heuristic to allocate time budgets.
result The proposed heuristic reduces the probability of exceeding allocated budgets.
Optimal crypto order execution using cross-exchange signals.
problem Maximizing order execution efficiency in cryptocurrency markets.
method Reinforcement learning applied to cross-exchange data.
result Cross-exchange signals improve optimal execution outcomes.
The paper uses machine learning to detect malicious executable files.
problem Detecting malicious executable files using static analysis.
method Pre-processing, cleaning, encoding, feature selection, and ensemble training of classifiers.
result An ensemble of classifiers effectively detects malicious executable files.
The paper analyzes trade execution strategies for large traders in a stochastic market environment.
problem Analyzing trade execution strategies in a stochastic market with price impact.
method Formulated a Markov game model and used backward induction method of dynamic programming.
result Explicit closed-form execution strategy at Markov perfect equilibrium.
Optimizes stock execution costs using stochastic control theory.
problem Minimizing execution costs in a market with discrete stock price movements.
method Discrete-time Stochastic Control Theory applied to a market model.
result Developed optimal allocation strategy for executing K stocks within T units.
Model for optimal execution with passive market impact.
problem Optimizing execution strategies in markets with passive price impact.
method Developed a mesoscopic model incorporating empirical price impact features.
result Obtained a passive impact rate that decays exponentially with quote distance.
Risk-averse trading policies learned from simulated market interactions.
problem Minimizing execution cost in limit order book markets with market impact.
method Risk-sensitive Q-learning applied to Markov Decision Process in a market simulator.
result Derived decision-tree-based execution policies that minimize cost variance.
What is a systematic way to efficiently apply a wide spectrum of advanced ML programs to industrial scale problems, using Big Models (up to 100s of billions of parameters) on Big Data (up to terabytes or petabytes)? Modern parallelization strategies employ fine-grained operations and scheduling beyond the classic bulk-…
Optimized execution model using interbank and internal liquidity.
problem Minimizing market impact in trading.
method Integrates interbank limit and market orders with internal market-making liquidity.
result Reduces market impact and improves execution efficiency.
Paper analyzes the cost and execution of limit orders in a market with random walk price behavior.
problem Cost and execution of limit orders in markets with random walk price behavior.
method Exact solution for the cost of static passive slice execution, derivation of risk and execution probability functions.
result No optimal limit level for order execution in a market with random walk price behavior.