Pipeline parallelism (PP) when training neural networks enables larger models to be partitioned spatially, leading to both lower network communication and overall higher hardware utilization. Unfortunately, to preserve the statistical efficiency of sequential training, existing PP techniques sacrifice hardware efficien…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,694 papers · 148 categories
Trend · papers per month
48 results for “pipeline parallelism”
Clapping reduces memory usage in distributed optimization by reusing data samples.
problem Significant communication overhead and impractical memory overhead in pipeline-parallel distributed optimization.
method Lazy sampling strategy to reuse data samples across steps, supporting convergence without unbiased gradient assumptions.
result Clapping achieves convergence in few-epoch or online training regimes without sample-size memory overhead.
PipeDream-2BW accelerates large model training by 20x with minimal memory usage.
problem Training large models requires memory beyond single accelerator capacity.
method Pipeline parallelism, weight gradient coalescing, double buffering.
result Accelerates large model training by up to 20x.
Theoretical Limits of Pipeline Parallel Optimization and Application to Distributed Deep Learningstat.ML
We investigate the theoretical limits of pipeline parallel learning of deep learning architectures, a distributed setup in which the computation is distributed per layer instead of per example. For smooth convex and non-convex objective functions, we provide matching lower and upper complexity bounds and show that a na…
OpTorch optimizes deep learning for resource-limited environments.
problem Resource constraints in deep learning training.
method Optimized deep learning pipelines in training time and memory.
result Achieved similar accuracy to existing libraries with reduced memory usage.
Pipelined Backpropagation trains large models without batches efficiently.
problem Training large models efficiently on hardware with limited batch sizes.
method Fine-grained Pipelined Backpropagation with Spike Compensation and Linear Weight Prediction.
result Fine-grained Pipelined Backpropagation with a batch size of one matches the accuracy of SGD for multiple networks.
This paper optimizes how deep learning models are distributed across different devices.
problem Optimizing how large, complex neural networks are split across multiple devices.
method Identified and solved an optimization problem for device placement of DNN operators.
result Automated algorithms that solve the device placement problem for modern pipelined settings.
Predicting the outcome of sales opportunities is a core part of successful business management. Conventionally, making this prediction has relied mostly on subjective human evaluations in the process of sales decision making. In this paper, we addressed the problem of forecasting the outcome of business to business (B2…
Mango automates hyperparameter tuning for large-scale ML training.
problem Manual hyperparameter tuning is tedious and inefficient for large-scale machine learning.
method Parallel hyperparameter tuning with intelligent search strategies and flexible abstractions.
result Mango achieves comparable performance to Hyperopt while supporting distributed computing.
DreamPropeller accelerates text-to-3D generation by 4.7x with minimal loss in quality.
problem Long generation times in text-to-3D generation algorithms degrade user experience.
method DreamPropeller uses Picard iterations generalized for non-ODE paths to accelerate parallel sampling.
result Empirically achieves up to 4.7x speedup with negligible quality loss.
Mozilla Research is developing Servo, a parallel web browser engine, to exploit the benefits of parallelism and concurrency in the web rendering pipeline. Parallelization results in improved performance for pinterest.com but not for google.com. This is because the workload of a browser is dependent on the web page it i…
New algorithms reduce communication in GNN training.
problem Higher communication costs in GNNs due to sparse connectivity.
method Parallel algorithms for sparse-dense matrix multiplication.
result Asymptotic reduction in communication compared to previous methods.
New approach speeds up optimization with repeated gradient steps on same batch.
problem Performance bottlenecks in massive parallel pipelines with large batch sizes.
method Data echoing, taking repeated gradient steps on the same batch.
result Data echoing affords speedups on curvature-dominated part of convergence rate.
TSML tackles anomaly detection and pattern discovery in industrial time series data.
problem Extracting and exploiting information from large industrial data to reduce downtimes and manufacturing errors.
method TSML uses a pipeline of lightweight filters to process industrial time series data in parallel.
result TSML effectively detects anomalies and discovers patterns in industrial time series data.
We present an LDA approach to entity disambiguation. Each topic is associated with a Wikipedia article and topics generate either content words or entity mentions. Training such models is challenging because of the topic and vocabulary size, both in the millions. We tackle these problems using a novel distributed infer…
NEST optimizes deep learning training by placing devices efficiently across networks and memory.
problem Inefficient device placement in distributed deep learning leads to high communication and memory overhead.
method NEST uses network-, compute-, and memory-aware dynamic programming to optimize device placement.
result NEST achieves up to 2.43 times higher throughput and better memory efficiency.
In this paper we study the real time deployment of deep learning algorithms in low resource computational environments. As the use case, we compare the accuracy and speed of neural networks for smile detection using different neural network architectures and their system level implementation on NVidia Jetson embedded p…
Deep Neural Networks (DNNs) have revolutionized numerous applications, but the demand for ever more performance remains unabated. Scaling DNN computations to larger clusters is generally done by distributing tasks in batch mode using methods such as distributed synchronous SGD. Among the issues with this approach is th…
This paper presents our contribution to PolEval 2019 Task 6: Hate speech and bullying detection. We describe three parallel approaches that we followed: fine-tuning a pre-trained ULMFiT model to our classification task, fine-tuning a pre-trained BERT model to our classification task, and using the TPOT library to find …
The evaluation of machine learning (ML) pipelines is essential during automatic ML pipeline composition and optimisation. The previous methods such as Bayesian-based and genetic-based optimisation, which are implemented in Auto-Weka, Auto-sklearn and TPOT, evaluate pipelines by executing them. Therefore, the pipeline c…
Deterministic Inference across Tensor Parallel Sizes That Eliminates Training-Inference Mismatchcs.LG
New method ensures consistent inference across different tensor parallel sizes for large language models.
problem Non-deterministic inference in large language models due to inconsistent reduction orders across GPUs.
method Tree-Based Invariant Kernels (TBIK) that align intra- and inter-GPU reduction orders through a unified hierarchical binary tree structure.
result Bit-wise identical results across different tensor parallel sizes for RL training.
Hyperparameter tuning of multi-stage pipelines introduces a significant computational burden. Motivated by the observation that work can be reused across pipelines if the intermediate computations are the same, we propose a pipeline-aware approach to hyperparameter tuning. Our approach optimizes both the design and exe…
Classical Machine Learning (ML) pipelines often comprise of multiple ML models where models, within a pipeline, are trained in isolation. Conversely, when training neural network models, layers composing the neural models are simultaneously trained using backpropagation. We argue that the isolated training scheme of ML…
Study proposes a statistical testing framework for evaluating clustering pipelines.
problem Quantifying the statistical reliability of clustering results from data analysis pipelines.
method Selective inference-based statistical testing framework for clustering pipelines.
result The proposed test controls the type I error rate and is effective in validating clustering results.
Investigates fairness in pipeline models where individuals may drop out.
problem Fairness in pipeline models where individuals may drop out and subsequent stages depend on remaining individuals.
method Rigorous framework for evaluating fairness guarantees, showing that naïve auditing is insufficient and dependence must exist between stages.
result Fairness in pipelines can be arbitrary, even with just two stages, and requires dependence between stages.
Data science relies on pipelines that are organized in the form of interdependent computational steps. Each step consists of various candidate algorithms that maybe used for performing a particular function. Each algorithm consists of several hyperparameters. Algorithms and hyperparameters must be optimized as a whole …
Backpropagation (BP) is the cornerstone of today's deep learning algorithms, but it is inefficient partially because of backward locking, which means updating the weights of one layer locks the weight updates in the other layers. Consequently, it is challenging to apply parallel computing or a pipeline structure to upd…
Automates supervised learning pipeline design with matrix and tensor factorization.
problem Designing effective supervised learning pipelines with many choices.
method Uses matrix and tensor factorization to model pipeline search space and develops greedy experiment design protocols.
result Demonstrates the effectiveness of the approach on real-world classification problems.
Data science relies on pipelines that are organized in the form of interdependent computational steps. Each step consists of various candidate algorithms that maybe used for performing a particular function. Each algorithm consists of several hyperparameters. Algorithms and hyperparameters must be optimized as a whole …
New model optimizes oil product distribution via pipelines.
problem Optimizing oil product distribution via pipelines.
method Discrete-time mixed integer linear programming model.
result Significant reductions in pipeline operational cost.
Pipeline decomposes portfolio optimization problems into smaller, solvable subproblems.
problem Large-scale portfolio optimization with constraints.
method Decomposition pipeline with preprocessing, clustering, and risk rebalancing.
result Pipeline reduces problem size by 80% and computation time.
Paper proposes a statistical test for feature selection pipelines using selective inference.
problem Assessing the significance of feature selection pipelines in data analysis.
method Selective inference technique applied to feature selection pipelines composed of various algorithms.
result The proposed statistical test controls false positive feature selection probabilities.
Machine Learning (ML) and Deep Learning (DL) innovations are being introduced at such a rapid pace that researchers are hard-pressed to analyze and study them. The complicated procedures for evaluating innovations, along with the lack of standard and efficient ways of specifying and provisioning ML/DL evaluation, is a …
Multilayer Network Analysis of the Drug Pipeline in the Global Pharmaceutical Industryphysics.soc-ph
Study reveals similarities in knowledge flows between pharmaceutical and AI industries.
problem Understanding the dynamics of drug pipelines in global pharmaceutical industry.
method Multilayer network analysis of drug pipeline, global supply chain, and ownership data.
result Proven similarities in knowledge flows between pharmaceutical and AI industries.
Modyn automates continuous ML model training on growing datasets.
problem Continuous model retraining is costly and impractical.
method Data-centric ML platform with policies for continuous training.
result Modyn enables high throughput training with sample-level data selection.
This research investigates the Statistical Machine Translation approaches to translate speech in real time automatically. Such systems can be used in a pipeline with speech recognition and synthesis software in order to produce a real-time voice communication system between foreigners. We obtained three main data sets …
TODS automates time series outlier detection with customizable pipelines.
problem Automated detection of outliers in time series data.
method Modular system with 70 primitives for data processing, time series analysis, and detection algorithms. GUI and data-driven searcher for pipeline design.
result Automated discovery and construction of effective outlier detection pipelines.
Hungarian text processing improved with efficient, accurate NLP pipelines.
problem Improving text processing for Hungarian language.
method Implemented in spaCy framework, focusing on efficiency and accuracy.
result Near state-of-the-art performance in all text preprocessing steps.
Deep Modulation Embeddingeess.SP
Deep neural network has recently shown very promising applications in different research directions and attracted the industry attention as well. Although the idea was introduced in the past but just recently the main limitation of using this class of algorithms is solved by enabling parallel computing on GPU hardware.…
A rigorous ML pipeline for binary classification in biomedical studies, focusing on pancreatic cancer.
problem Handling bias in ML models for complex biomedical data.
method Customizable ML analysis pipeline with 9 algorithms, hyperparameter optimization, and thorough evaluation.
result Comparison of ML algorithms to ExSTraCS, highlighting interpretability and bias handling.
Pipeline for comparing trading algorithms in finance and crypto.
problem Disconnected research and applications in algorithmic trading.
method General pipeline for designing, programming, and evaluating trading strategies.
result Systematic comparison of trading algorithms in finance and crypto.
RL pipeline simplifies knot diagrams, including very hard unknots.
problem Simplifying complex knot diagrams, especially very hard unknots.
method Reinforcement learning for move proposals and heuristic navigation of Reidemeister moves.
result Trained agent simplifies diagrams, including a link to a three-step unknotting process.
This paper explores how LLMs can improve pipeline-based conversational agents.
problem Limitations of pipeline-based conversational agents in human-like conversations.
method Investigated LLMs' capabilities in two phases: design and development, and operations.
result LLMs can enhance pipeline-based agents in various tasks like data generation, intent classification, and auto-correction.
Much of the work in metalearning has focused on classifier selection, combined more recently with hyperparameter optimization, with little concern for data preprocessing. Yet, it is generally well accepted that machine learning applications require not only model building, but also data preprocessing. In other words, p…
Pipeline selects best RL policy from limited data.
problem Selecting best RL policy from small datasets.
method Task- and method-agnostic pipeline using multiple data splits.
result Pipeline produces higher-performing policies.
Machine learning pipeline potentially consists of several stages of operations like data preprocessing, feature engineering and machine learning model training. Each operation has a set of hyper-parameters, which can become irrelevant for the pipeline when the operation is not selected. This gives rise to a hierarchica…
Machine learning tasks entail the use of complex computational pipelines to reach quantitative and qualitative conclusions. If some of the activities in a pipeline produce erroneous or uninformative outputs, the pipeline may fail or produce incorrect results. Inferring the root cause of failures and unexpected behavior…
We analyze the Hessian spectra of large models up to 100B parameters.
problem Accurate Hessian spectra of large foundation models are difficult to obtain.
method We use shard-local finite-difference Hessian vector products and stochastic Lanczos quadrature.
result We produce the first large-scale spectral density estimates of foundation models.