Research
On-device research index

arXiv research

A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.

168,742 papers · 148 categories

Trend · papers per month

14284256 · May 202619922001200920172026
48 results for pipeline orchestration

Training machine learning (ML) models on large datasets requires considerable computing power. To speed up training, it is typical to distribute training across several machines, often with specialized hardware like GPUs or TPUs. Managing a distributed training job is complex and requires dealing with resource contenti…

2019-03-24abs ↗pdf ↗

FinHEAR combines LLMs with human expertise for better financial decision-making.

problem Challenges in financial decision-making for language models.
method Multi-agent framework with specialized LLMs for historical analysis, event interpretation, and expert retrieval.
result FinHEAR outperforms baselines in financial tasks with higher accuracy and risk-adjusted returns.

Pioneers a new network slicing solution for beyond-5G networks.

problem Ensuring diverse service level agreements (SLAs) in a multi-tenant environment with varying latency and throughput requirements.
method Leverages a multi-armed-bandit-based (MAB) orchestrator, LACO, that makes adaptive resource slicing decisions based on system structure information.
result Near-optimal results in resource slicing decisions with no prior knowledge of traffic demand or channel quality statistics.

Orchestrating the Twin Transition in GBS: A Socio-Technical Framework

problem MNCs' need to harmonize digital efficiency with environmental stewardship
method Technology Roadmapping (TRM) with ICT-centric innovation ecosystem toolkit
result Central 'operational airlock' for GBS in harmonizing digital and environmental transformation

AGENTICAITA uses AI agents to autonomously trade markets without human intervention.

problem Inability of traditional trading systems to adapt to market complexity.
method Introduces an agentic AI framework with specialized LLM agents reasoning, negotiating, and acting.
result Demonstrated operational correctness and non-trivial inter-agent negotiation in live market conditions.

Photonic co-processor speeds up training of large neural networks.

problem Training large neural networks with backpropagation is inefficient and communication is a bottleneck.
method Direct Feedback Alignment (DFA) with a photonic accelerator.
result Photonic accelerator can compute random projections with trillions of parameters.

Reinforcement learning improves online matching by combining expert policies.

problem Efficient decision-making in complex systems like cloud services and marketplaces.
method Combines reinforcement learning with expert policies, using advantage-based weight updates.
result The orchestrated policy converges faster and yields higher efficiency than individual experts and conventional RL.

AVATAR uses a surrogate model to quickly evaluate ML pipelines, saving time and resources.

problem Time-consuming evaluation of ML pipelines limits exploration of complex models.
method AVATAR employs a surrogate model to assess pipeline validity without execution.
result AVATAR accelerates ML pipeline evaluation, improving efficiency in complex scenarios.

Hyperparameter tuning of multi-stage pipelines introduces a significant computational burden. Motivated by the observation that work can be reused across pipelines if the intermediate computations are the same, we propose a pipeline-aware approach to hyperparameter tuning. Our approach optimizes both the design and exe…

2019-03-12abs ↗pdf ↗

Federated learning (FL) is a machine learning setting where many clients (e.g. mobile devices or whole organizations) collaboratively train a model under the orchestration of a central server (e.g. service provider), while keeping the training data decentralized. FL embodies the principles of focused data collection an…

2019-12-10abs ↗pdf ↗

Study proposes a statistical testing framework for evaluating clustering pipelines.

problem Quantifying the statistical reliability of clustering results from data analysis pipelines.
method Selective inference-based statistical testing framework for clustering pipelines.
result The proposed test controls the type I error rate and is effective in validating clustering results.

Investigates fairness in pipeline models where individuals may drop out.

problem Fairness in pipeline models where individuals may drop out and subsequent stages depend on remaining individuals.
method Rigorous framework for evaluating fairness guarantees, showing that naïve auditing is insufficient and dependence must exist between stages.
result Fairness in pipelines can be arbitrary, even with just two stages, and requires dependence between stages.

Automates supervised learning pipeline design with matrix and tensor factorization.

problem Designing effective supervised learning pipelines with many choices.
method Uses matrix and tensor factorization to model pipeline search space and develops greedy experiment design protocols.
result Demonstrates the effectiveness of the approach on real-world classification problems.

Pipeline decomposes portfolio optimization problems into smaller, solvable subproblems.

problem Large-scale portfolio optimization with constraints.
method Decomposition pipeline with preprocessing, clustering, and risk rebalancing.
result Pipeline reduces problem size by 80% and computation time.

Paper proposes a statistical test for feature selection pipelines using selective inference.

problem Assessing the significance of feature selection pipelines in data analysis.
method Selective inference technique applied to feature selection pipelines composed of various algorithms.
result The proposed statistical test controls false positive feature selection probabilities.

Pipelined Backpropagation trains large models without batches efficiently.

problem Training large models efficiently on hardware with limited batch sizes.
method Fine-grained Pipelined Backpropagation with Spike Compensation and Linear Weight Prediction.
result Fine-grained Pipelined Backpropagation with a batch size of one matches the accuracy of SGD for multiple networks.

Study reveals similarities in knowledge flows between pharmaceutical and AI industries.

problem Understanding the dynamics of drug pipelines in global pharmaceutical industry.
method Multilayer network analysis of drug pipeline, global supply chain, and ownership data.
result Proven similarities in knowledge flows between pharmaceutical and AI industries.

TODS automates time series outlier detection with customizable pipelines.

problem Automated detection of outliers in time series data.
method Modular system with 70 primitives for data processing, time series analysis, and detection algorithms. GUI and data-driven searcher for pipeline design.
result Automated discovery and construction of effective outlier detection pipelines.

A rigorous ML pipeline for binary classification in biomedical studies, focusing on pancreatic cancer.

problem Handling bias in ML models for complex biomedical data.
method Customizable ML analysis pipeline with 9 algorithms, hyperparameter optimization, and thorough evaluation.
result Comparison of ML algorithms to ExSTraCS, highlighting interpretability and bias handling.

Pipeline parallelism (PP) when training neural networks enables larger models to be partitioned spatially, leading to both lower network communication and overall higher hardware utilization. Unfortunately, to preserve the statistical efficiency of sequential training, existing PP techniques sacrifice hardware efficien…

2019-10-09abs ↗pdf ↗

We introduce novel communication strategies in synchronous distributed Deep Learning consisting of decentralized gradient reduction orchestration and computational graph-aware grouping of gradient tensors. These new techniques produce an optimal overlap between computation and communication and result in near-linear sc…

2019-09-24abs ↗pdf ↗

Morpheo is a transparent and secure machine learning platform collecting and analysing large datasets. It aims at building state-of-the art prediction models in various fields where data are sensitive. Indeed, it offers strong privacy of data and algorithm, by preventing anyone to read the data, apart from the owner an…

2017-04-17abs ↗pdf ↗

RL pipeline simplifies knot diagrams, including very hard unknots.

problem Simplifying complex knot diagrams, especially very hard unknots.
method Reinforcement learning for move proposals and heuristic navigation of Reidemeister moves.
result Trained agent simplifies diagrams, including a 41#9104_1\#9_{10} link to a three-step unknotting process.

This paper explores how LLMs can improve pipeline-based conversational agents.

problem Limitations of pipeline-based conversational agents in human-like conversations.
method Investigated LLMs' capabilities in two phases: design and development, and operations.
result LLMs can enhance pipeline-based agents in various tasks like data generation, intent classification, and auto-correction.

Much of the work in metalearning has focused on classifier selection, combined more recently with hyperparameter optimization, with little concern for data preprocessing. Yet, it is generally well accepted that machine learning applications require not only model building, but also data preprocessing. In other words, p…

2018-10-23abs ↗pdf ↗

Future Connected and Automated Vehicles (CAV), and more generally ITS, will form a highly interconnected system. Such a paradigm is referred to as the Internet of Vehicles (herein Internet of CAVs) and is a prerequisite to orchestrate traffic flows in cities. For optimal decision making and supervision, traffic centres…

2019-07-01abs ↗pdf ↗

We introduce Microsoft Machine Learning for Apache Spark (MMLSpark), an ecosystem of enhancements that expand the Apache Spark distributed computing library to tackle problems in Deep Learning, Micro-Service Orchestration, Gradient Boosting, Model Interpretability, and other areas of modern computation. Furthermore, we…

2018-10-20abs ↗pdf ↗

Paper proposes EEIPU, a memoization-aware BO algorithm to reduce hyperparameter tuning costs.

problem High costs in GPU-days for training and fine-tuning language models.
method Memoization-aware Bayesian Optimization (EEIPU) algorithm in tandem with pipeline caching.
result EEIPU produces 103% more hyperparameter candidates and 108% more validation metric improvement.

Differentiable pipeline replaces non-differentiable CAE components for shape optimization.

problem Gradient-based optimization is limited by non-differentiable components in CAE workflows.
method Surrogate models replace non-differentiable pipeline components, enabling gradient-based optimization.
result Gradient-based shape optimization possible without differentiable solvers.

Paper introduces Modular Jets for diagnosing model decompositions in pipelines.

problem Evaluating model decompositions in pipelines for unique identification.
method Estimates empirical jets from module-level representations to diagnose mirage vs identifiable decompositions.
result Proves jet-identifiability theorem for two-module linear regression pipelines.

Autonomous driving requires 3D perception of vehicles and other objects in the in environment. Much of the current methods support 2D vehicle detection. This paper proposes a flexible pipeline to adopt any 2D detection network and fuse it with a 3D point cloud to generate 3D information with minimum changes of the 2D d…

2018-02-12abs ↗pdf ↗

To solve a machine learning problem, one typically needs to perform data preprocessing, modeling, and hyperparameter tuning, which is known as model selection and hyperparameter optimization.The goal of automated machine learning (AutoML) is to design methods that can automatically perform model selection and hyperpara…

2019-04-01abs ↗pdf ↗