New approach speeds up optimization with repeated gradient steps on same batch.
problem Performance bottlenecks in massive parallel pipelines with large batch sizes.
method Data echoing, taking repeated gradient steps on the same batch.
result Data echoing affords speedups on curvature-dominated part of convergence rate.
New machine learning pipeline solves dynamic vehicle routing problems efficiently.
problem Efficiently handling same day deliveries in e-commerce logistics.
method Combination of machine learning and combinatorial optimization.
result Ranked first in the EURO Meets NeurIPS Vehicle Routing Competition.
AVATAR uses a surrogate model to quickly evaluate ML pipelines, saving time and resources.
problem Time-consuming evaluation of ML pipelines limits exploration of complex models.
method AVATAR employs a surrogate model to assess pipeline validity without execution.
result AVATAR accelerates ML pipeline evaluation, improving efficiency in complex scenarios.
Paper introduces probabilistic approach to CO layers in ML.
problem Challenges in training CO layers with stochastic gradient descent.
method Probabilistic perspective, approximate differentiation, structured losses.
result InferOpt.jl package for training CO layers.
Hyperparameter tuning of multi-stage pipelines introduces a significant computational burden. Motivated by the observation that work can be reused across pipelines if the intermediate computations are the same, we propose a pipeline-aware approach to hyperparameter tuning. Our approach optimizes both the design and exe…
Differentiable clustering method using perturbed spanning forests.
problem Efficient clustering in trainable pipelines with noisy data.
method Stochastic perturbations of minimum-weight spanning forests.
result Method performs well even in challenging settings.
Classical Machine Learning (ML) pipelines often comprise of multiple ML models where models, within a pipeline, are trained in isolation. Conversely, when training neural network models, layers composing the neural models are simultaneously trained using backpropagation. We argue that the isolated training scheme of ML…
Study proposes a statistical testing framework for evaluating clustering pipelines.
problem Quantifying the statistical reliability of clustering results from data analysis pipelines.
method Selective inference-based statistical testing framework for clustering pipelines.
result The proposed test controls the type I error rate and is effective in validating clustering results.
RL applied to TCLs for power consumption control.
problem Optimizing power consumption using TCLs with RL.
method Modelica-based reinforcement learning (Q-learning) for stochastic TCLs.
result Q-learning parameters affect controller performance.
Investigates fairness in pipeline models where individuals may drop out.
problem Fairness in pipeline models where individuals may drop out and subsequent stages depend on remaining individuals.
method Rigorous framework for evaluating fairness guarantees, showing that naïve auditing is insufficient and dependence must exist between stages.
result Fairness in pipelines can be arbitrary, even with just two stages, and requires dependence between stages.
PipeMare enables efficient DNN training with minimal memory and pipeline sacrifices.
problem Sacrificing hardware efficiency to maintain statistical efficiency in pipeline parallel DNN training.
method PipeMare is a simple yet robust training method that tolerates asynchronous updates during pipeline parallelism without sacrificing pipeline utilization or memory.
result PipeMare achieves up to 2.7x less memory usage or 4.3x higher pipeline utilization compared to state-of-the-art synchronous PP training techniques.
Data science relies on pipelines that are organized in the form of interdependent computational steps. Each step consists of various candidate algorithms that maybe used for performing a particular function. Each algorithm consists of several hyperparameters. Algorithms and hyperparameters must be optimized as a whole …
Automates supervised learning pipeline design with matrix and tensor factorization.
problem Designing effective supervised learning pipelines with many choices.
method Uses matrix and tensor factorization to model pipeline search space and develops greedy experiment design protocols.
result Demonstrates the effectiveness of the approach on real-world classification problems.
Data science relies on pipelines that are organized in the form of interdependent computational steps. Each step consists of various candidate algorithms that maybe used for performing a particular function. Each algorithm consists of several hyperparameters. Algorithms and hyperparameters must be optimized as a whole …
New model optimizes oil product distribution via pipelines.
problem Optimizing oil product distribution via pipelines.
method Discrete-time mixed integer linear programming model.
result Significant reductions in pipeline operational cost.
Pipeline decomposes portfolio optimization problems into smaller, solvable subproblems.
problem Large-scale portfolio optimization with constraints.
method Decomposition pipeline with preprocessing, clustering, and risk rebalancing.
result Pipeline reduces problem size by 80% and computation time.
Paper proposes a statistical test for feature selection pipelines using selective inference.
problem Assessing the significance of feature selection pipelines in data analysis.
method Selective inference technique applied to feature selection pipelines composed of various algorithms.
result The proposed statistical test controls false positive feature selection probabilities.
New approach identifies and explains errors in machine learning pipelines.
problem Challenges in identifying and explaining errors in complex machine learning pipelines.
method Uses iteration and provenance to automatically infer root causes of failures.
result Significantly improves precision and recall compared to state-of-the-art methods.
Consider a device that is connected to an edge processor via a communication channel. The device holds local data that is to be offloaded to the edge processor so as to train a machine learning model, e.g., for regression or classification. Transmission of the data to the learning processor, as well as training based o…
We propose a systematic learning-based approach to the generation of massive quantities of synthetic 3D scenes and arbitrary numbers of photorealistic 2D images thereof, with associated ground truth information, for the purposes of training, benchmarking, and diagnosing learning-based computer vision and robotics algor…
Pipelined Backpropagation trains large models without batches efficiently.
problem Training large models efficiently on hardware with limited batch sizes.
method Fine-grained Pipelined Backpropagation with Spike Compensation and Linear Weight Prediction.
result Fine-grained Pipelined Backpropagation with a batch size of one matches the accuracy of SGD for multiple networks.
Study reveals similarities in knowledge flows between pharmaceutical and AI industries.
problem Understanding the dynamics of drug pipelines in global pharmaceutical industry.
method Multilayer network analysis of drug pipeline, global supply chain, and ownership data.
result Proven similarities in knowledge flows between pharmaceutical and AI industries.
Modyn automates continuous ML model training on growing datasets.
problem Continuous model retraining is costly and impractical.
method Data-centric ML platform with policies for continuous training.
result Modyn enables high throughput training with sample-level data selection.
TODS automates time series outlier detection with customizable pipelines.
problem Automated detection of outliers in time series data.
method Modular system with 70 primitives for data processing, time series analysis, and detection algorithms. GUI and data-driven searcher for pipeline design.
result Automated discovery and construction of effective outlier detection pipelines.
Hungarian text processing improved with efficient, accurate NLP pipelines.
problem Improving text processing for Hungarian language.
method Implemented in spaCy framework, focusing on efficiency and accuracy.
result Near state-of-the-art performance in all text preprocessing steps.
Sorting input objects is an important step in many machine learning pipelines. However, the sorting operator is non-differentiable with respect to its inputs, which prohibits end-to-end gradient-based optimization. In this work, we propose NeuralSort, a general-purpose continuous relaxation of the output of the sorting…
We introduce backdrop, a flexible and simple-to-implement method, intuitively described as dropout acting only along the backpropagation pipeline. Backdrop is implemented via one or more masking layers which are inserted at specific points along the network. Each backdrop masking layer acts as the identity in the forwa…
A rigorous ML pipeline for binary classification in biomedical studies, focusing on pancreatic cancer.
problem Handling bias in ML models for complex biomedical data.
method Customizable ML analysis pipeline with 9 algorithms, hyperparameter optimization, and thorough evaluation.
result Comparison of ML algorithms to ExSTraCS, highlighting interpretability and bias handling.
Pipeline for comparing trading algorithms in finance and crypto.
problem Disconnected research and applications in algorithmic trading.
method General pipeline for designing, programming, and evaluating trading strategies.
result Systematic comparison of trading algorithms in finance and crypto.
DeepLine automates ML pipeline generation using reinforcement learning.
problem Automatic generation of end-to-end ML pipelines combining multiple algorithms.
method Deep Reinforcement Learning with hierarchical actions filtering.
result DeepLine outperforms state-of-the-art approaches in accuracy and computational cost.
LLMs struggle to generate random numbers from statistical distributions, leading to biased results in applications.
problem LLMs' inability to generate random numbers accurately from specified distributions.
method Dual-protocol design: Batch Generation and Independent Requests, benchmarking 11 models across 15 distributions.
result Sampling fidelity degrades with distributional complexity and horizon, leading to systematic biases in downstream applications.
LAGS-SGD optimizes deep learning training by sparsifying gradients layer-wise.
problem Reduces long training times in large deep neural networks with distributed S-SGD.
method Layer-wise adaptive gradient sparsification combined with S-SGD.
result LAGS-SGD achieves convergence guarantees and outperforms vanilla S-SGD.
Paper develops methods to optimize policies directly from human feedback without reward inference.
problem Challenges in RLHF, including reward model overfitting and distribution shift.
method Develops two algorithms for RLHF without reward inference, using zeroth-order gradient approximators.
result Establishes polynomial convergence rates and outperforms existing methods in numerical experiments.
RankML predicts machine learning pipeline performance quickly and accurately.
problem Lack of machine learning experts and high computational costs.
method Meta-learning approach to predict pipeline performance.
result RankML outperforms or matches state-of-the-art approaches with less time and resources.
RL pipeline simplifies knot diagrams, including very hard unknots.
problem Simplifying complex knot diagrams, especially very hard unknots.
method Reinforcement learning for move proposals and heuristic navigation of Reidemeister moves.
result Trained agent simplifies diagrams, including a 41#910 link to a three-step unknotting process. This paper explores how LLMs can improve pipeline-based conversational agents.
problem Limitations of pipeline-based conversational agents in human-like conversations.
method Investigated LLMs' capabilities in two phases: design and development, and operations.
result LLMs can enhance pipeline-based agents in various tasks like data generation, intent classification, and auto-correction.
Much of the work in metalearning has focused on classifier selection, combined more recently with hyperparameter optimization, with little concern for data preprocessing. Yet, it is generally well accepted that machine learning applications require not only model building, but also data preprocessing. In other words, p…
Pipeline selects best RL policy from limited data.
problem Selecting best RL policy from small datasets.
method Task- and method-agnostic pipeline using multiple data splits.
result Pipeline produces higher-performing policies.
Machine learning is increasingly targeting areas where input data cannot be accurately described by a single vector, but can be modeled instead using the more flexible concept of random vectors, namely probability measures or more simply point clouds of varying cardinality. Using deep architectures on measures poses, h…
Machine learning pipeline potentially consists of several stages of operations like data preprocessing, feature engineering and machine learning model training. Each operation has a set of hyper-parameters, which can become irrelevant for the pipeline when the operation is not selected. This gives rise to a hierarchica…
Paper proposes EEIPU, a memoization-aware BO algorithm to reduce hyperparameter tuning costs.
problem High costs in GPU-days for training and fine-tuning language models.
method Memoization-aware Bayesian Optimization (EEIPU) algorithm in tandem with pipeline caching.
result EEIPU produces 103% more hyperparameter candidates and 108% more validation metric improvement.
Optimizes deep learning pipelines with novel algorithms for smooth and non-smooth functions.
problem Optimizing deep learning pipelines for smooth and non-smooth functions.
method Provided matching lower and upper bounds for smooth convex and non-convex functions, and developed PPRS for non-smooth convex functions.
result PPRS achieves near-linear speed-up and convergence time for non-smooth non-convex problems.
TempoPFN models for zero-shot time series forecasting using synthetic data.
problem Efficient long-horizon prediction and reproducibility in zero-shot time series forecasting.
method Linear RNNs pre-trained on synthetic data with GatedDeltaProduct architecture and state-weaving.
result Achieves top-tier competitive performance on various benchmarks.
Differentiable pipeline replaces non-differentiable CAE components for shape optimization.
problem Gradient-based optimization is limited by non-differentiable components in CAE workflows.
method Surrogate models replace non-differentiable pipeline components, enabling gradient-based optimization.
result Gradient-based shape optimization possible without differentiable solvers.
SRO optimizes decisions against worst-case sampler induced by generative models.
problem Operational uncertainty shifts from explicit probability law to sampler induced by learned generators.
method SRO optimizes decisions against the worst-case sampler induced by perturbing the learned generator.
result Empirical worst-case objective provides high-probability upper certificate for true population objective.
Clinical prognostic models derived from largescale healthcare data can inform critical diagnostic and therapeutic decisions. To enable off-theshelf usage of machine learning (ML) in prognostic research, we developed AUTOPROGNOSIS: a system for automating the design of predictive modeling pipelines tailored for clinical…
Paper introduces Modular Jets for diagnosing model decompositions in pipelines.
problem Evaluating model decompositions in pipelines for unique identification.
method Estimates empirical jets from module-level representations to diagnose mirage vs identifiable decompositions.
result Proves jet-identifiability theorem for two-module linear regression pipelines.
Autonomous driving requires 3D perception of vehicles and other objects in the in environment. Much of the current methods support 2D vehicle detection. This paper proposes a flexible pipeline to adopt any 2D detection network and fuse it with a 3D point cloud to generate 3D information with minimum changes of the 2D d…