AVATAR uses a surrogate model to quickly evaluate ML pipelines, saving time and resources.
problem Time-consuming evaluation of ML pipelines limits exploration of complex models.
method AVATAR employs a surrogate model to assess pipeline validity without execution.
result AVATAR accelerates ML pipeline evaluation, improving efficiency in complex scenarios.
Hyperparameter tuning of multi-stage pipelines introduces a significant computational burden. Motivated by the observation that work can be reused across pipelines if the intermediate computations are the same, we propose a pipeline-aware approach to hyperparameter tuning. Our approach optimizes both the design and exe…
Pipeline for comparing trading algorithms in finance and crypto.
problem Disconnected research and applications in algorithmic trading.
method General pipeline for designing, programming, and evaluating trading strategies.
result Systematic comparison of trading algorithms in finance and crypto.
Paper introduces Modular Jets for diagnosing model decompositions in pipelines.
problem Evaluating model decompositions in pipelines for unique identification.
method Estimates empirical jets from module-level representations to diagnose mirage vs identifiable decompositions.
result Proves jet-identifiability theorem for two-module linear regression pipelines.
Evaluating the computational reproducibility of data analysis pipelines has become a critical issue. It is, however, a cumbersome process for analyses that involve data from large populations of subjects, due to their computational and storage requirements. We present a method to predict the computational reproducibili…
Study proposes a statistical testing framework for evaluating clustering pipelines.
problem Quantifying the statistical reliability of clustering results from data analysis pipelines.
method Selective inference-based statistical testing framework for clustering pipelines.
result The proposed test controls the type I error rate and is effective in validating clustering results.
New approach identifies and explains errors in machine learning pipelines.
problem Challenges in identifying and explaining errors in complex machine learning pipelines.
method Uses iteration and provenance to automatically infer root causes of failures.
result Significantly improves precision and recall compared to state-of-the-art methods.
This work benchmarks neural embeddings for link prediction in evolving knowledge graphs.
problem Evaluating the robustness of neural embeddings in changing knowledge graphs.
method Proposes an open-source evaluation pipeline using relation-centric connectivity measures.
result Demonstrates the importance of simulating embedding accuracy for frequently updated knowledge graphs.
Paper proposes EEIPU, a memoization-aware BO algorithm to reduce hyperparameter tuning costs.
problem High costs in GPU-days for training and fine-tuning language models.
method Memoization-aware Bayesian Optimization (EEIPU) algorithm in tandem with pipeline caching.
result EEIPU produces 103% more hyperparameter candidates and 108% more validation metric improvement.
A rigorous ML pipeline for binary classification in biomedical studies, focusing on pancreatic cancer.
problem Handling bias in ML models for complex biomedical data.
method Customizable ML analysis pipeline with 9 algorithms, hyperparameter optimization, and thorough evaluation.
result Comparison of ML algorithms to ExSTraCS, highlighting interpretability and bias handling.
Investigates fairness in pipeline models where individuals may drop out.
problem Fairness in pipeline models where individuals may drop out and subsequent stages depend on remaining individuals.
method Rigorous framework for evaluating fairness guarantees, showing that naïve auditing is insufficient and dependence must exist between stages.
result Fairness in pipelines can be arbitrary, even with just two stages, and requires dependence between stages.
Pipelined Backpropagation trains large models without batches efficiently.
problem Training large models efficiently on hardware with limited batch sizes.
method Fine-grained Pipelined Backpropagation with Spike Compensation and Linear Weight Prediction.
result Fine-grained Pipelined Backpropagation with a batch size of one matches the accuracy of SGD for multiple networks.
Automatic machine learning (AutoML) is an area of research aimed at automating machine learning (ML) activities that currently require human experts. One of the most challenging tasks in this field is the automatic generation of end-to-end ML pipelines: combining multiple types of ML algorithms into a single architectu…
Hungarian text processing improved with efficient, accurate NLP pipelines.
problem Improving text processing for Hungarian language.
method Implemented in spaCy framework, focusing on efficiency and accuracy.
result Near state-of-the-art performance in all text preprocessing steps.
Framework evaluates privacy cost of non-private pre-processing in DP pipelines.
problem Privacy cost of non-private data-dependent pre-processing in DP machine learning pipelines.
method Establishes upper bounds on overall privacy guarantees using Smooth DP and bounded sensitivity.
result Explicit overall privacy guarantees for various pre-processing algorithms.
Study evaluates MRIQC pipeline's generalization on large multi-center datasets.
problem Generalizing MRI QC methods to new, unseen data.
method Evaluated MRIQC preprocessing steps on ABIDE and CATI datasets.
result Model without preprocessing yields best results on unseen data.
PortBench benchmarks LLMs for PM, revealing their weaknesses in diversification and robustness.
problem Lack of benchmarks for LLM-driven portfolio management, especially in diversification and robustness.
method Developed a comprehensive benchmark with a static QA dataset and a dynamic allocation pipeline, introducing metrics to evaluate correlation and robustness.
result 90% of LLMs fail to outperform a basic equal-weight allocation, highlighting their limitations in diversification and robustness.
Modular pipeline improves stock portfolio prediction robustness under regime changes.
problem Overfitting in deep learning models for non-stationary datasets.
method Modular machine learning pipeline with GBDT models and online learning techniques.
result GBDT models with dropout show high performance, robustness, and generalisability.
Pipeline combines ETF preprocessing with tabular model for cross-modal inference.
problem Transferability of tabular models across different modalities.
method Fixed comparison object, ETF preprocessing, in-context inference.
result Pipeline is broadly competitive, runs faster, and produces well-calibrated probabilities.
The paper proposes a ML workflow for B2B sales prediction.
problem Predicting business to business sales outcomes using subjective human evaluations.
method A two-pipeline ML approach on Azure ML, including data enrichment and model training.
result ML predictions improve accuracy and increase sales value.
Open dataset and pipeline for realistic OPE research.
problem Lack of realistic and reproducible OPE experimental studies.
method Public logged bandit dataset and Python software.
result Enables experimental comparisons of OPE estimators.
Transformer models improve arithmetic accuracy with number decomposition.
problem Transformer models struggle with arithmetic operations without decomposition.
method Fine-tuning models with a pipeline that decomposes numbers into units, tens, etc.
result Accuracy increased by 63% in five-digit addition tasks.
Rapidly estimating the remaining wall thickness (RWT) is paramount for the non-destructive condition assessment evaluation of large critical metallic pipelines. A robotic vehicle with embedded magnetism-based sensors has been developed to traverse the inside of a pipeline and conduct inspections at the location of a br…
The explosion of digital data has created multiple opportunities for organizations and individuals to leverage machine learning (ML) to transform the way they operate. However, the shortage of experts in the field of machine learning -- data scientists -- is often a setback to the use of ML. In an attempt to alleviate …
AgensFlow learns multi-agent coordination policies from experience.
problem Difficult coordination choices in multi-agent systems built on LLMs.
method Online policy learning from repeated trajectories, treating decisions as learnable.
result Learned routing improves coordination-heavy workflows over static wiring.
Functional Magnetic Resonance Imaging (fMRI) relies on multi-step data processing pipelines to accurately determine brain activity; among them, the crucial step of spatial smoothing. These pipelines are commonly suboptimal, given the local optimisation strategy they use, treating each step in isolation. With the advent…
Data-driven methods such as convolutional neural networks (CNNs) are known to deliver state-of-the-art performance on image recognition tasks when the training data are abundant. However, in some instances, such as change detection in remote sensing images, annotated data cannot be obtained in sufficient quantities. In…
Real world observational data, together with causal inference, allow the estimation of causal effects when randomized controlled trials are not available. To be accepted into practice, such predictive models must be validated for the dataset at hand, and thus require a comprehensive evaluation toolkit, as introduced he…
A new property fixes look-ahead bias in backtesting and trading pipelines.
problem Fixing look-ahead bias in backtesting and trading pipelines.
method Developed a pipeline calculus separating availability from reference time, and a type-and-effect system for the value-independent fragment.
result The check scales linearly and catches all leaks, including those missed by differential and tiling detectors.
Study shows noisy data collection in ImageNet leads to biased model performance.
problem Noisy data collection in ImageNet datasets causes biased model performance.
method Human studies on ImageNet dataset to investigate design choices and their impact.
result Systematic misalignment between ImageNet benchmark and real-world task.
As machine learning is applied more widely, data scientists often struggle to find or create end-to-end machine learning systems for specific tasks. The proliferation of libraries and frameworks and the complexity of the tasks have led to the emergence of "pipeline jungles" - brittle, ad hoc ML systems. To address thes…
We study the AutoML problem of automatically configuring machine learning pipelines by jointly selecting algorithms and their appropriate hyper-parameters for all steps in supervised learning pipelines. This black-box (gradient-free) optimization with mixed integer & continuous variables is a challenging problem. We pr…
State-of-the-art speaker diarization systems utilize knowledge from external data, in the form of a pre-trained distance metric, to effectively determine relative speaker identities to unseen data. However, much of recent focus has been on choosing the appropriate feature extractor, ranging from pre-trained i−vectors…
CausalTime generates realistic time-series for TSCD evaluation.
problem Lack of realistic synthetic datasets for TSCD performance evaluation.
method Harnessing deep neural networks and normalizing flow for dynamics, extracting causal graphs, and deriving ground truth causal graphs.
result Generated datasets accurately reflect real data and ground truth causal graphs.
Hill-climbing is a powerful baseline for NAS, even with reduced noise.
problem Noise in evaluating neural architectures.
method Hill-climbing algorithm, denoising training pipeline.
result Hill-climbing outperforms state-of-the-art NAS algorithms with reduced noise.
This work improves AutoML systems by dynamically evaluating fitness to reduce overfitting.
problem Overfitting in AutoML systems, especially during internal cross-validation iterations.
method Introducing dynamic fitness evaluations that approximate repeated k-fold cross-validation.
result Significant improvement in generalization/testing performance over baseline methods.
Classical Machine Learning (ML) pipelines often comprise of multiple ML models where models, within a pipeline, are trained in isolation. Conversely, when training neural network models, layers composing the neural models are simultaneously trained using backpropagation. We argue that the isolated training scheme of ML…
Hybrid pipeline detects spike-and-wave discharges in long-term EEG recordings.
problem Manual identification of spike-and-wave discharges in long-term EEG recordings is labour-intensive and error-prone.
method A hybrid pipeline that combines analytical features with a shallow ANN for SWD detection.
result The method correctly detected 384 out of 392 annotated SWD events, achieving high sensitivity and specificity.
Large scale machine learning (ML) systems such as the Alexa automatic speech recognition (ASR) system continue to improve with increasing amounts of manually transcribed training data. Instead of scaling manual transcription to impractical levels, we utilize semi-supervised learning (SSL) to learn acoustic models (AM) …
Interpretable survival analysis improves heart failure risk prediction.
problem Improving heart failure risk prediction using survival analysis.
method Survival stacking, ControlBurn, Explainable Boosting Machines.
result Achieves state-of-the-art performance and provides novel insights.
Machine Learning (ML) is increasingly applied in real-life scenarios, raising concerns about bias in automatic decision making. We focus on bias as a notion of opinion exclusion, that stems from the direct application of traditional ML pipelines to infer subjective properties. We argue that such ML systems should be ev…
Neural networks struggle with TSP beyond small instances, requiring new approaches.
problem Neural networks struggle to generalize to larger instances of the TSP.
method Unified pipeline to identify inductive biases and promote generalization.
result Zero-shot generalization requires rethinking neural combinatorial optimization.
MLModelScope streamlines ML/DL model evaluation and benchmarking.
problem Challenges in evaluating and benchmarking ML/DL models.
method Open-source, framework/hardware agnostic platform with distributed design.
result Demonstrates the impact of model evaluation pipelines and HW/SW choices.
Data science relies on pipelines that are organized in the form of interdependent computational steps. Each step consists of various candidate algorithms that maybe used for performing a particular function. Each algorithm consists of several hyperparameters. Algorithms and hyperparameters must be optimized as a whole …
Paper predicts stock volatility using ESG news, showing deep learning's effectiveness.
problem Predicting stock volatility using ESG news.
method ESG news extraction, news representations, and Bayesian inference of deep learning models.
result Deep learning models predict stock volatility better than traditional methods.
Automates supervised learning pipeline design with matrix and tensor factorization.
problem Designing effective supervised learning pipelines with many choices.
method Uses matrix and tensor factorization to model pipeline search space and develops greedy experiment design protocols.
result Demonstrates the effectiveness of the approach on real-world classification problems.
Data science relies on pipelines that are organized in the form of interdependent computational steps. Each step consists of various candidate algorithms that maybe used for performing a particular function. Each algorithm consists of several hyperparameters. Algorithms and hyperparameters must be optimized as a whole …
New model optimizes oil product distribution via pipelines.
problem Optimizing oil product distribution via pipelines.
method Discrete-time mixed integer linear programming model.
result Significant reductions in pipeline operational cost.