New framework assesses and benchmarks ML methods for multivariate time series.
problem Benchmarking and explaining performance of machine learning methods.
method Proposes a new framework with systematized performance-explainability characteristics.
result Illustrates application to multivariate time series classifiers.
The paper clarifies conditions for using benchmark scores in machine learning.
problem Using benchmark scores to draw scientific inferences about learning problems.
method Developing conditions of construct validity inspired by psychological measurement theory.
result Clarifies conditions under which benchmark scores support diverse scientific claims.
Paper benchmarks machine learning for detecting process curve drifts.
problem Detecting drifts in multivariate manufacturing process data.
method Synthetic data generation and evaluation score introduction.
result Existing algorithms often fail with complex drift scenarios.
DIGEN benchmark provides synthetic datasets for ML algorithm evaluation.
problem Understanding and comparing machine learning algorithms' performance.
method Synthetic datasets generated using 40 mathematical functions to evaluate machine learning algorithms.
result DIGEN resource facilitates understanding why algorithms perform poorly and provides ideas for improvement.
Study finds little progress in medical machine learning benchmarks over 3 years.
problem Lack of meaningful progress in medical machine learning benchmarks for structured healthcare data.
method Comprehensive review and meta-analysis of benchmarks in medical machine learning for structured data.
result Deep recurrent models perform only better than logistic regression on certain clinical prediction tasks.
Machine learning (ML) has become a vital part in many aspects of our daily life. However, building well performing machine learning applications requires highly specialized data scientists and domain experts. Automated machine learning (AutoML) aims to reduce the demand for data scientists by enabling domain experts to…
Molecular machine learning has been maturing rapidly over the last few years. Improved methods and the presence of larger datasets have enabled machine learning algorithms to make increasingly accurate predictions about molecular properties. However, algorithmic progress has been limited due to the lack of a standard b…
R package for machine learning in survival analysis.
problem Limited machine learning interfaces for survival analysis.
method Provides a comprehensive machine learning interface for survival analysis.
result Systematic infrastructure for survival modeling and evaluation.
Benchopt automates machine learning benchmarking across languages and hardware.
problem Limited transparency and tedious re-implementation work in machine learning validation.
method A collaborative framework for automating, reproducing, and publishing optimization benchmarks.
result Demonstrates practical findings that highlight the importance of details in machine learning validation.
New benchmark tests machine learning's ability to learn causal overhypotheses.
problem Machine learning's difficulty in understanding causal overhypotheses.
method Adapted blicket detector environment for machine learning agents to test causal overhypotheses.
result Many state-of-the-art methods struggle with causal overhypotheses in the new benchmark.
New model reveals significant impact of data and parameter variations on machine learning benchmarks.
problem Variance in machine learning benchmarks due to data sampling, initialization, and hyperparameters.
method Modeling the benchmarking process, analyzing comparison methods, and studying error rates.
result Adding more sources of variation reduces compute cost while improving performance.
GAMA is a user-friendly AutoML system for machine learning pipeline optimization.
problem Empowering users to control and optimize machine learning pipelines.
method Modular AutoML system with three search algorithms and two post-processing steps.
result GAMA allows users to track and control the search process for optimal machine learning pipelines.
The study examines dataset usage patterns in machine learning research.
problem Lack of attention to dataset dynamics in machine learning research.
method Analysis of dataset usage patterns across machine learning subcommunities and time periods (2015-2020).
result Increasing concentration on fewer and fewer datasets, significant adoption from other tasks, and concentration across the field on datasets introduced by elite institutions.
Machine learning research depends on objectively interpretable, comparable, and reproducible algorithm benchmarks. We advocate the use of curated, comprehensive suites of machine learning tasks to standardize the setup, execution, and reporting of benchmarks. We enable this through software tools that help to create an…
Paper introduces OARF benchmark suite for federated learning systems.
problem Limited diversity in federated learning benchmarks.
method Characterizes OARF benchmark suite with diverse data and applications.
result Federated learning can effectively increase end-to-end throughput.
CausalBench aims to advance causal learning research with a transparent platform.
problem Lack of unified benchmark datasets, algorithms, metrics, and evaluation interfaces for causal learning.
method Introduces CausalBench, a flexible benchmark framework for causal analysis and machine learning.
result Promotes scientific collaboration, reproducibility, and awareness in causal learning research.
Machine learning workflow development is anecdotally regarded to be an iterative process of trial-and-error with humans-in-the-loop. However, we are not aware of quantitative evidence corroborating this popular belief. A quantitative characterization of iteration can serve as a benchmark for machine learning workflow d…
FedML aims to improve FL research by providing a library and benchmark.
problem Inconsistent FL algorithm development and performance comparison.
method FedML offers an open research library and benchmark supporting diverse computing paradigms and flexible API design.
result FedML facilitates fair algorithm comparison and development in federated learning.
Paper compiles ML algorithm performance benchmarks on OpenML datasets.
problem Finding optimal hyperparameters for ML algorithms efficiently.
method Generated benchmark data for 7 ML algorithms on 39 datasets, fixed hyperparameters before testing.
result Comprehensive dataset of ML algorithm performance sensitivity.
FLBench automates federated learning benchmarking.
problem Manual dataset partitioning fails to simulate real-world isolated data islands.
method Develops a federated learning benchmark suite with three domains.
result Automates evaluation of federated learning systems and algorithms.
MLPerf, an emerging machine learning benchmark suite strives to cover a broad range of applications of machine learning. We present a study on its characteristics and how the MLPerf benchmarks differ from some of the previous deep learning benchmarks like DAWNBench and DeepBench. We find that application benchmarks suc…
GraphBench creates a unified benchmark for graph learning tasks.
problem Fragmented benchmarking practices and inconsistent evaluation protocols in graph learning.
method Developed a comprehensive benchmark suite with standardized evaluation protocols.
result Established principled baselines for future research in graph learning.
This paper predicts weekly stock market movements using machine learning and introduces a new benchmark.
problem Predicting stock market movements using daily data and various ML models.
method Focuses on weekly movements, introduces random traders as a benchmark, uses additional features, and adjusts training datasets.
result Trained models, especially MLP, show good performance across different trends.
This paper benchmarks algorithms for training fair DNNs, addressing real-world fairness constraints.
problem Training deep neural networks with fairness constraints.
method Benchmarking stochastic approximation algorithms for fairness-constrained DNN training.
result Demonstrates the use of a new benchmark for comparing fairness-improving algorithms.
Time series classification problems have drawn increasing attention in the machine learning and statistical community. Closely related is the field of functional data analysis (FDA): it refers to the range of problems that deal with the analysis of data that is continuously indexed over some domain. While often employi…
Benchmark for math reasoning models from human proofs.
problem Measuring and accelerating machine learning models in high-level mathematical reasoning.
method Built a non-synthetic dataset from theorem prover proofs, defined a task for model to fill in missing propositions, used hierarchical transformer to improve performance.
result Neural models can capture non-trivial mathematical reasoning, hierarchical transformer outperforms baseline.
The paper quantifies uncertainty in aggregated machine learning metrics.
problem Uncertainty in summarizing model performance across multiple tasks.
method Statistical methodologies including bootstrapping and Bayesian modeling.
result Insights into model performance dominance for specific tasks.
A new machine learning model uses matrix exponentials for universal approximation.
problem Developing a robust and efficient machine learning model.
method Introduces a novel architecture using matrix exponentials as the only nonlinearity.
result The model achieves universal approximation properties and outperforms other models on benchmark tasks.
Researchers create benchmarks to compare graph inference methods.
problem Comparing graph inference methods is difficult due to varying downstream tasks.
method Developed benchmarks for various graph tasks.
result Contrasted prominent graph inference techniques.
NeSS combines neural and symbolic approaches for better compositional generalization.
problem Lack of compositional generalization in deep learning models.
method NeSS uses a neural network to generate traces, executed by a symbolic stack machine with sequence manipulation.
result Achieves 100% generalization performance across multiple domains.
Reduces survival analysis to common regression tasks.
problem Applying standard machine learning tools to survival analysis.
method Various reduction techniques to simplify survival analysis.
result Benchmark analysis shows improved predictive performance.
Electroencephalograms (EEG) are often contaminated by artifacts which make interpreting them more challenging for clinicians. Hence, automated artifact recognition systems have the potential to aid the clinical workflow. In this abstract, we share the first results on applying various machine learning algorithms to the…
This paper benchmarks privacy-preserving machine learning on medical images.
problem Ensuring privacy in medical image analysis while maintaining model accuracy.
method Comparing Local-DP and DP-SGD for differential privacy in medical imagery.
result Theoretical privacy guarantees do not fully align with real-world performance.
Proposes CLRS benchmark to evaluate algorithmic reasoning.
problem Difficulty in transferring results across publications due to targeted algorithmic data.
method Develops a comprehensive benchmark covering various algorithmic tasks.
result Demonstrates performance of algorithmic reasoning baselines on the CLRS benchmark.
WRENCH benchmarks weak supervision datasets for machine learning.
problem Lack of standardized evaluation for weak supervision datasets.
method Developed a comprehensive benchmark platform WRENCH.
result Demonstrated the efficacy of WRENCH as a benchmark platform.
New IRT method identifies useful datasets for ML classifier evaluation.
problem Lack of standard evaluation strategy for ML benchmarks.
method Applied Item Response Theory (IRT) to OpenML-CC18 benchmark.
result Not all datasets are useful for evaluating classifiers.
This paper explores hyperparameter optimization for machine learning models.
problem Finding the best hyper-parameters for machine learning models.
method Introduces state-of-the-art optimization techniques and discusses their application.
result Comparison of different optimization methods on benchmark datasets.
OGB provides diverse graph datasets for robust ML research.
problem Challenges in scalable and robust graph machine learning.
method Unified evaluation protocol, diverse datasets, and automated pipeline.
result Significant scalability and generalization challenges identified.
CleverHans is a software library that provides standardized reference implementations of adversarial example construction techniques and adversarial training. The library may be used to develop more robust machine learning models and to provide standardized benchmarks of models' performance in the adversarial setting. …
Deep neural networks solve Raven's Progressive Matrices with high accuracy.
problem Testing relational reasoning in machine learning systems.
method Combining Wild Relation Networks with Multi-Layer Relation Networks and introducing Magnitude Encoding.
result Deep neural networks achieve 98.0 percent accuracy, significantly improving over previous methods.
Progress of machine learning in critical care has been difficult to track, in part due to absence of public benchmarks. Other fields of research (such as computer vision and natural language processing) have established various competitions and public benchmarks. Recent availability of large clinical datasets has enabl…
L2O uses machine learning to design optimization methods.
problem Designing efficient optimization methods for specific problem distributions.
method Data-driven approach to automate optimization method design.
result L2O methods are practical for specific problem distributions but fail on out-of-distribution problems.
The paper aims to define a benchmark for deep learning recommendation models.
problem Insufficient benchmarking for deep learning recommendation models.
method Synthesizes modeling strategies, defines desirable characteristics, and summarizes advice from the MLPerf Recommendation Advisory Board.
result Defines an industry-relevant benchmark for deep learning recommendation models.
WOODS benchmarks improve understanding of time series OOD generalization.
problem Limited understanding of OOD generalization in time series.
method Presented eight open-source time series benchmarks and revised OOD algorithms.
result Large room for improvement in OOD generalization algorithms for time series.
Quantum datasets improve QML performance.
problem Benchmarking QML on classical datasets is uncertain.
method Introduced NTangled dataset of quantum states with varying entanglement.
result QML models trained on NTangled dataset outperform classical models.
Deep learning models (aka Deep Neural Networks) have revolutionized many fields including computer vision, natural language processing, speech recognition, and is being increasingly used in clinical healthcare applications. However, few works exist which have benchmarked the performance of the deep learning models with…
AutoML frameworks outperform human data scientists on 7 out of 12 OpenML tasks.
problem Evaluating if AutoML can outperform human data scientists.
method Comparison of four AutoML frameworks on 12 popular OpenML datasets (6 supervised classification, 6 supervised regression).
result AutoML frameworks perform better or equal to human data scientists in 7 out of 12 tasks.
SubseasonalClimateUSA dataset improves subseasonal weather forecasting.
problem Challenges in subseasonal weather forecasting, especially skill of physics-based models and integration of local and global variables.
method Curated dataset for training and benchmarking subseasonal forecasting models, including various methods.
result Benchmarking suggests simple and effective ways to improve current operational models.