This work introduces a new benchmark to compare neural network training algorithms.
problem Lack of reliable benchmarks to compare training algorithms effectively.
method Developed a new benchmark called AlgoPerf: Training Algorithms benchmark.
result Demonstrated the feasibility of the benchmark and set a provisional state-of-the-art.
Proposes CLRS benchmark to evaluate algorithmic reasoning.
problem Difficulty in transferring results across publications due to targeted algorithmic data.
method Develops a comprehensive benchmark covering various algorithmic tasks.
result Demonstrates performance of algorithmic reasoning baselines on the CLRS benchmark.
A benchmark for simulation-based inference methods.
problem Lack of a public benchmark for 'likelihood-free' algorithms.
method Provided a benchmark with tasks and performance metrics, including neural networks and ABC methods.
result State-of-the-art algorithms have room for improvement, and neural network-based approaches generally perform better.
The optimization of algorithm (hyper-)parameters is crucial for achieving peak performance across a wide range of domains, ranging from deep neural networks to solvers for hard combinatorial problems. The resulting algorithm configuration (AC) problem has attracted much attention from the machine learning community. Ho…
Study benchmarks TSC algorithms in distinguishing diffusions using the likelihood ratio test.
problem Benchmarking optimality of TSC algorithms in distinguishing diffusion processes.
method Proposes to benchmark TSC algorithms using the likelihood ratio test (LRT).
result LRT benchmarks are computationally efficient and can be applied to various time series types.
PHYRE benchmark tests physics learning algorithms.
problem Developing efficient algorithms for physics reasoning.
method Simple classical mechanics puzzles in a 2D environment.
result Modern algorithms perform poorly on PHYRE.
Proposes CLRS-Text, a new benchmark for evaluating LM reasoning capabilities.
problem Lack of transferable benchmarks for evaluating reasoning capabilities of language models.
method Developed a textual version of the CLRS benchmark, generating diverse algorithmic tasks.
result Demonstrates a novel challenge for the LM reasoning community and validates prior work.
In this report, we present a new reinforcement learning (RL) benchmark based on the Sonic the Hedgehog (TM) video game franchise. This benchmark is intended to measure the performance of transfer learning and few-shot learning algorithms in the RL domain. We also present and evaluate some baseline algorithms on the new…
Bayesian framework for comparing trading algorithms using cost analysis.
problem Comparing trading algorithms using cost analysis.
method Bayesian framework, hierarchical models, standardized benchmarks, fat tails, skewness, heteroscedasticity.
result Effective calculation of trading benchmarks with limited data.
Benchmarked over 70 graph clustering algorithms.
problem Lack of comprehensive performance comparison for graph clustering algorithms.
method Evaluated 70+ graph clustering programs for runtime and quality on weighted and unweighted graphs, analyzed ground truth characteristics.
result Supply a start point for engineers and viewpoint for researchers.
This paper benchmarks algorithms for training fair DNNs, addressing real-world fairness constraints.
problem Training deep neural networks with fairness constraints.
method Benchmarking stochastic approximation algorithms for fairness-constrained DNN training.
result Demonstrates the use of a new benchmark for comparing fairness-improving algorithms.
DIGEN benchmark provides synthetic datasets for ML algorithm evaluation.
problem Understanding and comparing machine learning algorithms' performance.
method Synthetic datasets generated using 40 mathematical functions to evaluate machine learning algorithms.
result DIGEN resource facilitates understanding why algorithms perform poorly and provides ideas for improvement.
Paper benchmarks MBRL algorithms with new environments.
problem Lack of standardized benchmarking for MBRL algorithms.
method Proposes 18 new benchmarking environments and standardizes problem settings.
result Characterizes three key research challenges for MBRL.
Develops a framework for consistent clustering algorithm benchmarking.
problem Lack of consistent benchmarking and diverse datasets for clustering algorithms.
method Consistent methodology, aggregated and polished datasets, interactive explorer.
result Introduces a comprehensive framework for testing clustering algorithms.
The study corrects misconceptions in GBDT speed benchmarks.
problem Misleading speed benchmarks of GBDT algorithms.
method Explained and criticized several straightforward benchmarking methods, outlined fair benchmark requirements.
result A fair GBDT speed benchmark requires specific conditions.
Paper compiles ML algorithm performance benchmarks on OpenML datasets.
problem Finding optimal hyperparameters for ML algorithms efficiently.
method Generated benchmark data for 7 ML algorithms on 39 datasets, fixed hyperparameters before testing.
result Comprehensive dataset of ML algorithm performance sensitivity.
OpenDataVal benchmarks data valuation algorithms for diverse datasets.
problem Improving model performance and mitigating biases in training datasets.
method Unified benchmark framework for data valuation algorithms.
result No single algorithm performs uniformly best across all tasks.
FedML aims to improve FL research by providing a library and benchmark.
problem Inconsistent FL algorithm development and performance comparison.
method FedML offers an open research library and benchmark supporting diverse computing paradigms and flexible API design.
result FedML facilitates fair algorithm comparison and development in federated learning.
New benchmarks for algorithm control using RL.
problem Optimizing algorithm performance through online hyperparameter tuning.
method Formulated as a contextual MDP, solved with reinforcement learning.
result Reinforcement learning outperforms black-box approaches for longer sequences.
Olympus benchmarks optimization algorithms for noisy experiments.
problem Benchmarking optimization algorithms on realistic experimental scenarios is challenging.
method Introduces Olympus, a software package for benchmarking optimization algorithms on synthetic experiments.
result Mitigates barriers in benchmarking optimization algorithms on realistic experimental scenarios.
Molecular machine learning has been maturing rapidly over the last few years. Improved methods and the presence of larger datasets have enabled machine learning algorithms to make increasingly accurate predictions about molecular properties. However, algorithmic progress has been limited due to the lack of a standard b…
New natural environment benchmarks for RL to improve robustness.
problem Current RL benchmarks lack complexity and real-world applicability.
method Developed three new RL benchmark domains with natural complexity.
result Proposed benchmarks improve generalization and algorithm robustness.
NATS-Bench benchmarks NAS algorithms for architecture topology and size.
problem Incomparable performance of NAS algorithms due to different search spaces and training setups.
method Unified benchmarking platform for architecture topology and size searching.
result Validated benchmark for 15,625 topology and 32,768 size candidates.
A new benchmark task for evaluating policy learning in complex, high-dimensional action spaces.
problem Lack of a commonly accepted benchmark for evaluating policy learning in hierarchical tasks with high-dimensional action spaces.
method Proposed DinerDash Gym benchmark and Decomposed Policy Graph Modelling (DPGM) algorithm.
result DPGM achieves significant improvement over baselines and effectively injects domain knowledge.
Deep learning benchmarks ECG analysis with strong performance.
problem Lack of appropriate datasets and evaluation procedures for ECG interpretation.
method Benchmarking on PTB-XL and ICBEB2018 datasets using convolutional neural networks.
result Convolutional neural networks, especially resnet- and inception-based architectures, outperform feature-based algorithms.
BAT benchmark for autobidding tasks in RTB auctions.
problem Lack of comprehensive datasets and benchmarks for autobidding.
method Developed a benchmark for two auction formats, implemented robust baselines.
result Provides a framework for developing and refining autobidding algorithms.
This work benchmarks diffusion model-based samplers for Bayesian inverse problems.
problem Optimizing diffusion models for uncertainty quantification in Bayesian inverse problems.
method Introduces three benchmark problems and a unified framework for diffusion model-based posterior sampling.
result Provides insights into strengths and limitations of diffusion model-based samplers.
We give an explicit formulaic algorithm and source code for building long-only benchmark portfolios and then using these benchmarks in long-only market outperformance strategies. The benchmarks (or the corresponding betas) do not involve any principal components, nor do they require iterations. Instead, we use a multif…
Benchpress streamlines benchmarking structure learning algorithms for probabilistic models.
problem Benchmarking structure learning algorithms for probabilistic graphical models.
method Snakemake workflow for scalable, reproducible benchmarks.
result Demonstrates applicability in five typical data scenarios.
We created financial benchmarks for distribution shifts in crude oil prices and volatility.
problem Scarcity of task-labeled time-series benchmarks in finance.
method Transformed asset price data into volatility proxies, generated task labels based on distribution shifts, and made datasets publicly available.
result Inclusion of task labels improves continual learning algorithms' performance on real-world data.
This work introduces benchmarks for evaluating nanophotonic structures in design simulations.
problem Design and understanding of nanophotonic structures for various applications.
method Development of frameworks and benchmarks for evaluating nanophotonic structures in parametric design problems.
result Strategic use of evaluation fidelity in enhancing structure designs.
This work benchmarks MARL algorithms in cooperative tasks.
problem Lack of evaluation tasks and criteria for comparing MARL algorithms.
method Systematic evaluation of three MARL algorithm classes in diverse cooperative tasks.
result Insights into the effectiveness of different learning approaches.
NAS-Bench-Suite simplifies NAS evaluation across diverse tasks.
problem Limited and inconsistent NAS benchmarks hinder research reproducibility.
method Developed a comprehensive, extensible NAS benchmark suite.
result Many NAS conclusions do not generalize across different benchmarks.
Efficiently predict LLM benchmarks using feature selection and regression.
problem Predicting full benchmark scores with minimal question subsets.
method Multiple regression with feature selection, using kernel ridge regression and mRMR.
result Improved prediction accuracy and ranking correlation across various benchmarks.
We study online prediction where regret of the algorithm is measured against a benchmark defined via evolving constraints. This framework captures online prediction on graphs, as well as other prediction problems with combinatorial structure. A key aspect here is that finding the optimal benchmark predictor (even in hi…
This paper benchmarks batch RL algorithms on Atari, finding DQN and partially-trained policies perform best.
problem Deep RL algorithms fail in batch setting.
method Benchmarked batch RL algorithms on Atari using a single partially-trained policy.
result Many batch RL algorithms underperform DQN and partially-trained policies.
Machine learning research depends on objectively interpretable, comparable, and reproducible algorithm benchmarks. We advocate the use of curated, comprehensive suites of machine learning tasks to standardize the setup, execution, and reporting of benchmarks. We enable this through software tools that help to create an…
This paper introduces a scalable benchmark for evaluating local posterior sampling in neural networks.
problem Degeneracy in neural network loss landscapes and its impact on SGMCMC algorithms.
method Development of a scalable benchmark for local posterior sampling.
result RMSProp-preconditioned SGLD is most effective at representing the local geometry of the posterior distribution.
MLaut automates machine learning benchmarking.
problem Benchmarking machine learning algorithms on diverse datasets.
method High-level workflow interface, database of datasets, scikit-learn and keras integration.
result Deep neural networks perform poorly on standard supervised learning tasks.
Benchmark assesses fairness in algorithmic uncertainty, revealing consistent and calibrated estimates improve fairness.
problem Challenges in managing uncertainty in fairness evaluations for predictive algorithms.
method Introduces FairlyUncertain, an axiomatic benchmark for evaluating uncertainty in fairness.
result Consistent and calibrated uncertainty estimates improve fairness without explicit fairness interventions.
CSTS benchmarks time series clustering by evaluating correlation structures.
problem Lack of validated ground truth for objectively assessing clustering quality.
method Synthetic benchmark CSTS for evaluating correlation structures in multivariate time series data.
result CSTS enables precise diagnosis of methodological limitations in correlation-based time series clustering.
FLBench automates federated learning benchmarking.
problem Manual dataset partitioning fails to simulate real-world isolated data islands.
method Develops a federated learning benchmark suite with three domains.
result Automates evaluation of federated learning systems and algorithms.
QCircuitBench provides a dataset for evaluating AI's ability to design quantum algorithms.
problem Lack of datasets for evaluating AI's capability in designing quantum algorithms.
method Developed a comprehensive benchmark dataset with 120,290 data points, including 25 algorithms and 3 task suites.
result LLMs exhibit consistent error patterns and fine-tuning does not always outperform few-shot learning.
A fundamental challenge in calcium imaging has been to infer the timing of action potentials from the measured noisy calcium fluorescence traces. We systematically evaluate a range of spike inference algorithms on a large benchmark dataset recorded from varying neural tissue (V1 and retina) using different calcium indi…
Paper benchmarks machine learning for detecting process curve drifts.
problem Detecting drifts in multivariate manufacturing process data.
method Synthetic data generation and evaluation score introduction.
result Existing algorithms often fail with complex drift scenarios.
This paper evaluates how different imputation methods affect predictive models.
problem The impact of different imputation methods on predictive models' performance.
method Systematic evaluation of various imputation methods for different data sets and machine learning algorithms.
result Recommendation of a general method for empirical benchmarking of imputation methods.
WOODS benchmarks improve understanding of time series OOD generalization.
problem Limited understanding of OOD generalization in time series.
method Presented eight open-source time series benchmarks and revised OOD algorithms.
result Large room for improvement in OOD generalization algorithms for time series.
Paper defines benchmarks for learning new tasks sequentially.
problem Efficient evaluation of continual few-shot learning.
method Theoretical framework and flexible benchmarks.
result Introduction of SlimageNet64 for efficient evaluation.