Benchmarked over 70 graph clustering algorithms.
problem Lack of comprehensive performance comparison for graph clustering algorithms.
method Evaluated 70+ graph clustering programs for runtime and quality on weighted and unweighted graphs, analyzed ground truth characteristics.
result Supply a start point for engineers and viewpoint for researchers.
Open-source benchmark for AutoML systems.
problem Difficulty in comparing AutoML systems.
method Open-source benchmark framework with public datasets and results.
result Compared 4 AutoML systems across 39 datasets.
This study provides benchmarks for different implementations of LSTM units between the deep learning frameworks PyTorch, TensorFlow, Lasagne and Keras. The comparison includes cuDNN LSTMs, fused LSTM variants and less optimized, but more flexible LSTM implementations. The benchmarks reflect two typical scenarios for au…
FedML aims to improve FL research by providing a library and benchmark.
problem Inconsistent FL algorithm development and performance comparison.
method FedML offers an open research library and benchmark supporting diverse computing paradigms and flexible API design.
result FedML facilitates fair algorithm comparison and development in federated learning.
New method compares classifiers using GSD-front, addressing statistical uncertainty and robustness.
problem Comparing classifiers with multiple quality metrics and statistical uncertainty.
method Proposes GSD-front and statistical tests for robust comparisons.
result Reliable method for comparing classifiers with statistical uncertainty and robustness.
This paper compares and evaluates methods for evaluating statistical models using benchmarking data and simulations.
problem Choosing between benchmarking data sets and simulation studies for method comparison studies.
method Borrowing ideas from mixed methods research and Clinical Scenario Evaluation, the paper investigates and develops new approaches to evaluate methods.
result Develops new approaches to evaluate methods by combining the strengths of benchmarking data sets and simulation studies.
Benchmark for DL inference on embedded HWAs, focusing on autonomous driving.
problem Lack of comprehensive benchmarks for DL hardware.
method Developed a benchmark for inference on embedded HWAs, focusing on autonomous driving. Proposed new granularity, benchmark procedures, and performance indicators.
result Identifies mismatches between HWAs and DL models.
New benchmark evaluates BDL methods in medical retinopathy diagnosis.
problem Evaluate robustness and scalability of BDL methods in medical applications.
method Developed a new benchmark with real-world diabetic retinopathy tasks.
result Some BDL techniques overfit uncertainty to datasets, underperforming on new benchmark.
This work benchmarks MARL algorithms in cooperative tasks.
problem Lack of evaluation tasks and criteria for comparing MARL algorithms.
method Systematic evaluation of three MARL algorithm classes in diverse cooperative tasks.
result Insights into the effectiveness of different learning approaches.
Introduces AMLB, an open benchmark for AutoML frameworks.
problem Challenges in comparing AutoML frameworks.
method Open benchmark with 9 AutoML frameworks, 71 classification, 33 regression tasks, multi-faceted analysis, Bradley-Terry trees.
result Differences in AutoML frameworks' performance and trade-offs.
Study benchmarks 19 survival models on 34 datasets, finding Cox model still best.
problem Quantitative comparison of survival models on low-dimensional data.
method Comprehensive benchmarking of 19 models on 34 datasets, tuning and evaluating using 6 metrics.
result Cox Proportional Hazards model remains best overall for low-dimensional, right-censored data.
New depth function for partial orders helps compare machine learning algorithms.
problem Comparing machine learning algorithms using non-standard data types.
method Adapted simplicial depth to partial orders, using ufg depth for comparison.
result Demonstrates promising variety of analysis approaches based on ufg methods.
New model reveals significant impact of data and parameter variations on machine learning benchmarks.
problem Variance in machine learning benchmarks due to data sampling, initialization, and hyperparameters.
method Modeling the benchmarking process, analyzing comparison methods, and studying error rates.
result Adding more sources of variation reduces compute cost while improving performance.
Foundation models improve on econometric benchmarks for forecasting volatility, but vary widely across models.
problem Comparing pretrained time series foundation models to econometric benchmarks for volatility forecasting.
method Systematic comparison of nine zero-shot TSFMs against eight econometric specifications on 50 assets across 3 markets and 3 horizons.
result Tiny Time Mixers (TTM) is the only model that consistently beats the Log-HAR benchmark, but performance varies widely across models.
ROVAE uses noisy pairwise comparisons to disentangle factors in VAEs.
problem Disentangling factors in VAEs requires an inductive bias.
method Robust Ordinal VAE (ROVAE) incorporates noisy pairwise ordinal comparisons to disentangle factors.
result ROVAE outperforms existing methods and is more robust to noisy comparisons.
OceanForecastBench offers a comprehensive benchmark for data-driven ocean forecasting models.
problem Lack of open-source, standardized benchmarks for data-driven ocean forecasting models.
method Proposes OceanForecastBench, a benchmark with high-quality data and evaluation pipeline.
result Offers the most comprehensive benchmarking framework for data-driven ocean forecasting.
Paper introduces an efficient comparison operator for robust multi-objective optimization with uncertain objectives.
problem Optimizing with uncertain objectives in multi-objective problems.
method Empirical approach to compare solutions with arbitrary distributions of uncertain objectives.
result Higher optimization quality achieved at lower overheads compared to existing techniques.
A scalable DL benchmarking platform for evaluating and comparing models, frameworks, and hardware.
problem Lack of a uniform DL benchmarking platform for evaluating and comparing innovations.
method Identified 10 design features for a DL benchmarking platform, proposed MLModelScope, and implemented as an open-source project.
result Demonstrated how model, hardware, and framework selection affect accuracy and performance under different scenarios.
In this paper, we propose an active learning algorithm and models which can gradually learn individual's preference through pairwise comparisons. The active learning scheme aims at finding individual's most preferred choice with minimized number of pairwise comparisons. The pairwise comparisons are encoded into probabi…
Recent anomaly detection benchmarks are flawed, potentially misleading progress.
problem Flawed benchmark datasets create misleading progress reports.
method Identified four flaws in benchmark datasets and introduced a new archive.
result Published comparisons may be unreliable due to flaws in benchmark datasets.
Benchmarking graph neural networks for diverse datasets.
problem Quantifying progress in graph neural networks.
method Diverse graph collection, fair model comparison, open-source framework.
result Benchmark framework has spurred interest in graph positional encoding.
This study analyzes NAS benchmarks and finds that only a subset of operations is crucial for generating high-performing architectures.
problem NAS benchmarks lack generability and provide skewed performance distributions, leading to unreliable comparisons.
method Empirical analysis of widely used NAS benchmarks (101, 201, TransNAS-Bench-101) focusing on operation importance and generability.
result Only a subset of operations is necessary to generate architectures close to the upper-bound performance range, and convolution layers have the highest impact.
Study compares 29 emulators across 60 test functions and 40 datasets.
problem Comparing the strengths and weaknesses of different emulators.
method Large-scale, fully reproducible comparison using R package duqling.
result Detailed empirical insights into emulator strengths and weaknesses.
Bayesian framework for comparing trading algorithms using cost analysis.
problem Comparing trading algorithms using cost analysis.
method Bayesian framework, hierarchical models, standardized benchmarks, fat tails, skewness, heteroscedasticity.
result Effective calculation of trading benchmarks with limited data.
This work compares data reduction criteria for online Gaussian Processes.
problem The computational complexity of Gaussian Processes limits their applicability to small datasets and streaming scenarios.
method Unified comparison of several data reduction criteria, analyzing computational complexity and reduction behavior.
result Practical guidelines for choosing a suitable data reduction criterion for online Gaussian Processes.
New benchmarks for RNA 3D structure-function modeling.
problem Lack of standardized benchmarks for RNA deep learning.
method Developed seven benchmark datasets, provided tools for data handling, and offered a user-friendly environment for model comparison.
result Demonstrated utility with baseline results using a relational graph neural network.
Benchmarking TPU, GPU, and CPU for deep learning models.
problem Improving performance in deep learning training.
method ParaDnn benchmark suite for FC, CNN, and RNN models on TPU, GPU, and CPU.
result TPU, GPU, and CPU have unique strengths for different types of models.
Evaluates six ETSC algorithms on various datasets.
problem Comparing ETSC algorithms for different data characteristics.
method Empirical comparison of six ETSC algorithms on public and new datasets.
result Provides insights into algorithm performance on real-life applications.
WeatherBench provides a dataset and metrics for comparing data-driven weather forecasts.
problem Lack of a common dataset and evaluation metrics for data-driven weather forecasting.
method Publicly available dataset derived from ERA5, simple evaluation metrics.
result Baseline scores from various forecasting methods provided for comparison.
Paper creates benchmarks for neural hyperparameter search.
problem Difficulty in comparing HPO methods due to high computational costs.
method Developed benchmarks for a feed forward neural network on four regression datasets.
result Exhaustive comparison of HPO methods on the benchmarks.
The paper examines how optimizer comparisons in deep learning are influenced by hyperparameter tuning.
problem The sensitivity of optimizer comparisons to hyperparameter tuning protocols.
method Empirical comparisons of optimizers with and without varying hyperparameter search spaces.
result Inclusion relationships between optimizers matter in practice and can contradict recent empirical comparisons.
New benchmark for non-rigid 3D human shape retrieval.
problem Distinguishing between body shapes of 3D human models.
method Extended benchmark with 145 new models and FAUST dataset.
result Improved comparison of 25 shape retrieval methods.
Paper quantifies uncertainty in pairwise comparison models.
problem Uncertainty quantification in sparse Bradley-Terry-Luce models.
method Unified proof strategy for MLE and spectral estimator.
result Sharp and uniform non-asymptotic expansions for estimators.
Many deep models have been recently proposed for anomaly detection. This paper presents comparison of selected generative deep models and classical anomaly detection methods on an extensive number of non--image benchmark datasets. We provide statistical comparison of the selected models, in many configurations, archite…
New method uses model comparison signals to improve LLM evaluation accuracy.
problem Limited benchmark sizes and model stochasticity in evaluating LLMs' mathematical reasoning.
method Combines standard labeled outcomes with model comparison signals to design a statistically efficient evaluation framework.
result Semiparametric estimator achieves the semiparametric efficiency bound and substantially improves ranking accuracy.
SEMASIA provides a large dataset of latent representations for model comparison.
problem Difficulty in comparing semantic structures across different neural network models.
method Collection of latent representations from 1700 pretrained models across various benchmarks.
result Consistent semantic organization across models and datasets.
Paper addresses challenges in benchmarking stream learning algorithms with real-world data.
problem Lack of publicly available non-stationary real-world datasets for evaluating stream algorithms.
method Proposes a new public data repository for benchmarking stream algorithms with real-world data.
result Mitigates problems related to dataset choice in experimental evaluation of stream classifiers and drift detectors.
The paper develops a method to estimate consumer preferences from observed rankings.
problem Estimating consumer preferences from partial ranking information.
method Interpreting observed rankings as pairwise comparisons, modeling latent utility, and correcting for selection bias.
result The method improves recommendation performance, especially for previously unconsumed products.
ECG improves graph clustering and resolves resolution limit issues.
problem Graph clustering resolution limit issue.
method ECG uses consensus clustering to improve graph clustering.
result ECG alleviates the resolution limit issue and improves partition stability.
AutoML serves as the bridge between varying levels of expertise when designing machine learning systems and expedites the data science process. A wide range of techniques is taken to address this, however there does not exist an objective comparison of these techniques. We present a benchmark of current open source Aut…
While current benchmark reinforcement learning (RL) tasks have been useful to drive progress in the field, they are in many ways poor substitutes for learning with real-world data. By testing increasingly complex RL algorithms on low-complexity simulation environments, we often end up with brittle RL policies that gene…
New meta-score EPP interprets model performance differences.
problem Lack of interpretable benchmarks for model performance.
method Elo-based Predictive Power (EPP) meta-score, logistic regression.
result EPP scores have probabilistic interpretation and can be compared between data sets.
Enhances portfolio optimization under uncertainty using robust multi-objective methods.
problem Uncertainties in real-world portfolio optimization scenarios.
method Robust multi-objective optimization with benchmark comparisons.
result More reliable and adaptable portfolio strategies for market uncertainties.
Public dataset for benchmarking deep learning CT reconstruction methods.
problem Lack of a fair benchmark for comparing deep learning CT reconstruction methods.
method Processed and simulated over 40,000 CT scan slices from the LIDC/IDRI Database.
result First baseline results provided for comparison.
We provide a benchmark dataset for hand gesture recognition using force myography.
problem Lack of publicly available benchmark data for force myography hand gesture recognition.
method Collected data from 20 persons covering 18 unique gestures using a commercially available sensor setup.
result Improved gesture recognition accuracy through transfer learning.
Self-consistency improves the accuracy of model comparison methods.
problem Improving the accuracy of model comparison methods when simulation models are misspecified.
method Supplement traditional simulation-based training with a self-consistency loss on unlabeled real data.
result Self-consistency training improves model comparison accuracy, especially in open-world scenarios.
It is common that a trained classification model is applied to the operating data that is deviated from the training data because of noise. This paper demonstrates that an ensemble classifier, Diversified Multiple Tree (DMT), is more robust in classifying noisy data than other widely used ensemble methods. DMT is teste…
Benchmark evaluates financial misinformation detection models, revealing weaknesses without external context.
problem Detecting financial misinformation without external references.
method RFC Bench at paragraph level, two tasks: reference-free detection and comparison-based diagnosis.
result Performance improves with comparative context, revealing model weaknesses in reference-free settings.