Wiki-CS dataset benchmarks Graph Neural Networks using Wikipedia articles.
problem Benchmarking Graph Neural Networks on a new domain with structural differences.
method Derived from Wikipedia, nodes represent Computer Science articles, edges from hyperlinks, 10 classes for different branches, evaluated semi-supervised node classification and link prediction.
result Graph Neural Networks perform well on Wiki-CS, showing structural differences from earlier benchmarks.
New benchmarks for offline RL from diverse datasets.
problem Measuring progress in offline RL due to lack of suitable benchmarks.
method Developed benchmarks tailored for offline RL, focusing on diverse dataset properties.
result Revealed deficiencies in existing offline RL algorithms.
Study introduces a benchmark suite for evaluating neural MI estimators on real-world unstructured datasets.
problem Lack of comprehensive evaluation methods for neural MI estimators on real-world unstructured datasets.
method Developed a benchmark suite using same-class sampling and a binary symmetric channel trick.
result Showed accurate manipulation of true MI values of real-world datasets.
Benchmark study evaluates 8 clustering methods on 99 UCR time series datasets.
problem Assessing the performance of clustering methods on time series data.
method Examines 8 clustering methods across 3 categories and 3 distance measures on 99 UCR datasets.
result Provides a comprehensive dataset-level assessment of clustering methods.
MNIST-Nd offers synthetic datasets to benchmark clustering across dimensions.
problem Clustering performance degrades with high-dimensional data.
method Training mixture variational autoencoders on MNIST to create synthetic datasets with varying latent dimensions.
result Leiden clustering algorithm is most robust as dimensionality grows.
Framework generates multimodal datasets with known MI for benchmarking.
problem Benchmarking mutual information estimators and SSL techniques.
method Flow-based generative model and structured causal framework.
result Regression performance improves with increasing MI between modalities.
TUDataset provides benchmark datasets for graph learning.
problem Lack of meaningful benchmark datasets and standardized evaluation procedures for graph learning.
method Collection and standardization of 120 graph datasets from various applications.
result Standardized evaluation procedures and baseline experiments provided.
OGB provides diverse graph datasets for robust ML research.
problem Challenges in scalable and robust graph machine learning.
method Unified evaluation protocol, diverse datasets, and automated pipeline.
result Significant scalability and generalization challenges identified.
Develops a framework for consistent clustering algorithm benchmarking.
problem Lack of consistent benchmarking and diverse datasets for clustering algorithms.
method Consistent methodology, aggregated and polished datasets, interactive explorer.
result Introduces a comprehensive framework for testing clustering algorithms.
Enhances graph classification models on small datasets.
problem Over-fitting and undergeneralization on small-scale benchmark datasets.
method Data augmentation via graph structure transformation and model evolution framework.
result Average improvement of 3 - 13% accuracy on graph classification tasks.
Natural experiment dataset reveals inconsistent treatment effect estimators.
problem Inconsistent results from over 20 estimators on a new dataset.
method Created a benchmark to evaluate estimator accuracy, derived variance formula, introduced new estimator.
result Doubly robust estimators outperform others by orders of magnitude.
In this paper, we propose a novel method for generating a synthetic dataset obeying Gaussian distribution. Compared to the commonly used benchmark datasets with unknown distribution, the synthetic dataset has an explicit distribution, i.e., Gaussian distribution. Meanwhile, it has the same characteristics as the benchm…
GIFT-Eval benchmarks time series forecasting models across diverse datasets.
problem Lack of comprehensive benchmarks for evaluating time series foundation models.
method Developed GIFT-Eval, a benchmark with 23 datasets, 177 million data points, and 144,000 time series.
result Promotes evaluation of foundation models across various domains and frequencies.
FLAIR dataset for federated learning benchmarks.
problem Lack of suitable federated learning datasets.
method Curated large-scale annotated image dataset for multi-label classification.
result FLAIR captures real-world federated learning challenges.
New benchmark for EEG-eye movement reconstruction from functional data.
problem Reconstructing eye movements from EEG data.
method Functional neural networks and open challenges for evaluation.
result Baseline results for consumer-grade and research-grade hardware.
Paper compiles ML algorithm performance benchmarks on OpenML datasets.
problem Finding optimal hyperparameters for ML algorithms efficiently.
method Generated benchmark data for 7 ML algorithms on 39 datasets, fixed hyperparameters before testing.
result Comprehensive dataset of ML algorithm performance sensitivity.
DIGEN benchmark provides synthetic datasets for ML algorithm evaluation.
problem Understanding and comparing machine learning algorithms' performance.
method Synthetic datasets generated using 40 mathematical functions to evaluate machine learning algorithms.
result DIGEN resource facilitates understanding why algorithms perform poorly and provides ideas for improvement.
Paper introduces a fraud detection dataset benchmark.
problem Unique challenges in fraud detection datasets.
method Compilation of publicly available fraud datasets.
result Demonstrates applications of the Fraud Dataset Benchmark.
The paper introduces a framework to select efficient datasets for preserving model rankings.
problem Efficient evaluation of machine learning models on small, representative datasets.
method Bootstrap aggregation, clustering, design criteria, random baselines, and greedy farthest-first (FAFI).
result Several selection strategies improve rank preservation compared to random subsets, especially in time series classification.
We provide a benchmark dataset for hand gesture recognition using force myography.
problem Lack of publicly available benchmark data for force myography hand gesture recognition.
method Collected data from 20 persons covering 18 unique gestures using a commercially available sensor setup.
result Improved gesture recognition accuracy through transfer learning.
New benchmarks for RNA 3D structure-function modeling.
problem Lack of standardized benchmarks for RNA deep learning.
method Developed seven benchmark datasets, provided tools for data handling, and offered a user-friendly environment for model comparison.
result Demonstrated utility with baseline results using a relational graph neural network.
BREEDS benchmarks assess model robustness to subpopulation shifts.
problem Measuring model robustness to novel subpopulation shifts.
method Controlled synthesis of realistic distribution shifts using class structure.
result Validated model sensitivity and effectiveness of robustness interventions.
Progress of machine learning in critical care has been difficult to track, in part due to absence of public benchmarks. Other fields of research (such as computer vision and natural language processing) have established various competitions and public benchmarks. Recent availability of large clinical datasets has enabl…
New IRT method identifies useful datasets for ML classifier evaluation.
problem Lack of standard evaluation strategy for ML benchmarks.
method Applied Item Response Theory (IRT) to OpenML-CC18 benchmark.
result Not all datasets are useful for evaluating classifiers.
TimeGraph creates synthetic datasets for robust time-series causal discovery.
problem Lack of reliable synthetic benchmark datasets for robust time-series causal discovery.
method Developed comprehensive synthetic datasets with temporal properties, including trends, seasonality, and noise.
result Demonstrated significant variations in algorithm performance under realistic temporal conditions.
We report a series of robust empirical observations, demonstrating that deep Neural Networks learn the examples in both the training and test sets in a similar order. This phenomenon is observed in all the commonly used benchmarks we evaluated, including many image classification benchmarks, and one text classification…
Gradient matching method improves domain generalization across various datasets.
problem Machine learning's inability to generalize to unseen domains.
method Inter-domain gradient matching objective and first-order algorithm Fish.
result Fish method produces competitive results and surpasses baselines on 4 datasets.
A new large-scale tabular benchmark for Learning from Label Proportions.
problem Lack of a large-scale open benchmark for tabular Learning from Label Proportions.
method Proposed LLP-Bench, a suite of 70 datasets (62 feature bag and 8 random bag) from real-world tabular data.
result Demonstrated the effectiveness of 9 SOTA and popular tabular LLP techniques on 62 feature bag datasets.
Study links different drug property prediction methods and datasets.
problem Unclear which method is best for drug property prediction.
method Empirical study linking multiple datasets and methods.
result Best method depends on dataset, classical methods often outperform deep learning.
The study examines dataset usage patterns in machine learning research.
problem Lack of attention to dataset dynamics in machine learning research.
method Analysis of dataset usage patterns across machine learning subcommunities and time periods (2015-2020).
result Increasing concentration on fewer and fewer datasets, significant adoption from other tasks, and concentration across the field on datasets introduced by elite institutions.
New dataset abla2DFT for drug-like molecules benchmarks neural network potentials.
problem Lack of large, diverse datasets for training neural network potentials in quantum chemistry.
method Developed a new dataset abla2DFT containing energies, forces, and molecular properties for drug-like molecules. result First dataset with relaxation trajectories for drug-like molecules.
We created financial benchmarks for distribution shifts in crude oil prices and volatility.
problem Scarcity of task-labeled time-series benchmarks in finance.
method Transformed asset price data into volatility proxies, generated task labels based on distribution shifts, and made datasets publicly available.
result Inclusion of task labels improves continual learning algorithms' performance on real-world data.
UCFE benchmarks LLMs in financial tasks with human feedback.
problem Evaluating LLMs' financial task performance and user satisfaction.
method Hybrid approach combining human expert evaluations and dynamic interactions.
result Significant alignment between benchmark scores and human preferences (Pearson correlation coefficient of 0.78).
New framework assesses graph-learning datasets for better evaluation.
problem Insufficient evaluation of graph-learning datasets and methods.
method Introduces Rings framework for dataset ablations and proposes performance separability and mode complementarity measures.
result Demonstrates utility of Rings framework for graph-learning dataset evaluation.
Molecular machine learning has been maturing rapidly over the last few years. Improved methods and the presence of larger datasets have enabled machine learning algorithms to make increasingly accurate predictions about molecular properties. However, algorithmic progress has been limited due to the lack of a standard b…
SurvHTE-Bench benchmarks HTE estimation in survival analysis with diverse datasets.
problem Challenges in estimating HTEs from right-censored survival data.
method Modular synthetic datasets, semi-synthetic datasets, and real-world datasets.
result First rigorous comparison of survival HTE methods under diverse conditions.
Graph learning is often unnecessary for common benchmarks, as node features can suffice.
problem The necessity of graph learning in common graph benchmarks is often assumed.
method We compared graph learning to feature-only models on seven datasets and found that graph structure often adds little to performance.
result Node features can often suffice for common graph benchmarks, challenging the orthodoxy.
OpenDataVal benchmarks data valuation algorithms for diverse datasets.
problem Improving model performance and mitigating biases in training datasets.
method Unified benchmark framework for data valuation algorithms.
result No single algorithm performs uniformly best across all tasks.
Deep learning benchmarks ECG analysis with strong performance.
problem Lack of appropriate datasets and evaluation procedures for ECG interpretation.
method Benchmarking on PTB-XL and ICBEB2018 datasets using convolutional neural networks.
result Convolutional neural networks, especially resnet- and inception-based architectures, outperform feature-based algorithms.
Temporal information impacts only a fraction of time series datasets, skewing benchmark evaluations.
problem Temporal information's impact on time series classification is often overestimated.
method Permutation tests on UCR archive to identify datasets where temporal info is irrelevant.
result Many tabular datasets perform well without temporal info, skewing benchmark evaluations.
We introduce a new molecular dataset, named Alchemy, for developing machine learning models useful in chemistry and material science. As of June 20th 2019, the dataset comprises of 12 quantum mechanical properties of 119,487 organic molecules with up to 14 heavy atoms, sampled from the GDB MedChem database. The Alchemy…
Paper addresses challenges in benchmarking stream learning algorithms with real-world data.
problem Lack of publicly available non-stationary real-world datasets for evaluating stream algorithms.
method Proposes a new public data repository for benchmarking stream algorithms with real-world data.
result Mitigates problems related to dataset choice in experimental evaluation of stream classifiers and drift detectors.
BAT benchmark for autobidding tasks in RTB auctions.
problem Lack of comprehensive datasets and benchmarks for autobidding.
method Developed a benchmark for two auction formats, implemented robust baselines.
result Provides a framework for developing and refining autobidding algorithms.
WRENCH benchmarks weak supervision datasets for machine learning.
problem Lack of standardized evaluation for weak supervision datasets.
method Developed a comprehensive benchmark platform WRENCH.
result Demonstrated the efficacy of WRENCH as a benchmark platform.
Paper proposes a new dataset for group anomaly detection in physics.
problem Group anomaly detection in physics.
method Development of a synthetic benchmark dataset and comparison of anomaly detection techniques.
result Demonstrated performance of anomaly detection techniques on the new dataset.
PerturBench benchmarks ML models for cellular perturbation analysis.
problem Standardizing benchmarking in modeling single cell transcriptomic responses to perturbations.
method Modular platform, diverse datasets, metrics, extensive evaluation, rank metrics.
result Simpler models are competitive and scale well with larger datasets.
The recent breakthroughs in computer vision have benefited from the availability of large representative datasets (e.g. ImageNet and COCO) for training. Yet, robotic vision poses unique challenges for applying visual algorithms developed from these standard computer vision datasets due to their implicit assumption over…
This work benchmarks counterfactual methods in time series classification.
problem Lack of benchmarking studies for counterfactual methods in time series classification.
method Redesign metrics for sparsity, plausibility, and consistency; systematically benchmark 6 CF methods on 30 datasets.
result Performance of CF methods varies across metrics and models.