New IPC data set for graph learning tasks.
problem Benchmarking graph-based machine learning methods.
method Compilation from International Planning Competitions (IPC).
result Distinctly different characteristics from popular benchmarks.
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
169,341 papers · 148 categories
Trend · papers per month
48 results for “Benchmark data sets”
This paper surveys and benchmarks AutoML frameworks.
problem Reducing the need for data scientists in building machine learning applications.
method Survey and benchmark of popular AutoML frameworks on real data sets.
result Evaluation of 137 data sets from established AutoML benchmark suits.
New meta-score EPP interprets model performance differences.
problem Lack of interpretable benchmarks for model performance.
method Elo-based Predictive Power (EPP) meta-score, logistic regression.
result EPP scores have probabilistic interpretation and can be compared between data sets.
LEAF benchmarks federated learning challenges.
problem Federated learning challenges with scale and heterogeneity.
method Modular benchmarking framework with open-source datasets and implementations.
result Captures obstacles and intricacies of practical federated environments.
Unified benchmarks assess data poisoning and backdoor attacks.
problem Unclear danger and effectiveness of data poisoning methods.
method Developed standardized benchmarks for data poisoning and backdoor attacks.
result Existing methods may not generalize to realistic settings.
Generates synthetic data for benchmarking unsupervised outlier detection.
problem Difficulty in benchmarking unsupervised outlier detection due to rare and varied outliers in real data.
method Proposes a generic process to generate synthetic data with insightful characteristics.
result Demonstrates practicality of the generic process through a benchmark with state-of-the-art detection methods.
Paper defines benchmarks for learning new tasks sequentially.
problem Efficient evaluation of continual few-shot learning.
method Theoretical framework and flexible benchmarks.
result Introduction of SlimageNet64 for efficient evaluation.
The study initiates a theoretical analysis of dynamic benchmarking models.
problem Lack of theoretical foundation and empirical studies in dynamic benchmarks.
method Examined two realizations of dynamic benchmarking: sequential and hierarchical dependency models.
result Sequential dynamic benchmarks show initial performance improvement but can stall after three rounds due to label noise.
A Python tool generates synthetic data for cluster analysis from high-level descriptions.
problem Creating synthetic data for cluster analysis is laborious and requires detailed geometric parameters.
method Proposes natural language-based synthetic data generation and implements it in a Python package.
result Makes it easy to set up interpretable and reproducible benchmarks for cluster analysis.
New benchmarks for offline RL from diverse datasets.
problem Measuring progress in offline RL due to lack of suitable benchmarks.
method Developed benchmarks tailored for offline RL, focusing on diverse dataset properties.
result Revealed deficiencies in existing offline RL algorithms.
This paper benchmarks batch RL algorithms on Atari, finding DQN and partially-trained policies perform best.
problem Deep RL algorithms fail in batch setting.
method Benchmarked batch RL algorithms on Atari using a single partially-trained policy.
result Many batch RL algorithms underperform DQN and partially-trained policies.
This paper compares and evaluates methods for evaluating statistical models using benchmarking data and simulations.
problem Choosing between benchmarking data sets and simulation studies for method comparison studies.
method Borrowing ideas from mixed methods research and Clinical Scenario Evaluation, the paper investigates and develops new approaches to evaluate methods.
result Develops new approaches to evaluate methods by combining the strengths of benchmarking data sets and simulation studies.
Paper introduces OARF benchmark suite for federated learning systems.
problem Limited diversity in federated learning benchmarks.
method Characterizes OARF benchmark suite with diverse data and applications.
result Federated learning can effectively increase end-to-end throughput.
SurvHTE-Bench benchmarks HTE estimation in survival analysis with diverse datasets.
problem Challenges in estimating HTEs from right-censored survival data.
method Modular synthetic datasets, semi-synthetic datasets, and real-world datasets.
result First rigorous comparison of survival HTE methods under diverse conditions.
The study creates benchmarks for clinical time series data to evaluate machine learning models.
problem Lack of publicly available benchmark data sets for healthcare research.
method Proposed four clinical prediction benchmarks using MIMIC-III data, evaluated various deep supervision and multitask training methods.
result Demonstrated the effectiveness of deep supervision, multitask training, and data-specific architectural modifications on neural models.
FLBench automates federated learning benchmarking.
problem Manual dataset partitioning fails to simulate real-world isolated data islands.
method Develops a federated learning benchmark suite with three domains.
result Automates evaluation of federated learning systems and algorithms.
We provide a benchmark dataset for hand gesture recognition using force myography.
problem Lack of publicly available benchmark data for force myography hand gesture recognition.
method Collected data from 20 persons covering 18 unique gestures using a commercially available sensor setup.
result Improved gesture recognition accuracy through transfer learning.
Study proposes new methods to calculate probabilistic benchmarks in noisy data.
problem Identifying opportunities for improvement in comparable units with noisy data.
method 2-step methodology involving undersampling and relevance vector machine.
result Higher discrimination power achieved with macro-economic environment variables.
GeneDisco benchmarks experimental design for drug discovery.
problem Vast experimental design space in drug discovery.
method Machine learning for optimal experimental design.
result Standardised benchmark suite for active learning.
Fashion-MNIST replaces MNIST for machine learning benchmarks.
problem No new problem introduced.
method No new method introduced.
result Fashion-MNIST serves as a direct replacement for MNIST.
A new sparse benchmark metabench identifies key abilities from large benchmarks.
problem Redundancy and compression in existing benchmarks.
method Data from 5000+ LLMs to identify most informative items, distilling a sparse benchmark.
result Sparse benchmark metabench captures underlying abilities with high accuracy.
A framework and benchmark for deep batch active learning in neural networks.
problem Efficiently acquiring labels for neural network regression.
method Framework of base kernels, transformations, and selection methods; use of sketched finite-width neural tangent kernels and clustering.
result Proposed method outperforms state-of-the-art on benchmark, scales to large data sets.
Paper benchmarks real-world noisy labels and proposes a method to improve deep learning performance.
problem Understanding deep learning with real-world noisy labels.
method Develops a simple method to handle both synthetic and real noisy labels.
result Method achieves best results on benchmark datasets and web noise.
Method for creating synthetic multi-fidelity data sets.
problem Lack of representative synthetic datasets for multifidelity optimisation benchmarks.
method Systematic generation of synthetic fidelities from preexisting datasets.
result Allows systematic investigation of lower fidelity proxies' influence.
New method tackles RL with observational data, confounders.
problem Learning good policies from historical data with unobserved confounders.
method Extends Actor-Critic method to deconfounding variant.
result Proposed algorithms outperform traditional RL methods in confounded environments.
New method improves OSSL by learning from all unlabeled data.
problem Handling open-set semi-supervised learning with unknown classes.
method Self-supervision and energy-based score for all unlabeled data.
result State-of-the-art results on benchmark problems.
This paper evaluates how different imputation methods affect predictive models.
problem The impact of different imputation methods on predictive models' performance.
method Systematic evaluation of various imputation methods for different data sets and machine learning algorithms.
result Recommendation of a general method for empirical benchmarking of imputation methods.
Similar models predict similarly, reducing overfitting risk.
problem Excessive reuse of test data in machine learning.
method Proved model similarity mitigates overfitting and provided a generalization bound.
result Model similarity reduces the risk of overfitting, even when accuracy levels suggest otherwise.
A fundamental challenge in calcium imaging has been to infer the timing of action potentials from the measured noisy calcium fluorescence traces. We systematically evaluate a range of spike inference algorithms on a large benchmark dataset recorded from varying neural tissue (V1 and retina) using different calcium indi…
New benchmark for non-rigid 3D human shape retrieval.
problem Distinguishing between body shapes of 3D human models.
method Extended benchmark with 145 new models and FAUST dataset.
result Improved comparison of 25 shape retrieval methods.
A new benchmark system evaluates MCMC samplers using real data.
problem The evaluation of new MCMC samplers is inadequate with common methods.
method Meta-learning approach to generate benchmark examples from data sets and models, using flexible density models.
result New insights into effective sample size and estimation efficiency of samplers.
CausalRivers benchmarks causal discovery methods on real-world river discharge data.
problem Lack of in-the-wild evaluation of causal discovery methods on complex, real-world data.
method Introduces CausalRivers, a large-scale dataset of river discharge data for benchmarking.
result Demonstrates the utility of CausalRivers in evaluating causal discovery methods.
This study benchmarks data augmentation schemes to improve CNN performance.
problem Lack of training data for deep learning models.
method Various geometric and photometric data augmentation schemes evaluated on a CNN.
result Cropping in geometric augmentation significantly improves CNN task performance.
We present a novel approach to learn binary classifiers when only positive and unlabeled instances are available (PU learning). This problem is routinely cast as a supervised task with label noise in the negative set. We use an ensemble of SVM models trained on bootstrap resamples of the training data for increased rob…
FinStressTS creates synthetic benchmarks for financial forecasting, revealing model weaknesses.
problem Limited failure attribution in real-world financial benchmarks.
method Synthetic benchmark with 30 diagnostic environments linked to six mechanism families.
result Model performance varies by mechanism type, with autoregressive models often outperforming Transformers.
Efficiently predict LLM benchmarks using feature selection and regression.
problem Predicting full benchmark scores with minimal question subsets.
method Multiple regression with feature selection, using kernel ridge regression and mRMR.
result Improved prediction accuracy and ranking correlation across various benchmarks.
New benchmarks measure image generation models' ability to generalize beyond training data.
problem Trivially memorizing training data yields better scores than state-of-the-art models on current benchmarks.
method Developed neural network divergences (NNDs) as evaluation metrics requiring large samples.
result Implemented and validated a black-box metric that measures diversity, sample quality, and generalization.
MLaut automates machine learning benchmarking.
problem Benchmarking machine learning algorithms on diverse datasets.
method High-level workflow interface, database of datasets, scikit-learn and keras integration.
result Deep neural networks perform poorly on standard supervised learning tasks.
Adaptive robust strategy improves online portfolio selection by managing market trends and costs.
problem Optimizing sequential investment decisions in volatile markets.
method Robust optimization with adaptive parameter adjustment.
result Adaptive scheme outperforms existing strategies in cumulative returns and Sharpe ratios.
L2O uses machine learning to design optimization methods.
problem Designing efficient optimization methods for specific problem distributions.
method Data-driven approach to automate optimization method design.
result L2O methods are practical for specific problem distributions but fail on out-of-distribution problems.
AutoML benchmarks compare different frameworks.
problem No objective comparison of AutoML frameworks exists.
method Benchmarked four open-source AutoML solutions using open-source datasets.
result auto-sklearn outperformed TPOT in classification and regression tasks respectively.
Study characterizes harmful low-fidelity data sources for surrogate models.
problem Identifying which low-fidelity data sources to use in constructing surrogate models.
method Employed benchmark filtering techniques to assess harmful sources using limited data.
result Provided guidelines for using low-fidelity sources in an industrial setting.
New protocol benchmarks deep learning methods' data efficiency.
problem Measuring data efficiency of deep learning methods.
method Proposed experimental protocol to benchmark CNNs and HiGSFA.
result HiGSFA outperforms CNNs on smaller datasets.
This work benchmarks and theorizes robust NAS under adversarial training.
problem Lack of benchmark evaluations and theoretical guarantees for robust NAS architectures under adversarial training.
method Released a comprehensive data set and established a generalization theory using the neural tangent kernel.
result Established a generalization theory for robust NAS architectures under adversarial training.
OGB provides diverse graph datasets for robust ML research.
problem Challenges in scalable and robust graph machine learning.
method Unified evaluation protocol, diverse datasets, and automated pipeline.
result Significant scalability and generalization challenges identified.
Study benchmarks 19 survival models on 34 datasets, finding Cox model still best.
problem Quantitative comparison of survival models on low-dimensional data.
method Comprehensive benchmarking of 19 models on 34 datasets, tuning and evaluating using 6 metrics.
result Cox Proportional Hazards model remains best overall for low-dimensional, right-censored data.
New algorithms improve causal graph discovery with adaptive interventions, even under worst-case interventional costs.
problem Discover causal relationships from data with adaptive interventions and node-dependent costs.
method Define new benchmarks and provide adaptive search algorithms for causal graph discovery.
result Logarithmic approximations achieved under various settings: atomic, bounded size interventions and generalized cost objectives.
Benchmarking Minimax Linkagestat.ML
Minimax linkage improves clustering interpretability by minimizing maximum distance to prototypes.
problem Improving clustering interpretability and performance.
method Uses distances to prototypes for cluster formation, evaluated on multiple metrics.
result Minimax linkage often produces the smallest maximum minimax radius, but not always best across all metrics.